Methodology

 

Download Cookbook

Testimonials video

Play Testimonials

Want to learn more?

Find more case details and research results in our member area

Assessment Methodology

In business, performance of an organisation is measured by key performance indicators (KPIs). KPIs are different from business to business and need to be identified first. Due to the fact that CC&TH is still in its infancy, CC&TH KPIs do not exist yet. The ACT project takes lead in identifying CC&TH KPIs, and more importantly, the drivers for those KPIs. This chapter describes the evaluation framework that captures the KPIs (outcomes) that have been assessed throughout the ACT programme.  

In this section we describe the method for data collection of the baseline assessment and iterative assessments during the ACT programme. The steps are summarised in the figure below.?

Indicator Framework

The Evaluation Framework refers to a holistic approach for assessing qualitative and quantitative effectiveness and efficiency) performance of CC&TH services deployed in a local healthcare system. The framework of measurement indicators is structured in domains and subdomains over several areas. 

 

Organisation / Coordination

Driver

  • Coordination of care
  • Organisational structure & function
  • Technology
  • Care pathways

Population Stratification

Driver

  • Method
  • Disease
  • Deprivation index
  • Past use of healthcare

Clinical Stratification

Driver

  • Method
  • Health status
  • Frailty
  • Capabilities
 

 

 

Staff Engagement

Driver

  • Leadership
  • Awareness
  • Motivation
  • Workforce Development
  • Creating psychological ownership
  • Organisational change

Patient Adherence pre

Driver

  • Introduction
  • Belief
  • Experience with TH technology

Patient Adherence post

Driver

  • Adherence
  • Satisfaction
  • Acceptance
  • Benefits

Efficiency &
Efficacy

Outcome

  • Case ascertainment
  • Health outcomes
  • Clinical management goals
  • Process outcomes
  • Service utilisation
  • Economic outcomes

Holistic Evaluation

The holistic framework is depicted in Figure 32. The drivers represent all the real world aspects around CC&TH that are included in the framework, i.e. care coordination and organisation, staff engagement, patient adherence, and stratification. We measure these concepts by means of surveys: care coordination and workflow (CCWF), staff engagement (SE), programme manager (PM), frontline staff (FS), and patient adherence (PA).  We use these surveys for qualitative analysis to measure programme performance, do stakeholder comparison, correlated questions, and causal dependencies. Similarly, we measure a whole range of outcomes such as admissions, diagnosis, treatment, coverage, and so on. These quantitative measures, reported in excel templates, are used to measure programme performance and perform programme comparison. 

In this framework, we distinguish two types of outcomes: intermediate outcomes and final outcomes.

Intermediate Outcomes

Intermediate outcomes are outcomes measured by surveys. They capture the qualitative aspects of the care coordination and telehealth deployment. Intermediate outcomes have been collected at baseline and the second iteration and are available for analysis.

Final Outcomes (minimal data set)

Final outcomes are the efficiency & efficacy outcomes. These are quantitative data from the minimal data set, reported by the regions. They address the clinical and economical perspectives of the evaluation. Final outcomes have been collected in the first iteration; however, additional effort is required to detect the suitable elements for comparison. In the second iteration, follow-up data for final outcomes is collected to allow comparison over time.

In this project, our aim was to get a good view on the data that is currently collected by the regions, and to work with what is currently available. Regions started the data collection using an extensive list of indicators covering all domains in the Indicator Framework. This list of indicators was defined by the consortium, assuming full availability of data. This approach allowed us to develop an understanding of current data availability. Based on these findings, a minimum set of indicators has been defined. We follow the standard of measures reported by the National Quality Forum. At this stage we have put limited restrictions on inclusion and exclusion criteria in the definitions, and also the numerator and denominator statements are loosely defined to cater for collection of available data in the regions.

Case Ascertainment
The extent and precision with which the service manages its population.

Population Coverage

Case ascertainment indicator

Health Outcomes
The effect of the CC&TH service is on the patient’s physical and mental health (and deaths), functional limitations, and quality of life. Health outcomes have a direct meaning for the patient.

Mortality

Health outcome indicator

Polypharmacy

Health outcome indicator

Resource Utilisation
The effect of the CC&TH service on the utilisation of health care services. The CC&TH services should be cost effective. Here we measure the burden on the health care system. We measure the use of health care services in numbers, from which economic outcomes can be derived.

Hospitalisation days

Resource utilisation indicator

30 Day readmission

Resource utilisation indicator

Economic Outcomes
The cost of health care service utilisation. The CC&TH services should be cost effective. Here we measure the burden on the health care system in terms of cost.

Total cost per patient

Economic outcomes indicator

Transition towards primary care

Economic outcomes indicator

Visualisations

Outcome Scores

The outcomes score visualisations consist of two parts: Indicator Score and central Data Availability Score

In Figure 1, the central score represents the data availability for the programme, with scores 1-4 for each segment. The subscripts state how many indicators are available— a data availability score of 3.9 means that data was available for the programme, for multiple years, and 9 (out of 10) indicators have been reported. The outcome indicator score is visualised in each segment. Here we distinguish whether a programme is performing under or below its reference value. Also, whether or not the outcomes are improving over the years. In this example, the 30-day readmission is 4. Which ranks above average, with improvement (lower 30 day readmission) over the years.

The driver scores in Figure 2 have a similar shape. The central score refers to the intermediate outcomes—driver indicators that are considered to be an outcome by itself. The segments display the score of all indicators related to particular domain. All driver scores are related to the 5-pt Likert scales of the questionnaires.

Is is not possible, nor is it the intention, to benchmark the programmes. This is due to differences in data collection and differences in data maturity. Instead, the visuals should help show the relative strengths and weaknesses for each of the programmes. In particular, within a region.

Relation Drivers & Outcomes

The evaluation framework captures the complexity and heterogeneity of the data. The framework defines the indicators within their domains and subdomains. A scoring system for intermediate outcomes (drivers) and final outcomes allows quantification and comparison of the collected data. The linking mechanism for intermediate outcomes and final outcomes allows investigation of the relation between drivers and outcomes. Figure 39 depicts the framework visualizing the link between intermediate outcomes. In this figure we display the relation between the programme manager (PM) intermediate outcomes and final outcomes. The scores in each of the areas of the final outcomes are displayed in the table. It is possible to select only those programs that have data available (data maturity >2) and sort the programs on the drivers scores or the score for a particular outcome.