Module 8 — Systems Evaluation and Intervention Strategies
This module introduces evaluation as a continuous process embedded within improvement cycles. Students learn methods to assess whether interventions solve intended problems and how to recognise unintended effects. Emphasis is placed on context-sensitivity, adaptation, and real-world evidence. Students examine how to scale innovations responsibly, allowing for feedback-driven redesign rather than rigid replication.
Learning Outcomes –
- Apply evaluation concepts to assess whether interventions generate expected system-level change.
- Compare evaluation methodologies suitable for dynamic health environments.
- Analyse contextual constraints that influence implementation success.
- Propose system-focused strategies for scaling and sustaining interventions.
1) Evaluation Frameworks
Evaluation frameworks provide structured approaches to measure whether policies, programs, or interventions achieve their intended impact. In systems thinking, evaluation must account for complexity—capturing not only what changed, but how, for whom, and under what conditions change took place.
Traditional evaluation often focuses on outcomes alone; however, in complex health systems, outcomes arise from multiple interacting influences. Systems-oriented evaluation frameworks examine program context, relationships, feedback loops, and emergent behaviours. They recognise that interventions may perform differently depending on the organisational, cultural, and policy environment.
Several major frameworks are used in health systems. Formative evaluation guides early program development by testing feasibility, acceptability, and readiness. Process evaluation examines how implementation occurs, identifying system facilitators and barriers. Outcome evaluation measures immediate and longer-term effects. Developmental evaluation supports innovation under rapidly changing conditions, emphasising real-time learning rather than static judgment.
Systems evaluation encourages continuous feedback rather than one-time assessment. It considers intended and unintended consequences, including how interventions may shift workload, behaviour, or resource utilisation. It also highlights the importance of assessing differential effects across populations, identifying inequities early.
Table: Types of Evaluation Frameworks
| Evaluation Type | Primary Focus | Typical Use Case | Value in Complex Systems |
|---|---|---|---|
| Formative | Feasibility, early design | Intervention planning | Supports readiness + design refinement |
| Process | How implementation occurs | Program rollout | Uncovers barriers + enablers |
| Outcome | Effect of intervention | Post-implementation impact | Shows what changed |
| Developmental | Real-time learning | Innovation + adaptation | Supports rapid evolution |
Evaluation frameworks help determine whether an intervention is both effective and system-aligned, informing decisions about improvement, adaptation, or expansion.
2) Implementation Science
Implementation science examines how evidence-based practices are adopted, embedded, and sustained in real-world settings. It acknowledges that even well-designed interventions may fail if system context, culture, workflow, or leadership are not aligned.
The goal of implementation science is to understand why some interventions succeed while others fail, and to refine strategies that enhance uptake, fidelity, and long-term benefit. It studies not only the intervention, but also the environment into which it is introduced—and how both evolve over time.
Key concepts include:
• Fidelity — the degree to which an intervention is delivered as intended
• Adaptation — modification of an intervention to fit local context
• Acceptability — how stakeholders perceive usefulness and relevance
• Feasibility — practicality given available resources and constraints
Frameworks such as CFIR (Consolidated Framework for Implementation Research) highlight how characteristics of the intervention, individuals involved, and organisational context shape success. Implementation strategies include training, coaching, workflow redesign, incentives, communication campaigns, and leadership engagement.
Systems thinking enhances implementation science by emphasising relationships and feedback. Rather than assuming change is linear, it recognises that interventions interact with existing structures and may require iterative refinement.
Table: Key Factors Studied in Implementation Science
| Factor | Description | Example |
|---|---|---|
| Fidelity | Consistency with original design | Standardised care protocol |
| Adaptation | Modification for context | Cultural tailoring |
| Acceptability | Perceived suitability | Clinician endorsement |
| Feasibility | Practicality in setting | Resource availability |
Implementation science ensures that innovations translate into practice in a realistic, sustainable way.
3) Realist and Mixed-Method Approaches
Realist evaluation seeks to answer not just whether interventions work, but how and why they work for certain groups and under particular circumstances. It recognises that interventions interact with context; therefore, outcomes vary widely. The realist approach uses the Context–Mechanism–Outcome (CMO) framework to explain how context activates mechanisms that produce outcomes. This supports nuanced understanding rather than generalised claims.
For example, a community diabetes prevention program may work well when leadership is strong and peer networks are active (context), motivating behaviour change through shared learning (mechanism), leading to reduced risk factors (outcome). Where trust is low or engagement weak, the same program may fail.
Mixed-method approaches combine quantitative data (such as clinical measures or utilisation patterns) with qualitative insight (such as patient or provider experiences). This combination provides a more complete picture of system performance. Quantitative data shows patterns; qualitative data explains why they occur.
Mixed-method designs are particularly valuable in health systems because they capture both measurable impact and lived experience. They help evaluate complex interventions where numbers alone cannot explain outcomes. For example, reductions in hospital readmissions might be linked to improved discharge processes (quantitative), but interviews may reveal that family support was a critical enabling factor (qualitative).
Table: Realist vs Mixed-Method Approaches
| Aspect | Realist Evaluation | Mixed-Method Approaches |
|---|---|---|
| Core Question | What works, for whom, in what context, and why? | What is happening and why, supported by multiple data types? |
| Key Framework | C–M–O | Integrated qualitative + quantitative |
| Strength | Reveals causal pathways within context | Holistic understanding |
| Use Case | Complex social/health interventions | Deep evaluation across multiple outcomes |
These approaches support deeper interpretation of system behaviour, improving decision-making and intervention alignment.
4) Scaling & Sustainability
caling refers to expanding a successful intervention beyond its original context to increase reach and population benefit. Sustainability refers to maintaining its effectiveness over time. While many pilot programs show early promise, few scale successfully; even fewer sustain impact in diverse, changing environments.
Scaling requires understanding which elements of an intervention are core components that must be preserved, and which components can be adapted. Systems thinking helps identify enabling conditions—such as leadership support, financing structures, technology capacity, incentives, and workforce readiness—that must be in place for success.
Four common approaches to scaling include:
• Replication — implementing the same model in new settings
• Adaptation — modifying the model to suit local conditions
• Integration — embedding into existing systems or workflows
• Spread through networks — scaling via partnerships or communities of practice
Sustainability requires continuous evaluation, financial support, stakeholder engagement, and supportive culture. Over time, new contextual shifts—policy changes, workforce turnover, new technologies—may require further adaptation. Sustainable interventions become part of routine practice and retain value even as conditions change.
Key risks to scaling and sustainability include loss of fidelity, insufficient resources, weak governance, and failure to adapt. Systems thinking guides proactive planning by recognising where interventions interact with system structures and dynamics.
Table: Scaling vs Sustainability
| Feature | Scaling | Sustainability |
|---|---|---|
| Focus | Expanding reach | Long-term continuity |
| Primary Goal | Benefit larger populations | Maintain effectiveness over time |
| Strategy | Replicate/adapt | Integrate + refine |
| Success Indicator | Wider adoption | Continued value + resilience |
Successful scaling and sustainability processes acknowledge that systems evolve; interventions must therefore evolve alongside them.