Week 9: Impact Evaluation Methods

The main topics
• What is Evaluation?
• Theory of change
• Measurement
• Why randomise?
• How to randomise?
• Sampling and sample size
• Threats and analysis
• Generalisability
• Start to finish
EFN423 Health Economics 3
Why do we need Impact Evaluation?
• Billions of dollars are spent on social and health programs every year:
• “.. $2.3 trillion has been spent by the world’s wealthies nations on
poverty reduction over the past fifty years…” Karlan, Dean and Appel,
Jacob (2011) “More than Good Intentions”. Dutton, NY.
• However, we know very little about the performance and the impact of these
programs
• We know very little about what works
• There is a knowledge gap: “Evaluation Gap”
EFN423 Health Economics 4
Components of a Program
All programs/projects have (implicitly or explicitly):
¨ Objectives
¨ Key outcomes and targets
¨ Target areas and target population
¨ The intervention(s) and the delivery mechanism(s)
¨ Participant eligibility criteria and process for selecting participants
¨ A conceptual framework or program theory presenting the causal
chain to be induced by the program to change the outcome
¨ Implementation plan: start date, duration, deployment plan
Inputs Processes Outputs Outcomes
Resources:

• Staff
• Supplies
• Equipment
• Infrastructur
  e
  Activities:
• Training
• Logistics
• IEC
  Services:
• % facilities with
  health services
• # trained staff
• # services
  delivered
  Utilization:
• # new users
• # continuing
  users
  Health
  conditions:
• Incidence
• Mortality
  Program Level Population
  Level
  Components of a Health Program
  (Logical Framework)
  Final
  Outcomes
  Behaviors:
• Condom
  use
  Sometimes
  called Impact
  EFN423 Health Economics 6
  Impact evaluation
  ¨ Objectives:
• How much of the observed change in the outcome can be
  attributed to the program and not to other factors?
  ¨ Characteristics:
• Key issues: causality, quantification of program effect
  Note: Program monitoring tells you that a change occurred; impact
  evaluation tells you whether it was due to the program
  Important: Impact evaluation is not about monitoring final outcomes!
  EFN423 Health Economics 7
  Impact evaluation and Program evaluation
  Program Evaluation
  Impact Evaluation
  RCTs
  EFN423 Health Economics 8
  What is the impact of this program?
  Primary
  Outcome
  Program starts
  Time
  EFN423 Health Economics 9
  What is the impact of this program?
  ¨Positive
  ¨Negative
  ¨Zero
  ¨Not enough info
  EFN423 Health Economics 10
  What is the impact of this program?
  Time
  Primary Outcome
  Impact
  Counterfactual
  Program starts
  EFN423 Health Economics 11
  How to measure impact?
  ¨Impact is defined as a comparison between:

1. the outcome some time after the program has been
   introduced (the “factual”)
2. the outcome at that same point in time had the
   program not been introduced (the “counterfactual”)
   EFN423 Health Economics 12
   Impact: What is it?
   Time Primary Outcome
   Impact
   Counterfactual
   Program starts
   EFN423 Health Economics 13
   Impact: What is it?
   Time Primary Outcome
   Impact
   Counterfactual
   Program starts
   EFN423 Health Economics 14
   Counterfactual
   The counterfactual represents the state of the world that
   program participants would have experienced in the absence
   of the program
   Problem: Counterfactual cannot be observed
   Solution: We need to “mimic” or construct the counterfactual
   EFN423 Health Economics 15
   Constructing the counterfactual
   • Usually done by selecting a group of individuals that did
   not participate in the program
   • This group is usually referred to as the control group or
   comparison group
   • How this group is selected is a key decision in the
   design of any impact evaluation
   EFN423 Health Economics 16
   Selecting the comparison group
   ¨Idea: Comparability
   ¨Goal: Attribution
   EFN423 Health Economics 17
   Evaluation design: Finding a good
   counterfactual
   ¨Good evaluation designs address:
   I. Multiple factors
   II. Selection bias
   III. Spillovers
   IV. Contamination
   V. Heterogeneous impacts
   VI. Timing of effects
   ¨They are needed for internal validity:
   your estimate of program impact is really a good measure (approximation) of
   the real causal relationship between the intervention and the outcome in the
   sample of analysis.
   EFN423 Health Economics 18
   Finding a good counterfactual – Key issues to
   consider
   I. Multiple factors influence the behavior or health outcome of interest.
   The program is only one factor among many.
   You need a conceptual framework.
   Simple Conceptual Framework of Health
   Outcomes
   Individual / Household

• Age
• Education
• Household wealth/SES
• Risk aversion
• Biological endowments
  Health Service Supply/
  Community
• Health facilities Access
  Price
  Quality
• Fieldworkers Number
  Quality
• Program
• Sanitation
• Schools
  Healthy
  Behavior
  Healthy
  Outcomes
  Simple Conceptual Framework of Health Outcomes
  Which factors do we observe?
  Individual / Household
• Age
• Education
• Household wealth/SES
• Risk aversion
• Biological endowments
  Healthy
  Behavior
  Healthy
  Outcomes
  Observed?
  Yes
  Yes
  Yes
  No
  No
  Observed ?
  Yes
  No
  No
  Yes
  No
  Yes
  No
  No
  Problem:
  Incomplete information
  Question:
  How do we control for
  observed and unobserved
  factors in order to isolate
  program impact?
  Health Service Supply/
  Community
• Health facilities Access
  Price
  Quality
• Fieldworkers Number
  Quality
• Program
• Sanitation
• Schools
  EFN423 Health Economics 21
  Finding a good counterfactual – Key issues to
  consider
  II. Self Selection
  ¨Two usual features,
• Individuals decide to participate into the program
• Program managers target the program to particular communities
  ¨Therefore, there are two selection processes:
• Self-selection of individuals
• Selection of program intervention areas or “intervention communities” by
  program managers
  ¨Consequence:
  Program participants are most likely different from the non-participants.
  Individual / Household
• Age
• Education
• Household wealth/SES
• Risk aversion
• Biological endowments
  Health Service Supply /
  Community
  Program
  participation
  Healthy
  outcome
  Observed?
  Yes
  Yes
  Yes
  No
  No
  If unobserved factors influence both the outcome and program participation,
  and you do not control for them, the estimation of program effect
  might be incorrect, because changes in the outcome will be attributed to program
  participation, when they were due to the effect of the unobserved factors.
  Individual self-selection into the program – Observed and
  Unobserved factors
  EFN423 Health Economics 23
  Example 1 – two seemingly identical persons
  Characteristics Person 1 Person 2
  Observed Age 27 27
  Education 10 10
  SES Low Low
  Place residence Rural Rural
  Unobserved Risk-aversion Yes Yes
  Biological conditions Frail Healthy
  Observed Program participation Yes No
  Outcome Health status Bad Good
  Example 2: Targeted program in community –
  Observed and Unobserved factors
  Individual / Household
  Community
• Schools
• Non-program health
  service sources
• Community leaders of
  opinion and influence
• Sanitation
  Program
  presence
  Healthy
  outcome
  Observed?
  Yes
  Yes
  No
  No
  If unobserved factors influence both the outcome and program presence, and you do
  not control for them, the estimation of program effect might be incorrect, because
  changes in the outcome will be attributed to changes in participation, when they
  were due to the effect of the unobserved factors.
  EFN423 Health Economics 25
  Characteristics Person 1 Person 2
  Observed Age 27 27
  Education 10 10
  SES Low Low
  Place residence Rural Rural
  Unobserved Sanitation Bad Good
  Observed Program presence Yes No
  Outcome Health status Bad Good
  • A simple analysis, using only observed characteristics, will find a negative program
  impact, associating Program Presence to Bad Outcomes
  • But, the Bad outcome is caused by the unobserved factor.
  Example 2 – two seemingly identical persons
  EFN423 Health Economics 26
  Finding a good counterfactual – Key issues to
  consider
  III. Spillover
  They occur when the intervention has an impact on individuals “outside” the
  program intervention group
  — e.g., vaccination, information/education, cash transfer programs
  Consequences:
  • Underestimates the total population influenced by the program
  • Underestimates program impact when the spillover affects individuals of
  the comparison group.
  If spillovers are possible, evaluations should be designed to account for them.
  EFN423 Health Economics 27
  Finding a good counterfactual – Key issues to
  consider
  IV. Contamination
  It occurs when external factors affect either the intervention or comparison group.
  Examples: Other programs, Rains, price changes
  Consequences
  • If contamination is non-differential (affects T and C similarly), no bias
  • If contamination is differential (affects more T than C), the impact estimate could
  be biased; If you know it, you can add interaction terms and hope for no selectivity,
  but you have less statistical power
  • If contamination is complete in T, you cannot separate the effect of the external
  factor from the program impact
  Actions:
  • Avoid contamination. Monitor T and C groups and measure external factors.
  EFN423 Health Economics 28
  Finding a good counterfactual – Key issues to
  consider
  V. Heterogenous Impact
  • Program could have different impacts on different population groups.
  Examples: – By education level
• By SES: poor and non-poor
• By rural and urban areas
• By quality services where program operates
  • Consequences
• Average program impact for the entire treatment group hides the high (or low)
  impact on particular subgroups
• The average program impact may not be informative for policy decisions.
  • Actions
• Define subgroups of interest in evaluation design and in sampling design; power
  calculation for each subgroup (larger n and budget)
• Model the heterogeneity of impact in estimation strategy
  EFN423 Health Economics 29
  VI. Timing
  Design should allow enough time for program to have a measurable effect on the
  outcome. Ex. Should you estimate impact at midline or at endline?
  Program
  start
  Program
  end
  Outcome
  Program
  Impact
  With
  program
  Without
  program
  Midline
  Smaller
  Impact
  Finding a good counterfactual – Key issues to
  consider
  EFN423 Health Economics 30
  Finding a good counterfactual – Key issues to
  consider
  An additional issue: External validity
  • Program impact estimate is valid for the whole target population of the program,
  for other population groups in the country, or in other countries.
  • How to address external validity?
• Sample of analysis is representative of population of interest
• Analysis of similarities between the program and analysis sample and those in
  other regions or countries
• Conceptual framework or program theory is widely accepted
• Causal chain links are tested
  • Note, causal chain analysis is also included as a requirement for internal validity.
  EFN423 Health Economics 31
  External Validity
  Two: Serious Objections Three: Navel Gazing
  ≠
  EFN423 Health Economics 32
  Important – Avoid two commonly used
  counterfactuals
  • Before and after only (pre-post) – no comparison group
  Data on the program group before and after the program.
  • Simple comparison of enrolled versus not-enrolled
  This compares apples with oranges. We don’t know why one group
  enrolled and the other did not. Most likely those two groups are
  different, even in absence of the program.
  Program
  start
  Program
  midpoint or end
  Time
  Outcome
  Before and After: Can we attribute to the program the whole
  change we observe in the outcome?
  Program
  impact ?
  With
  program
  Without
  program
  Program
  start
  Program
  midpoint or end
  Time
  Outcome
  Program
  Impact
  Effect of
  other factors
  Before and After: Can we attribute to the program
  the whole change we observe in the outcome? No
  Group 1
  (Treatment)
  Population
  Basic Framework for Evaluation of Program Impact
  Group 2
  (Control)
  PROGRAM
  Group 1
  (Treatment)
  Group 2
  (Control)
  If “Random
  Assignment”: Group 1 = Group 2
  If Non-random
  Assignment:
  (Survey data)
  Group 1  Group 2
  Group 1  Group 2 Difference
  due to program
  Group 1  Group 2
  Difference due to program and
  pre-existing differences
  (observed and unobserved)
  EFN423 Health Economics 36
  Individuals are randomly assigned into a Treatment group and a Control
  group
  If well implemented and sample is large enough, random assignment
  makes the pre-program treatment and control groups similar on
  observed and unobserved characteristics.
  To estimate program impact:
  Program Impact = Average(“Treatment”) – Average(“Control”)
  Experiments control for problem of incomplete information and
  selection.
  Experimental Design
  EFN423 Health Economics 37
  Few conditions are needed:
  • Assignment to program is random
  • Program “intensity” is homogenous in treatment group
  • No spillover effect
  • Individuals do not change behavior because of participation/non-participation in
  experiment
  • There is no selective participation
  • Individuals remain during duration of experiment or at least there is no selective attrition
  • External factors influence both groups equally
  Randomised controlled trial (RCT) frequently used for clinical trials, vaccines, new drugs.
  Several examples in social policy.
• Classic experimental design, if well implemented, is the gold standard.
  Experimental Design (contd.)
  EFN423 Health Economics 38
  However,
  • Difficult implementation (practical, political or ethical)
  • Contamination is possible:
• spillover effects from the program into the control group
• from other programs into the treatment group or control group
  • Behavior might be modified by experiment
  • Selective participation
  • Selective attrition
  • Programs are complex interventions with multiple components and stages so, in which
  stage to randomise?
  • Generalisation is questionable as they might be applied in conditions that do not occur
  in routine operation of program
  Experimental Design (contd.)
  EFN423 Health Economics 39
  What is an RCT?
  The basics
  Start with simple case:
  ¨Take a sample of program applicants
  ¨Randomly assign them to either:
   Treatment Group – is offered treatment
   Control Group – not allowed to receive treatment (during the
  evaluation period)
  EFN423 Health Economics 40
  Why randomise? Conceptual Argument
  If properly designed and conducted, randomised experiments provide
  the most credible method to estimate the impact of a program on
  the study population
  Why most credible?
  Because members of the groups (treatment and control) do not differ
  systematically at the outset of the experiment, any difference that
  subsequently arises between them can be attributed to the program
  rather than to other factors.
  EFN423 Health Economics 41
  RCT: Key advantages of experiments
  Because members of the groups (treatment and control) do
  not differ systematically at the outset of the experiment, any
  difference that subsequently arises between them can be
  attributed to the program rather than to other factors.
  EFN423 Health Economics 42
  Two RCTs
  • RAND Health Insurance Experiment (HIE)
  • Oregon Medicaid Experiment
  EFN423 Health Economics 43
  RAND HIE
  ¨Randomly assigned 2,000 families from six US
  cities to different insurance coverage plans
  • Copayments groups:
  –Free, 25%, 50%, and 95%
  ¨Tracked utilization of health care (Q) in each
  copayment plan (P)
  • Copayment acts as the marginal cost that each
  family faces when buying care
  EFN423 Health Economics 44
  Oregon Medicaid Experiment
  ¨Compared two groups of low-income adults
  • Medicaid lottery winners vs. lottery losers
  ¨Lottery winners got to apply for public health
  insurance through Medicaid
  • So they faced lower out-of-pocket prices for care
  ¨Lottery losers could not get Medicaid (but might have
  purchased outside insurance)
  EFN423 Health Economics 45
  Results?
  ¨Health care demand curves are downward sloping
  (economic theory prevails!)
  ¨ Price changes affected demand for health care
  EFN423 Health Economics 46
  Different measures of care
  ¨Outpatient Care
  • Def: any medical care that does not involve an overnight
  hospital stay
  – E.g. runny noses, twisted ankles, minor broken bones
  ¨Inpatient Care
  • Def: medical care requiring overnight stays
  – E.g. More serious surgeries or conditions that require overnight
  recovery or monitoring
  ¨ER Care
  • Def: care involving the emergency room
  – E.g. heart attacks, strokes
  EFN423 Health Economics 47
  Outpatient care
  ¨RAND HIE
  • As patient cost-sharing (P) increases, number of episodes
  (Q) of outpatient care decreases
  • Holds for both acute and chronic conditions
  Data from Keeler et al. (1988)
  EFN423 Health Economics 48
  Outpatient care
  • Oregon Medicaid Study
  o Lottery winners have more outpatient visits than lottery losers
  Both the RAND HIE and the Oregon Medicaid Study
  find downward-sloping demand for outpatient care!
  EFN423 Health Economics 49
  Inpatient care
  ¨RAND HIE Oregon Medicaid
  Study
  No significant difference
  in usage rates between
  lottery winners and lottery
  losers
  Demand is still downward-sloping but less elastic than demand
  for outpatient care
  (Data from Keeler, 1988)
  EFN423 Health Economics 50
  ER care
  • RAND HIE • Oregon Medicaid Study
  No significant difference in ER
  care for lottery winners vs.
  lottery losers
  Even for emergency room care
  – likely the most urgent kind –
  those on the highest copayment
  plan in the RAND HIE were less
  likely to buy care!
  (Data from Newhouse, 1993)
  EFN423 Health Economics 51
  Does price for care affect
  health?
  ¨RAND HIE:
  • Generally, no health
  differences between
  people on free plan
  vs. cost-sharing!
  **Only statistically
  significant difference
  between plans were
  in blood pressure,
  myopia, &
  presbyopia
  EFN423 Health Economics 52
  Does price for care affect
  health?
  ¨Oregon Medicaid Experiment
  • Lottery winners self-reported better overall health,
  more healthy days, and lower rates of depression
  ¨Discrepancy with RAND HIE may be because
  Oregon Medicaid Study worked with the very lowincome,
  while RAND HIE studied a broader crosssection
  of the U.S.
  EFN423 Health Economics 53
  Some variations on the basics
  • Assigning to multiple treatment groups
  • Assigning of units other than individuals or households
   Health Centers
   Schools
   Local Governments
   Villages
  EFN423 Health Economics 54
  Key Steps in conducting an experiment

1. Design the evaluation and intervention carefully
2. Randomly assign people to treatment or control
3. Collect baseline data
4. Verify that assignment looks random
5. Monitor process so that integrity of experiments is not
   compromised
   EFN423 Health Economics 55
   Key Steps in conducting an experiment
   (contd.)
6. Collect follow-up data for both the treatment and control
   groups
7. Estimate program impacts by comparing mean outcomes of
   treatment group vs mean outcomes of the control group
8. Assess whether program impacts are statistically significant
   and practically significant