Public
Edited
Jul 19, 2023
Importers
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
gw_ni_cea = FileAttachment("gw_ni_cea.csv").csv()
Insert cell
sections = ["Individuals 5-14 years old", "Individuals 15-49 years old", "Individuals 50-74 years old", "Counterfactual value of spending from non-philanthropic actors (units of value per dollar)", "Probability of scenarios in absence of New Incentives' spending", "What fraction of the program would still happen?", "Probability of scenarios in absence of New Incentives' spending", "What fraction of the program would still happen?"]
Insert cell
param_imports = {
let x = {}
let currentSection = null;
gw_ni_cea.forEach(entry => {
const parameter_name = entry["Conditional cash transfers to increase infant vaccination in North West Nigeria - New Incentives"].replace("\"", "")
if(sections.includes(parameter_name)){
currentSection = parameter_name;
}
else if (parameter_name === ""){
currentSection = null;
}
let value = entry["Overall"].replace(",","")
if(value === "Not specified"){
value = 0
}
if(!Number.isNaN(parseFloat(value))){
if(currentSection === null){
x[parameter_name] = parseFloat(value)
}
else{
if(x[currentSection]) {
x[currentSection][parameter_name] = parseFloat(value)
}
else {
x[currentSection] = {[parameter_name]: parseFloat(value)
}
}
}
}
})
return x;
}
Insert cell
Insert cell
viewof ni_givewell_params = cell(`ni_params = givewell_params
0`, givewell_params)
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
### Adjustment for Self-Reporting Bias in the Randomised Controlled Trial (RCT) of New Incentives' Program
Following are adjustments for self-reporting bias, which [GiveWell expects](https://www.givewell.org/international/technical/programs/new-incentives#Self-reporting_bias) to contribute to a net overestimation of vaccination in the control group, leading to an underestimation of the effect of New Incentives' program on vaccination rates. Vaccination coverage in control areas is a key input for cost analysis so its overestimation can also affect calculation of cost-effectiveness. Overall, self-report bias - as estimated by comparison of carers' reports of children receiving the BCG vaccine to children's rates of scarring from the BCG vaccine - adjusts the increase in vaccination rate from ${formatFloat2(run(`mean(vaccination_increase_raw * 100)`, vaccination_increase_raw).value)} pp to approx. ${formatFloat2(run(`mean(est_true_effect_based_on_bcg_scarring * 100)`, est_true_effect_based_on_bcg_scarring).value)} pp. This estimation is higher than GiveWell's 20pp. This may be because the IDInsight's report presents a 95% CI for each finding as well as a 'value' which is not the average of the upper and lower bounds. This suggests an asymmetric distribution which GiveWell has accounted for by using the main value presented while we have used the entire range, approximating it to a log normal distribution. This could be improved by reviewing the type of distribution used.

Approximately ${formatFloat2(run(`mean(bcg_scarring_control * 100)`, adjustment_self_report_bias).value)}% of infants who receive the BCG vaccine will bear a small scar at the injection site which can be used as evidence of their vaccination. However, sometimes infants may be identified as having a scar when they do not or visa versa and this must also be adjusted for. The proportion of children who scar from the BCG vaccine appears to be higher in the areas treated by the program than in the control areas. The reason for this is uncertain but it may indicate some infants receiving the vaccine more than once.

- ${refName({adjustment_self_report_bias}, `bcg_scarring_control`)}: The proportion of BCG *vaccinated* children in the control group who displayed scars
This value has a high level of uncertainty but is based on review of literature from around the world with scarring rates ranging from 80.4% in Pakistan to 98.6% in Peru. GiveWell assumes the rate in North West Nigeria to be comparable to the 90% found in [Dhanawade et al.'s 2015 report](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4535100/) in infants in India, although it may be worth noting that lower birthweight infants in India had scarring rates as low as 45%. Low birthweight can be contributed to by the birth-giver being underweight or underage, or having poor nutrition in pregnancy. Poorer people are at a greater risk of these factors.
- ${refName({adjustment_self_report_bias}, `bcg_scarring_treatment`)}: The proportion of BCG *vaccinated* children in the treatment group who displayed scars
This value has been calculated by comparing scars to self-reports, Child Healthcare Cards, and clinic immunisation records. [GiveWell attributes](https://www.givewell.org/international/technical/programs/new-incentives#footnote16_9txuuzy) the ostensibly higher rate of scarring in the treatment group than the control to "some infants in the treatment group receiving BCG more than once, either due to errors (e.g. unclear child health cards leading clinic staff to believe that an infant had not yet received BCG when they in fact had) or repeat enrollment, though this is speculative." Since infants would have a certain probability of scaring each time they received the vaccine, a portion of the children being vaccinated multiple times would increase the proportion of vaccinated children with a scar.

I've set the distribution for this to a beta distribution with a mean of 0.97 and standard deviations of 0.02 so that the 95% confidence interval does not spill over 1, as this would be impossible.
- ${refName({adjustment_self_report_bias}, `scarring_detection_rate`)} (sensitivity): The proportion of BCG scars the researchers expected to be able to detect.

Some infants may have scarred but had their scars either not detected by research staff or mistaken for something else, leading to an underestimation of BCG scarring and therefore of vaccination - if this was calculated based on scarring. This parameter accounts for that source of error.
I've taken the 95% CI for this from [IDInsights report](https://files.givewell.org/files/DWDA%202009/NewIncentives/IDinsight_Impact_Evaluation_of_New_Incentives_Final_Report.pdf). This resulted in a mean lower than the value used by GiveWell.
- ${refName({adjustment_self_report_bias}, `scarring_identification_correct`)} (specificity): The proportion of BCG scars detected that were actually BCG scars, or equivalently, the proportion of infants who did not scar who were correctly detected as 'not scarring'.

Some infants recorded as having scars from the BCG vaccine may not have actually. What the researchers identified as a BCG scar may have actually been a birth mark or a scar from another source. It's also possible researchers could have mistakenly counted the same child with a scar twice. These mistakes would lead to an overestimation of BCG scarring and therefore of vaccination - if this was calculated based on scarring. This parameter accounts for that source of error.
I've taken the 95% CI for this from [IDInsights report](https://files.givewell.org/files/DWDA%202009/NewIncentives/IDinsight_Impact_Evaluation_of_New_Incentives_Final_Report.pdf). This resulted in a mean lower than the value used by GiveWell.
- ${refName({adjustment_self_report_bias}, `bcg_scars_detected_in_control`)}: The proportion of *all* children in the control cohort with BCG scars detected
I've taken the 95% CI for this from [IDInsights report](https://files.givewell.org/files/DWDA%202009/NewIncentives/IDinsight_Impact_Evaluation_of_New_Incentives_Final_Report.pdf). This resulted in a mean higher than the value used by GiveWell.
- ${refName({adjustment_self_report_bias}, `bcg_scars_detected_in_treatment`)}: The proportion of *all* children in the treatments cohort with BCG scars detected

I've taken the 95% CI for this from [IDInsights report](https://files.givewell.org/files/DWDA%202009/NewIncentives/IDinsight_Impact_Evaluation_of_New_Incentives_Final_Report.pdf). This resulted in a mean higher than the value used by GiveWell.
- ${refName({est_bcg_vaccination_in_control})}: The proportion of children in the control group estimated to have received the BCG vaccine (based on scarring)
- ${refName({est_bcg_vaccination_in_treatment})}: The proportion of children in the treatment group estimated to have received the BCG vaccine (based on scarring)
- ${refName({est_true_effect_based_on_bcg_scarring})}: The estimated true effect of the program, i.e. the increase in vaccination as a result of the program, calculated by the differences in vaccination rates (as estimated based on scarring) between the control and treatment groups.
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
After adjusting up for self-report bias and vaccines with multiple doses and down towards a skeptical prior, the adjusted increase in vaccination rates due to NI is expected to be about ${formatFloat2(run(`mean(adjusted_vaccination_increase_by_ni) * 100`, adjusted_vaccination_increase_by_ni).value)} pp.
Insert cell
Insert cell
Insert cell
A donation of $${run(`donation_size`, donation_size).value.value} usd to NI, in addition to government and Gavi spending on vaccination, would facilitate the counterfactual vaccination of ${project.getResult(infants_in_cohort_couterfactually_vaccinated).value} infants.

I have set the default ${refName({donation_size})} to $1 usd, so that the output gives an indication of benefit per dollar, but the model is interactive and the size of the donation, as well as government and Gavi contributions may be adjusted below to roughly calculate the effect of any donation to NI.

Regarding NI's cost-effectiveness, it is important to consider what is already being done by other party's to achieve their primary goal of increasing vaccination in North West Nigeria. The total money being dedicated to the cause is greater than NI's contribution alone, potentially increasing how efficiently their funds can be used. This may increase cost effectiveness. However, we must also consider that if NI enrols a cohort of a certain size and offers all of them CCT's to incentivise vaccination, many of these infants may have been vaccinated anyway, due to the efforts of the government or Gavi, meaning that not all of the CCT's will result in a *counterfactual* vaccination. This consideration would likely decrease cost-effectiveness and I expect it's effect to be much greater than possible efficiency gained by combining funds, meaning that the net impact of external funds would be a decrease in NI's impact per dollar, and failing to adjust for it would lead to an overestimation of cost-effectiveness.

- `government` here refers to the Nigerian government, which already has [programs and policies](https://nimr.gov.ng/nimr/wp-content/uploads/2021/10/Nigeria-Vaccine-Policy-2021.pdf) in place to facilitate vaccination across Nigeria.
- `gavi`'s, or [*Gavi, The Vaccine Alliance*](https://www.gavi.org), partners with "the World Health Organization, UNICEF, the World Bank and the Bill & Melinda Gates Foundation, and plays a critical role in strengthening primary health care (PHC), [particularly increasing vaccination,] and bringing the countries they works in closer to the Sustainable Development Goal (SDG) of Universal Health Coverage (UHC). [They] also work with donors, including sovereign governments, private sector foundations and corporate partners; NGOs, advocacy groups, professional and community associations, faith-based organisations and academia; vaccine manufacturers, including those in emerging markets; research and technical health institutes; and implementing country governments." [Gavi contributes significantly]( https://www.gavi.org/programmes-impact/country-hub/africa/nigeria) to increasing vaccination in Nigeria.


**Sam could you please make this say the number for infants_in_cohort_couterfactually_vaccinated?**

[Costs](https://docs.google.com/spreadsheets/d/1gAy4_Y4SvvE-R2sTEE7jio3pf4Ak1JPw78NhWJKFWR4/edit#gid=2130000819)
- ${refName({param}, `usd_cost_program_per_infant_ni`)}, ${refName({param}, `usd_cost_counterfactual_full_immunisation_government`)} and ${refName({param}, `usd_cost_counterfactual_full_immunisation_gavi`)} were calculated to be 29.52, 48.98 and 40. 91 respectively, based on [data](https://docs.google.com/spreadsheets/d/1gAy4_Y4SvvE-R2sTEE7jio3pf4Ak1JPw78NhWJKFWR4/edit#gid=2130000819) from Jun 2019 to Aug 2021. The government and Gavi values in the spreadsheet are qualified with a note that "we are unsure about the extent to which the [plan](https://nigeriahealthwatch.com/wp-content/uploads/bsk-pdf-manager/2019/09/18.04.2018_Nigeria-Strategy-for-Immunization-and-PHC-Strengthening_3rd_Version-Final.pdf) from 2018 to scale down Gavi spending and scale up government spending on routine immunisation has been followed. As a rough guess, we estimate that the proportion of costs covered by government and Gavi in 2022-2024 are an average of the historical costs covered by each actor in 2014-2018 and the planned proportion covered by each actor for 2022-2024 from [NIGERIA STRATEGY FOR IMMUNISATION AND PHC SYSTEM STRENGTHENING [NSIPSS] 2018 –2028](https://nigeriahealthwatch.com/wp-content/uploads/bsk-pdf-manager/2019/09/18.04.2018_Nigeria-Strategy-for-Immunization-and-PHC-Strengthening_3rd_Version-Final.pdf), Pg 24, Figure 8."
- I have presented ${refName({param}, `usd_cost_program_per_infant_ni`)} as a log normal distribution based on the min and max values across that time period. This is a rough way to estimate the uncertainty in these values and the distributions could be made more accurate by including the calculations with the uncertainty presented for each of the parameters used. It's worth noting that the mean of this distribution is about 17% lower than the value used by GiveWell. The government and Gavi values have both been presented as distribution between values 25% above and below their estimated value because the proportion of their funding each are directing to immunisation appears surprisingly uncertain.
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
${refName({param}, `vaccine_preventable_disease_mortality_unvaccinated_north_west_nigeria_children_under_5`)} =
${formatFloat2(run(`mean(vaccine_preventable_disease_direct_and_indirect_mortality_unvaccinated_children_under_5_north_west_nigeria)`, param).value)}

Ratio of the reduction in vaccine-preventable disease mortality to the reduction in vaccine-preventable disease = ${formatFloat2(run(`mean(ratio_vaccine_preventable_disease_mortality_reduction_to_vaccine_preventable_disease_incidence_reduction)`, param).value)}
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
exportSampleSet(project, `units_of_value_generated_per_dollar_spent_by_new_incentives(ni_params)`, ni_givewell_params, results_after_leverage_or_funging_adjustment)
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell
Insert cell

One platform to build and deploy the best data apps

Experiment and prototype by building visualizations in live JavaScript notebooks. Collaborate with your team and decide which concepts to build out.
Use Observable Framework to build data apps locally. Use data loaders to build in any language or library, including Python, SQL, and R.
Seamlessly deploy to Observable. Test before you ship, use automatic deploy-on-commit, and ensure your projects are always up-to-date.
Learn more