Falk Lieder

This project could be split into two parts: a function that performs cost-effectiveness for a given intervention and an analysis that uses that function to evaluate the value of research.

The value of research would lie in improving either the effect size or the scalability. We could then explore under which conditions the probability and magnitude of such improvements are large enough to warrant the cost of the research.

Moreover, the analysis of the cost-effectiveness of existing interventions would establish a baseline that is considered when determining whether the new intervention is more effective than the best existing intervention.

The baseline for the online intervention considered here would have to be an online intervention too. Very few of the previous interventions were online interventions. Perhaps the one by Baumsteiger was the only one? In that case, that one would become the baseline, and if that one is clearly lower, then the current analysis is correct. But then the model of the process is that we can create online interventions that are as effective as school-based interventions and much more scalable. This could be considered very optimistic rather than conservative.

The distinction between online and school-based interventions is important and should be taken into account. In each step it should be clear what kind of intervention is being assumed and those assumptions should be justified.

I think it would make sense for us to address these four questions sequentially instead of taking four steps at once to answer the fourth question right away. I think this will greatly simplify the presentation of the results because it will allow us to incrementally build on each of the previous steps. In that way, each of the steps might be small enough that it can be presented in a forum post that is short and simple enough that the typical reader can follow.

Three related questions are "How cost-effective are school-based interventions?", "Are digital interventions more cost-effective than school-based interventions?", and "How cost-effective would it be to develop more effective school-based interventions.".

to do: give an example of such an intervention
to do: add a link for randomized controlled trials
to do: complete the references

to do: currently, the effect sizes for mesurado include the effect size from a different study on one of the interventions that was also reviewed in Shin and Lee; I think that one should be removed

to do: check whether and, if so, how often multiple studies included in Shin and Lee tested the same intervention and then correct for that in the count for the number of interventions

to do: address the publication bias in the studies reviewed by Shin and Lee

to do: hunt down the individual effect sizes from the meta-analysis by Shin that are not already included in the meta-analysis by Mesurado.

The current standard error of the effect-size for the study by Baumsteiger (2019) is a placeholder that was calculated under the assumption that the correlation between pretest and posttest measures of prosocial behavior is 0.5. I calculated it as follows:

sqrt(SE_g_pre^2+SE_g_post^2-2*rho*SE_g_pre*SE_g_post),

where SE_g_pre and SE_g_post are the standard errors of Hedge's g for the comparisons between the two groups before the intervention and after the intervention, respectively.

[hedges_g_pre,SE_g_pre] = calculateEffectSize(4.08,4.62,1.28,1.41,43,32)
[hedges_g_post,SE_g_post] = calculateEffectSize(4.8,4.84,1.41,1.49,43,32)

Using a mixture distribution is imperfect because it doesn't take into account that the new intervention's effect size could be different from any of the previously observed effect sizes. Therefore, it would be better to fit a parametric distribution to the mixture distribution. Doing so should improve the extrapolation to unseen interventions.

It would be more principles to use the method of moments or maximum likelihood estimation. However, the stdev function is not implemented yet. So we are matching the 95% credible interval as a temporary solution.

to do: we need to explain (or point to) how the decay of the effect over time was modelled and how the half-life parameter was estimated

to do: we need to explain the integration over time or point to it.

Matching the 95% credible interval seems imperfect. I think it would be more principled to perform maximum-likelihood estimation of the distribution's parameters.

Given that this is one of the steps, it would make more sense to use a diagram that includes it.

It would be nice to convert the cost-effectiveness into WELBY/$. This could be implemented as a reusable function in the CEA library.

It would be nice to convert the cost-effectiveness estimates into WELBYs/USD.

They didn't report the cost-effectiveness in hours of happiness per dollar. So we might want to explain how it was converted into these units.

net_effect_of_research is a crucial function that needs to be explained. I wonder whether it makes sense to explain it here or whether the reader should be introduced to the application-independent fundamentals first in a separate document. We might first explain the fundamentals in the library notebook and then present this CEA as a an example of how those functions can be used in practice.

I think the way the net-effect of research function is used here assumes that ce_prev is the most cost-effective pre-existing intervention. However, there is some uncertainty about how cost-effective pre-existing interventions for promoting prosocial behavior are. There is a chance that one of those is more cost-effective than StrongMinds. This should be taken into account here.

I chose the assumed target group because I was envisioning an intervention for promoting prosocial purpose. However, other parts of the analysis are not specific to prosocial purpose.