A collection of GiveWell related Squiggle documents

Hannah Rokebrand

Tanae Rao

Sam Nolan

Quantifying Uncertainty in GiveWell Cost-Effectiveness Analyses

Givewell CEAs

> Epistemic Status: Still has fairly glaring numeric errors.

 The modesty is good, but you should also at least mention the case for a how you are still adding value. Also by "numeric errors" sounds like something that should be easily correctable, but I assume it isn't otherwise you would've corrected it. Rephrase?

david reinstein

It is easily correctable, and I've gone about correcting it now. I'm going to spend more time ironing out the correctness problems, this is quite a beast of a project to check!

> we use our uncertainty quantification to calculate the Expected Value of Perfect Information on all GiveWell Top Charities 

Reading through this, really curious what the basis for this calculation would be, maybe give a hint or a link here?

Note to self: add line explaining that we have country-level models as well

AMF is wrong here, need to fix. Would be good to include sensitivity analysis to find out where all the uncertainty is coming from.

This doesn't find the value of researching ALL charities, only one

I think value of information might be higher if we split by charity-country pairings rather than the global mean of each charity--maybe this is something to add to the room for improvement in our models section?

(Because this is how GiveWell allocates funds)

Maybe rewrite the presentation of VOI? (new stuff in asterisks or brackets for deletions)

> In the context of *choosing among discrete alternatives*, information is more valuable when:

> It is more likely to change our mind *about the best alternative* [] 
> *the value of the alternatives we are choosing among is likely to be larger*
> The environment is high-stakes, implying a greater potential for producing value or harm, given the resources devoted to it.

Trying to edit this so as to not be confused with a case where the context is high-stakes, but we are trying to choose between 'purple bednets and pink bednets' (nearly equivalent value)

> We call case (a) - where we donate to the expected best charity without researching -

I would say "without researching any *further*" ... obviously the above is based on a bunch of research

> The maximum cost-effectiveness we could expect to obtain in the uninformed state (U), in units of value per dollar, is simply the highest expected value of all the interventions considered:

I'd add a footnote embodying the idea that this is based on the distribution obtained *after* incorporating our prior beliefs about the effectiveness of interventions, and about the reliabiilty of the evidence in the presence of publication bias etc. (I think this ties into Joel's point in our conversation.)

The subscript indices on the X's seem to serve a double purpose here. Maybe better to use "X^{\ast}" or something, for X_i with the highest EV?

Your above comment: "This doesn't find the value of researching ALL charities, only one" -- is that still relevant or has it now been fixed? I think the latter. 

The VOI simulation calculation discussion you give ("This expectation of value obtainable in the informed state) is super-interesting to me, not sure I've seen this laid out before, I love it!. 

Maybe you could add 1 or 2 more lines or footnotes to make it a bit clearer; I had to think through 'yeah these distributions represent our beliefs over the probability of each actual value for each charity ... so sampling from these reflects what we can expect to learn.' Maybe just took me a moment because I didn't start out as a Bayesian.

Is there any basis for 'spend 1/10th of the max'? It seems super-arbitrary. Any evidence or theoretical justification for this?

On a related note, one could make a case that the 'benefit of research for informing Givewell's decisions' might understate the global benefits, if the research informs other research and decisionmaking outside of GW.

> amount of money we have to available (A), and the cost of research (B), is

use easier-to-remember variable names?

I would replace 'budget' with 'funds they expect to influence'; considering the time horizon for this is difficult though, it probably requires a model of how the benefit of the research evolves over time, and some sort of explore/exploit modeling

Finger-exercise: If we think GW influences 500 million per year over the next 10 years, or 5 billion in total, 11% is 550 million, 1% 55 million. ... or 5.5 million per year. (But as I said before, I have no idea where that '1/10th rule' came from). 


I guess this assumes that all of the money they shift are allocated to the single top-EV charity.

https://www.givingwhatwecan.org/charities/givewell-unrestricted-fund suggests their operating expenses are about $11 million per year. But:
- that's not all research obviously 
- on the other hand, there are many others doing research in this space
- on the other hand,  there are benefits outside GW

Tough one, but it's not obvious to me they are underspending!

But wait, I forgot about the arrival of new charities! That might be worth modeling, if we had access to some relevant data and figures. (OK you are starting this below, cool.)

The CEA this is based on still needs a lot of work

I see a potential 'buried headline' here. It seems as though Helen Keller dominates the more or less 'dominates' the other charities: even its bottom 10th percentile is above the 50th percentile for all the others, and above the 75th percentile of all but New Incentives (and the latter has extremely wide intervals). 

Stepping outside the model itself, how confident are you in this result?

Feel free to respond here or on the forum, when you have time (https://forum.effectivealtruism.org/posts/Nb2HnrqG4nkjCqmRg/quantifying-uncertainty-in-givewell-cost-effectiveness?commentId=6fsXJnbo2npCKcKe2#comments) if you prefer.

Changed moral uncertainty write-up; would appreciate someone to look it over!

Correlated uncertainty driven by *using the same parameters across models*: I'd love to see this, I expect it will effect the VOI calculations. 

Also could be a good gateway towards the more difficult correlated uncertainty: the correlation between parameters *within* a model. I expect some of these to be highly correlated, and this might matter a lot. In fact, in the fistula model, some of these might largely be different takes on the same uncertain underlying variable? Here one might need to report on the VOI for something like a 'cluster of variables'?

But this is more challenging; I guess it would probably start with some calibrated judgement/educated guessing on the correlation parameter?

> We do this by assuming the charities that we don't want to research have already reduced all their uncertainty and are set to their expected value,

Seems strange to me. Wouldn't it make more sense to assume those charities have *not* been researched any further?