It also allows users to choose their own parameters to generate predictions relevant to their own decisions.

I wouldn't say "Google Sheet and therefore" ... you could probably represent uncertainty with a google sheet, although it can be awkward, messy, and limited. Not ehte 'how to measure everything' book tries to do this (annoyingly) with standard spreadsheets.

Some grammar fixes (not sure how I can edit without annoying forking) ... "that change" --> "that changes"

Did you actually use formal Bayesian procedures anywhere? (Note "bayesian --> Bayesian" as far as I know)

This is not fair because the percentages cannot be compared in such a way. We should to total transfer over total spending

The beginning of this needs to make it clear it's talking from the perspective of Givewell's specific model, not Givedirectly in general.

Another link to the model here would be good

In an ideal world (and I know this could take a lot of time and engineering) I'd like to see a very clear statement of the full equation up top, with each component allowed to expand, and then each base component hyperlinked/ roll-over-views to the section discussing/defining it. Something like the thing at the bottom ``` value_per_dollar_after_all_adjustments = total_adjustment_factor * value_per_dollar_after_charity_adjustments ``` "value_per_dollar_after_all_adjustments" would expand into... ``` total_adjustment_factor = changes_in_ppp * developmental_effects * reduced_morbidity * child_mortality_effects ``` But maybe with abbreviations to enable a simple statement in latex math, with the roll-over hyperlinks to definitions and discussions

Good point Ozzie! Fixed it up a tad. That would be pretty cool David! However, That would take a long time and I'd want to be sure that this representation would be worth all the time and effort it would take to create it. So I might not pursue this path for now.

That's fair. Main thing is 'seeing forest for the trees'. Perhaps there's a halfway version of this you could do, where you at least state the equation or key equations up top, but don't fully let them expand or link them?

But I think it's something I could do with maybe 4-8 hours, if its easy to write on this platform.

Yeah, I was thinking of adding the math under some headers. I'll look into that

"GiveWell models start with a donation size in USD." -> ?? All GiveWell models? This specific one?

All of them

It must be this certain class of model. Like, GiveWell must have made some other kinds of models somewhere. At least condition this somehow, like "GiveWell cost-effectiveness models typically begin with a fixed donation size. The unit is set to USD."

Oh, do you mean like non-cost effectiveness models? This is true, I'm probably talking about "All Cost Effectiveness Models", fixed.

"We came up" -> Who is we? Sam?

Can you get the footnoes to hover?

Even a tiny bit more reasoning transparency would be helpful here. What part of your analysis gave you the 90% CI of 0.75 to 0.9?

I certainly could! There's many many different ways I could refine this

Maybe this could be named something more specific, like "household_grant_size"?

Small note, but a well-labeled histogram of household size would be more useful.

I take it there's some bug here?

Yes there is! It's the first thing I want to fix. Reloading this cell will fix it

Fixed!!!! Didn't even require a PR!

The scientific notation for relatively small numbers seems unecessarily illegible?

Yea; I think we clearly have improvement to do here, but that would happen more on the Squiggle level, as opposed for this notebook alone.

Hypothetically, could this have been done symbolically in Guesstimate? Just a symbolic answer for the pointwiseProduct of two gamma distributions?

Also, I think this formal bayesian update might have been overkill for this model.

Also, another way of doing this might have been to use bootstrapping from the underlying data

This is true, it is a bit overkill. You probably could do this with pointwise operations? I don't fully grok it yet to trust it though (especially thinking about how multiple observations are compounded to update the prior)

Entirely my error though! I'll think about this one

This title isn't very informative

I think this is a decent resolution

"pay off" -> It might be "payoff", I'm not sure. https://www.wordwebonline.com/en/PAYOFF

"The distribution was estimated as beta(32, 50) based on a mean of 39%." -> This really isn't clear to me; how did you come up with the stdev? I guess it would be better to have functions for this in Squiggle (maybe a bit later), like, ``beta({mean:a, stdev:b})``

Would be good to have that feature (`beta({mean: a, stdev: b})`), I think it also serves a communication purpose in Squiggle. As for the stardard deviation, it was guessed/eyeballed by Nuño. I have not much else to say

Another bug? It's blank

Can we have the heading/sections as a siderbar which I can continue to see as I scroll down? It helps to see where i am, in context.

I would love that! I was looking Sadly, I don't think we can with observable.

I was looking to do this before going with a table of contents

Worth adding as a feature?

Oh, that's not something I control. I mean it's not something I can do with this platform (Observable) not Squiggle. To add this I would need to submit a major piece of work to this: https://github.com/observablehq

got it. I guess I need to understand what Observable is/does

Wow! Maybe Joel McGuire of HLI can give his perspective on these considerations?

Would love to have Joel's thoughts on this model. The considerations above are all just internal inconsistencies (As in, the model is saying two things that are inconsistent, which I love nitpicking), and I think Joel is more familiar with the external ones. Would be nice to put Joel's thoughts here.

Personally (and obviously I'm not the funder here) I'd like to see at least 1 more GW model, so we can really enable comparisons. .. and maybe that suggests a need for side-by-side views. Also, for a robust PoC it would be helpful to include at least one 'more complicated' GW model, as GD is sort of one-dimension simpler

I agree! I just want to verify that GW agrees with us before going further xD

I really like this and where it's going! One thing I am missing a little bit here is a sort of 'dashboard view' ... "adjust all inputs and see how the output changes" ... as in the Causal app. See their example here: https://my.causal.app/models/69949 for a pretty nice instance of this.

I'm not really sure what to do here, as the GiveDirectly model doesn't actually have any obvious "Inputs"?. I mean possibly, there is: - Donation size, which is arbitrary, and ends up just being multiplied by doublings of consumption per dollar to create, which is already at the top - Discount rate, which I think has been considered well enough (has low enough uncertainty) to not be considered a model input. All the other GiveWell models do have things like moral weights, but that's because they are all being compared to GiveDirectly, so they need the conversion factor (lives saved to doublings of consumption). In this case, GiveDirectly is the baseline, this model is mostly economical and not really ethical

OK good point. > Donation size, which is arbitrary, and ends up just being multiplied by doublings of consumption per dollar to create, which is already at the top Yeah this one would be nice to remove if it were feasible, as I've noted > Discount rate, which I think has been considered well enough (has low enough uncertainty) to not be considered a model input. I think it probably *should* be a model input ... but it's not a *moral* parameter here, is it, as this is not about 'pure discounting of future utility in a value function' > In this case, GiveDirectly is the baseline, this model is mostly economical and not really ethical I think one probably *could* rework this in a way that involves moral parameters, but it's probably not the best use of time? Would be better to do one more model ... bednets or some such, where its much easier to bring in the moral uncertainty parameters.

I was about to disagree with you about donation size, but now I've thought about it and now agree with you. I'll make that change. Discount rates are not a moral parameter here at all, it's a calculation done to include the increase of consumption in other countries due to people getting richer. I might bring it up the document and turn it into paramater. As for GiveDirectly having moral parameters, you absolutely *could*, but then it wouldn't really be a baseline. If I was to make another model (Which I would love to, mainly waiting on whether GiveWell would appreciate it) I would include moral weights and uncertainty.

In fact, GiveDirectly use to have a moral weight of the value of an e increase in consumption. Although it was a dubious one, as it was actually just 1.44, as they defined value as a doubling of consumption, so was just changing the base of the log. I chose to just use the correct base for the log so that it didn't look like it was a considered moral weight when it wasn't (see note 4 from changes to GiveDirectly's model)

Also, there are other externalities that are put at the bottom of the model that could be uncertain and moral. Like, how much is the reduction of mortality from GiveDirectly worth vs the increase in consumption? I could extend in that direction.

that would be interesting

> As for GiveDirectly having moral parameters, you absolutely *could*, but then it wouldn't really be a baseline. If I was to make another model (Which I would love to, mainly waiting on whether GiveWell would appreciate it) I would include moral weights and uncertainty. I'm thinking about this. It's true you need to have something as a benchmark, or something "externally defined" like ... cost of saving 1 life of a child under age 8, or 1 DALY lost/gained. Maybe I find it a bit strange that the 'unit of value' is defined as a GD transfer of a certain size (or is it a certain amount of income gained?) ... but if that's the case then I guess it's true that adjusting the 'value of the income gain' doesn't have a strong meaning ... in what units would it even be? But once we are comparing it to another charity/model -- or other outcomes (like those external ones above) ... then the moral parameters (something something value of additional income) probably will be more meaningful. And you would be able to apply them 'within the GD model', if it's phrased in terms of "value in terms of equivalent DALY" or some such. Without such a comparison the moral parameters in the GD model have no real impact on the 'ultimate outcome' ... *but* they might affect the sensitivity of the model to other things. (Less interesting but still worth considering). E.g., if the marginal value of income diminishes more steeply in income, then the 'size of the family sharing the transfer' or 'size of average transfer' parameter would be more important.

My favorite 'killer apps' here 1. Enabling explicit uncertainty allowings reasoning-transparent and justifiable quantitative evaluations of *a wider set of interventions*. We can evaluate cases where we have less scientific data than for the ones GW targets. We don't have to only 'look under the streetlamp'. 2. Allowing users to input their own 'moral parameters' ... of their social welfare function (how much do you value years added at different ages, increases in population, etc.)

I would honestly love to see a "Squiggle showcase" notebook, where we go about giving examples of a large amount of different cool things you can do with Squiggle (forecasting and scoring, value of information, moral parameters etc)

Quick flag, that it would be nice if there were a section in the appendix or similar, listing all of the things Squiggle could improve on that would help this model.

Lol, there is

Appendix A

The graph box is much broader than the text box on my screen; it'd look neat if they had the same margins.

I think this is fixed, at least for me

Is it necessary to show this code? I would slightly prefer to just see the ouputs.

Could you explain what you're doing here? It seems cool! What are you specifying about the prior? I can't tell at a glance.

So whenever you say distribution(x, y) that typically means 95% CI x - y? Sorry if you explained this earlier and I didn't see. If you didn't, I think it's worth trying to help out the lazy skimmers out there, because they make up most readers.

In this case, it means a beta distribution with the alpha parameter being 32 and the beta parameter being 50. However, I'm about to fix this so it's easier to see where it comes from.

Did you just choose symmetrical / normalish distributions for ease? I assume this would be more right skewed, with more people investing less than than the mean than above it.

I choose symmetric distributions because that's the shape of a beta distribution, and a beta distribution is usually a default choice for portions and probabilities. I'm not sure why you would expect this number to be skewed? Keep in mind that this isn't the portion invested by people, but the distribution of the expected amount invested per person.

Why are you doing a relatively complicated thing if you don't think it'll change things? Why not just say "lognormal(mean = observed_consumption, SD = observed_SD)"?

I think positive spillovers seem pretty plausible, so this is one of the parameters where uncertainty specification is a real boon!

Does the 1.03 mean you're adjusting upwards based on the possibility of leakage / cash not reaching? I'm not sure that makes sense if so...

It'd be really good to know how how much of a difference these make!

Unfortunately, I learned quickly not to click on footnotes because it reloaded the entire page (which I expect will not be instant for most people). This was too much friction so I decided to ignore them.

"This is very convenient, as without this, as it is very performant and easy to calculate." I'm not sure this sentence makes sense.

> GiveWell represents their GiveDirectly model in a Google Sheet and *therefore* does not represent uncertainty with their calculation. The 'therefore' is not entirely correct. There are ways of embodying uncertainty in sheets, and there may even be plugins. But it doesn't come naturally

The hyperlink https://www.squiggle-language.com/docs/Features/Functions#lognormal-distribution is a dead link

Was looking it up because I was surprised the default was lognormal and not normal ... although I guess that makes sense for something bounded like the transfer rate

I've just updated this notebook considerably. It has a lot of nicer features now! Like equations and functions! I'll talk to you about this in our meeting.

How would an average user know this Beta distribution is the equivalent of lognormal with 0.75 - 0.90 90% CI?

This is halfway through a large edit. I'll need to update the text

--> "goes to each household" ? "this is important because of assumed diminishing returns" (and what else?) Important to understanding 'why this matters' in setting these parameters. That is another reason to try to put the 'expandable equation' (there is another fancy word for this I am forgetting, a math person told me) at the top.

^ It was Quinn Dougherty I spoke to about that

If you refresh, you'll get the expandable equations or at least a version of them