2c:null 2d:["$","title",null,{"children":"Quantifying Uncertainty in GiveWell's GiveDirectly Cost Effectiveness Analysis | Sam Nolan | Observable"}] 32:T614,> As for GiveDirectly having moral parameters, you absolutely *could*, but then it wouldn't really be a baseline. If I was to make another model (Which I would love to, mainly waiting on whether GiveWell would appreciate it) I would include moral weights and uncertainty. I'm thinking about this. It's true you need to have something as a benchmark, or something "externally defined" like ... cost of saving 1 life of a child under age 8, or 1 DALY lost/gained. Maybe I find it a bit strange that the 'unit of value' is defined as a GD transfer of a certain size (or is it a certain amount of income gained?) ... but if that's the case then I guess it's true that adjusting the 'value of the income gain' doesn't have a strong meaning ... in what units would it even be? But once we are comparing it to another charity/model -- or other outcomes (like those external ones above) ... then the moral parameters (something something value of additional income) probably will be more meaningful. And you would be able to apply them 'within the GD model', if it's phrased in terms of "value in terms of equivalent DALY" or some such. Without such a comparison the moral parameters in the GD model have no real impact on the 'ultimate outcome' ... *but* they might affect the sensitivity of the model to other things. (Less interesting but still worth considering). E.g., if the marginal value of income diminishes more steeply in income, then the 'size of the family sharing the transfer' or 'size of average transfer' parameter would be more important.33:T589,This notebook is a [Squiggle](https://squiggle-language.com/) model for [GiveWell's GiveDirectly cost-effectiveness analysis](https://docs.google.com/spreadsheets/d/1Kq6iHSQFr3eRz1p9KclHuJTQiaJYkpViOyneSd9KCJc/edit#gid=1680005064). GiveWell represents their GiveDirectly model in a Google Sheet and does not represent uncertainty with their calculations. In contrast, this model handles uncertainties using probability distributions. This notebook is interactive. Changing the parameters and calculations will update the results. It is also a [living document](https://en.wikipedia.org/wiki/Living_document) as you can [leave comments](https://observablehq.com/@observablehq/comments) and [submit changes](https://observablehq.com/@observablehq/fork-suggest-merge). For the most part, this model is a transposition of GiveWell's GiveDirectly model. I add uncertainty to parameters with two methods: - If there was enough information about the parameter, a formal Bayesian update was performed to arrive at a fairly accurate estimate of the uncertainty. - If there wasn't as much information, I guessed the uncertainty with the help of Nuño Sempere, a respected forecaster. These estimates were basic and could be better estimated in the future. I also found [an internal inconsistency](#cell-1830) that changes the output in GiveDirectly's model. You can fix this by checking a box at the bottom of the notebook.34:T4b0,## Motivation for this work GiveWell's CEAs are considered somewhat of a gold standard within EA. This work was originally motivated by wanting to see how far forward we can push that standard. There are a number of things that could be done to improve GiveWell CEAs. This document focuses on Quantified Uncertainty. This notebook attempts to quantify the uncertainty around GiveDirectly's CEA. We do this by letting parameters in the calculations be represented by probability distributions, and seeing how those probability distributions combine to represent uncertainty in the final cost effectiveness. This serves a dual purpose of demonstrating Squiggle. I'm very excited for the possibilities available with Squiggle and Squiggle notebooks. Squiggle notebooks allow you to quantify uncertainty, but also allow you to write and explain the assumptions of the model in text form. They include links, images, tables, and can pull data from other locations. It's also much easier to use and learn than a full probabilistic programming language. I hope that others will be inspired to create their own Squiggle notebooks for their evaluations and estimates, and enjoy the process as much as I did.35:T4d5,## Notes on the implementation To see the code that was used to create this notebook, hover over a cell and click the caret to the left. The text cells will mostly show you markdown, but the Squiggle cells will show you JavaScript. Observable is a JavaScript notebook platorm, and we've used it to embed Squiggle notebooks inside. You can change the text in the cells to see how it would change the model. Please note that currently this isn't that performant. [We are working on improving the performance to make a better experience](https://github.com/quantified-uncertainty/squiggle/issues/465). Changing the text does not change the model for anyone else but yourself, and refreshing the browser will remove your changes. If you wish to persist your changes, you can fork this model by clicking the fork button in the top right of the notebook. Then look into the JavaScript and edit the squiggle string passed into the `cell` or `partial` function. If you wish, you can even suggest changes that I may merge into this one, making this notebook a collaborative document. This is all provided by observable. If you want to learn more about Squiggle notebooks, check out [our tutorial](https://observablehq.com/@hazelfire/squiggle).36:T811,$`\begin{aligned} \text{grant size} &= \frac{\param{household grant size}}{\param{household size}} \\ \text{amount invested} &= \param{portion invested} \cdot \ref{grant size} \\ \text{amount consumed} &= (1 - \param{portion invested}) \cdot \ref{grant size} \\ u(x) &= ${isoelastic_utility ? `\\frac{x^{1 - \\mu} - 1}{1 - \\mu} \\\\ \\mu = 1.59` : `\\log_e x`}\\ \text{initial consumption} & = (\href{#cell-2000}{u}(\frac{\text{\href{#amount_invested}{amount consumed}}}{\param{duration initial consumption}} + \param{baseline consumption}) \\ &- u(\param{baseline consumption})) \cdot \param{duration initial consumption} \\ PV(p, d, t) &= p \int_0^t (1 + d)^{-x} dx = p\frac{1 - (1 + d)^{-t}}{\log_e(1 + d)}\\ \text{value from roi} &= \href{#utility}{u}(\param{return on investment} \cdot \ref{amount invested} + \param{baseline consumption}) \\ &- \href{#utility}{u}(\param{baseline consumption}) \\ \text{value from investment} &= PV(\text{value from roi}, \param{discount rate}, \param{investment duration} - 1) \\ \text{consumption last year} &= \textit{\param{baseline consumption}} + \ref{amount invested} \\ & \cdot (\textit{\param{return on investment}} + \param{portion investment returned})\\ \text{value resell} &= (\href{#utility}{u}(\text{consumption last year}) - u(\param{baseline consumption})) \\ &/ (1 + \param{discount rate})^{\param{investment duration}} \\ \text{total value including spillover} &= \ref{total value} \cdot \param{negative spillover} \\ \text{value per household} &= \text{total value including spillover} \cdot \param{household size} \\ \text{value per dollar} &= \param{transfer efficiency} \cdot \frac{\text{value per household}}{\param{household grant size}} \\ \text{value doubling consumption} &= u(\param{baseline consumption} \cdot 2) - u(\param{baseline consumption}) \\ \text{doubling consumption per dollar} &= \frac{\text{value per dollar}}{\text{value doubling consumption}} \\ \text{value after adjustments} &= \text{doubling consumption per dollar} \cdot \prod_{c \in \param{adjustments}} c \end{aligned} `37:T4bf,The recipient is then modelled to invest the other portion of their grant. The return on investment from the grant was estimated in GiveWell analysis. Here's a discussion from GiveWell: > **Notes from former GiveWell Research Analyst Chris Smith**: I would guess that roofs are an unusually good investment item. I expect that most other assets GiveDirectly's recipients invest in offer lower returns for fewer years. > > It's also important to account for the fact that many items counting as assets (for the sake of the suggested 39% input for "Percentage of transfers invested") aren't interest-bearing (at least in the conventional sense). For example, many people use transfers to buy furniture. > **Notes from GiveWell Senior Fellow Caitlin McGugan**: > Taking the conservative side estimate of the return to iron roofs. I agree with Chris that there’s probably a mix of investments, some of which is in durable consumption goods that don’t earn a return, bringing the average rate of return down. But I also think this is partially compensated by people selecting into non-roof investments (such as livestock) because they believe they’ll be more successful than average in that type of investment. 38:T516,The investment period runs over a duration. This duration is a very uncertain value because follow up studies are yet to be performed by GiveDirectly. Here are some discussions on the value by GiveWell staff: > **Notes on this input from GiveWell Senior Fellow Caitlin McGugan**: > This input is very uncertain as we lack sufficiently long follow-up evidence for GiveDirectly recipients. I'm choosing a value that I think is relatively optimistic but somewhat below the expected lifespan of a roof or a goat. > On the one hand, I think the transfers and some investments made using them can help buffer households from income shocks, prolonging the time until they’re forced to consider selling assets. On the other hand, I think negative income shocks are frequent in these populations, and there’s a good chance that households won’t be able to hang onto their investment over the long term. The balance of evidence I'm aware of from other types of transfer programs suggests that it's challenging for households to maintain asset investments over the very long term. > **Notes on this input from GiveWell Senior Advisor Michael Eddy**: Recent evidence (from Uganda, Ethiopia & Nairobi) makes me generally pessimistic on the long-term (> 5 year) impacts of GiveDirectly-style cash transfers.39:T4c4,The main part of the analysis is done. In the next section, we go through possible positive and negative adjustments to the impact based on other considerations that are not inside the model. Consider all numbers here to be percentages that get multiplied by the final result, like an adjustment of 1 means 100%, and is actually no adjustment. Similar adjustments are considered for all investments, as of such a number of these adjustments are simply not applicable for GiveDirectly. ### Risks of wastage The adjustments are defined by GiveWell as follows: > **double_treatment** The program is delivered to people who don't benefit or minimally benefit because they have received the program from another source. > > Not applicable for GiveDirectly > **ineffective_goods** Pills, LLINs, etc. are poorly made, counterfeit, or otherwise largely or fully ineffective at achieving the expected outcome. > > Not applicable for GiveDirectly > **cash_doesnt_reach_participants** Cash doesn't reach recipients or is taken back from them inappropriately > > we use the amount that GiveDirectly sends to recipients, not adjusted for cash that doesn't ultimately reach recipients through bribery/theft or cases of staff fraud.3a:T57a,These considerations are defined by GiveWell as follows: > **change_of_priorities** Charity decides to use some of the funding we directed to them to fund some research or other work that they see as being related to the program, but we don't find valuable. This would affect our cost per output analyses once it happened, so in many cases, no additional adjustment is needed. May be worth adjusting for for charities that we have less experience with. > > Our cost per dollar delivered calculation captures spending on priorities other than delivery of cash transfers. > > **non_funding_bottlenecks** Charity encounters non-funding bottlenecks to allocating funding and holds funding for a long time. At an extreme, charity is unable to spend money, draws down funding over time on work to identify opportunities to spend money for a time, and then shifts to another program. > > Has been able to spend funding quickly. > > **within_org_fungibility** Charity runs multiple programs and shifts funds from non-GiveWell sources or fundraising effort out of the program that we want to fund as a result of receiving GiveWell-directed funding. > > All funds support cash transfer programs, however some programs may be less cost-effective than the program we model in our cost-effectiveness analysis and there may be some fungibility across programs, particularly in how fundraising staff spend their time.3b:T67f,viewof read_params = cell(`params(x) = ({ "change_of_priorities": x["Change of priorities"], "non_funding_bottlenecks": x["Non-funding bottlenecks"], "within_org_fungibility": x["Within-org fungibility"], "misappropriation": x["Misappropriation without monitoring results"], "bad_monitoring": x["False monitoring results"], "changes_in_ppp": x["Changes in PPP and exchange rates"], "developmental_effects": x["Developmental effects"], "reduced_morbidity": x["Reduced morbidity"], "child_mortality_effects": x["Child mortality effects"], "double_treatment": x["Double treatment"], "ineffective_goods": x["Ineffective goods"], "cash_doesnt_reach_participants": x["Goods purchased and left in storage until they expire / Cash doesn't reach recipients or is taken back from them inappropriately"], "household_grant_size": x["Total size of transfer per household (nominal USD)"], "household_size": x["Average household size"], "discount_rate": x["Discount rate"], "portion_invested": x["Percentage of transfers invested"], "transfer_efficiency": x["Transfers as a percentage of total cost"], "return_on_investment": x["Return on investment"], "portion_investment_returned": x["Percent of investment returned when benefits end"], "negative_spillover": x["Discount to the benefits of cash from potential negative spillover effects on consumption"], "baseline_consumption": x["Baseline annual consumption per capita (in nominal USD)"], "duration_initial_consumption": 1, "investment_duration": x["Duration of investment benefits (in years)"] }) givewell_params = params($params) 0`, [], {jsImports: {params: param_imports}})3c:T74a,viewof my_params = cell(`arbitrary(x) = (0.9 to 1.1) * x uncertain_absolute(x, r) = normal({ p5: x - r, p95: x + r }) dictMap(x, f) = Dict.fromList(map(Dict.keys(x), {|key| [key, f(key, x[key])]})) adjustment_uncertainties = { change_of_priorities: 0.0001, // transfer_efficiency already includes this uncertainty non_funding_bottlenecks: 0.02, // Spends money fairly quickly within_org_fungibility: 0.05, // Possibly funges money through fundraising misappropriation: 0.01, // Very well monitored bad_monitoring: 0.01, changes_in_ppp: 0.05, // Not sure, likely to be small developmental_effects: 0.05, // Developmental effects could make a big change reduced_morbidity: 0.04, // Morbidity could also be a big change child_mortality_effects: 0.04, // Not sure about mortality effects "double_treatment": 0.0001, "ineffective_goods": 0.0001, "cash_doesnt_reach_participants": 0.01 } portion_stdev = { "discount_rate": 0.01, "portion_invested": 0.02, "transfer_efficiency": 0.05, "negative_spillover": 0.03, } lognormal_stdev = { "household_grant_size": 50, "return_on_investment": 0.03, "portion_investment_returned": 0.03, "duration_initial_consumption": 0.3, "investment_duration": 4, "household_size": 0.7, } rest = { "baseline_consumption": SampleSet.fromDist(lognormal({mean: givewell_params.baseline_consumption, stdev: 20}))} adjustment_params = dictMap(adjustment_uncertainties, {|key, v| uncertain_absolute(givewell_params[key], v)}) portion_params = dictMap(portion_stdev, {|key, v| beta({mean: givewell_params[key], stdev: v})}) lognormal_params = dictMap(lognormal_stdev, {|key, v| lognormal({mean: givewell_params[key], stdev: v})}) givedirectly_params = Dict.merge(adjustment_params, Dict.merge(portion_params, Dict.merge(lognormal_params, rest))) 0`, [household_size, baseline_consumption, read_params]) 3d:T406,## Discussion Quantifying uncertainty eventually found that the cost effectiveness of GiveDirectly varies by about an order of magnitude and is approximately lognormally distributed. The cost effectiveness however doesn't have particularly large tails. The benefits of the quantification of uncertainty have not yet been fully realised with this document. From this, it could be possible to calculate the [value of information](https://en.wikipedia.org/wiki/Value_of_information). In the future, I would love to continue this work in quantifying uncertainty for the rest of the GiveWell models, or possibly bringing back quantification for the [ACE models](https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/#ce). I had a lot of fun writing this Squiggle Notebook. It offers a large collection of advantages over other alternatives: - Ability to write detail inbetween cells - External links - Ability to quantify uncertainty - Ability to create calculators and interactive analysis 3e:T5be,### Isoelastic Utility The [calculated discount rate assumes](https://docs.google.com/document/d/1g2tf7Fu8-iG_8TgnJkEx9G5TlEyrcv7PXPokmPZaUMc/edit) that welfare is an [isoelastic](https://en.wikipedia.org/wiki/Isoelastic_utility) to consumption with ${tex`\eta=1.59`}, but the rest of the model assumes that welfare is isoelastic to consumption with ${tex`\eta = 1`} (aka, logarithmic). [To quote](https://docs.google.com/document/d/1g2tf7Fu8-iG_8TgnJkEx9G5TlEyrcv7PXPokmPZaUMc/edit): > Increases in consumption over time meaning marginal increases in consumption in the future are less valuable. We chose a rate of 1.7% based on an expectation that economic consumption would grow at 3% each year, and the function through which consumption translates to welfare is [isoelastic](https://en.wikipedia.org/wiki/Isoelastic_utility) with eta=1.59. Considering ${tex`\eta`} has been [estimated as 1.5 in the United Kindom](https://link.springer.com/article/10.1007/s10640-018-0242-z) and [1.4 in other OECD countries](https://www.jstor.org/stable/24440019), this model should also probably consider welfare to be an isoelastic function of consumption. This would increase the effectiveness of cash transfers. I've created a [desmos calculator to explore this concept](https://www.desmos.com/calculator/a0xtpziga4). Changing the utility function to be isoelastic is an enourmous change. **Fixing this increases GiveDirectly's cost effectiveness by a factor of 11%**.3f:T4d5,1. This model sometimes double counts consumption increases. For instance, if the duration of the and the investment returns come during the same period of time, however the increases of consumption during that period come from the same baseline, therefore double counting the increase in marginal consumption over the first year. I don't believe fixing this problem would change the results that much, and decided not to include it. 2. The adjustments section do not have any uncertainty associated with them, even though these parts are probably the most uncertain. These adjustments are also moraly uncertain. 3. There are two other people who have done extremely similar work, [Cole Haus](https://forum.effectivealtruism.org/posts/gMxTEMvh8RttX9Nt4/uncertainty-and-sensitivity-analyses-of-givewell-s-cost) and [Joel McGuire](https://www.happierlivesinstitute.org/report/cash-transfers-cost-effectiveness-analysis/). It would be valuable to compare this work to their results. 4. It would be good to find out exactly how much uncertainty comes from our estimates vs more formal bayesian analysis. 5. It could be possible to look into the value of further research in GiveDirectly based on comparing it to other GiveWell interventions.40:T451,appendixc = md`## Appendix B, Notes on Bayesian Inference Most parameter choices here are based on estimates. I worked with Nuño Sempere to refine these estimates and make sure they were appropriate. However, there were two parameters (household sizes and baseline consumption) which we had enough data to do Bayesian inference on. This appendix goes over the methods used to run the Bayesian inference. Household sizes were assumed to follow a [Poisson Distribution](https://en.wikipedia.org/wiki/Poisson_distribution). A Poisson distribution takes one parameter, ${tex`\lambda`}, that is the expected value of the distribution. Because we care about the distribution of the expected value, we want to find out what the distribution of ${tex`\lambda`} is based on the available data. For baseline consumption, I assumed that income was lognormally distributed. However, if you log a lognormal distribution you arrive at a normal distribution. So I simply log the observations and run bayesian inference over a normal distribution. Details on the math are provided on the following external notebook.`41:T425,compare_givewell = (name, cell, index) => { let formatPercent = (x) => new Intl.NumberFormat('en-US', {style: 'percent', maximumSignificantDigits: 3, signDisplay: "always"}).format(x) let result = project.getResult(cell).value let env = project.getEnvironment() let givewell = parseFloat(gw_gd_cea.filter(x => x["Cash Transfers - GiveDirectly"] === name)[index ?? 0]["Overall"].replace(",","")) if(result.tag === "Number"){ let value = result.value; let difference = (value - givewell) / givewell; return Inputs.table([{ "GiveWell": formatFloat(givewell) ,"Value": formatFloat(value), "Difference": formatPercent(difference)}]) } else { let dist = result.value let mean = dist.mean(env).value let lowCI = dist.inv(env, 0.025).value let highCI = dist.inv(env, 0.975).value let difference = (mean - givewell) / givewell return Inputs.table([{ "GiveWell": formatFloat(givewell) ,"Mean": formatFloat(mean), "95% CI": `${formatFloat(lowCI)} - ${formatFloat(highCI)}`, "Difference": formatPercent(difference)}]) } }42:[{"id":"8ad0773214d8e7f14ef0a9adcc4679f7e6e1bd9a5f9a6037dbb852d52a33c0f68537840d1c83df3fd2ed396a66edb1bd472f694308052c3e50c3c04ed354c198","href":"https://static.observableusercontent.com/files/8ad0773214d8e7f14ef0a9adcc4679f7e6e1bd9a5f9a6037dbb852d52a33c0f68537840d1c83df3fd2ed396a66edb1bd472f694308052c3e50c3c04ed354c198","name":"household size analysis - Family size by household.csv","type":"text/csv","lastModified":1651337283252.673,"size":1634},{"id":"cabadb0a43d285c8374d6031d96779e0d7bb43f4841026b58605bf1fbf06279fa6aed08a46a6b657a4c9b6ab637bece0251be36658cb3e5fa3321be04ef3d579","href":"https://static.observableusercontent.com/files/cabadb0a43d285c8374d6031d96779e0d7bb43f4841026b58605bf1fbf06279fa6aed08a46a6b657a4c9b6ab637bece0251be36658cb3e5fa3321be04ef3d579","name":"gw_gd_cea.csv","type":"text/csv","lastModified":1666755892307.117,"size":4124}] 30:{"id":"38cfea3f6eaa5311","slug":"givewells-givedirectly-cost-effectiveness-analysis","title":"Quantifying Uncertainty in GiveWell's GiveDirectly Cost Effectiveness Analysis","thumbnail":"cfbe44ec27a23a5771699c1b2b8a04e26297bcff6ddfeecea4200f2ecf0e5b4e","access_level":"public","create_time":"$D2022-03-18T03:26:29.559Z","update_time":"$D2023-01-12T05:06:53.228Z","view_time":null,"first_public_time":"$D2022-03-18T11:53:57.781Z","fork":false,"stars":6,"starred":false,"owner":{"type":"team","id":"1fff94fffef46139","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":"starter_2024"},"version":5383,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}],"members":[{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null,"role":"editor"}],"comments":[{"id":"6912541f2b37d720","reply_to_id":null,"content":"It also allows users to choose their own parameters to generate predictions relevant to their own decisions.","create_time":"$D2022-04-30T20:47:52.558Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"12eb0aac05ce8647","reply_to_id":null,"content":"This is not fair because the percentages cannot be compared in such a way. We should to total transfer over total spending","create_time":"$D2022-05-03T18:50:32.649Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"ed19e4380ff51322","reply_to_id":null,"content":"The beginning of this needs to make it clear it's talking from the perspective of Givewell's specific model, not Givedirectly in general.","create_time":"$D2022-05-05T21:38:26.380Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"93dd3d1a40d598ea","reply_to_id":"ed19e4380ff51322","content":"Another link to the model here would be good","create_time":"$D2022-05-05T21:38:45.535Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"fd01b58f4f5b5bc8","reply_to_id":null,"content":"\"GiveWell models start with a donation size in USD.\" -> ?? All GiveWell models? This specific one?","create_time":"$D2022-05-05T21:44:08.782Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"13e76feadfe4e4cd","reply_to_id":null,"content":"\"We came up\" -> Who is we? Sam?","create_time":"$D2022-05-05T21:47:04.246Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"a060630d8aced81e","reply_to_id":null,"content":"Maybe this could be named something more specific, like \"household_grant_size\"?","create_time":"$D2022-05-05T21:59:29.115Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"ef4eec564ab927c6","reply_to_id":null,"content":"Small note, but a well-labeled histogram of household size would be more useful.","create_time":"$D2022-05-05T22:00:47.972Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"2167d9860e710afe","reply_to_id":null,"content":"I take it there's some bug here?","create_time":"$D2022-05-05T22:06:32.569Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"6eb6cbc8f575b5a6","reply_to_id":null,"content":"Hypothetically, could this have been done symbolically in Guesstimate? Just a symbolic answer for the pointwiseProduct of two gamma distributions?","create_time":"$D2022-05-05T22:07:44.410Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"cdf6f3cc691bb7d6","reply_to_id":"6eb6cbc8f575b5a6","content":"Also, I think this formal bayesian update might have been overkill for this model.","create_time":"$D2022-05-05T22:10:08.108Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"69eef2ce7bb90b3e","reply_to_id":"6eb6cbc8f575b5a6","content":"Also, another way of doing this might have been to use bootstrapping from the underlying data","create_time":"$D2022-05-05T22:12:59.505Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"82a2adf5966c8cc3","reply_to_id":null,"content":"This title isn't very informative","create_time":"$D2022-05-05T22:15:07.328Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"95d4130a7b157577","reply_to_id":null,"content":"\"pay off\" -> It might be \"payoff\", I'm not sure. https://www.wordwebonline.com/en/PAYOFF","create_time":"$D2022-05-05T22:18:30.510Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"d16d8023545b3b51","reply_to_id":null,"content":"Another bug? It's blank","create_time":"$D2022-05-05T22:20:54.361Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"bdd94aec364c0d14","reply_to_id":"95d4130a7b157577","content":"\"The distribution was estimated as beta(32, 50) based on a mean of 39%.\" -> This really isn't clear to me; how did you come up with the stdev? \n\nI guess it would be better to have functions for this in Squiggle (maybe a bit later), like, ``beta({mean:a, stdev:b})``","create_time":"$D2022-05-05T22:22:37.482Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"92468cb2cbf77994","reply_to_id":"fd01b58f4f5b5bc8","content":"All of them","create_time":"$D2022-05-06T15:18:23.392Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"67f97fbad895ae14","reply_to_id":"2167d9860e710afe","content":"Yes there is! It's the first thing I want to fix. Reloading this cell will fix it","create_time":"$D2022-05-06T15:19:09.833Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"42c5c063881428e9","reply_to_id":"ed19e4380ff51322","content":"In an ideal world (and I know this could take a lot of time and engineering) I'd like to see a very clear statement of the full equation up top, with each component allowed to expand, and then each base component hyperlinked/ roll-over-views to the section discussing/defining it. \n\nSomething like the thing at the bottom\n\n```\nvalue_per_dollar_after_all_adjustments = total_adjustment_factor * value_per_dollar_after_charity_adjustments\n\n```\n\n\"value_per_dollar_after_all_adjustments\" would expand into...\n\n```\n\ntotal_adjustment_factor = changes_in_ppp * developmental_effects * reduced_morbidity * child_mortality_effects\n\n```\n\nBut maybe with abbreviations to enable a simple statement in latex math, with the roll-over hyperlinks to definitions and discussions","create_time":"$D2022-05-08T16:02:41.820Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"e85f56775e9788a3","reply_to_id":null,"content":"Can we have the heading/sections as a siderbar which I can continue to see as I scroll down? It helps to see where i am, in context.","create_time":"$D2022-05-08T16:05:41.198Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"6acfb92ad92f658b","reply_to_id":null,"content":"Wow! Maybe Joel McGuire of HLI can give his perspective on these considerations?","create_time":"$D2022-05-08T16:07:11.422Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"0a8c314c845ceba3","reply_to_id":null,"content":"Personally (and obviously I'm not the funder here) I'd like to see at least 1 more GW model, so we can really enable comparisons. .. and maybe that suggests a need for side-by-side views.\n\nAlso, for a robust PoC it would be helpful to include at least one 'more complicated' GW model, as GD is sort of one-dimension simpler","create_time":"$D2022-05-08T16:08:44.919Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"b277d46ca627fe3c","reply_to_id":null,"content":"I really like this and where it's going! One thing I am missing a little bit here is a sort of 'dashboard view' ... \"adjust all inputs and see how the output changes\" ... as in the Causal app. \n\nSee their example here: https://my.causal.app/models/69949 for a pretty nice instance of this.","create_time":"$D2022-05-08T16:10:17.218Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"aad44782ff7878e6","reply_to_id":null,"content":"My favorite 'killer apps' here \n\n\n1. Enabling explicit uncertainty allowings reasoning-transparent and justifiable quantitative evaluations of *a wider set of interventions*. We can evaluate cases where we have less scientific data than for the ones GW targets. We don't have to only 'look under the streetlamp'.\n\n2. Allowing users to input their own 'moral parameters' ... of their social welfare function (how much do you value years added at different ages, increases in population, etc.)","create_time":"$D2022-05-08T16:15:58.002Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"a058463ea5571267","reply_to_id":"fd01b58f4f5b5bc8","content":"It must be this certain class of model. Like, GiveWell must have made some other kinds of models somewhere. At least condition this somehow, like \"GiveWell cost-effectiveness models typically begin with a fixed donation size. The unit is set to USD.\"","create_time":"$D2022-05-08T22:00:45.488Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"d4f4e50521981199","reply_to_id":null,"content":"Quick flag, that it would be nice if there were a section in the appendix or similar, listing all of the things Squiggle could improve on that would help this model.","create_time":"$D2022-05-08T22:05:18.200Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"651bb8047be32100","reply_to_id":"b277d46ca627fe3c","content":"I'm not really sure what to do here, as the GiveDirectly model doesn't actually have any obvious \"Inputs\"?. I mean possibly, there is:\n- Donation size, which is arbitrary, and ends up just being multiplied by doublings of consumption per dollar to create, which is already at the top\n- Discount rate, which I think has been considered well enough (has low enough uncertainty) to not be considered a model input.\nAll the other GiveWell models do have things like moral weights, but that's because they are all being compared to GiveDirectly, so they need the conversion factor (lives saved to doublings of consumption). In this case, GiveDirectly is the baseline, this model is mostly economical and not really ethical","create_time":"$D2022-05-09T14:30:28.274Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"d19aef4d9b7d143e","reply_to_id":"e85f56775e9788a3","content":"I would love that! I was looking Sadly, I don't think we can with observable.","create_time":"$D2022-05-09T14:38:14.316Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"1b007196d7af7498","reply_to_id":"e85f56775e9788a3","content":"I was looking to do this before going with a table of contents","create_time":"$D2022-05-09T14:38:39.343Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"e1c9868c146de188","reply_to_id":"aad44782ff7878e6","content":"I would honestly love to see a \"Squiggle showcase\" notebook, where we go about giving examples of a large amount of different cool things you can do with Squiggle (forecasting and scoring, value of information, moral parameters etc)","create_time":"$D2022-05-09T14:46:54.374Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"f7921917f51a555c","reply_to_id":"ed19e4380ff51322","content":"Good point Ozzie! Fixed it up a tad.\n\nThat would be pretty cool David! However, That would take a long time and I'd want to be sure that this representation would be worth all the time and effort it would take to create it. So I might not pursue this path for now.","create_time":"$D2022-05-09T14:52:21.256Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"9d4811b9227cdf52","reply_to_id":"fd01b58f4f5b5bc8","content":"Oh, do you mean like non-cost effectiveness models? This is true, I'm probably talking about \"All Cost Effectiveness Models\", fixed.","create_time":"$D2022-05-09T14:54:27.808Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"3de4518ce1c39357","reply_to_id":"82a2adf5966c8cc3","content":"I think this is a decent resolution","create_time":"$D2022-05-09T16:02:08.732Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"07f6759b39f2e597","reply_to_id":"2167d9860e710afe","content":"Fixed!!!! Didn't even require a PR!","create_time":"$D2022-05-09T16:36:23.156Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"53c5999a595018f3","reply_to_id":"95d4130a7b157577","content":"Would be good to have that feature (`beta({mean: a, stdev: b})`), I think it also serves a communication purpose in Squiggle. As for the stardard deviation, it was guessed/eyeballed by Nuño. I have not much else to say","create_time":"$D2022-05-09T16:50:07.040Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"47a730366e071219","reply_to_id":"d4f4e50521981199","content":"Lol, there is","create_time":"$D2022-05-09T16:53:16.909Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"9ffcdadcdeee63d3","reply_to_id":"d4f4e50521981199","content":"Appendix A","create_time":"$D2022-05-09T16:53:33.257Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"ce143f269feb3277","reply_to_id":"6eb6cbc8f575b5a6","content":"This is true, it is a bit overkill. You probably could do this with pointwise operations? I don't fully grok it yet to trust it though (especially thinking about how multiple observations are compounded to update the prior)","create_time":"$D2022-05-09T16:55:18.365Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"ae2c34be5f075664","reply_to_id":"6eb6cbc8f575b5a6","content":"Entirely my error though! I'll think about this one","create_time":"$D2022-05-09T16:56:14.976Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"5a6ed12cad1b1f2a","reply_to_id":"6acfb92ad92f658b","content":"Would love to have Joel's thoughts on this model. The considerations above are all just internal inconsistencies (As in, the model is saying two things that are inconsistent, which I love nitpicking), and I think Joel is more familiar with the external ones. Would be nice to put Joel's thoughts here.","create_time":"$D2022-05-09T16:58:34.390Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"f315d4058298ae74","reply_to_id":"b277d46ca627fe3c","content":"OK good point.\n\n> Donation size, which is arbitrary, and ends up just being multiplied by doublings of consumption per dollar to create, which is already at the top\n\nYeah this one would be nice to remove if it were feasible, as I've noted\n\n> Discount rate, which I think has been considered well enough (has low enough uncertainty) to not be considered a model input.\n\nI think it probably *should* be a model input ... but it's not a *moral* parameter here, is it, as this is not about 'pure discounting of future utility in a value function' \n\n> In this case, GiveDirectly is the baseline, this model is mostly economical and not really ethical\n\nI think one probably *could* rework this in a way that involves moral parameters, but it's probably not the best use of time? Would be better to do one more model ... bednets or some such, where its much easier to bring in the moral uncertainty parameters.","create_time":"$D2022-05-09T17:05:32.088Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"747edcf3a174d7df","reply_to_id":"b277d46ca627fe3c","content":"I was about to disagree with you about donation size, but now I've thought about it and now agree with you. I'll make that change.\n\nDiscount rates are not a moral parameter here at all, it's a calculation done to include the increase of consumption in other countries due to people getting richer. I might bring it up the document and turn it into paramater.\n\nAs for GiveDirectly having moral parameters, you absolutely *could*, but then it wouldn't really be a baseline. If I was to make another model (Which I would love to, mainly waiting on whether GiveWell would appreciate it) I would include moral weights and uncertainty.","create_time":"$D2022-05-09T17:14:33.487Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"139c5bf166a64ff4","reply_to_id":"b277d46ca627fe3c","content":"In fact, GiveDirectly use to have a moral weight of the value of an e increase in consumption. Although it was a dubious one, as it was actually just 1.44, as they defined value as a doubling of consumption, so was just changing the base of the log. I chose to just use the correct base for the log so that it didn't look like it was a considered moral weight when it wasn't (see note 4 from changes to GiveDirectly's model)","create_time":"$D2022-05-09T17:21:52.922Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"5e63f7c884d46ed6","reply_to_id":"b277d46ca627fe3c","content":"Also, there are other externalities that are put at the bottom of the model that could be uncertain and moral. Like, how much is the reduction of mortality from GiveDirectly worth vs the increase in consumption? I could extend in that direction.","create_time":"$D2022-05-09T17:23:25.701Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"adcd594b473f5731","reply_to_id":"0a8c314c845ceba3","content":"I agree! I just want to verify that GW agrees with us before going further xD","create_time":"$D2022-05-09T17:38:03.247Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"40a5ddd08de5972e","reply_to_id":"b277d46ca627fe3c","content":"that would be interesting","create_time":"$D2022-05-09T19:32:27.565Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"cfe9d9ebd1ab287a","reply_to_id":"b277d46ca627fe3c","content":"$32","create_time":"$D2022-05-09T19:40:13.351Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"809663325ec194be","reply_to_id":"e85f56775e9788a3","content":"Worth adding as a feature?","create_time":"$D2022-05-09T19:40:31.573Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"8e347050055134ed","reply_to_id":"ed19e4380ff51322","content":"That's fair. Main thing is 'seeing forest for the trees'. \nPerhaps there's a halfway version of this you could do, where you at least state the equation or key equations up top, but don't fully let them expand or link them?","create_time":"$D2022-05-09T19:41:38.849Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"2498eadaf455d40a","reply_to_id":"e85f56775e9788a3","content":"Oh, that's not something I control. I mean it's not something I can do with this platform (Observable) not Squiggle. To add this I would need to submit a major piece of work to this: https://github.com/observablehq","create_time":"$D2022-05-09T19:46:32.001Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"3b3a486b54c82c2f","reply_to_id":null,"content":"The graph box is much broader than the text box on my screen; it'd look neat if they had the same margins.","create_time":"$D2022-05-30T16:49:35.192Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"8b9908a25e18f4be","reply_to_id":null,"content":"Is it necessary to show this code? I would slightly prefer to just see the ouputs.","create_time":"$D2022-05-30T16:53:09.093Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"a0c254a85a610369","reply_to_id":"2167d9860e710afe","content":"The scientific notation for relatively small numbers seems unecessarily illegible?","create_time":"$D2022-05-30T16:54:57.340Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"2c3077c5f8059a3a","reply_to_id":null,"content":"Could you explain what you're doing here? It seems cool! What are you specifying about the prior? I can't tell at a glance.","create_time":"$D2022-05-30T16:57:11.363Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"e2f9f1d023673843","reply_to_id":null,"content":"So whenever you say distribution(x, y) that typically means 95% CI x - y? Sorry if you explained this earlier and I didn't see. If you didn't, I think it's worth trying to help out the lazy skimmers out there, because they make up most readers.","create_time":"$D2022-05-30T17:03:47.672Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"cd9989f01ee3b754","reply_to_id":null,"content":"Did you just choose symmetrical / normalish distributions for ease? I assume this would be more right skewed, with more people investing less than than the mean than above it.","create_time":"$D2022-05-30T17:05:27.544Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"f9285d8d662da173","reply_to_id":null,"content":"Why are you doing a relatively complicated thing if you don't think it'll change things? Why not just say \"lognormal(mean = observed_consumption, SD = observed_SD)\"?","create_time":"$D2022-05-30T17:09:21.957Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"73814fe78e0523d7","reply_to_id":null,"content":"I think positive spillovers seem pretty plausible, so this is one of the parameters where uncertainty specification is a real boon!","create_time":"$D2022-05-30T17:18:28.773Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"d0117e849e70b921","reply_to_id":null,"content":"Does the 1.03 mean you're adjusting upwards based on the possibility of leakage / cash not reaching? I'm not sure that makes sense if so...","create_time":"$D2022-05-30T17:21:12.940Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"3988286d67ae0227","reply_to_id":null,"content":"It'd be really good to know how how much of a difference these make!","create_time":"$D2022-05-30T17:25:59.471Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"1148bcc1477039c0","reply_to_id":null,"content":"Unfortunately, I learned quickly not to click on footnotes because it reloaded the entire page (which I expect will not be instant for most people). This was too much friction so I decided to ignore them.","create_time":"$D2022-05-30T17:27:05.885Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"f2e8d2eddf96aae8","reply_to_id":null,"content":"\"This is very convenient, as without this, as it is very performant and easy to calculate.\" I'm not sure this sentence makes sense.","create_time":"$D2022-05-30T17:28:38.573Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"99bf89bca49474b5","login":"jpmos","name":"Joel McGuire","avatar_url":"https://avatars-next.observableusercontent.com/c1d42967106d5d0a3378b0b18a43ffab038202d44f8b1911d9b5ee9d0eee28c7","tier":null}},{"id":"d06d1c894bfdddc2","reply_to_id":"e85f56775e9788a3","content":"got it. I guess I need to understand what Observable is/does","create_time":"$D2022-06-10T12:58:10.755Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"6aca3d77ed8e428c","reply_to_id":null,"content":"> GiveWell represents their GiveDirectly model in a Google Sheet and *therefore* does not represent uncertainty with their calculation.\n\nThe 'therefore' is not entirely correct. There are ways of embodying uncertainty in sheets, and there may even be plugins. But it doesn't come naturally","create_time":"$D2022-06-10T13:00:03.705Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"c947df120434db27","reply_to_id":"ed19e4380ff51322","content":"But I think it's something I could do with maybe 4-8 hours, if its easy to write on this platform.","create_time":"$D2022-06-10T13:01:28.227Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"23680f8cc5f75da0","reply_to_id":"2167d9860e710afe","content":"Yea; I think we clearly have improvement to do here, but that would happen more on the Squiggle level, as opposed for this notebook alone.","create_time":"$D2022-06-14T14:36:06.627Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"af52f2fc91624257","login":"oagr","name":"Ozzie Gooen","avatar_url":"https://avatars-next.observableusercontent.com/737b0788d4fee013e1f3a4a504d610359cbc19c380ca5d71a8ee543af7b1ff46","tier":null}},{"id":"b131d3fce44a34b6","reply_to_id":"ed19e4380ff51322","content":"Yeah, I was thinking of adding the math under some headers. I'll look into that","create_time":"$D2022-06-15T04:34:36.632Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"75fd45e8ffa8ea6c","reply_to_id":"e2f9f1d023673843","content":"In this case, it means a beta distribution with the alpha parameter being 32 and the beta parameter being 50. However, I'm about to fix this so it's easier to see where it comes from.","create_time":"$D2022-07-12T02:58:22.173Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"f152064b65cac84b","reply_to_id":"cd9989f01ee3b754","content":"I choose symmetric distributions because that's the shape of a beta distribution, and a beta distribution is usually a default choice for portions and probabilities. I'm not sure why you would expect this number to be skewed? Keep in mind that this isn't the portion invested by people, but the distribution of the expected amount invested per person.","create_time":"$D2022-07-12T03:01:57.631Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"7ce0eaffd1c2a80c","reply_to_id":"3b3a486b54c82c2f","content":"I think this is fixed, at least for me","create_time":"$D2022-07-20T02:18:56.514Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"f88c030d18cd2e6a","reply_to_id":null,"content":"The hyperlink https://www.squiggle-language.com/docs/Features/Functions#lognormal-distribution is a dead link","create_time":"$D2022-07-20T08:03:40.182Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"56bea789289193f1","reply_to_id":"f88c030d18cd2e6a","content":"Was looking it up because I was surprised the default was lognormal and not normal ... although I guess that makes sense for something bounded like the transfer rate","create_time":"$D2022-07-20T08:04:31.025Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"9a0642e2bd1ffc86","reply_to_id":"6912541f2b37d720","content":"I wouldn't say \"Google Sheet and therefore\" ... you could probably represent uncertainty with a google sheet, although it can be awkward, messy, and limited. Not ehte 'how to measure everything' book tries to do this (annoyingly) with standard spreadsheets.","create_time":"$D2022-07-20T08:07:10.456Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"3d3a03293ab9b205","reply_to_id":"6912541f2b37d720","content":"Some grammar fixes (not sure how I can edit without annoying forking) ... \"that change\" --> \"that changes\"","create_time":"$D2022-07-20T08:08:14.097Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"12413c731b04e5f1","reply_to_id":"f88c030d18cd2e6a","content":"I've just updated this notebook considerably. It has a lot of nicer features now! Like equations and functions! I'll talk to you about this in our meeting.","create_time":"$D2022-07-20T08:08:46.295Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"389c2eb905b6af79","reply_to_id":"6912541f2b37d720","content":"Did you actually use formal Bayesian procedures anywhere? (Note \"bayesian --> Bayesian\" as far as I know)","create_time":"$D2022-07-20T08:08:54.791Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"471d2e7b47c93376","reply_to_id":"13e76feadfe4e4cd","content":"Can you get the footnoes to hover?","create_time":"$D2022-07-20T08:11:08.779Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"41fdc39cd3410d8f","reply_to_id":"13e76feadfe4e4cd","content":"Even a tiny bit more reasoning transparency would be helpful here. What part of your analysis gave you the 90% CI of 0.75 to 0.9?","create_time":"$D2022-07-20T08:12:16.392Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"b975b2518c2348ff","reply_to_id":null,"content":"How would an average user know this Beta distribution is the equivalent of lognormal with 0.75 - 0.90 90% CI?","create_time":"$D2022-07-20T08:13:13.714Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"793dc2d0bffb7fea","reply_to_id":null,"content":"--> \"goes to each household\"\n? \"this is important because of assumed diminishing returns\" (and what else?) \n\nImportant to understanding 'why this matters' in setting these parameters. That is another reason to try to put the 'expandable equation' (there is another fancy word for this I am forgetting, a math person told me) at the top.","create_time":"$D2022-07-20T08:15:28.353Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"2ef0dccccac495f4","reply_to_id":"b975b2518c2348ff","content":"This is halfway through a large edit. I'll need to update the text","create_time":"$D2022-07-20T08:16:04.191Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"80c3e93bbe5d5778","reply_to_id":"793dc2d0bffb7fea","content":"^ It was Quinn Dougherty I spoke to about that","create_time":"$D2022-07-20T08:17:18.270Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"79600481d4d5bc6d","login":"79600481d4d5bc6d","name":"david reinstein","avatar_url":"https://avatars-next.observableusercontent.com/13628adbf8a1e8255d053c5fb8ccaf2404efa8643ff0f06d3023cec62c646af9","tier":null}},{"id":"4560d6175006195f","reply_to_id":"13e76feadfe4e4cd","content":"I certainly could! There's many many different ways I could refine this","create_time":"$D2022-07-20T08:18:43.771Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}},{"id":"9b57d53671bb3a21","reply_to_id":"793dc2d0bffb7fea","content":"If you refresh, you'll get the expandable equations or at least a version of them","create_time":"$D2022-07-20T08:22:03.567Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"36700787ef9d8445","login":"hazelfire","name":"Sam Nolan","avatar_url":"https://avatars-next.observableusercontent.com/f13fffd8528d3edcbc144dc7134ab9d3b207a9e34df9066de75ab4827ed14c4d","tier":null}}],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":11,"collection":null,"type":"notebook","body":{"title":"Quantifying Uncertainty in GiveWell's GiveDirectly Cost Effectiveness Analysis","cells":[{"id":124,"value":"# Quantifying Uncertainty in GiveWell's GiveDirectly Cost Effectiveness Analysis","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1557,"value":"**Disclaimer**: I (Sam Nolan) do not represent GiveWell or the Quantfied Uncertainty Research Institute. All opinions and mistakes are my own.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1518,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":407,"value":"## Introduction","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":58,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":641,"value":"## What is Squiggle?\nThis notebook uses Squiggle to do calculations using uncertainty. \n\n[Squiggle](https://squiggle-language.com/) is a simple estimation programming language for forecasters and evaluators. It's in a pre-alpha stage, and comes with a healthy [collection of bugs](https://github.com/quantified-uncertainty/squiggle/issues?q=is%3Aissue+is%3Aopen+label%3Abug+) and [missing features](https://github.com/quantified-uncertainty/squiggle/discussions). Squiggle is made by the [Quantified Uncertainty Research Institute](https://quantifieduncertainty.org/).\n\nIf you want to use Squiggle, I recommend checking out the [playground](https://www.squiggle-language.com/playground/). For Observable support, check out the [demo Squiggle notebook](https://observablehq.com/@hazelfire/squiggle). Also feel free to directly get in contact with me via my [calendly](https://calendly.com/sam_nolan/30min) or the [EA Forum](https://forum.effectivealtruism.org/users/hazelfire).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":646,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":674,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":324,"value":"import {cellGen, newProject, stylesheet, saveProject, runGen, exportSampleSet} from '@hazelfire/squiggle'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2923,"value":"saveProject(project)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1898,"value":"stylesheet","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2831,"value":"project = newProject()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2835,"value":"cell = cellGen(project)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3016,"value":"run = runGen(project)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":651,"value":"# GiveDirectly model","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1567,"value":"GiveDirectly creates wellbeing through direct cash transfers. In [GiveWell's model](https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models), wellbeing created is measured by doublings of consumption. According to the model, if you double the consumption of an individual for a year, you get 1 unit of value.\n\nThe model starts by working out how much of your donation goes to each individual. First by subtracting charity costs, then dividing the rest among households, then dividing that among every member of the household.\n\nFrom there, the recipient is modelled to invest a portion of their money and use a portion of their money. These effects are then added together. This creates the amount of value before adjustments.\n\nFinally, the value is adjusted for other adjustments that are not included within the model. For instance, the likelihood on whether this could decrease child mortality.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2650,"value":"For the brave hearted, the full formula of the model is below. Italics represent parameters, and each term will link to where it is defined in Squiggle.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2591,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":413,"value":"## Transfer size","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1964,"value":"$$`\\text{grant size} = \\frac{\\param{household grant size}}{\\param{household size}}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1969,"value":"viewof grant_size = cell(`grant_size(x) = x.household_grant_size / x.household_size\ngrant_size(givedirectly_params)`, [givedirectly_params], {tickFormat: \"$,f\", title: \"Size of transfer per person (USD)\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3144,"value":"compare_givewell(\"Size of transfer per person (nominal USD)\", grant_size)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":188,"value":"The first important measure is how much money goes to a household. A GiveDirectly grant is usually [given to a household](https://www.givewell.org/charities/give-directly#Overview). GiveDirectly aims to give different amounts to different demographics based on inflation and other considerations. GiveWell gives a point estimate of $1000 based on talking with GiveDirectly:\n\n> [Transfer sizes for the standard lump sum projects: Kenya: $1,085, Rwanda: $970, Uganda: $963.](https://www.givewell.org/charities/give-directly/supplementary-information#footnote2_0h0watm)\n>\n> February 20, 2018, [Joe Huston](https://www.givedirectly.org/author/joe-huston/) GiveDirectly CFO","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":194,"value":"viewof household_grant_size = cell(`givedirectly_params.household_grant_size`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":349,"value":"The money given to a participant is shared among a household. We measure how much good we do for each person, so we'll need the average household size. A point estimate of 4.7 is provided by GiveWell from their [analysis on household size](https://files.givewell.org/files/DWDA%202009/GiveDirectly/household%20size%20analysis.xls) which uses data from the data from the Siaya 2012 census. I made a copy of this spreadsheet (You can see and download the copy by clicking on the paperclip on the right) and used it to do infer the mean household size. More details about this can be seen in [Appendix C](#appendixc).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":420,"value":"## Recipient usage of funds","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":198,"value":"Recipients can choose to spend the money now or invest it in something that gives them returns in the future. The point estimate for GiveWell's estimate is that participants invest 39% of the grant. An investment in this case is simply a purchase that has a payoff over a period of time, with a common example being tin roofs or livestock. This figure is retrieved from [Haushofer & Shapiro 2013](https://files.givewell.org/files/DWDA%202009/Interventions/Cash%20Transfers/haushofer_shapiro_uct_2013.11.16.pdf). Pg 20.\n> \"Households invest part of the transfers: the value of non-land assets increased by USD 279 on average; this represents 58 percent of the control group mean USD 478, and 39 percent of the average transfer.\"","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":423,"value":"### Amount invested / consumed\n\n${$`\\begin{aligned}\n\\text{amount invested} &= \\param{portion invested} \\cdot \\ref{grant size} \\\\\n\\text{amount consumed} &= (1 - \\param{portion invested}) \\cdot \\ref{grant size}\n\\end{aligned}\n`}\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1989,"value":"viewof amount_invested = cell(`amount_invested(x) = x.portion_invested * grant_size(x)\namount_consumed(x) = (1 - x.portion_invested) * grant_size(x)\n{invested: amount_invested(givedirectly_params), consumed: amount_consumed(givedirectly_params)}`, [grant_size, givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":206,"value":"viewof portion_invested = cell(`givedirectly_params.portion_invested`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2807,"value":"### Utility\nThe default utility function is logarithmic, but can be changed to isoelastic below. This means that people who are richer value marginal increases in money less than people that are poorer, a core assumption when giving to the poor.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2000,"value":"$${tex`u(x) = ${isoelastic_utility ? `\\\\frac{x^{1 - \\\\mu} - 1}{1 - \\\\mu} \\\\\\\\ \\\\mu = 1.59` : `\\\\log_e x`}`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2002,"value":"viewof utility = cell(`utility(x) = ${isoelastic_utility ? `(x ^ (1 - 1.59) - 1) / (1 - 1.59)` : `log(x)`}`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1376,"value":"### Log increase in initial consumption","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1952,"value":"$${$`\\begin{aligned}\n\\text{initial consumption} & = (\\href{#cell-2000}{u}(\\frac{\\text{\\href{#amount_invested}{amount consumed}}}{\\param{duration initial consumption}} + \\param{baseline consumption}) \\\\ \n&- u(\\param{baseline consumption})) \\cdot \\param{duration initial consumption}\n\\end{aligned}`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2231,"value":"viewof value_initial_consumption = cell(`value_initial_consumption(x) = (utility(amount_consumed(x)/x.duration_initial_consumption + x.baseline_consumption) - utility(x.baseline_consumption)) * x.duration_initial_consumption\nvalue_initial_consumption(givedirectly_params)`, [utility, amount_invested, givedirectly_params]) ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":5302,"value":"compare_givewell(\"Total immediate increase in ln(consumption) per person made possible by funds transferred\", value_initial_consumption)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":230,"value":"One way that the recipient gets value is by spending the money that they receive immediately. The initial consumption period is implicitly one year in GiveWell's analysis. This parameter is implicit within GiveWell's analysis. Here it's made explicit. From this, we can calculate how much utility we create by subtracting the utility gained from the grant from the utility of the baseline consumption.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":239,"value":"viewof duration_initial_consumption = cell(`givedirectly_params.duration_initial_consumption`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":430,"value":"### Recipient Investments\n${$`\\begin{aligned}\npv(p, d, t) &= p \\int_0^t (1 + d)^{-x} dx = p\\frac{1 - (1 + d)^{-t}}{\\log_e(1 + d)}\\\\\n\\text{value from roi} &= \\href{#utility}{u}(\\param{return on investment} \\cdot \\ref{amount invested} + \\param{baseline consumption}) \\\\ &- \\href{#utility}{u}(\\param{baseline consumption}) \\\\\n\\text{value from investment} &= pv(\\text{value from roi}, \\param{discount rate}, \\param{investment duration} - 1)\n\\end{aligned}`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2252,"value":"viewof value_investment = cell(`pv(payment, discount, duration) = payment * (1 - (1 + discount) ^ (-duration)) / ${continuous_present_value ? `log(1 + discount)` : `discount`}\nvalue_from_roi(x) = utility(x.return_on_investment * amount_invested(x) + x.baseline_consumption) - utility(x.baseline_consumption)\nvalue_investment(x) = pv(value_from_roi(x), x.discount_rate, x.investment_duration - 1)\nvalue_investment(givedirectly_params)`, [utility, amount_invested, givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3100,"value":"compare_givewell(\"Present value of future increases in ln(consumption) per person (excluding the last year of expected benefits)\", value_investment)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":438,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":258,"value":"viewof return_on_investment = cell(`givedirectly_params.return_on_investment`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":454,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":281,"value":"viewof investment_duration = cell(`givedirectly_params.investment_duration`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":479,"value":"GiveWell has a [full discussion of their discount rate elsewhere](https://docs.google.com/document/d/1g2tf7Fu8-iG_8TgnJkEx9G5TlEyrcv7PXPokmPZaUMc/edit?usp=sharing). Note that the discount rate is not because there is an inherent value on getting utility now, but is made from three components:\n- Uncertainty in the future\n- Increase in monetary consumption due to other effects, making marginal log increases in consumption later more difficult.\n- Increase in non-monetary consumption, again making marginal log increases in consumption more difficult.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":467,"value":"viewof discount_rate = cell(`givedirectly_params.discount_rate`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":485,"value":"This is a continuous present value formula, it is different from the one used by GiveWell. I prefer it because I think GiveDirectly investments are probably not well modelled by discrete payouts^{[6](#footnotecontinuouspv)}. It comes from the following formula for present value:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2407,"value":"cell(`value_investment(givedirectly_params)`, [givedirectly_params, value_investment])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":561,"value":"Where is ${tex`p`} the payment, ${tex`t`} is the duration, ${tex`d`} is the discount_rate ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":503,"value":"### Benefits from reselling investments\n${$`\n\\begin{aligned}\n\\text{consumption last year} &= \\textit{\\param{baseline consumption}} + \\ref{amount invested} \\\\\n & \\cdot (\\textit{\\param{return on investment}} + \\param{portion investment returned})\\\\\n\\text{value resell} &= (\\href{#utility}{u}(\\text{consumption last year})\n - u(\\param{baseline consumption})) \\\\\n&/ (1 + \\param{discount rate})^{\\param{investment duration}}\n\\end{aligned}\n`}\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2277,"value":"viewof value_resell = cell(`value_resell(x) = (utility(x.baseline_consumption + amount_invested(x) * (x.return_on_investment + x.portion_investment_returned)) - utility(x.baseline_consumption)) / ((1 + x.discount_rate) ^ x.investment_duration)\nvalue_resell(givedirectly_params)`,[utility, amount_invested, givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":5308,"value":"compare_givewell(\"Present value of ln(consumption) increase in the last year of expected benefits\", value_resell)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2275,"value":"GiveWell then considers the benefits from reselling investments. Comments from GiveWell:\n> In some cases, items purchased with cash transfers might be resold when they cease providing returns. We are extremely uncertain about the appropriate value for this parameter. It's our best guess that most investment items will not retain much of their original value. We have not prioritized a thorough investigation into possible values for this parameter.^{[7](#footnotesellinginvestments)}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":290,"value":"viewof percent_investment_returned = cell(`givedirectly_params.portion_investment_returned`, givedirectly_params);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":508,"value":"We can then add the three different sources of value together to create the total increase in pv from cash transfers.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":298,"value":"viewof total_value = cell(`total_value(x) = value_initial_consumption(x) + value_investment(x) + value_resell(x)\ntotal_value(givedirectly_params)`, [value_initial_consumption, value_investment, value_resell, givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3058,"value":"compare_givewell(\"Total present value of cash transfer in terms of increases in ln(consumption)\", total_value)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2637,"value":"$$`\\text{total value} = \\ref{initial consumption} + \\ref{returns from investment} + \\ref{value resell}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":545,"value":"## Negative Spillover and Total Value before adjustments\n${$`\\begin{aligned}\n\\text{total value including spillover} &= \\ref{total value} \\cdot \\param{negative spillover} \\\\\n\\text{value per household} &= \\text{total value including spillover} \\cdot \\param{household size} \\\\\n\\text{value per dollar} &= \\param{transfer efficiency} \\cdot \\frac{\\text{value per household}}{\\param{household grant size}} \\\\\n\\text{value doubling consumption} &= u(\\param{baseline consumption} \\cdot 2) - u(\\param{baseline consumption}) \\\\\n\\text{doubling consumption per dollar} &= \\frac{\\text{value per dollar}}{\\text{value doubling consumption}}\n\\end{aligned}\n`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":5319,"value":"viewof donation_size = cell(`donation_size = 100000`)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2298,"value":"viewof doubling_consumptions = cell(`total_value_including_spillover(x) = total_value(x) * (1 - x.negative_spillover)\nvalue_per_household(x) = total_value_including_spillover(x) * x.household_size\ndonation_size = 100000\ntransfers_made(x) = donation_size / x.household_grant_size\nln_consumptions_per_dollar(x) = x.transfer_efficiency * value_per_household(x) * transfers_made(x)\nvalue_doubling_consumption(x) = utility(x.baseline_consumption * 2) - utility(x.baseline_consumption)\ndoubling_consumptions(x) = ln_consumptions_per_dollar(x) / value_doubling_consumption(x)\ndoubling_consumptions(givedirectly_params)`, [total_value, utility, givedirectly_params, donation_size])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3053,"value":"compare_givewell(\"Total units of value generated\", doubling_consumptions)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":529,"value":"The negative spillover represents possible negative externalities that come from direct cash transfers. GiveWell has an [analysis of negative spillover from cash transfers](https://docs.google.com/document/d/1C4nX3LWM-TeNMFxmNAKRbi4vtt_4ZaLoIGZzlaHmvuI/edit#).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":512,"value":"viewof negative_spillover = cell(`givedirectly_params.negative_spillover`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":177,"value":"The proportion of funding available is the funding that ends up in the hands of grantees, after any required administration costs. This value is given a point value of 83% by GiveWell, and is calculated from [an analysis of GiveDirectly spending in February 2018](https://docs.google.com/spreadsheets/d/1L03SQuAeRRfjyuxy20QIJByOx6PEzyJ-x4edz2fSiQ4/edit#gid=537899494). Based on this analysis, with the help of Nuño Sempere, I constructed an estimate of a beta distribution wih ${tex`\\mu=0.83`} and ${tex`\\sigma = 0.05`} ^{[1](#footnoteto)} for GiveDirectly's efficiency.^{[2](#footnotetransfer)}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":179,"value":"viewof transfer_efficiency = cell(`givedirectly_params.transfer_efficiency`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":539,"value":"Finally, we total up the amount of value created for each household/transfer","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":849,"value":"## Charity level adjustments\n${$`\\text{value after adjustments} = \\prod^{c \\in \\param{adjustments}} c \\cdot \\text{doubling consumption per dollar}`}\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2323,"value":"viewof value_after_adjustments = cell(`charity_adjustments(x) = 1 - (x.double_treatment + x.ineffective_goods + x.cash_doesnt_reach_participants + x.misappropriation + x.bad_monitoring + x.change_of_priorities + x.non_funding_bottlenecks + x.within_org_fungibility)\nintervention_adjustments(x) = 1 + x.changes_in_ppp + x.developmental_effects + x.reduced_morbidity + x.child_mortality_effects\nvalue_after_adjustments(x) = doubling_consumptions(x) * intervention_adjustments(x) * charity_adjustments(x)\nvalue_after_adjustments(givedirectly_params)`, [doubling_consumptions, givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3173,"value":"compare_givewell(\"Total units of value generated, after accounting for all adjustments\", value_after_adjustments)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2321,"value":"$39","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":852,"value":"viewof risk_of_wastage_adjustments = cell(`{ double_treatment: givedirectly_params.double_treatment, ineffective_goods: givedirectly_params.ineffective_goods, cash_doesnt_reach_participants: givedirectly_params.cash_doesnt_reach_participants}`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":859,"value":"### Quality of monitoring and evaluation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1062,"value":"md`These adjustments are described by GiveWell as follows:\n> **misappropriation** Charity sends money/goods to a country/partner and the country/partner reports back that they carried out the program and reached a certain number of people, but these results are partially or fully fabricated and we do not have monitoring results from those locations.\n> \n> We've consistently received comprehensive monitoring results from GiveDirectly. (We chose not to revisit the GiveDirectly's monitoring in 2019 and therefore have not updated this.)\n\n> **false_monitoring** The monitoring results that we've received are partially or fully fabricated or significantly biased in some way we are unaware of, leading us to believe that results are better than they actually were.\n> \n> We have relatively high confidence in GiveDirectly's monitoring, which involves multiple rounds of checks and audits by different teams. (We chose not to revisit the GiveDirectly's monitoring in 2019 and therefore have not updated this.)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":862,"value":"viewof monitoring_and_evaluation_adjustments = cell(`{ misappropriation: givedirectly_params.misappropriation, bad_monitoring: givedirectly_params.bad_monitoring}`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":868,"value":"### Confidence of funds being used for intended purpose","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1072,"value":"$3a","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":870,"value":"viewof confidence_of_funds_adjustments = cell(`arbitrary(x) = (0.9 to 1.1) * x\n{ change_of_priorities: givedirectly_params.change_of_priorities, non_funding_bottlenecks: givedirectly_params.non_funding_bottlenecks, within_org_fungibility: givedirectly_params.within_org_fungibility}`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":909,"value":"## Intervention Level Adjustments","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1076,"value":"GiveWell then specifies [possible effects that are not included in the model](https://docs.google.com/spreadsheets/d/1Kq6iHSQFr3eRz1p9KclHuJTQiaJYkpViOyneSd9KCJc/edit#gid=819415156). Due to lack of strong evidence, none of these effects are considered likely to be large.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":913,"value":"viewof adjustments = cell(`{ changes_in_ppp: givedirectly_params.changes_in_ppp, developmental_effects: givedirectly_params.developmental_effects, reduced_morbidity: givedirectly_params.reduced_morbidity, child_mortality_effects: givedirectly_params.child_mortality_effects}`, givedirectly_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3104,"value":"$3b","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":5326,"value":"viewof givewell_params = cell(`givedirectly_params = givewell_params\n0`, [read_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":5334,"value":"viewof arbitrary_params = cell(`arbitrary(x) = x == 0 ? 0 : (x > 0 ? lognormal({mean: x, stdev: x / 3}): -1 * lognormal({mean: -1 * x, stdev: -1 * x / 3}))\ndictMap(x, f) = Dict.fromList(map(Dict.keys(x), {|key| [key, f(x[key])]}))\ngivedirectly_params = dictMap(givewell_params, arbitrary)\n0`, read_params)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2336,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2365,"value":"viewof final_value_per_dollar = cell(`final_value_per_dollar(x) = value_after_adjustments(x) / donation_size\nfinal_value_per_dollar(givedirectly_params)`, [givedirectly_params, value_after_adjustments, donation_size])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3043,"value":"compare_givewell(\"Units of value generated per philanthropic dollar spent, after accounting for all adjustments\", final_value_per_dollar)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3027,"value":"exportSampleSet(project, `final_value_per_dollar(givedirectly_params)`, [final_value_per_dollar,givedirectly_params])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2973,"value":"sensitivity(`value_after_adjustments`, `givedirectly_params`, [givedirectly_params, value_after_adjustments])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1746,"value":"Equivalently, the mean cost per doubling of consumption is as below:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2392,"value":"viewof dollars_per_double = cell(`1 / value_after_adjustments(givedirectly_params)`, [givedirectly_params, value_after_adjustments], {showSummary: true, tickFormat: \"$,f\", title: \"Cost of doubling someone's consumption\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":777,"value":"$3d","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1370,"value":"### Inference of baseline consumption","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":220,"value":"We measure the increase in welfare by doubling consumption. The poorer a recipient the more effective cash transfers become. The baseline consumption for recipients is given as 285.92 USD per year by GiveWell. To quote GiveWell:\n\n> Taken from [Haushofer and Shapiro 2013](https://files.givewell.org/files/DWDA%202009/Interventions/Cash%20Transfers/haushofer_shapiro_uct_2013.11.16.pdf), Pg 49. Non-durable expenditure (USD) in the control group is measured as 157.40 a month. This is multiplied by 12 to get an annual estimated, adjusted for purchasing power parity, and divided by average household size. See [calculations](https://files.givewell.org/files/DWDA%202009/Interventions/GiveWell_cost-effectiveness_analysis_2016.xlsx) [Parameters > Cash] for more details.\n\nI calculate the expected value of baseline consumption through Bayesian Inference, see [Appendix C](#appendixc) for more details.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1282,"value":"First, we get the mean consumption from the [Haushofer and Shapiro 2013](https://files.givewell.org/files/DWDA%202009/Interventions/Cash%20Transfers/haushofer_shapiro_uct_2013.11.16.pdf), Pg 49","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1141,"value":"monthly_consumption_per_household_ppp = 157.40","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1290,"value":"To convert it out of [PPP](https://en.wikipedia.org/wiki/Purchasing_power_parity), we use a multiplier. This value was taken from [GiveWell's parameter calculations](https://files.givewell.org/files/DWDA%202009/Interventions/GiveWell_cost-effectiveness_analysis_2016.xlsx). To quote:\n> Derived using the fact that the authors equate 1085 USD with 1,525 PPP","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1143,"value":"ppp_multiplier = 1.405529954","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1297,"value":"We then get the mean household size.^{[3](#footnotemean)}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2856,"value":"viewof mean_household_size = cell(`mean(household_size)`, household_size)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1148,"value":"household_size_mean = 4.7","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1304,"value":"We can then calculate the consumption per person per year that GiveWell arrived at.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1154,"value":"consumption_per_person_per_year_usd = monthly_consumption_per_household_ppp / household_size_mean / ppp_multiplier * 12","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1312,"value":"To do bayesian inference, we'll need the number of samples that was done in the study. Retrieved from [Haushofer and Shapiro 2013](https://files.givewell.org/files/DWDA%202009/Interventions/Cash%20Transfers/haushofer_shapiro_uct_2013.11.16.pdf), Pg 49.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1140,"value":"consumption_sample_count = 1372","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1318,"value":"We also need the standard deviation of the sample, also based on [Haushofer and Shapiro 2013](https://files.givewell.org/files/DWDA%202009/Interventions/Cash%20Transfers/haushofer_shapiro_uct_2013.11.16.pdf), Pg 49.^{[4](#footnoteknownvariance)}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1160,"value":"consumption_samples_stdev = 82.18","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1324,"value":"We can then generate a collection of observations (These observations are fictitious, all the mean, but it doesn't matter because bayesian inference only uses the sum and length of this array, see [Appendix C](#appendixc)).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1254,"value":"consumption_observations = Array(consumption_sample_count).fill(Math.log(consumption_per_person_per_year_usd))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1330,"value":"I then choose an appropriate prior. This is somewhat arbitrary and should not impact the result, as we have enough samples for the prior to be not considered very strongly. It comes from guessing the mean consumption to be somewhere from `100 to 500` on squiggle:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1167,"value":"consumption_prior = ({mean: 5.409889142205142, stdev: 0.48923438720106327 })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1366,"value":"cell(`lognormal($consumption_prior.mean, $consumption_prior.stdev)`, [], {jsImports: {consumption_prior: consumption_prior}, tickFormat: \"$,f\", title: \"Baseline consumption prior (USD)\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1334,"value":"We then calculate the posterior based on the prior and observations^{[5](#footnotetosample)}.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1170,"value":"consumption_posterior = normal_normal_bayes_update(consumption_prior, consumption_observations, Math.log(consumption_samples_stdev))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":228,"value":"viewof baseline_consumption = cell(`baseline_consumption = SampleSet.fromDist(lognormal($consumption_posterior.mean, $consumption_posterior.stdev))`, [], {jsImports: {consumption_posterior: consumption_posterior}})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":574,"value":"household_size_data = FileAttachment(\"household size analysis - Family size by household.csv\").csv()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":582,"value":"household_sizes = household_size_data.map(x => parseFloat(x[\"Total household size\"])).filter(x => x !== 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1686,"value":"FrequencyChart(household_sizes)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1114,"value":"household_sizes_prior = ({shape: 3, scale: 1.5})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2823,"value":"cell(`gamma($prior.shape, $prior.scale)`, undefined, {jsImports: {prior: household_sizes_prior}})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1125,"value":"household_sizes_posterior = poisson_gamma_bayes_update(household_sizes_prior, household_sizes)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":353,"value":"viewof household_size = cell(`household_size = gamma($household_sizes_posterior.shape, $household_sizes_posterior.scale)`, [], {jsImports: {household_sizes_posterior: household_sizes_posterior}})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1851,"value":"## Problems and room for improvement with this model\nHere I go through some larger changes to the model. Most of these models are here to ensure that the model remains internally consistent.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":3140,"value":"givedirectly_params = my_params","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":5373,"value":"### Continuous Present Value\nWe use the continuous version of present value, with the formula detailed above. This is because I think it models the situation better.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":5378,"value":"viewof continuous_present_value = Inputs.toggle({label: \"Use Continuous Present Value\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1830,"value":"$3e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1810,"value":"I'm only about 75% sure that this change is appropriate. If you would like to use an isoelastic utility function with ${tex`\\eta = 1.59`}, please tick the following:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1814,"value":"viewof isoelastic_utility = Inputs.toggle({label: \"Use isoelastic utility function\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1837,"value":"### Other changes not added but desired","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1771,"value":"$3f","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":934,"value":"## Minor changes made to GiveWell's model\nI made a number of changes in this model to correct what I see as a collection of errors made in the model design. None of these errors change the results by any large degree. They are mostly stylistic and I doubt are contraversial. With these changes, the model should still remain mostly comparable to the original.\n\n${list(minor_changes)}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2961,"value":"minor_changes = [`We use the continuous version of present value, with the formula detailed above. This is because I think it models the situation better.`, `We multiply, rather than add the adjustment percentages. This is because [you can't add percentages](https://towardsdatascience.com/most-people-screw-up-multiple-percent-changes-heres-how-to-do-get-them-right-b86bd6ef4b72)`, `We the [implicit parameter for duration of initial consumption](https://forum.effectivealtruism.org/posts/WmQwtYEajNDuPdyZx/type-checking-givewell-s-givedirectly-cost-effective#Results) explicit. *This makes no changes to the result*.`, `We remove the donation size parameter. The choice of this parameter is entirely arbitrary and does not impact the result at all. It is simply multiplied then divided out to calculate the doublings of consumption per dollar. On discussion with David Reinstein, I thought it better to remove it.`]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":822,"value":"## Acknowledgements\nBoth Squiggle and this work was done with help and funding from the [Quantified Uncertainty Research Institute](https://quantifieduncertainty.org/). I would like to thank Ozzie Gooen for his feedback, and Nuño Sempere for contributing their feedback and estimation expertise to this project. I'd like to thank the rest of the Squiggle development team: Ozzie Gooen, Nuño Sempere and Quinn Dougerty. I'd further like to thank David Reinstein for his feedback and interest on this project.\n\n![Quantified Uncertainty Research Institute](https://super-static-assets.s3.amazonaws.com/09bb1362-5e3f-4724-8ffd-f3235f67356f/images/6151ac3e-aed7-44c7-9827-399fe6e9222b.png)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":688,"value":"## Footnotes","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":709,"value":"footnote = (referencingCellName, number, text) => {\n return md`

^ ${text}

`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1038,"value":"footnoteto = footnote(\"cell-177\", 1, `The \"to\" syntax here is [syntax sugar for a lognormal distribution with a 90% confidence interval between the two numbers](https://www.squiggle-language.com/docs/Features/Functions#lognormal-distribution). This is borrowed from guesstimate.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1088,"value":"footnotetransfer = footnote(\"cell-177\", 2, `I feel like there's almost certainly a better way to do this than this estimate. We have all the spending data that we could possibly use, and we could probably do some form of bayesian inference to get the actual value of this parameter. I'm just not that sure on how to do that.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1695,"value":"footnotemean = footnote(\"cell-1297\", 3, `This is mildly problematic, because it would be better if we were to use the entire distribution rather than the point estimate of the mean. However Squiggle does not [yet support distributions as parameters to other distributions](https://github.com/quantified-uncertainty/squiggle/discussions/482). However, due to the household size having a small variance, I doubt this would have a very large impact on the final result.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1406,"value":"footnoteknownvariance = footnote(\"cell-1318\", 4, `\nIn this bayesian inference, I assume we know the variance. In truth, I don't think we actually know the variance, because I'm getting my variance value from the observations. I'm just not currently good enough at bayesian inference to do this properly. However, I doubt it would make much of a chance on the final result.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":722,"value":"footnotetosample = footnote(\"cell-220\", 5, `In Squiggle, under allmost all situations, distributions are assumed to be indipendent. This becomes an issue, for instance when you are measing log increase in consumption: \\`log(baseline_consumption + amount_consumed) - log(baseline_consumption) \\`.\n\nHere, the distributions that are being subtracted from each other are not indipendent, because they both rely on the same variable \\`baseline_consumption\\`. If they were, you would get nonsense results such as having probability mass below 0.\n\nHowever, sample sets do not have this property, and allow you to do operations on distributions that are not indipendent. This is why I convert \\`baseline_consumption\\` to a sample set\n\nTo take a look at the different types of distributions available in Squiggle, [check out our documentation](https://www.squiggle-language.com/docs/Discussions/Three-Formats-Of-Distributions)`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":840,"value":"footnotecontinuouspv = footnote(\"cell-485\", 6, `Another reason that I prefer the continuous format is because I love Dimensional Analysis (checking units of calcualtions), and the continuous version is [much easier to solve units for](https://forum.effectivealtruism.org/posts/WmQwtYEajNDuPdyZx/type-checking-givewell-s-givedirectly-cost-effective#Problems_and_caveats).`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":753,"value":"footnotesellinginvestments = footnote(\"cell-503\",7,`I'm not sold that this is an important consideration and could complicate the model. Removal of this consideration would be a change I would make.`)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":551,"value":"## Appendix A, Squiggle features that could improve this model.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":138,"value":"Current rooms for improvement, particularly within Squiggle development:\n - The performance is pretty slow at this point. It would be valuable to improve it.\n - Allowing parameters of distribution to be themselves distributions. Would be useful for bayesian inference of household sizes above","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1187,"value":"$40","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2941,"value":"import {normal_normal_bayes_update, poisson_gamma_bayes_update} from \"@hazelfire/bayesian-updating-with-conjugate-priors\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1462,"value":"### Utility Functions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1119,"value":"sum = x => x.reduce((a, b) => a + b)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1456,"value":"formatFloat = x => new Intl.NumberFormat('en-US', {maximumSignificantDigits: 3}).format(x)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1458,"value":"formatCurrency = x => new Intl.NumberFormat('en-US', { style: 'currency', currency: 'USD' }).format(x)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1620,"value":"import { BarChart } from \"@d3/bar-chart\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1514,"value":"import {toc} from \"@harrystevens/toc\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1673,"value":"freq = data => data.reduce((acc, d) => {\n if (!acc[d]) {\n acc[d] = 0;\n }\n acc[d]++;\n return acc\n }, {});","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1675,"value":"FrequencyChart = data => BarChart(Object.entries(freq(data)), {\n x: d => d[0],\n y: d => d[1],\n xDomain: Object.keys(freq(data)), // sort by descending frequency\n yFormat: \"g\",\n yLabel: \"↑ Count\",\n width,\n height: 300,\n color: \"steelblue\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2964,"value":"list = (items) => items.map((x, i) => `${i + 1}. ${x}`).join(\"\\n\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1996,"value":"$$ = tex.options({trust: true, displayMode: true, macros: {\"\\\\param\": context => { \n const args = context.consumeArgs(1);\n const name = args[0].reverse().map(x => x.text).join(\"\")\n const url = name.replaceAll(\" \", \"_\")\n return `\\\\textit{\\\\href{\\\\#${url}}{${name}}}`}, \"\\\\ref\": context => { \n const args = context.consumeArgs(1);\n const name = args[0].reverse().map(x => x.text).join(\"\")\n const url = name.replaceAll(\" \", \"_\")\n return `\\\\text{\\\\href{\\\\#${url}}{${name}}}` }\n}})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3036,"value":"gw_gd_cea = FileAttachment(\"gw_gd_cea.csv\").csv()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3111,"value":"param_imports = {\n let x = {}\n gw_gd_cea.forEach(entry => {\n const parameter_name = entry[\"Cash Transfers - GiveDirectly\"].replace(\"\\\"\", \"\")\n let value = entry[\"Overall\"].replace(\",\",\"\")\n if(value === \"Not specified\"){\n value = 0\n }\n if(!Number.isNaN(parseFloat(value))){\n x[parameter_name] = parseFloat(value)\n }\n\n \n })\n return x;\n}\n ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3033,"value":"$41","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":3063,"value":"sensitivity = (func, params, b) => {\n let result = run(`cov(x,y) = mean(x * y) - mean(x) * mean(y)\ncorr(x,y) = cov(x, y) / stdev(x) / stdev(y)\nr_squared(x, y) = corr(x, y) ^ 2\nall_to_sample_set(x) = Dict.fromList(map(Dict.keys(x), {|param| [param, SampleSet.fromDist(x[param])]}))\nsensitivity(f, params) = {\n ss_params = all_to_sample_set(params)\n result = f(ss_params)\n Dict.fromList(map(Dict.keys(ss_params), {|param| [param, r_squared(ss_params[param], result)]}))\n}\nsensitivity(${func}, ${params})`, b).value.value.entries()\n return Plot.plot({\n marks: [\n Plot.barX(result, {x: d => d[1].value, y: d => d[0], sort: {y: \"x\", reverse: true}}),\n Plot.ruleX([0])\n ],\n marginLeft: 200\n })\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W42","stdlib":"1","titling":"auto"}} 31:"$30:body"