2d:null 2e:["$","title",null,{"children":"Foretold Scoring Tutorial | Jacob Lagerros | Observable"}] 33:T694,md`### 1. Measuring information If you don't already know how to do it, "measuring information" can sound both daunting and amazing. However, the tools of the mathematical field of [information theory](https://en.wikipedia.org/wiki/Information_theory) give us a susprisingly elegant and easy way to manage this. To start intuitively, it seems that _any_ quantitative notion of "information" should satisfy the following criteria... 1. If you're _absolutely certain_, 100%, that something will happen -- and it in fact _does_ happen -- you don't learn anything. You gain 0 information. 2. The less certain you are about something (the closer to 0% confident you are), the more information you gain when it happens. Put another way, the more _surprising_ an event is, the more informative. For example, you gain more information about an upcoming movie by learning its director than by learning its genre, because there are many more directors than genres, so you'll be less certain about who directed it. 3. There is no limit to how much information you can gain. As your certainty approaches 0%, information gain approaches _infinity_. For example, it would be surprising to me if I was suddenly picked up by a helicopter crew through my window while writing this. But it would still be even more surprising if I was picked up by a dragon, or if I suddenly observed a suspension of gravity on earth and all my belongings drifting around in the air. Getting a sense for the relation between intuition and surprise can take some time, but it's a very fruitful analogy once it clicks. One function which satisfies all of the above criteria is the following [1]: `34:T429,md`### 3. Dealing with uncertain outcomes One of the most powerful features of Foretold is the ability to predict and resolve questions using custom distributions. However, to understand how we score these, it will be easier to start with the binary case. Suppose you've made a prediction of ${p_uncertain}% that it will rain 🌧 tomorrow , against a crowd aggregate 📈 of ${a_uncertain}% . However, when tomorrow comes, it's not clear-cut whether that's true... * ...maybe it rained for 80% of the day but was sunny otherwise. ⛅🌧🌧🌧🌧 * ...maybe you didn't go outside until the afternoon, and can't verify what the weather was like before then. 🏠 * ...maybe there was a confusing mix of some drops of water but also lots of sun, and you don't really know whether that counts as rain. 💧🌞 In these cases you might want to resolve the question not as **True** (or 100%), but as only ${o_uncertain}% true, to reflect your uncertainty. In this case, your score is simply a weighted combination of both the rain and no-rain outcomes [2]: `35:Ta22,rel_score_graph = { let target = DOM.element('div'); functionPlot({ width: math.min(width,max_width), xAxis: { domain: [0, 1], label: "Prediction"}, yAxis: { domain: [-2, 1], label: "Scores"}, data: [ { fn: (scope) => score(scope.x, undefined, o_uncertain/100, "percentage", "nonMarketScore"), graphType: 'polyline', attr: {"stroke-width": stroke_width} }, { fn: (scope) => score(scope.x, a_uncertain/100, o_uncertain/100, "percentage"), graphType: 'polyline', attr: {"stroke-width": stroke_width}, color: 'black' }, { points: [ [p_uncertain/100, score(p_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")], [p_uncertain/100, score(a_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")] ], fnType: 'points', graphType: 'polyline', color: 'green', attr: {"stroke-width": stroke_width} }, { points: [ [p_uncertain/100, score(a_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")], [a_uncertain/100, score(a_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")] ], fnType: 'points', graphType: 'polyline', attr: {"stroke-width": 0.3}, color: 'green' }, { points: [ [0, 0], [0, score(p_uncertain/100, a_uncertain/100, o_uncertain/100, "percentage")] ], fnType: 'points', graphType: 'polyline', color: 'green', attr: {"stroke-width": stroke_width} }, { points: [ [p_uncertain/100, score(p_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")] ], fnType: 'points', graphType: 'scatter', color: 'red', attr: {"stroke-width": point_width} }, { points: [ [a_uncertain/100, score(a_uncertain/100, undefined, o_uncertain/100, "percentage", "nonMarketScore")] ], fnType: 'points', graphType: 'scatter', color: 'orange', attr: {"stroke-width": point_width} }, { points: [ [p_uncertain/100, score(p_uncertain/100, a_uncertain/100, o_uncertain/100, "percentage")] ], fnType: 'points', graphType: 'scatter', color: 'green', attr: {"stroke-width": point_width} } ], target }); return target; }36:Tcf9,vegalite({ data: { values: convertToDiagram(pointScoreDistribution.data.points, time_input.startingTime, overTime.pointScoreDistribution("MarketScore").data.finalX) }, height: 350, config: { axis: { tickSize: 2, grid: true, gridColor: "#f6f6f6" } }, "layer": [{ mark: {"type": "area", "interpolate": "step-after"}, width: math.min(width,max_width_px), encoding: { x: { field: "x", type: "quantitative", scale: {domain: [0,10.5]} }, y: { field: "y", type: "quantitative", scale: {domain: [-0.2,1.4]} }, color: {value: "rgba(180, 180, 180, 0.6)"} } }, { mark: {type: "circle"}, data: {values: time_view_ps[0]}, encoding: { x: { field: "x", type: "quantitative" }, y: { field: "y", type: "quantitative" }, size: {value: 80}, color: {value: "red"} } }, { mark: {type: "circle"}, data: {values: time_view_ps[1]}, encoding: { x: { field: "x", type: "quantitative" }, y: { field: "y", type: "quantitative" }, size: {value: 80}, color: {value: "orange"} } }, { data: { values: [{data: overTime.averagePointScore("MarketScore", time_input.startingTime).data}] }, mark: "rule", encoding: { y: { field: "data", type: "quantitative", axis: {title: "Score"}, }, strokeDash: {value: [5,5]}, strokeCap: {value: "butt"}, color: {value: "green"}, size: {"value": 1} } }, { data: { values: [{"Title": `Average score ${overTime.averagePointScore("MarketScore", time_input.startingTime).data.toFixed(2)}`}] }, mark: { "type": "text", "dx": overTime.averagePointScore("MarketScore", time_input.startingTime).data, "align": "center", "baseline": "bottom" }, "encoding": { "text": {"field": "Title", "type": "nominal"} } }, { data: { values: [{data:0}] }, mark: "rule", encoding: { y: { field: "data", type: "quantitative", }, color: {value: "#666"}, } }, { data: { values: [{tt: time_input.startingTime}] }, mark: "rule", encoding: { x: { field: "tt", type: "quantitative", axis: { "title": "Time" } }, color: {"value": "#666"}, size: {"value": 2} } }, { data: { values: [{tt: time_input.measurements.resolution.time}] }, mark: "rule", encoding: { x: { field: "tt", type: "quantitative", }, color: {"value": "#666"}, size: {"value": 2} }, }, { data: { values: [{"Threshold": "Resolution time"}] }, mark: { "type": "text", "angle": "90", "dy": -30*time_input.measurements.resolution.time, "align": "center", "baseline": "bottom" }, "encoding": { "text": {"field": "Threshold", "type": "nominal"} } } ] })37:T646,md` You'll be pleased to learn that the scoring process for these works in basically the same way as for binary questions. The key to seeing this lies in understanding the section above called "Dealing with uncertain outcomes". Basically, we can think of a Yes/No question as a Number-question with only two outcomes. It's kind of like the question "how many of my shots at goal will I actually score?" except that when the outcomes are True and False those have a special status of importance among binary events. When a Yes/No question resolves, we see how much probability you put on each outcome, and weigh that by how much probability the resolition put on each outcome. Distribution scoring works in the same way, only that now instead of 2 outcomes, we have an _infinite_ number of outcomes. In this case, the equivalent of the log-score is something known as the [Kullback-Leibler Divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence). It is a way of measuring how similar one distribution is to another, reference distribution. You can also think of it in terms of surprise as, "the expected increase in surprise of switching to using one model of the world, compared to your current model". In the context of scoring, it answers the question "if the resolution is the true model of the world, how much did our surprise increase by listening to your prediction instead?" -- just like for the log score! _Try playing around with the inputs below to get a sense of how this works, using the [Foretold input format](https://observablehq.com/@oagr/foretold-inputs)._ `38:T44e,time_input = ({ measurements: { agentPredictions: [ {time: t_p, measurement: {data: 0.1, dataType: "percentage"}}, {time: 3., measurement: {data: 0.3, dataType: "percentage"}}, {time: 9., measurement: {data: 0.2, dataType: "percentage"}}, ], marketPredictions: [ {time: 1., measurement: {data: 0.6, dataType: "percentage"}}, {time: 2.1, measurement: {data: 0.5, dataType: "percentage"}}, {time: 3.1, measurement: {data: 0.44, dataType: "percentage"}}, {time: 3.6, measurement: {data: 0.40, dataType: "percentage"}}, {time: 4.3, measurement: {data: 0.32, dataType: "percentage"}}, {time: 7.2, measurement: {data: 0.35, dataType: "percentage"}}, {time: 8.8, measurement: {data: 0.27, dataType: "percentage"}}, {time: 9.2, measurement: {data: 0.12, dataType: "percentage"}}, {time: 10.2, measurement: {data: 0.1, dataType: "percentage"}}, ], resolution: {time: t_res, measurement: {data: 0.0, dataType: "percentage"}} }, startingTime: 0, })39:[] 31:{"id":"455b5655170cdca1","slug":"foretold-scoring-tutorial","title":"Foretold Scoring Tutorial","thumbnail":"41eb86ccd81204815a9ad1066651b11b141cbdea450b6e67748b9b7a91f75a03","access_level":"public","create_time":"$D2019-10-06T13:31:45.409Z","update_time":"$D2020-08-11T06:26:47.277Z","view_time":null,"first_public_time":"$D2019-10-06T18:12:05.245Z","has_unpublished_edits":null,"fork":true,"stars":1,"starred":false,"owner":{"type":"team","id":"6787ecb194e8d603","login":"jjj","name":"Jacob Lagerros","avatar_url":"https://avatars-next.observableusercontent.com/5637ffcecfc0d41d940cd15adca85029ea1ebdf9974dc2a822d37f1b2d7269e6","tier":"starter_2024"},"version":2159,"publish_version":2159,"role":null,"authored":false,"authors":[{"type":"individual","id":"89c462454a840e10","login":"jjj","name":"Jacob Lagerros","avatar_url":"https://avatars-next.observableusercontent.com/5637ffcecfc0d41d940cd15adca85029ea1ebdf9974dc2a822d37f1b2d7269e6","tier":null}],"members":[{"type":"individual","id":"89c462454a840e10","login":"jjj","name":"Jacob Lagerros","avatar_url":"https://avatars-next.observableusercontent.com/5637ffcecfc0d41d940cd15adca85029ea1ebdf9974dc2a822d37f1b2d7269e6","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":6,"collection":{"id":"042e10cad1b70bd9","slug":"foretold-scoring","title":"Foretold scoring","owner":{"id":"6787ecb194e8d603","login":"jjj","name":"Jacob Lagerros"}},"type":"notebook","body":{"title":"Foretold Scoring Tutorial","cells":[{"id":0,"value":"md`# Foretold Scoring Tutorial`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":948,"value":"md`\n## Overview\n_This interactive tutorial walks through how predictions are scored on Foretold. It focuses on building up intuition, step-by-step, with relatively little math._\n\n_If you just want a page quickly summarising the system, you can find that [here](https://observablehq.com/d/72c0744e89e7a297)._ \n\n_This tutorial works best on a desktop._\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":660,"value":"md`\n## Basics of the scoring system\nAt its core, the Foretold scoring system answers the question: \n\n> How much true information did a forecaster provide, relative to what the prediction system already knew, over a specific period of time?\n\nTo answer this we need:\n1. A way of **measuring information**\n2. A way of **comparing new vs. prior information**\n3. A way of **dealing with cases where the final answer is uncertain**\n4. A way of **incorporating predictions that change over time**\n\nIn brief, this is implemented as follows, with numbers indicating which of the above criteria is met by each component:\n\n> You score is the logarithm (1) of your prediction divided by the aggregate prediction (2), weighed by the resolution (3), and averaged over time (4).\n\nThe rest of the tutorial walks through each component step by step.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":632,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2017,"value":"{\n let target = DOM.element('div');\n functionPlot({\n width: math.min(width,max_width),\n xAxis: { \n domain: [0, 1],\n label: \"Prediction of X\"\n },\n yAxis: { \n domain: [0, 10], \n label: \"Info gain if X happens\"\n },\n \n data: [\n {\n fn: \"-log2(x)\",\n graphType: 'polyline',\n attr: {\"stroke-width\": stroke_width}\n }\n ],\n target\n });\n return target;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":991,"value":"md`\nThe units of this function are called \"bits\". Now that we have a way of measuring information, we can construct a basic prediction score.\n\nThe key intuition is as follows. Your prediction is good to the extent that, if we take your word for it, we don't learn much from the event itself. If you say 50%, and the event happens, we gain 1 bit of information (see the graph above). If you say 70%, and the event happens, we gain roughly 0.5 bits. As your prediction becomes better, we gain less information from the event itself -- instead getting that information from your prediction. \n\nThis is part of what it _means_ to be a good predictor. Rather than having to wait until the event happens, we can just check with you.\n\nSo we give you a score where you lose more points the more surprised we were after having listened to you. That is, just the negative of the previous graph. ` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1724,"value":"md`_Try moving the slider to see how the score changes._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2111,"value":"log_score_graph = {\n let target = DOM.element('div');\n functionPlot({\n width: math.min(width,max_width),\n xAxis: { \n domain: [0, 1],\n label: \"Prediction of X\"\n },\n yAxis: { \n domain: [-10, 0], \n label: \"Score if X happens\"\n },\n \n data: [\n {\n fn: \"log2(x)\",\n graphType: 'polyline',\n attr: {\"stroke-width\": stroke_width}\n },\n {\n points: [\n [p_first/100, Math.log2(p_first/100)]\n ],\n fnType: 'points',\n graphType: 'scatter',\n attr: {\"stroke-width\": point_width}\n }\n ],\n target\n });\n return target;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1016,"value":"firstId = {\n var w = math.min(width,max_width_px)\n var w_string = `width: ${w}px`\n return bind(range(80, \"firstId\", w_string), viewof p_first);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":998,"value":"tex`\nlog\\_score(${p_first}\\%) = \\log_2(${p_first}\\%) = ${score(p_first/100, undefined, 1, \"percentage\", \"nonMarketScore\").toFixed(2)}\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1155,"value":"md`This function is actually just a logarithm, and so we simply call your score a \"log score\".`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":648,"value":"md`*[1] This is not the only function that satisfies these criteria. Moreover, these criteria alone are not sufficient to exactly pin down a rigorous notion of information. Nonetheless, they serve the purposes of this trial well, and if you want to go more in-depth on the full explanation, I'd recommend [E. T. Jaynes' _Probability Theory: The Logic of Science_, chapter 11: \"Discrete prior probabilities: the entropy principle\"](http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf).*`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":645,"value":"md`### 2. Comparing with prior knowledge\nThe above is not sufficient for scoring. If we just gave you points for how close you brought us to the truth, you could just copy the predictions of others, and get as many points as they did.\n\nMoreover, since we're never perfectly certain, there will always be some surprise. And as we're penalising _any_ surprise, you'll always get negative points. Not that fun. \n\nTo solve these and other problems, we take the difference between your score and an _aggregate_ score, based on the combined predictions of all other users. Thereby we're measuring how much information you provided over and above what the crowd already knew (or how much you removed, in case you made the aggregate _less_ accurate).`\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1124,"value":"tex.block`\\begin{aligned}\n score(you=x\\%, market=y\\%) = log\\_score(x\\%) - log\\_score(y\\%) = \\log_2(\\tfrac{x\\%}{y\\%})\n\\end{aligned}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1163,"value":"md`_(We're moving from taking the difference between the scores, to then dividing them. If you recall high school maths, this is just [a neat property of how logarithms work](https://people.richland.edu/james/lecture/m116/logs/properties.html).)_`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":658,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":694,"value":"tex.block`\n\\begin{aligned}\nscore(p=${p_uncertain}\\%,📈 = ${a_uncertain}\\%, 🌧=${o_uncertain}\\%) =\n${o_uncertain}\\%\\log_2( \\tfrac{${p_uncertain}\\%}{${a_uncertain}\\%}) + ${100-o_uncertain}\\%\\log_2(\\tfrac{${100-p_uncertain}\\%}{${100-a_uncertain}\\%}) =\n${score(p_uncertain/100, a_uncertain/100, o_uncertain/100, \"percentage\").toFixed(2)}\n\n\\end{aligned}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":715,"value":"md`_Try playing around with the values in the textboxes above or sliders below to see how the equation changes._` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":860,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":193,"value":"md`**Your prediction:** ${p_uncertain}%`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":120,"value":"playerId = bind(range(80, \"playerId\", slider_width_string), viewof p_uncertain)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1929,"value":"viewof aggregate = md`**Aggregate prediction:** ${a_uncertain}%`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1926,"value":"agg_id = bind(range(60, \"aggId\", slider_width_string), viewof a_uncertain)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1933,"value":"viewof resolution = md`**Resolution:** ${o_uncertain}%`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1931,"value":"resId = bind(range(100, \"resId\", slider_width_string), viewof o_uncertain)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2142,"value":"md`_[2] Note that in the current version of Foretold uncertain resolutions are only supported for Number or distribution questions, not for binary Yes/No questions._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1237,"value":"md`### 4. Updating over time\n\nAbove we looked at cases where you make one prediction against the current aggregate. However, in the real world things are different. \n\nAs you think, read, build models, or discuss with others, you might change your mind about the question. So will everyone else, and as a result the aggregate prediction will also be updated. \n\nIn order to score these updates, we first look at each point in time and compute your score just as we did above. But then we take the average of your score over all time-points.\n\nThis is shown in the graph below, where the difference between your score and the aggregate is the shaded grey area.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1922,"value":"md`_Play around with the sliders to adjust the time of a player prediction of 10%, and the time the question resolved as \"False\"._`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1239,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1272,"value":"t_player = md`**Prediction time:** ${t_p}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1904,"value":"timeId = {\n var w = math.min(width,max_width_px);\n var w_string = `width: ${w}px`;\n return bind(range(2., \"timeId\", w_string, 10., 0.1), viewof t_p);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1886,"value":"t_resolution = md`**Resolution time:** ${t_res}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1900,"value":"resTimeId = {\n var w = math.min(width,max_width_px);\n var w_string = `width: ${w}px`;\n return bind(range(10, \"resTimeId\", w_string, 11., 0.1), viewof t_res);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1435,"value":"table(p_t_table, {\n sortable: true,\n columns: { \n time: {\n formatter(val, i, row) { return styler(val, row, true) }\n },\n \n prediction: {\n formatter(val, i, row) { return styler(`${100*val}%`, row) }\n },\n \n log_score: {\n formatter(val, i, row) { return styler(val, row) }\n },\n \n predictor: {\n formatter(val, i, row) { return styler(val, row) }\n }\n \n }\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1256,"value":"md`Moreover, currently, the \"clock starts ticking\" from the moment the question opens. So if the question asks whether it will rain in a month, but you only start predicting two weeks in, you'll only be able to make half as many points as you would otherwise.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":302,"value":"md`## Number-question scoring`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":807,"value":"md`Above we derived some intuition for how we score \"Yes/No\" questions. \n\nNow we'll consider Number-questions, like \"How many times will I visit the gym this week?\" or \"How fast will the fastest Olympian run in 2020?\"\n\nAnswers to these questions capture our uncertainty using distributions: \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":823,"value":"viewof distribution_showcase = input(\n \"2 to 7\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":827,"value":"toPdfDiagram(\n guesstimatorInputToCdf({\n guesstimatorInput: distribution_showcase,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n }))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":820,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":306,"value":"playerScores = columns({\n title: md`**Your prediction**`, \n log_score: md`Log score: ${score(playerCdf, aggregateCdf, resolutionCdf, \"floatCdf\", \"nonMarketScore\").toFixed(2)}`, \n rel_score: md`Score vs aggregate: ${score(playerCdf, aggregateCdf, resolutionCdf, \"floatCdf\", \"marketScore\").toFixed(2)}`})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":308,"value":"viewof player_distribution = input(\n \"2 to 7\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":310,"value":"toPdfDiagram(\n guesstimatorInputToCdf({\n guesstimatorInput: player_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n }))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":364,"value":"md`___`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":349,"value":"aggregateScores = columns({\n title: md`**Aggregate prediction**`, \n log_score: md`Log score: ${score(aggregateCdf, playerCdf, resolutionCdf, \"floatCdf\", \"nonMarketScore\").toFixed(2)}`, \n rel_score: md`Score vs you: ${score(aggregateCdf, playerCdf, resolutionCdf, \"floatCdf\", \"marketScore\").toFixed(2)}`})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":336,"value":"viewof aggregate_distribution = input(\n \"1 to 9\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":337,"value":"toPdfDiagram(\n guesstimatorInputToCdf({\n guesstimatorInput: aggregate_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n }))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":369,"value":"md`___`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":352,"value":"resolutionScores = columns({\n title: md`**Resolution**`, \n log_score: md`Log score: ${score(resolutionCdf, playerCdf, resolutionCdf, \"floatCdf\", \"nonMarketScore\").toFixed(2)}`, \n rel_score: md`Score vs aggregate: ${score(resolutionCdf, aggregateCdf, resolutionCdf, \"floatCdf\", \"marketScore\").toFixed(2)}`})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":339,"value":"viewof resolution_distribution = input(\n \"=normal(3,1)\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":341,"value":"toPdfDiagram(\n guesstimatorInputToCdf({\n guesstimatorInput: resolution_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n }))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2133,"value":"md`___\n\n___`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":284,"value":"md`## Variables (used for interactive components)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":736,"value":"md`### Views`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":999,"value":"viewof p_first = new View(60)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":286,"value":"viewof p_player = new View(0.8)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":289,"value":"viewof p_agg = new View(0.6)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":292,"value":"viewof p_res = new View(1)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":752,"value":"viewof p_uncertain = new View(60)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":756,"value":"viewof a_uncertain = new View(50)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":757,"value":"viewof o_uncertain = new View(80)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1264,"value":"viewof t_p = new View(2.)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1304,"value":"viewof t_res = new View(10.)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1384,"value":"time_view_ps = [\n pointScoreDistribution.data.points.filter( obj => player_predictions.includes( obj.x ) ),\n pointScoreDistribution.data.points.filter( obj => !player_predictions.includes( obj.x ) )\n ]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1090,"value":"md`### Design`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1091,"value":"max_width = 650","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1113,"value":"max_width_px = 620","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1191,"value":"stroke_width = 2","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1196,"value":"point_width = 7","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2125,"value":"slider_width_string = {\n const w = math.min(width,max_width_px);\n return `width: ${w}px`\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":742,"value":"md`### Other`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":314,"value":"playerCdf = guesstimatorInputToCdf({\n guesstimatorInput: player_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":316,"value":"aggregateCdf = guesstimatorInputToCdf({\n guesstimatorInput: aggregate_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":318,"value":"resolutionCdf = guesstimatorInputToCdf({\n guesstimatorInput: resolution_distribution,\n simulationSampleCount: 10000,\n outputSampleCount: 2000,\n smoothingWidth: 80\n })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1317,"value":"arr = pointScoreDistribution.data","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1247,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1314,"value":"player_predictions = time_input.measurements.agentPredictions.map( obj => obj.time )","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1462,"value":"p_t_table = { \n var player_t_ps = overTime.data[1][0].map( obj => ({\"time\": obj[0], \"prediction\": obj[1], \"log_score\": Number(score(obj[1], undefined, 0, \"percentage\", \"nonMarketScore\").toFixed(2)), \"predictor\": \"player\" }) );\n \n var agg_t_ps = overTime.data[1][1].map( obj => ({\"time\": obj[0], \"prediction\": obj[1], \"log_score\": Number(score(obj[1], undefined, 0, \"percentage\", \"nonMarketScore\").toFixed(2)), \"predictor\": \"aggregate\" }) );\n \n var res_t_ps = ({\"time\": overTime.data[1][2][0], \"prediction\": overTime.data[1][2][1], \"log_score\": 0, \"predictor\": \"resolution\"});\n \n var unsorted = player_t_ps.concat(agg_t_ps, res_t_ps)\n \n return unsorted.sort( (a,b) => (a.time < b.time) ? -1 : ((b.time > a.time) ? 1 : 0)) \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1482,"value":"res_t_ps = ({\"time\": overTime.data[1][2][0], \"prediction\": overTime.data[1][2][1], \"log_score\": 0, \"predictor\": \"resolution\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1249,"value":"overTime = new predictionAnalysis.PredictionResolutionOverTime(time_input.measurements);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1243,"value":"pointScoreDistribution = overTime.pointScoreDistribution(\"MarketScore\",3)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1445,"value":"p_t_table","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":747,"value":"md`### CDFs`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":744,"value":"df = [\n {\"Predictor\": \"You\", \"scoreType\": \"Log score\", \"score\": score(p_player, p_agg, p_res, \"percentage\", \"nonMarketScore\")},\n {\"Predictor\": \"Aggregate\", \"scoreType\": \"Log score\", \"score\": score(p_agg, p_player, p_res, \"percentage\", \"nonMarketScore\")},\n {\"Predictor\": \"You\", \"scoreType\": \"Relative score\", \"score\": -score(p_agg, p_player, p_res, \"percentage\")},\n];","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":233,"value":"md`## Helper functions`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":322,"value":"md`### Numerical`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":243,"value":"score = (player, aggregate, resolution, type=\"floatCdf\", marketType=\"marketScore\") => {\n if (type == \"floatCdf\") {\n return new predictionAnalysis.PredictionResolutionGroup(\n {agentPrediction: {\n data: { xs: player[\"xs\"], ys: player[\"ys\"] }, \n dataType: type},\n marketPrediction: (marketType == \"marketScore\")? {\n data: { xs: aggregate[\"xs\"], ys: aggregate[\"ys\"] }, \n dataType: type} : undefined,\n resolution: {\n data: { xs: resolution[\"xs\"], ys: resolution[\"ys\"] }, \n dataType: type}\n }).pointScore(predictionAnalysis[marketType])[\"data\"]}\n else {\n return new predictionAnalysis.PredictionResolutionGroup(\n {agentPrediction: {\n data: player, \n dataType: type},\n marketPrediction: (marketType == \"marketScore\")? {\n data: aggregate, \n dataType: type} : undefined,\n resolution: {\n data: resolution, \n dataType: type}\n }).pointScore(predictionAnalysis[marketType])[\"data\"]}\n }","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":324,"value":"md`### html`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":245,"value":"range = (value, id, style=\"\", max=100, step=1) => {return html``}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":320,"value":"input = (value) => html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1559,"value":"style = (val, shade=true, color=\"black\", bold=false) => {\n // Define strings used to make text bold\n var bold_open = (bold) ? \"\" : \"\"\n var bold_close = (bold) ? \"\" : \"\"\n \n if (shade) {\n return html`${bold_open}${val}${bold_close}`\n } \n else {\n return html`${bold_open}${val}${bold_close}`} \n }","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1616,"value":"styler = (val, row, time=false) => {\n var new_val = (time) ? val : row.time\n // Check if prediction in shaded grey area\n if (t_p <= new_val && new_val <= t_res) {\n // Check if player or aggregate prediction\n if (row.predictor === \"player\") {\n // Final argument checks if controllable player prediction and makes bold if so\n // We also want timestamps not to have a color, for ontological reasons :)\n var p_color = (time)? \"black\" : \"red\"\n return style(val, true, p_color, (row.time == t_p) )\n \n } else if (row.predictor === \"aggregate\") {\n var a_color = (time)? \"black\" : \"darkorange\"\n return style(val, true, a_color)\n } else { \n return style(val, true, \"black\", (row.time == t_res)) \n }\n } else {\n // Predictions outside the shaded grey area\n return style(val, false)\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":330,"value":"md`### Other`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1241,"value":"convertToDiagram = (ar, startingX, endingX) => {\n // Pads input array on both sides\n let initialX = ar[0].x;\n let initialY = ar[0].y;\n let lastX = ar[ar.length-1].x\n let lastY = ar[ar.length-1].y;\n let lastArray = [{x: startingX, y: 0}, {x: initialX, y: initialY}].concat(ar, [{x: endingX, y: lastY}, {x: endingX, y: 0}]);\n return lastArray\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":214,"value":"md`## Imports`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"import {View, bind} from \"@mbostock/synchronized-views\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":208,"value":"import {guesstimatorInputToCdf, toPdfDiagram} from \"@oagr/guesstimator-library\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":210,"value":"predictionAnalysis = require('@foretold/prediction-analysis@0.0.7/dist/index.js')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2149,"value":"import {columns} from \"@bcardiff/observable-columns\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":375,"value":"vegalite = require(\"@observablehq/vega-lite@0.2\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":428,"value":"import {vl} from '@vega/vega-lite-api'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2152,"value":"d3 = require(\"d3@3\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":852,"value":"functionPlot = d3, require(\"function-plot@1/dist/function-plot.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1024,"value":"math = require('https://cdnjs.cloudflare.com/ajax/libs/mathjs/1.5.2/math.min.js')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1438,"value":"import { table } from \"@tmcw/tables@513\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W39","stdlib":"1","titling":"auto"}} 32:"$31:body"