43:["$","title",null,{"children":"Shapley Value Experiments. Part I | Nuño Sempere | Observable"}] 4b:T1409,md `## Commentary: What is happening here? ### Setup Background: We are using the data from [an amplification experiment](https://observablehq.com/@jjj/untitled/2) and cutting the results in a different way We are taking a prediction (P_initial), and for every prediction (Pn) that follows, we are asking: - Is (Pn) closer to the last aggregate before it (AGn), or to (P_initial)? - How much better, or worse, does (Pn) do in comparison to (AGn)? Call this quantity X. - If (Pn) is closer to (P_initial), and (Pn) does better than (AGn), increase or decrease the helpfulness of (P_initial)'s user by X - If there are m (P_initial)s such that (Pn) is closer to (P_initial) than to the (AGn), then instead of X, use X/m With this, what we want to answer is: Do predictions influenced by someone do better? Is this person a positive influence? ### Comments - Predicting earlier and wrong is penalized much more strongly than predicting late and wrong - This set-up penalizes users who make a lot of predictions because of the same issues that the normal scoring system has: people don't outperform the market. But this isn't the only factor; compare Misha Yagudin and JK, and geesh and holomanga - The following users are particularly interesting: - Reprisal - Individually, he did pretty badly in: - [Unbound Prometheus: Europe had a lower birthrate than China [1]](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15?state=closed) - [Unbound Prometheus: The Han Dynasty in China established a state monopoly on salt and metals [1] ](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15/m/050b48c9-022e-4b53-95ec-a8f371292b93) - [Unbound Prometheus: pre-Industrial Revolution, average French wage was what percent of the British wage? [2]](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15/m/7a2774d8-6b6e-468c-84cd-d91104ebefbb) - etc. - But correctly in: - [Unbound Prometheus: Pre-Industrial Britain had a legal climate more favorable to industrialization than continental Europe [5] ](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15/m/bbe62da4-e100-4935-a419-30477a167540), where people moved in his direction. - [https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15/m/e827ff9f-7327-4243-9e72-cbf9e6da9376](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15/m/e827ff9f-7327-4243-9e72-cbf9e6da9376) - etc. - Because in the first case he predicted relatively late, and because his errors were not repeated. i.e. he didn't have much influence, he gets a small positive score. Personally I would be very curious to see how well he does if he participates in the next rounds, and in particular if he learns to use wider intervals. - holomanga & geesh. They're the users who earnt the most money, but both had an (unadjusted) negative score. They also tend to predict earlier, and so get hit hard. But holomanga gets hit (much) harder. - NunoSempere. This is myself. This position depends on the specific distance (see below), and that depends on a number of judgement calls; with other distances I am slightly negative. - Elizabeth. Her answer to [Unbound Prometheus: Just before the Industrial Rev…ere less friendly towards science than Europe [4]" id: "1f56fcf1-21c8-4149-8abf-4feed882d31c](https://www.foretold.io/c/f19015f5-55d8-4fd6-8621-df79ac072e15?state=closed) is substantially different from her prior, and this is interesting because her prior wasn't neutral. I think that Elizabeth's Bayes factor, i.e., the direction of the update is more interesting in this case than the resolution. In general, I would have wanted there to be more cases in which Elizabeth, as opposed to Priors Bot, had made an initial prediction (or, are both the same)? - Note on degrees of freedom & judgement calls: Geesh is in general "more helpful" than holomanga, but the degree to which this is so depends on the distance used. Elizabeth is generally negative. I am sometimes slightly negative. Reprisal is usually near the top, but can go down - To compute the distance between cdfs, I am using: Integral(| cdf 1 - cdf2|). A next step, and perhaps using a better distance (KL divergence?), would be to make the influence of one prediction on another depend on the distance. But if you multiply by a factor of 1/(1+constant1.distance), or of (constant2 - constant3.distance), you already have an infinite number of degrees of freedom. The principled way to do this, Shapley values, involves information which we do not have. - Relationship to Shapley value: Tenuous. This is Shapley value with a lot of terms missing, or counterfactual value dividing by the number of participants. - The first order effect is missing. - I get the impression that getting something similar to Shapley values from here wouldn't be that difficult? - A very convenient assumption to make would be that later users whose predictions are very similar to someone else's previous prediction would have predicted the aggregate or something closer to the aggregate instead.`4c:Tc1f,async function getQuestionData(questionID){ let query = ` { measurements(measurableId:"${questionID}" last:50){ total edges{ node{ id description createdAt updatedAt value{ floatCdf{ xs ys } } valueText competitorType agent{ id name } } } } } ` let req = await request(query) let total = req.data.measurements.total let edges = req.data.measurements.edges function getCreationTimeInitial(){ var result = [] for(let i =0; i creationTimeInitial[b] ? 1 : 0; }); return indices } let arrayIndicesSortedByCreationTime = getArrayIndicesSortedByCreationTime() function sortBy(arrayActedUpon, arrayWithIndices){ if(arrayActedUpon.length != arrayWithIndices.length){ return "Error; the arrays should be of the same length" }else{ var result = [] for(let i=0; i=0){ d0 = distance(originalData, data.values[locations[i]-1].floatCdf) } while(loc+2 < totalpredictions-1 && loc+2 < nextLocation){ let previousAggregation = data.values[loc+1].floatCdf //console.log(data.agentNames[loc+1], "should always be Aggregation Bot") let nextPrediction = data.values[loc+2].floatCdf let d1 = distance(originalData, nextPrediction) let d2 = distance(previousAggregation, nextPrediction) //console.log("Distance to agent", d1) //console.log("Distance agent-aggregate;", d2) // d2-d1 > 0 ) // && (d1<300) // d2 - d1 > radius // d2-d1 > 0 if((d2 - d1 > radius) && (d1 radius){ //console.log(data.agentNames[loc+2],data.creationTimes[loc+2]) //console.log(data.agentNames[loc+1], data.creationTimes[loc+2]) //data.creationTimes[loc+1] let contextLocation = ({ agentPrediction: { data: { xs: data.values[loc+2].floatCdf.xs, ys: data.values[loc+2].floatCdf.ys }, dataType: "floatCdf" }, marketPrediction: { data: { xs: data.values[loc+1].floatCdf.xs, ys: data.values[loc+1].floatCdf.ys }, dataType: "floatCdf" }, resolution: resolution.measurement }) //console.log(contextLocation) let PA = new predictionAnalysis.PredictionResolutionGroup(contextLocation) if(PA != undefined && PA.pointScore("MarketScore") != undefined){ //cv.push({ cv: PA.pointScore("MarketScore").data * (1/(1+(d1/Math.min(d0,100)))), location: loc+2}) cv.push({ cv: PA.pointScore("MarketScore").data, location: loc+2}) //console.log(PA.pointScore("MarketScore").data) } }else{ cv.push({cv: 0, location: loc+2}) } loc +=2 } } return(cv) }4e:T4ed,async function getSVofQuestionsInChannel(radius){ let queryChannel = ` { measurables(channelId: "f19015f5-55d8-4fd6-8621-df79ac072e15" last:100){ total edges{ node{ id name state min max } } } }` let r2= await request(queryChannel) function getQuestions(){ let edges2 = r2.data.measurables.edges let result = [] for(let edge of edges2){ result.push({id: edge.node.id, name: edge.node.name, state: edge.node.state, min: edge.node.min, max:edge.node.max}) } return result } let questions = getQuestions() let result = [] for(let question of questions){ if(question.state =="JUDGED"){ // && question.min == 0 && question.max ==100){ var sv = undefined try { sv = await getSVofPeopleInQuestion(question.id, radius) }catch(error){ console.log(error) console.log(`Error in question ${question.name}`) sv = undefined } if(sv !=undefined){ result.push({ name: question.name, id: question.id, sv: sv }) } } } return result }4f:[] 47:{"id":"9095daea1eca9bdc","slug":"shapley-value-experiments-part-i","title":"Shapley Value Experiments. Part I","thumbnail":"4927b51ab986e2e41ffc053e3ce00b7c3de7cd7f46c9a3e6f0e681c0aeff5d5e","access_level":"public","create_time":"$D2019-11-21T09:48:39.926Z","update_time":"$D2020-10-30T19:32:59.798Z","view_time":null,"first_public_time":"$D2019-11-21T13:16:41.972Z","has_unpublished_edits":null,"fork":true,"stars":1,"starred":false,"owner":{"type":"team","id":"07a813a05afbeeaf","login":"nunosempere","name":"Nuño Sempere","avatar_url":"https://avatars-next.observableusercontent.com/512eae7d32fc25bba414910bc3c00fac559004bc68c3e14fa07fa7c6544b630f","tier":"starter_2024","bio":"Researcher at the Quantified Uncertainty Research Institute","followed_by_you":false,"follows_you":false},"version":1329,"publish_version":1329,"role":null,"authored":false,"authors":[{"type":"individual","id":"cee02f8e9acb9aee","login":"nunosempere","name":"Nuño Sempere","avatar_url":"https://avatars-next.observableusercontent.com/bc840e103afe5117def6829a8a8476725da032e7bb7cc3cb8c49257be439a233","tier":null,"bio":"Researcher at the Quantified Uncertainty Research Institute","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"cee02f8e9acb9aee","login":"nunosempere","name":"Nuño Sempere","avatar_url":"https://avatars-next.observableusercontent.com/bc840e103afe5117def6829a8a8476725da032e7bb7cc3cb8c49257be439a233","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Shapley Value Experiments. Part I","cells":[{"id":0,"value":"md`# Shapley Value Experiments. Part I`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1121,"value":"md `# Results`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":791,"value":"table(SortedBySV(SV), {\n sortable: true,\n columns: {\n name: {\n title: \"Name\",\n },\n sv : {\n title: \"~ helpfulness\",\n },\n numberofquestions: {\n title: \"Number of questions / 7\"\n }\n \n \n }\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1133,"value":"$4b","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1256,"value":"md `# Helper functions`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":380,"value":"md `### Distance function `","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":390,"value":"function getFunctionFromCdf(cdf){\n let min = cdf.xs[0]\n let max = cdf.xs[ cdf.xs.length -1]\n function result(p){\n if(pmax){\n return 1;\n }else{\n let s1 = 0\n let s2 = cdf.xs.length -1\n \n let compare = Math.floor((s1+s2)/2)\n for(let i=0; i<10; i++){ // This is enough to search 1000 predictions, because 2^10 = 1000.\n if(compare > p){\n s1 = compare\n }else if(compare < p ){\n s2 = compare\n }\n }\n return cdf.ys[compare]\n }\n }\n return result\n\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":384,"value":"function distance(cdf1, cdf2){ // this distance function can be directly replaced by the KL function, as long as it's expressed in terms of cdfs.\n let f1 = getFunctionFromCdf(cdf1)\n let f2 = getFunctionFromCdf(cdf2)\n \n //let min = 0\n let min = cdf1.xs[0] > cdf2.xs[0] ? cdf1.xs[0] : cdf2.xs[0]\n //let max = 100\n let l1 = cdf1.xs.length\n let l2 = cdf2.xs.length\n let max = cdf1.xs[l1-1] > cdf2.xs[l2-1] ? cdf1.xs[l1-1] : cdf2.xs[l2-1]\n \n let h = (max-min) / 2000\n \n let d = 0;\n let p = min\n for(let i = 0; i<2000; i++){\n d+= Math.abs(f1(p) - f2(p));\n p+=h\n }\n return d\n\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":494,"value":"md `### Get the data`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":498,"value":"$4c","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1032,"value":"getQuestionData(\"1f56fcf1-21c8-4149-8abf-4feed882d31c\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":436,"value":"md `### Get the locations where a person predicts` ","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":483,"value":"function findLocations(name, data){\n let locations = []\n for(let i = 0; i a.indexOf(v) === i); \n for(let uniqueName of uniqueNames){\n if(uniqueName != \"Priors Bot\" && uniqueName != \"Aggregation Bot\"){\n result.push({\n name: uniqueName,\n cvarray: (getCVofPersonInQuestion(findLocations(uniqueName, data), data.total, data, resolution, radius)) //.reduce((a, b) => a + b, 0)\n // Remove the reduce to find out where they come from.\n })\n }\n }\n return {result: result, total: data.total}\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1052,"value":"md `### Get the SVs from the CVs`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1058,"value":"async function getSVofPeopleInQuestion(questionID, radius){\n let CVs = await getCVofPeopleInQuestion(questionID, radius)\n let total = CVs.total\n CVs = CVs.result\n let arrayLocation = Array(total)\n arrayLocation.fill(0,0,total)\n \n for(let i =0; i a + b, 0)}\n \n }\n \n \n \n return CVs\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1067,"value":"getSVofPeopleInQuestion(\"5be0fba1-5701-4a83-816b-0ee4ab71073a\", 100)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":613,"value":"md `### Do this for every question in a channel`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":656,"value":"$4e","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":697,"value":"SV = getSVofQuestionsInChannel(100)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1266,"value":"md `### Order the result`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":785,"value":"function SortedBySV(SV){\n function getUniqueNames(SV){ \n let names= []\n for(let a of SV){\n let b = a.sv\n for(let i=0; i a.indexOf(v) === i); \n return uniqueNames\n }\n \n let result = []\n let uniqueNames = getUniqueNames(SV)\n for(let uniqueName of uniqueNames){\n var sv= 0;\n var numberofquestions = 0;\n for(let a of SV){\n let b = a.sv\n for(let i=0; i{return b.sv-a.sv})\n return(result)\n\n }\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":820,"value":"md `## Present the result in a table`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1273,"value":"import {table} from \"@tmcw/tables@513\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1128,"value":"table(SortedBySV(SV), {\n sortable: true,\n columns: {\n name: {\n title: \"Name\",\n },\n sv : {\n title: \"~ Shapley value ~ helpfulness\",\n },\n numberofquestions: {\n title: \"Number of questions / 7\"\n }\n \n \n }\n })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":77,"value":"md `## Foretold functions`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"questionID = \"bbe62da4-e100-4935-a419-30477a167540\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"query = `\n{\n measurements(measurableId:\"bbe62da4-e100-4935-a419-30477a167540\" last:500){\n total\n edges{\n node{\n id\n description\n createdAt\n updatedAt\n value{\n floatCdf{\n xs\n ys\n }\n }\n\n valueText\n competitorType\n agent{\n id\n name\n }\n \n }\n }\n }\n}\n`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"main_url = \"https://api.foretold.io/graphql?token=\";","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"HEADERS = ({\n \"Accept-Encoding\": \"gzip, deflate, br\",\n \"Content-Type\": \"application/json\",\n Accept: \"application/json\",\n Connection: \"keep-alive\",\n DNT: \"1\",\n Origin: \"https://api.foretold.io\"\n});\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":17,"value":"\noptions = query => ({\n method: \"POST\",\n headers: HEADERS,\n body: JSON.stringify({ query })\n});\n\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":20,"value":"async function request(query) {\n \n let foretold_token = \"\"\n \n return await fetch(main_url + foretold_token, options(query)).then(response =>\n response.json()\n );\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"predictionAnalysis = require('@foretold/prediction-analysis@0.0.7/dist/index.js')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4f","stdlib":"1"}} 42:null