25:["$","title",null,{"children":"P-Hacking | Philipp Burckhardt | Observable"}] 4b:T45e,md`Regularly, you encounter newspaper articles presenting the results of some newly published study, which found an association between eating some kind of food and a desired health outcome. Yet, chances are high that these bold announcements are untenable und could not be replicated in a controlled study. Often, they are obtained by mining existing data sets for correlations between different variables. Due to the sheer number of comparisons that can be made, it is just natural that by pure chance, one is going to find significant correlations, even if the variables were all independent! The folks at FiveThirtyEight wrote an [article](http://fivethirtyeight.com/features/you-cant-trust-what-you-read-about-nutrition/) on the shady practice of p-hacking, which refers to data mining methods people use to obtain significant research findings. As an exploratory step to generate hypotheses, such an approach has its merits, but more often than not people present their findings without disclosing the process of how they were obtained nor accounting for the multitude of hypothesis tests that were carried out.`4c:[] 47:{"id":"22e67ae78c1828e6","slug":"p-hacking","title":"P-Hacking","thumbnail":"d718c06dc6aef7a0084f1c543125a5fb1fc883c548bf3ab238dc6696c2ca635f","access_level":"public","create_time":"$D2018-04-12T20:00:33.902Z","update_time":"$D2018-10-17T10:39:47.979Z","view_time":null,"first_public_time":"$D2018-04-12T22:26:12.738Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"0079603d1a1150ec","login":"planeshifter","name":"Philipp Burckhardt","avatar_url":"https://avatars-next.observableusercontent.com/8456e6044dafd635770ffd6fc90d200a75eacd1bcedd4826ccce11ae14179985","tier":"starter_2024","bio":"Statistics & Data Science. Ascetic-ecstatic Node.js hacker.","followed_by_you":false,"follows_you":false},"version":194,"publish_version":194,"role":null,"authored":false,"authors":[{"type":"individual","id":"935ee8ecd82e8cf3","login":"planeshifter","name":"Philipp Burckhardt","avatar_url":"https://avatars-next.observableusercontent.com/8456e6044dafd635770ffd6fc90d200a75eacd1bcedd4826ccce11ae14179985","tier":null,"bio":"Statistics & Data Science. Ascetic-ecstatic Node.js hacker.","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"935ee8ecd82e8cf3","login":"planeshifter","name":"Philipp Burckhardt","avatar_url":"https://avatars-next.observableusercontent.com/8456e6044dafd635770ffd6fc90d200a75eacd1bcedd4826ccce11ae14179985","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":3,"collection":null,"type":"notebook","body":{"titling":"auto","title":"P-Hacking","cells":[{"id":0,"value":"md`# P-Hacking`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":19,"value":"$4b","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":23,"value":"md` In this section, we are getting our own hands diry in trying to find spurious correlations between the variables in the data used by the FiveThirtyEight authors. Seeing how easy it is to obtain essentially meaningless results, the results of this exercise (don't repeat it at home!) should serve as a reminder to be cautious when dredging through massive amounts of data. Finally, we will discuss some remedies one can employ when faced with the task of running a large number of hypothesis tests.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"nutrition = d3.csv(\"https://raw.githubusercontent.com/fivethirtyeight/data/master/nutrition-studies/raw_anonymized_data.csv\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"md`The data comes from a survey of 54 respondents, who answered questions about their eating habits (\"[food frequency questionnaire](https://en.wikipedia.org/wiki/Food_frequency_questionnaire)\") other questions about themselves. The survey in question is semi-quantitative since it collected information on both the frequency and quantity of the consumed foods. For the present analysis, we will not make any distinction between the two.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":38,"value":"md`**Exercise**: Start by extracting the variable names for the respondent's characterstics and store them in an array (the second to 27th key in each object is a personal characteristic)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":35,"value":"''","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":41,"value":"md`**Exercise:** Also extract the 1066 FFQ variable names and store them in an array (they can be found starting with the 28th key in each object)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":44,"value":"''","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":48,"value":"md`We will use [two-sample t-tests](https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/stats/ttest2) to check whether there is a significant difference in terms of food intake for respondent's with different characteristics. For example, we could check whether there is a difference in the consumption of green salad among those with diabetes and those without. Let us start by splitting the data set into the two groups:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":89,"value":"bifurcated = stdlib.bifurcateBy( nutrition, function( x ) { return x.diabetes === 'Yes'; } )","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":98,"value":"md`Extracting the frequencies of green salad consumption and running a t-test, we obtain:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":52,"value":"salad_diabetes = bifurcated[ 0 ].map( function( x ) { return parseInt( x.GREENSALADFREQ ); } )","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":60,"value":"salad_no_diabetes = bifurcated[ 1 ].map( function( x ) { return parseInt( x.GREENSALADFREQ ); } )","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":62,"value":"salad_test = stdlib.ttest2( salad_diabetes, salad_no_diabetes )","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":68,"value":"md`Let's have a look at the formatted output:`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":71,"value":"salad_test.print()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":73,"value":"md`Almost significant at the 5% level! Too bad, but surely we will have more luck if we repeat this exercise a bunch of times. Recall: Even if there was no real association whatsoever, when using a significance level of 5% (as is the standard), you expect to detect an average of five significant results per 100 tests (which corresponds to a false positive rate of 5%). But even if you were to detect a true effect, there is still a question of whether that effect is of any practical significance. As sample sizes become larger, hypothesis tests become better at detecting even very small differences across groups. Finally, we should keep in mind that *correlation does not imply causation*, so there always might be a confounder that affects both variables and is thus responsible for the observed difference!`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":78,"value":"md`**Exercise:** Iterate over the food questionnaire items and run a sequence of t-tests looking for a difference between those with diabates and those without. Save the p-values for each result and order them according to their value in ascending order. How many significant results did you obtained (at ${tex`\\alpha = 0.05`})?`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"''","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":82,"value":"md`**Solution:**`\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":84,"value":"pvalues = {\n var fq = Object.keys( nutrition[ 0 ] ).filter( function( x, i ) { return i > 27; } )\n var arr = new Array( fq.length );\n for ( var i = 0; i < fq.length; i++ ) {\n var diabetes = stdlib.pluck( bifurcated[ 0 ], fq[ i ] ).map( function( x ) { return parseInt( x ) } );\n var nodiabetes = stdlib.pluck( bifurcated[ 1 ], fq[ i ] ).map( function( x ) { return parseInt( x ) } );\n var out = stdlib.ttest2( diabetes, nodiabetes ); \n arr[ i ] = out.pValue;\n }\n return arr.sort();\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":109,"value":"md`### Bonferroni Correction`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":113,"value":"md`One solution for the problem of multiple comparisons is the so-called [Bonferroni correction](https://en.wikipedia.org/wiki/Bonferroni_correction). The procedure is quite simple: Instead of testing each individual hypothesis at a significance level of ${tex`\\alpha`}, test it at ${tex`\\alpha /m`}, where ${tex`m`}\\ is the number of hypotheses. Then, it can be shown that the familywise error rate (FWER), i.e. the probability of rejecting at least one true null hypothesis, is ${tex`\\le \\alpha`}.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":118,"value":"md`**Exercise:** Had you used a Bonferroni correction in above question, at what significance level should you have carried out your tests?` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":121,"value":"md`### Controlling the False Discovery Rate (FDR)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":128,"value":"md`One problem with the Bonferroni correction is that it is overly conservative. Hence, its use often coincides with a large increase in the false negative rate. An alternative that is often suggested is the the Benjamini–Hochberg procedure, which controls the [False Discovery Rate (FDR)](https://en.wikipedia.org/wiki/False_discovery_rate) at a signficiance level ${tex`\\alpha`}. The Benjamini–Hochberg procedure works like this:\n\n1. Order the p-values in ascending order and denote them as ${tex`P_{(1)} \\ldots P_{(m)}`}.\n2. For a given ${tex`\\alpha`}, find the largest ${tex`k`} such that ${tex`P_{(k)} \\leq {\\frac {k}{m}}\\alpha`}.\n3. Reject the null hypothesis for the k hypotheses corresponding to the k lowest p-values`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":140,"value":"md`Graphically, the procedure can be carried out as follows: Plot the ordered p-values ${tex`P_{(k)}`} against ${tex`k`} and draw a line through the origin with slope ${tex`\\frac{\\alpha}{m}`}. Declare as significant all hypotheses that are to the left of where the points cross the line.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":145,"value":"md`**Exercise:** Plot the ordered p-values you obtained early like this. How many significant results do you obtain when controlling the FDR at ${tex`\\alpha = 0.05`}?`\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":147,"value":"plt = {\n var x = stdlib.incrspace( 0, 1065, 1 );\n var y = x.map( function( x ) { return x * 0.05/1065 } );\n return stdlib.plot( [ x, x ],[ y, pvalues ], {\n 'xLabel': 'k',\n 'yLabel': 'P_k',\n 'width': 600,\n 'xMax': xRange[ 0 ],\n 'yMax': xRange[ 1 ]\n });\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":169,"value":"viewof xRange = {\n var form = html``;\n form.value = [ form.xMax, form.yMax ].map(i => i.valueAsNumber);\n form.xMax.oninput = () => form.value[0] = form.xMax.valueAsNumber;\n form.yMax.oninput = () => form.value[1] = form.yMax.valueAsNumber;\n return form;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":188,"value":"html`${plt.render('html')}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"md`\n* * *\n\n## Appendix\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6,"value":"md`\nAn appendix containing functionality used in running this notebook.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":16,"value":"d3 = require(\"https://d3js.org/d3.v5.min.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"stdlib = require( \"https://unpkg.com/@stdlib/stdlib@0.0.34/dist/stdlib-flat.min.js\" )","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"md`\n* * *\n\n## License\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":12,"value":"md`\nCopyright (c) 2018 The Stdlib Authors.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4c","stdlib":"1"}} 24:null