31:[] 2f:{"title":"Introduction to Inference","cells":[{"id":0,"value":"md`# Introduction to Inference\n\nIFT6758, Fall 2019\n\nReading: [ISLR](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf) 3.1.1, 3.1.2 and [Bayesian Basics](https://m-clark.github.io/bayesian-basics) (intro - regression models)\n\nOptional reading: [MSMB](http://web.stanford.edu/class/bios221/book/Chap-Testing.html) 6.1 - 6.6 and [Statistics for Hackers](https://speakerdeck.com/jakevdp/statistics-for-hackers?slide=138)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"slide`### Inference\n\n* Meta-Algorithms: Evaluation of processes people use to learn from data\n* Science: How to go from the finite to the general?\n\n\"It is easy to lie with statistics, but hard to tell the truth without them.\" `","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"slide`### Hypothetical Reasoning\n\n* We've mostly been thinking about recognizing patterns occuring in observed data\n* But *hypothetical reasoning* is important. How could the world look intead?\n* Statistical inference provides quantitative machinery to help hypothetical reasoning\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"slide`### Plan\n\n* Inference for coin clips\n - Both mathematical and computational approaches\n* Inference in linear regression\n* Revisit problems from Bayesian perspective\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"slide`### Is your coin fair?\n\nHypothesis testing gives a formal framework for answering questions like,\n\n* How large a discrepancy would you need to see?\n* How many flips would convince you?\n\n${coin_flipping}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"slide`### Is your coin fair?\n\nHypothesis testing gives a formal framework for answering questions like,\n\n* How will you measure discrepancies?\n* How will you tell if it's meaningfully large?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":65,"value":"slide`### Model of Reality\n\n* A null model defines a default simulator of reality\n* Our goal is to determine whether an observed dataset is consistent with that reality\n\n

\n\nCounts of cancer deaths. 5 come from null model, where there is no spatial correlation.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":477,"value":"slide`### Model of Reality\n\n* A null model defines a default simulator of reality\n* Our goal is to determine whether an observed dataset is consistent with that reality\n\n

\n\nDistance vs. angle in three pointers from LA Lakers. 9 are from null model that there is a quadratic relationship.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":467,"value":"slide`### Measuring Discrepancies\n\n* We'll look in the data for potential discrepancies, and try to judge whether they are meaningfully large\n\n

\n\nIf you didn't have the error bars, you might think the two treatments are quite different.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"slide`### Challenges\n\n* All theories are approximations to reality\n* If your approximation isn't that believable, you will obviously reject the null (and no one will care)\n* Most critical assumption: Independence\n - Thinking you have more evidence than you do is usually much worse than small distributional differences\n\n

\n\nIf you had sampled twice as frequently, will you really have doubled your sample size?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"slide`### Measuring Discrepancies\n\n* The goal is to identify a statistic (any function of the data) that detects problems in your simulation of reality\n* For coin tossing, it makes sense to use\n${mtex_block`\n\\begin{aligned}\n\\hat{p}_n\\left(X_1, \\dots, X_n\\right) &= \\frac{1}{n}\\sum_{i = 1}^{n} X_i \\\\\n&= \\text{fraction heads}\n\\end{aligned}\n`}\nand sound an alarm if ${mtex`\\hat{p}_n`} is very far from 0.5\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":86,"value":"slide`### Not always obvious\n\n* What if you saw HHHHHHHHHHTTTTTTTTTT?\n* Seems fair according to our metric....\n* Different choices of test statistic are sensitive to different departures from your theory\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":68,"value":"slide`### Reference Distribution: Theory\n\n* Say you observed 59 heads in 100 flips.\n* How will you tell whether the measured discrepancy is meaningfully large?\n* For coin tossing, supposing a binomial simulator\n\n${mtex_block`\n\\Parg{\\hat{p}_n = \\frac{k}{n}} = {n \\choose k}p^k\\left(1 - p\\right)^{n - k}\n`}\n\n

\n\nThis image (and others) from [MSMB](https://www.huber.embl.de/msmb/Chap-Testing.html#multiple-testing).\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"slide`### Reference Distribution: Simulation\n\n* Alternatively, we can directly simulate from our model of the world\n* Computing test statistic on simulated data provides a reference distribution\n * Can _always_ be done when there's a generative model\n * Especially useful when no closed form analysis\n\n

\n\nSame reference distribution, from ${mtex`10^4`} simulations\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":84,"value":"slide`### Rejection Region\n\n* Signifance level: Want to keep false alarm rate below ${mtex`\\alpha`}%\n* Power: Want to be sensitive to small discrepancies\n\n

\n\nBy rejecting in the tails, we can reject as many types of outcomes as possible while making sure we rarely reject when the null (${mtex`p = 0.5`}) is actually true.\n\nDefinition: The ${mtex`p`}-value gives the area in the reference distribution of discrepancies that are \"more extreme\" than the one you've observed.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":557,"value":"slide`### Confidence Intervals\n\n* For your dataset, what are all the possible null hypotheses that you wouldn't have been able to reject?\n* This is richer information than just the test outcome\n\n

\n\nWe wouln't have rejected this ${mtex`\\theta_0`} because the test statistic lies in the bulk of the reference distribution.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":577,"value":"slide`### Confidence Intervals\n\n* For your dataset, what are all the possible null hypotheses that you wouldn't have been able to reject?\n* This is richer information than just the test outcome\n\n

\n\nWe start rejecting once we get past the red reference distributions. All the ${mtex`\\theta`}'s between these belong to our confidence interval.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":70,"value":"slide`### Recap: Overall Approach\n\n* Determine the effect of interest, and design a suitable test statistic\n* Define a null hypothesis, and come up with a null distribution for that statistic\n* Define the rejection region\n* Do the experiment and draw conclusions\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"slide`## Difference in Means\n\n* Common situation: Have two distributions, and we want to see whether they have the same mean\n* We don't have to assume they are normally distributed\n\n\n

\n\n[Follow along](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3#scrollTo=vrleqMPDxzGS).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":100,"value":"slide`### Measuring Discrepancy\n\n* Common to use a normalized difference, e.g. \n${mtex_block`\n\\hat{t}_n\\left(X_1, \\dots, X_n\\right) = \\frac{\\bar{x}_1 - \\bar{x}_2}{\\widehat{\\text{s.e.}}\\left(\\bar{x}_1 - \\bar{x}_2\\right)}\n`}\nwhere ${mtex`\\widehat{\\text{s.e.}}`} is an estimate of the standard error\n* If the means in the groups are the same, this is approximately ${mtex`t`}-distributed\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":491,"value":"slide`### Measuring Discrepancy\n\nThese are the distributions for ${mtex`\\bar{x}_1`} and ${mtex`\\bar{x}_2`}. Even though the original data are relatively far from gaussian, the central limit theorem kicks in.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":494,"value":"slide`### Measuring Discrepancy\n\nThe resulting normalized difference is about ${mtex`t`}-distributed.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"slide`### Evaluating Discrepancy\n\n* You don't have to use theoretical reference distribution\n* Permutation: Randomly reassign group labels and recompute the statistic\n\nThe separate histograms (one per group) become indistinguishable -- it's a simulation from the null for the raw data. Computing difference in means gives a reference distribution.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":510,"value":"slide`### Evaluating Discrepancy\n\n* You don't have to use theoretical reference distribution\n* Simulation: Randomly reassign group labels and recompute the statistic\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":147,"value":"slide` ## Linear Regression\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":150,"value":"slide`### Analogies\n\n* ${mtex`p \\longleftrightarrow `} use ${mtex`\\bar{x}`} in binomial model\n* ${mtex`\\mu_1 - \\mu_2 \\longleftrightarrow`} use ${mtex`{\\hat{t}}`} and central limit theorem\n* ${mtex`\\beta \\longleftrightarrow`} use ${mtex`\\hat{\\beta}`} in linear regression with iid gaussian errors\n\nGenerally: Parameter of interest ${mtex`\\longleftrightarrow`} reference distribution of a statistic under a model\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":166,"value":"slide`### Types of Questions\n\n* Association: Is there are relationship between ${mtex`x`} and ${mtex`y`}?\n* Model comparison: Is a model using additional features actually better?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":221,"value":"slide`### Testing association [one predictor]\n\nDefault to there being no association between ${mtex`x`} and ${mtex`y`}. Formally,\n\n${mtex_block`\n\\begin{aligned}\nH_0: \\beta_1 = 0 \\\\\nH_1: \\beta_1 \\neq 0\n\\end{aligned}\n`}\n\nHow should we measure the discrepancy?\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":190,"value":"slide`### Discrepancy\n\n**Measuring**: Continuous version of the earlier two-groups statistic,\n\n${mtex_block`\n\\begin{aligned}\n\\hat{t} = \\frac{\\hat{\\beta}_1}{\\widehat{\\text{s.e}}\\left(\\hat{\\beta}_1\\right)}\n\\end{aligned}\n`}\n\n**Evaluating**: Also has a ${mtex`t`}-distribution if you assume i.i.d. gaussian errors ${mtex`\\epsilon_i \\sim \\mathcal{N}\\left(0, \\sigma^2\\right)`}. We'll see some simulation-based alternatives in later lectures.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":233,"value":"slide`### Comment: Experimental Design\n\nThe formula for the standard error is illuminating,\n\n${mtex_block`\n\\begin{aligned}\n\\widehat{\\text{s.e.}}\\left(\\hat{\\beta}_1\\right) &= \\frac{\\sigma^2}{\\sum \\left(x_i - \\bar{x}\\right)^2}\n\\end{aligned}\n`}\n\n* Makes you want to spread out ${mtex`x_i`}\n * ... but make sure you can still check linearity!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":245,"value":"slide`### Model comparison\n\n* Suppose you want to compare a model that only uses first ${mtex`j`} features with the one that uses all ${mtex`p`}.\n* The relevant test is,\n\n${mtex`\nH_0: \\beta_{j + 1} = \\dots = \\beta_p = 0 \\\\\nH_1: \\beta_{j + 1}, \\dots, \\beta_p \\text{ arbitrary}\n`}\n\n\nFor example,\n* Are the higher-order terms in a polynomial regression actually helping?\n* Accounting for differences in backgrounds, does taking a data science course increase your income?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":259,"value":"slide`### Measuring Discrepancy\n\n* Full model's error, ${mtex_block`RSS = \\sum \\left(y_i - x_i^{T}\\hat{\\beta}\\right)^{2}`}\n* Submodel's error is, ${mtex_block`RSS_0 = \\sum \\left(y_i - x_i^{T}\\hat{\\beta}^0\\right)^{2}`}\nwhere ${mtex`\\hat{\\beta}^0`} is fit assuming the last ${mtex`p - j`} coordinates are 0.\n* Small model always has larger error rate, but is it really _that_ much worse?\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"slide`### Evaluating Discrepancy\n* Compare the error rates between the models,\n${mtex_block`\n\n\\hat{F} = \\frac{\\frac{1}{p - j}\\left(RSS_0 - RSS\\right)}{\\frac{1}{n - p - 1}RSS}`}\n\n* Under same regression assumptions, statistic follows an ${mtex`F`}-distribution\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":295,"value":"slide`### Nuances\n\n* A model comparison with one coefficient extra looks for an association with ${mtex`x_j`}, after controlling for other features\n * Question: I have a ${mtex`p`}-value for ${mtex`\\beta_j`}. I introduce a new variable to my regression. How does the ${mtex`p`}-value for the original variable change?\n* If you reject when there are many additional coefficients, you won't be able to pinpoint *which* of the ${mtex`\\beta_j`}'s are responsible\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":304,"value":"slide`## Bayesian inference\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":566,"value":"slide`### Main Idea\n\n* Instead of accepting or rejecting states of the world ${mtex`H`}, place probabilities over them\n* Instead of arguing whether coin is fair, define a distribution over plausible probabilities\n`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":308,"value":"slide`### Posterior Updating\n\n* Key logical device is Bayes rule,\n${mtex_block`\np\\left(H \\vert \\text{data}\\right) \\propto p\\left(\\text{data} \\vert H\\right)p\\left(H\\right)`}\n* New belief about world = new data ${mtex`\\times`} old belief\n* \"Update the prior to the posterior\"\n* Stronger priors are harder to overcome\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"slide` ### Some healthy perspective...\n

\n\n...\n\n

\n\nFrom Leo Breiman in [Nail finders, edifices, and Oz](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=2ahUKEwiZkoW66u3kAhVSSN8KHeS1BI4QFjABegQIABAC&url=http%3A%2F%2Fdigitalassets.lib.berkeley.edu%2Fsdtr%2Fucb%2Ftext%2F32.pdf&usg=AOvVaw1w1bS2VMuELCWdCoYZyntz)\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":321,"value":"slide`### Revisiting Coin Flips\n\n* Prior assigns probabilities to ${mtex`p \\in \\left[0, 1\\right]`}\n* We leave the update to a black box inference engine\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":351,"value":"slide`### Revisiting Coin Flips\n\n* Go to example [here](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":323,"value":"slide`### Revisiting Linear Regression\n\n* Almost exact same mechanics translates to linear regression\n* Go through example [here](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":325,"value":"slide`### Posterior Predictive Checks\n\n* Once you've fitted a model, try generating data from it\n* Do the simulated data look like your real data?\n* Do they match specific properties of your real data?\n * (means, quantiles, correlations, ...)\n* This idea is useful in frequentist settings too\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"slide`### Practical considerations\n\n* Modular, flexible ways of building models and drawing inference\n * E.g. [hierarchical modeling](https://mc-stan.org/users/documentation/case-studies/radon.html)\n* Prior choice is arbitrary, but you can measure sensitivity of inferences to that choice\n* Freedom in design can outweigh complexity of computation\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":46,"value":"import {coin_flipping} from \"@krisrs1128/sufficiency-illustrated\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":196,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":17,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}