31:T6e3,slide`### Rejection Region **Some definitions** * ${mtex`p`}-value: the probability (i.e., the area in the reference distribution) of obtaining results at least as extreme as the observed results of a statistical hypothesis test, assuming that the null hypothesis is correct. - A smaller ${mtex`p`}-value means that there is stronger evidence in favor of the alternative hypothesis. * Signifance level ${mtex`\alpha`}: it defines the strength of evidence (i.e., significance) in probabilistic terms. ${mtex`\alpha`} represents the probability that tests will produce statistically significant results when the null hypothesis is correct. - e.g.: ${mtex `\alpha = 0.05`} means your analysis has a ${mtex `5`}% chance of producing a significant result when the null hypothesis is correct. * Type-I error: Rejecting a true null hypothesis. ${mtex `\alpha`} equals the Type-I error rate. - Interpretation: You can think of this error rate as the *probability of a false positive*. The test results lead you to believe that an effect exists when it actually does not exist. * Type-II error: Failing to reject a false null hypothesis (i.e., when the alternative hypothesis is true). - Interpretation: You can think of this error rate as the *probability of a false negative*. * Power (of a test): it is the probability that we make the right decision when the null is not correct (i.e. we correctly reject it). Power = 1- Prob(Type-II error).

By rejecting in the tails (red regions, each area equals to ${mtex `\alpha`}), we can reject as many types of outcomes as possible while making sure we rarely reject when the null (${mtex`p = 0.5`}) is actually true. `32:[] 2f:{"title":"Introduction to Inference","cells":[{"id":0,"value":"md`# Introduction to Inference\n\nIFT6758, Fall 2020\n\nReading: [ISLR](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf) 3.1.1, 3.1.2 and [Bayesian Basics](https://m-clark.github.io/bayesian-basics) (intro - regression models)\n\nOptional reading: [MSMB](http://web.stanford.edu/class/bios221/book/Chap-Testing.html) 6.1 - 6.6, [Statistics for Hackers](https://speakerdeck.com/jakevdp/statistics-for-hackers?slide=138) and \n[CASI](https://web.stanford.edu/~hastie/CASI_files/PDF/casi.pdf) Chapter 3\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"slide`### Inference\n\n* Meta-Algorithms: Evaluation of processes people use to learn from data\n* Science: How to go from particular to general cases?\n\n\"It is easy to lie with statistics, but hard to tell the truth without them.\" `","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"slide`### Hypothetical Reasoning\n\n* We've mostly been thinking about recognizing patterns occuring in observed data\n* But *hypothetical reasoning* is important. How could the world look intead?\n* Statistical inference provides quantitative machinery to help hypothetical reasoning\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"slide`### Topics to be covered\n\n* Inference for coin clips\n - Both mathematical and computational approaches\n* Inference in linear regression\n* Revisit problems from Bayesian perspective\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"slide`### Is your coin fair?\n\nHypothesis testing gives a formal framework for answering questions like,\n\n* How will you measure discrepancies?\n* How will you tell if it's meaningfully large?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":65,"value":"slide`### Model of Reality\n\n* A null model defines a default simulator of reality\n* Our goal is to determine whether an observed dataset is consistent with that model\n\n

\n\nCounts of cancer deaths. 5 come from null model that there is no spatial correlation.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":477,"value":"slide`### Model of Reality\n\n* A null model defines a default simulator of reality\n* Our goal is to determine whether an observed dataset is consistent with that model\n\n

\n\nDistance vs. angle in three pointers from LA Lakers. 9 are from null model that there is a quadratic relationship.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":467,"value":"slide`### Measuring Discrepancies\n\n* We'll look in the data for potential discrepancies, and try to judge whether they are meaningfully large\n\n

\n\nAre these two treatments quite different? \n\nIf you didn't have the error bars, you might think *yes*. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"slide`### Challenges\n\n* All theories are approximations to reality (null hypothesis)\n* If your approximation isn't that believable, you will reject the null.\n\n* One should be careful *not to make any error in rejecting or accepting such approximation*. \n* Most critical assumption: Independence\n - Thinking you have more evidence than you do is usually much worse than small distributional differences\n\n

\n\nIf you had sampled twice as frequently, will you really have doubled your sample size?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"slide`### Measuring Discrepancies\n\n* The goal is to identify a statistic (any function of the data) that detects problems in your simulation of reality\n* For coin tossing, it makes sense to use\n${mtex_block`\n\\begin{aligned}\n\\hat{p}_n\\left(X_1, \\dots, X_n\\right) &= \\frac{1}{n}\\sum_{i = 1}^{n} X_i \\\\\n&= \\text{fraction heads}\n\\end{aligned}\n`}\nand sound an alarm if ${mtex`\\hat{p}_n`} is very far from 0.5\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":86,"value":"slide`\n\n* What if you saw HHHHHHHHHHTTTTTTTTTT?\n* Seems fair according to our metric....\n* Different choices of test statistic are sensitive to different departures from your theory\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":68,"value":"slide`### Reference Distribution: Theory\n\n* Say you observed 59 heads in 100 flips.\n* How will you tell whether the measured discrepancy is meaningfully large?\n* For coin tossing, supposing a binomial simulator\n\n${mtex_block`\n\\Parg{\\hat{p}_n = \\frac{k}{n}} = {n \\choose k}p^k\\left(1 - p\\right)^{n - k}, \\quad n=100,\\, p=0.5\n`}\n\n

\n\nBlue line: number of heads apearing in 100 flips.\n\nThis image (and others) from [MSMB](https://www.huber.embl.de/msmb/Chap-Testing.html#multiple-testing).\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"slide`### Reference Distribution: Simulation\n\n* Alternatively, we can directly simulate from our model of the world\n* Computing test statistic on simulated data provides a reference distribution\n * Can _always_ be done when there's a generative model\n * Especially useful when no closed form analysis\n\n

\n\nSame reference distribution, from ${mtex`10^4`} simulations\n\nSample with replacement 100 times and repeat ${mtex`10^4`} times\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":84,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":557,"value":"slide`### Confidence Intervals\n\n* For your dataset, what are all the possible null hypotheses that you wouldn't have been able to reject?\n* This is richer information than just the test outcome\n\n

\n\nWe wouln't have rejected this ${mtex`\\theta_0`} because the test statistic lies in the bulk of the reference distribution.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":577,"value":"slide`### Confidence Intervals\n\n* For your dataset, what are all the possible null hypotheses that you wouldn't have been able to reject?\n* This is richer information than just the test outcome\n\n

\n\nWe start rejecting once we get past the red reference distributions. All the ${mtex`\\theta`}'s between these belong to our confidence interval.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":70,"value":"slide`### Recap: Overall Approach\n\n* Determine the effect of interest, and design a suitable test statistic\n* Define a null hypothesis, and come up with a null distribution for that statistic\n* Define the rejection region\n* Do the experiment and draw conclusions\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"slide`## Difference in Means\n\n* Common situation: Have two distributions, and we want to see whether they have the same mean\n* We don't have to assume they are normally distributed\n\n\n

\n\n[An example](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3#scrollTo=vrleqMPDxzGS).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":100,"value":"slide`### Measuring Discrepancy\n\n* Common to use a normalized difference, e.g. \n${mtex_block`\n\\hat{t}_n\\left(X_1, \\dots, X_n\\right) = \\frac{\\bar{x}_1 - \\bar{x}_2}{\\widehat{\\text{s.e.}}\\left(\\bar{x}_1 - \\bar{x}_2\\right)}\n`}\nwhere ${mtex`\\widehat{\\text{s.e.}}`} is an estimate of the standard error (standard deviation of sample mean)\n* If the means in the groups are the same, this is approximately ${mtex`t`}-distributed\n\n**${mtex `t`}-distribution**: similar to the normal distribution with its bell shape but has heavier tails. The shape depends on *degrees of freedom*, ${mtex `df = n-1`}.\n\n\n\nCheck out mathematical expressions.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":491,"value":"slide`### Measuring Discrepancy\n\nThese are the distributions for ${mtex`\\bar{x}_1`} and ${mtex`\\bar{x}_2`}. Even though the original data are relatively far from gaussian, the central limit theorem kicks in.\n\n**Central limit theorem**\n\nIt states that sampling distribution of the sample means approaches a normal distribution as the sample size gets larger — *no matter what the shape of the population distribution*\n\n**Mathematical statement**\n\nLet ${mtex `\\bar X_n`} be the *mean* of ${mtex `n`} random samples, each of size ${mtex `n`} (i.e., ${mtex `\\bar X_n`} is the *sample mean*). The population has mean ${mtex `\\mu`} and finite variance ${mtex `\\sigma^2`}. Then the variable ${mtex `Z = (\\bar X_n - \\mu)/(\\sigma/\\sqrt{n})`} follows *standard normal distribution* as ${mtex `n \\rightarrow \\infty`}.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":494,"value":"slide`### Measuring Discrepancy\n\nThe resulting normalized difference is about ${mtex`t`}-distributed.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"slide`### Evaluating Discrepancy\n\n* You don't have to use theoretical reference distribution\n* Permutation: Randomly reassign group labels and recompute the statistic\n\nThe separate histograms (one per group) become indistinguishable -- it's a simulation from the null for the raw data. Computing difference in means gives a reference distribution.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":510,"value":"slide`### Evaluating Discrepancy\n\n* You don't have to use theoretical reference distribution\n* Simulation: Randomly reassign group labels and recompute the statistic\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":147,"value":"slide` ## Linear Regression\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":150,"value":"slide`### Analogies\n\n* ${mtex`p \\longleftrightarrow `} use ${mtex`\\bar{x}`} in binomial model\n* ${mtex`\\mu_1 - \\mu_2 \\longleftrightarrow`} use ${mtex`{\\hat{t}}`} and central limit theorem\n* ${mtex`\\beta \\longleftrightarrow`} use ${mtex`\\hat{\\beta}`} in linear regression with iid gaussian errors\n\nGenerally: Parameter of interest ${mtex`\\longleftrightarrow`} reference distribution of a statistic under a model\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":166,"value":"slide`### Types of Questions\n\n* Association: Is there are relationship between ${mtex`x`} and ${mtex`y`}?\n* Model comparison: Is a model using additional features actually better?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":221,"value":"slide`### Testing association [one predictor]\n\nDefault to there being no association between ${mtex`x`} and ${mtex`y`}. Formally,\n\n${mtex_block`\n\\begin{aligned}\nH_0: \\beta_1 = 0 \\\\\nH_1: \\beta_1 \\neq 0\n\\end{aligned}\n`}\n\nHow should we measure the discrepancy?\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":190,"value":"slide`### Discrepancy\n\n**Measuring**: Continuous version of the earlier two-groups statistic,\n\n${mtex_block`\n\\begin{aligned}\n\\hat{t} = \\frac{\\hat{\\beta}_1}{\\widehat{\\text{s.e}}\\left(\\hat{\\beta}_1\\right)}\n\\end{aligned}\n`}\n\n**Evaluation**: ${mtex `\\hat t`} has a ${mtex`t`}-distribution if you assume i.i.d. gaussian errors ${mtex`\\epsilon_i \\sim \\mathcal{N}\\left(0, \\sigma^2\\right)`}. We'll see some simulation-based alternatives in later lectures.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":233,"value":"slide`### Comment: Experimental Design\n\nThe formula for the standard error is illuminating,\n\n${mtex_block`\n\\begin{aligned}\n\\widehat{\\text{s.e.}}\\left(\\hat{\\beta}_1\\right) &= \\frac{\\sigma^2}{\\sum \\left(x_i - \\bar{x}\\right)^2}\n\\end{aligned}\n`}\n\n* Makes you want to spread out ${mtex`x_i`}\n * ... but make sure you can still check linearity!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":245,"value":"slide`### Model comparison\n\n* Suppose you want to compare a model that only uses first ${mtex`j`} features with the one that uses all ${mtex`p`}.\n* The relevant test is,\n\n${mtex`\nH_0: \\beta_{j + 1} = \\dots = \\beta_p = 0 \\\\\nH_1: \\beta_{j + 1}, \\dots, \\beta_p \\text{ arbitrary}\n`}\n\n\nFor example,\n* Are the higher-order terms in a polynomial regression actually helping?\n* Accounting for differences in backgrounds, does taking a data science course increase your income?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":259,"value":"slide`### Measuring Discrepancy\n\n* Full model's error, ${mtex_block`RSS = \\sum_i \\left(y_i - x_i^{T}\\hat{\\beta}\\right)^{2}`} \n - Another relevant metric: MSE = RSS/${mtex `\\nu`}, where ${mtex `\\nu`} is degree of freedom of residuals, ${mtex `\\nu = n -`} # fitted parameters)\n - ${mtex `MSE = RSS/(n-p-1)`}\n\n* Submodel's error is, ${mtex_block`RSS_0 = \\sum_i \\left(y_i - x_i^{T}\\hat{\\beta}^0\\right)^{2}`}\nwhere ${mtex`\\hat{\\beta}^0`} is fit assuming the last ${mtex`p - j`} coordinates are 0.\n - ${mtex `MSE_0 = RSS_0/(n-j-1)`}\n\n* Small model always has larger error rate, but is it really _that_ much worse?\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"slide`### Evaluating Discrepancy\n* Compare the error rates between the models,\n${mtex_block`\n\n\\hat{F} = \\frac{(RSS_0 - RSS)/(\\nu \\text{ of difference in residuals})}{MSE} \\\\\n= \\frac{\\frac{1}{p - j}\\left(RSS_0 - RSS\\right)}{\\frac{1}{n - p - 1}RSS}`}\n\n* Under same regression assumptions, statistic follows an ${mtex`F`}-distribution\n - the random variable ${mtex`X`} of the ${mtex`F`}-distribution can be written as\n ${mtex_block `X = \\frac{s^2_1/\\sigma^2_1}{s^2_2/\\sigma^2_2}`}\nwhere ${mtex `s^2_i = RSS_i/d_i`}; ${mtex `RSS_i`} is sum of squares of ${mtex `d_i`} random variables drawn from ${mtex `\\mathcal N(0, \\sigma^2_i)`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":295,"value":"slide`### Nuances\n\n* Regression coefficients represent the mean change in the response variable for one unit of change in the predictor variable while holding other predictors in the model constant.\n - Question: I have a ${mtex`p`}-value for ${mtex`\\beta_j`}. I introduce a new variable to my regression. How does the ${mtex`p`}-value for the original variable change?\n\n* *${mtex `t`}-test* can assess only one regression coefficient at a time. \n - Use this to pinpoint *which* of the ${mtex`\\beta_j`}'s are not significant in the predictor-response relationship.\n\n* *${mtex `F`}-test* can assess multiple coefficients simultaneously.\n - Use this to assess the significance of all coefficients of the overall model. \n - If you don't have any significant ${mtex `p`}-values for the individual coefficients in your model, the overall ${mtex `F`}-test won't be significant either.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":304,"value":"slide`## Bayesian inference\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":566,"value":"slide`### Main Idea\n\n* Instead of accepting or rejecting states of the world ${mtex`H`}, place probabilities over them\n* Instead of arguing whether coin is fair, define a distribution over plausible probabilities\n`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":308,"value":"slide`### Posterior Updating\n\n* Key logical device is Bayes rule,\n${mtex_block`\np\\left(H \\vert \\text{data}\\right) \\propto p\\left(\\text{data} \\vert H\\right)p\\left(H\\right)`}\n* New belief about world = new data ${mtex`\\times`} old belief\n* \"Update the prior to the posterior\"\n* Stronger priors are harder to overcome\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"slide` ### Does Bayesian approach make sense?\n\n\n...\n\n

\n\nFrom Leo Breiman in [Nail finders, edifices, and Oz](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=2ahUKEwiZkoW66u3kAhVSSN8KHeS1BI4QFjABegQIABAC&url=http%3A%2F%2Fdigitalassets.lib.berkeley.edu%2Fsdtr%2Fucb%2Ftext%2F32.pdf&usg=AOvVaw1w1bS2VMuELCWdCoYZyntz)\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":321,"value":"slide`### Revisiting Coin Flips\n\n* Prior assigns probabilities to ${mtex`p \\in \\left[0, 1\\right]`}\n* We leave the update to a black box inference engine\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":351,"value":"slide`### Revisiting Coin Flips\n\n* Go to example [here](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3#scrollTo=IAfN5aP1-s2h)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":323,"value":"slide`### Revisiting Linear Regression\n\n* Almost exact same mechanics translates to linear regression\n* Go through example [here](https://colab.research.google.com/drive/1vdHEKqxmQE8tvQbDlGmDMhDVf4IbKPI3)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":325,"value":"slide`### Posterior Predictive Checks\n\n* Once you've fitted a model, try generating data from it\n* Do the simulated data look like your real data?\n* Do they match specific properties of your real data?\n * (means, quantiles, correlations, ...)\n* This idea is useful in frequentist settings too\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"slide`### Practical considerations\n\n* Modular, flexible ways of building models and drawing inference\n * E.g. [hierarchical modeling](https://mc-stan.org/users/documentation/case-studies/radon.html)\n* Prior choice is arbitrary, but you can measure sensitivity of inferences to that choice\n* Freedom in design can outweigh complexity of computation\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":46,"value":"import {coin_flipping} from \"@krisrs1128/sufficiency-illustrated\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":196,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":17,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}