31:[] 2f:{"title":"The Bootstrap","cells":[{"id":0,"value":"md`# The Bootstrap\n\nIFT6758, Fall 2020\n\nReading: [ISLR](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf) 5.2 - 5.3\n\nOptional reading: [CASI](https://web.stanford.edu/~hastie/CASI_files/PDF/casi.pdf) Chapter 1\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":23,"value":"slide`### Computation for Inference\n\n* Increased computation has revolutionized our ability to fit functions to data\n* It has also profoundly influenced inference\n* The bootstrap is one important reflection of this\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"slide`### Algorithms vs. Inference\n\n* Algorithms help you construct useful reductions of data (for knowledge or decision making).\n* Inference helps you gauge how reliable your reductions are\n\n\"It is a surprising, and crucial, aspect of statistical theory that the same data that supplies an estimate can also be used to assess it's accuracy.\"\n\nQ: Given the data, can we have an idea about how 'confident' we are about a statistic?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":44,"value":"slide`### Example: Sample Mean\n\nThere is a closed-form formula for the standard error of the sample mean. Suppose ${mtex`x_i \\stackrel{i.i.d.}{\\sim} F`}, a distribution with variance ${mtex`\\sigma^2`}. Then,\n\n${mtex_block`\n\\begin{aligned}\n\\Vararg{\\bar{x}} \n&= \\Vararg{\\frac{1}{n}\\sum_{i = 1}^{n}x_i} \\\\\n&= \\frac{1}{n^2}\\sum_{i = 1}^{n}\\Vararg{x_i} \\\\\n&= \\frac{1}{n^2}n\\sigma^{2} \\\\\n&= \\frac{\\sigma^{2}}{n}\n\n\\end{aligned}`}\n`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":67,"value":"slide`### Estimated Standard Error\n\nTherefore, if we had a way of estimating ${mtex`\\sigma^2`}, then we can estimate the standard error of the mean by plugging this estimate in.\n\n${mtex_block`\n\\Var\\left[\\bar{x}\\right] \\approx \\frac{\\hat{\\sigma}^2}{n}\n`}\n\nThis would give us a way of understanding to what degree we can trust our estimate of the mean, computed entirely from the raw data.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"slide`### Estimated Standard Error\n\nOf course, if ${mtex`x_i \\stackrel{i.i.d.}{\\sim} F`}, then a reasonable estimate for ${mtex`\\sigma^2`} is\n\n${mtex_block`\n\\hat{\\sigma}^2 := \\frac{1}{n} \\sum_{i = 1}^{n} \\left(x_i - \\bar{x}\\right)^2\n`}\n\nSo in summary, to estimate the variance of the mean, we\n1. Compute an estimate of ${mtex`\\sigma^2`}\n2. Plug that the true expression for ${mtex`\\Vararg{\\bar{x}}`}\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":84,"value":"slide`### Abstraction: estimate the variance of the statistic from the data\n\nThis is a more abstract way of describing this process.\n\n1. Define a statistic ${mtex`\\hat{\\theta}\\left(x_1, \\dots, x_n\\right)`} of the data\n2. Do math to get an expression for ${mtex`\\Vararg{\\hat{\\theta}}`}\n3. Replace unknowns in the expression with estimates, and argue that the result is ${mtex`\\approx \\Vararg{\\hat{\\theta}}`}. \n * If you want to be precise, you could call this new estimator ${mtex`\\widehat{\\Var}\\left[\\hat{\\theta}\\right]`}\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":90,"value":"slide` ### Goals\n\nIn a lot of cases, step (2) is intractable.\n\n* ${mtex`\\hat{\\theta}`} is a more complex function of the ${mtex`x_i`}\n - Ratio between eigenvalues in your PCA\n - Derived statistics, like ${mtex`\\log\\mu`} in some model\n* You ran some iterative algorithm to compute ${mtex`\\hat{\\theta}`}\n - Robust regression (e.g. [Least Trimmed Squares](https://en.wikipedia.org/wiki/Least_trimmed_squares))\n - Random forests\n\n**We'd like a recipe that works even then, and which doesn't have to be rederived for every single problem.**\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":129,"value":"slide`### Example: Portfolio Optimization\n\n* Two assets that you can invest in, ${mtex`X`} and ${mtex`Y`}.\n* Distribute ${mtex`\\alpha`} fraction of funds to ${mtex`X`}, and the rest to ${mtex`Y`}, i.e., invest\n${mtex_block`\\alpha X + \\left(1 - \\alpha\\right)Y`}\n* The best strategy (in the sense of minimizing variance) can be shown to be\n${mtex_block`\n\\begin{aligned}\n\\alpha &= \\frac{\\sigma^2_Y - \\sigma_{XY}}{\\sigma^2_X + \\sigma^2_Y - 2\\sigma_{XY}}\n\\end{aligned}\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":138,"value":"slide`### Example: Portfolio Optimization\n\n* ${mtex`\\alpha`} is unknown in practice, so we estimate it,\n${mtex_block`\n\\hat{\\alpha}\\left(x_1, \\dots, x_n, y_1, \\dots, y_n\\right) = \\frac{\\hat{\\sigma}^2_Y - \\hat{\\sigma}_{XY}}{\\hat{\\sigma}^2_X + \\hat{\\sigma}^2_Y - 2\\hat{\\sigma}_{XY}}\n`}\n* Now we want to know, how variable is this estimator?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":143,"value":"slide`### A Thought Experiment\n\n* Suppose you had a window into parallel universes\n* How much does the estimator change across different samples?\n* ${mtex_block `\\sigma^2_X = 1,\\, \\sigma^2_Y = 1.25, \\, \\sigma_{XY} = 0.5;\\, \\alpha = 0.6`}\n\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":147,"value":"slide`### Thought Experiment\n\nIf you simulate 1000 datasets in this way, you can get a different ${mtex`\\hat{\\alpha}_r`}, for ${mtex`r = 1, 2, \\dots, 1000`}. \n\n${mtex_block`\n\\bar{\\alpha} = \\frac{1}{1000}\\sum_{r = 1}^{1000} \\hat{\\alpha}_r = 0.5996\n`}\nwhich is very close to the underlying 0.6.\n\nTo get a sense of the variability across datasets, we can use\n${mtex_block`\n\\widehat{\\Var}\\left[\\hat{\\alpha}\\right] = \\frac{1}{999}\\sum_{r = 1}^{1000}\\left(\\hat{\\alpha}_r - \\bar{\\alpha}\\right)^2 \\approx 0.083^2,\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":163,"value":"slide`### Bootstrap Idea\n\n* **Key idea**: The population is to the sample as the sample is to the bootstrap samples.\n\n* In our simulation, we were able to sample as many new datasets ${mtex`F`} as we wanted. In reality, we see only one.\n* But we can simulate as many datasets from the empirical distribution ${mtex`\\hat{F}`} as we want\n* It turns out that if you use ${mtex`\\hat{F}`} in place of ${mtex`F`}, the approach from the thought experiment *still works*\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"slide`### Sampling from ${mtex`F`}\n\nIn our ideal simulation world, we're able to generate many datasets from ${mtex`F`} and see how our estimator ${mtex`\\hat{\\theta}`} changes.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":336,"value":"slide`### Sampling from ${mtex`F`}\n\nIn our ideal simulation world, we're able to generate many datasets from ${mtex`F`} and see how our estimator ${mtex`\\hat{\\theta}`} changes.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":338,"value":"slide`### Sampling from ${mtex`F`}\n\n\nThe properties of the final sampling distribution for ${mtex`\\hat{\\theta}`} can be used to guide inference.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":346,"value":"slide`### Sampling from ${mtex`\\hat{F}`} \n\nIn reality, we can't just generate new datasets. We *can* draw samples from the empirical distribution, however.\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":355,"value":"slide`### Sampling from ${mtex`\\hat{F}`} \n\nIn reality, we can't just generate new datasets. We *can* draw samples from the empirical distribution, however.\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":357,"value":"slide`### Sampling from ${mtex`\\hat{F}`} \n\nIn reality, we can't just generate new datasets. We *can* draw samples from the empirical distribution, however.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":215,"value":"slide`### Bootstrap vs. Simulation Estimates\n\n* The left are estimates ${mtex`\\hat{\\alpha}_r`} when you simulate from ${mtex`F`} (impossible in practice), while the right are when you simulate from ${mtex`\\hat{F}`} (possible in practice).\n* The estimated variances are very similar\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"slide`### The Bootstrap\n\n* Input: A statistic ${mtex`\\hat{\\theta}`}, number of desired simulations ${mtex`B`}\n* For ${mtex`b = 1, \\dots, B`},\n * Simulate ${mtex`x_1^{b}, \\dots, x_n^b \\stackrel{i.i.d.}{\\sim} \\hat{F}`}\n * Compute ${mtex`\\hat{\\theta}^b := \\hat{\\theta}\\left(x_1^b, \\dots, x_n^b\\right)`}\n* Estimate the variance of the original ${mtex`\\hat{\\theta}`} by looking at the variance in the simulation output,\n${mtex_block`\n\\Var\\left[\\hat{\\theta}\\left(x_1, \\dots, x_n\\right)\\right] \\approx \\frac{1}{B - 1}\\sum_{b = 1}^{B}\\left(\\hat{\\theta}^b - \\bar{\\theta}\\right)^2,\n`}\nwhere ${mtex`\\bar{\\theta}`} is the average of all the ${mtex`\\hat{\\theta}^b`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":226,"value":"slide`### Plug-in Principle\n\nThe original estimator is constructed according to\n${mtex_block`\nF \\xrightarrow{sample} x_{1}, \\dots, x_{n} \\xrightarrow{estimate} \\hat{\\theta}\\left(x_1, \\dots, x_n\\right)\n`}\nThis is only done once, so you can't estimate the standard error from it alone.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"slide`### Plug-In Principle\n\nIf we plug in ${mtex`\\hat{F}`} for ${mtex`F`}, we get\n${mtex_block`\n\\hat{F} \\xrightarrow{resample} x_{1}^b, \\dots, x_{n}^b \\xrightarrow{estimate} \\hat{\\theta}\\left(x_1^b, \\dots, x_n^b\\right) := \\hat{\\theta}^b\n`}\n\nThis can be done as much as we want, so we can estimate the variance across ${mtex`\\hat{\\theta}^b`}. \n\nThe more is the number of samples, the better the plug-in principle works.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":259,"value":"slide`### The Resampling Perspective\n\n* Draws from ${mtex`\\hat{F}`}: up or downweight original samples.\n * Some ${mtex`x_i`} may not be included, other might be included multiple times\n * View this as points on the simplex (space of weights that sum to ${mtex`n`})\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":276,"value":"slide`### The Resampling Perspective\n\n* The bootstrap measures the sensitivity of ${mtex`\\hat{\\theta}`} to different weightings of the original points\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"slide`### Bootstrap Confidence Intervals \n\n* Optional read: [Bootstrap CI](http://users.stat.umn.edu/~helwig/notes/bootci-Notes.pdf)\n\n* A lot of times, we use estimates of the variance to build confidence intervals\n\n* Larger sample size leads to tighter confidence intervals\n* If ${mtex`\\hat{\\theta}`} is approximately normal, it can be shown that\n\n ${mtex_block`\n\\hat{\\theta} \\pm 1.96 \\sqrt{\\Var\\left(\\hat{\\theta}\\right)}\n`}\n is a valid confidence interval\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":289,"value":"slide`### Approach 1\n\nTherefore, if we can use the bootstrap to estimate the variance, we can directly use it to make a confidence interval\n\n\n ${mtex_block`\n\\hat{\\theta} \\pm 1.96 \\sqrt{\\widehat{\\Var}\\left(\\hat{\\theta}\\right)}\n`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":296,"value":"slide`### Approach 2\n\n* What if ${mtex`\\hat{\\theta}`} isn't approximately normal?\n* In classical stats, you'd need new theory to find an alternative confidence interval\n* However, the bootstrap gives us access to something close to the distribution of ${mtex`\\hat{\\theta}`}.\n - Get more samples and apply (bootsprap) central limit theorem.\n - Use [Bias corrected and accelerated bootstrap](https://blogs.sas.com/content/iml/2017/07/12/bootstrap-bca-interval.html#:~:text=The%20main%20advantage%20to%20the,less%20than%20the%20observed%20statistic.) method\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":308,"value":"slide`### Approach 2\n\n* Main idea is to directly use quantiles of the simulated ${mtex`\\hat{\\theta}^b`}\n* No longer requires normality (or even symmetry)\n* However, requires more simulation samples, since quantiles are harder to estimate than variances\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":376,"value":"slide`### Approach 2\n\n* Main idea is to directly use quantiles of the simulated ${mtex`\\hat{\\theta}^b`}\n* No longer requires normality (or even symmetry)\n* However, requires more simulation samples, since quantiles are harder to estimate than variances\n\n

\n\n* Python (e.g., to get 95% confidence interval): *numpy.percentile(bootstrap_dist, 2.5)*\n*numpy.percentile(bootstrap_dist, 97.5)*\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":384,"value":"slide`### Additional Examples: Difference in Means\n\n* Suppose we want to test the difference in means between two groups.\n* We can define a reference distribution using the bootstrap\n* Idea is to sample repeatedly from ${mtex`\\hat{F}_1`} and ${mtex`\\hat{F}_2`}, and look at the distribution in the difference in means\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":396,"value":"slide`### Additional Examples: Difference in Means\n\n* Suppose we want to test the difference in means between two groups.\n* We can define a reference distribution using the bootstrap\n* Idea is to sample repeatedly from ${mtex`\\hat{F}_1`} and ${mtex`\\hat{F}_2`}, and look at the distribution in the difference in means\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":386,"value":"slide`### Additional Examples: Regression\n\n* You can bootstrap regression models as well\n* Fit the regression many times, across resampled versions of the data, evaluate confidence intervals\n* Works for variants of regression with no analytical s.e. formula\n\n

\n\nRef: [Bootstrapping Regression Models](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=2ahUKEwi8oJKQjPrkAhXNVN8KHUf1DaUQFjAAegQIABAC&url=http%3A%2F%2Fstatweb.stanford.edu%2F~owen%2Fcourses%2F305-1314%2FFoxOnBootingRegInR.pdf&usg=AOvVaw0Rn0cjZPcEqxnTE8LdGs1U)\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":388,"value":"slide`### Additional Examples: Regression\n\nIt can even be applied to nonparametric regression models, e.g., lowess regressions,\n\n

\n\nand even [Random Forests](https://arxiv.org/abs/1311.4555).\n\n* Fit models using bootstrap samples and evaluate the confidence intervals on the test statistic\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":11,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":14,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}