31:[] 2f:{"title":"Cross Validation and Model Selection","cells":[{"id":0,"value":"md`\n# Cross Validation and Model Selection\n\nIFT6758, Fall 2019\n\nReading: [ISLR](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf) section 5.1 and [PDS](https://tanthiamhuat.files.wordpress.com/2018/04/pythondatasciencehandbook.pdf) pg. 359 - 375`","pinned":false,"hidden":false,"mode":"ojs"},{"id":31,"value":"slide`### Daily Choices for Data Scientists\n\nKnowing how to fit models is not enough, if you want to solve a real-world problem.\n\n* How should you select between model families?\n* Which parameters are best within a model family?\n* Should you be trying to improve the data?\n - More samples? Richer features?\n - Less missingness, fewer outliers, ...\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":42,"value":"slide`### Transitioning to Inference\n\n* We'll be more introspective, trying to understand properties of our algorithms\n* The heart of inference: Being critical of the processes people use to learn from data\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"slide`### Reminder: Bias-Variance Tradeoff\n\n* Ultimately, you want your model to perform well on out-of-sample data\n* If you only evaluate on in-sample data, you will underestimate the out-of-sample error\n\n

\n`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"slide`### Validation Sets\n\n* To approximate the out-of-sample error, we can use a validation set.\n* Randomly divide your sample into two pieces, one to train and another to validate\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"slide`### Validation Sets\n\nIf you run this over models with different degrees of complexity, you can see the bias-variance tradeoff.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":234,"value":"slide`### Validation Sets\n\nIf you run this over models with different degrees of complexity, you can see the bias-variance tradeoff.\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":53,"value":"slide`### Complexity Regimes\n\nEven if you only evaluate the train / validation error for a model of a given complexity, you get useful information.\n\n* Training ${mtex`\\ll`} validation error ${mtex`\\rightarrow`} Model is overfit\n* Training ${mtex`\\approx`} validation error ${mtex`\\rightarrow`} Model is underfit (or OK)\n\nCommon heuristic: Overfit the data first, then regularize.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":63,"value":"slide`### Learning Curves\n\n* As you gather more data, how much better do your models get?\n* This can guide the decision to collect more data.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":79,"value":"slide`### Learning Curves\n\n* Models of different complexities have different learning curves\n* Larger models don't saturate as quickly. They are,\n - worse than small models on small datasets\n - better than small models on large datasets\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":82,"value":"slide`### Evaluation and Randomness\n\n* We are only _estimating_ out-of-sample error\n* These estimates might be good or bad\n - Have randomness from choice of validation set\n - Have randomness from dataset collection\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":246,"value":"slide`### Evaluation and Randomness\n\n* We are only _estimating_ out-of-sample error\n* These estimates might be good or bad\n - Have randomness from choice of validation set\n - Have randomness from dataset collection\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":90,"value":"slide`### Bias and Variance in Validation Error\n\n* Variance: Different validation sets give different estimates\n* Bias: Training on subset leads to worse expected performance (remember learning curves)\n* Bias: You might overfit to the test set\n\nThere are a few alternatives to validation sets. We'll talk about,\n* Leave-One-Out Cross Validation [LOOCV]\n* K-Fold Cross Validation.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":100,"value":"slide`### Alternatives: LOOCV\n\n1. Fit your model without sample ${mtex`\\left(x_i, y_i\\right)`}. Call the fit ${mtex`\\hat{f}_{-i}`}.\n2. Compute holdout ${mtex`\\widehat{MSE}_{i} := \\left(y_{i} - \\hat{f}_{-i}\\left(x_i\\right)\\right)^{2}`}\n3. Estimate the out-of-sample error by averaging this over all possible holdouts coming from (1) and (2),\n${mtex_block`\n\\widehat{MSE} = \\frac{1}{n}\\sum_{i = 1}^{n} \\widehat{MSE}_{i}\n`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":252,"value":"slide`### Alternatives: LOOCV\n\n1. Fit your model without sample ${mtex`\\left(x_i, y_i\\right)`}. Call the fit ${mtex`\\hat{f}_{-i}`}.\n2. Compute holdout ${mtex`\\widehat{MSE}_{i} := \\left(y_{i} - \\hat{f}_{-i}\\left(x_i\\right)\\right)^{2}`}\n3. Estimate the out-of-sample error by averaging this over all possible holdouts coming from (1) and (2),\n${mtex_block`\n\\widehat{MSE} = \\frac{1}{n}\\sum_{i = 1}^{n} \\widehat{MSE}_{i}\n`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":104,"value":"slide`### LOOCV\n\nAdvantages\n * Lower bias. We use almost all the training data, so we don't underestimate performance.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":257,"value":"slide`### LOOCV\n\nDisdvantages\n * High computational complexity (except linear regression)\n * The trained models are correlated \n - The ${mtex`\\widehat{MSE}_i`} are correlated\n - The average of correlated variables has larger variance than the average of independent ones\n - The out-of-sample estimate has higher variance\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":101,"value":"slide`### Alternatives: K-Fold CV\n\n\n1. Randomly partition samples into one of ${mtex`K`} folds, ${mtex`\\{S_{1}, \\dots, S_{K}\\}`}.\n1. Fit your model without fold ${mtex`S_{k}`}. Call the fit ${mtex`\\hat{f}_{-k}`}.\n2. Compute holdout ${mtex`\\widehat{MSE}_{k} := \\sum_{i \\in S_{k}} \\left(y_{i} - \\hat{f}_{-k}\\left(x_i\\right)\\right)^{2}`}\n3. Estimate the out-of-sample error by averaging over folds,\n${mtex_block`\n\\widehat{MSE} = \\frac{1}{K}\\sum_{k = 1}^{K} \\widehat{MSE}_{k}\n`}\n\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":151,"value":"slide`### K-Fold CV\n\nAdvantages\n* More computationally tractable\n* Learns less correlated models\n - The estimates ${mtex`\\widehat{MSE}_k`} are less correlated\n - The estimate ${mtex`\\widehat{MSE}`} has lower variance\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":263,"value":"slide`### K-Fold CV\n\nDisadvantages\n* We don't train using the full training set\n* We bias our estimates upwards \n - Model on full dataset is actually better than estimated\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":276,"value":"slide`### Estimation Quality: LOOCV and K-Fold\n\n* The blue curves are known out-of-sample MSE's from a simulation experiment\n* Black and orange are LOOCV and K-Fold estimates, respectively\n* Note: Even when estimates of out-of-sample MSE is poor, the estimate of the minimum might be good\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":174,"value":"slide`### Hyperparameter Search\n\n* We will often have many parameters to tune simultaneously\n - Model parameters: Polynomial degree, # trees, ...\n - Training parameters: Learning rate, subsampling, ...\n - Preprocessing: Normalization, outlier removal, ...\n* No single \"model complexity\" parameter\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":187,"value":"slide`### Search Options\n\n* Manual search\n* Grid search\n* Random search\n* Combinations of these\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":192,"value":"slide`### Manual Search\n\n* Relate all the parameters to overall model complexity\n * e.g., more iterations ${mtex`\\rightarrow`} higher complexity\n* Guide your choice of parameters by which regime (over vs. underfitting) you are in\n* Advantage: Uses bias-variance tradeoff information\n* Disadvantage: Tedious, not fully reproducible\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"slide`### Grid Search\n\n* Compute out-of-sample error on all combinations of parameters\n* Advantage: Automatic, easy to implement\n* Disadvantage: Exponentially many parameter configurations\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":203,"value":"slide`### Random Search\n\n* Compute out-of-sample error on random samples of parameters\n* Advantage: Automatic, easy to implement. Relevant parameters become clear quickly.\n* Disadvantage: Still suffers when very many parameters.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":217,"value":"slide` ### Combinations\n\n* Can fix a few parameters manually, and use random search for others\n* Can use [\"multiscale\"](https://arxiv.org/abs/1206.5533) search. Automatically search over predefined grids, but manually set the grids to more promising regions.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":17,"value":"import {mtex} from \"@krisrs1128/function-fitting\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":19,"value":"import {mtex_block} from \"@krisrs1128/function-fitting\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}