31:T4b0,slide = { function slide() { const container = document.createElement("div"); container.className = "slide"; container.appendChild(md.apply(this, arguments)); return container; } function code(strings) { const container = document.createElement("div"); const pre = container.appendChild(document.createElement("pre")); const code = pre.appendChild(document.createElement("code")); let string = strings[0] + "", i = 0, n = arguments.length; while (++i < n) string += arguments[i] + "" + strings[i]; code.textContent = string.trim(); container.className = "slide slide--code"; return container; } slide.code = code; slide.js = function() { const container = code.apply(this, arguments); const content = container.firstChild.firstChild; content.className = "js hljs javascript"; hl.highlightBlock(content); return container; }; slide.img = function(strings) { const img = new Image; let string = strings[0] + "", i = 0, n = arguments.length; while (++i < n) string += arguments[i] + "" + strings[i]; img.src = string.trim(); img.className = "slide slide--img"; return img; }; return slide; }32:[] 2f:{"title":"Function Fitting","cells":[{"id":0,"value":"md`# Function Fitting\n\nIFT6758, Fall 2019\n\nReading: [ISLR](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf) sections 2.1, 3.2.1, 3.5\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"slide`\n### An Abstraction: Input → Output\n\nAcross applications we often want to model an output variable as a function of many inputs,\n* Genetic profile → Chance of developing disease\n* Person’s characteristics → Whether they’ll vote\n* Marketing plan → Total sales amount\n* Image pixel values → What’s in the image\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"slide`\n### An Abstraction: Input → Output\n\nAcross applications we often want to model an output variable as a function of many inputs,\n* Genetic profile → Chance of developing disease\n* Person’s characteristics → Whether they’ll vote\n* Marketing plan → Total sales amount\n* Image pixel values → What’s in the image\n\n${mtex_block`\n\\begin{aligned}\nx_i &= \\left(x_{i1}, \\dots, x_{ip}\\right) \\leftarrow \\text{all the inputs}\\\\\ny_i &= f\\left(x_i\\right) \\leftarrow\\text{inputs to output relationship}\n\\end{aligned}`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":56,"value":"slide`### A Picture\n\nWe allow for ${mtex`\\color{824848}y_i`} to be a different from ${mtex`\\color{113f91}f`}, perturbed by some observation-specific noise ${mtex`\\color{356583}\\eps_i`},\n\n${mtex_block`\n\\begin{aligned}\n{\\color{824848}{y_i}} = {\\color{113f91}f}\\left({\\color{237f9e}x_i}\\right) + {\\color{356583}\\eps_i}\n\\end{aligned}`\n}\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":109,"value":"slide`### Why this decomposition?\n\n* ${mtex`\\color{113f91}f`} describes systematic variation in ${mtex`\\color{824848}y_i`}, as a function of observed inputs. \n* ${mtex`\\color{356583}\\eps_i`} reflects variation whose source is unknown to us. Consider [coin tosses](https://www.npr.org/templates/story/story.php?storyId=1697475).\n\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":94,"value":"slide`### Why are we doing this?\n\n* Why not just visualize the data we have?\n* Reason 1: **Prediction**\n * We may want the ${mtex`\\color{824848}y_i`} corresponding to an input ${mtex`\\color{237f9e}x_i`}\n * Inputs may be much easier to collect than outputs\n* Reason 2: **Inference**\n * We may care about the form of ${mtex`\\color{113f91}f`}, for personal understanding\n * e.g., is a particular input relevant at all?\n* We want quantitative estimates, not visual summaries\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"slide`### Estimates and Predictions\n\n* In reality, we won’t know ${mtex`\\color{113f91}f`}\n* We estimate it from data and call the result ${mtex`\\color{f7b42c}\\hat{f}`}\n* To predict ${mtex`\\color{824848}y_i`} for some input ${mtex`\\color{237f9e}x_i`}, we'd use ${mtex`{\\color{f7b42c}\\hat{y}_i} = {\\color{f7b42c}\\hat{f}}\\left({\\color{237f9e}{x_i}}\\right)`}\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":127,"value":"slide`### Sources of Error\n\nThis process introduces two sources of error,\n\n* Approximation error: ${mtex`\\hat{f}`} isn't close to ${mtex`f`}\n * This error is *reducible* (use a better algorithm)\n* Irreducible error: ${mtex`y_i`} isn't close to ${mtex`f\\left(x_i\\right)`}\n * Incur this error even if ${mtex`f`} were perfectly known\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":178,"value":"slide`### Sources of Error\n\nThis process introduces two sources of error,\n\n* **Approximation error: ${mtex`\\hat{f}`} isn't close to ${mtex`f`}**\n * This error is *reducible* (use a better algorithm)\n* Irreducible error: ${mtex`y_i`} isn't close to ${mtex`f\\left(x_i\\right)`}\n * Incur this error even if ${mtex`f`} were perfectly known\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":181,"value":"slide`### Sources of Error\n\nThis process introduces two sources of error,\n\n* Approximation error: ${mtex`\\hat{f}`} isn't close to ${mtex`f`}\n * This error is *reducible* (use a better algorithm)\n* **Irreducible error: ${mtex`y_i`} isn't close to ${mtex`f\\left(x_i\\right)`}**\n * Incur this error even if ${mtex`f`} were perfectly known\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":189,"value":"slide` ### Extending the Picture\n\n* The abstraction is much more general\n* It applies to high-dimensional ${mtex`x_i`} and general ${mtex`y_i`}\n\n

\n\nHere, ${mtex`x_i`} are two dimensional.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":217,"value":"slide` ### Extending the Picture\n\n* The abstraction is much more general\n* It applies to high-dimensional ${mtex`x_i`} and general ${mtex`y_i`}\n\n

\n\nHere, the response ${mtex`y_i`} is binary (blue or orange).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":195,"value":"slide`### How to find ${mtex`\\hat{f}`}?\n\nGenerally have two steps,\n* Propose a model family ${mtex`\\F`}\n * E.g., set of all linear functions of ${mtex`x_i`}\n* Define a procedure to choose ${mtex`\\hat{f} \\in \\F`} based on the data\n * E.g., the choice ${mtex`\\hat{f}`} that minimizes ${mtex`\\sum_{i} \\left(y_i - \\hat{f}\\left(x_i\\right)\\right)^{2}`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":250,"value":"slide`### How to find ${mtex`\\hat{f}`}?\n\nGenerally have two steps,\n* Propose a model family ${mtex`\\F`}\n * E.g., set of all linear functions of ${mtex`x_i`}\n* Define a procedure to choose ${mtex`\\hat{f} \\in \\F`} based on the data\n * E.g., the choice ${mtex`\\hat{f}`} that minimizes ${mtex`\\sum_{i} \\left(y_i - \\hat{f}\\left(x_i\\right)\\right)^{2}`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":262,"value":"slide`### How to find ${mtex`\\hat{f}`}?\n\nGenerally have two steps,\n* Propose a model family ${mtex`\\F`}\n * E.g., set of all linear functions of ${mtex`x_i`}\n* Define a procedure to choose ${mtex`\\hat{f} \\in \\F`} based on the data\n * E.g., the choice ${mtex`\\hat{f}`} that minimizes ${mtex`\\sum_{i} \\left(y_i - \\hat{f}\\left(x_i\\right)\\right)^{2}`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":266,"value":"slide`### How to find ${mtex`\\hat{f}`}?\n\nGenerally have two steps,\n* Propose a model family ${mtex`\\F`}\n * E.g., set of all linear functions of ${mtex`x_i`}\n* Define a procedure to choose ${mtex`\\hat{f} \\in \\F`} based on the data\n * E.g., the choice ${mtex`\\hat{f}`} that minimizes ${mtex`\\sum_{i} \\left(y_i - \\hat{f}\\left(x_i\\right)\\right)^{2}`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":251,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n\n* There is a bias-variance tradeoff\n* Finding the best function in a large class ${mtex`\\F`} can be hard\n * High variance: Different samples ${mtex`x_i, y_i`} might result in very different ${mtex`\\hat{f}`}, even when ${mtex`f`} hasn't changed\n * This variance gets worse the higher-dimensional your ${mtex`x_i`} are (the \"curse of dimensionality\")\n * Incurring some bias, for the sake of better stability, can lead to overall better predictions\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":416,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n* Ultimately, you want your model to perform well on out-of-sample data\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":274,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n\n* There is a bias-variance tradeoff\n* Finding the best function in a large class ${mtex`\\F`} can be hard\n * High variance: Different samples ${mtex`x_i, y_i`} might result in very different ${mtex`\\hat{f}`}, even when ${mtex`f`} hasn't changed\n * Incurring some bias, for the sake of better stability, can lead to overall better predictions\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":276,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n\n* There is a bias-variance tradeoff\n* Finding the best function in a large class ${mtex`\\F`} can be hard\n * High variance: Different samples ${mtex`x_i, y_i`} might result in very different ${mtex`\\hat{f}`}, even when ${mtex`f`} hasn't changed\n * Incurring some bias, for the sake of better stability, can lead to overall better predictions\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n\n* There is a bias-variance tradeoff\n* Finding the best function in a large class ${mtex`\\F`} can be hard\n * High variance: Different samples ${mtex`x_i, y_i`} might result in very different ${mtex`\\hat{f}`}, even when ${mtex`f`} hasn't changed\n * Incurring some bias, for the sake of better stability, can lead to overall better predictions\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":285,"value":"slide`### Why ever use smaller ${mtex`\\F`}'s?\n\n* There is a bias-variance tradeoff\n* Finding the best function in a large class ${mtex`\\F`} can be hard\n * High variance: Different samples ${mtex`x_i, y_i`} might result in very different ${mtex`\\hat{f}`}, even when ${mtex`f`} hasn't changed\n * Incurring some bias, for the sake of better stability, can lead to overall better predictions\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":401,"value":"slide`### Discussion Questions\n* What are the advantages of a more vs. a less flexible regression model? When would you prefer one vs. the other?\n * How does your answer depend on the input dimension of the ${mtex`x_i`}?\n * How does your answer depend on the sample size?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":344,"value":"slide`### Model Flexibility -- Takeaways\n\nThis tradeoff can be summarized with a few takeaways,\n* If you don't have too many samples, you should prefer a simpler model\n* If you have many samples, you can afford a more complex model\n* We'll need some sort of mechanism to tell which regime we're in\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":224,"value":"slide`### Examples of Model Families\n\n* Before discussing specific algorithms in detail, let's get a feel for how different model families look like\n* We'll dive into their details in the next few lectures\n* We're using the \\`Advertising\\` dataset (how does advertising affect sales?)\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":300,"value":"slide`### Examples of Model Families\n\n* Before discussing specific algorithms in detail, let's get a feel for how different model families look like\n* We'll dive into their details in the next few lectures\n* We're using the \\`Advertising\\` dataset (how does advertising affect sales?)\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":229,"value":"slide`### Linear Regression\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"slide`### K-Nearest Neighbors\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":331,"value":"slide`### Linear Regression with Interaction\n\n

\n\nNotice that each line has a different slope.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":242,"value":"slide`### Trees\n\n

\n\nThis vaguely resembles the KNN fit.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":351,"value":"slide`### Linear Regression with Nonlinear Basis\n\n

\n\nA *linear* combination of *nonlinear* functions will be *nonlinear*.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"slide`### Generalized Additive Modes\n\n

\n\nIt looks quite similar to the interaction model, but is a little more wiggly.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":409,"value":"slide`### Discussion\nWhich of the function families above would you guess is parametric? Which would you guess is nonparametric?\n\n(I don't expect you to know the answer yet, but make an educated guess based on the pictures)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":246,"value":"slide`### Post-training Analysis\n* There's a certain set of checks you should always do after you fit a model, no matter what family it is\n * Yes, even deep learning models\n* You can do better than looking at the validation loss\n* Residual analysis, error modeling, outliers, high-leverage\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":369,"value":"slide`### Residual Analysis\n\n* Make a histogram of errors ${mtex`e_i = y_i - \\hat{y}_i`}\n* Plot them against a few input variables\n* If you notice systematic variation in them, this is information you can squeeze into ${mtex`f`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":380,"value":"slide`### Residual Analysis\n\n* Make a histogram of errors ${mtex`e_i = y_i - \\hat{y}_i`}\n* Plot them against a few input variables\n* If you notice systematic variation in them, this is information you can squeeze into ${mtex`f`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":385,"value":"slide` ### Error Modeling\n\n* You can use models to seek out systematic variation in ${mtex`e_i`}\n* Cluster the ${mtex`x_i`} associated with large errors ${mtex`\\left|e_i\\right|`}\n* Train a model with ${mtex`e_i`} as a response\n * E.g., if you use a tree, the first few split variables still have information that you haven't made use of\n (we'll describe trees next lecture)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":391,"value":"slide`### Outliers and Leverage\n\n* Look for outliers either in the ${mtex`x`} or ${mtex`y`} directions\n* High leverage points are those that, if they were perturbed slightly, would dramatically alter the fit\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":4,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"hl = require(\"@observablehq/highlight.js@1.0.0/highlight.min.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":37,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\eps{\\epsilon}\n\\def\\F{\\mathcal{F}}\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\eps{\\epsilon}\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}