31:[] 2f:{"title":"Large-Scale Inference and Experimental Design","cells":[{"id":0,"value":"md`# Large-Scale Inference and Experimental Design\n\nIFT6758, Fall 2019\n\nReading: MSMB 6.7 - 6.11, 13.1 - 13.4.\n\nOptional reading: CASI 15.1 - 15.3\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":23,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nAre any of these genes associated with a disease of interest?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nIs anywhere in the cosmic microwave background nongaussian?\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":67,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nAmong many possible changes to interface design, do any actually affect user behavior?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":60,"value":"slide` ### Alternatively\n\nIf these examples are too serious for you\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"slide`### Abstraction\n\n* You need to scan through a large number of features to extract those that are most relevant\n* The ones that are interesting deviate from some pre-existing model of the system\n* Let's associate feature ${mtex`i`} with a hypothesis test,\n\n${mtex_block`\n\\begin{aligned}\nH_{i0}:& \\text{ Feature follows expected behavior}\\\\\nH_{i1}:& \\text{ Feature is of interest}\n\\end{aligned}\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":58,"value":"slide`### False Discoveries\n\n* By design, even if everything really is null, some fraction of tests will result in rejections \n\n

\n\nBy chance, you will end up in the red area ${mtex`\\approx \\alpha`}\\% of the time. This is the point of the XKCD cartoon.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"slide`### Family-Wise Error Rate\n\n* How can we protect against this?\n* One idea is to control the probability of any falsely rejected hypothesis, when they are all in fact null\n${mtex_block`\n\\Psubarg{\\cap H_{0i}}{\\text{any } H_{i0} \\text{ is rejected }} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":102,"value":"slide`### Notation\n\n* How can we protect against this?\n* One idea is to control the probability of any falsely rejected hypothesis, when they are all in fact null\n${mtex_block`\n\\Psubarg{\\cap H_{0i}}{\\text{any } H_{i0} \\text{ is rejected }} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":103,"value":"slide` ### Notation\n\nWe need some new notation, suited to this point of view.\n\n| | Null is true | Null is false | Total |\n|:------------:|--------------|---------------|-------|\n| Rejected | ${mtex`V`} | ${mtex`S`} | ${mtex`R`} |\n| Not Rejected | ${mtex`U`} | ${mtex`T`} | ${mtex`m - R`} |\n| Total | ${mtex`m_0`} | m - ${mtex`m_0`} | ${mtex`m`} |\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"slide` ### Notation\n\nSome interpretations, so it doesn't become too confusing.\n\n| | Null is true | Null is false | Total |\n|:------------:|--------------|---------------|-------|\n| Rejected | False alarm | Correct detection | Number alarms |\n| Not Rejected | Proper silence | Missed signal | Number silent |\n| Total | Actually uninteresting | Actually relevant | Number scanned |\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":29,"value":"slide`### Family-Wise Error Rate\n\nWith this notation, the goal of FWER is to find a way to evaluating hypothesis so that\n${mtex_block`\n\\Parg{V > 0} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":149,"value":"slide`### Probability of no false positives\n\nWith many hypotheses, you'll almost definitely have at least one false alarm.\n\n${mtex_block`\n\\begin{aligned}\n\\Parg{V > 0} &= 1 - \\Parg{V = 0} \\\\\n&= 1 - \\left(1 - \\alpha\\right)^{m_0} \\\\\n&\\xrightarrow{m_0 \\to \\infty} 1,\n\\end{aligned}\n`}\nexcept when ${mtex`\\alpha = 0`} (and we never reject anything).\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"slide`### Bonferroni Correction\n\n* One way to do this is to make the original tests more stringent.\n* What rejection threshold ${mtex`\\alpha`} would we need, to control the FWER below ${mtex`0.05`}? Here's a picture when ${mtex`m_0 = 10,000`}.\n\n

\n\nThe black line is the function ${mtex`1 - \\left(1 - \\alpha\\right)^{m}`}. The crossing happens at ${mtex`\\frac{0.05}{10,000}`}. The idea of using ${mtex`\\frac{\\alpha}{m}`} as a rejection threshold is called the \"Bonferroni correction.\"\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":157,"value":"slide`### Justification\n\nWhy does this work?\n${mtex_block`\n\\begin{aligned}\n \\Parg{V > 0} &= \\Parg{\\cup_{i = 1}^{m} \\{H_{0i} \\text{ is rejected}\\}} \\\\\n&\\leq \\sum_{i = 1}^{m} \\Parg{H_{0i} \\text{ is rejected}} \\\\\n&= \\sum_{i = 1}^{m} \\frac{\\alpha}{m} = \\alpha\n\\end{aligned}\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":31,"value":"slide`### Issues with FWER\n\n* This approach becomes very conservative with even a moderate number of tests\n* Even if there *are* a few interesting hypotheses, you'll never reject them with such a stringent threshold\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":27,"value":"slide`### False Discovery Rate\n\nInstead of trying to prevent the occurrence of any false positive, just try to control their frequency.\n\nFormally, let's provide a guarantee on\n\n${mtex_block`\nFDR := \\Earg{\\frac{V}{R \\vee 1}},\n`}\nthe expected proportion of false positives, among all the rejected hypothesis.\n\n*The number of false alarms is allowed to grow with the number of hypotheses tested.*\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"slide`### Estimating FDR\n\nConsider the histogram of the ${mtex`p`}-values across all hypothesis, both those that are true and those that are not.\n\n* If a test statistic is drawn from the null, the associated ${mtex`p`}-value will be uniform\n* Otherwise, you will expect it to be small\n\n

\n\nFrom [this post](http://varianceexplained.org/statistics/interpreting-pvalue-histogram/)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":461,"value":"slide`### Digression: Uniformity under null\n\n* Under the null, a ${mtex`p`}-value has 5% chance of being ${mtex`\\leq 0.05`}\n * It gives the probability of a false alarm under the null\n* More generally, it has ${mtex`\\alpha`} chance of being smaller than ${mtex`\\alpha`}.\n* Therefore, under the null, ${mtex`p`}-values are uniform\n\n

\n\nFrom [this post](http://varianceexplained.org/statistics/interpreting-pvalue-histogram/)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":207,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an setimate of the ${mtex`FDR`}\n\n

\n\nMost of the rejection (green) region is blue, so the FDR will be small.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":497,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an setimate of the ${mtex`FDR`}\n\n

\n\nNow the FDR is larger.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":500,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an setimate of the ${mtex`FDR`}\n\n

\n\nAnd now even larger.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":216,"value":"slide`### Benjamini-Hochberg (BH) procedure\n\n* The BH procedure lets us control the FDR\n* Intuition: Make the rectangle as large as you can (make the most rejections), while limiting the estimated FDR\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":224,"value":"slide`### Formal Definition\n\n* Sort the ${mtex`p`}-values, ${mtex`p_{(1)}, \\dots, p_{(m)}`}\n* Find the last of the ${mtex`p_{(k)}`} below the line ${mtex`\\frac{\\alpha k}{m}`}\n* Reject all hypotheses with ${mtex`p`}-values smaller than that\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":237,"value":"slide`### Justification\n\n* This procedure is not as arbitrary as it might seem at first\n* We need to quantify the intuition from the histogram\n\nSince the histogram is has both a null (uniform) and an alternative component, we can think of it as a mixture density,\n\n${mtex_block`\nf\\left(p\\right) = \\pi_0 \\text{Unif}\\left[0, 1\\right] + \\left(1 - \\pi_0\\right)f_{alt}\\left(p\\right)\n`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":249,"value":"slide`### Justification\n\nLet's represent the \"instantaneous\" false discovery rate associated with any point on the histogram using\n${mtex_block`\nfdr\\left(p\\right) = \\frac{\\text{null part}}{\\text{rejection total}} = \\frac{\\pi_0}{f\\left(p\\right)},\n`}\nwhere ${mtex`f\\left(p\\right)`} is the density associated with the mixture histogram.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":257,"value":"slide`### Justification\n\nLet ${mtex`F`} represent the CDF associated with this mixture. Then we can define a false discovery rate in terms of the ratios of the areas in the histograms,\n\n${mtex_block`\nFdr\\left(p\\right) = \\frac{\\text{null part to left}}{\\text{rejection total to left}} = \\frac{\\pi_0 p}{F\\left(p\\right)},\n`}\nwhere ${mtex`F\\left(p\\right)`} is the CDF associated with the mixture histogram.\n\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":260,"value":"slide`### Justification\n\nWe'll estimate the FDR by plugging in the empirical ${mtex`p`}-value distribution in for ${mtex`F`},\n\n${mtex_block`\n\\widehat{Fdr}\\left(p\\right) = \\frac{\\pi_0 p}{\\hat{F}\\left(p\\right)}\n`}\n\nWe want the largest ${mtex`p`} which still satisfies ${mtex`\\widehat{Fdr}\\left(p\\right) \\leq \\alpha`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":267,"value":"slide`### Justification\n\n\nThink about the smallest ${mtex`p`}-value, ${mtex`p_{(1)}`}. Notice that,\n\n${mtex_block`\n\\widehat{Fdr}\\left(p_{(1)}\\right) = \\frac{\\pi_0 p_{(1)}}{1 / m},\n`}\n\nso that this estimate is smaller than ${mtex`\\alpha`} if and only if ${mtex_block`\n\\begin{aligned}\np_{(1)} \\leq \\frac{\\alpha}{m \\pi_0}.\n\\end{aligned}\n`}\n\nSo if ${mtex`p_{(1)} \\leq \\frac{\\alpha}{m}`}, that's good enough (because ${mtex`\\pi_0 \\leq 1`}).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":287,"value":"slide`### Justification\n\nSimilarly, for the ${mtex`k^{th}`}-largest ${mtex`p`}-value, ${mtex_block`\\widehat{Fdr}\\left(p_{(k)}\\right) = \\frac{\\pi_0 p_{(k)}}{k / m} \\leq \\alpha`} if and only if,\n\n${mtex_block`\n\\begin{aligned}\np_{(k)} \\leq \\frac{k\\alpha}{m \\pi_0},\n\\end{aligned}\n`}\nand it would be good enough if ${mtex`p_{(k)} \\leq \\frac{k\\alpha}{m}`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":296,"value":"slide`### Justification\n\nTherefore, the BH procedure is just trying to use the observed histogram of ${mtex`p`}-values to select the largest ${mtex`p_{(k)}`} so that the estimated ${mtex`\\widehat{Fdr}`} is less than ${mtex`\\alpha`}.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"slide`### The Multiple Testing ~~problem~~ opportunity\n\n* Usually, people think of the the fact that we have to screen many hypotheses as a problem\n * (because you can get false positives)\n* However, all these extra (mostly null) hypotheses give us extra information\n* We were able to use them to estimate the FDR, and it's in fact possible to use to estimate reference distributions\n * No need for theory-driven references\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":40,"value":"slide`## Experimental Design\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":183,"value":"slide`### Why Experimental Design?\n\n* The effects you observe can be driven by many factors\n* To draw effective inferences, need to minimize potential for alternative explanations\n* The best data scientists have an understanding of (or some control over) the collection process\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":568,"value":"slide`### Basic Principles\n\n* Confounding\n* Blocking and pairing\n* Power analysis\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":571,"value":"slide`### Confounding\n\nIf there are factors that may influence both the treatment assignment and the observed outcome, then they can't be ruled out as potential explanations for the outcome.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":574,"value":"slide`### Blocking and Pairing\n\n* Large, systematic sources of variation should be \"blocked\"\n* This makes it possible to see the residual variation\n\n

\n\nHere, the effect of the treatment would get drowned out by inter-individual variation, unless you explicitly looked at the differences on a per-person level.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":577,"value":"slide`### Blocking\n\nStory: You are asked to evaluate whether a new type of shoe sole will last longer than the previous design. How should you design your experiment?\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":580,"value":"slide`### Power Analysis\n\nThe power of any study is driven by\n\n1) The sample size used to estimate effects of interest\n\n2) The variability in the outcome of interest, within otherwise comparable samples\n\n3) The underlying effect size on the outcome of interest\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":583,"value":"slide`### Power Analysis\n\nUnfortunately, (3) is very hard to know before starting the study. There are a few common ways around this,\n\n* Simulate: Generate data that look like what you expect to see, under a few different effect sizes. See how data collection and analysis choices influence the downstream power, across effect sizes.\n* Pilot study: Get a small preliminary study, for the explicit purpose of getting a ballpark effect size estimate.\n* Sequential design: Conduct your study in stages, getting better estimates of the effect over time.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":341,"value":"slide`### Types of Experiments (informal)\n\n* Controlled Experiment: You have control over all factors that may influence the outcome\n* Study: You do not have control over some factors\n* Meta-Analysis: You combine the results from many experiments\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":344,"value":"slide`### Variants: Observational Study\n\n* In these studies, you control neither the samples that are collected nor the treatments they were assigned\n* Be wary that \"correlation is not causation\"\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":353,"value":"slide`### Randomized Control Trial\n\n* You have no control over most factors, but you **can** randomize the assignment of the treatment\n* By randomizing the assignment, other influences will cancel out, on average\n* Here, correlation *does* imply causation (assuming that the model is correct...)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":532,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n* ${mtex`t_i`} is the treatment,\n* ${mtex`x_i`} are measured influencing factors\n* ${mtex`\\epsilon_i`} are i.i.d. with mean 0 (from unmeasured factors)\n\n

\n\nIn theory, each sample could have two outcomes, depending on whether they are given the treatment or not.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":549,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n* ${mtex`t_i`} is the treatment,\n* ${mtex`x_i`} are measured influencing factors\n* ${mtex`\\epsilon_i`} are i.i.d. with mean 0 (from unmeasured factors)\n\n

\n\nIn reality, you can observe each sample under either the treatment or the control (not both).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":529,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\n* Controlled experiment: Can manipulate all ${mtex`x_i, t_i`}, and ${mtex`\\epsilon_i \\equiv 0`}.\n* Study: Able to manipulate some of the ${mtex`x_i`} and / or ${mtex`t_i`}\n* Observational study: Cannot manipulate any of ${mtex`x_i`} or ${mtex`t_i`}\\\n* Randomized controlled study: Randomize over ${mtex`t_i`}, can't manipulate ${mtex`x_i`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":545,"value":"slide`### Controlled Experiment\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nCan manipulate all ${mtex`x_i, t_i`}, and ${mtex`\\epsilon_i \\equiv 0`}. In the example here, we get both a treatment and a control sample on an even grid of ${mtex`x`}'s.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":559,"value":"slide`### Observational Study\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nHave to guard against confounders.\n\n

\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":555,"value":"slide`### Randomized Controlled Trial\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nRandomize over ${mtex`t_i`}, can't manipulate ${mtex`x_i`}. When the treatment effect is constant over ${mtex`x`}, any effect caused by ${mtex`x_i`} gets averaged out, leaving only the overall treatment effect. \n\nWhen it's not constant over ${mtex`x`} (like the figure below), there is some approximation error.\n\n

\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":356,"value":"slide`### Data Quality\n\nData quality is a fuzzy term.\n\nInstead, consider your data's *fitness for purpose*.\n\nWhat you want to do will guide your data collection and analysis strategies.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":426,"value":"slide`### Data Quality\n\n

\n\nA famous [story](https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d) about Abraham Wald (founder of decision theory). Where should you put the armor?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":420,"value":"slide`### Data Science in context\n\n

\n\nFigure from [this post](https://simplystatistics.org/2019/04/17/tukey-design-thinking-and-better-questions/). Some other useful references are,\n\n* [How to share data with a statistician](https://github.com/jtleek/datasharing)\n* [Design Study Methodology](https://www.cs.ubc.ca/labs/imager/tr/2012/dsm/)\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":412,"value":"slide`### Where to next?\n\n* We've reviewed the roles of prediction, unsupervised learning, and statistical inference in data science\n* We always assumed that our data are tabular\n * Samples drawn ${mtex`x_i \\stackrel{i.i.d.}{\\sim} F`}\n * Generic structure between coordinates of ${mtex`x_i`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":481,"value":"slide`### Where to next?\n\n* What if there are features that might be useful, but which weren't directly included?\n\n

\n\nFrom the winning solution described [here](https://stanford.edu/~lmackey/alsprize4life.html).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":440,"value":"slide`### Where to next?\n\n* What if your samples have relationships with one another?\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":445,"value":"slide`### Where to next?\n\n* What if the correlation has very specific types of structure?\n\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":451,"value":"slide`### Where to next?\n\n(to be continued...)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\Psubarg#1#2{\\P_{#1}\\left[{#2}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":73,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\Psubarg#1#2{\\P_{#1}\\left[{#2}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}