31:[] 2f:{"title":"Large-Scale Inference and Experimental Design","cells":[{"id":0,"value":"md`# Large-Scale Inference and Experimental Design\n\nIFT6758, Fall 2020\n\nReading: [MSMB](http://web.stanford.edu/class/bios221/book/introduction.html) 6.7 - 6.11, 13.1 - 13.4.\n\nOptional reading: [CASI](https://web.stanford.edu/~hastie/CASI_files/PDF/casi.pdf) 15.1 - 15.3\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":23,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nAre any of these genes associated with a disease of interest?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nIs anywhere in the cosmic microwave background nongaussian?\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":67,"value":"slide`### The Modern Regime\n\n* We have high-throughput data collection schemes, and it's now routine to collect many measurements per sample.\n* These sensors collect many aspects of each sample, and it's no longer possible (or desirable) to specify the relevant features to collect in advance\n\n

\n\nAmong many possible changes to interface design, do any actually affect user behavior?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":60,"value":"slide` ### Let's take a fun example\n\n[Source](https://xkcd.com/882/)\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"slide`### Abstraction\n\n* You need to scan through a large number of features to extract those that are most relevant\n* The ones that are interesting deviate from some pre-existing model of the system\n* Let's associate feature ${mtex`i`} with a hypothesis test,\n\n${mtex_block`\n\\begin{aligned}\nH_{i0}:& \\text{ Feature follows expected behavior}\\\\\nH_{i1}:& \\text{ Feature is of interest}\n\\end{aligned}\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":58,"value":"slide`### False Discoveries\n\n* By design, even if everything really is null, some fraction of tests will result in rejections \n\n

\n\nBy chance, you will end up in the red area ${mtex`\\approx \\alpha`}\\% of the time. Recall: jelly beans!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"slide`### Family-Wise Error Rate\n\n* How can we protect against this?\n* One idea is to control the probability of any falsely rejected hypothesis, when they are all in fact null\n${mtex_block`\n\\Psubarg{\\cap H_{i0}}{\\text{any } H_{i0} \\text{ is rejected }} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":102,"value":"slide`### Notation\n\n* How can we protect against this?\n* One idea is to control the probability of any falsely rejected hypothesis, when they are all in fact null\n${mtex_block`\n\\Psubarg{\\cap_m H_{i0}}{\\text{any } H_{i0} \\text{ is rejected }} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":103,"value":"slide` ### Notation\n\nWe need some new notation, suited to this point of view.\n\n| | Null is true | Null is false | Total |\n|:------------:|--------------|---------------|-------|\n| Rejected | ${mtex`V`} | ${mtex`S`} | ${mtex`R`} |\n| Not Rejected | ${mtex`U`} | ${mtex`T`} | ${mtex`m - R`} |\n| Total | ${mtex`m_0`} | m - ${mtex`m_0`} | ${mtex`m`} |\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"slide` ### Notation\n\nSome interpretations, so it doesn't become too confusing.\n\n| | Null is true | Null is false | Total |\n|:------------:|--------------|---------------|-------|\n| Rejected | False alarm | Correct detection | Number alarms |\n| Not Rejected | Proper silence | Missed signal | Number silent |\n| Total | Actually uninteresting | Actually relevant | Number scanned |\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":29,"value":"slide`### Family-Wise Error Rate\n\nWith this notation, the goal of FWER is to find a way to evaluating hypothesis so that\n${mtex_block`\n\\Parg{V > 0} \\leq \\alpha\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":149,"value":"slide`### Probability of false positives\n\nWith many hypotheses, you'll almost definitely have at least one false alarm. ${mtex `\\alpha`}: probability of false alarm in one sample\n\n${mtex_block`\n\\begin{aligned}\n\\Parg{V > 0} &= 1 - \\Parg{V = 0} \\\\\n&= 1 - \\left(1 - \\alpha\\right)^{m_0} \\\\\n&\\xrightarrow{m_0 \\to \\infty} 1,\n\\end{aligned}\n`}\nexcept when ${mtex`\\alpha = 0`} (and we never reject anything).\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"slide`### Bonferroni Correction\n\n* One way to do this is to make the original tests more stringent.\n* What rejection threshold ${mtex`\\alpha`} would we need, to control the FWER below ${mtex`0.05`}? Here's a picture when ${mtex`m_0 = 10,000`}.\n\n

\n\nThe black line is the function ${mtex`1 - \\left(1 - \\alpha\\right)^{m}`}. The crossing happens at ${mtex`\\frac{0.05}{10,000}`}. The idea of using ${mtex`\\frac{\\alpha}{m}`} as a rejection threshold is called the \"Bonferroni correction.\"\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":157,"value":"slide`### Justification\n\nWhy does this work?\n${mtex_block`\n\\begin{aligned}\n \\Parg{V > 0} &= \\Parg{\\cup_{i=0}^{m} \\{H_{i0} \\text{ is rejected}\\}} \\\\\n&\\leq \\sum_{i=0}^{m} \\Parg{H_{i0} \\text{ is rejected}} \\\\\n&= \\sum_{i=0}^{m} \\frac{\\alpha}{m} = \\alpha\n\\end{aligned}\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":31,"value":"slide`### Issues with FWER\n\n* This approach becomes very conservative with even a moderate number of tests\n* Even if there are *few* interesting hypotheses, you'll never reject them with such a stringent threshold\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":27,"value":"slide`### False Discovery Rate\n\nInstead of trying to prevent the occurrence of any false positive, just try to control their frequency.\n\nFormally, let's provide a guarantee on\n\n${mtex_block`\nFDR := \\Earg{\\frac{V}{R + 1}},\n`}\nthe expected proportion of false positives, among all the rejected hypothesis.\n\n*The number of false alarms is allowed to grow with the number of hypotheses tested.*\n\n* We want to reject those hypetheses for which ${mtex `\\text{FDR}(p)`} is less than ${mtex `\\alpha`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"slide`### Estimating FDR\n\nConsider the histogram of the ${mtex`p`}-values across all hypothesis, both those that are true and those that are not.\n\n* If a test statistic is drawn from the null, the associated ${mtex`p`}-value will be uniform\n* Otherwise, you will expect it to be small\n\n\nRef: [How to interpret a p-value histogram](http://varianceexplained.org/statistics/interpreting-pvalue-histogram/)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":461,"value":"slide`### Digression: Uniformity under null\n\n* Under the null, a ${mtex`p`}-value has 5% chance of being ${mtex`\\leq 0.05`}\n * It gives the probability of a false alarm under the null\n* More generally, it has ${mtex`\\alpha`} chance of being smaller than ${mtex`\\alpha`}.\n* Therefore, under the null, ${mtex`p`}-values are uniform\n\nRef: [How to interpret a p-value histogram](http://varianceexplained.org/statistics/interpreting-pvalue-histogram/)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":207,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an estimate of the ${mtex`FDR`}\n\n

\n\nMost of the rejection (green) region is blue, so the FDR will be small.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":497,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an setimate of the ${mtex`FDR`}\n\n

\n\nNow the FDR is larger.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":500,"value":"slide`### Estimating FDR\n\n* You will reject all hypotheses to the left of the vertical bar\n* The fraction of the rejection area that seems to come from the uniform part is an setimate of the ${mtex`FDR`}\n\n

\n\nAnd now even larger.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":216,"value":"slide`### Benjamini-Hochberg (BH) procedure\n\n* When the ${mtex `p`}-value histogram is almost uniform, at most a small percentage of hypotheses are non-null \n* The BH procedure to identify that percentage. It lets us control the FDR\n* Intuition: Make the rectangle as large as you can (make the most rejections), while limiting the estimated FDR\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":224,"value":"slide`### Formal Definition\n\n* Sort the ${mtex`p`}-values, ${mtex`p_{(1)}, \\dots, p_{(m)}`}\n* Find the last of the ${mtex`p_{(k)}`} below the line ${mtex`\\frac{\\alpha k}{m}`}\n* Reject all hypotheses with ${mtex`p`}-values smaller than that\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":237,"value":"slide`### Justification\n\n* This procedure is not as arbitrary as it might seem at first\n* We need to quantify the intuition from the histogram\n\nSince the histogram is has both a null (uniform) and an alternative component, we can think of it as a mixture density,\n\n${mtex_block`\nf\\left(p\\right) = \\pi_0 \\text{Unif}\\left[0, 1\\right] + \\left(1 - \\pi_0\\right)f_{alt}\\left(p\\right)\n`}\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":249,"value":"slide`### Justification\n\nLet's represent the \"instantaneous\" false discovery rate associated with any point on the histogram using\n${mtex_block`\nfdr\\left(p\\right) = \\frac{\\text{null part}}{\\text{rejection total}} = \\frac{\\pi_0}{f\\left(p\\right)},\n`}\nwhere ${mtex`f\\left(p\\right)`} is the density associated with the mixture histogram.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":257,"value":"slide`### Justification\n\nLet ${mtex`F`} represent the CDF associated with this mixture. Then we can define a false discovery rate in terms of the ratios of the areas in the histograms,\n\n${mtex_block`\nFdr\\left(p\\right) = \\frac{\\text{null part to left}}{\\text{rejection total to left}} = \\frac{\\pi_0 p}{F\\left(p\\right)},\n`}\nwhere ${mtex`F\\left(p\\right)`} is the CDF associated with the mixture histogram.\n\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":260,"value":"slide`### Justification\n\nWe'll estimate the FDR by plugging in the empirical ${mtex`p`}-value distribution in for ${mtex`F`},\n\n${mtex_block`\n\\widehat{Fdr}\\left(p\\right) = \\frac{\\pi_0 p}{\\hat{F}\\left(p\\right)}\n`}\n\nWe want the largest ${mtex`p`} which still satisfies ${mtex`\\widehat{Fdr}\\left(p\\right) \\leq \\alpha`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":267,"value":"slide`### Justification\n\n\nThink about the smallest ${mtex`p`}-value, ${mtex`p_{(1)}`}. Notice that,\n\n${mtex_block`\n\\widehat{Fdr}\\left(p_{(1)}\\right) = \\frac{\\pi_0 p_{(1)}}{1 / m},\n`}\n\nso that this estimate is smaller than ${mtex`\\alpha`} if and only if ${mtex_block`\n\\begin{aligned}\np_{(1)} \\leq \\frac{\\alpha}{m \\pi_0}.\n\\end{aligned}\n`}\n\nSo if ${mtex`p_{(1)} \\leq \\frac{\\alpha}{m}`}, that's good enough (because ${mtex`\\pi_0 \\leq 1`}).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":287,"value":"slide`### Justification\n\nSimilarly, for the ${mtex`k^{th}`}-largest ${mtex`p`}-value, ${mtex_block`\\widehat{Fdr}\\left(p_{(k)}\\right) = \\frac{\\pi_0 p_{(k)}}{k / m} \\leq \\alpha`} if and only if,\n\n${mtex_block`\n\\begin{aligned}\np_{(k)} \\leq \\frac{k\\alpha}{m \\pi_0},\n\\end{aligned}\n`}\nand it would be good enough if ${mtex`p_{(k)} \\leq \\frac{k\\alpha}{m}`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":296,"value":"slide`### Justification\n\nTherefore, the BH procedure is just trying to use the observed histogram of ${mtex`p`}-values to select the largest ${mtex`p_{(k)}`} so that the estimated ${mtex`\\widehat{Fdr}`} is less than ${mtex`\\alpha`}.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"slide`### The Multiple Testing ~~problem~~ opportunity\n\n* Usually, people think of the the fact that we have to screen many hypotheses as a problem\n * (because you can get false positives)\n* However, all these extra (mostly null) hypotheses give us extra information\n* We were able to use them to estimate the FDR, and it's in fact possible to use to estimate reference distributions\n * No need for theory-driven references\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":40,"value":"slide`## Experimental Design\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":183,"value":"slide`### Why Experimental Design?\n\n* The effects you observe can be driven by many factors\n* To draw effective inferences, need to minimize potential for alternative explanations\n* The best data scientists have an understanding of (or some control over) the collection process\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":568,"value":"slide`### Basic Principles\n\n* Confounding\n* Blocking \n* Power analysis\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":571,"value":"slide`### Confounding\n\n\n${mtex `X`} and ${mtex `Y`} are *confounded* by ${mtex `Z`} whenever ${mtex `Z`} causally influences both ${mtex `X`} and ${mtex `Y`}.\n\n\n\n\n\n\nSo, effect of ${mtex `Z`} should be accounted for while computing the effect of ${mtex `X`} on ${mtex `Y`}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":574,"value":"slide`### Blocking \n\nBlocking: arranging of experimental units in groups (blocks) that are similar to one another\n\nAim: reduce unexplained variability\n\nPrinciple: variability which cannot be overcome is confounded\n\n* Large, systematic sources of variation should be \"blocked\"\n* This makes it possible to see the residual variation\n\n\n\nExample: An experiment is designed to test a new drug on patients. There are two levels of the treatment, drug, and placebo. \n\n**Blocking factor**: gender of the patients\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":577,"value":"slide`### Blocking\n\nYou are asked to evaluate whether a new type of shoe sole will last longer than the previous design. How should you design your experiment? \n\n* Predictors (e.g. raw materials)? Response (e.g. longevity)?\n* What are possible blocking factors (e.g., location, climate)?\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":580,"value":"slide`### Power Analysis\n\nThe power of any study is driven by\n\n1) The sample size used to estimate effects of interest\n\n2) The variability in the outcome of interest, within otherwise comparable samples\n\n3) The underlying [effect size](https://www.leeds.ac.uk/educol/documents/00002182.htm#:~:text=Effect%20size%20is%20a%20simple,confounding%20this%20with%20sample%20size.&text=A%20number%20of%20alternative%20measures%20of%20effect%20size%20are%20described.) on the outcome of interest\n - a way of **quantifying the size of the difference between two groups**: *standardized mean difference* between *experimental group and control group*\n - Effect size and ${mtex `Z`}-score\n - Effect size and its confidence interval\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":583,"value":"slide`### Power Analysis\n\nUnfortunately, (3) is very hard to know before starting the study. There are a few common ways around this,\n\n* Simulate: Generate data that look like what you expect to see, under a few different effect sizes. See how data collection and analysis choices influence the downstream power, across effect sizes.\n* Pilot study: Get a small preliminary study, for the explicit purpose of getting a ballpark effect size estimate.\n* Sequential design: Conduct your study in stages, getting better estimates of the effect over time.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":341,"value":"slide`### Types of Experiments (informal)\n\n* Controlled Experiment: You have control over all factors that may influence the outcome\n* Study: You do not have control over some factors\n* Meta-Analysis: You combine the results from many experiments\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":344,"value":"slide`### Variants: Observational Study\n\n* In these studies, you control neither the samples that are collected nor the treatments they were assigned\n* Be wary that \"correlation is not causation\"\n - Examples of illogically inferring causation from correlation\n 1. The faster that windmills are observed to rotate, the more wind is observed.\nTherefore, wind is caused by the rotation of windmills.\n 2. Children that watch a lot of TV are the most violent. Clearly, TV makes children more violent.\n\n - Ref: [Correlation does not imply causation](https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":353,"value":"slide`### Randomized Control Trial\n\n* You have no control over most factors, but you **can** randomize the assignment of the treatment\n* By randomizing the assignment, other influences will cancel out, on average\n* Here, correlation *does* imply causation (assuming that the model is correct...)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":532,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n* ${mtex`t_i`} is the treatment,\n* ${mtex`x_i`} are measured influencing factors\n* ${mtex`\\epsilon_i`} are i.i.d. with mean 0 (from unmeasured factors)\n\n

\n\nIn theory, each sample could have two outcomes, depending on whether they are given the treatment or not.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":549,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n* ${mtex`t_i`} is the treatment,\n* ${mtex`x_i`} are measured influencing factors\n* ${mtex`\\epsilon_i`} are i.i.d. with mean 0 (from unmeasured factors)\n\n

\n\nIn reality, you can observe each sample under either the treatment or the control (not both).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":529,"value":"slide`### Types of Experiments (formal)\n\nFormally, there is a process influencing the outcome,\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\n* Controlled experiment: Can manipulate all ${mtex`x_i, t_i`}, and ${mtex`\\epsilon_i \\equiv 0`}.\n* Study: Able to manipulate some of the ${mtex`x_i`} and / or ${mtex`t_i`}\n* Observational study: Cannot manipulate any of ${mtex`x_i`} or ${mtex`t_i`}\\\n* Randomized controlled study: Randomize over ${mtex`t_i`}, can't manipulate ${mtex`x_i`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":545,"value":"slide`### Controlled Experiment\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nCan manipulate all ${mtex`x_i, t_i`}, and ${mtex`\\epsilon_i \\equiv 0`}. In the example here, we get both a treatment and a control sample on an even grid of ${mtex`x`}'s.\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":559,"value":"slide`### Observational Study\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nHave to guard against confounders.\n\n

\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":555,"value":"slide`### Randomized Controlled Trial\n\n${mtex_block`\ny_i = f\\left(x_i, t_i\\right) + \\epsilon_i\n`}\n\nRandomize over ${mtex`t_i`}, can't manipulate ${mtex`x_i`}. When the treatment effect is uniform over ${mtex`x`}, this randomization causes any effect of ${mtex`t_i`} to get averaged out, leaving only the overall effect of ${mtex`x`}. \n\n${mtex_block `\\text{Chain rule: } \\quad P(y|x) = \\sum_{t} P(y|x, t)P(t|x)`}\n\nWhen it's not constant over ${mtex`x`} (like the figure below), there is some approximation error.\n\n

\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":356,"value":"slide`### Data Quality\n\nData quality is a vague term.\n\nInstead, consider your data's *fitness for purpose*.\n\nWhat you want to do will guide your data collection and analysis strategies.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":426,"value":"slide`### Data Quality\n\n

\n\nA famous [story](https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d) about Abraham Wald (founder of decision theory). Where should you put the armor?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":420,"value":"slide`### Data Science in context\n\n

\n\nFigure from [this post](https://simplystatistics.org/2019/04/17/tukey-design-thinking-and-better-questions/). Some other useful references are,\n\n* [How to share data with a statistician](https://github.com/jtleek/datasharing)\n - e.g., raw data, clean data, metadata\n\n* [Design Study Methodology](https://www.cs.ubc.ca/labs/imager/tr/2012/dsm/)\n - 9 stages of design\n - common pitfalls \n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":412,"value":"slide`### Where to next?\n\n* We've reviewed the roles of prediction, unsupervised learning, and statistical inference in data science\n\n* We assumed the following\n * Samples drawn ${mtex`x_i \\stackrel{i.i.d.}{\\sim} F`}\n * Generic structure between coordinates of ${mtex`x_i`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":481,"value":"slide`### Where to next?\n\n* What if there are features that might be useful, but which weren't directly included?\n* What if your samples have relationships with one another?\n* What if the correlation has very specific types of structure?\n\n

\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"import {slide} from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"slide_style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"function mtex_block() {\n return tex.block`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\Psubarg#1#2{\\P_{#1}\\left[{#2}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":73,"value":"function mtex() {\n return tex`\n\\def\\reals{\\mathbb{R}}\n\\def\\E{\\mathbb{E}} % Expectation symbol\n\\def\\Var{\\text{Var}} % Expectation symbol\n\\def\\P{\\mathbb{P}} % Expectation symbol\n\\def\\Earg#1{\\E\\left[{#1}\\right]}\n\\def\\Vararg#1{\\V\\left[{#1}\\right]}\n\\def\\Esubarg#1#2{\\E_{#1}\\left[{#2}\\right]}\n\\def\\Parg#1{\\P\\left[{#1}\\right]}\n\\def\\Psubarg#1#2{\\P_{#1}\\left[{#2}\\right]}\n\\def\\eps{\\epsilon}\n\\newcommand\\mbb[1]{\\mathbb{#1}}\n\\def\\indic#1{\\mbb{I}\\left({#1}\\right)} % Indicator function\n\\def\\Vararg#1{\\Var\\left[{#1}\\right]}\n\\def\\absarg#1{\\left|#1\\right|}\n\\renewcommand{\\exp}[1]{\\operatorname{exp}\\left(#1\\right)} % Exponential\n${String.raw.apply(String, arguments)}\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}