2b:null 2c:["$","title",null,{"children":"Maximum likelihood - Bernoulli data | Mattias Villani | Observable"}] 31:T5b4,The interactive graph to the left below shows the Bernoulli distribution with the height of the spikes representing the probabilities of failure (${tex`x=0`}) and success (${tex`x=1`}) according to your selected probability of success ${tex`\theta`} using the slider. The proportion of successes in the data are also shown as the dark red horizontal line. Click on *show ML fitting* to add an optimal model fit according to the maximum likelihood (ML) method. This method chooses the ${tex`\theta`} that maximizes the average probability (likelihood) of the observed data. For the Bernoulli distribution, this optimal estimator of ${tex`\theta`} is the sample proportion of successes ${tex`s/n`} in the data. We prove this mathematically in an appendix at the end of the notebook. As you can see by interacting with the graph, each value of ${tex`\theta`} provides an *average probability* for the data. We use a [geometric mean](https://en.wikipedia.org/wiki/Geometric_mean), which is more suitable for an average of probabilities. In the figure to the right, we plot all these values as a function of ${tex`\theta`}. The maximum likelihood estimate of ${tex`\theta`} is the value that maximizes this function. The orange dot in the graph indicates the average probability of the observed for the ${tex`\theta`} value that you have chosen with the slider. When it is at the peak of the function to the right, you have found the maximum likelihood estimate!32:T592,> *This section requires a bit more mathematics and can be skipped if you are not accustomed to the technique of maximizing a function by finding the point where the derivative of the function is zero.* In the figures above, we maximized the geometric mean of the probability for the observed data. Maximizing this average probability is equivalent to maximizing the joint probability for the entire dataset ${tex`P(x_1,x_2,\ldots,x_n \vert \theta)=\prod_{i=1}^n P(x_i \vert \theta)`} where we have used that the observations are assumed to be independent and therefore the joint probability becomes a product of the probability for each observation, ${tex`P(x_i \vert \theta)`}. Note that data is written with lowercase letters because here we are interested in maximizing the probability for the dataset that we have **observed**. In the expression above, we have explicitly written out that the probability of observing a certain dataset depends on ${tex`\theta`}; we want to find the value for ${tex`\theta`} that maximizes the probability for the observed data. The joint probability for the data ${tex`P(x_1,x_2,\ldots,x_n \vert \theta)`} considered as a function of the parameter ${tex`\theta`} for a fixed (observed) dataset is called the **likelihood function**, or just the **likelihood**. The maximum likelihood estimate of ${tex`\theta`} is the value of the parameter that maximizes the likelihood function. 33:T5a8,In the case where data comes from a Bernoulli distribution, the probability function for an observation is simply ${tex`P(x) = \theta^x (1-\theta)^{1-x}`}. Because of independence, the likelihood function is therefore the product ${tex`P(x_1,x_2,\ldots,x_n \vert \theta) = \prod_{i=1}^n \theta^{x_i} (1-\theta)^{1-x_i}=\theta^s (1-\theta)^{n-s},`} where ${tex`s=\sum_{i=1}^n x_i`} is the number of successes.
Hence, the log-likelihood is: ${tex`\ell(\theta) = s\log \theta + {(n-s)}\log(1-\theta)`} We know from mathematical analysis that the maximum of a function ${tex`f(x)`} is found by setting the first derivative to zero and solving for ${tex`x`}. The first derivative of the log-likelihood is ${tex`\frac{d}{d\theta}\ell(\theta) = \frac{s}{\theta} -\frac{n-s}{1-\theta}`} Setting the first derivative to zero ${tex`\frac{s}{\theta} -\frac{n-s}{1-\theta} = 0`} and solving for ${tex`\theta`} gives the solution ${tex`\theta = s/n`}, the fraction of successes in the sample. We can verify that this is indeed a maximum by checking whether the second derivative is negative at ${tex`\theta = s/n`}. The second derivative is ${tex`\frac{d^2}{d\theta^2}\ell(\theta) = -\frac{s}{\theta^2} - \frac{n-s}{(1-\theta)^2}`} which is negative for all ${tex`\theta`}. The maximum likelihood estimator of the parameter ${tex`\theta`} in the Bernoulli model is therefore the sample proportion of successes ${tex`\hat\theta = s/n`}. 😅🥳34:[] 2f:{"id":"9642c03f61368c7b","slug":"maximum-likelihood-bernoulli-data","title":"Maximum likelihood - Bernoulli data","thumbnail":"78297402bf2d929494b51c2142d11b5fd8d62780756032e6817b3d087175b971","access_level":"public","create_time":"$D2023-11-18T13:10:49.016Z","update_time":"$D2025-02-20T12:34:06.581Z","view_time":null,"first_public_time":"$D2023-11-18T14:21:27.738Z","fork":{"id":"630c0c263bea1ca0","slug":"maximum-likelihood-poisson-regression","title":"Maximum likelihood - Poisson regression","owner":{"id":"7592a7ab121f74a0","login":"mattiasvillani","name":"Mattias Villani"}},"stars":0,"starred":false,"owner":{"type":"team","id":"7592a7ab121f74a0","login":"mattiasvillani","name":"Mattias Villani","avatar_url":"https://avatars-next.observableusercontent.com/a440aaf39a3abefb86cbcc758fd7a5ab42555ae68d6825b52ec006f385a94d1d","tier":"pro_2024"},"version":2421,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"56499be8cf43f0ef","login":"mattiasvillani","name":"Mattias Villani","avatar_url":"https://avatars-next.observableusercontent.com/a440aaf39a3abefb86cbcc758fd7a5ab42555ae68d6825b52ec006f385a94d1d","tier":null}],"members":[{"type":"individual","id":"56499be8cf43f0ef","login":"mattiasvillani","name":"Mattias Villani","avatar_url":"https://avatars-next.observableusercontent.com/a440aaf39a3abefb86cbcc758fd7a5ab42555ae68d6825b52ec006f385a94d1d","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":1,"collection":{"id":"209e13808f44ed11","slug":"sda1","title":"SDA1","owner":{"id":"7592a7ab121f74a0","login":"mattiasvillani","name":"Mattias Villani"}},"type":"notebook","body":{"title":"Maximum likelihood - Bernoulli data","cells":[{"id":38,"value":"# Maximum likelihood - Bernoulli data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1158,"value":"Assume that we have binary data modeled as independent Bernoulli variables with success probability ${tex`\\theta`}:\n\n${tex`X_1, X_2, \\ldots, X_n \\sim \\text{Bern}(\\theta)`}\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1175,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":2145,"value":"modell = tex`\\text{Model: }X_1,X_2,\\ldots,X_{${n}} \\overset{\\mathrm{indep}}{\\sim}\\mathrm{Bern}(\\theta)`\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1913,"value":"viewof n = Inputs.range([1,100], {value: 5, step: 1, label: tex`\\text{trials }n`})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2259,"value":"viewof s = Inputs.range([0,n], {value: 2, step: 1, label: tex`\\text{successes }s`})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":118,"value":"viewof settings = Inputs.form([\n Inputs.range([0.001, 0.999], {value: 0.5, step: 0.01, label: tex`\\text{parameter value }\\theta`}),\n Inputs.toggle({label: \"show ML fit\", value: false})\n ])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1121,"value":"textslh = {\n if (!settings[1]){\n return html`Average probability for observed data with the model \n Bern(θ = ${theta}) is ${geomean_like(s, n, theta).toPrecision(4)}`\n }\n else{\n return html`Average probability for observed data with the model \n Pois(θ = ${theta}) is ${geomean_like(s, n, theta).toPrecision(4)}
\n Average probability for observed data with the ML estimate \n Pois(θ = ${mle.toPrecision(4)}) is ${geomean_like(s, n ,mle).toPrecision(4)}\n`\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1218,"value":"bothplots = html`\n \n \n \n

\n ${pltdata}\n

\n ${pltlikelihood}\n

\n \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1233,"value":"### Appendix - derivation of the maximum likelihood estimator","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1235,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1274,"value":"It often turns out to be easier to maximize the *logarithm* of the likelihood function, the so-called **log-likelihood**:\n\n${tex`\\ell(\\theta) = \\log P(x_1,x_2,\\ldots,x_n \\vert \\theta)=\\sum_{i=1}^n \\log P(x_i \\vert \\theta)`}.\n\nSince the logarithm is a monotonically increasing function, the same ${tex`\\theta`} will maximize both the likelihood and the log-likelihood. The log-likelihood for independent observations is easier to work with because observations enter additively (sum) rather than multiplicatively (product) on the log scale. And it's easier to calculate the derivative of a sum than the derivative of a product.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1277,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":109,"value":"
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n\n---\n#### 🙈 Appendix 🙈\n\nHär följer bara kod och detaljer som du antagligen inte vill läsa.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":221,"value":"#### Imports","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":26,"value":"jstat = require('jstat')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":877,"value":"mathjs = require('mathjs')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1231,"value":"#### Dolda grafer som visas ovan","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":780,"value":"pltdata = Plot.plot({\n x: {\n label: \"x\",\n axis: true,\n domain: [-0.5,1.5],\n ticks: [0,1],\n tickFormat: \".0\"\n },\n y: {\n label: \"P(X=x)\",\n domain: [0, 1.05]\n },\n marks: [\n Plot.ruleY([0]),\n Plot.ruleY([s/n],{stroke: \"#B85450\"}),\n Plot.text([{x: 1.45, y: s/n}], {x: \"x\", y: \"y\", text: [\"s/n\"], dx: 0, dy: -7, fill: \"#B85450\"}),\n Plot.ruleX([-0.5]),\n Plot.ruleX([{x: 0, y: 1-theta},{x: 1, y: theta}], {x: \"x\", y: \"y\", dx:-5, stroke: \"steelblue\", strokeWidth: 2}), \n Plot.dot([{x: 0, y: 1-theta},{x: 1, y: theta}], {x: \"x\", y: \"y\", dx:-5, fill: \"steelblue\", r: 5}),\n Plot.text([{x: 0, y: 1-theta},{x: 1, y: theta}], {x: \"x\", y: \"y\", text: \"y\", dy: -15}),\n Plot.ruleX([{x: 0, y: 1-mle},{x: 1, y: mle}], {filter: d => settings[1], x: \"x\", y: \"y\", dx:+5, stroke: \"#5F9EA0\", strokeWidth: 2}),\n Plot.dot([{x: 0, y: 1-mle},{x: 1, y: mle}], {filter: d => settings[1], x: \"x\", y: \"y\", dx:+5, fill: \"#5F9EA0\", r: 5}),\n ]\n })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1062,"value":"pltlikelihood = Plot.plot({\n y: {label: \"average probability for single observation\"},\n x: {label: \"theta\"},\n marks: [\n Plot.ruleX([mle], {filter: d => settings[1], stroke: \"#5F9EA0\", strokeWidth: 2}),\n Plot.ruleY([0]),\n Plot.ruleX([0]),\n Plot.lineY(meanprob_grid, {x: \"theta\", y: \"geomean\", stroke: \"steelblue\", strokeWidth: 3}),\n Plot.dot([{x: theta, y: geomean_like(s, n, theta)}], {x: \"x\", y: \"y\", fill: \"orange\", marker: \"circle\", r: 7})\n ]\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":223,"value":"#### Data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1881,"value":"theta = settings[0]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":17,"value":"binomialpmf = {\n const x = d3.range(0, n, 1);\n const data = x.map(x => ({x: x, pdf: jstat.binomial.pdf(x, n, theta)}));\n return data\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":978,"value":"binomialpmf_mle = {\n const x = d3.range(0, n, 1);\n const data = x.map(x => ({x: x, pdf: jstat.binomial.pdf(x, n, mle)}));\n return data\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":981,"value":"mle = s/n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1327,"value":"meanprob_grid = {\n const thetaGrid = d3.range(0.001, 0.999, 0.001);\n const data2 = thetaGrid.map(theta => ({theta: theta, geomean: geomean_like(s, n, theta)}));\n return data2;\n}\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":961,"value":"#### Functions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":849,"value":"function geomean_like(s, n, theta){\n const likelihood = theta**s*(1-theta)**(n-s);\n return Math.pow(likelihood, 1/n)\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":229,"value":"#### Style","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":202,"value":"","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":58,"value":"viewof md_colour = Inputs.color({label: \"Highlight color cells\", value: \"#F5F5DC\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 30:"$2f:body"