24:null 25:["$","title",null,{"children":"ML | 3 Linear Classification 3.2 Logistic Regression Naive Bayesian Gaussian Discriminant Analysis (GDA) | Zhihao Yao | Observable"}] 2a:T5a2,${tex.block` \left.\begin{array}{c}p_{1}=P(y=1 \mid x)=\sigma\left(w^{T} x\right)=\frac{1}{1+e^{-w^{T} x}}=\varphi(x ; w) \\~\\ p_{0}=P(y=0 \mid x)=1-P(y=1 \mid x)=\frac{e^{-w^{T} x}}{1+e^{-w^{T} x}}=1-\varphi(x ; w)\end{array}\right\} p(y \mid x)=p_{1}^{y} p_{0}^{1-y} \\~\\ `} > *Maximum Likelihood Estimation*: ${tex.block` \hat{w}=\underset{w}{\operatorname{argmax}} \log \underbrace{P(Y \mid X)}_{\text {likelihood }} \\~\\ =\underset{w}{\operatorname{argmax}} \log \prod_{i=1}^{N} P\left(y_{i} \mid x_{i}\right) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N} \log P\left(y_{i} \mid x_{i}\right) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N}\left(y_{i} \log p_{1}+\left(1-y_{i}\right) \log p_{0}\right) \\~\\ =\underset{w}{\operatorname{argmax}} \underbrace{\sum_{i=1}^{N}\left(y_{i} \log \varphi(x ; w)+\left(1-y_{i}\right) \log (1-\varphi(x ; w))\right.}_{-\text {cross entropy }} \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N} \left[ y_i \log \frac{\varphi(x ; w)}{1 - \varphi(x ; w)} + \log (1 - \varphi(x ; w)) \right] \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N} \left[y_i (w^T x) - \log (1 + e^{(w^T x)} ) \right] \\~\\ `} > Maximum Likelihood Estimation ${tex`\Longleftrightarrow`} minimizing the cross-entropy

> Solve by gradient descent: ${tex.block` \frac{\partial L}{\partial w}=-\sum_{i=1}^n (y_i-\varphi(x_i))x_i^* \\~\\ w:=w-\eta \frac{\partial L}{\partial w} `}2b:T1d24,

## Gaussian Discriminant Analysis (GDA)
> In Gaussian discriminant analysis the class ${tex`y`} is Bernoulli distributed, and the sample data in each class is multivariate Gaussian distributed:
${tex.block` \left.y \sim \text { Bernoulli }(\phi) \Leftrightarrow \begin{array}{c} \phi^{y}, y=1 \\~\\ (1-\phi)^{1-y}, y=0 \end{array}\right\} P(y)=\phi^{y} \phi^{1-y} \\~\\ \left.\begin{array}{l} x \mid y=1 \sim N\left(\mu_{1}, \Sigma\right) \\~\\ x \mid y=0 \sim N\left(\mu_{2}, \Sigma\right) \end{array}\right\} P(x \mid y)=N\left(\mu_{1}, \Sigma\right)^{y} N\left(\mu_{2}, \Sigma\right)^{1-y} \\~\\ `} We assume the two Gaussian distributions have the same covariance matrix.

**Loss function**
MLE to estimate the parameter ${tex`\theta \left(\mu_{1}, \mu_{2}, \Sigma, \phi\right)`}
${tex.block` L(\theta)=\log \prod_{i=1}^{N} P\left(x_{i}, y_{i}\right) \\~\\ =\sum_{i=1}^{N} \log P\left(x_{i}, y_{i}\right) \\~\\ =\sum_{i=1}^{N} \log P\left(x_{i} \mid y_{i}\right) P\left(y_{i}\right) \\~\\ =\sum_{i=1}^{N}\left[\log P\left(x_{i} \mid y_{i}\right)+\log P\left(y_{i}\right)\right] \\~\\ =\sum_{i=1}^{N}\left[\log N\left(\mu_{1}, \Sigma\right)^{y_{i}} N\left(\mu_{2}, \Sigma\right)^{1-y_{i}}+\log \phi^{y_{i}} \phi^{1-y_{i}}\right] \\~\\ =\sum_{i=1}^{N}[\underbrace{\log N\left(\mu_{1}, \Sigma\right)^{y_{i}}}_{(1)}+\underbrace{\log N\left(\mu_{2}, \Sigma\right)^{1-y_{i}}}_{(2)}+\underbrace{\log \phi^{y_{i}} \phi^{1-y_{i}}}_{(3)}] \\~\\ \hat{\theta}=\underset{\theta}{\operatorname{argmax}} L(\theta) `}

> **To solve ${tex`\phi`}**
>> The number of samples with label = 1 is ${tex`N_1`}, The number of samples with label = 0 is ${tex`N_2`}. Therefore, ${tex`N_1 + N_2 = N`}
${tex.block` \text { (3) }=\sum_{i=1}^{N}\left[y_{i} \log \phi+\left(1-y_{i}\right) \log (1-\phi)\right] \\~\\ \frac{\partial(3)}{\partial \phi}=\sum_{i=1}^{N}\left[y_{i} \frac{1}{\phi}-\left(1-y_{i}\right) \frac{1}{1-\phi}\right]=0 \\~\\ \Rightarrow \sum_{i=1}^{N}\left[y_{i}(1-\phi)-\left(1-y_{i}\right) \phi\right]=0 \\~\\ \Rightarrow \sum_{i=1}^{N}\left(y_{i}-\phi\right)=0 \\~\\ \Rightarrow \sum_{i=1}^{N} y_{i}-N \phi=0 \\~\\ \hat{\phi}=\frac{1}{N} \sum_{i=1}^{N} y_{i}=\frac{N_{1}}{N} \\~\\~\\ `}
> **To solve ${tex`\mu_1, \mu_2`}**
${tex.block` \text { (1) }=\sum_{i=1}^{N} y_{i} \log \frac{1}{(2 \pi)^{p / 2}|\Sigma|^{1 / 2}} \exp \left\{-\frac{1}{2}\left(x_{i}-\mu_{1}\right)^{T} \Sigma^{-1}\left(x_{i}-\mu_{1}\right)\right\} \\~\\ \mu_{1}=\underset{\mu_{1}}{\arg \max }(1) \\~\\ =\underset{\mu_{1}}{\operatorname{argmax}} \sum_{i=1}^{N} y_{i}\left[\log \frac{1}{(2 \pi)^{p / 2}|\Sigma|^{1 / 2}}-\frac{1}{2}\left(x_{i}-\mu_{1}\right)^{T} \Sigma^{-1}\left(x_{i}-\mu_{1}\right)\right] \\~\\ =\underset{\mu_{1}}{\operatorname{argmax}} \sum_{i=1}^{N} y_{i}\left[-\frac{1}{2}\left(x_{i}-\mu_{1}\right)^{T} \Sigma^{-1}\left(x_{i}-\mu_{1}\right)\right] \\~\\ =\underset{\mu_{1}}{\operatorname{argmax}} \Delta \\~\\ \Delta=\sum_{i=1}^{N} y_{i}\left[-\frac{1}{2}\left(x_{i}-\mu_{1}\right)^{T} \Sigma^{-1}\left(x_{i}-\mu_{1}\right)\right] \\~\\~\\ =-\frac{1}{2} \sum_{i=1}^{N} y_{i}\left(x_{i}^{T} \Sigma^{-1}-\mu_{1}^{T} \Sigma^{-1}\right)\left(x_{i}-\mu_{1}\right) \\~\\ =-\frac{1}{2} \sum_{i=1}^{N} y_{i}\left(x_{i}^{T} \Sigma^{-1} x_{i}-2 \mu_{1}^{T} \Sigma^{-1} x_{i}+\mu_{1}^{T} \Sigma^{-1} \mu_{1}\right) \\~\\ \frac{\partial \Delta}{\partial \mu_{1}}=-\frac{1}{2} \sum_{i=1}^{N} y_{i}\left(-2 \Sigma^{-1} x_{i}+2 \Sigma^{-1} \mu_{1}\right)=0 \\~\\ \Rightarrow \sum_{i=1}^{N} y_{i}\left(\Sigma^{-1} \mu_{1}-\Sigma^{-1} x_{i}\right)=0 \\~\\ \Rightarrow \sum_{i=1}^{N} y_{i}\left(\mu_{1}-x_{i}\right)=0 \\~\\ \Rightarrow \sum_{i=1}^{N} y_{i} \mu_{1}=\sum_{i=1}^{N} y_{i} x_{i} \\~\\ \hat{\mu_{1}}=\frac{\sum_{i=1}^{N} y_{i} x_{i}}{\sum_{i=1}^{N} y_{i}}=\frac{\sum_{i=1}^{N} y_{i} x_{i}}{N_{1}}\\~\\ `} >> Similarly ${tex.block` \hat{\mu_{2}}=\frac{\sum_{i=1}^{N} y_{i} x_{i}}{N_{2}} `}
> **To solve ${tex`\Sigma`}**
>> linear algebra prerequisites: ${tex` \operatorname{tr}(A)`} refers to the trace of matrix

${tex.block` \frac{\partial \operatorname{tr}(A B)}{\partial A}=B^{T} \\~\\ \frac{\partial|A|}{\partial A}=|A| A^{-1} \\~\\ \operatorname{tr}(A B)=\operatorname{tr}(B A) \\~\\ \operatorname{tr}(A B C)=\operatorname{tr}(C A B)=\operatorname{tr}(B C A) \\~\\ `} >> The two classes of data are represented as two sets:
${tex.block` C_{1} =\left\{x_{i} \mid y_{i}=1, i=1,2, \cdots, N\right\} \\~\\ C_{2} =\left\{x_{i} \mid y_{i}=0, i=1,2, \cdots, N\right\} \\~\\ \left|C_{1}\right| =N_{1},\left|C_{2}\right|=N_{2}, N_{1}+N_{2}=N \\~\\~\\ `} >> ${tex`\hat{\Sigma}=\underset{\Sigma}{\operatorname{argmax}}((1)+(2)) \\~\\`} > ${tex.block` (1)+(2)=\sum_{x_{i} \in C_{1}} \log N\left(\mu_{1}, \Sigma\right)+\sum_{x_{i} \in C_{2}} \log N\left(\mu_{2}, \Sigma\right)\\~\\ \sum_{i=1}^{N} \log N(\mu, \Sigma)=\sum_{i=1}^{N} \log \frac{1}{(2 \pi)^{p / 2}|\Sigma|^{1 / 2}} \exp \left\{-\frac{1}{2}\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\left(x_{i}-\mu\right)\right\} \\~\\ =\sum_{i=1}^{N}\left[\log \frac{1}{(2 \pi)^{p / 2}}+\log |\Sigma|^{-\frac{1}{2}}-\frac{1}{2}\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\left(x_{i}-\mu\right)\right] \\~\\ =\sum_{i=1}^{N}\left[C-\frac{1}{2} \log |\Sigma|-\frac{1}{2}\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\left(x_{i}-\mu\right)\right] \\~\\ =C-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} \sum_{i=1}^{N}\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\left(x_{i}-\mu\right) \\~\\ =C-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} \sum_{i=1}^{N} \operatorname{tr}\left[\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\left(x_{i}-\mu\right)\right] \\~\\ \text { (The trace of a real number is equal to itself) } \\~\\ =C-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} \sum_{i=1}^{N} \operatorname{tr}\left[\left(x_{i}-\mu\right)\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\right] \\~\\ =C-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} \operatorname{tr}\left[\sum_{i=1}^{N}\left(x_{i}-\mu\right)\left(x_{i}-\mu\right)^{T} \Sigma^{-1}\right] \\~\\ =C-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} \operatorname{tr}\left(N S \Sigma^{-1}\right) \\~\\ \left(S=\frac{1}{N} \sum_{i=1}^{N}\left(x_{i}-\mu\right)\left(x_{i}-\mu\right)^{T}, \text { Covariance matrix }\right) \\~\\ =-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} N \operatorname{tr}\left(S \Sigma^{-1}\right)+C \\~\\ \text { (1) + (2)=- } \frac{1}{2} N_{1} \log |\Sigma|-\frac{1}{2} N_{1} \operatorname{tr}\left(S_{1} \Sigma^{-1}\right)-\frac{1}{2} N_{2} \log |\Sigma|-\frac{1}{2} N_{2} \operatorname{tr}\left(S_{2} \Sigma^{-1}\right)+C \\~\\ =-\frac{1}{2} N \log |\Sigma|-\frac{1}{2} N_{1} \operatorname{tr}\left(S_{1} \Sigma^{-1}\right)-\frac{1}{2} N_{2} \operatorname{tr}\left(S_{2} \Sigma^{-1}\right)+C \\~\\ =-\frac{1}{2}\left[N \log |\Sigma|+N_{1} \operatorname{tr}\left(S_{1} \Sigma^{-1}\right)+N_{2} \operatorname{tr}\left(S_{2} \Sigma^{-1}\right)\right]+C \\~\\~\\ \frac{\partial(1)+(2)}{\partial \Sigma} =-\frac{1}{2}\left[N \frac{1}{|\Sigma|}|\Sigma| \Sigma^{-1}+N_{1} \frac{\partial \operatorname{tr}\left(\Sigma^{-1} S_{1}\right)}{\partial \Sigma}+N_{2} \frac{\partial \operatorname{tr}\left(\Sigma^{-1} S_{2}\right)}{\partial \Sigma}\right] \\~\\ =-\frac{1}{2}\left[N \Sigma^{-1}+N_{1} S_{1}^{T}(-1) \Sigma^{-2}+N_{2} S_{2}^{T}(-1) \Sigma^{-2}\right] \\~\\ =-\frac{1}{2}\left(N \Sigma^{-1}-N_{1} S_{1}^{T} \Sigma^{-2}-N_{2} S_{2}^{T} \Sigma^{-2}\right)=0 \\~\\ N \Sigma-N_{1} S_{1}-N_{2} S_{2}=0 \\~\\ \hat{\Sigma}=\frac{N_{1} S_{1}+N_{2} S_{2}}{N} `}2c:Tf16,data3_2_1 = d3.csvParse(`x1,x2,y 34.62365962451697,78.0246928153624,0 30.28671076822607,43.89499752400101,0 35.84740876993872,72.90219802708364,0 60.18259938620976,86.30855209546826,1 79.0327360507101,75.3443764369103,1 45.08327747668339,56.3163717815305,0 61.10666453684766,96.51142588489624,1 75.02474556738889,46.55401354116538,1 76.09878670226257,87.42056971926803,1 84.43281996120035,43.53339331072109,1 95.86155507093572,38.22527805795094,0 75.01365838958247,30.60326323428011,0 82.30705337399482,76.48196330235604,1 69.36458875970939,97.71869196188608,1 39.53833914367223,76.03681085115882,0 53.9710521485623,89.20735013750205,1 69.07014406283025,52.74046973016765,1 67.94685547711617,46.67857410673128,0 70.66150955499435,92.92713789364831,1 76.97878372747498,47.57596364975532,1 67.37202754570876,42.83843832029179,0 89.67677575072079,65.79936592745237,1 50.534788289883,48.85581152764205,0 34.21206097786789,44.20952859866288,0 77.9240914545704,68.9723599933059,1 62.27101367004632,69.95445795447587,1 80.1901807509566,44.82162893218353,1 93.114388797442,38.80067033713209,0 61.83020602312595,50.25610789244621,0 38.78580379679423,64.99568095539578,0 61.379289447425,72.80788731317097,1 85.40451939411645,57.05198397627122,1 52.10797973193984,63.12762376881715,0 52.04540476831827,69.43286012045222,1 40.23689373545111,71.16774802184875,0 54.63510555424817,52.21388588061123,0 33.91550010906887,98.86943574220611,0 64.17698887494485,80.90806058670817,1 74.78925295941542,41.57341522824434,0 34.1836400264419,75.2377203360134,0 83.90239366249155,56.30804621605327,1 51.54772026906181,46.85629026349976,0 94.44336776917852,65.56892160559052,1 82.36875375713919,40.61825515970618,0 51.04775177128865,45.82270145776001,0 62.22267576120188,52.06099194836679,0 77.19303492601364,70.45820000180959,1 97.77159928000232,86.7278223300282,1 62.07306379667647,96.76882412413983,1 91.56497449807442,88.69629254546599,1 79.94481794066932,74.16311935043758,1 99.2725269292572,60.99903099844988,1 90.54671411399852,43.39060180650027,1 34.52451385320009,60.39634245837173,0 50.2864961189907,49.80453881323059,0 49.58667721632031,59.80895099453265,0 97.64563396007767,68.86157272420604,1 32.57720016809309,95.59854761387875,0 74.24869136721598,69.82457122657193,1 71.79646205863379,78.45356224515052,1 75.3956114656803,85.75993667331619,1 35.28611281526193,47.02051394723416,0 56.25381749711624,39.26147251058019,0 30.05882244669796,49.59297386723685,0 44.66826172480893,66.45008614558913,0 66.56089447242954,41.09209807936973,0 40.45755098375164,97.53518548909936,1 49.07256321908844,51.88321182073966,0 80.27957401466998,92.11606081344084,1 66.74671856944039,60.99139402740988,1 32.72283304060323,43.30717306430063,0 64.0393204150601,78.03168802018232,1 72.34649422579923,96.22759296761404,1 60.45788573918959,73.09499809758037,1 58.84095621726802,75.85844831279042,1 99.82785779692128,72.36925193383885,1 47.26426910848174,88.47586499559782,1 50.45815980285988,75.80985952982456,1 60.45555629271532,42.50840943572217,0 82.22666157785568,42.71987853716458,0 88.9138964166533,69.80378889835472,1 94.83450672430196,45.69430680250754,1 67.31925746917527,66.58935317747915,1 57.23870631569862,59.51428198012956,1 80.36675600171273,90.96014789746954,1 68.46852178591112,85.59430710452014,1 42.0754545384731,78.84478600148043,0 75.47770200533905,90.42453899753964,1 78.63542434898018,96.64742716885644,1 52.34800398794107,60.76950525602592,0 94.09433112516793,77.15910509073893,1 90.44855097096364,87.50879176484702,1 55.48216114069585,35.57070347228866,0 74.49269241843041,84.84513684930135,1 89.84580670720979,45.35828361091658,1 83.48916274498238,48.38028579728175,1 42.2617008099817,87.10385094025457,1 99.31500880510394,68.77540947206617,1 55.34001756003703,64.9319380069486,1 74.77589300092767,89.52981289513276,1 `, ({x1, x2, y})=>({ x1: +x1, x2: +x2, y: +y }))2d:T41a,// train = { // const X = math.matrix(data3_2_1.map(d=>[1, d.x1, d.x2])); // const y = math.matrix(data3_2_1.map(d=>d.y)); // const w0 = math.matrix(new Array(X.size()[1]).fill(0)); // return {X:X, y:y, w0:w0} // } // sigmoid = (z) => { // return 1 / (1 + math.exp(-z)) // } // lossFunction = (w, X, y) => { // const m = y.size()[0]; // const h = math.matrix(math.multiply(X, w).toArray().map(d=>sigmoid(d))); // const J = // ( // math.multiply( math.transpose(math.log(h)), y ) + // math.multiply( math.transpose(math.log(math.subtract(math.transpose(math.matrix(new Array(h.size()[0]).fill(1))) , h))), math.subtract(math.transpose(math.matrix(new Array(y.size()[0]).fill(1))) , y) ) // ) / m * (-1.0); // return isNaN(J) ? Infinity : J; // } // gradient = (w, X, y) => { // const m = y.size()[0]; // const h = math.matrix(math.multiply(X, w).toArray().map(d=>sigmoid(d))); // const grad = (math.multiply(math.transpose(X), math.subtract(h, y))).toArray().map(d=>d/m); // return grad; // }2e:[] 28:{"id":"250411f4e912c688","slug":"ml-3-linear-classification-3-2","title":"ML | 3 Linear Classification 3.2 Logistic Regression Naive Bayesian Gaussian Discriminant Analysis (GDA)","thumbnail":"5f2be07d39eeae406a7de71be6f1c1aa54662c2b5c445e8fe8fded66e03b90d1","access_level":"public","create_time":"$D2022-03-29T19:11:19.113Z","update_time":"$D2022-11-02T11:51:46.353Z","view_time":null,"first_public_time":"$D2022-03-31T14:38:10.141Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"610db6fe9b10a76c","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":"starter_2024"},"version":564,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"01b5d984f2428b1a","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":null}],"members":[{"type":"individual","id":"01b5d984f2428b1a","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":{"id":"102982dc3961aa5a","slug":"machine-learning-notes","title":"Machine Learning Notes","owner":{"id":"610db6fe9b10a76c","login":"howardyao","name":"Zhihao Yao"}},"type":"notebook","body":{"title":"ML | 3 Linear Classification 3.2 Logistic Regression Naive Bayesian Gaussian Discriminant Analysis (GDA)","cells":[{"id":0,"value":"# ML | 3 Linear Classification

3.2
Logistic Regression
Naive Bayesian
Gaussian Discriminant Analysis (GDA)\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10,"value":"> ${tex`\n \\large \\text{Linear Regression} \\xRightarrow[\\text{function}]{\\text{activation}} \\text{Linear Classification} \\\\~\\\\\n`}\n\n> ${tex`\n\\ y=f\\left(w^{T} x+b\\right), y \\in\\left\\{\\begin{array}{l}\n\\{0,1\\}, \\text { Hard classification (decision boundary) } \\\\\n{[0,1], \\text { Soft classification (probability)}}\n\\end{array}\\right. \\\\~\\\\\n\n\\text { Activation function } f: w^{T} x+b \\mapsto\\{0,1\\} /[0,1] \\\\~\\\\\n\\text { Link function } f^{-1} : \\{0,1\\} /[0,1] \\mapsto w^{T} x+b \\\\~\\\\\n`}\n\n

\n\n> ${tex`\n\\text{Linear Classification} \\left\\{\\begin{array}{l}\n\\text {Hard classification} \n \\left\\{\\begin{array}{l}\n \\text {Linear Discriminant Analysis (LDA)} \\\\\n \\text {Perceptron} \n \\end{array}\\right. \\\\~\\\\\n {\\color {steelblue} \\text { Soft classification}}\n \\left\\{\\begin{array}{l}\n {\\color {steelblue} \\text {Gaussian Discriminant Analysis (GDA)}} \\\\\n {\\color {steelblue} \\text {Naive Bayesian}} \\\\\n {\\color {steelblue} \\text {Logistic Regression}}\n \\end{array}\\right. \\\\\n\\end{array}\\right.\n`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":14,"value":"

\n## Logistic Regression\n
\n> Logistic regression is a binary classification algorithm. The linear combination ${tex`w^T x`} is compressed to between 0 and 1 by a sigmoid activation function to represent the probability of belonging to a particular classification.

\n> Data:\n${tex.block`\\left\\{\\left(x_{i}, y_{i}\\right)\\right\\}_{i=1}^{N}, x_{i} \\in \\mathbb{R}^{p}, y_{i} \\in\\{0,1\\}`}\n
\n\n> **Sigmoid activation function**\n> ${tex.block`\\sigma(z)=\\frac{1}{1+e^{-z}} \\\\`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":29,"value":"{\n let w = 700;\n let p = plotter({\n // xDomain: [-5, 5],\n yDomain: [1, 0],\n grid: false,\n width: w,\n height: w / 2,\n xLabel: '',\n yLabel: '',\n yTickCount: 8,\n xTickCount: 5,\n });\n p.func(x => 1 / (1 + Math.exp(-x)), { stroke: 'grey', width: 2});\n return p.node;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":42,"value":"$2a","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":163,"value":"> In logistic regression, we use the sigmoid function and perform a non-linear transformation to obtain the probabilities. Squaring this non-linear transformation will lead to non-convexity with local minimums. Finding the global minimum in such cases using gradient descent is not possible. Due to this reason, MSE is not suitable for logistic regression. Cross-entropy or log loss is used as a cost function for logistic regression.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":213,"value":"> **Example**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":525,"value":"viewof pars3_2_1= Inputs.form({\n iter: Inputs.range([0, lr_results.get('grad_all').length-1], {step: 1, value: 0, label: 'Iteration'}),\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":536,"value":"htl.html`Cost Function: ${lr_results.get('cost_all')[pars3_2_1.iter].toFixed(5)}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":482,"value":"plot_3_2_1 = {\n const w0 = lr_results.get('grad_all')[pars3_2_1.iter][0];\n const w1 = lr_results.get('grad_all')[pars3_2_1.iter][1];\n const w2 = lr_results.get('grad_all')[pars3_2_1.iter][2];\n\n const xmin = d3.min(data3_2_1, d=>d.x1) > 0 ? 0 : d3.min(data3_2_1, d=>d.x1);\n const xmax = d3.max(data3_2_1, d=>d.x1);\n const ymin = d3.min(data3_2_1, d=>d.x2) > 0 ? 0 : d3.min(data3_2_1, d=>d.x2) ;\n const ymax = d3.max(data3_2_1, d=>d.x2);\n const margin = 5\n \n const p = plotter({\n xDomain: [xmin-margin, xmax+margin],\n yDomain: [ymax+margin, ymin-margin],\n xLabel:'x1', \n yLabel:'x2',\n grid: false,\n });\n \n for (const v of data3_2_1) {\n const colorCode = v.y === 1 ? 'orange' : 'green';\n p.circle(v.x1, v.x2, 1, \n { fill: colorCode, stroke: colorCode}\n )\n }\n \n p.func(t => (w2/w1)*t*(-1.0) + (-1.0)*(w0/w1), \n { stroke: 'blue', width: 1 });\n\n return p.node;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":555,"value":"train = {\n const X = data3_2_1.map(d=>[1, d.x1, d.x2]);\n const y = data3_2_1.map(d=>d.y);\n\n return {X:X, y:y}\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":558,"value":"lr = py`import numpy as np\nimport numpy as np\nfrom scipy.optimize import minimize\n\ndef sigmoid(z):\n return(1 / (1 + np.exp(-z)))\n\ndef costFunction(theta, X, y):\n m = y.size\n h = sigmoid(X.dot(theta))\n \n J = -1.0*(1.0/m)*(np.log(h).T.dot(y)+np.log(1-h).T.dot(1-y))\n \n if np.isnan(J[0]):\n return(np.inf)\n return J[0]\n\ndef gradient(theta, X, y):\n m = y.size\n h = sigmoid(X.dot(theta.reshape(-1,1)))\n \n grad =(1.0/m)*X.T.dot(h-y)\n\n return(grad.flatten())\n\ndef run(X, y):\n\n X = np.array(X)\n y = np.array(y) \n\n initial_theta = np.zeros(X.shape[1])\n cost = costFunction(initial_theta, X, y)\n grad = gradient(initial_theta, X, y)\n \n grad_all = [grad]\n cost_all = [cost]\n def history(x):\n grad_all.append(x)\n cost_all.append(costFunction(x, X, y))\n \n res = minimize(costFunction, initial_theta, args=(X,y), jac=gradient, callback=history, options={'maxiter':400, 'disp':True})\n\n return {'res':res, 'grad_all':grad_all, 'cost_all':cost_all}\n\nrun`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":61,"value":"

\n## Naive Bayesian\n
\n> The basic assumption of Naive Bayesian is each feature is independent:\n${tex.block`\nP(X =x | Y=c_{k} )=P\\left(X^{(1)}=x^{(1)}, \\cdots, X^{(n)}=x^{(n)} | Y=c_{k}\\right) \\\\~\\\\ =\\prod_{j=1}^{n} P\\left(X^{(j)}=x^{(j)} | Y=c_{k}\\right)\n`}\n
\nGiven ${tex`x`}, the category of ${tex`y`} can be determined by the category with the highest probability:\n${tex.block`\n\\hat{y}=\\underset{y}{\\operatorname{argmax}} P(y \\mid x) \\\\~\\\\\n=\\underset{y}{\\operatorname{argmax}} \\frac{P(x, y)}{P(x)} \\\\~\\\\\n=\\operatorname{argmax} \\frac{P(y) P(x \\mid y)}{P(x)}\n`}","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":71,"value":"$2b","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":32,"value":"

\n## Appendix","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":435,"value":"lr_results = py`import pandas as pd\n${lr}(pd.read_json(${JSON.stringify(train.X)}),pd.read_json(${JSON.stringify(train.y)}))`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":185,"value":"$2c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"import { plotter } from '50dadfdec01c15a8'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":201,"value":"math = require('https://cdn.jsdelivr.net/npm/mathjs@10.5.1/lib/browser/math.min.js')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":422,"value":"import {py, pyodide} from \"@howardyao/machine-learning-notes-2\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":412,"value":"$2d","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W2e","stdlib":"1","titling":"auto"}} 29:"$28:body"