2d:null 2e:["$","title",null,{"children":"[AML] Applied Machine Learning | T.E. Schouten | Observable"}] 33:T487,md`#### *Finding optimal value: gradient descent* Another way to find an (sometimes sub-)optimal value, is by *gradient descent*. > ${tex` \begin{aligned} \text{new weights} &= \text{old weights} - \underbrace{\text{step size}}_{\text{learning rate * slope}} \\ w_{t+1} &= w_t - \gamma \frac{d}{d w_t} J (w_t) \\ \end{aligned}`} - We start with random weights ${tex`w_0`}. - With each step, we compute a derivate and update ${tex`w`} - We take steps into the right direction ***TO DO *** *Pros:* - *scales relatively well* - *highly optimized* *Cons:* - *Memory* - *Slow convergence* - *Not guaranteed to find the optimal value* #### *Stochastic gradient descent* However, with fe 1e6 inputs, this becomes rather slow, and there's a chance of overfitting the model. A better type of gradient descent is *batch gradient descent*; bgd takes only a batch of inputs in every step, and uses this batch to compute the gradient. *Pros:* - *Prevents overfitting by generalization* - *Faster! Because it uses fewer entries.* - *Memory efficient* *Cons:* - *Requires tuning learning rate* - *Optimalization techniques are less advanced* `34:T551,anchor_generalization = md` ---- ## Generalization (underfitting and overfitting) The real goal is not the objective function, but to generalize. A perfect score on our loss function, doesn't necessarily mean the model is good at predicting new values. Say we have a polynomial regression model: > ${tex`t = f_n(x,w) = x^tw + (x^2)^T+...+(x^n)^Tw`} If we try to fit a linear model to a complex dataset, we probably underfit. If we use a high order polynomial, we might overfit. How do we find the optimal polynomial degree? - We can use weight decay - We can add more data(!). This significantly lowers the chance of overfitting. - We train the model on our training set, and evaluate on the validating set. #### *Weight decay* We add a penalty (hyperparam lambda) to the learning objective to prevent overfitting. > ${tex`\underbrace{\frac{1}{2} \sum{(t_n - {x_n}^T}w)^2}_{\text{still minimize this loss}} +\underbrace{\lambda R(w)}_{\text{penalty} \sum_dw_d^2}`} The derivative of the penalty is: > ${tex`\lambda 2 w`} By changing ${tex `\lambda`} we influence how much we want to penalize the weights. We chose to square the weights because this makes the values positive (-10 and +10 are equally 'bad'), and also because it extra-penalizes high values: (a small change in the input leads to a huge change in the output, we can restrict this.) `35:T5e8,anchor_ensembly = md`## Combining models Note that combining models is key to get the best outcomes (though it may not scale well). Random forest classification is a nice example of using various models to improve the outcomes of a model. #### *Bootstrap sampling* For a trainingset D containing N examples, create a training subset D' by drawing N examples at random with replacement from D. #### *Bagging* 1. Create K bootstrap subsets 2. Train a classifier for each subset 3. Classify test example by majority vote over subset classifiers #### *Boosting (Adaptive Boosting -> AdaBoost)* In standard bagging/ensembling, models are trained independently. The goal of boosting is to 'learn' from the errors of previous models. 1. Attach a weight to each training example 2. Select a "weak classifier" with the lowest training error 3. Increate weight of incorrectly classified training examples 4. Select new best weak classifier with lower weight

One of the nice properties of AdaBoost is that it is not very sensitive to overfitting. An improvement in the training set usually means an improvement on the test set as well. However, a too-strong classifier leads to no training errors, and hence no benefits. A too-weak classifier might not converge to an answer / iterate too long. #### *In practise..* Advantage of standard ensembling over boosting: it can be run in parralel. ` 36:T500,md` #### *Training goal and overview* Good resource: https://hmkcode.com/ai/backpropagation-step-by-step/ A neural network consists of inputs, outputs, weights and biases. The goal is to find values for the weights and biases, so the predicted output (ŷ) is closest to the ground truth (y). We learn the weights and biases through forward-backward propagation: 1. Forward: map input to predicted output 2. Loss: compare predicted output to ground truth 3. Backward: correct weights and biases by propagating gradients #### *Forward propagation / Forward pass* Start from input, multiply with weights, sum, add bias. Repeat for all layers. The result is a linear function. To add non-linearity, an activation like Sigmoid or ReLU function is added. ReLU: > ${tex`\begin{aligned} R(z) &= \max (0,z)\\ \frac{d}{dz} R(z) &= 1 \text{ if z} \geq 0 \text{ else } 0 \end{aligned}` } They both have their pro's: | Sigmoid | ReLU | |------------------------- |------------------------------------------ | | Bounded between 0 and 1 | Stronger gradient signal (in backprop) | | 'Sexy' math | Easier to implement | | | More non-linear | `37:T92c,md`## Representing Text ### *Main ML applications on Textual Data* - Text clustering → Grouping texts that are topic-related (unsupervised) - Text classification → Extracting text sentiment or topic. An application is ranking and question answering. - Digital assistance → Meaningful communicate with user. Application: Siri/Cortana/Alexa/Google Now. ### *How do we go from language to information?* #### 1. Lexical representation of text ##### Bag of words Categorical features can be one-hot-encoded. We can do this with words/sentences/documents as well. We have a list of words (corpus) which can either be the entire dictionary or a subset. A word can then be represented with [0 0 0 ... 1 ... 0 ]. Where every bit is a word in the corpus. Drawback: the word has to be in the corpus! Vocabulary (corpus): {and, ate, cat, dog, hat, on, sat, the} "cat" → {0, 0, 1, 0, 0, 0, 0, 0} "the cat sat on the hat" → {0, 0, 1, 0, 1, 1, 1, 2} "the dog ate the cat and the hat" → {1, 1, 1, 1, 1, 0, 0, 3} ##### Document frequency [lecture 6.1, ~36:00] This 'bag of words' approach does not quantify the importance of words. One way to solve this, is by making using of **document frequency**. - Terms that occur in few documents are more informative - Terms that occur in many documents are less informative Definition: > ${tex`df(t) := \red{\text{TO DO}} `} Translate these document frequencies into term weights is commonly done by counting the log of the inverse of the doc frequency (IDF). [The less frequently we see a word, the more informative it is] > ${tex`idf(t) = \log \frac{n}{df(t)} = \log n - \log df(t) \\ \small n: \text{number of documents} \\ t: \text{term} \\ df(t): \text{term frequency}`} ##### Tokenization Forming words from a sequence of characters. This is harder than it seems: Just splitting by whitespace is often not enough - Word combinations: World War II → should be one entry from the corpus - Hyphenation: e-bay → ebay? - Numbers: quicktime 6.5 pro → one entry again - Periods: Don't always identify the end of a sentence. ##### Stemming Finds the 'root' of a word. For example: books → book. Two basic types: - Dictionary-ased: uses a list of related words - Algorithmic: uses program to determine related words (fe naively removing the s from plural forms) `38:T604,anchor_cnn = md` ## Representing Images (Convolutional Neural Networks) [Resource](https://cs231n.github.io/convolutional-networks/) Using an image as input for our neural network results in a few problems: - Too many params to learn. WxHx3 (excl. bias) - We lose spatial information (as its one vector) - Creating a neuron for R/G/B/ values loses the coherence (RGB shouldnt be interpreted naively). One way to solve this problem is to use convolutions and pooling. #### *Convolutions* Transform datapoints (fe an image) using a kernel that also looks at neighboring pixels.

Properties of this kernel - Weighted average to prevent blow-up of the values (we don't want it to exceed RGB max of 255 fe). - Note that the weight is given by the sum of all kernel pixels, not by the count of pixels! (so kernel [[0,1,2][3,4,5],[6,7,8]] is not divided by 9, but by 0+1+2+.. = 36) - Uneven because we need a center-value. - Smaller/larger filter size will store more/less spatial data. There are a few solutions to prevent loss on the edges:

`39:T5a3,md` #### *Pooling* While convolutions embed spatial information into each value, it doesn't solve the problem of dimensionality. Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map. Often used methods are Max pooling and Avg pooling. Maxpooling with filter size 2x2 and stride 2:

#### *Convolutions + Pooling* In practise: - Color image of 100x100x3 (3 for R,G,B). - Three layers (k=3) - Five filters k1, ten filters k2, fifteen filters k3 (Chosen values: hyperparam) - Each filter accounts for zero padding. - Between each convolution we use a 2x2 maxpool. The filtersize in each convolution has the same 'depth' as the depth of the input. (So 3 at conv1, 5 at conv2, 10 at conv 3) The process would look like this: ${tex`\underbrace{\boxed{\text{Input}}}_{80 ⋅ 80 ⋅ 3} \rightarrow \underbrace{\boxed{\text{Conv1}}}_{80 ⋅ 80 ⋅ 5} \rightarrow \underbrace{\boxed{\text{MaxP1}}}_{40 ⋅ 40 ⋅ 5} \rightarrow \underbrace{\boxed{\text{Conv2}}}_{40 ⋅ 40 ⋅ 10} \rightarrow \underbrace{\boxed{\text{MaxP2}}}_{20 ⋅ 20 ⋅ 10} \rightarrow \underbrace{\boxed{\text{Conv3}}}_{20 ⋅ 20 ⋅ 15} \rightarrow \underbrace{\boxed{\text{MaxP3}}}_{10 ⋅ 10 ⋅ 15} \rightarrow \boxed{\text{Input Neural Network}}\\`} `3a:T6af,md`## Representing Graphs (Graph Neural Networks) https://pytorch-geometric.readthedocs.io/en/latest/ Data in the 'real world' often doesn't have perfect structures. Social networks, road networks, molecules, unstructured data etc. can often be defined with nodes and edges: in a graph network. We can't 'out-of-the-box' plug these graphs into our NN. This is where graph neural networks (GNNs) come in. GNN's are to Graph Networks what CNNs are to images. A graph can be represented with an adjacency matrix and a feature matrix. The adjacency matrix can be filled with binary (connected/not connected) or with the weigth of the nodes:

In a CNN we use a filter that looks at the neighbouring pixels. In a GNN we don't know how many neighbours we have, and we don't have an order. In each layer we _update_ the value of a node by looking at itself (· weight₀) and its neighbors (· weight₁). We can either sum or average over the adjacent nodes. Lastly we apply a non-linearity like ReLu.

By doing this, each consequent layer takes in 'information' of more neighbors. The final output can be used for node classification(what class does this node belong to?), graph classification (is this molecule toxic?), link prediction (are nodes linked?)

` 3b:T512,md`## Hyperparameters Parameters we cannot learn. We set them manually and depending on the data. Fe: number of layers, number of neurons in each layer, etc. You have to find out yourself. Most importantly: Never use test set and labels to tweak best settings. The results won't translate well into the real world. What *can* we do? #### *Validation* Take out a portion of the training set to be used for evaluation. The downside is that we'll have to set-aside a part of the data (and thus reducing the set on which we can train our model). This is the preffered method when data is sufficient and/or models are complex. When we do have a limited number of entries, (or sufficient computational power) we can alternatively we use crossvalidation: #### *Crossvalidation* We split the dataset into K portions (folds) and we the K validation test. In each test, we hold out one of our folds. The total number of tests is equal to the number of folds times the number of hyperparameters to tune. The number of folds is a hyperparameter- usually around 5-8. If we have more hyperparameters then the computational costs increases. We can decrease the number of folds.

`3c:T57c,md`## Evaluating Performance Different groups have different goals. Performance should be compared to the goals of the user. For a search engine we can measure the rate of succesful returned links. For an advertiser this could be the clickthrough rate. From a machine-learning perspective we're mostly interested in our scores on the test-set. #### *Classification accuracy* Accuracy is the fraction of correct decisions (how many total vs how many correct). However, accuracy isnt always a good measure. - If there is a high price for false {positives, negatives} - FE you test -neg for a certain disease while you *do* have it - When the dataset is not balanced - FE 98% of our entries are negative. Classifying everything negative results in 98% accurate! #### *Recall and precision* Recall: relevant items predicted / total number relevant items in collection. Precision: relevant items predicted / total number of items predicted. Say we have a classifier that makes 100 predictions. It predicts 60 as false, 40 as true. From these false, 10 were actually true. From the true, 5 were actually false. TP->35 FP->5 TN-> 30 FN-> 10. Precision is then calculated with TP/(TP+FP) = 35/(35+5) = 35/40 = .875 Recall is calculated with TP/(TP+FN) = 35/(35+10) = 35/45 = .778

`3d:T77e,{ let height = 80 let width = 600 let margin = {top:10, left:50, bottom:10, right: 50} const svg = d3.create("svg") .attr("width", width) .attr("height", height + margin.top + margin.bottom) let data = [{rank: 1, relevant: true, label:'[1/1]*1'}, {rank: 2, relevant: true, label:'[2/2]*1'}, {rank: 3, relevant: false, label:'[2/3]*0'}, {rank: 4, relevant: true, label:'[3/4]*1'}, {rank: 5, relevant: false, label:'[3/5]*0'}, {rank: 6, relevant: false, label:'[3/6]*0'}] let x = d3.scaleLinear() .domain([0, data.length]) .range([margin.left, width-margin.right]) let s = d3.min([(margin.left, width-margin.right)/data.length - 30, height]) svg.selectAll("rect") .data(data) .join("rect") .attr("x", (d, i) => x(i)) .attr("y", 20) .attr("width", s) .attr("height", s) .attr("fill", d => d.relevant ? "#11644D" : "#F24E4E" ) .attr("fill-opacity", .3) .attr("stroke", d => d.relevant ? "#11644D" : "#F24E4E" ) svg.selectAll("Label") .data(data) .join("text") .attr("font-family", "Consolas") .attr("font-size", "13") .attr("x", (d, i) => x(i) + s/2) .attr("y", s-10) .attr("text-anchor", "middle") .text(d => d.label) svg.selectAll("Rank") .data(data) .join("text") .attr("font-family", "Consolas") .attr("font-size", "15") .attr("x", (d, i) => x(i) + s/2) .attr("y", 12) .attr("text-anchor", "middle") .text(d => d.rank) // svg.selectAll("rect") // .data(dataset) // .join("rect") // .attr("x", (d,i) => (w/(dataset.length+2) * (i+1)) ) // .attr("y", (d) => h-d.value*6) // note that 0 is at the top // .attr("width", 15) // .attr("height", d => d.value*6) // .. and we draw from top to bottom return svg.node() }3e:[{"id":"6df7f6fbafbf32c22466198d17da83e54f33d109b7e77a7a4814fdfad73030191b1b107b86a9a342ffe51ca106f08e88167170c7716ab69846c1e258b68e63b6","href":"https://static.observableusercontent.com/files/6df7f6fbafbf32c22466198d17da83e54f33d109b7e77a7a4814fdfad73030191b1b107b86a9a342ffe51ca106f08e88167170c7716ab69846c1e258b68e63b6","name":"1d_conv.PNG","type":"image/png","lastModified":1605618949946.114,"size":117001},{"id":"80c3d0495102aff5110842d20e44b52e71b396e6965fb39cb477d1dc945fafebd7d6a3cbffc9de386bcfa50b02df494da3333c985909f8cce45732aca60242d3","href":"https://static.observableusercontent.com/files/80c3d0495102aff5110842d20e44b52e71b396e6965fb39cb477d1dc945fafebd7d6a3cbffc9de386bcfa50b02df494da3333c985909f8cce45732aca60242d3","name":"zero_pad_alt.PNG","type":"image/png","lastModified":1605619409089.357,"size":1241368},{"id":"35fe75ff4bc523a14a325b1f4fc11af382c10c56de7f5012318c73c6e344a37f251efe671a3124d1c2abdc9076e2f4a200510c47e8a987bb0a29ba2c71a89758","href":"https://static.observableusercontent.com/files/35fe75ff4bc523a14a325b1f4fc11af382c10c56de7f5012318c73c6e344a37f251efe671a3124d1c2abdc9076e2f4a200510c47e8a987bb0a29ba2c71a89758","name":"2d_conv.jpeg","type":"image/jpeg","lastModified":1605619926351.86,"size":31739},{"id":"52c2a2a27884f6df6adb4b4b4901100715a9ed04e71c8a642c37f3d755fb99d84fec84d6d4f0e7cad3141a6ddf9fd5c608f2379061f0f7cc94ace094bb7e3266","href":"https://static.observableusercontent.com/files/52c2a2a27884f6df6adb4b4b4901100715a9ed04e71c8a642c37f3d755fb99d84fec84d6d4f0e7cad3141a6ddf9fd5c608f2379061f0f7cc94ace094bb7e3266","name":"crossval.png","type":"image/png","lastModified":1606052706863.831,"size":45148},{"id":"0b8a95ce39a07f72e98b957ab1031e39c04ea42870b90914cb253981ed56ed3970bd3f07adce57450e36ec71a5bcc786ebe168ee76d67b6bb48498d83dd7dba3","href":"https://static.observableusercontent.com/files/0b8a95ce39a07f72e98b957ab1031e39c04ea42870b90914cb253981ed56ed3970bd3f07adce57450e36ec71a5bcc786ebe168ee76d67b6bb48498d83dd7dba3","name":"roc_curve.PNG","type":"image/png","lastModified":1606058742283.229,"size":288245},{"id":"489a3ff2e69e6dddad04a5a059ea1dba52dd1c58dfd13c612ecffb73a73ac6c25a9438c8a5f33692cbb1e714b56105375f35d3b4ea672bf292d5de16469a34c5","href":"https://static.observableusercontent.com/files/489a3ff2e69e6dddad04a5a059ea1dba52dd1c58dfd13c612ecffb73a73ac6c25a9438c8a5f33692cbb1e714b56105375f35d3b4ea672bf292d5de16469a34c5","name":"roc_curve 2.png","type":"image/png","lastModified":1606059753574.498,"size":225621},{"id":"fb89639fea355e674bc48aa72b64671b6d981dd075f91b1be2d25f7848b5c634cf958db5ccc4666ea1b4e4e10077f7e1db192a90ece051cda31dd7962c75c8b2","href":"https://static.observableusercontent.com/files/fb89639fea355e674bc48aa72b64671b6d981dd075f91b1be2d25f7848b5c634cf958db5ccc4666ea1b4e4e10077f7e1db192a90ece051cda31dd7962c75c8b2","name":"recall@3.PNG","type":"image/png","lastModified":1606055571350.784,"size":160254},{"id":"a81ad3f3e72174479feb1f75f23c2d51afa52f81cdc6bb250518d7017165c2e0dade13a5c766167e1be0d187f8d14529d980a8d80faace8baa16cf67cae257d7","href":"https://static.observableusercontent.com/files/a81ad3f3e72174479feb1f75f23c2d51afa52f81cdc6bb250518d7017165c2e0dade13a5c766167e1be0d187f8d14529d980a8d80faace8baa16cf67cae257d7","name":"graph-representations3.png","type":"image/png","lastModified":1606125210470.88,"size":26595},{"id":"410942f5d0d5b5e1debbee10acbcc8985a9e1e025ba5dfbc1bdb28fd8b9e54fe4d3e0483ebe4000ba01252e0a03fe1bef9e1158c2d956a7d90b0047684e3e7dc","href":"https://static.observableusercontent.com/files/410942f5d0d5b5e1debbee10acbcc8985a9e1e025ba5dfbc1bdb28fd8b9e54fe4d3e0483ebe4000ba01252e0a03fe1bef9e1158c2d956a7d90b0047684e3e7dc","name":"GNN2.PNG","type":"image/png","lastModified":1606205824334.003,"size":126843},{"id":"355fb99990a763e431b398c487d5e1180c6d5b15df7532a97d4057070c76b78a7b1d08ee7e2ae3860ae37bc3dee8bf6608127711b6b3aad747dba9d33e601bf5","href":"https://static.observableusercontent.com/files/355fb99990a763e431b398c487d5e1180c6d5b15df7532a97d4057070c76b78a7b1d08ee7e2ae3860ae37bc3dee8bf6608127711b6b3aad747dba9d33e601bf5","name":"GNN@1.PNG","type":"image/png","lastModified":1606207317902.006,"size":195322},{"id":"5823a4e577993307e93181d387a3df062dc56c0177abf72eb794116028e00e46824eb054d363407c9da3e0c5367178fe583424f5f0ea9d3873684979cc8cc32c","href":"https://static.observableusercontent.com/files/5823a4e577993307e93181d387a3df062dc56c0177abf72eb794116028e00e46824eb054d363407c9da3e0c5367178fe583424f5f0ea9d3873684979cc8cc32c","name":"Maxpool.PNG","type":"image/png","lastModified":1606212749632.221,"size":20341},{"id":"73fb53922d889c5c3208edf7e0c364ae54228d9d16e0ebe6960426369d6040237089ff5c009ec9281e4b23e920825c8d24120b74659ad72f6e2ecb5a4a55f1a8","href":"https://static.observableusercontent.com/files/73fb53922d889c5c3208edf7e0c364ae54228d9d16e0ebe6960426369d6040237089ff5c009ec9281e4b23e920825c8d24120b74659ad72f6e2ecb5a4a55f1a8","name":"hinge_loss.PNG","type":"image/png","lastModified":1604913610722.134,"size":61611},{"id":"e9952a010f758d0ed391436d648a9732ee209ba0d35203392d18e903882784a1441412a0838d00f3327d3bd6ee5e7d3fb7b03c601c88176840525904f3c8484b","href":"https://static.observableusercontent.com/files/e9952a010f758d0ed391436d648a9732ee209ba0d35203392d18e903882784a1441412a0838d00f3327d3bd6ee5e7d3fb7b03c601c88176840525904f3c8484b","name":"multiclass_linear_classifier.PNG","type":"image/png","lastModified":1604916291980.94,"size":150889},{"id":"6e72748ad074de4a21ee8135927c7e676b7c405577df2d6fb682b7ed4b9244ef592b498ed86cc9bfad1077ae51c9220d1277f315b0f63efdeada089ce4b2c1f6","href":"https://static.observableusercontent.com/files/6e72748ad074de4a21ee8135927c7e676b7c405577df2d6fb682b7ed4b9244ef592b498ed86cc9bfad1077ae51c9220d1277f315b0f63efdeada089ce4b2c1f6","name":"nearest neighbor.PNG","type":"image/png","lastModified":1604917327541.342,"size":63620},{"id":"2ed230a49d3086fbd01944912afd46cb09b8f70d204f4827e314688bccc72fd7511643442c8498506fb55e1b2b0982acf904a376ac017265bb9b9318a3519ea7","href":"https://static.observableusercontent.com/files/2ed230a49d3086fbd01944912afd46cb09b8f70d204f4827e314688bccc72fd7511643442c8498506fb55e1b2b0982acf904a376ac017265bb9b9318a3519ea7","name":"information_gain.PNG","type":"image/png","lastModified":1605021423943.469,"size":159153},{"id":"9ee49f187ed5c7b48bf7395d588621b5feb58bdd0d246b7394905b057ba55f6b09e5802ce99a2d62c1a60314b2f1beb951450b4f5f0c23a5c89a0b2a4aa7c940","href":"https://static.observableusercontent.com/files/9ee49f187ed5c7b48bf7395d588621b5feb58bdd0d246b7394905b057ba55f6b09e5802ce99a2d62c1a60314b2f1beb951450b4f5f0c23a5c89a0b2a4aa7c940","name":"random_forest.PNG","type":"image/png","lastModified":1605029058643.32,"size":63623},{"id":"cc11170a6c0129bb7dc42a23fe24195b6b8d588652e25802ef596dfcaa8803f34d3062875bf04caf63d6c40e023e9e78058b5c9faff00e3d15c423070e8234ca","href":"https://static.observableusercontent.com/files/cc11170a6c0129bb7dc42a23fe24195b6b8d588652e25802ef596dfcaa8803f34d3062875bf04caf63d6c40e023e9e78058b5c9faff00e3d15c423070e8234ca","name":"history.PNG","type":"image/png","lastModified":1605207945762.318,"size":99761},{"id":"a6dc32e75f4e70e6ef12519b75e30ac03ca6eff6759c4ce7c903f9a9a8eef4baf3b964e75da87ce0bbadef62e01a98a7769e03ac9cf2354b0ad9783085dc0f7c","href":"https://static.observableusercontent.com/files/a6dc32e75f4e70e6ef12519b75e30ac03ca6eff6759c4ce7c903f9a9a8eef4baf3b964e75da87ce0bbadef62e01a98a7769e03ac9cf2354b0ad9783085dc0f7c","name":"machine_learning.PNG","type":"image/png","lastModified":1607793199721.573,"size":576769},{"id":"88f6177f5a519994bb71ebb51f4c59162a57de242cf7b059bbdee7091eaabb2ad30cfcd396c292ae17ea73db89ff291ed159107be9f6de31f32bbb9d6d968b50","href":"https://static.observableusercontent.com/files/88f6177f5a519994bb71ebb51f4c59162a57de242cf7b059bbdee7091eaabb2ad30cfcd396c292ae17ea73db89ff291ed159107be9f6de31f32bbb9d6d968b50","name":"nearest_mean.PNG","type":"image/png","lastModified":1604918329847.117,"size":45416},{"id":"53ad77b759f2090dc365ba2dc6c33f2f19a27603f0ccb67e4d133d65cdd3fd8b5db0b1683568b317e47e3ff9dbe2fbec4d9732eeb59155e763af8030b386ac08","href":"https://static.observableusercontent.com/files/53ad77b759f2090dc365ba2dc6c33f2f19a27603f0ccb67e4d133d65cdd3fd8b5db0b1683568b317e47e3ff9dbe2fbec4d9732eeb59155e763af8030b386ac08","name":"adaboost.jpg","type":"image/jpeg","lastModified":1605031120539.243,"size":29701},{"id":"f0b322522a2a364e67c4885378d87e500596f0b734754c36fcf7a55bb9b112041fe790ca29ead8915be61042a5ac64714a0d5b0fb89be8e2f17dd4a0f87311f5","href":"https://static.observableusercontent.com/files/f0b322522a2a364e67c4885378d87e500596f0b734754c36fcf7a55bb9b112041fe790ca29ead8915be61042a5ac64714a0d5b0fb89be8e2f17dd4a0f87311f5","name":"cbow_skipgram.PNG","type":"image/png","lastModified":1608111377360.811,"size":72724}] 31:{"id":"dfb0b99814117329","slug":"aml-applied-machine-learning","title":"[AML] Applied Machine Learning","thumbnail":"be8721fa1f8dab4c4d70d2202153ad991c9752f68c6f791a3bfb1d093efc2749","access_level":"public","create_time":"$D2020-11-07T16:42:09.191Z","update_time":"$D2020-12-16T18:12:34.485Z","view_time":null,"first_public_time":"$D2020-12-16T08:50:02.264Z","has_unpublished_edits":null,"fork":false,"stars":9,"starred":false,"owner":{"type":"team","id":"d19a230dabb035d0","login":"thijsschouten","name":"T.E. Schouten","avatar_url":"https://avatars-next.observableusercontent.com/091c69f0b7b396df70696f26f5738dc8c19b50db5102fc472c06d6e7458dd1ec","tier":"starter_2024"},"version":1581,"publish_version":1581,"role":null,"authored":false,"authors":[{"type":"individual","id":"b8802c07e4374e6f","login":"thijsschouten","name":"T.E. Schouten","avatar_url":"https://avatars-next.observableusercontent.com/091c69f0b7b396df70696f26f5738dc8c19b50db5102fc472c06d6e7458dd1ec","tier":null}],"members":[{"type":"individual","id":"b8802c07e4374e6f","login":"thijsschouten","name":"T.E. Schouten","avatar_url":"https://avatars-next.observableusercontent.com/091c69f0b7b396df70696f26f5738dc8c19b50db5102fc472c06d6e7458dd1ec","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":8,"collection":null,"type":"notebook","body":{"title":"[AML] Applied Machine Learning","cells":[{"id":343,"value":"md`\n\n# [AML] Applied Machine Learning \n## Course notes I\n\n---\n\n---\n---\n### ToC\n\n[Machine Learning](#anchor_machine_learning) \n Machine Learning basics.\n\n\n[Regression](#anchor_regression) \n linear regression, loss function, (stochastic) gradient descent \n\n[Generalization](#anchor_generalization) \n polynomial regression, weight decay \n\n[Classification](#anchor_classification) \n logistic regression, sigmoid function, support vector machines, decision trees, random forests \n\n[Combining models](#anchor_ensembly) \n bagging, ensembling, boosting, adaboost, bootstrapping \n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1471,"value":"anchor_machine_learning = md`\n----\n## Machine Learning Context\n#### *Defs*\n*AI*: the engineering of making intelligent machines and programs. \nCourse focus is on Supervised Learning. \n*ML*: Ability to learn without being explicitly programmed. \n*DL*: Learning based on Deep Neural Networks. \n\nDL is part of ML is part of AI. But AI is broader than ML which is broader than DL.\n\n

\n\nBasics of Machine Learning (Tom Mitchell, ML, 1997)\n> for some tasks *T*; \nand performance measure *P*; \nthe value *P* increases for *T* with experience *E*\n\nGiven input x → do some fancy computing → predict output y.\n\n#### *Datatypes*\n*Structured:* sensor readings, tax forms, excel tables \n*Unstructured:* Images, free text, speech\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":349,"value":"anchor_regression = md`\n----\n\n## Regression\n\nGoal: to predict a continuous output value ${tex`t`} (fe price) from a given input vector ${tex`x`} (fe m2, nr_bathrooms, etc). To find the best fit, we need to optimize the weights ${tex`w`}.\n\n> ${tex`y = f(x,w)`}\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":0,"value":"md`### Linear regression \n*[lecture 2.1, ~5:00]*\n\nIn linear regression, the model can be written like:\n> ${tex`tz = f(x,w) = x^Tw`} \n\n#### *Loss function*\nThere are various scoring methods (*learning objectives, or loss functions*). One of them: mean squared error (MSE). For each of the ${tex`N`} training examples, we look at the difference between the true value and the predicted value ${tex`(t_n - x_n)^2`}. (Note: ${tex`x_n`} is often written as ${tex`\\hat{t}_n`}.) The sum of all these values is the score of the weights, given by:\n\n> ${tex`J(w) = \\frac{1}{2} \\sum{(t_n - {x_n}^T}w)^2`}\n\nTo find the the value of ${tex`w`} (the slope of the regression model) that minimizes our MSE, we need to find the derivative of ${tex`s(w)`} with respect to ${tex`w`}:\n\n> ${tex`\\frac{\\partial w}{\\partial J(w)}= - \\sum{(t_n - w^Tx_n)}x_n = \\underbrace{-(T-W^TX)^TX^T}_\\text{Matrix form}`} \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"md`#### *Finding optimal value: closed form*\n\nWe can find the optimal value with calculus, by setting the derivate to 0. \n\n> ${tex`\\begin{aligned}\n -(T-W^TX)^TX^T &= 0 \\\\ \n & \\vdots \\\\\n W &= (XX^T)^{-1}XT^T\n \n\\end{aligned}`}\n\n*Pros: direct minimum and solution in one step. \nCons: doesn't scale well*\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":227,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":281,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":388,"value":"anchor_classification = md`\n---- \n\n## Classification\n\nGoal: predict discrete output from given inputs. \nFind a line that seperates inputs of different classes.\n\n\n### Logistic regression\nStarts from a linear model like linear regression. \nAdds an activation function that transforms output into \"probability\".\n\nEasiest solution is a step function. But: this is not differentiable. \nBetter solution is an activation function that transforms an output into a value between 0 and 1 (or -1 and 1). \n\nExample activation function is a sigmoid function:\n\n> ${tex`\\sigma(z) = \\frac{1}{1+e^{-z}}`}\n\nAnd the derivative:\n\n> ${tex`\\frac{d}{dz}\\sigma(z) = \\sigma(z)(1-\\sigma(z))`}\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":417,"value":"md`\n\n#### *Logistic regression with gradient descent*\n\nSteps:\n1. Linear map: ${tex`h=x^Tw+b`}\n2. Sigmoid activation: ${tex`f=\\sigma(h)`}\n3. Compute error between sigmoid output (between 0 and 1) and label (0 or 1)\n4. Update values of w and b based on error\n\nA suitable error / costfunction is *negative log likelihood*. The goal is to output low sigmoid values for label 0, and high for label 1.\n\n> ${tex`\\text{L} = -\\frac{1}{N}\\sum{(y_i \\log(f_i)+(1-y_i)\\log(1-f_i))}`}\n\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":453,"value":"md`\n#### *Classification that maximizes the margins : hinge loss*\nWe need a new loss function: hinge loss. Correctly classified datapoints have a loss of 0, incorrectly classified datapoints have a large loss, depending on the distance to the truth.\n\n

\n\n> ${tex`\\text{L} = \\sum| 1 - y_nf(x_n,w) |=\\sum\\max(0,1-y_nf(x_n,w))\\\\\n\\small x_n: \\text{datapoint}\\\\ \ny_n: \\text{ground truth label}\\\\ \nf(x_n,w): \\text{the prediction}`} \n\n\n\n`\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":505,"value":"md`\n#### *Support Vector Machines*\n\nA classifier that optimizes both a maximal margin as well as a minimal error. Precise workings are beyond the scope of this course.\n\n> ${tex`min \\frac{1}{2}||w||^2+c\\sum{\\xi_i}`} \n...\n`\n\n\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":522,"value":"md`\n#### *Multiclass linear models*\n\nWe can use these concepts to create a multiclass classifier. Say we have three classes:\n\n

\n\n- First we create three lines (binary classifiers), the class vs the other two classes.\n- Then, we add a decision rule: we calculate the score of each datapoint with respect to each classifier. The highest score 'wins' (or least negative, within the triangle). Formula:\n\n> ${tex`\\hat{y}=\\arg \\underset{k}{\\max} w_k x + b_k\\\\\n\\small \\hat{y}: \\text{target label}\\\\ \nw_k: \\text{weight for each class}\\\\ \nb_k: \\text{bias/offset for each class}\\\\ \nx: \\text{input x}\\\\\n\\text{arg max}: \\text{index for which the value is highest}`} \n\n\n`\n\n\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":560,"value":"md`\n#### *Nearest Neighbor Classifier*\n\nLooks at k nearest neighbors. Most seen label will be the predicted class.\n

\n\nEasily visualisable with a voronoi diagram. k Is a hyper parameter, not trainable.\n\nWe can avoid a tie in binary classification by setting k to an odd value. In multi-class classification, we can incorporate the weight inversely proportional to the distance (so the smaller the distance, the larger the weight).\n\nPros: \n- we don't need to do any 'learning'/'training' \n- non-linear (complex boundaries)\n- same for binary and multiclass\n\nCons:\n- Not very scalable, need to look at _all_ examples for every new classification. (Instead of just training one model)\n- Sensitive to outliers (especially with low k values)\n\n\n\n`\n\n\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":584,"value":"md`\n#### *Nearest Class Mean Classifier*\n\nCalculates the mean of each class. Then calculates the distance to the mean for each x. The lowest value will be the predicted label. \n

\n\n`\n\n\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":643,"value":"md`\n#### *Decision trees*\n\nBenefits: \n- Clear reasoning behind decision\n- Works on discrete and continuous features\n- Works on incomparable features\n- Fast to train and to evaluate\n\nSo how do we decide what we split the tree on?\n- We want the subtree as pure as possible (most information gain = decreases entropy the most)\n\nWe can do this by calculating the entropy (probability of 50% = entropy 1. probability 0% or 100% = entropy 0):\n\n> ${tex`E(S) = -P \\log_2(P) - N \\log_2(N)\\\\ \\small\nS: \\text{total set}\\\\ \nP: \\text{ratio of positive}\\\\ \nN: \\text{ratio of negatives}`}\n\n\n\n`\n\n\n\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":670,"value":"md`\n#### *Multiclass Decision trees*\n\nEntropy for multiple classes:\n> ${tex`E(S) = -P_1 \\log_2(P_1) - P_2 \\log_2(P_2) - \\dots - P_n \\log_2(P_n) = -\\sum P_i\\log_2(P_i)`}\n\nAnd the weighted average of entropies at a certain split (A):\n> ${tex`E(S) = -\\sum \\frac{|S_i|}{|S|}E(S_i)`}\n\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":695,"value":"md`\n#### *Random Forest Classifier*\nDecision trees have downsides. With many features and examples, the tree may become too deep. This could lead to perfect seperation but poor generalization.\n\nInstead of training one tree perfectly, we can train multiple trees imperfectly (on a random subset of features and/or entries (usually the first)).\n\nTraining process: \n(with hyperparameters K trees, D depth, R features per split)\n1. Intialize forest F\n2. For tree k=1,..,k \n - For depth d=1,..,D \n - Randomly select R features\n - Split node based on features\n - Add tree (k) to forest (f)\n\n

\n\nFeatures can then be predicted by traversing each tree and averaging the scores. (Or by majority vote).\n\nClassical descision tree classification result in low bias high varience predictions. Random forests solve the high variance. \n\n` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":729,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":761,"value":"anchor_1 = md`\n------\n## Deep Learning\n\n#### *History*\n\n- Isn't new. Started in 1958 with the invention of the 'perceptron' by Rosenblatt. \n- Now a hype because of advancements in: data, hardware, open-source software, tricks, more than just recognition\n\n

\n\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":779,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":828,"value":"md`\n#### *Eror calculation & reduction*\nAfter the forward pass, we can calculate how well the model performs. This can be done by comparing the actual output with the ground truth. \n\n> ${tex`MSE = \\frac{1}{2}(\\text{prediction} - \\text{actual})^2`\n}\n\nNote:\n- Error is 0 when prediction = ground truth\n- Error is always positive because of the square\n- 1/2 is added only for ease of calculating the derivate\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":816,"value":"md`\n#### *Backward propagation*\n Next we need to update the weights and biases iteratively using backward propagation. This is done through calculation of the gradient of the error functions with respect to the weights. \n\n\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":887,"value":"md`\n#### *Hyperparameters*\nThere are no rules for setting hyperparameters. We need to get a 'feel' for it by practising. Some important hyperparameters to set:\n\n- Number of layers and weights per layer\n- Number of trianing iterations\n- Learning rate for (stochastic) gradient descent\n- Regularization\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":896,"value":"md`\n#### *Tricks to improve the network:*\n##### DropOut\nRandomly drop neurons per iteration (with a certain probability p (like 0.05)). This is done during training. Motivation to do so:\n\n- Avoid learning co-occurrences (fe: all our training pictures with dogs, also contain sofas. The model might learn that sofa -> dog)\n- It actually simplifies the model, but there's way more variations.\n- It becomes harder for the model to 'memorize' images\n\n##### Weight decay\n... see ^ \n##### BatchNorm\nRe-normalize features in each layer.\n##### Residuals\n...\n\n\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":912,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":986,"value":"md`\n##### 2. Dimensionality Reduction: Latent Semantic Analysis\n[lecture 6.2, ~03:55] \nRepresenting text as data is high-dimensional. Fe, the size of each datapoint is equal to the total length of the whole corpus. At the same time, the data is very sparse. (Many 0's between the 1's representing the words/characters)\n\nSVD / Singular Value Decomposition is used to reduce this dimensionality. \nSVD represents the input matrix A as a product of three seperate matrices. \n\n> ${tex`A_{[m*n]} = U_{[m * r]}\\sum_{[r * r]}(V_{[n*r]})^T \\small \\\\\n A: \\text{input data matrix with m documents and n terms} \\\\\n U: \\text{left singular vectors with m documents and r concepts/topics} \\\\\n \\sum: \\text{singular values sorted descending, with r as rank of the matrix A} \\\\\n V: \\text{Right singular vectors, with n terms and r concepts/topics}`}\n\nWe can think of the diagonal elements on ${tex`\\sum`} as an indication of the 'strength' of each concept.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1545,"value":"md`##### 3. Word and document embeddings\nSkip-grams and continuous bag of words (CBOW).\n\n