2c:null 2d:["$","title",null,{"children":"Decision trees | Sandraviz | Observable"}] 32:T49b,## Attribute Selection Measure The best attribute or feature is selected using the Attribute Selection Measure(ASM). The attribute selected is the root node feature. ASM is a technique used for the selecting best attribute for discrimination among tuples. It gives rank to each attribute and the best attribute is selected as splitting criterion. In terms of decision tree, Entropy helps us to build an appropriate tree for selecting the best splitter it always lies between 0 to 1. It measures the purity of the split. It is hard to draw conclusions from the information when the entropy increases. #### Entropy Entropy is a measure of disorder or uncertainty. The entropy of a distribution describes the degree of randomness; namely the higher entropy, the less we are certain about the value of the random variable. Consider the Bernoulli distribution ${tex `\text{Ber}(p)`} where ${tex `p=[0, 1]`}. Recall that its sampling space is ${tex `\Omega = \{0, 1\}`} and ${tex `p(X=1)=p, p(X=0)=1-p`}. We can see how the mean parameter ${tex `p`} governs the randomness in the distribution. From the visualisation below, each dot is an independent Bernoulli random variable.33:T7ba,If there are total 100 instances in our class in which 30 are positive and 70 are negative then, ${tex `P(+) = 3/10 `} and ${tex `P(-) = 7/10 `}. The entropy can be calculated as followed. ${tex ` H(s)= -3/10 * log2 (3/10) - 7/10 * log2 ( 7/10) ≈ 0.88`} Imagine each house would belong to SF, then the entropy would be 0, if 50% of the houses belong to SF and 50% to NY there is a lot uncertainty/randomness. In this case the entropy would be at the highest. ${tex ` H(s) = -[ ( 5/10) * log2 (5/10) + (5/10) * log2 (5/10) ] = 1`} ### Information Gain Information gain is a decrease in entropy. Decision trees make use of information gain and entropy to determine which feature to split into nodes to get closer to predicting the target and also to determine when to stop splitting. As seen above imagine a tree using a variable with 50% in one group and 50% in another group, hence the highest level of entropy, which is 1. Choosing another variable which has 40% and 60% as the first split and then checking on each side the original variable we get for the 60% (group) ${tex ` H(s) = -[ ( 4/6) * log2 ( 4/6) + (2/6) * log2 (2/6) ] = 0.917`} for the 40% (group) ${tex ` H(s)= - [ ( 3/4) * log2 (3/4) + (1/4) * log2 (1/4) ] = 0.811`} The net entropy is therefore ${tex `(6/10) * 0.917 + (4/10) * 0.811 = 0.874`} The total reduction in entropy due to the split ${tex `1 - 0.874 = 0.126 `} is called information gain ### Gini Index Gini Index, also known as Gini impurity, calculates the amount of probability of a specific feature that is classified incorrectly when selected randomly. If all the elements are linked with a single class then it can be called pure. It measures the impurity of the node and is calculated for binary values only. Gini = ${tex `1 - (x/n)2 - (y/n)2`} The highest value is also related to the highest randomness of a variable, but only reaches 0.5 and the lowest is 0. ${tex `1 - (5 / 10)2 - (5 / 10)2 = 0.5`} 34:T533,## Bias-Variance Trade-off of the NY/SF model Rebuilding the decision tree from the use case above using a different visualization technique. In this example the minimum node size was 1, therefore the tree reaches high levels of complexity and a tendency of overfitting.

A model that is overly-simplistic is just as problematic as one that is overly-scrupulous. Errors due to bias and those due to variance are distinct. Understanding the tradeoff between bias and variance (and how different model types let you balance the two) is foundational to modeling well.

### Conclusions - No single algorithm will solve all your machine learning problems better than every other algorithm. - Make sure you completely understand a machine learning problem and the data involved before selecting an algorithm to use. - All models are only as good as the assumptions that they were created with and the data that was used to train them. - The best models for a given problem are somewhere in the middle of the two bias-variance extremes (see image above) - To find a good model for a problem, you may have to try different models and compare them using a robust cross-validation strategy.35:T49c,## Assignment 1 ### Part 1: Build the decision tree model Please, build a decision tree for the HR department to support application filtering. Previous data (use it for construction of the tree) is in the attached file hr_train_1.csv The tree is supposed to set a flag ”ignore”:True on the applications which are not interesting to the department. It’s okay to use sklearn library in the assignment, if you wish. Test your solution on the persons: - Senior, Java, with tweets, no PhD - Senior, Python, no tweets, with PhD - Mid, Java, with tweets, no PhD - Junior, Python, with tweets, no PhD - Explore the rules of the decision tree. ### Part 2: Biased dataset Now imagine that the HR department started to collect the gender of the applicants. Add column ”gender” to the train dataset and fill it (programmatically) with invented (you constract it as you wish) data (values 0,1,2) so that two-thirds of gender 2 will be marked with the ”ignore” flag in tests. How can you construct a test to demonstrate that the tree trained on this updated dataset will be biased? How can you show the bias to the HR department? Implement your answer in code.36:T482,## Appendix ### Books [Data Science from Scratch](https://github.com/joelgrus/data-science-from-scratch) Joel Grus 2019 (2nd edition) ISBN-13: 978-1491901427

### Research Papers [Three Machine Learning Solutions to the Bias-Variance Dilemma](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3588594). Marcos López de Prado. Cornell University. Operations Research & Industrial Engineering. 2020 ### Web links - [A visual introduction to machine learning](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/) - [Decision tree visual explained](https://mlu-explain.github.io/decision-tree/) - [Random forest visual explained](https://mlu-explain.github.io/random-forest/) - [Model Tuning and the Bias-Variance Tradeoff](http://www.r2d3.us/visual-intro-to-machine-learning-part-2/) - [Calculate the bias variance trade off](https://machinelearningmastery.com/calculate-the-bias-variance-trade-off/) - [Bias-variance visual explained](https://mlu-explain.github.io/bias-variance/) 37:T558,38:[{"id":"ee8df399ed62803341cb062c4bb3d6edf9df382a7354725c8f3361670ab79331247f79f4bdd5161c662c4bce614a93c98de109b2af393176da33afd9f27779d1","href":"https://static.observableusercontent.com/files/ee8df399ed62803341cb062c4bb3d6edf9df382a7354725c8f3361670ab79331247f79f4bdd5161c662c4bce614a93c98de109b2af393176da33afd9f27779d1","name":"dt@1.jpg","type":"image/jpeg","lastModified":1631545451847.929,"size":126933},{"id":"2fb7f85d6a9a1a6d4bb75c10af3c0f441aefd5703f6778eccd5b83e77f2ce5eda7d80874a6515769f2d4077c717788b59bdf1048e66e752aa9b118c3c02ecd59","href":"https://static.observableusercontent.com/files/2fb7f85d6a9a1a6d4bb75c10af3c0f441aefd5703f6778eccd5b83e77f2ce5eda7d80874a6515769f2d4077c717788b59bdf1048e66e752aa9b118c3c02ecd59","name":"training.jpg","type":"image/jpeg","lastModified":1631546974558.576,"size":101195},{"id":"bf8a795499b3d1da0b11319552780bf6e6145c5db78a295bacba521fd8f7940d86dc0290e2274c02bb9fb2f87bfac5ec6211c79fa64f0a3f01f86d2ead8380bb","href":"https://static.observableusercontent.com/files/bf8a795499b3d1da0b11319552780bf6e6145c5db78a295bacba521fd8f7940d86dc0290e2274c02bb9fb2f87bfac5ec6211c79fa64f0a3f01f86d2ead8380bb","name":"data science .jpeg","type":"image/jpeg","lastModified":1632488352169.275,"size":716254},{"id":"b8e61abbdffc28c51a227763d36edc8d0ae0fe25ed573ca5840241d76b20e53e3c7a517f8f63aa68cb399a4a115423607b9f66e2dd97a7b55797db8238d13161","href":"https://static.observableusercontent.com/files/b8e61abbdffc28c51a227763d36edc8d0ae0fe25ed573ca5840241d76b20e53e3c7a517f8f63aa68cb399a4a115423607b9f66e2dd97a7b55797db8238d13161","name":"decision tree.png","type":"image/jpeg","lastModified":1662650639031.799,"size":36872},{"id":"5f0f521ff0bf3c938b20ee3b76a675b3203089ee5c52a27b80afda3efeae9f7daa8dac9dd4602589a9713d725c9b8581d77318d82d9085df29728cd375376dfd","href":"https://static.observableusercontent.com/files/5f0f521ff0bf3c938b20ee3b76a675b3203089ee5c52a27b80afda3efeae9f7daa8dac9dd4602589a9713d725c9b8581d77318d82d9085df29728cd375376dfd","name":"PYTHON.png","type":"image/png","lastModified":1631546408926.912,"size":58981},{"id":"93309eaa7dec2b4b0abc5767fa2f781063c1dfdb095c7f6e5358720d5213b4722b84b9261a9cd56c29491f9fa509637bf0a676be97b1ee9eab4ba1a2a258e839","href":"https://static.observableusercontent.com/files/93309eaa7dec2b4b0abc5767fa2f781063c1dfdb095c7f6e5358720d5213b4722b84b9261a9cd56c29491f9fa509637bf0a676be97b1ee9eab4ba1a2a258e839","name":"BV2.jpg","type":"image/jpeg","lastModified":1632726398299.288,"size":77838},{"id":"509cd0e5b3aa8734ca7105424030e9dd76158d9df2571beb7d94c5df8058046a05b5e9f2e7c5e744fd99818493def580461635f70aed0ce46638510c7890ee00","href":"https://static.observableusercontent.com/files/509cd0e5b3aa8734ca7105424030e9dd76158d9df2571beb7d94c5df8058046a05b5e9f2e7c5e744fd99818493def580461635f70aed0ce46638510c7890ee00","name":"BV.jpg","type":"image/jpeg","lastModified":1632726386848.315,"size":64329},{"id":"a8b993d74e7e9ad5a820aaf3ecb278c6eb450e47c4e1e45239512db30cbb723d2848ac5cd913a136d0acea1619848c8b60d9064a5360857e37dfb6f5610a3754","href":"https://static.observableusercontent.com/files/a8b993d74e7e9ad5a820aaf3ecb278c6eb450e47c4e1e45239512db30cbb723d2848ac5cd913a136d0acea1619848c8b60d9064a5360857e37dfb6f5610a3754","name":"irisdt.jpg","type":"image/jpeg","lastModified":1632728050027.915,"size":389127}] 30:{"id":"cb25966488991ce5","slug":"decision-trees","title":"Decision trees","thumbnail":"2a2317eb7b6c52f831145d5eb98187e839362e3526390c2886b76a2a59408181","access_level":"public","create_time":"$D2021-09-13T15:04:11.847Z","update_time":"$D2023-11-28T12:32:34.851Z","view_time":null,"first_public_time":"$D2021-09-13T15:06:32.176Z","has_unpublished_edits":null,"fork":{"id":"fe4c72b0f9bf01af","slug":"limitations-of-ai","title":"Limitations of AI","owner":{"id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz"}},"stars":5,"starred":false,"owner":{"type":"team","id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/e6a2480f89a40dcfbe75f19d5081ae2296848971c4b934ed0499fd3b7f1973eb","tier":"starter_2024"},"version":1705,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"3d1beb3a90fbfe98","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/9b19f1a074469dcaafa9bd24c7b08f4bc969889b551fe70c209497f1d03c6514","tier":null}],"members":[{"type":"individual","id":"3d1beb3a90fbfe98","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/9b19f1a074469dcaafa9bd24c7b08f4bc969889b551fe70c209497f1d03c6514","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":{"id":"eab5d593efbf8501","slug":"ai","title":"AI ","owner":{"id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz"}},"type":"notebook","body":{"titling":"auto","title":"Decision trees","cells":[{"id":0,"value":"# Decision trees\n\nIn 2021 and 2022 I taught remotely at the **University Arcada (Finland)** responsible and ethical uses of machine learning practice, which is a fundamental precondition to trustworthy development of AI. You can find all the material and links in this [Github Repository](https://github.com/sandravizz/Decision-support-systems-verification-Python).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1654,"value":"

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1284,"value":"\n- One of the supervised machine learning algorithms\n- White box type of algorithm, internal decision making is transparent\n- Usually considered human-interpretable as it is close to how humans think and make decisions\n- Can handle a lot of input\n- Problem: Difficult to find the optimal path, tend to overfit\n- Regression trees have numerical and classification trees have categorial outputs","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1448,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1444,"value":"viewof p","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1439,"value":"chart","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1483,"value":"entropyView","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1676,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1363,"value":"## Example: Classifying homes \n\nDecision trees look at one variable at a time and are an accessible machine learning method. In computer science, trees grow up upside down, from the top to the bottom and use if-then statements to define patterns in data. \n\nBased on this example, if a home's elevation is above some number, then the home is probably in San Francisco. The top item is the question called root nodes, just like with real trees, everything starts there. That question has two possible answers, so answers are two branches nodes leading out of the tree. The goal is to build the tree until all homes are classified correctly. \n\n

\n\nRe-running our training dataset through the tree, we get 100% accuracy. \n\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1264,"value":"## Examples in python\n\n

\n\nYou can find a collection of notebook examples under this [link](https://github.com/sandravizz/Decision-support-systems-verification/tree/main/Decision%20tree)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1404,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1690,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1357,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1437,"value":"import { chart, viewof p, entropyView } from \"@heytitle/uncertainty-and-entropy\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":18,"value":"$37","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"}],"files":"$W38","stdlib":"1"}} 31:"$30:body"