2b:null 2c:["$","title",null,{"children":"Algorithmic bias | Sandraviz | Observable"}] 31:T91e,## Human Biases Before we look at the different "Machine Biases", let's have a deeper look at how humans tend to make biased decisions based on cognitive biases. - Which ones come to your mind? - Do you remember having made biased decisions? ### Types of errors A continually evolving list of cognitive biases has been identified over the last six decades of research on human judgment and decision-making in cognitive science, social psychology, and behavioural economics. #### Magic Answer Fallacy The first bias is so pervasive, that it's not formally recognised as a cognitive bias. It's the tendency to respond to questions with a simple "magic answer". For instance, how many days are in a year? Most people would just say 365. But the true length of a year on Earth is ~365.2422 days. That is why we have leap years, but even leap years don't fully account for it. The first step is to accept that most questions don't have magic answers. Most answers are uncertain, and to capture the uncertainty we draw it as a distribution. A distribution has the possibilities on the x-axis and the probabilities on the y-axis. Now, begin thinking in distributions instead of magic answers. #### Bayes rule Decisions are based on two parts: prior belief and likelihood of evidence. When making guesses, sometimes people use the new evidence and forget to include their prior knowledge, this is called Base Rate Neglect. Sometimes they stick too firmly to prior knowledge and forget to include the evidence, this is called Bayesian Conservatism. #### Overconfidence Bias The overconfidence bias is the tendency people have to be more confident in their own abilities, such as driving, teaching, or spelling, than is objectively reasonable. This overconfidence also involves matters of character. Because of the overconfidence bias, people will often take ethical issues lightly. They simply assume that they have good character and will therefore do the right thing when they encounter ethical challenges. In fact, studies show that the overconfidence bias causes people to overestimate how much, and how often, they will donate money or volunteer their time to charities. So, overconfidence in our own moral character can cause us to act without proper reflection. And that is when we are most likely to act unethically.32:T42b,## Machine bias Read through the following article and think about the questions listed below then discuss in your group. [

](https://parametric.press/issue-01/the-myth-of-the-impartial-machine/) - What are biased data? - What is the difference between a sampling and a non sampling error? - Think about three examples of representative but biased data (similar to the Amazon example)? - How can algorithms themselves amplify biases? - What are runaway loops, think about another example similar to the one mentioned in the article? - What is the idea behind de-biasing data, what is your opinion about it? - What is the idea behind de-biasing algorithms, what is your opinion about it? - Which other than accuracy metrics are mentioned to evaluate a model, what is your opinion about it? - What is your view on ethical guidelines? Do you think they are useful? For background information or example inspiration: [Algorithmic bias - Wiki](https://en.wikipedia.org/wiki/Algorithmic_bias) 33:T5a0,### What is Machine Bias? Algorithmic bias is difficult to define. It is often understood as a systematic and repeatable error in computational systems, that is responsible for unfair, wrongful results of data processing. Bias can emerge at every stage of developing and using an algorithm: from data collection, through coding, to application, and interpretation of output. Moreover, there may be feedback, that reinforces, amplifies the bias – which means in every cycle of algorithm application the bias gets stronger. When training data are in a form of written language, the algorithm will reflect, and potentially reinforce the human biases included in texts used for training, transfer the biases of their authors – as it was in the case of Microsoft chatbot Tay, launched in 2016. Tay was trained with data from Twitter (it learnt from conversations with human users of Twitter). In just 16 hours after its release it was shut down, because it generated racist and other offensive massages. The chatbot just “internalized” opinions of Twitter users. - Regarding the Tay example, at which stage emerge the bias? - What can we learn from this example/bias? Algorithmic bias is found across platforms, including but not limited to search engine results and social media platforms, and can have impacts ranging from inadvertent privacy violations to reinforcing social biases of race, gender, sexuality, and ethnicity. 34:T147b,### Types of Biases There are two key ways bias can be introduced and amplified during the machine learning process: by using non-representative data and while fitting and training models. #### Biased data In machine learning, models must be trained with data. Training data bias is probably the most widely discussed type of algorithmic bias. It takes place when input data are biased. Usually they are not representative for the population to which the algorithm is applied. A non-representative sample where some groups are over- or under-represented inevitably introduces bias in the statistical analysis. A dataset may be non-representative due to sampling error and non-sampling errors. ##### Sampling errors - Why is it less talked about today? They refer to the difference between a population value and a sample estimate that exists only because of the sample that happened to be selected. Sampling errors are especially problematic when the sample size is small relative to the size of the population. For example, suppose we sample 100 residents to estimate the average household income. A sample that happened to include one very rich houshold would result in an overestimate, while a sample that happened to include predominantly low-income households would result in an underestimate. ##### Non-sampling errors - Ideas for these type of errors? They are typically more serious and may arise from many different sources such as errors in data collection, non-response, and selection bias. #### Algorithmic processing bias Different methods of statistical prediction and decision making (Support Vector Machine, Naive Bayes models, logistic regression, decision trees and others), may give various results. So bias can be included in the way the data are processed. Or, one may argue, it is always embedded in data processing, because there is no neutral, objective way of data processing, each transforms it differently. Algorithms in and of themselves are not inherently unfair, of course; they follow code and are statistical-based tools. The bias stems from the datasets, collated by human hands, on which those algorithms are trained. These are often flawed and of low-quality, i.e. incomplete, skewed or unrepresentative of the population they are trying to make predictions for. ##### Biases that are amplified through algorithms - Even with non biased data how can be the outcome biased? Machine learning algorithms themselves may amplify bias if they make predictions that are more skewed than the training data. Such amplification often occurs through two mechanisms: - incentives to predict observations as belonging to the majority group and - runaway feedback loops. ##### Predict the majority group In order to maximize predictive accuracy when faced with an imbalanced dataset, machine learning algorithms are incentivized to put more learning weight on the majority group, thus disproportionately predicting observations to belong to that majority group. Therefore algorithms may not work well on minority groups in the population. For example, facial recognition algorithms are trained often with majority population data, and that’s why they perform poorly on minority, underrepresented groups. It happens in case of African Americans in USA as well as Asians in East Asia. This situation generates inequality. Sometimes algorithms used by public institutions are bought from private companies which use different input (training) data. ##### Runaway feedback loops In the previous image-classification example, bias amplification stops at the model prediction stage. However, in machine learning models where the prediction is fed back into the model as inputs for the next round of predictions, bias can be amplified further in the form of a feedback loop. Feedback loops are especially problematic when sub-groups in the training data exhibit large statistical differences (e.g. one precinct has a much higher crime rate than others); a model trained on such data will quickly “run away” and make predictions that fall into the majority group only, thereby generating ever-more lopsided data that are fed back into the model. #### Bias in data and in algorithms are interrelated It should be clear by this point that bias in data and algorithms are interrelated. When an algorithm is fed training data where one group dominates the sample, it is incentivized to prioritize learning about the dominant group and over-predict the number of observations that belong to the dominant group. This tendency is exacerbated when the model’s predictive accuracy is relatively low. Conversely, if the data were balanced relative to the predictive accuracy, the model would have nothing to gain by over-predicting the dominant group. Bias can also be perpetuated through a feedback loop if the model’s own biased predictions are repeatedly fed back into it, becoming its own biased source data for the next round of predictions. In the machine learning context, we no longer just face the risk of garbage in, garbage out—when there’s garbage in, more and more garbage may be generated through the ML pipeline if one does not monitor and address potential sources of bias.35:T863,### How to limit algorithmic bias? Each data transformation produces some bias, and we cannot change it. However, it is important to be aware of its nature, limitations, and its consequences. It is difficult, in many cases even impossible, but we should try make systems based on algorithm more transparent, endeavor to understand them better. Finally, we should try to fight blackboxing. There are engineering projects run by top research centers, aimed at minimizing blackboxing in machine learning algorithms. #### Tradeoff sensitive data vs accuracy The design of the model is incomplete. We do not have access to all the data. Therefore we try to uses other variable to capture the missing information. We organize society into certain categories: gender, race, nationality and many others. When the algorithm is designed, people decide which categories should be included – therefore some information about population, some of its characteristics, are selected and others are rejected. The output of algorithm may depend on categories that were included. Sometimes there may be a conflict of standards – for example we exclude some sensitive data from privacy reason (like sexual orientation), but if we did include it, the model would work better, with better accuracy. Therefore, there is a conflict of legal or ethical standard versus statistical standard. #### Technical solutions There have been several attempts to create methods and tools that can detect and observe biases within an algorithm. These emergent fields focus on tools which are typically applied to the (training) data used by the program rather than the algorithm's internal processes. These methods may also analyze a program's output and its usefulness and therefore may involve the analysis of its confusion matrix (or table of confusion). Explainable AI to detect algorithm Bias is a suggested way to detect the existence of bias in an algorithm or learning model. Using machine learning to detect bias is called, "conducting an AI audit", where the "auditor" is an algorithm that goes through the AI model and the training data to identify biases.36:T11a1,## How algorithmic bias impacts society? "Algorithmic decision-making is purported by the mainstream media to reduce the impact of human bias in our social systems. Yet the insidious and secretive nature of algorithms allows them to replicate and amplify the social inequalities brought about by ingrained systemic oppression. As a society, we seem to be content with the knowledge that we are not only failing to resolve systematic inequalities such as racism but promoting them. Such technology excuses us from addressing these complex issues by retreating behind the facade of objectivity. Far from advancing equity, the disturbing reality is that algorithms are racist unless you are white." [link](https://littledatascientistcom.wordpress.com/2019/06/30/the-compas-algorithm-its-not-that-black-and-white/) ### Discrimination Discrimination takes place when some group of people is, with no justified reasons, treated worse than whole population, for example they are deprived of some rights, just because they are members of that group. Examples: FaceApp is an application for editing photographs of a face using different filters, one of which was “hotness” filter. The filter made a person “hotter” by lightening skin tones, implying, that the “whiter” you are, the “hotter” your image is. This case can be considered racial discrimination. After it was revealed, the company’s founder and CEO Yaroslav Goncharov explained: "We are deeply sorry for this unquestionably serious issue. It is an unfortunate side effect of the underlying neural network caused by the training set bias, not intended behaviour. To mitigate the issue, we have renamed the effect to exclude any positive connotation associated with it." In USA criminal justice system black people are tend to be treated more severely by courts than white people who committed the same crimes, because they are judged with the use of algorithms trained with historical data, showing bigger crime rate in black community. Criminal justice algorithms also consider personal characteristics like age, sex, neighborhood, family background. As author of an article on algorithms in the US Criminal Justice System summarizes Two people accused of the same crime may receive sharply different bail or sentencing outcomes based on inputs that are beyond their control. Hiring algorithms of one of American companies designed to limit workforce turnover took into account the area where an applicant lives. It was based on an assumption, that commuters who lived further were more likely to change a job. Accidentally, the algorithm excluded from the recruitment process candidates living in poor neighborhoods, no matter what their qualifications were. Amazon’s recruiting tool algorithm was revealed to discriminate woman. The tool preferred men not because of their competencies, but because they dominate tech industry, so it decided, that just being a men is a success factor. Recruiting algorithms often use historical data to predict what characteristics are correlated with good job performance. ### Manipulation Algorithms of recommender systems strongly influence consumer decisions, or sometimes impose choices. They generate playlists for music services, select content for online media platforms and social media platforms, recommend products in online stores, recommend various services, match partners in online dating platforms, etc. Search engine results, which are always selective – selected by algorithm – impact peoples’ choices. Special emphasis is put on analysis of voting decisions. Research suggests, that manipulations of search results may strongly change the preferences of voters. and influence elections. Biases may also impact how society shapes itself around the data points that algorithms require. For example, if data shows a high number of arrests in a particular area, an algorithm may assign more police patrols to that area, which could lead to more arrests. Because of their convenience and authority, algorithms are theorized as a means of delegating responsibility away from humans.This can have the effect of reducing alternative options, compromises, or flexibility. Sociologist Scott Lash has critiqued algorithms as a new form of "generative power", in that they are a virtual means of generating actual ends. Where previously human behavior generated data to be collected and studied, powerful algorithms increasingly could shape and define human behaviors 37:T10c0,## Hands-on The number of examples are increasing and by analyzing each case, we can rediscover some of the problems mentioned above or even new ones. ### Choose an example Below you find recent article explaining different situations of AI biases. Choose the one that you find most interesting. - [How researchers are using Reddit and Twitter data to forecast suicide rates](https://www.vox.com/recode/2020/2/24/21150196/twitter-reddit-suicide-cdc-artificial-intelligence) Researchers at the CDC and Georgia Tech are using a whole lot of data, including social media, to forecast the suicide rate, a statistic that can lag by up to two years. - [San Francisco banned facial recognition tech. Here’s why other cities should too](https://www.vox.com/future-perfect/2019/5/16/18625137/ai-facial-recognition-ban-san-francisco-surveillance) San Francisco has become the first US city to ban the use of facial recognition technology by the police and local government agencies. This is a huge win for those who argue that the tech — which can identify an individual by analyzing their facial features in images, in videos, or in real time — carries risks so serious that they far outweigh any benefits. - [It’s disturbingly easy to trick AI into doing something deadly](https://www.vox.com/future-perfect/2019/4/8/18297410/ai-tesla-self-driving-cars-adversarial-machine-learning) Artificial intelligence researchers have a big problem. Even as they design powerful new technologies, hackers are figuring out how to trick the tech into doing things it was never meant to — with potentially deadly consequences. - [AI experts say research into algorithms that claim to predict criminality must end](https://www.theverge.com/2020/6/24/21301465/ai-machine-learning-racist-crime-prediction-coalition-critical-technology-springer-study) A coalition of AI researchers, data scientists, and sociologists has called on the academic world to stop publishing studies that claim to predict an individual’s criminality using algorithms trained on data like facial scans and criminal statistics. - [Some AI just shouldn’t exist](https://www.vox.com/future-perfect/2019/4/19/18412674/ai-bias-facial-recognition-black-gay-transgender) Human bias can seep into AI systems. Amazon abandoned a recruiting algorithm after it was shown to favor men’s resumes over women’s; researchers concluded an algorithm used in courtroom sentencing was more lenient to white people than to black people; a study found that mortgage algorithms discriminate against Latino and African American borrowers. - [Mathematicians urge colleagues to boycott police work in wake of killings](https://www.nature.com/articles/d41586-020-01874-9) More than 1,400 researchers have signed a letter calling on the discipline to stop working on predictive-policing algorithms and other models. - [Sexist and biased? How credit firms make decisions](https://www.bbc.com/news/business-50432634) Jamie Heinemeier Hansson had a better credit score than her husband, tech entrepreneur David. They have equal shares in their property and file joint tax returns. Yet David was given permission to borrow 20 times the amount on his Apple Card than his wife was granted. - [HUD Sues Facebook Over Housing Discrimination and Says the Company’s Algorithms Have Made the Problem Worse](https://www.propublica.org/article/hud-sues-facebook-housing-discrimination-advertising-algorithms) The Department of Housing and Urban Development announced Thursday it is suing Facebook for violating the Fair Housing Act by allowing advertisers to limit housing ads based on race, gender and other characteristics. - [Home Office drops 'racist' algorithm from visa decisions](https://www.bbc.com/news/technology-53650758) The Home Office has agreed to stop using a computer algorithm to help decide visa applications after allegations that it contained "entrenched racism". ### Discussion Now think first a moment by yourself about the following questions and then discuss them with your group. - What is the situation? - What kind of "bias" is discussed? - Any suggested solutions? - Do you think the article is reflecting on the real issue? - What are your thoughts on the problem? - What are your ideas regarding solutions? 38:T556,## Appendix ### Web links - [AI’s Islamophobia problem](https://www.vox.com/future-perfect/22672414/ai-artificial-intelligence-gpt-3-bias-muslim) - [Predictive policing algorithms are racist. They need to be dismantled.](https://www.technologyreview.com/2020/07/17/1005396/predictive-policing-algorithms-racist-dismantled-machine-learning-bias-criminal-justice/) - [Machine bias risk assessments in criminal sentencing](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) - [Anti-racism, algorithmic bias, and policing: a brief introduction](https://towardsdatascience.com/anti-racism-algorithmic-bias-and-policing-a-brief-introduction-bafa0dc75ac6). - [Algorithmic Solutions to Algorithmic Bias: A Technical Guide](https://towardsdatascience.com/algorithmic-solutions-to-algorithmic-bias-aef59eaf6565). - [Cognitive Bias](https://cassandraxia.com/cogbiases/) ### Books

[The Alignment Problem](https://brianchristian.org/the-alignment-problem/) Brian Christian ISBN-13: 978-0393635829 ### Research papers [When Machine Learning Goes Off the Rails](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3881422). Boris Babic. Independent I. Glenn Cohen. Harvard Law School 202139:T558,3a:[{"id":"40d8ee2cba6618b934a63b9f01c0633b584386a1d17837465b032d59cb7a381419ce9802b7af5946c4b12841d8dddcc02b2744da1abb6ae93eb3ecf3fa712b92","href":"https://static.observableusercontent.com/files/40d8ee2cba6618b934a63b9f01c0633b584386a1d17837465b032d59cb7a381419ce9802b7af5946c4b12841d8dddcc02b2744da1abb6ae93eb3ecf3fa712b92","name":"im.png","type":"image/png","lastModified":1634029092519.331,"size":449052},{"id":"4293f5e4b88f90fa9e5c4133cc4c924e5de3d6f5bc9c8c46f6c7d7024115ec97f6597cfe4ceaeb418f9c87d5722b34ac7167bd23cd0fc889ddc4b4ba28a5e70f","href":"https://static.observableusercontent.com/files/4293f5e4b88f90fa9e5c4133cc4c924e5de3d6f5bc9c8c46f6c7d7024115ec97f6597cfe4ceaeb418f9c87d5722b34ac7167bd23cd0fc889ddc4b4ba28a5e70f","name":"Algorithm_Bias_Baer.jpeg","type":"image/jpeg","lastModified":1632151993929.992,"size":364945},{"id":"7da358cefc0f968175efad60fcbc1b19c8b90eff0414b17b6ca2ffe2035f16d8b2d240f09255401c952a1143f55c769c24e146e921b965c34daafe3b1fc2d876","href":"https://static.observableusercontent.com/files/7da358cefc0f968175efad60fcbc1b19c8b90eff0414b17b6ca2ffe2035f16d8b2d240f09255401c952a1143f55c769c24e146e921b965c34daafe3b1fc2d876","name":"align.jpeg","type":"image/jpeg","lastModified":1632901859291.747,"size":95351}] 2f:{"id":"40142e2d05a2c116","slug":"algorithmic-biases","title":"Algorithmic bias","thumbnail":"449ae6433ec63f0acc28df9d654b5e09835f26d334497c296e2cf50b3f9d8d5e","access_level":"public","create_time":"$D2021-09-01T14:14:02.264Z","update_time":"$D2023-11-28T12:32:01.684Z","view_time":null,"first_public_time":"$D2021-09-01T14:23:39.311Z","fork":{"id":"f3fe1705621dea2b","slug":"recommender-systems","title":"Recommender systems","owner":{"id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz"}},"stars":2,"starred":false,"owner":{"type":"team","id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/e6a2480f89a40dcfbe75f19d5081ae2296848971c4b934ed0499fd3b7f1973eb","tier":"starter_2024"},"version":1598,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"3d1beb3a90fbfe98","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/9b19f1a074469dcaafa9bd24c7b08f4bc969889b551fe70c209497f1d03c6514","tier":null}],"members":[{"type":"individual","id":"3d1beb3a90fbfe98","login":"sandraviz","name":"Sandraviz","avatar_url":"https://avatars-next.observableusercontent.com/9b19f1a074469dcaafa9bd24c7b08f4bc969889b551fe70c209497f1d03c6514","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":2,"collection":{"id":"eab5d593efbf8501","slug":"ai","title":"AI ","owner":{"id":"6661932e6ac85aef","login":"sandraviz","name":"Sandraviz"}},"type":"notebook","body":{"title":"Algorithmic bias","cells":[{"id":0,"value":"Decision_Support = md`# Algorithmic bias\n\nIn 2021 and 2022 I taught remotely at the **University Arcada (Finland)** responsible and ethical uses of machine learning practice, which is a fundamental precondition to trustworthy development of AI. You can find all the material and links in this [Github Repository](https://github.com/sandravizz/Decision-support-systems-verification-Python).\n\n** The road to a civilized world goes through the dark woods of biases.** - Abhijit Naskar\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":181,"value":"Index = toc(\"h2, h3, h4, h5\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1393,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1521,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1194,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1409,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1315,"value":"### Complexity of algorithmic bias\n\nThough well-designed algorithms frequently determine outcomes that are equally (or more) equitable than the decisions of human beings, cases of bias still occur, and are difficult to predict and analyze. The complexity of analyzing algorithmic bias has grown alongside the complexity of programs and their design. Decisions made by one designer, or team of designers, may be obscured among the many pieces of code created for a single program; over time these decisions and their collective impact on the program's output may be forgotten.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1251,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1240,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1427,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1262,"value":"### Use case: Google search \n\nWhile users generate results that are \"completed\" automatically, Google has failed to remove sexist and racist autocompletion text. For example, Safiya Noble notes an example of the search for \"black girls\", which was reported to result in pornographic images. Google claimed it was unable to erase those pages unless they were considered unlawful.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1297,"value":"#### The Autocomplete Truth\n\nA series of ads, developed as a creative idea for UN Women by Memac Ogilvy & Mather Dubai, uses genuine Google searches to reveal the widespread prevalence of sexism and discrimination against women. Based on searches dated 9 March, 2013 the ads expose negative sentiments ranging from stereotyping as well as outright denial of women’s rights. [link](https://www.unwomen.org/en/news/stories/2013/10/women-should-ads)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1270,"value":"","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"},{"id":1305,"value":"##### How Algorithms Know What You’ll Type Next \n\nTo be able to make useful predictions, a text predictor needs as much knowledge about language as possible, often done by machine learning. We will look at a simple yet effective algorithm called k Nearest Neighbours. This works by looking at the last few words you wrote and comparing these to all groups of words seen during the training phase. It outputs the best guess of what followed groups of similar words in the past. [link](https://pudding.cool/2019/04/text-prediction/)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":1494,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":179,"value":"import {toc} from \"@nebrius/indented-toc\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":1184,"value":"$39","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined"}],"files":"$W3a","stdlib":"1","titling":"auto"}} 30:"$2f:body"