2b:null 2c:["$","title",null,{"children":"Working with ObservableHQ | Cade Kane | Observable"}] 31:T5cf,If we are not sure about which column to use, we can scan all the values for empty strings and add a new column marking each record if it is "good" or "bad". Here is an example of using _map_ to create that additional column:

data_diff = data.map(x => 
                     {
                       let x_copy = { ...x};
                       for (let key in x_copy) 
                        { if (x_copy[key] == "") 
                          {
                            x_copy.bad = true; 
                            break;
                          }
                        }
                       if (!('bad' in x_copy)) x_copy.bad = false;
                      return x_copy;
                     })

In this example, I chose not to change the objects in the _data_ array by adding another another property (often, it would be desirable to change them directly) but to create copies for the objects. In this code `let x_copy = { ...x};` I use the [spread operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax) which will 'unravel' all of the keys and values of the original object, then the brackets 'collect' them back into a new object. Cade Notes: - x_copy.bad creates a new key in the object/dictionary, that is true or false. - the second if statement checks if I indeed have a key called bad in x_copy - meaning I did not create the key, and it is not a bad object/dictionary (no missing values)32:[{"id":"33c1528a757659d321afd7d77928134bd6897bb59c7f60849da5ccaaa94aa4ac78cb0cc56172d619170a74d983b4c3656c61a240a13c2877a5f8da9542e8bb3c","href":"https://static.observableusercontent.com/files/33c1528a757659d321afd7d77928134bd6897bb59c7f60849da5ccaaa94aa4ac78cb0cc56172d619170a74d983b4c3656c61a240a13c2877a5f8da9542e8bb3c","name":"fileattach-1.png","type":"image/png","lastModified":1726775056543.143,"size":225633},{"id":"a715ceae46c7a5660adfdc73cb51c4ad4167ee1801cd2c7f0e1f53cf0663d34f58ea72f64b040f68e7c7f57e74ffc891624b7abc8bbbe6f9210c25374d0a751a","href":"https://static.observableusercontent.com/files/a715ceae46c7a5660adfdc73cb51c4ad4167ee1801cd2c7f0e1f53cf0663d34f58ea72f64b040f68e7c7f57e74ffc891624b7abc8bbbe6f9210c25374d0a751a","name":"SurveyResponses24.csv","type":"text/csv","lastModified":1726775056543.143,"size":4437}] 2f:{"id":"5c631cf3d584425e","slug":null,"title":"Working with ObservableHQ","thumbnail":"273936f553ed51edf4e56f85db797dad7ccca83ad3f5a9ba9a741a0cb103586c","access_level":"public","create_time":"$D2024-09-19T19:44:16.543Z","update_time":"$D2024-09-19T20:17:10.672Z","view_time":null,"first_public_time":"$D2024-09-19T19:44:16.543Z","has_unpublished_edits":null,"fork":{"id":"79906d113a06b6db","slug":null,"title":"Working with ObservableHQ","owner":{"id":"a03ba8d7c36f4ece","login":"nurki-kirshenbaum-ws","name":"Nurki Kirshenbaum's Workspace"}},"stars":0,"starred":false,"owner":{"type":"team","id":"b11cd9b6338a06c7","login":"cade-kane","name":"Cade Kane","avatar_url":"https://avatars-next.observableusercontent.com/09a42aa5084e45c92da788171f2ad373a706f624ead90136d53b5122e9f5aea4","tier":"starter_2024","bio":""},"version":152,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"3ea30de20c0f2944","login":"cade-kane","name":"Cade Kane","avatar_url":"https://avatars-next.observableusercontent.com/e2b35fa9749047a6c6a29b0c8531e67e03e15c7c5afa6c6091142bd4f0934b46","tier":null,"bio":""}],"members":[{"type":"individual","id":"3ea30de20c0f2944","login":"cade-kane","name":"Cade Kane","avatar_url":"https://avatars-next.observableusercontent.com/e2b35fa9749047a6c6a29b0c8531e67e03e15c7c5afa6c6091142bd4f0934b46","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Working with ObservableHQ","cells":[{"id":0,"value":"# Working with ObservableHQ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"ObservableHQ is a [computational notebook](https://en.wikipedia.org/wiki/Notebook_interface) written for Javascript programming, primarily with the D3js library for visualizations.\n\nObservable lets you create text cells that use [Markdown](https://res.cloudinary.com/practicaldev/image/fetch/s--U_TEB7k7--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/i/5zhubbpov3m3ly9a1t9c.png) (like this cell) or Javascript cells. Note that there are a few differences between working within a cell and a regular script file. They provide some information about the differences in this [collection of Observable Notebooks](https://observablehq.com/@observablehq/observable-javascript?collection=@observablehq/javascript-in-observable). You can also create cells in HTML format which can be useful for embedding videos.\n\nIn addition, you can import [data files](https://observablehq.com/@observablehq/introduction-to-data) directly into the notebook.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":7,"value":"Every cell as a source box and an output box. When the source box is visible it is considered __*pinned*__ and it can be hidden by unpinning (see the thumbtack icon next to a source box or the menu expanded from the three dot menu in the output box.)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":9,"value":"

In addition

You can create cells that are HTML and put HTML tags in them.

Like this cell.

Note how it is wrapped in a div.

\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":14,"value":"# Get Some Data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":16,"value":"If we need to, we can define data directly in a cell, for example:\n```\npowers = [\n {name: \"Invisibility\", value: 7},\n {name: \"Flying\", value: 9},\n {name: \"Super strength\", value: 2},\n {name: \"Telekinesis\", value: 12},\n {name: \"Talk to animals\", value: 2}\n]\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":19,"value":"powers = [\n {name: \"Testing\", value: 7},\n {name: \"Flying\", value: 9},\n {name: \"Super strength\", value: 2},\n {name: \"Telekinesis\", value: 12},\n {name: \"Talk to animals\", value: 2}\n]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":24,"value":"We can also upload files to the notebook. I used the paperclip icon on the menu to the right of the notebook to add the file attachment _SurveyResponses.24.csv_.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":21,"value":"image = FileAttachment(\"fileattach-1.png\").image({width: 300, style: \"border: 1px solid black\"})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":28,"value":"We can read the file we attached into an array using\n```\nfromFile = FileAttachment(\"SurveyResponses24.csv\").csv()\n```\nThis will turn every row to an element in the array that is itself an array of strings (the values from the file). \n\n[More about file attachments.](https://observablehq.com/@observablehq/file-attachments)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":31,"value":"fromFile = FileAttachment(\"SurveyResponses24.csv\").csv()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":35,"value":"But we can also include the content of the file directly into the body of the notebook as a table by finding the file in the _Files_ panel, clicking the ... menu, and choosing _insert into notebook_.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":37,"value":"surveyresponses24 = await FileAttachment(\"SurveyResponses24.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"surveyresponses24","database":"$undefined"},{"id":38,"value":"When the data is inserted in this way, we can use direct manipulation on the table to filter and slice the data.\nYou can see that by default it is saved to a variable with the file name `surveyresponses24`. You can use this as a variable throughout the notebook. \n\n(Note that it takes into account filters, if you want, you can duplicate this table, apply a filter and change the name of the variable to indicate it is filtered.)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":44,"value":"You can also use the D3 functions for reading data. For example, to read the file from a remote repository you can use:\n\n`data = d3.csv(\"https://raw.githubusercontent.com/nuritk/data/main/SurveyResponses24.csv\")`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":47,"value":"data = d3.csv(\"https://raw.githubusercontent.com/nuritk/data/main/SurveyResponses24.csv\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"# Working with Data","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":51,"value":"Now that we have the data in the notebook we can manipulate it. For example, let's look at the column headers. We can do that by getting the keys of the first element in our data array. I edited the names of the columns in our original file so they are more managable.\n\n`Object.keys(data[0])`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":53,"value":"Object.keys(data[0])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":56,"value":"We may want to clean our data. In this case, I noticed there are several records that are missing values, for example for the \"Lat\" column. We can remove those incomplete records using a _filter_ operation:\n\n`CleanedData = data.filter(x => x.Lat != \"\")`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":61,"value":"CleanedData = data.filter((x) => x.Lat != \"\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":63,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":70,"value":"data_diff = data.map(x => \n {\n let x_copy = { ...x};\n for (let key in x_copy) \n { if (x_copy[key] == \"\") \n {\n x_copy.bad = true; \n break;\n }\n }\n if (!('bad' in x_copy)) x_copy.bad = false;\n return x_copy;\n })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":72,"value":"It turns out that there are 4 bad records and not 3, so I can clean the data by filtering using the new \"bad\" column.\n\n`CleanerData = data_diff.filter(x => !x.bad)`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":74,"value":"CleanerData = data_diff.filter(x => !x.bad)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":78,"value":"Now I can use _CleanerData_ to calculate something. For example, let's calculate the total commute time for everyone in the class. We can use _reduce_ to do that.\n\n```TotalCommute = CleanedData.reduce((total, x) => \n total + parseInt(x.Commute), 0)```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":82,"value":"TotalCommute = CleanedData.reduce((total, x) => \n total + parseInt(x.Commute), 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":84,"value":"Alternatively, we can use one of [D3's array functions](https://d3js.org/d3-array) to do the same thing.\n\n`d3.fsum(CleanedData, (d) => d.Commute)`","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":86,"value":"d3.fsum(CleanedData, (d) => d.Commute)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"We can do more complicated things. The code below, for example, will create an object with the different superpowers as keys, and the count of students that chose them as values.\n\n

\nsuperpowerCount = {\n  let superpowers = CleanerData.map(x => x.Superpower)\n  let unique = [...new Set(superpowers)]; // every value should only appear once\n  let count = unique.map(x => \n                         CleanerData.reduce( \n                           (count, record) => {\n                             if (record.Superpower == x) \n                               count++; \n                             return count;\n                           }, 0))\n  return unique.reduce ( (obj, x, index) => {obj[x] = count[index]; return obj}, {});\n}\n

\n\nCan you follow the use of _map_ and _reduce_ above?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":92,"value":"superpowerCount = {\n let superpowers = CleanerData.map(x => x.Superpower)\n let unique = [...new Set(superpowers)]; // every value should only appear once\n let count = unique.map(x => \n CleanerData.reduce( \n (count, record) => {\n if (record.Superpower == x) \n count++; \n return count;\n }, 0))\n return unique.reduce ( (obj, x, index) => {obj[x] = count[index]; return obj}, {});\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":88,"value":"# Practice","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":96,"value":"A. Use _filter_ to find the records of students that like both Soda and Sports.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":129,"value":"CleanerData","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":131,"value":"CleanerData.filter((x) => {\n return (x.Soda == \"Yes\" && x.Sports == \"Yes\")\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":99,"value":"B. Find the average Age of students in our class.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":136,"value":"d3.mean(CleanerData, (d) => d.Age)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":103,"value":"C. Derive a new column called _BagColorBinary_ which is marked with \"black\" when the bag has any black in it or \"not\" for every other color.\n\nYou can use a condition like `if (x.BagColor.includes(\"black\") || x.BagColor.includes(\"Black\"))` to check if a string contains the substring for \"black\".","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":144,"value":"CleanerData.map((x) => {\n if (x.BagColor.includes(\"black\") || x.BagColor.includes(\"Black\")) {\n x.BagColorBinary = \"black\"\n }\n else {x.BagColorBinary = \"not\"}\n})\n// confused which function to use!","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":110,"value":"D. Count how many students were born in each month.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":112,"value":"E. Use `d3.min()` and `d3.max()` to find the range of Heights in the class.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"}],"files":"$W32","stdlib":"1"}} 30:"$2f:body"