2b:null 2c:["$","title",null,{"children":"Structure in data | johnx | Observable"}] 31:Tbb2,md`# Structure in data Data can be 'structured' or 'unstructured' - that is, organised in a standardised way, or not. The Data Management Association specifies that unstructured data "has not been tagged or otherwise structured into rows and columns or records." ([Geospatial Commission 2020](https://www.gov.uk/government/publications/geospatial-glossary/geospatial-glossary)) This is relevant when we try to automate the processing of data - computers work very well with structured data, but have a harder time processing unstructured data. Structured data organises information in a formal, consistent structure. These datasets are composed of 'records', individual 'observations' described in the data. These observations might represent any range of things - geographic features like road segments, train stations or postcodes; financial transactions; human beings - just about anything you can imagine, physical or intangible. Structured data comes in many forms, but it is often useful to imagine it as a table with rows and columns. In properly structured data - ['tidy' data](https://vita.had.co.nz/papers/tidy-data.pdf), as defined by [Hadley Wickham](https://twitter.com/hadleywickham) in 2014 - each row represents an individual _observation_ (or _record_), and each column represents a _dimension_ to the observation, or a _variable_ that describes it. If we think of a table where each observation represents a person, a tidy, structured dataset might look like this: | \`id\` | \`name\` | \`age\` | \`favorite_color\` | \`has_smartphone\` | | --- | --- | --- | --- | --- | | 0 | "Helena Espinosa" | \`29\` | "blue" | \`true\` | | 1 | "Dorian Beach" | \`17\` | "green" | \`false\` | | 2 | "Lukas Craft" | \`44\` | "purple" | \`false\` | | 3 | "Vishal Needham" | \`83\` | "grey" | \`true\` | _(Names randomly generated using [name-generator.org.uk](https://www.name-generator.org.uk/))._ As this data is structured in a standardised way, computer programs can parse and analyse this dataset much more easily than if the data was unstructured, like this: > "We met four people the other day. Helena was 28 before her birthday last week. She loves the color green, but her favorite is blue. She called Vishal on her smartphone - he's 83, and lives in Dorset. They met up with Dorian (17) and Lukas (44) at the park for a walk. Dorian and Lukas needed to use Vishal's smartphone for their walking directions, since they don't have smartphones. The three like green, purple and grey respectively." Both datasets contain the same information, but the structured data is organised in such a way that analysing the information programmatically is quite straightforward. Processes and techniques - like [natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing) - for using software programs to reliably extract these details from the unstructured dataset are maturing, but in general algorithms have an easier time handling structured data. `32:T504,md`### Data Types, Briefly A key insight: on computers, all of this information is represented in _binary format_. Data is encoded as sequences of **b**inary dig**its**, or **bits**. A bit is "a basic unit of information in information theory" ([Wikipedia](https://en.wikipedia.org/wiki/Bit#:~:text=The%20bit%20is%20a%20basic,a%20portmanteau%20of%20binary%20digit.)), a logical state with exactly two possible values. We can imagine these values as 1 or 0, yes or no, true or false, on or off, + or -. Sets of eight bits are called **bytes**. So how do sequences of bits represent the incredible complexity of information that can be accessed, analysed and visualised by computers? Computers work with data that has been encoded. Image, audio and video information, as well as characters like letters or other symbols, are converted in a standardised way to be represented as these binary sequences. An easy example: letters and numbers are encoded into binary using a character encoding standard, like ASCII or UTF-8. To convert the binary encoded data back into the characters represented, a computer just needs to have the encoding rules - it can then parse the binary sequence and derive the character sequence. So how would we represent a string of characters in binary? `33:T81e,md`Meaning - in UTF-8 encoding - "${textstring}" is stored and transmitted as the ${ textstring.length } byte binary sequence: ${stringToByteString(textstring)} _(Conversion performed in JavaScript based on [this StackOverflow answer](https://stackoverflow.com/questions/18729405/how-to-convert-utf8-string-to-byte-array).)_ UTF-8 is a global standard used across the Internet. Try pasting the sequence above into the "binary bits" field on [onlineutf8tools.com](https://onlineutf8tools.com/convert-binary-to-utf8). As long as the software has the UTF-8 encoding, it will correctly decode this binary sequence as "${textstring}". (Note that lower case and capital letters must be represented as different bytes.) Similarly, audio can be represented digitally by using audio encoding formats like MP3, AAC, FLAC. These formats encode the "audio information in terms of frequencies and amplitudes" ([Wikipedia](https://en.wikipedia.org/wiki/MP3)). Images can also be represented digitally in binary - we'll go over how shortly. Interestingly - _the same byte can have different meanings depending on the encoding!_ For example, the byte 0100 1111 might mean the character "O" in UTF-8 - but it also represents the unsigned integer 79. The computer processes each bit in this context - according to its 'data type'. In strongly-typed programming languages, when a variable is declared, developers need to declare the data type of that variable - uint, float, char, bool. Dynamically-typed languages like JavaScript and Python are more flexible, can be easier to code in, but are more error prone. This is because variables can be reassigned to a value of another type - and the computer can *usually* adapt accordingly. For the encoded information to be useful, it needs to be parsed and interpreted by a computer processor. This would include displaying the characters or images visually on a screen, or sending a signal to an audio speaker that would create the sounds encoded. `34:T4e8,md`### 🌐 Enter spatial So, what is _spatial data_? Spatial data is any data that contains information about where observations are. Usually, this means that **one of the dimensions associated with each observation describes that record's position in space.** So, if we look back at the table we saw earlier - but as a spatial dataset: | \`id\` | \`name\` | \`age\` | \`favorite_color\` | \`has_smartphone\` | \`coordinates\` | \`city\` | | --- | --- | --- | --- | --- | --- | --- | | 0 | "Helena Espinosa" | 29 | "blue" | true | [-5.930, 54.596] | "Belfast" | | 1 | "Dorian Beach" | 17 | "green" | false | [3.189, 55.953] | "Edinburgh" | | 2 | "Lukas Craft" | 44 | "purple" | false | [-3.183, 51.483] | "Cardiff" | | 3 | "Vishal Needham" | 83 | "grey" | true | [-0.128, 51.507] | "London" | _(Coordinates from Wikipedia.)_ By including the geographic coordinates in the dataset, it becomes spatial data. With this additional dimension, much deeper insights about the records in a dataset - and their relationships - can often be drawn. In fact, the spatial dimension is frequently the key to understanding relationships and answering questions about things an analyst might want to understand.`35:[] 2f:{"id":"8a5637f3e2a6d7e3","slug":"structure-in-data","title":"Structure in data","thumbnail":"6cd45eece2f35bf678ce20924fa865fcd33c6eeecb671f6193208278086306e7","access_level":"public","create_time":"$D2021-04-26T15:17:32.424Z","update_time":"$D2021-04-27T15:07:03.239Z","view_time":null,"first_public_time":"$D2021-04-27T15:03:47.990Z","fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"694a515e88970367","login":"johnx25bd","name":"johnx","avatar_url":"https://avatars-next.observableusercontent.com/2913455d7c8937d6db139dd4363a27c3d0ca15eb7cf0dc8a317fbde440d8a165","tier":"starter_2024"},"version":47,"publish_version":47,"role":null,"authored":false,"authors":[{"type":"individual","id":"a7851601ec4f2322","login":"johnx25bd","name":"johnx","avatar_url":"https://avatars-next.observableusercontent.com/2913455d7c8937d6db139dd4363a27c3d0ca15eb7cf0dc8a317fbde440d8a165","tier":null}],"members":[{"type":"individual","id":"a7851601ec4f2322","login":"johnx25bd","name":"johnx","avatar_url":"https://avatars-next.observableusercontent.com/2913455d7c8937d6db139dd4363a27c3d0ca15eb7cf0dc8a317fbde440d8a165","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":{"id":"6debf9a42039b3ad","slug":"a-guide-to-spatial-data","title":"A Guide to Spatial Data","owner":{"id":"694a515e88970367","login":"johnx25bd","name":"johnx"}},"type":"notebook","body":{"title":"Structure in data","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":13,"value":"viewof textstring = text({\n placeholder: \"A string of characters\",\n description: \"Type a sequence of characters here to see each in UTF-8 below.\",\n value: \"Ordnance Survey\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":16,"value":"md`${utf8table(textstring)}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":31,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":37,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"md`**Next, read about [Raster and Vector →](https://observablehq.com/@johnx25bd/raster-and-vector)**\n\n**← Or go back to the [table of contents](https://observablehq.com/@johnx25bd/a-guide-to-spatial-data)**`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"md`## Appendices`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":11,"value":"import { text } from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":20,"value":"md`### Functions`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"// From https://stackoverflow.com/questions/24337260/javascript-a-byte-is-supposed-to-be-8-bits - 🙏\nfunction byteString(n) {\n if (n < 0 || n > 255 || n % 1 !== 0) {\n throw new Error(n + \" does not fit in a byte\");\n }\n return (\"000000000\" + n.toString(2)).substr(-8);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":23,"value":"function utf8table(string) {\n let rows = [...string].map((c) => {\n return `| \"${c}\" | ${byteString(\n c.charCodeAt()\n )}| | ${c.charCodeAt()} |\\n`;\n });\n\n let table = `| Character | UTF-16 (binary) | UTF-16 (character code) | \n| --- | ---- | --- |\\n`;\n\n return table + rows.join(\"\");\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":33,"value":"function stringToByteString(string) {\n let bytes = \"\";\n for (let i = 0; i < string.length; i++) {\n bytes += byteString(string.charCodeAt(i)) + \" \";\n }\n return bytes.trim();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W35","stdlib":"1","titling":"auto"}} 30:"$2f:body"