2e:null 2f:["$","title",null,{"children":"Data Science for Aspiring Professionals - Details | Srinivas Rampalli | Observable"}] 34:[] 32:{"id":"58ac0d0d8209c869","slug":"data-science-for-aspiring-professionals-details","title":"Data Science for Aspiring Professionals - Details","thumbnail":"5e5c78f731a18d17f573baca612d80893e0e092d7e9b54842165ee5f08cc58af","access_level":"public","create_time":"$D2020-04-19T06:45:40.770Z","update_time":"$D2020-04-19T19:07:24.490Z","view_time":null,"first_public_time":"$D2020-04-19T19:07:54.930Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"57ba91783a6a250f","login":"rampalli-github","name":"Srinivas Rampalli","avatar_url":"https://avatars-next.observableusercontent.com/fd315dac01a3e313b27f55bbf9a5919d1d36c3b775e19582abf8da9a3db82dc4","tier":"starter_2024"},"version":357,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"64df3cbc722323a9","login":"rampalli-github","name":"Srinivas Rampalli","avatar_url":"https://avatars-next.observableusercontent.com/fd315dac01a3e313b27f55bbf9a5919d1d36c3b775e19582abf8da9a3db82dc4","tier":null}],"members":[{"type":"individual","id":"64df3cbc722323a9","login":"rampalli-github","name":"Srinivas Rampalli","avatar_url":"https://avatars-next.observableusercontent.com/fd315dac01a3e313b27f55bbf9a5919d1d36c3b775e19582abf8da9a3db82dc4","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Data Science for Aspiring Professionals - Details","cells":[{"id":0,"value":"md`# Data Science for Aspiring Professionals - Details`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"html `In the earlier article Introduction we had looked at the basics of what constitutes data science and few examples. Time to go a little bit into the details.` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"md `What are the different things you can do when you have *data* and *relevant tools* with you?`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":15,"value":"dot`digraph { \"Explore\" -> \"One off Insight\" -> \"Document\"; \"Explore\" -> \"Repeatable Insight\" -> \"Scheduled Insight Generator\"; \"Explore\" -> \"Predictable Insight\" -> \"Prediction event generator\"}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"md `As an example, let us look at the [titanic dataset](https://github.com/datasciencedojo/datasets/blob/master/titanic.csv). First thing we would like to know is what data is available. A certain summary of data, what columns are available, how many rows of data is available. If it is spatial data, other considerations may apply, for now we are dealing with non-spatial data in this case.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"table(titanicData.slice(0,3))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"md `**First technique** is to look at the first few rows of the data in a tabular form (3 in this case to keep it simple). It may so happen that there are too many columns *(happens when denormalized data is taken for exploration, large number of columns end up in the dataset and it becomes very difficult to understand)*`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":55,"value":"md `Now that we took the first 3 rows in the data, we get a picture of what the dataset contains. From here emerge our first set of questions *pertaining to understanding the data itself*!` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":61,"value":"html `

What do those columns mean?
What does the value in the column mean? For example, the name column may be self-evident (although assuming anything with respect to the dataset can easily lead to misleading conclusions) , others like Survived seems to have 0 and 1, now what does 0 mean? and what does 1 mean?

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":76,"value":"md `Answers to these questions would usually come from the data dictionary that the source of the dataset provides. If appropriate answers are not available, we could still explore but those datasets are *not reliable* to draw any kinds of insights because everything could change depending on the way the information needs to be interpreted.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":82,"value":"html `The data dictionary for this dataset is available here kaggle`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":88,"value":"md `Now the next question usually is to get a summary of the data. How many columns are there? what are their data types? If they are numbers what are their value ranges and so on.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":102,"value":"md `Total number of rows: **${titanicData.length}**\n\nTotal number of columns: **${titanicData.columns.length}**\n\nNow for each column, let us get some information.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":173,"value":"table(groupedData)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":182,"value":"md `Another issue is datatypes, in this case all are detected as strings. Other than that, we have 12 columns, of which couple of columns have ${titanicData.length} distinct values same as the number of rows in the data. Others have varying values depending on how many of them repeat. \n\nIf the datatypes were detected, we could also find the ranges of the values that these columns take`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":190,"value":"md `Basic information about the data has been obtained, now we can try to explore the data. Usually each dataset has a certain prominent aspect that would be the initial object of analysis. In this case, the prominent aspect is Survival, 0 means the person did not survive, 1 means the person survived. The initial questions that someone would want to know is to find if there was a pattern that can be identified with respect to survival `","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":239,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":343,"value":"md `A groupBy on the dataset gives us this graph of those who could survive and those who could not. This dataset has been analysed so many times in various ways, let us leave it here and not do further analysis on this. In the next article we would go into other details related to the exploration and insights with other datasets.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":179,"value":"md ` **----- content below is code to make the above document interactive and live ---------**`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"titanicData = d3.csv(\"https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":94,"value":"groupedData = titanicData.columns.map(g => {return {name: g, value: d3.rollup(titanicData,v=>v,d=>d[g])}}).map(h=> {return {name: h.name, distinctValues: h.value.size, dataType: toType(h.value.keys().next()['value'])}})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":12,"value":"dot = require(\"@observablehq/graphviz@0.2\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":29,"value":"d3 = require(\"d3@5\", \"d3-array@2\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"import {table} from \"@tmcw/tables@513\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":113,"value":"toType = function(obj) {\n return ({}).toString.call(obj).match(/\\s([a-zA-Z]+)/)[1].toLowerCase()\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":253,"value":"html ``\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":206,"value":"BriteCharts = require('https://cdn.jsdelivr.net/npm/britecharts@2.15.1/dist/bundled/britecharts.min.js')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":234,"value":"survival = d3.rollup(titanicData,v=>v.length,d=>d['Survived']);","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":308,"value":"survivalDataForBar = Array.from(survival.keys()).map(f=> {return {name: surviveOrExpire(f), value: survival.get(f)}});","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":288,"value":"container = d3.select('#chart');","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":290,"value":"barChart = BriteCharts.bar();","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":313,"value":"surviveOrExpire = function(f){\n if(f == 0) return 'Expired'\n else return 'Survived'\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":268,"value":"container.datum(survivalDataForBar).call(barChart);\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 33:"$32:body"