2b:null 2c:["$","title",null,{"children":"Movie viewers' preferences on Documentary and Fantasy have the least correlation with other genres | Fengrui Gan (Simon) | Observable"}] 31:T5f7,## What ### Data The dataset is the [100k MovieLens dataset](https://grouplens.org/datasets/movielens/100k/) that is used by some machine learning developers to train their models. This dataset is relatively old, being release in 1998, but it is the only dataset that contains user information such as their age and occupation, which I originally intended to explore, but failed to meaningful significant insight from them. ### Dataset type The entire dataset contains 3 main tables, the ratings data, the movies data, and the users data. The ratings data is a temporal table and the movies and users data are just tables. ### Attributes #### Ratings - **UserId** (_Categorical_): The id of the users, numbered consecutively from 1. - **itemId** (_Categorical_): The id of the movies, numbered consecutively from 1. - **rating** (_Quantitative_ _Sequential_): The movie rating given by the users. Ranged from 1 to 5 #### Movies - **movie id** (_Categorical_): The id of the movies, numbered consecutively from 1. - **genres** (_Categorical_): There are 19 fields corresponding to 18 genres and 1 for unknown genre. A value of 1 indicates the movie is of that genre, a 0 indicates it is not. Movies can be in several genres at once. ### Derived Attributes - **averageRatingOfUser** (_Quantitative_ _Sequential_): The average ratings of the user on each genre. Derived by mapping the user' rating of each movie onto its respective genres and taking an average over the number of times the user has rated on respective genre.32:T665,## Example Insight 1. Users from the same states rated movies similarly. - I will need the _averageRatingOfUser (quantitative)_ and the _state (categorical)_ - To show how closely the users have rated the movies, I will need to create clustering of users based on their movie ratings, and I need to have a way to connect users based on state. So a force-directed node-link graph could work, with users as nodes, and the states as links. 2. Users who are engineers, programmers, and scientists rated movies similar to each other, while users who are artists, writers, and educators rated similar to each other. - I will need the _averageRatingOfUser (quantitative)_ and the _occupation (categorical)_ - To show how closely the users have rated the movies, I will need to create clustering of users based on their movie ratings, and I need to have a way to connect users based on their occupations. So a force-directed node-link graph could work, with users as nodes, and the occupation as links. Or an adjacency matrix with the rating similarity as the connection value might work. 3. Users rated older movies more similarly but rated on newer movies more differently. - I will need the _averageRatingOfUser (quantitative)_ and the _occupation (categorical)_ - To show how closely the users have rated the movies, I will need to create clustering of users based on their movie ratings. An adjacency matrix with the rating similarity as the connection value might work. Then I can use faceting to build adjacency matrices for movies of different ranges of release date. ### Insight choice I will attempt to implement insight 233:T455,## How For [genreView](https://observablehq.com/d/aab42d3bb8179e95#genreView) Marks: - point Channels: |Attributes|Type & Direction|Channel|Rationale| |---|---|---|---| |userId|Categorical|Horizontal position, Separate Order and Align|Since the userId is categorical, one should use SOA| |genre|Categorical|Vertical position, SOA|Same as user, genre is categorical so one should use SOA| |rating|Ordinal, divergent|Hue and Saturation|Because the scale is divergent, I used different hues to signify the two directions, and saturation to signify the distance from the logical middle point| For [similarityPlot](https://observablehq.com/d/aab42d3bb8179e95#similarityPlot) Marks: - point Channels: |Attributes|Type & Direction|Channel|Rationale| |---|---|---|---| |userId|Categorical|Vertical and Horizontal position, Separate Order and Align|Since the userId is categorical, one should use SOA| |similarity|Quantitative, divergent|Hue and Saturation|Because the scale is divergent, I used different hues to signify the two directions, and saturation to signify the distance from the logical middle point|34:T1845,getState = (zipString) => { /* Ensure param is a string to prevent unpredictable parsing results */ if (typeof zipString !== 'string') { console.error('Must pass the zipcode as a string.'); return; } /* Ensure we have exactly 5 characters to parse */ if (zipString.length !== 5) { console.error('Must pass a 5-digit zipcode.'); return; } /* Ensure we don't parse strings starting with 0 as octal values */ const zipcode = parseInt(zipString, 10); let st; let state; /* Code cases alphabetized by state */ if (zipcode >= 35000 && zipcode <= 36999) { st = 'AL'; state = 'Alabama'; } else if (zipcode >= 99500 && zipcode <= 99999) { st = 'AK'; state = 'Alaska'; } else if (zipcode >= 85000 && zipcode <= 86999) { st = 'AZ'; state = 'Arizona'; } else if (zipcode >= 71600 && zipcode <= 72999) { st = 'AR'; state = 'Arkansas'; } else if (zipcode >= 90000 && zipcode <= 96699) { st = 'CA'; state = 'California'; } else if (zipcode >= 80000 && zipcode <= 81999) { st = 'CO'; state = 'Colorado'; } else if ((zipcode >= 6000 && zipcode <= 6389) || (zipcode >= 6391 && zipcode <= 6999)) { st = 'CT'; state = 'Connecticut'; } else if (zipcode >= 19700 && zipcode <= 19999) { st = 'DE'; state = 'Delaware'; } else if (zipcode >= 32000 && zipcode <= 34999) { st = 'FL'; state = 'Florida'; } else if ( (zipcode >= 30000 && zipcode <= 31999) || (zipcode >= 39800 && zipcode <= 39999) ) { st = 'GA'; state = 'Georgia'; } else if (zipcode >= 96700 && zipcode <= 96999) { st = 'HI'; state = 'Hawaii'; } else if (zipcode >= 83200 && zipcode <= 83999 && zipcode != 83414) { st = 'ID'; state = 'Idaho'; } else if (zipcode >= 60000 && zipcode <= 62999) { st = 'IL'; state = 'Illinois'; } else if (zipcode >= 46000 && zipcode <= 47999) { st = 'IN'; state = 'Indiana'; } else if (zipcode >= 50000 && zipcode <= 52999) { st = 'IA'; state = 'Iowa'; } else if (zipcode >= 66000 && zipcode <= 67999) { st = 'KS'; state = 'Kansas'; } else if (zipcode >= 40000 && zipcode <= 42999) { st = 'KY'; state = 'Kentucky'; } else if (zipcode >= 70000 && zipcode <= 71599) { st = 'LA'; state = 'Louisiana'; } else if (zipcode >= 3900 && zipcode <= 4999) { st = 'ME'; state = 'Maine'; } else if (zipcode >= 20600 && zipcode <= 21999) { st = 'MD'; state = 'Maryland'; } else if ( (zipcode >= 1000 && zipcode <= 2799) || (zipcode == 5501) || (zipcode == 5544 ) ) { st = 'MA'; state = 'Massachusetts'; } else if (zipcode >= 48000 && zipcode <= 49999) { st = 'MI'; state = 'Michigan'; } else if (zipcode >= 55000 && zipcode <= 56899) { st = 'MN'; state = 'Minnesota'; } else if (zipcode >= 38600 && zipcode <= 39999) { st = 'MS'; state = 'Mississippi'; } else if (zipcode >= 63000 && zipcode <= 65999) { st = 'MO'; state = 'Missouri'; } else if (zipcode >= 59000 && zipcode <= 59999) { st = 'MT'; state = 'Montana'; } else if (zipcode >= 27000 && zipcode <= 28999) { st = 'NC'; state = 'North Carolina'; } else if (zipcode >= 58000 && zipcode <= 58999) { st = 'ND'; state = 'North Dakota'; } else if (zipcode >= 68000 && zipcode <= 69999) { st = 'NE'; state = 'Nebraska'; } else if (zipcode >= 88900 && zipcode <= 89999) { st = 'NV'; state = 'Nevada'; } else if (zipcode >= 3000 && zipcode <= 3899) { st = 'NH'; state = 'New Hampshire'; } else if (zipcode >= 7000 && zipcode <= 8999) { st = 'NJ'; state = 'New Jersey'; } else if (zipcode >= 87000 && zipcode <= 88499) { st = 'NM'; state = 'New Mexico'; } else if ( (zipcode >= 10000 && zipcode <= 14999) || (zipcode == 6390) || (zipcode == 501) || (zipcode == 544) ) { st = 'NY'; state = 'New York'; } else if (zipcode >= 43000 && zipcode <= 45999) { st = 'OH'; state = 'Ohio'; } else if ((zipcode >= 73000 && zipcode <= 73199) || (zipcode >= 73400 && zipcode <= 74999) ) { st = 'OK'; state = 'Oklahoma'; } else if (zipcode >= 97000 && zipcode <= 97999) { st = 'OR'; state = 'Oregon'; } else if (zipcode >= 15000 && zipcode <= 19699) { st = 'PA'; state = 'Pennsylvania'; } else if (zipcode >= 300 && zipcode <= 999) { st = 'PR'; state = 'Puerto Rico'; } else if (zipcode >= 2800 && zipcode <= 2999) { st = 'RI'; state = 'Rhode Island'; } else if (zipcode >= 29000 && zipcode <= 29999) { st = 'SC'; state = 'South Carolina'; } else if (zipcode >= 57000 && zipcode <= 57999) { st = 'SD'; state = 'South Dakota'; } else if (zipcode >= 37000 && zipcode <= 38599) { st = 'TN'; state = 'Tennessee'; } else if ( (zipcode >= 75000 && zipcode <= 79999) || (zipcode >= 73301 && zipcode <= 73399) || (zipcode >= 88500 && zipcode <= 88599) ) { st = 'TX'; state = 'Texas'; } else if (zipcode >= 84000 && zipcode <= 84999) { st = 'UT'; state = 'Utah'; } else if (zipcode >= 5000 && zipcode <= 5999) { st = 'VT'; state = 'Vermont'; } else if ( (zipcode >= 20100 && zipcode <= 20199) || (zipcode >= 22000 && zipcode <= 24699) || (zipcode == 20598) ) { st = 'VA'; state = 'Virginia'; } else if ( (zipcode >= 20000 && zipcode <= 20099) || (zipcode >= 20200 && zipcode <= 20599) || (zipcode >= 56900 && zipcode <= 56999) ) { st = 'DC'; state = 'Washington DC'; } else if (zipcode >= 98000 && zipcode <= 99499) { st = 'WA'; state = 'Washington'; } else if (zipcode >= 24700 && zipcode <= 26999) { st = 'WV'; state = 'West Virginia'; } else if (zipcode >= 53000 && zipcode <= 54999) { st = 'WI'; state = 'Wisconsin'; } else if ( (zipcode >= 82000 && zipcode <= 83199) || zipcode == 83414 ) { st = 'WY'; state = 'Wyoming'; } else { st = 'none'; state = 'none'; console.log('No state found matching', zipcode); } return st; }35:[{"id":"340628bb6824380c50a10df98d06dcc639c2a322f7619c85e9b06a6fd0ed38ee3af80bcf8f5b8e33a666f74b4f74ae6906e00da0d5acfc4d5a991e5bbc022e8b","href":"https://static.observableusercontent.com/files/340628bb6824380c50a10df98d06dcc639c2a322f7619c85e9b06a6fd0ed38ee3af80bcf8f5b8e33a666f74b4f74ae6906e00da0d5acfc4d5a991e5bbc022e8b","name":"u.genre","type":"application/octet-stream","lastModified":1679417231639.972,"size":202},{"id":"37307d8e18340ef6d46f3f86be406bcb2c971db4bbda11c9d3dbd9f5a75579272c3c4b6bd8e371d89ce220126bb5562b81fae4e7e5ef1dba1b57a91e4dce8c20","href":"https://static.observableusercontent.com/files/37307d8e18340ef6d46f3f86be406bcb2c971db4bbda11c9d3dbd9f5a75579272c3c4b6bd8e371d89ce220126bb5562b81fae4e7e5ef1dba1b57a91e4dce8c20","name":"u.data","type":"application/octet-stream","lastModified":1679417233628.519,"size":1979173},{"id":"ada9f961e305a0d0623701c7e5c23b9a83dff37e3ad748c59f542e01407ef0c1ddd07a83999396ebd8431006a8bc7b61d0e106f9ce8ee27828f72ac2dc64f3ec","href":"https://static.observableusercontent.com/files/ada9f961e305a0d0623701c7e5c23b9a83dff37e3ad748c59f542e01407ef0c1ddd07a83999396ebd8431006a8bc7b61d0e106f9ce8ee27828f72ac2dc64f3ec","name":"u.item","type":"application/octet-stream","lastModified":1679417231996.589,"size":236344},{"id":"8cae1e7a31432ad259f325a8f2a9b3853bfaf2135ba1772a8f9595a3664e451378a2c3c680bff995096edbdbc5f41c44e3e0f683dff8c9934726627f3e7810a8","href":"https://static.observableusercontent.com/files/8cae1e7a31432ad259f325a8f2a9b3853bfaf2135ba1772a8f9595a3664e451378a2c3c680bff995096edbdbc5f41c44e3e0f683dff8c9934726627f3e7810a8","name":"u.user","type":"application/octet-stream","lastModified":1679417231652.664,"size":22628}] 2f:{"id":"aab42d3bb8179e95","slug":null,"title":"Movie viewers' preferences on Documentary and Fantasy have the least correlation with other genres","thumbnail":"f38e9775aa319fe4ae5514ea3fae6e8213f4fbae956d4492ed53aeeaf97b4abf","access_level":"public","create_time":"$D2023-03-21T16:22:07.495Z","update_time":"$D2023-03-23T23:44:16.111Z","view_time":null,"first_public_time":"$D2023-03-21T16:22:07.495Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"a48bdab08888437c","login":"fengruigan","name":"Fengrui Gan (Simon)","avatar_url":"https://avatars-next.observableusercontent.com/73027a1b3f09dc8313d125b3d1f54464475e75f162e6d1db0259249352073c1d","tier":"starter_2024"},"version":1284,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"6d58559bc5b22c08","login":"fengruigan","name":"Fengrui Gan (Simon)","avatar_url":"https://avatars-next.observableusercontent.com/73027a1b3f09dc8313d125b3d1f54464475e75f162e6d1db0259249352073c1d","tier":null}],"members":[{"type":"individual","id":"6d58559bc5b22c08","login":"fengruigan","name":"Fengrui Gan (Simon)","avatar_url":"https://avatars-next.observableusercontent.com/73027a1b3f09dc8313d125b3d1f54464475e75f162e6d1db0259249352073c1d","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Movie viewers' preferences on Documentary and Fantasy have the least correlation with other genres","cells":[{"id":0,"value":"# Movie viewers' preferences on Documentary and Fantasy have the least correlation with other genres","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":852,"value":"### Average rating of each genre given by movie viewers\n\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1004,"value":"viewof sortby = Inputs.select([\"none\", ...genres], {label: \"Sort Genre\", value: \"none\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":564,"value":"genreView = {\n return vl\n .markRect()\n .encode(\n vl.y().fieldN(\"genre\"),\n vl.x().fieldN(\"userId\").axis({title: \"users\"}).sort(false),\n vl.color().fieldN(\"rating\").scale({scheme: \"redblue\"})\n )\n .data(overviewData)\n .width(600)\n .height(600)\n .render()\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1240,"value":"The ratings were originally scaled from 1 to 5, and 0 if no ratings were given. The scale has been adjusted to a scale of -2 to 2, and the original rating of 3 is considered the same as 0, meaning neutral.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":913,"value":"### Similarity between users\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":250,"value":"similarityPlot = vl\n .markRect()\n .encode(\n vl.x().fieldN(\"source.userId\").axis({title: \"Users\", labelExpr: \"\"}).sort({order: \"ascending\", op: \"average\", field: \"value\"}),\n vl.y().fieldN(\"target.userId\").axis({title: \"Users\", labelExpr: \"\"}).sort({order: \"ascending\", op: \"average\", field: \"value\"}),\n vl.color().fieldQ(\"value\").legend({title: \"Similarity\"}).scale({scheme: \"redblue\"}),\n vl.tooltip([\n vl.fieldN(\"source.occupation\"),\n vl.fieldN(\"source.state\"),\n vl.fieldQ(\"value\")\n ])\n )\n .data(links)\n .width(600)\n .height(600)\n .render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1243,"value":"The axes are users, and the values indicate the cosine similarity between users. Both axes are sorted in ascending order based on the average similarity of a user to others.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1245,"value":"- Created by [Fengrui Gan](https://fengruigan.github.io/fengruigan-homepage/)\n- Class [CS 7295: Special Topics Course / Information Visualization](https://johnguerra.co/classes/data_visualization_spring_2023/)\n- Professor Name : [John Alexis Guerra Gómez](https://johnguerra.co)\n- Video: https://youtu.be/Ys81jeeyaP4\n- tweet: https://twitter.com/FengruiGan/status/1639045124339695616","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1132,"value":"## Introduction\n\nThe MovieLens Dataset is a popular dataset used by many in attempts to create movie recommendation systems. So I have taken the opportunity to explore the dataset and try to create some interesting visualizations, hoping to get some insights to what we might take into account if we were to build a movie recommendation system.\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":125,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":116,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":140,"value":"## Why\nThe tasks are:\n- T1: Summarize the similarities of user occupations (_occupation_) based on their movie ratings (_rating_) (**Summarize / Similarity**)\n- T2: Identify any outliers in the similarities user occupations (_occupation_) and movie ratings (_rating_) (**Identify / Outlier**)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1262,"value":"## Target Audience\nThe visualization is for people who wants to develop movie recommendation systems or those who want to know more about how people's preferences towards movies or movie genres look like.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1252,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":920,"value":"# Actual Insights\n\nThe picture below shows the average ratings of users on each movie genre. From the plot we can see that the Fantasy genre is the most neutral genre, Documentaries is the second one. But most of the users seems to like all other genres evenly.\n\nThe second plot shows the similarity between users. We can also see that most people rated movies similar to each other. We can see some outlier users who rated the movies the opposite way from others, which are the highlighted yellow lines, but that still reflects that their ratings do not show a preference. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":70,"value":"## Data Processing\n\nBecause the original dataset is too large for visualization, I truncated the data to only consider a subset of users and movies","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1269,"value":"### Cosine similarity and vector operations","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":199,"value":"vecSum = (a, b) => {\n if (a.length != b.length) {\n throw Error(\"Two vectors have different lengths\") \n }\n return a.map((el, idx) => el + b[idx])\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":219,"value":"vecScale = (a, scalar) => a.map(el => el * scalar)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":496,"value":"vecNormalizeElToOne = (a) => a.map(el => el !== 0? 1 : 0)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":143,"value":"vecDot = (a, b) => {\n if (a.length != b.length) {\n throw Error(\"Two vectors have different lengths\") \n }\n return a.map((el, idx) => el * b[idx]).reduce((acc, val) => acc + val, 0)\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":151,"value":"cosSim = (a, b) => {\n return vecDot(a, b) / (Math.sqrt(vecDot(a, a)) * Math.sqrt(vecDot(b, b)));\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1276,"value":"### Nodes and \"links\"","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":238,"value":"nodes = users.filter(el => userSet.has(el.userId))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":241,"value":"links = {\n const arr = []\n for (let i = 0; i < userAverages.length; i++) {\n for (let j = 0; j < userAverages.length; j++) {\n arr.push({\n source: users.filter(el => Number(el.userId) == userAverages[i].userId)[0],\n target: users.filter(el => Number(el.userId) == userAverages[j].userId)[0],\n // value: cosSim(userAverages[i].ratings.map(el => el.rating), userAverages[j].ratings.map(el => el.rating))\n value: cosSim(userAverages[i].ratings.map(el => el.rating), userAverages[j].ratings.map(el => el.rating))\n })\n }\n }\n return arr\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1283,"value":"### Averaging and rating scale shifting","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":287,"value":"centerRating = (rating) => rating === 0? rating : rating - 3","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":988,"value":"grouping = genres.map(genre => d3.group(matrixData, d => d.genre).get(genre)).map(ratings => ratings.sort((a, b) => a.rating - b.rating)).reduce((acc, val, idx) => {\n acc[genres[idx]] = val\n return acc\n}, {})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1009,"value":"overviewData = sortby === \"none\"\n ? matrixData\n : [\n ...grouping[sortby],\n ...genres\n .filter((el) => el !== sortby)\n .map((el) => grouping[el])\n .reduce((acc, val) => {\n acc.push(...val);\n return acc;\n }, [])\n ]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":489,"value":"genreRatingCounts = mappedGenreRatings.reduce((acc, val) => {\n if (acc[val.userId] == null) {\n acc[val.userId] = new Array(genres.length).fill(0)\n }\n acc[val.userId] = vecSum(acc[val.userId], vecNormalizeElToOne(val.ratings))\n return acc\n}, {})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":190,"value":"mappedGenreRatings = truncatedData.map(rate => {\n const movie = items[Number(rate.itemId)]\n return {\n userId: rate.userId,\n ratings: genres.map(genre => centerRating(Number(movie[genre] * Number(rate.rating))))\n }\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":156,"value":"userSums = mappedGenreRatings.reduce((acc, val) => {\n if (acc[val.userId] == null) {\n acc[val.userId] = new Array(genres.length).fill(0);\n }\n acc[val.userId] = vecSum(acc[val.userId], val.ratings)\n return acc\n}, {})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":426,"value":"matrixData = userAverages.reduce((acc, val) => {\n const user = nodes.filter(node => node.userId === val.userId)[0]\n const curr = val.ratings.map(el => ({\n userId: val.userId,\n occupation: user.occupation,\n state: getState(user[\"zip code\"]),\n gender: user.gender,\n // rating: quantizeRating(el.rating),\n rating: d3.scaleQuantize([-2, 2], [-2, -1, 0, 1, 2])(el.rating),\n // rating: el.rating,\n genre: el.genre\n }))\n acc.push(...curr)\n return acc\n}, []).sort((a, b) => a.userId - b.userId || a.rating - b.rating)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":217,"value":"userAverages = Object.keys(userSums).map((id) => {\n const normalizedRatings = userSums[id].map(\n (el, idx) => genreRatingCounts[id][idx] === 0? 0 : el / genreRatingCounts[id][idx]\n );\n return {\n userId: id,\n ratings: normalizedRatings.map((rating, idx) => ({\n genre: genres[idx],\n rating: rating\n }))\n };\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1278,"value":"### Preprocessing raw and select subset","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":82,"value":"truncatedData = data.filter(el => userSet.has(el.userId) && itemSet.has(el.itemId))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":84,"value":"userNumber = 500","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":102,"value":"userSet = new Set(users.slice(0, userNumber).map(user => user.userId))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":471,"value":"truncatedUserIds = [...userSet]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":86,"value":"itemNumber = items.length","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"itemSet = new Set(items.slice(0, itemNumber).map(item => item[\"movie id\"]))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":55,"value":"items = processRaw(rawItems, \"\\n\", \"|\", `movie id | movie title | release date | video release date | IMDb URL | unknown | Action | Adventure | Animation | Children's | Comedy | Crime | Documentary | Drama | Fantasy | Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | Thriller | War | Western`.split(\" | \")).filter(el => el.unknown != 1)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":41,"value":"users = processRaw(rawUsers, \"\\n\", \"|\", \"userId | age | gender | occupation | zip code\".split(\" | \"))\n.slice(0, -200)\n// .filter(user => user.occupation !== 'other' && user.occupation !== 'none' && user.occupation !== 'retired')\n.map(el => ({\n state: getState(el[\"zip code\"]),\n ...el\n}))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"data = processRaw(rawData, \"\\n\", \"\\t\", \"userId | itemId | rating | timestamp\".split(\" | \")).sort((a, b) => (a.userId - b.userId) || (a.itemId - b.itemId)).slice(1)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":164,"value":"genres = processRaw(rawGenres, \"\\n\", \"|\", [\"genre\", \"idx\"])\n .slice(0, -2)\n // .reduce((acc, val) => {\n // acc[val[\"genre\"]] = Number(val[\"idx\"]);\n // return acc;\n // }, {})\n .map(el => el.genre).filter(el => el !== 'unknown')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":47,"value":"processRaw = (raw, rowSep, colSep, headers) => {\n return raw.split(rowSep).map(row => row.split(colSep)).map(row => {\n const rowOut = {}\n headers.forEach((header, idx) => {\n rowOut[header] = row[idx]\n })\n return rowOut\n })\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":28,"value":"rawData = FileAttachment(\"u.data\").text()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"rawUsers = FileAttachment(\"u.user\").text()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"rawItems = FileAttachment(\"u.item\").text()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":162,"value":"rawGenres = FileAttachment(\"u.genre\").text()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":450,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":233,"value":"netClustering = require(\"netclustering\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":313,"value":"forceInABox = require('force-in-a-box@1.0.0/dist/forceInABox.min.js').catch(\n () => window.forceInABox\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W35","stdlib":"1","titling":"auto"}} 30:"$2f:body"