2a:null 2b:["$","title",null,{"children":"Week 2 Exercise: Data Questions and Transformations (Due Sept. 17) | Najim Rhalmi | Observable"}] 30:T92a,# Week 2 Exercise: Data Questions and Transformations (Due Sept. 17) In this week's lecture, we focused on four core elements of data transformations: _binning_, _aggregation_, _filtering_ , and _normalization_. In the in-class lab session that accompanied the lecture, we only covered _binning_ and _aggregation_. In this week's exercise, we will be focusing on the other two: _filtering_ and _normalization_. Like in our in-class lab, we will be focusing on a movie dataset provided by Vega-Lite. Let's observe some other nuances hidden in the data.

[Note]

Some parts of this exercise will involve code concepts that are more globally common in programming, not just contained within Vega-Lite. This includes using string expressions to write out code logic (e.g. "x > 10") and some light JavaScript coding. We will do our best to provide summaries and documentation where necessary. Please read all instructions carefully.

## Assignment Instructions This assignment is due **September 17, 2026 @ 11:59pm**. We base our grading on the last updated timestamp of your notebook; please take care to not modify your notebook at any point after this submission time. Otherwise, please follow these instructions: 1. Fork this notebook to make sure that you are working on your own copy! 2. Fill your name and NetID in the code block below. **You will not receive credit if you do not fill this part properly.** 3. All required code exercises are marked with a [TO-DO]. Please fill these out to the best of your ability. - You may refer to either Week 2 Lab for reference or additional online resources such as documentation and public forums for help with these questions. - **Please be mindful of our academic integrity policies when completing this assignment.** 4. Make this notebook either public or unlisted. **You will not receive credit if you do not accomplish this step.** 5. On Brightspace, submit the URL to this notebook under the relevant assignment.31:T431,When it comes to plotting, you generally want to follow these guidelines: 1. Titles should give readers an idea of what the chart is depicting, such as if it is exploring a relationship between variables. 2. If possible, axes titles should also inform what units they are in (e.g. currency, meters) 3. Axes titles should indicate any transformations that have occurred (e.g. if an axes is the `log` value rather than the raw value). With Vega-Lite, we can add axes and chart titles using the `.title()` where appropriate. - If adding a chart title, add `.title()` at the same level as `.data()` and `.encode()`. - If adding an axis title, add `.title()` to each `vl.x()` or `vl.y()`. inside of `.encode()`. For example, you can add a axis title like so: ``` ... .encode( vl.x().fieldQ().title(), vl.y(). ... ) ... ``` - You can even add titles to your color legend with the same idea: ``` .encode( ..., vl.color().fieldN().title() ) ```32:T511,**Filtering** is a very common data transformation that removes unwanted data from analysis. We typically require some filtering to, for example, remove samples with missing or noisy data. We may also want to use filters to remove irrelevant data. This section will cover both use cases for filtering data from our movie dataset. In Vega-Lite, any modification to data requires `.transform()`, right next to `.data()` and `.encode()`. We'll add arguments into `.transform()` to tell Vega-Lite _how_ we want to transform the data. In this case, we want to call `vl.filter()` as an argument inside `.transform()`; this is similar to how we call `vl.x()` for any x-axis operations inside of `.encode()`. When you filter data using `vl.filter()`, this `` is actually something we call a **"string expression"**. Consider it actual code, just written in string form; the API will do the heavy-lifting of interpreting what the string is trying to say and work with it. An example: consider we want to remove all Drama movies from our `movies` dataset. Here, we would type something similar to the following (pay attention to the string used inside `vl.filter()`): ```javascript .data(...) .transform( vl.filter('datum.Major_Genre != "Drama"') ) .encode(...) .render()) ```33:T591,

Be careful about single and double quotations: Notice that because we're dealing with a string value ("Drama") inside of the string expression, we used single quotations here. Some programming languages such as JavaScript and Python allow you to use single quotes to define multi-character strings. However, most programming languages will consider single quotations as character data types (e.g. 'c', 'a').

If you are working with data whose feature names have spaces (such as our `movies` dataset, which uses feature names like "Major Genre" instead of something like "Major_Genre"), then you can use this alternative syntax: ``` vl.filter('datum["Major Genre"] != "Drama"') ``` Regardless of syntax, this expression effectively tells Vega-Lite to only include any samples where the `Major_Genre` is not "Drama". If you want to add multiple conditions, you can combine both conditions into a singular string expression, such as: ```javascript vl.filter('datum["Major Genre"] != "Drama" && datum.Director != null') ``` In this, not only are we filtering out dramatic movies but also we're making sure that the movie also has a `Director` listed. For more information, here's the documentation for `vl.filter()`: documentation.34:T492,### 2A. Simple Filter: Only Use Popular Movies... According to IMDb (8%) Let's start off with a very simple filter: only include movies where `IMDB Rating` is greater than or equal to `8.0`. Let's also use this time to get used to some **binning** and **aggregation**, which we covered in lecture.

[TO-DO]

Produce a bar chart that counts the number of movies per month so that:

New .transform() and vl.filter() methods make it so that we only look at movies where IMDB Rating is greater than or equal to 8.0. (3%)
The chart has a width of 500px and a height of 300px. (2%)
The chart's title reflects the updated filter with a new title: "IMDb Ratings >= 8.0". (1%)
The X-axis has a renamed axis title of "Release Month". (1%)
The Y-axis has a renamed axis title of "# of Records". (1%)

There is no need to modify the code provided inside of .encode() already.

Solution:

${await FileAttachment("e2_2a.png").image({width:500})}

35:T484,--- ### 2B. Filter by a Range (1970-1999) (9%) Let's use the new `vl.filter()` method to isolate movie "eras". In particular, we're interested only in movies made between 1970 and 1999 (1970 <= y < 2000)

[TO-DO]

Produce a bar chart that counts the number of movies made between 1970 and 1999. You must use .transform() and vl.filter() to create a filter, and then use year(datum["Release Date"]) to extract the year from the date feature (you can read more about Vega-Lite's TimeUnit operations if you're interested).

A bar chart is used. (1%)
.transform() and vl.filter() are used with the correct filter. (3%)
The X-axis represents each year and is titled "Release Year". (2%)
The Y-axis is renamed "# of Movies". (1%)
The chart is 600px wide and 300px tall. (2%)

Solutions:

${await FileAttachment("e2_2b.png").image({width:600})}

36:T630,---- ### 2C. The Curious Case of Rom-Coms Across Eras (12%) With this `movies` dataset at our disposal, alongside our newfound ability to filter the data that we use in charts, we can begin to really look deeper into individual trends across specific features. A curious case may be found in how people view and rate movies across genres. Do people attitudes towards different genres depend on which era the movies come out in? Let's look at the classic **"Romantic Comedy** (Rom-Com) as our case study.

[TO-DO]

Generate two charts that have Rotten Tomatoes Rating along the X-axis and IMDB Rating along the Y-axis. Both charts must follow these conditions:

Both charts must plot only Rom-Coms. (2%)
Both charts must only show movies that have BOTH Rotten Tomatoes and IMDb ratings (i.e. both are not null). (2%)
Both charts must only show movies with a release date (AKA don't consider movies that are missing their release dates). (2%)
Both charts must be 300px wide and tall. (2%)
The first chart specifically looks at Rom-Coms made between 1970 and 1999; the second chart looks at Rom-Coms made in and after 2000. (2%)
The first chart (1970-1999) must be colored "blue"; the second chart (2000+) must be colored "red". (2%)

Solutions

${await FileAttachment("e2_2c1.png").image({width:300})} ${await FileAttachment("e2_2c2.png").image({width:300})}

37:T60a,Here's an example of a simple operation that adds two columns (`A` and `B`) together to make a third column (`C`): ```javascript vl.markCircle() .data(...) .transform( vl.calculate('datum.A + datum.B').as('C') ) .encode(...) .render() ``` Let's take this logical feature to the next step and apply it to data. Let's generate a new feature called `Worldwide Profit` that is calculated by subtracting `Worldwide Gross` and `Production Budget`.

[TO-DO]

Generate a bar chart depicting release year on the X-axis and the median Worldwide Profit on the Y-axis. Use .calculate() and .as() to generate this new Worldwide Profit feature, by subtracting Production Budget from Worldwide Gross.

The chart is a bar chart. (1%)
Using .transform() and vl.filter(), any rows where either Worldwide Gross or Production Budget are null are filtered. (2%)
Using .transform() and vl.calculate(), a new Worldwide Profit data is calculated from Worldwide Gross and Production Budget. (2%)
The X-axis represents the release year. (1%)
The Y-axis represents the median Worldwide Profit. (1%)

Solution

${await FileAttachment("e2_3a.png").image({width:400})}

38:T755,--- ### 3C. Profit v.s. Movie Ratings - Are Modern Movies Good AND Profitable? (18%) With our `rated_movies` dataset, the burning question is thus: **Do well-rated movies (at least, by IMDb and Rotten Tomatoes standards) tend to also be profitable**? Let's find some solace by charting out a possible response. For this question, we define "modern" to be _movies made in or after 2000_.

[TO-DO]

Create a circle plot that compares Rotten Tomatoes Rating (X-axis) with IMDB Rating (Y-axis). (3%)
The data should only consider movies that were created in or after the year 2000. (2%)
The data should only consider movies that have both non-null US Gross and Production Budget values. (2%)
A new US Profit feature is calculated by subtracting Production Budget from US Gross. (2%)
Both the X- and Y-axes must use .bin() that sets maxbins to 50 bins for each axis. (4%)
Points are colored by the median US Profit. The color scheme must be set to "redblue" and have the domainMid set to 0. (4%)
The plot must be 500px wide and 400px tall. (1%)

Solution:

${await FileAttachment('e2_3c.png').image({width:500})}

[Note]

You can solve requirements #5 and #6 with some tips:

#5 requires .bin(), while #6 requires .scale()
Both .bin() and .scale() can have their behavior modified with the following syntax: .bin({ property:value, ...}).

39:T584,--- ## Part 4: Plain JavaScript + Normalization (30%) Data **normalization** is the practice of re-scaling data for more fair comparisons. This is likely to occur depending on situational contexts. Common examples include: - Scaling prices in the past to match today's inflation levels, to observe relative costs of goods and services compared to household incomes. - Health monitoring & scaling multiple patients' physiological conditions to standardized metrics. - Grading scales across different districts or inter-school classes, to either provide grading curves or identify poor-performing or well-performing classes. - Demographics in urban settings, such as voter populations or the number of diseased individuals relative to per-capita or population density. The most common type of data normalization is z-score normalization, or the number of standard deviations a raw value is above or below the mean of the data distribution. Z-scores allow us to standardize raw scores into a common ground, even if the distributions of two different features are different (e.g. they have different means and variances). In our `movies` dataset, we will use z-score normalization to create some new insights with our data. To do so, we need to tap into pure JavaScript and do some basic scripting outside of Vega-Lite.3a:T78e,--- ### 4C. Comparing Production Budgets and Running Times (10%) Let's now apply the same logic for z-score normalization with two features of different scales: Production Budgets, and Running Times. We are dealing with different units here ($ v.s. Minutes), so using z-score normalization should provide some interesting relationships between the two metrics.

[TO-DO]

Perform a dual operation creating a new variant of the movies dataset that calculates the z-scores of the production budget and running times of movies, then graphs these z-scores.

Create a new movies_normalized dataset that uses pure JavaScript to calculate the z-scores of both Production Budget (called Budget Z) and Running Time min (called Running Time Z. Prior to calculating z-scores, any rows with null values for the running time or production budget must be filtered out. (5%)
Create a scatter plot that plots Budget Z on the X-axis and Running Time Z on the Y-axis. (2%)
The scatter plot must be 400px tall and wide, and be titled "Budget and Running Time Distributions". (2%)
The scatter plot must have a tooltip that shows the title, raw budget, and raw running time of each movie. (1%)

[NOTE]

If you want to include multiple fields within a tooltip, simply pass an array with the features you want inside of the .tooltip([...]). For example, if you want to include both the title and director of a movie in a tooltip, you'd use .tooltip(["Title","Director"]) instead of just .tooltip().fieldN("Title").

Solution:

${await FileAttachment("e2_4c.png").image({width:400})}

3b:T46d,Indeed, the shape of the data relationship is generally the same. However, here are some things that the z-score version shows that this raw data chart does not: - **Mean and Standard Deviation**: The raw chart version does not give you any indication of how the movies relate to each other in relation to each metric's mean. We can't tell what the average movie budget and running time is, and consequently we can't analyze these movies based on their averaged qualities. - **Distributions Ground Visual Observations**: With the raw chart version, we can _roughly_ come to the conclusion that a majority of movies within the `movies` datasets have budgets less than $80,000,000; this can in turn indicate an average within this visual clump. The z-score version solidifies that assumption by directly informing viewers which movies are more or less expensive than the average (which happens to be around $48,000,000 on average). There are some genuinely unique and applicable reasons why z-scores can be highly informative; building up the intuition to know when to use them is something you can really only gain through practice.3c:[{"id":"d42e129d12ac3849ee3fdc7cf3cdcc9a2e466d6988af3b9080590810834c287492bda739f6a137a6a699f95572ad725845ece0b4ecd0d21f09aa746e5716bf9c","href":"https://static.observableusercontent.com/files/d42e129d12ac3849ee3fdc7cf3cdcc9a2e466d6988af3b9080590810834c287492bda739f6a137a6a699f95572ad725845ece0b4ecd0d21f09aa746e5716bf9c","name":"e2_2a.png","type":"image/png","lastModified":1788637528278,"size":51690},{"id":"ef1053b80e60ecf7aaeec165f1f2a1a512c0fa864e90aa0d6d8e8c80e307ebc921ed12eae87144f76f13764c683a7e2c818502f6a287a5cfff9a93a501eb27e3","href":"https://static.observableusercontent.com/files/ef1053b80e60ecf7aaeec165f1f2a1a512c0fa864e90aa0d6d8e8c80e307ebc921ed12eae87144f76f13764c683a7e2c818502f6a287a5cfff9a93a501eb27e3","name":"movies.json","type":"application/json","lastModified":1788562184319,"size":1399981},{"id":"e9f6eac339f33df90ec80c4e7c6dbef44282bd4929f44583bffb9dd1c5c0cdcb3377929d7ce95b0ccecf8a221c1eee3b5b73887783ec8727c0cbf8f7070e61ed","href":"https://static.observableusercontent.com/files/e9f6eac339f33df90ec80c4e7c6dbef44282bd4929f44583bffb9dd1c5c0cdcb3377929d7ce95b0ccecf8a221c1eee3b5b73887783ec8727c0cbf8f7070e61ed","name":"e2_3c.png","type":"image/png","lastModified":1788674342022,"size":46592},{"id":"5cd0496da7f60c450d45fb3c6c33161332ac60aec4762839d584084a602aef86849aa83489b6f115004bbf152c6b897a07e5235ecbb5db0113f667d8253fdeae","href":"https://static.observableusercontent.com/files/5cd0496da7f60c450d45fb3c6c33161332ac60aec4762839d584084a602aef86849aa83489b6f115004bbf152c6b897a07e5235ecbb5db0113f667d8253fdeae","name":"e2_4b.png","type":"image/png","lastModified":1788709530153,"size":20540},{"id":"fb66f2690635ee9fb05c635b800c8d0999acc713ba039133c02110dcddbe8eec96f9f4154353a28a6c8b9f3803762aa770f66a6f42a7c2e5122516f9d2090dc2","href":"https://static.observableusercontent.com/files/fb66f2690635ee9fb05c635b800c8d0999acc713ba039133c02110dcddbe8eec96f9f4154353a28a6c8b9f3803762aa770f66a6f42a7c2e5122516f9d2090dc2","name":"e2_4c.png","type":"image/png","lastModified":1788714987087,"size":48326},{"id":"c15966f54bc805f7d22f7f5f5924de8aa91c9c1c77c2e0e3c989ef695356c692ff9484413336f7c3503ec97d45003bdc129ca26d180ef14be6ab743ead6c684b","href":"https://static.observableusercontent.com/files/c15966f54bc805f7d22f7f5f5924de8aa91c9c1c77c2e0e3c989ef695356c692ff9484413336f7c3503ec97d45003bdc129ca26d180ef14be6ab743ead6c684b","name":"e2_2c1.png","type":"image/png","lastModified":1788663746489,"size":11102},{"id":"6d7a7b4f92e563f9609a14dbb2e31049ec392b2e958fef7c97c5782dbf59effe6e946293b7c48efd26129e597ce7b589b090c4cf2f8383f6f65ecca756c15280","href":"https://static.observableusercontent.com/files/6d7a7b4f92e563f9609a14dbb2e31049ec392b2e958fef7c97c5782dbf59effe6e946293b7c48efd26129e597ce7b589b090c4cf2f8383f6f65ecca756c15280","name":"e2_2c2.png","type":"image/png","lastModified":1788663754724,"size":13194},{"id":"4b2a9b0a4839e4ee9f33e8cdfb50f9b446918cb021eef2cfbe8d0719b3621b9028765dff46312d88d876799d26bb0fb4e99c1d8d007f08be947b0e480861577c","href":"https://static.observableusercontent.com/files/4b2a9b0a4839e4ee9f33e8cdfb50f9b446918cb021eef2cfbe8d0719b3621b9028765dff46312d88d876799d26bb0fb4e99c1d8d007f08be947b0e480861577c","name":"e2_3a.png","type":"image/png","lastModified":1788667238958,"size":25479},{"id":"7a80e38b20d26ff35cda808e1a1fdcb0c7b035995bf25b768c8fd48f6891f951bb196bcecd390c3ccb39d513214dbe94461694cb2ee2e08968489c7660a96e0a","href":"https://static.observableusercontent.com/files/7a80e38b20d26ff35cda808e1a1fdcb0c7b035995bf25b768c8fd48f6891f951bb196bcecd390c3ccb39d513214dbe94461694cb2ee2e08968489c7660a96e0a","name":"e2_3b.png","type":"image/png","lastModified":1788669039820,"size":11314},{"id":"8a8cf044ed09630012fb7e2da849ac0cfb7abdeca213311bf702549cff18c49fd305a13db27f25c4abd37fc1f2ad3aa7310d6552471d688b9b52f440b319f36e","href":"https://static.observableusercontent.com/files/8a8cf044ed09630012fb7e2da849ac0cfb7abdeca213311bf702549cff18c49fd305a13db27f25c4abd37fc1f2ad3aa7310d6552471d688b9b52f440b319f36e","name":"e2_1.png","type":"image/png","lastModified":1788635867613,"size":108064},{"id":"96a62d477a020aa59d44081ff413ac95f7fc6bc5f6492ba4614819cd638a896633f23c986a63903776b0f7acab23895e470bea0a39e02d9d15629c012adccf10","href":"https://static.observableusercontent.com/files/96a62d477a020aa59d44081ff413ac95f7fc6bc5f6492ba4614819cd638a896633f23c986a63903776b0f7acab23895e470bea0a39e02d9d15629c012adccf10","name":"e2_2b.png","type":"image/png","lastModified":1788669554483,"size":13290}] 2e:{"id":"dd2deabceac86585","slug":"week-2-exercise-data-questions-and-transformations-due-sept-17","title":"Week 2 Exercise: Data Questions and Transformations (Due Sept. 17)","thumbnail":"aafb1918cdf0f5d4c985a199d69a43173511295a08a7754453c328b56d1178f5","access_level":"public","create_time":"$D2026-09-16T06:03:28.565Z","update_time":"$D2026-09-17T07:24:23.601Z","view_time":null,"first_public_time":"$D2026-09-16T06:06:52.593Z","has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"85059ad626ed3d01","login":"najim-rhalmi","name":"Najim Rhalmi","avatar_url":"https://avatars-next.observableusercontent.com/319ab4b6f628b94b79607cee456ea6c770361d60152dae9fab2c4950cc360fb3","tier":"starter_2024"},"version":1429,"publish_version":1429,"role":null,"authored":false,"authors":[{"type":"individual","id":"4bfe50431c572ac9","login":"najim-rhalmi","name":"Najim Rhalmi","avatar_url":"https://avatars-next.observableusercontent.com/319ab4b6f628b94b79607cee456ea6c770361d60152dae9fab2c4950cc360fb3","tier":null}],"members":[{"type":"team","id":"85059ad626ed3d01","login":"najim-rhalmi","name":"Najim Rhalmi","avatar_url":"https://avatars-next.observableusercontent.com/319ab4b6f628b94b79607cee456ea6c770361d60152dae9fab2c4950cc360fb3","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Week 2 Exercise: Data Questions and Transformations (Due Sept. 17)","cells":[{"id":0,"value":"$30","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":674,"value":"---\n\n

[TO-DO]

Fill your name and NetID:

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":716,"value":"student_name = \"Rhalmi Najim\";","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":717,"value":"student_netid = \"N16213529\";","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":719,"value":"---\n## Part 0: Importing the Movies Dataset (5%)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":721,"value":"

[TO-DO]

Import the movies dataset provided at this Google Drive link as a JSON, not a CSV.\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":718,"value":"movies = FileAttachment(\"movies.json\").json()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":720,"value":"If you imported the data successfully, you should see a table below depicting the `movies` data.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":17,"value":"movies\n ? Inputs.table(movies)\n : null","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":723,"value":"---\n## Part 1: Modifying Axis and Chart Titles in Vega-Lite (5%)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":724,"value":"So far, we've worked with charts that pre-determined the axes titles based on the field names used. We also never really gave our charts any titles. Here, we'll briefly add titles to our charts and individual axes.\n\nLet's assume we produced the following chart below:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":725,"value":"// Do not modify\n{\n if (movies == null) return;\n return vl.markBoxplot()\n .data(movies)\n .encode(\n vl.x().fieldQ(\"US Gross\"),\n vl.y().fieldN(\"MPAA Rating\"),\n )\n .width(500)\n .height(400)\n .render()\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":726,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":727,"value":"Let's modify this chart so that it has some more informative axes and chart titles.\n\n

[TO-DO]

\n Modify the provided chart so that:\n

The chart is titled \"US Gross Income per MPAA Rating Level\". (1%)
The X-axis is titled \"US Gross Income ($)\". (1%)
The Y-axis is titled \"Rating\". (1%)
The chart is colored based on MPAA Rating and its color legend is titled \"Rating Colors\". (2%)

Solution:

\n ${await FileAttachment(\"e2_1.png\").image({width:400})}\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":728,"value":"\n// TO-DO\nchart1 = vl.markBoxplot()\n .data(movies)\n .encode(\n vl.x().fieldQ(\"US Gross\").title(\"US Gross Income ($)\"),\n vl.y().fieldN(\"MPAA Rating\").title(\"Rating\"),\n vl.color().fieldN(\"MPAA Rating\").title(\"Rating Colors\")\n )\n .title(\"US Gross Income per MPAA Rating Level\")\n .width(500)\n .height(400)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":729,"value":"chart1?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":722,"value":"---\n## Part 2: Transform + Filter (29%)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":19,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":730,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":281,"value":"\n// TO-DO\nchart2 = vl.markBar()\n .data(movies)\n .transform(\n vl.filter('datum[\"IMDB Rating\"] >= 8.0')\n )\n .encode(\n vl.x().fieldT(\"Release Date\").timeUnit(\"month\").title(\"Release Month\"),\n vl.y().count().title(\"# of Records\")\n )\n .title(\"IMDb Ratings >= 8.0\")\n .width(500)\n .height(300)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":731,"value":"// Do not modify\nchart2?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":56,"value":"\n// TO-DO\nchart3 = vl.markBar()\n .data(movies)\n .transform(\n vl.filter('year(datum[\"Release Date\"]) >= 1970 && year(datum[\"Release Date\"]) < 2000')\n )\n .encode(\n vl.x().fieldT(\"Release Date\").timeUnit(\"year\").title(\"Release Year\"),\n vl.y().count().title(\"# of Movies\")\n )\n .width(600)\n .height(300)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":732,"value":"// Do not modify\nchart3?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":400,"value":"\n// TO-DO: Rom-Coms (1970-1999)\nchart4 = vl.markCircle({color: \"blue\"})\n .data(movies)\n .transform(\n vl.filter('datum[\"Major Genre\"] == \"Romantic Comedy\" && datum[\"Rotten Tomatoes Rating\"] != null && datum[\"IMDB Rating\"] != null && datum[\"Release Date\"] != null && year(datum[\"Release Date\"]) >= 1970 && year(datum[\"Release Date\"]) < 2000')\n )\n .encode(\n vl.x().fieldQ(\"Rotten Tomatoes Rating\"),\n vl.y().fieldQ(\"IMDB Rating\")\n )\n .width(300)\n .height(300)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":733,"value":"// Do not modify\nchart4?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":402,"value":"\n// TO-DO: Rom-Coms (>=2000)\nchart5 = vl.markCircle({color: \"red\"})\n .data(movies)\n .transform(\n vl.filter('datum[\"Major Genre\"] == \"Romantic Comedy\" && datum[\"Rotten Tomatoes Rating\"] != null && datum[\"IMDB Rating\"] != null && datum[\"Release Date\"] != null && year(datum[\"Release Date\"]) != null && year(datum[\"Release Date\"]) >= 2000')\n )\n .encode(\n vl.x().fieldQ(\"Rotten Tomatoes Rating\"),\n vl.y().fieldQ(\"IMDB Rating\")\n )\n .width(300)\n .height(300)\n\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":734,"value":"// Do not modify\nchart5?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":26,"value":"---\n## Part 3: Transform + Calculate (31%)\n\nAlongside just filtering, let's take a brief moment to consider another aspect of data transformation: **feature generation**, or deriving new features from existing ones.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":735,"value":"### 3A. Using Calculation to Generate Features (7%)\nIn Vega-Lite, we use `vl.calculate()`. This is very similar to `vl.filter()` in that we need to call it within `.transform()`. The premise here is that any kind of data calculation is considered a transformation, assuming that the coder wants to conduct some operation on one or more existing features and create new features from that. The Vega-Lite API documentation is rather curt, but the general gist is to do the following:\n\n1. Use a string expression to conduct an operation on existing features (similar to `vl.filter()`), and\n2. Store the result of that operation into a new feature defined by an accompanying `.as()`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":76,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":439,"value":"\n// TO-DO\nchart6 = vl.markBar()\n .data(movies)\n .transform(\n vl.filter('datum[\"Worldwide Gross\"] != null && datum[\"Production Budget\"] != null'),\n vl.calculate('datum[\"Worldwide Gross\"] - datum[\"Production Budget\"]').as(\"Worldwide Profit\")\n )\n .encode(\n vl.x().fieldT(\"Release Date\").timeUnit(\"year\").title(\"Release Year\"),\n vl.y().fieldQ(\"Worldwide Profit\").aggregate(\"median\")\n )\n .width(600)\n .height(300)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":736,"value":"// To not modify\nchart6?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":450,"value":"---\n### 3B. What Happened in 2011? (6%)\n\nWhat happened in 2011?! Clearly something is wrong with the data. Let's investigate. There's clearly some movies that are causing the issue...\n\n

[TO-DO]

\n Create a simple square plot that shows which movies in 2011 are causing the profit calculation issue shown in Part 3A.\n

Create a square plot. (1%)
Filter the data to only show movies in 2011 and whose worldwide profit (see Part 3A) is less than 0. (2%)
The X-axis should represent the Production Budget. (1%)
The Y-axis should represent the Worldwide Gross. (1%)
A tooltip should show the Title. (1%)

Solution

\n ${await FileAttachment(\"e2_3b.png\").image({width:500})}\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":737,"value":"\n// TO-DO\nchart7 = vl.markSquare()\n .data(movies)\n .transform(\n vl.filter('datum[\"Worldwide Gross\"] != null && datum[\"Production Budget\"] != null && year(datum[\"Release Date\"]) == 2011'),\n vl.calculate('datum[\"Worldwide Gross\"] - datum[\"Production Budget\"]').as(\"Worldwide Profit\"),\n vl.filter('datum[\"Worldwide Profit\"] < 0')\n )\n .encode(\n vl.x().fieldQ(\"Production Budget\"),\n vl.y().fieldQ(\"Worldwide Gross\"),\n vl.tooltip().fieldN(\"Title\")\n )\n .width(600)\n .height(300)\n\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":738,"value":"chart7?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":469,"value":"According to our dataset, only two records of movies from 2011 exist that are worrying. Of these, both have a `Worldwide_Gross` of 0. This dataset appears to have two core issues we must consider in future charts: missing data, and the lack of entries in certain conditions.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":193,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":232,"value":"\n// TO-DO\nchart8 = vl.markCircle()\n .data(movies)\n .transform(\n vl.filter('year(datum[\"Release Date\"]) >= 2000 && datum[\"US Gross\"] != null && datum[\"Production Budget\"] != null'),\n vl.calculate('datum[\"US Gross\"] - datum[\"Production Budget\"]').as(\"US Profit\")\n )\n .encode(\n vl.x().fieldQ(\"Rotten Tomatoes Rating\").bin({maxbins: 50}),\n vl.y().fieldQ(\"IMDB Rating\").bin({maxbins: 50}),\n vl.color()\n .fieldQ(\"US Profit\")\n .aggregate(\"median\")\n .scale({scheme: \"redblue\", domainMid: 0})\n )\n .width(500)\n .height(400)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":740,"value":"// Do not modify\nchart8?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":739,"value":"When you generate your graph, consider the primary question: **do well-rated movies tend to also be profitable**? You don't have to submit an answer for this assignment, but we encourage you to consider how the chart we just created can illuminate your response.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":546,"value":"$39","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":567,"value":"### 4A. Custom Datasets with Pure JavaScript: IMDb Ratings (15%)\n\nThe first thing to do is to create a new variant of the movies dataset - let's call it `movies_imdb`. Here, we need to conduct 4 operations:\n\n1. Filter any rows where either `Production Budget` or `US Gross` are `null` (i.e. empty).\n2. Calculate the mean and standard deviation of the `Production Budget`.\n3. Calculate the mean and standard deviation of the `US Gross`\n4. Using these four values, create **two** new features called `Budget Z` and `Gross Z`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":748,"value":"#### Observable Notebook Quirks with Cells\n\nTo conduct this, we need to understand one thing about Observable: _Observable expects each cell to **return** a value_. Consequently, we can't just write a bunch of lines and leave it at that. Observe the following:\n\n```\nconst a = 1500;\nconst b = 173;\nconst c = a - b;\n```\n\nIf you leave this code inside of an Observable notebook cell, you'll get an error. We need to re-write the cell like so:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":742,"value":"c = {\n const a = 1500;\n const b = 173;\n return a - b;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":749,"value":"Multi-line operations must be wrapped with brackets `{}`. Then, you must return a value within the brackets for the cell to work. Remember that Observable cells always expect each cell to either return a value or set a variable (e.g. how we've always saved our charts using `chart1`, `chart2`, etc.).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":743,"value":"#### D3.js operations for Mean and Standard Deviation\n\nWe will also need to think about how we can calculate the mean and standard deviation of column values. Here, we'll actually **use D3.js** to help us. Observe the following code:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":745,"value":"budget_mean = d3.mean(movies, d => d[\"Production Budget\"])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":746,"value":"D3.js is very helpful because its sole purpose is to help us manage large data arrays. The operations it provides helps us with these operations without needing to write up custom functions. Here, we need to use two specific D3.js oeprations:\n\n- `d3.mean(, row => row[])`\n- `d3.deviation(, row => row[])`\n\nThese operations basically are asking D3.js to calculate the mean or standard deviation by taking our `` and then only using `` from each `row`. If your `` is just a one-dimensional list (e.g. a list of numbers), then you don't need the arrow function `row => row[]`. However, if each item in `` is not a single number (e.g. each item in `` is an object `{}` or nested array `[]`), then we gotta use the arrow function to extract only the relevant feature from each item.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":750,"value":"#### Mapping\n\nThe concept of mapping is pretty agnostic in the programming world; it means that we are applying some operation on each item of an array or list. Unlike using a filter which removes items that don't satisfy a specific condition, mapping will create a variant of your `` without removing any items or modifying the original data. Observe the code below:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":751,"value":"movies.map(row => ({\n ...row,\n \"US Profit\": row[\"US Gross\"] - row[\"Production Budget\"]\n}))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":752,"value":"Consider what we're doing here:\n\n1. We call `.map()`, which means we want to apply some operation on each item in `movies`\n2. We use the arrow function `row => ({ ... })` as shorthand to say \"for each `row`, we're going to do something\".\n3. `...row` means we are directly copying the existing feature:value pairs from the original row into the new row.\n4. `US Profit` is a new feature we're adding (on top of all the existing ones, added by `...row`) that is extracted by some operation with `US Gross` and `Production Budget`.\n\nYou will need this syntax for this upcoming part of the assignment.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":753,"value":"#### Filtering\n\nUp to this point, we've used Vega-Lite to filter our rows using `vl.filter()` inside of `.transform()`. Instead, we can do this at the same time as we're using `.map()`. Consider the following:\n\n```\n.transform(\n vl.filter('datum[\"Worldwide Gross\"] >= 100')\n)\n...\n```\n\nThis operation in Vega-Lite asks the chart to ignore data rows where the `Worldwide Gross` is less than 100. In pure JavaScript, we can do something similar:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":754,"value":"movies.filter(d => d[\"Worldwide Gross\"] >= 100)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":755,"value":"In this, we went from 3201 rows to 3147 rows.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":747,"value":"

[TO-DO]

\n Create a new movies_imdb that uses the operations we covered above to create a variant of movies. This new variant contains all the original features of movies alongside a new feature: IMDB Rating Z that represent the normalized z-score values of the IMDB Rating feature. Before calculating IMDB Rating Z, make sure to filter out any rows where there are empty or null IMDB Rating.\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":569,"value":"\n// TO-DO\nmovies_imdb = {\n const filtered = movies.filter(d =>\n d[\"IMDB Rating\"] != null && d[\"IMDB Rating\"] !== \"\"\n );\n\n const mean = d3.mean(filtered, d => d[\"IMDB Rating\"]);\n const deviation =d3.deviation(filtered,d => d[\"IMDB Rating\"]);\n\n return filtered.map(d => ({\n ...d,\n \"IMDB Rating Z\": (d[\"IMDB Rating\"] - mean) / deviation\n })\n );\n}\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":572,"value":"### 4B. Plotting Z-Scores Against Raw Values (5%)\n\nLet's do a basic test: plotting z-scores against their raw values. This is a helpful exercise to understanding what z-scores can tell you about the distribution of numerical data.\n\n

[TO-DO]

\n Construct a very basic scatter plot that shows the relationship between raw IMDB Rating and the new IMDB Rating Z we created with movies_imdb.\n

The X-axis should represent IMDB Rating. (1%)
The Y-axis should represent IMDB Rating Z. (1%)
The chart should be titled \"IMDB Scores: Raw v.s. Z-score\". (1%)
The chart should be 400px wide and tall. (2%)

Solution:

\n ${await FileAttachment(\"e2_4b.png\").image({width:400})}\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":602,"value":"\n// TO-DO\nchart9 = vl.markCircle()\n .data(movies_imdb)\n .encode(\n vl.x().fieldQ(\"IMDB Rating\"),\n vl.y().fieldQ(\"IMDB Rating Z\")\n )\n .title(\"IIMDB Scores: Raw v.s. Z-score\")\n .width(400)\n .height(400)\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":756,"value":"// Do not modify\nchart9?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":608,"value":"Let's ask the main question: **Why even do this**? We need this analysis because it can tell us a lot of interesting things about how `IMDB Rating` scores are distributed.\n\n- The Z-score ratings mean is represented at Y = 0. Y-values represent the number of standard deviations away from the mean.\n- The Z-score ratings show that there is an *asymmetric*, uneven distribution around the mean. The highest-rated movies are approximately 2 standard deviations above the mean, while the lowest-rated movies are close to 4 standard deviations below the mean.\n- The mean of IMDB Ratings is approximately around 6.25. On average, movies on IMDB are rated somewhat positively.\n\n

[NOTE]

Z-scores can be a useful tool to understanding the distribution of raw data.

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":584,"value":"$3a","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":586,"value":"\n// TO-DO\nmovies_normalized = {\n const filtered = movies.filter(d =>\n d[\"Production Budget\"] != null &&\n d[\"Running Time min\"] != null\n );\n\n const budgetMean = d3.mean(filtered,d => d[\"Production Budget\"]);\n const budgetDeviation = d3.deviation(filtered, d => d[\"Production Budget\"]);\n\n const runningMean = d3.mean(filtered, d => d[\"Running Time min\"]);\n const runningDeviation = d3.deviation(filtered,d => d[\"Running Time min\"]);\n\n return filtered.map(d => ({\n ...d,\n \"Budget Z\": (d[\"Production Budget\"] - budgetMean) / budgetDeviation,\n \"Running Time Z\": (d[\"Running Time min\"] - runningMean) / runningDeviation\n }));\n}\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":588,"value":"\n// Do not modify:\nchart10 = {\n if (movies_normalized == null) return null;\n\n// TO-DO from here\n return vl.markCircle()\n .data(movies_normalized)\n .encode(\n vl.x().fieldQ(\"Budget Z\"),\n vl.y().fieldQ(\"Running Time Z\"),\n vl.tooltip([\"Title\", \"Production Budget\", \"Running Time min\"])\n )\n .title(\"Budget and Running Time Distributions\")\n .width(400)\n .height(400)\n\n// Do not modify\n}\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":590,"value":"// Do not modify\nchart10?.render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":619,"value":"An interesting question: Why not just plot `Production Budget` and `Running Time min` by themselves? It seems like it'll show the same overall shape. The plot for that is below:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":647,"value":"// Do not modify\n{\n if (movies_normalized == null) return;\n return vl.markCircle()\n .data(movies_normalized)\n .encode(\n vl.x().fieldQ(\"Production Budget\"),\n vl.y().fieldQ(\"Running Time min\")\n )\n .width(400).height(400).title(\"Raw Budgets and Running Times\")\n .render()\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":758,"value":"$3b","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":713,"value":"---\n## Appendix\n\n

Do no modify anything below this point

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":714,"value":"import { css, vl } from \"@rk2546/infovis-cse-backend\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":715,"value":"css","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":760,"value":"submission = ({\n student_name, student_netid,\n movies, movies_imdb, movies_normalized,\n chart1, chart2, chart3, chart4, chart5, chart6, chart7, chart8, chart9, chart10\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3c","stdlib":"2","titling":"auto"}} 2f:"$2e:body"