2b:null 2c:["$","title",null,{"children":"Week 2 Exercise: Data Questions and Transformations | New York University | Observable"}] 31:T51e,# Week 2 Exercise: Data Questions and Transformations In this week's [lecture](https://ctsilva.github.io/2025-InfoVis-CSE/slides/week2-data-transformation.html), we focused on four core elements of data transformations: _binning_, _aggregation_, _filtering_ , and _normalization_. In the [in-class lab session](https://ctsilva.github.io/2025-InfoVis-CSE/labs/week2-lab.html) that accompanied the lecture, we only covered _binning_ and _aggregation_. In this week's exercise, we will be focusing on the other two: _filtering_ and _normalization_. Like in our in-class lab, we will be focusing on a movie dataset provided by **Vega-Lite** (we'll abbreviate **Vega-Lite** to **VL** from now on). Let's observe some other nuances hidden in the data. This exercise is divided into 3 core components: 1. Transform + Filter (25%) 2. Transform + Calculate (35%) 3. Plain JavaScript + Normalization (40%) ### _Note: Thinking a bit outside Vega-Lite_ Some parts of this exercise will involve code concepts that are more globally common in programming, not just contained within **VL**. This includes using string expressions to write out code logic (e.g. "x > 10") and some light JavaScript coding. We will do our best to provide summaries and documentation where necessary. Please read all instructions carefully.32:T6bf,## Assignment Instructions **_Fork this notebook to make sure that you are working on your own copy!_** Work your way down the notebook, making sure to run code blocks whenever they appear. Some parts may require you to fill out markdown blocks. Others will require code work. And some are not interactive and are just for instructions! All exercises will be marked with a **[TO-DO]**, so pay attention to them throughout the notebook. You may refer to either [Week 2 Lab](https://observablehq.com/@rk2546/2025-infovis-cse_week-2-lab) for reference or additional online resources such as documentation and public forums for help with these questions. Please be mindful of our academic integrity policies when completing this assignment. ## Submission Instructions This assignment is due **September 18, 2025 @ 11:59pm**. Please follow the submission instructions: 1. _Make the notebook visible to the public._ This is accomplished by clicking the "Share" button at the top of the page and changing the "Access" settings for the Public to "Can view". 2. _Copy the Notebook URL in the same "Share" panel._ Feel free to use either an automatically-generated or custom URL. 3. In the markdown block below, _please fill out the following information._ The "Notebook URL" is the URL you copied in Step 2. **Not filling out the details below will result in a 0 for this assignment!** 4. _Download the notebook as a `.tgz` zip file._ This can be done by: 1. Scroll up to the top of the notebook, and click the "..." button. 2. `Export` -> `Download Code`. This produces a `.tgz` zip file. 3. On Brightspace, _navigate to "Assignments" and locate the corresponding assignment submission page._ 4. _Upload the `.tgz` zip file._33:T92c,--- ## Part 1: Transform + Filter (25%) --- **Filtering** is a very common data transformation that removes unwanted data from analysis. We typically require some filtering to, for example, remove samples with missing or noisy data. We may also want to use filters to remove irrelevant data. This section will cover both use cases for filtering data from our movie dataset. In **VL**, any modification to data requires `.transform()`, right next to `.data()` and `.encode()`. We'll add arguments into `.transform()` to tell **VL** _how_ we want to transform the data. In this case, we want to call `vl.filter()` as an argument inside `.transform()`; this is similar to how we call `vl.x()` for any x-axis operations inside of `.encode()`. When you filter data using `vl.filter()`, this `` is actually something we call a **"string expression"**. Consider it actual code, just written in string form; the API will do the heavy-lifting of interpreting what the string is trying to say and work with it. An example: consider we want to remove all Drama movies from our `movies` dataset. Here, we would type something similar to the following (pay attention to the string used inside `vl.filter()`): ```javascript .data(...) .transform( vl.filter('datum.Major_Genre != "Drama"') ) .encode(...) .render()) ``` **Be careful about single and double quotations**. Notice that because we're dealing with a string value ("Drama") inside of the string expression, we used _single quotations_ here. Some programming languages such as JavaScript and Python allow you to use single quotes to define multi-character strings. However, most programming languages will consider single quotations as _character_ data types (e.g. `'c'`, `'a'`). Nonetheless, this expression effectively tells **VL** to only include any samples where the `Major_Genre` is not "Drama". If you want to add multiple conditions, you can combine both conditions into a singular string expression, such as: ```javascript vl.filter('datum.Major_Genre != "Drama" && datum.Director != null') ``` In this, not only are we filtering out dramatic movies but also we're making sure that the movie also has a `Director` listed. For more information, here's the documentation for `vl.filter()`: [documentation](https://vega.github.io/vega-lite-api/api/filter.html).34:T532,---- ### 1C. The Curious Case of Rom-Coms Across Eras (5%) With this `movies` dataset at our disposal, alongside our newfound ability to filter the data that we use in charts, we can begin to really look deeper into individual trends across specific features. A curious case may be found in how people view and rate movies across genres. Do people attitudes towards different genres depend on which era the movies come out in? Let's look at the classic **"Romantic Comedy** (Rom-Com) as our case study. #### [TO-DO] Generate two charts that have `Rotten_Tomatoes_Rating` along the X-axis and `IMDB_Rating` along the Y-axis. Both charts must follow these conditions: 1. Both charts must plot only Rom-Coms. 2. Both charts must only show movies that have BOTH Rotten Tomatoes and IMDb ratings. 3. Both charts must only show movies with a release date (AKA don't consider movies that are missing their release dates) 4. The first chart specifically looks at Rom-Coms made between 1970 and 1999; the second chart looks at Rom-Coms made in and after 2000. We preclude movies made pre-1970 as the number of samples for each genre are too small for consideration. In your code answer, do not use facets at this time; instead, generate individual charts for this analysis. Your charts should look similar to the provided sample below.35:T5e4,### 2A. Using Calculation to Generate Features (5%) In **VL**, we use the `vl.calculate()` method alongside `vl.filter()` inside of `.calculate()`. The premise here is that any kind of data calculation is considered a transformation, assuming that the coder wants to conduct some operation on one or more existing features and create new features from that. The [**VL** API documentation](https://vega.github.io/vega-lite-api/api/calculate.html) is rather curt, but the general gist is to do the following: 1. Use a string expression to conduct an operation on existing features (similar to `vl.filter()`), and 2. Store the result of that operation into a new feature defined by an accompanying `.as()`. Here's an example of a simple operation that adds two columns (`A` and `B`) together to make a third column (`C`): ```javascript vl.markCircle() .data(...) .transform( vl.calculate('datum.A + datum.B').as('C') ) .encode(...) .render() ``` Let's take this logical feature to the next step and apply it to data. Let's generate a new feature called `Worldwide_Profit` that is calulated by subtracting `Worldwide_Gross` and `Production_Budget`. #### [TO-DO] Generate a bar chart depicting release year on the x-axis and the median `Worldwide_Profit` on the y-axis. Use `.calculate()` and `.as()` to generate this new `Worldwide_Profit` feature, by subtracting `Production_Budget` from `Worldwide_Gross`. You should be able to generate a new bar chart like the one below. 36:T9d8,--- ### 2B: Using JavaScript for Field Generation and Filtering (5%) Several times already, we have touched upon _string expressions_ to define the logic behind `vl.filter()` and `vl.calculate()`. In this part, we'll practice with plain JavaScript code to create a new variant of our `movies` dataset. If you recall in Part 1C of [Exercise #1](https://observablehq.com/@rk2546/2025-infovis-cse_exercise-1), we actually already engaged with JavaScript by creating a modified version of our camera dataset. A copy of the code question: as a reminder: ```javascript cameras = camera_data.map(d => ({ model: d["Model"], release_date: d["Release date"], max_resolution: d["Max resolution"], // [TO-DO] })) ``` This was only possible because: 1. **We are running a JavaScript environment**: Observable has a JavaScript compiler and executor built into it, similar to how a Python compiler is needed to run Jupyter Notebook or Google Colab notebooks. 2. **Datasets imported in Observable are treated as Arrays**: If you scroll up to where we imported the `movies` dataset, notice that the outputted log shows that `movies` is an `Array(...)`. JavaScript has some in-built method unique to Arrays - namely, `.filter()` and `.map()`. These methods require a second method you define elsewhere to _handle_ each entry in your array, as `.filter()` and `.map()` will iterate across your array entries. Alternatively, you can use **arrow functions** to define the handler logic directly in the method - this is most similar to [lambda functions in Python](https://www.geeksforgeeks.org/python/python-lambda-anonymous-functions-filter-map-reduce/). Here's the documentation for both [`.filter()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter) and [`.map()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map) - I strongly recommend you read them before moving to the next part of this question. That being said, let's say we want to generate an entirely new array, `rated_movies`, that takes in the original `movies` array and does the following: 1. Filters out rows that have `null` or `0` values in some of their features (`.filter()`), and 2. Both renames and calculates new features using existing features (`.map()`). #### [TO-DO] Complete the JavaScript code block below to create our new `rated_movies` modification. Some of the code is already provided to you, for your reference. _They should look familiar to you._37:T64f,--- ### 2D: Profit v.s. Movie Ratings - Are Modern Movies Good AND Profitable? (20%) With our `rated_movies` dataset, the burning question is thus: **Do well-rated movies (at least, by IMDb and Rotten Tomatoes standards) tend to also be profitable**? Let's find some solace by charting out a possible response. #### [TO-DO] (10%) In this question, we define "modern" to be _movies made in or after 2000_. Generate a chart of your own design that attempts to solve this question. Though a sample is provided, you need not necessarily follow its example - there are a multitude of different approaches to this question. You are free to do the following: - Use any chart type (histograms, bar charts, scatter plots, or something entirely different). - Use any filter or transformation logic via `.transform()`. - Refer to Rotten Tomatoes ratings, IMDb ratings, or both to answer this question. - Create your own variant of the `movies` dataset that is better suited to your needs. A sample for what a possible chart could look like is provided below. #### [TO-DO] (10%) Answer the primary question: **Do well-rated movies tend to also be profitable**? Your answer must go beyond just a "Yes" or "No" to this question - elaborate on how your figure emphasizes your final response. You are also allowed to refer to external media and resources to **corroborate** on findings you discover via your own - i.e. you can refer to another website to verify your findings or counter them, but you cannot use an external resource as the sole rationale behind your answer. Remember to cite sources, if you decide to use them.38:T794,--- ## Part 3: Plain JavaScript + Normalization (40%) --- Data **normalization** is the practice of re-scaling data for more fair comparisons. This is likely to occur depending on situational contexts. Common examples include: - Scaling prices in the past to match today's inflation levels, to observe relative costs of goods and services compared to household incomes. - Health monitoring & scaling multiple patients' physiological conditions to standardized metrics. - Grading scales across different districts or inter-school classes, to either provide grading curves or identify poor-performing or well-performing classes. - Demographics in urban settings, such as voter populations or the number of diseased individuals relative to per-capita or population density. The most common type of data normalization is [**z-score**](https://en.wikipedia.org/wiki/Standard_score), or the number of standard deviations a raw value is above or below the mean of the data distribution. Z-scores allow us to standardize raw scores into a common ground, even if the distributions of two different features are different (e.g. they have different means and variances). In our `movies` dataset, we will use data normalization to solve some major lingering issues with our data. Most notably, we will attempt to **compare movie success across genres using normalized US Gross**. The core topic at hand is to compare which movie genres are generally more profitable with regard to their U.S. performance. We normalize across U.S. Gross instead of looking at raw profits because of an uneven distribution of movies across the genres: some genres have more movies than others. However, before we get there, we need to establish some basic JavaScript code for helping us conduct data normalization. We'll spend some time learning how to conduct data normalization in plain Javascript, and we'll carry that logic over to answering our main question.39:T561,--- ### 3C: Normalizing via Z-Scores (10%) Let us generate a variant of `movies_us_profit` that feature normalized data (remember: we're keeping in line with good data principles by not modifying `movies_us_profit` directly). This new dataset, `normalized_movies`, is an Array type that should be of same size to `movies_us_profit`. The only difference is that we will be using a `for` loop to iterate across `genre_groups` and iteratively adding each movie into `normalized movies`. The general flow of logic is: 1. Create an empty `normalized_movies` Array. 2. Loop through each genre (key) in `genre_groups`. For each genre: 1. Extract the array of movies associated with that genre, `group`. 2. Isolate `profits` as a new Array only consisting of the `Profit` cell value (i.e. no headers, no other features, just an array of numbers) 3. Calculate the mean and standard deviation from `profits`. 4. Loop through each movie in `group`. For each movie: 1. Add an entry into `normalized_movies` for that movie. An entry is a JSON object with the following features: `Title`, `Genre`, `Profit`, and `Z-Score`. To aid you, two JavaScript codes are provided to you as helper functions: one to calculate the **mean** of an Array of values, and another to calculate the **standard deviation** of an Array of values. You can call them any time in your implementation.3a:[{"id":"232edd4bac9ed0c32465ced55abf38800d2681774140d548b4838e6d127dbcc1e7d9e7c5137bf1c5ba67eb0f98e9de26d56bfba87c24fc23813ed271d7a0e8e5","href":"https://static.observableusercontent.com/files/232edd4bac9ed0c32465ced55abf38800d2681774140d548b4838e6d127dbcc1e7d9e7c5137bf1c5ba67eb0f98e9de26d56bfba87c24fc23813ed271d7a0e8e5","name":"3D.png","type":"image/png","lastModified":1758227133375.565,"size":19123},{"id":"8243027020bd844e3e5acfdeff2cae619996290e5394c418fe921f6fb26948de4a10216d74bb57e00dfb6d1da59b996e8e8cf4e98d46795157a790273d256d5c","href":"https://static.observableusercontent.com/files/8243027020bd844e3e5acfdeff2cae619996290e5394c418fe921f6fb26948de4a10216d74bb57e00dfb6d1da59b996e8e8cf4e98d46795157a790273d256d5c","name":"3D_alt.png","type":"image/png","lastModified":1758227133375.565,"size":20289},{"id":"eb784c1ca65a0fdf0856143cba840afa058daf22cbb9f7b4aa2cb304f531d42b35625fd3e370e78bdc19f5cf36b23283a33a1dd586b3e384504dcc1e8aa76264","href":"https://static.observableusercontent.com/files/eb784c1ca65a0fdf0856143cba840afa058daf22cbb9f7b4aa2cb304f531d42b35625fd3e370e78bdc19f5cf36b23283a33a1dd586b3e384504dcc1e8aa76264","name":"E2@1.1As.png","type":"image/png","lastModified":1758227133375.565,"size":14325},{"id":"544132512811b179d8dbd28ed8867952321a966a06b8cda8bdebf374056ef01054b6010650291c5a0f5d71f7ea440bc8d86ae0041a35022c75fb2ee8acb1577e","href":"https://static.observableusercontent.com/files/544132512811b179d8dbd28ed8867952321a966a06b8cda8bdebf374056ef01054b6010650291c5a0f5d71f7ea440bc8d86ae0041a35022c75fb2ee8acb1577e","name":"E2.2C.png","type":"image/png","lastModified":1758227133375.565,"size":16412},{"id":"2bc4c02dd86fbb6a42bba081f8c2a1c05d8083523a72b7a8051dc70f68e28d78fed0930bee63cfd5d40cf1930ce3c6bd95a1a15ee305831549ca8afe8e267153","href":"https://static.observableusercontent.com/files/2bc4c02dd86fbb6a42bba081f8c2a1c05d8083523a72b7a8051dc70f68e28d78fed0930bee63cfd5d40cf1930ce3c6bd95a1a15ee305831549ca8afe8e267153","name":"E2.2D.png","type":"image/png","lastModified":1758227133375.565,"size":91293},{"id":"7e4aea9e17bdcaaf50daf11c9a2da3da1ce25ce661275dbc7ffb015a854bd0965ebb311b558ca33fbe1d634678a7dd909fbf23880b0751a8d7ec5f27d7640f3f","href":"https://static.observableusercontent.com/files/7e4aea9e17bdcaaf50daf11c9a2da3da1ce25ce661275dbc7ffb015a854bd0965ebb311b558ca33fbe1d634678a7dd909fbf23880b0751a8d7ec5f27d7640f3f","name":"E2.2011.png","type":"image/png","lastModified":1758227133375.565,"size":15997},{"id":"010d9d2e3b4c5d22db558ec59c7207d1b76c54c3057df6d4776463ca9eef8bcd01f8cf9654620f1c5093dfcea3db2d68afee2d96d112a1e87be4a493f159e055","href":"https://static.observableusercontent.com/files/010d9d2e3b4c5d22db558ec59c7207d1b76c54c3057df6d4776463ca9eef8bcd01f8cf9654620f1c5093dfcea3db2d68afee2d96d112a1e87be4a493f159e055","name":"E2@1.2A.png","type":"image/png","lastModified":1758227133375.565,"size":18476},{"id":"5663273a8872e3cab347ea4458c1c7f2eff135a6e16b7f6848235ce5c9d4b8ad6024ecd127400c9b3ba2fc0c6e5a743d5a29504ea8ee58290a74a3fa2dda8d63","href":"https://static.observableusercontent.com/files/5663273a8872e3cab347ea4458c1c7f2eff135a6e16b7f6848235ce5c9d4b8ad6024ecd127400c9b3ba2fc0c6e5a743d5a29504ea8ee58290a74a3fa2dda8d63","name":"E2.1C.png","type":"image/png","lastModified":1758227133375.565,"size":36557},{"id":"78c571fab85feb3eebc3bc46230aabf57965e6af3618f637f94e6964f1d73e7ba941e8169b3aaa829e5c028fba813e70f2b21ef96414602fa02ed831fa596422","href":"https://static.observableusercontent.com/files/78c571fab85feb3eebc3bc46230aabf57965e6af3618f637f94e6964f1d73e7ba941e8169b3aaa829e5c028fba813e70f2b21ef96414602fa02ed831fa596422","name":"E2.1B.png","type":"image/png","lastModified":1758227133375.565,"size":43311}] 2f:{"id":"518368f16a58bc17","slug":null,"title":"Week 2 Exercise: Data Questions and Transformations","thumbnail":"3f1eaf58e01c824d7fc037881387d3967f57d3e4d39798992b340b15427962dd","access_level":"public","create_time":"$D2025-09-18T20:25:33.375Z","update_time":"$D2025-09-18T23:01:01.600Z","view_time":null,"first_public_time":"$D2025-09-18T23:00:09.970Z","has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"3e0135dbcb9af4fe","login":"new-york-university21","name":"New York University","avatar_url":"https://avatars-next.observableusercontent.com/d97d076a94403ded92e5811f3b293e5a00e959f42fc5252221bd57904d711438","tier":"starter_2024","bio":""},"version":773,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"6b3b6472db850914","login":"sm12505","name":"Sm12505","avatar_url":"https://avatars-next.observableusercontent.com/25d7e16f268c88a291fc61a344dcc836598b5be17f4329fda0ef29d45aae18d2","tier":null,"bio":""}],"members":[{"type":"team","id":"3e0135dbcb9af4fe","login":"new-york-university21","name":"New York University","avatar_url":"https://avatars-next.observableusercontent.com/d97d076a94403ded92e5811f3b293e5a00e959f42fc5252221bd57904d711438","tier":"starter_2024","role":"editor"},{"type":"individual","id":"6b3b6472db850914","login":"sm12505","name":"Sm12505","avatar_url":"https://avatars-next.observableusercontent.com/25d7e16f268c88a291fc61a344dcc836598b5be17f4329fda0ef29d45aae18d2","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Week 2 Exercise: Data Questions and Transformations","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":672,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":674,"value":"---\n\n#### [TO-DO]\n\nFill out the following submission details:\n\n* **Name:** <Siddh Mandirwala>\n* **NetID:** <sm12505>\n* **SHARE URL:** <[NOTEBOOK URL](https://observablehq.com/d/518368f16a58bc17)>\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"import {vl} from \"@vega/vega-lite-api\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"import {printTable} from \"@uwdata/data-utilities\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":15,"value":"movies = (await require('vega-datasets@1'))['movies.json']()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":17,"value":"printTable(movies.slice(0, 5))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":19,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":34,"value":"---\n### 1A. Simple Filter: Only Use Popular Movies... According to IMDb (5%)\nLet's start off with a very simple filter: only include movies where `IMDb_Rating` is greater than or equal to `8.0`. Let's also use this time to get used to some **binning** and **aggregation**, which we covered in lecture.\n\n#### [TO-DO]\nModify the provided bar chart code that counts the number of movies per month. Use the new `.transform()` and `vl.filter()` methods make it so that we only look at movies where `IMDb_Rating` is greater than or equal to `8.0`. There is no need to modify the code provided inside of `.encode()` already. A sample of what your chart may look like is provided.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":299,"value":"FileAttachment(\"E2@1.1As.png\").image({width:350})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":281,"value":"// TO-DO\nvl.markBar()\n .data(movies)\n // START TO-DO\n .transform(\n vl.filter('datum.IMDB_Rating >= 8.0')\n )\n // END TO-DO\n .encode(\n vl.x().month(\"Release_Date\"),\n vl.y().count(),\n )\n .title(\"1A: IMDb Ratings >= 8.0\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"---\n### 1B. Filter by Era (<1970, 1970-1999, >=2000) (5%)\n\nLet's use the new `vl.filter()` method to isolate movie \"eras\". Let's say we want to form 3 groups:\n\n- All movies made before 1970 (y < 1970)\n- All movies made between 1970 and 1999 (1970 <= y < 2000)\n- All movies made after 2000 (y >= 2000)\n\nWe can use three separate charts to showcase these eras: one era for each chart.\n\n#### [TO-DO]\nProduce three bar charts that count the number of movies made per month, each chart representing each era above. You must use `vl.filter()` to achieve this. This process requires that you need to extract the year from `Release_Date`; you can achieve this by using `year(datum.Release_Date)`. This converts `Release_Date` into a VL [TimeUnit](https://vega.github.io/vega-lite/docs/timeunit.html) that contains only the year value. Your three charts might look like those provided below (_the provided solution uses facets to visualize the data; your solution must not use facets at this time_).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":332,"value":"FileAttachment(\"E2.1B.png\").image({width:1000})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":56,"value":"// TO-DO: Movie Counts per Month, <1970\nvl.markBar()\n .data(movies)\n .transform(\n vl.filter('year(datum.Release_Date) < 1970')\n )\n .encode(\n vl.x().month(\"Release_Date\"),\n vl.y().count(),\n )\n .title(\"Movies Before 1970\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":58,"value":"// TO-DO: Movie Counts per Month, 1970-1999\nvl.markBar()\n .data(movies)\n .transform(\n vl.filter('year(datum.Release_Date) >= 1970 && year(datum.Release_Date) < 2000')\n )\n .encode(\n vl.x().month(\"Release_Date\"),\n vl.y().count(),\n )\n .title(\"Movies 1970-1999\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":85,"value":"// TO-DO: >=2000\nvl.markBar()\n .data(movies)\n .transform(\n vl.filter('year(datum.Release_Date) >= 2000')\n )\n .encode(\n vl.x().month(\"Release_Date\"),\n vl.y().count(),\n )\n .title(\"Movies 2000 and After\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":398,"value":"FileAttachment(\"E2.1C.png\").image({width:700})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":400,"value":"// TO-DO: Rom-Coms (1970-1999)\nvl.markCircle()\n .data(movies)\n .transform(\n vl.filter('datum.Major_Genre == \"Romantic Comedy\" && datum.Rotten_Tomatoes_Rating != null && datum.IMDB_Rating != null && datum.Release_Date != null && year(datum.Release_Date) >= 1970 && year(datum.Release_Date) < 2000')\n )\n .encode(\n vl.x().fieldQ(\"Rotten_Tomatoes_Rating\"),\n vl.y().fieldQ(\"IMDB_Rating\"),\n )\n .title(\"1970-1999\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":402,"value":"// TO-DO: Rom-Coms (>=2000)\nvl.markCircle()\n .data(movies)\n .transform(\n vl.filter('datum.Major_Genre == \"Romantic Comedy\" && datum.Rotten_Tomatoes_Rating != null && datum.IMDB_Rating != null && datum.Release_Date != null && year(datum.Release_Date) >= 2000')\n )\n .encode(\n vl.x().fieldQ(\"Rotten_Tomatoes_Rating\"),\n vl.y().fieldQ(\"IMDB_Rating\"),\n )\n .title(\"2000+\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":405,"value":"---\n### Part 1D: Comparing Ratings Across Eras (10%)\n#### [TO-DO]\nEither individually or in a single paragraph, answer the following questions about the charts you made in Part 1C:\n\n1. What are similarities or differences in ratings for Rom-Coms pre- and post-2000? E.g. are their trends between IMDb and Rotten Tomatoes ratings consistent across time?\n2. Do other movie genres display the same trends? (_Feel free to mess around with the code yourself to find an answer to this question._)\n3. Are Rom-Coms (or other genres you explored in #2) that are older (made pre-2000) better than movies made more recently (made post-2000)? Or are there other possible effects at play (e.g. nostalgia bias)?\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":411,"value":"===============\n\n1. Pre-2000 Rom-Coms cluster tighter with fewer bad movies, while post-2000 has way more data points including lots of lower-rated films. The older ones seem \"better\" but that's probably just because only memorable ones made it into the dataset.\n \n2. Probably yes - we just have way more movie data from 2000+ because the industry expanded and digital tracking got better.\n \n3. Not really. It's survivorship bias - we only see the good old movies that people remembered, but we see ALL the recent movies including the forgettable ones.\n \n\n===============","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":26,"value":"---\n\n## Part 2: Transform + Calculate (35%)\n\n---\n\nAlongside just filtering, let's take a brief moment to consider another aspect of data transformation: **feature generation**, or deriving new features from existing ones. You may have done something similar in practice when generating new nominal labels for quantitative data (e.g. marking each sample as a binary 1 or 0 depending on some condition in the data) or adding/subtracting/multiplying features together. We'll spend just a bit of time focusing on how to do these operations both in **VL** and in raw JavaScript. The JavaScript portion will focus on creating a variant of our original `movies` dataset without mutating it (remember the general recommendations covered in lecture about data handling).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":76,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":445,"value":"FileAttachment(\"E2@1.2A.png\").image({width:350})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":439,"value":"// TO-DO\nvl.markBar()\n .data(movies)\n .transform(\n vl.calculate('datum.Worldwide_Gross - datum.Production_Budget').as('Worldwide_Profit')\n )\n .encode(\n vl.x().year(\"Release_Date\"),\n vl.y().median(\"Worldwide_Profit\"),\n )\n .title(\"Median Worldwide Profit by Year\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":450,"value":"---\n\n### What Happened in 2011?\nWhat happened in 2011?! Clearly something is wrong with the data.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":467,"value":"FileAttachment(\"E2.2011.png\").image({width:500})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":469,"value":"According to our dataset, only two records of movies from 2011 exist. Of these, both have a `Worldwide_Gross` of 0. This dataset appears to have two core issues we must consider in future charts: missing data, and the lack of entries in certain conditions.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":476,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":482,"value":"rated_movies = movies\n .filter(d =>\n d[\"IMDB_Rating\"] != null \n && d[\"Rotten_Tomatoes_Rating\"] != null \n // TO-DO: check that both `Worldwide_Gross` and `Production_Budget` are neither `null` nor 0. \n && d[\"Worldwide_Gross\"] != null \n && d[\"Worldwide_Gross\"] != 0\n && d[\"Production_Budget\"] != null \n && d[\"Production_Budget\"] != 0\n && d[\"Release_Date\"] != null\n // TO-DO: Check that `Release_Date` is not `null`\n \n )\n .map(d => ({\n Title: d.Title,\n Release_Year: new Date(d[\"Release_Date\"]).getFullYear(),\n IMD: +d[\"IMDB_Rating\"], \n // TO-DO: re-label `Rotten_Tomatoes_Rating` to `Rotten_Tomatoes` \n Rotten_Tomatoes: +d[\"Rotten_Tomatoes_Rating\"],\n Worldwide_Profit: +d[\"Worldwide_Gross\"] - +d[\"Production_Budget\"]\n // TO-DO: create a new `Worldwide_Profit` entry by subtracting `Production_Budget` from `Worldwide_Gross`\n \n }));","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":491,"value":"---\n### 2C: Re-Plotting Median Profits (5%)\n#### [TO-DO]\nRe-plot your median profit chart from Part 2A, but this time using the modified dataset `rated_movies`. Keep in mind the changes we've made to some of the features. For example, we've isolated `Release_Date` to `Release_Year` in our `.map()` code for you, but there is no longer any `Release_Date` feature in `rated_movies`. Your chart should look similar to the sample provided:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":525,"value":"FileAttachment(\"E2.2C.png\").image({width:350})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":497,"value":"// TO-DO\nvl.markBar()\n .data(rated_movies)\n .encode(\n vl.x().fieldQ(\"Release_Year\"),\n vl.y().median(\"Worldwide_Profit\"),\n )\n .title(\"Median Profit by Year\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":193,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":541,"value":"FileAttachment('E2.2D.png').image({width:500})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"// TO-DO\nvl.markCircle()\n .data(rated_movies)\n .transform(\n vl.filter('datum.Release_Year >= 2000')\n )\n .encode(\n vl.x().fieldQ(\"Rotten_Tomatoes\").title(\"Rotten Tomatoes Rating\"),\n vl.y().fieldQ(\"IMD\").title(\"IMDb Rating\"),\n vl.color().fieldQ(\"Worldwide_Profit\").scale({scheme: \"redbwhitelue\", range: [\"red\",\"white\", \"lightblue\", \"blue\"]}).title(\"Profit ($)\"),\n vl.size().value(50),\n vl.opacity().value(0.7)\n )\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":532,"value":"===============\n\n> Yes, there's a clear positive relationship. The blue dots (high profit) cluster in the upper right with good ratings, while red dots (low profit) are mostly in the lower left with bad ratings. Good movies get better word-of-mouth and make more money, though there are some outliers where marketing and star power matter too.\n\n===============","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":546,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":567,"value":"### 3A: First Steps: Custom Dataset (5%)\n\nThe question implies that we want to normalize `Profit` across each `Major_Genre`. This means that, for each genre, we need to identify which movies are tied to that genre and extract their profit value. To even begin with this, we need to create a new variant of our `movies` dataset, but this time using `US_Gross` instead of `Worldwide_Gross`.\n\nGenerate a new `movies_us_profit` dataset that calculates `Profit` from subtracting `US_Gross` and `Production_Budget`. This is an adaptation of Part 2B, and you are free to adapt it for this question.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":569,"value":"// TO-DO\nmovies_us_profit = movies\n .filter(d =>\n d[\"IMDB_Rating\"] != null \n && d[\"Rotten_Tomatoes_Rating\"] != null \n && d[\"US_Gross\"] != null \n && d[\"US_Gross\"] != 0\n && d[\"Production_Budget\"] != null \n && d[\"Production_Budget\"] != 0\n && d[\"Release_Date\"] != null\n && d[\"Major_Genre\"] != null\n )\n .map(d => ({\n Title: d.Title,\n Genre: d.Major_Genre,\n Release_Year: new Date(d[\"Release_Date\"]).getFullYear(),\n Profit: d[\"US_Gross\"] - d[\"Production_Budget\"],\n IMDB_Rating: +d[\"IMDB_Rating\"],\n Rotten_Tomatoes: +d[\"Rotten_Tomatoes_Rating\"]\n }))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":572,"value":"---\n### 3B: Second Steps: Grouping Movies by Genre (10%)\n\nWe now need to group movies by their `Major_Genre`. This also requires some manual, plain JavaScript code. The idea here is the following:\n\n1. Start with an empty JavaScript (or JSON) object - this will be a _dictionary_ that maps `Major_Genre` to an array of rows in our `movies_us_profit` dataset.\n2. Loop through each row of `movies_us_profit`. In each loop iteration:\n 1. Identify the genre associated with that movie.\n 2. If that genre isn't present in our dictionary, add a new entry.\n 3. Add the movie into the corresponding dictionary entry, by genre.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":599,"value":"#### _A quick note on running JavaScript on Observable_\n\nUnlike Python, where you can run Python without any major issues in each cell, Observable requires some minor changes to your logic stream. Namely:\n\n- Each code block expects some variable or function to be the focus of each code block - i.e. you are defining a function, or you are setting a variable. It does not allow just JavaScript operations themselves as a standalone code segment.\n- JavaScript operations must be wrapped with parentheses `{` and `}` and feature a `return` at the end of it.\n\nSee [this notebook](https://observablehq.com/d/0254e43ad2e19ee3) as an example for how to do a `for` loop in Observable JavaScript. For now, you can refer to the code sample below as an example of how to set variables in Observable JavaScript and how to define a `for` loop:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":602,"value":"example_variable = {\n // JavaScript operations in Observable must be defined using `{` and `}`\n let sum = 0;\n for (let i = 0; i < 10; i++) {\n sum += i;\n }\n // A `return` is expected\n return sum;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":608,"value":"#### [TO-DO]:\nUsing plain JavaScript, use the pseudo-code above to create a new `genre_groups` dictionary object.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":574,"value":"// TO-DO\ngenre_groups = {\n let groups = {};\n \n for (let movie of movies_us_profit) {\n const genre = movie.Genre;\n \n if (!groups[genre]) {\n groups[genre] = [];\n }\n \n groups[genre].push(movie);\n }\n \n return groups;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":584,"value":"$39","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":586,"value":"function mean(arr) {\n return arr.reduce((a, b) => a + b, 0) / arr.length;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":588,"value":"function stddev(arr) {\n const mu = mean(arr);\n const variance = arr.reduce((sum, val) => sum + (val - mu) ** 2, 0) / arr.length;\n return Math.sqrt(variance);\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":616,"value":"#### [TO-DO]\nFollow the pseudo-code above to generate a new `normalized_movies` array of equal size to `movies_us_profits`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":590,"value":"// TO-DO\nnormalized_movies = {\n let normalized = [];\n \n for (let genre in genre_groups) {\n const group = genre_groups[genre];\n const profits = group.map(movie => movie.Profit);\n const mu = mean(profits);\n const sigma = stddev(profits);\n \n for (let movie of group) {\n const zscore = (movie.Profit - mu) / sigma;\n \n normalized.push({\n Title: movie.Title,\n Genre: genre,\n Profit: movie.Profit,\n \"Z-Score\": zscore\n });\n }\n }\n \n return normalized;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":619,"value":"---\n### 3D: Plotting Normalized Values (5%)\n\nWith our new `normalized_movies` set up, we can now plot and compare `Profit` and z-scored `Profit`.\n\n#### [TO-DO]\nGenerate two bar charts from `normalized_movies`. The first chart plots the mean `Profit` across each Genre. The second chart plots the z-scores of each Genre. Sort them along the x-axis in descending order of `Profit` or z-score. Two examples of what the z-scored `Profit` looks like is shown below, depending on which filters you used:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":647,"value":"FileAttachment(\"3D.png\").image({width:400})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":712,"value":"FileAttachment(\"3D_alt.png\").image({width:400})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":632,"value":"// TO-DO\nvl.markBar()\n .data(normalized_movies)\n .transform(\n vl.aggregate([{op: \"mean\", field: \"Profit\", as: \"Mean_Profit\"}])\n .groupby([\"Genre\"]),\n vl.window([{op: \"rank\", as: \"rank\"}])\n .sort([{field: \"Mean_Profit\", order: \"descending\"}])\n )\n .encode(\n vl.x().fieldN(\"Genre\").sort({field: \"Mean_Profit\", order: \"descending\"}),\n vl.y().fieldQ(\"Mean_Profit\"),\n )\n .title(\"Mean Profit by Genre\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":621,"value":"// TO-DO\nvl.markBar()\n .data(normalized_movies)\n .transform(\n vl.aggregate([{op: \"mean\", field: \"Z-Score\", as: \"Mean_ZScore\"}])\n .groupby([\"Genre\"]),\n )\n .encode(\n vl.x().fieldN(\"Genre\").sort({field: \"Mean_ZScore\", order: \"descending\"}),\n vl.y().fieldQ(\"Mean_ZScore\"),\n )\n .title(\"Mean Z-Score by Genre\")\n .render()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":637,"value":"---\n\n### 3E: Profit Analysis: Z-Score vs Raw Profit (10%)\n\nAnswer the following questions either individually or in a single paragraph.\n\n1. Highlight which genres have consistently high or low normalized scores.\n2. Which genres tend to overperform or underperform relative to their peers? Are there outliers?\n3. How does normalization change your interpretation compared to using raw values?\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":650,"value":"===============\n1. Drama and Action have the highest z-scores (overperform). Musical has the lowest (underperforms badly). Comedy, Rom-Com, and Western all underperform.\n \n2. Drama is the biggest overperformer. Musical, Western, and Black Comedy are the biggest underperformers.\n \n3. Raw profits show which genres have big budgets. Z-scores show which genres are actually good at making money relative to expectations. Musical has high raw profits but terrible z-scores - some make tons but most lose money. Drama consistently does well, which is more useful info.\n \n===============","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"}],"files":"$W3a","stdlib":"1"}} 30:"$2f:body"