34:["$","title",null,{"children":"Project 2 - Spotify - Post-Feedback Corrections | John Cambefort | Observable"}] 3c:T505,md`A couple of interesting things to note here: - Most of the songs in this dataset have a stream count of roughly 5 million and higher. That being said, some songs have as low as ~4.1 million streams, such as Lizzo's "Good as Hell". - The data seems to have a few erroneous values for popularity. It seems impossible that Selena Gomez would put out "Look at Her Now" with a stream count of over 5 million, yet a popularity of zero. If we look closely, we see that the song's release date, along with all the other songs with similar low popularities, is in early August 2021. If we look on Kaggle, we see that the dataset was updated on August 16th 2021. Consequently, we can assume that for songs to score high on the 'popularity' chart, they must have been around for some time, even if they smashed the charts right after release. - Popularity ranges widely for songs with likewise low stream counts (e.g., songs with between 5 and 10 million streams have wildly different popularities), but for any additional stream counts, popularity seems to shoot up. Why might this be? There must be variables other than Streams that go into defining popularity for songs with popularities between 40 and 85. Next, we'll try comparing popularity against a song's highest charting position. `3d:T731,It seems as though there are some songs for which certainly a high stream count correlates with a high charting position, however for many other songs, this is not the case as evidenced in the straight line that plateaus at 5 million streams, at the bottom of the graph. Perhaps songs that have these relatively low stream counts have high charting positions only temporarily, and soon after drop off the high end of the charts? ### Reverse Engineering Popularity using a Random Forest For my (John's) Statistical Learning course, I chose to reverse-engineer the Spotify algorithm ([see the complete report here](https://johncambefort.github.io/spotify_popularity/)) that determines a song's Popularity using the Random Forest model. After optimizing hyperparameters, the model was able to predict Popularity with an R-Mean Squared Error of ~4.2; the model's Rsquared value was of 0.8, meaning that our variables could explain 80% of variance in predictions. This is pretty good! The `ranger` package was used to build the Random Forest model and accessed via the wrapper `caret` package `train` function. After excluding the variables `Genre` and `Weeks Charted` (pre-processing these for the model was too complex to achieve by the assignment deadline), and after preserving only songs between January 1st 2020 and July 21st 2021, the model found the following order of variable importance: ``` Variable Importance 1. Number.of.Times.Charted 100.0000 2. HighStart 82.0593 3. Release.Date 23.5023 4. Streams 21.0299 5. Highest.Charting.Position 17.9335 ``` (*Note*: HighStart is the Date at which a song reached its maximum peak on the charts). The top 4 variables are largely related with time. Let's plot the relationship between a few of these and Popularity.3e:T477,{ const startdate = new Date("2020-01-01"); const enddate = new Date("2021-07-21") - 15; const graph = vl .layer([ vl .markCircle({ size: 80, opacity: 0.5 }) .encode( vl.x().fieldT("Release Date").timeUnit("yearmonthdate"), vl.y().fieldQ("Popularity"), vl.tooltip([ "Song Name", "Artist", "Release_Date", "Number_of_Times_Charted" ]) ), vl .markLine({ color: "red" }) .transform(vl.regression("Popularity").on("Release Date")) .encode( vl.x().fieldT("Release Date").timeUnit("yearmonthdate"), vl.y().fieldQ("Popularity") ) ]) .data( spotify.filter( (d) => (d["Release_Date"] > startdate) & (d["Release_Date"] < enddate) & (d["Week_of_Highest_Charting"] > startdate) ) ) .transform([ vl.filter("datum.Popularity > 0"), vl.timeUnit("month").field("date").as("month") ]) .title("Song Popularity & Release Date") .height(500) .width(700); return graph.render(); }3f:T405,{ const startdate = new Date("2020-01-01"); const enddate = new Date("2021-07-21") - 15; const graph = vl .layer([ vl.markCircle({ size: 80, opacity: 0.5 }).encode( vl.y().fieldQ("Number of Times Charted"), vl.x().fieldQ("Popularity"), vl .color() .fieldT("Release Date") .timeUnit("yearmonthdate") .scale({ range: ["orange", "purple"] }), vl.tooltip([ "Song Name", "Artist", "Release_Date", "Number_of_Times_Charted", "Popularity" ]) ) ]) .data( spotify.filter( (d) => (d["Release_Date"] > startdate) & (d["Release_Date"] < enddate) & (d["Week_of_Highest_Charting"] > startdate) ) ) .transform([ vl.filter("datum.Popularity > 0"), vl.timeUnit("month").field("date").as("month") ]) .title("Number of Times Charted & Popularity") .height(500) .width(700); return graph.render(); }40:T410,The relationship here is less clearly linear, although the regression line seems to suggest that there might be some correlation. In any case, this is our most revealing graph. Here, we see that new songs (colored in purple) are generally quite popular, whereas older songs (in yellow) are not so popular. Then, we notice a few old songs (yellow) that have a pretty high popularity metric and that have charted many times. We'll call these songs our "oldies but goodies" -- these are songs that have stubornly stayed on the charts (and thus, "charted" "many" times), and so Spotify allows their popularity metric to stay high. If a song hasn't charted in a while, Spotify decays its popularity value. This makes sense -- popular songs should be songs that are hot and that folks are listening to *right now*. We can say with some degree of confidence that variables which are representative of time seem to be good predictors of the Popularity variable, and that Spotify decays a song's Popularity over time if the song is not charting.41:T52d,{ const startdate = new Date("2020-01-01"); const enddate = new Date("2021-07-21") - 15; return vl .layer([ vl .markCircle({ opacity: 0.75 }) .transform([ vl .calculate( "datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]" ) .as("firstweek"), vl.calculate("datum['Genre'][0]").as("firstgenre"), // vl.filter("datum['Release_Date'] < datum['firstweek']"), // what's up with these data points? vl.timeUnit("month").field("date").as("month") ]) .encode( vl.y().fieldT("Week_of_Highest_Charting").timeUnit("yearmonthdate"), vl.x().fieldT("Release Date").timeUnit("yearmonthdate"), vl.tooltip(["Song Name", "Popularity", "Genre", "Release_Date"]), vl.color().fieldN("is_pop") ), vl .markRule({ color: "yellow" }) .encode( vl.y().value(5), vl.x().value(600), vl.y2().value(600), vl.x2().value(10) ) ]) .data( spotify.filter( (d) => (d["Release_Date"] > startdate) & (d["Week_of_Highest_Charting"] > startdate) ) ) .title("Release Date vs Week of Higest Charting") .width(600) .height(600) .render(); }42:T6f1,{ const startdate = new Date("2020-01-01"); return vl .layer([ vl .markCircle({ size: 70, opacity: 0.75 }) .transform([ vl .calculate( "datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]" ) .as("firstweek"), vl .calculate( "(datum['Week_of_Highest_Charting']-datum['Release_Date'])/(1000 * 3600 * 24)" ) .as("Climbing_Time"), vl .calculate( "(datum['Weeks_Charted'][0] - datum['Week_of_Highest_Charting'])/(1000 * 3600 * 24)" ) .as("Falling_Time"), vl.filter( "datum['Climbing_Time'] < 280 && datum['Falling_Time'] < 280" ), vl.filter("datum['Release_Date'] < datum['firstweek']") ]) .encode( vl.x().fieldQ("Climbing_Time"), vl.y().fieldQ("Falling_Time"), vl .color() .fieldQ("Highest_Charting_Position") .sort("descending") .scale({ range: ["purple", "yellow"] }), vl.tooltip(["Song Name", "Artist", "Highest_Charting_Position"]) ), vl .markRule({ color: "lightblue" }) .encode( vl.y().value(0), vl.x().value(230), vl.y2().value(600), vl.x2().value(230) ), vl .markRule({ color: "lightblue" }) .encode( vl.y().value(435), vl.x().value(0), vl.y2().value(435), vl.x2().value(600) ) ]) .data(spotify.filter((d) => d["Release_Date"] > startdate)) .title("Time it takes for a song to rise vs fall (in days)") .width(600) .height(600) .render(); }43:[{"id":"a20ac3e4f9a561c7fceda3e442f49b56d28cd17c3f01123ba52df52ed72f52fcf3698b07f44479c23c85bd5bea667b5796291a383c93b415f20c933de90f280b","href":"https://static.observableusercontent.com/files/a20ac3e4f9a561c7fceda3e442f49b56d28cd17c3f01123ba52df52ed72f52fcf3698b07f44479c23c85bd5bea667b5796291a383c93b415f20c933de90f280b","name":"spotify_dataset.csv","type":"text/csv","lastModified":1635520657367.338,"size":738124}] 38:{"id":"f800758d76c5b7c7","slug":"project-2-spotify-post-feedback-corrections","title":"Project 2 - Spotify - Post-Feedback Corrections","thumbnail":"7ba471df180ad473e59220c6c7d0f7cab0998300cffc6b59b7b156eb725a9fb8","access_level":"public","create_time":"$D2021-10-29T15:17:37.367Z","update_time":"$D2021-12-29T09:58:29.976Z","view_time":null,"first_public_time":"$D2021-12-28T12:05:54.373Z","has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"45c865f8cbea953c","login":"johncambefort","name":"John Cambefort","avatar_url":"https://avatars-next.observableusercontent.com/9768e632bc0471ac818fbae15471293f499dd102c7a500fecd4f0b8a94a1ec85","tier":"starter_2024","bio":"Computer Science student @ Middlebury College","followed_by_you":false,"follows_you":false},"version":1573,"publish_version":1573,"role":null,"authored":false,"authors":[{"type":"individual","id":"a5e68e21b7abc8ef","login":"johncambefort","name":"John Cambefort","avatar_url":"https://avatars-next.observableusercontent.com/9768e632bc0471ac818fbae15471293f499dd102c7a500fecd4f0b8a94a1ec85","tier":null,"bio":"Computer Science student @ Middlebury College","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"83455f38fe8f6883","login":"christopherpandrews","name":"Christopher Andrews","avatar_url":"https://avatars-next.observableusercontent.com/e2f445a7bad2204845ceaf2d487e500674cd2ac7f8bd2544c1da00d1edb25a2c","tier":null,"role":"editor"},{"type":"individual","id":"7e7aeb139125af67","login":"hjarai","name":"Hirona Jacqueline Arai","avatar_url":"https://avatars-next.observableusercontent.com/b4a47f66241f0eb207847c30119859a92420cb2df61181c852a28e0e5e8c6d43","tier":null,"role":"editor"},{"type":"individual","id":"a5e68e21b7abc8ef","login":"johncambefort","name":"John Cambefort","avatar_url":"https://avatars-next.observableusercontent.com/9768e632bc0471ac818fbae15471293f499dd102c7a500fecd4f0b8a94a1ec85","tier":null,"role":"editor"}],"comments":[{"id":"f449f6dae072e170","reply_to_id":null,"content":"Yes, this is an improvement on your first pass. I'm glad you took the time to go back to it. I'm also interested to hear that you decided to tear the same dataset apart in statistical learning...","create_time":"$D2021-11-12T04:21:33.701Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"83455f38fe8f6883","login":"christopherpandrews","name":"Christopher Andrews","avatar_url":"https://avatars-next.observableusercontent.com/e2f445a7bad2204845ceaf2d487e500674cd2ac7f8bd2544c1da00d1edb25a2c","tier":null}}],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Project 2 - Spotify - Post-Feedback Corrections","cells":[{"id":0,"value":"# Project 2 - Spotify - Post-Feedback Corrections\nHirona Arai & John Cambefort\n\nThis notebook was submitted for an in-class assignment in our Data Visualizations course. It contains a few exploratory graphs on the dataset, with a few findings!\n\nOriginal project 2 report containing Prof. Andrew's comments: https://observablehq.com/d/0825e001302bd681#comment-43302f2e0e52a3b1\n\nLink to Kaggle dataset with description: https://www.kaggle.com/sashankpillai/spotify-top-200-charts-20202021","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":846,"value":"md`# Popular paparazzi...`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":597,"value":"md`An initial glance at the dataset shows an interesting variable titled 'Popularity', ranging from 0 (less popular) to 100 (most popular). The authors of the dataset decided that certain variables should determine a song's popularity.\nWhat are some of these predictor variables? Let's see if we can uncover a few.\n\nLet's start by looking at whether stream count is a reasonable predictor of this popularity metric.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":593,"value":"// Curious about the 'Popularity' metric rated from 0 to 100: how did the dataset authors define 'Popularity'?\n// What if we tried plotting Popularity against Streams\n\n{\n const graph = vl\n .layer([\n vl\n .markCircle({ size: 80, opacity: 0.35 })\n .encode(\n vl.x().fieldQ(\"Popularity\"),\n vl.y().fieldQ(\"Streams\"),\n vl.tooltip([\"Song Name\", \"Artist\", \"Release Date\", \"Streams\"])\n ),\n vl\n .markRule({ color: \"red\", strokeWidth: 2 })\n .encode(\n vl.x().value(0),\n vl.y().value(400),\n vl.x2().value(700),\n vl.y2().value(400)\n ),\n vl\n .markRule({ color: \"red\", strokeWidth: 2 })\n .encode(\n vl.x().value(0),\n vl.y().value(450),\n vl.x2().value(700),\n vl.y2().value(450)\n )\n ])\n .data(spotify)\n .title(\"Relationship between Popularity and Streams\")\n .height(500)\n .width(700);\n\n return graph.render();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":609,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":737,"value":"{\n const graph = vl\n .layer([\n vl\n .markCircle({ size: 80, opacity: 0.5 })\n .encode(\n vl.x().fieldQ(\"Popularity\"),\n vl.y().fieldQ(\"Highest_Charting_Position\"),\n vl.tooltip([\n \"Song Name\",\n \"Artist\",\n \"Release Date\",\n \"Number_of_Times_Charted\"\n ])\n ),\n vl\n .markRule({ color: \"red\", strokeWidth: 2 })\n .encode(\n vl.x().value(400),\n vl.y().value(0),\n vl.x2().value(400),\n vl.y2().value(500)\n ),\n vl\n .markRule({ color: \"red\", strokeWidth: 2 })\n .encode(\n vl.x().value(600),\n vl.y().value(0),\n vl.x2().value(600),\n vl.y2().value(500)\n )\n ])\n .data(spotify)\n .transform([vl.filter(\"datum.Popularity > 0\")])\n .title(\"Song Popularity against Highest Charting Position\")\n .height(500)\n .width(700);\n\n return graph.render();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":743,"value":"It seems as though highest charting position is not a very good predictor of song popularity. There are many songs that have wildly different highest charting positions, yet the same popularity metric (see all of the song within the two red bars, for instance).\nLet's see what happens if we visualize both highest charting position and stream count against popularity.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":451,"value":"vl\n .markCircle({ size: 80, opacity: 0.5 })\n .data(spotify)\n .transform([vl.filter(\"datum.Popularity > 0\")])\n .encode(\n vl.x().fieldQ(\"Highest_Charting_Position\").sort(\"descending\"),\n vl.y().fieldQ(\"Streams\"),\n vl\n .color()\n .fieldQ(\"Popularity\")\n .scale({ range: [\"yellow\", \"orange\", \"red\"] }),\n vl.tooltip([\"Song Name\", \"Artist\", \"Release Date\", \"Popularity\"])\n )\n .title(\n \"Song Stream Count and Highest Charting Position, colored by Popularity\"\n )\n .height(500)\n .width(700)\n .render()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":647,"value":"$3d","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1472,"value":"$3e","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1493,"value":"The relationship between Release Date and Popularity is one of the most linear we've seen so far.\n\nLet's try `Number of Times Charted` and Popularity, but we'll color by release date.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1499,"value":"$3f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1509,"value":"$40","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":532,"value":"# Time to Peak\n\nWe are also curious about how long it takes songs to reach its maximum popularity after being released. \nBecause we only have data about charting from Jan 2020 - July 2021, we're only going to only look at songs released in the same time range.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":185,"value":"$41","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":649,"value":"First weird thing: some songs seem to chart before they were released (everything in the bottom triangle!), so we have filtered out for those songs.\nAlso, most songs seem to chart the same week that they're released.... We will calculate the time it takes for songs to chart: perhaps genre has something to do with how quickly a song will gain popularity.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":681,"value":"{\n //climbing time is # of days difference b/w release date and week of highest chart\nconst startdate = new Date(\"2020-01-01\");\n\nreturn vl.markBar()\n.data(spotify.filter(d => d['Release_Date'] > startdate))\n.transform([\n vl.calculate(\"datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]\").as(\"firstweek\"),\n vl.calculate(\"(datum['Week_of_Highest_Charting']-datum['Release_Date'])/(1000 * 3600 * 24)\").as(\"Climbing_Time\"),\n vl.filter(\"datum['Release_Date'] < datum['firstweek']\")\n])\n.encode(\n vl.x().fieldQ(\"Climbing_Time\").bin({step:3}),\n vl.y().count(\"Total\"),\n vl.color().fieldN(\"is_pop\")\n)\n.title('Distribution of time it took for released songs to reach highest position')\n.width(900)\n.height(600)\n.render()\n\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":909,"value":"Two things:\n- The total distribution looks like it forms a y = 1/x curve\n- Can't really tell whether pop is more likely to climb faster... honestly the proportions look pretty much even across climbing_time\n\nLet's also encode \n1. What that peak position on the chart was\n2. for how many weeks the song charted","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":982,"value":"{\n const startdate = new Date(\"2020-01-01\");\n\n return vl\n .markBar()\n .data(spotify.filter((d) => d[\"Release_Date\"] > startdate))\n .transform([\n vl\n .calculate(\"datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]\")\n .as(\"firstweek\"),\n vl\n .calculate(\n \"(datum['Week_of_Highest_Charting']-datum['Release_Date'])/(1000 * 3600 * 24)\"\n )\n .as(\"Climbing_Time\"),\n vl.filter(\"datum['Release_Date'] < datum['firstweek']\")\n ])\n .encode(\n vl.x().fieldQ(\"Climbing_Time\").bin({ step: 3 }),\n vl.y().count(\"Total\"),\n vl.color().fieldQ(\"Highest_Charting_Position\").sort(\"descending\")\n )\n .title(\n \"Distribution of time it took for released songs to reach highest position (highest charting position)\"\n )\n .width(800)\n .height(600)\n .render();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":970,"value":"{\nconst startdate = new Date(\"2020-01-01\");\n\nreturn vl.markBar()\n.data(spotify.filter(d => d['Release_Date'] > startdate))\n.transform([\n vl.calculate(\"datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]\").as(\"firstweek\"),\n vl.calculate(\"(datum['Week_of_Highest_Charting']-datum['Release_Date'])/(1000 * 3600 * 24)\").as(\"Climbing_Time\"),\n vl.filter(\"datum['Release_Date'] < datum['firstweek']\")\n])\n.encode(\n vl.x().fieldQ(\"Climbing_Time\").bin({step:3}),\n vl.y().count(\"Total\"),\n vl.color().fieldQ(\"Number_of_Times_Charted\").scale({range:[\"yellow\", \"red\"]})\n)\n.title('Distribution of time it took for released songs to reach highest position (# weeks charted)')\n.width(1000)\n.height(600)\n.render()\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":979,"value":"Interestingly, it seems like the majority of songs in the first columns (took 0 weeks for song to reach its peak position) didn't stay on the chart for very long. (Darker hues live proportionally more on the right side of the graph) This makes sense, since the songs would have had to take time climbing up while staying on the chart, but it seems to also signify that the songs drop off of the chart quickly, as well.\n\nWhat about climbing time vs falling time?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1037,"value":"$42","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1030,"value":"The lack of points in the x >100 and 100> y >60 section tells us that once a song takes more than 100 days to rise in the charts, it is unlikely for it to stay on the charts for longer than 60 more days.\n\n\n\nCan we visualize the correlation between the highest charting position vs the number of times the song charted?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":997,"value":"{\nconst startdate = new Date(\"2020-01-01\");\n\nreturn vl.markCircle({size: 45, opacity: 0.5})\n.data(spotify.filter(d => d['Release_Date'] > startdate))\n.transform([\n vl.calculate(\"datum['Weeks_Charted'][datum['Weeks_Charted'].length-1]\").as(\"firstweek\"),\n vl.calculate(\"datum['Genre'][0]\").as(\"firstgenre\"),\n vl.filter(\"datum['Release_Date'] < datum['firstweek']\"),\n vl.timeUnit(\"month\").field(\"date\").as(\"month\")\n])\n.encode(\n vl.y().fieldQ(\"Highest_Charting_Position\").sort('descending'),\n vl.x().fieldQ(\"Number_of_Times_Charted\"),\n vl.tooltip(['Song Name',\n 'Popularity',\n 'Genre']),\n )\n\n.title('Longevity and Highest charting position')\n.width(700)\n.height(600)\n.render()\n\n}\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1025,"value":"The higher the highest charting position is, the longer it tends to have been on the charts. It's unlikely for a song to have charted over 20 times and not reach at least top 50 on the rankings","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"md`# Appendix & Data Imports`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"import {vl} from '@vega/vega-lite-api-v5'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":326,"value":"spotify_original = FileAttachment(\"spotify_dataset.csv\").csv()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":483,"value":"spotify = spotify_original.map((item) => {\n return {...item, \n Genre: eval(item['Genre']),\n Highest_Charting_Position: +item['Highest Charting Position'],\n Number_of_Times_Charted: +item['Number of Times Charted'],\n Artist_Followers: +item['Artist Followers'],\n Streams: +item['Streams'].replaceAll(\",\", \"\"),\n Popularity: +item['Popularity'],\n Danceability: +item['Danceability'],\n Energy: +item['Energy'],\n Loudness: +item['Loudness'],\n Speechiness: +item['Speechiness'],\n Acousticness: +item['Acousticness'],\n Liveness: +item['Liveness'],\n Tempo: +item['Tempo'],\n Duration: +item['Duration (ms)'],\n Valence: +item['Valence'],\n Weeks_Charted: item['Weeks Charted'].split(\"\\n\")\n .map(item => new Date(item.split(\"--\")[0])), //just need the start of the week\n\n Release_Date: new Date(item['Release Date']),\n Week_of_Highest_Charting: new Date(item['Week of Highest Charting'].split(\"--\")[0]),\n is_pop: item['Genre'].includes('pop')\n }}); \n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W43","stdlib":"1"}} 33:null