html`

Introduction

The broad goal of our project is to explore correlations between the stock price of Facebook and how it is talked about in TechCrunch articles. Stock price is incredibly unpredictable but with limited information, most investors look towards news reports to gauge how they should invest. This is our motivation in quantifying news article sentimentality and see how well it matches up with the market.

Related Work

Text sentimentality is a huge field in natural language processing and many exciting projects exist that portray it. For example, the following app: https://www.csc2.ncsu.edu/faculty/healey/tweet_viz/tweet_app/ will plot user inputted text onto different visualization formats for text sentimentality. Out of the many possible ways we explored to map sentimentality, we chose a format similar to this project: https://beta.observablehq.com/@mbostock/d3-calendar-view as we wanted to see data that spanned a full year.

Methodology

For our project, we first needed a dataset containing a list of relevant news article links. We considered using google news API, but we quickly realized that they only provide one month’s worth of data. The advantage there however was that they collect data from multiple news sources. However, since we wanted to see trends over multiple years, we decided to use a web scraper. The web scraper we used is actually from a previous hackathon project to scrape from TechCrunch. TechCrunch is special because of the specific way they list their url’s that makes it simple to scrape. Once getting the list of url’s, we parsed out the relevant ones by looking for a substring of “facebook” in the article title. Then, for each url, we retrieved the full article text along with the title using Beautiful Soup (a python library) and ran the tokenized text through a sentimentality analyzer (NLTK). The sentiment is defined as -1 for purely negative sentiment, and +1 for purely positive sentiment (with 0 meaning neutral). Our final data format was the date the article is published, the link to the article, the title of the article, and the sentiment score. Our data files and code can be found here: https://github.com/wanlinxie/dataviztry2 The second part of the data we needed was the stock data. This was simply retrieved from Yahoo Finance where we downloaded daily historical data. For our graph, we decided to only use the close price (the price the stock ends on when the market closes each day) to simplify the visualization.

Design

When choosing our design, we wanted to provide a clear correlation between the stock data and news article sentiment. The format of the stock data we decided should be a line graph as that is the most recognizable format in viewing stock price. It also makes sense as we want to see the trend over time. However, for the news sentiment, we explored a couple options. Our first idea was to plot the sentiment as a different line graph over the stock price (after scaling it to match the stock price). However, due to the high frequency of the news articles, the line was extremely intelligible and this idea was quickly abandoned. We also tried converting the sentiment to a scatter plot to see if it would look less messy. However, it was extremely hard to see a correlation and the sentiment points looked like noise. Finally, we settled on a calendar-view of a heat map. A heat map for encoding sentiment makes sense since we are trying to encode negativity and positivity. Therefore, we chose a scale from red to green. The calendar view also enables us to encode the date of article. This enables the user to see sentiment trend over time instead of a single article on its own. This also helps us clean up the data as we are able to average sentiment scores if multiple articles are published on one day. To provide the extra information (clickable links), we originally decided on a pop-up that would appear once a user clicked on the heat map square. However, we realized that the pop-up was distracting and eventually moved it to a fixed side panel.

Implementation

For our implementation, we used Python to create our dataset and parse through our data. For our visualization, everything is made using D3 in this Observable notebook.

Discussion

We feel that our final presentation was successful as this is a topic that has very useful and practical applications, especially for investors. There was a lot of interest into how this tool could be expanded and improved to be even more indicative of the market. We hope that our work provides new insight about one of the factors in stock price and is a fun exploratory tool for people interested in the market.

Future Work

We would love to improve this project in the future so it could perhaps one day be used in fields like quantitative trading. Because we were only able to scrape data from TechCrunch, we saw that our data was leaning more positive in general. By adding links from more datasets, we can expand our knowledge base which could lead to more interesting discoveries. We would love to add data from sites like MarketWatch or the Wall Street Journal and observe how it affects the overall sentiment. Due to the way we have written our code and structured this notebook, adding new data is extremely easy once we are given access to the links.

Acknowledgements or References

Thank you so much to Professor Agnes Chang and the TAs Conder Shou and Jeevan Farias for providing us guidance this semester. Also thank you to everyone who was at our mid-project review and the final showcase that gave us their insight. `