2b:null 2c:["$","title",null,{"children":"Manuscript Outline | William Pang | Observable"}] 31:T465,md` **Figure** 1B: The purpose of this graph is to create a visualization representing the data changes/fixes that are made to the dataset, which provides more context into the complexity of the data we're working with. We would still need to flesh out how this ties in with the Hidden Markov Model. A sample graph could be something akin to the heat tile map we created back in the fall, with the edges of the boxes ${textcolor('highlighted', 'yellow')} representing data that got edited.

States editing data.

Or something Marcus [created](https://observablehq.com/@marcusrussi/state-performance)?

Marcus state performance "subway" map. Photo: [Marcus's Observable Notebook](https://observablehq.com/@marcusrussi/state-performance)

`32:T99d,md` *Last Updated: Feb 3* ## Motivation
- Case and death data reported by states and counties have been used by public health officials and policy makers to make decisions on how to respond to the pandemic - *In addition*, the public also uses this published information, which is heavily used in media outlets and other online tools, to understand the trajectory of the pandemic - Many of these data, however, contains ["bad"](https://blog.covidestim.org/posts/data-pipeline/) or anomalous data that, without *proper interpretation*, can cause confusion and even incorrect decision making. Examples of bad data include: - Non-Reporting Days: Basically '0s' in the data reports -- generally weekends or public holidays - Expected Dumps: Large spikes after non-reporting days (for example, the day after Martin Luther King Jr. Day) - Dumps: Generic unexpected big spikes in the case or detah datas ## Existing solutions, and why they need to be improved upon
- Use "rolling averages" (such as the 7-day rolling averages) to smooth over anomalies - Does not given context to anaomalous data, which might contain relationships that we are interested in - Cannot respond to sudden fluctations (i.e., imagine there was a sudden up and then a sudden down in the curve -- the curve would simply get smoothed over) - Definition of moving average can sometimes be arbitrary -- why not 14 days, why not 21 days moving average? In fact, the New York Times does [extend](https://github.com/nytimes/covid-19-data/tree/master/rolling-averages) their moving average when no data is reported - Anamoly detection tools - Examples include: - Time Series Anomaly Detection: Basically looks at a trend and if it seems something that doesn't match the trend, the tool would "flag" the anamolous datapoint - Machine Learning Tools: Usually requires large sets of "labelled" data in order to study patterns and flag anomalies given some threshold definition (e.g., the probability that this data point being 'nominal' in our distribution of data points is very low) - The challenge with these tools is that they don't provide context to the data (i.e., good at giving a binary conclusion of whether the data is anamolus or not, but not able to tell us if the data is non-reporting, expected dumps, dumps, etc.) - The advantage of a hidden markov model (HMM) is that it is simple to understand, define, and easy to implement ## Our Proposal `33:[{"id":"f04bfba0062cc29408717bed16b2ad26ec1c7eab57353ba8899c99b474969b47eaa62d505494345171724e86e61f81b2b5dbbc021b6a78d3cb7854ab6fced59e","href":"https://static.observableusercontent.com/files/f04bfba0062cc29408717bed16b2ad26ec1c7eab57353ba8899c99b474969b47eaa62d505494345171724e86e61f81b2b5dbbc021b6a78d3cb7854ab6fced59e","name":"1BMarcus.png","type":"image/png","lastModified":1645371476933.549,"size":440192},{"id":"f05c64d8af2d686eefc3082a546cc3c579c23ac91c4a1f9c2ca835b7a4492b95320d9403fceefe4a909bf1f84594eace2a89640550f4fd423cf6bc0d93a68ded","href":"https://static.observableusercontent.com/files/f05c64d8af2d686eefc3082a546cc3c579c23ac91c4a1f9c2ca835b7a4492b95320d9403fceefe4a909bf1f84594eace2a89640550f4fd423cf6bc0d93a68ded","name":"Rplot@1.png","type":"image/png","lastModified":1645372011873.758,"size":135513},{"id":"c3b5e27b8bb079ca883d819d76e31bec98ff0ee4b85a3670f6eab3171b643965d59677f68a0fa8f6571a1d0eb152f0b7ee3d5909be9b43eead809b11c5ac2fbb","href":"https://static.observableusercontent.com/files/c3b5e27b8bb079ca883d819d76e31bec98ff0ee4b85a3670f6eab3171b643965d59677f68a0fa8f6571a1d0eb152f0b7ee3d5909be9b43eead809b11c5ac2fbb","name":"HMM.png","type":"image/png","lastModified":1645372247294.918,"size":153957},{"id":"a29886907c669498d4b6934bc278c82af69646dbb2f99fb0915dd5c83e1e62c6dbf066c6c468c171958487fe37f323123f1f5ccbb93095bf7d7e7e8b35c0bced","href":"https://static.observableusercontent.com/files/a29886907c669498d4b6934bc278c82af69646dbb2f99fb0915dd5c83e1e62c6dbf066c6c468c171958487fe37f323123f1f5ccbb93095bf7d7e7e8b35c0bced","name":"NonInformative.png","type":"image/png","lastModified":1645374059309.279,"size":369099},{"id":"980348f4883696d72c422c9e02d664385e3eb3a390a46e8a633d3736f4d042cac25a885cebb9fbf3e03061f96ea7d7dde96459c16a909d4f1c7fca675ace7be4","href":"https://static.observableusercontent.com/files/980348f4883696d72c422c9e02d664385e3eb3a390a46e8a633d3736f4d042cac25a885cebb9fbf3e03061f96ea7d7dde96459c16a909d4f1c7fca675ace7be4","name":"Anomaly_Graph_Azure.png","type":"image/png","lastModified":1644929430340.68,"size":126734},{"id":"0de3b09269d55f319c997e27d18461e096383bf917cac1fae7a85c7b57920f0aee96f4b7ec4f56fc81c61c653d63a8c7f7eab3a1c7d97006f6eb166596e80b26","href":"https://static.observableusercontent.com/files/0de3b09269d55f319c997e27d18461e096383bf917cac1fae7a85c7b57920f0aee96f4b7ec4f56fc81c61c653d63a8c7f7eab3a1c7d97006f6eb166596e80b26","name":"Heatmap_Marcus.png","type":"image/png","lastModified":1644930099114.12,"size":160602},{"id":"c3b5e27b8bb079ca883d819d76e31bec98ff0ee4b85a3670f6eab3171b643965d59677f68a0fa8f6571a1d0eb152f0b7ee3d5909be9b43eead809b11c5ac2fbb","href":"https://static.observableusercontent.com/files/c3b5e27b8bb079ca883d819d76e31bec98ff0ee4b85a3670f6eab3171b643965d59677f68a0fa8f6571a1d0eb152f0b7ee3d5909be9b43eead809b11c5ac2fbb","name":"Rplot.png","type":"image/png","lastModified":1645371260404.185,"size":153957},{"id":"238b8e4bc53e4d6a122d4a95aab8d289f8e1cf8542234ac7bc0aac570c1935cf26c3382a9af71e353e100c173a54e27ed7f8b9443ba493dd3bb43c5536510aa2","href":"https://static.observableusercontent.com/files/238b8e4bc53e4d6a122d4a95aab8d289f8e1cf8542234ac7bc0aac570c1935cf26c3382a9af71e353e100c173a54e27ed7f8b9443ba493dd3bb43c5536510aa2","name":"anomalies.csv","type":"text/csv","lastModified":1643567545935.976,"size":190260}] 2f:{"id":"040bc8c473890b45","slug":"manuscriptdraft","title":"Manuscript Outline","thumbnail":"61c90b75797ea2ac0a44e828e8caaa6812d5d997b8b2fa1325bff56de0b52962","access_level":"public","create_time":"$D2022-01-22T16:18:47.076Z","update_time":"$D2022-02-23T02:07:47.569Z","view_time":null,"first_public_time":"$D2023-01-29T16:21:03.740Z","fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"88df4b53b2fce8d2","login":"will-pang","name":"William Pang","avatar_url":"https://avatars-next.observableusercontent.com/4e5603477ef111cf1fc74cb0aab0dd58ddab37219e1b2a4e5e9c3d082c5532a0","tier":"starter_2024"},"version":425,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"30424cd7adf510bb","login":"williampangbest1","name":"William Pang","avatar_url":"https://avatars-next.observableusercontent.com/4e5603477ef111cf1fc74cb0aab0dd58ddab37219e1b2a4e5e9c3d082c5532a0","tier":null}],"members":[{"type":"individual","id":"30424cd7adf510bb","login":"williampangbest1","name":"William Pang","avatar_url":"https://avatars-next.observableusercontent.com/4e5603477ef111cf1fc74cb0aab0dd58ddab37219e1b2a4e5e9c3d082c5532a0","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":1,"collection":{"id":"c3b0b311dad03829","slug":"state-performance-1","title":"COVIDestim","owner":{"id":"88df4b53b2fce8d2","login":"will-pang","name":"William Pang"}},"type":"notebook","body":{"title":"Manuscript Outline","cells":[{"id":0,"value":"# Manuscript Outline","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":264,"value":"md`\n## Figures (Update Feb 15)\n*The layout of these figures is based on Marcus's [slides](https://drive.google.com/file/d/1UxVJwhsnO_MzwYhlZkht7hr92eJo9rFX/view?usp=sharing) back in September on the rationale behind doing a data anaomly detector.*\n\nA few main points we want our figures to tell us:\n- Contextualizing the problem by showing the problems with the raw input data (**Figure** 1A)\n- Show how states end up revising their data sets as well, adding a layer of complexity (**Figure** 1B)\n- Demonstrate labelled/contextualized anomalies using a Hidden Markov Model (**Figure** 2)\n- As a *proof of concept*, show how contextualized anomalies can inform decisions to modify time series (**Figure** 3B) and comparing it with unlabelled/unmodified time series graphs (**Figure** 3A)\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":288,"value":"md`\n**Figure** 1A: This graph should be *fairly* easy to make and perhaps more thought should be placed on the best way to visualize the state of the problem.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":296,"value":"md`\n

Anomaly Detection for Time Series Data. Photo: [Github](https://github.com/DHI/tsod)

\nWe could do something like this, where we show time series data (with the confidence bounds included) and then point out time points where there are spikes.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":408,"value":"inputData = d3.json('https://api.covidestim.org/latest_state_inputs?state=eq.Massachusetts&date=gt.2020-02-01&date=lt.2021-11-01&select=date,cases').then(d => d.map(({date, ...rest}) => ({date: new Date(date), ...rest})))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":413,"value":"Plot.plot({width:width, marks: [Plot.line(inputData, {x:'date',y:'cases'})]})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":351,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":376,"value":"md`\n**Figure** 2: This graph should demonstrate that we are able to contextualize data anomalies based on the four classifications that we specified. \n\nWe could do something similar to what Marcus created, where different tile colors 🟩 would represent the hidden states.\n\n

Visualizing HMM classifications. Photo: [Marcus's Slides](https://drive.google.com/file/d/1UxVJwhsnO_MzwYhlZkht7hr92eJo9rFX/view)

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":402,"value":"md`\n**Figure** 3: Would need Marcus to help flesh this out a little more, but the basic idea is to show as a proof-of-concept that making changes to the model with extra context can be useful in removing \"non-informative\" information.\n\n

Applications of input data classifier. Photo: [Marcus's Slides](https://drive.google.com/file/d/1UxVJwhsnO_MzwYhlZkht7hr92eJo9rFX/view)

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":337,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":5,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":130,"value":"md`\n- Use a [hidden markov model](https://observablehq.com/@marcusrussi/input-data-classifier) with 4 defined states (that we cannot see) and 4 defined observations (that we can empirically see from the data)\n- In Marcus's implenetation, he made eduated guesses about the transmission and emission matrix, but as discussed with Ted we should come up with a more empircal way (when possible) to estimate the values in these matricies\n- A rough sketch of the idea is to:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":136,"value":"dot`\ndigraph{\nrankdir=\"LR\";\n\na -> b \nb -> c\nc -> d\n\n\na [shape=\"rectangle\",label=\"Create a labelled dataset\\n for period \\u03B1 in US State \\u03A9\"]\nb [shape=\"rectangle\",label=\"Compute Emission and Transmission\\n Probabilites for period \\u03B1 in US State \\u03A9\"]\nc [shape=\"rectangle\",label=\"Use the HMM to label a dataset for\\n period \\u03B2 in US State \\u03A9\"]\nd [shape=\"rectange\",label=\"Validation\\n (Needs to be fleshed out)\"]\n}\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":172,"value":"md`\n- In order to create a labelled dataset:\n - We need to set some \"rules\" on how we define the four states\n - We would also need to *manually* label anamolies that are rare (such as unexpected dumps). After discussion with Marcus, we decided it was best to pick areas where we think reporting of anomalies is most robust. We decided to zero in on the State of New York for now. Anomaly reporting can be found at:\n - New York Times compiles anomalies in a CSV file which can be downloaded on [Github](https://github.com/nytimes/covid-19-data/tree/master/rolling-averages). A (not too extensive) list specifically looking at NYC & NY State can be found [here](https://www.nytimes.com/interactive/2021/us/new-york-city-new-york-covid-cases.html).\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":195,"value":"viewof selection = Inputs.table(nyt,{columns:[\"date\",\"state\",\"type\",\"description\"]}) //I have chosen for only some columns to be shown in the table to make it cleaner","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":204,"value":"selection //If you want to view the description you can use the check the box in the table and then find the complete description in the array","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":232,"value":"md`\n## Limitations
\n- The robustness of HMM really depends on how well we are able to \"label\" the data -- in other words, this implementation would work well in states/counties that are transparent with their reporting\n- I'm sure there are other limitations that we can expand on framed under a call for action to have better data reporting...\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":96,"value":"md ` ## Miscallenous\n- I wrote a short explainer on Hidden Markov Models and the Viterbi algorithim, which can be acessed [here](https://observablehq.com/@williampangbest1/hidden-markov-models?collection=@williampangbest1/half-baked).\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":27,"value":"references = md`### References\n\n

${DOM.text(d.url.replace(/^http[s]\\:\\/\\//, \"\"))}`)}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":104,"value":"md`\n### Observbale Stuff\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":117,"value":"import {reference} from \"@mbostock/reference\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":21,"value":"superscript = \"⁰¹²³⁴⁵⁶⁷⁸⁹\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":30,"value":"mutable data = []","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":191,"value":"nyt=FileAttachment(\"anomalies.csv\").csv({typed:true})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":352,"value":"import {textcolor} from \"@observablehq/text-color-annotations-in-markdown\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W33","stdlib":"1","titling":"auto"}} 30:"$2f:body"