2f:T5f8,{ // If this page is outside of Observable, set the default styles to approximate // the appearance of the Observable Look and Feel. if (ui === "custom") { return htl.html`

Note: This is an archived version of an Observable Notebook supplement for the RAMPVIS PhilTransA submission titled: 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"
For the live version, see here: https://observablehq.com/d/e635bbb87b89b463

`; } else { return htl.html`

Note: This is a live version of the Observable Notebook supplement for the RAMPVIS PhilTransA submission titled 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"

`; } }30:T4de,md` **NOTE**: This section is identical between all notebooks in the generic team. The approach does not impose any restrictions on the data types. There are a variety of data types used in the generic visualization system including multivariate time series, cumulative time series, and matrix. The data comes from different sources such as the SCRC data platform, Public Health Scotland, Our World in Data, and the Office for National Statistics. Some examples include: - [Weekly COVID-19 related deaths in different settings](https://statistics.gov.scot/resource?uri=http%3A%2F%2Fstatistics.gov.scot%2Fdata%2Fdeaths-involving-coronavirus-covid-19) - [Cumulative Number of Cases Tested Positive for COVID-19 in NHS Ayrshire & Arran](https://statistics.gov.scot/slice?dataset=http%3A%2F%2Fstatistics.gov.scot%2Fdata%2Fcoronavirus-covid-19-management-information&http%3A%2F%2Fstatistics.gov.scot%2Fdef%2Fdimension%2Fvariable=http%3A%2F%2Fstatistics.gov.scot%2Fdef%2Fconcept%2Fvariable%2Ftesting-cumulative-people-tested-for-covid-19-positive) - Correlation of ICU patients between different NHS health boards in Scotland (derived from raw data) It is currently being expanded to support visualizations of geographical and network data as well. `31:[] 2d:{"title":"RAMPVIS : Dynamic Data Scheduling Agents for Visualization","cells":[{"id":0,"value":"md`# RAMPVIS : Dynamic Data Scheduling Agents for Visualization`","pinned":false,"hidden":false,"mode":"ojs"},{"id":756,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs"},{"id":360,"value":"---\n## 1. TASK & USERS - User Story","pinned":false,"hidden":false,"mode":"md"},{"id":652,"value":"md`\nThere are different users interested in the dynamic characteristics of the data visualized in the generic visualization system.\n
For example:\n\n - As **a regional healthcare manager**,\n - I would like to see my regional data (new cases, number of ICU patients, etc) every day,\n - so that I can quickly keep myself up-to-date with the current situation, spot anomalies, and trends and do planning appropriately.\n\nAlternatively:\n\n - As **a modeling scientist**,\n - I would like to monitor my model predictions (e.g, number of COVID-related deaths) against real data daily,\n - so that I can understand how accurate and reliable my model is, how it behaves over time.\n\nIn both examples, there is a need to prepare up-to-date data for the systems over time.\nThere is also a need to run computations over the dynamic data such as calculating performance metrics for the models.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":128,"value":"---\n\n## 2. DATA","pinned":false,"hidden":false,"mode":"md"},{"id":663,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"---\n\n## 3. Dynamic Data Scheduling Agents for Visualization","pinned":false,"hidden":false,"mode":"md"},{"id":668,"value":"md`\n### Updating dynamic data for visualization\n\nRheingans (2003) outlined the challenges of visualizing dynamic data. Forbes et al. (2010) described a software framework for enabling the implementation of dynamic data visualization. However, the framework is not publicly available. We used software agents to enable the automatic updating of dynamic data streams as well as the application of analytical algorithms to these data streams. A software agent is a program that runs independently from the main system, receives its own inputs and acts upon them: Jennings (2000). Technically, we implemented several scheduled jobs that run periodically (e.g., daily) to fetch data and compute required analytical measurements. All our agent software is available open-source.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":371,"value":"---\n## 4. ORIGIN & CONTEXT","pinned":false,"hidden":false,"mode":"md"},{"id":673,"value":"md`\nResearch in dynamic data visualization has different angles. Rheingans (2003) focused on making interaction dynamic or updating the visualization as soon as the user interacts with the visual representation (e.g., during mouse movement rather than mouse leave). Others focused on depicting relationships between entities that can change over time such as Cakmak (2021). Fekete (2016) proposed a computation paradigm for analysis of big data and computationally expensive analysis. Turkay (2016) discussed design of a progressive high-dimensional data analysis system. We focused on the dynamic characteristic of the data itself, i.e., ensuring the underlying data input is up to date over time. We use software agents to update data automatically as reviewed in Jennings (2000).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"___\n\n## 5. EXAMPLES","pinned":false,"hidden":false,"mode":"md"},{"id":708,"value":"md `Many COVID-related daily charts maitain their data up to date:\n- https://www.worldometers.info/coronavirus/country/uk/\n- https://ourworldindata.org/covid-vaccinations?country=GBR\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":53,"value":"---\n## 6. DESCRIPTION & USE","pinned":false,"hidden":false,"mode":"md"},{"id":712,"value":"In our approach, a Downloader Agent automatically fetches data from the dynamically changing data sources within the SCRC data infrastructure. This enables the visualizations to display the most current data when they are called.\n\nData used in the generic visualization system comes from different sources including SCRC data platform, Public Health Scotland, Our World in Data, and the Office for National Statistics. Each source is assumed to update its data periodically.\n \nIn order to keep our visualizations to display up-to-date data, we implement a scheduler that are set to run every midnight to fetch data from those sources.","pinned":false,"hidden":false,"mode":"md"},{"id":410,"value":"---\n## 7. REFLECTION","pinned":false,"hidden":false,"mode":"md"},{"id":678,"value":"md`\nOften in VIS, a paper may deal with a single complex dataset. Although the methodology may apply to similar datasets, the software implementation may not. With COVID-19 data visualization, the software must respond to daily data updates automatically. We have not only achieved this automation, but also the automation in applying analytical algorithms to generate derived data for analytical visualization. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":415,"value":"### 7.1 What Worked\nThe upstream data is updated frequently, mainly on a daily basis. During the COVID pandemic, it is essential for the policy makers and the public to have an up-to-date view of the progress of the pandemic. As the number of data streams that needs to be visualised is large and the data also comes from different sources, manually updating them to thousands of visualisation are not feasible with limited resources. Our scheduling agents automate this data download process.\n\nMany visualisations make use of derived data which might be computationally expensive. To reduce the latency of the visualisations, we also implement a scheduling agent to do these computations once new data comes.\n\nIn the early stage of the voluntary project when the data infrastructure wasn’t available to provide data, we also implemented an agent to simulate streaming data to test our visualizations. This allows the visualizations to be implemented in an agile way without completely depending on the data team.","pinned":false,"hidden":false,"mode":"md"},{"id":434,"value":"### 7.2 What Didn't Work\nIt is inevitable that the agents are broken at some time due to programming bugs or server issues. It is important to have a mechanism to manage the agents, providing the ability to check their statuses or start/stop the services. We do not have an intuitive interface to manage these.","pinned":false,"hidden":false,"mode":"md"},{"id":446,"value":"### 7.3 Evidence\nThe visualisations do display the latest data that the data servers provide.","pinned":false,"hidden":false,"mode":"md"},{"id":461,"value":"### 7.4 Recommendations & Opportunities\nAutomation streamlines the notorious and manual processes. Using scheduling agents for repetitive tasks such as downloading data is recommended. Also, we need to have a good overview of the status of all the agents and a clear error logging to make the agents robust and able to respond to the issues quickly","pinned":false,"hidden":false,"mode":"md"},{"id":624,"value":"### 7.5 Who Was Involved?\n","pinned":false,"hidden":false,"mode":"md"},{"id":682,"value":"md`Nguyen, Phong and Khan, Saiful and Turkay, Cagatay and Chen, Min and Reeve, Richard and the SCRC data team`","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"---\n\n## 8. REFERENCES","pinned":false,"hidden":false,"mode":"md"},{"id":688,"value":"md`\nRheingans, P. (2002).
\nAre we there yet? Exploring with dynamic visualization.
\nIEEE Computer Graphics and Applications, vol. 22, no. 1, 6-10.
\nhttps://doi.org/10.1109/38.974511 \n\nForbes, A., Höllerer, T., and George, L. (2010).
\nBehaviorism: A framework for Dynamic Data Visualization.
\nIEEE Transactions on Visualization and Computer Graphics, vol. 16, no. 6, 1164-1171.
\nhttps://doi.org/10.1109/tvcg.2010.126 \n\nCakmak, E., Schlegel, U., Jackle, D., Keim, D., Schreck, T. (2021).
\nMultiscale Snapshots: Visual Analysis of Temporal Summaries in Dynamic Graphs.
\nIEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, 517-527.
\nhttps://doi.org/10.1109/tvcg.2020.3030398 \n\nJennings, N. R. (2000).
On agent-based software engineering.
Artificial Intelligence, vol. 117, no. 2, pp. 277–296, 2000\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":749,"value":"---\n### Imports","pinned":false,"hidden":false,"mode":"md"},{"id":753,"value":"import { ui } from \"@mootari/environment\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}