31:T624,{ // If this page is outside of Observable, set the default styles to approximate // the appearance of the Observable Look and Feel. if (ui === "custom") { return htl.html`

Note: This is an archived version of an Observable Notebook supplement for the RAMPVIS PhilTransA submission titled: 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"
For the live version, see here: https://observablehq.com/@henryqw/represent-error-and-uncertainty

`; } else { return htl.html`

Note: This is a live version of the Observable Notebook supplement for the RAMPVIS PhilTransA submission titled 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"

`; } }32:T4b2,There are many examples of using different visual designs and techniques to represent uncertainty. [Fua et al. (1999)](http://ieeexplore.ieee.org/document/809866/) use a hierarchical parallel coordinates combined with colours and transparency to depict uncertainty in a fatal accident dataset. Extent scaling, a technique to uniformly scale overlapping bands to reveal relative sizes of the extents, is applied to reduce occlusions. See figure below, left (without extent scaling) vs. right (with extent scaling). ${await FileAttachment("fua1999.png").image({width:600})} [Jones (2003)](https://onlinelibrary.wiley.com/doi/10.1002/mrm.10331) uses glyphs with orientations to encode uncertainty from MRI data. In the figure below, a) shows an MRI scan of fiber orientation, b) shows the encoded visualization of the selected region in a, aiming to identify the relationship between uncertainty in fiber orientation and anisotropy, c) shows a zoomed view of a selected region in b, where the uncertainty is high. The radius of the cone encodes uncertainty in the orientation of bundles.

33:T470,**Sedlmair et al. (2014)**: Sedlmair, M., Heinzl, C., Bruckner, S., Piringer, H., & Moller, T. (2014). Visual Parameter Space Analysis: A Conceptual Framework. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2161–2170. https://doi.org/10.1109/TVCG.2014.2346321 **Bonneau et al. (2014)**: Bonneau, G.-P., Hege, H.-C., Johnson, C. R., Oliveira, M. M., Potter, K., Rheingans, P., & Schultz, T. (2014). Overview and State-of-the-Art of Uncertainty Visualization. In C. D. Hansen, M. Chen, C. R. Johnson, A. E. Kaufman, & H. Hagen (Eds.), Scientific Visualization (pp. 3–27). Springer London. https://doi.org/10.1007/978-1-4471-6497-5_1 **Jones (2003)**: Jones, D. K. (2003). Determining and visualizing uncertainty in estimates of fiber orientation from diffusion tensor MRI. Magnetic Resonance in Medicine, 49(1), 7–12. https://doi.org/10.1002/mrm.10331 **Fua et al. (1999)**: Ying-Huey Fua, Ward, M. O., & Rundensteiner, E. A. (1999). Hierarchical parallel coordinates for exploration of large datasets. Proceedings Visualization ’99 (Cat. No.99CB37067), 43–508. https://doi.org/10.1109/VISUAL.1999.809866 34:[{"id":"a6c203c891687e9778d3426a2c6f8495c03ebce4c65cbd91fcc16d561ae19b75d55e7c6f9f73050a16e8d232f316f19b7f970b9fa1f3c3f4eb9dff850fac03eb","href":"https://static.observableusercontent.com/files/a6c203c891687e9778d3426a2c6f8495c03ebce4c65cbd91fcc16d561ae19b75d55e7c6f9f73050a16e8d232f316f19b7f970b9fa1f3c3f4eb9dff850fac03eb","name":"ensemble_line_chart@2.png","type":"image/png","lastModified":1630703936974.339,"size":73217},{"id":"7e618496796003131b74f8707c3538958d614eb33923a1e7388eae52be1693f3aa130eb9848f510b86ea285d07047c98a03f5e661a3cf4192868a5340b696291","href":"https://static.observableusercontent.com/files/7e618496796003131b74f8707c3538958d614eb33923a1e7388eae52be1693f3aa130eb9848f510b86ea285d07047c98a03f5e661a3cf4192868a5340b696291","name":"fua1999.png","type":"image/png","lastModified":1628788570022.448,"size":2332025},{"id":"462bac22864baa56d4db12c41138ec135944b05216aee3dad5f40a8db3978d3fd676f20235c3f50de7d45b82c42c3f31e90413285479e407d10d1904a6cbab32","href":"https://static.observableusercontent.com/files/462bac22864baa56d4db12c41138ec135944b05216aee3dad5f40a8db3978d3fd676f20235c3f50de7d45b82c42c3f31e90413285479e407d10d1904a6cbab32","name":"line_ci.png","type":"image/png","lastModified":1631831752388.06,"size":169812},{"id":"8d481dc1d08d145142ad19631629580c890d463bfdb1406c30f3b4ba14b7a242fe3a4b657c0d488ab8d613de7f2fced3464f62ab59c6e82b62d7277997765fd9","href":"https://static.observableusercontent.com/files/8d481dc1d08d145142ad19631629580c890d463bfdb1406c30f3b4ba14b7a242fe3a4b657c0d488ab8d613de7f2fced3464f62ab59c6e82b62d7277997765fd9","name":"line_2.png","type":"image/png","lastModified":1631831752008.448,"size":47449},{"id":"c2e7e02af32149da17fb553cefb0e2bffca80f8208f11377df07abac22c93f18d7405e56768f4ea60771c1e0e824dac8748659b88f4d3f9f2619df038753409c","href":"https://static.observableusercontent.com/files/c2e7e02af32149da17fb553cefb0e2bffca80f8208f11377df07abac22c93f18d7405e56768f4ea60771c1e0e824dac8748659b88f4d3f9f2619df038753409c","name":"line_3.png","type":"image/png","lastModified":1631831751873.275,"size":56901}] 2f:{"title":"RAMPVIS : Representing error and uncertainty","cells":[{"id":0,"value":"md`# RAMPVIS : Representing error and uncertainty`","pinned":false,"hidden":false,"mode":"ojs"},{"id":822,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":360,"value":"---\n## 1. TASK & USERS - User Story","pinned":false,"hidden":false,"mode":"md"},{"id":334,"value":"* As **a visualization researcher**\n* I would like to \n * assist epidemiological model builders(EMB) in **adjusting models**\n* so that EMBs can\n * **identify error and uncertainty** through interactive comparison of models with **different input, output and configurations**, and adjust the model accordingly\n","pinned":false,"hidden":false,"mode":"md"},{"id":128,"value":"---\n\n## 2. DATA","pinned":false,"hidden":false,"mode":"md"},{"id":346,"value":"Input data:\n\n* 160 different parameter configurations, each includes 16 parameters:\n * https://github.com/ScottishCovidResponse/EnsembleVis-UI/blob/master/public/assets/posterior_parameters.csv\n\nOutput data:\n\n* Average value from 1,000 runs for each configuration:\n * https://github.com/ScottishCovidResponse/EnsembleVis-UI/tree/master/public/assets/data/output","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"---\n\n## 3. Representing error and uncertainty","pinned":false,"hidden":false,"mode":"md"},{"id":43,"value":"We aim to **represent error and uncertainty** from multiple configurations through multiple linked views and interactions.\n\nThere are **aleatoric/statistical uncertainty**, **structural uncertainty** and **prediction uncertainty** from the output ([Sedlmair et al., 2014)](https://doi.org/10.1109/TVCG.2014.2346321).","pinned":false,"hidden":false,"mode":"md"},{"id":371,"value":"---\n## 4. ORIGIN & CONTEXT","pinned":false,"hidden":false,"mode":"md"},{"id":376,"value":"\nVisualization plays an important role in facilitating understanding of the errors and uncertainty that describe key characteristics used to interpret the data ([Bonneau et al., 2014](https://link.springer.com/chapter/10.1007/978-1-4471-6497-5_1)). By representing the errors and uncertainty that arise from the simulation data, eg. noisy data, human error, or incorrect model parameters, visualization illustrates the potential to assist EMBs in producing better models that are natural to the phenomenon.\n\nWe did not start with a specific type of error or uncertainty in mind, as the project progressed, we started to observe abnormalities in the output, which then prompted further investigations. \n","pinned":false,"hidden":false,"mode":"md"},{"id":13,"value":"___\n\n## 5. EXAMPLES","pinned":false,"hidden":false,"mode":"md"},{"id":388,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":53,"value":"---\n## 6. DESCRIPTION & USE","pinned":false,"hidden":false,"mode":"md"},{"id":400,"value":"Errors from the output can be easily observed in the line chart below. **Around day 21, the data appeared to be abnormal.**\n\n${await FileAttachment(\"ensemble_line_chart@2.png\").image({width:600})}\n\nOther abnormal outputs (eg. flat curves) were also observed during our demonstration with the EMBs.\n\n${await FileAttachment(\"line_2.png\").image({width:600})}\n${await FileAttachment(\"line_3.png\").image({width:600})}\n\n\nIn some outputs, uncertainty is encoded with confidence bands.\n${await FileAttachment(\"line_ci.png\").image({width:600})}\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":410,"value":"---\n## 7. REFLECTION","pinned":false,"hidden":false,"mode":"md"},{"id":446,"value":"### 7.0 Reflective Discussion ","pinned":false,"hidden":false,"mode":"md"},{"id":415,"value":"### 7.1 What Worked","pinned":false,"hidden":false,"mode":"md"},{"id":427,"value":"* We have introduced a dashboard-like system to explore the input and output data.\n* Interactions enable filtering and comparison between parameter configurations and simulation output.\n* Multiple linked views facilitate data exploration and insight generation. According to the EMBs, the use of Multiple Linked Views to represent error and uncertainty with interactions, is a first under the epidemiological setting. \n","pinned":false,"hidden":false,"mode":"md"},{"id":434,"value":"### 7.2 What Didn't Work","pinned":false,"hidden":false,"mode":"md"},{"id":438,"value":"* The large dataset size (256,000,000 data points) means we have to preprocess the data and produce a derived dataset, this may affect the accuracy of what we are trying to represent.\n* We did not have enough time to polish our system (UI/UX).\n* We did not have enough opportunities to consult the EMBs, our end users.\n","pinned":false,"hidden":false,"mode":"md"},{"id":461,"value":"### 7.3 Recommendations & Opportunities","pinned":false,"hidden":false,"mode":"md"},{"id":425,"value":"\n* We need to consolidate and optimize our data processing pipeline, in order to improve performance and reduce overhead\n* We are in the process of redesigning our user interface to improve the user experience\n* Support the comparison of multiple selected parameter configurations\n","pinned":false,"hidden":false,"mode":"md"},{"id":624,"value":"### 7.4 Who Was Involved?\n","pinned":false,"hidden":false,"mode":"md"},{"id":628,"value":"Rita Borgo, Hui Fang, Elif Firat, Robert Laramee, Cagatay Turkay, Tom Torsney-Weir, Qiru Wang","pinned":false,"hidden":false,"mode":"md"},{"id":16,"value":"---\n\n## 8. REFERENCES","pinned":false,"hidden":false,"mode":"md"},{"id":489,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":202,"value":"---\n\n### Imports\n","pinned":false,"hidden":false,"mode":"md"},{"id":824,"value":"import { ui } from \"@mootari/environment\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W34","stdlib":"1","titling":"auto"}