2f:T5f8,{ // If this page is outside of Observable, set the default styles to approximate // the appearance of the Observable Look and Feel. if (ui === "custom") { return htl.html`

Note: This is an archived version of an Observable Notebook supplement for the RAMPVIS PhilTransA submission titled: 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"
For the live version, see here: https://observablehq.com/d/7d339207ef90c483

`; } else { return htl.html`

Note: This is a live version of the Observable Notebook supplement for the RAMPVIS PhilTransA submission titled 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"

`; } }30:T5b6,Linked parallel coordinates for parameter space analysis uses two separate parallel coordinates views: one to show the distribution of input parameters and one to show the distribution of output metrics. These two views are linked so that filtering for specific parameter sets in one view shows the corresponding values in the other view - see the [RAMPVIS : Multiple Linked Views for Visual Exploration](https://observablehq.com/@henryqw/multiple-linked-views) notebook. The general goal of parameter space analysis seeks to understand the relationship between input parameters (aka independent variables) and output metrics (aka dependent variables) in a model. Often the goal is to find a specific output value from a simulation (optimization, history matching, etc). In addition, there are often multiple, competing objectives to optimise. Another task of parameter space analysis is understanding this tradeoff. Models do not explicitly have to be traditional simulations. Here the term *model* includes traditional simulations, performance metrics based on adjusting meta-parameters, statistical regression models, etc. Linked parallel coordinates seeks to accomplish the goals of parameter space analysis through interactive selection filters. Users can adjust their selection of the input parameters to see which output metrics are produced or filter in the output metrics to understand what input combinations will possibly lead to the desired effect.31:T696,(Visual) parameter space analysis is closely related to the concept of response surface methodology which dates back to [Box and Wilson (1951)](https://doi.org/10.1111/j.2517-6161.1951.tb00067.x). This methodology focused on exploring the relationship between input and output parameters. Visual parameter space analysis uses interactive visualization (rather than statistical measures) as that primary mode of explaining a complex model to a user. This was later extended to incorporate the idea of emulators as a way to quickly estimate results from a computer simulation ([Sacks et al., 1989](https://doi.org/10.1214/ss/1177012413)). The original concept of emulators was designed around experiments typically run on super computers. Each run of a simulation is expensive in terms of money and time but statistical analysis of simulation runs requires many samples. The idea of emulators was introduced to address this issue. An emulator, essentially a regression model, would be built on a small number of carefully chosen samples (e.g. space-filling design) of the simulation. Statistical analysis is then performed using the emulator as a stand-in for the full simulation model. If needed, additional samples can be identified and run in the complex simulation. [Sedlmair et al. (2014)](https://doi.org/10.1109/TVCG.2014.2346321) organized existing solutions and developed a taxonomy for visual parameter space analysis. Visual parameter space analysis has a number of challenges: * multi-dimensional input space * multiple outputs from the simulation * complex (non-linear), and often slow-running, simulation model * representing multi- and high-dimensional spaces on screen32:T510, An early example of this in the visualization community is from [Matkovic et al. (2008)](https://doi.org/10.1109/tvcg.2008.145). This work used the idea of *coordinated multiple views* between separate plots of the input and output dimensions. However, these used different visual encodings for the input and output space (see [Berger and Piringer, 2010](https://doi.org/10.1109/VAST.2010.5651694); [Matkovic et al., 2009](https://doi.org/10.1109/TVCG.2009.155); [Zhao et al., 2017](https://doi.org/10.1109/tvcg.2017.2744738) ) or only viewed one space at a time (see, e.g., [Konyha et al., 2006](https://doi.org/10.1109/TVCG.2006.99); [Demir and Westermann, 2013](https://doi.org/10.1111/cgf.12089); [Torsney-Weir, Sedlmair & Moeller, 2017](https://doi.org/10.1111/cgf.13177); [Torsney-Weir et al., 2018](https://doi.org/10.1111/cgf.13415) ). Using the same visual encoding should, theoretically, ease the training process due to the use of small multiples [Munzner (2014)](https://www.routledge.com/Visualization-Analysis-and-Design/Munzner/p/book/9781466508910). This was employed by [Berger et al. (2011)](https://doi.org/10.1111/j.1467-8659.2011.01940.x) who used two sets of parallel coordinates: one for the input space and one for the output space (panels b and f in the image below).33:T6c4,In terms of the design process, joining in to the weekly meetings of an Uncertainty Quantification (UQ) group within the SCRC consortium that consists of a mix of modellers and uncertainty quantification researchers, has been instrumental in developing an understanding of what is needed within the applied context and in informing our choice of using linked parallel coordinates for visual parameter space exploration. Within the initial weekly meetings, we mostly listened silently in the background -- which we are referring to as "lurking" in meetings in another Notebook -- to get a feel for the domain and problem specification before recommending any solutions. In order to build trust and a mutual understanding and to avoid any misunderstandings, we explained that visualisation researchers will remain silent for a while and only listen. The other UQ group members referred to some of us as "visualisation spies" in a playful way to recognise that we are joining the calls but mostly remaining silent and observing. This silent phase transformed into a more engaged and active participation from the visualisation researchers as we build a better understanding and develop a number of ideas joining the calls but mostly remaining silent and observing. This silent phase transformed into a more engaged and active participation from the visualisation researchers as we build a better understanding and develop a number of ideas and prototypes. The learnings from the silent listening phase enabled us to introduce our designs in a much more informed manner by using a vocabulary that will resonate with the rest of the team, underpinning a healthier dialogue within the collaboration. 34:T55f,Knowledge of the visual parameter space framework helped greatly in terms of understanding which mid-level tasks would be relevant to the high-level task of "understanding the model" (see [pos #1](https://observablehq.com/d/7d339207ef90c483#cell-695)). It encouraged the identification of input (independent) and output (dependent) parameters. In addition, it provided some common terminology and understanding of available statistical techniques. Some good examples are "space filling design" for sampling and the use of "emulators" which is a model that can be quickly queried to get an estimate of the more complex model. The advantage of using linked parallel coordinates views is that software and knowledge to create and link the views is readily available as opposed to more novel techniques, like slices ([Torsney-Weir, Sedlmair, Moeller](https://doi.org/10.1111/cgf.13177)), which are much less common and not part of the collective intelligence of the visualization community. Thus shortening developer and tutorial time. Different users may prefer purely visual/interactive or purely statistical techniques. A mismatch of user and technique can greatly affect user engagement (see [neg #1](https://observablehq.com/d/7d339207ef90c483#cell-684)). In addition, these techniques (statistical vs visual) seemed to be viewed as separate and not integral in our case.35:Tc28,**Box, G.E.P. and Wilson, K.B. (1951)**.
*On the Experimental Attainment of Optimum Conditions*.
Journal of the Royal Statistical Society: Series B (Methodological), 13, 1-38
https://doi.org/10.1111/j.2517-6161.1951.tb00067.x **Sacks, J., Welch, W.J., Mitchell, T.J., & Wynn H.P. (1989)**.
*Design and Analysis of Computer Experiments*.
Statistical Science, 4(4), 409-423.
https://doi.org/10.1214/ss/1177012413 **Konyha, Z., Matkovic, K., Gracanin, D., Jelovic, M. & Hauser, H. (2006)**.
*Interactive Visual Analysis of Families of Function Graphs*.
IEEE Transactions on Visualization and Computer Graphics, 12(6), 1373–1385.
https://doi.org/10.1109/TVCG.2006.99 **Matkovic, K., Gracanin, D., Jelovic, M., & Hauser H. (2008)**.
*Interactive Visual Steering - Rapid Visual Prototyping of a Common Rail Injection System*.
IEEE Transactions on Visualization and Computer Graphics, 14(6), 1699-1706.
https://doi.org/10.1109/TVCG.2008.145 **Matkovic, K., Gracanin, D., Klarin, B., & Hauser, H. (2009)**.
*Interactive Visual Analysis of Complex Scientific Data as Families of Data Surfaces*.
IEEE Transactions on Visualization and Computer Graphics, 15(6), 1351–1358.
https://doi.org/10.1109/tvcg.2009.155 **Berger, W. and Piringer, H. (2010)**.
*Interactive Visual Analysis of Multiobjective Optimizations*.
In 2010 IEEE Symposium on Visual Analytics Science and Technology.
https://doi.org/10.1109/vast.2010.5651694. **Berger, W., Piringer, H., Filzmoser, P., & Gröller, E. (2011)**.
*Uncertainty-Aware Exploration of Continuous Parameter Spaces Using Multivariate Prediction*.
Computer Graphics Forum, 30(3) 911–920.
https://doi.org/10.1111/j.1467-8659.2011.01940.x **Demir, I. & Westermann, R. (2013)**.
*Progressive High-Quality Response Surfaces for Visually Guided Sensitivity Analysis*.
Computer Graphics Forum, 32(3), 21–30.
https://doi.org/10.1111/cgf.12089 **Sedlmair, M., Heinzl, C., Bruckner, S., Piringer, H., Möller, T. (2014)**.
*Visual Parameter Space Analysis: A Conceptual Framework*.
IEEE Transactions on Visualization and Computer Graphics, 20(12), 2161-2170.
https://doi.org/10.1109/TVCG.2014.2346321 **Munzner, T. (2014)**.
*Visualization Analysis and Design*.
A K Peters/CRC Press. **Torsney-Weir, T., Sedlmair, M., Möller, T. (2017)**.
*Sliceplorer: 1D slices for multi-dimensional continuous functions*.
Computer Graphics Forum, 36(3), 167-177.
https://doi.org/10.1111/cgf.13177 **Zhao, X., Wu, Y., Cui, W., Du, X., Chen, Y., Wang, Y., Lee, D.L., & Qu, H. (2017)**.
*SkyLens: Visual Analysis of Skyline on Multi-Dimensional Data*.
IEEE Transactions on Visualization and Computer Graphics, 24(1), 246-255.
https://doi.org/10.1109/tvcg.2017.2744738 **Torsney-Weir, T., Möller, T., Sedlmair, M., & Kirby, R.M. (2018)**.
*Hypersliceplorer: Interactive Visualization of Shapes in Multiple Dimensions*.
Computer Graphics Forum, 37(3), 229-240.
https://doi.org/10.1111/cgf.13415 36:[{"id":"eb7f45c3c12b5d7c19537c07093cfab8f72b97a343a9bef5c3ffe9404706094b7bb2de39e2dfcbc6a667c75cc528dcae1a4ab4591a802375b28c64c715ada24a","href":"https://static.observableusercontent.com/files/eb7f45c3c12b5d7c19537c07093cfab8f72b97a343a9bef5c3ffe9404706094b7bb2de39e2dfcbc6a667c75cc528dcae1a4ab4591a802375b28c64c715ada24a","name":"posterior_parameters.csv","type":"text/csv","lastModified":1630674361925.247,"size":28640},{"id":"47f4c33a7f8e3ffab8e9eb535c2044c900184ea2c78083754dc18970cf27c38bacf0c828ba3ce0a780e570a1e75e5b2197051880db6d432d716d73472ad03420","href":"https://static.observableusercontent.com/files/47f4c33a7f8e3ffab8e9eb535c2044c900184ea2c78083754dc18970cf27c38bacf0c828ba3ce0a780e570a1e75e5b2197051880db6d432d716d73472ad03420","name":"overview.png","type":"image/png","lastModified":1630675771919.828,"size":900949},{"id":"4c12689a2b637fa9c48485fe4c3fe800bbc75bcf4c31e7d8888286dd2100dca0f39abb11fbc4d1ebc28cac95dbd5a5cbbf71034717c0f065dd304e463338f203","href":"https://static.observableusercontent.com/files/4c12689a2b637fa9c48485fe4c3fe800bbc75bcf4c31e7d8888286dd2100dca0f39abb11fbc4d1ebc28cac95dbd5a5cbbf71034717c0f065dd304e463338f203","name":"multi-pcp.png","type":"image/png","lastModified":1631615259948.74,"size":277928},{"id":"9ca7e7e23cd9e91b6e57177a6c9832232e992721b040e21173776787e3a0eec41c6c8a1d0fd72aceaa75bceb2830b6885afd73e1e384dfad593ccda419baead1","href":"https://static.observableusercontent.com/files/9ca7e7e23cd9e91b6e57177a6c9832232e992721b040e21173776787e3a0eec41c6c8a1d0fd72aceaa75bceb2830b6885afd73e1e384dfad593ccda419baead1","name":"optim.png","type":"image/png","lastModified":1631622134219.54,"size":169306},{"id":"f9c83c73f20e8d094be1e8e1a4126b2afadd25d954c2f147029daf6468d2bc1dd432be574200a194e14a7708398771db5ee6492af9fcaf264d9f4554012cec13","href":"https://static.observableusercontent.com/files/f9c83c73f20e8d094be1e8e1a4126b2afadd25d954c2f147029daf6468d2bc1dd432be574200a194e14a7708398771db5ee6492af9fcaf264d9f4554012cec13","name":"nimbus.png","type":"image/png","lastModified":1631627904356.283,"size":345157},{"id":"8dfe98f6334119ebb63fc626de0f464677af8ac6b0c85276303a50e616c975bc1e812c066f49c3546bcfb5cdeb35ab21edcd4a603975b97a7ecfe20ad69255cb","href":"https://static.observableusercontent.com/files/8dfe98f6334119ebb63fc626de0f464677af8ac6b0c85276303a50e616c975bc1e812c066f49c3546bcfb5cdeb35ab21edcd4a603975b97a7ecfe20ad69255cb","name":"output_prediction_full_11-09-2020_18-57-41.short.txt","type":"text/plain","lastModified":1631628732350.643,"size":41942998}] 2d:{"title":"RAMPVIS : Linked Parallel Coordinates for Visual Parameter Space Exploration","cells":[{"id":0,"value":"# RAMPVIS : Linked Parallel Coordinates for Visual Parameter Space Exploration","pinned":false,"hidden":false,"mode":"md"},{"id":986,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs"},{"id":360,"value":"---\n## 1. TASK & USERS - User Story","pinned":false,"hidden":false,"mode":"md"},{"id":676,"value":"md`* As **an epidemiological model builder**\n* I would like to:\n * understand which parameters have the most/least effect on model outcomes, esp hospitalizations and deaths\n * understand which parameters have different effects on different outcomes (e.g. which parameters are more important for hospitalizations vs deaths)\n * how much of an effect does the stochasticity aspect produce\n * what inputs will fit the model to observed data?\n * Which inputs lead to certain outcomes (e.g. hospitalizations over x)\n* So that I can *(3 different user stories)*:\n 1. Understand which parameters are the most important to adjust to try and match observed data\n 2. Understand which parts of the model are responsible for which output metrics\n 3. Understand which parameters (and ranges thereof) can I change to affect a desired change\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":128,"value":"---\n\n## 2. DATA","pinned":false,"hidden":false,"mode":"md"},{"id":664,"value":"md `Data is technically the model itself, which can be seen as a function of input parameters to output metrics. The model is a compartmental model. The representation of the model that we have right now is a table of samples. There is one sample for each:\n* day (200 days)\n* initial parameter configuration (~130)\n* stochastic variation (~100)\n\nThe **inputs** are 14 scalar parameters controlling, among other aspects, the initial state of the various compartments in the model.\n\nThe **outputs** are the final states of the various compartments, e.g. number in hospital or infected.\n\nOutputs can be aggregated over the days and stochastic variation to essentially produce a ${tex `R^n \\rightarrow R^m`} mapping.\n`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":815,"value":"## File layout\n\nInput parameters are stored in a single file called `posterior_parameters.csv`. Each parameter run is stored in a separate directory called `output_row_n/output_prediction_full_.txt`, where n refers to the corresponding row in the input parameter file. Both files are CSV files. Thus the file layout is:\n\n```\n├── posterior_parameters.csv\n├── output_row_0\n│ ├── output_prediction_full_11-09-2020_18-57-41.txt\n│ ├── output_prediction_simu_11-09-2020_18-57-41.txt\n│ └── run_11-09-2020_18-57-41.log\n├── output_row_1\n│ ├── output_prediction_full_11-09-2020_18-58-05.txt\n│ ├── output_prediction_simu_11-09-2020_18-58-05.txt\n│ └── run_11-09-2020_18-58-05.log\n├── output_row_2\n│ ├── output_prediction_full_11-09-2020_19-01-40.txt\n│ ├── output_prediction_simu_11-09-2020_19-01-40.txt\n│ └── run_11-09-2020_19-01-40.log\n...\n```","pinned":false,"hidden":false,"mode":"md"},{"id":823,"value":"## Input file","pinned":false,"hidden":false,"mode":"md"},{"id":831,"value":"inputs = FileAttachment(\"posterior_parameters.csv\").csv({typed: true})","pinned":false,"hidden":false,"mode":"ojs"},{"id":855,"value":"Inputs.table(inputs)","pinned":false,"hidden":false,"mode":"ojs"},{"id":837,"value":"## Output file\n\nThere is one of these per row (run) in the input parameters file","pinned":false,"hidden":false,"mode":"md"},{"id":840,"value":"output = FileAttachment(\"output_prediction_full_11-09-2020_18-57-41.short.txt\").csv({typed:true})\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":860,"value":"Inputs.table(output)","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"---\n\n## 3. Linked parallel coordinates for visual parameter space exploration","pinned":false,"hidden":false,"mode":"md"},{"id":680,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":371,"value":"---\n## 4. ORIGIN & CONTEXT","pinned":false,"hidden":false,"mode":"md"},{"id":719,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":932,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":999,"value":"await FileAttachment(\"multi-pcp.png\").image({width:800});","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"___\n\n## 5. EXAMPLES","pinned":false,"hidden":false,"mode":"md"},{"id":383,"value":"**[Cajot et al. (2019)](https://doi.org/10.3389/fict.2018.00032)** use linked parallel coordinates to understand how adjusting the range of inputs would influence possible solutions. In this case, the selection is only possible in one direction, from selecting input ranges to viewing possible outputs.","pinned":false,"hidden":false,"mode":"md"},{"id":1005,"value":"await FileAttachment(\"optim.png\").image({width:600});","pinned":false,"hidden":false,"mode":"ojs"},{"id":952,"value":"**www-nimbus** uses parallel coordinates to communicate possible solutions in a multi-objective problem (as is done in the VPSA case as well) but only looks at the output space.\n\ndemo: https://wwwnimbus.it.jyu.fi/\n\npaper: [Miettinen and Maekelae (2000)](https://doi.org/10.1016/S0305-0548(99%2900115-X)","pinned":false,"hidden":false,"mode":"md"},{"id":1009,"value":"await FileAttachment(\"nimbus.png\").image({width:600})","pinned":false,"hidden":false,"mode":"ojs"},{"id":53,"value":"---\n## 6. DESCRIPTION & USE","pinned":false,"hidden":false,"mode":"md"},{"id":867,"value":"The linked parallel coordinates view is shown here. The top PCP shows input parameter combinations while the bottom one shows output parameter configurations. To **show which outputs correspond to which inputs**, modellers would select particular combinations of input parameters in the top and view which lines are highlighted in the lower plot. ","pinned":false,"hidden":false,"mode":"md"},{"id":1013,"value":"await FileAttachment(\"overview.png\").image({width:800})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1016,"value":"Correspondingly, the modeller can filter in the lower, output parameter plot and see which combinations of input parameters will lead to the selected set of outputs. This helps them to understand **which parts of the model and which input paramter combinations will lead to a desired outcome**.\n\nBy filtering in on a particular set of output runs, the modeller could understand, for each input parameter, what are the posible ranges that could lead to that outcome. Parameters with a narrow range mean that the output is very sensitive to those parameters. Thus, to **affect a desired outcome** these particular parameters need to be adjusted.","pinned":false,"hidden":false,"mode":"md"},{"id":410,"value":"---\n## 7. REFLECTION","pinned":false,"hidden":false,"mode":"md"},{"id":446,"value":"### 7.0 Reflective Discussion ","pinned":false,"hidden":false,"mode":"md"},{"id":686,"value":"\nVisual parameter space analysis requires the development of multiple, integrated, linked views of a dataset. Datasets may be large and thus the filtering and other interactions must be performant. In addition, visualization techniques are ideally augmented by statistical modelling such as emulation, sensitivity analysis, correlation, and dimension reduction. Effective combinations of these techniques are user- and model-specific. Because of this, the loop from idea to development to feedback back to new ideas is very long and requires extensive development time (see [neg #3](https://observablehq.com/d/7d339207ef90c483#cell-684)). ","pinned":false,"hidden":false,"mode":"md"},{"id":966,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":757,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":415,"value":"### 7.1 What Worked","pinned":false,"hidden":false,"mode":"md"},{"id":695,"value":"1. Visual parameter space analysis was a good reference for mid-level tasks and terminology\n2. Parallel coordinates and interactive brushing and linking (specific implementation of VPSA) are commonly implemented in the visualization community, thus required little instruction in terms of implementation\n3. Interactive linking allows a single tool to be used for different tasks (multi-objective optimization, sensitivity analysis, finding inputs that generate a particular output)\n\nWith a single set of 2 plots we could address the main 3 user stories above.","pinned":false,"hidden":false,"mode":"md"},{"id":434,"value":"### 7.2 What Didn't Work","pinned":false,"hidden":false,"mode":"md"},{"id":684,"value":"1. User engagement\n2. Effective layout and coordination of multiple plots is not simple to define\n3. Development time made it difficult to develop alternative prototypes and incorporate statistical techniques\n4. Workflow of the tool was not clear. Using a more general interactive technique meant it was not obvious how to, e.g. perform sensitivity analysis, using the tool.
*Yes, this is basically the opposite of what I put for positive #3 above but this the tradeoff between general and specific*","pinned":false,"hidden":false,"mode":"md"},{"id":461,"value":"### 7.3 Recommendations & Opportunities","pinned":false,"hidden":false,"mode":"md"},{"id":713,"value":"The biggest problem with techniques like these is that they are great for initially defining the project, finding what's important, etc but they lack specific guidance about what visualization techniques and interactions will be most effective for which mid-level tasks. \n\n#### Future developments:\n* more linking of tasks to techniques\n* studying how interaction can support analysis tasks (i.e. not just visualization techniques)\n* rapid experimentation with novel interaction and visualization methods (e.g. slicing)\n\nExamples of these tools are often complex and rarely available online which makes dissemination difficult","pinned":false,"hidden":false,"mode":"md"},{"id":624,"value":"### 7.4 Who Was Involved?\n","pinned":false,"hidden":false,"mode":"md"},{"id":691,"value":"Tom Torsney-Weir, Rita Borgo, Cagatay Turkay, Hui Fang, Qiru Wang, Elif Firat, Robert Laramee","pinned":false,"hidden":false,"mode":"md"},{"id":16,"value":"---\n\n## 8. REFERENCES","pinned":false,"hidden":false,"mode":"md"},{"id":901,"value":"$35","pinned":false,"hidden":false,"mode":"md"},{"id":1020,"value":"---\n\n### Imports","pinned":false,"hidden":false,"mode":"md"},{"id":982,"value":"import {ui} from \"@mootari/environment\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}