2b:null 2c:["$","title",null,{"children":"RAMPVIS : Integrated Algorithmic Tools for Visual Analytics | lborohfang | Observable"}] 31:T660,{ // If this page is outside of Observable, set the default styles to approximate // the appearance of the Observable Look and Feel. if (ui === "custom") { return htl.html`

Note: This is an archived version of an Observable Notebook supplement for the RAMPVIS PhilTransA submission titled: 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"
For the live version, see here: https://observablehq.com/@lborohfang/rampvis-idiom-integrated-algorithmic-tools-for-visual-ana

`; } else { return htl.html`

Note: This is a live version of the Observable Notebook supplement for the RAMPVIS PhilTransA submission titled 
"Visualization for Epidemiological Modelling: Challenges, Solutions, Reflections & Recommendations"

`; } }32:T5b0,The key focus of a successful visual analytics (VA) system is to understand the relationship between input and output spaces. For a VA system to analyse modern state-of-the-art machine learning algorithms [1,2,3], e.g., convolutional neural networks, this insight improves the interpretability of the highly non-linear mapping between feature space and annotated labels. Whilst for a VA system to investigate agent-based modelling (ABM) simulators [4], this information facilitates better configurations of important causal factors to generate realistic simulations.
Despite a well-motivated task, designing an elegant visual analytics system is challenging because both input and output spaces are high-dimensional and there are too many interesting patterns for visualization so that it is a hard decision how to allocate powerful visual channels, e.g., colours and shapes, to various attributes for users to gain insightful knowledge of the models. In this exploration, we aim to integrate many classical machine learning algorithms, including unsupervised clustering algorithms, dimensional reduction methods and regression models, into a visual analytics system to serve the purpose of understanding an epidemiology simulation model, EERA [5]. Both the output uncertainty for each corresponding configuration set and sensitivity caused by each configuration parameter can be analysed well with the help of various algorithmic tools.
33:T471,As we have 160,000 output simulated time-series records by using 160 input configuration sets, it is impossible to colour-code the configuration sets to find out the relationship between the input and output spaces. To facilitate the VA on both uncertainty and sensitivity analysis, we propose to use an unsupervised clustering algorithm - K-means method to group the 160,000 time-series records first. After using the elbow method to select an optimal cluster number, which is set to six in our experiments, we assign a colour map to represent these clusters to understand the connections between the 160 configuration sets as well as the variations in each configuration set. In addition, to deal with the high dimensionality of each time-series record, we investigate a dimensional reduction method - Principal Component Analysis, to project the 199 dimensions into 2-D space. As the first two coefficients preserve ~90% variations, we believe the 2-D feature representatively reflects its original time-series signal. After this, we create 10X16 grid-based scatter plots to visualize the distributions of samples in the output space.34:T539,With the visualisation tool by using clustering and dimensional reduction methods, it could further help us to identify different patterns the simulator behaves with each configuration set. For example, the subplots [3,3] (i.e. [column index, row index]) [3,10] [5,2] have mixed colours. These show that in the high dimensional simulated output space, these three configuration sets generated samples which are represented by several clusters and the simulated patterns are more complicated than other configuration sets. While the subplots [7,6] [2,3] [12,10] [4,6] have relatively wider distributions compared to other sets but with the same clustering colour. It means that in the high dimensional space, the simulated data can be clustered well but there exist larger variations in the simulation compared to other configurations. On contrary, subplots [4,1] [15,3] shows linear patterns. It indicates that the first and the second largest components of the PCA have a strong correlation in the simulated data in these configuration sets. In addition, we also investigated the use of various clustering algorithms, e.g., DBSCAN and agglomeration clustering, and dimensional reduction methods, including TSNE and UMAP, to analyze the distributions, uncertainty, and how sensitivity of the output related to the input parameter sets. 35:T4bc, Integration of various machine learning-based algorithmic tools helps to build a powerful visual analytics system. In this approach, we practice the use of classical unsupervised learning methods, including clustering, dimensional reduction and regression methods, to tackle the high-dimensional data challenge in both input and output spaces and facilitate a powerful VA system to investigate the uncertainty and sensitivity of an epidemiological model. Specifically, the unsupervised clustering algorithm could help us to find similarities and variations of the simulated data by using different configuration settings so that we could further evaluate the sensitivity of the causal factors in the input space. Further, the dimensional reduction technique allows us to visualize the high-dimensional data for analysis. In RAMPVIS consortium, we believe there are other developments that could be categorized into this approach as the integration of algorithmic tools is a common practice in VIS community although they do not explicitly describe their designs in such a manner. While our approach could potentially provide a data processing pipeline for other VA platforms to standardise the process stage.36:T56a,In the current version of the developed system, we have not fully explored the algorithmic tools in an interactive manner although some patterns could be clearly spotted. Currently, we have not conducted a comprehensive study to select the most suitable tools for the task. We will investigate the possible bias which may be incurred due to the selection of different tools. At the current stage, it is not a tight integration of algorithmic tools for the VA system. However, it has been proved to be a useful tool for prompt analysis even with this loose integration system. This is our next stage once these tools are properly integrated to the RAMP-VIS platform and it will work better with elegant interactions. There are several reasons for the partial implementation: (1) there are many different combinations of methods that we could use. We have implemented several classical methods, for example, K-mean and DBScan for clustering, PCA, TSNE and UMAP for dimensional reduction, and regression models for emulators. While it will take more time to integrate them into the platform. (2) we have 8 ageing groups as well as simulated curves with various attributes, e.g., hospitalisation, exposures, infections, ICU usage and death toll etc. A more developed user interaction process is needed so that the users will not lose their focus when they have so many options to select. 37:T8ed,[1] Tam, Kwok-Leung, Fang, H., Aubrey, Andrew, Grant, P. W., Rosin, Paul L., Marshall, Daniel and Chen, M. (2011). Visualization of time-series data in parameter space for understanding facial dynamics. Computer Graphics Forum 30 (3), pp. 901-910.
https://doi.org/10.1111/j.1467-8659.2011.01939.x [2] T. Spinner, U. Schlegel, H. Schäfer and M. El-Assady (2020). "explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning," in IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, pp. 1064-1074, Jan.
https://doi.org/10.1109/TVCG.2019.2934629. [3] G. K. L. Tam, V. Kothari and M. Chen, (2017). "An Analysis of Machine- and Human-Analytics in Classification," in IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 1, pp. 71-80, Jan. 2017,
https://doi.org/10.1109/TVCG.2016.2598829. [4] Slingsby, A. , Hyde, J. and Turkay, C. (2019). Visual Analysis of Reactionary Train Delay from an Agent-Based Model. Poster presented at the EuroVis 2019, 3-7 Jun 2019, Porto, Portugal.
https://doi.org/10.2312/eurp.20191153 [5] Covid-19 EERA model (2021).
https://scottishcovidresponse.github.io/Covid19_EERAModel/site/model_documentation.html. [6] Wenskovitch, John, et al. (2017). "Towards a systematic combination of dimension reduction and clustering in visual analytics." IEEE transactions on visualization and computer graphics 24.1: 131-141.
https://doi.org/10.1109/TVCG.2017.2745258. [7] Packer, Eli, et al. (2013). "Visual analytics for spatial clustering: Using a heuristic approach for guided exploration." IEEE Transactions on Visualization and Computer Graphics 19.12 2179-2188.
https://doi.org/10.1109/TVCG.2013.224 [8] Ali, Mohammed, et al. (2019). "Clustering and classification for time series data in visual analytics: A survey." IEEE Access 7: 181314-181338.
https://doi.org/10.1109/ACCESS.2019.2958551. [9] Gibson, F., Fabbro, R., Rahat, A., Torsney-Weir, T., Archambault, D., Gravenor, M. and Lucini, B., (2021). July. An interactive tool for enhancing hospital capacity predictions using an epidemiological model. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 55-56).
https://doi.org/10.1145/3449726.3462738.38:[{"id":"55d4b5d7b9b9c8bba6dd652113bef54633a1274c517219cfeafb4f7c09f35a30bcaf4fd71ff1f1350c168452f349d85304e0e723d6a594551b5c7a34be247a57","href":"https://static.observableusercontent.com/files/55d4b5d7b9b9c8bba6dd652113bef54633a1274c517219cfeafb4f7c09f35a30bcaf4fd71ff1f1350c168452f349d85304e0e723d6a594551b5c7a34be247a57","name":"tool (2).png","type":"image/png","lastModified":1630673550180.84,"size":1065552},{"id":"2e0ce81cee8b6402e851c15a6d00e140192d5f359a5bbdec324181ca8dca4ba2f2037ab93df6f3e97cdbdee5cc4924bd3e6f691302eb142161cd15daa76f4393","href":"https://static.observableusercontent.com/files/2e0ce81cee8b6402e851c15a6d00e140192d5f359a5bbdec324181ca8dca4ba2f2037ab93df6f3e97cdbdee5cc4924bd3e6f691302eb142161cd15daa76f4393","name":"image@1.png","type":"image/png","lastModified":1629155936638.732,"size":552347},{"id":"3567fe6a605656acdc14eaefb7cd335310c6dc9a069596a0437b6a4d487787026f25d79f4f75cd9c4bbce902ec396f4a35537cb03b08756cfa436cec22d14601","href":"https://static.observableusercontent.com/files/3567fe6a605656acdc14eaefb7cd335310c6dc9a069596a0437b6a4d487787026f25d79f4f75cd9c4bbce902ec396f4a35537cb03b08756cfa436cec22d14601","name":"image@2.png","type":"image/png","lastModified":1630781216893.759,"size":349898},{"id":"ec7aef8175f1bb4e4fe06d2f193c4465aae6221d796b03ff746c70206a1b4eec408500ed7b4e3d702e6b02c00037b888bde1f03093f1cc4fea568367accb0bc6","href":"https://static.observableusercontent.com/files/ec7aef8175f1bb4e4fe06d2f193c4465aae6221d796b03ff746c70206a1b4eec408500ed7b4e3d702e6b02c00037b888bde1f03093f1cc4fea568367accb0bc6","name":"image@5.png","type":"image/png","lastModified":1630783879843.992,"size":9336}] 2f:{"id":"43358599cbcc009d","slug":"rampvis-idiom-integrated-algorithmic-tools-for-visual-ana","title":"RAMPVIS : Integrated Algorithmic Tools for Visual Analytics","thumbnail":"102421e0f1accd6f1bb75b916997374e5b184a819b42cbff815c281d53975fdc","access_level":"public","create_time":"$D2021-08-16T17:33:24.659Z","update_time":"$D2022-02-23T11:24:33.202Z","view_time":null,"first_public_time":"$D2021-08-17T00:09:23.501Z","has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"b641acc8c3622ee9","login":"lborohfang","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c46ee6e6aa5472e68f8b9b02e644bcb97e90a873b88b2cf4374ef95fc3b614b8","tier":"starter_2024"},"version":968,"publish_version":968,"role":null,"authored":false,"authors":[{"type":"individual","id":"1ec75da286f18a47","login":"lborohfang","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c46ee6e6aa5472e68f8b9b02e644bcb97e90a873b88b2cf4374ef95fc3b614b8","tier":null}],"members":[{"type":"individual","id":"7059effe79eeddde","login":"almarahat","name":"Alma Rahat","avatar_url":"https://avatars-next.observableusercontent.com/5bf5739d77f1295b7c9c67950dc2dd9a5581701d2840874ce949b98b3792fda9","tier":null,"role":"editor"},{"type":"individual","id":"c589264854a252b0","login":"effepivi","name":"","avatar_url":"https://avatars.githubusercontent.com/u/22597124?v=4","tier":null,"role":"editor"},{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null,"role":"editor"},{"type":"individual","id":"1ec75da286f18a47","login":"lborohfang","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c46ee6e6aa5472e68f8b9b02e644bcb97e90a873b88b2cf4374ef95fc3b614b8","tier":null,"role":"editor"}],"comments":[{"id":"9fdc9f76c21826a5","reply_to_id":null,"content":"Clear, but can you frame this as a User Story please?\nIs this adequate?\nPlease work on this!\n\nAs an epidemiological modeller simulating the spread of Covid\nI would like to explore the relationships between factors in agent-based models and simulated time-series\nso that I can select and optimize models to match real World observations","create_time":"$D2021-08-25T11:19:33.590Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"a6d7cab95c21fe98","reply_to_id":null,"content":"OK - please add direct links to the data if you can.","create_time":"$D2021-08-25T11:20:30.566Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"b95909487dde14af","reply_to_id":"a6d7cab95c21fe98","content":"Can you explain what the figure is trying to show and how? Colours?","create_time":"$D2021-08-25T11:20:50.927Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"68dad631f42d3273","reply_to_id":null,"content":"I hope you can find some examples of this approach being used elsewhere?","create_time":"$D2021-08-25T11:21:30.120Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"cd515835e912e341","reply_to_id":null,"content":"OK, so you are trying to find a combination of algorithms, visual designs and interactions that address the User Story. You don't really say which existing solution you are using, but useful that you link to candidate methods that you can incorporate. Fine!","create_time":"$D2021-08-25T11:24:52.328Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"ff719c6c14f8c2a8","reply_to_id":null,"content":"Slingsby.\n\nCan you make reference to a few more solutions that informed your design and solution?\nThis would be a great way to connect the scientific community to the VIS literature. See below also! Which published work, including your own, influenced the design?","create_time":"$D2021-08-25T11:26:29.062Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"c4ed328edef7af87","reply_to_id":null,"content":"The picture is pretty, but I don't know what it shows from the description. Can you be more explicit and specific.\n\nDo the 6 classes have names? \nHow are the 160 (10x16) output spaces ordered?\nWhy not use partially opaque symbols so that we can see density?\nWhat does a blue outlying dot in a red cluster (very salient) mean?","create_time":"$D2021-08-25T11:31:10.701Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"00289e2544d8d13a","reply_to_id":"c4ed328edef7af87","content":"There is a bit of an inconsistency here in that section 6 does not describe the Integrated Algorithmic Tools for Visual Analytics. There is no tool described here (is there?). It seems to describe a visualization of a clustering algorithm in output space. So, this 'description and use' section really addresses the user story ...\n\nAs an epidemiological modeller simulating the spread of Covid\nI would like to see how sampled model input parameters map to output spaces\nso that I can understand the relationships between factors in agent-based models and simulated time-series\n\nI can't really figure out what the output spaces are from the description - help! 😬😉","create_time":"$D2021-08-25T11:37:31.568Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"51bdf31e3cf674b7","reply_to_id":null,"content":"\"more insightful analysis about the relationship between input and output spaces is achieved. \"\nSo - can you tell us what you found in Section 6?","create_time":"$D2021-08-25T11:39:02.985Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"d27049e3d00bd29a","reply_to_id":"51bdf31e3cf674b7","content":"\"It is easily to identify which parameter settings generate more variations in its simulation and which settings generate more similar outputs. These findings provide better configurations as well as find out which causal factors are important to drive the changes in the output space.\"\n\nOK - sounds good, but you have to show us which these are and what this means in section 6!","create_time":"$D2021-08-25T11:39:25.338Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"3eafab8594af4e5c","reply_to_id":null,"content":"So only partial integration?\nwhat was the reason? Time, Technology, Data Complexity ... let us know!","create_time":"$D2021-08-25T11:40:19.913Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"2730c35fc2c8d3be","reply_to_id":null,"content":"Please name all contributors explicitly ... and discuss these comments with them!","create_time":"$D2021-08-25T11:44:17.772Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"1080978de412f683","reply_to_id":"9fdc9f76c21826a5","content":"Sounds OK _ as long as you are happy (you took my uninformed suggestion directly!) Feel free to revise and refine!","create_time":"$D2021-09-07T08:49:12.138Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"86b50c30ca9a61c3","reply_to_id":"a6d7cab95c21fe98","content":"Much better - much clearer description - thanks!","create_time":"$D2021-09-07T08:49:51.115Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"455ba015eb6add0b","reply_to_id":null,"content":"Unpin!","create_time":"$D2021-09-07T08:50:01.690Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"eaf32d48cf166585","reply_to_id":"cd515835e912e341","content":"There is a nice paper in one of the notebooks on levels of integration (\"Interactive Visual Steering – Rapid Visual Prototyping of a Common Rail Injection System\" Matkovic et al). Does figure 1 help here, can you use this to identify the level of interactive steering?\nJust a thought!","create_time":"$D2021-09-07T08:54:19.358Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"1f48de520eac935c","reply_to_id":"ff719c6c14f8c2a8","content":"Typo - \"SLINGSBY\" needs a N\n\nThis is the weakest section - if you can add images of examples, with descriptions, this would be helpful to readers.","create_time":"$D2021-09-07T08:56:17.782Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"4de494d215a5a8c5","reply_to_id":"ff719c6c14f8c2a8","content":"Actually - ignore this - you do it nicely below! Sorry!","create_time":"$D2021-09-07T08:56:47.931Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"f50ee59988bda389","reply_to_id":"c4ed328edef7af87","content":"This is a nice explanation - much better, great!\nThe semi-opaque dote work well - try smaller more transparent dots to see more!\nThe thing that is missing is a description of the layout used for the 16*10 grid. I still don't understand this. What do row and column mean? How are they ordered?","create_time":"$D2021-09-07T09:00:31.845Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"44020e8dc9b21c98","reply_to_id":"c4ed328edef7af87","content":"Could you say something about what the different structures mean - we are trying to persuade people of the use of this! \n - straight line [4,1] [15,3]\n - linear patterns [7,4] [5,1]\n - mixed colours [3,3] [3,10]\n - outliers [7,6] [2,3] [12,10] [4,6]\n - coloured outliers [8,8] [7,3] [6,1] [10,4]\nThis would really help!","create_time":"$D2021-09-07T09:05:52.698Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"264971a456aa01c8","reply_to_id":null,"content":"\"More\" is relative - more powerful than what?\nThe more you can focus on the user story - the need you are trying to address, the better!\nuse this as your benchmark, even if your evidence is solely reflective.\nHow well did the methods you used work in the context of the user story once you had used them?","create_time":"$D2021-09-07T09:07:28.459Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"bc00d6600607b032","reply_to_id":"51bdf31e3cf674b7","content":"Again, \"more\" is relative. More than what? ;)\n\"Better\"is relative - better than what ;)","create_time":"$D2021-09-07T09:09:02.294Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"feaa278c533149c4","reply_to_id":"51bdf31e3cf674b7","content":"I really like this:\n\"Our approach supports the identification of which parameter settings generate more variations in its simulation and which settings generate more similar outputs\"\n\nIf your user story is about \"the identification of which parameter settings generate more variations in its simulation and which settings generate more similar outputs\" then this is great!\nAnd if you can really clearly and definitively describe \"which parameter settings generate more variations in its simulation and which settings generate more similar outputs\" in section 6 (and you get pretty close already!) this will work very well. Which are they - and how do the graphics and the tight integration between vis and analytics help with that?","create_time":"$D2021-09-07T09:10:52.590Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"86cdc15e1ffd8ee8","reply_to_id":"3eafab8594af4e5c","content":"This is a slight problem as the idiom is about tight integration - and you don't have tight integration!\nMaybe you have shown that loose integration works well (as a first step in a pandemic, when time and resources are scarce and needs are unknown)?\nCan you reflect on this?","create_time":"$D2021-09-07T09:12:08.585Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"4c17cf70a31bc59a","reply_to_id":null,"content":"Updates are useful, I have added some comments too. Really this is about a system, that combines multiple techniques and idioms. There's lots that's good in here, but the more you can focus on one of the documented methods the easier it will be to reflect and make claims. Tight integration is proposed as important, but actually you didn't achieve this and there isn't really any evidence that the tight integration was necessary (is there?). Instead, it seems that loose integration as actually pretty effective and enabled you to understand a complex parameter space! This may be worth thinking about! Hopefully the comments help you think that through and line things up a bit. This is very rich and very useful! And impressive! I'm sure it will make a nice systems paper on its own!","create_time":"$D2021-09-07T09:18:28.176Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"06b19c0b0d721c18","reply_to_id":"9fdc9f76c21826a5","content":"The one you suggested was identical to the earlier description we wrote in this box, so worked perfectly.","create_time":"$D2021-09-15T13:44:31.215Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"6b1f9966e7dea692","login":"cagatayturkay","name":"","avatar_url":"https://avatars-next.observableusercontent.com/a40227523694e1e50f6ea8dca5c2774a65e88c0442050bbc77c84dcd8144d038","tier":null}},{"id":"ee989cab1cb20205","reply_to_id":"c4ed328edef7af87","content":"As we have 160 configuration sets, it is a straight way to organise them into 16X10 or 10X16. But once the interaction is added, we will consider how we could show few figures each time and make it more interactive.","create_time":"$D2021-09-18T11:16:16.738Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"1ec75da286f18a47","login":"lborohfang","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c46ee6e6aa5472e68f8b9b02e644bcb97e90a873b88b2cf4374ef95fc3b614b8","tier":null}},{"id":"66bd3c320d01c46f","reply_to_id":"3eafab8594af4e5c","content":"Agree with it. Yes, it is not a tight integration at the current stage.","create_time":"$D2021-09-18T11:30:47.595Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"1ec75da286f18a47","login":"lborohfang","name":"","avatar_url":"https://avatars-next.observableusercontent.com/c46ee6e6aa5472e68f8b9b02e644bcb97e90a873b88b2cf4374ef95fc3b614b8","tier":null}},{"id":"959033b23c717eb1","reply_to_id":"c4ed328edef7af87","content":"Sure - but there are 10! ways to arrange the columns (3,628,800) and 6! (720) ways to arrange the rows.\nSo which of these 2.6 billion arrangements did you use?\nAnd why choose that one?\n What do X and Y or R and C mean in the small multiple layout?!\n🤔","create_time":"$D2021-09-18T17:31:33.190Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"c063e248b443d9e0","reply_to_id":null,"content":"It would be great to get full names here!","create_time":"$D2022-02-16T10:28:41.876Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"3ffef3cae07de8e5","reply_to_id":null,"content":"Can you add references in the markdown format shown here? https://observablehq.com/@jsndyks/rampvis-idiom-template#cell-27","create_time":"$D2022-02-16T10:30:52.928Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"a65fa5c0cde7ae5e","reply_to_id":"a6d7cab95c21fe98","content":"CT : Size image by putting it in a separate block with:\n\nawait FileAttachment(\"image@1.png\").image({ width: 600 })","create_time":"$D2022-02-16T11:01:36.937Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"421e4b99bbf8a98b","reply_to_id":"a6d7cab95c21fe98","content":"CT : And below!","create_time":"$D2022-02-16T11:01:56.374Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"00e17abe884f93d3","reply_to_id":"cd515835e912e341","content":"\"In our study\" sounds as though this is a paper, rather than a description of what \"Integrated Algorithmic Tools for Visual Analytics\" is and where it comes from. See the Matkovic et al paper for ideas.","create_time":"$D2022-02-16T14:50:13.072Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"36a276c0b5a676c9","reply_to_id":null,"content":"Reduce image size with \nawait FileAttachment(\"tool (2).png\").image({width:600})","create_time":"$D2022-02-16T14:52:34.927Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}},{"id":"7ca36b292a19aa78","reply_to_id":null,"content":"---\n### Imports","create_time":"$D2022-02-16T14:54:11.352Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"955e37df990d4e95","login":"jsndyks","name":"jason dykes","avatar_url":"https://avatars-next.observableusercontent.com/ce8c740e3f4f161e440b04f5d1f16368eb63cba559ce4c7bdec538f4316764d8","tier":null}}],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"RAMPVIS : Integrated Algorithmic Tools for Visual Analytics","cells":[{"id":0,"value":"md`# RAMPVIS : Integrated Algorithmic Tools for Visual Analytics`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":905,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":360,"value":"---\n## 1. TASK & USERS - User Story","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"\n- As an **epidemiological modeller simulating the spread of Covid**\n- I would like to:\n - explore the relationships between factors in agent-based models and simulated time-series\n- so that I can **select and optimize models to match real World observations**\n\n*Further note:* - The work described here involves the ensemble vis team in which we focus on developing the visual analytics platform as a container and the visual analytics team in which we are investigating algorithmic tools. While these two teams worked separately and worked on different data sets, there is overlap between targeted users and the ways that the idea presented in this notebook operates within this context.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":128,"value":"---\n\n## 2. DATA","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":339,"value":"The data we use in this notebook is 160,000 time-series simulated records from 160 configurations evenly sampled from input space. Each time-series record has 199 dimensions which refer to 199 days' infection and hospitalisation simulation. When we plot all the time-series data illustrated in the following figure, it is difficult to understand how they correspond to the input space where a set of parameters is used to drive a compartment model to generate these curves. Here, each color represents the curves generated from one configuration set among 160 sets. Therefore, it is hard to have a clear view of how these input configurations could make impact on the simulated curves due to the cluttering output curves. \n\n

\n\nThe following is the link to EERA model that we use to generate our data:\nhttps://scottishcovidresponse.github.io/Covid19_EERAModel/\n
\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":862,"value":"md`A data source: [https://coronavirus.data.gov.uk/details/healthcare](https://coronavirus.data.gov.uk/details/healthcare)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"---\n\n## 3. Integrated Algorithmic Tools for Visual Analytics","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":371,"value":"---\n## 4. ORIGIN & CONTEXT","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"Visual analytics systems are popular to provide a better understanding of models, e.g., a machine learning-based facial expression recognition model introduced by [Tam et al (2011)](https://dl.acm.org/doi/10.1111/j.1467-8659.2011.01939.x) and an agent-based modelling model by [Slingsby et al (2019)](https://doi.org/10.2312/eurp.20191153). In many related works, various algorithmic tools are investigated and integrated to facilitate insightful analysis of different types of models. Some recent visual analytics papers, such as [6, 7, 8], discussed how to efficiently combine clustering, classification and dimensional reduction method to improve the effectiveness of VA systems.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":13,"value":"___\n\n## 5. EXAMPLES","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":852,"value":"md`A concrete example of integrating algorithmic tools for VA is in [9]. We used short term (e.g. two weeks) measurements of hospital admissions and optimised a simple compartmental simulator of Covid-19 to fit this data using an evolutionary optimiser. We, therefore, were able to identify a range of plausible vectors of parameters that could have given rise to the observed data using the simulator. The best fit parameter vector runs the simulator to produce a forecast for the next two weeks. The user can then interact with the tool by changing a slider that controls a parameter relating to the contact rate within a given population (Figure below; copied from [9]). \n. \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":855,"value":"await FileAttachment(\"tool (2).png\").image({width:600})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":859,"value":"md`We are currently working on an extension of this tool, where we plot all the vectors of parameters that fit the data well on a parallel coordinate plot. The user can then interactively decide to brush some of the vectors of parameters away, perhaps based on their ancillary data and knowledge of forthcoming policy decisions, and investigate forecasts accordingly. This tool will help users make better decisions on how best to prepare given the forecasts from the simulator based on their belief of what is plausible. \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":53,"value":"---\n## 6. DESCRIPTION & USE","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":170,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":946,"value":"await FileAttachment(\"image@2.png\").image({width:800})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":954,"value":"From the figure, we observe that some closest configuration sets generate outputs belonging to different clusters. One could further analyze the difference between these configuration sets. Another insight is that some input configuration sets, e.g., configuration set No.20 (5th scatterplot from the left in the second row) [0.384, 0.476, 63.3, 0.267, 0.978, 0.36, 1, 0.0007, 12.677, 12.49, 5.30, 10.18, 10000, 0.327 ] has high variations as it produces outputs which are grouped into three clusters. This indicates that there is some sensitive parameter to make an impact on the output curves. A zoom-in is shown as below:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":951,"value":"await FileAttachment(\"image@5.png\").image({width:600})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":944,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":446,"value":"## 7 Reflective Discussion ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":451,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":415,"value":"### 7.1 What Worked","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":427,"value":"Based on observations from the visualization supported by these integrated algorithmic tools, insightful analysis about the relationship between input and output spaces is achieved. Our approach supports the identification of which parameter settings generate more variations in its simulation and which settings generate similar outputs. These findings provide better configurations as well as indicating potential causal factors that are important to drive the changes in the output space.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":434,"value":"### 7.2 What Didn't Work","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":438,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":461,"value":"### 7.3 Recommendations & Opportunities","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":425,"value":"First, we will improve our system in a more interactive manner. This could further help us to understand detailed and subtle patterns in both the input and the output spaces as well as their mapping relationship. We will provide a holistic embedding space so that users could find out how clustered outputs of each configuration setting distribute on the holistic view of the simulated data. Second, we will investigate what clustering algorithms and dimensional reduction methods are more suitable for the analysis. Finally, these will be implemented into the VA platform that our ensemble vis team is developing to enhance the uncertainty and sensitivity analysis of the models.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":624,"value":"### 7.4 Who Was Involved?\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":896,"value":"Hui Fang, Alma Rahat, Cagatay Turkay, Franck Vidal, Rita Borgo, Shaunave Santhirasekaran\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":16,"value":"---\n\n## 8. REFERENCES","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":769,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":932,"value":"---\n### Imports","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":902,"value":"\nimport { ui } from \"@mootari/environment\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W38","stdlib":"1"}} 30:"$2f:body"