2b:null 2c:["$","title",null,{"children":"WebAPI* Commons, a COVID-19 Study | Jonas Almeida | Observable"}] 31:T7ba,html`

WebAPI* Commons, a COVID-19 Study

A study of how APIs* organically migrate to the Web stack under distributed duress, creating nimble, democratized, epidemiological data commons in real-time.

*Application Programming Interface, HTTP REST

Figures

Trajectory - Raw confirmed, recovered and death case count in a 3D plot.

Progression - Infection progression accessed by proportion of new cases plotted agains letal count.

Reporting Lag - QAQC plot checking timelyness of reported data.

Introduction

For reference epidemiology information about corona virus COVID-19 please refer to coronavirus.gov. This Observable Notebook is a portable computational exploration of real-time visualization and analysis of epidemiology data, for which the coronavirus epidemic is the pressing real-world scenario. To that effect, a data retrieval, wrangling and visualization library was developed to support applications in a variety of forms, from regular web pages to nodejs FaaS (server-side), including literate programming in observable notebooks like this one.

`32:Tc9b,html`

Data

Arguably, the most impactfull and comprehensive data dissemination resource about COV-19 epidemic was the one communicated by the report from Johns Hopkins' School of Engineering.

Dong, Ensheng, et al. “An Interactive Web-Based Dashboard to Track COVID-19 in Real Time.” The Lancet Infectious Diseases, Feb. 2020, doi:10.1016/S1473-3099(20)30120-1 [PMID:32087114].

This reference data resource is updated daily at the Whitting School of Engineering, Center for Systems Science and Engineering Center at Johns Hopkins University using GitHub. As a consequence of servind the data from the same git infrastructure where it is versioned, Github.com/CSSEGISandData/COVID-19 keeps both present and past versions of the reference daily updates and the time series records. The interoperability mechanism is now that of fecthing csv formated data as detailed in this pull request. As discussed below, the data ingestion and indexing is transferred to the clienct side enviornment, along the same lines outlined in Figure 1 of

Almeida JS, J Hajagos, J Saltz J, M Saltz (2019) Serverless OpenHealth at data commons scale—traversing the 20 million patient records of New York’s SPARCS dataset in real-time. PeerJ 7:e6230 [PMID:30671301].

CDC

A number of reference data assets are also emerging as part of public dashboards describing the development of the pandemic. They generaly follow the same approach of maintaining a static file, which the client application ingests in what becomes a reference visualization. This is illustrated, for example, by the map at the end of the coronavirus.gov page, which displays the data kept at cdc.gov/coronavirus/2019-ncov/map-data-cases.csv. As noted in teh reference above, this is an approach that shifts data ingestion and indexing in a database as a sitributed process handed to the Web Broser stack, which now includes both standard relational database and a state of the art assynchronous NOSQL data store, IndexedDB.

`33:Tbf9,trajectoryPlot = { let cc = await corona.progression(); let div = DOM.element('div'); //div.style.height = "100%"; //debugger; div.id = 'trajectoryPlotId'; let trace = (country = "China", color) => { let c = cc[country]; //debugger; return { type: 'scatter3d', name: country, z: c.active, //.map((x, i) => x / c.confirmed[i]), y: c.deaths, x: c.recovered, //mode: 'markers', marker: { size: 3, line: { //color: 'rgba(217, 217, 217, 0.9)', width: 0.5 }, color: color, opacity: 0.2 } }; }; //debugger; Plotly.plot( div, [ trace('Egypt', 'rouge'), trace('Sweden', 'cider'), trace('Belgium', 'silver'), trace('Netherlands', 'olive'), trace('Canada', 'plum'), trace('Philippines', 'chartreuse yellow'), trace('Indonesia', 'periwinkle'), trace('Korea South', 'green'), trace('Japan', 'purple'), trace('Portugal', 'lime'), trace('Brazil', 'magenta'), trace('India', 'pink'), trace('Iran', 'hue'), trace('Germany', 'brown'), trace('United Kingdom', 'violet'), trace('France', 'cyan'), trace('Spain', 'gold'), trace('Italy', 'orange'), trace('China', 'red'), trace('US', 'blue') //trace('Nigeria', 'chartreuse yellow'), //trace('Russia', 'chartreuse yellow') ], { title: `Progression trajectory, raw data
(to stop animation and rotate manually, hover figure with pointer and click home)`, legend: { traceorder: 'reversed' }, scene: { width: 1000, height: 700, xaxis: { title: 'Recovered', type: 'log', autorange: true }, yaxis: { title: 'Deaths', type: 'log', autorange: true }, zaxis: { title: 'Active', type: 'log', autorange: true }, //id: 'trajectoryScene' camera: { center: { x: -0.2, y: -0.2, z: -0.2 }, eye: { x: 1.5, y: 1.5, z: 1 } } } } ); //console.log('scene div', div.querySelector('#scene')); setTimeout(function() { console.log('plotly:', Plotly); function run() { rotate('scene', Math.PI / 180); //rotate('scene2', -Math.PI / 180); requestAnimationFrame(run); } run(); function rotate(id, angle) { var eye0 = div.layout[id].camera.eye; var rtz = xyz2rtz(eye0); rtz.t += angle; var eye1 = rtz2xyz(rtz); Plotly.relayout(div, id + '.camera.eye', eye1); } function xyz2rtz(xyz) { return { r: Math.sqrt(xyz.x * xyz.x + xyz.y * xyz.y), t: Math.atan2(xyz.y, xyz.x), z: xyz.z }; } function rtz2xyz(rtz) { return { x: rtz.r * Math.cos(rtz.t), y: rtz.r * Math.sin(rtz.t), z: rtz.z }; } //corona.rotate3D(div); }, 1000); return div; }34:T7f8,progressionPlot = { let div = DOM.element('div'); let trace = (country, color) => { let xx = progression[country]; let period = 7; return { name: country, type: 'scatter', mode: 'lines+markers', marker: { color: color, size: 6, line: { //color: 'rgba(217, 217, 217, 0.9)', width: 0.5 } }, x: xx.deaths.slice(period), //.map((x, i) => x / xx.confirmed[i]), y: xx.confirmed .slice(period) .map((x, i) => Math.max(0, (100 * (x - xx.confirmed[i])) / x)), text: xx.times.slice(period) }; debugger; }; Plotly.newPlot( div, [ trace('Egypt', 'rouge'), trace('Nigeria', 'chartreuse blue'), trace('Sweden', 'cider'), trace('Belgium', 'silver'), trace('Netherlands', 'olive'), trace('Canada', 'plum'), trace('Philippines', 'chartreuse yellow'), trace('Indonesia', 'periwinkle'), trace('Korea South', 'green'), trace('Japan', 'purple'), trace('Portugal', 'lime'), trace('Brazil', 'magenta'), trace('India', 'pink'), trace('Iran', 'hue'), trace('Russia', 'chartreuse red'), trace('Germany', 'brown'), trace('United Kingdom', 'black'), trace('France', 'cyan'), trace('Spain', 'gold'), trace('Italy', 'orange'), trace('China', 'red'), trace('US', 'blue') ], { xaxis: { title: 'deaths', type: 'log', range: [3, 5.5] }, yaxis: { title: '# past week cases as % of total', type: 'log', range: [-1, 2] }, height: 550, width: 650, legend: { traceorder: 'reversed' }, title: `Infection progression [${Date() .toString() .slice( 0, 33 )}]
(move mouse-over series to see dates, single/double click legend to select countries)` } ); return div; }35:T413,md`

Figure legend: Different countries will test COVID-19 differently so the rate of change in the numer of cases will be more informative about the momentum of the infection than the absolute numbers (for that see previous figure). Y-AXIS: The proportion of total cases observed so far that were reported in the last week was chosen to represent that rate because of the straigh-forward reading. For example, 50% will indicate that the number of positive cases doubled in the last week, whereas 75% indicates that it quadrupled, 80% that it quintupled, and so on. X-AXIS: number of letal cases is the one absolute count that is the same for all countries. Accordingly, the representation of rate of change in the number of cases as a function of letality, will provide a visualization of the momentum of the infection. The time scale is assessed by the number of markers (one per day). Moving the mouse over individual markers will display the date of the observation.

`36:T78d,timelyness = { let xx = dailyUpdate; let t = dailyUpdate.map(x => x["Last_Update"]); let traceCountry = (country, clr) => { let xx = dailyUpdate.filter(x => x["Country_Region"] == country); let confirmed = xx.map(x => x.Confirmed); let text = xx.map(x => { if (x["Country_Region"].length < 2) { return x["Country_Region"]; } else if (x["Province_State"].length > 1) { return x["Province_State"]; } else { return x["Country_Region"]; } }); let traceConfirmed = { name: country, x: t, y: confirmed, text: text, mode: 'markers+text', textposition: 'right', textfont: { size: 8, color: 'gray', orientation: 30 }, type: 'scatter', marker: { color: clr, size: 6 } }; return traceConfirmed; }; let div = DOM.element('div'); let data = [ traceCountry('China', 'red'), traceCountry('US', 'blue'), traceCountry('India', 'cyan'), traceCountry('Australia', 'green'), traceCountry('Japan', 'purple'), traceCountry('Italy', 'orange'), traceCountry('Germany', 'brown'), traceCountry('France', 'pink'), traceCountry('Spain', 'gold'), traceCountry('Portugal', 'lime'), traceCountry('United Kingdom', 'black'), //traceCountry('Netherlands', 'gray'), traceCountry('Brazil', 'magenta'), traceCountry('Nigeria', 'olive'), traceCountry('Egypt', 'plum') ]; Plotly.newPlot(div, data, { title: `Latest data updates (real time) from selected countries and regions
(lagging updates will trail to the left of concentrated vertical)`, yaxis: { title: 'Confirmed cases', type: 'log', autorange: true }, xaxis: { title: 'Last update' } }); return div; }37:[] 2f:{"id":"c5e721654ffced23","slug":"corona","title":"WebAPI* Commons, a COVID-19 Study","thumbnail":"32bf40600b169203940cf0f7945b9716609c663d0306616b76a266432a5e26bc","access_level":"public","create_time":"$D2020-03-11T01:32:52.304Z","update_time":"$D2022-11-08T13:47:56.727Z","view_time":null,"first_public_time":"$D2020-03-11T01:33:00.556Z","fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"0f2e596d5bd6ef6e","login":"episphere","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/3a0ff5abff6da0cdbd100da305ba323e12d43c8fcb7604275e6c72591db9dc69","tier":"starter_2024"},"version":788,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"4355fe13cb71b11d","login":"episphere","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/3a0ff5abff6da0cdbd100da305ba323e12d43c8fcb7604275e6c72591db9dc69","tier":null}],"members":[{"type":"individual","id":"4355fe13cb71b11d","login":"episphere","name":"Jonas Almeida","avatar_url":"https://avatars-next.observableusercontent.com/3a0ff5abff6da0cdbd100da305ba323e12d43c8fcb7604275e6c72591db9dc69","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":3,"collection":null,"type":"notebook","body":{"title":"WebAPI* Commons, a COVID-19 Study","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":15,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"md`\n

Results

\nA support javascript library, corona.js was developed (see Methods) to assess the ability to ingest COVID-19 epidemiology data generated, versioned, and served from github. This library is loaded below in the Methods section with
corona = require('https://episphere.github.io/corona/corona.js'),
and will be used here to retrieve and display the data. The same library is loaded via script tag by a regular web application at episphere.github.io/corona:
<script src=\"https://episphere.github.io/corona/corona.js\" ></script>.\n

Daily reports

\nThe daily reports can be retrieved in real time by corona.getDaily() ...\n

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":43,"value":"dailyUpdate = corona.getDaily()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"md`\n... which gives us the opportunity to assess the timelyness of individual updates for selected countries and regions. A plot of lag times is included in the Methods section\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":240,"value":"md`\n

Time series

\nThe reference data resource managed in GitHub by Johns Hopkins School of Engineering (See Data) includes a folder with updated times series. As with the daily updates above, the exploratory study of emerging distributed data commons will approach that versioned time series resource. The method that delivers that functionality is corona.getSeries() and can be inspected directly at the corona.js source. Let's use them here to get the latest times series for confirmed, ceceased, and recovered cases:\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":52,"value":"deaths = corona.getGlobalSeries('deaths')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"confirmed = corona.getGlobalSeries('confirmed')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":54,"value":"recovered = corona.getGlobalSeries('recovered')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":353,"value":"md`\n

\nAs before, all data is cached in the Web Browser's NOSQL assynchronous database, IndexedDB. That is, the Web browser's persistent layer will play the role traditionally assigned to the remote server-side, in essence working as a within browser local server-side. As emphacized in the conclusions, architecturally this ammounts to a distributed server-side model where the distribution is parceled to the user client-side computational resources. At this point the data is indexed locally, and more complex visulizations can now be attempted.\n

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":367,"value":"progression = corona.progression()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":376,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":451,"value":"md`\n

\nThe plot above was produced with data wrangled by corona.progression(). The aggregation by country has its own method, corona.agregateByCountry().The 3D plot suggests that different populations follow different trajectories that start at diferent points in time. That data includes also the number of confirmed cases, which can be used as the denominator of a progression axis.\n

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":455,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":629,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"html`\n

Methods

\nSupport javascript library maintained at github.com/episphere/corona.\n

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"corona = require('https://episphere.github.io/corona/corona.js')\n//corona = require('http://localhost:8000/corona/corona.js')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":123,"value":"Plotly = require(\"https://cdn.plot.ly/plotly-latest.min.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":133,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":626,"value":"md`\n

Discussion

\nThe evolution of FAIR interoperability models during public health emergencies highlights its most effective components by necessity. This process faces the same FAIR challanges of those faced by slower moving epidemiologica systems: how to simultaneously enunciate and demonstrate portable technical solutions that verify them. However, faster epidemics do so without contemplation for the architectural bias of a central authority. That is also the reason why we report this exercise through literate programming in an Observable Notebook: such that the observations can be verified by live desintermediated computational constructs that users can engage without any reliance on analytical resources beyond those deployable in the modern Web Browser.\n

...

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":257,"value":"md`\n

Conclusions

\nFiguratively speaking, what we observe in this exploration, as with other computational tools tracking the CV-ID19 pandemic (see for example nextstrain.org/ncov), is the migration of the remote (Cloud-hosted) server-side data base and data wrangling functionalities to local within-Browser persistent resources. This appears to fit the general patthern towards progressive web applications (PWAs), by relying on persistent indexed storage the modern web browser now provides.\n

...

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W37","stdlib":"1","titling":"auto"}} 30:"$2f:body"