32:Tbb6,md`# Visualizing Temporal Graph Networks Zach Stoebner & Cole Sawyer Project notebook: [https://observablehq.com/@zstoebs/cs-8395-project](https://observablehq.com/@zstoebs/cs-8395-project) GitHub Repo: [https://github.com/zstoebs/tgn](https://github.com/zstoebs/tgn) ## Usage 1. Clone the repo. 2. \`\`cd viz/\`\` 3. \`\`python server.py\`\` 4. Open the project notebook in Chrome. Safari will likely not work with the server. Tips: - If the view cells aren't rendered, run the server endpoint cells. - It's a big, complex viz so it may take a while to compute the full graph to a point where you can interact with the views. ## Description **We propose to design visualizations that present, reveal, and clarify information contained in and produced by temporal graph networks [TGNs].** For educational purposes, we want to better understand this type of GNN, from both a high and low-level point of view. To fulfill that vision, we need as much flexibility as possible in a visualization tool, permitting interaction with the underlying model and yielding different qualitative and quantitative views to easily and precisely comprehend its inner workings. ## Background Back in 2009, Scarselli et al published the original GNN paper [1]. Their proposed model addressed a niche type of dataset -- a graph. It is common knowledge that many structures in diverse fields, ranging from computer vision to social sciences, are graphs, irrefutably. Instead of preprocessing away from graph structure, the authors invented a new model that operated directly on the graph. Derived from recursive neural nets, the supervised learning algorithm conducts a random walk across the graph, computing a context embedding for each node based on its neighborhood and an output from two separate feed-forward functions. After determining the node’s loss, the weights are updated according to a backpropagation-through-time algorithm. Although they are unique and interesting, GNNs are not without their limitations. For one, they assume the Banach fixed point theorem which guarantees the existence and uniqueness of a solution, making iterative learning exponentially fast. Unfortunately, the theorem also limits random walks; if the graph is changed, i.e., a vertex is added, then the entire training algorithm has to be rerun from scratch because learning is transductive. Fortunately, GraphSage, proposed by Hamilton et al, implements inductive learning on the vanilla GNN architecture, mitigating the aforementioned problem [2]. However, the vanilla GNN always assumed a static dataset, limiting the potential for the GNN to adjust dynamically in a time efficient manner. On top of that, computational complexity and, in particular, memory requirements can drastically increase with growing graphs. Hence, in 2009, the infrastructure required for effective implementation of GNNs may not yet have been established, frustrating the quick adoption of the GNN by machine learning researchers. ` 33:T5da,md`Before its time perhaps, the GNN was essentially a sleeper for a decade. If you ask people in the field, more than a few would probably expect that GNNs were published only in the last couple of years. However, this misconception is likely due to their resurgent popularity, corresponding with the increasing prominence of graph structured data and their ability to learn complex relationships between input examples. Some subject areas that naturally have graphical data include: - Social network analysis - Image processing - Molecules Given this obvious trope in GNN theory, the assumption of static data spurred research into dynamic graph models, which ultimately, and recently, lead to the innovation of temporal GNNs, proposed by Rossi et al in 2020 [3]. Holding onto the components of the original GNN, TGNs improve upon these components in cunning ways to account for dynamic updates to the dataset. TGNs improve vanilla GNN’s message passing by incorporating memory modules into the workflow that keep a ledger for each node of all its time-stamped interactions. That way, when computing the context embedding for the node, the optimization also considers the neighborhood’s states at previous timepoints. Compressed memory representations allow the model to better estimate the context of nodes, outperforming regular GNNs. Even though a node’s memory is only updated when an event pertaining to that node occurs, the embedding module accounts for staleness, thankfully. ` 34:T10d0,md`## Tasks To understand the TGN from a high-level perspective, we first want to understand the output in order to understand the results, verify that the model works as expected and validate that our model is truly learning the temporal nature of the data. To explore potential outcomes, live updates to the model would be useful. Not only would the experience of tweaking the hyperparameters of the model in real-time introduce users to the intricacies of their model but it would also help them understand the range of outcomes possible. On a low-level perspective, a trained TGN can be interpreted as the learned context embeddings for each of the nodes. Visualizing these embeddings in space relative to one another is a critical piece to comprehending relationships within the dataset and how separate TGNs differ comparatively. In tandem, visualizing the temporal evolution of the graph over a given time course and inspecting the projections, as well as their raw embeddings, can reveal more information about the dynamic nature of the graph at hand. It is important to mention, considering the limitations of GNNs and TGNs, expected complications that we may face. First and foremost, the amount of memory storage for the trained model will likely be large. We will have to be careful and clever about implementing our model to ensure that the GPU can hold the entire graph and parameters while training and that our Python server backend can access enough data. Secondly, appropriate hardware solutions will need to be sourced if necessary, i.e., ACCRE, Google CoLab, or a personal GPU, in order to efficiently train the TGN. Finally, we need to be cognizant of the complexity of our visualization; the graph can become so large and dense that if we try to visualize it 1-to-1 then understanding the graph could be a steep learning curve. ## Data We will primarily use the Wikipedia dataset that was originally used in the TGN paper. This dataset consists of user edits. Each entry contains the user ID, page ID, timestamp, a state label, and a vectorized list of features. This data is already preprocessed so the focus can be on the visualization rather than on the model itself. In addition, we will use a dataset of Venmo transactions mined by Dr. Tyler Derr at Vanderbilt. This dataset is currently being used by the Vanderbilt Network & Data Science Lab in TGN research. This dataset consists of a list of transactions that includes the sender and receiver information, a timestamp, and the message. In order to properly input this data into the TGN, the message must be converted into a numeric vector using an NLP neural network, such as BERT. The original TGN network code provided by Twitter Research provides preprocessing for the data, once put in the form of \`\`Entry(source, target, timestamp, feature_vector)\`\`. ## Model The main model that we are using is the Temporal Graph Network developed by Twitter Research [3]. The model uses nodes and edgewise messages which are then converted to embeddings. The nodewise message function consists of the previous state, the current time, and the incoming entry. Edgewise messages are split into two messages if the graph is undirected, and one if the graph is directed, consisting of the previous states of the sender, the change in time, and the interaction event. These messages are then aggregated into batches, then encoded using various methods, including temporal graph attentions, which use a time-embedded query, key value system. These embeddings are then translated into edge probabilities. This model can run both a supervised edge prediction problem and a semi-supervised node classification problem. ## Designs Visualizing Graph neural networks can be difficult due to the extremely high dimensional nature of graphs and the models. For our project we will be visualizing both the output graph, as well as the model itself. For the node classification problem, we will highlight improperly classified nodes so the user can view the errors of the model. In addition, we plan to implement a temporal version of GNNVis [2]. This paper is the best current attempt to explain GNNs visually. To add a temporal aspect to this viz, we can add a time slider to demonstrate predictions based on a given time. ` 35:T1bf8,md`# Writeup ## Introduction Our project, TGNVis, is a visual analytic tool to help better understand the output of link prediction using temporal graph networks. Our project was loosely based on GNNVis, a tool created to visualize classification output on graph neural networks. This implementation was insufficient for our purposes, as link prediction has very different visualization needs than classification. ## Modules/Technical Details ### Preliminaries & Back end: To generate the graph using ACCRE, we ran the TGN on the Wikipedia dataset from http://snap.stanford.edu/jodie/ on both the basic link prediction task for the probability data and the node classification task for the context data. To preprocess and arrange the graph, we used NetworkX. To serve the data, we used Flask and, to compute t-SNE, we used SciKit-Learn. All backend code was written in Python and can be cloned from https://github.com/zstoebs/tgn. ### Front end: TGNVis currently has 3 main views: a full graph view, a subgraph view, and a subgraph scatterplot view. The full graph view consists of a link-node diagram that displays all of the nodes in the validation set. Each link color is scaled logarithmically based on the timestamp of the link occurring, with brighter and more saturated colors representing later timestamps. Link opacity linearly encodes the probability of the link occurring according to our model, with more opaque links signifying a higher predicted probability. Users can pan and zoom around the full graph view as well as brush when holding the Alt key (Windows / Linux) or Command key (macOS) to select a subset of nodes that will populate the subgraph view and the scatterplot. When nodes are brushed in the full graph, they will change colors from black to purple. Brushing in d3.js does not coexist well with most other interactions so it is inherently bugged; we suggest waiting for the force simulation to settle and trying to repeatedly to brush. The subgraph view consists of nodes that have been brushed on the full graph view, plus all of their 1-hop connections. Once the subgraph is populated, hovering over a node or a link will provide the available information about the node or link being displayed. Once again, users can pan and zoom around the subgraph to inspect specific elements. Upon brushing again in the full graph, the subgraph and scatterplots will repopulate. The scatterplot view contains the 2-component t-SNE dimensionality reduction view of the context embeddings available for the source and destination nodes of links on the subgraph. In order to retain the most information possible for each link, we provide two subplots that display reduced source and destination embeddings, left to right respectively. We also provide brushing functionality for our scatterplot view on the source node plot. When the plot is brushed, the brushed points will change color to blue, and the corresponding points in the destination scatterplot will also change color to red and lines are drawn to identify the links between selected nodes in the scatterplots. In addition, the subgraph view will update the colors of the selected nodes accordingly with one caveat: if it corresponds to both a source and destination embedding, it will change to purple. Nodes are often involved in multiple link events and therefore have multiple instances in the scatterplot view. To account for this edge case, we mix the color accordingly to purple. ## Justification Our final product had several deviations from the original GNNVis modules. We decided to include a full graph view and work down from there because it helps the user understand the underlying graph structure better without requiring interpretation at the extremely complicated level of the full graph. We originally were going to populate our subgraph view by manually selecting nodes and including their neighbors and 1-hop neighbors, but ultimately decided to go with the brushing technique instead due to the poor interpretability of the full graph. We noticed that manually entering nodes was relatively tedious and did not give us an intuitive understanding of what we were selecting. In addition, we would potentially have disconnected or very loosely connected components in our subgraph, which would increase confusion. In our scatterplot view, we considered combining our source and destination embeddings into one plot, but thought that this view would lose too much information in our reduction since the embeddings live in different spaces. We reconciled the increase of complexity by including brushing functionality to help better understand the relation between source and destination embeddings. We considered including an embedding feature matrix, but thought that this view would scale very poorly. In addition, the TGN architecture does not correspond well to a fine-grained feature matrix, since the embeddings are piped to an MLP to produce probability outputs, which means that the embeddings have a very non-linear relation to the probability output, reducing the effectiveness of a feature matrix. All in all, we utilized brushing as a central interactive component as a way to break down the bulk over an initially complicated structure, elucidating some information at the subgraph level that likely would not be noticed at the full graph level. ## Assessment Our visualization attempts to explain the output of TGNs. This problem is relatively difficult to solve due to the black-box nature of TGNs as well as the high dimensionality of the output data. In our visualizations, we sacrifice fine-grained analysis to help better understand the overarching structure of the network output. We believe that our visualization could be easily ported to other GNN architectures that solve link prediction problems. Our project can be used for tasks such as error analysis or could be run on multiple models to provide a model comparison. In error analysis, the user can find positive examples with low probability, and determine why this embedding had a lower probability based on the scatterplot data. A user could run two instances of our visualization on separate models to compare edge probabilities and similarities of node embeddings. For example, if two nodes have similar embeddings in one model but are very different in another, the user will know that the neighborhoods of these nodes provide good example data that is sensitive to the differences in parameters between the models. We believe that our model can still be improved upon in future work. Firstly, debugging the coexistence of brushing and other interactions on the same view, as well as optimizing our backend and force simulations for speed, would provide a massive quality-of-life improvement to our visualization. Beyond the basics, we envision a view that allows for hierarchical clustering of nodes to be able to view predictions on subsets of the graph as a whole. This view would require a novel implementation of aggregated link prediction, as, to our knowledge, there is no currently available algorithm to view links between and among aggregated nodes. `36:Td27,plot = { mutable info = []; mutable brushed_nodes = []; mutable brushed_source = new Set(); mutable corresponding_dest = new Set(); const links = graph.edges; const nodes = graph.nodes; const simulation = d3.forceSimulation(nodes) .force("link", d3.forceLink(links).id(d => d.id)) .force("charge", d3.forceManyBody()) .force("center", d3.forceCenter(width / 2, height / 2)); const svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) const g = svg.append('g') const link = g.append("g") .attr("stroke", "#999") .attr("stroke-opacity", 0.6) .selectAll("line") .data(links, d => [d.source, d.target]) .join("line") .attr('stroke',d => edge_color_scale(d.timestamp)) .attr('opacity', d => edge_opacity_scale(d.prob)) const globalNode = g.append("g") .attr('fill',d3.hcl(0,0,0)) .attr("stroke", "#fff") .attr("stroke-width", 1.5) const node = globalNode .selectAll("circle") .data(nodes) .join(enter => enter.append("circle")) .attr("r", 5) .attr('id', d=> d.id); node.append("title") .text(d => d.id); simulation.on("tick", () => { link .attr("x1", d => d.source.x) .attr("y1", d => d.source.y) .attr("x2", d => d.target.x) .attr("y2", d => d.target.y); node .attr("cx", d => d.x) .attr("cy", d => d.y); }); invalidation.then(() => simulation.stop()); // BRUSH ///////////////////////////////////////////////////////////////////////////////////////// let brush = d3.brush() .extent([[0,0],[width,height]]) .filter(event => event.metaKey || event.altKey) .on('start brush end', brushed) let brushed_data = [] function brushed({selection}) { if (selection) { const [[x0, y0], [x1, y1]] = selection; node.attr('fill',d3.hcl(0,0,0)) .filter(d => x0 <= d.x && d.x < x1 && y0 <= d.y && d.y < y1) .attr('fill',d3.hcl('purple')); brushed_data = graph.nodes.filter(d => x0 <= d.x && d.x < x1 && y0 <= d.y && d.y < y1); mutable brushed_nodes = brushed_data.length > 0 ? brushed_data.map(d => d.id) : []; } else { node.attr('fill',d3.hcl(0,0,0)) } } g.call(brush) .call(gr => gr.select(".overlay").style("cursor", "default")); ///////////////////////////////////////////////////////////////////////////////////////// // PAN & ZOOM // https://observablehq.com/@scax/force-directed-graph-with-zoom-and-stroke-scaling-svg ///////////////////////////////////////////////////////////////////////////////////////// let zoomLvl = 1; let lastK = 0; svg.call(d3.zoom() .extent([[0, 0], [width, height]]) .scaleExtent([1, 80]) .filter(event => !event.metaKey && !event.ctrlKey && !event.button) .on("zoom", zoomed)); function zoomed(e) { // d3.selectAll(".brush-container, .brush").call(brush.clear()); if(e.transform.k > 2 && lastK != e.transform.k){ lastK = e.transform.k; zoomLvl =Math.log2(e.transform.k); globalNode.attr("stroke-width", 1.5/zoomLvl ); link.attr("stroke-width", d => Math.sqrt(d.value)/(zoomLvl)); } g.attr("transform", e.transform); } ///////////////////////////////////////////////////////////////////////////////////////// return svg.node(); }37:T10f0,plot_subgraph = { const links = subgraph.edges; const nodes = subgraph.nodes; const simulation = d3.forceSimulation(nodes) .force("link", d3.forceLink(links).id(d => d.id)) .force("charge", d3.forceManyBody()) .force("center", d3.forceCenter(width / 2, height / 2)); const svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) const g = svg.append('g') const link = g.append("g") .attr("stroke", "#999") .attr("stroke-opacity", 0.6) .selectAll("line") .data(links, d => [d.source, d.target]) .join("line") .attr('stroke',d => edge_color_scale(d.timestamp)) .attr('stroke-opacity', d => edge_opacity_scale(d.prob)) .on('mouseover',linkover) .on('mouseout',linkoff) const globalNode = g.append("g") .attr('fill',d3.hcl(0,0,0)) .attr("stroke", "#fff") .attr("stroke-width", 1.5) const node = globalNode .selectAll("circle") .data(nodes) .join(enter => enter.append("circle")) .attr("r", 5) .style('fill', d => { let isSource = brushed_source.has(d.id); let isDest = corresponding_dest.has(d.id); if(isSource && isDest){ return d3.hsl(282, 1, .5); } else if(isSource){ return d3.hsl(236,1, .5); } else if(isDest){ return d3.hsl(0,1, .5); } else{ return d3.hsl(0, 0, 0); } }) .on('mouseover',nodeover) .on('mouseout',nodeoff) node.append("title") .text(d => d.id); simulation.on("tick", () => { link .attr("x1", d => d.source.x) .attr("y1", d => d.source.y) .attr("x2", d => d.target.x) .attr("y2", d => d.target.y); node .attr("cx", d => d.x) .attr("cy", d => d.y); }); invalidation.then(() => simulation.stop()); // HOVER ///////////////////////////////////////////////////////////////////////////////////////// function nodeover(e,d) { let cx = d3.select(this).attr('cx') let cy = d3.select(this).attr('cy') d3.select(this) .attr('fill', d3.hcl(0,0,75)) .attr('r', 9) let id = d.id.toString() mutable info = ['Node ID: ' + id] g.append('rect') .attr('x',cx-10) .attr('y',cy-10) .attr('width',id.length*10) .attr('height',12) .attr('fill',d3.hcl(0,0,100)) .classed('text_box',true) g.append('text') .attr('x',cx) .attr('y',cy) .attr('font-size',12) .text(() => id) .classed('node_id',true) } function nodeoff(e,d) { d3.select(this).attr('r',5).attr('fill',d3.hcl('black')) g.selectAll('.node_id').remove() g.selectAll('.text_box').remove() mutable info = [] } function linkover(e,d) { let x1 = d3.select(this).attr('x1') let x2 = d3.select(this).attr('x2') let y1 = d3.select(this).attr('y1') let y2 = d3.select(this).attr('y2') d3.select(this).attr('stroke-width',6) mutable info = ['Source: ' + d.source.id.toString(), 'Target: '+d.target.id.toString(), 'Prob: '+d.prob.toFixed(3).toString(), 'Timestamp: '+d.timestamp.toString()] } function linkoff(e,d) { d3.select(this).attr('stroke-width',1) mutable info = [] } ///////////////////////////////////////////////////////////////////////////////////////// // PAN & ZOOM // https://observablehq.com/@scax/force-directed-graph-with-zoom-and-stroke-scaling-svg ///////////////////////////////////////////////////////////////////////////////////////// let zoomLvl = 1; let lastK = 0; svg.call(d3.zoom() .extent([[0, 0], [width, height]]) .scaleExtent([1, 80]) .filter(event => !event.metaKey && !event.ctrlKey && !event.button) .on("zoom", zoomed)); function zoomed(e) { // d3.selectAll(".brush-container .brush").call(brush.clear()); if(e.transform.k > 2 && lastK != e.transform.k){ lastK = e.transform.k; zoomLvl =Math.log2(e.transform.k); globalNode.attr("stroke-width", 1.5/zoomLvl ); link.attr("stroke-width", d => Math.sqrt(d.value)/(zoomLvl)); } g.attr("transform", e.transform); } ///////////////////////////////////////////////////////////////////////////////////////// return svg.node(); }38:T17ac,scatterplot = { let svg = d3.create('svg').attr('width', width).attr('height', local_height).property("value", []); svg.append('text').attr('x', 0).attr('y', 15).attr('fill', d3.hcl(0,0,0)) .attr('font-size', '18px').text('Source Nodes') svg.append('text').attr('x', margin+width/2).attr('y', 15).attr('fill', d3.hcl(0,0,0)) .attr('font-size', '18px').text('Destination Nodes') let source_g = svg.append('g') .attr('transform', `translate(${margin},${margin})`) .attr('width',local_width) .attr('height',local_height-margin) let dest_g = svg.append('g') .attr('transform', `translate(${margin+width/2},${margin})`) .attr('width',local_width) .attr('height',local_height-margin) let source_nodes = source_g.selectAll('circle') .data(source_data) .enter() .append('circle') .attr('cx', d => source_x_scale(d.x)) .attr('cy', d => source_y_scale(d.y)) .attr('r', 4) .attr('fill', d3.hcl(0,0,75)) .attr('opacity',0.75) .attr('stroke', d3.hcl(0,0,0)) .attr('stroke-opacity', .5) let dest_nodes = dest_g.selectAll('circle') .data(dest_data) .enter() .append('circle') .attr('cx', d => dest_x_scale(d.x)) .attr('cy', d => dest_y_scale(d.y)) .attr('r', 4) .attr('fill', d3.hcl(0,0,75)) .attr('opacity',0.75) .attr('stroke', d3.hcl(0,0,0)) .attr('stroke-opacity', .5) source_nodes.raise() dest_nodes.raise() // BRUSH ///////////////////////////////////////////////////////////////////////////////////////// let brush = d3.brush() .extent([[margin,margin],[local_width+local_margin,local_height]]) .on('start brush end', brushed); function brushed({selection}) { svg.selectAll('.edge').remove() if (selection) { const [[x0, y0], [x1, y1]] = selection; source_nodes.attr('fill',d3.hcl(0,0,75)) .filter(d => x0 <= source_x_scale(d.x)+margin && source_x_scale(d.x)+margin < x1 && y0 <= source_y_scale(d.y)+margin && source_y_scale(d.y)+margin < y1) .attr('fill',d3.hcl('blue')); // get source selection let sources = source_data.filter(d => x0 <= source_x_scale(d.x)+margin && source_x_scale(d.x)+margin <= x1 && y0 <= source_y_scale(d.y)+margin && source_y_scale(d.y)+margin <= y1); mutable brushed_source = new Set(sources.map(d => d.node)); // get dest selection & update color on dest scatter let inds = source_data.map(d => x0 <= source_x_scale(d.x)+margin && source_x_scale(d.x)+margin <= x1 && y0 <= source_y_scale(d.y)+margin && source_y_scale(d.y)+margin <= y1); let dests = dest_data.filter((d,i) => inds[i]); let edges = svg.selectAll('edge') .data(sources) .enter() .append('line') .attr('x1', d => source_x_scale(d.x)+margin) .attr('x2', (d,i) => dest_x_scale(dests[i].x)+local_width+margin+local_margin+10) .attr('y1', d => source_y_scale(d.y)+margin) .attr('y2', (d,i) => dest_y_scale(dests[i].y)+margin) .attr('stroke',d3.hcl(0,0,50)) .attr('stroke-opacity',0.6) .classed('edge',true) edges.lower() dest_nodes.attr('fill',d3.hcl(0,0,75)) .filter((d,i) => inds[i]) .attr('fill', d3.hcl('red')); mutable corresponding_dest = new Set(dests.map(d => d.node)); } else { source_nodes.attr('fill',d3.hcl(0,0,75)); dest_nodes.attr('fill',d3.hcl(0,0,75)); svg.selectAll('.edge').remove(); mutable brushed_source = new Set(); mutable corresponding_dest = new Set(); } } svg.call(brush) .call(gr => gr.select(".overlay").style("cursor", "default")); ///////////////////////////////////////////////////////////////////////////////////////// // AXES ///////////////////////////////////////////////////////////////////////////////////////// // create axes from our scales source_g.append('g').attr('id', 'xaxis') .call(d3.axisBottom(source_x_scale).ticks(5)) source_g.append('g').attr('id', 'yaxis') .call(d3.axisLeft(source_y_scale).ticks(5)) // clear domains and tone down colors source_g.selectAll('#xaxis,#yaxis').select('.domain').remove() source_g.selectAll('#xaxis,#yaxis').selectAll('text').attr('fill', d3.hcl(0,0,50)) source_g.selectAll('#xaxis,#yaxis').selectAll('line').attr('stroke', d3.hcl(0,0,50)) // tick guides source_g.select('#xaxis').selectAll('.tick').append('line') .attr('y1', local_height-margin-local_margin).attr('y2', margin) .attr('stroke', d3.hcl(0,0,70)).attr('stroke-width', .5) source_g.select('#yaxis').selectAll('.tick').append('line') .attr('x1', margin).attr('x2', local_width-margin) .attr('stroke', d3.hcl(0,0,70)).attr('stroke-width', .5) // create axes from our scales dest_g.append('g').attr('id', 'xaxis') .call(d3.axisBottom(dest_x_scale).ticks(5)) dest_g.append('g').attr('id', 'yaxis') .call(d3.axisLeft(dest_y_scale).ticks(5)) // clear domains and tone down colors dest_g.selectAll('#xaxis,#yaxis').select('.domain').remove() dest_g.selectAll('#xaxis,#yaxis').selectAll('text').attr('fill', d3.hcl(0,0,50)) dest_g.selectAll('#xaxis,#yaxis').selectAll('line').attr('stroke', d3.hcl(0,0,50)) // tick guides dest_g.select('#xaxis').selectAll('.tick').append('line') .attr('y1', local_height-margin-local_margin).attr('y2', margin) .attr('stroke', d3.hcl(0,0,70)).attr('stroke-width', .5) dest_g.select('#yaxis').selectAll('.tick').append('line') .attr('x1', margin).attr('x2', local_width-margin) .attr('stroke', d3.hcl(0,0,70)).attr('stroke-width', .5) ///////////////////////////////////////////////////////////////////////////////////////// return svg.node() }39:T47c,md`# References [1] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” IEEE Trans. Neural Networks, vol. 20, no. 1, pp. 61–80, 2009, doi: 10.1109/TNN.2008.2005605. [2] W. L. Hamilton, “Inductive Representation Learning on Large Graphs,” no. Nips, pp. 1–19, 2017. [3] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, and M. Bronstein, “Temporal Graph Networks for Deep Learning on Dynamic Graphs,” in ICML, 2020, pp. 1–16. [4] R. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, “GNNExplainer: Generating explanations for graph neural networks,” arXiv, no. iii, 2019. [5] Z. Jin, Y. Wang, Q. Wang, Y. Ming, T. Ma, and H. Qu, “GNNVis : A Visual Analytics Approach for Prediction Error Diagnosis of Graph Neural Networks,” vol. XX, no. Xx, pp. 1–14, 2020. [Michael Bostock's Temporal Force-Directed Graph](https://observablehq.com/@d3/temporal-force-directed-graph?collection=@d3/d3-force) [Scax's Force-Directed Graph with Zooming](https://observablehq.com/@scax/force-directed-graph-with-zoom-and-stroke-scaling-svg) `3a:Tc8a,// plot = { // const simulation = d3.forceSimulation() // .force("charge", d3.forceManyBody()) // .force("link", d3.forceLink().id(d => d.id)) // .force("x", d3.forceX()) // .force("y", d3.forceY()) // .on("tick", ticked); // const svg = d3.create("svg") // .attr("viewBox", [-width / 2, -height / 2, width, height]); // let link = svg.append("g") // .attr("stroke", "#999") // .attr("stroke-opacity", 0.6) // .attr('stroke-widht',1) // .selectAll("line") // function nodeover(d,i) { // let cx = d3.select(this).attr('cx') // let cy = d3.select(this).attr('cy') // d3.select(this) // .attr('fill', d3.hcl(0,0,75)) // .attr('r', 9) // svg.append('text') // .classed('node_id',true) // .attr('x', () => cx) // .attr('y', () => cy) // .attr('font-size',12) // .text(() => d.id.toString()) // } // function nodeoff(d,i) { // d3.select(this).attr('r',5).attr('fill',d3.hcl('black')) // svg.selectAll('.node_id').remove() // } // function linkover(d,i) { // let x1 = d3.select(this).attr('x1') // let x2 = d3.select(this).attr('x2') // let y1 = d3.select(this).attr('y1') // let y2 = d3.select(this).attr('y2') // d3.select(this).attr('stroke-width',6) // let msg = timeframe.edges[i].prob.toString()+'\n' // +timeframe.edges[i].timestamp.toString()+'\n' // +timeframe.edges[i].ground_truth.toString() // svg.append('text') // .classed('edge_msg',true) // .attr('x', () => (x1+x2)/2) // .attr('y', () => (y1+y2)/2) // .attr('font-size',12) // .text(() => msg) // } // function linkoff(d,i) { // d3.select(this).attr('stroke-width',1) // svg.selectAll('.edge_msg').remove() // } // let node = svg.append("g") // .attr("stroke", "#fff") // .attr("stroke-width", 1.5) // .selectAll("circle"); // function ticked() { // node.attr("cx", d => d.x) // .attr("cy", d => d.y); // link.attr("x1", d => d.source.x) // .attr("y1", d => d.source.y) // .attr("x2", d => d.target.x) // .attr("y2", d => d.target.y); // } // invalidation.then(() => simulation.stop()); // return Object.assign(svg.node(), { // update({nodes, links}) { // const old = new Map(node.data().map(d => [d.id, d])); // nodes = nodes.map(d => Object.assign(old.get(d.id) || {}, d)); // links = links.map(d => Object.assign({}, d)); // node = node // .data(nodes, d => d.id) // .join(enter => enter.append("circle") // .attr("r", 5) // .call(drag(simulation))) // .on('mouseover',nodeover) // .on('mouseout',nodeoff); // link = link // .data(links, d => [d.source, d.target]) // .join("line") // .attr('stroke',d => edge_color_scale(d.timestamp)) // .on('mouseover',linkover) // .on('mouseout',linkoff); // simulation.nodes(nodes); // simulation.force("link").links(links); // simulation.alpha(1).restart().tick(); // ticked(); // render now! // } // }); // }3b:[{"id":"a34d2d8c4f1cd2ff8c1109515d9c02e98ab57c98c1aa172b2bfd211e55df158d0f84b2c6f091939a42a0eb5ead349a6af289c2aec977ae50bcba141758e26643","href":"https://static.observableusercontent.com/files/a34d2d8c4f1cd2ff8c1109515d9c02e98ab57c98c1aa172b2bfd211e55df158d0f84b2c6f091939a42a0eb5ead349a6af289c2aec977ae50bcba141758e26643","name":"gnnvis.jpg","type":"image/jpeg","lastModified":1615441923936.06,"size":394368},{"id":"fab0c21f6763456be722916b370ef1bfb45cd3ff140c8367df9de29083ceca5d6e305413b62a8f4471fcd0d24e89b0e606cfff5eaff96d39cc02a2320ac97846","href":"https://static.observableusercontent.com/files/fab0c21f6763456be722916b370ef1bfb45cd3ff140c8367df9de29083ceca5d6e305413b62a8f4471fcd0d24e89b0e606cfff5eaff96d39cc02a2320ac97846","name":"graph_example.jpg","type":"image/jpeg","lastModified":1615441923875.108,"size":103340},{"id":"6c1aed8d7e095650dd53454ba0e6ea220491185351030878a0629d8770dbf95de23371b50c279b809b2fae64a8496f26df26038c8bc9b0db424723e3a92e2a4c","href":"https://static.observableusercontent.com/files/6c1aed8d7e095650dd53454ba0e6ea220491185351030878a0629d8770dbf95de23371b50c279b809b2fae64a8496f26df26038c8bc9b0db424723e3a92e2a4c","name":"equations.jpg","type":"image/jpeg","lastModified":1615441923828.716,"size":68119},{"id":"341904a530e746190a034cb2448fc733f611aa64f638456b87ed34e7f5d618b8656385b8ddfde205067f3d59da96725130bf8d74a94859c435a8e2bea28bace6","href":"https://static.observableusercontent.com/files/341904a530e746190a034cb2448fc733f611aa64f638456b87ed34e7f5d618b8656385b8ddfde205067f3d59da96725130bf8d74a94859c435a8e2bea28bace6","name":"tgn.jpg","type":"image/jpeg","lastModified":1615441923877.026,"size":110311},{"id":"d842dd05684c29d4bcec4f379f74bf2ed1e68f942f686d5d532bcbe3c295beee0e60b55dd72136b852d72125c7ea3156529155cdaba9e9ae773e8181e57fdd0a","href":"https://static.observableusercontent.com/files/d842dd05684c29d4bcec4f379f74bf2ed1e68f942f686d5d532bcbe3c295beee0e60b55dd72136b852d72125c7ea3156529155cdaba9e9ae773e8181e57fdd0a","name":"design_example.jpeg","type":"image/jpeg","lastModified":1615441923930.396,"size":239449}] 30:{"title":"Visualizing Temporal Graph Networks","cells":[{"id":6,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"graph_example_jpg = FileAttachment(\"graph_example.jpg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"equation_jpg = FileAttachment(\"equations.jpg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"tgn_jpg = FileAttachment(\"tgn.jpg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":47,"value":"md`Despite the general, fundamental nature of graphs in computer science, GNNs and TGNs are quite complex, which hinders explainability and therefore makes effective visualization challenging. Additionally, they are not well-vetted due to their recent popularity; only a handful of visualization papers, such as GNNExplainer (Ying et al 2019) [4] and GNNVis (Jin et al 2020) [5], have been published. To our knowledge, no TGN visualization papers have been published yet. At the end of the semester, our project could be the first visualization of TGNs. \n`\n\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":58,"value":"gnnvis_jpg = FileAttachment(\"gnnvis.jpg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":54,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":1079,"value":"md`# Design Concept`","pinned":false,"hidden":false,"mode":"ojs"},{"id":40,"value":"design_example_jpg = FileAttachment(\"design_example.jpeg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":1693,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":1070,"value":"md`# Full Graph View\n*Let the graph settle before brushing (15-30 seconds)*\n- Drag to pan. \n- Scroll / pinch to zoom.\n- Alt (Windows/Linux) / Cmd (Mac) + drag to brush for a subgraph.\n - **Brush & zoom do not coexist well in d3.js; try repeatedly.**\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":466,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":1073,"value":"md`# Subgraph View\n- Drag to pan. \n- Scroll to zoom. \n- Hover over nodes and edges to populate information.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":788,"value":"info_box = {\n\n let w = 200\n let h = 100\n let margin = 20\n let svg = d3.create(\"svg\").attr('width',w+margin).attr('height',h+margin)\n \n svg.append('text').attr('x', 0).attr('y', 17).attr('fill', d3.hcl(0,0,0))\n .attr('font-size', '18px').text('Hover Information')\n\n let g = svg.append('g').attr('transform',`translate(0,${margin})`)\n\n let n = info.length\n\n for (var i = 0 ; i < n; ++i) {\n g.append('text')\n .attr('x',0)\n .attr('y',(h-10)/n * (i+1))\n .attr('font-size', 12)\n .attr('stroke',d3.hcl(0,0,0))\n .text(() => info[i])\n \n }\n\n return svg.node()\n \n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":285,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":1084,"value":"md`# Scatterplot View\nBrush over source nodes to identify destination nodes.\n\n*The corresponding nodes will change color in th subgraph.*\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":253,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":1286,"value":"md`# Config`","pinned":false,"hidden":false,"mode":"ojs"},{"id":785,"value":"mutable info = []","pinned":true,"hidden":false,"mode":"ojs"},{"id":968,"value":"mutable brushed_nodes = []","pinned":true,"hidden":false,"mode":"ojs"},{"id":1095,"value":"mutable brushed_source = new Set()","pinned":true,"hidden":false,"mode":"ojs"},{"id":1393,"value":"mutable corresponding_dest = new Set()","pinned":true,"hidden":false,"mode":"ojs"},{"id":121,"value":"height = 680","pinned":true,"hidden":false,"mode":"ojs"},{"id":425,"value":"margin = 20","pinned":true,"hidden":false,"mode":"ojs"},{"id":1341,"value":"local_margin = 10","pinned":true,"hidden":false,"mode":"ojs"},{"id":339,"value":"local_height = 500","pinned":true,"hidden":false,"mode":"ojs"},{"id":342,"value":"local_width = width / 2 - margin","pinned":true,"hidden":false,"mode":"ojs"},{"id":1289,"value":"md`# Server Endpoints\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":267,"value":"subgraph = {\n \n function success(resp){\n return resp;\n }\n \n return await $.ajax({\n url:'http://localhost:5000/get_subgraph_by_node_id',\n dataType: 'json',\n type: 'POST',\n data: JSON.stringify(Array.from(brushed_nodes)),\n contentType: \"application/json\",\n success: success\n });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":309,"value":"subgraph_nodes = subgraph.nodes.map(d => d.id)","pinned":true,"hidden":false,"mode":"ojs"},{"id":125,"value":"timestamps = {\n return await $.ajax({\n url: 'http://localhost:5000//get_timestamps',\n dataType: 'json',\n contentType: 'application/json;charset=UTF-8',\n type: 'GET'\n })\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":246,"value":"graph = {\n\n return await $.ajax({\n url:'http://localhost:5000/get_graph',\n dataType: 'json',\n type: 'GET',\n contentType: \"application/json;charset=UTF-8\",\n });\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":325,"value":"tsne = {\n function success(resp){\n return resp;\n }\n\n if (subgraph_nodes.length > 0) {\n \n return await $.ajax({\n url:'http://localhost:5000/perform_tsne2',\n dataType: 'json',\n type: 'POST',\n data: JSON.stringify(subgraph_nodes),\n contentType: \"application/json\",\n success: success\n });\n }\n\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":412,"value":"source_data = tsne.source_nodes.map( (d,i) => {\n let ret = new Object()\n ret.node = d\n ret.x = tsne.source_x[i]\n ret.y = tsne.source_y[i]\n return ret\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":415,"value":"dest_data = tsne.dest_nodes.map( (d,i) => {\n let ret = new Object()\n ret.node = d\n ret.x = tsne.dest_x[i]\n ret.y = tsne.dest_y[i]\n return ret\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1292,"value":"md`# Scales`","pinned":false,"hidden":false,"mode":"ojs"},{"id":329,"value":"dest_x_ext = d3.extent(tsne.dest_x)","pinned":true,"hidden":false,"mode":"ojs"},{"id":366,"value":"dest_x_scale = d3.scaleLinear().domain(dest_x_ext).range([margin, local_width-margin])","pinned":true,"hidden":false,"mode":"ojs"},{"id":331,"value":"dest_y_ext = d3.extent(tsne.dest_y)","pinned":true,"hidden":false,"mode":"ojs"},{"id":370,"value":"dest_y_scale = d3.scaleLinear().domain(dest_y_ext).range([local_height-margin-local_margin, margin])","pinned":true,"hidden":false,"mode":"ojs"},{"id":332,"value":"source_x_ext = d3.extent(tsne.source_x)","pinned":true,"hidden":false,"mode":"ojs"},{"id":372,"value":"source_x_scale = d3.scaleLinear().domain(source_x_ext).range([margin, local_width-margin])","pinned":true,"hidden":false,"mode":"ojs"},{"id":334,"value":"source_y_ext = d3.extent(tsne.source_y)","pinned":true,"hidden":false,"mode":"ojs"},{"id":374,"value":"source_y_scale = d3.scaleLinear().domain(source_y_ext).range([local_height-margin-local_margin,margin])","pinned":true,"hidden":false,"mode":"ojs"},{"id":171,"value":"prob_ext = d3.extent(graph.edges, d => d.prob)","pinned":true,"hidden":false,"mode":"ojs"},{"id":321,"value":"timestamps_ext = d3.extent(timestamps)","pinned":true,"hidden":false,"mode":"ojs"},{"id":355,"value":"edge_color_scale = d3.scaleLog().domain(timestamps_ext).range([d3.hsl(106,.2, .3),d3.hsl(106,1,.6)])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1491,"value":"edge_opacity_scale = d3.scaleLinear().domain(prob_ext).range([0.1,1])","pinned":true,"hidden":false,"mode":"ojs"},{"id":62,"value":"$39","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"d3 = require('d3@6')","pinned":true,"hidden":false,"mode":"ojs"},{"id":97,"value":"import {drawdom} from '91007ee9d5fd152b'","pinned":true,"hidden":false,"mode":"ojs"},{"id":67,"value":"import {vl} from '@vega/vega-lite-api'","pinned":true,"hidden":false,"mode":"ojs"},{"id":73,"value":"import { jQuery as $ } from \"@ddspog/useful-libs\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":69,"value":"import {Button, Checkbox, Toggle, Radio, Range, Select, Text, Textarea, Search, Table} from \"@observablehq/inputs\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":118,"value":"import {Scrubber} from \"@mbostock/scrubber\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":649,"value":"md`# Backup Code`","pinned":false,"hidden":false,"mode":"ojs"},{"id":135,"value":"// drag = simulation => {\n \n// function dragstarted(event,d) {\n// if (!event.active) simulation.alphaTarget(0.3).restart();\n// d.fx = event.subject.x;\n// d.fy = event.subject.y;\n// }\n \n// function dragged(event,d) {\n// d.fx = event.x;\n// d.fy = event.y;\n// }\n \n// function dragended(event,d) {\n// if (!event.active) simulation.alphaTarget(0);\n// d.fx = null;\n// d.fy = null;\n// }\n \n// return d3.drag()\n// .on(\"start\", dragstarted)\n// .on(\"drag\", dragged)\n// .on(\"end\", dragended);\n// }","pinned":true,"hidden":false,"mode":"ojs"},{"id":123,"value":"// viewof time = Scrubber(timestamps, {\n// delay: 100, \n// loop: true,\n// }) // TIMESTAMP SCRUBBER","pinned":false,"hidden":false,"mode":"ojs"},{"id":80,"value":"$3a","pinned":true,"hidden":false,"mode":"ojs"},{"id":129,"value":"// update = {\n// let nodes = timeframe.nodes\n// let links = timeframe.edges\n// plot.update({nodes,links});\n// }","pinned":false,"hidden":false,"mode":"ojs"},{"id":438,"value":"// tsne1 = {\n// function success(resp){\n// return resp;\n// }\n \n// return await $.ajax({\n// url:'http://localhost:5000/perform_tsne1',\n// dataType: 'json',\n// type: 'POST',\n// data: JSON.stringify(timeframe_nodes),\n// contentType: \"application/json\",\n// success: success\n// });\n\n// }","pinned":true,"hidden":false,"mode":"ojs"},{"id":88,"value":"// timeframe = {\n// function success(resp){\n// return resp;\n// }\n \n// return await $.ajax({\n// url:'http://localhost:5000/get_timeframe',\n// dataType: 'json',\n// type: 'POST',\n// data: JSON.stringify(timestamps.indexOf(time)),\n// contentType: \"application/json\",\n// success: success\n// });\n\n// }","pinned":true,"hidden":false,"mode":"ojs"},{"id":319,"value":"// timeframe_nodes = timeframe.nodes.map(d => d.id)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1100,"value":"// SCATTER HOVERING\n// hovering interaction\n // function mouseover_response(d,i) {\n // d3.select(this)\n // .attr('fill', d3.hcl('blue'))\n // d3.select(this).attr('r',6)\n \n // svg.append('text')\n // .classed('node_id',true)\n // .attr('x', d3.select(this).attr('cx'))\n // .attr('y', d3.select(this).attr('cy') - 5)\n // .attr('font-size',12)\n // .text(() => d.node)\n // }\n\n // function destover_response(d,i) {\n // d3.select(this)\n // .attr('fill', d3.hcl('blue'))\n // d3.select(this).attr('r',6)\n \n // dest_g.append('text')\n // .classed('node_id',true)\n // .attr('x', d3.select(this).attr('cx'))\n // .attr('y', d3.select(this).attr('cy') - 5)\n // .attr('font-size',12)\n // .text(() => d.node)\n // }\n \n // function mouseout_response(d,i) {\n // d3.select(this).attr('fill', d3.hcl(0,0,75))\n // d3.select(this).attr('r',4)\n // svg.selectAll('.node_id').remove()\n // }","pinned":true,"hidden":false,"mode":"ojs"},{"id":257,"value":"viewof id = Text({label: \"Enter NodeID\", placeholder: \"0\", submit: true, multiple: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":262,"value":"selected_nodes = new Set()","pinned":true,"hidden":false,"mode":"ojs"},{"id":263,"value":"selected_nodes.add(id)","pinned":true,"hidden":false,"mode":"ojs"},{"id":100,"value":"viewof plot_view = drawdom(plot,5)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1481,"value":"viewof subgraph_view = drawdom(plot_subgraph,5)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1483,"value":"viewof scatter_view = drawdom(scatterplot,5)","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W3b","stdlib":"1","titling":"auto"}