Using three.js for 2D data visualization This notebook was created for the Cloudera Fast Forward Labs Blog. I'd love to hear any comments on twitter @grantcuster or grantcuster at gmail dot com. This notebook shows the results of my efforts to use three.JS to make a 2D data visualization. I was making an interactive t-SNE visualization, but these methods will be useful for anyone who wants to make an interactive visualization of a lot of points (in my project it was over 50,000). This demo is only using 1000 points, but you can check out my working Codepen demo featuring 100,000. I used three.js because it can render more points more smoothly than SVG or canvas. Because three.js is not designed with these types of visualizations in mind, there was some friction in getting everything set up the way I wanted. This notebook documents my finished set-up and solutions to the issues I ran into. The main features I'm going to highlight are: Mouse-directed zoom and pan (using d3-zoom) Rendering circular points (three.js defaults to squares) Coloring points Displaying a tooltip when you hover over a point I show the finished interactive visualization below and then walk through the code that makes it work. If you really want to dig in and modify the code. I'd recommend forking the notebook so you can take advantage of code changes syncing across browser windows.
The visualization
{ renderer.setSize(viz_width, height); renderer.setPixelRatio(devicePixelRatio); try { while (true) { renderer.render(scene, camera); let html_render = html`<div style="position: relative; overflow: hidden;"> ${renderer.domElement} <div style="display: ${tooltip_state.display}; position: absolute; pointer-events: none; left: ${tooltip_state.left}px; top: ${tooltip_state.top}px; font-size: 13px; width: 120px; text-align: center; line-height: 1; padding: 6px; background: white; font-family: sans-serif;"><div style="padding: 4px; margin-bottom: 4px;">${tooltip_state.name}</div><div style="padding: 4px; background: ${color_array[tooltip_state.group]}">Group ${tooltip_state.group}</div> </div>` yield html_render; } } finally { renderer.dispose(); } }
0. Camera and render set-up Before we get into zoom, we should briefly talk about how this visualization exists in the three.js world. Objects in three.js exist in a three-dimensional space with x, y, and z axes. Your view of those points is determined by the camera, which also exists in three-dimensional space. This allows a bunch of stuff like zooming past 3D objects, or rotating around them, that we don't need for this 2D visualization. What I want to do is put all my points at 0 on the z axis and then pull my camera back so I can see them all. Zoom then involves moving the camera closer to the points. Besides the camera's position, you also need to set certain parameters, like field of view angle (fov), aspect ratio, and the near and far plane thresholds, on set-up. I made another notebook breaking down what those settings mean. We're also going to use the fov, near and far plane thresholds in our zoom handler so I've broken those out into their own cells.
fov = 40
near = 10
far = 100
camera = { let aspect = viz_width / height; return new THREE.PerspectiveCamera(fov, aspect, near, far + 1); }
Note: The far + 1 adjustment is because I was having issues with the hovered points not rendering when zoomed out. Theoretically it should not have been an issue, but since adding 1 to the threshold was a simple fix that doesn't hurt anything I didn't dig any further into it.
You'll also need to make a scene and add your points (which we'll define later) to it
scene = { let scene = new THREE.Scene(); scene.background = new THREE.Color(0xefefef); scene.add(points); return scene; }
Finally, you'll need to set up a rendering loop and a canvas element (for this demo that code is behind the visualization block). The three.js docs are a good resource on how to do that. There are no 2D specific adjustments you need to make for the render step so I'm not going to discuss it here.
1. Mouse-directed zoom and pan To get zoom and pan working I needed something to translate mouse and touch events (like scrolling and pinch-to-zoom) into zoom and movement values. I used d3's zoom behavior for this because I was familiar with it from past projects and I knew it had a feel I liked (in this case "feel" is how fast it zooms in response to mouse events). To get d3-zoom to work I needed to translate d3-zoom terms into three.js terms. d3-zoom starts with a scale of 1 and increases that scale as you zoom in (this works out of the box for svg and canvas transforms). As discussed above, in three.js the scale is determined by the z position of the camera. I needed a way to map d3's scale to three.js's camera position. I found the solution in the code for the three.map.control plugin, which showed how to use the field of view angle, the camera position, and trigonometry to find the current scale of the visualization.
function getScaleFromZ (camera_z_position) { let half_fov = fov/2; let half_fov_radians = toRadians(half_fov); let half_fov_height = Math.tan(half_fov_radians) * camera_z_position; let fov_height = half_fov_height * 2; let scale = height / fov_height; // Divide visualization height by height derived from field of view return scale; }
I go into detail about the math behind this in my other notebook. We'll also need a function that does the reverse: translating from d3's scale value into a camera z position.
function getZFromScale(scale) { let half_fov = fov/2; let half_fov_radians = toRadians(half_fov); let scale_height = height / scale; let camera_z_position = scale_height / (2 * Math.tan(half_fov_radians)); return camera_z_position; }
With those two functions in hand, we're ready to make our zoom handler. It takes the transform supplied by d3-zoom, which includes x, y, and k (scale) values and transforms them into a coordinates for the three.js camera. Crucially, the x and y coordinates get us mouse directed zoom, where the zoom follows where the mouse is pointed (you're probably familiar with this from google maps), often I see this intuitive feature left out of zoom set-ups. The x and y settings are automatically used for click-and-drag panning as well. We find the z value from the scale using the function above. We have to make some additional adjustments to the x and y values to translate d3's x and y values, which locate (0,0) at the top left, to three.js's, which put (0,0) in the middle of the visualization. We also divide the x and y values by the scale to remove the scale adjustment built into the d3 transform.
function zoomHandler(d3_transform) { let scale = d3_transform.k; let x = -(d3_transform.x - viz_width/2) / scale; let y = (d3_transform.y - height/2) / scale; let z = getZFromScale(scale); camera.position.set(x, y, z); }
We place our zoomHandler within d3 zoom handler. Here we also set the scaleExtent for d3.zoom using the values from our near and far plane settings translated from camera positions to scale by our getScaleFromZ function. Really you don't need these to eactly equal each other, you just need to make sure your thresholds contain the scale extent (otherwise your points will disappear when you cross the threshold), but it feels nice to have them exactly synced up.
zoom = { let d3_zoom = d3.zoom() .scaleExtent([getScaleFromZ(far), getScaleFromZ(near)]) .on('zoom', () => { let d3_transform = d3.event.transform; zoomHandler(d3_transform); }); return d3_zoom; }
After you have your d3-zoom behavior all set up you attach it to the view DOM element. The last thing you need to do (which tripped me up for a bit) is sync the initial position of d3 zoom transform and the camera position. This means setting their scale and z position to be equivalent, and offsetting d3's x and y position to compensate for three.js's different coordinate system.
view = d3.select(renderer.domElement)
{ function setUpZoom() { view.call(zoom); let initial_scale = getScaleFromZ(far); var initial_transform = d3.zoomIdentity.translate(viz_width/2, height/2).scale(initial_scale); zoom.transform(view, initial_transform); camera.position.set(0, 0, far); } setUpZoom(); return setUpZoom; }
With that, you've got a mouse-directed zoom and pan set-up that works across desktop and mobile.
2. Rendering circular points A less math-y, but still important, issue was getting three.js to render points as circles, rather than the default squares. From three.js examples, it was clear I needed to use a sprite to get a circle render. The main challenge was figuring out what kind of sprite would get the best looking circle across different colors. A lot of the examples used sprites with drop shadows. I did not want a drop shadow so I made a simple flat white circle (the white will take the colors we assign below). The biggest issue was determining how to handle the edge. By default photoshop will use anti-aliasing on the edges of a circle to smooth things out. This caused issues for me when I assigned colors, so I turned antialiasing off. An alternative suggested solution is to use a sprite with anti-aliasing and set the alphaTest property to around 0.5. I included both approaches below. You can switch the config used in sprite_settings and check the visualization to see the difference (I have a hard time seeing any).
Circle: anti-aliasing off
html`<div style="background: #222; padding: 20px"> <image src="http://blog.fastforwardlabs.com/images/2018/02/circle-1518727951930.png" /></div>`
circle_sprite= new THREE.TextureLoader().load( "https://blog.fastforwardlabs.com/images/2018/02/circle-1518727951930.png" )
Circle: antia-aliasing on
html`<div style="background: #222; padding: 20px"> <image src="http://blog.fastforwardlabs.com/images/2018/02/circle_aa-1518730700478.png" /></div>`
circle_sprite_aa= new THREE.TextureLoader().load( "https://blog.fastforwardlabs.com/images/2018/02/circle_aa-1518730700478.png" )
sprite_settings = { // Non anti-aliased settings let circle = { map: circle_sprite, transparent: true } // Anti-aliased settings let circle_aa = { map: circle_sprite_aa, transparent: true, alphaTest: 0.5 } // Switch out which settings object is returned to change settings return circle; }
After a bit of experimentation, I settled on 256x256 for the size of the sprite. I'd love to know more about how different sizes affect appearance or peformance. I think the sprites in the final visualization turned out pretty well but sometimes I still feel the edges look a bit rough on high-resolution screens.
3. Coloring points In three.js, you create a Points object using pointsGeometry and pointsMaterial. To color points you set an array of colors on pointsGemoetry and you set the vertexColors property on the pointsMaterial. Below is the code for setting up the points for the visualiztion. I'm using fake generated data (generated_data below) in this example.
points = { let pointsGeometry = new THREE.Geometry(); let colors = []; for (let datum of generated_points) { // Set vector coordinates from data let vertex = new THREE.Vector3(datum.position[0], datum.position[1], 0); pointsGeometry.vertices.push(vertex); let color = new THREE.Color(color_array[datum.group]); colors.push(color); } pointsGeometry.colors = colors; let pointsMaterial = new THREE.PointsMaterial({ size: 8, sizeAttenuation: false, vertexColors: THREE.VertexColors, }); // Add settings from sprite_settings for (let setting in sprite_settings) { pointsMaterial[setting] = sprite_settings[setting]; } return new THREE.Points(pointsGeometry, pointsMaterial); }
In this example I'm coloring my data based on their group. The group is an integer from 0-6, I'm using that to select from a premade array of colors and pushing the color for each item into the colors array, which I then set on pointsGeometry. It took me a while to get used to the three.js specific syntax (like remembering which things I set on geometry and which on material), but, overall, coloring by category was relatively straightforward.
A note on sizeAttenuation In the above code for pointsMaterial I've set sizeAttenuation to false. When set to true (the default value) sizeAttenuation rescales the points based on the camera's z-position so that they appear to get bigger as you get closer to them. This makes sense for a simulation of real objects, but for data visualizations it's often preferable to have the points remain the same size. It gives you a better view of global structure when zoomed out and local structure when zoomed in. When I've done similar, non-three.js, visualizations I've achieved the same effect by rescaling the points with zoom to make them stay the same size. In three.js you basically get that for free. There's an open issue on whether the default should be false that also has more info on just what is happening. Turning sizeAttenuation off can make zooming in close feel a little weird – since the points spread out it can feel like the one you were hovering over is running away from you. Partly for that reason I think the best user experience is to have the points stay mostly the same size but rescale to get a bit bigger as you zoom in very close. I set that up in the project this is based on but am leaving it out of this example. Basically on zoom you look at the camera z position and choose certain thresholds and size targets. Then you can set up functions to tween those size targets on zoom. I first saw this technique in Mapbox GL where they have it set up so you can determine the size of labels at different zoom levels. I think there is a lot of this going on in maps to make things feel smooth.
4. Hover interaction To add hover interaction to the points in our visualization, we first need to figure out when we're hovering over a point. Since we're rendering on a canvas element, the points are not DOM elements and don't have their own mouse events. If you were doing your own custom canvas thing, you'd have to devise a way to keep track of zoom and object position and then compare that to the mouse position (an interesting but time-consuming route). Thankfully, three.js has a built-in method that takes care of things for us. The idea is to project a ray from the camera through the mouse and find which objects it intersects with. This evidently comes up often enough that the three.js maintainers made a class for it called Raycaster. To get started we create a raycaster we can reuse and attach a function to the view to check for intersects whenever the mouse moves over the visualization.
raycaster = new THREE.Raycaster()
{ view.on("mousemove", () => { let [mouseX, mouseY] = d3.mouse(view.node()); let mouse_position = [mouseX, mouseY]; checkIntersects(mouse_position); }); return view; }
To get the intersect we first tramslate the d3 mouse coordinates into a three.js vector.
function mouseToThree(mouseX, mouseY) { return new THREE.Vector3( mouseX / viz_width * 2 - 1, -(mouseY / height) * 2 + 1, 1 ); }
The raycaster takes the mouse vector and the camera and returns a list of intersects. We then sort those intersects by the property distanceToRay. This is a gotcha I'll talk more about below. The first object in the sorted intersect array is the point we're hovering over. We get the index from that point and look up the data for it by using the index to select the relevant object from our original points data set. An alternative approach would be to add the relevant fields to the three.js points when you create them. See this StackOverflow discussion for more information on that.
function checkIntersects(mouse_position) { let mouse_vector = mouseToThree(...mouse_position); raycaster.setFromCamera(mouse_vector, camera); let intersects = raycaster.intersectObject(points); if (intersects[0]) { let sorted_intersects = sortIntersectsByDistanceToRay(intersects); let intersect = sorted_intersects[0]; let index = intersect.index; let datum = generated_points[index]; highlightPoint(datum); showTooltip(mouse_position, datum); } else { removeHighlights(); hideTooltip(); } }
A note on ray point threshold and sorting by distanceToRay After I got my ray working I was stumped by an issue where the first intersect object was not the one directly under my mouse. If I highlighted all the points returned as intersections they formed an area defined around my mouse position, but much bigger to it. Searching led me to this StackOverflow answer which suggested adjusting the point threshold property on the raycaster params to match the size of your points (it defaults to 1).
{ raycaster.params.Points.threshold = 1; return raycaster; }
Changing the point threshold definitely had some effect (making it bigger made the threshold bigger) but setting it to match the size of the points did not mean it only triggered the intersects when you were directly over a point. It seems like the threshold scales with the camera position, while my points, because I've set sizeAttenuation to false, do not. Anyway, it turned out to be kind of a moot point because I figured out I could sort the intersects by the property distanceToRay. After sorting the first point is the one nearest the mouse. The fact that the ray is larger than the points turns into a feature: it means that when I'm near a point I see a tooltip, rather than having to be directly over the point. Since the points in the visualization are pretty small this is more convenient for the user. Still, it would be nice to figure out exactly how the threshold interacts with zoom so you could have precise control over the interaction (please let me know if you do).
function sortIntersectsByDistanceToRay(intersects) { return _.sortBy(intersects, "distanceToRay"); }
4.1 Enlarging a point on hover Soon we're going to add a tooltip to on hover, but it's also good to add some indicator to the point itself that it's registered the hover. If we had CSS at our disposal we could do something like add a black border on with the :hover selector. We are far from CSS, however. In fact, we don't want to change the point we've already rendered at all. Instead we want to draw a new point (with the style we want) directly on top of the point we're hovering over. For me, getting used to three.js was a lot about getting used to the idea that is often easier to draw a completely new thing than modify an existing one. To handle the points we add on hover we're going to create an Object3D to use as a container for them and add that to the scene. This makes it easier to keep track of and clear the objects we add on hover.
hoverContainer = new THREE.Object3D()
{ scene.add(hoverContainer); return scene; }
When we hover over a point we create a new point and add it to hoverContainer. This point is styled in the same way as the existing points, just at a larger size. You don't have to use a point object here, in the prototype this code is based on I used a mesh object to show a book cover when you hovered over a point.
function highlightPoint(datum) { removeHighlights(); let geometry = new THREE.Geometry(); geometry.vertices.push( new THREE.Vector3( datum.position[0], datum.position[1], 0 ) ); geometry.colors = [ new THREE.Color(color_array[datum.group]) ]; let material = new THREE.PointsMaterial({ size: 26, sizeAttenuation: false, vertexColors: THREE.VertexColors }); // Add settings from sprite settings for (let setting in sprite_settings) { material[setting] = sprite_settings[setting]; } let point = new THREE.Points(geometry, material); hoverContainer.add(point); }
One nice thing about this method compared to doing similar things with CSS and the DOM is that there you often have issues where the enlarged hover object intercepting your hover and blocking it from reaching nearby objects. Here you are only checking for intersections on the original points, not anything you draw on hover. To clear objects when you un-hover, we just use the hoverContainer's remove function. You run it every time you run highlight point to clear the highlights from previous runs.
function removeHighlights() { hoverContainer.remove(...hoverContainer.children); }
You also want to be sure to remove highlights when the mouse leaves the visualization.
{ view.on("mouseleave", () => { removeHighlights() }) return view; }
4.2 Hover tooltip To add the tooltip we step out of the world of three.js and back into HTML, appending a div over the visualization and using CSS absolute positioning to pin it to the mouse. This is the approach the three.js docs recommend, I think because working with text in three.js is difficult. In my project I was using React, so I passed the relevant info (mouse position and point data) up into the containing component and used that to set the state for the HTML div. The code below does a similar thing but is a more minimal technique designed just to work in the notebook (note the default div styling is set up in the code block for the visualization at the top of the page). The important thing to note is how to get the information from the point, after that you can set it in React of JQuery or any way you like.
tooltip_state = { return { display: "none" } }
function showTooltip(mouse_position, datum) { let tooltip_width = 120; let x_offset = -tooltip_width/2; let y_offset = 30; tooltip_state.display = "block"; tooltip_state.left = mouse_position[0] + x_offset; tooltip_state.top = mouse_position[1] + y_offset; tooltip_state.name = datum.name; tooltip_state.group = datum.group; }
function hideTooltip() { tooltip_state.display = "none"; }
5. Extensions and caveats 5.1 Click on a point to zoom I figured this example is long enough, but I did want to point the way towards another common feature for interactive visualizations: clicking on a point to zoom (and probably show more detail information in a sidebar). Basically all the components for that are in this example. You'll use raycaster to find which point is clicked, and then you'll want to use a library like tween.js to animate the camera's zoom in. There are several three.js + tween.js guides out there like this one. 5.2 GeometryBuffer I set the geometry in for the points in this example using the Geometry class. For better performance the three.js docs recommend using BufferGeometry, which requires you to be a bit more specific in how you set it up. I used BufferGeometry in my final project but I'm not going to go into it here. You can always start with Geometry and make the move to BufferGeometry as part of your optimizations. 5.3 renderOrder Probably the biggest issue I had that I don't feel like I really ever got my head around has to do with renderOrder. three.js does not guarantee that the objects will render in the same order they are added to the scene. This can cause issues when you add objects on top of other objects (like the enlarged points I render on hover). I don't have any issues with this example, but as I was doing more complex things in a project I did run into problems. There is a method, suggested by WestLangley here to make sure an object always renders on top: mesh.renderOrder = 999;mesh.onBeforeRender = function( renderer ) { renderer.clearDepth(); }; I got this to work in some situations, but ran into trouble when I was working with objects within different containers. I found workarounds for the specific case, but I would love to look at (and will perhaps sometimes make) a reduced test case of how and when it works.
6. Wrapping up I hope this was enough to get people started who, like me, are moving into three.js from other visualization frameworks. I think interest in what machine learning systems are doing is going to continue to grow, and that by developing better interfaces for visualizing it we can increase understanding and open doors to human-machine collaboration. I hope this helps people develop those visualizations and interfaces. Also a last big thank you to all the people who work on three.js and d3 and answer questions about them on StackOverflow.
– Grant (@grantcuster, grantcuster at gmail dot com)
7. The rest of the code
function toRadians (angle) { return angle * (Math.PI / 180); }
mutable viz_width = Math.min(width, 700)
height = 350
generated_points = { let radius = 25; // Random point in circle code from https://stackoverflow.com/questions/32642399/simplest-way-to-plot-points-randomly-inside-a-circle function randomPosition(radius) { var pt_angle = Math.random() * 2 * Math.PI; var pt_radius_sq = Math.random() * radius * radius; var pt_x = Math.sqrt(pt_radius_sq) * Math.cos(pt_angle); var pt_y = Math.sqrt(pt_radius_sq) * Math.sin(pt_angle); return [pt_x, pt_y]; } let data_points = []; for (let i = 0; i < point_num; i++) { let position = randomPosition(radius); let name = 'Point ' + i; let group = Math.floor(Math.random() * 6); let point = { position, name, group }; data_points.push(point); } return data_points; }
point_num = 1000
color_array = [ "#1f78b4", "#b2df8a", "#33a02c", "#fb9a99", "#e31a1c", "#fdbf6f", "#ff7f00", "#6a3d9a", "#cab2d6", "#ffff99" ]
renderer = new THREE.WebGLRenderer({antialias: true})
THREE = require("three@0.89.0/build/three.min.js")
d3 = require("https://d3js.org/d3.v5.min.js")
_ = require('https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.5/lodash.min.js')