29:null 2a:["$","title",null,{"children":"A Comparative Overview of Dimension Reduction Methods | Chris North | Observable"}] 2f:T56c,## Principal Component Analysis Principal Component Analysis (PCA) is one of the most popular dimension reduction methods that is based on the idea of principal components. It simplifies complexity in high-dimensional data while preserving trends and patterns. It does this by transforming the data into fewer dimensions, which act as summaries of features. Each principal component (PC) captures as much information as possible about the dataset based on the variance in the original dimension, allowing us to reduce the data into lower dimensions while minimizing information loss. PCA reduces data by geometrically projecting them onto lower dimensions called principal components, with the goal of finding the best summary of the data using a limited number of PCs. The first PC is chosen to minimize the total distance between the data and their projection onto the PC. The subsequent components are selected in a similar fashion, with the requirement that they have no correlation with all previous PCs. ### 1. Standardize the data The first step of PCA is to standardize the data. Standardized data has an average value of 0 and a standard deviation of 1. Since PCA is sensitive to variance in the features, it is important to standardize the data to ensure that all features are on the same scale. Otherwise, features with large ranges end up dominating those with small ranges.30:T7e5,{ const svg = d3.create("svg") .attr("font-size", "11pt") .attr("viewBox", [0, 0, width, 440]); yield svg.node(); const table = new SVGTable(svg) .size([width, 350]) .fixedRows(0) .fixedColumns(1) .defaultNumberFormat(",.2f") .defaultColumnWidth(cov_grid_size) .style({ border: false, highlight: "cell", textColor: "black", highlightBackground: "#FFE720" }) .data(corr_matrix) .heatmapPalette(cov_color_scale) .heatmap(true) .onhighlight((e, context) => { update(context); }) .render(); var caption1 = svg.append("text") .attr("x", 0) .attr("y", 360+pad) var caption2 = svg.append("text") .attr("x", 0) .attr("y", 380+pad) var caption3 = svg.append("text") .attr("x", 0) .attr("y", 400+pad) caption1.text(`Hover over a cell to get some more information!`); function update(context) { const row = features[context.cell.rowIndex]; const col = context.column.name; const cov = context.cell.value.toFixed(2); const same_feature = (col === row); const pos_neg_text = (cov > 0) ? `The positive correlation indicates that as the ${row} value increases, ${col} increases.` : `The negative corrrelation indicates that as the ${row} value increases, ${col} decreases.`; var magnitude_text = (cov > .7 | cov < -.7) ? `The large magnitude of ${cov} indicates that the relationship between ${row} and ${col} is strong.` : `The small magnitude of ${cov} indicates that the relationship between ${row} and ${col} is weak.`; if (same_feature) { magnitude_text = `Comparing the same feature to itself yields a perflectly positive relationship because a feature is linear with itself.` } caption1.text(`${row} and ${col} have a correlation of ${cov}.`); caption2.text(`${pos_neg_text}`); caption3.text(`${magnitude_text}`); } svg.selectAll("text").style("font-family", "Verdana") return svg.node(); }31:T65f,### 3. Identify the principal components Using the correlation matrix, PCA then finds the principal components of the data. Principal components can be thought of as new features of the data that are constructed as linear combinations of the initial features. These linear combinations are done in such a way that the principal components are uncorrelated and most of the information about the data is compressed into the first few components. In other words, principal components represent the directions of the data that capture the maximum amount of variance. We find principal components by taking the eigenvectors and eigenvalues of the correlation matrix. Each eigenvector represents the direction of the axes where there is the most variance, and the associated eigenvalue represents the amount of variance in that principal component. ### 4. Select the principal components. By ranking the eigenvectors in order of their eigenvalues, highest to lowest, we get the principal components in order of significance of the amount of variance each one holds. Since each principal component is an axis of the data, we can choose to keep only the first p components, where p is the number of dimensions of the final projection, in order to preserve as much information as possible while reducing the number of dimensions. We then create a matrix of the eigenvectors of the selected components called the feature vector. ### 5. Recast the data along the new axes. The final step is to reorient the data along the new axes. This is done by multiplying the transpose of the original dataset by the transpose of the feature vector. 32:T18b5,pca = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width + pad*8; canvas.height = indiv_width; // SVG STUFF var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); const whole_group = svg.append("g"); let [x, y] = await new Promise(res => res(get_scales(animals_PCA_Y, indiv_width))); const selected_color = '#FFE720' const deselected_color = '#808080' const gs = whole_group.selectAll("point") .data(animals_PCA_Y) .enter() .append("g") .attr("class", "point") .attr("fill", (d, i) => color(classes[i])) .attr("stroke", (d, i) => color(classes[i])) .attr("name", (d, i) => classes[i]) .attr("original_color", (d, i) => color(classes[i])) // .on('click', (event, d) => { // const selected = d3.select(event.currentTarget) // // If deselecting a point // if (selected.attr("selected") == "true") { // svg.selectAll(".point").attr("selected", false) // .style('fill', d => d.original_color) // .style('stroke', d => d.original_color); // svg.selectAll("text").attr("selected", false) // .style('fill', d => d.original_color) // .style('stroke', d => d.original_color); // } else { // // If selecting a point // svg.selectAll(".point").attr("selected", false) // .style('fill', d => deselected_color) // .style('stroke', d => deselected_color); // //color of labels // svg.selectAll("text").attr("selected", false) // .style('fill', d => deselected_color) // .style('stroke', d => deselected_color); // selected.attr("selected", true) // .style('fill', selected_color) // .style('stroke', 'black') // .raise(); // //color of labels // selected.selectAll("text").attr("selected", true) // .style('fill', "black") // .style('stroke', "black"); // } // }); gs.append("text") .attr("x", 0) .attr("y", -10) .style("font-size", 12) .style("font-family", "Verdana") .text((d, i) => { return classes[i] }) gs.append("path") .attr("d", (d, i) => circle) svg.selectAll(".point") .data(animals_PCA_Y) .attr("transform", (([px, py], i) => `translate(${x(px)}, ${y(py)})`)) .attr("id", "points") return container.node() } /* mds = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width + pad*4; canvas.height = indiv_width; var MDS_coords = druid.Matrix.from(ld_coords); // SVG STUFF var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); let [x, y] = await new Promise(res => res(get_scales(MDS_coords, indiv_width))); var heat = simpleheat(canvas); // set point radius and blur radius (25 and 15 by default) heat.radius(25, 20); heat.gradient({0.33: 'white', 0.66: 'lightblue', 1: '#6bb8d2'}); heat.max(1); var selected_point_heatmap = [] const selected_color = '#FFE720' const deselected_color = '#808080' const gs = svg.selectAll("point") .data(MDS_coords) .enter() .append("g") .attr("class", "point") .attr("fill", (d, i) => color(classes[i])) .attr("stroke", (d, i) => color(classes[i])) .attr("name", (d, i) => classes[i]) .attr("original_color", (d, i) => color(classes[i])) .on('click', (event, d) => { const selected = d3.select(event.currentTarget) // If deselecting a point if (selected.attr("selected") == "true") { svg.selectAll(".point").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); svg.selectAll("text").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); heat.clear(); heat.draw(); } else { // If selecting a point heat.clear(); svg.selectAll(".point").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); //color of labels svg.selectAll("text").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); selected.attr("selected", true) .style('fill', selected_color) .style('stroke', 'black') .raise(); //color of labels selected.selectAll("text").attr("selected", true) .style('fill', "black") .style('stroke', "black"); selected_point_heatmap = get_mds_heatmap( selected.attr('name'), MDS_coords, x, y); const stress_list = selected_point_heatmap.map(function(sublist) { return sublist[2]; }); const max = Math.max(...stress_list); const min = Math.min(...stress_list); selected_point_heatmap.map(function(sublist) { sublist[2] = normalize(sublist[2], max, min); }); heat.data(selected_point_heatmap); heat.draw(); } }) gs.append("text") .attr("x", 0) .attr("y", -10) .style("font-size", 14) .style("font-family", "Verdana") .text((d, i) => { return classes[i] }); gs.append("path") .attr("d", (d, i) => circle); svg.selectAll(".point") .data(MDS_coords) .attr("transform", (([px, py], i) => `translate(${x(px)}, ${y(py)})`)) .attr("id", "points") return container.node() } */33:Te36,mds = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width + pad*8; canvas.height = indiv_width; var MDS_coords = druid.Matrix.from(ld_coords); // SVG STUFF var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); let [x, y] = await new Promise(res => res(get_scales(MDS_coords, indiv_width))); var heat = simpleheat(canvas); // set point radius and blur radius (25 and 15 by default) heat.radius(25, 20); heat.gradient({0.33: 'white', 0.66: 'lightblue', 1: '#4AABCD'}); // heat.gradient({0.25: 'white', 0.50: '#9ED9EE', 0.75: '#87CBE3', 1: '#4AABCD'}); heat.max(1); var selected_point_heatmap = [] const selected_color = '#FFE720' const deselected_color = '#808080' const gs = svg.selectAll("point") .data(MDS_coords) .enter() .append("g") .attr("class", "point") .attr("fill", (d, i) => color(classes[i])) .attr("stroke", (d, i) => color(classes[i])) .attr("name", (d, i) => classes[i]) .attr("original_color", (d, i) => color(classes[i])) .on('click', (event, d) => { const selected = d3.select(event.currentTarget) // If deselecting a point if (selected.attr("selected") == "true") { svg.selectAll(".point").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); svg.selectAll("text").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); heat.clear(); heat.draw(); } else { // If selecting a point heat.clear(); svg.selectAll(".point").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); //color of labels svg.selectAll("text").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); selected.attr("selected", true) .style('fill', selected_color) .style('stroke', 'black') .raise(); //color of labels selected.selectAll("text").attr("selected", true) .style('fill', "black") .style('stroke', "black"); selected_point_heatmap = get_mds_heatmap( selected.attr('name'), MDS_coords, x, y); const stress_list = selected_point_heatmap.map(function(sublist) { return sublist[2]; }); const max = Math.max(...stress_list); const min = Math.min(...stress_list); selected_point_heatmap.map(function(sublist) { sublist[2] = normalize(sublist[2], max, min); }); heat.data(selected_point_heatmap); heat.draw(); } }) gs.append("text") .attr("x", 0) .attr("y", -10) .style("font-size", 14) .style("font-family", "Verdana") .text((d, i) => { return classes[i] }); gs.append("path") .attr("d", (d, i) => circle); svg.selectAll(".point") .data(MDS_coords) .attr("transform", (([px, py], i) => `translate(${x(px)}, ${y(py)})`)) .attr("id", "points") return container.node() }34:T612,mds_stress_legend = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width; canvas.height = indiv_width/10; var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); var xScale = d3.scaleLinear() .domain([1, 10]) .range([10,390]); var xAxis = d3.axisBottom(d3.scalePoint().domain(["Low", "Medium", "High" ]).range([0, indiv_width-pad*2])); var axisLabel = svg.append("g") .append("text") .attr("fill", "black")//set the fill here .attr("transform","translate("+0+", "+canvas.height+")") .style("font-size", 16) .style("font-family", "Verdana") .text("Relative Stress of Selected Point 2D Projection"); var axis = svg.append("g") .attr('class', 'axis') .attr("transform","translate("+pad+", "+pad*2+")") .style("font-size", 14) .style("font-family", "Verdana") .call(xAxis) var mds_stress_color = d3.piecewise(d3.interpolateRgb.gamma(2.2), ["white", "lightblue", "#4AABCD"]) //var mds_stress_color = d3.piecewise(d3.interpolateRgb.gamma(2.2), ["white", '#9ED9EE', '#87CBE3', '#4AABCD']) const n = indiv_width-pad for (let i = 0; i < n; ++i) { context.fillStyle = mds_stress_color(i / (n - 1)); context.fillRect(i, 0, 1, pad*2); } return container.node() }35:T81e,## t-Distributed Stochastic Neighbor Embedding (tSNE) t-Distributed Stochastic Neighbor Embedding (t-SNE) is a dimension reduction method that models the similarity between data points by comparing the conditional probabilities that points are neighbors in the high-dimensional and low-dimensional spaces. Points that are similar to each other have a high probability of being neighbors and are more likely to be closer to each other in the lower dimensional plot. ### 1. Calculate the probability that high-dimensional points are neighbors. The first step of t-SNE is to find the distances between each pair of points in the higher dimensional space using the Euclidean distance function. t-SNE then calculates the conditional probabilities that the high-dimensional points are neighbors by placing a normal distribution on top of each point. The normal distribution is used to calculate the probability that any pair of points are neighbors to each other. Points that are closer to each other have a higher probability of being neighbors. Higher probabilities indicate higher similarity, while lower probabilities indicate dissimilarity. ### 2. Calculate the probability that low-dimensional points are neighbors. Similar to MDS, t-SNE then randomly initializes the low-dimensional points. Then, t-SNE calculates the conditional probabilities that the low-dimensional points are neighbors the same as the previous step except with a t-distribution instead of a normal distribution. ### 3. Minimize the difference between the high-dimensional and low-dimensional probability distributions. The last step of t-SNE is to optimize the low-dimensional mapping of the high-dimensional data. There is a high-dimensional and low-dimensional probability distribution which represents our data. t-SNE alters the randomly initialized low-dimensional data to minimize the differences between these probability distributions. This is done by performing gradient descent on the Kullback-Leibler Divergence, which measures how close the two probability distributions are to each other. 36:Te0b,tsne = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width + pad*8; canvas.height = indiv_width + pad; // SVG STUFF var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); let [x, y] = await new Promise(res => res(get_scales(animals_TSNE_Y, indiv_width))); var heat = simpleheat(canvas); // set point radius and blur radius (25 and 15 by default) heat.radius(40, 40); heat.gradient({0.33: 'white', 0.66: 'lightblue', 1: '#4AABCD'}); // heat.gradient({0.25: 'white', 0.50: '#9ED9EE', 0.75: '#87CBE3', 1: '#4AABCD'}); heat.max(1); var selected_point_heatmap = [] const selected_color = '#FFE720' const deselected_color = '#808080' const gs = svg.selectAll("point") .data(animals_TSNE_Y) .enter() .append("g") .attr("class", "point") .attr("fill", (d, i) => color(classes[i])) .attr("stroke", (d, i) => color(classes[i])) .attr("name", (d, i) => classes[i]) .attr("original_color", (d, i) => color(classes[i])) .on('click', (event, d) => { const selected = d3.select(event.currentTarget) // If deselecting a point if (selected.attr("selected") == "true") { svg.selectAll(".point").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); svg.selectAll("text").attr("selected", false) .style('fill', d => d.original_color) .style('stroke', d => d.original_color); heat.clear(); heat.draw(); } else { // If selecting a point heat.clear(); svg.selectAll(".point").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); //color of labels svg.selectAll("text").attr("selected", false) .style('fill', d => deselected_color) .style('stroke', d => deselected_color); selected.attr("selected", true) .style('fill', selected_color) .style('stroke', 'black') .raise(); //color of labels selected.selectAll("text").attr("selected", true) .style('fill', 'black') .style('stroke', 'black'); selected_point_heatmap = get_tsne_heatmap( selected.attr('name'), animals_TSNE_Y, x, y); const stress_list = selected_point_heatmap.map(function(sublist) { return sublist[2]; }); const max = Math.max(...stress_list); const min = Math.min(...stress_list); selected_point_heatmap.map(function(sublist) { sublist[2] = normalize(sublist[2], max, min); }); heat.data(selected_point_heatmap); heat.draw(); } }) gs.append("text") .attr("x", 0) .attr("y", -10) .style("font-size", 14) .style("font-family", "Verdana") .text((d, i) => { return classes[i] }) gs.append("path") .attr("d", (d, i) => circle) svg.selectAll(".point") .data(animals_TSNE_Y) .attr("transform", (([px, py], i) => `translate(${x(px)}, ${y(py)})`)) .attr("id", "points") return container.node() }37:T618,tsne_stress_legend = { const container = d3.select(DOM.element('div')); container.style("position", "relative"); const canvas = container.append('canvas').node(); const context = canvas.getContext('2d'); canvas.width = indiv_width; canvas.height = indiv_width/10; var svg = container.append('svg') .attr("width", canvas.width) .attr("height", canvas.height) .style("position", "absolute") .style("top", '0px') .style("left", '0px'); var xScale = d3.scaleLinear() .domain([1, 10]) .range([10,390]); var xAxis = d3.axisBottom(d3.scalePoint().domain(["Low", "Medium", "High" ]).range([0, indiv_width-pad*2])); var axisLabel = svg.append("g") .append("text") .attr("fill", "black")//set the fill here .attr("transform","translate("+0+", "+canvas.height+")") .style("font-size", 16) .style("font-family", "Verdana") .text("Relative Stress of Selected Point 2D Projection"); var axis = svg.append("g") .attr('class', 'axis') .attr("transform","translate("+pad+", "+pad*2+")") .style("font-size", 14) .style("font-family", "Verdana") .call(xAxis) var tsne_stress_color = d3.piecewise(d3.interpolateRgb.gamma(2.2), ["white", "lightblue", "#4AABCD"]) // var tsne_stress_color = d3.piecewise(d3.interpolateRgb.gamma(2.2), ["white", '#9ED9EE', '#87CBE3', '#4AABCD']) const n = indiv_width-pad for (let i = 0; i < n; ++i) { context.fillStyle = tsne_stress_color(i / (n - 1)); context.fillRect(i, 0, 1, pad*2); } return container.node() }38:T1780,explanation = { if (comparison == "Complete animal data"){ return md`This is the complete animal dataset. The distance function for MDS has been set to Euclidean distance (L2 norm) and t-SNE is set at 500 iterations.` } if (comparison == "Low variance data"){ return md`In a dataset with low variance, the data points tend to be close to the mean, or in other words, similar to each other. To show a projection of low variance data, we’ve taken all the water animals from the dataset, where our definition of a water animal is an animal whose water value is greater than 50. In all of the projections, we see some clustering of similar animals. All three have placed the humpback whale, blue whale and walrus in one group, and the killer whale, dolphin, and seal are all relatively close together. The beaver and otter, who are primarily land animals, have been grouped together and projected further away from the rest of the points. However, PCA does not have clusters that are as well defined as in the other two projection, and the points are more spread out. Because the principal components in PCA are selected to maximize variance, the features that contain the most dissimilarity among the points have a larger influence on the projection. As a result, the features that contain the most similarity among the points, such as “water” in this case, are not considered as strongly, and similar points are projected further apart. On the other hand, because MDS and t-SNE look at pairwise similarity to determine projection coordinates, all of the features are considered equally in the projection. As MDS and t-SNE go through each iteration, they pull similar points closer together and push dissimilar points further apart, creating more well defined clusters. ` } if (comparison == "High variance data"){ return md`A high variance dataset has data points that are all generally far from the mean, meaning that they are all different from each other. To show the performance of each method, we've chosen a subset of the animals that have distinctly different characteristics. While the PCA and MDS projection has some clustering, the t-SNE projection places all the points at about the same distance from each other. Because t-SNE is based on the probability of similarity between points, and no pairs of points are similar to each other in this dataset, we have a relatively equidistant projection that doesn’t provide much information about the global similarity of the points. On the other hand, while there is clustering in PCA, the clustering doesn't seem to make much sense. How is a gorilla similar to a hamster? Remember that PCA selects principal components that maximize the variance. In some cases, these principal components can result in strange clustering, especially if there are a few points that could skew the variance a certain way. In other words, the features that we’d think would separate the gorilla and hamster, such as a size, did not actually contain much variance and so did not have much influence on the projection. MDS also has some strange clustering, placing the hamster near the zebra. Because MDS tries to minimize the stress of the overall projection, it will sometimes place points at suboptimal coordinates in order to preserve distances between some of the other surrounding points. In this case, trying to preserve the distances between hamster and chihuahua, while also placing them as far as possible from the larger animals such as gorilla and blue whale places the hamster close to a dissimilar animal. ` } if (comparison == "Data with outliers"){ return md`Outliers are a common feature of many datasets and can affect the way the data is projected into lower dimensions. To compare how PCA, MDS and t-SNE project outliers, we’ve taken all of the water animals (animals whose water value is greater than 50) and added in the lion and chimpanzee as outliers. Notice that while all of the projections have formed clusters, PCA has made the strongest distinction between animals that live entirely underwater and those that live primarily on land, even given the otter and beaver who have high water values. Because PCA is based on the idea of maximizing variance, outliers can have a strong influence on the selection of principal components since they tend to skew the variance for certain features. Because of this, PCA is considered to be sensitive to outliers, and any differences between points can be inflated. The clustering in MDS and t-SNE is not as strong since both methods consider pairwise similarity, rather than variance among the entire dataset. This means that differences in any single feature will push two points away from each other, even if they are otherwise very similar. As a result, the projections have a larger spread, but it still is easy to identify significant outliers. ` } if (comparison == "Manifold data"){ return md`Manifold learning is a growing field of machine learning, in which we assume that our data lies on a low-dimensional manifold embedded in a higher dimensional space. For example, an image of a sphere looks 2D to us on a screen, but in reality, it has points in the 3D space. To show how PCA, MDS and t-SNE work on manifold data, we've generated a random set of points that take the shape of concentric circles. Manifold datasets often have an underlying non-linear structure, so it is important that we use a dimension reduction method that can preserve this structure. As such, PCA would not be a good choice, as it is strictly a linear method. t-SNE and MDS are both considered nonlinear dimension reduction techniques, so they work the best with manifold data. A linear dimension reduction technique, such as PCA, maps high-dimensional data to low-dimensional data through linear functions. Because t-SNE and MDS can map high-dimensional data to low dimensional data through nonlinear functions, they can preserve nonlinear structure in data. ` } }39:T5ee,function variance_binary_search(i, pw_dists, entropy_tol = 1e-3, max_attempts = 40){ // grab the distances for point i const dists_from_i = pw_dists.row(i); let variance_min = 0, // lower bound variance_max = Infinity, // upper bound variance = 1, // initial value searching = true, attempts = 0, probs_given_variance, current_entropy; // number of iterations through while looking const target_entropy = Math.log(perplexity) while(searching){ attempts++ // increment out attempts counter // calculate the probabilities and entropy given the current beta. probs_given_variance = calc_gaussian_probs(dists_from_i, variance); current_entropy = calc_entropy(probs_given_variance); // Entropy is below where we want, so try a higher variance if(current_entropy < target_entropy){ variance_min = variance; // move our min bound up variance = variance_max === Infinity ? variance*2 : (variance+variance_max)/2; } else { // distribution is too narrow. variance_max = variance; // move our max bound down variance = variance_min === -Infinity ? variance/2 : (variance+variance_min)/2; } // check some termination conditions const too_many_attempts = attempts > max_attempts; const close_to_target = Math.abs(current_entropy - target_entropy) < entropy_tol; searching = !(too_many_attempts || close_to_target); } return probs_given_variance }3a:Tb2f,/* Code for an individual visualization in the comparison (gets run three times) */ function add_visualization(svg, data, current_classes, technique_type) { let [translation, background_fill] = get_vis_parameters(technique_type) let [scale_x, scale_y] = get_scales(data, comp_width); // Group & Background const brush = d3.brush() .on("start brush end", brushed); const group = svg.append("g") .attr('transform', translation); const background = group.append('g'); background.append('rect') .attr('x', 0) .attr('y', 0) .attr('width', comp_width) .attr('height', comp_width) .attr('fill', 'white') .attr('stroke', 'lightgray') const title = background.append('text') .attr('x', comp_width/3) .attr('y', -2) .style("font-size", 16) .style("font-family", "Verdana") .attr("font-weight", 600) .text((comparison == "Manifold data") ? `Manifold with ${technique_type}` : `Animals with ${technique_type}`) .attr('fill', 'black'); // Points const points_group = background.append("g").call(brush); const points = points_group.selectAll(".point-comp") .data(data) .enter() .append("g") .attr("class", "point-comp") .attr("name", (d, i) => current_classes[i]) .attr("isBrushed", false) .attr("techniqueType", technique_type); points.append("text") .attr("y", -.5*pad) .style("text-anchor", "middle") .style("font-size", 12) .style("font-family", "Verdana") .text((d, i) => { return current_classes[i] }) points.append("path") .attr("d", circle) points_group.selectAll(".point-comp") .data(data) .attr("transform", ((d, i) => `translate(${scale_x(d[0])}, ${scale_y(d[1])})`)) .attr("stroke", (d, i) => get_color(d[2])) .attr("fill", (d, i) => get_color(d[2])) function brushed({selection}) { // IF BRUSH ADDED if (selection) { const [[x0, y0], [x1, y1]] = selection; points_group.selectAll(".point-comp") .filter((d, i) => x0 <= scale_x(d[0]) && scale_x(d[0]) < x1 && y0 <= scale_y(d[1]) && scale_y(d[1]) < y1) .attr("isBrushed", true) .data() .forEach(function(d, i) { let name = classes[d[2]] if (!brushedClasses.includes(name)){ brushedClasses.push(name) }}) svg.selectAll(".point-comp") .attr("fill", (d, i) => get_color(d[2])) .attr("stroke", (d, i) => get_color(d[2])); } // IF BRUSH DELETED else { while(brushedClasses.length) { brushedClasses.pop() } svg.selectAll(".point-comp") .attr("isBrushed", false); svg.selectAll(".point-comp") .attr("fill", (d, i) => get_color(d[2])) .attr("stroke", (d, i) => get_color(d[2])) } } }3b:[{"id":"b697f1dab23ab004c2e72917e76189131068ae959614505d8b706f0bca3b9d391c6d7892da2d881c10aea323889c61145e69af70d7148c2d58f9f7205d12ec8c","href":"https://static.observableusercontent.com/files/b697f1dab23ab004c2e72917e76189131068ae959614505d8b706f0bca3b9d391c6d7892da2d881c10aea323889c61145e69af70d7148c2d58f9f7205d12ec8c","name":"animal.csv","type":"text/csv","lastModified":1639710815025.966,"size":9439},{"id":"6bf6a60d20a76e12039aa3315838081c548f5f144a7c23422256aa36b85b176e2869995d8be1f760d1fd4ee4ccc52a497658ae42111fe4bb4c611934281c4ee6","href":"https://static.observableusercontent.com/files/6bf6a60d20a76e12039aa3315838081c548f5f144a7c23422256aa36b85b176e2869995d8be1f760d1fd4ee4ccc52a497658ae42111fe4bb4c611934281c4ee6","name":"manifold.csv","type":"text/csv","lastModified":1639710815025.966,"size":1296}] 2d:{"id":"778ef7f5c849f4cc","slug":null,"title":"A Comparative Overview of Dimension Reduction Methods","thumbnail":"85662d9f2d1f45181b5b2c5648a0520400b7b4ef2706fb05e2ac14477e588208","access_level":"public","create_time":"$D2021-12-17T03:13:35.025Z","update_time":"$D2021-12-17T03:13:35.025Z","view_time":null,"first_public_time":"$D2025-09-03T01:46:24.986Z","fork":{"id":"6356533826a18c85","slug":"a-comparative-overview-of-dimension-reduction-methods","title":"A Comparative Overview of Dimension Reduction Methods","owner":{"id":"0e216e195731055d","login":"3e787c1af6c5a437","name":"Elizabeth Christman"}},"stars":0,"starred":false,"owner":{"type":"team","id":"944d12de66c65f0a","login":"chrislnorth","name":"Chris North","avatar_url":"https://avatars-next.observableusercontent.com/4ccf47b2182cd891232f3949528d386f98cf55e9749ceb65960723d6b963c5a2","tier":"starter_2024"},"version":11761,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"2bd52d2c49b80248","login":"chrislnorth","name":"Chris North","avatar_url":"https://avatars-next.observableusercontent.com/4ccf47b2182cd891232f3949528d386f98cf55e9749ceb65960723d6b963c5a2","tier":null}],"members":[{"type":"individual","id":"2bd52d2c49b80248","login":"chrislnorth","name":"Chris North","avatar_url":"https://avatars-next.observableusercontent.com/4ccf47b2182cd891232f3949528d386f98cf55e9749ceb65960723d6b963c5a2","tier":null,"role":"editor"},{"type":"team","id":"944d12de66c65f0a","login":"chrislnorth","name":"Chris North","avatar_url":"https://avatars-next.observableusercontent.com/4ccf47b2182cd891232f3949528d386f98cf55e9749ceb65960723d6b963c5a2","tier":"starter_2024","role":"viewer"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":{"id":"b3520c7ea056fdd9","slug":"cs-3654","title":"VT CS 3654","owner":{"id":"944d12de66c65f0a","login":"chrislnorth","name":"Chris North"}},"type":"notebook","body":{"title":"A Comparative Overview of Dimension Reduction Methods","cells":[{"id":47,"value":"# A Comparative Overview of Dimension Reduction Methods","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10848,"value":"by Elizabeth Christman, Avi Seth, Mia Taylor","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10871,"value":"### Introduction","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5135,"value":"md`In data analysis, we often work with high dimensional datasets, where each data point has many values, or features, associated with it. Sometimes the number of features can even be greater than the number of observations in the entire data set. As we develop new ways of collecting data, we’re gathering more information than ever before, and high dimensional datasets are becoming more and more common. However, one challenge of working with high dimensional datasets is that it can be difficult to create visualizations to understand the data. Each feature represents a dimension of the data, so when we plot a dataset with n features, we end up with an n dimensional plot, which is impossible to visualize. To solve this problem, data scientists often use a technique called dimension reduction to transform the original data into something that is more easily visualized and understood.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5156,"value":"md`### What is dimension reduction?`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5140,"value":"md`Dimension reduction is the transformation of data from a higher dimension into a lower, easily visualized dimension, while still maintaining as much of the original properties of the data as possible. There are many different techniques to achieve this, each with their own algorithm to transform the data. Each one has its own strengths and weaknesses, and some might perform better on certain datasets than others.\n\nIn this notebook, we present three of the most common dimension reduction techniques: Principal Component Analysis (PCA), Multidimensional Scaling (MDS), and t-Distributed Stochastic Neighbor Embedding (t-SNE). We briefly explain the algorithms supporting each one, and then compare the three to see their visualizations on different types of data.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":400,"value":"md`### The Dataset`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":405,"value":"md`To explain dimension reduction, we will use the animals dataset shown below. There are 50 animals, each with 36 features. The value of the feature is the amount of the characteristic that each animal has. Values range from 0 to 100. For example, the humpback whale, an animal that lives underwater, has the highest water value at 89.95.\n\nExplore the table to see the values of each animals’ features. Click on the column headers to sort the data.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6908,"value":"// Inputs.table(animal_data)\n{\n const svg = d3.create(\"svg\")\n .attr(\"font-size\", \"11pt\")\n .attr(\"viewBox\", [0, 0, width, 360]);\n \n yield svg.node();\n \n new SVGTable(svg)\n .size([width, 350])\n .fixedRows(0)\n .fixedColumns(1) \n .defaultNumberFormat(\",.4\")\n .style({ border: true })\n .data(animal_data) \n .render(); \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3,"value":"$2f","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10915,"value":"### 2. Compute the correlation matrix\nThe next step of PCA is to create a correlation matrix in order to discover the relationships between the features. A positive correlation means the two variables will increase or decrease together. The maximum correlation value is 1. A negative correlation means that the two variables will move in opposite directions, so one will increase while the other decreases. The minimum correlation value is -1. The magnitude of the correlation value indicates the strenght of the relationship between features. A magnitude of 0.7 or greater indicates a strong relationship.\nThe table below shows the correlation matrix of the animal dataset. A darker shade indicates a stronger positive correlation. Notice that the correlation value between the same feature, such as Water and Water, is equal to 1. This indicates a perfectly positive correlation.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10410,"value":"$30","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10927,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10929,"value":"For a more detailed explanation of the math behind PCA, visit the links below: \nhttp://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf\nhttps://sebastianraschka.com/Articles/2015_pca_in_3_steps.html\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10942,"value":"### PCA Projection","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1885,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3042,"value":"md`## Multidimensional Scaling (MDS)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":293,"value":"Multidimensional scaling (MDS) is a dimension reduction method that is based on the distance between data points. The similarity between points is reflected as distance, and points that are closer to each other in the plot will be more similar to each other.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3058,"value":"md`### 1. Calculate distance in high dimensional space.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3066,"value":"The first step of MDS is to find the distances between each pair of points in the higher dimensional space using a distance function. These values can be stored in a distance matrix, which we have shown below. There are various distance functions that can be used depending on the type of data, but Manhattan (L1 norm) and Euclidean (L2 norm) are the most common.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":10962,"value":"The shading of each cell in the distance matrix below indicates the distance between the two animals in high dimensional space. A darker shade means they are farther apart, while a lighter shade indicates they are closer together.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":6897,"value":"// Inputs.table(distance_matrix(animals, mds_metric(MDS_distanceFunc)))\n{\n const svg = d3.create(\"svg\")\n .attr(\"font-size\", \"11pt\")\n .attr(\"viewBox\", [0, 0, width, 360]);\n\n const distMat = distance_matrix(animals, mds_metric(MDS_distanceFunc));\n \n yield svg.node();\n \n const table = new SVGTable(svg)\n .size([width, 350])\n .fixedRows(0)\n .fixedColumns(1) \n .defaultNumberFormat(\".2\")\n .style({ \n highlight: \"cell\"\n })\n .heatmapPalette(d3.interpolateReds)\n .heatmap(true)\n .data(distMat) \n .render(); \n\n \n svg.selectAll(\"text\").style(\"font-family\", \"Verdana\")\n return svg.node()\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3081,"value":"### 2. Calculate distance in low dimensional space.\nThe second step of MDS randomly initializes the points in two dimensions and calculates another distance matrix with the low dimensional points. Often this is done using a different distance function than in step 1, as distance functions that are good for lower dimensions may not be best suited for higher dimensional data.\n\n### 3. Minimize the difference between the high dimensional and low dimensional distances.\nFinally, MDS optimizes the low dimensional projection by minimizing the error between the high dimensional and low dimensional distances. This error is measured using a stress function, and with each iteration, MDS will move the low dimensional points to a place that further minimizes stress. The final projection will then show the relative distances between the points, indicating their similarity.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":4583,"value":"## MDS Projection","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3099,"value":"viewof MDS_distanceFunc = select({\n title: \"Distance Function\",\n description: \"Change the distance function to see how the projection changes. The new calculations will also be reflected in the distance matrix above.\",\n value: \"Euclidean (L2)\", \n options: [\"Manhattan (L1)\", \"Euclidean (L2)\"],\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1390,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8492,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3944,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3981,"value":"### t-SNE Hyperparameter\n\nNote that t-SNE has a hyperparameter called perplexity. Perplexity loosely controls the number of neighbors a point has which affects how \"clustered\" the final visualization appears. Typical values are between 5 and 50.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3987,"value":"Because t-SNE uses gradient descent, we provide a slider that controls the number of iterations to display the low-dimensional mapping below. Note that when iterations is 0, there is a seemingly random clustering of the data. This is because the low-dimensional data is randomly initialized. As you drag the number of iterations up to 500, you will begin to see clusters that make more sense intuitively, such as different types of whales being clustered together in the top left corner!","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1277,"value":"## t-SNE Projection","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1332,"value":"viewof perplexity = slider({\n min: 2, \n max: 100, \n step: 1, \n value: 20, \n title: \"Perplexity\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1301,"value":"viewof iterations = slider({\n min: 0, \n max: 500, \n step: 1, \n value: 0, \n title: \"Iterations\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1306,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9781,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":696,"value":"md`## Let's Compare`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":943,"value":"Here are the projections of all three dimension reduction methods again. Brush over any of the points to highlight their locations in each of the other projections. You can do this by clicking and dragging a rectangle over a certain area in one of the projections. Click anywhere within the projections to remove the selection. Use the dropdown menu to select the primary projection for the gradient coloring.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5338,"value":"md`Use the dropdown menu to select different types of datasets to see how these methods perform.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5170,"value":"viewof comparison = select({\n title: \"Dataset\",\n description: \"Pick a dataset to compare the three methods.\",\n options: [\"Complete animal data\", \"Low variance data\", \"High variance data\", \"Data with outliers\", \"Manifold data\"],\n value: \"Complete animal data\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7315,"value":"viewof primary_technique = select({\n title: \"Primary Technique\",\n description: \"Pick the projection that controls the gradient coloring.\",\n options: [\"PCA\", \"MDS\", \"t-SNE\"],\n value: \"PCA\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6131,"value":"general_comp.node()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6887,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":76,"value":"## Appendix","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7738,"value":"numeric = require('https://bundle.run/numericjs@1.2.6')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":181,"value":"druid = require(\"@saehrimnir/druidjs@^0.3.7\");","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1303,"value":"import {slider, select, checkbox} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8348,"value":"import {Legend, Swatches} from \"@d3/color-legend\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6893,"value":"import {SVGTable} from '@analyzer2004/svgtable'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1356,"value":"/*\n Scale functions between druid x,y coordinates and d3 x,y locations\n*/\nget_scales = function(data, w) {\n let x_extent = d3.extent(data, d => d[0]);\n let y_extent = d3.extent(data, d => d[1]);\n let x_span = x_extent[1] - x_extent[0];\n let y_span = y_extent[1] - y_extent[0];\n const offset = Math.abs(x_span - y_span) / 2;\n \n let x;\n let y;\n \n if (x_span > y_span) {\n x_extent[0] -= offset;\n x_extent[1] += offset;\n } else {\n y_extent[0] -= offset;\n y_extent[1] += offset;\n }\n\n return [\n d3.scaleLinear().domain(x_extent).rangeRound([pad, w - pad]).nice(),\n d3.scaleLinear().domain(y_extent).rangeRound([pad, w - pad]).nice(),\n ]\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3236,"value":"pad = 20\n// padding for comparative visualizations","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1358,"value":"// color = d3.scaleSequential(d3.interpolateViridis) // working viridis\ncolor = d3.scaleOrdinal(d3.schemeDark2)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":4082,"value":"// Pixel width for single DRT visualization\nindiv_width = 4*width/5","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6284,"value":"/*\n Heatmap package\n*/\nsimpleheat = {\n const response = await fetch(\"https://raw.githubusercontent.com/mourner/simpleheat/gh-pages/simpleheat.js\");\n const blob = await response.blob();\n return require(URL.createObjectURL(blob)).catch(() => window.simpleheat);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1367,"value":"circle = \"M5.641895835477563,0A5.641895835477563,5.641895835477563,0,1,1,-5.641895835477563,0A5.641895835477563,5.641895835477563,0,1,1,5.641895835477563,0\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8744,"value":"### Data Processing","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10751,"value":"function average(data){\n var sum = data.reduce(function(sum, value){\n return sum + value;\n }, 0);\n\n var avg = sum / data.length;\n return avg;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10749,"value":"function standardDeviation(values){\n var avg = average(values);\n \n var squareDiffs = values.map(function(value){\n var diff = value - avg;\n var sqrDiff = diff * diff;\n return sqrDiff;\n });\n \n var avgSquareDiff = average(squareDiffs);\n\n var stdDev = Math.sqrt(avgSquareDiff);\n return stdDev;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5402,"value":"data_normed = function(data, norm=true){\n const attrs = Object.keys(data[0]).filter(\n a => {\n return (typeof data[0][a] === \"number\") && !(new Set([\"Name\"])).has(a)\n }\n );\n \n if (norm) {\n const maxes = attrs.map(a => {\n return d3.max(data, d => d[a] + 10**-100);\n })\n\n return data.map(d => attrs.map((a, i) => d[a] / maxes[i]));\n } else {\n return data.map(d => attrs.map((a, i) => d[a] + 10**-100));\n } \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":283,"value":"animal_data = FileAttachment(\"animal.csv\").csv({typed: true})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10766,"value":"standardized_animal_data = {\n var standardized_animal_data = animal_data.map(a => {return {...a}}); // hard copy of objects\n const num_animals = animal_data.length;\n \n for (let i = 0; i < features.length; i++) {\n console.log(\"Feature: \" + features[i]);\n var values = standardized_animal_data.map(a => a[features[i]]);\n \n const sum = values.reduce(v=> {\n return v\n }, 0);\n\n const mean = sum / values.length;\n \n const std_dev = standardDeviation(values);\n\n values = values.map(a => ((a-mean)/std_dev));\n\n standardized_animal_data.map((a, j) => {\n a[features[i]] = Number.parseFloat(values[j].toFixed(3));\n return a;\n });\n }\n\n return standardized_animal_data;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10747,"value":"// Sanity check for Mia that standardized animal data is standardized\nis_data_standardized = {\n var is_data_standardized = true;\n const num_animals = standardized_animal_data.length;\n \n for (let i = 0; i < features.length; i++) {\n // console.log(\"Feature: \" + features[i]);\n const values = standardized_animal_data.map(a => a[features[i]]);\n \n const sum = values.reduce(v=> {\n return v\n }, 0);\n\n const mean = sum / values.length;\n // console.log(\"Mean \" + mean);\n if (mean != 0) is_data_standardized = false;\n \n const std_dev = standardDeviation(values);\n // console.log(\"Std dev \" + std_dev);\n const str_std_dev = `${std_dev}`\n if (!(str_std_dev === \"1\" || str_std_dev.includes(\"0.999\") || str_std_dev.includes(\"1.000\"))) is_data_standardized = false;\n }\n\n return is_data_standardized;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":391,"value":"animals = data_normed(animal_data)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":289,"value":"classes = animal_data.map(d => d?.Name)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1961,"value":"md`### PCA`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10071,"value":"features = Object.keys(animal_data[0]).splice(1)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10199,"value":"corr_matrix = {\n const matrix_data = standardized_animal_data;\n const animals_matrix = druid.Matrix.from(data_normed(matrix_data, false));\n const matrix = new Array();\n min_max_mean_cov[0] = get_cov(animals_matrix.col(0), animals_matrix.col(0));\n min_max_mean_cov[1] = -2;\n var sum_cov = 0;\n \n for (let i = 0; i < features.length; i++) {\n matrix.push({\n \"Feature\": features[i]\n });\n }\n \n for (let i = 0; i < features.length; i++) {\n for (let j = i; j < features.length; j++) {\n var cov_ij;\n if (i == j) cov_ij = 1;\n else cov_ij = get_cov(animals_matrix.col(i), animals_matrix.col(j));\n \n matrix[i][features[j]] = cov_ij;\n matrix[j][features[i]] = cov_ij;\n\n if (min_max_mean_cov[0] > cov_ij) min_max_mean_cov[0] = cov_ij;\n if (min_max_mean_cov[1] < cov_ij) min_max_mean_cov[1] = cov_ij;\n sum_cov += cov_ij;\n }\n }\n min_max_mean_cov[2] = sum_cov / ((Math.pow(features.length, 2)/2));\n return matrix;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10328,"value":"// used for normalizing for heatmap coloring\nmin_max_mean_cov = Array(3);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10242,"value":"// Get covariance of two features (1D arrays)\nfunction get_cov(X, Y) {\n const X_mean = X.reduce((a,b) => a + b, 0) / X.length;\n const Y_mean = Y.reduce((a,b) => a + b, 0) / Y.length;\n var cov_top = 0;\n const cov_bottom = X.length - 1;\n\n for (let i = 0; i < X.length; i++) {\n cov_top += ((X[i] - X_mean)*(Y[i] - Y_mean));\n }\n return cov_top/cov_bottom;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9997,"value":"cov_grid_size = Math.floor(indiv_width / features.length)*1.5","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9966,"value":"cov_color_scale = d3.interpolateRdBu","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9976,"value":"cov_colors = [\"white\", \"#163c48\"]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":10039,"value":"previous_d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5438,"value":"get_PCA = function(data){\n const X = druid.Matrix.from(data); // X is the data as object of the Matrix class.\n const DR = druid[\"PCA\"]; // DR is the selected DR class\n return new DR(X);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1877,"value":"get_animals_PCA = {\n const X = druid.Matrix.from(animals); // X is the data as object of the Matrix class.\n const DR = druid[\"PCA\"]; // DR is the selected DR class\n return new DR(X);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1881,"value":"animals_PCA_Y = get_animals_PCA.transform();","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1818,"value":"md`### MDS`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":11671,"value":"min_max_mean_dist = {\n const arr = new Array(3);\n const dist_matrix = distance_matrix(animals, mds_metric(MDS_distanceFunc));\n console.log(dist_matrix);\n arr[0] = 5;\n arr[1] = -1;\n var sum = 0;\n var num_animals = dist_matrix.length;\n var num_values = Object.values(dist_matrix[0]).slice(1).length;\n var num_skipped = 0;\n\n for (let i = 0; i < num_animals; i++) {\n var values = Object.values(dist_matrix[i]).slice(1);\n \n for (let j = 0; j < values.length; j++) {\n const val = values[j];\n if (val == 0) {\n num_skipped++;\n continue;\n }\n console.log(val);\n if (arr[0] > val) arr[0] = val;\n if (arr[1] < val) arr[1] = val;\n sum += val;\n }\n }\n console.log(sum);\n arr[2] = sum / ((num_animals * num_values) - num_skipped);\n return arr;\n}\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7739,"value":"//https://www.benfrederickson.com/multidimensional-scaling/\n/// given a matrix of distances between some points, returns the\n/// point coordinates that best approximate the distances\nget_MDS = function(distances, dimensions) {\n dimensions = dimensions || 2;\n\n // square distances\n var M = numeric.mul(-.5, numeric.pow(distances, 2));\n\n // double centre the rows/columns\n function mean(A) { return numeric.div(numeric.add.apply(null, A), A.length); }\n var rowMeans = mean(M),\n colMeans = mean(numeric.transpose(M)),\n totalMean = mean(rowMeans);\n\n for (var i = 0; i < M.length; ++i) {\n for (var j =0; j < M[0].length; ++j) {\n M[i][j] += totalMean - rowMeans[i] - colMeans[j];\n }\n }\n\n // take the SVD of the double centred matrix, and return the\n // points from it\n var ret = numeric.svd(M),\n eigenValues = numeric.sqrt(ret.S);\n return ret.U.map(function(row) {\n return numeric.mul(row, eigenValues).splice(0, dimensions);\n });\n};","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":11062,"value":"mds_metric = function(distFunc){\n if(distFunc == \"Euclidean\") return euclidean;\n if(distFunc == \"Manhattan\") return manhattan;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7761,"value":"//redefined stress function\nstress = function(distHD, dist2D){\n let top_sum = 0;\n let bottom_sum = 0;\n for (let i = 0; i < distHD.length; ++i) {\n for (let j = i + 1; j < distHD.length; ++j) {\n top_sum += Math.pow(distHD[i][j] - dist2D[i][j], 2);\n bottom_sum += Math.pow(distHD[i][j], 2);\n }\n }\n return Math.sqrt(top_sum / bottom_sum);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7741,"value":"hd_dist = {\n if(MDS_distanceFunc == \"Euclidean (L2)\") return raw_distance_matrix(animals, euclidean)\n if(MDS_distanceFunc == \"Manhattan (L1)\") return raw_distance_matrix(animals, manhattan)\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7754,"value":"ld_coords = get_MDS(hd_dist, 2)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7760,"value":"ld_dist = raw_distance_matrix(ld_coords, MDS_distanceFunc)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7083,"value":"/*\nReturn normalized version of value given max and min\n*/\nfunction normalize(val, max, min) {\n if(max - min === 0) return 1;\n return (val - min) / (max - min); \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7022,"value":"/*\nN = Num classes\ndist_matrix = Distance matrix of high-dimensional points\nY = Low-dimensional data (projection) as druid Matrix\n*/\nget_MDS_stress = function(N, dist_matrix, Y) {\n const d_X = dist_matrix; //high dim distance \n const d_Y = new druid.Matrix(); //fill with low dim distance\n\n d_Y.shape = [N, N, (i, j) => {\n return i < j ? euclidean(Y.row(i), Y.row(j)) : d_Y.entry(j, i); // calculates low dim euclidean dist\n }]\n \n let top_sum = 0;\n let bottom_sum = 0;\n for (let i = 0; i < N; ++i) {\n for (let j = i + 1; j < N; ++j) {\n top_sum += Math.pow(d_X.entry(i, j) - d_Y.entry(i, j), 2);\n bottom_sum += Math.pow(d_X.entry(i, j), 2);\n }\n }\n return Math.sqrt(top_sum / bottom_sum);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6979,"value":"/*\nName = animal name\nProjection = low-dimensional input data\nscale_x, scale_y = functions that convert low-dimensional x,y to d3 x,y coordinates\n*/\nfunction get_mds_heatmap(name, projection, scale_x, scale_y) {\n var heat = new Array();\n var row = classes.indexOf(name)\n var granularity = 35;\n const X = druid.Matrix.from(animals)\n const N = X.shape[0]\n // const dist_matrix = druid.distance_matrix(X, euclidean) //druid library is hard coded to euclidean, use the redefined function\n const dist_matrix = druid.Matrix.from(hd_dist)\n \n for(var x = pad; x < indiv_width - pad; x += indiv_width/granularity) { //x coordinates of vis\n var altered_projection = projection.clone();\n \n for (var y = pad; y < indiv_width - pad; y += indiv_width/granularity) {\n \n altered_projection.set_row(row, [scale_x.invert(x), scale_y.invert(y)])\n heat.push([x, y, get_MDS_stress(N, dist_matrix, altered_projection)]); //data copy has druid coords\n }\n }\n return heat;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1267,"value":"### t-SNE","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5410,"value":"function get_TSNE(data, it, perplex = 20, eps = 15) {\n return new druid[\"TSNE\"](druid.Matrix.from(data), perplex, eps).transform(it);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8643,"value":"/*\nName = animal name\nProjection = low-dimensional input data\nscale_x, scale_y = functions that convert low-dimensional x,y to d3 x,y coordinates\n*/\nfunction get_tsne_heatmap(name, projection, scale_x, scale_y) {\n var heat = new Array();\n var row = classes.indexOf(name);\n console.log(\"Getting heatmap for row \" + row);\n var granularity = 20;\n const X = druid.Matrix.from(animals);\n const N = X.shape[0];\n const P = find_P(N, X);\n \n for(var x = pad; x < indiv_width - pad; x += indiv_width/granularity) { //x coordinates of vis\n var altered_projection = projection.clone();\n \n for (var y = pad; y < indiv_width - pad; y += indiv_width/granularity) {\n \n altered_projection.set_row(row, [scale_x.invert(x), scale_y.invert(y)])\n heat.push([x, y, get_KLD(P, N, altered_projection)]); //data copy has druid coords\n }\n }\n console.log(\"Heatmap: \" + heat);\n return heat;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8653,"value":"function get_KLD(P, N, Y, proj_dist_matrix) {\n const Q = find_Q(N, Y, proj_dist_matrix);\n var cost = 0;\n \n for (let i = 0; i < N; i++) {\n for (let j = 0; j < N; j++) {\n cost += P.entry(i, j) * Math.log(P.entry(i, j)/Q.entry(i, j));\n }\n }\n return cost;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":8645,"value":"function find_P(N, X) { \n console.log(\"Finding P\");\n const P = new druid.Matrix(N, N);\n\n let p_row, i, j;\n const P_i_given_j = new druid.Matrix(N, N);\n \n for(i = 0; i < N; i++){\n const pairwise_distances = get_pairwise_distances(N, X);\n console.log(\"Pairwise distances: \");\n console.log(pairwise_distances)\n p_row = variance_binary_search(i, pairwise_distances) // probs from best variance. \n console.log(\"P row: \" + p_row);\n for(j=0;j {\n const curr_prob = dist !== 0 ? gaussian_prob(dist, variance): 0;\n sum_probs += curr_prob; // accumulate the sums \t\n return curr_prob; // send probability to our array \n }).map(p => p/sum_probs) // return the prob vector after normalizing it using sum.\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9706,"value":"function calc_entropy(probs){\n return probs.reduce( \n (sum, p) => sum - (p>1e-7 ? p*Math.log(p): 0), // super small probabilities can be ignored\n 0\n )\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1309,"value":"animals_TSNE_Y = get_animals_TSNE_selected_hyperparam(iterations);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1311,"value":"function get_animals_TSNE_selected_hyperparam(it) {\n return get_TSNE(animals, it, perplexity);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1795,"value":"md` #### Methods for Distance Matrix`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7745,"value":"raw_distance_matrix = function(A, metric = euclidean) {\n if (metric === undefined) return undefined;\n let n = A.length;\n let D = new Array(n);\n for (let i = 0; i < n; ++i) {\n D[i] = new Array(n);\n }\n for (let i = 0; i < n; ++i) {\n for (let j = i; j < n; ++j) {\n if(i === j){\n D[i][j] = 0;\n }\n else{\n if(metric == euclidean){\n D[i][j] = D[j][i] = euclidean(A[i], A[j]);\n }\n if(metric == manhattan){\n D[i][j] = D[j][i] = manhattan(A[i], A[j]);\n }\n }\n }\n }\n return D;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":319,"value":"distance_matrix = function(A, metric = euclidean) {\n if (metric === undefined) return undefined;\n let n = A.length;\n let D = new Array(n);\n for (let i = 0; i < n; ++i) {\n D[i] = new Array(n);\n }\n for (let i = 0; i < n; ++i) {\n for (let j = i; j < n; ++j) {\n if(i === j){\n D[i][j] = 0;\n }\n else{\n D[i][j] = D[j][i] = metric(A[i], A[j]);\n }\n }\n }\n for(let i =0;i= Math.abs(summand)) {\n compensation += (sum - t) + summand;\n } else {\n compensation += (summand - t) + sum;\n }\n sum = t;\n }\n return sum + compensation;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":323,"value":"euclidean_squared = function(a, b) {\n if (a.length != b.length) return undefined\n let n = a.length\n let s = new Array(n);\n for (let i = 0; i < n; ++i) {\n let x = a[i];\n let y = b[i]\n s[i] = ((x - y) * (x - y))\n }\n return neumair_sum(s);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":325,"value":"euclidean = function(a, b) {\n return Math.sqrt(euclidean_squared(a, b));\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3176,"value":"manhattan = function(a, b) {\n if (a.length != b.length) return undefined\n let n = a.length\n let sum = 0;\n for (let i = 0; i < n; ++i) {\n sum += Math.abs(a[i] - b[i])\n }\n return sum\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2929,"value":"### Comparison","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6834,"value":"data_subset_raw = { \n var high_var_data = [\"Grizzly Bear\", \"Blue Whale\", \"Gorilla\", \"Hamster\", \"Bobcat\", \"Rhinocerous\", \"Zebra\", \"Cow\", \"Chihuahua\"]; \n \n var low_var_data = [\"Killer Whale\" , \"Blue Whale\", \"Humpback Whale\", \"Walrus\", \"Dolphin\", \"Beaver\", \"Seal\", \"Otter\"];\n\n var outlier_data = [\"Killer Whale\", \"Blue Whale\", \"Humpback Whale\", \"Walrus\", \"Dolphin\", \"Beaver\", \"Seal\", \"Otter\", \"Lion\", \"Chimpanzee\"]; \n\n var manifold_raw = FileAttachment(\"manifold.csv\").csv({typed :true})\n \n \n if(comparison == \"Complete animal data\") return animal_data;\n if(comparison == \"High variance data\"){\n return animal_data.filter(animal => high_var_data.includes(animal.Name))\n }\n if(comparison == \"Low variance data\"){\n return animal_data.filter(animal => low_var_data.includes(animal.Name))\n }\n if(comparison == \"Manifold data\"){\n return manifold_raw\n }\n if(comparison == \"Data with outliers\"){\n return animal_data.filter(animal => outlier_data.includes(animal.Name))\n }\n \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6837,"value":"data_subset = data_normed(data_subset_raw)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6840,"value":"data_subset_classes = (comparison != \"Manifold data\") ? data_subset_raw.map(d => d?.Name) : []","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2976,"value":"/*\n General comparison svg which adds each comparison visualization\n*/\ngeneral_comp = {\n var svg = d3\n .create(\"svg\")\n .attr(\"width\", width)\n .attr(\"height\", comp_width*2 + pad*4);\n\n while(brushedClasses.length > 0) brushedClasses.pop(); // reset brushed classes everytime visualization changes\n comparison; // using comparison var here forces vis redraw when comparison value changes, KEEP!\n\n [\"PCA\", \"MDS\", \"t-SNE\"].forEach(technique => {\n const data = include_class_index(projectDR(data_subset, technique), data_subset_classes);\n add_visualization(svg, data, data_subset_classes, technique);\n });\n \n return svg;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":2924,"value":"// Width for comparative visualization windows\ncomp_width = Math.min(500, width/2 - pad*4)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6831,"value":"projectDR = function(data, dr_method){\n var data_dist_matrix = raw_distance_matrix(data, euclidean);\n if(dr_method == \"PCA\"){\n return get_PCA(data).transform();\n }\n else if(dr_method == \"MDS\"){\n return druid.Matrix.from(get_MDS(data_dist_matrix, 2));\n }\n else if(dr_method == \"t-SNE\"){\n return get_TSNE(data, 500);\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3005,"value":"function get_vis_parameters(vis) {\n var dx;\n var dy;\n var translation;\n var background_fill;\n \n if (vis == \"PCA\") {\n dx = 0\n dy = pad\n background_fill = 'lightgray'\n } else if (vis == \"MDS\") {\n dx = comp_width + pad\n dy = pad\n background_fill = 'lightgray'\n } else {\n dx = comp_width/2\n dy = comp_width + 3*pad\n background_fill = 'lightgray'\n }\n\n translation = `translate(${dx}, ${dy})`\n return [translation, background_fill]\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3199,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7351,"value":"// Automatically updates when primary visualization technique is changed\n// Key is animal name\n// Value is primary techinique's y value for that animal\nprimary_technique_map = {\n const pt_map = new Map();\n const primary_technique_data = include_class_index(projectDR(data_subset, primary_technique), data_subset_classes);\n let [scale_x, scale_y] = get_scales(primary_technique_data, comp_width);\n primary_technique_data.to2dArray.forEach((d, i) => {\n const normalized_y = scale_y(d[1])/comp_width;\n pt_map.set(classes[d[2]], comp_color(normalized_y));}\n )\n return pt_map;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6173,"value":"/*\n Automatically updates to include classes that are currently brushed in the comparison visualizations\n*/\nbrushedClasses = [] ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5752,"value":"// Get html color for a single point on comparative visualizations\nfunction get_color(i) {\n const name = classes[i]\n const isBrushed = brushedClasses.includes(name)\n const display_color = primary_technique_map.get(name)\n\n // Return display color if the name is brushed or if no classes are brushed, otherwise gray\n return (brushedClasses.length > 0 && isBrushed || brushedClasses.length == 0) ? display_color : \"gray\"\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":7324,"value":"comp_color = d3.scaleSequential(d3.interpolateViridis)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":6513,"value":"function include_class_index(data, subset_classes = []) {\nvar data_with_class = Array()\n\n if (subset_classes.length == 0){\n // Uses full dataset\n data.to2dArray\n .forEach(function(d, i) {\n data_with_class.push(d[0],d[1],i)\n });\n } else {\n // Uses subset of data\n data.to2dArray\n .forEach(function(d, i) {\n data_with_class.push(d[0],\n d[1],\n classes.indexOf(subset_classes[i])\n )});\n }\n \n Array.prototype.chunk = function(n) {\n if (!this.length) {\n return [];\n }\n return [this.slice(0, n)].concat(this.slice(n).chunk(n));\n };\n \n return data = druid.Matrix.from(data_with_class.chunk(3))\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9670,"value":"mathjs = require(\"mathjs\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3b","stdlib":"1","titling":"auto"}} 2e:"$2d:body"