29:null 2a:["$","title",null,{"children":"Resizing segmented vectors | S. Scott Whitmore | Observable"}] 2f:[{"id":"e6d5a8b97feb4999660764c12af9670d458ae98ee33892ca713f767b0231f11325e8defb8741413628ba0925cb0bc8054be3f04b957fa405a6c7445aa466c038","href":"https://static.observableusercontent.com/files/e6d5a8b97feb4999660764c12af9670d458ae98ee33892ca713f767b0231f11325e8defb8741413628ba0925cb0bc8054be3f04b957fa405a6c7445aa466c038","name":"bscan_simulation.png","type":"image/png","lastModified":1634146954453.785,"size":382668}] 2d:{"id":"24787dc6785f7fdb","slug":"resizing-segmented-vectors","title":"Resizing segmented vectors","thumbnail":"66188268d3971d142cdab161f1a56179d749029efbd12216922391f49cfd34e0","access_level":"public","create_time":"$D2021-10-12T14:46:24.822Z","update_time":"$D2023-02-08T15:34:17.411Z","view_time":null,"first_public_time":"$D2021-10-13T18:09:48.911Z","has_unpublished_edits":null,"fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"9f9ba4a25d6eb4e6","login":"barefootbiology","name":"S. Scott Whitmore","avatar_url":"https://avatars-next.observableusercontent.com/d6de548b1e86c3eebe1ecbd5fe21dedd4bc39e7a2fdfa579b00cd99d549f6e0a","tier":"starter_2024"},"version":710,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"dd2940d75505b352","login":"barefootbiology","name":"S. Scott Whitmore","avatar_url":"https://avatars-next.observableusercontent.com/d6de548b1e86c3eebe1ecbd5fe21dedd4bc39e7a2fdfa579b00cd99d549f6e0a","tier":null}],"members":[{"type":"individual","id":"dd2940d75505b352","login":"barefootbiology","name":"S. Scott Whitmore","avatar_url":"https://avatars-next.observableusercontent.com/d6de548b1e86c3eebe1ecbd5fe21dedd4bc39e7a2fdfa579b00cd99d549f6e0a","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Resizing segmented vectors","cells":[{"id":0,"value":"# Resizing segmented vectors","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":568,"value":"## Motivation\n\nI study inherited retinal diseases. My projects often involve images collected in the ophthalmology clinic. I segment these images to identify different layers of the retina. I'm building a model that let's me artificially reduce some of these layers. This model will let me simulate some diseases.\n\nTo visualize the results, I need to modify each column of pixels in the original retinal image. Using R, I can produce the decent looking simulations:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":658,"value":"FileAttachment(\"bscan_simulation.png\").image()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":672,"value":"But R isn't great for interactivity--at least, not without a lot of work in Shiny. So now I'm trying to port my R code to Observable.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":16,"value":"## The problem simplified: One vector of pixel values\n\nSuppose we have a vector of voxel intensities. For simplicity, I'm going to make the voxel intensity values range from 0 to 9. This also happens to be the index of each voxel:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":3,"value":"// v0 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\nv0 = d3.range(0, 30)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":20,"value":"We have some initial segmentation of this vector. These initial segmentations will always be positioned at the inner surface of a layer, that is, where the layer begins. For now, we'll use integer positions*:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":13,"value":"s0 = [0, 5, 9, 12, 17, 25]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":179,"value":"## From surface positions to layer thicknesses","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":64,"value":"We can also compute the thickness of each layer from the surface vector. To do this, we need to know the length of the data vector:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":89,"value":"compute_thickness = function(layer_starts, length) {\n \n let result = layer_starts.map((d,i) => {\n if(i < (layer_starts.length - 1)) {\n return layer_starts[i+1] - layer_starts[i]\n }\n return length - layer_starts[i]\n })\n\n return result\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":103,"value":"t0 = compute_thickness(s0, v0.length)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":160,"value":"As a sanity check, we should get the original vector length if we sum up all the thickness values:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":123,"value":"v0.length == d3.sum(t0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":644,"value":"## From layer thicknesses to layers\n\nGiven what we know about the thicknness of each layer and the total length of the data vector, we can compute a new vector that represents the identity of each layer. We'll use a function that's like the `inverse.rle` function from the R programming language.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":643,"value":"// Inverse of run-length-encoded data, a la inverse.rle from R\ninverse_rle = function(lengths, values) {\n // TASK: ERROR if lengths.length != values.length\n\n // Create an array to hold the result\n let result_length = d3.sum(lengths)\n\n let result = new Array(result_length)\n\n // Position in result\n let r = 0;\n // For each of the values...\n for (let i = 0; i < values.length; i++) {\n // ... add it the number of times specified in `lengths`\n for(let j = 0; j < lengths[i]; j++) {\n result[r] = values[i]\n r++\n }\n }\n \n return result\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":642,"value":"layers = inverse_rle(t0, d3.range(0, t0.length))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":184,"value":"## Modifying layer thicknesses","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":186,"value":"Once we can represent the layers by their thickness values, we can modify the thickness values. Suppose we wanted to reduce the thickness of the 4th layer by 25%. Our constraint is that the total layer thickness must remain constant. So whatever we lose in thickness in the fourth layer must be added somewhere else. For this example, we'll add that missing thickness to the 1st layer.\n\nFirst, we'll construct a vector to indicate the change in layer thicknesses:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":196,"value":"t_reduce = [1, 1, 1, 0.75, 0, 1]\n// t_reduce = [1, 1, 1, 0, 1]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":198,"value":"Now we'll multiply the thickness vector by the reduction vector:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":202,"value":"t1 = t0.map((d, i) => d * t_reduce[i])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":207,"value":"Next we'll compute how much thickness has been lost:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":209,"value":"t1_lost = d3.sum(t0) - d3.sum(t1)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":216,"value":"add_missing_thickness = function(thickness_vector, missing_thickness, index) {\n return thickness_vector.map((d, i) => {\n if(i == index) {\n return d + missing_thickness\n } else {\n return d\n }\n })\n}\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":225,"value":"t2 = add_missing_thickness(t1, t1_lost, 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":221,"value":"Now the total thickness should be correct:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":219,"value":"d3.sum(t0) == d3.sum(t2)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":237,"value":"## From modified layer thicknesses to modified vectors","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":241,"value":"Now we can use the data we have to generate a modified version of the original vector. To do this, we'll have to make some assumptions about what values should occur within each segment.\n\nFor my motivating example, I have to goals:\n\n1. Maintain the order of elements within a segment (layer) of the vector.\n2. Allow for some randomness within the result.\n\nTo do this, we'll upsample or downsample the vector as needed, but we'll use a random order to do it. Our procedure, however, is complicated by the fact that after modification we no longer have integer thicknesses: some layers are partial. We'll need a way to deal with this problem.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":261,"value":"We have a couple of options. \n\n1. The easiest option is to round the thickness values to the nearest integer, then adjust the thickness of the first layer as needed. However, this procedure will mean that we'll get a bit of difference between our [new surface positions] and the new data vector.\n2. We can combine data vector elements based on how much they overlap in adjacent layers. For instance, [SHOW EXAMPLE].","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":246,"value":"## Randomly remove elements in a vector to shrink the vector","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":305,"value":"// x = vector\n// n = number of elements in new vector. Should be n <= x.length\nrandomly_remove = function(x, new_length){\n let n0 = x.length\n let index0 = d3.range(0, n0)\n\n let random_indices = new Array(n0 - new_length)\n\n let temp_index = index0.map(d => d)\n\n for(let i = 0; i < (n0 - new_length); i++) {\n let random_index = d3.randomInt(0, temp_index.length)()\n temp_index.splice(random_index, 1)\n }\n \n return temp_index.map((d, i) => x[d])\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":313,"value":"randomly_remove(d3.range(0, 10), 8)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":338,"value":" The returned array should always be sorted from lowest to highest index.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":352,"value":"// Unit test: The returned array should always be sorted from lowest to highest index.\n{ \n let test_input = d3.range(0, 10)\n let test_output = randomly_remove(test_input, 10)\n let test_output_sorted = d3.sort(test_output)\n let test_results = test_output_sorted.map((d, i) => +(d == test_input[i]))\n \n return (d3.sum(test_results) == test_input.length)\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":366,"value":"## Randomly inject elements in a vector to enlarge the vector\n\nWe can make a similar function to randomly inject values:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":369,"value":"randomly_inject = function(x, new_length) {\n let n0 = x.length\n let index0 = d3.range(0, n0)\n \n let result = new Array(x.length)\n\n for(let i = 0; i < x.length; i++) {\n result[i] = x[i]\n }\n\n \n // Where in the result vector should random values be injected?\n let inject_index = d3.range(0, new_length - x.length).map(d => d3.randomInt(0, new_length)())\n\n let inject_index_sorted = d3.sort(inject_index)\n\n // Which values will we inject into the result vector?\n let inject_values = d3.range(0, new_length - x.length).map(d => x[d3.randomInt(0, x.length)()])\n\n for(let j = 0; j < inject_index_sorted.length; j++) {\n result.splice(inject_index_sorted[j], 0, inject_values[j])\n }\n\n return result\n // return {\"result\": result, \"inject_index_sorted\": inject_index_sorted, \"inject_values\": inject_values}\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":398,"value":"randomly_inject(d3.range(0, 4), 9)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":394,"value":"new Array(10)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":452,"value":"We can now make a function that can either randomly remove or inject based on the new input length:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":457,"value":"randomly_inject_remove = function(x, new_length) {\n if(x.length == new_length) {\n return x\n } else if(x.length < new_length) {\n return randomly_inject(x, new_length)\n } \n return randomly_remove(x, new_length)\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":460,"value":"randomly_inject_remove(d3.range(0, 10), 10)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":462,"value":"randomly_inject_remove(d3.range(0, 10), 6)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":466,"value":"randomly_inject_remove(d3.range(0, 10), 12)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":426,"value":"## Dealing with non-integer thickness values\n\nThe simplest method for dealing with non-integer thickness values is to round the thickness values to the nearest integer. After rounding, we'll need to deal with any residual thickness by adjusting the first layer thickness.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":435,"value":"t3 = t2.map(d => Math.round(d))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":440,"value":"t3_lost = d3.sum(t3) - d3.sum(t2)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":443,"value":"t4 = add_missing_thickness(t3, t3_lost, 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":446,"value":"Now we have integer thickness values, at the expense of true layer thickness loss. For our motivating use case, this shouldn't matter much.*","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":469,"value":"## From layer thicknesses to surface positions\n\nWe can now generate the surface positions from the layer thicknesses:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":473,"value":"s4 = d3.cumsum(t3)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":477,"value":"s4","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":450,"value":"modify_vector = function(x, surfaces_original, thickness_modified){\n let result = new Array(x.length)\n\n let thickness_original = compute_thickness(surfaces_original, x.length)\n \n // Index in the result\n let j = 0\n\n // For each layer in the original\n for (let l = 0; l < thickness_modified.length; l++) {\n // Cut the values from the original vector in the original layer.\n let original_data = x.slice(surfaces_original[l], surfaces_original[l] + thickness_original[l])\n \n // Re-sample the values based on the new thickness.\n let new_data = randomly_inject_remove(original_data, thickness_modified[l])\n\n // For each value in the new data\n for(let i = 0; i < new_data.length; i++) {\n // Insert the results into the result vector\n result[j] = new_data[i]\n j++\n }\n }\n \n return result\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":583,"value":"t4","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":590,"value":"v4 = modify_vector(v0, s0, t4)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":703,"value":"s0","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":704,"value":"t4","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":536,"value":"## Visualizing the result\n\nTo visualize the result, let's put the relevant pieces of data into an object:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":528,"value":"data = d3.range(0, v0.length)\n .map(d => { return { \n \"index\": d, \n \"0_original_vector\": v0[d], \n \"1_original_layers\": layers[d], \n \"2_modified_layers\": inverse_rle(t4, d3.range(0, t4.length))[d], \n \"3_modified_vector\": v4[d] \n }})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":544,"value":"import {Wrangler, op} from '@observablehq/data-wrangler'","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":551,"value":"// Wrangler(data)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":549,"value":"import { aq } from '@uwdata/arquero'","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":563,"value":"// To use copied code replace \"data\" with your own variable\ndata_folded = aq.from(data)\n\t.fold(['0_original_vector','1_original_layers','2_modified_layers','3_modified_vector'])\n\t// .objects() // Uncomment to return an array of objects","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":576,"value":"t_reduce","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":543,"value":"plot_data = Plot.plot({\n marks: [\n Plot.cell(data_folded, {\n x: \"key\",\n y: \"index\",\n fill: \"value\"\n })\n ]\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":685,"value":"I'm now ready to apply these methods to my simulation model.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W2f","stdlib":"1","titling":"auto"}} 2e:"$2d:body"