2a:null 2b:["$","title",null,{"children":"Latent Flowers GANden | Zihou Ng | Observable"}] 30:T5d5,md`### Picking an architecture and dataset So the goal is to be able to render a frame in 16ms, here are some major considerations: 1. **Model size or number of parameters**. While smaller model means shorter waiting time, it would also be great to fit in the 15MB limitation of Observable attachments. Let's say a model with 4 millions \`float32\` parameters, the size is 16MB, after [quantization](https://pytorch.org/docs/stable/quantization.html) (if applicable) it will be roughly 4MB. 2. **Architecture complexity**. While beasts like StyleGAN surely not plausible, slightly sophisticated architecture like [SAGAN](https://arxiv.org/abs/1805.08318) also not a good fit here, just imagine how much pingponging needed between FBOs. DCGAN would be a good first step, since the layers are just strided convolutions and batchnorms. 3. **Dataset**. So what is an interesting dataset that could be captured by a DCGAN with 4 million parameters? Browsing the [catalog of standard datasets](https://www.tensorflow.org/datasets/catalog/overview), I chose the [Oxford 102 category flowers](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/) dataset. It has 8k images at 256x256 resolution, should be easy to learn and fit into a small model. Even if it can not, broken imagery should still be interesting with flowers. ${html`

`}`31:Tb6f,{ const w = 140, h = 15, gap = 30; const root = d3.select(DOM.svg(width, 580)); const main = root.append("g").attr("transform", `translate(100,5)`); const styles = { text: t => t.attr("font-family", "Open Sans") .attr("font-size", 10) .attr("dominant-baseline", "hanging"), layer: t => t.attr("fill", "hsl(0,0%,95%)").attr("stroke", "hsl(0,0%,50%)"), arrow: t => t.attr("fill", "hsl(0,0%,35%)").attr("stroke", "hsl(0,0%,35%)") }; const netG = main.append("g"); netG.append("text").call(styles.text) .attr("font-size", 16) .attr("x", 30) .text("Generator"); const netGInner = netG.append("g").attr("transform", "translate(0,25)"); const addBlock = (parent, loc, size, layers) => parent.append("g") .attr("transform", `translate(0,${loc})`) .call(g => { layers.forEach((name, i) => { g.append("g").attr("transform", `translate(0,${h * i})`) .call(g => { g.append("rect").call(styles.layer).attr("width", w).attr("height", h); g.append("text").call(styles.text).text(name).attr("x", 5).attr("y", 2); }); }); g.append("text").call(styles.text).text(size) .attr("y", h * (layers.length ) + 3); }); const line = d3.line(); const addArrow = (parent, loc, n) => { parent.append("g").call(g => { g.append("line") .attr("x1", w / 2).attr("y1", loc + h * n + 3) .attr("x2", w / 2).attr("y2", loc + h * n + gap - 3) .call(styles.arrow); g.append("path") .attr("d", line([[w / 2, loc + h * n + gap - 3], [w / 2 - 3, loc + h * n+ gap - 8], [w / 2 + 3, loc + h * n + gap - 8]])) .call(styles.arrow); }); }; let cur = 0; netGInner.call(addBlock, cur, "1 x 1 x 128", ["Input"]); netGInner.call(addArrow, cur, 1); cur += h + gap; for (let i = 2; i <= 7; i++) { const s = 2 ** i, d = 2 ** (9 - i); netGInner.call(addBlock, cur, `${s} x ${s} x ${d}`, ["ConvTranspose2d", "BatchNorm2d", "ReLU"]); netGInner.call(addArrow, cur, 3); cur += h * 3 + gap; } netGInner.call(addBlock, cur, "256 x 256 x 3", ["ConvTranspose2d", "Tanh"]); const netD = main.append("g").attr("transform", `translate(${w + 140},1)`); netD.append("text").call(styles.text) .attr("font-size", 16) .attr("x", 17) .text("Discriminator"); const netDInner = netD.append("g").attr("transform", "translate(0,25)"); cur = 0; netDInner.call(addBlock, cur, "256 x 256 x 3", ["Input"]); netDInner.call(addArrow, cur, 1); cur += h + gap; for (let i = 2; i <= 7; i++) { const s = 2 ** (9 - i), d = 2 ** (i + 1); netDInner.call(addBlock, cur, `${s} x ${s} x ${d}`, ["Conv2d", "LeakyReLU"]); netDInner.call(addArrow, cur, 2); cur += h * 2 + gap; } netDInner.call(addBlock, cur, "1 x 1 x 1", ["Conv2d"]); return root.node(); }32:T55c,md`### Running the model with WebGL In WebGL the manageable memory unit is textures, you can think of them as some 3D tensors with maximum depth of 4. Doing neural network computation with WebGL is all about fitting N-dimensional tensors in such constrained spaces. General purpose libraries like [TFJS](https://js.tensorflow.org/) handle these much complexity for users in many general cases. But in our case the problem is well defined enough, only [\`ConvTranspose2d\`](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html), [\`BatchNorm2d\`](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) and [\`Tanh\`](https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html) are involved. The most important is that we want to rule out every single bit of overhead to strive for 60FPS performance, so we have to do the computation from scratch. The trickiest part is the properly align the data for less texture reads, and the goal here is to leveage [\`dot\`](https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/dot.xhtml) as much as possible. A key observation is that the middle \`ConvTranspose2d\` layers are stride-1 with 4x4 kernels. Recall the [arithmetics of convolution](https://github.com/vdumoulin/conv_arithmetic), for such a convolution, each output element has 4 corresponding input elements, for each channel: `33:T8a3,{ const csize = 15, skew = 15; const cos = Math.cos(skew * math.DEG2RAD), sin = Math.sin(skew * math.DEG2RAD); const [h, w] = [8, 8]; const bg = "hsl(0,0%,70%)"; const padding = 15; const root = d3.select(DOM.svg(width, csize * h * (1 + sin) + padding * 2)); const main = root.append("g").attr("transform", `translate(100,${padding})`); const inputTensor = main.append("g") .attr("transform",`translate(0,${csize * w * sin}) skewY(-${skew})`); const highlight = new Set([[1,1],[1,3],[3,1],[3,3]].map(([i, j]) => i * w + j)); const kernel = new Set(d3.range(4).map(i => d3.range(4).map(j => j + i * w)).flat()); inputTensor.selectAll(".cell").data(d3.range(h * w)).join("rect") .attr("x", d => (d % w) * csize * cos) .attr("y", d => (d / w | 0) * csize) .attr("width", csize * cos).attr("height", csize) .attr("fill", i => highlight.has(i) ? bg : "none") .attr("stroke", "hsl(0,0%,50%)") .attr("stroke-dasharray", i => kernel.has(i) ? "1,0" : "1,3"); const outputTensor = main.append("g") .attr("transform",`translate(300,${csize * w * sin}) skewY(-${skew})`); outputTensor.selectAll(".cell").data(d3.range(h * w)).join("rect") .attr("x", d => (d % w) * csize * cos) .attr("y", d => (d / w | 0) * csize) .attr("width", csize * cos).attr("height", csize) .attr("fill", i => i == 0 ? bg : "none") .attr("stroke", "hsl(0,0%,50%)") .attr("stroke-dasharray", i => i == 0 ? "1,0" : "1,3"); main.append("line") .attr("x1", 0).attr("y1", csize * h * sin) .attr("x2", 300).attr("y2", csize * h * sin) .attr("stroke", bg); main.append("line") .attr("x1", (h / 2) * csize * cos).attr("y1", csize * (h / 2) * sin) .attr("x2", 300 + csize * cos).attr("y2", csize * sin * (h - 1)) .attr("stroke", bg); main.append("line") .attr("x1", 0).attr("y1", csize * h * sin + csize * (h / 2)) .attr("x2", 300).attr("y2", csize * (h * sin + 1)) .attr("stroke", bg); main.append("line") .attr("x1", (h / 2) * csize * cos).attr("y1", csize * (h / 2) * (sin + 1)) .attr("x2", 300 + csize * cos).attr("y2", csize * (sin * (h - 1) + 1)) .attr("stroke", bg); return root.node(); }34:Tcca,{ const csize = 15, skew = 15; const cos = Math.cos(skew * 2 * math.DEG2RAD), sin = Math.sin(skew * math.DEG2RAD); const [h, w] = [6, 6]; const bg = "hsl(0,0%,50%,0.6)"; const styles = { text: t => t.attr("font-family", "Open Sans") .attr("font-size", 12) .attr("dominant-baseline", "hanging") .attr("fill", "hsl(0,0%,20%)"), }; const padding = 15, gap = 60; const root = d3.select(DOM.svg(width, (csize * h * 2 + 20) * (1 + sin) + padding * 2)); const main = root.append("g").attr("transform", `translate(100,${padding})`); const inputTensor = main.append("g"); for (let i = 0; i < 4; i++) { inputTensor.append("g") .attr("transform",`translate(${i * 8},${csize * w * sin}) skewY(-${skew})`) .selectAll(".cell").data(d3.range(h * w)).join("rect") .attr("x", d => (d % w) * csize * cos) .attr("y", d => (d / w | 0) * csize) .attr("width", csize * cos).attr("height", csize) .attr("fill", i => i == 0 ? bg : "white") .attr("stroke", bg); } main.append("text").call(styles.text) .text("Input") .attr("x", csize * 2 + 10).attr("y", csize * h + 30); const kernel = main.append("g").attr("transform", `translate(0, ${csize * h + gap})`); for (let i = 0; i < 4; i++) { kernel.append("g") .attr("transform",`translate(${i * 8 + 15},${csize * w * sin}) skewY(-${skew})`) .selectAll(".cell").data(d3.range(16)).join("rect") .attr("x", d => (d % 4) * csize * cos) .attr("y", d => (d / 4 | 0) * csize) .attr("width", csize * cos).attr("height", csize) .attr("fill", i => i == 0 ? bg : "white") .attr("stroke", bg); } main.append("text").call(styles.text) .text("Kernel") .attr("x", csize * 2 + 8).attr("y", csize * h * 2 + gap + 5); const output = main.append("g").attr("transform", `translate(300, 60)`); for (let i = 0; i < 4; i++) { output.append("g") .attr("transform",`translate(${i * 8},${csize * w * sin}) skewY(-${skew})`) .selectAll(".cell").data(d3.range(w * h)).join("rect") .attr("x", d => (d % w) * csize * cos) .attr("y", d => (d / w | 0) * csize) .attr("width", csize * cos).attr("height", csize) .attr("fill", j => j == 0 && i == 3 ? "hsl(0,0%,70%)" : "white") .attr("stroke", bg); } main.append("text").call(styles.text) .text("Output") .attr("x", 300 + csize * 2 + 8).attr("y", csize * h + 95); main.append("text").call(styles.text) .attr("font-size", 14).attr("font-family", "Courier") .text("dot()") .attr("x", 190).attr("y", 85); main.append("line") .attr("x1", 24 + csize / 2).attr("y1", csize / 2 + csize * h * sin) .attr("x2", 180).attr("y2", 90) .attr("stroke", bg); main.append("line") .attr("x1", 24 + csize * 1.5).attr("y1", csize * h + gap + 30) .attr("x2", 180).attr("y2", 94) .attr("stroke", bg); main.append("line") .attr("x1", 240).attr("y1", 92) .attr("x2", 295).attr("y2", 92) .attr("stroke", bg); main.append("path") .attr("d", d3.line()([[295, 92], [290, 87], [290, 97]])) .attr("fill", "hsl(0,0%,50%)"); return root.node(); }35:T6c2,show = new class { constructor() { const [W, H] = params.size; this.program = defProgram(` const float size = 256.; const float large = 400.; float coord(float i, float s) { return (i + 0.5) / s; } vec3 getPixel(vec2 p) { p += vec2(1.0); float uvx = coord(p.x, size); float r = texture2D(tex, vec2(uvx, coord(p.y, ${H}.))).r; float g = texture2D(tex, vec2(uvx, coord(p.y + ${W}., ${H}.))).r; float b = texture2D(tex, vec2(uvx, coord(p.y + ${W * 2}., ${H}.))).r; return vec3(r, g, b); } vec3 adjustSaturation(vec3 color, float value) { // https://www.w3.org/TR/WCAG21/#dfn-relative-luminance const vec3 luminosityFactor = vec3(0.2126, 0.7152, 0.0722); vec3 grayscale = vec3(dot(color, luminosityFactor)); return mix(grayscale, color, 1.0 + value); } vec3 adjustContrast(vec3 color, float value) { return 0.5 + (1.0 + value) * (color - 0.5); } vec3 adjustBrightness(vec3 color, float value) { return color + value; } void main() { vec2 uv1 = floor(vec2(v_uv.x, 1. - v_uv.y) * large); vec2 uv2 = uv1 * size / large; vec2 frac = fract(uv2); vec2 uv3 = floor(uv2); vec2 tl = uv3; vec2 tr = min(vec2(tl.x + 1., tl.y), size - 1.); vec2 bl = min(vec2(tl.x, tl.y + 1.), size - 1.); vec2 br = min(tl + 1., size - 1.); // bilinear interpolation vec3 vl = mix(getPixel(tl), getPixel(bl), frac.y); vec3 vr = mix(getPixel(tr), getPixel(br), frac.y); vec3 color = mix(vl, vr, frac.x); color = adjustBrightness(color, 0.02); color = adjustSaturation(color, 0.05); color = adjustContrast(color, 0.05); gl_FragColor = vec4(color, 1.0); }`, { tex: ["sampler2D", 0] }); } draw(tex) { this.program.textures = [tex]; webgl.draw(this.program); } }36:Tfe9,controller = state => { const ncol = 8, lh = 15, cw = 50, sSize = [(ncol + 1) * cw, Math.ceil(params.nz / ncol) * lh]; const rcSize = 150, radius = rcSize / 2; const root = d3.select(html`

`); let cancel = noop, walk = true, tid; const stopWalk = () => { cancel(); clearTimeout(tid); walk = false; }; const sMain = root.select('.sliders'); const scale = d3.scaleLinear([-3.5,3.5],[2,cw]).clamp(true); const sliders = sMain.selectAll(".sliders").data(indexed(state.value)).join("rect") .attr("x", (_, i) => (i % ncol) * (cw + 5)) .attr("y", (_, i) => (i / ncol | 0) * lh) .attr("width", d => scale(d[1])) .attr("height", lh-4) .attr("fill", "hsl(0,0%,95%)") .on("mouseover", e => d3.select(e.target).attr("fill", "hsl(0,0%,70%)")) .on("mouseout", e => d3.select(e.target).attr("fill", "hsl(0,0%,95%)")) .call(d3.drag().on("drag", (e, d) => { stopWalk(); const i = d[0]; const dx = e.x - (i % ncol) * (cw + 5); state.value[i] = scale.invert(dx); state.value = state.value; })); const chart = new Chart(radius, [-3.5, 3.5]); const chartSvg = root.select(".chart"); chartSvg.call(chart.setup); const scales = { r: d3.scaleLinear([-3.5, 3.5], [0, radius - 1]).clamp(true), index: d3.scaleQuantize([-Math.PI, Math.PI], d3.range(params.nz)) }; const center = [radius, radius]; chartSvg.call(d3.drag().on("drag", e => { const angle = v.angleBetween2(v.Y2, v.sub2(null, [e.x, e.y], center)); const idx = scales.index(angle); state.value[idx] = scales.r.invert(v.dist2(center, [e.x,e.y])); chart.update(state.value); state.value = state.value; stopWalk(); })); const btnRand = root.select("#random"); const chkInf = root.select("#walk"); const unsub = state.observe(value => { chkInf.property("checked", walk); chart.update(value); sliders.data(indexed(value)).attr("width", d => scale(d[1])); }); const randInt = d3.randomInt(embeddings.shape[0]); const randT = d3.randomUniform(3000, 8000); const tmp = new Float32Array(params.nz); btnRand.on("click", e => { stopWalk(); const i = randInt(); const cur = state.value.slice(); const target = embeddings.data .slice(i * params.nz, (i + 1) * params.nz) .map(x => x + randn() * 0.01); cancel = raf(transition(t => { state.value = slerp(tmp, cur, target, d3.easeCubicOut(t)); }, 1000)); }); let target = embeddings.data.slice(10 * params.nz, 11 * params.nz); let cur = state.value.slice(); const run = () => { let t = 0; cur = state.value.slice(); const reset = () => { const i = randInt(); const temp = embeddings.data.slice(i * params.nz, (i + 1) * params.nz); slerp(target, state.value, temp, 0.5); cur = state.value.slice(); t = 0; tid = setTimeout(reset, randT()); }; reset(); cancel = raf(() => { t += 0.007; state.value = slerp(tmp, cur, target, t);; }); }; chkInf.on("change", e => { cancel(); clearTimeout(tid); if (walk = e.target.checked) run(); }); run(); return [root.node(), () => { unsub(); stopWalk(); chartSvg.on(".drag", null); btnRand.on("click", null); chkInf.on("change", null); sliders .on("mouseover", null) .on("mouseout", null) .on(".drag", null); }]; }37:T56e,class Chart { constructor(radius, valRange = [0,1]) { this.radius = radius; this.r = d3.scaleLinear(valRange, [0, radius - 1]).clamp(true); this.setup = this.setup.bind(this); this.upddate = this.update.bind(this); } setup(main) { const radius = this.radius; main.append("rect") .attr("width", radius * 2) .attr("height", radius * 2) .attr("stroke", "hsl(0,0%,85%)") .attr("fill", "none"); const g = main.append("g").attr("transform", `translate(${radius},${radius})`); g.append("g").classed("axis-r", true) .selectAll("circle").data(this.r.ticks(5).slice(1)) .join("circle") .attr("r", this.r) .attr("fill", "none") .attr("stroke", "hsl(0,0%,40%)") .attr("stroke-dasharray", "1,2"); g.append("g").classed("axis-a", true) .selectAll("line").data(d3.range(0, 360, 30)) .join("line") .attr("transform", d => `rotate(${-d})`) .attr("x2", radius - 5) .attr("stroke", "hsl(0,0%,40%)") .attr("stroke-dasharray", "1,2"); this.path = g.append("path").classed("line", true) .attr("fill", "none") .attr("stroke", "hsl(0,0%,90%)"); this.line = d3.lineRadial().curve(d3.curveBasisClosed) .angle((_, i, l) => i / l.length * math.TAU) .radius(this.r); } update(data) { this.path.attr("d", this.line(data)); } }38:T4d7,html``39:T4cf,class Generator { constructor() { this.input = new Buffer(128, 1); this.output1 = new Buffer(params.size[0] * 4, params.size[1]); this.output2 = new Buffer(params.size[0] * 4, params.size[1]); this.convStem = new ConvTStem(layers[0].params, layers[1].params, this.input); this.convBlock1 = new ConvTBlock(layers[2].params, layers[3].params, [5, 5]); this.convBlock2 = new ConvTBlock(layers[4].params, layers[5].params, [9, 9]); this.convBlock3 = new ConvTBlock(layers[6].params, layers[7].params, [17, 17]); this.convBlock4 = new ConvTBlock(layers[8].params, layers[9].params, [33, 33]); this.convBlock5 = new ConvTBlock(layers[10].params, layers[11].params, [65, 65]); this.convHead = new ConvTHead(layers[12].params, [129,129]); } forward(data) { this.input.update(data); this.convStem.forward(this.output1, this.input); this.convBlock1.forward(this.output2, this.output1); this.convBlock2.forward(this.output1, this.output2); this.convBlock3.forward(this.output2, this.output1); this.convBlock4.forward(this.output1, this.output2); this.convBlock5.forward(this.output2, this.output1); this.convHead.forward(this.output1, this.output2); } }3a:Ta35,class ConvTHead { constructor({ weight, bias }, inputSize, totSize = params.size) { const [HH, WW] = totSize; const [H, W] = inputSize; const [C, K, R, S] = weight.shape;; const [P, Q] = [H * 2 - 1, W * 2 - 1]; this.texSize = [P, Q * K]; this.fs = ` const float H = ${H}., W = ${W}.; const float P = ${P}., Q = ${Q}., K = ${K}., C = ${C}.; float coord(float i, float s) { return (i + 0.5) / s; } float outputIndex(vec2 xy, float size) { vec2 t = step(1e-6, xy) * (1.0 - step(size, xy)); return mix(-1.0, (xy.y - 1.0) * (size - 1.0) + xy.x - 1.0, t.x * t.y); } vec2 inputIndex(float idx, float size) { float r = floor(idx / size), c = mod(idx, size); return floor((vec2(c, r) + 1.0) / 2.0); } float kernelIndex(float idx, float size) { float r = floor(idx / size), c = mod(idx, size); return mod(r, 2.0) * 2.0 + mod(c, 2.0); } vec2 spatialIndex(vec2 uv, float P, float Q, float K) { float y = mod(floor(uv.y * K * P), P); float x = floor(uv.x * Q); return vec2(x, y); } float collect(float idx) { float res = 0.0; float kid = kernelIndex(idx, Q - 1.0); vec2 iid = inputIndex(idx, Q - 1.0); for (int c = 0; c < int(C); c++) { vec4 f = texture2D(weight, vec2(coord(float(c) * 4.0 + kid, C * 4.0), v_uv.y)); vec4 v = texture2D(inputs, vec2(coord(iid.x, ${HH}.), coord(float(c) * H + iid.y, ${WW}.))); res += dot(f, v); } return res; } vec4 tanh(vec4 x) { vec4 e = exp(x); return (e - 1./e) / (e + 1./e); } void main() { vec2 xy = spatialIndex(v_uv, P, Q, K); float idx0 = outputIndex(xy, Q); float idx1 = outputIndex(xy + vec2(1.0, 0.0), Q); float idx2 = outputIndex(xy + vec2(0.0, 1.0), Q); float idx3 = outputIndex(xy + vec2(1.0, 1.0), Q); float b = texture2D(bias, vec2(0.5, v_uv.y)).r; vec4 result = vec4(0.0); result[0] = (collect(idx0) + b) * step(0., idx0); result[1] = (collect(idx1) + b) * step(0., idx1); result[2] = (collect(idx2) + b) * step(0., idx2); result[3] = (collect(idx3) + b) * step(0., idx3); result = tanh(result); result += 1.0; result *= 0.5; gl_FragColor = result; }`; this.weight = new Buffer(C * R * S, K, prepareKernel421(weight.transpose([1,0,2,3]).flip([2,3]).data)) this.bias = new Buffer(4, K, dilate(bias.data)); this.program = defProgram(this.fs, { inputs: ["sampler2D", 0], weight: ["sampler2D", 1], bias: ["sampler2D", 2], bnParams: ["sampler2D", 3] }); } forward(output, input) { this.program.textures = [input.tex, this.weight.tex, this.bias.tex]; output.draw(this.program, this.texSize); } }3b:Tbc9,class ConvTBlock { constructor(conv, bn, inputSize, totSize = params.size) { const [HH, WW] = totSize; const [H, W] = inputSize; const [C, K, R, S] = conv.weight.shape;; const [P, Q] = [H * 2 - 1, W * 2 - 1]; this.texSize = [P, Q * K]; this.fs = ` const float H = ${H}., W = ${W}.; const float P = ${P}., Q = ${Q}., K = ${K}., C = ${C}.; float coord(float i, float s) { return (i + 0.5) / s; } float outputIndex(vec2 xy, float size) { vec2 t = step(1e-6, xy) * (1.0 - step(size, xy)); return mix(-1.0, (xy.y - 1.0) * (size - 1.0) + xy.x - 1.0, t.x * t.y); } vec2 inputIndex(float idx, float size) { float r = floor(idx / size), c = mod(idx, size); return floor((vec2(c, r) + 1.0) / 2.0); } float kernelIndex(float idx, float size) { float r = floor(idx / size), c = mod(idx, size); return mod(r, 2.0) * 2.0 + mod(c, 2.0); } vec2 spatialIndex(vec2 uv, float P, float Q, float K) { float y = mod(floor(uv.y * K * P), P); float x = floor(uv.x * Q); return vec2(x, y); } float collect(float idx) { float res = 0.0; float kid = kernelIndex(idx, Q - 1.0); vec2 iid = inputIndex(idx, Q - 1.0); for (int c = 0; c < int(C); c++) { vec4 f = texture2D(weight, vec2(coord(float(c) * 4.0 + kid, C * 4.0), v_uv.y)); vec4 v = texture2D(inputs, vec2(coord(iid.x, ${HH}.), coord(float(c) * H + iid.y, ${WW}.))); res += dot(f, v); } return res; } float batchnorm(float x, vec4 params) { x -= params.x; x /= params.y; x *= params.z; x += params.w; return x; } void main() { vec2 xy = spatialIndex(v_uv, P, Q, K); float idx0 = outputIndex(xy, Q); float idx1 = outputIndex(xy + vec2(1.0, 0.0), Q); float idx2 = outputIndex(xy + vec2(0.0, 1.0), Q); float idx3 = outputIndex(xy + vec2(1.0, 1.0), Q); float b = texture2D(bias, vec2(0.5, v_uv.y)).r; vec4 bn = texture2D(bnParams, vec2(0.5, v_uv.y)); // mu, sigma, weight, bias vec4 result = vec4(0.0); result[0] = batchnorm(collect(idx0) + b, bn) * step(0., idx0); result[1] = batchnorm(collect(idx1) + b, bn) * step(0., idx1); result[2] = batchnorm(collect(idx2) + b, bn) * step(0., idx2); result[3] = batchnorm(collect(idx3) + b, bn) * step(0., idx3); result = max(result, 0.0); gl_FragColor = result; }`; this.weight = new Buffer(C * R * S, K, prepareKernel421(conv.weight.transpose([1,0,2,3]).flip([2,3]).data)) this.bias = new Buffer(4, K, dilate(conv.bias.data)); const params = Tensor.cat([ bn.mu.unsqueeze(1), bn.sigma.sqrt().unsqueeze(1), bn.weight.unsqueeze(1), bn.bias.unsqueeze(1) ], 1); this.bnParams = new Buffer(params.shape[1], params.shape[0], params.data); this.program = defProgram(this.fs, { inputs: ["sampler2D", 0], weight: ["sampler2D", 1], bias: ["sampler2D", 2], bnParams: ["sampler2D", 3] }); } forward(output, input) { this.program.textures = [input.tex, this.weight.tex, this.bias.tex, this.bnParams.tex]; output.draw(this.program, this.texSize); } }3c:T9d4,class ConvTStem { constructor(conv, bn) { const [C, K, R, S] = conv.weight.shape;; const [P, Q] = [5, 5]; this.texSize = [P, Q * K]; this.fs = ` const float P = ${P}., Q = ${Q}., K = ${K}., C = ${C}.; const int C4 = ${Math.ceil(C / 4)}; float coord(float i, float s) { return (i + 0.5) / s; } float outputIndex(vec2 xy, float size) { vec2 t = step(1e-6, xy) * (1.0 - step(size, xy)); return mix(-1.0, (xy.y - 1.0) * (size - 1.0) + xy.x - 1.0, t.x * t.y); } vec2 spatialIndex(vec2 uv, float P, float Q, float K) { float y = mod(floor(uv.y * K * P), P); float x = floor(uv.x * Q); return vec2(x, y); } float collect(float idx) { float res = 0.0; for (int c = 0; c < C4; c++) { vec4 f = texture2D(weight, vec2(coord(float(c) + idx * float(C4), float(C4) * ${R * S}.), v_uv.y)); vec4 v = texture2D(inputs, vec2(coord(float(c), float(C4)), v_uv.y)); res += dot(f, v); } return res; } float batchnorm(float x, vec4 params) { x -= params.x; x /= params.y; x *= params.z; x += params.w; return x; } void main() { vec2 xy = spatialIndex(v_uv, P, Q, K); float idx0 = outputIndex(xy, Q); float idx1 = outputIndex(xy + vec2(1.0, 0.0), Q); float idx2 = outputIndex(xy + vec2(0.0, 1.0), Q); float idx3 = outputIndex(xy + vec2(1.0, 1.0), Q); float b = texture2D(bias, vec2(0.5, v_uv.y)).r; vec4 bn = texture2D(bnParams, vec2(0.5, v_uv.y)); // mu, sigma, weight, bias vec4 result = vec4(0.0); result[0] = batchnorm(collect(idx0) + b, bn) * step(0., idx0); result[1] = batchnorm(collect(idx1) + b, bn) * step(0., idx1); result[2] = batchnorm(collect(idx2) + b, bn) * step(0., idx2); result[3] = batchnorm(collect(idx3) + b, bn) * step(0., idx3); result = max(result, 0.0); gl_FragColor = result; }`; this.weight = new Buffer(C * R * S, K, conv.weight.transpose([1,2,3,0]).data); this.bias = new Buffer(4, K, dilate(conv.bias.data)); const params = Tensor.cat([ bn.mu.unsqueeze(1), bn.sigma.sqrt().unsqueeze(1), bn.weight.unsqueeze(1), bn.bias.unsqueeze(1) ], 1); this.bnParams = new Buffer(params.shape[1], params.shape[0], params.data); this.program = defProgram(this.fs, { inputs: ["sampler2D", 0], weight: ["sampler2D", 1], bias: ["sampler2D", 2], bnParams: ["sampler2D", 3] }); } forward(output, input) { this.program.textures = [input.tex, this.weight.tex, this.bias.tex, this.bnParams.tex]; output.draw(this.program, this.texSize); } }3d:Td34,class Tensor { static stridesOf(shape) { const strides = [1]; for (let i = shape.length - 1; i > 0; i--) { strides.unshift(shape[i] * strides[0]); } return strides; } static cat(tensors, dim) { const n = tensors.length; const shape = tensors[0].shape, strides = tensors[0].strides; const m = shape.reduce((a, b) => a * b, 1); const data = new Float32Array(m * n); let ii = 0; const steps = shape.slice(dim).reduce((a, b) => a * b, 1); const rec = (base, d) => { if (d == dim) { for (let i = 0; i < n; i++) { for (let j = 0; j < steps; j++) { data[ii++] = tensors[i].data[base + j]; } } } else { for (let i = 0; i < shape[d]; i++) rec(base + i * strides[d], d + 1); } }; rec(0, 0); return new Tensor(data, shape.map((d,i) => d * (i == dim ? n : 1))); } constructor(data, shape) { this.data = data; this.shape = shape; this.strides = Tensor.stridesOf(shape); } pad(pads, value = 0) { const { shape, strides, data } = this; assert(0 < pads.length && pads.length / 2 <= shape.length, "Invalid padding length"); assert(pads.length % 2 == 0, "Padding length must be divisible by 2"); const shape1 = d3.reverse(shape).map((d, i) => { if (i * 2 < pads.length) d += pads[i * 2] + pads[i * 2 + 1]; return d; }).reverse(); const prePads = d3.reverse(shape).map((d, i) => i * 2 < pads.length ? pads[i * 2] : 0).reverse(); const strides1 = Tensor.stridesOf(shape1); const newData = new Float32Array(array.prod(shape1)); newData.fill(value); const n = data.length; for (let i = 0; i < n; i++) { let k = i; const res = []; for (let j = shape.length - 1; j >= 0; j--) { res.unshift(k % shape[j]); k = Math.floor(k / shape[j]); } const idx = res.reduce((a, s, i) => a + (s + prePads[i]) * strides1[i], 0); newData[idx] = data[i]; } return new Tensor(newData, shape1); } unsqueeze(dim) { const shape = this.shape.slice(); shape.splice(dim, 0, 1); return new Tensor(this.data.slice(), shape); } sqrt() { return new Tensor(this.data.map(x => Math.sqrt(x + 1e-9)), this.shape); } transpose(dims) { const { data, shape, strides } = this; assert(dims.length == shape.length, "dims should be equal to shape"); const newData = new Float32Array(data.length); let ii = 0; const rec = (base, d) => { if (d == dims.length) { newData[ii++] = data[base]; } else { for (let i = 0; i < shape[dims[d]]; i++) rec(base + i * strides[dims[d]], d + 1); } }; rec(0, 0); return new Tensor(newData, dims.map(i => shape[i])); } flip(dims) { const { data, shape, strides } = this; const dimSet = new Set(dims); const newData = new Float32Array(data.length); let ii = 0; const rec = (base, d) => { if (d == shape.length) { newData[ii++] = data[base]; } else if (dimSet.has(d)) { for (let i = shape[d] - 1; i >= 0; i--) rec(base + i * strides[d], d + 1); } else { for (let i = 0; i < shape[d]; i++) rec(base + i * strides[d], d + 1); } }; rec(0, 0); return new Tensor(newData, shape); } }3e:T516,fetchProgress = { function concat(arrays) { let i = 0, n = 0; for (const a of arrays) n += a.length; const concat = new Uint8Array(n); for (let a of arrays) concat.set(a, i), i += a.length; return concat.buffer; } const progressText = (cur, total) => { const max = total > 0 ? `${total / 1024 | 0}` : '?'; return `${cur / 1024 | 0} / ${max} kb`; }; return (name, url, init) => { const root = html`

${name} waiting...

`; const progress = root.children[1]; const text = root.children[2]; fetch(url, init).then(async response => { const max = response.headers.get("content-length"); progress.max = max || Number.MAX_SAFE_INTEGER; text.textContent = progressText(0, max); const reader = response.body.getReader(); const values = []; while (true) { const {done, value} = await reader.read(); if (done) break; progress.value += value.length; text.textContent = progressText(progress.value, max); values.push(value); } root.value = concat(values); root.dispatchEvent(new CustomEvent("input")); }); return root; }; }3f:T740,class View { static create(initialize) { return new Promise(resolve => { const view = new View(); let resolved = false; const dispose = initialize((val, state = val) => { view.value = val; view.state = state; if (!resolved) { resolved = true; resolve(Generators.disposable(view, () => { dispose && dispose(); view.dispose(); })); }; }) }); } static state(view, defaults) { return !!view ? view.state : defaults; } static subs(unsubs) { return () => unsubs.forEach(unsub => unsub()); } constructor(value) { Object.defineProperties(this, { _value: { value, writable: true }, _list: { value: {}, writable: true }, _subs: { value: [], writable: true }, }); const dispatch = e => this._subs.forEach(sub => sub(e.value)); this.addEventListener('input', dispatch); this.dispose = () => this.removeEventListener("input", dispatch); }; get value() { return this._value; } set value(val) { this._value = val; this.notify(); } observe(sub) { this._subs.push(sub); sub(this.value); // give out the current value on subscription return () => this._subs = this._subs.filter(l => l !== sub); } notify() { this.dispatchEvent({ type: "input", value: this.value }); } addEventListener(type, listener) { if (!this._list[type]) this._list[type] = []; if (this._list[type].includes(listener)) return; this._list[type] = [listener].concat(this._list[type]); } removeEventListener(type, listener) { this._list[type] = this._list[type].filter(l => l !== listener); } dispatchEvent(event) { if (!this._list[event.type]) return; const p = Promise.resolve(event); this._list[event.type].forEach(l => p.then(l)); } }40:[{"id":"17258116c06fdba792afddf23457b8104a5a940a18d1cecea0db7c44859e46391abf2dee2d32a1716b780c50029c7d576e1176e67a4d75a3ed64dccba2eed26e","href":"https://static.observableusercontent.com/files/17258116c06fdba792afddf23457b8104a5a940a18d1cecea0db7c44859e46391abf2dee2d32a1716b780c50029c7d576e1176e67a4d75a3ed64dccba2eed26e","name":"random_embedding_epoch_1400_s.png","type":"image/png","lastModified":1612588272214.984,"size":1026050},{"id":"64dd2197ba31896cf0f0bfb278cc8d0691a6279bad69d6714d858cf5f9ad8df8432ed9aeae00b926ec002b042584c83da4896674bb87169a17d5d504bdc68997","href":"https://static.observableusercontent.com/files/64dd2197ba31896cf0f0bfb278cc8d0691a6279bad69d6714d858cf5f9ad8df8432ed9aeae00b926ec002b042584c83da4896674bb87169a17d5d504bdc68997","name":"g_emb_1399.msgpack","type":"application/octet-stream","lastModified":1612541951775.431,"size":5951785},{"id":"3e6196d347480174ab8171fe405b49434b12bbddce216949b8481938022aef8f2076fdefc3219da587408b0a65fc0b3e5691f8a151227036a68564fe01d829cd","href":"https://static.observableusercontent.com/files/3e6196d347480174ab8171fe405b49434b12bbddce216949b8481938022aef8f2076fdefc3219da587408b0a65fc0b3e5691f8a151227036a68564fe01d829cd","name":"train_samples.png","type":"image/png","lastModified":1612589010800.248,"size":983796}] 2e:{"id":"26cf19b15d6c4657","slug":"latent-flowers-ganden","title":"Latent Flowers GANden","thumbnail":"448df1bc823fb3815224a79bcc6d865d77cf3ae341624076ddcbbdc53e99fa41","access_level":"public","create_time":"$D2021-02-04T03:37:38.634Z","update_time":"$D2021-04-14T03:46:06.327Z","view_time":null,"first_public_time":"$D2021-02-06T11:18:12.653Z","has_unpublished_edits":null,"fork":true,"stars":9,"starred":false,"owner":{"type":"team","id":"7fad80bf128915e9","login":"stwind","name":"Zihou Ng","avatar_url":"https://avatars-next.observableusercontent.com/ff8a8f73ad220dcaec180c1e813951d692777837355004f062692b49042d2458","tier":"starter_2024"},"version":10749,"publish_version":10749,"role":null,"authored":false,"authors":[{"type":"individual","id":"dbb85afd665c9e5e","login":"stwind","name":"Zihou Ng","avatar_url":"https://avatars-next.observableusercontent.com/ff8a8f73ad220dcaec180c1e813951d692777837355004f062692b49042d2458","tier":null}],"members":[{"type":"individual","id":"dbb85afd665c9e5e","login":"stwind","name":"Zihou Ng","avatar_url":"https://avatars-next.observableusercontent.com/ff8a8f73ad220dcaec180c1e813951d692777837355004f062692b49042d2458","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":4,"collection":{"id":"bbe30167c2f9fa03","slug":"machine-learning","title":"Machine Learning","owner":{"id":"7fad80bf128915e9","login":"stwind","name":"Zihou Ng"}},"type":"notebook","body":{"titling":"auto","title":"Latent Flowers GANden","cells":[{"id":0,"value":"md`# Latent Flowers GANden`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8883,"value":"md`An experiment of realtime latent space navigation of [GAN](https://en.wikipedia.org/wiki/Generative_adversarial_network). \n\nI trained a [DCGAN](https://arxiv.org/abs/1511.06434) on the [Oxford 102 Category Flowers](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/) dataset and managed to squeeze the generator to the brower for realtime controls.\n\n_Currently only desktop browsers are supported._\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9432,"value":"viewof modelData = fetchProgress(\"model\", await FileAttachment(\"g_emb_1399.msgpack\").url())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":124,"value":"viewof gl = {\n const canvas = DOM.canvas(params.width, params.height);\n const gl = canvas.getContext('webgl');\n gl.getExtension('OES_texture_float');\n gl.getExtension('WEBGL_draw_buffers');\n gl.getExtension('WEBGL_color_buffer_float');\n // gl.getExtension('EXT_color_buffer_float');\n \n const [elm, dispose] = controller(viewof z);\n const root = html`

${canvas}

\n ${elm}\n

`;\n root.value = gl;\n invalidation.then(dispose);\n return root;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9466,"value":"md`---\n## Notes`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9477,"value":"md`### Motivation\n\nThe latent space of GAN is fascinating, interpolation between is always a pleasure to watch. But due to the long generation time, almost any interesting results are in video form, thus a linear experience. I was wondering is it possible to have latent space walking experience in realtime (at least 30+ FPS), beyond simple datasets like MNIST , and at higher resolutions like 256x256.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9495,"value":"$30","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9561,"value":"md`### Training the GAN\n\nA technique I learned from [CodeParade](https://github.com/HackerPoet/Avant-Garfield) is to place an [embedding](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html) layer before the generator. The learned embedding from training samples could serve as anchor points in the latent space, these points generally represent more recognizable features than a random z vector.\n\nOther modifications include\n\n* Use [Spectral Normalization](https://arxiv.org/abs/1802.05957) on the discriminator.\n* Use [Wasserstein loss](https://arxiv.org/abs/1701.07875) with [relativistic factor](https://arxiv.org/abs/1807.00734) to stabilize training.\n* An extra [SSIM](https://www.cns.nyu.edu/~lcv/ssim/) term on the loss function as slight push toward the real sample.\n\nAfter some trials and errors, here is the architecutre I end up using:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9629,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10051,"value":"md`The model was trained for 1400 epochs on Colab for 28 hours. Here are some generated samples from embedding vectors:\n\n${html`

Left: random embedding samples. Right: some enlarged.

`}\n\nMany of them has recognizable flower features, and the broken ones have some special atmosphere like a painting. To me that is quite accetable for such a small model. Actually at ~800 epochs the results are good enough, there are not much improvements after that.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10036,"value":"md`The generator has 0.2 [GMAC](https://github.com/sovrasov/flops-counter.pytorch/issues/16)s and ~1.22M parameters, and the 128-dimensional embeddings for 8k training samples has ~1M parameters. So the generator alone has size of ~4MB, which is acceptable so we are not going to quantize it. The embeddings size is also 4MB and it is ok to have less precision, so we quantized it to 1MB. The final total size is ${modelData.byteLength/1024|0} kb.\n\nThe training code is available on this [Colab notebook](https://colab.research.google.com/github/stwind/gan-experiments/blob/master/notebooks/wgan_specnorm_flower102.ipynb).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9579,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10125,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10451,"value":"md`Remember that a texture has 4 channels, that is where take advantage of. By carefully aligning tensors this way, we could finish one step of convolution in a single \\`dot()\\` operation with two texture fetches (one for input, another for weights).`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10490,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10674,"value":"md`### UI\n\nThree levels of control are supported\n\n* A drawable radar chart for gross movement.\n* Sliders for more fine-grain control of each dimension.\n* Random walking.\n\n### Results\n\nThe final result was bilinearly scaled to 400x400, the artifacts are acceptable. On my Macbook Pro 2017 it was running fine with around 50~55FPS. \n\nFor the generate images, many chances are that it comes with lots of high frequency noises, the reason could possibily be how the model is regularized.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":715,"value":"md`---\n## Code`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":3176,"value":"render = {\n generator.forward(z);\n \n gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);\n gl.clearColor(0, 0, 0, 1.0);\n gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);\n\n show.draw(generator.output1.tex);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7357,"value":"viewof z = new View(new Float32Array(d3.range(params.nz).map(randn)))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":3171,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8285,"value":"generator = new Generator()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":128,"value":"params = ({\n width: 400, height: 400,\n size: [257, 1280],\n nz: 128\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9469,"value":"md`### Data`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9439,"value":"layers = decoded.layers.map(obj => {\n const params = {};\n for (const name in obj.params) {\n params[name] = new Tensor(new Float32Array(new Int8Array(obj.params[name].data).buffer),\n obj.params[name].shape);\n }\n\n return { params, type: obj.type };\n});","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9437,"value":"embeddings = dequantizeTensor(decoded.embedding)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9434,"value":"decoded = msgpack.decode(modelData)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9456,"value":"md`### UI`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7370,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7663,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8495,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":182,"value":"md`### Neural Network Modules`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8267,"value":"$39","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5890,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5669,"value":"$3b","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":4817,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":119,"value":"md`### GPGPU`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1344,"value":"class Buffer {\n constructor(width, height, data = null) {\n this.width = width;\n this.height = height;\n this.tex = webgl.defTextureFloat(gl, data, width / 4, height);\n this.fbo = webgl.defFBO(gl, { tex: [this.tex] });\n }\n \n read(size) {\n const { format, type } = this.tex;\n const out = new Float32Array(array.prod(size) * 4);\n this.fbo.bind();\n gl.readPixels(0, 0, size[0], size[1], gl.RGBA, gl.FLOAT, out);\n this.fbo.unbind();\n return out;\n }\n \n update(data) {\n const { target, format, type, size } = this.tex;\n this.tex.bind();\n gl.texSubImage2D(target, 0, 0, 0, size[0], size[1], gl.RGBA, type, data, 0);\n this.tex.unbind();\n }\n \n draw(program, size) {\n this.fbo.bind();\n gl.viewport(0, 0, ...size);\n webgl.draw(program);\n this.fbo.unbind();\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"readTex = (tex, size) => {\n const fbo = webgl.defFBO(gl, { tex: [tex] });\n const [w, h] = size;\n const n = w * h;\n const out = new Float32Array(n * 4);\n \n fbo.bind();\n gl.readPixels(0, 0, w, h, gl.RGBA, tex.type, out);\n fbo.unbind();\n fbo.release();\n \n return out;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8012,"value":"blit = {\n const program = defProgram(`\nvoid main() {\n gl_FragColor = texture2D(tex, v_uv);\n}`, { tex: [\"sampler2D\", 0]} );\n \n return tex => {\n program.textures = [tex];\n webgl.draw(program);\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":135,"value":"defProgram = (fs, uniforms = {}, shaderOpts = null, spec = null) => webgl.compileModel(gl, {\n ...webgl.defQuadModel(),\n shader: webgl.defShader(gl, {\n fs, uniforms,\n vs: `\nvoid main() {\n v_uv = uv;\n gl_Position = vec4(position, 0.0, 1.0);\n}`, \n declPrefixes: { v: \"v_\" },\n attribs: {\n position: \"vec2\",\n uv: \"vec2\",\n },\n varying: { uv: \"vec2\" },\n state: { blend: false },\n ...shaderOpts\n }),\n ...spec\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":24,"value":"md`### Utils`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7552,"value":"noop = () => {}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7115,"value":"randn = d3.randomNormal()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5704,"value":"prepareKernel421 = data => {\n const n = data.length;\n const res = new Float32Array(n);\n let ii = 0;\n for (let i = 0; i < n; i += 16) {\n for (let j of [0,1,4,5]) {\n for (let k of [0,2,8,10]) {\n res[ii++] = data[i + j + k];\n }\n }\n }\n return res;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":164,"value":"$3d","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":962,"value":"// https://leimao.github.io/article/Neural-Networks-Quantization/\ndequantizeTensor = {\n const dequantize = (xQ, alpha, beta, alphaQ = -128, betaQ = 127) => {\n const s = (beta - alpha) / (betaQ - alphaQ)\n const z = Math.round((beta * alphaQ - alpha * betaQ) / (beta - alpha));\n const res = new Float32Array(xQ.length);\n xQ.forEach((v, i) => res[i] = s * (v - z));\n return res;\n };\n \n return ({ data, shape, alpha, beta }) => {\n const vals = dequantize(new Int8Array(data), alpha, beta);\n return new Tensor(vals, shape);\n };\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1476,"value":"dilate = (arr, stride = 1, padding = 3, val = 0) => {\n if (padding == 0) return arr;\n \n const n = arr.length;\n const p = Math.floor(n / stride);\n const m = (stride + padding) * p + (n % stride);\n const res = new Float32Array(m);\n let ii = 0;\n for (let i = 0; i < n; i++) {\n res[ii++] = arr[i];\n if ((i + 1) % stride == 0) {\n for (let j = 0; j < padding; j++) res[ii++] = val;\n }\n }\n return res;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1522,"value":"undilate = (arr, stride = 1, padding = 3) => {\n if (padding == 0) return arr;\n \n const n = arr.length, step = (stride + padding);\n const m = Math.floor(n / step) * stride + (n % step);\n \n const res = new Float32Array(m);\n let ii = 0;\n for (let i = 0; i < n; i++) {\n if (i % step >= stride) i += padding - 1;\n else res[ii++] = arr[i];\n }\n return res;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8054,"value":"norm = data => Math.sqrt(d3.sum(data, d => d * d));","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6615,"value":"slerp = (out, a, b, t) => {\n const n = a.length;\n const normA = norm(a) + 1e-9, normB = norm(b) + 1e-9;\n let d = 0;\n a.forEach((x, i) => d += (x / normA) * (b[i] / normB));\n\n if (Math.abs(d) >= 1.0) {\n a.forEach((x, i) => out[i] = x);\n return out;\n }\n\n const omega = Math.acos(d);\n const so = Math.sin(omega);\n const a1 = Math.sin((1 - t) * omega) / so;\n const b1 = Math.sin(t * omega) / so;\n const res = out || new Float32Array(n);\n for (let i = 0; i < n; i++) \n res[i] = a1 * a[i] + b1 * b[i];\n return res;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":52,"value":"imshow = {\n const toABGR = (r, g = r, b = r, a = 255) => a << 24 | b << 16 | g << 8 | r;\n \n return (data, size, scale = 2) => {\n const [w, h] = size;\n const ctx = DOM.context2d(w * scale, h * scale, 1);\n const img = ctx.createImageData(ctx.canvas.width, ctx.canvas.height);\n const dst = new Uint32Array(img.data.buffer);\n const scaleV = d3.scaleLinear(d3.extent(data), [0, 255]);\n\n const row = w * scale;\n for (let i = 0; i < h; i++) {\n for (let j = 0; j < w; j++) {\n const v = toABGR(scaleV(data[i * w + j]));\n\n for (let r = 0; r < scale; r++) {\n for (let c = 0; c < scale; c++) {\n const idx = (i * scale + r) * row + (j * scale + c);\n dst[idx] = v;\n }\n }\n }\n }\n ctx.putImageData(img, 0, 0);\n return ctx.canvas;\n };\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":139,"value":"plotPalette = (colors, size = 25) => {\n const root = d3.select(DOM.svg(size * colors.length, size));\n root.selectAll(\".color\").data(colors).join('rect').classed(\"color\", true)\n .attr(\"width\", size).attr(\"height\", size)\n .attr(\"x\", (_, i) => i * size)\n .attr(\"fill\", d => d);\n return root.node();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":620,"value":"$3e","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8979,"value":"indexed = arr => d3.map(arr, (d, i) => [i, d])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6921,"value":"array = ({\n prod: arr => arr.reduce((a, b) => a * b, 1),\n eq: (a, b, eps = 1e-6) => d3.range(a.length).every(i => Math.abs(a[i] - b[i]) <= eps)\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"$3f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"transition = (f, duration, eps = 1e-6) => {\n const dt = 1000 / 60 / duration;\n let cur = 0;\n return () => {\n cur += dt;\n f(cur);\n return !math.eqDelta(cur, 1, eps);\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":85,"value":"raf = (f, done) => {\n let handle = requestAnimationFrame(function frame(t) {\n if (f(t) === false) {\n done && done();\n } else {\n handle = requestAnimationFrame(frame);\n }\n });\n return () => cancelAnimationFrame(handle);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":564,"value":"assert = (cond, msg) => { if (!cond) throw new Error(msg); }","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":89,"value":"fps = {\n let t = performance.now(), delta = 0;\n const last = [t];\n return cell => {\n last.length >= 10 && last.shift();\n t = performance.now();\n delta = (t - last[0]) / last.length;\n last.push(t);\n return Math.round(1000 / delta);\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"md`### Appendix`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":8,"value":"d3 = require(\"d3@6\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":110,"value":"webgl = require('https://bundle.run/@thi.ng/webgl@3.3.13')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6578,"value":"v = require('https://bundle.run/@thi.ng/vectors@4.9.1')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6963,"value":"math = require(\"https://bundle.run/@thi.ng/math@3.1.0\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"msgpack = require(\"@msgpack/msgpack@2.3.0\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":148,"value":"md`---\n## References\n\n* [[1802.05957] Spectral Normalization for Generative Adversarial Networks](https://arxiv.org/abs/1802.05957)\n* [[1701.07875] Wasserstein GAN](https://arxiv.org/abs/1701.07875)\n* [[1511.06434] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://arxiv.org/abs/1511.06434)\n* [Avant-Garfield - Creating New Comics With Neural Networks - YouTube](https://www.youtube.com/watch?v=wXWKWyALxYM)`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W40","stdlib":"1"}} 2f:"$2e:body"