Activation: 1.00

2a:null 2b:["$","title",null,{"children":"HINF 485 Demo: Feedforward Networks | Adam Krawitz | Observable"}] 31:T1908,32:T494,### Training When the network is initiated, the weights are assigned random values, so for most stimuli the actual response is unlikely to be very close to the desired response. So how do we adjust the weights to improve the responses? Since we are using supervised learning, we will use the difference between the actual and desired response, the error, to inform our adjustment of weights. We start with the output layer, adjusting the weights of the incoming connections to each output node so that the activation is closer to the desired value. We now move backward layer by layer repeating the process again and again until we reach the input layer. This process of the **backward propagation** of errors from the output layer to the input layer, **backprop**, is how we learn the connection weights in a bottom-up, implicit manner. Out of the complete set of data (stimulus-response pairs), part of it (half by default) is used to train the network, the **training data**, and the rest is used to test it, the **testing data**. In each **epoch**, the network is trained on all of the training data and then evaluated on both the training data and the testing data.33:T646,### Network Graph The network graph provides a visual representation of how the network is wired up. The layers proceed from left to right, with the input layer on the left, the output layer on the right, and any hidden layers in between. Within each layer, the nodes are represented as rounded rectangles. The nodes in the input and output layers are labelled as per the particular data set. And each output node also has a color associated with it, as indicated by the small colored circle right before the label. The connections are represented as dashed lines running between nodes. The strength of a connection is represented by its thickness and color. The *oranges* indicate positive weights, while the *blues* indicate negative weights. The greater the magnitude of the weight, the more *saturated* the color and the *thicker* the line. The bias is represented as an additional input to the node following the same color and thickness coding. Note that if you hover the cursor over a connection or the bias, a *tooltip* will provide the exact value. When you hover over a node, it is temporarily selected, as indicated by the darker outline. If you want to commit to the selection, just click the node, and now it will stay selected when you remove the cursor - unless, of course, you hover or click a different node. Note that when you reset a simulation the selection will return to the default, which is the first output node. Within each node is a minimap. In order to understand the minimaps and the controls to the left of each input node, we first need to understand the input-output map.34:Tc94,### Input-Output Map The last component of the visualization is the input-output map. The *x*- and *y*-axes of the map represent the level of activation for two of the inputs. By default the *x*-axis represents the first input and the *y*-axis represents the second input. For examples where the input vector has more than two dimensions, this can be altered with the radio buttons in front of the input nodes. Whichever input node is selected in the first column will be assigned to the *x*-axis, and whichever is selected in the second column will be assigned to the *y*-axis. If there are only two input nodes, this map shows the entire *input space*, but in cases where there are more than two input nodes, this map is only showing a two-dimensional slice through the overall higher-dimensional space. The particular slice is determined by the values for the other input nodes as set by the sliders that are found next to the radio buttons and the nodes themselves. By dragging these sliders, you can change which particular slice of the input space you are viewing. The color at each location in the map indicates the activation of the currently selected node, for that particular combination of inputs (as determined by the *x*- and *y*-coordinates, and the sliders for the other inputs, if there are any). High saturation *orange* indicates a strong activation (close to 1), gray indicates a medium activation (close to 0.5), and high saturation *blue* indicates a low activation (close to 0). Also plotted on the map are the data points in the training set. They are plotted as a white circle with a color in the middle which indicates which output node *should* be active for this input. If none of the output nodes should be active, then the color used is grey. Note that this color coding only makes sense if the outputs are categorical with localist coding. For example, the input is being classified/categorized as belonging to one of three categories, where the output would be (1, 0, 0) to indicate the first category, (0, 1, 0) for the second category, and (0, 0, 1) for the third category. If the outputs are not categorical with localist coding, well then, this may not be the visualization for you! All of the data points in the training set are shown, but when the input has more than two dimensions, those data points may or may not be in the particular slice currently being shown. This *distance* away from the current slice is indicated by *blurring* the point. The further the point is away from the surface of the map, the blurrier the point will be shown. For example, is the map is showing the first two input dimensions, and a third input dimension is set to 0.5, then the data point (1, 0, 0.5) would not be blurry at all, but the points (1, 0, 0.25) and (1, 0, 0.75) would be blurry, and the points (1, 0, 0) and (1, 0, 1) would be even blurrier. If you then adjust the slice by moving the slider for the third input dimension, you will be bringing some of the data points in to focus (i.e. into the plane of the map) and others will get blurrier (i.e. further away from the plan of the slice). You could think of this as a rough, multidimensional analogy to the focus on a microscope.35:T52e,### Graph/Map Interactions The network graph and the input-output map work together: - The *minimap* within each node shows the activation of that node as a function of the currently selected inputs. In other words, it is a miniature version of the input-output map described above, if that was the selected node. Whenever you change the currently selected node by hovering or clicking it, then its' minimap becomes the one that is shown in the big map! This can be handy both for size, and also because then you can see the data points overlaid on it. - If you hover or click on a data point in the map, then it will be highlighted with a black circle, and the minimaps in the graph are each replaced with a single activation value showing you how that node behaves for the selected input. We might think of this as ***activation*** mode for the graph as compared to ***map*** mode. (To return to map mode, click anywhere on the map outside of a data point.) - In activation mode, you can adjust the sliders for any of the inputs to change the input activations to the network. This will be reflected in the input-output map by the black circle for the current data point moving to reflect the new input values. In this way, you can try out the network on any arbitrary input, even if that input is not in the actual dataset.36:T4eb,## Net 3. Quadrants So far we've only been looking at problems with a very small number of very distinct stimuli, and a single output node. It's time to move on to problems where the stimuli are drawn from distributions that span more of the feature space, and there are multiple output nodes representing multiple distinct categories that those stimuli might belong to. In other words, instead of a simple logical operation, our network will be doing categorization. In this example, there are still two input features, X and Y, but now the input activations are more varied (instead of just 0 or 1) and there are four output nodes representing four distinct categories corresponding to the four quadrants of the feature space. Here are a few examples: | In: X | In: Y | Out: Upper Right | Out: Lower Right | Out: Lower Left | Out: Upper Left | | :-: | :-: | :-: | :-: | :-: | :-: | | 0.1 | 0.2 | 0 | 0 | 1 | 0 | | 0.7 | 0.9 | 1 | 0 | 0 | 0 | | 0.8 | 0.3 | 0 | 1 | 0 | 0 | | 0.2 | 0.6 | 0 | 0 | 0 | 1 | | 0.8 | 0.8 | 1 | 0 | 0 | 0 | The network is set up with a single hidden layer of 2 nodes, which is sufficient to learn this mapping. So, go ahead and run the simulation for 400 or so epochs, which should be enough to reduce the error to close to 0.01.37:T4b9,FNNSim = function(sim, controls, {percTrainData = 50, numSamples = 480, dataset = null, hiddenShape = [4], learningRate = 0.5} = {}) { if (controls.reset || !sim) { // (Re)set the simulation sim = {}; // Initialize data sim.percTrainData = percTrainData; sim.numSamples = numSamples; sim.dataset = dataset; sim.data = sim.dataset(sim.numSamples); shuffleData(sim.data); const splitIndex = Math.floor(sim.data.length * sim.percTrainData / 100); sim.trainData = sim.data.slice(0, splitIndex); sim.testData = sim.data.slice(splitIndex); // Initialize network sim.hiddenShape = hiddenShape; sim.shape = [sim.data[0].inputs.length, ...sim.hiddenShape, sim.data[0].outputs.length]; sim.network = buildNetwork(sim.shape, sim.data.inputIds, sim.data.outputIds); sim.learningRate = learningRate; sim.epoch = 0; } else { // Train for one epoch sim.epoch++; for (let index = 0; index < sim.trainData.length; index++) { const point = sim.trainData[index]; forwardProp(sim.network, point.inputs); backProp(sim.network, point.outputs); updateWeights(sim.network, sim.learningRate); } } return sim; }38:T7ab,backProp = function(network, targets) { // The output layer is a special case. const outputLayer = network[network.length - 1]; for (let nodeIndex = 0; nodeIndex < outputLayer.length; nodeIndex++) { outputLayer[nodeIndex].outputDer = outputLayer[nodeIndex].output - targets[nodeIndex]; } // Go through the layers backwards. for (let layerIndex = network.length - 1; layerIndex >= 1; layerIndex--) { let currentLayer = network[layerIndex]; // Compute the error derivative of each node with respect to: // 1) its total input // 2) each of its input weights. for (let nodeIndex = 0; nodeIndex < currentLayer.length; nodeIndex++) { let node = currentLayer[nodeIndex]; // Use sigmoid activation function let output = 1 / (1 + Math.exp(-node.totalInput)); let der = output * (1 - output); node.inputDer = node.outputDer * der; node.accInputDer += node.inputDer; node.numAccumulatedDers++; } // Error derivative with respect to each weight coming into the node. for (let nodeIndex = 0; nodeIndex < currentLayer.length; nodeIndex++) { let node = currentLayer[nodeIndex]; for (let linkIndex = 0; linkIndex < node.inputLinks.length; linkIndex++) { let link = node.inputLinks[linkIndex]; link.errorDer = node.inputDer * link.source.output; link.accErrorDer += link.errorDer; link.numAccumulatedDers++; } } if (layerIndex === 1) { continue; } let prevLayer = network[layerIndex - 1]; for (let nodeIndex = 0; nodeIndex < prevLayer.length; nodeIndex++) { let node = prevLayer[nodeIndex]; // Compute the error derivative with respect to each node's output. node.outputDer = 0; for (let linkIndex = 0; linkIndex < node.outputLinks.length; linkIndex++) { let output = node.outputLinks[linkIndex]; node.outputDer += output.weight * output.dest.inputDer; } } } } 39:T43f,buildNetwork = function(shape, inputIds, outputIds) { const numLayers = shape.length; let id = 1; const network = []; for (let layerIndex = 0; layerIndex < numLayers; layerIndex++) { const isOutputLayer = layerIndex === numLayers - 1; const isInputLayer = layerIndex === 0; const currentLayer = []; network.push(currentLayer); const numNodes = shape[layerIndex]; for (let nodeIndex = 0; nodeIndex < numNodes; nodeIndex++) { let nodeId = id.toString(); if (isInputLayer) { nodeId = inputIds[nodeIndex]; } else if (isOutputLayer) { nodeId = outputIds[nodeIndex]; } else { id++; } let node = makeNode(nodeId); currentLayer.push(node); if (layerIndex >= 1) { for (let linkIndex = 0; linkIndex < network[layerIndex - 1].length; linkIndex++) { let prevNode = network[layerIndex - 1][linkIndex]; let link = makeLink(prevNode, node); prevNode.outputLinks.push(link); node.inputLinks.push(link); } } } } return network; }3a:T6a09,// Adjust label width based on name length // Transparent overlays for links and biases (bigger targets) // Better labels // Better tooltips // Better cursors FNNViz = function(viz, sim, controls, {} = {}) { // Layer settings const numLayers = sim.shape.length; const layerSpacing = 120; // Node settings const numInputs = sim.shape[0]; const numOutputs = sim.shape[numLayers - 1]; const maxNodes = math.max(sim.shape); const nodeColor = colors.gray.darker(1); const nodeSelectedColor = colors.gray.darker(3); const nodeSize = 40; const nodeSpacing = 60; // Label settings const inputLabelAdjustment = -30; const outputLabelAdjustment = -5; const labelGap = 6; // Link settings const linkMarkerColor = colors.gray.darker(3); const linkColorScale = d3.scaleDiverging( [-5, 0, 5], d3.piecewise(d3.interpolateHcl, [colors.blue, colors.gray, colors.orange]), ).clamp(true); const linkWidthScale = d3.scaleLinear() .domain([-10, 0, 10]) .range([8, 1, 8]) .clamp(true); const linkSpacing = 5; const linkShift = 5; // Bias settings const biasLength = 10; const biasShift = 6; // Heatmap settings const heatmapPadding = 10; const heatmapAxisPadding = 40; const heatmapSize = 240; const heatmapDensity = 99; const heatmapXSampleScale = d3.scaleLinear().domain([0, 1]).range([0, heatmapDensity - 1]); const heatmapYSampleScale = d3.scaleLinear().domain([0, 1]).range([heatmapDensity - 1, 0]); const heatmapXPlotScale = d3.scaleLinear().domain([0, 1]).range([0, heatmapSize]); const heatmapYPlotScale = d3.scaleLinear().domain([0, 1]).range([heatmapSize, 0]); const heatmapThresholds = 30; const heatmapColorScale = d3.scaleDiverging( [0, 0.5, 1], d3.piecewise(d3.interpolateHcl, [colors.blue, colors.gray, colors.orange]), ); const heatmapXAxis = d3.axisBottom(heatmapXPlotScale); const heatmapYAxis = d3.axisLeft(heatmapYPlotScale); // Minimap settings const minimapDownsample = 3; const minimapDensity = heatmapDensity / minimapDownsample; // Data settings const numData = sim.trainData.length; const datumSize = 5; const datumBorder = 2; const datumSelectedBorder = 3; const datumCategoryColors = [colors.red, colors.cyan, colors.green, colors.yellow, colors.purple, colors.pink, colors.brown, colors.black]; const datumNoCategoryColor = colors.gray.darker(1); const datumColor = colors.white; const datumSelectedColor = colors.gray.darker(3); const datumOpacityScale = d3.scaleLinear() .domain([0, Math.sqrt(numInputs)]) .range([1, 0]); const datumBlurScale = d3.scaleSqrt() .domain([0, Math.sqrt(numInputs)]) .range([0, 2]); const createViz = () => { // Network const networkContainer = d3.select(viz).append('div') .classed('network', true) .style('overflow-x', 'auto') .style('display', 'inline-flex'); // Controls const controlContainer = networkContainer.append('div') .classed('controls', true) .style('display', 'inline-flex') .style('flex-flow', 'column nowrap') .style('justify-content', 'center') .style('height', `${maxNodes * nodeSpacing}px`); const controls = controlContainer.selectAll('.control').data(sim.network[0]).join('div') .classed('control', true) .property(viz.value.inputIndex, (node, nodeIndex) => { return nodeIndex; }) .style('display', 'flex') .style('flex-flow', 'row nowrap') .style('padding', `${(nodeSpacing - nodeSize) / 2}px 0`); controls.append('input') .attr('type', 'radio') .attr('name', 'x') .attr('id', (node) => { return `radio-x-${node.id}`; }) .attr('value', (node, nodeIndex) => { return nodeIndex; }) .on('input', (event) => { viz.value.inputXIndex = viz.value.inputIndex.get(event.target); updateAxes(); updateViz(); updateHeatmap(); }); controls.append('input') .attr('type', 'radio') .attr('name', 'y') .attr('id', (node) => { return `radio-y-${node.id}`; }) .attr('value', (node, nodeIndex) => { return nodeIndex; }) .on('input', (event) => { viz.value.inputYIndex = viz.value.inputIndex.get(event.target); updateAxes(); updateViz(); updateHeatmap(); }); controls.append('input') .attr('type', 'range') .attr('orient', 'vertical') .style('-webkit-appearance', 'slider-vertical') .attr('name', 'activation') .attr('id', (node) => { return `range-${node.id}`; }) .attr('min', 0) .attr('max', 1) .attr('step', 0.01) .style('width', `20px`) .style('height', `${nodeSize}px`) .style('padding', `0`) .style('margin', `0 2px`) .on('input', (event) => { viz.value.inputs[viz.value.inputIndex.get(event.target)] = Number.parseFloat(event.target.value); viz.value.selectedInputs[viz.value.inputIndex.get(event.target)] = Number.parseFloat(event.target.value); updateAxes(); updateViz(); updateHeatmap(); updateActivations(); }); // Input label length let maxInputLabelLength = 0; const root = d3.selection().select('body'); for (let inputIndex = 0; inputIndex < numInputs; inputIndex++) { const inputLabel = root.append('text') .style('font-family', 'var(--sans-serif)') .text(sim.network[0][inputIndex].id); const inputLabelLength = inputLabel.node().getBoundingClientRect().width; maxInputLabelLength = (inputLabelLength > maxInputLabelLength) ? inputLabelLength : maxInputLabelLength; inputLabel.remove(); } const inputLabelSpacing = maxInputLabelLength + inputLabelAdjustment; // Output label length let maxOutputLabelLength = 0; for (let outputIndex = 0; outputIndex < numOutputs; outputIndex++) { const outputLabel = root.append('text') .style('font-family', 'var(--sans-serif)') .text(sim.network[numLayers - 1][outputIndex].id); const outputLabelLength = outputLabel.node().getBoundingClientRect().width; maxOutputLabelLength = (outputLabelLength > maxOutputLabelLength) ? outputLabelLength : maxOutputLabelLength; outputLabel.remove(); } const outputLabelSpacing = maxOutputLabelLength + outputLabelAdjustment; // Network SVG const networkSvg = networkContainer.append('svg') .classed('networkSvg', true) .style('min-width', `${(numLayers * layerSpacing) + inputLabelSpacing + outputLabelSpacing}px`) .style('height', `${maxNodes * nodeSpacing}px`) .attr('preserveAspectRatio', 'xMinYMin slice') .attr('viewBox', ` ${-((layerSpacing / 2) + inputLabelSpacing)} ${-(maxNodes * nodeSpacing) / 2} ${(numLayers * layerSpacing) + inputLabelSpacing + outputLabelSpacing} ${maxNodes * nodeSpacing} `); // Defs const defs = networkSvg.append('defs'); // Clip path for nodes const clipPath = defs.append('clipPath') .attr('id', 'nodeClipPath'); clipPath.append('rect') .attr('x', -nodeSize/2) .attr('y', -nodeSize/2) .attr('width', nodeSize) .attr('height', nodeSize) .attr('rx', 5); // Marker for links const marker = defs.append('marker') .attr('id', 'markerArrow') .attr('markerWidth', 10) .attr('markerHeight', 12) .attr('refX', 3) .attr('refY', 6) .attr('orient', 0) .attr('markerUnits', 'userSpaceOnUse'); marker.append('path') .attr('d', 'M2,12 L8,6 L2,0') .attr('fill', linkMarkerColor); // Input labels const inputLabelContainer = networkSvg.append('g') .classed('input-labels', true) .attr('transform', `translate(${-((nodeSize / 2) + labelGap)} 0)`); const inputLabels = inputLabelContainer.selectAll('.input-label').data(sim.network[0]).join('g') .classed('input-label', true) .attr('transform', (node, nodeIndex) => { return `translate(0 ${(nodeIndex - (numInputs - 1) / 2) * nodeSpacing})`; }) inputLabels.append('text') .attr('text-anchor', 'end') .attr('dominant-baseline', 'middle') .style('font-family', 'var(--sans-serif)') .text((node) => { return node.id; }); // Layers const layerContainer = networkSvg.append('g') .classed('layers', true); const layers = layerContainer.selectAll('.layer').data(sim.network).join('g') .classed('layer', true) .property(viz.value.layerIndex, (layer, layerIndex) => { return layerIndex; }) .property(viz.value.numNodes, (layer, layerIndex) => { return sim.shape[layerIndex]; }) .attr('transform', (layer, layerIndex) => { return `translate(${layerIndex * layerSpacing} 0)`; }); // Node containers const nodeContainers = layers.selectAll('.node').data((layer) => { return layer; }).join('g') .classed('node', true) .property(viz.value.nodeIndex, (node, nodeIndex) => { return nodeIndex; }) .property(viz.value.numLinks, (node, nodeIndex) => { return node.inputLinks.length; }) .attr('transform', (node, nodeIndex, layer) => { return `translate(0 ${(nodeIndex - (layer.length - 1) / 2) * nodeSpacing})`; }); // Links const linkContainers = nodeContainers.append('g') .classed('links', true) .attr('transform', `translate(${-nodeSize / 2} 0)`); const links = linkContainers.selectAll('path').data((node) => { return node.inputLinks; }).join('path') .property(viz.value.linkIndex, (link, linkIndex) => { return linkIndex; }) .style('cursor', 'help') .attr('fill', 'none') .attr('stroke-dasharray', '9 1') .attr('marker-start', 'url(#markerArrow)') .attr('d', (link, linkIndex, elements) => { const x0 = -(layerSpacing - nodeSize); const y0 = ((linkIndex - (viz.value.numLinks.get(elements[linkIndex]) - 1) / 2) - (viz.value.nodeIndex.get(elements[linkIndex]) - (viz.value.numNodes.get(elements[linkIndex]) - 1) / 2)) * nodeSpacing; const x1 = 0; const y1 = (linkIndex - (viz.value.numLinks.get(elements[linkIndex]) - 1) / 2) * linkSpacing - linkShift; return `M ${x0},${y0} C ${x0 / 2},${y0} ${x0 / 2},${y1} ${x1},${y1}`; }); links.append('title'); // Biases const biases = nodeContainers.filter((node, nodeIndex, elements) => { return viz.value.layerIndex.get(elements[nodeIndex]) !== 0; }).append('g') .classed('bias', true) .attr('transform', `translate(${-nodeSize / 2} ${nodeSize / 2})`); const biasPaths = biases.append('path') .style('cursor', 'help') .attr('fill', 'none') .attr('stroke-dasharray', '9 1') .attr('d', (node) => { const x0 = -biasLength; const y0 = -biasShift; const x1 = 0; const y1 = -biasShift; return `M ${x0},${y0} L ${x1},${y1}`; }); biasPaths.append('title'); // Minimaps const minimaps = nodeContainers.append('g') .classed('minimap', true) .attr('clip-path', 'url(#nodeClipPath)'); const minimapForeignObjects = minimaps.append('foreignObject') .attr('x', -nodeSize / 2) .attr('y', -nodeSize / 2) .attr('width', nodeSize) .attr('height', nodeSize); minimapForeignObjects.append('xhtml:canvas') .attr('width', minimapDensity) .attr('height', minimapDensity) .style('position', 'fixed') .style('z-index', -1) .style('width', `${nodeSize}px`) .style('height', `${nodeSize}px`); // Nodes const nodes = nodeContainers.append('rect') .style('cursor', 'pointer') .attr('x', -nodeSize / 2) .attr('y', -nodeSize / 2) .attr('width', nodeSize) .attr('height', nodeSize) .attr('rx', 5) .attr('fill', colors.white) .attr('fill-opacity', 0) .attr('stroke-width', 2) .on('click', (event) => { viz.value.outputColorLayerIndex = viz.value.layerIndex.get(event.target); viz.value.outputColorNodeIndex = viz.value.nodeIndex.get(event.target); viz.value.selectedOutputColorLayerIndex = viz.value.outputColorLayerIndex; viz.value.selectedOutputColorNodeIndex = viz.value.outputColorNodeIndex; updateHeatmap(); }) .on('mouseenter', (event) => { viz.value.outputColorLayerIndex = viz.value.layerIndex.get(event.target); viz.value.outputColorNodeIndex = viz.value.nodeIndex.get(event.target); updateHeatmap(); }) .on('mouseleave', (event) => { viz.value.outputColorLayerIndex = viz.value.selectedOutputColorLayerIndex; viz.value.outputColorNodeIndex = viz.value.selectedOutputColorNodeIndex; updateHeatmap(); }); nodes.append('title'); // Output labels const outputLabelContainer = networkSvg.append('g') .classed('output-labels', true) .attr('transform', `translate(${((numLayers - 1) * layerSpacing) + (nodeSize / 2) + labelGap} 0)`); const outputLabels = outputLabelContainer.selectAll('.output-label').data(sim.network[numLayers - 1]).join('g') .classed('output-label', true) .attr('transform', (node, nodeIndex) => { return `translate(0 ${(nodeIndex - (numOutputs - 1) / 2) * nodeSpacing})`; }); const outputTexts = outputLabels.append('text') .attr('text-anchor', 'start') .attr('dominant-baseline', 'middle') .style('font-family', 'var(--sans-serif)'); outputTexts.append('tspan') .attr('fill', (node, nodeIndex) => { return datumCategoryColors[nodeIndex]; }) .text('⬤ '); outputTexts.append('tspan') .text((node) => { return node.id; }) // Heatmap const heatmapSvg = d3.select(viz).append('svg') .classed('heatmap', true) .attr('width', heatmapSize + heatmapPadding + heatmapAxisPadding) .attr('height', heatmapSize + heatmapPadding + heatmapAxisPadding); const heatmapContainer = heatmapSvg.append('g') .attr('transform', `translate(${heatmapAxisPadding}, ${heatmapPadding})`); const heatmapForeignObject = heatmapContainer.append('foreignObject') .attr('x', 0) .attr('y', 0) .attr('width', heatmapSize) .attr('height', heatmapSize); heatmapForeignObject.append('xhtml:canvas') .attr('width', heatmapDensity) .attr('height', heatmapDensity) .style('cursor', 'pointer') .style('position', 'fixed') .style('z-index', -1) .style('width', `${heatmapSize}px`) .style('height', `${heatmapSize}px`) .on('click', (event) => { viz.value.dataIndex = null; viz.value.selectedDataIndex = null; updateAxes(); updateViz(); updateHeatmap(); updateActivations(); }) // X-axis heatmapContainer.append('g') .classed('x-axis', true) .attr('transform', `translate(0, ${heatmapSize})`) .call(heatmapXAxis); heatmapContainer.append('text') .classed('x-title', true) .attr('transform', `translate(${heatmapSize / 2}, ${heatmapSize + 28})`) .attr('text-anchor', 'middle') .attr('dominant-baseline', 'middle') .style('font-family', 'var(--sans-serif)'); // Y-axis heatmapContainer.append('g') .classed('y-axis', true) .call(heatmapYAxis); heatmapContainer.append('text') .classed('y-title', true) .attr('transform', `translate(${-32}, ${heatmapSize / 2}) rotate(-90)`) .attr('text-anchor', 'middle') .attr('dominant-baseline', 'middle') .style('font-family', 'var(--sans-serif)'); // Data points const dataPointContainer = heatmapContainer.append('g') .classed('points', true); const dataPoints = dataPointContainer.selectAll('circle').data(sim.trainData).join('circle') .property(viz.value.datumIndex, (datum, datumIndex) => { return datumIndex; }) .style('cursor', 'pointer') .attr('fill', (datum) => { const outputIndex = datum.outputs.findIndex((element) => { return element === 1; }); return (outputIndex >= 0) ? datumCategoryColors[outputIndex] : datumNoCategoryColor; }) .attr('cx', heatmapXPlotScale(0)) .attr('cy', heatmapYPlotScale(0)) .attr('r', datumSize) .attr('stroke-width', datumBorder) .attr('stroke', datumColor) .on('click', (event) => { viz.value.dataIndex = viz.value.datumIndex.get(event.target); viz.value.selectedDataIndex = viz.value.dataIndex; for (let inputIndex = 0; inputIndex < viz.value.inputs.length; inputIndex++) { viz.value.inputs[inputIndex] = sim.trainData[viz.value.dataIndex].inputs[inputIndex]; viz.value.selectedInputs[inputIndex] = sim.trainData[viz.value.dataIndex].inputs[inputIndex]; } updateAxes(); updateViz(); updateHeatmap(); updateActivations(); }) .on('mouseenter', (event) => { viz.value.dataIndex = viz.value.datumIndex.get(event.target); for (let inputIndex = 0; inputIndex < viz.value.inputs.length; inputIndex++) { viz.value.inputs[inputIndex] = sim.trainData[viz.value.dataIndex].inputs[inputIndex]; } updateAxes(); updateViz(); updateHeatmap(); updateActivations(); }) .on('mouseleave', (event) => { viz.value.dataIndex = viz.value.selectedDataIndex; for (let inputIndex = 0; inputIndex < viz.value.inputs.length; inputIndex++) { viz.value.inputs[inputIndex] = viz.value.selectedInputs[inputIndex]; } updateAxes(); updateViz(); updateHeatmap(); updateActivations(); }); dataPoints.append('title') .text((datum) => { return `Inputs: [${datum.inputs.map(d3.format(".2f"))}]\nOutputs: [${datum.outputs.map(d3.format(".1f"))}]`; }); // Test Point const testPoint = heatmapContainer.append('circle') .classed('test-point', true) .attr('fill', colors.white) .attr('fill-opacity', 0) .attr('r', datumSize) .attr('stroke-width', datumSelectedBorder) .attr('stroke', datumSelectedColor) .style('pointer-events', 'none'); testPoint.append('title'); } const updateAxes = () => { // Heatmap const heatmapSvg = d3.select(viz).select('.heatmap'); // Distances const inputs = viz.value.inputs.slice(); for (let datumIndex = 0; datumIndex < numData; datumIndex++) { const datum = sim.trainData[datumIndex]; inputs[viz.value.inputXIndex] = datum.inputs[viz.value.inputXIndex]; inputs[viz.value.inputYIndex] = datum.inputs[viz.value.inputYIndex]; datum.distance = math.distance(inputs, datum.inputs); } // Axes heatmapSvg.select('.x-title') .text(sim.network[0][viz.value.inputXIndex].id); heatmapSvg.select('.y-title') .text(sim.network[0][viz.value.inputYIndex].id); // Data points heatmapSvg.select('.points').selectAll('circle') .sort((datumA, datumB) => { return (datumB.distance - datumA.distance); }) .transition().duration(250).ease(d3.easeSinInOut) .style('filter', (datum) => { return `blur(${datumBlurScale(datum.distance)}px)`; }) .attr('opacity', (datum) => {return datumOpacityScale(datum.distance); }) .transition().duration(750).ease(d3.easeSinInOut) .attr('cx', (datum) => { return heatmapXPlotScale(datum.inputs[viz.value.inputXIndex]); }) .attr('cy', (datum) => { return heatmapYPlotScale(datum.inputs[viz.value.inputYIndex]); }); // Test Point heatmapSvg.select('.test-point') .attr('opacity', viz.value.dataIndex ? 1 : 0) // .transition().duration(750).ease(d3.easeSinInOut) .attr('cx', (datum) => { return heatmapXPlotScale(viz.value.inputs[viz.value.inputXIndex]); }) .attr('cy', (datum) => { return heatmapYPlotScale(viz.value.inputs[viz.value.inputYIndex]); }); } const updateViz = () => { // Heatmap data const inputs = viz.value.inputs.slice(); // Create local input array for (let row = 0; row < heatmapDensity; row++) { inputs[viz.value.inputYIndex] = heatmapYSampleScale.invert(row); for (let column = 0; column < heatmapDensity; column++) { inputs[viz.value.inputXIndex] = heatmapXSampleScale.invert(column); forwardProp(sim.network, inputs); for (let layer = 0; layer < numLayers; layer++) { const numNodes = sim.shape[layer]; for (let node = 0; node < numNodes; node++) { viz.value.heatmaps[layer][node].heatmap[row][column] = sim.network[layer][node].output; } } } } // Controls const controls = d3.select(viz).select('.controls').selectAll('.control'); controls.select('input[name="x"]') .property('checked', (node, nodeIndex) => { return (nodeIndex === viz.value.inputXIndex); }) .property('disabled', (node, nodeIndex) => { return (nodeIndex === viz.value.inputYIndex); }); controls.select('input[name="y"]') .property('checked', (node, nodeIndex) => { return (nodeIndex === viz.value.inputYIndex); }) .property('disabled', (node, nodeIndex) => { return (nodeIndex === viz.value.inputXIndex); }); controls.select('input[name="activation"]') .property('disabled', (node, nodeIndex) => { return ((nodeIndex === viz.value.inputXIndex || nodeIndex === viz.value.inputYIndex) && viz.value.dataIndex === null); }) .property('value', (node, nodeIndex) => { return viz.value.inputs[nodeIndex]; }); // Network const networkSvg = d3.select(viz).select('.networkSvg'); // Node containers const nodeContainers = networkSvg.selectAll('.layer').selectAll('.node') // Minimap images nodeContainers.select('canvas') .each((node, nodeIndex, elements) => { const minimapContext = elements[nodeIndex].getContext('2d'); const minimapImage = minimapContext.createImageData(minimapDensity, minimapDensity); const heatmap = viz.value.heatmaps[viz.value.layerIndex.get(elements[nodeIndex])][nodeIndex].heatmap; let imageIndex = 0; for (let row = 0; row < heatmapDensity; row += minimapDownsample) { for (let column = 0; column < heatmapDensity; column += minimapDownsample) { const color = d3.rgb(heatmapColorScale(heatmap[row][column])); minimapImage.data[imageIndex++] = color.r; minimapImage.data[imageIndex++] = color.g; minimapImage.data[imageIndex++] = color.b; minimapImage.data[imageIndex++] = 255; } } minimapContext.putImageData(minimapImage, 0, 0); // Hack to get Chrome to put canvas below SVG elements minimapContext.getImageData(0, 0, 1, 1); }); // Links const links = nodeContainers.select('.links').selectAll('path') .attr('stroke-dashoffset', -sim.epoch / 2) .attr('stroke', (link) => { return linkColorScale(link.weight); }) .attr('stroke-width', (link) => { return linkWidthScale(link.weight); }); links.select('title') .text((link) => { return `Weight: ${link.weight.toFixed(2)}`; }); // Biases const biasPaths = nodeContainers.select('.bias').select('path') .attr('stroke-dashoffset', -sim.epoch / 2) .attr('stroke', (node) => { return linkColorScale(node.bias); }) .attr('stroke-width', (node) => { return linkWidthScale(node.bias); }); biasPaths.select('title') .text((node) => { return `Bias: ${node.bias.toFixed(2)}`; }); }; const updateHeatmap = () => { // Network const networkSvg = d3.select(viz).select('.networkSvg'); // Node containers const nodeContainers = networkSvg.selectAll('.layer').selectAll('.node') .attr('stroke', (node, nodeIndex, elements) => { return (viz.value.layerIndex.get(elements[nodeIndex]) === viz.value.outputColorLayerIndex) && (nodeIndex === viz.value.outputColorNodeIndex) ? nodeSelectedColor : nodeColor; }); // Heatmap const heatmapSvg = d3.select(viz).select('.heatmap'); // Heatmap image const heatmapContext = heatmapSvg.select('canvas').node().getContext('2d'); const heatmapImage = heatmapContext.createImageData(heatmapDensity, heatmapDensity); const heatmap = viz.value.heatmaps[viz.value.outputColorLayerIndex][viz.value.outputColorNodeIndex].heatmap; let imageIndex = 0; for (let row = 0; row < heatmapDensity; row++) { for (let column = 0; column < heatmapDensity; column++) { const color = d3.rgb(heatmapColorScale(heatmap[row][column])); heatmapImage.data[imageIndex++] = color.r; heatmapImage.data[imageIndex++] = color.g; heatmapImage.data[imageIndex++] = color.b; heatmapImage.data[imageIndex++] = 255; } } heatmapContext.putImageData(heatmapImage, 0, 0); // Hack to get Chrome to put canvas below SVG elements heatmapContext.getImageData(0, 0, 1, 1); } const updateActivations = () => { // Network const networkSvg = d3.select(viz).select('.networkSvg'); if (viz.value.dataIndex) { forwardProp(sim.network, viz.value.inputs); } // Nodes const nodes = networkSvg.selectAll('.layer').selectAll('.node').select('rect') .attr('fill', (node) => { return viz.value.dataIndex ? heatmapColorScale(node.output) : colors.white; }) .attr('fill-opacity', (node) => { return viz.value.dataIndex ? 1 : 0; }); nodes.select('title') .text((node) => { return viz.value.dataIndex ? `Activation: ${node.output.toFixed(2)}` : ''; }); }; // Reset if (controls.reset || !viz) { // Visualization container viz = d3.create('div') .style('display', 'flex') .style('flex-flow', 'row wrap') .node(); viz.value = { // D3 local variables layerIndex: d3.local(), numNodes: d3.local(), nodeIndex: d3.local(), numLinks: d3.local(), linkIndex: d3.local(), datumIndex: d3.local(), inputIndex: d3.local(), // Inputs mapped to X and Y coordinates in minimaps & heatmap inputXIndex: 0, inputYIndex: 1, // Values of inputs inputs: math.dotMultiply(math.ones([numInputs]), 0.5), selectedInputs: math.dotMultiply(math.ones([numInputs]), 0.5), // Currently selected data point for activations dataIndex: null, selectedDataIndex: null, // Node (hovered or selected) whose I/O map is shown in heatmap outputColorLayerIndex: numLayers - 1, outputColorNodeIndex: 0, // Currently selected node for heatmap selectedOutputColorLayerIndex: numLayers - 1, selectedOutputColorNodeIndex: 0, // 2d ragged array of heatmap data: [layer, node] heatmaps: math.map(sim.network, (layer, [layerIndex]) => { return {heatmap: math.zeros([heatmapDensity, heatmapDensity])}; }), }; createViz(); updateAxes(); } updateViz(); updateHeatmap(); updateActivations(); return viz; }3b:T52f,FNNHistViz = async function(viz, sim, controls) { // (Re)set the history let history = viz?.history; if (controls.reset || !history) { history = []; } history.push({ epoch: sim.epoch, lossTrain: getLoss(sim.network, sim.trainData), lossTest: getLoss(sim.network, sim.testData), }); // Reset if (controls.reset || !viz) { const view = await vl.markLine({point: false}) .data(vl.data(history).name('history')) .transform( vl.fold('lossTrain', 'lossTest').as('data', 'loss'), ) .encode( vl.x().fieldQ('epoch').title('Epoch').scale({nice: false}).axis({tickMinStep: 1}), vl.color().fieldN('data').sort(['lossTrain', 'lossTest']).legend({labelExpr: "{'lossTrain': 'Training data', 'lossTest': 'Testing data'}[datum.label]"}).title(' '), vl.y().fieldQ('loss').title('Error'), vl.tooltip([ {field: 'epoch', title: 'Epoch'}, {field: 'loss', title: 'Error', format: '.3f'}, ]), ) .width(width) .height(120) .autosize({type: 'fit-x', contains: 'padding', resize: true}) .render({renderer: 'svg'}); viz = html`

${view}

`; viz.value = {view}; viz.history = history; } await viz.value.view.value.data('history', history).runAsync(); return viz; }3c:T43c,generateDataQuadrants = function(numSamples) { const spacing = 0.05; const points = []; points.inputIds = ['X', 'Y']; points.outputIds = ['Upper Right', 'Lower Right', 'Lower Left', 'Upper Left']; for (let index = 0; index < numSamples;) { const input1 = math.random(0, 1); const input2 = math.random(0, 1); const outputIndex = (input1 > 0.5 && input2 > 0.5) ? 0 : (input1 > 0.5 && input2 < 0.5) ? 1 : (input1 < 0.5 && input2 < 0.5) ? 2 : (input1 < 0.5 && input2 > 0.5) ? 3 : null; const output1 = (outputIndex === 0) ? 1 : 0; const output2 = (outputIndex === 1) ? 1 : 0; const output3 = (outputIndex === 2) ? 1 : 0; const output4 = (outputIndex === 3) ? 1 : 0; if ((input1 > (0.5 - spacing) && input1 < (0.5 + spacing)) || (input2 > (0.5 - spacing) && input2 < (0.5 + spacing))) continue; points.push({ inputs: [input1, input2], outputs: [output1, output2, output3, output4], }); index++; } return points; }3d:[] 2e:{"id":"2c9214b38205117e","slug":"hinf-485-demo-feedforward-networks","title":"HINF 485 Demo: Feedforward Networks","thumbnail":"0b9e198a64ab846aab129debbf5d0d094d0ef1c311bd91834d70fb8342620f56","access_level":"public","create_time":"$D2023-05-05T22:54:20.011Z","update_time":"$D2023-05-09T01:15:42.113Z","view_time":null,"first_public_time":"$D2023-05-10T18:31:46.645Z","fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"3861495db0d6917b","login":"akrawitz","name":"Adam Krawitz","avatar_url":"https://avatars-next.observableusercontent.com/232e39a655b3dffeaad79a6e590471290f026876224f3f7c9825f12c3f0147bc","tier":"starter_2024"},"version":4333,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"f056c267619fba67","login":"akrawitz","name":"Adam Krawitz","avatar_url":"https://avatars-next.observableusercontent.com/8b0aba291db32e92917b2b37e94c3c5568c23f50e1e769e8f76508b16293220c","tier":null}],"members":[{"type":"individual","id":"f056c267619fba67","login":"akrawitz","name":"Adam Krawitz","avatar_url":"https://avatars-next.observableusercontent.com/8b0aba291db32e92917b2b37e94c3c5568c23f50e1e769e8f76508b16293220c","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":{"id":"8ee1672695697eae","slug":"uvic-hinf-485","title":"UVic HINF 485 Demonstrations","owner":{"id":"3861495db0d6917b","login":"akrawitz","name":"Adam Krawitz"}},"type":"notebook","body":{"title":"HINF 485 Demo: Feedforward Networks","cells":[{"id":0,"value":"# HINF 485 Demo: Feedforward Networks","pinned":false,"hidden":false,"mode":"md"},{"id":2994,"value":"assignmentToc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":2591,"value":"## Introduction\nIn this demonstration we will begin our exploration of artificial neural networks by focusing on supervised feedforward networks. These are classic connectionist models in the spirit of Rumelhart and McClelland's seminal volumes on [*Parallel Distributed Processing: Explorations in the Microstructure of Cognition*](https://mitpress.mit.edu/books/parallel-distributed-processing-volume-1).","pinned":false,"hidden":false,"mode":"md"},{"id":3003,"value":"### Architecture\nThe networks in this assignment consist of layers of one or more artificial neurons, i.e. nodes. The nodes in each layer are connected unidirectionally to all of the nodes in the next layer. The first layer is the **input** layer, the last layer is the **output** layer, and any intervening layers are **hidden** layers.\n\nNodes in the input layer are unique in that they do not have any incoming connections from other nodes. And nodes in the output layer are unique in that they do not have any outgoing connections to other nodes. All other nodes, i.e. those in hidden layers, have incoming connections from all of the nodes in the previous layer and outgoing connections to all of the nodes in the next layer.","pinned":false,"hidden":false,"mode":"md"},{"id":3242,"value":"$31","pinned":false,"hidden":false,"mode":"html"},{"id":3314,"value":"Each node has an **activation level**, *a_n*, between 0 and 1, which might be thought of as it's firing rate. Each connection between nodes has a **connection weight**, *w_m,n*, between −∞ and ∞, which might be thought of as the strength of the synaptic connection between the two neurons. In addition, each node has a baseline level called the **bias**, *b_n*.","pinned":false,"hidden":false,"mode":"md"},{"id":3233,"value":"### Operation\nThe activation level of the nodes in the input layer is determined by the current **stimulus** presented to the network. The activation level of every other node is calculated based on its inputs from the previous layer. Thus, to calculate the activations, one must start at the input layer and work forward through the network one layer at a time.\n\nThe amount of input that a node, *n*, receives from a node, *m*, in the previous layer is determined by the activation of that previous node multiplied by the weight of the connection between the two nodes, *a_m* ⋅ *w_m,n*. The total input to a node is the bias plus the sum of all the inputs from the previous layer. The activation level is then determined by passing the total input through the **activation function**, *f()*, which for us is a [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function). Putting it together:","pinned":false,"hidden":false,"mode":"md"},{"id":3194,"value":"a_n = f(b_n + \\sum_m a_m \\cdot w_{m,n})","pinned":false,"hidden":false,"mode":"tex"},{"id":3183,"value":"This activation then gets passed along all of the outgoing connections from this node to the next layer of nodes. And on it goes until our **forward propagation** reaches the output layer. The activation levels of the nodes in the output layer can be thought of as the network's **response**. ","pinned":false,"hidden":false,"mode":"md"},{"id":3323,"value":"### Evaluation\nEach stimulus is defined as a pattern of activation for the nodes in the input layer. In other words, if there are *x* input nodes, then the stimulus can be expressed as an *x*-dimensional vector. For example, if there are two input nodes, then the stimulus could be (1, 0), or (0.5, 0.25), or (1, 1). Each response is defined as a pattern of activation for the nodes in the output layer. If there are *y* output nodes, then the response can be expressed as a *y*-dimensional vector. With four output nodes, the response could be (0, 0, 1, 0), or (0.73, 0.29, 0.55, 0.88), etc…\n\nWhen a stimulus is presented to the network, the resulting response is calculated by propagating activation through the network as described above. This *actual* response can then be compared to the *desired* response.","pinned":false,"hidden":false,"mode":"md"},{"id":3350,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":3364,"value":"## Visualizing Networks\nThe visualization has three components (plus the controls for running the simulation, described below).","pinned":false,"hidden":false,"mode":"md"},{"id":4071,"value":"### Training History\nThe plot of training history shows how well the network's responses match the correct responses - the fit. At the end of each training epoch, the mean squared error (MSE) is calculated for the training data and the test data. The error is calculated as follows:","pinned":false,"hidden":false,"mode":"md"},{"id":3393,"value":"MSE = \\frac{\\sum_{o=1}^{O} (c_o - a_o)^2}{O}","pinned":false,"hidden":false,"mode":"tex"},{"id":3409,"value":"Where *c_n* is the correct activation for node *n*, *a_n* is actual activation of node *n*, capital *O* is the total number of nodes in the output layer, and lowercase *o* indexes those output nodes.\n\nThis error is calculated and plotted for both the training data and the testing data. Typically these two values will be similar, but if the network has been *overtrained*, then they can diverge. This is like overfitting in statistics, where you are accounting for patterns in the sample data that are not representative of the overall population data.\n\nIn general, the error should tend toward zero, as the network learns the correct weights. Indeed, the goal of training is to minimize this error. How close it can/should get to zero depends on both the network and the data.","pinned":false,"hidden":false,"mode":"md"},{"id":3424,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":3437,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":3454,"value":"$35","pinned":false,"hidden":false,"mode":"md"},{"id":3369,"value":"## Running Networks\nWe can use our standard controls to run the network simulation:\n- To reset to the initial state, click **Reset**. This will reset all of the connection weights to random values, and resample the data from the dataset.\n- To advance the simulation one epoch, click **Step**. In an epoch, every datum (stimulus) in the training set is presented to the network once, the response is calculated using forward propagation, and the weights are adjusted using backprop. Then the network is evaluated on the training set and the testing set.\n- To run lots of epochs in succession, click **Start**, and then **Stop** when you want to stop it.\n- To run the simulation faster or slower, drag the **Speed** slider (leftward = slower, rightward = faster). The actual speed depends on both the speed setting and the size of the network. Larger networks generally run slower due to the complexity of the visualization. ","pinned":false,"hidden":false,"mode":"md"},{"id":3487,"value":"## Net 1. AND\nAlright, enough with the chitchat. Let's get our hands dirty and run a model!\n\nOur first model is simple. There are two nodes in the input layer, one node in the output layer, and no hidden layers. There are exactly four distinct data points:\n\n| Input A | Input B | Output |\n| :-: | :-: | :-: |\n| 0 | 0 | 0 |\n| 0 | 1 | 0 |\n| 1 | 0 | 0 |\n| 1 | 1 | 1 |\n\nIn other words, we are training our network to perform the logical **AND** operation! The output neuron is active if and only if both of the input neurons are active.\n\nRun the model for 20 epochs. This should be sufficient to train the model to have a very small error — on the order of 0.005 MSE.","pinned":false,"hidden":false,"mode":"md"},{"id":2671,"value":"viewof conAnd = controls()","pinned":false,"hidden":false,"mode":"ojs"},{"id":2674,"value":"FNNHistViz(this, simAnd, conAnd)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2678,"value":"FNNViz(this, simAnd, conAnd)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2668,"value":"simAnd = FNNSim(this, conAnd, {dataset: generateDataAnd, hiddenShape: []})","pinned":true,"hidden":false,"mode":"ojs"},{"id":3574,"value":"## Net 2. XOR\nOur second network looks remarkably familiar. The architecture is the same as before. And here are the data points:\n\n| Input A | Input B | Output |\n| :-: | :-: | :-: |\n| 0 | 0 | 0 |\n| 0 | 1 | 1 |\n| 1 | 0 | 1 |\n| 1 | 1 | 0 |\n\nThis is the ***XOR*** operation — exclusive OR. The output neuron is active if either, but not both, of the inputs is active.\n\nRun this model for 20 epochs and take a look…","pinned":false,"hidden":false,"mode":"md"},{"id":2613,"value":"viewof conXor = controls()","pinned":false,"hidden":false,"mode":"ojs"},{"id":2622,"value":"FNNHistViz(this, simXor, conXor)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2632,"value":"FNNViz(this, simXor, conXor)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2616,"value":"simXor = FNNSim(this, conXor, {dataset: generateDataXor, hiddenShape: [2]})","pinned":true,"hidden":false,"mode":"ojs"},{"id":3635,"value":"$36","pinned":false,"hidden":false,"mode":"md"},{"id":2716,"value":"viewof conQuadrants = controls()","pinned":false,"hidden":false,"mode":"ojs"},{"id":2719,"value":"FNNHistViz(this, simQuadrants, conQuadrants)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2721,"value":"FNNViz(this, simQuadrants, conQuadrants)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2725,"value":"simQuadrants = FNNSim(this, conQuadrants, {numSamples: 80, dataset: generateDataQuadrants, hiddenShape: [2]})","pinned":true,"hidden":false,"mode":"ojs"},{"id":4209,"value":"## Postscript\nIn this demonstration you have explored basic feedforward neural networks trained with backpropagation. The neural network approach is:\n\n* numerical (used numbers as activations and weights)\n* bottom-up (the weights were adjusted based on learning, not set by you)\n* implicit (the rules are buried in the resulting configuration of connection weights)\n\nAn interesting set of questions centers on how higher-level symbolic representations can arise from lower-level connectionist representations. If this sounds interesting, starting points might include [Smolensky (1988)](https://psycnet.apa.org/doi/10.1017/S0140525X00052432) on the more theoretical side or [Rougier et al. (2005)](https://doi.org/10.1073/pnas.0502455102) on the more applied side.","pinned":false,"hidden":false,"mode":"md"},{"id":604,"value":"## Definitions","pinned":false,"hidden":false,"mode":"md"},{"id":1170,"value":"### Simulation","pinned":false,"hidden":false,"mode":"md"},{"id":613,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":684,"value":"forwardProp = function(network, inputs) {\n const inputLayer = network[0];\n // Update the input layer\n for (let nodeIndex = 0; nodeIndex < inputLayer.length; nodeIndex++) {\n inputLayer[nodeIndex].output = inputs[nodeIndex];\n }\n // Update the other layers\n for (let layerIndex = 1; layerIndex < network.length; layerIndex++) {\n let currentLayer = network[layerIndex];\n // Update all the nodes in this layer\n for (let nodeIndex = 0; nodeIndex < currentLayer.length; nodeIndex++) {\n const node = currentLayer[nodeIndex];\n node.totalInput = node.bias;\n // Gather all inputs\n for (let linkIndex = 0; linkIndex < node.inputLinks.length; linkIndex++) {\n let link = node.inputLinks[linkIndex];\n node.totalInput += link.weight * link.source.output;\n }\n // Sigmoid activation function!\n node.output = 1 / (1 + Math.exp(-node.totalInput));\n }\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":703,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":721,"value":"updateWeights = function(network, learningRate) {\n for (let layerIndex = 1; layerIndex < network.length; layerIndex++) {\n let currentLayer = network[layerIndex];\n for (let nodeIndex = 0; nodeIndex < currentLayer.length; nodeIndex++) {\n let node = currentLayer[nodeIndex];\n // Update the node's bias.\n if (node.numAccumulatedDers > 0) {\n node.bias -= learningRate * node.accInputDer / node.numAccumulatedDers;\n node.accInputDer = 0;\n node.numAccumulatedDers = 0;\n }\n // Update the weights coming into this node.\n for (let linkIndex = 0; linkIndex < node.inputLinks.length; linkIndex++) {\n let link = node.inputLinks[linkIndex];\n if (link.numAccumulatedDers > 0) {\n // Update the weight based on dE/dw.\n link.weight = link.weight - (learningRate / link.numAccumulatedDers) * link.accErrorDer;\n link.accErrorDer = 0;\n link.numAccumulatedDers = 0;\n }\n }\n }\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":678,"value":"$39","pinned":false,"hidden":false,"mode":"ojs"},{"id":673,"value":"makeNode = function(id) {\n return {\n id,\n inputLinks: [],\n bias: 1,\n outputLinks: [],\n totalInput: 0,\n output: 0,\n outputDer: 0,\n inputDer: 0,\n accInputDer: 0,\n numAccumulatedDers: 0,\n };\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":676,"value":"makeLink = function(source, dest) {\n return {\n id: source.id + \"-\" + dest.id,\n source,\n dest,\n weight: Math.random() * 5 - 2.5,\n errorDer: 0,\n accErrorDer: 0,\n numAccumulatedDers: 0,\n };\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1175,"value":"### Visualization","pinned":false,"hidden":false,"mode":"md"},{"id":2974,"value":"Visualization of feedforward neural network.","pinned":false,"hidden":false,"mode":"md"},{"id":853,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs"},{"id":2978,"value":"Visualization of training history.","pinned":false,"hidden":false,"mode":"md"},{"id":788,"value":"$3b","pinned":false,"hidden":false,"mode":"ojs"},{"id":2929,"value":"CSS styles for the history visualization.","pinned":false,"hidden":false,"mode":"md"},{"id":804,"value":"","pinned":false,"hidden":false,"mode":"html"},{"id":2982,"value":"Calculated the difference between actual and correct outputs.","pinned":false,"hidden":false,"mode":"md"},{"id":770,"value":"function getLoss(network, data) {\n let loss = 0;\n for (let dataIndex = 0; dataIndex < data.length; dataIndex++) {\n let point = data[dataIndex];\n forwardProp(network, point.inputs);\n // Uses squared error\n const outputLayer = network[network.length - 1];\n for(let outputIndex = 0; outputIndex < point.outputs.length; outputIndex++) {\n loss += Math.pow(outputLayer[outputIndex].output - point.outputs[outputIndex], 2);\n }\n }\n return loss / data.length;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1178,"value":"### Data","pinned":false,"hidden":false,"mode":"md"},{"id":742,"value":"shuffleData = function(array) {\n let counter = array.length;\n let temp = 0;\n let index = 0;\n // For each element in the array\n while (counter > 0) {\n // Pick a random index\n index = Math.floor(Math.random() * counter);\n // Decrease counter by 1\n counter--;\n // And swap the last element with it\n temp = array[counter];\n array[counter] = array[index];\n array[index] = temp;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":2643,"value":"generateDataAnd = function(numSamples) {\n const items = [];\n \n items.push({\n inputs: [0, 0], \n outputs: [0],\n });\n items.push({\n inputs: [0, 1], \n outputs: [0],\n });\n items.push({\n inputs: [1, 0], \n outputs: [0],\n });\n items.push({\n inputs: [1, 1], \n outputs: [1],\n });\n\n const points = rep(items, {each: numSamples / 4})\n \n points.inputIds = ['Input A', 'Input B'];\n points.outputIds = ['Output'];\n \n return points;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":2708,"value":"generateDataXor = function(numSamples) {\n const items = [];\n \n items.push({\n inputs: [0, 0], \n outputs: [0],\n });\n items.push({\n inputs: [0, 1], \n outputs: [1],\n });\n items.push({\n inputs: [1, 0], \n outputs: [1],\n });\n items.push({\n inputs: [1, 1], \n outputs: [0],\n });\n\n const points = rep(items, {each: numSamples / 4})\n \n points.inputIds = ['Input A', 'Input B'];\n points.outputIds = ['Output'];\n \n return points;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":738,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs"},{"id":18,"value":"## Resources","pinned":false,"hidden":false,"mode":"md"},{"id":2594,"value":"Load assignment styles.","pinned":false,"hidden":false,"mode":"html"},{"id":2598,"value":"import {assignmentStyles} from '@akrawitz/assignment-styles'","pinned":false,"hidden":false,"mode":"ojs"},{"id":2601,"value":"assignmentStyles","pinned":false,"hidden":false,"mode":"ojs"},{"id":2990,"value":"Load assignment table of contents.","pinned":false,"hidden":false,"mode":"md"},{"id":2987,"value":"import {assignmentToc} from '@akrawitz/assignment-toc'","pinned":false,"hidden":false,"mode":"ojs"},{"id":577,"value":"Load simulation controls.","pinned":false,"hidden":false,"mode":"md"},{"id":580,"value":"import {controls} from '@akrawitz/simulation-controls'","pinned":false,"hidden":false,"mode":"ojs"},{"id":583,"value":"Load pleasant colors.","pinned":false,"hidden":false,"mode":"md"},{"id":586,"value":"import {colors} from '@akrawitz/pleasant-colors'","pinned":false,"hidden":false,"mode":"ojs"},{"id":2649,"value":"Load repeater.","pinned":false,"hidden":false,"mode":"md"},{"id":2651,"value":"import {rep} from '@akrawitz/repeater'","pinned":false,"hidden":false,"mode":"ojs"},{"id":2488,"value":"Load [ObservableHQ range input](https://observablehq.com/@observablehq/input-range).","pinned":false,"hidden":false,"mode":"md"},{"id":1257,"value":"import {Range} from \"@observablehq/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":1080,"value":"Load [math.js](https://mathjs.org/), \"an extensive math library for JavaScript and Node.js\".","pinned":false,"hidden":false,"mode":"md"},{"id":1083,"value":"math = require(\"mathjs\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":789,"value":"Load the [Vega-Lite JavaScript API](https://github.com/vega/vega-lite-api/) which \"provides a convenient way to write [Vega-Lite](https://vega.github.io/vega-lite) specifications in a programmatic fashion\".","pinned":false,"hidden":false,"mode":"md"},{"id":793,"value":"import {vl} from '@vega/vega-lite-api'","pinned":false,"hidden":false,"mode":"ojs"},{"id":589,"value":"Load [d3](https://d3js.org/), \"a JavaScript library for manipulating documents based on data\".","pinned":false,"hidden":false,"mode":"md"},{"id":895,"value":"d3 = require('d3', 'd3-scale', 'd3-interpolate')","pinned":false,"hidden":false,"mode":"ojs"},{"id":600,"value":"License for [Tensorflow Playground](https://github.com/tensorflow/playground), whose code provided the initial basis for this notebook.","pinned":false,"hidden":false,"mode":"md"},{"id":480,"value":"tfp = `\n/* Copyright 2016 Google Inc. All Rights Reserved.\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n http://www.apache.org/licenses/LICENSE-2.0\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"## References\n* [TensorFlow Playground](https://playground.tensorflow.org/)\n * [GitHub repository](https://github.com/tensorflow/playground)","pinned":false,"hidden":false,"mode":"md"}],"files":"$W3d","stdlib":"1","titling":"auto"}} 2f:"$2e:body"