34:["$","title",null,{"children":"Client-Side Video Encoding with h264-mp4-encoder | Paolo Andrenacci | Observable"}] 3c:T13d8,md`# Client-Side Video Encoding with h264-mp4-encoder This notebook imports [Trevor Sundberg](https://twitter.com/trevorsundberg)'s really nicely done [h264-mp4-encoder](https://www.npmjs.com/package/h264-mp4-encoder) module and exports a thin wrapper which facilitates usage in Observable. It allows you to plug in a sequence of input values (e.g. time), preview generated frames, and download a video. It accepts raster image pixel values, nothing more, nothing less. As such, it does not depend on WebGL, canvas, or any particular input. I direct all credit toward [Trevor Sundberg](https://twitter.com/trevorsundberg) for the implementation, [Matt Deslauriers](https://twitter.com/mattdesl) and [Felix Faire](https://twitter.com/FelixFaire) for tracking it down and [suggesting it](https://twitter.com/mattdesl/status/1333751152261935106), and of course many upstream authors whose libraries and tools made this not only possible but very easy! Improvements and modifications are welcome! Update: [Lionel Radisson](https://twitter.com/MAKIO135) pointed out that he's also been [using this on Observable](https://observablehq.com/@makio135/bnb) 👏. ## Usage You can import this utility into a notebook with: \`\`\`js import { encodeMP4 } from '@rreusser/h264-mp4-encoder' \`\`\` Usage requires three things. - An array or iterable containing data which controls the animation. This may be an array of time values, for example. - A function which returns a \`Uint8Array\` of size \`width * height * 4\` containing RGBA image data. Alpha is ignored. - Configuration specifying at least a \`width\` and \`height\`. Your function may return data synchronously or asynchronously, but the buffer is *used* synchronously so that you can (and perhaps should) reuse it. \`\`\`js viewof videoBytes = encodeMP4( timeArray, // Array or iterable containing data (times, perhaps) fed to computeFrame function computeFrame (time) { return rgbaImageBytes // May return bytes asynchronously }, { // Wrapper configuration: disabled: boolean; // Disables rendering when true, default: false beforeStart: function; // Optional callback executed prior to starting/restarting rendering renderImmediately: boolean; // Begin rendering when cell is evaluated, default: false dryRun: boolean; // If true, a dry run with no actual encoding, default: false flipY: boolean; // Perform a vertical flip on each frame, default: false displayOutput: boolean; // Display output on a canvas. So that the screen is not blank, // this will cause the first frame to be rendered twice (but // encoded once) // Configuration forwarded to h264-mp4-encoder: width: number; // Must be multiple of two, required height: number; // Must be multiple of two, required frameRate: number; // Frame rate of the output video, default 30 kbps: number; // The bitrate in kbps relative to the frame_rate. Overwrites // quantization_parameter if not 0, default: 0 speed: number; // Speed where 0 means best quality and 10 means fastest // speed [0..10], default: 0 quantizationParameter: number; // Higher = better compression, lower = better quality // [10..51], default: 33 groupOfPictures: number; // Key frame period, default: 20 temporalDenoise: boolean; // Use temporal noise supression, default: false desiredNaluBytes: number; // Each NAL unit will be approximately capped at this size // (0 means unlimited), default: 0 debug: boolean; // Prints extra debug information, default: false } ) \`\`\` Browser iteration through the rendering of frames is capped to 60fps so that the UI may remain responsive. If this does not match your needs, keep in mind that this wrapper is only a very thin wrapper over the underlying node module and that you may find better peformance by using the API directly. For a full description of options and usage, see the [h264-mp4-encoder API](https://www.npmjs.com/package/h264-mp4-encoder#api). ## Feature Wishlist - I know there's some ability to convert SVG to a raster image. To what extent is it possible to use this for SVG? - Clean up async cell output and update events (if this were any larger, I'd be much more thoughtful about state management in the widget, but it's small enough we can probably get aay with it being a bit messy.) - Conform the UI a bit better to the development process. ~~A "dry run" checkbox so you can test it out more quickly without modifying the code?~~ Display the resulting video in a video element? - ~~A \`flipY\` configuration parameter to account for (0, 0) in top left vs. bottom left.~~ - ~~Fix progress bar for mobile Safari~~ `3d:T1721,async function encodeMP4(frameDataArray, computeFrame, opts) { opts = opts || {}; if (typeof opts.width === "undefined" || typeof opts.height === "undefined") { throw new Error('Missing required options `width` and `height`'); } const width = opts.width; const height = opts.height; let displayOutput = !!opts.displayOutput; let renderImmediately = !!opts.renderImmediately; let performEncoding = !(typeof opts.dryRun === "undefined" ? false : !!opts.dryRun); let enable = !(typeof opts.disabled === "undefined" ? false : !!opts.disabled); let encoder, outputFilename; let ctx, imageData; let videoBytes; let canvas = ''; let resolve, reject; if (displayOutput) { canvas = DOM.canvas(width, height); canvas.style.maxWidth = '100%'; ctx = canvas.getContext('2d'); imageData = new ImageData(width, height); } const progressBar = html`

`; const renderBtn = html``; const downloadBtn = html``; const reversedText = html`

`; const dryRunBox = html``; const view = html`

${canvas}

${reversedText} ${progressBar}

${renderBtn} ${downloadBtn} ${dryRunBox}Dry run

`; if (enable) { renderBtn.addEventListener('click', computeVideoData); renderBtn.removeAttribute('disabled'); } else { renderBtn.setAttribute('disabled', true); } downloadBtn.addEventListener('click', downloadVideoData); downloadBtn.setAttribute('disabled', true); dryRunBox.addEventListener('change', function() { performEncoding = !dryRunBox.checked; }); function downloadVideoData() { var blob = new Blob([videoBytes], { type: "video/mp4" }); var link = document.createElement('a'); link.href = window.URL.createObjectURL(blob); link.download = outputFilename; link.click(); } function updateProgressText(cur, total, str) { if (typeof cur === 'string') { str = cur; } else { if (total === undefined) { progressBar.style.width = '100%'; progressBar.style.background = `repeating-linear-gradient(45deg, #606dbc, #606dbc 10px, #465298 10px, #465298 20px)`; progressBar.style.backgroundPosition = `${-cur}px 0`; } else { progressBar.style.width = `${((cur / total) * 100).toFixed(1)}%`; } } progressBar.textContent = str; reversedText.textContent = str; } function halt() { breakOut = true; } let videoData = null; let breakOut = false; async function computeVideoData() { if (encoder) throw new Error('Encoding already in progress'); if (opts.enable === false) return; breakOut = false; view.value = videoData = null; view.dispatchEvent(new CustomEvent("input")); if (opts.beforeStart) { await opts.beforeStart(); } renderBtn.textContent = 'Stop'; renderBtn.removeEventListener('click', computeVideoData); renderBtn.addEventListener('click', halt); dryRunBox.setAttribute('disabled', true); if (performEncoding) { encoder = await H264MP4Encoder.createH264MP4Encoder(); Object.assign(encoder, opts); outputFilename = encoder.outputFilename; encoder.initialize(); } downloadBtn.setAttribute('disabled', true); updateProgressText(0, 1, ''); function finalize() { if (performEncoding && encoder) { encoder.finalize(); videoBytes = encoder.FS.readFile(encoder.outputFilename); resolve(videoBytes); view.value = videoBytes; view.dispatchEvent(new CustomEvent("input")); encoder.delete(); encoder = null; downloadBtn.removeAttribute('disabled'); updateProgressText(outputFilename); } renderBtn.removeEventListener('click', halt); renderBtn.addEventListener('click', computeVideoData); renderBtn.textContent = 'Render'; dryRunBox.removeAttribute('disabled'); } let curFrame = 0; let totalFrames = frameDataArray.length; for (let datum of frameDataArray) { if (breakOut) { breakOut = false; break; } const frameData = await computeFrame(datum); curFrame++; if (opts.flipY) flipY(frameData, width, height); if (totalFrames === undefined) { updateProgressText(curFrame, totalFrames, `Frame ${curFrame}`); } else { updateProgressText( curFrame, totalFrames, `Frame ${curFrame} / ${totalFrames}` ); } if (displayOutput) { imageData.data.set(frameData); ctx.putImageData(imageData, 0, 0); } if (performEncoding) { encoder.addFrameRgba(frameData); } await new Promise(requestAnimationFrame); } finalize(); } // This is super weird, I'd like to get rid of it. view.value = new Promise(function(_resolve, _reject) { resolve = _resolve; reject = _reject; }); if (!renderImmediately && displayOutput) { let frameData = await computeFrame(frameDataArray[0]); if (opts.flipY) flipY(frameData, width, height); imageData.data.set(frameData); ctx.putImageData(imageData, 0, 0); } if (renderImmediately) { computeVideoData(); } if (!performEncoding) resolve(null); return view; }3e:Ta7b,draw = regl({ frag: ` precision highp float; uniform vec2 iResolution; uniform float iTime; const float dt = 2.0 / 60.0; const int aa = 3; float hypot (vec2 z) { float x = abs(z.x); float y = abs(z.y); float t = min(x, y); x = max(x, y); t = t / x; return x * sqrt(1.0 + t * t); } vec2 cdiv (vec2 a, vec2 b) { float e, f; float g = 1.0; float h = 1.0; if( abs(b.x) >= abs(b.y) ) { e = b.y / b.x; f = b.x + b.y * e; h = e; } else { e = b.x / b.y; f = b.x * e + b.y; g = e; } return (a * g + h * vec2(a.y, -a.x)) / f; } vec2 cmul (vec2 a, vec2 b) { return vec2( a.x * b.x - a.y * b.y, a.y * b.x + a.x * b.y ); } float checkerboard (vec2 xy) { vec2 f = fract(xy * 0.5) * 2.0 - 1.0; return f.x * f.y > 0.0 ? 1.0 : 0.0; } float ease(float t) { return t < 0.5 ? +8.0 * pow(t, 4.0) : -8.0 * pow(t - 1.0, 4.0) + 1.0; } vec2 cpow (vec2 z, float x) { float r = hypot(z); float theta = atan(z.y, z.x) * x; return vec2(cos(theta), sin(theta)) * pow(r, x); } vec2 f (vec2 z, float n) { const vec2 ONE = vec2(1, 0); vec2 a = cpow(z + ONE, n); vec2 b = cpow(z - ONE, n); return cdiv(a + b, a - b); } float color (vec2 z, float t, float n) { vec2 fz = f(z, n); fz = cdiv(fz, f(vec2(t < 0.5 ? -1.001 : 1.001, 0), n)); return checkerboard(n * 2.0 * fz - vec2(2.0 * n, 0)); } float hardstep (float edge0, float edge1, float x) { return clamp((x - edge0) / (edge1 - edge0), 0.0, 1.0); } void main() { bool vert = fract(iTime * 0.5) < 0.5; vec2 uv = ((gl_FragCoord.xy / iResolution.xy - 0.0) - 0.5) * vec2(iResolution.x / iResolution.y, 1); float sum = 0.0; for (int i = 0; i < aa; i++) { for (int j = 0; j < aa; j++) { vec2 offset = (vec2(i + 1, j + 1) / float(aa + 1) - 0.5) / iResolution.xy; // This is an awful way to compute random offset for motion blur! float tOffset = fract(23.61803398875 * float(i + j * aa) / float(aa * aa)); float tt = fract(iTime + tOffset * dt); float n = 1.0 + ease(hardstep(0.06, 0.44, tt)) - ease(hardstep(0.56, 0.94, tt)); vec2 z = (uv + offset) * 4.0; if (vert) z = vec2(-z.y, -z.x); sum += color(z, tt, n); } } sum /= float(aa * aa); gl_FragColor = vec4(vec3(pow(sum, 0.454)), 1); }` })3f:T6b0,md`## License This notebook uses [minih264](https://github.com/TrevorSundberg/minih264) and [libmp4v2](https://github.com/TrevorSundberg/libmp4v2) via the [h264-mp4-encoder](https://github.com/TrevorSundberg/h264-mp4-encoder) module. The minih264 library is licensed under [Creative Commons Zero v1.0 Universal](https://github.com/TrevorSundberg/minih264/blob/master/LICENSE) while libmp4v2 is licensed under [MPL 1.1](https://github.com/TrevorSundberg/libmp4v2/blob/master/COPYING). The license text of the h264-mp4-encoder module is reproduced below.

${md` MIT License Copyright (c) 2020 Trevor Sundberg Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. `}

`40:[] 38:{"id":"f73384fe9f1c7b04","slug":"client-side-video-encoding-with-h264-mp4-encoder","title":"Client-Side Video Encoding with h264-mp4-encoder","thumbnail":"ccd6fa86c59415b8a68e8df1a315239fc1c8dd4ce38a3f898b4d08ac283e5cae","access_level":"public","create_time":"$D2021-01-07T15:20:59.001Z","update_time":"$D2021-03-17T19:04:17.696Z","view_time":null,"first_public_time":"$D2021-03-17T19:04:50.717Z","has_unpublished_edits":null,"fork":true,"stars":1,"starred":false,"owner":{"type":"team","id":"3b42508407aca5a3","login":"zawinul","name":"Paolo Andrenacci","avatar_url":"https://avatars-next.observableusercontent.com/5cf0f05427bb66405cd76829c2ade969967efb177b4ffefd635433c153ce8f46","tier":"starter_2024","bio":"","followed_by_you":false,"follows_you":false},"version":910,"publish_version":910,"role":null,"authored":false,"authors":[{"type":"individual","id":"ed8740f4700202b7","login":"zawinul","name":"Paolo Andrenacci","avatar_url":"https://avatars-next.observableusercontent.com/5cf0f05427bb66405cd76829c2ade969967efb177b4ffefd635433c153ce8f46","tier":null,"bio":"","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"ed8740f4700202b7","login":"zawinul","name":"Paolo Andrenacci","avatar_url":"https://avatars-next.observableusercontent.com/5cf0f05427bb66405cd76829c2ade969967efb177b4ffefd635433c153ce8f46","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Client-Side Video Encoding with h264-mp4-encoder","cells":[{"id":507,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":462,"value":"md`## Example`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":226,"value":"viewof videoData = encodeMP4(\n t, // Input data\n computeFrameData, // Frame computation callback\n {\n // wrapper configuration\n displayOutput: false,\n flipY: true,\n\n // h264-mp4-encoder configuration\n frameRate: 30,\n width: shape.width,\n height: shape.height,\n outputFilename: 'video.mp4'\n }\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":392,"value":"t = Array.from(Array(120).keys()).map((d, i) => i / 60)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":286,"value":"computeFrameData = {\n // Usage of this buffer is synchronous so that we may reuse it from frame to frame\n const data = new Uint8Array(shape.width * shape.height * 4);\n\n return function computeFrameData(time) {\n regl.poll();\n fbo.use(() => {\n blit({ time }, () => {\n draw();\n regl.read(data);\n });\n });\n return data; // (Alternatively, Promise.resolve(data) or make this an async function)\n };\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":493,"value":"videoData","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":452,"value":"md`## Wrapper implementation`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":10,"value":"H264MP4Encoder = await require('https://unpkg.com/h264-mp4-encoder/embuild/dist/h264-mp4-encoder.web.js').catch(\n () => window[\"HME\"]\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"$3d","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":833,"value":"function flipY(data, width, height) {\n let tmp = new Uint8Array(width * 4);\n let rowLength = width * 4;\n for (let row = Math.floor(height / 2) - 1; row >= 0; row--) {\n var offset1 = row * width * 4;\n var offset2 = (height - 1 - row) * width * 4;\n var view1 = data.subarray(offset1, offset1 + rowLength);\n var view2 = data.subarray(offset2, offset2 + rowLength);\n tmp.set(view1);\n view1.set(view2);\n view2.set(tmp);\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":283,"value":"md`## Example implementation`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":539,"value":"md`The code below adapts [this shadertoy example](https://www.shadertoy.com/view/wd3BDH) to the [regl](https://github.com/regl-project/regl) library. You can see a standalone version of this shadertoy-like example in [this notebook](https://observablehq.com/d/c4683a2f48b0bbe6).\n\nIn this case, so that I can test debug output, my primary display canvas is just a ${tex`1 \\times 1`} canvas. I'm rendering to an offscreen framebuffer and then reading the pixel values to a typed array. You may instead wish to render directly to the screen, in which case you may need \\`preserveDrawingBuffer: true\\` in the canvas' context creation attributes.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":65,"value":"regl = createREGL({ canvas: DOM.canvas(1, 1) })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":66,"value":"shape = ({ width: 512, height: 512 })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":63,"value":"createREGL = require('regl')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":98,"value":"fbo = regl.framebuffer({ ...shape })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":77,"value":"blit = regl({\n vert: `\n precision highp float;\n attribute vec2 xy;\n void main () {\n gl_Position = vec4(xy, 0, 1);\n }`,\n attributes: {\n xy: [-4, -4, 4, -4, 0, 4]\n },\n uniforms: {\n iTime: (ctx, props) =>\n !props || props.time === undefined ? ctx.time : props.time,\n iResolution: ctx => [ctx.framebufferWidth, ctx.framebufferHeight]\n },\n depth: { enable: false },\n count: 3\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":75,"value":"$3e","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":862,"value":"$3f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W40","stdlib":"1"}} 33:null