2b:null 2c:["$","title",null,{"children":"Zero-copy Apache Arrow with WebAssembly | Kyle Barron | Observable"}] 31:T450,## Copying data is a bottleneck for WebAssembly I'm a huge believer in the potential of WebAssembly. Access to native dependencies and reliable performance combine for an exciting future of data-intensive browser applications. WebAssembly's security model is designed to effectively sandbox code and prevent it from reading arbitrary JavaScript memory. Thus it provides WebAssembly programs an entirely separate memory structure apart from JavaScript memory. This means that most applications _copy_ data back and forth across the JS-WebAssembly boundary when calling Wasm functions. Aside from a speed penalty, copying data on every call is a huge _memory_ penalty. In a browser environment where code could be running on a mobile device with low physical resources, minimizing memory use is essential. Therefore by removing copies, we can reliably bring larger datasets to the browser. This post will explain how [Apache Arrow](https://arrow.apache.org/) enables zero-copy data exchange, why it's useful in the browser, and give a live example of reading Arrow memory with no copy from WebAssembly.32:T475,## Arrow C Data Interface The Apache Arrow specification [defines](https://arrow.apache.org/docs/format/Columnar.html) a common, language-agnostic in-memory representation for columnar data. Since _every_ implementation of Apache Arrow uses the same memory layout, with only a bit of metadata about where in memory to look and what data type an array has, _any Arrow implementation can understand existing memory it hasn't created_. The [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) is designed to fill this gap by defining minimal metadata to describe and locate arrays in memory. The C Data Interface has already enabled exciting cross-language functionality. For example, the Python bindings of [Polars](https://github.com/pola-rs/polars), a Rust data frame library built on Arrow, sometimes use `pyarrow` (which itself uses the Arrow C++ bindings internally) for loading data. Arrow and its C Data Interface enables efficiently moving this memory from C++ to Rust via a Python API. But up to this point I'm unaware of anyone using the C Data Interface in the browser ^{[[1](#cell-450)]}. 33:T67f,### Faster Data Loading I'm excited about bringing bigger datasets to the browser, and I've written before about how I believe [Apache Parquet](https://parquet.apache.org/) is the [ideal serialization format](https://observablehq.com/@kylebarron/geoparquet-on-the-web) for tabular data. To that end, I maintain [`parquet-wasm`](https://github.com/kylebarron/parquet-wasm) — WebAssembly bindings of the Rust Arrow and Parquet libraries for efficiently and reliably decoding the complex Parquet format. `parquet-wasm` first parses a Parquet file into an Apache Arrow table within WebAssembly memory, but then it still needs to expose that table to JS. In the existing version of `parquet-wasm` (v0.3.0) we use the [Arrow IPC format](https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc), to pass data back to JS. This works, but requires a few copies: - One copy on the Rust side from the Parquet buffer to the Arrow table. - Another copy from the Rust Arrow table into the Rust IPC buffer. - Then another copy of the Rust IPC buffer to JS memory. By implementing support for the Arrow C Data Interface, we can remove at least one and up to _two_ copies. We no longer need a copy from the Rust Arrow table into the Rust IPC buffer, and, if JS only needs read-only access to the table, we no longer need to copy it to JS memory at all! An application like [`deck.gl`](https://deck.gl/) that [supports Arrow](https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl) can view the Arrow table in Wasm memory directly, and in the `deck.gl` case can copy data directly from Wasm memory to WebGL memory!34:Tb0b,### Exposing an Arrow Table from Rust The `arrow2` Rust crate contains an [`ffi` module](https://docs.rs/arrow2/latest/arrow2/ffi/index.html) with the functions [`export_array_to_c`](https://docs.rs/arrow2/latest/arrow2/ffi/fn.export_array_to_c.html) and [`export_field_to_c`](https://docs.rs/arrow2/latest/arrow2/ffi/fn.export_field_to_c.html), which return the defined `ArrowArray` or `ArrowSchema` structs, respectively. It's important to note that these functions _do not copy data_, and the `ArrowArray` and `ArrowSchema` structs _do not contain data_. Rather, they only contain _pointers_ to where the data reside in memory. The `ArrowArray` and `ArrowSchema` structs are well-defined by the C Data Interface. Therefore, all Rust needs to communicate to JS are the _pointers to these structs_! Additionally we have to ensure that these structs and the underlying data are not freed until JS is done with them. While Wasm doesn't have a garbage collector, Rust's allocator will free the data if those structs go out of scope. This means JS needs to have a handle on the struct to control persisting the data. We can accomplish this by wrapping the `ArrowArray` and `ArrowSchema` structs in [`wasm-bindgen`](https://github.com/rustwasm/wasm-bindgen) structs: ```rs /// Wrapper around an ArrowArray FFI struct in Wasm memory. #[wasm_bindgen] pub struct FFIArrowArray(Box); /// Function to construct this `FFIArrowArray` struct from an /// `arrow2` Array by calling `ffi::export_array_to_c` on it impl From> for FFIArrowArray { fn from(array: Box) -> Self { Self(Box::new(ffi::export_array_to_c(array))) } } /// Implement a function for JS to see the pointer to the contained `ArrowArray` #[wasm_bindgen] impl FFIArrowArray { #[wasm_bindgen] pub fn addr(&self) -> *const ffi::ArrowArray { self.0.as_ref() as *const _ } } ``` The Arrow C Data interface provides definitions only for _individual_ array chunks and schema fields. Arrow libraries often operate on higher abstractions: a `Chunk` or `RecordBatch` as a collection of arrays, a `Schema` as a collection of individual fields, and a `Table` as a collection of `RecordBatches` with an associated `Schema`. At this point I believe it's required for individual programs to provide this contract on either side of its FFI boundary. The Rust side also needs to provide a way for JS to manually free the underlying memory when it's done. I _believe_ this is entirely handled using the `free` method that is automatically added by `wasm-bindgen` on its structs but need to double check this in the future. You can view the entire Rust FFI implementation in `parquet-wasm` [here](https://github.com/kylebarron/parquet-wasm/blob/4b38990cdfe54cde1f6ca77501bca44c1de6832b/src/arrow2/ffi.rs).35:T9d4,### Viewing data from JavaScript Since Wasm memory is backed by a normal `ArrayBuffer`, parsing Wasm memory behaves identically to parsing an `ArrayBuffer` in JS memory. We can use all the standard low-level JS tools to work on that buffer, including reading individual values with a [`DataView`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView) and creating views on ranges of memory using [typed arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray). Next we have to implement the C ABI layout for the `ArrowSchema` and `ArrowArray` definitions in the [C Data Interface spec](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions). The C ABI for WebAssembly is [described here](https://github.com/WebAssembly/tool-conventions/blob/main/BasicCABI.md) — in general, types are the same as outside the browser, noting that WebAssembly is currently limited to 32 bits, and therefore pointer types are `uint32`. Using the knowledge of Wasm's C ABI, reading the structs is straightforward, though tedious. For each `int64_t` type, we call [`DataView.getBigInt64`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView/getBigInt64), and for each pointer type we call [`DataView.getUint32`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView/getUint32). To make views onto arrays, we can instantiate typed arrays onto the relevant regions of Wasm memory, as defined by the pointers in the `ArrowArray` struct. For a JS application that's content with read-only access to the Arrow table, we can take views of the underlying arrow data directly. If the JS side wants to own the data, we can copy the underlying buffers directly. This should still be more efficient than using Arrow IPC for serialization, as it removes the need for a copy on the Wasm side. The C Data Interface structs also define a [release callback](https://arrow.apache.org/docs/format/CDataInterface.html#memory-management) to mark that underlying memory is no longer needed. I haven't yet tried to implement this as I need to dig in further to how function pointers work in Wasm. For now, releasing memory is done manually with a `wasm-bindgen` function call. I've [started a repository](https://github.com/kylebarron/arrow-js-ffi) to fully implement the above JS processing, though right now it's still alpha (doesn't support nested types, needs tests).36:T687,parseField = { /** Minimal parsing of an arrow Field from an ArrowSchema FFI struct */ function parseField(buffer, ptr) { const dataView = new DataView(buffer); // Parse format to an Arrow JS DataType const format = parseFormat(dataView, ptr); // Parse name into a JS string const namePtr = dataView.getUint32(ptr + 4, true); const name = parseNullTerminatedString(dataView, namePtr); // Extra 4 to be 8-byte aligned const flags = dataView.getBigInt64(ptr + 16, true); const nullable = Boolean(flags & 0x00000001n); return arrow.Field.new(name, format, nullable); } /** Parse a pointer to the format struct attribute into an Arrow JS DataType */ function parseFormat(dataView, ptr) { const formatPtr = dataView.getUint32(ptr, true); const format = parseNullTerminatedString(dataView, formatPtr); // Strings from https://arrow.apache.org/docs/format/CDataInterface.html#data-type-description-format-strings // In this minimal implementation, only include these two types const formatMapping = { S: new arrow.Uint16(), f: new arrow.Float32() }; return formatMapping[format]; } /** Parse a C-style null-terminated string to a JS string */ function parseNullTerminatedString(dataView, ptr) { // First find the end of the null-terminated string let end = ptr; while (end < dataView.byteLength && dataView.getUint8(end) !== 0) { end += 1; } // Take a Uint8Array view of that buffer range and decode to a UTF8 string return new TextDecoder("utf-8").decode( new Uint8Array(dataView.buffer, ptr, end - ptr) ); } return parseField; }37:T752,parseArray = { /** Parse Array from ArrowArray FFI */ return function parseArray(buffer, ptr, dataType) { const dataView = new DataView(buffer); // These functions assume bigint support // Older browsers could work with each half and cast to a number const length = dataView.getBigInt64(ptr, true); const nullCount = dataView.getBigInt64(ptr + 8, true); const offset = dataView.getBigInt64(ptr + 16, true); const nBuffers = dataView.getBigInt64(ptr + 24, true); const nChildren = dataView.getBigInt64(ptr + 32, true); // We have a pointer to an array of pointers, where each of those points to an actual array of data const ptrToBuffers = dataView.getUint32(ptr + 40, true); const bufferPtrs = []; for (let i = 0; i < nBuffers; i++) { bufferPtrs.push(dataView.getUint32(ptrToBuffers + i * 4, true)); } // For simplicity in this example, we only support primitive types // For primitive types, the first pointer is the validity and the second is the data: // https://arrow.apache.org/docs/format/Columnar.html#buffer-listing-for-each-layout const validityPtr = bufferPtrs[0]; // In an actual implementation you'd want to parse this validity bitmap 😄 // https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps const nullBitmap = validityPtr === 0 ? null : null; // Then take a view of the data // Here ArrayType is Uint16Array and Float32Array, respectively const dataPtr = bufferPtrs[1]; const data = new dataType.ArrayType( dataView.buffer, dataPtr, Number(length) ); // Then create Arrow JS Data objects from these low-level arrays return arrow.makeData({ type: dataType, offset: Number(offset), length: Number(length), nullCount: Number(nullCount), nullBitmap, data }); }; }38:[] 2f:{"id":"ceb57dd2a6136bfe","slug":"zero-copy-apache-arrow-with-webassembly","title":"Zero-copy Apache Arrow with WebAssembly","thumbnail":"5ee9fea919b7168aa6b2635d29528625e038e98ecdc158ff609aeea06f96ee47","access_level":"public","create_time":"$D2022-08-18T02:28:27.317Z","update_time":"$D2024-04-17T16:58:09.130Z","view_time":null,"first_public_time":"$D2022-08-22T09:48:26.331Z","fork":false,"stars":41,"starred":false,"owner":{"type":"team","id":"43d330d6f3e9ea97","login":"kylebarron","name":"Kyle Barron","avatar_url":"https://avatars-next.observableusercontent.com/75d37333b9e7068d0c7abf571919ecfa40a57510d4b2cf55d65c9b3bec1f2a9f","tier":"starter_2024"},"version":478,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"d003201354321004","login":"kylebarron","name":"Kyle Barron","avatar_url":"https://avatars-next.observableusercontent.com/0096ac4ec8894f01af5a4d303fe3463061511b44dda0e4803aafc9ede3f46507","tier":null}],"members":[{"type":"individual","id":"d003201354321004","login":"kylebarron","name":"Kyle Barron","avatar_url":"https://avatars-next.observableusercontent.com/0096ac4ec8894f01af5a4d303fe3463061511b44dda0e4803aafc9ede3f46507","tier":null,"role":"editor"}],"comments":[{"id":"60684d2b56cd2c8c","reply_to_id":null,"content":"why don't we use reference type specification of wasm to access arrow table in JS?","create_time":"$D2022-08-23T21:10:53.224Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}},{"id":"737a303975c45e37","reply_to_id":"60684d2b56cd2c8c","content":"am I missing here, I am curious your take on reference type spec that already available in browser I suppose","create_time":"$D2022-08-23T21:12:13.161Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}},{"id":"586f55922cc3f5e8","reply_to_id":"60684d2b56cd2c8c","content":"I think there's a use case for storing the memory on the WebAssembly side, and also potentially a future use case for storing the memory on the JS side and accessing it via reference types. But my understanding is that reference types are new and not simple to use. At least the Rust tooling is not fully developed, I think, to use reference types","create_time":"$D2022-08-24T09:10:35.121Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"d003201354321004","login":"kylebarron","name":"Kyle Barron","avatar_url":"https://avatars-next.observableusercontent.com/0096ac4ec8894f01af5a4d303fe3463061511b44dda0e4803aafc9ede3f46507","tier":null}},{"id":"240e828f48067098","reply_to_id":"60684d2b56cd2c8c","content":"wasm-bindgen support reference type via --reference-types fyi, anyways great work, pal :)","create_time":"$D2022-08-24T09:41:19.148Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}},{"id":"0f6042b104f822e3","reply_to_id":"60684d2b56cd2c8c","content":"It's true that wasm-bindgen supports reference types, but that doesn't really help on its own. The Arrow C Data interface only references _other places in memory_, so even with reference types, Wasm can't reference other places in JS memory, and thus Wasm can't do zero-copy reads from JS (without a copy on the JS side into an IPC buffer).","create_time":"$D2022-08-24T14:29:58.866Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"d003201354321004","login":"kylebarron","name":"Kyle Barron","avatar_url":"https://avatars-next.observableusercontent.com/0096ac4ec8894f01af5a4d303fe3463061511b44dda0e4803aafc9ede3f46507","tier":null}},{"id":"98ba6b4aa892e03b","reply_to_id":"60684d2b56cd2c8c","content":"thats true, reference type can hold of that Js object alone, if there is no connection from externalRef to memory you are interested in Js, then its useless, I am not expert of arrow, but found it odd then Js arrow table can't also reach that memory you are interested in Js too, right?","create_time":"$D2022-08-24T15:10:16.761Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}},{"id":"06e85342d6c17f7f","reply_to_id":"60684d2b56cd2c8c","content":"to have clear understanding, your goal is to use arrow fns in browser via webassembly that meant to parse arrow parqute format by implementing Arrow C Data interfaces in parser itself, no need of additional Js or wasm library access the data from parsed parqute in-memory in wasm, right ? :)","create_time":"$D2022-08-24T15:55:17.796Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}},{"id":"7fc311f3ab60fd21","reply_to_id":"60684d2b56cd2c8c","content":"so your parqute-wasm not only parse dot Arrow file and provide arrow fns via Arrow C Data interface implementation as side-car to your parquet parser/db wasm","create_time":"$D2022-08-24T15:57:15.903Z","update_time":null,"resolved":true,"user":{"type":"individual","id":"48b74598db64e1b5","login":"samkumar15","name":"SAMBATH KUMAR LOGAKRISHNAN","avatar_url":"https://avatars-next.observableusercontent.com/ddbdaeaffbca6b6c8371352ebb6f5e854f3c56adc56ec7b59eababa4f03d6711","tier":null}}],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":5,"collection":{"id":"12189391458021a4","slug":"geoparquet","title":"Cloud-Native Geospatial","owner":{"id":"43d330d6f3e9ea97","login":"kylebarron","name":"Kyle Barron"}},"type":"notebook","body":{"title":"Zero-copy Apache Arrow with WebAssembly","cells":[{"id":0,"value":"# Zero-copy Apache Arrow with WebAssembly\n## _Implementing the Arrow C Data Interface in JavaScript for zero-copy data access across the WebAssembly boundary_\n\nKyle Barron
\nAugust 22, 2022","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":476,"value":"_Edits_\n\n> July 9, 2023: [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi), a library that implements the FFI described in this post, has been [published to NPM](https://www.npmjs.com/package/arrow-js-ffi).\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":7,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":15,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":27,"value":"## Use Cases in the Browser\n\nIf we implement support for the C Data Interface in JavaScript, JS programs can read Arrow memory from WebAssembly without a copy, making existing use cases more performant and new ones possible.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":70,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":72,"value":"### Persist data in WebAssembly\n\n[Polars](https://github.com/pola-rs/polars) is a fast data frame library written in Rust and built on Apache Arrow. While Polars already has bindings to Python and Node.js, one friction discussed in proposed Wasm bindings included data copies to/from JS. Zero-copy views on Wasm memory make it possible to _always keep the data on the Wasm side_ across data frame operations, ensuring only WebAssembly mutates the data and the JS side is read-only.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":236,"value":"### Better WebAssembly memory utilization?\n\nIn the case of copying data from Wasm to JS, when serializing to the IPC format, all Arrow memory must be contiguous so that it can be sent to JS as a single buffer. But when exposed through the C Data Interface, each underlying array chunk can be stored separately in memory. In a constrained memory environment, I _think_ this might mean the Wasm allocator can utilize its full memory space more effectively.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":38,"value":"## How WebAssembly memory works\n\nThere are [various](https://radu-matei.com/blog/practical-guide-to-wasm-memory/) [excellent](https://wasmbyexample.dev/examples/webassembly-linear-memory/webassembly-linear-memory.rust.en-us.html), [in-depth](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/Memory) references out there on how WebAssembly memory works.\n\nFor our purposes, all we need to know is that Wasm memory is just a normal JS `ArrayBuffer`! The Wasm program has the onus of managing allocation of data into that `ArrayBuffer` and freeing memory as needed. Pointers are just integer indexes within the `ArrayBuffer`. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":92,"value":"## Implementation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":17,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":19,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":23,"value":"## Demo!\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":181,"value":"For this demo we'll:\n\n- Load the Arrow JS and `parquet-wasm` libraries.\n- Create an example Arrow JS `Table` in JS memory.\n- Pass that table into WebAssembly memory.\n- Read zero-copy views of the data from WebAssembly.\n- Assert that the Arrow JS table and the view of the Wasm table are equal.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":185,"value":"### Load libraries\n\nFirst, let's load the Arrow JS and `parquet-wasm` libraries:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":98,"value":"arrow = import(\"https://cdn.skypack.dev/apache-arrow@9\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":225,"value":"When using the ESM entry point of `parquet-wasm` or other libraries created with [`wasm-bindgen`](https://github.com/rustwasm/wasm-bindgen), we always need to `await` the default export. \n\nBut in this case we _also_ need to retain the `WebAssembly.Memory` object returned, because that serves as a reference to the underlying `ArrayBuffer` that WebAssembly is using as its memory space.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":208,"value":"// Load the parquet-wasm library\nparquetModule = {\n const parquetModule = await import(\n \"https://unpkg.com/parquet-wasm@0.4.0-beta.5/esm/arrow2.js\"\n );\n // Need to await the default export first to initialize the WebAssembly code\n const {memory} = await parquetModule.default();\n return [parquetModule, memory];\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":469,"value":"parquet = parquetModule[0]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":471,"value":"wasmMemory = parquetModule[1]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":191,"value":"### Create Arrow JS Table\n\nTo keep it simple for this example, we'll create an Arrow table using the Arrow JS library with just two columns: one with dtype `uint16` and the other with dtype `float32`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":102,"value":"arrowTable = {\n // An index column with ordered integers from 0 up to 10,000 t\n const indexArray = new Uint16Array(10000);\n for (let i = 0; i < indexArray.length; ++i) {\n indexArray[i] = i;\n }\n\n // Random float values between 0 and 100\n const valuesArray = Float32Array.from({ length: 10000 }, () =>\n Number((Math.random() * 100).toFixed(1))\n );\n\n // Create the Arrow Table\n return arrow.tableFromArrays({\n index: indexArray,\n values: valuesArray\n });\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":193,"value":"### Create Table in WebAssembly Memory\n\nSince I've implemented preliminary FFI support in `parquet-wasm`, an easy way to get the above Arrow table into WebAssembly memory is to first encode it as Parquet and then read it back via FFI.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":123,"value":"parquetBuffer = parquet.writeParquet(arrow.tableToIPC(arrowTable, \"file\"));","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":127,"value":"arrowTableFFI = parquet._readParquetFFI(parquetBuffer);","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":240,"value":"This `arrowTableFFI` object is now a handle onto a collection of Arrow FFI structs in Wasm memory!","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":195,"value":"### Interpret WebAssembly Memory\n\nNow comes the exciting part! I've started [a JS library](https://github.com/kylebarron/arrow-js-ffi) to simplify interpreting Arrow memory, but here we'll copy in some relevant bits ourselves to see how things work at a low level.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":249,"value":"Note that the Arrow C Data Interface specification only defines individual schema fields and array chunks. In order to describe a complete logical table, we need a collection of fields and array chunks. This API to access the pointers where each `ArrowSchema` and `ArrowArray` struct is located is custom and _not_ part of the Arrow C Data Interface.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":253,"value":"There are two fields in this schema:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":130,"value":"arrowTableFFI.schemaLength()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":255,"value":"Calling `schemaAddr` with each field index gives the indices into WebAssembly memory where the `ArrowSchema` structs are defined.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":132,"value":"schemaPointers = [arrowTableFFI.schemaAddr(0), arrowTableFFI.schemaAddr(1)]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":259,"value":"To interpret the memory from these two locations, we can use `DataView` to read memory according to the C Data Interface spec","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":136,"value":"$36","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":403,"value":"Now we'll parse those two `schemaPointers` into Arrow JS `Field` objects! These act just the same way as normal ones created in Arrow JS.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":145,"value":"arrowFields = [\n parseField(wasmMemory.buffer, schemaPointers[0]),\n parseField(wasmMemory.buffer, schemaPointers[1])\n]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":264,"value":"Now let's move on to parsing the array data. We have just one record batch/chunk:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":262,"value":"arrowTableFFI.chunksLength()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":268,"value":"And two arrays in that first chunk:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":266,"value":"arrowTableFFI.chunkLength(0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":406,"value":"Let's get the pointers to each low-level `ArrowArray` FFI struct:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":270,"value":"arrayPointers = [arrowTableFFI.arrayAddr(0, 0), arrowTableFFI.arrayAddr(0, 1)]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":408,"value":"Now we can parse those `ArrowArray` structs and make an Arrow table. Note that for simplicity this doesn't parse every field described in [the spec](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":272,"value":"$37","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":411,"value":"Parse our two array objects:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":286,"value":"arrowArrays = [\n parseArray(wasmMemory.buffer, arrayPointers[0], arrowFields[0].type),\n parseArray(wasmMemory.buffer, arrayPointers[1], arrowFields[1].type)\n]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":413,"value":"Now finally, using both the `arrowFields` and the `arrowArrays`, we can construct an Arrow JS `Table` object! All underlying data still exists in Wasm memory! We've only taken views onto wasm memory!","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":294,"value":"arrowTableFromFFI = {\n const schema = new arrow.Schema(arrowFields);\n const recordBatch = new arrow.RecordBatch(\n schema,\n arrow.makeData({ type: new arrow.Struct(), children: arrowArrays })\n );\n return new arrow.Table(schema, recordBatch);\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":417,"value":"Voilà! As a reminder, this is a simple example to show how the low level process works. I have a (work-in-progress) library at [kylebarron/arrow-js-ffi](https://github.com/kylebarron/arrow-js-ffi).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":200,"value":"### Assert Both Tables are Equal","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":394,"value":"A quick check to verify that the arrays in the owned JS table and the viewed Wasm table are in fact equal:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":322,"value":"arraysEqual = {\n // https://stackoverflow.com/a/16436975\n return function arraysEqual(a, b) {\n if (a === b) return true;\n if (a == null || b == null) return false;\n if (a.length !== b.length) return false;\n\n // If you don't care about the order of the elements inside\n // the array, you should sort both arrays here.\n // Please note that calling sort on an array will modify that array.\n // you might want to clone your array first.\n\n for (var i = 0; i < a.length; ++i) {\n if (a[i] !== b[i]) return false;\n }\n return true;\n }\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":306,"value":"arraysEqual(\n arrowTable.getChild(\"index\").toArray(),\n arrowTableFromFFI.getChild(\"index\").toArray()\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":328,"value":"arraysEqual(\n arrowTable.getChild(\"values\").toArray(),\n arrowTableFromFFI.getChild(\"values\").toArray()\n)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":21,"value":"## Caveats\n\nA few caveats to this approach:\n\n- I don't believe Wasm can read JS memory, so if your data _starts_ in JS, you'll still need an initial copy into Wasm.\n- In general, data in the Wasm side should be considered read-only. Mutating the underlying array could easily corrupt memory. That said, mutations _could_ safely be done in place from the JS side as long as the length of any buffer does not need to change.\n- But the JS Arrow table isn't _actually immutable_, so the onus is on the programmer to handle memory management correctly.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":29,"value":"## Future Work\n\n- Cleaner FFI API that prevents shooting yourself in the foot. As it is, careful attention to memory management is required.\n- Verify that freeing memory from JS is working correctly\n- Implement FFI handling for more complex data types, including nested types, dates/times, nulls.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":433,"value":"## Footnotes","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":450,"value":"1. [^](#cell-15) It's hard to tell but [`duckdb-wasm`](https://github.com/duckdb/duckdb-wasm) _might_ use the C Data Interface. They seem to [define functions for it on the C++ side](https://github.com/duckdb/duckdb-wasm/blob/03daa5df333de302d0f5ce621bc5e6ad66b7829e/lib/include/duckdb/web/arrow_bridge.h), but on the JS side they seem to [always copy data](https://github.com/duckdb/duckdb-wasm/blob/cf13a28980340325065b7afd177ce80fc6ccf7c0/packages/duckdb-wasm/src/bindings/bindings_base.ts#L169), and [expect an IPC stream](https://github.com/duckdb/duckdb-wasm/blob/cf13a28980340325065b7afd177ce80fc6ccf7c0/packages/duckdb-wasm/src/bindings/connection.ts#L31). Their [blog post](https://duckdb.org/2021/10/29/duckdb-wasm.html#looks-like-arrow-to-me) does seem to say my zero-copy approach is unwise 😅.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":31,"value":"## Acknowledgments\n\nThanks to Avi Bryant for hosting me at [Gradient Retreat](https://www.gradientretreat.com/) a few weeks ago, where I came up with this idea while chatting with Tom MacWright. Thanks to all the Arrow contributors for making the libraries so great.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":437,"value":"footnote = (referencingCellName, number, text) => {\n return html`

^ ${text}

`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W38","stdlib":"1","titling":"auto"}} 30:"$2f:body"