Kyle Barron

Cloud-Native Geo

why don't we use reference type specification of wasm to access arrow table in JS?

SAMBATH KUMAR LOGAKRISHNAN

am I missing here, I am curious your take on reference type spec that already available in browser I suppose

I think there's a use case for storing the memory on the WebAssembly side, and also potentially a future use case for storing the memory on the JS side and accessing it via reference types. But my understanding is that reference types are new and not simple to use. At least the Rust tooling is not fully developed, I think, to use reference types

wasm-bindgen support reference type via --reference-types fyi, anyways great work, pal :)

It's true that wasm-bindgen supports reference types, but that doesn't really help on its own. The Arrow C Data interface only references _other places in memory_, so even with reference types, Wasm can't reference other places in JS memory, and thus Wasm can't do zero-copy reads from JS (without a copy on the JS side into an IPC buffer).

thats true, reference type can hold of that Js object alone, if there is no connection from externalRef to memory you are interested in Js, then its useless, I am not expert of arrow, but found it odd then Js arrow table can't also reach that memory you are interested in Js too, right?

to have clear understanding, your goal is to use arrow fns in browser via webassembly that meant to parse arrow parqute format by implementing Arrow C Data interfaces in parser itself, no need of additional Js or wasm library access the data from parsed parqute in-memory in wasm, right ? :)

so your parqute-wasm not only parse dot Arrow file and provide arrow fns via Arrow C Data interface implementation as side-car to your parquet parser/db wasm