Éric Mauvière

Rather than use dictionary types, have you tried just exporting your grouping vals from R as integers directly? In my experience, even decoding short unicode strings using Arrow JS can be quite expensive and it seems like the data here is just numeric anyway, and could just be packed into a single bit. You'd lose the ability to pull the keys without a scan, but you could put that into the column metadata if you really wanted; and I wouldn't be surprised if things got yet faster.

Benjamin Schmidt

I'd say that factor encoding in R is precisely storing values as integers, and levels as metadata (attributes), so i think that when reading the arrow format exported from R, we are just decoding integers for those dictionary-encoded columns. Anyway, eventually, decoding is really fast (10 ms, see just below), so it will be difficult to do better ?

It wouldn't speed up the initial load, but if you try to access the attributes when iterating over an arrow table, it will perform a (strangely costly) utf-8 conversion for each item. This is why the 'unpack = true' speed thing; it handles the unpacking in a smarter way than the native arrow functions at the cost of using some javascript memory--I think the unpack code in arquero is the fastest of the methods in the benchmarking I did here: https://observablehq.com/@bmschmidt/faster-arrow-dictionary-unpacking.

So if you use ints instead of dictionaries, you could

1. Maybe speed up the arrow filtering function might be faste because (I think) arrow's 'scan' may be doing 30 million utf-8 conversions.
2. Avoid the time in the arquero loop of having to unpack the table from native arrow format, because unpack only really helps if there's a utf-8 operation.
3. Reduce memory usage, because an array of 6 million strings takes up some real space.

Arrow's native countby is already fast because it works on the integer keys, but it seems like the effort to develop those internal computer functions has kind of stalled out.

But I'm not totally sure, which is why I'm curious.

I agree with you, builtin Arrow toArray() is really slow and we should avoid using it, so does, very smartly, Arquero. 

I am pretty sure that, conversely, Arrow filter() is well optimized, because filtering on 3 categorical columns takes only 300 ms, though for instance with my big table, arrow_tb.getColumn("CS1_8").toArray() would take 7 s.

Gnerally speaking, i would delay conversion to JavaScript as much as possible. Usually, when dealing with a big table like this one, we want to dramatically reduce the number of rows, by filtering and summarising. So my point is: let's use Arrow built-in functions to run filters and rollups, and then let's enjoy Arquero's elegance and simplicity of use.

Anyway, and i have exchanged with Jeffrey Heer about this, i have no doubt Arquero will be able, soon, to leverage built-in Arrow fonctions, in a transparent way.

Finally, if performance were the ultimate critreria, i would embrace your "conversion to integers" idea, at the price of extra coding. 

But for the sake of simplicity, using factors when exporting from R, and dictionary decoding with javascript appears to me as a decent all-terrain tradeoff

Cool, you've convinced me that the predicate functions are working on dictionary keys inside arrow with filter(), although I can't quickly track down where in the code it happens.

I'm more pessimistic about general interop with the built-in arrow functions is that right now sometimes they work quickly, sometimes they work slowly, and sometimes they're broken! And with so little maintenance on the typescript repo right now, it's hard to tell which is which, especially because there are edge failures. E.g.--toArray() fails entirely on filters (think this was in your github issue) or on arrays with nulls (which I have an issue up on arrow JIRA that is unaddressed). 'countby' is fast but only works on dictionaries. Kind of a mess. I wouldn't be surprised if the future of compute kernels in the browser ends up being WASM compilations of stuff in C or Rust rather than the arrowjs project.

The rollup example you give at the end is great, BTW. I'm surprised it's so much faster than the arquero equivalent even excluding the unpack time, and not fully sure why.

Thank you for that stimulating exchange.I am a priori inclined to be confident about Arrow, considering the actors who have been supporting this project since 2016, such as Hadley Wickham (R/tidyverse) or Wes McKinney (Python/Pandas). The release of arrow 2.0 this fall is a really important step, the format now seems to be stabilized, and McKinney announces future advances. Let's hope that the js library won't be left behind...

Hey Benjamin - you're right sadly Arrow JS hasn't gotten a lot of attention since Paul and I both changed jobs back in 2018. I tend to agree with you that compiling C++ or Rust to WASM is a better long-term option for compute in the browser. We'll still likely need some Arrow JS lib that can read compute results and display them.

Even arquero is probably a better option at this point, the only thing that arrow compute does well that's missing in arquero is optimize some operations on dictionary-encoded columns. So maybe the medium-term solution is add that ability to arquero (discussed in https://github.com/uwdata/arquero/issues/51#issuecomment-765738331).

In the meantime, I'm happy to help fix Arrow JS bugs and/or review PRs, please ping me on PRs (github id: TheNeuralBit) or jiras (ASF id: bhulette).

FWIW here is the code that optimizes equality checks on dictionary-encoded columns: https://github.com/apache/arrow/blob/205e6da57671da3107eaedfef1a6470f2947820a/js/src/compute/predicate.ts#L182-L205