Christoph Pahmeyer

Collection of notebooks related to data science in Observable

Observable

Data Science in Observable

Examples of how Data Scientists are using Observable

Data Science 

Statistics in Observable

Nicolas Lambert

I couldn't find it on the npm. Really like this one, not sure why the author don't put it on the Github.

Guten

Hey everyone, author here. There's multiple reasons the package isn't on Github & npm yet. First I wanted to get some more feedback and potential issues out of the way before publishing. Second the Wilkinson formula parser is super rudimentary, and the whole library lacks testing. I was afraid that someone might be using it, gets bad & wrong results without receiving any errors, and then potentially works with these results unknowing that they might be wrong. Third is that at some point I forgot to upload the code to Github, haven't done so since because it's messy 😊  Fourth is that I'm more and more convinced that a preferred way to approach this topic is to build the library fundamentals in Rust, and then create bindings to JS and Python so everyone is happy and can consume a stable and fast library. Re-inventing the data science wheel in JS is definitely doable, but I think it would be more sustainable to jump on existing projects in lower level languages!

That said the main missing link so far is the Wilkinson parser. Formulaic (https://matthewwardrop.github.io/formulaic/) looks great, but it's entirely written in Python so would need a Rust/C re-write. BASFs rormula (https://github.com/basf/rormula) goes in the right direction, it's a Rust package with Python bindings. It's still a very early project though and missing a lot of features.

Can summary work with categorical data? Or at least ignore it?

That is do we do with something that d3.autoType can't convert to a number? For example, this crashes:
summary([{m:1,n:2,c:'red'}])

Mark McClure

Hey Mark! That's a great find! I had to double check what R does when a data frame contains categorial data: it counts the number of characteristics values in the column.
I made a quick fix, so it should work with categorial data now (see below for your example, as well as a larger one using artificial data).

The current implementation is really inefficient though, and will throw "Maximum call stack size exceeded" if you feed it more than 100k rows. I started working on a fix (drastically reducing the number of loops through the dataset), but it's not quite done yet. 

Once this is done, I'll try to work on parsing arithmetic expressions in the formula string. Once this works, interaction terms should be relatively quick to implement!

Is there a particular feature you are missing or would like to see?

I just finished re-writing the summary function for data frames. A remaining performance culprit currently is filtering the unique keys from the array of objects. Right now, I'm just using the keys from the first (header) row. But in JS land the key might have been omitted if no data was present in that particular row, leading to a missing column in our summary table.

Before, I've been using this ES6 construct: [...new Set(...dataframe.map(row => Object.keys(row)))], but that's giving me the "Maximum call stack size exceeded" error one we exceed 100k rows.

For now I switched to the following

const columnNames = [];
    for (let i = 0; i < dataframe.length; i++) {
      const keys = Object.keys(dataframe[i]);
      for (let j = 0; j < keys.length; j++) {
        if (columnNames.indexOf(keys[j]) === -1) columnNames.push(keys[j]);
      }
    }

I'll ask around if there is a more performant way though...

Thanks to @Vladimir Agafonkin there's now a more performant approach!