Christoph Pahmeyer

Collection of notebooks related to data science in Observable

Observable

Data Science in Observable

Examples of how Data Scientists are using Observable

Data Science 

Statistics in Observable

Nicolas Lambert

Can summary work with categorical data? Or at least ignore it?

That is do we do with something that d3.autoType can't convert to a number? For example, this crashes:
summary([{m:1,n:2,c:'red'}])

Mark McClure

Hey Mark! That's a great find! I had to double check what R does when a data frame contains categorial data: it counts the number of characteristics values in the column.
I made a quick fix, so it should work with categorial data now (see below for your example, as well as a larger one using artificial data).

The current implementation is really inefficient though, and will throw "Maximum call stack size exceeded" if you feed it more than 100k rows. I started working on a fix (drastically reducing the number of loops through the dataset), but it's not quite done yet. 

Once this is done, I'll try to work on parsing arithmetic expressions in the formula string. Once this works, interaction terms should be relatively quick to implement!

Is there a particular feature you are missing or would like to see?

I just finished re-writing the summary function for data frames. A remaining performance culprit currently is filtering the unique keys from the array of objects. Right now, I'm just using the keys from the first (header) row. But in JS land the key might have been omitted if no data was present in that particular row, leading to a missing column in our summary table.

Before, I've been using this ES6 construct: [...new Set(...dataframe.map(row => Object.keys(row)))], but that's giving me the "Maximum call stack size exceeded" error one we exceed 100k rows.

For now I switched to the following

const columnNames = [];
    for (let i = 0; i < dataframe.length; i++) {
      const keys = Object.keys(dataframe[i]);
      for (let j = 0; j < keys.length; j++) {
        if (columnNames.indexOf(keys[j]) === -1) columnNames.push(keys[j]);
      }
    }

I'll ask around if there is a more performant way though...

Thanks to @Vladimir Agafonkin there's now a more performant approach!