Another thing: since there is so much overplotting in this dataset, the input order of the data points has a big effect. Shuffling the data randomly inside the plot mitigates this bias.
Is this too wonky? I feel like the chart above is a nice takeaway. Unsure if this adds much to the notebook / collection (though many people won't traverse the full collection).
I think this fits here, as an extention. IE - the conclusion from above is that:
a - there is clearly a correlation between temp & energy distribution, with peaks for both colder & hotter temperatures.
b - the facets suggest that the periods matter too, in particular, the hourlies.
But... what do we do next to test this? Let's switch our point of view, to focus on fixing the temp bands, rather than the day.
We could also link to Discovering Date Patterns, as this is essentially a variation on that technique of augmenting our data with categoricals, except for the x-facet we've constructed categories by binning to 5-degree bands instead of binning by time period.
It's pretty clear from the graph that the correlation between temp & energy is non-linear, so it feels strange to me to be applying a linear correlation function: https://observablehq.com/@uwdata/arquero-cookbook#corr.
The other day you suggested that the difference from ~15 would be interesting. I agree.
So, one alternative would be to calculate the linear correlation between demand and the absolute value difference between the temp and 13.6 degrees (we could even calculate the correlation for abs value from values between 13.1 to 15.9 to find the bottom - I tried to do this but ran into an arquero error message).
I've added two charts showing what the regression line looks like (from Python) for both temp itself and abs(temp - 13.6)
we don't want to show these charts here (too many dots, don't have a good way to calculate the regression line as far as I can tell)... just adding to illustrate for you.
I think we should drop everything below this line.
I like the direction you're going w/ correlations... but it doesn't make sense to calculate a linear correlation on this data given what we saw in the first half. There might be things worth doing (the correlation of abs(temp - 13.6), but I want to be more careful about getting this right.
This is the first time we're really going into any sort of algorithmic calculation. And, if we're going to go that direction, I want to do it thoughtfully (not at 10pm at night when my todo list is too long, and knowing that you're ooo next two days). The risk I see is making some obvious mistake, and losing cred.
And, previously we were already debating if this last part fit. I think the direction you're going both has the potential for a stronger story... and means that it's more of a second story.
---
I'm going just going to be decisive right now.
I'm going fork to save this work. Then remove this content from this notebook. And, if we want to do this, let's do it as a fast follow (even before publication).