Would it make sense to bin data on the triangular net? Each bin would cover the same surface area, which hexagons don't do here since they're cut on the edges.

Yes, triangular bins would be better than hexagonal bins, especially if the hexagon radius is large. I might look into triangular binning support later. I couldn't find much on triangular binning in d3, only this GitHub repo. https://github.com/armollica/triangle-bin and I suppose d3-hexbin could be modified to use triangles instead of hexagons.

My solution https://observablehq.com/@fil/ternary-binning

Wow, thanks Fil! This looks really great! I'll see if integrate it into my nascent d3-ternary-plot module if you're ok with that, if so could you add a license to the notebook? Thanks a lot

sure… I've added a MIT license