2f:T4af,# Purescript incremental relational lenses 1/n Understanding and implementing incremental relational lenses in Purescript. - Part 1 - [Part 2](https://observablehq.com/d/c893cda26d4f49bf) In this first part we: - Adapted a few basic pieces of kit from the talk to purescript (rows, project, record set, - Started reading the paper and the paper it builds on top of. - Identified the key algorithm for put incrementality - Realised this work didn't include view maintenance - Looked at other haskell/purescript DB apis to see how and where to declare functional dependencies - Identified Selda as a good building block - Found a minimal example to work with for the simplest cases - Started to lay down some thoughts about how to deal with a pure version with extensible effects ## Sources - https://www.youtube.com/watch?v=e0JST9vmhFc&t=896s - https://github.com/rudihorn/haskell-lenses This is a great visual summary of what this is all about from [the paper](https://arxiv.org/pdf/1807.01948.pdf) Horn, Rudi et al. “Incremental relational lenses.” Proceedings of the ACM on Programming Languages 2 (2018): 1 - 30. ![image@1.png](${await FileAttachment("image@1.png").url()}) 30:T1c14,### Functional dependencies They are between fields in a table and need to be defined by the user with their schema. Here's an example of the haskell syntax from the [Scratch.hs](https://github.com/rudihorn/haskell-lenses/blob/master/Scratch.hs) file. ```haskell type Albums = '[ '("album", String), '("quantity", Int)] albums = prim @"albums" @Albums @'[ '["album"] --> '["quantity"]] ``` Oh actually, `albums` is a lens. Could/should the functional dependency be instead a schema annotation? I thought this was a relational algebra concept. Let's take a look at the paper. In [Horn, Rudi et al. “Incremental relational lenses.” Proceedings of the ACM on Programming Languages 2 (2018): 1 - 30.](https://arxiv.org/pdf/1807.01948.pdf) > These relational lenses are defined using put functions which map the source database and an updated view to the updated source database. Bohannon et al.’s work showed that it is possible in principle for databases to support updatable views including joins, provided the type system track integrity constraints on the data, such as functional dependencies > The lens keyword makes a base table into a basic lens, and allows specifying additional functional dependencies beyond key constraints. There's a note here saying that these extra constrains aren't checked in their `links` implementation. > These functional dependencies are as specified by Bohannon et al. [2006a], but result in a database with some redundancy; however, we keep the example as is for ease of comparison with prior work. This also suggests that there might be more to dig into here. That might take me on a tangent, but I'd like to see how other database layers in FP handle constraints in general. Let's look at: - [Haskell's Beam](https://haskell-beam.github.io/beam/): uses HKDs and type classes with an associated datatype "PrimaryKey". Foreign key constraints embed the PrimaryKey of the referred to table. That's clean. - [Rel8](https://rel8.readthedocs.io/en/latest/) (a layer on top of [OpalEye](https://hackage.haskell.org/package/opaleye)): [rel8 doesn't seem to deal with key constraints](https://github.com/circuithub/rel8/issues/186#issuecomment-1172265389) and neither does opaleye AFAICS. - project-m36 (closest to the relational algebra spirit): [Project:M36 includes first-class support for functional dependencies ](https://github.com/agentm/project-m36/blob/bdd130e1135e7353483d69dcd1713210bc710140/docs/tutd_tutorial.markdown?plain=1#L606C23-L606C91) - they're not modelled with haskell fundeps though. - tangent: haskell fundeps according to the [GHC User Guide](https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/exts/functional_dependencies.html#) come from the [Type Classes with Functional Dependencies paper from Mark P.Jones](https://web.cecs.pdx.edu/~mpj/pubs/fundeps.html) > A particular novelty of this paper is the application of ideas from the theory of relational databases to the design of type systems. Ha, so FP fundeps are inspired from relational algebra. Doesn't mean they map 1 to 1, or that using FP fundeps will help with modeling RA fundeps. But I think Rudi is saying that the haskell type system can help to ensure RA fundeps are respected. - Maybe something in purescript - Categorical databases and catlab in Julia Thinking about it from an engineering standpoint, I don't think we can introspect/reflect functional dependencies. It would also mean having to define class for tables which seems wrong. So RA fundeps (maybe we should rename them constraints here) need to be expressed with their own thing. In [Rudi's work it's a data type](https://github.com/rudihorn/haskell-lenses/blob/master/FunDep.hs): ```haskell data FunDep where FunDep :: [Symbol] -> [Symbol] -> FunDep ``` Personally I like the HKD approach a lot, because it provides a lot of reusability for the domain datatypes. I think they can be derived in purescript too like with barbies and higgledy in haskell. - There's Jordan's reference https://jordanmartinez.github.io/purescript-jordans-reference-site/content/31-Design-Patterns/43-Higher-Kinded-Data.html - Someone [seems to have tried to use generics with HKDs](https://discourse.purescript.org/t/generics-rep-for-higher-kinded-data/1811) but not clear if it's what I'm looking for (deriving an HKD from a normal data type, but to be fair that's not really so important right now). The [Reducing Boilerplate](https://jordanmartinez.github.io/purescript-jordans-reference-site/content/31-Design-Patterns/43-Higher-Kinded-Data.html#reducing-boilerplate) example is really cool. So does this mean we're heading the Beam way? If so: - Can we convert key constraints into the FunDep representation? - Can we make these constraints available to the lens when we build it? Meaning instead of the example above we would have something like: ```haskell data AlbumT f = Album { "album" :: Columnar f String , "quantity" :: Columnar f Int } instance Table AlbumT where data PrimaryKey AlbumT f = AlbumKey (Columnar f TexT) deriving (Generic, Beamable) primaryKey album = AlbumKey <$> album type Album = AlbumT Identity -- first VTA is the name of the DB table I think, albums = prim @"albums" @Album -- could we completely let go of passing the contraint here? -- @'[ '["album"] --> '["quantity"]] ``` Part of me likes this, but another part of me finds this pretty verbose. Let's continue looking at other DSLs. - This is pretty https://github.com/joneshf/purescript-typelevel-sql/blob/master/src/Main.purs - Nate's `purescript-convertable-options` is also worth getting inspiration from. https://github.com/natefaubion/purescript-convertable-options (the `poly` case is an HKD). I think if AlbumT was just the canonical HKD (AlbumF?) I would be ok with that. So something like: ```haskell data AlbumF f = Album { "album" :: f String , "quantity" :: f Int } -- type AlbumT f = AlbumF (Columnar f) -- Not sure this kind checks ? I think I need the polynomial functor kit. ``` Maybe we use a row type with orthogonal annotation for serialisation, contraints/fundeps,... In the spirit of Joran's recordOrVariant? I think it's even more higher order than that... ```haskell -- same AlbumF type AlbumCol = ``` I need to understand `Columnar` [ha](https://github.com/haskell-beam/beam/blob/86916c1274dbfb570c59f2b8f7cb48c7ef87f133/beam-core/Database/Beam/Schema/Tables.hs#L447) > A type family that we use to "tag" columns in our table datatypes. > This is what allows us to use the same table type to hold table data, describe table settings, > derive lenses, and provide expressions. I wonder if that's again a byproduct of not having row types...? Or maybe more precisely, it's a way to define a set of patterns for HKDs for Beam. Yes, it's a bit like what I intuit with "orthogonal annotations". So it does feel like effect handling, and maybe a hierarchical free monad of sorts to express that some part of the code depend on an annotation set described in some kind of variant/product? Hurts my brain a bit. Interesting that they have lenses there too. I don't think those are relational lenses though. 31:Tcfd,### A detour via purescript-selda Ok I had forgotten about https://github.com/Kamirus/purescript-selda - Oh cool there's a paper https://github.com/Kamirus/purescript-selda/blob/master/selda.pdf - There's sqlite3 support This is neat and terse ```haskell people ∷ Table (id ∷ Int, name ∷ String, age ∷ Maybe Int) people = Table { name: "people" } ``` What about keys? Yeah not handled it seems. > Also, we have not mentioned that id is a primary key, because for the purpose of this EDSL we do not need this information And > Please note that purescript-selda does not handle schema modification, such as table creation, we are doing it manually using postgresql-client. So it is important to correctly define the tables and types for its columns. That's fine. If we just add the fun deps maybe it's enough for relational lenses? (and maybe later it could be use for schema creation/modification). Oh! Haskell's [selda](https://selda.link/) does pass this along! ```haskell people :: Table Person people = table "people" [#name :- primary] ``` We're on the right track 😊 `primary` is defined [here](https://github.com/valderman/selda/blob/ab9619db13b93867d1a244441bb4de03d3e1dadb/selda/src/Database/Selda/Table.hs#L172) as ```haskell primary :: Attribute Group t a primary = Attribute [Primary, Required] ``` Interesting. `#name` is a field selector. I suppose we can use VTAs for this in purescript with the label symbol? Ah and `:-` is [constructor for `Attr`](https://github.com/valderman/selda/blob/ab9619db13b93867d1a244441bb4de03d3e1dadb/selda/src/Database/Selda/Table.hs#L49-L55C10) which is what `table` expects a list of. It's very fancy that it also allows group selectors 😮. Ah yes, because it's used for queries too. How does `purescript-selda` handle this? - Oh wow, the error message handling is so good! - Ok, I might be wrong, but I think field names are in scope because of how queries are just lambdas over the record type. So we don't need fancy syntax to access those. #### Refresher on existentials continuing from https://github.com/Kamirus/purescript-selda/blob/master/selda.pdf ```haskell data CounterF a = CounterF { val :: a , get :: a → Int , inc :: a → a } type Counter = Exists CounterF -- data Exists f -- mkExists :: forall f a . f a -> Exists f -- runExists :: forall f r. (forall a. f a -> r) -> Exists f -> r myInitCounter :: Counter myInitCounter = mkExists $ CounterF { val: 0, get: \i -> i, inc : \i -> i + 1 } myGet :: Counter -> Int myGet counter = runExists getC counter where getC :: forall a. CounterF a -> Int getC (CounterF { val, get }) = get val myInc :: Counter -> Counter myInc counter = runExists incC counter where incC :: forall a. CounterF a -> Counter incC (CounterF { val, get, inc }) = mkExists $ CounterF { val: inc val, get, inc } ``` The paper uses existentials for Columns as heterogeneous lists, which is still the case in code, but now the [expression type also has existentials](https://github.com/Kamirus/purescript-selda/blob/master/src/Selda/Expr.purs#L52-L55). Seems [they were first added](https://github.com/Kamirus/purescript-selda/blame/996e8e0ebf8cf87b3da91551e4fd1195bac28bd3/src/Expr.purs#L36) to support BinaryOp? I think to hide the type of the inner expressions. Ok let's zoom back out.32:T17bf,### How to go about implementing table contraints in purescript-selda If we have a good representation for this that we know will work with selda, we can then piggy back on that for the relational lens implementation. So we could extend `table` so that it takes attrs. Let's start with this thought. See how far we can go building a `table'` wrapping selda's `table`. Ok well we'll need to change the `Table` type to add attributes. Let's try. There are 2 types of table attrs in haskell's selda: - combinedAttrs: `Unique` and `Indexed` - pkAttr: `Primary` and `Required` Columns also can have ForeignKey attributes. Not sure if I'll need that for lenses. The fundep "treeform "check example in haskell-lenses's scratch doesn't seem to cover foreign key checks. This strangely [uses quantity as a dependency](https://github.com/rudihorn/haskell-lenses/blob/master/Scratch.hs#L117C6-L118). Oh https://en.wikipedia.org/wiki/Functional_dependency > Functional dependencies however should not be confused with inclusion dependencies, which are the formalism for foreign keys; even though they are used for normalization, functional dependencies express constraints over one relation (schema), whereas inclusion dependencies express constraints between relation schemas in a database schema. Interesting https://stackoverflow.com/a/35532653 > A table's superkeys (UNIQUE NOT NULL sets of columns) are the column sets that "functionally determine every other attribute in the table". So I suppose that in haskell-lenses, the `join` lens deals with inclusion dependencies. Can't find it in the paper. Ah it's mentioned in the original paper on relational lenses https://www.cis.upenn.edu/~bcpierce/papers/dblenses-pods.pdf but ass future work: > The choice of schema language is a fundamental issue in designing relational lenses, significantly constraining the design space for lens primitives. It may be fruitful to consider extensions to the schema language we have proposed here. Possibilities include multivalued dependencies and foreign key constraints—or, more generally, inclusion dependencies. Multivalued dependencies would allow us to support join lenses with wider domains. Among other benefits, inclusion constraints would allow us to define lenses for database normalization. Ok well, it would be good to have some better sense of [where the research is](https://scholar.google.com/scholar?hl=en&as_sdt=2005&sciodt=0%2C5&cites=9450561434602576893&scipsc=1&q=inclusion+dependencies&btnG=&oq=inclusion+depen) but it's too much to take on right now. Going back to fundeps. Bohannon seems to use them first in `Single-Dependency Record Revision`. I think this corresponds to the example given by Horn in `3.3 Relational Revision` (which is a great compact summary of Bohannon): > For example, given F = {A → B} and M = {{A = 1, B = 2}, {A = 2, B = 3}} and N = {{A = 1, B = 42}}, the result of revising M to be consistent with N and F is {{A = 1, B = 42}, {A = 2, B = 3}}. That's a good minimal example to workshop on top of. Let's try to write it in the haskell-lens way: ```haskell type U = '[ '("A", Int), '("B", Int)] m :: U m = rows @U [ ("A", 1), ("B", 2) ] type F = '[ ["A"] --> '["B"]] -- n is the modification to the view, which should allow us -- to infer m' the modified source of the view. n :: U n = rows @U [ ("A", 1), ("B", 42) ] -- it's implied that we have an identity lens here, but with the fundep F -- we also imagine we have a table `u` in the DB l = prim @"u" @U F -- the put assumes a db connection... feels like it would be good to have a pure version too... -- maybe a good case for purescript-run. At least returning the result of the lens! -- as `IO U` or something. revision :: IO () revision = test_put l n -- some IO modification of n which modifies a previously inserted m into m' ``` Anyhow, the work horse is in `3.3.2`: - If m and some n have the same value for X in the fundep (in our example `("A", 1)`) - we return a copy of m whose Y attributes have been updated with those from `n[Y]` (in our example `42`). Ok so, what if we had a fundep `A -> BC` and we only modify B? Yeah that's fine `n[Y]` then is only the B modification and leaves C intact. Ok let's find this algorithm in the haskell-lens code. I think [that's it for the simple select case](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Put/Classic.hs#L76C55-L76C55) ```haskell put_classic_select c (HPred p) (l :: Lens s) _ n = do m <- get c l let unsat = SR.filter (DP.not p) m let m0 = merge @(TopologicalSort (Fds s)) unsat n return m0 ``` Right then `merge` [is implemented in terms of `revise`](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Sorted.hs#L106-L107) (the names seems to indicate we're on the right track. Yes, then revise iterates on the fundeps and for each of them [it does](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Sorted.hs#L106-L107): ```haskell revise_fd :: forall (fd :: FunDep) rt rt'. (RevisableFd fd rt rt') => RecordsSet rt -> RecordsSet rt' -> RecordsSet rt revise_fd m n = Set.map update m where map = Map.fromList $ Set.toList $ Set.map f_row n f_row r = (R.project @(Left fd) r, R.project @(AdjustOrder (Right fd) (VarsEnv rt)) r) update r = case Map.lookup (R.project @(Left fd) r) map of Nothing -> r Just s -> R.revise r s ``` Which maps an update function over the incoming record, which: - checks that the projection of the left hand side of the fun dep exists in the `map` - which is the projection of the fundep lhs of the updated row n. - such that after the lookup, `Just s` is a "small" projected row of n over all the non root values I think. So in our example `[("B", 42)]` - if it does it calls `R.revise` [in Lens.Record.Base](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Base.hs#L203-L204) - which is called with each source row r and overwrites it with the value s of the "non-root" fields. ![image@2.png](${await FileAttachment("image@2.png").url()})33:T1309,### How about the incremental version That's a good moment too [look at how `put_delta`](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Put/Incremental.hs#L191-L205) does it! ```haskell put_delta c (Select (HPred p) l) delta_n wif = do unsat <- Set.fromList <$> query_ex @c @(Rt s) Proxy c tbls cols pred let delta_m0 = ((Delta.fromSet $ merge @(TopologicalSort (Fds s)) unsat (positive delta_n)) #- Delta.fromSet unsat) #- (Delta.fromSet $ negative delta_n) let delta_nh = (SR.filter p $ positive delta_m0, SR.filter p $ negative delta_m0) #- delta_n put_delta c l (delta_m0 #- delta_nh) wif where pred = DP.conjunction [affected @(Fds s) $ Delta.positive delta_n, query_predicate l, DP.not $ p] cols = column_map l tbls = recover_tables @(Ts s) Proxy ``` More involved as expected 🤓 Oh at first I kind of liked the indentation style, but now it totally confused me. But all good. So `unsat` is the application of the select, so we ignore that (UPDATE: nevermind, it's actually quite important, part of an optimisation and described in more details below). The main things are: - delta_m0, which also does the `merge @(TopologicalSort (Fds s)) unsat` we saw in the classic version. which is applied to `positive delta_n` so probably additions. But where does `delta_n` come from Ah yes, the main `put` first computes a delta and passes it to `put_delta` ```haskell put :: forall c s. LensPut c s => c -> Lens s -> RecordsSet (Rt s) -> IO () put c l rs = do unchanged <- query c l let delta = Delta.fromSet rs #- Delta.fromList unchanged put_delta c l delta False ``` I guess that's were we'd want a pure version. the query gets `m` and the `put_delta` does the `m'` modification hidden as implicit state inside `IO ()`. Ah so `#-` seems to be a difference operator that yields deltas between records. [Right](https://github.com/rudihorn/haskell-lenses/blob/master/Delta.hs#L31-L32) it's the `merge` (different merge) of a delta with the negation of another delta. using `Data.Set` union and `\\` difference. Interesting that rows are converted to deltas and the difference is on the deltas... Worth thinking about more later. Particularly if we need to hook this into a reactive/streaming abstraction. Oh the `Delta` datatype is a pair of positive and negatives. So pretty much exactly an ORSet... That's food for thought for future CRDT integration. Also since one of the one paper I saw on CRDTs and semantic web used a modification on ORSet to deal with tuple merging. That's the paper https://inria.hal.science/hal-00686484/document > Luis Daniel Ibáñez, Hala Skaf-Molli, Pascal Molli, Olivier Corby. Synchronizing Semantic Stores with Commutative Replicated Data Types. SWCS - Semantic Web Collaborative Spaces Workshop - 2012, Apr 2012, Lyon, France. pp.1091-1096, ff10.1145/2187980.2188246ff. ffhal-00686484f Hmm. It's only extending ORSet to SUSet because of the delete-insert atomic operation. Ah, I see. In RDF that could be a simple "row" update but span a lot of triples, and since we don't have "tables", the update could be large. Hence their comment > We could modify the operations to send the relevant set of triples to affect one by one, but that could flood the network with traffic considering the potential size of an RDF-Graph. Anyway, that's for later, but interesting to consider. Ok so back to `delta_m0` - It's doing the merge of these positive deltas (which are the changed rows). - then removes the non selected (`unsat`) rows - and removes the negative deltas (i.e. these rows that where deleted from the view). There is then `delta_nh`. Let's take a look at the paper. I think this is the juicy bit. Oh man, that looks so much like differentiation... wonder if there are results from AD or Zippers that could apply here. Also thinking about conflicts and transactions is going to be tough and interesting. ![image@3.png](${await FileAttachment("image@3.png").url()}) Oh ha ha > The δ getℓ direction simply performs incremental view maintenance, which is not our main concern here Oh yeah, indeed there is no incremental view maintenance code in haskell-lenses that I can see 😅 Oh well, hopefully having a incremental update maintenance implementation means we can draw the rest of the owl... Ok so we're looking at the optimised version. - Q is unsat (running a query with `pred` which select affected rows, which are the ones that are "touched" in the view, i.e. present in delta_n). It feels like building this for but also on top of `purescript-selda` is going to make some code much more pleasant. Ok so delta_nh is about rows which didn't satisfy the select predicate before the udpate, but do after. That's fine. I trust it. Ok so of course that's a lot of code, but I think I get the gist of it.34:Ta2c,### What next? So I guess before I break here, let's see what we have. - I don't see a problem in defining the fundeps in the schema. That's going to be a dependency of lenses anyway, so it shouldn't be an issue. - To be honest though, maybe starting with fundeps in the lenses will make integration with selda easier? And should be easier to code by following along the haskell code too... There's going to be tons of difficulties there anyway with type level stuff, so maybe a good way to break things into chunks. - Do we want to tackle the effect/interpreter side of things early? Maybe by implementing all this first as pure functions. The moment then when we wire the database will be a good moment to add this flexibility (probably with purescript-run). - Starting pure also means we don't have to deal with Effect in lenses, which I'm not sure are a thing in purescript. Oh there is https://github.com/sigilion/purescript-run-profunctor-lenses/blob/master/src/Run/Lenses.purs (oh interesting other repo https://github.com/brownplt/B2T2 about typechecker usability for databases, I wonder how haskell and purescript seldas would fare). Very tempting... - It would make sense to go in this order: - start with the classic lenses - start with the prim lens - then the select lens - then decide where to go. Ok let's just lay down some shapes before breaking. Oh sad. Run.Lenses is not in the default package set. I don't want to move to a local try purescript compiler yet though. So let's continue without it but with Run and with Data.Lens. It also feels like I could implement a version without the sophisticated type checking. And gradually implement it. Ok leaving it in this state for the next session 👋 ```haskell -- we'll want some DB effect and the ability to interpret lenses in it. -- let's start from STATE data Db s a = Db (s -> s) (s -> a) derive instance functorDb :: Functor (Db s) type DB s r = (db :: Db s | r) _db :: Proxy "db" _db = Proxy type Predicate a = a -> Boolean -- we're not changing the shape of things with a select -- in RA it's only a filter, projection is separate select :: forall s. Predicate a -> Lens' s a -> Lens' s a select pred _ = lens getSelect putSelect where getSelect :: s -> s getSelect s = s putSelect :: s -> s -> s putSelect _ s = s get :: forall s a r. Lens' s a -> Run (db :: DB s | r) ~> Run (db :: DB s | r) put :: forall s a r. Lens' s a -> Run (db :: DB s | r) ~> Run (db :: DB s | r) runDBPure :: forall s a. s -> Run (DB + r) a -> Run r s runDBPure = unsafeCrashWith "todo" ``` 35:T5ce,part1 = `module Main where import Prelude import Data.Lens (Lens, view, set) import Data.Lens.Record (prop) import Data.Foldable (class Foldable, intercalate) import Data.Set (Set, fromFoldable) import Data.Symbol (class IsSymbol) import Record (get, insert) as R import Prim.Row (class Cons, class Lacks) import Type.Proxy (Proxy(..)) type RecordSet rt = Set (Record rt) fetch :: forall @l a r r' . IsSymbol l => Cons l a r' r => Record r -> a fetch = view (prop (Proxy @l) :: Lens (Record r) (Record r) a a) update :: forall @l a r r' . IsSymbol l => Cons l a r' r => a -> Record r -> Record r update = set (prop (Proxy @l) :: Lens (Record r) (Record r) a a) update' :: forall @l b r1 r2. IsSymbol l => Lacks l r1 => Cons l b r1 r2 => b -> Record r1 -> Record r2 update' = R.insert (Proxy @l) project :: forall @l a r' r1 r2. IsSymbol l => Cons l a r' r1 => Cons l a () r2 => Record r1 -> Record r2 project (r :: Record r1) = R.insert (Proxy @l) (R.get (Proxy @l) r) {} rows :: forall f rows. Foldable f => Ord (Record rows) => f (Record rows) -> RecordSet rows rows = fromFoldable -- scratchpad type R = { "A" :: String, "B" :: String } rec :: R rec = { "A": "hello", "B": "world" } rws :: RecordSet ( "A" :: String, "B" :: String) rws = rows [ rec, rec, update @"A" "test" rec ] val :: String val = intercalate "\\n" [ fetch @"A" rec , show $ update @"A" "test" rec , show $ update' @"C" "test" rec , show $ project @"A" rec , show $ rws ] `36:[{"id":"6f40deb936bd75eae42893d683764ccb88a71c9bcc3e15c785b56a0afdfce87b9cd53e67885a65b409a003015d21dcd239bad13d3a8511f551e35de51ea399bb","href":"https://static.observableusercontent.com/files/6f40deb936bd75eae42893d683764ccb88a71c9bcc3e15c785b56a0afdfce87b9cd53e67885a65b409a003015d21dcd239bad13d3a8511f551e35de51ea399bb","name":"image.png","type":"image/png","lastModified":1703266068314.494,"size":14317},{"id":"9874c835503942a39b84b1a9855f88ccaca3d35d6b3b778c098e09cb36ea78073b92be809087f3a9c73fbdb526674d17d5922e943ba2e3170067452f73d9d460","href":"https://static.observableusercontent.com/files/9874c835503942a39b84b1a9855f88ccaca3d35d6b3b778c098e09cb36ea78073b92be809087f3a9c73fbdb526674d17d5922e943ba2e3170067452f73d9d460","name":"image@1.png","type":"image/png","lastModified":1703345410903.457,"size":177777},{"id":"e35027122bccc6670d55c77b18f4c4ec2f8ea5bd3f1002c0d3a5fe2f82f7123d4068966959c435c5002cbe4fd9d49cd4e49c996239554da3b178cb0a2df92941","href":"https://static.observableusercontent.com/files/e35027122bccc6670d55c77b18f4c4ec2f8ea5bd3f1002c0d3a5fe2f82f7123d4068966959c435c5002cbe4fd9d49cd4e49c996239554da3b178cb0a2df92941","name":"image@2.png","type":"image/png","lastModified":1703355555782.492,"size":49592},{"id":"2de0c5733d3a0cb9ff3050fca8dfaba9c7408a0a1863536d04747fdbad357dca196ecfd7b47f60b8c97bd9bba261fa235a8bbffd4a1fa8af0a0207c6369d7129","href":"https://static.observableusercontent.com/files/2de0c5733d3a0cb9ff3050fca8dfaba9c7408a0a1863536d04747fdbad357dca196ecfd7b47f60b8c97bd9bba261fa235a8bbffd4a1fa8af0a0207c6369d7129","name":"image@3.png","type":"image/png","lastModified":1703355666986.596,"size":121114}] 2d:{"title":"Purescript incremental relational lenses 1/n","cells":[{"id":0,"value":"$2f","pinned":false,"hidden":false,"mode":"md"},{"id":890,"value":"mdPlus`## TODO\n\n- [x] convert type level heterogeneous lists to row types\n- [x] decide on which purescript lens library to build on\n - [x] let's try purescript-profunctor-lenses (Data.Lens)\n - make sense to use it since it has the generics deriving.\n - [x] let's use the PR which implements VTA for nice syntax.\n- [x] let's try and start from the record types slide (at 15:20 on the video) [code](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Base.hs)\n - [x] fetch @\"A\" rec\n - [x] simple update\n - [ ] type changing update\n- [ ] decide on encoding of schema properties (fundeps <-> unique not null)\n`","pinned":true,"hidden":false,"mode":"ojs"},{"id":908,"value":"Ok, easy enough\n```haskell\nmodule Main where\n\nimport Data.Lens (view)\nimport Data.Lens.Record.VTA (prop)\n\n\nrec :: { \"A\" :: String, \"B\" :: String }\nrec = { \"A\": \"hello\", \"B\": \"world\" }\n\nval :: String\nval = view (prop @\"A\") rec\n```","pinned":false,"hidden":false,"mode":"md"},{"id":911,"value":"Oh it might be best to not depend on my local try purescript and just do VTA in fetch.\nYup that works, although it was not easy finding out which type annotation was needed (it was basically telling the type checker we didn't need a type changing lens)\n```haskell\nfetch \n :: forall @l a r r' \n . IsSymbol l\n => Cons l a r' r\n => Record r \n -> a\nfetch = view (prop (Proxy @l) :: Lens (Record r) (Record r) a a)\n```","pinned":false,"hidden":false,"mode":"md"},{"id":916,"value":"### Let's do update\n\nUh oh, just noticed that the 2nd update example adds a row field... Wonder how that's gonna go.\n\nIndeed, the basic `set` was easy enough:\n```haskell\nupdate :: forall @l a r r' . IsSymbol l => Cons l a r' r => a -> Record r -> Record r\nupdate = set (prop (Proxy @l) :: Lens (Record r) (Record r) a a)\n```\n\nBut adding a field doesn't work like that. Let's look at the purescript library as well as how that's implemented in Rudi's code.\n - Tried with a simple type changing setter but didn't work.\n - Indeed the [base case](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Base.hs#L155) for the `SetType` type family, adds the field if it wasn't found.\n\nThis is touching the edge of my skill level. But my intuition is that since the resulting type can be either the same as the input record type, or a new record type with the added field, we're going to need some type level shenanigans. \n","pinned":false,"hidden":false,"mode":"md"},{"id":924,"value":"mdPlus`\n- [ ] Maybe let's implement the one that changes types separately first.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":927,"value":"Ohh. I think I get it. `prop` actually is implemented in a way that requires the field to exist on both sides. Let's try and see if we can implement an alternative version `prop'`.\n\nOh well I can't do that, because the `view` part of the lens won't work. So let's try and move it inline of `update'`. I do wonder about the theoretical implications of this. Are they still well behaved if there's this asymmetry? To be fair, the slide doesn't say those are lenses... Ha ha, it's just `Record.get` and `Record.set` I think 🤦🏻‍♂️\n\nYes it's `Record.insert`\n\nOk so for now, let's continue and see what's next.","pinned":false,"hidden":false,"mode":"md"},{"id":934,"value":"### Record Projection\n\nI think that's also mostly in the purescript libraries and just a byproduct of Haskell lacking row types (and the temptation to write an extensible record implementation rather than picking an existing one).\n\nProjecting is \"just\" relying on the compiler Row type magic. So something like this\n```haskell\nproject :: forall @l a r' r1 r2. IsSymbol l => Cons l a r' r1 => Cons l a () r2 => Record r1 -> Record r2\nproject (r :: Record r1) = R.insert (Proxy @l) (R.get (Proxy @l) r) {}\n```","pinned":false,"hidden":false,"mode":"md"},{"id":938,"value":"### Record Set and Tables\n\n~~I think Record Set are like purescript RowLists.~~ No it's the type of a query result. So they could just be also a Data.Set of purescript records. There's a bit of term overloading here, since a purescript typelevel Row is not a database row, we're modeling database rows as records... 😬 And for some reason Rudi calls that Set a RecordSet... (ah yes I see the [hesitation here](https://github.com/rudihorn/haskell-lenses/blob/master/Lens/Record/Sorted.hs#L113))\n\n`project` on record set is `Set.map`\n```haskell\nprojects :: forall @l r r' a s. IsSymbol l => Cons l a r' r => Cons l a () s => Ord (Record r) => Ord (Record s) => RecordSet r -> RecordSet s\nprojects rs = S.map (project @l) rs\n```\n\nThere's some prior art here https://github.com/joneshf/purescript-typelevel-sql/blob/master/src/Type/SQL/Table.purs\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":945,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":952,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":960,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":969,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":979,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":85,"value":"## Source code","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"viewof editor = PurescriptEditor('', \n // [myDefaultTheme]\n)\n// codeJar\n// viewof editor = Editor({ \n// language: 'haskell',\n// debounced: 2000\n// })\n// viewof source = Inputs.textarea()","pinned":false,"hidden":false,"mode":"ojs"},{"id":801,"value":"Inputs.button([\n [\"Compile\", () => viewof editor.dispatchEvent(new Event('input', {bubbles: true}))],\n])","pinned":false,"hidden":false,"mode":"ojs"},{"id":647,"value":"||\n|---|---|---|\n|Compilation status|","pinned":false,"hidden":false,"mode":"md"},{"id":842,"value":"$${renderCompilationStatus(source, warnings, errors)}","pinned":false,"hidden":false,"mode":"html"},{"id":566,"value":"||\n| --- |--- | --- |\n|Displaying the `render` value as html|","pinned":false,"hidden":false,"mode":"md"},{"id":504,"value":"html = unsafe_html`${module.render}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":684,"value":"||\n| --- |--- | --- |\n|Output `val`|","pinned":false,"hidden":false,"mode":"md"},{"id":685,"value":"module.val","pinned":true,"hidden":false,"mode":"ojs"},{"id":686,"value":"||\n| --- |--- | --- |\n|Run the `greet` function|","pinned":false,"hidden":false,"mode":"md"},{"id":687,"value":"module.greet(\"me\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":690,"value":"||\n| --- |--- | --- |\n|Run `main`, a console effect, check your devtools.|","pinned":false,"hidden":false,"mode":"md"},{"id":689,"value":"module.main()","pinned":true,"hidden":false,"mode":"ojs"},{"id":155,"value":"mdPlus`## Todo\n\nReach out on the [purescript discord](https://purescript.org/chat) if you have ideas or want to help!\n\n- [x] Improve warnings and errors rendering\n- [x] Added a version of the original try purescript code sample.\n- [x] Make this notebook into a nicely importable notebook. https://observablehq.com/@jmatsushita/purescript\n- [x] Add syntax highlighting.\n- [ ] How would multiple modules work?\n- [ ] Display spinner while compiling`","pinned":false,"hidden":false,"mode":"ojs"},{"id":91,"value":"## Internals\n","pinned":false,"hidden":false,"mode":"md"},{"id":364,"value":"### Import the module\n","pinned":false,"hidden":false,"mode":"md"},{"id":618,"value":"ps = await Purescript(source)","pinned":true,"hidden":false,"mode":"ojs"},{"id":621,"value":"// module = ps.module\nmodule = ps.module","pinned":true,"hidden":false,"mode":"ojs"},{"id":623,"value":"errors = ps.errors","pinned":true,"hidden":false,"mode":"ojs"},{"id":635,"value":"warnings = ps.warnings","pinned":true,"hidden":false,"mode":"ojs"},{"id":274,"value":"renderCompilationStatus = (source, warnings, errors) => {\n const renderedWarnings = warnings && warnings.length >0 ? \nhtl.html`

Compiler warnings

\n ${toColumns(warnings).map(col => htl.html``)}\n \n \n ${toTable(warnings, source).map(row => htl.html`\n ${Object.values(row).map(cell => htl.html``)}\n `)}\n \n

${col}
${cell}

\n` : htl.html`No warnings
`\n\n const renderedErrors = errors ? htl.html`

Compiler errors

\n \n \n ${toColumns(errors).map(col => htl.html``)}\n \n \n \n ${toTable(errors, source).map(row => htl.html`\n ${Object.values(row).map(cell => htl.html``)}\n `)}\n \n

${col}
${cell}

\n` : htl.html`No errors`\n return htl.html`${renderedWarnings}${renderedErrors}`\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":571,"value":"import { Purescript } from '@jmatsushita/purescript'","pinned":true,"hidden":false,"mode":"ojs"},{"id":576,"value":"Purescript","pinned":true,"hidden":false,"mode":"ojs"},{"id":168,"value":"And there we have it. A compiled module that's accessible from the notebook 🎉","pinned":false,"hidden":false,"mode":"md"},{"id":366,"value":"### Use an embedded code editor with Haskell syntax highlighting","pinned":false,"hidden":false,"mode":"md"},{"id":192,"value":"Reuse https://observablehq.com/@cmudig/editor, an inline code editor with syntax highlighting","pinned":false,"hidden":false,"mode":"md"},{"id":176,"value":"// import {Editor} from '@jmatsushita/editor'\nimport {CodeMirror, myDefaultTheme} from '@jmatsushita/codemirror'","pinned":false,"hidden":false,"mode":"ojs"},{"id":824,"value":"PurescriptEditor = async (doc = '', extensions = []) => {\n const {StreamLanguage} = await import(`https://jspm.dev/@codemirror/language@6.9.3`)\n const { haskell } = await import(`https://jspm.dev/@codemirror/legacy-modes/mode/haskell`);\n return CodeMirror(doc, {\n extensions: [myDefaultTheme, StreamLanguage.define(haskell), ...extensions]\n });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":457,"value":"### Support persistence in local storage","pinned":false,"hidden":false,"mode":"md"},{"id":379,"value":"import {localStorageView} from '@tomlarkworthy/local-storage-view'","pinned":true,"hidden":false,"mode":"ojs"},{"id":462,"value":"If the local storage key doesn't exist, use this example as a default value.","pinned":false,"hidden":false,"mode":"md"},{"id":406,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":445,"value":"viewof source = localStorageView(\"part1\", { defaultValue: part1 })","pinned":true,"hidden":false,"mode":"ojs"},{"id":717,"value":"// creating a fake source to subscribe from, together with a bound localStorageView didn't work.\n// Uncaught TypeError: e.closest is not a function\n// at inputs.min.js:40:1444\n// viewof source = Inputs.input()\n// viewof sourceStorage = localStorageView(\"sourceStorage\")\n// Inputs.bind(viewof source, viewof sourceStorage)\n// See reproducer here https://observablehq.com/d/f7e7508476855e8b#cell-264","pinned":true,"hidden":false,"mode":"ojs"},{"id":726,"value":"Inputs.bind(viewof editor, viewof source)","pinned":true,"hidden":false,"mode":"ojs"},{"id":382,"value":"// Note you need to get these the right way round to have the page load work correctly\n// Inputs.bind(viewof source, viewof sourceStorage)","pinned":true,"hidden":false,"mode":"ojs"},{"id":438,"value":"// set helper\nfunction set(input, value) {\n input.value = value;\n input.dispatchEvent(new Event(\"input\", {bubbles: true}));\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":371,"value":"### Make warnings and errors readable\n\nUse a bunch of utility functions to render the warnings and errors in an html table (using the built-in `Inputs.table` didn't seem to allow multiline cells)","pinned":false,"hidden":false,"mode":"md"},{"id":195,"value":"toTable = (array, source) => array ? array.map(row => objectFilter(objectMap(row, toReadableRow(source)), hideColumns)) : []","pinned":true,"hidden":false,"mode":"ojs"},{"id":290,"value":"hideColumns = k => k !== \"allSpans\" && k !== \"moduleName\" && k !== \"filename\" && k !== \"errorLink\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":205,"value":"toColumns = array => array && array[0] ? Object.keys(array[0]).filter(hideColumns) : []","pinned":true,"hidden":false,"mode":"ojs"},{"id":237,"value":"toReadableRow = source => (v,k) => {\n switch(k) {\n case \"allSpans\": \n return JSON.stringify(v)\n\n case \"position\":\n // console.log(source.split(\"\\n\"))\n // console.log(source.split(\"\\n\").slice(v.startLine - 1, v.endLine))\n return source.split(\"\\n\").slice(v.startLine - 1, v.endLine).join(\"\\n\")\n \n default:\n return v;\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":221,"value":"toTable(warnings, source)","pinned":true,"hidden":false,"mode":"ojs"},{"id":210,"value":"toColumns(warnings)","pinned":true,"hidden":false,"mode":"ojs"},{"id":200,"value":"warnings","pinned":true,"hidden":false,"mode":"ojs"},{"id":263,"value":"errors.contents","pinned":true,"hidden":false,"mode":"ojs"},{"id":285,"value":"toColumns(errors.contents)","pinned":true,"hidden":false,"mode":"ojs"},{"id":287,"value":"toTable(errors.contents, source)","pinned":true,"hidden":false,"mode":"ojs"},{"id":231,"value":"objectMap = (obj, fn) =>\n Object.fromEntries(\n Object.entries(obj).map(\n ([k, v], i) => [k, fn(v, k, i)]\n )\n )\n ","pinned":false,"hidden":false,"mode":"ojs"},{"id":295,"value":"objectFilter = (obj, fn) => Object.fromEntries(Object.entries(obj).\n filter(([key, val]) => fn(key)));","pinned":false,"hidden":false,"mode":"ojs"},{"id":667,"value":"import {mdPlus} from \"@tmcw/bonus-markdown-flavor\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":521,"value":"function unsafe_html() {\n const span = document.createElement(\"span\");\n span.innerHTML = String.raw.apply(this, arguments);\n return span;\n}","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}