2b:null 2c:["$","title",null,{"children":"Teaching an Old Store New Triples: Implementing RDF 1.2 / SPARQL 1.2 in Dydra SPOCQ | datagenous | Observable"}] 31:T3205,# Teaching an Old Store New Triples: Implementing RDF 1.2 / SPARQL 1.2 in Dydra SPOCQ *A development retrospective — 2026-07-26* RDF 1.2 and SPARQL 1.2 add a small number of features with an outsized reach into a triple store: **triple terms** (`<<( s p o )>>`, an unasserted value that *is* a triple), **reified triples** (`<< s p o ~r >>`, a triple given a named occurrence), **annotation syntax** (`S P O {| … |}`), **base-direction language tags** (`"hello"@en--ltr`), and the `VERSION` directive. None of them is large on paper and yet they touch the parser, the dictionary, the index, the query compiler, the evaluator, and every serializer. This is the story of bringing them into SPOCQ — a SPARQL 1.0/1.1 engine over LMDB — from a clean 1.1 baseline to a stable state against the W3C 1.2 test suite, and of decisions along the way. ## The starting point SPOCQ stores RDF in a content-addressed dictionary and a set of LMDB quad indices. Every term — IRI, blank node, literal — is interned to a term-number, and the quad indices, the BGP compiler, and the revision visibility filters all speak term-numbers, not values. A SPARQL request arrives at the operator `graph-store-request`, flows through `pipe-query` into task decoding and parsing, gets compiled into native code through an algebra of macros, runs in a worker thread, and is serialized as a response according to the negotiated media type. The 1.1 engine managed all terms defined by the recommendations as first class objects, but had no notion of a term that is itself a triple. The first and most important design move was to recognize that there are two representation realms: the *value realm* (what the parser builds and the evaluator consumes — native atomic values and structures whose slots hold other structures) and the *term-number realm* (what the dictionary produces, the indices comprise, and the algebra reduces — integers). Triple terms exist in both realms. We introduced one struct, `triple-term`, with complementary slots for child *values* and child *term-numbers*, either pair resolvable lazily from the other. That struct is the bridge, and many decisions in the project turned out to be a question of how a value crosses the bridge. ## Storage: a triple is a string is a number A triple term is stored like any other dictionary entry — same record layout as an IRI, a term-number from the same monotonic counter — but its content-addressed key is the canonical fixed-width hex encoding of its three children, tagged so it cannot collide with a literal of the same lexical form: ``` "" → subject · predicate · object ``` Because the dictionary key is just a string, the existing content-addressing machinery deduplicates it for free: the SHA-1 of the `urn:triple:`-tagged three-child key maps a value to exactly one term-number, its *tt-id*, on every code path. The three children alone determine the term's *identity*: a triple term **identifies with** its `(s p o)`, and the SHA-1 dictionary key is computed over exactly those three constituents. The `` addition keeps that hash space disjoint from any literal whose lexical form happens to be the same hex string. Any context in which the term appears does *not* contribute to this key. The abstract terms are independent of repository content. This independence together with the desire to limit string decoding while operating on triple term patterns argued to add a permuted triple index to the global dictionary's SHA-1 index. Any context introduced by reification references in the repository appears in reified statements as the context of quad permutation index keys, but it is not part of the content-addressed triple term identity. In a repository, a reified triple is represented by two artefacts: one abstract value and one asserted statement. The abstract value is a `:triple-term` record, reified as `<< s p o ~r >>`, identified by its three children and carrying `r` in its context slot. The concrete statement is an asserted `(r rdf:reifies tt-id)` triple in the respective quad graph. The reifier is captured by that assertion, available as the context value of the triple index, but not folded into the term-number. Getting that distinction right (a "triple reified by r" is not a "triple in named graph r") informs the storage model. An abstract `<<( s p o )>>` is also present in the repository triple index with a record context slot which holds the *default-context marker* — the four-byte representation of −1, `FFFFFFFF`. For a , the reference holds the reifier's term-number. Because pattern matching over the children needs more than equality lookup, the *per-repository* triple term index also gets three rotational permutation indices (`tt:s.p.o`, `tt:p.o.s`, `tt:o.s.p`), each keyed by the children with the reifier tail. Those indices are *local*; the dictionary record is *global*. That asymmetry can matter. The question of equality concerns also the nature of constituent equality. It is not just identity. `<<( :a :b 123 )>>` and `<<( :a :b 123.0 )>>` must be equal under `=` (value comparison) but distinct under `sameTerm` (identity). Since `123` and `123.0` are value-equal but have different term-numbers, the fast path that decides equality by integer-comparing term-numbers had to be bypassed for triple terms and for value-equal-but-distinct numerics. `=` recurses component-wise by value; `sameTerm` may use the id. ## The TRIPLE() saga, or: when to intern The most instructive arc concerned a single function. The principle is clean: `TRIPLE(s, p, o)` should construct a value, and only compilation or import should ever touch the dictionary. The code didn't match — `TRIPLE()` eagerly interned a record. So we made it value-only. It broke immediately. A `BIND(TRIPLE(...) AS ?triple)` came back with `?triple` unbound. The reason was subtle: a SELECT solution column stores a term-*number*, and the natural place to defer the crossing — the projection collector — runs in a worker thread whose dynamic context can't intern, so the deferred crossing signalled and a `handler-case` quietly dropped the binding. We tried a second variant (value-only plus a record-only lazy crossing); it failed identically, for the same reason. Then the user posed the question that settled it: in one query, `{ :s :p <<( :a :b :c )>> }` and `{ ?r rdf:reifies <<( :a :b :c )>> }` must resolve the *same* `<<( :a :b :c )>>` to the *same* term-number, or the two index lookups can't join. They do — because the term-number is content-addressed. An ephemeral, transaction-local id is by construction *not* content-addressed, so two occurrences interned on different paths would silently fail to coincide. That reframed everything. The record-only intern isn't a stopgap; it is the correct identity mechanism. A triple term that can participate in matching, joining, `=`/`sameTerm`, or projection *must* resolve to the content-addressed persistent id. So `TRIPLE()` establishes that id at evaluation time (record only — no permutation indices, not an assertion), and projection just reads it. The "value-only" ideal survives only as a narrow, low-value optimisation for triple terms provably never materialised — and even that would have to mint the same content-addressed id, never an ephemeral one. Two failed attempts to do the "obvious" thing taught us why the existing behaviour was right. ## Decoupling the record from the index The same investigation exposed a real gap. The global record is content-addressed and interned once; the permutation indices are per-repository. So when a term is asserted, its record may already exist — from an earlier value crossing, or from assertion into a different repository — while *this* repository's index entries do not. An assertion path that skipped indexing because "the record already exists" would leave an unindexed, unmatchable triple. The fix is to decouple the two degrees of interning: the assertion path writes the index entries unconditionally and idempotently, never gating on record novelty and never short-circuiting on a cached id. ## The annotated-repository reckoning The last twist came from a test that *crashed* rather than failed. A triple-term pattern against a hand-picked repository signalled "an error, because the triple term permutation indices, and the scan method that reads them, exist only on an annotated repository, and the default repository class is not annotated. The test harness had been quietly creating annotated repositories all along; an ordinary one had never been exercised. A crash is a gift: it forces a decision. We settled on a clean asymmetry, modelled on how an absent constant behaves: - **Reading** a triple-term pattern against a non-annotated repository yields no result — silently, never an error, exactly as a string constant that isn't in the store contributes no solution. - **Storing** a triple term into a non-annotated repository fails loudly, with a distinct `triple-term-storage-error` (a request error, rendered as HTTP 400) that names the repository — because the store genuinely cannot hold the term, and silently mis-storing it would be worse than refusing. Reads are forgiving; writes are honest. Whether the *default* repository class should simply become annotated — so triple terms work everywhere without ceremony, at the cost of three extra databases per repository — is a deployment policy we deliberately left open. ## Proving it: the W3C suite and EARL To know where we actually stood, we ran the W3C `sparql12` test collection test-by-test, each against a freshly provisioned repository, and recorded the outcome as an EARL report. The current standing: > **241 passed / 11 failed / 18 not-comparable** across 270 tests in ten > manifests. The 18 "not-comparable" are harness gaps, not engine defects — SPARQL-XML result comparison, graph isomorphism for `CONSTRUCT`, update store-state — reported honestly as `earl:cantTell`. The 11 failures are all negative-syntax or static-semantic checks which the parser does not enforce (out-of-range codepoint escapes, nested aggregates, group-by scope, duplicated `VALUES` variables) — plus the long-string `VERSION` distinction, which we intentionally do not enforce: a string is a string, and that lexical distinction is arbitrary for a version label. Recording these as deliberate non-goals, rather than letting them lurk as mysterious red, was itself a decision. Running the suite also drove a documentation discipline: as the implementation clarified the spec, the requirements document was revised in lockstep (it now carries a revision history with several authors, including more than one AI model — this was a human-plus-AI engineering effort, and the reports say so). The requirements and the implementation report are the durable record; this post is the story they don't tell. ## Where it stands, and what's left The implementation is complete across all five topics — abstract representation, parsing, encoding, storage, and query processing. Triple terms parse, intern (content-addressed, with decoupled indexing), match (multi-index BGP with the rewrite phases), compare by value and by identity, order, and serialize across JSON, XML, CSV, TSV, and the Turtle-family graph formats. Reified triples, annotation syntax, base-direction literals, and `VERSION` all work. The open issues fall into honest buckets: - **Deferred by upstream:** TriX and JSON-LD triple-term encoders (JSON-LD awaits 1.3); C++-side recognition of the `:triple-term` tag (the Lisp side is self-contained via the `T:`-prefixed SHA-1 and the canonical key). - **Policy:** whether the default repository class becomes annotated. - **Intentional non-goals (for now):** the negative-syntax and static-semantic rejections that account for every one of the 11 W3C failures. ## What the project taught us Three things stand out. First, content-addressing is identity. This insight informs how the storage model establishes triple term identity. Second, the value realm and the term-number realm need a real boundary and most of the hard questions concerned where and how a term crosses it — in a worker thread without an intern context, during projection, or at assertion. Third, the tests earned their keep not just by passing, but also by failing in instructive ways: a crash on a plain repository motivated a cleaner design than any amount of up-front planning would have. Also, not to be neglected, the discipline of recording each decision — and each deliberate non-decision — in the requirements and implementation reports as it was made let a feature this cross-cutting reach a stable, explainable state. 32:[] 2f:{"id":"ceb09bb1772f7d60","slug":"teaching-an-old-store-new-triples","title":"Teaching an Old Store New Triples: Implementing RDF 1.2 / SPARQL 1.2 in Dydra SPOCQ","thumbnail":"a9ec6ad22c448efd87a93496377beebf1d5c8614f6e3c903fbd600d5f3fe4aa7","access_level":"public","create_time":"$D2026-07-26T22:18:59.113Z","update_time":"$D2026-07-26T22:19:30.644Z","view_time":null,"first_public_time":"$D2026-07-26T22:20:20.510Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"ad42203b295b2120","login":"datagenous","name":"datagenous","avatar_url":"https://avatars-next.observableusercontent.com/c8136cc7228bf2ad689462f6a332e51dc9984c845619ac0f34ca730875bf0c7a","tier":"pro"},"version":5,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null}],"members":[{"type":"team","id":"ad42203b295b2120","login":"datagenous","name":"datagenous","avatar_url":"https://avatars-next.observableusercontent.com/c8136cc7228bf2ad689462f6a332e51dc9984c845619ac0f34ca730875bf0c7a","tier":"pro","role":"editor"},{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Teaching an Old Store New Triples: Implementing RDF 1.2 / SPARQL 1.2 in Dydra SPOCQ","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"}],"files":"$W32","stdlib":"1","titling":"auto"}} 30:"$2f:body"