2b:null 2c:["$","title",null,{"children":"Opinions on a LISP API for LMDB | James Anderson | Observable"}] 31:T755,md ` what benefits and risks does a lisp binding for lmdb embody? the principal matter is how well the binding mediates the capabilities of the lmdb library. this is to be weighed against ways a binding might constrain or modulate those capabilities to bring access to them in line with best - or even just customary, practice, to mitigate dangers a foreign library introduces or to facilitate access patterns prevalent in a given application. the [lmdb library](http://www.lmdb.tech/doc/group__mdb.html) manages data through four entities - [environments](http://www.lmdb.tech/doc/group__internal.html#structMDB__env) afford access to the composite database - [transactions](http://www.lmdb.tech/doc/group__internal.html#structMDB__txn) effect acid database behaviour - [databases](http://www.lmdb.tech/doc/group__internal.html#structMDB__db) mediate operations on single (b-tree index) components - [cursors](http://www.lmdb.tech/doc/group__internal.html#structMDB__cursor) support database index traversal so long as references to each instance of these data structures observe the constraints on object dominance and containment which are implicit in the model and api, all instances may have indefinite extent. as the lmdb data model embodies just indirect containment rules (eg, an environment comprises databases, but the references are first-class), just dominance matters. that means, for example, so long as an environment is open, any of its databases can remain open and be passed to library operators, but once the environment has been closed, any database references become invalid. similar constraints apply to the relation between transactions and environments and between cursors and transactions. this permits an implementation pattern in which all entity references are independent of functional structure and may occur anywhere within the application. `32:T1118,md` this approach affords the most flexible access to lmdb capabilites, but it puts the application in a position where it must implement measure to minimize risks. - as read transactions with extended lifetimes can lead to undesirable storage consumption because the references to older revisions are not released until the transaction aborts, cursors must be managed in accord with transaction begin/end cycles. - entities which are subject to begin/end or open/close cycles must be managed. if references are lost, the entities will remain, which will lead to eventual memory exhaustion. if they are recycled too slowly - for example via termination, their extent of dependents on an exvironment can rapidly exceed library limitations. if the application is exceedingly simple, an unconstrained api would reduce time spent creating entities, but the benefit would be compromised by the complexity required to mitigate risk. in order to balance a potential performance benefit against the paranoia which that flexibility engenders, consider the costs of various operations: | entity | operation | time | |--------------------|-----------------|------| | transaction (clos) | make | 3µs | | transaction | begin/abort | 2µs | | cursor | make | 1µS | | cursor | make/open/close | 2µS | taken together, the complete transaction+cursor stack requires about 8µs. on one hand, this means that program structures which entail additional operations in order to constrain entity extent would add that to each composite get operation. on the other, where the typical composite operation is double-digit ms, it could be appropriate to trade-off the 8µs overhead in exchange for reduced risk. the the issues are the usual ones : scope and extent; what relations between the operators are to be permitted

entity	unconstrained			constrained
entity	instance extent	binding scope	binding extent	instance extent	binding scope	binding extent
environment	indefinite	indefinite	indefinite	indefinite	indefinite	indefinite	associated with a cataloged object. gc finalization
transaction	indefinite	indefinite	indefinite	indefinite	dynamic	dynamic	gc finalization
database	indefinite	indefinite	indefinite	object	lexical	indefinite re environment or dynamic	owned by clos environment instance; its slot value rebound lexically, but without closure; open either indefinite or dynamic
cursor	indefinite	indefinite	indefinite	dynamic	lexical	lexical	without closure

this still leaves risks - simultaneous db open : there is no way to control between processes and within one process there can be hundreds of threads which attempt to open a database. (caveat: "Opening a database can fail if another process is opening or closing it at exactly the same time.") - simultaneous environment open : there is no way to limit this between processes. (caveat: opening an environment can fail if another thread is opening it at the same time. at one instance over one hundred requests appeared for the same environment withing a few milliseconds. this led to simultaneous opens. open not the issue, but the step to clean up stale readers - of which there should have been none, triggered some sort of race condition in that operator, which led to y reported failure and a corrupt database - despite being read only.) - database extent : if databases are owned by environments, their extent needs to be that of the environment. this which means they need to be closed when its extent ends, otherwise a successive environment instance will attempt to open the db again. (caveat : "Do not have open an LMDB database twice in the same process at the same time.”) under these constraints, the application structure exhibits a restricted form `33:[{"id":"5b5915fdbf0aad29806cd9ce615e9e279de9e28a5d43036de8338e30193b5db63af0c85e88c27aec6bb1550ac80b4bd6b69992cb35fae46c96c902fe4c908312","href":"https://static.observableusercontent.com/files/5b5915fdbf0aad29806cd9ce615e9e279de9e28a5d43036de8338e30193b5db63af0c85e88c27aec6bb1550ac80b4bd6b69992cb35fae46c96c902fe4c908312","name":"unconstrained-lmdb.svg","type":"image/svg+xml","lastModified":1598286022437.482,"size":370583},{"id":"6236991a0bf3c41a294756dd318db1355e1a56b61a74f212a9b328409403297c99cf8d5a775f766e9194944758dc8a4e9ca0531d305966e968e40d47ef08e199","href":"https://static.observableusercontent.com/files/6236991a0bf3c41a294756dd318db1355e1a56b61a74f212a9b328409403297c99cf8d5a775f766e9194944758dc8a4e9ca0531d305966e968e40d47ef08e199","name":"constrained-lmdb@1.svg","type":"image/svg+xml","lastModified":1598346070214.649,"size":482471}] 2f:{"id":"661537a4ed74b03a","slug":"opinions-on-a-lisp-api-for-lmdb","title":"Opinions on a LISP API for LMDB","thumbnail":"f297703144cd10ec34ba32a575076a6cacdd9acf9dedf9b36dee619316c199c8","access_level":"public","create_time":"$D2020-08-24T03:13:02.344Z","update_time":"$D2020-08-25T09:05:45.341Z","view_time":null,"first_public_time":"$D2020-08-24T16:29:55.116Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"b452fd69977f5701","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":"starter_2024"},"version":65,"publish_version":65,"role":null,"authored":false,"authors":[{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null}],"members":[{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Opinions on a LISP API for LMDB","cells":[{"id":0,"value":"md`# Opinions on a LISP API for LMDB`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":2,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":9,"value":"\nFileAttachment(\"unconstrained-lmdb.svg\").image()\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":6,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":45,"value":"FileAttachment(\"constrained-lmdb@1.svg\").image()\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":59,"value":"md `this makes several constraints implicit in application patterns:\n- environments are catalogued, instantiated by name and their initialization protocol includes opening the respective lmdb entity through a serialized control path.\n- databases are contained in environments, which are specialized for respective database complement, are opened when the environment is instantiated, and are closed when it is closed.\n- environments are deaccessioned and closed actively when unused\n- transactions are managed with dynamic extent only. their bindings permit indefinite scope in order to simplify implementation and the operators include a lexical binding, but nothing closes indefinitely over it.\n- cursors are managed with dynamic extent only. they are manipulated through lexical bindings, but never through closures.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W33","stdlib":"1","titling":"auto"}} 30:"$2f:body"