2e:null 2f:["$","title",null,{"children":"Firebase Realtime Database Wire Protocol Explorer | Tom Larkworthy | Observable"}] 34:T575,## But Why? Building a [causally consistent](https://en.wikipedia.org/wiki/Causal_consistency), latency compensated, local persistence backed sync protocol is *hard*. There have been a few different attempts since Firebase, for example, *[replicache](https://replicache.dev/)*, *[supabase](https://supabase.com/)*, *[appwrite](https://appwrite.io/)*, but they all miss various important features which makes them only partial replacements. Luckily, the major engineering is in the client, which has been MIT licensed, so let's leverage that to build a true, wire-compatible substitution. Also, since Firebase SDK version 9, the SDK has been modularized, so now we are at quite a nice position where we can depend on just the production Realtime Database SDK and swap its upstream server, without accidentally including any Google specific services. I prefer the RTDB over Firestore, it has a lower latency and faster startup time due to technology choice, and while it hasn't much query expressivity, it's a high performance replicated Key-Value store so it's an excellent primitive to build upon. As you will see in this notebook, the core Realtime Database sync protocol is pretty simple. I hope that by demonstrating how easy it is to substitute your own server that interested you will see an open source server is not a huge undertaking. Follow along at: ${tweet('1471850731032485896')} 35:Ta94,#### Initial handshake Our freshly initialized client (configured for longpolling) first makes a HTTP request to ``` https://s-euw1b-nss-202.europe-west1.firebasedatabase.app/.lp?start=t&ser=63492315&cb=1&v=5&p=1:1986724398:web:9b8bc33895b45dd2e095bf&ns=endpointservice-eu ``` Here is the meaning of each part of the URL ([source](https://github.com/firebase/firebase-js-sdk/blob/fb3e35965b23f88e318dd877fabade16cdcb6385/packages/database/src/realtime/Constants.ts#L18)) | URL | meaning | |-------------|-------| | s-euw1b-nss-202.europe-west1.firebasedatabase.app | Host server location (GCP region `europe-west1`) | | /.lp | longpoll endpoint | | cb=... | callback suffix | | ser=... | cache buster | | v=5 | VERSION_PARAM | | p | APPLICATION_ID_PARAM | | ns | namespace, the database name | The server responds by setting up two [JSONP](https://en.wikipedia.org/wiki/JSONP#:~:text=JSONP%2C%20or%20JSON%2DP%20(,by%20Bob%20Ippolito%20in%202005.) callbacks, one for stream control and one for data messages. JSONP is an old technique for cross-origin communication that works with very old browsers (remember that the longpolling transport is a fallback for when websockets are not detected) ```js function pLPCommand(c, a1, a2, a3, a4) { parent.window["pLPCommand1"] && parent.window["pLPCommand1"](c, a1, a2, a3, a4); } function pRTLPCB(pN, data) { parent.window["pRTLPCB1"] && parent.window["pRTLPCB1"](pN, data); } pLPCommand('start','5124373','WQLLoCT4we'); pRTLPCB(0,[{"t":"c","d":{"t":"h","d":{ "ts":1638559030693,"v":"5", "h":"s-euw1b-nss-202.europe-west1.firebasedatabase.app", "s":"CellW7myawI0dIl4X4u9szYUYDyg6AWx" }}}]); ``` The actual number in the literals, like pLPCommand1, is matched to the cb parameter. Thus a client is able to hold multiple longpoll channels open (e.g. for multiple apps) by assigning each a unique ID. The server sends two messages immediately. First, a command message to start the stream which sets an ID and password ( [source](https://github.com/firebase/firebase-js-sdk/blob/fb3e35965b23f88e318dd877fabade16cdcb6385/packages/database/src/realtime/BrowserPollConnection.ts#L171)). ``` pLPCommand('start', ID, PASSWORD) ``` Second, the [SERVER_HELLO](https://github.com/firebase/firebase-js-sdk/blob/fb3e35965b23f88e318dd877fabade16cdcb6385/packages/database/src/realtime/Connection.ts#L62) packet which includes a server provided timestamp that completes the [handshake](https://github.com/firebase/firebase-js-sdk/blob/fb3e35965b23f88e318dd877fabade16cdcb6385/packages/database/src/realtime/Connection.ts#L378). Once these messages are processed the logging considers the connection 'healthy' 36:T704,### Outbound Data Message Framing A long poll is a vanilla HTTP GET request, and is the only client -> server transport. As a database, the client will sometimes need to send up to 1MB of data, but this exceeds the capacity of a URL. So, an outbound request is chunked into "segments", which is what "seg0 ts0 and d0" encode ([source](https://github.com/firebase/firebase-js-sdk/blob/fb3e35965b23f88e318dd877fabade16cdcb6385/packages/database/src/realtime/BrowserPollConnection.ts#L625)). `d` is the data portion, which is URL safe base64 encoded. When the d0 parameter above is decoded it turns out to be JSON ``` {"t":"d","d":{"r":1,"a":"s","b":{"c":{"sdk.js.9-5-0":1}}}} ``` The first d object matches what is printed in the logs! The outer `"t":"d"` seems to denote "data" message (instead of a control message) and is not printed in the logs. The inner `"d":{"r":1,"a":"s","b":{"c":{"sdk.js.9-5-0":1}}}` is a wire protocol data massage. By looking at the source we can figure out meaningful names. The first set of objects are a common message envelope. | property | meaning | VALUE | meaning | |----------|-------------|--------|--------------| | r | REQUEST_SEQ | number | | | a | ACTION | s | REPORT_STATS | | b | BODY | JSON | depends on a | The r number is auto incremented by the client so messages can be decoded in order server side. The a is the ACTION, in this case, REPORT_STATS, which is encoding what type of event the client is sending. The BODY is any associated data with the ACTION, for reporting stats the client send its SDK version under a property 'c'. 37:T5d3,### Inbound Data Message Framing Most outbound commands returned data or need acknowledgement. The server will send back something like ~~~js pRTLPCB(2,[{"t":"d","d":{"r":1,"b":{"s":"ok","d":""}}}]); ~~~ This is a JSONP response. The first number '`2`' is the message sequence number. The client orders returned packets by this number ([source](https://github.com/firebase/firebase-js-sdk/blob/7354a0ed438f4e3df6577e4927e8c8f8f1fbbfda/packages/database/src/realtime/polling/PacketReceiver.ts#L20)) before processing. Note these features means the long poll transport is a bidi ordered transport like a websocket. The websocket does not need these features as its naive to the transport. We then have a message envelope | property | meaning | VALUE | meaning | |----------|-------------|--------|--------------| | t | TYPE | d | DATA | | | | t | CONTROL | | d | DATA | JSON | the message | And then the actual message | property | meaning | VALUE | meaning | |----------|-------------|--------|--------------| | r | RESPOSNE_NUM | number | | | b | BODY | JSON | depends what is being responded to | The message is similar to an outbound messages, except there is no action. The r is used to match a message to the outbound request, so the action is implied by the outbound request that triggered it. Often the body is of the form "s" for status (e.g. "OK"), "d" for data.38:T605,``` p:0: Listen called for /@tomlarkworthy/rtdb-protocol-explorer/rw default p:0: Listen on /@tomlarkworthy/rtdb-protocol-explorer/rw for default p:0: {"r":2,"a":"q","b":{"p":"/@tomlarkworthy/rtdb-protocol-explorer/rw","h":""}} p:0: handleServerMessage d {"p":"@tomlarkworthy/rtdb-protocol-explorer/rw","d":"hi"} event: /@tomlarkworthy/rtdb-protocol-explorer/rw:value:"hi" p:0: from server: {"r":2,"b":{"s":"ok","d":{}}} p:0: listen response {"s":"ok","d":{}} ``` client issues ``` https://s-euw1b-nss-209.europe-west1.firebasedatabase.app/.lp?id=9751&pw=BglKoMFz30&ser=60177916&ns=larkworthy-dfb11-default-rtdb&seg0=1&ts0=1&d0=eyJ0IjoiZCIsImQiOnsiciI6MiwiYSI6InEiLCJiIjp7InAiOiIvQHRvbWxhcmt3b3J0aHkvcnRkYi1wcm90b2NvbC1leHBsb3Jlci9ydyIsImgiOiIifX19 ``` the payload decodes to ``` {"t":"d","d":{"r":2,"a":"q","b":{"p":"/@tomlarkworthy/rtdb-protocol-explorer/rw","h":""}}} ``` Server respondes ``` pRTLPCB(4,[{"t":"d","d":{"b":{"p":"@tomlarkworthy/rtdb-protocol-explorer/rw","d":"hi"},"a":"d"}}]); ``` Client then sends ``` https://s-euw1b-nss-209.europe-west1.firebasedatabase.app/.lp?id=9751&pw=BglKoMFz30&ser=60177917&ns=larkworthy-dfb11-default-rtdb ``` Server responds ``` pRTLPCB(5,[{"t":"d","d":{"r":2,"b":{"s":"ok","d":{}}}}]); ``` Client follows up with ``` https://s-euw1b-nss-209.europe-west1.firebasedatabase.app/.lp?id=9751&pw=BglKoMFz30&ser=60177918&ns=larkworthy-dfb11-default-rtdb ``` So, after requesting a listen, the server sends all the data, and responds OK when the client has received the initial data set.39:Tffb,{ // State to carry between requests const connections = {}; return endpoint("longpoll", async (req, res) => { debugger; const callbackId = req.query.cb; if (req.query.start) { // New session initialized const cid = Math.random().toString(16).substring(3); connections[cid] = { responseId: 0, sessionId: Math.random().toString(16).substring(3), password: Math.random().toString(16).substring(3), active: null, forClient: [] }; res.header("content-type", "application/javascript"); res.send(` function pLPCommand(c, a1, a2, a3, a4) { parent.window["pLPCommand${callbackId}"] && parent.window["pLPCommand${callbackId}"](c, a1, a2, a3, a4); } function pRTLPCB(pN, data) { parent.window["pRTLPCB${callbackId}"] && parent.window["pRTLPCB${callbackId}"](pN, data); } pLPCommand('start','${cid}','${connections[cid].password}') pRTLPCB(${connections[cid].responseId++},${JSON.stringify([ { t: "c", d: { t: "h", d: { ts: Date.now(), v: "5", h: "webcode.run", s: connections[cid].sessionId } } } ])})`); } else if (req.query.dframe) { res.header("content-type", "text/html"); res.send(``); } else { const connection = connections[req.query.id]; // If there is an existing long poll close the previous if (connection.active) { connection.active.send(`pRTLPCB(${connection.responseId++},[])`); connection.active = null; } res.header("content-type", "application/javascript"); if (req.query.d0) { debugger; const data = JSON.parse(atob(req.query.d0.replaceAll(".", "="))); if (data.d.a === "g") { const path = data.d.b.p; let body; if (path === "/") body = { s: "permission_denied", d: "Permission denied" }; if (path === "/@tomlarkworthy/rtdb-protocol-explorer/rw") body = { s: "ok", d: "hi!" }; // Action: GET res.send( `pRTLPCB(${connection.responseId++},${JSON.stringify([ { t: "d", d: { r: data.d.r, b: body } } ])});` ); } else if (data.d.a === "q") { // Actions: Listen/Query // Initial data update sent immediately res.send( `pRTLPCB(${connection.responseId++},${JSON.stringify([ { t: "d", d: { a: "d", // data action b: { p: "@tomlarkworthy/rtdb-protocol-explorer/rw", d: "hi!" } } } ])});` ); // Follow up with a OK to the listen connection.forClient.push( `pRTLPCB(${connection.responseId++},${JSON.stringify([ { t: "d", d: { r: data.d.r, b: { s: "ok", d: "" } } } ])});` ); } else if (data.d.a === "s") { // Actions: Stats res.send( `pRTLPCB(${connection.responseId++},${JSON.stringify([ { t: "d", d: { r: data.d.r, b: { s: "ok", d: "" } } } ])});` ); } } else { if (connection.forClient.length > 0) { res.send(connection.forClient.shift()); } else { connection.active = res; } } } }); }3a:[{"id":"01c2b756326d0b35dee0b5a1fc0e5a8c56059c12f917f3bd2a48592856a0829b168b79a626e27c579fe247b86cb81bedb38f386b409e6bf44c21b8b9d52a48b7","href":"https://static.observableusercontent.com/files/01c2b756326d0b35dee0b5a1fc0e5a8c56059c12f917f3bd2a48592856a0829b168b79a626e27c579fe247b86cb81bedb38f386b409e6bf44c21b8b9d52a48b7","name":"image.png","type":"image/png","lastModified":1639486084051.928,"size":181283}] 32:{"id":"6a287b937f2ed9b1","slug":"rtdb-protocol","title":"Firebase Realtime Database Wire Protocol Explorer","thumbnail":"07130f4cc9f08f098d3d0495b89e2785f40d7aad85a5dbe6d1f527b212812fb2","access_level":"public","create_time":"$D2021-12-03T09:33:54.666Z","update_time":"$D2024-05-28T19:05:03.323Z","view_time":null,"first_public_time":"$D2021-12-17T14:24:52.090Z","has_unpublished_edits":null,"fork":false,"stars":18,"starred":false,"owner":{"type":"team","id":"7db5ed2b0697d645","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":"starter_2024"},"version":1622,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null}],"members":[{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null,"role":"editor"}],"comments":[{"id":"5bb26dec72bfee18","reply_to_id":null,"content":"auth code: VECWKoM_ix7cZSV2bO5q","create_time":"$D2021-12-07T22:07:56.368Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null}},{"id":"e92eda5f6d64014c","reply_to_id":"5bb26dec72bfee18","content":"auth code: 0kAc1iSQlEQxt4JfMSOG","create_time":"$D2021-12-12T11:34:03.984Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null}},{"id":"92345a7b6d257e03","reply_to_id":"5bb26dec72bfee18","content":"public auth code: ahiRnCtmY5Y1HTnXPMin","create_time":"$D2022-03-22T17:24:59.569Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null}}],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":3,"collection":{"id":"6950d6449373532b","slug":"firebase","title":"firebase","owner":{"id":"7db5ed2b0697d645","login":"tomlarkworthy","name":"Tom Larkworthy"}},"type":"notebook","body":{"title":"Firebase Realtime Database Wire Protocol Explorer","cells":[{"id":0,"value":"# [Firebase Realtime Database](https://firebase.google.com/docs/database) Wire Protocol Explorer\n\nGoogle kindly open sourced the Firebase Client SDKs, but there is no open source server implementation! This notebook is step 1 to remedying that omission by **documenting the wire protocol**.\n\nIn this *interactive* notebook, we instrument the [Realtime Database](https://firebase.google.com/docs/database) client, and create a number of repeatable in-browser experiments to understand the wire protocol. We then confirm our understanding by **bringing up our own Fake Realtime Database Server**, and connecting it to an unmodified SDK.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1578,"value":"🚀 *update 2022-04-29: a prototype Firebase compatible server can found [here](https://observablehq.com/@tomlarkworthy/firebase-server-prototype-1)* 🚀","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1598,"value":"🍾 This work was presented at [HYTRADBOI conference](https://www.hytradboi.com/2022/creating-a-wire-compatible-firebase-realtime-database-substitute)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1594,"value":"","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1482,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1027,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1050,"value":"### Related work\n\nIn an earlier work I wrote a serverless [IndieAuth Oauth 2.0 notebook](https://observablehq.com/@endpointservices/auth) capable to minting Custom Firebase Auth tokens. These projects will hopefully combine.\n\nI created [webcode.run](https://webcode.run) to make building serious infrastructure in public web notebooks possible.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1069,"value":"### About me\n\nI worked on the Firebase Realtime Database from 2014 to 2018. I am careful to only use information derived from public sources, including the realtime databases SDKs own logging, and the MIT licensed source code hosted on Github.\n\nI am part of the [SDIAlliance](https://sdialliance.org/) technical steering board with the mission of delivering a sustainable digital infrastructure for Europe. My working hypothesis is that trust in digital services has broken down due to lack of transparency and auditability, so now I am on a mission trying to build secure systems that operate in the open, *by design*, e.g. from public notebook source code.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1539,"value":"## Firebase Realtime Database Protocol Sources\n\n- Urish reversed engineered the wire protocol once and make a testing server https://github.com/urish/firebase-server\n- The [client source code is public](https://github.com/firebase/firebase-js-sdk/blob/master/packages/database/src/core/ServerActions.ts)\n- The SDK will emit protocol information when you call \"enableLogging\"\n- Your Browser developer tools will log the unencrypted network requests. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1564,"value":"tweet(\"1506214338503364613\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1060,"value":"## How to use this Notebook\n\nWhen you enable logging (below), there will be a readout of the Realtime database SDK's [logging](https://firebase.google.com/docs/reference/node/firebase.database#enablelogging) output. This provides a high level description of messages exchanged over the wire protocol. The notebook contains buttons that when pressed, will call SDK methods. The aim of the notebook is to understand how the client SDK communicates with the server.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1525,"value":"\nWhile the logging is extremely useful, to understand the low level details you need to open DevTools and inspect the messages in the *network* tab. By unticking \"websocket\" and refreshing the webpage, you can force the SDK to use a *longpolling* transport fallback, which uses a remote HTTP endpointpoint prefixed by `.lp/`\n\n${await FileAttachment(\"image.png\").image({width: Math.min(width, 620)})}\n\nIn the first half of the notebook we probe an **reference** Firebase SDK pointing to a normal Firebase project. \nIn the second half we initialize a **fake** Firebase server and configure a Firebase SDK to point to that instead. To my delight, no source modifications of the client are required to point it at a 3rd party server implementation.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1515,"value":"viewof showOverlay = Inputs.bind(\n Inputs.toggle({ label: \"show SDK logging?\" }),\n localStorageView(\"logger\")\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1198,"value":"## Common Code","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":14,"value":"app = import(\"https://www.gstatic.com/firebasejs/9.5.0/firebase-app.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":945,"value":"auth = import(\"https://www.gstatic.com/firebasejs/9.5.0/firebase-auth.js\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":29,"value":"rtdb = (toggleWebsockets,\nawait import(\"https://www.gstatic.com/firebasejs/9.5.0/firebase-database.js\"))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1178,"value":"## Reference Firebase Server\n\nThe reference Firebase is wired to a normal Firebase project.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":67,"value":"viewof initializeGate = buttonGate(\"ref initializeApp\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"referenceApp = (initializeGate,\nrtdb,\napp.initializeApp(\n {\n /* apiKey & authDomain required for authentication, but you can talk to an RTDB unathenticated without these */\n apiKey: \"AIzaSyBN4bxw6d0cM0CGPNzRrkRlBqwFQnPLdN4\",\n authDomain: \"larkworthy-dfb11.firebaseapp.com\",\n databaseURL:\n \"https://larkworthy-dfb11-default-rtdb.europe-west1.firebasedatabase.app\"\n // \"https://s-euw1b-nss-205.europe-west1.firebasedatabase.app?ns=larkworthy-dfb11-default-rtdb\"\n },\n \"reference\"\n))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1166,"value":"### CONNECT/REPORT_STATS","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":38,"value":"referenceDatabase = {\n rtdb.enableLogging((log) => {\n viewof rtdbLogs.value = viewof rtdbLogs.value + \"\\n\" + log;\n viewof rtdbLogs.dispatchEvent(new Event(\"input\", { bubbles: true }));\n });\n return rtdb.getDatabase(referenceApp);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":46,"value":"refRoot = rtdb.ref(referenceDatabase, \"/\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":63,"value":"When an app is initialized, and a reference to the database is made, three major things happen in sequence.\n\n1. A connection is negotiated\n2. A disconnect channel is setup.\n3. The SDK reports stats (which seems to be just be which SDK version is in use)\n\nThe logs show some of this, but you need to look at the network tab to really see what is happening.\n\n~~~\np:0: Browser went online. \np:0: Making a connection attempt \ngetToken() completed. Creating connection. \nc:0:0: Connection created \nc:0:0:0 Connecting via long-poll to https://s-euw1b-nss-200.europe-west1.firebasedatabase.app/.lp?start=t&ser=71903273&cb=1&v=5&ns=larkworthy-dfb11-default-rtdb \nc:0:0: Realtime connection established. \np:0: connection ready \np:0: reportStats {\"c\":{\"sdk.js.9-5-0\":1}} \np:0: {\"r\":1,\"a\":\"s\",\"b\":{\"c\":{\"sdk.js.9-5-0\":1}}} \nc:0:0: Primary connection is healthy. \np:0: from server: {\"r\":1,\"b\":{\"s\":\"ok\",\"d\":\"\"}} \n~~~\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1222,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1248,"value":"#### Page unload handler\n\nThe clients now communicates with the server using the provided ID and password, and requests a disconnect SCRIPT \n\n~~~\nhttps://s-euw1b-nss-200.europe-west1.firebasedatabase.app/.lp?dframe=t&id=228625&pw=JjiEOkOhrA&ns=larkworthy-dfb11-default-rtdb\n~~~\n\nTo which the server responses with HTML \n\n~~~\n\n~~~\n\nYou can see if that script is unloaded (i.e. page closed), the client will send a message to the server first.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1264,"value":"#### REPORT_STATS\n\nThe client send its SDK version to the server after connecting. This looks like an outbound requests\n\n~~~\nhttps://s-euw1b-nss-209.europe-west1.firebasedatabase.app/.lp?\n id=28078&pw=QoaXoLj7Nf&\n ser=50262031&\n ns=larkworthy-dfb11-default-rtdb&\n seg0=0&\n ts0=1&\n d0=eyJ0IjoiZCIsImQiOnsiciI6MSwiYSI6InMiLCJiIjp7ImMiOnsic2RrLmpzLjktNS0wIjoxfX19fQ..\n~~~\n\nThe server responds with \n~~~js\npRTLPCB(2,[{\"t\":\"d\",\"d\":{\"r\":1,\"b\":{\"s\":\"ok\",\"d\":\"\"}}}]);\n~~~\n\nTo understand the outbound request we need to look at the source code. We will now attempt to explain the message framing that is common to messages after the initial handshake.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1279,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1294,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":851,"value":"### GET\n\nGET, action, \"a\", code \"g\". The client sends a path \"p\", and optionally a query filter \"q\".\n\n```\np:0: {\"r\":6,\"a\":\"g\",\"b\":{\"p\":\"/@tomlarkworthy/rtdb-protocol-explorer/rw\",\"q\":{}}} \n```\n\nThe server responds with the status, \"s\", and if the GET is successful, the data, \"d\", as a JSON\n\n```\np:0: from server: {\"r\":6,\"b\":{\"s\":\"ok\",\"d\":\"hi\"}}\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":767,"value":"viewof refGetOk = buttonGate(\"ref.get('/...') ok\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":774,"value":"(refGetOk,\n(\n await rtdb.get(\n rtdb.child(refRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\")\n )\n).val())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":56,"value":"viewof refGetDenied = buttonGate(\"ref.get('/..') permission_denied\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1341,"value":"If the a GET cannot be executed it is returned as status \"permission_denied\"\n\n```\np:0: from server: {\"r\":7,\"b\":{\"s\":\"permission_denied\",\"d\":\"Permission denied\"}} \n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":759,"value":"(refGetDenied, rtdb.get(rtdb.child(refRoot, \"subpath\")))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":846,"value":"### PUT","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":831,"value":"PUT, action \"a\" code \"p\", places the data \"d\" at the path \"p\"\n```\np:0: {\"r\":7,\"a\":\"p\",\"b\":{\"p\":\"/@tomlarkworthy/rtdb-protocol-explorer/rw\",\"d\":\"hi\"}} \n```\nThe server response with the status \"s\" and no data \"d\"\n```\np:0: from server: {\"r\":7,\"b\":{\"s\":\"ok\",\"d\":\"\"}} \n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":790,"value":"viewof refSetSuccess = buttonGate(\"ref.set('/...') success\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":792,"value":"(refSetSuccess,\nrtdb.set(rtdb.child(refRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\"), \"hi\"))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":932,"value":"### MERGE","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":936,"value":"A MERGE, action \"a\" code \"m\", takes a root path \"p\", and mixes in data payload.\n\n```\np:0: {\"r\":2,\"a\":\"m\",\"b\":{\"p\":\"/@tomlarkworthy/rtdb-protocol-explorer\",\"d\":{\"rw\":\"hi\"}}}\n```\nThe server repsonds with the status and no data.\n```\np:0: from server: {\"r\":2,\"b\":{\"s\":\"ok\",\"d\":\"\"}} \n```\n\na:m > action: merge","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":928,"value":"viewof refPatchSuccess = buttonGate(\"ref.update('/...') success\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":930,"value":"(refPatchSuccess,\nrtdb.update(rtdb.child(refRoot, \"@tomlarkworthy/rtdb-protocol-explorer\"), {\n rw: \"hi\"\n}))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":856,"value":"### LISTEN/QUERY\n\nListen, action code \"l\" or query, action code \"q\" are the more complex because often multiple messages need to be send for a single. When a LISTEN is attached,the server first sends the data, and then responds ok against the original request.\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":761,"value":"viewof refOnValueSuccess = buttonGate(\"ref.onValue('...')\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":820,"value":"Generators.observe((notify) => {\n refOnValueSuccess,\n rtdb.onValue(\n rtdb.child(refRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\"),\n (snap) => {\n notify(snap.val());\n }\n );\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":835,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1382,"value":"### UNLISTEN\n\nUNLISTEN is action \"a\" code \"n\", which requires a path \"p\". The server will acknowledge.\n~~~\np:0: Unlisten on /@tomlarkworthy/rtdb-protocol-explorer/rw for default \np:0: {\"r\":4,\"a\":\"n\",\"b\":{\"p\":\"/@tomlarkworthy/rtdb-protocol-explorer/rw\"}} \np:0: from server: {\"r\":4,\"b\":{\"s\":\"ok\",\"d\":\"\"}}\n~~~","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1390,"value":"viewof refOff = buttonGate(\"ref.off('...')\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1385,"value":"(refOff,\nrtdb.off(rtdb.child(refRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\")))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1363,"value":"### DISCONNECT","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":115,"value":"Inputs.button(\"goOffline\", {\n reduce: () => rtdb.goOffline(referenceDatabase)\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":739,"value":"```\nInterrupting connection for reason: repo_interrupt \nc:0:0: Closing realtime connection. \nc:0:0: Shutting down all connections \nc:0:0:0 Longpoll is being closed. \np:0: data client disconnected \n0: onDisconnectEvents \n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":976,"value":"### AUTH\n\nAUTH, action \"a\" code \"auth\" sends a signed [JWT access_token](https://datatracker.ietf.org/doc/html/rfc7519) to the server for use with authentication rules.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":996,"value":"viewof refSignIn = buttonGate(\"ref.signInAnonymously\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":950,"value":"(refSignIn, auth.signInAnonymously(await auth.getAuth(referenceApp)))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":979,"value":"```\np:0: {\"r\":2,\"a\":\"auth\",\"b\":{\"cred\":\"eyJ...tAg\"}} \np:0: from server: {\"r\":1,\"b\":{\"s\":\"ok\",\"d\":\"\"}} \n```\n\nThe decoded token payload looks something like \n```\n{\n \"provider_id\": \"anonymous\",\n \"iss\": \"https://securetoken.google.com/larkworthy-dfb11\",\n \"aud\": \"larkworthy-dfb11\",\n \"auth_time\": 1639411696,\n \"user_id\": \"NIZX38015decs0zX2LMLXezgPQN2\",\n \"sub\": \"NIZX38015decs0zX2LMLXezgPQN2\",\n \"iat\": 1639411696,\n \"exp\": 1639415296,\n \"firebase\": {\n \"identities\": {},\n \"sign_in_provider\": \"anonymous\"\n }\n}\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":985,"value":"### UNAUTH\n\nUNAUTH is action \"a\" code \"unath\", no data is sent","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":988,"value":"```\np:0: {\"r\":3,\"a\":\"unauth\",\"b\":{}} \np:0: from server: {\"r\":3,\"b\":{\"s\":\"ok\",\"d\":\"unauthenticated\"}} \n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1002,"value":"viewof refSignOut = buttonGate(\"ref.signOut\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":962,"value":"(refSignOut, auth.signOut(await auth.getAuth(referenceApp)))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":104,"value":"## Fake Realtime Database Server\n\nIn this section we create a server endpoint capable of replying to an unmodified client.\n\nNote For webcode live coding to work (which is extremely helpful as you can breakpoint the server), you need websockets *turned on* in the log overlay.\n\nNote Only the notebook owner can do live coding, but this notebook is forkable. After forking and logging into the server, you should should be able enable live coding and set DevTools debugging breakpoints with \"debugger\" in the server source code. You may need to refresh the page after the URL changes.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":242,"value":"$39","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1407,"value":"### Configuring a vanilla SDK to a 3rd party server","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":132,"value":"There are a few ways the SDK decodes the Database URL. You can include an \"ns\" parameter for instance ([source](https://github.com/firebase/firebase-js-sdk/blob/f5a17143b1930b7b7b45d431ad21f50b236e4f80/packages/database/src/core/util/libs/parser.ts#L176)).\n\nWe use the ns parameter to route the inbound long poll connections to a webcode endpoint hosted on [Observable](https://observablehq.com) (handler [source](https://github.com/endpointservices/webcode.run/blob/ec897913df5b3507910a329f91fce1cb042e2e6d/rtdb.mjs#L3))\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1423,"value":"THIS_NOTEBOOK = /(@|d)[^/]*\\/[^/#?]*/.exec(html``.href)[0]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1446,"value":"FAKE_SERVER_URL = `https://webcode.run?ns=${THIS_NOTEBOOK.replace(\n \"@\",\n \"\"\n).replace(\"/\", \"|\")};longpoll`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":109,"value":"viewof initializeFakeGate = buttonGate(\"initialize Fake Firebase Client\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":107,"value":"fakeApp = (initializeFakeGate,\nconsole.log(\"initialize fake app\"),\napp.initializeApp(\n {\n databaseURL: FAKE_SERVER_URL\n },\n Math.random().toString(16) // We randomize the name so we can rerun initialization\n))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1170,"value":"### CONNECT/REPORT_STATS","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":118,"value":"fakeDatabase = {\n rtdb.enableLogging((log) => {\n viewof rtdbLogs.value = viewof rtdbLogs.value + \"\\n\" + log;\n viewof rtdbLogs.dispatchEvent(new Event(\"input\", { bubbles: true }));\n });\n return rtdb.getDatabase(fakeApp);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":482,"value":"fakeRoot = rtdb.ref(fakeDatabase, \"/\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":250,"value":"Inputs.button(\"goOffline()\", {\n reduce: () => rtdb.goOffline(fakeDatabase)\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":877,"value":"### GET","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":295,"value":"viewof fakeGetDenied = buttonGate(\"get('/') permission_denied\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":751,"value":"(fakeGetDenied, rtdb.get(fakeRoot))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":870,"value":"viewof fakeGetOk = buttonGate(\"get('/') ok\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":866,"value":"(fakeGetOk,\n(\n await rtdb.get(\n rtdb.child(fakeRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\")\n )\n).val())","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":900,"value":"### LISTEN","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":898,"value":"viewof fakeOnValueSuccess = buttonGate(\"ref.onValue('...')\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":905,"value":"Generators.observe((notify) => {\n fakeOnValueSuccess,\n rtdb.onValue(\n rtdb.child(fakeRoot, \"@tomlarkworthy/rtdb-protocol-explorer/rw\"),\n (snap) => {\n notify(snap.val());\n }\n );\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":729,"value":"## UI Functions","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":353,"value":"toggleWebsockets = {\n if (!useWebsocket) window.WebSocket = undefined;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"function buttonGate(label) {\n const button = htl.html`

`;\n return button;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":179,"value":"viewof rtdbLogs = Inputs.input([])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":664,"value":"viewof useWebsocket = Inputs.bind(\n Inputs.toggle({ label: \"websocket?\" }),\n localStorageView(\"websocket\", {\n json: true\n })\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":813,"value":"#### Overlay implementation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":595,"value":"overlay = {\n if (!showOverlay) return;\n const sidebar = view`

\n ${Inputs.bind(\n Inputs.textarea({\n width: \"100%\",\n rows: 10,\n disabled: true\n }),\n viewof rtdbLogs\n )}\n

\n ${Inputs.button(\"clear logs\", {\n reduce: () => (\n (viewof rtdbLogs.value = []),\n viewof rtdbLogs.dispatchEvent(new Event(\"input\", { bubbles: true }))\n )\n })}\n ${[\"useWebsocket\", viewof useWebsocket]}\n \n

`;\n const container = html`