31:T715,This is a prototype **Firebase wire compatible** Realtime Database Server. Vanilla Firebase web clients are able to connect and sync data with it. The server is written in [Observable](https://observablehq.com/) to simplify programmable customizability, you can one-click fork to host your own ([*really*](https://observablehq.com/@observablehq/fork-suggest-merge)). I think an end user programmable database server has much potential, for example, you could use ordinary code to authorize requests, you could even call out to external APIs, checking signatures *etc.* You could create synthetic data views to federated data sources *on-demand*, or fanout writes to additional storage engines. You can collocate the database near to users for incredibly low latency. This is just a prototype though, and **not ready for production use**. However, it is a step in the right direction. The code layout has been optimized for observability, readability and developer ergonomics. Through use of [webcode.run](https://webcode.run) live coding, live traffic can be execution in the developers browser. You can attach the Chrome DevTools debugger and REPL traffic live. Furthermore, through the use of [flowQueue](/@tomlarkworthy/flow-queue)s, tunnelled requests leave their live execution trace in the variables of the notebook, so you can see the processing steps without special tooling. See the accompanying [Youtube video](https://observablehq.com/@tomlarkworthy/firebase-server-prototype-1) about some of the innovative development features. 32:T8a2,incomingLongpollRequestAction = { const req = incomingLongpollRequest.req; const res = incomingLongpollRequest.res; const callbackId = req.query.cb; if (req.query.start) { // New session initialized const cid = Math.random().toString(16).substring(3); sessions[cid] = { id: cid, responseId: 0, sessionId: Math.random().toString(16).substring(3), password: Math.random().toString(16).substring(3), hanginingRequest: null, serverToClientQueue: [] }; res.header("content-type", "application/javascript"); res.send(newSessionResponse(callbackId, sessions[cid])); } else if (req.query.dframe) { res.header("content-type", "text/html"); res.send(disconnectFrame(sessions[req.query.id])); } else { const session = sessions[req.query.id]; if (!session) { return viewof incomingLongpollRequest.reject( new Error("Unrecognized connection id for " + req.query.id) ); } // If there is an existing long poll close the previous if (session.hangingRequest) { session.hangingRequest.send(`pRTLPCB(${session.responseId++},[])`); session.hangingRequest = null; } res.header("content-type", "application/javascript"); var commandIndex = 0; while (req.query[`d${commandIndex}`]) { const data = JSON.parse( atob(req.query[`d${commandIndex}`].replaceAll(".", "=")) ); commandIndex++; console.log("Incoming request", data); try { const response = await viewof incomingRequest.send({ session, request: data }); session.serverToClientQueue.push({ t: "d", d: { r: data.d.r, b: response } }); } catch (err) { console.error(err); session.serverToClientQueue.push({ t: "d", d: { s: "fail", d: err.message } }); } } if (session.serverToClientQueue.length > 0) { res.send( `pRTLPCB(${session.responseId++},${JSON.stringify([ session.serverToClientQueue.shift() ])});` ); } else { session.hangingRequest = res; } } viewof incomingLongpollRequest.respond(); }33:T43c,incomingPutAction = { const path = incomingPut.command.p; const data = incomingPut.command.d; try { await redis.sendCommand([ "SET", "firebase-server-prototype-1" + path, JSON.stringify(data) ]); // We also need to tell all interested parties const listeners = pathToListeners[path] || []; listeners.forEach((session) => { session.serverToClientQueue.push({ t: "d", d: { a: "d", b: { p: path, d: data } } }); // TODO, we should not be doing long poll specific stuff here if (session.hangingRequest) { const res = session.hangingRequest; session.hangingRequest = null; res.send( `pRTLPCB(${session.responseId++},${JSON.stringify([ session.serverToClientQueue.shift() ])});` ); } }); const response = { s: "ok", d: "" }; viewof incomingPut.respond(response); return response; } catch (err) { viewof incomingPut.reject(err); return err; } }34:[] 2f:{"title":"Hackable Firebase Realtime Database Server Prototype #1","cells":[{"id":0,"value":"

Hackable Firebase Realtime Database Server Prototype #1

\n${gfx}\n

Prototype #1

Focus areas: Firebase Client compatibility, Redis Persistence, GET/PUT/LISTEN

","pinned":false,"hidden":false,"mode":"html"},{"id":992,"value":"*Let's build a 3rd party wire compatible Firebase Database Server.*","pinned":false,"hidden":false,"mode":"md"},{"id":866,"value":"toc(\"h2,h3,h4,h5,h6\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":674,"value":"## Introduction","pinned":false,"hidden":false,"mode":"md"},{"id":378,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":475,"value":"### What this prototype does\n\n- Works with vanilla Firebase clients over public internet.\n- Key based Listen/Get/Put\n- Persist data to Redis\n- HTTP long poll transport\n\n### What this prototype doesn't do\n\n- JSON interpolation\n- OrderBy/limit\n- onDisconnect semantics\n- websocket transport\n- REST API\n- Redis security\n- horizontal scalability\n- long poll big payload segment assembly\n- query indexes\n- server values\n","pinned":false,"hidden":false,"mode":"md"},{"id":477,"value":"## Background\nIn a previous work we developed a tool for [reverse engineering the Firebase Realtime Database wire protocol](https://observablehq.com/@tomlarkworthy/rtdb-protocol), which included a very primitive Firebase Server Fake. Since then, we have created a [Redis driver](https://observablehq.com/@tomlarkworthy/redis) so notebooks can persist data to Redis directly, and so now we are in a position to put it all together. \n","pinned":false,"hidden":false,"mode":"md"},{"id":999,"value":"## Follow along\n[twitter](https://twitter.com/tomlarkworthy) or \n","pinned":false,"hidden":false,"mode":"md"},{"id":900,"value":"## Database URL","pinned":false,"hidden":false,"mode":"md"},{"id":566,"value":"With a specially formatted database URL we can send a vanilla Firebase client to a 3rd party server implementation.","pinned":false,"hidden":false,"mode":"md"},{"id":560,"value":"databaseURL = `https://webcode.run?ns=${thisNotebooksNamespace}|${thisNotebooksSlug};server`","pinned":false,"hidden":false,"mode":"ojs"},{"id":748,"value":"## Live Tests\n\nThe tests are meant to be read. They illustrate what features are working and furthermore, they are executed in **your** browser so you know it works because you have seen it work. Pop open DevTools and take a look when the clients are reset.\n\nIn this early stage the tests are a bit flakey and you might need to restart the clients, refresh the page, and try a few times until everything is warmed up 😳.","pinned":false,"hidden":false,"mode":"md"},{"id":100,"value":"clientConfig = ({\n firebaseServer: {\n url: databaseURL\n }\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":342,"value":"### Firebase Test Clients","pinned":false,"hidden":false,"mode":"md"},{"id":502,"value":"import {\n viewof restartClient,\n rtdb,\n rootA,\n rootB\n} with { clientConfig as config } from \"@tomlarkworthy/firebase-server-test-clients\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":882,"value":"### Test Results","pinned":false,"hidden":false,"mode":"md"},{"id":114,"value":"viewof suite = (server,\nredis,\nflowQueue,\ncreateSuite({\n name: null\n}))","pinned":false,"hidden":false,"mode":"ojs"},{"id":878,"value":"### Restart Test Clients & Tests","pinned":false,"hidden":false,"mode":"md"},{"id":578,"value":"viewof restartClient","pinned":false,"hidden":false,"mode":"ojs"},{"id":914,"value":"### Smoke Tests","pinned":false,"hidden":false,"mode":"md"},{"id":920,"value":"#### Read your write","pinned":false,"hidden":false,"mode":"md"},{"id":124,"value":"readYourWriteTest = suite.test(\"Read your own write - String\", async () => {\n const payload = randomString();\n const location = rtdb.child(rootA, \"collection/readyourownwrite\");\n await rtdb.set(location, payload);\n const response = (await rtdb.get(location)).val();\n expect(response).toEqual(payload);\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":919,"value":"#### Read other's write","pinned":false,"hidden":false,"mode":"md"},{"id":174,"value":"readTheirWriteTest = suite.test(\"Read their write - String\", async () => {\n const payload = randomString();\n const locationA = rtdb.child(rootA, \"collection/readtheirwrite\");\n const locationB = rtdb.child(rootB, \"collection/readtheirwrite\");\n await rtdb.set(locationA, payload);\n const response = await rtdb.get(locationB);\n expect(response.val()).toEqual(payload);\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":931,"value":"#### Listen notified of other's write","pinned":false,"hidden":false,"mode":"md"},{"id":535,"value":"listenNotifiedOfWrite = suite.test(\n \"onValue is notified of other's write\",\n async (done) => {\n const payload = randomString();\n const locationA = rtdb.child(rootA, \"collection/subscribetheirwrite\");\n const locationB = rtdb.child(rootB, \"collection/subscribetheirwrite\");\n rtdb.onValue(locationB, (snap) => {\n const val = snap.val();\n console.log(val);\n if (_.isEqual(val, payload)) {\n done();\n }\n });\n\n await rtdb.set(locationA, payload);\n }\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":510,"value":"## Custom Firebase Realtime Server","pinned":false,"hidden":false,"mode":"md"},{"id":385,"value":"### Storage Engine: Redis\n\nRedis the the storage engine for this prototype. We are using a custom [redis-web](https://observablehq.com/@tomlarkworthy/redis) client.","pinned":false,"hidden":false,"mode":"md"},{"id":6,"value":"redis = createClient({\n socket: redisConfig\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":572,"value":"redisConfig = ({\n host: \"redis.webcode.run\",\n port: 443,\n tls: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"import { createClient } from \"@tomlarkworthy/redis\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":941,"value":"### Long Poll Endpoint\n\nFirebase clients make long lived connection to this endpoint","pinned":false,"hidden":false,"mode":"md"},{"id":23,"value":"server = endpoint(\"server\", async (req, res) => {\n redis, flowQueue; // make sure these are setup first\n try {\n await viewof incomingLongpollRequest.send({ req, res }); // handle in to poll pipeline (unfold across Dataflow)\n } catch (err) {\n res.status(err.code || 500).send(err.message);\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":359,"value":"### Long Poll Request Pipeline\n\nIf you are live tunnelling the server endpoint, these cells will evaluate in response to production traffic (and will handle production traffic).","pinned":false,"hidden":false,"mode":"md"},{"id":365,"value":"viewof incomingLongpollRequest = flowQueue({\n timeout_ms: 10000\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":372,"value":"incomingLongpollRequest","pinned":true,"hidden":false,"mode":"ojs"},{"id":183,"value":"sessions = ({})","pinned":false,"hidden":false,"mode":"ojs"},{"id":452,"value":"newSessionResponse = (callbackId, session) => `\n function pLPCommand(c, a1, a2, a3, a4) {\n parent.window[\"pLPCommand${callbackId}\"] && parent.window[\"pLPCommand${callbackId}\"](c, a1, a2, a3, a4);\n }\n function pRTLPCB(pN, data) {\n parent.window[\"pRTLPCB${callbackId}\"] && parent.window[\"pRTLPCB${callbackId}\"](pN, data);\n }\n pLPCommand('start','${session.id}','${session.password}')\n pRTLPCB(${session.responseId++},${JSON.stringify([\n {\n t: \"c\",\n d: {\n t: \"h\",\n d: {\n ts: Date.now(),\n v: \"5\",\n h: \"webcode.run\",\n s: session.sessionId\n }\n }\n }\n])})`","pinned":false,"hidden":false,"mode":"ojs"},{"id":460,"value":"disconnectFrame = (session) => ``","pinned":false,"hidden":false,"mode":"ojs"},{"id":369,"value":"$32","pinned":true,"hidden":false,"mode":"ojs"},{"id":355,"value":"### Data Request Pipeline","pinned":false,"hidden":false,"mode":"md"},{"id":186,"value":"viewof incomingRequest = flowQueue({\n timeout_ms: 10000\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":219,"value":"incomingRequest","pinned":false,"hidden":false,"mode":"ojs"},{"id":224,"value":"requestRouter = {\n const session = incomingRequest.session;\n const request = incomingRequest.request;\n const commandHandler = {\n s: viewof incomingStats,\n p: viewof incomingPut,\n q: viewof incomingQuery,\n g: viewof incomingGet\n }[request.d.a];\n\n if (!commandHandler) {\n return viewof incomingRequest.reject(\n new Error(\"Unrecognised server action \" + request.d.a)\n );\n }\n\n commandHandler\n .send({\n session,\n command: request.d.b\n })\n .then((handlerResponse) => {\n viewof incomingRequest.respond(handlerResponse);\n })\n .catch(viewof incomingRequest.reject);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":310,"value":"#### STATS","pinned":false,"hidden":false,"mode":"md"},{"id":318,"value":"viewof incomingStats = flowQueue({\n timeout_ms: 5000\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":320,"value":"incomingStats","pinned":true,"hidden":false,"mode":"ojs"},{"id":322,"value":"incomingStatsAction = {\n incomingStats; // Trigger\n const response = { s: \"ok\", d: \"\" };\n viewof incomingStats.respond(response);\n return response;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":235,"value":"#### PUT","pinned":false,"hidden":false,"mode":"md"},{"id":230,"value":"viewof incomingPut = flowQueue({\n timeout_ms: 5000\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":232,"value":"incomingPut","pinned":true,"hidden":false,"mode":"ojs"},{"id":242,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":269,"value":"#### GET","pinned":false,"hidden":false,"mode":"md"},{"id":273,"value":"viewof incomingGet = flowQueue({\n timeout_ms: 5000\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":728,"value":"incomingGet","pinned":true,"hidden":false,"mode":"ojs"},{"id":280,"value":"incomingGetAction = {\n const path = incomingGet.command.p;\n const query = incomingGet.command.q;\n\n // TODO permissions check\n // { s: \"permission_denied\", d: \"Permission denied\" };\n try {\n const value = await redis.sendCommand([\n \"GET\",\n \"firebase-server-prototype-1\" + path\n ]);\n const response = { s: \"ok\", d: JSON.parse(value) };\n viewof incomingGet.respond(response);\n return response;\n } catch (err) {\n viewof incomingGet.reject(err);\n return err;\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":522,"value":"#### QUERY","pinned":false,"hidden":false,"mode":"md"},{"id":712,"value":"pathToListeners = ({})","pinned":false,"hidden":false,"mode":"ojs"},{"id":531,"value":"viewof incomingQuery = flowQueue({\n timeout_ms: 5000\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":529,"value":"incomingQuery","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"incomingQueryAction = {\n // Actions: Listen/Query\n const session = incomingQuery.session;\n const command = incomingQuery.command;\n\n try {\n const initial = await redis.sendCommand([\n \"GET\",\n \"firebase-server-prototype-1\" + command.p\n ]);\n\n // Initial data update sent immediately\n session.serverToClientQueue.push({\n t: \"d\",\n d: {\n a: \"d\",\n b: {\n p: command.p,\n d: JSON.parse(initial)\n }\n }\n });\n\n // Now register as a listener\n pathToListeners[command.p] ||= [];\n pathToListeners[command.p].push(session);\n\n // Follow up with a OK to the query\n const response = { s: \"ok\", d: \"\" };\n viewof incomingQuery.respond(response);\n return response;\n } catch (err) {\n viewof incomingQuery.reject(err);\n return err;\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":439,"value":"### Utils","pinned":false,"hidden":false,"mode":"md"},{"id":134,"value":"randomString = () => Math.random().toString(16).substring(3)","pinned":false,"hidden":false,"mode":"ojs"},{"id":546,"value":"thisNotebooksNamespace = {\n if (thisNotebooksSlug === HEALTH_CHECK_FALLBACK_SLUG)\n return HEALTH_CHECK_FALLBACK_NAMESPACE;\n else return /@([^/]*)\\//.exec(html``.href)[1];\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":551,"value":"thisNotebooksSlug = {\n const thisSlug = /\\/([^/]*)\\?/.exec(html``.href)[1];\n if (thisSlug === \"healthcheck\") return HEALTH_CHECK_FALLBACK_SLUG;\n else return thisSlug;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":824,"value":"### Remote notebook state\n\nWe expose an endpoint for reading the remote notebooks state, which can be useful for debugging remote errors.","pinned":false,"hidden":false,"mode":"md"},{"id":813,"value":"import { notebookSnapshot } from \"@tomlarkworthy/notebook-snapshot\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":811,"value":"trackingVariable_e3366d24de62 = true","pinned":false,"hidden":false,"mode":"ojs"},{"id":809,"value":"endpoint(\"variables\", async (req, res) => {\n res.json(\n (await notebookSnapshot(\"trackingVariable_e3366d24de62\")).map(\n (variable) => ({\n state: variable.state,\n name: variable.name,\n // Note these cells might contain personal information, so we only allow errors values to leave the environment\n ...(variable.state === \"rejected\" && {})\n })\n )\n );\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":843,"value":"### [Health check](https://observablehq.com/@endpointservices/healthcheck)\n\nThe healthcheck is actively monitored and includes the smoke tests.\n\n| Region | URL |\n| ----------- | ----------- |\n| us-central1 | https://webcode.run/regions/us-central1/observablehq.com/@endpointservices/healthcheck?target=%40tomlarkworthy%2Ffirebase-server-prototype-1&wait=50 |\n| europe-west4 | https://webcode.run/regions/europe-west4/observablehq.com/@endpointservices/healthcheck?target=%40tomlarkworthy%2Ffirebase-server-prototype-1&wait=50 |","pinned":false,"hidden":false,"mode":"md"},{"id":954,"value":"When running inside the healthcheck notebook, the slug is detected incorrectly as \"healthcheck\" which means the clients connect to the wrong place. So we have manual fix.","pinned":false,"hidden":false,"mode":"md"},{"id":949,"value":"HEALTH_CHECK_FALLBACK_SLUG = \"firebase-server-prototype-1\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":961,"value":"HEALTH_CHECK_FALLBACK_NAMESPACE = \"tomlarkworthy\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":447,"value":"### Notebook Error Monitoring, Analytics and Backups","pinned":false,"hidden":false,"mode":"md"},{"id":587,"value":"footer","pinned":false,"hidden":false,"mode":"ojs"},{"id":443,"value":"### Dependancies","pinned":false,"hidden":false,"mode":"md"},{"id":654,"value":"import { gfx } from \"@tomlarkworthy/hackable-realtime-database-title-graphic\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"import { endpoint } from \"@endpointservices/webcode\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":82,"value":"import { expect, createSuite } from \"@tomlarkworthy/testing\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":156,"value":"import { flowQueue } from \"@tomlarkworthy/flow-queue\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":212,"value":"import { footer } from \"@endpointservices/footer-with-backups\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":861,"value":"import { toc } from \"@bryangingechen/toc\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W34","stdlib":"1","titling":"auto"}