29:null 2a:["$","title",null,{"children":"Proxying binary data through a Google Apps Script web app | Martin Stålberg | Observable"}] 2f:T917,# Proxying binary data through a Google Apps Script web app ## _or_ a special case of string serialization of binary data In a web app with maps for an event in my sports club [KFUM Uppsala Survival IF](https://www.uppsalasurvival.se/) I wanted to present map data from a [Google My Maps](https://www.google.com/maps/about/mymaps/) map, easily editable by everyone in the club. I found out how to download the map data as kml/kmz, but wanted the web app to get the latest data directly, and unfortunately the URLs I found were served with forbidding [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) policies. I later [found out](https://support.google.com/maps/thread/159694892?hl=en) that I could remove the `/u/1` part of the URLs to avoid the problem. But before that I built a proxy [Google Apps Script web app](https://developers.google.com/apps-script/guides/web) (_GASWA_ in this text) to avoid the CORS restrictions. A GASWA accepts HTTP GET and HTTP POST requests and can respond with either text or HTML. [kml](https://developers.google.com/kml/) is a text file format, so that's easy enough to proxy, just pass it as text. [kmz](https://developers.google.com/kml/documentation/kmzarchives) files are zip archives containing a kml file and supplementary files, e.g. image files for map marker icons. So we need to somehow serialize a zip archive into either HTML or text. Let's go with text. My understanding of [Google Apps Script](https://www.google.com/script/start/) is that it is essentially Javascript, running in an environment with some special builtins. The GASWA fetches the kmz file with [URLFetchApp.fetch(url)](https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app) and receives the result in an [HTTPResponse](https://developers.google.com/apps-script/reference/url-fetch/http-response). A straightforward way to serialize the binary data is simply [Utilities.base64Encode](https://developers.google.com/apps-script/reference/utilities/utilities?hl=en#base64Encode(Byte)(response.[getContent()](https://developers.google.com/apps-script/reference/url-fetch/http-response?hl=en#getContent()). The base64 string can then be passed as text, and the receiver can then use JSZip to unpack the file with something like30:T566,## Serialization in Google Apps Script One might think that we could just do HTTPResponse.getContentText() and be done with it, but that returns a very short string that does not encode all of the data (at least with my data). Instead, we can access the kmz binary data with [HTTPResponse.getContent()](https://developers.google.com/apps-script/reference/url-fetch/http-response?hl=en), which returns it as `byte[]` (an array of bytes), supposedly. The result is displayed in the apps script log as

${bytesJsonStringified}

Note the negative values. I think the byte values are interpreted as numbers in [two's-complement](https://en.wikipedia.org/wiki/Two%27s_complement) representation. My first idea was to convert this "byte array" to a string with a one-to-one mapping between elements in the array and characters in the string. For this we should be able to just do [String.fromCharCode(...charCodes)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/fromCharCode), but it turns out that in Google Apps Script, with my data, this function stops early and returns a very short string that doesn't encode all of the data.31:[{"id":"fa181256aa6ab5de8297b9796ae291065f647c05b08032298fabb31fe51117636331c1ee1d4ea859e679c254d8696de1f23833133bcadf638fd59356178ee6bc","href":"https://static.observableusercontent.com/files/fa181256aa6ab5de8297b9796ae291065f647c05b08032298fabb31fe51117636331c1ee1d4ea859e679c254d8696de1f23833133bcadf638fd59356178ee6bc","name":"kmzBase64.txt","type":"text/plain","lastModified":1649872552003.352,"size":8148},{"id":"262d6f72de23884abdd0706323ac1bd51a66b6e50c316f8673334ac1ee49a1c06ae8d91d7fd17c534d0644ae702f393bd63a4a2e7912cec0751ab87db33e65d0","href":"https://static.observableusercontent.com/files/262d6f72de23884abdd0706323ac1bd51a66b6e50c316f8673334ac1ee49a1c06ae8d91d7fd17c534d0644ae702f393bd63a4a2e7912cec0751ab87db33e65d0","name":"kmz bytes google apps script json stringified.txt","type":"text/plain","lastModified":1649871965379.414,"size":21814}] 2d:{"id":"d4e4e9221ebb7d63","slug":"proxying-binary-data-through-a-google-apps-script-web-app","title":"Proxying binary data through a Google Apps Script web app","thumbnail":"8cdd305f1c8742ca52adcfc935883d6e6950b49a8363465317e9856eddcdea2b","access_level":"public","create_time":"$D2022-04-13T15:38:09.932Z","update_time":"$D2022-04-13T18:37:42.752Z","view_time":null,"first_public_time":"$D2022-04-13T18:36:47.245Z","has_unpublished_edits":null,"fork":true,"stars":1,"starred":false,"owner":{"type":"team","id":"846cb38877f13a5e","login":"mast4461","name":"Martin Stålberg","avatar_url":"https://avatars-next.observableusercontent.com/60cf8849f11993dfaf06d6e0634bbcecffce9e5a71f2bdcaa587389505b28994","tier":"starter_2024"},"version":1409,"publish_version":1409,"role":null,"authored":false,"authors":[{"type":"individual","id":"44de236d09dec6e7","login":"mast4461","name":"Martin Stålberg","avatar_url":"https://avatars-next.observableusercontent.com/60cf8849f11993dfaf06d6e0634bbcecffce9e5a71f2bdcaa587389505b28994","tier":null}],"members":[{"type":"individual","id":"44de236d09dec6e7","login":"mast4461","name":"Martin Stålberg","avatar_url":"https://avatars-next.observableusercontent.com/60cf8849f11993dfaf06d6e0634bbcecffce9e5a71f2bdcaa587389505b28994","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"Proxying binary data through a Google Apps Script web app","cells":[{"id":0,"value":"$2f","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1149,"value":"JSZip.loadAsync(kmzBase64, { base64: true })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1133,"value":"The base64 approach works well, and is very simple, but because I enjoy optimization and problem solving I wanted to see if I can come up with some way to serialize the binary data more compactly.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1051,"value":"$30","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1230,"value":"My first guess was that maybe the negative numbers are the problem, so maybe we could get around this by adding an offset to the values before String.fromCharCode(), like this:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":847,"value":"viewof charCodeOffset = Inputs.range([0, 500], {\n label: \"Charcode offset\",\n value: 289,\n width: \"100%\",\n step: 1\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1241,"value":"bytesEncodedWithOffset = String.fromCharCode(...bytes.map((d) => d + charCodeOffset))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1385,"value":"{\n const node =\n this || html`

`; // This speeds up re-rendering enormously\n node.textContent = bytesEncodedWithOffset;\n return node;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1283,"value":"I tried an offset of 289, and that worked, so that's what I used. Here's how to deserialize:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1324,"value":"function deserializeWithOffset(bytesEncodedWithOffset, charCodeOffset) {\n const n = bytesEncodedWithOffset.length;\n const bytes = new Uint8Array(n);\n for (let i = 0; i < n; ++i) {\n bytes[i] = bytesEncodedWithOffset.charCodeAt(i) - charCodeOffset;\n }\n return bytes;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1296,"value":"decodedBytes = deserializeWithOffset(bytesEncodedWithOffset, charCodeOffset)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1293,"value":"And now, to verify with JSZip that it works:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1303,"value":"JSZip.loadAsync(decodedBytes)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1406,"value":"## Do we _really_ need an offset?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1314,"value":"If you try an offset of 0 and see that it works, you might think that the offset is unnecessary, but here's what the GASWA returns with offset 0:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1329,"value":"bytesEncodedWithOffset0 = \"PK\\u0003\\u0004\\u0014\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1337,"value":"Unsurprisingly, that short string does not encode all of the data:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":1250,"value":"deserializeWithOffset(bytesEncodedWithOffset0, 0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1401,"value":"---\n\n## Appendix","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":995,"value":"","pinned":true,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":492,"value":"JSZip = (await import(\"https://cdn.skypack.dev/jszip@3.9.1?min\")).default","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1097,"value":"bytesJsonStringified = FileAttachment(\n \"kmz bytes google apps script json stringified.txt\"\n).text()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1242,"value":"bytes = JSON.parse(bytesJsonStringified)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1144,"value":"kmzBase64 = FileAttachment(\"kmzBase64.txt\").text()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W31","stdlib":"1","titling":"auto"}} 2e:"$2d:body"