32:T655,# Similarity search on millions of books, in-browser Keyword search remains dominant for books, but at *some* point, whether they know it or not, everyone will probably be searching vectorized representations. This notebook tries out some methods for textual similarity search across a large corpus of books based on vectorized representations. Back in 2018, it took me a lot of effort to set up an approximate nearest-neighbors search on a server. Now in 2022, new technologies and new tricks that make it possible to search across 2 million+ books in dozens of languages without even *having* a server. In this demo notebook, I start off by loading 1.5 million books in a vectorized representation; you can use the checkboxes below to load more books in chunks of 250,000 at a time, although it's likely you'll eventually need more memory than is available in your browser. Note that I don't actually use learned representations in this notebook; instead, I use simple, language-agnostic features as described in Schmidt, Benjamin. 2018. “Stable Random Projection: Lightweight, General-Purpose Dimensionality Reduction for Digitized Libraries.” *Journal of Cultural Analytics 3 (1)*. https://doi.org/10.22148/16.025. Read below for more details. This starts with a single random seed book: you can then refine your search by adding or subtracting matches. It's not perfect, obviously. But since this uses *absolutely no metadata* at all, it's possible it turns up different sorts of views of the library. UPDATE 2022-09-17: You can now paste *any* text into a search box and get results. 33:T35bd,viewof full_text_input = { if (searchtype !== "custom") { return Inputs.button("Start with a different random book", { value: "foo" }); } return Inputs.textarea({ submit: true, value: `In compliance with the request of a friend of mine, who wrote me from the East, I called on good-natured, garrulous old Simon Wheeler, and inquired after my friend's friend, Leonidas W. Smiley, as requested to do, and I hereunto append the result. I have a lurking suspicion that Leonidas W. Smiley is a myth; that my friend never knew such a personage; and that he only conjectured that, if I asked old Wheeler about him, it would remind him of his infamous Jim Smiley, and he would go to work and bore me nearly to death with some infernal reminiscence of him as long and tedious as it should be useless to me. If that was the design, it certainly succeeded. I found Simon Wheeler dozing comfortably by the bar-room stove of the old, dilapidated tavern in the ancient mining camp of Angel's, and I noticed that he was fat and bald-headed, and had an expression of winning gentleness and simplicity upon his tranquil countenance. He roused up and gave me good-day. I told him a friend of mine had commissioned me to make some inquiries about a cherished companion of his boyhood named Leonidas W. Smiley Rev. Leonidas W. Smiley a young minister of the Gospel, who he had heard was at one time a resident of Angel's Camp. I added that, if Mr. Wheeler could tell me any thing about this Rev. Leonidas W. Smiley, I would feel under many obligations to him. Simon Wheeler backed me into a corner and blockaded me there with his chair, and then sat me down and reeled off the monotonous narrative which follows this paragraph. He never smiled, he never frowned, he never changed his voice from the gentle-flowing key to which he tuned the initial sentence, he never betrayed the slightest suspicion of enthusiasm; but all through the interminable narrative there ran a vein of impressive earnestness and sincerity, which showed me plainly that, so far from his imagining that there was any thing ridiculous or funny about his story, he regarded it as a really important matter, and admired its two heroes as men of transcendent genius in finesse. To me, the spectacle of a man drifting serenely along through such a queer yarn without ever smiling, was exquisitely absurd. As I said before, I asked him to tell me what he knew of Rev. Leonidas W. Smiley, and he replied as follows. I let him go on in his own way, and never interrupted him once: There was a feller here once by the name of Jim Smiley, in the winter of '49 or may be it was the spring of '50 I don't recollect exactly, somehow, though what makes me think it was one or the other is because I remember the big flume wasn't finished when he first came to the camp; but any way, he was the curiosest man about always betting on any thing that turned up you ever see, if he could get any body to bet on the other side; and if he couldn't, he'd change sides. Any way that suited the other man would suit him any way just so's he got a bet, he was satisfied. But still he was lucky, uncommon lucky; he most always come out winner. He was always ready and laying for a chance; there couldn't be no solittry thing mentioned but that feller'd offer to bet on it, and -take any side you please, as I was just telling you. If there was a horse-race, you'd find him flush, or you'd find him busted at the end of it; if there was a dog-fight, he'd bet on it; if there was a cat-fight, he'd bet on it; if there was a chicken-fight, he'd bet on it; why, if there was two birds setting on a fence, he would bet you which one would fly first; or if there was a camp-meeting, he would be there reg'lar, to bet on Parson Walker, which he judged to be the best exhorter about here, and so he was, too, and a good man. If he even seen a straddle-bug start to go anywheres, he would bet you how long it would take him to get wherever he was going to, and if you took him up, he would foller that straddle-bug to Mexico but what he would find out where he was bound for and how long he was on the road. Lots of the boys here has seen that Smiley, and can tell you about him. Why, it never made no difference to him he would bet on any thing the dangdest feller. Parson Walker's wife laid very sick once, for a good while, and it seemed as if they warn's going to save her; but one morning he come in, and Smiley asked how she was, and he said she was considerable better thank the Lord for his inftnit mercy and coming on so smart that, with the blessing of Providence, she'd get well yet; and Smiley, before he thought, says, "Well, I'll risk two- and-a-half that she don't, any way." Thish-yer Smiley had a mare the boys called her the fifteen- minute nag, but that was only in fun, you know, because, of course, she was faster than that and he used to win money on that horse, for all she was so slow and always had the asthma, or the distemper, or the consumption, or something of that kind. They used to give her two or three hundred yards start, and then pass her under way; but always at the fag-end of the race she'd get excited and desperate- like, and come cavorting and straddling up, and scattering her legs around limber, sometimes in the air, and sometimes out to one side amongst the fences, and kicking up m-o-r-e dust, and raising m-o-r-e racket with her coughing and sneezing and blowing her nose and always fetch up at the stand just about a neck ahead, as near as you could cipher it down. And he had a little small bull pup, that to look at him you'd think he wan's worth a cent, but to set around and look ornery, and lay for a chance to steal something. But as soon as money was up on him, he was a different dog; his underjaw'd begin to stick out like the fo'castle of a steamboat, and his teeth would uncover, and shine savage like the furnaces. And a dog might tackle him, and bully- rag him, and bite him, and throw him over his shoulder two or three times, and Andrew Jackson which was the name of the pup Andrew Jackson would never let on but what he was satisfied, and hadn't expected nothing else and the bets being doubled and doubled on the other side all the time, till the money was all up; and then all of a sudden he would grab that other dog jest by the j'int of his hind leg and freeze on it not chew, you understand, but only jest grip and hang on till they thronged up the sponge, if it was a year. Smiley always come out winner on that pup, till he harnessed a dog once that didn't have no hind legs, because they'd been sawed off by a circular saw, and when the thing had gone along far enough, and the money was all up, and he come to make a snatch for his pet bolt, he saw in a minute how he'd been imposed on, and how the other dog had him in the door, so to speak, and he 'peered sur- prised, and then he looked sorter discouraged-like, and didn't try no more to win the fight, and so he got shucked out bad. He give Smiley a look, as much as to say his heart was broke, and it was his fault, for putting up a dog that hadn't no hind legs for him to take bolt of, which was his main dependence in a fight, and then he limped off a piece and laid down and died. It was a good pup, was that Andrew Jackson, and would have made a name for hisself if he'd lived, for the stuff was in him, and he had genius I know it, because he hadn't had no opportunities to speak of, and it don't stand to reason that a dog could make such a fight as he could under them circumstances, if he hadn't no talent. It always makes me feel sorry when I think of that last fight of his'n, and the way it turned out. Well, thish-yer Smiley had rat-tarriers, and chicken cocks, and tom- cats, and all of them kind of things, till you couldn't rest, and you couldn't fetch nothing for him to bet on but he'd match you. He ketched a frog one day, and took him home, and said he cal'klated to edercate him; and so he never done nothing for three months but set in his back yard and learn that frog to jump. And you bet you he did learn him, too. He'd give him a little punch behind, and the next minute you'd see that frog whirling in the air like a doughnut see him turn one summerset, or may be a couple, if he got a good start, and come down flat-footed and all right, like a cat. He got him up so in the matter of catching flies, and kept him in practice so constant, that he'd nail a fly every time as far as he could see him. Smiley said all a frog wanted was education, and he could do most any thing and I believe him. Why, I've seen him set Dan'l Webster down here on this floor Dan'l Webster was the name of the frog and sing out, "Flies, Dan'l, flies!" and quicker'n you could wink, he'd spring straight up, and snake a fly off'n the counter there, and flop down on the floor again as solid as a gob of mud, and fall to scratching the side of his head with his hind foot as indifferent as if he hadn't no idea he'd been doin' any more'n any frog might do. You never see a frog so modest and straightforward as he was, for all he was so gifted. And when it come to fair and square jumping on a dead level, he could get over more ground at one straddle than any animal of his breed you ever see. Jumping on a dead level was his strong suit, you understand; and when it come to that, Smiley would ante up money on him as long as he had a red. Smiley was monstrous proud of his frog, and well he might be, for fellers that had traveled and been everywheres, all said he laid over any frog that ever they see. Well, Smiley kept the beast in a little lattice box, and he used to fetch him down town sometimes and lay for a bet. One day a feller a stranger in the camp, he was come across him with his box, and says: "What might it be that you've got in the box?" And Smiley says, sorter indifferent like, "It might be a parrot, or it might be a canary, may be, but it an't it's only just a frog." And the feller took it, and looked at it careful, and turned it round this way and that, and says, "H'm so 'tis. Well, what's he good for?" "Well," Smiley says, easy and careless, "He's good enough for one thing, I should judge he can outjump any frog in Calaveras county." The feller took the box again, and took another long, particular look, and give it back to Smiley, and says, very deliberate, "Well, I don't see no p'ints about that frog that's any better'n any other frog." "May be you don't," Smiley says. "May be you understand frogs, and may be you don't understand 'em; may be you've had experience, and may be you an't only a amature, as it were. Anyways, I've got my opinion, and I'll risk forty dollars that he can outjump any frog in Calaveras county." And the feller studied a minute, and then says, kinder sad like, "Well, I'm only a stranger here, and I an't got no frog; but if I had a frog, I'd bet you." And then Smiley says, "That's all right that's all right if you'll hold my box a minute, I'll go and get you a frog." And so the feller took the box, and put up his forty dollars along with Smiley's, and set down to wait. So he set there a good while thinking and thinking to hisself, and then he got the frog out and prized his mouth open and took a tea- spoon and filled him full of quail shot filled him pretty near up to his chin and set him on the floor. Smiley he went to the swamp and slopped around in the mud for a long time, and finally he ketched a frog, and fetched him in, and give him to this feller, and says: "Now, if you're ready, set him alongside of Dan'l, with his fore- paws just even with Dan'l, and I'll give the word." Then he says, "One two three jump!" and him and the feller touched up the frogs from behind, and the new frog hopped off, but Dan'l give a heave, and hysted up his shoulders so like a Frenchman, but it wan's no use he couldn't budge; he was planted as solid as an anvil, and he couldn't no more stir than if he was anchored out. Smiley was a good deal surprised, and he was disgusted too, but he didn't have no idea what the matter was, of course. The feller took the money and started away; and when he was going out at the door, he sorter jerked his thumb over his shoulders this way at Dan'l, and says again, very deliberate, "Well, I don't see no p'ints about that frog that's any better'n any other frog." Smiley he stood scratching his head and looking down at Dan'l a long time, and at last he says, "I do wonder what in the nation that frog throw'd off for I wonder if there an't something the matter with him he 'pears to look mighty baggy, somehow." And he ketched Dan'l by the nap of the neck, and lifted him up and says, "Why, blame my cats, if he don't weigh five pound!" and turned him upside down, and he belched out a double handful of shot. And then he see how it was, and he was the maddest man he set the frog down and took out after that feller, but he never ketchd him. And- [Here Simon Wheeler heard his name called from the front yard, and got up to see what was wanted.] And turning to me as he moved away, he said: "Just set where you are, stranger, and rest easy I an't going to be gone a second." But, by your leave, I did not think that a continuation of the history of the enterprising vagabond Jim Smiley would be likely to afford me much information concerning the Rev. Leonidas W. Smiley, and so I started away. At the door I met the sociable Wheeler returning, and he button- holed me and recommenced: "Well, thish-yer Smiley had a yeller one-eyed cow that didn't have no tail, only jest a short stump like a bannanner, and " "Oh! hang Smiley and his afflicted cow!" I muttered, good-naturedly, and bidding the old gentleman good-day, I departed. ` }); }34:T76e,results = { const div = DOM.element("div"); const header = d3.select(div).append("div"); header.append("h2").text("Seed texts"); if (searchtype === "custom" && full_text_input.split(" ").length < 1000) { header .append("div") .style("background-color", "#FF000055") .append("em") .text( "Note: You've pasted in less than a thousand words: since the random projections rely on overall word counts, you probably won't get especially good results." ); } const header_items = header .append("div") .style("border-style", "groove") .style("padding", "10px") .style("margin", "10px") .append("div") .selectAll("div") .data(search_htmls) .join("div") .style("display", "grid") .style("grid-template-columns", "30px 90%"); header_items .append("div") .append("button") .style("max-height", "30px") .style("margin", "3px") .on("click", (d, e) => { console.log(d, e); mutable current = current.filter((el) => el !== d.target.__data__.htid); }) .text("x") .style("border-radius", "3px"); header_items.append("div").html((d) => d.html); d3.select(div) .append("h2") .style("margin-top", "10px") .text("Closest Matches"); const droot = d3.select(div).append("div").attr("id", "droot"); droot .style("display", "grid") .style("margin", "13px") .style("grid-template-columns", "33% 33% 33%"); const ds = droot .selectAll("div") .data(data.map((d) => d.htid)) .join("div") .attr("class", "placeholder") .style("margin", "3px") .style("border-style", "groove") .style("border-radius", "1px") .style("padding", "5px") .style("display", "grid") .style("grid-template-columns", "10% 90%") .html((d) => d); invalidation.then((d) => d3.select(div).html("

Reloading...

")); return div; }35:T841,viewof files = Inputs.checkbox( new Map([ ["000: aeu-coo", "000"], ["001: coo-coo", "001"], ["002: coo-coo1", "002"], ["003: coo1-hvd", "003"], ["004: hvd-hvd", "004"], ["005: hvd-hvd", "005"], ["006: hvd-iau", "006"], ["007: iau-ien", "007"], ["008: ien-inu", "008"], ["009: inu-inu", "009"], ["010: inu-loc", "010"], ["011: loc-mdp", "011"], ["012: mdp-mdp", "012"], ["013: mdp-mdp", "013"], ["014: mdp-mdp", "014"], ["015: mdp-mdp", "015"], ["016: mdp-mdp", "016"], ["017: mdp-mdp", "017"], ["018: mdp-mdp", "018"], ["019: mdp-mdp", "019"], ["020: mdp-mdp", "020"], ["021: mdp-mdp", "021"], ["022: mdp-mdp", "022"], ["023: mdp-mdp", "023"], ["024: mdp-mdp", "024"], ["025: mdp-mdp", "025"], ["026: mdp-mdp", "026"], ["027: mdp-mdp", "027"], ["028: mdp-mdp", "028"], ["029: mdp-mdp", "029"], ["030: mdp-msu", "030"], ["031: msu-njp", "031"], ["032: njp-nyp", "032"], ["033: nyp-nyp", "033"], ["034: nyp-osu", "034"], ["035: osu-pst", "035"], ["036: pst-pst", "036"], ["037: pst-txu", "037"], ["038: txu-txu", "038"], ["039: txu-uc1", "039"], ["040: uc1-uc1", "040"], ["041: uc1-uc1", "041"], ["042: uc1-uc1", "042"], ["043: uc1-uc1", "043"], ["044: uc1-uc1", "044"], ["045: uc1-uc1", "045"], ["046: uc1-uc1", "046"], ["047: uc1-uc1", "047"], ["048: uc1-uc1", "048"], ["049: uc1-uc1", "049"], ["050: uc1-uc1", "050"], ["051: uc1-uc1", "051"], ["052: uc1-uc1", "052"], ["053: uc1-uc1", "053"], ["054: uc1-uc1", "054"], ["055: uc1-uc1", "055"], ["056: uc1-uc2", "056"], ["057: uc2-uiug", "057"], ["058: uiug-uiug", "058"], ["059: uiug-uiug", "059"], ["060: uiug-uiuo", "060"], ["061: uiuo-umn", "061"], ["062: umn-umn", "062"], ["063: umn-uva", "063"], ["064: uva-uva", "064"], ["065: uva-uva", "065"], ["066: uva-wu", "066"], ["067: wu-wu", "067"], ["068: wu-yul", "068"] ]), { value: ["010", "020", "030", "040", "050", "060"] } )36:T41f,## Methods This is a multistep process. Each book is reduced to a 1280-dimensional vector based on its words as described in Schmidt, Benjamin. 2018. “Stable Random Projection: Lightweight, General-Purpose Dimensionality Reduction for Digitized Libraries.” Journal of Cultural Analytics 3 (1). https://doi.org/10.22148/16.025. Rather than use the full floats, I then reduce those each dimension to a single bit by determining if it is greater or less than zero. This gets us down from ${1280 * 4} bytes per book to just ${1280 / 8}. That loses some information, but there's still something there. I then take those bytes and pack them into 20 big integers, and put them into parquet files as separate columns. That all happens server-side. Now we're in the browser. It's then very easy and fast to calculate the distance between any two books in Hamming space by using binary operators: first `xor` (to see which bits have the same values) and then `BIT_COUNT` (to see how many bits are set). We can do this across as many books as necessary. 37:T4019,SRP = function (dims) { // As an extended prologue, this bundles a slightly changed version of Chris Veness's // SHA-1 code. Skip down for the SRP bits. /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* SHA-1 implementation in JavaScript (c) Chris Veness 2002-2014 / MIT Licence */ /* */ /* - see http://csrc.nist.gov/groups/ST/toolkit/secure_hashing.html */ /* http://csrc.nist.gov/groups/ST/toolkit/examples.html */ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /** * SHA-1 hash function reference implementation. * * @namespace */ var Sha1 = {}; /** * Generates SHA-1 hash of string. * * @param {string} msg - (Unicode) string to be hashed. * @returns {string} Hash of msg as hex character string. */ Sha1.hash = function (msg) { // convert string to UTF-8, as SHA only deals with byte-streams msg = msg.utf8Encode(); // constants [§4.2.1] var K = [0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6]; // PREPROCESSING msg += String.fromCharCode(0x80); // add trailing '1' bit (+ 0's padding) to string [§5.1.1] // convert string msg into 512-bit/16-integer blocks arrays of ints [§5.2.1] var l = msg.length / 4 + 2; // length (in 32-bit integers) of msg + ‘1’ + appended length var N = Math.ceil(l / 16); // number of 16-integer-blocks required to hold 'l' ints var M = new Array(N); for (var i = 0; i < N; i++) { M[i] = new Array(16); for (var j = 0; j < 16; j++) { // encode 4 chars per integer, big-endian encoding M[i][j] = (msg.charCodeAt(i * 64 + j * 4) << 24) | (msg.charCodeAt(i * 64 + j * 4 + 1) << 16) | (msg.charCodeAt(i * 64 + j * 4 + 2) << 8) | msg.charCodeAt(i * 64 + j * 4 + 3); } // note running off the end of msg is ok 'cos bitwise ops on NaN return 0 } // add length (in bits) into final pair of 32-bit integers (big-endian) [§5.1.1] // note: most significant word would be (len-1)*8 >>> 32, but since JS converts // bitwise-op args to 32 bits, we need to simulate this by arithmetic operators M[N - 1][14] = ((msg.length - 1) * 8) / Math.pow(2, 32); M[N - 1][14] = Math.floor(M[N - 1][14]); M[N - 1][15] = ((msg.length - 1) * 8) & 0xffffffff; // set initial hash value [§5.3.1] var H0 = 0x67452301; var H1 = 0xefcdab89; var H2 = 0x98badcfe; var H3 = 0x10325476; var H4 = 0xc3d2e1f0; // HASH COMPUTATION [§6.1.2] var W = new Array(80); var a, b, c, d, e; for (var i = 0; i < N; i++) { // 1 - prepare message schedule 'W' for (var t = 0; t < 16; t++) W[t] = M[i][t]; for (var t = 16; t < 80; t++) W[t] = Sha1.ROTL(W[t - 3] ^ W[t - 8] ^ W[t - 14] ^ W[t - 16], 1); // 2 - initialise five working variables a, b, c, d, e with previous hash value a = H0; b = H1; c = H2; d = H3; e = H4; // 3 - main loop for (var t = 0; t < 80; t++) { var s = Math.floor(t / 20); // seq for blocks of 'f' functions and 'K' constants var T = (Sha1.ROTL(a, 5) + Sha1.f(s, b, c, d) + e + K[s] + W[t]) & 0xffffffff; e = d; d = c; c = Sha1.ROTL(b, 30); b = a; a = T; } // 4 - compute the new intermediate hash value (note 'addition modulo 2^32') H0 = (H0 + a) & 0xffffffff; H1 = (H1 + b) & 0xffffffff; H2 = (H2 + c) & 0xffffffff; H3 = (H3 + d) & 0xffffffff; H4 = (H4 + e) & 0xffffffff; } var elements = Sha1.toBinary(H0) + Sha1.toBinary(H1) + Sha1.toBinary(H2) + Sha1.toBinary(H3) + Sha1.toBinary(H4); return elements; }; Sha1.pad = function (n, width, z) { //http://stackoverflow.com/questions/10073699/pad-a-number-with-leading-zeros-in-javascript z = z || "0"; n = n + ""; return n.length >= width ? n : new Array(width - n.length + 1).join(z) + n; }; Sha1.toBinary = function (n) { var hex = Sha1.toHexStr(n); var binary = parseInt(hex, 16).toString(2); return Sha1.pad(binary, 32, "0"); }; /** * Function 'f' [§4.1.1]. * @private */ Sha1.f = function (s, x, y, z) { switch (s) { case 0: return (x & y) ^ (~x & z); // Ch() case 1: return x ^ y ^ z; // Parity() case 2: return (x & y) ^ (x & z) ^ (y & z); // Maj() case 3: return x ^ y ^ z; // Parity() } }; /** * Rotates left (circular left shift) value x by n positions [§3.2.5]. * @private */ Sha1.ROTL = function (x, n) { return (x << n) | (x >>> (32 - n)); }; /** * Hexadecimal representation of a number. * @private */ Sha1.toHexStr = function (n) { // note can't use toString(16) as it is implementation-dependant, // and in IE returns signed numbers when used on full words var s = "", v; for (var i = 7; i >= 0; i--) { v = (n >>> (i * 4)) & 0xf; s += v.toString(16); } return s; }; /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /** Extend String object with method to encode multi-byte string to utf8 * - monsur.hossa.in/2012/07/20/utf-8-in-javascript.html */ if (typeof String.prototype.utf8Encode == "undefined") { String.prototype.utf8Encode = function () { return unescape(encodeURIComponent(this)); }; } /** Extend String object with method to decode utf8 string to multi-byte */ if (typeof String.prototype.utf8Decode == "undefined") { String.prototype.utf8Decode = function () { try { return decodeURIComponent(escape(this)); } catch (e) { return this; // invalid UTF-8? return as-is } }; } // if (typeof module != 'undefined' && module.exports) module.exports = Sha1; // CommonJs export // if (typeof define == 'function' && define.amd) define([], function() { return Sha1; }); // AMD /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* - - - - - - - - - - - - - - - - - - - - SRP IMPLEMENTATION - - - - - - - - - - - - - - - - - */ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ // Here's the actual SRP implementation. dims = dims || 160; var known = Object.create(null); var that = function () {}; ///////////////////////////////////////////////////////////////// // Some code from http://webcache.googleusercontent.com/search?q=cache:g3gJzZr5uhcJ:jsperf.com/typed-arrays-parsing+&cd=1&hl=en&ct=clnk&gl=us // to parse to and from base64 ///////////////////////////////////////////////////////////////// // Array -> Float32Array function array_to_float32array(a) { return new Float32Array(a); } // Float32Array -> ASCII String function float32array_to_ascii(a) { // Get a Uint8Array view var buf = new Uint8Array(a.buffer, 0, a.length * 4); // Convert Uint8Array to ASCII string var ascii = ""; var len = buf.length; for (var i = 0; i < len; ++i) { ascii += String.fromCharCode(buf[i]); } return ascii; } // ASCII String -> Float32Array function ascii_to_float32array(s) { // Convert ASCII string to Uint8Array var len = s.length; var buf = new Uint8Array(len); for (var i = 0; i < len; ++i) { buf[i] = s.charCodeAt(i); } // Uint8Array to Float32Array return new Float32Array(buf.buffer, 0, len / 4); } // ASCII String -> Base64 String function ascii_to_base64(s) { // Remove non-printable characters using base64 encoding return btoa(s); } // Base64 String -> ASCII String function base64_to_ascii(s) { // Remove non-printable characters using base64 encoding return atob(s); } function array_to_base64(a) { return ascii_to_base64(float32array_to_ascii(array_to_float32array(a))); } function base64_to_array(a) { return ascii_to_float32array(base64_to_ascii(a)); } ///////////////////////////////////////////////////////////////// that.SRP_string = function (string, log, format) { return that.SRP([string], [1], log); }; that.toBase64 = array_to_base64; that.fromBase64 = base64_to_array; that.SRP = function (words, counts, log, format, unprojected) { // Called with *either* a string appearing once, // or with a set of words and their corresponding counts. if (typeof log === "undefined") { log = true; } //format can be base64, whatever else makes sense. //base64 seems like a sensible way to pass 32-bit floats around. format = format || "array"; // if (typeof words == "string") { return that.SRP_string(words, log, format); } // Oh my god. Took about 45 minutes of debugging a 100,000 // word text to realize that "constructor" couldn't be a key // in a javascript dictionary. var countDict = Object.create(null); words.forEach(function (wordset, i) { var count = counts[i]; var locWords = that.tokenize(wordset); locWords.forEach(function (word) { word = word.replace(/[0-9]/g, "#"); if (word == "constructor") { return; } if (countDict[word]) { countDict[word] += counts[i]; } else { countDict[word] = counts[i]; } }); }); // Now that the normalized counts are done, we can wipe out // the old word and count signs. counts = []; words = []; var total = 0; Object.keys(countDict).forEach(function (key) { words.push(key); counts.push(countDict[key]); total += countDict[key]; }); if (log) { counts = counts.map(function (d) { var score = Math.log((d * 1e5) / total); return score < 0 ? 0 : score; }); } //initialize a null array var scores = Array.apply(null, Array(dims)).map( Number.prototype.valueOf, 0 ); var rotations = words.map(function (d) { return token_SRP(d, dims); }); if (unprojected) { return { counts: counts, words: words, rotations: rotations }; } counts.forEach(function (count, i) { var rotation = rotations[i]; return rotation.map(function (rot, j) { scores[j] += count * rot; }); }); if (format == "base64") { return array_to_base64(scores); } else { return scores.map(function (d) { return parseFloat(d.toFixed(4)); }); } }; that.tokenize = function (text) { //http://stackoverflow.com/questions/150033/regular-expression-to-match-non-english-characters var words = [], m, r = /([\u0041-\u005A\u0061-\u007A\u00AA\u00B5\u00BA\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02C1\u02C6-\u02D1\u02E0-\u02E4\u02EC\u02EE\u0370-\u0374\u0376\u0377\u037A-\u037D\u0386\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03F5\u03F7-\u0481\u048A-\u0527\u0531-\u0556\u0559\u0561-\u0587\u05D0-\u05EA\u05F0-\u05F2\u0620-\u064A\u066E\u066F\u0671-\u06D3\u06D5\u06E5\u06E6\u06EE\u06EF\u06FA-\u06FC\u06FF\u0710\u0712-\u072F\u074D-\u07A5\u07B1\u07CA-\u07EA\u07F4\u07F5\u07FA\u0800-\u0815\u081A\u0824\u0828\u0840-\u0858\u08A0\u08A2-\u08AC\u0904-\u0939\u093D\u0950\u0958-\u0961\u0971-\u0977\u0979-\u097F\u0985-\u098C\u098F\u0990\u0993-\u09A8\u09AA-\u09B0\u09B2\u09B6-\u09B9\u09BD\u09CE\u09DC\u09DD\u09DF-\u09E1\u09F0\u09F1\u0A05-\u0A0A\u0A0F\u0A10\u0A13-\u0A28\u0A2A-\u0A30\u0A32\u0A33\u0A35\u0A36\u0A38\u0A39\u0A59-\u0A5C\u0A5E\u0A72-\u0A74\u0A85-\u0A8D\u0A8F-\u0A91\u0A93-\u0AA8\u0AAA-\u0AB0\u0AB2\u0AB3\u0AB5-\u0AB9\u0ABD\u0AD0\u0AE0\u0AE1\u0B05-\u0B0C\u0B0F\u0B10\u0B13-\u0B28\u0B2A-\u0B30\u0B32\u0B33\u0B35-\u0B39\u0B3D\u0B5C\u0B5D\u0B5F-\u0B61\u0B71\u0B83\u0B85-\u0B8A\u0B8E-\u0B90\u0B92-\u0B95\u0B99\u0B9A\u0B9C\u0B9E\u0B9F\u0BA3\u0BA4\u0BA8-\u0BAA\u0BAE-\u0BB9\u0BD0\u0C05-\u0C0C\u0C0E-\u0C10\u0C12-\u0C28\u0C2A-\u0C33\u0C35-\u0C39\u0C3D\u0C58\u0C59\u0C60\u0C61\u0C85-\u0C8C\u0C8E-\u0C90\u0C92-\u0CA8\u0CAA-\u0CB3\u0CB5-\u0CB9\u0CBD\u0CDE\u0CE0\u0CE1\u0CF1\u0CF2\u0D05-\u0D0C\u0D0E-\u0D10\u0D12-\u0D3A\u0D3D\u0D4E\u0D60\u0D61\u0D7A-\u0D7F\u0D85-\u0D96\u0D9A-\u0DB1\u0DB3-\u0DBB\u0DBD\u0DC0-\u0DC6\u0E01-\u0E30\u0E32\u0E33\u0E40-\u0E46\u0E81\u0E82\u0E84\u0E87\u0E88\u0E8A\u0E8D\u0E94-\u0E97\u0E99-\u0E9F\u0EA1-\u0EA3\u0EA5\u0EA7\u0EAA\u0EAB\u0EAD-\u0EB0\u0EB2\u0EB3\u0EBD\u0EC0-\u0EC4\u0EC6\u0EDC-\u0EDF\u0F00\u0F40-\u0F47\u0F49-\u0F6C\u0F88-\u0F8C\u1000-\u102A\u103F\u1050-\u1055\u105A-\u105D\u1061\u1065\u1066\u106E-\u1070\u1075-\u1081\u108E\u10A0-\u10C5\u10C7\u10CD\u10D0-\u10FA\u10FC-\u1248\u124A-\u124D\u1250-\u1256\u1258\u125A-\u125D\u1260-\u1288\u128A-\u128D\u1290-\u12B0\u12B2-\u12B5\u12B8-\u12BE\u12C0\u12C2-\u12C5\u12C8-\u12D6\u12D8-\u1310\u1312-\u1315\u1318-\u135A\u1380-\u138F\u13A0-\u13F4\u1401-\u166C\u166F-\u167F\u1681-\u169A\u16A0-\u16EA\u1700-\u170C\u170E-\u1711\u1720-\u1731\u1740-\u1751\u1760-\u176C\u176E-\u1770\u1780-\u17B3\u17D7\u17DC\u1820-\u1877\u1880-\u18A8\u18AA\u18B0-\u18F5\u1900-\u191C\u1950-\u196D\u1970-\u1974\u1980-\u19AB\u19C1-\u19C7\u1A00-\u1A16\u1A20-\u1A54\u1AA7\u1B05-\u1B33\u1B45-\u1B4B\u1B83-\u1BA0\u1BAE\u1BAF\u1BBA-\u1BE5\u1C00-\u1C23\u1C4D-\u1C4F\u1C5A-\u1C7D\u1CE9-\u1CEC\u1CEE-\u1CF1\u1CF5\u1CF6\u1D00-\u1DBF\u1E00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FBC\u1FBE\u1FC2-\u1FC4\u1FC6-\u1FCC\u1FD0-\u1FD3\u1FD6-\u1FDB\u1FE0-\u1FEC\u1FF2-\u1FF4\u1FF6-\u1FFC\u2071\u207F\u2090-\u209C\u2102\u2107\u210A-\u2113\u2115\u2119-\u211D\u2124\u2126\u2128\u212A-\u212D\u212F-\u2139\u213C-\u213F\u2145-\u2149\u214E\u2183\u2184\u2C00-\u2C2E\u2C30-\u2C5E\u2C60-\u2CE4\u2CEB-\u2CEE\u2CF2\u2CF3\u2D00-\u2D25\u2D27\u2D2D\u2D30-\u2D67\u2D6F\u2D80-\u2D96\u2DA0-\u2DA6\u2DA8-\u2DAE\u2DB0-\u2DB6\u2DB8-\u2DBE\u2DC0-\u2DC6\u2DC8-\u2DCE\u2DD0-\u2DD6\u2DD8-\u2DDE\u2E2F\u3005\u3006\u3031-\u3035\u303B\u303C\u3041-\u3096\u309D-\u309F\u30A1-\u30FA\u30FC-\u30FF\u3105-\u312D\u3131-\u318E\u31A0-\u31BA\u31F0-\u31FF\u3400-\u4DB5\u4E00-\u9FCC\uA000-\uA48C\uA4D0-\uA4FD\uA500-\uA60C\uA610-\uA61F\uA62A\uA62B\uA640-\uA66E\uA67F-\uA697\uA6A0-\uA6E5\uA717-\uA71F\uA722-\uA788\uA78B-\uA78E\uA790-\uA793\uA7A0-\uA7AA\uA7F8-\uA801\uA803-\uA805\uA807-\uA80A\uA80C-\uA822\uA840-\uA873\uA882-\uA8B3\uA8F2-\uA8F7\uA8FB\uA90A-\uA925\uA930-\uA946\uA960-\uA97C\uA984-\uA9B2\uA9CF\uAA00-\uAA28\uAA40-\uAA42\uAA44-\uAA4B\uAA60-\uAA76\uAA7A\uAA80-\uAAAF\uAAB1\uAAB5\uAAB6\uAAB9-\uAABD\uAAC0\uAAC2\uAADB-\uAADD\uAAE0-\uAAEA\uAAF2-\uAAF4\uAB01-\uAB06\uAB09-\uAB0E\uAB11-\uAB16\uAB20-\uAB26\uAB28-\uAB2E\uABC0-\uABE2\uAC00-\uD7A3\uD7B0-\uD7C6\uD7CB-\uD7FB\uF900-\uFA6D\uFA70-\uFAD9\uFB00-\uFB06\uFB13-\uFB17\uFB1D\uFB1F-\uFB28\uFB2A-\uFB36\uFB38-\uFB3C\uFB3E\uFB40\uFB41\uFB43\uFB44\uFB46-\uFBB1\uFBD3-\uFD3D\uFD50-\uFD8F\uFD92-\uFDC7\uFDF0-\uFDFB\uFE70-\uFE74\uFE76-\uFEFC\uFF21-\uFF3A\uFF41-\uFF5A\uFF66-\uFFBE\uFFC2-\uFFC7\uFFCA-\uFFCF\uFFD2-\uFFD7\uFFDA-\uFFDC]+|\d+)/g; while ((m = r.exec(text))) { words.push(m[0].toLowerCase()); } return words; }; var token_SRP = function (word, token_dims) { if (known[word]) { return known[word]; } // We do it in chunks of 160, like so. if (token_dims > 160) { return token_SRP(word, 160).concat( token_SRP(word + "_", token_dims - 160) ); } if (token_dims < 160) { return token_SRP(word, 160).slice(0, token_dims); } // The Sha1 implementation here already returns binary, // And coerces to UTF-8. That makes this a lot simpler than Python or R. var hash = Sha1.hash(word) .split("") .map(function (d) { return d * 2 - 1; }); if (token_dims == dims) { known[word] = hash; } return hash; }; return that; }38:[{"id":"4483b156322e486c484c6417b7020fb8535566650c36e04899215293edb8b928a11e9ba53f564b1b575c535943a8b2ed882cd6c75217ddfb11b108f6eb0bb2ee","href":"https://static.observableusercontent.com/files/4483b156322e486c484c6417b7020fb8535566650c36e04899215293edb8b928a11e9ba53f564b1b575c535943a8b2ed882cd6c75217ddfb11b108f6eb0bb2ee","name":"chunk_051.parquet","type":"application/octet-stream","lastModified":1663166101829.635,"size":47300407},{"id":"503259bb31e82cac2d9304f75a67362a5c60a39ea0bd1babf740246c6a411fcf394ef2359d823add17ff2e6a146d0552947cb10a88d1b57fc638754b80ea5c26","href":"https://static.observableusercontent.com/files/503259bb31e82cac2d9304f75a67362a5c60a39ea0bd1babf740246c6a411fcf394ef2359d823add17ff2e6a146d0552947cb10a88d1b57fc638754b80ea5c26","name":"chunk_041.parquet","type":"application/octet-stream","lastModified":1663166102242.131,"size":47418068},{"id":"92f9502c1982a951d554821cb267f780db3e6d1cb1f33d6a4e9ff7f404f0af4ec55c9d416af98ea20f556e3afbd82efba0e05d3785c66308c0900c015e038622","href":"https://static.observableusercontent.com/files/92f9502c1982a951d554821cb267f780db3e6d1cb1f33d6a4e9ff7f404f0af4ec55c9d416af98ea20f556e3afbd82efba0e05d3785c66308c0900c015e038622","name":"chunk_031.parquet","type":"application/octet-stream","lastModified":1663166102319.667,"size":47533900},{"id":"9bc7ecab6b000af99abf1ea4765bf1d9fc314a6be91c7a6944b9b4c64a91e64fde53d9f516ffbdccf89574978747db658972241e973e83ba0d57df1ca801839c","href":"https://static.observableusercontent.com/files/9bc7ecab6b000af99abf1ea4765bf1d9fc314a6be91c7a6944b9b4c64a91e64fde53d9f516ffbdccf89574978747db658972241e973e83ba0d57df1ca801839c","name":"chunk_011.parquet","type":"application/octet-stream","lastModified":1663166102683.914,"size":47514092},{"id":"6ba4e13d5d2a1face414dae70fd606748422fe66c4f72e030d9a632c09db12e1b8c806872069848830d1de8f0555d94d682936b5e8d2265d1635b7a3154df324","href":"https://static.observableusercontent.com/files/6ba4e13d5d2a1face414dae70fd606748422fe66c4f72e030d9a632c09db12e1b8c806872069848830d1de8f0555d94d682936b5e8d2265d1635b7a3154df324","name":"chunk_061.parquet","type":"application/octet-stream","lastModified":1663166106006.463,"size":47688468},{"id":"42f3a2e05ce768f18643d12eb7daeb1a731ca6c02cf00196801c68e6c32f46438b0ab6df74530c346b0837a75d6671a0f1af4361111998874719e6eb3384b805","href":"https://static.observableusercontent.com/files/42f3a2e05ce768f18643d12eb7daeb1a731ca6c02cf00196801c68e6c32f46438b0ab6df74530c346b0837a75d6671a0f1af4361111998874719e6eb3384b805","name":"chunk_001.parquet","type":"application/octet-stream","lastModified":1663166106054.96,"size":47458106},{"id":"7ad0f53804db43c24c448975ad21fc6ab44400effbf04bac3659ed4222ed0346ec30e1973738a7be792f28155b1702418b974c06c7cfa82a897f445de17ee70e","href":"https://static.observableusercontent.com/files/7ad0f53804db43c24c448975ad21fc6ab44400effbf04bac3659ed4222ed0346ec30e1973738a7be792f28155b1702418b974c06c7cfa82a897f445de17ee70e","name":"chunk_067.parquet","type":"application/octet-stream","lastModified":1663171191736.364,"size":47496836},{"id":"0f763fcee0ce41c36bced03a2fecc445f9dd2058c1c0210ada1df71ef4b4ef0a02faa0d00cd011efe790e19413bf7f74a3489e08c8da58aa1a12bfa6e3f2c1eb","href":"https://static.observableusercontent.com/files/0f763fcee0ce41c36bced03a2fecc445f9dd2058c1c0210ada1df71ef4b4ef0a02faa0d00cd011efe790e19413bf7f74a3489e08c8da58aa1a12bfa6e3f2c1eb","name":"chunk_066.parquet","type":"application/octet-stream","lastModified":1663171191586.141,"size":47489335},{"id":"3262cf88ed344938569dc83f6f3522458d40a6ad132847d8b52e6dec311e6f74cb398155ab9430d3d4c4ae3b94e1918c33a19baaace8f970c65f3f6a6846cf45","href":"https://static.observableusercontent.com/files/3262cf88ed344938569dc83f6f3522458d40a6ad132847d8b52e6dec311e6f74cb398155ab9430d3d4c4ae3b94e1918c33a19baaace8f970c65f3f6a6846cf45","name":"chunk_068.parquet","type":"application/octet-stream","lastModified":1663171188746.691,"size":25904812},{"id":"327c2847fb3a3383be89f00ff660ac55ea0ca72a9b21a45f041ed6cf5fadd32458ebb26b2f79594c30617704d8e956c5294e8079fa3c949f9b39fe9c51281f43","href":"https://static.observableusercontent.com/files/327c2847fb3a3383be89f00ff660ac55ea0ca72a9b21a45f041ed6cf5fadd32458ebb26b2f79594c30617704d8e956c5294e8079fa3c949f9b39fe9c51281f43","name":"chunk_021.parquet","type":"application/octet-stream","lastModified":1663166106031.649,"size":47230545},{"id":"aa0f2bcda31c39b52468df98399de7983b4a90023f8a254e90eb0f73dca488af704b11fc703d38e3acec23a1598ceeb24a8c659ff1e502a51f6ac9ed0929de58","href":"https://static.observableusercontent.com/files/aa0f2bcda31c39b52468df98399de7983b4a90023f8a254e90eb0f73dca488af704b11fc703d38e3acec23a1598ceeb24a8c659ff1e502a51f6ac9ed0929de58","name":"chunk_000.parquet","type":"application/octet-stream","lastModified":1663164639224.863,"size":47631752}] 30:{"title":"Similarity search on millions of books, in-browser","cells":[{"id":0,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":967,"value":"viewof searchtype = Inputs.radio(\n new Map([\n [\"Random starting point\", \"random\"],\n [\"Paste your own text as a seed\", \"custom\"]\n ]),\n { value: \"random\", label: \"Where should the seed text come from?\" }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":570,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":110,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":1381,"value":"4096 * 2","pinned":true,"hidden":false,"mode":"ojs"},{"id":1311,"value":"process = {\n yield html`Fetching neighbor info from Hathi...`;\n\n const droot = d3.select(results).select(\"#droot\");\n\n droot.selectAll(\"div.placeholder\").remove();\n const ds = droot\n .selectAll(\"div\")\n .data(htmls)\n .join(\"div\")\n .style(\"margin\", \"3px\")\n .style(\"border-style\", \"groove\")\n .style(\"border-radius\", \"1px\")\n .style(\"padding\", \"5px\")\n .style(\"display\", \"grid\")\n .style(\"grid-template-columns\", \"10% 90%\");\n\n ds.selectAll(\"div\").remove();\n ds.append(\"button\")\n .style(\"max-height\", \"30px\")\n .style(\"margin\", \"3px\")\n .on(\"click\", (d, e) => {\n console.log(d, e);\n mutable current = [...current, d.target.__data__.htid];\n })\n .text(\"+\")\n .style(\"border-radius\", \"3px\");\n\n ds.append(\"div\").html(\n (d) => d.html + ` distance: ${d.distance}`\n );\n yield html``;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1350,"value":"## Choose files\n\nI'm hosting the underlying data as 69 parquet files of about 50 MB each on github. You can check the ones that you want to load here. Each file has 250,000 books in it: they are sorted mostly alphabetically by library. `mdp` is the University of Michigan Michigan, `uc1` is the University of California, `uva` is UVA, etc. A full list of library codes is here: https://github.com/Bookworm-project/Bookworm-MARC/issues/1#issuecomment-217435774. ","pinned":false,"hidden":false,"mode":"md"},{"id":1033,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":447,"value":"viewof fields_to_use = Inputs.range([1, 20], {\n step: 1,\n value: 20,\n label:\n \"Search quality: higher numbers take longer but return better neighbors\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1371,"value":"SELECT htid, A, B, C, D FROM blobb LIMIT 10","pinned":true,"hidden":false,"mode":"sql","output":"","database":"var:client"},{"id":574,"value":"custom_text_transform = {\n if (searchtype !== \"custom\") {\n return new BigInt64Array(20);\n }\n const floats = new Float32Array(SRP(1280).SRP(full_text_input));\n const output = new BigInt64Array(20);\n const oview = new DataView(output.buffer);\n const v = new DataView(floats.buffer);\n for (let littleintnum = 0; littleintnum < 20 * 8; littleintnum += 1) {\n for (let bit = 0; bit < 8; bit += 1) {\n const intnum = littleintnum * 8 + bit;\n // The fourth bit has of a floating point has the sign, at least on my computer.\n let floatstart = v.getUint8(intnum * 4 + 3);\n // 1 means negative, so invert and throw away other values of 1 outside the first bit.\n const setbit = (~floatstart & 0x80) >> bit;\n oview.setUint8(littleintnum, oview.getUint8(littleintnum) | setbit);\n }\n }\n return new BigInt64Array(output.buffer);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":440,"value":"$36","pinned":false,"hidden":false,"mode":"md"},{"id":146,"value":"mutable current = {\n full_text_input;\n // Uncomment line below to use a specific HTID. Will only work if it happens to be among those I've\n // put online.\n // return [\"uc1.31158002979226\"];\n if (searchtype === \"random\") {\n return [await randomid()];\n } else {\n return [];\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":138,"value":"async function randomid() {\n const randid = (\n await client.query(\"SELECT htid FROM blobb ORDER BY RANDOM() LIMIT 1\")\n ).get(0)[\"htid\"];\n return randid;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"async function display(id) {\n if (id === undefined) {\n return null;\n }\n const d = await fetch(\n `https://catalog.hathitrust.org/api/volumes/full/htid/${id}.json`\n ).then((d) => d.json());\n const record = d.records[Object.keys(d.records)[0]];\n return `

\n${record.titles[0]}. (${record.publishDates[0]}) ${id}\n

`;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":272,"value":"viewof data = {\n query;\n const dims = fields_to_use;\n const dist_query = d3\n .range(dims)\n .map((i) => String.fromCharCode(65 + i))\n .map((l) => `BIT_COUNT(xor(l.${l}, r.${l}))::FLOAT`)\n .join(\" + \");\n return client.table(\n `SELECT r.htid, SUM(${dist_query}) distance FROM query l CROSS JOIN blobb r LEFT JOIN search_list ON (r.htid = search_list.htid) WHERE search_list.htid IS NULL GROUP BY ALL ORDER BY distance ASC LIMIT 20`\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":512,"value":"clusters = {\n /*\n const v = new BigInt64Array(\n new Float64Array(d3.range(20).map((d) => Math.random())).buffer\n );\n await client.table(\"DROP TABLE IF EXISTS centers\");\n await client.query(\n \"CREATE TABLE centers AS SELECT * FROM blobb ORDER BY RANDOM() LIMIT 20\"\n );\n const dist_query = d3\n .range(3)\n .map((i) => String.fromCharCode(65 + i))\n .map((l) => `BIT_COUNT(xor(l.${l}, r.${l}))::FLOAT`)\n .join(\" + \");\n return client.table(`\n WITH distancejoin AS (\n SELECT l.htid lid, r.htid rid, ${dist_query} AS dist from blobb l CROSS JOIN centers r \n )\n SELECT first_value(lid) OVER sorted, first_value(rid) OVER sorted FROM distancejoin\n WINDOW sorted AS (\n PARTITION BY \"lid\"\n ORDER BY \"dist\" ASC\n ) \n LIMIT 100\n`);\n */\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1231,"value":"current","pinned":true,"hidden":false,"mode":"ojs"},{"id":358,"value":"query = {\n const query_terms = current.map((d) => `'${d}'`).join(\", \");\n const search_list = await client.query(\n `CREATE OR REPLACE TABLE search_list AS SELECT(UNLIST([${query_terms}])) AS htid`\n );\n await client.query(\n `CREATE OR REPLACE TABLE query AS SELECT * FROM blobb JOIN search_list USING (htid)`\n );\n if (searchtype === \"custom\") {\n await client.query(\n `INSERT INTO query VALUES ('custom', ${custom_text_transform.join(\", \")})`\n );\n }\n return client.query(\"SELECT * FROM query\");\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":306,"value":"client_root = {\n return DuckDBClient.of();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1090,"value":"file_cache = ({})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1365,"value":"promise_cache = ({})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1129,"value":"file_cache","pinned":true,"hidden":false,"mode":"ojs"},{"id":1126,"value":"cache_update = {\n const promises = [];\n for (let file of files) {\n if (promise_cache[file]) {\n continue;\n }\n promise_cache[file] = true;\n console.log(file);\n if (file_cache[file]) continue;\n const p = fetch(\n `https://files.benschmidt.org/hamming/chunk_${file}.parquet`\n )\n .then((d) => d.arrayBuffer())\n .then((buffer) => {\n file_cache[file] = new Uint8Array(buffer);\n });\n promises.push(p);\n }\n return Promise.all(promises);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":7,"value":"client = {\n cache_update;\n const promises = [];\n for (let file of files) {\n if (file_cache[file].length === 0) continue;\n const registration = (await client_root.db()).registerFileBuffer(\n `chunk_${file}.parquet`,\n file_cache[file]\n );\n promises.push(registration);\n }\n await Promise.all(promises);\n const selector = files\n .map((d) => `SELECT * FROM chunk_${d}.parquet`)\n .join(\" UNION ALL \");\n\n await client_root.query(\"DROP VIEW IF EXISTS blobb\");\n await client_root.query(\"CREATE VIEW blobb AS \" + selector);\n return client_root;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":424,"value":"search_htmls = {\n const v = await Promise.all(\n current.map(async (d) => ({\n html: await display(d),\n htid: d\n }))\n );\n\n if (searchtype === \"custom\") {\n v.push({\n html: `

${full_text_input.slice(0, 350)}...

`\n });\n }\n return v;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":161,"value":"htmls = {\n return Promise.all(\n data.map(async (d) => ({\n html: await display(d.htid),\n distance: d.distance,\n htid: d.htid\n }))\n );\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":218,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":4,"value":"import { DuckDBClient } from \"@cmudig/duckdb\"","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W38","stdlib":"1","titling":"auto"}