2d:null 2e:["$","title",null,{"children":"N-gram generation Sci-fi text Try | ychsu6 | Observable"}] 33:T17ca,textOriginal = `“Indeed you are happy,” they answered. “How else can men be when they live for their brothers?” And now, sitting here in our tunnel, we wonder about these words. It is forbidden, not to be happy. For, as it has been explained to us, men are free and the earth belongs to them; and all things on earth belong to all men; and the will of all men together is good for all; and so all men must be happy. Yet as we stand at night in the great hall, removing our garments for sleep, we look upon our brothers and we wonder. The heads of our brothers are bowed. The eyes of our brothers are dull, and never do they look one another in the eyes. The shoulders of our brothers are hunched, and their muscles are drawn, as if their bodies were shrinking and wished to shrink out of sight. And a word steals into our mind, as we look upon our brothers, and that word is fear. There is fear hanging in the air of the sleeping halls, and in the air of the streets. Fear walks through the City, fear without name, without shape. All men feel it and none dare to speak. We feel it also, when we are in the Home of the Street Sweepers. But here, in our tunnel, we feel it no longer. The air is pure under the ground. There is no odor of men. And these three hours give us strength for our hours above the ground. Our body is betraying us, for the Council of the Home looks with suspicion upon us. It is not good to feel too much joy nor to be glad that our body lives. For we matter not and it must not matter to us whether we live or die, which is to be as our brothers will it. But we, Equality 7-2521, are glad to be living. If this is a vice, then we wish no virtue. Yet our brothers are not like us. All is not well with our brothers. There are Fraternity 2-5503, a quiet boy with wise, kind eyes, who cry suddenly, without reason, in the midst of day or night, and their body shakes with sobs they cannot explain. There are Solidarity 9-6347, who are a bright youth, without fear in the day; but they scream in their sleep, and they scream: “Help us! Help us! Help us!” into the night, in a voice which chills our bones, but the Doctors cannot cure Solidarity 9-6347. And as we all undress at night, in the dim light of the candles, our brothers are silent, for they dare not speak the thoughts of their minds. For all must agree with all, and they cannot know if their thoughts are the thoughts of all, and so they fear to speak. And they are glad when the candles are blown for the night. But we, Equality 7-2521, look through the window upon the sky, and there is peace in the sky, and cleanliness, and dignity. And beyond the City there lies the plain, and beyond the plain, black upon the black sky, there lies the Uncharted Forest. We do not wish to look upon the Uncharted Forest. We do not wish to think of it. But ever do our eyes return to that black patch upon the sky. Men never enter the Uncharted Forest, for there is no power to explore it and no path to lead among its ancient trees which stand as guards of fearful secrets. It is whispered that once or twice in a hundred years, one among the men of the City escape alone and run to the Uncharted Forest, without call or reason. These men do not return. They perish from hunger and from the claws of the wild beasts which roam the Forest. But our Councils say that this is only a legend. We have heard that there are many Uncharted Forests over the land, among the Cities. And it is whispered that they have grown over the ruins of many cities of the Unmentionable Times. The trees have swallowed the ruins, and the bones under the ruins, and all the things which perished. And as we look upon the Uncharted Forest far in the night, we think of the secrets of the Unmentionable Times. And we wonder how it came to pass that these secrets were lost to the world. We have heard the legends of the great fighting, in which many men fought on one side and only a few on the other. These few were the Evil Ones and they were conquered. Then great fires raged over the land. And in these fires the Evil Ones and all the things made by the Evil Ones were burned. And the fire which is called the Dawn of the Great Rebirth, was the Script Fire where all the scripts of the Evil Ones were burned, and with them all the words of the Evil Ones. Great mountains of flame stood in the squares of the Cities for three months. Then came the Great Rebirth. The words of the Evil Ones... The words of the Unmentionable Times... What are the words which we have lost? May the Council have mercy upon us! We had no wish to write such a question, and we knew not what we were doing till we had written it. We shall not ask this question and we shall not think it. We shall not call death upon our head. And yet... And yet... There is some word, one single word which is not in the language of men, but which had been. And this is the Unspeakable Word, which no men may speak nor hear. But sometimes, and it is rare, sometimes, somewhere, one among men find that word. They find it upon scraps of old manuscripts or cut into the fragments of ancient stones. But when they speak it they are put to death. There is no crime punished by death in this world, save this one crime of speaking the Unspeakable Word. We have seen one of such men burned alive in the square of the City. And it was a sight which has stayed with us through the years, and it haunts us, and follows us, and it gives us no rest. We were a child then, ten years old. And we stood in the great square with all the children and all the men of the City, sent to behold the burning. They brought the Transgressor out into the square and they led them to the pyre. They had torn out the tongue of the Transgressor, so that they could speak no longer. The Transgressor were young and tall. They had hair of gold and eyes blue as morning. They walked to the pyre, and their step did not falter. And of all the faces on that square, of all the faces which shrieked and screamed and spat curses upon them, theirs was the calmest and the happiest face. `34:T55d, root = { // create the root node let root = new Node("ROOT");//create the 最頂端 "ROOT" // loop over each word in the text // cuz it groups the words by every N words for (let j = 0; j < words.length-(N-1); j++) { // look at each sequence of N items, ngram is an Array of string, take a chunk of the words from the "words" array let ngram = words.slice(j, j + N); //an Array, start from the jth stop at the j+Nth(exclude j+N itself) // start with root as the 'current' node let current = root; //currently i'm working on the root node // loop over our N tokens for (let i = 0; i < ngram.length; i++) { //go over each element of the ngram 4 times, 0,1,2,3 let word = ngram[i];//take one element from ngram // does 'current' have this word as child already? if (!current.children[word]) { //if word is not an object in the Node already, which is the current // if not, then we create it current.children[word] = new Node(word);//so connect/add a new Node inside the ROOT } // now we increase the count for the child current.children[word].count++; //say now // now set the child to be the new 'current' current = current.children[word]; // say now the first element of Node children is the current } } return root; }35:[] 31:{"id":"3b569cb1bffa3f9a","slug":"n-gram-generation-sci-fi","title":"N-gram generation Sci-fi text Try","thumbnail":"df729248b084c0b27ba8c12d9da9594c7830f10d1096921837ab7d4556f7bd95","access_level":"public","create_time":"$D2021-05-14T15:20:06.021Z","update_time":"$D2021-05-18T16:38:59.327Z","view_time":null,"first_public_time":"$D2021-05-15T08:18:29.894Z","has_unpublished_edits":null,"fork":true,"stars":0,"starred":false,"owner":{"type":"team","id":"c3d91d7db2867b4d","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":"starter_2024"},"version":618,"publish_version":618,"role":null,"authored":false,"authors":[{"type":"individual","id":"8205ffe96d046f8c","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":null}],"members":[{"type":"individual","id":"8205ffe96d046f8c","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"N-gram generation Sci-fi text Try","cells":[{"id":0,"value":"md`# N-gram generation Sci-fi text Try`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":63,"value":"md`${generate()}${generate()}${generate()}${generate()}${generate()}`//sec, md`${generate()}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":258,"value":"N = 4;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":509,"value":"text = textOriginal.replace(\n /(Sofi)/g, \"I\"\n).replace(\n /(the girl)/g, \"my enemy\"\n).replace(\n /(Henry Ohm|Hen)/g, \"I\"\n).replace(\n /(robot|the machine|machine)/g, \"I\"\n).replace(\n /(machines|robots)/g, \"we\"\n).replace(\n /(Robots)/g, \"We\"\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":466,"value":"preNouns={ // to get a list of Noun from the input text to see what should be replaced\n let w = RiTa.tokenize(text)\n w = w.filter(x=> RiTa.isNoun(x))\n let u = {}; // object\n w.forEach(x=>{u[x]=true})// to remove the duplicated Noun\n return Object.keys(u);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":61,"value":"generate = function(n) {\n let next, result = [];\n let starts = Object.values(root.children).filter(n => /^[.?!]$/.test(n.word));\n let current = root.child('');\n while (!current.isLeaf()) {\n next = weightedChoice(Object.values(current.children));\n result.push(next);\n current = next;\n }\n //if the current Node is not a Leaf then keep doing this, until it becomes a leaf then stop it\n while(!/^<\\/s>$/.test(next.word))\t{\n let path = result.slice(-(N-1));\n current = search(path);\n next = weightedChoice(Object.values(current.children)); \n if (!next) break;\n result.push(next);\n }\n \n return RiTa.untokenize(result.map(n => n.word)).replace(/\"/g,'');\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":514,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":233,"value":"search = function(path) {\n let current = root;\n for (let i = 0; i < path.length; i++) {\n let next = current.child(path[i].word);\t\t\n current = next;\n }\n return current;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":391,"value":"weightedChoice = function(arr) {// get every elemetes of the array, sum is the total of element count\n let sum = arr.reduce((sofar, ele) => sofar + ele.count, 0); // \"(sofar, ele) => sofar + ele.count\" is a function that add ele.count into sofar\n //\"(sofar, ele) => sofar + ele.count\" is the 1st argument of the reduce() function, 0 is the 2nd argument, tell it to start from 0\n let rand = Math.random() * sum; // choose random from 0 ~ sum\n //find is an fuction that has another function as argument to find to get that single element that is bigger than the random number\n return arr.find(ele => (rand -= ele.count) < 0); \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":388,"value":"words = {\n let allWords = []\n let sents = RiTa.sentences(text);//make an array of sentences\n //for each \"s\", that \"s\" is in the array \"sents\"\n sents.forEach(s => {//go through the array and add ~~in the beginning;~~ in the end of the sentence\n let tokens = RiTa.tokenize(s);//tokennise it, tokens is an array\n allWords.push('~~');\n allWords.push(...tokens);//three dots before \"tokens\" in an array means to take all the elements in \"tokens\" arrays as \"Arguments\" in the push() function\n allWords.push('~~');\n });\n return allWords\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":109,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":380,"value":"class Node {\n constructor(word) {\n this.word = word;\n this.count = 0;\n this.children = {};\n }\n child(word) {\n return this.children[word];\n }\n isLeaf() {\n return Object.keys(this.children).length == 0;\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"RiTa = require('rita');","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":243,"value":"sec = pulse(6000)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":219,"value":"import {pulse} from '@dhowe/rita'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":48,"value":"gutenFetch = async function(url) {\n return fetch('https://observable-cors.glitch.me/' + url, {\n headers: {\n Accept: '*/*',\n 'Accept-Language': 'en-US,en;q=0.5',\n Referer: 'https://beta.observablehq.com/',\n },\n cache: 'no-store',\n mode: 'cors',\n redirect: 'follow',\n }).then(response => {\n if (!response.ok) throw new Error(response.status);\n return response.text();\n }).catch(error => {return error})\n};","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"url = 'https://www.gutenberg.org/files/63821/63821.txt'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":335,"value":"html`\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W35","stdlib":"1","titling":"auto"}} 32:"$31:body"