2d:null 2e:["$","title",null,{"children":"Leung Man Tao: \"Naai Gong\" or \"Gang Du\"｜梁文道是“奶共”還是“港獨” | John C | Observable"}] 33:T1173,html`

This is a technical experiment of using Markov model with a Chinese tokenizer. The source text is written by Leung Man Tao (梁文道). A writer and critic who is active in Mainland China, Hong Kong and Taiwan. He was attacked by some people from both Hong Kong and Mainland China for his neutral stand in the social movement in 2019. That's very interesting for me because he's the only one I know that received attack from both regions. So I collected his essays published in 2019 (the ones in traditional Chinese are published in Hong Kong, the ones in simplified Chinese are published in Mainland China), tokenize them with my Chinese tokenizer and put them into the same Markov model to try to generate new paragraph of text every sometime.

Most of Leung's articles in 2019 are about politics and culture, but after going through the Markov model, the generated text appears to be, somehow, meaningless, absurd, confusing, and ridiculous, reflecting on the fact that the Internet is deconstructing serious discussions. That is not necessarily a good thing or necessarily a bad thing. It all depends on how we react. With the deconstruction, can we jump out of the existing perceptions and stereotypes? And after the deconstruction, can we find a way back to the original question itself and settle down an answer?

Inspired by one essay in the input about Cantonese, I try to change some words in the output to words that sounds similar in Mandarin (change to simplified Chinese) or Cantonese (change to traditional Chinese) once in a while, to see how the output changes slightly through time.

What happened to the text is similar to what happened to Leung (and others) in real life: people quote, then 'interpret' and twist, what he wrote/said to support their opinions on him, then A's twisted quotation is quoted by B, and B's by C... So the meaning of the words in their quotation goes further and further away from the original idea that the writer wanted to express.

And sadly, that's quite a common phenomenon in Chinese cyberspace, in both Hong Kong and Mainland China, nowadays.

Pronunciation data collected from open Cantonese dictionary, CC-Canto database and CC-CEDICT database, pre-processed by me.

這是一個將Markov模型與中文分詞器結合使用的技術實驗。源文本來自梁文道，一位活躍於中國大陸，香港和台灣的作家和評論家。由於他在2019年的社會運動的中立立場，他同時遭到了部分香港人和部分內地人的攻擊。我覺得這十分有趣，因此，我收集了他在2019年發表的文章（繁體中文的文章是在香港發表的文章，簡體中文的文章是在內地發表的文章），用我的中文分詞器分析它們，並把它們放入一個Markov模型中，來嘗試生成新的文字。

梁文道在2019年發表的大多數文章都是關於政治和文化的。但是經過Markov模型的重塑之後，所生成的文本在某種程度上是無意義，荒謬，令人困惑和可笑的。這是互聯網消解嚴肅討論這一事實的影射。這一事實不一定是好的，也不一定是壞的。這完全取決於我們的反應。通過解構的過程，我們可以跳出現有的觀念和陳規嗎？而在解構之後，我們能否回到問題本身並找到一個答案？

受到源文本中一篇討論廣東話和普通話的文章的啟發，我嘗試不時地將生成的文字中的某些字詞更改為普通話中的近/同音字詞（該字詞會變為簡體中文）或廣東話中的近/同音字詞（該字詞會變為繁體中文），以使得生成的文字漸漸地隨時間變化。

我覺得這件作品中發生的事情與現實中發生在梁文道（和其他人）身上的事情類似：人們引用，並「解釋」與扭曲，他寫/說的內容，來支持他們對他的看法，然後乙引用甲的引文，而丙引用乙的引文……所以他們口中的作者的話語的意義與作者本身想表達的意義越行越遠。

而令人難過的是，如今，在香港和中國大陸的中文網絡空間中，這都是相當普遍的現象。

讀音數據收集自粵語開放詞典，CC-Canto和CC-CEDICT，經過處理

`34:Tc82,generateChineseSentences = function(model, nValue, noOfSentence, startWith, sth=true, includeStartWith=true, strictStart = true) { // helpers----------------------- let getRandChild = function(parentNode) { let sum = 0; for (let [word,child] of Object.entries(parentNode.ch)) { sum += child.c; } let rand = Math.random()*sum; for (let [word,child] of Object.entries(parentNode.ch)) { rand -= child.c; if (rand < 0) { return child; } } return; } let findMatch = function(toMatch=[]){ let returnedNode = model; while (toMatch.length > 0) { let w = toMatch.shift(); if (!returnedNode.ch[w]) return null; returnedNode = returnedNode.ch[w]; } return returnedNode; } let returnRandEle = function(arr){ let sum = arr.reduce((acc, node) => acc+node.c, 0); let rand = Math.random()*sum; return arr.find(node => (rand -= node.c) < 0); } //------------------------------ const CHINESE_END_OF_SENTENCE = /[。？！…]/ //->TODOs: improve end of sentence condition let startToken; if (startWith && typeof startWith === 'string' && startWith.length > 0){ let pp = findMatch(chineseTokenizer.tokenize(startWith)); if (pp !== null) { startToken = getRandChild(pp); } else if (pp === null && sth) { throw Error(`No sentence starts with "${startWith}" in the model`); } else if (pp ===null && !strictStart){ let tem = chineseTokenizer.tokenize(startWith); while (pp === null && tem.length > 0) { tem.shift(); pp = findMatch(tem); } if (!pp || pp.w === null) { let tochoose = Object.values(model.ch).filter(node => CHINESE_END_OF_SENTENCE.test(node.w)) let p = returnRandEle(tochoose) startToken = getRandChild(p); } } else if (pp === null) { let tochoose = Object.values(model.ch).filter(node => CHINESE_END_OF_SENTENCE.test(node.w)) let p = returnRandEle(tochoose) startToken = getRandChild(p); } } else { let tochoose = Object.values(model.ch).filter(node => CHINESE_END_OF_SENTENCE.test(node.w)) let p = returnRandEle(tochoose) startToken = getRandChild(p); } let resultArray = startWith && includeStartWith ? chineseTokenizer.tokenize(startWith).map(o => o.text) : chineseTokenizer.tokenize(startToken.w).map(o => o.text); let sentenceCount = 0; let currentToken = startToken; while (sentenceCount < noOfSentence) { let nextToken; if (Object.entries(currentToken.ch).length != 0) { nextToken = getRandChild(currentToken); } else { let pre = nValue > 1 ? resultArray.slice(-(nValue - 1)) : [];//fix n = 1 let tem = findMatch(pre); while (pre.length > 0 && tem === null) { pre.shift(); tem = findMatch(pre); }// if can't find sequence of n-1 tokens, try sequence of n-2 tokens ... if (tem === null) tem = model;// if reach the very end of the input, start again nextToken = getRandChild(tem); } if (CHINESE_END_OF_SENTENCE.test(nextToken.w)) sentenceCount++; resultArray.push(nextToken.w); currentToken = nextToken; } return resultArray; }35:T13c7,postProcess = function(array) { const OPENRE = /[‘“「『]/; const ENDRE = /[’”」』]/; const ENDS = /[。？！…]/; const LBRE = /^ *[$\{\[【（] *$/; const RBRE = /[$\]\}】）]/; const KOPENRE= /[《〈]/; const KENDRE= /[》〉]/; const PUNCT=/[。，；、：？！…—]/; const ENGRE=/^ *[A-Za-z0-9\- ]+ *$/; const SINGLECAP = /^ *[A-Z] *$/; const ALLNORE=/^ *[0-9]+ *$/; let result = []; let lbopenedWith = undefined; let openedWith = undefined; let kopenedWith = undefined; const pair = { '“': '”', '「': '」', '『': '』', '‘': '’' }; const bpair = { '(' : ')', '[' : ']', '{' : '}', '【' : '】', '（' : '）' }; const kpair = { '《': '》', '〈': '〉' }; for (let i = 0; i < array.length; i++) { let currentToken = array[i]; if (i == 0 && PUNCT.test(currentToken)){ continue; } if (openedWith == undefined && ENDRE.test(currentToken)) { //no open close quo continue; } if (openedWith == undefined && OPENRE.test(currentToken)) { //first open quo openedWith = OPENRE.exec(currentToken)[0].trim(); result.push(currentToken); continue; } if (openedWith != undefined && OPENRE.test(currentToken)) { // result.push(pair[openedWith]); // result.push('，'); // result.push(currentToken); // openedWith = OPENRE.exec(currentToken)[0].trim()currentToken.trim(); continue } if (openedWith != undefined && ENDS.test(currentToken)) { //end of sentence without close if (lbopenedWith == undefined && kopenedWith == undefined) { result.push(currentToken); result.push(pair[openedWith]); openedWith = undefined; continue; } else if (kopenedWith==undefined){ result.push(bpair[lbopenedWith]); result.push(currentToken); result.push(pair[openedWith]); openedWith = undefined; lbopenedWith = undefined; continue; } else if (lbopenedWith==undefined){ result.push(kpair[kopenedWith]); result.push(currentToken); result.push(pair[openedWith]); openedWith = undefined; kopenedWith = undefined; } else { result.push(kpair[kopenedWith]); result.push(bpair[lbopenedWith]); result.push(currentToken); result.push(pair[openedWith]); openedWith = undefined; lbopenedWith = undefined; kopenedWith = undefined; } continue; } if (openedWith != undefined && ENDRE.test(currentToken)) { result.push(pair[openedWith]); openedWith = undefined; continue; } //above quo //below brackets if (lbopenedWith == undefined && RBRE.test(currentToken)) { continue; } if (lbopenedWith == undefined && LBRE.test(currentToken)) { lbopenedWith = LBRE.exec(currentToken)[0].trim(); result.push(currentToken); continue; } if (lbopenedWith != undefined && LBRE.test(currentToken)) { result.push(bpair[lbopenedWith]); result.push('，'); result.push(currentToken); lbopenedWith = LBRE.exec(currentToken)[0].trim(); continue; } if (lbopenedWith != undefined && ENDS.test(currentToken)) { result.push(bpair[lbopenedWith]); result.push(currentToken); lbopenedWith = undefined; continue; } if (lbopenedWith != undefined && RBRE.test(currentToken)) { result.push(bpair[lbopenedWith]); lbopenedWith = undefined; continue; } //below 《〈〉》 if (kopenedWith == undefined && KENDRE.test(currentToken)) { continue; } if (kopenedWith == undefined && KOPENRE.test(currentToken)) { kopenedWith = KOPENRE.exec(currentToken)[0].trim(); result.push(currentToken); continue; } if (kopenedWith != undefined && KOPENRE.test(currentToken)) { result.push(kpair[kopenedWith]); result.push('，'); result.push(currentToken); kopenedWith = KOPENRE.exec(currentToken)[0].trim(); continue; } if (kopenedWith != undefined && PUNCT.test(currentToken)) { result.push(kpair[kopenedWith]); result.push(currentToken); kopenedWith = undefined; continue; } if (kopenedWith != undefined && KENDRE.test(currentToken)) { result.push(kpair[kopenedWith]); kopenedWith = undefined; continue; } // eng if (ENGRE.test(currentToken) && !ALLNORE.test(currentToken)) { if (SINGLECAP.test(currentToken) && ENGRE.test(array[i+1]) && !ALLNORE.test(array[i+1])){ result.push(" " + currentToken.trim() + array[i+1].trim() + " "); i++; continue; } else { result.push(" " + currentToken.trim() + " "); continue; } } result.push(currentToken); if (ENDS.test(currentToken)) { openedWith = undefined; lbopenedWith = undefined; kopenedWith = undefined; } } return result; }36:T730,changeWords = function(currentArray){ if (!currentArray) return []; let newArray = currentArray.map(e => e.replace(/<(span class="changeViaPinyin"|span class="changeViaJyutpin"|\/span)>/g, ""));//clean css let usedbyPinyin; let fail = []; //pinyin let failtime = 0; let pinyinIdx = randInt(currentArray.length); let newWord = findSimilarByPinyin(currentArray[pinyinIdx]); while (((newWord == null || newWord.length < 2) && failtime < 90) || (newWord == null && failtime < 100)) { fail.push(pinyinIdx); failtime ++; pinyinIdx = randInt(currentArray.length) while (fail.includes(pinyinIdx) && fail.length < Math.min(100,currentArray.length)) { pinyinIdx = randInt(currentArray.length); } newWord = findSimilarByPinyin(currentArray[pinyinIdx]); } if (newWord != null) { newArray[pinyinIdx] = "" + newWord + ""; //add css animation usedbyPinyin = pinyinIdx } // //reset fail = []; if (usedbyPinyin != undefined || usedbyPinyin != null) fail.push(usedbyPinyin) failtime = 0; //jyutpin let jyutpinIdx = randInt(currentArray.length); while (fail.includes(jyutpinIdx)){ jyutpinIdx = randInt(currentArray.length); } newWord = findSimilarByJyutpin(currentArray[jyutpinIdx]); while (((newWord == null || newWord.length < 2) && failtime < 90) || (newWord == null && failtime < 100)) { fail.push(jyutpinIdx); failtime ++; jyutpinIdx = randInt(currentArray.length); while (fail.includes(jyutpinIdx) && fail.length < Math.min(100,currentArray.length)) { jyutpinIdx = randInt(currentArray.length); } newWord = findSimilarByJyutpin(currentArray[jyutpinIdx]); } if (newWord != null) newArray[jyutpinIdx] = "" + newWord + "";//add css return newArray; }37:Tda3,findSimilarByPinyin = function(input){ const memorySize = 100; if (typeof input != 'string' || input.length == 0) return; if (!pyDict.hasOwnProperty(input)) return null; let pinyin = pyDict[input].p; if (typeof pinyin != 'string' || pinyin.length == 0) return null; pinyin = pinyin.replace(/[0-9]/g, "").trim(); let result = undefined; //same pronouncation if (pyDict.hasOwnProperty(pinyin)) { let list = pyDict[pinyin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (pyDict[chosen].s != input && chosen!=input && !recentReplacedPinyin.includes(chosen)) { result = chosen; recentReplacedPinyin.push(chosen); if(recentReplacedPinyin.length > memorySize) recentReplacedPinyin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return pyDict[result].s ? pyDict[result].s : result; //similar pronouncation -> changing one character's initial let pinyinArray = pinyin.split(' '); if (pinyinArray.length > 4) return null; // too long for (let i = 0; i < pinyinArray.length; i++){ let characterToChange = pinyinArray[i].trim(); let toChose = ['zh','ch','sh'].concat('bpmfdtnlgkhjqxzcs'.split()); if (/^[zsc]h/.test(characterToChange)) { for (let op of toChose) { let old = characterToChange; characterToChange.replace(/^[zcs]h/, op); if (characterToChange != old) { //got new one, try find let newPinyin = pinyinArray.slice(0,i); newPinyin.push(characterToChange); newPinyin = newPinyin.concat(pinyinArray.slice(i+1)); if (pyDict.hasOwnProperty(newPinyin)) { let list = pyDict[newPinyin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (pyDict[chosen].s != input && chosen!=input && !recentReplacedPinyin.includes(chosen)) { result = chosen; recentReplacedPinyin.push(chosen); if(recentReplacedPinyin.length > memorySize) recentReplacedPinyin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return pyDict[result].s ? pyDict[result].s : result; } } } else { for (let op of toChose) { let old = characterToChange; characterToChange.replace(/^[a-z]/, op); if (characterToChange != old){ //got new one, try find let newPinyin = pinyinArray.slice(0,i); newPinyin.push(characterToChange); newPinyin = newPinyin.concat(pinyinArray.slice(i+1)); if (pyDict.hasOwnProperty(newPinyin)) { let list = pyDict[newPinyin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (pyDict[chosen].s != input && chosen!=input && !recentReplacedPinyin.includes(chosen)) { result = chosen; recentReplacedPinyin.push(chosen); if(recentReplacedPinyin.length > memorySize) recentReplacedPinyin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return pyDict[result].s ? pyDict[result].s : result; } } } } return null }38:Td30,findSimilarByJyutpin = function(input){ const memorySize = 100; if (typeof input != 'string' || input.length == 0) return; if (!jpDict.hasOwnProperty(input)) return null; let jyutpin = jpDict[input].j; if (typeof jyutpin != 'string' || jyutpin.length == 0) return null; jyutpin = jyutpin.replace(/[0-9]/g, "").trim(); let result = undefined; //same pronouncation if (jpDict.hasOwnProperty(jyutpin)) { let list = jpDict[jyutpin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (chosen != input && jpDict[chosen].s != input && !recentReplacedJyutpin.includes(chosen)) { result = chosen; recentReplacedJyutpin.push(chosen); if (recentReplacedJyutpin.length > memorySize) recentReplacedJyutpin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return result; //similar pronouncation changing one character's initial let jyutpinArray = jyutpin.split(' '); if (jyutpinArray.length > 4) return null; // too long for (let i = 0; i < jyutpinArray.length; i++){ let characterToChange = jyutpinArray[i].trim(); let toChose = ['gw','kw','ng'].concat('bpmfdtnlgkhwsjzc'.split()); if(/^([gk]w|ng)[a-z]+/.test(characterToChange)){ for (let op in toChose){ let old = characterToChange; characterToChange = characterToChange.replace(/^[a-z][a-z]/, op); if (characterToChange != old) { let newJyutpin = jyutpinArray.slice(0,i); newJyutpin.push(characterToChange); newJyutpin.concat(jyutpinArray.slice(i+1)); if (jpDict.hasOwnProperty(newJyutpin)) { let list = jpDict[newJyutpin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (chosen != input && jpDict[chosen].s != input && !recentReplacedJyutpin.includes(chosen)) { result = chosen; recentReplacedJyutpin.push(chosen); if (recentReplacedJyutpin.length > memorySize) recentReplacedJyutpin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return result; } } } else { for (let op in toChose){ let old = characterToChange; characterToChange = characterToChange.replace(/^[a-z]/, op); if (characterToChange != old) { let newJyutpin = jyutpinArray.slice(0,i); newJyutpin.push(characterToChange); newJyutpin.concat(jyutpinArray.slice(i+1)); if (jpDict.hasOwnProperty(newJyutpin)) { let list = jpDict[newJyutpin]; let chosen = list[randInt(list.length)]; for(let i = 0; i < 100; i++) { if (chosen != input && jpDict[chosen].s != input && !recentReplacedJyutpin.includes(chosen)) { result = chosen; recentReplacedJyutpin.push(chosen); if (recentReplacedJyutpin.length > memorySize) recentReplacedJyutpin.shift(); break; } else { chosen = list[randInt(list.length)]; } } } if(result !== undefined) return result; } } } } return null }39:Tdf5,mainStyle = html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":21,"value":"import {chineseTokenizer} from '@real-john-cheung/chinese-tokenizer'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":28,"value":"import {Range} from \"@observablehq/inputs\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3a","stdlib":"1","titling":"auto"}} 32:"$31:body"