2d:null 2e:["$","title",null,{"children":"Tokenization Visualizer | Bairui Su | Observable"}] 33:T645,display(plot( "AI is not a substitute for human understanding. We do not want a chatbot where you can ask a question and get an opaque answer. The answer isn’t the point. The meaning of the answer — its implications, its nuances, its accuracy, its effect on your future assumptions and decisions — is the point. This means an agent should show its work and you should understand its derivation. Large language models are great at generating code; Observable’s job is to make that code more interpretable and malleable. That’s why we care about the computational medium, where you can see the state of a program as its runs, where you can instantly modify running programs, and quickly add interaction or visualization. The medium is where humans and computers work together. AI is not a substitute for human collaboration. We don’t want agents to replace human relationships. Observable is designed for collaboration — to quickly see what others are working on, to contribute to others’ work or riff on it, to reuse code, to ask for or receive feedback — and that extends to chats, too. AI is not a substitute for human creativity. Observable grew out of the nascent community of D3 users, and we continue to be inspired by the creativity and generosity of this community. As magical as it is to have an agent quickly sketch for you, an agent is not a community. A community does not merely suggest code; it provides support, empathy, and compassion. A community builds relationships that go beyond whatever tool you use. Read How D3 moved us for a few of those stories (seriously)." ));34:T41b,function plot(text) { const tokens = RiTa.tokenize(text); const analysis = RiTa.analyze(text); const pos = analysis.pos.split(" "); const I = d3.range(tokens.length); const colors = [...d3.schemeTableau10, ...d3.schemeObservable10]; const types = Array.from(new Set(pos.filter(d => !TYPES.includes(d)))); const domain = [...TYPES, ...types]; // I'm not sure about all the possible values. const color = d3.scaleOrdinal(domain, colors); const chart = Plot.plot({ width, marginLeft: 10, marginRight: 10, marginBottom: 10, x: {axis: "top"}, color: {domain: domain, range: colors}, marks: [ Plot.barX(pos, Plot.groupX({x: "count", sort: "count", fill: "x", reverse: true})), Plot.textX(pos, Plot.stackX(Plot.groupX( {x: "count", sort: "count", text: "x", reverse: true}, {tip: true} ))), ] }); const node = html`${I.map(i => html`${tokens[i]} `)}`; return html`${chart}

${node}

`; }35:[] 31:{"id":"b66dc2ec3dc4671c","slug":"tokenization-visualizer","title":"Tokenization Visualizer","thumbnail":"c718e6656847cce861c2266fa76cf20e2221838d494d9bd34b759e5998910024","access_level":"public","create_time":"$D2026-09-13T19:46:58.552Z","update_time":"$D2026-09-16T14:53:28.602Z","view_time":null,"first_public_time":"$D2026-09-14T22:54:05.177Z","has_unpublished_edits":null,"fork":false,"stars":5,"starred":false,"owner":{"type":"team","id":"3e931206c18366b5","login":"pearmini","name":"Bairui Su","avatar_url":"https://avatars-next.observableusercontent.com/0c55731913f2096b86eadf187f5e21ae0a951f3c4dd6bbde4bf2a8536c5b6e49","tier":"starter_2024","bio":"Creative technologist and researcher building tools for creative coding, data visualization, and HCI. Creator of Recho; previously led AntV G2 5.0."},"version":1003,"publish_version":1003,"role":null,"authored":false,"authors":[{"type":"individual","id":"76ec39dbb0bd8e40","login":"pearmini","name":"Bairui SU","avatar_url":"https://avatars-next.observableusercontent.com/0c55731913f2096b86eadf187f5e21ae0a951f3c4dd6bbde4bf2a8536c5b6e49","tier":null,"bio":"Creative technologist and researcher building tools for creative coding, data visualization, and HCI. Creator of Recho; previously led AntV G2 5.0."}],"members":[{"type":"team","id":"3e931206c18366b5","login":"pearmini","name":"Bairui Su","avatar_url":"https://avatars-next.observableusercontent.com/0c55731913f2096b86eadf187f5e21ae0a951f3c4dd6bbde4bf2a8536c5b6e49","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Tokenization Visualizer","cells":[{"id":1,"value":"# Tokenization Visualizer\n\nThis is a tokenization visualizer that processes a given text using [RiTa’s `tokenize` method](https://rednoise.org/rita/reference/RiTa/tokenize/index.html). The resulting tokens are labeled using the [Penn Treebank POS tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), although RiTa does not strictly follow Penn Treebank tokenization rules. Each token is then visually encoded according to its POS tag, while a stacked bar chart at the top shows the frequency distribution of the tags.\n\nThe following demo visualizes several inspiring opening lines from Sarah Groff-Palermo’s wonderful talk on artisanal data at [Eyeo 2014 Ignite](https://vimeo.com/111211477). The typing animation is adapted from Mike Bostock’s [Fit Text to Circle](https://observablehq.com/@mbostock/fit-text-to-circle).\n\nWait for the animation to finish, or type or paste your own text!","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":210,"value":"viewof replay = Inputs.button(\"Replay\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":206,"value":"viewof textstream = {\n replay;\n const random = d3.randomExponential(1 / 25);\n const value = text;\n const input = html`

`;\n  yield input;\n  for (let i = 0, n = value.length; i <= n; ++i) {\n    if (input.value !== value.substring(0, i - 1)) break;\n    input.value = value.substring(0, i);\n    yield Promises.delay(random(), input);\n  }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":200,"value":"display(plot(textstream));","pinned":false,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":228,"value":"## Example\n\nAnother visualization of Mike Bostock’s [thought-provoking insights on AI](https://observablehq.com/@observablehq/a-new-observable).","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":230,"value":"$33","pinned":false,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":189,"value":"## Appendix A: Implmentation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":2,"value":"const text = \"Most data visualization work tries to gather as many data points as possible, then extract meaning from the noise. In that kind of system, a person often appears only as a demographic point inside a much larger pattern. Groff-Palermo asks whether that is enough. If data is meant to help us understand ourselves, then perhaps we need ways to tell data stories that preserve individuality, emotion, and incompleteness. Her answer is literature. Literature works by inviting readers to identify with specific events happening to specific characters, even when those characters are unlike us. A novel or story is never complete on its own; readers bring imagination, empathy, and interpretation to it. That interaction is what makes the experience feel alive. Groff-Palermo suggests that artisanal data can work in a similar way. It does not need to represent everyone. It can instead present a precise, personal fragment and invite the viewer to enter it imaginatively.\"","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":227,"value":"const TYPES = RiTa.analyze(text).pos.split(\" \");","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":192,"value":"$34","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":74,"value":"import {RiTa} from \"https://esm.sh/rita\";","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":157,"value":"## Appendix B: Experiments","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"import {pipeline} from \"https://cdn.jsdelivr.net/npm/@xenova/transformers/dist/transformers.min.js\"","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"const words = display(text.split(/\\s/));","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"const freq = words.reduce((map, v) => (map[v] = map[v] ? map[v] + 1 : 1, map), {})\ndisplay(freq)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":38,"value":"const classifier = await pipeline(\"sentiment-analysis\")\ndisplay(classifier)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"await classifier(\"beautiful\")","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":46,"value":"const keys = display(Object.keys(freq))","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":48,"value":"const values = display(Object.values(freq))","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":45,"value":"const I = display(Array.from(keys, (_, i) => i))","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":50,"value":"const results = await Promise.all(I.map(i => classifier(keys[i])))\ndisplay(results)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":57,"value":"const labels = results.map(d => d[0].label)\ndisplay(labels)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":69,"value":"const scores = results.map(d => d[0].score)\ndisplay(scores)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":66,"value":"const color = d3.scaleOrdinal(new Set(labels), d3.schemeObservable10)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":62,"value":"const node = html`${I.map(i => html`<span style=\"background-color:${color(labels[i])}\">${keys[i]} </span>`)}`\ndisplay(node)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":75,"value":"const tokens = RiTa.tokenize(text);\ndisplay(tokens);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"const analysis = RiTa.analyze(text);\nconst pos = analysis.pos.split(\" \");\ndisplay(pos);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":87,"value":"const RI = d3.range(tokens.length)\ndisplay(RI)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":118,"value":"const types = Array.from(new Set(pos));\ndisplay(types);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"const color2 = d3.scaleOrdinal(types, d3.schemeObservable10)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":140,"value":"d3.schemeTableau10","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":93,"value":"const chart = Plot.plot({\n  width,\n  color: {legend: true, domain: types},\n  marks: [\n    Plot.barX(pos, Plot.groupX({x: \"count\", sort: \"count\", fill: \"x\", \"reverse\": true})),\n  ]\n});\ndisplay(chart);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":85,"value":"const node2 = html`${RI.map(i => html`<span style=\"background-color:${color2(pos[i])}\">${tokens[i]} </span>`)}`\ndisplay(node2)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":126,"value":"const rainbow = d3.range(types.length).map(i => d3.interpolateRainbow(i / types.length));\ndisplay(rainbow)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":116,"value":"const color3 = d3.scaleOrdinal(types, rainbow);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":132,"value":"const chart2 = Plot.plot({\n  width,\n  color: {legend: true, domain: types, range: rainbow},\n  marks: [\n    Plot.barX(pos, Plot.groupX({x: \"count\", sort: \"count\", fill: \"x\", \"reverse\": true})),\n  ]\n});\ndisplay(chart2);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":124,"value":"const node3 = html`${RI.map(i => html`<span style=\"background-color:${color3(pos[i])}\">${tokens[i]} </span>`)}`\ndisplay(node3)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":146,"value":"const merged = [...d3.schemeTableau10, ...d3.schemeObservable10]\ndisplay(merged)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":149,"value":"const color4 = d3.scaleOrdinal(types, merged);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":151,"value":"const chart2 = Plot.plot({\n  width,\n  color: {legend: true, domain: types, range: merged},\n  marks: [\n    Plot.barX(pos, Plot.groupX({x: \"count\", sort: \"count\", fill: \"x\", \"reverse\": true})),\n  ]\n});\ndisplay(chart2);","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":153,"value":"const node4 = html`${RI.map(i => html`<span style=\"background-color:${color4(pos[i])}\">${tokens[i]} </span>`)}`\ndisplay(node4)","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":161,"value":"{\n  const chart = Plot.plot({\n    width,\n    marginLeft: 10,\n    marginRight: 10,\n    marginBottom: 10,\n    x: {axis: \"top\"},\n    color: {domain: types, range: merged},\n    marks: [\n      Plot.barX(pos, Plot.groupX({x: \"count\", sort: \"count\", fill: \"x\", \"reverse\": true})),\n      Plot.textX(pos, Plot.stackX(Plot.groupX({x: \"count\", sort: \"count\", text: \"x\", \"reverse\": true}))),\n    ]\n  });\n\n  const node = html`${RI.map(i => html`<span style=\"background-color:${color4(pos[i])}\">${tokens[i]} </span>`)}`;\n\n  display(html`\n    ${chart}\n    <div style=\"margin: 0 10px 0 10px\">${node}</div>\n  `);\n}","pinned":true,"hidden":false,"mode":"ts","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W35","stdlib":"2"}}
32:"$31:body"
</article></body></html>