2a:null 2b:["$","title",null,{"children":"Tutorial: Generating with n-grams | Daniel Howe | Observable"}] 31:T555,html`If you're not yet familiar with RiTa, check out the website here ...

Other Resources

The RiTa reference in observable
An n-gram demo: phrases annotated with their sources
A tutorial on RiScript, RiTa's micro scripting language
Tools for RiTa in observable, including the rs template
Using RiScript and RiTa Grammars within Twine
RiTa's Lexicon and Letter-to-Sound engine
Generating with RiTa grammars

thanks to @real-john-cheung for porting this tutorial to Observable

`32:T8c6,html`
Again, we start from the left, at “The”, but after we get to “girl”, we have two choices about where to go next. Depending on which transition we take, we will end up with one of two different sentences. Here the probability of choosing either sentence is 50% or 0.5. The same idea can be further extended to a sequence of syllables, letters, words or sentences.

Now let’s change the sentences to make things slightly more interesting:

The girl went to a game after dinner.
The teacher went to dinner with a girl.

The word “went” can occur after either “girl” or “teacher”. The word (or token) that follows “girl” can be either “.” or “went”.
If we consider all the word pairs for the above two sentences (n=2), the outcome would look something like this (you can think of the → symbol as 'can be followed by' and the | symbol as OR).

The → girl | teacher

girl → went | .

went → to

to → a | dinner

a → game | girl

game → after

after → dinner

dinner → . | with

. → The

teacher → went

with → a

This way of looking at contiguous sequences of text, with a specific size (in our case, two words at a time), is known as an n-gram or Markov chain. And this idea can be extended to create models from sequences of arbitrary length, whether syllables, letters, words or sentences.
`33:T696,html`
Generating with n-grams

Generally n-grams are used as a means of analysing language, but here we are interested in generating new text. To do this, we can think of the table (or model) above as an action guide that the computer can follow to create new sentences.

For example, lets say the computer begins with the word “The”. Then it checks the guide for words that can be followed by “The”, and finds two options: “girl” and “teacher”. It flips a coin and picks “girl”. Then it checks for words that follow “girl”, and finds two options: “went” or “.” And so on...

With a longer text sample we could experiment with different n-values, looking at, say... 3 or 4 words at a time. The higher the n-value, the more similar the output will be to the input text (and the more processing/memory required). Often the easiest way to find the best value for n is just to experiment...

Here is an n-gram demo (using a corpus of contemporary prose poems), where each generated phrase is annotated with its source location and context.
`34:T641,html`
The RiMarkov Object

Generating text via n-grams with the RiMarkov object (included in the RiTa library) requires three steps:

First, we construct a model and set its n-value (the # of elements to consider). Let’s start with n=3...
${md`~~~js rm = RiTa.markov(3); ~~~`}

Second, we provide some text for RiTa to analyze. There are two functions to achieve this: fromJSON() and addText(). Let's start with addText(), which simply loads a string of text into the model.
${md`~~~js rm.addText("The girl went to a game after dinner.\\ The teacher went to dinner with a girl."); ~~~`}

Third, we generate from the Markov model. One can use generate() with different options to achieve different types of results. Below we simply ask for two sentences. Here are all three steps together:
${md`~~~js rm = RiTa.markov(3); rm.addText("The girl went to a game after dinner.\\ The teacher went to dinner with a girl."); sentences = rm.generate(2); ~~~`}

If we were to run this code, we would get an output like:

The teacher went to dinner.
The girl went to dinner with a game after dinner.

To get a better sense of how it all works, check out this example sketch which uses RiTa and p5.js to load and blend two different texts...

`35:Tdc1,advanced = html`
Advanced Options

The RiMarkov object includes some advanced options to give you more control over the generation process:

disableInputChecks

A potential problem with Markov-chains is that they may often include potentially long sequences of a input text in the output. Since we are trying to generate new text, this is not ideal. So, by default, RiMarkov will NOT generate any complete sentences which are found in the input. This mean that, at least at the sentence level, all the output from your model will be novel. If you wish to disable this constraint, simply pass the constructor option { disableInputChecks: true } when creating your RiMarkov object.

maxLengthMatch

This constructor option gives you further control over the degree to which input tokens are allowed to appears in the output. By definition, sequences of N tokens that the model generates exist in the input text(s). However the model may generate longer sequences which are also from the input; for example, five sequential words from the input when N=3. To control this, you can set the maxLengthMatch option, which specifies the maximum # of sequential tokens allowed from an input text. If, in the previous example, you set { maxLengthMatch: 4 }, then the model will reject any path of five or more tokens in the input. You can think of this option as a way to minimize overfitting of the model to the input data.

temperature

The temperature option for RiMarkov.generate() function acts like a knob to adjust the probability that input elements are chosen for the output. If no value is provided for temperature, then tokens are chosen in proportion to their relative frequency in the input (this is the standard way that Markov chains work). At higher temperature values, however, tokens that are infrequent in the input are more likely to be chosen, which increase the density of unusual (or at least less commonly-used) tokens in your output. Conversely, with lower temperature values the most frequent inputs are increasingly likely to be selected for the output.

Range: between 0 and positive infinity (excluding both)
Default: exactly mapped to frequencies in the input
Lower values move the highest-weighted output toward a probability of 1.0.
Higher values tend to even out all probabilities (toward random selection).

The temperature parameter functions similarly to its use with softmax in neural networks. See this page for a more in-depth explanation.

startTokens

If you would like the output to begin with a certain word of phrase, you can set the startTokens option which will then start each result with that text when calling RiMarkov.generate().

You can check the rest of the options for creating and generating with the RiMarkov object here: RiTa.markov() and here: RiMarkov.generate().

`36:[{"id":"575cfb6d127d4f5c30ae3a45062a69bfa1809817ad0cd54a9ed0f882016d4b5b00eca25850a0f4b60f3bad04ad904532708c49797979dccd49e6c44e8e2cf3fc","href":"https://static.observableusercontent.com/files/575cfb6d127d4f5c30ae3a45062a69bfa1809817ad0cd54a9ed0f882016d4b5b00eca25850a0f4b60f3bad04ad904532708c49797979dccd49e6c44e8e2cf3fc","name":"markov1.png","type":"image/png","lastModified":1615038123900.485,"size":12311},{"id":"505d4ccf573dc6e143ccbb11670d7bc11eff84d13dc4cbe85fccd586641a533dff749009f048ba74c618a81a594b4b113be9099c9309e26ff62a66f6287d4928","href":"https://static.observableusercontent.com/files/505d4ccf573dc6e143ccbb11670d7bc11eff84d13dc4cbe85fccd586641a533dff749009f048ba74c618a81a594b4b113be9099c9309e26ff62a66f6287d4928","name":"markov2.png","type":"image/png","lastModified":1615038123900.485,"size":25242},{"id":"1cb214207701e4f7180a95fdba9e968d0a58bf3209aca1087fb3f4b47f3c64335d12b5d676a7d9e604b3f0ecb606a19c9b0922323a150e22fd8fcdb611b0b655","href":"https://static.observableusercontent.com/files/1cb214207701e4f7180a95fdba9e968d0a58bf3209aca1087fb3f4b47f3c64335d12b5d676a7d9e604b3f0ecb606a19c9b0922323a150e22fd8fcdb611b0b655","name":"citiesandhighways.png","type":"image/png","lastModified":1615038123900.485,"size":164077}] 2e:{"id":"2a95ca0d029bd567","slug":"tut-rita-ngrams","title":"Tutorial: Generating with n-grams","thumbnail":"002be0e6112f22e06b2ae34c43bb6520e5f76cc0ebc1421b35979950eda7dcbe","access_level":"public","create_time":"$D2021-03-06T13:42:03.900Z","update_time":"$D2025-02-15T14:13:59.888Z","view_time":null,"first_public_time":"$D2021-03-06T17:43:44.611Z","fork":true,"stars":3,"starred":false,"owner":{"type":"team","id":"636cf123a07db681","login":"dhowe","name":"Daniel Howe","avatar_url":"https://avatars-next.observableusercontent.com/a5081aaa56ae9990aa27642040dc61de3ce8cea67f6f778a5b864e41111ba7a3","tier":"starter_2024"},"version":577,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"32869872eb6f2f82","login":"dhowe","name":"Daniel Howe","avatar_url":"https://avatars-next.observableusercontent.com/a5081aaa56ae9990aa27642040dc61de3ce8cea67f6f778a5b864e41111ba7a3","tier":null}],"members":[{"type":"individual","id":"32869872eb6f2f82","login":"dhowe","name":"Daniel Howe","avatar_url":"https://avatars-next.observableusercontent.com/a5081aaa56ae9990aa27642040dc61de3ce8cea67f6f778a5b864e41111ba7a3","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"subscribed":null,"forks":5,"collection":{"id":"3c04483074fcc200","slug":"rita","title":"RiTa","owner":{"id":"636cf123a07db681","login":"dhowe","name":"Daniel Howe"}},"type":"notebook","body":{"title":"Tutorial: Generating with n-grams","cells":[{"id":0,"value":"html`

Tutorial: Generating with n-grams

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":11,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"html`

Markov Chains

Markov chains (also called n-gram models) can be thought of as systems of states and transitions. An analogy might be a set of cities connected by highways, where each city is a state, and each highway is a transition, a way of getting from one city to another.\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":24,"value":"html`

Since we are concerned here with text, we can think of each state as a bit of text; a letter, a word, or a phrase (usually drawn inside a circle). A transition is a way of moving from one (text) state to another (usually drawn as an arrow). Here’s a simple example:

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"html`

In the image above we see a Markov chain for the sentence: “The girl created a game after school”. We can start at the first word, “The”, and follow the arrows to get to the final word, “school”. In this very simple chain, each word leads to exactly one subsequent word, so there are no choices we need to make.

But consider the following example (with 2 sentences):

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":44,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":54,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":59,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"html`

Appendix

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":320,"value":"html`\n\n`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}} 2f:"$2e:body"

The	→	girl \| teacher
girl	→	went \| .
went	→	to
to	→	a \| dinner
a	→	game \| girl
game	→	after
after	→	dinner
dinner	→	. \| with
.	→	The
teacher	→	went
with	→	a