2d:null 2e:["$","title",null,{"children":"How to do stemming and lemmatization? | WinkJS | Observable"}] 33:[] 31:{"id":"61dffd2bc61889c8","slug":"how-to-do-stemming-and-lemmatization","title":"How to do stemming and lemmatization?","thumbnail":"9342512d602f2747b18469e4cef3ba56f0ddd366686e6e02c48a1f45135546eb","access_level":"public","create_time":"$D2021-07-23T15:36:50.438Z","update_time":"$D2021-07-26T12:24:45.583Z","view_time":null,"first_public_time":"$D2021-07-26T17:18:34.157Z","has_unpublished_edits":null,"fork":false,"stars":9,"starred":false,"owner":{"type":"team","id":"062de6e3e5a0d590","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":"starter_2024"},"version":237,"publish_version":237,"role":null,"authored":false,"authors":[{"type":"individual","id":"c6b145bf0197f1e4","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":null}],"members":[{"type":"individual","id":"c6b145bf0197f1e4","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":7,"collection":{"id":"b1cd242ccf4e30b3","slug":"winknlp-recipes","title":"WinkNLP Recipes","owner":{"id":"062de6e3e5a0d590","login":"winkjs","name":"WinkJS"}},"type":"notebook","body":{"title":"How to do stemming and lemmatization?","cells":[{"id":0,"value":"# How to do stemming and lemmatization?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"[WinkNLP](https://github.com/winkjs/wink-nlp) has a rich set of features for text pre-processing or normalization. These include a stemmer and a lemmatizer. The stemmer is based on [Porter Stemmer algorithm version 2.0](https://snowballstem.org/algorithms/english/stemmer.html). The lemmatizer is inspired by [Wordnet's](https://wordnet.princeton.edu/) lemmatizer. As illustrated in the following example, both the operations can be easily accomplished using [`out()`]() method with [`its`]() helpers:","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":18,"value":"{\n const doc = nlp.readDoc( text );\n let display = init();\n \n display += startRow( '.out()' );\n doc.tokens().each( ( t ) => display += content( t, 'value' ) );\n display += endRow();\n \n display += startRow( '.out(its.lemma)' );\n doc.tokens().each( ( t ) => display += content( t, 'lemma' ) );\n display += endRow();\n \n display += startRow( '.out(its.stem)' );\n doc.tokens().each( ( t ) => display += content( t, 'stem' ) );\n display += endRow();\n \n display += end();\n return html`${display}`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":73,"value":"There are interesting observations about lemma and stem in the above example:\n

Lemmatization returned have as the original form of had, whereas stemmer failed to do so.\n
Stemmer stemmed beautiful to beauti, which is not a valid English word; same thing happened to the word voice.\n
Stemmer converted the singing to sing, whereas lemmatizer did not perform any conversion since it is a noun and not a verb. \n

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":65,"value":"Lemmatization requires morphological analysis of a word in the given context i.e. its part-of-speech. Stemming is a naive way to lemmatize — it does not consider the context of the word and algorithmically replaces affixes. A typical example is to drop the \"-ing\" or \"-ed\" suffix to derive the stem. It may not always yield a proper dictionary word.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":68,"value":"The choice between lemmatization or stemming depends on the context. Consider an application that is using word vectors. Here stemming is not a good choice as it sometimes yields an invalid word, which will not have a vector. With lemmatization this situation will never arise. Performance could be another consideration. Stemming speed is generally faster as it does not depend on part-of-speech. For many applications stemming is good enough — typical example is TF-IDF or BM25 text search.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":23,"value":"md`---\n### Setup`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":26,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":29,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":32,"value":"nlp = winkNLP( model );","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":35,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":81,"value":"text = 'My mother had a beautiful singing voice.';","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":86,"value":"cssClass = {\n let cssc = Object.create( null );\n cssc.had = 'had';\n cssc.beautiful = 'beautiful';\n cssc.voice = 'beautiful';\n cssc.singing = 'singing';\n return cssc;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"init = function () {\n return `

Example sentence

${text}

`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":132,"value":"startRow = function ( h ) {\n return ``;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":102,"value":"content = function ( t, itsType ) {\n return ``;;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":140,"value":"endRow = function () {\n return '';\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":146,"value":"end = function () {\n return '

${h}	${t.out( its[itsType] )}

';\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":213,"value":"md`---\n### CSS`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":159,"value":"html`\n `","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":20,"value":"md`---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W33","stdlib":"1","titling":"auto"}} 32:"$31:body"