34:null 35:["$","title",null,{"children":"How to split text into sentences? | WinkJS | Observable"}] 3b:[] 39:{"id":"d254aeb3d88363a8","slug":"how-to-split-text-into-sentences","title":"How to split text into sentences?","thumbnail":"11e53e84603030701c0ae4002168f25e6d95a110a1875894346d0c4f672d5332","access_level":"public","create_time":"$D2021-06-25T07:35:20.486Z","update_time":"$D2021-07-07T05:26:29.173Z","view_time":null,"first_public_time":"$D2021-06-25T09:20:41.628Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"062de6e3e5a0d590","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":"starter_2024","bio":"Developer friendly Natural Language Processing ✨","following":false},"version":106,"publish_version":106,"role":null,"authored":false,"authors":[{"type":"individual","id":"c6b145bf0197f1e4","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":null,"bio":"Developer friendly Natural Language Processing ✨","following":false}],"members":[{"type":"individual","id":"c6b145bf0197f1e4","login":"winkjs","name":"WinkJS","avatar_url":"https://avatars-next.observableusercontent.com/bf53408604fdcc88d0d3dec6ca4b4b6583ed36cd57fa9107cfea284a28d0fd52","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":6,"collection":{"id":"b1cd242ccf4e30b3","slug":"winknlp-recipes","title":"WinkNLP Recipes","owner":{"id":"062de6e3e5a0d590","login":"winkjs","name":"WinkJS"}},"type":"notebook","body":{"titling":"auto","title":"How to split text into sentences?","cells":[{"id":0,"value":"# How to split text into sentences?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":2,"value":"md`After tokenization, splitting a text into sentences is a key step towards making sense out of the text i.e. the natural language understanding. It is also usually the foundation for other NLP tasks such as negation handling or pos tagging. Sentence boundary identification, according to [Wikipedia](https://en.wikipedia.org/wiki/Sentence_boundary_disambiguation), can be challenging due to the potential ambiguity of punctuation marks; about 47% of the periods in the Wall Street Journal corpus denote abbreviations and not the end-of-sentence. Emoticons can further complicate the problem. [WinkNLP](https://github.com/winkjs/wink-nlp) allows you to easily split text into sentences, on which down stream NLP tasks can be performed.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":82,"value":"md`### Example Text`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":45,"value":"text = `Sen. Edward Kennedy (D., Mass.) said, \"It's a bottom-line issue\".\nThe Nasdaq 100 rose 7.08 to 445.23. (Are parenthesis part of a sentence?)\n\"This is a quoted... sentence.\" \"(This is a quoted sentence within parenthesis.)\"\n('Like the previous one!') AI Inc. is focussing on AI. I work for AI Inc. \nMy mail is r2d2@yahoo.com! U.S.A is my birth place.`;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":85,"value":"md`### Split into sentences`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":49,"value":"{\n const doc = nlp.readDoc( text );\n // Place every sentence in a new row of the table by using .markup() api.\n doc.sentences()\n .each((e) => e.markup('', ''));\n // Return as HTML.\n return html`${''+doc.out(its.markedUpText)+'

Sentences

'}`;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":18,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":28,"value":"md`### Setup`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":31,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":37,"value":"nlp = winkNLP( model );","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":40,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":78,"value":"md`---\n### Styles`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":64,"value":"html`\n `","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":22,"value":"md`---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W3b","stdlib":"1"}} 3a:"$39:body"