3e:null 3f:["$","title",null,{"children":"How to visualize key sentences in a document? | Penn | Observable"}] 45:T4dd,Automatic identification of important sentences and their effective visualization in a document helps to speed read and gain a quick insight. [**WinkNLP**](https://github.com/winkjs/wink-nlp) can be used to build such a tool easily with the following capabilities: 1. Highlight the key sentences using 3 levels of intensities based on their relative importance in the document. 2. Control the number of sentences to be highlighted using a slider. 3. Toggle the focus mode for better reading experience by fading the balance text. Mouseover on a sentence automatically brightens it. Here is a simple implementation in about 35 lines of code. This implementation is inspired by the hypothesis that *content salience is proportional to the frequency of part-of-speech n-grams* as outlined in the paper titled, [Examining the Content Load of Part of Speech Blocks for Information Retrieval](https://dl.acm.org/doi/10.5555/1273073.1273142). For implementation details refer to **`sentenceWiseNormalizedWeights`** under the setup section below. The [markup()](https://winkjs.org/wink-nlp/visualizing-markup.html) method of [**winkNLP**](https://github.com/winkjs/wink-nlp) has been used to create color intensity coded visualization.
46:Tbf0,text = `Children living in Japan’s hottest city will be given specially designed umbrellas to protect them from the heat, after a summer that saw record-breaking temperatures in many parts of the country.Local authorities in Kumagaya in Saitama prefecture have devised an umbrella that keeps out the rain and doubles as a parasol, the Mainichi Shimbun reported. The umbrellas, which bear the city’s logo and weigh just 336 grams, will be distributed to 9,000 primary schoolchildren next week, the newspaper said. Kumagaya, a city of about 195,000 located 60km north of Tokyo, regularly records the highest temperatures in Japan partly as a result of warm downslope winds created by the Foehn Effect. The city’s government has for the past two years advised younger children to shield themselves from the sun with regular umbrellas on their way to and from school to prevent heatstroke, but some questioned their ability to block out sunlight. Alarmed by a rise in the number of days when the mercury rises to at least 35C, the city decided to hand out the yellow fibreglass umbrellas, including to children who live in Kumagaya but attend schools outside the city, the Mainichi said. The heat-busting brollies will also force children to maintain a reasonable distance from each other, eliminating the need for them to wear masks to prevent the spread of the coronavirus, it added. The measure has come a little late in the day, however. Japan battled its worst heatwave since records began in 1875 in late June, after a premature end to the rainy season. The city of Isesaki, north of Tokyo, registered the country’s highest-ever temperature for that month, at 40.2C, beating the previous June record of 39.8C set in 2011. Tokyo experienced several consecutive days of 35C-plus heat, prompting the government to warn people to save energy or face power cuts, while Kumagaya and five other locations marked highs above 40C on 1 July. Kumagaya’s reputation for furnace-like temperatures was sealed in July 2018, when it battled an all-time high temperature of 41.1C – an unenviable record it shares with the city of Hamamatsu in central Japan. On Friday, the maximum temperature for Kumagaya was a far more comfortable 26C, according to the meteorological agency, although it forecast a rerun to the low 30s next week. Officials had hoped to distribute the umbrellas before the school summer holidays began were delayed by the Covid-19 pandemic. Global heating has prompted Japan’s government to take extra measures and issue a slew of advice on how to prevent heatstroke. Almost all classrooms in public primary and middle schools now have air conditioners, according to the Asahi Shimbun, while the education ministry last year urged teachers to instruct children to wear cool clothing and hats, and to keep hydrated when they travel to and from school. The pandemic has frustrated attempts to keep children cool at school, however, with teachers reporting that many are reluctant to remove their masks, even with encouragement from staff.`;47:Tbd7,sentenceWiseNormalizedWeights = { // Define open class part-of-speeches; used to compute intitial information content const openClassPOS = Object.create(null); openClassPOS.ADJ = true; openClassPOS.ADV = true; openClassPOS.INTJ = true; openClassPOS.NOUN = true; openClassPOS.PROPN = true; openClassPOS.VERB = true; openClassPOS.NUM = true; openClassPOS.SYM = true; // N-gram to use to construct a pos group. const NGram = 4; // Used to build table of weights of pos groups. Apart from frequency, it also maintains // (a) array of sentences, where a given pos group was found, (b) total weight computed as // frequency minus count of closed class part-of-speech in the group. const posGroupWeightTable = Object.create( null ); // Build pos group frequency table. doc.sentences().each( s => { // Ignore punctuation and space (tab, CRLF etc). const pos = s.tokens().out( its.pos ).filter( p => p !== 'SPACE' && p !== 'PUNCT' ); // Ignore sentences where we cannot build NGram i.e. sentences shorter than NGram. if ( pos.length < 4 ) return; // Determine NGrams; for ( let k = 0; k+ NGram - 1 < pos.length; k += 1 ) { const pos4Gram = pos.slice( k, k + NGram ); // Used to compute the weight for a pos group. const initInfoContent = pos4Gram.reduce( ( pv, cv ) => pv - ( ( openClassPOS[cv] ) ? 0 : 1 ), 0 ); const posGroup = pos4Gram.join( '_' ); posGroupWeightTable[ posGroup ] = posGroupWeightTable[ posGroup ] || Object.create( null ); posGroupWeightTable[ posGroup ].group = posGroup; posGroupWeightTable[ posGroup ].sentences = posGroupWeightTable[ posGroup ].sentences || []; posGroupWeightTable[ posGroup ].sentences.push( s.index() ); posGroupWeightTable[ posGroup ].weight = ( posGroupWeightTable[ posGroup ].weight === undefined ) ? initInfoContent + 1 : ( posGroupWeightTable[ posGroup ].weight + 1 ); posGroupWeightTable[ posGroup ].iv = initInfoContent; } } ); // Transform object into an array, and filter out elements with weight <= 0. const posGroupWeights = Object.keys( posGroupWeightTable ) .map( e => posGroupWeightTable[ e ] ) .filter( e => e.weight > 0 ); // This is an array index by each sentence's index and would contain the total weight // computed by adding all the weights of each pos group found in that sentence. let sentenceWiseWeights = new Array( doc.sentences().length() ); sentenceWiseWeights.fill( 0 ); posGroupWeights.forEach( pgw => { pgw.sentences.forEach( e => { sentenceWiseWeights[ e ] += pgw.weight } ); }); // Normalize weights by first halving weight and the dividing by the max. // Halving reduces the relative differnce between weights. sentenceWiseWeights = sentenceWiseWeights.map( e=>e/2 ); const max = Math.max( ...sentenceWiseWeights ); return sentenceWiseWeights.map( e => e / max ); }48:[] 43:{"id":"cacb848b9eae9755","slug":null,"title":"How to visualize key sentences in a document?","thumbnail":"6d09c491d90ed5557d2611bc60f9f6cd9725443ca0f598bef7cc93b560824c0f","access_level":"public","create_time":"$D2023-02-08T10:13:25.861Z","update_time":"$D2023-02-08T10:13:25.861Z","view_time":null,"first_public_time":"$D2023-02-08T10:13:25.861Z","has_unpublished_edits":null,"fork":{"id":"58106b32a86f3278","slug":"how-to-visualize-key-sentences-in-a-document","title":"How to visualize key sentences in a document?","owner":{"id":"062de6e3e5a0d590","login":"winkjs","name":"WinkJS"}},"stars":0,"starred":false,"owner":{"type":"team","id":"7dd579cd728bc7fb","login":"alwayscry","name":"Penn","avatar_url":"https://avatars-next.observableusercontent.com/ed7f2516457eb636384ee646fd29739a5990570a3e61524d2972fba33013fa42","tier":"starter_2024","bio":"","following":false},"version":440,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"fe17292941b86055","login":"alwayscry","name":"Penn","avatar_url":"https://avatars-next.observableusercontent.com/ed7f2516457eb636384ee646fd29739a5990570a3e61524d2972fba33013fa42","tier":null,"bio":"","following":false}],"members":[{"type":"individual","id":"fe17292941b86055","login":"alwayscry","name":"Penn","avatar_url":"https://avatars-next.observableusercontent.com/ed7f2516457eb636384ee646fd29739a5990570a3e61524d2972fba33013fa42","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"How to visualize key sentences in a document?","cells":[{"id":0,"value":"# How to visualize key sentences in a document?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":369,"value":"$45","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":175,"value":"viewof focus = Inputs.toggle({label: \"Focus\", value: false})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":209,"value":"viewof percentage = Inputs.range([10, 60], {value: 20, step: 5, label: \"Highlighted sentences %age\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":32,"value":"{\n doc.sentences().each( (s, k) => s.markup(beginTag(k),endTag()));\n return html`

${doc.out(its.markedUpText).replaceAll('\\n','
')}

`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6,"value":"---\n### Setup","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":253,"value":"**`// Load winkNLP.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":8,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":243,"value":"**`// Load winkNLP's language model.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":10,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":257,"value":"**`// Instantiate winkNLP using the language model.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":12,"value":"nlp = winkNLP( model, ['sbd', 'pos'] );","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":261,"value":"**`// Obtain winkNLP's its helper.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":14,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":263,"value":"**`// Text to be processed, you can replace it by your own text.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":16,"value":"$46","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":265,"value":"**`// Process the text through the nlp pipe.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":18,"value":"doc = {\n // Ensure code is triggered every time percentage slider value changes.\n percentage;\n // Same way for the focus checkbox.\n if ( focus ) return nlp.readDoc( text );\n else return nlp.readDoc( text );\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":270,"value":"**`// Compute relative importance of each sentence as a numerical value between 0 and 1.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":20,"value":"$47","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":272,"value":"**`// markup() helper function, uses \"mark\" tag style to control highlighting.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":39,"value":"beginTag = function ( k ) {\n const a = (sentenceWiseNormalizedWeights[k]>cutoff)? sentenceWiseNormalizedWeights[k]:0;\n// if (a) return ``\n if (a){\n if (a>0.9){\n return ``\n } else if ( a > 0.75) {\n return ``\n } else if ( a > 0.5) {\n return ``\n } else {\n return ``\n }\n }\n if (focus) return ``;\n return ``\n};","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":41,"value":"endTag = function () {\n return ``;\n};","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":274,"value":"**`// Compute the importance cutoff on the basis of quantile using \"d3\", which is used in \"beginTag() above.`**","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":215,"value":"cutoff = d3.quantile(sentenceWiseNormalizedWeights, (100 - percentage)/100)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":284,"value":"","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":277,"value":"---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"}],"files":"$W48","stdlib":"1"}} 44:"$43:body"