27:null 28:["$","title",null,{"children":"Data pipeline group selection feud | Jonathan St-Onge | Observable"}] 2d:T6a9,## In words #### Data pipeline _Goal: We want to extract all the interactions among authors and theories in all articles that participated in the feud. Contrary to traditional bibliometric approaches that focus on citations, we aim to resolve the many interactions embedded in the content of mentions in articles._ 1. We were missing most of the content for the group selection debate so we had to collect pdfs anew. 2. We had to parse these pdfs. 3. The initial analysis with GPT-3.5 were really satisfying, so we decided to try to fine-tune it. 4. To fine tune, we then needed to label data. We focus on sentences with entities. 5. We identified sentences with entities (authors and theoretical constructs) with Spacy to accelerate the labelling process. 6. [...] #### Once we have the interactions, we'll be able to: _Goal: We want to characterize the sentiments in the above interactions using LLMs. Contrary to traditional approaches in NLP that average sentiments over hand-labelled dictionaries of words, we use LLMs to take into account the various nuances in scientific writing._ 1. Create `[author x tokens]` matrix to do Ousiometry. - Language authors in the feud use to describe other authors and/or theory. Since we focus on tokens that have some valence in it, we hope that this will depict opposition between camps in a feud. 1. Compare corpora from the 1960s v. 2000s using Allotaxonometry 1. Time series of sentiments at paragraph level for each article; how negative are criticisms in science? Do they get worst as feuds perdure in time? 1. Higher-order networks of author-author interactions. Can we reconstruct invisible colleges now that we have high resolution interactions data?2e:T919,#### Example fine tune 2 Would be nice but expensive: - `classification_context_10:` "Yang et al. (2018) also argued that rank(A ) is as high as vocabulary size V based on the following two assumptions: 1. Natural language is highly context-dependent. In addition, since we can imagine many kinds of contexts, it is difficult to assume a basis that represents a conditional probability distribution for any contexts. In other words, compressing U is difficult. 2. Since we also have many kinds of semantic meanings, it is difficult to assume basic meanings that can create all other semantic meanings by such simple operations as addition and subtraction; compressing V is difficult. In summary, Yang et al. (2018) indicated that D h N is much smaller than rank(A) because its scale is usually 10 2 and vocabulary size V is at least 10 4 . ---------------------------------- **PROPOSED METHOD: DIRECT OUTPUT CONNECTION** To construct a high-rank matrix, Yang et al. (2018) proposed Mixture of Softmaxes (MoS). MoS computes multiple probability distributions from the hidden state of final RNN layer h N and regards the weighted average of the probability distributions as the final distribution." - `gold_context:` "Yang et al. (2018) also argued that rank(A ) is as high as vocabulary size V based on the following two assumptions: 1. Natural language is highly context-dependent. In addition, since we can imagine many kinds of contexts, it is difficult to assume a basis that represents a conditional probability distribution for any contexts. 2. Since we also have many kinds of semantic meanings, it is difficult to assume basic meanings that can create all other semantic meanings by such simple operations as addition and subtraction; compressing V is difficult. In summary, Yang et al. (2018) indicated that D h N is much smaller than rank(A) because its scale is usually 10 2 and vocabulary size V is at least 10 4." Cheaper version - Only train `gpt3.5` to detect ["differences background", "background differences", "differences", "differences motivation", "differences similarities motivation", "uses differences similarities", "extends differences background", "extends differences motivation"] (n=687; mean average token count=41)2f:T465,

His[ERRINGTON] work is only casually mentioned by W.-E. On the whole, considering the enormous size of the book and the many rather peripheral topics treated, it is surprising how incompletely the really pertinent points are reviewed.{'neg': 0.0, 'neu': 0.899, 'pos': 0.101, 'compound': 0.4215}
I think it may be said of the book, with a certain amount of truth, that most of what is sound in it is not new, and most of which is new is not sound.{'neg': 0.0, 'neu': 0.882, 'pos': 0.118, 'compound': 0.5267}
This in itself might not have been a very serious objection.{'neg': 0.137, 'neu': 0.863, 'pos': 0.0, 'compound': -0.1513}
If it had contained a good, well-digested, and balanced summary of facts and views on animals density regulation in relation to social behaviour, together with a clear statement of the problems (and on this foundation a statement of the more extreme theories of the author), it might have been a highly useful book.{'neg': 0.045, 'neu': 0.811, 'pos': 0.144, 'compound': 0.7178}

30:[{"id":"d92bf93eb74551b11d35b56fea6643c9a3b72a182cd45ba6dbcf51c630600bfa65fd3480bf98d7e67f8651711245ab989c8a9cc295f140618b9abe4f3973b3fb","href":"https://static.observableusercontent.com/files/d92bf93eb74551b11d35b56fea6643c9a3b72a182cd45ba6dbcf51c630600bfa65fd3480bf98d7e67f8651711245ab989c8a9cc295f140618b9abe4f3973b3fb","name":"image@2.png","type":"image/png","lastModified":1682180313448.616,"size":17883},{"id":"c4a0230c74dc9aec86ca35e96c8005ef10654f93b06b138b0d889b8fa24233a0473eeb164ffc925f1dff5944cef21ed4df54e30bfb9152da7d03bbd2a12de1f7","href":"https://static.observableusercontent.com/files/c4a0230c74dc9aec86ca35e96c8005ef10654f93b06b138b0d889b8fa24233a0473eeb164ffc925f1dff5944cef21ed4df54e30bfb9152da7d03bbd2a12de1f7","name":"image@3.png","type":"image/png","lastModified":1682180449713.835,"size":22973},{"id":"e15d2404aff89e1e578e686aad5e0265ae8451481adf8d9ece1d95b4221778fc2d5ca181ab0ba9e530ff7f84ec60ea1f58302e8d29e8d2e65f2f774cd510cecc","href":"https://static.observableusercontent.com/files/e15d2404aff89e1e578e686aad5e0265ae8451481adf8d9ece1d95b4221778fc2d5ca181ab0ba9e530ff7f84ec60ea1f58302e8d29e8d2e65f2f774cd510cecc","name":"image@4.png","type":"image/png","lastModified":1682186401367.974,"size":59913},{"id":"4319fe43b8b4c7f84bb4304516676aaf2bf6d09175d0577c5ba216e714dbabf550272ad5522807133f46aba4e9d44c4851effeded6c7d080ac1fd61d947d50c3","href":"https://static.observableusercontent.com/files/4319fe43b8b4c7f84bb4304516676aaf2bf6d09175d0577c5ba216e714dbabf550272ad5522807133f46aba4e9d44c4851effeded6c7d080ac1fd61d947d50c3","name":"image@5.png","type":"image/png","lastModified":1682603871793.741,"size":99602},{"id":"6afbc4ef17e85bdb750fc707d3c05dceb96b51b3d975d8e5cb32fa247118e45b8493677a0e61d73d2bf44ff4e980d94484d9a06272862fdf7fca8b55bfef9a95","href":"https://static.observableusercontent.com/files/6afbc4ef17e85bdb750fc707d3c05dceb96b51b3d975d8e5cb32fa247118e45b8493677a0e61d73d2bf44ff4e980d94484d9a06272862fdf7fca8b55bfef9a95","name":"image@6.png","type":"image/png","lastModified":1682687455597.126,"size":31794}] 2b:{"id":"40bddb2c10be8425","slug":null,"title":"Data pipeline group selection feud","thumbnail":"26787e243bbc21083eead7ceb7118e9d387cdb14e5f2e4cbe34104f0b5ee04cd","access_level":"public","create_time":"$D2023-04-22T15:20:06.016Z","update_time":"$D2023-05-09T14:53:17.985Z","view_time":null,"first_public_time":"$D2023-04-22T15:20:06.016Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"4fabdbefe825d266","login":"jstonge","name":"Jonathan St-Onge","avatar_url":"https://avatars-next.observableusercontent.com/48507f1ab99162ceb5bc1fccad23b163ba525a2a9a76b045557676a38f2a7014","tier":"starter_2024"},"version":438,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"7cb18c5b30f73a44","login":"jstonge","name":"Jonathan St-Onge","avatar_url":"https://avatars-next.observableusercontent.com/48507f1ab99162ceb5bc1fccad23b163ba525a2a9a76b045557676a38f2a7014","tier":null}],"members":[{"type":"individual","id":"7cb18c5b30f73a44","login":"jstonge","name":"Jonathan St-Onge","avatar_url":"https://avatars-next.observableusercontent.com/48507f1ab99162ceb5bc1fccad23b163ba525a2a9a76b045557676a38f2a7014","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"Data pipeline group selection feud","cells":[{"id":0,"value":"# Data pipeline group selection feud\n## Extracting high-resolution interactions for articles that participated to the feud, using fine-tuned gpt3.5","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":4,"value":"mermaid`graph LR\nsubgraph Data-Collection\n direction TB\n d[\"Missing articles\\nfrom feud\"] -->|Collect PDFs|a[Raw PDFs]\n a[Raw PDFs]-->|GROBID OCR|b[Parsed papers\\nparagraph-level]\nend\nsubgraph Model-tuning\n direction TB\n m(GPT3's Currie model)--> n(Fine-tuned GPT3A)\n v(Abstracts annotated\\nwith stance) --> n\n v(Abstracts annotated\\nwith stance) --> i[SciBert-based]\n n(Fine-tuned GPT3A) --> g(Fine-tuned GPT3B)\n z(Snippets around cite nannotated\\nwith relevant context) --> g\nend\nsubgraph Paragraph-level\ndirection TB\n subgraph Context-level\n direction TB\n subgraph Word-level\n direction TB\n g[Fine-tuned GPT3B] --> h[Author-author\\nsentiment/stance]\n g --> e[Author-author\\nsentiment/stance]\n b --> e\n b --> h\n end\n i --> s[Context overall\\nsentiment/stance]\n end\n i --> f[Paragraph overall\\nsentiment/stance]\n g --> f\n i --> f\n b --> f\n b --> f\nend\n`\n// classDef Textclass fill:#a162","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":245,"value":"This week, we found out about:\n - https://github.com/DominikBeese/DidAIGetMoreNegativeRecently\n - This sentiment classifier sits on top of `scibert` and has been fine-tuned on 1500 labelled abstracts from nlp and cs.\n - The classifier seems to perform well on our corpus as well. We ran it on all paragraphs that we have, not just abstracts.\n - TODO: perhaps we can run that on sentences themselves? \n - https://github.com/allenai/multicite\n - Based on Jurgens et al. (2018), this work provide a taxonomy of the different types of citations; _background_, _motivation_, _uses_, _extensions_, _comparison or contrast future_\n - We train the model again in the hope of classifying mentions in our corpus. ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":162,"value":"### Codebook\n1. Slighly positive interactions between authors include 'confirmatory statements'; when Bob cited Alice about X in a confirmatory way we take that Bob is positively inclined towards Alice, e.g.\n\n - ![image@2.png](${await FileAttachment(\"image@2.png\").url()})\n - ![image@4.png](${await FileAttachment(\"image@4.png\").url()})\n\nThis is different than truly neutral statements such as:\n\n - ![image@3.png](${await FileAttachment(\"image@3.png\").url()})\n\nThe difference between the two is that in the first there is an assessment of the importance of the results provided by Adams.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":112,"value":"$2d","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":251,"value":"Fine-tuning gpt3.5\n\n![image@6.png](${await FileAttachment(\"image@6.png\").url()})","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":255,"value":"What if we follow the following pipeline:\n - Fine tune #1: Fine tune on stance detection using dataset from Beese et al (2023;l n=1265 if we only show (sent < -.1 | sent > 0.1)). \n - Fine tune #2: Contextual citation analysis (CCA) on my hand-labelled dataset, without sentiments (since we use dataset in #1). The idea we'll be to extract something similar than discussed in Lauscher et al. (2022). Or we use gold context of Lauscher et al. 2022 to reverse engineer the CCA.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":264,"value":"$2e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":222,"value":"## VADER example","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":215,"value":"$2f","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":232,"value":"## More examples","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":234,"value":"image5 = FileAttachment(\"image@5.png\").image()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W30","stdlib":"1","titling":"auto"}} 2c:"$2b:body"