28:null 29:["$","title",null,{"children":"LLM Case Study | Clinical Text Classification Using Transformer Embeddings + MedDRA | Zhihao Yao | Observable"}] 2e:T5ce,# LLM Case Study | Clinical Text Classification Using Transformer Embeddings + MedDRA --- Transformer models have fundamentally reshaped natural language processing (NLP), achieving state-of-the-art performance across a wide range of tasks—including conversation, translation, classification, and generation. Unlike earlier models that process text in a fixed sequence, transformers rely on a powerful innovation called the **self-attention mechanism**, which allows the model to dynamically assign importance to different words in a sentence, regardless of their position. This parallel and context-aware processing enables transformers to capture deep semantic relationships within text—making them especially effective for domain-specific applications like **clinical text classification**. In our tutorial, we demonstrate how to leverage pretrained transformer models, such as **MiniLM** and **ClinicalBERT**, to embed free-text notes and classify them against standard terminologies like **MedDRA**. By computing **cosine similarity** between embedded phrases, we can map real-world clinical narratives (e.g., “patient didn’t cooperate”) to structured categories (e.g., “REFUSAL”), even when terminology varies. Such techniques open up exciting possibilities in healthcare, including **automated data normalization**, **clinical decision support**, and **real-time patient feedback classification**—all without the need for full retraining of large models. ---2f:T97e,results = { // Step 1: Normalize the input text entries const normalizedTexts = texts.map(normalize); // Step 2: Normalize each phrase in the categories for consistent comparison const normalizedCategories = {}; for (const [key, phrases] of Object.entries(categories)) { normalizedCategories[key] = phrases.map(normalize); } // Step 3: Generate vector embeddings for each category phrase using the transformer const categoryEmbeddings = {}; for (const [label, phrases] of Object.entries(normalizedCategories)) { categoryEmbeddings[label] = []; for (const phrase of phrases) { const out = await embedder(phrase); // Get model output const vec = meanPoolFromTensor(out); // Mean pooling to get a vector categoryEmbeddings[label].push(vec); // Store for the label } } // Step 4: Initialize arrays to store the final results const outputAvg = []; const outputMax = []; // Step 5: For each input sentence, compare it with all category embeddings for (const sentence of normalizedTexts) { const sentOut = await embedder(sentence); const sentVec = meanPoolFromTensor(sentOut); // ---- AVERAGE SIMILARITY ---- let bestAvg = { label: "", score: -1 }; for (const [label, reps] of Object.entries(categoryEmbeddings)) { // Calculate cosine similarity to all example vectors, then average const avgScore = reps.map(rep => cosineSimilarity(sentVec, rep)) .reduce((a, b) => a + b, 0) / reps.length; if (avgScore > bestAvg.score) { bestAvg = { label, score: avgScore }; } } outputAvg.push({ sentence, category: bestAvg.label || "(none)", similarity: bestAvg.score.toFixed(2) }); // ---- MAXIMUM SIMILARITY ---- let bestMax = { label: "", score: -1 }; for (const [label, reps] of Object.entries(categoryEmbeddings)) { // Find the highest similarity to any example vector in the category const maxScore = Math.max(...reps.map(rep => cosineSimilarity(sentVec, rep))); if (maxScore > bestMax.score) { bestMax = { label, score: maxScore }; } } outputMax.push({ sentence, category: bestMax.label || "(none)", similarity: bestMax.score.toFixed(2) }); } // Step 6: Return both classification outputs return { outputAvg, outputMax }; }; 30:T690,--- ## Discussion

| **Method** | **Benefit** | **Solution** | | ----------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Add more category phrases | Most effective for immediate score boost | Expand the `Medical Reason` category with more representative phrases like `"unfit for event"`, `"sick during event"`, `"missed due to illness"` to improve matching coverage. | | Switch to `all-mpnet-base-v2` | Semantic depth; better for long inputs | Use the Hugging Face model `sentence-transformers/all-mpnet-base-v2` for richer embeddings that better capture complex medical language. | | Normalize/abstract text | More robust embeddings for comparison | Preprocess inputs and category phrases by lowercasing, removing stopwords, normalizing synonyms (e.g., "assessment", "scan", "test" → "event") before computing embeddings. | | Fine-tune on labeled data | Best accuracy in domain-specific language | Collect labeled examples from your actual dataset, then fine-tune a transformer model using Hugging Face's `Trainer` or `PEFT` with `sentence-transformers` to capture clinical phrasing patterns. |31:T863,--- ### Update normalization by removing the line: ```js .replace(/\b(patient|subject|participant)\b/g, '') ``` **increased your similarity score** and **corrected the category** from `"SITE_ISSUE"` to the more appropriate `"REFUSAL"`. Here's why this happens: --- ### What That Line Does This line removes important identity terms from your input sentence during preprocessing: ```js "Echo not done – patient didn’t cooperate." ↓ becomes ↓ "echo not done – didn’t cooperate." ``` So the model never sees the word **"patient"** anymore. --- ### Impact on Embedding & Similarity BERT-based models like `all-MiniLM-L6-v2` or `ClinicalBERT` use **contextual embeddings**. That means: * Removing **"patient"** strips away context that is **semantically relevant**. * The model now sees a vaguer sentence: “didn’t cooperate,” but **who didn’t cooperate?** The agent is missing. * This makes the sentence embedding **drift** away from phrases like "patient refused" or "participant declined", reducing cosine similarity with the `"REFUSAL"` category vectors. In fact, it may now resemble more general operational issues like site errors or protocol deviation (`"SITE_ISSUE"`), which is why it gets misclassified. --- ### Why Keeping "patient" Helps By keeping `"patient"`: ```js "Echo not done – patient didn’t cooperate." → (no modification) ``` The model can now align this sentence more closely with category phrases like: ```js ["patient refused", "participant declined"] ``` So: * **Semantic proximity increases** * **Cosine similarity improves** from 0.29 → 0.46 * And the correct category (`"REFUSAL"`) is chosen --- ### Takeaway for Normalization Functions Removing generic terms like `"patient"` may **reduce noise** in some cases, but in **clinical free-text**, those identity terms often **carry key semantic meaning**. #### Guideline: * Remove only **truly redundant stopwords** (e.g., “the”, “and”, “of”). * Avoid stripping role-based terms like `"patient"`, `"doctor"`, `"investigator"` if they are **relevant to the action** (e.g., refusing, reporting, diagnosing). --- 32:[] 2c:{"id":"a56a2dbbe035d523","slug":null,"title":"LLM Case Study | Clinical Text Classification Using Transformer Embeddings + MedDRA","thumbnail":"7d4e8b847f6f471d0fae8645f134334e67c55463cdd50ed9000a666aca28fcdd","access_level":"public","create_time":"$D2025-05-31T01:01:16.905Z","update_time":"$D2025-06-24T13:58:16.112Z","view_time":null,"first_public_time":"$D2025-06-02T19:42:28.847Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"610db6fe9b10a76c","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":"starter_2024"},"version":329,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"01b5d984f2428b1a","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":null}],"members":[{"type":"individual","id":"01b5d984f2428b1a","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":null,"role":"editor"},{"type":"team","id":"610db6fe9b10a76c","login":"howardyao","name":"Zhihao Yao","avatar_url":"https://avatars-next.observableusercontent.com/0bac389abf8df9b4d7c6e09e8f96b464d2c01a64946e8b88cd7e143da5750c0e","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":{"id":"102982dc3961aa5a","slug":"machine-learning-notes","title":"Machine Learning Notes","owner":{"id":"610db6fe9b10a76c","login":"howardyao","name":"Zhihao Yao"}},"type":"notebook","body":{"titling":"auto","title":"LLM Case Study | Clinical Text Classification Using Transformer Embeddings + MedDRA","cells":[{"id":221,"value":"$2e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":226,"value":"## Step 1: Normalize the Input\n\nMake both free-text and template phrases comparable by removing noise and harmonizing terminology. All input sentences and category labels are cleaned (e.g., punctuation removed, lowercased, synonyms collapsed) for consistency.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":229,"value":"function normalize(text) {\n return text\n .toLowerCase()\n .replace(/[^a-z0-9 ]+/g, '') // Remove punctuation\n // .replace(/\\b(patient|subject|participant)\\b/g, '') // Remove redundant synonyms\n .replace(/\\b(scan|test|procedure|assessment)\\b/g, 'event') // Standardize clinical events\n .replace(/\\s+/g, ' ') // Collapse whitespace\n .trim();\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":234,"value":"---\n\n## Step 2: Generate Sentence Embeddings\n\nThe `embedder()` function uses a pretrained transformer (e.g., MiniLM or ClinicalBERT) to convert text into high-dimensional vectors.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":238,"value":"tranformers=import(\"https://cdn.jsdelivr.net/npm/@xenova/transformers@2\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":242,"value":"pipeline=tranformers.pipeline","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":245,"value":"embedder = await pipeline(\"feature-extraction\", \"Xenova/all-MiniLM-L6-v2\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":250,"value":"The model splits the input sentence:\n\n```js\n[\"[CLS]\", \"the\", \"scan\", \"was\", \"missed\", \"due\", \"to\", \"transportation\", \"[SEP]\"]\n```\n\nThis gives 9 tokens, including special tokens like [CLS] and [SEP]. \n\n| Token | Purpose | Position | Use Case |\n| ------- | ------------------------------ | ---------------------- | -------------------------------- |\n| `[CLS]` | Represents the **whole input** | Always at start | Used for classification output |\n| `[SEP]` | Separates **sentences** | Between/after segments | Used in pair tasks or single-end |\n\nEach token is converted to a dense vector of size 384, which comes from the embedding layer of the model (in this case hidden_size = 384).\n\nThe output tensor has shape `[1, tokens, hiddenSize]`, e.g. `[${vectorDemo.dims.join(' , ')}]`.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":247,"value":"vectorDemo = await embedder(\" scan was missed due to transportation\");","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":254,"value":"---\n\n## Step 3: Mean Pooling\n\nThis function averages the token embeddings to form a single vector representation for the full sentence.The output tensor from the transformer has dimensions [tokens, hidden_size]. We average over all tokens to get a single embedding vector per sentence.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":258,"value":"function meanPoolFromTensor(tensor) {\n const { data, dims } = tensor;\n const [batch, tokens, hiddenSize] = dims;\n const pooled = new Array(hiddenSize).fill(0);\n for (let i = 0; i < tokens; i++) {\n for (let j = 0; j < hiddenSize; j++) {\n pooled[j] += data[i * hiddenSize + j];\n }\n }\n return pooled.map(v => v / tokens);\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":261,"value":"---\n\n## Step 4: Cosine Similarity\n\n**Formula:**\n\n${tex.block`\n\\text{cosine\\_similarity}(A, B) = \\frac{A \\cdot B}{\\|A\\| \\cdot \\|B\\|}\n`}\n\n\nThis measures semantic similarity between a free-text sentence and a MedDRA category prototype.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":264,"value":"function cosineSimilarity(a, b) {\n const dot = a.reduce((sum, ai, i) => sum + ai * b[i], 0);\n const normA = Math.sqrt(a.reduce((sum, ai) => sum + ai * ai, 0));\n const normB = Math.sqrt(b.reduce((sum, bi) => sum + bi * bi, 0));\n return dot / (normA * normB);\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":267,"value":"---\n\n## Step 5: Define MedDRA-Like Categories\n\n```js\ncategories = {\n \"DRA_Refusal\": [\"subject refused\", \"patient declined\", \"refusal to continue\"],\n \"DRA_Scheduling\": [\"scheduling conflict\", \"missed appointment\", \"visit not scheduled\"],\n \"DRA_Transport\": [\"transportation issue\", \"no ride\", \"missed due to transit\"],\n ...\n}\n```\n\nEach label corresponds to a MedDRA-like standardized category. These phrases are used as semantic references.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":270,"value":"categories = ({\n \"DRA_REFUSAL\": [ // Subject Refusal\n \"patient refused\", \n \"participant declined\", \n \"subject unwilling to continue\", \n \"refusal to comply\"\n ],\n \"DRA_SCHEDULING\": [ // Scheduling Conflict\n \"scheduling issue\", \n \"calendar conflict\", \n \"missed appointment\", \n \"visit not scheduled in time\"\n ],\n \"DRA_SITE_ISSUE\": [ // Site-related Problems\n \"staff unavailable\", \n \"site error\", \n \"technical failure at site\", \n \"equipment malfunction\"\n ],\n \"DRA_MEDICAL\": [ // Medical or Clinical Events\n \"felt unwell\", \n \"adverse effect\", \n \"experienced symptoms\", \n \"illness prevented visit\"\n ],\n \"DRA_TRANSPORT\": [ // Social or Logistic Barriers\n \"no transportation\", \n \"public transit delay\", \n \"car broke down\", \n \"unable to reach site\"\n ]\n});","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":273,"value":"---\n\n## Step 6: Classify Based on Similarity\n\nLoop over normalized input and compare with category embeddings:\n\n```js\nfor (const sentence of normalizedTexts) {\n const sentVec = meanPoolFromTensor(await embedder(sentence));\n \n for (const [label, reps] of Object.entries(categoryEmbeddings)) {\n const score = cosineSimilarity(sentVec, rep);\n ...\n }\n}\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":278,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"},{"id":276,"value":"---\n\n## Output

\n\n\n \n \n \n \n ${results.outputMax.map((r,i) => html`\n \n \n \n \n \n \n `)}\n \n

Sentence	Normalized Input	Category	Similarity
${texts[i]}	${r.sentence}	${r.category}	${r.similarity}

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":295,"value":"$30","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":213,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":319,"value":"## Reference\n\nCho, H. N., Jun, T. J., Kim, Y.-H., Kang, H., Ahn, I., Gwon, H., Kim, Y., Seo, J., Choi, H., Kim, M., Han, J., Kee, G., Park, S., & Ko, S. (2024). *Task-specific transformer-based language models in health care: Scoping review*. JMIR Medical Informatics. [https://doi.org/10.2196/55213](https://doi.org/10.2196/55213)\n**PMID:** 39556827 • **PMCID:** PMC11612605\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":208,"value":"---\n\n## Appendix","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"","database":"$undefined"},{"id":102,"value":"texts = [\n \"Patient refused to complete the assessment.\",\n \"Echo not done – patient didn’t cooperate.\",\n \"Visit missed due to scheduling issue.\",\n \"Scan skipped as subject felt unwell.\",\n \"Procedure canceled - site staff unavailable.\",\n \"Participant declined test due to anxiety.\"\n]","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W32","stdlib":"1"}} 2d:"$2c:body"