31:T7c2,How accurate are LLMs in reasoning over unstructured data, especially when confronted with information that may not be in their training data? The existing [needle-in-the-haystack benchmarks](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) are a good start, but they are a bit limited for two reasons: 1. The documents are likely to be in the LLMs' training set (e.g., Paul Graham's essays) 2. The questions and inserted facts are not relevant to any particular task So, I decided to make my own small task and evaluate the models! The dataset consists of ~85 transcripts of video calls between doctors and their patients, pulled from `train.csv` and `valid.csv` in [this repository](https://github.com/wyim/aci-bench/tree/main/data/challenge_data). I ask 3 questions--a simple question, medium-difficulty question, and a hard question--to 8 instruction-tuned models: | Model | Input Token Limit | Last Updated | |--------------------------|---------|---------------| | OpenAI GPT-3.5-Turbo-16k | 16k | July 2023 | | OpenAI GPT-4 11-06 preview | 128k | November 2023 | | Anthropic Haiku | 200k | March 2024 | | Anthropic Sonnet | 200k | February 2024 | | Anthropic Opus | 200k | February 2024 | | Gemini Pro | 32k | February 2024 | | Mistral-7B | 128k | March 2024 | | Gemma-7B | 8k | March 2024 | | DBRX-Instruct | 32k | March 2024 | In this report, I show each model's performance, as well as the responses for each question. You can slice and dice the responses for each question, as well as hover over the the scatter plots to see models' answers vs the reference answer. At the end, I run the synthetic needle-in-a-haystack evals. **Note that I manually created the reference answer, with a _lot_ of data wrangling, so there are bound to be mistakes. I'm not trying to have perfect data; that would take ages :-)** 32:T5e7,easyPlot = {let aggregatedData = d3.rollups(first_name, v => d3.mean(v, d => d.correctness), // Aggregate function d => d.model, d => d.idx_bin) // Grouping keys .map(([model, bins]) => bins.map(([idx_bin, correctness]) => ({ model, idx_bin, correctness }))) .flat(); const colorScale = d3.scaleSequential() .interpolator(d3.interpolateViridis) // Changed colormap .domain([0, 1]); // Placeholder domain let p = Plot.plot({ style: { fontSize: 16 }, padding: 0, marginLeft: 120, width: document.body.clientWidth, // This sets the plot to take up the full width of the body height: 400, // You can adjust the height as needed x: {axis: "bottom", label: "Position in Transcript"}, y: {label: "Model"}, color: {type: "linear", scheme: "Viridis"}, marks: [ Plot.cell(aggregatedData, {x: "idx_bin", y: "model", fill: "correctness", inset: 0.5}), Plot.text(aggregatedData, {x: "idx_bin", y: "model", text: (d) => d.correctness?.toFixed(2), fill: d => { // Convert the color to RGB, then determine if it's light or dark const color = d3.rgb(colorScale(d.correctness)); // Simple luminance calculation for readability const luminance = (0.299 * color.r + 0.587 * color.g + 0.114 * color.b); return luminance > 186 ? "black" : "white"; // Choose text color for contrast }, title: "title"}) ], title: "Model Accuracy", }) return p }33:T411,nameScatterPlot = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(first_name, {x: "idx", y: "correctness", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Correctness ${d.correctness}`, r: 6, }), Plot.frame(), Plot.tip(first_name, Plot.pointer({ x: "idx", y: "correctness", stroke: "model", title: (d) => [`REFERENCE:\n${d.first_name}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: first_name, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Correctness", domain: d3.extent(first_name, d => d.correctness), // Ensure Y-axis covers the full recall range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Correctness vs. Position of Reference Answer in Transcript", style: { fontSize: 16 }, }); return scatterPlot; }34:T46f,### Takeaways _NOTE: The idx/position here is the average of the positions of each medication in the transcript, so that's why you see floats_ * The task, listing medications, simply asks the model to regurgitate various medications from the transcript * Medications currently taken are distributed mostly in the first third of the transcript * Opus is slightly better than GPT-4 in my metrics. But if you hover over the bad answers in the scatterplot for GPT-4, you'll find that GPT-4 rephrases the medications slightly * Claude models follow a super consistent and slightly more verbose structure? * Personally I would prefer Opus to GPT-4 here * All models degrade in performance when the answer is in the second and third segments of the transcript (note that segments are equally sized) * Gemini fails to return any meaningful result (e.g., it returns empty string or N/A) for several of the calls * Haiku and Sonnet look eerily similar in performance (but clearly have different responses) * DBRX has super long responses (and parts of the first part of the response seem to be half system prompt? maybe i prompted incorrectly)35:T437,medScatterPlot = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(medications, {x: "idx", y: "recall", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Recall ${d.recall}`, r: 6, }), Plot.frame(), Plot.tip(medications, Plot.pointer({ x: "idx", y: "recall", stroke: "model", // channels: {medications: "medications", output: "result"}, title: (d) => [`REFERENCE:\n${d.medications}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: medications, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Recall", domain: d3.extent(medications, d => d.recall), // Ensure Y-axis covers the full recall range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Recall vs. Position of Reference Answer in Transcript", style: { fontSize: 16 }, }); return scatterPlot; }36:T6b9,medBoxPlot = { // Calculate normalized bin counts const binCounts = d3.rollup(medications, v => v.length, d => d.idx_bin); const totalCount = medications.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(medications, v => d3.mean(v, d => d.recall), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); medications.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Recall", grid: true, }, facet: { data: medications, y: "bin_label", inset: 5, }, marks: [ // Box plots for recall distribution within each idx_bin and model Plot.boxY(medications, {x: "model", y: "recall", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), // Overlay individual recall values as points Plot.dot(medications, {x: "model", y: "recall", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), Plot.frame(), ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, // width: document.body.clientWidth, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top", }, title: "Recall Distributions by Transcript Segment", style: { fontSize: 16 }, }); return plot; }37:T409,sentenceScatterPlot = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(sentences, {x: "idx", y: "similarity", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Similarity ${d.similarity}`, r: 6, }), Plot.frame(), Plot.tip(sentences, Plot.pointer({ x: "idx", y: "similarity", stroke: "model", title: (d) => [`REFERENCE:\n${d.longest_sentence}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: sentences, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Similarity", domain: d3.extent(sentences, d => d.similarity), // Ensure Y-axis covers the full range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Similarity vs. Position of Reference Answer in Transcript", style: { fontSize: 16 }, }); return scatterPlot; }38:T6a9,sentenceBoxPlot = { // Calculate normalized bin counts const binCounts = d3.rollup(sentences, v => v.length, d => d.idx_bin); const totalCount = sentences.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(sentences, v => d3.mean(v, d => d.similarity), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); sentences.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Similarity", grid: true, }, facet: { data: sentences, y: "bin_label", inset: 5 }, marks: [ // Box plots for similarity distribution within each idx_bin and model Plot.boxY(sentences, {x: "model", y: "similarity", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nSimilarity: ${d.similarity}`}), // Overlay individual similarity values as points Plot.dot(sentences, {x: "model", y: "similarity", fill: "model", title: d => `Model: ${d.model}\nSimilarity: ${d.similarity}`}), Plot.frame() ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top" }, title: "Similarity Distributions by Transcript Segment", style: {fontSize: 16} }); return plot; }39:T488,### Takeaways _NOTE: The max idx is larger here (~13k) because all transcripts contain an answer. In previous questions, the super-long transcripts didn't contain any answers, so they weren't included in the analysis._ - None of the models are perfect! But other peoples' evals show perfect results? I guess this medical transcript data wasn't in the training set, so who knows what the result would be. - GPT 3.5 is the best!! That is quite surprising. I double and triple checked my code; maybe there is a bug. But I couldn't find one. Also the GPT-4 responses are quite verbose, which aligns with my expectations. - The models either get all 3 or get none, with the exception of Gemma, which occasionally retrieves 2 of the 3?? - Claude models are worse than Mistral - Surprisingly Mistral gets the ingredients, but you can see that its responses are a bit verbose - Gemini is quite bad, but many of the outputs are empty so that explains the poor performance - Sometimes the models are self-aware, and say that the transcript is not about pizza :-) - DBRX completely flops on all but 1!! It does not even output anything...again, maybe i'm sampling wrong3a:T67c,syntheticBoxPlot = { // Calculate normalized bin counts const binCounts = d3.rollup(synthetic, v => v.length, d => d.idx_bin); const totalCount = synthetic.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(synthetic, v => d3.mean(v, d => d.recall), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); synthetic.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Recall", grid: true, }, facet: { data: synthetic, y: "bin_label", inset: 5 }, marks: [ // Box plots for recall distribution within each idx_bin and model Plot.boxY(synthetic, {x: "model", y: "recall", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), // Overlay individual recall values as points Plot.dot(synthetic, {x: "model", y: "recall", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), Plot.frame(), ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top" }, title: "Recall Distributions by Transcript Segment", style: {fontSize: 16} }); return plot; }3b:T449,# Needle in the Synthetic World What happens when we run the same experiments as above, but we use GPT-4 to generate _synthetic_ transcripts between patients and providers? I generated 35 synthetic transcripts with a prompt that included a random patient name (from some list of names I queried GPT-4) and random set of medications (again, from a list of medications I queried GPT-4). The prompt to generate synthetic transcripts is as follows: ``` Here is an example of a transcript of a patient's visit to the doctor:\n\n{transcript}\n\nPlease generate a completely different synthetic transcript containing at least {num_words} words, representing a patient's visit to the doctor. The patient's name is {patient_name}, and they are taking the medications {medications}. Make sure the transcript references the patient name and the medications (and dosages) they are taking. Return ONLY the synthetic transcript. ``` For each of the 35 LLM calls to generate a synthetic transcript, I randomly selected a real transcript to provide as the example transcript (and `num_words`) in the prompt.3c:T625,easyPlotSynthetic = {let aggregatedData = d3.rollups(first_name_synthetic, v => d3.mean(v, d => d.correctness), // Aggregate function d => d.model, d => d.idx_bin) // Grouping keys .map(([model, bins]) => bins.map(([idx_bin, correctness]) => ({ model, idx_bin, correctness }))) .flat(); const colorScale = d3.scaleSequential() .interpolator(d3.interpolateViridis) // Changed colormap .domain([0, 1]); // Placeholder domain let p = Plot.plot({ style: { fontSize: 16 }, padding: 0, marginLeft: 120, width: document.body.clientWidth, // This sets the plot to take up the full width of the body height: 400, // You can adjust the height as needed x: {axis: "bottom", label: "Position in Transcript"}, y: {label: "Model"}, color: {type: "linear", scheme: "Viridis"}, marks: [ Plot.cell(aggregatedData, {x: "idx_bin", y: "model", fill: "correctness", inset: 0.5}), Plot.text(aggregatedData, {x: "idx_bin", y: "model", text: (d) => d.correctness?.toFixed(2), fill: d => { // Convert the color to RGB, then determine if it's light or dark const color = d3.rgb(colorScale(d.correctness)); // Simple luminance calculation for readability const luminance = (0.299 * color.r + 0.587 * color.g + 0.114 * color.b); return luminance > 186 ? "black" : "white"; // Choose text color for contrast }, title: "title"}) ], title: "Model Accuracy on Synthetic Transcripts (GPT-4 generated)", }) return p }3d:T46c,nameScatterPlotSynthetic = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(first_name_synthetic, {x: "idx", y: "correctness", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Correctness ${d.correctness}`, r: 6, }), Plot.frame(), Plot.tip(first_name_synthetic, Plot.pointer({ x: "idx", y: "correctness", stroke: "model", title: (d) => [`REFERENCE:\n${d.first_name}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: first_name_synthetic, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Correctness", domain: d3.extent(first_name_synthetic, d => d.correctness), // Ensure Y-axis covers the full recall range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Correctness vs. Position of Reference Answer in Transcript (for Synthetically-Generated Transcripts)", style: { fontSize: 16 }, }); return scatterPlot; }3e:T483,medScatterPlotSynthetic = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(medications_synthetic, {x: "idx", y: "recall", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Recall ${d.recall}`, r: 6, }), Plot.frame(), Plot.tip(medications_synthetic, Plot.pointer({ x: "idx", y: "recall", stroke: "model", // channels: {medications: "medications", output: "result"}, title: (d) => [`REFERENCE:\n${d.medications}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: medications_synthetic, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Recall", domain: d3.extent(medications_synthetic, d => d.recall), // Ensure Y-axis covers the full recall range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Recall vs. Position of Reference Answer in Transcript (on Synthetic Transcripts)", style: { fontSize: 16 }, }); return scatterPlot; }3f:T723,medBoxPlotSynthetic = { // Calculate normalized bin counts const binCounts = d3.rollup(medications_synthetic, v => v.length, d => d.idx_bin); const totalCount = medications_synthetic.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(medications_synthetic, v => d3.mean(v, d => d.recall), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); medications_synthetic.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Recall", grid: true, }, facet: { data: medications_synthetic, y: "bin_label", inset: 5, }, marks: [ // Box plots for recall distribution within each idx_bin and model Plot.boxY(medications_synthetic, {x: "model", y: "recall", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), // Overlay individual recall values as points Plot.dot(medications_synthetic, {x: "model", y: "recall", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), Plot.frame(), ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, // width: document.body.clientWidth, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top", }, title: "Recall Distributions by Transcript Segment (on Synthetic Transcripts)", style: { fontSize: 16 }, }); return plot; }40:T4ae,{ const sentenceLengths = []; const sentenceDelimiter = /(? { const sentences = d.transcript.split(sentenceDelimiter); sentences.forEach(sentence => { sentenceLengths.push({length: sentence.length, type: "synthetic", transcriptIdx: idx}); }); }); // Process real sentences sentences.forEach((d, idx) => { const sentences = d.transcript.split(sentenceDelimiter); sentences.forEach(sentence => { sentenceLengths.push({length: sentence.length, type: "real", transcriptIdx: idx}); }); }); return Plot.plot({ x: { nice: true, ticks: 10, transform: (d) => Math.pow(10, d), type: "log", tickFormat: "~s", label: "# characters in a sentence (log scale) →" }, marks: [ Plot.rectY(sentenceLengths, Plot.binX({y: "count"}, {x: (d) => Math.log10(d.length || NaN), fill: "type", mixBlendMode: "screen"})), Plot.ruleY([0]) ], y: {grid: true}, color: {legend: true}, title: "Histogram of Sentence Lengths in Transcripts" }); // TODO: Create a hexbin plot/2D hist }41:T455,sentenceScatterPlotSynthetic = { const scatterPlot = Plot.plot({ marks: [ Plot.dot(sentences_synthetic, {x: "idx", y: "similarity", fill: "model", title: d => `${d.model}: Idx ${d.idx}, Similarity ${d.similarity}`, r: 6, }), Plot.frame(), Plot.tip(sentences_synthetic, Plot.pointer({ x: "idx", y: "similarity", stroke: "model", title: (d) => [`REFERENCE:\n${d.longest_sentence}`, `OUTPUT:\n${d.result}`].join("\n\n") // Custom tooltip format })) ], height: 800, marginRight: 90, marginLeft: 110, grid: true, facet: { data: sentences_synthetic, y: "model" }, x: { label: "Idx →", nice: true }, y: { label: "↑ Similarity", domain: d3.extent(sentences_synthetic, d => d.similarity), // Ensure Y-axis covers the full range nice: true, inset: 5 }, color: { legend: false // Optional, remove if you prefer not to show a legend }, title: "Similarity vs. Position of Reference Answer in Transcript (on Synthetic Transcripts)", style: { fontSize: 16 }, }); return scatterPlot; }42:T6f8,sentenceBoxPlotSynthetic = { // Calculate normalized bin counts const binCounts = d3.rollup(sentences_synthetic, v => v.length, d => d.idx_bin); const totalCount = sentences_synthetic.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(sentences_synthetic, v => d3.mean(v, d => d.similarity), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); sentences_synthetic.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Similarity", grid: true, }, facet: { data: sentences_synthetic, y: "bin_label", inset: 5 }, marks: [ // Box plots for similarity distribution within each idx_bin and model Plot.boxY(sentences_synthetic, {x: "model", y: "similarity", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nSimilarity: ${d.similarity}`}), // Overlay individual similarity values as points Plot.dot(sentences_synthetic, {x: "model", y: "similarity", fill: "model", title: d => `Model: ${d.model}\nSimilarity: ${d.similarity}`}), Plot.frame() ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top" }, title: "Similarity Distributions by Transcript Segment", style: {fontSize: 16} }); return plot; }43:T6e6,syntheticBoxPlotSynthetic = { // Calculate normalized bin counts const binCounts = d3.rollup(synthetic_synthetic, v => v.length, d => d.idx_bin); const totalCount = synthetic_synthetic.length; const binCountsNormalized = Array.from(binCounts, ([key, value]) => ({ bin: key, percentage: (value / totalCount) * 100 })); // Order models by overall mean recall const orderedModels = Array.from(d3.rollup(synthetic_synthetic, v => d3.mean(v, d => d.recall), d => d.model)) .sort((a, b) => d3.descending(a[1], b[1])) .map(d => d[0]); synthetic_synthetic.forEach(d => { const binInfo = binCountsNormalized.find(b => b.bin === d.idx_bin); if (binInfo) { // Creating a new property that combines bin name and normalized count d.bin_label = `${d.idx_bin} (${binInfo.percentage.toFixed(1)}%)`; } }); const plot = Plot.plot({ y: { label: "↑ Recall", grid: true, }, facet: { data: synthetic_synthetic, y: "bin_label", inset: 5 }, marks: [ // Box plots for recall distribution within each idx_bin and model Plot.boxY(synthetic_synthetic, {x: "model", y: "recall", stroke: "model", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), // Overlay individual recall values as points Plot.dot(synthetic_synthetic, {x: "model", y: "recall", fill: "model", title: d => `Model: ${d.model}\nRecall: ${d.recall}`}), Plot.frame(), ], color: { legend: false }, marginLeft: 60, marginRight: 150, marginTop: 45, x: { label: "Model", domain: orderedModels, // Ensure models are ordered as calculated previously axis: "top" }, title: "Recall Distributions by Transcript Segment (on Synthetic Transcripts)", style: {fontSize: 16} }); return plot; }44:[{"id":"1f99e7a5cdc1f6611b986c1ea2a7464bc8f993c9cb74d29757bc87c4b96dd0f1fa3c8bad01a76b381f94d3f3dc9ea4b4d628adc0ded47f49aa98e566835e15d8","href":"https://static.observableusercontent.com/files/1f99e7a5cdc1f6611b986c1ea2a7464bc8f993c9cb74d29757bc87c4b96dd0f1fa3c8bad01a76b381f94d3f3dc9ea4b4d628adc0ded47f49aa98e566835e15d8","name":"first_name@5.csv","type":"text/csv","lastModified":1712124754139.449,"size":6698825},{"id":"1d52950b627ba019b2acbe21283b62996f777b07fb215571f36179f7f47f0c3d542c7609013a484629fac840618e6a66d3a9ae4c9d30665ea6a78afd725f43ce","href":"https://static.observableusercontent.com/files/1d52950b627ba019b2acbe21283b62996f777b07fb215571f36179f7f47f0c3d542c7609013a484629fac840618e6a66d3a9ae4c9d30665ea6a78afd725f43ce","name":"longest_patient_sentence@5.csv","type":"text/csv","lastModified":1712125018898.375,"size":7741692},{"id":"e3592f4afcc05bdcfc0a87dc4bcf7c9475cb06f94f6918fc793e68ecbe3842f25801f74ebf531802d6a088b0919049a793e844927d7360b1bc9bc61f671ff540","href":"https://static.observableusercontent.com/files/e3592f4afcc05bdcfc0a87dc4bcf7c9475cb06f94f6918fc793e68ecbe3842f25801f74ebf531802d6a088b0919049a793e844927d7360b1bc9bc61f671ff540","name":"medications@7.csv","type":"text/csv","lastModified":1712125057781.394,"size":2977385},{"id":"1d52950b627ba019b2acbe21283b62996f777b07fb215571f36179f7f47f0c3d542c7609013a484629fac840618e6a66d3a9ae4c9d30665ea6a78afd725f43ce","href":"https://static.observableusercontent.com/files/1d52950b627ba019b2acbe21283b62996f777b07fb215571f36179f7f47f0c3d542c7609013a484629fac840618e6a66d3a9ae4c9d30665ea6a78afd725f43ce","name":"longest_patient_sentence@6.csv","type":"text/csv","lastModified":1712125059437.831,"size":7741692},{"id":"831c8128690c52fc647c0faef29d83fdf4468851de8d58f3322060d4b5e416170c8b555d2f5f8bccf8eebd7dbae507f2438c21135a7d9991b065cc47455e457c","href":"https://static.observableusercontent.com/files/831c8128690c52fc647c0faef29d83fdf4468851de8d58f3322060d4b5e416170c8b555d2f5f8bccf8eebd7dbae507f2438c21135a7d9991b065cc47455e457c","name":"first_name@6.csv","type":"text/csv","lastModified":1712901028243.414,"size":1911340},{"id":"e7ba31954476e7d38d9e4343645083c4f61332d02cef450c0d0d551e7e190e44e272a3fe691aee14373980d3ad60a7dbf4e625dd4731d813ff56013da0b83dc0","href":"https://static.observableusercontent.com/files/e7ba31954476e7d38d9e4343645083c4f61332d02cef450c0d0d551e7e190e44e272a3fe691aee14373980d3ad60a7dbf4e625dd4731d813ff56013da0b83dc0","name":"medications@8.csv","type":"text/csv","lastModified":1712901500621.8,"size":1940929},{"id":"a9451d129baa0f3753ca2c3830bef4c0010d1100e37f9bec37848efb64db6ff811ae671a756ea30c4f1ce6c647f53541ed7dac781fbdad08a1c9b7187d1a789c","href":"https://static.observableusercontent.com/files/a9451d129baa0f3753ca2c3830bef4c0010d1100e37f9bec37848efb64db6ff811ae671a756ea30c4f1ce6c647f53541ed7dac781fbdad08a1c9b7187d1a789c","name":"longest_patient_sentence@7.csv","type":"text/csv","lastModified":1712901963581.225,"size":1988920},{"id":"b6d749bbdc616e8c3b0e60a8f28b58162dde8269850ee56ae843bff1420276a0fe16e04904cbb82b58250c765066f53b9ef35c31ce20e647261c765e2fee593f","href":"https://static.observableusercontent.com/files/b6d749bbdc616e8c3b0e60a8f28b58162dde8269850ee56ae843bff1420276a0fe16e04904cbb82b58250c765066f53b9ef35c31ce20e647261c765e2fee593f","name":"synthetic@4.csv","type":"text/csv","lastModified":1712902683531.539,"size":1995366},{"id":"a4759060642be1b8f34754db92edc26bb8d83759554d90c31cd8ef81e9a79f9af9cc263789f8e826791467c3de8f61f7243e5958f77f813e7faac8b461807abf","href":"https://static.observableusercontent.com/files/a4759060642be1b8f34754db92edc26bb8d83759554d90c31cd8ef81e9a79f9af9cc263789f8e826791467c3de8f61f7243e5958f77f813e7faac8b461807abf","name":"medications.csv","type":"text/csv","lastModified":1711419816525.126,"size":2096315},{"id":"b79e713b7cddb0838597c56f8b0a44b1345b9b0a98b4483512a7c948ac1ea741623310687514945a317938f47b341d7625f813a0fc611838c274aaabe44def44","href":"https://static.observableusercontent.com/files/b79e713b7cddb0838597c56f8b0a44b1345b9b0a98b4483512a7c948ac1ea741623310687514945a317938f47b341d7625f813a0fc611838c274aaabe44def44","name":"first_name.csv","type":"text/csv","lastModified":1711419818065.868,"size":4466304},{"id":"726d16decb6f1a63fccd7a0cd84cce11982c45220670e707b1a7fce44095a73f85c5108a932f2ba91a4605a192e99d5592f7c6dc407eb087f06a3b2a6baa770c","href":"https://static.observableusercontent.com/files/726d16decb6f1a63fccd7a0cd84cce11982c45220670e707b1a7fce44095a73f85c5108a932f2ba91a4605a192e99d5592f7c6dc407eb087f06a3b2a6baa770c","name":"longest_patient_sentence.csv","type":"text/csv","lastModified":1711419818375.617,"size":5154124},{"id":"b79e713b7cddb0838597c56f8b0a44b1345b9b0a98b4483512a7c948ac1ea741623310687514945a317938f47b341d7625f813a0fc611838c274aaabe44def44","href":"https://static.observableusercontent.com/files/b79e713b7cddb0838597c56f8b0a44b1345b9b0a98b4483512a7c948ac1ea741623310687514945a317938f47b341d7625f813a0fc611838c274aaabe44def44","name":"first_name@1.csv","type":"text/csv","lastModified":1711419913557.893,"size":4466304},{"id":"915560d8513ebce8541fd4c94c44d61e36c70fdd40d689b6c58d98751f19c507cc309dfe0b0f689329c2b9930cd25d71697b66e19c7f4ad09d90748c0357a323","href":"https://static.observableusercontent.com/files/915560d8513ebce8541fd4c94c44d61e36c70fdd40d689b6c58d98751f19c507cc309dfe0b0f689329c2b9930cd25d71697b66e19c7f4ad09d90748c0357a323","name":"synthetic@3.csv","type":"text/csv","lastModified":1712191060749.28,"size":7568799},{"id":"a4759060642be1b8f34754db92edc26bb8d83759554d90c31cd8ef81e9a79f9af9cc263789f8e826791467c3de8f61f7243e5958f77f813e7faac8b461807abf","href":"https://static.observableusercontent.com/files/a4759060642be1b8f34754db92edc26bb8d83759554d90c31cd8ef81e9a79f9af9cc263789f8e826791467c3de8f61f7243e5958f77f813e7faac8b461807abf","name":"medications@1.csv","type":"text/csv","lastModified":1711428122830.433,"size":2096315},{"id":"726d16decb6f1a63fccd7a0cd84cce11982c45220670e707b1a7fce44095a73f85c5108a932f2ba91a4605a192e99d5592f7c6dc407eb087f06a3b2a6baa770c","href":"https://static.observableusercontent.com/files/726d16decb6f1a63fccd7a0cd84cce11982c45220670e707b1a7fce44095a73f85c5108a932f2ba91a4605a192e99d5592f7c6dc407eb087f06a3b2a6baa770c","name":"longest_patient_sentence@1.csv","type":"text/csv","lastModified":1711433100633.294,"size":5154124},{"id":"dbfc0639c1a56753a943072fbadd32c9d7bf460936b556fbb2c2bc70cecec0df5851179ec684f9df4e4209e6bd8f8c43699a9b906065cc9b96f5f10370293591","href":"https://static.observableusercontent.com/files/dbfc0639c1a56753a943072fbadd32c9d7bf460936b556fbb2c2bc70cecec0df5851179ec684f9df4e4209e6bd8f8c43699a9b906065cc9b96f5f10370293591","name":"medications@2.csv","type":"text/csv","lastModified":1711436002527.527,"size":2096187},{"id":"95004e079597f8889715859e89471ff65292bc0ffaac40a9dfe2231a49c7d984694817e46163d2916fc62ecaca77fcf113278af86cf2fcf5071bdb814e49d2d7","href":"https://static.observableusercontent.com/files/95004e079597f8889715859e89471ff65292bc0ffaac40a9dfe2231a49c7d984694817e46163d2916fc62ecaca77fcf113278af86cf2fcf5071bdb814e49d2d7","name":"medications@3.csv","type":"text/csv","lastModified":1711436237881.24,"size":1982364},{"id":"36306a8b4904180192ac763538a62f6624ee39cce30bd2eb526a47d041c9feab3de53ff43d4516dbb224683f15af1d6581573cd1a15f8379a08b5995985b6ea0","href":"https://static.observableusercontent.com/files/36306a8b4904180192ac763538a62f6624ee39cce30bd2eb526a47d041c9feab3de53ff43d4516dbb224683f15af1d6581573cd1a15f8379a08b5995985b6ea0","name":"longest_patient_sentence@2.csv","type":"text/csv","lastModified":1711437370178.27,"size":5154767},{"id":"62cf4931750b832bf9bcf1b93a0fa566876dfb0ee42e0fd0966df06ef9f631146654163d0e2ac153f88408b8b7f9c77089144175d54ad0b3fa4f319a1dbf6b0a","href":"https://static.observableusercontent.com/files/62cf4931750b832bf9bcf1b93a0fa566876dfb0ee42e0fd0966df06ef9f631146654163d0e2ac153f88408b8b7f9c77089144175d54ad0b3fa4f319a1dbf6b0a","name":"first_name@2.csv","type":"text/csv","lastModified":1711487519749.545,"size":5944429},{"id":"3e612f05cc8bbf6756157e7b1d80a5f77af6f7e88e079aae39fa6ecde27d537189af961a15979e25b08a664ff44cfde64f0f5ca0d452c79eec303535cdeb529e","href":"https://static.observableusercontent.com/files/3e612f05cc8bbf6756157e7b1d80a5f77af6f7e88e079aae39fa6ecde27d537189af961a15979e25b08a664ff44cfde64f0f5ca0d452c79eec303535cdeb529e","name":"medications@4.csv","type":"text/csv","lastModified":1711487620154.548,"size":2636477},{"id":"6394e7791e95eb1bb5b635524891cebc285f7836316bb61c7dbea5663dbe8451c7dfe0b0be2edda394b38cf7a2a1096e9c77bfaca51ed6f3a4b4ff4c537d7f0f","href":"https://static.observableusercontent.com/files/6394e7791e95eb1bb5b635524891cebc285f7836316bb61c7dbea5663dbe8451c7dfe0b0be2edda394b38cf7a2a1096e9c77bfaca51ed6f3a4b4ff4c537d7f0f","name":"first_name@3.csv","type":"text/csv","lastModified":1711487762364.529,"size":5940173},{"id":"cdb2d7eff333aeb4f64f5ff6137953afdab317606a059b503b32949af8a2d69cf3df38ecab1322d140c9b77708fcca30c92e402e41d9bd0f9d54e723f76406e9","href":"https://static.observableusercontent.com/files/cdb2d7eff333aeb4f64f5ff6137953afdab317606a059b503b32949af8a2d69cf3df38ecab1322d140c9b77708fcca30c92e402e41d9bd0f9d54e723f76406e9","name":"medications@5.csv","type":"text/csv","lastModified":1711487775257.014,"size":2634629},{"id":"3b98d4cb3bc4de0ba94013037d3203faa8091573fe627cc68de6070968fdca9f8d931382d6a98f985b89d8366b059770cfcb8e3913748901aa8309f487e10691","href":"https://static.observableusercontent.com/files/3b98d4cb3bc4de0ba94013037d3203faa8091573fe627cc68de6070968fdca9f8d931382d6a98f985b89d8366b059770cfcb8e3913748901aa8309f487e10691","name":"longest_patient_sentence@3.csv","type":"text/csv","lastModified":1711487892145.124,"size":6848274},{"id":"463f795a70f882fe6ca665856fa0c0359e4cd8d6efa7a0f1ecbd090a18b1797031190571b69890a791d933c86b343bb12dacf945c8a4f4d9fbb2d95b156650f5","href":"https://static.observableusercontent.com/files/463f795a70f882fe6ca665856fa0c0359e4cd8d6efa7a0f1ecbd090a18b1797031190571b69890a791d933c86b343bb12dacf945c8a4f4d9fbb2d95b156650f5","name":"synthetic.csv","type":"text/csv","lastModified":1711516660727.71,"size":234096},{"id":"a23f64aadc5d91fe086d64fb64e6852e531400fc9651235e838cb76184e26d78b8f03a6e68e6d24619d3bfd57f86bcfee05f6d545c443a3ace07dc3a9bdd791c","href":"https://static.observableusercontent.com/files/a23f64aadc5d91fe086d64fb64e6852e531400fc9651235e838cb76184e26d78b8f03a6e68e6d24619d3bfd57f86bcfee05f6d545c443a3ace07dc3a9bdd791c","name":"synthetic@1.csv","type":"text/csv","lastModified":1711516726739.395,"size":6744891},{"id":"74226d064ea642e3eea44cfa96171cf8e93c50fb43988cfea650c7c7dbd96cfe53efa7f32f84a6b890138bbc19a822ad6f2e5ffa2d3a2f1df79c4eca58c4f569","href":"https://static.observableusercontent.com/files/74226d064ea642e3eea44cfa96171cf8e93c50fb43988cfea650c7c7dbd96cfe53efa7f32f84a6b890138bbc19a822ad6f2e5ffa2d3a2f1df79c4eca58c4f569","name":"synthetic@2.csv","type":"text/csv","lastModified":1711519055165.605,"size":6758207},{"id":"6394e7791e95eb1bb5b635524891cebc285f7836316bb61c7dbea5663dbe8451c7dfe0b0be2edda394b38cf7a2a1096e9c77bfaca51ed6f3a4b4ff4c537d7f0f","href":"https://static.observableusercontent.com/files/6394e7791e95eb1bb5b635524891cebc285f7836316bb61c7dbea5663dbe8451c7dfe0b0be2edda394b38cf7a2a1096e9c77bfaca51ed6f3a4b4ff4c537d7f0f","name":"first_name@4.csv","type":"text/csv","lastModified":1711519344921.458,"size":5940173},{"id":"9c0994ff150b12801f3ec6fbe7b4a96452fd0bc0bee194b2ed1e6682ba27085b8a23afd3b6e7f2d9b68e3380a51b81844f14b735a659da030601fbe3689bf51d","href":"https://static.observableusercontent.com/files/9c0994ff150b12801f3ec6fbe7b4a96452fd0bc0bee194b2ed1e6682ba27085b8a23afd3b6e7f2d9b68e3380a51b81844f14b735a659da030601fbe3689bf51d","name":"medications@6.csv","type":"text/csv","lastModified":1711519360124.467,"size":2639793},{"id":"73f9349c47f99dd942edaba55bd1ac0d9b3eeb59853a273841f191213294d2084cd6512e7d095d501a7e8a5e47192ae1d6e1bf1deb781d2ed88fac4b024b9b25","href":"https://static.observableusercontent.com/files/73f9349c47f99dd942edaba55bd1ac0d9b3eeb59853a273841f191213294d2084cd6512e7d095d501a7e8a5e47192ae1d6e1bf1deb781d2ed88fac4b024b9b25","name":"longest_patient_sentence@4.csv","type":"text/csv","lastModified":1711519397801.685,"size":6861606}] 2f:{"title":"Needle in the Real World","cells":[{"id":490,"value":"# Needle in the Real World","pinned":false,"hidden":false,"mode":"md"},{"id":330,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":306,"value":"## Simple: What is the first name of the patient?","pinned":false,"hidden":false,"mode":"md"},{"id":638,"value":"easyQuestion = html`

\n ${easyPlot}\n

\n ${nameScatterPlot}\n

`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":630,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":313,"value":"### Takeaways\n\n* gemma is pretty bad; the rest of the models are good\n* typically the name is mentioned in the first couple sentences of the transcript. if not, the name is likely mentioned in the end of the transcript (e.g., \"Bye [name], have a good day\")\n* DBRX gives super long answers?\n* this is the most basic of questions to make sure the pipeline works, and TLDR: it works","pinned":false,"hidden":false,"mode":"md"},{"id":424,"value":"#### Full Table of Responses","pinned":false,"hidden":false,"mode":"md"},{"id":610,"value":"first_name = await FileAttachment(\"first_name@5.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":309,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":283,"value":"## Medium: What medications is the patient currently taking?\n\nI realize this question is grammatically incorrect; oh well.\n\nRecall is defined as the proportion of words in the reference answer that sound like medicine names and are longer than 5 characters, which are also found in the model's output. I did a lot of manual blacklisting of words/curation to get words that sound like medicine names, but I bet my logic isn't exhaustive.","pinned":false,"hidden":false,"mode":"md"},{"id":278,"value":"mediumQuestion = html`

\n ${avgRecallByModel}\n ${medBoxPlot}\n

\n ${medScatterPlot}\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":437,"value":"#### Full Table of Responses","pinned":false,"hidden":false,"mode":"md"},{"id":613,"value":"medications = await FileAttachment(\"medications@7.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":126,"value":"avgRecallByModel = {let averageRecallByModel = d3.rollups(medications, \n v => d3.mean(v, d => d.recall), \n d => d.model)\n .map(([model, recall]) => ({ model, recall }));\n\nreturn Plot.plot({\n style: {\n fontSize: 16\n },\n x: {\n padding: 0.4,\n label: \"→ Model\",\n },\n y: {\n label: \"↑ Recall\", \n },\n color: {\n legend: true,\n },\n marks: [\n Plot.barY(averageRecallByModel, {x: \"model\", y: \"recall\", dx: 2, dy: 2}),\n Plot.barY(averageRecallByModel, {x: \"model\", y: \"recall\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(averageRecallByModel, Plot.pointerX({x: \"model\", y: \"recall\"}))\n ],\n title: \"Avg Recall of Reference Answer in Model Outputs\",\n marginBottom: 50,\n marginTop: 45\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":113,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":311,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":215,"value":"avgSimilarityByModel = {const averageSimilarityByModel = d3.rollups(sentences, \n v => d3.mean(v, d => d.similarity), \n d => d.model)\n .map(([model, similarity]) => ({ model, similarity }));\n\n console.log(averageSimilarityByModel);\n console.log(alphabet);\n\nreturn Plot.plot({\n x: {padding: 0.4},\n tip: true,\n marks: [\n Plot.barY(averageSimilarityByModel, {x: \"model\", y: \"similarity\", dx: 2, dy: 2}),\n Plot.barY(averageSimilarityByModel, {x: \"model\", y: \"similarity\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(averageSimilarityByModel, Plot.pointerX({x: \"model\", y: \"similarity\"}))\n ],\n title: \"Avg Similarity Between Outputs and Reference Answer\",\n style: {fontSize: 16},\n marginBottom: 50,\n marginTop: 45,\n marginLeft: 50\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":272,"value":"## Hard: What is the longest sentence the patient said?\n\nSimilarity is Jaccard similarity","pinned":false,"hidden":false,"mode":"md"},{"id":234,"value":"hardQuestion = html`\n \n \n \n

\n ${avgSimilarityByModel}\n ${sentenceBoxPlot}\n

\n ${sentenceScatterPlot}\n

\n \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":326,"value":"### Takeaways\n\n* GPT-4 seems significantly better than the other models, and the top two models are openai models\n* All models degrade in performance when the answer is in the second and third segments of the transcript (note that segments are equally sized)\n* Transcript quality is _really bad_, often the quotes that are labeled \"patient\" seem like they are coming from the doctor (and vice versa). I wonder what would happen if we flip the patient-doctor labels and rerun this experiment :-) \n* Mistral seems to attach a number of words to its response? no idea why, but these are not always correct\n* The better-performing models give a consistent structure\n* Gemini is just...bad?\n* DBRX is _really_ bad...maybe i prompted it wrong? i did copy the system prompt they gave in huggingface","pinned":false,"hidden":false,"mode":"md"},{"id":439,"value":"#### Full Table of Responses","pinned":false,"hidden":false,"mode":"md"},{"id":618,"value":"sentences = await FileAttachment(\"longest_patient_sentence@5.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":220,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":225,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":383,"value":"## A Final Test: Synthetic Eval\n\nHow do the models perform on the standard needle-in-the-haystack eval, where the haystack is our transcripts?\n\nTo test this, we leverage a similar experimental setup to [what's been done before](https://blog.langchain.dev/multi-needle-in-a-haystack/), but we focus on retrieving a single needle.\n\nFor each transcript, we randomly select 3 toppings from a list of 10 pizza toppings, and concatenate them into a sentence. E.g., _The secret ingredients needed to build the perfect pizza are: Espresso-soaked dates, Lemon and Goat cheese._\n\nNext, we (uniformly) randomly pick an index in the document to insert this sentence.\n\nThen, we ask each model the following question: _What are the secret ingredients needed to build the perfect pizza?_\n\nFor each model and document, we compute the recall of secret ingredients (i.e., 0, 0.33, 0.67, or 1.0).\n","pinned":false,"hidden":false,"mode":"md"},{"id":396,"value":"syntheticViz = html`\n \n \n \n

\n ${avgSyntheticRecallByModel}\n ${syntheticBoxPlot}\n

\n ${syntheticScatterPlot}\n

\n \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":414,"value":"$39","pinned":false,"hidden":false,"mode":"md"},{"id":445,"value":"#### Full Table of Responses","pinned":false,"hidden":false,"mode":"md"},{"id":648,"value":"synthetic = await FileAttachment(\"synthetic@3.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":391,"value":"avgSyntheticRecallByModel = {const avgSyntheticRecallByModel = d3.rollups(synthetic, \n v => d3.mean(v, d => d.recall), \n d => d.model)\n .map(([model, recall]) => ({ model, recall }));\n\n\nreturn Plot.plot({\n x: {padding: 0.4},\n tip: true,\n marks: [\n Plot.barY(avgSyntheticRecallByModel, {x: \"model\", y: \"recall\", dx: 2, dy: 2}),\n Plot.barY(avgSyntheticRecallByModel, {x: \"model\", y: \"recall\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(avgSyntheticRecallByModel, Plot.pointerX({x: \"model\", y: \"recall\"}))\n ],\n title: \"Avg Recall of Needles\",\n style: {fontSize: 16}\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":399,"value":"syntheticScatterPlot = {\n const scatterPlot = Plot.plot({\n marks: [\n Plot.dot(synthetic, {x: \"idx\", y: \"recall\", fill: \"model\", title: d => `${d.model}: Idx ${d.idx}, Recall ${d.recall}`, r: 6,\n}),\n Plot.frame(),\n Plot.tip(synthetic, Plot.pointer({\n x: \"idx\",\n y: \"recall\",\n stroke: \"model\",\n title: (d) => [`REFERENCE:\\n${d.needles}`, `OUTPUT:\\n${d.result}`].join(\"\\n\\n\") // Custom tooltip format\n }))\n ],\n height: 800,\n marginRight: 90,\n marginLeft: 110,\n grid: true,\n facet: {\n data: synthetic,\n y: \"model\"\n },\n x: {\n label: \"Idx →\",\n nice: true\n },\n y: {\n label: \"↑ Recall\",\n domain: d3.extent(synthetic, d => d.recall), // Ensure Y-axis covers the full recall range\n nice: true,\n inset: 5\n },\n color: {\n legend: false\n },\n title: \"Recall vs. Position of Needle in Transcript\",\n style: {fontSize: 16}\n});\n\n return scatterPlot;\n\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":404,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs"},{"id":665,"value":"$3b","pinned":false,"hidden":false,"mode":"md"},{"id":669,"value":"## What is the first name of the patient?","pinned":false,"hidden":false,"mode":"md"},{"id":682,"value":"easyQuestionSynthetic = html`

\n ${easyPlotSynthetic}\n

\n ${nameScatterPlotSynthetic}\n

`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":725,"value":"### Takeaways\n\nThis is near 100% accuracy for all models.","pinned":false,"hidden":false,"mode":"md"},{"id":671,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs"},{"id":679,"value":"$3d","pinned":false,"hidden":false,"mode":"ojs"},{"id":673,"value":"first_name_synthetic = await FileAttachment(\"first_name@6.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":686,"value":"## Medium: What medications is the patient currently taking?","pinned":false,"hidden":false,"mode":"md"},{"id":697,"value":"mediumQuestionSynthetic = html`

\n ${avgRecallByModelSynthetic}\n ${medBoxPlotSynthetic}\n

\n ${medScatterPlotSynthetic}\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":727,"value":"### Takeaways\n\nThis is also basically 100% accuracy for all models.","pinned":false,"hidden":false,"mode":"md"},{"id":691,"value":"avgRecallByModelSynthetic = {let averageRecallByModel = d3.rollups(medications_synthetic, \n v => d3.mean(v, d => d.recall), \n d => d.model)\n .map(([model, recall]) => ({ model, recall }));\n\nreturn Plot.plot({\n style: {\n fontSize: 16\n },\n x: {\n padding: 0.4,\n label: \"→ Model\",\n },\n y: {\n label: \"↑ Recall\", \n },\n color: {\n legend: true,\n },\n marks: [\n Plot.barY(averageRecallByModel, {x: \"model\", y: \"recall\", dx: 2, dy: 2}),\n Plot.barY(averageRecallByModel, {x: \"model\", y: \"recall\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(averageRecallByModel, Plot.pointerX({x: \"model\", y: \"recall\"}))\n ],\n title: \"Avg Recall of Reference Answer in Model Outputs (on Synthetic Transcripts)\",\n marginBottom: 50,\n marginTop: 45\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":693,"value":"$3e","pinned":false,"hidden":false,"mode":"ojs"},{"id":688,"value":"medications_synthetic = await FileAttachment(\"medications@8.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":695,"value":"$3f","pinned":false,"hidden":false,"mode":"ojs"},{"id":699,"value":"## Hard: What is the longest sentence the patient said?","pinned":false,"hidden":false,"mode":"md"},{"id":712,"value":"hardQuestionSynthetic = html`\n \n \n \n

\n ${avgSimilarityByModelSynthetic}\n ${sentenceBoxPlotSynthetic}\n

\n ${sentenceScatterPlotSynthetic}\n

\n \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":729,"value":"### Takeaways\n\nI am surprised by how much _worse_ performance is on this synthetic transcript set than the real transcript. My hypothesis is that in the synthetic transcripts, the sentences are more or less of the same length, so it's actually pretty difficult to get exactly the right longest sentence (see the histogram below, where there are lots of blue outliers). Also, I guess I wasn't precise in the question--is it the longest sentence by # words or by # characters? My \"ground truth\" is computed with # characters.","pinned":false,"hidden":false,"mode":"md"},{"id":751,"value":"$40","pinned":false,"hidden":false,"mode":"ojs"},{"id":703,"value":"avgSimilarityByModelSynthetic = {const averageSimilarityByModel = d3.rollups(sentences_synthetic, \n v => d3.mean(v, d => d.similarity), \n d => d.model)\n .map(([model, similarity]) => ({ model, similarity }));\n\n console.log(averageSimilarityByModel);\n console.log(alphabet);\n\nreturn Plot.plot({\n x: {padding: 0.4},\n tip: true,\n marks: [\n Plot.barY(averageSimilarityByModel, {x: \"model\", y: \"similarity\", dx: 2, dy: 2}),\n Plot.barY(averageSimilarityByModel, {x: \"model\", y: \"similarity\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(averageSimilarityByModel, Plot.pointerX({x: \"model\", y: \"similarity\"}))\n ],\n title: \"Avg Similarity Between Outputs and Reference Answer (on Synthetic Transcripts)\",\n style: {fontSize: 16},\n marginBottom: 50,\n marginTop: 45,\n marginLeft: 50\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":707,"value":"$41","pinned":false,"hidden":false,"mode":"ojs"},{"id":710,"value":"$42","pinned":false,"hidden":false,"mode":"ojs"},{"id":701,"value":"sentences_synthetic = await FileAttachment(\"longest_patient_sentence@7.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":731,"value":"## Synthetic Eval on Synthetic Transcripts :-)\n\nWe use the same needle-in-a-haystack pizza-related eval.","pinned":false,"hidden":false,"mode":"md"},{"id":741,"value":"syntheticVizSynthetic = html`\n \n \n \n

\n ${avgSyntheticRecallByModelSynthetic}\n ${syntheticBoxPlotSynthetic}\n

\n ${syntheticScatterPlotSynthetic}\n

\n \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":749,"value":"### Takeaways\n\nOk, this is really weird. Performance got worse for all models except GPT-3.5 and Mistral. Can't explain why...\n\nInspecting the outputs, it seems like most outputs where where recall = 0 reference something like \"The conversation is focused on a conversation between a doctor and patient...\"\n\nI wonder if, because these synthetic transcripts are, by definition, in-distribution for LLMs, inserting some phrase related to the secret ingredients for pizza is quite \"jarring\" for the models.\" But who knows.","pinned":false,"hidden":false,"mode":"md"},{"id":735,"value":"avgSyntheticRecallByModelSynthetic = {const avgSyntheticRecallByModel = d3.rollups(synthetic_synthetic, \n v => d3.mean(v, d => d.recall), \n d => d.model)\n .map(([model, recall]) => ({ model, recall }));\n\n\nreturn Plot.plot({\n x: {padding: 0.4},\n tip: true,\n marks: [\n Plot.barY(avgSyntheticRecallByModel, {x: \"model\", y: \"recall\", dx: 2, dy: 2}),\n Plot.barY(avgSyntheticRecallByModel, {x: \"model\", y: \"recall\", fill: \"green\", dx: -2, dy: -2}),\n Plot.tip(avgSyntheticRecallByModel, Plot.pointerX({x: \"model\", y: \"recall\"}))\n ],\n title: \"Avg Recall of Needles (on Synthetic Transcripts)\",\n style: {fontSize: 16}\n})}","pinned":false,"hidden":false,"mode":"ojs"},{"id":737,"value":"syntheticScatterPlotSynthetic = {\n const scatterPlot = Plot.plot({\n marks: [\n Plot.dot(synthetic_synthetic, {x: \"idx\", y: \"recall\", fill: \"model\", title: d => `${d.model}: Idx ${d.idx}, Recall ${d.recall}`, r: 6,\n}),\n Plot.frame(),\n Plot.tip(synthetic_synthetic, Plot.pointer({\n x: \"idx\",\n y: \"recall\",\n stroke: \"model\",\n title: (d) => [`REFERENCE:\\n${d.needles}`, `OUTPUT:\\n${d.result}`].join(\"\\n\\n\") // Custom tooltip format\n }))\n ],\n height: 800,\n marginRight: 90,\n marginLeft: 110,\n grid: true,\n facet: {\n data: synthetic_synthetic,\n y: \"model\"\n },\n x: {\n label: \"Idx →\",\n nice: true\n },\n y: {\n label: \"↑ Recall\",\n domain: d3.extent(synthetic_synthetic, d => d.recall), // Ensure Y-axis covers the full recall range\n nice: true,\n inset: 5\n },\n color: {\n legend: false\n },\n title: \"Recall vs. Position of Needle in Transcript (on Synthetic Transcripts)\",\n style: {fontSize: 16}\n});\n\n return scatterPlot;\n\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":739,"value":"$43","pinned":false,"hidden":false,"mode":"ojs"},{"id":733,"value":"synthetic_synthetic = await FileAttachment(\"synthetic@4.csv\").csv({typed: \"auto\"})","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W44","stdlib":"1","titling":"auto"}