3e:null 3f:["$","title",null,{"children":"Evaluating LLM Cost on Basis of Price of Performance | Kallol Chatterjee | Observable"}] 45:Tb22,

This scatter plot visualizes the relationship between the operational output cost of leading Large Language Models (LLMs) and their actual performance across key business metrics. By plotting the Overall Pass Rate (based on an average score threshold of >80) against the Output Cost per 1M tokens, the chart highlights the aggressive price scaling associated with proprietary "celebrity" models compared to high-value alternatives.

Key Insights

The Output Value Gap: While input costs are often comparable, output costs reveal the true operational expense of these models. Krista LLM demonstrates a significant advantage, maintaining an 83.4% pass rate at an industry-leading output cost of only $0.10 per 1M tokens.
The Efficiency Frontier: Open-source models like gpt-oss-120b (84.2% pass rate at $0.60) provide superior performance-to-price ratios compared to proprietary "mini" models with similar price points.
Premium Reasoning Costs: Top-tier performers such as gemini-3-pro-preview (87.5% pass rate) and claude-sonnet-4.5 (84.3% pass rate) sit at the far right of the scale, with output costs reaching $12.00 to $15.00 per 1M tokens.
Obsolescence in the Mid-Tier: Legacy models like gpt-3.5-turbo show a clear lack of utility in the modern landscape, appearing in the "low-value" zone with lower performance and higher output costs than several state-of-the-art alternatives.

46:T689,data = [ { model: "gemini-3-pro-preview", passRate: 87.5, outputCost: 12.00, type: "Closed" }, { model: "o3-mini (Reasoning)", passRate: 87.0, outputCost: 4.40, type: "Closed" }, { model: "gpt-5.2", passRate: 85.7, outputCost: 14.00, type: "Closed" }, { model: "gpt-4.1-mini", passRate: 85.7, outputCost: 1.60, type: "Closed" }, { model: "claude-sonnet-4.5", passRate: 84.3, outputCost: 15.00, type: "Closed" }, { model: "gpt-oss-120b", passRate: 84.2, outputCost: 0.60, type: "Open" }, { model: "gpt-4.1", passRate: 83.8, outputCost: 8.00, type: "Closed" }, { model: "Krista LLM", passRate: 83.4, outputCost: 0.10, type: "Closed" }, { model: "Kimi-K2-Instruct", passRate: 83.3, outputCost: 2.50, type: "Open" }, { model: "claude-sonnet-4", passRate: 83.3, outputCost: 15.00, type: "Closed" }, { model: "gpt-oss-20b", passRate: 83.1, outputCost: 0.30, type: "Open" }, { model: "Llama-3.3-70B-Instruct", passRate: 81.7, outputCost: 0.90, type: "Open" }, { model: "gpt-5", passRate: 81.6, outputCost: 10.00, type: "Closed" }, { model: "gpt-5-mini", passRate: 81.5, outputCost: 2.00, type: "Closed" }, { model: "grok-4-1-fast", passRate: 81.2, outputCost: 0.50, type: "Closed" }, { model: "gpt-5.1", passRate: 81.1, outputCost: 10.00, type: "Closed" }, { model: "Grok-4-fast", passRate: 81.0, outputCost: 0.50, type: "Closed" }, { model: "Llama-4-Maverick", passRate: 80.6, outputCost: 0.88, type: "Open" }, { model: "gpt-4o", passRate: 80.2, outputCost: 10.00, type: "Closed" }, { model: "gpt-4o-mini", passRate: 74.6, outputCost: 0.60, type: "Closed" }, { model: "gpt-3.5-turbo", passRate: 64.2, outputCost: 1.50, type: "Closed" } ]47:[] 43:{"id":"baf4c47d17adecc1","slug":null,"title":"Evaluating LLM Cost on Basis of Price of Performance","thumbnail":"078ee01c75a0903ed33376dfabb0cd3d79c73751e37c00b22f7255996faf1a85","access_level":"public","create_time":"$D2026-03-30T18:37:45.087Z","update_time":"$D2026-03-31T12:18:48.977Z","view_time":null,"first_public_time":"$D2026-03-31T12:15:16.557Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"dc97a2766ef9be06","login":"k-chat","name":"Kallol Chatterjee","avatar_url":"https://avatars-next.observableusercontent.com/f31a56f01fed3c9a0499394b5920853e37295852e119e35d12c6a0f8848a4d84","tier":"starter_2024","bio":"","following":false},"version":26,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"a092fa1ed8517f3f","login":"k-chat","name":"Kallol Chatterjee","avatar_url":"https://avatars-next.observableusercontent.com/f31a56f01fed3c9a0499394b5920853e37295852e119e35d12c6a0f8848a4d84","tier":null,"bio":"","following":false}],"members":[{"type":"individual","id":"a092fa1ed8517f3f","login":"k-chat","name":"Kallol Chatterjee","avatar_url":"https://avatars-next.observableusercontent.com/f31a56f01fed3c9a0499394b5920853e37295852e119e35d12c6a0f8848a4d84","tier":null,"role":"editor"},{"type":"team","id":"dc97a2766ef9be06","login":"k-chat","name":"Kallol Chatterjee","avatar_url":"https://avatars-next.observableusercontent.com/f31a56f01fed3c9a0499394b5920853e37295852e119e35d12c6a0f8848a4d84","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Evaluating LLM Cost on Basis of Price of Performance","cells":[{"id":0,"value":"# Evaluating LLM Cost on
Basis of Price of Performance\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":12,"value":"Plot.plot({\n grid: true,\n width: 1500,\n height: 1000,\n marginTop: 40,\n marginRight: 100,\n x: { \n label: \"Output Cost ($ per 1M tokens) →\", \n type: \"log\", \n domain: [0.08, 20]\n },\n y: { \n label: \"↑ Overall Pass Rate (%)\", \n domain: [60, 95]\n },\n color: {\n legend: true,\n domain: [\"Closed\", \"Open\"],\n range: [\"#2196F3\", \"#4CAF50\"]\n },\n marks: [\n Plot.ruleY([80], {strokeDasharray: \"4,4\", strokeOpacity: 0.5}), // Benchmark\n Plot.dot(data, {\n x: \"outputCost\",\n y: \"passRate\",\n fill: \"type\",\n r: 6,\n tip: true\n }),\n Plot.text(data, {\n x: \"outputCost\",\n y: \"passRate\",\n text: \"model\",\n dy: -14,\n fontSize: 12,\n fontWeight: \"bold\"\n })\n ]\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"$45","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":10,"value":"$46","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W47","stdlib":"1"}} 44:"$43:body"