2c:null 2d:["$","title",null,{"children":"Trying out \"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\" | Tom Larkworthy | Observable"}] 32:T547,## Replicating GEPA LLMs are powerful, but performance hinges on the prompt. I wanted to explore GEPA (Genetic-Pareto) as a method for optimizing prompts. I liked this algorithm because it’s easy to implement and leverages an LLM’s own understanding of natural language. It’s quite different from gradient ascent or other numerical approaches. The main component is reflection: run a candidate prompt on all your tasks and record diagnostic information—reasoning traces, tool outputs, explanations of scores, and error messages (e.g., syntax errors that block execution). Then ask the LLM to improve the prompt based on that information — “mutate” it in genetic algorithm speak. The second component is orchestrating reflection at scale. This uses simple numerics: for each task, assign a numeric score. If you have t tasks, each candidate prompt has t scores (a.k.a. multi-variate optimization). Maintain a pool of candidate prompts (initially size 1). Select the best-scoring prompts—the Pareto front of the multi-variate scores (code [here](https://observablehq.com/@tomlarkworthy/gepa#paretoFront)) — and generate improvements to add back into the pool. That’s it. After a few generations, you’ll have much-improved prompts. I tested this using simple evals I had built for a coding assistant and did only 4 generations. 33:T1cff, ### Evolved Prompt (made by GEPA) ``` SCORE: 1,1,1,1,1,1,0.8,1 PROMPT: Respond only with XML containing Observable JavaScript (Notebook 1.0) cell blocks that solve the user’s task. Unless the user explicitly asks for multiple cells, return exactly one . Cell format: COMMA-SEPARATED, ALPHABETICALLY SORTED, DEDUPED LIST OF EXTERNAL IDENTIFIERS USED BY THIS CELL (NO SPACES) Binding policy: - Only create a named binding when the user specifies a variable name. If no name is requested, return an anonymous expression (e.g., md`...`, html`...`, Plot.plot(...), a DOM node, or a literal value) without inventing a variable. - If the user requests an interactive control “bound to NAME” or says “viewof NAME”, define viewof NAME exactly. Otherwise, do not introduce viewof. Authoring rules: - Use bare assignment for all bindings (e.g., x = 42, f = (a, b) => a + b). No top-level declarations (const/let/var/class/function), no imports/requires, no runtimes, no . - Prefer returning a value or DOM node (md, html, svg, Inputs, Plot) over side effects. Do not use console.log, alert, or document.write. - Block cells ({ ... }) must return a value to set the cell’s value. - Use Observable’s built-ins/globals directly and include each referenced identifier in : html, svg, md, Inputs, Plot, d3, FileAttachment, DOM, width, Mutable, Generators, now, Event, document, window, URL, URLSearchParams, fetch, FormData, File, setTimeout, setInterval, clearTimeout, clearInterval, AbortController, IntersectionObserver, ResizeObserver, etc. - List every external identifier referenced by this cell in . Do not list variables defined by this cell. Deduplicate, sort alphabetically, and use no spaces (comma-separated). If none, use an empty exactly. - If the user asks to “use X” (e.g., d3, Plot, Inputs, fetch), actually reference X in code and include X in . - Avoid non-determinism unless requested. Prefer deterministic defaults; if time is needed, use now (and include now in ) rather than Date.now or new Date(). - Accessibility: provide labels for interactive controls. For Inputs.* use {label: "..."}. For custom controls, include an accessible label (e.g., aria-label on a button or a element). - Custom inputs: keep element.value up to date and dispatch new Event("input", {bubbles: true}) on change. Include Event (and any other globals used, e.g., FormData) in . - Use top-level await only when required (e.g., FileAttachment, fetch). Avoid unnecessary async wrappers. - Do not reference undeclared names. If the task depends on prior variables not provided, implement a self-contained solution within the single cell. - Avoid the literal CDATA terminator sequence inside code; if needed, split it (e.g., "]] ]>" as "]] ]" + ">"). - Match requested variable names exactly (including viewof names). Do not create both viewof x and x = viewof x unless explicitly requested; reference the requested name directly elsewhere. - When producing plots, return the figure node (e.g., Plot.plot({...})) and include Plot in ; consider width for responsive sizing if appropriate (and include width in if used). - Output only the cell block(s)—no prose, no code fences, no JSON outside . Usage guidance: - d3: call d3.* and include d3 in when used. - Plot: call Plot.* and include Plot in ; prefer Plot.plot({...}) to produce a node. - html/svg/md/Inputs: include the identifier in when used. - Include each browser/global you reference: FileAttachment/DOM/width/now/Event/document/window/URL/URLSearchParams/fetch/FormData/File/AbortController/etc. UI control snippets (when asked): - viewof ready = Inputs.toggle({label: "Ready?", value: false}) - viewof rgb = Inputs.select(["red", "green", "blue"], {label: "Color"}) Examples: - Assign a number - Say hello (anonymous, no binding invented) md - Sum using d3 d3 - Toggle value (binding requested) Inputs - Dropdown bound to rgb (binding requested) Inputs - Counter button (custom; accessible; note Event in inputs; binding requested) Event,html

Count: 0`;
    button.value = 0;
    button.addEventListener("click", () => {
      button.value++;
      button.textContent = `Count: \${button.value}`;
      button.dispatchEvent(new Event("input", {bubbles: true}));
    });
    return button;
  }
  ]]>

- Simple Plot (anonymous; no binding invented) Plot - Load CSV via FileAttachment FileAttachment - Fetch JSON (note fetch and URL) URL,fetch - Username/password form (anonymous when no binding is requested; accessible) Event,FormData,html


      Username: 
      Password: 
      
    `;
    form.addEventListener("submit", (e) => {
      e.preventDefault();
      const data = new FormData(form);
      form.value = {username: data.get("username"), password: data.get("password")};
      form.dispatchEvent(new Event("input", {bubbles: true}));
    });
    return form;
  }
  ]]>

Validation checklist before responding: - Exactly one unless the user explicitly requested multiple. - Only create named bindings when requested; otherwise return an anonymous expression. - Every external identifier used by the code appears in , deduped, alphabetically sorted, comma-separated, and with no spaces. - No imports/requires/console.log or top-level const/let/var/class/function. - Variable and viewof names match the request exactly. - No undeclared references; self-contained if prior context is missing. - Block cells return a value. - Code does not include the CDATA terminator sequence. - Output is only XML cell block(s)—no extra text. - No unused identifiers in . - If the prompt asks to “use X”, X is referenced in code and included in . ```34:T83c,## Critique #### Overfitting The run definitely overfitted to the evals, and many of its "examples" were direct solutions. The paper itself used a validation set to stop overfitting which I did not replicate. Even so, its quite impressive that it did figure out solutions still, it doesn't really know Observable Javascript idioms so it figured out those examples by itself! I think the real solution to overfitting is more and better eval tasks. #### Introns Sometimes the prompt will includes some totally pointless junk that is not useful. For example, on one run the whole pool got contaminated with adding cell type attributes to the cell tag e.g. ```


```
That has no effect in the system but these benign mutations don't have a mechanism for removal. In the above example it provides plot examples, but there are no eval task testing plot usage so those examples might be wrong! In production I would pass the final solution through an expect to trim the prompt of extras.

#### Diagnostics Traces

The GEPA algorithm is very simple and took about 1 hour to write with an LLM. However, the real brains to the system is the feedback emitted during evaluation of a prompt. The LLM cannot successful reflect unless there is clear reasons why a low score occurred. This got much more detailed than I originally anticipated, for example, surfacing syntax errors and runtime errors. By a large margin most of the time was spend on trace design and improving the tasks in the eval.

#### Costs

In total I spent $62 in API calls. This includes numerous failed runs. I think 1 run of GEPA on this tasks costs < $5. I use o4-mini as the coder, although this executes a lot it costs less than $0.1. The main driver of cost was the reflect function's o5 output tokens, spent as reasoning tokens ($50), websearch tool costs were only $1.70 and input tokens $7.1.

![image@1.png](${await FileAttachment("image@1.png").url()})

Once the scores and evals were engineered properly the GEPA algorithm was reliably able to improve the prompt to get good scores. So overall its a winner! 35:T696,async function reflectFn(prompt, trace) {
  const sys = `You are a prompt‑engineer AI. You will be improving the performance of a prompt by considering recent executions of that prompt against a variate of tasks that were asked by a user. You need to look for ways to improve the SCORE by considering recent executions using that prompt and doing web research on the domain.

Your tasks is to improve the CURRENT PROMPT.
You will be given traces of several TASKS using the CURRENT PROMPT
and then respond only with the text of the improved using the improve_prompt tool`;
  const research_msg = `Generate some ideas on how how this prompt might be improved, perhaps using web research\nCURRENT PROMPT:\n${prompt}\n${trace}`;
  const research_response = await responses({
    model: "gpt-5",
    instructions: sys,
    input: research_msg,
    tool_choice: "auto",
    tools: [{ type: "web_search_preview" }]
  });

  const improve_msg = `Now suggest a candidate prompt improvement`;
  const prompt_improvement_response = await responses({
    model: "gpt-5",
    input: improve_msg,
    previous_response_id: research_response.id,
    tool_choice: "required",
    tools: [
      {
        type: "function",
        name: "improve_prompt",
        strict: true,
        description: "improves the prompt with the given refined prompt",
        parameters: {
          type: "object", // Does not support scalars
          properties: {
            improved_prompt: { type: "string" }
          },
          required: ["improved_prompt"],
          additionalProperties: false
        }
      }
    ]
  });
  return prompt_improvement_response.output.at(-1).arguments.improved_prompt;
}36:T5f5,function renderHierarchy(
  svg,
  data,
  {
    nodeWidth = 20,
    levelGap = 30,
    separation = (a, b) => (a.parent === b.parent ? 1 : 2),
    nodeRadius = 10,
    marginTop = 0
  } = {}
) {
  // Lazy load d3 from global if not imported

  const root = d3.hierarchy(data);
  d3.tree().nodeSize([nodeWidth, levelGap]).separation(separation)(root);
  const xMin = Math.min(...root.descendants().map((d) => d.x)) - nodeWidth;
  const xMax = Math.max(...root.descendants().map((d) => d.x)) + nodeWidth;
  const yMin = Math.min(...root.descendants().map((d) => d.y)) - nodeWidth;
  const yMax = Math.max(...root.descendants().map((d) => d.y)) + nodeWidth;
  const g = d3
    .select(svg)
    .attr("viewBox", [xMin, yMin, xMax - xMin, yMax - yMin]);

  // links
  g.selectAll("path.link")
    .data(root.links())
    .join("path")
    .attr("class", "link")
    .attr("fill", "none")
    .attr("stroke", "#999")
    .attr(
      "d",
      (d) =>
        `M${d.source.x},${d.source.y}V${(d.source.y + d.target.y) / 2}H${
          d.target.x
        }V${d.target.y}`
    );

  // nodes
  const node = g
    .selectAll("g.node")
    .data(root.descendants())
    .join("g")
    .attr("class", "node")
    .attr("transform", (d) => `translate(${d.x},${d.y})`);

  node
    .append("circle")
    .attr("r", nodeRadius)
    .attr("fill", "#fff")
    .attr("stroke", "#333");

  node
    .append("text")
    .attr("dy", 1)
    .attr("font-size", 3)
    .attr("text-anchor", "middle")
    .text((d) => d.data.name);
  return svg;
}37:[{"id":"d01f33e522e8e15412b1c2e319f8dfad08f38a2dff95dd1d3a03e3a2017bc79e2bcccb1e31a0f91c5ff46dbd7c7ac16dbda292ab6e7d2a8473efca9bfcc7a37c","href":"https://static.observableusercontent.com/files/d01f33e522e8e15412b1c2e319f8dfad08f38a2dff95dd1d3a03e3a2017bc79e2bcccb1e31a0f91c5ff46dbd7c7ac16dbda292ab6e7d2a8473efca9bfcc7a37c","name":"image@1.png","type":"image/png","lastModified":1755027031602.098,"size":57588}]
30:{"id":"50e53b27a28761aa","slug":"gepa","title":"Trying out \"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\"","thumbnail":"c5c1de2847c0598bf9461d86eebd1e6dd1fd6b98c5b5a9b379fca2fa999dea7a","access_level":"public","create_time":"$D2025-08-01T08:41:36.234Z","update_time":"$D2025-11-09T16:19:27.414Z","view_time":null,"first_public_time":"$D2025-08-12T19:45:55.094Z","has_unpublished_edits":null,"fork":false,"stars":4,"starred":false,"owner":{"type":"team","id":"7db5ed2b0697d645","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":"starter_2024"},"version":784,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null}],"members":[{"type":"individual","id":"5215f6ec4a999d40","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":null,"role":"editor"},{"type":"team","id":"7db5ed2b0697d645","login":"tomlarkworthy","name":"Tom Larkworthy","avatar_url":"https://avatars-next.observableusercontent.com/47327a8bc1966f2186dcb3ebf4b7ee6e4e7ab9a5c2a07405aff57200ea778f71","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":5,"collection":{"id":"ef392f92725a92d7","slug":"ai","title":"AI","owner":{"id":"7db5ed2b0697d645","login":"tomlarkworthy","name":"Tom Larkworthy"}},"type":"notebook","body":{"titling":"auto","title":"Trying out \"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\"","cells":[{"id":0,"value":"# Trying out \"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\"\n## https://arxiv.org/abs/2507.19457 25 Jul 2025\n_Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, Omar Khattab_ -- UC Berkeley, Stanford University, BespokeLabs.ai, Databricks, MIT","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":554,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":643,"value":"\n### Seed prompt (human made)\n```\nRespond in Observable Javascript (Notebook 1.0) inside an\nXML code tag to solve the question.\nfor example\n\n\n\n\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":636,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":663,"value":"## Tasks\n\nThe tasks used to evaluate are found in [@tomlarkworthy/robocoop-eval](https://observablehq.com/@tomlarkworthy/robocoop-eval). They are 8 very simple Observable coding challenges","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":652,"value":"## Noted on implementation\n\nNo cross-over was used.\n\nWe enabled websearch when reflecting on a new prompt (code [here](https://observablehq.com/@tomlarkworthy/gepa#reflectFn)). This was important because the LLM was able to source actual documentation on the Observable programming model when trying to fix misunderstandings.\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":690,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":755,"value":"---\n\nFor the rest of the notebook to run you need to enter a \"OPENAI_API_KEY\" in the Robocoop 2.0 modal, but you can take the algorithm and rewire it to something else if you don't want to run *this* particular experiment.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":762,"value":"viewof run_example = Inputs.toggle({\n  label: \"run GEPA\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":493,"value":"renderHierarchy(svg``, heirachy)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":327,"value":"### Math","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":120,"value":"function dominates(a, b) {\n  const betterInAny = a.scores.some((v, i) => v > b.scores[i]);\n  const notWorse = a.scores.every((v, i) => v >= b.scores[i]);\n  return betterInAny && notWorse;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":118,"value":"function paretoFront(pop) {\n  return pop.filter((c) => !pop.some((o) => o !== c && dominates(o, c)));\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":309,"value":"## GEPA","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":100,"value":"class GEPA {\n  constructor({\n    initialPrompts,\n    evalFn,\n    reflectFn,\n    populationSize = 12,\n    children = 2,\n    generations = 8\n  }) {\n    this.evalFn = evalFn;\n    this.reflectFn = reflectFn;\n    this.populationSize = populationSize;\n    this.children = children;\n    this.generations = generations;\n    this.population = initialPrompts.map((p) => ({\n      parent: null,\n      prompt: p,\n      scores: null,\n      trace: \"\"\n    }));\n  }\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":303,"value":"async function initialize(gepa) {\n  // evaluate seed prompts\n  await Promise.all(\n    gepa.population.map(async (c) => {\n      const r = await gepa.evalFn(c.prompt);\n      c.scores = r.scores;\n      c.trace = r.trace;\n    })\n  );\n  return gepa;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":301,"value":"async function step(gepa) {\n  const frontier = paretoFront(gepa.population);\n\n  const tasks = frontier.flatMap((parent) =>\n    Array.from({ length: gepa.children }, async () => {\n      try {\n        const childPrompt = await gepa.reflectFn(parent.prompt, parent.trace);\n        const r = await gepa.evalFn(childPrompt);\n        return {\n          parent,\n          prompt: childPrompt,\n          scores: r.scores,\n          trace: r.trace\n        };\n      } catch (e) {\n        console.error(e);\n        return null;\n      }\n    })\n  );\n\n  const offspring = (await Promise.all(tasks)).filter(Boolean);\n\n  gepa.population = paretoFront([...frontier, ...offspring]);\n  if (gepa.population.length > gepa.populationSize)\n    gepa.population.length = gepa.populationSize;\n\n  return gepa;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":139,"value":"### LLM integration (callFn)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":109,"value":"import { responses } from \"@tomlarkworthy/openai-responses-api\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":123,"value":"async function callFn(prompt, task) {\n  const output = await runPrompt(task, {\n    system_prompt: prompt\n  });\n  output.trace = `USER: ${task.query}\\nLLM: ${JSON.stringify(\n    output.response\n  )}\\nASSISTANT: ${JSON.stringify(output.cells, null, 2)}`;\n  return output;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":187,"value":"#### test callFn","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":127,"value":"test_callFn = {\n  const output = await callFn(system_prompt, assign_literal);\n  expect(output.cells.length).toBeGreaterThan(0);\n  expect(output.trace.length).toBeGreaterThan(0);\n  return {\n    seed_prompt,\n    ...output\n  };\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":255,"value":"test_callFn.trace","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":144,"value":"### Scoring Tasks (scoreFn, evalFn)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":151,"value":"scoreFn = async (output, task) => {\n  const experiment = await runScore({\n    response: output,\n    caseTask: task\n  });\n\n  const overall =\n    (experiment.score.responded ? 1 : 0) +\n    (experiment.score.compiles ? 1 : 0) +\n    (experiment.score.applies ? 1 : 0) +\n    (experiment.score.correct ? 1 : 0) +\n    (experiment.score.errors.length == 0 ? 1 : 0);\n  return {\n    score: overall / 5.0,\n    trace: JSON.stringify(experiment.score, null, 2)\n  };\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"test_scoreFn = {\n  const output = test_callFn;\n  const scoreResponse = await scoreFn(output, assign_literal);\n\n  return scoreResponse.trace;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":111,"value":"/**\n * buildEvalFn – create an evalFn from a task list.\n * tasks: [{ name, input, ... }]\n * callFn(prompt, task)  -> output\n * scoreFn(output, task) -> number | number[]\n * aggregator(scores[])  -> number[] (default: single objective mean)\n */\nfunction buildEvalFn({ tasks, callFn, scoreFn, aggregator = (s) => s }) {\n  return async function evalFn(prompt) {\n    const taskScores = [];\n    const traceParts = [];\n    const outputs = [];\n    await Promise.all(\n      tasks.map(async (task) => {\n        const output = await callFn(prompt, task);\n        const score = await scoreFn(output, task);\n        outputs.push(output);\n        taskScores.push(score.score);\n        traceParts.push(\n          `TASK: ${task.name}\\nSCORE: ${score.trace}\\nOVERALL: ${score.score}\\nTRACE: ${output.trace}\\n`\n        );\n      })\n    );\n    const scores = aggregator(taskScores);\n    return { scores, outputs, trace: traceParts.join(\"\\n---\\n\") };\n  };\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":149,"value":"evalFn = buildEvalFn({\n  tasks: [...cases.values()],\n  callFn,\n  scoreFn\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":153,"value":"test_evalFn = {\n  const result = await evalFn(seed_prompt);\n  expect(result.scores.length).toEqual([...cases.values()].length);\n  expect(result.trace).toBeDefined();\n  return result;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":245,"value":"test_evalFn.trace","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":262,"value":"## Reflection\n\nLearning how to answer the prompt requires research, so the reflect function first uses a web tool and then answers with an improved prompt.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":264,"value":"$35","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":266,"value":"test_reflectFunction = {\n  const reflection = reflectFn(system_prompt, test_evalFn.trace);\n\n  return reflection;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":286,"value":"## Run GEPA","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":563,"value":"seed_prompt = `Respond in Observable Javascript (Notebook 1.0) inside an XML code tag to solve the question. for example\n\n\n\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":288,"value":"test_gepa_initialization = {\n  const state = new GEPA({\n    initialPrompts: [seed_prompt],\n    evalFn,\n    reflectFn,\n    populationSize: undefined,\n    children: undefined,\n    generations: undefined\n  });\n  return state;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":295,"value":"test_gepa_initialize = run_example && initialize(test_gepa_initialization)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":320,"value":"pInit = recordFront(paretoFront(test_gepa_initialize.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":584,"value":"printFront(pInit)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":316,"value":"test_gepa_step_0 = step(test_gepa_initialization)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":322,"value":"p0 = recordFront(paretoFront(test_gepa_step_0.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":587,"value":"printFront(p0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":318,"value":"test_gepa_step_1 = step(test_gepa_step_0)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":330,"value":"p1 = recordFront(paretoFront(test_gepa_step_1.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":589,"value":"printFront(p1)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":332,"value":"test_gepa_step_2 = step(test_gepa_step_1)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":334,"value":"p2 = recordFront(paretoFront(test_gepa_step_2.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":615,"value":"printFront(p2)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":367,"value":"test_gepa_step_3 = step(test_gepa_step_2)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":501,"value":"p3 = recordFront(paretoFront(test_gepa_step_3.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":617,"value":"printFront(p3)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":499,"value":"test_gepa_step_4 = step(test_gepa_step_3)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":372,"value":"p4 = recordFront(paretoFront(test_gepa_step_4.population))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":504,"value":"printFront(p4)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":521,"value":"printFront = (front) =>\n  Inputs.textarea({\n    rows: 100,\n    value: front\n      .map(\n        (s) =>\n          `SCORE: ${s.scores}\\nPROMPT: \\n${s.prompt}\\nTRACE:\\n\\n${s.trace}\\n`\n      )\n      .join(\"\")\n  })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":382,"value":"function recordFront(front) {\n  viewof generations.value.push(front);\n  viewof generations.dispatchEvent(new Event(\"input\"));\n  return front;\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":376,"value":"viewof generations = Inputs.input([])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":392,"value":"generations","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":405,"value":"function toHierarchy(generations) {\n  // Map candidates to node shells\n  const nodeMap = new Map();\n  generations.flat().forEach((c) => {\n    nodeMap.set(c, { name: c.scores.join(\",\"), data: c, children: [] });\n  });\n  // Wire up children\n  generations.flat().forEach((c) => {\n    if (c.parent && nodeMap.has(c.parent)) {\n      nodeMap.get(c.parent).children.push(nodeMap.get(c));\n    }\n  });\n  // Roots = generation 0\n  const roots = generations[0].map((c) => nodeMap.get(c));\n  return roots.length === 1 ? roots[0] : { name: \"root\", children: roots };\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":407,"value":"heirachy = toHierarchy([generations.at(-1)])","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":415,"value":"renderHierarchy(svg``, heirachy)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":396,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":350,"value":"## Evals","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":134,"value":"import {\n  cases,\n  runCase,\n  assign_literal,\n  load_all,\n  system_prompt,\n  runScore,\n  runPrompt\n} from \"@tomlarkworthy/robocoop-eval\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":363,"value":"data = [...cases.values()]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":775,"value":"Inputs.table(data, {\n  layout: \"auto\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":346,"value":"# Dev tools","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":55,"value":"\n  ${ui}\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":60,"value":"sayHello = {\n  return \"hello\";\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"test_hello = expect(sayHello).toEqual(\"hello\")","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"import { ui, failing_tests, expect } from \"@tomlarkworthy/agentic-planner\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":34,"value":"import { exporter } from \"@tomlarkworthy/exporter-2\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"exporter()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"import { context_menu } from \"@tomlarkworthy/editor-4\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":41,"value":"//context_menu","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"import {} from \"@tomlarkworthy/gepa\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":168,"value":"load_all","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W37","stdlib":"1"}}
31:"$30:body"