2b:null 2c:["$","title",null,{"children":"ChatGPT Q&A Generator | Tanner Hobson | Observable"}] 31:T9dc,htl.html`

ChatGPT Q&A Generator

The GPT-4 Model Card was recently published. I found the sections in Appendix A, B, and C particularly interesting (see Reference section below). These sections describe a question-and-answer (Q&A) form that is used to classify the AI's own responses into several categories. They used it for training primarily, but I'm interested in the use of Q&A for classifying things on its own (rather than as input to the training algorithm).

The primary goals of this notebook are to explore this concept a bit more and to generalize the concept of a Q&A form for general use, i.e. as a function or a library that can be used in the future.

I've envision a couple of specific use cases that I would like to explore:

When summarizing long documents, I would like to know if each additional "page" of original text is important or if its content can be ignored. This would be helpful for limiting the number of requests needed to summarize or answer questions from long documents.
When evaluating my own GPT fine-tunings, it would be helpful to have some external quality metric. This is similar to the use cases from the GPT-4 model card, but I would not be incorporating the results into training directly; it would just be a metric for me to look at externally.
When interpreting user commands, it would be helpful to have a way to classify types of commands that could be interpreted using more specialized instructions. This would be like this blog post (that is currently missing?) which aims to use ChatGPT as an interpreter for a natural language interface to a reminder application.

Existing Q&A Forms

The GPT-4 System Card appendices are included below.

GPT-4 System Card - Appendix A

${viewof GPT_4_System_Card_Appendix_A}

GPT-4 System Card - Appendix B

${viewof GPT_4_System_Card_Appendix_B}

GPT-4 System Card - Appendix C

${viewof GPT_4_System_Card_Appendix_C}

From these, we can see that there are two different kinds of Q&A form. The first one, from Appendix A, is for handling a cross-product of multiple categories. The second one, from Apendix B and C, is for handling a single set of categories. `/* htl.html */32:T4a3,viewof ROUND_1_SUMMARIZATIONS = { await (yield button('Execute')).promise; const fragments = SPLIT(TRANSCRIPT, { maxTokens: 512, tokenOverlap: 64 }); const summaries = []; for (const fragment of fragments) { const request = new Request('https://api.openai.com/v1/chat/completions', { method: 'POST', mode: 'cors', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify({ model: 'gpt-3.5-turbo', messages: [ { role: 'system', content: SUMMARIZATION_CONFIG.INTRODUCTION }, { role: 'user', content: `... ${fragment} ...` }, ], }), }); const response = await FETCH(request); const summary = response.choices[0].message.content; summaries.push(summary); } return yield Inputs.form(fragments.map((fragment, i) => { const summary = summaries[i]; return Inputs.form({ fragment: Inputs.textarea({ rows: 12, cols: 40, value: fragment, }), summary: Inputs.textarea({ rows: 12, cols: 40, value: summary, }), }); })); }33:T69d,viewof PROMPT = { const value = VALUE({ informations: INFORMATIONS, choices: CHOICES, }); return Inputs.textarea({ rows: 24, cols: 80, value, }); function VALUE({ choices, informations=null }={}) { const prompt = []; prompt.push(String.raw`Read the entire message above and answer the multiple-choice question about it.`); if (Array.isArray(informations) && informations.length) { prompt.push(String.raw`Here is some additional information that may help for classifying the message.`); for (const information of informations) { prompt.push(information); } } prompt.push(String.raw`Which of the following choices best describes the Assistant response? Your choice should not be affected by the content of the User prompt. Answer the question by selecting one of the following options:`); if (Array.isArray(choices)) { const keys = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; choices = Object.fromEntries(choices.map((choice, i) => ([keys.substring(i, i+1), choice]))); } const keys = Array.from(Object.keys(choices)); keys.sort((a, b) => a.localeCompare(b)); for (const key of keys) { prompt.push(String.raw`- (${key}) ${choices[key]}`); } prompt.push(String.raw`Your response should start with only the single character "${keys.join('" or "')}" (without quotes or punctuation) on its own line followed by an explanation of your answer on the next line. Your explanation should take the reader through your reasoning step-by-step, culminating in the correct answer. Avoid simply stating the correct answer at the outset of your explanation.`); return prompt.join('\n'); } }34:[{"id":"fb12f63a4717fd2bb125bb58e9a25e449ddc8e384e37ca95c74f09e7df1dc2ae761ba29445a369415fad666ff8b541779bba396b72f2d76107f28eea3ecf6126","href":"https://static.observableusercontent.com/files/fb12f63a4717fd2bb125bb58e9a25e449ddc8e384e37ca95c74f09e7df1dc2ae761ba29445a369415fad666ff8b541779bba396b72f2d76107f28eea3ecf6126","name":"GPT-4 System Card - Appendix C.txt","type":"text/plain","lastModified":1679333530765.589,"size":3298},{"id":"b454ab69b14f34b35f86db948b48f5d574a5059fcb94158316c3d1e32dbf6a745c7d506f09de8d13a8057c753d9705f68b84578c6fbfe1ba2438914f7399081d","href":"https://static.observableusercontent.com/files/b454ab69b14f34b35f86db948b48f5d574a5059fcb94158316c3d1e32dbf6a745c7d506f09de8d13a8057c753d9705f68b84578c6fbfe1ba2438914f7399081d","name":"GPT-4 System Card - Appendix A@1.txt","type":"text/plain","lastModified":1679337892392.329,"size":5925},{"id":"4f8924fa33687ea895a067cb909f419ec4b30e802a383ebf3487a22e2e4ff1a1708d2a868187d086aef2ce671c02b6ae5ab4d121f359f2763d6fbb20a27298f2","href":"https://static.observableusercontent.com/files/4f8924fa33687ea895a067cb909f419ec4b30e802a383ebf3487a22e2e4ff1a1708d2a868187d086aef2ce671c02b6ae5ab4d121f359f2763d6fbb20a27298f2","name":"GPT-4 System Card - Appendix B@1.txt","type":"text/plain","lastModified":1679338078046.878,"size":3748}] 2f:{"id":"345f198b0372e678","slug":null,"title":"ChatGPT Q&A Generator","thumbnail":"17649a8d6fd2640321a92bcbbc5fec8d0b1c36f5235e2b716602c831fe7591fe","access_level":"public","create_time":"$D2023-03-20T16:36:21.814Z","update_time":"$D2023-03-22T17:48:32.617Z","view_time":null,"first_public_time":"$D2023-03-20T16:36:21.814Z","fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"7343f790779e6735","login":"player1537","name":"Tanner Hobson","avatar_url":"https://avatars-next.observableusercontent.com/16170a1a32e364e1ff6054c6045de6814a032d01331aba01288f82afceeaaec2","tier":"starter_2024"},"version":156,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"e0a627f6b6f3beb0","login":"player1537","name":"Tanner Hobson","avatar_url":"https://avatars-next.observableusercontent.com/16170a1a32e364e1ff6054c6045de6814a032d01331aba01288f82afceeaaec2","tier":null}],"members":[{"type":"individual","id":"e0a627f6b6f3beb0","login":"player1537","name":"Tanner Hobson","avatar_url":"https://avatars-next.observableusercontent.com/16170a1a32e364e1ff6054c6045de6814a032d01331aba01288f82afceeaaec2","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":{"id":"d21bf020f9cae8af","slug":"work-in-progress-promising","title":"Work-in-Progress (promising)","owner":{"id":"7343f790779e6735","login":"player1537","name":"Tanner Hobson"}},"type":"notebook","body":{"title":"ChatGPT Q&A Generator","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":123,"value":"viewof OPENAI_API_KEY = Inputs.text({\n label: 'OpenAI API Key',\n});","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":71,"value":"viewof YOUTUBE_CONFIG = Inputs.form({\n BASE_URL: Inputs.text({\n label: 'MAB YouTube API Base URL',\n value: 'https://youtube.is.mediocreatbest.xyz',\n }),\n AUTH: Inputs.text({\n label: 'youtube.m-at.best auth (user:pass)',\n }),\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":80,"value":"viewof TRANSCRIPT = {\n if (!YOUTUBE_CONFIG.AUTH) throw 'need youtube auth';\n let transcript = await YOUTUBE('AvjlbxS71XY', 'text/plain');\n transcript = transcript.replaceAll(/\\s+/g, ' ');\n return Inputs.textarea({\n rows: 24,\n cols: 80,\n value: transcript,\n });\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":117,"value":"viewof SUMMARIZATION_CONFIG = Inputs.form({\n INTRODUCTION: Inputs.textarea({\n label: 'INTRODUCTION',\n rows: 24,\n cols: 80,\n value: `You are a YouTube video transcript summarizer. Your job is to summarize the user's transcript into a series of easily understandable bullet points. You take special care to keep your summarizations terse and focus more on the breadth of the summary than on its depth.`,\n }),\n});","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":104,"value":"$32","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":60,"value":"viewof INFORMATIONS = listOfItems([\n String.raw`A good summarization includes a breadth of information that covers all topics within the original source text.`,\n]);","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":51,"value":"viewof CHOICES = listOfItems([\n String.raw`The message is a good summary of the original text.`,\n String.raw`The message has too much repeated information.`,\n]);","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":5,"value":"$33","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":27,"value":"viewof GPT_4_System_Card_Appendix_A = Inputs.textarea({\n rows: 24,\n cols: 80,\n value: (await FileAttachment(\"GPT-4 System Card - Appendix A@1.txt\").text()).trim(),\n});","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":30,"value":"viewof GPT_4_System_Card_Appendix_B = Inputs.textarea({\n rows: 24,\n cols: 80,\n value: (await FileAttachment(\"GPT-4 System Card - Appendix B@1.txt\").text()).trim(),\n});","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":32,"value":"viewof GPT_4_System_Card_Appendix_C = Inputs.textarea({\n rows: 24,\n cols: 80,\n value: (await FileAttachment(\"GPT-4 System Card - Appendix C.txt\").text()).trim(),\n});","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":53,"value":"import { listOfItems } from \"@eeeps/variable-length-sortable-list-input\";","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":73,"value":"async function YOUTUBE(videoId, mimeType) {\n const response = await fetch(`${YOUTUBE_CONFIG.BASE_URL}/transcript/${videoId}`, {\n headers: {\n 'Accept': `${mimeType}`,\n 'Authorization': `Basic ${btoa(YOUTUBE_CONFIG.AUTH)}`,\n },\n });\n\n if (mimeType === 'text/plain') {\n return await response.text();\n } else if (mimeType === 'application/json') {\n return await response.json();\n }\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":95,"value":"SPLIT(TRANSCRIPT, { maxTokens: 512, overlapTokens: 64 });","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":90,"value":"function SPLIT(document, {\n maxTokens=null,\n nonOverlapTokens=null,\n overlapTokens=0,\n tokenizer=new GPT3Tokenizer({ type: 'gpt3' }),\n}={}) {\n if (maxTokens === null && nonOverlapTokens === null) {\n throw 'one of [maxTokens | nonOverlapTokens] is required';\n } else if (maxTokens === null && nonOverlapTokens !== null) {\n maxTokens = nonOverlapTokens + overlapTokens;\n } else if (maxTokens !== null && nonOverlapTokens !== null) {\n if (maxTokens !== nonOverlapTokens + overlapTokens) {\n throw 'if both maxTokens and nonOverlapTokens are passed, then they need to be consistent';\n }\n }\n \n const { text } = tokenizer.encode(document);\n\n const fragments = [];\n for (let i=0, n=text.length; i