2c:null 2d:["$","title",null,{"children":"DIVERSE-CDT : Replication Study 03/04The Stimuli / Design, The Instrument / Approach, The Panel, The Pilot | cagatayturkay | Observable"}] 32:T658,### Some measurement categories **Performance measures — what users can do** - Task completion rate, time-on-task, error rate, number of clicks/steps, etc... - Objective and easy to compare across conditions, but don't tell you why something happened **Physiological/behavioral measures — what users involuntarily do** - Eye-tracking (fixations, saccades, scanpaths), EEG, skin conductance, heart rate variability... - High(er) ecological validity for attention and cognitive load, but expensive and often require lab settings **Subjective measures — what users think and feel** - Established survey _instruments_: NASA-TLX (cognitive load), SUS (usability), AttrakDiff (UX), UMUX, VLAT (vis literacy), (remember -- PREVis: Perceived Readability Evaluation for Visualizations?) **Behavioral/log data — what users actually did** - Interaction logs, mouse/eye-tracking heatmaps, - Unobtrusive and scalable, but context-poor

Q: What is the effect you will measure? What is a reasonable effect size to expect?

Q: What makes a **good** measurement & instrument pair? Q: Can you triangulate with multiple (types of) measures?

Q: are you able to measure the full/complete range? **Ceiling effect — **participants score at or near the maximum possible value, so the measure can't distinguish between high performers or detect improvements in already-easy conditions. **Floor effect — **participants score at or near the minimum, so the measure can't distinguish between low performers or detect further degradation in already-hard conditions.33:[{"id":"a6f7dad8650a8241f5354c46eee5c459bfd0264761c5d4118fa2fa30012c5f825c0abe040ea0114f562c187450df5581545d579b1979a91dd834f498844e3dc9","href":"https://static.observableusercontent.com/files/a6f7dad8650a8241f5354c46eee5c459bfd0264761c5d4118fa2fa30012c5f825c0abe040ea0114f562c187450df5581545d579b1979a91dd834f498844e3dc9","name":"Screenshot 2026-05-21 at 08.08.38.png","type":"image/png","lastModified":1779347337451.894,"size":255463}] 30:{"id":"1f51b612fad03b7b","slug":"diverse-cdt-replication-study-03-04-methods","title":"DIVERSE-CDT : Replication Study 03/04The Stimuli / Design, The Instrument / Approach, The Panel, The Pilot","thumbnail":"f61c195b613d571d126fc0ea1592fc25e93978c86d8c28c5eacefd6e85f1951c","access_level":"public","create_time":"$D2026-05-20T15:59:06.614Z","update_time":"$D2026-05-21T11:35:13.615Z","view_time":null,"first_public_time":"$D2026-05-21T11:36:55.166Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"b80915b6d16242f5","login":"cagatayturkay","name":"","avatar_url":"https://avatars-next.observableusercontent.com/a40227523694e1e50f6ea8dca5c2774a65e88c0442050bbc77c84dcd8144d038","tier":"starter_2024","bio":""},"version":177,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"6b1f9966e7dea692","login":"cagatayturkay","name":"","avatar_url":"https://avatars-next.observableusercontent.com/a40227523694e1e50f6ea8dca5c2774a65e88c0442050bbc77c84dcd8144d038","tier":null,"bio":""}],"members":[{"type":"individual","id":"6b1f9966e7dea692","login":"cagatayturkay","name":"","avatar_url":"https://avatars-next.observableusercontent.com/a40227523694e1e50f6ea8dca5c2774a65e88c0442050bbc77c84dcd8144d038","tier":null,"role":"editor"},{"type":"team","id":"b80915b6d16242f5","login":"cagatayturkay","name":"","avatar_url":"https://avatars-next.observableusercontent.com/a40227523694e1e50f6ea8dca5c2774a65e88c0442050bbc77c84dcd8144d038","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":null,"type":"notebook","body":{"titling":"auto","title":"DIVERSE-CDT : Replication Study 03/04The Stimuli / Design, The Instrument / Approach, The Panel, The Pilot","cells":[{"id":0,"value":"# DIVERSE-CDT : [Replication Study](https://observablehq.com/@jsndyks/diverse-cdt-replication-study-homepage) 03/04
The Stimuli / Design, The Instrument / Approach, The Panel, The Pilot\n---\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":46,"value":"## 1. Plan for today\n\n The plan at the end of the day (or EOW if EoD not possible):\n\n** Have study ready to run for a first small pilot study**\n\n... and we will\n\n- know about the different considerations in recruiting study participants\n- develop a plan for participant recruitment\n- know the process for setting up a study panel on a provider platform\n- **prepare for a pilot study by creating stimuli, tasks, questions, measurement instruments**\n- **refine study protocol in-line with stimuli and measurements**\n- know how to deploy the pilot study on a suitable platform and prepare to run a pilot with local sample\n- **develop a plan/strategy on what to test in a pilot study**\n- run pilot study for data collection and analysis\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":56,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":44,"value":"## 2. Group Checkins!\n\n - **progress** - what has happened? How are you doing against the workplan?\n - **problems** - is anything stopping progress?\n - **priorities** - what is going to happen next?\n - **plan** - any changes to the plan required given the above?\n\n\n\n**Q:Are you comfortable with your research questions, variables of interest, factors you are studying?**\n- _have you identified your independent variables (what you manipulate) and dependent variables (what you measure)_\n- _(in your follow-up) have you reviewed prior literature to anchor your hypotheses?_\n \n\n![image](${await FileAttachment(\"Screenshot 2026-05-21 at 08.08.38.png\").url()})\n_From: Hornbæk, Kasper, Per Ola Kristensson, and Antti Oulasvirta, 'Experiments', Introduction to Human-Computer Interaction (Oxford, 2025; online edn, Oxford Academic, 21 Aug. 2025),_\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":54,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":130,"value":"**Q: Do you have relevant/suitable tasks?**\n\nOne decision concerns the activities that participants will engage in, which are often prescribed as tasks. One may select tasks in many ways. One approach is to select tasks that are representative of what users would do outside the experiment. Munzner [*] discussed task selection in information visualization and wrote\n\n>A study is not very interesting if it shows a nice result for a task that nobody will ever actually do, or a task much less common or important than some other task. You need to convince the reader that your tasks are a reasonable abstraction of the real-world tasks done by your target users.\n\nFrom: _Hornbæk, K., Kristensson, P.O. and Oulasvirta, A., 2025. Introduction to Human-Computer Interaction. Oxford University Press._,\n\n[*] T. Munzner. Process and pitfalls in writing information visualization research papers. In Information visualization, pages 134–153. Springer, 2008.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":51,"value":"## 3. On Measurements, Measuring & Effects (sizes)\n\n- What will you measure in your (follow-up) studies?\n- What **can** you measure in your (follow-up) studies?\n\n \n\n\n \n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":62,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":109,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":118,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":64,"value":"## 4. Study Protocol (flow) & Stimuli Generation and Distributions\n\n### Study Protocol (flow) \n\nDifferent design choices — in-between, within group\n\n- **Between-subjects (or between-groups) study design:** different people test each condition, so that each person is only exposed to a single type of stimulus.\n- **Within-subjects (or repeated-measures) study design:** the same person tests all the conditions (i.e., all the combinations).\n\nYou will also need to ... \n- decide how many repetitions each participant does\n- which stimuli each participant will look at (e.g., if comparing pairs, think about the *true effect*)\n- have you considered **ordering effects**? \n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":100,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":139,"value":"\nWhen every participant tests multiple interface versions, the order they experience them can skew the results (e.g., learning effects or fatigue). A **counter-balancing strategy** will enable you to distribute stimuli across participants in a rigorous way that minimizes the potential impact of ordering. \n\n- Randomisation is one approach\n- A more structured approach is **Latin Square** -- evenly distributes these biases across all conditions so they cancel each other out\n\n\n

\"Reading — [https://web.mit.edu/6.813/www/sp16/classes/11-experiment-design/](https://web.mit.edu/6.813/www/sp16/classes/11-experiment-design/)

\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":66,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":159,"value":"### Stimuli Generation\n\n- do you have a robust/systematic mechanism to generate stimuli \n- what is the distribution of the true effects in your stimuli pool\n- ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":165,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":68,"value":"## 5. Participants, Recruitment, Platforms\n\n- specifying a panel & considerations on recruitment -- a panel provider -- Prolific\n- are there special considerations in recruiting your participants? How to ensure diversity and diversity in what?\n- do you need to screen for relevant characteristics: prior experience, age, accessibility needs, visual literacy, etc.?\n\n\n- Protocols for poor performance and response quality - how to identify poor performance, non-engagement -- you need to outline them clearly beforehand","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":73,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":71,"value":"# Part-2: At your own pace as groups\n\nPrepare for a first pilot (for C&M group -- to be run within the cohort (we can't run with externals yet, no ethics approval))\n- stimuli\n- platforms / technology\n- study logic/flow\n- what do you expect to achieve in the pilot, what to watch out for?\n\n**C&M groups:**\n- Generate stimuli -- think about naming conventions, think about matching with _true effects_\n- Start a study on Qualtrics\n- Implement the study protocol/logic\n- Get your stimuli into Qualtrics\n- Think about the data/measurement\n\n**By EoW be ready to run a pilot study: **\nA pilot run (even with 2–3 people) can reveal:\n\n- Ambiguous task instructions\n- Technical failures in your apparatus\n- Ceiling or floor effects in your measures\n- Session length issues\n\n\n**BallotMap group**\n- Generate stimuli \n- Think about measurements, effects you are computing -- are you prepared for the analysis \n- \n ","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":173,"value":"

\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":28,"value":"---\n\n## Appendices","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"import { Refs } from \"@gicentre/references-class\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":11,"value":"bib = Refs.create(\n await d3.text(\n \"https://jsndyks.github.io/diverse-cdt/bib/zoteroDIVERSE-CDT.bib\"\n )\n)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"import { readData, table, T } from \"@jwolondon/cdt-utilities\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":20,"value":"import {\n aims,\n info,\n annotation,\n task,\n learning\n} from \"@jwolondon/schema-teaching2025\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":18,"value":"import { setFonts, setPageStyle, spacer } from \"@jwolondon/cdt-utilities\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":36,"value":"setFonts()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"setPageStyle()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"import { icons } from \"6ccb7a5a129abf60\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W33","stdlib":"1"}} 31:"$30:body"