2f:T8da,# key typing dataset from: https://userinterfaces.aalto.fi/136Mkeystrokes/ The keystrokes file is 1.46GB, there are 168594 PARTICIPANTS with each typing 15 sentences. Resulting in a test file between 51000 and 540000 bytes making an uncompressed file size of 17.5GB Some used like 180 key presses to type 37 characters, so there are some really bad error rates and I should filter the **metadata_participants.txt** file to remove the noisy data. The **metadata_participants.txt** needs to be filtered to reduce the bad data. - **ERROR_RATE** Uncorrected error rate - **ECPC** Error Corrections per Character - **AVG_WPM_15** Words per minute averaged over the 15 typed sentences ### Filter Values >ERROR_RATE < 0.5 ECPC <= 0.05 AVG_WPM_15 > 10 That reduces the number of files to 32059. The **(_participant number_)_keystrokes.txt** is the keystroke-by-keystroke log.

participant number_keystrokes.txt is a heavy structure as each keystroke contains the complete sentence to be typed as well as the complete finished sentence.

Example of data:

  {
  PARTICIPANT_ID: 32
  TEST_SECTION_ID: 173
  SENTENCE: "The Senate defied President Bill Clinton on Thursday."
  USER_INPUT: "The senate defied president Bill Clinton On Thursday"
  KEYSTROKE_ID: 7679
  PRESS_TIME: 1471934742126
  RELEASE_TIME: 1471934742491
  LETTER: "SHIFT"
  KEYCODE: 16
},
{
  PARTICIPANT_ID: 32
  TEST_SECTION_ID: 173
  SENTENCE: "The Senate defied President Bill Clinton on Thursday."
  USER_INPUT: "The senate defied president Bill Clinton On Thursday"
  KEYSTROKE_ID: 7682
  PRESS_TIME: 1471934742393
  RELEASE_TIME: 1471934742493
  LETTER: "T"
  KEYCODE: 84
}...

1. I take each participants test and extract the unique sentences they took. 2. I then build a json object with the participants ID and there test and the keys they pressed. 3. JSON key codes inflate the file too, so I have renamed them to smaller ones, abbreviations will be in a legend. 4. Times are in MS since 1970 and could be shorter. I subtract the initial keypress start time all the other keypress times. These changes result in the JSON file being upto 50% smaller than the TSV 30:T844,--- ## original files **metadata_participants.txt**: demographic data of users and aggregate statistics such as speed and errors

PARTICIPANT_ID          Unique ID of participant  
AGE                     0 - 120
GENDER                  female, male, none
HAS_TAKEN_TYPING_COURSE Whether the participant has taken a typing course (1) or not (0)  
COUNTRY                 ISO Alpha-2 code  https://www.nationsonline.org/oneworld/country_code_list.htm
KEYBOARD_LAYOUT         QWERTY, AZERTY or QWERTZ layout of keyboard used  
NATIVE_LANGUAGE         ISO language codes  https://www.w3docs.com/learn-html/html-language-codes.html
FINGERS                 Choice between 1-2, 3-4, 5-6, 7-8 and 9-10 fingers used for typing  
TIME_SPENT_TYPING       Number of hours spent typing everyday  
KEYBOARD_TYPE           Full (desktop), laptop, small physical (e.g on phone) or touch keyboard  
ERROR_RATE(%)           Uncorrected error rate  
AVG_WPM_15              Words per minute averaged over the 15 typed sentences  
AVG_IKI                 Average inter-key interval   
ECPC                    Error Corrections per Character  
KSPC                    Keystrokes per Character  
AVG_KEYPRESS            Average Keypress duration  
ROR                     Rollover ratio

**number_keystrokes.txt**: the keystroke-by-keystroke log for all test sentences attempted by the **user ID/PARTICIPANT_ID** = _number_.

PARTICIPANT_ID      Unique ID of participant  
TEST_SECTION_ID     Unique ID of the presented sentence  
SENTENCE            Sentence shown to the user  
USER_INPUT          Sentence typed by the user after pressing Enter or Next button  
KEYSTROKE_ID        Unique ID of the keypress   
PRESS_TIME          Timestamp of the key down event (in ms)   
RELEASE_TIME        Timestamp of the key release event (in ms)  
LETTER              The typed letter  
KEYCODE             The javascript keycode of the pressed key

Note: For some users, Keystrokes are not logged or not displayed correctly. The corresponding **javascript keycode** is used instead: _**KEYCODE**_ 31:[{"id":"b2093ffb967d292f2d05802335545a8b6c960380d98e2deb73893c1ac6fab66bc0274d2fa5de18dcae3f68b6d34793d836940eaf3bc585a2da2e82e20e1c1b51","href":"https://static.observableusercontent.com/files/b2093ffb967d292f2d05802335545a8b6c960380d98e2deb73893c1ac6fab66bc0274d2fa5de18dcae3f68b6d34793d836940eaf3bc585a2da2e82e20e1c1b51","name":"32_keystrokes.txt","type":"text/plain","lastModified":1647750658872.252,"size":110899},{"id":"0d46ba17b1727f093905ba5cd9da8713e0928605c85b8fb404377a6761069f362af4d078532c360cd162911f30bfd5cd128e41d8fd3aafc047cc2ae961f91a61","href":"https://static.observableusercontent.com/files/0d46ba17b1727f093905ba5cd9da8713e0928605c85b8fb404377a6761069f362af4d078532c360cd162911f30bfd5cd128e41d8fd3aafc047cc2ae961f91a61","name":"metadata_participants.txt","type":"text/plain","lastModified":1647750818676.253,"size":21255894},{"id":"3ad93c6410b7ed8c1f0903be9c0eda423729c9394cc619c67f44f7fe2372e7deac01b2d6547152c053a5b1596493bb4a133473c29aed791ad47ee94442eb136f","href":"https://static.observableusercontent.com/files/3ad93c6410b7ed8c1f0903be9c0eda423729c9394cc619c67f44f7fe2372e7deac01b2d6547152c053a5b1596493bb4a133473c29aed791ad47ee94442eb136f","name":"readme.txt","type":"text/plain","lastModified":1647750655584.766,"size":3468},{"id":"49d060296319895fb577602ae54d94540a306bd29d32ef2e051d4ca2f6b924f4c435760702557c8ba557a5c67982618092b58aa6371b0ce85a28e361980e528f","href":"https://static.observableusercontent.com/files/49d060296319895fb577602ae54d94540a306bd29d32ef2e051d4ca2f6b924f4c435760702557c8ba557a5c67982618092b58aa6371b0ce85a28e361980e528f","name":"ddl.txt","type":"text/plain","lastModified":1647912788395.099,"size":80360}] 2d:{"title":"key typing dataset","cells":[{"id":0,"value":"$2f","pinned":false,"hidden":false,"mode":"md"},{"id":276,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":403,"value":"---\n\n# large dataset","pinned":false,"hidden":false,"mode":"md"},{"id":6,"value":"meta_participants = FileAttachment('metadata_participants.txt').tsv({typed: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":662,"value":"## filter the metadata_participants.txt file.","pinned":false,"hidden":false,"mode":"md"},{"id":673,"value":"participants = _.filter(meta_participants, d=> {\n return ((d.ERROR_RATE < 0.5) && (d.ECPC <= 0.05) && (d.AVG_WPM_15 >55) )})","pinned":true,"hidden":false,"mode":"ojs"},{"id":381,"value":"Inputs.table(participants)","pinned":true,"hidden":false,"mode":"ojs"},{"id":483,"value":"Plot.plot({\n color: {\n legend: true,\n },\n y: {\n grid: true\n },\n marks: [\n Plot.rect(participants, Plot.bin({fill:'count'}, {x: 'ERROR_RATE', y: radio, title: (d) =>`${d.GENDER} ${d.PARTICIPANT_ID} \\n ${radio}: ${d[radio]}`})),\n Plot.ruleY([0])\n ], \n width:width,\n height:800\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":618,"value":"viewof radio = Inputs.radio(ALL_KEYS, {label: 'Select one',value:'AVG_WPM_15'})","pinned":false,"hidden":false,"mode":"ojs"},{"id":504,"value":"Plot.plot({\n grid: true,\n facet: {\n data: participants,\n y: 'GENDER'\n },\n marks: [\n Plot.rectY(participants, Plot.binX({y: 'count'}, {x: radio, fill: 'GENDER',\n title: (d) =>\n `${d.GENDER} ${d.PARTICIPANT_ID} \\n ${radio}: ${d[radio]}` \n })),\n Plot.ruleY([0])\n ], \n width:width\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":635,"value":"ALL_KEYS = ['PARTICIPANT_ID','AGE','GENDER','HAS_TAKEN_TYPING_COURSE','COUNTRY','LAYOUT','NATIVE_LANGUAGE','FINGERS','TIME_SPENT_TYPING','KEYBOARD_TYPE','ERROR_RATE','AVG_WPM_15','AVG_IKI','ECPC','KSPC','ROR']","pinned":true,"hidden":false,"mode":"ojs"},{"id":414,"value":"function refact(keystrokes) {\n\n let UniqueSentances = _.uniqBy(keystrokes.map( d=>{\n return {TEST_ID:d.TEST_SECTION_ID, SENTENCE:d.SENTENCE, USER_INPUT:d.USER_INPUT,}\n }),'SENTENCE')\n\n\n let teTests = []\n for (let sent of UniqueSentances) {\n\n let TH = _.filter(keystrokes,{TEST_SECTION_ID:sent.TEST_ID})\n\n let startTime = TH[0].PRESS_TIME\n sent.TYPED = TH.map( d=> {\n return {\n // ksID: d.KEYSTROKE_ID, // don't really need this one\n pTIME: d.PRESS_TIME-startTime,\n rTIME: d.RELEASE_TIME-startTime,\n // CHAR: d.LETTER, // not all files contain the letter but KEYCODE is valid\n kCODE: d.KEYCODE}\n })\n teTests.push(sent)\n }\n\n let participant={}\n participant.PARTICIPANT_ID = keystrokes[0].PARTICIPANT_ID\n \n participant.tests = teTests\n \n return participant\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":247,"value":"---\n\n# I can Read in the 1.4GB keystroke dataset \n\nAfter filtering the keystroke dataset to remove troll data I ended up with 20628 files to process.\nThat still takes a lot of time even with a fast m.2 SSD and I had to add a delay in the process save routine as it would ddos the os and stop after saving 6000 files.\n\nSo, I have converted the data to 20628 JSON files. But it's still too big to upload to observable (116MB compressed 562MB uncompressed).\n\nThis means I will make another notebook to process those files with the goal of looking at how long it takes to transition from one keyboard key to another.\nThat resulting dataset could then be used to simulate typing at different skills of typing.\n","pinned":false,"hidden":false,"mode":"md"},{"id":219,"value":"viewof keyStrokesZIP_file = Inputs.file({label: 'Load in file keystrokes.zip',accept: \".zip\" , required: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":222,"value":"zfile = keyStrokesZIP_file.zip()","pinned":true,"hidden":false,"mode":"ojs"},{"id":706,"value":"zipNames = zfile.filenames","pinned":true,"hidden":false,"mode":"ojs"},{"id":718,"value":"keystrokeFiles = participants.map( d =>{\n return `Keystrokes/files/${d.PARTICIPANT_ID}_keystrokes.txt`\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1270,"value":"I had issues with processing breaking with not all files processed. \nSo I made a file containing all the files that had been processed so I could skip them and only do the unprocessed files.","pinned":false,"hidden":false,"mode":"md"},{"id":1168,"value":"FilesProcessed = {\n let f = await FileAttachment(\"ddl.txt\").text()\n f = _.split(f,\",\")\n return _.sortBy(f.map(d=>+d))\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1152,"value":"The code below only downloaded 6937 files of the 20628 files.\n\n\nI need to make a list of files that have been processed and then only process the files not done.\n\nAdded a sleep to not overload the OS\n\n# TODO\n\nThe meta data needs to also be filtered to remove junk data (ie hours per day spent typing has many that a +18 and +24 hours)\n\nadd a button to enable saving, so it doesn't try and make files on it's own.\n\ncurrently the save is commented out.","pinned":true,"hidden":false,"mode":"md"},{"id":724,"value":"ks = {\n\n for (const file of keystrokeFiles) {\n\n let filename = `${_.trim(file,'Keystrokes/files/txt')}json`\n let fileNumber = _.trim(filename,'_keystrokes.json')\n\n const found = FilesProcessed.find(d => d == fileNumber); // temp test file\n if(found == undefined ) {\n // let blooby = serialize(refact(await zfile.file(file).tsv( {typed: true})))\n // FileSaver.saveAs(blooby, filename); \n await sleep(200) // have a sleep so OS doesnt get over loaded\n }\n }\n return \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1127,"value":"sleep = ms => new Promise(r => setTimeout(r, ms));","pinned":true,"hidden":false,"mode":"ojs"},{"id":1240,"value":"# TODO: more data filtering.\n\nSome of the files contain a NULL character code. It looks like the person entering the data has stopped or there session was cut off.\nI am guessing there is around a 100-200 files of the 20628 that could have this problem.\n","pinned":true,"hidden":false,"mode":"md"},{"id":243,"value":"---\n# Appendix\n\nFileSaver is a useful tool to automatically save data to your local drive. \nSerialize converts a json into a Blob for saving to disk. \nI don't use the ASCII_TABLE but was looking at using it to convert the KeyCodes into ASCII and return words for things like Space, Shift, Caps Lock....","pinned":false,"hidden":false,"mode":"md"},{"id":1133,"value":"FileSaver = require('https://bundle.run/file-saver@2.0.5') // I can auto save files.","pinned":true,"hidden":false,"mode":"ojs"},{"id":376,"value":"import {serialize} from '@palewire/saving-json'","pinned":true,"hidden":false,"mode":"ojs"},{"id":971,"value":"import {ASCII_TABLE} from \"@hellonearthis/ascii-table\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1258,"value":"### Sample raw keystrokes file.","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"keystrokes32 = FileAttachment('32_keystrokes.txt').tsv( {typed: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":236,"value":" ","pinned":true,"hidden":false,"mode":"html"},{"id":232,"value":"# LICENSE AND ATTRIBUTION of Keystroke Dataset\n\nYou are free to use this data for non-commercial use in your own research or projects with attribution to the authors. \n\nPlease cite: \n\nVivek Dhakal, Anna Maria Feit, Per Ola Kristensson, Antti Oulasvirta \nObservations on Typing from 136 Million Keystrokes. \nIn Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM, 2018. \n\n@inproceedings{dhakal2018observations, \nauthor = {Dhakal, Vivek and Feit, Anna and Kristensson, Per Ola and Oulasvirta, Antti}, \nbooktitle = {Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)}, \ntitle = {{Observations on Typing from 136 Million Keystrokes}}, \nyear = {2018} \npublisher = {ACM} \ndoi = {https://doi.org/10.1145/3173574.3174220} \nkeywords = {text entry, modern typing behavior, large-scale study} \n} ","pinned":false,"hidden":false,"mode":"md"},{"id":240,"value":"FileAttachment('readme.txt').text()","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}