25:["$","title",null,{"children":"Tracking WaniKani Progress Per Day | Alexey Komissarouk | Observable"}] 4b:T1745,## Longer Story I've been studying Kanji for about a year now, since [moving to Japan last year](https://www.linkedin.com/posts/alexeymk_after-3-years-at-masterclass-it-is-my-turn-activity-7049383108445425664-DNqj?utm_source=share&utm_medium=member_desktop). Early on, [Mat](https://matschaffer.com/) recommended [WaniKani](https://www.wanikani.com/) as a great way to pick up the ~2k characters needed for most reading. Early on, I was using Duolingo, Anki _and_ Wanikani, all spaced repetition apps, and I have to say - WaniKani is the only one I was excited to stick with. I'm Level 17 now, about a 3rd of the way through their "60 levels" now, meaning the entire process will likely take me a little over 3 years, at about 30 minutes a day. I know about 600 Kanji characters "well enough." ### 100 Reviews per Day? One streak I have been trying to hold myself accountable to is reviewing at least 100 items per day. This in term made me want to see a chart of "items reviewed per day" to stay motivated. WaniKani has some basic stats itself and seems to have a popular stats website, WkStats.com

${await FileAttachment("Screenshot 2023-12-24 at 2.37.03 PM.png").image()} However, this visualization is one of **outputs** - IE, how much progress I am making, not **inputs** - IE, how hard am I working, which is ultimately the thing I can control. Luckily, WaniKani has an API. Having some free time over the holidays, I decided I'd try to spin up Observable & see what I could make for myself. Googling around, [Fedor](https://observablehq.com/@indutny/wanikani-stats) had this idea before me and made some cool visualizations, saving me the work of getting the API working. Thanks Fedor! ### The Problem: fetching reviews. Unfortunately, the thing I want relies on seeing my review history, and the [WaniKani API](https://docs.api.wanikani.com/20170710/#reviews) at some point turned off the Reviews endpoint, likely due to scaling issues.

${await FileAttachment("Screenshot 2023-12-24 at 2.42.02 PM.png").image()} ### Workarounds 1. **Try fetch the reviews independently**, by `review_id`: `GET https://api.wanikani.com/v2/reviews/`. However, as far as I can tell, the `review_id` is not referenced elsewhere in the API - and also, it feels like bad manners given even the aggregate call seems to be costly. 2. **Undocumented APIs**? Next, I tried to see if any of the WaniKani front-end uses either Review IDs or an undocumented API I could rely on. However, from what I can tell, WaniKani does the old-school thing and prepares all of its data server-side, avoiding server-side calls during the review session (super convenient for working on trains with spotty wifi). Props for that - but unfortunately, no good for me. 3. **Give up and compromise.** Ultimately, I accepted the same compromise that ChatGPT suggested: graph "passes" and "burns" - IE, learning a term well enough that it is "solid" and "perfect" known. This is still an **output** metric, but it's a much closer proxy to effort than level progression. 4. **Daily Cron of Assignments. ** Thinking about it more, if I scraped assignments daily, I could see which ones have changed and use the delta to determine the amount of reviews that the user conducted in the past 24 hours. However, this does not work retrospectively. If I do go down that route, it looks like these days Observable has a [cron](https://observablehq.com/@endpointservices/cron) and [simple storage](https://observablehq.com/@endpointservices/cellstore) solution. 5. **Ask Wanikani for help** - which was part of the motivation for this post, actually. Is there an easy way you guys can make daily review statistics accessible, either via API or just as something that can easily be seen in your review UI? ## Observable Plot & ChatGPT Data visualization has always something I've always been fascinated by but not particularly talented at or having the patience for. Between [Observable.Plot](https://observablehq.com/plot/) and ChatGPT, the work required to get the visualization to work (and add colors, and a granularity metric) got an order of magnitude easier. Rather than googling around various places to find the answer, I left that to ChatGPT and instead focused on fixing and adjusting the code samples it sent. ### Hallucinations My favorite ChatGPT error, actually, was when it assumed that assignments _must_ have a `reviewed_at date:

${await FileAttachment("Screenshot 2023-12-24 at 6.11.05 PM.png").image()} It simply assumed that there _must_ have been a reasonable API - not actually correct, but certainly reasonable in terms of its intuition for API design. I still don't fully understand Plot, and I don't think ChatGPT does other - but we **were** able to put together a pretty decent tool, and I don't think we'd have particular trouble adapting it as more functionality becomes available. In the future, the "technical enough PM" will be incredibly empowered. ### Prior Work Poking around the WaniKani forums, I see https://public.dasding.online/wanikanitm2/ having taken a similar approach, potentially with a slightly nicer chart. Hopefully we will all have access to full # reviews one day and can upgrade. It looks like these extensions (https://community.wanikani.com/t/userscript-wanikani-workload-graph/49731, https://community.wanikani.com/t/userscript-wanikani-heatmap/34947/2089) was in a similar boat and got blocked in April as well. ### If you got this far... You might as well subscribe to my newsletter, or check out the blog at [alexeymk.com](https://alexeymk.com). It's about Growth Engineering. 4c:T482,async function fetchPaged(url, type) { const storageKey = `${API_KEY}:${url}:v1`; let lastUpdated = localStorage.getItem(`${storageKey}:lastUpdated`); let result = JSON.parse(localStorage.getItem(`${storageKey}:data`) ?? '[]'); let current = url; if (lastUpdated) { current = `${current}?updated_after=${lastUpdated}`; } while (current) { const res = await fetch(current, { cache: 'no-store', headers: { authorization: `Bearer ${API_KEY}`, 'wanikani-revision': '20170710', }, }); if (!res.ok) { // Throw an error with the status text, which contains the error message throw new Error(`HTTP error! status: ${res.status}, statusText: ${res.statusText}`); } const body = await res.json(); result = result.concat(body.data); lastUpdated = body.data_updated_at; current = body.pages.next_url; } if (lastUpdated) { localStorage.setItem(`${storageKey}:lastUpdated`, lastUpdated); } localStorage.setItem(`${storageKey}:data`, JSON.stringify(result)); return result.map(({ data }) => ({ ...data, type, created_at: new Date(data.created_at) })); }4d:[{"id":"049ab238320644e4a1191c42ed1a13583e245d5dfab4b1a850debffc00f10267d5f6c7aef6ccafaaea9c3f5e9301b6ac2485fb61b485ca5276f2109f5b5f8b3b","href":"https://static.observableusercontent.com/files/049ab238320644e4a1191c42ed1a13583e245d5dfab4b1a850debffc00f10267d5f6c7aef6ccafaaea9c3f5e9301b6ac2485fb61b485ca5276f2109f5b5f8b3b","name":"Screenshot 2023-12-24 at 6.11.05 PM.png","type":"image/png","lastModified":1703409096985.871,"size":193973},{"id":"9812f21d051d2215d02690269123c5d865e3715a2e5e5697cdde14481551e60e0ba2ea2785a53b1fa4e843cb6cd706ed42c502d8ab43cef5743790200a1ed1b6","href":"https://static.observableusercontent.com/files/9812f21d051d2215d02690269123c5d865e3715a2e5e5697cdde14481551e60e0ba2ea2785a53b1fa4e843cb6cd706ed42c502d8ab43cef5743790200a1ed1b6","name":"Screenshot 2023-12-24 at 2.37.03 PM.png","type":"image/png","lastModified":1703396230317.269,"size":197912},{"id":"b5ce1036289d0f16ab0a4d32af2c5a5e61192fc7fad916e7aa95b30faeb4975fb167af8eea308eefcc71516990fe606795694265e6ba477de47ea439b1b59fcf","href":"https://static.observableusercontent.com/files/b5ce1036289d0f16ab0a4d32af2c5a5e61192fc7fad916e7aa95b30faeb4975fb167af8eea308eefcc71516990fe606795694265e6ba477de47ea439b1b59fcf","name":"Screenshot 2023-12-24 at 2.42.02 PM.png","type":"image/png","lastModified":1703396531104.563,"size":66935}] 47:{"id":"b4b78c921f8898da","slug":"wanikani-progress","title":"Tracking WaniKani Progress Per Day","thumbnail":"8c5a6819d3647462e96d70356c514b58e52384a8be3d10f4ba088957b1793100","access_level":"public","create_time":"$D2023-12-23T06:03:46.950Z","update_time":"$D2023-12-24T14:09:25.196Z","view_time":null,"first_public_time":"$D2023-12-23T06:03:46.950Z","has_unpublished_edits":null,"fork":{"id":"d92866f05017cd7e","slug":"wanikani-stats","title":"WaniKani Stats","owner":{"id":"45d16fa014230070","login":"indutny","name":"Fedor Indutny"}},"stars":0,"starred":false,"owner":{"type":"team","id":"e14ef6c8566ebea1","login":"alexeymk","name":"Alexey Komissarouk","avatar_url":"https://avatars-next.observableusercontent.com/0f6d561d4d7c5f25de8ea2c87156dea3b3b50b961c90f1294c4c5cf8ca6fdb8f","tier":"starter_2024","bio":"","followed_by_you":false,"follows_you":false},"version":1150,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"0c73579b7872f3ef","login":"alexeymk","name":"Alexey Komissarouk","avatar_url":"https://avatars-next.observableusercontent.com/0f6d561d4d7c5f25de8ea2c87156dea3b3b50b961c90f1294c4c5cf8ca6fdb8f","tier":null,"bio":"","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"0c73579b7872f3ef","login":"alexeymk","name":"Alexey Komissarouk","avatar_url":"https://avatars-next.observableusercontent.com/0f6d561d4d7c5f25de8ea2c87156dea3b3b50b961c90f1294c4c5cf8ca6fdb8f","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Tracking WaniKani Progress Per Day","cells":[{"id":0,"value":"# Tracking WaniKani Progress Per Day\n**TL;DR:** I made a chart to visualize your Japanese Kanji study progress - see below!","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":49,"value":"viewof API_KEY = Inputs.text({\n label: \"API Key\",\n type: 'password',\n value: localStorage.getItem(\"api_key\") ?? \"ed970ca5-2131-460d-bd8a-95ad4ff6d29c\",\n description:\"defaulted to Alexey's key so there is interesting data right away\" \n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1036,"value":"_Defaulting to Alexey's (read-only) key_ \n\nIf you're using WaniKani to memorize Kanji, see your version of these stats by grabbing your API key: https://www.wanikani.com/settings/personal_access_tokens","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":878,"value":"Plot.plot({\n marks: [\n Plot.rectY(_.concat(passedAtFormatted, burnedAtFormatted), {interval: timeGranularity, x: \"date\", y: \"count\", fill: \"type\"}),\n\n ],\n x: {\n label: \"Date →\",\n tickFormat: d3.timeFormat(\"%Y-%m-%d\")\n },\n y: {\n grid: true,\n label: \"Terms Mastered\"\n },\n width: 800,\n height: 500,\n color: {\n legend: true,\n domain: [\"kana_vocabulary guru\", \"kanji guru\", \"radical guru\", \"vocabulary guru\", \n \"kana_vocabulary burned\", \"kanji burned\", \"radical burned\", \"vocabulary burned\"],\n range: [\"rgb(0, 128, 0)\", \"rgb(60, 179, 113)\", \"rgb(34, 139, 34)\", \"rgb(50, 205, 50)\", \n \"rgb(255, 0, 0)\", \"rgb(178, 34, 34)\", \"rgb(205, 92, 92)\", \"rgb(139, 0, 0)\"]\n // hardcoding colors because multiple scales was unintuitive.\n }\n})\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1045,"value":"viewof timeGranularity = Inputs.select([\"day\", \"week\", \"month\", \"quarter\"], {value: \"month\", label: \"Time Granularity\"})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":925,"value":"$4b","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":790,"value":"## Appendix: The Code","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1079,"value":"// how many of each type do i know at least as well as foo","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":939,"value":"passedAtFormatted = _(assignmentsPassedPerDay).toPairs().map(([date_and_type, count]) => { \n const [date, type] = date_and_type.split(\",\")\n return { date: new Date(date), type: `${type} guru`, count} }\n).sortBy('type')\n.value()\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":999,"value":"burnedAtFormatted = _(assignments)\n .filter((r) => r.burned_at)\n .groupBy((r) => [new Date(r.burned_at).toDateString(), r.subject_type])\n .mapValues('length') \n .toPairs().map(([date_and_type, count]) => { \n const [date, type] = date_and_type.split(\",\")\n return { date: new Date(date), type: `${type} burned`, count} }\n).sortBy('type')\n.value()\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":482,"value":"---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":11,"value":"reviews = await fetchPaged('https://api.wanikani.com/v2/reviews', 'review') \n// broken according to the WK API","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":843,"value":"reviewStats = await fetchPaged('https://api.wanikani.com/v2/review_statistics', 'review_statistics')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":848,"value":"reviewsByDay = _.groupBy(reviewStats, (r) => new Date(r.created_at).toDateString())\n\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":871,"value":"correctByDay = _.mapValues(reviewsByDay, (oneDaysReviews) => {\n // return _.sumBy(oneDaysReviews, (r) => r.meaning_correct + r.reading_correct)\n return oneDaysReviews.length\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":874,"value":"levelProgressions = await fetchPaged('https://api.wanikani.com/v2/level_progressions', 'levelProgression')\n// left behind from fork but may be useful to add to chart","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":852,"value":"assignments = fetchPaged('https://api.wanikani.com/v2/assignments', 'assignments')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":927,"value":"assignmentsPassedPerDay = _(assignments)\n .filter((r) => r.passed_at)\n .groupBy((r) => [new Date(r.passed_at).toDateString(), r.subject_type]) \n .mapValues( assignmentsCompletedOnDay => assignmentsCompletedOnDay.length) \n .value()\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":152,"value":"$4c","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"{\n localStorage.setItem('api_key', API_KEY);\n return html`Status: persisted`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1083,"value":"------\n# Scratch","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1136,"value":"

\n ${progress_plots[0]}\n

\n ${progress_plots[1]}\n

\n ${progress_plots[2]}\n

\n ${progress_plots[3]}\n

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":1085,"value":"progress_plots = [\"kana_vocabulary\", \"kanji\", \"radical\", \"vocabulary\"].map(target_subject_type => {\n return Plot.plot({\n marks: [\n Plot.rectY(srs_progress_by_type.filter(p => p.subject == target_subject_type), \n {x: \"subject\", y: \"count\", fill: \"srs_stage\"}),\n // Plot.tip(_(srs_progress_by_type).reverse().filter(p => p.subject == target_subject_type), \n // {x: \"subject\", y: \"count\", fill: \"srs_stage\"})\n // TODO from here: make the plots nicer, and figure out why the tip is in reverse order - also probably add a color scale.\n ],\n x: {\n label: \"Type\",\n },\n y: {\n grid: true,\n label: target_subject_type\n },\n height: 1200,\n color: {\n legend: true\n }\n })\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1088,"value":"srs_progress_by_type = _(assignments)\n .groupBy((r) => [r.subject_type, r.srs_stage])\n .mapValues('length')\n .toPairs()\n .map(([subject_comma_stage, count]) => { \n const [subject, srs_stage] = subject_comma_stage.split(\",\")\n return { subject, srs_stage, count }\n })\n .sortBy('subject', 'srs_stage')\n .reverse()\n .value()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1117,"value":"srs_at_least_level_type_type = aggregateAtLeast(srs_progress_by_type)\n\n","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1122,"value":"// Function to aggregate data on an \"at least\" basis - ChatGPT\naggregateAtLeast = (data) => {\n // Group data by subject\n const groupedBySubject = _.groupBy(data, 'subject');\n\n return _.map(groupedBySubject, (entries, subject) => {\n // Sort entries by SRS stage in descending order\n const sortedEntries = _.orderBy(entries, entry => parseInt(entry.srs_stage), 'desc');\n\n // Accumulate counts\n return sortedEntries.reduce((acc, entry, index) => {\n if (index === 0) {\n acc.push(entry);\n } else {\n acc.push({\n subject: entry.subject,\n srs_stage: entry.srs_stage,\n count: entry.count + acc[index - 1].count\n });\n }\n return acc;\n }, []);\n }).flat();\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4d","stdlib":"1"}} 24:null