29:null 2a:["$","title",null,{"children":"Software Engineering Visualization with GitHub's API | Steve Kasica | Observable"}] 2f:T52a,diffVis = { const width = 960, height = 450; const dur = 500; const y = d3.scaleLinear().domain([0, 150]).range([0, height * 4]); const x = 50; const lines = atob(readmeVersions[currCommit].content) .split('\n') .map((d,i) => ({ text: d, opacity: 1, line: i + 1, id: hash(d) // Generate a unique id for object constancy })); const svg = d3.select(this || DOM.svg(width, height)); svg.attr('font-family', "monospace") .attr("font-size", "10"); const text = svg.selectAll('text') .data(lines, d => d.id); // pass the hash values id in the key function // Update lines text.attr('class', 'updated') .transition() .duration(dur) .attr('y', d => y(d.line)); // Create new lines text.enter() .append('text') .attr('fill-opacity', 0) .attr('class', 'created') .attr('id', d => d.id) .attr('x', width) .attr('y', d => y(d.line)) .text(d => d.text) .transition() .duration(dur) .attr('fill-opacity', 1) .attr('x', x); // Remove old lines text.exit() .attr('class', 'removed') .transition() .duration(dur) .attr('x', 0) .style("fill-opacity", 0) .remove(); return svg.node(); }30:T54e,md`### Rate Limit Quotas GitHub's API has three levels of authentication: [Basic](https://developer.github.com/v3/#basic-authentication), [OAuth](https://developer.github.com/v3/#oauth2-token-sent-in-a-header), and unauthenticated. The Basic (and OAuth) levels give your application **5000!** requests per hour, but the API also allows up to 60 unauthenticated requests per hour (per IP address). See the [Rate Limit docs](https://developer.github.com/v3/rate_limit) for more about rate limit status. Unauthenticated requests will be fine for this notebook, unless you get really aggressive with the bubble chart above. It makes a new request each time you select a different repo. But if you use GitHub's API data in a project, you'll likely need more. So below is how you format request headers for Basic authentication with the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). The *requestInit* cell below contains authorization information, but privately stored in [Secrets](https://beta.observablehq.com/@observablehq/secrets), which are sort of like environment variables in Observable Notebooks. You'll see that *requestInit* is passed to each async request for data via the second argument to *d3.json*. Secrets are only accessable in private notebooks, so the *Secret* function below is just for demonstration purposes.`31:[] 2d:{"id":"caac4b42d1ccd321","slug":"software-engineering-visualization-with-githubs-api","title":"Software Engineering Visualization with GitHub's API","thumbnail":"d270d6a4d18f10cd3824ba26398b23218fd168971c83793752085b42b944a437","access_level":"public","create_time":"$D2018-11-12T18:27:06.211Z","update_time":"$D2022-01-19T04:20:02.026Z","view_time":null,"first_public_time":"$D2018-11-16T15:55:27.010Z","has_unpublished_edits":null,"fork":false,"stars":13,"starred":false,"owner":{"type":"team","id":"1cebfd5d1e47fa9b","login":"stvkas","name":"Steve Kasica","avatar_url":"https://avatars-next.observableusercontent.com/b10311feda71ac109d4c36f5ef7f30ba700e7c0f8160bc5750596e4c28d9e1ab","tier":"starter_2024"},"version":1469,"publish_version":1469,"role":null,"authored":false,"authors":[{"type":"individual","id":"a483da0cdef190cc","login":"stvkas","name":"Steve Kasica","avatar_url":"https://avatars-next.observableusercontent.com/b10311feda71ac109d4c36f5ef7f30ba700e7c0f8160bc5750596e4c28d9e1ab","tier":null}],"members":[{"type":"individual","id":"a483da0cdef190cc","login":"stvkas","name":"Steve Kasica","avatar_url":"https://avatars-next.observableusercontent.com/b10311feda71ac109d4c36f5ef7f30ba700e7c0f8160bc5750596e4c28d9e1ab","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":7,"collection":{"id":"0bccc3c8a41f5b31","slug":"github-on-observable","title":"Github on Observable","owner":{"id":"f35c755083683fe5","login":"observablehq","name":"Observable"}},"type":"notebook","body":{"title":"Software Engineering Visualization with GitHub's API","cells":[{"id":0,"value":"md`# Software Engineering Visualization with GitHub's API\n\n## Or How to Achieve Marks and Profit in CPSC 410!\n\n*This notebook was used in a mini-lecture I gave to UBC students in CPSC 410: Advanced Software Engineering on Nov. 15, 2018*`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":564,"value":"md` $\"Get$
Hat tip to the DSL group that made States Pro for introducing me to this meme\n\nIf you're thinking of visualizing static properties or meta-property analysis for your software visualization project in CPSC 410, then consider GitHub's [REST API v3](https://developer.github.com/v3). Copious amounts of data on commits, file sizes, and version history are just a GET request away. \n\nIt was sooooo easy to make a SE vis like this!\n\n### Example 1: Whose Repo is This Anyway?\n\nBelow is a bubble chart visualizing commits by contributor for a given repo. The circle's radius, and hence its area, is a function of the total number of commits for a given contributor. This category of visualization excels at relative comparisons, when the differences are large enough.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":994,"value":"viewof bubbleRepo = html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1326,"value":"BubbleChart(contributors, {\n value: (d) => d.value,\n group: (d) => d.group,\n label: (d) => `${d.name}\\n${d.value}`,\n link: (d) => `http://github.com/${d.title}`,\n width: width\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":843,"value":"import { BubbleChart } from \"@d3/bubble-chart\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1167,"value":"md`The data in the *contributors* cell came from a [endpoint of repo statistics](https://developer.github.com/v3/repos/statistics). Besides contributor stats, this endpoint also includes data that GitHub uses in its visualizations.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":972,"value":"contributors = d3.json(\"https://api.github.com/repos/\" + bubbleRepo + \"/stats/contributors\").then((data) => {\n return data.map((d) => ({\n name: d.author.login,\n title: d.author.login,\n group: d.author.type,\n value: d.total\n }));\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":817,"value":"md`### Example #2: The Humble File Diff\n\nWhile not the fanciest visualizations, file diffs are arguably the most important because of their ubiquity. This is what file diff applications looked like ${new Date().getFullYear() - 1996} years ago. Pretty familiar, huh?\n\n $\"File$
Ball T., Eick, S. (1996) Software Visualization in the Large. In IEEE Computer, Vol 29, No. 4. April 1996, 33-43\n\n#### File Diffs by Data Joins\n\nTo put a new spin on file diffs, I wanted to visualize a file changing through time as a kinetic typography experiment. If you think of a text file as a dataset and each line is a data point, then you can do a file diff via [data joins in D3](https://bost.ocks.org/mike/join/).\n\nBelow is the recent history of Linux's README file.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1119,"value":"commiter = md`Commit made by ${readmeVersions[currCommit].author.name} on ${dateFormat(new Date(readmeVersions[currCommit].author.date))}.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":179,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":420,"value":"style = html``","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":534,"value":"md `#### Reconstructing File History\n\nYou can reconstruct the history of a file through [the commits endpoint](https://developer.github.com/v3/repos/commits/) and [contents endpoint](https://developer.github.com/v3/repos/contents/). The commits endpoint gives you the history of commits for a given file and the contents endpoint gives you a base64 encoded string of the file at the time of a particular commit. \n\nThe cell below does both using [Promises](https://beta.observablehq.com/@mbostock/introduction-to-promises).`\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1100,"value":"readmeVersions = {\n return d3.json(\"https://api.github.com/repos/torvalds/linux/commits?path=README\", requestInit)\n .then((data) => {\n let res = data.reverse() // commit history is sorted in reverse chronological order\n .filter((d,i) => i !== 25) // Some changes from initial commits are not available\n .slice(19)\n .map((d) => {\n return d3.json(\"https://api.github.com/repos/torvalds/linux/readme?ref=\" + d.sha, requestInit)\n .then((content) => {\n content.author = d.commit.author;\n return content;\n });\n });\n \n return Promise.all(res);\n });\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":812,"value":"md` ##### Sidenote about Observable Notebooks\n\nThe *currCommit* cell is a [generator](https://beta.observablehq.com/@mbostock/introduction-to-generators) that cycles through commits in *readmeVersions*. When this variable changes, the current version of the file also changes.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":199,"value":"currCommit = {\n let i = 0;\n const interval = 2000; // how fast the animation cycles\n while (true) {\n yield Promises.delay(interval, i);\n i++;\n i = i % readmeVersions.length; \n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":466,"value":"md` ##### Sidenote about D3\n\nD3 needs a unique id for each data point to maintain [object consistency](https://bost.ocks.org/mike/constancy). Basically, D3 isn't going to know which line is being added, retained, or removed unless we assign a unique key to each string. Since summing the character codes could result in duplicates, a good hash function is the only tool for the job. While SHA-1 is [not collision free](https://www.theregister.co.uk/2017/02/23/google_first_sha1_collision/), it's good enough for our purposes.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"hash = require('object-hash') // You can ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1077,"value":"md`### Example #3: Organizing Source Code By Subsystem and Size\n\nTreemaps display hierarchical data in as sets of nested rectangles. Unlike a node-link tree, a treemap can also encode a specific dimension of the data in each rectangle's size. In software visualization, you can use this technique to communicate the sizes of different components in hierarchical data.\n\nThe visualization below show file sizes in FiveThirtyEight's Data repo and grouped by parent directory, which represents the article that features these data. These data came from [the trees endpoint](https://developer.github.com/v3/git/trees) on the GitHub API.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1342,"value":"Treemap(tree, {\n path: (d) => d.path,\n value: (d) => d?.size,\n label: (d) => `${d.path.split(\"/\")[0]}\\n${d.path.split(\"/\")[1]}`,\n group: (d) => d.path.split(\"/\")[0],\n width: width,\n height: width\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1065,"value":"import { Treemap } from \"@d3/treemap\"\n// import { chart as treeMap } with {\n// tree as data,\n// treeMapColor as color\n// } from \"@mbostock/d3-treemap\"","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1091,"value":"md`Another cool thing about Observable Notebooks is that you can [import other people's cells](https://beta.observablehq.com/@mbostock/introduction-to-imports) into your notebook. All I had to do to create the treemap and bubble chart was shoehorn data from GitHub into the right format. For the treemap, I also swapped out the function that color codes each group.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1051,"value":"tree = d3\n .json(\n \"https://api.github.com/repos/fivethirtyeight/data/git/trees/9e9cee37d0695ccc6866c67f38373675231758ab?recursive=1\",\n requestInit\n )\n .then((data) => data.tree)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":35,"value":"$30","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1266,"value":"requestInit = {\n return {\n // GITHUB_LOGIN = :\n headers: { authorization: \"Basic \" + btoa(Secret(\"GITHUB_LOGIN\")) }\n };\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1290,"value":"function Secret(name) {\n return null;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":459,"value":"md`### Conclusion and a Public Service Announcement\n\nHopefully this demo gave you some ideas about the data and visualization tools you can use in your software engineering visualization projects. Please see the [official GitHub API docs](https://developer.github.com/v3/), [D3's wiki](https://github.com/d3/d3/wiki), shoot me an email ([kasica@cs.ubc.ca](mailto:kasica@cs.ubc.ca)), if you have any questions.\n\nIf you're interest in learning more about these notebooks, Mike Bostock will be at talking about Observable Notebook at UBC on **Thursday, Feb. 7, 2019**, from 3:30-5:00 p.m. in Hugh Dempster Pavilion Room 110. This event is part of the [Distinguished Lecture Series](https://www.cs.ubc.ca/sites/cs/files/shared/ubc-cs-dls-2018-2019-final-sm.pdf)\n

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":1127,"value":"dateFormat = d3.timeFormat(\"%B %d, %Y\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"d3 = require('d3')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W31","stdlib":"1","titling":"auto"}} 2e:"$2d:body"