31:T42d,md`## Multi-Webcam 3D Pose tracking application Whether it's for games, motion analysis in health and fitness, virtual reality development, or medical examinations, many applications require that people and their movements are captured digitally in 3D in real-time. Until now, this was possible only with expensive systems of multiple cameras, or by having people wear special suits and tracking gears. This project offers the possibility of turn smartphones and webcams into a multi-view body tracking system without using any trackers. The long term goal of this project is to interact in the VR world with no controllers nor wearables. By detecting 3D gestures in non-costly ways, we hope to reduce the need for body trackers, hardware usages, and also computational power and make 3D interaction in VR more accessible. Full-body tracking for skeletal animation allows for more user expression, which could humanize users in the VR world. There are endless possible use cases including hand motion interaction with data, video games, and virtual meetings in 3D. `32:[{"id":"1c7bfd8bb1ed195927a0bcba73c67eb9720ba8923a23ee85b8cf4bca50ea8892f97c27a309f79b77d59be74768dc2a36b04bac29a3cad3638a744d09b9477ba4","href":"https://static.observableusercontent.com/files/1c7bfd8bb1ed195927a0bcba73c67eb9720ba8923a23ee85b8cf4bca50ea8892f97c27a309f79b77d59be74768dc2a36b04bac29a3cad3638a744d09b9477ba4","name":"camera_setup.png","type":"image/png","lastModified":1628047549760.824,"size":118579}] 2f:{"title":"Webcam 2: PoseNet AI Powered 3D Human Pose Tracking and Analysis","cells":[{"id":0,"value":"md`# Webcam 2: PoseNet AI Powered 3D Human Pose Tracking and Analysis`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1445,"value":"md` This notebook is the webcam 2 client-side of the [AI-Powered 3D Human Pose Tracking and Analysis Project](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1579,"value":"md`** Goal:**\n\n * Detect 2D pose using PoseNet\n * Send the skeleton key points to the server through Socket.IO \n * Save the data in a CSV file`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1446,"value":"md`\nHow to run this project: \n1. Run [Server](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio/blob/main/README.md)\n2. Open [Receiver](https://observablehq.com/@mt-cs/posenet-receiver)\n3. Run [Webcam 1](https://observablehq.com/@mt-cs/posenet-webcam-1) and this Webcam 2`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1452,"value":"md`## 2D Pose Detection with PoseNet TensorFlow.js`","pinned":false,"hidden":false,"mode":"ojs"},{"id":312,"value":"md`[Posenet](https://github.com/tensorflow/tfjs-models/tree/master/posenet) is a pre-trained machine learning library that can estimate human poses. It was released by Google Creative Lab, and built on Tensorflow.js. It's powerful and fast enough to estimate human poses in real time, and works entirely in the browser. Even better, it has a relatively simple API. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":1466,"value":"links = html`

Compare with MoveNet

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1455,"value":"md`## Tracking two angles to get 3D pose tracking\n\nBy positioning two webcams, pointed at orthogonal directions, we can combine the two 2D poses into a single 3D pose stream.\n\nTraditional 3D motion capture systems require a sophisticated setup and are very costly. This system makes 3D pose capture accessible to much more people. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":1458,"value":"html`

\n ${await FileAttachment(\"camera_setup.png\").image()}\n

Image: Camera Setup

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1470,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":1473,"value":"md`---\n## Let's Run Webcam 2`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1475,"value":"md`***Turn on live feed from your webcam:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":248,"value":"viewof webcam = toggle({\n title_off: \"Webcam off\",\n title: \"Webcam on\",\n value: false\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":461,"value":"md`***If your image has multiple figures in it, flip this switch to let Posenet know:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":463,"value":"viewof multiple = toggle({ title_off: 'One figure', title: 'Multiple Figures', value: false })","pinned":false,"hidden":false,"mode":"ojs"},{"id":250,"value":"source = {\n if (webcam) {\n const vid = html`

`;\n navigator.getUserMedia(\n { video: true, audio: false },\n (stream) => {\n vid.srcObject = stream;\n vid.height = vid.offsetHeight;\n vid.play();\n },\n () => {}\n );\n return vid;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":80,"value":"md`Posenet will run your image through a machine learning model and output an object (or an array of objects, if multiple):`","pinned":false,"hidden":false,"mode":"ojs"},{"id":82,"value":"pose = {\n let pose;\n let frame;\n let poses = [];\n\n if (webcam && !multiple) {\n return getSinglePose();\n } else if (webcam && multiple) {\n return getMultiplePoses();\n } else if (!webcam && multiple) {\n pose = model.estimateMultiplePoses(source);\n } else {\n pose = model.estimateSinglePose(source);\n }\n return pose;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1494,"value":"md`## Pose Detection Controls`","pinned":true,"hidden":false,"mode":"ojs"},{"id":268,"value":"viewof toggleImage = toggle({ title_off: 'Background image off', title: 'Background image on', value: false })","pinned":false,"hidden":false,"mode":"ojs"},{"id":373,"value":"md`Overall Confidence Score: **${Math.floor(pose.score*100)}%**`","pinned":false,"hidden":false,"mode":"ojs"},{"id":333,"value":"md`Ok, so it's not perfect. Some of Roger's elbows, knees, and ankles are a bit off. But it's still a pretty accurate representation of his pose. Posenet's confidence score for this image was **${Math.floor(pose.score*100)}%**. We can also fiddle with the accuracy by loading a different model multiplier (warning: this could take a while and/or take a toll on your GPU):`","pinned":false,"hidden":false,"mode":"ojs"},{"id":351,"value":"viewof modelMultiplier = select({\n title: \"Model Multiplier\",\n description: \"See how different model multipliers affect the result\",\n options: ['1.01', '1', '0.75', '0.5'],\n value: '1'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":377,"value":"md`Each keypoint comes with its own confidence score. Let's only render the points that Posenet is relatively sure of:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":380,"value":"viewof confidence = slider({\n min: 0, \n max: 1, \n step: 0.01,\n value: 0.75,\n format: \".0%\",\n description: \"Minimum Confidence Level\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":410,"value":"md`Excellent. Now we can show just the key points that we're pretty confident in. Let's make our pose representation look a little bit more like a human figure by joining the collection of points:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":417,"value":"stickFigure = {\n const canvas = DOM.canvas(source.width, source.height);\n const ctx = canvas.getContext(\"2d\");\n const scale = 1;\n\n if (toggleImage || webcam) {\n ctx.drawImage(source, 0, 0, source.width, source.height);\n }\n\n if (multiple) {\n pose.forEach((figure) => {\n drawSkeleton(figure.keypoints, confidence, ctx, scale);\n drawKeypoints(figure.keypoints, confidence, ctx, scale);\n });\n } else {\n drawSkeleton(pose.keypoints, confidence, ctx, scale);\n drawKeypoints(pose.keypoints, confidence, ctx, scale);\n }\n\n return canvas;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":584,"value":"viewof figColor = color({\n value: \"#f44242\",\n title: \"Color\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":793,"value":"viewof pointRadius = slider({\n min: 1, \n max: 8, \n step: 1, \n value: 4, \n title: \"Point Radius\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":581,"value":"viewof lineWidth = slider({\n min: 1, \n max: 5, \n step: 1, \n value: 2, \n title: \"Skeleton Line Width\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":642,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1502,"value":"md`## Socket.io`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1504,"value":"md`[Socket.IO](https://socket.io/docs/v4/index.html) is a library that enables real-time, bidirectional, and event-based communication between the browser and the server. It is divided into two parts:\n* Server-Side: it is a Node.js server\n* Client-Side: it is a Javascript client library for the browser (which can be also run from Node.js), In this project the client is both webcam 1 and webcam 2 ObservableHQ notebook.\n\nWhile data can be sent in multiple forms, JSON is the simplest.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1507,"value":"viewof serverUrl = text({\n placeholder: \"https://summerproject.kineviz.com:9000\",\n description: \"Configure the socket server URL\",\n value: \"https://summerproject.kineviz.com:9000\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1212,"value":"socket = io(serverUrl, {\n withCredentials: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":880,"value":"io = require(\"socket.io-client@2.3.0/dist/socket.io.dev.js\")\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":1378,"value":"md`---\n## Saving Data`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1533,"value":"md`We are saving the skeleton data points in an array for the purpose of project demo and further data analysis.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1536,"value":"md`***Run to make sure the array contains skeleton data:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1385,"value":"xyWebcam2","pinned":true,"hidden":false,"mode":"ojs"},{"id":1541,"value":"md`***Then run cell below to download the CSV file:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1380,"value":"DOM.download(serialize(xyWebcam2), \"posenet_webcam2\", \"Download CSV\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":622,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":617,"value":"md`*Utility functions below taken from the [Posenet repo](https://github.com/tensorflow/tfjs-models/blob/master/posenet/demos/demo_util.js):*`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1090,"value":"function drawKeypoints(keypoints, minConfidence, ctx, scale) {\n let skeleton = [];\n for (let i = 0; i < keypoints.length; i++) {\n const keypoint = keypoints[i];\n\n if (keypoint.score < minConfidence) {\n continue;\n }\n const { y, x } = keypoint.position;\n const xyPoints = {\n xPoint: x,\n yPoint: y,\n bodyPart: keypoint.part\n };\n skeleton.push(xyPoints);\n drawPoint(ctx, y * scale, x * scale, pointRadius, color);\n }\n\n const message = {\n webcam: 2,\n data: keypoints,\n timestamp: Date.now(),\n model: \"posenet\"\n };\n\n xyWebcam2.push(skeleton);\n\n socket.emit(\"skeleton\", message);\n}\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":634,"value":"async function* getSinglePose() {\n let pose;\n let i = 0;\n while (true) {\n if (i % 1500 == 0) {\n pose = await model.estimateSinglePose(source);\n yield pose;\n }\n i = i + 1;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1356,"value":"async function* getMultiplePoses() {\n let pose;\n while (true) {\n pose = await model.estimateMultiplePoses(source);\n yield pose;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":564,"value":"function drawBoundingBox(keypoints, ctx) {\n const boundingBox = posenet.getBoundingBox(keypoints);\n\n ctx.rect(boundingBox.minX, boundingBox.minY,\n boundingBox.maxX - boundingBox.minX, boundingBox.maxY - boundingBox.minY);\n\n ctx.stroke();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":607,"value":"function drawSkeleton(keypoints, minConfidence, ctx, scale) {\n const adjacentKeyPoints = posenet.getAdjacentKeyPoints(\n keypoints, minConfidence);\n\n adjacentKeyPoints.forEach((keypoints) => {\n drawSegment(toTuple(keypoints[0].position), toTuple(keypoints[1].position), figColor, scale, ctx);\n });\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":572,"value":"function drawPoint(ctx, y, x, r, color) {\n ctx.beginPath();\n ctx.arc(x, y, r, 0, 2 * Math.PI);\n ctx.fillStyle = figColor;\n ctx.fill();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":575,"value":"function drawSegment([ay, ax], [by, bx], color, scale, ctx) {\n ctx.beginPath();\n ctx.moveTo(ax * scale, ay * scale);\n ctx.lineTo(bx * scale, by * scale);\n ctx.lineWidth = lineWidth;\n ctx.strokeStyle = figColor;\n ctx.stroke();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":604,"value":"function toTuple({y, x}) {\n return [y, x];\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1382,"value":"mutable xyWebcam2 = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":391,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":69,"value":"md`## Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":402,"value":"model = await posenet.load(+modelMultiplier)","pinned":true,"hidden":false,"mode":"ojs"},{"id":66,"value":"posenet = require('@tensorflow-models/posenet@1.0.3')","pinned":true,"hidden":false,"mode":"ojs"},{"id":62,"value":"import { file, slider, select, text, color } from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":273,"value":"import {toggle} from \"@kelleyvanevert/little-things\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1376,"value":"import { serialize } from \"@palewire/saving-csv\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1374,"value":"json2csv = require(\"json2csv\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1553,"value":"md`---\n\n## Credits\n\nDeveloped for the [Kineviz](https://kineviz.com/) and [University of San Francisco](https://www.usfca.edu/arts-sciences/graduate-programs/computer-science?gclid=Cj0KCQjwu7OIBhCsARIsALxCUaOnomqGePhLPOYnIdNZNowVP2aSLsepdMYOe9W5Gb3InnI8b5lDKxAaAk03EALw_wcB) Summer Project by:\n* [Marisa Tania](https://www.linkedin.com/in/marisatania/)\n* [Barbora Novakova](https://www.linkedin.com/in/barbora-novakova-666029126/)\n\nSponsor: Weidong Yang\n\nCode credits:\n* PoseNet: [Loren Riesenfeld](https://observablehq.com/@lorenries/estimating-pose-in-the-browser-with-posenet-and-tensorflow-)\n* MoveNet: [Taras Novak](https://observablehq.com/@randomfractals/tensorflow-movenet-webcam)\n* Socket.io: [Sylvain Lesage](https://observablehq.com/@severo/hello-socket-io)\n---\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1556,"value":"md`## Acknowledgments\n\nWe sincerely thank:\n* [Weidong Yang](https://www.linkedin.com/in/yangweidong/) and the [Kineviz](https://www.kineviz.com/) Team for our biweekly meetings\n* [Nikko Sacramento](https://www.linkedin.com/in/nikkosac/) for plotting the 3D skeleton in GraphXR and running the data in VR\n* Sean Li for debugging the cors issue\n* [Thom Lauret](https://www.linkedin.com/in/thomlauret/) for the more robust AWS server\n* Professor [Mario Lim](https://www.linkedin.com/in/mario-lim-243556/) for helpful crossover network connection discussions\n* [Alex Law](https://www.linkedin.com/in/alexandria-law/) for posting the blogpost and voice over\n* [Cynthia Yu](https://www.shuhuancynthiayu.com/about) for being our model`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1558,"value":"md`---\n\n## Contact\n\nThe notebook was implemented by [Barbora Novakova](mailto:barunka838@gmail.com) and [Marisa Tania](mailto:mt.marisatania@gmail.com). Feel free to reach out!\n\n---`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}