31:T777,md`[MoveNet Model](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection/src/movenet) is the latest pre-trained machine learning library [released](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html) by [TensorFlow](https://www.tensorflow.org) team as part of a larger [Pose Detection](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection) [TensorFlow.js models](https://www.tensorflow.org/js/models) set. MoveNet is a convolutional neural network model that runs on RGB images and predicts human joint locations of a single person that runs faster than real-time ([30+ FPS](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html)). The [pose Animator](https://blog.tensorflow.org/2020/05/pose-animator-open-source-tool-to-bring-svg-characters-to-life.html) meshes SVG, [face landmarks detection](https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection), and motion capture with [pose estimation](https://www.tensorflow.org/lite/examples/pose_estimation/overview) to allow [real-time human pose estimation in the browser](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5). We encouraged you to check our [2D Webcam Pose Detection With PoseNet](https://observablehq.com/@mt-cs/posenet-webcam-1) to compare the robustness of the two pose detection models. Compared to TensorFlow's older model - PoseNet, the MoveNet model has the combination of speed and accuracy needed that allows a more accurate depiction of difficult poses and motions. The new [Pose Detection](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection) API supports multiple models for running real-time pose estimation by simply estimating where key body joints are. Although impressive, the pose is still in 2D.`32:T437,md`## Multi-Webcam 3D Pose tracking application Whether it's for games, motion analysis in health and fitness, virtual reality development, or medical examinations, many applications require that people and their movements be captured digitally in 3D in real-time. Until now, this was possible only with expensive systems using multiple cameras and/or by having people wear special suits and tracking points or tracking devices. This project offers the possibility to turn smartphones and webcams into multi-view body tracking systems without using any trackers. The long-term goal of this project is to interact in the VR world without controllers or wearables. By detecting 3D gestures in non-costly ways, we hope to reduce the need for body trackers, hardware usage, and also computational power and make 3D interaction in VR more accessible. Full-body tracking for skeletal animation allows for more user expression that could humanize users in the VR world. There are endless use cases including hand motion interaction with data, video games, and virtual meetings in 3D.`33:[{"id":"1c7bfd8bb1ed195927a0bcba73c67eb9720ba8923a23ee85b8cf4bca50ea8892f97c27a309f79b77d59be74768dc2a36b04bac29a3cad3638a744d09b9477ba4","href":"https://static.observableusercontent.com/files/1c7bfd8bb1ed195927a0bcba73c67eb9720ba8923a23ee85b8cf4bca50ea8892f97c27a309f79b77d59be74768dc2a36b04bac29a3cad3638a744d09b9477ba4","name":"camera_setup.png","type":"image/png","lastModified":1628013159312.393,"size":118579}] 2f:{"title":"Webcam 1: MoveNet AI Powered 3D Human Pose Tracking and Analysis","cells":[{"id":0,"value":"md`# Webcam 1: MoveNet AI Powered 3D Human Pose Tracking and Analysis`","pinned":false,"hidden":false,"mode":"ojs"},{"id":472,"value":"md` This notebook is the webcam 1 client-side of the [AI-Powered 3D Human Pose Tracking and Analysis Project](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":750,"value":"md`** Goal:**\n\n * Detect 2D pose using MoveNet\n * Send the skeleton key points to the server through Socket.IO \n * Save the data in a CSV file`","pinned":false,"hidden":false,"mode":"ojs"},{"id":566,"value":"md` ## How to run this project \n1. Run [Server](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio/blob/main/README.md)\n2. Open [Receiver](https://observablehq.com/@mt-cs/movenet-3d-pose-tracking-receiver)\n3. Run Webcam 1 and this [Webcam 2](https://observablehq.com/@mt-cs/movenet-3d-pose-tracking-webcam-2)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":478,"value":"md`## 2D Pose Detection with MoveNet TensorFlow.js`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":266,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":645,"value":"links = html`

Compare with PoseNet

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":269,"value":"md`## Tracking two angles to get 3D pose tracking\n\nBy positioning two webcams, pointed at orthogonal directions, we can combine the two 2D poses into a single 3D pose stream.\n\nTraditional 3D motion capture systems require a sophisticated setup and are very costly. This system makes 3D pose capture accessible to much more people. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":558,"value":"html`

\n ${await FileAttachment(\"camera_setup.png\").image()}\n

Image: Camera Setup

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":491,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":493,"value":"md`---\n# Let's Run Webcam 1`","pinned":false,"hidden":false,"mode":"ojs"},{"id":320,"value":"md`***Turn on live feed from your webcam:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":11,"value":"viewof webcam = toggle({\n title_off: \"Webcam off\",\n title: \"Webcam on\",\n value: false\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":323,"value":"md`Toggle on and off to reset webcam if you get the video element has not loaded data:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":293,"value":"viewof resetWebcam = toggle({\n title_off: \"Reset webcam\",\n title: \"Toggle off\",\n value: false\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":149,"value":"source = { /* return webcam video as source */\n if (webcam) {\n const vid = html`

`;\n navigator.getUserMedia(\n { video: true, audio: false },\n (stream) => {\n vid.srcObject = stream;\n vid.height = vid.offsetHeight;\n vid.play();\n },\n () => {}\n );\n return vid;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":335,"value":"md`Movenet will run webcam feed through a machine learning model and output an object stored in the first element of the array below:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":147,"value":"pose = { /* return pose estimation with 17 skeleton points */\n let pose;\n let frame;\n let poses = [];\n\n // Without these if else statement, the webcam would not reset and load data\n if (webcam && !resetWebcam) { \n return getPose();\n } else if (webcam && resetWebcam) {\n return getPose();\n } else if (!webcam && resetWebcam) {\n pose = detector.estimatePoses(source);\n } else {\n pose = detector.estimatePoses(source);\n }\n return pose;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"md`## Pose Detection Controls`","pinned":false,"hidden":false,"mode":"ojs"},{"id":391,"value":"viewof pointColor = color({\n value: \"#feffad\", //'#00ff4c',\n description: \"Point/Line Color\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":406,"value":"viewof textColor = color({\n value: \"#5bbeaa\", //'#00ffff',', //'#00ffff',\"#eeeeee\"\n description: \"Text Color\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":397,"value":"viewof lineWidth = Inputs.range([1, 5], {\n value: 2,\n step: 1,\n label: \"Skeleton Line Width\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":70,"value":"viewof showFacePoints = Inputs.toggle({label: \"Show Face Points\", value: true})","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"viewof showLabels = Inputs.toggle({label: \"Show Point Labels\", value: true})","pinned":false,"hidden":false,"mode":"ojs"},{"id":162,"value":"stickFigure = {\n /* Plots the 17 keypoints on webcam image stream as canvas */\n const canvas = DOM.canvas(source.width, source.height); //Type error width will disappear once the webcam is on\n const ctx = canvas.getContext(\"2d\");\n const scale = 1;\n\n if (webcam) {\n // draw canvas\n ctx.drawImage(source, 0, 0, source.width, source.height);\n }\n // mark poses on the webcam canvas\n markPoses(canvas, pose, showLabels, showFacePoints, pointColor, textColor);\n\n // sending keypoints to server\n sendKeypoints(pose[0].keypoints, 1, ctx, scale);\n return canvas;\n}\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":413,"value":"md`## Socket.io`","pinned":false,"hidden":false,"mode":"ojs"},{"id":443,"value":"md`[Socket.IO](https://socket.io/docs/v4/index.html) is a library that enables real-time, bidirectional, and event-based communication between the browser and the server. It is divided into two parts:\n* Server-Side: it is a Node.js server\n* Client-Side: it is a Javascript client library for the browser (which can be also run from Node.js), In this project the client is both webcam 1 and webcam 2 ObservableHQ notebook.\n\nWhile data can be sent in multiple forms, JSON is the simplest.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":428,"value":"md`***Enter your server URL:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":421,"value":"viewof serverUrl = text({\n placeholder: \"https://summerproject.kineviz.com:9000\",\n description: \"Configure the socket server URL\",\n value: \"https://summerproject.kineviz.com:9000\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":184,"value":"socket = io(serverUrl, {\n withCredentials: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":182,"value":"io = require(\"socket.io-client@2.3.0/dist/socket.io.dev.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":202,"value":"function sendKeypoints(keypoints, minConfidence, ctx, scale) {\n /* store and emit data */\n let skeleton = [];\n // loop through the 17 keypoints\n for (let i = 0; i < keypoints.length; i++) {\n const keypoint = keypoints[i];\n\n // store x, y and bodyPart in an object to be pushed to CSV\n const xyPoints = {\n xPoint: keypoint.x,\n yPoint: keypoint.y,\n bodyPart: keypoint.name\n };\n skeleton.push(xyPoints);\n }\n\n // create message object to be emitted\n const message = {\n webcam: 1,\n data: keypoints,\n timestamp: Date.now()\n };\n\n // save skeleton to an array\n xyWebcam1.push(skeleton);\n\n // emit message through socket.io to server\n socket.emit(\"skeleton\", message);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":229,"value":"md`## Saving Data`","pinned":false,"hidden":false,"mode":"ojs"},{"id":601,"value":"md`We are saving the skeleton data points in an array for the purpose of project demo and further data analysis.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":614,"value":"md`***Run to make sure the array contains skeleton data:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":239,"value":"xyWebcam1","pinned":true,"hidden":false,"mode":"ojs"},{"id":606,"value":"md`***Then run cell below to download the CSV file:***`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":234,"value":"DOM.download(serialize(xyWebcam1), \"movenet_webcam1\", \"Download CSV\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":705,"value":"md`---\n*Utility functions below:*`","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"function markPoses(\n canvas,\n poses,\n showLabels,\n showFacePoints,\n pointColor,\n textColor\n) {\n const context = canvas.getContext(\"2d\");\n context.font = \"16px _sans\";\n context.textBaseline = \"top\";\n context.fillStyle = textColor;\n context.strokeStyle = pointColor;\n context.lineWidth = lineWidth;\n if (poses) {\n poses.forEach((pose) =>\n drawPose(context, pose, showLabels, showFacePoints)\n );\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":153,"value":"async function* getPose() { /* Asynchronous function to estimate pose */\n let pose;\n let i = 0;\n while (true) {\n if (i % 1500 == 0) { // limiting pose to yield\n pose = await detector.estimatePoses(source);\n yield pose;\n }\n i = i + 1;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"mutable xyWebcam1 = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":92,"value":"md`---\n## Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":23,"value":"model = poseDetection.SupportedModels.MoveNet","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"detector = await poseDetection.createDetector(model)","pinned":false,"hidden":false,"mode":"ojs"},{"id":32,"value":"poseDetection = requireTF(\"@tensorflow-models/pose-detection@0.0.3/dist/pose-detection.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":29,"value":"import {\n tf,\n requireTF,\n drawPose,\n drawLine,\n drawPoint,\n facePoints,\n sparkbar\n} from \"@mt-cs/movenet-posenet-image-pose-tracking\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":27,"value":"import { file, slider, text, select, color } from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":144,"value":"import {toggle} from '@kelleyvanevert/little-things'","pinned":false,"hidden":false,"mode":"ojs"},{"id":237,"value":"import { serialize } from \"@palewire/saving-csv\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":433,"value":"md`---\n\n## Credits\n\nDeveloped for the [Kineviz](https://kineviz.com/) and [University of San Francisco](https://www.usfca.edu/arts-sciences/graduate-programs/computer-science?gclid=Cj0KCQjwu7OIBhCsARIsALxCUaOnomqGePhLPOYnIdNZNowVP2aSLsepdMYOe9W5Gb3InnI8b5lDKxAaAk03EALw_wcB) Summer Project by:\n* [Marisa Tania](https://www.linkedin.com/in/marisatania/)\n* [Barbora Novakova](https://www.linkedin.com/in/barbora-novakova-666029126/)\n\nSponsor: Weidong Yang\n\nCode credits:\n* PoseNet: [Loren Riesenfeld](https://observablehq.com/@lorenries/estimating-pose-in-the-browser-with-posenet-and-tensorflow-)\n* MoveNet: [Taras Novak](https://observablehq.com/@randomfractals/tensorflow-movenet-webcam)\n* Socket.io: [Sylvain Lesage](https://observablehq.com/@severo/hello-socket-io)\n---\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":671,"value":"md`## Acknowledgments\n\nWe sincerely thank:\n* [Weidong Yang](https://www.linkedin.com/in/yangweidong/) and the [Kineviz](https://www.kineviz.com/) Team for our biweekly meetings\n* [Nikko Sacramento](https://www.linkedin.com/in/nikkosac/) for plotting the 3D skeleton in GraphXR and running the data in VR\n* Sean Li for debugging the cors issue\n* [Thom Lauret](https://www.linkedin.com/in/thomlauret/) for the more robust AWS server\n* Professor [Mario Lim](https://www.linkedin.com/in/mario-lim-243556/) for helpful crossover network connection discussions\n* [Alex Law](https://www.linkedin.com/in/alexandria-law/) for posting the blogpost and voice over\n* [Cynthia Yu](https://www.shuhuancynthiayu.com/about) for being our model`","pinned":false,"hidden":false,"mode":"ojs"},{"id":666,"value":"md`---\n\n## Contact\n\nThe notebook was implemented by [Barbora Novakova](mailto:barunka838@gmail.com) and [Marisa Tania](mailto:mt.marisatania@gmail.com). Feel free to reach out!\n\n---`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}