31:T576,md`[Socket.IO](https://socket.io/docs/v4/index.html) is a library that enables real-time, bidirectional, and event-based communication between the browser and the server. Socket.IO is **NOT** a WebSocket implementation. Although Socket.IO indeed uses WebSocket as transport when possible, it adds additional metadata to each packet. That is why a WebSocket client will not be able to successfully connect to a Socket.IO server, and a Socket.IO client will not be able to connect to a plain WebSocket server either. **How does that work?** Please see the documentation for **socket.io** [here](https://socket.io/docs/v4/index.html) Socket.IO is a library that enables real-time, bidirectional, and event-based communication between the browser and the server. It is divided into two parts: * Server-Side: it is a Node.js server * Client-Side: it is a Javascript client library for the browser (which can be also run from Node.js), In this project the client is both webcam 1 and webcam 2 ObservableHQ notebook. The client will try to establish a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) connection if possible and will fall back on HTTP long-polling if not. WebSocket is a communication protocol that provides a full-duplex and low-latency channel between the server and the browser. More information can be found [here](https://en.wikipedia.org/wiki/WebSocket).`32:T829,calculateData = (data) => { // store data let avgKeypoints = []; let coordinates = []; let x1 = 0, x2 = 0, y1 = 0, y2 = 0, score1 = 0, score2 = 0, bodyPart = ""; // edge case if both data comes from same webcam if (data2 == 1 && data1 == 1) { return; } // updates data if (data.webcam == "1") { data1[0] = data; } else if (data.webcam == "2") { data2[0] = data; } if (data1[0] != undefined && data2[0] != undefined) { mutable count++; // loop through the 17 keypoints for (let i = 0; i < 17; i++) { x1 = data1[0].data[i].x; x2 = data2[0].data[i].x; y1 = data1[0].data[i].y; y2 = data2[0].data[i].y; score1 = data1[0].data[i].score; score2 = data2[0].data[i].score; bodyPart = data1[0].data[i].name; // calclulate and store data average in an object for 2D plotting in Receiver const keypoint = { score: (score1 + score2) / 2, part: bodyPart, position: { x: (x1 + x2) / 2, y: (y1 + y2) / 2 } }; avgKeypoints.push(keypoint); // store x, y, z coordinates and bodyPart in an object to be pushed to CSV const xyz_points = { pointX: x1, pointY: y1, pointZ: x2, bodyPart: bodyPart }; // save xyz_points to an array coordinates.push(xyz_points); } // save coordinates (17 keypoints) to an array xyzArr.push(coordinates); // drawing the average 2D keypoints on the canvas const ctx = canvas.getContext("2d"); ctx.clearRect(0, 0, canvas.width, canvas.height); ctx.font = "16px _sans"; ctx.textBaseline = "top"; ctx.fillStyle = "#ff0000"; ctx.strokeStyle = figColor; ctx.lineWidth = 2; // loop through the 17 keypoints and draw the skeleton points for (let i = 0; i < avgKeypoints.length; i++) { const keypoint = avgKeypoints[i]; drawPoints(ctx, keypoint, showLabels, showFacePoints); } drawPoses(ctx, avgKeypoints, showLabels, showFacePoints); } return data; }33:T4de,md`For this Project we are using [Kineviz GraphXR](https://www.kineviz.com/?gclid=CjwKCAjwmK6IBhBqEiwAocMc8oAyRP8ZHw55wveQw3luEs4AMI9BLxG7-6yeqDvd5Rb70hV8Z-fRuRoC_kgQAvD_BwE) tool for 3D visualization **How to run collected data in GraphXR** If you want to visualize the CSV data in 3D through Kineviz GraphXR, follow these steps: * Create an account on [GraphXR Next](https://graphxrnext.kineviz.com/register) * Download skel.grove and skelly.graphxr from [GitHub](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio/tree/main/graphXR) folder * Create a new Project in GraphXR Next * Open the recently created project * Click on the Project icon on left and then Extensions and pick Grove * Click on the View attached files and change into desired .csv data * Drag skelly.graphxr and paste it into the background * Hold right-click to rotate the Skeleton as needed * Go to Settings to adjust the visualization * If you want to escape into Virtual Reality World, you can put on your goggles and watch your skeleton in GraphXR VR Watch the full Tutorial [here](https://user-images.githubusercontent.com/55717978/128306293-05b63fce-02df-4850-b2de-71b35a593f3f.mp4) for more details. `34:T773,md`For this project implementation, we chose a configuration that minimizes the mathematical computation complexity, but it's not necessary to have this restriction. • **Placement of the camera**\n The next step is to move away from the orthogonal placement of the webcams, we can experimenting placing the camera in different angles and reconstruct the Z point using epipolar geometry. We can deal with lost frames using the time-domain filter to improve accuracy and precision. • **Improve the stability**\n For improving accuracy and reliability, the [Kalman filter](https://en.wikipedia.org/wiki/Kalman_filter) is one of the most commonly used linear estimators [minimum mean-square-error](https://en.wikipedia.org/wiki/Minimum_mean_square_error). • **Directly connect Receiver to GraphXR**\n In this project, we collect data in CSV before we input them to GraphXR for development and analysis purposes. For future improvements, we can stream data from the receiver directly to Kineviz GraphXR in [Grove](https://www.kineviz.com/grovesample) using Socket.IO. By doing this step, we can open up potential gesture control to interact with data in the 3D virtual environment, as explained below. • **Gesture Recognition in the Virtual World**\n We can use this project to collect **Machine Learning data for Gesture Recognition** and be able to use those gestures in the **Virtual Environment**. Gestures can be used for interactivity with different objects. Most Extended Reality (XR) tracking is done from the glasses so often users do not see their hands. with the two webcams placement in the front users, we can see the hand and full-body gestures. This gives us the ability to immerse ourselves in the virtual environment and integrate the virtual body with the real body. Recognizing simple natural gestures like push, pull, grab, drag can be a game-changer in interactivity in XR.`35:T7ea,// this function draw the line between all the bodyParts function drawPoses(context, pose, showLabels, showFacePoints) { // key points by name const points = new Map(); if (pose != undefined){ for (let j = 0; j < pose.length; j++) { const element = pose[j]; let part = element.part; let position = element.position; points.set(part, position); } // draw pose lines: https://github.com/tensorflow/tfjs-models/tree/master/pose-detection#keypoint-diagram drawLines(context, points.get("left_shoulder"), points.get("right_shoulder")); drawLines(context, points.get("left_hip"), points.get("right_hip")); // left arm drawLines(context, points.get("left_shoulder"), points.get("left_elbow")); drawLines(context, points.get("left_elbow"), points.get("left_wrist")); // left side drawLines(context, points.get("left_shoulder"), points.get("left_hip")); // left leg drawLines(context, points.get("left_hip"), points.get("left_knee")); drawLines(context, points.get("left_knee"), points.get("left_ankle")); // right arm drawLines(context, points.get("right_shoulder"), points.get("right_elbow")); drawLines(context, points.get("right_elbow"), points.get("right_wrist")); // right side drawLines(context, points.get("right_shoulder"), points.get("right_hip")); // right leg drawLines(context, points.get("right_hip"), points.get("right_knee")); drawLines(context, points.get("right_knee"), points.get("right_ankle")); // connect face points if (showFacePoints) { drawLines(context, points.get("right_ear"), points.get("right_eye")); drawLines(context, points.get("right_eye"), points.get("nose")); drawLines(context, points.get("nose"), points.get("left_eye")); drawLines(context, points.get("left_eye"), points.get("left_ear")); } } // draw points and text labels for (let j = 0; j < pose.length; j++) { drawPoint(context, pose[j], showLabels, showFacePoints); } }36:[{"id":"543777dc37c4f26055bcce1bd6053178965467a9fdbc0c5f380ebee0d8569fdeef4ebe32f0266f72005bb4dda92b251a5a35a1d450dad661eaf5185584391bcd","href":"https://static.observableusercontent.com/files/543777dc37c4f26055bcce1bd6053178965467a9fdbc0c5f380ebee0d8569fdeef4ebe32f0266f72005bb4dda92b251a5a35a1d450dad661eaf5185584391bcd","name":"3Dplane","type":"image/jpeg","lastModified":1628014892199.249,"size":17317}] 2f:{"title":"Receiver: MoveNet AI Powered 3D Human Pose Tracking and Analysis","cells":[{"id":0,"value":"md`# Receiver: MoveNet AI Powered 3D Human Pose Tracking and Analysis`","pinned":false,"hidden":false,"mode":"ojs"},{"id":324,"value":"md`This notebook is the Receiver client-side of the [AI-Powered 3D Human Pose Tracking and Analysis Project](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio).\\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":924,"value":"md`** Goal:**\n\n* Combine two corresponding points from *webcam1* and *webcam2* \n* Save the data in a CSV file`","pinned":false,"hidden":false,"mode":"ojs"},{"id":548,"value":"md`\n## How to run this project \n1. Run [Server](https://github.com/kineviz-3D-human-pose-tracking/streamer-receiver-socketio/blob/main/README.md)\n2. Open this Receiver\n3. Run [Webcam 1](https://observablehq.com/@mt-cs/movenet-3d-pose-tracking-webcam-1)\n4. Run [Webcam 2](https://observablehq.com/@mt-cs/movenet-3d-pose-tracking-webcam-2) \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":647,"value":"md`## Three-dimensional space`","pinned":false,"hidden":false,"mode":"ojs"},{"id":717,"value":"html`

\n${await FileAttachment(\"3Dplane\").image()}\n

Image source: ck-12

\n `","pinned":false,"hidden":false,"mode":"ojs"},{"id":623,"value":"md`\nPositioning two webcams pointed at orthogonal directions is the simplest way to combine the two 2D poses into a single 3D pose stream. Both cameras are placed at a 90 degrees angle to get the ***x, y, z coordinates*** for plotting in [Three-dimensional space](https://en.wikipedia.org/wiki/Three-dimensional_space).\n\n*For more information about the proper **camera setup** check [Webcam Notebook](https://observablehq.com/d/823f2f9c4daa588c).*`","pinned":false,"hidden":false,"mode":"ojs"},{"id":636,"value":"md`---\n## Let's Run Receiver`","pinned":false,"hidden":false,"mode":"ojs"},{"id":354,"value":"md`***Enter your server URL:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"viewof serverUrl = text({\n placeholder: \"https://summerproject.kineviz.com:9000\",\n description: \"Configure the socket server URL\",\n value: \"https://summerproject.kineviz.com:9000\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":453,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":338,"value":"md`## Pose Detection Controls`","pinned":false,"hidden":false,"mode":"ojs"},{"id":369,"value":"viewof figColor = color({\n value: \"#5bbeaa\",\n title: \"Color\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":364,"value":"viewof pointRadius = slider({\n min: 1,\n max: 8,\n step: 1,\n value: 4,\n title: \"Point Radius\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"viewof lineWidth = slider({\n min: 1,\n max: 5,\n step: 1,\n value: 2,\n title: \"Skeleton Line Width\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":560,"value":"viewof showFacePoints = Inputs.toggle({\n label: \"Show Face Points\",\n value: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":403,"value":"viewof showLabels = Inputs.toggle({ label: \"Show Point Labels\", value: true })","pinned":false,"hidden":false,"mode":"ojs"},{"id":459,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":426,"value":"md` ## Combined Streams from Webcam 1 and Webcam 2\nWhen **Server**, **Webcam 1,** and **Webcam 2** are running, you can watch the resulting skeleton points combined from 2 webcams [MoveNet](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection/src/movenet) streams, that are detecting 17 key points of a body below:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":14,"value":"canvas = DOM.canvas(640, 480)","pinned":false,"hidden":false,"mode":"ojs"},{"id":417,"value":"md`---\n## Socket.io`","pinned":false,"hidden":false,"mode":"ojs"},{"id":420,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"socket = {\n const socket = io(serverUrl);\n socket.on(\"skeleton\", onDataReceived);\n return socket;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":58,"value":"io = require(\"socket.io-client@2.3.0/dist/socket.io.dev.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":582,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":503,"value":"md`## Calculating Data\nWe successfully capture data from both webcams with an average of 26 frames/second. For plotting purposes, we decided to calculate the *arithmetic mean* between the points from both of the webcam streams to smooth out unstable readings. Then with the proper position of both cameras *(check [Webcam Notebook](https://observablehq.com/d/823f2f9c4daa588c) for more information)* we can get the most accurate results combining the **x, y, z coordinates and body parts** for plotting in 3D plane using [GraphXR](https://www.kineviz.com/?gclid=CjwKCAjwr56IBhAvEiwA1fuqGh4PbQhDFUb_ulQpOE_MuE8dLcr4RX6IC7Wba4RdBh1t64cBR_OC3BoC2twQAvD_BwE).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":784,"value":"md`\n* x1 = webcam1 x point\n* y1 = webcam1 y point\n* x2 = webcam2 x point\n* y2 = webcam2 y point\n* z = proportionate to x2`","pinned":false,"hidden":false,"mode":"ojs"},{"id":773,"value":"md`In this receiver, we are taking an ongoing average of both y points. For a more accurate way to calculate x, y, z points, we recommend finding one calibration point where y1 and y2 have the same value and calibrate the x and z accordingly.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":74,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":462,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"md`## Saving Data`","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"md`We are saving *{pointX, pointY, pointZ, bodyPart}* in an array for the purpose of plotting in 3D plane and further data analysis. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":592,"value":"md`***Run to make sure the array contains points:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":211,"value":"xyzArr","pinned":false,"hidden":false,"mode":"ojs"},{"id":611,"value":"md`***Then run cell below to download the CSV file:***`","pinned":false,"hidden":false,"mode":"ojs"},{"id":208,"value":"DOM.download(serialize(xyzArr), \"movenet_receiver3D\", \"Download CSV\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":433,"value":"md`---\n## 3D Visualization in GraphXR`","pinned":false,"hidden":false,"mode":"ojs"},{"id":856,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":902,"value":"md`---\n## Future Improvements and Applications`","pinned":false,"hidden":false,"mode":"ojs"},{"id":898,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":464,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs"},{"id":437,"value":"md`*Utility functions below:*`","pinned":false,"hidden":false,"mode":"ojs"},{"id":127,"value":"// this function skeleton points besed on part and position (x, y)\nfunction drawPoints(context, point, showLabels, showFacePoints) {\n if (showFacePoints) {\n context.beginPath();\n context.arc(point.position.x, point.position.y, 2, 0, 2 * Math.PI);\n context.stroke();\n context.closePath();\n if (showLabels) {\n context.fillText(point.part, point.position.x, point.position.y);\n }\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":132,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"// this function draw the line between all the bodyParts\nfunction drawLines(context, pointA, pointB) {\n if (pointA && pointB) {\n context.beginPath();\n context.moveTo(pointA.x, pointA.y);\n context.lineTo(pointB.x, pointB.y);\n context.stroke();\n context.closePath();\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":320,"value":"// this function draw 17 keypoints\nfunction drawPoint(ctx, y, x, r, color) {\n ctx.beginPath();\n ctx.arc(x, y, r, 0, 2 * Math.PI);\n ctx.fillStyle = figColor;\n ctx.fill();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":43,"value":"// this function listen for incoming socket\nfunction onDataReceived(data) {\n calculateData(data);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":40,"value":"// helper method that save position points into tuple\nfunction toTuple({ y, x }) {\n return [y, x];\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":516,"value":"data1 = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":518,"value":"data2 = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":522,"value":"mutable xyzArr = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"mutable count = 0","pinned":false,"hidden":false,"mode":"ojs"},{"id":343,"value":"md`---\n## Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":490,"value":"model = poseDetection.SupportedModels.MoveNet","pinned":false,"hidden":false,"mode":"ojs"},{"id":47,"value":"detector = await poseDetection.createDetector(model)","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"poseDetection = requireTF(\"@tensorflow-models/pose-detection@0.0.3/dist/pose-detection.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":53,"value":"import {\n tf,\n requireTF,\n facePoints,\n sparkbar\n} from \"@mt-cs/movenet-posenet-image-pose-tracking\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":215,"value":"import { serialize } from \"@palewire/saving-csv\";","pinned":false,"hidden":false,"mode":"ojs"},{"id":387,"value":"import { file, slider, text, select, color } from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":341,"value":"md`---\n\n## Credits\n\nDeveloped for the [Kineviz](https://kineviz.com/) and [University of San Francisco](https://www.usfca.edu/arts-sciences/graduate-programs/computer-science?gclid=Cj0KCQjwu7OIBhCsARIsALxCUaOnomqGePhLPOYnIdNZNowVP2aSLsepdMYOe9W5Gb3InnI8b5lDKxAaAk03EALw_wcB) Summer Project by:\n* [Marisa Tania](https://www.linkedin.com/in/marisatania/)\n* [Barbora Novakova](https://www.linkedin.com/in/barbora-novakova-666029126/)\n\nSponsor: Weidong Yang\n\nCode credits:\n* PoseNet: [Loren Riesenfeld](https://observablehq.com/@lorenries/estimating-pose-in-the-browser-with-posenet-and-tensorflow-)\n* MoveNet: [Taras Novak](https://observablehq.com/@randomfractals/tensorflow-movenet-webcam)\n* Socket.io: [Sylvain Lesage](https://observablehq.com/@severo/hello-socket-io)\n---\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":737,"value":"md`## Acknowledgments\n\nWe sincerely thank:\n* [Weidong Yang](https://www.linkedin.com/in/yangweidong/) and the [Kineviz](https://www.kineviz.com/) Team for our biweekly meetings\n* [Nikko Sacramento](https://www.linkedin.com/in/nikkosac/) for plotting the 3D skeleton in GraphXR and running the data in VR\n* Sean Li for debugging the cors issue\n* [Thom Lauret](https://www.linkedin.com/in/thomlauret/) for the more robust AWS server\n* Professor [Mario Lim](https://www.linkedin.com/in/mario-lim-243556/) for helpful crossover network connection discussions\n* [Alex Law](https://www.linkedin.com/in/alexandria-law/) for posting the blogpost and voice over\n* [Cynthia Yu](https://www.shuhuancynthiayu.com/about) for being our model`","pinned":false,"hidden":false,"mode":"ojs"},{"id":730,"value":"md`## Contact\n\nThe notebook was implemented by [Barbora Novakova](mailto:barunka838@gmail.com) and [Marisa Tania](mailto:mt.marisatania@gmail.com). Feel free to reach out!`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}