36:["$","title",null,{"children":"Object detection and sonification | Suraj Mane | Observable"}] 3e:T738,

1. Object Detection

The YOLOv3 is a convolutional neural network that interprets the image pixel by pixel, learns the features of an image such as the edges and corners in the initial layers and the association between the feature and the class label. Below image showcases the working principle of object detection using YOLO. ![image.png](${await FileAttachment("image (1).png").url()}) [source](https://stats.stackexchange.com/questions/339754/what-is-the-need-of-multiple-bounding-boxes-per-grid-cell-in-yolo-v1) - Why specifically YOLO? 1. Computation time is optimal(few milliseconds to 1 second) which serves the purpose for the towards achieving end goal of sonification in real time. 2. YOLO works even when objects are overlapping by using anchor boxes which allows one grid cell to detect multiple objects of different classes. 3. Techniques such as Non-Maximal Suppression and Intersection Over detects the unique perfect match out multiple bounding boxes. - Which Dataset is the most suitable for our task? 1. COCO dataset is the most suitable in our task as it represents a handful of objects we encounter in our day to day life and hence is used for training the object detection model. 2. The COCO dataset contains 328K images and approximately 81 object categories. Typically 83K images are used for conducting training and 41K for validation.

2. Sonification

The results are sonified through a Speech Utterance API which takes input text and reads it out at the specified pitch, voice, volume and rate in the chosen language. 3f:T500,

Downstream Validations:

Considering our end-user and keeping in mind the experience we need to deliver we started with various design ideas and iterated over to reach the final conclusion as below: 1. Simple object detection in a virtual environment or from the image and presenting it would not help the visually impaired audience. The perceptual and cognitive abilities would make no sense for them. 2. We improved our design guidelines and thought to present the information in a textual format but again it would be a visualization problem for our user. 3. The final idea where we thought the design would make real sense for the user is sonifying the text so that the user can use the auditory cognitive abilities to comprehend the image or the virtual scene. 4. We doubled down on this idea and came up with certain mathematical rules to deliver the information which makes more sense for the user. 5. We also have given four different ways in which the end user can percept the information to their liking, i.e., by order, by prominence, by count and by uniqueness. Below is the link to the source code where object detection is performed: https://github.com/shreyas21295/ObjectDetectionAndSonification40:T519,

Contributions

- Sheeban: Implementation of YOLO in Python to get the image with bounding boxes and confidence scores for each of the objects present in the image with the data in a csv file. - Suraj: Worked on creating logic for functions/buttons and implementing them along with implementation of text to speech in the observable using the CSV generated. Styling the notebook and content creation. - Shreyas: Worked on proof of concept for perception of information to get objects by uniqueness, count, prominence and order in python. Worked on documenting how the different soring and filtering functions work. Tried integration of python in d3 using the pyodide framework to get dynamic data. (https://pyodide.org/en/stable/). - Shruti: Worked on analysis of YOLO object detection model from various perspectives. Worked on incorporating bounding boxes in observable using anchor points and other available parameters. Worked on narration part of observable and presentation along with other teammates. - Madhuvanti: Worked on real-time inference of an object detection model in python. Contributed to the ideation and conceptualization of the end-to-end pipeline. Documented the evolution of the project in the final deliverables. 41:[{"id":"10b134d4e16ef084cde608043fd8441edee55c8e7856ae38df92833f51c2222a64df39e6bd3b5e27ee4196cf2df000c85200fdc8c7fdc02f3312667573fc9a3b","href":"https://static.observableusercontent.com/files/10b134d4e16ef084cde608043fd8441edee55c8e7856ae38df92833f51c2222a64df39e6bd3b5e27ee4196cf2df000c85200fdc8c7fdc02f3312667573fc9a3b","name":"dog.csv","type":"text/csv","lastModified":1639193705348.021,"size":185},{"id":"9ff0336030586d5c39c9234d576b1256a05ea094000da47eab6cb3e18537bb499b537df24ca929547ad0f809d188bacc1920d1fd00580b47e25d36a18d42d1de","href":"https://static.observableusercontent.com/files/9ff0336030586d5c39c9234d576b1256a05ea094000da47eab6cb3e18537bb499b537df24ca929547ad0f809d188bacc1920d1fd00580b47e25d36a18d42d1de","name":"dog.png","type":"image/png","lastModified":1639193705348.021,"size":1135589},{"id":"4fcd2f00d02d425eb9d460faacb0f513ed74a2bf9797608faeb2d76df76f6cd7ef7f7fddde69c772cab7ce333ce9c2bd28907820505d4c98f18d6428e3cf8b3a","href":"https://static.observableusercontent.com/files/4fcd2f00d02d425eb9d460faacb0f513ed74a2bf9797608faeb2d76df76f6cd7ef7f7fddde69c772cab7ce333ce9c2bd28907820505d4c98f18d6428e3cf8b3a","name":"dog_notags.png","type":"image/png","lastModified":1639193705348.021,"size":1139013},{"id":"ee39cab7551777de035d12506cf39fc4c256018d234a73f8fd816f58f93ec8586f206d946abd84212e4c2807402e8065aaf21d9a6f9268d8d16a1a1fbf8ad97a","href":"https://static.observableusercontent.com/files/ee39cab7551777de035d12506cf39fc4c256018d234a73f8fd816f58f93ec8586f206d946abd84212e4c2807402e8065aaf21d9a6f9268d8d16a1a1fbf8ad97a","name":"person.png","type":"image/png","lastModified":1639193705348.021,"size":1180095},{"id":"1da2eced26f628173c03e21da8815a172de30d106541ffc31b78ca1cb6f80c75617a11992be46cb2418c810ce312ac2959c9a5e9a713b4baebc9a9a3e2218a87","href":"https://static.observableusercontent.com/files/1da2eced26f628173c03e21da8815a172de30d106541ffc31b78ca1cb6f80c75617a11992be46cb2418c810ce312ac2959c9a5e9a713b4baebc9a9a3e2218a87","name":"person_notags.png","type":"image/png","lastModified":1639193705348.021,"size":1177614},{"id":"8274009e092bc26a11c39cfed92f02c5090b28d8d416567ca063ec779ecb4bdb39d1d68f44acb7c4dd8f837b3cb847636da7e29baa5641dc0e75625b50ec1d7d","href":"https://static.observableusercontent.com/files/8274009e092bc26a11c39cfed92f02c5090b28d8d416567ca063ec779ecb4bdb39d1d68f44acb7c4dd8f837b3cb847636da7e29baa5641dc0e75625b50ec1d7d","name":"image (1).png","type":"image/png","lastModified":1639193705348.021,"size":222907},{"id":"4d2df4cfeec7aef1771eb396483f6fdb224057ff4e53c999b0d25acc4281d4ee7ebf4b0444d8cd2fb70bf6e5438359d801d1e4f371d7a1fe817ab5e7454b43b7","href":"https://static.observableusercontent.com/files/4d2df4cfeec7aef1771eb396483f6fdb224057ff4e53c999b0d25acc4281d4ee7ebf4b0444d8cd2fb70bf6e5438359d801d1e4f371d7a1fe817ab5e7454b43b7","name":"busstand_boxed.png","type":"image/png","lastModified":1639354529430.024,"size":1267547},{"id":"8ceba20cf851413e6928cd86a003237dd1168bd890d48badec6a69b0f4972ee36d4209d12a3e209d1dd190b7af709c250fa5061986b18944b0542c6176aa1c24","href":"https://static.observableusercontent.com/files/8ceba20cf851413e6928cd86a003237dd1168bd890d48badec6a69b0f4972ee36d4209d12a3e209d1dd190b7af709c250fa5061986b18944b0542c6176aa1c24","name":"busstand_nobound.png","type":"image/png","lastModified":1639354529714.902,"size":1265630},{"id":"b92d17877f2aba626c25066a872d6ce3d85ac10986a2b5705a0bad08eafe78d3efc591902fd4aceee2cc2db8ee31181965f047b2da4082af603f3f3c05a0adb9","href":"https://static.observableusercontent.com/files/b92d17877f2aba626c25066a872d6ce3d85ac10986a2b5705a0bad08eafe78d3efc591902fd4aceee2cc2db8ee31181965f047b2da4082af603f3f3c05a0adb9","name":"file.csv","type":"text/csv","lastModified":1639354342397.317,"size":981}] 3a:{"id":"bbdb83a35bac6f6a","slug":"object-detection-and-sonification/2","title":" Object detection and sonification","thumbnail":"71dd0de03cf1f37e529594fcebf549debd01de8ed028562407a1d147ab5c215d","access_level":"public","create_time":"$D2021-12-11T03:35:05.348Z","update_time":"$D2021-12-13T20:52:12.660Z","view_time":null,"first_public_time":"$D2021-12-11T03:35:45.487Z","has_unpublished_edits":null,"fork":true,"stars":1,"starred":false,"owner":{"type":"team","id":"485d69bcdabb092d","login":"surajmane","name":"Suraj Mane","avatar_url":"https://avatars-next.observableusercontent.com/8f31e17c86876f9ca88dbd354021f4821e4e26479be8a78343fb46900b2f8cdc","tier":"starter_2024","bio":"","followed_by_you":false,"follows_you":false},"version":1248,"publish_version":1248,"role":null,"authored":false,"authors":[{"type":"individual","id":"66d53226e49cf368","login":"surajmane","name":"Suraj Mane","avatar_url":"https://avatars-next.observableusercontent.com/8f31e17c86876f9ca88dbd354021f4821e4e26479be8a78343fb46900b2f8cdc","tier":null,"bio":"","followed_by_you":false,"follows_you":false}],"members":[{"type":"individual","id":"66d53226e49cf368","login":"surajmane","name":"Suraj Mane","avatar_url":"https://avatars-next.observableusercontent.com/8f31e17c86876f9ca88dbd354021f4821e4e26479be8a78343fb46900b2f8cdc","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":4,"collection":null,"type":"notebook","body":{"titling":"auto","title":" Object detection and sonification","cells":[{"id":0,"value":"

Object detection and sonification

How do we detect objects in an image?

Problem Statement

Existing Visualization

Design Issues

1. The order in which they are found

2. The most prominent objects in the image

3. A count of all objects in the image

4. Just the unique objects detected

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":560,"value":"The user can choose the orientation they desire by clicking on the relevant buttons in this notebook.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":627,"value":"Through the selection of objects we eliminate visual and audio clutter and retrieve the results the user finds relevant.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":566,"value":"How does this work under the hood?","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":572,"value":"$3e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":988,"value":"We've shown below intermediate outputs of the object detection model. One is with just the bounding boxes and the other one is with both the bounding boxes and confidence score.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":741,"value":"image1 = FileAttachment(\"busstand_nobound.png\").image()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":961,"value":"image1.width = 550","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":749,"value":"image2 = FileAttachment(\"busstand_boxed.png\").image()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":963,"value":"image2.width = 550","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1240,"value":"

Click on each button to hear the results of the above image

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":6,"value":"viewof byOrder = Inputs.button(html``);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":603,"value":"Reads out objects by the order of occurrence in the image determined by the anchor points of the bounding box.\nBy order means here the objects would be read out from as they occur in the image from left to right. The anchor points are obtained from the attached csv file.\n\nSo we sort the objects according to the anchor points in descending order and give it to the speech synthesizer to read. (by_order)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":151,"value":"by_order = {\n byOrder;\n let value = [...data].slice().sort((a, b) => d3.ascending(a.AnchorPoint, b.AnchorPoint))\n let value2 = value.map(a => a.Object)\n return value2\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":224,"value":"{\n//var size = by_prominence.length\nmsg_order.text = 'The order of the objects is '+by_order + 'in the image'\nwindow.speechSynthesis.speak(msg_order);\n}\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":96,"value":"viewof byProminance = Inputs.button(html``);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":609,"value":"Vocalizes the most prominent objects based on the area of the object in the entire image.\nSo here we are giving the user the option to sonify the objects by the size that they acquire in the image. Here the objects are sorted in descending order according to the area they cover in the image. The area for each object is defined in the obtained csv file. We've added a condition which would filter and return just the top 4 objects, although it's totally changeable and will depend on the image.(by_prominence)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":187,"value":"by_prominence = {\n byProminance;\n let value = [...data].slice().sort((a, b) => d3.descending(a.Area, b.Area))\n let value2 = d3.map(value, d => d.Object)\n return value2\n\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":245,"value":"{\nvar message = by_prominence.slice(0,4)\n//var size = by_prominence.length\nmsg_prominence.text = 'The prominent objects in the image are '+message\nwindow.speechSynthesis.speak(msg_prominence);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":100,"value":"viewof ByUniqueness = Inputs.button(html``);","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":621,"value":"Lists the distinct objects present in the image. To obtain the distinct objects in the image, we are using the results obtained from the order function and putting them in a SET to just get the distinct objects.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":176,"value":"by_uniqueness = {\n ByUniqueness;\n let values = d3.map([...data], x => x.Object);\n values = [...new Set(values)]\n msg_uniqueness.text = 'There is a '+values+' in the image.';\n window.speechSynthesis.speak(msg_uniqueness);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":98,"value":"viewof byCount = Inputs.button(html``)","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":615,"value":"Counts the occurrence of an object in the scene. To obtain the count we take all the objects and count them using d3.count after using d3.rollup to get the count of the objects by the index. (by_count)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":145,"value":"by_count = {\n byCount;\n let value = d3.rollup(data, v=> d3.count(v, d => d.Index), d=>d.Object)\n return value\n \n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":679,"value":"{\nvar speech = []\n var arr2 = d3.map([...data], x => x.Object)\n arr2 = [...new Set(arr2)]\n for(var i=0; i < arr2.length; i++){\n var temp = by_count.get(arr2[i])\n var count = ''\n if (temp > 1){\n count = \"There are \"\n speech = speech+temp + \" \" + arr2[i] + \" \"\n \n }\n else{\n count = \"There is \"\n speech = speech+temp + \" \" + arr2[i] + \" \"\n }\n }\n msg_count.text = count+speech\n window.speechSynthesis.speak(msg_count)\n\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":828,"value":"$3f","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":845,"value":"

Future work

\n- Currently this is a two step process wherein first the user needs to run the inference on the image through the IPYNB file. Once that's done a CSV is generated which is attached to this notebook and then the remaining operations are performed.\n- We'd like to simplify this by integrating the backend inference task with this notebook either by using NodeJS or if there's any extension of observable that allows this. Once that's done, all that a user will need to do is give a reference of the image in the notebook and click those buttons to hear the results.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":863,"value":"$40","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"","database":"$undefined"},{"id":195,"value":"d3 = require(\"d3@6\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":216,"value":"msg = new SpeechSynthesisUtterance();\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":235,"value":"msg_prominence = new SpeechSynthesisUtterance();","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":232,"value":"msg_order = new SpeechSynthesisUtterance();","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":241,"value":"msg_count = new SpeechSynthesisUtterance();","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":268,"value":"d3Fetch = require('d3-fetch')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":243,"value":"msg_uniqueness = new SpeechSynthesisUtterance();","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":1015,"value":"data = FileAttachment(\"file.csv\").csv()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W41","stdlib":"1"}} 35:null