2c:null 2d:["$","title",null,{"children":"Advanced pretrained models (object detection) | Jeya Balaji Balasubramanian | Observable"}] 32:T53d,## Object detection In [computer vision](https://en.wikipedia.org/wiki/Computer_vision), [*object detection*](https://en.wikipedia.org/wiki/Object_detection) is the task of locating objects of interest in an image or a video. Object detection models are often part of a larger workflow that involve other downstream data transformations or models. For example, Google Maps uses an object detection model to locate people's faces in a street-view image and follow up with another module that blurs the detected faces in order to anonymize them. Object detection models return the location of the detected objects using *bounding boxes* and assign a *confidence score* to each box indicating the model's confidence that the box indeed contains the detected object. The bounding boxes are specified by providing the top-left and the bottom-right coordinates of a rectangle in the [canvas coordinate space](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Drawing_shapes#the_grid) that overlays the image. Outputs from object detection models may require some post-processing because they often return a lot of sub-standard detections. [*Non-maximum suppression* (NMS)](https://www.youtube.com/watch?v=VAo84c1hQX8) is one such post-processing method, where boxes below a certain confidence score threshold are filtered out.33:Te3a,### Neural architectures for object detection Neural network-based models excel at the task of object detection but require complex architectures to solve them. Consider a simple extension to a neural network classifier to predict both the object category and the bounding box coordinates of the object that it predicts. Such a neural network would be both a classifier (object class) and a regressor (object location). This approach only works when there is a single instance of the object (e.g. a single bird in an image). If there are multiple instances of the same object (e.g. a flock of birds) or if there are multiple objects of interest (e.g. detect both birds and people in an image), our simplistic neural network runs into problems. ML researchers have come up with smart architectures to solve this problem. One such solution is [**R-CNNs** (not RNNs, which is another class of neural architectures) i.e. convolutional neural networks with region proposals](https://arxiv.org/abs/1311.2524). R-CNNs first uses an algorithm to generate (~2000) region proposals i.e., regions in the image where an object is likely located. Next, they engineer visual features from the input observations using a convolutional neural network. Finally, they classify the object within the proposed region using a simple model such as a support vector machines. The advantage of R-CNNs is— 1) multiple objects of different scales can be detected, and 2) it does not require an unproductive sliding window to traverse the whole image. The disadvantage of R-CNNs is— 1) it is slow because it runs the classifier for each proposed region, and 2) it requires three (region proposal, feature extraction, and classification) models to be independently trained, which can lead to sub-optimal models. Some later architectures, such as [Faster R-CNN](https://arxiv.org/abs/1506.01497) overcame this shortcoming by enabling end-to-end training of all the three modules simultaneously. Alternative solutions to R-CNNs for this problem include [**YOLO**](https://arxiv.org/abs/1506.02640) (You Only Look Once) and [**SSD**](https://arxiv.org/abs/1512.02325) (Single Shot Multi-box Detector). Both these architectures are extremely fast because, as their names imply, they classify all objects present in the image in one single pass. SSDs have three modules— 1) *feature extraction*, a stack of convolutional neural networks that generates engineered visual features to classify objects at different scales (because the objects can exist at different scales), 2) *detection heads*, also a convolutional neural network that looks at multiple default bounding box locations, at different scales, to estimate the confidence for whether each predicted box contains an object or just contains a background image, and 3) *non-maximum suppression* (NMS), to remove redundant detections of the same object. We will take a close look at NMS in this notebook. [Here is a YouTube video that does a great job of describing the SSD architecture](https://www.youtube.com/watch?v=F-irLP2k3Dk). [The linked blog here also does a great job of explaining this architecture](https://machinethink.net/blog/object-detection/). We will try to employ one such SSD architecture in this notebook called SSD-MobileNetV2. It uses MobileNetV2 (similar to the one we saw in [the previous notebook](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js)) to engineer visual features for object classification. Its detection head has about 1917 default boxes to encompass objects. It uses NMS to remove redundant boxes that detect the same object.34:T560,cocoClasses = ["person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant", "-", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "-", "backpack", "umbrella", "-", "-", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "-", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", "-", "dining table", "-", "-", "toilet", "-", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "-", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"];35:T457,{ // div container with 2 children— img and a canvas placed over it const div = htl.html`

`; // set canvas at absolute position and set left to 0 to overlay canvas over img, else canvas is placed next to img by default const canvas = htl.html`