2b:null 2c:["$","title",null,{"children":"Pretrained models (image classification) | Jeya Balaji Balasubramanian | Observable"}] 31:T6b8,# Pretrained models (image classification) By this point in the [TF.js series](https://observablehq.com/collection/@jeyabbalas/tensorflow-js), you have learned to encode data into *Tensors* and to transform these tensors into other tensors using "*Flow*" operations. A statistical model (e.g. a deep neural network) is nothing but a sequence of operations that transforms some input tensor (e.g. an image) into some meaningful output tensor (e.g. object contained in the image). So, a statistical model can be thought of as one big tensor operator. Even before we learn to train our own models from data, with our newly developed tensor skills, we can already utilize models trained by others on their datasets. These are called *pretrained models*. In this notebook, we will learn about one such resource to find pretrained TF.js models, called [TensorFlow Hub](https://tfhub.dev/). TensorFlow Hub is an example of a *model zoo*. A model zoo is an online resource that maintains a curated collection of pretrained models along with their associated metadata. The metadata contains information about the model and directions on how to utilize it. To my knowledge, TensorFlow Hub is the only model zoo that contains TF.js models. Other zoos contain models developed in other frameworks such as PyTorch, ONNX, and TensorFlow's Python API. We will visit those model zoos in a future notebook. Models in the wild differ in many ways including— the task they were built for, the domain data they were trained on, the framework used to develop them, and the model format. At the minimum, we must recognize all of these in order to correctly employ these models. Let us start by building some vocabulary to talk about them.32:T67d,## Table of Contents - [What are models?](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-609) - [Artificial intelligence (AI)](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-609) - [Machine learning (ML)](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-642) - [ML tasks](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-54) - [Problem domains (modality)](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-604) - [Model format](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-606) - [Model zoos](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-52) - [Model metadata](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-52) - [Hosting TF.js models](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-52) - [Image classification (MobileNet V3)](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-322) - [MobileNets](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-322) - [Practicum](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-510) - [Conclusion](https://observablehq.com/@jeyabbalas/pretrained-models?collection=@jeyabbalas/tensorflow-js#cell-56)33:Tbb7,## What are models? ### Artificial intelligence (AI) [*Artificial intelligence*](https://en.wikipedia.org/wiki/Artificial_intelligence) (AI) is the study of intelligence. You cannot claim to understand intelligence unless you can recreate it from the scratch. So, being able to build *intelligent systems* is at the heart of AI. An *intelligent system* is a machine capable of exhibiting intelligence in at least some task. AI is a widely interdisciplinary field that includes expertise from many scientific disciplines. Some of them are listed below along with the main questions they seek to answer with regards to the development of AI. 1. Philosophy: What is intelligence? What is knowledge? How do we gain new knowledge? What is true and what is false? How do thoughts arise from the physical brain? What is consciousness? What are the consequences of an action? 2. Cognitive science: How to intelligent agents think? 3. Robotics: How do intelligent agents interact with the world? 4. Computer science and engineering: How do we build an intelligent agent? 5. Mathematics, statistics: What is the calculus of learning new knowledge? How do we learn knowledge from observations? What can be estimated? How do we reason under uncertainty? How do we build a **statistical model** that learns from observations? 6. Linguistics: How does language relate to intelligence? 7. Computer vision: How do we visually perceive the world around us? 8. Neuroscience: How do brains work? 9. Evolutionary biology: How does evolution influence intelligence? 10. Economics: How do we make decisions to maximize payoffs? How do we incentivize collaboration? There is no consensus on what it means to be intelligent. A famous yet simple test comes from Alan Turing (1950) who called it [*the Imitation Game*](https://academic.oup.com/mind/article/LIX/236/433/986238?login=false). It is more commonly known as the *Turing test*. In this test, a human interrogator converses with a human and a machine, if the interrogator cannot determine which responses come from the human and which come from the machine, then the machine passes the Turing test. The machine can then be considered intelligent. In light of recent events, where [a Google AI, called LaMDA, was mistaken for a sentient human by one of their engineers](https://www.washingtonpost.com/technology/2022/06/17/google-ai-lamda-turing-test/), the Turing test has come under fresh scrutiny. AI systems can beat the test by simply trying to "imitate" humans without exhibiting any human-level *thinking*, [a concept Alan Turing considered irrelevant to the discussion of intelligence](https://techpolicy.press/artifice-and-intelligence/). [Google Duplex is another example of a conversational AI system that would beat the Turing test](https://www.youtube.com/watch?v=JvbHu_bVa_g). However, whether or not these systems actually exhibit thinking, remains an active topic of debate.34:T765,### Machine learning (ML) [*Machine learning*](https://en.wikipedia.org/wiki/Machine_learning) (ML) is a sub-discipline of AI concerned with learning from observations. The models discussed in [this TF.js series](https://observablehq.com/collection/@jeyabbalas/tensorflow-js) will constrain to only ML problems. [Dr. Pedro Domingos, in his book 'The Master Algorithm', suggested that there are at least five major schools of thought, i.e. "tribes", in ML](https://learning.acm.org/techtalks/machinelearning). Each tribe differ in how they perceive that new knowledge can be learned from observations. The suggested five tribes were— 1. Symbolists: new knowledge can be learned using [inverse deduction](https://en.wikipedia.org/wiki/Deductive_reasoning). 2. Connectionists: new knowledge is learned using [backpropagation](https://en.wikipedia.org/wiki/Backpropagation). 3. Evolutionaries: new knowledge is learned using [genetic algorithms](https://en.wikipedia.org/wiki/Genetic_algorithm). 4. Bayesians: new knowledge is learned using probabilistic inference. E.g. [Bayes theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem). 5. Analogizers: new knowledge is learned by seeking other similar examples. E.g. [support vector machine](https://en.wikipedia.org/wiki/Support_vector_machine). [Knowledge representation](https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning), i.e. how to encode observations before passing them into the above listed algorithms, and how to encode the knowledge inferred by them, is an open research topic of huge significance. This TF.js series will only concern with connectionist models. Specifically, we will work with [neural network models](https://en.wikipedia.org/wiki/Neural_network). Neural networks differ in the tasks that they specialize in. We will take a look at them in the next section.35:T79c,### ML tasks Each model specializes in one or more tasks. Some major learning tasks include— 1. [Supervised learning](https://en.wikipedia.org/wiki/Supervised_learning): each observation has associated labels. The task is to learn to predict the labels of an unseen observation. This task has two major subtypes— * [Classification](https://en.wikipedia.org/wiki/Statistical_classification): the labels to predict are categorical. E.g. is the observed image that of a cat or a dog? * [Regression](https://en.wikipedia.org/wiki/Regression_analysis): the labels to predict are continuous. E.g. what is the age of the person in the image? 2. [Unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning): the observations have no associated labels. Tasks here vary considerably. Some important tasks include— * [Generative modeling](https://en.wikipedia.org/wiki/Generative_model): learn to generate observations similar to the ones previously seen. E.g. [generate images from a text prompt](https://huggingface.co/spaces/dalle-mini/dalle-mini). * [Dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction): learn to reduce the number of features of the input. E.g. [visualize your data in 3D for exploratory data analysis](https://projector.tensorflow.org/). * [Clustering](https://en.wikipedia.org/wiki/Cluster_analysis): learn to group similar observations. E.g. group similar images together. 3. [Reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning): the goal here is to interact with the environment, perform some actions, and iteratively learn from the mistakes in order to maximize a reward function. E.g. [teaching a machine to read maps](https://arxiv.org/abs/1711.07479). There are many more types of ML tasks. [Click here for an extensive list](https://en.wikipedia.org/wiki/Machine_learning#Approaches).36:T573,### Problem domains (modality) The *problem domain* or [*modality*](https://en.wikipedia.org/wiki/Modality_%28human%E2%80%93computer_interaction%29) refers to the data type of the observations with respect to the type of sensory input information that it encodes. The typical modalities include— 1. Images: e.g. images of common objects, pathology images, etc. 2. Text: e.g. wikipedia articles, clinical text, etc. 3. Video: e.g. movies, echocardiogram, etc. 4. Audio: e.g. human speech, heart murmurs, etc. 5. Tabular: e.g. census data, electronic health records, etc. Usually, ML models tend to be trained on observations from one modality (*unimodal*). However, recently powerful models have emerged by training models on multiple modalities. Such a learning task is called [multimodal learning](https://en.wikipedia.org/wiki/Multimodal_learning). A recently popular example of model trained with multimodal learning is [DALL-E](https://huggingface.co/spaces/dalle-mini/dalle-mini), which generates images based on text prompts. This model was trained on both text and image data. [Foundation models](https://arxiv.org/abs/2108.07258) are models trained on multimodal data and are trained to perform multiple ML tasks. An example of a foundation model is [PaLM by Google](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html).37:Tae7,### Model format The model format is determined by the framework used to develop the model. This can be [PyTorch](https://pytorch.org/), [ONNX](https://onnx.ai/), [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html), [TensorFlow](https://www.tensorflow.org/), or [JAX](https://jax.readthedocs.io/en/latest/index.html), among others. As we have come to realize, TensorFlow models themselves come in different formats. They include— 1. [TensorFlow](https://www.tensorflow.org/): built for server-side compute using the Python API. 2. [TensorFlow.js](https://www.tensorflow.org/js): built for client-side (browser API) or server-side (Node.js API) compute. 3. [TensorFlow Lite](https://www.tensorflow.org/lite): built for Internet of Things (IoT) devices. 4. [Coral](https://coral.ai/): also built for IoT devices. TensorFlow Hub contains TensorFlow models for all these architectures. The server-side distributions of TensorFlow models are typically in the [SavedModel format](https://www.tensorflow.org/guide/saved_model). The older [Keras's HDF5 format](https://www.tensorflow.org/tutorials/keras/save_and_load#hdf5_format) is also still supported. IoT device distributions (TFLite and Coral) are in the [TFLite format](https://www.tensorflow.org/lite/guide#2_convert_the_model). **TF.js models** come in two formats. They are— 1. Layers model: relatively more transparent and modifiable but computationally inefficient. 2. Graph model: relatively less transparent, unmodifiable but computationally efficient. If you are a Python programmer, Layers model is meant to be a drop-in replacement for the Keras API. Once you have determined the type of TF.js model that you are about to consume, you can use the appropriate load function from TF.js. If the model is a Layers model, you can load it using [`tf.loadLayersModel(url)`](https://js.tensorflow.org/api/latest/#loadLayersModel). If the model is a Graph model, you can load it using [`tf.loadGraphModel(url)`](https://js.tensorflow.org/api/latest/#loadGraphModel). The model metadata usually contains this information. If you are unsure, you can try them both and see which works. If you are ingesting the TF.js model from TF Hub, you need to pass an additional parameter `{fromTFHub: true}` into the load function. This tells the loader that the model is being read from TF Hub. I understand that the mentioned differences between the TF.js model formats are vague. When we build neural networks from the scratch, the differences will become clear. For now, it is enough to know that two formats of TF.js models exist and that they are ingested differently in our code. Also, to my knowledge, all TF.js models in TF Hub are Graph models.38:Taad,## Model zoos Model zoos are an online resource that maintain a curated list of pretrained models. I think, ideally, model zoos should aspire to become *data commons*. [*Data commons* "co-locates data with cloud computing infrastructure and commonly used software services, tools & applications for managing, analyzing and sharing data to create an interoperable resource for a research community"](https://medium.com/@rgrossman1/a-proposed-end-to-end-principle-for-data-commons-5872f2fa8a47). For a specific research domain, all relevant models, the data they were trained on, the preprocessing pipeline required to prepare the data for the model, and the post-processing pipeline to interpret model output, should all be co-located with a cloud computing infrastructure. This would rapidly accelerate the development of powerful, reproducible, and reusable ML models in the research domain. Examples of model zoos include— 1. [Hugging Face](https://huggingface.co/models) 2. [ModelZoo](https://modelzoo.co/) 3. [ONNX Model Zoo](https://github.com/onnx/models) 4. [Papers with code](https://paperswithcode.com/sota): under my definition, this would qualify as a model zoo. 5. [TensorFlow Hub](https://tfhub.dev/). Under the definition of model zoos, there are two places where current model zoos lack— 1) API (OAuth'ed if necessary) access to data (and metadata) which was used to train the model, and 2) a comprehensive and standardized metadata. ### Model metadata When considering standardizing the metadata for model zoos, I believe the following must be included— 1. Author/publisher with contact information if possible. 2. Paper/blog/wiki describing the purpose of the model. Comparison to state-of-the-art. 3. Expected data input format. 4. Interpretation of the model output. 5. Model format. Environment type. 6. Example code snippet showing how to consume the model. Should include data preprocessing pipeline. 7. Usage statistics. ### Hosting TF.js models TF.js models are typically hosted using a public URL pointing to a model JSON file. This file contains the network architecture of the model. On the same URL path the weights (BIN) files must be co-located and also be publicly accessible. The weights file is often fragmented into shards to allow asynchronous consumption of the model on the client-side. See below, [an example of a TF.js model hosted in a public URL in Google Cloud Storage](https://console.cloud.google.com/storage/browser/tfjs-models/tfjs/iris_v1). | ![Intersection over Union](${await FileAttachment("tfjs_public_bucket.png").url()}) | |-| [Here is a list of some other models hosted in this bucket](https://console.cloud.google.com/storage/browser/tfjs-models/tfjs).39:Tb4a,## Image classification (MobileNet V3) In [computer vision](https://en.wikipedia.org/wiki/Computer_vision), image classification is the task of categorizing images by their main subject. We will try to classify images using a popular model called [MobileNet V3](https://arxiv.org/abs/1905.02244v5). It is an image classifier developed by Google researchers that runs efficiently on low memory devices like mobile phones. Yet, it manages to achieve classification accuracies comparable to much larger neural networks. The [original MobileNet paper](https://arxiv.org/abs/1704.04861v1) and [this blog](https://machinethink.net/blog/googles-mobile-net-architecture-on-iphone/) go into some details about how MobileNets achieve this feat. ### MobileNets I will quickly and rather poorly summarize MobileNets in this section. You can skip this section without hurting your understanding on how to utilize MobileNets. Image analysis is typically done by a class of neural networks called *convolutional neural networks*. We will look at them in more detail in a future notebook. The most important tensor operator in this neural network is a [*convolution*](https://en.wikipedia.org/wiki/Convolution). A convolution operator requires a large number of multiplications, which are computationally expensive. To improve the computational performance of convolutional neural networks, [MobileNetV1](https://arxiv.org/abs/1704.04861v1) proposed *depth-wise convolutions*, which is a trick to reduce the number of multiplications made by the convolutional operator. MobileNetV1 managed to achieve predictive performance comparable to traditional convolutional neural networks, while requiring much less space and compute power. [MobileNetV2](https://arxiv.org/abs/1801.04381) improved upon V1 by introducing new blocks (sequence of neural layers that are repeated) called *inverted residuals with linear bottleneck*. The idea here was to improve the chance of finding the [manifold of interest](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) by first increasing the number of channels of the features in the hidden layers, and then filtering the signals using depth-wise convolutional layers, and then finally constraining the number of channels in the output layer of the bottleneck block. The factor that constrains the number of channels output by the bottleneck block is called the *depth multiplier*, which is a term that will appear in the practicum section below. [MobileNetV3](https://arxiv.org/abs/1905.02244) improved upon V2 by adding squeeze-and-excitation modules to the bottleneck layer. The goal of this module is to improve the interdependencies between the channels at almost no compute cost. The papers get increasingly complicated. However, as I explained at the beginning of this section, understanding them is not necessary to be able to skillfully utilize them.3a:T962,### Practicum We start by trying to locate MobileNet V3 on TF Hub. First, go to [TF Hub](https://tfhub.dev/). There are thousands of models on TF Hub. On the left-hand-side panel of TF Hub, we have a few options to filter the models. Let's try to [filter the `Model format` to `TF.js` models](https://tfhub.dev/s?deployment-format=tfjs) because we plan to run it entirely on the client-side. We then [filter the `Problem domain` to `Image classification`](https://tfhub.dev/s?deployment-format=tfjs&module-type=image-classification) because that is the task of interest in this notebook. We can see that we have plenty of models that qualify these two filters. We additionally [filter the neural network architectures to only MobileNet V3](https://tfhub.dev/s?deployment-format=tfjs&module-type=image-classification&network-architecture=mobilenet-v3). We are still left with 4 models. The MobileNet V3 models seem to be of 2 major types— `large` and `small`. A quick look at [the MobileNet V3 paper](https://arxiv.org/abs/1905.02244v5) tells us that `large` models are slightly more accurate but more bulky (~15 or 20 MB). The `small` models are lighter (~7 or 9 MB) at the cost of losing some accuracy. We choose large models because we can easily accommodate them on the web browser. We are still left with 2 models— one with `depth multiplier` of `0.75` and another with `depth multiplier` of `1.0`. Again, the paper tells us that `depth multiplier` of `1.0` is relatively more accurate at the cost of the model being more bulky (~20 MB). Since we can accommodate this on the browser, we choose the depth multiplier of 1. The link to this model is here: https://tfhub.dev/google/imagenet/mobilenet_v3_large_100_224/classification/5. The TF.js model is located under the `JS` tab: https://tfhub.dev/google/tfjs-model/imagenet/mobilenet_v3_large_100_224/classification/5/default/1. We find all the metadata associated with the model on this page. Some models don't contain a `JS` tab. Those models cannot be directly ingested into JavaScript. We still have one trick up our sleeve— we can [convert Python models into TF.js](https://codelabs.developers.google.com/codelabs/tensorflowjs-convert-python-savedmodel#0). This will be the subject of a future notebook in this series. The URL to the model is provided in the orange button to the right. We copy and paste it into the variable below.3b:T4bc,## Conclusion TensorFlow models can be thought of as one large tensor operator. It converts an input tensor into some meaningful output tensor. We looked into TensorFlow Hub, a rich resource of TensorFlow models. We saw how to look for and utilize a TF.js model, for a specific task, from TF Hub. This model was a simple example. Some models in TF Hub are quite advanced and requires some background knowledge for us to be able to correctly utilize them. This is the subject of [the next notebook](https://observablehq.com/@jeyabbalas/object-detection?collection=@jeyabbalas/tensorflow-js), where we will try to utilize an object detection model from TF Hub. **Note:** In this notebook, I ignored [my earlier advice](https://observablehq.com/@jeyabbalas/tensors?collection=@jeyabbalas/tensorflow-js) of cleaning up the tensors after we are done using them. I did this for didactic reasons, so that we can investigate each step of the process. This is bad production design because, for each image predicted, we accumulate orphan tensors. In an actual application, we must load the model, preprocess data, and make prediction inside `tf.tidy()` or `dispose()` the tensors we create after we are done using them.3c:[{"id":"fcd5bd8043494f7dac50ef6a679cb833c95fc2086e6281c8c3795546fc07d46235923b379c765122ca1e89348078a5a930e3e4f949eb32e787932e473b5239c4","href":"https://static.observableusercontent.com/files/fcd5bd8043494f7dac50ef6a679cb833c95fc2086e6281c8c3795546fc07d46235923b379c765122ca1e89348078a5a930e3e4f949eb32e787932e473b5239c4","name":"Ramphodon_naevius_with_flower.jpeg","type":"image/jpeg","lastModified":1666151474865.682,"size":298870},{"id":"fe3fb74d0560e8307eab17dbb5786d6daa3a0cd164dafb580892355dec15304d5e420b7175857f93b07daf5a76c448eb04b0d18c7f0c632dcbaf744364c6b55f","href":"https://static.observableusercontent.com/files/fe3fb74d0560e8307eab17dbb5786d6daa3a0cd164dafb580892355dec15304d5e420b7175857f93b07daf5a76c448eb04b0d18c7f0c632dcbaf744364c6b55f","name":"tfjs_public_bucket.png","type":"image/png","lastModified":1667359266152.879,"size":264937}] 2f:{"id":"b8b443d8e7e12304","slug":"pretrained-models","title":"Pretrained models (image classification)","thumbnail":"cab95bef9bb2640186cbd292c62e7b07fba67670d50679c85d81fd8ac7bd0e54","access_level":"public","create_time":"$D2022-09-24T23:14:50.002Z","update_time":"$D2022-11-15T17:17:14.883Z","view_time":null,"first_public_time":"$D2022-10-26T12:19:19.013Z","fork":false,"stars":5,"starred":false,"owner":{"type":"team","id":"7c99fef7c866716c","login":"jeyabbalas","name":"Jeya Balaji Balasubramanian","avatar_url":"https://avatars-next.observableusercontent.com/80f51d00c7e14ae878fca50bb97e3d82c91a9c13d88c71a3bd41f878c83c135e","tier":"starter_2024"},"version":785,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"352a9430837886b4","login":"jeyabbalas","name":"Jeya Balaji Balasubramanian","avatar_url":"https://avatars-next.observableusercontent.com/80f51d00c7e14ae878fca50bb97e3d82c91a9c13d88c71a3bd41f878c83c135e","tier":null}],"members":[{"type":"individual","id":"352a9430837886b4","login":"jeyabbalas","name":"Jeya Balaji Balasubramanian","avatar_url":"https://avatars-next.observableusercontent.com/80f51d00c7e14ae878fca50bb97e3d82c91a9c13d88c71a3bd41f878c83c135e","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"subscribed":null,"forks":3,"collection":{"id":"a734a023a656c47c","slug":"tensorflow-js","title":"TensorFlow.js","owner":{"id":"7c99fef7c866716c","login":"jeyabbalas","name":"Jeya Balaji Balasubramanian"}},"type":"notebook","body":{"title":"Pretrained models (image classification)","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":782,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":609,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":642,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":54,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":604,"value":"$36","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":606,"value":"$37","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":52,"value":"$38","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":322,"value":"$39","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":510,"value":"$3a","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"mobileNetPath = \"https://tfhub.dev/google/tfjs-model/imagenet/mobilenet_v3_large_100_224/classification/5/default/1\"","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":524,"value":"The code snippet in model metadata tells us that this TF.js model is a Graph model. So, we load it using [`tf.loadGraphModel(url, {fromTFHub: true})`](https://js.tensorflow.org/api/latest/#loadGraphModel). We need to pass in the parameter `{fromTFHub: true}` to be able to ingest TF Hub models. Try removing this option below to see the kind of error message we would get otherwise.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":19,"value":"mobileNet = tf.loadGraphModel(mobileNetPath, { fromTFHub: true })","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":530,"value":"The Graph model object contains some useful information about the model. For example, we can take a look at the shape of the input tensor that it expects. `[-1, 224, 224, 3]` tells us that it takes in a batch of any number of images (`-1`), each of size ${tex`224\\times 224`}, and has `3` (RGB) channels.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":306,"value":"mobileNet.inputs[0].shape","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":533,"value":"The Graph model object also contains the weights matrices, as shown below. This contains a list of all the neural network layers and its parameter values in the model.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":309,"value":"mobileNet.weights","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":537,"value":"Let's inspect the first hidden layer more closely. It appears to be of shape `[3, 3, 3, 16]`. Table 1 in [the MobileNet V3 paper](https://arxiv.org/abs/1905.02244v5) confirms that the first layer in MobileNet V3-Large is a neural network layer called a \"bottleneck layer\" of shape `[3, 3, 3, 16]`. It is a convolutional layer with ${tex`3\\times 3`} receptive field, with 3 channels and 16 feature maps. Understanding convolutional layers in more detail will be the subject of a future notebook.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":324,"value":"mobileNet.weights[Object.keys(mobileNet.weights)[0]]","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":336,"value":"mobileNet.weights[Object.keys(mobileNet.weights)[0]][0].array()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":544,"value":"The model object contains a metadata variable that contains the indexed class names of the object classes that the model predicts. A quick look at the model metadata tells us that MobileNet V3 was trained on the ImageNet [ILSVRC 2012 dataset](https://www.image-net.org/challenges/LSVRC/2012/) with `1001` classes. The classes are listed below.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":32,"value":"mobileNetClassNames = mobileNet.metadata.classNames","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":546,"value":"We go back to [the TF Hub page of the model](https://tfhub.dev/google/tfjs-model/imagenet/mobilenet_v3_large_100_224/classification/5/default/1) and copy the code snippet for preprocessing one image tensor to prepare it for feeding it into MobileNet V3. The code shows that it crops the image and resizes it to the input shape that the model expects i.e. ${tex`224\\times 224`}. It also expands its dimensions so that it generates a batch of size 1. If we have multiple images to predict, we can stack them up and feed it into MobileNet V3.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":28,"value":"function preprocess(imageTensor) {\n const widthToHeight = imageTensor.shape[1] / imageTensor.shape[0];\n let squareCrop;\n if (widthToHeight > 1) {\n const heightToWidth = imageTensor.shape[0] / imageTensor.shape[1];\n const cropTop = (1-heightToWidth) / 2;\n const cropBottom = 1 - cropTop;\n squareCrop = [[cropTop, 0, cropBottom, 1]];\n } else {\n const cropLeft = (1-widthToHeight) / 2;\n const cropRight = 1 - cropLeft;\n squareCrop = [[0, cropLeft, 1, cropRight]];\n }\n // Expand image input dimensions to add a batch dimension of size 1.\n const crop = tf.image.cropAndResize(imageTensor.expandDims(), squareCrop, [0], [224, 224]);\n \n return crop.div(255);\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":347,"value":"Please enter a URL of an image you wish to classify using MobileNet V3. The default image is the one we used to learn about image tensors in [the previous notebook](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js). The main subject of our default image is a hummingbird.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":447,"value":"viewof imgUrl = Inputs.text({\n type: \"url\", \n label: \"Enter an image URL:\", \n value: `${await FileAttachment(\"Ramphodon_naevius_with_flower.jpeg\").url()}`, \n placeholder: \"Enter an image URL\"\n})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":186,"value":"

A Saw-billed Hermit (Ramphodon naevius). Author:\thttps://www.flickr.com/photos/dariosanches/. Source: Wikimedia Commons.

This file is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic license. license.

","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":140,"value":"img = new Promise((resolve, reject) => {\n const img = htl.html``;\n img.src = imgUrl;\n img.crossOrigin = 'anonymous';\n img.onload = () => resolve(img);\n img.onerror = reject;\n})","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":554,"value":"We convert the image into a tensor.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":149,"value":"imageTensor = tf.browser.fromPixels(img)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":153,"value":"imageTensor.shape","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":556,"value":"We process the image tensor using the provided preprocessing script.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":151,"value":"processedImageTensor = preprocess(imageTensor)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":558,"value":"We can see that the tensor has been reshaped to ${tex`224\\times 224`} and is a batch of size 1.","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":164,"value":"processedImageTensor.shape","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":158,"value":"{\n const canvas = htl.html`