25:["$","title",null,{"children":"Image tensors | Jeya Balaji Balasubramanian | Observable"}] 4b:T4df,# Image tensors Images are a good data type to get some early hands-on experience with deep learning models in TensorFlow. Imaging deep learning models are highly successful in real-world applications. Images are intuitive because they provide a visual feedback on what each data transformation operation does. Deep learning models for [natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing) are also highly successful in the real-world but they require at least some linguistics background, which might often distract too much from the core ideas of data modeling. *Image tensors* contain the value of each pixel of the image that it encodes. If you took a magnifying glass and looked closely at an LCD monitor displaying white light, you will notice that in close-up, each pixel is really composed of three fully-lit sub-pixel bulbs of colors: red, green, and blue. You can see them in the image below. To display the color black, all the three bulbs are switched off. To display the color orange, as seen in the image below, only the red and the green bulbs are lit and the blue bulb is switched off. By controlling the intensities of the three colored bulbs, each pixel can display a full spectrum of colors.4c:T604,The data type of image tensors are either `Int32` or `Float32`. If they are encoded in `Int32`, the pixel values range from `0` to `255`. A single pixel is encoded with 1 byte of information i.e., 8 bits, which can take ${tex`2^8 = 256`} different values (`0-255`). If they are encoded in `Float32`, the pixel values range from `0` to `1`. Images are typically encoded as either grayscale or RGB (red, green, blue). If the image tensors encode a grayscale image, then each pixel only requires one value i.e., the intensity of the color white. If the image tensor is of type `Int32`, the value `0` encodes the color black and the value `255` encodes the color white. The values in ranges `1-254` encode the "254 shades of gray". If the image tensor is of type `Float32`, the value `0.0` encodes the color black and the value `1.0` encodes the color white. All values in between range `0.0-1.0` encode the different shades of gray. If the image tensors encode an RGB image, then each pixel contains three values i.e., the intensities of the red, green, and blue sub-pixels. If the image tensor is of type `Int32`, the value `[0, 0, 0]` encodes the color black and the value `[255, 255, 255]` encodes the color white. The different combinations of the three values encode a full spectrum of colors. Similarly, if the image tensor is of type `Float32` the value `[0.0, 0.0, 0.0]` encodes the color black and the value `[1.0, 1.0, 1.0]` encodes the color white. All the other different combinations of values encode a full spectrum of colors.4d:T58d,## Table of Contents - [Image tensor dimensions](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-88) - [Rendering image tensors as images](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-33) - [Encoding images as image tensors](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-119) - [Converting a DOM element image into an image tensor](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-274) - [Converting an in-memory image into an image tensor](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-276) - [Image tensor manipulations](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-329) - [Resizing](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-463) - [Cropping](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-465) - [Mirroring](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-468) - [Rotating](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-496) - [Conclusion](https://observablehq.com/@jeyabbalas/image-tensors?collection=@jeyabbalas/tensorflow-js#cell-674)4e:T662,## Image tensor dimensions The values needed to encode each pixel in an image tensor are called *channels*. Since grayscale images only encode the intensity of the color white, each pixel only contains 1 value. So, grayscale image tensors have 1 channel. Each pixel in RGB image tensors encode the intensities of the colors red, green, and blue. So, RGB tensors have 3 channels. In some rare occasions images may be encoded as RGBA (red, green, blue, alpha) values, which contain 4 channels. However, in this notebook, we will only explore grayscale and RGB encodings. The *rank* of image tensors is usually 3. The first rank corresponds to the pixel height of the image. The second rank corresponds to the pixel width of the image. The third rank corresponds to the channels in the image. For grayscale images, the third rank has only 1 value, so sometimes grayscale image tensors are only encoded using rank-2 tensors. RGB image tensors are rank-3, where the third rank contains 3 (RGB intensity) values. **Note**: The way that tensor dimensions are encoded can be a bit counter-intuitive. Typically, when someone tells you that an image has dimensions ${tex`1024\times 768`}, they mean that the number of pixels in the width of the image is `1024` and the number of pixels in the height of the image is `768`. If this was an RGB image, the tensor dimensions would be ${tex`768\times 1024\times 3`}. Here, the first rank corresponds to the number of pixels in the height of the image and the second rank corresponds to the number of pixels in the width of the image. So, remember this when you think of images in terms of tensors.4f:T77b,## Rendering image tensors as images The method [`tf.browser.toPixels()`](https://js.tensorflow.org/api/latest/#browser.toPixels) renders an input tensor as an image onto an HTML canvas element. It takes a tensor as an input. The tensor could be 2D (implies grayscale) or 3D (grayscale if the 3rd rank has 1 channel, RGB if it has 3 channels, or RGBA if it has 4 channels). The method takes in an optional second parameter, a canvas element. The method returns a Promise of a `Uint8ClampedArray`. If a canvas is specified, the image in the byte array is rendered there. If no canvas is specified, the returned array can be used to render at a later time. In the code below, we create a 3D tensor with 3 channels, all randomly initialized between 0 and 1 (type `Float32` image). We also create a canvas, where we want our tensor to be rendered into an image. We pass the tensor and the canvas as parameters to the `tf.browser.toPixels()` method and return the canvas. The rendered image should contain pixels taking many different colors of the visual spectrum. Try setting the number of channels to 1 ([`tf.randomUniform([height, width, 1])`](https://js.tensorflow.org/api/latest/#randomUniform)) or pass a 2D tensor (`tf.randomUniform([height, width])`). This would turn the tensor into a grayscale image. What happens when you fill the tensor with zeros ([`tf.zeros([height, width, 3])`](https://js.tensorflow.org/api/latest/#zeros))? You should expect all pixels in the image to turn black. What if you fill with ones ([`tf.ones([height, width, 3])`](https://js.tensorflow.org/api/latest/#ones))? You should expect all pixels in the image to turn white. Note that we dispose off the tensor as soon as we are done with it. [In the previous notebook](https://observablehq.com/@jeyabbalas/tensors?collection=@jeyabbalas/tensorflow-js), we saw the importance of doing this. We do not want to create orphan tensors.50:T477,## Encoding images as image tensors The method [`tf.browser.fromPixels()`](https://js.tensorflow.org/api/latest/#browser.fromPixels) encodes an image into a tensor. This method is quite versatile because the input image can be of any of the following types— PixelData, [ImageData](https://developer.mozilla.org/en-US/docs/Web/API/ImageData), [HTMLImageElement (``)](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement), [HTMLCanvasElement (`