2d:null 2e:["$","title",null,{"children":"Visualization in Deep Learning | Kai Middlebrook | Observable"}] 33:T6b3,md`# Motivation [Humans can process visual information much faster and better than tabular or numerical information](http://misrc.umn.edu/workingpapers/fullpapers/1986/8611.pdf). Given this fact, it isn't suprising that many inflection points in the history of science came after moments when science “zoomed in.” At these points, we develop a visualization or tool that allows us to see the world in a new level of detail or from a new perspective. This can even lead to the development of new fields of science focused on studying the world through a new lens.

Hooke’s Micrographia revealed a rich microscopic world as seen through a microscope, including the initial discovery of cells (Left, Center). Images from the National Library of Wales. The Golgi stain first allowed for the visualization of individual neurons. (Right). Image from the Wikipedia Commons.

For example, microscopes let us see cells, leading to cellular biology. Following the invention of the microscope, Camillo Golgi invented a silver staining technique which revealed the intricate structures of individual neurons. Science zoomed in. These transitions weren’t just a change in precision: they were qualitative changes in what the objects of scientific inquiry are. For example, neuroscience isn’t just more careful cellular biology. It’s a new kind of inquiry that dramatically shifts what we can understand. `34:T4ac,md`

Visualizatizing learned features in layer mixed4d of GoogLeNet. Image from [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/).

Just as the early microscopes and staining methods hinted at a new world of cells and neural systems, visualizations of artificial neural networks have revealed hints of a rich inner world within our models (e.g. [[1](https://arxiv.org/pdf/1506.02078.pdf), [2](https://www.researchgate.net/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network), [3](https://distill.pub/2017/feature-visualization/), [4](https://arxiv.org/pdf/1312.6034.pdf), [5](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html), [6](https://arxiv.org/pdf/1704.03296.pdf), [7](https://arxiv.org/pdf/1906.02715.pdf), [8](https://distill.pub/2019/activation-atlas/)] ). This promts us to wonder: could deep learning be at a similar, although more modest, transition point?

`35:T956,md`

Example of feature visualization from layer mixed4d of GoogLeNet. Image from [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/).

Example of feature attribution output when the input image on the left is feed to GoogleLeNet. Image from [Feature Visualization](https://distill.pub/2017/feature-visualization/).

Example of two dimensionality reduction techniques used to visualize learned embeddings on MNIST. Image from [t-SNE](http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf) (top) and [UMAP](https://umap-learn.readthedocs.io/en/latest/supervised.html) (bottom).

There is a growing sense that neural networks need to be more interpretable. Recently, the field of neural network interpretability has formed in response to these concerns. From it, three powerful visualization techniques have developed: feature visualization (e.g., [[2](https://www.researchgate.net/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network) ,[3](https://distill.pub/2017/feature-visualization/)] ), attribution (e.g., [[9](https://arxiv.org/pdf/1703.01365.pdf), [10](https://distill.pub/2020/attribution-baselines/)] ), and dimensionality reduction (e.g., [[11](http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf), [12](https://arxiv.org/pdf/1802.03426.pdf)] ). In this post, we will briefly look at two of these techniques: feature visualization and feature attribution. `36:T1970,html`

Image: Feature Visualization of GoogLeNet trained on ImageNet.

Feature visualization techniques aim to answers questions about what a network — or parts of a network — are looking for by generating examples.

How can we visualize how our model (e.g., GoogLeNet) sees an image? As a first step, we could try feeding an image into our network and collect the activations — the numerical values of how much each neuron fired — from the layer of interest. After all, we know that if a neuron is excited by what it is shown, it’s activation value will be positive [3].

Unfortunatley, the vectors of activation values we just collected are just vectors of unitless numbers. Humans can't really interpret these vectors. This is where feature visualization comes in handy!

At a high level, we can think of feature visualization as creating an idealized image of what the network thinks would produce a particular activation vector. This is different from how we normally use neural networks: to transform an image into an activation vector and ultimately make a prediction. In feature visualization we go in the opposite direction. Starting with an activation vector at a particular layer, we create an image through an iterative optimization process. [2].

At a lower level, to understand how feature visualization works we need to remember that neural networks are differentialable with respect to their inputs. This means we can leverage the power of derivatives to find what kind of input would cause a certain behavior — whether that’s an internal neuron firing or the final output behavior. In other words, we optimize an input (image, text, audio, etc.) to activate a particular neuron or group of neurons (e.g., the optimization objective).

If derivaties, differentiation, and optimization sound like gibberish that's ok. You don't need to know exactly how the process works for this post. Just remember that feature visualization tries to create an image of what our network thinks would produce a particular activation vector (e.g., a layer or neuron within a layer). Still confused? It may be useful to think of feature visualization as a language translator (metaphorically speaking of course). But rather than our translator translating between two human languages, she is translating between a human language and a machine language. Her translations help us understand what our models say and see.

Here are a few examples of feature visualizations for different optimization objectives.



Neuron objective `layer_n[x,y,z]`	Channel objective `layer_n[:,:,z]`	Layer objective `layer_n[:,:,:]²`

From the above feature visualizations we can start developing hypotheses about our network: Neuron objective layer_n[x,y,z] may be learning to identify dog eyes or noses, Channel objective layer_n[:,:,z] may learn to identify complex symmetric hexigonal like shapes and defined lines. The most informative visualization seems to be the Layer objective at layer_n[:,:,:]². As a unit, this layer seems to be excited by dog heads and possibly green grass.

`37:T458c,md`

While feature visualization is a powerful and useful tool, it can only help us understand what our network detects. What if we want to know how or why our network arrived at a decision?

This is when feature attribution can be useful! Feature attribution tries to explain the relationships between neurons. There are a wide variety of ways one can approach attribution, and it is still unclear which technique is best. Some researchers even argue that attribution is not useful at all [13]. However, the majority of deep learning visualization researchers do not share this feeling. However, there is ample opportunity for more research to take place on attribution methods.

Example of Saliency Map (middle) and RectGrad attributions (right) methods. Although the saliency map has more noise, both attribution methods suggest that the model thinks the mantis's body is most important to properly label this input image.

Feature attribution methods can be separated into three groups:

General Feature Attribution: Evaluates the contribution of each input feature to the output of a model.
Layer Attribution: Evaluates the contribution of each neuron in a given layer to the output of the model.
Neuron Attribution: Evaluates the contribution of each input feature on the activation of a particular hidden neuron.

The most common interface for attribution is called a saliency map. Saliency maps are simple heatmaps that highlight pixels of the input image that most caused the output classification [14].

Saliency maps are generally easy to interpret. But they have two main weaknesses. First, it may be inappropriate to use individual pixels as the primary attribution. We know that the human brain doesn't make decisions by considering items in isolation, should neural networks be any different? Moreover, pixel level importance measures are not robust to simple visual transforms (e.g., brightness, contrast, etc.), and are far-removed from high-level concepts like the output class [14]. Second, traditional saliency maps are a very limited type of interface — they only display the attribution for a single class at a time, and do not allow you to investigate individual points more deeply. This is because they do not explicitly deal with hidden layers, making it difficult to explore fully their capabilities.

Due to these limitations, research is ongoing to find more robust and flexible methods to attribute importance across layers, channels, and neurons. Some recently proposed methods communication interpretability through heatmaps just like the popular saliency method we saw (e.g., center heatmap in the mantis picture). However, other methods use bar charts to communicate importance. Bar charts may not be as fun to look at compared to heatmaps but they can be as effective or more at communicating attribution importance.

Example of feature attribution visualized by a barchart. The chart displayes the average feature importance of features from the Titanic Dataset.

From the feature attribution chart, we gain interesting insights regarding the importance of various features. We can see that the strongest features appear to be age and being male, which are negatively correlated with survival. In general, embarking at Queenstown and the number of parents / children appear to be less important features. Note that we could also visualize the importance of individual layers or neurons within a layer towards classification.

Attribution methods that use bar charts are not perfect either. In fact, all attribution methods are sensitive to the distribution of the data they received to generate attribution scores. We can't conclude with confidence that the average attribution scores we calculated will be true on a different dataset. This can even be true when we measure attribution scores on a train and test set. In other words, it is possible that features have very different attributions for different examples in the data. Nevertheless, these methods can help us better understand our networks.

`38:T971,md`

Model behavior is extremely complex, and our current visualization methods limit us to show only speciﬁc aspects of it. An important direction for future interpretability research will be developing techniques that achieve broader coverage of model behavior. For instance, visualizing rich connections between layers or neurons. In additon, in the evolving field of deep learning interpretability, it will be crucial to develop interfaces that do not mislead. Engaging with visualizations should not cause users to implicitly draw incorrect conclusions about the model. Insuring this will not be easy. But it is an incredibly important goal we must strive towards.

Research on model interpretability relies heavily on qualitative methods. Unfortunately, some members of the machine learning and broader computer science communities do not take this work very seriously. The common misplaced assumption is that this work is too qualitative and so lacks scientific rigour. As a result, some interpretability researchers may feel they are not as respected or listened to compared to their peers. But do not fear, history suggests we are doing important work!

Hooke’s Micrographia revealed a rich microscopic world as seen through a microscope, including the initial discovery of cells (Left, Center). Images from the National Library of Wales. The Golgi stain first allowed for the visualization of individual neurons. (Right). Image from the Wikipedia Commons.

In many respects the microscope is a symbol of science. But this wasn't always the case. In fact, the invention went relatively unnoticed for about fifty years! This all changed when Robert Hooke published Micrographia, consisting of a collection of drawings of things he had seen using a microscope, including the first picture of a cell. The discovery of cells was a qualitative research result. But that didn't stop it from changing the world.

`39:[] 31:{"id":"e7ab4b4e779c41de","slug":"visualization-in-deep-learning","title":"Visualization in Deep Learning","thumbnail":"5f4d468867bd0ee373d567111608751f384f10449a16b1c4eaf46dcbc1f60ac0","access_level":"public","create_time":"$D2020-04-01T23:23:12.763Z","update_time":"$D2020-04-06T19:07:56.365Z","view_time":null,"first_public_time":"$D2020-04-06T10:08:43.172Z","has_unpublished_edits":null,"fork":false,"stars":20,"starred":false,"owner":{"type":"team","id":"e0b1764046d92b32","login":"krmiddlebrook","name":"Kai Middlebrook","avatar_url":"https://avatars-next.observableusercontent.com/b53a4ca49319b278e0a4baf77bbf124e87e11a3a664602ba4eebc90d17aa9801","tier":"starter_2024","bio":"Data Science and Music --\r\nUniversity of San Francisco"},"version":552,"publish_version":552,"role":null,"authored":false,"authors":[{"type":"individual","id":"84b19e1938bdcb89","login":"krmiddlebrook","name":"Kai Middlebrook","avatar_url":"https://avatars-next.observableusercontent.com/b53a4ca49319b278e0a4baf77bbf124e87e11a3a664602ba4eebc90d17aa9801","tier":null,"bio":"Data Science and Music --\r\nUniversity of San Francisco"}],"members":[{"type":"individual","id":"84b19e1938bdcb89","login":"krmiddlebrook","name":"Kai Middlebrook","avatar_url":"https://avatars-next.observableusercontent.com/b53a4ca49319b278e0a4baf77bbf124e87e11a3a664602ba4eebc90d17aa9801","tier":null,"role":"editor"}],"comments":[{"id":"3415ed874b4723ad","reply_to_id":null,"content":"The first image appears to be broken. I'd recommend to add it as a file attachment to the notebook and include it from there.","create_time":"$D2023-05-21T11:57:06.331Z","update_time":null,"resolved":false,"user":{"type":"individual","id":"07362516b5994994","login":"mootari","name":"Fabian Iwand","avatar_url":"https://avatars-next.observableusercontent.com/0d7defa821f38094c03bad23b9b360a5364e6e97e21fc238c39ddc48db7994ad","tier":null}}],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":5,"collection":{"id":"cc47632928450280","slug":"a-i-artificial-intelligence","title":"Artificial Intelligence","owner":{"id":"f35c755083683fe5","login":"observablehq","name":"Observable"}},"type":"notebook","body":{"titling":"auto","title":"Visualization in Deep Learning","cells":[{"id":0,"value":"md`\n# Visualization in Deep Learning\n### Towards Understanding Deep Neural Networks\nThis notebook will introduce you to feature visualization techniques for deep neural networks.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":39,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":56,"value":"md`\n# Interpretability\nWhat are our models learning?\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":72,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":167,"value":"md`\n # Feature Visualization\n What are our models looking for?\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":263,"value":"$36","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":415,"value":"md`\n # Attribution Visualization\n Why and how are our models making decision?\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":250,"value":"$37","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":475,"value":"md`\n# Conclusion \nA bright future ahead?\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":485,"value":"$38","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":534,"value":"md`## Resources`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":422,"value":"md`\nIf these visualization techniques interest you, I highly recommend these great interactive papers:\n* Feature Visualization\n* Attribution Visualization\n* Dimensionality Reduction Visualizations (mentioned briefly here): t-SNE and UMAP\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":509,"value":"md`## About Me\n* **Name**: Kai Middlebrook,\n* **Degree**: Data Science (May 2020), \n* **Interests**: Deep Learning, Music and Machine Learning, Neuroscience, Psychology, History\n* **Why this topic**: Deep learning models achieve great success on image recognition, speech processing, and language translation tasks. Yet, relatively little is known about why these networks work. Perhaps more importantly, there is no easy method to identify factors that cause models to produce unfavorable results. To advance deep learning safely and ethically, we must develop tools to better understand how our models work. Visualization techniques will play a crucial role in this effort. \n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W39","stdlib":"1"}} 32:"$31:body"