md`

While feature visualization is a powerful and useful tool, it can only help us understand what our network detects. What if we want to know how or why our network arrived at a decision?

This is when feature attribution can be useful! Feature attribution tries to explain the relationships between neurons. There are a wide variety of ways one can approach attribution, and it is still unclear which technique is best. Some researchers even argue that attribution is not useful at all [13]. However, the majority of deep learning visualization researchers do not share this feeling. However, there is ample opportunity for more research to take place on attribution methods.

Example of Saliency Map (middle) and RectGrad attributions (right) methods. Although the saliency map has more noise, both attribution methods suggest that the model thinks the mantis's body is most important to properly label this input image.

Feature attribution methods can be separated into three groups:

General Feature Attribution: Evaluates the contribution of each input feature to the output of a model.
Layer Attribution: Evaluates the contribution of each neuron in a given layer to the output of the model.
Neuron Attribution: Evaluates the contribution of each input feature on the activation of a particular hidden neuron.

The most common interface for attribution is called a saliency map. Saliency maps are simple heatmaps that highlight pixels of the input image that most caused the output classification [14].

Saliency maps are generally easy to interpret. But they have two main weaknesses. First, it may be inappropriate to use individual pixels as the primary attribution. We know that the human brain doesn't make decisions by considering items in isolation, should neural networks be any different? Moreover, pixel level importance measures are not robust to simple visual transforms (e.g., brightness, contrast, etc.), and are far-removed from high-level concepts like the output class [14]. Second, traditional saliency maps are a very limited type of interface — they only display the attribution for a single class at a time, and do not allow you to investigate individual points more deeply. This is because they do not explicitly deal with hidden layers, making it difficult to explore fully their capabilities.

Due to these limitations, research is ongoing to find more robust and flexible methods to attribute importance across layers, channels, and neurons. Some recently proposed methods communication interpretability through heatmaps just like the popular saliency method we saw (e.g., center heatmap in the mantis picture). However, other methods use bar charts to communicate importance. Bar charts may not be as fun to look at compared to heatmaps but they can be as effective or more at communicating attribution importance.

Example of feature attribution visualized by a barchart. The chart displayes the average feature importance of features from the Titanic Dataset.

From the feature attribution chart, we gain interesting insights regarding the importance of various features. We can see that the strongest features appear to be age and being male, which are negatively correlated with survival. In general, embarking at Queenstown and the number of parents / children appear to be less important features. Note that we could also visualize the importance of individual layers or neurons within a layer towards classification.

Attribution methods that use bar charts are not perfect either. In fact, all attribution methods are sensitive to the distribution of the data they received to generate attribution scores. We can't conclude with confidence that the average attribution scores we calculated will be true on a different dataset. This can even be true when we measure attribution scores on a train and test set. In other words, it is possible that features have very different attributions for different examples in the data. Nevertheless, these methods can help us better understand our networks.