flashtorch：卷积神经网络的可视化

GitHub：FlashTorch

saliency maps论文地址： Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

FlashTorch简介

A Python visualization toolkit, built with PyTorch, for neural networks in PyTorch.

Neural networks are often described as “black box”. The lack of understanding on how neural networks make predictions enables unpredictable/biased models, causing real harm to society and a loss of trust in AI-assisted systems.

Feature visualization is an area of research, which aims to understand how neural networks perceive images. However, implementing such techniques is often complicated.

FlashTorch was created to solve this problem!

You can apply feature visualization techniques (such as saliency maps and activation maximization) on your model, with as little as a few lines of code.

It is compatible with pre-trained models that come with torchvision, and seamlessly integrates with other custom models built in PyTorch.

FlashTorch安装

使用命令行，通过pip安装。安装代码如下：

pip install flashtorch

安装结果如下：

如果要安装到Anaconda的Python下，参见博客：python+pip——使用pip将函数库安装到Python环境或Anaconda环境。

flashtorch中demo的实现

saliency maps（显著图）

import matplotlib.pyplot as plt
import torchvision.models as  modelsfrom flashtorch.utils import apply_transforms, load_image
from flashtorch.saliency import Backprop
image = load_image('C:/Users/73416/PycharmProjects/flashtorch-master/examples/images/peacock.jpg')# ------------------------显示图像------------------------
plt.imshow(image)
plt.title('Original image')
plt.axis('off')
plt.show()          # 自加# ------------------------可视化------------------------
peacock = apply_transforms(image)model = models.alexnet(pretrained=True)backprop = Backprop(model)backprop.visualize(peacock, 84, guided=True, use_gpu=True)
plt.show()# 标号不对时，产生的warning：
# C:\Users\73416\PycharmProjects\flashtorch-master\flashtorch\saliency\backprop.py:111: UserWarning: The predicted class index 251 does notequal the target class index 96. Calculatingthe gradient w.r.t. the predicted class.
#   'the gradient w.r.t. the predicted class.'

解析：

网络模型采用ImageNet的预训练模型，此处采用AlexNet。model = models.alexnet(pretrained=True)。
调用.visualize()或plt.imshow()显示之后，需要使用plt.show()来显示图像。
记得更改路径。

标号不对时，产生的warning：

C:\Users\73416\PycharmProjects\flashtorch-master\flashtorch\saliency\backprop.py:111: UserWarning: The predicted class index 251 does not equal the target class index 96. Calculatingthe gradient w.r.t. the predicted class.

结果显示：

activation maximization（激活最大化）

初始化

# %matplotlib inline
# %config InlineBackend.figure_format = 'retina'import torchvision.models as models
from flashtorch.activmax import GradientAscent
import matplotlib.pyplot as plt# ------------------------------自加显示网络结构的函数------------------------------
def show_architecture(architecture):
# input: list of network architecturefor i in range(len(architecture)):print(architecture[i])# ------------------------------Load a pre-trained Model------------------------------
model = models.vgg16(pretrained=True)
architecture=list(model.features.named_children())       # Print layers and corresponding indicies
show_architecture(architecture)

解析：

首先选择网络模型，为VGG16。model = models.vgg16(pretrained=True)。之后以list类型，显示网络结构。

注意网络是经过预训练的，也就是说VGG16是在ImageNet上训练好的。

结果如下所示：

E:\Anaconda\python.exe "C:/Users/73416/PycharmProjects/flashtorch-master/try_Activation Maximization.py"('0', Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('1', ReLU(inplace))
('2', Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('3', ReLU(inplace))
('4', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
('5', Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('6', ReLU(inplace))
('7', Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('8', ReLU(inplace))
('9', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
('10', Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('11', ReLU(inplace))
('12', Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('13', ReLU(inplace))
('14', Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('15', ReLU(inplace))
('16', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
('17', Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('18', ReLU(inplace))
('19', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('20', ReLU(inplace))
('21', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('22', ReLU(inplace))
('23', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
('24', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('25', ReLU(inplace))
('26', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('27', ReLU(inplace))
('28', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
('29', ReLU(inplace))
('30', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))

可视化：特定卷积层的特定滤波器

选择要显示的卷积层和滤波器

# ------------------------------Specify layers and filters------------------------------
conv1_2 = model.features[2]
conv1_2_filters = [17, 33, 34, 57]conv2_1 = model.features[5]
conv2_1_filters = [27, 40, 68, 73]conv3_1 = model.features[10]
conv3_1_filters = [31, 61, 147, 182]conv4_1 = model.features[17]
conv4_1_filters = [238, 251, 338, 495]conv5_1 = model.features[24]
conv5_1_filters = [45, 271, 363, 409]

以前两行为例，解说代码含义。

conv1_2 = model.features[2]是指显示的是model的编号为2的feature，参见上面的网络结构，可知其为('2', Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))，可知要显示的第一卷积层。

根据该卷积层的信息，可知in_channels=64, out_channels=64。所以该卷积层共有64个filter。conv1_2_filters = [17, 33, 34, 57]指明要显示的是fliter的编号是17，33，34，57。

实例化 GradientAscent

# Creating an instance of GradientAscent class
g_ascent = GradientAscent(model.features)

可视化所选定滤波器

# ------------------------------Optimize and visualize filters------------------------------
# optimization and visualization
g_ascent.visualize(conv1_2, conv1_2_filters, title='conv1_2')
plt.show()
g_ascent.visualize(conv2_1, conv2_1_filters, title='conv2_1')
plt.show()
g_ascent.visualize(conv3_1, conv3_1_filters, title='conv3_1')
plt.show()
g_ascent.visualize(conv4_1, conv4_1_filters, title='conv4_1')
plt.show()
g_ascent.visualize(conv5_1, conv5_1_filters, title='conv5_1')
plt.show()

要显示的卷积层由conv5_1来指定，要显示的滤波器的编号由conv5_1_filters给出。

可以看到，越低级的卷积层，可提取到的特征越简单。越高的卷积层，提取到的特征越复杂，比如conv5_1的filter_45会提取到眼睛的信息。

可视化：特定卷积层的随机滤波器

# ----------------------GradientAscent.visualize: randomly select filters----------------------
g_ascent.visualize(conv5_1, title='Randomly selected filters from conv5_1')
plt.show()

要显示的卷积层由conv5_1来指定，通过不传入滤波器编号来表示随机显示。

可视化：特定卷积层的一个滤波器

# -------------------------GradientAscent.visualize: plot one filter-------------------------
g_ascent.visualize(conv5_1, 3, title='conv5_1 filter 3')
plt.show()

可视化并返回 image tensor

# ------------------------GradientAscent.visualize: return image tensor------------------------
output = g_ascent.visualize(conv5_1, 3, title='conv5_1 filter 3', return_output=True)
plt.show()
print('num_iter:', len(output))
print('optimized image:', output[-1].shape)

deepdream

# ------------------------GradientAscent.deepdream: create DeepDream------------------------
g_ascent.deepdream('C:/Users/73416/PycharmProjects/flashtorch-master/examples/images/jay.jpg', conv5_1, 33)
plt.show()

perform optimization only

# -----------GradientAscent.optimize: perform optimization only (no visualization)-----------
output = g_ascent.optimize(conv5_1, 3)
print('num_iter:', len(output))
print('optimized image:', output[-1].shape)

结果如下：

num_iter: 30
optimized image: torch.Size([1, 3, 224, 224])