?? Author ：Horizon Max
? 編程技巧篇：各種操作小結(jié)
?? 機器視覺篇：會變魔術(shù) OpenCV
?? 深度學(xué)習(xí)篇：簡單入門 PyTorch
?? 神經(jīng)網(wǎng)絡(luò)篇：經(jīng)典網(wǎng)絡(luò)模型
?? 算法篇：再忙也別忘了 LeetCode
?? Squeeze-and-Excitation Networks
Squeeze ：擠壓 ? ? Excitation ：激勵 ；
Squeeze-and-Excitation Networks 簡稱 SENet ，由 Momenta 和 牛津大學(xué) 的Jie Hu等人 提出的一種新的網(wǎng)絡(luò)結(jié)構(gòu)；
目標是通過建模 卷積特征通道之間的相互依賴關(guān)系 來提高網(wǎng)絡(luò)的表示能力；
在2017年最后一屆 ImageNet 挑戰(zhàn)賽(ILSVRC) classification 任務(wù)中獲得 冠軍，將錯誤率降低到 2.251% ；
?? 論文地址：Squeeze-and-Excitation Networks
?? SENet 詳解
?? Squeeze-and-Excitation block
對于任意給定的變換： Ftr ：X → U ，其中 X ∈ R H’xW’xC’ ， U ∈ R HxWxC ，Ftr 用作一個卷積算子 ；
?? Squeeze: Global Information Embedding
擠壓：全局信息嵌入
（1）Squeeze ：特征U通過 squeeze 壓縮操作，將跨空間維度H × W的特征映射進行聚合，生成一個通道描述符，HxWxC → 1x1xC ；
將 全局空間信息 壓縮到上述 通道描述符 中，使來這些 通道描述符 可以被 其輸入的層 利用，這里采用的是 global average pooling ；
?? Excitation: Adaptive Recalibration
激勵：自適應(yīng)調(diào)整
（2）Excitation ：每個通道通過一個 基于通道依賴 的自選門機制 來學(xué)習(xí)特定樣本的激活，使其學(xué)會使用全局信息，有選擇地強調(diào)信息特征，并抑制不太有用的特征，這里采用的是 sigmoid ，并在中間嵌入了 ReLU 函數(shù)用于限制模型的復(fù)雜性和幫助訓(xùn)練 ；
通過 兩個全連接層(FC) 構(gòu)成的瓶頸來參數(shù)化門控機制，即 W1 用于降低維度，W2 用于維度遞增 ；
（3）Reweight ：將 Excitation 輸出的權(quán)重通過乘法逐通道加權(quán)到輸入特征上；
總的來說 SE Block 就是在 Layer 的輸入和輸出之間添加結(jié)構(gòu)： global average pooling - FC - ReLU - FC- sigmoid ；
SE block 的靈活性意味著它可以直接應(yīng)用于標準卷積以外的轉(zhuǎn)換，通過將 SE block 集成到任何復(fù)雜模型當(dāng)中來開發(fā)SENet；
?? 在非殘差網(wǎng)絡(luò)中的應(yīng)用
應(yīng)用于 非殘差網(wǎng)絡(luò) Inception network 當(dāng)中，形成 SE-Inception module ；
Scale ： 改變（文字、圖片）的尺寸大小
?? 在殘差網(wǎng)絡(luò)中的應(yīng)用
應(yīng)用于 殘差網(wǎng)絡(luò) Residual network 當(dāng)中，形成 SE-ResNet module ；
論文中對 SE block 的應(yīng)用用于實驗對比：
SE-ResNet-50 網(wǎng)絡(luò)的準確性優(yōu)于 ResNet-50 和模型深化版的 ResNet101 網(wǎng)絡(luò) ；
對于224 × 224像素的輸入圖像，ResNet-50 需要164 ms，而 SE-ResNet-50 需要167 ms ；
?? SENet 復(fù)現(xiàn)
這里實現(xiàn)的是 SE-ResNet 系列網(wǎng)絡(luò) ：文章來源：http://www.zghlxwxcb.cn/news/detail-807521.html
# Here is the code :
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchinfo import summary
class SE_Block(nn.Module): # Squeeze-and-Excitation block
def __init__(self, in_planes):
super(SE_Block, self).__init__()
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.conv1 = nn.Conv2d(in_planes, in_planes // 16, kernel_size=1)
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(in_planes // 16, in_planes, kernel_size=1)
self.sigmoid = nn.Sigmoid()
def forward(self, x):
x = self.avgpool(x)
x = self.conv1(x)
x = self.relu(x)
x = self.conv2(x)
out = self.sigmoid(x)
return out
class BasicBlock(nn.Module): # 左側(cè)的 residual block 結(jié)構(gòu)（18-layer、34-layer）
expansion = 1
def __init__(self, in_planes, planes, stride=1): # 兩層卷積 Conv2d + Shutcuts
super(BasicBlock, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3,
stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,
stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.SE = SE_Block(planes) # Squeeze-and-Excitation block
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != self.expansion*planes: # Shutcuts用于構(gòu)建 Conv Block 和 Identity Block
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes)
)
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
SE_out = self.SE(out)
out = out * SE_out
out += self.shortcut(x)
out = F.relu(out)
return out
class Bottleneck(nn.Module): # 右側(cè)的 residual block 結(jié)構(gòu)（50-layer、101-layer、152-layer）
expansion = 4
def __init__(self, in_planes, planes, stride=1): # 三層卷積 Conv2d + Shutcuts
super(Bottleneck, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,
stride=stride, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.conv3 = nn.Conv2d(planes, self.expansion*planes,
kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(self.expansion*planes)
self.SE = SE_Block(self.expansion*planes) # Squeeze-and-Excitation block
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != self.expansion*planes: # Shutcuts用于構(gòu)建 Conv Block 和 Identity Block
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes)
)
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = F.relu(self.bn2(self.conv2(out)))
out = self.bn3(self.conv3(out))
SE_out = self.SE(out)
out = out * SE_out
out += self.shortcut(x)
out = F.relu(out)
return out
class SE_ResNet(nn.Module):
def __init__(self, block, num_blocks, num_classes=1000):
super(SE_ResNet, self).__init__()
self.in_planes = 64
self.conv1 = nn.Conv2d(3, 64, kernel_size=3,
stride=1, padding=1, bias=False) # conv1
self.bn1 = nn.BatchNorm2d(64)
self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1) # conv2_x
self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2) # conv3_x
self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2) # conv4_x
self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2) # conv5_x
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.linear = nn.Linear(512 * block.expansion, num_classes)
def _make_layer(self, block, planes, num_blocks, stride):
strides = [stride] + [1]*(num_blocks-1)
layers = []
for stride in strides:
layers.append(block(self.in_planes, planes, stride))
self.in_planes = planes * block.expansion
return nn.Sequential(*layers)
def forward(self, x):
x = F.relu(self.bn1(self.conv1(x)))
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.layer4(x)
x = self.avgpool(x)
x = torch.flatten(x, 1)
out = self.linear(x)
return out
def SE_ResNet18():
return SE_ResNet(BasicBlock, [2, 2, 2, 2])
def SE_ResNet34():
return SE_ResNet(BasicBlock, [3, 4, 6, 3])
def SE_ResNet50():
return SE_ResNet(Bottleneck, [3, 4, 6, 3])
def SE_ResNet101():
return SE_ResNet(Bottleneck, [3, 4, 23, 3])
def SE_ResNet152():
return SE_ResNet(Bottleneck, [3, 8, 36, 3])
def test():
net = SE_ResNet50()
y = net(torch.randn(1, 3, 224, 224))
print(y.size())
summary(net, (1, 3, 224, 224))
if __name__ == '__main__':
test()
輸出結(jié)果：文章來源地址http://www.zghlxwxcb.cn/news/detail-807521.html
torch.Size([1, 1000])
===============================================================================================
Layer (type:depth-idx) Output Shape Param #
===============================================================================================
SE_ResNet -- --
├─Conv2d: 1-1 [1, 64, 224, 224] 1,728
├─BatchNorm2d: 1-2 [1, 64, 224, 224] 128
├─Sequential: 1-3 [1, 256, 224, 224] --
│ └─Bottleneck: 2-1 [1, 256, 224, 224] --
│ │ └─Conv2d: 3-1 [1, 64, 224, 224] 4,096
│ │ └─BatchNorm2d: 3-2 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-3 [1, 64, 224, 224] 36,864
│ │ └─BatchNorm2d: 3-4 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-5 [1, 256, 224, 224] 16,384
│ │ └─BatchNorm2d: 3-6 [1, 256, 224, 224] 512
│ │ └─SE_Block: 3-7 [1, 256, 1, 1] 8,464
│ │ └─Sequential: 3-8 [1, 256, 224, 224] 16,896
│ └─Bottleneck: 2-2 [1, 256, 224, 224] --
│ │ └─Conv2d: 3-9 [1, 64, 224, 224] 16,384
│ │ └─BatchNorm2d: 3-10 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-11 [1, 64, 224, 224] 36,864
│ │ └─BatchNorm2d: 3-12 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-13 [1, 256, 224, 224] 16,384
│ │ └─BatchNorm2d: 3-14 [1, 256, 224, 224] 512
│ │ └─SE_Block: 3-15 [1, 256, 1, 1] 8,464
│ │ └─Sequential: 3-16 [1, 256, 224, 224] --
│ └─Bottleneck: 2-3 [1, 256, 224, 224] --
│ │ └─Conv2d: 3-17 [1, 64, 224, 224] 16,384
│ │ └─BatchNorm2d: 3-18 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-19 [1, 64, 224, 224] 36,864
│ │ └─BatchNorm2d: 3-20 [1, 64, 224, 224] 128
│ │ └─Conv2d: 3-21 [1, 256, 224, 224] 16,384
│ │ └─BatchNorm2d: 3-22 [1, 256, 224, 224] 512
│ │ └─SE_Block: 3-23 [1, 256, 1, 1] 8,464
│ │ └─Sequential: 3-24 [1, 256, 224, 224] --
├─Sequential: 1-4 [1, 512, 112, 112] --
│ └─Bottleneck: 2-4 [1, 512, 112, 112] --
│ │ └─Conv2d: 3-25 [1, 128, 224, 224] 32,768
│ │ └─BatchNorm2d: 3-26 [1, 128, 224, 224] 256
│ │ └─Conv2d: 3-27 [1, 128, 112, 112] 147,456
│ │ └─BatchNorm2d: 3-28 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-29 [1, 512, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-30 [1, 512, 112, 112] 1,024
│ │ └─SE_Block: 3-31 [1, 512, 1, 1] 33,312
│ │ └─Sequential: 3-32 [1, 512, 112, 112] 132,096
│ └─Bottleneck: 2-5 [1, 512, 112, 112] --
│ │ └─Conv2d: 3-33 [1, 128, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-34 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-35 [1, 128, 112, 112] 147,456
│ │ └─BatchNorm2d: 3-36 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-37 [1, 512, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-38 [1, 512, 112, 112] 1,024
│ │ └─SE_Block: 3-39 [1, 512, 1, 1] 33,312
│ │ └─Sequential: 3-40 [1, 512, 112, 112] --
│ └─Bottleneck: 2-6 [1, 512, 112, 112] --
│ │ └─Conv2d: 3-41 [1, 128, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-42 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-43 [1, 128, 112, 112] 147,456
│ │ └─BatchNorm2d: 3-44 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-45 [1, 512, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-46 [1, 512, 112, 112] 1,024
│ │ └─SE_Block: 3-47 [1, 512, 1, 1] 33,312
│ │ └─Sequential: 3-48 [1, 512, 112, 112] --
│ └─Bottleneck: 2-7 [1, 512, 112, 112] --
│ │ └─Conv2d: 3-49 [1, 128, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-50 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-51 [1, 128, 112, 112] 147,456
│ │ └─BatchNorm2d: 3-52 [1, 128, 112, 112] 256
│ │ └─Conv2d: 3-53 [1, 512, 112, 112] 65,536
│ │ └─BatchNorm2d: 3-54 [1, 512, 112, 112] 1,024
│ │ └─SE_Block: 3-55 [1, 512, 1, 1] 33,312
│ │ └─Sequential: 3-56 [1, 512, 112, 112] --
├─Sequential: 1-5 [1, 1024, 56, 56] --
│ └─Bottleneck: 2-8 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-57 [1, 256, 112, 112] 131,072
│ │ └─BatchNorm2d: 3-58 [1, 256, 112, 112] 512
│ │ └─Conv2d: 3-59 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-60 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-61 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-62 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-63 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-64 [1, 1024, 56, 56] 526,336
│ └─Bottleneck: 2-9 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-65 [1, 256, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-66 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-67 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-68 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-69 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-70 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-71 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-72 [1, 1024, 56, 56] --
│ └─Bottleneck: 2-10 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-73 [1, 256, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-74 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-75 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-76 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-77 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-78 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-79 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-80 [1, 1024, 56, 56] --
│ └─Bottleneck: 2-11 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-81 [1, 256, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-82 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-83 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-84 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-85 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-86 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-87 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-88 [1, 1024, 56, 56] --
│ └─Bottleneck: 2-12 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-89 [1, 256, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-90 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-91 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-92 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-93 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-94 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-95 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-96 [1, 1024, 56, 56] --
│ └─Bottleneck: 2-13 [1, 1024, 56, 56] --
│ │ └─Conv2d: 3-97 [1, 256, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-98 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-99 [1, 256, 56, 56] 589,824
│ │ └─BatchNorm2d: 3-100 [1, 256, 56, 56] 512
│ │ └─Conv2d: 3-101 [1, 1024, 56, 56] 262,144
│ │ └─BatchNorm2d: 3-102 [1, 1024, 56, 56] 2,048
│ │ └─SE_Block: 3-103 [1, 1024, 1, 1] 132,160
│ │ └─Sequential: 3-104 [1, 1024, 56, 56] --
├─Sequential: 1-6 [1, 2048, 28, 28] --
│ └─Bottleneck: 2-14 [1, 2048, 28, 28] --
│ │ └─Conv2d: 3-105 [1, 512, 56, 56] 524,288
│ │ └─BatchNorm2d: 3-106 [1, 512, 56, 56] 1,024
│ │ └─Conv2d: 3-107 [1, 512, 28, 28] 2,359,296
│ │ └─BatchNorm2d: 3-108 [1, 512, 28, 28] 1,024
│ │ └─Conv2d: 3-109 [1, 2048, 28, 28] 1,048,576
│ │ └─BatchNorm2d: 3-110 [1, 2048, 28, 28] 4,096
│ │ └─SE_Block: 3-111 [1, 2048, 1, 1] 526,464
│ │ └─Sequential: 3-112 [1, 2048, 28, 28] 2,101,248
│ └─Bottleneck: 2-15 [1, 2048, 28, 28] --
│ │ └─Conv2d: 3-113 [1, 512, 28, 28] 1,048,576
│ │ └─BatchNorm2d: 3-114 [1, 512, 28, 28] 1,024
│ │ └─Conv2d: 3-115 [1, 512, 28, 28] 2,359,296
│ │ └─BatchNorm2d: 3-116 [1, 512, 28, 28] 1,024
│ │ └─Conv2d: 3-117 [1, 2048, 28, 28] 1,048,576
│ │ └─BatchNorm2d: 3-118 [1, 2048, 28, 28] 4,096
│ │ └─SE_Block: 3-119 [1, 2048, 1, 1] 526,464
│ │ └─Sequential: 3-120 [1, 2048, 28, 28] --
│ └─Bottleneck: 2-16 [1, 2048, 28, 28] --
│ │ └─Conv2d: 3-121 [1, 512, 28, 28] 1,048,576
│ │ └─BatchNorm2d: 3-122 [1, 512, 28, 28] 1,024
│ │ └─Conv2d: 3-123 [1, 512, 28, 28] 2,359,296
│ │ └─BatchNorm2d: 3-124 [1, 512, 28, 28] 1,024
│ │ └─Conv2d: 3-125 [1, 2048, 28, 28] 1,048,576
│ │ └─BatchNorm2d: 3-126 [1, 2048, 28, 28] 4,096
│ │ └─SE_Block: 3-127 [1, 2048, 1, 1] 526,464
│ │ └─Sequential: 3-128 [1, 2048, 28, 28] --
├─AdaptiveAvgPool2d: 1-7 [1, 2048, 1, 1] --
├─Linear: 1-8 [1, 1000] 2,049,000
===============================================================================================
Total params: 28,080,344
Trainable params: 28,080,344
Non-trainable params: 0
Total mult-adds (G): 63.60
===============================================================================================
Input size (MB): 0.60
Forward/backward pass size (MB): 2691.18
Params size (MB): 112.32
Estimated Total Size (MB): 2804.10
===============================================================================================
到了這里，關(guān)于[ 注意力機制 ] 經(jīng)典網(wǎng)絡(luò)模型1——SENet 詳解與復(fù)現(xiàn)的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
