1 簡介

論文指出，有兩種主要的計算高分辨率特征的方式：1 從ResNet等網絡輸出的低分辨率特征恢復高分辨率特征，同時可以得到中分辨率特征，如Hourglass、SegNet、DeconvNet、U-Net、encoder-decoder等。此時，可以使用上采樣網絡得到高分辨率特征，上采樣網絡應當和下采樣網絡互為對稱網絡（也有不對稱的上采樣過程，如RefineNet等）。2 通過高分辨率特征及並行的低分辨率特征來保持高分辨率特征。另一方面，空洞卷積可以用於計算中分辨率的特征。

HRNet能夠保持高分辨率的特征，而不是從低分辨率的特征恢復高分辨率的特征。

2 網絡結構

2.1 總體結構

論文的網絡結構如下圖所示，該圖有4個階段，第2、3、4階段均為重復的多分辨率模塊（modularized multi-resolution blocks）。在每個多分辨率模塊之前，有一個交換層（Translation layer），該層才會出現額外的特征圖。而多分辨率模塊（多分辨率分組卷積+多分辨率卷積）沒有額外的特征圖出現。

注意：論文中的網絡結構和代碼中的稍有差異，但是本質上是一樣的。如下面網址：

https://github.com/HRNet/HRNet-Object-Detection/issues/3

問題：

作者回答：

如下所示，左側的論文結構實際上等效於右側的代碼實現。d2為c2的下采樣，但是由於c2是a2和b2的全連接，因而也可以認為d2是a2和b2的全連接。

2.2 多分辨率模塊

多分辨率模塊包括多分辨率分組卷積（multi-resolution group convolution）和多分辨率卷積（multi-resolution convolution）。多分辨率分組卷積如下圖a所示，其是分組卷積的擴展，將輸入的通道分成了不同的子通道，每個子通道使用常規的卷積。多分辨率卷積如下圖b所示，輸入和輸出子集使用類似全連接的方式，實現上是常見的卷積。為了保持分辨率匹配，降低分辨率時使用的是多個stride=2的3*3conv，增加分辨率時使用的是上采樣（bilinear或者nearest neighbor）。下圖c左側為常規卷積，等效於右側的全連接多分支卷積。

==================================================

以下內容出自HRNetV1：

假設某階段輸入特征為$\left\{ {{\mathbf{X}}_{1}},{{\mathbf{X}}_{2}},\cdots ,{{\mathbf{X}}_{s}} \right\}$，輸出特征為$\left\{ {{\mathbf{Y}}_{1}},{{\mathbf{Y}}_{2}},\cdots ,{{\mathbf{Y}}_{s}} \right\}$，輸出特征的分辨率和寬高都和輸入的對應相等。每個輸出均為輸入的加權和${{\mathbf{Y}}_{k}}=\sum\nolimits_{i=1}^{s}{a\left( {{\mathbf{X}}_{i}},k \right)}$。Translation layer還有一個額外的特征${{\mathbf{Y}}_{s+1}}$：${{\mathbf{Y}}_{s+1}}=a\left( {{\mathbf{Y}}_{s}},s+1 \right)$，即為2.1中注意的地方。

$a\left( {{\mathbf{X}}_{i}},k \right)$包含從第i分辨率到第k分辨率的上采樣（使用最近鄰差值進行上采樣，並使用1*1conv保證通道一致）或者下采樣（1個3*3，stride=2的conv得到2x的下采樣，2個3*3，stride=2的conv得到4x的下采樣）操作。當i=k時，$a\left( \cdot ,\cdot \right)$為恆等連接（identify connection），即$a\left( {{\mathbf{X}}_{i}},k \right)={{\mathbf{X}}_{i}}$。

==================================================

2.3 HRNetV2和HRNetV1的區別

HRNetV1的最后一個階段，只有最高分辨率的特征作為輸出，如下圖a所示，這意味着最后一個階段只有高分辨率的特征會被利用，其他的低分辨率特征會被丟棄。HRNetV2最后一個階段，使用bilinear插值將低分辨率的特征上采樣到高分辨率，並拼接后作為最終的高分辨率特征，如下圖b所示。b的方式直接用於分割、人臉關鍵點定位。多於目標檢測，則將最后一個階段的高分辨率特征使用平均池化下采樣，得到多尺度特征，如下圖c所示，簡記為HRNet2p。

3 代碼

HighResolutionNet代碼如下：

  1 blocks_dict = {
  2     'BASIC': BasicBlock,
  3     'BOTTLENECK': Bottleneck
  4 }
  5 
  6 
  7 class HighResolutionNet(nn.Module):
  8 
  9     def __init__(self, config, **kwargs):
 10         self.inplanes = 64
 11         extra = config.MODEL.EXTRA
 12         super(HighResolutionNet, self).__init__()
 13 
 14         # stem net
 15         self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1, bias=False)    # 2個3*3的conv
 16         self.bn1 = BatchNorm2d(64, momentum=BN_MOMENTUM)
 17         self.conv2 = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1, bias=False)
 18         self.bn2 = BatchNorm2d(64, momentum=BN_MOMENTUM)
 19         self.relu = nn.ReLU(inplace=True)
 20         self.sf = nn.Softmax(dim=1)
 21 
 22         self.layer1 = self._make_layer(Bottleneck, 64, 64, 4)  # ResNet的第一個layer  # 輸入64，輸出256維度
 23 
 24         self.stage2_cfg = extra['STAGE2']
 25         num_channels = self.stage2_cfg['NUM_CHANNELS']   # 每個分辨率分支特征維數
 26         block = blocks_dict[self.stage2_cfg['BLOCK']]    # BasicBlock還是Bottleneck
 27         num_channels = [num_channels[i] * block.expansion for i in range(len(num_channels))]  # 每個分辨率分支特征維數
 28         self.transition1 = self._make_transition_layer([256], num_channels)   # self.layer1輸出為256維，因而此處為[256]
 29         self.stage2, pre_stage_channels = self._make_stage(self.stage2_cfg, num_channels)
 30 
 31         self.stage3_cfg = extra['STAGE3']
 32         num_channels = self.stage3_cfg['NUM_CHANNELS']
 33         block = blocks_dict[self.stage3_cfg['BLOCK']]
 34         num_channels = [num_channels[i] * block.expansion for i in range(len(num_channels))]
 35         self.transition2 = self._make_transition_layer(pre_stage_channels, num_channels)
 36         self.stage3, pre_stage_channels = self._make_stage(self.stage3_cfg, num_channels)
 37 
 38         self.stage4_cfg = extra['STAGE4']
 39         num_channels = self.stage4_cfg['NUM_CHANNELS']
 40         block = blocks_dict[self.stage4_cfg['BLOCK']]
 41         num_channels = [num_channels[i] * block.expansion for i in range(len(num_channels))]
 42         self.transition3 = self._make_transition_layer(pre_stage_channels, num_channels)
 43         self.stage4, pre_stage_channels = self._make_stage(self.stage4_cfg, num_channels, multi_scale_output=True)
 44 
 45         final_inp_channels = sum(pre_stage_channels)    # 拼接后最終的特征維數
 46 
 47         self.head = nn.Sequential(                      # 1*1 conv(維度不變) + BN + ReLU + 1*1 conv(降維)用於得到面部關鍵點數量的熱圖
 48             nn.Conv2d(in_channels=final_inp_channels, out_channels=final_inp_channels, kernel_size=1,
 49                 stride=1, padding=1 if extra.FINAL_CONV_KERNEL == 3 else 0),
 50             BatchNorm2d(final_inp_channels, momentum=BN_MOMENTUM),
 51             nn.ReLU(inplace=True),
 52             nn.Conv2d(in_channels=final_inp_channels, out_channels=config.MODEL.NUM_JOINTS,
 53                 kernel_size=extra.FINAL_CONV_KERNEL, stride=1, padding=1 if extra.FINAL_CONV_KERNEL == 3 else 0)
 54         )
 55 
 56     def _make_transition_layer(self, num_channels_pre_layer, num_channels_cur_layer):
 57         num_branches_cur = len(num_channels_cur_layer)  # 當前層的分辨率個數  num_channels_pre_layer, num_channels_cur_layer均為list
 58         num_branches_pre = len(num_channels_pre_layer)  # 之前層的分辨率個數  num_channels_pre_layer, num_channels_cur_layer均為list
 59 
 60         transition_layers = []
 61         for i in range(num_branches_cur):   # 依次遍歷當前層的每個分辨率
 62             if i < num_branches_pre:        # 之前層有當前子層的分辨率
 63                 if num_channels_cur_layer[i] != num_channels_pre_layer[i]:  # 當前子層的通道數量和之前子層的通道數量不同，stage2的transition1為16!=256
 64                     transition_layers.append(nn.Sequential(                 # 3*3 conv + BN + ReLU 用於維度匹配
 65                         nn.Conv2d(num_channels_pre_layer[i], num_channels_cur_layer[i], 3, 1, 1, bias=False),
 66                         BatchNorm2d(num_channels_cur_layer[i], momentum=BN_MOMENTUM),
 67                         nn.ReLU(inplace=True)))
 68                 else:
 69                     transition_layers.append(None)    # 當前子層的通道數量和之前子層的通道數量相同，給出的模型均這樣
 70             else:
 71                 conv3x3s = []
 72                 # 如果num_branches_cur=num_branches_pre+1，則一個[conv(num_channels_pre_layer[-1], num_channels_cur_layer[-1])_stride_2]
 73                 # 如果num_branches_cur=num_branches_pre+2，則（目前下面的情況均不存在）:
 74                 #     [  conv(pre[-1],cur[-1])_stride_2,                                    ，會出現當前分辨率寬高降低的情況
 75                 #        conv(pre[-1],pre[-1])_stride_2 + conv(pre[-1],cur[-1])_stride_2，  ] 會出現當前分辨率寬高降低的情況
 76                 # 如果num_branches_cur=num_branches_pre+3，則:
 77                 #     [  conv(pre[-1],cur[-1])_stride_2,                                    ，會出現當前分辨率寬高降低的情況
 78                 #       conv(pre[-1],pre[-1])_stride_2 + conv(pre[-1],cur[-1])_stride_2     ，會出現當前分辨率寬高降低的情況
 79                 #       conv(pre[-1],pre[-1])_stride_2 + conv(pre[-1],pre[-1])_stride_2 + conv(pre[-1],cur[-1])_stride_2，  ] 會出現當前分辨率寬高降低的情況
 80                 for j in range(i + 1 - num_branches_pre):      # stride=2的降維卷積
 81                     inchannels = num_channels_pre_layer[-1]
 82                     outchannels = num_channels_cur_layer[i] if j == i - num_branches_pre else inchannels
 83                     conv3x3s.append(nn.Sequential(
 84                         nn.Conv2d(inchannels, outchannels, 3, 2, 1, bias=False),
 85                         BatchNorm2d(outchannels, momentum=BN_MOMENTUM),
 86                         nn.ReLU(inplace=True)))
 87                 transition_layers.append(nn.Sequential(*conv3x3s))
 88 
 89         return nn.ModuleList(transition_layers)
 90 
 91     def _make_layer(self, block, inplanes, planes, blocks, stride=1):  # 輸入64，輸出256維度
 92         downsample = None
 93         if stride != 1 or inplanes != planes * block.expansion:
 94             downsample = nn.Sequential(                # 64==》64*4的1*1卷積，用於維度變換
 95                 nn.Conv2d(inplanes, planes * block.expansion, kernel_size=1, stride=stride, bias=False),
 96                 BatchNorm2d(planes * block.expansion, momentum=BN_MOMENTUM),
 97             )
 98 
 99         layers = []
100         layers.append(block(inplanes, planes, stride, downsample))  # 只有一個_make_layer，(64=>64)=>(64=>64)=>(64=>64*4)的1個Bottleneck
101         inplanes = planes * block.expansion
102         for i in range(1, blocks):
103             layers.append(block(inplanes, planes))     # (64*4=>64)=>(64=>64)=>(64=>64*4)的3個Bottleneck
104 
105         return nn.Sequential(*layers)
106 
107     def _make_stage(self, layer_config, num_inchannels, multi_scale_output=True):
108         num_modules = layer_config['NUM_MODULES']      # 當前層重復次數
109         num_branches = layer_config['NUM_BRANCHES']    # 當前層的分辨率分支數
110         num_blocks = layer_config['NUM_BLOCKS']        # 每個分辨率重復block的個數
111         num_channels = layer_config['NUM_CHANNELS']    # 當前層各分辨率通道個數
112         block = blocks_dict[layer_config['BLOCK']]     # BasicBlock還是Bottleneck
113         fuse_method = layer_config['FUSE_METHOD']      # SUM
114 
115         modules = []
116         for i in range(num_modules):   # 重復當前多分辨率分支的次數
117             # multi_scale_output is only used last module
118             if not multi_scale_output and i == num_modules - 1:  # multi_scale_output==True，HRNet V2中此處永遠不會執行
119                 reset_multi_scale_output = False
120             else:
121                 reset_multi_scale_output = True                  # HRNet V2中此處恆為True
122             modules.append(                                      # [num_inchannels] = [num_channels*block.expansion]
123                 HighResolutionModule(num_branches, block, num_blocks, num_inchannels,
124                                      num_channels, fuse_method, reset_multi_scale_output))
125             num_inchannels = modules[-1].get_num_inchannels()
126 
127         return nn.Sequential(*modules), num_inchannels
128 
129     def forward(self, x):
130         # h, w = x.size(2), x.size(3)
131         x = self.conv1(x)
132         x = self.bn1(x)
133         x = self.relu(x)
134         x = self.conv2(x)
135         x = self.bn2(x)
136         x = self.relu(x)
137         x = self.layer1(x)
138 
139         x_list = []
140         for i in range(self.stage2_cfg['NUM_BRANCHES']):   # NUM_BRANCHES為當前層的分辨率分支數
141             if self.transition1[i] is not None:            # stage2的transition1的兩個分辨率分支都不為None(一個維度匹配，一個降分辨率)
142                 x_list.append(self.transition1[i](x))
143             else:
144                 x_list.append(x)
145         y_list = self.stage2(x_list)                       # 在對應stage中會重復NUM_MODULES個HighResolutionModule
146 
147         x_list = []
148         for i in range(self.stage3_cfg['NUM_BRANCHES']):   # NUM_BRANCHES為當前層的分辨率分支數
149             if self.transition2[i] is not None:
150                 x_list.append(self.transition2[i](y_list[-1]))  # 將前一個分辨率的最后一個分支輸入當前transition_layer
151             else:
152                 x_list.append(y_list[i])                   # 前一個分辨率的其他分支直接輸入當前分辨率對應分支
153         y_list = self.stage3(x_list)                       # 在對應stage中會重復NUM_MODULES個HighResolutionModule
154 
155         x_list = []
156         for i in range(self.stage4_cfg['NUM_BRANCHES']):   # NUM_BRANCHES為當前層的分辨率分支數
157             if self.transition3[i] is not None:
158                 x_list.append(self.transition3[i](y_list[-1]))  # 將前一個分辨率的最后一個分支輸入當前transition_layer
159             else:
160                 x_list.append(y_list[i])                   # 前一個分辨率的其他分支直接輸入當前分辨率對應分支
161         x = self.stage4(x_list)                            # 在對應stage中會重復NUM_MODULES個HighResolutionModule
162 
163         # Head Part
164         height, width = x[0].size(2), x[0].size(3)         # 得到寬高
165         x1 = F.interpolate(x[1], size=(height, width), mode='bilinear', align_corners=False)  # 其他分辨率差值到x[0]寬高
166         x2 = F.interpolate(x[2], size=(height, width), mode='bilinear', align_corners=False)  # 其他分辨率差值到x[0]寬高
167         x3 = F.interpolate(x[3], size=(height, width), mode='bilinear', align_corners=False)  # 其他分辨率差值到x[0]寬高
168         x = torch.cat([x[0], x1, x2, x3], 1)      # 拼接特征
169         x = self.head(x)                          # 1*1 conv(維度不變) + BN + ReLU + 1*1 conv(降維)用於得到面部關鍵點數量的熱圖
170 
171         return x
172 
173     def init_weights(self, pretrained=''):
174         logger.info('=> init weights from normal distribution')
175         for m in self.modules():
176             if isinstance(m, nn.Conv2d):
177                 # nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
178                 nn.init.normal_(m.weight, std=0.001)
179                 # nn.init.constant_(m.bias, 0)
180             elif isinstance(m, nn.BatchNorm2d):
181                 nn.init.constant_(m.weight, 1)
182                 nn.init.constant_(m.bias, 0)
183         if os.path.isfile(pretrained):
184             pretrained_dict = torch.load(pretrained)
185             logger.info('=> loading pretrained model {}'.format(pretrained))
186             model_dict = self.state_dict()
187             pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict.keys()}
188             for k, _ in pretrained_dict.items():
189                 logger.info('=> loading {} pretrained model {}'.format(k, pretrained))
190             model_dict.update(pretrained_dict)
191             self.load_state_dict(model_dict)

View Code

上面extra = config.MODEL.EXTRA位於experiments/alfw（300w，cofw，wflw）/face_alignment_aflw_hrnet_w18.yaml中，如下：

 1 EXTRA:
 2     FINAL_CONV_KERNEL: 1
 3     STAGE2:
 4       NUM_MODULES: 1
 5       NUM_BRANCHES: 2
 6       BLOCK: BASIC
 7       NUM_BLOCKS:
 8         - 4
 9         - 4
10       NUM_CHANNELS:
11         - 18
12         - 36
13       FUSE_METHOD: SUM
14     STAGE3:
15       NUM_MODULES: 4
16       NUM_BRANCHES: 3
17       BLOCK: BASIC
18       NUM_BLOCKS:
19         - 4
20         - 4
21         - 4
22       NUM_CHANNELS:
23         - 18
24         - 36
25         - 72
26       FUSE_METHOD: SUM
27     STAGE4:
28       NUM_MODULES: 3
29       NUM_BRANCHES: 4
30       BLOCK: BASIC
31       NUM_BLOCKS:
32         - 4
33         - 4
34         - 4
35         - 4
36       NUM_CHANNELS:
37         - 18
38         - 36
39         - 72
40         - 144
41       FUSE_METHOD: SUM

View Code

_make_layer實際上和ResNet的layer一致。

_make_transition_layer結構如下：

HighResolutionModule如下：

  1 class HighResolutionModule(nn.Module):
  2     def __init__(self, num_branches, blocks, num_blocks, num_inchannels,
  3                  num_channels, fuse_method, multi_scale_output=True):
  4         '''     num_branches                分辨率分支個數
  5                 block                       BasicBlock還是Bottleneck
  6                 num_blocks                  重復block的個數
  7                 num_inchannels              [輸入通道個數]  # [num_inchannels] = [num_channels*block.expansion]
  8                 num_channels                [當前通道個數]  # num_channels=layer_config['NUM_CHANNELS']
  9                 fuse_method                 SUM
 10                 reset_multi_scale_output    multi_scale_output==True'''
 11         super(HighResolutionModule, self).__init__()
 12         self._check_branches(num_branches, blocks, num_blocks, num_inchannels, num_channels)
 13 
 14         self.num_inchannels = num_inchannels  # [self.num_inchannels] = [num_channels*block.expansion]
 15         self.fuse_method = fuse_method
 16         self.num_branches = num_branches
 17         self.multi_scale_output = multi_scale_output   # 恆為True
 18 
 19         self.branches = self._make_branches(num_branches, blocks, num_blocks, num_channels)
 20         self.fuse_layers = self._make_fuse_layers()
 21         self.relu = nn.ReLU(inplace=True)
 22 
 23     def _check_branches(self, num_branches, blocks, num_blocks, num_inchannels, num_channels):
 24         if num_branches != len(num_blocks):
 25             error_msg = 'NUM_BRANCHES({}) <> NUM_BLOCKS({})'.format(num_branches, len(num_blocks))
 26             logger.error(error_msg)
 27             raise ValueError(error_msg)
 28 
 29         if num_branches != len(num_channels):
 30             error_msg = 'NUM_BRANCHES({}) <> NUM_CHANNELS({})'.format(num_branches, len(num_channels))
 31             logger.error(error_msg)
 32             raise ValueError(error_msg)
 33 
 34         if num_branches != len(num_inchannels):
 35             error_msg = 'NUM_BRANCHES({}) <> NUM_INCHANNELS({})'.format(num_branches, len(num_inchannels))
 36             logger.error(error_msg)
 37             raise ValueError(error_msg)
 38 
 39     def _make_one_branch(self, branch_index, block, num_blocks, num_channels, stride=1):
 40         downsample = None  # BasicBlock時downsample=None  Bottleneck時進行維度匹配
 41         if stride != 1 or self.num_inchannels[branch_index] != num_channels[branch_index] * block.expansion:  # stride == 1
 42             downsample = nn.Sequential(           # [self.num_inchannels] = [num_channels*block.expansion]   1*1 conv維度匹配
 43                 nn.Conv2d(self.num_inchannels[branch_index], num_channels[branch_index] * block.expansion,
 44                           kernel_size=1, stride=stride, bias=False),
 45                 BatchNorm2d(num_channels[branch_index] * block.expansion, momentum=BN_MOMENTUM),
 46             )
 47 
 48         layers = []
 49         # BasicBlock：(num_channels*block.expansion=>num_channels) => (num_channels=>num_channels*block.expansion)  # block.expansion=1
 50         # Bottleneck：(num_channels*block.expansion=>num_channels) => (num_channels=>num_channels) =>
 51         #             (num_channels=>num_channels*block.expansion)                                                  # block.expansion=4
 52         layers.append(block(self.num_inchannels[branch_index], num_channels[branch_index], stride, downsample))     # 1個維度匹配的block
 53         # 經過上面block，輸出維度可能變化了，因而此處更新接下來block的輸入維度
 54         self.num_inchannels[branch_index] = num_channels[branch_index] * block.expansion
 55         for i in range(1, num_blocks[branch_index]):                                                                # 3個維度不變的block
 56             layers.append(block(self.num_inchannels[branch_index], num_channels[branch_index]))                # 此處輸入維度和輸出維度一樣
 57 
 58         return nn.Sequential(*layers)
 59 
 60     def _make_branches(self, num_branches, block, num_blocks, num_channels):
 61         branches = []
 62         for i in range(num_branches):   # 依次將不同分辨率分支append到branches中
 63             branches.append(self._make_one_branch(i, block, num_blocks, num_channels))                             # 依次增加一個分辨率分支
 64 
 65         return nn.ModuleList(branches)
 66 
 67     def _make_fuse_layers(self):
 68         # stage 3，3個branch，BasicBlock時，num_inchannels=[18, 36, 72]
 69         #  i   j   k   fuse_layer
 70         #
 71         #  0   0   x   None
 72         #  0   1   x   1*1_conv(36, 18) + BN                          # M1
 73         #  0   2   x   1*1_conv(72, 18) + BN                          # M2
 74         #
 75         #  1   0   0   3*3_conv(18, 36)_stride_2 + BN                 # M3
 76         #  1   1   x   None
 77         #  1   2   x   1*1_conv(72, 36) + BN                          # M4
 78         #
 79         #  2   0   0   3*3_conv(18, 72)_stride_2 + BN + ReLU          # M5
 80         #  2   0   1   3*3_conv(18, 72)_stride_2 + BN                 # M6
 81         #  2   1   0   3*3_conv(36, 72)_stride_2 + BN                 # M7
 82         #  2   2   x   None
 83 
 84         if self.num_branches == 1:   # 分辨率分支個數，目前不會出現1，此處不執行
 85             return None
 86 
 87         num_branches = self.num_branches         # 分辨率分支個數
 88         num_inchannels = self.num_inchannels     # [輸入通道個數]    # [self.num_inchannels] = [num_channels*block.expansion]
 89         fuse_layers = []
 90         for i in range(num_branches if self.multi_scale_output else 1):   # self.multi_scale_output==True，for i in range(num_branches）
 91             fuse_layer = []
 92             for j in range(num_branches):
 93                 if j > i:
 94                     fuse_layer.append(nn.Sequential(
 95                         nn.Conv2d(num_inchannels[j], num_inchannels[i], 1, 1, 0, bias=False),
 96                         BatchNorm2d(num_inchannels[i], momentum=BN_MOMENTUM)))
 97                     # nn.Upsample(scale_factor=2**(j-i), mode='nearest')))
 98                 elif j == i:
 99                     fuse_layer.append(None)
100                 else:
101                     conv3x3s = []
102                     for k in range(i - j):
103                         if k == i - j - 1:
104                             num_outchannels_conv3x3 = num_inchannels[i]
105                             conv3x3s.append(nn.Sequential(
106                                 nn.Conv2d(num_inchannels[j], num_outchannels_conv3x3, 3, 2, 1, bias=False),
107                                 BatchNorm2d(num_outchannels_conv3x3, momentum=BN_MOMENTUM)))
108                         else:
109                             num_outchannels_conv3x3 = num_inchannels[j]
110                             conv3x3s.append(nn.Sequential(
111                                 nn.Conv2d(num_inchannels[j], num_outchannels_conv3x3, 3, 2, 1, bias=False),
112                                 BatchNorm2d(num_outchannels_conv3x3, momentum=BN_MOMENTUM),
113                                 nn.ReLU(inplace=True)))
114                     fuse_layer.append(nn.Sequential(*conv3x3s))
115             fuse_layers.append(nn.ModuleList(fuse_layer))
116 
117         return nn.ModuleList(fuse_layers)
118 
119     def get_num_inchannels(self):
120         return self.num_inchannels     # 最終輸出的維度就是這個
121 
122     def forward(self, x):
123         if self.num_branches == 1:    # 分辨率分支個數，目前不會出現1，此處不執行
124             return [self.branches[0](x[0])]  # self.branches和x均為list，因而此處取self.branches[0]和x[0]
125 
126         for i in range(self.num_branches):   # 將x[i]輸入self.branches[i]，得到第i個分辨率分支的輸出x[i]
127             x[i] = self.branches[i](x[i])
128 
129         # stage 3，3個branch，BasicBlock時，num_inchannels=num_inchannels=[18, 36, 72]
130         # 下面的x[i]均為上面 第i個分辨率分支的輸出x[i]。最終每個分辨率的輸出，均是x[0]、x[1]、x[2]融合的結果(下面的三個輸出)
131         #  i    y1               j    y
132         #
133         #  0    x[0]             1    x[0] + interpolate(M1(x[1]))                                 中間
134         #  0    x[0]             2    ReLU(x[0] + interpolate(M1(x[1])) + interpolate(M2(x[2])))   輸出
135         #
136         #  1    M3(x[0])         1    M3(x[0]) + x[1]                                              中間
137         #  1    M3(x[0])         2    ReLU(M3(x[0]) + x[1] + interpolate(M4(x[2])))                輸出
138         #
139         #  2    M6(M5(x[0]))     0    M6(M5(x[0])) + M7(x[1])                                      中間
140         #  2    M6(M5(x[0]))     1    ReLU(M6(M5(x[0])) + M7(x[1]) + x[2])                         輸出
141         x_fuse = []
142         for i in range(len(self.fuse_layers)):  # 依次遍歷每個分辨率分支，進行融合
143             y = x[0] if i == 0 else self.fuse_layers[i][0](x[0])         # y1
144             for j in range(1, self.num_branches):
145                 if i == j:
146                     y = y + x[j]                                         # y 中間
147                 elif j > i:
148                     y = y + F.interpolate(self.fuse_layers[i][j](x[j]),  # y 中間
149                                           size=[x[i].shape[2], x[i].shape[3]], mode='bilinear')
150                 else:
151                     y = y + self.fuse_layers[i][j](x[j])                 # y 中間
152             x_fuse.append(self.relu(y))                                  # y 輸出
153 
154         return x_fuse