Pytorch GRU/LSTM 權重參數初始化

pytorch模型訓練表現不佳, 很有可能是參數初始化的問題

GRU weights采用正交初始化, bias采用0初始化

       self.gru = nn.GRU(10, 20, 2, dropout=0.2, bidirectional=True)
        # use orthogonal init for GRU layer0 weights
        weight_init.orthogonal(self.gru.weight_ih_l0)
        weight_init.orthogonal(self.gru.weight_hh_l0)
        # use zero init for GRU layer0 bias
        self.gru.bias_ih_l0.zero_()
        self.gru.bias_hh_l0.zero_()

　

self.gru.bias_ih_l0.zero_()

--> RunTime Error： A leaf Variable that requires grad has been used in an inplace operation. ???

>>> self.gru.bias_ih_l0.is_leaf
>>> True

>>> import torch.nn.parameter.Parameter as parameter

>>> self.gru.bias_ih_l0 = parameter(torch.zeros_like(self.gru.bias_ih_l0))