pytorch在損失函數中為權重添加L1正則化

L1正則化可以使權重變稀疏，應用場景：對one-hot詞袋模型中的詞表進行裁剪時，根據權重weight篩選，此時需要權重越稀疏越好；

L1_Weight為超參數，可設定為1e-4

def train(model, iterator, optimizer, criteon):
    avg_acc, avg_loss = [], []
    model.train()     

    for batch in tqdm(iterator):
        text, label = batch[0].cuda(), batch[1].cuda()         

        pred = model(text)    
        l1_penalty = L1_Weight * sum([p.abs().sum() for p in model.fc.parameters()])
        loss = criteon(pred, label.long())        
        loss_with_penalty = loss + l1_penalty

        acc = utils.binary_acc(torch.argmax(pred.cpu(), dim=1), label.cpu().long())  
        avg_acc.append(acc)
        avg_loss.append(loss.item())

        optimizer.zero_grad()
        loss_with_penalty.backward()
        #loss.backward()
        optimizer.step()

    avg_acc = np.array(avg_acc).mean()
    avg_loss = np.array(avg_loss).mean()
    train_metrics = {'train_acc': avg_acc,
                     'train_loss': avg_loss
                     }
    logging.info(train_metrics)
    return avg_acc, avg_loss