首先在ctrl+c后出現這些錯誤
訓練后卡在
torch.distributed.init_process_group(backend='nccl', init_method='env://',world_size=2, rank=args.local_rank)
這句之前，使用ctrl+c后出現
torch.distributed.elastic.multiprocessing.api.SignalException: Process 214426 got signal: 2
torch.distributed.elastic.multiprocessing.api.SignalException: Process 214426 got signal: 2
:torch.distributed.elastic.agent.server.api:Received 2 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 214465 closing signal SIGINT
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 214466 closing signal SIGINT
^CWARNING:torch.distributed.elastic.multiprocessing.api:Sending process 214465 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 214466 closing signal SIGTERM
解決
網上都是說添加
torch.distributed.init_process_group(backend='nccl', init_method='env://',world_size=2, rank=args.local_rank)
os.environ['MASTER_ADDR'] = '127.0.0.1'
# os.environ['MASTER_PORT'] = '62222'#一機多卡不需要這個，有這個會導致不能開始
但是實際上一機多卡不需要'MASTER_PORT'這個變量注釋掉就可以，或者使用另外一個初始化方法
torch.distributed.init_process_group(backend='gloo', init_method='file:///home/user/switch.txt',world_size=2, rank=args.local_rank)
使用這個初始化方法不需要網絡主機等環境變量