on_policy_distillation训练不稳定

[Jim2016713]

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)
训练过程非常不稳定，W1218 12:19:35.164000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7756 via 1, forcefully exiting via 9
W1218 12:19:35.824000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7757 via 1, forcefully exiting via 9
W1218 12:19:36.496000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7759 via 1, forcefully exiting via 9
W1218 12:19:37.204000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7760 via 1, forcefully exiting via 9
W1218 12:19:37.790000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7761 via 1, forcefully exiting via 9
W1218 12:19:38.483000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7762 via 1, forcefully exiting via 9
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 905, in
main()
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 357, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 143, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 715, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 879, in _invoke_run
time.sleep(monitor_interval)
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 7689 got signal: 1

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)

Additional context
Add any other context about the problem here(在这里补充其他信息)

[hjh0119]

Please provide more information, such as a reproduction script, full error stack trace, and environment details

[Jim2016713]

Please provide more information, such as a reproduction script, full error stack trace, and environment details
reproduction script:

source activate swift
pip install ./code/flash_attn-2.8.1+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
pip install deepspeed
pip install msgspec
unset https_proxy
unset http_proxy

pip install vllm

CUDA_VISIBLE_DEVICES=7 \

swift rollout \

--model Qwen/Qwen3-8B-Base \

--vllm_max_model_len 24192

STUDENT_MDOEL=./model/Qwen3-8B-Base
TEACHER_MODEL=./model_hub/Qwen/Qwen3-32B
NPROC_PER_NODE=7
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6
swift rlhf
--rlhf_type gkd
--model $STUDENT_MDOEL
--teacher_model $TEACHER_MODEL
--train_type full
--dataset ./data/OpenThoughts3-1.2M/data \
--seq_kd false
--lmbda 1
--beta 1
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 1
--learning_rate 1e-5
--gradient_accumulation_steps 1
--save_steps 100
--save_total_limit 2
--logging_steps 1
--max_length 16000
--max_completion_length 8192
--output_dir output
--warmup_ratio 0.05
--save_only_model true
--dataloader_num_workers 64
--dataset_num_proc 4
--deepspeed zero2
--teacher_deepspeed zero3
--attn_impl flash_attn
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8000

error stack:

Train: 1%| | 14/1429 [16:32<28:07:10, 71.54s/it]
Train: 1%| | 15/1429 [16:55<22:23:07, 56.99s/it]

{'loss': 0.60909599, 'grad_norm': 10.63460827, 'learning_rate': 2.08e-06, 'epoch': 0.01, 'global_step/max_steps': '15/1429', 'percentage': '1.05%', 'elapsed_time': '16m 55s', 'remaining_time': '1d 2h 36m 11s', 'memory(GiB)': 70.42, 'train_speed(iter/s)': 0.014764}

Train: 1%| | 15/1429 [16:56<22:23:07, 56.99s/it]
Train: 1%| | 15/1429 [16:56<22:23:07, 56.99s/it]
Train: 1%| | 16/1429 [18:50<29:14:36, 74.51s/it]

{'loss': 0.30126953, 'grad_norm': 6.76516056, 'learning_rate': 2.22e-06, 'epoch': 0.01, 'global_step/max_steps': '16/1429', 'percentage': '1.12%', 'elapsed_time': '18m 51s', 'remaining_time': '1d 3h 45m 6s', 'memory(GiB)': 70.42, 'train_speed(iter/s)': 0.014143}

Train: 1%| | 16/1429 [18:51<29:14:36, 74.51s/it]
Train: 1%| | 16/1429 [18:51<29:14:36, 74.51s/it]W1218 12:19:04.366000 7689 site-packages/torch/distributed/elastic/agent/server/api.py:723] Received 1 death signal, shutting down workers
W1218 12:19:04.369000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7755 closing signal SIGHUP
W1218 12:19:04.370000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7756 closing signal SIGHUP
W1218 12:19:04.370000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7757 closing signal SIGHUP
W1218 12:19:04.371000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7759 closing signal SIGHUP
W1218 12:19:04.372000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7760 closing signal SIGHUP
W1218 12:19:04.372000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7761 closing signal SIGHUP
W1218 12:19:04.373000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 7762 closing signal SIGHUP

Train: 1%| | 17/1429 [19:18<23:40:50, 60.38s/it]

{'loss': 0.25760326, 'grad_norm': 5.22085428, 'learning_rate': 2.36e-06, 'epoch': 0.01, 'global_step/max_steps': '17/1429', 'percentage': '1.19%', 'elapsed_time': '19m 18s', 'remaining_time': '1d 2h 43m 49s', 'memory(GiB)': 70.42, 'train_speed(iter/s)': 0.014673}

Train: 1%| | 17/1429 [19:18<23:40:50, 60.38s/it]
Train: 1%| | 17/1429 [19:18<23:40:50, 60.38s/it]W1218 12:19:34.374000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7755 via 1, forcefully exiting via 9
W1218 12:19:35.164000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7756 via 1, forcefully exiting via 9
W1218 12:19:35.824000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7757 via 1, forcefully exiting via 9
W1218 12:19:36.496000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7759 via 1, forcefully exiting via 9
W1218 12:19:37.204000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7760 via 1, forcefully exiting via 9
W1218 12:19:37.790000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7761 via 1, forcefully exiting via 9
W1218 12:19:38.483000 7689 site-packages/torch/distributed/elastic/multiprocessing/api.py:919] Unable to shutdown process 7762 via 1, forcefully exiting via 9
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 905, in
main()
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 357, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 143, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 715, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 879, in _invoke_run
time.sleep(monitor_interval)
File "/opt/miniconda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 7689 got signal: 1

environment details:
GPU:A100-80G
ms_swift : 3.12.0.dev0
python：3.12

base image:nvcr.io/nvidia/pytorch:24.08-py3@sha256:60cc7c9cf24b423c28f621287779ebe5e52cd39cfd284df04983df52566c1e39