|
|
发表于 2025-12-12 11:19:54
手机版
|
显示全部楼层
连接主机...
连接主机成功
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.6.87.2-microsoft-standard-WSL2 x86_64)
• Documentation: https://help.ubuntu.com
• Management: https://landscape.canonical.com
• Support: https://ubuntu.com/pro
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.
To restore this content, you can run the 'unminimize' command.
Last login: Fri Dec 12 02:25:44 2025 from 172.18.0.1
(diffusion-pipe) root@1d615dc8ebd0:~# deepspeed --num_gpus=1 diffusion-pipe/train.py --deepspeed --config diffusion-pipe/examples/main_FLUX.1-dev.toml
[2025-12-12 03:17:03,522] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-12-12 03:17:07,168] [WARNING] [runner.py:215:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2025-12-12 03:17:07,169] [INFO] [runner.py:605:main] cmd = /root/miniconda3/envs/diffusion-pipe/bin/python3.12 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None diffusion-pipe/train.py --deepspeed --config diffusion-pipe/examples/main_FLUX.1-dev.toml
[2025-12-12 03:17:08,413] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-12-12 03:17:10,671] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0]}
[2025-12-12 03:17:10,671] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=1, node_rank=0
[2025-12-12 03:17:10,671] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2025-12-12 03:17:10,671] [INFO] [launch.py:164:main] dist_world_size=1
[2025-12-12 03:17:10,671] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0
[2025-12-12 03:17:10,672] [INFO] [launch.py:256:main] process 35732 spawned with command: ['/root/miniconda3/envs/diffusion-pipe/bin/python3.12', '-u', 'diffusion-pipe/train.py', '--local_rank=0', '--deepspeed', '--config', 'diffusion-pipe/examples/main_FLUX.1-dev.toml']
[2025-12-12 03:17:13,140] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-12-12 03:17:16,324] [INFO] [comm.py:675:init_distributed] cdb=None
[2025-12-12 03:17:16,324] [INFO] [comm.py:706:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[rank0]: Traceback (most recent call last):
[rank0]: File "/root/diffusion-pipe/train.py", line 303, in <module>
[rank0]: model = flux.FluxPipeline(config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/diffusion-pipe/models/flux.py", line 183, in __init__
[rank0]: transformer = diffusers.FluxTransformer2DModel.from_pretrained(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/diffusers/models/modeling_utils.py", line 1179, in from_pretrained
[rank0]: resolved_model_file, sharded_metadata = _get_checkpoint_shard_files(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/diffusers/utils/hub_utils.py", line 393, in _get_checkpoint_shard_files
[rank0]: raise FileNotFoundError(
[rank0]: FileNotFoundError: /root/diffusion/models/FLUX.1-dev/transformer/ does not appear to have a file named /root/diffusion/models/FLUX.1-dev/transformer/diffusion_pytorch_model-00001-of-00003.safetensors which is required according to the checkpoint index.
[rank0]:[W1212 03:17:18.789442383 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[2025-12-12 03:17:20,674] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 35732
[2025-12-12 03:17:20,674] [ERROR] [launch.py:325:sigkill_handler] ['/root/miniconda3/envs/diffusion-pipe/bin/python3.12', '-u', 'diffusion-pipe/train.py', '--local_rank=0', '--deepspeed', '--config', 'diffusion-pipe/examples/main_FLUX.1-dev.toml'] exits with return code = 1 |
|