Skip to content

[RFC]: Command "dreamverse-server --host 0.0.0.0 --port 8009": GPU 0 warmup failed #1685

Description

@nanwanj

Motivation.

We followed https://github.com/hao-ai-lab/FastVideo/blob/main/apps/dreamverse/README.md#quick-start-local-gpu to start dreamverse server on a B200 vm.

when running command "dreamverse-server --host 0.0.0.0 --port 8009", there will be error like:

1;36m(Worker pid=418572)�[0;0m INFO 08-06 00:16:34.816 [lora_pipeline.py:252] Converting transformer to LoRA Transformer
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584] WorkerMultiprocProc failed.
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584] Traceback (most recent call last):
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]   File "/home/ubuntu/FastVideo/fastvideo/worker/multiproc_executor.py", line 566, in worker_main
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]     worker.worker_busy_loop()
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]   File "/home/ubuntu/FastVideo/fastvideo/worker/multiproc_executor.py", line 692, in worker_busy_loop
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]     output_batch = self.worker.execute_forward(forward_batch, fastvideo_args)
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]   File "/home/ubuntu/FastVideo/fastvideo/worker/gpu_worker.py", line 75, in execute_forward
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]     output_batch = self.pipeline.forward(forward_batch, self.fastvideo_args)
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]   File "/home/ubuntu/FastVideo/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]     return func(*args, **kwargs)
�[1;36m(Worker pid=418572)�[0;0m ERROR 08-06 00:16:34.820 [multiproc_executor.py:584]            ^^^^^^^^^^^^^^^^^^^^^

Full log will be attached.

log.txt

Proposed Change.

Can you please help and let's know what's going on here?

Thanks.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    installationInstallation and setup issuesscope: attentionAttention backends (VSA, STA, Flash, etc.)scope: inferenceInference pipeline, serving, CLIscope: trainingTraining pipeline, methods, configs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions