EcoVideo

EcoVideo is the open-source inference path for frame-level entropy-orchestrated cloud-edge video generation. The released code keeps the main text-to-video pipeline:

scripts/run_full_pipeline.py
  -> vdit.generators.wan_t2v / vdit.generators.cogvideo_t2v
  -> third_party/wan21 or third_party/wan22
  -> vdit.pipeline.run_iframe
  -> EDEN interpolation + interval scoring + greedy refinement
  -> final video

Demo

Generated by EcoVideo (Wan2.1-T2V-14B + EDEN interpolation):

demo_1.mp4	demo_2.mp4	demo_3.mp4
demo_4.mp4	demo_5.mp4	demo_6.mp4

To view the corresponding prompt words for the gallery, please click here

Features

• Attention-entropy-based keyframe selection.
• Optional non-keyframe context for temporally consistent keyframe denoising.
• EDEN-based interpolation from sparse keyframes to target FPS.
• Optional RAFT-based motion/occlusion interval scoring.
• Wan2.1, Wan2.2 and CogVideoX inference backends.

Tested environment

• Python 3.10
• PyTorch installed separately according to CUDA version
• CUDA-capable GPU for full generation
• pip install -r requirements/base.txt for EcoVideo runtime dependencies

Installation

conda create -n ecovideo python=3.10 -y
conda activate ecovideo

# Choose the PyTorch command that matches your CUDA driver.
# Example for CUDA 12.1 wheels:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

pip install -r requirements/base.txt

# Optional: install xformers only if a compatible wheel exists for your torch/CUDA.
# pip install xformers

export PYTHONPATH=$PWD/src:$PYTHONPATH

Check the environment:

PYTHONPATH=src python tools/check_env.py

Model preparation

Download the pretrained model checkpoints from HuggingFace:

Model	HuggingFace Link
CogVideoX-5B	THUDM/CogVideoX-5b
CogVideoX-2B	THUDM/CogVideoX-2b
Wan2.1-T2V-14B	Wan-AI/Wan2.1-T2V-14B
Wan2.1-T2V-1.3B	Wan-AI/Wan2.1-T2V-1.3B
Wan2.2-T2V-A14B	Wan-AI/Wan2.2-T2V-A14B

You can download them via huggingface-cli:

pip install "huggingface_hub[cli]"

huggingface-cli download THUDM/CogVideoX-5b --local-dir ./CogVideoX-5b
huggingface-cli download THUDM/CogVideoX-2b --local-dir ./CogVideoX-2b
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B
huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir ./Wan2.1-T2V-1.3B
huggingface-cli download Wan-AI/Wan2.2-T2V-A14B --local-dir ./Wan2.2-T2V-A14B

Prepare checkpoints and export paths:

export WAN21_CKPT=/path/to/Wan2.1-T2V-14B
export WAN22_CKPT=/path/to/Wan2.2-T2V-A14B
export COGVIDEO_CKPT=/path/to/CogVideoX-5b
export EDEN_CKPT=/path/to/eden.pt
export RAFT_CKPT=/path/to/raft-things.pth   # optional

configs/eden_infer.yaml is a template. The helper below creates a local config with the correct EDEN checkpoint path:

python tools/make_eden_config.py \
  --template configs/eden_infer.yaml \
  --eden_ckpt "$EDEN_CKPT" \
  --output outputs/eden_infer.local.yaml

Quick start

Wan2.1

WAN21_CKPT=/path/to/Wan2.1-T2V-14B \
EDEN_CKPT=/path/to/eden.pt \
RAFT_CKPT=/path/to/raft-things.pth \
bash scripts/infer_wan21.sh

Wan2.2

WAN22_CKPT=/path/to/Wan2.2-T2V-A14B \
EDEN_CKPT=/path/to/eden.pt \
RAFT_CKPT=/path/to/raft-things.pth \
bash scripts/infer_wan22.sh

CogVideoX

COGVIDEO_CKPT=/path/to/CogVideoX-5b \
EDEN_CKPT=/path/to/eden.pt \
RAFT_CKPT=/path/to/raft-things.pth \
bash scripts/infer_cogvideo.sh

If RAFT_CKPT is omitted, RAFT-based flow/occlusion scores are disabled. RGB and EDEN-difference scores still run.

Manual command example

PYTHONPATH=src python scripts/run_full_pipeline.py \
  --generator wan \
  --wan_version 2.1 \
  --ckpt_dir /path/to/Wan2.1-T2V-14B \
  --prompt "A boat sailing smoothly on a calm lake." \
  --wan_task t2v-14B \
  --wan_size "1280*720" \
  --wan_frame_num 81 \
  --wan_sample_steps 50 \
  --wan_sample_solver unipc \
  --wan_guide_scale 5.0 \
  --wan_keyframe_by_entropy \
  --wan_entropy_steps 5 \
  --wan_entropy_mode ema \
  --wan_keyframe_target_fps 8 \
  --wan_use_nonkey_context \
  --eden_config outputs/eden_infer.local.yaml \
  --raft_ckpt /path/to/raft-things.pth \
  --target_fps 24 \
  --keyframe_mode all \
  --output_path outputs/wan21_ecovideo.mp4

Reproducing paper-style results

The full paper reproduction requires the exact released checkpoints, prompts, hardware, and evaluation scripts. The minimum reproducible workflow is:

1. Generate sparse keyframes with entropy-based selection.
2. Run EDEN interpolation to the target FPS.
3. Save *.metrics.json for latency and pipeline statistics.
4. Evaluate generated videos with VBench or the metric suite used in the paper.

Example commands are provided in scripts/infer_wan21.sh, scripts/infer_wan22.sh, and scripts/infer_cogvideo.sh.

Evaluation

We use VBench for video generation quality evaluation.

Install evaluation dependencies

pip install -r requirements/eval.txt

# Optional: required by some VBench dimensions (e.g., object_class)
pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

First generate videos using the inference scripts (see Quick start), then evaluate:

python evaluation/eval_vbench.py \
  --videos_path outputs/my_videos \
  --prompt_file examples/prompts.txt \
  --output_dir evaluation_results \
  --name ecovideo_wan21

Or use the shell script:

VIDEOS_PATH=outputs/my_videos \
PROMPT_FILE=examples/prompts.txt \
NAME=ecovideo_wan21 \
bash evaluation/eval_script.sh

Evaluation options

Argument	Description
--videos_path	Path to folder containing generated videos
--prompt_file	Text file (one prompt per line) or JSON dict {video_filename: prompt}
--dimension_list	VBench dimensions to evaluate (default: all custom-input supported)
--output_dir	Directory to save results (default: evaluation_results)
--device	Device for evaluation (default: cuda)
--name	Name tag for this evaluation run (default: ecovideo)

Results are saved as JSON in --output_dir and printed as a summary table to stdout.

Code structure

EcoVideo/
  assets/                  # pipeline and qualitative figures
  configs/                 # EDEN config template
  docs/                    # installation, reproduction, troubleshooting notes
  evaluation/              # VBench evaluation scripts
  examples/                # example prompts
  requirements/            # dependency groups
  scripts/                 # public inference scripts
  src/vdit/                # EcoVideo pipeline code
  third_party/             # vendored Wan/RAFT components
  tools/                   # release/environment helper scripts

Citation

If you find this project useful, please cite our paper:

@article{chen2025ecovideo,
  title   = {EcoVideo: Entropy-Orchestrated Video Generation Paradigm in Cloud-Edge Dynamics},
  author  = {Jiayu Chen and Hengyi Zhang and Maoliang Li and Minyu Li and Zihao Zheng and Xuanzhe Liu and Guojie Luo and Xiang Chen},
  booktitle={European Conference on Computer Vision (ECCV)},
  year    = {2026},
}

License

EcoVideo original code is released under the Apache License 2.0.

Third-party code under third_party/ remains subject to their respective upstream licenses. See NOTICE for details.