VGen

Use this repo skill when a task concerns the VGen repository, its YAML-driven video generation launchers, or one of its named model families: ModelScope text-to-video, I2VGen-XL, VideoComposer-style conditioning, HiGen, TF-T2V, InstructVideo, DreamVideo, VideoLCM, or SR600.

Do not use this skill for generic Diffusers usage, ordinary image classification/detection/segmentation, unrelated video editing, or pure deployment tasks that do not involve VGen configs, checkpoints, data lists, or runtime modules.

First checks

1. Read references/repo-provenance.md before deciding whether this skill is current for a checkout.
2. Confirm the task's workflow family and route to the smallest matching sub-skill.
3. Check the target YAML's TASK_TYPE; VGen dispatch is config-driven.
4. Validate prompt/data lists before a long GPU run.
5. Confirm CUDA, checkpoints, ffmpeg, and optional workflow dependencies are present.

A minimal preflight from this skill root:

python scripts/check_runtime.py --repo-root /path/to/VGen --require-cuda
python scripts/dispatch_config.py --repo-root /path/to/VGen --dry-run --cfg configs/t2v_infer.yaml

Setup

A typical local setup is:

# install a CUDA-matched PyTorch wheel first, then the repo requirements
pip install -r requirements.txt
# InstructVideo reward fine-tuning also needs the extra list in configs/instructvideo/requirements.txt
pip install -r configs/instructvideo/requirements.txt

If you are matching the legacy README environment exactly, follow the repository README's Python 3.8 / Torch 1.12 CUDA notes instead of mixing package versions. No editable install is required; the bundled helpers assume the checkout itself is on disk and can be pointed to with --repo-root.

Sub-skill routing

• sub-skills/text-to-video/: ModelScope T2V, HiGen, TF-T2V, VideoLCM, VideoComposer-style conditioning, SR600 upscaling, generic train_net.py / inference.py config dispatch, T2V dataset/model smoke helpers.
• sub-skills/image-to-video/: I2VGen-XL image-to-video inference, image|||caption input lists, person-specialized I2VGen config, Cog predictor and Gradio demo as optional/reference-only surfaces.
• sub-skills/dreamvideo/: DreamVideo subject learning, motion learning, joint subject+motion inference, adapter-key inspection, and DreamVideo metric calculation.
• sub-skills/instructvideo/: InstructVideo reward fine-tuning, HPSv2 reward setup, LoRA/base evaluation presets, and WebVid reward-list preparation.

Shared references

• references/overview.md: route map, workflow families, included/excluded evidence, and shared helper index.
• references/configuration.md: Config, TASK_TYPE, registry dispatch, _BASE/vldm_cfg layering, and CLI override caveats.
• references/data-formats.md: prompt-only, path-caption, VideoComposer-style, DreamVideo metric, and InstructVideo list schemas.
• references/troubleshooting.md: cross-cutting CUDA, ffmpeg, OpenCV/NumPy, registry, checkpoint, and list-file failures.
• references/repo-routing-metadata.json: structured router metadata for managed repo-skill import.

Shared scripts

• scripts/check_runtime.py: import, CUDA, ffmpeg, and heavy tools registration preflight.
• scripts/dispatch_config.py: safe dry-run or execution wrapper for VGen YAMLs.
• scripts/inspect_list_file.py: list-file validator for common VGen schemas.
• scripts/dump_unet_key_sets.py: root-level alias for the DreamVideo temporal/spatial UNet key exporter.

Environment guidance

The README documents a legacy Python 3.8 / Torch 1.12 CUDA 11.3 setup and also expects ffmpeg. Current checkouts may need a modern CUDA PyTorch stack compatible with xformers, open-clip-torch, fairscale, diffusers, transformers, OpenCV, and reward extras such as piq and scikit-image. Treat CPU-only imports as insufficient for generation, customization, reward fine-tuning, and metric proof.

Operating guardrails

• Prefer copied YAML edits for numeric, boolean, list, dict, and nested config changes; keep positional CLI overrides for string-safe paths.
• Do not run demo wrappers (gradio_app.py, predict.py) unless the user explicitly wants ModelScope/Gradio/Cog deployment and accepts network/deployment dependencies.
• Do not silently switch workflow families when checkpoints or list files are missing.
• Keep generated previews, run logs, model downloads, and review/test artifacts outside this runtime skill tree.