Full Examples

[deepbeepmeep]

Hi

Thank you for very interesting work.

I am currently working on a low VRAM implementation of VAP for WanGP (https://github.com/deepbeepmeep/Wan2GP) a low VRAM Wan, Qwen, Flux ... Video Gen.

In order to be able to check if my code works properly I would be grateful if you could answer these questions :

• how should we choose the reference prompt versus the target prompt ? it seems when one tries to copy an effect in the reference video both the ref prompt and target will be similar. If that's the case why do we need two prompts ?
• the video prompt seems to have much less weight than the target video text prompt in the output result : I have tried to use a completely unrelated video to the target text prompt as a reference video and still got the desired output described in the text prompt. A general guide on how to pick the video and text would be greatly appreciated
• can we use a number frames higher than 49 ?

[yxbian23]

Hi

Thank you for very interesting work.

I am currently working on a low VRAM implementation of VAP for WanGP (https://github.com/deepbeepmeep/Wan2GP) a low VRAM Wan, Qwen, Flux ... Video Gen.

In order to be able to check if my code works properly I would be grateful if you could answer these questions :

• how should we choose the reference prompt versus the target prompt ? it seems when one tries to copy an effect in the reference video both the ref prompt and target will be similar. If that's the case why do we need two prompts ?
• the video prompt seems to have much less weight than the target video text prompt in the output result : I have tried to use a completely unrelated video to the target text prompt as a reference video and still got the desired output described in the text prompt. A general guide on how to pick the video and text would be greatly appreciated
• can we use a number frames higher than 49 ?

Thanks so much for your interest and for working on a low-VRAM adaptation — that’s really cool to hear! 🚀
I’ll try to clarify how to think about prompts and reference video selection:

1. How to choose the reference vs. target prompts

In order to preserve the original capabilities of the underlying text-to-video backbone, we keep the same style of video caption prompts that the base models (Wan, CogVideoX, Flux, etc.) were trained on.
This typically means describing: Who / what is in the scene? What motion or action is happening? Camera setting.Environment or style cues.

So:

• Reference prompt → describes the existing reference video, including its appearance of the subject and the environment + the motion or stylistic aspect you want to transfer.

• Target prompt → describes the new video you want to generate, replacing only the subject / setting with your new ones, while keeping the semantic structure similar.

2. The roles of the video prompts and the text prompts

During training, the text prompt and video prompt were always used together.
Our observations suggest something like:

1. Text prompt establishes the high-level semantic intent (e.g., “transfer motion”, “apply stylization”, “change identity”, etc.)

2. The model then looks into the reference video to find the corresponding structure and carries that over.

So, if the reference video is unrelated to the intent described in the target prompt, the model will:

• First follow the text prompt, since it defines the semantic goal

• Then fail to find matching cues in the reference video
  → Resulting in little or no transfer, so the output looks mainly guided by the prompt.

This is expected behavior under the current training design.

3. About using more than 49 frames

This depends on the backbone:

Backbone	Max Frames Supported	Notes
Wan2.1 (I2V)	Supports longer sequences	Our finetuning was limited to 49 frames due to dataset constraints, but longer outputs are possible in practice.
CogVideoX (I2V)	Limited to 49 frames	So our CogVideoX-based VAP version is also limited to 49 frames.

[deepbeepmeep]

Thank you very much for your detailed answer.

I have tried to use much more generic text prompts (that didnt describe the content of the video prompt) and the video prompt was still ignored : the output video was a basic continuation of the reference image.

It seems the text prompt is the only input that has a real weight on the output.

As I understand the model received during training redundent information about the expected output video (a video prompt and an output text prompt that happens to describe both the prompt video and the output video at the same time), it is likely the model learned mainly from the text prompt which contained enough information to do the matching with the output video (principle of least effort).

If you have the chance to refine your model in the future, it would be great to try to discard the two text prompts (or keep only one text prompt which is very generic and not specific to the prompt video). Not only it may increase the generalization capabilities of the model but the model will be much more practical to use as it wont require the user to write detailed text prompts.

[yxbian23]

Thank you very much for your detailed answer.

I have tried to use much more generic text prompts (that didnt describe the content of the video prompt) and the video prompt was still ignored : the output video was a basic continuation of the reference image.

It seems the text prompt is the only input that has a real weight on the output.

As I understand the model received during training redundent information about the expected output video (a video prompt and an output text prompt that happens to describe both the prompt video and the output video at the same time), it is likely the model learned mainly from the text prompt which contained enough information to do the matching with the output video (principle of least effort).

If you have the chance to refine your model in the future, it would be great to try to discard the two text prompts (or keep only one text prompt which is very generic and not specific to the prompt video). Not only it may increase the generalization capabilities of the model but the model will be much more practical to use as it wont require the user to write detailed text prompts.

Thank you for the thoughtful follow-up — and you raised a very good observation! ❤️
It seems there may still be a bit of misunderstanding, so I’ll try to clarify the key point more clearly:

1. Text prompt and reference video should be used together

As mentioned earlier, in the current VAP design the model does not treat the text prompt and reference video independently.
Instead, the model typically:

1. Uses the text prompt to establish the high-level semantic intent first
   (e.g., transfer motion, apply stylization, change identity, expand like a balloon, etc.)
2. Then looks into the reference video to find the fine-grained structure that corresponds to this intent, and transfers that information.

So, the text prompt ≈ what to transfer
and the reference video ≈ how it looks / moves / forms.

2. Why generic prompts or mismatched reference videos reduce the effect

If:

• The reference video does not contain cues related to the semantic intent in the text prompt, or
• The text prompt is too generic to express the intended semantics,

then the model cannot find a meaningful match between the two.
In this case, the output will lean more toward the text prompt alone, because the reference video cannot contribute relevant structure.

So it’s not that the model ignores the reference video —
It is that the model needs both the text (coarse intent) and the video (fine detail) to align.

3. To illustrate this behavior with a minimal example

We simplified the text prompt in infer/cog_vap.py to contain only the core semantic intent — “expand into a balloon and float upward” — without describing any appearance details or motion nuances.

Even with this very minimal text, the model was still able to retrieve the fine-grained visual and motion details from the reference video itself, because those cues exist clearly in the reference video.

Here is the core code and prompts used:

output_frames = pipe(
    image=image,
    ref_videos=[ref_frames],
    prompt="The horse begins to inflate, become a balloon-like form, and lift silently from the ground.",
    prompt_mot_ref=[
        "The sneaker begins to inflate, become a balloon-like form, and lift silently from the ground."
    ],
    height=480,
    width=720,
    num_frames=49,
    frames_selection="evenly",
    use_dynamic_cfg=True,
).frames[0]

And the generated result:

cog_vap.mp4

This demonstrates that the text prompt provides the coarse semantic intent, while the reference video supplies the fine structure, which also supports the potential effectiveness of a future instruction-based version that could make this process much more intuitive.

4. We strongly agree with your suggestion on making prompts easier 🎯

You are absolutely right — requiring two descriptive prompts is not the most user-friendly design.
In fact, in our technical report we highlighted this exact limitation.

Our next step (and what we see as the ideal direction) is to move toward instruction-based prompting, e.g.:

• “Transfer only the motion from the reference video”
• “Copy the Ghibli style from the reference video”
• “Use the Hitchcock camera movement from the reference video”

This would: Reduce user effort, Improve interpretability, Improve generalization, Make model much easier and more intuitive to use

Thanks again for the thoughtful discussion — your feedback aligns directly with the improvements we want to pursue.