Thanks for sharing this fascinating work and open-sourcing the codebase! I really appreciate the novel perspective of using physics-based control signals for video generation.
While exploring the code, I had a few specific questions regarding the implementation details and method design. I would be very grateful if you could share your insights:
-
Point Force Injection (src/data/controlnet_datasets.py)
Looking around line 274 in src/data/controlnet_datasets.py, my understanding is that you pre-calculate the object's position at each timestep based on the initial force, and then render a 3D Gaussian blob at those corresponding positions to serve as the control condition. Could you please confirm if my understanding here is correct?
-
Handling Object Mass in Demos
In the demo featuring the laundry basket, applying the same force to objects of different weights results in different movement speeds. Could you clarify how this is implemented? Is the visual control signal pre-scaled based on the object's weight before being fed into the network?
-
Continuous vs. Instantaneous Force
The force control signal generation in the code appears to be continuous across frames. However, the Blender simulation seems to rely on an instantaneous force (impulse). Is there a gap between the simulation logic and the diffusion control injection, or were environmental factors like friction removed in the simulation so that an impulse naturally approximates continuous motion?
-
Connection to Trajectory-guided Generation
Regarding the method design: if my understanding of the pre-calculated positions (Question 1) is correct, could Force-Prompting be conceptually viewed as a physics-based variant of trajectory-guided video generation (similar to methods like Wan-Move or MoFA-Video)? The main distinction would be that your method derives the trajectory mathematically from physical force inputs, rather than relying on manual user drawings. I would love to hear your thoughts on this comparison.
Thanks again for your time and the great work! Looking forward to your thoughts.
Thanks for sharing this fascinating work and open-sourcing the codebase! I really appreciate the novel perspective of using physics-based control signals for video generation.
While exploring the code, I had a few specific questions regarding the implementation details and method design. I would be very grateful if you could share your insights:
Point Force Injection (src/data/controlnet_datasets.py)
Looking around line 274 in src/data/controlnet_datasets.py, my understanding is that you pre-calculate the object's position at each timestep based on the initial force, and then render a 3D Gaussian blob at those corresponding positions to serve as the control condition. Could you please confirm if my understanding here is correct?
Handling Object Mass in Demos
In the demo featuring the laundry basket, applying the same force to objects of different weights results in different movement speeds. Could you clarify how this is implemented? Is the visual control signal pre-scaled based on the object's weight before being fed into the network?
Continuous vs. Instantaneous Force
The force control signal generation in the code appears to be continuous across frames. However, the Blender simulation seems to rely on an instantaneous force (impulse). Is there a gap between the simulation logic and the diffusion control injection, or were environmental factors like friction removed in the simulation so that an impulse naturally approximates continuous motion?
Connection to Trajectory-guided Generation
Regarding the method design: if my understanding of the pre-calculated positions (Question 1) is correct, could Force-Prompting be conceptually viewed as a physics-based variant of trajectory-guided video generation (similar to methods like Wan-Move or MoFA-Video)? The main distinction would be that your method derives the trajectory mathematically from physical force inputs, rather than relying on manual user drawings. I would love to hear your thoughts on this comparison.
Thanks again for your time and the great work! Looking forward to your thoughts.