[FEAT] Support GLM4.1V series - #534
Conversation
|
Ready for review. |
This comment was marked as resolved.
This comment was marked as resolved.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for GLM-4.1V models, including training scripts, monkey patching, dataset position ID handling, and input preprocessing. The changes primarily focus on enabling training and inference with GLM-4.1V models and ensuring compatibility with their specific requirements. I have added comments to address potential issues related to code correctness and efficiency.
|
Ready for review. @hiyouga |
| temperature: float = 1.0, | ||
| **kwargs, | ||
| ) -> tuple | Glm4vCausalLMOutputForPPO: | ||
| from verl.utils.experimental.torch_functional import FusedLinearForPPO |
There was a problem hiding this comment.
Removed for no used.
This PR is modified by PR #3291 of VeRL, with removed 'padding_free' tech.
| QWEN2_VL_MODELS = ("qwen2_vl", "qwen2_5_vl") | ||
| QWEN3_VL_MODELS = ("qwen3_vl", "qwen3_vl_moe") | ||
| # TODO support monkey patch for glm4.1v | ||
| GLM_VL_MODELS = ("glm4.1v_base", "glm4.1v_thinking") |
There was a problem hiding this comment.
The 'padding_free' haven been tested, I suggest putting it in the future
|
The swanlab log with |
|
Done for 2nd review. |
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Code Review
This pull request adds support for GLM-4.1V models, introducing new training scripts and model-specific handling for position IDs and input preprocessing. The changes are well-structured. However, I've identified a critical bug in the video position ID generation logic that could lead to incorrect embeddings. Additionally, there's a potential KeyError in the FSDP worker and a minor performance concern in the new get_rope_index function. Addressing these issues will improve the robustness and correctness of the implementation.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2 participants
This pull request adds support for GLM-4.1V models in the codebase, including training scripts, monkey patching, dataset position ID handling, and input preprocessing. The changes primarily focus on enabling training and inference with GLM-4.1V models and ensuring compatibility with their specific requirements.
GLM-4.1V Model Support:
glm4.1v_base_geo3k_grpo.shandglm4.1v_thinking_geo3k_grpo.sh, enabling quick setup for training on the Geometry3K dataset. [1] [2]verl/models/monkey_patch.pyto recognize GLM-4.1V models in theGLM_VL_MODELStuple for future monkey patching support.Dataset and Preprocessing Adjustments:
verl/utils/dataset.pyto add special handling for GLM-4.1V position IDs, using only 3D vision position IDs for rotary embedding as required by these models.verl/workers/fsdp_workers.pyto ensure that pixel values from certain image processors (such as GLM-4.1V) are properly squeezed to remove the batch dimension.