[MOEB] model: add ViCLIP video-language model (L-14, B-16)#5013
Open
Rakshitha-Ireddi wants to merge 4 commits into
Open
[MOEB] model: add ViCLIP video-language model (L-14, B-16)#5013Rakshitha-Ireddi wants to merge 4 commits into
Rakshitha-Ireddi wants to merge 4 commits into
Conversation
Adds ViCLIPWrapper and ModelMeta for OpenGVLab/ViCLIP-L-14-hf and OpenGVLab/ViCLIP-B-16-hf. Model from InternVid (ICLR 2024, arXiv:2307.06942). Covers T2V and V2T retrieval tasks in MOEB. Closes embeddings-benchmark#5012, part of embeddings-benchmark#4842.
Contributor
Author
|
Hello @AdnanElAssadi56 and @Samoed . can you please check this |
Member
|
Can you evaluate model on tasks to match performance from papper? |
Samoed
reviewed
Jul 25, 2026
Comment on lines
+18
to
+19
| _VICLIP_MEAN = (0.48145466, 0.4578275, 0.40821073) | ||
| _VICLIP_STD = (0.26862954, 0.26130258, 0.27577711) |
Samoed
reviewed
Jul 25, 2026
Comment on lines
+98
to
+105
| if not isinstance(feat, torch.Tensor): | ||
| feat = ( | ||
| feat.pooler_output | ||
| if hasattr(feat, "pooler_output") | ||
| else next(iter(feat.values())) | ||
| ) | ||
| if feat.dim() == 1: | ||
| feat = feat.unsqueeze(0) |
Member
There was a problem hiding this comment.
I don't think tat we need this part
Comment on lines
+129
to
+134
| if not isinstance(features, torch.Tensor): | ||
| features = ( | ||
| features.pooler_output | ||
| if hasattr(features, "pooler_output") | ||
| else next(iter(features.values())) | ||
| ) |
Samoed
added
the
new model
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ViCLIP video-language embedding model to MTEB, as part of the MOEB initiative (#4842).
Closes #5012.
Models added
OpenGVLab/ViCLIP-L-14-hfOpenGVLab/ViCLIP-B-16-hfAbout ViCLIP
ViCLIP is a video-text contrastive model introduced in the InternVid paper (ICLR 2024, arXiv:2307.06942). It extends CLIP's ViT backbone with spatiotemporal attention to
encode video clips (8 frames at 224×224) and aligns them with text via contrastive learning on InternVid-10M-FLT (10M video-text pairs).
Tasks covered: T2VRetrieval, V2TRetrieval — directly applicable to MOEB tasks including MSR-VTT, MSVD, VALOR-32K, VSC2022, and ActivityNet.
Implementation notes
ViCLIP uses a non-standard API (custom BPE tokenizer via
model.tokenizer,get_vid_features()/get_text_features()) so a customViCLIPWrapperwas implemented following thexclip_models.pypattern.Requires
trust_remote_code=True.Checklist
mteb.get_model(model_name, revision)andmteb.get_model_meta(model_name, revision)