Skip to content

Problème avec passage tokenizer #22

Description

@Rollybre
tokenizer = AutoTokenizer.from_pretrained("camembert-base", use_fast=True)

fb_emotion = Fabula(scorer=TransformersScorer(model="astrosbd/french_emotion_camembert"), 
                    segmenter=SlidingWindowTokenSegmenter(tokenizer = tokenizer,window_tokens=512, stride_tokens=256),
                    analysis="emotion")

Output :
Token indices sequence length is longer than the specified maximum sequence length for this model (14046 > 512). Running this sequence through the model will result in indexing errors

Je suppose un problème au niveau du self.tokeize dans segment.py, j'investigue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions