Chap 3: Fault about TransformerForSequenceClassification

[S3nnyK]

Information

The question or comment is about chapter:

• Introduction
• Text Classification
• Transformer Anatomy
• Multilingual Named Entity Recognition
• Text Generation
• Summarization
• Question Answering
• Making Transformers Efficient in Production
• Dealing with Few to No Labels
• Training Transformers from Scratch
• Future Directions

Question or comment

In TransformerForSequenceClassification, x = self.encoder(x)[:, 0, :] means [CLS] token is included in the inputs. However, in the beginning of this chapter, inputs is defined as tokenizer(text, return_tensors="pt", add_special_tokens=False), without special_tokens. Hence, the 0-th is "time", not "[CLS]".