Questions on the evaluation process

[mutiann]

Hello,

Thank you for your great work! I'm trying to reproduce the work and trying to look deeper into the evaluation process, particularly on inputs longer than the training condition (which is 4K). As far as I understand, evaluation with length extrapolation (e.g. on LongBench) will not involve any update to the model itself on linear attention and SWA models. However, on the transformer++ baseline, I assume that some expansion of the original RoPE embeddings will be necessary. Hence I wonder what strategy is actually adopted in your experiments that report the results on transformer.

Thank you!

[sustcsonglin]

Sorry for the super late reply :(
We didn't use any length extension techniques for the Transformer++ baseline, which is why its score is very low. We should have applied some techniques, such as post-training on longer sequences first, for these evaluations. This was an oversight on our part.

[mutiann]

Sorry for the super late reply :( We didn't use any length extension techniques for the Transformer++ baseline, which is why its score is very low. We should have applied some techniques, such as post-training on longer sequences first, for these evaluations. This was an oversight on our part.

This is reassuring :) I was thinking about if I did anything wrong in my experiment. Thanks for the explanation!

[mutiann]

Thank you so much for the clarifications! While recently I also encountered some issues regarding the evaluations on the zero-shot in-context recall-intensive evaluations. I tried to follow the instructions in README to use the scripts from Just Read Twice to evaluate the models and to obtain the Table 4 in the Gated DeltaNet paper. To align with the paper, I set context_length to 2000, and evaluated with a single pass (without twice), with bf16 models.

To my surprise, the delta_net-1.3B-100B model can achieve 35.8% average score, compared to the 26.2% result of DeltaNet and 30.6% of Gated DeltaNet reported in Table 4. This is quite close to a transformer-1.3B-100B model, with 37.0% in both my evaluation (using the model on fla-hub) and also in Table 4. Therefore I wonder if I am using the correct evaluation protocol, and if I can know more about the exact evaluation protocol and models used in the evaluations reported in the paper.

Thank you!