PPL logits

[szwedek]

I noticed that you used the performer's logits to compute PPL, but the paper says it should be the observer's.

https://github.com/ahans30/Binoculars/blob/main/binoculars/detector.py#L91

I've tested both approaches and the performer's logits result in better classification.

I think it should be marked or corrected somehow.

[inspire22]

Thanks for sharing this @ your llama cpp code, I got a lot of use out of it.

I did try both and the original PPL method worked best for me. Though I had tried a lot of models to find a set that worked well (LLama-3.2-1B actually worked even better than mistral 8b for me surprisingly). I'm doing poetry AI detection though, so it might have some different aspects.

Unsloth's FastLanguageModel (with load_in_4bit) was about 7x faster than llama_cpp's 4-bit GGUFs interestingly.

from unsloth import FastLanguageModel
self.observer_model, tokenizer = FastLanguageModel.from_pretrained("unsloth/Llama-3.2-1B", load_in_4bit=True)
self.performer_model, self.tokenizer = FastLanguageModel.from_pretrained("unsloth/Llama-3.2-1B-Instruct", load_in_4bit=True)

[szwedek]

@inspire22 you need to know that other models need different thresholds. The thresholds have to be adjusted if you change the original model.

I tested the original PPL method and "the right one" on 1k polish samples(human/AI) . The original method is much better, so it should be marked in the paper somehow or tested by somebody on different samples.