Bug:
AttributeError: 'FinetuningTokenGenerator' object has no attribute 'pad_id' occurs whilst attempting to create preprocessor for a image-caption dataset.
I believe this is from the Cerebras modelzoo code for this snippet finetuning_token_generator.py:74-79
self.eos_token = (
self.tokenizer.convert_ids_to_tokens(self.pad_id)
if self.eos_id is None
else self.tokenizer.convert_ids_to_tokens(self.eos_id)
)
self.pad_id = pad_id
self.pad_id is being used just before its assignment.
In my case moving the assignment just above this snippet fixed the problem ensuring it was set before use. The error causing line self.tokenizer.convert_ids_to_tokens(self.pad_id) was added slightly later to this section so might explain where the issue has arisen from.
Bug:
AttributeError: 'FinetuningTokenGenerator' object has no attribute 'pad_id'occurs whilst attempting to create preprocessor for a image-caption dataset.I believe this is from the Cerebras modelzoo code for this snippet finetuning_token_generator.py:74-79
self.pad_id is being used just before its assignment.
In my case moving the assignment just above this snippet fixed the problem ensuring it was set before use. The error causing line
self.tokenizer.convert_ids_to_tokens(self.pad_id)was added slightly later to this section so might explain where the issue has arisen from.