Skip to content

Add DisallowedSpecialTokenError (ValueError subclass) for disallowed special tokens (#290) - #590

Open
JayYarlagadda wants to merge 1 commit into
openai:mainfrom
JayYarlagadda:fix/disallowed-special-token-error
Open

Add DisallowedSpecialTokenError (ValueError subclass) for disallowed special tokens (#290)#590
JayYarlagadda wants to merge 1 commit into
openai:mainfrom
JayYarlagadda:fix/disallowed-special-token-error

Conversation

@JayYarlagadda

Copy link
Copy Markdown

Summary

This adds a dedicated DisallowedSpecialTokenError for the case where encode() encounters text corresponding to a special token that was not allowed.

The exception is exported from the top-level tiktoken package, so callers can handle this specific condition without matching the existing error message:

try:
    encoding.encode(text)
except tiktoken.DisallowedSpecialTokenError:
    # Handle disallowed special-token input specifically.
    ...

Why

Until now, this path raised a plain ValueError. That made it difficult for callers to distinguish a disallowed special token from other invalid input without relying on the exception message.

DisallowedSpecialTokenError subclasses ValueError, so existing code that catches ValueError continues to work unchanged. The message and detection behavior are also unchanged; only the concrete exception type is made more specific.

Implementation

  • Add DisallowedSpecialTokenError(ValueError) in tiktoken.core.
  • Raise it from the existing centralized raise_disallowed_special_token() helper, covering all current encode paths.
  • Re-export it from tiktoken as part of the public API.
  • Extend the existing special-token test to assert the dedicated exception while retaining coverage that it is catchable as ValueError.

Validation

  • Focused special-token test: 1 passed.
  • Full test suite with a deterministic Hypothesis seed: 33 passed.
  • Verified the public exception MRO includes ValueError.

Closes #290

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use a custom exception ValueError subclass for the special tokens warning

1 participant