`tokenizer.bos_token = cfg["boq"]` may fail if token not in vocab

[hty993]

conversion/convert_to_hf.py:105-110:

def set_tokenizer_special_tokens(tokenizer, cfg: dict):
    if "boq" in cfg:
        tokenizer.bos_token = cfg["boq"]
    if "eoa" in cfg:
        tokenizer.eos_token = cfg["eoa"]
    return tokenizer

HuggingFace tokenizers will accept setting bos_token to a string that isn't in the vocab — it just adds it as an added token. This silently grows the vocab past what the model head was built for, which then fails during the next forward pass with an out-of-range token id error.

prepare_sft_data.py uses tok.token_to_id(name) to validate each special token exists (lines 62-66), and would be the right pattern here too:

def set_tokenizer_special_tokens(tokenizer, cfg: dict):
    for attr, key in (("bos_token", "boq"), ("eos_token", "eoa")):
        if key in cfg:
            tok = cfg[key]
            if tokenizer.convert_tokens_to_ids(tok) == tokenizer.unk_token_id:
                raise ValueError(f"special token {tok!r} not in vocab")
            setattr(tokenizer, attr, tok)
    return tokenizer

Severity: Medium — the HF export looks correct on first inspection, but loaded models would fail on first inference.