pyiqa.archs.qrealign.qwen3_5_src.tokenization_qwen3_5¶

Tokenization classes for Qwen3.5.

Module Contents¶

class pyiqa.archs.qrealign.qwen3_5_src.tokenization_qwen3_5.Qwen3_5Tokenizer(vocab: str | dict[str, int] | None = None, merges: str | list[str] | None = None, vocab_file=None, merges_file=None, unk_token: str = '<|endoftext|>', bos_token=None, eos_token: str = '<|endoftext|>', pad_token: str = '<|endoftext|>', add_prefix_space=None, **kwargs)[source]¶: Bases: pyiqa.archs.tokenization_utils_tokenizers.TokenizersBackend