Class Tokenizer

java.lang.Object
smile.llm.tokenizer.Tiktoken
smile.llm.model.qwen.Tokenizer
All Implemented Interfaces:
Tokenizer
public class Tokenizer extends Tiktoken
Qwen3.5 chat tokenizer (byte-level BPE via Tiktoken).

Loads from a checkpoint directory. Prefers HuggingFace tokenizer.json (includes added_tokens with the real <|im_start|> / <|im_end|> ids) over vocab.json alone.

  • Constructor Details

    • Tokenizer

      public Tokenizer(Map<Bytes,Integer> ranks)
      Constructor.
      Parameters:
      ranks - token byte sequence → id map.

    • Tokenizer

      public Tokenizer(Map<Bytes,Integer> ranks, String bos, String eos, String... specialTokens)
      Constructor.
      Parameters:
      ranks - token → id map.
      bos - beginning-of-sequence token.
      eos - end-of-sequence token.
      specialTokens - additional special tokens.

  • Method Details

    • pad

      public int pad()
      Padding token id (<|endoftext|>).
      Returns:
      pad token id.

    • stopTokens

      public int[] stopTokens()
      Stop token ids used during generation.
      Returns:
      stop token id array.

    • setChatTemplate

      public void setChatTemplate(Qwen3ChatTemplate.Variant variant)
      Declares the chat template of the checkpoint, so encodeDialog(Message...) renders prompts like it.
      Parameters:
      variant - the template's behaviours (Qwen3ChatTemplate.Variant.detect(String)), or null for the legacy hand-built prompt.

    • chatTemplate

      public Qwen3ChatTemplate.Variant chatTemplate()
      Returns the declared chat template behaviours.
      Returns:
      the variant, or null when none was declared.

    • registerAtomicTokens

      public void registerAtomicTokens(Collection<String> contents)
      Registers added tokens (control tokens plus <tool_call>-style markers) that must stay a single token when they occur in text, as the reference tokenizer does. Tokens missing from the vocabulary are ignored.
      Parameters:
      contents - the token strings.

    • encodeWithSpecials

      public int[] encodeWithSpecials(String text)
      Encodes rendered chat text in which control tokens (<|im_start|>, <think>, <tool_call>, the vision placeholders, ...) stand for single tokens; everything between them is ordinary text.
      Parameters:
      text - a rendered prompt.
      Returns:
      token ids.

    • renderDialog

      public String renderDialog(Message[] dialog, ChatOptions options)
      Renders a dialog with the checkpoint's chat template.
      Parameters:
      dialog - the messages.
      options - chat options (tools, thinking mode, reasoning effort); may be null.
      Returns:
      the prompt text with the assistant turn open, or null when no template was declared.

    • encodeDialog

      public int[] encodeDialog(Message... dialog)
      Encodes a chat dialog in the Qwen chat-template format and leaves the assistant header open for completion.
      Parameters:
      dialog - conversation turns.
      Returns:
      token ids.

    • encodeDialog

      public int[] encodeDialog(Message[] dialog, ChatOptions options)
      Encodes a chat dialog with optional tool-calling options.
      Parameters:
      dialog - conversation turns.
      options - chat options; when tools are present (or the dialog already contains tool turns), uses Qwen3ChatTemplate.
      Returns:
      token ids.

    • of

      public static Tokenizer of(String checkpointDir) throws IOException
      Loads a tokenizer from a HuggingFace checkpoint directory.

      Order matters: tokenizer.json is preferred because it carries added_tokens with the true chat special ids. Loading vocab.json alone and then appending specials at maxId+1 remaps <|im_start|> past vocab_size and crashes embedding gather on GPU.

      Parameters:
      checkpointDir - model directory.
      Returns:
      tokenizer instance.
      Throws:
      IOException - if no supported tokenizer files are found.

    • loadTokenizerJson

      public static Map<Bytes,Integer> loadTokenizerJson(Path tokenizerJson) throws IOException
      Extracts vocab from HuggingFace tokenizer.json (model.vocab plus added_tokens).
      Parameters:
      tokenizerJson - path to tokenizer.json.
      Returns:
      token byte sequence → id map (GPT-2 keys converted to raw bytes).
      Throws:
      IOException - if the file is missing or not a valid tokenizer.json.

    • requireChatSpecialsInVocab

      public void requireChatSpecialsInVocab(int vocabSize)
      Ensures chat control tokens lie in [0, vocabSize). Call after construction against the model config vocab size.
      Parameters:
      vocabSize - embedding / lm_head vocabulary size from model config.