Class MiniMaxTokenizer

java.lang.Object

smile.llm.tokenizer.Tiktoken

smile.llm.model.minimax.MiniMaxTokenizer

All Implemented Interfaces:

Tokenizer

public class MiniMaxTokenizer
extends Tiktoken

MiniMax-M3 tokenizer: NFC normalization, then byte-level BPE with the model's case-aware split regex. Control tokens (]~b] BOS, [e~[ EOS, ]!p~[ pad, ...) are taken from tokenizer.json's added_tokens; only specials actually present are registered so no id can fall outside the embedding table.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	MiniMaxTokenizer.AddedToken	One entry of added_tokens in tokenizer.json.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BOS	Beginning-of-turn token.
static final String	EOS	End-of-turn token.
static final String	PAD	Padding token.

Fields inherited from class Tiktoken

ranks, specialTokens

Constructor Summary

Constructors

Constructor	Description
MiniMaxTokenizer(Map<Bytes,Integer> ranks)	Constructor for a vocabulary whose added tokens are the built-in MiniMax set.
MiniMaxTokenizer(Map<Bytes,Integer> ranks, List<MiniMaxTokenizer.AddedToken> added)	Constructor.

Method Summary

Modifier and Type	Method	Description
int[]	encode(String text, boolean bos, boolean eos)	Encodes a string into a list of token IDs.
int[]	encodeWithSpecials(String text)	Encodes template text in which markers such as ]~b], [e~[ and ]<]minimax[>[ stand for single special tokens, as the reference tokenizer does for text containing them.
static MiniMaxTokenizer	of(String checkpointDir)	Loads the tokenizer from a checkpoint directory containing tokenizer.json.
int	pad()	Padding token id.
int[]	stopTokens()	Stop token ids used during generation.
String	tryDecode(int[] tokens, boolean skipSpecial)	Decodes tokens.

Methods inherited from class Tiktoken

allowSpecialTokens, decode, encode, isSpecialTokenAllowed, load, size, specialToken, tokenize, tryDecode

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BOS

public static final String BOS

Beginning-of-turn token.

See Also:

• Constant Field Values

EOS

public static final String EOS

End-of-turn token.

See Also:

• Constant Field Values

PAD

public static final String PAD

Padding token.

See Also:

• Constant Field Values

Constructor Details

MiniMaxTokenizer

public MiniMaxTokenizer(Map<Bytes,Integer> ranks)

Constructor for a vocabulary whose added tokens are the built-in MiniMax set.

Parameters:

ranks - token byte sequence to id map, including added tokens.

Throws:

IllegalArgumentException - if the BOS, EOS or pad token is missing.

MiniMaxTokenizer

public MiniMaxTokenizer(Map<Bytes,Integer> ranks,
 List<MiniMaxTokenizer.AddedToken> added)

Constructor.

Every added token is atomic when encoding text, special or not (as in the reference tokenizer), which is what makes <tool_call> and ]<]minimax[>[ single tokens. Tokens flagged special are hidden when decoding for display.

Parameters:

ranks - token byte sequence to id map, including added tokens.

added - the added_tokens of tokenizer.json.

Throws:

IllegalArgumentException - if the BOS, EOS or pad token is missing.

Method Details

of

public static MiniMaxTokenizer of(String checkpointDir)
                           throws IOException

Loads the tokenizer from a checkpoint directory containing tokenizer.json.

Parameters:

checkpointDir - model directory.

Returns:

tokenizer.

Throws:

IOException - if tokenizer.json is missing or malformed.

encode

public int[] encode(String text,
 boolean bos,
 boolean eos)

Description copied from interface: Tokenizer

Encodes a string into a list of token IDs.

Specified by:

encode in interface Tokenizer

Overrides:

encode in class Tiktoken

Parameters:

text - The input string to be encoded.

bos - Whether to prepend the beginning-of-sequence token.

eos - Whether to append the end-of-sequence token.

Returns:

A list of token IDs.

encodeWithSpecials

public int[] encodeWithSpecials(String text)

Encodes template text in which markers such as ]~b], [e~[ and ]<]minimax[>[ stand for single special tokens, as the reference tokenizer does for text containing them.

Parameters:

text - a rendered chat prompt.

Returns:

token ids.

tryDecode

public String tryDecode(int[] tokens,
 boolean skipSpecial)
                 throws CharacterCodingException

Decodes tokens. With skipSpecial, control tokens (]~b], [e~[, padding, ...) are dropped, the reasoning tags <mm:think> and </mm:think> are rendered as <think> and </think> (the form chat clients style as reasoning), and the tool-call marker ]<]minimax[>[ stays so tool calls can be parsed from the text.

Specified by:

tryDecode in interface Tokenizer

Overrides:

tryDecode in class Tiktoken

Parameters:

tokens - The list of token IDs to be decoded.

skipSpecial - when true, registered special-token IDs are omitted (useful for streaming chat so headers/eot are not shown as literal <|...|> text).

Returns:

The decoded string.

Throws:

CharacterCodingException - If the byte sequence is not legal UTF-8.

pad

public int pad()

Padding token id.

Returns:

pad token id.

stopTokens

public int[] stopTokens()

Stop token ids used during generation.

Returns:

the end-of-turn token id.