[API] Public way to construct SentencePieceTokenizer (Unigram) from tokenizer.json / in-memory pieces+scores

[ericstj]

[API] Public way to construct SentencePieceTokenizer (Unigram) from tokenizer.json / in-memory pieces+scores

Summary

Microsoft.ML.Tokenizers fully implements the SentencePiece Unigram model
(SentencePieceUnigramModel), but the only public way to obtain a
SentencePieceTokenizer is SentencePieceTokenizer.Create(Stream), which parses a
SentencePiece protobuf (.model / ModelProto). There is no public API to build a
Unigram tokenizer from a Hugging Face tokenizer.json (or from in-memory pieces + scores).

Many modern HF models ship a JSON-only Unigram tokenizer (tokenizer.json with
model.type == "Unigram", model.vocab as [piece, score] pairs, model.unk_id, and a
normalizer/pre_tokenizer) and no sentencepiece.model/spiece.model/tokenizer.model
protobuf. For these models there is currently no supported way to construct the tokenizer.

Current behavior

• SentencePieceTokenizer's only constructor is internal SentencePieceTokenizer(ModelProto modelProto, ...).
• The Sentencepiece.* generated protobuf types (ModelProto, TrainerSpec,
  NormalizerSpec, …) are internal, so callers can't build a ModelProto directly.
• The public factory SentencePieceTokenizer.Create(Stream, bool addBeginOfSentence, bool addEndOfSentence, ...)
  requires a serialized SentencePiece protobuf stream.
• By contrast, BpeTokenizer / WordPieceTokenizer expose vocab-file/stream factories, so the
  asymmetry is Unigram-specific.

Request

A public factory to construct a Unigram SentencePieceTokenizer without a protobuf, e.g. one of:

1. From tokenizer.json — SentencePieceTokenizer.CreateFromTokenizerJson(Stream json, ...)
   (parse model.vocab pieces+scores, model.unk_id, and the normalizer/pre_tokenizer
   precompiled charsmap + metaspace settings).
2. From in-memory pieces+scores —
   Create(IEnumerable<(string Piece, float Score)> vocab, int unkId, ReadOnlySpan<byte> precompiledCharsMap, bool addDummyPrefix, bool escapeWhitespaces, ...).

Either would let callers load JSON-Unigram models that have no .model protobuf.

Workaround we're using

Since the only public entry is a protobuf stream, we synthesize a SentencePiece ModelProto
on the fly from tokenizer.json and feed the bytes to Create(Stream):

• pieces ← model.vocab [piece, score] (mapping special tokens to CONTROL/UNKNOWN types).
• trainer_spec.model_type = UNIGRAM, trainer_spec.unk_id = model.unk_id (+ bos/eos/pad ids).
• normalizer_spec.precompiled_charsmap ← the Precompiled normalizer's precompiled_charsmap
  bytes from tokenizer.json (gives byte-exact NFKC parity), plus
  add_dummy_prefix / escape_whitespaces from the Metaspace pre-tokenizer.

This works, but it requires hand-writing a SentencePiece protobuf encoder and re-deriving the
wire schema, which is exactly the kind of thing the library could expose directly. It's also
fragile across schema changes.

Why it matters

JSON-only Unigram tokenizers are common (multilingual static-embedding models, several HF
encoder models). Without a JSON/in-memory factory, every consumer must either ship a converted
.model or reimplement the protobuf synthesis above.

Repro context

• Model: a potion-multilingual-128M-style static embedding model — tokenizer.json only,
  model.type == "Unigram", ~500k-entry custom vocab, [PAD]=0 / [UNK]=1, with a
  Sequence→Precompiled normalizer (precompiled_charsmap present) and a Metaspace
  pre-tokenizer. No sentencepiece.model.
• Microsoft.ML.Tokenizers main (commit 901da3e).

[github-actions]

🔍 No similar issues found with high confidence.

---

AI-powered similar issue detection

[ericstj]

/cc @tarekgh

[github-actions]

Triage Notes

Label: enhancement — this is a well-described request for a new public API, not a bug. The library currently works correctly for .model protobuf inputs; the gap is that there is no supported path for HuggingFace JSON-only Unigram models.

Area: Microsoft.ML.Tokenizers — specifically SentencePieceTokenizer and SentencePieceUnigramModel.

---

Implementation Plan

The core blocker is that both SentencePieceTokenizer's constructor and SentencePieceBaseModel's constructor are tightly coupled to ModelProto (the internal generated protobuf type). A minimal, additive fix would involve two layers:

1. Add a raw-data constructor to SentencePieceBaseModel / SentencePieceUnigramModel

SentencePieceBaseModel (src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs) extracts a handful of scalar fields from ModelProto (bos/eos/unk tokens and IDs, addDummyPrefix, escapeWhitespaces, byteFallback, precompiledCharsmap). A second internal constructor that accepts those fields directly — with no ModelProto dependency — would allow the Unigram model to be built from any source.

SentencePieceUnigramModel (src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs) similarly iterates modelProto.Pieces to populate its _vocab/_vocabReverse/_trie. A companion constructor accepting IReadOnlyList<(string Piece, float Score, PieceType Type)> (or a simpler (string, float)[] for normal tokens) alongside the scalar config would complete the internal change.

2. Expose two new public factories on SentencePieceTokenizer

In src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:

// Option A – from HuggingFace tokenizer.json (preferred for the reported use case)
public static SentencePieceTokenizer CreateFromHuggingFaceJson(
    Stream jsonStream,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    IReadOnlyDictionary<string, int>? specialTokens = null);

// Option B – from in-memory pieces + scores (building block, and useful on its own)
public static SentencePieceTokenizer Create(
    IEnumerable<(string Piece, float Score)> vocab,
    int unkId,
    ReadOnlySpan<byte> precompiledCharsMap = default,
    bool addDummyPrefix = true,
    bool escapeWhiteSpaces = true,
    bool byteFallback = false,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    IReadOnlyDictionary<string, int>? specialTokens = null);

CreateFromHuggingFaceJson would parse the well-known HF schema (model.type, model.vocab as [piece, score] pairs, model.unk_id, normalizer.precompiled_charsmap, pre_tokenizer Metaspace settings) using System.Text.Json and delegate to the in-memory Create overload. Note that SentencePieceNormalizer already accepts a raw precompiledCharsMap span, so no normalizer changes are needed.

Files likely affected:

File	Change
src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs	Add non-ModelProto internal constructor
src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs	Add non-ModelProto internal constructor
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs	Add CreateFromHuggingFaceJson + in-memory Create overload; update existing internal constructor to route through either path
test/Microsoft.ML.Tokenizers.Tests/	Add round-trip tests using a real JSON-only Unigram tokenizer.json

The BPE path (SentencePieceBpeModel) has similar coupling and could follow the same pattern in a follow-up if needed, but the issue is Unigram-specific.

Generated by Issue Triage for issue #7624 · ● 374.7K · ◷

[tarekgh]

@ericstj SentencePiece supports different tokenizer models, such as BPE and Unigram. As a result, Hugging Face tokenizer JSON files can have different formats depending on the underlying tokenizer model.

For example, the Llama 1 tokenizer uses SentencePiece BPE, and its tokenizer JSON follows a format similar to a generic BPE tokenizer (e.g., the Llama tokenizer JSON https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/raw/main/tokenizer.json. However, this format cannot be loaded directly by the SentencePiece tokenizer and must first be converted into the SentencePiece model format. This is why many models ship a tokenizer.model file alongside the tokenizer.json file.

In contrast, if you look at a SentencePiece Unigram tokenizer, such as the one used by paraphrase-multilingual-MiniLM-L12-v2, you'll notice that its tokenizer JSON has a different structure that reflects the Unigram model representation https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/blob/main/tokenizer.json.