Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json

[Copilot]

SentencePieceTokenizer only exposed Create(Stream) requiring a SentencePiece protobuf (.model), making it impossible to load Hugging Face JSON-only Unigram tokenizers that have no .model file.

New public APIs

From in-memory vocab:

SentencePieceTokenizer.Create(
    IEnumerable<(string Piece, float Score)> vocab,
    int unkId,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    ReadOnlySpan<byte> precompiledCharsMap = default,
    bool addDummyPrefix = true,
    bool escapeWhiteSpaces = true,
    bool treatWhitespaceAsSuffix = false,
    IReadOnlyDictionary<string, int>? specialTokens = null)

From tokenizer.json:

SentencePieceTokenizer.CreateFromTokenizerJson(
    Stream tokenizerJsonStream,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    IReadOnlyDictionary<string, int>? specialTokens = null)

CreateFromTokenizerJson reads model.vocab, model.unk_id, extracts precompiled_charsmap from a Precompiled or Sequence normalizer, and reads Metaspace pre-tokenizer settings (add_prefix_space, replacement, prepend_scheme). It validates model.type == "Unigram".

Internal changes

• SentencePieceBaseModel: new constructor taking individual config parameters instead of ModelProto
• SentencePieceUnigramModel: new constructors building vocab from IReadOnlyList<(string, float)>; BOS/EOS/PAD IDs auto-detected by piece name (<s>, </s>, <pad>) with SentencePiece-conventional positional fallbacks

Note on token IDs

HF tokenizer.json typically uses a different special-token ordering than the SentencePiece protobuf (e.g. <s>=0, <pad>=1, </s>=2, <unk>=3 vs. <unk>=0, <s>=1, </s>=2). Piece strings produced are identical; numeric IDs will differ by the vocab offset introduced by the extra special tokens.

[ericstj]

Thanks for adding this — the in-memory Create(vocab, ...) overload is clean and the ID-preservation (JSON vocab index = token id) is the right call. I implemented this same JSON-only Unigram capability recently against real Hugging Face models, and hit two correctness issues that this PR's single test model happens to mask. Details inline; summary here.

Bugs (untested by the current suite)

1. BOS/EOS positional fallback corrupts real pieces. FindSpecialTokenId(pieces, "<s>", 1) / ("</s>", 2) fall back to positions 1/2 when the vocab has no piece literally named <s>/</s>. Many HF Unigram tokenizers don't use those names — e.g. minishlab/potion-multilingual-128M (bge-m3 family) has unk_id=1, vocab [0]="[PAD]", [1]="[UNK]", [2]=",". There eosId→2 marks "," as Control and drops it from the Viterbi trie (it can never be emitted), and bosId→1 collides with unkId and clobbers the unknown entry. This is structural (independent of addBos/addEos).

2. Normalizer steps beyond Precompiled are silently dropped. ExtractPrecompiledCharsMap extracts only the charsmap and discards sibling normalizers. Real Unigram models often have a richer chain (potion/bge-m3: Sequence[Precompiled, Replace(punctuation spacing), Replace("\\s+"->" "), Strip]), which SentencePieceNormalizer cannot reproduce — so CreateFromTokenizerJson silently yields different tokens than HF. Since the charsmap must run before those Replace steps, they can't just be reordered into SP; at minimum this should throw on unrecognized normalizer types rather than silently ignore them.

Why the test stays green

Paraphrase-multilingual-MiniLM-L12-v2 names its specials (<s>=0, </s>=2) so the fallback never fires, and its normalizer is a single Precompiled, so the dropped-sibling path is never hit. Recommend adding fixtures that (a) place specials at non-conventional positions / omit <s>/</s>, and (b) use a Sequence normalizer with Replace/Strip, asserting against HF reference ids.

Minor

• removeExtraWhitespaces is hard-coded true in both factories rather than derived from the JSON.
• added_tokens from the JSON aren't auto-wired; correctness depends on the caller passing specialTokens. Worth documenting or reading them.

(Posting as comments only — not an approval or change request.)

[ericstj]

@copilot please address feedback

[ericstj]

@copilot address feedback

[codecov]

Codecov Report

❌ Patch coverage is 83.29238% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.61%. Comparing base (548d4d0) to head (7e32c05).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs	69.06%	28 Missing and 15 partials ⚠️
...t.ML.Tokenizers/Model/SentencePieceUnigramModel.cs	86.00%	8 Missing and 6 partials ⚠️
...soft.ML.Tokenizers/Model/SentencePieceBaseModel.cs	71.79%	10 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7625      +/-   ##
==========================================
+ Coverage   69.59%   69.61%   +0.01%     
==========================================
  Files        1484     1484              
  Lines      273606   274013     +407     
  Branches    27949    27999      +50     
==========================================
+ Hits       190410   190745     +335     
- Misses      75832    75885      +53     
- Partials     7364     7383      +19     

Flag	Coverage Δ
Debug	69.61% <83.29%> (+0.01%)	⬆️
production	63.85% <75.53%> (+0.01%)	⬆️
test	89.65% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
test/Microsoft.ML.Tokenizers.Tests/UnigramTests.cs	94.52% <100.00%> (+1.55%)	⬆️
...soft.ML.Tokenizers/Model/SentencePieceBaseModel.cs	79.81% <71.79%> (-0.63%)	⬇️
...t.ML.Tokenizers/Model/SentencePieceUnigramModel.cs	63.13% <86.00%> (+2.74%)	⬆️
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs	77.62% <69.06%> (-14.88%)	⬇️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR extends SentencePieceTokenizer to support Hugging Face JSON-only Unigram tokenizers by adding new public factory APIs that can construct a Unigram tokenizer from either an in-memory vocab list or a tokenizer.json stream, avoiding the current requirement for a SentencePiece .model protobuf.

Changes:

• Add SentencePieceTokenizer.Create(IEnumerable<(string Piece, float Score)> vocab, ...) for constructing a Unigram tokenizer directly from a vocab list.
• Add SentencePieceTokenizer.CreateFromTokenizerJson(Stream tokenizerJsonStream, ...) for parsing HF tokenizer.json (Unigram) including vocab, unk_id, precompiled charsmap, and Metaspace settings.
• Add internal constructors/refactoring to build a SentencePieceUnigramModel from vocab pieces and config values, plus new tests covering these creation paths.

Show a summary per file

File	Description
test/Microsoft.ML.Tokenizers.Tests/UnigramTests.cs	Adds unit tests for vocab-based and tokenizer.json-based Unigram construction and behavior parity checks.
src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs	Adds Unigram model constructors that build vocab/trie from (piece, score) inputs and detect special tokens by name.
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs	Adds public factories for vocab and tokenizer.json, plus JSON parsing helpers for normalizer/pre-tokenizer extraction.
src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs	Adds a new base-model constructor taking explicit config/token IDs instead of ModelProto.

Copilot's findings

• Files reviewed: 4/4 changed files
• Comments generated: 4