Blog: Granite Switch in Mellea#59
Conversation
Introduces Granite Switch as a delivery mechanism for Mellea intrinsics — single vLLM-served checkpoint, no adapter-weight lifecycle management. Shows answerability checking and hallucination detection end-to-end. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Hub-and-spoke SVG showing one Granite Switch checkpoint serving five intrinsic capabilities (answerability, hallucination, citations, req. check, guardian), colour-coded by library family (RAG/Core/Safety). Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Flags vLLM/Linux as the validated path and notes omlx/vmlx as unvalidated macOS alternatives pending end-to-end smoke-test. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
- Add granite-switch plugin package install (registers GraniteSwitchForCausalLM architecture with vLLM; without it vLLM crashes immediately on model load) - Add required vLLM flags: --enable-auto-tool-choice and --tool-call-parser granite4 (without the latter, adapter dispatch silently fails) - Replace "What this post does" callout with prerequisites callout - Fix faithfulness output values: faithful/unfaithful (lowercase) — confirmed against live ibm-granite/granite-switch-4.1-3b-preview - Fix pip install quoting: pip install 'mellea[switch]' - Remove "(OpenAI-compatible)" qualifier from table Runtime cell - Strengthen hook: matrix framing; "zero adapter files to manage" - Strengthen trade-off section: eliminate lifecycle management entirely - Strengthen operational cost section: N×M matrix argument; adding a new intrinsic is a code change, not an infrastructure change - Strengthen vision section: "just a request parameter", clear path to broad runtime support, same code path in prod and on a laptop Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
planetf1
left a comment
planetf1
left a comment
There was a problem hiding this comment.
Code and content review — all runnable snippets validated against a live ibm-granite/granite-switch-4.1-3b-preview vLLM instance.
What's landing well: the intrinsics-vs-Switch distinction in the second section is sharp and exactly right. The demo flows cleanly from setup → answerability → hallucination detection on the same backend object — that's the core "same call, different intrinsic" point in action.
- Callout now flags the server-side plugin as a prerequisite - Setup section distinguishes server environment (granite-switch plugin) from application environment (mellea[switch]) - Inline comment explains api_key="EMPTY" for readers unfamiliar with vLLM's API key handling Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
- "twelve-plus intrinsics" → "a dozen" (exact count from model_ids.py
is 12; future-proofed phrasing as the curated set evolves)
- Adapter config download description now mentions JSON + YAML, matching
the later paragraph that names adapter_index.json and per-adapter
io.yaml
- Linked granitelib-{rag,core,guardian} HF repos at first mention
- Linked the granite-switch plugin package on PyPI at first mention
- Try-it section now points at the top-level Mellea repo and the
intrinsics overview docs in addition to the OpenAI-backend page
Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
The earlier draft singled out two specific Apple Silicon MLX servers (omlx, vmlx) as candidates. The investigation is broader than that — the open question is "what's the macOS story for Switch given Ollama doesn't support it", not a comparison of named MLX runtimes. Reword the reviewer note to match that framing. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
The earlier draft led with "managing adapter weights is painful" — but
that hook only resonates with readers already running intrinsics on
PEFT. Most readers come in cold and want to know what they get.
Reframed the post around the value users actually see: dropping
validation and requirement checks into a Mellea program is one
function call. The hook now shows the trio of calls
(check_answerability, flag_hallucinated_content, requirement_check)
and the rest of the post follows: what Switch is, how to run it, the
demo, and where it fits.
Cuts:
- "What it costs to ship" — its core point ("a code change, not an
infrastructure change") moved into the hook; the matrix-of-binaries
detail was mellea-internal noise
- "Where this is going" — issue/epic references and roadmap detail
read as project status and made the post feel WIP-heavy
- "When to reach for Switch vs PEFT" comparison table — replaced with
a short "When this fits" closer that links to the intrinsics
overview docs for readers who want the full picture
- The intrinsics-vs-Switch terminology section is now a single tight
"How it works" paragraph
The reader still gets enough to run it end-to-end; everything beyond
that lives behind the docs links in "Try it".
Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Previous version said the validators are "baked into the model weights" but stopped there. The genuinely distinctive thing about Switch — the reason it can be drop-in where alternatives can't — is that the *routing* between validators is part of the model itself, not an orchestration layer wrapped around it. Spelled that out by contrasting with the two mechanisms a reader is most likely to know (LoRA hot-swap and LLM-as-judge), in one short paragraph each. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
The previous single reviewer note covered macOS only. Cross-checking against the granite-switch repo README, the HF model card, and the mellea OpenAI backend doc surfaced two more environment questions that need verification before merge: 1. Whether to lead with `granite-switch[vllm20]` (CUDA 13+) or `granite-switch[vllm]` (broader CUDA 12.x compat). Upstream prefers `[vllm]`; we've only validated `[vllm20]` on LSF. 2. Whether `--enable-auto-tool-choice --tool-call-parser granite4` is actually required. Upstream sources all omit it, but internal testing on bvllm needed it to make intrinsics dispatch. Each callout now sits at the install/serve line it relates to, so a reviewer reading the post can resolve them in place. The macOS note keeps its existing position. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Retitle to "one checkpoint, every intrinsic" — drops "adapter wrangling" framing the rewritten hook no longer leans on, and matches the SVG tagline. Removes the trailing "the dispatch happens inside OpenAIBackend" aside which is mellea-internal detail readers don't need. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
ajbozarth
left a comment
ajbozarth
left a comment
There was a problem hiding this comment.
Overall looks good, only one note outside the items you've already called out, and making sure to update the date to the actual expected date when you're ready to schedule.
|
|
||
| ## Setting it up | ||
|
|
||
| The [`granite-switch`](https://pypi.org/project/granite-switch/) plugin package |
There was a problem hiding this comment.
The macOS-not-supported note currently lives only inside the reviewer-note callout below, which gets deleted before merge. Worth surfacing it at the top of this section in some form so macOS readers can bail before pip install.
There was a problem hiding this comment.
It's my intent to sort out the macOS limitation if I can as noted in the pr. I'll keep this conv. open as a reminder of that action item, since all the callouts need removing and the overall body needs to flow. I just need to try it out on macOS
|
Terminology / timing note The post uses Plan: build the example against the current code rather than waiting on the rename. For this pass we'll just get the positioning terminology right — lead with "adapter function" (the term already shipped in the public Granite Switch material), and define "intrinsic" as the earlier name. The example's class names stay as-is for now; we'll update the blog to the new symbols once the epic lands (the deprecation plan keeps the current code runnable in the meantime). Separately, one dependency on the epic side: the epic's own naming work still needs to realign with the shipped Granite Switch vocabulary. Its current target ( |
|
Blog updated: terminology + background Reader-facing prose now leads with "adapter function" to match the public Granite Switch vocabulary; "intrinsic" is defined once as the earlier name. Code symbols are kept as the current code ( Also added:
Open items parked for the next pass (previously inline reviewer notes, removed from the draft so they don't ship):
|
2 participants
Closes #58
Summary
Granite Switch bakes a curated set of Granite intrinsics into a single
vLLM-served checkpoint. This post leads with the end-user value
(adding validation, requirement checks, hallucination detection to a
Mellea program is a single function call), shows two intrinsics
running end-to-end against a live vLLM server, and explains the
architectural shift that makes it work — the routing between
behaviours is part of the model itself, not orchestration around it.
Content checklist
LLM-as-judge framed as the alternatives the reader knows)
"answerable"/"unanswerable")Technical validation
ibm-granite/granite-switch-4.1-3b-preview: both code snippets runclean; output values (
"answerable"/"unanswerable","faithful"/"unfaithful") match the post exactly.Verification status
bvllm)To run vLLM for validation (internal tooling —
bvllm):Launches vLLM on an IBM LSF cluster and returns an OpenAI-compatible
endpoint.
Open questions for the reviewer
Each of these has a callout at the relevant point in the post; this
list is the consolidated todo. The post is honest about being preview
software, but several setup details are stricter than upstream docs
and need a re-test before publish.
granite-switch[vllm20]vsgranite-switch[vllm]— upstreamgranite-switchREADME leads with[vllm](broad CUDA 12.x compat,vLLM 0.19.x); we've only validated
[vllm20](CUDA 13+, vLLM 0.20+)on LSF. Decide which to lead with, and consider showing only one to
keep install simple.
--enable-auto-tool-choice --tool-call-parser granite4— theseflags were added after
bvllmtesting showed intrinsics didn'tdispatch without them. The upstream
granite-switchREADME, the HFmodel card, and
docs/docs/integrations/openai.mdall omit them.Re-test against a vanilla
vllm serve <model>invocation; ifdispatch works without them, drop to match upstream.
macOS path — Switch doesn't run under Ollama. A macOS option is
under investigation; nothing confirmed. Expand the setup section if
a macOS path lands before merge.
Claims avoided
coarse-grained expert switching).
OpenAIBackendonly onmaintoday.
a runtime API call.
-preview.Notes for reviewer
All code snippets match the logic in
docs/examples/granite-switch/in the main Mellea repository — they'retested by the e2e suite and confirmed against a live vLLM instance.
The three reviewer callouts in the post (one per open question above)
are tagged
**Reviewer note —**so they're easy to grep before publish.FYI @ajbozarth — this is a candidate to publish soon; flagging early in
case you want to weigh in on framing or timing.