RubricBasedEvaluator _normalize_text too basic — fails on judge model markdown output

[tottenjordan]

Description

_normalize_text() in rubric_based_evaluator.py only does text.lower().strip(). When LLM judge models return rubric verdicts with markdown formatting (bullets, smart quotes, extra whitespace, non-ASCII characters), the exact-match rubric lookup fails silently, producing incorrect scores.

Reproduction

Judge model returns: "• The response correctly identifies the tool"
Expected rubric: "the response correctly identifies the tool"
_normalize_text() produces: "• the response correctly identifies the tool"
Result: No match → score defaults to 0 or lowest rubric

Common Patterns That Fail

• Leading bullets: •, *, -
• Smart quotes: "...", '...'
• Non-ASCII: accented characters, em-dashes
• Multi-space: "the response" vs "the response"
• Trailing whitespace/newlines

Suggested Fix

Enhanced normalization:

def _normalize_text(text: str) -> str:
    if not isinstance(text, str):
        return ""
    text = re.sub(r'^[\s*•\-]+', '', text)   # Strip leading bullets
    text = re.sub(r'[\s*•\-]+$', '', text)   # Strip trailing
    text = re.sub(r'\s+', ' ', text)          # Collapse whitespace
    text = text.encode('ascii', 'ignore').decode()  # Remove non-ASCII
    return text.lower().strip()

Additionally, a substring fallback when exact match fails would prevent silent scoring failures:

# If exact match fails, try substring match
for rubric_text, score in rubric_map.items():
    if normalized_response in rubric_text or rubric_text in normalized_response:
        return score

Impact

Without this fix, GEPA optimization produces unreliable rubric-based scores (rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1), leading to suboptimal prompt evolution.

Environment

• google-adk 2.2.0
• Judge models: gemini-2.5-pro, gemini-3.5-flash

[surajksharma07]

@tottenjordan Thanks for the clear writeup — reproduced this on 2.2.0 with a quick offline test: every pattern you listed (bullets, bold, double spaces, smart quotes) misses the exact-match lookup and the rubric just gets dropped from the average with only a warning log.

Your normalization direction works, though I'd suggest two tweaks: swap the ascii-ignore step for NFKC normalization plus a smart-quote/dash mapping so non-English rubrics don't get mangled, and add a uniqueness guard to the substring fallback so two similar rubrics can't cross-match.

Since _normalize_text is applied to both sides of the comparison, strengthening it is fairly low-risk.

Could you try that variant as a workaround and verify it fixes the scores on your GEPA runs?

If it holds up after thorough testing, it would make a solid PR — and a raw failing judge transcript from your runs would be a great regression fixture to include!

[tottenjordan]

Thanks @surajksharma07 — tried both tweaks, they work well together.

1. NFKC + smart-char mapping instead of ascii-ignore:

Replaced .encode('ascii', 'ignore').decode('ascii') with unicodedata.normalize('NFKC', text) plus an explicit translation table for smart quotes/dashes:

import unicodedata, re

_SMART_CHARS = {
    0x2018: "'", 0x2019: "'",  # smart single quotes
    0x201C: '"', 0x201D: '"',  # smart double quotes
    0x2013: "-", 0x2014: "-",  # en/em dashes
    0x2026: "...",             # ellipsis
}

def _fuzzy_normalize(text):
    text = unicodedata.normalize('NFKC', text)
    text = text.translate(_SMART_CHARS)
    text = re.sub(r'^[\s*•\-"\']+', '', text)
    text = re.sub(r'[\s*•\-"\']+$', '', text)
    text = re.sub(r'\s+', ' ', text)
    return text.lower().strip()

This preserves accented characters (e.g. réponse stays réponse instead of rponse) while still normalizing all the judge garbling patterns we've seen:

Input	Normalized	Match
- The response correctly uses tools	the response correctly uses tools	✅
* **The response correctly uses tools**	the response correctly uses tools	✅
“The response correctly uses tools”	the response correctly uses tools	✅
— The response correctly uses tools	the response correctly uses tools	✅
The response correctly uses tools	the response correctly uses tools	✅

2. Uniqueness guard on substring fallback:

Changed the substring fallback from first-match-wins to only-if-unique:

# Before (fragile — first match wins, could cross-match similar rubrics)
for config_text, r in normalized_map.items():
    if config_text in norm_text or norm_text in config_text:
        rubric = r
        break

# After (only accept if exactly one rubric matches)
candidates = [r for ct, r in normalized_map.items()
              if ct in norm_text or norm_text in ct]
if len(candidates) == 1:
    rubric = candidates[0]

Verified: when two rubrics share a common substring (e.g., "uses tools correctly" vs "uses tools efficiently"), a judge response of just "uses tools" correctly gets rejected instead of arbitrarily matching the first one.

Test results: 107 tests pass on our production codebase with both changes applied. We'll run this through our next GEPA optimization pipeline to validate on real judge transcripts. Happy to include a failing transcript as a regression fixture once we capture one.