feat(evaluation): add VLMMetrics

[davidberenstein1957]

Add ImageRewardMetric for evaluating image-text alignment using ImageReward library.

[begumcig]

Thank you so much David! I have asked some questions because I am not really familiar with litellm and the requirements using this specific framework brings. I think the metric classes themselves look really good, just needs some tweaks here and there! And really apologies for the delay in reviewing 💞

[codacy-production]

Not up to standards ⛔

🔴 Issues 2 critical · 16 high · 19 medium · 63 minor

Alerts:
⚠ 100 issues (≤ 0 issues of at least minor severity)

Results:
100 new issues

Category	Results
UnusedCode	1 medium
Documentation	62 minor
ErrorProne	2 high
Security	14 high
CodeStyle	1 minor
Complexity	2 critical 18 medium

View in Codacy

🟢 Metrics 462 complexity · 83 duplication

Metric	Results
Complexity	462
Duplication	83

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}

[begumcig]

Hey David, great updates on these! I left some specific comments below, but the general theme is that there’s a bit of a mismatch between the current implementation and the original research methodologies.

The VLM infrastructure you’ve built is a great foundation, but we’re currently missing some of the logic that actually defines these benchmarks. Without those steps, our results might be hard to compare with the official papers. Let me know what you think!!

[llcnt]

@cursor review

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: aggregation parameter stored but never applied in scoring
  • QAAccuracyMetric now validates and applies aggregation, using strict binary scoring for all_or_nothing instead of always averaging per-question scores.
• ✅ Fixed: Chinese language heuristic misclassifies EN-only rows
  • The OneIG language heuristic now only infers Chinese from prompt-only rows when actual CJK characters are present, preventing EN prompt-only rows from being routed to _zh Q_D files.
• ✅ Fixed: Text score uses unnormalized distance favoring short texts
  • TextScoreMetric now stores normalized Levenshtein distance by dividing by ground-truth character length so the reported mean matches character error rate behavior across varying text lengths.

Or push these changes by commenting:

@cursor push 6e88aefdac

Preview (6e88aefdac)

diff --git a/src/pruna/data/datasets/prompt.py b/src/pruna/data/datasets/prompt.py
--- a/src/pruna/data/datasets/prompt.py
+++ b/src/pruna/data/datasets/prompt.py
@@ -139,11 +139,15 @@
     lang = row.get("language") or row.get("lang")
     if isinstance(lang, str) and lang.lower() in {"zh", "zh-cn", "zh_cn", "chinese", "cn"}:
         return True
-    if row.get("prompt_zh"):
+    if row.get("prompt_zh") or row.get("prompt_cn"):
         return True
     prompt = row.get("prompt")
     prompt_en = row.get("prompt_en")
-    return bool(prompt and not (isinstance(prompt_en, str) and prompt_en.strip()))
+    if not (isinstance(prompt, str) and prompt.strip()):
+        return False
+    if isinstance(prompt_en, str) and prompt_en.strip():
+        return False
+    return any("\u4e00" <= ch <= "\u9fff" for ch in prompt)
 
 
 def _oneig_qd_prefix(row: dict) -> str:

diff --git a/src/pruna/evaluation/metrics/metric_qa_accuracy.py b/src/pruna/evaluation/metrics/metric_qa_accuracy.py
--- a/src/pruna/evaluation/metrics/metric_qa_accuracy.py
+++ b/src/pruna/evaluation/metrics/metric_qa_accuracy.py
@@ -105,6 +105,11 @@
         self.call_type = get_call_type_for_single_metric(call_type, self.default_call_type)
         self.add_state("scores", [])
         self.aggregation = kwargs.pop("aggregation", "mean")
+        if self.aggregation not in {"mean", "all_or_nothing"}:
+            raise ValueError(
+                "qa_accuracy aggregation must be one of {'mean', 'all_or_nothing'}. "
+                f"Got: {self.aggregation!r}."
+            )
 
     def _extract_questions(self, gt: Any, n: int) -> List[List[str]]:
         if isinstance(gt, (list, tuple)) and len(gt) >= n:
@@ -151,7 +156,10 @@
                 ["Yes"] * len(questions),
                 response_format=self.response_format,
             )
-            score = float(np.mean(scores))
+            if self.aggregation == "all_or_nothing":
+                score = float(all(float(s) == 1.0 for s in scores))
+            else:
+                score = float(np.mean(scores))
             self.scores.append(score)
 
     def compute(self) -> MetricResult:

diff --git a/src/pruna/evaluation/metrics/metric_text_score.py b/src/pruna/evaluation/metrics/metric_text_score.py
--- a/src/pruna/evaluation/metrics/metric_text_score.py
+++ b/src/pruna/evaluation/metrics/metric_text_score.py
@@ -172,7 +172,7 @@
 @MetricRegistry.register("text_score")
 class TextScoreMetric(_BaseVLMOCRTextMetric):
     """
-    OCR then mean Levenshtein distance to ground truth (lower is better).
+    OCR then mean normalized Levenshtein distance (character error rate, lower is better).
 
     Registry: ``ocr_levenshtein`` (descriptive) and ``text_score`` (legacy).
 
@@ -240,7 +240,8 @@
     def _accumulate_sample(self, text_gt: str, ocr_text: str) -> None:
         norm_gt = normalize_text_simple(text_gt)
         norm_ocr = normalize_text_simple(ocr_text)
-        self.scores.append(levenshtein(norm_ocr, norm_gt))
+        gt_len = max(len(norm_gt), 1)
+        self.scores.append(float(levenshtein(norm_ocr, norm_gt) / gt_len))
 
     def _compute_result_value(self) -> float:
         if not self.scores:

diff --git a/tests/data/test_oneig_loader.py b/tests/data/test_oneig_loader.py
--- a/tests/data/test_oneig_loader.py
+++ b/tests/data/test_oneig_loader.py
@@ -34,6 +34,18 @@
     assert prompt_mod._oneig_qd_prefix(row) == "anime_zh"
 
 
+def test_oneig_qd_prefix_prompt_only_en_row_stays_en() -> None:
+    """Prompt-only EN rows must not be misclassified as Chinese."""
+    row = {
+        "category": "General_Object",
+        "id": "001",
+        "prompt": "a red apple on a table",
+        "prompt_en": "",
+        "class": "None",
+    }
+    assert prompt_mod._oneig_qd_prefix(row) == "object"
+
+
 def test_to_oneig_record_multilingualism_fills_questions() -> None:
     """Synthetic Multilingualism row resolves Q_D from merged index."""
     qb = {"multilingualism_zh_000": {"questions": {"1": "现场是不是颁奖典礼？"}, "dependencies": {"1": [0]}}}

diff --git a/tests/evaluation/test_vlm_metrics.py b/tests/evaluation/test_vlm_metrics.py
--- a/tests/evaluation/test_vlm_metrics.py
+++ b/tests/evaluation/test_vlm_metrics.py
@@ -146,6 +146,22 @@
 
 
 @pytest.mark.cpu
+def test_qa_accuracy_aggregation_modes() -> None:
+    mock_vlm = MagicMock(spec=BaseVLM)
+    mock_vlm.score.return_value = [1.0, 0.0]
+    images = _dummy_image(batch=1)
+    aux = [{"questions": {"1": "Q1", "2": "Q2"}}]
+
+    mean_metric = QAAccuracyMetric(vlm=mock_vlm, vlm_type="litellm", device="cpu", aggregation="mean")
+    mean_metric.update(["a prompt"], aux, images)
+    assert mean_metric.compute().result == pytest.approx(0.5)
+
+    strict_metric = QAAccuracyMetric(vlm=mock_vlm, vlm_type="litellm", device="cpu", aggregation="all_or_nothing")
+    strict_metric.update(["a prompt"], aux, images)
+    assert strict_metric.compute().result == pytest.approx(0.0)
+
+
+@pytest.mark.cpu
 def test_get_vlm_returns_custom() -> None:
     custom = MagicMock(spec=BaseVLM)
     out = get_vlm(vlm=custom, vlm_type="litellm", model_name="gpt-4o")
@@ -183,6 +199,19 @@
 
 
 @pytest.mark.cpu
+def test_text_score_uses_normalized_edit_distance() -> None:
+    mock_vlm = MagicMock(spec=BaseVLM)
+    mock_vlm.generate.side_effect = [["abxde"], ["ax"]]
+    metric = TextScoreMetric(vlm=mock_vlm, vlm_type="litellm", device="cpu")
+
+    metric.update(["p1"], ["abcde"], _dummy_image(batch=1))
+    metric.update(["p2"], ["ab"], _dummy_image(batch=1))
+
+    assert metric.scores == pytest.approx([0.2, 0.5])
+    assert metric.compute().result == pytest.approx(0.35)
+
+
+@pytest.mark.cpu
 def test_text_score_registry_aliases() -> None:
     from pruna.evaluation.metrics.registry import MetricRegistry

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7435679. Configure here.}

[cursor]

aggregation parameter stored but never applied in scoring

High Severity

QAAccuracyMetric accepts and stores self.aggregation (e.g. "all_or_nothing" for GenEval) but never reads it. The update method always uses np.mean(scores) on line 154, giving partial credit to every image. For GenEval, where Task.from_benchmark explicitly passes aggregation="all_or_nothing", this produces inflated scores instead of the official binary pass/fail (1 only if every atomic question passes, 0 otherwise).

Additional Locations (2)

• src/pruna/evaluation/task.py#L108-L111
• src/pruna/evaluation/benchmark_vlm_integration.py#L128-L130

 

^{Reviewed by Cursor Bugbot for commit 7435679. Configure here.}

[cursor]

Chinese language heuristic misclassifies EN-only rows

Medium Severity

_oneig_alignment_language_zh falls through to a heuristic on line 146 that returns True (Chinese) when prompt is non-empty but prompt_en is absent or empty. Rows in the EN config (OneIG-Bench) that use prompt as their primary text field without a separate prompt_en column would be misclassified as Chinese, causing _oneig_qd_prefix to select *_zh question-dependency files instead of the English ones.

 

^{Reviewed by Cursor Bugbot for commit 7435679. Configure here.}

[cursor]

Text score uses unnormalized distance favoring short texts

Medium Severity

TextScoreMetric._accumulate_sample stores raw Levenshtein edit distance without normalizing by text length. Since the metric is lower_is_better, longer ground-truth texts inherently produce higher (worse) scores than shorter texts with the same number of errors, making scores incomparable across samples of different length. The benchmark description mentions "mean character error rate" but the implementation returns raw edit counts.

 

^{Reviewed by Cursor Bugbot for commit 7435679. Configure here.}

[llcnt]

The code change (>5K) is too large for me :( Could you point me to some place where my review can be particularly useful ? I can definitely dedicate some time to review some files, but not all 😓

[davidberenstein1957]

Hi @llcnt, if you have time. You can take a look at the LLM2CLIP implementation :)

[davidberenstein1957]

---

Code review

Found 2 issues:

1. qa_accuracy ignores aggregation (including GenEval all_or_nothing). The metric stores aggregation in __init__ but update always uses np.mean(scores) per image and compute always uses np.mean(self.scores), so partial credit remains and the final score is not strict per-image all-or-nothing. This conflicts with the GenEval benchmark text and Task.from_benchmark wiring that pass aggregation=\"all_or_nothing\".

pruna/src/pruna/evaluation/metrics/metric_qa_accuracy.py

Lines 107 to 168 in 7435679

	self.aggregation = kwargs.pop("aggregation", "mean")
	def _extract_questions(self, gt: Any, n: int) -> List[List[str]]:
	if isinstance(gt, (list, tuple)) and len(gt) >= n:
	out = []
	for i in range(n):
	v = gt[i]
	if isinstance(v, dict) and "questions" in v:
	qs = v["questions"]
	out.append(list(qs.values()) if isinstance(qs, dict) else list(qs))
	else:
	out.append([])
	return out
	return [[]] * n
	def update(self, x: List[Any] | torch.Tensor, gt: torch.Tensor, outputs: torch.Tensor) -> None:
	"""
	Update the metric with new batch data.
	Parameters
	----------
	x : List[Any] | torch.Tensor
	The input data.
	gt : torch.Tensor
	The ground truth (questions per image).
	outputs : torch.Tensor
	The output images.
	"""
	inputs = metric_data_processor(x, gt, outputs, self.call_type)
	images = _process_images(inputs[0])
	auxiliaries = inputs[1] if len(inputs) > 1 else []
	questions_per_image = self._extract_questions(auxiliaries, len(images))
	for i, image in enumerate(images):
	questions = questions_per_image[i] if i < len(questions_per_image) else []
	if not questions:
	aux = auxiliaries[i] if i < len(auxiliaries) else {}
	raise ValueError(
	"qa_accuracy requires 'questions' in auxiliaries. "
	"Use a benchmark that provides it (e.g. GenEval, DPG, OneIG). "
	f"Got aux keys: {list(aux.keys()) if isinstance(aux, dict) else 'not a dict'}."
	)
	scores = self.vlm.score(
	[image] * len(questions),
	questions,
	["Yes"] * len(questions),
	response_format=self.response_format,
	)
	score = float(np.mean(scores))
	self.scores.append(score)
	def compute(self) -> MetricResult:
	"""
	Compute the QA accuracy score.
	Returns
	-------
	MetricResult
	The mean QA accuracy across all updates.
	"""
	if not self.scores:
	return MetricResult(self.metric_name, self.__dict__, 0.0)
	return MetricResult(self.metric_name, self.__dict__, float(np.mean(self.scores)))

pruna/src/pruna/evaluation/benchmarks.py

Lines 219 to 227 in 7435679

	Benchmark(
	name="GenEval",
	description=(
	"Compositional text-to-image benchmark with 6 categories: single object, two object, "
	"counting, colors, position, color attributes. Uses atomic yes/no questions per prompt; "
	"``Task.from_benchmark`` wires ``qa_accuracy`` with strict per-image aggregation "
	"(all questions must pass) plus ``clip_score``. For holistic VQAScore-style scoring "
	"use GenAI Bench with ``vqa``."
	),

2. _extract_questions uses [[]] * n, which aliases the same inner list n times. If the fallback branch runs with n > 1, all indices share one list object (classic Python pitfall); use e.g. [[] for _ in range(n)].

pruna/src/pruna/evaluation/metrics/metric_qa_accuracy.py

Lines 109 to 120 in 7435679

	def _extract_questions(self, gt: Any, n: int) -> List[List[str]]:
	if isinstance(gt, (list, tuple)) and len(gt) >= n:
	out = []
	for i in range(n):
	v = gt[i]
	if isinstance(v, dict) and "questions" in v:
	qs = v["questions"]
	out.append(list(qs.values()) if isinstance(qs, dict) else list(qs))
	else:
	out.append([])
	return out
	return [[]] * n

(No CLAUDE.md in repo root or modified directories; review used PR head 7435679005957aaa13413714e7af9aca27938812.)

Generated with Claude Code

[davidberenstein1957]

Update on review #545 (comment)

1. aggregation / GenEval all_or_nothing — Already implemented on this branch: in QAAccuracyMetric.update, per-image score is 1.0 only when every per-question score passes (all_or_nothing), otherwise mean over question scores for the default. Covered by tests/evaluation/test_vlm_metrics.py (test_qa_accuracy_all_or_nothing_*).

2. _extract_questions fallback — Fixed: replaced return [[]] * n with return [[] for _ in range(n)] so the fallback path does not alias one empty list across n slots (src/pruna/evaluation/metrics/metric_qa_accuracy.py).