Summary
Two retry-related fields on TaskProtocol are exposed in the public API and documented as functional, but the execution loop never reads them. They are silent no-ops for users.
Affected fields
-
TaskProtocol.max_retries — maseval/core/task.py:45-52. Docstring says "Maximum retry attempts for transient failures (not timeouts)." Grepping maseval/core/ shows the only references are the dataclass field itself and its to_dict() serialization. Benchmark._execute_task_repetition (maseval/core/benchmark.py:1069-1255) never reads it.
-
TimeoutAction.RETRY and TimeoutAction.EXTEND — maseval/core/task.py:21-26. The TaskTimeoutError handler at benchmark.py:1139-1159 records the timeout and returns. It does not branch on task.protocol.timeout_action. So SKIP, RETRY, and EXTEND all behave identically (skip).
Reproduction
```python
from maseval import Task
from maseval.core.task import TaskProtocol, TimeoutAction
task = Task(
query="...",
protocol=TaskProtocol(max_retries=5, timeout_action=TimeoutAction.RETRY, timeout_seconds=1.0),
)
Run a benchmark where this task raises a transient error or times out.
Observed: a single failed report is produced, max_retries / RETRY are ignored.
Expected: up to 5 retries on transient errors; one retry on timeout.
```
Suggested resolution
Either:
- Wire the fields into `_execute_task_repetition` (retry loop around the execution `try` block; branch on `timeout_action` in the timeout handler), or
- Mark the fields as "not implemented" in their docstrings and raise on non-default values, so downstream users do not silently rely on them.
Summary
Two retry-related fields on
TaskProtocolare exposed in the public API and documented as functional, but the execution loop never reads them. They are silent no-ops for users.Affected fields
TaskProtocol.max_retries—maseval/core/task.py:45-52. Docstring says "Maximum retry attempts for transient failures (not timeouts)." Greppingmaseval/core/shows the only references are the dataclass field itself and itsto_dict()serialization.Benchmark._execute_task_repetition(maseval/core/benchmark.py:1069-1255) never reads it.TimeoutAction.RETRYandTimeoutAction.EXTEND—maseval/core/task.py:21-26. TheTaskTimeoutErrorhandler atbenchmark.py:1139-1159records the timeout and returns. It does not branch ontask.protocol.timeout_action. SoSKIP,RETRY, andEXTENDall behave identically (skip).Reproduction
```python
from maseval import Task
from maseval.core.task import TaskProtocol, TimeoutAction
task = Task(
query="...",
protocol=TaskProtocol(max_retries=5, timeout_action=TimeoutAction.RETRY, timeout_seconds=1.0),
)
Run a benchmark where this task raises a transient error or times out.
Observed: a single failed report is produced, max_retries / RETRY are ignored.
Expected: up to 5 retries on transient errors; one retry on timeout.
```
Suggested resolution
Either: