Skip to content

TaskProtocol.max_retries and TimeoutAction.RETRY/EXTEND are declared but never consumed #62

@cemde

Description

@cemde

Summary

Two retry-related fields on TaskProtocol are exposed in the public API and documented as functional, but the execution loop never reads them. They are silent no-ops for users.

Affected fields

  1. TaskProtocol.max_retriesmaseval/core/task.py:45-52. Docstring says "Maximum retry attempts for transient failures (not timeouts)." Grepping maseval/core/ shows the only references are the dataclass field itself and its to_dict() serialization. Benchmark._execute_task_repetition (maseval/core/benchmark.py:1069-1255) never reads it.

  2. TimeoutAction.RETRY and TimeoutAction.EXTENDmaseval/core/task.py:21-26. The TaskTimeoutError handler at benchmark.py:1139-1159 records the timeout and returns. It does not branch on task.protocol.timeout_action. So SKIP, RETRY, and EXTEND all behave identically (skip).

Reproduction

```python
from maseval import Task
from maseval.core.task import TaskProtocol, TimeoutAction

task = Task(
query="...",
protocol=TaskProtocol(max_retries=5, timeout_action=TimeoutAction.RETRY, timeout_seconds=1.0),
)

Run a benchmark where this task raises a transient error or times out.

Observed: a single failed report is produced, max_retries / RETRY are ignored.

Expected: up to 5 retries on transient errors; one retry on timeout.

```

Suggested resolution

Either:

  • Wire the fields into `_execute_task_repetition` (retry loop around the execution `try` block; branch on `timeout_action` in the timeout handler), or
  • Mark the fields as "not implemented" in their docstrings and raise on non-default values, so downstream users do not silently rely on them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions