clusterd-test-driver: headless frontend to clusterd for scripted compute tests

[antiguru]

Motivation

environmentd couples the compute protocol to the full SQL and catalog stack, which makes targeted compute experiments slow to set up and hard to control.
This adds a generic headless frontend to clusterd that drives the compute protocol directly from a script, so a test controls the exact persist state, the exact commands the replica receives, and the exact timestamps.
Design doc: doc/developer/design/20260612_headless_clusterd_test_driver.md.

Description

New crate mz-clusterd-test-driver, split into a generic mechanism, a dataflow builder, and a JSON scripting layer.

The mechanism hosts the persist PubSub server, performs the controller handshake over CTP, and exposes a Driver that sends any ComputeCommand, submits dataflows without auto-scheduling, observes merged frontiers and the raw response stream, and peeks.
DataflowBuilder assembles dataflows from generic parts (import_persist, build over MIR, export_index), owning the MIR-to-LIR lowering, the RenderPlan conversion, the CollectionMetadata attachment, and the SqlRelationType-versus-ReprRelationType bookkeeping; a RenderPlan cannot be hand-built outside mz-compute-types, so it uses the real lowering pipeline.

The driver is a JSON command reader: it reads commands from DRIVER_SCRIPT (or stdin) and writes one JSON response per line, exiting non-zero if any command failed.
A script declares relations with define_schema, writes synthetic or explicit rows, submits and schedules dataflows, drives allow_compaction, awaits frontiers, and asserts peek counts with expect_count.
reconnect re-handshakes and stops in the reconciliation window (before initialization_complete), and expect_error runs a nested command and asserts it fails; the builder returns errors instead of panicking on malformed input, so bad script input is assertable.
Explicit row values parse through cell_from_json (JSON value + ColumnType -> Datum), the literal shim a future full-MIR define will reuse.
The scenarios — index, deep-history, side-effects, multi-dataflow, reconciliation, and error-behavior — are JSON scripts under test/clusterd-test-driver/scripts/, not compiled Rust.

The handshake version comes from mz_persist_client::BUILD_INFO so it matches the connected clusterd, and the peek response stash is disabled so peeks return their rows inline.

Verification

mzcompose runs each scenario script against a real clusterd, with the composition directory mounted at /workdir so the scripts are readable in the container.
The asserting scripts fail the run on a wrong count, error-behavior asserts a set of expected failures via expect_error, and multi-dataflow reproduces a current limitation and exits 0 by design.
Crate unit tests cover the direct-write round trip, the frontier merge, the lowered dataflow structure, schema and datum parsing, and command serialization.
The local runner run-local.py drives the same scripts on the host for profiling.

[antiguru]

What we're missing is a specific language to script the headless driver. Encoding the interactions in Rust is a good first step, but ultimately we want something that is easier to iterate on. I could even imagine Python scripts that we load through pyo3 or so.

[antiguru]

Agreed on a scripting language being the goal. I captured it under "Future work" in the design doc: the mechanism is already a thin scriptable surface (send, submit_dataflow, schedule, expect_frontier, peek, subscribe_raw), so a declarative script or Python-via-pyo3 layer would bind to it rather than replace it. Left it out of this PR to keep the first step focused.

[def-]

I have once built something very similar at a previous company, also using json inputs to test an execution engine directly instead of going through the entire database stack. Back then the test framework turned out to not be very useful since it was easier to write unit tests and full system tests instead of the middle-ground headless json tests.

Do we have some concrete examples of regression tests we've wanted to build and failed before, which would be possible with clusterd-test-driver?

I still think it's a worthy experiment, and I can especially imagine that building a fuzzer for clusterd-test-driver with some invariants could lead us to interesting bugs.