feat(depot): watermark-aligned delta reclaim and retention engine#5221
Conversation
Collapse SQLite Depot history retention onto the hot watermark: - DELTA rows are reclaimed at or below the watermark with no per-shard or PIDX proof; the install that advances the watermark is the coverage proof. - COMMITS/VTX below the watermark survive only as keep-set islands (pins plus retained PITR interval representatives); superseded SHARD versions are GC'd once no covered txid reads through them. - Snapshot targets (pins, forks, restores) are fenced to covered-or-unfolded txids; versionstamp targets snap down to the newest covered point. - Truncate preserves shard history via pruned versions instead of deleting it; folding is per-coverage-txid and truncate-aware. - Bucket-fork pins materialize on covered points with a fail-safe fallback. - PITR policy scope is denormalized onto the branch record (vbare v2) to fix the plan-vs-validate livelock and drop the global pointer scan. - Hardening: bounded/paged reclaim scans, quota credit on delete, immediate drain re-arm, orphan-stage cleanup queue, fail-closed PIDX decode, future-clock interval coverage.
|
Stack for rivet-dev/rivet
Get stack: |
Code Review — watermark-aligned delta reclaim and retention engineOverviewThis PR replaces the old per-shard coverage-proof approach to delta reclaim with a simpler watermark rule: any DELTA at or below the hot watermark is deletable, with no PIDX or shard-version proof required. The constraint that makes this safe is a new fork-alignment fence that prevents forks, pins, and restores from targeting uncovered below-watermark txids. The PR also fixes a plan/validate livelock caused by reading the default PITR policy at planning time, adds quota credits on delete, and overhauls truncate to preserve shard history. The design is coherent and the documentation in Issues
The existing rule says:
The new truncate implementation keeps old SHARD versions intact for PITR history and instead writes a new pruned version at
fn now_ms() -> Result<i64> {
let millis = SystemTime::now()...
}Every other call site in this file threads
fn tracked_row_bytes(key: &[u8], value: &[u8]) -> i64 {
i64::try_from(key.len() + value.len()).unwrap_or(i64::MAX)
}
i64::try_from(key.len().saturating_add(value.len())).unwrap_or(i64::MAX)Staged shard cleanup hardcodes In udb::compare_and_clear(
tx,
&keys::branch_compaction_stage_hot_shard_key(
input.database_branch_id,
input.job_id,
output_ref.shard_id,
output_ref.as_of_txid,
0, // chunk_idx always 0?
),
staged_blob,
);This is consistent with the existing read (also chunk 0), so it's correct if staged shards are always single-chunk. If that invariant holds, a
let pending = state.pending_stage_cleanups.remove(0);This shifts all remaining elements on each drain. The queue is expected to be tiny (one or two entries), but Observations / Questions
The old code returned Shard GC cursor is opaque raw key bytes
The comment about "Rejected executes still commit this transaction, which is load-bearing: pin materializations must persist" is correct and important. It might be worth surfacing this in Positive highlights
🤖 Generated with Claude Code |
1 participant
Collapse SQLite Depot history retention onto the hot watermark:
PIDX proof; the install that advances the watermark is the coverage proof.
retained PITR interval representatives); superseded SHARD versions are GC'd
once no covered txid reads through them.
txids; versionstamp targets snap down to the newest covered point.
folding is per-coverage-txid and truncate-aware.
the plan-vs-validate livelock and drop the global pointer scan.
drain re-arm, orphan-stage cleanup queue, fail-closed PIDX decode,
future-clock interval coverage.