feat(sidecar): cross-process locking + atomic writes, and wire provenance/history CLI (V-L2-F4, #150)

CHANGELOG.adoc

@@ -8,6 +8,27 @@ All notable changes to verisimiser will be documented in this file.
 This format is based on https://keepachangelog.com/en/1.1.0/[Keep a Changelog],
 and this project adheres to https://semver.org/spec/v2.0.0.html[Semantic Versioning].
 
+== [0.2.0] - 2026-05-30
+
+=== Added
+* JSON-family sidecar storage backend (`[sidecar].storage = "json"`) with a
+  `[sidecar].format` key — `plain` | `ld` (JSON-LD) | `ndjson` — at full
+  runtime parity with the SQLite path: hash-chained provenance (incl. forks),
+  temporal versioning, drift, and gc (#146, V-L2-F3).
+* `sidecar::StorageKind` — single source of truth resolving
+  `[sidecar].storage` (+ `format`) to a backend; `validate`/`generate`/
+  `drift`/`gc` all defer to it.
+* Cross-process write locking for the JSON sidecar (advisory `<path>.lock`
+  with stale-steal) plus documented atomic-rename durability (#150, V-L2-F4).
+* `provenance <entity>` and `history <entity> [--at <RFC3339>]` CLI
+  subcommands now query the sidecar (sqlite + json) instead of stubbing
+  (#150, V-L2-F4).
+* `examples/json-sidecar/` manifest demonstrating the NDJSON sidecar.
+
+=== Changed
+* `verisimiser generate` emits a `sidecar_schema.{json,jsonld,ndjson}`
+  scaffold for the json family (SQL backends still emit DDL).
+
 == [0.1.0] - 2026-03-21
 
 === Phase 1 — RSR Compliance Sweep

Cargo.toml

@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: PMPL-1.0-or-later
 [package]
 name = "verisimiser"
-version = "0.1.0"
+version = "0.2.0"
 edition = "2024"
 rust-version = "1.85"
 authors = ["Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>"]

README.adoc

@@ -266,6 +266,55 @@ The `verisimiser octad` subcommand prints the active concerns from your
 manifest; `verisimiser doctor` checks that sidecars, thresholds, and
 retention bounds are configured consistently.
 
+=== Sidecar storage backends (Phase 1 schema)
+
+The Phase 1 manifest uses a single `[sidecar]` section. All octad data
+(provenance log, temporal versions, lineage edges, access policies) lives
+in one sidecar store — never in your target database:
+
+[source,toml]
+----
+[sidecar]
+# storage backend: "sqlite" (default), "postgres"/"postgresql", or "json"
+storage = "json"
+# json on-disk encoding (ignored for sql backends): "plain" | "ld" | "ndjson"
+format  = "ndjson"
+path    = ".verisim/sidecar.ndjson"
+----
+
+* **sqlite** (default) — the reference store; embedded and transactional.
+* **postgres** — the same overlay schema, emitted as PostgreSQL DDL by `generate`.
+* **json** — an append-only document store mirroring the `verisimdb_*` tables,
+  in one of three encodings:
++
+--
+** **plain** — one JSON object keyed by table name.
+** **ld** — JSON-LD (`@context` + a `@graph` of typed, `@id`-addressed nodes), for linked-data tooling.
+** **ndjson** — newline-delimited JSON, one record per line.
+--
+
+The JSON store has full parity with the SQLite runtime path: hash-chained
+provenance (including first-class forks), temporal versioning, drift, and
+gc. Writes are crash-safe (atomic temp-file + `rename`) and serialised
+across processes by an advisory lock file (`<path>.lock`).
+
+[cols="2,1,1,1",options="header"]
+|===
+| Command | sqlite | postgres | json
+
+| `generate`               | DDL | DDL | scaffold
+| `provenance` / `history` | ✓   | —   | ✓
+| `drift` / `gc`           | ✓   | —   | ✓
+|===
+
+`generate` writes `sidecar_schema.sql` for SQL backends and a
+`sidecar_schema.{json,jsonld,ndjson}` scaffold for the json family.
+`verisimiser provenance <entity>` prints the entity's chain and its
+verification status (plus any fork points); `verisimiser history <entity>
+[--at <RFC3339>]` lists temporal versions, or the point-in-time snapshot per
+modality. PostgreSQL *runtime* reads (`provenance`/`history`/`drift`/`gc`)
+are not yet implemented and refuse explicitly rather than silently no-op.
+
 == Architecture
 
 [source]

examples/README.adoc

@@ -1 +1,10 @@
+// SPDX-License-Identifier: PMPL-1.0-or-later
 = examples Pillar
+
+Worked examples of `verisimiser.toml` manifests.
+
+* **blog-db/** — augment a typical blog database (SQLite target) with the
+  octad dimensions, using the default SQLite sidecar.
+* **json-sidecar/** — the same octad augmentation, but stored in a
+  JSON-family (NDJSON) sidecar instead of SQLite. Shows `[sidecar].storage =
+  "json"` + `format`.

examples/blog-db/verisimiser.toml

@@ -23,4 +23,7 @@ enable-simulation = false
 
 [sidecar]
 storage = "sqlite"
+# To use the JSON document store instead (see examples/json-sidecar):
+#   storage = "json"
+#   format  = "ndjson"   # "plain" | "ld" | "ndjson"
 path = ".verisim/sidecar.db"

examples/json-sidecar/README.adoc

@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: PMPL-1.0-or-later
+= JSON sidecar example
+
+Demonstrates the JSON-family sidecar backend (V-L2-F3/F4): `[sidecar].storage
+= "json"` with `format = "ndjson"`. The octad data (provenance, temporal,
+lineage, access policies) is stored as an append-only NDJSON document store
+mirroring the `verisimdb_*` tables, with the same runtime parity as SQLite —
+hash-chained provenance, temporal versioning, drift, and gc.
+
+[source,sh]
+----
+# Validate the manifest (checks storage + format resolve):
+verisimiser validate -m examples/json-sidecar/verisimiser.toml
+
+# Emit the JSON sidecar scaffold:
+verisimiser generate -m examples/json-sidecar/verisimiser.toml -o .verisim
+
+# After data accrues, query it:
+verisimiser provenance -m examples/json-sidecar/verisimiser.toml <entity-id>
+verisimiser history    -m examples/json-sidecar/verisimiser.toml <entity-id> --at 2026-01-01T00:00:00Z
+verisimiser gc         -m examples/json-sidecar/verisimiser.toml --dry-run
+----
+
+Set `format` to `"ld"` for JSON-LD (`@context` + `@graph`) or `"plain"` for a
+single keyed JSON object. Writes are crash-safe (atomic rename) and serialised
+across processes by a `<path>.lock` advisory lock.

examples/json-sidecar/verisimiser.toml

@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: PMPL-1.0-or-later
+# VeriSimiser manifest: JSON-family (NDJSON) sidecar example.
+#
+# Same octad augmentation as the blog-db example, but the sidecar is an
+# append-only NDJSON document store instead of SQLite — handy when you want
+# the provenance/temporal data as plain, diff-/grep-able text. Switch
+# `format` to "ld" for JSON-LD (linked-data tooling) or "plain" for a single
+# keyed JSON object.
+
+[project]
+name = "json-sidecar-example"
+version = "0.1.0"
+description = "Example: octad augmentation with a JSON-family (NDJSON) sidecar"
+
+[database]
+backend = "sqlite"
+connection-string-env = "APP_DATABASE_URL"
+# schema-source = "schema.sql"   # optional; omitted here for a self-contained example
+
+[octad]
+enable-provenance = true
+enable-lineage = true
+enable-temporal = true
+enable-access-control = true
+enable-simulation = false
+
+[sidecar]
+storage = "json"
+format  = "ndjson"   # "plain" | "ld" | "ndjson"
+path = ".verisim/sidecar.ndjson"

src/gc.rs

@@ -65,15 +65,20 @@ fn run_gc_json(
     dry_run: bool,
     format: crate::sidecar::JsonFormat,
 ) -> Result<GcReport> {
+    use crate::sidecar::json::{self, JsonStore};
     let sidecar_path = &manifest.sidecar.path;
-    let mut store = crate::sidecar::json::JsonStore::open(sidecar_path, format)
-        .with_context(|| format!("opening json sidecar at {}", sidecar_path))?;
-    let counts = store.gc_purge(&manifest.retention, dry_run);
-    if !dry_run {
-        store
-            .save()
-            .with_context(|| format!("saving json sidecar at {}", sidecar_path))?;
-    }
+    let retention = &manifest.retention;
+    let counts = if dry_run {
+        // Read-only: count purge candidates without locking or writing.
+        let mut store = JsonStore::open(sidecar_path, format)
+            .with_context(|| format!("opening json sidecar at {}", sidecar_path))?;
+        store.gc_purge(retention, true)
+    } else {
+        // Mutating: hold the cross-process write lock across load→purge→save.
+        json::with_locked(sidecar_path, format, |store| {
+            Ok(store.gc_purge(retention, false))
+        })?
+    };
     Ok(GcReport {
         sidecar: sidecar_path.clone(),
         dry_run,