Release: merge dev into main (feature docs v4–v114, 130+ headless utils)

.gitignore

@@ -13,6 +13,7 @@ __pycache__/
 build/
 develop-eggs/
 dist/
+node_modules/
 downloads/
 eggs/
 .eggs/

CLAUDE.md

@@ -229,7 +229,7 @@ Run before every commit; fix all new findings:
 pip install ruff pylint bandit radon
 ruff check je_auto_control/
 pylint je_auto_control/
-bandit -r je_auto_control/ -x je_auto_control/test
+bandit -c pyproject.toml -r je_auto_control/   # uses [tool.bandit] excludes/skips
 radon cc je_auto_control/ -a -nc   # flags functions with CC >= C (>10)
 ```
 

autocontrol-lsp/vscode/package.json

@@ -97,7 +97,7 @@
   "devDependencies": {
     "@types/node": "^20.0.0",
     "@types/vscode": "^1.85.0",
-    "esbuild": "^0.25.0",
+    "esbuild": "^0.28.1",
     "typescript": "^5.4.0",
     "vscode-languageclient": "^9.0.1"
   }

dev_requirements.txt

@@ -9,8 +9,14 @@ qt-material==2.17
 mss==10.2.0
 defusedxml==0.7.1
 
+# Office I/O ([office] extra) — exercised by the headless Office tests.
+openpyxl==3.1.5
+python-docx==1.2.0
+python-pptx==1.0.2
+
 # Quality tooling — used by .github/workflows/quality.yml and locally.
 ruff==0.15.14
 bandit==1.9.4
 pytest==9.0.3
 pytest-timeout==2.4.0
+pytest-rerunfailures==15.1

docs/source/Eng/doc/new_features/v100_features_doc.rst

@@ -0,0 +1,46 @@
+Near-Duplicate Text Detection (SimHash / MinHash)
+=================================================
+
+``fuzzy.fuzzy_dedupe`` is O(n²) pairwise ``SequenceMatcher`` with no stable
+fingerprint, and ``image_dedup`` only hashes pixels. This adds text
+fingerprints — SimHash (Hamming-distance near-dup) and MinHash (estimated
+Jaccard) — that scale and give a reusable signature, the text analog of the
+perceptual image hash.
+
+Pure standard library (``hashlib`` / ``re``); imports no ``PySide6``. A fixed
+hash (``blake2b``, not the salted built-in ``hash()``) keeps fingerprints
+deterministic across runs and CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (
+        simhash, near_duplicates, minhash_signature, minhash_similarity,
+    )
+
+    h1 = simhash("the quick brown fox jumps over the lazy dog")
+    h2 = simhash("the quick brown fox jumps over the lazy dogs")
+    # small Hamming distance ⇒ near-duplicate
+
+    clusters = near_duplicates(docs, max_distance=12)   # groups of indices
+
+    sig_a = minhash_signature(text_a)
+    minhash_similarity(sig_a, minhash_signature(text_b))  # ~ Jaccard
+
+``simhash`` returns a ``bits``-wide fingerprint from word shingles;
+``hamming_distance`` (shared with ``image_dedup``) measures bit difference.
+``near_duplicates`` clusters texts whose SimHashes are within ``max_distance``
+bits, returning a partition of indices (singletons included).
+``minhash_signature`` / ``minhash_similarity`` give a MinHash signature and a
+Jaccard estimate for set-overlap style dedup. Run ``normalize_text`` first for
+accent/form-insensitive fingerprints.
+
+Executor commands
+-----------------
+
+``AC_simhash`` returns ``{simhash}`` for a ``text``; ``AC_near_duplicates``
+returns ``{clusters}`` for ``texts`` within ``max_distance``. Both are exposed
+as MCP tools (``ac_simhash`` / ``ac_near_duplicates``) and as Script Builder
+commands under **Data**.

docs/source/Eng/doc/new_features/v101_features_doc.rst

@@ -0,0 +1,41 @@
+Single-Series Anomaly Detection
+===============================
+
+``data_drift`` answers "did the *distribution* shift between two batches" — it
+cannot point at *which* value in one live series is anomalous — and
+``slo.burn_alerts`` only thresholds error-budget burn, not arbitrary metric
+values (latency spikes, cost spikes, CPU). This flags outliers in a single
+series via z-score, robust MAD (modified z-score), and an EWMA control chart.
+
+Pure standard library (``math`` / ``statistics``); imports no ``PySide6``. Every
+function is pure (values in, flags out), so it is fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import detect_anomalies, mad_anomalies, ewma_control
+
+    series = [10, 11, 9, 10, 12, 10, 95, 11, 10]    # index 6 is the spike
+    mad_anomalies(series)                            # [6]  (robust)
+    detect_anomalies(series, method="mad")
+    # [{index, value, score, is_anomaly}, ...]
+
+    ewma_control(values, alpha=0.5, target_mean=10, target_sigma=1)  # shift indices
+
+``detect_anomalies`` scores each value (``mad`` default, or ``zscore``) and flags
+those past the threshold (3.5 for MAD, 3.0 for z-score). ``mad_anomalies`` /
+``zscore_anomalies`` return just the flagged indices, and ``mad_scores`` /
+``zscore_scores`` the raw scores. MAD (Iglewicz-Hoaglin modified z-score) is
+robust to outliers inflating the spread, so it stays sensitive where a plain
+z-score would not. ``ewma_control`` is an EWMA control chart for sustained
+level shifts — pass ``target_mean`` / ``target_sigma`` for an in-control
+baseline (else the series' own stats).
+
+Executor command
+----------------
+
+``AC_detect_anomalies`` takes a ``values`` list (optional ``method`` /
+``threshold``) and returns ``{results}``. It is exposed as the MCP tool
+``ac_detect_anomalies`` and as a Script Builder command under **Data**.

docs/source/Eng/doc/new_features/v102_features_doc.rst

@@ -0,0 +1,35 @@
+Moving-Average Smoothing
+========================
+
+``stats.describe`` summarises a whole sample and ``timeseries`` rolls counters
+into rates, but nothing smoothed a noisy signal or weighted recent points. This
+adds trailing simple / weighted / exponentially-weighted moving averages and a
+generic rolling reducer.
+
+Pure standard library; imports no ``PySide6``. Every function is pure (values
+in, list out), so it is fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import sma, wma, ewma, rolling
+
+    sma([1, 2, 3, 4], 2)            # [1.0, 1.5, 2.5, 3.5]
+    ewma([1, 2, 3], alpha=0.5)      # [1.0, 1.5, 2.25]
+    wma(values, [1, 2, 3])          # weights align to the latest points
+    rolling(values, 5, max)         # generic trailing-window reduction
+
+``sma`` averages each trailing window of ``window`` points; ``wma`` applies the
+given weights (latest-aligned); ``ewma`` smooths with factor ``alpha`` in
+``(0, 1]``; ``rolling`` applies any reducer over each trailing window. All
+return a same-length list, so the result lines up with the input timeline (a
+``resource_profiler`` FPS/CPU series, a latency stream, etc.).
+
+Executor commands
+-----------------
+
+``AC_sma`` returns ``{series}`` for ``values`` over a ``window``; ``AC_ewma``
+returns ``{series}`` for an ``alpha``. Both are exposed as MCP tools (``ac_sma``
+/ ``ac_ewma``) and as Script Builder commands under **Data**.

docs/source/Eng/doc/new_features/v103_features_doc.rst

@@ -0,0 +1,43 @@
+Idempotency-Key Store
+=====================
+
+``resilience.RetryPolicy`` *re-executes* on retry and ``work_queue`` dedups only
+in-flight references — nothing cached the first result so a duplicate request
+returns the *same* response without re-running the side effect. This is the
+Stripe idempotency pattern: register a key, run the work once, and replay the
+stored response for any duplicate.
+
+Pure standard library (``hashlib`` / ``json``); imports no ``PySide6``. The
+clock is injectable and the store is in-memory with JSON persistence, so TTL
+expiry and replay are fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import IdempotencyStore, request_fingerprint
+
+    store = IdempotencyStore(ttl=86400)
+    state = store.begin("order-42", request_fingerprint(payload))
+    if state["status"] == "completed":
+        return state["response"]            # replay — do not re-run
+    result = charge(payload)                 # run the side effect once
+    store.complete("order-42", result)
+
+``begin`` returns ``{status, response}`` where status is ``new`` (first time),
+``in_progress`` (a duplicate before completion), or ``completed`` (replay the
+stored response); reusing a key with a different ``request`` fingerprint raises
+``IdempotencyConflict`` (Stripe's HTTP-400 behaviour). ``complete`` records the
+response, ``get`` reads a live record, and ``save`` / ``load`` persist the store
+as JSON. ``request_fingerprint`` is a stable, order-independent SHA-256 of a
+payload.
+
+Executor commands
+-----------------
+
+``AC_idempotency_begin`` registers/looks up a ``key`` in a named store (optional
+``request`` for conflict detection); ``AC_idempotency_complete`` stores the
+``response``. Both use a named-instance registry (like circuit breakers /
+bulkheads) and are exposed as MCP tools (``ac_idempotency_begin`` /
+``ac_idempotency_complete``) and as Script Builder commands under **Flow**.

docs/source/Eng/doc/new_features/v104_features_doc.rst

@@ -0,0 +1,37 @@
+Time-Windowed Deduplication
+===========================
+
+``work_queue`` dedups only ``new`` / ``in_progress`` references — once an item
+completes, the same reference enqueues again, and redelivered webhooks are
+reprocessed. This adds the missing "seen this id in the last N seconds → drop
+it" inbox that converts at-least-once delivery to exactly-once-in-window.
+
+Pure standard library; imports no ``PySide6``. The clock is injectable, so TTL
+eviction is fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import DedupWindow
+
+    inbox = DedupWindow(ttl_s=3600)
+    if inbox.check_and_mark(event_id):
+        process(event)           # first time within the window
+    else:
+        skip(event)              # duplicate / redelivery
+
+``check_and_mark`` atomically returns ``True`` the first time an id is seen
+within the window (and marks it) or ``False`` for a duplicate. ``seen`` / ``mark``
+are the separate query/record halves, ``purge_expired`` drops stale entries, and
+``size`` reports the live count. Entries older than ``ttl_s`` are evicted on each
+operation, so the window stays bounded.
+
+Executor command
+----------------
+
+``AC_dedup_check`` check-and-marks a ``message_id`` in a named window (TTL
+``ttl_s``) and returns ``{first_seen, size}``. It uses a named-instance registry
+and is exposed as the MCP tool ``ac_dedup_check`` and as a Script Builder
+command under **Flow**.

docs/source/Eng/doc/new_features/v105_features_doc.rst

@@ -0,0 +1,39 @@
+Per-Stream Sequence-Gap Detection
+=================================
+
+Nothing tracked per-stream monotonic sequence numbers to detect missing,
+out-of-order, or duplicate messages. ``dedup_window`` says "seen this id
+before"; this complements it by classifying each sequence number and tracking
+the outstanding gaps and high-water mark per stream.
+
+Pure standard library; imports no ``PySide6``. State is in-memory and fully
+injectable, so detection is deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import SequenceTracker
+
+    tracker = SequenceTracker()
+    tracker.observe("orders", 1)      # {"status": "ok", ...}
+    tracker.observe("orders", 4)      # {"status": "gap", "missing": [2, 3]}
+    tracker.observe("orders", 3)      # {"status": "reorder", "missing": [2]}
+    tracker.gaps("orders")            # [2]
+    tracker.high_water("orders")      # 4
+
+``observe`` returns ``{status, seq, missing}`` where status is ``ok`` (next in
+order or the first seen), ``duplicate`` (already seen), ``gap`` (numbers were
+skipped — they are recorded as missing), or ``reorder`` (a late earlier number,
+which fills a gap when applicable). ``gaps`` lists the outstanding missing
+numbers and ``high_water`` is the highest seen. Streams are tracked
+independently by ``stream_id``.
+
+Executor command
+----------------
+
+``AC_sequence_observe`` observes a ``seq`` on a ``stream_id`` in a named tracker
+and returns the classification. It uses a named-instance registry and is exposed
+as the MCP tool ``ac_sequence_observe`` and as a Script Builder command under
+**Flow**.

docs/source/Eng/doc/new_features/v106_features_doc.rst

@@ -0,0 +1,42 @@
+Optimistic-Concurrency Versioned Store
+======================================
+
+``http_conditional`` uses ETag for *read* caching (``If-None-Match`` / 304) but
+never for *write* concurrency (``If-Match`` / version check). There was no local
+compare-and-swap / versioned record store for "update only if the version is
+unchanged". This fills the write side of the ETag story.
+
+Pure standard library (``json``); imports no ``PySide6``. The version is a
+monotonic int and the store is in-memory with JSON persistence, so behaviour is
+fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import VersionedStore, VersionConflict, if_match_header
+
+    store = VersionedStore()
+    version = store.put("db.host", "prod-1")          # version 1
+    record = store.get("db.host")                      # {"value": ..., "version": 1}
+    try:
+        store.put("db.host", "prod-2", expected_version=record["version"])
+    except VersionConflict:
+        reload_and_retry()
+    header = if_match_header(version)                  # '"1"' for an HTTP If-Match
+
+``put`` writes only when ``expected_version`` matches the current version
+(``0`` requires the key to be absent, omitting it is a blind write) and returns
+the new version, raising ``VersionConflict`` on a stale write. ``get`` returns
+``{value, version}``; ``delete`` is likewise guarded; ``save`` / ``load``
+persist as JSON. ``if_match_header`` / ``check_if_match`` bridge to real HTTP
+``If-Match`` writes alongside ``http_conditional``.
+
+Executor commands
+-----------------
+
+``AC_cas_put`` returns ``{ok, version}`` (or ``{ok: false, error}`` on conflict);
+``AC_cas_get`` returns ``{record}``. Both use a named-instance registry and are
+exposed as MCP tools (``ac_cas_put`` / ``ac_cas_get``) and as Script Builder
+commands under **Flow**.

docs/source/Eng/doc/new_features/v107_features_doc.rst

@@ -0,0 +1,46 @@
+Transactional Outbox
+====================
+
+``events.cloud_events`` posts immediately and synchronously — a crash between
+"did the work" and "sent the event" loses it, and a network blip drops it (no
+durability, no retry, no replay). The transactional-outbox pattern persists each
+event first and drains it later with at-least-once delivery and a dead-letter
+cap, so events survive sink outages.
+
+Pure standard library (``json``); imports no ``PySide6``. The delivery ``sink``
+is injected and the store is in-memory with JSON persistence, so draining is
+fully deterministic in CI.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import Outbox
+
+    box = Outbox()
+    box.enqueue({"type": "order.created", "id": 7})    # buffered, pending
+    box.enqueue({"type": "order.paid", "id": 7})
+
+    result = box.drain(post_to_webhook, max_batch=100, max_attempts=5)
+    # {"sent": 2, "failed": 0, "remaining": 0}
+
+    box.pending()        # entries still awaiting delivery
+    box.dead_letters()   # entries that exhausted their attempts
+
+``enqueue`` appends an event as pending and returns its id. ``drain`` delivers
+up to ``max_batch`` pending entries through the injected ``sink``; a sink
+exception leaves the entry pending for retry until ``max_attempts``, after which
+it is dead-lettered (recorded with its error). Delivery is at-least-once: a sink
+that succeeds but is interrupted before the entry is marked sent will be retried.
+``save`` / ``load`` persist the whole buffer as JSON so events outlive the
+process.
+
+Executor commands
+-----------------
+
+``AC_outbox_enqueue`` returns ``{id, pending}``; ``AC_outbox_pending`` returns
+``{pending}``. Both use a named-instance registry and are exposed as MCP tools
+(``ac_outbox_enqueue`` / ``ac_outbox_pending``) and as Script Builder commands
+under **Flow**. Draining requires a callable sink, so it stays a headless / API
+operation.