Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-22) — JSON-Schema Compatibility Checking](#whats-new-2026-06-22--json-schema-compatibility-checking)
- [What's new (2026-06-22) — Typed Configuration Schema](#whats-new-2026-06-22--typed-configuration-schema)
- [What's new (2026-06-22) — OTLP/JSON Span Export](#whats-new-2026-06-22--otlpjson-span-export)
- [What's new (2026-06-22) — Canonical Log Lines & Structured Logging](#whats-new-2026-06-22--canonical-log-lines--structured-logging)
Expand Down Expand Up @@ -148,6 +149,12 @@

---

## What's new (2026-06-22) — JSON-Schema Compatibility Checking

Classify schema changes as backward/forward/full. Full reference: [`docs/source/Eng/doc/new_features/v96_features_doc.rst`](docs/source/Eng/doc/new_features/v96_features_doc.rst).

- **`check_compatibility` / `diff_schemas` / `is_backward_compatible` / `is_forward_compatible` / `is_full_compatible`** (`AC_check_compatibility`): we could validate against and generate JSON Schemas but couldn't answer "will an old consumer still read new data?". This classifies changes (added-required field, removed field, narrowed/widened type, enum add/remove) under Confluent/Avro backward/forward/full rules over the object subset. Pure-stdlib, deterministic.

## What's new (2026-06-22) — Typed Configuration Schema

Validate config into a typed object. Full reference: [`docs/source/Eng/doc/new_features/v95_features_doc.rst`](docs/source/Eng/doc/new_features/v95_features_doc.rst).
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-22) — JSON-Schema 兼容性检查](#本次更新-2026-06-22--json-schema-兼容性检查)
- [本次更新 (2026-06-22) — 具类型的配置结构](#本次更新-2026-06-22--具类型的配置结构)
- [本次更新 (2026-06-22) — OTLP/JSON Span 导出](#本次更新-2026-06-22--otlpjson-span-导出)
- [本次更新 (2026-06-22) — 标准日志行与结构化日志](#本次更新-2026-06-22--标准日志行与结构化日志)
Expand Down Expand Up @@ -147,6 +148,12 @@

---

## 本次更新 (2026-06-22) — JSON-Schema 兼容性检查

把结构变更分类为 backward/forward/full。完整参考:[`docs/source/Zh/doc/new_features/v96_features_doc.rst`](../docs/source/Zh/doc/new_features/v96_features_doc.rst)。

- **`check_compatibility` / `diff_schemas` / `is_backward_compatible` / `is_forward_compatible` / `is_full_compatible`**(`AC_check_compatibility`):我们能依结构验证并生成结构,但无法回答「旧消费者是否仍能读新数据?」。本功能依 Confluent/Avro backward/forward/full 规则,在对象子集上分类变更(新增必填字段、移除字段、收窄/放宽类型、enum 增减)。纯标准库、确定。

## 本次更新 (2026-06-22) — 具类型的配置结构

把配置验证成具类型的对象。完整参考:[`docs/source/Zh/doc/new_features/v95_features_doc.rst`](../docs/source/Zh/doc/new_features/v95_features_doc.rst)。
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-22) — JSON-Schema 相容性檢查](#本次更新-2026-06-22--json-schema-相容性檢查)
- [本次更新 (2026-06-22) — 具型別的設定結構](#本次更新-2026-06-22--具型別的設定結構)
- [本次更新 (2026-06-22) — OTLP/JSON Span 匯出](#本次更新-2026-06-22--otlpjson-span-匯出)
- [本次更新 (2026-06-22) — 標準日誌行與結構化日誌](#本次更新-2026-06-22--標準日誌行與結構化日誌)
Expand Down Expand Up @@ -147,6 +148,12 @@

---

## 本次更新 (2026-06-22) — JSON-Schema 相容性檢查

把結構變更分類為 backward/forward/full。完整參考:[`docs/source/Zh/doc/new_features/v96_features_doc.rst`](../docs/source/Zh/doc/new_features/v96_features_doc.rst)。

- **`check_compatibility` / `diff_schemas` / `is_backward_compatible` / `is_forward_compatible` / `is_full_compatible`**(`AC_check_compatibility`):我們能依結構驗證並產生結構,但無法回答「舊消費者是否仍能讀新資料?」。本功能依 Confluent/Avro backward/forward/full 規則,在物件子集上分類變更(新增必填欄位、移除欄位、收窄/放寬型別、enum 增減)。純標準函式庫、具決定性。

## 本次更新 (2026-06-22) — 具型別的設定結構

把設定驗證成具型別的物件。完整參考:[`docs/source/Zh/doc/new_features/v95_features_doc.rst`](../docs/source/Zh/doc/new_features/v95_features_doc.rst)。
Expand Down
46 changes: 46 additions & 0 deletions docs/source/Eng/doc/new_features/v96_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
JSON-Schema Compatibility Checking
==================================

We can *validate against* a JSON Schema (``json_schema``) and *generate* one
(``action_lint/schema``) but could not answer "will a consumer on the old
schema still read data written under the new schema?" — i.e. classify changes
(added-required field, removed field, narrowed type, removed enum value) under
the Confluent/Avro backward / forward / full rules. This adds that classifier.

Scope: the object-schema subset ``json_schema`` understands —
``properties`` / ``required`` / ``type`` / ``enum``. Pure standard library;
imports no ``PySide6``. Each function is pure (two schema dicts in, report out),
so it is fully deterministic in CI.

Headless API
------------

.. code-block:: python

from je_auto_control import (
check_compatibility, is_backward_compatible, diff_schemas,
)

report = check_compatibility(old_schema, new_schema, mode="backward")
# {"compatible": False, "mode": "backward",
# "changes": [...], "breaking": [{"path": "email", "kind": "field_added",
# "breaks": ["backward"]}]}

if not is_backward_compatible(old_schema, new_schema):
block_release()

``diff_schemas`` classifies every change as a ``SchemaChange`` (``path``,
``kind``, ``breaks``). Backward-breaking changes include a new required field, a
narrowed type, and a removed enum value; forward-breaking changes include a
removed required field, a widened type, and an added enum value.
``check_compatibility`` filters those by ``mode`` (``backward`` / ``forward`` /
``full``); ``is_backward_compatible`` / ``is_forward_compatible`` /
``is_full_compatible`` are boolean shortcuts.

Executor command
----------------

``AC_check_compatibility`` takes ``old`` / ``new`` schemas and an optional
``mode`` and returns ``{compatible, mode, changes, breaking}``. It is exposed as
the MCP tool ``ac_check_compatibility`` and as a Script Builder command under
**Data**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v93_features_doc
doc/new_features/v94_features_doc
doc/new_features/v95_features_doc
doc/new_features/v96_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
38 changes: 38 additions & 0 deletions docs/source/Zh/doc/new_features/v96_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
JSON-Schema 相容性檢查
=====================

我們能*依*某個 JSON Schema 驗證(``json_schema``)並*產生*結構(``action_lint/schema``),但無法回答
「使用舊結構的消費者,是否仍能讀取以新結構寫入的資料?」—— 也就是依 Confluent/Avro 的 backward /
forward / full 規則分類變更(新增必填欄位、移除欄位、收窄型別、移除 enum 值)。本功能補上此分類器。

範圍:``json_schema`` 理解的物件結構子集 —— ``properties`` / ``required`` / ``type`` / ``enum``。純標準
函式庫;不匯入 ``PySide6``。每個函式皆為純函式(兩個結構 dict 輸入、報告輸出),因此在 CI 中完全具決定性。

無頭 API
--------

.. code-block:: python

from je_auto_control import (
check_compatibility, is_backward_compatible, diff_schemas,
)

report = check_compatibility(old_schema, new_schema, mode="backward")
# {"compatible": False, "mode": "backward",
# "changes": [...], "breaking": [{"path": "email", "kind": "field_added",
# "breaks": ["backward"]}]}

if not is_backward_compatible(old_schema, new_schema):
block_release()

``diff_schemas`` 把每個變更分類為 ``SchemaChange``(``path``、``kind``、``breaks``)。會破壞 backward 的
變更包含新增必填欄位、收窄型別、移除 enum 值;會破壞 forward 的變更包含移除必填欄位、放寬型別、新增
enum 值。``check_compatibility`` 依 ``mode``(``backward`` / ``forward`` / ``full``)篩選;
``is_backward_compatible`` / ``is_forward_compatible`` / ``is_full_compatible`` 為布林捷徑。

執行器命令
----------

``AC_check_compatibility`` 接受 ``old`` / ``new`` 結構與選用的 ``mode``,回傳
``{compatible, mode, changes, breaking}``。它以 MCP 工具 ``ac_check_compatibility`` 以及 Script Builder
中 **Data** 分類下的命令提供。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v93_features_doc
doc/new_features/v94_features_doc
doc/new_features/v95_features_doc
doc/new_features/v96_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
7 changes: 7 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,11 @@
from je_auto_control.utils.data_drift import (
categorical_drift, detect_drift, ks_two_sample, psi,
)
# JSON-Schema compatibility (backward / forward / full classification)
from je_auto_control.utils.schema_compat import (
SchemaChange, check_compatibility, diff_schemas, is_backward_compatible,
is_forward_compatible, is_full_compatible,
)
# Tabular row-set diff (CDC-style added/removed/changed by key)
from je_auto_control.utils.dataset_diff import (
cell_changes, diff_rows, summarize_diff,
Expand Down Expand Up @@ -880,6 +885,8 @@ def start_autocontrol_gui(*args, **kwargs):
"extract_fields", "mask_rows", "validate_rows",
"infer_schema", "profile_rows",
"categorical_drift", "detect_drift", "ks_two_sample", "psi",
"SchemaChange", "check_compatibility", "diff_schemas",
"is_backward_compatible", "is_forward_compatible", "is_full_compatible",
"cell_changes", "diff_rows", "summarize_diff",
"check_accepted_values", "check_foreign_key", "check_row_count",
"check_unique_key",
Expand Down
13 changes: 13 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -1888,6 +1888,19 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None:
),
description="Categorical drift: chi-square + total-variation distance.",
))
specs.append(CommandSpec(
"AC_check_compatibility", "Data", "Schema Compat: Check",
fields=(
FieldSpec("old", FieldType.STRING,
placeholder='{"properties": {"id": {"type": "integer"}}}'),
FieldSpec("new", FieldType.STRING,
placeholder='{"properties": {"id": {"type": "integer"}}, '
'"required": ["id"]}'),
FieldSpec("mode", FieldType.STRING, optional=True,
placeholder="backward | forward | full"),
),
description="Classify JSON-Schema changes as backward/forward/full.",
))
specs.append(CommandSpec(
"AC_diff_rows", "Data", "Dataset Diff: Rows by Key",
fields=(
Expand Down
13 changes: 13 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3243,6 +3243,18 @@ def _explain_config(layers: Any, key: str) -> Dict[str, Any]:
"layer": trace.layer}}


def _check_compatibility(old: Any, new: Any,
mode: str = "backward") -> Dict[str, Any]:
"""Adapter: classify JSON-Schema compatibility (backward/forward/full)."""
import json
from je_auto_control.utils.schema_compat import check_compatibility
if isinstance(old, str):
old = json.loads(old)
if isinstance(new, str):
new = json.loads(new)
return check_compatibility(old, new, mode)


def _detect_drift(reference: Any, current: Any,
threshold: Any = 0.25, bins: Any = 10) -> Dict[str, Any]:
"""Adapter: numeric distribution drift report (PSI + KS)."""
Expand Down Expand Up @@ -4435,6 +4447,7 @@ def __init__(self):
"AC_parse_sse": _parse_sse,
"AC_resolve_config": _resolve_config,
"AC_explain_config": _explain_config,
"AC_check_compatibility": _check_compatibility,
"AC_detect_drift": _detect_drift,
"AC_categorical_drift": _categorical_drift,
"AC_diff_rows": _diff_rows,
Expand Down
19 changes: 18 additions & 1 deletion je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -3548,6 +3548,23 @@ def dataset_diff_tools() -> List[MCPTool]:
]


def schema_compat_tools() -> List[MCPTool]:
return [
MCPTool(
name="ac_check_compatibility",
description=("Classify JSON-Schema changes from 'old' to 'new' under "
"'mode' (backward / forward / full). Returns "
"{compatible, mode, changes, breaking}."),
input_schema=schema(
{"old": {"type": "object"}, "new": {"type": "object"},
"mode": {"type": "string"}},
["old", "new"]),
handler=h.check_compatibility,
annotations=READ_ONLY,
),
]


def data_drift_tools() -> List[MCPTool]:
seq_schema = {"type": "array"}
return [
Expand Down Expand Up @@ -5363,7 +5380,7 @@ def media_assert_tools() -> List[MCPTool]:
trace_context_tools, baggage_tools, canonical_log_tools, otlp_export_tools,
secret_ref_tools, config_schema_tools, config_redaction_tools,
data_profile_tools, http_problem_tools, dotenv_tools,
sse_client_tools, layered_config_tools, data_drift_tools,
sse_client_tools, layered_config_tools, data_drift_tools, schema_compat_tools,
dataset_diff_tools, referential_tools, link_header_tools, multipart_tools,
http_content_tools, cookie_jar_tools, http_conditional_tools,
saga_tools, decision_table_tools, locator_repair_tools,
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1864,6 +1864,12 @@ def explain_config(layers, key):
return _explain_config(layers, key)


def check_compatibility(old, new, mode="backward"):
from je_auto_control.utils.executor.action_executor import (
_check_compatibility)
return _check_compatibility(old, new, mode)


def detect_drift(reference, current, threshold=0.25, bins=10):
from je_auto_control.utils.executor.action_executor import _detect_drift
return _detect_drift(reference, current, threshold, bins)
Expand Down
10 changes: 10 additions & 0 deletions je_auto_control/utils/schema_compat/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
"""JSON-Schema compatibility classification for AutoControl."""
from je_auto_control.utils.schema_compat.schema_compat import (
SchemaChange, check_compatibility, diff_schemas, is_backward_compatible,
is_forward_compatible, is_full_compatible,
)

__all__ = [
"SchemaChange", "check_compatibility", "diff_schemas",
"is_backward_compatible", "is_forward_compatible", "is_full_compatible",
]
Loading
Loading