Standardize JSONL log envelopes across AWF runtime logs#3790
Conversation
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (2 files)
Coverage comparison generated by |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
This PR standardizes AWF’s JSONL runtime logs onto a consistent top-level envelope (timestamp in ISO-8601 ms, event, and _schema) to make downstream parsing and cross-log correlation less brittle.
Changes:
- Updated Squid
audit.jsonlemission to usetimestamp(ISO-8601 ms) and an explicitevent. - Updated log parsing/tests to accept both new (
timestamp) and legacy (ts) audit records. - Extended/added JSON Schemas + docs for the standardized envelope across token usage, OTel spans, and CLI proxy access logs.
Show a summary per file
| File | Description |
|---|---|
| src/squid/config-generator.ts | Updates Squid audit_jsonl logformat to emit timestamp (ISO ms) and event. |
| src/squid-config-features.test.ts | Adjusts config generation test expectations for new audit JSONL fields. |
| src/logs/log-parser.ts | Updates audit JSONL parser to support ISO timestamp while keeping legacy ts support. |
| src/logs/log-parser.test.ts | Updates/extends tests for current and legacy audit JSONL timestamp formats. |
| containers/cli-proxy/server.js | Renames CLI proxy access log field ts → timestamp. |
| containers/api-proxy/token-tracker.schema.test.js | Updates token-usage schema tests to include/require event. |
| containers/api-proxy/token-persistence.js | Adds event to token-usage persistence record + runtime validation requirements. |
| containers/api-proxy/otel.js | Adds _schema + event to file-exported OTel span JSONL records. |
| containers/api-proxy/otel.test.js | Adds test coverage for OTel JSONL envelope fields (timestamp/event/_schema). |
| schemas/audit.schema.json | Updates audit schema to require timestamp and event (dropping ts). |
| schemas/token-usage.schema.json | Requires event: "token_usage" in token-usage records. |
| schemas/otel-span.schema.json | Adds a new schema for otel.jsonl span records. |
| schemas/cli-proxy-access.schema.json | Adds a new schema for CLI proxy access.jsonl records. |
| schemas/README.md | Updates schema index docs and examples to reflect standardized envelope + new schema artifacts. |
| docs/awf-config-spec.md | Updates spec to make standardized JSONL envelope normative and lists new schema artifacts. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 15/15 changed files
- Comments generated: 6
| let timestamp = 0; | ||
| if (typeof obj.timestamp === 'string') { | ||
| const parsed = Date.parse(obj.timestamp); | ||
| if (!Number.isNaN(parsed)) { | ||
| timestamp = parsed / 1000; | ||
| } | ||
| } else if (typeof obj.ts === 'number') { | ||
| timestamp = obj.ts; | ||
| } |
| const required = [ | ||
| ['_schema', 'string'], | ||
| ['timestamp', 'string'], | ||
| ['event', 'string'], | ||
| ['request_id', 'string'], | ||
| ['provider', 'string'], |
| }, | ||
| "event": { | ||
| "type": "string", | ||
| "enum": ["http_access", "http_access_allowed", "http_access_denied"], |
| All AWF JSONL records **MUST** include the following top-level fields: | ||
|
|
||
| - `timestamp` (string, required): ISO 8601 UTC with milliseconds (`YYYY-MM-DDTHH:mm:ss.SSSZ`). | ||
| - `event` (string, required): Stable snake_case record discriminator. | ||
| - `_schema` (string, required): Schema identifier in the form `<record-type>/v<version>`. | ||
|
|
| "type": "object", | ||
| "required": ["timestamp", "event"], | ||
| "additionalProperties": true, | ||
| "properties": { | ||
| "timestamp": { | ||
| "type": "string", | ||
| "format": "date-time", | ||
| "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z$", | ||
| "description": "ISO 8601 UTC timestamp with millisecond precision." | ||
| }, | ||
| "event": { | ||
| "type": "string", | ||
| "pattern": "^[a-z0-9]+(?:_[a-z0-9]+)*$", | ||
| "description": "Stable snake_case event discriminator (e.g. exec_start, exec_done)." | ||
| } |
| * Each line is a self-contained JSON object for easy parsing. | ||
| */ | ||
| function accessLog(entry) { | ||
| const record = { ts: new Date().toISOString(), ...entry }; | ||
| const record = { timestamp: new Date().toISOString(), ...entry }; | ||
| const line = JSON.stringify(record); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@copilot address review feedback |
All six review issues addressed in commit
|
🔧 BYOK Smoke Test ResultsStatus: PARTIAL PASS
Note: Running in BYOK offline mode ( Current PR: "Standardize JSONL log envelopes across AWF runtime logs"
|
|
Smoke Test Results ✅ GitHub MCP: PR #3786 "Update Status: FAIL cc
|
🔬 Smoke Test: API Proxy OpenTelemetry TracingStatus: ✅ All scenarios validated Test Results
Architecture Validation✅ Span lifecycle: Conclusion: OTEL tracing integration is fully operational. All hook points validated, test coverage complete.
|
|
Smoke Codex: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
Service Connectivity Test Results❌ Redis: TIMEOUT/ERROR Overall: FAIL — services not reachable via host.docker.internal
|
Gemini Engine Smoke Test Results
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Chroot Runtime Version ComparisonTest results from comparing runtime versions between host and chrooted environment:
Overall Result: Not all tests passed (1/3 matches) Analysis
The version mismatches indicate the chrooted environment is seeing different runtime installations than the host, which is expected behavior when using selective bind mounts rather than a full filesystem overlay.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS All build and test operations completed successfully across all ecosystems.
|
AWF JSONL outputs used inconsistent timestamp keys/formats (
tsepoch vstimestampISO) and uneven event discrimination, which made cross-log parsing and correlation unnecessarily brittle. This change aligns runtime log records on a common envelope:timestamp(ISO-8601 ms),event, and_schema.Emitter format alignment
audit.jsonl(Squid): switched totimestampISO-8601 with milliseconds, added top-levelevent.token-usage.jsonl(api-proxy): addedevent: "token_usage"to persisted records.otel.jsonl(api-proxy file exporter): added_schema+event: "otel_span"and standardized timestamp envelope.access.jsonl(cli-proxy): renamedts→timestamp.Parser migration support
timestamp(ISO string), andts(epoch float),preserving compatibility during rollout.
Schema contract updates
schemas/audit.schema.jsonto requiretimestamp(ISO ms) andevent.schemas/token-usage.schema.jsonto requireevent(token_usage).schemas/otel-span.schema.jsonschemas/cli-proxy-access.schema.jsonSpec/docs normalization
timestamp/event/_schemaa normative JSONL requirement.otel-spanandcli-proxy-accessschema artifacts.Example standardized envelope shape now emitted across JSONL records:
{ "timestamp": "2026-05-25T09:00:00.000Z", "event": "token_usage", "_schema": "token-usage/v0.23.1" }