feat(sync-rules): railroad diagram generation from EBNF grammars by Sleepful · Pull Request #535 · powersync-ja/powersync-service

Sleepful · 2026-02-27T08:04:51Z

This PR goes in hand with powersync-ja/powersync-docs#372

Summary

Tool that parses W3C EBNF grammar files and generates railroad diagram documentation (SVG + MDX + HTML).

Script at packages/sync-rules/scripts/generate-grammar-docs.ts
Local review: flat HTML with inline SVGs and clickable cross-references
Docs output: flat single-page MDX per grammar with co-located SVGs — for the Mintlify docs site

What it generates

From two EBNF grammars (bucket-definitions.ebnf and sync-streams-compiler.ebnf):

41 SVG railroad diagrams (15 Sync Rules + 26 Sync Streams)
Flat MDX pages: one per grammar with all productions, cross-reference links (Used by / References), operator table, and lexical rules — ready for Mintlify
Flat HTML review pages: all diagrams on one page with clickable cross-references and inline SVGs
Resolved EBNF files: committed for diffability of inlining changes
Operator precedence tables embedded in ScalarExpr sections
Lexical rules with token summary tables and per-rule descriptions

Workflow

From packages/sync-rules/:

# Generate local review output (flat HTML + MDX + resolved EBNF)
pnpm generate:grammar

# Also generate docs output (flat MDX + co-located SVGs for the docs site)
pnpm exec tsx scripts/generate-grammar-docs.ts --outdir /path/to/powersync-docs/sync/grammar --base-url /sync/grammar

# Preview in Mintlify (from the docs repo)
cd /path/to/powersync-docs && npx mintlify dev --port 3333

--outdir — where docs MDX + SVG files are written (per-grammar subdirectories: sync-rules/, sync-streams/)
--base-url — URL prefix for absolute <img> src paths in MDX (must match the docs site structure)
Local review output always goes to packages/sync-rules/grammar/docs/ regardless of --outdir

Key design decisions

Inlining is controlled by a fixed config (GRAMMARS array), not heuristics
SVGs are static files co-located with MDX (not inline SVG — Mintlify MDX parser strips <style> and <text> elements)
Three coverage guards ensure every grammar term is accounted for
Each production section includes Used by (parent terms) and References (child terms) cross-links

Docs repo PR

The generated MDX pages are committed in the docs repo: powersync-ja/powersync-docs#372

Files changed

packages/sync-rules/scripts/generate-grammar-docs.ts — the generation script (~1400 lines)
packages/sync-rules/grammar/README.md — explains the grammar directory
packages/sync-rules/grammar/docs/*.resolved.ebnf — committed resolved grammars
packages/sync-rules/package.json — added scripts + devDependencies
packages/sync-rules/.gitignore — gitignore generated output except resolved EBNF

changeset-bot · 2026-02-27T08:05:01Z

⚠️ No Changeset found

Latest commit: f051a72

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

rkistner

I did not review the script in detail, but the output looks good to me

Script at packages/sync-rules/scripts/generate-grammar-docs.ts parses W3C EBNF grammar files and generates railroad diagram documentation: - SVG diagrams with configurable production inlining - Flat HTML review page with inline SVGs and clickable cross-references - Flat MDX page for docs site consumption - Resolved EBNF files showing post-inlining grammar (committed for diffability) - descriptions.yaml extracted from EBNF comments Run: pnpm --filter='./packages/sync-rules' generate:grammar-flat

…cing, and improve README - Add two coverage guards: assertNoSkippedTerms and assertAllRefsAreDiagrammed - Add lexical rule diagrams (Identifier, StringLiteral, IntegerLiteral, NumericLiteral) with summary table - Add HTML review features: inlining notes per production, inlined-only terms table, clickable NonTerminal links - Fix terminal modifier bug ("NOT"? now correctly renders as optional NOT) - Rebalance diagrams: decompose FromSource, Predicate, CaseExpr into smaller sections - Extract CaseCondition from SearchedCaseExpr to reduce width (1821px -> 1325px) - Inline SVGs in HTML for text selection and link interactivity - Add xmlns attribute to SVGs for browser rendering - Update grammar/README.md with inlining config, lexical rules, and coverage checks documentation

- Remove comment preprocessor and descriptions.yaml output (docs written directly in MDX) - Add stale config check (assertNoStaleInlineRefs) to catch orphaned inline references - Decompose grammars: promote TableValuedCall, Reference, WhereAtom, MemberSuffix, ScalarBinaryOp, CaseExpr variants (WhenCaseExpr, WhenScalarExpr) to own diagrams - Merge TableRef into Reference, remove Literal wrapper, extract ScalarBinaryOp - Inline Predicate into WhereAtom for sync-streams only - Promote ParameterQuery variants to top-level diagrams in bucket-definitions

Add per-production MDX pages, lexical rules page, and index page for each grammar (sync-streams, sync-rules). SVGs are generated with split-mode links baked in; flat HTML replaces them with #anchor links. - Per-production pages with diagram, TODO placeholder, inlines note, referenced terms list with cross-links - ScalarExpr pages include embedded operator precedence table - Lexical rules page with prose descriptions and EBNF code blocks - Index page per grammar listing all productions and lexical rules - Coverage guards updated: lexical and operator-table rules exempt - Lexical rules no longer get SVG diagrams (EBNF text only) - Add generate:grammar-docs pnpm script (split mode) - Resolved EBNF updated for upstream grammar changes

Lexical rules section now shows a summary table (Token | Examples | Rule) followed by per-rule description subsections. All lexical rule links in SVG diagrams point to the Lexical Rules heading so the table is the first thing seen. Fixed flat HTML to also link lexical rules (previously stripped).

…it-mode MDX Replace inline SVG approach (which fought Mintlify MDX parser stripping <style> and <text> elements) with co-located .svg files referenced via <img> tags. Add --base-url CLI param for absolute URL paths. Strip <a> links from static SVGs since they do not work inside <img> tags.

…g in split MDX Remove repeated ## heading from production pages (title in frontmatter is sufficient). Escape | in lexical rule patterns so markdown tables render correctly. Remove redundant name from description frontmatter.

…n compatibility

Remove --mode/split from grammar docs generator. The docs repo now gets flat single-page MDX per grammar with co-located SVGs via --outdir. Each production section includes Used by / References cross-links. Lexical term links point to the Lexical Rules heading. Pipe and angle bracket escaping applied to flat MDX operator table.

Backtick code spans don't decode HTML entities, so < > were rendering as literal text. Use raw < > characters instead.

…rminology Replace internal jargon with conventional SQL names across grammar files, the docs generator config, and the test helper. Examples: CompilerStreamQuery → SelectStatement, ScalarExpr → Expression, MatchExpr → WhereClause, MemberSuffix → PropertyAccess, etc.

…Used by" refs - Add SQL example snippets above each railroad diagram for quick comprehension - Collapse "Used by" cross-references into accordion/details toggle (less important than "References") - Reorder cross-refs: "References" first, "Used by" second in both MDX and HTML output - Simplify SubquerySource example to avoid confusing column alias syntax

Add assertExamplesCoverage() to catch stale/missing example keys when productions are renamed or added. Backport from docs PR review (powersync-ja/powersync-docs#372): - Change title format to "Grammar Reference (Label)" - Add introFile config field for external .intro.mdx content injected after frontmatter - Create intro files for both grammars with page context and "When to use this page" guidance

…ess examples The WithQuery name was misleading — the production is just the SELECT body inside a CTE definition, not a full WITH…SELECT statement. Also fix PropertyAccess examples to show only the suffix chain (->…->>…) rather than including the base expression or unsupported dot notation.

- PredicateTail: IN (1,2,3) → IN (SELECT ...) to match InSource alternatives - Predicate (bucket-definitions): age > 18 → status IS NOT NULL (> is not in PredicateTail) - Subquery, CteDefinition, TableParameterQuery: differentiate from other examples

…types The generate:grammar script now uses Node 22+ built-in TypeScript stripping instead of the tsx dependency.

rkistner

Added minor comments, but happy overall.

- README.md: generate:grammar-flat → generate:grammar (split mode removed) - generate-grammar-docs.ts: identifiers normalize to lower case, not uppercase - Minor indentation cleanup in error message continuations

Sleepful force-pushed the grammar-svgs branch 3 times, most recently from 8e9a4a6 to c8b0324 Compare March 4, 2026 10:24

Sleepful marked this pull request as ready for review March 5, 2026 04:46

Sleepful mentioned this pull request Mar 5, 2026

Add grammar reference with railroad syntax diagrams powersync-ja/powersync-docs#372

Merged

Sleepful force-pushed the grammar-svgs branch from 4191637 to c61c08d Compare March 5, 2026 04:53

rkistner reviewed Mar 5, 2026

View reviewed changes

Comment thread packages/sync-rules/package.json Outdated

Comment thread packages/sync-rules/grammar/docs/bucket-definitions.resolved.ebnf

Sleepful added 18 commits March 13, 2026 01:21

fix(sync-rules): escape pipe characters in operator table for markdow…

db60aa4

…n compatibility

style(sync-rules): fix prettier formatting in grammar docs generator

956125c

style(sync-rules): fix prettier formatting in grammar README

f38aa0e

fix(sync-rules): remove HTML entity escaping for < > in operator table

5c50617

Backtick code spans don't decode HTML entities, so < > were rendering as literal text. Use raw < > characters instead.

refactor(sync-rules): drop tsx in favor of node --experimental-strip-…

24c40dd

…types The generate:grammar script now uses Node 22+ built-in TypeScript stripping instead of the tsx dependency.

Sleepful force-pushed the grammar-svgs branch from c077def to 24c40dd Compare March 13, 2026 07:21

Sleepful requested a review from rkistner March 13, 2026 07:34

rkistner reviewed Mar 16, 2026

View reviewed changes

Comment thread packages/sync-rules/grammar/README.md Outdated

Comment thread packages/sync-rules/scripts/generate-grammar-docs.ts Outdated

fix(sync-rules): correct README script name and Identifier lexical note

f051a72

- README.md: generate:grammar-flat → generate:grammar (split mode removed) - generate-grammar-docs.ts: identifiers normalize to lower case, not uppercase - Minor indentation cleanup in error message continuations

Sleepful force-pushed the grammar-svgs branch from 310272d to f051a72 Compare March 17, 2026 07:31

Sleepful requested a review from rkistner March 17, 2026 07:57

rkistner approved these changes Mar 17, 2026

View reviewed changes

Sleepful merged commit 5303940 into powersync-ja:main Mar 17, 2026
61 of 62 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sync-rules): railroad diagram generation from EBNF grammars#535

feat(sync-rules): railroad diagram generation from EBNF grammars#535
Sleepful merged 19 commits intopowersync-ja:mainfrom
Sleepful:grammar-svgs

Sleepful commented Feb 27, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Feb 27, 2026 •

edited

Loading

Uh oh!

rkistner left a comment

Uh oh!

Uh oh!

Uh oh!

rkistner left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Sleepful commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What it generates

Workflow

Key design decisions

Docs repo PR

Files changed

Uh oh!

changeset-bot Bot commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sleepful commented Feb 27, 2026 •

edited

Loading

changeset-bot Bot commented Feb 27, 2026 •

edited

Loading