Skip to content

Add pure-PHP SQLite engine compatibility for WordPress and WooCommerce#431

Draft
adamziel wants to merge 12 commits into
WordPress:trunkfrom
adamziel:pure-php-engine-pr
Draft

Add pure-PHP SQLite engine compatibility for WordPress and WooCommerce#431
adamziel wants to merge 12 commits into
WordPress:trunkfrom
adamziel:pure-php-engine-pr

Conversation

@adamziel

@adamziel adamziel commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

What it does

Adds a pure-PHP SQLite-compatible engine path so the SQLite integration can run without pdo_sqlite.

This PR also expands compatibility for WordPress and WooCommerce-style SQL:

  • Wires WP_PHP_Engine_PDO into the MySQL-on-SQLite driver and install/bootstrap paths.
  • Adds file-backed persistence, transaction snapshots, write locking, PDO fetch compatibility, and virtual SQLite metadata tables.
  • Covers more MySQL-on-SQLite patterns: multi-query execution, joined updates that read from information_schema, views, triggers, routines/events metadata, logical database metadata, CTAS, defaults, ALTER TABLE cases, and legacy SQLite fallbacks.
  • Adds pure-PHP engine test variants for the driver, PDO API, metadata, query, information schema, and file persistence suites.

Rationale

WordPress Playground and similar environments may not have native SQLite/PDO support available. A pure-PHP fallback makes the integration usable in those environments while preserving the existing PDO-based path where native SQLite is available.

Concurrency is intentionally conservative. The pure-PHP engine is not trying to emulate SQLite’s full pager, WAL mode, MVCC, or cross-process read/write scheduling. It serializes writes with a database-level lock and uses transaction snapshots to avoid lost updates. That is enough for the fallback runtime this PR targets, but it is not a high-concurrency database engine and should not be presented as one.

Implementation

The pure-PHP path stores database state as PHP arrays and persists it for file-backed databases. WP_PHP_Engine_PDO exposes enough PDO behavior for the existing driver stack, including legacy PHP/PDO fetch quirks covered by the package matrix.

Concurrency support is limited to correctness-oriented safeguards. For a file-backed database, overlapping writes behave like this:

  1. The first writer takes an exclusive flock() on the database file.
  2. Before applying the write, it reloads the latest committed file state so it does not overwrite changes committed by a previous connection.
  3. The write runs against the in-memory PHP array state.
  4. On success, the full database state is serialized back to the file with a new generation token, then the lock is released.
  5. A second writer that starts while the first writer holds the lock waits up to the configured busy timeout (PDO::ATTR_TIMEOUT, or PRAGMA busy_timeout in milliseconds). If the lock is released in time, it reloads the just-written generation before applying its own changes. If not, it fails with SQLite-compatible code 5, database is locked.

Explicit transactions hold that exclusive lock from BEGIN until COMMIT or ROLLBACK. Reads from other file-backed connections take a shared lock and use the same busy-timeout behavior: they wait while a write transaction is open, then either reload the committed generation before reading or fail with database is locked if the timeout expires. ROLLBACK restores the transaction snapshot and releases the lock without exposing intermediate writes.

For :memory: databases, this is just a per-connection in-memory engine. There is no cross-connection shared database and therefore no cross-connection write coordination.

Known concurrency shortcomings:

  • no row/page-level locks;
  • no WAL-style concurrent writer plus readers model;
  • no full MVCC snapshot isolation across independent processes;
  • no deadlock detection, fairness guarantee, or lock wait tuning beyond the busy timeout;
  • write-heavy workloads will bottleneck on the single database lock.

Testing instructions

./vendor/bin/phpunit --configuration packages/mysql-on-sqlite/phpunit.xml.dist --colors=never
./vendor/bin/phpcs --standard=phpcs.xml.dist \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine.php \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine-parser.php \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine-evaluator.php \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine-functions.php \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine-pdo.php \
  packages/mysql-on-sqlite/src/php-engine/class-wp-php-engine-pdo-statement.php \
  packages/mysql-on-sqlite/src/sqlite/class-wp-pdo-mysql-on-sqlite.php
git diff --check

CI is expected to pass across the full package PHP/SQLite matrix, WordPress PHPUnit, WordPress e2e, code style, and proxy tests.

adamziel and others added 7 commits May 27, 2026 11:39
## What?

Adds a browser-runtime compatibility check for the `wp_mysql_parser`
WASM side modules and wires it into both WASM workflows:

- `wasm-spike.yml`
- `publish-wasm-extension-artifact.yml`

The check compares each side module's `env` function imports against the
matching Playground web PHP.wasm runtime exports. It runs for PHP 8.0
through 8.5 so the published manifest cannot claim browser support for a
PHP version that will crash during extension startup.

Also updates the native extension README/demo URL to use the live
`blueprint.json` location and documents PHP 8.0-8.5 browser-runtime
coverage instead of the temporary PHP 8.3 pin.

## Why?

The previous Node-based smoke test was insufficient: PHP 8.4 loaded in
the Node/CLI runtime but crashed in the browser runtime because the web
runtime did not export Zend symbols imported by the extension side
module.

This PR makes that class of failure visible in CI before publishing a
WASM manifest.

## Dependency

This depends on WordPress/wordpress-playground#3690 and updated
Playground web PHP.wasm artifacts. Current Playground web builds are
known to miss required exports for PHP 8.0, 8.1, 8.4, and 8.5; the new
CI check is expected to pass once those runtime exports ship.

## Testing

- `node --check
packages/php-ext-wp-mysql-parser/wasm-spike/check-playground-web-compat.mjs`
- `git diff --check`
- Verified locally with the Playground checkout:
  - PHP 8.3 currently passes.
- PHP 8.4 currently fails with missing `zend_declare_class_constant_ex`
and `zend_register_internal_class_ex`, matching the browser crash root
cause.

---------

Co-authored-by: Jan Jakeš <jan@jakes.pro>
Co-authored-by: Chloe Pomegranate <chloehoughtonjenkins+git@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Francesco Bigiarini <francesco.bigiarini@gmail.com>
Co-authored-by: Wojtek Naruniec <wojtek@naruniec.me>
Co-authored-by: wpfuse <113634078+wp-fuse@users.noreply.github.com>
Co-authored-by: Ashish Kumar <ashfame@users.noreply.github.com>
Co-authored-by: Jon Surrell <sirreal@users.noreply.github.com>
Co-authored-by: John Blackbourn <john@johnblackbourn.com>
Co-authored-by: Jonathan Desrosiers <359867+desrosj@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Introduce WP_PHP_Engine — an SQLite-compatible database engine written
entirely in PHP, with no dependency on the pdo_sqlite or sqlite3
extensions. Tables live in PHP arrays, with optional persistence to a
plain file.

The engine plugs in underneath the existing MySQL-on-SQLite driver via
a PDO-compatible facade. Because the driver already translates MySQL to
SQLite SQL and registers its MySQL function library as PHP callbacks,
the entire driver stack — translation, information schema, transactions,
session semantics — runs unchanged on top of the new engine:

    $pdo        = new WP_PHP_Engine_PDO( 'php-engine::memory:' );
    $connection = new WP_SQLite_Connection( array( 'pdo' => $pdo ) );
    $driver     = new WP_SQLite_Driver( $connection, 'wp' );

The engine implements the SQLite dialect surface the driver emits, and
then some: SELECT with joins/subqueries/CTEs/window functions/compound
queries, INSERT (incl. upserts), UPDATE (incl. UPDATE ... FROM and
multi-column assignments), DELETE, full DDL incl. STRICT tables and
AUTOINCREMENT, AFTER triggers, foreign keys with CASCADE/SET NULL/SET
DEFAULT, transactions and savepoints (O(1) snapshots via PHP COW),
sqlite_master, the PRAGMA introspection interface, type affinities,
collations (BINARY/NOCASE/RTRIM), and the core/date-time/aggregate
function library.

Conformance was driven by differential testing: a corpus of 37,548
SQLite queries captured from the driver test suites replays identically
(0 diffs) against the real SQLite and the PHP engine, including result
values, column names, error messages, and PDO column metadata. See
tests/tools/php-engine-differential.php.

The existing driver test suites now run twice — once against SQLite and
once against the PHP engine — via a small create_pdo() factory hook in
the test classes. All 1,171 tests (1.44M assertions) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Persist DDL statements (CREATE TABLE/INDEX/TRIGGER/VIEW, DROP, ALTER)
  for file-backed databases. Persistence is now centralized in the
  execute() entry point instead of sprinkled across handlers, so every
  write statement is durable once no transaction is open.

- Prevent lost updates between concurrent connections. The database file
  now carries a generation header, and every connection accesses it
  through one persistent handle with flock()-based locking: write
  statements (and whole transactions, from BEGIN to COMMIT/ROLLBACK)
  hold an exclusive lock around their read-modify-write cycle and reload
  the state if another process saved in the meantime; reads take a
  shared lock and reload when stale. The engine also refuses to open
  files in unrecognized formats, so it can never overwrite a real SQLite
  database file.

- Wire the engine into production as a fallback: when the pdo_sqlite
  driver is missing (or WP_SQLITE_FORCE_PHP_ENGINE is set),
  WP_SQLite_Connection now transparently uses WP_PHP_Engine_PDO, and the
  db.php drop-in bootstraps the new SQLite driver on top of it instead
  of dying. Plugin activation and admin notices now only require PDO
  itself; composer moves ext-pdo_sqlite to "suggest".

- bindParam() now binds by reference and reads the variable at
  execute() time, like PDO.

- INSERT ... ON CONFLICT DO NOTHING now reports 0 affected rows, and
  DO UPDATE ... WHERE that filters out the update also reports 0.

New WP_PHP_Engine_Tests cover file-backed DDL/data/AUTOINCREMENT
persistence across reopens, rollback durability, concurrent-connection
lost-update prevention, cross-connection visibility, foreign-file
refusal, upsert row counts, and bindParam reference semantics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@adamziel

Copy link
Copy Markdown
Collaborator Author

Fair local benchmark (same WordPress SQLite driver stack, only backend swapped):

  • Machine: Apple M4 Max, arm64
  • PHP: 8.4.5 CLI, opcache disabled, JIT disabled
  • Storage: in-memory for both backends (sqlite::memory: vs php-engine::memory:)
  • Method: one warmup run excluded; 5 measured runs; median reported
  • Workload per run through WP_SQLite_Driver: create 5 schema objects, 250 wp_options inserts, 250 point selects, one grouped select, 250 wp_posts inserts, 250 wp_postmeta inserts, one join with ORDER BY ... LIMIT, one delete, one count.
Backend Median Runs
SQLite PDO 0.1386s 0.1374, 0.1386, 0.1382, 0.1393, 0.1395
Pure-PHP engine 0.4960s 0.4898, 0.6819, 0.4921, 0.5002, 0.4960

Result: on this small in-memory WordPress-ish workload, the pure-PHP engine is ~3.6x slower than SQLite PDO. This is intended as an apples-to-apples backend comparison through the same driver/translation layer, not as a production throughput claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant