Add native Rust-based MySQL parser extension

[adamziel]

What it does

Adds an optional Rust PHP extension for the MySQL lexer/parser. When the extension is loaded, the existing public PHP API stays the same, but WP_MySQL_Lexer and WP_MySQL_Parser delegate lexing/parsing to native Rust code.

php -d extension=/path/to/libwp_mysql_parser.so your-script.php

require 'packages/mysql-on-sqlite/src/load.php';

$driver = new WP_PDO_MySQL_On_SQLite( 'mysql-on-sqlite:path=:memory:;dbname=wp;' );
$parser = $driver->create_parser( 'SELECT ID, post_title FROM wp_posts WHERE ID IN (1, 2, 3)' );

$parser->next_query();
$ast = $parser->get_query_ast();

echo $ast->rule_name; // query

Without the extension, the same code uses the existing PHP parser.

Rationale

The PHP parser is correct but expensive on large query sets. On the MySQL corpus, the native path measured against trunk's PHP implementation was:

Scenario	Trunk PHP	Native extension
Parse only	14.3114s / 4,862 QPS / 68.0MB	1.1574s / 60,105 QPS / 30.0MB
Parse + walk	20.0804s / 3,465 QPS / 70.0MB	2.9751s / 23,383 QPS / 48.0MB

That is about 12.37x faster for parse-only and 6.75x faster for parse+walk. Raw numbers are in #381 (comment).

Implementation

The extension lives in packages/php-ext-wp-mysql-parser/ and exports WP_MySQL_Native_Lexer, WP_MySQL_Native_Token_Stream, WP_MySQL_Native_Parser, and WP_MySQL_Native_Parser_Node.

The SQLite driver selects the native path only when the native lexer class is active:

$lexer = new WP_MySQL_Lexer( $sql );

if ( $lexer instanceof WP_MySQL_Native_Lexer ) {
    $tokens = $lexer->native_token_stream();
} else {
    $tokens = $lexer->remaining_tokens();
}

Native AST nodes are lazy PHP wrappers over a Rust-owned AST. Wrapper identity is stable through a per-AST cache, and Rust state is stored in a Rust-side registry keyed by the PHP wrapper object pointer. That avoids the previous PHP/Rust reference cycle while keeping repeated child reads referentially stable.

The pure-PHP parser remains the fallback and WP_MySQL_Parser remains an instanceof WP_Parser.

Testing instructions

Run the PHP suite normally:

cd packages/mysql-on-sqlite
php ./vendor/bin/phpunit -c ./phpunit.xml.dist

Run it against the extension:

cd packages/php-ext-wp-mysql-parser
cargo build

cd ../mysql-on-sqlite
WP_SQLITE_REQUIRE_NATIVE_PARSER_EXTENSION=1 \
php -d extension=../php-ext-wp-mysql-parser/target/debug/libwp_mysql_parser.so \
  ./vendor/bin/phpunit -c ./phpunit.xml.dist

CI currently passes on bde34d5 for PHP 7.2-8.5, including extension-loaded SQLite integration tests on PHP 8.0-8.5 and WordPress PHPUnit with the extension loaded.

[JanJakes]

The native AST accessors appear to create fresh PHP wrappers for child nodes/tokens on each read. That changes WP_Parser_Node semantics: child object identity is no longer stable, and mutations to a child returned by get_first_child_node() / get_children() are not visible when traversing from the parent again. Since WP_Parser_Node exposes public mutators and this PR aims to keep the public parser API unchanged, can we either cache/materialize child wrappers consistently or explicitly account for this compatibility change?

[adamziel]

#391 and #392 address that. I hope :D I'll get the Ci to run against all the same tests as the PHP driver and scrutinize once that works

[adamziel]

Perf comparison against trunk's PHP implementation:

Trunk PHP was measured from a detached origin/trunk worktree at fa5a7ba with the native-AST benchmark helper staged only in /tmp for measurement. Runtime: local PHP 8.2.29, same mysql-server-tests-queries.csv corpus, three sequential runs.

Scenario	Trunk PHP avg	This PR native	Speedup
Parse only	14.3114s / 4,862 QPS / 68.0MB	1.1574s / 60,105 QPS / 30.0MB	12.37x by duration
Parse + walk	20.0804s / 3,465 QPS / 70.0MB	2.9751s / 23,383 QPS / 48.0MB	6.75x by duration

Raw trunk PHP runs:

parse-only: 13.9360s / 4,991 QPS / 68.0MB
parse-only: 14.6215s / 4,757 QPS / 68.0MB
parse-only: 14.3767s / 4,838 QPS / 68.0MB
walk:       19.5978s / 3,549 QPS / 70.0MB
walk:       20.6002s / 3,377 QPS / 70.0MB
walk:       20.0432s / 3,470 QPS / 70.0MB

The native numbers are from the latest release-extension perf run on this PR (13f131b): https://github.com/WordPress/sqlite-database-integration/actions/runs/25219828811.

[adamziel]

Let's land this! It shouldn't affect the PHP implementation on hosts without the native extension so it's safe to land. There is a few more rough edges, e.g. comments similar to:

this is exactly the kind of state the reviewer worried

that make no sense after the PR is merged. Also, overly verbose code style such as:

		if ( $lexer instanceof WP_MySQL_Native_Lexer ) {
			$tokens = $lexer->native_token_stream();
			return $this->reset_or_create_parser( $tokens );
		}

		$tokens = $lexer->remaining_tokens();
		return $this->reset_or_create_parser( $tokens );

that could be just

		$tokens = $lexer instanceof WP_MySQL_Native_Lexer
			? $lexer->native_token_stream()
			: $lexer->remaining_tokens();
		return $this->reset_or_create_parser( $tokens );

Or even just this, if we reused the remaining_tokens() method to return the token stream.

		return $this->reset_or_create_parser( $lexer->remaining_tokens() );

So let's address those in a follow up. Thank you @JanJakes!