Add bidirectional index/codec compatibility harness for Lucene, #270 - #1351
Open
paulirwin wants to merge 7 commits into
Open
Add bidirectional index/codec compatibility harness for Lucene, #270#1351paulirwin wants to merge 7 commits into
paulirwin wants to merge 7 commits into
Conversation
…pache#270 Adds a two-way check that Lucene.NET and Apache Lucene 4.8.1 (Java) can each read an index written by the other. A new self-contained Maven project under src/java/index-compat generates and reads indexes; the .NET side is covered by TestJavaCompatibility in the _I-J test shard. Both runtimes write the same deterministic document set (mirroring the schema in TestBackwardsCompatibility), then the other runtime opens it, runs CheckIndex (the codec integrity gate over the per-file checksums), and asserts the contents match. No byte-for-byte comparison is done, since some header fields (e.g. java.vendor) legitimately differ between runtimes. No index fixtures are committed: every index is generated fresh into the gitignored work/ folder. run-compat.sh / run-compat.ps1 drive both directions end to end and return a non-zero exit code on failure. In .NET a missing compat index makes the cross-runtime test inconclusive (a JDK may be absent); in Java a missing .NET index fails the test. The .NET writer forces the real Lucene46Codec, since LuceneTestCase otherwise randomizes Codec.Default to test-only impostors a stock Lucene cannot load. Also adds a hand-maintained GitHub Actions workflow (not generated by Generate-TestWorkflows.ps1, since it needs a JDK and a Maven <-> dotnet orchestration) that runs both directions on a matrix of ubuntu/windows x net10.0/net8.0/net472/net48. Each .NET step asserts "Passed: 1" so a skipped or filtered-out test fails the job instead of going green silently. The workflow uses its own 'compat-' prefixed NuGet cache key to avoid colliding with the generated workflows' full-solution cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NightOwl888
requested changes
Jun 16, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a bidirectional Lucene.NET ↔ Apache Lucene 4.8.1 (Java) index/codec compatibility harness. The harness deterministically generates equivalent indexes in both runtimes, validates them via CheckIndex, and semantically verifies document/content/docvalues parity, with scripts and CI orchestration to run both directions.
Changes:
- Added a shared deterministic document/index contract (
CompatDocs) and .NET tests to (a) round-trip locally, (b) read Java-generated indexes, and (c) write .NET indexes for Java to read. - Added a self-contained Maven project (
src/java/index-compat) that writes Java indexes and JUnit-tests reading .NET indexes. - Added cross-platform driver scripts plus a dedicated GitHub Actions workflow to execute both directions on an OS/TFM matrix.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Lucene.Net.Tests/Support/Index/TestJavaCompatibility.cs | .NET-side tests for round-trip, Java→.NET read, and .NET→Java write, including semantic verification. |
| src/Lucene.Net.Tests/Support/Index/CompatDocs.cs | Deterministic shared document set and index writer logic for the cross-runtime contract. |
| src/java/index-compat/src/test/java/org/apache/lucenenet/compat/TestDotNetCompatibility.java | JUnit test that validates Lucene 4.8.1 can read a Lucene.NET-written index and contents match contract. |
| src/java/index-compat/src/main/java/org/apache/lucenenet/compat/GenerateIndex.java | Java CLI entry point to generate Java-written compatibility indexes. |
| src/java/index-compat/src/main/java/org/apache/lucenenet/compat/CompatDocs.java | Java mirror of the deterministic contract and index writer / verifier helpers. |
| src/java/index-compat/run-compat.sh | Bash entry-point wrapper that invokes the PowerShell driver. |
| src/java/index-compat/run-compat.ps1 | Cross-platform orchestration script to run both directions end-to-end. |
| src/java/index-compat/run-compat.bat | Windows CMD wrapper for invoking the PowerShell driver. |
| src/java/index-compat/README.md | Documentation for running the harness and explaining the contract. |
| src/java/index-compat/pom.xml | Maven project definition for Lucene 4.8.1 compatibility harness and JUnit execution. |
| src/java/index-compat/mvnw.cmd | Maven wrapper (Windows). |
| src/java/index-compat/mvnw | Maven wrapper (POSIX). |
| src/java/index-compat/.mvn/wrapper/maven-wrapper.properties | Maven wrapper configuration. |
| .rat-excludes | Excludes Maven wrapper artifacts from RAT checks. |
| .gitignore | Ignores Java build outputs and harness-generated work/ directories. |
| .github/workflows/Lucene-Net-Index-Compatibility.yml | Dedicated CI workflow to run both directions across OS/TFM matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add bidirectional index/codec compatibility harness for Lucene
Fixes #270
Description
Adds a two-way check that Lucene.NET and Apache Lucene 4.8.1 (Java) can each read an index written by the other. A new self-contained Maven project under src/java/index-compat generates and reads indexes; the .NET side is covered by TestJavaCompatibility in the _I-J test shard.
Both runtimes write the same deterministic document set (mirroring the schema in TestBackwardsCompatibility), then the other runtime opens it, runs CheckIndex (the codec integrity gate over the per-file checksums), and asserts the contents match. No byte-for-byte comparison is done, since some header fields (e.g. java.vendor) legitimately differ between runtimes.
No index fixtures are committed: every index is generated fresh into the gitignored
work/folder.run-compat.sh/run-compat.ps1drive both directions end to end and return a non-zero exit code on failure. In .NET a missing compat index makes the cross-runtime test inconclusive (i.e. if you just cloned the code and haven't run the compat script yet); in Java a missing .NET index fails the test. The .NET writer forces the real Lucene46Codec, since LuceneTestCase otherwise randomizes Codec.Default to test-only impostors a stock Lucene cannot load.Also adds a hand-maintained GitHub Actions workflow (not generated by Generate-TestWorkflows.ps1, since it needs a JDK and a Maven <-> dotnet orchestration) that runs both directions on a matrix of ubuntu/windows x net10.0/net8.0/net472/net48. Each .NET step asserts "Passed: 1" so a skipped or filtered-out test fails the job instead of going green silently. The workflow uses its own 'compat-' prefixed NuGet cache key to avoid colliding with the generated workflows' full-solution cache.
AI: Written with Claude Code, Opus 4.8, and reviewed/edited by hand.