JCA conformance harness: jtreg run + exclusions - #552
Closed
WillChilds-Klein wants to merge 2 commits into
Closed
Conversation
First branch of issue corretto#550. Wires up a harness for running the JDK's own jtreg crypto conformance tests against ACCP, to catch behavioral divergence from the JDK-default providers (SunJCE, SunEC, SunRsaSign). This commit lands the scaffolding only: - tests/jca-conformance/run-jtreg.sh: driver script. Runs in --smoke mode by default, validating that ACCP installs as the highest-priority provider under the JDK under test and that the config files parse. The --full mode (real OpenJDK test-tree checkout + jtreg run) is a placeholder delivered in branch 2. - tests/jca-conformance/jtreg-test-roots.txt: the JCA-relevant test/jdk subtrees to run (ciphers, Mac, Signature, KeyFactory, EC/RSA, etc.), excluding areas ACCP does not implement (TLS/JSSE, PKCS#11, Kerberos). - tests/jca-conformance/exclusions/{common,jdk17,jdk21}.txt: per-JDK exclusion lists, empty to start. A failing test gets an entry here (with a required rationale comment linking DIFFERENCES.md or an issue) so CI stays green while the divergence backlog stays explicit. - tests/jca-conformance/README.md: design, exclusion model, JCK note. - .github/workflows/jca-conformance.yml: GHA job, matrix JDK 17/21, builds ACCP and runs the harness in --smoke mode (non-gating wiring until branch 2 enables --full). Tracking: corretto#550
Second branch of issue corretto#550. Turns the scaffolding into a working conformance run and switches CI to it. Harness (run-jtreg.sh --full): - Builds jtreg from source at a pinned openjdk/jtreg tag (jtreg-version.txt). No third-party prebuilt binaries. - Sparse-checks-out the OpenJDK test content for the JDK under test from a pinned per-version GA tag (jdk-tags.txt), using a partial + sparse + shallow clone so only the crypto test subtrees and the jtreg support dirs (test/lib, test/jtreg-ext) are pulled, not the whole JDK source tree. Per-version pins because the test set differs by JDK version. - Installs ACCP at provider priority 1 by emitting a renumbered java.security override (ACCP at 1, JDK defaults shifted to 2..N+1). A bare security.provider.1=ACCP would OVERWRITE the SUN provider and break JDK-internal SecureRandom bootstrap (InternalError: SHA-1 not available, since ACCP doesn't register SUN's legacy SHA/SHA1 alias); renumbering keeps every default present so those aliases fall through. - Runs jtreg over the configured roots minus the exclusion lists and exits nonzero on any non-excluded failure. Config: - jdk-tags.txt: openjdk/jdk17u@jdk-17.0.13-ga, openjdk/jdk21u@jdk-21.0.5-ga - jtreg-version.txt: jtreg-7.5.1+1 Seeded exclusions (all triaged as non-ACCP-conformance issues): - common.txt: DefaultProviderList, Versions, SecurityProviderModularTest -- meta-tests asserting the JDK's exact default provider set, which ACCP intentionally perturbs by installing at priority 1. - jdk21.txt: CryptoPermissions/InconsistentEntries -- writes a policy file into the read-only JDK install dir (AccessDeniedException); environmental, would fail the same against SunJCE. Local results against ACCP: - JDK17: 533 passed, 3 excluded, exit 0. - JDK21: 567 passed, 4 excluded, exit 0. Workflow: jca-conformance.yml now runs --full on the JDK 17/21 matrix and uploads the jtreg report as an artifact. Tracking: corretto#550
WillChilds-Klein
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #552 +/- ##
============================================
+ Coverage 78.58% 78.66% +0.07%
- Complexity 1211 1213 +2
============================================
Files 127 127
Lines 7734 7734
Branches 981 981
============================================
+ Hits 6078 6084 +6
+ Misses 1126 1122 -4
+ Partials 530 528 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second of two PRs for #550, stacked on #551. Turns the scaffolding into a working conformance run against ACCP and switches CI to it.
Note
Stacked on #551 (branch-1 scaffolding). Until #551 merges, this PR's diff includes the scaffolding commit too; review #551 first. Once #551 merges to main, this collapses to just the
--fulldelta.Harness (
run-jtreg.sh --full)jtreg-version.txt(openjdk/jtreg). No third-party prebuilt binaries.jdk-tags.txt), via partial (--filter=blob:none) + sparse + shallow clone — only the crypto subtrees + jtreg support dirs (test/lib,test/jtreg-ext), not the whole JDK source (~226 MB vs multi-GB). Per-version pins because the test set differs by JDK version.java.securityoverride (ACCP at 1, JDK defaults shifted to 2..N+1).Why renumber rather than
security.provider.1=ACCPA bare
security.provider.1=ACCPoverwrites slot 1 (theSUNprovider), evicting it — which breaks JDK-internalSecureRandombootstrap withInternalError: SHA-1 not available(ACCP doesn't register SUN's legacySHA/SHA1alias) and fails nearly every test. Renumbering keeps every default present so legacy aliases fall through to SUN. This was the main harness bug found and fixed while bringing the run up.Seeded exclusions
All triaged as not ACCP algorithm-conformance issues:
common.txt:DefaultProviderList,Versions,SecurityProviderModularTest— meta-tests asserting the JDK's exact default provider set, which ACCP intentionally perturbs by installing at priority 1.jdk21.txt:CryptoPermissions/InconsistentEntries— writes a policy file into the read-only JDK install dir (AccessDeniedException); environmental, would fail the same against SunJCE.Each entry carries a rationale comment per the exclusion model.
Results against ACCP (local)
Workflow
jca-conformance.ymlnow runs--fullon the JDK 17/21 matrix and uploads the jtreg report as a CI artifact.Test plan
shellcheckcleanTracking: #550