Skip to content

JCA conformance harness: jtreg run + exclusions - #557

Open
WillChilds-Klein wants to merge 1 commit into
jca-conformance-scaffoldingfrom
jca-conformance-jtreg-run
Open

JCA conformance harness: jtreg run + exclusions#557
WillChilds-Klein wants to merge 1 commit into
jca-conformance-scaffoldingfrom
jca-conformance-jtreg-run

Conversation

@WillChilds-Klein

@WillChilds-Klein WillChilds-Klein commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Second of two PRs for #550, stacked on #556. Turns the scaffolding into a working conformance run against ACCP and switches CI to it.

Harness (run-jtreg.sh --full)

  • Builds jtreg from source at the tag pinned in jtreg-version.txt (openjdk/jtreg). No third-party prebuilt binaries.
  • Sparse-checks-out the OpenJDK test content for the JDK under test from a pinned per-version GA tag (jdk-tags.txt), via partial (--filter=blob:none) + sparse + shallow clone — only the crypto subtrees + jtreg support dirs (test/lib, test/jtreg-ext), not the whole JDK source (~226 MB vs multi-GB). Per-version pins because the test set differs by JDK version.
  • Installs ACCP at provider priority 1 via a renumbered java.security override (ACCP at 1, JDK defaults shifted to 2..N+1).

Why renumber rather than security.provider.1=ACCP

A bare security.provider.1=ACCP overwrites slot 1 (the SUN provider), evicting it — which breaks JDK-internal SecureRandom bootstrap with InternalError: SHA-1 not available (ACCP doesn't register SUN's legacy SHA/SHA1 alias) and fails nearly every test. Renumbering keeps every default present so legacy aliases fall through to SUN. This was the main harness bug found and fixed while bringing the run up.

Seeded exclusions

All triaged as not ACCP algorithm-conformance issues:

  • common.txt: DefaultProviderList, Versions, SecurityProviderModularTest — meta-tests asserting the JDK's exact default provider set, which ACCP intentionally perturbs by installing at priority 1.
  • jdk21.txt: CryptoPermissions/InconsistentEntries — writes a policy file into the read-only JDK install dir (AccessDeniedException); environmental, would fail the same against SunJCE.

Each entry carries a rationale comment per the exclusion model.

Results against ACCP (local)

JDK passed excluded exit
17 533 3 0
21 567 4 0

Workflow

jca-conformance.yml now runs --full on the JDK 17/21 matrix and uploads the jtreg report as a CI artifact.

Test plan

  • shellcheck clean
  • JDK17 full run green (533 passed, 3 excluded)
  • JDK21 full run green (567 passed, 4 excluded)
  • CI: full job green on JDK 17 and 21

Tracking: #550

Second branch of issue #550. Turns the scaffolding into a working
conformance run and switches CI to it.

Harness (run-jtreg.sh --full):
- Builds jtreg from source at a pinned openjdk/jtreg tag
  (jtreg-version.txt). No third-party prebuilt binaries.
- Sparse-checks-out the OpenJDK test content for the JDK under test
  from a pinned per-version GA tag (jdk-tags.txt), using a partial +
  sparse + shallow clone so only the crypto test subtrees and the
  jtreg support dirs (test/lib, test/jtreg-ext) are pulled, not the
  whole JDK source tree. Per-version pins because the test set differs
  by JDK version.
- Installs ACCP at provider priority 1 by emitting a renumbered
  java.security override (ACCP at 1, JDK defaults shifted to 2..N+1).
  A bare security.provider.1=ACCP would OVERWRITE the SUN provider and
  break JDK-internal SecureRandom bootstrap (InternalError: SHA-1 not
  available, since ACCP doesn't register SUN's legacy SHA/SHA1 alias);
  renumbering keeps every default present so those aliases fall through.
- Runs jtreg over the configured roots minus the exclusion lists and
  exits nonzero on any non-excluded failure.

Config:
- jdk-tags.txt: openjdk/jdk17u@jdk-17.0.13-ga, openjdk/jdk21u@jdk-21.0.5-ga
- jtreg-version.txt: jtreg-7.5.1+1

Seeded exclusions (all triaged as non-ACCP-conformance issues):
- common.txt: DefaultProviderList, Versions, SecurityProviderModularTest
  -- meta-tests asserting the JDK's exact default provider set, which
  ACCP intentionally perturbs by installing at priority 1.
- jdk21.txt: CryptoPermissions/InconsistentEntries -- writes a policy
  file into the read-only JDK install dir (AccessDeniedException);
  environmental, would fail the same against SunJCE.

Local results against ACCP:
- JDK17: 533 passed, 3 excluded, exit 0.
- JDK21: 567 passed, 4 excluded, exit 0.

Workflow: jca-conformance.yml now runs --full on the JDK 17/21 matrix
and uploads the jtreg report as an artifact.

Tracking: #550
@WillChilds-Klein
WillChilds-Klein force-pushed the jca-conformance-jtreg-run branch from 91dd324 to 09eb4ff Compare July 31, 2026 17:15
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.56%. Comparing base (6822ebc) to head (09eb4ff).

Additional details and impacted files
@@                        Coverage Diff                        @@
##             jca-conformance-scaffolding     #557      +/-   ##
=================================================================
- Coverage                          78.56%   78.56%   -0.01%     
+ Complexity                          1218     1217       -1     
=================================================================
  Files                                127      127              
  Lines                               7760     7759       -1     
  Branches                             983      982       -1     
=================================================================
- Hits                                6097     6096       -1     
- Misses                              1132     1133       +1     
+ Partials                             531      530       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant