JCA conformance harness: jtreg run + exclusions

.github/workflows/jca-conformance.yml

@@ -0,0 +1,61 @@
+name: JCA Conformance
+
+# Runs the JCA/JCE-relevant subset of the OpenJDK jtreg suite against ACCP to
+# catch behavioral divergence from the JDK-default providers. See
+# tests/jca-conformance/README.md and issue #550.
+#
+# The harness builds jtreg from pinned openjdk/jtreg, sparse-checks-out the
+# pinned per-version OpenJDK test content, installs ACCP at provider priority 1,
+# and runs the configured JCA roots with the exclusion lists applied. Tests that
+# fail for a triaged, accepted reason are listed in
+# tests/jca-conformance/exclusions/ with a rationale; CI is green as long as no
+# non-excluded test fails.
+
+on:
+  push:
+    branches:
+      - '*'
+  pull_request:
+    branches:
+      - '*'
+
+permissions:
+  contents: read
+
+jobs:
+  jca-conformance:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        jdk: ['17', '21']
+    name: jtreg (JDK ${{ matrix.jdk }})
+    steps:
+      - name: Set up Corretto ${{ matrix.jdk }}
+        uses: actions/setup-java@v4
+        with:
+          java-version: ${{ matrix.jdk }}
+          distribution: 'corretto'
+
+      - uses: actions/checkout@v4
+        with:
+          submodules: true
+
+      - name: Build ACCP
+        env:
+          TEST_JAVA_HOME: ${{ env.JAVA_HOME }}
+        run: ./gradlew build -x test
+
+      - name: Run JCA conformance harness
+        env:
+          TEST_JAVA_HOME: ${{ env.JAVA_HOME }}
+          ACCP_JAR: ${{ github.workspace }}/build/cmake/AmazonCorrettoCryptoProvider.jar
+        run: ./tests/jca-conformance/run-jtreg.sh --full
+
+      - name: Upload jtreg report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: jtreg-report-jdk${{ matrix.jdk }}
+          path: build/jca-conformance/jtreg-report
+          if-no-files-found: ignore

tests/jca-conformance/README.md

@@ -0,0 +1,141 @@
+# JCA/JCE Conformance Testing
+
+This directory holds the harness that runs JDK-provided cryptography conformance
+tests against ACCP, to catch behavioral divergence between ACCP and the
+JDK-default JCA/JCE providers (SunJCE, SunEC, SunRsaSign, etc.).
+
+Tracking issue: https://github.com/corretto/amazon-corretto-crypto-provider/issues/550
+
+## Why
+
+ACCP registers ahead of the JDK-default providers and overrides many JCA SPIs
+(`Cipher`, `Mac`, `MessageDigest`, `Signature`, `KeyFactory`, `KeyPairGenerator`,
+`KeyAgreement`, `KeyGenerator`, `SecretKeyFactory`, `KEM`, ...). The JDK ships an
+extensive regression suite (`jtreg`) under `test/jdk/.../crypto`,
+`test/jdk/.../security` that exercises concrete provider behavior — exception
+types, parameter-spec acceptance, key-encoding round-trips, and KAT vectors.
+Running those tests with ACCP installed surfaces divergences our own unit tests
+don't.
+
+## Layout
+
+```
+tests/jca-conformance/
+  README.md            -- this file
+  run-jtreg.sh         -- driver: build jtreg, fetch tests, run against ACCP
+  jdk-tags.txt         -- per-JDK pin: repo + GA tag for the test content
+  jtreg-version.txt    -- pinned openjdk/jtreg tag to build the harness from
+  jtreg-test-roots.txt -- which test/jdk subtrees to run (JCA-relevant only)
+  exclusions/
+    common.txt         -- tests excluded on all JDKs
+    jdk17.txt          -- tests excluded only on JDK 17
+    jdk21.txt          -- tests excluded only on JDK 21
+```
+
+## Harness model
+
+`run-jtreg.sh --full`:
+
+1. Resolves the JDK under test (`TEST_JAVA_HOME`) and its major version.
+2. Builds `jtreg` from source at the tag pinned in `jtreg-version.txt`
+   (`openjdk/jtreg`). No third-party prebuilt binaries.
+3. Sparse-checks-out the OpenJDK test content for this JDK major from the repo
+   and GA tag pinned in `jdk-tags.txt` (e.g. `openjdk/jdk17u @ jdk-17.0.13-ga`),
+   using a partial (`--filter=blob:none`) + sparse + shallow clone so only the
+   crypto test subtrees and the jtreg support dirs (`test/lib`,
+   `test/jtreg-ext`) are pulled — not the whole JDK source tree. The test set
+   genuinely differs per JDK version, which is why each major is pinned
+   separately.
+4. Builds ACCP (or consumes a prebuilt `AmazonCorrettoCryptoProvider.jar`) and
+   installs it at provider priority 1 (see below).
+5. Runs jtreg over the roots in `jtreg-test-roots.txt`, minus the tests named in
+   the applicable `exclusions/*.txt`, and exits nonzero if any non-excluded test
+   fails.
+
+`run-jtreg.sh --smoke` (the default) validates the wiring without the OpenJDK
+checkout: it confirms ACCP installs as provider #1 under the JDK under test and
+that the config files parse.
+
+### Adding / updating
+
+- **Update test content for a version:** bump its tag in `jdk-tags.txt`.
+- **Add a JDK version:** add a line to `jdk-tags.txt` AND a matrix entry in
+  `.github/workflows/jca-conformance.yml`.
+- **Update the harness:** bump the tag in `jtreg-version.txt`.
+
+### Provider installation
+
+ACCP is installed at provider priority 1 via a generated `java.security`
+override passed to every test JVM with `-Djava.security.properties=`.
+
+Important subtlety: a bare `security.provider.1=<ACCP>` does **not** prepend —
+it *overwrites* slot 1 (the `SUN` provider on a stock JDK), evicting it. That
+breaks JDK-internal `SecureRandom` bootstrap, which needs SUN's legacy `SHA`
+(a.k.a. `SHA1`) `MessageDigest` alias that ACCP does not register, surfacing as
+`InternalError: SHA-1 not available` on nearly every test. The harness therefore
+discovers the JDK's default provider order at runtime and emits a renumbered
+list: ACCP at 1, the JDK defaults shifted to 2..N+1, so nothing is evicted and
+legacy aliases fall through to SUN.
+
+The native library is self-extracted from `AmazonCorrettoCryptoProvider.jar` by
+ACCP's `Loader`, so only the jar needs to be on the (boot)classpath — no
+separate `java.library.path` is required.
+
+## Exclusions
+
+A test belongs in an `exclusions/*.txt` file when it fails under ACCP for a
+reason we have triaged and accepted. Two categories:
+
+- **Intentional behavioral difference.** ACCP deliberately diverges from the
+  JDK-default provider (documented in [`DIFFERENCES.md`](../../DIFFERENCES.md)).
+  The exclusion entry MUST reference the relevant section or a tracking issue.
+- **Known bug, not yet fixed.** A real ACCP defect we have not addressed. The
+  exclusion entry MUST reference a tracking issue so it can be removed once
+  fixed.
+
+This keeps CI green while making the backlog of divergences explicit and
+reviewable in source. CI fails if a test that is NOT excluded fails, and (once
+enabled) if an excluded test unexpectedly passes — so stale exclusions get
+flagged.
+
+### Exclusion file format
+
+One test selector per line. Blank lines and lines beginning with `#` are
+ignored. Every non-comment line MUST be preceded by a comment giving the
+rationale and a link (DIFFERENCES.md section or issue URL):
+
+```
+# AES/GCM rejects reusing an IV across encryptions; SunJCE allows it.
+# See DIFFERENCES.md "AES-GCM IV reuse".
+com/sun/crypto/provider/Cipher/AEAD/GCMIvReuse.java
+
+# BUG: ECDSA signature with a truncated public key throws a different
+# exception type than SunEC. https://github.com/corretto/.../issues/NNN
+sun/security/ec/SignatureKAT.java
+```
+
+The selector is a path relative to the OpenJDK `test/jdk` root, matching the
+form jtreg uses in its reports.
+
+## Running locally
+
+```bash
+# Smoke (default): validate wiring only. Fast; uses the JDK on PATH (17+).
+./tests/jca-conformance/run-jtreg.sh
+
+# Full: build jtreg, fetch the pinned test content, run the suite against ACCP.
+# Builds ACCP if no jar is supplied. First run is slow (jtreg build + checkout).
+ACCP_JAR=/path/to/AmazonCorrettoCryptoProvider.jar \
+  TEST_JAVA_HOME=/path/to/jdk21 \
+  ./tests/jca-conformance/run-jtreg.sh --full
+
+# Enumerate failures without failing (used when seeding exclusion lists):
+KEEP_GOING=true ./tests/jca-conformance/run-jtreg.sh --full
+```
+
+## JCK
+
+Oracle's JCK (and the OpenJDK Community TCK) require a signed access agreement
+and cannot be fetched anonymously from CI. Onboarding JCK is tracked separately
+in issue #550; once access is provisioned, a sibling `run-jck.sh` will live here
+and reuse the same exclusion model. For now this harness covers jtreg only.

tests/jca-conformance/exclusions/common.txt

@@ -0,0 +1,31 @@
+# JCA/JCE conformance test exclusions (common).
+#
+# Each non-comment line is a jtreg test selector (path relative to the OpenJDK
+# test/jdk root) that fails under ACCP for a triaged, accepted reason. Every
+# entry MUST be preceded by a comment giving the rationale and a link to either
+# a DIFFERENCES.md section or a tracking issue. See ../README.md for details.
+#
+# Entries here apply to ALL JDK versions. JDK-specific exclusions go in
+# exclusions/jdk<major>.txt.
+
+# The three tests below are meta-tests that assert the JDK's *exact default
+# provider configuration*. ACCP deliberately installs itself at provider
+# priority 1 (ahead of the JDK defaults) -- that is the entire point of the
+# provider -- so these tests cannot pass with ACCP installed. None of them
+# exercise algorithm behavior, so excluding them does not reduce conformance
+# coverage.
+
+# Asserts the running provider list equals the JDK default list. ACCP is
+# intentionally present at position 1. (Also trips a ServiceLoader
+# instantiation path under the test's classloader.)
+java/security/Provider/DefaultProviderList.java
+
+# Verifies JDK provider version strings are consistent with the JDK license/
+# code. ACCP at position 1 changes what the test observes about the provider
+# set; not an ACCP algorithm concern.
+java/security/misc/Versions.java
+
+# Exercises modular (JPMS) provider loading of the JDK default providers. Does
+# not compile/run meaningfully with ACCP injected via -Xbootclasspath/a in this
+# harness; it tests the JDK's own modular provider plumbing, not ACCP.
+java/security/Provider/SecurityProviderModularTest.java

tests/jca-conformance/exclusions/jdk17.txt

@@ -0,0 +1,9 @@
+# JCA/JCE conformance test exclusions (jdk17).
+#
+# Each non-comment line is a jtreg test selector (path relative to the OpenJDK
+# test/jdk root) that fails under ACCP for a triaged, accepted reason. Every
+# entry MUST be preceded by a comment giving the rationale and a link to either
+# a DIFFERENCES.md section or a tracking issue. See ../README.md for details.
+#
+# This file starts empty; branch 2 of issue #550 populates it from the first
+# real run of the harness against ACCP.

tests/jca-conformance/exclusions/jdk21.txt

@@ -0,0 +1,16 @@
+# JCA/JCE conformance test exclusions (jdk21).
+#
+# Each non-comment line is a jtreg test selector (path relative to the OpenJDK
+# test/jdk root) that fails under ACCP for a triaged, accepted reason. Every
+# entry MUST be preceded by a comment giving the rationale and a link to either
+# a DIFFERENCES.md section or a tracking issue. See ../README.md for details.
+#
+# Entries here apply only to JDK 21. Cross-version exclusions go in common.txt.
+
+# Environmental, not ACCP: this test writes a crypto policy file into the JDK
+# installation tree (<jdk>/conf/security/policy/testlimited), which fails with
+# AccessDeniedException on a read-only/system-installed JDK. It would fail the
+# same way against SunJCE; it exercises JDK crypto-policy plumbing, not ACCP.
+# (The CryptoPermissions policy tests differ by JDK version, so this is scoped
+# to JDK 21 rather than common.)
+javax/crypto/CryptoPermissions/InconsistentEntries.java

tests/jca-conformance/jdk-tags.txt

@@ -0,0 +1,16 @@
+# Per-JDK-major pin of the OpenJDK update repo and GA tag from which the jtreg
+# test content (test/jdk crypto subtrees) is fetched. The test set differs by
+# JDK version, so each supported major is pinned to a specific General
+# Availability tag for reproducibility.
+#
+# Format (whitespace-separated):  <jdk-major> <github-repo> <git-tag>
+#
+# To bump test content for a version, change its tag. To add a JDK version, add
+# a line here AND a matrix entry in .github/workflows/jca-conformance.yml.
+#
+# Tags are GA releases from the openjdk update-release repos:
+#   https://github.com/openjdk/jdk17u  (tags like jdk-17.0.13-ga)
+#   https://github.com/openjdk/jdk21u  (tags like jdk-21.0.5-ga)
+
+17  openjdk/jdk17u  jdk-17.0.13-ga
+21  openjdk/jdk21u  jdk-21.0.5-ga

tests/jca-conformance/jtreg-test-roots.txt

@@ -0,0 +1,29 @@
+# OpenJDK jtreg test roots to run against ACCP, relative to the `test/jdk`
+# directory of the OpenJDK source tree. One root per line; blank lines and
+# lines starting with `#` are ignored.
+#
+# Scope: only JCA/JCE areas where ACCP overrides a default-provider SPI. We
+# deliberately exclude TLS/JSSE (ACCP does not provide an SSLContext), PKCS#11,
+# Kerberos, and other areas ACCP does not touch.
+#
+# This list is intentionally conservative to start; branch 2 of issue #550
+# populates and trims it based on what actually runs against ACCP. Keep entries
+# scoped to directories that exercise algorithm SPIs ACCP implements.
+
+# Symmetric ciphers, AEAD, key wrap, KDFs, Mac, SecretKeyFactory (SunJCE).
+com/sun/crypto/provider
+
+# Public JCE API conformance: Cipher, Mac, KeyAgreement, KEM, KeyGenerator, etc.
+javax/crypto
+
+# MessageDigest, Signature, KeyFactory, KeyPairGenerator API conformance.
+java/security
+
+# EC/EdDSA provider tests (SunEC).
+sun/security/ec
+
+# RSA provider tests (SunRsaSign).
+sun/security/rsa
+
+# Shared provider-level tests (SecureRandom, algorithm parameters, etc.).
+sun/security/provider

tests/jca-conformance/jtreg-version.txt

@@ -0,0 +1,4 @@
+# Pinned openjdk/jtreg tag to build the harness from. jtreg is built from source
+# (no third-party prebuilt binaries) for upstream provenance and reproducibility.
+# https://github.com/openjdk/jtreg/tags
+jtreg-7.5.1+1