TCK challenge for adding support for running Security 3.0 signature TCK tests on Java 21

[scottmarlow]

When running the https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.1.zip tests on Java 21, the following signature test failures occur due to Java 21 added a few new JDK methods:

12-01-2023 11:32:57:********** Status Report 'jakarta.security.enterprise.identitystore.openid' **********
12-01-2023 11:32:57:SignatureTest report
Base version: 3.0.0
Tested version: 3.0.0
Check mode: src [throws normalized]
Constant checking: on

Added Superclasses or Superinterfaces

jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String>
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>

Added Methods

jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed()
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)

12-01-2023 11:32:57:********** Package 'jakarta.security.enterprise.identitystore.openid' - FAILED (STATIC MODE) **********

12-01-2023 11:32:58:********** Status Report 'jakarta.security.enterprise.identitystore.openid' **********

12-01-2023 11:32:58:SignatureTest report
Base version: 3.0.0
Tested version: 3.0.0
Check mode: src [throws normalized]
Constant checking: on

Added Superclasses or Superinterfaces

jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String>
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>

Added Methods

jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed()
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)

12-01-2023 11:32:58:SigTestEE.signatureTest() failed!, diffs found
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.252 s <<< FAILURE! - in ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT
[ERROR] ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest Time elapsed: 6.074 s <<< ERROR!
ee.jakarta.tck.security.signaturetest.SigTestEE$Fault: SigTestEE.signatureTest() failed!, diffs found
at security.tck.sigtest@3.0.0/ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest(SecurityAPISigTestIT.java:355)

There are two possible fixes, the easiest one line change would be to exclude the specific updated JDK classes on the command line when invoking the sigtest tool. I'll look into that next year.

The better change would be to update https://github.com/jtulach/netbeans-apitest to be able to avoid checking any SPEC API classes that are not in the jakarta namespace which I think would avoid validation of any JDK classes and instead focus only on checking jakarta.* classes. I'd like to do this change for Jakarta EE 11+.

[scottmarlow]

Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the challenge label. Thanks!

[scottmarlow]

I can confirm that the jakartaee/platform-tck#1206 change solved ^ running the (Jakarta EE 10) Platform TCK Signature tests on Java 21 and the same change should help the Security TCK.

[scottmarlow]

Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the challenge label. Thanks!

^ is done and the updated Platform TCK 10.0.3 TCK is released as mentioned on https://jakarta.ee/specifications/platform/10

[scottmarlow]

Could one of the Security project committers please add the challenge label and initiate process to accept or reject this TCK challenge.

[arjantijms]

@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.

See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest

[scottmarlow]

@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.

See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest

@arjantijms to be clear, I think that we need a https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.2.zip release that includes the #305 fix as well.

[arjantijms]

@scottmarlow okay, that should be doable

[arjantijms]

I've staged 3.0.2 here, but not checked it yet:

https://www.eclipse.org/downloads/download.php?file=/security/jakartaee10/staged/eftl/jakarta-security-tck-3.0.2.zip

[scottmarlow]

I've staged 3.0.2 here, but not checked it yet:

https://www.eclipse.org/downloads/download.php?file=/security/jakartaee10/staged/eftl/jakarta-security-tck-3.0.2.zip

Thanks, I'm working on running it locally with latest WildFly on Java 17 and will then run again on Java 21.