Need to define how specification TCKs that reference other specification containers are handled

[starksm64]

Is your feature request related to a problem? Please describe.
We have a problem with circular dependencies in TCKs for specifications like CDI that in addition having a base Java SE behavior, define behaviors and have TCK tests for many other specification containers; EJBs, JMS, JPA, JTA, Servlet, etc.

Describe the solution you'd like
The CDI team is in the process of separating out the TCK tests into two sets; a Java SE base and a Web profile base. We need this separation of dependencies in general for specifications that are targeting the core profile and to break current circular dependencies in the overall release process as it relates to TCKs. A natural place for these separated TCK tests would be the lowest profile/platform that encompasses the required containers.

Describe alternatives you've considered
The specification projects could have separate TCK release artifacts that are extensions of the profile/platform TCK that are new standalone TCKs that need to be run as part of the profile/platform TCK.

Additional context
None

[kwsutter]

A natural place for these separated TCK tests would be the lowest profile/platform that encompasses the required containers.

Just to clarify this item... When you mention a "natural place", I hope you are referring to a logical place and not a physical place. For example, we already have the problem with the Platform TCK containing loads of tests for individual specs (ie, ejb). And, we have Issue #333 to help define how to break these apart into separate components. So, to clarify, these separated TCKs that are mentioned in this Issue would still be physically housed with the individual Specification project. But, the collection and execution of these tests may be performed by the lowest profile (ie, core or web or platform). Do I have your intent correct?

[starksm64]

The problem is how do cross cutting concern specs like CDI, security, transactions, etc. define TCK tests that validate expected behaviors in a given container. For example, all of these specs define expectations when running in a servlet container. Does the servlet spec take on the responsibility for these tests and the spec text in collaboration with the other spec team, or are these cross cutting concern tests that live in the profile that is incorporating the composition of specifications.

[scottmarlow]

As mentioned in the January, 2022 Platform call minutes, we had TCK call #6 to discuss handling of split-out-tcks.

At the end of the recorded call, we noted the action item to be taken in the minutes (search for Action: Platform TCK team and new split out TCK (Spec API) team).

Describe the actions to be taken for removing duplicate tests from Platform TCK

• After new SPEC API TCK is considered complete (e.g. new TCK will definitely be included) for EE release
  • Platform TCK pull request is needed to eliminate any Platform TCK tests not needed to validate Platform SPEC requirements.
    • Send email to Platform, Platform TCK, relevant SPEC API mailing lists with PR link for removing duplicate tests.
      • If there is disagreement about removing duplicate tests, the Platform committer team can be asked to vote on the removal of the tests.
      • The pull request will remain open during the discussion period of two weeks.
    • The relevant EE release will block on waiting for the Platform TCK to be updated to remove tests not needed to validate Platform SPEC requirements.

Guidance for creating the Platform TCK pull request to remove tests not needed to validate Platform SPEC requirements

• Ensure that some validation of SPEC API is done for each tested EE container/vehicle.
• Review each test and make a determination if there is a specific Platform requirement being tested beyond just using any parts of SPEC API in specified containers.
• If the new SPEC API TCK validates the Platform requirements (with required EE container/vehicles), all of the Platform TCK tests specific to that SPEC API may be removed.
• Once the Platform TCK refactoring is complete, revise the guidance here to instead update the Platform TCK to depend directly on the relevant SPEC API TCK tests.

Guidance for how specification TCKs that reference other specification containers are handled

• If your specification TCK depends on other specifications (for TCK testing) that may change (e.g. public class methods/fields removed) after your specification is released as final, your specification may need to be released again with TCK test updates and verified that a compatible implementation still passes.
  • The reason to release again is for proving that a compatible implementation can pass.
• There must be at least one Jakarta Platform implementation that passes your specification TCK EE mode tests.
  • As per the Platform TCK user guide stated requirements.
  • There must be at least one Jakarta Platform implementation that passes your specification TCK SE mode tests.
    • If the TCK EE mode tests include running the SE mode tests, there is no need to separately run the SE mode tests.