Develop common reporting / summary format for TCK summaries

[edbratt]

Tracking issue for the platform team to produce recommendation for output format from each TCK and each vehicle summary that will be required for certification reports.

It has been noted that the output from different TCKs can be challenging for developers to track down -- then to determine what to include in the certification report.

This enhancement will track the discussion and development for a recommended reporting format for each TCK to generate -- for each test suite or vehicle that must be used to validate a certification.

For interested readers -- one collection of these data can be viewed from the Eclipse GlassFish 9.1 Platform Certification report.. Today, these summary data are collected as a manual test output review.

A goal of this enhancement will be to designate an output format and common output or summary file (or test output designation delimiters) that will allow an implementation developer to gather these data and include it for the required compatibility certification report.

This issue was initiated after discussion at the Oct. 26 Jakarta EE Platform Committer team meeting.

AB#75

[scottmarlow]

Thanks @PatrickHuang888 for volunteering to help with this issue. I don't think that the Eclipse or JakartaEE github rules allow me to assign this issue to you so I will assign it to myself with this note that the issue is to be considered to be assigned to @PatrickHuang888.

[edburns]

@PatrickHuang888 wrote, on the tck-dev-list:

have checked the current TCK execution results, for instance, running the XA module using run_jakataeetck.sh xa. The XA module will generate a summary.txt file in the directory $jakartaeetck-report/xa/text/, which includes the results. This file lists cases that have passed, failed, or encountered errors.
I think we can execute a command/program after completing all tests to aggregate the summary.txt files from all testing modules. The goal is to generate a Markdown file similar to https://github.com/apusic/aas-tck/blob/master/aas_v10_ee9_full.md. This file can be posted when announcing the certification results.
However, note that my investigation does not cover modules such as DI, CDI, Bean Validation, and Debugging. These modules require separate research to determine the handling approach.
Patrick