Testing of the toolkit

[aditigopalan]

Get feedback on testing and sustainability reports from Jineta, Adam, Orion

Here’s a list of scenarios you could test for the toolkit:

You should be able to follow the README to execute

• Test execution with a single tool
• Test execution involving different tools
• Test using a sample sheet
• Test the consistency and repeatability of the AI-generated reports
• Did you see any errors?
• How did you find the documentation?
• Any other feedback?

[jaybee84]

Tested execution on a single tool: https://cancercomplexity.synapse.org/Explore/Tools/DetailsPage?toolName=AMBRA_low

Generated AI report: AMBRA_low_ai_analysis.html

Did you see any errors?

no
The AI report was comprehensive.
The JOSS report was generated: joss_report_AMBRA_low.json

How did you find the documentation?

Documentation was adequate to setup Synapse agent and link auth token to Nextflow secrets

Any other feedback?

1. It would be great to include extraction of SYNAPSE_AUTH_TOKEN from env variables if not provided.
2. For the run log:

 N E X T F L O W   ~  version 24.04.1

Launching `main.nf` [modest_fermat] DSL2 - revision: 5d709b8730

executor >  local (5)
[9a/e5ab93] ProcessRepo (1)                 [100%] 1 of 1 ✔
[ea/6d6a3b] RunAlmanack (1)                 [100%] 1 of 1 ✔
[7c/a71ce3] TestExecutor (1)                [100%] 1 of 1 ✔
[4a/edd21d] AnalyzeJOSSCriteria (AMBRA_low) [100%] 1 of 1 ✔
[76/e33ce8] AIAnalysis (1)                  [100%] 1 of 1 ✔
Completed at: 11-Jul-2025 13:04:47
Duration    : 3m 34s
CPU hours   : 0.1
Succeeded   : 5

It would be great to include the name and path of the output file for each of the above processes to make it easy for the user to find the right output file.

3. Doc update suggestion: Suggest changing almanack_results.json: Detailed metrics from Almanack analysis to almanack_results_<repo_name>.json: Detailed metrics from Almanack analysis to maintain consistency with output_file style.

Great work @aditigopalan !

[d33bs]

Nice job @aditigopalan (and team)! Providing some feedback below for thoughts. I'd love to follow up on this with another review if I can try out the AI-based feedback (I couldn't seem to get this to work).

Task	Feedback
Test execution with a single tool	Here's my use of the single tool execution as a notebook (cells prefixed with ! are executed as though through the terminal). I used uv via uvx --from jupyterlab jupyter-lab to run an ephemeral Jupyter Lab instance within the repository's context. I leveraged jq to "pretty print" the JSON results to screen. It seemed like --synapse_agent_id might be now required (we can't run without using this). Consider making this concrete within the readme to help avoid confusion. I was able to find a synapse_agent_id to use from the documentation but I wasn't sure what these might mean. Would it make sense to provide a small portion of documentation telling the audience what these ID's represent? Repo URL's appeared to require the .git extension at the end. The feedback from the error was helpful! I wonder at the same time if passing in a repo URL without the .git might be a more common default (I typically copy these from the browser navigation bar directly). The results from test_results_CytoTable.json looked like they might be trying to execute tests for the project. The error looks like it ran into a missing import for duckdb which is used in the cytotable project to help with data management. Running tests for an external project might be challenging because of differences in the envrionment management for Python and other languages. I haven't looked in-depth into how the test execution design works within cckp-toolkit-workflow but at a high-level I'd recommend using code coverage report data (for ex, coverage.py) from the repositories or GitHub Actions execution success data (which may be inferred to mean that the environment built and was used for testing, though it may not always be used this way). Both of these are provided through the almanack checks (repo-code-coverage-percent and repo-gh-workflow-success-ratio) - so it's possible that you could read from the almanack JSON and report back more specifically on what happened. Given all the different JSON reports I felt it might be helpful to store these in a joint JSON file of some kind or perhaps store them as JSON values within an embedded database like DuckDB which supports JSON storage and querying over SQL. Maybe each repository could have it's own DuckDB result (e.g. cckp-report-cytotable.duckdb), or you could abstract to "run id's" which can capture one or many results (which I imagine I might find more detail about below! 🙂). Using a single data structure might make it quick to make dynamic queries over the data (whether through JSON/jq-like querying, SQL, or perhaps graph-based). I noticed that I might be missing the AI-based reports and saw an error: [da/a5f75a] NOTE: Missing output file(s) `CytoTable_ai_analysis.html` expected by process `AIAnalysis (1)` -- Error is ignored I wasn't sure how to troubleshoot this exactly - I might have missed a step! Let me know if I can try again here with a different configuration.
Test execution involving different tools	Here's my use of the different tool execution as a notebook. See earlier results for an explanation of how I did this in the context of this testing. I made a guess that I was supposed to try multiple tools by using the --repo_url flag. I think I did this wrong! The result was that the workflow seemed to take the last --repo_url provided and it showed an error: NOTE: Missing output file(s) `cosmicqc_ai_analysis.html` expected by process `AIAnalysis (1)` -- Error is ignored While working through this I had to do a double take on the results vs test-results directory. If test-results are a part of the tests for this project, consider nesting them under the tests directory to make clear that these results don't pertain to non-software-test results.
Test using a sample sheet	Here's my use of the sample sheet execution as a notebook. See earlier results for an explanation of how I did this in the context of this testing. This was neat to see in action and really shows how this can scale! I mentioned this in an earlier comment and wondered here again: it could be helpful to structure the data in a way that it can be queried all at once. I experimented with doing this through the DuckDB CLI near the end of the notebook to demonstrate one way of doing this (perhaps this is already being done somewhere internally). If you do opt for this type of structure another thing which could help would be to lean into a loose JSONSchema to validate the structures or perhaps using Parquet files (which can be read as multi-file datasets using directories). If you use Parquet or a similar format you could use the results as the distributed database of the system (updates could be applied at the file level within object storage and you could use query engine to grab / display data as needed). This might also address change data capture (to understand how the projects change over time, through for instance, s3 object versioning).
Test the consistency and repeatability of the AI-generated reports	I couldn't see the results here so I might need to follow up once I understand this better or get some help! Sorry about this.
Did you see any errors?	There were errors with the AI-based results which I mentioned earlier. Otherwise, all human-errors with command line usage.
How did you find the documentation?	I was able to read through the readme and found it generally really helpful! I have further feedback below.
Any other feedback?	Python 3.8 doesn't appear to be in use anymore. There are no git installation instructions. I recognize these may be implied but it could be nice to follow the cadence with how the other libraries have installation details provided. Under Optional Dependencies there are a number of Synapse related items which might benefit from links or guides. I think these all exist somewhere and it's a matter of looking it up but we could save a few minutes for readers by linking directly. The Python dependencies and requirements.txt seem to no longer be present. Consider removing this from the readme (it seems that the dependencies might be managed through the Docker image instead). What permissions are needed for the Synapse personal token? Consider adding this level of detail to the setup. I'll try to follow up with some improvements to the readme since I mentioned them here (stay tuned)!

[Bankso]

Excellent work on this, @aditigopalan ! Here is my testing feedback:

---

I tested the workflow by cloning the current repo and running nextflow commands from the repo root directory

• Test execution with a single tool
  • tested with TARGet, results matched Example Repositories section
  • tested with POT, results matched Example Repositories section
• Test using a sample sheet
  • tested with example-input.csv, toolkit was successfully run for both TARGet and POT through this method
• Test the consistency and repeatability of the AI-generated reports
  • Overall, the AI-generated report contents conveyed the same results when comparing two independently generated reports for the same tool
  • Report information seemed aligned with the metrics/information contained in the other results files
  • Reports seem consistent between runs for POT, except the JOSS readiness section was less informative in the second report (initial report indicated the tool would likely be publishable as is; second report only stated that the repository meets most criteria)
  • Reports for TARGet differed between runs, where the second run included a response from the agent at the top of the page and a repository overview section that was missing from both the initial TARGet report and the POT reports
  • The report contents differ for TARGet and POT:
    • POT reports did not contain the individual JOSS review criteria scores
    • For both runs, the POT reports indicated that the repo score was out of 1.0, whereas TARGet reports indicated the repo score was out of 5
• Did you see any errors?
  • when trying to install dependencies into my conda environment with pip install -r requirements.txt, I received an error message that the requirements file was not present in the repo.
• How did you find the documentation?
  • The documentation was sufficient to install nextflow, set up Synapse integration, run single tool and sample sheet-based analysis
  • Recommendations:
    • provide instructions on accessing the toolkit (cloning the repo, downloading the docker image, etc.) in the README
    • have people generate their .synapseConfig file using the synapseclient CLI command synapse config (described in the docs here)
    • maybe note that the docker client must be running in the background in order for it to be used by nextflow
• Any other feedback?
  • In the Example Repositories section of the README, it isn't clear to me how the results files relate to the Expected Status entries. E.g., what file/piece of information in the output files allows you to say that TARGet "Fails dependency and test checks"? I think noting how the outputs are related to interpretation would simplify a user's review of the output files.
  • For the Synapse integration section, there is a note about uploading report contents and needing an appropriate folder ID. I was not required to provide this to create AI-generated reports and it's not clear to me how I would provide the folder ID for use in the workflow