[REVIEW]: Extrae.jl:

[editorialbot]

Submitting author: @mofeing (Sergio Sánchez Ramírez)
Repository: https://github.com/bsc-quantic/Extrae.jl
Branch with paper.md (empty if default branch):
Version: v0.2.0
Editor: @luraess
Reviewers: @carstenbauer, @JBlaschke
Archive: Pending

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/fc8a16a70d639bc983ab00315b47a01c"><img src="https://proceedings.juliacon.org/papers/fc8a16a70d639bc983ab00315b47a01c/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/fc8a16a70d639bc983ab00315b47a01c/status.svg)](https://proceedings.juliacon.org/papers/fc8a16a70d639bc983ab00315b47a01c)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@carstenbauer & @JBlaschke, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @luraess know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @carstenbauer

📝 Checklist for @JBlaschke

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper source files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1137/141000671 is OK
- 10.1177/1094342010391989 is OK
- 10.1016/j.procs.2017.05.203 is OK
- 10.1109/SC.2002.10004 is OK
- 10.1145/3404397.3404440 is OK
- 10.1016/j.future.2024.07.050 is OK
- 10.21105/jcon.00077 is OK
- 10.1016/j.jcp.2021.110467 is OK
- 10.5281/zenodo.3996439 is OK
- 10.5334/jors.151 is OK
- 10.1016/j.softx.2015.10.004 is OK
- 10.1177/109434200001400303 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Paraver: A tool to visualize and analyze parallel ...
- No DOI given, and none found for title: Scalability of visualization and tracing tools
- No DOI given, and none found for title: Paraver: Reference Manual (Version 3.1)
- No DOI given, and none found for title: Extrae 4.2.3 documentation
- No DOI given, and none found for title: ScoreP.jl
- No DOI given, and none found for title: LIKWID.jl
- No DOI given, and none found for title: LinuxPerf.jl
- No DOI given, and none found for title: MPI.jl: Julia bindings for the Message Passing Int...
- No DOI given, and none found for title: IntelITT.jl
- No DOI given, and none found for title: MPITape.jl
- No DOI given, and none found for title: NVTX.jl
- No DOI given, and none found for title: Introduction to Paraver

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.98  T=0.02 s (1345.0 files/s, 183404.8 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
TeX                               8            313            197           2567
Julia                             8            115            114            532
YAML                              6              0              2            175
SVG                               1              0              0            157
XML                               1             18              1             87
Markdown                          1             28              0             46
Ruby                              1              8              4             45
Bourne Again Shell                1              8              2             26
TOML                              3              4              0             20
JSON                              1              0              0             17
Bourne Shell                      2              5              0              9
--------------------------------------------------------------------------------
SUM:                             33            499            320           3681
--------------------------------------------------------------------------------

Commit count by author:

   117	Sergio Sánchez Ramírez
     7	Marc Clascà
     2	Mosè Giordano

[editorialbot]

Paper file info:

🚨 Wordcount for paper.tex is 3091

🔴 Failed to discover a Statement of need section in paper

[editorialbot]

License info:

✅ License found: Apache License 2.0 (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[carstenbauer]

Review checklist for @carstenbauer

Conflict of interest

• I confirm that I have read the JuliaCon conflict of interest policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JCon for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/bsc-quantic/Extrae.jl?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@mofeing) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.pdf file include a list of authors with their affiliations?
• A statement of need: Does the paper clearly explain the problem it addresses, its importance to the broader community, the intended audience, and how it connects to existing work in the field?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

[carstenbauer]

Hi all, while I haven't completed all of my review works (for example, I haven't actually tested Extrae.jl yet), here comes my first batch of comments:

CONFLICT OF INTEREST

• @luraess, For clarity reasons, I should point out that I'm mentioned in the acknowledgements of the paper. However, I have not collaborated with the authors on Extrae.jl beyond talking/brainstorming about it.

GENERAL CHECKS

• Regarding "Contribution and authorship": Looking at the repo contributions, it's not obvious why @clasqui (Marc Clascà) isn't an author but @giordano (Mosè Giordano) is. Can you shortly comment on this?

DOCUMENTATION

• Statement of need is missing
• No installation instructions. Not for Extrae.jl and not for the parent tool (at least a link?).
• "Example usage": Would be great to have a few more examples (e.g. multithreading, MPI).
• "Automated tests": No CI setup.
• "Community guidelines": Not available.

PAPER

• See comments/notes in the attached PDF file: 10.21105.jcon.00180.pdf
• Expand statement of need: Why do we need Extrae, specifically? Why aren't the other, mentioned tools not sufficient?

OTHER COMMENTS

• Link to JuliaCon24 talk in the README.md?

[mofeing]

Regarding "Contribution and authorship": Looking at the repo contributions, it's not obvious why @clasqui (Marc Clascà) isn't an author but @giordano (Mosè Giordano) is. Can you shortly comment on this?

Sure. The development of Extrae.jl was carried out on different stages. I first started Extrae.jl to learn about Julia and BinaryBuilder, but I didn't really use it much.

A while later, @clasqui contacted me as he saw my package and was interested in using it for profiling an app that used Distributed.jl. He added a custom ClusterManager that did the same thing as the SSH Cluster Manager but emitted Extrae events. I think that it worked for him and we stopped developing it more.

Later, I did a major refactor of the code to keep it simpler: it dependend on Cassette.jl which has a lot of problems. Also, after talking with @jpsamaroo, I tried to make it compatible with new developments in Tracepoints.jl (currently known ProbeBase.jl) and DistributedNext.jl (which is adding a hooking system which will let us do what @clasqui was looking for). This included removing the custom ClusterManager.

So the code added by @clasqui, excluding bugfixes, didn't make it to the final version presented at JuliaCon but I'm happy to add him as co-author if reviewers see it appropiate.

The development of Extrae.jl was resumed when @giordano came as a visitor to my research group. We were both looking to make Extrae work on A64FX to profile code on Fujitsu machines. His work was mostly performed on the background, as he got in charge of fixing the Yggdrasil build for Extrae_jll. It was a time-consuming task because:

1. Extrae's build system had a couple of bugs preventing cross-compilation from working
2. It was the first Yggdrasil recipe that had both CUDA and MPI platform augmentations

No installation instructions. Not for Extrae.jl and not for the parent tool (at least a link?).
"Example usage": Would be great to have a few more examples (e.g. multithreading, MPI).
Expand statement of need: Why do we need Extrae, specifically? Why aren't the other, mentioned tools not sufficient?
Link to JuliaCon24 talk in the README.md?

I'm leaving here a list of tasks to do:

• Add examples
  • multithread
  • MPI
• Update README.md with JuliaCon24 talk and installation instructions
• Add statement of need
• Apply proposed changes and corrections