Paper: Advancing High Energy Physics Data Analysis with Julia: A Case for JuliaHEP

papers/ianna_osborne/main.md

@@ -0,0 +1,58 @@
+---
+title: Advancing High Energy Physics Data Analysis with Julia -- A Case for JuliaHEP
+abstract: |
+  For the past 25 years, the High-Energy Physics (HEP) community has steadily adopted Python as its primary language for data analysis, supported by compiled-backend libraries like NumPy [@numpy] and [Awkward Array](https://github.com/scikit-hep/awkward), along with other Python tools like [Uproot](https://github.com/scikit-hep/uproot5) for I/O. However, the Julia programming language offers a compelling alternative, addressing the two-language problem with C++-comparable performance and Python-like ease of use. [JuliaHEP](https://github.com/JuliaHEP/) is an informal organization that aims to unify effort in developing Julia projects related to HEP, outlining its advantages, ongoing developments, and integration with existing Python-based tools.
+---
+
+## Introduction
+
+The High-Energy Physics (HEP) community has long relied on Python and C++ for data analysis. While Python provides ease of use and a rich scientific ecosystem, it struggles with performance for large-scale analyses. C++, on the other hand, offers speed but comes with increased complexity and slower development cycles. Julia, a modern language designed for scientific computing, promises the best of both worlds: high-level expressiveness with near-native execution speed.
+
+:::{figure} heplanguages.png
+:label: fig:heplanguages
+HEP Software: it all revolves around a language, or several.
+:::
+
+JuliaHEP, with its emerging set of tools and libraries [@juliahep], aims to facilitate HEP data analysis in Julia. However, given the long timescales of HEP experiments and the substantial legacy software they depend on, introducing Julia alongside existing C++ and Python ecosystems inevitably creates a three-language problem — though this is intended as a transitional, rather than permanent, arrangement.
+
+To navigate this, three main development strategies have emerged within the JuliaHEP community. The first involves wrapping mature C++ libraries — for example, Geant4.jl provides a Julia interface to the widely used Geant4 particle transportation toolkit. The second approach opts for a complete reimplementation of existing tools in native Julia, as demonstrated by BAT.jl. The third, and the one we follow, focuses on integrating Julia with the Python-based Awkward Array library via its dedicated Julia backend, AwkwardArray.jl.
+
+This backend addresses one of the key challenges in multi-language environments:  data sharing across language boundaries. By exposing the underlying data buffers directly to Julia, AwkwardArray.jl avoids data copying and offers a gentle, incremental path for physicists to explore Julia’s capabilities within their existing Python workflows.
+
+
+## Awkward Array in Julia
+
+Awkward Array is a key library for handling complex and jagged HEP data in the Python ecosystem. To access this data from Julia, it is integrated via a dedicated backend, AwkwardArray.jl. What sets this backend apart is its tight integration with Julia’s type system.
+
+To bridge the Python and Julia ecosystems, we make extensive use of PythonCall.jl[@PythonCall.jl], a package that provides bi-directional interoperability between the two languages. This enables us to offload computationally heavy operations to Julia — following a pattern familiar from other JIT compilation tools in Python, such as Numba.
+
+We evaluated this integration by comparing performance and usability using Julia-native types like Vector of Vectors, which provide similar capabilities for representing nested, irregular data. Our results show no significant overhead when combining the two ecosystems. Additionally, we explored Julia’s interoperability and native packages to further optimize HEP computations, offering a practical pathway for gradually introducing Julia to the Python-based HEP community.
+
+## Performance Benefits
+
+We present the benefits of using Julia for HEP workloads by benchmarking the same operation with JuliaHEP tools and comparing the results to Python and C++ implementations. As shown in @fig:combinations, Julia offers a noticeable speedup with less boilerplate.
+
+:::{figure} combinations.png
+:label: fig:combinations
+This is the combinations performance plot, comparing the ak.combinations function with a Julia kernel on an awkward array. The Julia kernel uses JuliaMath/Combinatirics.jl[@Combinatorics.jl].
+:::
+
+ This is without exploiting Julia’s potential for parallel computing and built-in GPU support, which provides a pathway for scaling HEP computations.
+
+## Scaling it up
+
+While performance gains matter, integrating Julia into established HEP workflows also involves practical challenges. AwkwardArray.jl enables physicists to explore Julia’s capabilities alongside existing Python tools without disrupting current workflows. It acts as a transitional layer, allowing heavy computations to be offloaded to Julia when useful, while preserving compatibility with Python’s data analysis ecosystem.
+
+Naturally, adding a third language raises a long-term question. Will Julia replace Python and C++? It’s too soon to say. Legacy software must be maintained, Python remains dominant, and new options like Rust are emerging. For now, the goal is to provide a flexible, incremental path for physicists to use Julia where it offers clear advantages.
+
+The main technical hurdle is efficient data interoperability — a challenge addressed by the Awkward Array design, which uses contiguous data buffers and descriptive metadata. This model has also proven effective in related optimizations, such as its virtual array implementation.
+
+Data access remains central. ROOT TTrees are still the dominant format, with RNTuple expected to grow in use. Both formats are accessible as Awkward Arrays via uproot (Python) and unroot (Julia), ensuring consistency across languages.
+
+Managing a shared Python–Julia environment remains challenging. Conda for Python, juliaup, and Julia’s package manager integrate well, though keeping package versions aligned requires care. A coordination tool like pixi is currently under evaluation to simplify this.
+
+Overall, this approach offers a pragmatic, low-friction path for gradually introducing Julia into HEP workflows — letting physicists benefit from its strengths in JIT compilation, multiple dispatch, and GPU acceleration without leaving established ecosystems behind.
+
+## Acknowledgements
+
+This work was supported by the National Science Foundation under Cooperative Agreement PHY-2323298.

papers/ianna_osborne/mybib.bib

@@ -0,0 +1,124 @@
+# These references may be helpful:
+
+@inproceedings{jupyter,
+  abstract  = {It is increasingly necessary for researchers in all fields to write computer code, and in order to reproduce research results, it is important that this code is published. We present Jupyter notebooks, a document format for publishing code, results and explanations in a form that is both readable and executable. We discuss various tools and use cases for notebook documents.},
+  author    = {Kluyver, Thomas and Ragan-Kelley, Benjamin and Pérez, Fernando and Granger, Brian and Bussonnier, Matthias and Frederic, Jonathan and Kelley, Kyle and Hamrick, Jessica and Grout, Jason and Corlay, Sylvain and Ivanov, Paul and Avila, Damián and Abdalla, Safia and Willing, Carol and {Jupyter development team}},
+  editor    = {Loizides, Fernando and Scmidt, Birgit},
+  location  = {Netherlands},
+  publisher = {IOS Press},
+  url       = {https://eprints.soton.ac.uk/403913/},
+  booktitle = {Positioning and Power in Academic Publishing: Players, Agents and Agendas},
+  year      = {2016},
+  pages     = {87--90},
+  title     = {Jupyter Notebooks - a publishing format for reproducible computational workflows},
+}
+
+@article{matplotlib,
+  abstract     = {Matplotlib is a 2D graphics package used for Python for application development, interactive scripting, and publication-quality image generation across user interfaces and operating systems.},
+  author       = {Hunter, J. D.},
+  publisher    = {IEEE COMPUTER SOC},
+  year         = {2007},
+	doi          = {https://doi.org/10.1109/MCSE.2007.55},
+  journal      = {Computing in Science \& Engineering},
+  number       = {3},
+  pages        = {90--95},
+  title        = {Matplotlib: A 2D graphics environment},
+  volume       = {9},
+}
+
+@article{numpy,
+  author       = {Harris, Charles R. and Millman, K. Jarrod and van der Walt, Stéfan J. and Gommers, Ralf and Virtanen, Pauli and Cournapeau, David and Wieser, Eric and Taylor, Julian and Berg, Sebastian and Smith, Nathaniel J. and Kern, Robert and Picus, Matti and Hoyer, Stephan and van Kerkwijk, Marten H. and Brett, Matthew and Haldane, Allan and del Río, Jaime Fernández and Wiebe, Mark and Peterson, Pearu and Gérard-Marchant, Pierre and Sheppard, Kevin and Reddy, Tyler and Weckesser, Warren and Abbasi, Hameer and Gohlke, Christoph and Oliphant, Travis E.},
+  publisher    = {Springer Science and Business Media {LLC}},
+  doi          = {https://doi.org/10.1038/s41586-020-2649-2},
+  date         = {2020-09},
+  year         = {2020},
+  journal      = {Nature},
+  number       = {7825},
+  pages        = {357--362},
+  title        = {Array programming with {NumPy}},
+  volume       = {585},
+}
+
+@misc{pandas1,
+  author    = {{The Pandas Development Team}},
+  title     = {pandas-dev/pandas: Pandas},
+  month     = feb,
+  year      = {2020},
+  publisher = {Zenodo},
+  version   = {latest},
+  url       = {https://doi.org/10.5281/zenodo.3509134},
+}
+
+@inproceedings{pandas2,
+  author    = {Wes McKinney},
+  title     = {{D}ata {S}tructures for {S}tatistical {C}omputing in {P}ython},
+  booktitle = {{P}roceedings of the 9th {P}ython in {S}cience {C}onference},
+  pages     = {56 - 61},
+  year      = {2010},
+  editor    = {{S}t\'efan van der {W}alt and {J}arrod {M}illman},
+	doi       = {https://doi.org/10.25080/Majora-92bf1922-00a},
+}
+
+@article{scipy,
+  author  = {Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and
+            Haberland, Matt and Reddy, Tyler and Cournapeau, David and
+            Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and
+            Bright, Jonathan and {van der Walt}, St{\'e}fan J. and
+            Brett, Matthew and Wilson, Joshua and Millman, K. Jarrod and
+            Mayorov, Nikolay and Nelson, Andrew R. J. and Jones, Eric and
+            Kern, Robert and Larson, Eric and Carey, C J and
+            Polat, {\.I}lhan and Feng, Yu and Moore, Eric W. and
+            {VanderPlas}, Jake and Laxalde, Denis and Perktold, Josef and
+            Cimrman, Robert and Henriksen, Ian and Quintero, E. A. and
+            Harris, Charles R. and Archibald, Anne M. and
+            Ribeiro, Ant{\^o}nio H. and Pedregosa, Fabian and
+            {van Mulbregt}, Paul and {SciPy 1.0 Contributors}},
+  title   = {{{SciPy} 1.0: Fundamental Algorithms for Scientific
+            Computing in Python}},
+  journal = {Nature Methods},
+  year    = {2020},
+  volume  = {17},
+  pages   = {261--272},
+  adsurl  = {https://rdcu.be/b08Wh},
+	doi     = {https://doi.org/10.1038/s41592-019-0686-2},
+}
+
+@article{sklearn1,
+  author       = {Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  year         = {2011},
+  journal      = {Journal of Machine Learning Research},
+  pages        = {2825--2830},
+  title        = {Scikit-learn: Machine Learning in {P}ython},
+  volume       = {12},
+}
+
+@inproceedings{sklearn2,
+  author    = {Buitinck, Lars and Louppe, Gilles and Blondel, Mathieu and Pedregosa, Fabian and Mueller, Andreas and Grisel, Olivier and Niculae, Vlad and Prettenhofer, Peter and Gramfort, Alexandre and Grobler, Jaques and Layton, Robert and VanderPlas, Jake and Joly, Arnaud and Holt, Brian and Varoquaux, Gaël},
+  booktitle = {ECML PKDD Workshop: Languages for Data Mining and Machine Learning},
+  year      = {2013},
+  pages     = {108--122},
+  title     = {{API} design for machine learning software: experiences from the scikit-learn project},
+}
+
+@article{juliahep,
+  author  = {Graeme Andrew Stewart, Alexander Moreno Briceño, Philippe Gras, Benedikt Hegner, Uwe Hernandez Acosta, Tamas Gal, Jerry Ling, Pere Mato, Mikhail Mikhasenko, Oliver Schulz, Sam Skipsey},
+  title   = {Julia in HEP},
+  year    = {2025},
+  url     = {https://arxiv.org/abs/2503.08184},
+  doi     = {https://doi.org/10.48550/arXiv.2503.08184},
+}
+
+@misc{PythonCall.jl,
+  author = {Rowley, Christopher},
+  title  = {PythonCall.jl: Python and Julia in harmony},
+  year   = {2022},
+  url    = {https://github.com/JuliaPy/PythonCall.jl},
+}
+
+@misc{Combinatorics.jl,
+  title = {Combinatorics.jl: A combinatorics library for Julia},
+  url   = {https://github.com/JuliaMath/Combinatorics.jl},
+}
+
+@misc{Geant4.jl,
+Julia bindings to the Geant4 simulation toolkit 

papers/ianna_osborne/myst.yml

@@ -0,0 +1,40 @@
+version: 1
+extends: ../papers.yml
+project:
+  id: scipy-2025-ianna_osborne
+  title: Advancing High Energy Physics Data Analysis with Julia -- A Case for JuliaHEP
+  description: The computational challenges in HEP require optimized solutions for 
+    handling complex data structures, parallel computing, and Just-in-Time (JIT) compilation. 
+    While Python and C++ remain the standard, Julia presents an opportunity to improve 
+    performance while maintaining usability.
+  authors:
+    - name: Ianna Osborne
+      email: ianna.osborne@cern.ch
+      orcid: 0000-0002-6955-1033
+      affiliations:
+        - Princeton University
+      corresponding: true
+  keywords:
+    - Awkward Array
+    - Julia
+    - High-Energy Physics
+    - Data Analysis
+  # Add the abbreviations that you use in your paper here
+  abbreviations:
+    JIT: Just-in-Time compilation
+  # It is possible to explicitly ignore the `doi-exists` check for certain citation keys
+  error_rules:
+    - rule: doi-exists
+      severity: ignore
+      keys:
+        - Atr03
+        - terradesert
+        - jupyter
+        - sklearn1
+        - sklearn2
+  exports:
+    - id: pdf
+      format: typst
+      template: https://github.com/curvenote-templates/scipy.git
+      article: main.md
+      output: full_text.pdf