Software needed to be developed.

[DavidSagan]

Here is my idea for what software will be needed with the lattice standard. The boxes in the figure are:

• Simulation program: A simulation program.
• Interface layer: Code to interface between the structures defined by the program and the lattice structure defined by the Translator. The program maintainer(s) must create the interface layer since the code will be program specific. However, structure translation should (hopefully) be relatively straight forward.
• Translator: Package supplied with the lattice standard. The Translator has code to put lattice information into a "basic form". This includes expanding beam lines, evaluating expressions, etc.
• Reader / Writer: Package to read/write files of a standard format (JSON, YAML, etc). Packages to do this are widely available.
• Lattice File: A lattice file in a standard format such as JSON, YAML, etc.

Note: The Translator could be interfaced to multiple Reader/Writers so that multiple languages can be handled at once. The structure that the Translator passes to the Interface layer will be language independent so that the interface layer does not
have to deal with what language is being used directly (other than telling the Translator what language to use).

[DavidSagan]

[ax3l]

That structure makes sense to me.

For the reader / writer level, I would suggest we use something that can autocratically serialize/deserialize and validate dictionaries/lists/structs, e.g., pydantic:
https://github.com/ax3l/pydantic-mad (playground/demo project)

The translator on top of the reader / writer ([de]serializer) is probably one package.

[ax3l]

@ChristopherMayes commented:

We use Pydantic extensively in LUME project. For example, Genesis4 lattice elements: https://github.com/slaclab/lume-genesis/blob/master/genesis/version4/input/_lattice.py

[ax3l]

XSuite uses a package called Xdeps underneath their lattice parsing: https://github.com/xsuite/Xdeps

[rdemaria]

XSuite uses a package called Xdeps underneath their lattice parsing: https://github.com/xsuite/Xdeps

Xdeps implements several functionalities. The main is one to allow controlling element attributes by changing variables, inspecting and manipulating these relations and code generation. It is completely general and can control data in any valid Python container without modifications. Xdeps contains code to translate MAD-X expressions into xdeps framework and other MAD-X syntax parsing functions. As well, optimizers and tables used in xtrack and other components of xsuite.

[DavidSagan]

Needed will be translators for C, C++, Fortran, Python, and Julia. Maintenance is an issue and a possible headache. One thought is to write a C based package to handle the bookkeeping and then interface that to the other languages.

[ax3l]

An alternative is to expose C bindings and write in another language (e.g., C++).

But my feeling is we should first write a quick version in a scripting language like Python, especially since we want to exploit again the capabilities of pydantic (as done in LUME, xopt, etc.).

A quick implementation in any other language could be: use the scripts to prepare and validate files, then in the language of choice just read the format that is supported (e.g., there are many YAML/JSON/TOML readers available to maintainers). Not every language needs a validator for instance, that part can be done in a single, scripted implementation (as we do in openPMD-validator).

[ldeniau]

Nice to see that you converge to my proposal made during the first meeting (30.11.2024) :-)

I fully agree that a tool/library should be provided with the standard, written in C (at least exposed through a C API), in order to spread its use.

The parts I was proposing:

• Database / Object model: allow to store and retrieve information efficiently (sequences, elements, etc), evaluate expressions, handle hierarchy of classes (a-la-MAD-X).
• Parsers / Translators: read and write various formats to feed the Database.
• Validator: ensure consistency of attributes in elements, elements in sequences, sequence constructions, compliance with the standard, etc...
  External tools using this library would only need a C API to use components of these parts and build their own representation. If the database / object model is fast enough for queries, it may save the tool from building its own representation.

[DavidSagan]

There must have been some miscommunication here. This scheme was my thought all along and basically mirrors what was done for Accelerator Markup Language (AML).

[rdemaria]

The strategy we adopted in Xsuite was to have a basic object model in C (relatively simple with limited dynamic features, movable in memory, https://github.com/xsuite/xobjects) and leave the expressions for the host language/framework (in our case Python) to seamlessly meld with the goodies of the host ecosystem.

[DavidSagan]

This works well when using Python or Julia but unfortunately does not work well for programs based on C, C++, Fortran.

[DavidSagan]

For maximum flexibility, the program should be able to set how the translator does the translation. EG whether to evaluate expressions or not.

[mattsignorelli]

Moving #85 here:

---

I would like to discuss a bit more seriously the choice of using language-specific PALS implementations (e.g. PALS-python, future PALS-julia, etc) as opposed to a single shared C library.

In my view, there should be one single shared C library which contains all of the PALS routines, and then light wrappers of this C library in the various languages. C libraries are easily callable from most programming languages, making it the most widely usable immediately -- we wouldn't have to wait for each individual implementation to be completed. And when changes, modifications to the standard, or bug fixes need to be implemented, they would only need to be made to the C library, as opposed to each language specific implementation.

This really would ease long term maintenance, and make features immediately available to all users

---

Seems like @ldeniau already proposed basically this some time ago, if I understand correctly

[DavidSagan]

there is very little that this C API can provide that is more than what a target language's reference YAML/JSON/... reader provides.

Actually there is a lot that the C/C++ code can provide that is not in the reader. Stuff like lattice expansion, expression evalutation, global position calculations, searching for elements, reference time and energy calculation, etc. I think it is very important that we don't force people to keep reinventing the wheel to do stuff like this.

Language bindings to high-level languages from C: I have done this a lot, e.g., in C/C++ to Fortran, openPMD-api and elsewhere. In my experience, automatically generating language bindings on C APIs for Python/Fortran/Julia/... usually shows. There is more work needed to let an API feel and be productive (e.g., be "pythonic") than exposing C function calls 1:1.

I agree that it is probably better to use C++ with extern "C" linkage.

All in all, I think this is technically feasible, but I fear not the most productive thing to do, because so many language-specific standardlib implementations for reading the YAML/JSON/XML/TOML/... in concrete formats that PALS files are usually stored and exchanged in already exist and I see no urgent use case just yet why fast-to-write Julia and Python reference implementations would not address the important use cases that we need to knock out first.

The issue here is not reading/writing. The issue is the manipulations that need to be done to get the lattice as defined in the PALS lattice file into a form that is usable by a simulation program. To maximize the usefulness of PALS, we should strive to provide modules that do the common manipulations (lattice expansion, etc.) to minimize reinventing the wheel.

In the case of writing in Julia/Python, we only need one person that only has competence in one language, each.

And the problem with this is that if there are additions and/or changes to the standard, changes will have to be made in the Julia package, and the Python package, and any other package that is developed for C++, etc. And we will have to worry that the different implementations are consistent with each other. I know that developing the needed functionality in C++ is not great but my gut feeling is that it is better than trying to maintain multiple packages doing exactly the same thing.

[ax3l]

Actually there is a lot that the C/C++ code can provide that is not in the reader. Stuff like lattice expansion, expression evalutation, global position calculations, searching for elements, reference time and energy calculation, etc. I think it is very important that we don't force people to keep reinventing the wheel to do stuff like this.

I agree that there is some routines that are common and this list is very helpful to estimate.
Right now, my guesstimate is this manipulation and expansion logic can be implemented in less than 2k lines of Python and Julia, respectively. Compensated sums for global coordinates are in all these languages, too (or can be added in ~20lines).

It is hard for me to imagine a sensible/productive C API for all this book-keeping calc/logic, too (pointers everywhere). In the end, we will use this anyway from scripting layers, is what I suspect.

The issue is the manipulations that need to be done to get the lattice as defined in the PALS lattice file into a form that is usable by a simulation program.

Yes, I think that this discussion overestimates what part of this can be shared in the Translator box and how much is actually in the code that implements PALS. My estimate is the listed manipulations are less than 2k lines of code and we write it quickly in Julia and Python instead of suffering in C++. (And I write this as C++ developer.)

And the problem with this is that if there are additions and/or changes to the standard, changes will have to be made in the Julia package, and the Python package, and any other package that is developed for C++, etc.

I understand that concern, but think that we will nonetheless move way faster with both a Julia and Python implementation, at least for now, due to the other reasons (like lowering all new features to a C API is work, too, and then raising again to something that feels natural in Julia/Python).

I am not sure that we have yet a urgent need for a C++ package, e.g., neither BLAST nor XSuite should need it, because they will combine book-keeping logic on the scripting level.

[ax3l]

One can even see the needed two, independent implementations (Julia & Python) as a feature: it ensures that what we define in PALS is actually real-world interpretable and works in two independent implementations.

(That is for instance, how the ISO C++ committee operates. Before they standardize, they want to see a demonstration in two compilers/stdlibs.)

[mattsignorelli]

Thanks for all of the detailed replies.

While I agree that it would be nice to have all these features like global coordinate calculations, expansion, etc. etc., really what I need for SciBmad most urgently is a reader/writer. We already have our own lattice definition package with its own polymorphic and highly optimized data structures, so things like manipulating lattices and computing floor coordinates and the likes aren't needed from PALS by SciBmad. What is needed however is a way to write our data structures into a PALS format .yaml/.json/etc, and likewise read a PALS lattice file into our data structures. Everything after that is handled by our own code already. In my view, most codes also probably have a way of manipulating a lattice already, and so the first priority of PALS should be just reading/writing lattice .yaml/.json files, not actually manipulating them. Manipulations, and calculations of dependent variables (e.g. floor coordinates in some cases), while nice, I feel should be lower on the priority list. Furthermore, with manipulations, there is a lot more discussion to be had here, because then it raises the question of whether we can compute dependent variables lazily (a la MAD-NG and now SciBmad) which is much more performant in almost all cases, or eagerly (a la current Bmad). Then of course things like polymorphism of these calculations need to be discussed.

In the meeting yesterday it seemed I could either 1) have SciBmad read/write directly to these .json/.yaml formats, or 2) first translate my data structures to PALS's data structures, and then let PALS handle the rest. For the reasons we discussed (validation being a big one), 2 is the easier choice.

Therefore, if there were some data structures in C that I could just translate my Julia structures to, and then call the read/writing routines, we would be immediately in business. As someone who's coded a lot in Julia, and also wrapped a C library in Julia, my personal view is that it is a lot easier to wrap an already finished C library and then add Julianic ways of doing things with those data structures on top of that, than to start from scratch in Julia. But of course, this requires that someone actually already wrote the C library.

In terms of what I could personally contribute, I definitely do not have the time to write a Julia implementation of PALS and stay up-to-date with the latest changes / bug fixes in the standard. But I could certainly wrap a C library in Julia and provide basic features immediately in a Julia package, and then just update the C library whenever new versions are released.

[ax3l]

Thanks! Yes agree with the priorities.

I think for you the quickest way to go would be use a Julia yaml reader (for concrete/expanded PALS files) and translate in a loop the objects to your internal SciBmad representation, which is something that needs to be done in every code, as it is code specific.

Personally on the BLAST side, we have not enough time and no need for a C library, so we will focus on the Python implementation to move fast in the coming year. But if there are volunteers in the community to write a C API, and you have time to prepare a good spec of what exact C APIs you would need, people can def chime in.

[ldeniau]

I agree with matt (who agreed with my first remark ;-)

MAD-NG is already interfacing script level with C and C++ at large scale. Getting a clean pure C API is not difficult as soon as you follow several important principles described in slides 19-21 of https://github.com/CObjectSystem/COS/blob/master/doc/slides-design.pdf. Making such an API efficient is another story if the interface is targeting a low level of interaction, but sometimes it is needed and this is where technologies come to the rescue as explained for MAD-NG in https://indico.cern.ch/event/487416/contributions/2174904/attachments/1281079/1903600/ld_1605-mad_dev_slides.pdf

src % cloc --force-lang=Lua,mad *.mad *.lua *.[hc] *.[hc]pp 
     122 text files.
     122 unique files.                                          
       0 files ignored.

github.com/AlDanial/cloc v 2.06  T=0.51 s (239.4 files/s, 147613.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Lua                             66           7000           5248          39964
C                               30           2171           1765          12034
C/C++ Header                    25            794            826           2783
C++                              1            392            154           2090
-------------------------------------------------------------------------------
SUM:                           122          10357           7993          56871
-------------------------------------------------------------------------------