Update documentation

CHANGELOG.md

@@ -1,3 +1,10 @@
+0.7.0 (2024/12/05)
+
+Rewrite documentation
+Update to MAD-NG 1.1.1
+Handle opening and closing of MAD-NG process more robustly
+
+
 0.6.0 (2024/12/05)
 Remove `debug` input variable functionality, now it is only a boolean, and dictates whether the debug information is printed to the console. \
 Add `stdout` input to the `MAD` object, this allows the user to redirect the output of the MAD-NG process to a file. \
@@ -53,7 +60,7 @@ Set pymadng to now be in beta.
 Fix MADX issue
 Move binaries to the bin folder \
 Update MAD-NG binaries \
-Rename files to start with madp_... \
+Rename files to start with madp\_... \
 Completely refactor underlying process and remove reliance on mad objects, mad strings and `__last__`, now the process is completely self contained and can be separated into MAD-NG itself.
 
 List of changes to PyMAD-NG
@@ -70,4 +77,4 @@ Update MAD-NG binaries
 
 Update MAD-NG binaries \
 Fix bug with negative integer values \
-Initialise CHANGELOG 
+Initialise CHANGELOG

README.md

@@ -1,30 +1,100 @@
-# pymadng
-Python interface to MAD-NG running as subprocess
+# PyMAD-NG
 
-Install using below, see [The Python Package Index (PyPi)](https://pypi.org/project/pymadng/);
+**Python interface to MAD-NG running as a subprocess**
 
-`pip install pymadng`
+[![PyPI version](https://img.shields.io/pypi/v/pymadng.svg)](https://pypi.org/project/pymadng/)
+[![Documentation Status](https://readthedocs.org/projects/pymadng/badge/?version=latest)](https://pymadng.readthedocs.io/en/latest/)
+[![License](https://img.shields.io/github/license/MethodicalAcceleratorDesign/MADpy)](https://github.com/MethodicalAcceleratorDesign/MADpy/blob/main/LICENSE)
 
-Familiarising yourself with pymadng
-===================================
+---
 
-First, we recommend familiarising yourself with MAD-NG, documentation can be found [here](https://madx.web.cern.ch/releases/madng/html/). 
+## 🚀 Installation
 
-Then reading through the Low-Level Example Explained on the [pymadng documentation](https://pymadng.readthedocs.io/en/latest/) should be sufficient (alongside knowledge of MAD-NG), assuming you are not planning to use any "syntactic sugar". If you plan to use the available pythonic looking code, there are plenty of examples to look at. 
+Install via pip from [PyPI](https://pypi.org/project/pymadng/):
 
-In the documentation, [FODO Examples Explained](https://pymadng.readthedocs.io/en/latest/ex-fodo.html), is a chapter that goes into detail on what is happening on each line of the [FODO example](https://github.com/MethodicalAcceleratorDesign/MADpy/blob/main/examples/ex-fodo/ex-fodos.py), while [LHC Example](https://pymadng.readthedocs.io/en/latest/ex-lhc-couplingLocal.html) gives an example of loading the LHC and how to grab intermediate results from a match. 
+```bash
+pip install pymadng
+```
 
-The only other example that may be of use is the [ps-twiss](https://github.com/MethodicalAcceleratorDesign/MADpy/blob/main/examples/ex-ps-twiss/ps-twiss.py) example. This is an extremely simple example, extending the FODO example to perform a twiss on the PS sequence.
-If anything does not seem fully explained, initially check the [API Reference](https://pymadng.readthedocs.io/en/latest/pymadng.html#module-pymadng) and/or the [MAD-NG Documentation](https://mad.web.cern.ch/mad/releases/madng/html/), then feel free to open an [issue](https://github.com/MethodicalAcceleratorDesign/MADpy/issues) so improvements can be made.
+---
 
-Documentation
-=============
+## 🧠 Getting Started
 
-Documentation, including explanation of a couple of examples and the limitations of the API can be found [here](https://pymadng.readthedocs.io/en/latest/). 
+Before diving into PyMAD-NG, we recommend you:
 
-The API reference is also included in [this documentation](https://pymadng.readthedocs.io/en/latest/). You can also compile to documentation yourself by cloning the repository and running ``make html`` in the docs folder.
+1. Familiarise yourself with [MAD-NG](https://madx.web.cern.ch/releases/madng/html/) — understanding MAD-NG is essential.
+2. Read the [Quick Start Guide](https://pymadng.readthedocs.io/en/latest/) to see how to control MAD-NG from Python.
 
-Getting the examples working
-============================
+### Explore Key Examples
 
-You can run the example with `python3 EXAMPLE_NAME.py`
+- **[Low-Level Example Explained](https://pymadng.readthedocs.io/en/latest/)** – Learn the fundamentals line-by-line.
+- **[FODO Example Breakdown](https://pymadng.readthedocs.io/en/latest/ex-fodo.html)** – Annotated walkthrough of a FODO cell simulation.
+- **[LHC Matching Example](https://pymadng.readthedocs.io/en/latest/ex-lhc-couplingLocal.html)** – Real-world optics matching with intermediate feedback.
+- **[PS Twiss Example](https://github.com/MethodicalAcceleratorDesign/MADpy/blob/main/examples/ex-ps-twiss/ps-twiss.py)** – Minimal example applying `twiss()` to the Proton Synchrotron.
+
+If anything seems unclear:
+- Refer to the [API Reference](https://pymadng.readthedocs.io/en/latest/pymadng.html#module-pymadng)
+- Check the [MAD-NG Docs](https://madx.web.cern.ch/releases/madng/html/)
+- Or open an [issue](https://github.com/MethodicalAcceleratorDesign/MADpy/issues)
+
+---
+
+## 📚 Documentation
+
+Full documentation and example breakdowns are hosted at:
+[https://pymadng.readthedocs.io/en/latest/](https://pymadng.readthedocs.io/en/latest/)
+
+To build locally:
+
+```bash
+git clone https://github.com/MethodicalAcceleratorDesign/MADpy.git
+cd MADpy/docs
+make html
+```
+
+---
+
+## 🧪 Running Examples
+
+Examples are stored in the `examples/` folder.
+Run any script with:
+
+```bash
+python3 examples/ex-fodos.py
+```
+
+You can also batch-run everything using:
+
+```bash
+python3 runall.py
+```
+
+---
+
+## 💡 Features
+
+- High-level Python interface to MAD-NG
+- Access to MAD-NG functions, sequences, optics, and tracking
+- Dynamic `send()` and `recv()` communication
+- Python-native handling of MAD tables and expressions
+- Optional integration with `pandas` and `tfs-pandas`
+
+---
+
+## 🤝 Contributing
+
+We welcome contributions! See [`CONTRIBUTING.md`](docs/source/contributing.md) or the [Contributing Guide](https://pymadng.readthedocs.io/en/latest/contributing.html) in the docs.
+
+Bug reports, feature requests, and pull requests are encouraged.
+
+---
+
+## 📜 License
+
+PyMAD-NG is licensed under the [MIT License](https://github.com/MethodicalAcceleratorDesign/MADpy/blob/main/LICENSE).
+
+---
+
+## 🙌 Acknowledgements
+
+Built on top of MAD-NG, developed at CERN. This interface aims to bring MAD's power to the Python ecosystem with minimal friction.

docs/source/advanced_features.md

@@ -0,0 +1,204 @@
+```{eval-rst}
+.. currentmodule:: pymadng
+```
+
+# Advanced Features in PyMAD-NG
+
+This section covers some of the most powerful capabilities in PyMAD-NG. These features allow you to create scalable and complex accelerator workflows by combining the performance of MAD-NG with Python's expressiveness.
+
+```{contents}
+:depth: 1
+:local:
+```
+
+---
+
+## Understanding `_last[]` Temporary Variables
+
+In MAD-NG, when a command returns a value, it is not automatically captured unless explicitly assigned. PyMAD-NG handles this by assigning results to a set of reserved variables: `_last[1]`, `_last[2]`, etc.
+
+These are managed internally by PyMAD-NG using a helper class {class}`madp_last.last_counter`, and accessed in Python via references. This allows expressions like:
+
+```python
+result = mad.math.sqrt(2) + mad.math.log(10)
+```
+
+Behind the scenes, each intermediate operation is stored in a new `_last[i]` reference, then combined. You can access or evaluate the result using `.eval()`:
+
+```python
+print(result.eval())
+```
+
+These temporary variables are recycled unless manually stored using:
+
+```python
+mad["my_var"] = result
+```
+
+This is particularly useful in expressions, multi-step computations, and avoiding naming clutter.
+
+---
+
+## Function and Object References in MAD-NG
+
+In PyMAD-NG, accessing or calling any MAD-NG function or object returns a Python reference to that MAD-NG entity, rather than immediately executing or resolving it. This enables symbolic chaining and precise control over execution.
+
+### Example:
+```python
+r = mad.math.exp(1)
+print(type(r))  # high_level_mad_ref
+print(r.eval())  # 2.718...
+```
+
+You can delay evaluation until needed, allowing reuse:
+```python
+mad["result"] = mad.math.log(10) + mad.math.sin(1)
+```
+
+This keeps Python responsive and lets MAD-NG do the heavy lifting.
+
+---
+
+## Real-Time Feedback with Python During Matching
+
+MAD-NG supports callbacks and iterative evaluations, which can be tied into Python logic. One common use is during `match` procedures, where you want to receive intermediate updates.
+
+### Example Workflow:
+In MAD:
+```lua
+function twiss_and_send()
+  local tbl, flow = twiss {sequence=seq, method=4}
+  py:send({tbl.s, tbl.beta11})
+  return tbl, flow
+end
+```
+
+In Python:
+```python
+mad.match(
+  command=mad.twiss_and_send,
+  variables=[...],
+  equalities=[...],
+  objective={"fmin": 1e-3},
+  maxcall=100
+)
+
+while True:
+    data = mad.recv()
+    if data is None:
+        break
+    update_plot(data)
+```
+
+This is ideal for live visualization, feedback loops, or diagnostics during optimization.
+
+---
+
+## Using PyMAD-NG with Multiprocessing
+
+Because PyMAD-NG communicates with MAD-NG via pipes (not shared memory), you can launch multiple independent MAD processes using `os.fork()` or `multiprocessing`.
+
+### When to Use This:
+- Run parallel simulations or parameter scans
+- Avoid reloading large sequences repeatedly
+
+### Example:
+```python
+import os
+if os.fork() == 0:
+    mad = MAD()
+    mad.send("... long running setup ...")
+    os._exit(0)
+```
+
+Each process maintains its own MAD instance and data pipeline.
+
+---
+
+## Loading and Using External MAD Files and Modules
+
+MAD-X and MAD-NG models often consist of `.seq`, `.mad`, `.madx`, or `.str` files. You can load these via the high-level interface:
+
+```python
+mad.MADX.load("'lhc.seq'", "'lhc.mad'")
+mad.load("MADX", "lhcb1")
+```
+
+Or load additional MAD-NG modules:
+```python
+mad.load("MAD.gphys", "melmcol")
+```
+
+This loads extended libraries for magnet properties, tracking models, or optics algorithms.
+
+---
+
+## Exporting Results for External Use
+
+After running a Twiss or Survey, the results are stored in an `mtable`, which can be exported to a TFS file:
+
+```python
+mad.tbl.write("'results.tfs'", mad.quote_strings(["s", "beta11", "mu1"]))
+```
+
+You can read this file with `tfs-pandas` or use it as input to another tool.
+
+---
+
+## Combining with NumPy and Pandas
+
+PyMAD-NG integrates cleanly with Python’s data ecosystem:
+
+- Pass `numpy` arrays to MAD-NG using {func}`MAD.send`
+- Use {func}`.to_df` on MAD tables to get Pandas DataFrames
+- Use `tfs-pandas` for rich metadata support
+
+### Example:
+```python
+import numpy as np
+mad.send("my_array = py:recv()")
+mad.send(np.linspace(0, 1, 100))
+```
+
+This allows direct use of scientific computation tools in tandem with accelerator modeling.
+
+---
+
+## Managing Larger Workflows
+
+PyMAD-NG supports:
+- Loading full files with `mad.loadfile("mysetup.mad")`
+- Organising expressions using Python variables
+- Retaining command history using:
+
+```python
+print(mad.history())
+```
+
+For clean resource management, always use context blocks:
+```python
+with MAD() as mad:
+    mad.MADX.load("'lhc.seq'", "'lhc.mad'")
+```
+
+This ensures the MAD process is correctly shut down when finished.
+
+---
+
+## Summary of Advanced Features
+
+| Feature                         | Purpose                                          |
+|---------------------------------|--------------------------------------------------|
+| `_last[]` Variables             | Track intermediate return values symbolically    |
+| Reference Objects               | Access MAD-NG objects with delayed evaluation    |
+| Matching Feedback               | Monitor intermediate results during match        |
+| Multiprocessing                 | Run multiple MAD-NG simulations in parallel      |
+| File and Module Loading         | Import sequences, optics files, and Lua modules  |
+| Table Export                    | Write TFS files from MAD tables                  |
+| NumPy / Pandas Interoperability  | Pass data between Python and MAD-NG seamlessly   |
+| Project Structuring             | Use {func}`MAD.loadfile`, {func}`MAD.history`, and `with` block  |
+
+These tools are designed to give you complete control over your simulations while staying fast and maintainable.
+
+Next: head over to **Debugging & Troubleshooting** to diagnose and resolve common issues in real-world workflows.
+