Add transfer of dicts.

.github/workflows/python-publish.yml

@@ -37,8 +37,8 @@ jobs:
     - name: Get MAD Binaries
       run: |
         mkdir src/pymadng/bin 
-        curl https://madx.web.cern.ch/releases/madng/1.1/mad-linux-1.1.2 -o src/pymadng/bin/mad_Linux
-        curl https://madx.web.cern.ch/releases/madng/1.1/mad-macos-1.1.2 -o src/pymadng/bin/mad_Darwin
+        curl https://madx.web.cern.ch/releases/madng/1.1/mad-linux-1.1.3 -o src/pymadng/bin/mad_Linux
+        curl https://madx.web.cern.ch/releases/madng/1.1/mad-macos-1.1.3 -o src/pymadng/bin/mad_Darwin
         chmod +x src/pymadng/bin/mad_Linux src/pymadng/bin/mad_Darwin
     - name: Build package
       run: python -m build

.github/workflows/test-pymadng.yml

@@ -33,8 +33,8 @@ jobs:
     - name: Get MAD Binaries
       run: |
         mkdir ./src/pymadng/bin 
-        curl https://madx.web.cern.ch/releases/madng/1.1/mad-linux-1.1.2 -o ./src/pymadng/bin/mad_Linux
-        curl https://madx.web.cern.ch/releases/madng/1.1/mad-macos-1.1.2 -o ./src/pymadng/bin/mad_Darwin
+        curl https://madx.web.cern.ch/releases/madng/1.1/mad-linux-1.1.3 -o ./src/pymadng/bin/mad_Linux
+        curl https://madx.web.cern.ch/releases/madng/1.1/mad-macos-1.1.3 -o ./src/pymadng/bin/mad_Darwin
         chmod +x ./src/pymadng/bin/mad_Linux ./src/pymadng/bin/mad_Darwin
     - name: Install dependencies
       run: |

CHANGELOG.md

@@ -1,4 +1,14 @@
-0.6.3 (2024/30/05)
+0.7.0 (2025/06/05)
+Update to MAD-NG 1.1.3 \
+Breaking change: tables in lua are always returned as references, so you must use `eval` to get the value of the table. Or use the optional second argument in `py:send` such as `py:send(data, true)` to return the value of the table. \
+Dictionaries in python can now be sent to MAD-NG, and will be converted to a lua table. \
+Add an optional parameter to to_df and convert_to_dataframe methods to allow the user to specify to always return a pandas dataframe, instead of a tfs dataframe, when tfs is installed. \
+Update the documentation and examples to work again. \
+Remove iter restriction on MAD-NG objects that are not sequences, now all objects can be iterated over. \
+Renamed redirect_sterr to redirect_stderr in the MAD object, fixing a typo. \
+
+
+0.6.3 (2025/04/30)
 
 Update to MAD-NG 1.1.2
 

docs/source/advanced_features.md

@@ -67,7 +67,7 @@ MAD-NG supports callbacks and iterative evaluations, which can be tied into Pyth
 In MAD:
 ```lua
 function twiss_and_send()
-  local tbl, flow = twiss {sequence=seq, method=4}
+  local tbl, flow = twiss {sequence=seq}
   py:send({tbl.s, tbl.beta11})
   return tbl, flow
 end

docs/source/communication.md

@@ -89,8 +89,8 @@ If the object is not an `mtable`, a `TypeError` will be raised.
 ```
 
 See:
-```{literalinclude} ../../examples/ex-ps-twiss/ps-twiss.py
-:lines: 18, 24, 41-49
+```{literalinclude} ../../examples/ex-ps-twiss/ex-ps-twiss.py
+:lines: 19, 25, 42-53
 :linenos:
 ```
 

docs/source/contributing.md

@@ -64,8 +64,7 @@ python -m unittest tests/*.py
 ## Best Practices
 
 ### Code Style
-- Follow PEP8 (enforced via linters)
-- Use descriptive names for MAD objects (e.g. `tw`, `flow`, `seq`)
+- Use descriptive names for everything
 - Keep high-level user APIs separate from internal helpers
 - Use Ruff for code and import formatting.
 

docs/source/ex-lhc-couplingLocal.rst

@@ -5,7 +5,7 @@ LHC Example
    :local:
    :depth: 2
 
-The file :ref:`ex-lhc-couplingLocal/lhc-couplingLocal.py <ex-lhc>` contains an example of loading the required files to use and run the LHC, while including a method to receive and plot intermediate results of a match.
+The file :ref:`ex-lhc-couplingLocal/ex-lhc-couplingLocal.py <ex-lhc>` contains an example of loading the required files to use and run the LHC, while including a method to receive and plot intermediate results of a match.
 
 Loading the LHC 
 ---------------
@@ -14,8 +14,8 @@ The following lines loads the required variables and files for the example. ``as
 
 To grab variables from the MAD-X environment, we use ``mad.load("MADX", ...)``.
 
-.. literalinclude:: ../../examples/ex-lhc-couplingLocal/lhc-couplingLocal.py
-    :lines: 12-22
+.. literalinclude:: ../../examples/ex-lhc-couplingLocal/ex-lhc-couplingLocal.py
+    :lines: 20-28
     :linenos:
 
 Receving intermediate results
@@ -25,80 +25,80 @@ The most complicated part of the example includes the following set of lines.
 
 From lines 4 - 8 below, we define a function that will be invoked during the optimization process at each iteration. Within this function, we perform a twiss for the match function to use, while also sending some information on the twiss to python, on line 6. 
 
-From lines 10 - 21, we run a match, with a **reference** to the match result returned to the variable ``match_rtrn``. Line 22 is a very important line, as this is something you place in the pipe to MAD-NG for MAD-NG to execute once the match is done. Lines 23-25 receive the first result returned during the match, so that we can start plotting the results.
+From lines 10 - 23, we run a match, with a **reference** to the match result returned to the variable ``match_rtrn``. Line 24 is a very important line, as this is something you place in the pipe to MAD-NG for MAD-NG to execute once the match is done. Lines 23-25 receive the first result returned during the match, so that we can start plotting the results.
 
-The plotting occurs between lines 27 - 36, wtih the while loop continuing until twiss result is ``None``, which occurs when the match is done, as requested on line 22.
+The plotting occurs between lines 29 - 38, wtih the while loop continuing until twiss result is ``None``, which occurs when the match is done, as requested on line 24.
 
-Finally, on lines 38 and 39, we retrieve the results of the match from the variable ``match_rtrn``. Since ``match_rtrn`` is a *temporary variable*, there is a limit to how many of these that can be stored (see :doc:`/advanced_features` for more information on these), we delete the reference in python to clear the temporary variable so that is is available for future use.
+Finally, on lines 40 and 41, we retrieve the results of the match from the variable ``match_rtrn``. Since ``match_rtrn`` is a *temporary variable*, there is a limit to how many of these that can be stored (see :doc:`/advanced_features` for more information on these), we delete the reference in python to clear the temporary variable so that is is available for future use.
 
 .. important::
     As MAD-NG is running in the background, the variable ``match_rtrn`` contains *no* information and instead must be queried for the results. During the query, python will then have to wait for MAD-NG to finish the match, and then return the results. On the other hand, if we do not query for the results, the match will continue to run in the background, we can do other things in python, and then query for the results later.
 
-.. literalinclude:: ../../examples/ex-lhc-couplingLocal/lhc-couplingLocal.py
-    :lines: 39-77
+.. literalinclude:: ../../examples/ex-lhc-couplingLocal/ex-lhc-couplingLocal.py
+    :lines: 50-90
     :linenos:
 
-LHC Speed Tests
----------------
+.. LHC Speed Tests
+.. ---------------
 
-This file creates functions within MAD-NG, ``LHC_load``, which loads the LHC and ``reg_expr``, which looks through the MADX environment and places anything that is a deferred expression into the table ``expr``.
+.. This file creates functions within MAD-NG, ``LHC_load``, which loads the LHC and ``reg_expr``, which looks through the MADX environment and places anything that is a deferred expression into the table ``expr``.
 
-The ``LHC_load`` function sends a string ``"done"`` afterwards, so that Python can stay in sync and time the LHC correctly.
+.. The ``LHC_load`` function sends a string ``"done"`` afterwards, so that Python can stay in sync and time the LHC correctly.
 
-The code below just runs the function and times it.
+.. The code below just runs the function and times it.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 44-47
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 44-47
 
 
-The ``reg_expr`` fucntion is recursive so, after calling the function, to ensure Python stays in sync, Python is required to ask MAD-NG for the string ``"done"``.
+.. The ``reg_expr`` fucntion is recursive so, after calling the function, to ensure Python stays in sync, Python is required to ask MAD-NG for the string ``"done"``.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 50-53
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 50-53
 
-Next, we have the first of two methods to evaluate every deferred expression in the LHC and receive them, where Python performs a loop through the number of deferred expressions and has to ask MAD-NG everytime to receive the result.
+.. Next, we have the first of two methods to evaluate every deferred expression in the LHC and receive them, where Python performs a loop through the number of deferred expressions and has to ask MAD-NG everytime to receive the result.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 60-64
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 60-64
 
-The second method involves making MAD-NG do a loop at the same time as python and therefore does not require back on forth communication, which speeds up the transfer of data
+.. The second method involves making MAD-NG do a loop at the same time as python and therefore does not require back on forth communication, which speeds up the transfer of data
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 67-71
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 67-71
 
-The two methods above do not store the data, so the next bit of code is identical to above, but uses list comprehension to store the data into a list automatically, storing the lists into variables ``exprList1`` and ``exprList2``. The main point of seperating the methods above and below was to identify if storing the variables into a list was a bottleneck.
+.. The two methods above do not store the data, so the next bit of code is identical to above, but uses list comprehension to store the data into a list automatically, storing the lists into variables ``exprList1`` and ``exprList2``. The main point of seperating the methods above and below was to identify if storing the variables into a list was a bottleneck.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 74-77, 79-83
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 74-77, 79-83
 
 
-While we have the LHC loaded, the next example grabs the name of every element in the sequence ``lhcb1``, demonstrating the ability and speed of pymadng and MAD-NG, other attributes could also be grabbed, but for simplicity this code just gets the names. This bit of code also uses list comprehension while making MAD-NG loop at the same time as Python.
+.. While we have the LHC loaded, the next example grabs the name of every element in the sequence ``lhcb1``, demonstrating the ability and speed of pymadng and MAD-NG, other attributes could also be grabbed, but for simplicity this code just gets the names. This bit of code also uses list comprehension while making MAD-NG loop at the same time as Python.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 89-99
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 89-99
 
-Another method, not shown above could be to create an entire list on the side of MAD-NG and then send the entire list to python. Which is done below, where instead all the names from the sequence ``lhcb2`` are taken from MAD-NG.
+.. Another method, not shown above could be to create an entire list on the side of MAD-NG and then send the entire list to python. Which is done below, where instead all the names from the sequence ``lhcb2`` are taken from MAD-NG.
 
-.. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
-    :lines: 104-114
+.. .. literalinclude:: ../../examples/ex-recv-lhc/ex-defexpr.py
+..     :lines: 104-114
 
-You can run the file yourself to retrieve the timings, but below is one run on an Intel® Core™ i7-8550U CPU @ 1.80GHz × 8 in Ubuntu 22.04.1 LTS
+.. You can run the file yourself to retrieve the timings, but below is one run on an Intel® Core™ i7-8550U CPU @ 1.80GHz × 8 in Ubuntu 22.04.1 LTS
 
-.. code-block:: console
+.. .. code-block:: console
 
-    Load time: 0.3955872058868408  sec
-    reg_expr time: 0.034337759017944336  sec
+..     Load time: 0.3955872058868408  sec
+..     reg_expr time: 0.034337759017944336  sec
 
-For evaluating the deferred expressions
+.. For evaluating the deferred expressions
 
-.. code-block:: console
+.. .. code-block:: console
 
-    eval time method 1: 0.5888900756835938  sec
-    eval time method 2: 0.2224569320678711  sec
-    eval time method 3: 0.6652431488037109  sec
-    eval time method 4: 0.21885156631469727  sec
+..     eval time method 1: 0.5888900756835938  sec
+..     eval time method 2: 0.2224569320678711  sec
+..     eval time method 3: 0.6652431488037109  sec
+..     eval time method 4: 0.21885156631469727  sec
 
-.. code-block:: console
+.. .. code-block:: console
 
-    time to retrieve every element name in lhcb1 sequence 0.024236202239990234 sec
-    time to retrieve every element name in lhcb2 sequence 0.0245511531829834 sec
+..     time to retrieve every element name in lhcb1 sequence 0.024236202239990234 sec
+..     time to retrieve every element name in lhcb2 sequence 0.0245511531829834 sec

docs/source/examples.rst

@@ -19,7 +19,7 @@ PS Twiss Example
 
 .. _ex-ps-twiss:
 
-.. literalinclude:: ../../examples/ex-ps-twiss/ps-twiss.py
+.. literalinclude:: ../../examples/ex-ps-twiss/ex-ps-twiss.py
    :linenos:
 
 
@@ -28,7 +28,7 @@ LHC Example
 
 .. _ex-lhc:
 
-.. literalinclude:: ../../examples/ex-lhc-couplingLocal/lhc-couplingLocal.py
+.. literalinclude:: ../../examples/ex-lhc-couplingLocal/ex-lhc-couplingLocal.py
    :linenos:
 
 Managing References Example

docs/source/quickstartguide.md

@@ -53,7 +53,7 @@ mad.seq.beam = mad.beam()
 ### Low-Level:
 
 ```python
-mad.send("seq.beam = beam")
+mad.send("seq.beam = beam {}")
 ```
 
 ---
@@ -63,13 +63,13 @@ mad.send("seq.beam = beam")
 ### High-Level:
 
 ```python
-mad["tbl", "flow"] = mad.twiss(sequence=mad.seq, method=4)
+mad["tbl", "flw"] = mad.twiss(sequence=mad.seq)
 ```
 
 ### Low-Level:
 
 ```python
-mad.send("tbl, flow = twiss {sequence=seq, method=4}")
+mad.send("tbl, flw = twiss {sequence=seq}")
 mad.send("py:send(tbl)")
 tbl = mad.recv()
 ```

examples/ex-LowLevel/ex-send-multypes.py

@@ -1,7 +1,9 @@
-from pymadng import MAD
-import numpy as np
 import time
 
+import numpy as np
+
+from pymadng import MAD
+
 arr0 = np.zeros((10000, 1000)) + 1j  # 2*10000*1000*8 -> 160 MB
 
 mad = MAD()
@@ -20,46 +22,59 @@
     {0} = cm1 {1} {2}
     py:send({0})"""
 
-mad.send(cmatrixString.format("cm4", "*", 1)) ## Set cm4 to cm1 * 1 and send it to Python
-mad.send(cmatrixString.format("cm1", "*", 2)) ## Set cm1 to cm1 * 2 and send it to Python
-mad.send(cmatrixString.format("cm2", "*", 2)) ## Set cm2 to cm1 * 2 and send it to Python
-mad.send(cmatrixString.format("cm3", "/", 3)) ## Set cm3 to cm1 / 3 and send it to Python
+mad.send(
+    cmatrixString.format("cm4", "*", 1)
+)  ## Set cm4 to cm1 * 1 and send it to Python
+mad.send(
+    cmatrixString.format("cm1", "*", 2)
+)  ## Set cm1 to cm1 * 2 and send it to Python
+mad.send(
+    cmatrixString.format("cm2", "*", 2)
+)  ## Set cm2 to cm1 * 2 and send it to Python
+mad.send(
+    cmatrixString.format("cm3", "/", 3)
+)  ## Set cm3 to cm1 / 3 and send it to Python
 
 ## Create a vector in MAD and send it to Python
 mad.send("""
     local v1 = (MAD.vector(45):seq()*2 + 1)/3
     py:send(v1)
     """)
-start_time = time.time() # Start timer
+start_time = time.time()  # Start timer
 
 # Receive the matrices and vectors
-m1 = mad.recv() 
+m1 = mad.recv()
 cm4 = mad.recv()
 cm1 = mad.recv()
 cm2 = mad.recv()
 cm3 = mad.recv()
 v1 = mad.recv()
 
-print(time.time() - start_time) # Print time
+print(time.time() - start_time)  # Print time
 
 # Check if the matrices have been correctly sent
-print(np.all(cm1 == arr0*2)) 
-print(np.all(cm2 == arr0*2*2))
-print(np.all(cm3 == arr0*2/3))
+print(np.all(cm1 == arr0 * 2))
+print(np.all(cm2 == arr0 * 2 * 2))
+print(np.all(cm3 == arr0 * 2 / 3))
 print(np.all(cm4 == arr0))
 
 # Send a list to MAD and receive a changed version back
-myList = [[1, 2, 3, 4, 5, 6, 7, 8, 9]] * 2
+my_list = [[1, 2, 3, 4, 5, 6, 7, 8, 9]] * 2
 mad.send("""
-local list = py:recv()
+list = py:recv() -- Note: no local, so it can be accessed outside this block
 list[1][1] = 10
 list[2][1] = 10
 py:send(list)
 """)
-mad.send(myList)
-myList[0][0] = 10
-myList[1][0] = 10
-print("receiving lists", mad.recv() == myList)
+mad.send(my_list)
+my_list[0][0] = 10
+my_list[1][0] = 10
+mad_list = mad.recv("list")
+for i, inner_list in enumerate(mad_list):
+    for j, val in enumerate(inner_list):
+        print(
+            f"List value at [{i}][{j}]: {val} == {my_list[i][j]}", val == my_list[i][j]
+        )
 
 # Send an integer to MAD and receive a changed version back
 myInt = 4
@@ -94,14 +109,14 @@
 py:send(myNil)
 """)
 mad.send(None)
-print("Nil/None", mad.recv() == None)
+print("Nil/None", mad.recv() is None)
 
 # Receive ranges from MAD
 mad.send("""
 py:send(3..11..2)
 py:send(MAD.nrange(3.5, 21.4, 12))
 py:send(MAD.nlogrange(1, 20, 20))
 """)
-print("irng", mad.recv() == range(3  , 12  , 2)) #Py not inclusive, mad is
+print("irng", mad.recv() == range(3, 12, 2))  # Py not inclusive, mad is
 print("rng", mad.recv() == np.linspace(3.5, 21.4, 12))
-print("lrng", np.allclose(mad.recv(), np.geomspace(1, 20, 20)))
+print("lrng", np.allclose(mad.recv(), np.geomspace(1, 20, 20)))