Adding the KMOD summary outputs #560

mattiastefanelli · 2025-11-26T16:52:11Z

Adding a script to generate .tfs and .txt tables with the summary of the KMOD measurement imported with kmod_importer. It will generate one .tfs and one .txt table for each beam.
The .txt table has been made to be easily copy/pasted in the logbook keeping a readable format.

…to kmod_tabsum_import

…/omc3 into kmod_tabsum_import" This reverts commit 875598f, reversing changes made to 6612de9.

mattiastefanelli · 2025-11-27T09:41:04Z

The output .txt table for the Logbook looks like this without Tabulate package

B1_summary_logbook.txt

github-actions · 2025-11-27T10:25:37Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
omc3
kmod_importer.py					344-346
Project Total

_{This report was generated by python-coverage-comment-action}

fsoubelet

Thanks for the PR Mattia, this is going to be useful to have.

I left comments here and there about inconsistencies / code clarity etc. See below.

One thing I'm wondering is: could this be a stand-alone script? And then it could be run on previous folders with kmod measurements from before we had it (and the relevant part just imported and called into kmob_importer.py)?

fsoubelet · 2025-11-27T10:39:36Z

omc3/kmod_importer.py

    # return [round(bstar, 3) for bstar in df_model.loc[ip, [f"{BETA}X", f"{BETA}Y"]]]
    return df_model.loc[ip, [f"{BETA}X", f"{BETA}Y"]].tolist()

+def import_kmod_summary_table(


This writes out things so the naming feels off. Maybe something like output_kmod_summary_tables?

I realize import_kmod_data suffers the same problem. We should look to fix that as well.

(removed - updated below)

import_kmod_data makes a bit more sense, because it is importing it into the omc3 analysis folder in the same format as the other files there. Whereas this function really only outputs some text files, which actually do not really need to go into the omc3 results folder. But if there is a good naming suggestion, I am not strictly against renaming the import-data function as well.

fsoubelet · 2025-11-27T10:41:29Z

omc3/kmod_importer.py


+def import_kmod_summary_table(
+        meas_paths: Sequence[Path | str],
+        averaged_meas_paths: Sequence[tfs.TfsDataFrame],


Above at function call this is passed as averaged_results which is definitely not a Sequence[tfs.TfsDataFrame].

omc3/kmod_importer.py

fsoubelet · 2025-11-27T10:41:55Z

omc3/kmod_importer.py

+    """
+    Write the KMOD summary table from each results.tfs file.
+    It writes down the following files for each beam:
+      {beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.


Suggested change

{beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.

B{beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.

none of the filenames here are correct actually anyway. There is no X/Y it's all in one file.
sum is actually summary and tables is summary_logbook. I prefer tables though.
And make them all lower-case.

fsoubelet · 2025-11-27T10:42:04Z

omc3/kmod_importer.py

+    Write the KMOD summary table from each results.tfs file.
+    It writes down the following files for each beam:
+      {beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.
+      {beam}_kmod_sum_X.tfs: BETSTARY, ERRBETSTARY, BETWAISTY, ERRBETWAISTY, WAISTY, ERRWAISTY.


Suggested change

{beam}_kmod_sum_X.tfs: BETSTARY, ERRBETSTARY, BETWAISTY, ERRBETWAISTY, WAISTY, ERRWAISTY.

B{beam}_kmod_sum_Y.tfs: BETSTARY, ERRBETSTARY, BETWAISTY, ERRBETWAISTY, WAISTY, ERRWAISTY.

fsoubelet · 2025-11-27T11:11:56Z

omc3/kmod_importer.py

+        return df_x, df_y
+
+    grouped = {beam: [] for beam in beams}
+    for elem in meas_paths:


I'd prefer representative variable names. This elem is a path

fsoubelet · 2025-11-27T11:13:07Z

omc3/kmod_importer.py

+        """
+
+        if not av_res_flag:
+            file_name   = data_sngl_file_raw.parent.parent.name


This is not going to work is data_sngl_file_raw is a str, which the type hint says is accepted.

fsoubelet · 2025-11-27T11:16:50Z

omc3/kmod_importer.py

+                f.write("\n".join(txt_output))
+            tfs.write(output_dir/f"{beam}_kmod_summary.tfs", big_df)
+
+    LOG.info('KMOD summary outputs correctly saved.')


Suggested change

LOG.info('KMOD summary outputs correctly saved.')

LOG.info(f"KMOD summary outputs saved at {output_dir/f'{beam}_kmod_summary.tfs'}")

even better is the full output path is stored in a variable before

fsoubelet · 2025-11-27T11:17:59Z

tests/unit/test_kmod_importer.py

            lumi_filename = _get_lumi_filename(betas, ip_a=ips[0], ip_b=ips[1])
            assert (average_dir / lumi_filename).exists()

+


No need for this line

fsoubelet · 2025-11-27T11:18:14Z

tests/unit/test_kmod_importer.py

            lumi_filename = _get_lumi_filename(betas, ip_a=ips[0], ip_b=ips[1])
            assert (average_dir / lumi_filename).exists()

+


Could we add a little check that the written files are as expected?

JoschD

A few comments, but my main point is also, that this should be a standalone script, in case someone only wants the table without importing it anywhere.

When refactoring, also separate the output from the reading/transforming. There should be one function that creates the tables and simply returns them. Then another funtion/outer function that writes them out.

In my opinion the script/main function of the script should also have the following parameters:

outputdir -> optional, write the files ou (always return the tables)
average -> true/false, calls the functions to average the data (could also be boolean | path-to-the-file | averaged-dataframes)
logbook -> optional, specify a logbook (LHC_OMC, LHC_OP, true->LHC_OMC) uses the logbook uploader to create a new logbook entry.

And then you can import the function here and re-use it to create the table automatically on import.

When putting it into another script: consider which parts might benefit from being in their own little function as well, to make the code more readable/self-explainatory.

JoschD · 2025-11-27T11:09:30Z

omc3/kmod_importer.py

+            try:
+                label = data_sngl_file["LABEL"].iloc[0]
+                second_magnet = label.split("-")[1]
+                relevant = second_magnet.split(".")[-1]
+                ip_number = relevant[1:]
+                ip_name = f"IP{ip_number}" # --- it extracts the IP from the magnet label, check for typos
+            except KeyError as e:


Keep the try-part as short as possible (usually only one line), to make sure the Key error originates where you expect it to come from.

JoschD · 2025-11-27T11:10:54Z

omc3/kmod_importer.py

+    cols_x = ["BETSTARX", "ERRBETSTARX", "BETWAISTX", "ERRBETWAISTX", "WAISTX", "ERRWAISTX"]
+    cols_y = ["BETSTARY", "ERRBETSTARY", "BETWAISTY", "ERRBETWAISTY", "WAISTY", "ERRWAISTY"]


I'll mark it here, but this is a general comment for the whole function: Use the constants for the columns. Check the other kmod-related files for how it is done there.

JoschD · 2025-11-27T11:14:12Z

omc3/kmod_importer.py

+    all_av_x, all_av_y = [], []
+    for key, value in averaged_meas_paths.items():
+        LOG.debug(f"Reading averaged results: {key}")
+        df_av_x, df_av_y = output_table_single_beam(value[0], av_res_flag=True)
+        all_av_x.append(df_av_x)
+        all_av_y.append(df_av_y)


rename key and value to what the variables actually mean. (e.g. I have no idea what value[0] is supposed to be.

JoschD · 2025-11-27T11:31:15Z

omc3/kmod_importer.py

    # return [round(bstar, 3) for bstar in df_model.loc[ip, [f"{BETA}X", f"{BETA}Y"]]]
    return df_model.loc[ip, [f"{BETA}X", f"{BETA}Y"]].tolist()

+def import_kmod_summary_table(


(removed - updated below)

JoschD · 2025-11-27T11:44:44Z

omc3/kmod_importer.py

+    cols_y = ["BETSTARY", "ERRBETSTARY", "BETWAISTY", "ERRBETWAISTY", "WAISTY", "ERRWAISTY"]
+    beams = [f"{BEAM_DIR}{1}", f"{BEAM_DIR}{2}"]
+
+    def output_table_single_beam(


I don't think it makes sense to have this as a nested function.

Co-authored-by: Felix Soubelet <[email protected]>

mstefane added 10 commits August 28, 2025 15:23

sphinx, change the :noindex: to :no-index-entry:

89eefb4

First Draft Kmod summary table for Logbook

7dae640

First Draft Kmod summary table for Logbook

6612de9

Merge branch 'kmod_tabsum_import' of https://github.com/pylhc/omc3 in…

875598f

…to kmod_tabsum_import

Revert "Merge branch 'kmod_tabsum_import' of https://github.com/pylhc…

eb0be5c

…/omc3 into kmod_tabsum_import" This reverts commit 875598f, reversing changes made to 6612de9.

Reverting old noindex

2008130

Fixing

71ddc4d

Fixing

b88bab8

Fixing

0e21534

Table output updated .to_markdown()

220961f

mattiastefanelli requested a review from a team as a code owner November 26, 2025 16:52

mattiastefanelli marked this pull request as draft November 26, 2025 16:54

Exception Fixing.

934c3a6

mattiastefanelli marked this pull request as ready for review November 27, 2025 08:37

mattiastefanelli requested review from JoschD, emaclean and fsoubelet November 27, 2025 08:38

mstefane and others added 4 commits November 27, 2025 09:45

Fixinf Ruff suggestions.

5cf7e42

Fixing Ruff suggestions.

d67dae0

Merge branch 'master' into kmod_tabsum_import

b7283ab

.to_markdown() requires Tabulate Package. Changed to .to_string()

8d9e703

fsoubelet reviewed Nov 27, 2025

View reviewed changes

JoschD requested changes Nov 27, 2025

View reviewed changes

Update omc3/kmod_importer.py

cd8891a

Co-authored-by: Felix Soubelet <[email protected]>

	{beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.
	B{beam}_kmod_sum_X.tfs: BETSTARX, ERRBETSTARX, BETWAISTX, ERRBETWAISTX, WAISTX, ERRWAISTX.

	{beam}_kmod_sum_X.tfs: BETSTARY, ERRBETSTARY, BETWAISTY, ERRBETWAISTY, WAISTY, ERRWAISTY.
	B{beam}_kmod_sum_Y.tfs: BETSTARY, ERRBETSTARY, BETWAISTY, ERRBETWAISTY, WAISTY, ERRWAISTY.

	LOG.info('KMOD summary outputs correctly saved.')
	LOG.info(f"KMOD summary outputs saved at {output_dir/f'{beam}_kmod_summary.tfs'}")

		lumi_filename = _get_lumi_filename(betas, ip_a=ips[0], ip_b=ips[1])
		assert (average_dir / lumi_filename).exists()

		cols_x = ["BETSTARX", "ERRBETSTARX", "BETWAISTX", "ERRBETWAISTX", "WAISTX", "ERRWAISTX"]
		cols_y = ["BETSTARY", "ERRBETSTARY", "BETWAISTY", "ERRBETWAISTY", "WAISTY", "ERRWAISTY"]

Adding the KMOD summary outputs #560

Are you sure you want to change the base?

Adding the KMOD summary outputs #560

Uh oh!

Conversation

mattiastefanelli commented Nov 26, 2025

Uh oh!

mattiastefanelli commented Nov 27, 2025

Uh oh!

github-actions bot commented Nov 27, 2025

Coverage report

Uh oh!

fsoubelet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoschD Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoschD left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoschD Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoschD Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JoschD Nov 27, 2025 •

edited

Loading

JoschD Nov 27, 2025 •

edited

Loading

JoschD Nov 27, 2025 •

edited

Loading