esgpublish prints a huge amount of log output

[skremdwd]

A normal import process of around 900 files created a total output/log file of around 700 MiB on disk.

It seems like many lines are repeated over and over with the same content, getting another line longer after every imported dataset.

This may be related to #170?

Log output

Part of the output to demonstrate what it looks like, it seems like it is reprinting some parts but I am not sure why it would do that, I can't find a related issue in the publog code.

esgpublish --no-replica --project climatepredictionsde --map ~/import-test/map/ --config ~/.esg/projects/esg.climatepredictionsde.yaml --no-auth &> ~/log.txt

[...]
Making dataset...
[New dataset gets imported]
Making dataset...
Making dataset...
[New dataset gets imported]
Making dataset...
Making dataset...
Making dataset...
[New dataset gets imported]
Making dataset...
Making dataset...
Making dataset...
Making dataset...
[...]

Part/start of the log file

2024-07-12 08:05:01 INFO     No Globus UUID defined.
2024-07-12 08:05:01 INFO     No data transfer node defined.
2024-07-12 08:05:01 INFO     Converting mapfile...
2024-07-12 08:05:01 INFO     Running Extraction...
2024-07-12 08:05:01 INFO     Making dataset...
2024-07-12 08:05:01 WARNING  experiment does not agree! subseasonal != mfc20240708
2024-07-12 08:05:01 WARNING  product does not agree! restricted != output
2024-07-12 08:05:01 INFO     Updating...
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     Running index pub...
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709.sfcWind_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r77i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709.sfcWind_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r77i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     Done. Cleaning up.
2024-07-12 08:05:02 INFO     No Globus UUID defined.
2024-07-12 08:05:02 INFO     No data transfer node defined.
2024-07-12 08:05:02 INFO     Converting mapfile...
2024-07-12 08:05:02 INFO     Converting mapfile...
2024-07-12 08:05:02 INFO     Running Extraction...
2024-07-12 08:05:02 INFO     Running Extraction...
2024-07-12 08:05:02 INFO     Making dataset...
2024-07-12 08:05:02 INFO     Making dataset...
2024-07-12 08:05:02 WARNING  experiment does not agree! subseasonal != mfc20240708
2024-07-12 08:05:02 WARNING  experiment does not agree! subseasonal != mfc20240708
2024-07-12 08:05:02 WARNING  product does not agree! restricted != output
2024-07-12 08:05:02 WARNING  product does not agree! restricted != output
2024-07-12 08:05:02 INFO     Updating...
2024-07-12 08:05:02 INFO     Updating...
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     [SUCCESS] Number of records updated: 1
2024-07-12 08:05:02 INFO     INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     Running index pub...
2024-07-12 08:05:02 INFO     Running index pub...
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de
2024-07-12 08:05:02 INFO     Done. Cleaning up.
2024-07-12 08:05:02 INFO     Done. Cleaning up.
2024-07-12 08:05:02 INFO     No Globus UUID defined.
2024-07-12 08:05:02 INFO     No data transfer node defined.
2024-07-12 08:05:02 INFO     Converting mapfile...
2024-07-12 08:05:02 INFO     Converting mapfile...
2024-07-12 08:05:02 INFO     Converting mapfile...
2024-07-12 08:05:02 INFO     Running Extraction...
2024-07-12 08:05:02 INFO     Running Extraction...
2024-07-12 08:05:02 INFO     Running Extraction...
2024-07-12 08:05:02 INFO     Making dataset...
2024-07-12 08:05:02 INFO     Making dataset...
2024-07-12 08:05:02 INFO     Making dataset...
[...]

[sashakames]

I am able to reproduce the issue when running with --map <directory>/
two workarounds (1) use a shell loop to iterate through mapfiles and use --map <filename>.map
(2) --silent should suppress output.
Issue under investigation...