[cleaner] Make cleaner concurrent inside one archive

[pmoravec]

Draft version adding to the traditional sequential backend also sqlite3 and file based concurrent ones.

TL;DR: please review the idea, test it and comment or ack or nack the approach. Then I will fix the TODOs.

Let me explain various factors and reasons that affected the chosen implementation.

As very very first, I implemented both sqlite3 and files based approaches, see reasoning below. And left the original behaviour just for (performance) comparison (it must be run with -j 1).

First, mappings are very independent objects that do maintain their dataset on their own. Even adding an item to obfuscate (say, FQDN) means the dataset is updated a few times (by the host, FQDN and/or domain). I tried to respect this as I like that independency (and also changing it would require a lot of changes).

Also, the dataset is growing in time (as we discover more domains or IP networks) , which means some instances of lately discovered domain might not be obfuscated or that changing ordering of files to obfuscate result in different final mapping (also in the size of the final map). You can check it by yourself if you reorder the list from get_file_list. So running cleaner concurrently - which means dataset being populated non-deterministically - can end up in different final map. Dont be confused by this as I was.

I chose individual processes to sync over the pieces of information "we obfuscate item X as the first one, and item Y as the next". An option to exchange or sync on the whole mapping/datasets would mean 1) more data to sync and 2) altering the mapping classes "independent" behaviour.

This also means there are gaps in numbers and it is fine. The ordering number is the size of the dataset, which can be incremented more than by one when adding just one item. This is fine, as each and every process replays the same and adds the same stuff into its dataset. And also this allows a smart replay of the whole "how was the dataset created?" process at the end - see the archive.load_parser_entries() call.

Then, ProcessPoolExecutor has two limitations for us that I described directly in the code, plus one substantial one: passing the whole SoSCleaner class is not easily feasible. Doing so would prevent some code movement among classes and would be nicer to hook new code in, but it does not work easily. Trying that, I hit issues like "oh, cloning SoSCleaner into a child process means cloning there sos arguments parsing that overrides some __* method that blocks successful process spawn and iteration over a list". It is possible to fix or hack those traps in our code, but after iteratively doing that for five such traps, I gave up this rabbit hole journey.

Please, consider this as a draft only - see the number of TODO points, plus various methods need a better name. Anyway the code is functionally ready, works well(*) and scales fairly well.

(*) the only concern is that sqlite DB can lock itself. I did a few changes there, but still I seldomly hit a live-lock behaviour over locked DB on some artificial test cases (have 8 identical files each with 100 unique IP addresses inside a sosreport, and run cleaner with -j 8 is my "favourite" one).

While I like the sqlite approach more (it looks so professional!), we cant choose it until we fix these live-locks / DB locks.
Gladly, the file-based approach works smoothly and provides comparable (or even slightly better) performance.

When speaking about performance, some benchmark tests are running. Results from 8cores RHEL10 beta, median time from 3 runs each:

a small, 11MB packed sosreport:

• current cleaner run 0m59.799s
• sqlite-based cleaner run 0m59.956s (j=1), 0m34.710s (j=2), 0m22.761s (j=4) and 0m14.137s (j=8)
• files-based cleaner run 0m59.760s (j=1), 0m34.594s (j=2), 0m22.650s (j=4) and 0m14.111s (j=8)

220MB packed sosreport:

• current cleaner run 42m2.282s
• sqlite-based cleaner run 42m4.665s (j=1), 23m17.864s (j=2), 19m17.219s (j=4) and 10m26.143s (j=8)
• files-based cleaner run 42m0.994s (j=1), 23m12.987s (j=2), 19m23.130s (j=4) and 10m26.608s (j=8)

350MB packed sosreport:

• current cleaner run 65m30.090s
• sqlite-based cleaner run 66m59.097s (j=1), 42m5.778s (j=2), 29m3.550s (j=4) and 25m45.191s (j=8)
• files-based cleaner run 66m51.753s (j=1), 41m39.086s (j=2), 28m21.102s (j=4) and 24m20.798s (j=8)

770MB packed sosreport:

• current cleaner run 794m0.238s - yes, over 12 hours
• sqlite-based cleaner run 794m47.511s (j=1), 436m9.563s (j=2), 219m25.159s (j=4) and 164m10.205s (j=8) - speeded up from 12h to 2.5h
• files-based cleaner run 793m45.589s (j=1), 432m25.081s (j=2), 218m22.004s (j=4) and 165m10.775s (j=8) - again speeded up from 12h to 2.5h

1GB packed sosreport:

• current cleaner run 66m17.037s
• sqlite-based cleaner run 66m14.740s (j=1), 48m7.181s (j=2), 26m5.231s (j=4) and 23m20.612s (j=8)
• files-based cleaner run 65m56.254s (j=1), 47m57.850s (j=2), 25m50.313s (j=4) and 23m19.893s (j=8)

Closes: #3097

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo yum install -y dnf-plugins-core on RHEL 8
• sudo dnf install -y dnf-plugins-core on Fedora
• dnf copr enable packit/sosreport-sos-3988
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[pmoravec]

How to get the sqlite DB locks:

In an unpacked sosreport:

for i in $(seq 1 10); do
  for j in $(seq 1 10); do
    echo "${i}.${j}.${i}.${j}"
  done
done > hunderd_of_IPs.0.txt

for i in $(seq 1 8); do
  cp hunderd_of_IPs.0.txt hunderd_of_IPs.${i}.txt

And run sos clean -j 8 --concurrency-backend sql <DIR> (-j 4 should get the same outcome as well with smaller probability).

[TurboTurtle]

From a pure review standpoint, I like this approach and the code changes overall. We'd want to ensure that the --jobs option gets exposed to report for leveraging in-line cleaning, but that can come later.

I am unsure of the need to carry both files and sqlite as the concurrency mechanism though. Especially if the results are largely the same. I have a preference to only carry files, but am certainly open to hearing why we should prefer the sqlite approach (if we only carry one forward).

That said, on python 3.13 (my Fedora daily driver install), I get this error when trying to clean an archive:

sosreport-terra-2025-04-22-xicbmuo :               Beginning obfuscation...
Exception while processing sosreport-terra-2025-04-22-xicbmuo: cannot pickle 'BufferedReader' instances
No reports obfuscated, aborting...

and it is not immediately obvious to me where that BufferedReader is coming into play here but I suspect it is from the ProcessPoolExecutor instantiation.

What python version(s) are you able to get successful executions on?

[pmoravec]

BufferedReader: I can reproduce the same on a few Fedora versions and python versions, the problem is archive class instance is not pickable.

One more bug:

  File "/path/to/sos-pmoravec-concurrent-cleaner-TESTING/sos/cleaner/mappings/__init__.py", line 135, in add
    os.link(tmpfile.name, os.path.join(self.link_dir, f"{counter}"))
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmp9emqdshx' -> '/etc/sos/cleaner/cleaner_links/soshostnamemap/1'

Because /tmp is on a different lvolume/device than /etc. The tmpfile must be under self.link_dir (plus minus).

[pmoravec]

The cannot pickle 'BufferedReader' instances is resolved now (though in a way I dont like much; simply the archive class instance has to keep some open FD, to a file or a connection, but I was unable to identify either - while I knew I was explicitly dropping some such connections to sqlite). Previous testing was run on several RHELs (9 and 10 beta), it is strange it hasnt hit this issue despite same python version on Fedora hit it.

Invalid cross-device link: also fixed.

The concurrency mechanism does not need to be configurable, I think sticking to either sql xor files is fine. I like sql more but as it has scalability issues we cant afford, I would vote for files as well. I put it in the draft as an option to easy test and compare both.

The --jobs option gets exposed to report: good catch, I thought it is there but we use --threads instead. What option name would you suggest, anything better than --cleaner-concurrency? And it should go into Report Options:, not to Global Options:, right?

[pmoravec]

This change (and same in ipv6_test.py is tricky. Some previous test obfuscated tmp into obfuscatedword1. We replaced /etc/sos/cleaner/default_mapping by empty file, yet the cleaner's cache kept the mapping. So self.assertFileCollected(MOCK_FILE) fails as the filename is changed by cleaner in the tarball.

[pmoravec]

Various tests are failing the same way I fixed one instance of that failure in #3988 (comment) :

• https://github.com/sosreport/sos/blob/main/tests/cleaner_tests/existing_archive.py#L127 adds tmp to a keyword mapper, which is stored in static /etc/sos/cleaner/default_mapping and also "transient" [/var|]/tmp/sos_avocado_tests/cleaner_tests/soskeywordmap directory.
• various tests remove /etc/sos/cleaner/default_mapping but not the cleaner_cache dir, so they load var -> obfuscatedword0 and tmp -> obfuscatedword1 mapping
• so the tests replace tmp dir inside the archives they work on, causing errors like /var/tmp/avocado_yhobmqaosos_tests.py.rhbz1950350/sosreport-rhbz1950350/sosreport-host1-2025-06-18-ryovfyo/var/log/clean_config_test.txt does not exist

How to resolve it? A few ideas:

• restrict the https://github.com/sosreport/sos/blob/main/tests/cleaner_tests/existing_archive.py#L127 just for avocado keyword
• use --tmp-dir /var/tmp/foo/bar/ there and test for foo,bar,avocado keywords (or something more clever but unique than foo and bar, ofc - simply something we wont see inside any tested archive)
• remove [/var|]/tmp/sos_avocado_tests/cleaner_cache within tests (either after the existing_archive.py#L127 execution, or in setUp phase of any potentially affected test) - but such removal can step on other running tests toes that work over already built cleaner_cache dir
• put the auxiliary testing files directly to / instead of /tmp or /var/*, like I did for the ipv6 case
• enhance cleaner code by some routine to safely remove cleaner_cache when nobody uses it (needs some locking mechanism, doable). And tests will call that removal routine whenever they purge away default_mapping

Any other idea? Any preferences? I think the 1st or 2nd option is less painful.

[jcastill]

How to resolve it? A few ideas:

• restrict the https://github.com/sosreport/sos/blob/main/tests/cleaner_tests/existing_archive.py#L127 just for avocado keyword
• use --tmp-dir /var/tmp/foo/bar/ there and test for foo,bar,avocado keywords (or something more clever but unique than foo and bar, ofc - simply something we wont see inside any tested archive)
• remove [/var|]/tmp/sos_avocado_tests/cleaner_cache within tests (either after the existing_archive.py#L127 execution, or in setUp phase of any potentially affected test) - but such removal can step on other running tests toes that work over already built cleaner_cache dir
• put the auxiliary testing files directly to / instead of /tmp or /var/*, like I did for the ipv6 case
• enhance cleaner code by some routine to safely remove cleaner_cache when nobody uses it (needs some locking mechanism, doable). And tests will call that removal routine whenever they purge away default_mapping

Any other idea? Any preferences? I think the 1st or 2nd option is less painful.

I think that the first option is the best for now, with the idea of working on the fourth long term.

[pmoravec]

After some offline discussion, the best is:

• short-term solution in change from --keywords var,tmp,avocado to --tmp-dir /var/tmp/soscleanertest/banana --keywords avocado,banana,soscleanertest in https://github.com/sosreport/sos/blob/main/tests/cleaner_tests/existing_archive.py#L127
• long-term solution to have some method to safely remove cleaner_cache when no cleaner is running, and use this method in tests setUp

Until there will be another idea, I will follow the short-term solution in next few days.

[pmoravec]

/packit retest-failed

[arif-ali]

I've created 3000 users, and thought I would try that, the first iteration from the original code took 65 mins.

real	65m10.759s
user	62m56.780s
sys	0m16.806s

Second one had issues, the traceback/explanation is below

Installed the package from my local PPA, based on the change as per our discussion last week, and that was quicker on the real time, but user time was longer, results below

run1

real	36m19.024s
user	93m58.093s
sys	1m35.355s

run2

real	34m10.159s
user	86m45.177s
sys	1m22.987s

This was run in a container, so will try in a VM, and see if I get something different

The issue on the second run, I got the following traceback

Traceback (most recent call last):
  File "/usr/bin/sos", line 22, in <module>
    sos.execute()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/sos/__init__.py", line 187, in execute
    self._component.execute()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/__init__.py", line 384, in execute
    self.preload_all_archives_into_maps()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/__init__.py", line 669, in preload_all_archives_into_maps
    self._prepare_archive_with_prepper(archive, prepper)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/__init__.py", line 640, in _prepare_archive_with_prepper
    _parser.mapping.add(item)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/mappings/__init__.py", line 60, in add
    self.dataset[item] = self.sanitize_item(item)
                         ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/mappings/username_map.py", line 33, in sanitize_item
    return self.sanitize_item(item.lower())
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/mappings/username_map.py", line 33, in sanitize_item
    return self.sanitize_item(item.lower())
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/sos/cleaner/mappings/username_map.py", line 33, in sanitize_item
    return self.sanitize_item(item.lower())
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  [Previous line repeated 991 more times]
RecursionError: maximum recursion depth exceeded

I have user1 to user3000, and this traceback happens with a different username each time. This shouldn't happen, and will investigate what is going on. This is with the original code.

This new version didn't have the traceback, so hopefully we won't see the issue

[pmoravec]

IMHO that traceback can not happen in concurrent cleaner where I had to catch all exceptions to prevent child process termination (https://github.com/sosreport/sos/pull/3988/files#diff-64f7cecab583ab4d9d9a3d4d4d89fda30a4b0e5b8e9371cc8a156bdb626e2d0fR209-R211). In such a case, no sanitisation happens for the given line and parser.

But we should investigate also the root cause of infinite recursion - that sounds like generic problem that my code would rather hide than resolve.

[arif-ali]

So, just looking at those time fields, the real time is the right time, so the new code is deffo faster ,and my 3rd was even better, so from this perspective, looks good

real	24m34.495s
user	68m0.067s
sys	0m22.820s

Also, just realised, as we changed the tmp-dir for Debian/Ubuntu now in the policy to /tmp, not the obfuscated report and normal actually ended up in /var/tmp instead of /tmp, and this was unexpected :(; will debug this further

[pmoravec]

I investigated the RecursionError which would be an issue alone. The root cause of that bug is:

• some username is already mapped in /etc/sos/cleaner/default_mapping
• next sos clean execution both loads the username from the mapping file and from prepper
• prepper tries to obfuscate it (again), and enters the endless recursion

Just one common username in mapping file and in lastlog* files, and we have this reproducer.

This can't happen in concurrent cleaner, since SoSMap.add newly does not call self.dataset[item] = self.sanitize_item(item) in this situation: it tests presence in dataset and skips to provide the value directly. Purely side effect of a must-feature in concurrent cleaner :)

(so the presence of this issue is yet another reason to review this PR :) )

[pmoravec]

Hi mainly @TurboTurtle but also others - can you please do a tech review of the codechanges as well?

I would like to move this PR further / have it merged.

[TurboTurtle]

I'm having some inconsistencies in performance. I've had some executions that run better than current main, and others that sit for 15+ minutes, using default values.

I've not been able to track down what's happening here as of yet.

[pmoravec]

I'm having some inconsistencies in performance. I've had some executions that run better than current main, and others that sit for 15+ minutes, using default values.

I've not been able to track down what's happening here as of yet.

Can you provide the sosreport you get stuck cleaner on, so I can investigate? If you run cleaner with -vvv, does it print to stdout pid=12345: obfuscating /var/tmp/sosreport-foo...?

[pmoravec]

The new version contains the two Jake's comments applied.

[TurboTurtle]

Can you provide the sosreport you get stuck cleaner on, so I can investigate? If you run cleaner with -vvv, does it print to stdout pid=12345: obfuscating /var/tmp/sosreport-foo...?

Generally it would hang on the system journal collection, but I've not been able to reproduce the issue since then.

LGTM, would like to get any feedback from @arif-ali before merge, but also want to have this in before our release in August either way.

[arif-ali]

Went through stuff over the past couple of days, looks good to me