ctre: 'ctr: content digest sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: not found'

[vilitux]

While running ctre prune -f, sometimes we run into the following issue:

(deleted container) k8s.io mynamespace helloworld-01234567-xxxxx helloworld 2024-05-13T12:00:02.26628533Z
(deleted container) k8s.io mynamespace helloworld-34534534-xxxxx helloworld [sandbox] 2024-05-13T13:00:00.7284989Z
(deleted container) k8s.io mynamespace helloworld-34534534-xxxxx helloworld 2024-05-13T13:00:01.468378018Z
ctr: content digest sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: not found
Traceback (most recent call last):
  File "/usr/bin/ctre", line 700, in <module>
    main()
  File "/usr/bin/ctre", line 689, in main
    prune_content_references()
  File "/usr/bin/ctre", line 397, in prune_content_references
    ret = check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ctr', '-n', 'k8s.io', 'content', 'prune', 'references']' returned non-zero exit status 1.

When rerunning the same command, it completes successfully (so far).