Skip to content

common: make PersistedProgress._lock a private attribute #641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sigmaris
Copy link

About this change - What it does

Makes the _lock member of PersistedProgress into a Pydantic private attribute. This means it'll be excluded from (de)serialization.
This fixes the following problem on Pydantic 2.10.6, Python 3.12.3:

UploadEventProgressTracker Thread-11 DEBUG: Tracking upload event for file default/xlog/000000020000002E0000009D
TransferAgent Thread-9 ERROR: Problem in moving file: PosixPath('/var/lib/pghoard/default/xlog/000000020000002E000000A6'), need to retry
Traceback (most recent call last):
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pghoard/transfer.py", line 481, in handle_upload
    storage.store_file_object(
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/rohmu/object_storage/sftp.py", line 235, in store_file_object
    bytes_written = self._put_object(
                    ^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/rohmu/object_storage/sftp.py", line 262, in _put_object
    self.client.putfo(fl=fd, remotepath=target_path, callback=wrapper_upload_progress_fn)
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 716, in putfo
    size = self._transfer_with_callback(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 684, in _transfer_with_callback
    callback(size, file_size)
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/rohmu/object_storage/sftp.py", line 259, in wrapper_upload_progress_fn
    upload_progress_fn(bytes_written, total_bytes)
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/rohmu/object_storage/base.py", line 168, in wrapper
    cb(progress - last_progress)
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pghoard/transfer.py", line 485, in <lambda>
    upload_progress_fn=lambda n_bytes: upload_progress_fn(total_bytes_uploaded=n_bytes),
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pghoard/transfer.py", line 149, in increment
    persisted_progress = PersistedProgress.read(metrics=self.metrics)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pghoard/common.py", line 154, in read
    return cls()
           ^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py", line 339, in init_private_attributes
    default = private_attr.get_default()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pydantic/fields.py", line 1166, in get_default
    return _utils.smart_deepcopy(self.default) if self.default_factory is None else self.default_factory()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/pghoard/venv/lib/python3.12/site-packages/pydantic/_internal/_utils.py", line 345, in smart_deepcopy
    return deepcopy(obj)  # slowest way when we actually might need a deepcopy
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 151, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.lock' object

Why this way

This seems like the correct way to specify an attribute that isn't a field, should be excluded from serialization, and initialised by a default_factory according to https://docs.pydantic.dev/latest/concepts/models/#private-model-attributes

This avoids a 'TypeError: cannot pickle '_thread.lock' object' problem
on Pydantic 2.10.6.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant