DM-48690 : Implement Caching Butler Pool#165
DM-48690 : Implement Caching Butler Pool#165tcjennings merged 4 commits intotickets/DM-49324/releasefrom
Conversation
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
3 times, most recently
from
2f6d285 to
5aecda3
Compare
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
2 times, most recently
from
eb0c9da to
d55e8f9
Compare
tcjennings
changed the base branch from
tickets/DM-49324/release
to
tickets/DM-49522/deprecate_db
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
2 times, most recently
from
1d29e7c to
7f7275b
Compare
tcjennings
force-pushed
the
tickets/DM-49522/deprecate_db
branch
from
715a894 to
7e1fa74
Compare
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
8 times, most recently
from
f0b22c8 to
b69e742
Compare
tcjennings
force-pushed
the
tickets/DM-49324/release
branch
5 times, most recently
from
c5cc5cd to
71b4107
Compare
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
3 times, most recently
from
bb57a8b to
77973c7
Compare
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
2 times, most recently
from
93ca0ac to
e8a8e7d
Compare
| op.drop_index(op.f("ix_step_name"), table_name="step", if_exists=True) | ||
| op.drop_index(op.f("ix_campaign_spec_id"), table_name="campaign", if_exists=True) | ||
| op.drop_index(op.f("ix_campaign_spec_block_id"), table_name="campaign", if_exists=True) | ||
| op.drop_index(op.f("ix_campaign_parent_id"), table_name="campaign", if_exists=True) |
There was a problem hiding this comment.
Were these database schema changes included in this PR in error? They seem unrelated to the code change.
There was a problem hiding this comment.
No error, they're part of a bugfix to support more successful migration downgrades but not worth their own PR.
| without_datastore=True, | ||
| ) | ||
| butler = await to_thread.run_sync(butler_f) | ||
| butler = BUTLER_FACTORY.get_butler(butler_repo, collections=[input_coll, campaign_input_coll]) |
There was a problem hiding this comment.
This call to get get_butler can ultimately call Butler.from_config and butler.clone(), both of which do synchronous I/O. So this probably should be run in a thread pool.
There was a problem hiding this comment.
butler.clone() does sync IO? What is the nature of this IO?
There was a problem hiding this comment.
There is a "default data ID" feature (long story) that gets triggered when you pass the collections argument to the Butler constructor or Butler.clone(). That hits the database to find out which data IDs are associated with the collections.
The default data ID feature is the main thing that can be triggered in either place. But also the constructor can incidentally go hit the disk or network to load miscellaneous configuration files etc, and it's hard to predict exactly when that will happen because the I/O is scattered across multiple layers of the code and in some cases is triggered by dynamic loading of classes based on strings in the configuration files (or implicitly loaded configuration files).
Basically anything you ever do with the Butler should be done in a thread pool, full stop.
Automatically generated by python-semantic-release
tcjennings
force-pushed
the
tickets/DM-48690/butler_pool
branch
from
e8a8e7d to
584b3fe
Compare
2 participants
Implements a caching butler factory for the CM service such that