DRAFT, DO NOT MERGE 7450/loan availability updater#13067
Draft
benbdeitch wants to merge 10 commits into
Draft
Conversation
…7450) Adds a loan_availability_updater script that polls IA's loan changes API and atomically updates two new Solr fields on work documents: - ebook_availability: "available" | "unavailable" - ebook_becomes_available: ISO-8601 UTC loan expiry timestamp On startup (or after a full Solr re-index), the script binary-searches for the uid approximately 14 days ago so that all currently-active loans are reflected without replaying the entire history. Once caught up it polls every 30s; a full batch (1000 rows) is processed without sleeping. Expired loans are evicted each cycle via a Solr range query on ebook_becomes_available, so missed return/expire events self-heal. Files changed: - openlibrary/core/lending.py: add get_loan_changes() S3 GET helper - conf/solr/conf/managed-schema.xml: add ebook_availability / ebook_becomes_available fields - openlibrary/solr/solr_types.py: type-stub the two new fields - scripts/solr_updater/loan_availability_updater.py: main polling script - scripts/solr_updater/tests/test_loan_availability_updater.py: 20 unit tests Closes internetarchive#7450
Fixes the python_tests CI failure: solr_types.py was manually edited with a Literal type and inline comment that the auto-generator doesn't produce; regenerate to match (string → Optional[str]). Cleanup pass: - Remove 35-line module docstring → 8 lines - Remove section-banner comments throughout - Remove BINARY_SEARCH_ITERS module constant → inline range(40) - Remove redundant local type annotations - Collapse new_uid/last_uid into a single variable computed once - Use contextlib.suppress (ruff SIM105) - DRY resolve_work_keys into a dict comprehension - Trim all function docstrings to one paragraph - Remove unused _make_loan_changes_response test helper - Fix test call-count assertion to match inline range(40)
- Add lending.setup(infogami.config) in main() so config globals are populated
- Wrap find_start_uid() API calls in try/except for network resilience
- Quote identifiers in resolve_work_keys() Solr query (ia:("a" "b" "c"))
- Only set ebook_becomes_available for active loans with a parseable until value
- Move write_state() to after successful Solr commit to keep state consistent
- Add resolve_work_keys unit tests (empty input + basic mapping + query quoting)
Adds a loan_uid (plong) field to the Solr schema. The loan_availability_updater writes the processed event uid to every work it updates, so that on restart (or after a full re-index that wipes the field) the updater can query max(loan_uid) from Solr to resume where it left off — avoiding the need for a manual --reset flag after a re-index. Startup uid resolution order: state file → max(loan_uid) in Solr → binary search. Also adds plong to the types_generator type_map so future plong fields are handled automatically.
…use _root_ and _version_
for more information, see https://pre-commit.ci
mekarpeles
reviewed
Jul 2, 2026
Comment on lines
+166
to
+167
| <field name="ebook_availability" type="string" multiValued="false" stored="false" docValues="true" /> | ||
| <field name="ebook_becomes_available" type="pdate" multiValued="false" stored="false" docValues="true" /> |
Member
There was a problem hiding this comment.
- ebook_availability could be an enum int [0 / 1] (e.g.
unavailable,available) - ebook_next_available -> translate to
epochtime, cleanly an int
Member
|
If we switch from ptime to epoch for the loan_expiry field, then we'll want to confirm timezone which I believe it utc |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7450
When completed, it will add a Solr field for ebook lending availability, with an updater that runs at set intervals. This avoids the current issue of every request having to be sent through to the IA endpointt.
Technical
Testing
Screenshot
Stakeholders