Skip to content

Add AlchemicalArchive extraction - #486

Merged
dotsdl merged 15 commits into
mainfrom
feat/extract_archive
Jul 8, 2026
Merged

Add AlchemicalArchive extraction#486
dotsdl merged 15 commits into
mainfrom
feat/extract_archive

Conversation

@ianmkenney

Copy link
Copy Markdown
Member

closes #246

@dotsdl

dotsdl commented Mar 16, 2026

Copy link
Copy Markdown
Member

Awaiting new gufe release featuring AlchemicalArchive before we merge.

@dotsdl

dotsdl commented Apr 8, 2026

Copy link
Copy Markdown
Member

@ianmkenney would you like me to continue on this one? Otherwise happy to leave it to you!

@ianmkenney

Copy link
Copy Markdown
Member Author

@dotsdl you can take over if you'd like

@dotsdl

dotsdl commented Apr 10, 2026

Copy link
Copy Markdown
Member

Can do! Thanks for all your work on this! Will ask you for a review once I'm finished!

@dotsdl dotsdl self-assigned this Apr 10, 2026
dotsdl and others added 4 commits June 10, 2026 17:29
Add AlchemiscaleClient.get_network_archive / get_network_archives, which
bundle an AlchemicalNetwork with all successful ProtocolDAGResults for its
Transformations into a gufe AlchemicalArchive, with optional user-supplied
metadata. Not-found networks yield None.

Implemented client-side atop the existing get_network and
get_transformation_results machinery, replacing the abandoned server-side
/bulk/networks/archive endpoint (which referenced undefined names and a
non-existent statestore method). Also fixes a missing `json` import in
client.py and restores pdr_from_bytes in utils.py (zstd import + decode
fallback).

Closes #246

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.81%. Comparing base (829532e) to head (8be54a3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #486      +/-   ##
==========================================
+ Coverage   80.54%   80.81%   +0.27%     
==========================================
  Files          31       31              
  Lines        4877     4916      +39     
==========================================
+ Hits         3928     3973      +45     
+ Misses        949      943       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dotsdl dotsdl changed the title [DNM] Add AlchemicalArchive extraction Add AlchemicalArchive extraction Jul 7, 2026
Rework AlchemiscaleClient._get_network_archive to pair each Transformation
with its results by fetching results in parallel via get_network_results and
then retrieving each Transformation by its authoritative (server-side)
ScopedKey. A Transformation deserialized now may not reproduce the GufeKey it
had at ingestion (gufe tokenization can change across versions), so a
reconstructed ScopedKey is not a reliable join and could silently drop
results; the server-side ScopedKey is stable.

Also broaden metadata-serialization validation to catch ValueError (e.g.
circular references), and extend tests to cover string ScopedKey input, the
compress=False path, and per-network metadata ordering in the bulk method.

Document the GufeKey-instability gotcha in CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dotsdl

dotsdl commented Jul 7, 2026

Copy link
Copy Markdown
Member

Design note: client-side archive assembly

This PR builds AlchemicalArchive extracts client-side (AlchemiscaleClient.get_network_archive / get_network_archives), composing existing retrieval methods, rather than adding a server-side endpoint. The original [DNM] draft sketched a /bulk/networks/archive endpoint; we removed it and reimplemented on the client. Rationale and tradeoffs:

Why client-side

  • API worker memory. An archive is inherently bulk: it pulls every successful ProtocolDAGResult (the large objects in the system, backed by S3) for every Transformation in a network and serializes them into one object. Doing that server-side forces a shared API worker to hold the entire result set in memory for a single request. The workers are sized for many small, concurrent request/response cycles — archive generation is the opposite shape (few, huge, bursty). Supporting it server-side would mean provisioning every worker for a peak that is idle almost all the time, i.e. a large, near-permanent over-provision.
  • No new API-side bottleneck. A long-lived, memory-heavy request risks head-of-line blocking for all other users of that worker. Client-side, the weight lands in the user's own process, where a big archive is their RAM, not the server's.
  • Reuse of tested machinery. Assembly is just AlchemicalNetwork + per-Transformation ProtocolDAGResults + optional metadata. The client already has cached, parallelized retrieval for exactly those pieces, so there is nothing to reimplement server-side. AlchemicalArchive construction is gufe logic that runs identically on either side, so centralizing it buys little.
  • Smaller API surface. No new endpoint to authorize, scope-check, version, and maintain.

Tradeoffs accepted

  • More round trips. Client-side assembly means N per-transformation fetches instead of one request. Mitigated by using get_network_results, which fans the ProtocolDAGResult retrieval out across processes, so the dominant cost is parallel rather than serial. Acceptable for an infrequent, inherently-bulk export operation.
  • When an endpoint would earn its keep: a concrete need the client can't meet — e.g. server-mediated archival for clients that cannot hold the full result set. No such need today; if one arises, a streaming/paginated endpoint can be added without disturbing this client API.

Correctness note

Results are joined to Transformations via their authoritative server-side ScopedKeys (from get_network_results / get_network_transformations), then each Transformation is fetched by that ScopedKey. We deliberately do not reconstruct a ScopedKey from a freshly-deserialized Transformation's GufeKey: a ScopedKey carries the ingestion-time gufe_key, but a GufeTokenizable deserialized later may recompute a different GufeKey if gufe tokenization has changed since ingestion — so a key-reconstructed join can silently drop results rather than error.

The docs build failed to import the package: client.py now imports
gufe.archival, absent from the docs env's pinned gufe=1.3.0. Bump it to
1.10.0 to match the runtime pin.

That bump surfaced a latent conf.py issue: gufe>=1.10.0 evaluates
`PositiveFloat | None` at class-definition time in its settings models, which
raises "unsupported operand type(s) for |" when pydantic is mocked via
autodoc_mock_imports. Drop pydantic from the mock list (it is a real,
installed gufe dependency); autodoc then imports gufe cleanly.

Finally, render AlchemicalArchive as an inline literal in getting_started.rst
rather than an intersphinx cross-reference: the gufe inventory is pinned to
v1.2.0, which predates gufe.archival, so the reference cannot resolve.

Verified: `sphinx -W` builds with zero warnings in both the docs.yml env and
the full runtime env.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@dotsdl dotsdl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with this! Thanks for the initial work on this @ianmkenney!

@dotsdl
dotsdl requested review from dotsdl and mikemhenry and removed request for dotsdl July 7, 2026 15:09
Comment thread alchemiscale/interface/client.py
Comment thread CLAUDE.md
@dotsdl
dotsdl merged commit e04181e into main Jul 8, 2026
7 checks passed
@dotsdl
dotsdl deleted the feat/extract_archive branch July 8, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add AlchemiscaleClient method for producing archival-quality extracts of AlchemicalNetworks and their existing results

3 participants