Reduce mercurial work in the tool shed's tool source and install info paths

[mvdbeek]

Reduces the mercurial work the tool shed does when serving tool sources and install info. Two independent changes.

Materialize tool revisions locally instead of cloning over HTTP

_shed_tool_source_for cloned the tool shed's own repository over HTTP, through the shed's own hgweb front end, into a temporary directory — just to read a single tool XML. It did this while already holding the local repository handle and the changectx two lines above:

repo = repository_metadata.repository.hg_repo
ctx = get_changectx_for_changeset(repo, repository_metadata.changeset_revision)
work_dir = tempfile.mkdtemp(prefix="tmp-toolshed-tool_source")
cloned_ok, error_message = clone_repository(repository_clone_url, work_dir, str(ctx.rev()))

That's a network round trip plus a full changelog transfer per call. It now does a local hg archive of the target revision via archive_repository_revision, which was already in tool_shed/util/hg_util.py but had no callers.

The whole revision is still materialized rather than just the tool file, because tool XML may import macros from sibling files — a single-file extraction would silently break every macro-based wrapper. There's a new column_maker_with_macros fixture and test covering that.

repository_clone_url was only ever needed to feed the clone, so it's dropped from the manager functions that threaded it through.

Worth noting for reviewers: parsed_tool_model_cached_for caches the parsed model, but GET /api/tools/{tool_id}/versions/{tool_version}/tool_source calls tool_source_for directly and bypasses that cache, so it paid the clone on every request.

Resolve changeset revisions without scanning the changelog

get_changectx_for_changeset walked the entire changelog, materializing a changectx and hex-formatting it for every revision, to find one hash:

for changeset in repo.changelog:
    ctx = repo[changeset]
    if str(ctx) == changeset_revision:
        return ctx

It now resolves the hex node id prefix directly. Note that repo[changeset_revision] cannot be used here — it resolves revision symbols, so an all-digit hash such as 123456789012 would be read as a revision number. The null node and short prefixes are rejected so the result matches what the changelog scan matched. There are 8 call sites.

Two consequences on the get_install_info path:

• get_repo_info_dict no longer forks hg id via changeset2rev to compute ctx_rev.
• get_next_downloadable_changeset_revision fetched the downloadable revisions from the database and then walked the changelog again to recover ordering it had already asked for. It now orders them by changelog position directly. This one is called once per transitive dependency during RelationBuilder traversal.

A trap worth flagging

RepositoryMetadata.numeric_revision looks like the obvious source for ctx_rev — it's indexed, already populated, and means exactly that. It is not safe to use. repository_metadata_manager.py:1140 moves changeset_revision ahead to a new repository tip without updating numeric_revision, so the two can disagree. The column_maker_with_download_gaps fixture has a row with numeric_revision=1 whose real changelog position is 2. Since Galaxy feeds ctx_rev straight into hg clone -r, trusting that column would install the wrong revision.

This also means get_metadata_revisions sorts by a column that can be stale, which propagates to get_latest_downloadable_changeset_revision. Not addressed here — fixing it properly needs a data backfill, not just a write at the move-ahead site. Happy to open a separate issue.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.

New test/unit/tool_shed/test_hg_revision_lookup.py covers changeset resolution across both fixtures (including null-hash, short-prefix and unknown-hash rejection), next-downloadable ordering over the gap fixture, and that get_repo_info_dict's ctx_rev equals the real mercurial revision — that last one is the regression guard for the numeric_revision trap above.

test/unit/tool_shed/test_tool_source.py gains a macro-expansion case.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.