[bug] Unexpected Recipe Revision Resolution with Multiple Remotes in Conan 2

[jasal82]

Describe the bug

Description
When using multiple remotes in Conan 2 with --update enabled, the resolution mechanism for recipe revisions (rrevs) appears to deviate from the behavior in Conan 1. Specifically, it does not respect the order of remotes as defined in remotes.json. Instead, recipe revisions are sorted by their timestamps across all remotes, which can lead to unexpected behavior and Conan1-incompatible package resolution.

Steps to Reproduce

1. Configure two remotes:
   • Remote 1: Contains recipe revisions with older timestamps.
   • Remote 2: Contains the same recipe revisions re-uploaded, resulting in newer timestamps.
2. Require a package with the same revision distributed across these two remotes but with different timestamps.
3. Enable the --update flag while resolving the package.

Expected Behavior
Conan should prioritize recipe revisions from the first remote in remotes.json, regardless of timestamp differences. For example:

• If pkg@1.0.0#bcd exists in both remotes with different timestamps, it should be resolved from Remote 1 if it appears first in remotes.json.

Actual Behavior
With --update enabled, Conan 2 queries recipe revisions from all remotes and sorts them by timestamp. This results in the recipe from Remote 2 being chosen, even when it has the same content as the revision in Remote 1 but a newer timestamp.

Code Reference
The issue seems related to the _find_newest_recipe_in_remotes function in proxy.py.

Example
Given the following remote distribution:

- Remote 1:
  - pkg@1.0.0#abc @ 2025/01/01
  - pkg@1.0.0#bcd @ 2025/01/02
- Remote 2:
  - pkg@1.0.0#abc @ 2025/01/05

With the current behavior:

• pkg@1.0.0#abc from Remote 2 is resolved as the latest revision due to the newer timestamp.
  Expected behavior:
• pkg@1.0.0#bcd from Remote 1 should be resolved as the latest revision since it is the most recent recipe that appears first in remotes.json.

Use case
We have a cache remote which is used to upload binary packages for recipes that were built by CI. It is different from the remote where the recipes are stored, and this is intentional (because the release repository does not enable delete permissions for immutability, but cache remote needs delete permissions for periodic cleanup tasks). When uploading a binary package the recipe entry is automatically created in the cache remote and gets the timestamp of creation instead of the original recipe timestamp. Both remotes are used when building consumers, with the recipe remote having higher priority.

Proposed solution
This is a tough one. I don't really have an idea how to solve this properly. Maybe the Conan 1 way of resolving across remotes could be restored, but that would probably break existing Conan 2 solutions. Or maybe it would be possible to store the original timestamp of downloaded recipes in the local cache so that it can be preserved when uploading the recipe again to another remote (this seems to be the only viable solution in my opinion).

Environment
Conan Version: 2.x

[memsharded]

Hi @jasal82

Thanks for your report.

In principle, this is intentional and by design in Conan 2. The --update behavior in conan 1 got some criticisms for not being that clear, so the rule in Conan 2 is much simpler: "it will fetch the latest in the provided remotes". If some server got a latest timestamp, this is the one expected to be downloaded, and this seems to fullfill very much the user expectations so far, it has been working and understood quite well.

But there might be something that I am missing, because Conan 2 does not track the remote servers origins for packages. So the fact that the same recipe revision is downloaded from one remote or the other should not have any effect, as the final result in the Conan cache will be identical. But specially, because the package binary resolution is not coupled anymore in Conan2 to the recipe previous resolution, so Conan should be able to successfully get your binaries from the other server.

We have a cache remote which is used to upload binary packages for recipes that were built by CI. It is different from the remote where the recipes are stored, and this is intentional (because the release repository does not enable delete permissions for immutability, but cache remote needs delete permissions for periodic cleanup tasks). When uploading a binary package the recipe entry is automatically created in the cache remote and gets the timestamp of creation instead of the original recipe timestamp. Both remotes are used when building consumers, with the recipe remote having higher priority.

This is what I don't fully understand. Conan 2 does a second iteration over remotes, independent of the recipe iteration one, looking for the binary, so it should be able to find it. It is not very clear what is failing to resolve or how it fails in the process, if you could please clarify that. Thanks!

[jasal82]

Hi @memsharded , the problem is not related to the binary resolution, it's the recipe resolution which breaks. And that is because the timestamp is relevant for determining the latest revision. We're not downloading the same revision from one or the other remote, we're getting a completely different revision as the latest one because the timestamps are different between the two repositories.

[jasal82]

As I wrote above, if I download a specific recipe revision from one remote and upload it to another, the uploaded one has a different timestamp. As long as there is only that single recipe revision it doesn't matter, because both have the same contents. But if there are other recipe revisions and we're requiring only based on the version, the latest revision is determined via the timestamps and these might have a different order across the revisions in different remotes.

[jasal82]

The cause of our problem is the new simple rule "it will fetch the latest in the provided remotes". This makes the assumption that recipes with a more recent timestamp belong to a more recent revision of the recipe which is generally a wrong assumption in my opinion. The assumption is only correct if any given recipe revision with identical content is only uploaded exactly once and only to a single remote OR if their timestamps are synchronized across all remotes. With a single remote it would even be safe to reupload the exact same revision again because Conan would be able to check that the content of the recipe to upload are the same as the one already in the remote and reject the upload. But if we're using multiple remotes this does not work anymore because I can download an older revision (which is not the "latest") from remote 1 and upload it to another remote which does not yet contain that revision. This will discard the original timestamp from remote 1 and assign a new timestamp. When we now add this second remote the revision in remote 2 will actually be considered the latest one because it has the latest timestamp, but it's actually only a copy of an existing older revision in remote 1.

[jasal82]

@memsharded Do you need any additional information? This is a blocker for us as it completely breaks our binary package caching strategy.

[jasal82]

Which timestamp is used for determining the "newest" recipe on the remote in the search algorithm? It seems that Conan 2 uses the timestamp in the index.json on the remote (respectively the one in the .timestamp file, which is generated by the server during the upload). Shouldn't it be using the timestamp from the conanmanifest.txt? That one is preserved across a download/upload cycle as far as I can see.

[jasal82]

Ok, I checked the REST calls. The latest revision is determined on server side and queried using the v2/conans/<pkg>/<version>/<user>/<channel>/latest API. That one uses the timestamp from the index.json which is created by the server and is always equal to the upload time. However, it should use the timestamp in the conanmanifest.txt instead. Can you confirm that this is a bug?

[jasal82]

Possibly related to #14068

[jasal82]

Created a PR which addresses the immediate problems we're facing due to this issue. However, this is not a fix for the issue itself, because I think it is a fundamental design flaw which requires more than just a workaround.

[memsharded]

Hi @jasal82

Sorry for the delay, we have been investigating and discussing this.

The discussed behavior is:

• Remote R1 contains pkg/0.1#rev1%timestamp1
• Remote R2 contains pkg/0.1#rev2%timestamp2
• Remote R3 contains pkg/0.1#rev1%timestamp3
• timestamp3 > timestamp2 > timestamp1 (timestamp3 is the newest)
• A conan install --requires=pkg/0.1 --update will by default in Conan 2 resolve to pkg/0.1#rev1%timestamp3 in R3

It is reported as a bug that Conan should be resolving instead to pkg/0.1#rev2%timestamp2 in remote R2.

It seems the current default behavior wouldn't be a design flaw, but designed this way. This would be the rationale:

• It is a very simple behavior to explain and understand: "pick the revision from all the remotes with the newest timestamp". This is a very important point, the more difficult to explain and understand some behavior, the more problems it causes, the more support it needs. The full team aligns on the importance of this point and than the current default behavior makes sense, and that the suggested behavior will be more difficult to explain and understand and cause confusion among many users.
• There are already many users that have finalized the transition to Conan 2, and this has not been an issue, but rather the opposite, not a single problem so far. Furthermore, we have been asking for feedback to several different Conan users, specially those who have large or very large deployments and use multiple remotes in resolution in the last two weeks. The unanimous feedback from them points in the direction that the current behavior is the most expected one.
• Conan is decentralized by nature, which means that uploading some revision to a new server will create a newer timestamp, there is no way it could recover the existing timestamp from other repository at the time of upload. But, it is perfectly possible, and in fact the recommended way, to maintain the original timestamps. This is the process described in https://docs.conan.io/2/devops/package_promotions.html, which is running promotions. If instead of re-uploading the same revision pkg/0.1#rev1 that already exists in R1 to another server R3, the recommended way is to run a promotion from R1->R3. The test in Add art:promote tests for timestamps conan-extensions#173 proves that promotions respect the timestamps, so it will define pkg/0.1#rev1%timestamp1 in remote R3 and conan install --requires=pkg/0.1 --update will resolve to rev2 in remote R2
• Note the recommended approach of promotions also implies cost savings, as it is necessary less cpu and network transfer, saving cost and time in CI.

Said this, we are not opposed to explore other --update resolution strategies whenever it might make sense. I am already experimenting trying to implement the resolution that will result in pkg/0.1#rev2%timestamp2 in remote R2 even if the timestamp of R3 is the latest one, but it would be an explicit opt-in strategy.

[jasal82]

Hi @memsharded

thanks for the detailed answer and for your willingness to provide a remedy for our use case. I just want to add a few comments:

It is a very simple behavior to explain and understand: "pick the revision from all the remotes with the newest timestamp". This is a very important point, the more difficult to explain and understand some behavior, the more problems it causes, the more support it needs. The full team aligns on the importance of this point and than the current default behavior makes sense, and that the suggested behavior will be more difficult to explain and understand and cause confusion among many users.

I can understand this decision in principle, but the fact that the remotes are decentralized, in combination with the timestamp assignment by the servers, could lead to problems even with two servers with different system clocks. I realize that the existing mechanism can no longer be changed at this point - and that's not what I'm suggesting, just to be clear -, but in my opinion only the creator of the recipe can decide which revision is newer.

Conan is decentralized by nature, which means that uploading some revision to a new server will create a newer timestamp, there is no way it could recover the existing timestamp from other repository at the time of upload. But, it is perfectly possible, and in fact the recommended way, to maintain the original timestamps. This is the process described in https://docs.conan.io/2/devops/package_promotions.html, which is running promotions.

If revisions were only ever duplicated via promotion, everything would be fine, agreed. We're actually using this as a workaround now by "priming" the cache via promotion of the recipes before uploading the binary packages at the end of a build. But this doesn't take into account the fact that recipes can be downloaded and uploaded again across repositories, even unintentionally.

If nobody else has problems with this behavior, then it's ok for us. Your suggested opt-in switch would solve our current problems, but we may have to redesign the caching mechanism anyway under these circumstances.

[jasal82]

Here is a diagram that shows how this issue can manifest itself even with just two remotes. You have to follow the time axis from t1 to t4:

An important fact is that our component CI jobs never upload binaries, they just upload recipes. We do this because the set of configurations that a component is built for in its own CI is usually completely disjoint from the set of configurations that the consumers build the component for (mainly due to differences in the transitive requirement graphs and our use of full_mode for package ID calculation).

Binaries are uploaded by the consumer builds. This maximizes the probability, that the built binaries are gonna be re-used in other subsequent builds. The binaries go to a different remote (for several reasons, mainly permission management and cleanup considerations). This creates duplicates of the recipes (because the binary remote initially does not know about the original recipes) with new timestamps.