Skip to content

fix: don't lose an upload when an image host is slow on one screenshot - #330

Open
nitrobass24 wants to merge 4 commits into
mainfrom
fix/partial-image-host-failure
Open

fix: don't lose an upload when an image host is slow on one screenshot#330
nitrobass24 wants to merge 4 commits into
mainfrom
fix/partial-image-host-failure

Conversation

@nitrobass24

@nitrobass24 nitrobass24 commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

One image an image host refuses aborts the entire release upload, even when min_successful_image_uploads is satisfied. The workflow dies at review-uploads with upload dry run requires target tracker IDs, which describes an empty tracker set rather than the actual cause.

Observed with screens: 6, min_successful_image_uploads: 3:

image hosting: completed uploads host=reelflix tracker=RF wall_duration=4m12.140067534s mean_attempt_duration=42.023199529s attempts=6 succeeded=5 unsuccessful=1
image hosting: upload batch completed host=reelflix tracker=RF failures=1 successes=5
releaseworkflow: command=composite_upload operation=upload_execute stage=composite_upload state=failed cause=release workflow composite review-uploads: release workflow publish upload dry run: upload dry run requires target tracker IDs

Same failure on the host-fallback path, so it is not specific to one host or to fallback:

image hosting: upload failed ... host=pixhost err=pixhost upload failed with status 413
core: starting image upload fallback failed_hosts=pixhost fallback_hosts=imgbox trackers=[RF]
releaseworkflow: ... cause=release workflow composite review-uploads: release workflow publish upload dry run: upload dry run requires target tracker IDs

The dropped image was a client timeout waiting on the host:

image hosting: upload failed file=Example.Release.2026.2160p-GRP.mkv-02-ss_03569611.png host=<host> tracker=RF err=image hosting: send multipart request to https://<host>/api/1/upload: Post "https://<host>/api/1/upload": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

while awaiting headers means the request body was already sent and the deadline expired waiting on the host to answer, so this is host-side processing time rather than local transfer. The same timeout has been seen on pixhost and imgbox as well, so it is not specific to any one host.

Cause

Three separate defects.

1. min_successful_image_uploads is never read. It is parsed at internal/config/config.go:151 and shipped in internal/config/defaults/example.yaml, but the only other references in the tree are a persistence test and the e2e harness fixture. internal/imagehosting/service.go:707 fails the batch whenever any single image failed, regardless of the configured floor. The successfully published links are returned alongside that error and are still usable, but the error alone drives everything downstream.

2. Any image-hosting failure removes the tracker from the downstream set. internal/releaseworkflow/eligibility.go:317-321, for the descriptions and upload stages:

for trackerID := range base {
    if _, failed := TrackerImageHostFailure(media, trackerID); failed {
        delete(base, trackerID)
    }
}

With a single selected tracker, base empties, resolveDownstreamTrackerSet returns an empty set and a nil error, and the empty TrackerIDs only fails much later at pkg/api/workflow_contracts_validation.go:1283.

The filter does not distinguish a total host outage from a host that refused one image out of six, even though internal/releaseworkflow/module.go:1129-1141 already models that distinction as StageStatusPartial, and TrackerImageHostFailure's doc comment describes it as a terminal failure.

Reproduced against resolveDownstreamTrackerSet with a media snapshot carrying five hosted artifacts and one image-hosting failure for the same tracker:

media-stage tracker IDs = [RF]
upload-stage tracker IDs = []   err = <nil>
dry run validate err = upload dry run requires target tracker IDs

3. The image-host deadline leaves almost no headroom. internal/httpclient/httpclient.go:14 sets a fixed 60s whole-request deadline, and UploadTimeout is one constant shared by every host in the uploader registry. The batch above reported mean_attempt_duration=42.02s across 6 attempts, and since attemptDurations records failed attempts too, the 5 successes averaged around 38s. That is barely 1.5x headroom for an operation whose duration is set by how fast a remote host answers.

imgbox spends that same budget three times per batch: imgboxGetCsrfAndCookie and the token generation each clone the client at UploadTimeout before the upload itself runs, so a slow host burns the ceiling on requests that publish nothing.

Impact

Every host is slower to process a 5-8MB image than a 1080p screenshot, so this shows up as a 2160p problem across hosts rather than a per-host one. In a 25 item batch of mixed content every 1080p release succeeded and three 2160p releases failed this way. Since one dropped image is enough, a complete and otherwise valid upload is lost, and the surfaced error gives no indication that image hosting was involved.

Changes

Four commits, each can be backed out alone.

fix(core): honor min_successful_image_uploads when a host drops uploads

  • internal/config/config.go: ResolvedMinSuccessfulUploads(), matching the existing ResolvedMaxMenuItems() idiom. Zero disables the allowance and keeps the previous strict behavior.
  • internal/core/media.go: partialHostUploadIsUsable accepts a partially failed host batch once the target has published at least the floor. This lives in internal/core deliberately, since it is the only layer that sees reused plus newly published images for a target. internal/imagehosting only receives the missing subset, so a threshold there would under-count reuse and fail a target that already has enough images.
  • internal/config/defaults/example.yaml: comment describing what the knob does.
  • Table test covering above / at / below the floor, allowance disabled, and floor above the requested count. The disabled case pins the previous behavior.

fix(releaseworkflow): name image-host exhaustion instead of an empty tracker set

  • internal/releaseworkflow/eligibility.go: when image hosting is what removed the last downstream tracker, fail with ErrInvalidTransition naming the blocked trackers instead of returning an empty set that only breaks at contract validation. Sets that were already empty for other reasons are untouched.

fix(httpclient): raise image-host upload deadline to 120s

  • internal/httpclient/httpclient.go: 60s to 120s, restoring roughly 3x headroom over the observed host response time. UploadTimeout is consumed only by internal/imagehosting; every tracker upload path uses DefaultTimeout, so no other deadline widens.

test(core): cover repository link reuse in the partial image-upload allowance

  • Two cases pinning why the floor lives in internal/core: a host that publishes fewer images than the floor still succeeds once repository reuse makes up the difference, and reuse still short of the floor fails. Applying the floor in internal/imagehosting, which only receives the missing subset, would fail the first.

Notes for review

  • The 120s value is a judgement call. If you would rather this be configurable than a constant, say so and I will wire it through config instead, though that pulls in schema, defaults, env overrides and settings UI parity.
  • A genuinely hung host now stalls twice as long before erroring. With max_concurrent_uploads: 1 and 6 screenshots that is a 12 minute worst case per release instead of 6. Commit 1 limits the damage, since the release no longer dies from it.
  • Commit 1 changes upload semantics: with 5 of 6 published you now upload with 5 screenshots instead of failing. Tracker-side screenshot requirements are still enforced separately by refreshMutatedMediaStatus via requiredScreenshots, so that gate is intact, but you may prefer the floor to interact with tracker requirements rather than sit beside them.
  • Commit 1 also means no fallback host is attempted once the floor is met. That is intended, since the target already has enough images, but it does change when fallback fires.

Verification

  • make test-go full race suite green
  • make lint 0 issues
  • make logpolicy, make pathpolicy, make gofix-check-changed, git diff --check clean

Summary by CodeRabbit

  • New Features

    • Added configurable minimum-success thresholds for image uploads.
    • Partial upload failures can proceed when the required number of images uploads successfully.
  • Bug Fixes

    • Improved handling when image-host failures block all downstream release targets, with a clear validation error instead of continuing with no targets.
  • Documentation

    • Documented the minimum successful image upload setting, including zero-value behavior.
  • Improvements

    • Increased the image upload timeout to support slower uploads.

min_successful_image_uploads is parsed from config and shipped in the
example config, but nothing ever read it. Any single image an image host
failed to accept failed the whole host batch, which records a
tracker-scoped image-hosting failure and removes that tracker from the
downstream set. With one selected tracker that empties the upload plan.

Image hosts drop individual uploads under concurrency, and larger
screenshots are dropped far more often, so a 2160p release could lose an
otherwise complete upload to one refused image.

Resolve the configured floor in internal/core, which is the only layer
that sees reused plus newly published images for a target, and accept a
partially failed host batch once the target has published at least that
many images. Zero keeps the previous strict behavior, and a requested
count below the floor cannot clear it.
…tracker set

When image hosting fails for every downstream tracker,
resolveDownstreamTrackerSet returned an empty set with a nil error. That
set travels into the upload plan and only surfaces at contract
validation as "upload dry run requires target tracker IDs", which names
neither the stage that emptied it nor the cause.

Fail with ErrInvalidTransition naming the blocked trackers when image
hosting is what removed the last one. Sets that were already empty for
other reasons are untouched.
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The change adds a configurable minimum for successful image uploads. Qualifying partial uploads continue as successful. Upload failures that block every downstream tracker now return an invalid transition error listing the affected trackers. The upload timeout increases to 120 seconds.

Changes

Upload handling and release eligibility

Layer / File(s) Summary
Partial upload policy and validation
internal/config/config.go, internal/config/defaults/example.yaml, internal/httpclient/httpclient.go, internal/core/media.go, internal/core/media_test.go
The configuration resolves non-positive thresholds to zero. Direct and repository-reuse uploads accept partial results when the published count meets the positive threshold. Tests cover accepted and rejected thresholds. The upload timeout is 120 seconds.
Blocked tracker transition handling
internal/releaseworkflow/eligibility.go, internal/releaseworkflow/eligibility_test.go
Downstream resolution records trackers removed by image-host failures. When all trackers are blocked, it returns ErrInvalidTransition with sorted tracker IDs.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UploadFlow
  participant ImageHostingService
  participant ReleaseWorkflow
  UploadFlow->>ImageHostingService: upload requested images
  ImageHostingService-->>UploadFlow: published links and partial error
  UploadFlow->>UploadFlow: validate minimum successful count
  UploadFlow->>ReleaseWorkflow: provide upload result
  ReleaseWorkflow->>ReleaseWorkflow: remove trackers with image-host failures
  ReleaseWorkflow-->>UploadFlow: return invalid transition if all trackers are blocked
Loading

Suggested reviewers: audionut

Poem

A rabbit counts the images bright,
Keeps enough successes in sight.
If trackers all become blocked,
A clear transition error is logged.
Hop, hop—the flow is right!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: preserving uploads when an image host is slow or partially fails.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/partial-image-host-failure

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/core/media_test.go`:
- Around line 453-461: Extend the table-driven test around the mediaModule setup
to exercise repository link reuse before a partial upload error: configure a
test case with reusable links and newly published links, ensure their combined
count satisfies ScreenshotHandlingConfig.MinSuccessfulUploads, and assert the
operation succeeds. Keep the existing direct-upload cases unchanged and retain
coverage for the repository-reuse partial-upload path.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 95dd9d89-eb6c-4e52-ab3e-df163913f4d1

📥 Commits

Reviewing files that changed from the base of the PR and between 47bdb91 and d4ded9d.

📒 Files selected for processing (6)
  • internal/config/config.go
  • internal/config/defaults/example.yaml
  • internal/core/media.go
  • internal/core/media_test.go
  • internal/releaseworkflow/eligibility.go
  • internal/releaseworkflow/eligibility_test.go

Comment thread internal/core/media_test.go
@nitrobass24 nitrobass24 changed the title fix: don't lose an upload when an image host refuses one screenshot fix: don't lose an upload when an image host is slow on one screenshot Aug 8, 2026
UploadTimeout is one constant shared by every image host in the uploader
registry, and the 60s whole-request deadline was too tight for all of
them. The same timeout failure was seen on pixhost and imgbox before
reelflix hosting was configured, so this is not one host being slow.

The deadline is not a transfer budget. The failure looks like:

  Post "https://<host>/api/1/upload": context deadline exceeded
  (Client.Timeout exceeded while awaiting headers)

"while awaiting headers" means the multipart body was already sent and
the deadline expired waiting on the host to answer, so what the ceiling
has to cover is host-side processing of a 5-8MB image. Every host is
slower at that than at a 1080p screenshot, which is why this shows up as
a 2160p problem across hosts rather than a per-host one.

imgbox spends the same budget three times per batch: its CSRF fetch and
token generation each clone the client at UploadTimeout before the upload
itself runs, so a slow host burns the ceiling on requests that publish
nothing.

The batch with recorded timings reported mean_attempt_duration=42.02s
across 6 attempts, and since attemptDurations records failed attempts
too, the 5 successes averaged around 38s. That is barely 1.5x headroom
for an operation whose duration is set by how fast a remote host answers.
120s restores roughly 3x.

UploadTimeout is consumed only by internal/imagehosting; every tracker
upload path uses DefaultTimeout, so no other deadline widens.
…llowance

The floor lives in internal/core because that is the only layer that sees
reused plus newly published images for a target. Two cases pin that: a
host that publishes fewer images than the floor still succeeds once reuse
makes up the difference, and reuse that is still short of the floor
fails. Applying the floor in internal/imagehosting, which only receives
the missing subset, would fail the first case.
@nitrobass24
nitrobass24 force-pushed the fix/partial-image-host-failure branch from fb22fbd to 92df420 Compare August 8, 2026 04:19
@nitrobass24
nitrobass24 requested a review from Audionut August 8, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant