Skip to content

fix(youtube,tiktok): request identity encoding on the media reads - #1853

Merged
nevo-david merged 1 commit into
mainfrom
fix/youtube-chunk-transient-errors
Aug 7, 2026
Merged

fix(youtube,tiktok): request identity encoding on the media reads#1853
nevo-david merged 1 commit into
mainfrom
fix/youtube-chunk-transient-errors

Conversation

@giladresisi

@giladresisi giladresisi commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

What kind of change does this PR introduce?

Bug fix.

Why was this change needed?

The customer report

A customer reported (Discord) that ~1 GB YouTube uploads through Postiz fail often enough that they built a heartbeat agent just to check whether their posts actually went out. Their own investigation was accurate: the same hosted files alternated between passing and failing without being changed, an immediately-failed post succeeded a minute later after restarting the same row, and larger files fail more often without there being a fixed size limit.

Root cause

YoutubeProvider.youtubeMediaSize / youtubeChunkStream and TiktokProvider.tiktokChunkStream read the stored video back from media storage with their own HEAD and ranged-GET helpers, and none of them asked for identity encoding. Every other media read in the project does — #1835 added accept-encoding: identity to SocialAbstract.mediaSize / mediaChunk / mediaStream and to the providers that read media themselves.

That header matters here for two reasons, both documented by Cloudflare:

A 200-with-full-body is currently thrown as BadBody, which postWorkflowV106 treats as a permanent platform rejection: the post is marked ERROR immediately with no retry. A 1 GB video needs ~128 ranged reads, so one such answer kills the whole post even though the YouTube upload session itself is still perfectly resumable. That matches the reported symptom, including why it scales with file size and why an identical file behaves differently between attempts.

Why the header was missing here

This looks like an artifact of PR merge ordering rather than a decision. #1813 introduced these bespoke YouTube/TikTok helpers on 2026-08-03. #1835 swept identity encoding across the project on 2026-08-04, but it had been written against a tree that did not yet contain them — so X and LinkedIn (which had their own bespoke ranged reads at the time, from #1796 and #1786) were consolidated onto the shared helpers and fixed, while YouTube and TikTok were missed. SocialAbstract and every provider that reads media through it has had the header since; these three call sites were the only ones left without it.

Other information:

  • Scope: this PR only adds the header. Behaviour on a non-206 response is deliberately unchanged, so the fix is a prevention rather than a behaviour change, and its effect can be observed on its own.
  • Please monitor after deploy. The customer symptom (large YouTube uploads intermittently going to ERROR with "The media storage did not return the requested byte range") should stop. If it recurs, the follow-up is to stop treating that specific answer as permanent: a documented 200-with-full-body would be thrown as a plain Error instead of BadBody, which the v1.0.6 workflow already knows how to handle — it re-probes the upload session and resumes from the exact committed byte offset, with a bounded consecutive-failure budget. That behaviour was prototyped and verified end-to-end against a storage server injecting Cloudflare-style 200s: pre-change the post died at the third chunk; with the reclassification the same upload survived five injected failures, resuming byte-exactly each time, and published in 98 s. It is being held back deliberately in favour of trying the smaller, in-pattern fix first.
  • Suggested follow-ups (not included here):
    • Delete youtubeMediaSize and call SocialAbstract.mediaSize instead, as TikTok already does. The generic helper is a strict superset: it also rejects a non-ok HEAD and a zero/NaN length (a failed HEAD can still carry the Content-Length of its error body, which would poison the chunk-count math downstream).
    • Consolidate youtubeChunkStream / tiktokChunkStream into a single mediaChunkStream on SocialAbstract, next to mediaChunk (ranged, buffered) and mediaStream (whole file, streamed). The ranged-stream shape is the one combination the abstract does not offer today, which is why both providers hand-rolled it; a shared helper would have been swept automatically by fix: stream provider media uploads instead of buffering files in memory #1835 and would keep the non-206 classification in one place. Note the two callers do not want identical error semantics — YouTube can probe and resume its session, TikTok cannot — so the shared helper should throw one error type and let each caller classify it.

Checklist:

Put a "X" in the boxes below to indicate you have followed the checklist;

  • I have read the CONTRIBUTING guide.
  • I have signed the Contributor License Agreement (CLA) (ICLA for individuals, CCLA for entities).
  • I confirm I have not used AI to submit this PR or generate code for it.
  • I checked that there were no similar issues or PRs already open for this.
  • This PR fixes just ONE issue

🤖 Generated with Claude Code

@postiz-contribution postiz-contribution Bot added the contribution:approved Approved contributor label Aug 7, 2026
@postiz-agent

postiz-agent Bot commented Aug 7, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@giladresisi
giladresisi force-pushed the fix/youtube-chunk-transient-errors branch from 4ba5359 to 8b7cb0b Compare August 7, 2026 14:44
@giladresisi giladresisi changed the title fix(youtube): treat storage read failures as transient, not permanent fix(youtube): treat a non-206 media range read as transient, not permanent Aug 7, 2026
The YouTube and TikTok chunked uploaders read the stored video back from
media storage with their own HEAD/ranged GET helpers, and those helpers
never asked for identity encoding. Every other media read in the project
does: #1835 added `accept-encoding: identity` to SocialAbstract's
mediaSize / mediaChunk / mediaStream and to the providers that read media
themselves, because a transformed (compressed) response loses its
Content-Length - and Cloudflare answers a range request on an object with
no Content-Length by returning the full body with a 200 instead of the
requested 206.

That is very likely what has been failing ~1GB YouTube uploads: a single
200 out of the ~128 ranged reads a 1GB video needs is thrown as BadBody,
which the v1.0.6 workflow treats as a permanent platform rejection, so
the whole post is marked ERROR even though the upload session is still
resumable.

The gap looks like an artifact of merge ordering rather than a decision:
#1813 introduced these two bespoke helpers on Aug 3, and #1835 swept
identity encoding across the project on Aug 4 - by then the sweep had
been written against a tree that did not contain them, so YouTube and
TikTok were missed while X and LinkedIn (bespoke at the time too) were
consolidated onto the shared helpers and fixed.

Behaviour on a non-206 is deliberately left untouched: if the customer
report recurs after this ships, the next step is to reconsider treating
the documented 200-with-full-body answer as transient (a plain Error the
workflow retries and resumes from the committed byte offset) instead of
as a permanent failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@giladresisi
giladresisi force-pushed the fix/youtube-chunk-transient-errors branch from 8b7cb0b to 5a9b1cc Compare August 7, 2026 15:54
@giladresisi giladresisi changed the title fix(youtube): treat a non-206 media range read as transient, not permanent fix(youtube,tiktok): request identity encoding on the media reads Aug 7, 2026
@nevo-david
nevo-david added this pull request to the merge queue Aug 7, 2026
Merged via the queue into main with commit d1765d4 Aug 7, 2026
11 checks passed
@nevo-david
nevo-david deleted the fix/youtube-chunk-transient-errors branch August 7, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution:approved Approved contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants