fix: stream provider media uploads instead of buffering files in memory - #1835
Conversation
Workers were going OOM because social providers downloaded entire media files (arrayBuffer/Blob) into memory before uploading them. This moves all providers to streamed or chunked uploads: - social.abstract.ts: shared mediaSize (HEAD/stat), mediaChunk (ranged reads) and runStreamedUpload (retry/handleErrors classification for non-replayable streamed bodies) helpers - linkedin/x: chunked video uploads read one 1-2MB ranged part at a time; LinkedIn media now uploads sequentially instead of Promise.all - bluesky/skool/whop: size from HEAD, body streamed with duplex: 'half' - discord/mastodon/reddit/tumblr: multipart forms stream attachments with known lengths via form-data, rebuilt per retry attempt - tiktok: reuse the shared mediaSize helper All user-influenced fetches keep the SSRF-safe dispatcher. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| const { data: stream } = await axios.get(item.path, { | ||
| responseType: 'stream', | ||
| }); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Pull request overview
This PR addresses worker OOMs by refactoring multiple social-provider media upload paths to avoid buffering entire media files in memory, primarily by streaming uploads and/or uploading chunk-by-chunk.
Changes:
- Added shared streaming/chunking helpers to
SocialAbstract(mediaSize,mediaChunk,runStreamedUpload) and migrated providers to use them. - Updated chunked-upload providers (e.g., X, LinkedIn, TikTok) to read ranged chunks on-demand rather than loading full videos.
- Updated multipart/PUT upload providers (e.g., Discord, Mastodon, Reddit, Tumblr, Skool, Whop, Bluesky) to stream upload bodies and rebuild requests on retry where needed.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| libraries/nestjs-libraries/src/integrations/social.abstract.ts | Introduces shared helpers for streaming/chunked uploads and retry/error classification for one-shot streams. |
| libraries/nestjs-libraries/src/integrations/social/x.provider.ts | Switches X video upload to ranged-chunk reads and sequential media uploads to reduce memory pressure. |
| libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts | Streams LinkedIn video uploads in 2MB chunks and uploads media sequentially. |
| libraries/nestjs-libraries/src/integrations/social/tiktok.provider.ts | Replaces provider-local size logic with shared mediaSize. |
| libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts | Streams video upload bytes and uses HEAD-derived content-length instead of buffering video. |
| libraries/nestjs-libraries/src/integrations/social/discord.provider.ts | Streams attachments into multipart form and retries by rebuilding the request. |
| libraries/nestjs-libraries/src/integrations/social/mastodon.provider.ts | Streams media bytes into multipart form and rebuilds on retry to avoid replaying consumed streams. |
| libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts | Streams the upload file into Reddit’s S3 form with known length rather than buffering. |
| libraries/nestjs-libraries/src/integrations/social/tumblr.provider.ts | Streams media parts into multipart upload and rebuilds on retry to avoid replaying streams. |
| libraries/nestjs-libraries/src/integrations/social/whop.provider.ts | Streams PUT uploads and derives sizes via HEAD to avoid holding media in memory during long polling. |
| libraries/nestjs-libraries/src/integrations/social/skool.provider.ts | Streams PUT uploads and derives sizes via HEAD to avoid buffering media. |
Suppressed comments (1)
libraries/nestjs-libraries/src/integrations/social/discord.provider.ts:170
- Use
this.fetch(SSRF-safe dispatcher + consistent retry/error semantics) to stream attachment bytes instead ofaxios.get, which may follow redirects to internal addresses and isn’t protected bygetSsrfSafeDispatcher.
const fileSize = await this.mediaSize(item.path, this.identifier);
const { data: stream } = await axios.get(item.path, {
responseType: 'stream',
});
| { | ||
| segment_index: i / chunkSize, | ||
| media: await this.videoChunk(path, i, end), | ||
| segment_index: i, | ||
| media: await this.mediaChunk(path, start, end), | ||
| }, |
| await fetch(createFileResponse.upload_url, { | ||
| method: 'PUT', | ||
| headers: createFileResponse.upload_headers || {}, | ||
| body: fileBuffer, | ||
| }); | ||
| headers: { | ||
| // Whop's own upload_headers win if they ever include a length. | ||
| 'Content-Length': String(contentLength), |
| await fetch(createFileResponse.write_url, { | ||
| method: 'PUT', | ||
| headers: { | ||
| 'Content-Type': createFileResponse.content_type, | ||
| 'Content-Length': String(contentLength), |
| import axios from 'axios'; | ||
| import FormDataUpload from 'form-data'; |
|
@CodeRabbit review |
✅ Action performedReview finished.
|
|
Note
|
| Layer / File(s) | Summary |
|---|---|
Shared media access and retry primitives libraries/nestjs-libraries/src/integrations/social.abstract.ts, libraries/nestjs-libraries/src/integrations/social/tiktok.provider.ts |
SocialAbstract provides media size detection, ranged reads, media streams, and retryable streamed uploads. TikTok uses the shared size helper. |
Direct streaming provider uploads libraries/nestjs-libraries/src/integrations/social/{bluesky,skool,whop}.provider.ts |
Bluesky, Skool, and Whop validate media metadata and stream media with explicit content lengths. |
Multipart streamed provider uploads libraries/nestjs-libraries/src/integrations/social/{discord,mastodon,reddit,tumblr}.provider.ts |
Multipart uploads stream source media and rebuild forms for retry attempts. |
Chunked and sequential media workflows libraries/nestjs-libraries/src/integrations/social/{linkedin,x}.provider.ts |
LinkedIn and X process bounded chunks. Media processing runs sequentially where specified. X polls processing for up to 30 minutes. |
Estimated code review effort: 4 (Complex) | ~60 minutes
Sequence Diagram(s)
sequenceDiagram
participant Provider
participant SocialAbstract
participant MediaSource
participant PlatformUpload
Provider->>SocialAbstract: request size or stream
SocialAbstract->>MediaSource: perform SSRF-safe HEAD, range, or GET
MediaSource-->>SocialAbstract: metadata or readable stream
SocialAbstract->>PlatformUpload: send streamed media
PlatformUpload-->>SocialAbstract: upload response
SocialAbstract-->>Provider: success or classified error
Possibly related PRs
- gitroomhq/postiz-app#481: Modifies LinkedIn media upload and post handling.
- gitroomhq/postiz-app#654: Modifies TikTok media upload handling.
- gitroomhq/postiz-app#861: Modifies shared social upload and error handling.
Suggested reviewers: copilot
Poem
A rabbit sends streams through the night,
With bounded chunks measured right.
Forms rebuild after a retry,
Safe requests pass bytes nearby.
Hop! Each upload takes flight.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title clearly and concisely describes the main change: streaming provider media uploads to avoid buffering files in memory. |
| Docstring Coverage | ✅ Passed | No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. |
| Linked Issues check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
📝 Generate docstrings
- Create stacked PR
- Commit on current branch
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
fix/stream-provider-media-uploads
Warning
There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.
🔧 ESLint
If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.
libraries/nestjs-libraries/src/integrations/social.abstract.ts
ESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox.
libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts
ESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox.
libraries/nestjs-libraries/src/integrations/social/x.provider.ts
ESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts (1)
95-121: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winCheck the response body and the upload status before you poll the job.
Line 99 checks only
videoResponse.ok, sovideoResponse.bodycan still be null when it is used as the request body on line 118.skool.provider.ts:256andwhop.provider.ts:276guard both. Line 111 also ignores the upload status: if XuploadResponsereturns an error payload,jobStatus.jobIdisundefinedand the loop callsgetJobStatus({ jobId: undefined }), which hides the real cause.🛡️ Proposed guards
const videoResponse = await fetch(videoPath, { // `@ts-ignore` - undici-only option; blocks SSRF to internal IPs dispatcher: getSsrfSafeDispatcher(), }); - if (!videoResponse.ok) { + if (!videoResponse.ok || !videoResponse.body) { throw new Error(`Failed to fetch video: ${videoResponse.statusText}`); }const jobStatus = (await uploadResponse.json()) as AppBskyVideoDefs.JobStatus; + if (!jobStatus?.jobId && !jobStatus?.blob) { + throw new BadBody( + 'bluesky', + JSON.stringify(jobStatus ?? {}), + {} as any, + 'Bluesky rejected the video upload' + ); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts` around lines 95 - 121, In the video upload flow, update the guards after the initial fetch and upload request to require a non-null videoResponse.body before using it as the streaming request body, and verify uploadResponse.ok before parsing or polling the job. Throw descriptive errors containing the relevant response status information so getJobStatus is never called with an undefined job ID; follow the equivalent checks in skool.provider.ts and whop.provider.ts.
🧹 Nitpick comments (1)
libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts (1)
196-208: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winWrap the axios upload in
runStreamedUploadand guard the<Location>match.The upload no longer goes through
this.fetch, sohandleErrorsclassification and the 429/500 retry no longer apply to Reddit media uploads.discord.provider.ts:149andtumblr.provider.ts:517wrap the equivalent axios call inrunStreamedUpload, which rebuilds the form per attempt. Reddit setsmaxConcurrentJob = 1because of strict rate limits, so 429 handling matters on this path.Line 206 also indexes
matchAll(...)[0][1]directly. If the response carries no<Location>element, this throws aTypeErrorinstead of a classified error.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts` around lines 196 - 208, The Reddit media upload in the surrounding provider method must use runStreamedUpload for the axios request so retries and handleErrors classification apply; rebuild the upload form within each attempt as done by the equivalent Discord and Tumblr flows. Replace the direct matchAll index access with a guarded <Location> match, and propagate a classified error when no location is returned.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@libraries/nestjs-libraries/src/integrations/social.abstract.ts`:
- Around line 177-202: Update mediaSize and the related mediaChunk filesystem
access to resolve non-HTTP media paths through process.env.UPLOAD_DIRECTORY,
matching the existing getImageDimensions and updateMedia behavior. Preserve
direct handling for HTTP URLs and ensure statSync and createReadStream receive
the resolved filesystem path rather than the raw provider path.
In `@libraries/nestjs-libraries/src/integrations/social/discord.provider.ts`:
- Around line 166-176: Replace the axios streaming downloads with a shared
SSRF-safe SocialAbstract.mediaStream helper that uses the safe dispatcher and
returns a Node-readable stream, while preserving local-file handling and upload
error behavior. Add the helper next to mediaSize and mediaChunk, then use it in
discord.provider.ts lines 166-176, reddit.provider.ts lines 184-186, and
tumblr.provider.ts lines 527-529, retaining each upload part and knownLength
behavior.
In `@libraries/nestjs-libraries/src/integrations/social/skool.provider.ts`:
- Around line 259-269: Update the streamed PUT handling near the file-upload
flow to retain the fetch response and validate its success status before pushing
createFileResponse.file.id. If the PUT is unsuccessful, stop the attachment flow
and propagate or handle the upload failure so an empty attachment cannot be
published; only append the file ID after a successful response.
In `@libraries/nestjs-libraries/src/integrations/social/whop.provider.ts`:
- Around line 279-289: Update the streamed PUT in the upload flow around
createFileResponse.upload_url to retain the fetch response and immediately
validate that its status is 2xx. On any non-2xx response, fail the operation
with an appropriate error instead of continuing to the status-poll loop.
In `@libraries/nestjs-libraries/src/integrations/social/x.provider.ts`:
- Around line 459-494: Update the polling request in the video-processing loop
after `finalize` to use X’s v2 `/2/media/upload/status` endpoint with `media_id`
as the request parameter. Keep the existing `processing_info` response handling
and retry logic intact so polling continues until processing succeeds or fails.
---
Outside diff comments:
In `@libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts`:
- Around line 95-121: In the video upload flow, update the guards after the
initial fetch and upload request to require a non-null videoResponse.body before
using it as the streaming request body, and verify uploadResponse.ok before
parsing or polling the job. Throw descriptive errors containing the relevant
response status information so getJobStatus is never called with an undefined
job ID; follow the equivalent checks in skool.provider.ts and whop.provider.ts.
---
Nitpick comments:
In `@libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts`:
- Around line 196-208: The Reddit media upload in the surrounding provider
method must use runStreamedUpload for the axios request so retries and
handleErrors classification apply; rebuild the upload form within each attempt
as done by the equivalent Discord and Tumblr flows. Replace the direct matchAll
index access with a guarded <Location> match, and propagate a classified error
when no location is returned.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c34a4234-6f86-46db-8f50-0dfcc1c5a216
📒 Files selected for processing (11)
libraries/nestjs-libraries/src/integrations/social.abstract.tslibraries/nestjs-libraries/src/integrations/social/bluesky.provider.tslibraries/nestjs-libraries/src/integrations/social/discord.provider.tslibraries/nestjs-libraries/src/integrations/social/linkedin.provider.tslibraries/nestjs-libraries/src/integrations/social/mastodon.provider.tslibraries/nestjs-libraries/src/integrations/social/reddit.provider.tslibraries/nestjs-libraries/src/integrations/social/skool.provider.tslibraries/nestjs-libraries/src/integrations/social/tiktok.provider.tslibraries/nestjs-libraries/src/integrations/social/tumblr.provider.tslibraries/nestjs-libraries/src/integrations/social/whop.provider.tslibraries/nestjs-libraries/src/integrations/social/x.provider.ts
| // Resolves the total byte size of the media without loading it into memory: | ||
| // a HEAD request for remote URLs, statSync for local files. | ||
| protected async mediaSize(path: string, identifier = ''): Promise<number> { | ||
| if (path.indexOf('http') === 0) { | ||
| // the media path is user-influenced, keep the SSRF-safe dispatcher that | ||
| // this.fetch applies to every other outbound request | ||
| const head = await fetch(path, { | ||
| method: 'HEAD', | ||
| dispatcher: getSsrfSafeDispatcher(), | ||
| } as any); | ||
| const length = head.headers.get('content-length'); | ||
| // A failed HEAD can still carry a content-length (of the error body), | ||
| // which would register the media with a garbage size downstream. | ||
| if (!head.ok || !length) { | ||
| throw new BadBody( | ||
| identifier, | ||
| '{}', | ||
| Buffer.from('{}'), | ||
| 'Could not determine the media size for upload' | ||
| ); | ||
| } | ||
| return Number(length); | ||
| } | ||
|
|
||
| return statSync(path).size; | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect how media paths are produced and whether relative paths reach providers.
set -euo pipefail
fd -t f 'social.abstract.ts' libraries | xargs -r rg -n 'readOrFetch|FRONTEND_URL|statSync|mediaSize|mediaChunk'
# Find the readOrFetch helper and its path handling.
rg -n -C 8 'export (const|function|async function) readOrFetch' libraries
# Find where media paths are assigned before reaching providers.
rg -n -C 5 "indexOf\('http'\)" libraries appsRepository: gitroomhq/postiz-app
Length of output: 19327
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== social.abstract.ts relevant methods =="
sed -n '155,235p' libraries/nestjs-libraries/src/integrations/social.abstract.ts
echo
echo "== call sites for mediaSize/mediaChunk =="
rg -n -C 8 '\.(mediaSize|mediaChunk|getImageDimensions)\(' libraries/nestjs-libraries/src/integrations/social || true
echo
echo "== provider-specific media path handling snippets =="
for f in libraries/nestjs-libraries/src/integrations/social/tumblr.provider.ts \
libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts \
libraries/nestjs-libraries/src/integrations/social/discord.provider.ts \
libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts \
libraries/nestjs-libraries/src/integrations/social/x.provider.ts; do
if [ -f "$f" ]; then
echo "--- $f ---"
rg -n -C 10 'mediaSize|mediaChunk|getImageDimensions|getMediaUrl|media\.path|m\.path|path.*media' "$f" || true
fi
done
echo
echo "== media file model / assignments =="
rg -n -C 6 'model (Media|media)|path.*String|upload|multipart|fieldValue|UploadService|uploadTo' libraries apps | head -n 240Repository: gitroomhq/postiz-app
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate uploads/services/controllers =="
git ls-files | rg 'upload|asset|media|post|posts.*service|controller' | rg '\.ts$' | head -n 200
echo
echo "== focus: media upload path implementations =="
for f in $(git ls-files | rg 'upload|assets*|post' | rg 'providers|services|controllers|repository|schema' | grep '\.ts$' | head -n 80); do
case "$f" in
*uploads*|*asset*|*upload*|*post*)
echo "--- $f ---"
rg -n -C 4 'upload-directory|uploadDirectory|uploadDir|MEDIA_PATH|path:|filename|originalname|mimetype|files\\(' "$f" || true
;;
esac
done
echo
echo "== PostDetails/media DTO definitions =="
rg -n -C 6 "PostDetails|interface .*Media|type .*Media|path: string|files\\(|express-multer|Multer" libraries apps | head -n 260Repository: gitroomhq/postiz-app
Length of output: 34687
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== local.storage.ts =="
cat -n libraries/nestjs-libraries/src/upload/local.storage.ts | sed -n '1,220p'
echo
echo "== media.controller.ts =="
cat -n apps/backend/src/api/routes/media.controller.ts | sed -n '1,260p'
echo
echo "== posts.service.ts updateMedia paths =="
cat -n libraries/nestjs-libraries/src/database/prisma/posts/posts.service.ts | sed -n '330,430p'
echo
echo "== create media paths in posts.service.ts =="
rg -n -C 4 'create\\(|createMany\\(|create\\(\\{|media' libraries/nestjs-libraries/src/database/prisma/posts/posts.service.ts | sed -n '1,220p'Repository: gitroomhq/postiz-app
Length of output: 16653
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== media.repository.ts =="
cat -n libraries/nestjs-libraries/src/database/prisma/media/media.repository.ts | sed -n '1,220p'
echo
echo "== media.service.ts =="
cat -n libraries/nestjs-libraries/src/database/prisma/media/media.service.ts | sed -n '1,240p'
echo
echo "== posts.service.ts create/update calls around media =="
cat -n libraries/nestjs-libraries/src/database/prisma/posts/posts.service.ts | sed -n '620,750p'
cat -n libraries/nestjs-libraries/src/database/prisma/posts/posts.service.ts | sed -n '750,830p'
echo
echo "== provider upload functions around raw paths =="
cat -n libraries/nestjs-libraries/src/integrations/social/x.provider.ts | sed -n '490,525p'
cat -n libraries/nestjs-libraries/src/integrations/social/tiktok.provider.ts | sed -n '790,830p'
cat -n libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts | sed -n '160,190p'
cat -n libraries/nestjs-libraries/src/integrations/social/discord.provider.ts | sed -n '145,175p'
cat -n libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts | sed -n '255,280p'Repository: gitroomhq/postiz-app
Length of output: 23667
Resolve non-HTTP media paths before using them with the filesystem.
mediaSize and mediaChunk use statSync / createReadStream for any path that does not start with http, but providers pass raw media paths such as item.path or media.path. Local uploads are stored with URLs like /uploads/year/month/day/r…, not absolute filesystem paths, so these calls can resolve relative to the worker cwd and fail. Resolve these cases like getImageDimensions/updateMedia use process.env.UPLOAD_DIRECTORY for non-HTTP paths.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libraries/nestjs-libraries/src/integrations/social.abstract.ts` around lines
177 - 202, Update mediaSize and the related mediaChunk filesystem access to
resolve non-HTTP media paths through process.env.UPLOAD_DIRECTORY, matching the
existing getImageDimensions and updateMedia behavior. Preserve direct handling
for HTTP URLs and ensure statSync and createReadStream receive the resolved
filesystem path rather than the raw provider path.
| const finalize = await client.v2.post<{ | ||
| data: { | ||
| processing_info?: { | ||
| state: string; | ||
| check_after_secs?: number; | ||
| error?: { message?: string }; | ||
| }; | ||
| id: string; | ||
| processing_info?: { state: string; check_after_secs?: number }; | ||
| }; | ||
| }>('media/upload', { command: 'STATUS', media_id: mediaId }); | ||
|
|
||
| const info = response.data.processing_info; | ||
| if (!info || info.state === 'succeeded') { | ||
| return; | ||
| } | ||
| if (info.state === 'failed') { | ||
| throw new BadBody( | ||
| 'x-error-upload', | ||
| JSON.stringify(response.data), | ||
| Buffer.from('{}'), | ||
| `X media processing failed: ${info.error?.message || 'unknown error'}` | ||
| ); | ||
| } | ||
| await timer((info.check_after_secs || 1) * 1000); | ||
| await this.waitForMediaProcessing(client, mediaId); | ||
| } | ||
| }>(`media/upload/${mediaId}/finalize`); | ||
|
|
||
| // Resolves the total byte size of the video without loading it into memory: | ||
| // a HEAD request for remote URLs, statSync for local files. | ||
| private async videoSize(path: string): Promise<number> { | ||
| if (path.indexOf('http') === 0) { | ||
| const head = await fetch(path, { method: 'HEAD' }); | ||
| const length = head.headers.get('content-length'); | ||
| if (!length) { | ||
| let processing = finalize.data.processing_info; | ||
| let attempts = 0; | ||
| const maxAttempts = 100; // X drives the pace via check_after_secs (~1-5s each) | ||
| while (processing && processing.state !== 'succeeded') { | ||
| if (processing.state === 'failed' || attempts >= maxAttempts) { | ||
| throw new BadBody( | ||
| 'x-error-upload', | ||
| '{}', | ||
| this.identifier, | ||
| JSON.stringify(processing), | ||
| Buffer.from('{}'), | ||
| 'Could not determine the video size for X upload' | ||
| `X failed to process the uploaded video${ | ||
| (processing as any)?.error?.message | ||
| ? `: ${(processing as any).error.message}` | ||
| : '' | ||
| }` | ||
| ); | ||
| } | ||
| return Number(length); | ||
| } | ||
|
|
||
| return statSync(path).size; | ||
| } | ||
|
|
||
| // Returns only the [start, end] byte window of the video as a Buffer (a | ||
| // ranged GET for remote URLs, a ranged read for local files), so memory is | ||
| // bounded by the segment size. | ||
| private async videoChunk( | ||
| path: string, | ||
| start: number, | ||
| end: number | ||
| ): Promise<Buffer> { | ||
| if (path.indexOf('http') === 0) { | ||
| const response = await fetch(path, { | ||
| headers: { Range: `bytes=${start}-${end}` }, | ||
| }); | ||
| // A 200 means the origin ignored the Range header and is sending the | ||
| // whole file: uploading it as this segment would silently corrupt the | ||
| // video, so fail loudly instead. | ||
| if (response.status !== 206) { | ||
| throw new BadBody( | ||
| 'x-error-upload', | ||
| '{}', | ||
| Buffer.from('{}'), | ||
| `Media server ignored the ranged request (status ${response.status}); it must support HTTP Range requests for chunked X uploads` | ||
| ); | ||
| } | ||
| return Buffer.from(await response.arrayBuffer()); | ||
| await timer((processing.check_after_secs || 1) * 1000); | ||
| const status = await client.v2.get<{ | ||
| data: { | ||
| processing_info?: { state: string; check_after_secs?: number }; | ||
| }; | ||
| }>('media/upload', { command: 'STATUS', media_id: mediaId }); | ||
| processing = status.data.processing_info; | ||
| attempts++; | ||
| } | ||
|
|
||
| return new Promise((resolve, reject) => { | ||
| const chunks: Buffer[] = []; | ||
| createReadStream(path, { start, end }) | ||
| .on('data', (chunk) => chunks.push(chunk as Buffer)) | ||
| .on('end', () => resolve(Buffer.concat(chunks))) | ||
| .on('error', reject); | ||
| }); | ||
| return mediaId; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
X API v2 media upload chunked initialize append finalize status endpoint parameters
💡 Result:
The X API v2 uses a four-step chunked upload process to handle large media files such as videos, GIFs, and images. The process is broken down into the following endpoints and operations [1][2]. 1. INITIALIZE (POST /2/media/upload/initialize) This step initiates the upload session and returns a media_id [1][3]. Parameters (Body): - total_bytes: The total size of the media in bytes (up to 17,179,869,184 bytes) [3]. - media_type: The MIME type of the media (e.g., video/mp4, image/jpeg, image/gif) [3]. - media_category: The category of the media (e.g., amplify_video, tweet_video, tweet_image) [3]. - shared: A boolean indicating if the media is shared [3]. 2. APPEND (POST /2/media/upload/{id}/append) After obtaining a media_id, you use this endpoint to upload individual chunks of the file sequentially [1]. - Path Parameter: {id} — The media_id returned during the initialization phase [4]. - Body: The binary chunk of the file being uploaded (typically sent as multipart/form-data) [4]. 3. FINALIZE (POST /2/media/upload/{id}/finalize) Once all chunks are uploaded, call this endpoint to complete the upload process [1][5]. - Path Parameter: {id} — The media_id of the target media [5]. 4. STATUS (GET /2/media/upload?command=STATUS&media_id={id}) For asynchronous processing (common with videos), use this to check the status of the media [1]. - Parameters: - command: Must be set to STATUS [1]. - media_id: The ID of the media to query [1]. - This endpoint returns a processing_info object containing the current state (e.g., pending, in_progress, failed, succeeded) and, if applicable, the check_after_secs field, which advises how long to wait before polling again [1][5]. Once the status indicates the state is succeeded, the media_id can be used to attach the media to other resources, such as posts [1][2].
Citations:
- 1: https://x-preview.mintlify.app/x-api/media/quickstart/media-upload-chunked
- 2: https://x-preview.mintlify.app/x-api/media/introduction
- 3: https://x-preview.mintlify.app/x-api/media/initialize-media-upload
- 4: https://docs.x.com/x-api/media/append-media-upload
- 5: https://x-preview.mintlify.app/x-api/media/finalize-media-upload
🌐 Web query:
twitter-api-v2 1.29.0 media upload get media upload command STATUS
💡 Result:
In the twitter-api-v2 library, specifically as of version 1.29.0 [1][2], the media upload process has been migrated to use the dedicated X API v2 endpoints [3][4]. Regarding your query about the media upload command STATUS: 1. Migration from V1.1: The traditional V1.1 media upload process utilized a command-based system (e.g., using the command=STATUS query parameter) [4]. However, the twitter-api-v2 library has been updated to align with X's shift away from these legacy V1.1 commands [4]. 2. Current Implementation: The library's.uploadMedia method for V2 now interacts with the updated X API v2 media upload endpoints [3][4]. As indicated in the project's documentation and pull request history (such as PR #584), the library was updated to use these new dedicated endpoints, effectively removing the reliance on the legacy command parameter system [3][4]. 3. Status Checking: If you are using.uploadMedia in version 1.29.0, the library handles the underlying request structure [5]. For checking media processing status, the X API v2 now provides distinct endpoints (e.g., GET media/upload/status) rather than the old command-based query parameters [6]. If you are encountering issues or searching for the status check mechanism, ensure you are utilizing the updated methods provided in the V2 client rather than trying to pass a command parameter manually, as the library abstracts this to comply with the current API architecture [5][4].
Citations:
- 1: https://www.npmjs.com/package/twitter-api-v2
- 2: https://registry.npmjs.org/twitter-api-v2
- 3: plhery/node-twitter-api-v2@1.20.2...1.29.0
- 4: [bug] V2 uploadMedia (documentation, not working?) plhery/node-twitter-api-v2#569
- 5: https://github.com/plhery/node-twitter-api-v2/blob/HEAD/doc/v2.md
- 6: https://docs.x.com/x-api/media/get-media-upload-status
🌐 Web query:
site:github.com/PLhery/node-twitter-api-v2 "media/upload?command=STATUS" "twitter-api-v2"
💡 Result:
The media/upload endpoint using the command=STATUS query parameter is a legacy mechanism associated with the Twitter API v1.1 [1][2]. It is not used in the modern Twitter API v2 for media uploads [3][4]. In the context of the popular node-twitter-api-v2 library, media uploads are handled differently depending on the API version: 1. API v1.1: Still utilizes the legacy media/upload.json endpoint, which supports the command query parameter (e.g., INIT, APPEND, FINALIZE, STATUS) [1][2]. 2. API v2: X has deprecated the older chunked upload process that relied on the command query parameter [3][4]. Instead, the v2 media upload process now uses dedicated endpoints for each step of the upload process [3]: - POST.../2/media/upload/initialize [3] - POST.../2/media/upload/{id}/append [3] - POST.../2/media/upload/{id}/finalize [3] If you are working with the node-twitter-api-v2 library, you should use the built-in v2 methods (such as client.v2.uploadMedia) which have been updated to use these new dedicated v2 endpoints rather than the legacy command-based system [5][3][6].
Citations:
- 1: https://github.com/plhery/node-twitter-api-v2/blob/master/doc/examples.md
- 2: https://github.com/PLhery/node-twitter-api-v2/blob/master/doc/v1.md
- 3: Adapt uploadMedia to new endpoints for chunked media upload in V2 plhery/node-twitter-api-v2#584
- 4: [bug] V2 uploadMedia (documentation, not working?) plhery/node-twitter-api-v2#569
- 5: https://github.com/PLhery/node-twitter-api-v2/blob/master/doc/v2.md
- 6: Fix error handling and use segment_index starting from zero plhery/node-twitter-api-v2#589
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- candidate files ---'
fd -a 'x\.provider\.ts$|package\.json$|package-lock\.json$|pnpm-lock\.yaml$|yarn\.lock$' . | sed 's#^\./##' | head -100
printf '%s\n' '--- declaration ---'
rg -n '"twitter-api-v2"|twitter-api-v2' package.json package-lock.json pnpm-lock.yaml yarn.lock libraries apps 2>/dev/null || true
printf '%s\n' '--- x.provider outline ---'
ast-grep outline libraries/nestjs-libraries/src/integrations/social/x.provider.ts --view expanded || true
printf '%s\n' '--- uploadVideoInChunks section ---'
sed -n '420,500p' libraries/nestjs-libraries/src/integrations/social/x.provider.ts | cat -nRepository: gitroomhq/postiz-app
Length of output: 5500
Use X’s v2 media upload status endpoint.
Lines 459-488 call the v2 finalize endpoint, but then poll with client.v2.get('media/upload', { command: 'STATUS', media_id: mediaId }), which is the legacy v1.1 status shape. Use X’s v2 GET /2/media/upload/status path with media_id instead; otherwise processing_info may stay undefined, poll exits immediately, and the tweet can use a media id that has not finished processing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libraries/nestjs-libraries/src/integrations/social/x.provider.ts` around
lines 459 - 494, Update the polling request in the video-processing loop after
`finalize` to use X’s v2 `/2/media/upload/status` endpoint with `media_id` as
the request parameter. Keep the existing `processing_info` response handling and
retry logic intact so polling continues until processing succeeds or fails.
- Add a shared SSRF-safe mediaStream helper and use it in Discord, Reddit and Tumblr: the axios stream downloads bypassed the SSRF-safe dispatcher that the matching HEAD requests already used - Check the streamed PUT response status in Skool and Whop so a rejected upload fails fast instead of publishing an empty attachment (Skool) or burning the ~9 minute status poll (Whop) - Thread the provider identifier through mediaChunk so ranged-read failures are attributed to the right provider in Temporal Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the review findings in 1b121d0: Fixed
Not fixed, with reasons
|
|
@CodeRabbit review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (4)
libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts:101
videoResponse.bodycan be null (e.g., if the fetch implementation fails to provide a body). Passing a null body into the upload request will throw at runtime; check forbodypresence alongsideokbefore streaming the upload.
if (!videoResponse.ok) {
throw new Error(`Failed to fetch video: ${videoResponse.statusText}`);
}
libraries/nestjs-libraries/src/integrations/social.abstract.ts:200
mediaSizereturnsNumber(content-length)without validating that the header is a finite, positive integer. If the origin returnscontent-length: 0or a non-numeric value, callers will compute chunk counts/sizes incorrectly (e.g.,Math.ceil(NaN / chunk)), leading to hard-to-debug upload failures.
'Could not determine the media size for upload'
);
}
return Number(length);
}
libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts:186
- This S3 upload path now uses a one-shot stream but isn't wrapped in
runStreamedUpload, so errors won’t be classified/retried likethis.fetchand the stream can’t be recreated on retry. This risks losing existing Temporal retry semantics for transient S3/Reddit failures.
// S3 requires the exact part length, which comes from a HEAD request.
const fileSize = await this.mediaSize(path, this.identifier);
const stream = await this.mediaStream(path, this.identifier);
const upload = (fields as { name: string; value: string }[]).reduce(
libraries/nestjs-libraries/src/integrations/social/x.provider.ts:468
maxAttempts = 100caps processing polling to ~100–500 seconds for typicalcheck_after_secsvalues (1–5s). That introduces a new timeout compared to the previous behavior and can cause false failures for longer video processing jobs.
const maxAttempts = 100; // X drives the pace via check_after_secs (~1-5s each)
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts (3)
607-619: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winSelect the streaming path before reusing
media.buffer.If a video item contains both
pathandbuffer, this branch passes the full buffer touploadPictureand defeats bounded video uploads. Check for MP4 videos first, or explicitly rejectbufferfor videos.Proposed branch order
- if ( + if (hasExtension(media.path, 'mp4')) { + mediaBuffer = { path: media.path }; + } else if ( media && typeof media === 'object' && 'buffer' in media && Buffer.isBuffer(media.buffer) ) { mediaBuffer = (media as any).buffer; - } else if (hasExtension(media.path, 'mp4')) { - mediaBuffer = { path: media.path }; } else {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts` around lines 607 - 619, Update the mediaBuffer selection logic near the video handling branch to check hasExtension(media.path, 'mp4') before accepting media.buffer. Ensure MP4 items always use the streaming { path: media.path } representation, even when a buffer is also present, while preserving buffered handling for non-video media.
620-620: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winDon’t convert unbuffered PDFs to JPEG before Documents upload.
In
createMainPost, a PDF path withoutmedia.bufferruns throughprepareMediaBuffer, whose fallback sends it throughsharp(...).toFormat('jpeg'). The returned JPEG bytes are still uploaded onuploadPicturewith thebuffer.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts` at line 620, Update createMainPost so unbuffered PDF media does not pass through prepareMediaBuffer’s JPEG conversion before uploadPicture; preserve the original PDF bytes or reject the PDF at input validation, while keeping existing image conversion behavior unchanged. Add a regression test covering a PDF path without media.buffer.
327-350: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftUse each video upload instruction for its byte range.
The video loop uses only
uploadInstructions[0].uploadUrland creates its own 2 MB byte ranges. LinkedIn's Videos API can return multiple instructions, each with its ownuploadUrl,firstByte, andlastByte; send each part using the matching instruction before finalizinguploadedPartIdsin instruction order.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts` around lines 327 - 350, Update the video upload loop around uploadInstructions to iterate each instruction and use its uploadUrl, firstByte, and lastByte for the corresponding request body. Read or slice exactly the instructed byte range, then collect uploadedPartIds in the same order as uploadInstructions before finalizing; remove the fixed 2 MB range generation and uploadInstructions[0] usage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@libraries/nestjs-libraries/src/integrations/social/linkedin.provider.ts`:
- Around line 607-619: Update the mediaBuffer selection logic near the video
handling branch to check hasExtension(media.path, 'mp4') before accepting
media.buffer. Ensure MP4 items always use the streaming { path: media.path }
representation, even when a buffer is also present, while preserving buffered
handling for non-video media.
- Line 620: Update createMainPost so unbuffered PDF media does not pass through
prepareMediaBuffer’s JPEG conversion before uploadPicture; preserve the original
PDF bytes or reject the PDF at input validation, while keeping existing image
conversion behavior unchanged. Add a regression test covering a PDF path without
media.buffer.
- Around line 327-350: Update the video upload loop around uploadInstructions to
iterate each instruction and use its uploadUrl, firstByte, and lastByte for the
corresponding request body. Read or slice exactly the instructed byte range,
then collect uploadedPartIds in the same order as uploadInstructions before
finalizing; remove the fixed 2 MB range generation and uploadInstructions[0]
usage.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 35662690-22ae-477e-ae05-3de084e06b16
📒 Files selected for processing (8)
libraries/nestjs-libraries/src/integrations/social.abstract.tslibraries/nestjs-libraries/src/integrations/social/discord.provider.tslibraries/nestjs-libraries/src/integrations/social/linkedin.provider.tslibraries/nestjs-libraries/src/integrations/social/reddit.provider.tslibraries/nestjs-libraries/src/integrations/social/skool.provider.tslibraries/nestjs-libraries/src/integrations/social/tumblr.provider.tslibraries/nestjs-libraries/src/integrations/social/whop.provider.tslibraries/nestjs-libraries/src/integrations/social/x.provider.ts
🚧 Files skipped from review as they are similar to previous changes (7)
- libraries/nestjs-libraries/src/integrations/social.abstract.ts
- libraries/nestjs-libraries/src/integrations/social/skool.provider.ts
- libraries/nestjs-libraries/src/integrations/social/discord.provider.ts
- libraries/nestjs-libraries/src/integrations/social/reddit.provider.ts
- libraries/nestjs-libraries/src/integrations/social/tumblr.provider.ts
- libraries/nestjs-libraries/src/integrations/social/x.provider.ts
- libraries/nestjs-libraries/src/integrations/social/whop.provider.ts
- X: cap video-processing polling on accumulated wait time (30 min) instead of 100 attempts - main polled unbounded, so the attempt cap could fail long-processing videos that previously succeeded - mediaSize: reject zero/NaN content-length so chunk-count math never runs on a garbage size - Bluesky: guard against a null response body before streaming the upload Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Second round addressed in c2944e1: Fixed
Not fixed, with reasons
|
| return [ | ||
| ...(d.data as string).matchAll(/<Location>(.*?)<\/Location>/g), | ||
| ][0][1]; |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@CodeRabbit review |
✅ Action performedReview finished.
|
An empty or unexpected S3 body made the <Location> parse throw an opaque TypeError; fail with a clear BadBody instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Final round, addressed in b0bb1c8:
Copilot's latest review generated no new comments, and CodeRabbit had nothing new inside the diff. All other outstanding threads were dispositioned in the two comments above. |
A file-by-file comparison with origin/main confirmed no logic was
dropped by the streaming refactor, but surfaced side-effects:
- request identity encoding on every media HEAD/GET whose
content-length feeds an upload size: fetch transparently
decompresses encoded bodies, so a compressing origin would make the
declared Content-Length smaller than the streamed bytes and abort or
truncate the upload (impossible with the old buffered code)
- tumblr: validate the create-post response shape - axios parses JSON
silently, so a non-JSON 200 (proxy/WAF page) returned a string and
published the post as completed with an empty id; main threw here
- linkedin: restore the Buffer.from wrap on the GIF pass-through so
prepareMediaBuffer always returns a real Buffer (uploadPicture
branches on Buffer.isBuffer)
- runStreamedUpload: never hand handleErrors an empty string, matching
this.fetch's '|| {}' guard
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| totalRetries <= 2 && | ||
| (status === 429 || | ||
| (status === 500 && !handleError) || | ||
| handleError?.type === 'retry' || | ||
| json.includes('rate_limit_exceeded') || | ||
| json.includes('Rate limit')) | ||
| ) { | ||
| await timer(5000); | ||
| return this.runStreamedUpload(func, identifier, totalRetries + 1); |
There was a problem hiding this comment.
Bug: The retry condition in runStreamedUpload (totalRetries <= 2) allows 4 attempts, while this.fetch (totalRetries > 2) allows 3, contradicting the stated goal of matching behavior.
Severity: MEDIUM
Suggested Fix
Change the retry condition in runStreamedUpload at line 306 from totalRetries <= 2 to totalRetries < 2 or a similar condition that results in 3 total attempts, aligning it with the logic in this.fetch. This will make the retry behavior consistent.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: libraries/nestjs-libraries/src/integrations/social.abstract.ts#L306-L314
Potential issue: The retry logic in `runStreamedUpload` uses the condition `totalRetries
<= 2`, which results in 4 total attempts (initial call + 3 retries). This is
inconsistent with the `this.fetch` method, which uses `totalRetries > 2` and allows only
3 total attempts. This discrepancy violates the stated intent in the code comments and
PR description to maintain the "same retry... rules as this.fetch". This can lead to
different behavior for transient errors, adding an extra 5-second delay and an
unnecessary retry attempt for providers using `runStreamedUpload`.
| } catch (err: any) { | ||
| if (!err?.response) { | ||
| throw err; | ||
| } |
There was a problem hiding this comment.
Bug: Errors thrown when fetching media for runStreamedUpload lack a .response property, which bypasses the function's retry logic and prevents retries on transient network failures.
Severity: HIGH
Suggested Fix
When a media fetch fails with a non-ok status, throw an error object that includes the response property, similar to how axios errors are structured. This will allow the catch block in runStreamedUpload to correctly identify it as a retryable error.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: libraries/nestjs-libraries/src/integrations/social.abstract.ts#L292-L295
Potential issue: When fetching media from a source URL for providers like Mastodon, a
`new Error(...)` is thrown if the response status is not ok. This error object lacks a
`.response` property. The `runStreamedUpload` function, which wraps this logic, is
designed to only retry on errors that contain an `err?.response` property. As a result,
transient failures during media fetching (e.g., 429, 500, 503 status codes) will not be
retried, causing the upload to fail immediately instead of recovering.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (4)
libraries/nestjs-libraries/src/integrations/social/whop.provider.ts:292
- The media source fetch failure is thrown as a generic Error, which bypasses the SocialAbstract media helpers and yields inconsistent/non-truncated provider failures. Since you already added streaming helpers, prefer using mediaStream() here so SSRF-safe fetching and BadBody formatting are centralized and Temporal won’t waste retries on permanent 4xx media URLs.
const fileResponse = await fetch(item.path, {
headers: { 'accept-encoding': 'identity' },
// @ts-ignore - undici-only option; blocks SSRF to internal IPs
dispatcher: getSsrfSafeDispatcher(),
});
if (!fileResponse.ok || !fileResponse.body) {
throw new Error(`Failed to fetch media: ${fileResponse.statusText}`);
}
libraries/nestjs-libraries/src/integrations/social/skool.provider.ts:261
- If fetching the media fails, this throws a generic Error. Using the shared mediaStream() helper here would keep SSRF protection and error formatting consistent (BadBody) and avoid Temporal retries on permanent media URL failures.
const fileResponse = await fetch(item.path, {
headers: { 'accept-encoding': 'identity' },
// @ts-ignore - undici-only option; blocks SSRF to internal IPs
dispatcher: getSsrfSafeDispatcher(),
});
if (!fileResponse.ok || !fileResponse.body) {
throw new Error(`Failed to fetch media: ${fileResponse.statusText}`);
}
libraries/nestjs-libraries/src/integrations/social/mastodon.provider.ts:156
- Inside runStreamedUpload, a failed media fetch currently throws a plain Error, so runStreamedUpload cannot apply its retry/handleErrors classification (it only handles errors with err.response). Throw an axios-style { response: { status, data } } object here so transient 429/5xx can be retried and non-retryable failures are surfaced as BadBody/RefreshToken consistently.
const fileResponse = await fetch(fileUrl, {
// identity encoding so content-length matches the streamed bytes
headers: { 'accept-encoding': 'identity' },
// @ts-ignore - undici-only option; blocks SSRF to internal IPs
dispatcher: getSsrfSafeDispatcher(),
});
if (!fileResponse.ok || !fileResponse.body) {
throw new Error(`Failed to fetch media: ${fileResponse.statusText}`);
}
libraries/nestjs-libraries/src/integrations/social/bluesky.provider.ts:104
- When the video GET fails, this throws a generic Error rather than a provider BadBody. That makes failures less consistent with the other Bluesky upload errors (and can lead to noisy Temporal retries for permanent 4xx media URLs). Prefer throwing BadBody with the HTTP status so the failure is classified and truncated consistently.
const videoResponse = await fetch(videoPath, {
headers: { 'accept-encoding': 'identity' },
// @ts-ignore - undici-only option; blocks SSRF to internal IPs
dispatcher: getSsrfSafeDispatcher(),
});
if (!videoResponse.ok || !videoResponse.body) {
throw new Error(`Failed to fetch video: ${videoResponse.statusText}`);
}
The YouTube and TikTok chunked uploaders read the stored video back from media storage with their own HEAD/ranged GET helpers, and those helpers never asked for identity encoding. Every other media read in the project does: #1835 added `accept-encoding: identity` to SocialAbstract's mediaSize / mediaChunk / mediaStream and to the providers that read media themselves, because a transformed (compressed) response loses its Content-Length - and Cloudflare answers a range request on an object with no Content-Length by returning the full body with a 200 instead of the requested 206. That is very likely what has been failing ~1GB YouTube uploads: a single 200 out of the ~128 ranged reads a 1GB video needs is thrown as BadBody, which the v1.0.6 workflow treats as a permanent platform rejection, so the whole post is marked ERROR even though the upload session is still resumable. The gap looks like an artifact of merge ordering rather than a decision: #1813 introduced these two bespoke helpers on Aug 3, and #1835 swept identity encoding across the project on Aug 4 - by then the sweep had been written against a tree that did not contain them, so YouTube and TikTok were missed while X and LinkedIn (bespoke at the time too) were consolidated onto the shared helpers and fixed. Behaviour on a non-206 is deliberately left untouched: if the customer report recurs after this ships, the next step is to reconsider treating the documented 200-with-full-body answer as transient (a plain Error the workflow retries and resumes from the committed byte offset) instead of as a permanent failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What kind of change does this PR introduce?
Bug fix — memory-leak / OOM fix in the workers.
Why was this change needed?
Workers were running out of memory because most social providers downloaded entire media files into memory (
arrayBuffer()/Blob) before uploading them. With several concurrent jobs uploading large videos, the buffers stacked up and OOM-killed the worker (in Whop's case the buffer even stayed resident through the whole ~9-minute status poll).This PR makes every provider upload media without holding the whole file in memory:
mediaSize(HEAD request /statSync),mediaChunk(ranged reads for chunked-upload APIs), andrunStreamedUpload(retry +handleErrorsclassification for streamed bodies that can't be replayed — the whole request is rebuilt per attempt).Promise.all(parallel uploads held every file in memory at once).duplex: 'half'.form-data, rebuilt on every retry attempt.All user-influenced media fetches keep the SSRF-safe dispatcher, and error classification (retry / refresh-token / bad-body) matches the existing
this.fetchbehavior so Temporal retry semantics are unchanged. No workflow or activity signatures were touched.Other information:
Follow-up to the August 2026 worker OOM audit; the Temporal worker configuration findings from that audit are not part of this PR.
Checklist:
🤖 Generated with Claude Code
Summary by CodeRabbit