Skip to content

perf!: serialize requests once in batchAddRequests - #1051

Open
vdusek wants to merge 6 commits into
v3from
perf/batch-add-requests-single-serialization
Open

perf!: serialize requests once in batchAddRequests#1051
vdusek wants to merge 6 commits into
v3from
perf/batch-add-requests-single-serialization

Conversation

@vdusek

@vdusek vdusek commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Description

  • Mirroring fix: Make batch_add_requests split batches by serialized payload size apify-client-python#953.
  • batchAddRequests stringified every request twice: once in sliceArrayByByteLength to measure the batch, and again in the serializeRequest interceptor when sending. Axios then ran its default transformRequest on top, because transformRequest: undefined in the instance config falls back to the defaults, and that default validates a JSON string body by parsing it in full. Each batch body was stringified twice and parsed once more.
  • Each request is now serialized once up front. The byte lengths decide the batch boundaries (commas and brackets counted, which the old measurement skipped) and the same strings are joined into the batch body, sent with an explicit content-type: application/json. The interceptor passes a string body with an explicit content type through untouched, and the axios instance sets transformRequest and transformResponse to []. That also removes the validation re-parse for every JSON body the client sends.

Issue

Breaking changes

  • A string body declared as JSON but not valid JSON (for example setRecord with contentType: 'application/json' and a non-JSON string) is sent as it is. Axios used to double-encode it into a JSON string literal.
  • A request too large for the payload limit is rejected before any batch is sent. It used to be detected only when its batch came up, with earlier batches already in flight.
  • The protected _batchAddRequests and _batchAddRequestsWithRetries take the serialized entries, and _batchAddRequests no longer re-validates a batch the public method already validated. The API report is updated.

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 9, 2026
@vdusek vdusek self-assigned this Sep 9, 2026
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/34350259605#summary-102461507918

@vdusek vdusek changed the title perf: serialize requests once in batchAddRequests perf!: serialize requests once in batchAddRequests Sep 9, 2026
@vdusek
vdusek requested a review from barjin September 9, 2026 12:19
@vdusek
vdusek marked this pull request as ready for review September 9, 2026 12:19
@vdusek
vdusek requested a review from szaganek as a code owner September 9, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants