Skip to content

Commit 104b505

Browse files
committed
feat!: add retry/parallelism/byte-chunking to batch_add_requests; type request-queue lock/list/batch results
RequestQueueClient::batch_add_requests now matches the reference client's batchAddRequests: chunks by both request count (25) and JSON byte size (~9 MiB), sends chunks with bounded parallelism (default 5 in flight), and retries requests reported unprocessed (rate-limited) with exponential backoff (default 3 retries). Its second parameter is now BatchAddRequestsOptions instead of a bare forefront: bool. list_and_lock_head, list_requests, unlock_requests, prolong_request_lock and batch_delete_requests now return typed models (LockedRequestQueueHead, RequestQueueRequestsPage, UnlockRequestsResult, RequestLockInfo, BatchRequestsOperationResult) instead of serde_json::Value, matching every other resource client and the stable schemas the OpenAPI spec already documents for these endpoints. batch_delete_requests also now rejects more than 25 requests client-side instead of forwarding an oversized payload to the API. Breaking change, bumping 0.7.0 -> 0.8.0.
1 parent f088c82 commit 104b505

8 files changed

Lines changed: 714 additions & 76 deletions

File tree

CHANGELOG.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,31 @@ All notable changes to the Rust Apify API client are documented here. The format
44
based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project adheres
55
to [Semantic Versioning](https://semver.org/).
66

7+
## [0.8.0] - 2026-08-11
8+
9+
### Changed
10+
- `RequestQueueClient::batch_add_requests` now matches the reference client's `batchAddRequests`:
11+
it automatically splits large inputs into chunks that respect both the API's per-call
12+
request-count limit (25) and its request-body byte-size limit (~9 MiB), sends chunks with up
13+
to `BatchAddRequestsOptions::max_parallel` requests in flight at once (default 5), and retries
14+
requests an API call reports as `unprocessed` (typically rate-limited) with exponential
15+
backoff, up to `max_unprocessed_requests_retries` times (default 3). Its second parameter is
16+
now `BatchAddRequestsOptions` (was a bare `forefront: bool`), and it returns the typed
17+
`BatchRequestsOperationResult` (was `serde_json::Value`). **Breaking.**
18+
- `RequestQueueClient::batch_delete_requests` now returns the typed `BatchRequestsOperationResult`
19+
(was `serde_json::Value`), and rejects more than 25 requests per call with
20+
`ApifyClientError::InvalidArgument` instead of forwarding an oversized payload to the API
21+
(matching the reference client, which validates rather than auto-chunks deletes). **Breaking.**
22+
- `RequestQueueClient::list_and_lock_head` now returns the typed `LockedRequestQueueHead` (was
23+
`serde_json::Value`). **Breaking.**
24+
- `RequestQueueClient::list_requests` now returns the typed `RequestQueueRequestsPage` (was
25+
`serde_json::Value`). **Breaking.**
26+
- `RequestQueueClient::unlock_requests` now returns the typed `UnlockRequestsResult` (was
27+
`serde_json::Value`). **Breaking.**
28+
- `RequestQueueClient::prolong_request_lock` now returns the typed `RequestLockInfo` (was
29+
`serde_json::Value`). **Breaking.**
30+
- Bumped crate version to `0.8.0`.
31+
732
## [0.7.0] - 2026-08-10
833

934
### Added

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "apify-client"
3-
version = "0.7.0"
3+
version = "0.8.0"
44
authors = ["Apify Technologies <support@apify.com>"]
55
description = "An official, but experimental, AI-generated and AI-maintained Rust client for the Apify API (https://apify.com)."
66
license = "Apache-2.0"

docs/storages.md

Lines changed: 68 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -192,14 +192,14 @@ listed in `KeyValueStoreKeysPage::items`. Its fields:
192192
| `get_request(id)` | `&str` | `Option<RequestQueueRequest>` | Reads a request. |
193193
| `update_request(request, forefront)` | `&RequestQueueRequest`, `bool` | `RequestQueueOperationInfo` | Updates a request. |
194194
| `delete_request(id)` | `&str` | `()` | Deletes a request. |
195-
| `list_and_lock_head(lock_secs, limit)` | `i64`, `Option<i64>` | `Value` | Locks head requests. |
196-
| `batch_add_requests(requests, forefront)` | `&[RequestQueueRequest]`, `bool` | `Value` | Batch add. |
197-
| `batch_delete_requests(requests)` | `&[impl Serialize]` | `Value` | Batch delete. |
198-
| `list_requests(options)` | `ListRequestsOptions { limit, exclusive_start_id, cursor, filter }` | `Value` | List requests (cursor/filter pagination). |
195+
| `list_and_lock_head(lock_secs, limit)` | `i64`, `Option<i64>` | `LockedRequestQueueHead` | Locks head requests. |
196+
| `batch_add_requests(requests, options)` | `&[RequestQueueRequest]`, `BatchAddRequestsOptions` | `BatchRequestsOperationResult` | Batch add, with automatic chunking, bounded parallelism, and retries for rate-limited requests. |
197+
| `batch_delete_requests(requests)` | `&[impl Serialize]` | `BatchRequestsOperationResult` | Batch delete (max 25 requests per call; larger inputs are rejected client-side). |
198+
| `list_requests(options)` | `ListRequestsOptions { limit, exclusive_start_id, cursor, filter }` | `RequestQueueRequestsPage` | List requests (cursor/filter pagination). |
199199
| `paginate_requests(page_limit)` | `Option<i64>` | `RequestQueueRequestsIterator` | Lazy request iterator. |
200-
| `prolong_request_lock(id, lock_secs, forefront)` | `&str`, `i64`, `bool` | `Value` | Extend a lock. |
200+
| `prolong_request_lock(id, lock_secs, forefront)` | `&str`, `i64`, `bool` | `RequestLockInfo` | Extend a lock. |
201201
| `delete_request_lock(id, forefront)` | `&str`, `bool` | `()` | Release a lock. |
202-
| `unlock_requests()` || `Value` | Release all this client's locks. |
202+
| `unlock_requests()` || `UnlockRequestsResult` | Release all this client's locks. |
203203

204204
`paginate_requests(page_limit)` returns a `RequestQueueRequestsIterator` — a lazy, page-fetching
205205
iterator (parity with the Store iterator in [Store, users and logs](misc.md#apify-store--clientstore)).
@@ -241,22 +241,62 @@ while let Some(request) = iter.next().await? {
241241
# }
242242
```
243243

244-
The `forefront` boolean (on `add_request`, `update_request`, `batch_add_requests`,
245-
`prolong_request_lock`, `delete_request_lock`) controls queue ordering: `true` puts the
246-
request(s) at the **front** of the queue so they are handled before the existing backlog;
247-
`false` (the usual choice) appends them at the **back**.
244+
The `forefront` boolean (on `add_request`, `update_request`,
245+
`prolong_request_lock`, `delete_request_lock`) and the `BatchAddRequestsOptions::forefront` field
246+
(on `batch_add_requests`) control queue ordering: `true` puts the request(s) at the **front** of
247+
the queue so they are handled before the existing backlog; `false` (the usual choice) appends
248+
them at the **back**.
248249

249-
Some request-queue methods return an untyped `serde_json::Value` because the API responses are
250-
open-ended and most callers do not consume them structurally. Their shapes (read fields with
251-
`value.get("...")`):
250+
### `batch_add_requests`
252251

253-
- `list_and_lock_head` → an object with `items` (the locked head requests), `limit`,
254-
`queueModifiedAt`, `hadMultipleClients`, and the granted `lockSecs`.
255-
- `batch_add_requests` / `batch_delete_requests` → an object with `processedRequests` and
256-
`unprocessedRequests` arrays.
257-
- `list_requests` → an object with `items` (the page of requests), `count`, `limit`, and
258-
`exclusiveStartId` for cursor continuation.
259-
- `unlock_requests` → an object reporting how many locks were released (`unlockedCount`).
252+
`batch_add_requests(requests, options)` is the efficient way to add many requests at once —
253+
significantly cheaper than calling `add_request` in a loop. It mirrors the reference client's
254+
`batchAddRequests`:
255+
256+
- The input is automatically split into chunks that respect both the API's per-call request-count
257+
limit (25) and its request-body byte-size limit (~9 MiB), so there is no need to chunk manually.
258+
- Chunks are sent with up to `options.max_parallel` requests in flight at once (default 5).
259+
- Any request an API call reports as `unprocessed` (typically due to rate limiting) is retried
260+
automatically with exponential backoff, up to `options.max_unprocessed_requests_retries` times
261+
(default 3). A request still unprocessed after every retry is reported in
262+
`BatchRequestsOperationResult::unprocessed_requests` rather than failing the call.
263+
- Every request should set `RequestQueueRequest::unique_key` (or rely on the API's `url`
264+
fallback) so a retried request can be correlated back to the original input.
265+
266+
```rust,no_run
267+
use apify_client::models::RequestQueueRequest;
268+
use apify_client::BatchAddRequestsOptions;
269+
# use apify_client::ApifyClient;
270+
# async fn run(client: ApifyClient) -> Result<(), Box<dyn std::error::Error>> {
271+
let queue = client.request_queues().get_or_create(None).await?;
272+
let queue_client = client.request_queue(&queue.id);
273+
274+
let requests: Vec<RequestQueueRequest> = (0..3)
275+
.map(|i| RequestQueueRequest {
276+
id: None,
277+
url: format!("https://example.com/{i}"),
278+
unique_key: Some(format!("page-{i}")),
279+
method: Some("GET".to_string()),
280+
user_data: None,
281+
extra: Default::default(),
282+
})
283+
.collect();
284+
285+
let result = queue_client
286+
.batch_add_requests(&requests, BatchAddRequestsOptions::default())
287+
.await?;
288+
println!(
289+
"added {} request(s), {} unprocessed",
290+
result.processed_requests.len(),
291+
result.unprocessed_requests.len()
292+
);
293+
# Ok(())
294+
# }
295+
```
296+
297+
`batch_delete_requests` does **not** auto-chunk (matching the reference client): it accepts at
298+
most 25 requests per call, identified by `id` and/or `unique_key` (not the full
299+
`RequestQueueRequest` shape), and returns `ApifyClientError::InvalidArgument` for a larger input.
260300

261301
### `RequestQueueRequest` and request-queue return types
262302

@@ -304,6 +344,14 @@ Relevant return-type fields:
304344
`was_already_handled: bool`.
305345
- `RequestQueueHead`: `limit: i64`, `had_multiple_clients: bool`,
306346
`items: Vec<RequestQueueRequest>`, `extra: Extra` (any other fields returned by the API).
347+
- `LockedRequestQueueHead` (from `list_and_lock_head`): same fields as `RequestQueueHead` plus
348+
`lock_secs: i64`, `queue_has_locked_requests: Option<bool>`, `client_key: Option<String>`.
349+
- `RequestQueueRequestsPage` (from `list_requests`): `limit: i64`, `items: Vec<RequestQueueRequest>`,
350+
`cursor` / `next_cursor` / `exclusive_start_id` (all `Option<String>`) for pagination.
351+
- `BatchRequestsOperationResult` (from `batch_add_requests` / `batch_delete_requests`):
352+
`processed_requests: Vec<ProcessedRequest>`, `unprocessed_requests: Vec<UnprocessedRequest>`.
353+
- `RequestLockInfo` (from `prolong_request_lock`): `lock_expires_at: DateTime<Utc>`.
354+
- `UnlockRequestsResult` (from `unlock_requests`): `unlocked_count: i64`.
307355
- `KeyValueStoreKeysPage`: `limit: i64`, `is_truncated: bool`, `exclusive_start_key`,
308356
`next_exclusive_start_key` (both `Option<String>`), `items: Vec<KeyValueStoreKey>`.
309357

0 commit comments

Comments
 (0)