storages.md

Storages: datasets, key-value stores, request queues

Storage metadata models (Dataset, KeyValueStore, RequestQueue)

get and get_or_create on each storage collection/client return a metadata model from apify_client::models (Dataset, KeyValueStore, RequestQueue). All three share a common core; the .id field is what the examples read to build a per-storage client (client.dataset(&dataset.id), client.key_value_store(&store.id), client.request_queue(&queue.id)):

Field	Type	On	Description
id	String	all three	Unique storage ID (always present); pass to client.dataset(..) / client.key_value_store(..) / client.request_queue(..).
name	Option<String>	all three	Technical name, if the storage is named.
user_id	Option<String>	all three	ID of the owner.
created_at	Option<DateTime<Utc>>	all three	When the storage was created.
modified_at	Option<DateTime<Utc>>	all three	When the storage was last modified.
item_count	Option<i64>	Dataset only	Total number of items in the dataset.
total_request_count	Option<i64>	RequestQueue only	Total number of requests ever added.
extra	Extra	all three	Any other fields returned by the API.

# use apify_client::ApifyClient;
# async fn run(client: ApifyClient) -> Result<(), Box<dyn std::error::Error>> {
let dataset = client.datasets().get_or_create(None).await?;
// Use the metadata `id` to obtain a client for the storage itself.
let dataset_client = client.dataset(&dataset.id);
# let _ = dataset_client;
# Ok(())
# }

Datasets — client.datasets() / client.dataset(id)

DatasetCollectionClient: list(options: StorageListOptions), get_or_create(name: Option<&str>). StorageListOptions: offset, limit, desc, unnamed, ownership.

get_or_create takes Option<&str>: pass Some("my-name") to get-or-create a named storage (reused across runs), or None for an unnamed one. The same signature applies to the key-value-store and request-queue collections.

# use apify_client::ApifyClient;
# async fn run(client: ApifyClient) -> Result<(), Box<dyn std::error::Error>> {
// Named: returns the existing dataset if one with this name exists, else creates it.
let dataset = client.datasets().get_or_create(Some("my-results")).await?;
// Unnamed:
let scratch = client.datasets().get_or_create(None).await?;
# let _ = (dataset, scratch);
# Ok(())
# }

DatasetClient:

Method	Arguments	Returns	Description
get()	—	Option<Dataset>	Dataset metadata.
update(fields)	&impl Serialize	Dataset	Updates metadata.
delete()	—	()	Deletes the dataset.
list_items::<T>(options)	DatasetListItemsOptions	PaginationList<T>	Reads items (pagination via response headers).
push_items(items)	&impl Serialize	()	Appends items (object or array).
get_statistics()	—	Option<Value>	Field statistics.
download_items(format, options)	DownloadItemsFormat, DatasetDownloadOptions	Vec<u8>	Export items as JSON/CSV/XLSX/XML/RSS/HTML.
create_items_public_url(options, expires)	DatasetListItemsOptions, Option<i64>	String	Shareable (HMAC-signed for private) items URL.

DatasetListItemsOptions: offset, limit, desc, fields, output_fields, omit, skip_empty, skip_hidden, clean, unwind, flatten, view, simplified, skip_failed_pages. DatasetDownloadOptions adds attachment, bom, delimiter, skip_header_row, xml_root, xml_row, feed_title, feed_description.

Key-value stores — client.key_value_stores() / client.key_value_store(id)

KeyValueStoreCollectionClient: list(options: StorageListOptions), get_or_create(name: Option<&str>).

KeyValueStoreClient:

Method	Arguments	Returns	Description
get()	—	Option<KeyValueStore>	Store metadata.
update(fields)	&impl Serialize	KeyValueStore	Updates metadata.
delete()	—	()	Deletes the store.
list_keys(options)	ListKeysOptions	KeyValueStoreKeysPage	Lists keys (key-based pagination).
get_records(options)	GetRecordsOptions { collection, prefix, signature }	Vec<u8>	Downloads all records as a ZIP archive (raw bytes).
record_exists(key)	&str	bool	Whether a record exists (HEAD).
get_record(key)	&str	Option<KeyValueStoreRecord>	Reads a record's raw value.
set_record_raw(key, bytes, content_type)	&str, Vec<u8>, &str	()	Stores a raw record.
set_record_json(key, value)	&str, &impl Serialize	()	Stores a JSON record.
delete_record(key)	&str	()	Deletes a record.
get_record_with_options(key, options)	&str, GetRecordOptions { attachment, signature }	Option<KeyValueStoreRecord>	Reads a record with explicit attachment/signature options.
get_record_public_url(key)	&str	String	Shareable (HMAC-signed for private) record URL.
create_keys_public_url(expires)	Option<i64>	String	Shareable keys-list URL.

ListKeysOptions: limit, exclusive_start_key, prefix, collection, signature. KeyValueStoreRecord exposes value: Vec<u8>, content_type, plus as_text() and json::<T>() helpers.

Request queues — client.request_queues() / client.request_queue(id)

RequestQueueCollectionClient: list(options: StorageListOptions), get_or_create(name: Option<&str>).

RequestQueueClient (chainable with_client_key(key) for lock coordination):

Method	Arguments	Returns	Description
get()	—	Option<RequestQueue>	Queue metadata.
update(fields)	&impl Serialize	RequestQueue	Updates metadata.
delete()	—	()	Deletes the queue.
list_head(limit)	Option<i64>	RequestQueueHead	Requests at the head.
add_request(request, forefront)	&RequestQueueRequest, bool	RequestQueueOperationInfo	Adds a request.
get_request(id)	&str	Option<RequestQueueRequest>	Reads a request.
update_request(request, forefront)	&RequestQueueRequest, bool	RequestQueueOperationInfo	Updates a request.
delete_request(id)	&str	()	Deletes a request.
list_and_lock_head(lock_secs, limit)	i64, Option<i64>	Value	Locks head requests.
batch_add_requests(requests, forefront)	&[RequestQueueRequest], bool	Value	Batch add.
batch_delete_requests(requests)	&[impl Serialize]	Value	Batch delete.
list_requests(options)	ListRequestsOptions { limit, exclusive_start_id, cursor, filter }	Value	List requests (cursor/filter pagination).
paginate_requests(page_limit)	Option<i64>	RequestQueueRequestsIterator	Lazy request iterator.
prolong_request_lock(id, lock_secs, forefront)	&str, i64, bool	Value	Extend a lock.
delete_request_lock(id, forefront)	&str, bool	()	Release a lock.
unlock_requests()	—	Value	Release all this client's locks.

RequestQueueRequest and request-queue return types

RequestQueueRequest (from apify_client::models) is the value passed to add_request / update_request and returned by get_request / inside RequestQueueHead. Its fields:

Field	Type	Description
id	Option<String>	Request ID assigned by the API; leave None when adding a new request.
url	String	The URL to process (required).
unique_key	Option<String>	Dedup key (defaults to url server-side when omitted).
method	Option<String>	HTTP method (defaults to GET).
user_data	Option<serde_json::Value>	Arbitrary user data attached to the request.
extra	Extra	Any other fields returned by the API; use Default::default() when constructing.

Construct one and add it to a queue:

use apify_client::models::RequestQueueRequest;
# use apify_client::ApifyClient;
# async fn run(client: ApifyClient) -> Result<(), Box<dyn std::error::Error>> {
let queue = client.request_queues().get_or_create(None).await?;
let queue_client = client.request_queue(&queue.id);

let request = RequestQueueRequest {
    id: None,
    url: "https://example.com/".to_string(),
    unique_key: Some("example".to_string()),
    method: Some("GET".to_string()),
    user_data: None,
    extra: Default::default(),
};
let info = queue_client.add_request(&request, false).await?;
println!("added request {}", info.request_id);

let head = queue_client.list_head(Some(10)).await?;
println!("{} request(s) at the head", head.items.len());
# Ok(())
# }

Relevant return-type fields:

• RequestQueueOperationInfo: request_id: String, was_already_present: bool, was_already_handled: bool.
• RequestQueueHead: limit: i64, had_multiple_clients: bool, items: Vec<RequestQueueRequest>, extra: Extra (any other fields returned by the API).
• KeyValueStoreKeysPage: limit: i64, is_truncated: bool, exclusive_start_key, next_exclusive_start_key (both Option<String>), items: Vec<KeyValueStoreKey>.

Common list container — PaginationList<T>

Offset/limit-paginated list methods (list_items, the various collection list methods, …) return PaginationList<T>. Re-exported at the crate root (apify_client::PaginationList). Fields:

Field	Type	Description
total	i64	Total items available across all pages.
offset	i64	Items skipped at the start.
limit	i64	Max items the API would return for this request.
count	i64	Items actually returned in this page.
desc	bool	Whether the items are in descending order.
items	Vec<T>	The items of this page.

The storage clients are also reachable from a run via run.dataset(), run.key_value_store() and run.request_queue().