Skip to content

Commit 0746676

Browse files
feat(safePushData): detailed error reporting and result tracking (#4)
* safePushData: log which fields failed on each retry round Each failing round now reports the set of offending field paths (with the AJV keyword) instead of just an item count, split into fields we repaired and fields that forced a drop. The give-up log lists the fields still failing. The set is intentionally not a per-item breakdown: one bad field usually appears on many items in a batch, and array indices collapse into `/tags/[]` for the same reason. The list is capped at 20 entries with the overflow counted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ek1iV9QVKioLZ3EiL4cPYc * safePushData: return pushFn's result, rename result fields, fix repair logic BREAKING: the result shape now names what it holds — `pushed` -> `pushedCount`, `dropped` -> `droppedItems`, `attempts` -> `attemptCount`. `*Count` is a number, `*Items` is an array of objects. Whatever `pushFn` resolves to is handed back as `pushResult` (the successful call's value; absent when every item was dropped). `PushFn<T, R>` and `SafePushDataResult<T, R>` carry the type through. Logic fixes found while reviewing: - Hitting the attempt cap used to drop the *entire* remaining batch, including items the API never complained about — the exact data loss this wrapper exists to prevent. Now only the items still failing are dropped and the survivors get one final push of their own. - Sibling array elements were spliced front-to-back, so the second `/tags/N` path pointed at a shifted array: a valid element could be deleted while the invalid one stayed. Errors are now applied deepest- and highest-index-first. - A nested `required` deleted the whole parent object; it now gets the same placeholder treatment as a root-level one, at any depth. - An item whose errors are all unactionable (paths it doesn't have) was re-pushed unchanged until the cap, burning every attempt for the rest of the batch. It's dropped on the spot instead. - Give-up drops reported stale errors from an earlier round; the per-item errors are now reset each round, so they say why the item is failing *now*. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ek1iV9QVKioLZ3EiL4cPYc * safePushData: broaden the test suite to 53 cases Covers the gaps left by the previous round: additionalProperties at the root and nested, mixed error kinds in a single round, JSON Pointer escaping on placeholder paths, per-item state surviving a splice, dropped items reporting the caller's original, non-object items, invalid items the API sends no errors for, the salvage push (its pushResult, its rejection, non-schema errors from it), maxAttempts: 1, the guard's negative cases, log label formatting, and an end-to-end mixed batch. Each of the five logic fixes was mutation-checked: reverting the repair ordering, the pointer escaping, the salvage push, the no-progress drop, or the nested `required` handling makes the corresponding tests fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ek1iV9QVKioLZ3EiL4cPYc --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent f10f256 commit 0746676

3 files changed

Lines changed: 1109 additions & 160 deletions

File tree

README.md

Lines changed: 51 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,20 @@ await Actor.init();
1717
const result = await safePushData((batch) => Actor.pushData(batch), items);
1818

1919
console.log(result);
20-
// { pushed: 2, dropped: [...], attempts: 2 }
20+
// { pushedCount: 2, droppedItems: [...], attemptCount: 2, pushResult: undefined }
2121
```
2222

2323
Accepts a single item or an array. `pushFn` is the first positional arg
2424
and is required — the library itself never imports the Apify SDK, and a
2525
CI check forbids `.pushData(` from appearing anywhere in the source.
2626

27+
Whatever `pushFn` resolves to comes back as `pushResult`, so a push
28+
function with a meaningful return value stays usable:
29+
30+
```ts
31+
const { pushResult } = await safePushData((batch) => client.dataset(id).pushItems(batch), items);
32+
```
33+
2734
## Performance notes
2835

2936
The happy path is a single `try/await pushFn(items)` with **no extra
@@ -38,11 +45,12 @@ every AJV error per item:
3845

3946
| Error | Action |
4047
| ------------------------------------ | ---------------------------------------------------------------------------------- |
41-
| `required` at the root | Set the missing field to `null` and mark the path as a placeholder. |
42-
| `additionalProperties` at the root | Delete the unknown property. |
48+
| `required` (at any depth) | Set the missing field to `null` and mark the path as a placeholder. |
49+
| `additionalProperties` (any depth) | Delete the unknown property. |
4350
| `type` / `format` / etc. at the root | Item itself is the wrong shape → dropped. |
4451
| Constraint on a **placeholder** path | Replace with a type-aware default (see below). If no default is known → dropped. |
4552
| Constraint on **user-supplied** data | Delete the field. If the schema later marks it required, a placeholder takes over. |
53+
| Nothing on the item was actionable | Item is dropped — an unchanged item would fail identically on the next push. |
4654

4755
### Placeholder defaults
4856

@@ -73,19 +81,52 @@ The retry loop chases one layer of errors per round
7381
(`required``type` → push) until either the push succeeds
7482
or `maxAttempts` (default 5) is hit.
7583

84+
### When the attempt cap is hit
85+
86+
Items still failing on the last allowed attempt are dropped — but the rest
87+
of the batch is **not** lost with them. Because a rejected push stores
88+
nothing at all, the wrapper drops the incurable items and then makes one
89+
final push with the survivors, which the API already validated in the
90+
previous round. That final push is counted in `attemptCount`, so a run that
91+
exhausts `maxAttempts: 5` can report `attemptCount: 6`.
92+
93+
## Logging
94+
95+
Every failed round logs which fields went wrong, so you can fix the schema
96+
(or the scraper) without digging through the returned `droppedItems`:
97+
98+
```
99+
safePushData: schema validation failed on attempt 1: 12 invalid item(s); repaired fields: /age (type), /name (required), /tags/[] (type); dropped 2 item(s) on unfixable fields: /email (format); retrying with 10 item(s).
100+
safePushData: gave up after 5 attempts; dropped 3 item(s) still failing on fields: /sku (pattern); pushing the 9 valid item(s) left.
101+
```
102+
103+
The field list is a **set**, not a per-item breakdown — one bad field
104+
usually shows up on many items in a batch, and knowing which item had which
105+
problem rarely changes what you do about it. Array indices collapse
106+
(`/tags/0`, `/tags/7``/tags/[]`) for the same reason, and the list is
107+
capped at 20 entries with the rest reported as `(+N more)`.
108+
76109
## Options
77110

78-
| Option | Type | Default | Notes |
79-
| ------------- | -------- | ------- | -------------------- |
80-
| `maxAttempts` | `number` | `5` | Hard cap on retries. |
111+
| Option | Type | Default | Notes |
112+
| ------------- | -------- | ------- | -------------------------------------------------------------- |
113+
| `maxAttempts` | `number` | `5` | Cap on repair rounds, plus the final salvage push if it's hit. |
81114

82115
## Return shape
83116

117+
Names say what they hold: `*Count` is a number, `*Items` is an array of
118+
objects.
119+
84120
```ts
85-
interface SafePushDataResult<T> {
86-
pushed: number;
87-
dropped: { item: T; errors: ValidationError[] }[];
88-
attempts: number;
121+
interface SafePushDataResult<T, R = unknown> {
122+
/** How many of the caller's items made it into the dataset. */
123+
pushedCount: number;
124+
/** The items we couldn't repair, each with the errors that doomed it. */
125+
droppedItems: { item: T; errors: ValidationError[] }[];
126+
/** How many times `pushFn` was actually called. */
127+
attemptCount: number;
128+
/** What the successful `pushFn` call resolved to; absent if none did. */
129+
pushResult?: R;
89130
}
90131
```
91132

0 commit comments

Comments
 (0)