feat(llmobs): add dataset record tags - #9810
Conversation
Overall package sizeSelf size: 8.2 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.72 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
BenchmarksBenchmark execution time: 2026-08-14 20:07:02 Comparing candidate commit 71c9b0d in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2278 metrics, 14 unstable metrics.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #9810 +/- ##
==========================================
- Coverage 98.53% 93.68% -4.85%
==========================================
Files 975 959 -16
Lines 144058 141841 -2217
Branches 12379 10848 -1531
==========================================
- Hits 141941 132884 -9057
- Misses 2117 8957 +6840 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
7f7f5fa to
f1e0c3b
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f916375225
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (!Array.isArray(tags)) return result | ||
| for (const tag of tags) { | ||
| const separator = tag.indexOf(':') | ||
| if (separator > 0) result[tag.slice(0, separator)] = tag.slice(separator + 1) |
There was a problem hiding this comment.
When a dataset record has multiple tags with the same key, which the new string[] tag API allows (for example topic:math and topic:logic), this assignment overwrites the earlier value. Both the LLMObs annotate path and the fallback toSpan() path use this helper, so experiment spans only carry the last record tag and cannot be filtered or grouped by the full record tag set. Please preserve repeated keys, or append the original record tag strings, instead of collapsing them into one object property.
Useful? React with 👍 / 👎.
| this.#pendingTagOperations.delete(recordId) | ||
| const update = this.#updatedRecordsById.get(recordId) | ||
| if (update) { | ||
| delete update.tagOperations | ||
| if (Object.keys(update).length === 1) this.#updatedRecordsById.delete(recordId) |
There was a problem hiding this comment.
If an existing record tagged split:eval is pushed after removeTags(0, ['split:eval']) and the caller re-adds that tag before the push resolves, mergeTagOperations() returns empty and this branch drops the current update. After the in-flight batch commits the removal, #clearCommittedChanges() has no pending update to compare, so no compensating add is sent and the local dataset says the tag exists while the backend record has lost it. Please retain enough state to reconcile against the in-flight payload instead of deleting the pending tag operation outright.
Useful? React with 👍 / 👎.
| return output | ||
| } | ||
|
|
||
| function serializedTagOperations (operations) { |
There was a problem hiding this comment.
why do we need to serialize methods
There was a problem hiding this comment.
thats the api interface for tags: https://github.com/DataDog/dd-source/pull/320916
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 71c9b0dcd2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| records, | ||
| datasetVersion, | ||
| latestVersion | ||
| datasetVersion ?? latestVersion, |
There was a problem hiding this comment.
When pullDataset() is called without an explicit version while dataset metadata lags ingestion—the exact case this change is handling—the unversioned records request can return the newest snapshot, but this fallback still assigns the older metadata latestVersion to the returned dataset. Experiment.run() subsequently sends that value as dataset_version, associating the experiment with an older snapshot while executing the newer records; resolve the version of the records response or leave it unset instead of substituting stale metadata.
Useful? React with 👍 / 👎.
| input: record.inputData, | ||
| expectedOutput: record.expectedOutput ?? null, | ||
| metadata: record.metadata ?? {}, | ||
| ...(record.tags === undefined ? {} : { tags: record.tags }), |
There was a problem hiding this comment.
When LLMObs is disabled or credentials are missing and the caller omits the optional tags argument, this conditional leaves record.tags undefined, whereas enabled datasets always expose tags: [] and the public Dataset.records() type declares tags: string[]. Code that safely iterates record.tags in enabled mode can therefore throw only in the no-op path; initialize it to a copied empty array so disabling the subsystem remains graceful.
AGENTS.md reference: AGENTS.md:L222-L225
Useful? React with 👍 / 👎.
| if (Object.hasOwn(operations, 'replace')) { | ||
| const replaced = new Set(operations.replace) | ||
| for (const tag of tags) { | ||
| if (operation === 'add') replaced.add(tag) |
There was a problem hiding this comment.
the op can only be add or delete at this point?
2 participants
Summary
update()anddelete()operations with batched insert/update/delete/tag mutations.nullvalues in sparse update payloads and reconcile local changes safely after concurrent pushes.API interface
createDataset()record options also accepttags?: string[]. Tag operations and record updates/deletes are kept in local state and sent together throughbatchUpdateDatasetRecords()onpush().Test plan
dd-trace-js
nullvalues, deletes, batching, retries, and concurrent local edits.llm-observability examples
The runnable validation scripts are tracked in llm-observability PR #93, on branch
feat/nodejs-dataset-operations-tags. The PR extends the dataset example with the dataset operations and tag flows covered by this change.cd experiments/nodejs npm install npm run dataset npm run validate:allThe dataset example reports initial and generated IDs, initial record tags,
addTags(),removeTags(),replaceTags(), tag-filtered pulls,update(),delete(), batched pushes, version changes, follow-up pulls, and experiment execution over a tagged dataset slice. It avoids assertion-heavy output and exits non-zero only for SDK or backend errors. Live validation should be run with Datadog credentials, for example:This pull request is no longer stacked on #9809; the local dataset refactor has been merged into this branch and the tags work is based on current
master.