⚗️ Partial view updates (experimental) - #4201
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
Bundles Sizes Evolution
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 9e2c9e6 | Docs | Datadog PR Page | Give us feedback! |
1e1e4f8 to
b9f667f
Compare
|
Nice work on this — the diff engine design looks solid, and the REPLACE/MERGE/APPEND strategy categorization is clean. I've been prototyping a parallel implementation against our staging backend and wanted to share some observations. High severity1. No post-assembly strip (~500-650B wasted per view_update)The diff in
These fields have REPLACE semantics — they don't change between updates. In my prototype I added a second pass in 2. No periodic full VIEW refresh (no recovery from dropped events)If any Suggestion: force a full 3. No full VIEW on view endWhen Suggestion: always emit a full Medium severity4.
|
1950458 to
4ca0710
Compare
40aee0f to
c60c158
Compare
|
/to-staging |
|
View all feedbacks in Devflow UI.
Commit c60c158a5c will soon be integrated into staging-12.
Commit c60c158a5c has been merged into staging-12 in merge commit f3d007064e. Check out the triggered DDCI request. If you need to revert this integration, you can use the following command: |
Integrated commit sha: c60c158 Co-authored-by: mormubis <adrian.delarosa@datadoghq.com>
c60c158 to
52a76bb
Compare
|
/to-staging |
|
View all feedbacks in Devflow UI.
Commit 52a76bbd97 will soon be integrated into staging-14.
Commit 52a76bbd97 has been merged into staging-14 in merge commit 1cf29217c5. Check out the triggered DDCI request. If you need to revert this integration, you can use the following command: |
Integrated commit sha: 52a76bb Co-authored-by: mormubis <adrian.delarosa@datadoghq.com>
75e714c to
5e79702
Compare
|
/to-staging |
|
View all feedbacks in Devflow UI.
Commit 08a05063bc will soon be integrated into staging-16.
Commit 08a05063bc has been merged into staging-16 in merge commit cd062ff830. If you need to revert this integration, you can use the following command: |
Integrated commit sha: 08a0506 Co-authored-by: mormubis <adrian.delarosa@datadoghq.com>
08a0506 to
279d277
Compare
1ce910d to
c6f3d12
Compare
| if (viewId !== lastSentView?.view.id) { | ||
| lastSentView = serverRumEvent | ||
| viewUpdatesSinceCheckpoint = 0 | ||
| batch.upsert(serverRumEvent, viewId) |
There was a problem hiding this comment.
This sound relevant, does the backend supports this properly?
| export function assembleViewUpdateEvent( | ||
| current: AssembledRumEvent, | ||
| last: AssembledRumEvent | ||
| ): AssembledRumEvent | undefined { |
There was a problem hiding this comment.
suggestion: Types around this function could be greatly improved like this:
export function assembleViewUpdateEvent(current: RumViewEvent, last: RumViewEvent): RumViewUpdateEvent | undefined {
const diff = diffMerge(current, last, {
// context, connectivity, usr, device, privacy are objects — use REPLACE to avoid partial updates
replaceKeys: new Set(['view.custom_timings', 'context', 'connectivity', 'usr', 'device', 'privacy']),
appendKeys: new Set(['_dd.page_states']),
// Ignore always-required fields — they are added back via combine regardless of changes
ignoreKeys: new Set([
'date',
'type',
'application',
'session',
'view.id',
'view.url',
'_dd.document_version',
'_dd.format_version',
]),
})
if (!diff) {
return undefined
}
// Restore the ignoreKeys — backend needs them on every event
return combine(diff, {
type: RumEventType.VIEW_UPDATE,
date: current.date,
application: current.application,
session: current.session,
view: {
id: current.view.id,
url: current.view.url,
},
_dd: {
document_version: current._dd.document_version,
format_version: current._dd.format_version,
},
})
}By using the proper types RumViewEvent and RumViewUpadetEvent, you can remove all casts.
| }), | ||
| }) | ||
|
|
||
| let lastSentView: AssembledRumEvent | undefined |
There was a problem hiding this comment.
issue: keeping track of a single view is an issue because it is possible to get updates for a view after it becomes inactive
There was a problem hiding this comment.
As we discussed offline, there can only be one active view. When startView is called, the previous view gets is_active: false immediately. The view-end always sends a full upsert before the diff logic. One edge case: if a late view-end for A arrives after B started, it clears B's lastSentView. Not a correctness issue but B's next update would be a full upsert instead of a diff. I can fix that by only clearing when the ending view matches the tracked view.
| // Use setViewName to trigger unthrottled view updates (unlike addAction which is | ||
| // throttled to THROTTLE_VIEW_UPDATE_PERIOD=3s, setViewName calls triggerViewUpdate directly). | ||
| // We need more than PARTIAL_VIEW_UPDATE_CHECKPOINT_INTERVAL (100) updates to trigger a checkpoint. | ||
| // All calls are batched in a single evaluate to avoid 102 round-trips to the browser. |
There was a problem hiding this comment.
suggestion: This is likely a bug, no? setViewName should schedule a view update instead of sending the update directly.
There was a problem hiding this comment.
I think you're right. setViewName calls triggerViewUpdate directly instead of scheduleViewUpdate. I'll file it separately and fix forward.
| // Should have at least one view_update | ||
| const viewUpdateEvents = intakeRegistry.rumViewUpdateEvents | ||
| expect(viewUpdateEvents.length).toBeGreaterThanOrEqual(1) | ||
|
|
||
| // All events share the same view.id | ||
| const viewId = viewEvents[0].view.id | ||
| for (const update of viewUpdateEvents) { | ||
| expect(update.view.id).toBe(viewId) | ||
| } |
There was a problem hiding this comment.
suggestion: add an assertion to show that a view update contains the updated view.action.count field, so we understand why we have an addAction above
|
|
||
| export type AssembledRumEvent = ( | ||
| | RumViewEvent | ||
| | RumViewUpdateEvent |
There was a problem hiding this comment.
suggestion: This types depicts the RUM events that are assembled by assembly.ts. Since ViewUpdate are not assembled by assembly.ts, it shouldn't be listed there.
| }) | ||
| }) | ||
|
|
||
| describe('startRumBatch partial_view_updates routing', () => { |
There was a problem hiding this comment.
suggestion: this describe is empty, you could remove it
- Add PARTIAL_VIEW_UPDATES to ExperimentalFeature enum - Add VIEW_UPDATE to RumEventType, RawRumViewUpdateEvent, and RawRumEvent union - Add viewDiff.ts: isEqual and diffMerge utilities implementing MERGE / REPLACE / APPEND strategies for computing minimal diffs between assembled view events
When partial_view_updates is enabled, startRumBatch intercepts assembled view events and sends view_update diffs instead of full views for intermediate updates. Key design: diff runs post-assembly so beforeSend always sees full view events (backward-compatible). view_update events intentionally bypass the assembly pipeline — they are a bandwidth optimization, not a customer-visible event type. - computeAssembledViewDiff: diffs two assembled view events, always including required routing fields (view.id, view.url, _dd.document_version, format_version) - Routing state machine: handles new view / view-end / checkpoint / diff cases - view-end events (is_active: false) always sent as full view - Full view checkpoint every 100 updates for backend recovery - Exclude view_update from trackEventCounts and assembly beforeSend guard - Add E2E tests covering all routing cases
…t or APPEND array
…e safety fallback
5a7aa93 to
9e2c9e6
Compare
4 participants
Motivation
Every periodic view update was sending the full view payload even when only one counter changed. Benchmarks showed 50–90% of the data was redundant. This implements the partial view updates RFC to reduce bandwidth by sending only changed fields in subsequent view events.
Aligned with rum-events-format #355 (now merged).
Changes
When
partial_view_updatesis enabled, the SDK sends the first event perview.idas a fullview, then sendsview_updatediffs with only changed fields. The diff runs post-assembly instartRumBatch.tssobeforeSendalways sees the full event (backward-compatible).view_updateevents bypass the assembly pipeline intentionally, they are a bandwidth optimization and not a customer-visible event type.A full
viewcheckpoint is sent every 100 updates for backend recovery. Checkpoints can be disabled withpartial_view_updates_no_checkpoint.view_updateevents usebatch.addinstead ofupsert, so a batch can contain a fullviewfollowed byview_updateevents. This is intentional: if we consolidated them, we wouldn't be able to tell if the backend missed an intermediate update or it was never sent.Test instructions
Or manually:
Checklist