Skip to content

Commit 094dc6e

Browse files
author
Jade Wang
committed
docs: consolidate implementation into 2 phases (Thrift gaps first, SEA second)
Co-authored-by: Isaac
1 parent 0a27469 commit 094dc6e

File tree

1 file changed

+37
-30
lines changed

1 file changed

+37
-30
lines changed

docs/designs/fix-telemetry-gaps-design.md

Lines changed: 37 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -631,37 +631,44 @@ The following proto fields are **not applicable** to the C# ADBC driver and will
631631

632632
## Implementation Priority
633633

634-
### Phase 1: E2E Test Infrastructure (Test-First)
634+
### Phase 1: Thrift Telemetry Gaps (Missing Fields, ChunkDetails, Behavioral Changes)
635+
636+
Fix all gaps in the existing Thrift telemetry pipeline first, since the infrastructure is already in place.
637+
638+
**E2E Tests (test-first):**
635639
1. Build E2E test infrastructure using `CapturingTelemetryExporter` to assert proto field values
636-
2. Write E2E tests for all currently populated proto fields (Thrift) - these establish the baseline
637-
3. Write failing E2E tests for missing fields (auth_type, WorkspaceId, runtime_vendor, etc.) - these drive Phase 2-5
638-
4. Write failing E2E tests for SEA telemetry (expect telemetry events from SEA connections) - these drive Phase 2
639-
5. Write failing E2E tests for ChunkDetails fields - these drive Phase 4
640-
641-
### Phase 2: SEA Telemetry (Highest Priority - Zero Coverage Today)
642-
6. Extract `TelemetryHelper` from `DatabricksConnection` for shared use
643-
7. Wire `InitializeTelemetry()` into `StatementExecutionConnection` with `mode=SEA`
644-
8. Add `EmitTelemetry()` to `StatementExecutionStatement`
645-
9. Wire telemetry dispose/flush into `StatementExecutionConnection.Dispose()`
646-
10. Verify SEA E2E tests from Phase 1 now pass
647-
648-
### Phase 3: Missing Fields (Low Risk)
649-
11. Populate `runtime_vendor` and `client_app_name` in DriverSystemConfiguration
650-
12. Populate `auth_type` on root log
651-
13. Populate additional DriverConnectionParameters (enable_arrow, rows_fetched_per_block, etc.)
652-
14. Set `WorkspaceId` in TelemetrySessionContext
653-
15. Verify missing-field E2E tests from Phase 1 now pass
654-
655-
### Phase 4: ChunkDetails Wiring (Medium Risk - Crosses Component Boundaries)
656-
16. Add `ChunkMetrics` aggregation to `CloudFetchDownloader`
657-
17. Expose metrics via `CloudFetchReader.GetChunkMetrics()`
658-
18. Call `SetChunkDetails()` in `DatabricksStatement.EmitTelemetry()` and `StatementExecutionStatement.EmitTelemetry()`
659-
19. Verify ChunkDetails E2E tests from Phase 1 now pass
660-
661-
### Phase 5: Other Behavioral Changes (Medium Risk)
662-
20. Track `retry_count` on SqlExecutionEvent
663-
21. Mark internal calls with `is_internal_call = true`
664-
22. Add metadata operation telemetry (GetObjects, GetTableTypes)
640+
2. Write E2E tests for all currently populated proto fields (Thrift) - establish the baseline
641+
3. Write failing E2E tests for missing fields (auth_type, WorkspaceId, runtime_vendor, client_app_name, etc.)
642+
4. Write failing E2E tests for ChunkDetails fields
643+
5. Write failing E2E tests for metadata operations and internal call tracking
644+
645+
**Implementation:**
646+
6. Populate `runtime_vendor` and `client_app_name` in DriverSystemConfiguration
647+
7. Populate `auth_type` on root log
648+
8. Populate additional DriverConnectionParameters (enable_arrow, rows_fetched_per_block, etc.)
649+
9. Set `WorkspaceId` in TelemetrySessionContext
650+
10. Add `ChunkMetrics` aggregation to `CloudFetchDownloader`
651+
11. Expose metrics via `CloudFetchReader.GetChunkMetrics()`
652+
12. Call `SetChunkDetails()` in `DatabricksStatement.EmitTelemetry()`
653+
13. Track `retry_count` on SqlExecutionEvent
654+
14. Mark internal calls with `is_internal_call = true`
655+
15. Add metadata operation telemetry (GetObjects, GetTableTypes)
656+
16. Verify all Phase 1 E2E tests pass
657+
658+
### Phase 2: SEA Telemetry (Wire Telemetry into StatementExecutionConnection)
659+
660+
Once Thrift telemetry is complete, extend coverage to the SEA protocol using the shared `TelemetryHelper`.
661+
662+
**E2E Tests (test-first):**
663+
17. Write failing E2E tests for SEA telemetry (expect telemetry events from SEA connections)
664+
665+
**Implementation:**
666+
18. Extract `TelemetryHelper` from `DatabricksConnection` for shared use (already done - verify coverage)
667+
19. Wire `InitializeTelemetry()` into `StatementExecutionConnection` with `mode=SEA`
668+
20. Add `EmitTelemetry()` to `StatementExecutionStatement`
669+
21. Wire telemetry dispose/flush into `StatementExecutionConnection.Dispose()`
670+
22. Wire `SetChunkDetails()` in `StatementExecutionStatement.EmitTelemetry()` for SEA CloudFetch
671+
23. Verify all Phase 2 SEA E2E tests pass
665672

666673
---
667674

0 commit comments

Comments
 (0)