[da-vinci] Add server fallback blob transfer metrics - #2921
Open
shresthhh wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds richer observability for Da Vinci server-fallback blob transfer by tracking which source (Da Vinci peer vs Venice server) was used per request and recording bounded outcomes/reasons across Tehuti and OpenTelemetry, while keeping existing discovery/transfer control-flow unchanged.
Changes:
- Extend blob peer discovery responses to preserve provenance of which discovered hosts are Venice servers.
- Add new blob-transfer metrics: source/outcome request counters + Kafka fallback reason counter (OTel), and source success/fail request counters (Tehuti).
- Introduce a typed HTTP exception to retain response status codes for outcome classification and add/extend unit tests accordingly.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/venice-common/src/test/java/com/linkedin/venice/blobtransfer/ServerBlobFinderTest.java | Asserts server host provenance is populated in discovery response. |
| internal/venice-common/src/test/java/com/linkedin/venice/blobtransfer/ServerAndDaVinciBlobFinderTest.java | Expands coverage for server-host provenance and host normalization behavior. |
| internal/venice-common/src/test/java/com/linkedin/venice/blobtransfer/MetadataBasedServerBlobFinderTest.java | Validates server-host provenance is set for metadata-based discovery. |
| internal/venice-common/src/main/java/com/linkedin/venice/blobtransfer/ServerBlobFinder.java | Populates serverHostNames for server-based discovery. |
| internal/venice-common/src/main/java/com/linkedin/venice/blobtransfer/ServerAndDaVinciBlobFinder.java | Preserves server-host provenance across composite discovery and normalizes hostnames. |
| internal/venice-common/src/main/java/com/linkedin/venice/blobtransfer/MetadataBasedServerBlobFinder.java | Populates serverHostNames from discovered replicas. |
| internal/venice-common/src/main/java/com/linkedin/venice/blobtransfer/BlobPeersDiscoveryResponse.java | Adds serverHostNames field to carry server provenance through discovery. |
| internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensionsTest.java | Adds dimension-name assertions for new blob-transfer dimensions. |
| internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceBlobTransferSourceTest.java | New test fixture for the blob-transfer source dimension values. |
| internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceBlobTransferOutcomeTest.java | New test fixture for the blob-transfer outcome dimension values. |
| internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensions.java | Registers new OTel dimensions for blob transfer source/outcome. |
| internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceBlobTransferSource.java | New enum dimension for request source (Da Vinci peer vs Venice server). |
| internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceBlobTransferOutcome.java | New enum dimension for bounded outcomes/reasons. |
| internal/venice-client-common/src/main/java/com/linkedin/venice/exceptions/VeniceBlobTransferHttpException.java | New exception type to preserve HTTP status for metric classification. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/ServerMetricEntityTest.java | Updates expected metric entity count after adding new metrics. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/BlobTransferStatsTestUtils.java | Adds helpers to build request/fallback OTel attribute sets. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/BlobTransferStatsTest.java | Adds Tehuti tests for request counts by source/success. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/BlobTransferOtelStatsTest.java | Adds OTel tests for request counts and Kafka fallback reasons. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/BlobTransferOtelMetricEntityTest.java | Validates new metric entity definitions and dimensions. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/AggVersionedBlobTransferStatsTest.java | Verifies dual recording (Tehuti + OTel) for new request metric. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/blobtransfer/TestP2PFileTransferClientHandler.java | Updates assertions to expect the new typed HTTP exception. |
| clients/da-vinci-client/src/test/java/com/linkedin/davinci/blobtransfer/TestNettyP2PBlobTransferManager.java | Extends coverage for per-attempt source/outcome recording and fallback reasons. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/BlobTransferStatsReporter.java | Registers new Tehuti gauges for request counts by source/success. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/BlobTransferStats.java | Adds Tehuti sensors/counts and APIs for request counts by source/success. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/BlobTransferOtelStats.java | Adds OTel recording APIs for request counts and Kafka fallback reasons. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/BlobTransferOtelMetricEntity.java | Defines new OTel metric entities for request count and Kafka fallback count. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/AggVersionedBlobTransferStats.java | Adds aggregate recording methods for request counts and Kafka fallback reasons. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/NettyP2PBlobTransferManager.java | Classifies per-attempt source/outcome, aggregates failure reasons, and records fallback metrics. |
| clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/client/P2PFileTransferClientHandler.java | Throws typed HTTP exception for non-OK responses to retain status codes for classification. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+304
to
+308
| Throwable cause = unwrap(ex); | ||
| if (cause instanceof VenicePeersConnectionException) { | ||
| // error case 3: failed to connect to the peer, move to the next possible host | ||
| LOGGER.error(PEER_CONNECTION_EXCEPTION_MSG, replicaId, chosenHost, ex.getMessage()); | ||
| } else if (ex.getCause() instanceof VeniceBlobTransferFileNotFoundException) { | ||
| } else if (cause instanceof VeniceBlobTransferFileNotFoundException) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
Stateful CDC clients can now fall back from Da Vinci peers to Venice servers for blob bootstrap.
Existing blob-transfer metrics predate this multi-source flow and cannot show whether the new server-fallback path is
being used, how reliably each source performs, or why blob transfer ultimately falls back to Kafka.
This PR adds the observability required to certify, roll out, and operate the feature safely by attributing attempts to
their source, recording bounded outcomes, and reporting Kafka fallback reasons.
Solution
Add narrowly scoped observability without changing discovery order, transfer sequencing, retry behavior, feature
flags, or concurrency limits:
DAVINCI_PEERorVENICE_SERVERand a bounded outcome.failures.
Code changes
No configuration was added. The new error logs are emitted only if metric recording itself throws; they follow the
existing stats error-path pattern and are not emitted for normal transfer failures.
Concurrency-Specific Checks
Both reviewer and PR author to verify
synchronized,RWLock) are used where needed.ConcurrentHashMap,CopyOnWriteArrayList).The new attempt aggregation is confined to the existing sequential per-partition future chain. Metric recording is
best effort and does not alter transfer completion or retry control flow.
How was this PR tested?
Ran
spotlessApply,spotlessCheck, and focused unit tests for blob-transfer discovery, manager behavior, clientresponse handling, Tehuti/OTel stats, metric definitions, and metric dimensions.
Does this PR introduce any user-facing or breaking changes?