feat(ourlog): Add vercel log drain endpoint #5212

AbhiPrasad · 2025-10-03T21:26:34Z

resolves https://linear.app/getsentry/issue/LOGS-389/add-vercel-log-drain-endpoint-to-relay

Building on top of the vercel log transform added in #5209, this PR adds an endpoint for the vercel log drain.

This endpoint is featured flagged, you can see the options automator PR here: https://github.com/getsentry/sentry-options-automator/pull/5367

linear · 2025-10-03T21:26:37Z

LOGS-389 Add Vercel Log Drain Endpoint to Relay

Dav1dde · 2025-10-09T12:39:30Z

relay-server/src/processing/logs/integrations/vercel.rs

+fn parse_logs_data(payload: &[u8]) -> Result<Vec<VercelLog>> {
+    // Try parsing as JSON array first
+    if let Ok(logs) = serde_json::from_slice::<Vec<VercelLog>>(payload) {
+        return Ok(logs);
+    }


Is there no way to differentiate the payload format based on the request, a header or something?

If we can't, you can peek at the first byte ([) to figure out if it is an array or not.

yeah we differentiate based on content type, done with a3db213

Dav1dde · 2025-10-09T12:40:41Z

relay-server/src/processing/logs/integrations/vercel.rs

+    let logs: Vec<VercelLog> = payload_str
+        .lines()
+        .filter_map(|line| serde_json::from_str::<VercelLog>(line.trim()).ok())
+        .collect();


Instead of collecting to a vector here, you could use the iterator and call produce for each item instead. This reduces the amount of necessary allocations.

Dav1dde · 2025-10-09T12:41:24Z

relay-server/src/processing/logs/integrations/vercel.rs

+    if logs.is_empty() {
+        relay_log::debug!("Failed to parse any logs from vercel log drain payload");
+    }


Is that correct? Maybe we just got an empty content? Also the log line is not emitted for [].

I adjusted this in 8879201.

Dav1dde · 2025-10-09T12:44:23Z

relay-server/src/processing/logs/integrations/vercel.rs

+    }
+
+    // Fall back to NDJSON parsing
+    let payload_str = std::str::from_utf8(payload).map_err(|e| {


Conversion to utf8 first isn't necessary and means scanning an extra time across the data. slice::split is probably enough.

Agree, alongside the iterator improvements. Changed with 8879201

Dav1dde · 2025-10-09T12:45:11Z

tests/integration/fixtures/__init__.py

+
+        response = self.post(url, headers=headers, data=data)
+
+        response.raise_for_status()


Suggested change

response.raise_for_status()

response.raise_for_status()

return response

Dav1dde · 2025-10-09T12:47:56Z

tests/integration/test_vercel_logs.py

+    project_config["config"]["retentions"] = {
+        "log": {"standard": 30, "downsampled": 13 * 30},
+    }


Fine to omit, we can expect retentions to work for an integration if it works for the core logs.

Agree, generally moved some test items around in 488f625

Dav1dde · 2025-10-09T12:48:49Z

tests/integration/test_vercel_logs.py

+        headers={"Content-Type": "application/json"},
+    )
+
+    # Check that the items are properly processed via items_consumer


Dav1dde · 2025-10-09T12:49:20Z

tests/integration/test_vercel_logs.py

+        headers={"Content-Type": "application/x-ndjson"},
+    )
+
+    # Check that the items are properly processed via items_consumer


What else (again)?

Dav1dde · 2025-10-09T12:49:32Z

tests/integration/test_vercel_logs.py

+    project_config["config"]["retentions"] = {
+        "log": {"standard": 30, "downsampled": 13 * 30},
+    }


Also mentioned below, we can omit this.

Dav1dde · 2025-10-09T12:49:46Z

tests/integration/test_vercel_logs.py

+        },
+    ]
+
+    # Check outcomes


As opposed to what?

cursor · 2025-10-10T22:42:38Z

relay-server/src/processing/logs/integrations/vercel.rs

+                    count += 1;
+                    let ourlog = relay_ourlogs::vercel_log_to_sentry_log(log);
+                    produce(ourlog);
+                }


Bug: Inconsistent Error Handling in NDJSON Parsing

The NDJSON parser silently drops individual lines that fail to parse, unlike the JSON array parser which fails the entire payload on any error. This inconsistent error handling can lead to silent data loss and make debugging challenging.

AbhiPrasad · 2025-10-10T22:44:34Z

relay-server/src/processing/logs/integrations/vercel.rs

+    }
+
+    if count == 0 {
+        relay_log::debug!("Failed to parse any logs from vercel log drain payload");


This does output when you get empty arrays, but this is unexpected from the log drain endpoint.

After implementing this I also realized that we could return the count from expand, and use that accordingly, we can make a similar refactor to the otlp integration.

Dav1dde · 2025-10-13T14:00:47Z

relay-server/src/integrations/mod.rs

+    // Vercel Log Drain data in a json array payload
+    Json,
+    // Vercel Log Drain data in a newline delimited json payload
+    NDJson,


Suggested change

NDJson,

NdJson,

Even if it looks wrong, according to Rust code this is the proper capitalization.

tests/integration/fixtures/__init__.py

Dav1dde · 2025-10-13T14:05:54Z

tests/integration/test_vercel_logs.py

+TEST_CONFIG = {
+    "outcomes": {
+        "emit_outcomes": True,
+        "batch_size": 1,
+        "batch_interval": 1,
+        "aggregator": {
+            "bucket_interval": 1,
+            "flush_interval": 1,
+        },
+    },
+    "aggregator": {
+        "bucket_interval": 1,
+        "initial_delay": 0,
+    },
+}


Are you sure, because you're just repeating the default config:

relay/tests/integration/fixtures/relay.py

Lines 147 to 158 in ca7e20d

"outcomes": {

"batch_size": 1,

"batch_interval": 1,

"aggregator": {

"bucket_interval": 1,

"flush_interval": 0,

},

},

"aggregator": {

"bucket_interval": 1,

"initial_delay": 0,

},

This didn't used to be the default, I changed that 2 weeks ago, maybe you tried before that?

Dav1dde · 2025-10-13T14:11:29Z

relay-server/src/processing/logs/integrations/vercel.rs

+                    continue;
+                }
+
+                if let Ok(log) = serde_json::from_slice::<VercelLog>(line) {


I think we wouldn't want to swallow errors here and instead emit a single error.

This would mean you need to pass in the RecordKeeper and emit an error. But we can also tackle this separately/afterwards.

We should then also make a test which has a single invalid line.

Then this maybe also addresses the issue with the count afterwards.

AbhiPrasad self-assigned this Oct 3, 2025

AbhiPrasad force-pushed the abhi-vercel-log-drain-endpoint branch from 837b7d7 to 1dac424 Compare October 5, 2025 02:20

jjbayer mentioned this pull request Oct 6, 2025

feat(ourlog): Add vercel log drain transform #5209

Merged

AbhiPrasad force-pushed the abhi-vercel-log-drain-integration branch from 3810607 to ec8cfbf Compare October 6, 2025 23:24

Base automatically changed from abhi-vercel-log-drain-integration to master October 8, 2025 18:02

AbhiPrasad force-pushed the abhi-vercel-log-drain-endpoint branch 6 times, most recently from 771fc43 to c95f053 Compare October 9, 2025 12:29

feat(ourlog): Add vercel log drain endpoint

d25fc99

AbhiPrasad force-pushed the abhi-vercel-log-drain-endpoint branch from c95f053 to d25fc99 Compare October 9, 2025 12:30

AbhiPrasad marked this pull request as ready for review October 9, 2025 12:30

AbhiPrasad requested a review from a team as a code owner October 9, 2025 12:30

This comment was marked as outdated.

Sign in to view

Dav1dde requested changes Oct 9, 2025

View reviewed changes

AbhiPrasad added 3 commits October 10, 2025 13:53

store and use format for vercel logs

a3db213

clean up expand for vercel logs

8879201

clean up integration tests

488f625

cursor bot reviewed Oct 10, 2025

View reviewed changes

AbhiPrasad commented Oct 10, 2025

View reviewed changes

Dav1dde reviewed Oct 13, 2025

View reviewed changes


		response = self.post(url, headers=headers, data=data)

		response.raise_for_status()

	response.raise_for_status()
	response.raise_for_status()
	return response

	"outcomes": {
	"batch_size": 1,
	"batch_interval": 1,
	"aggregator": {
	"bucket_interval": 1,
	"flush_interval": 0,
	},
	},
	"aggregator": {
	"bucket_interval": 1,
	"initial_delay": 0,
	},

feat(ourlog): Add vercel log drain endpoint #5212

Are you sure you want to change the base?

feat(ourlog): Add vercel log drain endpoint #5212

Uh oh!

Conversation

AbhiPrasad commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear bot commented Oct 3, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Oct 10, 2025

Choose a reason for hiding this comment

Bug: Inconsistent Error Handling in NDJSON Parsing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AbhiPrasad commented Oct 3, 2025 •

edited

Loading