feat(sigil): Add sigil.receiver and sigil.write components#6029
feat(sigil): Add sigil.receiver and sigil.write components#6029alexander-akhmetov wants to merge 1 commit intomainfrom
Conversation
|
💻 Deploy preview available (feat(sigil): Add sigil.receiver and sigil.write components): |
db40e88 to
9a73970
Compare
c2d91a7 to
1f1475c
Compare
1f1475c to
ab41f6a
Compare
sigil.receiver listens on POST /api/v1/generations:export and forwards raw request bytes to one or more downstream GenerationsReceiver targets. sigil.write implements GenerationsReceiver by forwarding to remote Sigil ingest endpoints with configurable backoff/retry and tenant header rewriting. Both components are marked StabilityExperimental.
ab41f6a to
7606264
Compare
|
Hey thanks for the pr. Mind explaining why we would want something like this in alloy? AFAICT can see this is just adding a generic proxy to alloy. From sigil docs:
So you should be able to ingest this with otel pipelines right? But I am probably missing something |
|
Hi! Yes, this is basically a proxy. I based it mostly on The reason I thought it could make sense in Alloy is that Sigil is not just OTLP. Traces and metrics go through the normal OTel pipeline, but the full generation data (messages, tools, thinking information, etc.) goes through a separate Sigil ingest API. So even if someone already uses Alloy for OTel, they still need to configure the app separately or add another place for auth, tenant headers, retries, and routing for that generation ingest path. My idea was to let Alloy handle that too. But I also agree the obvious question is whether the right fix is to improve Sigil and upstream OTel standards to make them work together instead of adding this path to Alloy. That is a fair concern. I mainly opened this PR because the separate Sigil ingest path exists today, and this seemed like a practical integration point for users who already run Alloy. So I totally understand if this isn't something you want in Alloy, feel free to close the PR if so :) |
Yeah so while the profiles in pyroscope is just a byte blob it still have labels that you can e.g. perform relabeling on them.
I can see this being annoying / not ideal but you would still have to configure the app to send it to the endpoint you expose with correct port etc but there is a value in centralize e.g. authentication.
Do you think this "generation data" will ever become something like a native otel signal or something we would want to do processing on before storing at the final destination? |
I think it could, but I'm not sure if / when that could happen.
I think processing could be useful, even if just relabeling, filtering or routing based on labels / metadata, etc. but I made it just a raw proxy to start with. |
Brief description of Pull Request
Add
sigil.receiverandsigil.writecomponents for proxying Sigil AI generation ingest traffic through Alloy.sigil.receiverlistens onPOST /api/v1/generations:exportand forwards raw request bytes to downstreamGenerationsReceivertargets.sigil.writeimplementsGenerationsReceiverby forwarding to remote Sigil ingest endpoints with configurable backoff/retry, tenant header rewriting, and fan-out to multiple backends.Both components use the raw-proxy pattern (like
pyroscope.receive_http). Request and response bodies are forwarded as opaque bytes without deserialization. Both are markedStabilityExperimental.Pull Request Details
Sigil SDKs currently send generation records directly to Sigil ingest, bypassing Alloy. This means operators who route all observability traffic through Alloy have a gap in their telemetry gateway for AI signals.
These components close that gap: SDKs point at Alloy's
sigil.receiver, Alloy proxies the traffic throughsigil.writeto the upstream, applying auth header rewriting, retry/backoff, and multi-backend fan-out transparently.Notes to the Reviewer
pyroscope.receive_http(receiver) andpyroscope.write(writer).PR Checklist