Skip to content

Commit ab41f6a

Browse files
alexander-akhmetovalexander-akhmetov
committed
feat(sigil): Add sigil.receiver and sigil.write components
sigil.receiver listens on POST /api/v1/generations:export and forwards raw request bytes to one or more downstream GenerationsReceiver targets. sigil.write implements GenerationsReceiver by forwarding to remote Sigil ingest endpoints with configurable backoff/retry and tenant header rewriting. Both components are marked StabilityExperimental.
1 parent b6cb5da commit ab41f6a

File tree

20 files changed

+1537
-0
lines changed

20 files changed

+1537
-0
lines changed

‎.github/ISSUE_TEMPLATE/blank.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ body:
199199
- remote.kubernetes.secret
200200
- remote.s3
201201
- remote.vault
202+
- sigil.receiver
203+
- sigil.write
202204
# End components list
203205
- type: textarea
204206
attributes:

‎.github/ISSUE_TEMPLATE/bug_report.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ body:
199199
- remote.kubernetes.secret
200200
- remote.s3
201201
- remote.vault
202+
- sigil.receiver
203+
- sigil.write
202204
# End components list
203205
- type: textarea
204206
attributes:

‎.github/ISSUE_TEMPLATE/docs.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,8 @@ body:
202202
- remote.kubernetes.secret
203203
- remote.s3
204204
- remote.vault
205+
- sigil.receiver
206+
- sigil.write
205207
# End components list
206208
- type: textarea
207209
id: feedback

‎.github/ISSUE_TEMPLATE/feature_request.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ body:
199199
- remote.kubernetes.secret
200200
- remote.s3
201201
- remote.vault
202+
- sigil.receiver
203+
- sigil.write
202204
# End components list
203205
- type: textarea
204206
attributes:

‎.github/ISSUE_TEMPLATE/proposal.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ body:
199199
- remote.kubernetes.secret
200200
- remote.s3
201201
- remote.vault
202+
- sigil.receiver
203+
- sigil.write
202204
# End components list
203205
- type: textarea
204206
attributes:

‎docs/sources/reference/components/sigil/_index.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/sigil/
3+
description: Learn about the sigil components in Grafana Alloy
4+
title: sigil
5+
weight: 100
6+
---
7+
8+
# `sigil`
9+
10+
This section contains reference documentation for the `sigil` components.
11+
12+
{{< section >}}

‎docs/sources/reference/components/sigil/sigil.receiver.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/sigil/sigil.receiver/
3+
description: Learn about sigil.receiver
4+
labels:
5+
stage: experimental
6+
products:
7+
- oss
8+
title: sigil.receiver
9+
---
10+
11+
# `sigil.receiver`
12+
13+
{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}
14+
15+
`sigil.receiver` receives Sigil AI generation export requests over HTTP and forwards them to `sigil.write` components or other components capable of receiving generations.
16+
17+
The component starts an HTTP server that accepts `POST /api/v1/generations:export` requests.
18+
Request and response bodies are forwarded as opaque bytes without deserialization.
19+
20+
## Usage
21+
22+
```alloy
23+
sigil.receiver "<LABEL>" {
24+
forward_to = <RECEIVER_LIST>
25+
}
26+
```
27+
28+
## Arguments
29+
30+
You can use the following arguments with `sigil.receiver`:
31+
32+
| Name | Type | Description | Default | Required |
33+
| --------------- | ---------------------------- | ---------------------------------------------------- | -------- | -------- |
34+
| `forward_to` | `list(GenerationsReceiver)` | List of receivers to send generations to. | | yes |
35+
| `max_request_body_size` | `string` | Maximum allowed request body size (e.g. `"20MiB"`). | `"20MiB"`| no |
36+
37+
## Blocks
38+
39+
You can use the following blocks with `sigil.receiver`:
40+
41+
{{< docs/alloy-config >}}
42+
43+
| Block | Description | Required |
44+
| ---------------------- | -------------------------------------------------- | -------- |
45+
| [`http`][http] | Configures the HTTP server that receives requests. | no |
46+
| `http` > [`tls`][tls] | Configures TLS for the HTTP server. | no |
47+
48+
[http]: #http
49+
[tls]: #tls
50+
51+
{{< /docs/alloy-config >}}
52+
53+
### `http`
54+
55+
{{< docs/shared lookup="reference/components/server-http.md" source="alloy" version="<ALLOY_VERSION>" >}}
56+
57+
### `tls`
58+
59+
The `tls` block configures TLS for the HTTP server.
60+
61+
{{< docs/shared lookup="reference/components/server-tls-config-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
62+
63+
## Exported fields
64+
65+
`sigil.receiver` doesn't export any fields.
66+
67+
## Component health
68+
69+
`sigil.receiver` is reported as unhealthy if it's given an invalid configuration.
70+
71+
## Debug metrics
72+
73+
| Metric | Type | Description |
74+
| ----------------------------------------------- | --------- | ---------------------------------------------------- |
75+
| `sigil_receiver_requests_total` | Counter | Total number of generation export requests received. |
76+
| `sigil_receiver_request_body_bytes_total` | Counter | Total bytes received in generation export requests. |
77+
| `sigil_receiver_request_duration_seconds` | Histogram | Duration of generation export request handling. |
78+
79+
## Example
80+
81+
This example creates a `sigil.receiver` that listens on the default address and forwards generation records to a `sigil.write` component.
82+
83+
```alloy
84+
sigil.receiver "default" {
85+
forward_to = [sigil.write.default.receiver]
86+
}
87+
88+
sigil.write "default" {
89+
endpoint {
90+
url = "https://sigil.grafana.net"
91+
92+
basic_auth {
93+
username = env("SIGIL_USER")
94+
password = env("SIGIL_API_KEY")
95+
}
96+
97+
headers = {
98+
"X-Scope-OrgID" = env("SIGIL_TENANT_ID"),
99+
}
100+
}
101+
}
102+
```

‎docs/sources/reference/components/sigil/sigil.write.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/sigil/sigil.write/
3+
description: Learn about sigil.write
4+
labels:
5+
stage: experimental
6+
products:
7+
- oss
8+
title: sigil.write
9+
---
10+
11+
# `sigil.write`
12+
13+
{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}
14+
15+
`sigil.write` receives Sigil AI generation records from other components and forwards them to remote Sigil ingest endpoints.
16+
17+
Request and response bodies are forwarded as opaque bytes without deserialization.
18+
When multiple `endpoint` blocks are provided, generation records are concurrently forwarded to all configured locations.
19+
20+
You can specify multiple `sigil.write` components by giving them different labels.
21+
22+
## Usage
23+
24+
```alloy
25+
sigil.write "<LABEL>" {
26+
endpoint {
27+
url = "<SIGIL_URL>"
28+
}
29+
}
30+
```
31+
32+
## Arguments
33+
34+
`sigil.write` has no top-level arguments.
35+
36+
## Blocks
37+
38+
You can use the following blocks with `sigil.write`:
39+
40+
{{< docs/alloy-config >}}
41+
42+
| Block | Description | Required |
43+
| -------------------------------------------------- | ---------------------------------------------------------- | -------- |
44+
| [`endpoint`][endpoint] | Location to send generations to. | no |
45+
| `endpoint` > [`authorization`][authorization] | Configure generic authorization to the endpoint. | no |
46+
| `endpoint` > [`basic_auth`][basic_auth] | Configure `basic_auth` for authenticating to the endpoint. | no |
47+
| `endpoint` > [`oauth2`][oauth2] | Configure OAuth 2.0 for authenticating to the endpoint. | no |
48+
| `endpoint` > `oauth2` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
49+
| `endpoint` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
50+
51+
[endpoint]: #endpoint
52+
[authorization]: #authorization
53+
[basic_auth]: #basic_auth
54+
[oauth2]: #oauth2
55+
[tls_config]: #tls_config
56+
57+
{{< /docs/alloy-config >}}
58+
59+
### `endpoint`
60+
61+
The `endpoint` block describes a single location to send generation records to.
62+
Multiple `endpoint` blocks can be provided to fan out to multiple locations.
63+
64+
The following arguments are supported:
65+
66+
| Name | Type | Description | Default | Required |
67+
| ------------------------ | ------------------- | ------------------------------------------------------------------------------------------------ | --------- | -------- |
68+
| `url` | `string` | Full URL of the Sigil ingest endpoint. | | yes |
69+
| `bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no |
70+
| `bearer_token` | `secret` | Bearer token to authenticate with. | | no |
71+
| `enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no |
72+
| `follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no |
73+
| `headers` | `map(string)` | Extra headers to deliver with the request. | | no |
74+
| `http_headers` | `map(list(secret))` | Custom HTTP headers to be sent along with each request. The map key is the header name. | | no |
75+
| `max_backoff_period` | `duration` | Maximum backoff time between retries. | `"5m"` | no |
76+
| `max_backoff_retries` | `int` | Maximum number of retries. 0 to retry infinitely. | `10` | no |
77+
| `min_backoff_period` | `duration` | Initial backoff time between retries. | `"500ms"` | no |
78+
| `name` | `string` | Optional name to identify the endpoint in metrics. | | no |
79+
| `no_proxy` | `string` | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. | | no |
80+
| `proxy_connect_header` | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests. | | no |
81+
| `proxy_from_environment` | `bool` | Use the proxy URL indicated by environment variables. | `false` | no |
82+
| `proxy_url` | `string` | HTTP proxy to send requests through. | | no |
83+
| `remote_timeout` | `duration` | Timeout for requests made to the URL. | `"10s"` | no |
84+
| `tenant_id` | `string` | Tenant ID to use for `X-Scope-OrgID` header. Overrides the value from the incoming request. | | no |
85+
86+
At most, one of the following can be provided:
87+
88+
* [`authorization`][authorization] block
89+
* [`basic_auth`][basic_auth] block
90+
* [`bearer_token_file`][endpoint] argument
91+
* [`bearer_token`][endpoint] argument
92+
* [`oauth2`][oauth2] block
93+
94+
{{< docs/shared lookup="reference/components/http-client-proxy-config-description.md" source="alloy" version="<ALLOY_VERSION>" >}}
95+
96+
Requests are retried on HTTP 429, 408, and 5xx responses with exponential backoff. Requests that receive other 4xx responses are not retried.
97+
98+
### `authorization`
99+
100+
{{< docs/shared lookup="reference/components/authorization-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
101+
102+
### `basic_auth`
103+
104+
{{< docs/shared lookup="reference/components/basic-auth-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
105+
106+
### `oauth2`
107+
108+
{{< docs/shared lookup="reference/components/oauth2-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
109+
110+
### `tls_config`
111+
112+
{{< docs/shared lookup="reference/components/tls-config-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
113+
114+
## Exported fields
115+
116+
The following fields are exported and can be referenced by other components:
117+
118+
| Name | Type | Description |
119+
| ---------- | ---------- | --------------------------------------------------------------- |
120+
| `receiver` | `receiver` | A value that other components can use to send generations to. |
121+
122+
## Component health
123+
124+
`sigil.write` is only reported as unhealthy if given an invalid configuration.
125+
In those cases, exported fields are kept at their last healthy values.
126+
127+
## Debug metrics
128+
129+
| Metric | Type | Description |
130+
| ------------------------------------- | --------- | -------------------------------------------------------------- |
131+
| `sigil_write_sent_bytes_total` | Counter | Total number of bytes sent to Sigil endpoints. |
132+
| `sigil_write_dropped_bytes_total` | Counter | Total number of bytes dropped by Sigil write. |
133+
| `sigil_write_requests_total` | Counter | Total number of requests sent to Sigil endpoints. |
134+
| `sigil_write_retries_total` | Counter | Total number of retries to Sigil endpoints. |
135+
| `sigil_write_latency` | Histogram | Write latency for sending generations to Sigil endpoints. |
136+
137+
All metrics include an `endpoint` label identifying the specific endpoint URL.
138+
139+
## Example
140+
141+
This example forwards generation records to two Sigil endpoints with different tenant IDs.
142+
143+
```alloy
144+
sigil.write "default" {
145+
endpoint {
146+
url = "https://sigil.grafana.net"
147+
148+
basic_auth {
149+
username = env("SIGIL_USER")
150+
password = env("SIGIL_API_KEY")
151+
}
152+
153+
headers = {
154+
"X-Scope-OrgID" = env("SIGIL_TENANT_ID"),
155+
}
156+
}
157+
158+
endpoint {
159+
url = "https://sigil-staging.internal"
160+
tenant_id = "staging"
161+
}
162+
}
163+
164+
sigil.receiver "default" {
165+
forward_to = [sigil.write.default.receiver]
166+
}
167+
```

‎internal/component/all/all.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,4 +181,6 @@ import (
181181
_ "github.com/grafana/alloy/internal/component/remote/kubernetes/secret" // Import remote.kubernetes.secret
182182
_ "github.com/grafana/alloy/internal/component/remote/s3" // Import remote.s3
183183
_ "github.com/grafana/alloy/internal/component/remote/vault" // Import remote.vault
184+
_ "github.com/grafana/alloy/internal/component/sigil/receiver" // Import sigil.receiver
185+
_ "github.com/grafana/alloy/internal/component/sigil/write" // Import sigil.write
184186
)

‎internal/component/sigil/receiver/arguments.go‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
package receiver
2+
3+
import (
4+
"github.com/alecthomas/units"
5+
fnet "github.com/grafana/alloy/internal/component/common/net"
6+
"github.com/grafana/alloy/internal/component/sigil"
7+
)
8+
9+
type Arguments struct {
10+
Server *fnet.ServerConfig `alloy:",squash"`
11+
ForwardTo []sigil.GenerationsReceiver `alloy:"forward_to,attr"`
12+
MaxRequestBodySize units.Base2Bytes `alloy:"max_request_body_size,attr,optional"`
13+
}
14+
15+
// SetToDefault implements syntax.Defaulter.
16+
func (a *Arguments) SetToDefault() {
17+
*a = Arguments{
18+
Server: fnet.DefaultServerConfig(),
19+
MaxRequestBodySize: 20 * units.MiB,
20+
}
21+
}

0 commit comments

Comments
 (0)