Skip to content

Commit be4149e

Browse files
feat: Support configuring behavior of loki.source.syslog setting the year of an incoming timestamp (#2755)
* migrate syslog code from loki * Update syslog behavior to allow configuration of currentyear behavior for rfc3164 * Changelog entry * refactor to do year parsing in a way that works at year borders * Fix changelog * Apply suggestions from code review Co-authored-by: Clayton Cornell <[email protected]> * fix changelog * Add comments about message framing in syslog parser * Update docs/sources/reference/components/loki/loki.source.syslog.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/loki/loki.source.syslog.md Co-authored-by: Clayton Cornell <[email protected]> --------- Co-authored-by: Clayton Cornell <[email protected]>
1 parent 486cc27 commit be4149e

File tree

11 files changed

+375
-29
lines changed

11 files changed

+375
-29
lines changed

CHANGELOG.md

+2
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ Main (unreleased)
1616

1717
### Enhancements
1818

19+
- Add `rfc3164_default_to_current_year` argument to `loki.source.syslog` (@dehaansa)
20+
1921
- Add `connection_name` support for `prometheus.exporter.mssql` (@bck01215)
2022

2123
- Add livedebugging support for `prometheus.scrape` (@ravishankar15, @wildum)

docs/sources/reference/components/loki/loki.source.syslog.md

+17-12
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ You can use the following blocks with `loki.source.syslog`:
4949

5050
| Name | Description | Required |
5151
| --------------------------------------- | --------------------------------------------------------------------------- | -------- |
52-
| [`listener`][listener] | Configures a listener for IETF Syslog (RFC5424) messages. | no |
52+
| [`listener`][listener] | Configures a listener for Syslog messages. | no |
5353
| `listener` > [`tls_config`][tls_config] | Configures TLS settings for connecting to the endpoint for TCP connections. | no |
5454

5555
The > symbol indicates deeper levels of nesting.
@@ -65,17 +65,18 @@ The `listener` block defines the listen address and protocol where the listener
6565
The following arguments can be used to configure a `listener`.
6666
Only the `address` field is required and any omitted fields take their default values.
6767

68-
| Name | Type | Description | Default | Required |
69-
| ------------------------ | ------------- | ----------------------------------------------------------------------------- | --------- | -------- |
70-
| `address` | `string` | The `<host:port>` address to listen to for syslog messages. | | yes |
71-
| `idle_timeout` | `duration` | The idle timeout for TCP connections. | `"120s"` | no |
72-
| `label_structured_data` | `bool` | Whether to translate syslog structured data to Loki labels. | `false` | no |
73-
| `labels` | `map(string)` | The labels to associate with each received syslog record. | `{}` | no |
74-
| `max_message_length` | `int` | The maximum limit to the length of syslog messages. | `8192` | no |
75-
| `protocol` | `string` | The protocol to listen to for syslog messages. Must be either `tcp` or `udp`. | `tcp` | no |
76-
| `syslog_format` | `string` | The format for incoming messages. Must be either `rfc5424` or `rfc3164`. | `rfc5424` | no |
77-
| `use_incoming_timestamp` | `bool` | Whether to set the timestamp to the incoming syslog record timestamp. | `false` | no |
78-
| `use_rfc5424_message` | `bool` | Whether to forward the full RFC5424-formatted syslog message. | `false` | no |
68+
| Name | Type | Description | Default | Required |
69+
| --------------------------------- | ------------- | -------------------------------------------------------------------------------------- | --------- | -------- |
70+
| `address` | `string` | The `<host:port>` address to listen to for syslog messages. | | yes |
71+
| `idle_timeout` | `duration` | The idle timeout for TCP connections. | `"120s"` | no |
72+
| `label_structured_data` | `bool` | Whether to translate syslog structured data to Loki labels. | `false` | no |
73+
| `labels` | `map(string)` | The labels to associate with each received syslog record. | `{}` | no |
74+
| `max_message_length` | `int` | The maximum limit to the length of syslog messages. | `8192` | no |
75+
| `protocol` | `string` | The protocol to listen to for syslog messages. Must be either `tcp` or `udp`. | `tcp` | no |
76+
| `rfc3164_default_to_current_year` | `bool` | Whether to default the incoming timestamp of an `rfc3164` message to the current year. | `false` | no |
77+
| `syslog_format` | `string` | The format for incoming messages. Must be either `rfc5424` or `rfc3164`. | `rfc5424` | no |
78+
| `use_incoming_timestamp` | `bool` | Whether to set the timestamp to the incoming syslog record timestamp. | `false` | no |
79+
| `use_rfc5424_message` | `bool` | Whether to forward the full RFC5424-formatted syslog message. | `false` | no |
7980

8081
By default, the component assigns the log entry timestamp as the time it was processed.
8182

@@ -86,6 +87,10 @@ All header fields from the parsed RFC5424 messages are brought in as internal la
8687
If `label_structured_data` is set, structured data in the syslog header is also translated to internal labels in the form of `__syslog_message_sd_<ID>_<KEY>`.
8788
For example, a structured data entry of `[example@99999 test="yes"]` becomes the label `__syslog_message_sd_example_99999_test` with the value `"yes"`.
8889

90+
The `rfc3164_default_to_current_year` argument is only relevant when `use_incoming_timestamp` is also set to `true`.
91+
`rfc3164` message timestamps don't contain a year, and this component's default behavior is to mimic Promtail behavior and leave the year as 0.
92+
Setting `rfc3164_default_to_current_year` to `true` sets the year of the incoming timestamp to the current year using the local time of the {{< param "PRODUCT_NAME" >}} instance.
93+
8994
### `tls_config`
9095

9196
{{< docs/shared lookup="reference/components/tls-config-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

internal/component/loki/process/stages/util.go

+3-8
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ import (
66
"strconv"
77
"strings"
88
"time"
9+
10+
"github.com/grafana/alloy/internal/util"
911
)
1012

1113
var (
@@ -150,14 +152,7 @@ func parseTimestampWithoutYear(layout string, location *time.Location, timestamp
150152
return parsedTime, fmt.Errorf(ErrTimestampContainsYear, timestamp)
151153
}
152154

153-
// Handle the case we're crossing the New Year's Eve midnight
154-
if parsedTime.Month() == 12 && now.Month() == 1 {
155-
parsedTime = parsedTime.AddDate(now.Year()-1, 0, 0)
156-
} else if parsedTime.Month() == 1 && now.Month() == 12 {
157-
parsedTime = parsedTime.AddDate(now.Year()+1, 0, 0)
158-
} else {
159-
parsedTime = parsedTime.AddDate(now.Year(), 0, 0)
160-
}
155+
util.SetYearForLimitedTimeFormat(&parsedTime, now)
161156

162157
return parsedTime, nil
163158
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
package config
2+
3+
import (
4+
"time"
5+
6+
promconfig "github.com/prometheus/common/config"
7+
"github.com/prometheus/common/model"
8+
)
9+
10+
type SyslogFormat string
11+
12+
const (
13+
// A modern Syslog RFC
14+
SyslogFormatRFC5424 = "rfc5424"
15+
// A legacy Syslog RFC also known as BSD-syslog
16+
SyslogFormatRFC3164 = "rfc3164"
17+
)
18+
19+
// SyslogTargetConfig describes a scrape config that listens for log lines over syslog.
20+
type SyslogTargetConfig struct {
21+
// ListenAddress is the address to listen on for syslog messages.
22+
ListenAddress string `yaml:"listen_address"`
23+
24+
// ListenProtocol is the protocol used to listen for syslog messages.
25+
// Must be either `tcp` (default) or `udp`
26+
ListenProtocol string `yaml:"listen_protocol"`
27+
28+
// IdleTimeout is the idle timeout for tcp connections.
29+
IdleTimeout time.Duration `yaml:"idle_timeout"`
30+
31+
// LabelStructuredData sets if the structured data part of a syslog message
32+
// is translated to a label.
33+
// [example@99999 test="yes"] => {__syslog_message_sd_example_99999_test="yes"}
34+
LabelStructuredData bool `yaml:"label_structured_data"`
35+
36+
// Labels optionally holds labels to associate with each record read from syslog.
37+
Labels model.LabelSet `yaml:"labels"`
38+
39+
// UseIncomingTimestamp sets the timestamp to the incoming syslog messages
40+
// timestamp if it's set.
41+
UseIncomingTimestamp bool `yaml:"use_incoming_timestamp"`
42+
43+
// UseRFC5424Message defines whether the full RFC5424 formatted syslog
44+
// message should be pushed to Loki
45+
UseRFC5424Message bool `yaml:"use_rfc5424_message"`
46+
47+
// Syslog format used at the target. Acceptable value is rfc5424 or rfc3164.
48+
// Default is rfc5424.
49+
SyslogFormat SyslogFormat `yaml:"syslog_format"`
50+
51+
// MaxMessageLength sets the maximum limit to the length of syslog messages
52+
MaxMessageLength int `yaml:"max_message_length"`
53+
54+
TLSConfig promconfig.TLSConfig `yaml:"tls_config,omitempty"`
55+
56+
// When parsing an RFC3164 message, should the year be defaulted to the current year?
57+
// When false, the year will default to 0.
58+
RFC3164DefaultToCurrentYear bool `yaml:"rfc3164_default_to_current_year"`
59+
}
60+
61+
func (config SyslogTargetConfig) IsRFC3164Message() bool {
62+
return config.SyslogFormat == SyslogFormatRFC3164
63+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
package syslogparser
2+
3+
import (
4+
"bufio"
5+
"fmt"
6+
"io"
7+
"time"
8+
9+
"github.com/grafana/alloy/internal/util"
10+
"github.com/leodido/go-syslog/v4"
11+
"github.com/leodido/go-syslog/v4/nontransparent"
12+
"github.com/leodido/go-syslog/v4/octetcounting"
13+
"github.com/leodido/go-syslog/v4/rfc3164"
14+
)
15+
16+
// ParseStream parses a rfc5424 syslog stream from the given Reader, calling
17+
// the callback function with the parsed messages. The parser automatically
18+
// detects octet counting.
19+
// The function returns on EOF or unrecoverable errors.
20+
func ParseStream(isRFC3164Message bool, useRFC3164DefaultYear bool, r io.Reader, callback func(res *syslog.Result), maxMessageLength int) error {
21+
buf := bufio.NewReaderSize(r, 1<<10)
22+
23+
b, err := buf.ReadByte()
24+
if err != nil {
25+
return err
26+
}
27+
_ = buf.UnreadByte()
28+
cb := callback
29+
if isRFC3164Message && useRFC3164DefaultYear {
30+
cb = func(res *syslog.Result) {
31+
if res.Message != nil {
32+
rfc3164Msg := res.Message.(*rfc3164.SyslogMessage)
33+
if rfc3164Msg.Timestamp != nil {
34+
util.SetYearForLimitedTimeFormat(rfc3164Msg.Timestamp, time.Now())
35+
}
36+
}
37+
callback(res)
38+
}
39+
}
40+
41+
// See https://datatracker.ietf.org/doc/html/rfc6587 for details on message framing
42+
// If a syslog message starts with '<' the first piece of the message is the priority, which means it must use
43+
// an explicit framing character.
44+
if b == '<' {
45+
if isRFC3164Message {
46+
nontransparent.NewParserRFC3164(syslog.WithListener(cb), syslog.WithMaxMessageLength(maxMessageLength), syslog.WithBestEffort()).Parse(buf)
47+
} else {
48+
nontransparent.NewParser(syslog.WithListener(cb), syslog.WithMaxMessageLength(maxMessageLength), syslog.WithBestEffort()).Parse(buf)
49+
}
50+
// If a syslog message starts with a digit, it must use octet counting, and the first piece of the message is the length
51+
} else if b >= '0' && b <= '9' {
52+
if isRFC3164Message {
53+
octetcounting.NewParserRFC3164(syslog.WithListener(cb), syslog.WithMaxMessageLength(maxMessageLength), syslog.WithBestEffort()).Parse(buf)
54+
} else {
55+
octetcounting.NewParser(syslog.WithListener(cb), syslog.WithMaxMessageLength(maxMessageLength), syslog.WithBestEffort()).Parse(buf)
56+
}
57+
} else {
58+
return fmt.Errorf("invalid or unsupported framing. first byte: '%s'", string(b))
59+
}
60+
61+
return nil
62+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
package syslogparser_test
2+
3+
import (
4+
"io"
5+
"strings"
6+
"testing"
7+
"time"
8+
9+
"github.com/grafana/alloy/internal/component/loki/source/syslog/internal/syslogtarget/syslogparser"
10+
"github.com/leodido/go-syslog/v4"
11+
"github.com/leodido/go-syslog/v4/rfc3164"
12+
"github.com/leodido/go-syslog/v4/rfc5424"
13+
"github.com/stretchr/testify/require"
14+
)
15+
16+
var (
17+
defaultMaxMessageLength = 8192
18+
)
19+
20+
func TestParseStream_OctetCounting(t *testing.T) {
21+
r := strings.NewReader("23 <13>1 - - - - - - First24 <13>1 - - - - - - Second")
22+
23+
results := make([]*syslog.Result, 0)
24+
cb := func(res *syslog.Result) {
25+
results = append(results, res)
26+
}
27+
28+
err := syslogparser.ParseStream(false, false, r, cb, defaultMaxMessageLength)
29+
require.NoError(t, err)
30+
31+
require.Equal(t, 2, len(results))
32+
require.NoError(t, results[0].Error)
33+
require.Equal(t, "First", *results[0].Message.(*rfc5424.SyslogMessage).Message)
34+
require.NoError(t, results[1].Error)
35+
require.Equal(t, "Second", *results[1].Message.(*rfc5424.SyslogMessage).Message)
36+
}
37+
38+
func TestParseStream_ValidParseError(t *testing.T) {
39+
// This message can not parse fully but is valid when using the BestEffort Parser Option.
40+
r := strings.NewReader("17 <13>1 First")
41+
42+
results := make([]*syslog.Result, 0)
43+
cb := func(res *syslog.Result) {
44+
results = append(results, res)
45+
}
46+
47+
err := syslogparser.ParseStream(false, false, r, cb, defaultMaxMessageLength)
48+
require.NoError(t, err)
49+
50+
require.Equal(t, 1, len(results))
51+
require.EqualError(t, results[0].Error, "expecting a RFC3339MICRO timestamp or a nil value [col 6]")
52+
require.True(t, results[0].Message.(*rfc5424.SyslogMessage).Valid())
53+
}
54+
55+
func TestParseStream_OctetCounting_LongMessage(t *testing.T) {
56+
r := strings.NewReader("8198 <13>1 - - - - - - First")
57+
58+
results := make([]*syslog.Result, 0)
59+
cb := func(res *syslog.Result) {
60+
results = append(results, res)
61+
}
62+
63+
err := syslogparser.ParseStream(false, false, r, cb, defaultMaxMessageLength)
64+
require.NoError(t, err)
65+
66+
require.Equal(t, 1, len(results))
67+
require.EqualError(t, results[0].Error, "message too long to parse. was size 8198, max length 8192")
68+
}
69+
70+
func TestParseStream_NewlineSeparated(t *testing.T) {
71+
r := strings.NewReader("<13>1 - - - - - - First\n<13>1 - - - - - - Second\n")
72+
73+
results := make([]*syslog.Result, 0)
74+
cb := func(res *syslog.Result) {
75+
results = append(results, res)
76+
}
77+
78+
err := syslogparser.ParseStream(false, false, r, cb, defaultMaxMessageLength)
79+
require.NoError(t, err)
80+
81+
require.Equal(t, 2, len(results))
82+
require.NoError(t, results[0].Error)
83+
require.Equal(t, "First", *results[0].Message.(*rfc5424.SyslogMessage).Message)
84+
require.NoError(t, results[1].Error)
85+
require.Equal(t, "Second", *results[1].Message.(*rfc5424.SyslogMessage).Message)
86+
}
87+
88+
func TestParseStream_InvalidStream(t *testing.T) {
89+
r := strings.NewReader("invalid")
90+
91+
err := syslogparser.ParseStream(false, false, r, func(_ *syslog.Result) {}, defaultMaxMessageLength)
92+
require.EqualError(t, err, "invalid or unsupported framing. first byte: 'i'")
93+
}
94+
95+
func TestParseStream_EmptyStream(t *testing.T) {
96+
r := strings.NewReader("")
97+
98+
err := syslogparser.ParseStream(false, false, r, func(_ *syslog.Result) {}, defaultMaxMessageLength)
99+
require.Equal(t, err, io.EOF)
100+
}
101+
102+
func TestParseStream_RFC3164Timestamp(t *testing.T) {
103+
r := strings.NewReader("<13>Dec 1 00:00:00 host Message")
104+
105+
results := make([]*syslog.Result, 0)
106+
cb := func(res *syslog.Result) {
107+
results = append(results, res)
108+
}
109+
110+
err := syslogparser.ParseStream(true, false, r, cb, defaultMaxMessageLength)
111+
require.NoError(t, err)
112+
113+
require.Equal(t, 1, len(results))
114+
require.NoError(t, results[0].Error)
115+
require.Equal(t, "Message", *results[0].Message.(*rfc3164.SyslogMessage).Message)
116+
require.Equal(t, "host", *results[0].Message.(*rfc3164.SyslogMessage).Hostname)
117+
require.Equal(t, time.Date(0, 12, 1, 0, 0, 0, 0, time.UTC), *results[0].Message.(*rfc3164.SyslogMessage).Timestamp)
118+
}
119+
120+
func TestParseStream_RFC3164TimestampWithYear(t *testing.T) {
121+
r := strings.NewReader("<13>Dec 1 00:00:00 host Message")
122+
123+
results := make([]*syslog.Result, 0)
124+
cb := func(res *syslog.Result) {
125+
results = append(results, res)
126+
}
127+
128+
err := syslogparser.ParseStream(true, true, r, cb, defaultMaxMessageLength)
129+
require.NoError(t, err)
130+
131+
require.Equal(t, 1, len(results))
132+
require.NoError(t, results[0].Error)
133+
require.Equal(t, "Message", *results[0].Message.(*rfc3164.SyslogMessage).Message)
134+
require.Equal(t, "host", *results[0].Message.(*rfc3164.SyslogMessage).Hostname)
135+
require.Equal(t, time.Date(time.Now().Year(), 12, 1, 0, 0, 0, 0, time.UTC), *results[0].Message.(*rfc3164.SyslogMessage).Timestamp)
136+
}

internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import (
1212
"time"
1313

1414
"github.com/go-kit/log"
15-
"github.com/grafana/loki/v3/clients/pkg/promtail/scrapeconfig"
15+
scrapeconfig "github.com/grafana/alloy/internal/component/loki/source/syslog/config"
1616
"github.com/grafana/loki/v3/clients/pkg/promtail/targets/target"
1717
"github.com/grafana/loki/v3/pkg/logproto"
1818
"github.com/leodido/go-syslog/v4"

0 commit comments

Comments
 (0)