Skip to content

Let product operators control product-specific vector config #794

Open
@nightkr

Description

@nightkr

Currently, op-rs embeds a lot of product-specific knowledge. For example, there are several configuration bits that are only relevant to our specific deployment of OPA that are always embedded in the configuration (

files_opa_bundle_builder:
type: file
include:
- {STACKABLE_LOG_DIR}/bundle-builder/current
files_opa_json:
type: file
include:
- {STACKABLE_LOG_DIR}/opa/current
transforms:
processed_files_opa_bundle_builder:
inputs:
- files_opa_bundle_builder
type: remap
source: |
parsed_event = parse_regex!(strip_whitespace(strip_ansi_escape_codes(string!(.message))), r'(?P<timestamp>[0-9]{{4}}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T(2[0-3]|[01][0-9]):[0-5][0-9]:[0-5][0-9].[0-9]{{6}}Z)[ ]+(?P<level>\w+)[ ]+(?P<logger>.+):[ ]+(?P<message>.*)')
.timestamp = parse_timestamp!(parsed_event.timestamp, "%Y-%m-%dT%H:%M:%S.%6fZ")
.level = parsed_event.level
.logger = parsed_event.logger
.message = parsed_event.message
processed_files_opa_json:
inputs:
- files_opa_json
type: remap
source: |
parsed_event = parse_json!(string!(.message))
keys = keys!(parsed_event)
if includes(keys, "timestamp") {{
.timestamp = parse_timestamp!(parsed_event.timestamp, "%Y-%m-%dT%H:%M:%S.%fZ")
}} else {{
.timestamp = parse_timestamp!(parsed_event.time, "%Y-%m-%dT%H:%M:%SZ")
}}
if includes(keys, "decision_id") {{
.logger = "decision"
}} else {{
.logger = "server"
}}
.level = upcase!(parsed_event.level)
.message = string!(parsed_event.msg)
del(parsed_event.time)
del(parsed_event.timestamp)
del(parsed_event.level)
del(parsed_event.msg)
.message = .message + "\n" + encode_key_value(object!(parsed_event), field_delimiter: "\n")
).

It makes sense to offer a few defaults ("this is how we typically configure log4j to behave"), but the final say should be up to the operator and we definitely shouldn't have single-product configurations in the framework crate.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions