Skip to content

advanced-config.md

Latest commit

 

History

History
350 lines (298 loc) · 18.8 KB

advanced-config.md

File metadata and controls

350 lines (298 loc) · 18.8 KB

The official Splunk documentation for this page is Configure the Splunk Distribution of OTel JS.

For instructions on how to contribute to the docs, see CONTRIBUTING.md.

Advanced Configuration

To configure the Splunk Distribution of OpenTelemetry JS, you can use a combination of environment variables and arguments passed to the start() function:

  • Environment variables

    For example: export OTEL_SERVICE_NAME='test-service'

  • Arguments passed to the start() function

    For example: start({ serviceName: 'my-node-service', });

start() arguments take precedence over the corresponding environment variables.

Configuration for each of the supported signals(Metrics, Profiling, and Tracing) is set with additional properties on the configuration object:

start({
  // general options like `serviceName` and `endpoint`
  metrics: {
    // configuration passed to metrics signal
  },
  profiling: {
    // configuration passed to profiling signal
  },
  tracing: {
    // configuration passed to tracing signal
  },
});

Instead of passing an object, boolean is also an valid for turning on the signals that are not enabled by default using the default configuration:

start({
  // general options like `serviceName` and `endpoint`
  metrics: true, // turn metrics on with default options
  profiling: true, // turn profiling on with default options
});

start() turns on all stable signals with one call, which means the list will change over time.

List of settings

This distribution supports all the configuration options supported by the components it uses with the defaults specified by the OTel Specification.

Tracing

Environment variable
start() argument		 Default value Support Notes
OTEL_ATTRIBUTE_COUNT_LIMIT Stable Maximum allowed span attribute count
OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT 12000* Stable Maximum allowed attribute value size
OTEL_EXPORTER_OTLP_ENDPOINT
endpoint		 http://localhost:4318 Stable The OTLP endpoint to export to.
OTEL_LOG_LEVEL Stable Log level to use in diagnostics logging.
OTEL_PROPAGATORS
tracing.propagators		 tracecontext,baggage Stable Comma-delimited list of propagators to use. Valid keys: baggage, tracecontext, b3multi, b3.
OTEL_RESOURCE_ATTRIBUTES Stable Comma-separated list of resource attributes added to every reported span.
Examplekey1=val1,key2=val2
OTEL_SERVICE_NAME
serviceName		 unnamed-node-service Stable The service name of this Node service.
OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT 128 Stable Maximum allowed span attribute count
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT Stable Maximum allowed attribute value size. Empty value is treated as infinity
OTEL_SPAN_EVENT_COUNT_LIMIT 128 Stable
OTEL_SPAN_LINK_COUNT_LIMIT 1000* Stable
OTEL_TRACES_EXPORTER
tracing.spanExporterFactory		 otlp Stable Chooses the trace exporters. Shortcut for setting spanExporterFactory. Comma-delimited list of exporters. Currently supported values: otlp, console, none.
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL http/protobuf Stable Metric exporter protocol. Supported values: http/protobuf, grpc.
OTEL_TRACES_SAMPLER_ARG Stable String value to be used as the sampler argument. Only be used if OTEL_TRACES_SAMPLER is set.
SPLUNK_ACCESS_TOKEN
accessToken		 Stable The optional access token for exporting signal data directly to SignalFx API.
SPLUNK_REALM
realm		 Stable The name of your organization's realm, for example, us0. When you set the realm, telemetry is sent directly to the ingest endpoint of Splunk Observability Cloud, bypassing the Splunk OpenTelemetry Collector. Overridden by settings that define a complete endpoint URL, like OTEL_EXPORTER_OTLP_ENDPOINT.
SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
tracing.serverTimingEnabled		 true Stable Enable injection of Server-Timing header to HTTP responses.
SPLUNK_REDIS_INCLUDE_COMMAND_ARGS false Stable Will include the full redis query in db.statement span attribute when using redis instrumentation.

*: Overwritten default value

Sampling configuration

| Environment variable | Default value | Support | Notes | ------------------------ | ------------------------ | -------------- | ------- | ----------- | | OTEL_TRACES_SAMPLER | always_on | Stable | Sampler to be used for traces. See Sampling

Splunk Distribution of OpenTelemetry JS supports all standard samplers as provided by OpenTelemetry JS SDK. In addition, the distribution adds the following samplers:

rules

This sampler allows to ignore individual endpoints and drop all traces that originate from them. It applies only to spans with SERVER kind.

For example, the following configuration results in all requests to /healthcheck to be excluded from monitoring:

export OTEL_TRACES_SAMPLER=rules
export OTEL_TRACES_SAMPLER_ARG=drop=/healthcheck;fallback=parentbased_always_on

All requests to downstream services that happen as a consequence of calling an excluded endpoint are also excluded.

The value of OTEL_TRACES_SAMPLER_ARG is interpreted as a semicolon-separated list of rules. The following types of rules are supported:

  • drop=<value>: The sampler drops a span if its url.path (or http.target for instrumentations using older semantic conventions) attribute has a substring equal to the provided value. You can provide as many drop rules as you want.
  • fallback=sampler: Fallback sampler used if no drop rule matched a given span. Supported fallback samplers are always_on and parentbased_always_on. Probability samplers such as traceidratio are not supported.

If several fallback rules are provided, only the last one will be in effect.

If OTEL_TRACES_SAMPLER_ARG is not provided or has en empty value, no drop rules are configured and always_on sampler is as default.

Additional tracing config options in start()

The following config options can be set by passing them as tracing arguments to start().

  • tracing.resourceFactory: A function that is invoked with the default resource detected from the environment. Can be used to change the detected attributes or return a completely new resource.

  • tracing.spanExporterFactory: A function that accepts the tracing options. Returns a new instance of SpanExporter. When set, this function is used to create a new exporter and the returned exporter will be used in the pipeline.

  • tracing.spanProcessorFactory: Returns a SpanProcessor instance or an array of SpanProcessor instances. When set, this function is be used to create one or more span processors. The returned processors are added to the global tracer provider and configured to process all spans generated by any tracer provider by the global provider. The function creates a new span exporter and uses it with each processor it creates. It may call options.spanExporterFactory(options) to create a new exporter as configured by the user.

  • tracing.propagatorFactory: A function that returns a new instance of a TextMapPropagator. Defaults to a composite propagator comprised of W3C Trace Context and Baggage propagators.

  • tracing.instrumentations: Can be used to enable additional instrumentation packages.

  • tracing.captureHttpRequestUriParams: Either a list of keys (case-sensitive) of HTTP query parameters to capture or a function that gets invoked with the current span and query parameters to set a custom span attribute. When using the former, parameters are set as span attributes as http.request.param.${key}. Attribute keys are normalized at capture time, meaning . is replaced with _ to avoid any attribute namespacing issues.

  • tracing.tracerConfig: An object that is merged into the default tracer config replacing any existing keys. It's passed to the tracer provider during initialization. This can be used to customize the tracer provider or tracer.

Metrics

Environment variable
start() argument		 Default value Support Notes
OTEL_SERVICE_NAME
serviceName		 unnamed-node-service Stable The service name of this Node service.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
endpoint		 http://localhost:4318 Stable The OTLP endpoint to export to.
OTEL_METRIC_EXPORT_INTERVAL
metrics.exportIntervalMillis		 30000 Stable The interval, in milliseconds, of metrics collection and exporting.
OTEL_METRICS_EXPORTER
metrics.metricReaderFactory		 otlp Stable Chooses the metric exporters. Comma-delimited list of exporters. Currently supported values: otlp, console, none.
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
metrics.metricReaderFactory		 http/protobuf Stable Chooses the metric exporter protocol. Currently supported values: grpc, http/protobuf.
OTEL_RESOURCE_ATTRIBUTES Stable The resource attributes to metric data.
Environment variable examplekey1=val1,key2=val2
SPLUNK_METRICS_ENABLED
n/a (enabled by calling start)		 false Experimental Sets up the metrics pipeline (global meter provider, exporters).
n/a
metrics.resourceFactory		 Experimental Callback which allows to filter the default resource or provide a custom one. The function takes one argument of type Resource which is the resource pre-filled by the SDK containing the service.name, environment, host and process attributes.
SPLUNK_RUNTIME_METRICS_ENABLED
metrics.runtimeMetricsEnabled		 true Experimental Enable collecting and exporting of runtime metrics.
SPLUNK_RUNTIME_METRICS_COLLECTION_INTERVAL
metrics.runtimeMetricsCollectionIntervalMillis		 5000 Experimental The interval, in milliseconds, during which GC and event loop statistics are collected. After the collection is done, the values become available to the metric exporter.
SPLUNK_DEBUG_METRICS_ENABLED
metrics.debugMetricsEnabled		 false Experimental Enable collection of various internal metrics (e.g. the profiler's internal performance). Only useful when troubleshooting issues and should not be switched on otherwise.

Profiling

Environment variable
start() argument		 Default value Support Notes
SPLUNK_PROFILER_ENABLED false Experimental Enable continuous profiling.
SPLUNK_PROFILER_MEMORY_ENABLED
profiling.memoryProfilingEnabled		 false Experimental Enable continuous memory profiling.
SPLUNK_PROFILER_LOGS_ENDPOINT
endpoint		 http://localhost:4318 Experimental The OTLP logs receiver endpoint used for profiling data.
OTEL_SERVICE_NAME
serviceName		 unnamed-node-service Stable Service name of the application.
OTEL_RESOURCE_ATTRIBUTES Stable Comma-separated list of resource attributes.
Exampledeployment.environment=demo,key2=val2

File based configuration

Status: Experimental

An OpenTelemetry configuration YAML file can be provided via OTEL_EXPERIMENTAL_CONFIG_FILE environment variable. When set, all of the configuration options will be loaded from the given path.

Other options provided via environment variables will be ignored. Programmatic options sill take preference over the file based configuration.

See the OpenTelemetry's getting started example or use the following as a starter configuration:

file_format: "1.0-rc.2"
disabled: false # When true, the instrumentation does nothing.
log_level: warn
distribution:
  splunk:
    use_bundled_instrumentations: true
    runtime_metrics:
      collection_interval: 30000
    instrumentations:
      http:
        trace_response_header_enabled: true # SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
      redis:
        include_command_args: true           # SPLUNK_REDIS_INCLUDE_COMMAND_ARGS
    profiling:
      exporter:
        otlp_log_http:
          endpoint: "http://localhost:4318/v1/logs" # SPLUNK_PROFILER_LOGS_ENDPOINT
      always_on:
        cpu_profiler:                        # SPLUNK_PROFILER_ENABLED
          sampling_interval: 1000            # SPLUNK_PROFILER_CALL_STACK_INTERVAL
        memory_profiler:                     # SPLUNK_PROFILER_MEMORY_ENABLED
      callgraphs:                            # SPLUNK_SNAPSHOT_PROFILER_ENABLED
        sampling_interval: 1                 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
        selection_probability: 0.01          # SPLUNK_SNAPSHOT_SELECTION_PROBABILITY
logger_provider:
  processors:
    - batch:
        schedule_delay: 5000
        export_timeout: 30000
        max_queue_size: 1024
        max_export_batch_size: 256
        exporter:
          otlp_http:
            endpoint: "http://:4318/v1/logs"
            compression: gzip
            timeout: 10000
            encoding: protobuf
  limits:
    attribute_value_length_limit: 8192
    attribute_count_limit: 512
meter_provider:
  readers:
    - periodic:
        interval: 60000
        timeout: 30000
        exporter:
          otlp_http:
            endpoint: http://localhost:4318/v1/metrics
            compression: gzip
            timeout: 10000
            encoding: protobuf
            temporality_preference: delta
            default_histogram_aggregation: base2_exponential_bucket_histogram
        cardinality_limits:
          default: 2000
          counter: 2000
          gauge: 2000
          histogram: 2000
          observable_counter: 2000
          observable_gauge: 2000
          observable_up_down_counter: 2000
          up_down_counter: 2000
  views:
    - selector:
        instrument_name: my-instrument
        instrument_type: histogram
        unit: ms
        meter_name: my-meter
        meter_version: 1.0.0
        meter_schema_url: https://opentelemetry.io/schemas/1.16.0
      stream:
        name: new_instrument_name
        description: new_description
        aggregation:
          explicit_bucket_histogram:
            boundaries:
              [
                0.0,
                5.0,
                10.0,
                25.0,
                50.0,
                75.0,
                100.0,
                250.0,
                500.0,
                750.0,
                1000.0,
                2500.0,
                5000.0,
                7500.0,
                10000.0
              ]
            record_min_max: true
        aggregation_cardinality_limit: 2000
        attribute_keys:
          included:
            - key1
            - key2
          excluded:
            - key3
propagator:
  composite:
    - tracecontext:
    - baggage:
    - b3:
    - b3multi:
  composite_list: "tracecontext,baggage,b3,b3multi,xray"
tracer_provider:
  processors:
    - batch:
        schedule_delay: 5000
        export_timeout: 30000
        max_queue_size: 2048
        max_export_batch_size: 256
        exporter:
          otlp_http:
            endpoint: http://localhost:4318/v1/traces
            headers:
              - name: api-key
                value: "1234"
            compression: gzip
            timeout: 10000
            encoding: protobuf
  sampler:
    parent_based:
      root:
        always_on:
      remote_parent_sampled:
        always_on:
      remote_parent_not_sampled:
        always_on:
      local_parent_sampled:
        always_on:
      local_parent_not_sampled:
        always_on:
resource:
  attributes:
    - name: service.name
      value: test_service
    - name: service.version
      value: "1.3.0"
    - name: string_key
      value: test_value
      type: string
    - name: bool_key
      value: true
      type: bool
    - name: int_key
      value: 1
      type: int
  detection/development:
    detectors:
      - container:
      - host:
      - process:
      - service:

Migrating from signal-specific start* functions

Older versions of @splunk/otel shipped with signal-specific start functions: startMetrics, startProfiling, and startTracing. Calling these functions is now deprecated and these functions will be removed in future versions.

If your code still uses these, merge them into one start call:

const { start } = require('@splunk/otel');

start({
  // accessToken,
  // endpoint,
  // serviceName,
  tracing: {
    // tracing-specific options here.
  },
  // profiling: true, // enable profiling signal
  /*
  metrics: { // enable metrics signal with specific configuration
    // exportInterval,
  },
  */
});