fixing the examples in data prepper trace analytics #11183
Conversation
Signed-off-by: Anton Rubin <[email protected]>
AntonEliatra
requested review from
AMoo-Miki,
dlvenable,
epugh,
kolchfa-aws,
natebower and
sumobrian
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
Signed-off-by: Anton Rubin <[email protected]>
| name: "raw-trace-pipeline" | ||
| - pipeline: | ||
| name: "entry-pipeline" | ||
| - pipeline: { name: "raw-trace-pipeline" } |
There was a problem hiding this comment.
We should use the old approach as that syntax is more familiar for users.
| batch_size: 64 | ||
| processor: | ||
| - otel_traces: | ||
| - otel_traces: {} |
There was a problem hiding this comment.
The {} is not needed and we don't encourage it to help reduce clutter.
| buffer_size: 10240 | ||
| batch_size: 160 | ||
| sink: | ||
| - pipeline: { name: "raw-trace-pipeline" } |
| source: | ||
| pipeline: { name: "entry-pipeline" } | ||
| processor: | ||
| - otel_traces: {} |
| service-map-pipeline: | ||
| source: | ||
| pipeline: { name: "entry-pipeline" } |
| otlp: | ||
| endpoint: data-prepper:21890 | ||
| tls: | ||
| insecure: true # plaintext to Data Prepper (matches ssl: false on the otel_trace_source) |
There was a problem hiding this comment.
Would it be possible to make this example work using SSL? Maybe we can disable hostname validation. We want to encourage as secure of defaults as possible.
| The following example demonstrates how to build a pipeline that supports the [OpenSearch Dashboards Observability plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines serve two different purposes and write to different OpenSearch indexes. The first pipeline prepares trace data for OpenSearch and enriches and ingests the span documents into a span index within OpenSearch. The second pipeline aggregates traces into a service map and writes service map documents into a service map index within OpenSearch. | ||
|
|
||
| Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_traces_prepper` processor. The `otel_traces` processor replaces the `otel_traces_prepper` processor and supports some of Data Prepper's recent data model changes. Instead, you should use the `otel_traces` processor. See the following YAML file example: | ||
| Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_traces_prepper` processor. The `otel_traces` processor replaces the `otel_traces_prepper` processor and supports some of Data Prepper's recent data model changes. See the following YAML file example: |
There was a problem hiding this comment.
Also, the otel_traces replaces the otel_trace_raw which customers have used before. The otel_trace_raw is still allowed in Data Prepper 2.0. We will remove it in 3.0.
| # We recommend to keep the same buffer_size for all pipelines. | ||
| # Make sure you configure sufficient heap | ||
| # default value is 512 | ||
| buffer_size: 512 |
There was a problem hiding this comment.
The default values are extremely low. We should include better values in the samples.
Perhaps:
buffer_size: 500000
batch_size: 10000
Signed-off-by: Anton Rubin <[email protected]>
|
@dlvenable Thank you for the review, thats been updated now |
3 participants
Description
fixing the examples in data prepper trace analytics
Version
all
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.