Export Snapshot Profiling Stack Traces As OpenTelemetry Log Messages #2237
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| /* | ||
| * Copyright Splunk Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.splunk.opentelemetry.profiler; | ||
|
|
||
| import com.google.common.annotations.VisibleForTesting; | ||
| import io.opentelemetry.api.logs.Logger; | ||
| import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties; | ||
| import io.opentelemetry.sdk.logs.LogRecordProcessor; | ||
| import io.opentelemetry.sdk.logs.SdkLoggerProvider; | ||
| import io.opentelemetry.sdk.logs.export.LogRecordExporter; | ||
| import io.opentelemetry.sdk.logs.export.SimpleLogRecordProcessor; | ||
| import io.opentelemetry.sdk.resources.Resource; | ||
| import java.util.function.Function; | ||
|
|
||
| public class OtelLoggerFactory { | ||
| private final Function<ConfigProperties, LogRecordExporter> logRecordExporter; | ||
|
|
||
| public OtelLoggerFactory() { | ||
| this(LogExporterBuilder::fromConfig); | ||
| } | ||
|
|
||
| @VisibleForTesting | ||
| public OtelLoggerFactory(Function<ConfigProperties, LogRecordExporter> logRecordExporter) { | ||
| this.logRecordExporter = logRecordExporter; | ||
| } | ||
|
|
||
| public Logger build(ConfigProperties properties, Resource resource) { | ||
| LogRecordExporter exporter = createLogRecordExporter(properties); | ||
| LogRecordProcessor processor = SimpleLogRecordProcessor.create(exporter); | ||
| return buildOtelLogger(processor, resource); | ||
| } | ||
|
|
||
| private LogRecordExporter createLogRecordExporter(ConfigProperties properties) { | ||
| return logRecordExporter.apply(properties); | ||
| } | ||
|
|
||
| private Logger buildOtelLogger(LogRecordProcessor logProcessor, Resource resource) { | ||
| return SdkLoggerProvider.builder() | ||
| .addLogRecordProcessor(logProcessor) | ||
| .setResource(resource) | ||
| .build() | ||
| .loggerBuilder(ProfilingSemanticAttributes.OTEL_INSTRUMENTATION_NAME) | ||
| .setInstrumentationVersion(ProfilingSemanticAttributes.OTEL_INSTRUMENTATION_VERSION) | ||
| .build(); | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| /* | ||
| * Copyright Splunk Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.splunk.opentelemetry.profiler.snapshot; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
| import java.util.concurrent.ConcurrentHashMap; | ||
| import java.util.concurrent.ConcurrentMap; | ||
| import java.util.function.Supplier; | ||
|
|
||
| class AccumulatingStagingArea implements StagingArea { | ||
|
Collaborator
There was a problem hiding this comment. Here you collect stack traces per trace id and when the trace is done you export them. To me the staging area abstraction doesn't seem ideal. When there is a long request you end up accumulating many stack traces, when the request is short you export a small batch with only one or two traces. When there are multiple concurrent short requests you'll export many small batches. With the non-snapshot profiler we already get the input data in batches and the interval between the batches is large so we don't need to handle this. Did you consider alternative solutions like for example what is used in https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/export/BatchSpanProcessorBuilder.java Place data in the queue, if queue has batch size items (there it is 512) export them, otherwise wait for a bit (there 5s) and export whatever is in the queue. If queue grows too large drop data.
Contributor
Author
There was a problem hiding this comment. This sounds like a very good idea. Would you be against doing this as a followup (this PR is a partial blocker to adding span correlation)? I don't think a time-based approach is necessarily the right approach however. What I'm envisioning is a This would control how many stack traces are bundled into a single log record. The frequency of the log record exporting is separately controlled by the
Collaborator
There was a problem hiding this comment.
sure
Flushing when there are no new stack traces in some time isn't that hard. When you write stacks from all ongoing traces to the same pprof then flushing on ending the entry span doesn't make sense.
Using |
||
| private final ConcurrentMap<String, List<StackTrace>> stackTraces = new ConcurrentHashMap<>(); | ||
| private final Supplier<StackTraceExporter> exporter; | ||
|
|
||
| AccumulatingStagingArea(Supplier<StackTraceExporter> exporter) { | ||
| this.exporter = exporter; | ||
| } | ||
|
|
||
| @Override | ||
| public void stage(String traceId, StackTrace stackTrace) { | ||
| stackTraces.compute( | ||
| traceId, | ||
| (id, stackTraces) -> { | ||
| if (stackTraces == null) { | ||
| stackTraces = new ArrayList<>(); | ||
| } | ||
| stackTraces.add(stackTrace); | ||
| return stackTraces; | ||
| }); | ||
| } | ||
|
|
||
| @Override | ||
| public void empty(String traceId) { | ||
| List<StackTrace> stackTraces = this.stackTraces.remove(traceId); | ||
| if (stackTraces != null) { | ||
| exporter.get().export(stackTraces); | ||
| } | ||
| } | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I needed access to the
AutoConfigureUtilclass defined in this subproject. Used inStackTraceExporterActivatorto access theConfigPropertiesandResourcefields in theAutoConfiguredOpenTelemetrySdkwhen configuring the OtelLogger.The
compileOnlyaccess already defined did not allow me to test that the configuration was taking place.