Support for ECS

[deepakmohanakrishnan07]

It would be great if the emitted structured log adhere to elastic common schema. This would enable the application to adopt ECS schema using this drop in replacement library with logbook encoder configuration changes. If everyone feels that would be a useful feature add, please feel free to provide your inputs/feedback/thumbsup :-)

[helmchen]

ECS Dokumentation can be found here.

[universalmind303]

I agree. I am actually in the process of converting the pattern to match ECS, and it is quite time consuming. Would be nice if it matched the schema with 0 configuration.

[will-sargent-eero]

Also see https://github.com/elastic/ecs-logging-java#mapping

[aaronrebak]

Bumping this.

There is a separate ticket open on elastic/ecs-logging-java#49.
Currently elastic's version of logback encoding is extremely limited and does not support custom key-value pairs at all.

This entirely defeats the purpose of the library as it only offers very basic support and doesn't allow additional fields to be logged in a nice way which can be indexed for monitoring/alerting in elastic.

There are are other libraries that support ecs log formatting such as quarkiverse extensions: https://github.com/quarkiverse/quarkus-logging-json/tree/main

Unfortunately these are only for applications in the quarkus framework and given how robust logstash-logback-encoder is it would be great to offer an out of the box encoder to configure with logback.

[aaronrebak]

I agree. I am actually in the process of converting the pattern to match ECS, and it is quite time consuming. Would be nice if it matched the schema with 0 configuration.

@universalmind303 did you ever get around to converting the pattern to match ECS (elastic common schema).
If so would you be able to share a snippet of this?

[koszta5]

@aaronrebak I was trying to make ECS APM agent working together with Marker supported in this project and managed to combine both.

1. Add both dependencies (see ECS APM java agent docs)
2. Use filebeat + encoder from ECS Java client to actually create logs
   public class MarkerSupportingEcsEncoder extends EcsEncoder
   (most importantly examine and copy encode method from the original encoder...
3. Adjust method serializeMarkers taking advantage of marker serialization from logback-logstash
4. Use the newly created encoder for your classic logback appender

[aaronrebak]

@aaronrebak I was trying to make ECS APM agent working together with Marker supported in this project and managed to combine both.

1. Add both dependencies (see ECS APM java agent docs)
2. Use filebeat + encoder from ECS Java client to actually create logs
   public class MarkerSupportingEcsEncoder extends EcsEncoder
   (most importantly examine and copy encode method from the original encoder...
3. Adjust method serializeMarkers taking advantage of marker serialization from logback-logstash
4. Use the newly created encoder for your classic logback appender

@koszta5 sounds great!
Are you able to provide a snippet to show this working in practice for the community?

[koszta5]

My code is a little specific to my use case but simply follow the approach above
1.

<dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
      <version>7.4</version>
    </dependency>
    <dependency>
      <groupId>co.elastic.logging</groupId>
      <artifactId>logback-ecs-encoder</artifactId>
      <version>1.5.0</version>
    </dependency>

2. Most of the code stolen from EcsEncoder; but here you can do your custom magic...

public class MarkerSupportingEcsEncoder extends EcsEncoder
@Override
	public byte[] encode(ILoggingEvent event) {
		Marker runtimeCreatedMarker = mergeMarkerWithAdditionalFields(event);
		StringBuilder builder = new StringBuilder(256);
		EcsJsonSerializer.serializeObjectStart(builder, event.getTimeStamp());
		EcsJsonSerializer.serializeLogLevel(builder, event.getLevel().toString());
		EcsJsonSerializer.serializeFormattedMessage(builder, event.getFormattedMessage());
		EcsJsonSerializer.serializeEcsVersion(builder);
		this.serializeMarkers(runtimeCreatedMarker, builder);
		EcsJsonSerializer.serializeServiceName(builder, this.serviceName);
		EcsJsonSerializer.serializeServiceVersion(builder, this.serviceVersion);
		EcsJsonSerializer.serializeServiceEnvironment(builder, this.serviceEnvironment);
		EcsJsonSerializer.serializeServiceNodeName(builder, this.serviceNodeName);
		EcsJsonSerializer.serializeEventDataset(builder, this.eventDataset);
		EcsJsonSerializer.serializeThreadName(builder, event.getThreadName());
         	EcsJsonSerializer.serializeLoggerName(builder, event.getLoggerName());
		EcsJsonSerializer.serializeAdditionalFields(builder, this.additionalFields);
		EcsJsonSerializer.serializeMDC(builder, event.getMDCPropertyMap());
		if (this.includeOrigin) {
			StackTraceElement[] callerData = event.getCallerData();
			if (callerData != null && callerData.length > 0) {
				EcsJsonSerializer.serializeOrigin(builder, callerData[0]);
			}
		}

		this.addCustomFields(event, builder);
		IThrowableProxy throwableProxy = event.getThrowableProxy();
		if (throwableProxy != null) {
			if (this.throwableConverter != null) {
				EcsJsonSerializer.serializeException(builder, throwableProxy.getClassName(), throwableProxy.getMessage(), this.throwableConverter.convert(event), this.stackTraceAsArray);
			} else if (throwableProxy instanceof ThrowableProxy) {
				EcsJsonSerializer.serializeException(builder, ((ThrowableProxy)throwableProxy).getThrowable(), this.stackTraceAsArray);
			} else {
				EcsJsonSerializer.serializeException(builder, throwableProxy.getClassName(), throwableProxy.getMessage(), this.throwableProxyConverter.convert(event), this.stackTraceAsArray);
			}
		}

		EcsJsonSerializer.serializeObjectEnd(builder);
		return builder.toString().getBytes(UTF_8);
	}

@NotNull
	private Marker mergeMarkerWithAdditionalFields(ILoggingEvent event) {
		Map<String, AdditionalField> additionalFields = new HashMap<>();
		if (MarkerSupportingEcsEncoder.additionalFieldHolderRegistry != null) {
			Optional<AdditionalFieldsHolder> holder = getHolder(event);
			if (holder.isPresent()) {
				additionalFields.putAll(holder.get()
											  .get()
											  .stream()
											  .filter(fld -> !fld.getKey().equals(AdditionalFieldsHolder.ADDITIONAL_FIELD.LOGGER.getFieldName()))
											  .collect(Collectors.toMap(AdditionalField::getKey, Function.identity(), (k1, k2) -> k2)));
			}
		}
		if (isConnectorLogging(event)){
			additionalFields.put(AdditionalFieldsHolder.ADDITIONAL_FIELD.PHASE.getFieldName(), new AdditionalField(AdditionalFieldsHolder.ADDITIONAL_FIELD.PHASE.getFieldName(),
																												   Log.IN_PROCESS));
		}
		if (event.getMarker() != null) {
			additionalFields.putAll(markersToAdditionalFields(event.getMarker()).stream()
																				.collect(Collectors.toMap(AdditionalField::getKey,
																										  Function.identity(),
																										  (k1, k2) -> k2)));
		}
		Marker runtimeCreatedMarker = new MapEntriesAppendingMarker(additionalFields.values()
																					.stream()
																					.collect(Collectors.toMap(AdditionalField::getKey,
																											  AdditionalField::getValue,
																											  (k1, k2) -> k2)));
		return runtimeCreatedMarker;
	}


// define all setters like this:
	@Override
	public void setServiceVersion(String serviceVersion) {
		super.setServiceVersion(serviceVersion);
		this.serviceVersion = serviceVersion;
	}

4. Then to configure logback:

 <appender name="FILE"
              class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${logFolder}/catalina.out</file>
        <encoder class="com.your.package.MarkerSupportingEcsEncoder">
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${logFolder}/archive/catalina.out.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
    </appender>