Release (#1701)

Author: renejeglinsky

java/operating-applications/observability.md

@@ -74,14 +74,14 @@ Similarly, no specific log output configuration is required for local developmen
 All logs are written, that have a log level greater or equal to the configured log level of the corresponding logger object.
 The following log levels are available:
 
-| Level    | Use case
-| :--------| :--------
-| `OFF`    | Turns off the logger
-| `TRACE`  | Tracks the application flow only
-| `DEBUG`  | Shows diagnostic messages
-| `INFO`   | Shows important flows of the application (default level)
-| `WARN`   | Indicates potential error scenarios
-| `ERROR`  | Shows errors and exceptions
+| Level   | Use case                                                 |
+| :------ | :------------------------------------------------------- |
+| `OFF`   | Turns off the logger                                     |
+| `TRACE` | Tracks the application flow only                         |
+| `DEBUG` | Shows diagnostic messages                                |
+| `INFO`  | Shows important flows of the application (default level) |
+| `WARN`  | Indicates potential error scenarios                      |
+| `ERROR` | Shows errors and exceptions                              |
 
 With Spring Boot, there are different convenient ways to configure log levels in a development scenario, which is explained in the following section.
 
@@ -144,20 +144,20 @@ curl -X POST -H 'Content-Type: application/json' -d '{"configuredLevel": "DEBUG"
 
 CAP Java SDK has useful built-in loggers that help to track runtime behavior:
 
-| Logger                         | Use case
-| :------------------------------| :--------
-| `com.sap.cds.security.authentication`  | Logs authentication and user information
-| `com.sap.cds.security.authorization`  | Logs authorization decisions
-| `com.sap.cds.odata.v2`  | Logs OData V2 request handling in the adapter
-| `com.sap.cds.odata.v4`  | Logs OData V4 request handling in the adapter
-| `com.sap.cds.handlers`  | Logs sequence of executed handlers as well as the lifecycle of RequestContexts and ChangeSetContexts
-| `com.sap.cds.persistence.sql` | Logs executed queries such as CQN and SQL statements (w/o parameters)
-| `com.sap.cds.persistence.sql-tx` | Logs transactions, ChangeSetContexts, and connection pool
-| `com.sap.cds.multitenancy`  | Logs tenant-related events and sidecar communication
-| `com.sap.cds.messaging`  | Logs messaging configuration and messaging events
-| `com.sap.cds.remote.odata`  | Logs request handling for remote OData calls
-| `com.sap.cds.remote.wire`  | Logs communication of remote OData calls
-| `com.sap.cds.auditlog`  | Logs audit log events
+| Logger                                | Use case                                                                                             |
+| :------------------------------------ | :--------------------------------------------------------------------------------------------------- |
+| `com.sap.cds.security.authentication` | Logs authentication and user information                                                             |
+| `com.sap.cds.security.authorization`  | Logs authorization decisions                                                                         |
+| `com.sap.cds.odata.v2`                | Logs OData V2 request handling in the adapter                                                        |
+| `com.sap.cds.odata.v4`                | Logs OData V4 request handling in the adapter                                                        |
+| `com.sap.cds.handlers`                | Logs sequence of executed handlers as well as the lifecycle of RequestContexts and ChangeSetContexts |
+| `com.sap.cds.persistence.sql`         | Logs executed queries such as CQN and SQL statements (w/o parameters)                                |
+| `com.sap.cds.persistence.sql-tx`      | Logs transactions, ChangeSetContexts, and connection pool                                            |
+| `com.sap.cds.multitenancy`            | Logs tenant-related events and sidecar communication                                                 |
+| `com.sap.cds.messaging`               | Logs messaging configuration and messaging events                                                    |
+| `com.sap.cds.remote.odata`            | Logs request handling for remote OData calls                                                         |
+| `com.sap.cds.remote.wire`             | Logs communication of remote OData calls                                                             |
+| `com.sap.cds.auditlog`                | Logs audit log events                                                                                |
 
 Most of the loggers are used on DEBUG level by default as they produce quite some log output. It's convenient to control loggers on package level, for example, `com.sap.cds.security` covers all loggers that belong to this package (namely `com.sap.cds.security.authentication` and `com.sap.cds.security.authorization`).
 
@@ -289,7 +289,7 @@ Step-by-step description on how to access a bash session in the application's co
 1. Locate java executable and JDBC driver:
 
    By default `JAVA_HOME` isn't set in the buildpack and contains minimal tooling, as it tries to minimize the container size. However, the default location of the `java` executable is `/layers/paketo-buildpacks_sap-machine/jre/bin`.
-   
+
    For convenience, store the path into a variable, for example, `JAVA_HOME`:
    ```sh
    export JAVA_HOME=/layers/paketo-buildpacks_sap-machine/jre/bin/
@@ -357,6 +357,10 @@ In addition, it's possible to add manual instrumentations using the [Open Teleme
 The configuration steps below assume that your application uses the [SAP Java Buildpack](https://help.sap.com/docs/btp/sap-business-technology-platform/sap-jakarta-buildpack).
 :::
 
+:::tip
+You can conveniently enhance your application with Open Telemetry observability by running the command `cds add cloud-logging --with-telemetry` in your project directory.
+:::
+
 Configure your application to enable the Open Telemetry Java Agent by adding or adapting the `JBP_CONFIG_JAVA_OPTS` parameter in your deployment descriptor:
 
 ::: code-group
@@ -461,7 +465,7 @@ The following steps describe the required configuration:
 By default, instrumentation for CAP-specific components is disabled, so that no traces and spans are created even if the Open Telemetry Java Agent has been configured. It's possible to selectively activate specific spans by changing the log level for a component.
 
 | Logger                                         | Required Level | Description                                                |
-|------------------------------------------------|----------------|------------------------------------------------------------|
+| ---------------------------------------------- | -------------- | ---------------------------------------------------------- |
 | `com.sap.cds.otel.span.ODataBatch`             | `INFO`         | Spans for individual requests of a OData $batch request.   |
 | `com.sap.cds.otel.span.CQN`                    | `INFO`         | Spans for executed CQN statement.                          |
 | `com.sap.cds.otel.span.OutboxCollector`        | `INFO`         | Spans for execution of the transactional outbox collector. |
@@ -602,12 +606,12 @@ To add actuator support in your application, add the following dependency:
 
 The following table lists some of the available actuators that might be helpful to understand the internal status of the application:
 
-| Actuator    | Description
-| :--------| :--------
-| `metrics`    | Thread pools, connection pools, CPU, and memory usage of JVM and HTTP web server
-| `beans`    | Information about Spring beans created in the application
-| `env`    | Exposes the full Spring environment including application configuration
-| `loggers`    | List and modify application loggers
+| Actuator  | Description                                                                      |
+| :-------- | :------------------------------------------------------------------------------- |
+| `metrics` | Thread pools, connection pools, CPU, and memory usage of JVM and HTTP web server |
+| `beans`   | Information about Spring beans created in the application                        |
+| `env`     | Exposes the full Spring environment including application configuration          |
+| `loggers` | List and modify application loggers                                              |
 
 By default, nearly all actuators are active. You can switch off actuators individually in the configuration. The following configuration turns off `flyway` actuator:
 

java/outbox.md

@@ -34,7 +34,7 @@ Once the transaction succeeds, the messages are read from the database table and
 
 To enable the persistence for the outbox, you need to add the service `outbox` of kind `persistent-outbox` to the `cds.requires` section in the _package.json_ or _cdsrc.json_, which will automatically enhance your CDS model in order to support the persistent outbox.
 
-```jsonc 
+```jsonc
 {
   // ...
   "cds": {
@@ -124,12 +124,12 @@ public class MySpringComponent {
 ... it must be ensured that there are no unprocessed entries left.
 
 Removing a custom outbox from the `cds.outbox.services` section doesn't remove the
-entries from the `cds.outbox.Messages` table. The entries remain in the `cds.outbox.Messages` table and isn't
+entries from the `cds.outbox.Messages` table. The entries remain in the `cds.outbox.Messages` table and aren't
 processed anymore.
 
 :::
 
-### Outbox Event Versions 
+### Outbox Event Versions
 
 In scenarios with multiple deployment versions (blue/green), situations may arise in which the outbox collectors of the older deployment cannot process the events generated by a newer deployment. In this case, the event can get stuck in the outbox, with all the resulting problems.
 
@@ -182,7 +182,7 @@ CqnService remoteS4 = ...;
 CqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4);
 ```
 
-If a method on the outboxed service has a return value, it will always return `null` since it is executed asynchronously. A common example for this are the `CqnService.run(...)` methods.
+If a method on the outboxed service has a return value, it will always return `null` since it's executed asynchronously. A common example for this are the `CqnService.run(...)` methods.
 To improve this the API `OutboxService.outboxed(Service, Class)` can be used, which wraps a service with an asynchronous suited API while outboxing it.
 This can be used together with the interface `AsyncCqnService` to outbox remote OData services:
 
@@ -212,7 +212,7 @@ A service wrapped by an outbox can be unboxed by calling the API `OutboxService.
 service are executed synchronously without storing the event in an outbox.
 
 ::: warning Java Proxy
-A service wrapped by an outbox is a [Java Proxy](https://docs.oracle.com/javase/8/docs/technotes/guides/reflection/proxy.html). Such a proxy only implements the _interfaces_ of the object it is wrapping. This means an outboxed service proxy can't be casted to the class implementing the underlying service object.
+A service wrapped by an outbox is a [Java Proxy](https://docs.oracle.com/javase/8/docs/technotes/guides/reflection/proxy.html). Such a proxy only implements the _interfaces_ of the object that it's wrapping. This means an outboxed service proxy can't be casted to the class implementing the underlying service object.
 :::
 
 ::: tip Custom outbox for scaling
@@ -262,7 +262,7 @@ The outbox by default retries publishing a message, if an error occurs during pr
 This behavior makes applications resilient against unavailability of external systems, which is a typical use case for outbox message processing.
 
 However, there might also be situations in which it is not reasonable to retry publishing a message.
-For example, when the processed message causes a semantic error - typically due to a 400 Bad request - on the external system.
+For example, when the processed message causes a semantic error - typically due to a `400 Bad request` - on the external system.
 Outbox messages causing such errors should be removed from the outbox message table before reaching the maximum number of retry attempts and instead application-specific
 counter-measures should be taken to correct the semantic error or ignore the message altogether.
 
@@ -307,13 +307,142 @@ void handleAuditLogProcessingErrors(OutboxMessageEventContext context) {
 
 [Learn more about `EventContext.proceed()`.](./event-handlers/#proceed-on){.learn-more}
 
-## Troubleshooting
+## Outbox Dead Letter Queue
 
-To manually delete entries in the `cds.outbox.Messages` table, you can either
-expose it in a service or programmatically modify it using the `cds.outbox.Messages`
-database entity.
+The transactional outbox tries to process each entry a specific number of times. The number of attempts is configurable per outbox by setting the configuration `cds.outbox.services.<key>.maxAttempts`.
+
+[Learn more about CDS Properties.](./developing-applications/properties){.learn-more}
+
+Once the maximum number of attempts is exceeded, the corresponding entry is not touched anymore and hence it can be regarded as dead. Dead outbox entries are not deleted automatically. They remain in the database and it's up to the application to take care of the entries. By defining a CDS service, the dead entries can be managed conveniently. Let's have a look, how you can develop a Dead Letter Queue for the transactional outbox.
+
+::: warning Changing configuration between deployments
+
+It's possible to increase the value of the configuration `cds.outbox.services.<key>.maxAttempts` in between of deployments. Older entries which have reached their max attempts in the past would be retried automatically after deployment of the new microservice version. If the dead letter queue has a large size, this leads to unintended load on the system.
+
+:::
+
+
+### Define the Service
+
+::: code-group
+
+```cds [srv/outbox-dead-letter-queue-service.cds]
+using from '@sap/cds/srv/outbox';
+
+@requires: 'internal-user'
+service OutboxDeadLetterQueueService {
+
+  @readonly
+  entity DeadOutboxMessages as projection on cds.outbox.Messages
+    actions {
+      action revive();
+      action delete();
+    };
+
+}
+```
+
+:::
+
+The `OutboxDeadLetterQueueService` provides an entity `DeadOutboxMessages` which is a projection on the outbox table `cds.outbox.Messages` that has two bound actions:
+
+- `revive()` sets the number of attempts to `0` such that the outbox entry is going to be processed again.
+- `delete()` deletes the outbox entry from the database.
+
+Filters can be applied as for any other CDS defined entity, for example, to filter for a specific outbox where the outbox name is stored in the field `target` of the entity `cds.outbox.Messages`.
+
+::: warning `OutboxDeadLetterQueueService` for internal users only
+
+It is crucial to make the service `OutboxDeadLetterQueueService` accessible for internal users only as it contains sensitive data that could be exploited for malicious purposes if unauthorized changes are performed.
+
+[Learn more about pseudo roles](../guides/security/authorization#pseudo-roles){.learn-more}
+
+:::
+
+### Filter for Dead Entries
+
+This filtering can't be done on the database since the maximum number of attempts is only available from the CDS properties.
+
+To ensure that only dead outbox entries are returned when reading `DeadOutboxMessages`, the following code provides the handler for the `DeadLetterQueueService` and the `@After-READ` handler that filters for the dead outbox entries:
+
+```java
+@Component
+@ServiceName(OutboxDeadLetterQueueService_.CDS_NAME)
+public class DeadOutboxMessagesHandler implements EventHandler {
+
+	@After(entity = DeadOutboxMessages_.CDS_NAME)
+	public void filterDeadEntries(CdsReadEventContext context) {
+		CdsProperties.Outbox outboxConfigs = context.getCdsRuntime().getEnvironment().getCdsProperties().getOutbox();
+		List<DeadOutboxMessages> deadEntries = context
+				.getResult()
+				.listOf(DeadOutboxMessages.class)
+				.stream()
+				.filter(entry -> entry.getAttempts() >= outboxConfigs.getService(entry.getTarget()).getMaxAttempts())
+				.toList();
+
+		context.setResult(deadEntries);
+	}
+}
+```
+
+[Learn more about event handlers.](./event-handlers/){.learn-more}
+
+### Implement Bound Actions
+
+```java
+@Autowired
+@Qualifier(PersistenceService.DEFAULT_NAME)
+private PersistenceService db;
+
+@On
+public void reviveOutboxMessage(DeadOutboxMessagesReviveContext context) {
+  CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel());
+  AnalysisResult analysisResult = analyzer.analyze(context.getCqn());
+  Map<String, Object> key = analysisResult.rootKeys();
+  Messages deadOutboxMessage = Messages.create((String) key.get(Messages.ID));
+
+  deadOutboxMessage.setAttempts(0);
+
+  this.db.run(Update.entity(Messages_.class).entry(key).data(deadOutboxMessage));
+  context.setCompleted();
+}
+
+@On
+public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) {
+  CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel());
+  AnalysisResult analysisResult = analyzer.analyze(context.getCqn());
+  Map<String, Object> key = analysisResult.rootKeys();
+
+  this.db.run(Delete.from(Messages_.class).byId(key.get(Messages.ID)));
+  context.setCompleted();
+}
+```
+
+The injected `PersistenceService` instance is used to perform the operations on the `Messages` entity since the entity `DeadOutboxMessages` is read-only. Both handlers first retrieve the ID of the entry and then they perform the corresponding operation on the database.
+
+[Learn more about CQL statement inspection.](./working-with-cql/query-introspection#cqnanalyzer){.learn-more}
 
 ::: tip Use paging logic
-Avoid to read all entries of the `cds.outbox.Messages` table at once, as the size of an entry is unpredictable
+Avoid to read all entries of the `cds.outbox.Messages` or `OutboxDeadLetterQueueService.DeadOutboxMessages` table at once, as the size of an entry is unpredictable
 and depends on the size of the payload. Prefer paging logic instead.
 :::
+
+## Observability using Open Telemetry
+
+The transactional outbox integrates Open Telemetry for logging telemetry data.
+
+[Learn more about observability with Open Telemetry.](./operating-applications/observability#open-telemetry){.learn-more}
+
+The following KPIs are logged in addition to the spans described in the [observability chapter](./operating-applications/observability):
+
+| KPI Name                                   | Description                                                                                            | KPI Type |
+| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ | -------- |
+| `com.sap.cds.outbox.coldEntries`           | Number of entries that could not be delivered after repeated attempts and will not be retried anymore. | Gauge    |
+| `com.sap.cds.outbox.remainingEntries`      | Number of entries which are pending for delivery.                                                      | Gauge    |
+| `com.sap.cds.outbox.maxStorageTimeSeconds` | Maximum time in seconds an entry was residing in the outbox.                                           | Gauge    |
+| `com.sap.cds.outbox.medStorageTimeSeconds` | Median time in seconds of an entry stored in the outbox."                                              | Gauge    |
+| `com.sap.cds.outbox.minStorageTimeSeconds` | Minimal time in seconds an entry was stored in the outbox.                                             | Gauge    |
+| `com.sap.cds.outbox.incomingMessages`      | Number of incoming messages of the outbox.                                                             | Counter  |
+| `com.sap.cds.outbox.outgoingMessages`      | Number of outgoing messages of the outbox.                                                             | Counter  |
+
+The KPIs are logged per microservice instance (in case of horizontal scaling), outbox, and tenant.