[release-1.8] RHIDP-9238: Troubleshooting Guide

assemblies/assembly-orchestrator-troubleshooting-serverless-workflows.adoc

@@ -0,0 +1,19 @@
+:_mod-docs-content-type: ASSEMBLY
+
+[id="assembly-orchestrator-troubleshooting-serverless-workflows_{context}"]
+
+= Diagnose and resolve serverless workflow issues
+
+Use the following information to diagnose and resolve serverless workflow and visibility issues.
+
+// HTTP error codes in workflows
+include::modules/orchestrator/ref-troubleshoot-workflow-http-error-codes.adoc[leveloffset=+1]
+
+// Workflow errors
+include::modules/orchestrator/proc-workflow-deployment-errors.adoc[leveloffset=+1]
+
+// Common SonataFlow configuration issues
+include::modules/orchestrator/proc-troubleshoot-sonataflow-cross-namespace-issues.adoc[leveloffset=+1]
+
+// Troubleshooting workflows missing from the {product-very-short} UI
+include::modules/orchestrator/proc-troubleshooting-missing-workflows.adoc[leveloffset=+1]

modules/orchestrator/proc-troubleshoot-sonataflow-cross-namespace-issues.adoc

@@ -0,0 +1,163 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-troubleshoot-sonataflow-cross-namespace-issues_{context}"]
+= Troubleshooting cross-namespace SonataFlow configuration and deployment issues
+
+Use this procedure to resolve configuration and deployment failures when SonataFlow workflows are installed in a namespace separate from the core services, or if the Data Index fails to connect to the PostgreSQL database.
+
+.Prerequisites
+* You have administrator privileges to access the OpenShift cluster.
+
+.Procedure
+
+. Identify required namespaces.
+
+* Retrieve the namespace value where {product-very-short} is running using `oc get backstage -A`.
+
+* Identify the SonataFlow Services Namespace by checking for either a `sonataflowclusterplatform` or `sonataflowplatform` instance.
++
+[NOTE]
+====
+By default, the SonataFlow namespace must be the same as the {product-very-short} namespace.
+====
+
+. If the workflow is deployed to a namespace outside the core SonataFlow services, configure network policies to permit the necessary inter-namespace traffic.
++
+[source,subs="+attributes,+quotes"]
+----
+# Example `NetworkPolicy` configuration to ingress traffic into the workflow namespace
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-allow-infra-ns-to-workflow-ns
+  # Sonataflow and Workflows are using the {product-very-short} target namespace.
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  podSelector: {}
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allow knative events to be delivered to workflows.
+            kubernetes.io/metadata.name: knative-eventing
+      - namespaceSelector:
+          matchLabels:
+            # Allow auxiliary knative function for workflow (such as m2k-save-transformation)
+            kubernetes.io/metadata.name: knative-serving
+      - namespaceSelector:
+          matchLabels:
+            # Allow communication between the serverless logic operator and the workflow namespace.
+            kubernetes.io/metadata.name: openshift-serverless-logic
+----
+
+. Add `SonataFlowClusterPlatform` Custom Resource as shown in the following configuration:
++
+[source,yaml]
+----
+oc create -f - <<EOF
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowClusterPlatform
+metadata:
+  name: cluster-platform
+spec:
+  platformRef:
+    name: sonataflow-platform
+    namespace: $RHDH_NAMESPACE
+----
+
+. To allow communication between {product-very-short} namespace and the workflow namespace, create the following network policies:
+
+.. Allow {product-very-short} services to accept traffic from workflows. Create an additional network policy within the {product-very-short} instance namespace as shown in the following configuration::
++
+[source,yaml]
+----
+oc create -f - <<EOF
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-external-workflows-to-rhdh
+  # Namespace where network policies are deployed
+  namespace: $RHDH_NAMESPACE
+spec:
+  podSelector: {}
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allow SonataFlow services to communicate with new/additional workflow namespace.
+            kubernetes.io/metadata.name: $ADDITIONAL_WORKFLOW_NAMESPACE
+----
+
+.. Allow traffic from {product-very-short}, SonataFlow and Knative. Create a network policy within the additional workflow namespace as shown in the following configuration:
++
+[source,yaml]
+----
+oc create -f - <<EOF
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-rhdh-and-knative-to-workflows
+  namespace: $ADDITIONAL_WORKFLOW_NAMESPACE
+spec:
+  podSelector: {}
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allows traffic from pods in the {product-very-short} namespace.
+            kubernetes.io/metadata.name: $RHDH_NAMESPACE
+      - namespaceSelector:
+          matchLabels:
+            # Allows traffic from pods in the Knative Eventing namespace.
+            kubernetes.io/metadata.name: knative-eventing
+      - namespaceSelector:
+          matchLabels:
+            # Allows traffic from pods in the Knative Serving namespace.
+            kubernetes.io/metadata.name: knative-serving
+----
+
+. (Optional) Create an `allow-intra-namespace` policy in the workflow namespace to enable unrestricted communication among all pods within that namespace.
+
+. If workflow persistence is required, perform the following configuration steps:
+
+.. Create a dedicated PostgreSQL Secret containing database credentials within the workflow namespace as shown in the following configuration:
++
+[source,yaml]
+----
+oc get secret sonataflow-psql-postgresql -n <your_namespace> -o yaml > secret.yaml
+sed -i '/namespace: <your_namespace>/d' secret.yaml
+oc apply -f secret.yaml -n $ADDITIONAL_NAMESPACE
+----
+
+.. Configure the workflow `serviceRef` property to correctly reference the PostgreSQL service namespace as shown in the following configuration:
++
+[source,yaml]
+----
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+  ...
+spec:
+  ...
+  persistence:
+    postgresql:
+      secretRef:
+        name: sonataflow-psql-postgresql
+        passwordKey: postgres-password
+        userKey: postgres-username
+      serviceRef:
+        databaseName: sonataflow
+        databaseSchema: greeting
+        name: sonataflow-psql-postgresql
+        namespace: $POSTGRESQL_NAMESPACE
+        port: 5432
+----
++
+`namespace`::
+Enter the namespace where the PostgreSQL server is deployed.
+
+. If the `sonataflow-platform-data-index-service` cannot connect to the PostgreSQL database on startup, perform the following diagnostic checks:
+
+.. Verify that the PostgreSQL Pod has fully transitioned to a `running` and operational status.
+Allow additional time for database initialization before expecting related service pods (`DataIndex`, `JobService`) to establish a connection.
+
+.. If the PostgreSQL Server operates in a dedicated namespace (for example, outside {product-very-short}), verify that network policies are configured to allow ingress traffic from the SonataFlow services namespace. Network policies might prevent the Data Index and Job Service pods from connecting to the database.

modules/orchestrator/proc-troubleshooting-missing-workflows.adoc

@@ -0,0 +1,117 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-troubleshooting-missing-workflows_{context}"]
+= Troubleshooting workflows missing from the {product-very-short} UI
+
+You can perform the following checks to verify the workflow status and connectivity when the deployed workflow is missing from the {product-very-short} Orchestrator UI.
+
+.Prerequisites
+
+* You have administrator privileges to access the OpenShift cluster where {product-very-short} and SonataFlow services are running.
+
+.Procedure
+
+. Verify if the workflow uses GitOps profile. The {product-very-short} Orchestrator UI displays only the workflows that use this profile. Make sure the workflow definition and the SonataFlow manifests use the GitOps profile.
+
+. Verify that the workflow pod has started and is ready. The readiness of a workflow pod depends on its successful registration with the Data Index. When a workflow initializes, it performs the following actions:
+.. It attempts to create its schema in the database (if persistence is active).
+.. It attempts to register itself to the Data Index. The workflow pod remains in an unready state until it successfully registers to the Data Index.
++
+Check the workflow deployment for additional status and error messages that might be unavailable in the pod log.
+
+. Check if the workflow pod can reach the Data Index service. Connect to the workflows pod and send the following GraphQL request to the Data Index:
++
+[source,subs="+attributes,+quotes"]
+----
+curl -g -k  -X POST  -H "Content-Type: application/json" \
+                    -d '{"query":"query{ ProcessDefinitions  { id, serviceUrl, endpoint } }"}' \
+                    http://sonataflow-platform-data-index-service.<your_namespace>/graphql
+----
++
+Use the Data Index service and namespace as defined in your environment. By default, this is the same namespace where {product-very-short} is installed. If your SonataFlow resources are installed in a separate namespace, use `<your_namespace>`. Check if the {product-very-short} pod can reach the workflow service by running the following command:
++
+[source,bash]
+----
+curl http://<workflow_service>.<workflow_namespace>/management/processes
+----
+
+. Connect to the {product-very-short} pod. Verify its connection to the Data Index service and inspect the {product-very-short} pod logs for messages from the Orchestrator plugin.
++
+To inspect the logs, identify the {product-very-short} pod and run the following *oc logs* command:
++
+[source,yaml]
+----
+oc get pods -n <your_namespace>
+oc logs <rhdh_pod_name> -n <your_namespace>
+----
++
+You must find messages indicating it is attempting to fetch workflow information from the Data Index, similar to the following:
++
+[source,yaml]
+----
+{"level":"\u001b[32minfo\u001b[39m","message":"fetchWorkflowInfos() called: http://sonataflow-platform-data-index-service.<your_namespace>","plugin":"orchestrator","service":"backstage","span_id":"fca4ab29f0a7aef9","timestamp":"2025-08-04 17:58:26","trace_flags":"01","trace_id":"5408d4b06373ff8fb34769083ef771dd"}
+----
++
+Notice the _"plugin":"orchestrator"_ that can help to filter the messages.
+
+. Make sure the Data Index properties are set in the `-managed-props` ConfigMap of the workflow as shown in the following configuration:
++
+[source,yaml]
+----
+kogito.data-index.health-enabled = true
+kogito.data-index.url = http://sonataflow-platform-data-index-service.<your_namespace>
+...
+mp.messaging.outgoing.kogito-processdefinitions-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/definitions
+mp.messaging.outgoing.kogito-processinstances-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/processes
+----
++
+[NOTE]
+====
+The `-managed-props` ConfigMap is located in the same namespace as the workflow and is generated by the Openshift Serverless Logic (OSL) Operator.
+====
++
+These properties, along with similar settings for the Job Services, indicate that the (OSL) Operator successfully registered the Data Index service.
+
+. Confirm that the workflow is registered in the Data Index database. Connect to the database used by the Data Index and run the following command from the PSQL instance pod:
++
+[source,bash]
+----
+PGPASSWORD=<psql password> psql -h localhost -p 5432 -U < user> -d sonataflow
+----
++
+Replace `<psql password>` and `<user>` with your database credentials.
++
+Run the following SQL commands to query the registered workflow definitions:
++
+[source,yaml]
+----
+sonataflow=# SET search_path TO "sonataflow-platform-data-index-service";
+sonataflow=# select id, name from definitions;
+----
++
+You must see your workflows listed in the query results.
+
+. Make sure you have enabled Data Index and Job Service in the `SonataFlowPlatform` custom resource (CR) as shown in the following configuration:
++
+[source,yaml]
+----
+services:
+    dataIndex:
+      enabled: true
+    jobService:
+      enabled: true
+----
++
+If you fail to enable the Data Index and the Job Services in the `SonataFlowPlatform` custom resource (CR), the Orchestrator plugin fails to fetch the available workflows.
++
+[NOTE]
+====
+You can also manually edit the `SonataFlowPlatform` CR instance to trigger the re-creation of workflow-related manifests.
+====
+
+. Set the RBAC permissions correctly. For more information, see {authorization-book-link}#managing-authorizations-by-using-the-rest-api[RBAC documentation].
+
+[role="_additional-resources"]
+.Additional resources
+
+* {monitoring-and-logging-book-link}#configuring-the-application-log-level-by-using-the-operator_assembly-rhdh-observability[Configuring the application log level by using the {product} Operator]

modules/orchestrator/proc-workflow-deployment-errors.adoc

@@ -0,0 +1,24 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-workflow-deployment-errors_{context}"]
+= Troubleshooting common workflow deployment errors
+
+Use these steps to diagnose and resolve common workflow deployment, connectivity, or configuration failures.
+
+.Procedure
+
+. If the workflow operation fails, examine the container log of the specific workflow instance to determine the cause by running the following command:
++
+[source,terminal]
+----
+$ oc logs my-workflow-xy73lj
+----
+
+. If the workflow fails to reach an HTTPS endpoint, check the pod log for an SSL certificate verification failure. This occurs if the target endpoint uses a Certificate Authority (CA) that the workflow cannot verify. The resulting error resembles the following:
++
+[source,yaml]
+----
+sun.security.provider.certpath.SunCertPathBuilderException - unable to find valid certification path to requested target
+----
+
+. To resolve the SSL certificate error, load the additional CA certificate into the running workflow container.

modules/orchestrator/ref-troubleshoot-workflow-http-error-codes.adoc

@@ -0,0 +1,25 @@
+:_mod-docs-content-type: REFERENCE
+
+[id="ref-troubleshoot-workflow-http-error-codes_{context}"]
+= Troubleshoot workflow HTTP error codes
+
+Workflow operations fail when a service endpoint returns an HTTP error code. The user interface displays the HTTP code and error message. See link:https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status[external documentation] for a complete list of HTTP status code meanings.
+
+The following table lists common HTTP errors encountered during workflow execution:
+
+[cols="25%,25%,50%", frame="all", options="header"]
+|===
+|HTTP code|Description|Possible cause
+
+|`401`
+|Unauthorized access
+|The token, password, or username provided for the endpoint might be incorrect or expired.
+
+|`403`
+|Forbidden
+|The server understood the request but refused to process it due to insufficient permissions to a resource or action.
+
+|`409`
+|Conflict
+|The workflow attempted to create or update a resource (for example, Kubernetes or OpenShift resources) that already exists.
+|===

titles/orchestrator/master.adoc

@@ -20,4 +20,6 @@ include::assemblies/assembly-install-rhdh-orchestrator-plugin-in-an-air-gapped-e
 
 include::assemblies/assembly-building-and-deploying-serverless-workflows.adoc[leveloffset=+1]
 
+include::assemblies/assembly-orchestrator-troubleshooting-serverless-workflows.adoc[leveloffset=+1]
+
 include::assemblies/assembly-orchestrator-plugins-technical-appendixes.adoc[leveloffset=+1]