Try a different approach at running the services

Author: lubomir

.tekton/README-INTEGRATION-TESTS.md

@@ -4,11 +4,7 @@ This directory contains integration test scenarios for the Compose Tracking Serv
 
 ## Overview
 
-The integration test scenario validates that the built CTS container image works correctly by:
-
-1. **Deploying a test environment** - Creates a PostgreSQL database and deploys the CTS service
-2. **Running integration tests** - Validates API endpoints and service functionality
-3. **Cleaning up** - Removes all test resources
+The integration test scenario validates that the built CTS container image works correctly by running tests within the Konflux CI environment's permission constraints.
 
 ## Files
 
@@ -21,25 +17,29 @@ The integration test scenario validates that the built CTS container image works
 
 ### Test Pipelines
 
-- **`integration-test-k8s.yaml`** - Main integration test pipeline (recommended)
-  - Deploys PostgreSQL database
-  - Deploys CTS service with the built image
-  - Runs comprehensive API tests
-  - Automatically cleans up resources
+**Note:** Konflux CI service accounts have limited permissions and cannot create Deployments or Services. The test pipelines below work within these constraints.
+
+- **`integration-test-container.yaml`** - Full integration test (recommended, currently in use)
+  - Uses podman within test container to run CTS + PostgreSQL
+  - Tests actual API functionality with database
+  - Validates all major endpoints
+  - Most comprehensive testing approach
 
-- **`integration-test-simple.yaml`** - Simplified validation pipeline
-  - Only validates image structure and metadata
-  - Faster but less comprehensive
-  - Useful for quick smoke tests
+- **`integration-test-pod.yaml`** - Sidecar-based approach (alternative)
+  - Uses Tekton sidecars to run CTS and PostgreSQL together
+  - Cleaner than podman approach
+  - May require sidecar support in your Konflux environment
 
-- **`integration-test-pipeline.yaml`** - Alternative podman-based approach
-  - Uses podman-in-podman for testing
-  - More complex, kept for reference
+- **`integration-test-basic.yaml`** - Minimal validation (fallback)
+  - Only validates image metadata and structure using skopeo
+  - No service deployment needed
+  - Fast and guaranteed to work, but doesn't test functionality
+  - Use as fallback if podman/sidecars don't work
 
 ### Test Scripts
 
 - **`../tests/integration_test.py`** - Standalone integration test script
-  - Can be run locally or in CI
+  - Can be run locally for development
   - Tests all major API endpoints
   - Usage: `CTS_URL=http://localhost:5005 python3 tests/integration_test.py`
 
@@ -67,27 +67,27 @@ The integration tests validate:
 4. Test pipeline deploys and validates the image
 5. Results are reported back to the PR/commit
 
-### Pipeline Execution Flow
+### Pipeline Execution Flow (container approach)
 
 ```
 ┌─────────────────────────────┐
 │ 1. Parse Snapshot           │  Extract image URL from Konflux snapshot
 └─────────────┬───────────────┘
               │
 ┌─────────────▼───────────────┐
-│ 2. Deploy PostgreSQL DB     │  Create database for testing
+│ 2. Start PostgreSQL         │  Launch postgres container with podman
 └─────────────┬───────────────┘
               │
 ┌─────────────▼───────────────┐
-│ 3. Deploy CTS Service       │  Deploy CTS with test image
+│ 3. Start CTS Service        │  Launch CTS container with podman
 └─────────────┬───────────────┘
               │
 ┌─────────────▼───────────────┐
-│ 4. Run Integration Tests    │  Execute API validation tests
+│ 4. Run Integration Tests    │  Test API endpoints with Python
 └─────────────┬───────────────┘
               │
 ┌─────────────▼───────────────┐
-│ 5. Cleanup (always runs)    │  Remove test resources
+│ 5. Cleanup                  │  Stop and remove containers
 └─────────────────────────────┘
 ```
 
@@ -121,6 +121,9 @@ python3 tests/integration_test.py --url http://localhost:5005 --timeout 120
 If you have a Kubernetes cluster with Tekton installed:
 
 ```bash
+# First, apply the pipeline definition
+kubectl apply -f .tekton/integration-test-container.yaml
+
 # Create a test snapshot (adjust IMAGE_URL)
 cat <<EOF | kubectl apply -f -
 apiVersion: tekton.dev/v1
@@ -129,7 +132,7 @@ metadata:
   name: cts-integration-test-manual
 spec:
   pipelineRef:
-    name: cts-integration-test-k8s
+    name: cts-integration-test-container
   params:
   - name: SNAPSHOT
     value: |
@@ -169,43 +172,57 @@ self._run_test("My new feature", self.test_my_new_feature)
 
 ### Modifying the Test Environment
 
-Edit `.tekton/integration-test-k8s.yaml` to change:
+Edit `.tekton/integration-test-container.yaml` to change:
 
-- Database configuration (in `deploy-test-database` task)
-- CTS environment variables (in `deploy-cts-service` task)
-- Test execution logic (in `run-integration-tests` task)
+- Database configuration (in the PostgreSQL podman run command)
+- CTS environment variables (in the CTS podman run command)
+- Test execution logic (in the Python test script)
 
 ## Troubleshooting
 
 ### Tests Failing in Konflux
 
 1. Check PipelineRun logs:
    ```bash
-   kubectl logs -n team-sp-rhelcmp-tenant -l tekton.dev/pipeline=cts-integration-test-k8s
+   kubectl get pipelinerun -n team-sp-rhelcmp-tenant -l test.appstudio.openshift.io/scenario=cts-integration-test
+   kubectl logs -n team-sp-rhelcmp-tenant <pipelinerun-name> --all-containers
    ```
 
 2. Check if image is accessible:
    ```bash
    skopeo inspect docker://YOUR_IMAGE_URL
    ```
 
-3. Check pod logs:
-   ```bash
-   kubectl logs -n team-sp-rhelcmp-tenant -l app=cts-test
-   ```
+3. Look for podman errors in the test logs - the pipeline includes container logs on failure
 
 ### Tests Timing Out
 
-- Increase timeout in IntegrationTestScenario
-- Check pod events: `kubectl describe pod -l app=cts-test`
-- Verify image pull is not failing
-- Check database readiness probe
+- Check if podman is available in the test container
+- Increase wait times in the pipeline script
+- Verify image pull is not failing (check for registry authentication)
+- Check if PostgreSQL container starts successfully
 
 ### Database Connection Issues
 
-- Verify PostgreSQL deployment is ready
-- Check service DNS resolution
-- Review CTS logs for connection errors
+- Review the PostgreSQL podman logs in the test output
+- Verify the connection string is correct
+- Check if the database is accepting connections on localhost:5432
+- Ensure CTS waits long enough for database to be ready
+
+### Podman Not Available
+
+If podman doesn't work in your Konflux environment:
+
+1. Switch to the basic validation approach:
+   ```yaml
+   # In cts-integration-test.yaml
+   pathInRepo: .tekton/integration-test-basic.yaml
+   ```
+
+2. Or try the sidecar approach:
+   ```yaml
+   pathInRepo: .tekton/integration-test-pod.yaml
+   ```
 
 ## Best Practices
 

.tekton/cts-integration-test.yaml

@@ -13,14 +13,14 @@ spec:
     - name: url
       value: https://github.com/release-engineering/cts
     - name: revision
-      value: main
+      value: main  # Change to your branch for testing, then back to main before merge
     - name: pathInRepo
-      value: .tekton/integration-test-k8s.yaml
+      value: .tekton/integration-test-container.yaml
   contexts:
   - name: service-validation
     description: |
-      Deploys CTS with PostgreSQL database and validates that:
-      - Service starts correctly
-      - Database migrations run successfully
-      - API endpoints respond correctly
-      - OpenAPI specification is accessible
+      Tests the CTS container image by:
+      - Starting CTS service with PostgreSQL database using podman
+      - Validating API endpoints respond correctly
+      - Testing OpenAPI specification availability
+      - Verifying database connectivity

.tekton/integration-test-basic.yaml

@@ -0,0 +1,130 @@
+apiVersion: tekton.dev/v1
+kind: Pipeline
+metadata:
+  name: cts-integration-test-basic
+spec:
+  description: |
+    Basic integration test for CTS that works within Konflux permissions.
+    Tests image structure and validates the built image without requiring
+    additional Kubernetes resources or privileged containers.
+
+  params:
+  - name: SNAPSHOT
+    description: JSON string representing the snapshot under test
+    type: string
+
+  tasks:
+  - name: validate-and-test
+    taskSpec:
+      params:
+      - name: SNAPSHOT
+        type: string
+      results:
+      - name: test-result
+        description: Overall test result
+      steps:
+      - name: test-image
+        image: quay.io/konflux-ci/appstudio-utils:5bd0d6efcc7c828b0c6f5667630e36a580801fe9
+        script: |
+          #!/usr/bin/env bash
+          set -euo pipefail
+
+          echo "=========================================="
+          echo "CTS Image Integration Test"
+          echo "=========================================="
+          echo ""
+
+          # Parse snapshot
+          IMAGE_URL=$(echo '$(params.SNAPSHOT)' | jq -r '.components[] | select(.name=="cts") | .containerImage')
+          echo "Image: $IMAGE_URL"
+          echo ""
+
+          # Install tools
+          dnf install -y skopeo jq
+
+          # Test 1: Image is accessible
+          echo "Test 1: Image Accessibility"
+          echo "----------------------------"
+          if skopeo inspect "docker://$IMAGE_URL" > /tmp/inspect.json; then
+            echo "✓ Image exists and is accessible"
+          else
+            echo "✗ Failed to access image"
+            exit 1
+          fi
+          echo ""
+
+          # Test 2: Image metadata
+          echo "Test 2: Image Metadata"
+          echo "----------------------"
+          NAME=$(jq -r '.Name' /tmp/inspect.json)
+          DIGEST=$(jq -r '.Digest' /tmp/inspect.json)
+          echo "  Name: $NAME"
+          echo "  Digest: $DIGEST"
+          echo "✓ Image metadata valid"
+          echo ""
+
+          # Test 3: Image labels
+          echo "Test 3: Image Labels"
+          echo "--------------------"
+          LABELS=$(jq -r '.Labels' /tmp/inspect.json)
+          if echo "$LABELS" | jq -e '.description' >/dev/null; then
+            DESC=$(echo "$LABELS" | jq -r '.description')
+            echo "  Description: $DESC"
+            echo "✓ Expected labels present"
+          else
+            echo "⚠ Warning: Some labels missing (non-fatal)"
+          fi
+          echo ""
+
+          # Test 4: Image layers
+          echo "Test 4: Image Structure"
+          echo "-----------------------"
+          LAYERS=$(jq -r '.LayersData | length' /tmp/inspect.json)
+          SIZE=$(jq -r '.Size' /tmp/inspect.json)
+          echo "  Layers: $LAYERS"
+          echo "  Size: $(($SIZE / 1024 / 1024)) MB"
+          if [ "$LAYERS" -gt 0 ]; then
+            echo "✓ Image has valid layer structure"
+          else
+            echo "✗ Invalid image structure"
+            exit 1
+          fi
+          echo ""
+
+          # Test 5: Image can be pulled and files inspected
+          echo "Test 5: Image Contents"
+          echo "----------------------"
+          if skopeo copy "docker://$IMAGE_URL" "dir:/tmp/image-contents" >/dev/null 2>&1; then
+            echo "✓ Image can be extracted"
+
+            # Check for expected files in the image manifest
+            if [ -f /tmp/image-contents/manifest.json ]; then
+              echo "✓ Image manifest valid"
+            fi
+          else
+            echo "⚠ Warning: Could not extract image (non-fatal)"
+          fi
+          echo ""
+
+          # Summary
+          echo "=========================================="
+          echo "All Tests Passed!"
+          echo "=========================================="
+          echo ""
+          echo "Image Details:"
+          echo "  URL: $IMAGE_URL"
+          echo "  Digest: $DIGEST"
+          echo "  Size: $(($SIZE / 1024 / 1024)) MB"
+          echo "  Layers: $LAYERS"
+          echo ""
+          echo "✓ Image is ready for deployment"
+
+          # Save result
+          echo "passed" | tee $(results.test-result.path)
+    params:
+    - name: SNAPSHOT
+      value: $(params.SNAPSHOT)
+
+  results:
+  - name: TEST_OUTPUT
+    value: $(tasks.validate-and-test.results.test-result)