Authenticate the integration tests against a local Keycloak realm

.github/workflows/run_integration_tests.yml

@@ -10,8 +10,14 @@ on:
         default: "latest"
     secrets:
         INTEGRATION_TEST_AZURE_CLIENT_SECRET:
-          required: true
-          description: "Azure client secret for the service principal used in integration tests with access to keyvault"
+          required: false
+          description: |
+            Azure client secret for the service principal with read access to the
+            integration-test key vault, which now holds only the MQTT credentials.
+            No longer used for authentication between the services: those tokens come
+            from a local Keycloak realm. Kept optional rather than removed because
+            this workflow is consumed at @main by every robotics repo and each one
+            declares it; removing it outright would break them all at once.
 
   workflow_dispatch:
     inputs:
@@ -25,6 +31,8 @@ permissions:
   contents: read
 
 env:
+  # Only MQTT credentials are still read from the key vault; service-to-service
+  # authentication runs against a local Keycloak realm.
   KEYVAULT_NAME: FlotillaTestsKv
   AZURE_TENANT_ID: 3aa4a235-b6e2-48d5-9195-7fcf05b459b0
   AZURE_CLIENT_ID: 17d7c036-e4ff-4df6-87fd-0d648a36a727
@@ -53,15 +61,12 @@ jobs:
         run: |
           echo "Retrieving secrets from Key Vault: $KEYVAULT_NAME"
 
-          # You can list the secret names here
+          # MQTT is username/password, validated by the broker against the hashed
+          # passwd_file committed in equinor/flotilla, so these remain real secrets.
           secrets=(
-            INTEGRATION-TESTS-CLIENT-SECRET
             FLOTILLA-MQTT-PASSWORD
-            FLOTILLA-AZURE-CLIENT-SECRET
             FLOTILLA-BROKER-SERVER-KEY
-            ISAR-AZURE-CLIENT-SECRET
             ISAR-MQTT-PASSWORD
-            SARA-AZURE-CLIENT-SECRET
             SARA-MQTT-PASSWORD
           )
 
@@ -120,13 +125,9 @@ jobs:
           GIT_REPOSITORY_FOR_MIGRATIONS_REF: ${{ steps.images.outputs.git_ref }}
           SARA_GIT_REPOSITORY_FOR_MIGRATIONS_REF: ${{ steps.images.outputs.git_ref }}
 
-          INTEGRATION_TESTS_CLIENT_SECRET: ${{ env.INTEGRATION_TESTS_CLIENT_SECRET }}
           FLOTILLA_MQTT_PASSWORD: ${{ env.FLOTILLA_MQTT_PASSWORD }}
-          FLOTILLA_AZURE_CLIENT_SECRET: ${{ env.FLOTILLA_AZURE_CLIENT_SECRET }}
           FLOTILLA_BROKER_SERVER_KEY: ${{ env.FLOTILLA_BROKER_SERVER_KEY }}
-          ISAR_AZURE_CLIENT_SECRET: ${{ env.ISAR_AZURE_CLIENT_SECRET }}
           ISAR_MQTT_PASSWORD: ${{ env.ISAR_MQTT_PASSWORD }}
-          SARA_AZURE_CLIENT_SECRET: ${{ env.SARA_AZURE_CLIENT_SECRET }}
           SARA_MQTT_PASSWORD: ${{ env.SARA_MQTT_PASSWORD }}
 
         run: uv run --frozen pytest -s -n auto robotics_integration_tests

README.md

@@ -11,6 +11,48 @@ The following components are currently included in the integration tests:
 - PostgreSQL database
 - Azure Blob Storage (emulated with Azurite)
 - [ISAR Robot](https://github.com/equinor/isar-robot) (your friendly neighbourhood mocked robot which provides the answers you need)
+- [SARA](https://github.com/equinor/sara) (storage and analysis of robot acquired data)
+- A local Keycloak realm (see [Authentication](#authentication))
+
+## Authentication
+The tests run with authentication **enabled and genuinely exercised**, but without Microsoft
+Entra ID. A [Keycloak](https://www.keycloak.org/) container is the OpenID Connect issuer for the
+whole stack: Flotilla and SARA run with `ASPNETCORE_ENVIRONMENT=IntegrationTest`, whose
+`appsettings.IntegrationTest.json` sets `Authentication:Provider=Oidc`; ISAR is pointed at the
+realm with `ISAR_OPENID_CONFIG_URL`; Flotilla acquires its downstream tokens from the realm too;
+and the test process mints its own from the same issuer.
+
+No app registrations, tenant or client secrets are needed. The fixtures assert that each service
+rejects unauthenticated callers before any test runs, and the mission tests attempt unauthorised
+interference mid-flight and then assert the mission completed unaffected.
+
+MQTT is the exception: username/password validated by the broker against the hashed
+`passwd_file` committed in `equinor/flotilla`, so those remain real secrets.
+
+### The realm
+`robotics_integration_tests/custom_realms/robotics-realm.json` is imported at startup. It is also
+what a developer mounts to run flotilla or sara against Keycloak locally, so a local run and a CI
+run exercise the same clients, scopes and roles.
+
+Unlike Entra, Keycloak does not derive the audience from the requested scope: each API has a
+client scope — `isar-api`, `sara-api`, `flotilla-api`, `pointilla-api` — carrying a single
+audience mapper onto `isar-test`, `sara-test` and so on. **Request exactly one API scope per
+token**; two audience mappers make Keycloak emit `aud` as an array, which ISAR rejects.
+
+Keycloak has no ad-hoc token endpoint, so a role set is chosen by picking the service account
+that holds it:
+
+| Client | Roles |
+| --- | --- |
+| `integration-tests` | every role the three services require |
+| `integration-tests-limited-role` | `Role.User.HUA` only — recognised, but insufficient |
+| `integration-tests-no-role` | none |
+| `flotilla-test` | used by Flotilla for its downstream ISAR/SARA calls |
+
+Three protocol mappers exist only to satisfy `fastapi-azure-auth`'s Entra-shaped token model,
+which ISAR uses: a hardcoded `ver`, a hardcoded `nbf` (Keycloak emits none, the library requires
+it) and a flat `roles` claim, since the default nested `realm_access.roles` maps to neither
+ISAR's `User.roles` nor .NET's `ClaimTypes.Role`.
 
 ## Run the integration tests through remote workflow call
 To run the integration tests in a remote repository, this [workflow](./.github/workflows/run_integration_tests.yml) has been set up. 
@@ -55,6 +97,9 @@ This snippet will enable you to run the integration tests manually and automatic
 INTEGRATION_TEST_AZURE_CLIENT_SECRET
 ```
 
+This secret now only grants read access to the MQTT credentials in the key vault. It is kept
+because every consuming repository declares it.
+
 The input `lane` determines which image tag should be applied to the internally developed packages like Flotilla and ISAR. If input is set as `lane=dev` the newest development images (corresponding to newest push to main branch) will be used while `lane=latest` will use the newest release. 
 
 ## Local development
@@ -63,20 +108,17 @@ Clone the repository and install dependencies with [uv](https://docs.astral.sh/u
 uv sync
 ```
 
-Ensure the following secrets are populated in your local environment, either as environment variables or in a `.env` file in the repository root directory.
+Ensure the following secrets are populated in your local environment, either as environment variables or in a `.env` file in the repository root directory. These are the MQTT credentials; no
+Azure app registration secrets are needed, and you do **not** need to be logged in with `az`.
 
 ```
-INTEGRATION_TESTS_CLIENT_SECRET
-FLOTILLA_AZURE_CLIENT_SECRET
 FLOTILLA_BROKER_SERVER_KEY
 FLOTILLA_MQTT_PASSWORD
-ISAR_AZURE_CLIENT_SECRET
 ISAR_MQTT_PASSWORD
-SARA_AZURE_CLIENT_SECRET
 SARA_MQTT_PASSWORD
 ```
 
-They may all be found in the integration test [keyvault](https://portal.azure.com/#@StatoilSRM.onmicrosoft.com/resource/subscriptions/c389567b-2dd0-41fa-a5da-d86b81f80bda/resourceGroups/FlotillaIntegrationTests/providers/Microsoft.KeyVault/vaults/FlotillaTestsKv/overview).
+They may be found in the integration test [keyvault](https://portal.azure.com/#@StatoilSRM.onmicrosoft.com/resource/subscriptions/c389567b-2dd0-41fa-a5da-d86b81f80bda/resourceGroups/FlotillaIntegrationTests/providers/Microsoft.KeyVault/vaults/FlotillaTestsKv/overview).
 
 You may now run the tests with
 
@@ -118,4 +160,4 @@ Two things worth knowing:
 - `isar-robot`'s `uv.lock` pins `isar` from PyPI, so the locally built `isar` wheel is installed
   over the released one.
 
-The mosquitto broker is always the published image.
+The mosquitto broker and Keycloak are always the published images.

pyproject.toml

@@ -11,13 +11,10 @@ dependencies = [
   "pytest",
   "testcontainers[postgres,azurite,mqtt]",
     "azure-storage-blob",
-    "azure-identity",
-    "azure-keyvault-secrets",
     "pydantic",
     "pydantic-settings",
     "loguru",
     "python-dotenv",
-    "msal",
     "requests"
 ]
 

robotics_integration_tests/armada.py

@@ -10,20 +10,20 @@
 )
 from robotics_integration_tests.custom_containers.isar import IsarRobot
 from robotics_integration_tests.custom_containers.mosquitto import FlotillaBroker
+from robotics_integration_tests.custom_containers.keycloak import Keycloak
 from robotics_integration_tests.custom_containers.postgres import (
     FlotillaDatabase,
     SaraDatabase,
 )
 from robotics_integration_tests.custom_containers.teams_webhook_receiver import (
     TeamsWebhookReceiver,
 )
-from robotics_integration_tests.utilities.keyvault import Keyvault
 
 
 class Armada:
     def __init__(self) -> None:
         self.network: Network | None = None
-        self.keyvault: Keyvault | None = None
+        self.keycloak: Keycloak | None = None
         self.test_id: str = ""
         self.flotilla_database: FlotillaDatabase | None = None
         self.flotilla_broker: FlotillaBroker | None = None