Issue test tokens from a local Keycloak realm

Replace the bespoke oauth2-mock-server image with Keycloak. The mock was
built for this suite alone; Keycloak is a conformant OpenID Connect
server that flotilla and sara can also be run against for local
development, and that an operator deploying outside Azure could use in
earnest. custom_realms/robotics-realm.json is the single realm
definition, mounted both here and by those repositories' compose files,
so a local run and a CI run exercise the same clients, scopes and roles.

Keycloak does not derive a token's audience from the requested scope the
way Entra does, so each API gets a client scope carrying one audience
mapper: isar-api yields aud=isar-test. Exactly one such scope may be
requested per token -- two audience mappers make Keycloak emit aud as an
array, which fastapi-azure-auth rejects outright.

Three protocol mappers exist only to satisfy that library's Entra-shaped
token model: a hardcoded ver, a hardcoded nbf (Keycloak emits none and
the library requires it) and a flat roles claim, since the default
nested realm_access.roles maps to neither ISAR's User.roles nor .NET's
ClaimTypes.Role.

Keycloak has no equivalent of the mock's /issue-token, so a role set is
chosen by picking the service account that holds it. The negative
assertions gain a client holding Role.User.HUA alone, kept distinct from
the client with no roles because Flotilla answers 403 for an
insufficient role but 401 for a token carrying none.

Two things the spike caught. Waiting on the discovery document is not
enough: it answers while clients are still importing, and a token minted
in that window comes back 401, so readiness now mints a real token. And
the client credentials are constants rather than settings, because a
setting named INTEGRATION_TESTS_CLIENT_SECRET is silently overridden by
the stale Entra value of that name still sitting in developers' .env
files -- which presents only as an unexplained 401.

Author: oysand

.github/workflows/run_integration_tests.yml

@@ -15,7 +15,7 @@ on:
             Azure client secret for the service principal with read access to the
             integration-test key vault, which now holds only the MQTT credentials.
             No longer used for authentication between the services: those tokens come
-            from a local OAuth2 mock issuer. Kept optional rather than removed because
+            from a local Keycloak realm. Kept optional rather than removed because
             this workflow is consumed at @main by every robotics repo and each one
             declares it; removing it outright would break them all at once.
 
@@ -32,7 +32,7 @@ permissions:
 
 env:
   # Only MQTT credentials are still read from the key vault. Authentication between
-  # the services is handled by a local OAuth2 mock issuer started by the test suite,
+  # the services is handled by a local Keycloak realm started by the test suite,
   # so no Entra ID app registrations are involved.
   KEYVAULT_NAME: FlotillaTestsKv
   AZURE_TENANT_ID: 3aa4a235-b6e2-48d5-9195-7fcf05b459b0
@@ -66,7 +66,7 @@ jobs:
           # against the hashed passwd_file committed in equinor/flotilla, so these
           # remain real secrets. The app-registration client secrets that used to
           # be fetched here are gone: service-to-service auth now runs against a
-          # local OAuth2 mock issuer.
+          # local Keycloak realm.
           secrets=(
             FLOTILLA-MQTT-PASSWORD
             FLOTILLA-BROKER-SERVER-KEY

README.md

@@ -12,29 +12,58 @@ The following components are currently included in the integration tests:
 - Azure Blob Storage (emulated with Azurite)
 - [ISAR Robot](https://github.com/equinor/isar-robot) (your friendly neighbourhood mocked robot which provides the answers you need)
 - [SARA](https://github.com/equinor/sara) (storage and analysis of robot acquired data)
-- A local OAuth2 mock issuer (see [Authentication](#authentication))
+- A local Keycloak realm (see [Authentication](#authentication))
 
 ## Authentication
 The tests run with authentication **enabled and genuinely exercised**, but without Microsoft
-Entra ID. A local [oauth2-mock-server](https://github.com/axa-group/oauth2-mock-server)
-container acts as the OpenID Connect issuer for the whole stack:
-
-- Flotilla and SARA run with `ASPNETCORE_ENVIRONMENT=IntegrationTest`, which selects their
-  `appsettings.IntegrationTest.json` and points token validation at the mock.
-- ISAR is pointed at the mock with `ISAR_OPENID_CONFIG_URL`.
-- Flotilla acquires its downstream ISAR/SARA tokens from the mock too, via a
-  `GenericOidcAuthorizationHeaderProvider` registered only in that environment.
+Entra ID. A [Keycloak](https://www.keycloak.org/) container acts as the OpenID Connect issuer
+for the whole stack:
+
+- Flotilla and SARA run with `ASPNETCORE_ENVIRONMENT=IntegrationTest`, whose
+  `appsettings.IntegrationTest.json` sets `Authentication:Provider=Oidc` and points token
+  validation at the realm.
+- ISAR is pointed at the realm with `ISAR_OPENID_CONFIG_URL`.
+- Flotilla acquires its downstream ISAR/SARA tokens from the realm too, via
+  `GenericOidcAuthorizationHeaderProvider`.
 - The test process mints its own tokens from the same issuer.
 
 This means **no app registrations, no tenant and no client secrets** are needed, and there is
 nothing to rotate. The container fixtures assert that each service rejects unauthenticated
 callers before any test runs, and the mission tests attempt unauthorised interference mid-flight
-(wrong audience, missing role) and then assert the mission completed unaffected — so a
+(wrong audience, insufficient role) and then assert the mission completed unaffected — so a
 misconfiguration cannot silently disable authentication.
 
 MQTT is the one exception: it uses username/password validated by the broker against the
 hashed `passwd_file` committed in `equinor/flotilla`, so those credentials remain real secrets.
 
+### The realm
+`robotics_integration_tests/custom_realms/robotics-realm.json` is imported at startup. It is
+also what a developer mounts to run flotilla or sara against Keycloak locally (see those
+repositories' READMEs), so a local run and a CI run exercise the same clients, scopes and roles.
+
+Tokens are minted with the client credentials grant. Unlike Entra, Keycloak does not derive the
+audience from the requested scope: each API has a client scope — `isar-api`, `sara-api`,
+`flotilla-api`, `pointilla-api` — carrying a single audience mapper onto `isar-test`,
+`sara-test` and so on.
+
+**Request exactly one API scope per token.** Two audience mappers make Keycloak emit `aud` as an
+array, which `fastapi-azure-auth` rejects, so ISAR would answer an opaque 401.
+
+Because Keycloak has no ad-hoc "issue me a token with these claims" endpoint, a role set is
+chosen by picking the service account that holds it:
+
+| Client | Roles |
+| --- | --- |
+| `integration-tests` | every role the three services require |
+| `integration-tests-limited-role` | `Role.User.HUA` only — recognised, but insufficient |
+| `integration-tests-no-role` | none |
+| `flotilla-test` | used by Flotilla for its downstream ISAR/SARA calls |
+
+The realm also carries three protocol mappers that exist only to satisfy `fastapi-azure-auth`'s
+Entra-shaped token model, which ISAR uses: a hardcoded `ver`, a hardcoded `nbf` (Keycloak does
+not emit one, and the library requires it) and a **flat** `roles` claim, since Keycloak's default
+nested `realm_access.roles` maps to neither ISAR's `User.roles` nor .NET's `ClaimTypes.Role`.
+
 ## Run the integration tests through remote workflow call
 To run the integration tests in a remote repository, this [workflow](./.github/workflows/run_integration_tests.yml) has been set up. 
 
@@ -142,5 +171,4 @@ Two things worth knowing:
 - `isar-robot`'s `uv.lock` pins `isar` from PyPI, so the locally built `isar` wheel is installed
   over the released one.
 
-The mosquitto broker is always the published image, and the OAuth2 mock is built automatically
-by the test fixtures.
+The mosquitto broker and Keycloak are always the published images.

robotics_integration_tests/armada.py

@@ -10,7 +10,7 @@
 )
 from robotics_integration_tests.custom_containers.isar import IsarRobot
 from robotics_integration_tests.custom_containers.mosquitto import FlotillaBroker
-from robotics_integration_tests.custom_containers.oauth2_mock import OAuth2Mock
+from robotics_integration_tests.custom_containers.keycloak import Keycloak
 from robotics_integration_tests.custom_containers.postgres import (
     FlotillaDatabase,
     SaraDatabase,
@@ -23,7 +23,7 @@
 class Armada:
     def __init__(self) -> None:
         self.network: Network | None = None
-        self.oauth_mock: OAuth2Mock | None = None
+        self.keycloak: Keycloak | None = None
         self.test_id: str = ""
         self.flotilla_database: FlotillaDatabase | None = None
         self.flotilla_broker: FlotillaBroker | None = None

robotics_integration_tests/conftest.py

@@ -43,9 +43,9 @@
     create_flotilla_broker_container,
     FlotillaBroker,
 )
-from robotics_integration_tests.custom_containers.oauth2_mock import (
-    OAuth2Mock,
-    create_oauth2_mock_container,
+from robotics_integration_tests.custom_containers.keycloak import (
+    Keycloak,
+    create_keycloak_container,
 )
 from robotics_integration_tests.custom_containers.postgres import (
     SaraDatabase,
@@ -66,8 +66,8 @@
 )
 from robotics_integration_tests.settings.settings import settings
 from robotics_integration_tests.utilities.authentication import (
-    configure_mock_issuer,
-    reset_mock_issuer,
+    configure_issuer,
+    reset_issuer,
 )
 from robotics_integration_tests.utilities.authentication_assertions import (
     assert_authentication_is_enforced,
@@ -100,6 +100,7 @@ def _pull_latest_images() -> None:
     native_images = [
         settings.POSTGRESQL_IMAGE,
         settings.AZURITE_IMAGE,
+        settings.KEYCLOAK_IMAGE,
     ]
 
     for image in amd64_images:
@@ -144,30 +145,34 @@ def network():
 
 
 @pytest.fixture
-def oauth_mock(network: Network, test_id: str):
+def keycloak(network: Network, test_id: str):
     """Local OpenID Connect issuer standing in for Azure Entra ID.
 
     Every service in the stack validates its access tokens against this
     container, and the test process mints its own tokens from it, so the suite
     needs no app registrations, no tenant and no client secrets.
+
+    The realm is imported from custom_realms/, the same file a developer mounts
+    to run flotilla or sara against Keycloak locally, so a local run and a CI run
+    exercise the same clients, scopes and roles.
     """
-    container, mock = create_oauth2_mock_container(
+    container, issuer = create_keycloak_container(
         network=network,
-        alias=settings.OAUTH_MOCK_ALIAS,
-        port=settings.OAUTH_MOCK_PORT,
+        alias=settings.KEYCLOAK_ALIAS,
+        port=settings.KEYCLOAK_PORT,
         test_id=test_id,
     )
     with container:
-        wait_for_port_mapping_to_be_available(container=container, port=mock.port)
-        mock.wait_until_ready()
+        wait_for_port_mapping_to_be_available(container=container, port=issuer.port)
+        issuer.wait_until_ready()
 
         # The API helpers build auth headers from module functions with no access
         # to fixtures, so the host URL is handed over globally.
-        configure_mock_issuer(mock.host_url)
+        configure_issuer(issuer.host_url)
         try:
-            yield mock
+            yield issuer
         finally:
-            reset_mock_issuer()
+            reset_issuer()
 
 
 @pytest.fixture
@@ -314,7 +319,7 @@ def teams_webhook_receiver(network: Network, test_id: str):
 @pytest.fixture
 def flotilla_backend(
     network: Network,
-    oauth_mock: OAuth2Mock,
+    keycloak: Keycloak,
     flotilla_database: FlotillaDatabase,
     teams_webhook_receiver: TeamsWebhookReceiver,
     test_id: str,
@@ -356,7 +361,7 @@ def flotilla_backend(
 @pytest.fixture
 def sara(
     network: Network,
-    oauth_mock: OAuth2Mock,
+    keycloak: Keycloak,
     sara_database: SaraDatabase,
     armada_storage: ArmadaStorage,
     test_id: str,
@@ -394,7 +399,7 @@ def sara(
 def armada_without_robots(
     network: Network,
     test_id: str,
-    oauth_mock: OAuth2Mock,
+    keycloak: Keycloak,
     flotilla_broker: FlotillaBroker,
     flotilla_database: FlotillaDatabase,
     flotilla_backend: FlotillaBackend,
@@ -407,7 +412,7 @@ def armada_without_robots(
 
     armada.network = network
     armada.test_id = test_id
-    armada.oauth_mock = oauth_mock
+    armada.keycloak = keycloak
     armada.sara_database = sara_database
     armada.sara = sara
     armada.flotilla_database = flotilla_database
@@ -451,7 +456,7 @@ def armada_with_single_successful_robot(armada_without_robots: Armada):
     blob_conn_data, blob_conn_metadata = _blob_connection_strings(armada)
     with create_isar_robot_container(
         network=armada.network,
-        openid_config_url=armada.oauth_mock.internal_openid_config_url,
+        openid_config_url=armada.keycloak.internal_openid_config_url,
         image=settings.ISAR_ROBOT_IMAGE,
         name=settings.ISAR_ROBOT_NAME,
         port=settings.ISAR_ROBOT_PORT,
@@ -486,7 +491,7 @@ def armada_with_single_failing_robot(armada_without_robots: Armada):
 
     with create_isar_robot_container(
         network=armada.network,
-        openid_config_url=armada.oauth_mock.internal_openid_config_url,
+        openid_config_url=armada.keycloak.internal_openid_config_url,
         image=settings.ISAR_ROBOT_IMAGE,
         name=settings.ISAR_ROBOT_NAME,
         port=settings.ISAR_ROBOT_PORT,
@@ -572,7 +577,7 @@ def armada_with_multiple_robots(armada_without_robots: Armada):
             container = stack.enter_context(
                 create_isar_robot_container(
                     network=armada.network,
-                    openid_config_url=armada.oauth_mock.internal_openid_config_url,
+                    openid_config_url=armada.keycloak.internal_openid_config_url,
                     image=settings.ISAR_ROBOT_IMAGE,
                     name=cfg["name"],
                     port=settings.ISAR_ROBOT_PORT,

robotics_integration_tests/custom_containers/isar.py

@@ -59,8 +59,8 @@ def create_isar_robot_container(
         .with_kwargs(platform="linux/amd64")
         .with_env("ISAR_MQTT_HOST", settings.FLOTILLA_BROKER_ALIAS)
         .with_env("ISAR_MQTT_PASSWORD", settings.ISAR_MQTT_PASSWORD)
-        # Validate access tokens against the local mock issuer rather than Entra
-        # ID. ISAR_AZURE_CLIENT_ID is the expected `aud`; note that settings.py
+        # Validate access tokens against the local Keycloak realm rather than
+        # Entra ID. ISAR_AZURE_CLIENT_ID is the expected `aud`; note that settings.py
         # lets a bare AZURE_CLIENT_ID override it, so that variable must not be
         # set here.
         .with_env("ISAR_OPENID_CONFIG_URL", openid_config_url)