SS 1254 Check app status using curl (#14)

* Add integration tests

* Move unit tests into a subdirectory named unit

* Git actions should only run unit tests, not integration tests

* Add a class for curl-like probing of user apps and associated tests.

* Add app probe method to status queue.

* Add structured well-defined types for status records, probe results and POST payload and refactor the main classes to use these types.

* Add more type hints. Remove deprecated code.

* Add app URL resolving for shiny proxy apps

* Introduce URL probing feature. Modify the status queue process logic to adhere to new rules for prelim running and deleted statuses.

* Add unit test for status queue probing logic

* Complete new logic including URL probing in the status queue process method. Correct timeouts used for k8s list_namespaced_pod when watch is False.

* Add CLI options for running the program in two additional modes: diagnostics and probetest.

* Display configuration of APP_PROBE_STATUSES at startup.

* Fix Dockerfile for base image vs curl and jq package versions.

* Rename all uses of new-status to status as this is now the new convention in StatusRecord. Rename to new-status just before sending the payload to the API as expected.

* Fix a timing issue in the status queue process method

* Add support for out of cluster URL resolution for local development. Introduce new env var PROBE_PF.

* Add a new env var TLS_SSL_VERIFICATION to control TLS or SSL verification, used by both the main program and integration tests.

Author: alfredeen

.github/workflows/ci.yaml

@@ -72,7 +72,7 @@ jobs:
             pip install -r requirements.txt
 
       - name: Run tests
-        run: python -m unittest discover -s tests
+        run: python -m unittest discover -s tests/unit
 
 
   build:

.github/workflows/publish.yaml

@@ -44,7 +44,7 @@ jobs:
           pip install -r requirements.txt
 
       - name: Run tests
-        run: python -m unittest discover -s tests
+        run: python -m unittest discover -s tests/unit
 
   publish:
     name: Publish to ghcr.io

Dockerfile

@@ -1,31 +1,28 @@
 FROM python:3.12-alpine3.20
 LABEL maintainer="serve@scilifelab.se"
 
+# create non-root user
 RUN addgroup -S serve && adduser -S -G serve serve
 
 WORKDIR /app
 
 COPY requirements.txt /app/requirements.txt
 
-# Install runtime deps
-RUN apk add --no-cache \
-        curl=8.12.1-r0 \
-        jq=1.7.1-r0 \
-    && pip install --no-cache-dir --upgrade "pip~=25.2" \
-    && pip install --no-cache-dir -r requirements.txt \
-    && rm requirements.txt
+# install runtime deps (unversioned to avoid Alpine repo rotation breakage)
+# hadolint ignore=DL3018
+RUN apk add --no-cache curl jq \
+ && pip install --no-cache-dir --upgrade "pip~=25.2" \
+ && pip install --no-cache-dir -r requirements.txt \
+ && rm requirements.txt
 
-# Copy source, owned by non-root user
+# copy source with correct ownership, owned by non-root user
 COPY --chown=serve:serve serve_event_listener/ /app/serve_event_listener/
 
-# Make sure the package root is importable
 ENV PYTHONPATH=/app
 
-# Drop privileges for runtime
 USER serve
 
-# Run the package module so imports resolve properly
+# run the package module so imports resolve properly
 ENTRYPOINT ["python3", "-m", "serve_event_listener.main"]
-
-# Sensible defaults; can override container.args
+# sensible defaults; can override container.args
 CMD ["--namespace", "default", "--label-selector", "type=app"]

README.md

@@ -50,7 +50,33 @@ export USERNAME=<admin username (Serve)>
 export PASSWORD=<admin password (Serve)>
 ```
 
-To retrieve additional log messages, set:
+Modify the `TLS_SSL_VERIFICATION` environment variable if needed to disable SSL verification or point to a self-signed cert. Options are:
+```bash
+- "1"/"true"  = verify (default)
+- "0"/"false" = do not verify (dev/self-signed)
+- "/path.pem" = verify using provided CA/cert bundle
+```
+
+#### URL probing
+URL probing can complement the k8s event stream to detect app status. Note that during development, if running the service outside of a cluster such as using docker compose, then URL probing may require port fowarding and extra hosts defined.  
+
+To enable URL probing, set the environment variable APP_PROBE_STATUSES to the status codes for which the probing should run.
+This is currently only available for the Running and Deleted statuses.
+
+```bash
+export APP_PROBE_STATUSES=Running,Deleted
+```
+
+An environment variable APP_PROBE_APPS controls the app types that the URL probing controls. Currently only available for shiny and shiny-proxy (default).
+
+```bash
+export APP_PROBE_APPS=shiny,shiny-proxy
+```
+
+There are additional environment variables with sensible defaults that control URL resolution. See app_urls.py for more information.
+
+#### Debug output
+To retrieve additional log messages during development, set:
 
 ```bash
 export DEBUG=True
@@ -64,10 +90,22 @@ Navigate to the project directory and execute the following command to run the s
 python3 -m serve_event_listener.main --namespace <some-namespace> --label-selector <some label selector>
 ```
 
-### Running the unit tests
+### Running the service in other modes
+
+The program can also be run at the command line in two other modes, diagnostics and probetest.
+
+diagnostics: The dignostics mode simply prints the effective settings and exits
+```bash
+python3 -m serve_event_listener.main --mode diagnostics
+```
+
+probetest: The probe test mode performs a single-shot URL probing test against a specified URL.
 
+Arguments:
+- Required argument: --probe-url
+- Optional arguments: --probe-insecure, --probe-connect-timeout, --probe-read-timeout
 ```bash
-python -m unittest discover -s tests
+python3 -m serve_event_listener.main --mode probetest --probe-url <url-to-probe>
 ```
 
 ## Docker Container Setup
@@ -98,3 +136,41 @@ The following are the main function arguments that can be passed to run the prog
 
 - `--namespace`: Kubernetes namespace to watch (default: `default`).
 - `--label-selector`: Label selector for filtering pods (default: `type=app`).
+
+## Testing
+
+This project contains both unit tests and integration tests.
+
+### Running the unit tests
+
+```bash
+python -m unittest discover -s tests/unit/
+```
+
+### Running the integration tests
+
+To run the integration tests, the variable RUN_INTEGRATION_TESTS must be set to 1.
+
+1. Start the target service (as defined by BASE_URL). Ensure that KUBECONFIG is set or can be resolved.
+
+2. Then run the integration tests:
+```bash
+RUN_INTEGRATION_TESTS=1 python -m unittest discover -v -s tests/integration/
+```
+
+If you have an existing app available for testing in your target k8s environment, then you can use it to run additional tests using env var PROBE_RELEASE, like so:
+
+```bash
+RUN_INTEGRATION_TESTS=1 PROBE_RELEASE=<app-release> NAMESPACE_UNDER_TEST=default python -m unittest discover -v -s tests/integration/
+```
+
+### Pytest
+
+If you instead prefer to use Pytest for nicer output etc:
+
+```bash
+pip install pytest
+pytest tests/unit -v
+RUN_INTEGRATION_TESTS=1 pytest tests/integration -v
+RUN_INTEGRATION_TESTS=1 PROBE_RELEASE=<app-release> pytest tests/integration -v
+```

serve_event_listener/app_urls.py

@@ -0,0 +1,90 @@
+from __future__ import annotations
+
+import os
+from typing import Optional
+from urllib.parse import urlunparse
+
+from serve_event_listener.el_types import StatusRecord
+
+# for non-cluster, development / testing
+_HOST_GATEWAY = "host.docker.internal"
+_PF = os.getenv("PROBE_PF")
+
+
+def _host_for(service: str, namespace: str) -> str:
+    """Build host according to DNS mode env settings."""
+    mode = (os.getenv("APP_URL_DNS_MODE", "short") or "short").lower()
+    suffix = os.getenv("APP_URL_DNS_SUFFIX")
+    if mode == "fqdn":
+        return f"{service}.{namespace}.svc.cluster.local"
+    if suffix:
+        return f"{service}.{namespace}.{suffix}"
+    # default short form: service.namespace
+    return f"{service}.{namespace}"
+
+
+def _port() -> str:
+    return os.getenv("APP_URL_PORT", "80")
+
+
+def _scheme() -> str:
+    return os.getenv("APP_URL_SCHEME", "http")
+
+
+def _pf_port_for_release(release: str) -> Optional[str]:
+    if not _PF:
+        return None
+    s = _PF.strip()
+    if s.isdigit():
+        return s
+    # parse mapping: rel:port,rel2:port2
+    for part in s.split(","):
+        if ":" in part:
+            rel, port = part.split(":", 1)
+            rel, port = rel.strip(), port.strip()
+            if rel == release and port.isdigit():
+                return port
+    return None
+
+
+def resolve_app_url(
+    rec: StatusRecord, *, fallback_namespace: Optional[str] = None
+) -> Optional[str]:
+    """
+    Return a cluster-internal HTTP URL for the given StatusRecord, or None if unknown.
+
+    Currently supports:
+      - app-type == 'shiny-proxy':
+          service: <release>-<SHINYPROXY_SERVICE_SUFFIX>
+          host:    per DNS mode (short/fqdn/custom suffix)
+          path:    <SHINYPROXY_PATH_PREFIX>/<release>/
+    """
+    app_type = (rec.get("app-type") or "").lower()
+    assert app_type is not None, "app_type should be set"
+    if not app_type:
+        return None
+
+    release = rec.get("release")
+    assert release is not None, "release should be set"
+    if not release:
+        return None
+
+    namespace = rec.get("namespace") or fallback_namespace or "default"
+
+    if app_type == "shiny-proxy":
+        pf_port = _pf_port_for_release(release)
+        if pf_port:
+            # bypass proxy: talk to the port-forward on the host
+            return f"http://{_HOST_GATEWAY}:{pf_port}/app/{release}/"
+
+        # apply the normal ingress/cluster logic
+        suffix = os.getenv("SHINYPROXY_SERVICE_SUFFIX", "shinyproxyapp")
+        path_prefix = os.getenv("SHINYPROXY_PATH_PREFIX", "/app").rstrip("/")
+        service = f"{release}-{suffix}"
+        host = _host_for(service, namespace)
+        path = f"{path_prefix}/{release}/"
+        netloc = f"{host}:{_port()}"
+        return urlunparse((_scheme(), netloc, path, "", "", ""))
+
+    # Other app types are not yet supported
+    return None

serve_event_listener/el_types.py

@@ -0,0 +1,69 @@
+"""Defines common data types"""
+
+from typing import Any, Literal, Mapping, NotRequired, Optional, TypedDict
+
+# Canonical status/app types
+Status = Literal["Running", "Pending", "Deleted", "Failed", "Unknown", "Succeeded"]
+AppType = Literal["shiny", "shiny-proxy"]
+ProbeStatus = Literal["Running", "Unknown", "NotFound"]
+
+# Result block produced by the AppAvailabilityProbe (optional on payload)
+ProbeBlockDict = TypedDict(
+    "ProbeBlockDict",
+    {
+        "status": ProbeStatus,  # e.g., "Running" | "Unknown" | "NotFound"
+        "port80_status": Optional[int],  # HTTP status code (if any)
+        "note": str,  # short diagnostic
+        "url": str,  # probed URL
+    },
+    total=False,
+)
+
+# Internal, per-release record kept by StatusData (snake-ish keys ok here)
+StatusRecord = TypedDict(
+    "StatusRecord",
+    {
+        "release": str,
+        "status": Status,
+        "event-ts": str,  # ISO8601 "YYYY-mm-ddTHH:MM:SS.sssZ"
+        "namespace": NotRequired[str],
+        "resource_version": NotRequired[str],
+        "pod": NotRequired[str],  # current pod name (if relevant)
+        "labels": NotRequired[Mapping[str, str]],
+        "reason": NotRequired[str],
+        "message": NotRequired[str],
+        "app-type": NotRequired[AppType],
+        "app-url": NotRequired[str],
+        "curl-probe": NotRequired[ProbeBlockDict],
+    },
+    total=False,
+)
+
+# External payload for POST-ing
+EventMsg = TypedDict(
+    "EventMsg",
+    {
+        "pod-msg": Optional[str],
+        "container-msg": Optional[str],
+    },
+    total=True,
+)
+
+PostPayload = TypedDict(
+    "PostPayload",
+    {
+        "release": str,
+        "new-status": Status,
+        "event-msg": EventMsg,
+        "event-ts": Optional[str],  # ISO8601 or None
+        "token": str,
+    },
+    total=False,
+)
+
+
+def validate_status_record(rec: Mapping[str, Any]) -> None:
+    """Lightweight runtime guard for required fields (raises ValueError)."""
+    missing = [k for k in ("release", "status", "event-ts") if k not in rec]
+    if missing:
+        raise ValueError(f"StatusRecord missing required fields: {missing}")