Merge branch 'idp5' into idp4

Author: TomiBelan

README.md

@@ -161,6 +161,66 @@ The key benefit of this design is that it requires zero changes to the back serv
     sudo -u {USER} /opt/shibboleth-idp/bin/plugin.sh -i $PWD/andrvotr-dist/target/idp-plugin-andrvotr-*-SNAPSHOT.tar.gz --noCheck
     ```
 
-<!-- TODO: ## Developing compatible front services -->
+## Developing front services
+
+Front services which want to use Andrvotr to connect to a back service must follow this procedure.
+
+Andrvotr currently does not have a reusable client library in any language. You must implement it yourself.
+
+You will need:
+
+- a SAML Service Provider module or library
+- an HTTP client library with cookie management
+- an HTML parser library
+
+### Login procedure
+
+1.  Read the SAML attribute containing the Andrvotr Authority Token.
+    - It is identified by `Name="tag:fmfi-svt.github.io,2024:andrvotr-authority-token"` and
+      `NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"`.
+    - How to read attributes depends on your SP software. In most cases it should be trivial.
+    - If you use mod_shib (Shibboleth SP), it is not trivial because it discards any unknown attributes. Edit `/etc/shibboleth/attribute-map.xml` and add this:
+      ```xml
+      <Attribute name="tag:fmfi-svt.github.io,2024:andrvotr-authority-token" id="ANDRVOTR_AUTHORITY_TOKEN" />
+      ```
+    - If the IdP does not send you the attribute, the process cannot continue.
+      This is most likely because the IdP is not configured correctly (andrvotr.allowedConnections, andrvotr.apiKeys).
+
+2.  Initialize an empty cookie jar. It will be used for all HTTP requests you send.
+
+3.  Send an HTTP request to the back service URL which initiates the login process.
+    Disable automatic HTTP redirect processing in your client library.
+    (If the URL is not known, use e.g. browser dev tools to see what happens when you click the login button.)
+
+4.  Process the response to decide what request should be sent next:
+    - If it is a HTTP 3xx redirect, the *next request* will be a GET of that Location header.
+    - If it is a pseudo-redirect (an invisible form which is immediately submitted by JavaScript), parse the HTML page.
+      The *next request* will be a POST to that form's action with the form's hidden inputs.
+      Detecting this case might need some finetuning, but searching for the string "`document.forms[0].submit()`" works well in practice.
+    - If it is a success page (back service specific), return success.
+    - Otherwise, return failure.
+
+5.  If the *next request* is `GET https://$your_idp/...`, instead set *next request* to `POST https://$your_idp/idp/profile/andrvotr/fabricate` with POST body parameters:
+    - `front_entity_id` = the SAML SP entity ID of the front service
+    - `api_key` = your Andrvotr API key (must match andrvotr.apiKeys)
+    - `andrvotr_authority_token` = token from the SAML attribute
+    - `target_url` = the original *next request* URL
+
+6.  Send the *next request*.
+
+7.  Go to step 4 to process the response.
+
+### Example implementations
+
+[demo/demo.py](/demo/demo.py) implements an Andrvotr client in 50 lines of Python.
+It is not integrated with an SP -- you must extract the token and run demo.py manually from the command line.
+
+```shell
+./demo.py "$back_service_target_url" "$front_entity_id" "$api_key" "$andrvotr_authority_token"
+```
+
+The demo requires some Python libraries. Either [install "uv"](https://docs.astral.sh/uv/getting-started/installation/) which will take care of it. Or `apt install python3-bs4 python3-requests` and run with `python3 demo.py ...`. Or create a venv with `beautifulsoup4` and `requests`.
+
+[Votr](https://github.com/fmfi-svt/votr/blob/master/fladgejt/login.py) contains another implementation of this procedure (also in Python).
 
 <!-- TODO: ## Similar projects -->

andrvotr-impl/src/main/java/io/github/fmfi_svt/andrvotr/AddressLookupStrategy.java

@@ -0,0 +1,85 @@
+package io.github.fmfi_svt.andrvotr;
+
+import jakarta.servlet.http.HttpServletRequest;
+import java.util.function.Function;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.shibboleth.idp.profile.context.SpringRequestContext;
+import net.shibboleth.shared.primitive.LoggerFactory;
+import net.shibboleth.shared.servlet.HttpServletSupport;
+import org.opensaml.profile.context.ProfileRequestContext;
+import org.slf4j.Logger;
+import org.springframework.webflow.context.ExternalContext;
+import org.springframework.webflow.execution.RequestContext;
+
+/// Works around session address binding.
+///
+/// Shibboleth IdP sessions remember the client IP address. If someone sends a request with a cookie value from a
+/// different address, they will have to reauthenticate. (Note that by "Shibboleth IdP session" we mean interface
+/// `IdPSession`, class `StorageBackedIdPSession`. and config option `idp.session.cookieName`, *not* `JSESSIONID`.)
+///
+/// That's a good thing, but it presents a problem for Andrvotr. The IdP session is initially created by the real user
+/// and bound to their real IP address. When /.../andrvotr/fabricate sends a nested request, its remote address will be
+/// localhost or similar, which is not what Shibboleth expects.
+///
+/// This class works around the issue by locally disabling the session address check during nested Andrvotr requests.
+/// Normal requests are unaffected.
+///
+/// This class is registered as "shibboleth.SessionAddressLookupStrategy" by `AddressLookupStrategyInjector`. It is
+/// called by "PopulateSessionContext" via authn-beans.xml and "ProcessLogout" via logout-beans.xml. Interestingly, it
+/// is only called when reading existing IdP sessions, not for new ones. When StorageBackedSessionManager creates a new
+/// session, it just calls HttpServletSupport.getRemoteAddr(). This might be a Shibboleth bug.
+public final class AddressLookupStrategy implements Function<ProfileRequestContext, String> {
+    private final @Nonnull Logger log = LoggerFactory.getLogger(AddressLookupStrategy.class);
+
+    private final @Nullable Function<ProfileRequestContext, String> nextStrategy;
+
+    public AddressLookupStrategy(@Nullable Function<ProfileRequestContext, String> nextStrategy) {
+        log.info("initialized andrvotr AddressLookupStrategy, nextStrategy = {}", nextStrategy);
+        this.nextStrategy = nextStrategy;
+    }
+
+    public @Nullable String apply(ProfileRequestContext prc) {
+        // Look up the necessary objects.
+        // There is no known situation where these exceptions are thrown.
+        SpringRequestContext shibSpringRequestContext = prc.getSubcontext(SpringRequestContext.class);
+        if (shibSpringRequestContext == null) {
+            throw new RuntimeException("SpringRequestContext is missing");
+        }
+        RequestContext webflowRequestContext = shibSpringRequestContext.getRequestContext();
+        if (webflowRequestContext == null) {
+            throw new RuntimeException("getRequestContext() is null");
+        }
+        ExternalContext externalContext = webflowRequestContext.getExternalContext();
+        if (externalContext == null) {
+            throw new RuntimeException("getExternalContext() is null");
+        }
+        Object nativeRequest = externalContext.getNativeRequest();
+        if (nativeRequest == null) {
+            throw new RuntimeException("getNativeRequest() is null");
+        }
+        if (!(nativeRequest instanceof HttpServletRequest)) {
+            throw new RuntimeException("getNativeRequest() is not a HttpServletRequest");
+        }
+        HttpServletRequest httpRequest = (HttpServletRequest) nativeRequest;
+
+        // If this is a nested request sent by our HttpController to ourselves, return null.
+        // When PopulateSessionContext sees that we returned null, it'll skip the checkAddress() call.
+        if (webflowRequestContext.getRequestScope().contains(Constants.ANDRVOTR_FABRICATION_TOKEN_OK)) {
+            log.info("forcing client address of nested request to null. original was {}", httpRequest.getRemoteAddr());
+            return null;
+        }
+
+        // Return the normal address. Logic copied from
+        // java-identity-provider/idp-session-impl/src/main/java/net/shibboleth/idp/session/impl/PopulateSessionContext.java.
+        if (nextStrategy != null) {
+            String result = nextStrategy.apply(prc);
+            log.trace("client address from nextStrategy is {}", result);
+            return result;
+        } else {
+            String result = HttpServletSupport.getRemoteAddr(httpRequest);
+            log.trace("client address from httpRequest is {}", result);
+            return result;
+        }
+    }
+}

andrvotr-impl/src/main/java/io/github/fmfi_svt/andrvotr/AddressLookupStrategyInjector.java

@@ -0,0 +1,45 @@
+package io.github.fmfi_svt.andrvotr;
+
+import net.shibboleth.shared.component.AbstractInitializableComponent;
+import org.springframework.beans.factory.config.BeanDefinition;
+import org.springframework.beans.factory.config.ConfigurableListableBeanFactory;
+import org.springframework.beans.factory.support.BeanDefinitionBuilder;
+import org.springframework.beans.factory.support.BeanDefinitionRegistry;
+import org.springframework.beans.factory.support.BeanDefinitionRegistryPostProcessor;
+
+/// Registers `AddressLookupStrategy` as the "shibboleth.SessionAddressLookupStrategy" bean.
+///
+/// The bean is documented at https://shibboleth.atlassian.net/wiki/spaces/IDP5/pages/3199506072/SessionConfiguration,
+/// though barely. It has no built-in definition - it is is intended as an optional extension point for administrators.
+/// In the unlikely event an administrator defines their own "shibboleth.SessionAddressLookupStrategy" in their
+/// configuration, this postprocessor renames it to another id and chains it after our implementation.
+public final class AddressLookupStrategyInjector extends AbstractInitializableComponent
+        implements BeanDefinitionRegistryPostProcessor {
+    private static final String TARGET_BEAN_ID = "shibboleth.SessionAddressLookupStrategy";
+    private static final String RENAMED_BEAN_ID = "andrvotr.original.shibboleth.SessionAddressLookupStrategy";
+
+    @Override
+    public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry registry) {
+        // Sadly no logging, because this class apparently runs too early for logging to work.
+
+        boolean exists = registry.containsBeanDefinition(TARGET_BEAN_ID);
+
+        if (exists) {
+            BeanDefinition originalDefinition = registry.getBeanDefinition(TARGET_BEAN_ID);
+            registry.removeBeanDefinition(TARGET_BEAN_ID);
+            registry.registerBeanDefinition(RENAMED_BEAN_ID, originalDefinition);
+        }
+
+        BeanDefinitionBuilder builder = BeanDefinitionBuilder.genericBeanDefinition(AddressLookupStrategy.class);
+        if (exists) {
+            builder.addConstructorArgReference(RENAMED_BEAN_ID);
+        } else {
+            builder.addConstructorArgValue(null);
+        }
+        registry.registerBeanDefinition(TARGET_BEAN_ID, builder.getBeanDefinition());
+    }
+
+    /// postProcessBeanFactory defaults to an empty method in Spring 6.1.0+, but our Spring is too old.
+    @Override
+    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) {}
+}

andrvotr-impl/src/main/java/io/github/fmfi_svt/andrvotr/Constants.java

@@ -15,6 +15,10 @@ private Constants() {}
     // Token value used for internal communication between HttpController and FabricationWebflowListener.
     public static final String ANDRVOTR_FABRICATION_TOKEN_VALUE = "andrvotr-fabrication-token";
 
+    // RequestContext request scope key used for internal communication between FabricationWebflowListener and
+    // AddressLookupStrategy.
+    public static final String ANDRVOTR_FABRICATION_TOKEN_OK = "andrvotr_fabrication_token_ok";
+
     // State and event names defined in the Shibboleth flow "SAML2/Redirect/SSO". Arguably an internal implementation
     // detail of Shibboleth. See class doc of FabricationWebflowListener.
     public static final String STATE_DECODE_MESSAGE = "DecodeMessage";

andrvotr-impl/src/main/java/io/github/fmfi_svt/andrvotr/FabricationWebflowListener.java

@@ -38,8 +38,6 @@
 /// java-identity-provider/idp-conf-impl/src/main/resources/net/shibboleth/idp/flows/saml/saml-abstract-flow.xml.
 public final class FabricationWebflowListener extends AbstractInitializableComponent implements FlowExecutionListener {
 
-    private static final String ANDRVOTR_FABRICATION_TOKEN_OK = "andrvotr_fabrication_token_ok";
-
     private final @Nonnull Logger log = LoggerFactory.getLogger(FabricationWebflowListener.class);
 
     private Config config;
@@ -96,7 +94,7 @@ public void requestSubmitted(RequestContext context) {
         }
 
         log.info("started {} as a nested request inside andrvotr/fabricate", request.getRequestURI());
-        context.getRequestScope().put(ANDRVOTR_FABRICATION_TOKEN_OK, new Object());
+        context.getRequestScope().put(Constants.ANDRVOTR_FABRICATION_TOKEN_OK, new Object());
         addTrace(context, Constants.TRACE_START);
     }
 
@@ -106,7 +104,7 @@ public void eventSignaled(RequestContext context, Event event) {
                 (HttpServletRequest) context.getExternalContext().getNativeRequest();
 
         // If the request does not have the Andrvotr-Internal-Fabrication-Token header, do nothing.
-        if (!context.getRequestScope().contains(ANDRVOTR_FABRICATION_TOKEN_OK)) return;
+        if (!context.getRequestScope().contains(Constants.ANDRVOTR_FABRICATION_TOKEN_OK)) return;
 
         // If we're leaving the "DecodeMessage" state with the "proceed" event (not an error), check whether our
         // configuration allows connections from the front entity ID (sent by HttpController in a header) to the back
@@ -139,7 +137,7 @@ public void eventSignaled(RequestContext context, Event event) {
     @Override
     public void stateEntered(RequestContext context, StateDefinition previousState, StateDefinition state) {
         // If the request does not have the Andrvotr-Internal-Fabrication-Token header, do nothing.
-        if (!context.getRequestScope().contains(ANDRVOTR_FABRICATION_TOKEN_OK)) return;
+        if (!context.getRequestScope().contains(Constants.ANDRVOTR_FABRICATION_TOKEN_OK)) return;
 
         // When moving from "HandleOutboundMessage" to "end", it is expected that the response is already sent, and we
         // can't add response headers anymore. Avoid the warning in addTrace.

andrvotr-impl/src/main/resources/META-INF/net.shibboleth.idp/postconfig.xml

@@ -34,4 +34,8 @@
         p:connectionDisregardTLSCertificate="%{andrvotr.httpclient.connectionDisregardTLSCertificate:false}"
         p:maxConnectionsTotal="%{andrvotr.httpclient.maxConnectionsTotal:100}"
         p:maxConnectionsPerRoute="%{andrvotr.httpclient.maxConnectionsPerRoute:100}" />
+
+    <!-- Spring will auto-detect and run this bean because it implements BeanDefinitionRegistryPostProcessor. -->
+    <bean class="io.github.fmfi_svt.andrvotr.AddressLookupStrategyInjector" />
+
 </beans>

demo/demo.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env -S uv run
+
+# /// script
+# dependencies = [
+#     "beautifulsoup4",
+#     "requests",
+# ]
+# ///
+
+import os
+import re
+import sys
+
+import requests
+import bs4
+
+session = requests.Session()
+
+VERIFY_TLS_CERTS = os.getenv("VERIFY_TLS_CERTS") != "false"
+
+url, front_entity_id, api_key, andrvotr_authority_token = sys.argv[1:]
+post_data = None
+
+while True:
+    print("Requesting", ("POST" if post_data else "GET"), url, *(["with", list(post_data)] if post_data else []))
+
+    if re.match(r'^https://[^/]+/idp/profile/', url):
+        idp_host = url.split("/")[2]
+        assert post_data is None, post_data
+        assert url.startswith("https://" + idp_host + '/idp/profile/SAML2/Redirect/SSO?'), url
+        post_data = { 'front_entity_id': front_entity_id, 'api_key': api_key, 'andrvotr_authority_token': andrvotr_authority_token, 'target_url': url }
+        url = "https://" + idp_host + "/idp/profile/andrvotr/fabricate"
+        print("Nevermind, requesting POST", url, "with", list(post_data))
+
+    response = session.request("POST" if post_data else "GET", url, data=post_data, verify=VERIFY_TLS_CERTS, allow_redirects=False)
+
+    print("Received", response.status_code, response.reason)
+    print()
+
+    if 300 <= response.status_code <= 399 and 'location' in response.headers:
+        url = response.headers['location']
+        post_data = None
+        continue
+
+    if response.status_code == 200 and b"document.forms[0].submit()" in response.content:
+        soup = bs4.BeautifulSoup(response.text)
+        url = soup.form['action']
+        post_data = { input['name']: input['value'] for input in soup.find_all('input') if input['type'] == 'hidden' }
+        continue
+
+    print("Final headers:", response.headers)
+    sys.stdout.buffer.write(b"Final content: [" + response.content + b"]\n")
+    break

resources/release-new-version.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+VERSION=$1
+
+if [[ $VERSION =~ ^v([0-9]+)\.([0-9]+)\.([0-9]+)(-.*)?$ ]]; then
+  X="${BASH_REMATCH[1]}"
+  Y="${BASH_REMATCH[2]}"
+  Z="${BASH_REMATCH[3]}"
+  SUFFIX="${BASH_REMATCH[4]}"
+else
+  echo "invalid version argument"
+  exit 1
+fi
+
+cd "$(dirname "$0")/.."
+
+if [[ "$(git status --porcelain)" ]]; then
+  echo "git status has non-empty output, is your repo dirty?"
+  exit 1
+fi
+
+sed -r -i "1,/<name>/ s@<version>.*</version>@<version>$X.$Y.$Z</version>@" pom.xml */pom.xml
+sed -r -i "s@version = .*@version = $X.$Y.$Z@" andrvotr-impl/src/main/resources/io/github/fmfi_svt/andrvotr/plugin.properties
+
+git commit -a -m "build: release $VERSION"
+git tag "$VERSION"
+
+((Z++))
+
+sed -r -i "1,/<name>/ s@<version>.*</version>@<version>$X.$Y.$Z-SNAPSHOT</version>@" pom.xml */pom.xml
+sed -r -i "s@version = .*@version = $X.$Y.$Z@" andrvotr-impl/src/main/resources/io/github/fmfi_svt/andrvotr/plugin.properties
+
+git commit -a -m "build: bump version after $VERSION"
+
+echo "Done"
+echo "Now push it with: git push --tags"
+echo "GitHub Actions should build it."
+echo "Then update plugins.properties on the meta branch (you can use GitHub web GUI)."
+