Merge origin/dev into sni-mtls-pop (resync #938 with base)

Bring the SNI mTLS PoP branch current with base dev (which had advanced with
PRs #911, #937, #940, #942), resolving the acquire_token_for_client overlap
between this branch's mtls_proof_of_possession feature and dev's new
forwarded_client_claims parameter (#937).

Conflicts resolved by integrating BOTH sides:
- msal/application.py acquire_token_for_client(): keep both the
  mtls_proof_of_possession and forwarded_client_claims parameters, both request
  setup blocks, and the fail-closed mtls_pop downgrade/binding_certificate
  post-processing (result = _clean_up(...) form).
- msal/application.py _acquire_token_for_client(): keep the is_mtls_pop client
  selection AND the client_claims merge into the OAuth claims parameter; the
  single popped request_data carries token_type/key_id and client_claims on the
  wire.
- tests/test_token_cache.py: union the imports (_key_id_to_str from this branch
  plus _parse_claims_or_raise/_merge_claims from dev).

No Task-1 SNI hardening dropped: the mTLS PoP feature, the
test_credential_x509_output_{pop,bearer} matrix, the region split (pop GLOBAL /
bearer REGIONAL westus3, no regional pop cell), the _call_graph helper, and the
downgrade fail-closed all remain. No escape hatch reintroduced.

Verified: py_compile clean; pytest collects 196 tests (e2e matrix + upstream
suites); 17 mtls_pop + forwarded_client_claims unit tests pass; 49 token_cache
tests pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Author: Robbie-Microsoft

.Pipelines/CI-AND-RELEASE-PIPELINES.md

@@ -26,10 +26,10 @@ including what each pipeline does, when it runs, and how to trigger a release.
 | Push / merge | `dev` |
 | Scheduled | Daily at 11:45 PM Pacific, `dev` branch (only when there are new changes) |
 
-Fast unit-test feedback for PRs targeting **other** branches (e.g. `release-x.y.z`)
-is provided separately by the GitHub Actions workflow
+Package build verification for PRs targeting any branch is provided separately
+by the GitHub Actions workflow
 [`.github/workflows/python-package.yml`](../.github/workflows/python-package.yml),
-which runs the package build and unit tests on every PR.
+which builds the package and runs `twine check` on every PR.
 
 ### Stages
 

.github/workflows/python-package.yml

@@ -1,15 +1,15 @@
-# Build verification + unit tests for the msal Python package.
+# Build verification for the msal Python package.
 #
 # This workflow runs on every PR (against any target branch) to give contributors
-# fast feedback that the package still builds and that the unit tests pass across
-# all supported Python versions.
+# fast feedback that the package still builds and that the distribution metadata
+# passes `twine check`.
 #
-# Post-merge validation on dev, E2E tests, benchmarks, SDL scans, and PyPI
-# publishing are NOT run here. Those run in the ADO pipelines:
-#   - azure-pipelines.yml             (PRs + pushes to dev: unit + E2E + SDL)
+# Unit tests, E2E tests, benchmarks, SDL scans, and PyPI publishing are NOT run
+# here. Those run in the IDDP/ADO pipelines:
+#   - azure-pipelines.yml             (PRs + pushes: unit + E2E + benchmarks + SDL)
 #   - .Pipelines/pipeline-publish.yml (manual release to TestPyPI / PyPI)
 
-name: Build and Unit Tests
+name: Build
 
 on:
   pull_request:
@@ -43,39 +43,3 @@ jobs:
         name: dist
         path: dist/
         retention-days: 7
-
-  ci:
-    name: Unit tests Python ${{ matrix.python-version }}
-    permissions:
-      contents: read
-
-    runs-on: ubuntu-22.04
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
-
-    steps:
-    - uses: actions/checkout@v4
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
-      # It automatically takes care of pip cache, according to
-      # https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#about-caching-workflow-dependencies
-      with:
-        python-version: ${{ matrix.python-version }}
-        cache: 'pip'
-
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        python -m pip install pytest
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-
-    - name: Run unit tests
-      # Skip benchmarks and E2E tests — those require lab credentials and run in ADO.
-      run: |
-        pytest tests/ \
-          --benchmark-skip \
-          --ignore=tests/test_e2e.py \
-          --ignore=tests/test_e2e_manual.py \
-          --ignore=tests/test_fmi_e2e.py

doc/python_version_support_policy.md

@@ -74,8 +74,8 @@ kept in sync with this policy:
 1. **`setup.cfg`** — `python_requires`, the `Programming Language ::
    Python :: 3.x` trove classifiers, and any `python_version`
    environment markers on optional dependencies (e.g. `pymsalruntime`).
-2. **`.github/workflows/python-package.yml`** — the `python-version`
-   matrix used by the `pytest` test job.
+2. **`azure-pipelines.yml`** and **`.Pipelines/template-pipeline-stages.yml`**
+   — the `python-version` matrix used by the PR-gate unit and E2E test stages.
 3. **`tests/test_cryptography.py`** — the N+3 ceiling test that enforces
    tracking the latest `cryptography` release. Newer `cryptography`
    versions routinely drop EOL Python versions, which is the most common

msal/application.py

@@ -253,6 +253,14 @@ def _obtain_token(  # The verb "obtain" is influenced by OAUTH2 RFC 6749
         _data.update(data or {})  # So the content in data param prevails
         _data = {k: v for k, v in _data.items() if v}  # Clean up None values
 
+        # "client_claims" is a cache-key-only pseudo-parameter: callers merge its
+        # value into the standard "claims" body parameter upstream, and it is kept
+        # in the request data solely so it contributes to the extended cache key.
+        # It must not be sent on the wire. Popping it here (from this method's own
+        # local copy) keeps the wire body clean while the caller's data dict — used
+        # for the cache-add event — still carries it.
+        _data.pop("client_claims", None)
+
         if _data.get('scope'):
             _data['scope'] = self._stringify(_data['scope'])
 

msal/oauth2cli/oauth2.py

@@ -74,14 +74,47 @@ class IdTokenAudienceError(IdTokenError):
 class IdTokenNonceError(IdTokenError):
     pass
 
+def _decode_id_token_claims(id_token):
+    """Decode an id_token and return its claims as a dictionary, WITHOUT validation.
+
+    MSAL does not validate the ID token. Per OpenID Connect, an ID token that is
+    obtained via direct communication between the client and the token endpoint
+    (which is how MSAL retrieves tokens) does not need to be validated by the
+    client. Validation, if needed, is the responsibility of the application,
+    which can perform it at the appropriate time (see the issue below).
+    https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/911
+
+    ID token claims would at least contain: "iss", "sub", "aud", "exp", "iat",
+    per `specs <https://openid.net/specs/openid-connect-core-1_0.html#IDToken>`_
+    and it may contain other optional content such as "preferred_username",
+    `maybe more <https://openid.net/specs/openid-connect-core-1_0.html#Claims>`_
+    """
+    return json.loads(decode_part(id_token.split('.')[1]))
+
+
 def decode_id_token(id_token, client_id=None, issuer=None, nonce=None, now=None):
     """Decodes and validates an id_token and returns its claims as a dictionary.
 
+    .. deprecated:: 1.38.0
+        MSAL's token acquisition no longer validates the ID token, because the
+        SDK should not perform any ID token validation
+        (`issue #911 <https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/911>`_).
+        This standalone helper is kept only for backward compatibility and it
+        still performs the legacy validations described below. Prefer the
+        ``id_token_claims`` that MSAL already returns alongside each token, or
+        decode the token yourself, and perform any validation your app requires.
+
     ID token claims would at least contain: "iss", "sub", "aud", "exp", "iat",
     per `specs <https://openid.net/specs/openid-connect-core-1_0.html#IDToken>`_
     and it may contain other optional content such as "preferred_username",
     `maybe more <https://openid.net/specs/openid-connect-core-1_0.html#Claims>`_
     """
+    warnings.warn(
+        "decode_id_token() is deprecated. MSAL's token acquisition no longer "
+        "validates the ID token; validation is the application's "
+        "responsibility. This legacy helper still performs some validation. "
+        "Prefer the id_token_claims returned alongside the token.",
+        DeprecationWarning, stacklevel=2)
     decoded = json.loads(decode_part(id_token.split('.')[1]))
     # Based on https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation
     _now = int(now or time.time())
@@ -157,18 +190,27 @@ class Client(oauth2.Client):
     """
 
     def decode_id_token(self, id_token, nonce=None):
-        """See :func:`~decode_id_token`."""
+        """See :func:`~decode_id_token`.
+
+        .. deprecated:: 1.38.0
+            MSAL's token acquisition no longer validates the ID token. This
+            method is kept for backward compatibility and still performs the
+            legacy validations. See :func:`~decode_id_token`.
+        """
         return decode_id_token(
             id_token, nonce=nonce,
             client_id=self.client_id, issuer=self.configuration.get("issuer"))
 
     def _obtain_token(self, grant_type, *args, **kwargs):
         """The result will also contain one more key "id_token_claims",
-        whose value will be a dictionary returned by :func:`~decode_id_token`.
+        whose value is a dictionary of the (non-validated) ID token claims.
         """
         ret = super(Client, self)._obtain_token(grant_type, *args, **kwargs)
         if "id_token" in ret:
-            ret["id_token_claims"] = self.decode_id_token(ret["id_token"])
+            # MSAL does not validate the ID token. It only decodes the claims,
+            # so that downstream components can build accounts, etc.
+            # https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/911
+            ret["id_token_claims"] = _decode_id_token_claims(ret["id_token"])
         return ret
 
     def build_auth_request_uri(self, response_type, nonce=None, **kwargs):

msal/oauth2cli/oidc.py

@@ -7,7 +7,7 @@
 import warnings
 
 from .authority import canonicalize
-from .oauth2cli.oidc import decode_part, decode_id_token
+from .oauth2cli.oidc import decode_part, _decode_id_token_claims
 from .oauth2cli.oauth2 import Client
 
 
@@ -92,10 +92,22 @@ def _compute_ext_cache_key(data):
     This ensures tokens acquired with different parameter values
     (e.g., different FMI paths) are cached separately.
 
+    The hash may also intentionally include cache-key-only pseudo-parameters
+    such as ``client_claims`` -- these are stripped from the wire body by the
+    oauth2 layer but are retained in *data* precisely so that different
+    client-originated claims route to separate cache entries.
+
     Returns an empty string when *data* has no hashable fields.
 
-    The algorithm matches the Go MSAL implementation (CacheExtKeyGenerator):
-    sorted key+value pairs are concatenated and SHA256 hashed, then base64url encoded.
+    The algorithm matches MSAL .NET's ``ComputeAccessTokenExtCacheKey``: sorted
+    key+value pairs are concatenated (no separators) and SHA256 hashed, then
+    base64url encoded. This keeps the hash byte-identical to MSAL .NET.
+
+    MSAL Go's ``CacheExtKeyGenerator`` has since switched to a length-prefixed
+    encoding (AzureAD/microsoft-authentication-library-for-go#629) to make it
+    injective; Python deliberately tracks .NET instead, so these hashes are not
+    byte-identical to current Go. Caches are not shared across languages, so the
+    difference does not affect runtime correctness.
     """
     if not data:
         return ""
@@ -105,14 +117,71 @@ def _compute_ext_cache_key(data):
     }
     if not cache_components:
         return ""
-    # Sort keys for consistent hashing (matches Go implementation)
+    # Sort keys, then concatenate key+value pairs with no separators. This
+    # matches MSAL .NET's ComputeAccessTokenExtCacheKey byte-for-byte. (See the
+    # docstring re: the Go #629 length-prefixed divergence.)
     key_str = "".join(
         k + cache_components[k] for k in sorted(cache_components.keys())
     )
     hash_bytes = hashlib.sha256(key_str.encode("utf-8")).digest()
     return base64.urlsafe_b64encode(hash_bytes).rstrip(b"=").decode("ascii").lower()
 
 
+def _parse_claims_or_raise(claims):
+    """Parse a claims JSON string into a dict, or raise a friendly ``ValueError``.
+
+    The raw claims value is never included in the error message because it may
+    contain sensitive data. Mirrors MSAL .NET's ``ClaimsHelper.ParseClaimsOrThrow``.
+    """
+    try:
+        parsed = json.loads(claims)
+    except (ValueError, TypeError) as ex:
+        # json.JSONDecodeError (malformed JSON) is a subclass of ValueError;
+        # TypeError is raised when *claims* is not a str/bytes/bytearray. Both
+        # are surfaced as the same friendly ValueError so every caller behaves
+        # consistently regardless of the bad input's type.
+        raise ValueError(
+            "The claims value is not valid JSON. "
+            "See https://openid.net/specs/openid-connect-core-1_0.html#ClaimsParameter."
+        ) from ex
+    if not isinstance(parsed, dict):
+        # A valid JSON array, scalar, or the literal "null" is not a claims object.
+        raise ValueError(
+            "The claims value is not a valid JSON object. "
+            "See https://openid.net/specs/openid-connect-core-1_0.html#ClaimsParameter.")
+    return parsed
+
+
+def _deep_merge_dict(base, overlay):
+    """Recursively merge ``overlay`` into ``base``, returning a new dict.
+
+    Nested dicts are merged; for any other value type, ``overlay`` wins.
+    """
+    result = dict(base)
+    for key, value in overlay.items():
+        if (key in result
+                and isinstance(result[key], dict) and isinstance(value, dict)):
+            result[key] = _deep_merge_dict(result[key], value)
+        else:
+            result[key] = value
+    return result
+
+
+def _merge_claims(claims_a, claims_b):
+    """Merge two claims JSON strings into a single JSON string.
+
+    If either side is empty/None, the other is returned as-is. Mirrors MSAL
+    .NET's ``ClaimsHelper.MergeClaimsObjects``.
+    """
+    if not claims_a:
+        return claims_b
+    if not claims_b:
+        return claims_a
+    merged = _deep_merge_dict(
+        _parse_claims_or_raise(claims_a), _parse_claims_or_raise(claims_b))
+    return json.dumps(merged)
+
+
 def is_subdict_of(small, big):
     return dict(big, **small) == big
 
@@ -350,6 +419,11 @@ def make_clean_copy(dictionary, sensitive_fields):  # Masks sensitive info
             data=make_clean_copy(event.get("data", {}), (
                 "password", "client_secret", "refresh_token", "assertion",
                 "user_federated_identity_credential",
+                # Client-originated claims may carry sensitive values; they are
+                # kept in data only for ext_cache_key computation, so redact them
+                # from the debug log (both the cache-key pseudo-param and the
+                # merged wire parameter).
+                "client_claims", "claims",
             )),
             response=make_clean_copy(event.get("response", {}), (
                 "id_token_claims",  # Provided by broker
@@ -392,8 +466,9 @@ def __add(self, event, now=None):
         refresh_token = response.get("refresh_token")
         id_token = response.get("id_token")
         id_token_claims = response.get("id_token_claims") or (  # Prefer the claims from broker
-            # Only use decode_id_token() when necessary, it contains time-sensitive validation
-            decode_id_token(id_token, client_id=event["client_id"]) if id_token else {})
+            # MSAL does not validate the ID token; it only decodes the claims.
+            # https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/911
+            _decode_id_token_claims(id_token) if id_token else {})
         client_info, home_account_id = self.__parse_account(response, id_token_claims)
 
         target = ' '.join(sorted(event.get("scope") or []))  # Schema should have required sorting