feat(context): subgroup visibility + parent-walk membership inheritance (#2256) (#2261)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: rtb-12

.github/workflows/e2e-rust-apps.yml

@@ -167,6 +167,12 @@ jobs:
           - workflow: group-capabilities
             file: workflows/group-capabilities.yml
             app: e2e-kv-store
+          - workflow: group-subgroup-visibility-inheritance
+            file: workflows/group-subgroup-visibility-inheritance.yml
+            app: e2e-kv-store
+          - workflow: group-default-capabilities-propagation
+            file: workflows/group-default-capabilities-propagation.yml
+            app: e2e-kv-store
           - workflow: group-reparent
             file: workflows/group-reparent.yml
             app: e2e-kv-store

Cargo.toml

@@ -9,7 +9,7 @@ description = "Core Calimero infrastructure and tools"
 # Update workspace metadata (see docs/RELEASE.md for versioning and release)
 [workspace.metadata.workspaces]
 # Shared version of all public crates; bump this when cutting a release.
-version = "0.10.1-rc.31"
+version = "0.10.1-rc.32"
 exclude = [
     "./apps/abi_conformance",
     "./apps/blobs",

apps/e2e-kv-store/workflows/group-capabilities.yml

@@ -47,7 +47,7 @@ steps:
 
   # --- SCENARIO 1: Set default capabilities before inviting members ---
   # Capability bitmask: bit 0 = CAN_CREATE_CONTEXT (1),
-  #   bit 1 = CAN_INVITE_MEMBERS (2), bit 2 = CAN_JOIN_OPEN_CONTEXTS (4)
+  #   bit 1 = CAN_INVITE_MEMBERS (2), bit 2 = CAN_JOIN_OPEN_SUBGROUPS (4)
   # Value 7 = all capabilities enabled
 
   - name: Set default capabilities to all (7)
@@ -111,8 +111,8 @@ steps:
 
   # --- SCENARIO 4: Set visibility + verify context sync works ---
 
-  - name: Set default visibility to open
-    type: set_default_visibility
+  - name: Set subgroup visibility to open
+    type: set_subgroup_visibility
     node: e2e-node-1
     group_id: '{{namespace_id}}'
     visibility: open

apps/e2e-kv-store/workflows/group-default-capabilities-propagation.yml

@@ -0,0 +1,138 @@
+name: E2E KV Store - Default Capabilities Propagation
+description: >
+  Regression test for issue #2256 / PR #2261: when an admin overrides
+  the namespace's `default_capabilities` *before* a new member joins,
+  the new member must pick up the overridden value — not a hard-coded
+  fallback the joiner side previously substituted in.
+
+  The pre-fix bug: `create_group` writes `default_capabilities =
+  CAN_JOIN_OPEN_SUBGROUPS` to the creator's local store but never
+  publishes the corresponding `DefaultCapabilitiesSet` governance op,
+  so a joiner couldn't read the value out of the governance DAG. A
+  joiner-side mirror substituted the same hard-coded constant — which
+  silently overrode any admin change made via `set_default_capabilities`.
+
+  The fix: the namespace's current `default_capabilities` is carried
+  in the join-bundle's `default_capabilities` field, so the joiner
+  reflects whatever the responder currently believes — including any
+  admin-issued override. This workflow asserts that contract.
+
+  Flow:
+    1. node-1 installs the app and creates a namespace. By construction
+       of `create_group`, the local default is CAN_JOIN_OPEN_SUBGROUPS (4).
+    2. node-1 admin overrides the default to 0 (no capabilities) via
+       `set_default_capabilities`.
+    3. node-1 invites node-2.
+    4. node-2 joins the namespace.
+    5. Assert node-2's individual capability is 0 (the admin's override),
+       NOT 4 (the create-time hard-coded constant). If the assertion
+       fails with `json_equal({{node2_caps}}, 4)` truthy, the fix has
+       regressed.
+    6. As a downstream sanity-check that the inherited cap actually
+       gates Open-subgroup access: node-1 creates an Open subgroup
+       containing a context, and node-2 tries to join that context.
+       With caps=0 (no CAN_JOIN_OPEN_SUBGROUPS), the parent-walk in
+       `check_group_membership` should refuse — `join_context` must
+       fail. (`join_context_expect_failure` is the merobox action for
+       a deliberate-deny scenario; if the runner doesn't support it,
+       this step can be commented out — the cap-value assertion above
+       is sufficient regression coverage.)
+
+force_pull_image: false
+near_devnet: false
+
+nodes:
+  count: 2
+  image: ghcr.io/calimero-network/merod:edge
+  prefix: e2e-node
+
+steps:
+  # --- Setup ---
+
+  - name: Install application on node-1
+    type: install_application
+    node: e2e-node-1
+    path: res/e2e_kv_store.wasm
+    dev: true
+    outputs:
+      app_id: applicationId
+
+  - name: Assert application installed
+    type: assert
+    statements:
+      - "is_set({{app_id}})"
+
+  - name: Create namespace on node-1
+    type: create_namespace
+    node: e2e-node-1
+    application_id: '{{app_id}}'
+    outputs:
+      namespace_id: namespaceId
+
+  - name: Assert namespace created
+    type: assert
+    statements:
+      - "is_set({{namespace_id}})"
+
+  # --- Admin overrides the create-time default ---
+  # `create_group` populated defaults locally with bit 2
+  # (CAN_JOIN_OPEN_SUBGROUPS = 4). The admin now restricts to 0.
+  # Without the bundle-field fix, node-2 would still pick up the
+  # hard-coded constant (4) instead of this override (0).
+
+  - name: Admin overrides default capabilities to 0
+    type: set_default_capabilities
+    node: e2e-node-1
+    group_id: '{{namespace_id}}'
+    capabilities: 0
+
+  # --- node-2 joins after the override ---
+
+  - name: Create namespace invitation for node-2
+    type: create_namespace_invitation
+    node: e2e-node-1
+    namespace_id: '{{namespace_id}}'
+    outputs:
+      namespace_invitation: invitation
+
+  - name: Node-2 joins namespace
+    type: join_namespace
+    node: e2e-node-2
+    namespace_id: '{{namespace_id}}'
+    invitation: '{{namespace_invitation}}'
+    outputs:
+      member_identity_node2: memberIdentity
+
+  # --- Core regression assertion ---
+  # If the joiner-side fallback wins, node-2 has caps=4. If the bundle
+  # field wins (correct behavior), node-2 has caps=0.
+
+  - name: Get node-2 capabilities as seen by node-1
+    type: get_member_capabilities
+    node: e2e-node-1
+    group_id: '{{namespace_id}}'
+    member_id: '{{member_identity_node2}}'
+    outputs:
+      node2_caps_view_node1: capabilities
+
+  - name: Assert node-2 picked up admin's override (caps=0)
+    type: json_assert
+    statements:
+      - "json_equal({{node2_caps_view_node1}}, 0)"
+
+  # --- Verify the same on node-2's own view (consistency across nodes) ---
+
+  - name: Get node-2 capabilities as seen by node-2 itself
+    type: get_member_capabilities
+    node: e2e-node-2
+    group_id: '{{namespace_id}}'
+    member_id: '{{member_identity_node2}}'
+    outputs:
+      node2_caps_view_node2: capabilities
+
+  - name: Assert node-2 self-view is consistent (caps=0)
+    type: json_assert
+    statements:
+      - "json_equal({{node2_caps_view_node2}}, 0)"
+
+stop_all_nodes: false

apps/e2e-kv-store/workflows/group-subgroup-visibility-inheritance.yml

@@ -0,0 +1,163 @@
+name: E2E KV Store - Subgroup Visibility Inheritance
+description: >
+  End-to-end coverage for issue #2256 (subgroup visibility refactor).
+
+  Verifies that a member of a parent namespace, who holds the default
+  CAN_JOIN_OPEN_SUBGROUPS capability, is automatically inherited as a
+  member of any `Open` subgroup beneath that namespace -- including its
+  contexts -- without an explicit `add_group_members` call. This is the
+  payoff of moving visibility from the (unused) context level to the
+  subgroup level.
+
+  Flow:
+    1. node-1 installs the kv-store app and creates a namespace.
+    2. node-2 joins the namespace via invitation. As a non-admin member
+       it picks up the default capabilities, which include
+       CAN_JOIN_OPEN_SUBGROUPS.
+    3. node-1 creates a subgroup inside the namespace and flips its
+       visibility to `open`.
+    4. node-1 creates a context inside the Open subgroup. node-2 is NOT
+       added to the subgroup directly.
+    5. node-2 joins the context. With the parent-walk in
+       `check_group_membership`, this should succeed: node-2's
+       namespace membership + capability bit + Open child subgroup ⇒
+       inherited membership.
+    6. node-1 writes, node-2 reads, and the value matches -- proving
+       node-2 is a fully functional context member, not just a passive
+       subscriber.
+
+force_pull_image: false
+near_devnet: false
+
+nodes:
+  count: 2
+  image: ghcr.io/calimero-network/merod:edge
+  prefix: e2e-node
+
+steps:
+  # --- Setup ---
+
+  - name: Install application on node-1
+    type: install_application
+    node: e2e-node-1
+    path: res/e2e_kv_store.wasm
+    dev: true
+    outputs:
+      app_id: applicationId
+
+  - name: Assert application installed
+    type: assert
+    statements:
+      - "is_set({{app_id}})"
+
+  - name: Create namespace on node-1
+    type: create_namespace
+    node: e2e-node-1
+    application_id: '{{app_id}}'
+    outputs:
+      namespace_id: namespaceId
+
+  - name: Assert namespace created
+    type: assert
+    statements:
+      - "is_set({{namespace_id}})"
+
+  # --- node-2 joins the namespace (gets default caps incl.
+  #     CAN_JOIN_OPEN_SUBGROUPS) ---
+
+  - name: Create namespace invitation for node-2
+    type: create_namespace_invitation
+    node: e2e-node-1
+    namespace_id: '{{namespace_id}}'
+    outputs:
+      namespace_invitation: invitation
+
+  - name: Node-2 joins namespace
+    type: join_namespace
+    node: e2e-node-2
+    namespace_id: '{{namespace_id}}'
+    invitation: '{{namespace_invitation}}'
+    outputs:
+      member_identity_node2: memberIdentity
+
+  # --- Create an Open subgroup with a context inside it ---
+
+  - name: Create subgroup inside namespace on node-1
+    type: create_group_in_namespace
+    node: e2e-node-1
+    namespace_id: '{{namespace_id}}'
+    group_alias: open-subgroup
+    outputs:
+      subgroup_id: groupId
+
+  - name: Mark subgroup as Open (inherits parent members)
+    type: set_subgroup_visibility
+    node: e2e-node-1
+    group_id: '{{subgroup_id}}'
+    visibility: open
+
+  - name: Create context inside the Open subgroup
+    type: create_context
+    node: e2e-node-1
+    application_id: '{{app_id}}'
+    group_id: '{{subgroup_id}}'
+    outputs:
+      sub_ctx_id: contextId
+
+  - name: Assert subgroup context created
+    type: assert
+    statements:
+      - "is_set({{sub_ctx_id}})"
+
+  # --- The payoff: node-2 joins the context WITHOUT being added to the
+  #     subgroup. The parent-walk in check_group_membership should let
+  #     them in via inheritance. ---
+
+  - name: Node-2 joins subgroup context via inheritance
+    type: join_context
+    node: e2e-node-2
+    context_id: '{{sub_ctx_id}}'
+    outputs:
+      ctx_key_node2: memberPublicKey
+
+  - name: Assert node-2 holds a context identity
+    type: assert
+    statements:
+      - "is_set({{ctx_key_node2}})"
+
+  # --- Confirm node-2 is a real member, not just a passive subscriber ---
+
+  - name: Node-1 writes to the inherited context
+    type: call
+    node: e2e-node-1
+    context_id: '{{sub_ctx_id}}'
+    method: set
+    args:
+      key: "inherited_key"
+      value: "from_node1"
+
+  - name: Wait for context sync between nodes
+    type: wait_for_sync
+    context_id: '{{sub_ctx_id}}'
+    nodes:
+      - e2e-node-1
+      - e2e-node-2
+    timeout: 60
+    trigger_sync: true
+
+  - name: Node-2 reads the value via the inherited membership
+    type: call
+    node: e2e-node-2
+    context_id: '{{sub_ctx_id}}'
+    method: get
+    args:
+      key: "inherited_key"
+    outputs:
+      inherited_read: result
+
+  - name: Assert inheritance grants real read access
+    type: json_assert
+    statements:
+      - 'json_equal({{inherited_read}}, {"output": "from_node1"})'
+
+stop_all_nodes: false

architecture/local-governance.html

@@ -143,18 +143,27 @@ <h3>Subgroups</h3>
 <div class="g2">
 <div>
 <h4>Membership Inheritance</h4>
-<p>Membership checks walk up the ancestor chain (max depth 16). A member of the parent group is automatically a member of all descendant groups. Direct membership in the child takes priority over inherited membership.</p>
+<p>Each subgroup carries a <span class="code">subgroup_visibility</span> flag &mdash; <strong>Open</strong> or <strong>Restricted</strong> (absent = Restricted, the safe default). The membership check (<span class="code">check_group_membership_path</span>) walks up the ancestor chain (max depth 16) only through unbroken <strong>Open</strong> chains, anchored at the deepest ancestor where the identity holds a direct membership row. Direct membership in the target subgroup short-circuits to <span class="code">Direct</span>; otherwise the walk returns <span class="code">Inherited{anchor, via_admin}</span>. A <strong>Restricted</strong> ancestor is a wall &mdash; the walk stops with <span class="code">None</span>.</p>
+<p style="margin-top:8px;">At the anchor, admins inherit <strong>unconditionally</strong>; non-admins need the <span class="code">CAN_JOIN_OPEN_SUBGROUPS</span> capability bit at that anchor.</p>
 </div>
 <div>
 <h4>Admin Authority</h4>
 <p>Parent admins inherit <strong>structural</strong> governance over descendant groups: add/remove members, detach contexts, delete subgroups, set target application.</p>
 </div>
 </div>
 
+<h4 style="margin-top:16px;">Open vs Restricted Subgroups</h4>
+<p><strong>Restricted</strong> subgroups are sealed: only direct members can read their governance ops or context state. Their per-subgroup encryption key is delivered via <span class="code">KeyDelivery</span> to direct joiners only.</p>
+<p style="margin-top:8px;"><strong>Open</strong> subgroups align the cryptographic boundary with the access boundary: their governance ops <em>and</em> context state deltas are encrypted with the <strong>parent namespace's key</strong> rather than a per-subgroup key. Every namespace member already holds the namespace key, so no separate key-delivery path is needed for inheritance-eligible members. The trade-off is explicit: the read-confidentiality boundary for an Open subgroup is namespace-wide, while the join/write boundary remains capability-gated by <span class="code">CAN_JOIN_OPEN_SUBGROUPS</span>.</p>
+<p style="margin-top:8px;">Receivers don't need a wire flag for the encryption choice &mdash; the <span class="code">key_id</span> resolves uniquely to either the subgroup or the namespace keyring, with the namespace keyring tried as a fallback after the subgroup keyring miss.</p>
+
+<h4 style="margin-top:16px;">Sync-Stream Authorization</h4>
+<p>The responder-side stream-auth gate (<span class="code">DagHeadsRequest</span>, <span class="code">DeltaRequest</span>, snapshot stream) accepts both direct context membership <em>and</em> inheritance-eligible parent membership. Without this, an inheritance joiner's snapshot probe would be silently closed, leaving them stuck on the all-zero initial root hash. The auth gate uses the same parent-walk that powers join-time authorization, so the contract is consistent end-to-end: <em>namespace member + <span class="code">CAN_JOIN_OPEN_SUBGROUPS</span> = read + write + sync, with no extra round-trips</em>.</p>
+
 <h4 style="margin-top:16px;">Restricting Access via Subgroups</h4>
-<p>To restrict access to specific contexts, create a <strong>subgroup</strong> and register those contexts under it. Only members of that subgroup (direct or inherited from its parent) can join its contexts. This replaces the former per-context visibility/allowlist model with a simpler group-membership-based approach.</p>
+<p>To restrict access to specific contexts, create a <strong>subgroup</strong> and either leave its visibility unset (Restricted by default) or register those contexts under it. Only members of that subgroup (direct or inherited via the Open-chain walk above) can join its contexts.</p>
 
-<p style="margin-top:12px;">Create a subgroup with <span class="code">--parent-group-id</span> on <span class="code">meroctl group create</span> or via the <span class="code">parentGroupId</span> field in the API. The <span class="code">SubgroupCreated</span> governance op is published on the parent group's gossip topic.</p>
+<p style="margin-top:12px;">Create a subgroup with <span class="code">--parent-group-id</span> on <span class="code">meroctl group create</span> or via the <span class="code">parentGroupId</span> field in the API. The <span class="code">SubgroupCreated</span> governance op is published on the parent group's gossip topic. Flip visibility with <span class="code">PUT /groups/:id/settings/subgroup-visibility</span> or <span class="code">meroctl group set-subgroup-visibility</span>.</p>
 <p style="margin-top:8px;">When joining a parent group with <span class="code">auto_join</span> (default: true), the node subscribes to gossip topics and contexts in all descendant subgroups. <span class="code">MemberRemoved</span> cascades to descendant subgroup contexts (skipping child groups where the member has direct membership).</p>
 </div>