Move shareable authz utilities into Tiled

CHANGELOG.md

@@ -6,9 +6,27 @@ Write the date in place of the "Unreleased" in the case a new version is release
 
 ## v0.1.0-b37 (Unreleased)
 
+### Added
+
+- The access tags compiler and db schema have been upstreamed into Tiled
+- API keys can now be restricted to specific access tags
+- New unit tests covering the new access policy and access control features
+
 ### Changed
 
 - Remove `SpecialUsers` principals for single-user and anonymous-access cases
+- Access control code is now in the `access_control` subdirectory
+- `SimpleAccessPolicy` has been removed
+- AuthN database can now be in-memory SQLite
+- Catalog database can now be shared when using in-memory SQLite
+- `TagBasedAccessPolicy` now supports anonymous access
+- `AccessTagsParser` is now async
+- `toy_authentication` example config now uses `TagBasedAccessPolicy`
+- Added helpers for setting up the access tag and catalog databases for `toy_authentication`
+
+### Fixed
+
+- Access control on container export was partially broken, now access works as expected.
 
 
 ## v0.1.0-b36 (2025-08-26)
@@ -29,7 +47,6 @@ Write the date in place of the "Unreleased" in the case a new version is release
 
 - The project ships with a pixi manifest (`pixi.toml`).
 
-
 ## v0.1.0-b34 (2025-08-14)
 
 ### Fixed
@@ -41,7 +58,6 @@ Write the date in place of the "Unreleased" in the case a new version is release
   should be re-run on any databases that could not be upgraded with the previous
   release.
 
-
 ## v0.1.0-b33 (2025-08-13)
 
 _This release requires a database migration of the catalog database._

docs/source/explanations/access-control.md

@@ -35,95 +35,43 @@ Rephrasing these two items now using the jargon of entities in Tiled:
    children (if any).
 
 This determination can be backed by a call to an external service or by a
-static configuration file. We demonstrate both here.
+static configuration file. We demonstrate the static file case here.
 
-First, the static configuration file. Consider this simple tree of data:
+Consider this simple tree of data:
 
-```{eval-rst}
-.. literalinclude:: ../../../tiled/examples/toy_authentication.py
-   :caption: tiled/examples/toy_authentication.py
+```
+/
+├── A  -> array=(10 * numpy.ones((10, 10))), access_tags=["data_A"]
+├── B  -> array=(10 * numpy.ones((10, 10))), access_tags=["data_B"]
+├── C  -> array=(10 * numpy.ones((10, 10))), access_tags=["data_C"]
+└── D  -> array=(10 * numpy.ones((10, 10))), access_tags=["data_D"]
 ```
 
-protected by this simple Access Control Policy:
+which will be protected by Tiled's "Tag Based" Access Control Policy. Note the
+"access tags" associated with each node. The tag-based access policy uses ACLs,
+which are compiled from provided access-tag definitions (example below), to make
+decisions based on these access tags.
 
 ```{eval-rst}
-.. literalinclude:: ../../../example_configs/toy_authentication.yml
-   :caption: example_configs/toy_authentication.yml
+.. literalinclude:: ../../../example_configs/access_tags/tag_definitions.yml
+   :caption: example_configs/access_tags/tag_definitions.yml
 ```
 
-Under `access_lists:` usernames are mapped to the keys of the entries the user may access.
-The section `public:` designates entries that an
-unauthenticated (anonymous) user may access *if* the server is configured to
-allow anonymous access. (See {doc}`security`.) The special value
-``tiled.adapters.mapping:SimpleAccessPolicy.ALL`` designates that the user may access any entry
-in the Tree.
+Under `tags`, usernames and groupnames are mapped to either a role or list of scopes.
+Roles are pre-defined lists of scopes, and are also defined in this file. This mapping
+confers these scopes to these users for data which is tagged with the corresponding tagname.
 
-```
-ALICE_PASSWORD=secret1 BOB_PASSWORD=secret2 CARA_PASSWORD=secret3 tiled serve config example_configs/config.yml
-```
+Tags can also inherit the ACLs of other tags, using the `auto_tags` field. There is also a
+`public` tag which is a special tag used to mark data as public (all users can read).
+
+Lastly, only "owners" of a tag can apply that tag to a node. Tag owners are defined in
+this same tag definitions file, under the `tag_owners` key.
 
-For large-scale deployment, Tiled typically integrates with an existing access management
-system. This is sketch of the Access Control Policy used by NSLS-II to
-integrate with our proposal system.
-
-```py
-import cachetools
-import httpx
-from tiled.queries import In
-from tiled.scopes import PUBLIC_SCOPES
-
-
-# To reduce load on the external service and to expedite repeated lookups, use a
-# process-global cache with a timeout.
-response_cache = cachetools.TTLCache(maxsize=10_000, ttl=60)
-
-
-class PASSAccessPolicy:
-    """
-    access_control:
-      access_policy: pass_access_policy:PASSAccessPolicy
-      args:
-        url: ...
-        beamline: ...
-    """
-
-    def __init__(self, url, beamline, provider):
-        self._client = httpx.Client(base_url=url)
-        self._beamline = beamline
-        self.provider = provider
-
-    def _get_id(self, principal):
-        for identity in principal.identities:
-            if identity.provider == self.provider:
-                return identity.id
-        else:
-            raise ValueError(
-                f"Principcal {principal} has no identity from provider {self.provider}. "
-                f"Its identities are: {principal.identities}"
-            )
-
-    def allowed_scopes(self, node, principal, authn_scopes):
-        return PUBLIC_SCOPES
-
-    def filters(self, node, principal, authn_scopes, scopes):
-        queries = []
-        id = self._get_id(principal)
-        if not scopes.issubset(PUBLIC_SCOPES):
-            return NO_ACCESS
-        try:
-            response = response_cache[id]
-        except KeyError:
-            response = self._client.get(f"/data_session/{id}")
-            response_cache[id] = response
-        if response.is_error:
-            response.raise_for_status()
-        data = response.json()
-        if ("nsls2" in (data["facility_all_access"] or [])) or (
-            self._beamline in (data["beamline_all_access"] or [])
-        ):
-            return queries
-        queries.append(
-            In("data_session", data["data_sessions"] or [])
-        )
-        return queries
+To try out this access control configuration, an example server can be prepped and launched:
+```
+# prep the access tags and catalog databases
+python example_configs/access_tags/compile_tags.py
+python example_configs/catalog/create_catalog.py
+# launch the example server, which loads these databases
+ALICE_PASSWORD=secret1 BOB_PASSWORD=secret2 CARA_PASSWORD=secret3 tiled serve config example_configs/toy_authentication.yml
 ```

docs/source/how-to/api-keys.md

@@ -49,6 +49,14 @@ in the example below with that address.
 ALICE_PASSWORD=secret1 tiled serve config example_configs/toy_authentication.yml
 ```
 
+Note that you will need to run these helper tools to prep the backing databases that Tiled needs,
+before you can use the example config shown above:
+```
+# prep the access tags and catalog databases
+python example_configs/access_tags/compile_tags.py
+python example_configs/catalog/create_catalog.py
+```
+
 Using the Tiled commandline interface, log in as `alice` using the password `secret1`.
 
 ```

docs/source/reference/authentication.md

@@ -28,6 +28,14 @@ is included with the tiled source code, and start a server like so.
    :caption: example_configs/toy_authentication.py
 ```
 
+Note that you will need to run these helper tools to prep the backing databases that Tiled needs:
+```
+# prep the access tags and catalog databases
+python example_configs/access_tags/compile_tags.py
+python example_configs/catalog/create_catalog.py
+```
+
+then, you can launch the server:
 ```
 ALICE_PASSWORD=secret1 BOB_PASSWORD=secret2 CARA_PASSWORD=secret3 tiled serve config example_configs/toy_authentication.yml
 ```

example_configs/access_tags/compile_tags.py

@@ -0,0 +1,30 @@
+from pathlib import Path
+
+from tiled.access_control.access_tags import AccessTagsCompiler
+from tiled.access_control.scopes import ALL_SCOPES
+
+
+def group_parser(groupname):
+    return {
+        "group_A": ["alice", "bob"],
+        "admins": ["cara"],
+    }[groupname]
+
+
+def main():
+    file_directory = Path(__file__).resolve().parent
+
+    access_tags_compiler = AccessTagsCompiler(
+        ALL_SCOPES,
+        Path(file_directory, "tag_definitions.yml"),
+        {"uri": f"file:{file_directory}/compiled_tags.sqlite"},
+        group_parser,
+    )
+
+    access_tags_compiler.load_tag_config()
+    access_tags_compiler.compile()
+    access_tags_compiler.connection.close()
+
+
+if __name__ == "__main__":
+    main()

example_configs/access_tags/tag_definitions.yml

@@ -0,0 +1,36 @@
+roles:
+  facility_user:
+    scopes: ["read:data", "read:metadata"]
+  facility_admin:
+    scopes: ["read:data", "read:metadata", "write:data", "write:metadata", "create", "register"]
+tags:
+  data_A:
+    groups:
+      - name: group_A
+        role: facility_user
+    auto_tags:
+      - name: data_admin
+  data_B:
+    users:
+      - name: alice
+        scopes: ["read:data", "read:metadata"]
+    auto_tags:
+      - name: data_admin
+  data_C:
+    users:
+      - name: bob
+        role: facility_user
+    auto_tags:
+      - name: data_admin
+  data_D:
+    auto_tags:
+      - name: data_admin
+      - name: public
+  data_admin:
+    users:
+      - name: cara
+        role: facility_admin
+tag_owners:
+  data_admin:
+    users:
+      - name: cara

example_configs/catalog/create_catalog.py

@@ -0,0 +1,34 @@
+from pathlib import Path
+
+import numpy
+import yaml
+
+from tiled._tests.utils import enter_username_password
+from tiled.client import Context, from_context
+from tiled.server.app import build_app_from_config
+
+CONFIG_NAME = "toy_authentication.yml"
+CATALOG_STORAGE = "data/"
+
+
+def main():
+    file_directory = Path(__file__).resolve().parent
+    config_directory = file_directory.parent
+    Path(file_directory, CATALOG_STORAGE).mkdir()
+
+    with open(Path(config_directory, CONFIG_NAME)) as config_file:
+        config = yaml.load(config_file, Loader=yaml.BaseLoader)
+    app = build_app_from_config(config)
+    context = Context.from_app(app)
+    with enter_username_password("admin", "admin"):
+        client = from_context(context, remember_me=False)
+    for n in ["A", "B", "C", "D"]:
+        client.write_array(
+            key=n, array=10 * numpy.ones((10, 10)), access_tags=[f"data_{n}"]
+        )
+    client.logout()
+    context.close()
+
+
+if __name__ == "__main__":
+    main()

example_configs/toy_authentication.yml

@@ -7,27 +7,29 @@ authentication:
         alice: ${ALICE_PASSWORD}
         bob: ${BOB_PASSWORD}
         cara: ${CARA_PASSWORD}
+        admin: "admin"
       confirmation_message: "You have logged in as {id}."
   tiled_admins:
     - provider: toy
-      id: alice
+      id: admin
 access_control:
-  access_policy: tiled.access_policies:SimpleAccessPolicy
+  access_policy: "tiled.access_control.access_policies:TagBasedAccessPolicy"
   args:
-    provider: toy  # matches provider above
-    access_lists:
-      alice:
-      - A
-      - B
-      bob:
-      - A
-      - C
-      cara: tiled.access_policies:SimpleAccessPolicy.ALL
+    provider: "toy"
     scopes:
     - "read:metadata"
     - "read:data"
-    public:
-    - D
+    - "write:metadata"
+    - "write:data"
+    - "create"
+    tags_db:
+      uri: "file:example_configs/access_tags/compiled_tags.sqlite"
+    access_tags_parser: "tiled.access_control.access_tags:AccessTagsParser"
 trees:
   - path: /
-    tree: tiled.examples.toy_authentication:tree
+    tree: catalog
+    args:
+      uri: "sqlite+aiosqlite:///./example_configs/catalog/catalog.db"
+      writable_storage: "./example_configs/catalog/data"
+      init_if_not_exists: true
+      top_level_access_blob: {"tags": ["public"]}

tiled/_tests/conftest.py

@@ -66,20 +66,18 @@ def buffer():
 
 
 @pytest.fixture(scope="function")
-def buffer_factory(request):
+def buffer_factory():
     buffers = []
 
     def _buffer():
         buf = io.BytesIO()
         buffers.append(buf)
         return buf
 
-    def teardown():
-        for buf in buffers:
-            buf.close()
+    yield _buffer
 
-    request.addfinalizer(teardown)
-    return _buffer
+    for buf in buffers:
+        buf.close()
 
 
 @pytest.fixture