Merge branch 'main' into closure-tables

Author: genematx

.github/workflows/ci.yml

@@ -88,6 +88,14 @@ jobs:
         # Opt in to LDAPAuthenticator tests.
         TILED_TEST_LDAP: 1
 
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v4
+      with:
+        name: ${{ inputs.python-version }}/${{ inputs.runs-on }}
+        files: cov.xml
+      env:
+        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
   windows_checks:
     name: Test on Windows
     runs-on: windows-latest

.gitignore

@@ -16,6 +16,15 @@ docs/source/reference/generated/
 # pytest
 .pytest_cache/
 
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.cache
+nosetests.xml
+coverage.xml
+cover/*
+
 # PyBuilder
 target/
 

CHANGELOG.md

@@ -2,8 +2,115 @@
 Write the date in place of the "Unreleased" in the case a new version is released. -->
 # Changelog
 
+## 0.1.0-b27 (2025-05-08)
 
-## Unreleased
+_This release requires a database migration of the catalog database._
+
+```none
+tiled catalog upgrade-database [postgresql://.. | sqlite:///...]
+```
+
+### Added
+
+- New access policy `TagBasedAccessPolicy` which introduces more robust
+  authorization based on the concept of tagging. When this policy is used,
+  access to data is controlled by the node's `access_blob` (i.e the tags applied
+  to that node).
+- Added new `access_blob` column to catalog database, in support of the new
+  authorization. This blob typically contains one of: resource owner (creator),
+  or a list of access tags.
+- Added new filter type `AccessBlobFilter` which filters nodes based upon their
+  `access_blob` contents. In support of the new authorization.
+
+### Changed
+- Tiled now accepts a single `access_control` configuraton for the entire
+  server, only. Access policies are now a server-wide singleton used for
+  all access requests. Access control can no longer be specified on
+  individual trees.
+- Removed `path_parts` arg from access policy signatures and related.
+- Effective scopes for the principal (from authN) are now threaded into
+  access policies and related.
+- Removed `access_policy` from `MapAdapter` and `CatalogAdapter`; accesss policies
+  are now set server-wide only.
+
+## 0.1.0-b26 (2025-05-07)
+
+### Added
+
+- New query parameter `drop_revision` on endpoints `PUT /metadata/{path}`
+  and `PATCH /metadata/{path}`. If set to true, the version replaced by
+  the update is _not_ saved as a revision. This is exposed in the Python
+  client via a new keyword-only argument `drop_revision` in
+  `update_metadata`, `patch_metadata`, and `replace_metadata`.
+
+### Fixed
+
+- A critical bug in the `mount_node` feature introduced in the
+  previous release prohibited the server from starting when
+  `mount_node` was used with a PostgreSQL database.
+
+## 0.1.0-b25 (2025-05-06)
+
+### Added
+
+- New optional parameter to catalog configuration, `mount_node`
+  enables mounting different sub-trees of one catalog database
+  at different prefixes. This is an advanced feature to facilitate
+  migration from many catalogs to one. See
+  `tiled/_tests/test_mount_node.py` for usage.
+
+## 0.1.0-b24 (2025-05-06)
+
+### Added
+
+- Support for reading numpy's on-disk format, `.npy` files.
+
+### Changed
+
+- In server configuration, `writable_storage` now takes a list of URIs,
+  given in order of decreasing priority.
+- Adapters should implement a `supported_storage` attribute, as specified
+  in `tiled.adapters.protocols.BaseAdapter`. This is optional, for
+  backward-compatiblity with existing Adapters, which are assumed to
+  use file-based storage.
+
+### Fixed
+
+- When using SQL-backed storage and file-backed storage, Tiled treated SQLite
+  or DuckDB files as if they were directories of readable files, and
+  included them superfluously in a check on whether assets were situated
+  in a readable area.
+- Update data_sources in the client after receiving a response from the server.
+  Removed the (unused) `data_source` parameter from the `PUT /data_source/`
+  endpoint; the id of the updated data source must be included in the structure
+  within the body of the request.
+
+## 0.1.0-b23 (2025-04-24)
+
+### Added
+
+- New query type `Like` enables partial string match using SQL `LIKE`
+  condition.
+
+### Changed
+
+- Exposed `Session.state` information from database to enhance custom access
+  control developments.
+
+## 0.1.0-b22 (2025-04-21)
+
+### Added
+
+- Tiled now retries HTTP requests that fail due to server-side (`5XX`) or
+  connection-level problems.
+- Support for `async` streaming serializers (exporters)
+
+### Changed
+
+- Iteration over a `Composite` client yields its (flattened) keys, not its
+  internal parts. This makes `__iter__` and `__getitem__` consistent.
+
+## 0.1.0-b21 (2025-04-15)
 
 ### Added
 
@@ -15,15 +122,17 @@ Write the date in place of the "Unreleased" in the case a new version is release
 
 - Adjust arguments of `print_admin_api_key_if_generated` and rename `print_server_info`
 - Allow `SQLAdapter.append_partition` to accept `pyarrow.Table` as its argument
+- Fix streaming serialization of tables keeping the dtypes of individual columns
 
 ### Maintenance
 
 - Extract API key handling
 - Extract scope fetching and checking
 - Refactor router construction
 - Adjust environment loading
-  - This is a breaking change if setting TILED_SERVER_SECRET_KEYS or TILED_ALLOW_ORIGINS,
-    TILED_SERVER_SECRET_KEYS is now TILED_SECRET_KEYS and these fields now require passing a json
+  - This is a breaking change if setting `TILED_SERVER_SECRET_KEYS` or
+    `TILED_ALLOW_ORIGINS`. `TILED_SERVER_SECRET_KEYS` is now
+    `TILED_SECRET_KEYS` and these fields now require passing a json
     list e.g. ``TILED_SECRET_KEYS='["one", "two"]'``
 - More type hinting
 - Refactor authentication router construction

docker-compose.yml

@@ -1,7 +1,7 @@
 version: "3.2"
 services:
   tiled:
-    image: ghcr.io/bluesky/tiled:0.1.0-b20
+    image: ghcr.io/bluesky/tiled:0.1.0-b27
     environment:
       - TILED_SINGLE_USER_API_KEY=${TILED_SINGLE_USER_API_KEY}
     ports:

docs/source/explanations/access-control.md

@@ -102,10 +102,10 @@ class PASSAccessPolicy:
                 f"Its identities are: {principal.identities}"
             )
 
-    def allowed_scopes(self, node, principal, path_parts):
+    def allowed_scopes(self, node, principal, authn_scopes):
         return PUBLIC_SCOPES
 
-    def filters(self, node, principal, scopes, path_parts):
+    def filters(self, node, principal, authn_scopes, scopes):
         queries = []
         id = self._get_id(principal)
         if not scopes.issubset(PUBLIC_SCOPES):

docs/source/how-to/retries.md

@@ -0,0 +1,37 @@
+# Control Retries
+
+Tiled uses the library [stamina](https://stamina.hynek.me/en/stable/) to
+implement HTTP retries. Retries are essential for making distributed systems
+resilient. See the
+[Motivation](https://stamina.hynek.me/en/stable/motivation.html) section of the
+stamina documentation for more.
+
+Tiled retries requests that fail to garner a response (e.g. due to connection
+problems) or receive a response indicating a server-side problem (HTTP
+status code `5XX`). By default, it retries for 10 attempts or 45 seconds,
+whichever it reaches first. These defaults can be tuned by setting
+environment variables:
+
+```
+TILED_RETRY_ATTEMPTS  # max number of attempts
+TILED_RETRY_TIMEOUT  # max total seconds
+```
+
+See following examples to control retry behavior in the context of development
+or testing.
+
+```python
+import stamina
+
+# Disable retries globally.
+stamina.set_active(False)
+
+# Check whether retries are active.
+stamina.is_active()
+
+# Disable delay between attempts
+stamina.set_testing(True, attempts=1)
+
+# Check whether test mode is enabled.
+stamina.is_testing()
+```

docs/source/index.md

@@ -33,6 +33,7 @@ how-to/metrics
 how-to/direct-client
 how-to/tiled-authn-database
 how-to/register
+how-to/retries
 ```
 
 ```{toctree}

docs/source/reference/queries.md

@@ -20,6 +20,7 @@ Follow the links in the table below for examples specific to each query.
    tiled.queries.NotEq
    tiled.queries.FullText
    tiled.queries.In
+   tiled.queries.Like
    tiled.queries.NotIn
    tiled.queries.Regex
    tiled.queries.SpecQuery

pyproject.toml

@@ -95,6 +95,7 @@ all = [
     "rich",
     "sparse",
     "sqlalchemy[asyncio] >=2",
+    "stamina",
     "starlette >=0.38.0",
     "tifffile",
     "uvicorn[standard]",
@@ -122,6 +123,7 @@ client = [
     "pyarrow",
     "rich",
     "sparse",
+    "stamina",
     "watchfiles",
     "xarray",
     "zstandard",
@@ -175,6 +177,7 @@ formats = [
 minimal-client = [
     "entrypoints",
     "rich",
+    "stamina",
     "watchfiles",
 ]
 # These are the requirements needed for basic server functionality.
@@ -304,3 +307,12 @@ exclude = '''
   | web-frontend
 )
 '''
+
+[tool.coverage.run]
+data_file = "/tmp/tiled.coverage"
+
+[tool.coverage.report]
+exclude_lines = ["if __name__ == '__main__':"]
+
+[tool.coverage.paths]
+source = ["."]

tiled/_tests/adapters/test_arrow.py

@@ -4,8 +4,9 @@
 import pytest
 
 from tiled.adapters.arrow import ArrowAdapter
+from tiled.storage import FileStorage
 from tiled.structures.core import StructureFamily
-from tiled.structures.data_source import DataSource, Management, Storage
+from tiled.structures.data_source import DataSource, Management
 from tiled.structures.table import TableStructure
 
 names = ["f0", "f1", "f2"]
@@ -38,7 +39,7 @@ def data_source_from_init_storage() -> DataSource[TableStructure]:
         structure=structure,
         assets=[],
     )
-    storage = Storage(filesystem=data_uri, sql=None)
+    storage = FileStorage(data_uri)
     return ArrowAdapter.init_storage(
         data_source=data_source, storage=storage, path_parts=[]
     )