Draft
Conversation
| @@ -205,11 +205,11 @@ def test_psql(adapter_psql_one_partition: SQLAdapter) -> None: | |||
| ], | |||
Contributor
There was a problem hiding this comment.
Could instead shrink all the very similar SQLAdapter fixtures into one fixture that fetches the right database scheme and number of partitions?
async def _postgresql_uri() -> AsyncGenerator[str, None]:
uri = os.getenv("TILED_TEST_POSTGRESQL_URI")
if uri is None:
pytest.skip("TILED_TEST_POSTGRESQL_URI is not set")
async with temp_postgres(uri) as uri_with_database_name:
yield uri_with_database_name
async def _database_uri(scheme: str, tmp_path: Path) -> Generator[str, None, None]:
if schema == "postgresql":
url = await postgresql_uri()
yield url
else:
yield f"{scheme}:///{tmp_path}/test.db"
@asyncio.fixture
async def adapter(tmp_path: Path, request: pytest.FixtureRequest) -> Generator[SQLAdapter, None, None]:
database_params: tuple[str, int] = request.params
uri = await database_uri(database_params[0], tmp_path)
data_source = data_source_from_init_storage(uri, database_params[1])
yield SQLAdapter(
data_source.assets[0].data_uri,
data_source.structure,
data_source.parameters["table_name"],
data_source.parameters["dataset_id"],
)
@pytest.mark.parametrize(
"adapter",
[
("sqlite", 1),
("duckdb", 1),
("postgresql", 1),
],
indirect=True
)
def test_write_read_one_batch_one_part(adapter: SQLAdapter) -> None:And adapter_psql_many_partitions becomes ("postgresql", 3) etc.?
Contributor
There was a problem hiding this comment.
(I just don't like the pattern of fetching the fixture inside the test, this way the skip is done before the first line of the test)
| # By giving that a name ("_") we enable requests to asking for the | ||
| # last N by requesting the sorting ("_", -1). | ||
| sorting = [SortingItem(key="_", direction=1)] | ||
| sorting = [SortingItem(key="_", direction=SortingDirection.ASCENDING)] |
Contributor
There was a problem hiding this comment.
Does this actually work? Or as discussed, is the type hint wrong and the method works with what it was previously receiving?
* Add missing asserts * Update xdi test * Update regular expression slightly * Add length check to ensure content but not order * Use \s in re * Add type ignore to expected warnings * Add assert for composite test * Add workaround for tests * revert test to main * Update custom format test * Update check * Add change log * Revert CHANGELOG changes * fix lint issue * Add re explanation
These were (per the comments) intended to be used for testing but do not appear to be used anywhere.
* add zarr route * ENH: basic zarr functionality * ENH: map tiled chunks to zarr blocks * MNT: Clean-up comments * ENH: support tables * add zarr route * ENH: basic zarr functionality * ENH: map tiled chunks to zarr blocks * MNT: Clean-up comments * ENH: support tables * ENH: Add data type to sparse * ENH: support units for numpy datetime types * MNT: removed unnecessary imports * ENH: add default value for units * ENH: update BuiltinDtype in pydantic * ENH: update BuiltinDtype in pydantic * ENH: add default value for units * TST: datetime dtypes in test_array * MNT: Update changelog * resolve conflict FIX: Recursion error when pickling with dill TST: Initial tests for zarr endpoints Clean, refactor, and lint TST: tests for arrays and tables TST: tests for arrays and tables ENH: restructure demo examples ENH: (partial) support for StructDtype TST: fix tests ENH: support for datetime types * Clean-up * MNT: gitignore alembic.ini * FIX: typo in comment * ENH: Add data type to sparse * FIX: assignment error * MNT: clean and lint * MNT: update changelog * MNT: fix changelog * FIX: tests with COOStructure * FIX: typing * BLD: add aiohttp package to server requirements * MNT: clean and lint * FIX: default value of units to empty string. * FIX: use None as the sentinel for the units kwarg * ENH: use np.datetime_data to extract units * TST: Fix failing authorization test -- empty password * MNT: format and lint * MNT: remove deprecated PatchedStreamingResponse * MNT: lint * TST: add authentication tests * FIX: ensure support for py3.8 * MNT: lint * MNT: add changelog entry * MNT: moved aiohttp from required to dev dependencies * ENH: add indx_data_type for sparse arrays * MNT: rename indx_data_type to coord_data_type * MNT: clean up * MNT: lint * TST: refactor ThreadedServer class for tests * TST: test no writing in read-only mode * MNT: resolve conflicts * MNT: lint * FIX: use default_factory for BuiltinDtype * ENH: incorporate changes from David * TST: fix existing tests after rebase * TST: update tests * FIX: do not specify media_type in zar group response * TST: xfail dtype tests * TST: bring back tests with complex and string types * ENH: add zarr v3 endpoints * FIX: variable names and default codec * ENH: support struct dtypes natively * ENH: remove zarr middleware * FIX: zattrs and codecs for zarr v2 * DOC: add tutorial on zarr * TST: use blosc lz4 codec for v2 and v3 * DOC: fix errors in building docs * TST: replace default zarr metadata when exporting to HDF5 * Update docs/source/tutorials/zarr-integration.md Co-authored-by: Dan Allan <daniel.b.allan@gmail.com> * DOC: update docs * MNT: update readme to match the new demo tree * Satisfy linter --------- Co-authored-by: Dan Allan <dallan@bnl.gov> Co-authored-by: Dan Allan <daniel.b.allan@gmail.com>
* starting locust load test * locust is working * clean up kubernetes config files for testing locust * add locust dev dep, and update locust readme * add a better locust simulator, trying to get the api key working * fix typo in docs * add test_locust.py because I am getting auth errors * update header * Update the ReadingUser, need to figure out how to get the WritingUser working * Add draft of writing user * found out that tiled-dev doesn't support writing * only load test reading in this PR * locust reading test are working well now * add locust tasks for the remaining GET endpoints * locust remove redendant debug logs * undo pyarrow version updates because it is being addressed in another pr * locust touchup * touch up locust README * default to localhost if --host is not given * locust reader touch up * add a task for each search type * get the search tasks working * add --container-name arg, and create the container if it doesn't exist * touch up
The 'args' field was over indented making it part of the description of the authenticator property. This would have prevented configurations from being validated correctly if not for the item/items typo that meant none of the authentication configuration was being checked.
* FIX: move constratint * FIX: migrating empty tables * MNT: add logs for migration * MNT: changelog
* Add pixi.toml * Update CHANGELOG * add deps to get the tests passing * Fix sorting and loosen pins --------- Co-authored-by: gbischof <bisc8233@gmail.com>
* ENH: sqlalchemy counting statement * ENH: implement length approximation mechanism * MNT: changelog * TST: fix sqlite tests * ENH: use exact length for len() on the client * MNT: bring back codecov * ENH: set the exact count limit in server configs * TST: add tests for approximate length
* Apply Hynek recommendations to Dockerfile * Avoid heredoc syntax, for podman compat * No custom entrypoint * No need for dev or client deps * Fix syntax * more syntax errors * more syntax...debugging via CI * More fixes after removing heredoc * Handle ARG scope correctly * No uv.lock is committed * Maybe this only works if there is a lockfile * whitespace * python is a build dep * whitespace * Use podman-compatibile feature set. * Name it Containerfile * Include python...-dev for building * Specify 'Containerfile' * Label stage * App needs to write to storage * Adjust containerfile to work as devcontainer * Use python base image to obtain desired python versions * Revert to py3.9 for devcontainer * Fix casing * No need to install Python * Adjust uv invocation to match catch in WORKDIR * Configure dev environment in devcontainer --------- Co-authored-by: Joseph Ware <53935796+DiamondJoseph@users.noreply.github.com>
* Add types to Generic sequence implementations * Fix type hints for python 3.9 and add changelog entry * Correct type hints * Add typing to instances of Maps * Use typehints on CachedMaps * Tighten types * Restore type var for rebasing * Update and prevent problems * Correct type hints * Correct changelog version
* Update to use a startup script for keycloak-oidc * Update health check for keycloak * Increase sleep time for keycloak * Update the startup time for compose yaml * Update client secret to secret * Add comments * Add more documentation * Add docs for logout * Add proper name for keycloak
* add external policy decision point * update access_policy test * Remove dls specific policy * Add docs for policy
* If server allows anonymous access, allow anon access to WS too. * fix: Test logic was not quite right * Extend timeouts (tests have been flaky) * tst: Test Subscription without API key
* tst: Register with a prefix * fix: Apply default key_for_filename * fix: Delete _contents_ of registration target, not target itself * chore: update CHANGELOG
When building a catalog adapter, a subprocess is used to initialise the
database, if this process fails, the error (written to stderr) was lost
making it difficult to determine the cause.
By wrapping the stderr in a new exception and setting it to be the cause
of the raised CalledProcessException, Python displays it as
<subprocess stderr content>
The above exception was the direct cause of the following exception:
<the stacktrace of the command being run as before>
and the cause is more clearly visible.
* ENH: force reshaping in HDF5 * TST: test hdf5 reshaping * ENH: generalize to all adapters * ENH: force rashape tiff, jpeg, and npy * MNT: Update Changelog
* sparipsa-fix: typos * sparipsa-fix: drop obsolete compose version key
* ENH: allow to configure global custom routers * MNT: changelog * ENH: add routers to the config schema * TST: test custom endpoints * ENH: rename /custom prefix * FIX: type conversion --------- Co-authored-by: Dan Allan <dallan@bnl.gov>
* fix a little bug with StreamingCache no defined * add missing dep to pixi.toml * increase the websocket max size * expost the max_size param * update docs * touch up * add more max_size params to subclasses * improve receiver handling of messages that are too big * touch ups * touch ups * move max_size param to start * disconnect when message too big * update max_size docs
* Added TTLCacheDatastore with potential PubSub * Added itertools.count to prevent race conditions * Added client property to TTLCache * Fixed publish key to "notify:" * Combine make_ws_handler for redis and ttlcache * Added weakref to PubSub and implemented get() * Apply docstrings from copilot code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Removed else from while statement in streaming.py and called super.init for TTLCache Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Not using strong ref self in the cleanup callback * Fixed seqence_counters bug * Added defaults to seq_ttl and data_ttl in TTLCacheDatastore * Added async lock to TTLCacheDatastore * Changed cache_settings to cache_config * Added ttlcache as default streaming datastore for --temp flags * Added a try except block around adding a message to queue * Added scalable check for ttlcache * Rename 'ttlcache' to 'memory' * Added in-memory (ttlcache) tests similar to redis tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Allan <dallan@bnl.gov>
* move tests from tiled/_tests to tests/ and update imports Co-Authored-By: sketch <hello@sketch.dev> Change-ID: se4bea6793b64567dk * fix remaining tiled._tests import references Co-Authored-By: sketch <hello@sketch.dev> Change-ID: sb4856f274ea6c525k * update coverage config for new test location Co-Authored-By: sketch <hello@sketch.dev> Change-ID: s52d270803d5ca1d3k * Update relative path to example_configs * touch up * Run formatters --------- Co-authored-by: sketch <hello@sketch.dev> Co-authored-by: gbischof <bisc8233@gmail.com>
* Add custom auth for tiled client * Add minor changes
* chore: Drop Python 3.9 (EOL) * Update CHANGELOG * MNT: remove py3.9-related import * MNT: fix imports in connection_pool * MNT: remove py39 dependencies from pixi.toml * MNT: remove importlib_resources from dev dependencies * Bump devcontainer version --------- Co-authored-by: Eugene M. <ymatviych@bnl.gov>
…1285) * Require setting explicit scopes to use access-tag restricted apikeys * Update apikey creation in access control tests * Update changelog
* And ALL_ACCESS queries * directly return the entry without filtering * make all_access `[]` * add ALL_ACCESS to `TagBasedAccessPolicy`
* FIX: regression in downlod * MNT: changelog * Please enter the commit message for your changes. Lines starting * FIX: pydantic version * Revert "FIX: pydantic version" This reverts commit 5dc54c5.
* FIX: refcator the delete method * FIX: refactor SQL * MNT: init alembic migration scripts * TST: fix tests * ENH: migration scripts * ENH: remove rebuilding of foreign key constraints for sqlite * FIX: Skip changing the constraint for SQLite * MNT: fmt and lint * MNT: update changelog * TST: revert comment * TST: revert comment * TST: revert comment * MNT: remove breakpoint * FIX: drop index in alembic migration * FIX: condition to delete storage assets * FIX: remove unnecessary index
* MNT: Update readme * ENH: add properties column to the DataSources table * MNT: Update changelog * DOC: update documentation * TST: test DataSource.properties
* Extend StrEnum sphinx workaround * Move dev deps from extras into dependency-groups * Add a Justfile for convenient docs building with uv * Switch to pydata-sphinx-theme and start Getting Started * Rebrand 'how to' as 'user guide' * Redesign landing page. * Start '10 minutes...' tutorial with myst_nb * Add docs deps * Docs build cleanly. * Include docs dependency group in CI * Fix fenceposts * Enable some use of toctree. * Remove broken references. * First pass at 'What is Tiled' * Refine 'What is Tiled' * Keep a more consistent scientist perspective. * Copy edits Co-authored-by: Eugene <ymatviych@bnl.gov> * Add missing words Co-authored-by: Eugene <ymatviych@bnl.gov> * Update docs/source/getting-started/what-is-tiled.md Co-authored-by: Eugene <ymatviych@bnl.gov> * More polishing of 'What is tiled' * Refine claim about generality * Hit point about generality stronger * More rework of 'What is Tiled' * Move footnote * Tighten landing page description * Better tab title * Remove footnote - too much noise * Integrate feedback * Add subheadings for scanability * Remove logo outline * Add scripts for populating tiled-demo * Progress filling out 10 minutes to Tiled * More 10 minutes... * Add from_catalog, wrapping SimpleTiledServer * Basic writing examples run in tutorial * Ignore more execution output * Mention web UI * Add in-memory streaming cache to SimpleTiledServer * Small improvements * Allow multiple servers. No strong reason to prohibit this. * Draft Stream and Register sections * Copyedits * Update CHANGELOG * Drop test associated with removed constraint. * Rename from_catalog to simple, and add a test. * Wording improvements Co-authored-by: Eugene <ymatviych@bnl.gov> * Address review comments * Add a section 'Slice remotely' * Avoid ambiguity with row filtering. --------- Co-authored-by: Eugene <ymatviych@bnl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist