v2.4.5

Updating UI version to 1.3.7

v2.4.4

Features

Add configurable separation for read and write pools for the database connection

A separate reader pool of connections can now be configured to point to a different database instance, to split the query load when necessary. This can be configured by the following environment variables

• USE_SEPARATE_READER_POOL set to "True"
• MF_METADATA_DB_READ_REPLICA_HOST to the host for the readonly connections.

Configurable SSL support for database connection

The connection to the database can now be configured to utilize SSL if required.
Setup is done with the following environment variables

• MF_METADATA_DB_SSL_MODE, possible values 'allow', 'prefer', 'require', 'verify-ca', 'verify-full'
• MF_METADATA_DB_SSL_CERT_PATH Path to cert file
• MF_METADATA_DB_SSL_KEY_PATH Path to key file
• MF_METADATA_DB_SSL_ROOT_CERT_PATH Path to CA

Improvements

UI v1.3.6

Updates the bundled UI version to 1.3.6

Update Python base images

Update base images versions to 3.11.6 for added security.

Relax thresholds for marking runs as failed.

In a previous release, in an effort to reduce the weight of database queries, the threshold for marking runs as failed due to an expired heartbeat was set to a very strict 1 minute. While configurable, this meant that by default any run that gets all of its tasks stuck in a scheduler would be marked as failed very quickly.

This release eases the threshold to a more reasonable 6 minutes, while still being fully configurable via the environment variable RUN_INACTIVE_CUTOFF_TIME

What's Changed

• fix: Correct project version in setup.py file by @tfurmston in #377
• Add INSTALLED_PLUGINS_BASE_DIR env variable by @obgibson in #378
• [ui] config root configurable by env vars by @valayDave in #381
• Docker changes to support M1 laptops by @obgibson in #380
• Use f-strings for log message by @rohanrebello in #382
• support PATH_PREFIX var in metadata service by @oavdeev in #384
• Upgrade Github actions used in dockerimage action by @tfurmston in #379
• chore: deprecated pkg_resources by @saikonen in #387
• In a previous commit, the detection of a failure became too aggressive. by @romain-intel in #386
• fix PATH_PREFIX handling in metadata service so it doesn't interfere with mfgui routes by @oavdeev in #388
• Configurable SSL Connection by @RikishK in #373
• fix: run_goose.py not working without SSL mode env variables. by @saikonen in #390
• fix: tone down run inactive cutoff default by @saikonen in #392
• Added the ability to separate out reads and writes into their own connection pools. by @pjoshi30 in #344
• chore: upgrade Python base images to 3.11.6 by @saikonen in #396
• chore: update UI version to 1.3.6 by @saikonen in #397
• Bump version to 2.4.4 by @saikonen in #398

New Contributors

• @tfurmston made their first contribution in #377
• @rohanrebello made their first contribution in #382
• @RikishK made their first contribution in #373
• @pjoshi30 made their first contribution in #344

Full Changelog: v2.4.3...v2.4.4

v2.4.3

What's Changed

• Fix/avoid expensive ops by @romain-intel in #338
• Add Google Cloud dependencies to UI backend service (#375) by @jbvaningen in #376

New Contributors

• @jbvaningen made their first contribution in #376

Full Changelog: v2.4.2...v2.4.3

v2.4.2

Improvements

Upgrade Python version to 3.11

After Python 3.7 reaching end of life, all of the services have now been moved over to Python 3.11.

The official Dockerfile has also seen some security upgrades by moving to a smaller base image.

Update bundled Metaflow-UI version to 1.3.3

Metaflow-UI has been updated to 1.3.3. See the Metaflow-UI Release Notes for details.

What's Changed

• chore: update python version by @saikonen in #367
• Release/2.4.2 by @saikonen in #372

Full Changelog: v2.4.1...v2.4.2

v2.4.1

Improvements

Fix issue with Metaflow-UI backend and MinIO

The Metaflow client dependency in service v2.4.0 had a bug that affects accessing resources on MinIO. As a result, the Metaflow-UI in v2.4.0 is unable to display logs or artifacts that are hosted with MinIO. This release fixes the underlying issue by depending on a later Metaflow client version.

Fixing deprecations in code base

Fixing various deprecations in the code base in preparation for a Python version upgrade.
Most noteworthy changes being

• Moving away from using LooseVersion
• Changing the way metadata-service and migration-service aiohttp processes are initialized.

What's Changed

• chore: fix deprecations by @saikonen in #297
• fix: add missing packaging dependency by @saikonen in #369
• chore: fix deprecated way of initialising aiohttp app in services by @saikonen in #298
• fix: S3 access for MinIO resources by @saikonen in #370
• Bump version to 2.4.1 by @saikonen in #371

Full Changelog: v2.4.0...v2.4.1

v2.4.0

Improvements

Improve query performance for Metaflow UI with full indexes

There was an issue with the partial indexes not being used by the Metaflow UI backend queries, resulting in poor performance on platforms with a majority of the resources having string format ID's (e.g. argo-testflow-abc132, sfn-testflow-5abc4, t-123abc).

This release includes a database migration to change the existing partial indexes for resources with string format ID's with full ones.

Considerations

The creation of the indexes in the migration has been set to happen concurrently, so as not to impact regular operations.
Depending on the size of the database this can take a substantial amount of time though, during which higher than average loads can occur as the index is built.

What's Changed

• Drop all partial indexes and replace them with full indexes by @sarthak in #341
• Bump version to 2.4.0 by @saikonen in #365

New Contributors

• @sarthak made their first contribution in #341

Full Changelog: v2.3.11...v2.4.0

v2.3.11

What's Changed

• UI v1.3.2 - Fixes realtime updates not working on the timeline for runs with string id's

Full Changelog: v2.3.10...v2.3.11

v2.3.10

• Update to v1.3.1 of metaflow-ui

Full Changelog: v2.3.9...v2.3.10

v2.3.9

What's Changed

• fix: GitHub test workflow stop containers issues by @saikonen in #359
• v1.2.8 of metaflow-ui

Full Changelog: v2.3.8...v2.3.9

v2.3.8

What's Changed

• Added release docs by @obgibson in #296
• Change cutoff times from 6 days to 1 day by @obgibson in #348
• feature: add CodeQL by @saikonen in #356
• feature: pin workflow steps by @saikonen in #357

Full Changelog: v2.3.7...v2.3.8