Skip to content

Fixes to migration from cals by block. #405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 46 commits into from
Mar 3, 2025
Merged

Conversation

cmccully
Copy link
Collaborator

This breaks the alembic migration into two parts so we shouldn't break the database even if back populating data takes a while. I've also restructured how the archive API calls are made so we don't get rate limited.

In testing I ran into significant issues with old versions of deps. I updated the deps to more modern versions.

@cmccully cmccully requested review from sfoale and jchate6 February 12, 2025 19:03
Copy link
Collaborator

@jchate6 jchate6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot of failing style checks and tests etc. Should we try to fix those?
I can help with the style things a least.

@cmccully
Copy link
Collaborator Author

While this should have been relatively simple, dependencies added a lot of complexity here. Changes over time to our devops + dependenency ancillary data for pytorch was making things untestable. As such, I've had to make sweeping changes to the build and dependency management for banzai. Our previous docker images were >14GB each. This PR gets us down to <2GB.

Conda has not kept up with modern enough versions of all our dependencies for that to be viable any longer. I've migrated to poetry for dependency management and packaging in lieu of pip. This was not without issues as poetry does not handle building non-pure python projects well. It also doesn't play well with tox and the adapter code between them is quite out of date so I've retired tox and simply rely on github actions. To test things locally without tox, I've been using a miniconda environment with only the version of python I want to test installed, pip install poetry and then let poetry manage the rest.

As part of this process, we have updated to a more modern python (3.12 is our default now). This and some of the dependencies (e.g. sqlalchemy and tenacity) needed api updates to make them work.

Copy link
Contributor

@sfoale sfoale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@cmccully cmccully merged commit 24afdf5 into main Mar 3, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants