V1.5.4 release #857
Draft
V1.5.4 release #857
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o be updated as needed.
* Harden images and reorg * fix timezone with package (#680) * Joyride tutorials update (#679) * Merged in latest v1.2.1 * Updates to the joyride tutorials and some minor bugfixes. * Revert Markdown update to fix current jest testing error * Updated React Markdown library and fixed the jest test errors. * Some modifications to remove the build and linting errors that came up after the merge with latest branch. * Load frontend config before rendering App Stop tests from leaking calls to globus.org * Some fixes to get the react front-end to build and get the postgres to start up. There are still errors with django session store. * Fixed some linting errors so frontend should build correctly. * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: downiec <[email protected]> * Staticbuild v1.3 fixes (#697) * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config (#707) * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: downiec <[email protected]> * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config Auto generate configuration documentation with pre-commit hooks * Revert yarn.lock, fix more tests * Includes some updates and minor fixes so that everything builds and runs correctly. Removed the GLOBUS_REDIRECT setting and did some cleanup of the manage_metagrid.sh functions. * Updated the test files to no longer use the act() functions, as they are deprecated. Cleaned up and fixed some failing tests. * Fix esgsearch json tests * Minor test fixes to see if tests will pass on github (they pass locally). --------- Co-authored-by: downiec <[email protected]> Co-authored-by: Zach Price <[email protected]> --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: bstrdsmkr <[email protected]> --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: Sasha Ames <[email protected]> Co-authored-by: bstrdsmkr <[email protected]>
…repeated settings-doc-markdown config, to see if that resolves the pre-commit failure.
* Fix broken link in Globus transfer panel * Globus link display update (#3) * Added a warning case for when a transfer request returns no failures or successes in the backend response. No successes indicates no transfer occurred although there wasn't any specified errors provided. Added new test for the added warning case. * Fix pre-commit prerequisites --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: Carlos Downie <[email protected]>
* Update version number, add version changelog (to be updated with the changes that will be included in this update) * Updated documentation references to config variables (#706) * Updated documentation, specifically references to the configuration parameters which have changed over time, to use the latest variable names currently used in configuration. Updated requirements to use the latest readthedocs them of Sphinx * Update version number, add version changelog (to be updated with the changes that will be included in this update) * fixed the pagination so that it will reset to page one when doing a new search (#705) * update for obs4MIPs (#711) Co-authored-by: Sasha Ames <[email protected]> * Updated the settings file to autoformat on save to hopefully reduce linting errors. Fixed linting errors for the initial_projects_data update. Fixed the status tooltip to correctly display the node's status. Updated test fixtures to accurately reflect the node status response (the node names included the https:// in front and thats why status tooltip wasn't finding them. (#714) * Added alert for the case that a collection search fails in the manage collections popup (#710) * Added alert for the case that a collection search fails in the collection management popup. Added a test to bring the coverage up to 100% for the search table component. Removed some unused code, the get endpoint function is not used. * Minor linting error fixes. * Fixed some backend tests based on the removed/unused functions * Small fix to test. * Minor update to update summary. * Made it so the globus ready column does not appear if no globus ready nodes are configured (as it was before). Changed ActiveFacets type to be more strict in order to avoid the issue of calling join on a string which was causing errors. Fixed and updated some tests related to the url to string and string to url functions in the utils file. * Fixed failing test. Updated the manage_metagrid.sh helper scripts to include commands for pre-commits, backend and frontend tests. Updated setup.cfg to allow backend tests to run correctly. Removed unneeded commands in backend README.md. Removed the autoformat setting on save. --------- Co-authored-by: Sasha Ames <[email protected]>
* Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. (#4) --------- Co-authored-by: Zach Price <[email protected]> Co-authored-by: downiec <[email protected]>
* Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. * Added a new configuration option for setting the footer of the metagrid frontend so that links and text can be displayed as desired. Updated the manage_metagrid script to include more developer actions, including run migrations, and installing packages for local development. Modified styling to better fit the current footer. Will need to add option for adjusting the footer height based on the height of the footer text. Updated tests and documentation for the custom footer options. * Final updates for the v1.3.0 release. Includes updates to the joyride tutorial to add the theme switch. Updated the manage_metagrid script and created a new configHelper script that sets the proper initial configuration settings for first time deployment or developers. Added tests for the theme buttons in the app. --------- Co-authored-by: Zach Price <[email protected]>
Co-authored-by: Zach Price <[email protected]>
* Include helm chart and healthchecks * Modified the traefik.yml config to separate between local and production, in order to attempt to get production traefik deployment to function as it did in another branch. Updated docker compose files to separate keycloak into a profile and remove docs service from production. Updated manage script to utilize new compose files for deployment and testing. * Updated some docs and the manage scripts * Merged in some minor changes from traefik-deployment-fixes branch to update app css and include other minor fixes. * Delete docker-compose.prod-overlay.yml * Pre-comfig fixes, update to scripts. * Some updates to docker deployment to test if they will work in production * Separated trafik.yml into one for production and one for local, to avoid using the conditional statements that were being used which seemed to not be working for traefik deployment. Updated the deployment script to make sure local and production custm overlays are applied last so their variables aren't overriden. * Removed unnecessary prod_traefik file. * Updated prod overlay to name the containers with prod to distinguish deployment container types. Updated the config helper script to generate production overlay * Updated changelog. Added dark theme styling to the React Joyride tutorials. * Implemented some suggestions from PR comments. Brightened the tutorial background in dark theme. * Minor fix to compose file. * Reverted container names to leave them be. * Updated obs4mips url and modified the display button. Remove obsolete updateProjects.sh script and updated the README's regarding updating project data. Added the update functions to the manage_metagrid.sh scripts instead. Still need to test in production, but should work. * Fix minor lint error. Added build step to project migration function. --------- Co-authored-by: Zach Price <[email protected]>
…as label, to Replicas Only.
…ges are updated and how the changelog is updated. The data for messages and changes is now located in a messageData.json file, removing the need to modify the messageDisplayData.ts file in multiple locations every time a new version and changelog entry is created.
…request to add option for obtaining object using request.body, this fixes the wget download which expects data in a different format than other requests. Updated the changelog.
…ay, to make version history collapsed as it was getting long.
… to increment/create a new version. It will create a new changelog file, update the package.json and update the messageData.json file to add the new version.
…a save steps to see if that will resolve globus transfer token auth loop issue in Firefox.
…a save steps to see if that will resolve globus transfer token auth loop issue in Firefox.
…globus permissions were requested. This seems to allow the path set redirect to request the appropriate permissions for allowing transfers to occur.
…ges, fix typos in ingress template (#743) Co-authored-by: Zach Price <[email protected]>
…rsion script so helm chart version is automatically updated when the version number is updated.
* Fixes hash for automatic deployment rolls * Fixes container trigger and pushing images * Fixes workflow tags * Fixes default filename for configHelper.sh * Fixes frontend makefile * Adds minimal frontend-config.js * Revert "Adds minimal frontend-config.js" This reverts commit 8856c9d. * Adds helm chart testing job * Fixes working directory * Updates workflow and adds test values * Fixes building dependency * Adds readme and helmfile to test locally * Fixes NOTES rendering * Adds defaults to support no-config install * Removes test.yaml since defaults were fixed * Fixes spelling error * Fixes workflow * Updates readme
* Some minor updates to config helper and manage_metagrid scripts. * Added option to cancel endpoint path selection from globus side and it will return user to metagrid properly. * Updated to the manage script for updating the version values in helm files. Fixes some linting errors. * Added a tooltip to the download button to hint to user why it's disabled. Updated download button behavior ro make sure user can't start a Globus download without selecting a collection or start a wget download without selecting a cart item.
* Created a DataBundlePersister class which will bundle key value pairs into a single object, stringify and then save to session storage as a single bundle. This will reduce requests to session storage by consolidating into a single request when values need to be persisted or loaded. * Minor updates to changelogs for consistent punctuation. * Some minor updates to persistent class * Updated the way session storage is used by the datasetdownload component to improve globus transfer reliability. Includes extensive refactoring of tests and test components to pass using the new DataBundlePersister class. Updated the cart summary to display info on selected cart items not just on the entire cart totals. * Some minor changes to the documentation to update the frontend project structure. Moved the reset tokens button to a menu in the globus transfer button. Globus transfer and wget download are separated into their own buttons using download type to select which is displayed. Updated tests as needed and added new ones for the reset token feature. Updated react joyride tour to mention changes on the summary table. * Made some minor updates, refactors and added new tests to increase test coverage
* Updated some packages to latest version. Updated the changelog for v1.5.0. * Updated axios and removed unused lib/axios (it has typescript support now). Updated react-markdown and react-hotjar, while addressing breaking changes. Updated a few other packages which didn't seem to need any further fixes. * Updated the js-pkce package and made necessary updates to the DatasetDownloads functions. * Updated antd library, migrated react-router-dom 7.5 to react-router 7.6, uuid to 11, eslint configs, and prettier. * Fixed linter errors and failing tests. * Updated some linting rules and packages. Fixed minor linting errors. * Some more updates to package.json for eslint and linting fixes. * Add speicifc timeout for test which is failing due to timeout error. * Update timeout for test which is failing due to timeout error. * Minor update to changlog.
… to create globus transfer tasks.
#812) * Updated the Search Table to have a results limit of 10000, to avoid errors when attempting to view a page containing results beyond 10000. Added a handler that will reset cached page to 1 if there is a 422 error (the error caused by requesting a page beyond supported limit). * Increased coverage, added test for new 422 error handling. Updated coverage in other places.
* Updated the Search Table to have a results limit of 10000, to avoid errors when attempting to view a page containing results beyond 10000. Added a handler that will reset cached page to 1 if there is a 422 error (the error caused by requesting a page beyond supported limit). * Increased coverage, added test for new 422 error handling. Updated coverage in other places. * Modified the createSearchRouteURL function so that the configured search setting is used for the JSON link, rather than the proxy path. * Made trivial change to the URL used as an example for the search url setting.
* Change heading to Site Administrators, clarify purpose in docs, DRCDP data_node (#816) * Update mkdocs.yml * clarify purpose of docs * DRCDP corrections add data_node correct facet group name * Hotfix/globus transfer (#819) * Hot fix for current globus transfer issues. Reverted to using get rather than post for the search_files helper function used when submitting transfers. Added some additional logic to help improve debugging. * Updated tests and added two more to cover the new test cases and logic that was added. Updated changelog * Rewrote entire backend globus login flow to eliminate the need for a nativeclient on the frontend, instead utilizing a confidential client on the backend. Simplifies the download process while improving reliability. Removed the need to persistently store the globus related tokens on the frontend. Need to do some cleanup to remove obsolete code. * Removed commented out code and deleted experimental functions that are no longer needed. Basic cleanup. * Removed obsolete GLOBUS_CLIENT_ID settings related code, as we only need the client key and secret now. Removed broken tests from backend, will create new ones later to bring coverage back to 100%. Removed more unused code. * Added some try/except statements on the globus api to provide more information if an error occurs. Fixed a minor bug that was causing the auth url not to be set when it was needed. * Due to the added testing complexity for the backend globus functions, reduced the 100% coverage requirements to 95% (will set it back to 100% when tests can be fully covered). Brought the test coverage for the backend up to 98%, will need to cleanup and write more tests to get the last 2% of coverage. Fixed all broken tests and removed obsolete tests for the frontend globus related features. Removed fixtures and code that are no longer needed in the frontend, now that globus transfer is mostly handled by the backend. * Updated the changelog with current changes in this branch. Switched from access tokens to refresh tokens and updated tests accordingly. Added function to reset stored tokens on the backend so that consents can be refreshed should the need ever arrive or for testing purposes. Added tests to cover new api function for token reset. * Minor update to make sure the url is reset when errors occur in globus transfer attempts. --------- Co-authored-by: Sasha Ames <[email protected]>
* Change heading to Site Administrators, clarify purpose in docs, DRCDP data_node (#816) * Update mkdocs.yml * clarify purpose of docs * DRCDP corrections add data_node correct facet group name * Hotfix/globus transfer (#819) * Hot fix for current globus transfer issues. Reverted to using get rather than post for the search_files helper function used when submitting transfers. Added some additional logic to help improve debugging. * Updated tests and added two more to cover the new test cases and logic that was added. Updated changelog * support multiple domains for citation API * Update test_views.py --------- Co-authored-by: Carlos Downie <[email protected]>
* Modified the search page copy and save search buttons to be a menu instead, where the copy search feature will be in a drop-down. Then added 2 new copy features related to esgpull, one for search and one for download. * Added Intake ESGF search option and created a function to generate the python code to run the search. The command assumes you have intake installed and ready to use. * cleaned up some code and fixed broken tests. * Added tootltips and updated the joyride tutorials to show the search copy menu items. Updated the changelog with notes about the update. Added several tests to increase coverage and verify the copy search options give the appropriate output. * Updates to esgpull search and download feature so that unnecessary facets are removed from valid list and missing ones were added. Added custom comments on esgpull commands to highlight any facets that were not applied. Other minor updates. * Adding an esgpull download options for a single dataset using the master_id attribute. * Fix some tests and increase coverage.
* Update initial_projects_data.py to change E3SM * change E3SM logic
remove extraneous and clarify
* Initial testing commit, merging in most significant changes from the previous STAC integration branches. * linting fixes. * Merged in the latest changes from v1.5.1 branch, fixed some broken code cuased by the STAC changes. Updated tests so that they pass. * Updated the tables to include file sizes and checksums for STAC results if the data is available. Removed the numResults requirement as it seems to be missing sometimes, instead counting the results features array directly. Updated the filename to utilize assets key when there is no id, title or name attribute in the asset object. * Added objects to map differences between CMIP6 facets and facets url. Created some functions to generate stac filters for and, or and equals operations. Updated search to utilize generated filters. Fixed minor issue with metadata values that weren't being displayed. Updated the tabs to utilize cmip6:further_info_url data for es-docs. * Fixed minor issue with filtering so that OR filters work properly. Updated search limit to 9999 rather than only 10. Still need to make page size work properly. Updated the query string to display the generated STAC filter string if project is a STAC project. * Removed the 'All except CMIP6' project as it seems to be broken now. Updated SearchesCards to handle the STAC Project searches. Modified STAC request url to include the project name and made it so future STAC projects can be added and will work. Adde some error checking for STAC search results so that status is returned after a search. * Modified the search page copy and save search buttons to be a menu instead, where the copy search feature will be in a drop-down. Then added 2 new copy features related to esgpull, one for search and one for download. * Added Intake ESGF search option and created a function to generate the python code to run the search. The command assumes you have intake installed and ready to use. * cleaned up some code and fixed broken tests. * Added tootltips and updated the joyride tutorials to show the search copy menu items. Updated the changelog with notes about the update. Added several tests to increase coverage and verify the copy search options give the appropriate output. * Implemented aggregations for STAC so that result counts are shown for specific facets. Added STAC icon and fixed table to display the STAC icon instead of dataset icon, for STAC datasets. Other changes. * Small fixes after v1.5.3-release merge. * Updates to esgpull search and download feature so that unnecessary facets are removed from valid list and missing ones were added. Added custom comments on esgpull commands to highlight any facets that were not applied. Other minor updates. * Adding an esgpull download options for a single dataset using the master_id attribute. * Updated postgres version to 16-alpine to remove the critical vulnerability in postgres-16. Minor fixes. * Fix some tests and increase coverage. * Removed the default filter for STAC * Removed the All (except CMIP6) project option since it appears to no longer work. Added logic to hide STAC projects if STAC URL is not configured. * Updated authentication options in the script and settings so that 'none' is now a deployment option. When none is set, there will not be a sign in button in the navigation bar. Updated keycloak deployment so that a basic keycloak instance is created which can be connected to for configuring it as an admin. The keycloak authentication doesn't work, so there are still issues with how the authentication is handled in the frontend. Updated some tests and added keycloak overlays to handle keycloak deployment while skipping it when none or globus options are selected. * Updated the cart, and saved searches to filter out STAC items when STAC is not active. Also updated the navbar to show the correct filtered count. Updated the search cards to display the STAC filter rather than the regular filter, when showing the query string. * Added several tests on various components and removed the facet route in order to improve overall test coverage. * Fixed issues with pagination and page size.
… transfer if the deployment has authentication method set to none. Minor updates to linter config settings to remove deprecated settings.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (87.33%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## master #857 +/- ##
==========================================
- Coverage 97.75% 95.08% -2.67%
==========================================
Files 46 50 +4
Lines 1823 2888 +1065
Branches 415 634 +219
==========================================
+ Hits 1782 2746 +964
- Misses 41 142 +101
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
* Updates backend database settings to be configurable * Allows helm testing to fail * Removes pydantic database settings, replaces with environ.db_url to parse DATABASE_URL * Updates chart to use new DATABASE_URL for backend and adds examples * Updates docker-compose to use DATABASE_URL * Moves helm chart testing to helm-chart workflow, adds workflow trigger * Updates backend workflow to use DATABASE_URL * Removes job requirements * Fixes workflow trigger * Removes pull_request trigger * Merges containers and helm-chart workflows * Removes bitnami repo * Fixes running migrate job on upgrade * Updates default search/wget urls * Adds job to test upgrade * Fixes liveness/readiness probe indentation, fixes duplication port labels
…xes for the helm deployment
* Fix some issues with precommit and startup * Fixes pre-commit workflow to run on all pull requests --------- Co-authored-by: Jason Boutte <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes # (issue)
Type of change
How Has This Been Tested?
Checklist