Export zip file with batch files and metadata - #8
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request introduces significant enhancements to the export workflow by adding configurable output settings, batch control, and comprehensive metadata tracking. The changes enable users to control file types, batch sizes, and whether to merge output files, while packaging all exports with detailed metadata into a single zip file.
Changes:
- Introduced
OutputSettingsmodel to centralize output file configuration (file type, batch size, merge option) - Refactored export process to create zip archives containing batch files and metadata.json
- Renamed all
consolidate_output_filesreferences tomerge_output_filesfor clarity - Enhanced metadata tracking with entry counts and search timestamps
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| src/nomad_actions/actions/entries/models.py | Added OutputSettings and ExportDatasetMetadata models; refactored ExportEntriesUserInput to use OutputSettings; renamed ConsolidateOutputFilesInput to MergeOutputFilesInput |
| src/nomad_actions/actions/entries/activities.py | Renamed consolidate_output_files to merge_output_files; refactored export_dataset_to_upload to create zip files with metadata; added timestamp tracking to search activity |
| src/nomad_actions/actions/entries/workflows.py | Updated workflow to use new OutputSettings model; added metadata collection during search loops; made merging conditional based on user settings |
| src/nomad_actions/actions/entries/utils.py | Renamed consolidate_files to merge_files with updated documentation |
| src/nomad_actions/actions/entries/init.py | Updated imports to use merge_output_files instead of consolidate_output_files |
| tests/actions/test_action.py | Updated imports to use merge_output_files instead of consolidate_output_files |
Comments suppressed due to low confidence (1)
tests/actions/test_action.py:15
- The test file is missing the import for
OutputSettingswhich is now required to constructExportEntriesUserInputproperly. AddOutputSettingsto the imports fromnomad_actions.actions.entries.models.
from nomad_actions.actions.entries.models import (
ExportEntriesUserInput,
SearchSettings,
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Pull Request Test Coverage Report for Build 21039089009Details
💛 - Coveralls |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main pain point resolved: Merging all the batch files into one big file can run into a high memory footprint (potentially crashing the pods). Here's the consolidation has been made optional, and the dataset files are exported as a zip. Additionally, the zip now contains a JSON file containing metadata of the export workflow.
Output settings and batch control
OutputSettingsmodel to allow users to specify output file type, batch size, and whether to merge output files. Integrated these settings into the workflow and models. [1] [2] [3]Enhanced metadata and export process
metadata.jsonfile inside the exported zip archive. [1] [2] [3]Output file merging and terminology updates
consolidate_output_filestomerge_output_filesthroughout the codebase, including activity, model, and utility function names, and updated related descriptions for clarity. [1] [2] [3] [4] [5] [6] [7] [8] [9]merge_output_filesactivity and model, and to conditionally merge output files based on user settings. [1] [2]Search activity improvements
These changes collectively make the export workflow more robust, configurable, and informative for users.