Skip to content

Conversation

@stephprince
Copy link
Contributor

Fix #154. Some updates to the parquet structure to help with querying and some general refactoring to make it easier to generate the database files locally without using the app.

Creating the new submodule makes it harder to see the changes so here is a summary:

  • pulled out dataclasses and parquet processing from app, moved to different module to run locally more easily outside of the app
  • added version filtering before concatenation to avoid issues with older results structures
  • added "machine_id" column to machine.parquet file
  • add new columns for parameter_case fields ("parameter_case_name", "parameter_case_https_url", "parameter_case_object_name", "parameter_case_slice_range").

Should environment parquet file also have a version column in case we add additional fields at any point?

@stephprince stephprince marked this pull request as ready for review September 18, 2025 17:56
@CodyCBakerPhD
Copy link
Collaborator

@stephprince Feel free to change whatever you need - my first attempt was just to get something roughly in the ballpark of some kind of usable shape

@CodyCBakerPhD CodyCBakerPhD merged commit 56bf848 into main Sep 18, 2025
3 checks passed
@CodyCBakerPhD CodyCBakerPhD deleted the add-database-module branch September 18, 2025 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update parameter case storage in data base

3 participants