Experiment Abstraction through config settings#1
Open
HashirA123 wants to merge 3 commits intocharmlab:mainfrom
Open
Experiment Abstraction through config settings#1HashirA123 wants to merge 3 commits intocharmlab:mainfrom
HashirA123 wants to merge 3 commits intocharmlab:mainfrom
Conversation
This better reflects their roles as objects better. Is also easier on the tongue.
Does so without needing the user to touch the code, this repo still needs alot of error handling incase bad configs are given.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request focuses on modularizing the configuration system, standardizing naming conventions, and improving extensibility for experiments. The changes enable experiment-level configuration overrides, introduce factory patterns for method and evaluation instantiation, and rename core classes for clarity.
Configuration and Experiment Orchestration
config_utils.pywith utilities for loading YAML configs and recursively merging experiment-level overrides into layer configs.experiments/experiment_roar_mlp_config.yml) that controls all aspects of an experiment, including data, model, method, and evaluation settings and their overrides.experiment.pyto orchestrate the full pipeline: loading configs, merging overrides, instantiating data/model/method/evaluation objects, selecting factuals, running counterfactual generation, and aggregating evaluation results.Class and File Renaming for Clarity
DataModuletoDataObjectandEvaluationModuletoEvaluationObjectthroughout the codebase, updating all imports and usages for consistency.ModelObjectand related references.Extensibility via Factory Patterns
evaluation_factory.pywith a registry and factory functions for instantiating evaluation metrics from config, supporting easy addition of new metrics via decorators.Layer Configuration and Overrides
DataObject,ModelObject, etc.) to accept merged config dictionaries, enabling experiment-level overrides without directly editing layer config files.Miscellaneous Improvements