Automatically determine/override config settings when loading from checkpoint

[jstjohn]

Currently checkpoint loading flow requires options to be specified in a list in sub-packages/bionemo-llm/src/bionemo/llm/model/biobert/model.py, specifically OVERRIDE_BIONEMO_CONFIG_DEFAULTS. We should come up with a CLI that "knows" what a user wants to change and applies those changes over the config in some easy to explain precedence procedure.

[jstjohn]

@skothenhill-nv let's include this as something we demonstrate we can improve on in our future CLI. Something where the runner is "smart" about which settings a user wants to override. Some kind of sane precedence rules. Maybe a case for something like a config with user-defined CLI overrides.

[trvachov]

This issue was filed against the bionemo-llm sub-package's checkpoint config override mechanism (OVERRIDE_BIONEMO_CONFIG_DEFAULTS / OVERRIDE_BIOBERT_CONFIG_DEFAULTS in bionemo.llm.model.config and bionemo.llm.model.biobert.model).

The bionemo-llm sub-package is now deprecated and being removed (PR #1491). The replacement bionemo-recipes architecture uses Hydra/OmegaConf for configuration, which natively provides exactly what this issue requested — users can override any config setting via CLI args with clear precedence rules (config.yaml → CLI overrides), without needing hardcoded override lists.

PR #276 added the necessary parallelism overrides as a workaround for the remaining lifetime of the legacy code. Closing as the underlying architecture has been superseded.