Skip to content

Bug: --config-overrides cannot set options for external plugin reporters #355

Description

@rafaelpereyra

The --config-overrides flag cannot be used to configure options for external plugin reporters (e.g., bedrock-summary-reporter). The reporters section of AshConfig is a fixed Pydantic model (ReporterConfigSegment) that only contains fields for built-in reporters. External plugin reporter configs have no corresponding field on this model, so overrides like reporters.bedrock-summary-reporter.options.model_id=... are silently ignored.

Steps to Reproduce

  1. Run:
ash report --format bedrock-summary-reporter \
  --output-dir .ash/ash_output \
  --config-overrides 'ash_plugin_modules=["automated_security_helper.plugin_modules.ash_aws_plugins"]' \
  --config-overrides 'reporters.bedrock-summary-reporter.options.model_id=us.anthropic.claude-haiku-4-5-20251001-v1:0' \
  --log-level VERBOSE
  1. Observe that the reporter still uses its hardcoded default model (us.anthropic.claude-3-7-sonnet-20250219-v1:0) instead of the overridden value.

Expected Behavior

--config-overrides should be able to set options for any registered reporter, including those loaded from external plugin modules.

Workaround

Use the ASH_BEDROCK_MODEL_ID environment variable instead:

ASH_BEDROCK_MODEL_ID="us.anthropic.claude-haiku-4-5-20251001-v1:0" ash report --format bedrock-summary-reporter ...

Root Cause

The ReporterConfigSegment Pydantic model in automated_security_helper/config/ash_config.py has statically defined fields for each built-in reporter (csv, html, markdown, etc.). External plugins registered via ash_plugin_modules don't get a corresponding field added to this model, so:

  1. The YAML config can't define reporters.bedrock-summary-reporter (it would fail validation)
  2. --config-overrides targeting that path has no effect

Suggested Fix

Options:

  1. Dynamic reporter config: Change ReporterConfigSegment to accept arbitrary additional keys (e.g., using model_config = ConfigDict(extra="allow")) so external plugin configs can be defined in the YAML and overridden via CLI.

  2. Plugin-specific override namespace: Add a separate config section like plugin_options that accepts arbitrary key-value pairs and is passed to plugins at initialization.

  3. Registry-based approach: After loading plugin modules, dynamically register their config schemas into the ReporterConfigSegment model before parsing the config file.

Environment

  • ASH version: v3 (installed via uvx)
  • OS: Linux (Amazon Linux)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions