add newa search subcommand

[kkaarreell]

Summary by Sourcery

Add a searchable CLI for NEWA state directories and reuse listing logic for richer search output.

New Features:

• Introduce a search subcommand to query NEWA state directories by text, event, erratum, RoG MR, or Jira metadata with regex support and configurable output detail levels.

Enhancements:

• Refactor state directory listing into a reusable helper that adds relative modification timestamps and supports brief vs detailed output modes.
• Clarify and expand documentation for the --prev-state-dir option, including multi-terminal behavior and examples.

Tests:

• Add extensive unit tests for the new search command covering matching behavior, regex handling, output formatting, and error cases.

[sourcery-ai]

Reviewer's Guide

Introduces a new search subcommand that scans NEWA state directories for regex-based patterns across metadata, refactors listing logic into reusable helpers with relative-time/brief output support, and documents the new behavior with extensive tests and README updates.

Sequence diagram for the new search subcommand

sequenceDiagram
    actor User
    participant main as main
    participant cmd_search as cmd_search
    participant ctx as CLIContext
    participant _search_object as _search_object
    participant print_state_dirs as print_state_dirs

    User->>main: newa search [options]
    main->>cmd_search: cmd_search(ctx, text, event, erratum, rog_mr, jira, brief, full)
    cmd_search->>ctx: enter_command
    cmd_search->>cmd_search: init_colors
    cmd_search->>cmd_search: compile regex patterns

    loop each state_dir in newa_statedir_topdir
        cmd_search->>ctx: state_dirpath = state_dir
        cmd_search->>cmd_search: evaluate search_specs
        alt search_type == text
            cmd_search->>cmd_search: state_dir.glob('*.yaml')
            cmd_search->>cmd_search: value_pattern.search(content)
        else search_type == jira
            cmd_search->>ctx: load_jira_jobs
            cmd_search->>_search_object: _search_object(jira_dict, key_pattern, value_pattern)
        else search_type in [event, erratum, rog]
            cmd_search->>ctx: load_artifact_jobs
            cmd_search->>_search_object: _search_object(obj_dict, key_pattern, value_pattern)
        end
        alt all specs matched
            cmd_search->>cmd_search: matching_state_dirs.append(state_dir)
        end
    end

    cmd_search->>cmd_search: determine show_brief / show_events / show_issues
    cmd_search->>print_state_dirs: print_state_dirs(ctx, matching_state_dirs, events, issues, False, False, False, brief)
    print_state_dirs->>ctx: load_artifact_jobs(filter_events=True)
    print_state_dirs->>print_state_dirs: _get_relative_time(state_dir)
    print_state_dirs->>print_state_dirs: print headers and details

    cmd_search-->>User: print summary of matches

Loading

Flow diagram for search output detail level selection

flowchart TD
    A[Start cmd_search] --> B{brief flag?}
    B -->|yes| C[show_brief = True]
    C --> D[show_events = False\nshow_issues = False]
    B -->|no| E{full flag?}
    E -->|yes| F[show_events = False\nshow_issues = False]
    E -->|no| G{jira option provided?}
    G -->|yes| H[show_issues = True\nshow_events = False]
    G -->|no| I[show_events = True\nshow_issues = False]
    D --> J[call print_state_dirs]
    F --> J
    H --> J
    I --> J
    J --> K[End cmd_search]

Loading

File-Level Changes

Change	Details	Files
Refactor list command to support reusable state-dir printing with relative timestamps and brief mode.	Extracted state-directory printing logic into a new print_state_dirs helper that can operate in detailed or brief modes. Added _get_relative_time to derive human-friendly modification times from .ppid files and included this in state-dir headers. Updated cmd_list to compute the set of state directories as before but delegate all output formatting to print_state_dirs, preserving event/issue/execution listing behavior.	newa/cli/commands/list_cmd.py
Add a regex-powered search CLI subcommand for querying NEWA state directories by text and structured objects.	Implemented _parse_search_spec and _search_object helpers to parse PATTERN vs KEY=PATTERN specs and recursively search nested objects using compiled regexes. Added cmd_search Click command with options for --text, --event, --erratum, --rog-mr, --jira, plus --brief and --full, including validation and dynamic default output detail. Within cmd_search, iterate state dirs from NEWA_STATEDIR_TOPDIR, apply AND-combined match specs over YAML contents, artifact jobs, and Jira jobs using NEWA’s loaders, and reuse print_state_dirs to render results at appropriate detail levels. Registered cmd_search with the main CLI entrypoint.	newa/cli/commands/search_cmd.py newa/cli/main.py
Document new search subcommand and improve --prev-state-dir behavior description.	Expanded README section for --prev-state-dir / -P to clarify per-terminal scoping and multi-session behavior with concrete examples. Added comprehensive documentation for the search subcommand, its options (--text, object-specific filters, --brief, --full), output formats, and usage scenarios.	README.md
Introduce unit tests covering search behavior and regex handling.	Created tests/unit/test_search.py to validate basic search, case-insensitive matching, no-match and empty-topdir behavior, and inclusion of state-dir descriptions. Added tests ensuring search scans multiple YAML files, supports partial matches, enforces required options, and handles complex/invalid regex patterns and object-specific use cases.	tests/unit/test_search.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 3 issues, and left some high level feedback:

• The --refresh/--refresh-all flags are still accepted and validated in cmd_list, but they are never used inside print_state_dirs (or elsewhere) to actually update Testing Farm request statuses anymore; consider wiring these options back to _update_all_tf_request_statuses (or equivalent) so the documented behavior is preserved.
• In cmd_search, the internal option name 'rog' is used when building error messages for the --rog-mr option (e.g., Invalid regular expression in --rog), which will be confusing to users; align the option label in error messages with the public flag name (e.g., --rog-mr).

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `--refresh`/`--refresh-all` flags are still accepted and validated in `cmd_list`, but they are never used inside `print_state_dirs` (or elsewhere) to actually update Testing Farm request statuses anymore; consider wiring these options back to `_update_all_tf_request_statuses` (or equivalent) so the documented behavior is preserved.
- In `cmd_search`, the internal option name `'rog'` is used when building error messages for the `--rog-mr` option (e.g., `Invalid regular expression in --rog`), which will be confusing to users; align the option label in error messages with the public flag name (e.g., `--rog-mr`).

## Individual Comments

### Comment 1
<location path="newa/cli/commands/list_cmd.py" line_range="352-360" />
<code_context>
+        else:
+            state_dirs = [Path(e.path) for e in sorted_entries[-last:]]
+
+    # Print all state directories
+    print_state_dirs(
+        ctx,
+        state_dirs,
+        events=events,
+        issues=issues,
+        refresh=refresh,
+        refresh_all=refresh_all,
+        specific_state_dir=specific_state_dir)
+
     # restore logger level and statedir
</code_context>
<issue_to_address>
**issue (bug_risk):** The --refresh / --refresh-all flags no longer trigger any TF status updates

These flags are still accepted and passed through, but nothing in `cmd_list` / `print_state_dirs` calls `_update_all_tf_request_statuses` based on them, so they currently have no effect. If this behavior change isn’t deliberate, please reintroduce the refresh logic (e.g., call `_update_all_tf_request_statuses` before printing when `refresh` / `refresh_all` is set). If it is deliberate, consider removing or repurposing the flags to avoid confusing users.
</issue_to_address>

### Comment 2
<location path="newa/cli/commands/list_cmd.py" line_range="27-36" />
<code_context>
+def _get_relative_time(state_dir: Path) -> str:
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Broad exception handling in _get_relative_time can hide underlying problems

Catching `Exception` here means issues like permission errors or bad mtimes are silently converted to `''`, which can mask real problems. Please narrow the exception types (e.g. `OSError`, `ValueError`) or at least log the exception before returning `''`. Also, you call `datetime.now().astimezone().tzinfo` twice; store it once (e.g. `tz = datetime.now().astimezone().tzinfo`) and reuse it to avoid minor inconsistencies if the time changes between calls.

Suggested implementation:

```python
def _get_relative_time(state_dir: Path) -> str:
    """Get relative time string for state directory.

    Uses the modification time of the .ppid file in the state directory.

    Args:
        state_dir: Path to the state directory

    Returns:
        Relative time string like "2 days ago", "3 hours ago", etc.
        Returns empty string if ppid file not found.

```

```python
    ppid_file = state_dir / ".ppid"
    if not ppid_file.exists():
        return ""

    try:
        tz = datetime.now().astimezone().tzinfo
        mtime = ppid_file.stat().st_mtime
        mtime_dt = datetime.fromtimestamp(mtime, tz=tz)
        now = datetime.now(tz)
        delta = now - mtime_dt
        return humanize.naturaltime(delta)
    except (OSError, ValueError):
        # Failed to read or interpret the ppid file; treat as no relative time.
        return ""

```

If this project has a logging convention (e.g. a module-level `logger`), you may optionally:
1. Import / obtain the logger in this module.
2. Log the caught exception inside the `except (OSError, ValueError):` block before returning `""`, for example: `logger.warning("Failed to compute relative time for %s", ppid_file, exc_info=True)`.
</issue_to_address>

### Comment 3
<location path="README.md" line_range="1107" />
<code_context>
+#### Option `--prev-state-dir`, `-P`

-Similar to `--state-dir`, however no directory is specified. Instead, `newa` will use the most recent (modified) directory used by `newa` process issued from the current shell (so the functionality won't collidate with `newa` processes from different terminals).
+Similar to `--state-dir`, however no directory is specified. Instead, `newa` will use the most recent (modified) directory used by `newa` process issued from the current shell.
+
+**Multi-Terminal Support:** This option is intentionally shell-scoped to support running multiple NEWA sessions in parallel across different terminals. Each terminal maintains its own "previous state-dir" context, allowing you to run different test campaigns simultaneously without interference:
</code_context>
<issue_to_address>
**nitpick (typo):** Clarify grammar around "newa process issued from the current shell"

Consider rephrasing to: "Instead, `newa` will use the most recently modified directory used by `newa` processes invoked from the current shell." This pluralizes "process" and replaces "issued from" with a more natural verb like "invoked from" or "run from."

```suggestion
Similar to `--state-dir`, however no directory is specified. Instead, `newa` will use the most recently modified directory used by `newa` processes invoked from the current shell.
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}