feat: add global --output-format json for structured CLI output - #773
feat: add global --output-format json for structured CLI output#773glenn-sq wants to merge 4 commits into
Conversation
Adds structured JSON output mode to `pat test` via `--output-format json`. When enabled, all logging is redirected to stderr and a single JSON object is printed to stdout containing summary counts, per-detection test results, failures, invalid specs, and skipped tests. Audit fixes included: - Clamp num_passed to zero to prevent negative counts in JSON summary - Guard setup_data_models print() with logging.error() to avoid stdout pollution - Emit JSON error envelope on early-exit paths (empty specs, no filter match) - Defensive error access in _serialize_function_result for non-dict errors - Use OutputFormat(str, Enum) for Typer-level validation and shell completion - Guard handler.setStream() with isinstance check for non-StreamHandler types - Buffer errored test results in _run_tests for consistent JSON output - Use compact JSON (no indent) for machine-consumable output - Revert cosmetic f-string and cast() changes to reduce diff noise - Add 19 unit tests covering all JSON serialization and output paths Version bump to 1.6.0. Made-with: Cursor
Adds a global --output-format {text,json} option (and PANTHER_OUTPUT_FORMAT
envvar) that enables structured JSON output across all CLI commands.
When enabled, all logging redirects to stderr and commands emit a single
JSON object to stdout. Commands with rich domain data (test, validate,
benchmark, check-packs, upload, delete, merge, migrate, enrich-test-data,
update-custom-schemas, init) produce custom JSON envelopes. All other
commands receive a generic {command, status, return_code} envelope
automatically via the call_and_exit wrapper.
Key implementation details:
- Global callback in Typer app sets module-level _output_format and
redirects logging StreamHandlers to stderr in JSON mode
- _COMMANDS_WITH_OWN_JSON frozenset routes command output correctly
- Test command uses flat schema with summary/results/failed/invalid/skipped
- num_passed calculation uses only detection-related invalid specs
- Interactive prompts (e.g. bulk delete confirmation) are skipped in
JSON mode with a log message to stderr
- stdout pollution from helper functions is suppressed in JSON mode
47 unit tests covering all JSON serialization, output functions, global
infrastructure, and command-level helpers.
Version bump to 1.6.0.
Closes panther-labs#634
Made-with: Cursor
PR SummaryMedium Risk Overview Implements command-specific JSON payloads for Written by Cursor Bugbot for commit 94e28e1. This will update automatically on new commits. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Extract output format state into panther_analysis_tool/output.py to break circular imports between main and command modules. This replaces lazy in-function imports with clean top-level imports from the new shared module. - Add missing json, Dict, Any imports in benchmark.py - Fix misplaced type: ignore comments after black reformatting - Add targeted pylint inline suppressions for complexity warnings introduced by JSON output code paths - Suppress invalid-name on OutputFormat enum members (lowercase by design) Made-with: Cursor
Commands in _COMMANDS_WITH_OWN_JSON suppress the generic JSON envelope in call_and_exit, so every return path must emit its own JSON in JSON mode. Previously, error paths in bulk_delete, benchmark, and migrate produced no stdout output, breaking CI/CD consumers. - bulk_delete: emit error JSON on all 5 failure paths - benchmark: add _emit_benchmark_error_json helper for all early returns - migrate: emit error JSON on EditorCommandNotFoundError - Apply isort/black formatting fixes Made-with: Cursor
1 participant
Summary
Adds a global
--output-format {text,json}option to all CLI commands, enabling structured JSON output for CI/CD integration and programmatic consumption. When enabled, all logging redirects to stderr and each command emits a single JSON object to stdout.This is a more complete implementation of the feature requested in #634, extending JSON output beyond just the
testcommand to cover the entire CLI surface. Supersedes #772.Architecture
--output-format json(orPANTHER_OUTPUT_FORMAT=jsonenvvar) is registered in the Typer app callback and applies to every command{command, status, return_code}envelope via thecall_and_exitwrapperCommands with custom JSON output
testvalidateuploadbenchmarkcheck-packsdeletemergemigrateenrich-test-dataupdate-custom-schemasinitAll other commands (
zip,release,fmt,check-connection,test-lookup-table,publish,explore,install,update) receive the generic envelope automatically.Changes
New/modified files:
panther_analysis_tool/main.py— Global--output-formatin app callback,_emit_json_result()generic envelope,_COMMANDS_WITH_OWN_JSONrouting, test command JSON output,_emit_check_packs_json(), upload/enrich/update-schemas JSON pathspanther_analysis_tool/command/standard_args.py—OutputFormat(str, Enum)definitionpanther_analysis_tool/command/validate.py—_emit_validate_json()with validation result/issuespanther_analysis_tool/command/benchmark.py—_emit_benchmark_json()with timing stats and performance ratingpanther_analysis_tool/command/bulk_delete.py—_emit_delete_json()with deletion resultspanther_analysis_tool/command/merge.py—_emit_merge_json()with update/conflict trackingpanther_analysis_tool/command/migrate.py—_emit_migrate_json()leveraging existingMigrationStatus.to_dict()panther_analysis_tool/command/init_project.py— JSON output with stdout suppression for helper functionstests/unit/panther_analysis_tool/test_json_output.py— 47 unit testsVersion bump: 1.5.2 → 1.6.0
Usage Examples
test— Run tests with full JSON output{ "summary": { "path": "rules/", "total": 1, "passed": 1, "failed": 0, "invalid": 0, "skipped": 0 }, "results": { "Crowdstrike.Detection.Passthrough": [ { "name": "Low Severity Finding", "passed": true, "errored": false, "functions": [ {"name": "rule", "status": "pass", "output": "true"}, {"name": "title", "status": "pass", "output": "Crowdstrike Alert: NGAV on macbook"}, {"name": "severity", "status": "pass", "output": "LOW"} ] } ] }, "failed": {}, "invalid": [], "skipped": [] }Extract just the summary:
Extract a specific test result:
validate— Validate detections against a Panther deploymentpanther_analysis_tool --output-format json validate --path rules/ 2>/dev/null{ "command": "validate", "return_code": 0, "status": "success", "data": { "valid": true, "error": null, "issues": [] } }On failure:
{ "command": "validate", "return_code": 1, "status": "error", "data": { "valid": false, "error": "Validation failed", "issues": [{"severity": "error", "message": "Invalid field reference"}] } }upload— Upload detections to Pantherpanther_analysis_tool --output-format json upload --path rules/ 2>/dev/null{ "command": "upload", "return_code": 0, "status": "success", "data": { "rules": {"new": 0, "modified": 2, "total": 15}, "policies": {"new": 0, "modified": 0, "total": 3} } }benchmark— Performance test a detection rulepanther_analysis_tool --output-format json benchmark \ --path rules/my_rule.py --iterations 10 2>/dev/null{ "command": "benchmark", "return_code": 0, "status": "success", "data": { "rule": "my_rule.yml", "hour": "2026-03-10T12:00:00", "iterations_completed": 10, "had_error": false, "read_time_seconds": {"mean": 0.45, "median": 0.42, "max": 0.78, "min": 0.31}, "processing_time_seconds": {"mean": 1.23, "median": 1.15, "max": 2.01, "min": 0.89}, "performance_rating": "highly_performant", "iterations": [{"read_time_nanos": 420000000, "processing_time_nanos": 1150000000}] } }check-packs— Validate pack completenessSuccess:
{ "command": "check-packs", "return_code": 0, "status": "success" }Missing items:
{ "command": "check-packs", "return_code": 1, "status": "error", "data": { "missing_items": [{"path": "packs/core.yml", "missing": ["Rule.Missing"]}] } }delete— Bulk delete detections or saved queriespanther_analysis_tool --output-format json delete \ --analysis-id Rule.Old Rule.Deprecated 2>/dev/null{ "command": "delete", "return_code": 0, "status": "success", "data": { "detections": ["Rule.Old", "Rule.Deprecated"] } }Note: Interactive confirmation is skipped in JSON mode. A log message is emitted to stderr.
merge— Merge analysis items with latest Panther contentpanther_analysis_tool --output-format json merge 2>/dev/null{ "command": "merge", "return_code": 0, "status": "success", "data": { "preview": false, "updated_items": ["AWS.CloudTrail.Root.Activity", "AWS.S3.BucketPolicy"], "merge_conflicts": ["Custom.SSO.Login"] } }migrate— Migrate detections to latest formatpanther_analysis_tool --output-format json migrate 2>/dev/null{ "command": "migrate", "return_code": 0, "status": "success", "data": { "single_item": false, "has_conflicts": false, "has_warnings": true, "empty": false, "migrated": ["Rule.A", "Rule.B"], "warnings": ["Rule.C: deprecated field 'Threshold'"] } }enrich-test-data— Enrich test data with live Panther datapanther_analysis_tool --output-format json enrich-test-data --path rules/ 2>/dev/null{ "command": "enrich-test-data", "return_code": 0, "status": "success", "data": { "enriched_ids": ["AWS.CloudTrail.Root.Activity", "AWS.S3.ServerAccess.Logging"] } }update-custom-schemas— Update custom log schemaspanther_analysis_tool --output-format json update-custom-schemas --path schemas/ 2>/dev/null{ "command": "update-custom-schemas", "return_code": 0, "status": "success", "data": { "results": [ {"failed": false, "summary": "Custom.MyLog: updated successfully"}, {"failed": false, "summary": "Custom.AuditLog: created"} ] } }init— Initialize a new Panther projectpanther_analysis_tool --output-format json init 2>/dev/null{ "command": "init", "return_code": 0, "status": "success", "data": { "pat_root_created": false } }Generic envelope — Commands without custom JSON (e.g.,
zip,fmt)panther_analysis_tool --output-format json zip --path rules/ --out ./dist 2>/dev/null{ "command": "zip", "return_code": 0, "status": "success", "message": "Created archive at ./dist/panther-analysis-all.zip" }On error:
{ "command": "check_connection", "return_code": 1, "status": "error", "errors": [{"error": "Connection refused"}] }Testing
47 unit tests covering:
_serialize_function_result,_serialize_test_result— pass/fail/error states, function name stripping_print_json_output,_print_json_error— all-passing, failures, invalid specs, skipped, negative-passed clamping, non-detection invalidsis_json_mode,get_output_format,_emit_json_result,_command_emits_own_json— mode switching, generic envelope, routing_emit_check_packs_json,_emit_validate_json,_emit_delete_json,_emit_merge_json— valid JSON output, edge casesOutputFormatvalues, string comparison, str subclassAll 630 existing tests pass with zero regressions.
Closes #634