[Feature]: Decouple Health event analyzer from mongodb

[XRFXLP]

Prerequisites

• I searched existing issues

Feature Summary

Currently, health event analyser is closely tied to mongodb as it accepts rules in form of mongodb pipelines, which break the DB agnosticity pattern of nvsentinel

Problem/Use Case

All modules should should have DB agnosticity

Proposed Solution

One of:

1. Parametrize the current health event analyzer config prioritizing expressiveness in order to be open for future analysis rules
2. TBD

Component

Fault Management

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[coderabbitai]

Warning

A plan generation is already in progress. Please wait for it to complete before requesting a new plan.

[coderabbitai]

Coding Plan

Approach

Ticket Summary

Decouple the health-events-analyzer from MongoDB by replacing MongoDB aggregation pipeline-based rules with a database-agnostic rule evaluation system that maintains expressiveness for complex event correlation and analysis patterns.

Codebase Summary

The health-events-analyzer is a Go-based service that correlates non-fatal health events to determine when they should escalate to fatal events. It currently evaluates rules defined as MongoDB aggregation pipelines in TOML configuration files, parsing and executing these pipelines via the DatabaseClient interface from store-client. The service includes the analyzer (XID burst detection), config (rule loading), parser (dynamic reference resolution), publisher (gRPC event publishing), and reconciler (core orchestration) packages. The fault-quarantine service provides a model for DB-agnostic design using CEL (Common Expression Language) for rule evaluation. The store-client package provides a robust abstraction layer supporting both MongoDB and PostgreSQL through query builders, change streams, and provider-specific implementations.

Observations

The health-events-analyzer is a Go-based service that correlates non-fatal health events to determine when they should escalate to fatal events. It currently evaluates rules defined as MongoDB aggregation pipelines in TOML configuration files, parsing and executing these pipelines via the DatabaseClient interface from store-client. The service includes the analyzer (XID burst detection), config (rule loading), parser (dynamic reference resolution), publisher (gRPC event publishing), and reconciler (core orchestration) packages. The fault-quarantine service provides a model for DB-agnostic design using CEL (Common Expression Language) for rule evaluation. The store-client package provides a robust abstraction layer supporting both MongoDB and PostgreSQL through query builders, change streams, and provider-specific implementations.

Assumptions

Assumption 1: Rule Expression Language Selection

Options Considered:

• CEL (Common Expression Language): General-purpose expression language that's fast, portable, safe, and optimized for compile-once/evaluate-many workflows. Already proven in fault-quarantine module. Excellent for boolean logic and field comparisons, but evaluates expressions that are similar to single-line functions, which may require custom extensions for aggregation operations.
• Extended Query Builder with in-memory evaluation: Fetch data using store-client query builders, then evaluate conditions in Go. Provides flexibility but requires implementing aggregation logic in application code.
• Provider-specific rule implementations with unified interface: Keep MongoDB pipelines for MongoDB, create equivalent PostgreSQL queries for PostgreSQL. Maximum expressiveness but duplicates rule definitions.

Chosen Option: CEL-based evaluation with custom aggregation functions and query builder integration

Rationale: This approach follows the established pattern from fault-quarantine (minimizing architectural divergence), provides a safe, sandboxed environment that's non-Turing complete, and can be extended with custom functions for aggregation operations. The two-phase approach (query builder fetches relevant events, CEL evaluates conditions) balances expressiveness with database agnosticity. This aligns with developer familiarity (fault-quarantine team already uses CEL) and maintains the system's principle of using abstract interfaces over database-specific implementations.

Assumption 2: Dynamic Reference Resolution Strategy

Options Considered:

• CEL variable binding: Pass the triggering event as a CEL variable (e.g., "event") and use CEL's native field access (event.healthevent.nodename)
• Template-based substitution before CEL evaluation: Replace "this." references before CEL compilation (similar to current parser approach)
• Custom CEL functions for event field access: Provide helper functions like getField("healthevent.nodename")

Chosen Option: CEL variable binding with the triggering event passed as context

Rationale: This is the idiomatic CEL pattern used across the ecosystem (Kubernetes, Firebase), provides type safety during CEL compilation, and simplifies the implementation by leveraging CEL's built-in field access capabilities rather than custom parsing logic.

Assumption 3: Data Fetching Strategy for Rule Evaluation

Options Considered:

• Rule-specific query specifications: Each rule defines query parameters (time windows, field filters) that the evaluator uses to fetch data
• Two-phase rule definition: Separate "fetch" configuration (query builder operations) from "evaluate" configuration (CEL expressions)
• Fixed fetch window with CEL filtering: Always fetch events from a standardized time window, use CEL to filter and aggregate

Chosen Option: Two-phase rule definition with explicit fetch and evaluate sections

Rationale: This provides clear separation of concerns, allows optimization of data fetching (using indexes, query builders), maintains expressiveness (fetch can use joins, time ranges, field filters), and makes rules easier to understand and debug. The fetch phase uses store-client query builders for DB agnosticity, while the evaluate phase uses CEL for portable business logic.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Rule Configuration Schema Evolution

Extend the rule configuration structure to support database-agnostic rule definitions while maintaining backward compatibility during migration.

Task 1: Define New Rule Configuration Schema

Create new rule configuration structures that separate data fetching from evaluation logic.

• Add new configuration structs in health-events-analyzer/pkg/config/rules.go for the two-phase rule format: FetchConfig (query parameters: time windows, field filters, limit) and EvaluateConfig (CEL expression, required variables)
• Define RuleV2 struct containing name, description, fetch configuration, evaluate configuration, recommended action, and message fields
• Add version field to TomlConfig to distinguish between v1 (MongoDB pipeline) and v2 (CEL-based) rules
• Maintain existing HealthEventsAnalyzerRule struct for backward compatibility
• Add validation logic to ensure required fields are present and CEL expressions compile successfully

Task 2: Implement Configuration Loading and Migration Support

Update configuration loading to handle both legacy and new rule formats.

• Extend LoadConfig() in health-events-analyzer/pkg/config/rules.go to detect rule version and load appropriate format
• Add configuration validation that compiles CEL expressions at load time to catch syntax errors early
• Create helper functions to convert common MongoDB pipeline patterns (time-based filtering, field matching, counting) to equivalent fetch/evaluate configurations
• Add logging to indicate which rule format is being used for observability during migration

🤖 Prompt for AI agents

Implement new rule configuration schema to support database-agnostic CEL-based
rules alongside existing MongoDB pipeline rules.

Create new configuration structures in
health-events-analyzer/pkg/config/rules.go:
- Add FetchConfig struct with time windows, field filters, and limit fields
- Add EvaluateConfig struct with CEL expression string and required variables
- Define RuleV2 struct with name, description, fetch config, evaluate config,
recommended action, and message
- Add version field to TomlConfig to distinguish v1 and v2 rules
- Keep HealthEventsAnalyzerRule struct unchanged for backward compatibility
- Add validation that checks required fields and compiles CEL expressions

Extend LoadConfig() to handle both rule formats:
- Detect rule version from configuration
- Route to appropriate loading logic based on version
- Compile CEL expressions at load time for early error detection
- Create helper functions to convert MongoDB pipeline patterns to fetch/evaluate
configs
- Add logging for rule format visibility during migration

Phase 2: CEL-Based Rule Evaluation Framework

Implement a CEL evaluation engine with custom functions for event correlation patterns, following the fault-quarantine model.

Task 1: Create Rule Evaluator Abstraction

Design the evaluation interface and base implementations.

• Create health-events-analyzer/pkg/evaluator/ package with RuleEvaluator interface defining Evaluate(ctx, triggerEvent, fetchedEvents) (matched bool, error) method
• Implement CelRuleEvaluator that compiles CEL expressions and evaluates them against event data
• Add CEL environment initialization with custom type declarations for HealthEvent protobuf messages
• Implement expression compilation and caching to avoid recompilation on every evaluation (following the most common pattern of parsing and checking expressions at config time, storing the AST, and evaluating repeatedly at runtime)
• Add error handling for CEL evaluation failures with detailed error messages

Task 2: Implement Custom CEL Functions for Aggregations

Extend CEL environment with domain-specific functions for event analysis.

• Add custom CEL functions in health-events-analyzer/pkg/evaluator/cel_extensions.go for common patterns: countEvents(events, condition) for counting matching events, timeWindow(event, duration) for time-based filtering, groupBy(events, field) for grouping operations, hasRepeatedValue(events, field, threshold) for repeated value detection
• Implement helper functions that operate on slices of HealthEvent objects
• Register these functions with the CEL environment during initialization
• Add unit tests for each custom function to verify correct behavior

Task 3: Implement Data Fetching Abstraction

Create the layer that translates fetch configurations to database queries.

• Create health-events-analyzer/pkg/evaluator/data_fetcher.go with EventFetcher interface defining FetchEvents(ctx, fetchConfig, triggerEvent) ([]HealthEvent, error) method
• Implement QueryBuilderFetcher that translates FetchConfig into store-client query builder operations
• Add support for common fetch patterns: time-based queries (events within X hours), field equality filters (nodename, errorcode), related entity queries (same GPU, same node), combined conditions using And/Or query builders
• Use the DatabaseClient.Find() method with dynamically built queries from store-client query package
• Add result limiting and pagination support to prevent excessive memory usage

🤖 Prompt for AI agents

Build CEL-based rule evaluation framework following fault-quarantine patterns.

Create health-events-analyzer/pkg/evaluator/ package with evaluation
abstractions:
- Define RuleEvaluator interface with Evaluate(ctx, triggerEvent, fetchedEvents)
method returning (bool, error)
- Implement CelRuleEvaluator struct that compiles and evaluates CEL expressions
- Initialize CEL environment with HealthEvent protobuf type declarations
- Cache compiled CEL ASTs to avoid recompilation (parse at config time, evaluate
at runtime)
- Add detailed error handling for CEL evaluation failures

Add custom CEL functions in cel_extensions.go:
- Implement countEvents(events, condition) for counting matches
- Implement timeWindow(event, duration) for time filtering
- Implement groupBy(events, field) for grouping
- Implement hasRepeatedValue(events, field, threshold) for repeated detection
- Register all custom functions with CEL environment
- Add unit tests for each function

Create data fetching layer in data_fetcher.go:
- Define EventFetcher interface with FetchEvents(ctx, fetchConfig, triggerEvent)
method
- Implement QueryBuilderFetcher that converts FetchConfig to query builder
operations
- Support time-based queries, field filters, related entity queries
- Use DatabaseClient.Find() with dynamically built queries
- Add result limiting and pagination

Phase 3: Query Builder Extensions for Event Correlation

Extend store-client query builders to support time-based queries and additional operators needed for rule evaluation.

Task 1: Add Time-Based Query Support

Enhance query builders with temporal filtering capabilities.

• Add time range methods to query builder in store-client/pkg/query/builder.go: WithinLast(duration), Between(start, end), After(timestamp), Before(timestamp)
• Implement ToMongo() translation for time operators using $gte/$lte with timestamp fields
• Implement ToSQL() translation for time operators using PostgreSQL timestamp comparison and JSONB field extraction
• Add support for dynamic time reference (e.g., "within last 24 hours from triggering event timestamp")

Task 2: Extend Query Builder Operators

Add additional comparison and logical operators for complex queries.

• Add Contains operator for substring matching in string fields
• Add NotIn operator as complement to existing In operator
• Add IsNull and IsNotNull operators for existence checks
• Ensure all new operators correctly translate to both MongoDB query syntax and PostgreSQL JSONB query syntax
• Add comprehensive unit tests validating both MongoDB and PostgreSQL output

🤖 Prompt for AI agents

Extend store-client query builders with time-based queries and additional
operators for event correlation.

Add temporal filtering to store-client/pkg/query/builder.go:
- Add WithinLast(duration) method for relative time ranges
- Add Between(start, end) method for absolute time ranges
- Add After(timestamp) and Before(timestamp) methods
- Implement ToMongo() translation using $gte/$lte operators
- Implement ToSQL() translation using PostgreSQL timestamp comparison and JSONB
extraction
- Support dynamic time references relative to trigger event timestamp

Add new comparison operators:
- Implement Contains operator for substring matching
- Implement NotIn operator as complement to In
- Implement IsNull and IsNotNull for existence checks
- Ensure all operators translate correctly to MongoDB and PostgreSQL syntax
- Add comprehensive unit tests for both database backends

Phase 4: Reconciler Integration

Refactor the reconciler to use the new CEL-based evaluation system while maintaining XID detector functionality.

Task 1: Refactor Rule Processing Pipeline

Replace MongoDB aggregation pipeline execution with CEL evaluation flow.

• Update processRule() method in health-events-analyzer/pkg/reconciler/reconciler.go to detect rule version
• For v2 rules: use EventFetcher to retrieve relevant events based on FetchConfig, pass trigger event and fetched events to CelRuleEvaluator, evaluate CEL expression against event data
• For v1 rules: maintain existing pipeline-based evaluation for backward compatibility
• Remove direct databaseClient.Aggregate() calls for v2 rule processing
• Update agent filter logic to apply during fetch phase rather than as pipeline stage

Task 2: Update Reconciler Initialization

Integrate the new evaluator components into reconciler startup.

• Add initialization in Start() method in health-events-analyzer/pkg/reconciler/reconciler.go to create CEL environment and compile all rule expressions at startup
• Create EventFetcher instance with DatabaseClient dependency
• Add rule validation that verifies CEL expressions compile successfully before starting event processing
• Update error handling to distinguish between fetch errors and evaluation errors
• Add metrics for CEL evaluation time and fetch query duration

Task 3: Maintain XID Detector Pattern

Ensure XID burst detection continues to work with provider-specific implementations.

• Keep existing XID detector logic in handleXidDetector() unchanged as it already follows provider-specific pattern
• Update XID detection to use v2 rule format if XID rules are migrated to CEL
• Ensure Go-based XID burst detector for PostgreSQL and pipeline-based approach for MongoDB remain functional
• Add integration tests verifying XID detection works correctly with both database backends

🤖 Prompt for AI agents

Integrate CEL-based evaluation into reconciler while maintaining backward
compatibility and XID detector functionality.

Refactor processRule() in health-events-analyzer/pkg/reconciler/reconciler.go:
- Detect rule version (v1 vs v2)
- For v2 rules: use EventFetcher to retrieve events based on FetchConfig, pass
trigger and fetched events to CelRuleEvaluator, evaluate CEL expression
- For v1 rules: keep existing pipeline-based evaluation
- Remove databaseClient.Aggregate() calls for v2 processing
- Move agent filter logic to fetch phase

Update reconciler initialization in Start() method:
- Create CEL environment and compile all v2 rule expressions at startup
- Initialize EventFetcher with DatabaseClient dependency
- Validate CEL expressions compile successfully before processing events
- Distinguish fetch errors from evaluation errors
- Add metrics for CEL evaluation time and fetch duration

Preserve XID detector functionality:
- Keep handleXidDetector() logic unchanged (already provider-specific)
- Support v2 rule format for XID rules if migrated
- Ensure both Go-based (PostgreSQL) and pipeline-based (MongoDB) XID detection
work
- Add integration tests for both database backends

Phase 5: Rule Migration and Validation

Provide tooling and documentation for migrating existing MongoDB pipeline rules to the new CEL-based format.

Task 1: Create Rule Migration Utility

Build a tool to assist in converting MongoDB pipeline rules to CEL expressions.

• Create health-events-analyzer/cmd/migrate-rules/ command-line tool that reads v1 TOML configurations and suggests v2 equivalents
• Implement pattern matching for common pipeline structures: $match stages to query builder filters, $count stages to countEvents() CEL functions, time-based $expr filters to time window queries, $group operations to groupBy() CEL functions
• Generate human-readable migration suggestions with explanations
• Add validation that compiles generated CEL expressions to catch errors

Task 2: Add Validation and Testing Infrastructure

Create comprehensive testing for rule evaluation correctness.

• Add integration tests in health-events-analyzer/pkg/reconciler/reconciler_test.go that verify v2 rules produce equivalent results to v1 rules for known scenarios
• Create test fixtures with sample events and expected evaluation outcomes
• Add validation tests that ensure fetch configurations produce correct query builder output
• Test both MongoDB and PostgreSQL execution paths to verify database agnosticity
• Add benchmarking tests to compare performance of pipeline-based vs CEL-based evaluation

Task 3: Update Documentation and Examples

Provide guidance for rule authors and operators.

• Update design document 007-event-correlation-and-analysis.md to describe the new CEL-based rule format with complete examples
• Create migration guide documenting the process of converting pipeline rules to CEL rules
• Add example rule configurations in tests/data/ showing v2 rule format for common patterns (repeated errors, time-window analysis, multi-condition correlation)
• Document custom CEL functions and their usage patterns
• Add troubleshooting section for common CEL evaluation errors

🤖 Prompt for AI agents

Create migration tooling, testing infrastructure, and documentation for
CEL-based rules.

Build migration utility in health-events-analyzer/cmd/migrate-rules/:
- Create CLI tool that reads v1 TOML configs and suggests v2 equivalents
- Implement pattern matching for common pipelines: $match to query filters,
$count to countEvents(), time-based $expr to time windows, $group to groupBy()
- Generate human-readable suggestions with explanations
- Validate generated CEL expressions compile successfully

Add comprehensive testing in
health-events-analyzer/pkg/reconciler/reconciler_test.go:
- Create integration tests verifying v2 rules match v1 results for known
scenarios
- Add test fixtures with sample events and expected outcomes
- Validate fetch configurations produce correct query builder output
- Test both MongoDB and PostgreSQL execution paths
- Add benchmarks comparing pipeline-based vs CEL-based performance

Update documentation:
- Update design doc 007-event-correlation-and-analysis.md with CEL rule format
and examples
- Create migration guide for converting pipeline to CEL rules
- Add example v2 configs in tests/data/ for common patterns
- Document custom CEL functions and usage
- Add troubleshooting section for CEL evaluation errors

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement new rule configuration schema to support database-agnostic CEL-based
rules alongside existing MongoDB pipeline rules.

Create new configuration structures in
health-events-analyzer/pkg/config/rules.go:
- Add FetchConfig struct with time windows, field filters, and limit fields
- Add EvaluateConfig struct with CEL expression string and required variables
- Define RuleV2 struct with name, description, fetch config, evaluate config,
recommended action, and message
- Add version field to TomlConfig to distinguish v1 and v2 rules
- Keep HealthEventsAnalyzerRule struct unchanged for backward compatibility
- Add validation that checks required fields and compiles CEL expressions

Extend LoadConfig() to handle both rule formats:
- Detect rule version from configuration
- Route to appropriate loading logic based on version
- Compile CEL expressions at load time for early error detection
- Create helper functions to convert MongoDB pipeline patterns to fetch/evaluate
configs
- Add logging for rule format visibility during migration
===============================================================================

Task: 2

Build CEL-based rule evaluation framework following fault-quarantine patterns.

Create health-events-analyzer/pkg/evaluator/ package with evaluation
abstractions:
- Define RuleEvaluator interface with Evaluate(ctx, triggerEvent, fetchedEvents)
method returning (bool, error)
- Implement CelRuleEvaluator struct that compiles and evaluates CEL expressions
- Initialize CEL environment with HealthEvent protobuf type declarations
- Cache compiled CEL ASTs to avoid recompilation (parse at config time, evaluate
at runtime)
- Add detailed error handling for CEL evaluation failures

Add custom CEL functions in cel_extensions.go:
- Implement countEvents(events, condition) for counting matches
- Implement timeWindow(event, duration) for time filtering
- Implement groupBy(events, field) for grouping
- Implement hasRepeatedValue(events, field, threshold) for repeated detection
- Register all custom functions with CEL environment
- Add unit tests for each function

Create data fetching layer in data_fetcher.go:
- Define EventFetcher interface with FetchEvents(ctx, fetchConfig, triggerEvent)
method
- Implement QueryBuilderFetcher that converts FetchConfig to query builder
operations
- Support time-based queries, field filters, related entity queries
- Use DatabaseClient.Find() with dynamically built queries
- Add result limiting and pagination
===============================================================================

Task: 3

Extend store-client query builders with time-based queries and additional
operators for event correlation.

Add temporal filtering to store-client/pkg/query/builder.go:
- Add WithinLast(duration) method for relative time ranges
- Add Between(start, end) method for absolute time ranges
- Add After(timestamp) and Before(timestamp) methods
- Implement ToMongo() translation using $gte/$lte operators
- Implement ToSQL() translation using PostgreSQL timestamp comparison and JSONB
extraction
- Support dynamic time references relative to trigger event timestamp

Add new comparison operators:
- Implement Contains operator for substring matching
- Implement NotIn operator as complement to In
- Implement IsNull and IsNotNull for existence checks
- Ensure all operators translate correctly to MongoDB and PostgreSQL syntax
- Add comprehensive unit tests for both database backends
===============================================================================

Task: 4

Integrate CEL-based evaluation into reconciler while maintaining backward
compatibility and XID detector functionality.

Refactor processRule() in health-events-analyzer/pkg/reconciler/reconciler.go:
- Detect rule version (v1 vs v2)
- For v2 rules: use EventFetcher to retrieve events based on FetchConfig, pass
trigger and fetched events to CelRuleEvaluator, evaluate CEL expression
- For v1 rules: keep existing pipeline-based evaluation
- Remove databaseClient.Aggregate() calls for v2 processing
- Move agent filter logic to fetch phase

Update reconciler initialization in Start() method:
- Create CEL environment and compile all v2 rule expressions at startup
- Initialize EventFetcher with DatabaseClient dependency
- Validate CEL expressions compile successfully before processing events
- Distinguish fetch errors from evaluation errors
- Add metrics for CEL evaluation time and fetch duration

Preserve XID detector functionality:
- Keep handleXidDetector() logic unchanged (already provider-specific)
- Support v2 rule format for XID rules if migrated
- Ensure both Go-based (PostgreSQL) and pipeline-based (MongoDB) XID detection
work
- Add integration tests for both database backends
===============================================================================

Task: 5

Create migration tooling, testing infrastructure, and documentation for
CEL-based rules.

Build migration utility in health-events-analyzer/cmd/migrate-rules/:
- Create CLI tool that reads v1 TOML configs and suggests v2 equivalents
- Implement pattern matching for common pipelines: $match to query filters,
$count to countEvents(), time-based $expr to time windows, $group to groupBy()
- Generate human-readable suggestions with explanations
- Validate generated CEL expressions compile successfully

Add comprehensive testing in
health-events-analyzer/pkg/reconciler/reconciler_test.go:
- Create integration tests verifying v2 rules match v1 results for known
scenarios
- Add test fixtures with sample events and expected outcomes
- Validate fetch configurations produce correct query builder output
- Test both MongoDB and PostgreSQL execution paths
- Add benchmarks comparing pipeline-based vs CEL-based performance

Update documentation:
- Update design doc 007-event-correlation-and-analysis.md with CEL rule format
and examples
- Create migration guide for converting pipeline to CEL rules
- Add example v2 configs in tests/data/ for common patterns
- Document custom CEL functions and usage
- Add troubleshooting section for CEL evaluation errors

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord or schedule a call!