feat(loki): Add regex-based log masking for sensitive data protection#491
Open
Marie673 wants to merge 21 commits intografana:mainfrom
Open
feat(loki): Add regex-based log masking for sensitive data protection#491Marie673 wants to merge 21 commits intografana:mainfrom
Marie673 wants to merge 21 commits intografana:mainfrom
Conversation
- Add MaskingConfig struct for log masking configuration - Add MaskingPattern struct for custom regex patterns - Support builtin pattern identifiers (email, phone, credit_card, etc.) - Support custom regex patterns with optional replacements - Add GlobalReplacement and HidePatternType options - Add unit tests for JSON serialization and field validation
- Add builtin pattern ID constants (email, phone, credit_card, ip_address, mac_address, api_key, jwt_token) - Register precompiled regex patterns in builtinPatterns map - Add GetBuiltinPattern/IsValidBuiltinPattern helper functions - Add comprehensive unit tests for each pattern matching
- Add ValidateMaskingConfig for pattern validation - Check pattern count limit (max 20) - Validate builtin pattern identifiers - Validate custom regex syntax - Add LogMasker struct with compiled patterns - Add NewLogMasker factory function - Add MaskEntries method for applying masks to log entries - Support GlobalReplacement override - Support HidePatternType option - Disable back-references (literal replacement) - Add comprehensive unit tests for: - Validation (valid/invalid configs, limits) - LogMasker creation and configuration - MaskEntries with builtin/custom patterns - Edge cases (empty, unicode, overlapping)
- Add Masking field to QueryLokiLogsParams struct - Add applyMaskingToEntries helper function - Validate masking config before log fetch (early failure) - Apply masking after log retrieval, before returning - Add comprehensive unit tests for integration
- Add TestQueryLokiLogs_WithMasking for masking functionality - Add TestQueryLokiLogs_WithoutMasking for backward compatibility - Add TestQueryLokiLogs_MaskingValidationError for error handling - Verify builtin and custom pattern combinations work correctly - Ensure validation errors prevent unmasked data leaks
- Add BenchmarkLogMasker_MaskEntries_100Entries for 100 entries with builtin patterns - Add BenchmarkLogMasker_MaskEntries_20Patterns for max pattern limit (20) - Add BenchmarkLogMasker_MaskEntries_SLO for SLO verification - Add BenchmarkLogMasker_PatternCompilation for compilation time - Add TestSLOCompliance to verify 100ms SLO (actual: ~1.5ms) - Add TestPatternCompilationOnce to verify pattern reuse - Add helper function generateTestLogEntries for realistic test data
- Remove redundant doc comments from types, functions, and variables - Remove task reference comments from test files - Remove Japanese comments from pattern definitions - Keep section separator comments for readability - Align comment style with existing codebase (prometheus.go)
- Eliminate outdated integration tests from loki_test.go and loki_masking_unit_test.go - Streamline test cases to focus on essential functionality and validation - Ensure backward compatibility and error handling are still covered in remaining tests
- Remove support for custom patterns in MaskingConfig for security and simplicity. - Update validation logic to only consider builtin patterns. - Refactor tests to align with the new configuration, focusing on builtin patterns only. - Streamline JSON serialization and pattern handling in the masking implementation.
- Introduce LokiMaskingConfig struct to manage log masking settings. - Add ExtractLokiMaskingFromEnv function to read configuration from environment variables. - Implement validation and filtering methods for masking patterns. - Create comprehensive unit tests for environment variable extraction and pattern validation. - Enhance CLI integration for Loki masking configuration with appropriate flags.
…ted functions - Add WithMasker and MaskerFromContext functions for managing LogMasker in context. - Update Composed*ContextFunc functions to support optional LogMasker parameter. - Enhance tests for LogMasker context functions to ensure proper retrieval and handling. - Modify main application logic to integrate LogMasker into server context setup.
- Add tests for LogMasker context management, ensuring proper storage and retrieval. - Implement tests for backward compatibility when no masker is set in context. - Validate masking behavior when a masker is applied via context. - Ensure log entries remain unchanged when no masker or a nil masker is present.
- Introduce new test cases for backward compatibility with nil masker in context. - Validate log entry structure and masking behavior across various scenarios. - Ensure proper handling of empty results and format validation for masked entries. - Enhance tests for single and multiple masking patterns, confirming expected behavior.
- Introduce a new section on log masking in the README, outlining its purpose and configuration. - Detail the command-line flags for enabling log masking and specifying masking patterns. - Provide a table of supported masking patterns and an example of environment variable setup for users.
- Modify test cases in loki_test.go to replace 'container' label with 'job' label in LogQL queries. - Ensure assertions reflect the change from container to job label values for consistency in testing.
…d masking function - Simplify the lokiMaskingConfig struct by reordering fields for clarity. - Remove the applyMaskingToEntries function as it is no longer needed. - Ensure the codebase remains clean and maintainable by eliminating redundant code.
- Eliminate the TestApplyMaskingToEntries function from loki_masking_unit_test.go as it is no longer needed. - Ensure the remaining tests continue to validate the masking functionality effectively.
- Modify test cases in loki_test.go to replace 'job' label with 'container' label in LogQL queries. - Update assertions to reflect the change from job to container label values for consistency in testing. - Ensure that tests validate the expected behavior with the new label context.
- Modify test cases in prometheus_test.go to incorporate label matching using the 'job' label. - Update assertions to ensure the expected number of label names and values are returned. - Enhance test reliability by directly specifying label filters in the test parameters.
- queryLokiLogs関数のドキュメントコメントを復元 理由: commit 971c482で誤って削除してしまったため
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(loki): Add server-side log masking for sensitive data protection
Summary
Add regex-based log masking capability to the Loki log query tool. This feature automatically masks sensitive information (PII, authentication tokens, etc.) in log data before returning results to AI assistants, enhancing security and compliance.
Key Design Decision: Masking configuration is set at server startup by system administrators. AI/LLM cannot modify or override masking settings via tool calls. This ensures organizational security policies are enforced consistently and prevents unintended data exposure.
Changes
New Files
tools/loki_masking.gotools/loki_masking_unit_test.goModified Files
tools/loki.gotools/loki_test.gomcpgrafana.goLokiMaskingConfigstruct,WithMasker/MaskerFromContextcontext functionscmd/mcp-grafana/main.gomcpgrafana_test.goREADME.mdexamples/tls_example.goFeatures
Server-Level Configuration
Masking is configured at server startup via environment variables:
Or via CLI flags:
Or via Docker Compose:
Builtin Patterns
The following sensitive data patterns are provided as predefined options:
emailuser@example.com→[MASKED:email]phone+819012345678→[MASKED:phone]credit_card4111-1111-1111-1111→[MASKED:credit_card]ip_address192.168.1.1→[MASKED:ip_address]mac_address00:1A:2B:3C:4D:5E→[MASKED:mac_address]api_keyapi_key=abc123...→[MASKED:api_key]jwt_tokeneyJ...→[MASKED:jwt_token]Security Model
Architecture
Context-Based Masker Injection
The masking is applied automatically when enabled - AI assistants receive masked logs without any ability to bypass or modify the masking configuration.
Technical Details
Performance
Error Handling
Backward Compatibility
LOKI_MASKING_ENABLED=false)query_loki_logstool API (no new parameters added)Breaking Changes
ComposedStdioContextFunc,ComposedSSEContextFunc,ComposedHTTPContextFuncsignatures changed (addedmasker anyparameter)nilas the second argument to maintain existing behaviorTest Plan
Unit Tests
LogMaskermasking behavior testsMaskingConfigvalidation testsIntegration Tests
Non-Goals (Intentionally Not Supported)