RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer#847
Open
RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer#847
Conversation
rameeshm
requested review from
Copilot,
dineshkumar-yadav,
kumaab,
mneethiraj,
pradeepagrawal8184 and
vperiasamy
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive Ranger Audit Server architecture that decouples audit collection from Ranger Admin by implementing:
- A centralized Audit Server that receives audits from plugins via REST API and produces them to Kafka
- Separate consumer services for SOLR and HDFS that consume from Kafka and write to their respective destinations
- Shared common infrastructure for Kafka consumer management with support for consumer groups and rebalancing
Changes:
- New audit server infrastructure with 3 microservices (audit-server, consumer-solr, consumer-hdfs) and shared common module
- Integration with existing Ranger plugins (HDFS, Hive) to support HTTP-based audit destination
- Docker support for testing the complete audit pipeline with Kerberos authentication
- Thread-safe audit handling with concurrent counter management
Reviewed changes
Copilot reviewed 110 out of 111 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| ranger-audit-server/scripts/* | Shell scripts for starting/stopping audit services |
| ranger-audit-server/ranger-audit-server-service/* | Core audit server service with REST API and Kafka producer |
| ranger-audit-server/ranger-audit-consumer-solr/* | SOLR consumer service with Kafka-to-SOLR pipeline |
| ranger-audit-server/ranger-audit-consumer-hdfs/* | HDFS consumer service with Kafka-to-HDFS pipeline |
| ranger-audit-server/ranger-audit-common/* | Shared utilities, base classes, and consumer registry |
| agents-audit/dest-auditserver/* | New audit destination for plugins to send to audit server |
| agents-common/src/main/java/org/apache/ranger/plugin/audit/* | Enhanced audit handler with service type in additional info |
| agents-audit/core/src/main/java/org/apache/ranger/audit/provider/* | Thread-safe counters using AtomicLong |
| dev-support/ranger-docker/* | Docker compose and Dockerfiles for audit server services |
| pom.xml, distro/* | Build configuration and assembly descriptors |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
added 3 commits
February 11, 2026 13:42
…nsumer - fix failing testing
…nsumer - fix pmd issue
…onsumer - Fix audit commit failure propagation and recovery in the consumers
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 110 out of 111 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…nsumer - audit server partition management enhancement
mneethiraj
reviewed
Feb 20, 2026
Contributor
mneethiraj
left a comment
mneethiraj
left a comment
There was a problem hiding this comment.
@rameeshm - sending comments of the portions I reviewed so far. Will continue reviewing.
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
agents-audit/core/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java
Outdated
Show resolved
Hide resolved
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
added 4 commits
February 23, 2026 00:20
…nsumer - Fix review comments
…nsumer - PojoMappingFeature for AuditEvent Object for serialization
…onsumer - Fix review comments set #2
…nsumer - Audit Batch processing and failure reprocessing improvement
rameeshm
commented
Feb 25, 2026
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
added 2 commits
February 25, 2026 16:36
…onsumer - Fix duplicate dependency error in the pom for sl4j
…nsumer - Fix ubuntu audit ranger module war file creation failure
mneethiraj
reviewed
Feb 28, 2026
...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java
Outdated
Show resolved
Hide resolved
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 117 out of 118 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (7)
ranger-audit-server/ranger-audit-server-service/src/main/java/javax/ws/rs/core/NoContentException.java:1
- Defining a class under the
javax.ws.rs.*namespace inside the project is very risky: it can conflict with the JAX-RS API/implementation provided by the container or dependency set, and can lead to classloading/linkage issues that are hard to diagnose. Prefer fixing this by adding/aligning the correct JAX-RS/Jersey dependency that providesjavax.ws.rs.core.NoContentException, or by relocating/shading any workaround class into a project-owned package and updating the reference point that needs it.
ranger-audit-server/ranger-audit-server-service/src/main/java/javax/ws/rs/core/NoContentException.java:1 - Defining a class under the
javax.ws.rs.*namespace inside the project is very risky: it can conflict with the JAX-RS API/implementation provided by the container or dependency set, and can lead to classloading/linkage issues that are hard to diagnose. Prefer fixing this by adding/aligning the correct JAX-RS/Jersey dependency that providesjavax.ws.rs.core.NoContentException, or by relocating/shading any workaround class into a project-owned package and updating the reference point that needs it.
hdfs-agent/scripts/install.properties:1 - The HDFS agent install properties set
XAAUDIT.AUDITSERVER.FILE_SPOOL_DIRto a Hive log path (/var/log/hive/...). This looks like a copy/paste error and will cause HDFS plugin audit spooling to go to the wrong directory. Update it to an HDFS-appropriate directory (consistent with other HDFS audit spool paths in this repo).
ranger-audit-server/ranger-audit-server-service/src/main/webapp/WEB-INF/security-applicationContext.xml:1 - CSRF is explicitly disabled while sessions are forced (
create-session=\"always\"). If this service is intended to be a REST-only API (JWT/delegation-token), prefer making it stateless (no forced sessions) and limiting CSRF disabling to only the endpoints/methods that require it; otherwise, leaving CSRF off while maintaining session state increases risk for browser-based clients. Consider switching to stateless session creation and/or enabling CSRF with appropriate exemptions for non-browser clients.
ranger-audit-server/ranger-audit-server-service/src/main/webapp/WEB-INF/security-applicationContext.xml:1 - CSRF is explicitly disabled while sessions are forced (
create-session=\"always\"). If this service is intended to be a REST-only API (JWT/delegation-token), prefer making it stateless (no forced sessions) and limiting CSRF disabling to only the endpoints/methods that require it; otherwise, leaving CSRF off while maintaining session state increases risk for browser-based clients. Consider switching to stateless session creation and/or enabling CSRF with appropriate exemptions for non-browser clients.
ranger-audit-server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/producer/kafka/AuditRecoveryManager.java:1 RecoveryStats.toString()starts withRecoveryStats{but never appends a closing}. This makes logs harder to read/parse. Append a closing brace before returning the string.
ranger-audit-server/ranger-audit-consumer-solr/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java:1- A new
ObjectMapperis created for every health check request.ObjectMapperis designed to be reused and can be expensive to construct. Prefer using a static/singletonObjectMapper(or injecting one via Spring) to reduce allocation overhead.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
agents-common/src/test/java/org/apache/ranger/plugin/audit/TestRangerDefaultAuditHandler.java
Outdated
Show resolved
Hide resolved
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java
Outdated
Show resolved
Hide resolved
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java
Show resolved
Hide resolved
agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
Outdated
Show resolved
Hide resolved
mneethiraj
reviewed
Mar 5, 2026
agents-audit/core/src/main/java/org/apache/ranger/audit/model/AuthzAuditEvent.java
Outdated
Show resolved
Hide resolved
agents-audit/core/src/main/java/org/apache/ranger/audit/provider/AuditHandler.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java
Outdated
Show resolved
Hide resolved
…nsumer - Fix Review comments set #3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Audit Server should have service which has kafka as a queue mechanism and have Kafka Producer to relay and store audits in Kafka Topic / Partition.
Audit Server should have Solr consumer running as a separate service to write audit into SOLR Index as a destination
Audit Server should have HDFS consumer running as a separate service to write audit into HDFS.
All the above mentioned service should be available and test in docker container.
How was this patch tested?