RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer

[rameeshm]

What changes were proposed in this pull request?

Audit Server should have service which has kafka as a queue mechanism and have Kafka Producer to relay and store audits in Kafka Topic / Partition.
Audit Server should have Solr consumer running as a separate service to write audit into SOLR Index as a destination
Audit Server should have HDFS consumer running as a separate service to write audit into HDFS.
All the above mentioned service should be available and test in docker container.

How was this patch tested?

• Testing done in Docker running the containers of Ranger Audit Server with Ranger and HDFS and HIVE for policy and audit creation. Documentation on the JIRA has the testing details as well
• https://docs.google.com/document/d/1a6ZZJyD6Mr7tIV1A7YknobjgSitcYzS9bDJ032MvjMc/edit?tab=t.0#heading=h.iitwqftxhxfq

[Copilot]

Pull request overview

This PR introduces a comprehensive Ranger Audit Server architecture that decouples audit collection from Ranger Admin by implementing:

• A centralized Audit Server that receives audits from plugins via REST API and produces them to Kafka
• Separate consumer services for SOLR and HDFS that consume from Kafka and write to their respective destinations
• Shared common infrastructure for Kafka consumer management with support for consumer groups and rebalancing

Changes:

• New audit server infrastructure with 3 microservices (audit-server, consumer-solr, consumer-hdfs) and shared common module
• Integration with existing Ranger plugins (HDFS, Hive) to support HTTP-based audit destination
• Docker support for testing the complete audit pipeline with Kerberos authentication
• Thread-safe audit handling with concurrent counter management

Reviewed changes

Copilot reviewed 110 out of 111 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ranger-audit-server/scripts/*	Shell scripts for starting/stopping audit services
ranger-audit-server/ranger-audit-server-service/*	Core audit server service with REST API and Kafka producer
ranger-audit-server/ranger-audit-consumer-solr/*	SOLR consumer service with Kafka-to-SOLR pipeline
ranger-audit-server/ranger-audit-consumer-hdfs/*	HDFS consumer service with Kafka-to-HDFS pipeline
ranger-audit-server/ranger-audit-common/*	Shared utilities, base classes, and consumer registry
agents-audit/dest-auditserver/*	New audit destination for plugins to send to audit server
agents-common/src/main/java/org/apache/ranger/plugin/audit/*	Enhanced audit handler with service type in additional info
agents-audit/core/src/main/java/org/apache/ranger/audit/provider/*	Thread-safe counters using AtomicLong
dev-support/ranger-docker/*	Docker compose and Dockerfiles for audit server services
pom.xml, distro/*	Build configuration and assembly descriptors

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 110 out of 111 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mneethiraj]

@rameeshm - sending comments of the portions I reviewed so far. Will continue reviewing.

[mneethiraj]

Requiring serviceName same as the authenticatedUser doesn't seem right. I guess the intention is that the caller is the identity of the service account for the specified serviceName. This identity should be obtained from the service configuration retrieved from Ranger admin - for example configurations service.admin.users. If necessary, we can introduce a new configuration service.account.user for this purpose.