Add FileCache Prune REST API endpoint #19321

x-INFiN1TY-x · 2025-09-17T07:12:09Z

Add FileCache Prune REST API

Introduces a REST API for on-demand pruning of OpenSearch's file cache. This production-ready implementation enables operators to efficiently manage disk space across warm node clusters.

Issue

#19322

Implementation Architecture

Implements OpenSearch's advanced TransportNodesAction pattern for sophisticated multi-node coordination:

REST Layer → Node Transport → Node Resolution → Parallel FileCache Operations

API Specification

Endpoint

POST /_cache/filecache/prune

Parameters

nodes (optional) — Comma-separated list of node IDs or node selectors (e.g., warm:true)
node (optional) — Single node ID for targeted operations
timeout (optional) — Operation timeout (e.g., 30s, 2m, 1h)

Response Format

{
  "acknowledged": true,
  "total_pruned_bytes": 2097152,
  "summary": {
    "total_nodes_targeted": 3,
    "successful_nodes": 2,
    "failed_nodes": 1,
    "total_cache_capacity": 32212254720
  },
  "nodes": {
    "warm-node-1": {
      "name": "opensearch-warm-1",
      "transport_address": "10.0.1.101:9300",
      "host": "warm-1.cluster.local",
      "ip": "10.0.1.101",
      "pruned_bytes": 1048576,
      "cache_capacity": 10737418240
    },
    "warm-node-2": {
      "name": "opensearch-warm-2", 
      "transport_address": "10.0.1.102:9300",
      "host": "warm-2.cluster.local",
      "ip": "10.0.1.102",
      "pruned_bytes": 1048576,
      "cache_capacity": 10737418240
    }
  },
  "failures": [
    {
      "node_id": "warm-node-3",
      "reason": "FileCache prune operation failed",
      "caused_by": "RuntimeException"
    }
  ]
}

Files Added

server/src/main/java/org/opensearch/action/admin/cluster/cache/
├── PruneCacheAction.java
├── PruneCacheRequest.java
├── PruneCacheResponse.java
├── NodePruneCacheResponse.java  [NEW - Advanced node response model]
├── TransportPruneCacheAction.java
└── package-info.java

server/src/main/java/org/opensearch/rest/action/admin/cluster/
└── RestPruneCacheAction.java

Files Modified

server/src/main/java/org/opensearch/action/ActionModule.java       # Action & REST handler registration
server/src/main/java/org/opensearch/node/Node.java                 # FileCache dependency injection
server/src/test/java/org/opensearch/action/ActionModuleTests.java  # Framework integration tests

Advanced Capabilities

Intelligent Node Targeting

# Target all warm nodes (default)
POST /_cache/filecache/prune

# Target specific warm nodes
POST /_cache/filecache/prune?nodes=warm-node-1,warm-node-2

# Target single node
POST /_cache/filecache/prune?node=warm-node-1

# With operation timeout
POST /_cache/filecache/prune?timeout=60s

Test Architecture

PruneCacheRequestResponseTests.java
TransportPruneCacheActionTests.java
RestPruneCacheActionTests.java

Behavior Details

Warm Nodes: Executes FileCache.prune() and returns actual freed bytes with capacity metrics
Non-Warm Nodes: Safely filtered out.
Null FileCache: Graceful handling returning zero metrics.
Exception Handling: FileCache layer exceptions properly surfaced with node context

Sequence Flow Diagram

- Implements POST /_cache/remote/prune endpoint for manual cache cleanup - Adds comprehensive action, transport, and REST handler implementation - Includes full test coverage for all components

github-actions · 2025-09-17T07:35:36Z

❌ Gradle check result for 7788eb6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

gaobinlong · 2025-09-17T07:36:16Z

Mind opening an issue to describe the purpose of this new API?

x-INFiN1TY-x · 2025-09-17T09:14:15Z

Gradle Check failed due to Flaky Tests : #17486

server/src/main/java/org/opensearch/action/ActionModule.java

server/src/main/java/org/opensearch/node/Node.java

server/src/main/java/org/opensearch/action/admin/cluster/cache/PruneCacheAction.java

server/src/main/java/org/opensearch/action/admin/cluster/cache/PruneCacheRequest.java

github-actions · 2025-09-19T12:53:12Z

❕ Gradle check result for 5f88ec4: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-09-19T12:53:34Z

Codecov Report

❌ Patch coverage is 84.13793% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.97%. Comparing base (7e0c6ea) to head (8fb90d0).
⚠️ Report is 35 commits behind head on main.

Files with missing lines	Patch %	Lines
...admin/cluster/cache/TransportPruneCacheAction.java	71.05%	8 Missing and 3 partials ⚠️
...on/admin/cluster/cache/NodePruneCacheResponse.java	63.63%	8 Missing ⚠️
...action/admin/cluster/cache/PruneCacheResponse.java	94.23%	0 Missing and 3 partials ⚠️
...est/action/admin/cluster/RestPruneCacheAction.java	93.75%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #19321    +/-   ##
==========================================
  Coverage     72.96%   72.97%            
- Complexity    69825    69870    +45     
==========================================
  Files          5667     5673     +6     
  Lines        320555   320700   +145     
  Branches      46348    46364    +16     
==========================================
+ Hits         233908   234035   +127     
- Misses        67769    67782    +13     
- Partials      18878    18883     +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-09-23T10:31:06Z

❌ Gradle check result for ca35f02: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-23T16:26:14Z

✅ Gradle check result for 8fb90d0: SUCCESS

x-INFiN1TY-x · 2025-09-23T16:43:43Z

Requesting Review

CC : @Harsh-87

Gagan6164

Apart from these minor comments, rest of the changes LGTM.

Gagan6164 · 2025-10-12T08:46:58Z

server/src/main/java/org/opensearch/action/admin/cluster/cache/NodePruneCacheResponse.java

+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (!(o instanceof NodePruneCacheResponse)) return false;


nit: The convention is to use == false instead of !.

Gagan6164 · 2025-10-12T09:15:35Z

server/src/main/java/org/opensearch/action/admin/cluster/cache/PruneCacheResponse.java

+    public boolean isPartiallySuccessful() {
+        return !getNodes().isEmpty() && !failures().isEmpty();
+    }
+
+    /**
+     * Check if the operation was completely successful (all targeted nodes succeeded)
+     *
+     * @return true if all targeted nodes succeeded
+     */
+    public boolean isCompletelySuccessful() {
+        return !getNodes().isEmpty() && failures().isEmpty();
+    }
+


Are we using these methods anywhere else except in tests? If not we can mark these @VisibleForTesting instead of making them public.

Gagan6164 · 2025-10-12T09:22:35Z

server/src/main/java/org/opensearch/action/admin/cluster/cache/TransportPruneCacheAction.java

+            );
+        }
+
+        if (warmNodes.isEmpty() && nodeIds.length == 0) {


nodeIds.length == 0 check is redundant here.

Gagan6164 · 2025-10-12T09:28:03Z

server/src/main/java/org/opensearch/action/admin/cluster/cache/TransportPruneCacheAction.java

+
+    @Override
+    protected NodePruneCacheResponse nodeOperation(NodeRequest nodeRequest) {
+        PruneCacheRequest request = nodeRequest.getRequest();


nit: I don't see request getting used here.

Gagan6164 · 2025-10-12T14:34:26Z

server/src/main/java/org/opensearch/node/Node.java

+                if (fileCache != null) {
+                    b.bind(FileCache.class).toInstance(fileCache);
+                } else {
+                    b.bind(FileCache.class).toProvider(() -> null);


Suggested change

b.bind(FileCache.class).toProvider(() -> null);

b.bind(FileCache.class).toProvider(Providers.of(null));

Gagan6164 · 2025-10-12T15:31:51Z

server/src/main/java/org/opensearch/action/admin/cluster/cache/TransportPruneCacheAction.java

+        assert request.concreteNodes() == null : "request concreteNodes shouldn't be set";
+
+        String[] nodeIds = clusterState.nodes().resolveNodes(request.nodesIds());
+        List<DiscoveryNode> warmNodes = Arrays.stream(nodeIds)


We can first collect Warm nodes and then take intersection with the nodeIds from request and we can reuse the same list at Line 85.

Also, if this PR get's merged first then we can get the same list directly fromDiscoveryNodes.getWarmNodes() as well.

Add FileCache Prune REST API endpoint

7788eb6

- Implements POST /_cache/remote/prune endpoint for manual cache cleanup - Adds comprehensive action, transport, and REST handler implementation - Includes full test coverage for all components

x-INFiN1TY-x marked this pull request as ready for review September 17, 2025 07:12

x-INFiN1TY-x requested a review from a team as a code owner September 17, 2025 07:12