Skip to content

Commit 0570e41

Browse files
arkmishCopilot
andcommitted
Backport the recursive snapshot summary tool
Adapt Apache ZooKeeper ZOOKEEPER-4566 by Szabolcs Bukros (05b2159). Reuse the comparer snapshot loader and fixtures, retaining subtree totals, depth behavior, launchers and CLI regression tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent e8661df commit 0570e41

5 files changed

Lines changed: 419 additions & 13 deletions

File tree

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
@echo off
2+
REM Licensed to the Apache Software Foundation (ASF) under one or more
3+
REM contributor license agreements. See the NOTICE file distributed with
4+
REM this work for additional information regarding copyright ownership.
5+
REM The ASF licenses this file to You under the Apache License, Version 2.0
6+
REM (the "License"); you may not use this file except in compliance with
7+
REM the License. You may obtain a copy of the License at
8+
REM
9+
REM http://www.apache.org/licenses/LICENSE-2.0
10+
REM
11+
REM Unless required by applicable law or agreed to in writing, software
12+
REM distributed under the License is distributed on an "AS IS" BASIS,
13+
REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
REM See the License for the specific language governing permissions and
15+
REM limitations under the License.
16+
17+
setlocal
18+
call "%~dp0zkEnv.cmd"
19+
20+
set ZOOMAIN=org.apache.zookeeper.server.SnapshotRecursiveSummary
21+
call %JAVA% -cp "%CLASSPATH%" %ZOOMAIN% %*
22+
23+
endlocal & exit /b %ERRORLEVEL%
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/usr/bin/env bash
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
ZOOBIN="${BASH_SOURCE-$0}"
19+
ZOOBIN="$(dirname "${ZOOBIN}")"
20+
ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"
21+
22+
if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
23+
. "$ZOOBINDIR"/../libexec/zkEnv.sh
24+
else
25+
. "$ZOOBINDIR"/zkEnv.sh
26+
fi
27+
28+
"$JAVA" -cp "$CLASSPATH" $JVMFLAGS \
29+
org.apache.zookeeper.server.SnapshotRecursiveSummary "$@"

‎zookeeper-docs/src/main/resources/markdown/zookeeperTools.md‎

Lines changed: 48 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ limitations under the License.
2424
* [zkTxnLogToolkit.sh](#zkTxnLogToolkit)
2525
* [zkSnapShotToolkit.sh](#zkSnapShotToolkit)
2626
* [zkSnapshotComparer.sh](#zkSnapshotComparer)
27+
* [zkSnapshotRecursiveSummaryToolkit.sh](#zkSnapshotRecursiveSummaryToolkit)
2728

2829
* [Testing](#Testing)
2930
* [Jepsen Test](#jepsen-test)
@@ -252,24 +253,58 @@ Use `-d`, `--debug` to display filtered paths and comparison details. Use
252253
The batch report visits each depth and sorts paths alphabetically within it.
253254
It retains the upstream empty label for the root in comparison lines.
254255
256+
<a name="zkSnapshotRecursiveSummaryToolkit"></a>
257+
258+
### zkSnapshotRecursiveSummaryToolkit.sh
259+
260+
Recursively summarize one snapshot subtree. This is the native backport of
261+
[ZOOKEEPER-4566](https://github.com/apache/zookeeper/commit/05b215994f5e145c2758c4089828b57ba471b329).
262+
263+
```bash
264+
bin/zkSnapshotRecursiveSummaryToolkit.sh snapshot.1 /app 1
265+
```
266+
267+
Usage: `SnapshotRecursiveSummary <snapshot_file> <starting_node> <max_depth>`.
268+
The starting node must be an existing absolute znode path. The maximum depth is
269+
a non-negative integer: 0 prints every non-leaf node, 1 prints the starting
270+
node and its non-leaf children, 2 adds another level, and so on.
271+
**Depth limits output only, not traversal or totals.**
272+
273+
`children` is the number of all descendants, not just immediate children.
274+
`data` is the sum of payload bytes for the node itself and all descendants.
275+
Null data contributes zero bytes. Leaves contribute to their ancestors' totals
276+
but are not printed; selecting a leaf produces no summary entries.
277+
278+
For example, if `/app` has 2 payload bytes, `/app/branch` has 3, and
279+
`/app/branch/leaf` has 5, the summary is:
280+
281+
```text
282+
/app
283+
children: 2
284+
data: 10
285+
-- /app/branch
286+
-- children: 1
287+
-- data: 8
288+
```
289+
255290
#### Snapshot analysis limitations
256291
257-
The comparer runs offline, reads files without modifying them, and supports
258-
uncompressed, `.gz`, and `.snappy` snapshots, including mixed formats. It
259-
validates snapshot checksums and reports unreadable or corrupt input rather
260-
than silently producing a successful analysis. Invalid arguments or file paths
261-
exit with code 2; snapshot read failures exit with code 1. A Windows launcher
262-
with a `.cmd` extension is included. Both launchers use the existing `zkEnv`
263-
configuration.
292+
Both tools run offline, read files without modifying them, and support
293+
uncompressed, `.gz`, and `.snappy` snapshots (including mixed formats in the
294+
comparer). They validate snapshot checksums and report unreadable or corrupt
295+
input rather than silently producing a successful analysis. Invalid arguments,
296+
file paths or summary starting paths exit with code 2; snapshot read failures
297+
exit with code 1. Windows launchers with the same names and a `.cmd` extension
298+
are also included. The launchers use the existing `zkEnv` configuration.
264299
265-
The comparer **includes ephemeral znodes** present in the snapshot; it does not
300+
Both tools **include ephemeral znodes** present in the snapshot; they do not
266301
report session records. This reflects the upstream traversal behavior, despite
267-
the original description claiming that ephemerals were ignored. It does not
268-
compare payload contents, ACLs, versions or other znode metadata.
302+
the original comparer's description claiming that ephemerals were ignored.
303+
Neither tool compares payload contents, ACLs, versions or other znode metadata.
269304
**Equal sizes/counts do not prove identical contents.** Snapshots may be fuzzy:
270-
the tool does not replay transaction logs, reconstruct point-in-time state,
271-
or establish transaction-consistent equality. It loads snapshots into memory,
272-
and recursive traversal visits the full snapshot.
305+
these tools do not replay transaction logs, reconstruct point-in-time state,
306+
or establish transaction-consistent equality. The tools load snapshots into
307+
memory, and recursive traversal still visits the full selected subtree.
273308
274309
<a name="Testing"></a>
275310
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
package org.apache.zookeeper.server;
20+
21+
import java.io.File;
22+
import java.io.IOException;
23+
import java.util.Collections;
24+
import java.util.Set;
25+
import java.util.TreeSet;
26+
import org.apache.yetus.audience.InterfaceAudience;
27+
import org.apache.zookeeper.ZKUtil;
28+
import org.apache.zookeeper.common.PathUtils;
29+
import org.apache.zookeeper.util.ServiceUtils;
30+
31+
/**
32+
* Recursively summarizes snapshot subtree data sizes and descendant counts.
33+
* Only non-leaf nodes are printed, but totals include all descendants and ephemeral nodes.
34+
* The maximum depth limits output, not traversal; zero means unlimited output depth.
35+
*
36+
* <p>Backported from Apache ZooKeeper commit 05b215994f5e145c2758c4089828b57ba471b329
37+
* (ZOOKEEPER-4566).
38+
*/
39+
@InterfaceAudience.Public
40+
public class SnapshotRecursiveSummary {
41+
42+
public static void main(String[] args) {
43+
if (args.length != 3) {
44+
System.err.println(getUsage());
45+
ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
46+
return;
47+
}
48+
try {
49+
new SnapshotRecursiveSummary().run(args[0], args[1], Integer.parseInt(args[2]));
50+
} catch (IllegalArgumentException e) {
51+
System.err.println(e.getMessage());
52+
System.err.println(getUsage());
53+
ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
54+
} catch (IOException e) {
55+
System.err.println("Unable to read snapshot: " + e.getMessage());
56+
ServiceUtils.requestSystemExit(ExitCode.UNEXPECTED_ERROR.getValue());
57+
}
58+
}
59+
60+
public void run(String snapshotFileName, String startingNode, int maxDepth) throws IOException {
61+
PathUtils.validatePath(startingNode);
62+
if (maxDepth < 0) {
63+
throw new IllegalArgumentException("max_depth must be a non-negative integer.");
64+
}
65+
String error = ZKUtil.validateFileInput(snapshotFileName);
66+
if (error != null) {
67+
throw new IllegalArgumentException(error);
68+
}
69+
DataTree dataTree = SnapshotComparer.getSnapshot(new File(snapshotFileName));
70+
if (dataTree.getNode(startingNode) == null) {
71+
throw new IllegalArgumentException("Starting node does not exist: " + startingNode);
72+
}
73+
StringBuilder builder = new StringBuilder();
74+
printZnode(dataTree, startingNode, builder, 0, maxDepth);
75+
System.out.println(builder);
76+
}
77+
78+
private long[] printZnode(DataTree dataTree, String name, StringBuilder builder, int level, int maxDepth) {
79+
DataNode node = dataTree.getNode(name);
80+
Set<String> children;
81+
long dataSize;
82+
synchronized (node) {
83+
dataSize = node.data == null ? 0 : node.data.length;
84+
children = new TreeSet<>(node.getChildren());
85+
}
86+
long[] result = {1L, dataSize};
87+
if (children.isEmpty()) {
88+
return result;
89+
}
90+
StringBuilder childBuilder = new StringBuilder();
91+
for (String child : children) {
92+
long[] childResult = printZnode(dataTree, name + (name.equals("/") ? "" : "/") + child,
93+
childBuilder, level + 1, maxDepth);
94+
result[0] += childResult[0];
95+
result[1] += childResult[1];
96+
}
97+
if (maxDepth == 0 || level <= maxDepth) {
98+
String indent = String.join("", Collections.nCopies(level, "--"));
99+
builder.append(indent).append(" ").append(name).append("\n");
100+
builder.append(indent).append(" children: ").append(result[0] - 1).append("\n");
101+
builder.append(indent).append(" data: ").append(result[1]).append("\n");
102+
builder.append(childBuilder);
103+
}
104+
return result;
105+
}
106+
107+
public static String getUsage() {
108+
String newLine = System.lineSeparator();
109+
return String.join(newLine,
110+
"USAGE:",
111+
"",
112+
"SnapshotRecursiveSummary <snapshot_file> <starting_node> <max_depth>",
113+
"",
114+
"snapshot_file: path to the zookeeper snapshot",
115+
"starting_node: the absolute path in the zookeeper tree where traversal should begin",
116+
"max_depth: non-negative output depth. 0 displays every non-leaf node; "
117+
+ "1 displays the starting node and its non-leaf children; 2 adds another level, and so on. "
118+
+ "This ONLY affects the level of details displayed, NOT the calculation.");
119+
}
120+
121+
}

0 commit comments

Comments
 (0)