Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,20 +94,21 @@ pertains to. If a message has no such context then the IP column will be empty:
There are 3 kinds of measurements:

* ping (delta receive) - delta between receiving heartbeats
* jitter (delta send) - delta between sending heartbeats
* timestamp delta - delta between remotely recorded sent timestamp and locally recorded receive time
* jitter (delta send) - the time that has passed between sending consecutive pings.
* timestamp delta - delta between remotely recorded timestamp when ping
was generated and a locally recorded timestamp when it was received.

Meshmonitor will print histograms of each of the three tracked values. All of these values need to be interpreted with
the `--ping` interval in mind (default 5ms) that is included in the measurement values. The values that are printed are
max, mean, and percentiles: 99th, 99.9th, and 99.99th:

```console
[ ] --------receive-(ms)--------- ---------delta-(ms)---------- ---------send-(ms)-----------
[ ] Max Mean 99 99.9 99.99| Max Mean 99 99.9 99.99| Max Mean 99 99.9 99.99
[ 172.31.10.72] 5.2 5.1 5.1 5.2 5.2| 0.2 0.0 0.0 0.1 0.2| 5.1 5.1 5.1 5.1 5.1
[ 172.31.14.3] 5.3 5.1 5.1 5.1 5.3| 0.4 0.2 0.2 0.2 0.4| 5.8 5.1 5.1 5.5 5.8
[ 172.31.9.146] 5.1 5.1 5.1 5.1 5.1| 5.1 2.6 5.0 5.1 5.1| 5.1 5.1 5.1 5.1 5.1
[ 172.31.5.177] 5.1 5.1 5.1 5.1 5.1| 5.2 2.8 5.2 5.2 5.2| 5.1 5.1 5.1 5.1 5.1
09:08:55 [ ] ----------ping-(ms)---------- ---------jitter-(ms)--------- ----timestamp-diff-(ms)------
09:08:55 [ ] Max Mean 99 99.9 99.99| Max Mean 99 99.9 99.99 | Max Mean 99 99.9 99.99
09:08:55 [ 172.31.10.72] 5.2 5.1 5.1 5.2 5.2| 5.1 5.1 5.1 5.1 5.1 | 0.2 0.0 0.0 0.1 0.2
09:08:55 [ 172.31.14.3] 5.3 5.1 5.1 5.1 5.3| 5.8 5.1 5.1 5.5 5.8 | 0.4 0.2 0.2 0.2 0.4
09:08:55 [ 172.31.9.146] 5.1 5.1 5.1 5.1 5.1| 5.1 5.1 5.1 5.1 5.1 | 5.1 2.6 5.0 5.1 5.1
09:08:55 [ 172.31.5.177] 5.1 5.1 5.1 5.1 5.1| 5.1 5.1 5.1 5.1 5.1 | 5.2 2.8 5.2 5.2 5.2
```

Measurements exceeding `--threshold` (default 20ms) will be printed in yellow. Those that exceed 1 second will be printed in
Expand All @@ -128,7 +129,7 @@ using [GraalVM Community Edition](https://github.com/graalvm/graalvm-ce-builds/r
dependencies and can be run as is.

A pure Java version in jar form (meshmonitor.jar) is also available. The Java version should work on
any platform with Java 17 or later installed (although it has only been tested on Linux).
any platform with Java 8 or later installed (although it has only been tested on Linux).

# Using Meshmonitor

Expand Down Expand Up @@ -230,11 +231,11 @@ Use the following command to run meshmonitor from the jar file:
java -jar meshmonitor.jar <ARGS>
```

It requires at least Java 17 to run.
Java 8 is enough run it but Java 11 is required to build and execute tests.

## Building

Java SDK is required to build and test the Meshmonitor. Version 17 or above.
Java SDK is required to build and test the Meshmonitor. Version 11 or above.
Maven is used as a build system but does not need to be installed locally.

The `mvnw` script (or `mvnw.cmd` on Windows) is used to bootstrap the build
Expand Down
8 changes: 4 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>

<maven.compiler.testSource>17</maven.compiler.testSource>
<maven.compiler.testTarget>17</maven.compiler.testTarget>
<maven.compiler.testSource>11</maven.compiler.testSource>
<maven.compiler.testTarget>11</maven.compiler.testTarget>

<imageName>meshmonitor</imageName>
<mainClass>org.voltdb.meshmonitor.cli.MeshMonitorCommand</mainClass>
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/voltdb/meshmonitor/ConsoleLogger.java
Original file line number Diff line number Diff line change
Expand Up @@ -69,14 +69,14 @@ private void log(String hostAddress, LogLevel logLevel, String message, Object..
String colourfulDateTime = CommandLine.Help.Ansi.AUTO.string(String.format("@|%s %s|@", logLevel, dateTime));
String colourfulHostAddress = CommandLine.Help.Ansi.AUTO.string(String.format("@|%s [%15s]|@", logLevel, hostAddress));

out.println(colourfulDateTime + " " + colourfulHostAddress + " " + message.formatted(args));
out.println(colourfulDateTime + " " + colourfulHostAddress + " " + String.format(message, args));
}

public void debug(InetSocketAddress socketAddress, String format, Object... args) {
String hostAddress = socketAddress.getAddress().getHostAddress();

if (enableDebugLogging) {
log(hostAddress, LogLevel.INFO, format.formatted(args));
log(hostAddress, LogLevel.INFO, String.format(format, args));
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,15 @@ public String getSimpleVersion() {
String maybeCommitId = Objects.toString(properties.get(GIT_COMMIT_ID_PROPERTY), "");

StringBuilder version = new StringBuilder();
if (!maybeTags.isBlank()) {
if (!maybeTags.trim().isEmpty()) {
version.append(maybeTags);
} else if (!maybeCommitId.isBlank()) {
} else if (!maybeCommitId.trim().isEmpty()) {
version.append(maybeCommitId);
} else {
version.append(VERSION_UNKNOWN);
}

if (!maybeBuildTime.isBlank()) {
if (!maybeBuildTime.trim().isEmpty()) {
version.append(" built on ").append(maybeBuildTime);
}

Expand Down
12 changes: 6 additions & 6 deletions src/main/java/org/voltdb/meshmonitor/HistogramLogger.java
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@ public void printResults(Monitor monitor, long minHiccupSize) {
MeshMonitorTimings currentTimings = monitor.getTimings();
InetSocketAddress remoteId = monitor.getRemoteId();

String receive = printAndReset(currentTimings.receiveHistogram(), minHiccupSize);
String delta = printAndReset(currentTimings.deltaHistogram(), minHiccupSize);
String send = printAndReset(currentTimings.sendHistogram(), minHiccupSize);
String ping = printAndReset(currentTimings.pingHistogram(), minHiccupSize);
String jitter = printAndReset(currentTimings.jitterHistogram(), minHiccupSize);
String timestampDiff = printAndReset(currentTimings.timestampDeltaHistogram(), minHiccupSize);

consoleLogger.log(remoteId, receive + "|" + delta + "|" + send);
consoleLogger.log(remoteId, ping + "|" + jitter + "|" + timestampDiff);
}

private String printAndReset(HistogramWithDelta histogram, long minHiccupSize) {
Expand All @@ -53,11 +53,11 @@ private String format(Histogram deltaHistogram, long minHiccupSizeMicroseconds)

private String getFormatWithColours(double value, double minHiccupSize) {
if (value > 999.9) {
String formatted = "%4.1fs".formatted(value / 1000.0);
String formatted = String.format("%4.1fs", value / 1000.0);
return CommandLine.Help.Ansi.AUTO.string("@|bold,red " + formatted + "|@");
}

String formatted = "%5.1f".formatted(value);
String formatted = String.format("%5.1f", value);
if (value > minHiccupSize) {
return CommandLine.Help.Ansi.AUTO.string("@|bold,yellow " + formatted + "|@");
}
Expand Down
70 changes: 59 additions & 11 deletions src/main/java/org/voltdb/meshmonitor/MeshMonitorTimings.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (C) 2024 Volt Active Data Inc.
* Copyright (C) 2024-2025 Volt Active Data Inc.
*
* Use of this source code is governed by an MIT
* license that can be found in the LICENSE file or at
Expand All @@ -9,25 +9,37 @@

import org.HdrHistogram.SynchronizedHistogram;

public record MeshMonitorTimings(
HistogramWithDelta receiveHistogram,
HistogramWithDelta sendHistogram,
HistogramWithDelta deltaHistogram) {
import java.util.Objects;

public final class MeshMonitorTimings {

public static final int NUMBER_OF_SIGNIFICANT_VALUE_DIGITS = 3;
public static final long HIGHEST_TRACKABLE_VALUE = 24 * 60 * 60 * 1000 * 1000L;

private final HistogramWithDelta pingHistogram;
private final HistogramWithDelta jitterHistogram;
private final HistogramWithDelta timestampDeltaHistogram;

public MeshMonitorTimings(
HistogramWithDelta pingHistogram,
HistogramWithDelta jitterHistogram,
HistogramWithDelta timestampDeltaHistogram) {
this.pingHistogram = pingHistogram;
this.jitterHistogram = jitterHistogram;
this.timestampDeltaHistogram = timestampDeltaHistogram;
}

public void pingReceived(long now, long lastReceiveTime, long timestampFromRemoteHost, long pingInterval) {
long valueToRecord = now - lastReceiveTime;
receiveHistogram.recordValueWithExpectedInterval(valueToRecord, pingInterval);
pingHistogram.recordValueWithExpectedInterval(valueToRecord, pingInterval);

// Abs because clocks can be slightly out of sync...
valueToRecord = Math.abs(now - timestampFromRemoteHost);
deltaHistogram.recordValueWithExpectedInterval(valueToRecord, pingInterval);
timestampDeltaHistogram.recordValueWithExpectedInterval(valueToRecord, pingInterval);
}

public void trackWakeupJitter(long observedInterval, long expectedInterval) {
sendHistogram.recordValueWithExpectedInterval(observedInterval, expectedInterval);
jitterHistogram.recordValueWithExpectedInterval(observedInterval, expectedInterval);
}

private static SynchronizedHistogram defaultHistogram() {
Expand All @@ -36,9 +48,45 @@ private static SynchronizedHistogram defaultHistogram() {

public static MeshMonitorTimings createDefault(ConsoleLogger logger) {
return new MeshMonitorTimings(
new HistogramWithDelta(logger, "receive", defaultHistogram()),
new HistogramWithDelta(logger, "send", defaultHistogram()),
new HistogramWithDelta(logger, "delta", defaultHistogram())
new HistogramWithDelta(logger, "ping", defaultHistogram()),
new HistogramWithDelta(logger, "jitter", defaultHistogram()),
new HistogramWithDelta(logger, "timestamp delta", defaultHistogram())
);
}

public HistogramWithDelta pingHistogram() {
return pingHistogram;
}

public HistogramWithDelta jitterHistogram() {
return jitterHistogram;
}

public HistogramWithDelta timestampDeltaHistogram() {
return timestampDeltaHistogram;
}

@Override
public boolean equals(Object obj) {
if (obj == this) return true;
if (obj == null || obj.getClass() != this.getClass())
return false;
MeshMonitorTimings that = (MeshMonitorTimings) obj;
return Objects.equals(this.pingHistogram, that.pingHistogram) &&
Objects.equals(this.jitterHistogram, that.jitterHistogram) &&
Objects.equals(this.timestampDeltaHistogram, that.timestampDeltaHistogram);
}

@Override
public int hashCode() {
return Objects.hash(pingHistogram, jitterHistogram, timestampDeltaHistogram);
}

@Override
public String toString() {
return "MeshMonitorTimings[" +
"pingHistogram=" + pingHistogram + ", " +
"jitterHistogram=" + jitterHistogram + ", " +
"timestampDeltaHistogram=" + timestampDeltaHistogram + ']';
}
}
5 changes: 3 additions & 2 deletions src/main/java/org/voltdb/meshmonitor/ServerManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

public class ServerManager {

Expand All @@ -30,14 +31,14 @@ public ServerManager(ConsoleLogger consoleLogger, MonitorFactory monitorFactory,
public synchronized List<Monitor> getMonitors() {
return monitors.stream()
.filter(Monitor::isRunning)
.toList();
.collect(Collectors.toList());
}

public synchronized List<InetSocketAddress> getConnections() {
return monitors.stream()
.filter(Monitor::isRunning)
.map(Monitor::getRemoteId)
.toList();
.collect(Collectors.toList());
}

public synchronized boolean createNewMonitorIfNotAlreadyPresent(SocketChannel channel, MeshMonitor meshMonitor, InetSocketAddress remoteId) {
Expand Down
19 changes: 11 additions & 8 deletions src/main/java/org/voltdb/meshmonitor/cli/MeshMonitorCommand.java
Original file line number Diff line number Diff line change
Expand Up @@ -121,14 +121,17 @@ public Integer call() {

System.out.println(
CommandLine.Help.Ansi.AUTO.string(
"""
@|green __ ___ __ _ __ \s
/ |/ /__ _____/ /_ ____ ___ ____ ____ (_) /_____ _____
/ /|_/ / _ \\/ ___/ __ \\/ __ `__ \\/ __ \\/ __ \\/ / __/ __ \\/ ___/
/ / / / __(__ ) / / / / / / / / /_/ / / / / / /_/ /_/ / / \s
/_/ /_/\\___/____/_/ /_/_/ /_/ /_/\\____/_/ /_/_/\\__/\\____/_/ \s
|@ %s
""".formatted(new GitPropertiesVersionProvider().getSimpleVersion())));
String.format(
"@|green __ ___ __ _ __ \n" +
" / |/ /__ _____/ /_ ____ ___ ____ ____ (_) /_____ _____\n" +
" / /|_/ / _ \\/ ___/ __ \\/ __ `__ \\/ __ \\/ __ \\/ / __/ __ \\/ ___/\n" +
" / / / / __(__ ) / / / / / / / / /_/ / / / / / /_/ /_/ / / \n" +
" /_/ /_/\\___/____/_/ /_/_/ /_/ /_/\\____/_/ /_/_/\\__/\\____/_/ \n" +
" |@ %s\n",
new GitPropertiesVersionProvider().getSimpleVersion()
)
)
);

ConsoleLogger consoleLogger = new ConsoleLogger(spec.commandLine().getOut(), enableDebugLogging);
ServerManager serverManager = new ServerManager(consoleLogger, Monitor::new, Duration.ofMillis(pingIntervalMilliseconds));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,21 @@ public void print(StringBuilder output, Monitor monitor) {
MeshMonitorTimings timings = monitor.getTimings();
InetSocketAddress remoteId = monitor.getRemoteId();

timings.receiveHistogram().getCumulativeHistogram(histogram ->
timings.pingHistogram().getCumulativeHistogram(histogram ->
histogramPrinter.printHistogram(output,
histogram,
remoteId,
"receive_seconds")
);

timings.deltaHistogram().getCumulativeHistogram(histogram ->
timings.timestampDeltaHistogram().getCumulativeHistogram(histogram ->
histogramPrinter.printHistogram(output,
histogram,
remoteId,
"delta_seconds")
);

timings.sendHistogram().getCumulativeHistogram(histogram -> {
timings.jitterHistogram().getCumulativeHistogram(histogram -> {
histogramPrinter.printHistogram(output,
histogram,
remoteId,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,48 +22,36 @@ public class GitPropertiesVersionProviderTest {
static Stream<Arguments> variousPropertiesContents() {
return Stream.of(
Arguments.of(
"""
git.tags=v1.0
git.build.time=2024-01-01
git.commit.id.abbrev=abc123
""",
"git.tags=v1.0\n" +
"git.build.time=2024-01-01\n" +
"git.commit.id.abbrev=abc123\n",
"version v1.0 built on 2024-01-01"
),
Arguments.of(
"""
git.tags=
git.build.time=2024-01-01
git.commit.id.abbrev=abc123
""",
"git.tags=\n" +
"git.build.time=2024-01-01\n" +
"git.commit.id.abbrev=abc123\n",
"version abc123 built on 2024-01-01"
),
Arguments.of(
"""
git.tags=
git.build.time=2024-01-01
git.commit.id.abbrev=
""",
"git.tags=\n" +
"git.build.time=2024-01-01\n" +
"git.commit.id.abbrev=\n",
"version unknown built on 2024-01-01"
),
Arguments.of(
"""
git.tags=
git.build.time=
git.commit.id.abbrev=
""",
"git.tags=\n" +
"git.build.time=\n" +
"git.commit.id.abbrev=\n",
"version unknown"
),
Arguments.of(
"""
git.build.time=2024-01-01
git.commit.id.abbrev=abc123
""",
"git.build.time=2024-01-01\n" +
"git.commit.id.abbrev=abc123\n",
"version abc123 built on 2024-01-01"
),
Arguments.of(
"""
git.build.time=2024-01-01
""",
"git.build.time=2024-01-01\n",
"version unknown built on 2024-01-01"
),
Arguments.of(
Expand Down
Loading