From fb1048fda6522ee3c59fba161c09483d1ae2b18d Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 21 Nov 2025 20:11:08 +0000 Subject: [PATCH 01/14] Enhance documentation with section numbering and improve formatting across multiple files --- AGENTS.md | 65 +++++- README.adoc | 8 +- src/main/adoc/architecture.adoc | 11 +- src/main/adoc/benchmark-lifecycle.adoc | 13 +- src/main/adoc/decision-log.adoc | 185 +++++++++++++++++- src/main/adoc/jlbh-cookbook.adoc | 15 +- src/main/adoc/project-requirements.adoc | 82 ++++---- .../adoc/results-interpretation-guide.adoc | 23 ++- src/main/adoc/systemProperties.adoc | 24 +++ 9 files changed, 342 insertions(+), 84 deletions(-) create mode 100644 src/main/adoc/systemProperties.adoc diff --git a/AGENTS.md b/AGENTS.md index 8e1995a8..dc1a9e9f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,9 +10,10 @@ LLM-based agents can accelerate development only if they respect our house rules | Requirement | Rationale | |--------------|-----------| -| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the University of Oxford style guide for reference. | -| **ASCII-7 only** (code-points 0-127). Avoid smart quotes, non-breaking spaces and accented characters. | ASCII-7 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect the 8th bit to be 0. | -| If a symbol is not available in ASCII-7, use a textual form such as `micro-second`, `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | +| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the [University of Oxford style guide](https://www.ox.ac.uk/public-affairs/style-guide) for reference. | +| **ISO-8859-1** (code-points 0-255). Avoid smart quotes, non-breaking spaces and accented characters. | ISO-8859-1 survives every toolchain Chronicle uses. | +| If a symbol is not available in ISO-8859-1, use a textual form such as `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | +| Tools to check ASCII compliance include `iconv -f ascii -t ascii` and IDE settings that flag non-ASCII characters. | These help catch stray Unicode characters before code review. | ## Javadoc guidelines @@ -26,8 +27,20 @@ noise and slows readers down. | Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | | Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | -The principle that Javadoc should only explain what is *not* manifest from the signature is well-established in the -wider Java community. +The principle that Javadoc should only explain what is *not* manifest from the +signature is well-established in the wider Java community. + +Inline comments should also avoid noise. The following example shows the +difference: + +```java +// BAD: adds no value +int count; // the count + +// GOOD: explains a subtlety +// count of messages pending flush +int count; +``` ## Build & test commands @@ -40,11 +53,18 @@ mvn -q verify ## Commit-message & PR etiquette -1. **Subject line <= 72 chars**, imperative mood: "Fix roll-cycle offset in `ExcerptAppender`". +1. **Subject line <= 72 chars**, imperative mood: Fix roll-cycle offset in `ExcerptAppender`. 2. Reference the JIRA/GitHub issue if it exists. 3. In *body*: *root cause -> fix -> measurable impact* (latency, allocation, etc.). Use ASCII bullet points. 4. **Run `mvn verify`** again after rebasing. +### When to open a PR + +* Open a pull request once your branch builds and tests pass with `mvn -q clean verify`. +* Link the PR to the relevant issue or decision record. +* Keep PRs focused: avoid bundling unrelated refactoring with new features. +* Re-run the build after addressing review comments to ensure nothing broke. + ## What to ask the reviewers * *Is this AsciiDoc documentation precise enough for a clean-room re-implementation?* @@ -53,6 +73,14 @@ mvn -q verify * Does the commit point back to the relevant requirement or decision tag? * Would an example or small diagram help future maintainers? +### Security checklist (review **after every change**) + +**Run a security review on *every* PR**: Walk through the diff looking for input validation, authentication, authorisation, encoding/escaping, overflow, resource exhaustion and timing-attack issues. + +**Never commit secrets or credentials**: tokens, passwords, private keys, TLS materials, internal hostnames, Use environment variables, HashiCorp Vault, AWS/GCP Secret Manager, etc. + +**Document security trade-offs**: Chronicle prioritises low-latency systems; sometimes we relax safety checks for specific reasons. Future maintainers must find these hot-spots quickly, In Javadoc and `.adoc` files call out *why* e.g. "Unchecked cast for performance - assumes trusted input". + ## Project requirements See the [Decision Log](src/main/adoc/decision-log.adoc) for the latest project decisions. @@ -84,7 +112,7 @@ This tight loop informs the AI accurately and creates immediate clarity for all When using AI agents to assist with development, please adhere to the following guidelines: -* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ASCII-7 guidelines outlined above. +* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ISO-8859-1 guidelines outlined above. Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. * **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. * **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. @@ -122,8 +150,7 @@ Date:: YYYY-MM-DD Context:: * What is the issue that this decision addresses? * What are the driving forces, constraints, and requirements? -Decision Statement:: -* What is the change that is being proposed or was decided? +Decision Statement :: What is the change that is being proposed or was decided? Alternatives Considered:: * [Alternative 1 Name/Type]: ** *Description:* Brief description of the alternative. @@ -159,3 +186,23 @@ section:: Top Level Section ### Emphasis and Bold Text In AsciiDoc, an underscore `_` is _emphasis_; `*text*` is *bold*. + +### Section Numbering + +Use automatic section numbering for all `.adoc` files. + +* Add `:sectnums:` to the document header. +* Do not prefix section titles with manual numbers to avoid duplication. + +```asciidoc += Document Title +Chronicle Software +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +The document overview goes here. + +== Section 1 Title +``` diff --git a/README.adoc b/README.adoc index 12c47f81..6d6f0380 100644 --- a/README.adoc +++ b/README.adoc @@ -4,6 +4,8 @@ Chronicle Software :toc: macro :toclevels: 2 :icons: font +:lang: en-GB +:source-highlighter: rouge image:https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh] image:https://javadoc.io/badge2/net.openhft/JLBH/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-wire/latest/index.html"] @@ -17,12 +19,12 @@ toc::[] == About -Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a microbenchmark. +Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a micro-benchmark. See <> for a series of articles introducing JLBH. -An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles.] +An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles]. link:src/main/adoc/project-requirements.adoc[The requirements document] contains detailed feature descriptions. -For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_7-glossary[the Glossary (section 7)]. +For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_glossary[the Glossary (section 7)]. Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread. To do this, use the JLBH.eventLoopHandler method rather than JLBH.start. diff --git a/src/main/adoc/architecture.adoc b/src/main/adoc/architecture.adoc index 46445443..cd223d26 100644 --- a/src/main/adoc/architecture.adoc +++ b/src/main/adoc/architecture.adoc @@ -1,15 +1,16 @@ = Chronicle JLBH - Architecture Overview Chronicle Software :toc: +:sectnums: :lang: en-GB :source-highlighter: rouge -== 1. Introduction +== Introduction Chronicle JLBH (Java Latency Benchmark Harness) is a library designed for measuring and analyzing the latency of Java applications "in context," particularly under specific throughput conditions and accounting for coordinated omission. This document outlines its high-level architecture, major components, and key operational flows. -== 2. Major Components +== Major Components The JLBH system is composed of several key classes and interfaces that work together to orchestrate and execute benchmarks: @@ -80,7 +81,7 @@ The `startTimeNs` is the calculated ideal start time for the iteration. ** Monitoring is enabled by default (`JLBHOptions.recordOSJitter` defaults to true) and can be disabled via `recordOSJitter(false)`. ** Repeatedly calls `System.nanoTime()` to detect delays greater than `recordJitterGreaterThanNs` and records these into a dedicated `Histogram`. -== 3. Execution Flow +== Execution Flow The harness follows a well-defined lifecycle, detailed visually in link:benchmark-lifecycle.adoc[benchmark lifecycle diagram]. A typical execution sequence is as follows: @@ -128,7 +129,7 @@ The harness may busy-wait (spin using `System.nanoTime()`) if the current time i ** The `JLBHTask.complete()` method is invoked for any final cleanup by the user's task. ** The `OSJitterMonitor` thread is terminated if it was running. -== 4. Threading Model +== Threading Model JLBH has a specific threading model that users and contributors should understand: @@ -161,7 +162,7 @@ Chronicle Core's `Histogram` has specific single-writer assumptions unless expli ** To safely access this result from a thread other than the main harness thread (e.g., an application thread wanting to query results after a test run), `JLBHResultConsumer.newThreadSafeInstance()` should be used. This creates a `ThreadSafeJLBHResultConsumer` which uses a `volatile` field to ensure proper publication of the immutable result. -== 5. Data Management and Results +== Data Management and Results JLBH pays careful attention to how latency data is recorded, structured, and reported: diff --git a/src/main/adoc/benchmark-lifecycle.adoc b/src/main/adoc/benchmark-lifecycle.adoc index 322d3bf4..afa893b3 100644 --- a/src/main/adoc/benchmark-lifecycle.adoc +++ b/src/main/adoc/benchmark-lifecycle.adoc @@ -1,7 +1,8 @@ = Benchmark Lifecycle +:toc: +:sectnums: :lang: en-GB :source-highlighter: rouge -:toc: [mermaid] ---- @@ -14,7 +15,7 @@ graph LR A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are: -== 1. Setup +== Setup This initial phase prepares the benchmark environment: @@ -25,7 +26,7 @@ This initial phase prepares the benchmark environment: * **OS Jitter Monitoring**: Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. * **Affinity**: If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. -== 2. Warmup +== Warmup Before any measurements are formally recorded, the harness executes a warm-up phase: @@ -36,7 +37,7 @@ Before any measurements are formally recorded, the harness executes a warm-up ph * **Task Notification**: After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. * **Optional Pause**: If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. -== 3. Execution (Measurement Runs) +== Execution (Measurement Runs) This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): @@ -59,7 +60,7 @@ This is the core phase where timed iterations are performed and latency data is *** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. *** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. -== 4. Reporting +== Reporting After all measurement runs are completed, the aggregated results are finalized and made available: @@ -70,7 +71,7 @@ After all measurement runs are completed, the aggregated results are finalized a ** `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. ** `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. -== 5. Cleanup +== Cleanup The final phase ensures that any resources are properly released: diff --git a/src/main/adoc/decision-log.adoc b/src/main/adoc/decision-log.adoc index 1113684c..96521f89 100644 --- a/src/main/adoc/decision-log.adoc +++ b/src/main/adoc/decision-log.adoc @@ -1,14 +1,23 @@ = Chronicle JLBH - Decision Log Chronicle Software -:revnumber: 1.0 -:revdate: 24 May 2025 :toc: :lang: en-GB +:source-highlighter: rouge -This document records key architectural and project choices made during the development of Chronicle JLBH. Entries follow the Decision Record Template so that contributors can trace why a change occurred. +This document records key architectural and project choices made during the development of Chronicle JLBH. +Identifiers use the `JL` project scope with Nine-Box tags (FN, NF-P, NF-O, DOC, OPS, RISK, TEST, UX) and link back to the requirements in `project-requirements.adoc`. -== Decisions +== Decision Index +* link:#JL-DOC-001[JL-DOC-001 Adopt AsciiDoc for project documentation] +* link:#JL-FN-002[JL-FN-002 Single-threaded harness with JLBHTask lifecycle] +* link:#JL-NF-P-003[JL-NF-P-003 Default coordinated-omission compensation and histograms] +* link:#JL-OPS-004[JL-OPS-004 OS jitter tracking as an optional probe] +* link:#JL-FN-005[JL-FN-005 Additional probes and NanoSampler API] +* link:#JL-OPS-006[JL-OPS-006 CSV serialisation and result retention] +* link:#JL-OPS-007[JL-OPS-007 CI integration via TeamCity statistics] + +[[JL-DOC-001]] === [JL-DOC-001] Adopt AsciiDoc for project documentation Date:: 2025-05-24 @@ -35,3 +44,171 @@ Impact & Consequences:: * Some external tools may expect Markdown, requiring conversion. Notes/Links:: ** https://asciidoctor.org[AsciiDoc project page] + +[[JL-FN-002]] +=== [JL-FN-002] Single-threaded harness with JLBHTask lifecycle + +Date:: 2025-11-14 +Context:: +* JLBH targets low-latency Java applications where benchmark overhead and scheduling noise must be minimised. +* The harness needs a clear lifecycle so that warm-up, steady-state runs and shutdown can be expressed consistently across benchmarks. +* Requirements: FN-001 Latency Sampling, FN-006 Event Loop Integration, NF-P-001 Performance, NF-O-001 Portability. +Decision Statement:: +* Run the JLBH harness logic on a single dedicated thread, driving user code through the `JLBHTask` lifecycle (`init`, `run`, `warmedUp`, `runComplete`, `complete`). +* Allow the harness thread to be supplied by the caller via `eventLoopHandler(EventLoop)` while preserving the single-threaded execution model. +Alternatives Considered:: +* Multi-threaded harness with worker pools :: +** *Pros:* Could saturate multi-core CPUs more easily in some scenarios. +** *Cons:* Adds scheduling variability, complicates reasoning about pauses and makes per-iteration latency harder to interpret. +* Delegating to a generic micro-benchmark framework (e.g. JMH) :: +** *Pros:* Reuses existing tooling and reporting. +** *Cons:* Less control over in-context execution, fewer hooks for probes and OS jitter, and different lifecycle assumptions than JLBH requires. +Rationale for Decision:: +* A single harness thread keeps timing behaviour predictable and avoids extra scheduling noise beyond what the benchmarked code introduces. +* The explicit lifecycle methods match the phases described in the requirements and README, making it easy to structure complex benchmarks. +Impact & Consequences:: +* Benchmarked code must respect the single-threaded driver model, spawning its own threads only when needed and understanding the impact on measurements. +* Event-loop integration is explicit: users can install JLBH onto an existing loop without changing the underlying lifecycle. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:benchmark-lifecycle.adoc[Benchmark lifecycle overview] + +[[JL-NF-P-003]] +=== [JL-NF-P-003] Default coordinated-omission compensation and high-resolution histograms + +Date:: 2025-11-14 +Context:: +* Co-ordinated omission can hide worst-case latency by skipping samples during pauses or back-pressure. +* JLBH must support high-resolution percentile analysis at extreme tails to satisfy low-latency users. +* Requirements: FN-001 Latency Sampling, FN-002 Coordinated-Omission Compensation, NF-P-001 Performance. +Decision Statement:: +* Enable compensation for co-ordinated omission by default via `accountForCoordinatedOmission(true)`, with an option to disable it when raw uncorrected timings are required. +* Use high-resolution histograms as the internal representation for recorded latencies so that percentiles can be reported accurately across a wide range of values. +Alternatives Considered:: +* No coordinated-omission compensation :: +** *Pros:* Simpler mental model; raw timings only. +** *Cons:* Under-reports tail latency in back-pressured systems, undermining the harness goals. +* External correction tooling applied after the run :: +** *Pros:* Keeps the core harness simpler. +** *Cons:* Splits logic across tools, increases risk of misconfiguration and makes it harder to reason about what the reported percentiles mean. +Rationale for Decision:: +* Default-on correction matches the expectations set in the JLBH documentation and literature on latency measurement accuracy. +* High-resolution histograms are a proven structure for representing latency distributions with minimal overhead. +Impact & Consequences:: +* Users must be aware that reported percentiles are corrected unless they explicitly disable the feature. +* Benchmark authors can rely on accurate tail reporting without bolting on extra tooling. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-004]] +=== [JL-OPS-004] OS jitter tracking as an optional probe + +Date:: 2025-11-14 +Context:: +* Kernel scheduling, interrupts and other background activity can introduce jitter that affects latency measurements. +* Users often need to distinguish between pauses caused by their code and pauses caused by the operating system or environment. +* Requirements: FN-004 OS Jitter Tracking, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide an optional OS jitter probe implemented as a background thread that records scheduler delays beyond a configured threshold, and summarise the results alongside core latency histograms. +* Enable OS jitter tracking by default, with configuration options such as `recordOSJitter(false)` to disable it when the overhead is not acceptable. +Alternatives Considered:: +* Rely on external OS-level tools or profilers :: +** *Pros:* No extra threads or complexity inside the harness. +** *Cons:* Harder to correlate jitter with benchmark iterations; adds extra setup steps for users. +* Always disabling jitter tracking in the core library :: +** *Pros:* Zero overhead from jitter measurement. +** *Cons:* Removes a key diagnostic feature that helps interpret unexpected latency spikes. +Rationale for Decision:: +* Integrating jitter measurement directly into the harness keeps benchmark setup simple and ensures results remain correlated with the workload. +* Making the feature optional and configurable manages overhead while preserving a sensible default for investigation. +Impact & Consequences:: +* When enabled, benchmarks incur some additional overhead from the jitter thread; users can trade this off against diagnostic value. +* CI and local runs can use the jitter probe to explain outliers and refine environment configuration (CPU affinity, power settings). +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-FN-005]] +=== [JL-FN-005] Additional probes and NanoSampler API + +Date:: 2025-11-14 +Context:: +* Benchmarks often need to record latencies for sub-stages (for example serialisation, network calls) without distorting the end-to-end latency distribution. +* JLBH must make it easy to capture these timings while keeping the main histogram focused on overall request latency. +* Requirements: FN-003 Additional Probes, NF-P-001 Performance, NF-UX-001 Usability. +Decision Statement:: +* Provide `addProbe(String name)` on `JLBH` that returns a `NanoSampler`, each backed by its own histogram and reported alongside the primary distribution. +* Keep the end-to-end `sample` / `sampleNanos` API unchanged so that additional probes never alter the core latency measurements. +Alternatives Considered:: +* Require users to maintain their own histograms outside the harness :: +** *Pros:* No extra API surface in JLBH. +** *Cons:* More boilerplate; higher risk of inconsistent reporting or incompatible histogram settings. +* Fold sub-stage timings into the primary histogram :: +** *Pros:* Single distribution to manage. +** *Cons:* Hides distinctions between phases and makes diagnosis of slow sub-components difficult. +Rationale for Decision:: +* A dedicated probe API gives clear separation between overall latencies and sub-stage metrics, supporting richer analysis without complicating the main flow. +* `NanoSampler` provides a lightweight abstraction that integrates cleanly with the existing sampling model. +Impact & Consequences:: +* Each additional probe incurs extra memory and processing cost; users should add probes selectively for meaningful stages. +* Documentation and examples must show how to name probes consistently so that dashboards and CI jobs can rely on stable identifiers. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-OPS-006]] +=== [JL-OPS-006] CSV serialisation and result retention + +Date:: 2025-11-14 +Context:: +* Users need to archive benchmark results for later analysis, trend tracking and comparison across builds or environments. +* Plain console tables are useful for humans but awkward for automated tooling and external analytics systems. +* Requirements: FN-005 CSV Serialisation, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Expose an immutable `JLBHResult` model for completed runs and provide `JLBHResultSerializer` to write results to CSV with a stable schema. +* Default to a simple file layout suitable for CI artefacts, while allowing callers to direct output to custom streams or paths. +Alternatives Considered:: +* Rely solely on console output :: +** *Pros:* No extra types or wiring. +** *Cons:* Difficult to parse reliably; fragile across formatting changes. +* Use a binary format (for example compressed histograms) only :: +** *Pros:* More compact; potentially faster to write. +** *Cons:* Harder to inspect manually; requires custom tooling for every consumer. +Rationale for Decision:: +* CSV strikes a balance between human readability and machine processing, working well with spreadsheets, scripting languages and CI systems. +* A dedicated serializer centralises schema decisions so that downstream tools can depend on consistent column names and ordering. +Impact & Consequences:: +* Schema changes to CSV output must be versioned and documented carefully to avoid breaking consumers. +* Large or long-running benchmarks may produce sizeable CSV files; operations teams should manage retention and storage policies. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-007]] +=== [JL-OPS-007] CI integration via TeamCity statistics + +Date:: 2025-11-14 +Context:: +* Many users run JLBH benchmarks in CI and need to gate builds on latency regressions or track trends over time. +* TeamCity provides a native mechanism (`##teamcity` service messages) for publishing numerical statistics from builds. +* Requirements: Product Functions (CI Metrics), NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide helper code to emit key JLBH metrics (for example selected percentiles and throughput) as TeamCity statistics lines, using predictable metric names derived from benchmark configuration. +* Treat CI integration as an optional layer that can be enabled or disabled without affecting the core harness behaviour. +Alternatives Considered:: +* Expect users to write their own CI integration scripts around console output or CSV files :: +** *Pros:* Keeps the core library smaller; no explicit dependency on CI conventions. +** *Cons:* Duplicated effort across projects; higher risk of inconsistent parsing and metrics. +* Integrate tightly with a single CI provider only :: +** *Pros:* Deep integration for that platform. +** *Cons:* Locks users into a specific CI and complicates usage elsewhere. +Rationale for Decision:: +* A lightweight TeamCity helper covers a common use case without constraining users who rely on other CI platforms. +* Publishing statistics directly simplifies setting latency thresholds and visualising trends in existing dashboards. +Impact & Consequences:: +* CI pipelines can fail fast when configured thresholds are exceeded, turning JLBH results into actionable quality gates. +* Additional documentation must describe metric naming conventions so that teams can map them to their CI configuration. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] diff --git a/src/main/adoc/jlbh-cookbook.adoc b/src/main/adoc/jlbh-cookbook.adoc index fa91cc6d..acc38047 100644 --- a/src/main/adoc/jlbh-cookbook.adoc +++ b/src/main/adoc/jlbh-cookbook.adoc @@ -1,13 +1,14 @@ = Chronicle JLBH Cookbook Chronicle Software :toc: +:sectnums: :lang: en-GB :source-highlighter: rouge A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. These recipes aim to provide practical starting points for your own benchmarks. -== 1. Timing a Simple Method Call +== Timing a Simple Method Call This is the most basic use case: measuring the latency of a self-contained Java method. @@ -81,7 +82,7 @@ Avoid including unrelated logic within the timed section. If `myMethod()` is very fast, high throughput might be achievable. If it's slower, adjust throughput accordingly. -== 2. Measuring a Network Round Trip (UDP Loopback) +== Measuring a Network Round Trip (UDP Loopback) This recipe demonstrates timing a UDP message send and receive on loopback. @@ -220,7 +221,7 @@ public class MainApp { * Consider TCP for connection-oriented tests; its performance characteristics (e.g., Nagle's algorithm, ACK delays) differ from UDP. * Packet loss and retransmissions (for UDP, if implementing reliability) can heavily affect latency distributions. -== 3. Running Within an Existing Event Loop +== Running Within an Existing Event Loop JLBH can integrate with Chronicle's event loop mechanism, useful for benchmarking components that are already part of an event-driven architecture. @@ -311,7 +312,7 @@ public class MainApp { * `JLBHOptions.accountForCoordinatedOmission(true)` (the default) is essential for `eventLoopHandler` to function correctly. --- -== 4. Using Multiple Probes for Staged Operations +== Using Multiple Probes for Staged Operations JLBH allows timing multiple stages within a single benchmark iteration using custom probes. @@ -405,9 +406,9 @@ public class MainApp { * This approach is excellent for identifying bottlenecks within a larger operation. * The JLBH output will include separate percentile summaries for "Stage1_Processing", "Stage2_Persistence", and the default "end to end" probe. ---- +''' -== 5. Recording and Exporting Results +== Recording and Exporting Results JLBH allows programmatic access to results and provides utilities for serialization, such as to CSV. @@ -489,7 +490,7 @@ public class MainApp { * The `JLBHResult` object is immutable and contains rich data for all probes and runs. * `JLBHResultSerializer` provides a convenient way to get data out for external analysis. -== 6. Enabling and Understanding OS Jitter Measurement +== Enabling and Understanding OS Jitter Measurement Operating system jitter can significantly impact low-latency applications. JLBH can measure this independently. diff --git a/src/main/adoc/project-requirements.adoc b/src/main/adoc/project-requirements.adoc index 2600c7ab..54ba15e5 100644 --- a/src/main/adoc/project-requirements.adoc +++ b/src/main/adoc/project-requirements.adoc @@ -1,24 +1,24 @@ = Chronicle JLBH - Software Requirements Specification Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB +:source-highlighter: rouge *_Abstract_*:: Chronicle JLBH is an open-source Java library, released under the Apache Licence 2.0, that provides a high-resolution latency benchmark harness. It allows developers to _measure_, _analyse_ and _regress-test_ latency behaviour of critical code paths "in context" rather than via isolated micro-benchmarks. Key features include compensation for *co-ordinated omission*, configurable throughput modelling, additional probes, optional operating-system jitter tracking, and serialisation of results for CI dashboards. -== 1. Introduction +== Introduction -=== 1.1 Purpose +=== Purpose This document specifies functional and non-functional requirements for Chronicle JLBH, ensuring that contributors and integrators share a common understanding of the product's goals and constraints. -=== 1.2 Scope +=== Scope The harness targets _low-latency_ Java applications (trading systems, micro-services, real-time analytics) that must quantify latencies at extreme percentiles under realistic load patterns. -=== 1.3 Definitions, Acronyms and Abbreviations +=== Definitions, Acronyms and Abbreviations |=== | *Term* | *Definition* @@ -30,7 +30,7 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv | CI | Continuous Integration environment (e.g. GitHub Actions, TeamCity). |=== -=== 1.4 References +=== References * Upstream repository: https://github.com/OpenHFT/JLBH[GitHub - OpenHFT/JLBH] * API reference: https://javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/core/jlbh/JLBH.html[Javadoc] @@ -38,16 +38,16 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv * Discussion of _co-ordinated omission_: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE[m.s. thread] * Chronicle blog post on JLBH in event loops: https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html[Vanilla-Java article] -=== 1.5 Overview +=== Overview Sections 2-5 describe the product context, interfaces, detailed system features, and quality attributes. Section 6 captures licensing, whilst Section 7 presents the glossary. -== 2. Overall Description +== Overall Description -=== 2.1 Product Perspective +=== Product Perspective JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. -=== 2.2 Product Functions +=== Product Functions * Generate high-resolution histograms (>=35 bits) of end-to-end sample latencies. * Optionally compensate for *co-ordinated omission* by adjusting the synthetic start time. @@ -59,7 +59,7 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * Integrate with CI (TeamCity helper emits `##teamcity` statistics lines). * Allow execution from any thread or installation onto a Chronicle *MediumEventLoop*. -=== 2.3 User Classes and Characteristics +=== User Classes and Characteristics |=== | *Actor* | *Description* | *Technical Expertise* @@ -69,28 +69,28 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro | CI System | Runs automated latency regression suite, ingests CSV or TeamCity stats. | N/A (automation). |=== -=== 2.4 Operating Environment +=== Operating Environment * Java 11 LTS and later; Java 17 recommended for current builds. * Linux x86-64 (primary), other POSIX OSes supported; macOS usable but CI baselines exclude it. * Typical CPU affinity facilities available via *OpenHFT Affinity* library. -=== 2.5 Design and Implementation Constraints +=== Design and Implementation Constraints * Single-threaded harness execution; benchmarked code may span threads. * Measurements rely on `System.nanoTime()` precision; hardware timers must be invariant. * JVM must be started with `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when using async-profiler concurrently. * Library is released under the Apache Licence 2.0; contributions must comply with CLA policy. -=== 2.6 Assumptions and Dependencies +=== Assumptions and Dependencies * SUT jars are available on the same JVM classpath. * Benchmark host clocks are monotonic and not subjected to aggressive power-saving states. * CI agents provision exclusive CPU cores to minimise jitter. -== 3. External Interface Requirements +== External Interface Requirements -=== 3.1 Public API +=== Public API * `JLBHOptions` - builder for benchmark parameters (throughput, warm-up, runs, etc.). * `JLBHTask` - user-defined callback with lifecycle methods `init`, `run`, `warmedUp`, `runComplete`, `complete`. @@ -98,78 +98,84 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * `JLBHResult` - immutable projection of measured data. * `JLBHResultSerializer` - CSV writer for analytics pipelines. -=== 3.2 User Interface +=== User Interface CLI usage is demonstrated in _ExampleJLBHMain_ and test fixtures; no dedicated GUI is provided. -=== 3.3 Hardware Interfaces +=== Hardware Interfaces None. The harness interacts with OS timers and CPU affinity via JNI where available. -=== 3.4 Software Interfaces +=== Software Interfaces * Build tool: *Apache Maven* (pom.xml defines BOM imports). * Logging: SLF4J (test scope). * Testing: JUnit 4 for unit and integration tests. * CI Metrics: TeamCity service messages. -== 4. System Features +== System Features -=== 4.1 FN-001 Latency Sampling +=== FN-001 Latency Sampling *Description*: Collect nanosecond deltas from user code via `sample(long)` or `sampleNanos(long)`. *Stimulus/Response*: Upon each iteration, `JLBHTask.run` calls `jlbh.sample(...)`; histogram updates occur; run summary printed at completion. *Priority*: *Essential*. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. -=== 4.2 FN-002 Coordinated-Omission Compensation +=== FN-002 Coordinated-Omission Compensation Ensures that queue back-log does not under-represent tail latency. Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. +*Decision reference*: link:decision-log.adoc#JL-NF-P-003[JL-NF-P-003]. -=== 4.3 FN-003 Additional Probes +=== FN-003 Additional Probes Developers may time sub-stages (e.g., serialisation, network round-trip) through `addProbe(String)` and record them independently. +*Decision reference*: link:decision-log.adoc#JL-FN-005[JL-FN-005]. -=== 4.4 FN-004 OS Jitter Tracking +=== FN-004 OS Jitter Tracking A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. Results are summarised alongside core latencies. +*Decision reference*: link:decision-log.adoc#JL-OPS-004[JL-OPS-004]. -=== 4.5 FN-005 CSV Serialisation +=== FN-005 CSV Serialisation Post-run, results can be persisted for offline analysis (`result.csv` by default). +*Decision reference*: link:decision-log.adoc#JL-OPS-006[JL-OPS-006]. -=== 4.6 FN-006 Event Loop Integration +=== FN-006 Event Loop Integration `eventLoopHandler(EventLoop)` allows benchmarks to operate inside Chronicle threading framework, avoiding extra threads. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. -== 5. Non-Functional Requirements +== Non-Functional Requirements -=== 5.1 NF-P-001 Performance +=== NF-P-001 Performance * Overhead per sample must remain below 100 ns when no additional probes are active. * Histogram generation must support >=200 M iterations without heap pressure. -=== 5.2 NF-R-001 Reliability +=== NF-R-001 Reliability * Harness must abort gracefully on thread interruptions or sample time-outs (`timeout(long)`). * Immutable result objects ensure thread-safe publication to external consumers. -=== 5.3 NF-UX-001 Usability +=== NF-UX-001 Usability * Fluent builder API; sensible defaults provide a runnable benchmark in ≤10 LOC. * ASCII table outputs are human-readable and CI-friendly. -=== 5.4 NF-O-001 Portability +=== NF-O-001 Portability * Pure-Java codebase; no native compilation steps. * JDK-specific optimisations (e.g., *Zing* support) are runtime-detected. -=== 5.5 NF-O-002 Maintainability +=== NF-O-002 Maintainability * 80 %+ unit-test line coverage with deterministic fixtures. * Code adheres to Chronicle parent POM style and SonarCloud quality gates. -=== 5.6 NF-S-001 Security +=== NF-S-001 Security No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code. -== 6. Licensing +== Licensing The project is released under the *Apache Licence 2.0* (_see_ `LICENSE.adoc`). Downstream consumers must preserve licence notices and may include JLBH in commercial or OSS products, subject to the terms therein. -== 7. Glossary +== Glossary Co-ordinated Omission:: Statistical artefact causing under-reporting of worst-case latency. Histogram:: Data structure that records frequency counts in logarithmic buckets, enabling percentile extraction. Percentile:: Value below which a given percentage of observations fall (e.g., 99th percentile). -== 8. Appendix A - Example Minimal Benchmark +== Appendix A - Example Minimal Benchmark [source,java] ---- @@ -189,7 +195,7 @@ public class NothingBenchmark implements JLBHTask { } ---- -== 9. Appendix B - Footnotes +== Appendix B - Footnotes * JLBH originated within the Chronicle Software open-source stack and is actively maintained. See https://github.com/OpenHFT/JLBH. * The API reference highlights the focus on _co-ordinated omission_ and event-loop support. See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/adoc/results-interpretation-guide.adoc index e2886473..6adc1723 100644 --- a/src/main/adoc/results-interpretation-guide.adoc +++ b/src/main/adoc/results-interpretation-guide.adoc @@ -1,15 +1,14 @@ = JLBH Results Interpretation Guide Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB -:icons: font +:source-highlighter: rouge *_Abstract_*:: This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. Effective interpretation is key to deriving meaningful insights from your latency benchmarks. -== 1. Understanding JLBH Console Output Structure +== Understanding JLBH Console Output Structure When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). Understanding this output is the first step: @@ -26,7 +25,7 @@ When JLBH runs, it typically prints information to the console (or a configured ** Columns for each run, showing the latency value for that percentile in that run. ** A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). -== 2. Interpreting Latency Histograms and Percentiles +== Interpreting Latency Histograms and Percentiles JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. These histograms are then used to derive percentile values. @@ -45,7 +44,7 @@ JLBH uses histograms to capture the full distribution of measured latencies, rat * **Units**: Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. Pay attention to the units indicated in the table headers. * **Custom Probes**: If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. -== 3. Co-ordinated Omission (CO) +== Co-ordinated Omission (CO) Co-ordinated Omission is a critical concept in latency measurement, especially for systems that process requests at a certain rate. @@ -61,14 +60,14 @@ Co-ordinated Omission is a critical concept in latency measurement, especially f ** *Without CO compensation*: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. This can be useful for understanding raw compute time but not overall system responsiveness under load. * **Comparison**: Always state whether CO compensation was enabled when comparing benchmark results. Comparing a CO-compensated run with a non-compensated run is misleading. JLBH prints "Correcting for co-ordinated:true/false" in each run's output. -== 4. OS Jitter Probe +== OS Jitter Probe JLBH can optionally measure Operating System (OS) jitter, which refers to delays caused by the OS scheduler or other system activities interrupting the benchmark process. * **Mechanism**: ** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. ** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. * **Interpreting Jitter Output**: ** The "OS Jitter" probe will have its own percentile table in the JLBH output. ** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." @@ -81,7 +80,7 @@ JLBH can optionally measure Operating System (OS) jitter, which refers to delays ** Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). ** Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). -== 5. Analysing Run-to-Run Variation +== Analysing Run-to-Run Variation Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. JLBH's summary output includes a "% Variation" column for each percentile. @@ -97,7 +96,7 @@ Latency measurements can vary between successive runs of the same benchmark on t *** **Benchmarked Code Instability**: The code itself might have inherent variability or be interacting with unstable external resources. * **Goal**: Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. Focus on understanding systematic performance rather than chasing the single "best" score from one run. Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. -== 6. The Throughput vs. Latency Relationship +== The Throughput vs. Latency Relationship A fundamental aspect of performance is the trade-off between throughput (how much work is done per unit of time) and latency (how long each piece of work takes). @@ -111,7 +110,7 @@ A fundamental aspect of performance is the trade-off between throughput (how muc ** This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. ** The results can inform capacity planning and help set realistic performance expectations. -== 7. Common Pitfalls in Interpretation +== Common Pitfalls in Interpretation Avoiding these common mistakes can help ensure your benchmark results are meaningful and your conclusions are sound: @@ -143,7 +142,7 @@ Avoiding these common mistakes can help ensure your benchmark results are meanin ** *Issue*: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. ** *Solution*: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. This often means runs lasting at least several seconds, or even minutes for deep stability tests. -== 8. Deriving Actionable Insights +== Deriving Actionable Insights The ultimate goal of benchmarking is to gain insights that can lead to improvements. diff --git a/src/main/adoc/systemProperties.adoc b/src/main/adoc/systemProperties.adoc new file mode 100644 index 00000000..4ded78aa --- /dev/null +++ b/src/main/adoc/systemProperties.adoc @@ -0,0 +1,24 @@ += JLBH System Properties +:toc: +:lang: en-GB +:source-highlighter: rouge + +== System Properties + +JLBH is a micro-benchmark harness that is sensitive to any extra work done by the JVM. +It checks the `jvm.resource.tracing` property to avoid running when intrusive resource tracing is enabled. + +.System properties +[cols="2a,1,3a,2a",options="header"] +|=== +| Property Key +| Default Value +| Description +| Usage (Type) + +| `jvm.resource.tracing` +| not set or `false` +| When this property is set with an empty value or `true`, JLBH prints a warning and terminates the JVM rather than running the benchmark, because resource tracing would distort latency and allocation measurements. Ensure this property is unset or explicitly `false` when running JLBH. +| Checked in the `JLBH` constructor (`String` value parsed as `boolean`) +|=== + From fc367db88c011c304a9befa51f774b66e429f2f1 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 21 Nov 2025 20:11:15 +0000 Subject: [PATCH 02/14] Refactor JLBH classes to improve thread safety and code clarity --- .../java/net/openhft/chronicle/jlbh/JLBH.java | 40 ++++++++--------- .../openhft/chronicle/jlbh/JLBHOptions.java | 44 +++++++++---------- .../jlbh/util/JLBHResultSerializer.java | 6 ++- 3 files changed, 45 insertions(+), 45 deletions(-) diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java index 68df6b41..11d97b68 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java @@ -22,6 +22,7 @@ import java.util.concurrent.ConcurrentSkipListMap; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; import java.util.function.Consumer; import java.util.stream.Collectors; @@ -80,8 +81,7 @@ public class JLBH implements NanoSampler { private final AtomicBoolean abortTestRun = new AtomicBoolean(); private final long mod; private final long length; - // Todo: Remove all concurrent constructs such as volatile and AtomicBoolean - private volatile long noResultsReturned; + private final AtomicLong noResultsReturned = new AtomicLong(); //Use non-atomic when so thread synchronisation is necessary private boolean warmedUp; private volatile Thread testThread; @@ -116,8 +116,8 @@ public JLBH(@NotNull JLBHOptions jlbhOptions, @NotNull PrintStream printStream, final String resourceTracing = System.getProperty("jvm.resource.tracing"); if (resourceTracing != null && (resourceTracing.isEmpty() || Boolean.parseBoolean(resourceTracing))) { - System.out.println("***** WARNING : JLBH can not be run if jvm.resource.tracing=" + resourceTracing + ", please remove all \"jvm.resource.tracing\" as this will corrupt your stats *****"); - System.exit(-1); + throw new IllegalStateException("JLBH can not be run if jvm.resource.tracing=" + resourceTracing + + ", please remove all \"jvm.resource.tracing\" as this will corrupt your stats"); } this.jlbhOptions = jlbhOptions; @@ -133,8 +133,9 @@ public JLBH(@NotNull JLBHOptions jlbhOptions, @NotNull PrintStream printStream, : jlbhOptions.iterations > 50_000_000 ? 20_000_000_000L : jlbhOptions.iterations > 10_000_000 ? 10_000_000_000L : 5_000_000_000L; - long mod2; - for (mod2 = 1000; mod2 <= jlbhOptions.iterations / 200; mod2 *= 10) { + long mod2 = 1000; + while (mod2 <= jlbhOptions.iterations / 200) { + mod2 *= 10; } this.mod = mod2; } @@ -251,7 +252,6 @@ public void start() { } else { if (latencyBetweenTasks > 2e6) { - long end = System.nanoTime() + latencyBetweenTasks; Jvm.pause(latencyBetweenTasks / 1_000_000 - 1); // account for jitter in Thread.sleep() and wait until a fixed point in time startTimeNs = busyWaitUntil(startTimeNs); @@ -367,7 +367,7 @@ private long warmup() { private void endOfAllRuns() { printPercentilesSummary("end to end", percentileRuns, printStream); - if (additionalPercentileRuns.size() > 0) { + if (!additionalPercentileRuns.isEmpty()) { additionalPercentileRuns.forEach((label, percentileRuns1) -> printPercentilesSummary(label, percentileRuns1, printStream)); } @@ -413,13 +413,11 @@ private void endOfRun(int run, long runStart) { printStream.printf("%-48s", format("End to End: (%,d)", endToEndHistogram.totalCount())); printStream.println(endToEndHistogram.toMicrosFormat()); - if (additionHistograms.size() > 0) { + if (!additionHistograms.isEmpty()) { additionHistograms.forEach((key, value) -> { List ds = additionalPercentileRuns.computeIfAbsent(key, i -> new ArrayList<>()); ds.add(value.getPercentiles()); -// if (value.totalCount() != jlbhOptions.iterations) -// warning = " WARNING " + value.totalCount() + "!=" + jlbhOptions.iterations; printStream.printf("%-48s", format("%s (%,d)", key, value.totalCount())); printStream.println(value.toMicrosFormat()); }); @@ -432,7 +430,7 @@ private void endOfRun(int run, long runStart) { jlbhOptions.jlbhTask.runComplete(); - noResultsReturned = 0; + noResultsReturned.set(0); additionHistograms.values().forEach(Histogram::reset); endToEndHistogram.reset(); osJitterMonitor.reset(); @@ -445,8 +443,9 @@ private void checkSampleTimeout() { while (true) { Jvm.pause(TimeUnit.SECONDS.toMillis(10)); - if (previousSampleCount < noResultsReturned) { - previousSampleCount = noResultsReturned; + long currentSampleCount = noResultsReturned.get(); + if (previousSampleCount < currentSampleCount) { + previousSampleCount = currentSampleCount; previousSampleTime = System.currentTimeMillis(); } else { if (previousSampleTime < (System.currentTimeMillis() - jlbhOptions.timeout)) { @@ -529,9 +528,9 @@ public void printPercentilesSummary( .append("\n"); double[] percentiles = Histogram.percentilesFor(jlbhOptions.iterations); boolean skipFirst = percentiles.length > 3; - if (jlbhOptions.skipFirstRun == JLBHOptions.SKIP_FIRST_RUN.SKIP) { + if (jlbhOptions.skipFirstRun == JLBHOptions.SkipFirstRun.SKIP) { skipFirst = true; - } else if (jlbhOptions.skipFirstRun == JLBHOptions.SKIP_FIRST_RUN.NO_SKIP) { + } else if (jlbhOptions.skipFirstRun == JLBHOptions.SkipFirstRun.NO_SKIP) { skipFirst = false; } PercentileSummary percentileSummary = new PercentileSummary(skipFirst, percentileRuns, percentiles); @@ -581,7 +580,8 @@ private String formatPercentile(double percentile) { * @param pr formatted percentile label (e.g. {@code "99.9:"}) * @param runs number of run value placeholders to append */ - private void addPrToPrint(@NotNull StringBuilder sb, String pr, int runs) { + // Visible for tests via reflection to verify percentile summary patterns. + void addPrToPrint(@NotNull StringBuilder sb, String pr, int runs) { sb.append(pr); for (int i = 0; i < runs; i++) { sb.append("%12.2f "); @@ -646,12 +646,12 @@ public void sampleNanos(long durationNs) { * @param durationNs latency in nanoseconds */ public void sample(long durationNs) { - noResultsReturned++; - if (noResultsReturned < jlbhOptions.warmUpIterations && !warmedUp) { + long sampleIndex = noResultsReturned.incrementAndGet(); + if (sampleIndex < jlbhOptions.warmUpIterations && !warmedUp) { endToEndHistogram.sample(durationNs); return; } - if (noResultsReturned == jlbhOptions.warmUpIterations && !warmedUp) { + if (sampleIndex == jlbhOptions.warmUpIterations && !warmedUp) { warmedUp = true; endToEndHistogram.reset(); if (!additionHistograms.isEmpty()) { diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java index ecb9b397..243de104 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java @@ -31,7 +31,7 @@ public class JLBHOptions { JLBHTask jlbhTask; int pauseAfterWarmupMS = 0; @NotNull - SKIP_FIRST_RUN skipFirstRun = SKIP_FIRST_RUN.NOT_SET; + SkipFirstRun skipFirstRun = SkipFirstRun.NOT_SET; boolean jitterAffinity; Supplier acquireLock = Affinity::acquireLock; long timeout; @@ -164,8 +164,8 @@ public JLBHOptions iterations(long iterations) { * @return Instance of the JLBHOptions to be used in the builder pattern. */ @NotNull - public JLBHOptions jlbhTask(JLBHTask JLBHTask) { - this.jlbhTask = JLBHTask; + public JLBHOptions jlbhTask(JLBHTask jlbhTask) { + this.jlbhTask = jlbhTask; return this; } @@ -189,7 +189,7 @@ public JLBHOptions pauseAfterWarmupMS(int pauseMS) { */ @NotNull public JLBHOptions skipFirstRun(boolean skip) { - skipFirstRun = skip ? SKIP_FIRST_RUN.SKIP : SKIP_FIRST_RUN.NO_SKIP; + skipFirstRun = skip ? SkipFirstRun.SKIP : SkipFirstRun.NO_SKIP; return this; } @@ -235,24 +235,22 @@ public JLBHOptions timeout(long timeout) { @Override public String toString() { - final StringBuffer sb = new StringBuffer("JLBHOptions{"); - sb.append("runs=").append(runs); - sb.append(", iterations=").append(iterations); - sb.append(", warmUpIterations=").append(warmUpIterations); - sb.append(", pauseAfterWarmupMS=").append(pauseAfterWarmupMS); - sb.append(", accountForCoordinatedOmission=").append(accountForCoordinatedOmission); - sb.append(", skipFirstRun=").append(skipFirstRun); - sb.append(", recordOSJitter=").append(recordOSJitter); - sb.append(", recordJitterGreaterThanNs=").append(recordJitterGreaterThanNs); - sb.append(", throughput=").append(throughput); - sb.append(", throughputTimeUnit=").append(throughputTimeUnit); - sb.append(", latencyDistributor=").append(latencyDistributor); - sb.append(", jitterAffinity=").append(jitterAffinity); - sb.append(", timeout=").append(timeout); - sb.append(", jlbhTask=").append(jlbhTask); - sb.append(", acquireLock=").append(acquireLock); - sb.append('}'); - return sb.toString(); + return "JLBHOptions{" + "runs=" + runs + + ", iterations=" + iterations + + ", warmUpIterations=" + warmUpIterations + + ", pauseAfterWarmupMS=" + pauseAfterWarmupMS + + ", accountForCoordinatedOmission=" + accountForCoordinatedOmission + + ", skipFirstRun=" + skipFirstRun + + ", recordOSJitter=" + recordOSJitter + + ", recordJitterGreaterThanNs=" + recordJitterGreaterThanNs + + ", throughput=" + throughput + + ", throughputTimeUnit=" + throughputTimeUnit + + ", latencyDistributor=" + latencyDistributor + + ", jitterAffinity=" + jitterAffinity + + ", timeout=" + timeout + + ", jlbhTask=" + jlbhTask + + ", acquireLock=" + acquireLock + + '}'; } /** @@ -266,7 +264,7 @@ public String toString() { *
  • {@link #NO_SKIP} - always include the first run.
    • * */ - enum SKIP_FIRST_RUN { + enum SkipFirstRun { NOT_SET, SKIP, NO_SKIP } } diff --git a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java index 1d4f35f7..5ff498f1 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java +++ b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java @@ -3,12 +3,12 @@ */ package net.openhft.chronicle.jlbh.util; +import java.io.OutputStreamWriter; import net.openhft.chronicle.jlbh.JLBHResult; import org.jetbrains.annotations.NotNull; import java.io.BufferedWriter; import java.io.IOException; -import java.io.PrintWriter; import java.io.Writer; import java.nio.file.Files; import java.nio.file.Paths; @@ -16,6 +16,8 @@ import java.util.Collections; import java.util.Optional; +import static java.nio.charset.StandardCharsets.ISO_8859_1; + /** * Utility class that writes the output of a {@link net.openhft.chronicle.jlbh.JLBH} * run to a CSV file. @@ -98,7 +100,7 @@ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, String * @throws IOException if the file cannot be written */ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, Iterable namesOfProbes, boolean includeOSJitter) throws IOException { - try (Writer pw = new BufferedWriter(new PrintWriter(Files.newOutputStream(Paths.get(fileName))))) { + try (Writer pw = new BufferedWriter(new OutputStreamWriter(Files.newOutputStream(Paths.get(fileName)), ISO_8859_1))) { writeHeader(pw); JLBHResult.ProbeResult probeResult = jlbhResult.endToEnd(); From 1219bfb54d9fdbb66212e23940cced9f82d13598 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 21 Nov 2025 20:11:23 +0000 Subject: [PATCH 03/14] Refactor JLBH classes for improved UTF-8 support and code clarity --- pom.xml | 2 +- .../chronicle/jlbh/ExampleJLBHMain.java | 6 +- .../jlbh/JLBHAdditionalCoverageTest.java | 6 +- .../jlbh/JLBHDeterministicFixtures.java | 5 +- .../chronicle/jlbh/JLBHIntegrationTest.java | 15 ++++- .../chronicle/jlbh/JLBHOptionsTest.java | 4 +- .../net/openhft/chronicle/jlbh/JLBHTest.java | 56 +++++++++++++++---- .../chronicle/jlbh/SimpleBenchmark.java | 10 ++-- .../jlbh/SimpleOSJitterBenchmark.java | 7 ++- .../jlbh/SimpleResultSerializerBenchmark.java | 16 +++--- 10 files changed, 87 insertions(+), 40 deletions(-) diff --git a/pom.xml b/pom.xml index 9c7aaead..dcd23499 100644 --- a/pom.xml +++ b/pom.xml @@ -31,7 +31,7 @@ net.openhft third-party-bom - 3.27ea5 + 3.27ea7 pom import diff --git a/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java b/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java index 23c9a230..da46681c 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java +++ b/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java @@ -26,7 +26,6 @@ */ public class ExampleJLBHMain implements JLBHTask { private int count = 0; - private double sin; //private NanoSampler nanoSamplerSin; //private NanoSampler nanoSamplerWait; private JLBH lth; @@ -46,13 +45,10 @@ public void run(long startTimeNS) { count++; if (count == 160_000) { System.out.println("PAUSE"); - //long now = System.nanoTime(); Jvm.pause(100); - //nanoSamplerWait.sampleNanos(System.nanoTime()-now); } - long now = System.nanoTime(); - sin = Math.sin(count); + Jvm.safepoint(); // used to keep Math.sin side-effect free path removed //nanoSamplerSin.sampleNanos(System.nanoTime()-now); lth.sample(System.nanoTime() - startTimeNS); diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java index 01e8fe1c..f7ca5c25 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java @@ -104,7 +104,11 @@ private static JLBHOptions newHarness(JLBHTask task) { } private static PrintStream silentPrintStream() { - return new PrintStream(new ByteArrayOutputStream()); + try { + return new PrintStream(new ByteArrayOutputStream(), true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } private static final class AbortOnRunTask implements JLBHTask { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java index e4b181ec..a183671b 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java @@ -11,7 +11,7 @@ public class JLBHDeterministicFixtures { static final int ITERATIONS = 9_000; private static final int THROUGHPUT = 1_000_000; private static final int RUNS = 3; - private final static String expectedOutput = "" + + private static final String expectedOutput = "" + "Warm up complete ...\n" + "-------------------------------- BENCHMARK RESULTS (RUN 1) us --------------------------------------\n" + "Run time: ...s, distribution: NORMAL\n" + @@ -89,14 +89,13 @@ static String withoutNonDeterministicFields(String content) { static class PredictableJLBHTask implements JLBHTask { int nanoTime = 1_000_000; - private int latency; private JLBH lth; private NanoSampler additionalSamplerA; private NanoSampler additionalSamplerB; @Override public void run(long startTimeNS) { - latency = 1000 + (++this.nanoTime % 11567); + int latency = 1000 + (++this.nanoTime % 11567); lth.sample(latency); additionalSamplerA.sampleNanos(latency - 1000); if (sampleB()) diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java index 74780255..29d3e44f 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java @@ -47,7 +47,12 @@ public void shouldMeasureLatency() { jlbh.start(); // then - String stdOut = outContent.toString(); + String stdOut; + try { + stdOut = outContent.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } resetSystemOut(); assertThat(stdOut, containsString("OS Jitter")); assertThat(stdOut, containsString("Warm up complete (500 iterations took ")); @@ -59,8 +64,12 @@ public void shouldMeasureLatency() { } private void redirectSystemOut() { - System.setOut(new PrintStream(outContent)); - System.setErr(new PrintStream(errContent)); + try { + System.setOut(new PrintStream(outContent, true, "UTF-8")); + System.setErr(new PrintStream(errContent, true, "UTF-8")); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } private void resetSystemOut() { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java index 4c8063a0..91cbff22 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java @@ -49,7 +49,7 @@ public void shouldApplyAllConfigurationOptions() throws Exception { assertEquals(1_000_000L, getField(options, "iterations", Long.class).longValue()); assertSame(task, getField(options, "jlbhTask", JLBHTask.class)); assertEquals(77, getField(options, "pauseAfterWarmupMS", Integer.class).intValue()); - assertEquals(JLBHOptions.SKIP_FIRST_RUN.SKIP, getField(options, "skipFirstRun", JLBHOptions.SKIP_FIRST_RUN.class)); + assertEquals(JLBHOptions.SkipFirstRun.SKIP, getField(options, "skipFirstRun", JLBHOptions.SkipFirstRun.class)); assertTrue(getField(options, "jitterAffinity", Boolean.class)); assertSame(customSupplier, getField(options, "acquireLock", Supplier.class)); assertEquals(9876L, getField(options, "timeout", Long.class).longValue()); @@ -63,7 +63,7 @@ public void shouldApplyAllConfigurationOptions() throws Exception { @Test public void shouldRespectSkipFirstRunFalse() throws Exception { JLBHOptions options = new JLBHOptions().skipFirstRun(false); - assertEquals(JLBHOptions.SKIP_FIRST_RUN.NO_SKIP, getField(options, "skipFirstRun", JLBHOptions.SKIP_FIRST_RUN.class)); + assertEquals(JLBHOptions.SkipFirstRun.NO_SKIP, getField(options, "skipFirstRun", JLBHOptions.SkipFirstRun.class)); } private static T getField(JLBHOptions options, String name, Class type) throws Exception { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java index 97fff900..cf6bc938 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java @@ -15,7 +15,6 @@ import org.junit.runners.Parameterized; import java.io.ByteArrayOutputStream; -import java.io.OutputStream; import java.io.PrintStream; import java.util.ArrayList; import java.util.Arrays; @@ -63,14 +62,25 @@ private void start(JLBH jlbh) { public void shouldWriteResultToTheOutputProvided() { // given - final OutputStream outputStream = new ByteArrayOutputStream(); - final JLBH jlbh = new JLBH(options(), new PrintStream(outputStream), resultConsumer()); + final ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); + final PrintStream ps; + try { + ps = new PrintStream(outputStream, true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } + final JLBH jlbh = new JLBH(options(), ps, resultConsumer()); // when start(jlbh); // then - String result = outputStream.toString().replace("\r", ""); + String result; + try { + result = outputStream.toString("UTF-8").replace("\r", ""); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } assertThat(result, containsString("OS Jitter")); assertThat(result, containsString("Warm up complete (500 iterations took ")); assertThat(result, containsString("Run time: ")); @@ -138,7 +148,6 @@ public void shouldProvideResultData() { assertEquals(5_106L, probeALastRunSummary.get50thPercentile().toNanos(), 20); assertEquals(8_708L, probeALastRunSummary.get90thPercentile().toNanos(), 30); assertEquals(9_516L, probeALastRunSummary.get99thPercentile().toNanos(), 30); -// assertEquals(9_604L, probeALastRunSummary.get9999thPercentile().toNanos()); assertEquals(9_604L, probeALastRunSummary.getWorst().toNanos(), 30); assertEquals(probeALastRunSummary.get50thPercentile(), probeALastRunSummary.percentiles().get(PERCENTILE_50TH)); assertEquals(probeALastRunSummary.get90thPercentile(), probeALastRunSummary.percentiles().get(PERCENTILE_90TH)); @@ -187,10 +196,20 @@ public void teamCityHelper() { // then final ByteArrayOutputStream baos = new ByteArrayOutputStream(); - try (final PrintStream printStream = new PrintStream(baos)) { - TeamCityHelper.teamCityStatsLastRun("prefix", jlbh, jlbhOptions.iterations, printStream); + try { + try (final PrintStream printStream = new PrintStream(baos, true, "UTF-8")) { + TeamCityHelper.teamCityStatsLastRun("prefix", jlbh, jlbhOptions.iterations, printStream); + } + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); } String extra = Jvm.isAzulZing() ? ".zing" : Jvm.isJava15Plus() ? ".java17" : ""; + String stats; + try { + stats = baos.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } assertEquals("##teamcity[buildStatisticValue key='prefix.end-to-end.0.5" + extra + "' value='8.072']\n" + "##teamcity[buildStatisticValue key='prefix.end-to-end.0.9" + extra + "' value='11.664']\n" + "##teamcity[buildStatisticValue key='prefix.end-to-end.0.99" + extra + "' value='12.464']\n" + @@ -204,7 +223,7 @@ public void teamCityHelper() { "##teamcity[buildStatisticValue key='prefix.B.0.5" + extra + "' value='0.100125']\n" + "##teamcity[buildStatisticValue key='prefix.B.0.9" + extra + "' value='0.100125']\n" + "##teamcity[buildStatisticValue key='prefix.B.0.99" + extra + "' value='0.100125']\n" + - "##teamcity[buildStatisticValue key='prefix.B.1.0" + extra + "' value='0.100125']\n", baos.toString().replace("\r", "")); + "##teamcity[buildStatisticValue key='prefix.B.1.0" + extra + "' value='0.100125']\n", stats.replace("\r", "")); } @Test @@ -212,14 +231,25 @@ public void histogramSummariesAreCorrect() { final JLBHResultConsumer resultConsumer = resultConsumer(); JLBHOptions jlbhOptions = options().jlbhTask(new PredictableJLBHTaskDifferentShape()).iterations(ITERATIONS * 2); final ByteArrayOutputStream baos = new ByteArrayOutputStream(); - final PrintStream printStream = new PrintStream(baos); + final PrintStream printStream; + try { + printStream = new PrintStream(baos, true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } final JLBH jlbh = new JLBH(jlbhOptions, printStream, resultConsumer); // when start(jlbh); System.out.println(baos); - assertTrue(baos.toString().replace("\r", "").contains( + String summary; + try { + summary = baos.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } + assertTrue(summary.replace("\r", "").contains( "-------------------------------- SUMMARY (B) us ----------------------------------------------------\n" + "Percentile run1 run2 run3 % Variation\n" + "50.0: 0.10 0.10 0.10 0.00\n" + @@ -307,6 +337,10 @@ private JLBHResultConsumer resultConsumer() { @NotNull private PrintStream printStream() { - return new PrintStream(new ByteArrayOutputStream()); + try { + return new PrintStream(new ByteArrayOutputStream(), true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java index 55988da6..d4d4a9f7 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java @@ -15,7 +15,8 @@ public static void main(String[] args) { .warmUpIterations(20_000) .iterations(1_000_000) .throughput(100_000) -// .accountForCoordinatedOmission(true) + // enable this if you need to compensate for coordinated omission + // .accountForCoordinatedOmission(true) .runs(2) .jlbhTask(new SimpleBenchmark()); new JLBH(lth).start(); @@ -28,11 +29,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to include work done inside this method. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); } } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java index a64b46d2..a611aacd 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java @@ -43,11 +43,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to measure the local work. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); myProbe.sampleNanos(delta); } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java index 3baebd05..b01b60f0 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java @@ -13,15 +13,16 @@ public class SimpleResultSerializerBenchmark implements JLBHTask { private JLBH jlbh; public static void main(String[] args) { - //Create the JLBH options you require for the benchmark + // Create the JLBH options you require for the benchmark JLBHOptions lth = new JLBHOptions() .warmUpIterations(20_000) .iterations(1_000_000) .throughput(100_000) -// .accountForCoordinatedOmission(true) + // enable this if you need to compensate for coordinated omission + // .accountForCoordinatedOmission(true) .runs(2) - .jlbhTask(new SimpleResultSerializerBenchmark() ); - new JLBH(lth, System.out,jlbhResult -> { + .jlbhTask(new SimpleResultSerializerBenchmark()); + new JLBH(lth, System.out, jlbhResult -> { try { System.out.println("Serializing result..."); JLBHResultSerializer.runResultToCSV(jlbhResult); @@ -40,11 +41,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to measure the local work. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); } } From 7db0935074fbad68a37e3ddac1c5393921940eda Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 21 Nov 2025 20:36:24 +0000 Subject: [PATCH 04/14] Add CLAUDE.md for project guidance and documentation structure --- CLAUDE.md | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..c2935bac --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,219 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +JLBH (Java Latency Benchmark Harness) is a specialised benchmarking library for measuring end-to-end latency in Java applications under realistic throughput conditions. Unlike JMH micro-benchmarks, JLBH is designed to benchmark code "in context" and account for coordinated omission. It is particularly suited for producer/consumer workloads where iteration start and completion may occur on different threads. + +## Build & Test Commands + +```bash +# Verify build and run all tests +mvn -q verify + +# Clean build from scratch +mvn -q clean verify + +# Run a specific test class +mvn -q test -Dtest=JLBHTest + +# Run a single test method +mvn -q test -Dtest=JLBHTest#shouldProvideResultData + +# Compile only (skip tests) +mvn -q compile + +# Skip tests during verify +mvn -q verify -DskipTests +``` + +## Language & Character-Set Requirements + +**Critical**: All code and documentation must use: +- **British English** spelling (`organisation`, `licence`) except for Java keywords like `synchronized` +- **ISO-8859-1** character encoding only (code points 0-255) +- **No Unicode characters**: Use ASCII equivalents like `>=` instead of special symbols +- Verify with: `iconv -f ascii -t ascii ` + +See AGENTS.md for complete language policy and rationale. + +## Architecture Overview + +### Core Components + +**JLBH** (src/main/java/net/openhft/chronicle/jlbh/JLBH.java) +- The main orchestrator that manages the entire benchmark lifecycle +- Runs on a single thread (annotated `@SingleThreaded`) +- Provides `sample(long durationNs)` to record end-to-end latencies +- Provides `addProbe(String)` to create additional `NanoSampler` instances for measuring sub-stages +- Can run standalone via `start()` or integrate with Chronicle EventLoop via `eventLoopHandler(EventLoop)` + +**JLBHOptions** (src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java) +- Builder-style configuration object defining all benchmark parameters +- Key settings: `throughput()`, `iterations()`, `runs()`, `warmUpIterations()`, `accountForCoordinatedOmission()`, `recordOSJitter()` +- Default throughput: 10,000 ops/sec; default runs: 3 + +**JLBHTask** (src/main/java/net/openhft/chronicle/jlbh/JLBHTask.java) +- User-implemented interface defining the workload to benchmark +- Lifecycle methods called by JLBH: + - `init(JLBH)`: Setup phase, register probes + - `run(long startTimeNs)`: Execute benchmark iteration (called for each iteration) + - `warmedUp()`: Notification after warmup completes + - `runComplete()`: Notification after each run + - `complete()`: Final cleanup + +**Histograms & Results** +- Each probe (end-to-end, custom probes, OS jitter) records samples into a `Histogram` from chronicle-core +- Histograms use logarithmic bucketing with 35-bit range and 8 significant figures +- Immutable result objects: `JLBHResult`, `ProbeResult`, `RunResult` +- Use `ThreadSafeJLBHResultConsumer` to retrieve results from another thread + +### Threading Model + +- **Single-threaded harness**: All JLBHTask lifecycle methods run on the single JLBH thread +- **User code may spawn threads**: The benchmarked code within `JLBHTask.run()` can be multi-threaded +- **Sample recording thread-safety**: Only the JLBH harness thread should call `jlbh.sample()` or probe samplers +- **Event loop integration**: Via `eventLoopHandler()` instead of `start()` (requires coordinated omission accounting enabled) + +### Execution Flow + +1. **Initialization**: User configures `JLBHOptions`, creates `JLBH` instance, calls `JLBHTask.init()` +2. **Warm-up**: Runs `warmUpIterations` times to allow JIT compilation (default ~12k iterations) +3. **Measurement runs**: Executes `runs` times (default 3), each with `iterations` samples +4. **Completion**: Prints summary, constructs immutable `JLBHResult`, calls `JLBHTask.complete()` + +See src/main/adoc/benchmark-lifecycle.adoc for visual flow diagram. + +### Coordinated Omission + +JLBH accounts for coordinated omission by default (`accountForCoordinatedOmission = true`): +- `startTimeNs` passed to `run()` is the *calculated ideal start time*, not `System.nanoTime()` +- Harness busy-waits if current time is before scheduled start time +- Disabling this feature means start time is simply based on throughput without waiting + +### OS Jitter Monitoring + +Enabled by default via `recordOSJitter(true)`: +- Background thread `OSJitterMonitor` repeatedly samples `System.nanoTime()` to detect scheduler delays +- Records jitter > `recordJitterGreaterThanNs` (default: 1000ns) into separate histogram +- Incurs overhead; disable with `recordOSJitter(false)` for minimal-overhead benchmarks + +## Key Design Patterns + +**Probes for Multi-Stage Benchmarks** +```java +class MyTask implements JLBHTask { + private JLBH jlbh; + private NanoSampler stage1Probe; + + public void init(JLBH jlbh) { + this.jlbh = jlbh; + this.stage1Probe = jlbh.addProbe("stage1"); + } + + public void run(long startTimeNs) { + long stage1Start = System.nanoTime(); + // ... stage 1 work ... + stage1Probe.sampleNanos(System.nanoTime() - stage1Start); + + // ... remaining work ... + jlbh.sample(System.nanoTime() - startTimeNs); + } +} +``` + +**Typical Benchmark Setup** +```java +JLBHOptions options = new JLBHOptions() + .warmUpIterations(100_000) + .iterations(1_000_000) + .throughput(50_000) + .runs(3) + .jlbhTask(myTask); +new JLBH(options).start(); +``` + +**Thread-Safe Result Retrieval** +```java +JLBHResultConsumer resultConsumer = JLBHResultConsumer.newThreadSafeInstance(); +JLBH jlbh = new JLBH(options, System.out, resultConsumer); +// Run on separate thread +new Thread(() -> jlbh.start()).start(); +// Retrieve results from main thread after completion +JLBHResult result = resultConsumer.get(); +``` + +## Documentation Structure + +All requirements and design decisions are in src/main/adoc/: +- **project-requirements.adoc**: Formal requirements specification with tagged requirements (FN-xxx, NF-P-xxx, etc.) +- **decision-log.adoc**: Architectural decision records (ADRs) following Nine-Box taxonomy +- **architecture.adoc**: High-level architecture, components, execution flow, threading model +- **benchmark-lifecycle.adoc**: Visual diagram of benchmark execution phases +- **jlbh-cookbook.adoc**: Common usage patterns and recipes +- **results-interpretation-guide.adoc**: How to interpret percentile output + +When making changes: +1. Update relevant .adoc files first (if changing requirements/design) +2. Update code and tests +3. Verify all three stay synchronised in same commit + +## Javadoc Standards + +Follow guidelines from AGENTS.md: +- Document *behavioural contracts*, edge cases, thread-safety, units, performance characteristics +- Do NOT restate the obvious ("Gets the value") +- First sentence must be concise (becomes summary line) +- Remove autogenerated Javadoc for trivial getters/setters +- Explain *why* and *how*, not just *what* + +## Test Examples + +Test files in src/test/java/net/openhft/chronicle/jlbh/ demonstrate usage: +- `ExampleJLBHMain`: Basic command-line harness demonstration +- `SimpleBenchmark`: Minimal benchmark example +- `SimpleOSJitterBenchmark`: Demonstrates OS jitter recording and custom probes +- `JLBHTest`: Unit test showing programmatic result extraction +- `JLBHIntegrationTest`: Example integration test + +## Performance Requirements + +From project-requirements.adoc (NF-P requirements): +- Overhead per sample: < 100 ns when no additional probes active +- Histogram generation: Support >= 200M iterations without heap pressure +- Warm-up rule of thumb: ~30% of run iterations (default uses JIT compile threshold * 6/5) + +## Dependencies + +Key dependencies from pom.xml: +- **chronicle-core**: Provides `Histogram`, `NanoSampler`, threading utilities +- **affinity**: CPU affinity control via `AffinityLock` +- **chronicle-threads**: Event loop integration (test scope) + +## Common Gotchas + +- JLBH is `@SingleThreaded` - the harness itself runs on one thread +- The `startTimeNs` parameter to `run()` is NOT `System.nanoTime()` - it's the calculated ideal start time +- Warm-up iterations use ~12k by default; adjust with `warmUpIterations()` if benchmark takes long to stabilise +- OS jitter monitoring is enabled by default and adds overhead; explicitly disable if needed +- When using `eventLoopHandler()`, coordinated omission must remain enabled +- CSV export: Use `JLBHResultSerializer.runResultToCSV(jlbhResult)` (writes to `result.csv` by default) + +## Non-Functional Quality Attributes + +From README.adoc summary: +- **Performance**: < 100 ns overhead per sample, >= 200M iterations supported +- **Reliability**: Graceful abort on interruption/timeout, immutable thread-safe results +- **Usability**: Fluent API, runnable in <= 10 LOC, human-readable ASCII output +- **Portability**: Pure Java, runtime-detected JDK optimisations +- **Maintainability**: >= 80% test coverage (current: ~83% line, ~78% branch per pom.xml) + +## Commit & PR Guidelines + +From AGENTS.md: +- Subject line <= 72 chars, imperative mood +- Body: root cause -> fix -> measurable impact +- Run `mvn -q verify` before opening PR +- Keep PRs focused; avoid bundling unrelated changes +- Re-run build after addressing review comments From 3fdfc8f80ddf33a5bf264b1767dffd87f1b4d0bf Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 14 Nov 2025 13:48:24 +0000 Subject: [PATCH 05/14] Add documentation for JLBH system properties --- systemProperties.adoc | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 systemProperties.adoc diff --git a/systemProperties.adoc b/systemProperties.adoc new file mode 100644 index 00000000..4ded78aa --- /dev/null +++ b/systemProperties.adoc @@ -0,0 +1,24 @@ += JLBH System Properties +:toc: +:lang: en-GB +:source-highlighter: rouge + +== System Properties + +JLBH is a micro-benchmark harness that is sensitive to any extra work done by the JVM. +It checks the `jvm.resource.tracing` property to avoid running when intrusive resource tracing is enabled. + +.System properties +[cols="2a,1,3a,2a",options="header"] +|=== +| Property Key +| Default Value +| Description +| Usage (Type) + +| `jvm.resource.tracing` +| not set or `false` +| When this property is set with an empty value or `true`, JLBH prints a warning and terminates the JVM rather than running the benchmark, because resource tracing would distort latency and allocation measurements. Ensure this property is unset or explicitly `false` when running JLBH. +| Checked in the `JLBH` constructor (`String` value parsed as `boolean`) +|=== + From c457b53440b23463df5bfb192bf38fe2750dd77c Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 14 Nov 2025 14:03:49 +0000 Subject: [PATCH 06/14] Updated documentation --- src/main/adoc/architecture.adoc | 1 + src/main/adoc/benchmark-lifecycle.adoc | 4 ++-- src/main/adoc/jlbh-cookbook.adoc | 1 + src/main/adoc/project-requirements.adoc | 4 +++- src/main/adoc/results-interpretation-guide.adoc | 7 +++++-- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/main/adoc/architecture.adoc b/src/main/adoc/architecture.adoc index cd223d26..2a40176e 100644 --- a/src/main/adoc/architecture.adoc +++ b/src/main/adoc/architecture.adoc @@ -4,6 +4,7 @@ Chronicle Software :sectnums: :lang: en-GB :source-highlighter: rouge +:sectnums: == Introduction diff --git a/src/main/adoc/benchmark-lifecycle.adoc b/src/main/adoc/benchmark-lifecycle.adoc index afa893b3..aaac9445 100644 --- a/src/main/adoc/benchmark-lifecycle.adoc +++ b/src/main/adoc/benchmark-lifecycle.adoc @@ -1,8 +1,8 @@ = Benchmark Lifecycle -:toc: -:sectnums: :lang: en-GB :source-highlighter: rouge +:toc: +:sectnums: [mermaid] ---- diff --git a/src/main/adoc/jlbh-cookbook.adoc b/src/main/adoc/jlbh-cookbook.adoc index acc38047..47749eac 100644 --- a/src/main/adoc/jlbh-cookbook.adoc +++ b/src/main/adoc/jlbh-cookbook.adoc @@ -7,6 +7,7 @@ Chronicle Software A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. These recipes aim to provide practical starting points for your own benchmarks. +:sectnums: == Timing a Simple Method Call diff --git a/src/main/adoc/project-requirements.adoc b/src/main/adoc/project-requirements.adoc index 54ba15e5..86a10845 100644 --- a/src/main/adoc/project-requirements.adoc +++ b/src/main/adoc/project-requirements.adoc @@ -1,9 +1,11 @@ = Chronicle JLBH - Software Requirements Specification Chronicle Software +:revnumber: 1.0 +:revdate: 23 May 2025 :toc: -:sectnums: :lang: en-GB :source-highlighter: rouge +:sectnums: *_Abstract_*:: Chronicle JLBH is an open-source Java library, released under the Apache Licence 2.0, that provides a high-resolution latency benchmark harness. diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/adoc/results-interpretation-guide.adoc index 6adc1723..fb78c3c0 100644 --- a/src/main/adoc/results-interpretation-guide.adoc +++ b/src/main/adoc/results-interpretation-guide.adoc @@ -1,8 +1,11 @@ = JLBH Results Interpretation Guide Chronicle Software +:revnumber: 1.0 +:revdate: 23 May 2025 :toc: -:sectnums: :lang: en-GB +:icons: font +:sectnums: :source-highlighter: rouge *_Abstract_*:: @@ -67,7 +70,7 @@ JLBH can optionally measure Operating System (OS) jitter, which refers to delays * **Mechanism**: ** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. ** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. * **Interpreting Jitter Output**: ** The "OS Jitter" probe will have its own percentile table in the JLBH output. ** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." From 443b978f97504688222082469772377f54e26bf5 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 14 Nov 2025 15:09:03 +0000 Subject: [PATCH 07/14] Enhance documentation with British English conventions and formatting updates --- AGENTS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index dc1a9e9f..eb499013 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,9 +11,9 @@ LLM-based agents can accelerate development only if they respect our house rules | Requirement | Rationale | |--------------|-----------| | **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the [University of Oxford style guide](https://www.ox.ac.uk/public-affairs/style-guide) for reference. | -| **ISO-8859-1** (code-points 0-255). Avoid smart quotes, non-breaking spaces and accented characters. | ISO-8859-1 survives every toolchain Chronicle uses. | -| If a symbol is not available in ISO-8859-1, use a textual form such as `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | -| Tools to check ASCII compliance include `iconv -f ascii -t ascii` and IDE settings that flag non-ASCII characters. | These help catch stray Unicode characters before code review. | +| **ISO-8859-1 (Latin-1) only** (code-points 0-255). Avoid smart quotes, non-breaking spaces and characters outside this range. | ISO-8859-1 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect predictable 8-bit characters. | +| If a symbol is not available in ISO-8859-1, use a textual form such as `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode outside Latin-1 must not be inserted. | Extended encodings outside ISO-8859-1 are *not* portable and are therefore disallowed. | +| Tools to check ISO-8859-1 compliance include `iconv -f ISO-8859-1 -t ISO-8859-1` and IDE settings that flag characters outside Latin-1. | These help catch stray characters before code review. | ## Javadoc guidelines From 93e18d3a1791d5ab8fd8a5ea6b82c3ce3b2d5339 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Sun, 16 Nov 2025 15:45:11 +0000 Subject: [PATCH 08/14] Update documentation --- src/main/adoc/results-interpretation-guide.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/adoc/results-interpretation-guide.adoc index fb78c3c0..bf4614c7 100644 --- a/src/main/adoc/results-interpretation-guide.adoc +++ b/src/main/adoc/results-interpretation-guide.adoc @@ -70,7 +70,7 @@ JLBH can optionally measure Operating System (OS) jitter, which refers to delays * **Mechanism**: ** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. ** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. * **Interpreting Jitter Output**: ** The "OS Jitter" probe will have its own percentile table in the JLBH output. ** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." From b8cd0f23762d673f4b76eb25fbba5a114049583f Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Mon, 17 Nov 2025 21:53:25 +0000 Subject: [PATCH 09/14] Add SpotBugs profile and enhance development guidelines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit introduces static analysis integration and improves development documentation for the JLBH module: * Add module-quality Maven profile (pom.xml) - Activates on JDK 11+ to avoid Java 8 compatibility issues - Configures SpotBugs Maven Plugin 4.9.8.1 with Max effort, Low threshold - Includes test sources in analysis - Tested successfully: 16 issues identified across priority levels * Enhance AGENTS.md with developer guidance - Add security checklist for PR reviews - Expand build commands to include module-quality profile - Add "When to open a PR" section - Include inline comment guidelines with examples - Add link to Oxford style guide for British English - Clarify ISO-8859-1 character set requirements * Add TODO.md for module documentation tracking - Comprehensive checklist for architecture documentation work - ISO 9001 and ISO 27001 compliance considerations - Links to parent ARCH_TODO.md roadmap Tested with: mvn -P module-quality clean verify (Java 11) All tests pass, SpotBugs analysis completes successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- AGENTS.md | 28 +++--- TODO.md | 258 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ pom.xml | 32 +++++++ 3 files changed, 308 insertions(+), 10 deletions(-) create mode 100644 TODO.md diff --git a/AGENTS.md b/AGENTS.md index eb499013..e9a440c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,11 +44,19 @@ int count; ## Build & test commands -Agents must verify that the project still compiles and all unit tests pass before opening a PR: +Agents must verify that the project still compiles and all unit tests pass before opening a PR. Running from a clean checkout avoids stale artifacts: ```bash # From repo root -mvn -q verify +mvn -q clean verify +``` +The command should exit with code `0` to indicate success. + +For module-local quality runs on JLBH with Java 11 or newer: + +```bash +cd JLBH +mvn -P module-quality clean verify ``` ## Commit-message & PR etiquette @@ -113,7 +121,7 @@ This tight loop informs the AI accurately and creates immediate clarity for all When using AI agents to assist with development, please adhere to the following guidelines: * **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ISO-8859-1 guidelines outlined above. - Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. +Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. * **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. * **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. @@ -146,12 +154,12 @@ To improve traceability, we adopt the Nine-Box taxonomy for requirement and deci ```asciidoc === [Identifier] Title of Decision -Date:: YYYY-MM-DD -Context:: +Date :: YYYY-MM-DD +Context :: * What is the issue that this decision addresses? * What are the driving forces, constraints, and requirements? Decision Statement :: What is the change that is being proposed or was decided? -Alternatives Considered:: +Alternatives Considered :: * [Alternative 1 Name/Type]: ** *Description:* Brief description of the alternative. ** *Pros:* ... @@ -160,14 +168,14 @@ Alternatives Considered:: ** *Description:* Brief description of the alternative. ** *Pros:* ... ** *Cons:* ... -Rationale for Decision:: +Rationale for Decision :: * Why was the chosen decision selected? * How does it address the context and outweigh the cons of alternatives? -Impact & Consequences:: +Impact & Consequences :: * What are the positive and negative consequences of this decision? * How does this decision affect the system, developers, users, or operations? - What are the trade-offs made? -Notes/Links:: +Notes/Links :: ** (Optional: Links to relevant issues, discussions, documentation, proof-of-concepts) ``` @@ -178,7 +186,7 @@ Notes/Links:: Do not rely on indentation for list items in AsciiDoc documents. Use the following pattern instead: ```asciidoc -section:: Top Level Section +section :: Top Level Section (Optional) * first level ** nested level ``` diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..2fb25b76 --- /dev/null +++ b/TODO.md @@ -0,0 +1,258 @@ +# JLBH - Repository TODO + +**📋 Part of:** [Chronicle Architecture Documentation](../ARCH_TODO.md) +**Module Layer:** Infrastructure (Benchmarking) +**Priority:** 🟡 P2 +**Last Updated:** 2025-11-16 + +## Purpose + +This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TODO.md](../ARCH_TODO.md). It helps break down the architecture documentation work into manageable, repository-specific chunks. + +## Related Main TODO Files + +- [../ARCH_TODO.md](../ARCH_TODO.md) - Master architecture documentation roadmap +- [../TODO_INDEX.md](../TODO_INDEX.md) - Index of all TODO files +- [../ADOC_TODO.md](../ADOC_TODO.md) - AsciiDoc standardization (affects this module) + +## Module Information for Architecture Overview + +### Basic Information +- [ ] **Module Name:** JLBH +- [ ] **Maven Artifact ID:** jlbh +- [ ] **Primary Purpose:** [1-2 sentence description] +- [ ] **Layer in Chronicle Stack:** Infrastructure (Benchmarking) +- [ ] **Dependencies (Chronicle modules):** [List key Chronicle dependencies] +- [ ] **Key Classes/Interfaces:** [List 3-5 most important public APIs] + +### Architecture Information for ARCH_TODO.md Stage 3 + +**Feeds into:** ARCH_TODO.md Stage 3 - Module Deep Dives (ARCH-MOD-JLBH) + +- [ ] **Core Abstractions:** [List primary abstractions this module provides] +- [ ] **Interactions with other modules:** [Which Chronicle modules does this use/integrate with?] +- [ ] **Typical use cases:** [List 2-3 common scenarios where this module is used] +- [ ] **Performance characteristics:** [Key performance metrics if applicable] +- [ ] **Design patterns used:** [e.g., flyweight, single writer, etc.] + +### Existing Documentation Audit + +- [ ] Check if `src/main/docs/architecture-overview.adoc` exists + - [ ] If yes: Review quality (compare to Chronicle-Bytes standard) + - [ ] If no: Note as gap for ARCH_TODO Stage 5.5 +- [ ] Check if `src/main/docs/project-requirements.adoc` exists + - [ ] If yes: Review for ARCH_TODO Stage 1.75 (Requirements Overview) + - [ ] If no: Note as gap for FUNC_TODO.md +- [ ] Check if `src/main/docs/decision-log.adoc` exists + - [ ] If yes: Review for ARCH_TODO Stage 1.85 (Decision Log Overview) + - [ ] If no: Note as gap for DECISION_TODO.md +- [ ] Check if `README.adoc` provides good module overview +- [ ] Check if `AGENTS.md` exists and follows canonical template + +### Documentation Gaps (for ARCH_TODO Stage 5.5) + +**Missing Documentation:** +- [ ] Architecture overview? [Y/N] +- [ ] Requirements documentation? [Y/N] +- [ ] Decision log? [Y/N] +- [ ] Security review? [Y/N] +- [ ] Testing strategy? [Y/N] +- [ ] Performance targets? [Y/N] + +**Documentation Quality Issues:** +- [ ] Missing `:toc:`, `:lang: en-GB`, or `:source-highlighter: rouge`? +- [ ] Manual section numbering instead of `:sectnums:`? +- [ ] Broken cross-references? +- [ ] Outdated information? + +## Requirements for Architecture Overview (ARCH_TODO Stage 1.75) + +**Feeds into:** Requirements Overview consolidation + +- [ ] **Identify key functional requirements:** [List 3-5 most important] +- [ ] **Identify key non-functional requirements:** + - [ ] Performance targets: [e.g., latency, throughput] + - [ ] Security obligations: [e.g., bounds checking, input validation] + - [ ] Operability requirements: [e.g., monitoring, logging] +- [ ] **Map requirements to architecture patterns:** [How do requirements drive design?] + +## Decisions for Architecture Overview (ARCH_TODO Stage 1.85) + +**Feeds into:** Decision Log Overview consolidation + +- [ ] **Identify key architectural decisions:** [List 2-4 major decisions] + - [ ] Decision ID (if in decision-log.adoc): + - [ ] Brief description: + - [ ] Rationale: + - [ ] Alternatives considered: +- [ ] **Identify decision patterns used:** + - [ ] Off-heap memory? [Y/N - explain] + - [ ] Single writer principle? [Y/N - explain] + - [ ] Reference counting? [Y/N - explain] + - [ ] Flyweight pattern? [Y/N - explain] + +## Glossary Terms (ARCH_TODO Stage 1.5) + +**Feeds into:** Cross-module glossary + +- [ ] **Module-specific terms to include in glossary:** + - [ ] Term 1: [Definition] + - [ ] Term 2: [Definition] + - [ ] [Add more as needed] + + +## ISO 9001 Quality Management Considerations + +**Reference:** [../COMPLIANCE_QUICK_REFERENCE.md](../COMPLIANCE_QUICK_REFERENCE.md) + +### Design Inputs (ISO 9001 Clause 8.3.3) +- [ ] **Functional requirements documented?** + - [ ] Location: `src/main/docs/project-requirements.adoc` + - [ ] Requirements use Nine-Box taxonomy? (JLBH-FN-NNN) + - [ ] Requirements are testable and verifiable? +- [ ] **Non-functional requirements documented?** + - [ ] Performance requirements (JLBH-NF-P-NNN) + - [ ] Security requirements (JLBH-NF-S-NNN) + - [ ] Operability requirements (JLBH-NF-O-NNN) + +### Design Outputs (ISO 9001 Clause 8.3.5) +- [ ] **Architecture documented?** + - [ ] Location: `src/main/docs/architecture-overview.adoc` + - [ ] Describes key components and their interactions? + - [ ] Includes interface specifications? +- [ ] **APIs and interfaces specified?** + - [ ] Public API documented (JavaDoc)? + - [ ] Integration points with other modules described? + +### Design Verification (ISO 9001 Clause 8.3.4) +- [ ] **Requirements traceable to tests?** + - [ ] Test classes reference requirement IDs in comments/docs? + - [ ] Coverage: What % of requirements have corresponding tests? +- [ ] **Test strategy documented?** + - [ ] Unit test approach + - [ ] Integration test approach + - [ ] Performance test approach (if applicable) +- [ ] **Code review evidence?** + - [ ] PR review process followed? + - [ ] Review comments addressed? + +### Design Changes (ISO 9001 Clause 8.3.4) +- [ ] **Architectural decisions documented?** + - [ ] Location: `src/main/docs/decision-log.adoc` + - [ ] Decisions include context, alternatives, rationale? + - [ ] Impact of changes assessed? +- [ ] **Change history maintained?** + - [ ] Git commit messages describe rationale? + - [ ] Breaking changes documented in release notes? + +## ISO 27001 Information Security Considerations + +**Reference:** [../ARCHITECTURE_RESEARCH_GUIDE.md](../ARCHITECTURE_RESEARCH_GUIDE.md) - Security Research Topics + +### Secure Coding (ISO 27001 Control A.8.28) +- [ ] **Input validation implemented?** + - [ ] Where are untrusted inputs received? [List entry points] + - [ ] How are malformed inputs handled? + - [ ] Size limits enforced? +- [ ] **Bounds checking implemented?** + - [ ] Buffer overflow prevention mechanisms? + - [ ] Array access validation? + - [ ] Off-heap memory bounds checked? +- [ ] **Static analysis performed?** + - [ ] Checkstyle violations reviewed? + - [ ] SpotBugs security patterns checked? + - [ ] Suppressions justified and documented? + +### Access Control (ISO 27001 Control A.8.3) +- [ ] **Access restrictions implemented?** + - [ ] Are there authentication/authorization mechanisms? [Y/N] + - [ ] If yes, where and how are they implemented? + - [ ] Principle of least privilege followed? +- [ ] **Privileged operations identified?** + - [ ] Which operations require elevated privileges? + - [ ] How are they protected? + +### Cryptographic Controls (ISO 27001 Control A.8.24) +- [ ] **Cryptography usage identified?** + - [ ] Is encryption used? [Y/N - where?] + - [ ] Is hashing used? [Y/N - which algorithms?] + - [ ] Is TLS/SSL used? [Y/N - configuration?] +- [ ] **Key management?** + - [ ] How are cryptographic keys managed? + - [ ] Are keys hardcoded? [Y/N - if yes, flag as risk] + +### Network Security (ISO 27001 Control A.8.22) +- [ ] **Network communication security?** + - [ ] Does this module communicate over network? [Y/N] + - [ ] If yes, is communication encrypted? + - [ ] How are network endpoints authenticated? +- [ ] **Network configuration?** + - [ ] Secure defaults configured? + - [ ] Insecure protocols disabled? + +### Vulnerability Management (ISO 27001 Control A.8.8) +- [ ] **Known vulnerabilities?** + - [ ] Any open security issues in GitHub? + - [ ] Any CVEs against dependencies? +- [ ] **Security testing?** + - [ ] Fuzz testing performed? + - [ ] Security-specific test cases? + - [ ] Penetration testing performed? + +### Security Documentation +- [ ] **Security review documented?** + - [ ] Location: `src/main/docs/security-review.adoc` + - [ ] Threat model documented? + - [ ] Security controls described? + - [ ] Known limitations documented? + +## Improvement Tasks (ARCH_TODO Stage 5.5) + +**Feeds into:** Improve Existing Module Documentation + +### High Priority +- [ ] Create missing architecture-overview.adoc (if needed) +- [ ] Add missing front-matter to existing docs +- [ ] Fix broken cross-references +- [ ] Add `:sectnums:` where appropriate + +### Medium Priority +- [ ] Expand brief architecture docs (if < 75 lines) +- [ ] Add "Trade-offs and Alternatives" section (following Chronicle-Bytes pattern) +- [ ] Add performance characteristics section +- [ ] Create decision log entries for undocumented decisions + +### Low Priority +- [ ] Add diagrams (PlantUML or draw.io) +- [ ] Create example code snippets +- [ ] Expand requirements documentation +- [ ] Add cross-references to other module docs + +## Code Quality Tasks + +**Reference:** [../QUALITY_PLAYBOOK.md](../QUALITY_PLAYBOOK.md) + +- [ ] Run Checkstyle scan and document violations +- [ ] Run SpotBugs scan and document issues +- [ ] Identify any code review follow-ups from CODE_REVIEW_STATUS.md + +## Notes + +[Add any module-specific notes, blockers, or context here] + +## Completion Checklist + +Before marking this repository's contribution to ARCH_TODO as complete: + +- [ ] All "Module Information" sections filled out +- [ ] Existing documentation audited +- [ ] Requirements identified for ARCH_TODO Stage 1.75 +- [ ] Decisions identified for ARCH_TODO Stage 1.85 +- [ ] Glossary terms identified for ARCH_TODO Stage 1.5 +- [ ] Documentation gaps documented +- [ ] Improvement tasks prioritized +- [ ] Information contributed to relevant ARCH_TODO stages + +--- + +**When complete, update:** [../ARCH_TODO.md](../ARCH_TODO.md) Stage 3 tracking matrix diff --git a/pom.xml b/pom.xml index dcd23499..8c70f528 100644 --- a/pom.xml +++ b/pom.xml @@ -142,6 +142,38 @@ 8 + + + + module-quality + + [11,) + + + + + com.github.spotbugs + spotbugs-maven-plugin + 4.9.8.1 + + Max + Low + true + true + + + + spotbugs-main + process-classes + + spotbugs + + + + + + + From a18896ea38eae01c0246d4130bf749d4b7f6f700 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Tue, 18 Nov 2025 13:12:49 +0000 Subject: [PATCH 10/14] Checkpoint --- TODO.md | 11 ++-- pom.xml | 55 ++++++++++++++++--- .../java/net/openhft/chronicle/jlbh/JLBH.java | 1 + .../jlbh/JLBHAdditionalCoverageTest.java | 1 + .../chronicle/jlbh/JLBHIntegrationTest.java | 1 + .../net/openhft/chronicle/jlbh/JLBHTest.java | 1 + 6 files changed, 59 insertions(+), 11 deletions(-) diff --git a/TODO.md b/TODO.md index 2fb25b76..24435f1d 100644 --- a/TODO.md +++ b/TODO.md @@ -2,7 +2,7 @@ **📋 Part of:** [Chronicle Architecture Documentation](../ARCH_TODO.md) **Module Layer:** Infrastructure (Benchmarking) -**Priority:** 🟡 P2 +**Priority:** 🟢 P3 **Last Updated:** 2025-11-16 ## Purpose @@ -232,13 +232,16 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD **Reference:** [../QUALITY_PLAYBOOK.md](../QUALITY_PLAYBOOK.md) -- [ ] Run Checkstyle scan and document violations -- [ ] Run SpotBugs scan and document issues +- [x] Run Checkstyle scan and document violations + - Latest command: `mvn checkstyle:check` from `JLBH` with Java 21 (see `verify-jlbh-java21-checkstyle-latest.log`); Checkstyle reports `You have 0 Checkstyle violations.` for this module. +- [x] Run SpotBugs scan and document issues + - A Java 21 SpotBugs run (`mvn com.github.spotbugs:spotbugs-maven-plugin:4.9.8.1:check -DskipTests`, see `verify-jlbh-java21-spotbugs-latest.log`) completes with `BugInstance size is 0` and `No errors/warnings found` for JLBH. - [ ] Identify any code review follow-ups from CODE_REVIEW_STATUS.md + - JLBH does not currently have a dedicated section in `CODE_REVIEW_STATUS.md`; any future review actions (for example around new benchmarks or harness features) should be recorded there and linked from this TODO. ## Notes -[Add any module-specific notes, blockers, or context here] +- 2025-11-18: JLBH is Checkstyle- and SpotBugs-clean on Java 21 (`verify-jlbh-java21-checkstyle-latest.log`, `verify-jlbh-java21-spotbugs-latest.log`). Remaining documentation/requirements/compliance TODOs are longer-running work and are tracked as deferred in `TODO_STATUS.md`. ## Completion Checklist diff --git a/pom.xml b/pom.xml index 8c70f528..fe12e549 100644 --- a/pom.xml +++ b/pom.xml @@ -143,14 +143,44 @@ - + - module-quality - - [11,) - + quality + + org.apache.maven.plugins + maven-checkstyle-plugin + 3.6.0 + + + validate + validate + + check + + + + + ${checkstyle.config.location} + true + true + true + ${checkstyle.violationSeverity} + + + + com.puppycrawl.tools + checkstyle + 10.26.1 + + + net.openhft + chronicle-quality-rules + 1.27.0-SNAPSHOT + + + com.github.spotbugs spotbugs-maven-plugin @@ -160,13 +190,24 @@ Low true true + + net/openhft/quality/spotbugs27/chronicle-spotbugs-include.xml + net/openhft/quality/spotbugs27/chronicle-spotbugs-exclude.xml + + + net.openhft + chronicle-quality-rules + 1.27.0-SNAPSHOT + + spotbugs-main - process-classes + + process-test-classes - spotbugs + check diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java index 11d97b68..9e9ad3ed 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java @@ -81,6 +81,7 @@ public class JLBH implements NanoSampler { private final AtomicBoolean abortTestRun = new AtomicBoolean(); private final long mod; private final long length; + // Todo: Remove all concurrent constructs such as volatile and AtomicBoolean private final AtomicLong noResultsReturned = new AtomicLong(); //Use non-atomic when so thread synchronisation is necessary private boolean warmedUp; diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java index f7ca5c25..57dd1b59 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java @@ -3,6 +3,7 @@ */ package net.openhft.chronicle.jlbh; +import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.util.NanoSampler; import org.junit.Test; diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java index 29d3e44f..f43d4f33 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java @@ -3,6 +3,7 @@ */ package net.openhft.chronicle.jlbh; +import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.OS; import org.junit.After; import org.junit.Assume; diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java index cf6bc938..e1ab097a 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java @@ -3,6 +3,7 @@ */ package net.openhft.chronicle.jlbh; +import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.Jvm; import net.openhft.chronicle.core.io.Closeable; import net.openhft.chronicle.core.threads.EventLoop; From 56ea0d32b691a527c158db2dd90a53cd54f000c7 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Wed, 19 Nov 2025 09:59:31 +0000 Subject: [PATCH 11/14] Update TODO.md with ISO alignment and trust zone details for multiple modules --- TODO.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index 24435f1d..6f626cf2 100644 --- a/TODO.md +++ b/TODO.md @@ -18,12 +18,17 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD ## Module Information for Architecture Overview ### Basic Information -- [ ] **Module Name:** JLBH -- [ ] **Maven Artifact ID:** jlbh -- [ ] **Primary Purpose:** [1-2 sentence description] -- [ ] **Layer in Chronicle Stack:** Infrastructure (Benchmarking) -- [ ] **Dependencies (Chronicle modules):** [List key Chronicle dependencies] -- [ ] **Key Classes/Interfaces:** [List 3-5 most important public APIs] +- [x] **Module Name:** JLBH +- [x] **Maven Artifact ID:** jlbh +- [x] **Primary Purpose:** Provide a Java latency benchmark harness for measuring end-to-end response times under realistic workloads rather than micro-benchmarks. +- [x] **Layer in Chronicle Stack:** Infrastructure (Benchmarking and performance analysis) +- [x] **Dependencies (Chronicle modules):** `affinity`, `chronicle-core`, `chronicle-threads` (test-only) +- [x] **Key Classes/Interfaces:** `JLBH`, `JLBHOptions`, `JLBHTask`, `JLBHResult`, `JLBHResultConsumer` + +### ISO Alignment and Trust Zone + +- [x] **Trust zone identified (Edge/Core/Foundation):** JLBH is a *Foundation (Zone C)* benchmarking and latency measurement tool used to characterise Chronicle components rather than process untrusted production traffic directly. +- [x] **Shared standards reviewed:** Align JLBH documentation with the shared standards in `Chronicle-Quality-Rules/src/main/docs`, focusing on its role in validating latency budgets and performance envelopes rather than enforcing security policies. ### Architecture Information for ARCH_TODO.md Stage 3 From 17a449ec4bde61d7aa0c6d9cf44434d1591acd26 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Fri, 21 Nov 2025 10:29:41 +0000 Subject: [PATCH 12/14] Code Analysis fixes --- .../net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java | 1 - .../java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java | 1 - src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java | 1 - 3 files changed, 3 deletions(-) diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java index 57dd1b59..f7ca5c25 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java @@ -3,7 +3,6 @@ */ package net.openhft.chronicle.jlbh; -import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.util.NanoSampler; import org.junit.Test; diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java index f43d4f33..29d3e44f 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java @@ -3,7 +3,6 @@ */ package net.openhft.chronicle.jlbh; -import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.OS; import org.junit.After; import org.junit.Assume; diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java index e1ab097a..cf6bc938 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java @@ -3,7 +3,6 @@ */ package net.openhft.chronicle.jlbh; -import java.nio.charset.StandardCharsets; import net.openhft.chronicle.core.Jvm; import net.openhft.chronicle.core.io.Closeable; import net.openhft.chronicle.core.threads.EventLoop; From 23cc1785e1fe84d7a256592c865b95c79cd0d492 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Mon, 24 Nov 2025 10:22:44 +0000 Subject: [PATCH 13/14] Remove unnecessary blank lines in various classes for improved code readability --- AGENTS.md | 224 +------ LICENSE.adoc | 5 +- README.adoc | 18 +- TODO.md | 1 - src/main/adoc/architecture.adoc | 200 ------- src/main/adoc/benchmark-lifecycle.adoc | 80 --- src/main/adoc/decision-log.adoc | 214 ------- src/main/adoc/jlbh-cookbook.adoc | 553 ------------------ src/main/adoc/project-requirements.adoc | 206 ------- .../adoc/results-interpretation-guide.adoc | 158 ----- .../chronicle/jlbh/JLBHResultConsumer.java | 1 - 11 files changed, 23 insertions(+), 1637 deletions(-) delete mode 100644 src/main/adoc/architecture.adoc delete mode 100644 src/main/adoc/benchmark-lifecycle.adoc delete mode 100644 src/main/adoc/decision-log.adoc delete mode 100644 src/main/adoc/jlbh-cookbook.adoc delete mode 100644 src/main/adoc/project-requirements.adoc delete mode 100644 src/main/adoc/results-interpretation-guide.adoc diff --git a/AGENTS.md b/AGENTS.md index e9a440c0..166412d8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,216 +1,8 @@ -# Guidance for AI agents, bots, and humans contributing to Chronicle Software's OpenHFT projects. - -LLM-based agents can accelerate development only if they respect our house rules. This file tells you: - -* how to run and verify the build; -* what *not* to comment; -* when to open pull requests. - -## Language & character-set policy - -| Requirement | Rationale | -|--------------|-----------| -| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the [University of Oxford style guide](https://www.ox.ac.uk/public-affairs/style-guide) for reference. | -| **ISO-8859-1 (Latin-1) only** (code-points 0-255). Avoid smart quotes, non-breaking spaces and characters outside this range. | ISO-8859-1 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect predictable 8-bit characters. | -| If a symbol is not available in ISO-8859-1, use a textual form such as `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode outside Latin-1 must not be inserted. | Extended encodings outside ISO-8859-1 are *not* portable and are therefore disallowed. | -| Tools to check ISO-8859-1 compliance include `iconv -f ISO-8859-1 -t ISO-8859-1` and IDE settings that flag characters outside Latin-1. | These help catch stray characters before code review. | - -## Javadoc guidelines - -**Goal:** Every Javadoc block should add information you cannot glean from the method signature alone. Anything else is -noise and slows readers down. - -| Do | Don't | -|----|-------| -| State *behavioural contracts*, edge-cases, thread-safety guarantees, units, performance characteristics and checked exceptions. | Restate the obvious ("Gets the value", "Sets the name"). | -| Keep the first sentence short; it becomes the summary line in aggregated docs. | Duplicate parameter names/ types unless more explanation is needed. | -| Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | -| Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | - -The principle that Javadoc should only explain what is *not* manifest from the -signature is well-established in the wider Java community. - -Inline comments should also avoid noise. The following example shows the -difference: - -```java -// BAD: adds no value -int count; // the count - -// GOOD: explains a subtlety -// count of messages pending flush -int count; -``` - -## Build & test commands - -Agents must verify that the project still compiles and all unit tests pass before opening a PR. Running from a clean checkout avoids stale artifacts: - -```bash -# From repo root -mvn -q clean verify -``` -The command should exit with code `0` to indicate success. - -For module-local quality runs on JLBH with Java 11 or newer: - -```bash -cd JLBH -mvn -P module-quality clean verify -``` - -## Commit-message & PR etiquette - -1. **Subject line <= 72 chars**, imperative mood: Fix roll-cycle offset in `ExcerptAppender`. -2. Reference the JIRA/GitHub issue if it exists. -3. In *body*: *root cause -> fix -> measurable impact* (latency, allocation, etc.). Use ASCII bullet points. -4. **Run `mvn verify`** again after rebasing. - -### When to open a PR - -* Open a pull request once your branch builds and tests pass with `mvn -q clean verify`. -* Link the PR to the relevant issue or decision record. -* Keep PRs focused: avoid bundling unrelated refactoring with new features. -* Re-run the build after addressing review comments to ensure nothing broke. - -## What to ask the reviewers - -* *Is this AsciiDoc documentation precise enough for a clean-room re-implementation?* -* Does the Javadoc explain the code's *why* and *how* that a junior developer would not be expected to work out? -* Are the documentation, tests and code updated together so the change is clear? -* Does the commit point back to the relevant requirement or decision tag? -* Would an example or small diagram help future maintainers? - -### Security checklist (review **after every change**) - -**Run a security review on *every* PR**: Walk through the diff looking for input validation, authentication, authorisation, encoding/escaping, overflow, resource exhaustion and timing-attack issues. - -**Never commit secrets or credentials**: tokens, passwords, private keys, TLS materials, internal hostnames, Use environment variables, HashiCorp Vault, AWS/GCP Secret Manager, etc. - -**Document security trade-offs**: Chronicle prioritises low-latency systems; sometimes we relax safety checks for specific reasons. Future maintainers must find these hot-spots quickly, In Javadoc and `.adoc` files call out *why* e.g. "Unchecked cast for performance - assumes trusted input". - -## Project requirements - -See the [Decision Log](src/main/adoc/decision-log.adoc) for the latest project decisions. -See the [Project Requirements](src/main/adoc/project-requirements.adoc) for details on project requirements. - -## Elevating the Workflow with Real-Time Documentation - -Building upon our existing Iterative Workflow, the newest recommendation is to emphasise *real-time updates* to documentation. -Ensure the relevant `.adoc` files are updated when features, requirements, implementation details, or tests change. -This tight loop informs the AI accurately and creates immediate clarity for all team members. - -### Benefits of Real-Time Documentation - -* **Confidence in documentation**: Accurate docs prevent miscommunications that derail real-world outcomes. -* **Reduced drift**: Real-time updates keep requirements, tests and code aligned. -* **Faster feedback**: AI can quickly highlight inconsistencies when everything is in sync. -* **Better quality**: Frequent checks align the implementation with the specified behaviour. -* **Smoother onboarding**: Up-to-date AsciiDoc clarifies the system for new developers. -* **Incremental changes**: AIDE flags newly updated files so you can keep the documentation synchronised. - -### Best Practices - -* **Maintain Sync**: Keep documentation (AsciiDoc), tests, and code synchronised in version control. Changes in one area should prompt reviews and potential updates in the others. -* **Doc-First for New Work**: For *new* features or requirements, aim to update documentation first, then use AI to help produce or refine corresponding code and tests. For refactoring or initial bootstrapping, updates might flow from code/tests back to documentation, which should then be reviewed and finalised. -* **Small Commits**: Each commit should ideally relate to a single requirement or coherent change, making reviews easier for humans and AI analysis tools. -- **Team Buy-In**: Encourage everyone to review AI outputs critically and contribute to maintaining the synchronicity of all artefacts. - -## AI Agent Guidelines - -When using AI agents to assist with development, please adhere to the following guidelines: - -* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ISO-8859-1 guidelines outlined above. -Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. -* **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. -* **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. - -## Company-Wide Tagging - -This section records **company-wide** decisions that apply to *all* Chronicle projects. All identifiers use the --xxx prefix. The `xxx` are unique across in the same Scope even if the tags are different. Component-specific decisions live in their xxx-decision-log.adoc files. - -### Tag Taxonomy (Nine-Box Framework) - -To improve traceability, we adopt the Nine-Box taxonomy for requirement and decision identifiers. These tags are used in addition to the existing ALL prefix, which remains reserved for global decisions across every project. - -.Adopt a Nine-Box Requirement Taxonomy - -|Tag | Scope | Typical examples | -|----|-------|------------------| -|FN |Functional user-visible behaviour | Message routing, business rules | -|NF-P |Non-functional - Performance | Latency budgets, throughput targets | -|NF-S |Non-functional - Security | Authentication method, TLS version | -|NF-O |Non-functional - Operability | Logging, monitoring, health checks | -|TEST |Test / QA obligations | Chaos scenarios, benchmarking rigs | -|DOC |Documentation obligations | Sequence diagrams, user guides | -|OPS |Operational / DevOps concerns | Helm values, deployment checklist | -|UX |Operator or end-user experience | CLI ergonomics, dashboard layouts | -|RISK |Compliance / risk controls | GDPR retention, audit trail | - -`ALL-*` stays global, case-exact tags. Pick one primary tag if multiple apply. - -### Decision Record Template - -```asciidoc -=== [Identifier] Title of Decision - -Date :: YYYY-MM-DD -Context :: -* What is the issue that this decision addresses? -* What are the driving forces, constraints, and requirements? -Decision Statement :: What is the change that is being proposed or was decided? -Alternatives Considered :: -* [Alternative 1 Name/Type]: -** *Description:* Brief description of the alternative. -** *Pros:* ... -** *Cons:* ... -* [Alternative 2 Name/Type]: -** *Description:* Brief description of the alternative. -** *Pros:* ... -** *Cons:* ... -Rationale for Decision :: -* Why was the chosen decision selected? -* How does it address the context and outweigh the cons of alternatives? -Impact & Consequences :: -* What are the positive and negative consequences of this decision? -* How does this decision affect the system, developers, users, or operations? -- What are the trade-offs made? -Notes/Links :: -** (Optional: Links to relevant issues, discussions, documentation, proof-of-concepts) -``` - -## Asciidoc formatting guidelines - -### List Indentation - -Do not rely on indentation for list items in AsciiDoc documents. Use the following pattern instead: - -```asciidoc -section :: Top Level Section (Optional) -* first level - ** nested level -``` - -### Emphasis and Bold Text - -In AsciiDoc, an underscore `_` is _emphasis_; `*text*` is *bold*. - -### Section Numbering - -Use automatic section numbering for all `.adoc` files. - -* Add `:sectnums:` to the document header. -* Do not prefix section titles with manual numbers to avoid duplication. - -```asciidoc -= Document Title -Chronicle Software -:toc: -:sectnums: -:lang: en-GB -:source-highlighter: rouge - -The document overview goes here. - -== Section 1 Title -``` +# Chronicle JLBH AGENTS + +- Follow repository `AGENTS.md` as the base rules; this file adds JLBH specifics. Durable docs live in `src/main/docs/` with the landing page at `README.adoc`. +- Module purpose: Java Latency Benchmark Harness for running context-aware latency/throughput benchmarks with Chronicle-friendly patterns. +- Build commands: full build `mvn -q clean verify`; module-only without tests `mvn -pl JLBH -am -DskipTests install`. +- Quality gates: keep Checkstyle/SpotBugs clean; ensure benchmarks remain deterministic and reproducible; guard against benchmark harness changes that alter measurement accuracy. +- Documentation: maintain Nine-Box IDs in `src/main/docs/project-requirements.adoc` and link decisions/tests/benchmarks to them; British English, ASCII/ISO-8859-1, `:source-highlighter: rouge`. +- Guardrails: avoid introducing benchmark code that masks coordinated omission; document any new CLI options or environment knobs in the docs. diff --git a/LICENSE.adoc b/LICENSE.adoc index 4c32dfd9..6d68b90a 100644 --- a/LICENSE.adoc +++ b/LICENSE.adoc @@ -1,5 +1,8 @@ == Copyright 2016-2025 chronicle.software +:toc: +:lang: en-GB +:source-highlighter: rouge Licensed under the *Apache License, Version 2.0* (the "License"); you may not use this file except in compliance with the License. @@ -11,4 +14,4 @@ Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. \ No newline at end of file +limitations under the License. diff --git a/README.adoc b/README.adoc index 6d6f0380..337c35e8 100644 --- a/README.adoc +++ b/README.adoc @@ -1,10 +1,14 @@ = Chronicle JLBH +:toc: +:lang: en-GB +:source-highlighter: rouge Chronicle Software -:css-signature: demo +:toc: :toc: macro -:toclevels: 2 :icons: font :lang: en-GB +:toclevels: 2 +:css-signature: demo :source-highlighter: rouge image:https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh] @@ -22,9 +26,9 @@ toc::[] Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a micro-benchmark. See <> for a series of articles introducing JLBH. An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles]. -link:src/main/adoc/project-requirements.adoc[The requirements document] contains detailed feature descriptions. +link:src/main/docs/project-requirements.adoc[The requirements document] contains detailed feature descriptions. -For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_glossary[the Glossary (section 7)]. +For terminology used throughout the project, see link:src/main/docs/project-requirements.adoc#_7-glossary[the Glossary (section 7)]. Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread. To do this, use the JLBH.eventLoopHandler method rather than JLBH.start. @@ -81,7 +85,7 @@ Commonly used option methods include: For a full list of configuration parameters see `src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java`. -For a visual overview of how a benchmark progresses, see the link:src/main/adoc/benchmark-lifecycle.adoc[benchmark lifecycle diagram]. +For a visual overview of how a benchmark progresses, see the link:src/main/docs/benchmark-lifecycle.adoc[benchmark lifecycle diagram]. == Additional Features @@ -129,10 +133,10 @@ This writes the output to `result.csv` by default, as defined in `JLBHResultSeri == Environment and Configuration -CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/adoc/project-requirements.adoc[requirements document]. +CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/docs/project-requirements.adoc[requirements document]. Histogramming of recorded latencies is precise. -As specified in the link:src/main/adoc/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. +As specified in the link:src/main/docs/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. This accuracy retains sub-nanosecond resolution over a wide range, so the reported percentiles closely reflect true end-to-end timings. == Sample Benchmarks and Tests diff --git a/TODO.md b/TODO.md index 6f626cf2..a011e0f6 100644 --- a/TODO.md +++ b/TODO.md @@ -105,7 +105,6 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD - [ ] Term 2: [Definition] - [ ] [Add more as needed] - ## ISO 9001 Quality Management Considerations **Reference:** [../COMPLIANCE_QUICK_REFERENCE.md](../COMPLIANCE_QUICK_REFERENCE.md) diff --git a/src/main/adoc/architecture.adoc b/src/main/adoc/architecture.adoc deleted file mode 100644 index 2a40176e..00000000 --- a/src/main/adoc/architecture.adoc +++ /dev/null @@ -1,200 +0,0 @@ -= Chronicle JLBH - Architecture Overview -Chronicle Software -:toc: -:sectnums: -:lang: en-GB -:source-highlighter: rouge -:sectnums: - -== Introduction - -Chronicle JLBH (Java Latency Benchmark Harness) is a library designed for measuring and analyzing the latency of Java applications "in context," particularly under specific throughput conditions and accounting for coordinated omission. -This document outlines its high-level architecture, major components, and key operational flows. - -== Major Components - -The JLBH system is composed of several key classes and interfaces that work together to orchestrate and execute benchmarks: - -* `JLBH` -** The primary orchestrator of the benchmark lifecycle. -** Manages warm-up phases, iteration loops, and multiple runs. -** Provides the `sample(long durationNs)` method for recording end-to-end latencies. -** Offers `addProbe(String name)` to create `NanoSampler` instances for measuring sub-components or stages within the benchmarked task. -** Responsible for printing summaries to a `PrintStream`. -** Implements `NanoSampler` itself for the main end-to-end timings. - -* `JLBHOptions` -** A builder-style configuration object used to define all parameters for a `JLBH` instance. -** Settings include: -*** Number of warm-up iterations (`warmUpIterations`). -*** Number of measurement iterations per run (`iterations`). -*** Number of benchmark runs (`runs`). -*** Target throughput and its time unit (`throughput`, `throughputTimeUnit`). -*** The `JLBHTask` to be executed. -*** Coordinated Omission accounting (`accountForCoordinatedOmission`). -*** OS jitter recording parameters (`recordOSJitter`, `recordJitterGreaterThanNs`, `jitterAffinity`). -*** Latency distribution strategy (`latencyDistributor`). -*** CPU affinity for the main benchmark thread (`acquireLock`). -*** Timeout for benchmark abortion if no samples are produced (`timeout`). - -* `JLBHTask` -** An interface implemented by the user to define the actual workload to be benchmarked. -** Key lifecycle methods invoked by `JLBH`: -*** `init(JLBH jlbh)`: Called once before any iterations, used for setup and to acquire `NanoSampler` probes via `jlbh.addProbe()`. -*** `run(long startTimeNs)`: Called for each benchmark iteration (both warm-up and measurement). -The `startTimeNs` is the calculated ideal start time for the iteration. -*** `warmedUp()`: Called after the warm-up phase is complete and before the first measurement run begins. -*** `runComplete()`: Called after each measurement run finishes. -*** `complete()`: Called once after all runs have finished, for cleanup. - -* `NanoSampler` -** An interface for recording timing samples. -** Implementations (like `JLBH` itself or those returned by `addProbe`) typically wrap a `Histogram`. -** Primary method: `sampleNanos(long durationNs)`. - -* `Histogram` -** A data structure used to record the distribution of latency samples for each probe (end-to-end, custom probes, OS jitter). -** Provided by the `chronicle-core` library. -** Key characteristics: -*** Stores frequencies of samples in logarithmically scaled buckets, enabling high precision over a wide dynamic range. -*** Configured by `JLBH` (via `createHistogram()`) for typically 35 bits of range and 8 significant figures. -*** Provides methods to extract percentile data (e.g., `getPercentiles()`). - -* `JLBHResult` -** An immutable interface representing the complete results of a benchmark execution. -** Provides access to: -*** `endToEnd()`: A `ProbeResult` for the main end-to-end measurements. -*** `probe(String probeName)`: An `Optional` for any additionally registered probes. -*** `osJitter()`: An `Optional` if OS jitter was recorded. -** Concrete immutable implementations include `ImmutableJLBHResult`, `ImmutableProbeResult`, and `ImmutableRunResult`. - -* `JLBHResultConsumer` -** A functional interface (`Consumer` and `Supplier`) used to decouple the production of `JLBHResult` from its consumption. -** `JLBHResultConsumer.newThreadSafeInstance()` provides a `ThreadSafeJLBHResultConsumer` that allows results to be safely retrieved by a different thread after the benchmark completes. - -* `LatencyDistributor` -** An optional strategy interface (`long apply(long averageLatencyNS)`) to shape the throughput by adjusting the delay between task invocations. -** Allows simulation of different load patterns beyond a constant rate. -** Predefined implementations are available in `LatencyDistributors` (e.g., `NORMAL`, `RANDOM`). - -* `OSJitterMonitor` -** An internal background thread responsible for measuring operating system scheduling jitter. -** Monitoring is enabled by default (`JLBHOptions.recordOSJitter` defaults to true) and can be disabled via `recordOSJitter(false)`. -** Repeatedly calls `System.nanoTime()` to detect delays greater than `recordJitterGreaterThanNs` and records these into a dedicated `Histogram`. - -== Execution Flow - -The harness follows a well-defined lifecycle, detailed visually in link:benchmark-lifecycle.adoc[benchmark lifecycle diagram]. -A typical execution sequence is as follows: - -* **Initialization**: -** The user configures `JLBHOptions` defining all benchmark parameters. -** A `JLBH` instance is created with these options, an optional `PrintStream` for output, and an optional `JLBHResultConsumer` for programmatic result access. -** The `JLBHTask.init(JLBH)` method is called, allowing the user's task to perform setup. -*** During `init`, custom `NanoSampler` probes can be registered via `jlbh.addProbe(String)`. -Each probe is backed by its own `Histogram`. -** If `recordOSJitter` is enabled in `JLBHOptions`, the `OSJitterMonitor` background thread is started. - -* **Warm-up Phase**: -** The `JLBHTask.run(System.nanoTime())` method is invoked `warmUpIterations` times. -*** The primary purpose is to allow the JVM's JIT compiler to optimize the code and for the system to reach a steady operational state. -*** Latency samples collected during this phase are typically added to histograms but are reset and discarded before measurement runs begin. -** After warm-up iterations, `JLBHTask.warmedUp()` is called. -** All probe histograms (end-to-end, custom, OS jitter) are reset. - -* **Measurement Runs**: -** This phase is repeated for the number of `runs` specified in `JLBHOptions`. -** For each run: -*** A loop executes for the configured number of `iterations`. -*** In each iteration: -**** The `JLBH` orchestrator calculates the `startTimeNs` for the task invocation. -***** This calculation is influenced by the configured `throughput` and the `LatencyDistributor` strategy. -***** If `accountForCoordinatedOmission` is true, `startTimeNs` represents the ideal scheduled time. -The harness may busy-wait (spin using `System.nanoTime()`) if the current time is before this ideal `startTimeNs` to ensure the task starts as close to the scheduled time as possible. -**** `JLBHTask.run(startTimeNs)` is invoked with this calculated start time. -***** The user's benchmark logic executes. -***** To record the end-to-end latency, the task calls `jlbh.sample(System.nanoTime() - startTimeNs)`. -***** To record latency for sub-stages, the task calls `sampleNanos()` on any custom `NanoSampler` probes obtained during `init`. -**** Each call to `sample()` or `sampleNanos()` on a sampler results in the duration being recorded in its corresponding `Histogram`. -*** After all iterations for the current run are complete: -**** The `JLBHTask.runComplete()` method is invoked. -**** Statistics for the completed run (percentile values from each histogram) are typically printed to the configured `PrintStream`. -**** The percentile data for this run is stored as part of the accumulating results. -**** All probe histograms are reset to prepare for the next measurement run (if any). - -* **Completion**: -** After all measurement runs have finished: -*** A final summary report, often comparing results across runs and showing variation, is printed to the `PrintStream`. -*** The complete, immutable `JLBHResult` object, containing all data for all probes across all runs, is constructed. -*** If a `JLBHResultConsumer` was provided, its `accept(JLBHResult)` method is called with the final result. -** The `JLBHTask.complete()` method is invoked for any final cleanup by the user's task. -** The `OSJitterMonitor` thread is terminated if it was running. - -== Threading Model - -JLBH has a specific threading model that users and contributors should understand: - -* **Primary Harness Thread**: -** When `JLBH.start()` is called, it orchestrates the entire benchmark lifecycle (initialization, warm-up, measurement runs, completion) on the *calling thread*. -** All `JLBHTask` lifecycle methods (`init`, `run`, `warmedUp`, `runComplete`, `complete`) are invoked sequentially on this single harness thread. -*** This implies that `JLBHTask` implementations generally do not need to be concerned about thread-safety for their internal state when accessed *only* by these JLBH-invoked lifecycle methods. -** CPU affinity for this thread can be suggested via `JLBHOptions.acquireLock`. - -* **Event Loop Integration**: -** As an alternative to `JLBH.start()`, the benchmark can be driven by an external `EventLoop` (e.g., from Chronicle Services) by calling `JLBH.eventLoopHandler(EventLoop)`. -** Coordinated omission accounting must remain enabled; calling `eventLoopHandler` with `accountForCoordinatedOmission(false)` throws an `UnsupportedOperationException`. -** In this mode, the warm-up and measurement iterations are scheduled and executed on the provided event loop's thread, using internal `EventHandler` implementations (`WarmupHandler`, `JLBHEventHandler`). - -* **User-Spawned Threads within `JLBHTask`**: -** The code being benchmarked *within* `JLBHTask.run()` (i.e., the "business logic") *can* be multi-threaded if the application itself is multi-threaded. -** However, critical interactions with the `JLBH` instance *must* be managed carefully. -*** Calling `jlbh.sample()` or methods on `NanoSampler` probes (like `probe.sampleNanos()`) should ideally be done only by the thread that `JLBH` used to call `JLBHTask.run()`. -*** The `JLBH` class is annotated `@SingleThreaded`, and its internal state (including the histograms) is generally not designed for concurrent modification from multiple user threads without external synchronization. -Chronicle Core's `Histogram` has specific single-writer assumptions unless explicitly used with synchronization. - -* **`OSJitterMonitor` Thread**: -** If OS jitter recording is enabled (`JLBHOptions.recordOSJitter == true`), a dedicated background thread (`OSJitterMonitor`) is started. -** This thread runs independently of the main harness thread. -** Its sole purpose is to repeatedly sample `System.nanoTime()` to detect scheduling delays or pauses exceeding `JLBHOptions.recordJitterGreaterThanNs` and record these durations into a dedicated `Histogram`. -** Its CPU affinity can be optionally controlled via `JLBHOptions.jitterAffinity`. - -* **Result Consumption**: -** The `JLBHResult` object containing all benchmark data is immutable by design. -** To safely access this result from a thread other than the main harness thread (e.g., an application thread wanting to query results after a test run), `JLBHResultConsumer.newThreadSafeInstance()` should be used. -This creates a `ThreadSafeJLBHResultConsumer` which uses a `volatile` field to ensure proper publication of the immutable result. - -== Data Management and Results - -JLBH pays careful attention to how latency data is recorded, structured, and reported: - -* **Latency Recording**: -** All latency durations are measured and recorded in nanoseconds. -** `System.nanoTime()` is the fundamental source for time measurements, chosen for its high resolution and (on most systems) monotonic behavior. -** For each active probe (end-to-end, user-defined additional probes, OS jitter), a `net.openhft.chronicle.core.util.Histogram` instance is maintained. -*** These histograms are configured by `JLBH.createHistogram()` for high precision (typically 8 significant figures of value) and a wide dynamic range (typically 35 bits, covering from nanoseconds to many seconds). - -* **Immutable Results**: -** A core design principle is the immutability of benchmark results. -Once a benchmark or a run is complete, the generated data is fixed. -** The `JLBHResult` interface and its constituent parts (`ProbeResult`, `RunResult`) are designed to be immutable. -** This is enforced by internal implementations like `ImmutableJLBHResult`, `ImmutableProbeResult`, and `ImmutableRunResult`. -** Immutability simplifies reasoning about results, ensures thread-safe access for consumers, and allows for reliable and repeatable reporting. - -* **Structure of Results (`JLBHResult`)**: -** The top-level `JLBHResult` object provides access to: -*** `endToEnd()`: Returns a `ProbeResult` for the overall end-to-end latency measurements. -*** `probe(String probeName)`: Returns an `Optional` for any user-defined probe, identified by its name. -*** `osJitter()`: Returns an `Optional` for the OS jitter measurements, if recorded. -** Each `JLBHResult.ProbeResult` object contains: -*** `summaryOfLastRun()`: A `RunResult` detailing the statistics of the final measurement run for that probe. -*** `eachRunSummary()`: A `List`, providing the statistics for every measurement run of that probe, in execution order. -** Each `JLBHResult.RunResult` object contains: -*** `percentiles()`: A `Map`, mapping enum keys (e.g., `PERCENTILE_50TH`, `PERCENTILE_99TH`, `WORST`) to their corresponding latency `Duration` values. -*** Convenience getter methods like `get50thPercentile()`, `get99thPercentile()`, `getWorst()`. - -* **Output and Serialization**: -** Human-readable summaries are printed to the configured `PrintStream` (default `System.out`) during and after benchmark runs. -This includes percentile tables for each probe and overall summaries. -** Programmatic access to the structured, immutable results is provided via the `JLBHResultConsumer` and the `JLBHResult` interface. -** Results can be serialized to CSV format using `net.openhft.chronicle.jlbh.util.JLBHResultSerializer`, facilitating offline analysis or import into spreadsheets and other tools. -** Integration with CI systems like TeamCity is supported via `net.openhft.chronicle.jlbh.TeamCityHelper`, which can emit build statistics. diff --git a/src/main/adoc/benchmark-lifecycle.adoc b/src/main/adoc/benchmark-lifecycle.adoc deleted file mode 100644 index aaac9445..00000000 --- a/src/main/adoc/benchmark-lifecycle.adoc +++ /dev/null @@ -1,80 +0,0 @@ -= Benchmark Lifecycle -:lang: en-GB -:source-highlighter: rouge -:toc: -:sectnums: - -[mermaid] ----- -graph LR - A[Setup] --> B(Warmup); - B --> C{Execution Per Run}; - C --> D[Reporting]; - D --> E[Cleanup]; ----- - -A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are: - -== Setup - -This initial phase prepares the benchmark environment: - -* **JLBH Configuration**: The user instantiates `JLBHOptions` to define all benchmark parameters (e.g., iterations, throughput, runs, coordinated omission settings). An instance of `JLBH` is then created using these options. -* **Task Initialization**: The `JLBHTask.init(JLBH jlbh)` method of the user-provided task is invoked. -** This allows the benchmark task to perform its own one-time setup, such as initializing resources or fixtures. -** Crucially, this is where additional measurement probes can be registered by calling `jlbh.addProbe("probeName")` to get `NanoSampler` instances for specific sub-sections of the task. -* **OS Jitter Monitoring**: Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. -* **Affinity**: If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. - -== Warmup - -Before any measurements are formally recorded, the harness executes a warm-up phase: - -* **Warmup Iterations**: The `JLBHTask.run(long startTimeNs)` method is called `warmUpIterations` times (as specified in `JLBHOptions`). The `startTimeNs` passed is typically just `System.nanoTime()` for this phase. -** The primary goal is to allow the Java Virtual Machine (JVM) to perform Just-In-Time (JIT) compilation and optimize the benchmarked code paths. -** It also helps in bringing caches to a "warm" state and letting the system reach a more stable performance profile. -* **Sample Discarding**: Latency samples generated during the warmup phase are usually recorded into the histograms but are *not* part of the final reported results. The histograms are reset after this phase. -* **Task Notification**: After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. -* **Optional Pause**: If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. - -== Execution (Measurement Runs) - -This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): - -* **Per Run**: -** The following steps are repeated for each configured run. -** **Iteration Loop**: The `JLBH` orchestrator executes a loop for the number of `iterations` specified in `JLBHOptions`. -*** **Start Time Calculation**: For each iteration, JLBH calculates an ideal `startTimeNs`. This calculation considers: -**** The configured `throughput` (e.g., messages per second). -**** The `LatencyDistributor` strategy, if one is set, to potentially vary the inter-iteration delay. -**** If `accountForCoordinatedOmission` is true (the default), `startTimeNs` represents the ideal scheduled time for the operation. JLBH will then busy-wait (spin) until `System.nanoTime()` reaches this `startTimeNs`, ensuring the task is dispatched as close as possible to its intended schedule, thus accounting for delays that might otherwise be missed. -*** **Task Invocation**: The `JLBHTask.run(startTimeNs)` method is invoked with the calculated (and potentially waited-for) `startTimeNs`. -*** **Sample Recording**: -**** Within the `JLBHTask.run()` method, the user's code is executed. -**** To record the primary end-to-end latency for the iteration, the task must call `jlbh.sample(System.nanoTime() - startTimeNs)`. -**** For any custom probes registered during setup, the task can call `myProbe.sampleNanos(duration)` to record latencies for specific sub-stages. -**** Each recorded sample is added to the respective `Histogram` associated with the sampler (either the main JLBH sampler or a custom probe's sampler). -** **Run Completion**: After all iterations for the current run are finished: -*** The `JLBHTask.runComplete()` method is invoked. -*** Statistics for this specific run (e.g., percentiles derived from the histograms) are typically printed to the configured `PrintStream`. -*** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. -*** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. - -== Reporting - -After all measurement runs are completed, the aggregated results are finalized and made available: - -* **Final Summary**: A comprehensive summary, often comparing results across all runs and highlighting percentile variations, is printed to the `PrintStream`. -* **Result Object Creation**: The final, immutable `JLBHResult` object is constructed. This object encapsulates all recorded data for all probes across all runs. -* **Result Consumption**: If a `JLBHResultConsumer` was provided during JLBH setup, its `accept(JLBHResult)` method is called, passing the `JLBHResult` object. This allows for programmatic access to the detailed results, for example, for assertion in automated tests or for custom serialization. -* **External Tools**: Users can further process the `JLBHResult` using utilities like: -** `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. -** `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. - -== Cleanup - -The final phase ensures that any resources are properly released: - -* **Task Cleanup**: The `JLBHTask.complete()` method is called, allowing the user's benchmark task to perform any necessary cleanup (e.g., closing files, releasing network connections). -* **OS Jitter Monitor**: If the `OSJitterMonitor` thread was started, it is signaled to terminate. -* **Affinity Release**: If a CPU affinity lock was acquired by JLBH at the start, it is released. diff --git a/src/main/adoc/decision-log.adoc b/src/main/adoc/decision-log.adoc deleted file mode 100644 index 96521f89..00000000 --- a/src/main/adoc/decision-log.adoc +++ /dev/null @@ -1,214 +0,0 @@ -= Chronicle JLBH - Decision Log -Chronicle Software -:toc: -:lang: en-GB -:source-highlighter: rouge - -This document records key architectural and project choices made during the development of Chronicle JLBH. -Identifiers use the `JL` project scope with Nine-Box tags (FN, NF-P, NF-O, DOC, OPS, RISK, TEST, UX) and link back to the requirements in `project-requirements.adoc`. - -== Decision Index - -* link:#JL-DOC-001[JL-DOC-001 Adopt AsciiDoc for project documentation] -* link:#JL-FN-002[JL-FN-002 Single-threaded harness with JLBHTask lifecycle] -* link:#JL-NF-P-003[JL-NF-P-003 Default coordinated-omission compensation and histograms] -* link:#JL-OPS-004[JL-OPS-004 OS jitter tracking as an optional probe] -* link:#JL-FN-005[JL-FN-005 Additional probes and NanoSampler API] -* link:#JL-OPS-006[JL-OPS-006 CSV serialisation and result retention] -* link:#JL-OPS-007[JL-OPS-007 CI integration via TeamCity statistics] - -[[JL-DOC-001]] -=== [JL-DOC-001] Adopt AsciiDoc for project documentation - -Date:: 2025-05-24 -Context:: -* Project requires easily maintainable, version-controlled documentation. -* Contributors need a text format that is readable in raw form and convertible to HTML. -Decision Statement:: -* Adopt AsciiDoc as the canonical format for all project documentation. -Alternatives Considered:: -* Markdown: -** *Description:* Common lightweight markup; widely supported. -** *Pros:* Familiar to many developers; simple syntax. -** *Cons:* Limited features for complex documents; inconsistent rendering across tools. -* Plain text files: -** *Description:* Minimalistic approach with no markup. -** *Pros:* Simplest possible format; no tooling required. -** *Cons:* Hard to express structure or cross references; not suitable for large docs. -Rationale for Decision:: -* AsciiDoc provides structured markup without sacrificing readability. -* The format integrates well with our build tooling and supports the Nine-Box tagging used in requirements. -Impact & Consequences:: -* Contributors must learn basic AsciiDoc syntax. -* Build pipeline processes `.adoc` files to publish HTML. -* Some external tools may expect Markdown, requiring conversion. -Notes/Links:: -** https://asciidoctor.org[AsciiDoc project page] - -[[JL-FN-002]] -=== [JL-FN-002] Single-threaded harness with JLBHTask lifecycle - -Date:: 2025-11-14 -Context:: -* JLBH targets low-latency Java applications where benchmark overhead and scheduling noise must be minimised. -* The harness needs a clear lifecycle so that warm-up, steady-state runs and shutdown can be expressed consistently across benchmarks. -* Requirements: FN-001 Latency Sampling, FN-006 Event Loop Integration, NF-P-001 Performance, NF-O-001 Portability. -Decision Statement:: -* Run the JLBH harness logic on a single dedicated thread, driving user code through the `JLBHTask` lifecycle (`init`, `run`, `warmedUp`, `runComplete`, `complete`). -* Allow the harness thread to be supplied by the caller via `eventLoopHandler(EventLoop)` while preserving the single-threaded execution model. -Alternatives Considered:: -* Multi-threaded harness with worker pools :: -** *Pros:* Could saturate multi-core CPUs more easily in some scenarios. -** *Cons:* Adds scheduling variability, complicates reasoning about pauses and makes per-iteration latency harder to interpret. -* Delegating to a generic micro-benchmark framework (e.g. JMH) :: -** *Pros:* Reuses existing tooling and reporting. -** *Cons:* Less control over in-context execution, fewer hooks for probes and OS jitter, and different lifecycle assumptions than JLBH requires. -Rationale for Decision:: -* A single harness thread keeps timing behaviour predictable and avoids extra scheduling noise beyond what the benchmarked code introduces. -* The explicit lifecycle methods match the phases described in the requirements and README, making it easy to structure complex benchmarks. -Impact & Consequences:: -* Benchmarked code must respect the single-threaded driver model, spawning its own threads only when needed and understanding the impact on measurements. -* Event-loop integration is explicit: users can install JLBH onto an existing loop without changing the underlying lifecycle. -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:benchmark-lifecycle.adoc[Benchmark lifecycle overview] - -[[JL-NF-P-003]] -=== [JL-NF-P-003] Default coordinated-omission compensation and high-resolution histograms - -Date:: 2025-11-14 -Context:: -* Co-ordinated omission can hide worst-case latency by skipping samples during pauses or back-pressure. -* JLBH must support high-resolution percentile analysis at extreme tails to satisfy low-latency users. -* Requirements: FN-001 Latency Sampling, FN-002 Coordinated-Omission Compensation, NF-P-001 Performance. -Decision Statement:: -* Enable compensation for co-ordinated omission by default via `accountForCoordinatedOmission(true)`, with an option to disable it when raw uncorrected timings are required. -* Use high-resolution histograms as the internal representation for recorded latencies so that percentiles can be reported accurately across a wide range of values. -Alternatives Considered:: -* No coordinated-omission compensation :: -** *Pros:* Simpler mental model; raw timings only. -** *Cons:* Under-reports tail latency in back-pressured systems, undermining the harness goals. -* External correction tooling applied after the run :: -** *Pros:* Keeps the core harness simpler. -** *Cons:* Splits logic across tools, increases risk of misconfiguration and makes it harder to reason about what the reported percentiles mean. -Rationale for Decision:: -* Default-on correction matches the expectations set in the JLBH documentation and literature on latency measurement accuracy. -* High-resolution histograms are a proven structure for representing latency distributions with minimal overhead. -Impact & Consequences:: -* Users must be aware that reported percentiles are corrected unless they explicitly disable the feature. -* Benchmark authors can rely on accurate tail reporting without bolting on extra tooling. -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:../README.adoc[Chronicle JLBH README] - -[[JL-OPS-004]] -=== [JL-OPS-004] OS jitter tracking as an optional probe - -Date:: 2025-11-14 -Context:: -* Kernel scheduling, interrupts and other background activity can introduce jitter that affects latency measurements. -* Users often need to distinguish between pauses caused by their code and pauses caused by the operating system or environment. -* Requirements: FN-004 OS Jitter Tracking, NF-O-001 Portability, NF-R-001 Reliability. -Decision Statement:: -* Provide an optional OS jitter probe implemented as a background thread that records scheduler delays beyond a configured threshold, and summarise the results alongside core latency histograms. -* Enable OS jitter tracking by default, with configuration options such as `recordOSJitter(false)` to disable it when the overhead is not acceptable. -Alternatives Considered:: -* Rely on external OS-level tools or profilers :: -** *Pros:* No extra threads or complexity inside the harness. -** *Cons:* Harder to correlate jitter with benchmark iterations; adds extra setup steps for users. -* Always disabling jitter tracking in the core library :: -** *Pros:* Zero overhead from jitter measurement. -** *Cons:* Removes a key diagnostic feature that helps interpret unexpected latency spikes. -Rationale for Decision:: -* Integrating jitter measurement directly into the harness keeps benchmark setup simple and ensures results remain correlated with the workload. -* Making the feature optional and configurable manages overhead while preserving a sensible default for investigation. -Impact & Consequences:: -* When enabled, benchmarks incur some additional overhead from the jitter thread; users can trade this off against diagnostic value. -* CI and local runs can use the jitter probe to explain outliers and refine environment configuration (CPU affinity, power settings). -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:../README.adoc#_additional_features[Additional features in the README] - -[[JL-FN-005]] -=== [JL-FN-005] Additional probes and NanoSampler API - -Date:: 2025-11-14 -Context:: -* Benchmarks often need to record latencies for sub-stages (for example serialisation, network calls) without distorting the end-to-end latency distribution. -* JLBH must make it easy to capture these timings while keeping the main histogram focused on overall request latency. -* Requirements: FN-003 Additional Probes, NF-P-001 Performance, NF-UX-001 Usability. -Decision Statement:: -* Provide `addProbe(String name)` on `JLBH` that returns a `NanoSampler`, each backed by its own histogram and reported alongside the primary distribution. -* Keep the end-to-end `sample` / `sampleNanos` API unchanged so that additional probes never alter the core latency measurements. -Alternatives Considered:: -* Require users to maintain their own histograms outside the harness :: -** *Pros:* No extra API surface in JLBH. -** *Cons:* More boilerplate; higher risk of inconsistent reporting or incompatible histogram settings. -* Fold sub-stage timings into the primary histogram :: -** *Pros:* Single distribution to manage. -** *Cons:* Hides distinctions between phases and makes diagnosis of slow sub-components difficult. -Rationale for Decision:: -* A dedicated probe API gives clear separation between overall latencies and sub-stage metrics, supporting richer analysis without complicating the main flow. -* `NanoSampler` provides a lightweight abstraction that integrates cleanly with the existing sampling model. -Impact & Consequences:: -* Each additional probe incurs extra memory and processing cost; users should add probes selectively for meaningful stages. -* Documentation and examples must show how to name probes consistently so that dashboards and CI jobs can rely on stable identifiers. -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:../README.adoc#_additional_features[Additional features in the README] - -[[JL-OPS-006]] -=== [JL-OPS-006] CSV serialisation and result retention - -Date:: 2025-11-14 -Context:: -* Users need to archive benchmark results for later analysis, trend tracking and comparison across builds or environments. -* Plain console tables are useful for humans but awkward for automated tooling and external analytics systems. -* Requirements: FN-005 CSV Serialisation, NF-O-001 Portability, NF-R-001 Reliability. -Decision Statement:: -* Expose an immutable `JLBHResult` model for completed runs and provide `JLBHResultSerializer` to write results to CSV with a stable schema. -* Default to a simple file layout suitable for CI artefacts, while allowing callers to direct output to custom streams or paths. -Alternatives Considered:: -* Rely solely on console output :: -** *Pros:* No extra types or wiring. -** *Cons:* Difficult to parse reliably; fragile across formatting changes. -* Use a binary format (for example compressed histograms) only :: -** *Pros:* More compact; potentially faster to write. -** *Cons:* Harder to inspect manually; requires custom tooling for every consumer. -Rationale for Decision:: -* CSV strikes a balance between human readability and machine processing, working well with spreadsheets, scripting languages and CI systems. -* A dedicated serializer centralises schema decisions so that downstream tools can depend on consistent column names and ordering. -Impact & Consequences:: -* Schema changes to CSV output must be versioned and documented carefully to avoid breaking consumers. -* Large or long-running benchmarks may produce sizeable CSV files; operations teams should manage retention and storage policies. -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:../README.adoc[Chronicle JLBH README] - -[[JL-OPS-007]] -=== [JL-OPS-007] CI integration via TeamCity statistics - -Date:: 2025-11-14 -Context:: -* Many users run JLBH benchmarks in CI and need to gate builds on latency regressions or track trends over time. -* TeamCity provides a native mechanism (`##teamcity` service messages) for publishing numerical statistics from builds. -* Requirements: Product Functions (CI Metrics), NF-O-001 Portability, NF-R-001 Reliability. -Decision Statement:: -* Provide helper code to emit key JLBH metrics (for example selected percentiles and throughput) as TeamCity statistics lines, using predictable metric names derived from benchmark configuration. -* Treat CI integration as an optional layer that can be enabled or disabled without affecting the core harness behaviour. -Alternatives Considered:: -* Expect users to write their own CI integration scripts around console output or CSV files :: -** *Pros:* Keeps the core library smaller; no explicit dependency on CI conventions. -** *Cons:* Duplicated effort across projects; higher risk of inconsistent parsing and metrics. -* Integrate tightly with a single CI provider only :: -** *Pros:* Deep integration for that platform. -** *Cons:* Locks users into a specific CI and complicates usage elsewhere. -Rationale for Decision:: -* A lightweight TeamCity helper covers a common use case without constraining users who rely on other CI platforms. -* Publishing statistics directly simplifies setting latency thresholds and visualising trends in existing dashboards. -Impact & Consequences:: -* CI pipelines can fail fast when configured thresholds are exceeded, turning JLBH results into actionable quality gates. -* Additional documentation must describe metric naming conventions so that teams can map them to their CI configuration. -Notes/Links:: -* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] -* link:../README.adoc[Chronicle JLBH README] diff --git a/src/main/adoc/jlbh-cookbook.adoc b/src/main/adoc/jlbh-cookbook.adoc deleted file mode 100644 index 47749eac..00000000 --- a/src/main/adoc/jlbh-cookbook.adoc +++ /dev/null @@ -1,553 +0,0 @@ -= Chronicle JLBH Cookbook -Chronicle Software -:toc: -:sectnums: -:lang: en-GB -:source-highlighter: rouge - -A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. -These recipes aim to provide practical starting points for your own benchmarks. -:sectnums: - -== Timing a Simple Method Call - -This is the most basic use case: measuring the latency of a self-contained Java method. - -*Goal*: Measure the execution time of `myMethod()`. - -*Implement the task (`CallBenchmark.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHTask; - -public class CallBenchmark implements JLBHTask { - private JLBH jlbh; - private int counter = 0; // Example state - - @Override - public void init(JLBH jlbh) { - this.jlbh = jlbh; - // Initialize any resources needed by myMethod() here - } - - @Override - public void run(long startTimeNs) { - myMethod(counter++); // Pass some varying input if appropriate - jlbh.sample(System.nanoTime() - startTimeNs); - } - - private void myMethod(int input) { - // Work to be timed. - // This should ideally be significant enough to dominate the - // overhead of System.nanoTime() and jlbh.sample(). - // For very short methods, consider batching or alternative microbenchmark tools. - String.valueOf(input).hashCode(); // Example of a small piece of work - } - - @Override - public void complete() { - // Cleanup resources if any were initialized - } -} ----- - -*Set up and run the harness (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; - -public class MainApp { - public static void main(String[] args) { - JLBHOptions options = new JLBHOptions() - .warmUpIterations(20_000) // Allow for JIT compilation and steady state - .iterations(200_000) // Number of measurements per run - .throughput(100_000) // Target operations per second - .jlbhTask(new CallBenchmark()); - new JLBH(options).start(); - } -} ----- - -*Key Considerations*: -* Ensure the work within `myMethod()` is the primary focus. -Avoid including unrelated logic within the timed section. -* The `throughput` setting influences how often `run()` is called. -If `myMethod()` is very fast, high throughput might be achievable. -If it's slower, adjust throughput accordingly. - -== Measuring a Network Round Trip (UDP Loopback) - -This recipe demonstrates timing a UDP message send and receive on loopback. - -*Goal*: Measure the latency of sending a UDP packet and receiving a response. - -*Prerequisite*: A UDP echo server must be running on `localhost:8080`. -A simple echo server (`SimpleUDPEchoServer.java`): - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import java.net.DatagramPacket; -import java.net.DatagramSocket; -import java.io.IOException; - -public class SimpleUDPEchoServer { - public static void main(String[] args) throws IOException { - System.out.println("UDP Echo Server starting on port 8080..."); - try (DatagramSocket socket = new DatagramSocket(8080)) { - byte[] buffer = new byte[1024]; - DatagramPacket packet = new DatagramPacket(buffer, buffer.length); - while (true) { - socket.receive(packet); - // Echo back to sender - socket.send(new DatagramPacket(packet.getData(), packet.getLength(), packet.getAddress(), packet.getPort())); - } - } - } -} ----- - -TIP: Run this server in a separate process before starting the JLBH client. - -*Implement the task (`LoopbackBenchmark.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHTask; - -import java.io.IOException; -import java.io.UncheckedIOException; -import java.net.InetSocketAddress; -import java.nio.ByteBuffer; -import java.nio.channels.DatagramChannel; - -public class LoopbackBenchmark implements JLBHTask { - private JLBH jlbh; - private DatagramChannel channel; - // Using a direct ByteBuffer can offer performance benefits for I/O - private final ByteBuffer outBuffer = ByteBuffer.allocateDirect(32); - private final ByteBuffer inBuffer = ByteBuffer.allocateDirect(32); - - @Override - public void init(JLBH jlbh) { - this.jlbh = jlbh; - try { - channel = DatagramChannel.open(); - channel.configureBlocking(true); // For simplicity; non-blocking could also be used - channel.connect(new InetSocketAddress("localhost", 8080)); - } catch (IOException e) { - throw new UncheckedIOException(e); // Consider more robust error handling - } - } - - @Override - public void run(long startTimeNs) { - try { - outBuffer.clear(); - outBuffer.putInt(0, (int) startTimeNs); // Send some data (optional) - outBuffer.limit(4); - channel.write(outBuffer); - - inBuffer.clear(); - channel.read(inBuffer); // Wait for echo - } catch (IOException e) { - // In a real benchmark, decide how to handle I/O errors. - // Re-throwing might halt the benchmark or skew results if not all iterations complete. - throw new UncheckedIOException(e); - } - jlbh.sample(System.nanoTime() - startTimeNs); - } - - @Override - public void complete() { - try { - if (channel != null) { - channel.close(); - } - } catch (IOException e) { - e.printStackTrace(); - } - } -} ----- - -*Set up the harness (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.core.Jvm; -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; - -import java.io.IOException; - - -public class MainApp { - public static void main(String[] args) { - // First, ensure SimpleUDPEchoServer is running. - // new Thread(() -> { try { SimpleUDPEchoServer.main(null); } catch (IOException e) { e.printStackTrace(); }}).start(); - // Jvm.pause(100); // Give server a moment to start if launched programmatically - - JLBHOptions options = new JLBHOptions() - .warmUpIterations(10_000) - .iterations(50_000) - // Lower throughput for network tests helps avoid queuing in network buffers, - // giving a clearer picture of the actual round-trip time. - .throughput(1_000) // Reduced from 10_000 for more stable network results - // Coordinated Omission is crucial for network tests to capture true send-to-receive latency - .accountForCoordinatedOmission(true) - .jlbhTask(new LoopbackBenchmark()); - new JLBH(options).start(); - } -} ----- - -*Key Considerations*: - -* For true network latency, run client and server on different machines connected via a relevant network. -* Consider TCP for connection-oriented tests; its performance characteristics (e.g., Nagle's algorithm, ACK delays) differ from UDP. -* Packet loss and retransmissions (for UDP, if implementing reliability) can heavily affect latency distributions. - -== Running Within an Existing Event Loop - -JLBH can integrate with Chronicle's event loop mechanism, useful for benchmarking components that are already part of an event-driven architecture. - -*Goal*: Execute a JLBH benchmark task on a pre-existing `EventLoop`. - -*Implement the task (`EventLoopTask.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHTask; - -public class EventLoopTask implements JLBHTask { - private JLBH jlbh; - - @Override - public void init(JLBH jlbh) { - this.jlbh = jlbh; - System.out.println("EventLoopTask initialized on thread: " + Thread.currentThread().getName()); - } - - @Override - public void run(long startTimeNs) { - // Perform work specific to the event loop context - jlbh.sample(System.nanoTime() - startTimeNs); - } - - @Override - public void warmedUp() { - System.out.println("EventLoopTask warmed up on thread: " + Thread.currentThread().getName()); - } -} ----- - -*Set up the harness (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.core.threads.EventLoop; -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; -import net.openhft.chronicle.jlbh.JLBHTask; -import net.openhft.chronicle.threads.MediumEventLoop; -import net.openhft.chronicle.threads.Pauser; - -public class MainApp { - public static void main(String[] args) { - // MediumEventLoop and Pauser are part of Chronicle Core/Threads - // Pauser.balanced() tries to balance latency and CPU usage. - EventLoop eventLoop = new MediumEventLoop(null, "myBenchmarkLoop", Pauser.balanced(), true, null); - eventLoop.start(); // Start the event loop thread - - JLBHTask task = new EventLoopTask(); - JLBHOptions options = new JLBHOptions() - .warmUpIterations(10_000) // Warmup happens on the event loop too - .iterations(100_000) - .throughput(50_000) - // accountForCoordinatedOmission defaults to true, which is compatible and required - // for eventLoopHandler() to function as expected. - .jlbhTask(task); - - JLBH jlbh = new JLBH(options); - // Instead of jlbh.start(), install handlers onto the event loop - jlbh.eventLoopHandler(eventLoop); - - // Keep the main thread alive or wait for completion if necessary. - // For this example, we'll let it run for a bit then stop the loop. - // In a real app, the event loop would run for the application's lifetime. - try { - Thread.sleep(15_000); // Let benchmark run - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } finally { - eventLoop.close(); // Cleanly shut down the event loop - } - } -} ----- - -*Key Considerations*: - -* The `eventLoopHandler` method installs handlers that drive the benchmark lifecycle on the `EventLoop`'s thread. -* This is useful when the code being benchmarked is designed to run on a specific event loop. -* `JLBHOptions.accountForCoordinatedOmission(true)` (the default) is essential for `eventLoopHandler` to function correctly. ---- - -== Using Multiple Probes for Staged Operations - -JLBH allows timing multiple stages within a single benchmark iteration using custom probes. - -*Goal*: Measure total operation latency and the latency of two distinct internal stages. - -*Implement the task (`MultiStageBenchmark.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.core.util.NanoSampler; -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHTask; - -import java.util.concurrent.TimeUnit; - -public class MultiStageBenchmark implements JLBHTask { - private JLBH jlbh; - private NanoSampler stage1Sampler; - private NanoSampler stage2Sampler; - - @Override - public void init(JLBH jlbh) { - this.jlbh = jlbh; - // Register custom probes for each stage - this.stage1Sampler = jlbh.addProbe("Stage1_Processing"); - this.stage2Sampler = jlbh.addProbe("Stage2_Persistence"); - } - - @Override - public void run(long startTimeNs) { - // Stage 1 - long stage1StartNs = System.nanoTime(); - performStage1Work(); - stage1Sampler.sampleNanos(System.nanoTime() - stage1StartNs); - - // Stage 2 - long stage2StartNs = System.nanoTime(); - performStage2Work(); - stage2Sampler.sampleNanos(System.nanoTime() - stage2StartNs); - - // Sample total end-to-end time - jlbh.sample(System.nanoTime() - startTimeNs); - } - - private void performStage1Work() { - // Simulate some processing work, e.g., data transformation - try { - TimeUnit.NANOSECONDS.sleep(100 + (System.nanoTime() % 50)); // Variable work - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - - private void performStage2Work() { - // Simulate some I/O or persistence work - try { - TimeUnit.NANOSECONDS.sleep(150 + (System.nanoTime() % 70)); // Variable work - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } -} ----- - -*Set up the harness (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; - -public class MainApp { - public static void main(String[] args) { - JLBHOptions options = new JLBHOptions() - .warmUpIterations(15_000) - .iterations(100_000) - .throughput(20_000) // Adjust based on expected stage latencies - .jlbhTask(new MultiStageBenchmark()); - new JLBH(options).start(); - } -} ----- - -*Key Considerations*: - -* The sum of individual probe latencies might not exactly equal the end-to-end latency due to the overhead of `System.nanoTime()` calls between stages. -* This approach is excellent for identifying bottlenecks within a larger operation. -* The JLBH output will include separate percentile summaries for "Stage1_Processing", "Stage2_Persistence", and the default "end to end" probe. - -''' - -== Recording and Exporting Results - -JLBH allows programmatic access to results and provides utilities for serialization, such as to CSV. - -*Goal*: Run a benchmark, retrieve its results, and save them to a CSV file. - -*Implement the task*: We can reuse the `CallBenchmark` from Recipe 1. - -[source,java] ----- -// Re-use CallBenchmark.java from Recipe 1 -package net.openhft.chronicle.jlbh.cookbook; -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHTask; -public class CallBenchmark implements JLBHTask { ... } ----- - -*Set up the harness and process results (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; -import net.openhft.chronicle.jlbh.JLBHResult; -import net.openhft.chronicle.jlbh.JLBHResultConsumer; -import net.openhft.chronicle.jlbh.util.JLBHResultSerializer; - -import java.io.IOException; -import java.time.Duration; - -public class MainApp { - public static void main(String[] args) { - JLBHResultConsumer resultConsumer = JLBHResultConsumer.newThreadSafeInstance(); - - JLBHOptions options = new JLBHOptions() - .warmUpIterations(10_000) - .iterations(50_000) - .throughput(100_000) - .jlbhTask(new CallBenchmark()) // Using CallBenchmark from Recipe 1 - .runs(2); // Multiple runs to see more in the result - - // Pass the consumer to the JLBH constructor - new JLBH(options, System.out, resultConsumer).start(); - - // After JLBH completes, retrieve the results - JLBHResult result = resultConsumer.get(); - - if (result != null) { - // Programmatically access results (example: last run's 99th percentile) - JLBHResult.RunResult lastRunEndToEnd = result.endToEnd().summaryOfLastRun(); - Duration p99 = lastRunEndToEnd.get99thPercentile(); - System.out.printf("Last run's End-to-End 99th Percentile: %d ns%n", p99.toNanos()); - - // Serialize all probe results to CSV - try { - String csvFileName = "benchmark_results.csv"; - JLBHResultSerializer.runResultToCSV(result, csvFileName, result.probeNames(), true); - System.out.println("Benchmark results saved to " + csvFileName); - // The CSV file will contain: - // ProbeName,50th p-le,90th p-le,99th p-le,999th p-le,9999th p-le,Worst - // endToEnd,... - // (any other custom probes)... - // OSJitter,... (if enabled) - } catch (IOException e) { - System.err.println("Error saving results to CSV: " + e.getMessage()); - e.printStackTrace(); - } - } else { - System.err.println("Benchmark did not produce results."); - } - } -} ----- - -*Key Considerations*: - -* `JLBHResultConsumer.newThreadSafeInstance()` is important if you plan to access results from a different thread than the one running `JLBH.start()`, though in this synchronous example it primarily serves as the hook. -* The `JLBHResult` object is immutable and contains rich data for all probes and runs. -* `JLBHResultSerializer` provides a convenient way to get data out for external analysis. - -== Enabling and Understanding OS Jitter Measurement - -Operating system jitter can significantly impact low-latency applications. -JLBH can measure this independently. - -*Goal*: Configure JLBH to record OS jitter and understand its output. - -*Implement the task*: Any simple task will do, as OS jitter is measured by a background thread. -We'll reuse `CallBenchmark`. - -[source,java] ----- -// Re-use CallBenchmark.java from Recipe 1 ----- - -*Set up the harness with OS Jitter options (`MainApp.java`)*: - -[source,java] ----- -package net.openhft.chronicle.jlbh.cookbook; - -import net.openhft.chronicle.jlbh.JLBH; -import net.openhft.chronicle.jlbh.JLBHOptions; - -public class MainApp { - public static void main(String[] args) { - JLBHOptions options = new JLBHOptions() - .warmUpIterations(5_000) // Shorter for quick demo - .iterations(10_000) // Shorter for quick demo - .throughput(50_000) - .jlbhTask(new CallBenchmark()) - // OS jitter recording is enabled by default; use recordOSJitter(false) - // if you want to disable the background monitor. - // Optional: Only record jitter events greater than a specific threshold (default 1_000 ns) - // .recordJitterGreaterThanNs(5_000) - // Optional: Pin the OS Jitter monitoring thread to a specific CPU core (if supported) - // .jitterAffinity(true) // Be cautious with affinity, ensure free isolated cores - .runs(1); - - new JLBH(options).start(); - } -} ----- - -*Interpreting OS Jitter Results*: - -* JLBH records OS jitter by default and starts a background thread that continuously measures the time elapsed between calls to `System.nanoTime()`. -Use `recordOSJitter(false)` only when you want to suppress this monitoring. -* If this measured gap exceeds `recordJitterGreaterThanNs`, it's considered an OS-induced pause/jitter event and is recorded in a special "OS Jitter" probe. -* The JLBH output (both console and `JLBHResult`) will include a section for the "OS Jitter" probe, showing its own percentile distribution. -* These jitter values represent pauses that are *not* caused by your application code but by the operating system or other processes interfering with the JLBH process's execution. -* High OS jitter values (e.g., milliseconds) can explain unexpected latency spikes in your main application probes, even if your application code itself is highly optimized. - -*Key Considerations*: - -* OS jitter measurement adds some (usually small) overhead. -Enable it when you suspect external factors are influencing your benchmark. -* The `recordJitterGreaterThanNs` option helps filter out minor, expected system noise. -* `jitterAffinity(true)` can provide more stable jitter measurements if you have isolated CPU cores available, but can be detrimental otherwise. -* Correlating high OS jitter percentiles with high application latency percentiles can be a powerful diagnostic tool. diff --git a/src/main/adoc/project-requirements.adoc b/src/main/adoc/project-requirements.adoc deleted file mode 100644 index 86a10845..00000000 --- a/src/main/adoc/project-requirements.adoc +++ /dev/null @@ -1,206 +0,0 @@ -= Chronicle JLBH - Software Requirements Specification -Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 -:toc: -:lang: en-GB -:source-highlighter: rouge -:sectnums: - -*_Abstract_*:: -Chronicle JLBH is an open-source Java library, released under the Apache Licence 2.0, that provides a high-resolution latency benchmark harness. -It allows developers to _measure_, _analyse_ and _regress-test_ latency behaviour of critical code paths "in context" rather than via isolated micro-benchmarks. -Key features include compensation for *co-ordinated omission*, configurable throughput modelling, additional probes, optional operating-system jitter tracking, and serialisation of results for CI dashboards. - -== Introduction - -=== Purpose -This document specifies functional and non-functional requirements for Chronicle JLBH, ensuring that contributors and integrators share a common understanding of the product's goals and constraints. - -=== Scope -The harness targets _low-latency_ Java applications (trading systems, micro-services, real-time analytics) that must quantify latencies at extreme percentiles under realistic load patterns. - -=== Definitions, Acronyms and Abbreviations - -|=== -| *Term* | *Definition* - -| JLBH | Java Latency Benchmark Harness - the library under specification. -| Co-ordinated Omission | Measurement error whereby pauses/back-pressure hide worst-case latencies. -| Probe | A named histogram that records an additional timing within the benchmark workflow. -| OS Jitter | Kernel or scheduler-induced timing variance measured by a background thread. -| CI | Continuous Integration environment (e.g. GitHub Actions, TeamCity). -|=== - -=== References - -* Upstream repository: https://github.com/OpenHFT/JLBH[GitHub - OpenHFT/JLBH] -* API reference: https://javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/core/jlbh/JLBH.html[Javadoc] -* Tutorial series: http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[RationalJava blog series] -* Discussion of _co-ordinated omission_: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE[m.s. thread] -* Chronicle blog post on JLBH in event loops: https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html[Vanilla-Java article] - -=== Overview -Sections 2-5 describe the product context, interfaces, detailed system features, and quality attributes. -Section 6 captures licensing, whilst Section 7 presents the glossary. - -== Overall Description - -=== Product Perspective -JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. - -=== Product Functions - -* Generate high-resolution histograms (>=35 bits) of end-to-end sample latencies. -* Optionally compensate for *co-ordinated omission* by adjusting the synthetic start time. -* Model load via configurable `throughput(int, TimeUnit)` and *LatencyDistributor*s. -* Provide additional named probes via `addProbe(String)`. -* Record OS-level jitter and present it as a separate probe. -* Output run summaries to `PrintStream` and expose structured results via `JLBHResult`. -* Serialise the final run to CSV (`JLBHResultSerializer`). -* Integrate with CI (TeamCity helper emits `##teamcity` statistics lines). -* Allow execution from any thread or installation onto a Chronicle *MediumEventLoop*. - -=== User Classes and Characteristics - -|=== -| *Actor* | *Description* | *Technical Expertise* - -| Performance Engineer | Designs latency benchmarks and analyses histograms. | Advanced Java & performance tuning. -| Developer | Adds JLBH tests to codebase, evaluates pull-request regressions. | Intermediate Java. -| CI System | Runs automated latency regression suite, ingests CSV or TeamCity stats. | N/A (automation). -|=== - -=== Operating Environment - -* Java 11 LTS and later; Java 17 recommended for current builds. -* Linux x86-64 (primary), other POSIX OSes supported; macOS usable but CI baselines exclude it. -* Typical CPU affinity facilities available via *OpenHFT Affinity* library. - -=== Design and Implementation Constraints - -* Single-threaded harness execution; benchmarked code may span threads. -* Measurements rely on `System.nanoTime()` precision; hardware timers must be invariant. -* JVM must be started with `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when using async-profiler concurrently. -* Library is released under the Apache Licence 2.0; contributions must comply with CLA policy. - -=== Assumptions and Dependencies - -* SUT jars are available on the same JVM classpath. -* Benchmark host clocks are monotonic and not subjected to aggressive power-saving states. -* CI agents provision exclusive CPU cores to minimise jitter. - -== External Interface Requirements - -=== Public API - -* `JLBHOptions` - builder for benchmark parameters (throughput, warm-up, runs, etc.). -* `JLBHTask` - user-defined callback with lifecycle methods `init`, `run`, `warmedUp`, `runComplete`, `complete`. -* `JLBH` - orchestrator class with `start()` and `eventLoopHandler(EventLoop)` entry points. -* `JLBHResult` - immutable projection of measured data. -* `JLBHResultSerializer` - CSV writer for analytics pipelines. - -=== User Interface -CLI usage is demonstrated in _ExampleJLBHMain_ and test fixtures; no dedicated GUI is provided. - -=== Hardware Interfaces -None. The harness interacts with OS timers and CPU affinity via JNI where available. - -=== Software Interfaces - -* Build tool: *Apache Maven* (pom.xml defines BOM imports). -* Logging: SLF4J (test scope). -* Testing: JUnit 4 for unit and integration tests. -* CI Metrics: TeamCity service messages. - -== System Features - -=== FN-001 Latency Sampling -*Description*: Collect nanosecond deltas from user code via `sample(long)` or `sampleNanos(long)`. -*Stimulus/Response*: Upon each iteration, `JLBHTask.run` calls `jlbh.sample(...)`; histogram updates occur; run summary printed at completion. -*Priority*: *Essential*. -*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. - -=== FN-002 Coordinated-Omission Compensation -Ensures that queue back-log does not under-represent tail latency. Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. -*Decision reference*: link:decision-log.adoc#JL-NF-P-003[JL-NF-P-003]. - -=== FN-003 Additional Probes -Developers may time sub-stages (e.g., serialisation, network round-trip) through `addProbe(String)` and record them independently. -*Decision reference*: link:decision-log.adoc#JL-FN-005[JL-FN-005]. - -=== FN-004 OS Jitter Tracking -A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. Results are summarised alongside core latencies. -*Decision reference*: link:decision-log.adoc#JL-OPS-004[JL-OPS-004]. - -=== FN-005 CSV Serialisation -Post-run, results can be persisted for offline analysis (`result.csv` by default). -*Decision reference*: link:decision-log.adoc#JL-OPS-006[JL-OPS-006]. - -=== FN-006 Event Loop Integration -`eventLoopHandler(EventLoop)` allows benchmarks to operate inside Chronicle threading framework, avoiding extra threads. -*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. - -== Non-Functional Requirements - -=== NF-P-001 Performance -* Overhead per sample must remain below 100 ns when no additional probes are active. -* Histogram generation must support >=200 M iterations without heap pressure. - -=== NF-R-001 Reliability -* Harness must abort gracefully on thread interruptions or sample time-outs (`timeout(long)`). -* Immutable result objects ensure thread-safe publication to external consumers. - -=== NF-UX-001 Usability -* Fluent builder API; sensible defaults provide a runnable benchmark in ≤10 LOC. -* ASCII table outputs are human-readable and CI-friendly. - -=== NF-O-001 Portability -* Pure-Java codebase; no native compilation steps. -* JDK-specific optimisations (e.g., *Zing* support) are runtime-detected. - -=== NF-O-002 Maintainability -* 80 %+ unit-test line coverage with deterministic fixtures. -* Code adheres to Chronicle parent POM style and SonarCloud quality gates. - -=== NF-S-001 Security -No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code. - -== Licensing - -The project is released under the *Apache Licence 2.0* (_see_ `LICENSE.adoc`). -Downstream consumers must preserve licence notices and may include JLBH in commercial or OSS products, subject to the terms therein. - -== Glossary - -Co-ordinated Omission:: Statistical artefact causing under-reporting of worst-case latency. -Histogram:: Data structure that records frequency counts in logarithmic buckets, enabling percentile extraction. -Percentile:: Value below which a given percentage of observations fall (e.g., 99th percentile). - -== Appendix A - Example Minimal Benchmark - -[source,java] ----- -public class NothingBenchmark implements JLBHTask { - private JLBH jlbh; - public void init(JLBH jlbh) { this.jlbh = jlbh; } - public void run(long startTimeNS) { - jlbh.sample(System.nanoTime() - startTimeNS); - } - public static void main(String[] args) { - new JLBH(new JLBHOptions() - .throughput(1_000_000) - .iterations(10_000) - .jlbhTask(new NothingBenchmark())) - .start(); - } -} ----- - -== Appendix B - Footnotes - -* JLBH originated within the Chronicle Software open-source stack and is actively maintained. See https://github.com/OpenHFT/JLBH. -* The API reference highlights the focus on _co-ordinated omission_ and event-loop support. See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. -* A multi-part blog series gives practical guidance on designing realistic latency tests. See http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html. -* Vanilla-Java's micro-services article demonstrates embedding JLBH in an event-driven architecture. See https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html. -* Original discussion of co-ordinated omission by Gil Tene motivates JLBH's default settings. See https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE. diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/adoc/results-interpretation-guide.adoc deleted file mode 100644 index bf4614c7..00000000 --- a/src/main/adoc/results-interpretation-guide.adoc +++ /dev/null @@ -1,158 +0,0 @@ -= JLBH Results Interpretation Guide -Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 -:toc: -:lang: en-GB -:icons: font -:sectnums: -:source-highlighter: rouge - -*_Abstract_*:: -This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. Effective interpretation is key to deriving meaningful insights from your latency benchmarks. - -== Understanding JLBH Console Output Structure - -When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). Understanding this output is the first step: - -* **Warm-up Messages**: Indicates the progress and completion of the warm-up phase (e.g., "Warm up complete (X iterations took Y s)"). -* **Per-Run Detailed Output**: For each measurement run, JLBH prints a section: -** Header indicating the run number (e.g., "BENCHMARK RESULTS (RUN 1) us"). -** Run metadata: Total run time, latency distribution strategy, coordinated omission status, target throughput. -** **Probe Statistics**: For each active probe (the default "End to End" probe, any custom probes added via `jlbh.addProbe()`, and the "OS Jitter" probe if enabled), a line showing: -*** Probe name and total samples recorded for that probe in the run. -*** A compact representation of key percentiles (e.g., "50/90 99/99.9 99.99 - worst was X / Y Z / A B - C"). These values are usually in microseconds. -* **Final Summary Tables**: After all runs are complete, JLBH prints summary tables for each probe: -** Header: "SUMMARY (probe_name) us". -** Rows for key percentiles (50.0, 90.0, 99.0, 99.9, etc., up to 'worst'). -** Columns for each run, showing the latency value for that percentile in that run. -** A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). - -== Interpreting Latency Histograms and Percentiles - -JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. These histograms are then used to derive percentile values. - -* **What is a Percentile?** -** A percentile indicates the value below which a given percentage of observations fall. For example, the 99th percentile latency is the value X where 99% of all measured latencies were less than or equal to X, and 1% were greater. -* **Key Percentiles in JLBH Output**: -** `50.0` (Median): The middle value; 50% of latencies are lower, 50% are higher. It gives a sense of the "typical" latency but can be misleading on its own if the distribution is skewed. -** `90.0`: 90% of requests were faster than this value. -** `99.0` (99th or "p99"): A common metric for Service Level Objectives (SLOs). Indicates that 1 out of 100 requests experienced this latency or worse. -** `99.9` (99.9th or "p999", "three nines"): Important for high-performance systems; 1 out of 1000 requests were this slow or slower. -** `99.99` (99.99th or "p9999", "four nines"): Critical for systems demanding very high consistency in performance. -** `worst`: The single highest latency recorded during the run. This can be affected by outliers but is important to note, especially if it's dramatically different from high percentiles. -* **Why Percentiles Matter**: -** Averages can be heavily skewed by a small number of very low or very high latencies, masking the true experience of most requests. -** Tail latencies (e.g., p99 and above) often drive user-perceived performance and system stability. A system with a good average latency but terrible tail latency can still feel unresponsive or unreliable. -* **Units**: Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. Pay attention to the units indicated in the table headers. -* **Custom Probes**: If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. - -== Co-ordinated Omission (CO) - -Co-ordinated Omission is a critical concept in latency measurement, especially for systems that process requests at a certain rate. - -* **The Problem**: If a system (or the benchmark harness itself) pauses or slows down, it might temporarily stop measuring. When it resumes, it might only measure the (potentially fast) service times of requests that are processed immediately, "omitting" the time that other requests would have spent waiting *during* the pause. This leads to unrealistically optimistic latency figures, particularly at the tail. -* **How JLBH Compensates**: -** JLBH addresses this by default (`accountForCoordinatedOmission = true` in `JLBHOptions`). -** For each iteration, JLBH calculates an *ideal scheduled start time* based on the target throughput. -** It then actively waits (busy-spins using `System.nanoTime()`) until this ideal start time is reached before invoking `JLBHTask.run(startTimeNs)` with that ideal time. -** The latency is then calculated as `System.nanoTime() - idealStartTimeNs`. -** This means if the system or harness was delayed, the measured latency for that sample will correctly include that delay, reflecting the experience a request would have had if it arrived at its scheduled time. -* **Impact on Results**: -** *With CO compensation (default)*: Histograms reflect a more accurate picture of the latency a user or client would experience, including any queuing or delays caused by the system struggling to keep up with the offered load. Tails are usually higher and more realistic. -** *Without CO compensation*: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. This can be useful for understanding raw compute time but not overall system responsiveness under load. -* **Comparison**: Always state whether CO compensation was enabled when comparing benchmark results. Comparing a CO-compensated run with a non-compensated run is misleading. JLBH prints "Correcting for co-ordinated:true/false" in each run's output. - -== OS Jitter Probe - -JLBH can optionally measure Operating System (OS) jitter, which refers to delays caused by the OS scheduler or other system activities interrupting the benchmark process. - -* **Mechanism**: -** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. -** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. -* **Interpreting Jitter Output**: -** The "OS Jitter" probe will have its own percentile table in the JLBH output. -** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." -** *Low jitter values* (e.g., a few microseconds) indicate that the benchmark process was generally ableto run without significant external interference from the OS. -* **Why it's Useful**: -** Helps distinguish between latency caused by your application code versus latency caused by the environment. -** If your application shows high tail latencies, and the OS Jitter probe also shows high values around the same magnitude, it's a strong indicator that external factors are at play. -* **Actions for High Jitter**: -** Ensure the benchmark is running on a quiet machine with minimal other load. -** Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). -** Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). - -== Analysing Run-to-Run Variation - -Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. JLBH's summary output includes a "% Variation" column for each percentile. - -* **The Variation Column**: This shows the percentage difference between the maximum and minimum values recorded for a given percentile across all included runs. -** By default (`JLBHOptions.skipFirstRun = NOT_SET`), if there are more than 3 runs, the first run is often excluded from this variation calculation as it might still be affected by late-stage JIT or other one-off initializations. You can control this explicitly with `skipFirstRun(boolean)`. -* **Interpreting Variation**: -** *Low Variation (e.g., < 5-10%)*: Suggests a stable benchmarking environment and that your application's performance is repeatable under the test conditions. This increases confidence in the results. -** *High Variation*: Indicates instability. Possible causes include: -*** **Insufficient Warm-up**: The JVM might still be optimizing code differently across runs. Try increasing `warmUpIterations`. -*** **Garbage Collection (GC) Pauses**: Infrequent but long GC pauses can hit some runs but not others, drastically affecting tail percentiles. Monitor GC activity. -*** **External System Noise**: Other processes on the machine, network fluctuations (if applicable), or even hardware power management. -*** **Adaptive JVM Behavior**: Some JVM optimizations are adaptive and can change behavior between runs if conditions shift slightly. -*** **Benchmarked Code Instability**: The code itself might have inherent variability or be interacting with unstable external resources. -* **Goal**: Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. Focus on understanding systematic performance rather than chasing the single "best" score from one run. Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. - -== The Throughput vs. Latency Relationship - -A fundamental aspect of performance is the trade-off between throughput (how much work is done per unit of time) and latency (how long each piece of work takes). - -* **JLBH's Role**: JLBH allows you to set a target `throughput` via `JLBHOptions`. This controls the rate at which `JLBHTask.run()` is invoked. -* **Typical Behavior**: -** At low throughputs, a system can often maintain low latency. -** As throughput increases, contention for resources (CPU, memory, network, locks) typically rises. -** Beyond a certain point, latency will start to increase, often non-linearly. Queues may build up, and the system struggles to keep pace. -* **Using JLBH to Explore**: -** Run your benchmark with different `throughput` settings to understand how your application's latency responds to varying load. -** This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. -** The results can inform capacity planning and help set realistic performance expectations. - -== Common Pitfalls in Interpretation - -Avoiding these common mistakes can help ensure your benchmark results are meaningful and your conclusions are sound: - -* **Insufficient Warm-up**: -** *Issue*: The JVM performs JIT compilation, class loading, and cache warming during initial execution. Short or inadequate warm-up can mean you're measuring pre-optimized code or including one-off startup costs. -** *Solution*: Ensure `warmUpIterations` is sufficient. A common rule of thumb is at least the JVM's compilation threshold (e.g., `Jvm.compileThreshold()` in Chronicle Core, typically 10,000-15,000 iterations) plus some buffer for application-specific warm-up. Monitor if results stabilize after a certain number of runs. -* **Ignoring Co-ordinated Omission Setting**: -** *Issue*: Comparing results where one run had CO compensation enabled and another didn't will lead to incorrect conclusions about performance changes. -** *Solution*: Always be aware of the `accountForCoordinatedOmission` setting (default is `true`) and report it alongside your results. -* **Benchmark Overhead**: -** *Issue*: The act of measuring itself (calling `System.nanoTime()`, `jlbh.sample()`) has a small overhead. For extremely fast operations (deep sub-microsecond), this overhead can become a significant percentage of the measured time. -** *Solution*: JLBH is designed for low overhead (aiming for <100ns). Be mindful of this when benchmarking trivial operations. Consider batching very small operations if appropriate, or if the raw speed of an isolated, tiny operation is critical, a more specialized microbenchmarking tool like JMH might be used for that specific part, with JLBH testing the larger context. -* **Unrealistic Throughput Targets**: -** *Issue*: Setting `throughput` far beyond what the system can handle will just measure a system under constant overload, with latencies dominated by queue times. Setting it too low might not expose bottlenecks that appear under moderate load. -** *Solution*: Test a range of throughputs relevant to your expected operational load and capacity limits. -* **Focusing Only on Averages**: -** *Issue*: Averages hide outliers and don't reflect the experience of users hitting tail latencies. -** *Solution*: Prioritize percentile analysis (p50, p90, p99, p99.9, worst) as JLBH encourages. -* **Ignoring Custom Probe Data**: -** *Issue*: If you've set up custom probes for different stages of an operation but only look at the end-to-end numbers, you miss valuable insights into internal bottlenecks. -** *Solution*: Analyze the percentile data for each custom probe. Compare their contributions to the overall latency. -* **Environmental Inconsistency**: -** *Issue*: Running benchmarks on different hardware, with varying background OS/application load, different JVM versions, or different system configurations will yield different results. -** *Solution*: For comparative analysis, always use a consistent, controlled, and quiet environment. Document the environment meticulously. -* **Thread Pinning (Affinity) Issues**: -** *Issue*: On multi-core systems, OS thread scheduling can move your benchmark thread between cores, causing cache misses and jitter. Forgetting to pin, or pinning to a busy/shared core, can skew results. -** *Solution*: Use `JLBHOptions.acquireLock(Affinity::acquireLock)` (from OpenHFT Affinity library) to attempt to pin the main benchmark thread to an isolated core. Similarly for `jitterAffinity` if using the OS Jitter probe. Ensure these cores are genuinely isolated. -* **Not Enough Iterations/Run Duration**: -** *Issue*: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. -** *Solution*: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. This often means runs lasting at least several seconds, or even minutes for deep stability tests. - -== Deriving Actionable Insights - -The ultimate goal of benchmarking is to gain insights that can lead to improvements. - -* **Identify Bottlenecks**: -** High end-to-end latency? Use custom probes to break down the operation and see which stage is the culprit. -** High p99 or p99.9 values? This often points to issues like GC, network spikes, lock contention, or OS jitter. Correlate with OS Jitter probe and GC logs. -* **Validate Optimizations**: Run benchmarks before and after a code change to quantify its impact on latency across different percentiles and throughputs. -* **Capacity Planning**: Determine the throughput your system can handle while meeting latency SLOs. -* **Regression Testing**: Integrate JLBH benchmarks into your CI/CD pipeline to catch performance regressions automatically. -* **Understand Trade-offs**: Use JLBH to explore the impact of different configurations, algorithms, or architectural choices on the latency profile. diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java index d0833540..b52f99cd 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java @@ -20,7 +20,6 @@ * safe cross-thread visibility. *

    */ - public interface JLBHResultConsumer extends Consumer, Supplier { /** From 84d5d1d73643f4e9e383b3dd493eb01313da1b19 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Mon, 24 Nov 2025 10:35:55 +0000 Subject: [PATCH 14/14] Remove unnecessary blank lines in various classes for improved code readability --- TODO.md | 73 +-- src/main/docs/architecture.adoc | 206 +++++++ src/main/docs/benchmark-lifecycle.adoc | 80 +++ src/main/docs/decision-log.adoc | 214 +++++++ src/main/docs/jlbh-cookbook.adoc | 551 ++++++++++++++++++ src/main/docs/project-requirements.adoc | 203 +++++++ .../docs/results-interpretation-guide.adoc | 154 +++++ 7 files changed, 1447 insertions(+), 34 deletions(-) create mode 100644 src/main/docs/architecture.adoc create mode 100644 src/main/docs/benchmark-lifecycle.adoc create mode 100644 src/main/docs/decision-log.adoc create mode 100644 src/main/docs/jlbh-cookbook.adoc create mode 100644 src/main/docs/project-requirements.adoc create mode 100644 src/main/docs/results-interpretation-guide.adoc diff --git a/TODO.md b/TODO.md index a011e0f6..5b0afd7e 100644 --- a/TODO.md +++ b/TODO.md @@ -34,25 +34,25 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD **Feeds into:** ARCH_TODO.md Stage 3 - Module Deep Dives (ARCH-MOD-JLBH) -- [ ] **Core Abstractions:** [List primary abstractions this module provides] -- [ ] **Interactions with other modules:** [Which Chronicle modules does this use/integrate with?] -- [ ] **Typical use cases:** [List 2-3 common scenarios where this module is used] -- [ ] **Performance characteristics:** [Key performance metrics if applicable] -- [ ] **Design patterns used:** [e.g., flyweight, single writer, etc.] +- [ ] [P2] [E:M] **Core Abstractions:** [List primary abstractions this module provides] +- [ ] [P2] [E:M] **Interactions with other modules:** [Which Chronicle modules does this use/integrate with?] +- [ ] [P2] [E:M] **Typical use cases:** [List 2-3 common scenarios where this module is used] +- [ ] [P2] [E:M] **Performance characteristics:** [Key performance metrics if applicable] +- [ ] [P2] [E:M] **Design patterns used:** [e.g., flyweight, single writer, etc.] ### Existing Documentation Audit -- [ ] Check if `src/main/docs/architecture-overview.adoc` exists +- [ ] [P2] [E:S] Check if `src/main/docs/architecture-overview.adoc` exists - [ ] If yes: Review quality (compare to Chronicle-Bytes standard) - [ ] If no: Note as gap for ARCH_TODO Stage 5.5 -- [ ] Check if `src/main/docs/project-requirements.adoc` exists +- [ ] [P2] [E:S] Check if `src/main/docs/project-requirements.adoc` exists - [ ] If yes: Review for ARCH_TODO Stage 1.75 (Requirements Overview) - [ ] If no: Note as gap for FUNC_TODO.md -- [ ] Check if `src/main/docs/decision-log.adoc` exists +- [ ] [P2] [E:S] Check if `src/main/docs/decision-log.adoc` exists - [ ] If yes: Review for ARCH_TODO Stage 1.85 (Decision Log Overview) - [ ] If no: Note as gap for DECISION_TODO.md -- [ ] Check if `README.adoc` provides good module overview -- [ ] Check if `AGENTS.md` exists and follows canonical template +- [ ] [P2] [E:S] Check if `README.adoc` provides good module overview +- [ ] [P2] [E:S] Check if `AGENTS.md` exists and follows canonical template ### Documentation Gaps (for ARCH_TODO Stage 5.5) @@ -74,33 +74,33 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD **Feeds into:** Requirements Overview consolidation -- [ ] **Identify key functional requirements:** [List 3-5 most important] +- [ ] [P1] [E:L] **Identify key functional requirements:** [List 3-5 most important] - [ ] **Identify key non-functional requirements:** - - [ ] Performance targets: [e.g., latency, throughput] - - [ ] Security obligations: [e.g., bounds checking, input validation] - - [ ] Operability requirements: [e.g., monitoring, logging] -- [ ] **Map requirements to architecture patterns:** [How do requirements drive design?] + - [ ] [P1] [E:M] Performance targets: [e.g., latency, throughput] + - [ ] [P1] [E:M] Security obligations: [e.g., bounds checking, input validation] + - [ ] [P1] [E:M] Operability requirements: [e.g., monitoring, logging] +- [ ] [P1] [E:L] **Map requirements to architecture patterns:** [How do requirements drive design?] ## Decisions for Architecture Overview (ARCH_TODO Stage 1.85) **Feeds into:** Decision Log Overview consolidation -- [ ] **Identify key architectural decisions:** [List 2-4 major decisions] +- [ ] [P1] [E:M] **Identify key architectural decisions:** [List 2-4 major decisions] - [ ] Decision ID (if in decision-log.adoc): - [ ] Brief description: - [ ] Rationale: - [ ] Alternatives considered: - [ ] **Identify decision patterns used:** - - [ ] Off-heap memory? [Y/N - explain] - - [ ] Single writer principle? [Y/N - explain] - - [ ] Reference counting? [Y/N - explain] - - [ ] Flyweight pattern? [Y/N - explain] + - [ ] [P1] [E:S] Off-heap memory? [Y/N - explain] + - [ ] [P1] [E:S] Single writer principle? [Y/N - explain] + - [ ] [P1] [E:S] Reference counting? [Y/N - explain] + - [ ] [P1] [E:S] Flyweight pattern? [Y/N - explain] ## Glossary Terms (ARCH_TODO Stage 1.5) **Feeds into:** Cross-module glossary -- [ ] **Module-specific terms to include in glossary:** +- [ ] [P2] [E:S] **Module-specific terms to include in glossary:** - [ ] Term 1: [Definition] - [ ] Term 2: [Definition] - [ ] [Add more as needed] @@ -210,27 +210,32 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD - [ ] Security controls described? - [ ] Known limitations documented? +## Automation Tasks + +- [x] [P1] [E:S] Create `scripts/generate-progress.sh` to track TODO completion. +- [x] [P1] [E:S] Create `scripts/housekeeping.sh` to archive completed tasks. + ## Improvement Tasks (ARCH_TODO Stage 5.5) **Feeds into:** Improve Existing Module Documentation ### High Priority -- [ ] Create missing architecture-overview.adoc (if needed) -- [ ] Add missing front-matter to existing docs -- [ ] Fix broken cross-references -- [ ] Add `:sectnums:` where appropriate +- [ ] [P1] [E:M] Create missing architecture-overview.adoc (if needed) +- [x] [P1] [E:S] Add missing front-matter to existing docs +- [ ] [P1] [E:M] Fix broken cross-references +- [x] [P1] [E:S] Add `:sectnums:` where appropriate ### Medium Priority -- [ ] Expand brief architecture docs (if < 75 lines) -- [ ] Add "Trade-offs and Alternatives" section (following Chronicle-Bytes pattern) -- [ ] Add performance characteristics section -- [ ] Create decision log entries for undocumented decisions +- [ ] [P2] [E:L] Expand brief architecture docs (if < 75 lines) +- [ ] [P2] [E:L] Add "Trade-offs and Alternatives" section (following Chronicle-Bytes pattern) +- [ ] [P2] [E:M] Add performance characteristics section +- [ ] [P2] [E:M] Create decision log entries for undocumented decisions ### Low Priority -- [ ] Add diagrams (PlantUML or draw.io) -- [ ] Create example code snippets -- [ ] Expand requirements documentation -- [ ] Add cross-references to other module docs +- [ ] [P3] [E:L] Add diagrams (PlantUML or draw.io) +- [ ] [P3] [E:M] Create example code snippets +- [ ] [P3] [E:L] Expand requirements documentation +- [ ] [P3] [E:M] Add cross-references to other module docs ## Code Quality Tasks @@ -240,7 +245,7 @@ This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TOD - Latest command: `mvn checkstyle:check` from `JLBH` with Java 21 (see `verify-jlbh-java21-checkstyle-latest.log`); Checkstyle reports `You have 0 Checkstyle violations.` for this module. - [x] Run SpotBugs scan and document issues - A Java 21 SpotBugs run (`mvn com.github.spotbugs:spotbugs-maven-plugin:4.9.8.1:check -DskipTests`, see `verify-jlbh-java21-spotbugs-latest.log`) completes with `BugInstance size is 0` and `No errors/warnings found` for JLBH. -- [ ] Identify any code review follow-ups from CODE_REVIEW_STATUS.md +- [ ] [P1] [E:S] Identify any code review follow-ups from CODE_REVIEW_STATUS.md - JLBH does not currently have a dedicated section in `CODE_REVIEW_STATUS.md`; any future review actions (for example around new benchmarks or harness features) should be recorded there and linked from this TODO. ## Notes diff --git a/src/main/docs/architecture.adoc b/src/main/docs/architecture.adoc new file mode 100644 index 00000000..a6abb38f --- /dev/null +++ b/src/main/docs/architecture.adoc @@ -0,0 +1,206 @@ += Chronicle JLBH - Architecture Overview +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +== Introduction + +Chronicle JLBH (Java Latency Benchmark Harness) is a library designed for measuring and analyzing the latency of Java applications "in context," particularly under specific throughput conditions and accounting for coordinated omission. +This document outlines its high-level architecture, major components, and key operational flows. + +== Major Components + +The JLBH system is composed of several key classes and interfaces that work together to orchestrate and execute benchmarks: + +* `JLBH` +** The primary orchestrator of the benchmark lifecycle. +** Manages warm-up phases, iteration loops, and multiple runs. +** Provides the `sample(long durationNs)` method for recording end-to-end latencies. +** Offers `addProbe(String name)` to create `NanoSampler` instances for measuring sub-components or stages within the benchmarked task. +** Responsible for printing summaries to a `PrintStream`. +** Implements `NanoSampler` itself for the main end-to-end timings. + +* `JLBHOptions` +** A builder-style configuration object used to define all parameters for a `JLBH` instance. +** Settings include: +*** Number of warm-up iterations (`warmUpIterations`). +*** Number of measurement iterations per run (`iterations`). +*** Number of benchmark runs (`runs`). +*** Target throughput and its time unit (`throughput`, `throughputTimeUnit`). +*** The `JLBHTask` to be executed. +*** Coordinated Omission accounting (`accountForCoordinatedOmission`). +*** OS jitter recording parameters (`recordOSJitter`, `recordJitterGreaterThanNs`, `jitterAffinity`). +*** Latency distribution strategy (`latencyDistributor`). +*** CPU affinity for the main benchmark thread (`acquireLock`). +*** Timeout for benchmark abortion if no samples are produced (`timeout`). + +* `JLBHTask` +** An interface implemented by the user to define the actual workload to be benchmarked. +** Key lifecycle methods invoked by `JLBH`: +*** `init(JLBH jlbh)`: Called once before any iterations, used for setup and to acquire `NanoSampler` probes via `jlbh.addProbe()`. +*** `run(long startTimeNs)`: Called for each benchmark iteration (both warm-up and measurement). +The `startTimeNs` is the calculated ideal start time for the iteration. +*** `warmedUp()`: Called after the warm-up phase is complete and before the first measurement run begins. +*** `runComplete()`: Called after each measurement run finishes. +*** `complete()`: Called once after all runs have finished, for cleanup. + +* `NanoSampler` +** An interface for recording timing samples. +** Implementations (like `JLBH` itself or those returned by `addProbe`) typically wrap a `Histogram`. +** Primary method: `sampleNanos(long durationNs)`. + +* `Histogram` +** A data structure used to record the distribution of latency samples for each probe (end-to-end, custom probes, OS jitter). +** Provided by the `chronicle-core` library. +** Key characteristics: +*** Stores frequencies of samples in logarithmically scaled buckets, enabling high precision over a wide dynamic range. +*** Configured by `JLBH` (via `createHistogram()`) for typically 35 bits of range and 8 significant figures. +*** Provides methods to extract percentile data (e.g., `getPercentiles()`). + +* `JLBHResult` +** An immutable interface representing the complete results of a benchmark execution. +** Provides access to: +*** `endToEnd()`: A `ProbeResult` for the main end-to-end measurements. +*** `probe(String probeName)`: An `Optional` for any additionally registered probes. +*** `osJitter()`: An `Optional` if OS jitter was recorded. +** Concrete immutable implementations include `ImmutableJLBHResult`, `ImmutableProbeResult`, and `ImmutableRunResult`. + +* `JLBHResultConsumer` +** A functional interface (`Consumer` and `Supplier`) used to decouple the production of `JLBHResult` from its consumption. +** `JLBHResultConsumer.newThreadSafeInstance()` provides a `ThreadSafeJLBHResultConsumer` that allows results to be safely retrieved by a different thread after the benchmark completes. + +* `LatencyDistributor` +** An optional strategy interface (`long apply(long averageLatencyNS)`) to shape the throughput by adjusting the delay between task invocations. +** Allows simulation of different load patterns beyond a constant rate. +** Predefined implementations are available in `LatencyDistributors` (e.g., `NORMAL`, `RANDOM`). + +* `OSJitterMonitor` +** An internal background thread responsible for measuring operating system scheduling jitter. +** Monitoring is enabled by default (`JLBHOptions.recordOSJitter` defaults to true) and can be disabled via `recordOSJitter(false)`. +** Repeatedly calls `System.nanoTime()` to detect delays greater than `recordJitterGreaterThanNs` and records these into a dedicated `Histogram`. + +== Execution Flow + +```mermaid +graph TD + A[Initialization] --> B(Warm-up Phase) + B --> C{Measurement Runs} + C --> D(Completion) + C -- loop --> C +``` + +The harness follows a well-defined lifecycle, detailed visually in link:benchmark-lifecycle.adoc[benchmark lifecycle diagram]. +A typical execution sequence is as follows: + +* **Initialization**: +** The user configures `JLBHOptions` defining all benchmark parameters. +** A `JLBH` instance is created with these options, an optional `PrintStream` for output, and an optional `JLBHResultConsumer` for programmatic result access. +** The `JLBHTask.init(JLBH)` method is called, allowing the user's task to perform setup. +*** During `init`, custom `NanoSampler` probes can be registered via `jlbh.addProbe(String)`. +Each probe is backed by its own `Histogram`. +** If `recordOSJitter` is enabled in `JLBHOptions`, the `OSJitterMonitor` background thread is started. + +* **Warm-up Phase**: +** The `JLBHTask.run(System.nanoTime())` method is invoked `warmUpIterations` times. +*** The primary purpose is to allow the JVM's JIT compiler to optimize the code and for the system to reach a steady operational state. +*** Latency samples collected during this phase are typically added to histograms but are reset and discarded before measurement runs begin. +** After warm-up iterations, `JLBHTask.warmedUp()` is called. +** All probe histograms (end-to-end, custom, OS jitter) are reset. + +* **Measurement Runs**: +** This phase is repeated for the number of `runs` specified in `JLBHOptions`. +** For each run: +*** A loop executes for the configured number of `iterations`. +*** In each iteration: +**** The `JLBH` orchestrator calculates the `startTimeNs` for the task invocation. +***** This calculation is influenced by the configured `throughput` and the `LatencyDistributor` strategy. +***** If `accountForCoordinatedOmission` is true, `startTimeNs` represents the ideal scheduled time. +The harness may busy-wait (spin using `System.nanoTime()`) if the current time is before this ideal `startTimeNs` to ensure the task starts as close to the scheduled time as possible. +**** `JLBHTask.run(startTimeNs)` is invoked with this calculated start time. +***** The user's benchmark logic executes. +***** To record the end-to-end latency, the task calls `jlbh.sample(System.nanoTime() - startTimeNs)`. +***** To record latency for sub-stages, the task calls `sampleNanos()` on any custom `NanoSampler` probes obtained during `init`. +**** Each call to `sample()` or `sampleNanos()` on a sampler results in the duration being recorded in its corresponding `Histogram`. +*** After all iterations for the current run are complete: +**** The `JLBHTask.runComplete()` method is invoked. +**** Statistics for the completed run (percentile values from each histogram) are typically printed to the configured `PrintStream`. +**** The percentile data for this run is stored as part of the accumulating results. +**** All probe histograms are reset to prepare for the next measurement run (if any). + +* **Completion**: +** After all measurement runs have finished: +*** A final summary report, often comparing results across runs and showing variation, is printed to the `PrintStream`. +*** The complete, immutable `JLBHResult` object, containing all data for all probes across all runs, is constructed. +*** If a `JLBHResultConsumer` was provided, its `accept(JLBHResult)` method is called with the final result. +** The `JLBHTask.complete()` method is invoked for any final cleanup by the user's task. +** The `OSJitterMonitor` thread is terminated if it was running. + +== Threading Model + +JLBH has a specific threading model that users and contributors should understand: + +* **Primary Harness Thread**: +** When `JLBH.start()` is called, it orchestrates the entire benchmark lifecycle (initialization, warm-up, measurement runs, completion) on the *calling thread*. +** All `JLBHTask` lifecycle methods (`init`, `run`, `warmedUp`, `runComplete`, `complete`) are invoked sequentially on this single harness thread. +*** This implies that `JLBHTask` implementations generally do not need to be concerned about thread-safety for their internal state when accessed *only* by these JLBH-invoked lifecycle methods. +** CPU affinity for this thread can be suggested via `JLBHOptions.acquireLock`. + +* **Event Loop Integration**: +** As an alternative to `JLBH.start()`, the benchmark can be driven by an external `EventLoop` (e.g., from Chronicle Services) by calling `JLBH.eventLoopHandler(EventLoop)`. +** Coordinated omission accounting must remain enabled; calling `eventLoopHandler` with `accountForCoordinatedOmission(false)` throws an `UnsupportedOperationException`. +** In this mode, the warm-up and measurement iterations are scheduled and executed on the provided event loop's thread, using internal `EventHandler` implementations (`WarmupHandler`, `JLBHEventHandler`). + +* **User-Spawned Threads within `JLBHTask`**: +** The code being benchmarked *within* `JLBHTask.run()` (i.e., the "business logic") *can* be multi-threaded if the application itself is multi-threaded. +** However, critical interactions with the `JLBH` instance *must* be managed carefully. +*** Calling `jlbh.sample()` or methods on `NanoSampler` probes (like `probe.sampleNanos()`) should ideally be done only by the thread that `JLBH` used to call `JLBHTask.run()`. +*** The `JLBH` class is annotated `@SingleThreaded`, and its internal state (including the histograms) is generally not designed for concurrent modification from multiple user threads without external synchronization. +Chronicle Core's `Histogram` has specific single-writer assumptions unless explicitly used with synchronization. + +* **`OSJitterMonitor` Thread**: +** If OS jitter recording is enabled (`JLBHOptions.recordOSJitter == true`), a dedicated background thread (`OSJitterMonitor`) is started. +** This thread runs independently of the main harness thread. +** Its sole purpose is to repeatedly sample `System.nanoTime()` to detect scheduling delays or pauses exceeding `JLBHOptions.recordJitterGreaterThanNs` and record these durations into a dedicated `Histogram`. +** Its CPU affinity can be optionally controlled via `JLBHOptions.jitterAffinity`. + +* **Result Consumption**: +** The `JLBHResult` object containing all benchmark data is immutable by design. +** To safely access this result from a thread other than the main harness thread (e.g., an application thread wanting to query results after a test run), `JLBHResultConsumer.newThreadSafeInstance()` should be used. +This creates a `ThreadSafeJLBHResultConsumer` which uses a `volatile` field to ensure proper publication of the immutable result. + +== Data Management and Results + +JLBH pays careful attention to how latency data is recorded, structured, and reported: + +* **Latency Recording**: +** All latency durations are measured and recorded in nanoseconds. +** `System.nanoTime()` is the fundamental source for time measurements, chosen for its high resolution and (on most systems) monotonic behavior. +** For each active probe (end-to-end, user-defined additional probes, OS jitter), a `net.openhft.chronicle.core.util.Histogram` instance is maintained. +*** These histograms are configured by `JLBH.createHistogram()` for high precision (typically 8 significant figures of value) and a wide dynamic range (typically 35 bits, covering from nanoseconds to many seconds). + +* **Immutable Results**: +** A core design principle is the immutability of benchmark results. +Once a benchmark or a run is complete, the generated data is fixed. +** The `JLBHResult` interface and its constituent parts (`ProbeResult`, `RunResult`) are designed to be immutable. +** This is enforced by internal implementations like `ImmutableJLBHResult`, `ImmutableProbeResult`, and `ImmutableRunResult`. +** Immutability simplifies reasoning about results, ensures thread-safe access for consumers, and allows for reliable and repeatable reporting. + +* **Structure of Results (`JLBHResult`)**: +** The top-level `JLBHResult` object provides access to: +*** `endToEnd()`: Returns a `ProbeResult` for the overall end-to-end latency measurements. +*** `probe(String probeName)`: Returns an `Optional` for any user-defined probe, identified by its name. +*** `osJitter()`: Returns an `Optional` for the OS jitter measurements, if recorded. +** Each `JLBHResult.ProbeResult` object contains: +*** `summaryOfLastRun()`: A `RunResult` detailing the statistics of the final measurement run for that probe. +*** `eachRunSummary()`: A `List`, providing the statistics for every measurement run of that probe, in execution order. +** Each `JLBHResult.RunResult` object contains: +*** `percentiles()`: A `Map`, mapping enum keys (e.g., `PERCENTILE_50TH`, `PERCENTILE_99TH`, `WORST`) to their corresponding latency `Duration` values. +*** Convenience getter methods like `get50thPercentile()`, `get99thPercentile()`, `getWorst()`. + +* **Output and Serialization**: +** Human-readable summaries are printed to the configured `PrintStream` (default `System.out`) during and after benchmark runs. +This includes percentile tables for each probe and overall summaries. +** Programmatic access to the structured, immutable results is provided via the `JLBHResultConsumer` and the `JLBHResult` interface. +** Results can be serialized to CSV format using `net.openhft.chronicle.jlbh.util.JLBHResultSerializer`, facilitating offline analysis or import into spreadsheets and other tools. +** Integration with CI systems like TeamCity is supported via `net.openhft.chronicle.jlbh.TeamCityHelper`, which can emit build statistics. diff --git a/src/main/docs/benchmark-lifecycle.adoc b/src/main/docs/benchmark-lifecycle.adoc new file mode 100644 index 00000000..afa893b3 --- /dev/null +++ b/src/main/docs/benchmark-lifecycle.adoc @@ -0,0 +1,80 @@ += Benchmark Lifecycle +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +[mermaid] +---- +graph LR + A[Setup] --> B(Warmup); + B --> C{Execution Per Run}; + C --> D[Reporting]; + D --> E[Cleanup]; +---- + +A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are: + +== Setup + +This initial phase prepares the benchmark environment: + +* **JLBH Configuration**: The user instantiates `JLBHOptions` to define all benchmark parameters (e.g., iterations, throughput, runs, coordinated omission settings). An instance of `JLBH` is then created using these options. +* **Task Initialization**: The `JLBHTask.init(JLBH jlbh)` method of the user-provided task is invoked. +** This allows the benchmark task to perform its own one-time setup, such as initializing resources or fixtures. +** Crucially, this is where additional measurement probes can be registered by calling `jlbh.addProbe("probeName")` to get `NanoSampler` instances for specific sub-sections of the task. +* **OS Jitter Monitoring**: Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. +* **Affinity**: If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. + +== Warmup + +Before any measurements are formally recorded, the harness executes a warm-up phase: + +* **Warmup Iterations**: The `JLBHTask.run(long startTimeNs)` method is called `warmUpIterations` times (as specified in `JLBHOptions`). The `startTimeNs` passed is typically just `System.nanoTime()` for this phase. +** The primary goal is to allow the Java Virtual Machine (JVM) to perform Just-In-Time (JIT) compilation and optimize the benchmarked code paths. +** It also helps in bringing caches to a "warm" state and letting the system reach a more stable performance profile. +* **Sample Discarding**: Latency samples generated during the warmup phase are usually recorded into the histograms but are *not* part of the final reported results. The histograms are reset after this phase. +* **Task Notification**: After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. +* **Optional Pause**: If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. + +== Execution (Measurement Runs) + +This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): + +* **Per Run**: +** The following steps are repeated for each configured run. +** **Iteration Loop**: The `JLBH` orchestrator executes a loop for the number of `iterations` specified in `JLBHOptions`. +*** **Start Time Calculation**: For each iteration, JLBH calculates an ideal `startTimeNs`. This calculation considers: +**** The configured `throughput` (e.g., messages per second). +**** The `LatencyDistributor` strategy, if one is set, to potentially vary the inter-iteration delay. +**** If `accountForCoordinatedOmission` is true (the default), `startTimeNs` represents the ideal scheduled time for the operation. JLBH will then busy-wait (spin) until `System.nanoTime()` reaches this `startTimeNs`, ensuring the task is dispatched as close as possible to its intended schedule, thus accounting for delays that might otherwise be missed. +*** **Task Invocation**: The `JLBHTask.run(startTimeNs)` method is invoked with the calculated (and potentially waited-for) `startTimeNs`. +*** **Sample Recording**: +**** Within the `JLBHTask.run()` method, the user's code is executed. +**** To record the primary end-to-end latency for the iteration, the task must call `jlbh.sample(System.nanoTime() - startTimeNs)`. +**** For any custom probes registered during setup, the task can call `myProbe.sampleNanos(duration)` to record latencies for specific sub-stages. +**** Each recorded sample is added to the respective `Histogram` associated with the sampler (either the main JLBH sampler or a custom probe's sampler). +** **Run Completion**: After all iterations for the current run are finished: +*** The `JLBHTask.runComplete()` method is invoked. +*** Statistics for this specific run (e.g., percentiles derived from the histograms) are typically printed to the configured `PrintStream`. +*** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. +*** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. + +== Reporting + +After all measurement runs are completed, the aggregated results are finalized and made available: + +* **Final Summary**: A comprehensive summary, often comparing results across all runs and highlighting percentile variations, is printed to the `PrintStream`. +* **Result Object Creation**: The final, immutable `JLBHResult` object is constructed. This object encapsulates all recorded data for all probes across all runs. +* **Result Consumption**: If a `JLBHResultConsumer` was provided during JLBH setup, its `accept(JLBHResult)` method is called, passing the `JLBHResult` object. This allows for programmatic access to the detailed results, for example, for assertion in automated tests or for custom serialization. +* **External Tools**: Users can further process the `JLBHResult` using utilities like: +** `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. +** `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. + +== Cleanup + +The final phase ensures that any resources are properly released: + +* **Task Cleanup**: The `JLBHTask.complete()` method is called, allowing the user's benchmark task to perform any necessary cleanup (e.g., closing files, releasing network connections). +* **OS Jitter Monitor**: If the `OSJitterMonitor` thread was started, it is signaled to terminate. +* **Affinity Release**: If a CPU affinity lock was acquired by JLBH at the start, it is released. diff --git a/src/main/docs/decision-log.adoc b/src/main/docs/decision-log.adoc new file mode 100644 index 00000000..ac51482b --- /dev/null +++ b/src/main/docs/decision-log.adoc @@ -0,0 +1,214 @@ += Chronicle JLBH - Decision Log +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +This document records key architectural and project choices made during the development of Chronicle JLBH. +Identifiers use the `JL` project scope with Nine-Box tags (FN, NF-P, NF-O, DOC, OPS, RISK, TEST, UX) and link back to the requirements in `project-requirements.adoc`. + +== Decision Index + +* link:#JL-DOC-001[JL-DOC-001 Adopt AsciiDoc for project documentation] +* link:#JL-FN-002[JL-FN-002 Single-threaded harness with JLBHTask lifecycle] +* link:#JL-NF-P-003[JL-NF-P-003 Default coordinated-omission compensation and histograms] +* link:#JL-OPS-004[JL-OPS-004 OS jitter tracking as an optional probe] +* link:#JL-FN-005[JL-FN-005 Additional probes and NanoSampler API] +* link:#JL-OPS-006[JL-OPS-006 CSV serialisation and result retention] +* link:#JL-OPS-007[JL-OPS-007 CI integration via TeamCity statistics] + +[[JL-DOC-001]] +=== [JL-DOC-001] Adopt AsciiDoc for project documentation + +Date:: 2025-05-24 +Context:: +* Project requires easily maintainable, version-controlled documentation. +* Contributors need a text format that is readable in raw form and convertible to HTML. +Decision Statement:: +* Adopt AsciiDoc as the canonical format for all project documentation. +Alternatives Considered:: +* Markdown: +** *Description:* Common lightweight markup; widely supported. +** *Pros:* Familiar to many developers; simple syntax. +** *Cons:* Limited features for complex documents; inconsistent rendering across tools. +* Plain text files: +** *Description:* Minimalistic approach with no markup. +** *Pros:* Simplest possible format; no tooling required. +** *Cons:* Hard to express structure or cross references; not suitable for large docs. +Rationale for Decision:: +* AsciiDoc provides structured markup without sacrificing readability. +* The format integrates well with our build tooling and supports the Nine-Box tagging used in requirements. +Impact & Consequences:: +* Contributors must learn basic AsciiDoc syntax. +* Build pipeline processes `.adoc` files to publish HTML. +* Some external tools may expect Markdown, requiring conversion. +Notes/Links:: +** https://asciidoctor.org[AsciiDoc project page] + +[[JL-FN-002]] +=== [JL-FN-002] Single-threaded harness with JLBHTask lifecycle + +Date:: 2025-11-14 +Context:: +* JLBH targets low-latency Java applications where benchmark overhead and scheduling noise must be minimised. +* The harness needs a clear lifecycle so that warm-up, steady-state runs and shutdown can be expressed consistently across benchmarks. +* Requirements: FN-001 Latency Sampling, FN-006 Event Loop Integration, NF-P-001 Performance, NF-O-001 Portability. +Decision Statement:: +* Run the JLBH harness logic on a single dedicated thread, driving user code through the `JLBHTask` lifecycle (`init`, `run`, `warmedUp`, `runComplete`, `complete`). +* Allow the harness thread to be supplied by the caller via `eventLoopHandler(EventLoop)` while preserving the single-threaded execution model. +Alternatives Considered:: +* Multi-threaded harness with worker pools :: +** *Pros:* Could saturate multi-core CPUs more easily in some scenarios. +** *Cons:* Adds scheduling variability, complicates reasoning about pauses and makes per-iteration latency harder to interpret. +* Delegating to a generic micro-benchmark framework (e.g. JMH) :: +** *Pros:* Reuses existing tooling and reporting. +** *Cons:* Less control over in-context execution, fewer hooks for probes and OS jitter, and different lifecycle assumptions than JLBH requires. +Rationale for Decision:: +* A single harness thread keeps timing behaviour predictable and avoids extra scheduling noise beyond what the benchmarked code introduces. +* The explicit lifecycle methods match the phases described in the requirements and README, making it easy to structure complex benchmarks. +Impact & Consequences:: +* Benchmarked code must respect the single-threaded driver model, spawning its own threads only when needed and understanding the impact on measurements. +* Event-loop integration is explicit: users can install JLBH onto an existing loop without changing the underlying lifecycle. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:benchmark-lifecycle.adoc[Benchmark lifecycle overview] + +[[JL-NF-P-003]] +=== [JL-NF-P-003] Default coordinated-omission compensation and high-resolution histograms + +Date:: 2025-11-14 +Context:: +* Co-ordinated omission can hide worst-case latency by skipping samples during pauses or back-pressure. +* JLBH must support high-resolution percentile analysis at extreme tails to satisfy low-latency users. +* Requirements: FN-001 Latency Sampling, FN-002 Coordinated-Omission Compensation, NF-P-001 Performance. +Decision Statement:: +* Enable compensation for co-ordinated omission by default via `accountForCoordinatedOmission(true)`, with an option to disable it when raw uncorrected timings are required. +* Use high-resolution histograms as the internal representation for recorded latencies so that percentiles can be reported accurately across a wide range of values. +Alternatives Considered:: +* No coordinated-omission compensation :: +** *Pros:* Simpler mental model; raw timings only. +** *Cons:* Under-reports tail latency in back-pressured systems, undermining the harness goals. +* External correction tooling applied after the run :: +** *Pros:* Keeps the core harness simpler. +** *Cons:* Splits logic across tools, increases risk of misconfiguration and makes it harder to reason about what the reported percentiles mean. +Rationale for Decision:: +* Default-on correction matches the expectations set in the JLBH documentation and literature on latency measurement accuracy. +* High-resolution histograms are a proven structure for representing latency distributions with minimal overhead. +Impact & Consequences:: +* Users must be aware that reported percentiles are corrected unless they explicitly disable the feature. +* Benchmark authors can rely on accurate tail reporting without bolting on extra tooling. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-004]] +=== [JL-OPS-004] OS jitter tracking as an optional probe + +Date:: 2025-11-14 +Context:: +* Kernel scheduling, interrupts and other background activity can introduce jitter that affects latency measurements. +* Users often need to distinguish between pauses caused by their code and pauses caused by the operating system or environment. +* Requirements: FN-004 OS Jitter Tracking, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide an optional OS jitter probe implemented as a background thread that records scheduler delays beyond a configured threshold, and summarise the results alongside core latency histograms. +* Enable OS jitter tracking by default, with configuration options such as `recordOSJitter(false)` to disable it when the overhead is not acceptable. +Alternatives Considered:: +* Rely on external OS-level tools or profilers :: +** *Pros:* No extra threads or complexity inside the harness. +** *Cons:* Harder to correlate jitter with benchmark iterations; adds extra setup steps for users. +* Always disabling jitter tracking in the core library :: +** *Pros:* Zero overhead from jitter measurement. +** *Cons:* Removes a key diagnostic feature that helps interpret unexpected latency spikes. +Rationale for Decision:: +* Integrating jitter measurement directly into the harness keeps benchmark setup simple and ensures results remain correlated with the workload. +* Making the feature optional and configurable manages overhead while preserving a sensible default for investigation. +Impact & Consequences:: +* When enabled, benchmarks incur some additional overhead from the jitter thread; users can trade this off against diagnostic value. +* CI and local runs can use the jitter probe to explain outliers and refine environment configuration (CPU affinity, power settings). +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-FN-005]] +=== [JL-FN-005] Additional probes and NanoSampler API + +Date:: 2025-11-14 +Context:: +* Benchmarks often need to record latencies for sub-stages (for example serialisation, network calls) without distorting the end-to-end latency distribution. +* JLBH must make it easy to capture these timings while keeping the main histogram focused on overall request latency. +* Requirements: FN-003 Additional Probes, NF-P-001 Performance, NF-UX-001 Usability. +Decision Statement:: +* Provide `addProbe(String name)` on `JLBH` that returns a `NanoSampler`, each backed by its own histogram and reported alongside the primary distribution. +* Keep the end-to-end `sample` / `sampleNanos` API unchanged so that additional probes never alter the core latency measurements. +Alternatives Considered:: +* Require users to maintain their own histograms outside the harness :: +** *Pros:* No extra API surface in JLBH. +** *Cons:* More boilerplate; higher risk of inconsistent reporting or incompatible histogram settings. +* Fold sub-stage timings into the primary histogram :: +** *Pros:* Single distribution to manage. +** *Cons:* Hides distinctions between phases and makes diagnosis of slow sub-components difficult. +Rationale for Decision:: +* A dedicated probe API gives clear separation between overall latencies and sub-stage metrics, supporting richer analysis without complicating the main flow. +* `NanoSampler` provides a lightweight abstraction that integrates cleanly with the existing sampling model. +Impact & Consequences:: +* Each additional probe incurs extra memory and processing cost; users should add probes selectively for meaningful stages. +* Documentation and examples must show how to name probes consistently so that dashboards and CI jobs can rely on stable identifiers. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-OPS-006]] +=== [JL-OPS-006] CSV serialisation and result retention + +Date:: 2025-11-14 +Context:: +* Users need to archive benchmark results for later analysis, trend tracking and comparison across builds or environments. +* Plain console tables are useful for humans but awkward for automated tooling and external analytics systems. +* Requirements: FN-005 CSV Serialisation, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Expose an immutable `JLBHResult` model for completed runs and provide `JLBHResultSerializer` to write results to CSV with a stable schema. +* Default to a simple file layout suitable for CI artefacts, while allowing callers to direct output to custom streams or paths. +Alternatives Considered:: +* Rely solely on console output :: +** *Pros:* No extra types or wiring. +** *Cons:* Difficult to parse reliably; fragile across formatting changes. +* Use a binary format (for example compressed histograms) only :: +** *Pros:* More compact; potentially faster to write. +** *Cons:* Harder to inspect manually; requires custom tooling for every consumer. +Rationale for Decision:: +* CSV strikes a balance between human readability and machine processing, working well with spreadsheets, scripting languages and CI systems. +* A dedicated serializer centralises schema decisions so that downstream tools can depend on consistent column names and ordering. +Impact & Consequences:: +* Schema changes to CSV output must be versioned and documented carefully to avoid breaking consumers. +* Large or long-running benchmarks may produce sizeable CSV files; operations teams should manage retention and storage policies. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-007]] +=== [JL-OPS-007] CI integration via TeamCity statistics + +Date:: 2025-11-14 +Context:: +* Many users run JLBH benchmarks in CI and need to gate builds on latency regressions or track trends over time. +* TeamCity provides a native mechanism (`##teamcity` service messages) for publishing numerical statistics from builds. +* Requirements: Product Functions (CI Metrics), NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide helper code to emit key JLBH metrics (for example selected percentiles and throughput) as TeamCity statistics lines, using predictable metric names derived from benchmark configuration. +* Treat CI integration as an optional layer that can be enabled or disabled without affecting the core harness behaviour. +Alternatives Considered:: +* Expect users to write their own CI integration scripts around console output or CSV files :: +** *Pros:* Keeps the core library smaller; no explicit dependency on CI conventions. +** *Cons:* Duplicated effort across projects; higher risk of inconsistent parsing and metrics. +* Integrate tightly with a single CI provider only :: +** *Pros:* Deep integration for that platform. +** *Cons:* Locks users into a specific CI and complicates usage elsewhere. +Rationale for Decision:: +* A lightweight TeamCity helper covers a common use case without constraining users who rely on other CI platforms. +* Publishing statistics directly simplifies setting latency thresholds and visualising trends in existing dashboards. +Impact & Consequences:: +* CI pipelines can fail fast when configured thresholds are exceeded, turning JLBH results into actionable quality gates. +* Additional documentation must describe metric naming conventions so that teams can map them to their CI configuration. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] diff --git a/src/main/docs/jlbh-cookbook.adoc b/src/main/docs/jlbh-cookbook.adoc new file mode 100644 index 00000000..c0d0f30d --- /dev/null +++ b/src/main/docs/jlbh-cookbook.adoc @@ -0,0 +1,551 @@ += Chronicle JLBH Cookbook +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. +These recipes aim to provide practical starting points for your own benchmarks. +:sectnums: + +== Timing a Simple Method Call + +This is the most basic use case: measuring the latency of a self-contained Java method. + +*Goal*: Measure the execution time of `myMethod()`. + +*Implement the task (`CallBenchmark.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHTask; + +public class CallBenchmark implements JLBHTask { + private JLBH jlbh; + private int counter = 0; // Example state + + @Override + public void init(JLBH jlbh) { + this.jlbh = jlbh; + // Initialize any resources needed by myMethod() here + } + + @Override + public void run(long startTimeNs) { + myMethod(counter++); // Pass some varying input if appropriate + jlbh.sample(System.nanoTime() - startTimeNs); + } + + private void myMethod(int input) { + // Work to be timed. + // This should ideally be significant enough to dominate the + // overhead of System.nanoTime() and jlbh.sample(). + // For very short methods, consider batching or alternative microbenchmark tools. + String.valueOf(input).hashCode(); // Example of a small piece of work + } + + @Override + public void complete() { + // Cleanup resources if any were initialized + } +} +---- + +*Set up and run the harness (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; + +public class MainApp { + public static void main(String[] args) { + JLBHOptions options = new JLBHOptions() + .warmUpIterations(20_000) // Allow for JIT compilation and steady state + .iterations(200_000) // Number of measurements per run + .throughput(100_000) // Target operations per second + .jlbhTask(new CallBenchmark()); + new JLBH(options).start(); + } +} +---- + +*Key Considerations*: +* Ensure the work within `myMethod()` is the primary focus. +Avoid including unrelated logic within the timed section. +* The `throughput` setting influences how often `run()` is called. +If `myMethod()` is very fast, high throughput might be achievable. +If it's slower, adjust throughput accordingly. + +== Measuring a Network Round Trip (UDP Loopback) + +This recipe demonstrates timing a UDP message send and receive on loopback. + +*Goal*: Measure the latency of sending a UDP packet and receiving a response. + +*Prerequisite*: A UDP echo server must be running on `localhost:8080`. +A simple echo server (`SimpleUDPEchoServer.java`): + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import java.net.DatagramPacket; +import java.net.DatagramSocket; +import java.io.IOException; + +public class SimpleUDPEchoServer { + public static void main(String[] args) throws IOException { + System.out.println("UDP Echo Server starting on port 8080..."); + try (DatagramSocket socket = new DatagramSocket(8080)) { + byte[] buffer = new byte[1024]; + DatagramPacket packet = new DatagramPacket(buffer, buffer.length); + while (true) { + socket.receive(packet); + // Echo back to sender + socket.send(new DatagramPacket(packet.getData(), packet.getLength(), packet.getAddress(), packet.getPort())); + } + } + } +} +---- + +TIP: Run this server in a separate process before starting the JLBH client. + +*Implement the task (`LoopbackBenchmark.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHTask; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.InetSocketAddress; +import java.nio.ByteBuffer; +import java.nio.channels.DatagramChannel; + +public class LoopbackBenchmark implements JLBHTask { + private JLBH jlbh; + private DatagramChannel channel; + // Using a direct ByteBuffer can offer performance benefits for I/O + private final ByteBuffer outBuffer = ByteBuffer.allocateDirect(32); + private final ByteBuffer inBuffer = ByteBuffer.allocateDirect(32); + + @Override + public void init(JLBH jlbh) { + this.jlbh = jlbh; + try { + channel = DatagramChannel.open(); + channel.configureBlocking(true); // For simplicity; non-blocking could also be used + channel.connect(new InetSocketAddress("localhost", 8080)); + } catch (IOException e) { + throw new UncheckedIOException(e); // Consider more robust error handling + } + } + + @Override + public void run(long startTimeNs) { + try { + outBuffer.clear(); + outBuffer.putInt(0, (int) startTimeNs); // Send some data (optional) + outBuffer.limit(4); + channel.write(outBuffer); + + inBuffer.clear(); + channel.read(inBuffer); // Wait for echo + } catch (IOException e) { + // In a real benchmark, decide how to handle I/O errors. + // Re-throwing might halt the benchmark or skew results if not all iterations complete. + throw new UncheckedIOException(e); + } + jlbh.sample(System.nanoTime() - startTimeNs); + } + + @Override + public void complete() { + try { + if (channel != null) { + channel.close(); + } + } catch (IOException e) { + e.printStackTrace(); + } + } +} +---- + +*Set up the harness (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.core.Jvm; +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; + +import java.io.IOException; + +public class MainApp { + public static void main(String[] args) { + // First, ensure SimpleUDPEchoServer is running. + // new Thread(() -> { try { SimpleUDPEchoServer.main(null); } catch (IOException e) { e.printStackTrace(); }}).start(); + // Jvm.pause(100); // Give server a moment to start if launched programmatically + + JLBHOptions options = new JLBHOptions() + .warmUpIterations(10_000) + .iterations(50_000) + // Lower throughput for network tests helps avoid queuing in network buffers, + // giving a clearer picture of the actual round-trip time. + .throughput(1_000) // Reduced from 10_000 for more stable network results + // Coordinated Omission is crucial for network tests to capture true send-to-receive latency + .accountForCoordinatedOmission(true) + .jlbhTask(new LoopbackBenchmark()); + new JLBH(options).start(); + } +} +---- + +*Key Considerations*: + +* For true network latency, run client and server on different machines connected via a relevant network. +* Consider TCP for connection-oriented tests; its performance characteristics (e.g., Nagle's algorithm, ACK delays) differ from UDP. +* Packet loss and retransmissions (for UDP, if implementing reliability) can heavily affect latency distributions. + +== Running Within an Existing Event Loop + +JLBH can integrate with Chronicle's event loop mechanism, useful for benchmarking components that are already part of an event-driven architecture. + +*Goal*: Execute a JLBH benchmark task on a pre-existing `EventLoop`. + +*Implement the task (`EventLoopTask.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHTask; + +public class EventLoopTask implements JLBHTask { + private JLBH jlbh; + + @Override + public void init(JLBH jlbh) { + this.jlbh = jlbh; + System.out.println("EventLoopTask initialized on thread: " + Thread.currentThread().getName()); + } + + @Override + public void run(long startTimeNs) { + // Perform work specific to the event loop context + jlbh.sample(System.nanoTime() - startTimeNs); + } + + @Override + public void warmedUp() { + System.out.println("EventLoopTask warmed up on thread: " + Thread.currentThread().getName()); + } +} +---- + +*Set up the harness (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.core.threads.EventLoop; +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; +import net.openhft.chronicle.jlbh.JLBHTask; +import net.openhft.chronicle.threads.MediumEventLoop; +import net.openhft.chronicle.threads.Pauser; + +public class MainApp { + public static void main(String[] args) { + // MediumEventLoop and Pauser are part of Chronicle Core/Threads + // Pauser.balanced() tries to balance latency and CPU usage. + EventLoop eventLoop = new MediumEventLoop(null, "myBenchmarkLoop", Pauser.balanced(), true, null); + eventLoop.start(); // Start the event loop thread + + JLBHTask task = new EventLoopTask(); + JLBHOptions options = new JLBHOptions() + .warmUpIterations(10_000) // Warmup happens on the event loop too + .iterations(100_000) + .throughput(50_000) + // accountForCoordinatedOmission defaults to true, which is compatible and required + // for eventLoopHandler() to function as expected. + .jlbhTask(task); + + JLBH jlbh = new JLBH(options); + // Instead of jlbh.start(), install handlers onto the event loop + jlbh.eventLoopHandler(eventLoop); + + // Keep the main thread alive or wait for completion if necessary. + // For this example, we'll let it run for a bit then stop the loop. + // In a real app, the event loop would run for the application's lifetime. + try { + Thread.sleep(15_000); // Let benchmark run + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + eventLoop.close(); // Cleanly shut down the event loop + } + } +} +---- + +*Key Considerations*: + +* The `eventLoopHandler` method installs handlers that drive the benchmark lifecycle on the `EventLoop`'s thread. +* This is useful when the code being benchmarked is designed to run on a specific event loop. +* `JLBHOptions.accountForCoordinatedOmission(true)` (the default) is essential for `eventLoopHandler` to function correctly. +--- + +== Using Multiple Probes for Staged Operations + +JLBH allows timing multiple stages within a single benchmark iteration using custom probes. + +*Goal*: Measure total operation latency and the latency of two distinct internal stages. + +*Implement the task (`MultiStageBenchmark.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.core.util.NanoSampler; +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHTask; + +import java.util.concurrent.TimeUnit; + +public class MultiStageBenchmark implements JLBHTask { + private JLBH jlbh; + private NanoSampler stage1Sampler; + private NanoSampler stage2Sampler; + + @Override + public void init(JLBH jlbh) { + this.jlbh = jlbh; + // Register custom probes for each stage + this.stage1Sampler = jlbh.addProbe("Stage1_Processing"); + this.stage2Sampler = jlbh.addProbe("Stage2_Persistence"); + } + + @Override + public void run(long startTimeNs) { + // Stage 1 + long stage1StartNs = System.nanoTime(); + performStage1Work(); + stage1Sampler.sampleNanos(System.nanoTime() - stage1StartNs); + + // Stage 2 + long stage2StartNs = System.nanoTime(); + performStage2Work(); + stage2Sampler.sampleNanos(System.nanoTime() - stage2StartNs); + + // Sample total end-to-end time + jlbh.sample(System.nanoTime() - startTimeNs); + } + + private void performStage1Work() { + // Simulate some processing work, e.g., data transformation + try { + TimeUnit.NANOSECONDS.sleep(100 + (System.nanoTime() % 50)); // Variable work + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + private void performStage2Work() { + // Simulate some I/O or persistence work + try { + TimeUnit.NANOSECONDS.sleep(150 + (System.nanoTime() % 70)); // Variable work + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } +} +---- + +*Set up the harness (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; + +public class MainApp { + public static void main(String[] args) { + JLBHOptions options = new JLBHOptions() + .warmUpIterations(15_000) + .iterations(100_000) + .throughput(20_000) // Adjust based on expected stage latencies + .jlbhTask(new MultiStageBenchmark()); + new JLBH(options).start(); + } +} +---- + +*Key Considerations*: + +* The sum of individual probe latencies might not exactly equal the end-to-end latency due to the overhead of `System.nanoTime()` calls between stages. +* This approach is excellent for identifying bottlenecks within a larger operation. +* The JLBH output will include separate percentile summaries for "Stage1_Processing", "Stage2_Persistence", and the default "end to end" probe. + +''' + +== Recording and Exporting Results + +JLBH allows programmatic access to results and provides utilities for serialization, such as to CSV. + +*Goal*: Run a benchmark, retrieve its results, and save them to a CSV file. + +*Implement the task*: We can reuse the `CallBenchmark` from Recipe 1. + +[source,java] +---- +// Re-use CallBenchmark.java from Recipe 1 +package net.openhft.chronicle.jlbh.cookbook; +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHTask; +public class CallBenchmark implements JLBHTask { ... } +---- + +*Set up the harness and process results (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; +import net.openhft.chronicle.jlbh.JLBHResult; +import net.openhft.chronicle.jlbh.JLBHResultConsumer; +import net.openhft.chronicle.jlbh.util.JLBHResultSerializer; + +import java.io.IOException; +import java.time.Duration; + +public class MainApp { + public static void main(String[] args) { + JLBHResultConsumer resultConsumer = JLBHResultConsumer.newThreadSafeInstance(); + + JLBHOptions options = new JLBHOptions() + .warmUpIterations(10_000) + .iterations(50_000) + .throughput(100_000) + .jlbhTask(new CallBenchmark()) // Using CallBenchmark from Recipe 1 + .runs(2); // Multiple runs to see more in the result + + // Pass the consumer to the JLBH constructor + new JLBH(options, System.out, resultConsumer).start(); + + // After JLBH completes, retrieve the results + JLBHResult result = resultConsumer.get(); + + if (result != null) { + // Programmatically access results (example: last run's 99th percentile) + JLBHResult.RunResult lastRunEndToEnd = result.endToEnd().summaryOfLastRun(); + Duration p99 = lastRunEndToEnd.get99thPercentile(); + System.out.printf("Last run's End-to-End 99th Percentile: %d ns%n", p99.toNanos()); + + // Serialize all probe results to CSV + try { + String csvFileName = "benchmark_results.csv"; + JLBHResultSerializer.runResultToCSV(result, csvFileName, result.probeNames(), true); + System.out.println("Benchmark results saved to " + csvFileName); + // The CSV file will contain: + // ProbeName,50th p-le,90th p-le,99th p-le,999th p-le,9999th p-le,Worst + // endToEnd,... + // (any other custom probes)... + // OSJitter,... (if enabled) + } catch (IOException e) { + System.err.println("Error saving results to CSV: " + e.getMessage()); + e.printStackTrace(); + } + } else { + System.err.println("Benchmark did not produce results."); + } + } +} +---- + +*Key Considerations*: + +* `JLBHResultConsumer.newThreadSafeInstance()` is important if you plan to access results from a different thread than the one running `JLBH.start()`, though in this synchronous example it primarily serves as the hook. +* The `JLBHResult` object is immutable and contains rich data for all probes and runs. +* `JLBHResultSerializer` provides a convenient way to get data out for external analysis. + +== Enabling and Understanding OS Jitter Measurement + +Operating system jitter can significantly impact low-latency applications. +JLBH can measure this independently. + +*Goal*: Configure JLBH to record OS jitter and understand its output. + +*Implement the task*: Any simple task will do, as OS jitter is measured by a background thread. +We'll reuse `CallBenchmark`. + +[source,java] +---- +// Re-use CallBenchmark.java from Recipe 1 +---- + +*Set up the harness with OS Jitter options (`MainApp.java`)*: + +[source,java] +---- +package net.openhft.chronicle.jlbh.cookbook; + +import net.openhft.chronicle.jlbh.JLBH; +import net.openhft.chronicle.jlbh.JLBHOptions; + +public class MainApp { + public static void main(String[] args) { + JLBHOptions options = new JLBHOptions() + .warmUpIterations(5_000) // Shorter for quick demo + .iterations(10_000) // Shorter for quick demo + .throughput(50_000) + .jlbhTask(new CallBenchmark()) + // OS jitter recording is enabled by default; use recordOSJitter(false) + // if you want to disable the background monitor. + // Optional: Only record jitter events greater than a specific threshold (default 1_000 ns) + // .recordJitterGreaterThanNs(5_000) + // Optional: Pin the OS Jitter monitoring thread to a specific CPU core (if supported) + // .jitterAffinity(true) // Be cautious with affinity, ensure free isolated cores + .runs(1); + + new JLBH(options).start(); + } +} +---- + +*Interpreting OS Jitter Results*: + +* JLBH records OS jitter by default and starts a background thread that continuously measures the time elapsed between calls to `System.nanoTime()`. +Use `recordOSJitter(false)` only when you want to suppress this monitoring. +* If this measured gap exceeds `recordJitterGreaterThanNs`, it's considered an OS-induced pause/jitter event and is recorded in a special "OS Jitter" probe. +* The JLBH output (both console and `JLBHResult`) will include a section for the "OS Jitter" probe, showing its own percentile distribution. +* These jitter values represent pauses that are *not* caused by your application code but by the operating system or other processes interfering with the JLBH process's execution. +* High OS jitter values (e.g., milliseconds) can explain unexpected latency spikes in your main application probes, even if your application code itself is highly optimized. + +*Key Considerations*: + +* OS jitter measurement adds some (usually small) overhead. +Enable it when you suspect external factors are influencing your benchmark. +* The `recordJitterGreaterThanNs` option helps filter out minor, expected system noise. +* `jitterAffinity(true)` can provide more stable jitter measurements if you have isolated CPU cores available, but can be detrimental otherwise. +* Correlating high OS jitter percentiles with high application latency percentiles can be a powerful diagnostic tool. diff --git a/src/main/docs/project-requirements.adoc b/src/main/docs/project-requirements.adoc new file mode 100644 index 00000000..3dbe1a0f --- /dev/null +++ b/src/main/docs/project-requirements.adoc @@ -0,0 +1,203 @@ += Chronicle JLBH - Software Requirements Specification +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +*_Abstract_*:: +Chronicle JLBH is an open-source Java library, released under the Apache Licence 2.0, that provides a high-resolution latency benchmark harness. +It allows developers to _measure_, _analyse_ and _regress-test_ latency behaviour of critical code paths "in context" rather than via isolated micro-benchmarks. +Key features include compensation for *co-ordinated omission*, configurable throughput modelling, additional probes, optional operating-system jitter tracking, and serialisation of results for CI dashboards. + +== Introduction + +=== Purpose +This document specifies functional and non-functional requirements for Chronicle JLBH, ensuring that contributors and integrators share a common understanding of the product's goals and constraints. + +=== Scope +The harness targets _low-latency_ Java applications (trading systems, micro-services, real-time analytics) that must quantify latencies at extreme percentiles under realistic load patterns. + +=== Definitions, Acronyms and Abbreviations + +|=== +| *Term* | *Definition* + +| JLBH | Java Latency Benchmark Harness - the library under specification. +| Co-ordinated Omission | Measurement error whereby pauses/back-pressure hide worst-case latencies. +| Probe | A named histogram that records an additional timing within the benchmark workflow. +| OS Jitter | Kernel or scheduler-induced timing variance measured by a background thread. +| CI | Continuous Integration environment (e.g. GitHub Actions, TeamCity). +|=== + +=== References + +* Upstream repository: https://github.com/OpenHFT/JLBH[GitHub - OpenHFT/JLBH] +* API reference: https://javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/core/jlbh/JLBH.html[Javadoc] +* Tutorial series: http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[RationalJava blog series] +* Discussion of _co-ordinated omission_: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE[m.s. thread] +* Chronicle blog post on JLBH in event loops: https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html[Vanilla-Java article] + +=== Overview +Sections 2-5 describe the product context, interfaces, detailed system features, and quality attributes. +Section 6 captures licensing, whilst Section 7 presents the glossary. + +== Overall Description + +=== Product Perspective +JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. + +=== Product Functions + +* Generate high-resolution histograms (>=35 bits) of end-to-end sample latencies. +* Optionally compensate for *co-ordinated omission* by adjusting the synthetic start time. +* Model load via configurable `throughput(int, TimeUnit)` and *LatencyDistributor*s. +* Provide additional named probes via `addProbe(String)`. +* Record OS-level jitter and present it as a separate probe. +* Output run summaries to `PrintStream` and expose structured results via `JLBHResult`. +* Serialise the final run to CSV (`JLBHResultSerializer`). +* Integrate with CI (TeamCity helper emits `##teamcity` statistics lines). +* Allow execution from any thread or installation onto a Chronicle *MediumEventLoop*. + +=== User Classes and Characteristics + +|=== +| *Actor* | *Description* | *Technical Expertise* + +| Performance Engineer | Designs latency benchmarks and analyses histograms. | Advanced Java & performance tuning. +| Developer | Adds JLBH tests to codebase, evaluates pull-request regressions. | Intermediate Java. +| CI System | Runs automated latency regression suite, ingests CSV or TeamCity stats. | N/A (automation). +|=== + +=== Operating Environment + +* Java 11 LTS and later; Java 17 recommended for current builds. +* Linux x86-64 (primary), other POSIX OSes supported; macOS usable but CI baselines exclude it. +* Typical CPU affinity facilities available via *OpenHFT Affinity* library. + +=== Design and Implementation Constraints + +* Single-threaded harness execution; benchmarked code may span threads. +* Measurements rely on `System.nanoTime()` precision; hardware timers must be invariant. +* JVM must be started with `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when using async-profiler concurrently. +* Library is released under the Apache Licence 2.0; contributions must comply with CLA policy. + +=== Assumptions and Dependencies + +* SUT jars are available on the same JVM classpath. +* Benchmark host clocks are monotonic and not subjected to aggressive power-saving states. +* CI agents provision exclusive CPU cores to minimise jitter. + +== External Interface Requirements + +=== Public API + +* `JLBHOptions` - builder for benchmark parameters (throughput, warm-up, runs, etc.). +* `JLBHTask` - user-defined callback with lifecycle methods `init`, `run`, `warmedUp`, `runComplete`, `complete`. +* `JLBH` - orchestrator class with `start()` and `eventLoopHandler(EventLoop)` entry points. +* `JLBHResult` - immutable projection of measured data. +* `JLBHResultSerializer` - CSV writer for analytics pipelines. + +=== User Interface +CLI usage is demonstrated in _ExampleJLBHMain_ and test fixtures; no dedicated GUI is provided. + +=== Hardware Interfaces +None. The harness interacts with OS timers and CPU affinity via JNI where available. + +=== Software Interfaces + +* Build tool: *Apache Maven* (pom.xml defines BOM imports). +* Logging: SLF4J (test scope). +* Testing: JUnit 4 for unit and integration tests. +* CI Metrics: TeamCity service messages. + +== System Features + +=== FN-001 Latency Sampling +*Description*: Collect nanosecond deltas from user code via `sample(long)` or `sampleNanos(long)`. +*Stimulus/Response*: Upon each iteration, `JLBHTask.run` calls `jlbh.sample(...)`; histogram updates occur; run summary printed at completion. +*Priority*: *Essential*. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. + +=== FN-002 Coordinated-Omission Compensation +Ensures that queue back-log does not under-represent tail latency. Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. +*Decision reference*: link:decision-log.adoc#JL-NF-P-003[JL-NF-P-003]. + +=== FN-003 Additional Probes +Developers may time sub-stages (e.g., serialisation, network round-trip) through `addProbe(String)` and record them independently. +*Decision reference*: link:decision-log.adoc#JL-FN-005[JL-FN-005]. + +=== FN-004 OS Jitter Tracking +A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. Results are summarised alongside core latencies. +*Decision reference*: link:decision-log.adoc#JL-OPS-004[JL-OPS-004]. + +=== FN-005 CSV Serialisation +Post-run, results can be persisted for offline analysis (`result.csv` by default). +*Decision reference*: link:decision-log.adoc#JL-OPS-006[JL-OPS-006]. + +=== FN-006 Event Loop Integration +`eventLoopHandler(EventLoop)` allows benchmarks to operate inside Chronicle threading framework, avoiding extra threads. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. + +== Non-Functional Requirements + +=== NF-P-001 Performance +* Overhead per sample must remain below 100 ns when no additional probes are active. +* Histogram generation must support >=200 M iterations without heap pressure. + +=== NF-R-001 Reliability +* Harness must abort gracefully on thread interruptions or sample time-outs (`timeout(long)`). +* Immutable result objects ensure thread-safe publication to external consumers. + +=== NF-UX-001 Usability +* Fluent builder API; sensible defaults provide a runnable benchmark in ≤10 LOC. +* ASCII table outputs are human-readable and CI-friendly. + +=== NF-O-001 Portability +* Pure-Java codebase; no native compilation steps. +* JDK-specific optimisations (e.g., *Zing* support) are runtime-detected. + +=== NF-O-002 Maintainability +* 80 %+ unit-test line coverage with deterministic fixtures. +* Code adheres to Chronicle parent POM style and SonarCloud quality gates. + +=== NF-S-001 Security +No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code. + +== Licensing + +The project is released under the *Apache Licence 2.0* (_see_ `LICENSE.adoc`). +Downstream consumers must preserve licence notices and may include JLBH in commercial or OSS products, subject to the terms therein. + +== Glossary + +Co-ordinated Omission:: Statistical artefact causing under-reporting of worst-case latency. +Histogram:: Data structure that records frequency counts in logarithmic buckets, enabling percentile extraction. +Percentile:: Value below which a given percentage of observations fall (e.g., 99th percentile). + +== Appendix A - Example Minimal Benchmark + +[source,java] +---- +public class NothingBenchmark implements JLBHTask { + private JLBH jlbh; + public void init(JLBH jlbh) { this.jlbh = jlbh; } + public void run(long startTimeNS) { + jlbh.sample(System.nanoTime() - startTimeNS); + } + public static void main(String[] args) { + new JLBH(new JLBHOptions() + .throughput(1_000_000) + .iterations(10_000) + .jlbhTask(new NothingBenchmark())) + .start(); + } +} +---- + +== Appendix B - Footnotes + +* JLBH originated within the Chronicle Software open-source stack and is actively maintained. See https://github.com/OpenHFT/JLBH. +* The API reference highlights the focus on _co-ordinated omission_ and event-loop support. See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. +* A multi-part blog series gives practical guidance on designing realistic latency tests. See http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html. +* Vanilla-Java's micro-services article demonstrates embedding JLBH in an event-driven architecture. See https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html. +* Original discussion of co-ordinated omission by Gil Tene motivates JLBH's default settings. See https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE. diff --git a/src/main/docs/results-interpretation-guide.adoc b/src/main/docs/results-interpretation-guide.adoc new file mode 100644 index 00000000..9dd72d92 --- /dev/null +++ b/src/main/docs/results-interpretation-guide.adoc @@ -0,0 +1,154 @@ += JLBH Results Interpretation Guide +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +*_Abstract_*:: +This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. Effective interpretation is key to deriving meaningful insights from your latency benchmarks. + +== Understanding JLBH Console Output Structure + +When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). Understanding this output is the first step: + +* **Warm-up Messages**: Indicates the progress and completion of the warm-up phase (e.g., "Warm up complete (X iterations took Y s)"). +* **Per-Run Detailed Output**: For each measurement run, JLBH prints a section: +** Header indicating the run number (e.g., "BENCHMARK RESULTS (RUN 1) us"). +** Run metadata: Total run time, latency distribution strategy, coordinated omission status, target throughput. +** **Probe Statistics**: For each active probe (the default "End to End" probe, any custom probes added via `jlbh.addProbe()`, and the "OS Jitter" probe if enabled), a line showing: +*** Probe name and total samples recorded for that probe in the run. +*** A compact representation of key percentiles (e.g., "50/90 99/99.9 99.99 - worst was X / Y Z / A B - C"). These values are usually in microseconds. +* **Final Summary Tables**: After all runs are complete, JLBH prints summary tables for each probe: +** Header: "SUMMARY (probe_name) us". +** Rows for key percentiles (50.0, 90.0, 99.0, 99.9, etc., up to 'worst'). +** Columns for each run, showing the latency value for that percentile in that run. +** A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). + +== Interpreting Latency Histograms and Percentiles + +JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. These histograms are then used to derive percentile values. + +* **What is a Percentile?** +** A percentile indicates the value below which a given percentage of observations fall. For example, the 99th percentile latency is the value X where 99% of all measured latencies were less than or equal to X, and 1% were greater. +* **Key Percentiles in JLBH Output**: +** `50.0` (Median): The middle value; 50% of latencies are lower, 50% are higher. It gives a sense of the "typical" latency but can be misleading on its own if the distribution is skewed. +** `90.0`: 90% of requests were faster than this value. +** `99.0` (99th or "p99"): A common metric for Service Level Objectives (SLOs). Indicates that 1 out of 100 requests experienced this latency or worse. +** `99.9` (99.9th or "p999", "three nines"): Important for high-performance systems; 1 out of 1000 requests were this slow or slower. +** `99.99` (99.99th or "p9999", "four nines"): Critical for systems demanding very high consistency in performance. +** `worst`: The single highest latency recorded during the run. This can be affected by outliers but is important to note, especially if it's dramatically different from high percentiles. +* **Why Percentiles Matter**: +** Averages can be heavily skewed by a small number of very low or very high latencies, masking the true experience of most requests. +** Tail latencies (e.g., p99 and above) often drive user-perceived performance and system stability. A system with a good average latency but terrible tail latency can still feel unresponsive or unreliable. +* **Units**: Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. Pay attention to the units indicated in the table headers. +* **Custom Probes**: If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. + +== Co-ordinated Omission (CO) + +Co-ordinated Omission is a critical concept in latency measurement, especially for systems that process requests at a certain rate. + +* **The Problem**: If a system (or the benchmark harness itself) pauses or slows down, it might temporarily stop measuring. When it resumes, it might only measure the (potentially fast) service times of requests that are processed immediately, "omitting" the time that other requests would have spent waiting *during* the pause. This leads to unrealistically optimistic latency figures, particularly at the tail. +* **How JLBH Compensates**: +** JLBH addresses this by default (`accountForCoordinatedOmission = true` in `JLBHOptions`). +** For each iteration, JLBH calculates an *ideal scheduled start time* based on the target throughput. +** It then actively waits (busy-spins using `System.nanoTime()`) until this ideal start time is reached before invoking `JLBHTask.run(startTimeNs)` with that ideal time. +** The latency is then calculated as `System.nanoTime() - idealStartTimeNs`. +** This means if the system or harness was delayed, the measured latency for that sample will correctly include that delay, reflecting the experience a request would have had if it arrived at its scheduled time. +* **Impact on Results**: +** *With CO compensation (default)*: Histograms reflect a more accurate picture of the latency a user or client would experience, including any queuing or delays caused by the system struggling to keep up with the offered load. Tails are usually higher and more realistic. +** *Without CO compensation*: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. This can be useful for understanding raw compute time but not overall system responsiveness under load. +* **Comparison**: Always state whether CO compensation was enabled when comparing benchmark results. Comparing a CO-compensated run with a non-compensated run is misleading. JLBH prints "Correcting for co-ordinated:true/false" in each run's output. + +== OS Jitter Probe + +JLBH can optionally measure Operating System (OS) jitter, which refers to delays caused by the OS scheduler or other system activities interrupting the benchmark process. + +* **Mechanism**: +** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. +** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. +** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +* **Interpreting Jitter Output**: +** The "OS Jitter" probe will have its own percentile table in the JLBH output. +** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." +** *Low jitter values* (e.g., a few microseconds) indicate that the benchmark process was generally ableto run without significant external interference from the OS. +* **Why it's Useful**: +** Helps distinguish between latency caused by your application code versus latency caused by the environment. +** If your application shows high tail latencies, and the OS Jitter probe also shows high values around the same magnitude, it's a strong indicator that external factors are at play. +* **Actions for High Jitter**: +** Ensure the benchmark is running on a quiet machine with minimal other load. +** Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). +** Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). + +== Analysing Run-to-Run Variation + +Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. JLBH's summary output includes a "% Variation" column for each percentile. + +* **The Variation Column**: This shows the percentage difference between the maximum and minimum values recorded for a given percentile across all included runs. +** By default (`JLBHOptions.skipFirstRun = NOT_SET`), if there are more than 3 runs, the first run is often excluded from this variation calculation as it might still be affected by late-stage JIT or other one-off initializations. You can control this explicitly with `skipFirstRun(boolean)`. +* **Interpreting Variation**: +** *Low Variation (e.g., < 5-10%)*: Suggests a stable benchmarking environment and that your application's performance is repeatable under the test conditions. This increases confidence in the results. +** *High Variation*: Indicates instability. Possible causes include: +*** **Insufficient Warm-up**: The JVM might still be optimizing code differently across runs. Try increasing `warmUpIterations`. +*** **Garbage Collection (GC) Pauses**: Infrequent but long GC pauses can hit some runs but not others, drastically affecting tail percentiles. Monitor GC activity. +*** **External System Noise**: Other processes on the machine, network fluctuations (if applicable), or even hardware power management. +*** **Adaptive JVM Behavior**: Some JVM optimizations are adaptive and can change behavior between runs if conditions shift slightly. +*** **Benchmarked Code Instability**: The code itself might have inherent variability or be interacting with unstable external resources. +* **Goal**: Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. Focus on understanding systematic performance rather than chasing the single "best" score from one run. Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. + +== The Throughput vs. Latency Relationship + +A fundamental aspect of performance is the trade-off between throughput (how much work is done per unit of time) and latency (how long each piece of work takes). + +* **JLBH's Role**: JLBH allows you to set a target `throughput` via `JLBHOptions`. This controls the rate at which `JLBHTask.run()` is invoked. +* **Typical Behavior**: +** At low throughputs, a system can often maintain low latency. +** As throughput increases, contention for resources (CPU, memory, network, locks) typically rises. +** Beyond a certain point, latency will start to increase, often non-linearly. Queues may build up, and the system struggles to keep pace. +* **Using JLBH to Explore**: +** Run your benchmark with different `throughput` settings to understand how your application's latency responds to varying load. +** This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. +** The results can inform capacity planning and help set realistic performance expectations. + +== Common Pitfalls in Interpretation + +Avoiding these common mistakes can help ensure your benchmark results are meaningful and your conclusions are sound: + +* **Insufficient Warm-up**: +** *Issue*: The JVM performs JIT compilation, class loading, and cache warming during initial execution. Short or inadequate warm-up can mean you're measuring pre-optimized code or including one-off startup costs. +** *Solution*: Ensure `warmUpIterations` is sufficient. A common rule of thumb is at least the JVM's compilation threshold (e.g., `Jvm.compileThreshold()` in Chronicle Core, typically 10,000-15,000 iterations) plus some buffer for application-specific warm-up. Monitor if results stabilize after a certain number of runs. +* **Ignoring Co-ordinated Omission Setting**: +** *Issue*: Comparing results where one run had CO compensation enabled and another didn't will lead to incorrect conclusions about performance changes. +** *Solution*: Always be aware of the `accountForCoordinatedOmission` setting (default is `true`) and report it alongside your results. +* **Benchmark Overhead**: +** *Issue*: The act of measuring itself (calling `System.nanoTime()`, `jlbh.sample()`) has a small overhead. For extremely fast operations (deep sub-microsecond), this overhead can become a significant percentage of the measured time. +** *Solution*: JLBH is designed for low overhead (aiming for <100ns). Be mindful of this when benchmarking trivial operations. Consider batching very small operations if appropriate, or if the raw speed of an isolated, tiny operation is critical, a more specialized microbenchmarking tool like JMH might be used for that specific part, with JLBH testing the larger context. +* **Unrealistic Throughput Targets**: +** *Issue*: Setting `throughput` far beyond what the system can handle will just measure a system under constant overload, with latencies dominated by queue times. Setting it too low might not expose bottlenecks that appear under moderate load. +** *Solution*: Test a range of throughputs relevant to your expected operational load and capacity limits. +* **Focusing Only on Averages**: +** *Issue*: Averages hide outliers and don't reflect the experience of users hitting tail latencies. +** *Solution*: Prioritize percentile analysis (p50, p90, p99, p99.9, worst) as JLBH encourages. +* **Ignoring Custom Probe Data**: +** *Issue*: If you've set up custom probes for different stages of an operation but only look at the end-to-end numbers, you miss valuable insights into internal bottlenecks. +** *Solution*: Analyze the percentile data for each custom probe. Compare their contributions to the overall latency. +* **Environmental Inconsistency**: +** *Issue*: Running benchmarks on different hardware, with varying background OS/application load, different JVM versions, or different system configurations will yield different results. +** *Solution*: For comparative analysis, always use a consistent, controlled, and quiet environment. Document the environment meticulously. +* **Thread Pinning (Affinity) Issues**: +** *Issue*: On multi-core systems, OS thread scheduling can move your benchmark thread between cores, causing cache misses and jitter. Forgetting to pin, or pinning to a busy/shared core, can skew results. +** *Solution*: Use `JLBHOptions.acquireLock(Affinity::acquireLock)` (from OpenHFT Affinity library) to attempt to pin the main benchmark thread to an isolated core. Similarly for `jitterAffinity` if using the OS Jitter probe. Ensure these cores are genuinely isolated. +* **Not Enough Iterations/Run Duration**: +** *Issue*: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. +** *Solution*: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. This often means runs lasting at least several seconds, or even minutes for deep stability tests. + +== Deriving Actionable Insights + +The ultimate goal of benchmarking is to gain insights that can lead to improvements. + +* **Identify Bottlenecks**: +** High end-to-end latency? Use custom probes to break down the operation and see which stage is the culprit. +** High p99 or p99.9 values? This often points to issues like GC, network spikes, lock contention, or OS jitter. Correlate with OS Jitter probe and GC logs. +* **Validate Optimizations**: Run benchmarks before and after a code change to quantify its impact on latency across different percentiles and throughputs. +* **Capacity Planning**: Determine the throughput your system can handle while meeting latency SLOs. +* **Regression Testing**: Integrate JLBH benchmarks into your CI/CD pipeline to catch performance regressions automatically. +* **Understand Trade-offs**: Use JLBH to explore the impact of different configurations, algorithms, or architectural choices on the latency profile.