Skip to content

benchmark-lifecycle.adoc

Latest commit

 

History

History
79 lines (62 loc) · 6.1 KB

benchmark-lifecycle.adoc

File metadata and controls

79 lines (62 loc) · 6.1 KB
Raw

Benchmark Lifecycle

Table of Contents
graph LR
    A[Setup] --> B(Warmup);
    B --> C{Execution Per Run};
    C --> D[Reporting];
    D --> E[Cleanup];

A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are:

1. Setup

This initial phase prepares the benchmark environment:

  • JLBH Configuration: The user instantiates JLBHOptions to define all benchmark parameters (e.g., iterations, throughput, runs, coordinated omission settings). An instance of JLBH is then created using these options.

  • Task Initialization: The JLBHTask.init(JLBH jlbh) method of the user-provided task is invoked.

    • This allows the benchmark task to perform its own one-time setup, such as initializing resources or fixtures.

    • Crucially, this is where additional measurement probes can be registered by calling jlbh.addProbe("probeName") to get NanoSampler instances for specific sub-sections of the task.

  • OS Jitter Monitoring: Jitter tracking is enabled by default (recordOSJitter defaults to true); unless disabled via recordOSJitter(false), the OSJitterMonitor background thread starts to measure operating system scheduling jitter independently.

  • Affinity: If an acquireLock supplier is configured in JLBHOptions, an attempt to acquire CPU affinity for the main benchmark thread might occur.

2. Warmup

Before any measurements are formally recorded, the harness executes a warm-up phase:

  • Warmup Iterations: The JLBHTask.run(long startTimeNs) method is called warmUpIterations times (as specified in JLBHOptions). The startTimeNs passed is typically just System.nanoTime() for this phase.

    • The primary goal is to allow the Java Virtual Machine (JVM) to perform Just-In-Time (JIT) compilation and optimize the benchmarked code paths.

    • It also helps in bringing caches to a "warm" state and letting the system reach a more stable performance profile.

  • Sample Discarding: Latency samples generated during the warmup phase are usually recorded into the histograms but are not part of the final reported results. The histograms are reset after this phase.

  • Task Notification: After all warmup iterations are complete, the JLBHTask.warmedUp() method is called.

  • Optional Pause: If JLBHOptions.pauseAfterWarmupMS is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase.

3. Execution (Measurement Runs)

This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by JLBHOptions.runs):

  • Per Run:

    • The following steps are repeated for each configured run.

    • Iteration Loop: The JLBH orchestrator executes a loop for the number of iterations specified in JLBHOptions.

      • Start Time Calculation: For each iteration, JLBH calculates an ideal startTimeNs. This calculation considers:

        • The configured throughput (e.g., messages per second).

        • The LatencyDistributor strategy, if one is set, to potentially vary the inter-iteration delay.

        • If accountForCoordinatedOmission is true (the default), startTimeNs represents the ideal scheduled time for the operation. JLBH will then busy-wait (spin) until System.nanoTime() reaches this startTimeNs, ensuring the task is dispatched as close as possible to its intended schedule, thus accounting for delays that might otherwise be missed.

      • Task Invocation: The JLBHTask.run(startTimeNs) method is invoked with the calculated (and potentially waited-for) startTimeNs.

      • Sample Recording:

        • Within the JLBHTask.run() method, the user’s code is executed.

        • To record the primary end-to-end latency for the iteration, the task must call jlbh.sample(System.nanoTime() - startTimeNs).

        • For any custom probes registered during setup, the task can call myProbe.sampleNanos(duration) to record latencies for specific sub-stages.

        • Each recorded sample is added to the respective Histogram associated with the sampler (either the main JLBH sampler or a custom probe’s sampler).

    • Run Completion: After all iterations for the current run are finished:

      • The JLBHTask.runComplete() method is invoked.

      • Statistics for this specific run (e.g., percentiles derived from the histograms) are typically printed to the configured PrintStream.

      • The collected histogram data for this run is processed and stored internally for the final JLBHResult.

      • All probe histograms are then reset to ensure that measurements for the next run (if any) are independent.

4. Reporting

After all measurement runs are completed, the aggregated results are finalized and made available:

  • Final Summary: A comprehensive summary, often comparing results across all runs and highlighting percentile variations, is printed to the PrintStream.

  • Result Object Creation: The final, immutable JLBHResult object is constructed. This object encapsulates all recorded data for all probes across all runs.

  • Result Consumption: If a JLBHResultConsumer was provided during JLBH setup, its accept(JLBHResult) method is called, passing the JLBHResult object. This allows for programmatic access to the detailed results, for example, for assertion in automated tests or for custom serialization.

  • External Tools: Users can further process the JLBHResult using utilities like:

    • JLBHResultSerializer to write results to a CSV file for analysis in spreadsheets or other tools.

    • TeamCityHelper to output statistics in a format suitable for TeamCity CI server integration.

5. Cleanup

The final phase ensures that any resources are properly released:

  • Task Cleanup: The JLBHTask.complete() method is called, allowing the user’s benchmark task to perform any necessary cleanup (e.g., closing files, releasing network connections).

  • OS Jitter Monitor: If the OSJitterMonitor thread was started, it is signaled to terminate.

  • Affinity Release: If a CPU affinity lock was acquired by JLBH at the start, it is released.