Skip to content

decision-log.adoc

Latest commit

 

History

History
229 lines (185 loc) · 12.7 KB

decision-log.adoc

File metadata and controls

229 lines (185 loc) · 12.7 KB
Raw

Chronicle Threads - Decision Log

Table of Contents

This file captures component-specific architectural and operational decisions for Chronicle Threads. Identifiers follow the THR-<Tag>-NNN pattern from the Nine-Box taxonomy (FN, NF-P, NF-S, NF-O, TEST, DOC, OPS, UX, RISK). Numbers are unique within the THR scope.

Decision Records

THR-FN-001 Event loops are single-threaded for handler execution

Date

2015-02-24

Context

  • Chronicle Threads aims to simplify concurrent programming for event-driven systems while delivering predictable low-latency behaviour.

  • Traditional multi-threaded handler models often require complex locking, which introduces contention, bugs, and unpredictable pauses on the hot path.

Decision Statement

  • Each EventLoop instance runs on a dedicated Java platform thread.

  • All EventHandler instances registered with a given EventLoop (except handlers with BLOCKING or CONCURRENT priorities, which use their own threading model within an EventGroup) are executed serially by that single thread.

Alternatives Considered
* Multi-threaded event loops

  • Description: Execute handlers from a shared loop across a thread pool.

  • Pros: Higher potential throughput if handlers are fully independent.

  • Cons: Requires thread-safe handlers, more locking, less predictable latency, and harder reasoning about shared state.

* Actor model per handler

  • Description: Each handler has its own queue and dedicated thread.

  • Pros: Strong isolation between handlers.

  • Cons: Higher resource usage (threads, queues) and higher message-passing overhead than serial invocation on one loop.

Rationale for Decision

  • Single-threaded loops make handler implementation simpler by removing most intra-loop locking requirements.

  • They support low-jitter latency by avoiding lock contention in the main execution path.

  • The model aligns with a common thread-per-core pattern used in low-latency systems.

Impact & Consequences

  • Positive:

    • Simplifies handler development and testing.

    • Improves latency predictability for all handlers on a given loop.

    • Makes it easier to reason about state owned by handlers on one loop.

  • Negative:

    • Throughput of a single EventLoop is bounded by one CPU core.

    • A misbehaving or blocking handler on a standard loop can starve other handlers on that loop. This is mitigated by using EventGroup with BLOCKING or CONCURRENT handlers on separate threads or thread pools.

Notes/Links

THR-NF-P-002 Busy-spin pauser for latency-critical fast threads

Date

2015-02-24

Context

  • For latency-critical fast threads (often pinned to isolated cores), minimising wake-up time when new work arrives is essential.

  • Standard pause strategies that yield or sleep introduce variable and often large delays before a thread resumes work.

Decision Statement

  • EventLoop instances configured for maximum performance (for example using PauserMode.BUSY or PauserMode.TIMED_BUSY) use a busy-spinning or near busy-spinning Pauser by default.

  • Other pauser modes (YIELDING, BALANCED, SLEEPY, MILLI) remain available and are recommended for less latency-sensitive loops or where CPU cores are constrained.

Alternatives Considered
* Yielding pauser as default

  • Description: Call Thread.yield() when idle.

  • Pros: More CPU-friendly than pure busy-spin.

  • Cons: Higher and less predictable wake-up latency.

* Sleeping pauser as default

  • Description: Sleep or park immediately when idle.

  • Pros: Lowest CPU usage when idle.

  • Cons: Highest wake-up latency and jitter.

* Adaptive pauser as universal default (for example BALANCED)

  • Description: Combine spinning, yielding, and sleeping adaptively.

  • Pros: Good general-purpose balance.

  • Cons: Not as fast as a dedicated busy-spin for the most latency-sensitive loops.

Rationale for Decision

  • Busy-spinning keeps the thread hot on its core and avoids scheduler and context-switch overhead, giving the minimum wake-up latency when events arrive.

  • Users can explicitly opt into less aggressive pausers when CPU utilisation must be reduced.

Impact & Consequences

  • Positive:

    • Minimises p99.99 and tail latency for designated fast threads.

    • Provides predictable entry into handler code once an event is visible.

  • Negative:

    • Consumes a full CPU core even when idle, so requires careful system configuration (isolated cores, enough headroom for other work).

    • Misuse (too many busy-spinning threads) can degrade overall system performance.

Notes/Links

THR-OPS-003 Background disk-space monitoring

Date

2018-05-11

Context

  • Chronicle Queue and other disk-backed Chronicle components can fail or lose data if the underlying storage fills up.

  • Operations teams need early, component-aware warning of low disk-space conditions to act before failure.

Decision Statement

  • Chronicle Threads provides a background monitoring singleton DiskSpaceMonitor.

  • The monitor periodically checks disk usage for paths associated with Chronicle components (for example queue directories as they are initialised).

  • When free space falls below a configurable threshold, a NotifyDiskLow service (default NotifyDiskLowLogWarn) logs warnings. The threshold is configured via system property chronicle.disk.monitor.threshold.percent.

Alternatives Considered
* No built-in monitoring

  • Description: Rely solely on external system-level disk monitoring.

  • Pros: Keeps the library simpler.

  • Cons: External monitoring might not be tuned to Chronicle usage; users can forget to set it up.

* More aggressive built-in actions

  • Description: Automatically halt writers when space is critically low.

  • Pros: Could prevent writes that would immediately fail.

  • Cons: Too intrusive for a general-purpose library; better handled by application policy.

Rationale for Decision

  • A built-in monitor provides a lightweight, Chronicle-aware early warning without enforcing policy.

  • Logging is non-intrusive and keeps responsibility for operational response with the application and operations teams.

  • The ServiceLoader based NotifyDiskLow contract allows custom notification or escalation strategies.

Impact & Consequences

  • Positive:

    • Gives early warning of storage exhaustion for Chronicle workloads.

    • Reduces the risk of unexpected failures caused by full disks.

  • Negative:

    • Adds minimal overhead (one background thread plus periodic I/O).

    • Effectiveness depends on log monitoring or a custom NotifyDiskLow implementation.

Notes/Links

  • Key classes: DiskSpaceMonitor, NotifyDiskLow, NotifyDiskLowLogWarn.

  • Properties: chronicle.disk.monitor.disable, chronicle.disk.monitor.threshold.percent (see also system properties table.)

THR-DOC-004 Nine-Box taxonomy for requirements and decisions

Date

2025-03-01

Context

  • The project needs a consistent, structured way to identify and trace requirements, decisions, tests, and possibly code artefacts.

  • Earlier schemes using ad-hoc or simple sequential numbering made it difficult to infer scope or intent from an identifier alone.

Decision Statement

  • All functional requirements (in project-requirements.adoc) and architectural decisions (in this log) use identifiers prefixed with THR-, followed by a Nine-Box tag (for example FN, NF-P, OPS, DOC) and a sequence number (for example THR-FN-001).

  • The Nine-Box tag definitions and usage guidelines are maintained in AGENTS.md.

Alternatives Considered
* Simple sequential numbering (for example REQ-001, DEC-001)

  • Description: Use only numeric sequence numbers.

  • Pros: Very easy to implement and explain.

  • Cons: Conveys no information about type or domain.

* Custom project-specific categorisation scheme

  • Description: Invent a bespoke set of categories for Chronicle Threads only.

  • Pros: Could be tailored closely to the project.

  • Cons: Extra effort to define and maintain; less transferable knowledge for staff working across multiple Chronicle projects.

Rationale for Decision

  • The Nine-Box taxonomy already provides a broadly applicable and documented set of categories.

  • Using the same taxonomy across Chronicle projects improves consistency and makes it easier for people to move between modules.

  • Including the tag in identifiers gives immediate context about what an item represents.

Impact & Consequences

  • Positive:

    • Better traceability between requirements, decisions, tests, and code.

    • Easier for team members to understand the role of an item from its ID.

    • Supports better organisation and searching in documentation and tooling.

  • Negative:

    • Requires familiarity with the Nine-Box taxonomy and consistent application.

    • Initial documentation and training overhead (captured in AGENTS.md).

Notes/Links

THR-FN-005 EventGroup priority sets gate handler registration

Date

2025-03-01

Context

  • EventGroup coordinates multiple child loops and routes handlers based on their declared HandlerPriority.

  • Not all deployments require every priority; some groups should expose a restricted set of priorities or be effectively disabled for application handlers.

  • Earlier behaviour allowed handlers to be added even when the group had not been configured explicitly with a corresponding priority set, making it harder to reason about which handlers were admissible.

Decision Statement

  • EventGroup instances are configured with a set of supported handler priorities.

  • Loops are only materialised for the priorities included in this set; other loops are omitted.

  • Attempts to register a handler whose HandlerPriority has no corresponding loop (for example because that priority is not in the configured set) fail fast with an IllegalStateException.

  • A group configured with an empty priority set retains only its monitor loop; attempts to add handlers with any other priority are rejected.

Alternatives Considered
* Implicit support for all priorities

  • Description: Treat all priorities as supported unless explicitly disabled, materialising loops lazily on first handler registration.

  • Pros: Minimal configuration, potentially simpler for small deployments.

  • Cons: Harder to predict resource usage and topology; accidental handler registration can silently create extra loops and threads.

* Soft failure for unsupported priorities

  • Description: Drop or log handlers whose priorities are not supported instead of throwing.

  • Pros: Avoids exceptions in misconfigured systems.

  • Cons: Silent misconfiguration is dangerous; handlers may never run with little or no visibility.

Rationale for Decision

  • An explicit priority set makes EventGroup topology predictable and keeps resource allocation under operator control.

  • Failing fast when a handler targets an unsupported priority surfaces configuration errors early in development and testing.

  • Retaining only the monitor loop when the set is empty supports scenarios where an EventGroup instance is used purely for monitoring infrastructure rather than application handlers.

Impact & Consequences

  • Positive:

    • Clearer reasoning about which handlers can attach to a given group.

    • Better alignment between configuration (builder calls) and the underlying loop topology.

    • Easier to construct priority-limited or monitor-only groups without surprising background loops.

  • Negative:

    • A stricter contract means some previously permissive configurations now fail fast and require explicit updates to their configured priority sets.

    • Callers that expect to use priorities that are not enabled for a given group must update their configuration or routing.

Notes/Links