Performance Tuning Guide

Comprehensive guide to HDF5 B-tree rebalancing and performance optimization

This guide explains B-tree rebalancing strategies for optimal HDF5 performance, particularly for deletion-heavy workloads with TB-scale scientific data.

Introduction
Understanding B-tree Rebalancing
Rebalancing Modes
Performance Characteristics
Workload-Specific Recommendations
Configuration Guide
Troubleshooting
Advanced Topics

Introduction

What is B-tree Rebalancing?

HDF5 uses B-tree v2 data structures to store dense attribute collections (8+ attributes) and other metadata. When you delete attributes, records are removed from B-tree nodes, potentially leaving them underutilized (sparse).

Rebalancing is the process of reorganizing B-tree nodes after deletions to:

Maintain balance: Keep all leaf nodes at the same depth
Optimize space: Merge underfull nodes to reduce overhead
Improve performance: Speed up searches by reducing tree depth

Why It Matters for HDF5 Performance

For scientific workloads processing TB-scale files with thousands of datasets:

Without rebalancing: B-trees become increasingly sparse → slower searches, wasted disk space
With naive rebalancing: Every deletion triggers expensive tree restructuring → 10-100x slower writes
With smart rebalancing: Batch or background processing → optimal balance of speed and efficiency

This library provides 4 rebalancing strategies to match your specific workload.

When to Use Each Mode

Quick Decision Tree:

Are you doing deletions?
├─ No → Use default (no rebalancing)
└─ Yes
   ├─ Small files (<100MB) → Use default (rebalancing overhead not worth it)
   └─ Large files (≥100MB)
      ├─ Batch deletions (delete many, then continue) → Use lazy rebalancing
      ├─ Continuous operations (can't afford pauses) → Use incremental rebalancing
      └─ Don't know pattern / want autopilot → Use smart rebalancing

Understanding B-tree Rebalancing

B-tree Basics

A B-tree v2 is a self-balancing tree structure used by HDF5 for:

Dense attribute storage (8+ attributes per object)
Link name index in groups
Other metadata collections

Key Properties:

All leaf nodes at same depth (balanced)
Each node ≥50% full (except root)
Records sorted by hash for fast lookup
Typical order: ~100-200 records per node

What Happens During Deletion

Without Rebalancing:

Initial B-tree (3 nodes, well-balanced):
    [Node A: 80% full]
         /        \
[Node B: 70% full] [Node C: 75% full]

After 50% deletion (NO rebalancing):
    [Node A: 40% full]  ← Underfull!
         /        \
[Node B: 35% full] [Node C: 40% full]  ← Both underfull!

Problem: Sparse tree, wasted space, slower searches

With Rebalancing:

After 50% deletion + rebalancing:
    [Node A: 75% full]
         |
    (Merged B + C → single well-filled node)

Result: Compact tree, efficient searches

The Performance Dilemma

Immediate Rebalancing (traditional approach):

✅ Keeps B-tree always optimal
❌ Very expensive: Each deletion triggers node merging, parent updates, disk writes
❌ 10-100x slower for batch deletion workloads

No Rebalancing (this library's default, like C library):

✅ Fast deletions: Just remove record, no restructuring
❌ B-tree becomes sparse over time
❌ Wastes disk space, slower subsequent operations

Solution: Deferred Rebalancing Strategies (this guide covers all options)

Rebalancing Modes

This library offers 4 rebalancing modes, from simplest to most sophisticated:

1. No Rebalancing (Default)

What it does: Never rebalances automatically. B-trees can become sparse.

When to use:

Append-only workloads (no deletions)
Small files (<100MB)
Read-heavy workloads where write performance is critical
You want identical behavior to HDF5 C library

Performance:

✅ Fastest deletion: 0% overhead
✅ Zero CPU cost for rebalancing
❌ B-tree becomes sparse if many deletions occur
❌ May waste disk space

Example:

// No options = no rebalancing (default, like C library)
fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
if err != nil {
    return err
}
defer fw.Close()

// Write and delete - no automatic rebalancing
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000})
ds.WriteAttribute("attr1", 42)
ds.DeleteAttribute("attr1")  // Fast! No rebalancing

2. Lazy Rebalancing (Batch Processing)

What it does: Accumulates deletions and rebalances in batches when threshold is reached.

How it works:

Track deletions (counts underflow nodes)
When (underflow_nodes / total_nodes) ≥ threshold, trigger batch rebalancing
Process multiple nodes in single operation
Also triggers after MaxDelay time to prevent indefinite delay

When to use:

Batch deletion workloads: Delete many attributes, then continue working
Medium to large files (100-500MB)
Moderate delete ratios (5-20% of operations)
You can tolerate occasional 100-500ms pauses for rebalancing

Performance:

✅ 10-100x faster than immediate rebalancing
✅ Batching amortizes restructuring cost
✅ Minimal overhead between batches (~1-2%)
⏸️ Occasional pauses (100-500ms) during batch rebalancing
✅ B-tree stays reasonably compact

Example:

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),           // Trigger at 5% underflow
        hdf5.LazyMaxDelay(5*time.Minute),   // Force rebalance after 5 min
        hdf5.LazyBatchSize(100),            // Process 100 nodes per batch
    ),
)
if err != nil {
    return err
}
defer fw.Close()

// Delete many attributes - rebalancing happens in batches
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000})
for i := 0; i < 1000; i++ {
    ds.WriteAttribute(fmt.Sprintf("attr_%d", i), i)
}

// Delete 100 attributes - lazy rebalancing automatically batches
for i := 0; i < 100; i++ {
    ds.DeleteAttribute(fmt.Sprintf("attr_%d", i))
    // Rebalancing happens automatically when threshold reached
}

Tuning Parameters:

Parameter	Default	Range	Effect
`LazyThreshold`	0.05 (5%)	0.01-0.20	Lower = more frequent rebalancing, tighter tree
`LazyMaxDelay`	5 minutes	1s-1h	Forces rebalance even if threshold not met
`LazyBatchSize`	100 nodes	10-1000	Larger = fewer rebalancing events, longer pauses

Recommendations:

Aggressive batching: Threshold(0.10), BatchSize(200) → fewer, longer pauses
Tight tree: Threshold(0.02), BatchSize(50) → more frequent, shorter rebalancing
Write-heavy: MaxDelay(10*time.Minute) → avoid interrupting long write sessions

3. Incremental Rebalancing (Background Processing)

What it does: Rebalances B-trees in the background using a goroutine with time budgets.

How it works:

Requires lazy rebalancing as prerequisite (tracks underflow nodes)
Background goroutine wakes up every Interval (default: 5 seconds)
Rebalances for Budget time (default: 100ms), then pauses
Continues until all underflow nodes processed
Zero user-visible pause - rebalancing happens between operations

When to use:

Large files (>500MB) where lazy rebalancing pauses are noticeable
High delete ratios (>20% of operations)
Continuous operation workloads (can't afford pauses)
TB-scale scientific data with strict latency requirements

Performance:

✅ Zero user-visible pause: All rebalancing in background
✅ Eventual consistency: B-tree optimized over time
✅ Tunable CPU impact (adjust Budget and Interval)
⚠️ ~2-5% overhead (background goroutine + synchronization)
⚠️ ~100MB memory overhead for background processing (configurable)

Example:

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(),  // Prerequisite!
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(100*time.Millisecond),  // 100ms per session
        hdf5.IncrementalInterval(5*time.Second),       // Every 5 seconds
        hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
            log.Printf("Rebalancing: %d/%d nodes (ETA: %v)\n",
                p.NodesRebalanced, p.NodesRemaining, p.EstimatedRemaining)
        }),
    ),
)
if err != nil {
    return err
}
defer fw.Close()  // Automatically stops background goroutine

// Delete operations never block - rebalancing happens in background
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000})
for i := 0; i < 10000; i++ {
    ds.WriteAttribute(fmt.Sprintf("attr_%d", i), i)
}

for i := 0; i < 5000; i++ {
    ds.DeleteAttribute(fmt.Sprintf("attr_%d", i))
    // ZERO PAUSE! Rebalancing happens in background goroutine
}

Tuning Parameters:

Parameter	Default	Range	Effect
`IncrementalBudget`	100ms	10ms-1s	Time spent rebalancing per session
`IncrementalInterval`	5 seconds	1s-1min	How often to run rebalancing session
`ProgressCallback`	nil	func	Optional callback for progress monitoring

Budget vs Interval Trade-offs:

Configuration	CPU Impact	Rebalancing Speed	Use Case
Budget: 50ms, Interval: 10s	Very low (~1%)	Slow (gradual)	Low-priority background cleanup
Budget: 100ms, Interval: 5s	Low (~2-3%)	Moderate	Default: balanced
Budget: 200ms, Interval: 2s	Medium (~5%)	Fast	Aggressive rebalancing

Recommendations:

Low CPU impact: Budget(50ms), Interval(10s) → minimal overhead
Fast rebalancing: Budget(200ms), Interval(2s) → aggressive cleanup
Monitoring: Always set ProgressCallback to track rebalancing progress

4. Smart Rebalancing (Auto-Tuning, Optional)

What it does: Automatically detects workload patterns and selects optimal rebalancing mode.

How it works:

Workload Detection: Tracks operation patterns (inserts, deletes, reads)
Feature Extraction: Computes metrics (delete ratio, batch size, operation rate)
Mode Selection: Uses decision rules to choose: none, lazy, or incremental
Auto-Switching: Can switch modes as workload changes (optional)
Explainability: Provides confidence scores and reasoning for decisions

When to use:

Unknown workload patterns: Don't know access patterns in advance
Mixed workloads: Combination of batch and continuous operations
Auto-pilot mode: Want library to optimize automatically
Research/experimental setups with varying workloads

Performance:

✅ Adapts to workload automatically
✅ No manual tuning required
⚠️ ~3-7% overhead (detection + evaluation)
⚠️ ~1MB memory for operation history (10,000 operations)
⚠️ Decision overhead every ReevalInterval (default: 5 minutes)

Example:

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithSmartRebalancing(
        hdf5.SmartAutoDetect(true),   // Auto-detect workload patterns
        hdf5.SmartAutoSwitch(true),   // Auto-switch between modes
        hdf5.SmartMinFileSize(10*hdf5.MB),  // Only for files >10MB
        hdf5.SmartAllowedModes("lazy", "incremental"),  // Don't use "none"
        hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
            log.Printf("Rebalancing mode: %s (confidence: %.1f%%)\n",
                d.SelectedMode, d.Confidence*100)
            log.Printf("Reason: %s\n", d.Reason)
        }),
    ),
)
if err != nil {
    return err
}
defer fw.Close()

// Library automatically selects optimal rebalancing mode!
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000})

// Workload changes → smart rebalancer adapts automatically
// Phase 1: Batch writes (auto-selects "none")
for i := 0; i < 1000; i++ {
    ds.WriteAttribute(fmt.Sprintf("attr_%d", i), i)
}

// Phase 2: Batch deletes (auto-switches to "lazy")
for i := 0; i < 500; i++ {
    ds.DeleteAttribute(fmt.Sprintf("attr_%d", i))
}

// Phase 3: Mixed operations (might switch to "incremental")
for i := 500; i < 1000; i++ {
    if i%2 == 0 {
        ds.DeleteAttribute(fmt.Sprintf("attr_%d", i))
    } else {
        ds.WriteAttribute(fmt.Sprintf("new_%d", i), i*2)
    }
}

Decision Factors:

The smart rebalancer considers:

File Size: Small files (<10MB) → usually "none"
Delete Ratio: High deletes (>20%) → "lazy" or "incremental"
Batch Size: Large batches (>100 ops) → "lazy"
Operation Rate: High rate (>1000 ops/sec) → "incremental"
Workload Stability: Stable patterns → confident decisions

Tuning Parameters:

Parameter	Default	Effect
`SmartAutoDetect`	true	Enable workload pattern detection
`SmartAutoSwitch`	true	Allow mode switching
`SmartMinFileSize`	10MB	Minimum file size for auto-rebalancing
`SmartAllowedModes`	all	Restrict which modes can be selected
`SmartOnModeChange`	nil	Callback when mode changes

When NOT to Use:

Known, stable workload (manual mode selection is faster)
Very small files (<10MB) where overhead isn't worth it
Need deterministic performance (smart mode adds variability)

Performance Characteristics

Benchmark Comparison

Test Setup:

1000 attributes created, then 500 deleted
Measured on modern desktop (AMD Ryzen 7, NVMe SSD)
Results averaged over 10 runs

Rebalancing Mode	Deletion Speed	Space Efficiency	CPU Overhead	Pause Time
None (default)	100% (baseline)	60% (sparse tree)	0%	None
Lazy (5% threshold)	95% (5% slower)	95% (tight tree)	~2%	100-500ms batches
Incremental (100ms budget)	92% (8% slower)	95% (tight tree)	~4%	None (background)
Smart (auto)	88% (12% slower)	90-95% (adapts)	~6%	Varies

Key Takeaways:

Lazy is 10-100x faster than immediate rebalancing (not shown: immediate = 1-5% baseline speed)
Incremental has zero user-visible pause (critical for TB-scale data)
Smart mode trades 6% overhead for automatic optimization
All modes vastly better than immediate rebalancing for batch deletes

Operations Per Second

Deletion Throughput (higher is better):

Mode	Small Files (<100MB)	Large Files (>500MB)	Notes
None	10,000 ops/sec	10,000 ops/sec	No rebalancing overhead
Lazy	9,500 ops/sec	9,200 ops/sec	Occasional batch pauses
Incremental	9,000 ops/sec	8,800 ops/sec	Background goroutine sync
Smart	8,500 ops/sec	8,300 ops/sec	Detection overhead

Memory Usage:

Mode	Overhead	Notes
None	0 MB	Just operation counters
Lazy	<1 MB	Underflow node tracking
Incremental	~100 MB	Background processing buffers
Smart	~1-2 MB	Operation history (10K ops)

Workload-Specific Recommendations

Append-Only Workloads

Characteristics:

Only inserts, no deletions
Examples: Logging, sensor data collection, append-only time series

Recommendation: No rebalancing (default)

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
// No options = no rebalancing

Why:

No deletions → B-tree never becomes sparse
Rebalancing has zero benefit, only overhead
Matches HDF5 C library behavior (users expect this)

Batch Deletion Workloads

Characteristics:

Write many attributes/objects
Delete many in batch
Continue working (can tolerate brief pauses)
Examples: Data cleaning, batch processing pipelines, ETL jobs

Recommendation: Lazy rebalancing

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),         // 5% underflow
        hdf5.LazyMaxDelay(5*time.Minute), // Force after 5 min
        hdf5.LazyBatchSize(100),          // 100 nodes per batch
    ),
)

Why:

10-100x faster than immediate rebalancing
Batching amortizes restructuring cost
100-500ms pauses are acceptable for batch jobs
B-tree stays reasonably compact

Tuning Tips:

For aggressive batching: Increase LazyBatchSize(200) and LazyThreshold(0.10)
For tighter tree: Decrease LazyThreshold(0.02) and LazyBatchSize(50)

Large Files with Moderate Deletes

Characteristics:

File size >500MB
10-20% of operations are deletes
Can afford small overhead for optimization
Examples: Long-running simulations, iterative data processing

Recommendation: Incremental rebalancing

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(),  // Prerequisite
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(100*time.Millisecond),
        hdf5.IncrementalInterval(5*time.Second),
    ),
)
defer fw.Close()  // Stops background goroutine

Why:

Zero user-visible pause (all rebalancing in background)
Critical for TB-scale files where lazy rebalancing pauses would be noticeable
~2-5% overhead is acceptable for large files
B-tree stays optimized without blocking operations

Tuning Tips:

For low CPU impact: IncrementalBudget(50ms), Interval(10s)
For faster rebalancing: IncrementalBudget(200ms), Interval(2s)
Always monitor: Set ProgressCallback to track rebalancing

Continuous Heavy-Delete Workloads

Characteristics:

High delete ratio (>20%)
Continuous operations (no natural pause points)
Cannot tolerate any pause
Examples: Real-time data processing, streaming ingestion with pruning

Recommendation: Incremental rebalancing (aggressive)

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.10),  // Higher threshold for incremental
    ),
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(200*time.Millisecond),  // Aggressive
        hdf5.IncrementalInterval(2*time.Second),       // Frequent
        hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
            if p.NodesRemaining > 1000 {
                log.Warn("Rebalancing backlog: %d nodes", p.NodesRemaining)
            }
        }),
    ),
)
defer fw.Close()

Why:

Must use incremental (lazy pauses unacceptable)
Aggressive settings prevent backlog buildup
Monitoring callback alerts if rebalancing can't keep up

Warning: If deletes far exceed rebalancing capacity, consider:

Increase IncrementalBudget further
Decrease IncrementalInterval
Dedicate more CPU to rebalancing (may impact main workload)

Mixed/Unknown Workloads

Characteristics:

Workload pattern unknown or varies over time
Research environment, exploratory analysis
Want "auto-pilot" optimization
Examples: Interactive notebooks, ad-hoc queries, research pipelines

Recommendation: Smart rebalancing

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithSmartRebalancing(
        hdf5.SmartAutoDetect(true),
        hdf5.SmartAutoSwitch(true),
        hdf5.SmartMinFileSize(10*hdf5.MB),
        hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
            log.Info("Mode: %s (%.1f%% confidence): %s",
                d.SelectedMode, d.Confidence*100, d.Reason)
        }),
    ),
)
defer fw.Close()

Why:

Library adapts to changing workload automatically
No manual tuning required
Callback provides transparency (know why mode was selected)

Trade-off: ~6% overhead for auto-tuning

Configuration Guide

Lazy Rebalancing Parameters

LazyThreshold

What it controls: When to trigger batch rebalancing

Formula: (underflow_nodes / total_nodes) ≥ threshold

Range: 0.01 (1%) to 0.20 (20%)

Default: 0.05 (5%)

Examples:

// Conservative (tight tree, more frequent rebalancing)
hdf5.LazyThreshold(0.02)  // Trigger at 2% underflow

// Default (balanced)
hdf5.LazyThreshold(0.05)  // Trigger at 5% underflow

// Aggressive (loose tree, less frequent rebalancing)
hdf5.LazyThreshold(0.10)  // Trigger at 10% underflow

When to adjust:

Decrease (0.02) if: Disk space is limited, search performance critical
Increase (0.10) if: Write performance is critical, disk space abundant

LazyMaxDelay

What it controls: Maximum time before forcing rebalancing

Purpose: Prevents indefinite delay in write-heavy workloads

Range: 1 second to 1 hour

Default: 5 minutes

Examples:

// Short delay (ensure timely rebalancing)
hdf5.LazyMaxDelay(1*time.Minute)

// Default
hdf5.LazyMaxDelay(5*time.Minute)

// Long delay (minimize interruptions)
hdf5.LazyMaxDelay(30*time.Minute)

When to adjust:

Decrease (1 min) if: Want predictable rebalancing, file size growth is concern
Increase (30 min) if: Long write sessions, can't afford interruptions

LazyBatchSize

What it controls: How many nodes to rebalance per batch

Trade-off: Larger batches = fewer events but longer pauses

Range: 10 to 1000 nodes

Default: 100 nodes

Examples:

// Small batches (shorter pauses, more frequent)
hdf5.LazyBatchSize(50)

// Default
hdf5.LazyBatchSize(100)

// Large batches (longer pauses, fewer events)
hdf5.LazyBatchSize(200)

Pause time estimate (approximate):

50 nodes: ~50-100ms pause
100 nodes: ~100-200ms pause
200 nodes: ~200-500ms pause

When to adjust:

Decrease (50) if: Very latency-sensitive, can tolerate more frequent pauses
Increase (200) if: Batch jobs, want fewer interruptions

Incremental Rebalancing Parameters

IncrementalBudget

What it controls: Time spent rebalancing per background session

Trade-off: Larger budget = more CPU per session, faster rebalancing

Range: 10ms to 1 second

Default: 100ms

Examples:

// Low CPU impact (minimal overhead, slower rebalancing)
hdf5.IncrementalBudget(50*time.Millisecond)

// Default (balanced)
hdf5.IncrementalBudget(100*time.Millisecond)

// High throughput (faster rebalancing, higher overhead)
hdf5.IncrementalBudget(200*time.Millisecond)

CPU overhead estimate:

50ms budget, 10s interval: ~0.5% CPU
100ms budget, 5s interval: ~2% CPU
200ms budget, 2s interval: ~10% CPU

When to adjust:

Decrease (50ms) if: CPU constrained, rebalancing is low priority
Increase (200ms) if: Need aggressive rebalancing, CPU available

IncrementalInterval

What it controls: How often to run rebalancing sessions

Trade-off: Shorter interval = more frequent rebalancing, higher overhead

Range: 1 second to 1 minute

Default: 5 seconds

Examples:

// Infrequent (low overhead, batching effect)
hdf5.IncrementalInterval(10*time.Second)

// Default (balanced)
hdf5.IncrementalInterval(5*time.Second)

// Frequent (aggressive rebalancing)
hdf5.IncrementalInterval(2*time.Second)

When to adjust:

Decrease (2s) if: High delete rate, need to prevent backlog
Increase (10s) if: Low delete rate, minimize overhead

IncrementalProgressCallback

What it controls: Callback for monitoring rebalancing progress

Optional: Can be nil (no progress reporting)

Example:

hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
    log.Printf("Rebalancing: %d/%d nodes (%.1f%% complete, ETA: %v)\n",
        p.NodesRebalanced,
        p.NodesRemaining,
        float64(p.NodesRebalanced)/float64(p.NodesRebalanced+p.NodesRemaining)*100,
        p.EstimatedRemaining,
    )

    // Alert if backlog is building up
    if p.NodesRemaining > 1000 {
        log.Warn("Large rebalancing backlog: %d nodes", p.NodesRemaining)
    }
})

Best Practices:

Always set callback for production systems (visibility into rebalancing)
Alert on large backlogs (may need to adjust Budget/Interval)
Log progress periodically (helps debug performance issues)

Smart Rebalancing Parameters

SmartAutoDetect

What it controls: Enable automatic workload pattern detection

Default: true

Example:

hdf5.SmartAutoDetect(true)  // Detect patterns (default)
hdf5.SmartAutoDetect(false) // Disable detection (manual mode selection)

SmartAutoSwitch

What it controls: Allow automatic mode switching

Default: true

Example:

hdf5.SmartAutoSwitch(true)  // Auto-switch modes (default)
hdf5.SmartAutoSwitch(false) // Initial mode selection only, no switching

When to disable: If mode switching causes performance jitter, disable to keep initial selection

SmartMinFileSize

What it controls: Minimum file size for enabling auto-rebalancing

Purpose: Avoid rebalancing overhead on small files where it's not beneficial

Default: 10 MB

Example:

hdf5.SmartMinFileSize(1*hdf5.MB)   // Aggressive (even small files)
hdf5.SmartMinFileSize(10*hdf5.MB)  // Default
hdf5.SmartMinFileSize(100*hdf5.MB) // Conservative (only large files)

SmartAllowedModes

What it controls: Restrict which rebalancing modes can be auto-selected

Default: All modes allowed ("none", "lazy", "incremental")

Example:

// Only lazy and incremental (never "none")
hdf5.SmartAllowedModes("lazy", "incremental")

// Only incremental (force background rebalancing)
hdf5.SmartAllowedModes("incremental")

Use case: Force specific modes for organizational policy

SmartOnModeChange

What it controls: Callback when rebalancing mode changes

Optional: Can be nil

Example:

hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
    log.Printf("Rebalancing mode changed: %s (confidence: %.1f%%)\n",
        d.SelectedMode, d.Confidence*100)
    log.Printf("Reason: %s\n", d.Reason)
    log.Printf("Factors: %+v\n", d.Factors)

    // Send metrics to monitoring system
    metrics.RecordModeChange(d.SelectedMode, d.Confidence)
})

Best Practices:

Always set callback to understand auto-tuning decisions
Log decisions for debugging performance issues
Record metrics for monitoring system health

Troubleshooting

Slow Deletion Performance

Symptom: Deletions are 10-100x slower than expected

Likely Cause: Immediate rebalancing enabled (not offered by this library, but possible with custom B-tree implementation)

Solution:

Check if using default mode (no rebalancing) - should be fast

If using lazy/incremental, verify configuration:

// Check if threshold is too aggressive
hdf5.LazyThreshold(0.05)  // Not 0.01!

Benchmark with no rebalancing to establish baseline:

fw, _ := hdf5.CreateForWrite("test.h5", hdf5.CreateTruncate)
// No options = no rebalancing

Excessive Disk Space Usage

Symptom: File size much larger than expected after deletions

Likely Cause: No rebalancing enabled, B-tree has become very sparse

Solution: Enable lazy rebalancing:

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),
    ),
)

Verify B-tree is compact:

// After deletions, check file size
info, _ := os.Stat("data.h5")
log.Printf("File size: %d MB", info.Size()/1024/1024)

// Compare with C library output (should be similar with lazy rebalancing)

Rebalancing Pauses Too Long

Symptom: Lazy rebalancing pauses are unacceptable (>500ms)

Likely Cause: Batch size too large or threshold too high

Solution 1: Decrease batch size:

hdf5.WithLazyRebalancing(
    hdf5.LazyBatchSize(50),  // Smaller batches = shorter pauses
)

Solution 2: Switch to incremental rebalancing:

hdf5.WithLazyRebalancing(),
hdf5.WithIncrementalRebalancing(
    hdf5.IncrementalBudget(100*time.Millisecond),
    hdf5.IncrementalInterval(5*time.Second),
)
// Zero user-visible pause!

High CPU Usage

Symptom: Background rebalancing consuming too much CPU

Likely Cause: Incremental rebalancing budget or interval too aggressive

Solution: Decrease CPU impact:

hdf5.WithIncrementalRebalancing(
    hdf5.IncrementalBudget(50*time.Millisecond),  // Lower budget
    hdf5.IncrementalInterval(10*time.Second),     // Less frequent
)

Verify CPU usage:

# Linux: Monitor CPU usage
top -p $(pgrep -f your_program)

# Check if incremental rebalancing goroutine is the culprit
# (Should show <5% CPU in most cases)

Rebalancing Backlog Building Up

Symptom: Progress callback reports NodesRemaining keeps increasing

Likely Cause: Delete rate exceeds rebalancing throughput

Solution: Increase rebalancing aggressiveness:

hdf5.WithIncrementalRebalancing(
    hdf5.IncrementalBudget(200*time.Millisecond),  // More time per session
    hdf5.IncrementalInterval(2*time.Second),       // More frequent sessions
    hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
        if p.NodesRemaining > 1000 {
            log.Warn("Backlog: %d nodes (may need to tune Budget/Interval)", p.NodesRemaining)
        }
    }),
)

Alternative: Switch to lazy rebalancing (batch processing):

hdf5.WithLazyRebalancing(
    hdf5.LazyThreshold(0.05),  // Will batch-process periodically
)
// Occasional pauses, but guaranteed to complete

Smart Mode Not Switching

Symptom: Smart rebalancing stays in "none" mode despite deletions

Likely Cause: File size below SmartMinFileSize

Solution: Lower threshold:

hdf5.WithSmartRebalancing(
    hdf5.SmartMinFileSize(1*hdf5.MB),  // Lower from default 10MB
)

Verify detection:

hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
    log.Printf("Mode: %s, Confidence: %.1f%%, Reason: %s",
        d.SelectedMode, d.Confidence*100, d.Reason)
    // Check decision reasoning
})

Unclear Auto-Tuning Decisions

Symptom: Smart mode switches modes unexpectedly

Solution: Enable callback to understand decisions:

hdf5.WithSmartRebalancing(
    hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
        log.Printf("Mode Change: %s → reason: %s", d.SelectedMode, d.Reason)
        log.Printf("Confidence: %.1f%%", d.Confidence*100)
        log.Printf("Factors: %+v", d.Factors)  // Shows detection metrics
        log.Printf("Timestamp: %v", d.Timestamp)
    }),
)

Interpret factors:

delete_ratio: Higher → more likely to choose lazy/incremental
batch_size: Larger → more likely to choose lazy
operation_rate: Higher → more likely to choose incremental
file_size: Smaller → more likely to choose none

Advanced Topics

Custom Thresholds for Scientific Workloads

Scenario: Processing TB-scale simulation data with specific patterns

Custom Configuration:

// Simulation data characteristics:
// - 1TB files
// - 30% delete ratio (trimming failed runs)
// - Batch processing (can tolerate 1-2s pauses)
// - Need tight tree (disk space expensive)

fw, err := hdf5.CreateForWrite("simulation.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.03),          // Tight threshold (3%)
        hdf5.LazyMaxDelay(2*time.Minute),  // Frequent forced rebalancing
        hdf5.LazyBatchSize(200),           // Large batches (acceptable pauses)
    ),
)

Why this works:

Low threshold (3%) → B-tree stays very compact (saves disk space)
Short MaxDelay (2 min) → Regular rebalancing (prevents excessive sparsity)
Large BatchSize (200) → Amortizes cost of rebalancing large trees

Multi-TB File Optimization

Challenge: Incremental rebalancing may not keep up with delete rate in multi-TB files

Solution: Hybrid Approach

// Strategy: Lazy for bulk deletions, incremental for continuous optimization
fw, err := hdf5.CreateForWrite("huge.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),           // Batch deletions
        hdf5.LazyMaxDelay(10*time.Minute),  // Don't interrupt long sessions
        hdf5.LazyBatchSize(500),            // Large batches (multi-TB files)
    ),
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(500*time.Millisecond),  // Aggressive budget
        hdf5.IncrementalInterval(2*time.Second),       // Frequent sessions
        hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
            // Monitor closely for multi-TB files
            log.Printf("Rebalancing: %d/%d nodes (ETA: %v)\n",
                p.NodesRebalanced, p.NodesRemaining, p.EstimatedRemaining)

            // Alert if backlog exceeds threshold
            if p.NodesRemaining > 10000 {
                log.Error("Critical: Rebalancing backlog at %d nodes", p.NodesRemaining)
                // Consider pausing deletions temporarily
            }
        }),
    ),
)
defer fw.Close()

Key Points:

Lazy handles bulk deletions quickly (batching amortizes cost)
Incremental cleans up gradually in background
Close monitoring essential (callback tracks backlog)
May need to throttle delete rate if rebalancing can't keep up

Memory vs. Performance Trade-offs

Incremental Rebalancing Memory Cost:

Incremental rebalancing requires ~100MB for background processing:

Node buffers for rebalancing operations
Work queue for pending nodes
Progress tracking structures

Tuning Memory Usage:

// Low-memory configuration (reduces memory, slower rebalancing)
hdf5.WithIncrementalRebalancing(
    hdf5.IncrementalBudget(50*time.Millisecond),  // Smaller budget
    hdf5.IncrementalInterval(10*time.Second),     // Less frequent
    // Reduces concurrent node processing → lower memory
)

// High-memory configuration (faster rebalancing, more memory)
hdf5.WithIncrementalRebalancing(
    hdf5.IncrementalBudget(500*time.Millisecond),  // Large budget
    hdf5.IncrementalInterval(1*time.Second),       // Very frequent
    // More concurrent processing → higher memory (~200-300MB)
)

When Memory Matters:

Embedded systems → Use lazy rebalancing (minimal memory overhead)
HPC clusters → Use high-memory incremental (maximize throughput)
Cloud environments → Balance cost (memory pricing) vs. performance

Monitoring and Metrics

Production Recommendations:

Always monitor rebalancing in production:

type RebalancingMetrics struct {
    NodesRebalanced   int64
    TotalPauseTime    time.Duration
    LastRebalanceTime time.Time
    BacklogSize       int
}

var metrics RebalancingMetrics

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(),
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalProgressCallback(func(p hdf5.RebalancingProgress) {
            atomic.AddInt64(&metrics.NodesRebalanced, int64(p.NodesRebalanced))
            metrics.LastRebalanceTime = time.Now()
            atomic.StoreInt(&metrics.BacklogSize, p.NodesRemaining)

            // Export to Prometheus/StatsD/etc.
            prometheusGauge.Set("rebalancing.backlog", float64(p.NodesRemaining))
            prometheusCounter.Add("rebalancing.nodes_processed", float64(p.NodesRebalanced))
        }),
    ),
)

// Periodic health check
go func() {
    ticker := time.NewTicker(1 * time.Minute)
    defer ticker.Stop()

    for range ticker.C {
        backlog := atomic.LoadInt(&metrics.BacklogSize)
        if backlog > 5000 {
            log.Warn("Rebalancing backlog high: %d nodes", backlog)
        }

        timeSinceRebalance := time.Since(metrics.LastRebalanceTime)
        if timeSinceRebalance > 1*time.Minute {
            log.Info("No rebalancing activity for %v (may be idle)", timeSinceRebalance)
        }
    }
}()

Key Metrics to Track:

Backlog size: Should stay <1000 nodes typically
Rebalancing frequency: Should match Interval configuration
Pause time (lazy): Should be <500ms per batch
CPU usage (incremental): Should be <5% typically

Comparison with HDF5 C Library

C Library Behavior:

Default: No automatic rebalancing (same as this library's default)
Manual API: H5Ocompact() to trigger manual compaction
Trade-off: Users responsible for rebalancing

This Library's Advantage:

Lazy mode: Automatic batch rebalancing (10-100x faster than naive approach)
Incremental mode: Background rebalancing (zero pause, unique to this library)
Smart mode: Auto-tuning (not available in C library)

Compatibility:

Default mode (no rebalancing) → 100% compatible with C library
Lazy/incremental modes → Files readable by C library (standard HDF5 format)
File format unchanged → Interoperability guaranteed

Summary

Quick Reference Card

Workload	Recommended Mode	Key Parameters	Expected Overhead
Append-only	None (default)	None	0%
Batch deletes	Lazy	`Threshold(0.05)`, `BatchSize(100)`	~2%, 100-500ms pauses
Large files, moderate deletes	Incremental	`Budget(100ms)`, `Interval(5s)`	~4%, no pauses
Continuous heavy deletes	Incremental (aggressive)	`Budget(200ms)`, `Interval(2s)`	~5%, no pauses
Unknown/mixed	Smart	`AutoDetect(true)`, `AutoSwitch(true)`	~6%, varies

Best Practices

Start with default (no rebalancing) unless you know you need it
Use lazy for batch deletion workloads (10-100x faster than immediate)
Use incremental for large files where pauses are unacceptable
Use smart only if workload is truly unknown or highly variable
Always set progress callbacks for production systems
Monitor backlog size for incremental rebalancing
Benchmark before deploying to production

Performance Tips

Lazy rebalancing: Increase BatchSize for fewer, longer pauses
Incremental rebalancing: Adjust Budget/Interval to balance CPU vs. throughput
Smart rebalancing: Lower MinFileSize if small files need optimization
All modes: Monitor metrics to verify rebalancing is beneficial

When to Ask for Help

Contact library maintainers if:

Rebalancing backlog continuously grows (may indicate bug)
CPU usage >10% from rebalancing (unexpected overhead)
File size doesn't decrease after lazy rebalancing (possible issue)
Smart mode makes poor decisions repeatedly (detection may need tuning)

Last Updated: 2025-11-13 Related Guides: Rebalancing API Reference, FAQ

FilesExpand file tree

PERFORMANCE_TUNING.md

Latest commit

History

PERFORMANCE_TUNING.md

File metadata and controls

Performance Tuning Guide

Table of Contents

Introduction

What is B-tree Rebalancing?

Why It Matters for HDF5 Performance

When to Use Each Mode

Understanding B-tree Rebalancing

B-tree Basics

What Happens During Deletion

The Performance Dilemma

Rebalancing Modes

1. No Rebalancing (Default)

2. Lazy Rebalancing (Batch Processing)

3. Incremental Rebalancing (Background Processing)

4. Smart Rebalancing (Auto-Tuning, Optional)

Performance Characteristics

Benchmark Comparison

Operations Per Second

Workload-Specific Recommendations

Append-Only Workloads

Batch Deletion Workloads

Large Files with Moderate Deletes

Continuous Heavy-Delete Workloads

Mixed/Unknown Workloads

Configuration Guide

Lazy Rebalancing Parameters

LazyThreshold

LazyMaxDelay

LazyBatchSize

Incremental Rebalancing Parameters

IncrementalBudget

IncrementalInterval

IncrementalProgressCallback

Smart Rebalancing Parameters

SmartAutoDetect

SmartAutoSwitch

SmartMinFileSize

SmartAllowedModes

SmartOnModeChange

Troubleshooting

Slow Deletion Performance

Excessive Disk Space Usage

Rebalancing Pauses Too Long

High CPU Usage

Rebalancing Backlog Building Up

Smart Mode Not Switching

Unclear Auto-Tuning Decisions

Advanced Topics

Custom Thresholds for Scientific Workloads

Multi-TB File Optimization

Memory vs. Performance Trade-offs

Monitoring and Metrics

Comparison with HDF5 C Library

Summary

Quick Reference Card

Best Practices

Performance Tips

When to Ask for Help