feat(differ): add comprehensive benchmark suite for differ package (#23)

erraggy · claude · web-flow · commit 7295c5c171d7 · 2025-11-20T19:51:42.000-08:00
* feat(differ): add comprehensive benchmark suite for differ package Add benchmark tests for differ package with 10 benchmarks covering all major operations and performance patterns. Update documentation to reflect new benchmarks and performance metrics. Changes: - Add differ/differ_bench_test.go with 10 comprehensive benchmarks - Convenience functions (Diff, DiffParsed) - Struct-based API (Differ.Diff, Differ.DiffParsed) - Different modes (Simple, Breaking) - Configuration options (WithInfo, WithoutInfo) - Edge cases (IdenticalSpecs) - Parse-once pattern efficiency - Update benchmarks.md with differ performance metrics - Add Differ Performance section with detailed results - Update benchmark count from 60+ to 70+ - Update version from v1.7.0 to v1.9.1 in overview - Add differ to benchmark suite listing - Add differ to running benchmarks examples - Update best practices to include differ - Update README.md with differ benchmarks - Update benchmark count from 60+ to 70+ - Update version from v1.7.0 to v1.9.1 - Add 58x faster DiffParsed performance metric - Add Diff operation to performance table - Update Makefile with differ support - Add bench-differ target for running differ benchmarks - Add ./differ to all benchmark targets (bench, bench-save, bench-baseline, bench-cpu, bench-mem, bench-profile) - Add bench-differ to .PHONY targets Performance Results: - DiffParsed is 58x faster than Diff (7.8μs vs 457μs) - Identical specs comparison is very fast (2.8μs) - fast path working - Breaking mode vs Simple mode has negligible difference (~0.15μs) This update completes the benchmark coverage for all five core packages (parser, validator, converter, joiner, differ). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(differ): update benchmarks to use Go 1.24+ for b.Loop() pattern Address claude-review feedback by updating all differ benchmarks to use the modern Go 1.24+ for b.Loop() pattern for consistency with other packages. Also add comprehensive benchmark testing guidance to CLAUDE.md. Changes: - Update all 10 benchmarks to use for b.Loop() instead of the old for i := 0; i < b.N; i++ pattern - Remove redundant b.ReportAllocs() calls (handled by b.Loop()) - Remove unnecessary b.ResetTimer() calls for trivial setup - Add package-level comment about b.Fatalf usage pattern - Add new "Benchmark Test Requirements" section to CLAUDE.md - Document the for b.Loop() pattern as the required standard - Provide correct and incorrect examples - Explain why consistency matters - Add guidance on when to include setup outside the loop This ensures consistency across all five benchmark files: - parser/parser_bench_test.go: Uses for b.Loop() ✅ - validator/validator_bench_test.go: Uses for b.Loop() ✅ - converter/converter_bench_test.go: Uses for b.Loop() ✅ - joiner/joiner_bench_test.go: Uses for b.Loop() ✅ - differ/differ_bench_test.go: Now uses for b.Loop() ✅ Benchmark results remain consistent with previous run, confirming the pattern change has no negative impact on performance measurement. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -468,6 +468,74 @@ func TestValidatorValidate(t *testing.T) { ... }
 func TestJoinerJoin(t *testing.T) { ... }
 func TestConverterConvert(t *testing.T) { ... }
 ```
+
+### Benchmark Test Requirements
+
+**CRITICAL: Use the Go 1.24+ `for b.Loop()` pattern for all benchmarks.**
+
+When writing benchmark tests, you MUST follow these requirements:
+
+1. **Use `for b.Loop()` Pattern** (Go 1.24+)
+   - **ALWAYS** use `for b.Loop()` instead of `for i := 0; i < b.N; i++`
+   - **DO NOT** call `b.ReportAllocs()` manually (handled automatically by `b.Loop()`)
+   - This is the modern, consistent pattern used across all packages
+
+   **Correct Pattern**:
+   ```go
+   func BenchmarkDiffParsed(b *testing.B) {
+       // Setup (parsing, creating instances, etc.)
+       source, _ := parser.Parse("file.yaml", false, true)
+       target, _ := parser.Parse("file2.yaml", false, true)
+
+       for b.Loop() {  // ✅ Correct - modern Go 1.24+ pattern
+           _, err := DiffParsed(*source, *target)
+           if err != nil {
+               b.Fatal(err)
+           }
+       }
+   }
+   ```
+
+   **Incorrect Pattern**:
+   ```go
+   func BenchmarkDiffParsed(b *testing.B) {
+       source, _ := parser.Parse("file.yaml", false, true)
+       target, _ := parser.Parse("file2.yaml", false, true)
+
+       b.ReportAllocs()  // ❌ Wrong - redundant with b.Loop()
+       for i := 0; i < b.N; i++ {  // ❌ Wrong - old pattern
+           _, err := DiffParsed(*source, *target)
+           if err != nil {
+               b.Fatal(err)
+           }
+       }
+   }
+   ```
+
+2. **No Manual Timer Resets**
+   - **DO NOT** call `b.ResetTimer()` for trivial setup (e.g., creating a Differ instance)
+   - `b.Loop()` handles timing automatically
+   - Only include expensive setup (like parsing) outside the loop
+
+3. **Package-Level Comment**
+   - Add this comment at the top of each benchmark file:
+   ```go
+   // Note on b.Fatalf usage in benchmarks:
+   // Using b.Fatalf for errors in benchmark setup or execution is an acceptable pattern.
+   // These operations (parse, diff, validate, etc.) should never fail with valid test fixtures.
+   // If they do fail, it indicates a bug that should halt the benchmark immediately.
+   ```
+
+4. **Consistency Across Packages**
+   - All benchmark files (`parser`, `validator`, `converter`, `joiner`, `differ`) use the same pattern
+   - This ensures maintainability and consistency with Go evolution
+
+**Why This Matters**:
+- **Consistency**: All existing benchmarks use `for b.Loop()`
+- **Best Practice**: Modern Go 1.24+ approach
+- **Automatic Handling**: `b.Loop()` manages allocations reporting and timing
+- **Future Compatibility**: Aligns with Go's benchmark evolution
+
 ## Submitting changes
 
 **Before Submitting Code:**
diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-.PHONY: build test lint clean install tidy check help bench bench-parser bench-validator bench-converter bench-joiner bench-save bench-compare bench-baseline bench-clean release-test release-clean
+.PHONY: build test lint clean install tidy check help bench bench-parser bench-validator bench-converter bench-joiner bench-differ bench-save bench-compare bench-baseline bench-clean release-test release-clean
 
 # Build variables
 BINARY_NAME=oastools
@@ -83,7 +83,7 @@ check: tidy fmt lint test
 ## bench: Run all benchmarks
 bench:
 	@echo "Running all benchmarks ($(BENCH_TIME) per benchmark)..."
-	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner
+	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner ./differ
 
 ## bench-parser: Run parser benchmarks only
 bench-parser:
@@ -105,19 +105,24 @@ bench-joiner:
 	@echo "Running joiner benchmarks..."
 	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./joiner
 
+## bench-differ: Run differ benchmarks only
+bench-differ:
+	@echo "Running differ benchmarks..."
+	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./differ
+
 ## bench-save: Run all benchmarks and save to timestamped file
 bench-save:
 	@echo "Running benchmarks and saving results..."
 	@TIMESTAMP=$$(date +%Y%m%d-%H%M%S); \
 	OUTPUT_FILE="benchmark-$${TIMESTAMP}.txt"; \
-	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner 2>&1 | tee "$${OUTPUT_FILE}"; \
+	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner ./differ 2>&1 | tee "$${OUTPUT_FILE}"; \
 	echo ""; \
 	echo "Benchmark results saved to: $${OUTPUT_FILE}"
 
 ## bench-baseline: Run benchmarks and update baseline file
 bench-baseline:
 	@echo "Running benchmarks and updating baseline..."
-	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner 2>&1 | tee benchmark-baseline.txt
+	@go test -bench=. -benchmem -benchtime=$(BENCH_TIME) ./parser ./validator ./converter ./joiner ./differ 2>&1 | tee benchmark-baseline.txt
 	@echo ""
 	@echo "Baseline updated: benchmark-baseline.txt"
 
@@ -145,7 +150,7 @@ bench-cpu:
 	@echo "Running benchmarks with CPU profiling..."
 	@TIMESTAMP=$$(date +%Y%m%d-%H%M%S); \
 	PROFILE_FILE="cpu-profile-$${TIMESTAMP}.prof"; \
-	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -cpuprofile="$${PROFILE_FILE}" ./parser ./validator ./converter ./joiner; \
+	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -cpuprofile="$${PROFILE_FILE}" ./parser ./validator ./converter ./joiner ./differ; \
 	echo ""; \
 	echo "CPU profile saved to: $${PROFILE_FILE}"; \
 	echo "Analyze with: go tool pprof $${PROFILE_FILE}"
@@ -155,7 +160,7 @@ bench-mem:
 	@echo "Running benchmarks with memory profiling..."
 	@TIMESTAMP=$$(date +%Y%m%d-%H%M%S); \
 	PROFILE_FILE="mem-profile-$${TIMESTAMP}.prof"; \
-	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -memprofile="$${PROFILE_FILE}" ./parser ./validator ./converter ./joiner; \
+	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -memprofile="$${PROFILE_FILE}" ./parser ./validator ./converter ./joiner ./differ; \
 	echo ""; \
 	echo "Memory profile saved to: $${PROFILE_FILE}"; \
 	echo "Analyze with: go tool pprof $${PROFILE_FILE}"
@@ -166,7 +171,7 @@ bench-profile:
 	@TIMESTAMP=$$(date +%Y%m%d-%H%M%S); \
 	CPU_PROFILE="cpu-profile-$${TIMESTAMP}.prof"; \
 	MEM_PROFILE="mem-profile-$${TIMESTAMP}.prof"; \
-	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -cpuprofile="$${CPU_PROFILE}" -memprofile="$${MEM_PROFILE}" ./parser ./validator ./converter ./joiner; \
+	go test -bench=. -benchmem -benchtime=$(BENCH_TIME) -cpuprofile="$${CPU_PROFILE}" -memprofile="$${MEM_PROFILE}" ./parser ./validator ./converter ./joiner ./differ; \
 	echo ""; \
 	echo "CPU profile saved to: $${CPU_PROFILE}"; \
 	echo "Memory profile saved to: $${MEM_PROFILE}"; \
diff --git a/README.md b/README.md
@@ -166,14 +166,15 @@ See [pkg.go.dev](https://pkg.go.dev/github.com/erraggy/oastools) for complete AP
 
 ## Benchmarks
 
-The library includes comprehensive performance benchmarking (60+ benchmarks across all packages). As of v1.7.0, significant optimizations have been implemented:
+The library includes comprehensive performance benchmarking (70+ benchmarks across all packages). As of v1.9.1, significant optimizations have been implemented:
 
 **Performance Highlights**:
 - **25-32% faster** JSON marshaling (v1.7.0 optimization)
 - **29-37% fewer** memory allocations
 - **30x faster** validation with `ValidateParsed` vs `Validate` (parse once, validate many)
 - **9x faster** conversion with `ConvertParsed` vs `Convert` (parse once, convert many)
 - **154x faster** joining with `JoinParsed` vs `Join` (parse once, join many)
+- **58x faster** diffing with `DiffParsed` vs `Diff` (parse once, diff many)
 
 **Document Processing Performance** (Apple M4, Go 1.24):
 
@@ -183,6 +184,7 @@ The library includes comprehensive performance benchmarking (60+ benchmarks acro
 | Validate         | 143 μs            | 1,160 μs            | 14,635 μs           |
 | Convert (OAS2→3) | 153 μs            | 1,314 μs            | -                   |
 | Join (2 docs)    | 115 μs            | -                   | -                   |
+| Diff (2 docs)    | 457 μs            | -                   | -                   |
 
 For detailed performance analysis, methodology, and optimization strategies, see [benchmarks.md](benchmarks.md).
 
diff --git a/benchmarks.md b/benchmarks.md
@@ -4,7 +4,7 @@ This document provides detailed performance analysis and benchmark results for t
 
 ## Overview
 
-As of v1.7.0, oastools includes comprehensive performance benchmarking infrastructure covering all major operations across the parser, validator, converter, and joiner packages. The library has undergone targeted optimizations to achieve significant performance improvements while maintaining correctness and code quality.
+As of v1.9.1, oastools includes comprehensive performance benchmarking infrastructure covering all major operations across the parser, validator, converter, joiner, and differ packages. The library has undergone targeted optimizations to achieve significant performance improvements while maintaining correctness and code quality.
 
 **Platform**: Apple M4, darwin/arm64, Go 1.24
 
@@ -35,7 +35,7 @@ The v1.7.0 release includes a major optimization to JSON marshaling that elimina
 
 ## Benchmark Suite
 
-The benchmark suite includes **60+ benchmarks** across four main packages:
+The benchmark suite includes **70+ benchmarks** across five main packages:
 
 ### Parser Package (32 benchmarks)
 
@@ -84,6 +84,16 @@ The benchmark suite includes **60+ benchmarks** across four main packages:
 - Array merge strategies
 - Tag deduplication
 
+### Differ Package (10 benchmarks)
+
+**Diffing Operations**:
+- Diff (parse + diff) vs DiffParsed (pre-parsed)
+- Simple mode vs Breaking mode
+- Convenience functions (Diff, DiffParsed) vs struct-based API
+- Configuration options (IncludeInfo)
+- Edge cases (identical specifications)
+- Parse-once pattern efficiency
+
 ## Current Performance Metrics
 
 ### Parser Performance
@@ -178,16 +188,43 @@ The benchmark suite includes **60+ benchmarks** across four main packages:
 
 **Observation**: JoinParsed is **154x faster** than Join for 2 small documents (747ns vs 115μs) because it skips parsing. The actual joining operation has minimal overhead.
 
+### Differ Performance
+
+**Diffing** (parse + diff):
+
+| Mode     | Time (μs) | Memory (KB) | Allocations |
+|----------|-----------|-------------|-------------|
+| Simple   | 457       | 609         | 7,156       |
+| Breaking | 458       | 611         | 7,157       |
+
+**Diffing pre-parsed documents** (DiffParsed):
+
+| Mode     | Time (μs) | Memory (KB) | Allocations |
+|----------|-----------|-------------|-------------|
+| Simple   | 7.82      | 6.89        | 138         |
+| Breaking | 7.97      | 6.96        | 137         |
+
+**Special cases**:
+
+| Case               | Time (μs) | Memory (KB) | Allocations |
+|--------------------|-----------|-------------|-------------|
+| Identical specs    | 2.82      | 2.08        | 79          |
+| With info changes  | 7.97      | 6.96        | 137         |
+| Without info       | 8.10      | 8.11        | 138         |
+
+**Observation**: DiffParsed is **58x faster** than Diff (7.8μs vs 457μs) because it skips parsing. The differ includes a fast path for identical specifications that runs in ~2.8μs. Breaking mode vs Simple mode has negligible performance difference (~0.15μs), making breaking change detection essentially free.
+
 ## Performance Best Practices
 
 ### For Library Users
 
-1. **Reuse parser/validator/converter/joiner instances** when processing multiple files with the same configuration
-2. **Use ParseOnce workflows**: For operations on the same document (validate, convert, join), parse once and pass the `ParseResult` to subsequent operations:
+1. **Reuse parser/validator/converter/joiner/differ instances** when processing multiple files with the same configuration
+2. **Use ParseOnce workflows**: For operations on the same document (validate, convert, join, diff), parse once and pass the `ParseResult` to subsequent operations:
    ```go
    result, _ := parser.Parse("spec.yaml", false, true)
    validator.ValidateParsed(result, true, false)   // 30x faster than Validate
    converter.ConvertParsed(result, "3.0.3")        // 9x faster than Convert
+   differ.DiffParsed(result1, result2)             // 58x faster than Diff
    ```
 3. **Disable reference resolution** (`ResolveRefs=false`) when not needed
 4. **Disable validation** during parsing (`ValidateStructure=false`) if you'll validate separately
@@ -220,12 +257,14 @@ make bench-parser
 make bench-validator
 make bench-converter
 make bench-joiner
+make bench-differ
 
 # Or individually
 go test -bench=. -benchmem ./parser
 go test -bench=. -benchmem ./validator
 go test -bench=. -benchmem ./converter
 go test -bench=. -benchmem ./joiner
+go test -bench=. -benchmem ./differ
 
 # Save baseline for comparison
 go test -bench=. -benchmem ./... > benchmark-baseline.txt
diff --git a/differ/bench-results.txt b/differ/bench-results.txt
@@ -0,0 +1,56 @@
+goos: darwin
+goarch: arm64
+pkg: github.com/erraggy/oastools/differ
+cpu: Apple M4
+BenchmarkDiffConvenience-10          	    2523	    436911 ns/op	  607112 B/op	    7156 allocs/op
+BenchmarkDiffConvenience-10          	    2776	    446545 ns/op	  607657 B/op	    7156 allocs/op
+BenchmarkDiffConvenience-10          	    2698	    439833 ns/op	  607993 B/op	    7156 allocs/op
+BenchmarkDiffConvenience-10          	    2757	    508917 ns/op	  608323 B/op	    7156 allocs/op
+BenchmarkDiffConvenience-10          	    2715	    451005 ns/op	  608371 B/op	    7156 allocs/op
+BenchmarkDiffParsedConvenience-10    	  156670	      7678 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDiffParsedConvenience-10    	  157692	      7662 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDiffParsedConvenience-10    	  156468	      7670 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDiffParsedConvenience-10    	  156780	      7728 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDiffParsedConvenience-10    	  155041	      7691 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferDiff-10               	    2600	    452266 ns/op	  609324 B/op	    7156 allocs/op
+BenchmarkDifferDiff-10               	    2640	    455860 ns/op	  610072 B/op	    7157 allocs/op
+BenchmarkDifferDiff-10               	    2637	    457806 ns/op	  610558 B/op	    7157 allocs/op
+BenchmarkDifferDiff-10               	    2600	    460511 ns/op	  610688 B/op	    7157 allocs/op
+BenchmarkDifferDiff-10               	    2638	    458362 ns/op	  610756 B/op	    7157 allocs/op
+BenchmarkDifferDiffParsed-10         	  153064	      7812 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferDiffParsed-10         	  153217	      7847 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferDiffParsed-10         	  154752	      7768 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferDiffParsed-10         	  153975	      7823 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferDiffParsed-10         	  153349	      7807 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferSimpleMode-10         	  149916	      7820 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferSimpleMode-10         	  153992	      7799 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferSimpleMode-10         	  154177	      7809 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferSimpleMode-10         	  153376	      7836 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferSimpleMode-10         	  151728	      7872 ns/op	    6891 B/op	     138 allocs/op
+BenchmarkDifferBreakingMode-10       	  150068	      7931 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferBreakingMode-10       	  150386	      8020 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferBreakingMode-10       	  149853	      7989 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferBreakingMode-10       	  151822	      7969 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferBreakingMode-10       	  151111	      7944 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithInfo-10           	  150004	      7974 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithInfo-10           	  148920	      7946 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithInfo-10           	  150672	      7988 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithInfo-10           	  150805	      7965 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithInfo-10           	  151230	      7977 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkDifferWithoutInfo-10        	  148821	      8086 ns/op	    8108 B/op	     138 allocs/op
+BenchmarkDifferWithoutInfo-10        	  148964	      8108 ns/op	    8108 B/op	     138 allocs/op
+BenchmarkDifferWithoutInfo-10        	  147596	      8111 ns/op	    8108 B/op	     138 allocs/op
+BenchmarkDifferWithoutInfo-10        	  148171	      8091 ns/op	    8108 B/op	     138 allocs/op
+BenchmarkDifferWithoutInfo-10        	  147568	      8129 ns/op	    8108 B/op	     138 allocs/op
+BenchmarkDifferIdenticalSpecs-10     	  425127	      2829 ns/op	    2081 B/op	      79 allocs/op
+BenchmarkDifferIdenticalSpecs-10     	  418862	      2822 ns/op	    2081 B/op	      79 allocs/op
+BenchmarkDifferIdenticalSpecs-10     	  429141	      2829 ns/op	    2081 B/op	      79 allocs/op
+BenchmarkDifferIdenticalSpecs-10     	  426162	      2821 ns/op	    2081 B/op	      79 allocs/op
+BenchmarkDifferIdenticalSpecs-10     	  421341	      2827 ns/op	    2081 B/op	      79 allocs/op
+BenchmarkParseOnceDiffMany-10        	  151274	      7975 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkParseOnceDiffMany-10        	  150036	      7994 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkParseOnceDiffMany-10        	  149776	      7988 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkParseOnceDiffMany-10        	  149809	      7984 ns/op	    6955 B/op	     137 allocs/op
+BenchmarkParseOnceDiffMany-10        	  149707	      8006 ns/op	    6955 B/op	     137 allocs/op
+PASS
+ok  	github.com/erraggy/oastools/differ	64.034s
diff --git a/differ/differ_bench_test.go b/differ/differ_bench_test.go
