Add LSI benchmark system for GSL vs native Ruby comparison by cardmagic · Pull Request #74 · cardmagic/classifier

cardmagic · 2025-12-27T07:24:13Z

Summary

Add benchmark/lsi_benchmark.rb script for performance testing
Add rake benchmark task to run with current configuration
Add rake benchmark:compare task to compare GSL vs native Ruby
Update Rakefile to reference README.md (renamed from .markdown)
Add Benchmarks section to README.md

Usage

# Run benchmark with current configuration
rake benchmark

# Compare GSL vs native Ruby performance
rake benchmark:compare

# Force native Ruby mode
NATIVE_VECTOR=true rake benchmark

Benchmark Results (Native Ruby)

============================================================
LSI Benchmark: 5 documents (Native Ruby)
============================================================

Operation                  User     System      Total
----------------------------------------------------
add_items                0.0004     0.0001     0.0004
build_index              0.0043     0.0010     0.0053
classify                 0.0081     0.0016     0.0098
search                   0.0032     0.0003     0.0035
find_related             0.0019     0.0002     0.0021
----------------------------------------------------
TOTAL                                          0.0210

============================================================
LSI Benchmark: 20 documents (Native Ruby)
============================================================

Operation                  User     System      Total
----------------------------------------------------
add_items                0.0001     0.0000     0.0001
build_index              0.5458     0.0020     0.5477
classify                 0.0139     0.0001     0.0140
search                   0.0133     0.0001     0.0134
find_related             0.0093     0.0001     0.0093
----------------------------------------------------
TOTAL                                          0.5846

Benchmark Operations

add_items - Adding documents to the index
build_index - Building the SVD index (dominates total time)
classify - Classification (100 iterations)
search - Search queries (100 iterations)
find_related - Finding related documents (100 iterations)

Test Plan

rake benchmark runs successfully
Handles SVD failures gracefully with helpful error messages
Reports GSL availability status
All 85 tests pass

Fixes #75

- Add SUPPRESS_GSL_WARNING=true env var to silence the notice - Update warning message with clear install instructions - Modernize README with platform-specific GSL installation steps - Rename README.markdown to README.md Closes #41

- Add benchmark/lsi_benchmark.rb with performance tests - Add rake benchmark and rake benchmark:compare tasks - Test add_items, build_index, classify, search, find_related - Handle native Ruby SVD numerical instability gracefully - Update Rakefile to reference README.md (renamed from .markdown) Note: Native Ruby SVD has numerical limits with larger document sets. GSL is recommended for production use with large collections.

Copilot

Pull request overview

This PR adds a comprehensive benchmarking system to compare LSI performance between GSL and native Ruby implementations. The benchmark script measures key operations (adding items, building index, classification, search, and finding related documents) across different document set sizes, with graceful handling of GSL availability and native Ruby SVD numerical stability limitations.

Key Changes:

New benchmarking infrastructure with single-run and comparison modes
Enhanced GSL availability warnings with suppression option
Updated documentation for GSL installation and usage

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
benchmark/lsi_benchmark.rb	Comprehensive benchmark script with subprocess-based comparison system for GSL vs native Ruby performance testing
Rakefile	Added `benchmark` and `benchmark:compare` rake tasks, updated README reference
lib/classifier/lsi.rb	Enhanced GSL warning message with suppression option
README.md	Expanded GSL installation instructions and documented warning suppression
.rubocop.yml	Excluded benchmark directory from RuboCop checks

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-27T07:56:20Z

+    puts "\n%-20s %10s %10s %10s" % ['Operation', 'User', 'System', 'Total']
+    puts '-' * 52
+    results.each do |name, bm|
+      puts "%-20s %10.4f %10.4f %10.4f" % [name, bm.utime, bm.stime, bm.total]
+    end
+
+    total = results.values.sum(&:total)
+    puts '-' * 52
+    puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"\\n%-20s %10s %10s %10s\", 'Operation', 'User', 'System', 'Total')

Suggested change

puts "\n%-20s %10s %10s %10s" % ['Operation', 'User', 'System', 'Total']

puts '-' * 52

results.each do |name, bm|

puts "%-20s %10.4f %10.4f %10.4f" % [name, bm.utime, bm.stime, bm.total]

end

total = results.values.sum(&:total)

puts '-' * 52

puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]

puts format("\n%-20s %10s %10s %10s", 'Operation', 'User', 'System', 'Total')

puts '-' * 52

results.each do |name, bm|

puts format("%-20s %10.4f %10.4f %10.4f", name, bm.utime, bm.stime, bm.total)

end

total = results.values.sum(&:total)

puts '-' * 52

puts format("%-20s %10s %10s %10.4f", 'TOTAL', '', '', total)

Copilot · 2025-12-27T07:56:21Z

+    puts "\n%-20s %10s %10s %10s" % ['Operation', 'User', 'System', 'Total']
+    puts '-' * 52
+    results.each do |name, bm|
+      puts "%-20s %10.4f %10.4f %10.4f" % [name, bm.utime, bm.stime, bm.total]
+    end
+
+    total = results.values.sum(&:total)
+    puts '-' * 52
+    puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"%-20s %10.4f %10.4f %10.4f\", name, bm.utime, bm.stime, bm.total)

Suggested change

puts "\n%-20s %10s %10s %10s" % ['Operation', 'User', 'System', 'Total']

puts '-' * 52

results.each do |name, bm|

puts "%-20s %10.4f %10.4f %10.4f" % [name, bm.utime, bm.stime, bm.total]

end

total = results.values.sum(&:total)

puts '-' * 52

puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]

puts format("\n%-20s %10s %10s %10s", 'Operation', 'User', 'System', 'Total')

puts '-' * 52

results.each do |name, bm|

puts format("%-20s %10.4f %10.4f %10.4f", name, bm.utime, bm.stime, bm.total)

end

total = results.values.sum(&:total)

puts '-' * 52

puts format("%-20s %10s %10s %10.4f", 'TOTAL', '', '', total)

Copilot · 2025-12-27T07:56:21Z

+
+    total = results.values.sum(&:total)
+    puts '-' * 52
+    puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"%-20s %10s %10s %10.4f\", 'TOTAL', '', '', total)

Suggested change

puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]

puts format("%-20s %10s %10s %10.4f", 'TOTAL', '', '', total)

Copilot · 2025-12-27T07:56:21Z

+    next unless native || gsl
+
+    puts "\n--- #{size} Documents ---"
+    puts "%-20s %12s %12s %12s" % ['Operation', 'Native', 'GSL', 'Speedup']


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"%-20s %12s %12s %12s\", 'Operation', 'Native', 'GSL', 'Speedup')

Copilot · 2025-12-27T07:56:21Z

+                  'N/A'
+                end
+
+      puts "%-20s %12s %12s %12s" % [op, native_str, gsl_str, speedup]


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"%-20s %12s %12s %12s\", op, native_str, gsl_str, speedup)

Copilot · 2025-12-27T07:56:21Z

+      gsl_total = operations.sum { |op| gsl[op] }
+      speedup = gsl_total > 0 ? native_total / gsl_total : 0
+      puts '-' * 58
+      puts "%-20s %12.4f %12.4f %11.1fx" % ['TOTAL', native_total, gsl_total, speedup]


Using % for string formatting with array arguments can be hard to read and maintain. Consider using Ruby's format method instead: format(\"%-20s %12.4f %12.4f %11.1fx\", 'TOTAL', native_total, gsl_total, speedup)

Copilot · 2025-12-27T07:56:22Z

+  puts "\n>>> Running Native Ruby benchmarks..."
+  sizes.each do |size|
+    docs = doc_sets[size]
+    docs_literal = docs.map { |d, c| "[#{d.inspect}, #{c.inspect}]" }.join(', ')


Constructing code strings by interpolating user-controlled data can lead to code injection vulnerabilities. Consider using a safer approach like passing data through stdin or a temporary file, or using JSON serialization instead of building Ruby literals.

Copilot · 2025-12-27T07:56:22Z

+  puts "\n>>> Running GSL benchmarks..."
+  sizes.each do |size|
+    docs = doc_sets[size]
+    docs_literal = docs.map { |d, c| "[#{d.inspect}, #{c.inspect}]" }.join(', ')


Constructing code strings by interpolating user-controlled data can lead to code injection vulnerabilities. Consider using a safer approach like passing data through stdin or a temporary file, or using JSON serialization instead of building Ruby literals.

- Catch ExceptionForMatrix::ErrDimensionMismatch in addition to Math::DomainError - Use diverse document vocabulary to reduce SVD instability - Reduce default benchmark sizes (5, 10, 15, 20) for native Ruby stability - Update test queries to match new document style

- Replace % string formatting with format() for readability - Fix code injection vulnerability by using JSON via stdin - Use Bundler.with_unbundled_env so subprocess can access GSL gem - Fix GSL::Vector missing count method (use to_a.count instead) - Update README with Ruby 3.4+ GSL installation instructions - Add real benchmark results showing 4x-116x speedup with GSL The benchmark comparison now works correctly, showing GSL provides significant speedups especially for build_index (SVD computation).

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-27T09:14:58Z

+    puts format("%-20s %10s %10s %10.4f", 'TOTAL', '', '', total)
+
+    { results: results, gsl: Classifier::LSI.gsl_available, success: true }
+  rescue Math::DomainError, ExceptionForMatrix::ErrDimensionMismatch => e


The exception class ExceptionForMatrix::ErrDimensionMismatch appears to be incorrect. Ruby's Matrix library uses Matrix::ErrDimensionMismatch (without 'Exception' prefix). This rescue clause will not catch dimension mismatch errors as intended.

Copilot · 2025-12-27T09:14:58Z

+
+  # Use Bundler.with_unbundled_env if available to access system gems (like GSL)
+  popen_block = lambda do
+    IO.popen([env, RbConfig.ruby, '-I', lib_path, '-W0', '-e', <<~'RUBY'], 'r+')


The -W0 flag suppresses all warnings, which could hide important runtime issues during benchmarking. Consider using -W1 (default warnings) or removing this flag to ensure warnings about potential problems are visible.

Copilot · 2025-12-27T09:14:58Z

+    result = run_subprocess(docs_json, NATIVE_VECTOR: 'true', SUPPRESS_GSL_WARNING: 'true')
+
+    if result[:error]
+      puts "  #{size} docs: FAILED (#{result[:error].class.name.split('::').last})"


The error message formatting logic result[:error].class.name.split('::').last is duplicated. Extract this into a helper method or variable to improve maintainability.

Copilot · 2025-12-27T09:14:58Z

+    result = run_subprocess(docs_json, SUPPRESS_GSL_WARNING: 'true')
+
+    if result[:error]
+      puts "  #{size} docs: FAILED (#{result[:error].class.name.split('::').last})"


The error message formatting logic result[:error].class.name.split('::').last is duplicated. Extract this into a helper method or variable to improve maintainability.

cardmagic mentioned this pull request Dec 27, 2025

Add LSI benchmark system for GSL vs native Ruby comparison #75

Closed

cardmagic added 3 commits December 26, 2025 23:51

Improve GSL warning with suppression option

0d74869

- Add SUPPRESS_GSL_WARNING=true env var to silence the notice - Update warning message with clear install instructions - Modernize README with platform-specific GSL installation steps - Rename README.markdown to README.md Closes #41

Exclude benchmark directory from RuboCop

c1b1b89

cardmagic force-pushed the feature/lsi-benchmark branch from 47ff4a4 to c1b1b89 Compare December 27, 2025 07:52

cardmagic requested a review from Copilot December 27, 2025 07:55

cardmagic self-assigned this Dec 27, 2025

Copilot AI reviewed Dec 27, 2025

View reviewed changes

cardmagic added 3 commits December 27, 2025 00:01

Add Benchmarks section to README with sample results

ff6ba7f

cardmagic requested a review from Copilot December 27, 2025 09:14

Copilot AI reviewed Dec 27, 2025

View reviewed changes

cardmagic merged commit ba36830 into master Dec 27, 2025
5 checks passed

cardmagic deleted the feature/lsi-benchmark branch December 27, 2025 09:16

	puts "%-20s %10s %10s %10.4f" % ['TOTAL', '', '', total]
	puts format("%-20s %10s %10s %10.4f", 'TOTAL', '', '', total)

Conversation

cardmagic commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Benchmark Results (Native Ruby)

Benchmark Operations

Test Plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cardmagic commented Dec 27, 2025 •

edited

Loading