Skip to content

CHANGELOG.md

Latest commit

 

History

History
180 lines (149 loc) · 8.77 KB

CHANGELOG.md

File metadata and controls

180 lines (149 loc) · 8.77 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

[0.4.0] - 2025-12-26

Changed

  • Normalized describe/1 to canonical schema format - Stage now returns a consistent schema:
    • name changed from string ("CrucibleBench.Stage") to atom (:bench)
    • Added __schema_version__: "1.0.0" marker for schema evolution
    • Added required, optional, types, defaults fields for option validation
    • Moved analysis metadata to __extensions__.bench namespace
    • Removed verbose mode (all metadata now in __extensions__)
  • Added crucible_framework dependency - Now requires crucible_framework ~> 0.5.0
  • Behaviour now mandatory - Uses @behaviour Crucible.Stage unconditionally with @impl true

Added

  • Schema version marker (__schema_version__: "1.0.0")
  • Conformance tests for stage contract (test/crucible_bench/conformance_test.exs)
  • Type specifications in schema for all options:
    • :tests - List of statistical test enums
    • :alpha - Float for significance level
    • :confidence_level - Float for confidence level
    • :bootstrap_iterations - Integer for bootstrap iterations
    • :data_source - Enum for data source selection

Documentation

  • README updated with Stage Contract section
  • README updated with Schema Introspection example
  • Installation version updated to 0.4.0

[0.3.2] - 2025-12-25

Added

  • Stage Multi-Group Support - Stage now supports multiple data layouts:
    • Two-group comparison: control and treatment keys for t-test, Mann-Whitney
    • Multi-group comparison: groups key for ANOVA, Kruskal-Wallis
    • Paired comparison: before and after keys for paired t-test, Wilcoxon
  • Stage Metrics Merging - Results automatically merged into context.metrics:
    • Summary stats: bench_n, bench_mean, bench_sd, bench_median
    • Test p-values: bench_ttest_p_value, bench_anova_p_value, etc.
  • Stage Type Specifications - Added @spec declarations for public functions
  • Stage Behaviour Compliance - Conditional @behaviour Crucible.Stage when available
  • Updated SVG - Professional bell curve design for crucible_bench.svg
  • Current State Documentation - Added docs/20251225/current_state.md with complete module reference
  • Gap Analysis - Added docs/20251225/gaps.md with improvement opportunities
  • Implementation Guide - Added docs/20251225/implementation_prompt.md for Stage enhancements

Fixed

  • Credo Strict Compliance - All 18 issues resolved:
    • Number formatting (20700 → 20_700)
    • Alphabetically sorted aliases across 7 files
    • Enum.map_join/3 instead of Enum.map/2 |> Enum.join/2
    • Reduced function arity using map parameters
    • Extracted helpers to reduce cyclomatic complexity and nesting depth
    • Replaced length/1 == 0 with Enum.empty?/1

Changed

  • Dependencies Updated:
    • crucible_ir upgraded from ~> 0.1.1 to ~> 0.2.0
    • eval_ex upgraded from ~> 0.1.2 to ~> 0.1.4
    • Added credo ~> 1.7 as dev/test dependency

Documentation

  • README updated with Advanced Stage Configuration section
  • README updated with Metrics Merging section

Testing

  • New test suite for two-group comparisons
  • New test suite for multi-group comparisons (ANOVA, Kruskal-Wallis)
  • New test suite for paired comparisons (paired t-test, Wilcoxon)
  • New test suite for metrics merging
  • New test suite for behaviour compliance

[0.3.1] - 2025-12-24

Added

  • EvalLog Schema: Inspect-ai compatible evaluation log structs
  • EvalEx Adapter: Convert EvalEx.Result into CrucibleBench.EvalLog
  • Metric Extraction: Helpers mirroring inspect-ai analysis extractors

Documentation

  • README updated with EvalLog adapter usage
  • Inspect-ai parity requirements updated with current status

[0.3.0] - 2025-11-26

Added

  • CrucibleIR Integration - Added dependency on crucible_ir ~> 0.1.1 for shared IR structures
  • CrucibleBench.Stage - New pipeline stage module for integration with crucible_framework
    • Implements stage behaviour for use in pipeline orchestration
    • Accepts CrucibleIR.Reliability.Stats configuration from experiment context
    • Extracts and analyzes data from pipeline context (:outputs or :metrics keys)
    • Returns comprehensive statistical analysis in :bench context key
    • Supports test selection, confidence intervals, and bootstrap analysis
    • Provides describe/1 function for stage introspection
  • IR Config Support - Main CrucibleBench.compare/3 function now accepts CrucibleIR.Reliability.Stats struct
    • Automatically converts IR configuration to internal options format
    • Maintains backwards compatibility with keyword list options
    • Maps IR test types (:ttest, :bootstrap, etc.) to CrucibleBench implementations

Changed

  • Version bumped to 0.3.0 (MINOR version due to new functionality)
  • Enhanced CrucibleBench module with multi-clause function definitions for config handling

Documentation

  • Complete API documentation for CrucibleBench.Stage module
  • Updated README with Stage usage examples
  • Enhanced main module documentation with IR config examples

Testing

  • New comprehensive test suite for CrucibleBench.Stage (18 tests)
  • Tests cover context processing, error handling, config conversion, and integration
  • Property-based validation of IR config acceptance

[0.2.1] - 2025-11-25

Added

  • Multiple Comparison Corrections - Controls Type I error rates when conducting multiple tests

    • Bonferroni correction (most conservative, controls FWER)
    • Holm step-down method (less conservative than Bonferroni, still controls FWER)
    • Benjamini-Hochberg FDR correction (controls false discovery rate, more powerful)
    • New module: CrucibleBench.Stats.MultipleComparisons
    • Integration with hyperparameter sweep experiments (automatic p-value adjustment)
    • Detailed correction results with original and adjusted p-values

  • Formal Normality Tests - Statistical tests for distribution assumptions

    • Shapiro-Wilk test (most powerful omnibus test, n = 3 to 5000)
    • Comprehensive normality assessment combining multiple approaches
    • Quick normality check using skewness/kurtosis thresholds
    • New module: CrucibleBench.Stats.NormalityTests

  • Variance Equality Tests - Validates homogeneity of variance assumptions

    • Levene's test (robust to non-normality, uses median-based deviations)
    • F-test for two groups (parametric, sensitive to normality)
    • Quick variance check using variance ratios
    • New module: CrucibleBench.Stats.VarianceTests

Changed

  • Hyperparameter sweep experiments now apply multiple comparison corrections by default (Holm method)
  • Export module updated to display adjusted p-values in pairwise comparison tables
  • Experiment results include correction method information

Documentation

  • Comprehensive enhancement design document in docs/20251125/enhancement_design.md
  • Complete API documentation for all new modules
  • Examples demonstrating each new feature
  • Best practices for multiple comparison handling

Testing

  • 3 new comprehensive test suites (multiple_comparisons_test, normality_tests_test, variance_tests_test)
  • Property-based testing for monotonicity and boundary conditions
  • Integration tests demonstrating end-to-end functionality

[0.2.0] - 2025-11-24

Added

  • Complete statistical testing framework with parametric (Student's, Welch's, paired t-tests, one-way ANOVA) and non-parametric (Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis) coverage using accurate distribution functions
  • Expanded effect size suite including Cohen's d, Hedges' g, Glass's delta, paired Cohen's d, eta-squared/omega-squared, and rank-biserial correlation with interpretation guidance
  • Confidence intervals (analytical and bootstrap) and power analysis (a priori and post-hoc for t-tests and ANOVA) with actionable recommendations
  • High-level analysis helpers for automatic test selection plus experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
  • Publication-ready exports to Markdown, LaTeX, and HTML with standardized result metadata

[0.1.0] - 2025-10-07

Added

  • Initial release
  • Comprehensive statistical testing framework for AI/ML research
  • Parametric tests (t-tests, ANOVA) and non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis)
  • Effect size measures (Cohen's d, Hedges' g, Glass's delta, eta-squared, omega-squared)
  • Power analysis with a priori and post-hoc calculations
  • Confidence intervals using bootstrap and analytical methods
  • High-level experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
  • Publication-ready export formats (Markdown, LaTeX, HTML)

Documentation

  • Comprehensive README with examples
  • API documentation for all statistical tests
  • Usage examples for common AI research scenarios
  • Best practices guide for statistical rigor in AI research