Skip to content

LILO hash computation and stamping for data freshness#4992

Open
ItsMrLin wants to merge 1 commit intofacebook:mainfrom
ItsMrLin:export-D95284287
Open

LILO hash computation and stamping for data freshness#4992
ItsMrLin wants to merge 1 commit intofacebook:mainfrom
ItsMrLin:export-D95284287

Conversation

@ItsMrLin
Copy link
Contributor

@ItsMrLin ItsMrLin commented Mar 6, 2026

Summary:
Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:

  • Add LILO_INPUT_HASH key to Keys enum in constants.py
  • Create ax/utils/common/hash_utils.py with compute_lilo_input_hash
    (standalone hash function) and get_current_lilo_hash (convenience helper
    that looks up the pairwise DerivedMetric on an experiment, extracts
    input_metric_names, and computes the hash — returns None if no pairwise
    metric is registered)
  • Stamp hash in LILOPairwiseMetric._compute_derived_values after producing labels
  • Add tests for hash determinism, sensitivity to data/message changes, stamping,
    and get_current_lilo_hash helper

Reviewed By: saitcakmak

Differential Revision: D95284287

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 6, 2026
@meta-codesync
Copy link

meta-codesync bot commented Mar 6, 2026

@ItsMrLin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D95284287.

ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 6, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 6, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
@codecov-commenter
Copy link

codecov-commenter commented Mar 6, 2026

Codecov Report

❌ Patch coverage is 3.70370% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.80%. Comparing base (3e8e4e7) to head (942b0f9).

Files with missing lines Patch % Lines
ax/utils/common/hash_utils.py 0.00% 26 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4992      +/-   ##
==========================================
- Coverage   96.84%   96.80%   -0.04%     
==========================================
  Files         601      602       +1     
  Lines       64732    64759      +27     
==========================================
+ Hits        62687    62689       +2     
- Misses       2045     2070      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 9, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 9, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 9, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
Summary:
Pull Request resolved: facebook#4992

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
ItsMrLin added a commit to ItsMrLin/Ax that referenced this pull request Mar 9, 2026
Summary:

Add hash-based data freshness tracking for LILO (Language-in-the-Loop)
pairwise preference labels.

When LILOPairwiseMetric produces labels, it now stamps a SHA-256 hash of the
experiment's LILO inputs (metric data for input_metric_names + LLM messages)
onto the trial's _properties. If any of these inputs change (new data arrives,
data is updated, or the user modifies LLM messages), the hash changes,
indicating that existing LILO labels are stale.

Changes:
- Add `LILO_INPUT_HASH` key to `Keys` enum in `constants.py`
- Create `ax/utils/common/hash_utils.py` with `compute_lilo_input_hash`
  (standalone hash function) and `get_current_lilo_hash` (convenience helper
  that looks up the pairwise `DerivedMetric` on an experiment, extracts
  `input_metric_names`, and computes the hash — returns `None` if no pairwise
  metric is registered)
- Stamp hash in `LILOPairwiseMetric._compute_derived_values` after producing labels
- Add tests for hash determinism, sensitivity to data/message changes, stamping,
  and `get_current_lilo_hash` helper

Reviewed By: saitcakmak

Differential Revision: D95284287
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants