Skip to content

Convert numeric interaction timestamps - #1210

Open
appleweiping wants to merge 3 commits into
lenskit:mainfrom
appleweiping:fix/1183-timestamp-conversion
Open

Convert numeric interaction timestamps#1210
appleweiping wants to merge 3 commits into
lenskit:mainfrom
appleweiping:fix/1183-timestamp-conversion

Conversation

@appleweiping

Copy link
Copy Markdown
Contributor

Fixes #1183.

This converts numeric interaction timestamps to Arrow timestamp types in both DatasetBuilder.add_interactions and from_interactions_df.

  • adds an explicit timestamp_unit option
  • infers integer values from seconds through nanoseconds when at least 95 percent fall between 1900 and 2099
  • treats floating point values as seconds by default, preserves millisecond precision, and converts NaN to null
  • preserves existing timestamp columns and promotes compatible units across batches
  • keeps timestamp attributes usable in SciPy sparse matrices as integer seconds

Validation:

  • data test suite: 438 passed, 40 skipped, 3 expected failures
  • focused timestamp and builder tests: 64 passed
  • Ruff lint and format checks passed

Developed with OpenAI Codex assistance. I reviewed and tested the change and remain responsible for it.

@appleweiping

Copy link
Copy Markdown
Contributor Author

The dev container check failed in tests/tuning/test_optuna_point.py::test_float_points with a Hypothesis FlakyFailure. A generated log scale float fell about 0.0011 below its lower bound, then the failure did not reproduce on Hypothesis retry. This test is outside the timestamp code changed here. I do not have permission to rerun upstream Actions. Could a maintainer rerun the failed job when convenient?

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.69767% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.42%. Comparing base (396c743) to head (4338381).

Files with missing lines Patch % Lines
src/lenskit/data/_builder.py 90.24% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1210      +/-   ##
==========================================
- Coverage   90.42%   90.42%   -0.01%     
==========================================
  Files         263      263              
  Lines       17559    17602      +43     
==========================================
+ Hits        15878    15916      +38     
- Misses       1681     1686       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mdekstrand mdekstrand left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for submitting this PR! The dev container test is flaky right now for unclear reasons, so I'm not worried about that one failing if everything else is passing. I've also kicked off the data CI tests (those require manual approval).

The strategy looks sound. I've added comments on some places where we need additional documentation, and one logic change (add_relationships needs to support timestamp inference too).

Comment thread tests/data/test_builder_interactions.py Outdated
@mark.parametrize(
("unit", "value"),
[
("s", 1_700_000_000),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are quite a few magic numbers in the new tests in this file — what is the purpose of selecting these values specifically? Please add code comments describing what these values are chosen to accomplish.

Comment thread src/lenskit/data/_builder.py Outdated
allow_repeats: bool = True,
default: bool = False,
remove_repeats: bool | Literal["exact"] = False,
timestamp_unit: TimestampUnit | None = None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put Literal[...] directly in the function signatures instead of defining a separate type — that way the acceptable values will be directly in the method documentation.

Comment thread src/lenskit/data/_builder.py Outdated
integer units are inferred and floating-point values are assumed to
be seconds. Numeric timestamps are converted to Arrow timestamps.
"""
data = _convert_interaction_timestamps(data, timestamp_unit)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add the timestamp_unit option to add_relationships as well, and move the conversion logic into that method. It is valid to use add_relationships to add interactions, and the timestamp logic should be identical.

Comment thread src/lenskit/data/_builder.py Outdated
Comment on lines +1245 to +1246
_TIMESTAMP_MIN_SECONDS = -2_208_988_800
_TIMESTAMP_MAX_SECONDS = 4_102_444_800

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarity, it would be better to define these by creating datetime objects from ISO date strings and extracting the UNIX timestamp, so that it is obvious what these values mean when they are defined.

@appleweiping

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I moved timestamp conversion into add_relationships and added timestamp_unit there. add_interactions now forwards the option. I also inlined the accepted values in the public signatures, replaced the numeric bounds with ISO datetimes, and explained the test values. I added direct add_relationships coverage as well. The complete data test suite passed with 440 tests passed, 42 skipped, and 3 expected failures. Ruff checks passed.

@appleweiping

Copy link
Copy Markdown
Contributor Author

The new test jobs are failing before project setup because the mise action is attempting to download version 2026.9.3 and receives a 404. LensKit tests do not start. The local full data suite remains green.

@appleweiping

Copy link
Copy Markdown
Contributor Author

The previous rerun completed with all project and dataset tests passing. Codecov's project check remained red on a 0.01% overall coverage change even though patch coverage passed.

I added direct coverage for the remaining timestamp conversion branches: an explicit integer unit, an all-null timestamp column, a nonnumeric timestamp column, and an invalid unit. The updated interaction test file passes all 26 tests, and Ruff passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Convert timestamp columns to timestamps in DatasetBuilder

2 participants