Allow more column types to be interpolated #421

BrianDeacon · 2025-03-17T18:09:53Z

linear and zero interpolation methods continue to require numeric column types, but ffill, bfill, and null will work on any column type.

Changes

Rather than a blanket rejection of all non-numeric column types, the requirements are applied on a per-column basis depending on the interpolation method required. ValueError is still thrown when the column type doesn't work, but the check is done at the time of attempting to interpolate the column.

Linked issues

Resolves #420

Functionality

Tests

manually tested
added unit tests
added integration tests
verified on staging environment (screenshot attached)

…the specific method used

R7L208 · 2025-03-25T15:11:54Z

Hey @BrianDeacon - Given the Databricks License and that our Databricks Labs workflow broke to accept external contributions; I'm not sure if this can be merged. I am checking on options for you

cc: @pohlposition @gueniai

BrianDeacon · 2025-03-25T17:58:38Z

Hey @BrianDeacon - Given the Databricks License and that our Databricks Labs workflow broke to accept external contributions; I'm not sure if this can be merged. I am checking on options for you

cc: @pohlposition @gueniai

I mean, I'm pretty sure this just fell off a truck, and you wrote it @R7L208

;)

R7L208 · 2025-05-27T18:51:59Z

@BrianDeacon, if you're still interested in authoring this, could you email me the below info to [email protected] to get you whitelisted?

Contributor’s name
Contributor’s email address
Contributor’s GitHub handle
Contributor’s static IP address

BrianDeacon · 2025-06-08T22:29:24Z

Contributor’s static IP address

I've emailed Lorin. Is the static ip address a requirement?

Lol, you're Lorin. :)

R7L208 · 2025-06-12T23:46:12Z

#425 should fix the mypy errors. Once that's merged then you can pull the latest from master into this branch

@kwang-databricks fyi

R7L208 · 2025-06-16T14:21:54Z

@BrianDeacon - #425 was merged into master. Can you update your fork & branch? That PR should resolve the lint issues that are unrelated to your changes

BrianDeacon · 2025-06-18T21:39:58Z

@BrianDeacon - #425 was merged into master. Can you update your fork & branch? That PR should resolve the lint issues that are unrelated to your changes

Done!

BrianDeacon · 2025-06-23T18:22:41Z

@R7L208 Anything else?

BrianDeacon · 2025-06-26T22:17:35Z

Looks like the pipeline uncovered that I had some unit test code that wasn't compatible with older versions of pyspark. That's fixed now.

R7L208

Found a few small things but otherwise looks great!

R7L208 · 2025-06-27T21:00:15Z

python/tempo/interpol.py

+
+        if not self._is_valid_method_for_column(series, method, target_col):
+            raise ValueError(
+                f"Interpolation method '{method}' is not supported for column '{target_col}' of type '{series.schema[target_col].dataType}'"


@BrianDeacon - can you clarify the error message to indicate the column must be of NumericType but instead received a non-numeric type of {}

R7L208 · 2025-06-27T21:02:01Z

python/tests/interpol_tests.py

+            ValueError,
+            self.interpolate_helper.interpolate,
+            simple_input_tsdf,
+            freq="30 seconds", func="ceil", method="linear", ts_col="event_ts",


This should be method="zero" IIUC. Can you update and make sure test still passes?

R7L208 · 2025-06-27T21:04:47Z

python/tests/base.py

+        if "decimal_convert" in self.df:
+            for decimal_col in self.df["decimal_convert"]:
+                if "." in date_col:
+                    col, field = date_col.split(".")
+                    convert_field_expr = sfn.col(col).getField(field).cast("decimal")
+                    df = df.withColumn(
+                        col, sfn.col(col).withField(field, convert_field_expr)
+                    )
+                else:
+                    df = df.withColumn(decimal_col, sfn.col(decimal_col).cast("decimal"))


I believe this is a bug here and you need decimal_col inside of the for loop instead of date_col for the if expression and call of split(). Can you double check this?

Live by the clipboard, die by the clipboard. :) Fixed.

Can you sanity check me? As far as I can tell, none of the test configs pass in data that would hit these if "." branches. I'm guessing these got pulled in from some other code base? I'm not suggesting to yank it out, but I just wanted to make sure I wasn't misunderstanding how this works.

It's used in the interpol tests to manage decimal accuracy when converting data to DecimalType

But do the other variations ever run into a column with a "." in the name? These _convert variations are all to process these entries, right? So I just got "lucky" that none of these bits in the config had a . in the name?

"date_convert": ["date_col"],

BrianDeacon · 2025-06-30T17:14:14Z

@R7L208 Ready for another look. 👍

R7L208 · 2025-06-30T17:19:54Z

python/tests/base.py

@@ -176,7 +176,7 @@ def as_sdf(self) -> DataFrame:
        if "decimal_convert" in self.df:
            for decimal_col in self.df["decimal_convert"]:
                if "." in date_col:


Suggested change

if "." in date_col:

if "." in decimal_col:

:face_palm:
Fixed

R7L208

One final change i think as long as test pass

R7L208

LGTM! Thanks @BrianDeacon!

BrianDeacon mentioned this pull request Mar 18, 2025

[FEATURE]: Support more column types for resampling and interpolating #420

Closed

1 task

BrianDeacon changed the title ~~Allow resampling to only reject column types that cannot work for the…~~ Allow interpolation to only reject column types that cannot work for the… Mar 20, 2025

BrianDeacon changed the title ~~Allow interpolation to only reject column types that cannot work for the…~~ Allow more column types to be interpolated Mar 20, 2025

BrianDeacon marked this pull request as ready for review March 21, 2025 20:09

Allow interpolation to only reject column types that cannot work for …

6021d6c

…the specific method used

BrianDeacon force-pushed the expanded-resample branch from ace43dd to 6021d6c Compare March 21, 2025 20:19

Merge branch 'databrickslabs:master' into expanded-resample

a20f7bf

Merge branch 'databrickslabs:master' into expanded-resample

844fcb7

Fix unit test incompatibility with dbr113

5ec455c

Merge branch 'databrickslabs:master' into expanded-resample

bad301a

R7L208 requested changes Jun 27, 2025

View reviewed changes

Fix tests and clarify error message

17cd07b

R7L208 requested changes Jun 30, 2025

View reviewed changes

Fix incorrect column reference in test

95a61e0

R7L208 requested changes Jun 30, 2025

View reviewed changes

R7L208 approved these changes Jun 30, 2025

View reviewed changes

R7L208 merged commit 74c2b07 into databrickslabs:master Jun 30, 2025
7 checks passed

Allow more column types to be interpolated #421

Allow more column types to be interpolated #421

Conversation

BrianDeacon commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Linked issues

Functionality

Tests

Uh oh!

R7L208 commented Mar 25, 2025

Uh oh!

BrianDeacon commented Mar 25, 2025

Uh oh!

R7L208 commented May 27, 2025

Uh oh!

BrianDeacon commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

R7L208 commented Jun 12, 2025

Uh oh!

R7L208 commented Jun 16, 2025

Uh oh!

BrianDeacon commented Jun 18, 2025

Uh oh!

BrianDeacon commented Jun 23, 2025

Uh oh!

BrianDeacon commented Jun 26, 2025

Uh oh!

R7L208 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BrianDeacon commented Jun 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

R7L208 left a comment

Choose a reason for hiding this comment

Uh oh!

R7L208 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

BrianDeacon commented Mar 17, 2025 •

edited

Loading

BrianDeacon commented Jun 8, 2025 •

edited

Loading