Skip to content

resample/interpolate doesn't handle columns with dot in name #361

@ghormann

Description

@ghormann

ISSUE

While maybe not the best practice, databricks does allow columns to have "." in the name. When doing a resample with interporlate, this results in a cannot resolve column name error

How to reproduce

  1. Create a TSDF with columns that include a "."
    image

  2. Attempt to resample and interporlate with

resample_tsdf = base_tsdf.resample(freq="30 seconds", func="mean").interpolate(method="ffill")

An error is produced

AnalysisException: Cannot resolve column name "Bundler.Status.CurMachSpeed" among (site, line, ts, Bundler.Status.CurMachSpeed, Bundler.Status.MachSpeed, agg_key); did you mean to quote the Bundler.Status.CurMachSpeed column?

Workaround

Rename the columns before resampling / interpolate

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions