Skip to content

perf(core): vectorize the date conversion in basemodel_to_df - #7622

Open
dexhunter wants to merge 1 commit into
OpenBB-finance:developfrom
dexhunter:feature/vectorize-basemodel-to-df-date-conversion
Open

perf(core): vectorize the date conversion in basemodel_to_df#7622
dexhunter wants to merge 1 commit into
OpenBB-finance:developfrom
dexhunter:feature/vectorize-basemodel-to-df-date-conversion

Conversation

@dexhunter

@dexhunter dexhunter commented Jul 31, 2026

Copy link
Copy Markdown

Description

basemodel_to_df runs three per-element Python passes over the date column of
every payload it converts:

if "date" in df.columns:
    df["date"] = df["date"].apply(to_datetime)
    if all(t.time() == time(0, 0) for t in df["date"]):
        df["date"] = df["date"].apply(lambda x: x.date())

Series.apply(to_datetime) calls pandas.to_datetime once per row. Each of
those calls receives a lone scalar with no format hint, so pandas also re-runs
_guess_datetime_format_for_array on every row. For a 1,000-row daily payload
that is 1,000 to_datetime entries and 1,000 format guesses. cProfile counts
84,000 regex searches inside the format guesser alone, and attributes
0.389 s of the call's 0.394 s to the two .apply passes.

This parses the column in a single call and uses vectorized comparisons for the
all-midnight test and the reduction to datetime.date.

Measurements

Payload built through EquityHistoricalData, so the date column carries what
the standard model actually produces after field validation. Median of 9,
Python 3.12, pandas 3.0.5, taken while holding an exclusive CPU lock.

payload before after saved
250 daily bars (~1 trading year) 8.92 ms 1.46 ms 7.46 ms
1,000 daily bars (~4 years) 32.93 ms 3.13 ms 29.80 ms
5,000 daily bars (~20 years) 162.25 ms 12.23 ms 150.02 ms

These are absolute figures for the conversion itself, not a share of a wider
request. For a command that fetches over the network, this is a small part of
end-to-end latency; for the local paths below it is most of the work.

Intraday payloads gain much less (1,000 bars: 4.35 ms → 3.46 ms) because the
all() short-circuits on the first non-midnight value, so only one of the three
passes ever runs.

Where this is called

Once per conversion, from 100 call sites: OBBject.to_dataframe(), which
backs the user-facing .to_df(), plus 26 technical_router commands, 26
econometrics_router commands, the quantitative/rolling/stats routers, and the
charting extension. Anything that turns list[Data] into a frame goes through
this function.

What this deliberately does not do

Faster variants exist. I measured each one and rejected it, and the reason
matters more than the number:

  • Plain to_datetime(df["date"]) with no fallback. Faster still, and
    wrong. One vectorized call cannot represent a column that mixes UTC
    offsets, so it raises ValueError: Mixed timezones detected on any series
    crossing a daylight-savings boundary. test_to_df_daylight_savings covers
    exactly that, and the per-element apply being replaced here is what makes
    the case work today. The fallback keeps behaviour the current code already
    has.
  • to_datetime(..., utc=True). Uniform and fast, but it rewrites the
    values: offsets get normalized to UTC instead of preserved per element.
  • Skipping the parse when the column already holds date/datetime
    objects.
    Cheapest of all, but it drops the normalization that string inputs
    rely on, so "2023-07-30" and date(2023, 7, 30) would stop converging.
  • .dt.time or .dt.floor("D") instead of .dt.normalize() for the
    midnight test. All three are correct and landed within a few percent of each
    other, which did not justify a less obvious spelling.

I have not touched df_to_basemodel, get_target_columns, or anything else in
this file. df_to_basemodel's to_json round-trip looks like it has real
headroom too, but that is a separate change with a separate risk profile and
belongs in its own PR if you want it.

How has this been tested?

  • pytest openbb_platform/core/tests/app/test_utils.py openbb_platform/core/tests/app/model/test_obbject.py
    51 passed, identical to the unmodified develop in the same
    environment. test_to_df_daylight_savings is in that set and passes.
  • Differential check against the current implementation across 15 payload
    shapes: daily, intraday, mixed midnight/non-midnight, non-midnight only in
    the last row, ISO strings, string datetimes, single row, no date column,
    index="date", index on another column, multiindex, tz-aware, mixed UTC
    offsets across a DST boundary, mixed offsets all at midnight, and a scalar
    Data of lists. Column dtypes, index identity and every cell compare equal.
  • ruff check, black --diff --check, codespell, mypy and pylint
    (10.00/10) all clean on the changed file.

Optimization search record

I ran an optimization search while working on this. Its trajectory is here:
https://dashboard.weco.ai/share/mAfGFdRF-ihwdTrtCnGoL8VsHTFXq8sl

That link is a record of the search, not an authority for the change. I wrote
and measured the diff against the tests above. The trajectory is also where the
daylight-savings constraint showed up: the evaluator rejected several candidate
patches that took the naive vectorized parse, against the mixed-offset case.

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have adhered to the GitFlow naming convention and my branch name is in
    the format of feature/feature-name or hotfix/hotfix-name.
  • I ensure that I am following the CONTRIBUTING guidelines.

basemodel_to_df ran three per-element Python passes over the date column of
every payload it converted. Series.apply(to_datetime) calls pandas.to_datetime
once per row, and because each call receives a lone scalar with no format hint,
pandas also re-runs its datetime format guesser on every row.

Parse the column in one call instead, and use vectorized comparisons for the
all-midnight test and the reduction to datetime.date.

The element-wise parse is kept as a fallback for one case that the vectorized
path cannot represent: a column mixing UTC offsets, such as a series crossing a
daylight-savings boundary. Parsing per element lets each value keep its own
offset and yields an object column of tz-aware Timestamps, whereas a single
to_datetime call over that column raises "Mixed timezones detected". That
behaviour is covered by test_to_df_daylight_savings and is preserved exactly.

Measured on a payload built through EquityHistoricalData, median of 9 under an
exclusive CPU lock:

  250 daily bars     8.92 ms -> 1.46 ms
  1,000 daily bars  32.93 ms -> 3.13 ms
  5,000 daily bars 162.25 ms -> 12.23 ms
@dexhunter
dexhunter force-pushed the feature/vectorize-basemodel-to-df-date-conversion branch from 840a963 to 621a6b5 Compare July 31, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant