perf(core): vectorize the date conversion in basemodel_to_df - #7622
Open
dexhunter wants to merge 1 commit into
Open
perf(core): vectorize the date conversion in basemodel_to_df#7622dexhunter wants to merge 1 commit into
basemodel_to_df#7622dexhunter wants to merge 1 commit into
Conversation
basemodel_to_df ran three per-element Python passes over the date column of every payload it converted. Series.apply(to_datetime) calls pandas.to_datetime once per row, and because each call receives a lone scalar with no format hint, pandas also re-runs its datetime format guesser on every row. Parse the column in one call instead, and use vectorized comparisons for the all-midnight test and the reduction to datetime.date. The element-wise parse is kept as a fallback for one case that the vectorized path cannot represent: a column mixing UTC offsets, such as a series crossing a daylight-savings boundary. Parsing per element lets each value keep its own offset and yields an object column of tz-aware Timestamps, whereas a single to_datetime call over that column raises "Mixed timezones detected". That behaviour is covered by test_to_df_daylight_savings and is preserved exactly. Measured on a payload built through EquityHistoricalData, median of 9 under an exclusive CPU lock: 250 daily bars 8.92 ms -> 1.46 ms 1,000 daily bars 32.93 ms -> 3.13 ms 5,000 daily bars 162.25 ms -> 12.23 ms
dexhunter
force-pushed
the
feature/vectorize-basemodel-to-df-date-conversion
branch
from
July 31, 2026 23:08
840a963 to
621a6b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
basemodel_to_dfruns three per-element Python passes over thedatecolumn ofevery payload it converts:
Series.apply(to_datetime)callspandas.to_datetimeonce per row. Each ofthose calls receives a lone scalar with no format hint, so pandas also re-runs
_guess_datetime_format_for_arrayon every row. For a 1,000-row daily payloadthat is 1,000
to_datetimeentries and 1,000 format guesses.cProfilecounts84,000 regex searches inside the format guesser alone, and attributes
0.389 s of the call's 0.394 s to the two
.applypasses.This parses the column in a single call and uses vectorized comparisons for the
all-midnight test and the reduction to
datetime.date.Measurements
Payload built through
EquityHistoricalData, so thedatecolumn carries whatthe standard model actually produces after field validation. Median of 9,
Python 3.12, pandas 3.0.5, taken while holding an exclusive CPU lock.
These are absolute figures for the conversion itself, not a share of a wider
request. For a command that fetches over the network, this is a small part of
end-to-end latency; for the local paths below it is most of the work.
Intraday payloads gain much less (1,000 bars: 4.35 ms → 3.46 ms) because the
all()short-circuits on the first non-midnight value, so only one of the threepasses ever runs.
Where this is called
Once per conversion, from 100 call sites:
OBBject.to_dataframe(), whichbacks the user-facing
.to_df(), plus 26technical_routercommands, 26econometrics_routercommands, the quantitative/rolling/stats routers, and thecharting extension. Anything that turns
list[Data]into a frame goes throughthis function.
What this deliberately does not do
Faster variants exist. I measured each one and rejected it, and the reason
matters more than the number:
to_datetime(df["date"])with no fallback. Faster still, andwrong. One vectorized call cannot represent a column that mixes UTC
offsets, so it raises
ValueError: Mixed timezones detectedon any seriescrossing a daylight-savings boundary.
test_to_df_daylight_savingscoversexactly that, and the per-element
applybeing replaced here is what makesthe case work today. The fallback keeps behaviour the current code already
has.
to_datetime(..., utc=True). Uniform and fast, but it rewrites thevalues: offsets get normalized to UTC instead of preserved per element.
date/datetimeobjects. Cheapest of all, but it drops the normalization that string inputs
rely on, so
"2023-07-30"anddate(2023, 7, 30)would stop converging..dt.timeor.dt.floor("D")instead of.dt.normalize()for themidnight test. All three are correct and landed within a few percent of each
other, which did not justify a less obvious spelling.
I have not touched
df_to_basemodel,get_target_columns, or anything else inthis file.
df_to_basemodel'sto_jsonround-trip looks like it has realheadroom too, but that is a separate change with a separate risk profile and
belongs in its own PR if you want it.
How has this been tested?
pytest openbb_platform/core/tests/app/test_utils.py openbb_platform/core/tests/app/model/test_obbject.py→ 51 passed, identical to the unmodified
developin the sameenvironment.
test_to_df_daylight_savingsis in that set and passes.shapes: daily, intraday, mixed midnight/non-midnight, non-midnight only in
the last row, ISO strings, string datetimes, single row, no
datecolumn,index="date", index on another column, multiindex, tz-aware, mixed UTCoffsets across a DST boundary, mixed offsets all at midnight, and a scalar
Dataof lists. Column dtypes, index identity and every cell compare equal.ruff check,black --diff --check,codespell,mypyandpylint(10.00/10) all clean on the changed file.
Optimization search record
I ran an optimization search while working on this. Its trajectory is here:
https://dashboard.weco.ai/share/mAfGFdRF-ihwdTrtCnGoL8VsHTFXq8sl
That link is a record of the search, not an authority for the change. I wrote
and measured the diff against the tests above. The trajectory is also where the
daylight-savings constraint showed up: the evaluator rejected several candidate
patches that took the naive vectorized parse, against the mixed-offset case.
Checklist
the format of
feature/feature-nameorhotfix/hotfix-name.