Skip to content

PERF: Speed up DataFrame.to_csv with a formatted DatetimeIndex - #71844

Open
idan-friedman wants to merge 5 commits into
pandas-dev:mainfrom
idan-friedman:codex/pr44908-fork-review
Open

idan-friedman wants to merge 5 commits into
pandas-dev:mainfrom
idan-friedman:codex/pr44908-fork-review

Conversation

@idan-friedman

Copy link
Copy Markdown

PR #44908 improved CSV datetime-index export time, but left room for improvement by using pandas' internal batch formatter.

Change

Formatted DatetimeIndex values currently use a per-timestamp loop in DataFrame.to_csv. This change uses the datetime-array CSV formatter already used for datetime columns. It retains scalar formatting for PeriodIndex and for years before 1000 to preserve existing warnings and output.

Measurements and validation

On the original 100,000-row PR #44908 workload, the complete to_csv call measured 469.0 ms before and 107.3 ms after on Linux (4.37×). Four other numeric formats improved 2.75× to 4.07×; a weekday-name format improved only 1.09×. Complete CSV output hashes matched for all six formats.

Focused CSV tests passed on macOS and Linux (141 passed, 10 skipped on each). Local pre-commit checks and git diff --check passed. The full pandas test suite has not run.

AI use

I used Codex with OpenAI gpt-6-sol (xhigh reasoning) to investigate PR #44908 and prepare the implementation, tests, release note, benchmarks, and initial PR text. I used Claude Code with Sonnet 5.5 (high thinking) to review the patch. Its feedback prompted additional format benchmarks and correctness checks. I reviewed and understood every change before submitting.

Check exactly one of the following, per the automated contributions policy:

  • I did not use AI to develop this pull request.
  • I used AI to develop this pull request. I prompted it to follow AGENTS.md, I have reviewed and understood every change, and I have described above how I used it and exactly which tool, model version, and effort setting.

@github-actions github-actions Bot added the Needs Issue Assignment PR whose author is not assigned to a linked issue label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thanks for the pull request, @idan-friedman! It's linked to #71843, but that issue isn't assigned to you yet. To make sure two people don't unknowingly work on the same thing, we ask contributors to claim an issue first. You can comment /take on #71843 to claim it, but be aware that certain issues cannot be taken if they are labeled e.g. Needs Triage — see the contributing guide for more details.

This pull request is unlikely to be reviewed until #71843 is assigned to you.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs Issue Assignment PR whose author is not assigned to a linked issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PERF: DataFrame.to_csv with date_format remains slow for a DatetimeIndex

1 participant