Skip to content

Introduce DataTableStore abstraction#7088

Open
jkschneider wants to merge 8 commits intomainfrom
jkschneider/data-table-store
Open

Introduce DataTableStore abstraction#7088
jkschneider wants to merge 8 commits intomainfrom
jkschneider/data-table-store

Conversation

@jkschneider
Copy link
Member

@jkschneider jkschneider commented Mar 21, 2026

Summary

  • Replace legacy in-memory Map on ExecutionContext with a pluggable DataTableStore interface
  • InMemoryDataTableStore (default), CsvDataTableStore (streaming to disk), NoOpDataTableStore
  • DataTableExecutionContextView for accessing the store following the existing view pattern
  • DataTable gains group (community tables) and instanceName (lazy, for disambiguation)
  • CSV files include @name/@instanceName/@group metadata comments via Univocity
  • Precondition suppression via DataTableStore.noop()
  • Common recipes set instance names from their options
  • Python/C#/TypeScript stores aligned with idiomatic conventions

Merge Order and Coordinated PRs

Phase 1 — Core (merge first)

  1. openrewrite/rewrite — THIS PR. Publish snapshot before proceeding.

Phase 2 — Parallel (after rewrite snapshot)

  1. openrewrite/rewrite-maven-plugin — Use CsvDataTableStore for data table export rewrite-maven-plugin#1125
  2. openrewrite/rewrite-gradle-plugin — Use CsvDataTableStore for data table export rewrite-gradle-plugin#430
  3. openrewrite/rewrite-prethink — Migrate to DataTableStore API rewrite-prethink#10
  4. moderneinc/rewrite-prethink — https://github.com/moderneinc/rewrite-prethink/pull/27
  5. moderneinc/moderne-cli — https://github.com/moderneinc/moderne-cli/pull/3544
  6. moderneinc/moderne-worker — https://github.com/moderneinc/moderne-worker/pull/1261

Phase 3 — SaaS (after CLI + Worker)

  1. moderneinc/moderne-saas — https://github.com/moderneinc/moderne-saas/pull/328

Impacts

Breaking: DATA_TABLES constant removed from ExecutionContext. Use DataTableExecutionContextView.

Breaking: RecipeRun.exportDatatablesToCsv() removed. Plugins now set CsvDataTableStore on ctx.

Breaking: DataTableDescriptor has new nullable fields (instanceName, group).

Filenames change: Data table CSV files now include sanitized instance name suffix with 4-char hash.

GraphQL additions (non-breaking): instanceName and group on DataTable types.

CSV metadata: Files start with # @name, # @instanceName, # @group comments.

Replace the legacy in-memory Map<DataTable<?>, List<?>> on ExecutionContext
with a pluggable DataTableStore interface. This enables different environments
to handle data table storage differently: in-memory for tests, streaming to
compressed CSV for the CLI, and file-based exchange for RPC.

Key changes:
- DataTableStore interface with insertRow, getRows (Stream), getDataTables
- InMemoryDataTableStore (default), CsvDataTableStore (file-based), NoOpDataTableStore
- DataTableExecutionContextView for accessing the store
- DataTable gains group (community tables) and instanceName (disambiguation)
- CsvDataTableStore writes @name/@instanceName/@group metadata comments
- Precondition suppression via DataTableStore.noop() and getMessage interception
- RecipeRunStats.flush() uses standard insertRow path
- Common recipes (FindMethods, Find, FindTypes, etc.) set instance names
- Python/C#/TypeScript stores aligned with idiomatic naming conventions
@github-project-automation github-project-automation bot moved this to In Progress in OpenRewrite Mar 21, 2026
jkschneider added a commit to openrewrite/rewrite-prethink that referenced this pull request Mar 21, 2026
ExportContext and GenerateCalmArchitecture read from
DataTableExecutionContextView instead of ctx.getMessage(DATA_TABLES).

Depends on: openrewrite/rewrite#7088
jkschneider added a commit to openrewrite/rewrite-maven-plugin that referenced this pull request Mar 21, 2026
Replace recipeRun.exportDatatablesToCsv() with CsvDataTableStore set on
the ExecutionContext before recipe execution.

Depends on: openrewrite/rewrite#7088
jkschneider added a commit to openrewrite/rewrite-gradle-plugin that referenced this pull request Mar 21, 2026
Replace recipeRun.exportDatatablesToCsv() with CsvDataTableStore set on
the ExecutionContext before recipe execution.

Depends on: openrewrite/rewrite#7088
…e API

- Regenerate recipes.csv to include instanceName/group fields in DataTableDescriptor
- DataTableDescriptor uses @JsonCreator for backward-compatible deserialization
- Update C# DataTableTests to use GetRows/GetDataTables instead of removed Rows/AcceptRows
}
headers.addAll(suffixColumns.keySet());

OutputStream os = outputStreamFactory.apply(path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

callout: File resource leak

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that I was wrong about this one. It's a delayed close inside of BucketWriter, so as long as BucketWriter gets closed, then things are good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants