Skip to content

10-create-compute-framework.md

Latest commit

 

History

History
83 lines (62 loc) · 3.57 KB

10-create-compute-framework.md

File metadata and controls

83 lines (62 loc) · 3.57 KB
Raw

Create a Compute Framework Plugin

Use this guide to add support for a new data processing library to mloda.

Decision Tree

Q1: Does your framework require a connection/session object?
    YES → Q2
    NO  → Q3

Q2: Is it a data lake table format (Iceberg, Delta, Hudi)?
    YES → Category 5: Data Lake
    NO  → Category 3: Stateful Connection

Q3: Does your framework use lazy evaluation?
    YES → Category 2: Stateless Lazy
    NO  → Q4

Q4: Does your framework have external dependencies?
    YES → Category 1: Stateless Eager
    NO  → Category 4: Zero Dependency

Q5: Do you need cross-framework conversion?
    YES → See 08-framework-transformer

Q6: Does your library have built-in PyArrow conversion?
    YES → Simplifies transformer (use .to_arrow()/.from_arrow())
    NO  → Manual conversion in 08-framework-transformer

Q7: Need to understand merge/join operations?
    YES → See 06-merge-engine

Q8: Do you need multi-column index support for joins?
    YES → See 06-merge-engine (Index with tuple)

Q9: Need to understand filter operations?
    YES → See 07-filter-engine

Q10: Should connections be auto-created or user-provided?
    AUTO  → Add fallback in set_framework_connection_object()
    USER  → Require via data_access_collection parameter

Q11: Ready to test your implementation?
    YES → See 09-testing-guide

Category Guides

Category When to Use
01-stateless-eager Simple in-memory frameworks (Pandas, PyArrow, Polars DataFrame)
02-stateless-lazy Lazy evaluation frameworks (Polars LazyFrame, Ibis)
03-stateful-connection Connection/session required (DuckDB, SQLite, Spark)
04-zero-dependency Pure Python, no external libs (Python dict)
05-data-lake Catalog-based table formats (Iceberg, Delta)

Concepts

Guide What It Covers
06-merge-engine Join operations (INNER, LEFT, OUTER, APPEND, UNION)
07-filter-engine Filter operations (range, equal, regex, categorical)
08-framework-transformer Cross-framework conversion (PyArrow as hub)
09-testing-guide Testing your implementation
10-data-type-extraction Mapping native column types to mloda DataType

Timezone Validation (Opt-In)

Merge and filter engines can opt in to a timezone guard that turns silent tz-aware vs tz-naive mismatches into a clear ValueError.

  • Opt in: set provides_column_semantics = True on your BaseMergeEngine / BaseFilterEngine subclass and implement _column_semantics(data, column) returning a ColumnSemantics.
  • Default False: guard skipped, hook never required.
  • Opted in without the hook: NotImplementedError.
  • As-of joins require the hook regardless of the flag.

See 06-merge-engine and 07-filter-engine for details, and the upstream comparison contract for the full model.