Skip to content

index.md

Latest commit

 

History

History
28 lines (23 loc) · 3.08 KB

index.md

File metadata and controls

28 lines (23 loc) · 3.08 KB
Raw

Data Operation Patterns

Data operations are built-in feature groups that transform existing columns: row-preserving analytics (binning, ranks, windows, offsets, percentiles), group-reducing aggregations, and element-wise string transforms.

These guides describe the contracts each category must honor, the cross-framework reference-implementation pattern, and the end-to-end recipe for adding a new operation.

Read them after you are comfortable with the feature-group patterns. They extend that model with the additional invariants that make a single feature name produce identical results on PyArrow, Pandas, Polars, DuckDB, and SQLite.

Guides

  1. Overview - Categories, naming patterns, and where the code lives
  2. Row-preserving contract - Output row count and order must match input
  3. Reference implementation pattern - PyArrow is the source of truth
  4. Supported ops per framework - Excluding ops a framework cannot express
  5. Binning - bin vs qbin, NTILE vs rank, NULL handling
  6. Window aggregation - Partitioned aggregates broadcast per row
  7. Percentile, rank, offset - The analytic window family
  8. Scalar aggregate, frame aggregate, scalar arithmetic, and point arithmetic - Global broadcast, rolling/expanding windows, element-wise column-vs-constant arithmetic, and element-wise column-vs-column arithmetic
  9. String operations - Element-wise string transforms
  10. Adding a new data operation - End-to-end recipe
  11. Time bucketization - floor / ceil / round a timestamp to a bucket interval (minute / hour / day / week / month / year)
  12. Forward fill by time - Carry the last non-null value forward across time gaps, per partition (row-preserving)
  13. EMA - Exponential moving average ({col}__ema_{span}), per partition; pandas / polars native, pyarrow / duckdb / sqlite not implemented (no backend)
  14. Resample - Collapse events onto a regular time grid ({col}__resample_{n}_{unit}_{agg}); the first row_changing operation
  15. Sessionization - Assign a gap-threshold session id on an ordered timestamp ({ts}__sessionize_{n}_{unit}), per partition (row-preserving)
  16. return_data_type_rule failure handling - The fail-fast, post-selection contract: why the type rules no longer catch extraction errors, how binning/resample matching was tightened so a selected feature never raises, and the completeness-test guard (#244 -> #265, core #485/#493)
  17. Known divergences - Audited cases where a framework would diverge from the PyArrow reference, with the mitigation for each
  18. Framework support matrix - Operation x framework capability table, generated from supported_*() test overrides