Skip to content

README.md

Latest commit

 

History

History
39 lines (31 loc) · 1.62 KB

README.md

File metadata and controls

39 lines (31 loc) · 1.62 KB
Raw

A tool for visual explanation of ETL pipelines of structured data

etl_visual_explained is a Jupyter tool for visual debugging and step-by-step analysis of Pandas (and potentially Polars) data pipelines, designed to make every transformation in ETL processes transparent.

Main Features

  • Magic command %%visual_chain for Jupyter notebooks: executes and visualizes each step of a Pandas method chain, showing the DataFrame state after every transformation.
  • Interactive HTML visualization: each step is displayed with details about code, execution time, data preview, and metadata (shape, columns, dtypes).
  • Detailed analysis: errors and performance are tracked step-by-step, making pipeline debugging and optimization easier.
  • Extensible: the modular structure (parser, executor, renderer) allows easy adaptation to new use cases or libraries.

Project Structure

  • etl_visual_explained/magics.py: defines the %%visual_chain cell magic for Jupyter.
  • etl_visual_explained/core/: contains the parsing logic (parser.py), step-by-step execution (executor.py), and data models (models.py).
  • etl_visual_explained/ui/renderer.py: generates the HTML visualization of pipeline steps.
  • try.ipynb: example notebook with data generation and magic usage.

Usage Example

%load_ext etl_visual_explained

# In a Jupyter cell:
%%visual_chain
df = (
	my_df
	.filter(...)
	.assign(...)
	# other transformations
)

Requirements

  • Python >= 3.9
  • pandas >= 1.5.0
  • ipython >= 8.0.0
  • anywidget >= 0.9.0 (optional)
  • traitlets >= 5.0.0

For details and customization, see the source code and the example notebook.