Skip to content

Normalize configuration doesn't support loader_file_format anymore #2641

Open
@bartcode

Description

@bartcode

dlt version

1.10.0

Describe the problem

This change removed the loader_file_format from the NormalizeConfiguration class.

According to the docs (e.g. here), you should be able to specificy loader_file_format="parquet" in the [normalize] section, and dlt would then write Parquet files to the destination. But this option doesn't work anymore now.

Expected behavior

The loader file format should be according to what's in .dlt/config.toml.

Steps to reproduce

Add this to your config.toml and see how the result is always written as jsonl after running a pipeline:

[normalize]
loader_file_format="parquet"

Operating system

Linux

Runtime environment

Local

Python version

3.13

dlt data source

Filesystem, SQL, REST API

dlt destination

No response

Other deployment details

No response

Additional information

No response

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Type

No type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions