Skip to content

Support creating empty files for streams with at least one SCHEMA message but no RECORD messages #119

Open
@edgarrmondragon

Description

@edgarrmondragon

I don't think that's possible with the way target-csv currently works. The logic to write to the file, and more importantly here create if it doesn't exist, lives in the process_batch implementation:

write_csv(
output_file,
context["records"],
self.schema,
escapechar=self.config.get("escape_character"),
)

My guess is that it's never called for empty streams because the method is only called when a "batch" is full, but that never happens for the stream in question. One fix I can think of is to add an optional early file creation step controlled by a new setting (e.g create_files_for_empty_streams), maybe within the Sink class __init__ method.

https://meltano.slack.com/archives/C069CQNHDNF/p1718436121705219?thread_ts=1718426268.425619&cid=C069CQNHDNF

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions