Finer control on file writing / coping with a GCS bucket without delete

[mkeskells]

We have a need to have a finer grain of control of the files written.
We want to configure GCS buckets without delete permission, so we want to be able to write files with a suffix or similar

Some background as to why we want this

• For us the downstream processes can cope with duplicate records, but cant cope with records going missing (e.g. you write a file, but commit fails, connect stops, and on restart the storage parameters have changed (or for any other reason we store less /different content in the file)
• If the file had been consumed (by a downstream process) and indexed then the index would not be valid
• Its also nicer to ensure that we cant accidentally delete data that we may legally have to keep for 7 years ...

Its a relatively simple process to detect and cope with I believe, but not a current capability that is configurable as I understand it.

For us any suffix on the filename would be OK, but there are probably other use cases that may want more specific control

What are you thoughts on this?

[Claudenw]

I want to see the ability to define the file pattern using multiple components.

I think this can be resolved as part of a migration to a better file name template. Specifically I think we should include the ability to add items that impact the file name should be specified in the file name pattern.

A partial list includes

• {{compression}} – the compression defaults to {{compression::ext}}
  • {{compression:name}} – e.g. GZIP
  • {{compression:ext} – e.g. .gz {{compression}} should default to this
• {{topic}} – the topic the message was from.
• {{topic:upper} - force upper case
• {{topic::lower} - force lower case
• {{partition}}
  • {{partition:digits}} – partition value left padded with zeros out to digits positions
• {{start-offset}}
  • {{start-offset:digits}} - offset partition value left padded with zeros out to digits positions
• {{timestamp}}
  • {{timestamp:digits}} - timestamp partition value left padded with zeros out to digits positions
  • {{timestamp:format_char}} – see Formatter (Java Platform SE 8 ) for a complete list of date/time format options.
• {{format}} The format type for the file (see commons main …/config/FormatType.java) Defaults to {{format::lower}}
  • {{format:upper}}
  • {{format::lower}
• {{prefix}} the prefix defined in the configuration
  • {{prefix::upper}}
  • {{prefix::lower}}

I am not certain that prefix is required but we currently have it defined in the system.

Using this you would be able to define an extension that was the timestamp (or part of the timestamp) Would this work, or do you need some sort of counter?

[mkeskells]

Hi We don't use your libraries any more. Take or ignore any ideas or code from any of my PRs Regards Mike Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Claude Warren ***@***.***> Sent: Wednesday, May 14, 2025 10:31:12 AM To: Aiven-Open/cloud-storage-connectors-for-apache-kafka ***@***.***> Cc: Mike Skells ***@***.***>; Author ***@***.***> Subject: Re: [Aiven-Open/cloud-storage-connectors-for-apache-kafka] Finer control on file writing / coping with a GCS bucket without delete (Discussion #339) I want to see the ability to define the file pattern using multiple components. I think this can be resolved as part of a migration to a better file name template. Specifically I think we should include the ability to add items that impact the file name should be specified in the file name pattern. A partial list includes * {{compression}} – the compression defaults to {{compression::ext}} * {{compression:name}} – e.g. GZIP * {{compression:ext} – e.g. .gz {{compression}} should default to this * {{topic}} – the topic the message was from. * {{topic:upper} - force upper case * {{topic::lower} - force lower case * {{partition}} * {{partition:}} – partition value left padded with zeros out to positions * {{start-offset}} * {{start-offset:}} - offset partition value left padded with zeros out to positions * {{timestamp}} * {{timestamp:}} - timestamp partition value left padded with zeros out to positions * {{timestamp:}} – see Formatter (Java Platform SE 8 )<https://docs.oracle.com/javase/8/docs/api/java/util/Formatter.html#syntax> for a complete list of date/time format options. * {{format}} The format type for the file (see commons main …/config/FormatType.java) Defaults to {{format::lower}} * {{format:upper}} * {{format::lower} * {{prefix}} the prefix defined in the configuration * {{prefix::upper}} * {{prefix::lower}} I am not certain that prefix is required but we currently have it defined in the system. Using this you would be able to define an extension that was the timestamp (or part of the timestamp) Would this work, or do you need some sort of counter? — Reply to this email directly, view it on GitHub<#339 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFJ6LZN5PWROPYSFAODTKSD26MEOBAVCNFSM6AAAAABRR42NGKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMJUGI3DKOA>. You are receiving this because you authored the thread.Message ID: ***@***.***>