Replies: 2 comments
-
I want to see the ability to define the file pattern using multiple components. I think this can be resolved as part of a migration to a better file name template. Specifically I think we should include the ability to add items that impact the file name should be specified in the file name pattern. A partial list includes
I am not certain that prefix is required but we currently have it defined in the system. Using this you would be able to define an extension that was the timestamp (or part of the timestamp) Would this work, or do you need some sort of counter? |
Beta Was this translation helpful? Give feedback.
-
Hi
We don't use your libraries any more. Take or ignore any ideas or code from any of my PRs
Regards
Mike
Get Outlook for Android<https://aka.ms/ghei36>
…________________________________
From: Claude Warren ***@***.***>
Sent: Wednesday, May 14, 2025 10:31:12 AM
To: Aiven-Open/cloud-storage-connectors-for-apache-kafka ***@***.***>
Cc: Mike Skells ***@***.***>; Author ***@***.***>
Subject: Re: [Aiven-Open/cloud-storage-connectors-for-apache-kafka] Finer control on file writing / coping with a GCS bucket without delete (Discussion #339)
I want to see the ability to define the file pattern using multiple components.
I think this can be resolved as part of a migration to a better file name template. Specifically I think we should include the ability to add items that impact the file name should be specified in the file name pattern.
A partial list includes
* {{compression}} – the compression defaults to {{compression::ext}}
* {{compression:name}} – e.g. GZIP
* {{compression:ext} – e.g. .gz {{compression}} should default to this
* {{topic}} – the topic the message was from.
* {{topic:upper} - force upper case
* {{topic::lower} - force lower case
* {{partition}}
* {{partition:}} – partition value left padded with zeros out to positions
* {{start-offset}}
* {{start-offset:}} - offset partition value left padded with zeros out to positions
* {{timestamp}}
* {{timestamp:}} - timestamp partition value left padded with zeros out to positions
* {{timestamp:}} – see Formatter (Java Platform SE 8 )<https://docs.oracle.com/javase/8/docs/api/java/util/Formatter.html#syntax> for a complete list of date/time format options.
* {{format}} The format type for the file (see commons main …/config/FormatType.java) Defaults to {{format::lower}}
* {{format:upper}}
* {{format::lower}
* {{prefix}} the prefix defined in the configuration
* {{prefix::upper}}
* {{prefix::lower}}
I am not certain that prefix is required but we currently have it defined in the system.
Using this you would be able to define an extension that was the timestamp (or part of the timestamp) Would this work, or do you need some sort of counter?
—
Reply to this email directly, view it on GitHub<#339 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFJ6LZN5PWROPYSFAODTKSD26MEOBAVCNFSM6AAAAABRR42NGKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMJUGI3DKOA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
We have a need to have a finer grain of control of the files written.
We want to configure GCS buckets without delete permission, so we want to be able to write files with a suffix or similar
Some background as to why we want this
Its a relatively simple process to detect and cope with I believe, but not a current capability that is configurable as I understand it.
For us any suffix on the filename would be OK, but there are probably other use cases that may want more specific control
What are you thoughts on this?
Beta Was this translation helpful? Give feedback.
All reactions