Group data every hour

[amit2103]

Hey, currently when we group by say key whats the time period ill which it will group ?

Is there a way to group data every hour and send to GCS ?

[DanielWozniak94]

Looks like this can be addressed with setting offset.flush.interval.ms. Only issue is, it needs to be done on worker level since on connector level the config wasn't getting picked up.

[ivanyu]

Hi @amit2103 and @DanielWozniak94
A very important note here is that grouping by key can't include other fields like timestamp. So, it's literally one value per file.

Apart from this, offset.flush.interval.ms is the only way to control this now. Unfortunately, it's Connect current limitation that it's not possible to set this per connector. The connector itself is a bit reactive in this sense: it doesn't have it's own thread or timer of any sort that could prompt it to flush on its own schedule.