Question: Retry logic/configuration for Connector

[timothydilbert]

I have a use case where I want the HTTP Connector to retry delivering failed messages. That part, I've figured out and have it configured to my liking.

The thing is, in my case, I want the HTTP Connector to retry delivering the message several times before giving up on that message - then evict/discard that message and move onto the next message in the Kafka Topic queue. Right now, after the error threshold, the Aiven HTTP Connector fails outright, and the Aiven HTTP Connector does not deliver any further messages, holding up the entire queue. Below is my configuration:

{
    "name": "EXAMPLE_MESSAGE",
    "config": {
        "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
        "topics.regex": "EXAMPLE_TOPIC",
        "http.authorization.type": "none",
        "http.url": "https://api.domain.local/path/to/my/api",
        "batching.enabled": "false",
        "batch.max.size": 1,
        "tasks.max": "1",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "group.id": "my_group_name",
        "retry.backoff.ms": 3000,
        "max.retries": 5,
        "consumer.override.auto.offset.reset": "latest"
    }
 }

How do I achieve the outcome I want? Where if message #1 fails after several attempts, the Aiven HTTP Connector will just move onto message #2.