Skip to content

Update documentation with results of testing update plugin #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
GregorioBlazquez wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update documentation with results of testing update plugin #57

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion doc/technical_configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,42 @@ Kafnus Connect plugins are automatically built into the Docker image and placed

This directory is **populated automatically** during the Docker build using the logic defined in the [Dockerfile](Dockerfile).

### 🔄 Plugin Upgrade Behavior

Kafka Connect stores connector definitions, status, and offsets in its internal Kafka topics:

- `connect-configs`
- `connect-offsets`
- `connect-status`

Because of this, connector configurations are not stored inside the Kafnus Connect container itself.

When upgrading the Kafnus Connect image (for example, to deploy a newer version of a plugin such as the JDBC Sink fork), existing connectors are automatically restored when the worker starts, provided that:

- The Kafka cluster remains available
- The internal Kafka Connect topics are preserved
- The connector class name remains unchanged
- Connector configuration remains compatible with the new plugin version

This means that, in normal upgrade scenarios:

1. Stop the existing Kafnus Connect container
2. Deploy the new Kafnus Connect image
3. Start the worker again

Existing connectors will be recovered automatically from Kafka and their tasks will be recreated using the updated plugin code.

No connector re-registration or recreation is required.

> ⚠️ Connector recreation may be required if a plugin upgrade introduces incompatible configuration changes or renames the connector class.

This behavior was validated during Kafnus Connect upgrade testing by:
- Creating and running JDBC sink connectors with a previous image version
- Producing data successfully
- Upgrading the Kafnus Connect image
- Verifying that connectors were automatically restored
- Confirming that data continued to be processed using the updated plugin implementation

### 1. JDBC Plugin for PostGIS

Includes:
Expand Down Expand Up @@ -782,7 +818,7 @@ This feature is typically used by *lastdata* flows to avoid overwriting newer ob

---

## ▶️ Registering Connectors (it will be updated)
## ▶️ Registering Connectors

From the `sinks/` directory, register each connector using `curl`:

Expand Down
Loading