feat(new source): Add ODBC source implementation by powerumc · Pull Request #24044 · vectordotdev/vector

powerumc · 2025-10-22T13:11:57Z

Summary

This PR implemented a new ODBC(Open Database Connectivity) Source.

Vector configuration

Manual configuration example

[sources.odbc]
type = "odbc"
connection_string = "driver={MySQL ODBC 8.0 ANSI Driver};server=localhost;port=13306;database=vector_db;uid=vector;pwd=vector;"
statement = "SELECT * FROM odbc_table WHERE id > ? LIMIT 1;"
schedule = "*/5 * * * * *"
schedule_timezone = "UTC"
last_run_metadata_path = "odbc_tracking.json"
tracking_columns = ["id"]
statement_init_params = { id = "0" }

[sinks.console]
type = "console"
inputs = ["odbc"]
encoding.codec = "json"

Create example sql file

cat << EOF > example.sql
DROP TABLE IF EXISTS odbc_table;
CREATE TABLE odbc_table
(
    id int auto_increment primary key,
    name varchar(255) null,
    datetime datetime null
);

INSERT INTO odbc_table (name, datetime) VALUES
('test1', now()),
('test2', now()),
('test3', now()),
('test4', now()),
('test5', now());
EOF

Configure ODBC and MariaDB Driver on MacOS

brew install unixodbc
brew install mariadb-connector-odbc

cat << EOF >> /opt/homebrew/etc/odbcinst.ini
[MariaDB Unicode]
Description=MariaDB Connector/ODBC v.3.0
Driver=/opt/homebrew/Cellar/mariadb-connector-odbc/3.2.7/lib/mariadb/libmaodbc.dylib
EOF

Run MariaDB docker container

docker run \
  --rm \
  --name mariadb \
  -e MYSQL_ROOT_PASSWORD=vector \
  -e MYSQL_USER=vector \
  -e MYSQL_PASSWORD=vector \
  -e MYSQL_DATABASE=vector_db \
  -v $(pwd)/example.sql:/docker-entrypoint-initdb.d/example.sql:ro \
  -p 3306:3306 \
  mariadb:latest

How did you test this PR?

I tested it with integration tests of two databases: MariaDB and Postgresql.
(Testing the MySQL container integration in an ARM64 architecture is not simple, so I tested it with MariaDB. Instead, I manually tested MySQL locally.)

cargo vdev integration test odbc-mariadb
cargo vdev integration test odbc-postgresql

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Related: Prototype MySQL Database Source Implementation and Roadmap Inquiry for Official Database Sources/Sinks Support #22931

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

…ling logic

…ata_path properties tests

…tests

# Conflicts: # Cargo.lock # Cargo.toml # scripts/integration/Dockerfile # src/internal_events/mod.rs # src/sources/mod.rs

…ostgreSQL services

…ion for an ODBC source

…cking

…for MariaDB usage

…th in OdbcConfig

… preserve exact values

…r file in OdbcConfig

…eturn Result type

…ics tracking

…ul shutdown

…lue_to_sql_parameter function for improved SQL binding

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c798594fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… OdbcConfig and enforce non-empty statement requirement

…sary processing

…l data loss

…ze errors and improve metric tracking

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a9e08785f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 26419942f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…to_timestamp_value function for improved DST and fallback management

…or and allow indefinite wait

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c08668179a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-17T13:46:03Z

  "sources-logstash",
  "sources-mqtt",
  "sources-nats",
+  "sources-odbc",


Install unixODBC in release runtimes

Adding sources-odbc to the default log sources pulls odbc-api/odbc-sys into the base release binary, so Linux images now need the system ODBC driver manager at load time. I checked the release Dockerfiles: the Debian image only installs ca-certificates tzdata systemd libsasl2-2, and the distroless-libc image only stages zlib, so their vector --version smoke tests and container startup can fail with a missing libodbc unless the runtime images/package dependencies include unixODBC or ODBC stays out of the default feature set.

Useful? React with 👍 / 👎.

…ues and improve documentation

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07cc01a7ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-17T14:21:35Z

+
+#[tokio::test]
+async fn query_executed_with_init_params() {
+    const LAST_RUN_METADATA_PATH: &str = "odbc_tracking-integration-tests.json";


Isolate ODBC integration test state

When cargo vdev int test odbc-* invokes cargo nextest run (see vdev/src/testing/runner.rs) with the new ODBC test.yaml filter and no --test-threads=1, these #[tokio::test]s can run concurrently. This checkpoint path is also used by query_executed_with_filepath, and both tests drop/recreate odbc_table, so one test can delete or advance the other's table/checkpoint while its scheduled source is still polling, making the integration suite flaky; use per-test table/file names or force these tests to run serially.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T14:21:35Z

+        .filter_map(|col| {
+            let value = map.get(col)?;
+            value_to_sql_parameter(value).map(|param| param.into_parameter())
+        })


Reject missing tracking parameter values

When tracking_columns is set, this filter_map silently drops declared columns that are absent from statement_init_params/metadata/last result row, or whose value is Null/unconvertible. For a query whose ? placeholders correspond to those columns, the next scheduled execution binds fewer ODBC parameters than the SQL expects and fails every tick instead of surfacing the bad tracking state when it is loaded or saved; return an error if any declared tracking column cannot be converted to a parameter.

Useful? React with 👍 / 👎.

…loss when last_run_metadata_path is set

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b79856da3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-17T15:15:02Z

+            let latest_result = extract_and_save_tracking(
+                cfg.last_run_metadata_path.as_deref(),
+                last.clone(),
+                tracking_columns,
+            )


Validate metadata path before sending rows

When last_run_metadata_path points to a missing parent directory or an unwritable file, this checkpoint write runs only after send_batch has accepted the rows. The write error is returned, prev_result is not advanced, and the next scheduled poll falls back to the same old metadata/init params, so the source re-emits the same database rows every tick until the path is fixed; validate or prepare the checkpoint path before sending rows, or avoid sending rows that cannot be checkpointed.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T15:15:02Z

+
+	classes: {
+		commonly_used: false
+		delivery:      "at_least_once"


Don't advertise at-least-once delivery

The component metadata says ODBC is at_least_once, but the source intentionally disables acknowledgements and src/sources/odbc/config.rs:345 documents at-most-once behavior when last_run_metadata_path is used. Users reading the generated reference will expect rows to survive crashes or downstream sink failures, while this source can advance its checkpoint once rows enter the topology; remove the at-least-once claim or mark the weaker delivery guarantee.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T15:15:02Z

+                cols.insert(key, value);
+            }
+
+            rows.push(Value::Object(cols))


Stream large ODBC result sets in bounded chunks

When a scheduled query returns a large result set, this loop keeps appending every fetched row to rows until the cursor is exhausted before decoding or sending anything. odbc_batch_size only limits the driver fetch buffer, so a broad query can still build one unbounded in-memory JSON payload and exhaust Vector memory; send/decode each fetched batch or enforce a configured result limit instead of accumulating the entire result set.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T15:15:02Z

+                return Value::Null;
+            };
+
+            naive_local_to_timestamp_value(NaiveDateTime::new(NaiveDate::default(), time), tz, s)


Preserve TIME values when binding tracking params

When tracking_columns includes a SQL TIME column, this maps the time-only value into a Timestamp anchored to the default date, and the next poll formats timestamps as a full YYYY-MM-DD HH:MM:SS parameter. Databases such as PostgreSQL will compare a TIME placeholder against 1970-01-01 15:30:00 instead of 15:30:00, which can fail the query or advance tracking incorrectly; keep time-only values as their original text for checkpoint/bind purposes.

Useful? React with 👍 / 👎.

pront · 2026-06-17T19:08:05Z

Hello @powerumc, we are very interested in this PR. There a few merge conflicts and unresolved threads though. Ping us if you need any assistance.

powerumc added 29 commits July 24, 2025 20:53

feat(odbc source): add ODBC source for scheduled database queries

9c793e2

test(odbc source): Integration tests

9e3f5d7

test(odbc source): prepare Dockerfile

f383c9f

feat(odbc source): implement ODBC metrics for event tracking

50fac18

test(odbc source): integration test configuration

fa4502d

refactor(odbc source): introduce graceful shutdown and improve schedu…

3090a4e

…ling logic

chore(odbc source): clean up formatting

e13b1a1

test(odbc source): ODBC init_params, tracking_columns, last_run_metad…

f39ee28

…ata_path properties tests

fix(odbc source): fix data type handling and comments

831eb45

test(odbc source): number types integration test

64fda4e

test(odbc source): string types, other types integration test

e9037aa

chore(odbc source): clean up formatting

b88c42e

feat(odbc source): add support for PostgreSQL and update integration …

aef4174

…tests

fix(odbc source): refactor test setup for consistency

32a566d

test(odbc source): add integration test for timestamp columns

8af2a13

docs(odbc source): add documentation for ODBC data source configuration

be3e3ef

Merge branch 'master' into powerumc/odbc-source

5dae7a7

# Conflicts: # Cargo.lock # Cargo.toml # scripts/integration/Dockerfile # src/internal_events/mod.rs # src/sources/mod.rs

test(odbc source): add external network configuration for MySQL and P…

19cf895

…ostgreSQL services

docs(odbc source): update source code documents.

15d4436

docs(odbc source): add comprehensive CUE configuration and documentat…

1a3efcb

…ion for an ODBC source

chore(odbc source): source code formatting

40d5795

refactor(odbc source): executes ODBC query processing with result tra…

c74a12a

…cking

chore(deps): update odbc-api dependency to version 19.1.0

cea0073

chore(odbc source): update LICENSE-3rdparty.csv

bb62426

docs(odbc source): improve documentation clarity and update examples …

6af628d

…for MariaDB usage

Merge branch 'master' into powerumc/odbc-source

f4830ed

chore(odbc source): update fmt and clippy

3cb3d8f

chore(vdev): apply vdev rust check fixes

2a12364

chore(vdev): apply vdev rust check fixes

0b9c191

github-actions Bot added the domain: sources Anything related to the Vector's sources label Oct 22, 2025

powerumc added 7 commits June 15, 2026 20:47

refactor(odbc): enforce requirement for statement or statement_filepa…

8db9893

…th in OdbcConfig

refactor(odbc): update value mapping for Decimal and Numeric types to…

c21ef25

… preserve exact values

refactor(odbc): enforce requirement for non-empty connection string o…

50d86c6

…r file in OdbcConfig

refactor(odbc): improve connection string handling in OdbcConfig to r…

5c79859

…eturn Result type

feat(odbc): include byte size in OdbcEventsReceived for improved metr…

459c8cf

…ics tracking

fix(odbc): handle schedule exhaustion in ODBC source to ensure gracef…

61db439

…ul shutdown

refactor(odbc): enhance parameter handling in order_params and add va…

0f1336c

…lue_to_sql_parameter function for improved SQL binding

chatgpt-codex-connector Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread src/sources/odbc/config.rs Outdated

Comment thread src/internal_events/odbc_metrics.rs Outdated

Comment thread src/sources/odbc/client.rs

Comment thread src/sources/odbc/client.rs

Comment thread src/sources/odbc/config.rs

powerumc added 4 commits June 15, 2026 22:08

refactor(odbc): improve error handling for SQL statement retrieval in…

dd6dd94

… OdbcConfig and enforce non-empty statement requirement

fix(odbc): handle empty rows in decode_rows method to prevent unneces…

03506e0

…sary processing

fix(odbc): add truncation check in query execution to handle potentia…

31f9193

…l data loss

refactor(odbc): enhance error handling in OdbcFailedError to categori…

3a9e087

…ze errors and improve metric tracking

chatgpt-codex-connector Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread src/sources/odbc/client.rs

Comment thread src/sources/odbc/client.rs Outdated

Comment thread src/sources/odbc/client.rs Outdated

Comment thread src/sources/odbc/client.rs

refactor(odbc): add login_timeout configuration

2641994

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread src/sources/odbc/config.rs Outdated

Comment thread src/sources/odbc/client.rs Outdated

powerumc added 3 commits June 17, 2026 22:21

refactor(odbc): enhance datetime handling by introducing naive_local_…

7db485c

…to_timestamp_value function for improved DST and fallback management

docs(odbc): update OdbcConfig documentation to clarify timeout behavi…

cea7290

…or and allow indefinite wait

fix(odbc): enforce odbc_batch_size to be greater than 0 in configuration

c086681

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

powerumc added 3 commits June 17, 2026 22:50

fix(odbc): preserve out-of-range integer values as bytes

538fba6

refactor(odbc): add validation for tracking_columns

07cc01a

refactor(odbc): update max_str_limit handling to support optional val…

fbec868

…ues and improve documentation

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

pront added the meta: awaiting author Pull requests that are awaiting their author. label Jun 17, 2026

powerumc added 2 commits June 17, 2026 23:46

fix(odbc): format tracking timestamps in local timezone for SQL binds

80175bc

docs(odbc): clarify behavior of can_acknowledge method regarding row …

9b79856

…loss when last_run_metadata_path is set

github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jun 17, 2026

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

pront added the meta: awaiting author Pull requests that are awaiting their author. label Jun 17, 2026

Conversation

powerumc commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

pront commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

powerumc commented Oct 22, 2025 •

edited

Loading