Skip to content

Conversation

@siiddhantt
Copy link
Contributor

@siiddhantt siiddhantt commented Oct 3, 2025

Description

This PR implements JDBC URL parameters and SSL/TLS configuration support for the MySQL connector, addressing the need for more flexible and secure database connections.

Changes:

  • Extended Config struct with JDBCURLParams (map[string]string) for custom connection parameters
  • Added SSLConfiguration field using existing utils.SSLConfig for certificate-based SSL/TLS
  • Updated URI() method to dynamically apply JDBC parameters and SSL settings to connection string
  • Added registerTLSConfig() helper method for loading and registering custom TLS certificates
  • Updated Validate() to check SSL configuration when provided
  • Enhanced spec.json with new optional jdbc_url_params and ssl fields
  • Updated README.md with examples and documentation
  • Maintained full backward compatibility with existing tls_skip_verify field

Fixes #378

Type of change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Unit Tests:

  • All 5 unit tests pass (TestConfig_URI_WithJDBCParams, TestConfig_URI_WithSSLDisabled, TestConfig_URI_WithSSLRequired, TestConfig_Validate_WithSSLConfig, TestConfig_URI_CombinedParams)

Integration Tests with Real MySQL Database:

  • Connection with JDBC parameters (charset=utf8mb4, parseTime=true) - SUCCEEDED
  • Connection with SSL config (mode: disable) - SUCCEEDED
  • Connection with combined JDBC params + SSL config - SUCCEEDED
  • Backward compatibility with legacy tls_skip_verify - SUCCEEDED

Code Quality:

  • Build successful (no compilation errors)
  • go vet passed (no issues)
  • Code properly formatted

Test Configuration Examples:

{
  "hosts": "localhost",
  "username": "mysql",
  "password": "secret",
  "database": "test_db",
  "port": 3306,
  "jdbc_url_params": {
    "charset": "utf8mb4",
    "parseTime": "true",
    "timeout": "10s"
  },
  "ssl": {
    "mode": "disable"
  }
}

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

@CLAassistant
Copy link

CLAassistant commented Oct 3, 2025

CLA assistant check
All committers have signed the CLA.

@vaibhav-datazip
Copy link
Collaborator

Hi @siiddhantt please rebase your PR to staging branch

- Add JDBCURLParams map for custom connection parameters
- Add SSLConfiguration field for certificate-based SSL/TLS
- Update URI() to dynamically apply JDBC params and SSL settings
- Add registerTLSConfig() for certificate handling
- Add comprehensive unit tests (7 tests, all passing)
- Update spec.json schema and README documentation
- Maintain backward compatibility with tls_skip_verify
@siiddhantt siiddhantt force-pushed the feat/JDBC-url-params branch from f386d3b to 3afa192 Compare October 5, 2025 00:33
@siiddhantt siiddhantt changed the base branch from master to staging October 5, 2025 00:39
@siiddhantt
Copy link
Contributor Author

Hi @vaibhav-datazip I've rebased it to the staging branch

- Remove redundant SSL nil check in buildTLSConfig()
- Change certificate handling from file paths to PEM string content
- Use maps.Copy() for efficient JDBC params copying
- Fix TLS config registration: buildTLSConfig() returns *tls.Config, registered inline
- Remove TLSSkipVerify field completely
- Simplify README examples per feedback
- Update spec.json descriptions for certificate fields
@vaibhav-datazip vaibhav-datazip added the hacktoberfest Issues open for Hacktoberfest contributors label Oct 7, 2025
@vaibhav-datazip
Copy link
Collaborator

LGTM

@hash-data
Copy link
Collaborator

Hi @siiddhantt, can you share the test list that you have done on this PR with MySQL?

@siiddhantt
Copy link
Contributor Author

siiddhantt commented Oct 28, 2025

Hi @hash-data, I've verified the implementation with comprehensive testing:

Unit Tests (5): All JDBC parameter and SSL config tests pass
Connection Tests (4): Real MySQL connections succeed with:

  • JDBC parameters only (charset=utf8mb4, parseTime=true)
  • SSL config (mode: disable/require)
  • Combined JDBC params + SSL config

All configurations generate correct DSN strings and connect successfully to MySQL 8.0.
If needed I could add those connection tests that would require MySQL to be running locally to verify!

@hash-data
Copy link
Collaborator

hash-data commented Oct 30, 2025

Hi @siiddhantt, once check the security CI, it is failing

@siiddhantt
Copy link
Contributor Author

Hi @siiddhantt, once check the security CI, it is failing

Hi @hash-data it should be resolved now, thanks!

@ImDoubD-datazip
Copy link
Collaborator

Hi @siiddhantt , Basically 3 things are required from your side.

  • Please provide the exact steps of how you have tested the SSL support.
  • Also provide the steps to reproduce the SSL configuration for the MYSQL source.
  • If possible, please provide an end-to-end video of you testing the SSL configuration as well.

@Akshay-datazip
Copy link
Collaborator

Hi @siiddhantt feel free to check our slack and in contributing-to-olake channel drop in your pr if you are facing any issues our devs would love to take the things from there too .
thanks
https://join.slack.com/t/getolake/shared_invite/zt-2uyphqf69-KQxih9Gwd4GCQRD_XFcuyw

@siiddhantt
Copy link
Contributor Author

Hi @ImDoubD-datazip and @Akshay-datazip , sure I'll provide those three and take a look into slack for updates regarding the PR. Thanks!

@siiddhantt
Copy link
Contributor Author

siiddhantt commented Oct 30, 2025

Hi @ImDoubD-datazip,

Here's exactly what I did to test the connection and how you can reproduce it:

What I Tested

I ran two types of tests to verify the JDBC parameters and SSL configuration work correctly:

  1. Unit Tests - Verify URI generation and configuration validation
  2. Integration Tests - Verify actual connections to MySQL with these configurations

Configuration 1: SSL Disabled

{
  "hosts": "localhost",
  "username": "mysql",
  "password": "secret1234",
  "database": "olake_mysql_test",
  "port": 3306,
  "ssl": {
    "mode": "disable"
  }
}

Configuration 2: JDBC Parameters Only

{
  "hosts": "localhost",
  "username": "mysql",
  "password": "secret1234",
  "database": "olake_mysql_test",
  "port": 3306,
  "jdbc_url_params": {
    "charset": "utf8mb4",
    "parseTime": "true"
  }
}

Configuration 3: Combined (JDBC + SSL)

{
  "hosts": "localhost",
  "username": "mysql",
  "password": "secret1234",
  "database": "olake_mysql_test",
  "port": 3306,
  "jdbc_url_params": {
    "charset": "utf8mb4",
    "parseTime": "true",
    "timeout": "10s"
  },
  "ssl": {
    "mode": "disable"
  }
}

How to Run the Tests Yourself

If you want to verify everything locally, here are the exact steps:

  • Start/Verify MySQL container running
/Applications/Docker.app/Contents/Resources/bin/docker ps | grep mysql
  • Run the Unit Tests
cd drivers/mysql
go test ./internal -run "^TestConfig" -v
  • Run the Integration Tests
go test ./internal -run "^TestConnection" -v

UPDATE

I've added comprehensive SSL/TLS testing for the MySQL connector. Here's what I did:

Generated test certificates using OpenSSL and stored them in ssl-certs (CA cert, server cert, client cert). Then spun up an SSL-enforced MySQL container on port 3307 using docker.

What I Tested :

Ran the full test suite with go test ./internal -run "^TestConnection" -v and covered:

  • JDBC URL parameters working correctly
  • All SSL modes (disable, verify-ca, verify-full)
  • Certificate-based authentication
  • Combined SSL + JDBC params
  • Invalid certificate handling (fallback behavior)
  • Config validation

I've attached the connection_test.go file in the Gist for your reference:
https://gist.github.com/siiddhantt/ecc4cc7ec41174610c32d0c9dae0da4a

@vaibhav-datazip
Copy link
Collaborator

@siiddhantt , will be reviewing soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hacktoberfest Issues open for Hacktoberfest contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add JDBC URL params and SSL config support to MySQL connector

6 participants