Skip to content

Extend query config with query.masked config with Type.PASSWORD for secure query masking#1593

Merged
KGaneshDatta merged 22 commits into10.9.xfrom
kdatta_query_masked_config_patch1
Dec 3, 2025
Merged

Extend query config with query.masked config with Type.PASSWORD for secure query masking#1593
KGaneshDatta merged 22 commits into10.9.xfrom
kdatta_query_masked_config_patch1

Conversation

@KGaneshDatta
Copy link
Contributor

@KGaneshDatta KGaneshDatta commented Nov 12, 2025

Problem

  • Introduced an internal query.masked configuration of Type.Password to ensure secure logging by masking sensitive values and preventing accidental exposure.
  • Enhanced the existing validation to enforce that the connector operates exclusively in either table.mode or query.mode.
  • Fixed the breaking connector validations in query mode with the new tableFilteringOptions introduced in 10.8.5. Issue

Solution

Does this solution apply anywhere else?
  • yes
  • no
If yes, where?

Test Strategy

Testing done:
  • Unit tests
  • Integration tests
  • System tests
  • Manual tests

Release Plan

@KGaneshDatta KGaneshDatta requested a review from a team as a code owner November 12, 2025 11:07
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link
Member

@akanimesh7 Animesh Kumar (akanimesh7) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few things KGaneshDatta

  1. Since query was never exposed on cloud before this, we should perform some manual testing to understand/validate the behaviour. (How frequently will the query be executed, What if the query is taking too long to exceute ? do we have any observability into this ? Can multiple queries be provided (select * from table1;select* from table2;))
  2. If user has specified both table include/exclude list and query, we don't know which way they wanna go. The validation exception message currently instructs them to remove table include/exclude configs. Which might be other way round also.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing validation -- If mode is query and timestamp.initial is provided validation passes.

@KGaneshDatta
Copy link
Contributor Author

KGaneshDatta commented Dec 2, 2025

Missing validation -- If mode is query and timestamp.initial is provided validation passes.

The query mode would be working in timestamp criteria also. More details here - https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/overview.html#specifying-a-where-clause-with-query-modes
This also has been tested on CCloud with timestamp mode. Hence, when query was working in timestamp mode, I feel it would be valid to configure timestamp.initial when it is used.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trim.sensitive.log is a config which was already existing in the sink connectors.
Now we are using the same config in source connectors. Although we have defined it in the ConfigDef, but there is no way to set a value for this config. Please check this.

@sonarqube-confluent
Copy link

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks KGaneshDatta

@KGaneshDatta KGaneshDatta merged commit a9d25c3 into 10.9.x Dec 3, 2025
3 checks passed
@KGaneshDatta KGaneshDatta deleted the kdatta_query_masked_config_patch1 branch December 3, 2025 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants