fix(redshift): use boundary-aware segment stitching for query reconstruction

[kyungsoo-datahub]

Redshift stores queries in fixed-width segments (200 chars for provisioned, 4000 for serverless). When using LISTAGG with per-segment RTRIM, tokens at boundaries merge without spaces (GROUP BY → GROUPBY).

Fix: Insert a space when trimmed segment length is less than segment size (indicates padding). Applied to all 6 LISTAGG expressions in both modes.

Also: Replace stl_query.querytxt (truncated to 4000 chars) with STL_QUERYTEXT CTE in provisioned scan-based lineage.

Added tests for boundary detection and segment size correctness.

[github-actions]

Linear: ING-1654

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[github-actions]

Linear: ING-1802

[rajatoss]

Connector Tests Results

Connector tests failed for commit e27b075

View full test logs →

Autogenerated by the connector-tests CI pipeline.

[sgomezvillamor]

My overall concern is that we’re adding quite a lot of complexity (and therefore risk of errors) to address what seems to be an anecdotal issue. How recurrent is this issue?

Beyond complexity, proposed solution is not perfect.

The keyword-adjacency heuristic is fragile. We’d need a comprehensive reserved word list, and edge cases will grow quickly. For example:

-- GROUP as a keyword — needs space
SELECT a, COUNT(*) FROM t GROUP<CHR1>BY a

-- GROUP as an identifier — space is harmless but detection logic
-- must correctly classify it, which requires parsing context
SELECT group<CHR1>_id FROM t   -- identifier, no space needed

This would require bidirectional lookahead around every marker, which increases complexity even more.

The reserved word list itself is also a maintenance burden. Redshift’s reserved words may change across versions, and include non-obvious entries. Keeping that list correct and up to date is operational overhead.

Additionally, there’s the risk of false positives from string literals. CHR(1) is the SOH (Start of Heading) ASCII control character. It’s exotic, but Redshift allows it inside string literals. Very unlikely, but still technically possible.

---

We currently read from SQL_QUERYTEXT (or SQL_QUERY_TEXT in serverless).

There is also SVL_STATEMENTTEXT: https://docs.aws.amazon.com/redshift/latest/dg/r_SVL_STATEMENTTEXT.html

That view is built on top of SQL_QUERYTEXT. I’m not sure if it’s available in serverless. The docs describe how to reconstruct statements there.

Does SVL_STATEMENTTEXT suffer from the same reconstruction issues as SQL_QUERYTEXT? If not, maybe we could switch to it.

If it’s not supported in serverless (given the 4K char limit there), is this issue still frequent enough that we must address it?

---

Another alternative would be audit logging to S3: https://docs.aws.amazon.com/redshift/latest/mgmt/db-auditing.html

From what I understand, the query statement is not split there. But this would be a completely different ingestion approach.

I agree this may not feasible in the short term.

---

My current view:

• The current proposal adds significant complexity and is still not a perfect solution.
• Can we consider a simpler, more naive concat? Not perfect either, but much less complex.
• And should we evaluate SVL_STATEMENTTEXT more seriously first?

IMO final decision depends on how recurrent is the issue.

[alwaysmeticulous]

🔴 Meticulous spotted visual differences in 1 of 1373 screens tested: view and approve differences detected.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Last updated for commit d156355. This comment will update as new commits are pushed.}

[sgomezvillamor]

LGTM
Just wondering how is the implementation in PR is slightly different to the one suggested here

https://docs.aws.amazon.com/redshift/latest/dg/r_SVL_STATEMENTTEXT.html

select LISTAGG(CASE WHEN LEN(RTRIM(text)) = 0 THEN text ELSE RTRIM(text) END, '') within group (order by sequence) AS query_statement 
from SVL_STATEMENTTEXT where pid=pg_backend_pid();

[kyungsoo-datahub]

AWS's approach doesn't fix the keyword merge problem. Their condition LEN(RTRIM(text)) = 0 only preserves completely empty segments. A segment like "GROUP " becomes "GROUP" (length 5, not 0), which merges with the next "BY" segment -> "GROUPBY"

Our approach uses segment size as a boundary detector: if LEN(RTRIM(text)) < segment_size, we add a space back because the segment likely reached a natural boundary (not just had trailing spaces stripped).

[sgomezvillamor]

Basically, we are just doing string contains validations

So I would remove the focus on "stitching" and make this just a generic test for redshift queries

• test_redshift_queries.py
• class TestProvisionedQueries
• class TestServerlessQueries
• ...

So we can add more validations in the future

[kyungsoo-datahub]

Revised.

[sgomezvillamor]

I don't see much value on these tests using _simulate_boundary_aware_stitching. We could just remove them.

If we really want to test this in an integration test, we could add a eg long enough select in https://github.com/acryldata/connector-tests/blob/main/smoke-test/integration/create_data/redshift_data.py

[kyungsoo-datahub]

Removed.

[sgomezvillamor]

LGTM