Composite and multi-dim Array Type Support for PG Replication Client

[peterxcli]

Summary

Currently, some pg type for example like Composite and Array Composite will fail cause:

moonlink/src/moonlink_connectors/src/pg_replicate/clients/postgres.rs

Lines 261 to 268 in db080d5

	// Fail fast on any type that we are not able to parse in try_from_str.
	let typ = Type::from_oid(type_oid).ok_or_else(|| {
	ReplicationClientError::UnsupportedType(
	name.clone(),
	type_oid,
	table_name.to_string(),
	)
	})?;

---

In addition, the multi-dim Array is also rejected cause the check:

moonlink/src/moonlink_connectors/src/pg_replicate/clients/postgres.rs

Lines 287 to 302 in db080d5

	let attndims: i32 = row
	.try_get("attndims")?
	.ok_or(ReplicationClientError::MissingColumn(
	"attndims".to_string(),
	"pg_attribute".to_string(),
	))?
	.parse()
	.map_err(|_| ReplicationClientError::TypeModifierColumnNotI32)?;
	if matches!(typ.kind(), Kind::Array(_)) && attndims > 1 {
	return Err(ReplicationClientError::UnsupportedType(
	name.clone(),
	type_oid,
	table_name.to_string(),
	));
	}

Context

Why is this issue important? Link any relevant discussions, designs, or documents.

Tasks

• Composite support
• Array of Composite support
• Nested Array in Composite's field
• Nested Composite
• Array of Array support (multi-dim array)

Notes

I got a draft patch which aims to add integration testing of pg row parsing and try to make PG Replication Client to support Composite and Array of Composite:
https://github.com/Mooncake-Labs/moonlink/compare/main...peterxcli:feat/support-customize-type-in-pg-replication-client?expand=1

Remember to follow the type matching rule introduced in #1301 or its following...

[peterxcli]

cc @dentiny, would you like to check if this makes sense? TIA!

[dentiny]

cc @dentiny, would you like to check if this makes sense? TIA!

Yes it's intentional, due to the lack of support at moonlink side;
once composite type and array type fully supported we should remove the check.

[dpxcc]

Note that pg array behavior is sometimes weird:

1. Limited multidimensional array type support duckdb/pg_duckdb#39

Postgres Arrays do not enforce the dimensionality of the schema and the values to match.
What this means is that a Postgres Array column can be marked as having the schema BIGINT[] and contains 2+

2. Handle PG columns with arrays with 0 dimensions correctly duckdb/pg_duckdb#616

As was found in duckdb/pg_duckdb#556 sometimes array columns in Postgres are listed to have 0 dimensions, that's obviously because of missing information.

Make sure we don't crash in these edge cases

[dentiny]

@dpxcc can we have regression tests on type at pgmooncake / moonlink side?

[dpxcc]

yea, prob in moonlink pg connector

[peterxcli]

Note that pg array behavior is sometimes weird:

1. Limited multidimensional array type support duckdb/pg_duckdb#39

Postgres Arrays do not enforce the dimensionality of the schema and the values to match.
What this means is that a Postgres Array column can be marked as having the schema BIGINT[] and contains 2+

2. Handle PG columns with arrays with 0 dimensions correctly duckdb/pg_duckdb#616

As was found in duckdb/pg_duckdb#556 sometimes array columns in Postgres are listed to have 0 dimensions, that's obviously because of missing information.

Make sure we don't crash in these edge cases

@dpxcc thanks for the info, let's move the multi-dim array support to #1462.