Support for writing to existing Iceberg tables in S3 from C# service in ECS

[MPeli]

Hello .NET for Apache Spark team,

We're exploring the possibility of using .NET for Apache Spark to optimize our current data pipeline. Our scenario is as follows:

Current setup:

• C# service running in Amazon ECS writes data to an Amazon Kinesis stream
• AWS Glue job reads from this stream and writes to Iceberg tables in S3

Desired setup:

• C# service running in Amazon ECS writes directly to the same Iceberg tables in S3 that our Glue jobs are currently writing to, bypassing Kinesis and Glue

Our main question:

Is it possible to use .NET for Apache Spark within a C# service running in ECS to write directly to existing Iceberg tables in S3? If so, could you provide a high-level overview of how this might be implemented?

We appreciate any insights or documentation you can provide to help us evaluate the feasibility of this approach.

Thank you for your time and assistance!