Hive file write rollback permission failure causing S3 API call volume to EXPLODE? #27665
tc-melissa-bilinski
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Running Trino on ECS via Fargate, 1 coordinator and 3 workers 2vCPUs and 16GB RAM each. Just archiving from MariaDB to Glue/S3 via the Hive connector. Noticing that periodically, a file will fail to write, and per the logs the hive connector attempts a rollback, but the ecs role for Trino does not have delete permissions on Glue/S3 (we'd prefer it didn't). After about an hour of those failure notifications on worker logs, the coordinator goes nuclear and starts spamming API ListRequests to the bucket. We've seen an hour period of 40,000+ requests per minute before we caught and killed it. The first two days this went unnoticed were costly. If anyone has any intel or suggestions on how to handle this behavior I'd be eternally grateful.
Beta Was this translation helpful? Give feedback.
All reactions