[Glue] Support OpenTableFormatInput for Iceberg table creation

[pomykalakyle]

Feature Request

Description

The AWS Glue CreateTable API supports an optional OpenTableFormatInput parameter for creating Apache Iceberg tables. This parameter is currently ignored by moto.

AWS Documentation

• CreateTable API - shows OpenTableFormatInput as an optional parameter
• OpenTableFormatInput - structure definition

Current Implementation

In moto/glue/responses.py, create_table only extracts DatabaseName and TableInput:

def create_table(self) -> str:
    database_name = self.parameters.get("DatabaseName")
    table_input = self.parameters.get("TableInput")
    table_name = table_input.get("Name")
    self.glue_backend.create_table(database_name, table_name, table_input)
    return ""

The OpenTableFormatInput parameter is never read from the request.

How to Reproduce

import boto3
from moto import mock_aws

@mock_aws
def test_create_iceberg_table():
    client = boto3.client("glue", region_name="us-east-1")
    
    client.create_database(DatabaseInput={"Name": "test_db"})
    
    # OpenTableFormatInput is silently ignored
    client.create_table(
        DatabaseName="test_db",
        TableInput={"Name": "my_table"},
        OpenTableFormatInput={
            "IcebergInput": {
                "MetadataOperation": "CREATE",
                "Version": "2"
            }
        }
    )
    
    # Table is created as a regular Hive table, not an Iceberg table

Related

• UpdateTable also supports OpenTableFormatInput (same gap in responses.py)
• The IcebergInput structure includes partition specs, schemas, and sort orders

[bblommers]

Hi @pomykalakyle! Is the OpenTableFormatInput actually exposed anywhere in the AWS API? I'm not seeing that parameter come back in any other operations (like get_table, where I would expect it).

Probably a related note: Moto (with a few exceptions) just mocks the AWS API, it doesn't actually expose a Iceberg table that you can connect to. The linked PR seems to try and execute SQL queries, and that will not work against Moto.

[linhr]

From what I've seen, when OpenTableFormatInput is used to create the table, the response for get_table would have information specific to Iceberg. (The columns in storage descriptor would have parameters such as iceberg.field.id. And the table parameter would have table_type of value iceberg.) Indeed, get_table does not return OpenTableFormatInput verbatim, so some knowledge of the Iceberg spec may be needed for creating the mock response.

I'm not sure about the efforts if we want to mock the get_table response with high fidelity. But Moto doesn't need to create an Iceberg table. As long as the table metadata is mocked in certain ways, the SQL engine would be able to use such information to work with the actual Iceberg table (e.g., a table stored in a mocked S3 service). It seems that Glue also creates Iceberg manifest files in the storage system (e.g., S3) according to the table location specified by OpenTableFormatInput. So this is more than simply mocking the response.

cc @pomykalakyle