SageMaker client `create_model_package` gives validation error when AdditionalS3DataSource is specified.

[shawnohare]

Describe the bug

The SDK documentation clearly states that an "AdditionalS3DataSource" is a valid parameter in an InferenceSpecification.Containers element object when calling the CreateModelPackage API. However, when attempting to specify this parameter in the input to a sagemaker client create_model_package method the following error is raised

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateModelPackage operation: AdditionalS3DataSource is not allowed in InferenceSpecification

If this is an AWS permissions issue related to my underlying work account I apologize in advance.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

That the AdditionalS3DataSource be accepted without a ValidationException.

Current Behavior

Assuming some dummy assets have been loaded into an s3 bucket, when specifying the AdditionalS3DataSource parameter we see the following error.

Traceback (most recent call last):
...
  File "/xxx/.venv/lib/python3.12/site-packages/botocore/client.py", line 570, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/.venv/lib/python3.12/site-packages/botocore/context.py", line 124, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/.venv/lib/python3.12/site-packages/botocore/client.py", line 1031, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateModelPackage operation: AdditionalS3DataSource is not allowed in InferenceSpecification

Reproduction Steps

The script below, with bucket name changed, is running in a work environment and so I cannot fully provide a complete SSCCE.

# create_model_package.py
"""Almost minimal example. Assumes the existence of some assets have previously
been pushed to s3, such as
- s3://{BUCKET}/model/model.tar.gz
- s3://{BUCKET}/model/additional/data.json

"""
import boto3
from botocore.exceptions import ClientError

MODEL_PACKAGE_GROUP_NAME = "my-model-group"
MODEL_PACKAGE_NAME = "my-model-package"
REGION = "us-west-2"
IMAGE_REPO: str = "huggingface-pytorch-inference"
IMAGE_TAG: str = "2.6.0-transformers4.49.0-cpu-py312-ubuntu22.04"
IMAGE_URI = f"763104351884.dkr.ecr.{REGION}.amazonaws.com/{IMAGE_REPO}:{IMAGE_TAG}"
BUCKET = "example-bucket"
S3_PREFIX = f"s3://{BUCKET}/model"
MODEL_TARBALL = f"{S3_PREFIX}/model.tar.gz"
ADDITIONAL_S3_DATA = f"{S3_PREFIX}/additional"

client = boto3.client("sagemaker", region_name=REGION)

# Create model package group
try:
    response = client.describe_model_package_group(ModelPackageGroupName=MODEL_PACKAGE_GROUP_NAME)
    model_group_arn = response["ModelPackageGroupArn"]
except ClientError:
    response = client.create_model_package_group(
        ModelPackageGroupName=MODEL_PACKAGE_GROUP_NAME,
    )
    model_group_arn = response["ModelPackageGroupArn"]


# Create Model Package
response = client.create_model_package(
    ModelPackageGroupName=MODEL_PACKAGE_GROUP_NAME,
    ModelPackageDescription="Model package with additional s3 data.",
    InferenceSpecification={
        "Containers": [
            {
                "Image": IMAGE_URI,
                "ModelDataSource": {
                    "S3DataSource": {
                        "S3Uri": MODEL_TARBALL,
                        "S3DataType": "S3Object",
                        "CompressionType": "Gzip",
                    }
                },
                "AdditionalS3DataSource": {
                    "S3Uri": ADDITIONAL_S3_DATA,
                    "S3DataType": "S3Prefix",
                    "CompressionType": "None"
                },
            }
        ],
        "SupportedContentTypes": ["application/json"],
        "SupportedResponseMIMETypes": ["application/json"]
    },
    CertifyForMarketplace=False
)

model_package_arn = response["ModelPackageArn"]
print(f"Registered Model Package: {model_package_arn}")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.37.37

Environment details (OS name and version, etc.)

macOS 15.4 (m1)

[RyanFitzSimmonsAK]

Hi @shawnohare, thanks for reaching out. I was able to reproduce this issue, and agree it looks like a bug. I've reached out to the service team for insights, and will let you know when I have any updates. Thanks!

Ticket # for internal use : P235884999

[RyanFitzSimmonsAK]

Thanks for your patience. The service team got back to me, and explained that AdditionalS3DataSource is not supported for both versioned and un-versioned model packages.

AdditionalS3DataSource is documented as an option for CreateModelPackage because it shares the same InferenceSpecification structure. This is a common nuance of AWS documentation. When a parameter is reused across multiple operations, the request and response syntax will show every possible value of that parameter. However, not every operation accepts or returns every possible value of that parameter, and you often need to check the parameter documentation, or reference an example, to get more information. In this case, the documentation is lacking, and the service team has plans to address that.

Thank you for pointing this out. Please let me know if you have any follow-up questions.

[shawnohare]

Thanks for your patience. The service team got back to me, and explained that AdditionalS3DataSource is not supported for both versioned and un-versioned model packages.

AdditionalS3DataSource is documented as an option for CreateModelPackage because it shares the same InferenceSpecification structure. This is a common nuance of AWS documentation. When a parameter is reused across multiple operations, the request and response syntax will show every possible value of that parameter. However, not every operation accepts or returns every possible value of that parameter, and you often need to check the parameter documentation, or reference an example, to get more information. In this case, the documentation is lacking, and the service team has plans to address that.

Thank you for pointing this out. Please let me know if you have any follow-up questions.

Thanks for the response and I appreciate your time spent on this. I suspected this nuance could be at play.

Are there any recommendations on source code to check so that one can infer exactly which fields of a parameter are supported by a particular endpoint? It seems there is some sort of source available parameter schema that is being validated against that is different from what advertised schema? Or is the only recourse for issues like this to hope that the documentation clarifies the difference?

Thanks again!

[RyanFitzSimmonsAK]

Are there any recommendations on source code to check so that one can infer exactly which fields of a parameter are supported by a particular endpoint?

Unfortunately, in this situation where documentation is misleading because of how structures are rendered, the request tends to hit server-side input validation, rather than client-side. You can distinguish them by whether the exception appears in the response body or not. There isn't a way to check the source code to determine if the parameter is actually supported.

It seems there is some sort of source available parameter schema that is being validated against that is different from what advertised schema?

Same answer as above; it's server-side validation.

Or is the only recourse for issues like this to hope that the documentation clarifies the difference?

Depending on the service and operation, examples might be help clarify it. Otherwise, you unfortunately have to hope that the documentation mentions it. This is definitely something I would like to improve about AWS documentation in the future.