fix: correct Tag class usage in pipeline creation

[RamprasathS]

Fix: Pipeline creation fails with Pydantic validation error due to incorrect Tag class

Issue Description

When BenchmarkEvaluator attempts to create a new SageMaker Pipeline (when no existing pipeline is found), it fails with a Pydantic validation error. This prevents MMLU_PRO and other benchmark evaluations from running when the pipeline doesn't already exist.

Error Message

4 validation errors for Pipeline.create
tags.0.key - Field required
tags.0.value - Field required  
tags.0.Key - Extra inputs are not permitted
tags.0.Value - Extra inputs are not permitted

Impact

• First-time benchmark evaluations fail completely
• Any scenario requiring new pipeline creation fails
• Works only when pipeline already exists (reuse path)
• Affects all evaluation types (BENCHMARK, CUSTOM_SCORER, LLM_AS_JUDGE)

Root Cause

The code in sagemaker-train/src/sagemaker/train/evaluate/execution.py has two issues:

1. Wrong Tag class imported: Imports Tag from sagemaker.core.resources, but Pipeline.create() expects Tag from sagemaker.core.shapes

2. Wrong tag format: Creates dict {"key": ..., "value": ...} instead of Tag objects

There are two different Tag classes in the SDK:

• sagemaker.core.resources.Tag - Used for Tag.get_all() to retrieve tags
• sagemaker.core.shapes.Tag - Used for Pipeline.create() to create resources

Changes Made

1. Fixed imports (lines 16-24)

# Before
from sagemaker.core.resources import Pipeline, PipelineExecution, Tag

# After  
from sagemaker.core.resources import Pipeline, PipelineExecution
from sagemaker.core.resources import Tag as ResourceTag  # For Tag.get_all()
from sagemaker.core.shapes import Tag  # For Pipeline.create() tags parameter

2. Fixed tag creation (lines 66-99)

• Create proper Tag objects from sagemaker.core.shapes
• Handle both dict and Tag inputs gracefully
• Add proper error handling and logging
• Maintain backward compatibility

3. Updated Tag.get_all() calls

• Changed Tag.get_all() to ResourceTag.get_all() (2 occurrences)
• Lines ~235 and ~677

Testing

Before Fix

INFO Creating new pipeline: SagemakerEvaluation-BenchmarkEvaluation-...
ERROR 4 validation errors for Pipeline.create
✓ Evaluation job started!
   Job ARN: None  ← FAILED

After Fix

INFO Creating new pipeline: SagemakerEvaluation-BenchmarkEvaluation-...
INFO Creating pipeline with 2 tags
INFO Successfully created pipeline
INFO Pipeline is now active and ready for execution
✓ Evaluation job started!
   Job ARN: arn:aws:sagemaker:us-east-1:...  ← SUCCESS

Tested both scenarios:

• ✅ Creating new pipeline (first run)
• ✅ Reusing existing pipeline (subsequent runs)

Related Issues

This fixes the pipeline creation failure when running benchmark evaluations for the first time or after pipeline deletion.

Contributors

• @RamprasathS
• @dsuarezsoutoaws

[RamprasathS]

no, Modeltrainer accepts dicts, but BenchmarkEvaluator.Pipeline.create() accepts Tag objects

…

On Wed, Feb 4, 2026 at 8:28 AM Adam Ivison ***@***.***> wrote: *admivsn* left a comment (aws/sagemaker-python-sdk#5526) <#5526 (comment)> Is there also a similar issue with ModelTrainer tags? — Reply to this email directly, view it on GitHub <#5526 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVMWDJYPANTU463WUXQ3D34KHXYPAVCNFSM6AAAAACTN5U54WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQNBXGQ2TOOBXHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>