feat: Add API and Connection timeout similar to OpenAI python SDK #2458

ranadeepsingh · 2025-12-16T03:54:13Z

What changes are proposed in this pull request?

Summary

Align SynapseML OpenAI connector timeout defaults with the OpenAI Python SDK and add user-configurable timeout parameters to prevent hanging operations and improve reliability for long-running requests.

Timeout Configuration Changes

Timeout Type	OpenAI SDK	SynapseML (before)	SynapseML (after)
API/Socket Timeout	600s (10 min)	360s (6 min)	600s (10 min)
Connection Timeout	5.0s	360s (same as socket)	5.0s
Operation Timeout	N/A	N/A	Configurable

New Parameters

Parameter	Type	Default	Description
`apiTimeout`	Double	600.0s	Per-request timeout waiting for API response
`connectionTimeout`	Double	5.0s	Timeout for establishing HTTP connection
`timeout`	Double	None	Global operation timeout - returns HTTP 408 for rows exceeding this limit

Key Features

Global Operation Timeout: When timeout is set, the entire DataFrame transformation will stop making new API calls after the specified duration, returning HTTP 408 (Request Timeout) errors for remaining rows. Useful for batch processing with time constraints.
Early Timeout Detection: Added early timeout check at partition start to avoid network errors (e.g., UnknownHostException) when using very short timeouts.

Configurable via OpenAIDefaults or Per-Transformer:

# Global defaults
from synapse.ml.services.openai.OpenAIDefaults import OpenAIDefaults
defaults = OpenAIDefaults()
defaults.set_api_timeout(600.0)
defaults.set_connection_timeout(5.0)
defaults.set_timeout(120.0)  # 2 minute operation timeout

# Or per-transformer
prompt = OpenAIPrompt()
    .setApiTimeout(300.0)
    .setConnectionTimeout(10.0)
    .setTimeout(60.0)

HTTP/1.1 Protocol Version: Fixed synthetic responses (e.g., timeout errors) to include proper HTTP/1.1 protocol version instead of NULL.

Files Changed

Core HTTP Layer:

HTTPTransformer.scala - Added three timeout parameters, broadcast start time for distributed timeout tracking, early timeout check
HTTPSchema.scala - Fixed stringToResponse, emptyResponse, binaryToResponse to use ProtocolVersionData("HTTP", 1, 1)
SimpleHTTPTransformer.scala - Pass timeout parameter to HTTPTransformer
Clients.scala - Added precomputedResponse field to RequestWithContext
HTTPClients.scala - Handle precomputed responses for timed-out requests

OpenAI Layer:

OpenAI.scala - Added OpenAITimeoutKey, OpenAIApiTimeoutKey, OpenAIConnectionTimeoutKey
OpenAIDefaults.scala - Added setters/getters/resetters for all three timeouts with validation
OpenAIDefaults.py - Added Python bindings for timeout methods
OpenAIPrompt.scala - Register timeout parameter with GlobalParams
CognitiveServiceBase.scala - Pass timeout parameters to SimpleHTTPTransformer

Tests:

OpenAIPromptSuite.scala - Added tests for long input handling, timeout configuration, short timeout behavior
OpenAIDefaultsSuite.scala - Added tests for timeout getters, resetters, and validation
test_OpenAIDefaults.py - Added Python tests for timeout setters/getters, resetters, and validation

Documentation:

OpenAI.ipynb - Added "Configuring Timeouts" section with examples

How is this patch tested?

I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Test Coverage

Scala Tests (OpenAIPromptSuite):

Long Input Handling - Tests behavior with normal, null, and very long (1000x, 10000x repetition) inputs
Timeout Configuration - Validates all three timeout parameters are properly set and retrieved
Short Timeout Returns Timeout Error - Validates HTTP 408 responses for timed-out operations

Scala Tests (OpenAIDefaultsSuite):

Test Getters - Validates all timeout getters return correct values
Test Resetters - Validates all timeout resetters clear values properly
Test Parameter Validation - Validates timeout values must be > 0

Python Tests (test_OpenAIDefaults.py):

test_setters_and_getters - Tests timeout set/get operations
test_resetters - Tests timeout reset operations
test_parameter_validation - Tests timeout validation (must be > 0)

Does this PR change any dependencies?

No. You can skip this section.

Does this PR add a new feature? If so, have you added samples on website?

Yes. Make sure you have added samples following below steps.

Documentation added to OpenAI.ipynb notebook with:

Explanation of all three timeout parameters
Code examples using OpenAIDefaults for global configuration
Code examples for per-transformer timeout configuration

github-actions · 2025-12-16T03:54:22Z

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

ranadeepsingh · 2025-12-16T04:46:40Z

/azp run

azure-pipelines · 2025-12-16T04:46:50Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov-commenter · 2025-12-16T05:01:22Z

Codecov Report

❌ Patch coverage is 93.84615% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.34%. Comparing base (bcf251c) to head (a05b321).

Files with missing lines	Patch %	Lines
...m/microsoft/azure/synapse/ml/io/http/Clients.scala	0.00%	2 Missing ⚠️
...crosoft/azure/synapse/ml/io/http/HTTPClients.scala	75.00%	1 Missing ⚠️
...icrosoft/azure/synapse/ml/io/http/HTTPSchema.scala	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2458      +/-   ##
==========================================
- Coverage   84.36%   84.34%   -0.02%     
==========================================
  Files         335      335              
  Lines       17734    17777      +43     
  Branches     1602     1633      +31     
==========================================
+ Hits        14961    14994      +33     
- Misses       2773     2783      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ranadeepsingh · 2025-12-16T09:57:59Z

/azp run

azure-pipelines · 2025-12-16T09:58:11Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh · 2025-12-16T10:10:27Z

/azp run

azure-pipelines · 2025-12-16T10:10:40Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh · 2025-12-16T10:58:20Z

/azp run

azure-pipelines · 2025-12-16T10:58:33Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR adds configurable timeout parameters to the SynapseML OpenAI connector to align with the OpenAI Python SDK defaults and improve reliability for long-running requests. The changes introduce three distinct timeout parameters: apiTimeout (per-request timeout for API responses), connectionTimeout (timeout for establishing HTTP connections), and timeout (global operation timeout for the entire DataFrame transformation).

Key changes:

Updated default timeout values: API timeout increased from 360s to 600s (10 minutes), connection timeout separated from API timeout and set to 5.0s to match OpenAI SDK
Added global operation timeout feature that stops processing new rows after a specified duration, returning HTTP 408 responses for remaining rows
Fixed synthetic HTTP responses to include proper HTTP/1.1 protocol version instead of NULL

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
HTTPTransformer.scala	Added three timeout parameters (apiTimeout, connectionTimeout, timeout), implemented global timeout tracking with broadcast variables, and added early timeout checks
HTTPSchema.scala	Fixed protocol version in stringToResponse, emptyResponse, and binaryToResponse to use HTTP/1.1 instead of null
SimpleHTTPTransformer.scala	Updated to pass all three timeout parameters to HTTPTransformer
Clients.scala	Added precomputedResponse field to RequestWithContext for handling timeout responses without making HTTP calls
HTTPClients.scala	Modified sendRequestWithContext to handle precomputed responses and separated connection timeout from API timeout
OpenAI.scala	Added global parameter keys for the three timeout types
OpenAIDefaults.scala	Added setters, getters, and resetters for timeout parameters with validation (must be > 0)
OpenAIDefaults.py	Added Python bindings for timeout methods with client-side validation
OpenAIPrompt.scala	Registered timeout parameters with GlobalParams and changed default from timeout to apiTimeout (600.0s)
CognitiveServiceBase.scala	Modified to pass timeout parameters to SimpleHTTPTransformer
OpenAIPromptSuite.scala	Added tests for long input handling, timeout configuration, and short timeout behavior
OpenAIDefaultsSuite.scala	Added tests for timeout getters, resetters, and validation
test_OpenAIDefaults.py	Added Python tests for timeout setters, getters, resetters, and validation
HTTPTransformerSuite.scala	Added tests for default timeout values, custom timeout values, and global timeout behavior
SimpleHTTPTransformerSuite.scala	Updated test to use setApiTimeout instead of setTimeout

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

core/src/test/scala/com/microsoft/azure/synapse/ml/io/split1/HTTPTransformerSuite.scala

core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala

…PTransformer.scala Co-authored-by: Copilot <[email protected]>

ranadeepsingh · 2025-12-18T04:56:45Z

/azp run

azure-pipelines · 2025-12-18T04:56:57Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh · 2025-12-18T04:57:14Z

/azp run

azure-pipelines · 2025-12-18T04:57:26Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh · 2025-12-18T05:35:10Z

/azp run

azure-pipelines · 2025-12-18T05:35:21Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh · 2025-12-18T10:27:25Z

/azp run

azure-pipelines · 2025-12-18T10:27:52Z

Azure Pipelines successfully started running 1 pipeline(s).

Add API timeout and Connection tmeout similar to OpenAI python SDK

57825d3

Push timeout changes

1f0b28d

Fix black python formatting

817851c

Update to Long input test

01c7238

ranadeepsingh requested review from BrendanWalsh, Copilot and levscaut December 18, 2025 03:44

Copilot started reviewing on behalf of ranadeepsingh December 18, 2025 03:44 View session

Copilot AI reviewed Dec 18, 2025

View reviewed changes

core/src/test/scala/com/microsoft/azure/synapse/ml/io/split1/HTTPTransformerSuite.scala Show resolved Hide resolved

core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala Outdated Show resolved Hide resolved

Update core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTT…

48617a9

…PTransformer.scala Co-authored-by: Copilot <[email protected]>

Merge branch 'master' into rana/updated-openai-timeouts

01559c4

Fix scala style issue

27ab5e1

ranadeepsingh added 2 commits December 18, 2025 02:14

Add timeouts to OpenAIEmbedding

25c0453

Add tests to OpenAIEmbeddignSuite and renamed it

a05b321

feat: Add API and Connection timeout similar to OpenAI python SDK #2458

Are you sure you want to change the base?

feat: Add API and Connection timeout similar to OpenAI python SDK #2458

Uh oh!

Conversation

ranadeepsingh commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

Summary

Timeout Configuration Changes

New Parameters

Key Features

Files Changed

How is this patch tested?

Test Coverage

Does this PR change any dependencies?

Does this PR add a new feature? If so, have you added samples on website?

Uh oh!

github-actions bot commented Dec 16, 2025

Uh oh!

ranadeepsingh commented Dec 16, 2025

Uh oh!

azure-pipelines bot commented Dec 16, 2025

Uh oh!

codecov-commenter commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ranadeepsingh commented Dec 16, 2025

Uh oh!

azure-pipelines bot commented Dec 16, 2025

Uh oh!

ranadeepsingh commented Dec 16, 2025

Uh oh!

azure-pipelines bot commented Dec 16, 2025

Uh oh!

ranadeepsingh commented Dec 16, 2025

Uh oh!

azure-pipelines bot commented Dec 16, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ranadeepsingh commented Dec 18, 2025

Uh oh!

azure-pipelines bot commented Dec 18, 2025

Uh oh!

ranadeepsingh commented Dec 18, 2025

Uh oh!

azure-pipelines bot commented Dec 18, 2025

Uh oh!

ranadeepsingh commented Dec 18, 2025

Uh oh!

azure-pipelines bot commented Dec 18, 2025

Uh oh!

ranadeepsingh commented Dec 18, 2025

Uh oh!

azure-pipelines bot commented Dec 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ranadeepsingh commented Dec 16, 2025 •

edited

Loading

codecov-commenter commented Dec 16, 2025 •

edited

Loading