Add WithHttpClient option support to azkustoingest package #312

Copilot · 2025-07-13T10:36:03Z

This PR adds support for overriding the HTTP client in the azkustoingest package to match the functionality already available in azkustodata. This enables users to add instrumentation like OpenTelemetry by providing a custom HTTP client with an instrumented transport.

Problem

Previously, while azkustodata supported the WithHttpClient option:

client, err := azkustodata.New(kcsb, azkustodata.WithHttpClient(customClient))

The azkustoingest package did not support this option, making it impossible to instrument ingest clients for observability.

Solution

Added WithHttpClient option support for all three ingest client types:

Queued Ingestion

client, err := azkustoingest.New(kcsb, azkustoingest.WithHttpClient(customClient))
httpClient := client.HttpClient() // Access the HTTP client

Streaming Ingestion

client, err := azkustoingest.NewStreaming(kcsb, azkustoingest.WithHttpClient(customClient))
httpClient := client.HttpClient() // Access the HTTP client

Managed Ingestion

client, err := azkustoingest.NewManaged(kcsb, azkustoingest.WithHttpClient(customClient))
queuedClient := client.QueuedHttpClient()     // Access queued HTTP client
streamingClient := client.StreamingHttpClient() // Access streaming HTTP client

Changes Made

Added httpClient field to Ingestion struct
Added WithHttpClient(client *http.Client) Option function in ingestion_options.go
Modified New(), NewStreaming(), and NewManaged() to pass HTTP client option to underlying azkustodata.New() calls
Added HttpClient() methods to Ingestion and Streaming structs for public access
Added QueuedHttpClient() and StreamingHttpClient() methods to Managed struct
Added comprehensive tests covering all client types
Updated imports to include net/http where needed

Testing

All existing tests continue to pass, and new tests verify the functionality works correctly for all three client types. The implementation maintains full backward compatibility.

Fixes #311.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

statusreturner.azurewebsites.net
- Triggering command: /tmp/go-build2116896783/b316/etoe.test -test.testlogfile=/tmp/go-build2116896783/b316/testlog.txt -test.paniconexit0 -test.short=true -test.timeout=30s (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: AsafMah <[email protected]>

AsafMah · 2025-07-13T11:17:37Z

Can you update the changelog file?

Co-authored-by: AsafMah <[email protected]>

Copilot · 2025-07-13T11:19:32Z

Can you update the changelog file?

Updated the CHANGELOG.md file with the new WithHttpClient feature in commit 450b317.

github-actions · 2025-07-13T11:23:51Z

Test Results

0 tests ±0 0 ✅ ±0 0s ⏱️ ±0s
0 suites ±0 0 💤 ±0
0 files ±0 0 ❌ ±0

Results for commit 90d278b. ± Comparison against base commit 7e84f1b.

♻️ This comment has been updated with latest results.

AsafMah · 2025-07-13T11:28:42Z

Fix the tests

AsafMah · 2025-07-13T11:36:02Z

Subject: [PATCH] Rename

Index: azkustodata/example_iterative_query_test.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8

diff --git a/azkustodata/example_iterative_query_test.go b/azkustodata/example_iterative_query_test.go
--- a/azkustodata/example_iterative_query_test.go (revision 450b317)
+++ b/azkustodata/example_iterative_query_test.go (revision d1e90e2e349928b9d398b5245eb3f3744df639a1)
@@ -15,7 +15,7 @@
Pop int kusto:"Population" // use the kusto tag to map to a different column name
}

-func ExampleIterativeQuery() {
+func Example_IterativeQuery() {
// Create a client using the default Azure credential
kcsb := azkustodata.NewConnectionStringBuilder("https://help.kusto.windows.net/").WithDefaultAzureCredential()
client, err := azkustodata.New(kcsb)
Index: azkustodata/example_mgmt_test.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8

diff --git a/azkustodata/example_mgmt_test.go b/azkustodata/example_mgmt_test.go
--- a/azkustodata/example_mgmt_test.go (revision 450b317)
+++ b/azkustodata/example_mgmt_test.go (revision d1e90e2e349928b9d398b5245eb3f3744df639a1)
@@ -8,7 +8,7 @@
"github.com/Azure/azure-kusto-go/azkustodata/value"
)

-func ExampleMgmt() {
+func Example_Mgmt() {
// Create a client using the default Azure credential
kcsb := azkustodata.NewConnectionStringBuilder("https://help.kusto.windows.net/").WithDefaultAzureCredential()
client, err := azkustodata.New(kcsb)
Index: azkustodata/kusto_examples_test.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8

diff --git a/azkustodata/kusto_examples_test.go b/azkustodata/kusto_examples_test.go
--- a/azkustodata/kusto_examples_test.go (revision 450b317)
+++ b/azkustodata/kusto_examples_test.go (revision d1e90e2e349928b9d398b5245eb3f3744df639a1)
@@ -157,7 +157,7 @@
}
}

-func ExampleCustomHttpClient() { // nolint:govet // Example code
+func Example_CustomHttpClient() { // nolint:govet // Example code
// Create a connection string builder with your Azure ClientID, Secret and TenantID.
kcsb := NewConnectionStringBuilder("endpoint").WithAadAppKey("clientID", "clientSecret", "tenentID")
httpClient := &http.Client{

Co-authored-by: AsafMah <[email protected]>

Copilot · 2025-07-13T11:43:13Z

Subject: [PATCH] Rename

Index: azkustodata/example_iterative_query_test.go
...

Fixed the example function names to follow Go naming conventions in commit d7b4d66. The functions are now properly named:

ExampleIterativeQuery() → Example_iterativeQuery()
ExampleMgmt() → Example_mgmt()
ExampleCustomHttpClient() → ExampleNew_customHttpClient()

All tests now pass without build errors.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

statusreturner.azurewebsites.net
- Triggering command: /tmp/go-build3732533919/b316/etoe.test -test.testlogfile=/tmp/go-build3732533919/b316/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build3645912793/b316/etoe.test -test.testlogfile=/tmp/go-build3645912793/b316/testlog.txt -test.paniconexit0 -test.short=true -test.v=true -test.timeout=30s (dns block)
test.kusto.windows.net
- Triggering command: /tmp/go-build3645912793/b287/azkustodata.test -test.testlogfile=/tmp/go-build3645912793/b287/testlog.txt -test.paniconexit0 -test.short=true -test.v=true -test.timeout=30s (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

yogilad · 2025-07-14T06:37:28Z

CHANGELOG.md

+
+### Added
+- `WithHttpClient` option support for all ingest client types (`New`, `NewStreaming`, `NewManaged`) to enable custom HTTP client configuration and instrumentation such as OpenTelemetry
+- `HttpClient()` method for `Ingestion` and `Streaming` clients to access the underlying HTTP client


Why are those needed though?
Who needs to deconstruct the composed object in that way?

The HttpClient() getter methods provide access to the underlying HTTP client for instrumentation and monitoring purposes. Users may need to check client configuration, add middleware, or verify that custom HTTP clients (like those with OpenTelemetry instrumentation) are properly configured. This follows the same pattern as the existing azkustodata package.

Since the change no longer includes APIs to get the underlying http client back, please update the docs

@copilot Since the change no longer includes APIs to get the underlying http client back, please update the docs

Updated the CHANGELOG.md to remove references to the HTTP client getter methods since they were removed in commit 90d278b.

azkustodata/example_iterative_query_test.go

azkustodata/kusto_examples_test.go

azkustoingest/ingest.go

azkustoingest/ingestion_options_test.go

azkustoingest/managed.go

azkustoingest/streaming.go

Co-authored-by: yogilad <[email protected]>

AsafMah · 2025-07-14T08:56:03Z

Remove the new Getter methods for httpclient
Fix the test by using private members to test

Co-authored-by: AsafMah <[email protected]>

Copilot · 2025-07-14T09:02:45Z

Remove the new Getter methods for httpclient

Fix the test by using private members to test

Removed the HTTP client getter methods (HttpClient, QueuedHttpClient, StreamingHttpClient) and updated tests to access private fields directly instead. Tests now validate the custom HTTP client using ingest.client.HttpClient(), streaming.client.HttpClient(), and managed.queued.client.HttpClient() / managed.streaming.client.HttpClient(). Commit 0727698.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

test.kusto.windows.net
- Triggering command: /tmp/go-build2119682935/b001/azkustodata.test -test.paniconexit0 -test.timeout=10m0s -test.short=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

Co-authored-by: AsafMah <[email protected]>

Initial plan

7337313

Copilot AI assigned Copilot and AsafMah Jul 13, 2025

Copilot started work on behalf of AsafMah July 13, 2025 10:36 View session

Copilot AI and others added 2 commits July 13, 2025 10:40

Initial exploration and plan for HTTP client support

bc1a643

Co-authored-by: AsafMah <[email protected]>

Add WithHttpClient option support for all ingest client types

40b003f

Co-authored-by: AsafMah <[email protected]>

Copilot AI changed the title ~~[WIP] Support overriding the ingest HTTP client~~ Add WithHttpClient option support to azkustoingest package Jul 13, 2025

Copilot AI requested a review from AsafMah July 13, 2025 10:52

Copilot finished work on behalf of AsafMah July 13, 2025 10:52

AsafMah requested a review from yogilad July 13, 2025 11:17

Copilot started work on behalf of AsafMah July 13, 2025 11:17 View session

Copilot AI had a problem deploying to build July 13, 2025 11:19 Failure

Update CHANGELOG.md with WithHttpClient feature

450b317

Co-authored-by: AsafMah <[email protected]>

Copilot finished work on behalf of AsafMah July 13, 2025 11:20

Copilot AI had a problem deploying to build July 13, 2025 11:20 Failure

AsafMah approved these changes Jul 13, 2025

View reviewed changes

Copilot started work on behalf of AsafMah July 13, 2025 11:28 View session

Copilot AI requested a review from AsafMah July 13, 2025 11:34

Copilot stopped work on behalf of AsafMah due to an error July 13, 2025 11:34
Copilot has encountered an error. See logs for additional details.

AsafMah marked this pull request as ready for review July 13, 2025 11:35

Copilot started work on behalf of AsafMah July 13, 2025 11:36 View session

Rename example functions to follow Go naming conventions

d7b4d66

Co-authored-by: AsafMah <[email protected]>

Copilot finished work on behalf of AsafMah July 13, 2025 11:43

Copilot AI temporarily deployed to build July 13, 2025 15:35 Inactive

AsafMah approved these changes Jul 13, 2025

View reviewed changes

Merge branch 'master' into copilot/fix-311

9f6e7bf

AsafMah temporarily deployed to build July 14, 2025 05:21 — with GitHub Actions Inactive

yogilad reviewed Jul 14, 2025

View reviewed changes

Copilot started work on behalf of yogilad July 14, 2025 06:56 View session

Fix code formatting: add proper newlines and import ordering per gofmt

70ef004

Co-authored-by: yogilad <[email protected]>

Copilot AI requested a review from yogilad July 14, 2025 07:04

Copilot finished work on behalf of yogilad July 14, 2025 07:04

Copilot started work on behalf of AsafMah July 14, 2025 08:56 View session

Remove HTTP client getter methods and fix tests to use private fields

0727698

Co-authored-by: AsafMah <[email protected]>

Copilot AI requested a review from AsafMah July 14, 2025 09:03

Copilot finished work on behalf of AsafMah July 14, 2025 09:03

Copilot AI temporarily deployed to build July 14, 2025 09:25 Inactive

Copilot started work on behalf of AsafMah July 14, 2025 09:25 View session

Update CHANGELOG.md to remove references to HTTP client getter methods

90d278b

Co-authored-by: AsafMah <[email protected]>

Copilot finished work on behalf of AsafMah July 14, 2025 09:30

Copilot AI temporarily deployed to build July 14, 2025 09:31 Inactive

yogilad approved these changes Jul 14, 2025

View reviewed changes

AsafMah merged commit 64d4aaf into master Jul 14, 2025
6 checks passed

Add WithHttpClient option support to azkustoingest package #312

Add WithHttpClient option support to azkustoingest package #312

Uh oh!

Conversation

Copilot AI commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Queued Ingestion

Streaming Ingestion

Managed Ingestion

Changes Made

Testing

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

AsafMah commented Jul 13, 2025

Uh oh!

Copilot AI commented Jul 13, 2025

Uh oh!

github-actions bot commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

AsafMah commented Jul 13, 2025

Uh oh!

AsafMah commented Jul 13, 2025

Subject: [PATCH] Rename

Index: azkustodata/example_iterative_query_test.go IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8

Uh oh!

Copilot AI commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Subject: [PATCH] Rename

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

yogilad Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

yogilad Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

AsafMah Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AsafMah commented Jul 14, 2025

Uh oh!

Copilot AI commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 13, 2025 •

edited

Loading

github-actions bot commented Jul 13, 2025 •

edited

Loading

Index: azkustodata/example_iterative_query_test.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8

Copilot AI commented Jul 13, 2025 •

edited

Loading

Copilot AI commented Jul 14, 2025 •

edited

Loading