Add request-aware connection routing and enhanced node selection

Introduces flexible connection routing system that allows intelligent node
selection based on request characteristics and node roles. This enables
performance optimizations by routing operations to the most appropriate nodes.

Features:
- RequestAwareSelector interface for operation-based node selection
- Role-based selector with options pattern: WithRequiredRoles(),
  WithExcludedRoles(), WithStrictMode(), WithFallback()
- SmartSelector with automatic operation detection for bulk and search requests
- Automatic routing of bulk operations to ingest nodes and search to data nodes
- Extend connection pool with RequestAwareConnectionPool interface
- Add documentation and guides for node discovery and role management

This new routing system is backward compatible and optional - existing
clients continue to use round-robin selection unless explicitly configured.

Fixes: opensearch-project#770
Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

Author: sean-

CHANGELOG.md

@@ -7,6 +7,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ### Added
 - Enhanced cluster readiness checking for improved test reliability: `ostest.NewClient()` now includes readiness validation (health + cluster state + nodes info)
 - Configuration option `IncludeDedicatedClusterManagers` for controlling cluster manager node routing ([#765](https://github.com/opensearch-project/opensearch-go/issues/765))
+- Request-aware connection routing for improved performance and service availability ([#770](https://github.com/opensearch-project/opensearch-go/pull/770))
+  - `RequestAwareSelector` interface for operation-based node selection
+  - Role-based selector with flexible options pattern: `NewRoleBasedSelector()` with `WithRequiredRoles()`, `WithExcludedRoles()`, `WithStrictMode()`, `WithFallback()` options
+  - `SmartSelector` with automatic operation detection for bulk and search requests
+  - Automatic routing of bulk operations to ingest nodes and search operations to data nodes
 
 ### Changed
 - Refactor Client struct to use embedded mutex pattern for improved thread safety ([#775](https://github.com/opensearch-project/opensearch-go/pull/775))

USER_GUIDE.md

@@ -396,4 +396,5 @@ func getCredentialProvider(accessKey, secretAccessKey, token string) aws.Credent
 - [Advanced Index Actions](guides/advanced_index_actions.md)
 - [Index Templates](guides/index_template.md)
 - [Data Streams](guides/data_streams.md)
+- [Node Discovery and Role Management](guides/node_discovery_and_roles.md)
 - [Retry and Backoff](guides/retry_backoff.md)

guides/bulk.md

@@ -15,8 +15,11 @@ import (
 	"fmt"
 	"os"
 	"strings"
+	"time"
 
+	"github.com/opensearch-project/opensearch-go/v4"
 	"github.com/opensearch-project/opensearch-go/v4/opensearchapi"
+	"github.com/opensearch-project/opensearch-go/v4/opensearchtransport"
 )
 
 func main() {
@@ -27,6 +30,7 @@ func main() {
 }
 
 func example() error {
+	// Basic client setup
 	client, err := opensearchapi.NewDefaultClient()
 	if err != nil {
 		return err
@@ -35,6 +39,37 @@ func example() error {
 	ctx := context.Background()
 ```
 
+### Advanced Setup: Optimized for Bulk Operations
+
+For high-throughput bulk operations, you can configure the client to automatically route requests to appropriate nodes:
+
+```go
+	// Advanced client setup with smart routing for mixed workloads
+	advancedClient, err := opensearch.NewClient(opensearch.Config{
+		Addresses: []string{"http://localhost:9200"},
+
+		// Enable node discovery to find all cluster nodes
+		DiscoverNodesOnStart:  true,
+		DiscoverNodesInterval: 5 * time.Minute,
+
+		// Configure smart routing: bulk operations go to ingest nodes, searches go to data nodes
+		Transport: &opensearchtransport.Config{
+			Selector: opensearchtransport.NewSmartSelector(
+				opensearchtransport.NewRoundRobinSelector(),
+			),
+		},
+	})
+	if err != nil {
+		return err
+	}
+
+	// This client will automatically route operations to appropriate nodes:
+	// - Bulk operations -> ingest nodes
+	// - Search operations -> data nodes
+	// - Other operations -> round-robin
+	_ = advancedClient
+```
+
 Next, create an index named `movies` and another named `books` with the default settings:
 
 ```go
@@ -217,6 +252,86 @@ The following code shows an example on how to look for errors in the response:
 	}
 ```
 
+## Performance Optimization for Bulk Operations
+
+### Automatic Ingest Node Routing
+
+For production environments with dedicated ingest nodes, you can optimize bulk operation performance by routing requests to the most appropriate nodes:
+
+```go
+	// Create a client optimized for bulk operations
+	bulkClient, err := opensearch.NewClient(opensearch.Config{
+		Addresses: []string{"http://localhost:9200"},
+
+		// Enable node discovery
+		DiscoverNodesOnStart:  true,
+		DiscoverNodesInterval: 5 * time.Minute,
+
+		// Use smart selector for automatic operation routing
+		Transport: &opensearchtransport.Config{
+			Selector: opensearchtransport.NewSmartSelector(
+				opensearchtransport.NewRoundRobinSelector(),
+			),
+		},
+	})
+	if err != nil {
+		return err
+	}
+
+	// This bulk request will automatically route to ingest nodes
+	bulkResp, err := bulkClient.Bulk(
+		ctx,
+		opensearchapi.BulkReq{
+			Body: strings.NewReader(`{ "index": { "_index": "movies", "_id": "perf-1" } }
+{ "title": "High Performance Bulk", "year": 2024 }
+{ "index": { "_index": "movies", "_id": "perf-2" } }
+{ "title": "Optimized Ingest", "year": 2024 }
+`),
+		},
+	)
+	if err != nil {
+		return err
+	}
+	fmt.Printf("Optimized bulk completed with %d items\n", len(bulkResp.Items))
+```
+
+### Choosing the Right Selector
+
+You can choose different routing strategies based on your cluster setup:
+
+```go
+	// Create a fallback selector (round-robin)
+	fallbackSelector := opensearchtransport.NewRoundRobinSelector()
+
+	// Option 1: Prefer ingest nodes, fallback to any available node
+	ingestPreferred := opensearchtransport.NewRoleBasedSelector(
+		opensearchtransport.WithRequiredRoles(opensearchtransport.RoleIngest),
+		opensearchtransport.WithFallback(fallbackSelector),
+	)
+
+	// Option 2: Only use ingest nodes, fail if none available (strict mode)
+	ingestOnly := opensearchtransport.NewRoleBasedSelector(
+		opensearchtransport.WithRequiredRoles(opensearchtransport.RoleIngest),
+		opensearchtransport.WithStrictMode(),
+	)
+
+	// Option 3: Automatically detect operation type and route appropriately
+	smartSelector := opensearchtransport.NewSmartSelector(fallbackSelector)
+
+	// Option 4: Use the generic selector for custom role combinations
+	customSelector := opensearchtransport.NewRoleBasedSelector(
+		opensearchtransport.WithRequiredRoles(opensearchtransport.RoleIngest),
+		opensearchtransport.WithExcludedRoles(opensearchtransport.RoleClusterManager),
+		opensearchtransport.WithFallback(fallbackSelector),
+	)
+```
+
+The smart selector automatically detects different operation types:
+- **Bulk operations** (`/_bulk`) -> Routes to ingest nodes
+- **Ingest pipeline operations** (`/_ingest/`) -> Routes to ingest nodes
+- **Search operations** (`/_search`) -> Routes to data nodes
+- **Other operations** -> Uses default routing
+
 ## Cleanup
 
 To clean up the resources created in this guide, delete the `movies` and `books` indices: