Merge branch 'master' into server-view-improvements-historical

Author: capistrant

docs/configuration/index.md

@@ -1362,6 +1362,7 @@ Middle Managers pass their configurations down to their child peons. The Middle
 |`druid.worker.baseTaskDirs`|List of base temporary working directories, one of which is assigned per task in a round-robin fashion. This property can be used to allow usage of multiple disks for indexing. This property is recommended in place of and takes precedence over `${druid.indexer.task.baseTaskDir}`.  If this configuration is not set, `${druid.indexer.task.baseTaskDir}` is used. For example, `druid.worker.baseTaskDirs=[\"PATH1\",\"PATH2\",...]`.|null|
 |`druid.worker.baseTaskDirSize`|The total amount of bytes that can be used by tasks on any single task dir. This value is treated symmetrically across all directories, that is, if this is 500 GB and there are 3 `baseTaskDirs`, then each of those task directories is assumed to allow for 500 GB to be used and a total of 1.5 TB will potentially be available across all tasks. The actual amount of memory assigned to each task is discussed in [Configuring task storage sizes](../ingestion/tasks.md#configuring-task-storage-sizes)|`Long.MAX_VALUE`|
 |`druid.worker.category`|A string to name the category that the Middle Manager node belongs to.|`_default_worker_category`|
+|`druid.worker.startAlwaysEnabled`|If true, the Middle Manager always starts in the enabled state. If false, a disabled state set via the worker disable API is persisted and restored across restarts.|`false`|
 |`druid.indexer.fork.property.druid.centralizedDatasourceSchema.enabled`| This config should be set when [Centralized Datasource Schema](#centralized-datasource-schema-experimental) feature is enabled. |false|
 
 #### Peon processing
@@ -1478,6 +1479,7 @@ For most types of tasks, `SegmentWriteOutMediumFactory` can be configured per-ta
 |`druid.worker.baseTaskDirSize`|The total amount of bytes that can be used by tasks on any single task dir. This value is treated symmetrically across all directories, that is, if this is 500 GB and there are 3 `baseTaskDirs`, then each of those task directories is assumed to allow for 500 GB to be used and a total of 1.5 TB will potentially be available across all tasks. The actual amount of memory assigned to each task is discussed in [Configuring task storage sizes](../ingestion/tasks.md#configuring-task-storage-sizes)|`Long.MAX_VALUE`|
 |`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available for ingestion processing. This is applied by automatically setting the `maxBytesInMemory` property on tasks.|Configured max JVM heap size / 6|
 |`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge operations that can run concurrently across all tasks.|`druid.worker.capacity` / 2, rounded down|
+|`druid.worker.startAlwaysEnabled`|If true, the Indexer always starts in the enabled state. If false, a disabled state set via the worker disable API is persisted and restored across restarts.|`false`|
 |`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`|
 |`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
 |`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on Indexer restart for restorable tasks to gracefully exit.|`PT5M`|

docs/development/extensions-core/druid-pac4j.md

@@ -55,6 +55,7 @@ druid.auth.authenticator.jwt.type=jwt
 |`druid.auth.pac4j.oidc.discoveryURI`|discovery URI for fetching OP metadata [see this](http://openid.net/specs/openid-connect-discovery-1_0.html).|none|Yes|
 |`druid.auth.pac4j.oidc.oidcClaim`|[claim](https://openid.net/specs/openid-connect-core-1_0.html#Claims) that will be extracted from the ID Token after validation.|name|No|
 |`druid.auth.pac4j.oidc.scope`| scope is used by an application during authentication to authorize access to a user's details.|`openid profile email`|No|
+|`druid.auth.pac4j.oidc.clientAuthenticationMethod`|The client authentication method to use when communicating with the OIDC provider. Supported values: `client_secret_basic`, `client_secret_post`, `client_secret_jwt`, `private_key_jwt`, `none`. If not specified, pac4j will auto-detect the method from the provider's discovery document. Set this explicitly if you need to use a specific method (e.g., when your provider advertises multiple methods but you want to use a particular one).|Auto-detected from provider|No|
 
 :::info
 Users must set a strong passphrase to ensure that an attacker is not able to guess it simply by brute force.

docs/ingestion/supervisor.md

@@ -193,29 +193,30 @@ The following example shows a supervisor spec with `lagBased` autoscaler:
 ```
 </details>
 
-**2. Cost-based autoscaler strategy (experimental)**
+**2. Cost-based autoscaler strategy**
 
-An autoscaler which computes the required supervisor task count via cost function based on ingestion lag and poll-to-idle ratio.
-Task counts are selected from a bounded range derived from the current partitions-per-task ratio,
-not strictly from factors/divisors of the partition count. This bounded partitions-per-task window enables gradual scaling while
-voiding large jumps and still allowing non-divisor task counts when needed.
+The cost-based autoscaler picks the number of ingestion tasks that minimizes a combined cost score. The score has two components:
 
-**It is experimental and the implementation details as well as cost function parameters are subject to change.**
+- **Lag cost** — how long it would take to drain the current backlog at the observed processing rate. More tasks reduce this cost.
+- **Idle cost** — how far the predicted idle ratio is from the target of ~25%. Tasks that are too busy (under-provisioned) or too idle (over-provisioned) both drive the score up. 
+The sweet spot is roughly 25% idle, giving headroom to absorb traffic spikes without wasting resources.
+
+At every evaluation interval, Druid computes the score for each candidate task count and picks the one with the lowest total cost.
 
 Note: Kinesis is not supported yet, support is in progress.
 
 The following table outlines the configuration properties related to the `costBased` autoscaler strategy:
 
-| Property|Description|Required|Default|
-|---------|-----------|--------|-------|
-|`scaleActionPeriodMillis`|The frequency in milliseconds to check if a scale action is triggered. | No | 600000 |
-|`lagWeight`|The weight of extracted lag value in cost function.| No| 0.25 |
-|`idleWeight`|The weight of extracted poll idle value in cost function. | No | 0.75 |
-|`useTaskCountBoundaries`|Enables the bounded partitions-per-task window when selecting task counts.|No| `false` |
-|`highLagThreshold`|Average partition lag threshold that triggers burst scale-up when set to a value greater than `0`. Set to a negative value to disable burst scale-up.|No|-1|
-|`minScaleUpDelay`|Minimum cooldown duration after a scale-up action before the next scale-up is allowed, specified as an ISO-8601 duration string.|No||
-|`minScaleDownDelay`|Minimum cooldown duration after a scale-down action before the next scale-down is allowed, specified as an ISO-8601 duration string.|No|`PT30M`|
-|`scaleDownDuringTaskRolloverOnly`|Indicates whether task scaling down is limited to periods during task rollovers only.|No|`false`|
+| Property | Description | Required | Default                   |
+|----------|-------------|----------|---------------------------|
+|`scaleActionPeriodMillis`|How often, in milliseconds, Druid evaluates whether to scale.|No| `600000` (10 min)         |
+|`lagWeight`|How much weight to give the lag cost relative to the idle cost. Higher values make the autoscaler more aggressive about adding tasks to drain backlog.|No| `0.4`                     |
+|`idleWeight`|How much weight to give the idle cost relative to the lag cost. Higher values make the autoscaler more aggressive about removing over-provisioned tasks.|No| `0.6`                     |
+|`useTaskCountBoundariesOnScaleUp`|Limits scale-up to a small step relative to the current task count, preventing large jumps. Disable to allow the autoscaler to jump directly to any task count.|No| `false`                   |
+|`useTaskCountBoundariesOnScaleDown`|Limits scale-down to a small step relative to the current task count, preventing large drops. Disable to allow the autoscaler to drop directly to any task count.|No| `true`                    |
+|`minScaleUpDelay`|Minimum cooldown after a scale-up before the next scale-up is allowed. Specified as an ISO-8601 duration.|No| `scaleActionPeriodMillis` |
+|`minScaleDownDelay`|Minimum cooldown after a scale-down before the next scale-down is allowed. Specified as an ISO-8601 duration.|No| `PT30M`                   |
+|`scaleDownDuringTaskRolloverOnly`|If `true`, scale-down actions are deferred until the next task rollover. This avoids disrupting in-progress ingestion.|No| `false`                   |
 
 The following example shows a supervisor spec with `costBased` autoscaler:
 
@@ -231,10 +232,10 @@ The following example shows a supervisor spec with `costBased` autoscaler:
       "autoScalerStrategy": "costBased",
       "taskCountMin": 1,
       "taskCountMax": 10,
+      "lagWeight": 0.4,
+      "idleWeight": 0.6,
       "minScaleUpDelay": "PT10M",
-      "minScaleDownDelay": "PT30M",
-      "lagWeight": 0.1,
-      "idleWeight": 0.9
+      "minScaleDownDelay": "PT30M"
     }
   }
 }

embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/autoscaler/CostBasedAutoScalerIntegrationTest.java

@@ -143,7 +143,7 @@ public void test_autoScaler_computesOptimalTaskCountAndProducesScaleUp()
       }
     });
 
-    // These values were carefully handpicked to allow that test to pass in a stable manner.
+    // These values were carefully handpicked to allow that test to pass stably.
     final CostBasedAutoScalerConfig autoScalerConfig = CostBasedAutoScalerConfig
         .builder()
         .enableTaskAutoScaler(true)
@@ -152,8 +152,8 @@ public void test_autoScaler_computesOptimalTaskCountAndProducesScaleUp()
         .taskCountStart(lowInitialTaskCount)
         .scaleActionPeriodMillis(500)
         .minTriggerScaleActionFrequencyMillis(1000)
-        .lagWeight(0.2)
-        .idleWeight(0.8)
+        .lagWeight(0.8)
+        .idleWeight(0.2)
         .build();
 
     final KafkaSupervisorSpec kafkaSupervisorSpec = createKafkaSupervisorWithAutoScaler(
@@ -192,11 +192,11 @@ public void test_autoScaler_scalesUpAndDown_withSlowPublish()
 
     final CostBasedAutoScalerConfig autoScalerConfig = CostBasedAutoScalerConfig
         .builder()
+        .enableTaskAutoScaler(true)
         .taskCountMin(1)
         .taskCountMax(4)
-        .lagWeight(1.0)
-        .idleWeight(1.0)
-        .enableTaskAutoScaler(true)
+        .lagWeight(0.5)
+        .idleWeight(0.5)
         .minTriggerScaleActionFrequencyMillis(10L)
         .scaleActionPeriodMillis(10L)
         .minScaleDownDelay(Duration.standardSeconds(1))

extensions-core/druid-catalog/src/main/java/org/apache/druid/catalog/sync/CacheNotifier.java

@@ -87,7 +87,7 @@ public void start()
           return;
         }
         catch (Throwable t) {
-          LOG.makeAlert(t, callerName + ": Error occured while handling updates.").emit();
+          LOG.makeAlert(t, callerName + ": Error occurred while handling updates.").emit();
         }
       }
     });

extensions-core/druid-catalog/src/main/java/org/apache/druid/catalog/sync/CatalogUpdateReceiver.java

@@ -85,7 +85,7 @@ public void start()
               LOG.debug("Scheduled catalog refresh is done");
             }
             catch (Throwable t) {
-              LOG.makeAlert(t, "Error occured while refreshing catalog.").emit();
+              LOG.makeAlert(t, "Error occurred while refreshing catalog.").emit();
             }
           }
       );

extensions-core/druid-pac4j/pom.xml

@@ -34,11 +34,11 @@
   </parent>
 
   <properties>
-    <pac4j.version>5.7.3</pac4j.version>
+    <pac4j.version>5.7.10</pac4j.version>
 
     <!-- Following must be updated along with any updates to pac4j version. One can find the compatible version of nimbus libraries in org.pac4j:pac4j-oidc dependencies-->
     <nimbus.lang.tag.version>1.7</nimbus.lang.tag.version>
-    <nimbus.jose.jwt.version>9.37.2</nimbus.jose.jwt.version>
+    <nimbus.jose.jwt.version>9.37.3</nimbus.jose.jwt.version>
     <oauth2.oidc.sdk.version>10.8</oauth2.oidc.sdk.version>
   </properties>
 

extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/OIDCConfig.java

@@ -44,20 +44,25 @@ public class OIDCConfig
   @JsonProperty
   private final String scope;
 
+  @JsonProperty
+  private final String clientAuthenticationMethod;
+
   @JsonCreator
   public OIDCConfig(
       @JsonProperty("clientID") String clientID,
       @JsonProperty("clientSecret") PasswordProvider clientSecret,
       @JsonProperty("discoveryURI") String discoveryURI,
       @JsonProperty("oidcClaim") String oidcClaim,
-      @JsonProperty("scope") @Nullable String scope
+      @JsonProperty("scope") @Nullable String scope,
+      @JsonProperty("clientAuthenticationMethod") @Nullable String clientAuthenticationMethod
   )
   {
     this.clientID = Preconditions.checkNotNull(clientID, "null clientID");
     this.clientSecret = Preconditions.checkNotNull(clientSecret, "null clientSecret");
     this.discoveryURI = Preconditions.checkNotNull(discoveryURI, "null discoveryURI");
     this.oidcClaim = oidcClaim == null ? DEFAULT_SCOPE : oidcClaim;
     this.scope = scope;
+    this.clientAuthenticationMethod = clientAuthenticationMethod;
   }
 
   @JsonProperty
@@ -89,4 +94,10 @@ public String getScope()
   {
     return scope;
   }
+
+  @JsonProperty
+  public String getClientAuthenticationMethod()
+  {
+    return clientAuthenticationMethod;
+  }
 }

extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/Pac4jAuthenticator.java

@@ -27,6 +27,7 @@
 import com.google.common.base.Suppliers;
 import com.google.common.primitives.Ints;
 import com.google.inject.Provider;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthenticationMethod;
 import com.nimbusds.oauth2.sdk.http.HTTPRequest;
 import org.apache.druid.server.security.AuthenticationResult;
 import org.apache.druid.server.security.Authenticator;
@@ -132,6 +133,10 @@ private Config createPac4jConfig(OIDCConfig oidcConfig)
     oidcConf.setSecret(oidcConfig.getClientSecret().getPassword());
     oidcConf.setDiscoveryURI(oidcConfig.getDiscoveryURI());
     oidcConf.setScope(oidcConfig.getScope());
+    if (oidcConfig.getClientAuthenticationMethod() != null) {
+      oidcConf.setClientAuthenticationMethod(
+          ClientAuthenticationMethod.parse(oidcConfig.getClientAuthenticationMethod()));
+    }
     oidcConf.setExpireSessionWithToken(true);
     oidcConf.setUseNonce(true);
     oidcConf.setReadTimeout(Ints.checkedCast(pac4jCommonConfig.getReadTimeout().getMillis()));

extensions-core/druid-pac4j/src/test/java/org/apache/druid/security/pac4j/OIDCConfigTest.java

@@ -46,6 +46,7 @@ public void testSerde() throws Exception
     Assert.assertEquals("testdiscoveryuri", conf.getDiscoveryURI());
     Assert.assertEquals("name", conf.getOidcClaim());
     Assert.assertEquals("testscope", conf.getScope());
+    Assert.assertNull(conf.getClientAuthenticationMethod());
   }
 
   @Test
@@ -72,4 +73,31 @@ public void testSerdeWithoutDefaults() throws Exception
     Assert.assertEquals("email", conf.getOidcClaim());
     Assert.assertEquals("testscope", conf.getScope());
   }
+
+  @Test
+  public void testSerdeWithClientAuthenticationMethod() throws Exception
+  {
+    ObjectMapper jsonMapper = new ObjectMapper();
+
+    String jsonStr = "{\n"
+                     + "  \"clientID\": \"testid\",\n"
+                     + "  \"clientSecret\": \"testsecret\",\n"
+                     + "  \"discoveryURI\": \"testdiscoveryuri\",\n"
+                     + "  \"oidcClaim\": \"email\",\n"
+                     + "  \"scope\": \"testscope\",\n"
+                     + "  \"clientAuthenticationMethod\": \"client_secret_post\"\n"
+                     + "}\n";
+
+    OIDCConfig conf = jsonMapper.readValue(
+        jsonMapper.writeValueAsString(jsonMapper.readValue(jsonStr, OIDCConfig.class)),
+        OIDCConfig.class
+    );
+
+    Assert.assertEquals("testid", conf.getClientID());
+    Assert.assertEquals("testsecret", conf.getClientSecret().getPassword());
+    Assert.assertEquals("testdiscoveryuri", conf.getDiscoveryURI());
+    Assert.assertEquals("email", conf.getOidcClaim());
+    Assert.assertEquals("testscope", conf.getScope());
+    Assert.assertEquals("client_secret_post", conf.getClientAuthenticationMethod());
+  }
 }