Merge pull request #259 from CerebriumAI/eli/cer-4656-add-evaluation-interval-to-the-clibackend

Author: elijah-rou

cerebrium/partner-services/deepgram.mdx

@@ -315,7 +315,7 @@ Deepgram services support independent scaling configurations:
 - **min_replicas**: Minimum number of instances to maintain (0 for scale-to-zero)
 - **max_replicas**: Maximum number of instances that can be created during high load
 - **replica_concurrency**: Number of concurrent requests each instance can handle
-- **cooldown**: Time in seconds that an instance remains active after processing its last request
+- **cooldown**: Time window (in seconds) that must pass at reduced concurrency before scaling down
 
 Adjust these parameters based on expected traffic patterns and latency requirements.
 

cerebrium/partner-services/index.mdx

@@ -36,7 +36,7 @@ Partner Services support independent scaling configurations:
 
 - Use the `min_replicas` and `max_replicas` parameters to control the number of instances
 - The `replica_concurrency` parameter determines how many concurrent requests each instance can handle
-- Adjust the `cooldown` parameter to control how long instances remain active after processing requests
+- Adjust the `cooldown` parameter to control the time window that must pass at reduced concurrency before scaling down
 - Adjust the `hardware` section to control the instance type which affects performance and/or cost
 
 For more information on specific Partner Services, see:

cerebrium/partner-services/rime.mdx

@@ -99,7 +99,7 @@ Rime services support independent scaling configurations:
 - **min_replicas**: Minimum instances to maintain (0 for scale-to-zero). Recommended: 1.
 - **max_replicas**: Maximum instances during high load.
 - **replica_concurrency**: Concurrent requests per instance. Recommended: 3.
-- **cooldown**: Seconds an instance remains active after last request. Recommended: 50.
+- **cooldown**: Time window (in seconds) that must pass at reduced concurrency before scaling down. Recommended: 50.
 - **compute**: Instance type. Recommended: `AMPERE_A10`.
 
 Adjust these parameters based on traffic patterns and latency requirements. Best would be to consult the Rime team

cerebrium/scaling/scaling-apps.mdx

@@ -16,7 +16,7 @@ The **number of requests** currently waiting for processing in the queue indicat
   See below for more information.
 </Info>
 
-As traffic decreases, instances enter a cooldown period after processing their last request. When no new requests arrive during cooldown, instances terminate to optimize resource usage. This automatic cycle ensures apps remain responsive while managing costs effectively.
+As traffic decreases, instances enter a cooldown period at reduced concurrency. If reduced concurrency is maintained for the cooldown duration, instances scale down to optimize resource usage. This automatic cycle ensures apps remain responsive while managing costs effectively.
 
 ## Scaling Configuration
 
@@ -40,7 +40,7 @@ The `max_replicas` parameter sets an upper limit on concurrent instances, contro
 
 ### Cooldown Period
 
-After processing a request, instances remain available for the duration specified by `cooldown`. Each new request resets this timer. A longer cooldown period helps handle bursty traffic patterns but increases instance running time and cost.
+The `cooldown` parameter specifies the time window (in seconds) that must pass at reduced concurrency before an instance scales down. This prevents premature scale-down during brief traffic dips that might be followed by more requests. A longer cooldown period helps handle bursty traffic patterns but increases instance running time and cost.
 
 ### Replica Concurrency
 
@@ -184,3 +184,99 @@ Since the config has specified `100` as a target for `concurrency_utilization` a
 the autoscaler will suggest a value of 1 replica for scale out. Since however, we have `scale_buffer=3`, the application will actually scale one more replica to **(1+3)=4**.
 In other words, the scale buffer will simply add a static amount of replicas to the number of replicas the autoscaler suggests using the scale target.
 Once this request has completed, the usual `cooldown` period will apply, and the app replica count will scale down back to the baseline of **1 replica**.
+
+## Evaluation Interval
+
+<Warning>Requires CLI version 2.1.5 or higher.</Warning>
+
+The `evaluation_interval` parameter controls the time window (in seconds) over which the autoscaler evaluates metrics before making scaling decisions. The default is 30 seconds, with a valid range of 6-300 seconds.
+
+```toml
+[cerebrium.scaling]
+evaluation_interval = 30  # Evaluate metrics over 30-second windows
+```
+
+A shorter interval makes the autoscaler more responsive to traffic spikes but may cause more frequent scaling events. A longer interval smooths out transient spikes but may delay scaling responses.
+
+<Info>
+  For bursty workloads, a shorter `evaluation_interval` (e.g., 10-15 seconds)
+  helps the system respond quickly to demand. For steady workloads, a longer
+  interval (e.g., 60 seconds) reduces unnecessary scaling churn.
+</Info>
+
+## Load Balancing
+
+<Warning>Requires CLI version 2.1.5 or higher.</Warning>
+
+The `load_balancing` parameter controls how incoming requests are distributed across your replicas. When not specified, the system automatically selects the best algorithm based on your `replica_concurrency` setting.
+
+```toml
+[cerebrium.scaling]
+load_balancing = "min-connections"  # Explicitly set load balancing algorithm
+```
+
+**Default behavior**: When `load_balancing` is not set, the system uses `first-available` for `replica_concurrency <= 3` (typical for GPU workloads) and `round-robin` for higher concurrency.
+
+### Available Algorithms
+
+#### round-robin
+
+Cycles through replicas starting from the last successful target. Each replica's concurrency limit is respected - if a replica is at capacity, the algorithm proceeds to the next one in rotation.
+
+| Characteristic       | Value                                                                   |
+| -------------------- | ----------------------------------------------------------------------- |
+| Selection complexity | O(1) typical, O(N) worst case when scanning for available capacity      |
+| Latency profile      | Consistent p50, good p90 under uniform load                             |
+| Strategy             | Stateful index rotation with mutex synchronization; skips full replicas |
+
+**Best for**: Workloads with predictable request times where you want even distribution across replicas over time.
+
+#### first-available
+
+Scans replicas from the start of the list and selects the first one with available capacity.
+
+| Characteristic       | Value                                                                         |
+| -------------------- | ----------------------------------------------------------------------------- |
+| Selection complexity | O(1) typical, O(N) worst case                                                 |
+| Latency profile      | Optimal p50 when load is light, may degrade p90 under high load               |
+| Strategy             | Linear scan from list start; returns first replica that accepts via Reserve() |
+
+**Best for**: GPU workloads with low concurrency (`replica_concurrency <= 3`). Maximizes utilization of warm replicas before spreading load, reducing cold starts and keeping models in VRAM.
+
+**Tradeoff**: Earlier replicas in the list handle more traffic. This is desirable for GPU workloads but may cause uneven distribution for CPU workloads.
+
+#### min-connections
+
+Linear scan to find the replica with the fewest in-flight requests, then attempts to reserve it. If that replica cannot accept (at capacity), falls back to trying other replicas in iteration order.
+
+| Characteristic       | Value                                                              |
+| -------------------- | ------------------------------------------------------------------ |
+| Selection complexity | Θ(N) - always scans all replicas to find minimum                   |
+| Latency profile      | Best p90/p99 tail latency                                          |
+| Strategy             | Single pass to find minimum in-flight; fallback in iteration order |
+
+**Best for**: Workloads with variable request times (e.g., LLM inference where output length varies). Routes new requests to the least busy replica, preventing fast requests from queuing behind slow ones.
+
+#### random-choice-2
+
+Implements the "Power of Two Choices" algorithm: randomly samples two replicas and routes to the one with lower weight (based on active request tracking). Ties are broken randomly.
+
+| Characteristic       | Value                                                       |
+| -------------------- | ----------------------------------------------------------- |
+| Selection complexity | Θ(1) - constant time regardless of replica count            |
+| Latency profile      | Good balance of p50 and p90                                 |
+| Strategy             | Sample 2 random replicas, compare weights, pick lighter one |
+
+**Best for**: High-throughput scenarios with many replicas where selection overhead matters. Research shows this achieves exponentially better load distribution than pure random selection.
+
+**Note**: Uses weight-based tracking rather than reservation-based concurrency limiting, making it suitable for unlimited concurrency scenarios.
+
+### Choosing an Algorithm
+
+| Scenario                                 | Recommended                 | Reason                                             |
+| ---------------------------------------- | --------------------------- | -------------------------------------------------- |
+| GPU inference, `replica_concurrency=1`   | `first-available` (default) | Maximizes GPU utilization, keeps models warm       |
+| LLMs with variable output lengths        | `min-connections`           | Prevents head-of-line blocking, best tail latency  |
+| High-throughput, many replicas           | `random-choice-2`           | Θ(1) selection with near-optimal distribution      |
+| Uniform request times, even distribution | `round-robin`               | Predictable rotation, no hot spots over time       |
+| Latency-sensitive with variable load     | `min-connections`           | Minimizes p90/p99 by routing to least busy replica |

migrations/mystic.mdx

@@ -78,7 +78,7 @@ gpu_count = 1             # Number of GPUs
 [cerebrium.scaling]
 min_replicas = 0         # Save costs when inactive and scale down your app
 max_replicas = 2         # Handle increased traffic and scale up where necessary
-cooldown = 60            # Time to wait before scaling down an idle instance
+cooldown = 60            # Time window at reduced concurrency before scaling down
 replica_concurrency = 1  # The number of requests a single container can support
 
 [cerebrium.dependencies.pip]

toml-reference/toml-reference.mdx

@@ -128,17 +128,19 @@ The `[cerebrium.hardware]` section defines compute resources.
 
 The `[cerebrium.scaling]` section controls auto-scaling behavior.
 
-| Option                    | Type    | Default                   | Description                                                                                                      |
-| ------------------------- | ------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------- |
-| min_replicas              | integer | 0                         | Minimum running instances                                                                                        |
-| max_replicas              | integer | 2                         | Maximum running instances                                                                                        |
-| replica_concurrency       | integer | 10                        | Concurrent requests per replica                                                                                  |
-| response_grace_period     | integer | 3600                      | Grace period in seconds                                                                                          |
-| cooldown                  | integer | 1800                      | Time to wait before scaling down an idle container                                                               |
-| scaling_metric            | string  | "concurrency_utilization" | Metric for scaling decisions (concurrency_utilization, requests_per_second, cpu_utilization, memory_utilization) |
-| scaling_target            | integer | 100                       | Target value for scaling metric (percentage for utilization metrics, absolute value for requests_per_second)     |
-| scaling_buffer            | integer | optional                  | Additional replica capacity above what scaling metric suggests                                                   |
-| roll_out_duration_seconds | integer | 0                         | Gradually send traffic to new revision after successful build. Max 600s. Keep at 0 during development.           |
+| Option                    | Type    | Default                   | CLI Requirement | Description                                                                                                                                                                                             |
+| ------------------------- | ------- | ------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| min_replicas              | integer | 0                         | 2.1.2+          | Minimum running instances                                                                                                                                                                               |
+| max_replicas              | integer | 2                         | 2.1.2+          | Maximum running instances                                                                                                                                                                               |
+| replica_concurrency       | integer | 10                        | 2.1.2+          | Concurrent requests per replica                                                                                                                                                                         |
+| response_grace_period     | integer | 3600                      | 2.1.2+          | Grace period in seconds                                                                                                                                                                                 |
+| cooldown                  | integer | 1800                      | 2.1.2+          | Time window (seconds) that must pass at reduced concurrency before scaling down. Helps avoid cold starts from brief traffic dips.                                                                       |
+| scaling_metric            | string  | "concurrency_utilization" | 2.1.2+          | Metric for scaling decisions (concurrency_utilization, requests_per_second, cpu_utilization, memory_utilization)                                                                                        |
+| scaling_target            | integer | 100                       | 2.1.2+          | Target value for scaling metric (percentage for utilization metrics, absolute value for requests_per_second)                                                                                            |
+| scaling_buffer            | integer | optional                  | 2.1.2+          | Additional replica capacity above what scaling metric suggests                                                                                                                                          |
+| evaluation_interval       | integer | 30                        | 2.1.5+          | Time window in seconds over which metrics are evaluated before scaling decisions (6-300s)                                                                                                               |
+| load_balancing            | string  | ""                        | 2.1.5+          | Algorithm for distributing traffic across replicas. Default: round-robin if replica_concurrency > 3, first-available otherwise. Options: round-robin, first-available, min-connections, random-choice-2 |
+| roll_out_duration_seconds | integer | 0                         | 2.1.2+          | Gradually send traffic to new revision after successful build. Max 600s. Keep at 0 during development.                                                                                                  |
 
 <Warning>
   Setting min_replicas > 0 maintains warm instances for immediate response but
@@ -241,6 +243,8 @@ response_grace_period = 3600
 cooldown = 1800
 scaling_metric = "concurrency_utilization"
 scaling_target = 100
+evaluation_interval = 30
+# load_balancing = ""  # Auto-selects based on replica_concurrency
 roll_out_duration_seconds = 0
 
 [cerebrium.dependencies.pip]