Skip to content

Commit 9b0f7d7

Browse files
elijah-rouelijah-rou
committed
new cooldown descriptions
1 parent d690b9e commit 9b0f7d7

File tree

6 files changed

+27
-19
lines changed

6 files changed

+27
-19
lines changed

‎cerebrium/partner-services/deepgram.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,7 +315,7 @@ Deepgram services support independent scaling configurations:
315315
- **min_replicas**: Minimum number of instances to maintain (0 for scale-to-zero)
316316
- **max_replicas**: Maximum number of instances that can be created during high load
317317
- **replica_concurrency**: Number of concurrent requests each instance can handle
318-
- **cooldown**: Time in seconds that an instance remains active after processing its last request
318+
- **cooldown**: Time window (in seconds) that must pass at reduced concurrency before scaling down
319319

320320
Adjust these parameters based on expected traffic patterns and latency requirements.
321321

‎cerebrium/partner-services/index.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ Partner Services support independent scaling configurations:
3636

3737
- Use the `min_replicas` and `max_replicas` parameters to control the number of instances
3838
- The `replica_concurrency` parameter determines how many concurrent requests each instance can handle
39-
- Adjust the `cooldown` parameter to control how long instances remain active after processing requests
39+
- Adjust the `cooldown` parameter to control the time window that must pass at reduced concurrency before scaling down
4040
- Adjust the `hardware` section to control the instance type which affects performance and/or cost
4141

4242
For more information on specific Partner Services, see:

‎cerebrium/partner-services/rime.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ Rime services support independent scaling configurations:
9999
- **min_replicas**: Minimum instances to maintain (0 for scale-to-zero). Recommended: 1.
100100
- **max_replicas**: Maximum instances during high load.
101101
- **replica_concurrency**: Concurrent requests per instance. Recommended: 3.
102-
- **cooldown**: Seconds an instance remains active after last request. Recommended: 50.
102+
- **cooldown**: Time window (in seconds) that must pass at reduced concurrency before scaling down. Recommended: 50.
103103
- **compute**: Instance type. Recommended: `AMPERE_A10`.
104104

105105
Adjust these parameters based on traffic patterns and latency requirements. Best would be to consult the Rime team

‎cerebrium/scaling/scaling-apps.mdx‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ The **number of requests** currently waiting for processing in the queue indicat
1616
See below for more information.
1717
</Info>
1818

19-
As traffic decreases, instances enter a cooldown period after processing their last request. When no new requests arrive during cooldown, instances terminate to optimize resource usage. This automatic cycle ensures apps remain responsive while managing costs effectively.
19+
As traffic decreases, instances enter a cooldown period at reduced concurrency. If reduced concurrency is maintained for the cooldown duration, instances scale down to optimize resource usage. This automatic cycle ensures apps remain responsive while managing costs effectively.
2020

2121
## Scaling Configuration
2222

@@ -40,7 +40,7 @@ The `max_replicas` parameter sets an upper limit on concurrent instances, contro
4040

4141
### Cooldown Period
4242

43-
After processing a request, instances remain available for the duration specified by `cooldown`. Each new request resets this timer. A longer cooldown period helps handle bursty traffic patterns but increases instance running time and cost.
43+
The `cooldown` parameter specifies the time window (in seconds) that must pass at reduced concurrency before an instance scales down. This prevents premature scale-down during brief traffic dips that might be followed by more requests. A longer cooldown period helps handle bursty traffic patterns but increases instance running time and cost.
4444

4545
### Replica Concurrency
4646

@@ -187,6 +187,10 @@ Once this request has completed, the usual `cooldown` period will apply, and the
187187

188188
## Evaluation Interval
189189

190+
<Warning>
191+
Requires CLI version 2.1.5 or higher.
192+
</Warning>
193+
190194
The `evaluation_interval` parameter controls the time window (in seconds) over which the autoscaler evaluates metrics before making scaling decisions. The default is 30 seconds, with a valid range of 6-300 seconds.
191195

192196
```toml
@@ -204,6 +208,10 @@ A shorter interval makes the autoscaler more responsive to traffic spikes but ma
204208

205209
## Load Balancing
206210

211+
<Warning>
212+
Requires CLI version 2.1.5 or higher.
213+
</Warning>
214+
207215
The `load_balancing` parameter controls how incoming requests are distributed across your replicas. When not specified, the system automatically selects the best algorithm based on your `replica_concurrency` setting.
208216

209217
```toml

‎migrations/mystic.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ gpu_count = 1 # Number of GPUs
7878
[cerebrium.scaling]
7979
min_replicas = 0 # Save costs when inactive and scale down your app
8080
max_replicas = 2 # Handle increased traffic and scale up where necessary
81-
cooldown = 60 # Time to wait before scaling down an idle instance
81+
cooldown = 60 # Time window at reduced concurrency before scaling down
8282
replica_concurrency = 1 # The number of requests a single container can support
8383

8484
[cerebrium.dependencies.pip]

‎toml-reference/toml-reference.mdx‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -128,19 +128,19 @@ The `[cerebrium.hardware]` section defines compute resources.
128128

129129
The `[cerebrium.scaling]` section controls auto-scaling behavior.
130130

131-
| Option | Type | Default | Description |
132-
| ------------------------- | ------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
133-
| min_replicas | integer | 0 | Minimum running instances |
134-
| max_replicas | integer | 2 | Maximum running instances |
135-
| replica_concurrency | integer | 10 | Concurrent requests per replica |
136-
| response_grace_period | integer | 3600 | Grace period in seconds |
137-
| cooldown | integer | 1800 | Time to wait before scaling down an idle container |
138-
| scaling_metric | string | "concurrency_utilization" | Metric for scaling decisions (concurrency_utilization, requests_per_second, cpu_utilization, memory_utilization) |
139-
| scaling_target | integer | 100 | Target value for scaling metric (percentage for utilization metrics, absolute value for requests_per_second) |
140-
| scaling_buffer | integer | optional | Additional replica capacity above what scaling metric suggests |
141-
| evaluation_interval | integer | 30 | Time window in seconds over which metrics are evaluated before scaling decisions (6-300s) |
142-
| load_balancing | string | "" | Algorithm for distributing traffic across replicas. Default: round-robin if replica_concurrency > 3, first-available otherwise. Options: round-robin, first-available, min-connections, random-choice-2 |
143-
| roll_out_duration_seconds | integer | 0 | Gradually send traffic to new revision after successful build. Max 600s. Keep at 0 during development. |
131+
| Option | Type | Default | CLI Requirement | Description |
132+
| ------------------------- | ------- | ------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
133+
| min_replicas | integer | 0 | 2.1.2+ | Minimum running instances |
134+
| max_replicas | integer | 2 | 2.1.2+ | Maximum running instances |
135+
| replica_concurrency | integer | 10 | 2.1.2+ | Concurrent requests per replica |
136+
| response_grace_period | integer | 3600 | 2.1.2+ | Grace period in seconds |
137+
| cooldown | integer | 1800 | 2.1.2+ | Time window (seconds) that must pass at reduced concurrency before scaling down. Helps avoid cold starts from brief traffic dips. |
138+
| scaling_metric | string | "concurrency_utilization" | 2.1.2+ | Metric for scaling decisions (concurrency_utilization, requests_per_second, cpu_utilization, memory_utilization) |
139+
| scaling_target | integer | 100 | 2.1.2+ | Target value for scaling metric (percentage for utilization metrics, absolute value for requests_per_second) |
140+
| scaling_buffer | integer | optional | 2.1.2+ | Additional replica capacity above what scaling metric suggests |
141+
| evaluation_interval | integer | 30 | 2.1.5+ | Time window in seconds over which metrics are evaluated before scaling decisions (6-300s) |
142+
| load_balancing | string | "" | 2.1.5+ | Algorithm for distributing traffic across replicas. Default: round-robin if replica_concurrency > 3, first-available otherwise. Options: round-robin, first-available, min-connections, random-choice-2 |
143+
| roll_out_duration_seconds | integer | 0 | 2.1.2+ | Gradually send traffic to new revision after successful build. Max 600s. Keep at 0 during development. |
144144

145145
<Warning>
146146
Setting min_replicas > 0 maintains warm instances for immediate response but

0 commit comments

Comments
 (0)