Skip to content

dynamic-configuration.mdx

Latest commit

 

History

History
251 lines (201 loc) · 29.5 KB

dynamic-configuration.mdx

File metadata and controls

251 lines (201 loc) · 29.5 KB
id dynamic-configuration
title Temporal Cluster dynamic configuration reference
sidebar_label Dynamic configuration
description Temporal Cluster offers dynamic configuration keys that you can update on the fly to optimize performance without service interruption. Customize these settings to meet specific Workflow, Activity, Namespace, or Task Queue requirements, and test thoroughly before deploying to production. For more details, visit the Temporal GitHub repository.
toc_max_heading_level 4
keywords
reference
tags
Reference

Temporal Cluster provides dynamic configuration keys that you can update and apply to a running Cluster without restarting your services.

The dynamic configuration keys are set with default values when you create your Cluster configuration. You can override these values as you test your Cluster setup for optimal performance according to your workload requirements.

For the complete list of dynamic configuration keys, see https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go. Ensure that you check server release notes for any changes to these keys and values.

For the default values of dynamic configuration keys, check the following links:

Setting dynamic configuration is optional. Change these values only if you need to override the default values to achieve better performance on your Temporal Cluster. Also, ensure that you test your changes before setting these in production.

Format

To override the default dynamic configuration values, specify your custom values and constraints for the dynamic configuration keys that you want to change in a YAML configuration file. Use the following format when creating your dynamic configuration file.

testGetBoolPropertyKey:
  - value: false
  - value: true
    constraints:
      namespace: 'your-namespace'
  - value: false
    constraints:
      namespace: 'your-other-namespace'
testGetDurationPropertyKey:
  - value: '1m'
    constraints:
      namespace: 'your-namespace'
      taskQueueName: 'longIdleTimeTaskqueue'
testGetFloat64PropertyKey:
  - value: 12.0
    constraints:
      namespace: 'your-namespace'
testGetMapPropertyKey:
  - value:
      key1: 1
      key2: 'value 2'
      key3:
        - false
        - key4: true
          key5: 2.0

Constraints

You can define constraints on some dynamic configuration keys to set specific values that apply on a Namespace or Task Queue level. Not defining constraints on a dynamic configuration key sets the values across the Cluster.

  • To set global values for the configuration key with no constraints, use the following:

    frontend.globalNamespaceRPS: # Total per-Namespace RPC rate limit applied across the Cluster.
  - value: 5000

  • For keys that can be customized at Namespace level, you can specify multiple values for different Namespaces in addition to one default value that applies globally to all Namespaces. To set values at a Namespace level, use namespace (String) as shown in the following example.

    frontend.persistenceNamespaceMaxQPS: # Rate limit on the number of queries the Frontend sends to the Persistence store.
  - constraints: {} # Sets default value that applies to all Namespaces
    value: 2000 # The default value for this key is 0.
  - constraints: { namespace: 'namespace1' } # Sets limit on number of queries that can be sent from "namespace1" Namespace to the Persistence store.
    value: 4000
  - constraints: { namespace: 'namespace2' }
    value: 1000

  • For keys that can be customized at a Task Queue level, you can specify Task Queue name and Task type in addition to Namespace. To set values at a Task Queue level, use taskQueueName (String) with taskType (optional; supported values: Workflow and Activity).

    For example if you have Workflow Executions creating a large number of Workflow and Activity tasks per second, you can add more partitions to your Task Queues (default is 4) to handle the high throughput of tasks. To do this, add the following to your dynamic configuration file. Note that if changing the number of partitions, you must set the same count for both read and write operations on Task Queues.

    matching.numTaskqueueReadPartitions: # Number of Task Queue partitions for read operations.
  - constraints: { namespace: 'namespace1', taskQueueName: 'tq' } # Applies to the "tq" Task Queue for both Workflows and Activities.
    value: 8 # The default value for this key is 4. Task Queues that need to support high traffic require higher number of partitions. Set these values in accordance to your poller count.
  - constraints: {
        namespace: 'namespace1',
        taskQueueName: 'other-tq',
        taskType: 'Activity',
      } # Applies to the "other_tq" Task Queue for Activities specifically.
    value: 20
  - constraints: { namespace: 'namespace2' } # Applies to all task queues in "namespace2".
    value: 10
  - constraints: {} # Applies to all other task queues in "namespace1" and all other Namespaces.
    value: 16
matching.numTaskqueueWritePartitions: # Number of Task Queue partitions for write operations.
  - constraints: { namespace: 'namespace1', taskQueueName: 'tq' } # Applies to the "tq" Task Queue for both Workflows and Activities.
    value: 8 # The default value for this key is 4. Task Queues that need to support high traffic require higher number of partitions. Set these values in accordance to your poller count.
  - constraints: {
        namespace: 'namespace1',
        taskQueueName: 'other-tq',
        taskType: 'Activity',
      } # Applies to the "other_tq" Task Queue for Activities specifically.
    value: 20
  - constraints: { namespace: 'namespace2' } # Applies to all task queues in "namespace2".
    value: 10
  - constraints: {} # Applies to all other task queues in "namespace1" and all other Namespaces.
    value: 16

{/* Note that the values set with most constraints take priority over values that are set with fewer constraints, regardless of the order in which they are set in the dynamic configuration key. */}

For more examples on how dynamic configuration is set, see samples-server.

Commonly used dynamic configuration keys

The following table lists commonly used dynamic configuration keys that can be used for rate limiting requests to the Temporal Cluster.

Setting dynamic configuration keys is optional. If you choose to update these values for your Temporal Cluster, ensure that you are provisioning enough resources to handle the load.

All values listed here are for Temporal server v1.21. Check server release notes to verify any potential breaking changes when upgrading your versions.

Service-level RPS limits

The Requests Per Second (RPS) dynamic configuration keys set the rate at which requests can be made to each service in your Cluster.

When scaling your services, tune the RPS to test your workload and set acceptable provisioning benchmarks. Exceeding these limits results in ResourceExhaustedError.

Dynamic configuration key Type Description Default value
Frontend
frontend.rps Int Rate limit (requests/second) for requests accepted by each Frontend Service host. 2400
frontend.namespaceRPS Int Rate limit (requests/second) for requests accepted by each Namespace on the Frontend Service. 2400
frontend.namespaceCount Int Limit on the number of concurrent Task Queue polls per Namespace per Frontend Service host. 1200
frontend.globalNamespaceRPS Int Rate limit (requests/second) for requests accepted per Namespace, applied across Cluster. The limit is evenly distributed among available Frontend Service instances. If this is set, it overrides the per-instance limit (frontend.namespaceRPS). 0
internal-frontend.globalNamespaceRPS Int Rate limit (requests/second) for requests accepted on each Internal-Frontend Service host applied across the Cluster. 0
History
history.rps Int Rate limit (requests/second) for requests accepted by each History Service host. 3000
Matching
matching.rps Int Rate limit (requests/second) for requests accepted by each Matching Service host. 1200
matching.numTaskqueueReadPartitions Int Number of read partitions for a Task Queue. Must be set with matching.numTaskqueueWritePartitions. 4
matching.numTaskqueueWritePartitions Int Number of write partitions for a Task Queue. 4

QPS limits for Persistence store

The Queries Per Second (QPS) dynamic configuration keys set the maximum number of queries a service can make per second to the Persistence store.

Persistence store rate limits are evaluated synchronously. Adjust these keys according to your database capacity and workload. If the number of queries made to the Persistence store exceeds the dynamic configuration value, you will see latencies and timeouts on your tasks.

Dynamic configuration key Type Description Default value
Frontend
frontend.persistenceMaxQPS Int Maximum number queries per second that the Frontend Service host can send to the Persistence store. 2000
frontend.persistenceNamespaceMaxQPS Int Maximum number of queries per second that each Namespace on the Frontend Service host can send to the Persistence store.
If the value set for this config is less than or equal to 0, the value set for frontend.persistenceMaxQPS will apply.		 0
History
history.persistenceMaxQPS Int Maximum number of queries per second that the History host can send to the Persistence store. 9000
history.persistenceNamespaceMaxQPS Int Maximum number of queries per second for each Namespace that the History host can send to the Persistence store.
If the value set for this config is less than or equal to 0, then the value set for history.persistenceMaxQPS will apply.		 0
Matching
matching.persistenceMaxQPS Int Maximum number of queries per second that the Matching Service host can send to the Persistence store. 9000
matching.persistenceNamespaceMaxQPS Int Maximum number of queries per second that the Matching host can send to the Persistence store for each Namespace.
If the value set for this config is less than or equal to 0, the value set for matching.persistenceMaxQPS will apply.		 0
Worker
worker.persistenceMaxQPS Int Maximum number of queries per second that the Worker Service host can send to the Persistence store. 100
worker.persistenceNamespaceMaxQPS Int Maximum number of queries per second that the Worker host can send to the Persistence store for each Namespace.
If the value set for this config is less than or equal to 0, the value set for worker.persistenceMaxQPS will apply.		 0
Visibility
system.visibilityPersistenceMaxReadQPS Int Maximum number queries per second that Visibility database can receive for read operations. 9000
system.visibilityPersistenceMaxWriteQPS Int Maximum number of queries per second that Visibility database can receive for write operations. 9000

Activity and Workflow default policy setting

You can define default values for Activity and Workflow Retry Policies at the Cluster level with the following dynamic configuration keys.

Dynamic configuration key Type Description Default value
history.defaultActivityRetryPolicy Map (key-value pair elements) Server configuration for an Activity Retry Policy when it is not explicitly set for the Activity in your code. Default values for retry Policy
history.defaultWorkflowRetryPolicy Map (key-value pair elements) Retry Policy for unset fields where the user has set an explicit RetryPolicy, but not specified all the fields. Default values for retry Policy

Size limit settings

The Persistence store in the Cluster has default size limits set for optimal performance. The dynamic configuration keys relating to some of these are listed below.

The default values on these keys are based on extensive testing. You can change these values, but ensure that you are provisioning enough database resources to handle the changed values.

For details on platform limits, see the Temporal Platform limits sheet.

Dynamic configuration key Type Description Default value
limit.maxIDLength Int Length limit for various Ids, including: Namespace, TaskQueue, WorkflowID, ActivityID, TimerID, WorkflowType, ActivityType, SignalName, MarkerName, ErrorReason/FailureReason/CancelCause, Identity, and RequestID. 1000
limit.blobSize.warn Int Limit, in bytes, for BLOBs size in an Event when a warning is thrown in the server logs. 512 KB (512 × 1024)
limit.blobSize.error Int Limit, in bytes, for BLOBs size in an Event when an error occurs in the transaction. 2 MB (2 × 1024 × 1024)
limit.historySize.warn Int Limit, in bytes, at which a warning is thrown for the Workflow Execution Event History size. 10 MB (10 × 1024 × 1024)
limit.historySize.error Int Limit, in bytes, at which an error occurs in the Workflow Execution for exceeding allowed size. 50 MB (50 × 1024 × 1024)
limit.historyCount.warn Int Limit, in count, at which a warning is thrown for the Workflow Execution Event History size. 10,240 Events
limit.historyCount.error Int Limit, in count, at which an error occurs in the Workflow Execution for exceeding allowed number of Events. 51,200 events
limit.numPendingActivities.error Int Maximum number of pending Activities that a Workflow Execution can have before the ScheduleActivityTask fails with an error. 2000
limit.numPendingSignals.error Int Maximum number of pending Signals that a Workflow Execution can have before the SignalExternalWorkflowExecution commands from this Workflow fail with an error. 2000
history.maximumSignalsPerExecution Int Maximum number of Signals that a Workflow Execution can receive before it throws an Invalid Argument error. 10000
limit.numPendingCancelRequests.error Int Maximum number of pending requests to cancel other Workflows that a Workflow Execution can have before the RequestCancelExternalWorkflowExecution commands fail with an error. 2000
limit.numPendingChildExecutions.error Int Maximum number of pending Child Workflows that a Workflow Execution can have before the StartChildWorkflowExecution commands fail with an error. 2000
frontend.visibilityMaxPageSize Int Maximum number of Workflow Executions shown from the ListWorkflowExecutions API in one page. 1000

Secondary visibility settings

Secondary visibility configuration keys enable Dual Visibility on your Temporal Cluster. This can be useful when migrating a Visibility database or creating a backup Visibility store.

Dynamic configuration key Type Description Default value
system.enableReadFromSecondaryVisibility Boolean Enables reading from the secondary visibility store, and can be set per Namespace. Allowed values are true or false. false
system.secondaryVisibilityWritingMode Enables writing Visibility data to the secondary Visibility store and can be set per Namespace. Setting this value to on disables write operations to the primary Visibility store. Allowed values:
off: Enables writing to primary Visibility store only.
on: Enables writing to secondary Visibility store only.
dual: Enables writing to both primary and secondary Visibility stores.		 off

Server version check settings

The Temporal server reports the server version and the version of the SDK that it is connected to in order to determine if the Web UI should show a banner that states a new version is available to install. This can be disabled by defining the following value or by setting the TEMPORAL_VERSION_CHECK_DISABLED environment variable to 1.

Dynamic configuration key Type Description Default value
frontend.enableServerVersionCheck Boolean Enables the Temporal server to report version information about the current server and SDK. Allowed values are true or false. true

Nexus settings

Settings related to the management of Nexus.

For more detailed explanation of these configs see the in-repo Nexus architecture doc.

Dynamic configuration key Type Description Default value
system.enableNexus Boolean Enables Nexus Features. Removed in 1.31.0 (Nexus is always enabled). true (since 1.27)
component.nexusoperations.useSystemCallbackURL String Controls whether to use temporal://system as the callback URL generated by the server. (Server 1.30 or newer) false
component.nexusoperations.callback.endpoint.template String Defines the URL template used to construct Nexus callback URLs. (Only required for the experimental external endpoint target feature or servers older than 1.30.) unset
component.callbacks.allowedAddresses Object Defines the security allow-list of callback URL patterns that the server will accept; used to restrict what callback endpoints can be invoked. (Only required for the experimental external endpoint target feature or servers older than 1.30.) unset