Skip to content

dynamic-property-configuration.adoc

Latest commit

 

History

History
200 lines (160 loc) · 8.51 KB

dynamic-property-configuration.adoc

File metadata and controls

200 lines (160 loc) · 8.51 KB
Raw

Dynamic Property Configuration

Table of Contents

Property Supplier

Decaton provides some properties for you to configure how it processes tasks. These properties don’t need to be hard-coded. Decaton lets you configure some of the properties so they can be loaded dynamically.

To externalize property, you need to implement PropertySupplier and configure it for Decaton processor.

Central Dogma Property Supplier

Central Dogma is a highly-available version-controlled service configuration repository. Decaton supports it as a property supplier out-of-the-box. If you already have a Central Dogma server, you only need to configure appropriate CentralDogmaPropertySupplier for the Decaton processor to use it.

Add the decaton-centraldogma dependency to your project. And we will use Central Dogma Java Client library to access Central Dogma here.

build.gradle
dependencies {
    implementation "com.linecorp.decaton:decaton-centraldogma:$DECATON_VERSION"
    implementation "com.linecorp.centraldogma:centraldogma-client-armeria:$CENTRALDOGMA_VERSION"
}

There are two ways to instantiate Central Dogma property supplier:

CentralDogmaPropertySupplier supplier = CentralDogmaPropertySupplier
                .register(centralDogma, PROJECT_NAME, REPOSITORY_NAME, "/properties.json"); // (1)

CentralDogmaPropertySupplier supplier = new CentralDogmaPropertySupplier(
                centralDogma, PROJECT_NAME, REPOSITORY_NAME, "/properties.json"); // (2)

  1. CentralDogmaPropertySupplier.register will create a property file with default values if it doesn’t exist on Central Dogma.

  2. If you already have your property file on Central Dogma, or don’t want it to be created automatically, you can instantiate it with new CentralDogmaPropertySupplier.

After creating a instance of CentralDogmaPropertySupplier, you have to configure it for Decaton processor:

CentralDogmaSupplierMain.java
ProcessorSubscription testProcessor =
        SubscriptionBuilder.newBuilder("testProcessor")
                           /* ... */
                           .properties(supplier)
                           .consumerConfig(consumerConfig)
                           .buildAndStart();

  1. When you instantiate the ProcessorSubscription, use SubscriptionBuilder#properties to set your property supplier.

The following is the full example that demonstrates how to use CentralDogmaPropertySupplier:

CentralDogmaSupplierMain.java
public class CentralDogmaSupplierMain {
    public static void main(String[] args) throws Exception {
        final CentralDogma centralDogma = new ArmeriaCentralDogmaBuilder()
                .host("127.0.0.1")
                .accessToken("accesstoken")
                .build();

        CentralDogmaPropertySupplier supplier = CentralDogmaPropertySupplier
                .register(centralDogma, "project", "repository", "/testProcessor.json");

        // ...

        ProcessorSubscription testProcessor =
                SubscriptionBuilder.newBuilder("testProcessor")
                                   .processorsBuilder(
                                           ProcessorsBuilder.consuming(
                                                   "my-decaton-topic",
                                                   new ProtocolBuffersDeserializer<>(PrintMessageTask.parser()))
                                                            .thenProcess(new PrintMessageTaskProcessor())
                                   )
                                   .properties(supplier)
                                   .consumerConfig(consumerConfig)
                                   .buildAndStart();
    }
}

Use YAML instead of JSON

From decaton v9.4.0, you can store the property file in YAML as well as JSON. Nothing changes in your code except the file‐name extension.

Note you cannot use YAML’s tag, anchor, or alias features. Just you can add comment in the file.

CentralDogmaPropertySupplier supplier =
        CentralDogmaPropertySupplier.register(
                centralDogma,
                "project",
                "repository",
                "/properties.yaml");   // (1)

  1. Use .yaml (or .yml) instead of .json. All other APIs and behaviours remain exactly the same.

You may keep the flat, dot‑separated keys.

# You should use the flat, dot-separated keys like JSON.
decaton.partition.concurrency: 8
decaton.processing.rate.per.partition: 50

As with JSON, you cannot use nested structures in YAML. Therefore, the following is NOT allowed:

# This style is not supported.
decaton:
  partition:
    concurrency: 8
  processing:
    rate:
      per:
        partition: 50

Comments (# like this) are allowed and ignored by decaton.

Multiple Property Suppliers

You can specify multiple property suppliers including one provides hardcoded properties. Suppliers are evaluated from the first to the last in given order where the first occurrence of a property for one definition is adopted. Hence you should put supplier with higher priority before others.

    .properties(
        StaticPropertySupplier.of(
            // This one is adopted even if below centralDogmaPropertySupplier contains the same property
            Property.ofStatic(ProcessorProperties.CONFIG_PARTITION_CONCURRENCY, 100),
            Property.ofStatic(...)),
        centralDogmaPropertySupplier);

Validate your configuration file with JSON Schema

Decaton ships a set of JSON Schema files that precisely describe every key available in CentralDogmaPropertySupplier including each key’s type and default value. Leveraging these schemas in your configuration files gives you two immediate benefits:

  1. IDE assistance: when the file begins with a $schema directive, most modern IDEs (IntelliJ IDEA, VS Code, etc.) will

    • offer auto-completion for property names,

    • flag typos instantly,

    • suggest default values where they exist, and

    • show descriptions for each property.

  2. CI validation: you can run any JSON Schema validator (e.g. ajv-validator/ajv, ajv-validator/ajv-cli, python-jsonschema/jsonschema) in a build or deploy pipeline to prevent invalid configurations from reaching production.

decaton-processor-dynamic-configuration.json
{
  "$schema": "https://raw.githubusercontent.com/line/decaton/vX.Y.Z/centraldogma/src/jsonschema/dist/decaton-processor-properties-central-dogma-schema-draft_7.json",
  "decaton.partition.concurrency": 10000,
  "decaton.processing.rate.per.partition": -1,
  ...
}
decaton-processor-dynamic-configuration.yaml
# $schema: https://raw.githubusercontent.com/line/decaton/vX.Y.Z/centraldogma/src/jsonschema/dist/decaton-processor-properties-central-dogma-schema-draft_7.json
# yaml-language-server: $schema=https://raw.githubusercontent.com/line/decaton/vX.Y.Z/centraldogma/src/jsonschema/dist/decaton-processor-properties-central-dogma-schema-draft_7.json
decaton.partition.concurrency: 10000
decaton.processing.rate.per.partition: -1
...

For example, you can use JSON Schema by adding a $schema directive at the top of the file as shown above follows. Of course, there may be other ways to use it. Replace vX.Y.Z with the exact Decaton version your application depends on. If you prefer living at HEAD, you can also reference master, but pinning to a release tag guarantees repeatable builds.

Json Schema Version

For every release Decaton publishes some schema variants like

  • …​-draft_7.json

  • …​-draft_2019_09.json

  • …​-draft_2020_12.json

Choose the draft that matches the capabilities of the validator you use.

By default, additionalProperties is set to false. If you want to allow additional keys for some reason, you can use the …​-allow-additional-properties.json variant.

You can find your favorite schema variant in the centraldogma/src/jsonschema/dist directory.