Skip to content

setup-configuration.rst

Latest commit

 

History

History
686 lines (509 loc) · 25.1 KB

setup-configuration.rst

File metadata and controls

686 lines (509 loc) · 25.1 KB
Raw

Configuration

The main configuration file is located at conf/dogma.json, with most properties set to their sensible defaults:

{
  "dataDir": "./data",
  "ports": [
    {
      "localAddress": {
        "host": "*",
        "port": 36462
      },
      "protocols": [
        "http"
      ]
    }
  ],
  "tls": null,
  "trustedProxyAddresses": null,
  "clientAddressSources": null,
  "numWorkers": null,
  "maxNumConnections": null,
  "requestTimeoutMillis": null,
  "idleTimeoutMillis": null,
  "maxFrameLength": null,
  "numRepositoryWorkers": 16,
  "maxRemovedRepositoryAgeMillis": null,
  "repositoryCacheSpec": "maximumWeight=134217728,expireAfterAccess=5m",
  "webAppEnabled": true,
  "webAppTitle": null,
  "gracefulShutdownTimeout": {
    "quietPeriodMillis": 1000,
    "timeoutMillis": 10000
  },
  "replication": {
    "method": "NONE"
  },
  "accessLogFormat": "common",
  "authentication": null,
  "cors": null,
  "pluginConfigs": [
    {
      "type": "com.linecorp.centraldogma.server.mirror.MirroringServicePluginConfig",
      "enabled": true,
      "numMirroringThreads": null,
      "maxNumFilesPerMirror": null,
      "maxNumBytesPerMirror": null,
      "trustedHostKeys": null
    }
  ],
  "management": {
    "address": "127.0.0.1",
    "port": 36463,
    "protocol": null,
    "path": null
  },
  "zone": null,
  "metadataProperties": null
}

Core properties

  • dataDir (string)

    • the path to the data directory which contains the repositories served by Central Dogma and other stateful data. Can be a relative path from the root directory of the distribution.

  • ports

    • the server ports that serve the incoming requests.
    • localAddress - the bind address of server socket.
      • host (string)
        • the IP address or * to bind to all network interfaces.
      • port (integer)
        • the port number. 36462 is preferred.
    • protocols (string array)
      • protocols which are served on the port. http, https and proxy are supported.

  • tls

    • the configuration for Transport Layer Security(TLS) support. Specify null to disable TLS. See :ref:`tls` for more information.

  • trustedProxyAddresses (string array)

    • the addresses or ranges of Classless Inter-domain Routing (CIDR) blocks of trusted proxy servers. e.g. 10.0.0.1 for a single address or 10.0.0.0/8 for a CIDR block. With trustedProxyAddresses and clientAddressSources properties, you can get a client address who initiated a request from the access log. If null or an empty array, the remote address of the connection is used as a client address.

  • clientAddressSources (string array)

    • the HTTP header names to be used for retrieving a client address. PROXY_PROTOCOL is a reserved keyword for getting the source address specified in a PROXY protocol message. By default, forwarded, x-forwarded-for and PROXY_PROTOCOL are used when trustedProxyAddresses is configured. Otherwise, the remote address of the connection is used as a client address.

  • numWorkers (integer)

    • the number of I/O worker threads. If null, the default value of <numCpuCores> * 2 is used.

  • maxNumConnections (integer)

    • the maximum number of TCP/IP connections that can be handled concurrently. Any connection attempts that make the number of connections exceed this value will be rejected immediately. If null, no limit is enforced.

  • requestTimeoutMillis (integer)

    • the maximum number of milliseconds allowed for handling a request. If a request takes more than this, the server may respond with a 503 Service Unavailable response. If null, the default value of '10000 milliseconds' (10 seconds) is used.

  • idleTimeoutMillis (integer)

    • the number of milliseconds before closing an idle connection. The server will close the connection if it stays idle for more than this without any pending requests. If null, the default value of '15000 milliseconds' (15 seconds) is used.

  • maxFrameLength (integer)

    • the maximum length of request content. If a client sends a request whose content is longer than this, the server may respond with a 413 Request Entity Too Large response. If null, the default value of '1048576 bytes' (1 MiB) is used.

  • numRepositoryWorkers (integer)

    • the number of worker threads dedicated to handling repository reads and writes. If null, the default value of '16 threads' is used.

  • maxRemovedRepositoryAgeMillis (integer)

    • the maximum allowed age of removed projects and repositories before they are purged. Set 0 to disable automatic purge. If null, the default value of '604800000 milliseconds' (7 days) is used.

  • repositoryCacheSpec (string)

    • the cache specification string which determines the capacity and behavior of the repository access cache. Refer to the Caffeine API documentation for more information. Note that the weight of the cache has been tuned to be roughly proportional to its memory usage.

  • webAppEnabled (boolean)

    • whether to enable the web-based administrative console. Enabled by default.

  • webAppTitle (string)

    • the title text which is displayed on the navigation bar of the web-based administrative console. If null, the default value of Central Dogma at {{hostname}} is used. Note that {{hostname}} will be replaced with the actual hostname that the server is running on.

  • gracefulShutdownTimeout

    • the amount of time to wait after the initiation of shutdown procedure for requests to go away before the actual shutdown of the server.
    • quietPeriodMillis (integer)
      • the number of milliseconds to wait for active requests to go end before shutting down. 0 means the server will stop right away without waiting.
    • timeoutMillis (integer)
      • the number of milliseconds to wait before shutting down the server regardless of active requests. This should be set to a time greater than quietPeriodMillis to ensure the server shuts down even if there is a stuck request.

  • replication

    • the replication configuration.
    • method (string)
      • the replication method. NONE indicates 'standalone mode' without replication. See :ref:`replication` to learn how to configure ZooKeeper-based multi-master replication.

  • accessLogFormat (string)

    • the format to be used for writing an access log. common and combined are pre-defined for NCSA common log format and NCSA combined log format, respectively. Also, a custom log format can be specified here. Read Writing an access log for more information. Specify null to disable access logging feature.

  • authentication

    • the authentication configuration. If null, the authentication is disabled. See :ref:`auth` to learn how to configure the authentication layer.

  • cors

    • specify null to disable CORS entirely.
    • allowedOrigins (string or string array)
      • the list of origins which are allowed a lenient CORS policy. If the literal value * is specified then all origins are allowed.
    • maxAgeSeconds (integer)
      • how long in seconds the results of a preflight request can be cached. If not specified then the default value 7200 is applied.

  • pluginConfigs

    • the list of plugin configuration. See :ref:`plugins` for more information.

  • management

    • the management server configuration. Read ManagementService API documentation to know more about the management service.
    • port (integer)
      • the port number of the management service. If 0, the management service uses the same port as the main service.
    • address (string)
      • the IP address of the management service. If null, the management will listen to all network interfaces.
      • this option is ignored if port is set to 0.
    • protocol
      • the protocol of the management service. http and https are supported. If not specified, http is used.
      • this option is ignored if port is set to 0.
    • path
      • the path of the management service. If not specified, the management service is mounted at /internal/management.

  • zone

    • the zone information of the server. If not specified, PluginTarget.ZONE_LEADER_ONLY can't be used.
    • currentZone (string)
      • the current zone name. If the value starts with env:, the environment variable is used as the zone name. For example, if the value is env:ZONE_NAME, the environment variable named ZONE_NAME is used as the zone name.
      • You can also dynamically load a zone name by implementing :api:`com.linecorp.centraldogma.server.ConfigValueConverter`.
    • allZones (string array)
      • the list of zone names.
      • the current zone name must be included in the list of zone names.

  • metadataProperties

    • the additional metadata properties of projects, repositories and app identities. Each field is a JSON Schema that the properties of the corresponding resource must conform to at creation time. Properties that are not declared in the schema's top-level properties keyword are silently dropped rather than rejected, so that a new property can be declared with a rolling restart. If the schema declares its shape in another way (e.g. $ref or allOf), nothing is dropped and the whole object is validated as is. If not specified, the creation APIs behave as before.

    • all replicas must be running a binary that understands metadataProperties before this section is enabled. Also note that declaring a required property for appIdentity effectively disables the deprecated POST /api/v1/tokens endpoint, which cannot carry properties.

    • project / repo / appIdentity (object)

      • the JSON Schema for the properties of a project, a repository and an app identity respectively. For example:

        {
  "metadataProperties": {
    "project": {
      "type": "object",
      "properties": {
        "serviceId": { "type": "string" }
      },
      "required": [ "serviceId" ]
    }
  }
}

      • the declared schemas are exposed via GET /api/v1/metadataProperties so that clients such as the web UI can render input forms for the declared properties. The web UI uses the standard title and description keywords of each property as the form label and its help text.

Configuring replication

Central Dogma features multi-master replication which allows a client to push commits to any of the available replicas, and thus it’s possible to update the settings of your application even when all replicas but one are down. The clients will automatically connect to an available replica.

Note

Central Dogma implements multi-master replication by embedding Apache ZooKeeper. You may find it useful to have some prior administrative knowledge of ZooKeeper although it is not required. For more information about ZooKeeper administration, see ZooKeeper administrator's guide

To enable replication, you need to update the replication section of conf/dogma.json. The following example shows the configuration of the first replica in a 3-replica cluster:

{
  ...
  "replication" : {
    "method": "ZOOKEEPER",
    "serverId": 1,
    "servers": {
      "1": {
        "host": "replica1.example.com",
        "quorumPort": 36463,
        "electionPort": 36464,
        "groupId": null,
        "weight": null
      },
      "2": {
        "host": "replica2.example.com",
        "quorumPort": 36463,
        "electionPort": 36464,
        "groupId": null,
        "weight": null
      },
      "3": {
        "host": "replica3.example.com",
        "quorumPort": 36463,
        "electionPort": 36464,
        "groupId": null,
        "weight": null
      }
    },
    "secret": "JqJAkZ!oZ6MNx4rBpIH8M*yuVWXDULgR",
    "additionalProperties": {},
    "timeoutMillis": null,
    "numWorkers": null,
    "maxLogCount": null,
    "minLogAgeMillis": null
  }
}

  • method (string)

    • the replication method. ZOOKEEPER indicates Central Dogma will provide multi-master replication by embedding Apache ZooKeeper.

  • serverId (integer)

    • the unique positive integer ID of the replica. Be careful not to use a duplicate ID or not to change this value after joining the cluster. If null or unspecified, the serverId is auto-detected from the server list in the servers section.

      Note

      Internally, this value is used as the myid of the embedded ZooKeeper peer.

  • servers

    • a map whose key is the serverId of a replica in the cluster and whose value is a map which contains the properties required to connect to each other:

      • host (string)
        • the host name or IP address of the replica
      • quorumPort (integer)
        • the TCP/IP port number which is used by ZooKeeper for reaching consensus
      • electionPort (integer)
        • the TCP/IP port number which is used by ZooKeeper for leader election
      • groupId (integer)
        • the group ID which is used by ZooKeeper for hierarchical quorums If null or unspecified, hierarchical quorums are disabled.
      • weight (integer)
        • the weight of the replica which is used by ZooKeeper for hierarchical quorums If null or unspecified, 1 is used by default. If groupId is null, this value will be ignored.

    • It is highly recommended to have more than 3, preferably odd number of, replicas because the consensus algorithm requires more than half of all replicas to agree with each other to function correctly. If you had 2 replicas, losing just one replica would make your cluster stop to function.

      Note

      See here or here if you are curious why odd number of replicas are preferred over even number of replicas.

  • secret (string)

    • the secret string which is used for replicas to authenticate each other. The replicas in the same cluster must have the same secret. This property is required when ZooKeeper replication is enabled. The secret must be at least 16 characters long. Use openssl rand -hex 32 to generate a secure random secret. You can secure this property by :ref:`hiding_sensitive_property_values`.

  • allowInsecureSecret (boolean)

    • if true, allows the use of an insecure or missing secret for backward compatibility. When enabled with no secret set, the legacy default value is used with a warning. This property is deprecated and will be removed in a future release. It is strongly recommended to set a proper secret instead of relying on this flag.

  • additionalProperties (map of string key-value pairs)

    • ZooKeeper configuration properties. Currently, initLimit, syncLimit, tickTime, syncEnabled, autopurge.snapRetainCount, autopurge.purgeInterval, and quorumListenOnAllIPs can be set. It is recommended to leave this properties empty because Central Dogma sets the sensible defaults.

  • timeoutMillis (integer)

    • the ZooKeeper timeout, in milliseconds. If null or unspecified, the default value of '1000 milliseconds' (1 second) is used.

  • numWorkers (integer)

    • the number of worker threads dedicated for replication. If null or unspecified, the default value of '16 threads' is used.

  • maxLogCount (integer)

    • the maximum number of log items to keep in ZooKeeper. Note that the log entries will still not be removed if they are younger than minLogAgeMillis. If null or unspecified, the default value of '1024 log entries' is used.

  • minLogAgeMillis (integer)

    • the minimum allowed age of log items before they are removed from ZooKeeper. If null or unspecified, the default value of '86400000 milliseconds' (1 day) is used.

Configuring TLS

Central Dogma supports TLS for its API and web pages. To enable TLS, a user may configure tls property in dogma.json as follows.

{
  "dataDir": "./data",
  "ports": [
    {
      "localAddress": {
        "host": "*",
        "port": 36462
      },
      "protocols": [
        "https"
      ]
    }
  ],
  "tls": {
    "keyCertChain": "file:./cert/centraldogma.crt",
    "key": "file:./cert/centraldogma.key",
    "keyPassword": null
  },
  "trustedProxyAddresses": null,
  "clientAddressSources": null,
  "numWorkers": null,
  "maxNumConnections": null,
  "requestTimeoutMillis": null,
  "idleTimeoutMillis": null,
  "maxFrameLength": null,
  "numRepositoryWorkers": 16,
  "repositoryCacheSpec": "maximumWeight=134217728,expireAfterAccess=5m",
  "webAppEnabled": true,
  "webAppTitle": null,
  "gracefulShutdownTimeout": {
    "quietPeriodMillis": 1000,
    "timeoutMillis": 10000
  },
  "replication": {
    "method": "NONE"
  },
  "accessLogFormat": "common",
  "authentication": null
}
  • tls
    • the configuration for TLS support. It will be applied to the port which is configured with https protocol. If null, a self-signed certificate will be generated for https protocol.
    • keyCertChain (string)
    • key (string)
    • keyPassword (string)
      • the password of the private key file. Specify null if no password is set. Note that null (no password) and "null" (password is 'null') are different. You can secure this property by :ref:`hiding_sensitive_property_values`.

If you run your Central Dogma with TLS, you need to enable TLS on the client side as well. In case of Java client, call the useTls() method when building a CentralDogma instance:

CentralDogma dogma = new ArmeriaCentralDogmaBuilder()
        .host("centraldogma.example.com", 8443)
        .accessToken("appToken-********")
        .useTls()
        .build();

Configuring plugins

Central Dogma supports installing a plugin that runs on Central Dogma servers. You can configure the plugin with pluginConfigs property in dogma.json as follows.

{
  "dataDir": "./data",
  "ports": [
    {
      "localAddress": {
        "host": "*",
        "port": 36462
      },
      "protocols": [
        "https"
      ]
    }
  ],
  ...
  "pluginConfigs": [
    {
      "type": "com.linecorp.centraldogma.server.mirror.MirroringServicePluginConfig",
      "enabled": true,
      "numMirroringThreads": null,
      "maxNumFilesPerMirror": null,
      "maxNumBytesPerMirror": null,
      "zonePinned": false,
      "trustedHostKeys": {
        "github.com": [
          "SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU"
        ]
      }
    }
  ]
}

Each configuration in pluginConfigs must have type that specifies the fully qualified class name of the plugin configuration class. The plugin configuration class must implement the PluginConfig interface that requires enabled property. The plugin configuration class can have additional properties that are specific to the plugin. The example above shows the configuration of the mirroring plugin and here are the properties that can be configured:

  • numMirroringThreads (integer)

    • the number of worker threads dedicated to periodic mirroring tasks. If null, the default value of '16 threads' is used.

  • maxNumFilesPerMirror (integer)

    • the maximum allowed number of files in a mirror. If a Git repository contains more files than this, Central Dogma will reject to mirror the Git repository. If null, the default value of '8192 files' is used.

  • maxNumBytesPerMirror (integer)

    • the maximum allowed number of bytes in a mirror. If the total size of the files in a Git repository exceeds this, Central Dogma will reject to mirror the Git repository. If null, the default value of '33554432 bytes' (32 MiB) is used.

  • zonePinned (boolean)

    • whether the mirroring plugin is pinned to a specific zone. If true, a mirroring task will be executed only in the specified zone. If false, the plugin will be executed in the leader replica.

  • trustedHostKeys (object)

    • a map of hostname to a list of trusted SSH host key fingerprints (SHA-256) for git+ssh:// mirrors. When an SSH mirror connects to a remote server, the server's host key fingerprint is verified against this list. If the fingerprint does not match any trusted key, the connection is rejected.

      If null or empty, SSH mirrors will accept any host key without verification and log a warning. It is strongly recommended to configure this property for all SSH mirror hosts.

      You can obtain a host's fingerprint by running:

      ssh-keyscan <hostname> 2>/dev/null | ssh-keygen -lf - -E sha256

      Example:

      "trustedHostKeys": {
  "github.com": [
    "SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU",
    "SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s"
  ],
  "git.example.com": [
    "SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8"
  ]
}

For more information about mirroring, refer to :ref:`mirroring`.

Hiding sensitive property values

Central Dogma supports property value substitution through the use of :api:`com.linecorp.centraldogma.server.ConfigValueConverter`. If you want to hide sensitive information such as tls.keyPasswords, you can implement the converting logic with your own :api:`com.linecorp.centraldogma.server.ConfigValueConverter` and register it via SPI:

  • Your configuration file:
{
  "tls": {
    "keyCertChain": "file:./cert/centraldogma.crt",
    "key": "file:./cert/centraldogma.key",
    "keyPassword": "encryption:encrypted-password"
  }
}
  • Your converter:
import com.linecorp.centraldogma.server.ConfigValueConverter;

public class MyConfigValueConverter implements ConfigValueConverter {
    @Override
    public List<String> supportedPrefixes() {
        return List.of("encryption");
    }

    @Override
    public String convert(String prefix, String value) {
        assert "encryption".equals(prefix);
        if ("encrypted-password".equals(value)) {
            // return the decrypted password that is stored in a safe place such as Vault.
        }
        ...
    }
}

This feature enables you to enhance the security of your Central Dogma configuration by avoiding the exposure of sensitive information. Below is a list of properties that can be substituted:

  • replication.secret
  • tls.keyCertChain
  • tls.key
  • tls.keyPassword
  • authentication.properties.keyStore.password (when SAML is used.)
  • authentication.properties.keyStore.keyPasswords (when SAML is used.)