Skip to content

INSTALL.md

Latest commit

 

History

History
237 lines (173 loc) · 11.7 KB

INSTALL.md

File metadata and controls

237 lines (173 loc) · 11.7 KB

How to Install

Dynatrace Snowflake Observability Agent comes in the form of a series of SQL scripts (accompanied with a few configuration files), which need to be deployed at Snowflake by executing them in the correct order.

This document assumes you are installing from the distribution package (dynatrace_snowflake_observability_agent-*.zip). If you are a developer and want to build from source, please refer to the CONTRIBUTING.md guide.

Prerequisites

Before you can deploy the agent, you need to ensure the following tools are installed on your system.

Windows Users

On Windows, it is necessary to install Windows Subsystem for Linux (WSL) version 2.0 or higher. The deployment scripts must be run through WSL. See Install WSL guide for more details.

All Users

You will need the following command-line tools:

  • bash: The deployment scripts are written in bash.
  • Snowflake CLI: For connecting to and deploying objects in Snowflake.
  • jq: For processing JSON files.
  • gawk: For text processing.

You can run the included ./setup.sh script, which will attempt to install these dependencies for you.

Alternatively, you can install them manually:

Snowflake CLI

Install using pipx (recommended):

# If you do not have pipx installed, run:
# on Ubuntu/Debian
sudo apt install pipx
# on macOS
brew install pipx

# With pipx installed, run:
pipx install snowflake-cli-labs

Or on macOS with Homebrew:

brew tap snowflakedb/snowflake-cli
brew install snowflake-cli

jq and gawk

On Ubuntu/Debian:

sudo apt install jq gawk

On macOS (with Homebrew):

brew install jq gawk

Deploying Dynatrace Snowflake Observability Agent

The default option to install Dynatrace Snowflake Observability Agent is from the distribution package. To deploy Dynatrace Snowflake Observability Agent, run the ./deploy.sh command:

./deploy.sh $config_name $file_prefix

The $config_name parameter is required and must match one of the previously created configuration files.

The $file_prefix parameter is optional and can take multiple values:

  • If left empty, all SQL scripts will be executed at once.
  • If set to config, the configuration table will be updated using the config file specified by the $config_name parameter.
  • If set to apikey, the Dynatrace Access Token (API Key) will be stored in Snowflake and made available only to the Dynatrace Snowflake Observability Agent stored procedure.
  • If set to manual, a complete SQL script with all commands necessary to deploy and set up Dynatrace Snowflake Observability Agent will be produced; use the Snowflake Web UI to execute it.
  • If set to teardown, Dynatrace Snowflake Observability Agent will be uninstalled from your Snowflake account.
  • If set to any other value, only the files with base names starting with that $file_prefix will be executed as part of the deployment procedure.

You should store the Access Token for your Dynatrace tenant (to which you want to send telemetry from your environment) as the environment variable DTAGENT_TOKEN. The token should have the following scopes enabled:

Scope ID Scope Name Comment
logs.ingest Ingest Logs
metrics.ingest Ingest Metrics
bizevents.ingest Ingest BizEvents
openpipeline.events OpenPipeline - Ingest Events
openTelemetryTrace.ingest Ingest OpenTelemetry Traces
events.ingest Ingest Events Not required version>=0.9.1

We strongly recommend to ensure your token is not recorded in shell script history; please find an example how to define DTAGENT_TOKEN environment variable on Linux or WSL below:

export HISTCONTROL=ignorespace
# make sure to put the space before the next command to ensure that the TOKEN is not recorded in bash history
 export DTAGENT_TOKEN="dynatrace-token"

If you do not set the DTAGENT_TOKEN environment variable, or if it does not contain a valid token value:

  • The Dynatrace Snowflake Observability Agent deployment process WILL NOT send self-monitoring BizEvents to your Dynatrace tenant to mark the start and finish of the deployment process.
  • The deployment process will not be able to set DTAGENT_API_KEY when deploying the complete configuration (./deploy.sh $config_name) or when updating just the API key (./deploy.sh $config_name apikey). In these cases, YOU WILL be prompted to provide the correct DTAGENT_TOKEN value during deployment.
  • The deployment process will not be able to send BizEvents to your Dynatrace tenant to mark the start and finish of the deployment process.

No additional objects need to be provided for the deployment process on the Snowflake side. Dynatrace Snowflake Observability Agent will build a database to store his information - DTAGENT_DB by default or DTAGENT_{TAG}_DB if tag is provided (see Multitenancy).

The complete log of deployment and the script executed during deployment is available as .logs/dtagent-deploy-$config_name-$current_date.log.

Setting up a profile

Before you deploy Dynatrace Snowflake Observability Agent to Snowflake, you need to configure a profile with necessary information to establish connection between single Snowflake account and single Dynatrace account. Dynatrace Snowflake Observability Agent enables to send telemetry from multiple Snowflake accounts to one or multiple (see Multitenancy) Dynatrace account. You will need to create a profile configuration file for each Snowflake-Dynatrace pair.

Creating profile configuration file for Snowflake-Dynatrace connection

You must create the deployment configuration file in the conf/ directory. The file must follow config-$config_name.json naming convention and the content as presented in the conf/config-template.json template. Make sure the CORE entries; you can skip the rest. Optionally you can adjust plugin configurations.

One of the configuration options for each Dynatrace Snowflake Observability Agent plugin is the SCHEDULE, which determines when the Snowflake task responsible for executing this plugin should start. By default, the majority of tasks are scheduled to execute on the hour or half hour to increase warehouse utilization and reduce costs. Dynatrace Snowflake Observability Agent allows you to configure the schedule for each plugin separately using one of three formats:

  • CRON format: USING CRON */30 * * * * UTC
  • Interval format: 30 MINUTES
  • Task graph definition: after DTAGENT_DB.APP.TASK_DTAGENT_QUERY_HISTORY_GRANTS

NOTE: Due to Snowflake limitations, using task graph definition (AFTER) requires all tasks in one graph to be owned by the same role.

Multi-configuration deployment

The configuration file is a JSON array containing multiple configuration definitions — one for each environment. This enables you to deploy to several environments, potentially with similar settings, in a single operation.

HINT: When specifying multiple configurations, the first configuration in the array serves as the base. All subsequent configurations inherit settings from the first, so you only need to specify the differences for each additional environment. This helps reduce duplication of common configuration values.

[
    {
        "CORE": {
            "DEPLOYMENT_ENVIRONMENT": "ENV1",
            /// ...
        },
        "OTEL": {
            /// ...
        },
        "PLUGINS": {
            /// ...
        }
    },
    {
        "CORE": {
            "DEPLOYMENT_ENVIRONMENT": "ENV2"
        }
    }
]

Multitenancy

If you want to deliver telemetry from one Snowflake account to multiple Dynatrace tenants, or require different configurations and schedules for some plugins, you can achieve this by creating configuration with multiple deployment environment (Dynatrace Snowflake Observability Agent instances) definitions. Specify a different CORE.DEPLOYMENT_ENVIRONMENT parameter for each instance. We strongly recommend also defining a unique CORE.TAG for each additional Dynatrace Snowflake Observability Agent instance running on the same Snowflake account.

Example:

[
    {
        //...
        "CORE": {
            //...
            "DEPLOYMENT_ENVIRONMENT": "TEST-MT001",
            //...
            "TAG": "MT001"
        },
        //...
    }
]

You can specify the configuration for each instance in a separate configuration file or use the multi-configuration deployment described above.

IMPORTANT

  • Running ./deploy.sh $config_name expects a conf/config-$config_name.json file with at least one configuration defined.
  • If you have multiple CORE.DEPLOYMENT_ENVIRONMENT values specified in a single configuration file, you will need a matching snow connection profile for each one.
  • If you plan to deploy configurations that send telemetry to different Dynatrace tenants, DO NOT define the DTAGENT_TOKEN environment variable in advance; you will be prompted to provide it separately for each configuration as needed.

Setting up connection to Snowflake

You must add a connection definition to Snowflake using the following command. The connection name must follow this pattern: snow_agent_$config_name. Only the required fields are necessary, as external authentication is used by default. These connection profiles are used ONLY during the deployment process.

To deploy Dynatrace Snowflake Observability Agent properly, the specified user must be able to assume the ACCOUNTADMIN role on the target Snowflake account.

HINT: Running ./setup.sh $config_name or ./deploy.sh $config_name will prompt you to create Snowflake connection profiles named snow_agent_$ENV, where ENV matches each CORE.DEPLOYMENT_ENVIRONMENT in your configuration. If these profiles do not exist, you will be prompted to create them.

WARNING: If you wish to use a different connection name or pattern, you must modify the ./deploy.sh script and update the --connection parameter in the snow sql call.

snow connection add --connection-name snow_agent_$ENV

To list your currently defined connections run:

snow connection list

Here is an example of how to fill in the form to configure connection based on external browser authentication, which is a recommended way for users authenticating with external SSO:

Snowflake account name: ${YOUR_SNOWFLAKE_ACCOUNT_NAME.REGION_NAME}
Snowflake username: ${YOUR_USERNAME}
Snowflake password [optional]: 
Role for the connection [optional]: 
Warehouse for the connection [optional]: 
Database for the connection [optional]: 
Schema for the connection [optional]: 
Connection host [optional]: 
Connection port [optional]: 
Snowflake region [optional]: 
Authentication method [optional]: externalbrowser
Path to private key file [optional]:

You can also run this command to fill in the required and recommended parts:

snow connection add --connection-name snow_agent_$config_name \
                    --account ${YOUR_SNOWFLAKE_ACCOUNT_NAME.REGION_NAME} \
                    --user ${YOUR_USERNAME} \
                    --authenticator externalbrowser

If you have any issues setting up the connection check the SnowCli documentation.