Skip to content

Troubleshooting Guide

Jump to bottom
Geri Jennings edited this page May 13, 2020 · 26 revisions

Note: devs work to maintain the following pages with up-to-date content as much as possible. If you are a Secretless user, your best bet for current info is to read the Secretless docs. The content below may be up-to-date with the code on master, or it may be lagging slightly behind. The documentation will always have up-to-date content for the latest stable release.

Table of Contents

Introduction

In each of the troubleshooting scenarios below, we describe the symptoms of the problem, the known causes, and strategies for resolution. Diagnosing the problem often requires looking at the client logs or at the Secretless logs. If you are running Secretless in a container, you access the logs the same way you access any container logs - by running docker logs [container ID] if you are running locally, or using kubectl logs if you are running in a Kubernetes-based environment.

NOTE It is expected that you will not be running Secretless in debug mode for production. In the Symptoms section for any given troubleshooting scenario you might see logs with the prefix [DEBUG]. Such logs can only be made available by running Secretless in debug mode.

Troubleshooting Configuration Problems

General Configuration Problems

Using a Secretless version that doesn't support v2 config

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/07/01 12:42:28 Trying to load configuration file: /etc/secretless/secretless.yml
2019/07/01 12:42:28 Failed to initialize configuration manager 'configfile': Unable to parse configuration: 'yaml: unmarshal errors:
  line 1: field version not found in type config.Config
  line 3: field services not found in type config.Config'

Known Causes

If you are using a v2 config file with an older version of Secretless, you will see an error and Secretless will be unable to start with your config.

Resolution

You can update your deployment to use a version of Secretless with version greater than or equal to v0.8.0 and redeploy your application.

Configuration not specified and no configuration file found in expected locations

Symptoms

  • In the Secretless logs, you can see that it attempts to find the configuration file but is unsuccessful, and the process dies on Failed to initialize configuration manager 
     2019/10/25 12:28:32 Secretless v1.2.0-906f9eb starting up...
 2019/10/25 12:28:32 Initializing health check on :5335...
 2019/10/25 12:28:32 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
 2019/10/25 12:28:32 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
 2019/10/25 12:28:32 Trying to load ./secretless.yml...
 2019/10/25 12:28:32 WARN: Could not load ./secretless.yml. Skipping...
 2019/10/25 12:28:32 Trying to load /home/secretless/.secretless.yml...
 2019/10/25 12:28:32 WARN: Could not load /home/secretless/.secretless.yml. Skipping...
 2019/10/25 12:28:32 Trying to load /etc/secretless.yml...
 2019/10/25 12:28:32 WARN: Could not load /etc/secretless.yml. Skipping...
 2019/10/25 12:28:32 ERROR: Unable to locate any working configuration files

Known Causes

No configuration file was provided in an expected location accessible to the Secretless process, and no alternative configuration mechanism was specified.

Resolution

Ensure that you have a readable and valid configuration in one of these paths:

  • ./secretless.yml
  • $HOME/.secretless.yml
  • /etc/secretless.yml

Configuration data in wrong format

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 13:24:38 Secretless v1.2.0-906f9eb starting up...
2019/10/25 13:24:38 Initializing health check on :5335...
2019/10/25 13:24:38 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 13:24:38 Trying to load configuration file: secretless.yml
2019/10/25 13:24:38 unable to load configuration when parsing version 2: 'yaml: unmarshal errors:
  line 4: cannot unmarshal !!seq into map[string]*v2.serviceYAML'

Known Causes

The configuration you've provided to Secretless is in an unexpected format.

Resolution

Ensure that you have a valid configuration in the provided path that was passed to Secretless Broker (-f <path> parameter). Log output should indicate what location the broker is trying to read.

File Configuration Manager Problems

Configuration specified but file not found

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 13:25:44 Secretless v1.2.0-906f9eb starting up...
2019/10/25 13:25:44 Initializing health check on :5335...
2019/10/25 13:25:44 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 13:25:44 Trying to load configuration file: notthere.yml
2019/10/25 13:25:44 error reading config file 'notthere.yml': 'open notthere.yml: no such file or directory'

Known Causes

Your configuration file is missing, is located at an unexpected path, or is unreadable by the Secretless process. Alternatively, you may have intended to provide an alternate file location or configuration source, but did not specify to the Secretless process using command line flags.

Resolution

Ensure that you have a readable and valid configuration in the provided path that was passed to secretless broker (-f <path> parameter). Log output should indicate what location the broker is trying to read.

Invalid configuration supplied for an earlier version of Secretless

Symptoms

  • On startup, Secretless hangs before starting any connectors. The Secretless logs look something like:

    2020/01/09 13:42:54 Secretless v1.4.1 starting up...
2020/01/09 13:42:54 Initializing health check on :5335...
2020/01/09 13:42:54 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2020/01/09 13:42:54 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2020/01/09 13:42:54 Trying to load configuration file: /secretless.yml
2020/01/09 13:42:54 Registering reload signal listeners...

    and there are no log lines that look like

    2020/01/09 13:54:19 [WARN]  Starting HTTP listener on 0.0.0.0:8080...

    You may also have trouble if you attempt to open a new connection.

Known Causes

Your configuration is invalid, and you are using a version of Secretless that supports v2 config pre-v1.4.2.

Resolution

Review your configuration to ensure it is valid. Upgrade Secretless to v1.4.2+. If you attempt to rerun Secretless using v1.4.2+ and your invalid configuration, you will see a fatal error that explains what is wrong with your configuration. For example, you may see something like this:

2020/01/09 13:54:19 [ERROR] Fatal error in 'HTTP Proxy on tcp://0.0.0.0:8080': unable to create HTTP proxy service on 'tcp://0.0.0.0:8080': 'authenticateURLsMatching' key has incorrect type, must be a string or list of strings

Kubernetes Configuration CRD Problems

Missing custom resource (CR)

Symptoms

  • You see the following output in your Secretless logs that looks something like: 
    2019/10/28 12:25:28 Secretless v1.2.0-906f9eb starting up...
2019/10/28 12:25:28 Initializing health check on :5335...
2019/10/28 12:25:28 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 12:25:28 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 12:25:28 k8s/crd: Using home dir config...
2019/10/28 12:25:28 k8s/crd: Registering CRD watcher...
2019/10/28 12:25:28 k8s/crd: Using home dir config...
2019/10/28 12:25:28 [INFO]  Waiting for new configuration...

Note: The last line in this output will not be visible without turning on the -debug flag!

Known Causes

Your configuration CRD is not accessible to Secretless or is not available under the expected name.

Resolution

  • Ensure that you have a Secretless CRD defined and accessible in the same namespace as the Secretless broker (kubectl get crd configurations.secretless.io).
  • Ensure that the name provided as the custom resource (CR) name matches the one defined in Kubernetes.

Note: Secretless broker will wait until a readable configuration matching the exact name is available and readable. You should usually not need to restart the broker unless there is a permission problem.

Custom resource (CR) configuration is not valid

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/28 12:31:30 Secretless v1.2.0-906f9eb starting up...
2019/10/28 12:31:30 Initializing health check on :5335...
2019/10/28 12:31:30 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 12:31:30 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 12:31:30 k8s/crd: Using home dir config...
2019/10/28 12:31:30 k8s/crd: Registering CRD watcher...
2019/10/28 12:31:30 k8s/crd: Using home dir config...
2019/10/28 12:31:39 k8s/crd: Add configuration event
2019/10/28 12:31:39 secretless-example-config2
2019/10/28 12:31:39 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 12:31:39 k8s/crd: WARN: New CRD could not be turned into a config.Config!

Known Causes

The configuration you've provided to Secretless is in an unexpected format.

Resolution

Ensure that the custom resource (CR)definition provided to Secretless broker is a valid configuration. You can use kubectl get configurations.secretless.io <name> --output=yaml to see what the current resource definition (CR) looks like to ensure validity.

Troubleshooting Connector Problems

General Connector Problems

Service connector not found

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 19:42:48 Secretless v1.2.0-906f9eb starting up...
2019/10/25 19:42:48 Initializing health check on :5335...
2019/10/25 19:42:48 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 19:42:48 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/25 19:42:48 [INFO]  Adding 'my_service_connector.so' as a plugin...
2019/10/25 19:42:48 [INFO]  Loading plugin 'my_service_connector'...
2019/10/25 19:42:48 [ERROR] my_service_connector: PluginInfo['type'] of 'not_connector.not_supported' is not supported
2019/10/25 20:01:17 [ERROR] my_service_connector: plugin 'my_service_connector' (API v0.1.2) is not a supported API version (v0.1.0)
2019/10/25 19:42:48 Trying to load configuration file: ./secretless.yml
2019/10/25 19:42:48 [INFO]  Waiting for new configuration...
2019/10/25 19:42:48 [DEBUG] Got new configuration
2019/10/25 19:42:48 Registering reload signal listeners...
2019/10/25 19:42:48 [INFO]  Validating services against available plugins: ssh,ssh-agent,pg,mysql,aws,basic_auth,conjur
2019/10/25 19:42:48 Failed to start services: services validation failed: my_service_name: missing service connector "my_service_connector".
2019/10/25 19:42:48 Registering reload signal listeners...

Known Causes

Secretless is unable to find a service connector in the collection of internal and external plugins.

Resolution

  • Ensure the .so file is in the specified plugins directory. A log of this form should be present to indicate that the plugin shared object file has been loaded: 
    2019/10/25 19:42:48 [INFO]  Adding 'my_service_connector.so' as a plugin...
2019/10/25 19:42:48 [INFO]  Loading plugin 'my_service_connector'...
  • Ensure a value for the id key is specified in the PluginInfo map otherwise the service connector will be silently ignored.
  • Ensure the value associated with the pluginAPIVersion key of the PluginInfo map matches the API version of the Secretless binary you're running. If it does not match you'll see a log of the form: 
    2019/10/25 20:01:17 [ERROR] my_service_connector: plugin 'my_service_connector' (API v0.1.2) is not a supported API version (v0.1.0)
  • Ensure the symbol PluginInfo is present, else you'll also see a log of the form: 
    2019/10/25 20:06:59 [ERROR] my_service_connector: plugin: symbol PluginInfo not found in plugin plugin/*
  • Ensure the symbol GetXXXPlugin is present, else you'll also see a log of the form: 
    2019/10/25 20:14:19 [ERROR] my_service_connector: plugin: symbol GetXXXPlugin not found in plugin plugin/*

Socket/port in use

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 14:01:16 Secretless v1.2.0-906f9eb starting up...
2019/10/25 14:01:16 Initializing health check on :5335...
2019/10/25 14:01:16 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 14:01:16 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/25 14:01:16 Trying to load configuration file: /secretless.yml
2019/10/25 14:01:16 Registering reload signal listeners...
2019/10/25 14:01:16 [PANIC] unable to create TCP service 'backend_staging': listen tcp 0.0.0.0:2222: bind: address already in use

Known Causes

Secretless is unable to configure itself to listen on a socket in its configuration because that socket is already in use by some other process.

Resolution

Ensure that the noted socket file or port is not in use already by another program

Unable to open socket/port

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 13:04:09 Secretless v1.2.0-906f9eb starting up...
2019/10/25 13:04:09 Initializing health check on :5335...
2019/10/25 13:04:09 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 13:04:09 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/25 13:04:09 Trying to load configuration file: /secretless.yml
2019/10/25 13:04:09 [PANIC] unable to create TCP service 'backend_production': listen tcp 0.0.0.0:1: bind: permission denied
panic: [PANIC] unable to create TCP service 'backend_production': listen tcp 0.0.0.0:1: bind: permission denied

Known Causes

Secretless attempted to open the socket specified in its configuration, but was unable to.

Resolution

  • Ensure that the user has sufficient permissions to create files (if using socket files) and permissions to open ports (if using TCP ports). You must have root privileges on *nix platforms to open ports lower than 1024.
  • Ensure that listening address and port combinations are valid in the configuration

Unreachable backend server

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/25 14:08:17 Secretless v1.2.0-906f9eb starting up...
2019/10/25 14:08:17 Initializing health check on :5335...
2019/10/25 14:08:17 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/25 14:08:17 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/25 14:08:17 Trying to load configuration file: /secretless.yml
2019/10/25 14:08:17 Registering reload signal listeners...
2019/10/25 14:08:21 Instantiating provider 'literal'
2019/10/25 14:08:21 [ERROR] backend_production: Failed on handle connection: failed on connect: dial tcp 127.0.0.1:1234: connect: connection refused
  • Sample PostgreSQL client log output messages: 
    psql: FATAL:  dial tcp 127.0.0.1:5436: connect: connection refused

    psql: FATAL:  dial tcp: missing address
  • Sample MySQL client log output messages: 
    ERROR: MySQL Error 2000 (HY000): #HY000dial tcp: lookup localhosts: no such host

    ERROR: MySQL Error 2000 (HY000): #HY000dial tcp 127.0.0.1:5433: connect: connection refused

Known Causes

The target service is unreachable at the expected location.

Resolution

  • Ensure that host and port fields in Secretless configuration point to a reachable server by connecting directly to it
  • Ensure that Secretless outgoing connections are not blocked

Generic HTTP Connector Problems

Connecting to Secretless as an HTTPS proxy

Symptoms

  • Your client receives a "405 Method Not Allowed" response when attempting to connect to the target service via Secretless. Sample client log output messages:

    curl:

    *   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 62160 (#0)
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 405 Method Not Allowed
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 06 May 2020 17:39:19 GMT
< Content-Length: 26
<
* Received HTTP code 405 from proxy after CONNECT
* Closing connection 0
curl: (56) Received HTTP code 405 from proxy after CONNECT

    Go client:

    Get https://httpbin.org/anything: Method Not Allowed

  • When running in "debug" mode, you see a CONNECT request in your Secretless logs that looks something like:

    2020/05/06 18:39:19 [DEBUG] Got request  httpbin.org:443 CONNECT //httpbin.org:443

Known Causes

This type of error occurs when the client attempts to use Secretless as an HTTPS proxy. Secretless can only act as an HTTP proxy. This error can happen for a few reasons:

  1. An explicit attempt to use Secretless as an HTTPS proxy
  2. Providing the client an HTTPS target when intending to proxy the connection through Secretless might result in the client attempting to use Secretless as an HTTPS proxy, as is the case with Go's standard library HTTP client.

Resolution

  • Ensure that target of your request is HTTP only; that is, rather than telling your client to connect to "https://httbin.org", tell it to connect to "http://httpbin.org". When using Secretless to connect to an HTTP-based target service, you may also drop the prefix altogether, e.g. direct your client to connect to "httpbin.org" with the HTTP proxy set to address / port of the configured Secretless connector.
  • Secretless does not support HTTPS between the client and Secretless, though it does support it between Secretless and the target. Do not use Secretless as an HTTPS proxy.
  • To make the connection between Secretless and the target an HTTPS connection you must set forceSSL: true in the Secretless service connector configuration.

HTTPS certificate verification failure when forceSSL is set

Symptoms

  • Your client receives a certificate error when attempting to connect to the target service via Secretless, such as "x509: certificate signed by unknown authority". Sample client log output messages:

    curl:

    * Rebuilt URL to: http://self-signed.badssl.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 62165 (#0)
> GET http://self-signed.badssl.com/ HTTP/1.1
> Host: self-signed.badssl.com
> User-Agent: curl/7.54.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 06 May 2020 17:39:20 GMT
< Content-Length: 46
<
{ [46 bytes data]
* Connection #0 to host 127.0.0.1 left intact
x509: certificate signed by unknown authority

    Go client:

    Status:
503 Service Unavailable

Body:
x509: certificate is valid for *.badssl.com, badssl.com, not wrong.host.badssl.com

  • When running Secretless in "debug" mode, you see an x509 certificate error in your Secretless logs that looks something like:

    2020/05/06 18:39:19 [DEBUG] Got request / self-signed.badssl.com GET http://self-signed.badssl.com/
2020/05/06 18:39:19 [DEBUG] Using connector 'test' for request http://self-signed.badssl.com/
2020/05/06 18:39:20 [DEBUG] Error: x509: certificate signed by unknown authority

Known Causes

This type of error occurs when the client attempts to connect to a target with a self-signed certificate, and there is some failure on verification. Secretless verifies all HTTPS connections to the target.

There are several reasons why verification might fail including:

  1. The signer of the target's certificate is not a CA that Secretless trusts
  2. The target's certificate is expired or is not yet valid
  3. The target's certificate is not valid for the host

Resolution

This type of error can be broken into 2 categories.

  1. The signer of the target's certificate is not a trusted CA
  2. All other issues

For (2), you must ensure that the target's certificate is valid and matches the host.

For (1) you will need to ensure that Secretless is aware of the root certificate authority (CA) that it should use to verify the server certificates when proxying requests. To do this, ensure that the SECRETLESS_HTTP_CA_BUNDLE environment variable is set in the Secretless runtime environment. The SECRETLESS_HTTP_CA_BUNDLE environment variable gives a path to the bundle of CA certificates that are appended to the certificate pool that Secretless uses for server certificate verification of all HTTP service connectors.

PostgreSQL Connector Problems

Bad/incompatible PostgreSQL client settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up...
2019/10/30 11:53:19 Initializing health check on :5335...
2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 11:53:19 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml
2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml
2019/10/30 11:53:19 Registering reload signal listeners...
2019/10/30 11:53:34 Instantiating provider 'literal'
2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: SSL not supported
  • Sample client log output messages: 
    psql: FATAL:  SSL not supported

Known Causes

Generally you see an error like this if you attempt to open the client to Secretless connection without disabling SSL.

Resolution

PG connections to Secretless from the client must have sslmode set to disabled (e.g. psql "host=pg-host port=5555 sslmode=disable dbname=postgres"). Ensure that sslmode is set to connect in this manner.

Incompatible PostgreSQL server settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up...
2019/10/30 11:53:19 Initializing health check on :5335...
2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 11:53:19 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml
2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml
2019/10/30 11:53:19 Registering reload signal listeners...
2019/10/30 11:53:34 Instantiating provider 'literal'
2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: the backend does not allow SSL connections
  • Sample client log output messages: 
    psql: FATAL:  the backend does not allow SSL connections

Known Causes

These errors can occur when the PostgreSQL server is unreachable at the configured host/port or when its configuration is incompatible with the configuration you provided the client and/or Secretless.

Resolution

  • Ensure that the backend details for PG in the Secretless configuration match the connection details for your database.
  • Attempt to connect directly to your database using the credentials to ensure that the backend is working properly.

Bad PostgreSQL authentication details

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up...
2019/10/30 11:53:19 Initializing health check on :5335...
2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 11:53:19 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml
2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml
2019/10/30 11:53:19 Registering reload signal listeners...
2019/10/30 11:53:34 Instantiating provider 'literal'
2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: pg: FATAL: password authentication failed for user "username"
  • Sample client log output messages 
    psql: FATAL:  password authentication failed for user "username"

Known Causes

This error can occur when you have provided Secretless with invalid credentials for the service in the configuration.

Resolution

  • Ensure that authentication details for your connection to Secretless in the configuration are valid.
  • Attempt to connect directly to the database using those connection details to ensure that they are correct.

Missing or Invalid Port for PostgreSQL

Symptoms

  • Your client displays an error because the port is not provided in the configuration:

    $ psql "host=localhost port=5432 sslmode=disable dbname=my_db"
psql: FATAL:  dial tcp: address HOST: missing port in address

  • Your client hangs as Secretless tries to connect. It eventually times out and displays an error because an invalid port was provided in the configuration:

    $ psql "host=localhost port=5432 sslmode=disable dbname=my_db"
psql: FATAL:  dial tcp HOST:INVALID_PORT: connect: operation timed out

Known Causes

You have not provided a valid port to Secretless.

Resolution

Ensure that the address provided in the Secretless configuration is of the form host:port where port is nonempty and valid for the PostgreSQL server

MySQL Connector Problems

Bad/incompatible MySQL client settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:27 [ERROR] pg-tcp: Failed on handle connection: failed on connect: ERROR: 2026 (HY000): SSL connection error: SSL is required but the server doesn't support it
  • Sample client log output messages 
    ERROR: MySQL Error 2026 (HY000): SSL connection error: SSL is required but the server doesn't support it

Known Causes

Generally you see an error like this if you attempt to open the client to Secretless connection without disabling SSL.

Resolution

MySQL connections to Secretless from the client must have ssl-mode set to disabled (e.g. mysql --ssl-mode=DISABLED ...). Ensure that ssl-mode is set to connect in this manner.

Incompatible MySQL server settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:27 [ERROR] pg-tls: Failed on handle connection: failed on connect: dial tcp X.X.X.X:XXXX: connect: connection refused
  • Sample client log output messages 
    ERROR 2000 (HY000): #HY000dial tcp X.X.X.X:XXXX: connect: connection refused
    or 
    ERROR 2013 (HY000): Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110 "Connection timed out"

Known Causes

These errors can occur when the MySQL server is unreachable at the configured host/port or when its configuration is incompatible with the configuration you provided the client and/or Secretless.

Resolution

  • Ensure that the backend details for MySQL in Secretless configuration match the connection details for your database.
  • Attempt to connect directly to your database using the credentials to ensure that the backend is working properly.

Bad MySQL authentication details

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:27 [ERROR] serivce_name: Failed on handle connection: failed on connect: ERROR: 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES)
  • Sample client log output messages 
    ERROR 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES)

Known Causes

This error can occur when you have provided Secretless with invalid credentials for the service in the configuration.

Resolution

  • Ensure that authentication details for your connection to Secretless in the configuration are valid.
  • Attempt to connect directly to the database using those connection details to ensure that they are correct.

Unsupported MySQL version

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:27 [ERROR] serivce_name: Failed on handle connection: failed on connect: ERROR: 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES)
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000EOF

Known Causes

You are using a version of MySQL that is not currently supported by Secretless.

Resolution

Ensure that your backend is a supported version of MySQL by Secretless

Missing or Invalid Port for MySQL

Symptoms

  • You see an error in your Secretless logs because the port was not provided that looks something like:

    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:27 [ERROR] serivce_name: Failed on handle connection: failed on connect: dial tcp XX.XX.XX.XX:INVALID_PORT: connect: connection refused

  • Your client hangs as Secretless tries to connect and eventually times out with an error, because Secretless was provided with an invalid port in its configuration:

    $ mysql -h localhost -P 3306 -D my_db
Creating a Session to 'myuser@localhost:3306/my_db'
Enter password:
ERROR: 2000 (HY000): #HY000dial tcp HOST:INVALID_PORT: connect: operation timed out

Known Causes

You have not provided a valid port to Secretless.

Resolution

Ensure that port is provided in the Secretless configuration and is nonempty and valid for the MySQL server

MsSQL Connector Problems

Bad/incompatible MsSQL client settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    [00] 2020/01/13 19:31:44 [WARN]  Starting TCP listener on 0.0.0.0:2223...
[00] 2020/01/13 19:31:44 [INFO]  mssql: Starting service
[00] 2020/01/13 19:31:44 [INFO]  Waiting for new configuration...
[00] 2020/01/13 19:44:11 Instantiating provider 'literal'
[00] 2020/01/13 19:44:11 [INFO]  mssql: New connection on x.x.x.x:2223.
[00] 2020/01/13 19:44:11 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:0': dial tcp x.x.x.x:0: connect: connection refused
  • Sample client log output messages 
    Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : SSL Provider: [error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:self signed certificate].
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Client unable to establish connection.

Known Causes

Generally you see an error like this if you attempt to open the client to Secretless connection without disabling SSL.

Resolution

MsSQL connections to Secretless from the client must have ssl-mode set to disabled (e.g. do not use -N when using SQLCMD). Ensure that ssl-mode is set to connect in this manner.

Incompatible MsSQL server settings

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    [00] 2020/01/13 21:19:34 Secretless v1.4.2-dev starting up...
...
[00] 2020/01/13 21:19:34 [WARN]  Starting TCP listener on 0.0.0.0:2223...
[00] 2020/01/13 21:19:34 [INFO]  mssql: Starting service
[00] 2020/01/13 21:19:34 [INFO]  Waiting for new configuration...
[00] 2020/01/13 21:19:34 Attaching filesystem notifier onto /secretless.yml
[00] 2020/01/13 21:19:34 Registering reload signal listeners...
[00] 2020/01/13 21:19:50 Instantiating provider 'literal'
[00] 2020/01/13 21:19:50 [INFO]  mssql: New connection on 192.168.16.3:2223.
[00] 2020/01/13 21:19:50 [ERROR] mssql: Failed on handle connection: failed on connect: lookup <host> on x.x.x.x:53: no such host
  • Sample client log output messages 
    Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Timeout error [258]. .
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Unable to complete login process due to delay in prelogin response.

Known Causes

These errors can occur when the MsSQL server is unreachable at the configured host/port or when its configuration is incompatible with the configuration you provided the client and/or Secretless.

Resolution

  • Ensure that the backend details for MsSQL in Secretless configuration match the connection details for your database.
  • Attempt to connect directly to your database using the credentials to ensure that the backend is working properly.

Bad MsSQL authentication details

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    [00] 2020/01/13 20:58:13 Secretless v1.4.2-dev starting up...
...
[00] 2020/01/13 20:58:13 [WARN]  Starting TCP listener on 0.0.0.0:2223...
[00] 2020/01/13 20:58:13 [INFO]  mssql: Starting service
[00] 2020/01/13 20:58:13 [INFO]  Waiting for new configuration...
[00] 2020/01/13 20:58:13 Attaching filesystem notifier onto /secretless.yml
[00] 2020/01/13 20:58:13 Registering reload signal listeners...
[00] 2020/01/13 20:59:05 Instantiating provider 'literal'
[00] 2020/01/13 20:59:05 [INFO]  mssql: New connection on 172.27.0.3:2223.
[00] 2020/01/13 20:59:05 [ERROR] mssql: Failed on handle connection: failed on connect: Login error: mssql: Login failed for user 'bad'.
  • Sample client log output messages 
    Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user 'sa'..

Known Causes

This error can occur when you have provided Secretless with invalid credentials for the service in the configuration.

Resolution

  • Ensure that authentication details for your connection to Secretless in the configuration are valid.
  • Attempt to connect directly to the database using those connection details to ensure that they are correct.

Invalid Port for MsSQL

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    [00] 2020/01/13 20:19:13 Secretless v1.4.2-dev starting up...
...
[00] 2020/01/13 20:19:13 Registering reload signal listeners...
[00] 2020/01/13 20:19:13 [WARN]  Starting TCP listener on 0.0.0.0:2223...
[00] 2020/01/13 20:19:13 [INFO]  mssql: Starting service
[00] 2020/01/13 20:19:13 [INFO]  Waiting for new configuration...
[00] 2020/01/13 20:21:13 Instantiating provider 'literal'
[00] 2020/01/13 20:21:13 [INFO]  mssql: New connection on x.x.x.x:2223.
[00] 2020/01/13 20:21:13 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:<invalid port>': dial tcp x.x.x.x:<invalid port>: connect: connection refused
  • Your client hangs as Secretless tries to connect and eventually times out with an error, because Secretless was provided with an invalid port in its configuration: 
    sqlcmd -S "localhost,2223" -U "x" -P "x" -Q "SELECT 1+1"
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Timeout error [258]. .
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Unable to complete login process due to delay in prelogin response.

Known Causes

You have not provided a valid port to Secretless.

Resolution

Ensure that port is provided in the Secretless configuration is valid for the MsSQL server

Server does not support TLS

Symptoms

  • You see an error in your Secretless logs that looks something like:
  2020/04/21 17:34:36 [INFO]  mssql: Starting service
  2020/04/21 17:34:36 Registering reload signal listeners...
  2020/04/21 17:34:45 Instantiating provider 'literal'
  2020/04/21 17:34:45 [ERROR] mssql: Failed on handle connection: failed on connect: TLS Handshake failed: x509: certificate is valid for <HostName>, not <Given HostName>
  Received a stop signal
  • Your client hangs as Secretless tries to connect and eventually times out with an error, because Secretless was provided with incorrect SSL settings and cannot complete the connection with encryption: 
    sqlcmd -S "localhost,2223" -U "x" -P "x" -Q "SELECT 1+1"
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Timeout error [258]. .
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Unable to complete login process due to delay in prelogin response.

Known Causes

  • Your server has TLS disabled, but your secretless configuration requires some form of encryption.

  • You have provided a hostname in your secretless configuration that does not correspond to the hostname in the root certificate provided in your secretless configuration

Resolution

  • Ensure that sslmode: disable is set in your Secretless configuration, or use a server that supports encryption.

  • Ensure that the value for host and sslrootcert reference the same host.

  • Use sslhost to provide an alternate value that corresponds to sslrootcert.

Server requires TLS encryption

Symptoms

  • You see an error in your Secretless logs that looks something like:
[00] 2020/04/21 17:43:37 [INFO]  mssql: Starting service
[00] 2020/04/21 17:43:37 [DEBUG] Waiting for new configuration...
[00] 2020/04/21 17:43:43 Instantiating provider 'literal'
[00] 2020/04/21 17:43:43 [DEBUG] mssql: New connection on X.X.X.X:2223.
[00] 2020/04/21 17:43:58 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:1433': dial tcp Y.Y.Y.Y:1433: i/o timeout
  • Your client hangs as Secretless tries to connect and eventually times out with an error, because Secretless was provided with incorrect SSL settings and cannot complete the connection without encryption: 
    sqlcmd -S "localhost,2223" -U "x" -P "x" -Q "SELECT 1+1"
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Timeout error [258]. .
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Unable to complete login process due to delay in prelogin response.

Known Causes

Your server has been configured to require encryption, but your secretless configuration is requesting to disable TLS.

Resolution

Ensure that sslmode is set to allow encryption, using require, verify-ca or verify-full, and providing any corresponding certificates information

Troubleshooting Credential Provider Problems

Conjur Provider Problems

Conjur provider improperly configured

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:34 Instantiating provider 'conjur'
2019/10/30 15:06:34 ERROR: Provider 'conjur' could not be used! ERROR: Unable to construct a Conjur provider client from the available credentials
# Program exit
    or 
    ...
2019/10/30 15:06:34 ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://localhost/authn/myorg/admin/authenticate: x509: certificate signed by unknown authority
  • Sample client log output messages 
    ERROR: MySQL Error 2003 (HY000): Can't connect to MySQL server on 'localhost' (61)

    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://localhost/authn/myorg/admin/authenticate: x509: certificate signed by unknown authority

Known Causes

You have failed to provide Secretless with a valid configuration to enable it to connect to the Conjur server.

Resolution

Ensure that you have all the required details needed for Conjur-based credential retrieval

Conjur server is unreachable

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:47 Instantiating provider 'conjur'
2019/10/30 15:06:47 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:47 ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://nopelocalhost/authn/myorg/admin/authenticate: dial tcp: lookup nopelocalhost: no such host
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://nopelocalhost/authn/myorg/admin/authenticate: dial tcp: lookup nopelocalhost: no such host

Known Causes

You have provided an invalid address for the Conjur server and/or the Conjur server is currently unreachable by Secretless.

Resolution

  • Ensure that the Conjur server is configured properly.
  • Ensure that CONJUR_APPLIANCE_URL is set to the correct and accessible server destination.

Specified Conjur variable not found

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:29 Instantiating provider 'conjur'
2019/10/30 15:06:29 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:29 ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'.

Known Causes

The Conjur variable referenced in the Secretless configuration does not exist, or the Secretless configuration refers to a variable with Secretless is not entitled to access.

Resolution

  • Ensure that Conjur has the specified variable ID stored.
  • Ensure that the API key can retrieve the specified variable ID

Insufficient permissions to read specified Conjur variable

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:44 Instantiating provider 'conjur'
2019/10/30 15:06:44 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:44 ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'.
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'.

Known Causes

Secretless does not have permissions to read the Conjur variable specified in its configuration.

Resolution

  • Ensure that the user specified for Conjur has the appropriate permissions on the requested variable ID.
  • Connect directly to Conjur with specified credentials to ensure that your backend configuration is correct.

Conjur Kubernetes authenticator configured with unreachable Conjur endpoint

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:14 Instantiating provider 'conjur'
2019/10/30 15:06:14 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:14 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:14 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:16 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:16 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:21 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:21 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:26 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:26 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:42 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:42 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:55 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:55 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:17 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:17 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:36 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:36 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:43 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:43 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:53 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:53 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:13 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:08:13 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:23 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:08:23 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:23 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur: Post http://nopehttps//conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert: dial tcp: lookup nopehttps on 10.7.240.10:53: no such host

Known Causes

You have provided an invalid address for the Conjur server and/or the Conjur server is currently unreachable by Secretless.

Resolution

  • Ensure that CONJUR_APPLIANCE_URL and CONJUR_AUTHN_URL are valid in Secretless sidecar configuration.
  • Ensure that there is network connectivity between sidecar containers and Conjur

Conjur Kubernetes authenticator configured with invalid authentication details

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:49 Instantiating provider 'conjur'
2019/10/30 15:06:49 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:49 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:49 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:50 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:50 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:54 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:54 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:03 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:03 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:19 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:19 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:36 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:36 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:05 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:05 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:16 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:16 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:25 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:25 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:45 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:45 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:56 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur:

Known Causes

The Conjur Kubernetes authenticator configuration provided to Secretless may be invalid.

Resolution

  • Ensure that the authenticator authn-k8s/<id> is enabled on Conjur.
  • Ensure that the host is defined as one that can log in through this authenticator.
  • Ensure that CONJUR_AUTHN_LOGIN and CONJUR_ACCOUNT variables for Secretless match expected values in Conjur.

Conjur Kubernetes authenticator configured with bad/invalid Conjur SSL certificate

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 16:34:45 Instantiating provider 'conjur'
2019/10/30 16:34:45 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 16:34:45 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:45 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:34:48 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:48 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:34:52 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:52 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:04 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:04 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:13 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:13 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:35 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:35 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:48 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:48 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:02 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:02 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:21 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:21 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:37 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:37 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:56 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur: Post https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert: x509: certificate signed by unknown authority
    or 
    2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:33 Instantiating provider 'conjur'
2019/10/30 15:06:33 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:33 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: At least one of CONJUR_SSL_CERTIFICATE and CONJUR_CERT_FILE must be provided

Known Causes

The Conjur SSL certificate provided to Secretless is invalid for the Conjur server it is attempting to access.

Resolution

  • Ensure that the SSL certificate for Conjur matches exactly the one in CONJUR_SSL_CERTIFICATE or CONJUR_CERT_FILE env variable.
  • Ensure that if there are load balancers in front of the follower that you use the balancer's SSL certificate as the one to use in Secretless configuration

Kubernetes Secrets Provider Problems

Invalid Kubernetes secret ID format

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/28 15:40:59 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:40:59 Initializing health check on :5335...
2019/10/28 15:40:59 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:40:59 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:40:59 k8s/crd: Using home dir config...
2019/10/28 15:40:59 k8s/crd: Registering CRD watcher...
2019/10/28 15:40:59 k8s/crd: Using home dir config...
2019/10/28 15:40:59 k8s/crd: Add configuration event
2019/10/28 15:40:59 secretless-example-config2
2019/10/28 15:40:59 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:41:06 Instantiating provider 'kubernetes'
2019/10/28 15:41:06 ERROR: Resolving credential 'kube-password' from provider 'kubernetes' failed: Kubernetes secret id must contain secret name and field name in the format secretName#fieldName, received 'kube-password'
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'kube-password' from provider 'kubernetes' failed: Kubernetes secret id must contain secret name and field name in the format secretName#fieldName, received 'kube-password'

Known Causes

The specified Kubernetes secret ID is in an invalid format, and cannot be parsed by Secretless.

Resolution

Ensure that the kubernetes is specified in the <secret_id>#<key> format

Kubernetes API server unreachable

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/28 15:58:35 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:58:35 Initializing health check on :5335...
2019/10/28 15:58:35 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:58:35 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:58:35 Trying to load configuration file: secretless-k8s-secret.yml
2019/10/28 15:58:35 WARN: 'protocol' key found on service 'mysql-socket'. 'protocol' is now deprecated and will be removed in a future release.
2019/10/28 15:58:35 [INFO]  Waiting for new configuration...
2019/10/28 15:58:35 [DEBUG] Got new configuration
2019/10/28 15:58:35 Registering reload signal listeners...
2019/10/28 15:58:35 [INFO]  mysql-socket: Starting service
2019/10/28 15:58:35 [INFO]  Waiting for new configuration...
2019/10/28 15:58:37 Instantiating provider 'literal'
2019/10/28 15:58:37 Instantiating provider 'kubernetes'
2019/10/28 15:59:03 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: connect: operation timed out
2019/10/28 15:59:03 [ERROR] mysql-socket: Failed on handle connection: failed on retrieve credentials: ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: connect: operation timed out
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: i/o timeout

Known Causes

Secretless is unable to communicate with the Kubernetes API to access the specified Kubernetes secret value.

Resolution

  • Ensure that Secretless can access the Kubernetes server
  • Connect to Kubernetes over kubectl to ensure that you have the provider works

Specified Kubernetes secret not found

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/28 15:43:49 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:43:49 Initializing health check on :5335...
2019/10/28 15:43:49 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:43:49 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:43:49 k8s/crd: Using home dir config...
2019/10/28 15:43:49 k8s/crd: Registering CRD watcher...
2019/10/28 15:43:49 k8s/crd: Using home dir config...
2019/10/28 15:43:49 k8s/crd: Add configuration event
2019/10/28 15:43:49 secretless-example-config2
2019/10/28 15:43:49 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:43:52 Instantiating provider 'kubernetes'
2019/10/28 15:43:52 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: could not find Kubernetes secret from 'mysql#password'
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: could not find Kubernetes secret from 'mysql#password'

Known Causes

The secret you referenced in the Secretless configuration either does not exist, or Secretless does not have access to it.

Resolution

  • Ensure that the wanted Kubernetes secret and its identifier match the specified configuration in Secretless configuration.
  • Ensure that Secretless has enough permissions to list and read the specified secret.
  • Ensure that Secretless is trying to read the secret from the correct namespace.
  • Connect to Kubernetes over kubectl to ensure that the secret is readable and accessible.

Kubernetes secret does not contain the expected key

Symptoms

  • You see an error in your Secretless logs that looks something like: 
    2019/10/28 15:46:33 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:46:33 Initializing health check on :5335...
2019/10/28 15:46:33 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:46:33 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:46:33 k8s/crd: Using home dir config...
2019/10/28 15:46:33 k8s/crd: Registering CRD watcher...
2019/10/28 15:46:33 k8s/crd: Using home dir config...
2019/10/28 15:46:33 k8s/crd: Add configuration event
2019/10/28 15:46:33 secretless-example-config2
2019/10/28 15:46:33 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:46:36 Instantiating provider 'kubernetes'
2019/10/28 15:46:36 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: could not find field 'password' in Kubernetes secret 'mysql'
  • Sample client log output messages 
    ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: could not find field 'password' in Kubernetes secret 'mysql'

Known Causes

You have provided a reference to a Kubernetes secret key in the Secretless configuration that cannot be found by Secretless.

Resolution

  • Ensure that the wanted Kubernetes secret and its identifier match the specified configuration in Secretless configuration.
  • Connect to Kubernetes over kubectl to ensure that the secret has the appropriate key id and a value assigned.

Template for Adding New Troubleshooting Scenarios

Please copy/paste the template below if you are adding a new troubleshooting scenario to this wiki.

### Title
#### Symptoms
-

#### Known Causes

#### Resolution

Clone this wiki locally