Windows could not start the AWSXrayDaemon service on Local Computer

[abivolmv]

I get this error when trying to setup and start xray daemon on local Windows 7:

Error 1067: The process terminated unexpectedly

Logs :

2018-10-10T09:55:41+02:00 [Info] Initializing AWS X-Ray daemon 3.0.0
2018-10-10T09:55:41+02:00 [Info] Using buffer memory limit of 1309 MB
2018-10-10T09:55:41+02:00 [Info] 20944 segment buffers allocated
2018-10-10T09:55:41+02:00 [Error] Unable to retrieve the region from the EC2 instance RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/placement/availability-zone: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.

2018-10-10T09:55:41+02:00 [Error] Cannot fetch region variable from config file, environment variables and ec2 metadata.

This is my modified cfg.yaml :

# Maximum buffer size in MB (minimum 3). Choose 0 to use 1% of host memory.
TotalBufferSizeMB: 0
# Maximum number of concurrent calls to AWS X-Ray to upload segment documents.
Concurrency: 8
# Send segments to AWS X-Ray service in a specific region
Region: "eu-west-1"
# Change the X-Ray service endpoint to which the daemon sends segment documents.
Endpoint: ""
Socket:
  # Change the address and port on which the daemon listens for UDP packets containing segment documents.
  UDPAddress: "127.0.0.1:2000"
  # Change the address and port on which the daemon listens for HTTP requests to proxy to AWS X-Ray.
  TCPAddress: "127.0.0.1:2000"
Logging:
  LogRotation: true
  # Change the log level, from most verbose to least: dev, debug, info, warn, error, prod (default).
  LogLevel: "prod"
  # Output logs to the specified file path.
  LogPath: ""
# Turn on local mode to skip EC2 instance metadata check.
LocalMode: true
# Amazon Resource Name (ARN) of the AWS resource running the daemon.
ResourceARN: ""
# Assume an IAM role to upload segments to a different account.
RoleARN: ""
# Disable TLS certificate verification.
NoVerifySSL: true
# Upload segments to AWS X-Ray through a proxy.
ProxyAddress: ""
# Daemon configuration file format version.
Version: 2

I also have set the env var AWS_DEFAULT_REGION

[yogiraj07]

Hi @abivolmv ,
From the error log, the daemon is not able to read region value. The daemon searches for the region value at from this location. Parsing logic : Link

From the error log, it looks like the configured cfg.yaml file is not read by the daemon.

Are you using executable for Windows service?

If yes, it is possible because of the user used to start the service. The daemon service looks for the file in the directory, where the service is running and not in the custom directory, where you are running the script.

Usually, the windows services run from C:\Windows\System32 location. You would need to place configured cfg.yaml file at this location and start the daemon service.

Alternatively, you can pass command line option -c with the absolute path to cfg.yaml file.

Note -c parameter in the below script.

sc.exe create AWSXRayDaemon binPath= "$daemonPath -f $daemonLogPath -c $configPath"

Please let me know if you have questions.
Thanks,
Yogi

[abivolmv]

Hi ,
I used the -c flag and it worked. But now I get this error (tell me if I should open a new issue for that) :

2018-10-11T09:22:12+02:00 [Info] Initializing AWS X-Ray daemon 3.0.0
2018-10-11T09:22:12+02:00 [Debug] Listening on UDP 127.0.0.1:2000
2018-10-11T09:22:12+02:00 [Info] Using buffer memory limit of 1309 MB
2018-10-11T09:22:12+02:00 [Info] 20944 segment buffers allocated
2018-10-11T09:22:12+02:00 [Debug] Using proxy address: 
2018-10-11T09:22:12+02:00 [Debug] Fetch region eu-west-1 from commandline/config file
2018-10-11T09:22:12+02:00 [Info] Using region: eu-west-1
2018-10-11T09:22:12+02:00 [Debug] ARN of the AWS resource running the daemon: 
2018-10-11T09:22:12+02:00 [Debug] No Metadata set for telemetry records
2018-10-11T09:22:12+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2018-10-11T09:22:12+02:00 [Debug] Telemetry initiated
2018-10-11T09:22:12+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2018-10-11T09:22:12+02:00 [Debug] Batch size: 50
2018-10-11T09:22:12+02:00 [Info] Starting proxy http server on 127.0.0.1:2000
2018-10-11T09:23:02+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:23:03+02:00 [Debug] processor: sending partial batch
2018-10-11T09:23:03+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2018-10-11T09:23:03+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:23:03+02:00 [Warn] Delaying sending of additional batches by 0 seconds
2018-10-11T09:23:12+02:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:24:12+02:00 [Debug] Failed to send telemetry 2 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:25:13+02:00 [Debug] Failed to send telemetry 3 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:26:08+02:00 [Debug] Segment batch: done!

It works when I run the process exe directly in a cmd :
xray_windows.exe -o -n eu-west-1
But I want it as a service.

[yogiraj07]

Hi @abivolmv ,
The reason for above error is, the AWSXRayDaemon service is not able to access AWS Credentials while sending the segment. This may be due to user permissions.

Please make sure, your AWSXRayDaemon service has right permissions to access ~/.aws folder.

Alternatively, you can set environment variable for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the daemon will automatically pick credentials.

Thanks,
Yogi

[abivolmv]

Thanks for the hint : I added reading permission to LOCAL SERVICE user on ~/.aws and it is now running.

[abivolmv]

No , actually I was to happy to close the issue. After adding that permission the service is starting but when there is a batch to be sent the same error appears :

2018-10-12T11:52:30+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-12T11:52:31+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-12T11:52:31+02:00 [Warn] Delaying sending of additional batches by 0 seconds

[abivolmv]

Setting the env vars worked. But is it possible to do it without the env vars ?

[yogiraj07]

Hi @abivolmv ,
The X-Ray daemon service under the hood uses AWS Go SDK to fetch credentials : Link

The AWS CLI uses credentials at path > dir "%UserProfile%\.aws". Refer : https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Another option can be adding your above directory path to PATH environment variable.

Thanks,
Yogi

[rbanerjiaws]

I am getting the following which i snot exactly the same as the errors above. Any idea why? My credentials are at "%UserProfile%.aws". My client application in .Net Core picks them up just fine.

2018-10-18T18:28:02-04:00 [Debug] Failed to send telemetry 30 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.

[yogiraj07]

Hi @rbanerjiaws ,
The reason for above error possibly is aws credentials are not available to the X-Ray daemon service.

For .NET Core application, AWS Dotnet sdk picks the credentials from %UserProfile%.aws location.
However, for X-Ray daemon which runs as a service locally, requires aws credentials to be made available using above methods as discussed, if the service is not able to recognize the aws credentials automatically.

1. Can you try setting the environment variables for aws credentials as discussed above.
2. Telemetry records reports daemon statistics to X-Ray service at regular interval. After step 1, can you check from the daemon logs whether the daemon is able to send Segments and Telemetry records? (you can follow script mentioned in the thread)

Please let me know if you have more questions.

Thanks,
Yogi

[rbanerjiaws]

It worked with setting environment variables. Just a minor note, the system environment variable needs to be set not the user's.

[arcban02]

I have followed everything on this thread but still getting below error:

Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[awssandra]

Hi, arcban02

Looks like a pretty common case: aws/aws-sdk-go#992

As per previous comment: The AWS CLI uses credentials at path > dir "%UserProfile%.aws". Refer : https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Can you post the daemon logs, I'd like to take a deeper look.

Can you confirm you have the ~/.aws folder, with a credentials file?
Can you confirm the AWSXRayDaemon service has right permissions to access ~/.aws folder?
Can you confirms the directory path above is on the PATH environment variable?
Did you try setting the environment variables, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?

Thanks,
Sandra