Docker on Elastic Beanstalk Not Connecting

[alarinel]

Hi,

I've been searching a ton for a solution to this problem, but no luck tracking down my issue.

I'm using Java and Spring Boot inside a docker container on elastic beanstalk. I added the code to expose the ports for xray.

EXPOSE 2000/tcp
EXPOSE 2000/udp

I added my ebextensions config file with this:

Files:
"/opt/elasticbeanstalk/tasks/taillogs.d/xray-daemon.conf": mode: "000644" owner: root group: root content: | /var/log/xray/xray.log "/etc/amazon/xray/cfg.yaml": mode: "000644" owner: root group: root content: | Logging: LogLevel: "debug" Version: 2 "/etc/awslogs/config/xray.conf": mode: "000755" owner: root group: root content: | [xray-log] log_group_name=/aws/elasticbeanstalk/llitd/docker-xray log_stream_name={instance_id} file=/var/log/xray/*.log file_fingerprint_lines=2-5

container_commands:
01_stop_tracing: command: yum remove -y xray ignoreErrors: true 02_copy_tracing: command: curl https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-3.x.rpm -o /home/ec2-user/xray.rpm 03_start_tracing: command: yum install -y /home/ec2-user/xray.rpm

and in my dockerrun.aws file i have opened this port:

{
  "ContainerPort": "2000",
  "HostPort": "2000"
}

---

In the log file for my xray stream I see:

[Info] Initializing AWS X-Ray daemon 3.2.0
[Debug] Listening on UDP 127.0.0.1:2000
  [Info] Using buffer memory limit of 38 MB
[Info] 608 segment buffers allocated
  [Debug] Using proxy address:
  [Debug] Fetch region us-east-1 from ec2 metadata
  [Info] Using region: us-east-1
  [Debug] ARN of the AWS resource running the daemon:
  [Debug] Using ip-172-31-41-65.ec2.internal hostname for telemetry records
  [Debug] Using i-0b2a13c006e33014f Instance Id for Telemetry records
  [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
 [Debug] Telemetry initiated
  [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-east-1.amazonaws.com
  [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
  [Debug] Batch size: 50
  [Info] Starting proxy http server on 127.0.0.1:2000
  [Debug] Skipped telemetry data as no segments found
  [Debug] Skipped telemetry data as no segments found

And it just keeps with skipping telemetry at the end:

In the docker log file I see tons of UDP Emitters running.

DEBUG c.a.x.e.UDPEmitter { "name" : "writeBehind", "id" : "0ab3649415ffbd7d", "start_time" : 1.600271771496E9, "trace_id" : "1-5f62359b-31d6ea0f342328bac438f7b8", "end_time" : 1.600271771497E9, "aws" : { "xray" : { "sdk_version" : "2.7.1", "sdk" : "X-Ray for Java" } }, "metadata" : { "ClassInfo" : { "Class" : "StoryCardCacheService" } }, "service" : { "runtime" : "OpenJDK 64-Bit Server VM", "runtime_version" : "14.0.2" }
  | 2020-09-16T11:56:11.717-04:00 | }
 
DEBUG c.a.x.e.UDPEmitter Sending UDP packet.

And earlier when it first loaded it was:

  DEBUG c.a.x.c.DaemonConfiguration TCPAddress is set to 127.0.0.1:2000.
DEBUG c.a.x.c.DaemonConfiguration UDPAddress is set to 127.0.0.1:2000.

So I know that the docker container via Spring AOP is sending to 127.0.0.1:2000 and I know that the docker daemon is running on elastic beanstalk and listening UDP on the same address, but for some reason they aren't actually talking to each other. I'm sure I'm missing something really simple I'm just out of ideas about what to try to get this working.

[awssandra]

Hi alarinel,

Are you following the instructions at https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-beanstalk.html ?
It looks like you are, just want to confirm.

[alarinel]

Yep @awssandra I used that code but added modifications to get it working (in the downloading and running section since I'm using docker) by adding in the version: 2 line and then I added my own logs so I could see them in Cloudwatch.

[awssandra]

I am not too familiar with Beanstalk configs myself, but I've reached out to the Beanstalk team to take a look. Will update when I have a better idea.

Sorry for the issues!

[LZY7977-zz]

Hi alarinel,

Can you provide more information about your docker environment, such as the platform branch and platform version(for example, AL2 Single container docker v3.1.2)?

Furthermore, are the ports you listed above the only ports defined in the dockerrun.aws file? If not, could you provide the full list of ports?

Thanks.

[alarinel]

hi @LZY7977

Docker running on 64bit Amazon Linux/2.15.4 - Single Container

The dockerfile is built originally with ubuntu with some customizations but single container for Spring Boot Java entrypoint

In the docker file and dockerrun I have:
8080
3000
3306
6379
2000

All open with host and container port.

I hope that answers all of the questions. I don't know how to find the platform branch or version

[LZY7977-zz]

Hi @alarinel

You can specify multiple container ports, but Elastic Beanstalk uses only the first one to connect your container to the host's reverse proxy and route requests from the public internet.

Please refer the "Ports" section from this doc(https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/single-container-docker-configuration.html).

[willarmiros]

@alarinel you can also check out our ECS sample app for an example of an app instrumented with X-Ray and running in a EB multi-container Docker environment: https://github.com/aws-samples/eb-java-scorekeep/tree/xray-ecs

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.