using PROMETHEUS_EXPORTER_PORT other than 9090 became port 0 on file_sd_config

[vcrini]

I've following configuration of docker-compose:

version: '3'
services:
  app:
    [...]
    ports:
      - "9090:9090"
    labels:
      PROMETHEUS_EXPORTER_PORT: "9090"

this correctly produces a file_sd_config

- targets:
  - 172.30.8.250:9090
  labels:
    [...]
    container_name: app
    

if instead of port 9090 I use any other, e.g. 8090

version: '3'
services:
  app:
    [...]
    ports:
      - "8090:9090"
    labels:
      PROMETHEUS_EXPORTER_PORT: "8090"

entry in file_sd_config is

- targets:
  - 172.30.8.250:0
  labels:
    [...]
    container_name: app
    

so 0 instead of port 8090, this prevents any metrics ingestion using ecs-discovery.
If I try to wget 172.30.8.250:8090 from ecs-discovery container it works returning the expected metric.

/prometheus # wget 172.30.8.250:8090 -O -
Connecting to 172.30.8.250:8090 (172.30.8.178:8090)
writing to stdout
# HELP system_connection Total connection result
# TYPE system_connection counter
# HELP queue_size Total of element in the queue to process
# TYPE queue_size gauge
queue_size{type="fdh",system="S3LAB.P3TW_AQO_AP",app="s3lab-ap-producer",} 0.0

Have you any insights?

[maxiwoj]

I actually had an issue with ECS-discovery resulting in the port being set to '0' when the port I specified was not accessible on the target container from the ECS-discovery service - so apart from the presented problem, It could be very useful to have some more meaningful message/logging added on connection error, etc.

[vcrini]

Hi!
I was able to fix this issue by change configuration to

version: '3'
services:
  app:
    [...]
    ports:
      - "8090:8090"
    labels:
      PROMETHEUS_EXPORTER_PORT: "8090"

and change listener server port from 9090 to 8090.
Is this the expected benaviour? In my opinion it should have worked with previous configuration (that avoided me to duplicate some code).

I have a pretty similar issue with the port becoming "0" in the export file. However i made sure the metrics endpoint is reachable via curl from the host where the discovery container is running, so it's definitely not a network issue. Any suggestions?