fix(testing): fix testing setup of prometheus

[VAveryanov8]

This adds scrape config for scylla-manager service itself.

---

Please make sure that:

• Code is split to commits that address a single change
• Commit messages are informative
• Commit titles have module prefix
• Commit titles have issue nr. suffix

[Michal-Leszczynski]

@VAveryanov8 what's the status of this PR?
I understand that in our test env our prometheus instance is not scrapping SM metrics (I couldn't find them when running local test env). I tried cherry picking this commit, but it didn't seem to work either.

[VAveryanov8]

@VAveryanov8 what's the status of this PR?
I understand that in our test env our prometheus instance is not scrapping SM metrics (I couldn't find them when running local test env). I tried cherry picking this commit, but it didn't seem to work either.

It works for the IPv4 (make sure you've restarted container with prometheus), but I wasn't able to verify it against IPv6 and that's why this PR is in draft

[Michal-Leszczynski]

It works for the IPv4 (make sure you've restarted container with prometheus), but I wasn't able to verify it against IPv6 and that's why this PR is in draft

Could you explain how does it work with IPV4? I tried to populate metrics by make pkg-integration-test and manually starting SM server, adding cluster, executing repair, but none of those approaches populated anything for the prometheus server at 192.168.100.98:9090. When starting SM server manually, I am able to scrape its metrics manually at localhost:5090/metrics.

[VAveryanov8]

Could you explain how does it work with IPV4? I tried to populate metrics by make pkg-integration-test and manually starting SM server, adding cluster, executing repair, but none of those approaches populated anything for the prometheus server at 192.168.100.98:9090. When starting SM server manually, I am able to scrape its metrics manually at localhost:5090/metrics.

Yes, when running tests with pkg-integration-test, in most of the integration tests (I guess in all, but I haven't checked) we don't serve metrics on :5090 nor we push metrics to running Prometheus server. I think this is something that worth improving - I'll take a look at this.

For me it works, when I'm starting scylla-manager manually and running some backup/restore commands using sctool. I think it was working the same before, isn't it?

[Michal-Leszczynski]

For me it works, when I'm starting scylla-manager manually and running some backup/restore commands using sctool. I think it was working the same before, isn't it?

Interesting! Anyway, do you know where is 192.168.100.2 coming from?

[VAveryanov8]

Interesting! Anyway, do you know where is 192.168.100.2 coming from?

I guess, from here:

❯ docker network inspect scylla_manager_public -f 'json' | jq -r '.[].Containers[] | {Name, IPv4Address}'
{
  "Name": "scylla_manager-dc2_node_1-1",
  "IPv4Address": "192.168.100.21/24"
}
{
  "Name": "scylla_manager-prometheus-1",
  "IPv4Address": "192.168.100.98/24"
}
{
  "Name": "scylla_manager-dc1_node_3-1",
  "IPv4Address": "192.168.100.13/24"
}
{
  "Name": "scylla_manager-dc1_node_2-1",
  "IPv4Address": "192.168.100.12/24"
}
{
  "Name": "scylla_manager-dc2_node_3-1",
  "IPv4Address": "192.168.100.23/24"
}
{
  "Name": "scylla_manager-second_cluster_dc1_node_2-1",
  "IPv4Address": "192.168.100.32/24"
}
{
  "Name": "scylla_manager-dc2_node_2-1",
  "IPv4Address": "192.168.100.22/24"
}
{
  "Name": "scylla_manager-dc1_node_1-1",
  "IPv4Address": "192.168.100.11/24"
}
{
  "Name": "scylla_manager_server",
  "IPv4Address": "192.168.100.2/24"
}
{
  "Name": "scylla_manager-second_cluster_dc1_node_1-1",
  "IPv4Address": "192.168.100.31/24"
}

[Michal-Leszczynski]

I don't see scylla_manager_server after:

• make start-dev-env
• ./scylla-manager.dev
  So I guess I will need to investigate why is that.

miles@fedora:~/scylla-manager$ docker network inspect scylla_manager_public -f 'json' | jq -r '.[].Containers[] | {Name, IPv4Address}'
{
  "Name": "scylla_manager-dc1_node_2-1",
  "IPv4Address": "192.168.100.12/24"
}
{
  "Name": "scylla_manager-dc1_node_3-1",
  "IPv4Address": "192.168.100.13/24"
}
{
  "Name": "scylla_manager-second_cluster_dc1_node_1-1",
  "IPv4Address": "192.168.100.31/24"
}
{
  "Name": "scylla_manager-dc2_node_2-1",
  "IPv4Address": "192.168.100.22/24"
}
{
  "Name": "scylla_manager-dc1_node_1-1",
  "IPv4Address": "192.168.100.11/24"
}
{
  "Name": "scylla_manager-dc2_node_3-1",
  "IPv4Address": "192.168.100.23/24"
}
{
  "Name": "scylla_manager-prometheus-1",
  "IPv4Address": "192.168.100.98/24"
}
{
  "Name": "scylla_manager-dc2_node_1-1",
  "IPv4Address": "192.168.100.21/24"
}
{
  "Name": "scylla_manager-second_cluster_dc1_node_2-1",
  "IPv4Address": "192.168.100.32/24"
}

[VAveryanov8]

./scylla-manager.dev
So I guess I will need to investigate why is that.

oh, that's because it's not in the container :) You need to run it using make run-server

[Michal-Leszczynski]

okay, I don't think I ever used make run-server, so it makes sense now, thanks!

[karol-kokoszka]

@VAveryanov8 what is the status of this PR ? Do we still need it ? Can it be closed ?