Publishing a hardened image on container.gov.de

[l3monKenji]

Hi we're using the postgres-exporter in a project connected to the german container.gov.initiative. We're thinking about publishing it there (which includes mandatory hardening and supplying a VEX for the vulnerabilities). If we're doing the work already, it's maybe nice to contribute this work upstream back to you. We're already doing this with a Kaniko Image.
Are you open for an exchange on this topic?

Best regards
Freddy

[SuperQ]

Without knowing what "hardening" means, it's impossible to be able to know what to do.

The Prometheus ecosystem is already pretty well "hardended".

• We use a minimized default container image.
• We publish distroless images as well.
• All projects get auto-updated dependencies.
• Most of the code is in Go and we build static binaries.

[Nexucis]

I got the same question about supporting hardened image perses/perses#4103. It gives more context about what is expected. And from my perspective, there are already tool or companies doing it and it's too much work to support as an opensource project.
Perhaps this is possible for a project like Prometheus because there are more maintainers, but I doubt it

[l3monKenji]

Thanks for answering, and for the questions. I think I left out some context, so let me explain a bit more.

We're using the postgres_exporter image in one of our projects, so we're hardening it anyway. What "hardening" means in our case is written up here: https://docs.devguard.org/tutorials/container-hardening/#definition-of-container-hardening

You're right that the base is already in good shape: minimal/distroless images, auto-updated deps, static Go binaries. That's most of the attack-surface work, and we're not trying to redo it. Where we'd add something is the layer on top by handling the remaining CVEs and supplying a VEX that states which of them actually apply to the image and which don't.

To make it more concrete: a trivy scan of prometheuscommunity/postgres-exporter:latest on reported 14 findings in the binary (1 critical, 11 high), almost all from the Go stdlib and x/crypto, and most already have a fixed version available. A lot of that is a rebuild on current Go + updated dependencies. For the rest the VEX would matter. Several of these are DoS paths in code the exporter doesn't reach, so they're false positives, not a real risk and can be deprioritized/ accepted. The VEX makes this visible to whoever consumes it, instead of everyone re-triaging the same 14 lines themselves.

For container.gov.de there's a specific checklist we have to follow: https://container.gov.de/docs/referenzen/container-hardening-checklist/ and part of that is supplying a VEX that says which CVEs actually apply to the image and which don't.

container.gov.de is a German government initiative that provides these hardened images to the public under a "public money, public code" motto. The images listed there are being used for development projects and hardened for security reasons by spending tax money. In this way, the public can benefit from public OSS projects like openDesk even more. There are also more images in the pipeline for being listed.

And yes @Nexucis , there's tooling for this already, we build some of it ourselves. For Disclosure: we are building DevGuard, an open-source software-security-platform, which is also published on container.gov.de as an image as well. So this isn't a pitch, it's just a reason where the hardening work comes from.
Our idea here is the following: since we're doing the work anyway, we'd rather contribute it back to you than keep it in our fork, same as we already do with the Kaniko image. And it would give postgres-exporter some visibility in the German public sector through the platform.

Would a short call make sense? Or if it's easier, we could maybe just open a draft PR / VEX against a specific image first, so there's something concrete to react to.

Best regards,
Freddy

[SuperQ]

The problem is Trivy and a lot of similar scanners generate an absolute stream of false positives. They're next to useless.

For Go code the standard is govulncheck, which actually scans the code calls to determine if there is an affected call.

In fact, as a project, we do not respond to security reports that don't come from govulncheck anymore.

x/crypto is a great example. We have gotten a ton of reports over the years for CVEs in x/crypto. But 99% of them were related to x/crypto/ssh, which isn't used.

So, I would say, that the VEX requirement is not something we're going to do anything about becasue the tools like Trivy are broken by design.

If the requirements were changed to use tools that are not prone to false positives we might be more willing.

[SuperQ]

Just to give an example, here is the govulncheck report of the latest (v0.20.1) postgres_exporter:

$ govulncheck -show verbose ./...
Fetching vulnerabilities from the database...

Checking the code against the vulnerabilities...

The package pattern matched the following 4 root packages:
  github.com/prometheus-community/postgres_exporter/config
  github.com/prometheus-community/postgres_exporter/collector
  github.com/prometheus-community/postgres_exporter/exporter
  github.com/prometheus-community/postgres_exporter/cmd/postgres_exporter
Govulncheck scanned the following 32 modules and the go1.27.1 standard library:
  github.com/prometheus-community/postgres_exporter
  github.com/alecthomas/kingpin/v2@v2.4.0
  github.com/alecthomas/units@v0.0.0-20240927000941-0f3dac36c52b
  github.com/beorn7/perks@v1.0.1
  github.com/blang/semver/v4@v4.0.0
  github.com/cespare/xxhash/v2@v2.3.0
  github.com/coreos/go-systemd/v22@v22.7.0
  github.com/golang-jwt/jwt/v5@v5.3.1
  github.com/google/uuid@v1.6.0
  github.com/jpillora/backoff@v1.0.0
  github.com/lib/pq@v1.10.9
  github.com/mdlayher/socket@v0.6.0
  github.com/mdlayher/vsock@v1.3.0
  github.com/munnerz/goautoneg@v0.0.0-20191010083416-a7dc8b61c822
  github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f
  github.com/prometheus/client_golang@v1.23.2
  github.com/prometheus/client_model@v0.6.2
  github.com/prometheus/common@v0.69.0
  github.com/prometheus/exporter-toolkit@v0.17.1
  github.com/prometheus/procfs@v0.16.1
  github.com/xhit/go-str2duration/v2@v2.1.0
  go.yaml.in/yaml/v2@v2.4.4
  golang.org/x/crypto@v0.53.0
  golang.org/x/net@v0.55.0
  golang.org/x/oauth2@v0.36.0
  golang.org/x/sync@v0.21.0
  golang.org/x/sys@v0.46.0
  golang.org/x/text@v0.38.0
  golang.org/x/time@v0.15.0
  google.golang.org/protobuf@v1.36.11
  gopkg.in/yaml.v2@v2.4.0
  gopkg.in/yaml.v3@v3.0.1

=== Symbol Results ===
No vulnerabilities found.

=== Package Results ===

Vulnerability #1: GO-2026-5970
    Infinite loop on invalid input in golang.org/x/text
  More info: https://pkg.go.dev/vuln/GO-2026-5970
  Module: golang.org/x/text
    Found in: golang.org/x/text@v0.38.0
    Fixed in: golang.org/x/text@v0.39.0

=== Module Results ===

Vulnerability #1: GO-2026-6355
    Prevent DoS on deadlocked established channel in golang.org/x/crypto/ssh
  More info: https://pkg.go.dev/vuln/GO-2026-6355
  Module: golang.org/x/crypto
    Found in: golang.org/x/crypto@v0.53.0
    Fixed in: golang.org/x/crypto@v0.56.0

Vulnerability #2: GO-2026-6354
    Prevent DoS on deadlocked undecided channel in golang.org/x/crypto/ssh
  More info: https://pkg.go.dev/vuln/GO-2026-6354
  Module: golang.org/x/crypto
    Found in: golang.org/x/crypto@v0.53.0
    Fixed in: golang.org/x/crypto@v0.56.0

Vulnerability #3: GO-2026-6303
    Source-address critical option not enforced for non-public-key auth
    callbacks in golang.org/x/crypto/ssh
  More info: https://pkg.go.dev/vuln/GO-2026-6303
  Module: golang.org/x/crypto
    Found in: golang.org/x/crypto@v0.53.0
    Fixed in: golang.org/x/crypto@v0.55.0

Vulnerability #4: GO-2026-5942
    Parsing an invalid SVCB or HTTPS RR can panic in
    golang.org/x/net/dns/dnsmessage
  More info: https://pkg.go.dev/vuln/GO-2026-5942
  Module: golang.org/x/net
    Found in: golang.org/x/net@v0.55.0
    Fixed in: golang.org/x/net@v0.56.0

Vulnerability #5: GO-2026-5932
    The golang.org/x/crypto/openpgp package is unmaintained, unsafe by design,
    and has known security issues
  More info: https://pkg.go.dev/vuln/GO-2026-5932
  Module: golang.org/x/crypto
    Found in: golang.org/x/crypto@v0.53.0
    Fixed in: N/A

Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 5
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.

As you can see, the x/crypto report is there, but none of the exporter code calls any of the vulnerable functions. Module level scanning is not good enough.

[SuperQ]

I read over the requirements on container.gov.de. Many of them we already have implemented. There are a few that are interesting and we may be interested in implementing. If we can change our setup to better support this without a lot of toil we're more than willing to accept PRs for it.

We can bring this up at our next developer summit if you would like to discuss it in detail.

But officially / directly supporting is not likely to happen unless there is a dedicated maintainer to sponsor all the work involved.