Problem with discovery endpoint

[monoflux]

Hello.

Hydra occasionally can't obtain signing keys when requesting discovery endpoint /.well-known/openid-configuration.

Most common error:
Could not ensure that signing keys for "hydra.openid.id-token" exists. A network error occurred, see error for specific details.

github.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).GetPublicKey
/project/jwk/jwt_strategy.go:70
github.com/ory/hydra/v2/oauth2.(*Handler).discoverOidcConfiguration
/project/oauth2/handler.go:476
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2136
github.com/ory/hydra/v2/driver.(*RegistryBase).RegisterRoutes.(*RegistryBase).OAuth2AwareMiddleware.Middleware.func2.1
/project/x/oauth2cors/cors.go:34
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2136
...

Recorded this once:
Could not ensure that signing keys for "hydra.openid.id-token" exists. If you are running against a persistent SQL database this is most likely because your "secrets.system" ("SECRETS_SYSTEM" environment variable) is not set or changed. When running with an SQL database backend you need to make sure that the secret is set and stays the same, unless when doing key rotation. This may also happen when you forget to run "hydra migrate sql..

github.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).GetPublicKey
/project/jwk/jwt_strategy.go:70
github.com/ory/hydra/v2/oauth2.(*Handler).discoverOidcConfiguration
/project/oauth2/handler.go:476
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2136
github.com/ory/hydra/v2/driver.(*RegistryBase).RegisterRoutes.(*RegistryBase).OAuth2AwareMiddleware.Middleware.func2.1
/project/x/oauth2cors/cors.go:34
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2136
...

Migration has already been successfully completed, no errors occurred during startup.
In both cases, the problem persists for a couple of seconds, then hydra works normally.
It could be a network issue, but since the migration error occurred, there may be something else.

We are using Hydra v2.2.0.

Thanks for any assistance.

[vinckr]

Hey, hard to say what the problem could be without access to your deployment, as you noted it's most likely a network error.

[monoflux]

Thanks for your reply. Do you have any tips on how to investigate this issue?

[vinckr]

Hm not great maybe but here are some ideas:

• could be related to this issue: GetOrGenerateKeys locking leads to significant service degredation with high request saturation #3863
• also mentioned in the issue above, if the tail latency of the database query to retrieve JWKs is above a certain threshold under highly concurrent conditions, it can cause problems
• measure the latency of queries related to JWK retrieval. High latency could be contributing to the issue. check CPU usage during that process, if CPU is maxing out it could be due to the key generation process.
• make sure you are not rotating keys too often unintentionally
• make sure SECRETS_SYSTEM environment variable is set and consistently across all instances.
• if possible try to reproduce it in a staging/dev environment with a load testing tool to make sure its really the cause

These are general suggestions based on the available information - mostly from the mentioned issue. Finding the cause might require more in-depth investigation specific to your deployment and usage.

[monoflux]

Thank you for the ideas!