-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kong DNS resolving not respecting properly configured overrides RES_OPTIONS and LOCALDOMAIN at times. #13301
Comments
And my resolv.conf default which seems to be what Kong ends up using during parts of the runtime when it should not:
|
I'll follow your steps to reproduce it, if finding something, I'll get back to you. |
@chobits sounds good, as long as you see in the warming up dns entries bit its not respecting the environment variable overrides w the dns client debug logs like shown above. Or if I am doing something wrong with the override ENV variables I am all ears there but I believe I used it correctly, as there are parts to Kong’s startup in dns logs that respected them(like the first portion of logs I included). |
reason for the problem
The first part of your provided log is generated by nginx master, see its pid 30.
The second part of your provided log is generated by nginx worker, see its pid 47. Why couldn't the nginx worker get DNS options from
how to mitigate this problem temporarily?configure your /etc/kong/kong.conf with the following options, let nginx workers inherit
BTW, I recommend configuring /etc/resolv.conf directly. $RES_OPTIONS doesn't work for anyone right now, and using it is very rare. Trying to fix it might introduce more logic to the DNS library, adding unnecessary risk. |
@chobits in our k8s environment in my company seems like /etc/resolv.conf gets set by the platform on pod/container startup. Its nice to be able to see the difference in performance and behavior native with the companies provided resolv.conf vs the optimizations via the ENV variables. I am not even sure if I can override the resolv.conf easily via maybe a config map file mount to the container or if k8s would still override that file again on container start? Not totally sure but thats why I use the ENV variable overrides. Would need to add:
To accomplish the full DNS override settings as expected. Still seems a worthy bug to me or need to update documentation that Kong no longer supports DNS override settings. And Kong DNS lookups are super slow with our default DNS settings. Should not need to wait sometimes as much as 30-60-90 seconds for DNS cache warmups to be complete in our instances on Kong startup, taking way too long with our default configs and we have some users complaining that in the hotpath of Kong runtime when it does do DNS lookups its causing larger latency spikes. |
Wow, yep not running in debug mode just yet again, but just by the time in dev it went for DNS warmup times from 20-30 seconds down to 1.4 seconds. The power of DNS optimizations is very real:
Relevant logs of proof of wayyy better performance. Running in debug mode now just to capture the dns client debug logs to confirm the better settings. |
And lastly the debug logs post fix confirming the proper behavior carried through to the workers:
@chobits The ENV variable patch approach worked as evidenced above. |
Even my stage environment with the fixes is super improved:
3.8 seconds when it used to take a full 1+ minute 🚀 . Thousands of service records needing FQDN resolved in those environments. |
Cool! great to hear that the ENV method worked. |
I'm closing this issue, feel free to reopen or left a comment if you still have the problem with this topic. |
@chobits Why would this be closed? Kong nginx workers do not respect the ENV variable settings mentioned in the conf ootb until Kong includes those ENV vars either in your default nginx template or in the conf itself? https://github.com/Kong/kong/blob/3.7.1/kong.conf.default#L1496 |
I have filed an internal ticket to track this, currently I think we should modify the documentation to mention the ENV method. /KAG-4845 |
@chobits ah gotcha, internal ticket and the option of updating documentation on how to achieve the settings seems reasonable too. |
Is there an existing issue for this?
Kong version (
$ kong version
)3.7.1
Current Behavior
When I start Kong in debug mode, I set a bunch of DNS override settings:
And confirmed in pod at runtime:
Per documentation:
https://github.com/Kong/kong/blob/3.7.1/kong.conf.default#L1496
I am here to report that the
RES_OPTIONS
ENV variable override sometimes does not get used during Kong's DNS calls on startup as evidenced by the logs.Portions of Startup logs below(these are good and look like the overrides are taking effect)
Now most of the settings look fine here, it duplicates the
(re)configuring dns client
block twice here, maybe thats because the diff between the two is the overriding the nameservers seen in resolve.conf to theKONG_DNS_RESOLVER
specific ones. This block isn't really the issue but wanted to present it as a block that looks mostly correct as its reconfiguring the DNS client to have the right nameservers I assume(and ndots, attempts, timeout etc. all look like the override is right there).My problem was as I kept watching the logs like a hawk I noticed the DNS warmup step come up later, which again when running Kong in debug mode tells me what its using for DNS settings here at the runtime, this time the output is not respecting the ENV overrides at all it seems other than the 2 nameserver override field(ndots wrong, search wrong, attempts wrong, timeout wrong). Looks like its falling back to settings from my resolv.conf that I don't want it to follow:
cc @chobits since we were talking DNS on that other issue.
Expected Behavior
I expect Kong to respect all ENV variable DNS overrides and maintain those settings persisting throughout Kong's runtime DNS behavior. That means respecting KONG_DNS_RESOLVER(seems it does), and
LOCALDOMAIN
andRES_OPTIONS
which right now seems Kong does not in debug mode.Steps To Reproduce
Anything else?
No response
The text was updated successfully, but these errors were encountered: