dnsdist: add redis KVStore implementation

[esensar]

Short description

Adds support for communicating with Redis, as well as a RedisKVStore implementation, backed by a redis client (which can be used independently in Lua).

Checklist

I have:

• read the CONTRIBUTING.md document
• read and accepted the Developer Certificate of Origin document, including the AI Policy, and added a "Signed-off-by" to my commits
• compiled this code
• tested this code
• included documentation (including possible behaviour changes)
• documented the code
• added or modified regression test(s)
• added or modified unit test(s)

---

Sponsored by Quad9

[esensar]

NOTE: this is just the first PR in sequence, if this is accepted. The actual implementation has a lot more commands and also caching support (all available in Quad9 fork: https://github.com/Quad9DNS/pdns-redis/tree/feature/redis-kvstore -- this is actually related to #17387 too, since it is included and mixed in https://github.com/Quad9DNS/pdns-redis/tree/redis-cuckoo-mmdb with this, but it was much easier to extract).

I wanted to get review on the general approach taken with Redis, which maybe has too many classes (each reply, command, lookup action and some variants of clients get a class), before introducing further PRs with more complex parts of this (like caching).

This is also probably missing some important parts, since I just yanked it out of the implementation in the fork, but I can fix these after review.

[rgacogne]

Thanks! On principle I don't see any reason not to add this functionality, it would be nice to be able to query redis without going to Lua, for example.
I skimmed over the code and I don't really see the point of the pipeline mechanism, but it's probably because I'm not very familiar with redis. Would you mind explaining a bit more what the use-case is?

[esensar]

I skimmed over the code and I don't really see the point of the pipeline mechanism, but it's probably because I'm not very familiar with redis. Would you mind explaining a bit more what the use-case is?

The idea is to batch commands in a single communication to redis, improving throughput. I believe for correct usage the user would have to tweak and play with the pipelining interval. When only using Redis from Lua, I think pipelining is useless, because it will block until the response arrives anyways and since Lua uses a single thread, no other request will be made in the meantime, but I think it can increase throughput when used from KVS, since it can then be used by multiple threads.

[rgacogne]

I skimmed over the code and I don't really see the point of the pipeline mechanism, but it's probably because I'm not very familiar with redis. Would you mind explaining a bit more what the use-case is?

The idea is to batch commands in a single communication to redis, improving throughput. I believe for correct usage the user would have to tweak and play with the pipelining interval. When only using Redis from Lua, I think pipelining is useless, because it will block until the response arrives anyways and since Lua uses a single thread, no other request will be made in the meantime, but I think it can increase throughput when used from KVS, since it can then be used by multiple threads.

How does that work? Isn't the KVS interface just waiting for the response as well? I don't see any async implementation that would suspend the query then resume it when the response comes up.

[esensar]

How does that work? Isn't the KVS interface just waiting for the response as well? I don't see any async implementation that would suspend the query then resume it when the response comes up.

My assumption was that KVS can be used from multiple threads, so in that case, there might be multiple queries lined up and sent in a single communication - all of them will block until these responses come though. Maybe this is not such a good idea 😄

[rgacogne]

My assumption was that KVS can be used from multiple threads, so in that case, there might be multiple queries lined up and sent in a single communication - all of them will block until these responses come though. Maybe this is not such a good idea 😄

We might have to do a quick benchmark in any case, because it feels like we are adding latency rather than reducing it, but it's hard to know without some numbers :)

[esensar]

My assumption was that KVS can be used from multiple threads, so in that case, there might be multiple queries lined up and sent in a single communication - all of them will block until these responses come though. Maybe this is not such a good idea 😄

We might have to do a quick benchmark in any case, because it feels like we are adding latency rather than reducing it, but it's hard to know without some numbers :)

Agreed. I will try to set something up to check this.

[esensar]

My assumption was that KVS can be used from multiple threads, so in that case, there might be multiple queries lined up and sent in a single communication - all of them will block until these responses come though. Maybe this is not such a good idea 😄

We might have to do a quick benchmark in any case, because it feels like we are adding latency rather than reducing it, but it's hard to know without some numbers :)

On the other hand, this probably makes more sense if we know we want to send multiple commands at once, but all lookup actions are based on just a single command, so maybe this is just a needless complication. I will remove it for now and if I figure out a case when it creates a noticeable performance gain, I will add it in separately.

[coveralls]

Coverage Report for CI Build 27819253316

Coverage decreased (-0.1%) to 70.94%

Details

• Coverage decreased (-0.1%) from the base build.
• Patch coverage: 378 uncovered changes across 7 files (28 of 406 lines covered, 6.9%).
• 53 coverage regressions across 13 files.

Uncovered Changes

File	Changed	Covered	%
pdns/dnsdistdist/redis.hh	131	0	0.0%
pdns/dnsdistdist/redis.cc	121	0	0.0%
pdns/dnsdistdist/dnsdist-kvs.cc	47	0	0.0%
pdns/dnsdistdist/dnsdist-lua-bindings-redis.cc	52	9	17.31%
pdns/dnsdistdist/dnsdist-configuration-yaml.cc	44	13	29.55%
pdns/dnsdistdist/dnsdist.cc	3	0	0.0%
pdns/dnsdistdist/redis-stats.hh	2	0	0.0%
Total (9 files)	406	28	6.9%

Coverage Regressions

53 previously-covered lines in 13 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
pdns/dnsdistdist/dnsdist-carbon.cc	11	59.56%
pdns/tcpiohandler.cc	9	59.89%
pdns/misc.cc	6	61.45%
pdns/recursordist/syncres.cc	6	81.08%
pdns/recursordist/test-syncres_cc1.cc	5	80.16%
pdns/recursordist/pdns_recursor.cc	3	75.8%
pdns/recursordist/test-syncres_cc2.cc	3	79.39%
pdns/snmp-agent.cc	3	84.42%
pdns/recursordist/negcache.hh	2	90.63%
pdns/signingpipe.cc	2	83.68%

---

Coverage Stats

Relevant Lines:	171634
Covered Lines:	133425
Line Coverage:	77.74%
Relevant Branches:	81984
Covered Branches:	46492
Branch Coverage:	56.71%
Branches in Coverage %:	Yes
Coverage Strength:	5601106.17 hits per line

---

💛 - Coveralls

[esensar]

I think this is mostly ready for review - but it is missing tests. Not sure how to best test this, since it would be best to have a running Redis instance beside it to add some regression test cases.