Improve performance and adding caching layer

[KVInventoR]

I discovered a problem: if you have a lot of SNMP metrics, your connection will be blocked until you have the necessary time to obtain responses.
Can a caching layer with Redis-based key expiration be added?
This can lessen the quantity of data that is retrieved, and it's likely that some metrics can be obtained with varying delays.

Additionally, the mechanism of obtaining metrics can be reviewed. For instance, if we first queryed an endpoint to obtain metrics, the system will automatically set up on-demand metrics retrieval and use Redis to store data. The metrics can then be returned from cache during the subsequent request.

[SuperQ]

We've talked a bit about this in #744.

In general I would love to have some kind of generalized SNMP walk cache that can be used.

Designing it to not be a configuration mess is the tricky part.

How do you handle a situation like this:

• exporter1 - module_foo - walk: 1.3.6.1.2.1.2
• exporter2 - module_bar - walk: 1.3.6.1.2.1.2.2.1.1

I guess you would only be able to do exact key matching on the configured walk in the snmp.yml.

Also what if multiple exporters declare different cache TTLs? This could get tricky if you have multiple exporters and rollout a change.

[lukeod]

I've had a look through the previous threads, and here's my 2c on caching more broadly:

• Focusing on labels over values seems sensible to me - broad use-cases with things like ifName/ifDescr/ifAlias type data that typically changes infrequently. Likely less contentious than caching counters.
• Cache invalidation should have at least two options:
  • One or more sentinel oids. For a typical use-case of doing high-frequency polling of a Cisco switch using IF-MIB::ifTable, you would probably want to invalidate the cache if either IF-MIB::ifTableLastChange OR CISCO-CONFIG-MAN-MIB::ccmHistoryRunningLastChanged changed, as ifTableLastChange only updates on new/removed interfaces, not changes to an ifAlias for instance.
  • Fallback TTL
• A sysUpTime decrease should probably always flush the target's entries as well - covers reboots on devices that don't implement the sentinels properly
• The auth context needs to be part of the cache key, not just target+oid - a v3 user or community can have a restricted view, so two credentials can legitimately see different data for the same oid on the same target.
• On redis specifically - I feel like an external cache should be deferred, with a v1 cache just being in-memory, per-instance. Keyed isolation in a shared store is a solved problem (although you'd probably want to key on a hash of auth contents rather than auth names, as the same auth name can be different credentials on two exporters - although maybe that's just bike shedding). It seems to me you get the lions share of the wins from just in-process cache. Seems in-process would be desirable either way, so focusing on it and keeping the interface generic enough to bolt on external implementations later would be pragmatic to me.
• Exact matching on the configured walk oid seems right to me. An issue with the generator: OID minimization folds the lookup columns into the parent table walk (the default if_mib module walks ifTable/ifXTable wholesale), so there's no distinct walk entry to attach a cache policy to. v1 probably needs to require cached oids to be their own walk entries, with the generator erroring otherwise
• The conflicting TTL concern gets a lot smaller if policies live in the module config and the cache flushes on config reload - worst case two modules disagree about the same subtree and the stricter one revalidates