Caching despite low threshold

[gerases]

Hi,

I have the following caching configuration now:

cache:
   # Type of caching. Valid: "mem", "memcache", "null"
   type: "mem"
   # Cache limit in megabytes
   size_mb: 0
   # Default cache timeout value. Identical to DEFAULT_CACHE_DURATION in graphite-web.
   defaultTimeoutSec: 10

Despite that, the new metrics are not discovered until about 30 minutes later. Is there something I'm missing in the configuration? What's a good way to troubleshoot this?

Thanks!

[deniszh]

If you think it's a cache issue - switch cache type to "null". I think 0 mb cache means "unlimited". Also, if you using go-carbon realtime metric discovery will work only if using carbonserver with realtime-index enabled. Otherwise new metric will become visible only after file is created, which can be delayed because of disk load or low max-creates-per-second value.

[gerases]

Thanks for the super quick reply. I will try null. I do seem to be using go-carbon (i inherited the cluster, so bear with me):

   backendsv2:
        backends:
          -
            groupName: "go-carbon"
            # supported:
            #    carbonapi_v2_pb - carbonapi 0.11 or earlier version of protocol.
            #    carbonapi_v3_pb - new protocol, http interface (native)
            #    carbonapi_v3_grpc - new protocol, gRPC interface (native)
            #    protobuf, pb, pb3 - same as carbonapi_v2_pb
            #    msgpack - protocol used by graphite-web 1.1 and metrictank
            #    auto - carbonapi will do it's best to guess if it's carbonapi_v3_pb or carbonapi_v2_pb
            #
            #  non-native protocols will be internally converted to new protocol, which will increase memory consumption
            protocol: "carbonapi_v3_pb"
           servers:
                - "http://some-host:8081"
                ...

Legacy graphite service does see the metrics almost immediately though. So the files are definitely there.

[deniszh]

If you post go-carbon config I can say what you need to enable

[gerases]

[common]
user = "carbon"
graph-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
max-cpu = 70
metric-interval = "1m0s"

[whisper]
data-dir = "/carbon/whisper"
schemas-file = "/etc/go-carbon/storage-schemas.conf"
aggregation-file = "/etc/go-carbon/storage-aggregation.conf"
workers = 140
max-updates-per-second = 0
max-creates-per-second = 0
hard-max-creates-per-second = false
sparse-create = false
flock = false
compressed = false
enabled = true
hash-filenames = true
remove-empty-file = false

[cache]
max-size = 1000000000
write-strategy = "noop"

[udp]
listen = ":2003"
enabled = true
buffer-size = 0

[tcp]
listen = ":2003"
enabled = true
buffer-size = 0
compression = ""

[pickle]
listen = ":2004"
max-message-size = 67108864
enabled = true
buffer-size = 0

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"

[grpc]
listen = "127.0.0.1:7003"
enabled = true

[tags]
enabled = false

[carbonserver]
listen = "myhost:8081"
enabled = true
query-cache-enabled = true
query-cache-size-mb = 0
find-cache-enabled = true
buckets = 10
max-globs = 100000
fail-on-max-globs = false
metrics-as-counters = false
trigram-index = true
internal-stats-dir = ""
cache-scan = false
max-metrics-globbed = 10000000
max-metrics-rendered = 1000000
trie-index = true
concurrent-index = false
realtime-index = 1
file-list-cache = ""
read-timeout = "1m0s"
idle-timeout = "1m0s"
write-timeout = "1m0s"
scan-frequency = "5m0s"

[dump]
enabled = false
path = "/var/lib/graphite/dump/"
restore-per-second = 0

[pprof]
listen = "127.0.0.1:7007"
enabled = false

[[logging]]
logger = ""
file = "/var/log/go-carbon/go-carbon.log"
level = "error"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"
sample-tick = ""
sample-initial = 0
sample-thereafter = 0

[prometheus]
enabled = false
endpoint = "/metrics"
[prometheus.labels]

[tracing]
enabled = false
jaegerEndpoint = ""
stdout = false
send_timeout = "10s"

Thank you for offering to analyze this!

[deniszh]

@gerases : sorry, missed your reply. Probably not relevant, but realtime-index = 65535 should fix your issue - it's not a flag, it's index size.