Confusion about max_background_jobs and max_background_compactions

[yatriks]

I ran into this issue when trying to tune compaction background threads, and noticing that even though the estimate_pending_compaction_bytes was getting severe as was L0 files buildup, still only 2 background compactions were being scheduled (I had max_background_jobs set to 8).

I tried, on a whim, to set max_background_compactions, and that did actually change the behaviour.

Perhaps something dynamic is happening here and my setting it was necessary for my write-heavy workload, but it contradicts some of the documentation:

Per kvrocks.conf:

# Maximum number of concurrent background jobs (compactions and flushes).
# For backwards compatibility we will set `max_background_jobs =
# max_background_compactions + max_background_flushes` in the case where user
# sets at least one of `max_background_compactions` or `max_background_flushes`
# (we replace -1 by 1 in case one option is unset).
rocksdb.max_background_jobs 4

Then looking in config.cc:

      {"rocksdb.max_background_compactions", false, new IntField(&rocks_db.max_background_compactions, 2, -1, 32)},
      {"rocksdb.max_background_flushes", true, new IntField(&rocks_db.max_background_flushes, 2, -1, 32)},

Which as I understand it, means rocksdb.max_background_compactions is not readonly (true in my experience) and max_background_compactions is not (also true in my experience)

I'm not sure if I got lucky, and max_background_compactions was supposed to be readonly but because of the nature of my workload I needed to override rocksdb's setting, but I don't understand enough about all the parts at play.

I'm happy to update the documentation if I can have some help understanding the behaviour here.

Thanks

[git-hulk]

@yatriks Thanks for reporting this.

The max background jobs wouldn't take effect if one of max_background_compactions / max_background_flushes was set:

  if (max_background_flushes == -1 && max_background_compactions == -1) {
    // for our first stab implementing max_background_jobs, simply allocate a
    // quarter of the threads to flushes.
    res.max_flushes = std::max(1, max_background_jobs / 4);
    res.max_compactions = std::max(1, max_background_jobs - res.max_flushes);
  } else {
    // compatibility code in case users haven't migrated to max_background_jobs,
    // which automatically computes flush/compaction limits
    res.max_flushes = std::max(1, max_background_flushes);
    res.max_compactions = std::max(1, max_background_compactions);
  }

See https://github.com/facebook/rocksdb/blob/main/db/db_impl/db_impl_compaction_flush.cc#L3125

Could you please use config get * to check the value of those configurations?

[yatriks]

Ah...I think I may have carried over an old version of my config file that (for experimentation) did have max_background_compactions already set which is why I was going into this other path.

I can set them to -1 and it should work, I think my confusion came from the fact that I had already had max_background_compactions set from a previous experiment.

Thanks again, and I apologize for the confusion. As I see it, this is all fine, and probably need to leave max_background_compactions as settable to allow people to migrate to the new config. I think everything as written is then correct, with the caveat that maybe it could mention why max_background_compactions is still settable -- but I leave that up to you (happy to make a PR if so). Thanks again 🙏 🙏

[git-hulk]

@yatriks Thanks for your confirmation.

with the caveat that maybe it could mention why max_background_compactions is still settable -- but I leave that up to you (happy to make a PR if so).

Yes, perhaps we could remove the max_background_compactions and max_background_flushes from the configuration in the future, because there are deprecated configurations in rocksdb.

[yatriks]

I just checked and I think I see where I got confused: if I set the values to -1 in the kvrocks.conf file, they are at -1. However, if I leave them unset in the kvrocks.conf file, both are at 2.
I know in the default docker image, for instance, the default kvrocks.conf has them set to -1 so things are fine.

So I think I understand that the way to deprecate the would be to update the default values in config.cc to -1? I think this would be fine because if you have an override set its noop but if you currently have it unset and its defaulting to 2 each, you're covered by max_background_jobs default. The only odd case is if (like me) you had it unset in kvrocks.conf and you have (perhaps mistakenly) set max_background_jobs and did not set max_background_flushes=max_background_compactions=-1, so updating the default would now cause that previously-ignored max_background_jobs to take effect. This feels like it would be unlikely?

I'm also curious if I should consistently be adjusting the default kvrocks.conf instead of assuming it has the defaults and only declaring overrides in my own kvrocks.conf?

[git-hulk]

I'm also curious if I should consistently be adjusting the default kvrocks.conf instead of assuming it has the defaults and only declaring overrides in my own kvrocks.conf?

Yes, we should make the default value to -1 in the codebase.

[yatriks]

I'll submit a PR 👍

[yatriks]

#3398 🙏