maxTotalWalSize not working

[ankitivo]

I have gone through previous issues on this. I need some help understanding why .log files won't be deleted.
The database I have has 57 column families:

S.No. CF Name Estimated Counts
1	  007	1,160,672
2	  008	0
3	  009	591,092
4	  030	931,658
5	  031	59,803
6	  037	1,141,936
7	  064	633,304
8	  065	1,064,757
9	  076	52,540
10	  089	1,118,027
11	  090	406,730
12	  091	62,844
13	  092	482,849
14	  094	101,055
15	  095	5,257
16	  096	118,042
17	  097	75,349
18	  111	1,720,528
19	  114	289,917
20	  127	722,298
21	  150	3,345,349
22	  220	1,007,907
23	  223	965,807
24	  225	766,441
25	  328	212,820
26	  offsets	3,582,655
27	  default	0
28	  scan_00	1,319,022
29	  scan_01	1,035,385
30	  scan_02	465,913
31	  scan_03	1,372,868
32	  scan_04	695,169
33	  scan_05	759,760
34	  scan_06	397,928
35	  scan_07	2,399,730
36	  scan_08	1,006,134
37	  scan_09	1,866,664
38	  scan_10	1,134,034
39	  scan_11	779,691
40	  scan_12	240,009
41	  scan_13	708,806
42	  scan_14	501,182
43	  scan_15	265,936
44	  scan_16	733,980
45	  scan_17	774,969
46	  scan_18	288,808
47	  scan_19	624,275
48	  scan_20	1,172,614
49	  scan_21	245,503
50	  scan_22	1,121,587
51	  scan_23	4,958

having the following RocksDB properties:

1 | CF.compaction_style | 1
-- | -- | --
2 | CF.level0_file_num_compaction_trigger | 10
3 | CF.level0_slowdown_writes_trigger | 40
4 | CF.max_bytes_for_level_base | 800MB
5 | CF.max_write_buffer_number | 10
6 | CF.memtable_prefix_bloom_size_ratio | 0.125
7 | CF.optimize_for_point_lookup | false
8 | CF.optimize_for_point_lookup.block_cache_size_mb | 50
9 | CF.optimize_for_small_db | false
10 | CF.report_bg_io_stats | false
11 | CF.target_file_size_base | 256MB
12 | CF.write_buffer_size | 256MB
13 | RO.readaheadSize | 10MB
14 | Table.block_cache_capacity | 800MB
15 | Table.block_size | 64KB
16 | enable_lru_cache | true
17 | maxTotalWalSize | 50000000000
18 | stats_dump_period_sec | 0

I have 260GB of space on disk occupied by log files, even though I mentioned 50GB as the maxTotalWalSize. As you can see, writes can vary across column families. However, after a reboot, things look normal and the disk size is reduced (probably because all column families are flushed?).

Please take a look and help me in fixing the disk space utilization.

[lizhanhui]

I used to run into a similar issue:

• ManualWalFlush=true;
• Modifying only a few keys and MemTable never got full/immutable;
• No manual flush of column family;

Our solution is to incur CF flush according to the number of writes and SST flush stats.

[kisshot288]

max_total_wal_size only works when the number of column family is larger than 1

there is a sample to make max_total_wal_size work (modified from examples/column_familes_example.cc)

#include <cstdio>
#include <filesystem>
#include <string>
#include <vector>

#include "rocksdb/db.h"
#include "rocksdb/options.h"
#include "rocksdb/slice.h"

#if defined(OS_WIN)
std::string kDBPath = "C:\\Windows\\TEMP\\rocksdb_column_families_example";
#else
std::string kDBPath = "/tmp/rocksdb_column_families_example";
#endif

using ROCKSDB_NAMESPACE::ColumnFamilyDescriptor;
using ROCKSDB_NAMESPACE::ColumnFamilyHandle;
using ROCKSDB_NAMESPACE::ColumnFamilyOptions;
using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::DBOptions;
using ROCKSDB_NAMESPACE::Options;
using ROCKSDB_NAMESPACE::ReadOptions;
using ROCKSDB_NAMESPACE::Slice;
using ROCKSDB_NAMESPACE::Status;
using ROCKSDB_NAMESPACE::WriteBatch;
using ROCKSDB_NAMESPACE::WriteOptions;

int main() {
  std::filesystem::remove_all(kDBPath);

  // open DB
  Options options;
  options.create_if_missing = true;
  options.max_total_wal_size = 1024;
  DB* db;
  Status s = DB::Open(options, kDBPath, &db);
  assert(s.ok());

  // create column family
  ColumnFamilyHandle* cf;
  s = db->CreateColumnFamily(ColumnFamilyOptions(), "new_cf", &cf);
  assert(s.ok());

  // close DB
  s = db->DestroyColumnFamilyHandle(cf);
  assert(s.ok());
  delete db;

  // open DB with two column families
  std::vector<ColumnFamilyDescriptor> column_families;
  // have to open default column family
  column_families.push_back(ColumnFamilyDescriptor(
      ROCKSDB_NAMESPACE::kDefaultColumnFamilyName, ColumnFamilyOptions()));
  // open the new one, too
  column_families.push_back(
      ColumnFamilyDescriptor("new_cf", ColumnFamilyOptions()));
  std::vector<ColumnFamilyHandle*> handles;
  s = DB::Open(options, kDBPath, column_families, &handles, &db);
  assert(s.ok());

  // db->SetDBOptions({{"max_total_wal_size", "1024"}});

  // put and get from non-default column family
  for (int i = 0; i < 1000000000; ++i) {
    // max_total_wal_size will work
    s = db->Put(WriteOptions(), handles[1], Slice("key"), Slice("value" + std::to_string(i)));
    assert(s.ok());
  }
  std::string value;
  s = db->Get(ReadOptions(), handles[1], Slice("key"), &value);
  assert(s.ok());

  // atomic write
  WriteBatch batch;
  batch.Put(handles[0], Slice("key2"), Slice("value2"));
  batch.Put(handles[1], Slice("key3"), Slice("value3"));
  batch.Delete(handles[0], Slice("key"));
  s = db->Write(WriteOptions(), &batch);
  assert(s.ok());

  // drop column family
  s = db->DropColumnFamily(handles[1]);
  assert(s.ok());

  // close db
  for (auto handle : handles) {
    s = db->DestroyColumnFamilyHandle(handle);
    assert(s.ok());
  }
  delete db;

  return 0;
}

[ankitivo]

If you see the number of column families is actually > 1.