v1.34.2

This is a patch release

It is strongly advised to update to this version if you use 1.34.0 / 1.34.1 due to regression when Dragonfly runs in cache mode

What's Changed

• fix(evicition): Don't accumulate deleted bytes if there is no RSS evi… by @BorysTheDev in #5908

Full Changelog: v1.34.1...v1.34.2

v1.34.1 (Use with caution - see known issues in description)

This is a patch release

Known issues

#5899
#5891

What's Changed

• fix: prometheus is unable to scrape the /metrics endpoint #5829

Full Changelog: v1.34.0...v1.34.1

v1.34.0 (Use with caution - see known issues in description)

Dragonfly v1.34.0

Known issues

#5899
#5891

Important stability fixes and changes:

• Replication fixes: #5727 #5700
• Geo search fixes: #5714 #5678 #5726 #5734
• SSD tiering improvements: #5802 #5803 #5810

What's Changed

• fix: prevent writeBucket if migration is canceled by @BorysTheDev in #5676
• chore: group memory prometheous metrics by @kostasrim in #5674
• fix: check snapshot_version in correct place by @BorysTheDev in #5687
• fix: stack corruption in MultiCommandSquasher by @romange in #5697
• fix: RdbSaver: destroy SliceSnapshot on its shard thread to fix flaky crash on ARM64 with --force_epoll by @vyavdoshenko in #5685
• server: fix max emission limit in throttle command by @abhijat in #5686
• fix: list ops transformation during replication by @BorysTheDev in #5700
• fix: command parsing in geo family by @kostasrim in #5714
• chore: add queue task index to the log by @romange in #5716
• chore(server): Semi-automatic flag updates by @dranikpg in #5666
• feat(search): Add indexes joining by @BagritsevichStepan in #5604
• feat(search_family): Add LOAD_FROM to the FT.AGGREGATE command by @BagritsevichStepan in #5703
• feat(search_family): Improve sorting and limiting in index joins by @BagritsevichStepan in #5717
• fix: macos and linux build by @vyavdoshenko in #5718
• feat: Implement GEORADIUS_RO command by @EricHayter in #5678
• test: skip failing tests by @vyavdoshenko in #5721
• docs: improve description of --maxmemory parameter (units/tiering/minPerThread) by @frivoire in #5696
• chore: update task queue fiber priority to high by @romange in #5722
• chore: check fail for empty listpacks in streams by @kostasrim in #5704
• feat: improve target node latency during migration by @BorysTheDev in #5715
• chore: Strict bounds when adding uknown command for INFO ALL stats by @mkaruza in #5735
• server: Drop contents of exec_info body on clear by @abhijat in #5740
• feat: Implement GEORADIUSBYMEMBER_RO command by @EricHayter in #5726
• fix: reject geosearch count 0 by @kostasrim in #5734
• fix: jsonPath parser for int values by @BorysTheDev in #5737
• feat(dfly_bench): Generate value with random length by @mkaruza in #5743
• fix: JSON RESP3 compatibility - remove extra nested arrays for most commands by @vyavdoshenko in #5747
• chore: rewrite mimalloc2 external lib dependency by @romange in #5751
• fix: replication for Z...STORE commands by @BorysTheDev in #5728
• server: Enable lua legacy float response for RESP3 by @abhijat in #5754
• fix: bitfield no operations command by @BorysTheDev in #5756
• test: unskip epoll cluster and replication tests by @BorysTheDev in #5757
• Pull helio by @dranikpg in #5760
• fix: search index restore when index names start with ‘:’ (e.g. :Order:index) by @vyavdoshenko in #5755
• Revert "test: unskip epoll cluster and replication tests (#5757)" by @BorysTheDev in #5761
• chore: add more logs for takeover by @BorysTheDev in #5765
• chore: implement the iterative fragmentation check by @romange in #5766
• server: Adds metrics for background defrag task by @abhijat in #5767
• server: metrics to track stored command sizes by @abhijat in #5764
• chore: unlink deletes hashes asynchronously by @romange in #5773
• fix: acl pubsub should only glob match the channel by @kostasrim in #5769
• chore: add self-testing code for heap->full_page_size computation by @romange in #5772
• chore: tune replication/migration settings by @romange in #5779
• facade/server: Use inline instead of extern to reduce loc by @abhijat in #5768
• fix: script error reply by @BorysTheDev in #5776
• chore: Resize vector in StringMap::RandomPairs by @mkaruza in #5781
• dfly_bench: Show error on failing to make address by @abhijat in #5787
• feat: SHUTDOWN: fast path for NOW/FORCE; unify SAVE/SAFE; support NOSAVE by @vyavdoshenko in #5783
• core: Replace sbf with hyperloglog by @abhijat in #5784
• server: Reject caching command if tracking is not enabled by @abhijat in #5786
• fix: data loss in partial sync from same source master by @kostasrim in #5727
• fix: Scan on non-existing key should return zero cursor by @mkaruza in #5780
• fix(snapshot): Replace sleeps with yield by @dranikpg in #5619
• fix(block_list): Speed up Split method by @BagritsevichStepan in #5748
• chore: pass to a string compact_object a key/value flag by @romange in #5795
• feat(range_tree): Add FinalizeInitialization method to the RangeTree by @BagritsevichStepan in #5793
• fix: stale pointer reference during slot migration and flush by @romange in #5798
• chore: Command category cleanup (part 2?) by @dranikpg in #5800
• fix: remove limitation for SSCAN by @BorysTheDev in #5806
• chore: Update helio submodule by @mkaruza in #5807
• chore(tiering): Fix asserts around reads for serialization by @dranikpg in #5802
• feat: prometheus metrics per database by @qlimenoque in #5770
• chore(rdb): Insert big values only after full construction by @dranikpg in #5796
• fix(tiering): Cooldown for DiskStorage::Grow() retries by @dranikpg in #5803
• ci: enable repeat tests to use release and force epoll by @abhijat in #5805
• chore: Clean up string_family by @dranikpg in #5801
• chore(tiering): Make lower bound configurable by @dranikpg in #5810
• fix: increase fiber stack by @BorysTheDev in #5817
• chore: support huffman encoding for string values by @romange in #5815
• chore: increase timeout to 80mins for reg tests by @kostasrim in #5819
• chore(server): Hide internal CO flags from output by @dranikpg in #5814
• feat: Add eviction based on rss memory usage by @BagritsevichStepan in #5791

Huge thanks to all the contributors! ❤️

New Contributors

• @frivoire made their first contribution in #5696
• @qlimenoque made their first contribution in #5770

Full Changelog: v1.33.0...v1.34.0

v1.33.1

This is a patch release.

It is strongly advised to update to this version if you use 1.33 due to regression below.

What's Changed

• fix : stack corruption in MultiCommandSquasher #5697
  It also fixed the io_uring compatibility issue with older kernels (#5693)

v1.33.0

Dragonfly v1.33.0

Important stability fixes and changes:

• Fix stability bugs in georadius #5629 #5634
• Cluster related improvements: #5614 #5621 and #5618

What's Changed

• server: Log warning if server stops accepting connections by @abhijat in #5596
• feat(server): support command help command by @guozhihao-224 in #5601
• chore: refactor JournalItem by @kostasrim in #5585
• feat(linear_search_map): Add keys method by @BagritsevichStepan in #5603
• core: Minor cleanups in bloom filters by @abhijat in #5595
• fix(tiering): Block on concurrent backing file grow by @dranikpg in #5609
• feat: use SET instead of RESTORE for strings during slot migrations by @BorysTheDev in #5614
• chore: introduce shard_thread_busy_polling_usec to control busy polling by @romange in #5624
• fix: UB in georadius by @BorysTheDev in #5629
• fix: test_throttle_on_commands_squashing_replies_bytes by @kostasrim in #5628
• chore: more pipeline latency coverage by @romange in #5632
• fix: ub in GEORADIUSBYMEMBER by @kostasrim in #5634
• chore: more pipeline latency coverage (#5632) by @romange in #5635
• feat: use PXAT flag instead of separate command for strings during migration by @BorysTheDev in #5631
• refactor: CmdArgParser::Error() by @BorysTheDev in #5636
• test(acl): Fix flaky ACL tests with long-running Lua scripts by @vyavdoshenko in #5627
• chore: string_view overload for BaseFamilyTest::Run by @kostasrim in #5637
• fix: FlushSlots test by @BorysTheDev in #5639
• fix: macos build geo_family.cc by @vyavdoshenko in #5640
• fix: preserve null characters in hash values after HEXPIRE by @vyavdoshenko in #5642
• test(acl): make ACL tests with concurrent scripts deterministic by @vyavdoshenko in #5641
• test: skip test_acl_del_user_while_running_lua_script becuase it constantly fails on CI by @vyavdoshenko in #5646
• chore: do not shutdown after takeover only in full cluster mode by @kostasrim in #5621
• chore: add more I/O run-time settings by @romange in #5644
• fix: Prevent race condition during concurrent SAVE operations by @vyavdoshenko in #5633
• core: Track objects skipped during defragmentation by @abhijat in #5647
• chore(tiered_storage): Rename flags, make them updatable by @dranikpg in #5608
• chore: allow replicaof in cluster mode when state is TAKEN_OVER by @kostasrim in #5618
• fix: timeout overflow during migration finalization by @BorysTheDev in #5652
• fix(json): support json keys containing dots in bracket notation by @vyavdoshenko in #5650
• fix: accounting of metrics under squash_stats_latency_lower_limit filter by @romange in #5659
• chore: sort per shard sorted sets for SINTER by @kostasrim in #5617
• fix: macos build tiered_storage.h by @vyavdoshenko in #5663
• fix: RoundRobinSharder for all proactors by @kostasrim in #5654
• workflows: Add a dispatch action which will run pytests on repeat by @abhijat in #5668
• fix(pytest): Don't delete and modify huge value keys in test_replication_all by @mkaruza in #5657
• chore: move tiering flags by @kostasrim in #5665
• fix(json_family): Remove expiry of existing keys in JSON.SET by @mkaruza in #5670
• fix: test_migration_timeout_on_sync by @BorysTheDev in #5667
• fix: refactor save synchronization to prevent deadlocks and crashes by @vyavdoshenko in #5655
• fix(pytest): Increase timeout in test_seeder_key_target by @mkaruza in #5677
• chore: calibrate regression tests running time by @kostasrim in #5658
• chore(lua): Add lua redis.log function by @mkaruza in #5672
• chore: add more detailed timings for slow squash hops by @romange in #5679

Full Changelog: v1.32.0...v1.33.0

v1.32.0

Dragonfly v1.32.0

Important stability fixes and changes:

• Lots of performance improvements around Search #5341#5327 #5350 #5545 #5533
• Add optional latency percentile stats #5372 and #5428
• Update mimalloc version to 2.24
• Added BZMPOP and BLMPOP commands #5436 and #5370
• Added CONFIG REWRITE command #5471

What's Changed

• test(search): add unified benchmarks for prefix/suffix/infix search by @vyavdoshenko in #5319
• chore: get rid of quicklist encoding by @romange in #5282
• feat: add flag pause_wait_timeout by @BorysTheDev in #5328
• test: enable test_multidim_knn python test by @vyavdoshenko in #5329
• test: make random determenistic in test_multidim_knn python test by @vyavdoshenko in #5335
• fix: data race when doing a snapshot in cache mode by @romange in #5336
• chore(search_family): Minor performance improvements to GetAllResults methods. FIRST PR by @BagritsevichStepan in #5341
• fix: JSON.DEL to return 0 for non-existing keys instead of error by @vyavdoshenko in #5343
• chore: improve test_full_sync test by @romange in #5340
• chore: Refactor CompactObject encoding by @dranikpg in #5342
• feat: Added Sort_RO Command. by @H4R5H1T-007 in #5332
• chore: avoid deduplicating docs ids when not needed by @romange in #5344
• chore: add hdr_histogram to the project by @romange in #5330
• chore(tiering_test): tune memory settings to be below 6GB by @romange in #5345
• chore: suffix search by @dranikpg in #5327
• test: Vector operations benchmark was added by @vyavdoshenko in #5353
• feat(replication): Aggregate data until threshold before replication by @mkaruza in #5308
• SimSIMD integration for Vector operations by @vyavdoshenko in #5348
• feat (server) : introduce no point in time replication by @adiholden in #5103
• chore: move 3rd party libraries into a separate cmake file by @romange in #5362
• fix(pytest): Increase allowed diff in test_partial_replication_on_same_source_master by @mkaruza in #5363
• fix(replication): reduce ERROR logging during normal shutdown by @vyavdoshenko in #5366
• chore: remove unneeded mimalloc-specific code by @romange in #5367
• chore: add mimalloc2 third_party library by @romange in #5364
• fix(replication): reduce ERROR to WARNING for connection issues during sync by @vyavdoshenko in #5369
• feat(search): Suffix/Infix search for Tags by @dranikpg in #5350
• fix(zset): expand parsing of rank arguments beyond int32 by @romange in #5377
• test: replication_test.py::test_bug_in_json_memory_tracking enabled by @vyavdoshenko in #5375
• chore: Disable test_replication_all cache mode by @mkaruza in #5385
• chore: Make fakeredis part of the core CI checks by @romange in #5380
• fix(zset_family): Return early if range is expected to be zero by @mkaruza in #5386
• fix: JSON.STRLEN behavior for non-existent keys by @vyavdoshenko in #5389
• chore: add psync metrics for replica by @kostasrim in #5365
• fix(server): fix serialization on move with big values by @adiholden in #5390
• feat(pytest): Set and log random seed used in pytest by @mkaruza in #5391
• chore: remove zset/t_zset files by @romange in #5387
• feat: add support for BLMPOP command by @EricHayter in #5370
• feat(pytest): Initialize random seed for each test by @mkaruza in #5395
• fix(values): fix typos in Helm README and values file by @adimoragbloch in #5400
• fix: no set/hmap expiration during serialization by @BorysTheDev in #5349
• feat(search_family): Add SortedVector<pair<DocId, double>> by @BagritsevichStepan in #5405
• feat: add snapshot-id field into dfs files by @BorysTheDev in #5399
• chore: Enable point_in_time_snapshot by default by @mkaruza in #5403
• chore: Increase BLMPopBlocking sleep duration by @mkaruza in #5408
• chore: skip flaky test_pipeline_cache_size by @kostasrim in #5410
• chore: introduce class scope fixtures by @romange in #5409
• fix(pytest): Increase REPLTAKOVER time in test_take_over_seeder by @mkaruza in #5415
• server_family: Expose command latency stats as part of info by @abhijat in #5372
• fix(search): Cut off results before serializing by @dranikpg in #5412
• test: Add Dragonfly search benchmark tests by @vyavdoshenko in #5406
• feat(search_family): Introduce RangeTree by @BagritsevichStepan in #5414
• fix: search_benchmark_utils on python 3.12 by @romange in #5422
• feat(search_family): Add RangeTree to NumericIndex by @BagritsevichStepan in #5416
• chore: switch to mimalloc2 by @romange in #5368
• feat(search_family): Add optimized merging logic for one and two blocks in RangeResult by @BagritsevichStepan in #5417
• chore(range_tree): Simplify RangeTree code by @BagritsevichStepan in #5418
• Feat config management by @boomballa in #5388
• Lua library Integer overflow can cause the DragonFly crash. (CVE-2020-14147) by @KIMDONGYEON00 in #5421
• chore(search): LIMIT=0 Fast path and remove LOAD parameter from FT.SEARCH by @dranikpg in #5419
• server_family: Support resetting latency histogram by @abhijat in #5428
• test: search benchmark updated to use class fixture for multiple tests by @vyavdoshenko in #5425
• chore: harden checks in _mi_page_malloc_zero by @romange in #5427
• fix(pytest): Wait replication to be in sync in test_stream_approximate_trimming by @mkaruza in #5430
• chore: avoid squashing when squashing_current_reply_size crosses limit by @kostasrim in #4924
• fix: acl log missing return after reply by @kostasrim in #5454
• chore: add acl help by @kostasrim in #5456
• chore: allow initializing a separate huffman table for string values by @romange in #5470
• feat(search_family): Introduce SingleBlockRangeResult and TwoBlocksRangeResult by @BagritsevichStepan in #5424
• chore: add tls_caching flags by @kostasrim in #5485
• chore: add more logs during replication by @BorysTheDev in #5489
• chore(set_family): Fast path SPOP with small count by @dranikpg in #5491
• fix: add a sleep to relax slot migration process by @romange in #5478
• chore: add tls_socket accept error metric by @romange in #5492
• chore: add io_read_cnt metric by @romange in #5494
• chore: add tls handshake statistics by @kostasrim in #5493
• feat: add more auxillary strings to the snapshot by @romange in #5411
• chore: log frequency for IsReplySizeOverLimit by @kostasrim in #5501
• fix: adjust BLMPOP command options and implementation for proper journaling by @EricHayter in #5503
• chore: add more metrics by @romange in https://github.com/dragonflydb/dragonfly/p...

v1.31.2

This is a patch release.

What's Changed

• fix: prevent idle disconnects for replica flow connections (#5506)
• feat: add debug segments command (#5505)
• server: Make memory defrag blocking, accept optional threshold (#5498)
• fix: allow eviction of zero-allocation entries in cache mode (#5500)
• chore: add more metrics (#5504)

Full Changelog: v1.31.1...v1.31.2

v1.31.1

This is a patch release.

What's Changed

• fix: two tls issues
  1. Excessive logging during tls handshake in some cases.
  2. TLS Short read that hinders read performance.
• chore: add tls handshake statistics (#5493)
• chore: add io_read_cnt metric (#5494)
• chore: add tls_socket accept error metric (#5492)
• fix: add sleep to relax slot migration process (#5478)
• fix(set_family): Fast path SPOP with small count (#5491)
• chore: add more logs during replication (#5489)
• chore: add tls_caching flags (#5485)

Full Changelog: v1.31.0...v1.31.1

v1.31.0

Dragonfly v1.31.0

This release is mostly focused on stability features and bug fixes.

Important stability fixes and changes:

1. Introduce partial sync feature that avoids full replica reload during master/replica short disconnects #5182
2. Add support for GAT/GATS commands #5257
3. Introduce experimental huffman encoding for keys #5179

What's Changed

• chore: support fetching a huffman encoded string from CompactObj by @romange in #5086
• fix(hset_family): Fix val being overwritten by TTL by @abhijat in #5094
• fix: memcached authentication handling by @vyavdoshenko in #5098
• fix: memcached port has been already used crash by @vyavdoshenko in #5097
• chore: Add suffix and infix search syntax support in query parser by @vyavdoshenko in #5091
• chore(server): cleanup unused noop opcode in journal by @adiholden in #5101
• chore: add arch type to dfly bench release binary by @kostasrim in #5090
• feat: add HELP SCAN and MINMSZ filter by @romange in #5099
• fix test: rdb test keep loading expired keys by @adiholden in #5105
• refactor: provide CommandContext into InvokeCmd by @BorysTheDev in #5100
• dfly_bench: Send auth if required before cluster setup by @abhijat in #5107
• chore: tls improvements by @romange in #5116
• fix: XTRIM command crash by @vyavdoshenko in #5121
• chore: update module versions by @romange in #5126
• chore: provide metrics for huffman encoding by @romange in #5117
• fix: make search keywords case insensitive by @romange in #5129
• chore: replica clean ups by @romange in #5127
• fix: collapsing logic in CapturingReplyBuilder by @romange in #5133
• fix: improved socket error logging for connection diagnostics by @vyavdoshenko in #5062
• fix: out of bounds SIMD access by @romange in #5140
• chore: improve logs around the parser error by @romange in #5144
• feat: introduce resp validator by @romange in #5147
• chore: minor fixes in geo_family by @romange in #5149
• refactor: geosearch cmd with CmdArgParser by @BorysTheDev in #5153
• feat: add Georadius command by @BorysTheDev in #5158
• pytest: Add sleep to let master stop propagate to replica by @abhijat in #5157
• chore: enable simd optimizations for aarch64 by @romange in #5150
• fix: prevent crash in GEORADIUS argument parsing by @vyavdoshenko in #5166
• chore: more tests around dash table by @romange in #5168
• chore: Add option to ignore parsing errors in replay tool by @vyavdoshenko in #5167
• fix: Prevent crash in PFCOUNT when using keys of different types by @vyavdoshenko in #5161
• chore: lower log level for harmless "Invalid Json path" warnings by @vyavdoshenko in #5163
• chore: remove global current_reply_size by @romange in #5057
• replication: Ensure snapshot fiber joined in all code paths by @abhijat in #5171
• chore: refactorings in dash code. by @romange in #5178
• chore(dash_table): pass memory resource to segment by @romange in #5176
• fix: show proactor kind on test failures by @kostasrim in #5170
• fix(parser): Parse correctly split simple response or error msg in client mode by @mkaruza in #5177
• chore: introduce azure snapshot storage skeleton by @romange in #5104
• fix: improve parsing of client facing responses by @romange in #5181
• server(fix) : SCAN command reduce latency by @vyavdoshenko in #5151
• chore(json_family): Clean up the memory tracking code for JSON mutate operations. FOURTH PR by @BagritsevichStepan in #5070
• feat(search_family): Ignore unsupported options in the FT.CREATE command by @BagritsevichStepan in #5188
• fix (server): fix replication partial sync flow by @adiholden in #5189
• feat: add flags support for huffman encoding by @romange in #5179
• fix server: expire if need check valid condition fix by @adiholden in #5193
• chore: minor dash fixes by @romange in #5191
• fix(json_family): Fix memory tracking for the JSON.DEL command. FIFTH PR by @BagritsevichStepan in #5075
• fix: prevent crash on KEYLOCK_ACQUIRED check for NO_KEY_TRANSACTIONAL commands by @vyavdoshenko in #5185
• fix(cluster_family): Cancel slot migration from incoming node on OOM by @mkaruza in #5000
• fix: timeout for test_seeder_key_target by @BorysTheDev in #5198
• fix: crash in HLL PFMERGE with corrupted data by @vyavdoshenko in #5201
• fix: ZCOUNT crash when min > max by @vyavdoshenko in #5205
• fix: SearchStats race condition crash during concurrent FT operations by @vyavdoshenko in #5207
• fix(pytest): Sleep in sporadically failing test_slot_migration_oom pytest by @mkaruza in #5214
• fix(transaction): DCHECK fail in non-atomic transactions by @romange in #5217
• fix: seeder hset commands generation by @BorysTheDev in #5220
• fix(http_api): Return valid JSON string by @mkaruza in #5210
• feat: add Lua GC flags by @BorysTheDev in #5194
• fix: Unsupported status for Copy/Move/Rename operations by @vyavdoshenko in #5223
• (string_family): Add priliminary support for GAT by @abhijat in #5229
• chore: reduce leaky interface of DashCursor by @romange in #5230
• chore: minor changes in dash code by @romange in #5224
• test: add migration_restart_test with another slots set by @BorysTheDev in #5231
• chore: bring back partial sync by @kostasrim in #5182
• fix: macos build by @vyavdoshenko in #5234
• chore: maintain meta information around continous transactions by @romange in #5232
• fix: dcheck trigger in JournalFlushGuard constructor by @kostasrim in #5241
• fix: typo in AclFamily by @vyavdoshenko in #5244
• fix(pubsub): Skip stale pubsub messages by @dranikpg in #5246
• chore: improve partial sync and test by @kostasrim in #5195
• chore(search): Add self time and resp3 format to ft.profile by @dranikpg in #5250
• fix: test_pubsub_unsubscribe test timeout increased by @vyavdoshenko in #5253
• chore: add corruption check for lpSetTotalBytes by @romange in #5252
• chore: Pull latest helio by @mkaruza in #5254
• fix: BITFIELD: overflow handling, type validation, and return values by @vyavdoshenko in #5242
• fix: prevent writing zero size fields into log file by @BorysTheDev in #5256
• fix(transaction): Fix non-transactional multi/exec transaction access by @dranikpg in #5260
• fix(search_family): Fix empty key bug by @BagritsevichStepan in #5264
• chore: remove old ZIPLIST encoding during snapshotting by @romange in #5267
• chore: stats for lsn buffer by @kostasrim in https://github.com/dragonflyd...

v1.30.3

This is a patch release.

What's Changed

• fix: reduce scan effect on latency of other operations - #5193 #5151
• fix replication partial sync flow (#5189)
• fix(search_family): Ignore unsupported options in the FT.CREATE command (#5188)

Full Changelog: v1.30.2...v1.30.3