v1.38.0

[romange]

Dragonfly v1.38.0

This release is a major one with lots of significant improvements and stability fixes.
It may behave differently due to changes in its core data structures.
We worked hard to avoid any regressions so please let us know if you observe degradation around item expiry behavior, replication or backups.

Important stability fixes and changes:

Top-K command family (TOPK.*): RedisBloom-compatible heavy-hitter sketch with
RDB persistence. (#6920, #6931, #6932, #6950)

Count-Min Sketch family (CMS.*): Full CMS with INITBYDIM, INITBYPROB,
INCRBY, QUERY, MERGE, INFO. RDB persistence. (#6867, #6888, #6896, #6897)

~26% memory reduction for TTL workloads: Expire table eliminated, expiry embedded
in CompactKey via SDS_TTL_TAG. 10M keys: 900 MB -> 665 MB. (#6923, #6925, #6933)

HTTL command: Returns remaining TTL for specific hash fields. (#6879)

VECTOR_RANGE search: Radius-based vector similarity queries in FT.SEARCH and
FT.AGGREGATE for both FLAT and HNSW indexes, with optional YIELD_DISTANCE_AS.
(#6880, #6898, #6938)

FT.AGGREGATE: FILTER, APPLY, DIALECT, KNN: Full expression-based filtering,
computed fields, dialect selection, and KNN steps in the aggregation pipeline.
(#6968, #6982, #7066)

Smarter KNN prefilter: Falls back to brute-force scan when prefilter result set
is small. (#6730)

Replica expired-key deletion: Replicas proactively delete expired keys on reads.
--replica_delete_expired (default true). (#6985)

New Prometheus metrics: Pipeline latency histogram (dragonfly_pipeline_latency_seconds),
TLS handshake counter (tls_handshakes_total), stream access patterns
(stream_accesses_total), pipeline blocking counter. (#6688, #7015, #6767, #6685)

EKS Pod Identity for S3: AWS credential chain supports EKS Pod Identity and ECS
task roles. (#6917)

Early TLS filter and TCP_DEFER_ACCEPT: Drops zombie connections at OS level before
allocating fiber/OpenSSL state. (#6857)

Memcached io_loop_v2 enabled by default*: Async I/O with deferred replies. (#6700)

Incremental SBF replication: Scalable Bloom Filter chunks flushed incrementally,
bounding peak memory. (#7034)

What's Changed

• chore(server): Extend script slowlog stats by @dranikpg in chore(server): Extend script slowlog stats #6656
• test(ci): exclude flaky BullMQ tests by @vyavdoshenko in test(ci): exclude flaky BullMQ tests #6673
• feat(fuzz): Add LLM-guided PR fuzzing with targeted seed generation by @vyavdoshenko in feat(fuzz): Add LLM-guided PR fuzzing with targeted seed generation #6666
• refactor(journal): replace Journal class with free functions by @romange in refactor(journal): replace Journal class with free functions #6672
• fix(search): Fix HNSW index creation failure, logs, drop by @dranikpg in fix(search): Fix HNSW index creation failure, logs, drop #6662
• refactor: clean includes by @BorysTheDev in refactor: clean includes #6679
• chore: remove facade/acl_commands.h by @kostasrim in chore: remove facade/acl_commands.h #6682
• fix: improve fuzz-pr crash detection and artifact packaging by @vyavdoshenko in fix: improve fuzz-pr crash detection and artifact packaging #6686
• feat(tiering): Use Future::IsResolved to check if delayed entry is resolved by @mkaruza in feat(tiering): Use Future::IsResolved to check if delayed entry is resolved #6665
• test: mark heavy skipped tests as large instead of skip by @vyavdoshenko in test: mark heavy skipped tests as large instead of skip #6675
• feat: add pipeline latency histogram and Prometheus SUMMARY export by @romange in feat: add pipeline latency histogram and Prometheus SUMMARY export #6688
• refactor: move methods from OAHEntry to OAHSet by @BorysTheDev in refactor: move methods from OAHEntry to OAHSet #6692
• chore: remove duplicate info reported by "memory stats" by @romange in chore: remove duplicate info reported by "memory stats" #6696
• server: Collect stats for why defragmentation was skipped by @abhijat in server: Collect stats for why defragmentation was skipped #6680
• fix: shrink memory usage for journaling items by @romange in fix: shrink memory usage for journaling items #6695
• ci: exclude flaky BullMQ getWorkers shared connection test by @vyavdoshenko in ci: exclude flaky BullMQ getWorkers shared connection test #6697
• fix(tiering): Track all keys mutation callback for cluster migration by @mkaruza in fix(tiering): Track all keys mutation callback for cluster migration #6691
• refactor: remove static fields from rdb_load by @BorysTheDev in refactor: remove static fields from rdb_load #6684
• fix(tiering): Increase write depth by @dranikpg in fix(tiering): Increase write depth #6705
• refactor(mc): async MGET/GAT by @glevkovich in refactor(mc): async MGET/GAT #6674
• chore(ci): Replace direct push with PR for helm chart updates by @vyavdoshenko in chore(ci): Replace direct push with PR for helm chart updates #6717
• fix: validate stream command arguments to prevent crashes by @vyavdoshenko in fix: validate stream command arguments to prevent crashes #6718
• fix(fuzz): skip missing RECORD files in replay_crash.py by @vyavdoshenko in fix(fuzz): skip missing RECORD files in replay_crash.py #6719
• chore: Skip KNN search if prefilter is empty by @mkaruza in chore: Skip KNN search if prefilter is empty #6720
• perf(seeder): batch STRING hashing with MGET to reduce tiering latency by @romange in perf(seeder): batch STRING hashing with MGET to reduce tiering latency #6721
• test: add test for replication HNSW index by @BorysTheDev in test: add test for replication HNSW index #6722
• skip tiering tests by @dranikpg in skip tiering tests #6723
• ci: migrate ioloop-v2-regtests to OIDC for AWS auth by @AdiCloudbit-DevOps in ci: migrate ioloop-v2-regtests to OIDC for AWS auth #6725
• test: skip test_hnsw_search_replication_with_network_disruptions by @BorysTheDev in test: skip test_hnsw_search_replication_with_network_disruptions #6729
• feat: add RM command skeleton (phase 1) by @romange in feat: add RM command skeleton (phase 1) #6731
• chore: add pipeline_blocking_commands_total counter by @kostasrim in chore: add pipeline_blocking_commands_total counter #6685
• feat(set_family): Reserve vector on OpInter in advance by @abhijat in feat(set_family): Reserve vector on OpInter in advance #6736
• ci: add memcache fuzzing to regular campaign by @vyavdoshenko in ci: add memcache fuzzing to regular campaign #6735
• test: consolidate slow marker into large by @vyavdoshenko in test: consolidate slow marker into large #6734
• feat(dash): segment merging by @kostasrim in feat(dash): segment merging #6676
• server: move blob loading logic into respective family classes by @romange in server: move blob loading logic into respective family classes #6713
• feat(facade): enable experimental_io_loop_v2 by default and clean CI by @glevkovich in feat(facade): enable experimental_io_loop_v2 by default and clean CI #6700
• core: Fix page usage accounting by @abhijat in core: Fix page usage accounting #6737
• chore: do not reconcile slots after dfly takeover if cluster config updated by @kostasrim in chore: do not reconcile slots after dfly takeover if cluster config updated #6728
• fix: orphan streams on error in OpCreate and OpAdd by @kostasrim in fix: orphan streams on error in OpCreate and OpAdd #6744
• fix: crash in SetFullJson when overwriting string key with JSON by @vyavdoshenko in fix: crash in SetFullJson when overwriting string key with JSON #6750
• fix(server): flush replies on encountering blocking command in pipeline by @abhijat in fix(server): flush replies on encountering blocking command in pipeline #6743
• server: Add summary view to memory arena by @abhijat in server: Add summary view to memory arena #6677
• chore: stream family boilerplate to a function by @kostasrim in chore: stream family boilerplate to a function #6748
• fix: prevent SIGFPE in DEBUG POPULATE when val_size is 0 by @vyavdoshenko in fix: prevent SIGFPE in DEBUG POPULATE when val_size is 0 #6754
• feat: implement RM scan-and-delete logic (phase 2) by @romange in feat: implement RM scan-and-delete logic (phase 2) #6732
• refactor: move stream functions from t_stream.c to their call sites by @romange in refactor: move stream functions from t_stream.c to their call sites #6703
• ci: add optional memcache smoke fuzzing to PR workflow by @vyavdoshenko in ci: add optional memcache smoke fuzzing to PR workflow #6758
• feat(server): Get and set byte at index position in string object by @mkaruza in feat(server): Get and set byte at index position in string object #6752
• refactor: move stream alloc/free into compact_object and fix rdb_load cleanup by @romange in refactor: move stream alloc/free into compact_object and fix rdb_load cleanup #6759
• fix: return WOULD_BLOCK for STATS/VERSION in MC pipeline by @vyavdoshenko in fix: return WOULD_BLOCK for STATS/VERSION in MC pipeline #6762
• ci: extract BullMQ skipped tests into a documented file by @vyavdoshenko in ci: extract BullMQ skipped tests into a documented file #6756
• chore: Retrieve single byte for GETBIT command by @mkaruza in chore: Retrieve single byte for GETBIT command #6757
• fix(bitops): Fetch only single byte for SETBIT operation by @mkaruza in fix(bitops): Fetch only single byte for SETBIT operation #6745
• fix: prevent empty hash on HINCRBYFLOAT with NaN/Inf by @vyavdoshenko in fix: prevent empty hash on HINCRBYFLOAT with NaN/Inf #6772
• chore: revert our custom check-fails in mimalloc by @romange in chore: revert our custom check-fails in mimalloc #6774
• json: Add stats for interned string pool by @abhijat in json: Add stats for interned string pool #6701
• feat(server): add stream access pattern metrics by @romange in feat(server): add stream access pattern metrics #6767
• feat: add deffered queue for hnsw write ops by @BorysTheDev in feat: add deffered queue for hnsw write ops #6746
• feat(search): Use filtered search when prefilter result size is small by @mkaruza in feat(search): Use filtered search when prefilter result size is small #6730
• chore: upload coverage by @kostasrim in chore: upload coverage #6747
• tests: Protect connections with mutex by @abhijat in tests: Protect connections with mutex #6777
• fix(search): use 2D vectors in HnswSubsetKnnTest/CompareWithFilteredKnn f or COSINE (macOS build fix) by @vyavdoshenko in fix(search): use 2D vectors in HnswSubsetKnnTest/CompareWithFilteredKnn f or COSINE (macOS build fix) #6790
• Fix pipeline starvation in AsyncFiber during async v2 loop transition by @glevkovich in Fix pipeline starvation in AsyncFiber during async v2 loop transition #6788
• test(tiering): add ABA stash-delete-restash test for OpManager by @romange in test(tiering): add ABA stash-delete-restash test for OpManager #6789
• fix(fuzz): treat AFL++ hangs as failures and upload artifacts by @vyavdoshenko in fix(fuzz): treat AFL++ hangs as failures and upload artifacts #6782
• feat: implement debug compact-table command by @kostasrim in feat: implement debug compact-table command #6741
• chore(tiering): move TieredColdRecord to tiering namespace, update op_manager types by @romange in chore(tiering): move TieredColdRecord to tiering namespace, update op_manager types #6792
• fix(ci): fix no-op cluster tests and move slow tests to large CI by @vyavdoshenko in fix(ci): fix no-op cluster tests and move slow tests to large CI #6795
• chore: simplify AsyncFiber routing logic and improve VLOGs by @glevkovich in chore: simplify AsyncFiber routing logic and improve VLOGs #6797
• CI: make sure jobs only run on the org repo by @abhijat in CI: make sure jobs only run on the org repo #6791
• fix(facade): improve connection setup and buffer capacity tracking by @romange in fix(facade): improve connection setup and buffer capacity tracking #6794
• fix: test_network_disconnect_small_buffer by @kostasrim in fix: test_network_disconnect_small_buffer #6793
• fix: restore HNSW index during replication with different shard counts by @vyavdoshenko in fix: restore HNSW index during replication with different shard counts #6733
• fix(fuzz): fix log spam from RECORD files and hang artifact upload failure by @vyavdoshenko in fix(fuzz): fix log spam from RECORD files and hang artifact upload failure #6798
• fix(pipeline): fix missing SetDeferredReply bugs by @glevkovich in fix(pipeline): fix missing SetDeferredReply bugs #6806
• refactor: deduplicate build steps into reusable composite action by @romange in refactor: deduplicate build steps into reusable composite action #6804
• CI: Remove the docker image and binary downloads by @abhijat in CI: Remove the docker image and binary downloads #6812
• refactor(tiering): introduce FragmentRef abstraction for serialization by @romange in refactor(tiering): introduce FragmentRef abstraction for serialization #6796
• feat: add pending keys to HNSW index during full sync phase by @BorysTheDev in feat: add pending keys to HNSW index during full sync phase #6805
• ci(fuzz): switch AFL++ compiler from afl-clang-lto to afl-clang-fast by @vyavdoshenko in ci(fuzz): switch AFL++ compiler from afl-clang-lto to afl-clang-fast #6817
• chore: remove shard_cnt/EXPIRED arguments in journal by @romange in chore: remove shard_cnt/EXPIRED arguments in journal #6818
• doc(serialization): introduce the serialization design document by @romange in doc(serialization): introduce the serialization design document #6820
• fix: improve HNSW index restoration correctness during replication by @BorysTheDev in fix: improve HNSW index restoration correctness during replication #6815
• fix: clear stale expiry when overwriting key with no expiration in AddOrUpdateInternal by @vyavdoshenko in fix: clear stale expiry when overwriting key with no expiration in AddOrUpdateInternal #6821
• chore: Set HNSW label with setExternalLabel by @mkaruza in chore: Set HNSW label with setExternalLabel #6847
• fix: finalize memory accounting before tiered stash during RDB load by @romange in fix: finalize memory accounting before tiered stash during RDB load #6823
• fix(search): Don't apply limit on results in prefilter knn search by @mkaruza in fix(search): Don't apply limit on results in prefilter knn search #6724
• feat(server): First coroutine for Del by @dranikpg in feat(server): First coroutine for Del #6706
• chore: Fix clang compilation issues for ZSetFamily structs by @mkaruza in chore: Fix clang compilation issues for ZSetFamily structs #6849
• fix(tiering): fix flaky op_manager_test by waiting for cancelled stash IOs by @romange in fix(tiering): fix flaky op_manager_test by waiting for cancelled stash IOs #6852
• chore(tiering): pass FragmentRef instead of PrimeValue in tiered storage by @romange in chore(tiering): pass FragmentRef instead of PrimeValue in tiered storage #6784
• fix(cluster): Handle number overflow in DFLYCLUSTER CONFIG parsing by @vyavdoshenko in fix(cluster): Handle number overflow in DFLYCLUSTER CONFIG parsing #6853
• feat(facade): add early TLS filter and TCP_DEFER_ACCEPT by @glevkovich in feat(facade): add early TLS filter and TCP_DEFER_ACCEPT #6857
• Add Claude Code skill for reproducing fuzz crashes by @romange in Add Claude Code skill for reproducing fuzz crashes #6859
• fix(cluster): Validate slot IDs in DFLYCLUSTER FLUSHSLOTS to prevent out-of-bounds crash by @vyavdoshenko in fix(cluster): Validate slot IDs in DFLYCLUSTER FLUSHSLOTS to prevent out-of-bounds crash #6861
• fix: make SwitchState return previous state to prevent double DFLY LOAD by @romange in fix: make SwitchState return previous state to prevent double DFLY LOAD #6855
• doc(serialization): document ordering invariants in code comments by @romange in doc(serialization): document ordering invariants in code comments #6845
• doc(serialization): expand shard serialization design with tiered analysis and roadmap by @romange in doc(serialization): expand shard serialization design with tiered analysis and roadmap #6827
• fix(tiering): Serialize delayed entries from SerializeBucket by @mkaruza in fix(tiering): Serialize delayed entries from SerializeBucket #6807
• chore: introduce SerializerBase by @kostasrim in chore: introduce SerializerBase #6854
• feat(server): Count unique strings across shards by @abhijat in feat(server): Count unique strings across shards #6811
• test: make test_hnsw_search_replication_with_network_disruptions stable by @BorysTheDev in test: make test_hnsw_search_replication_with_network_disruptions stable #6863
• core: Yield while traversing by @abhijat in core: Yield while traversing #6846
• Revert "refactor(string): migrate SET, APPEND, PREPEND to async SimpleContext" by @dranikpg in Revert "refactor(string): migrate SET, APPEND, PREPEND to async SimpleContext" #6851
• Revert "core: Add data structure for estimate which decays over time … by @romange in Revert "core: Add data structure for estimate which decays over time … #6866
• chore: another way to track per object memory by @romange in chore: another way to track per object memory #6864
• chore(core): add CMS implementation with unit tests by @romange in chore(core): add CMS implementation with unit tests #6867
• Document cluster node health feature by @Copilot in Document cluster node health feature #6502
• fix(rdb_load): Fix dry run with shard optimization by @dranikpg in fix(rdb_load): Fix dry run with shard optimization #6727
• fix(ci): Skip AWS auth step on forked PR by @dranikpg in fix(ci): Skip AWS auth step on forked PR #6875
• feat: add extract method into StringMap by @BorysTheDev in feat: add extract method into StringMap #6872
• Add Celery integration tests by @romange in Add Celery integration tests #6876
• chore(server): Use coroutines for MGET/GAT/INCR by @dranikpg in chore(server): Use coroutines for MGET/GAT/INCR #6869
• Add HTTL command for hash field TTL queries by @romange in Add HTTL command for hash field TTL queries #6879
• feat(search): implement VECTOR_RANGE operator for FLAT index by @vyavdoshenko in feat(search): implement VECTOR_RANGE operator for FLAT index #6880
• fix: json family memory tracking and orphans by @kostasrim in fix: json family memory tracking and orphans #6785
• fix(search): Fix null vector detection in FlatVectorIndex by @vyavdoshenko in fix(search): Fix null vector detection in FlatVectorIndex #6892
• chore: add CMS type in CompactObject and AclFamily by @kostasrim in chore: add CMS type in CompactObject and AclFamily #6888
• chore: add cms family by @kostasrim in chore: add cms family #6896
• chore(facade): Simplify parsed command replies by @dranikpg in chore(facade): Simplify parsed command replies #6878
• chore: update pytests for more modern redis-py client by @romange in chore: update pytests for more modern redis-py client #6889
• fix: macOS build by @vyavdoshenko in fix: macOS build #6900
• fix(facade): Support RESP in ioloopv2 by @dranikpg in fix(facade): Support RESP in ioloopv2 #6870
• chore: disk backpressure class utility by @kostasrim in chore: disk backpressure class utility #6020
• chore: switch CI to use ubuntu:24 by @romange in chore: switch CI to use ubuntu:24 #6873
• fix(search): don't crash on wrong-sized vector/numeric field values in FT.SEARCH by @vyavdoshenko in fix(search): don't crash on wrong-sized vector/numeric field values in FT.SEARCH #6908
• chore: remove unneeded http pages by @romange in chore: remove unneeded http pages #6905
• fix(search): two crash fixes — non-finite NUMERIC values and KeepTopKSorted OOB by @vyavdoshenko in fix(search): two crash fixes — non-finite NUMERIC values and KeepTopKSorted OOB #6911
• chore: CMS serialization by @kostasrim in chore: CMS serialization #6897
• fix(CI): conditionally use pip flags by @abhijat in fix(CI): conditionally use pip flags #6914
• chore(server): Remove non-coroutine async structs by @dranikpg in chore(server): Remove non-coroutine async structs #6877
• chore: add BullMQ/sidekiq tests, fix celery worker teardown timeout by @romange in chore: add BullMQ/sidekiq tests, fix celery worker teardown timeout #6901
• chore: stack reduction and small helio clean-ups by @romange in chore: stack reduction and small helio clean-ups #6916
• chore: switch actions to ubuntu-dev:24 by @romange in chore: switch actions to ubuntu-dev:24 #6922
• CI: allow commit to repeat test by @abhijat in CI: allow commit to repeat test #6919
• feat(core): embed TTL directly in CompactKey via SDS_TTL_TAG by @romange in feat(core): embed TTL directly in CompactKey via SDS_TTL_TAG #6925
• core: add dict_builder module with HLL compressibility estimation by @romange in core: add dict_builder module with HLL compressibility estimation #6903
• feat(search): Add HNSW vector range search (FT.SEARCH) by @vyavdoshenko in feat(search): Add HNSW vector range search (FT.SEARCH) #6898
• feat(core): introduce TOPK data structure (Stage 1) by @glevkovich in feat(core): introduce TOPK data structure (Stage 1) #6920
• chore: improve naming and get rid of references for mcflag by @romange in chore: improve naming and get rid of references for mcflag #6930
• feat(core): integrate TOPK into CompactObj by @glevkovich in feat(core): integrate TOPK into CompactObj #6931
• chore(server): remove expire table by @romange in chore(server): remove expire table #6923
• fix(server): fix active expiry dilution after expire table removal by @romange in fix(server): fix active expiry dilution after expire table removal #6934
• perf(server): optimize OpStrLen for tiered storage by @asherlaau in perf(server): optimize OpStrLen for tiered storage #6913
• chore: macos stub for DiskBackedQueue by @kostasrim in chore: macos stub for DiskBackedQueue #6935
• chore(server): remove expire table by @romange in chore(server): remove expire table #6933
• chore: fix release pipeline by @romange in chore: fix release pipeline #6937
• feat(aws): support EKS Pod Identity for S3 credentials by @kanaevad in feat(aws): support EKS Pod Identity for S3 credentials #6917
• fix(search): guard LoadEntry against freed DocIds in global HNSW KNN by @vyavdoshenko in fix(search): guard LoadEntry against freed DocIds in global HNSW KNN #6936
• feat(server): Use more coroutines in string_family by @claude in feat(server): Use more coroutines in string_family #6910
• fix(core): Optimize SortedMap::GetRange by @dranikpg in fix(core): Optimize SortedMap::GetRange #6942
• chore: SliceSnapshot inherits from SerializerBase by @kostasrim in chore: SliceSnapshot inherits from SerializerBase #6860
• feat(topk): implement RDB serialization for Top-K sketches by @glevkovich in feat(topk): implement RDB serialization for Top-K sketches #6932
• Exclude fork PRs from AWS credential steps in CI workflows by @Copilot in Exclude fork PRs from AWS credential steps in CI workflows #6946
• fix(search): cap GROUPBY nargs before reserve to prevent DoS crash by @vyavdoshenko in fix(search): cap GROUPBY nargs before reserve to prevent DoS crash #6949
• feat(search): support VECTOR_RANGE operator in FT.AGGREGATE with YIELD_DISTANCE_AS by @vyavdoshenko in feat(search): support VECTOR_RANGE operator in FT.AGGREGATE with YIELD_DISTANCE_AS #6938
• chore(core): retire LZ4 compression from qlist by @romange in chore(core): retire LZ4 compression from qlist #6952
• test: add logs for redis and valkey instances during tests by @BorysTheDev in test: add logs for redis and valkey instances during tests #6954
• feat(topk): add TOPK command family by @glevkovich in feat(topk): add TOPK command family #6950
• chore: fix qlist memory accounting bug by @romange in chore: fix qlist memory accounting bug #6958
• chore(deps): bump the actions group across 1 directory with 10 updates by @dependabot[bot] in chore(deps): bump the actions group across 1 directory with 10 updates #6957
• fix: ft.dropindex deadlock by @BorysTheDev in fix: ft.dropindex deadlock #6956
• chore(server): Simplify SerializerBase interface by @dranikpg in chore(server): Simplify SerializerBase interface #6948
• fix(test): Disable heartbeat during keys hash calculation in tiering replication test by @mkaruza in fix(test): Disable heartbeat during keys hash calculation in tiering replication test #6904
• fix(search): Simplify hnsw indexing by @dranikpg in fix(search): Simplify hnsw indexing #6962
• fix: update helio submodule by @vyavdoshenko in fix: update helio submodule #6961
• chore(fuzz): bundle repro.env with crash archives for reliable reproduction by @vyavdoshenko in chore(fuzz): bundle repro.env with crash archives for reliable reproduction #6953
• fix: disable BullMQ flaky test by @vyavdoshenko in fix: disable BullMQ flaky test #6963
• chore(core): add ZSTD dictionary compression infrastructure to qlist by @romange in chore(core): add ZSTD dictionary compression infrastructure to qlist #6955
• test(fakeredis): add topk (fakeredis) test suite by @glevkovich in test(fakeredis): add topk (fakeredis) test suite #6959
• refactor: DenseSet entry removing by @BorysTheDev in refactor: DenseSet entry removing #6966
• feat(server): Locking control from lua scripts by @dranikpg in feat(server): Locking control from lua scripts #6277
• fix: FT.AGGREGATE SORTBY crash with huge nargs value by @vyavdoshenko in fix: FT.AGGREGATE SORTBY crash with huge nargs value #6972
• feat(search): add FILTER expression parser and evaluator for FT.AGGREGATE by @vyavdoshenko in feat(search): add FILTER expression parser and evaluator for FT.AGGREGATE #6968
• chore: Disable memory_test::test_throttle_on_commands_squashing_replies_bytes by @mkaruza in chore: Disable memory_test::test_throttle_on_commands_squashing_replies_bytes #6976
• fix: null ptr deref in JournalXReadGroupIfNeeded for XREAD BLOCK with replication by @romange in fix: null ptr deref in JournalXReadGroupIfNeeded for XREAD BLOCK with replication #6980
• fix: JSON.NUMINCRBY negative result overflow by @romange in fix: JSON.NUMINCRBY negative result overflow #6981
• feat(search): add FILTER clause support in FT.AGGREGATE by @vyavdoshenko in feat(search): add FILTER clause support in FT.AGGREGATE #6982
• fix: memory calculations for sets by @BorysTheDev in fix: memory calculations for sets #6983
• chore(core): implement ZSTD dict based compression by @romange in chore(core): implement ZSTD dict based compression #6967
• fix: crash for lazy expired sets by @BorysTheDev in fix: crash for lazy expired sets #6979
• fix: SPOP crash when set members have expired per-member TTLs by @vyavdoshenko in fix: SPOP crash when set members have expired per-member TTLs #6990
• fix(test): Increase RSS memory usage in test_eviction_on_rss_treshold by @mkaruza in fix(test): Increase RSS memory usage in test_eviction_on_rss_treshold #6986
• fix: heap buffer overflow in StringSet::AddBatch when updating TTL by @vyavdoshenko in fix: heap buffer overflow in StringSet::AddBatch when updating TTL #6988
• fix: SPOP crash when randomly picked set members have expired TTLs by @vyavdoshenko in fix: SPOP crash when randomly picked set members have expired TTLs #6997
• test: Add missing fuzz seeds and mutator commands for missing functionality by @vyavdoshenko in test: Add missing fuzz seeds and mutator commands for missing functionality #6969
• Use SerializerBase in RestoreSteamer, unite tiered handling by @dranikpg in Use SerializerBase in RestoreSteamer, unite tiered handling #6970
• fix: crash in srandmember durin lazy expiration by @BorysTheDev in fix: crash in srandmember durin lazy expiration #6999
• feat: add distroless Docker image with built-in healthcheck by @vyavdoshenko in feat: add distroless Docker image with built-in healthcheck #6902
• Add compression stats and ratio threshold for ZSTD dict by @romange in Add compression stats and ratio threshold for ZSTD dict #7000
• feat(facade): Basic RESP support in ioloopv2, part 2 by @dranikpg in feat(facade): Basic RESP support in ioloopv2, part 2 #6885
• fix: replication of set lazy expiration by @BorysTheDev in fix: replication of set lazy expiration #6994
• chore(common): Some C++20 simplifications by @dranikpg in chore(common): Some C++20 simplifications #6883
• fix: preserve M and EF_CONSTRUCTION in HNSW index restore command by @vyavdoshenko in fix: preserve M and EF_CONSTRUCTION in HNSW index restore command #7014
• feat(server): Retire RdbSerializerBase by @abhijat in feat(server): Retire RdbSerializerBase #7016
• chore(facade): improve IoLoopV2 readability and simplify recv flow by @romange in chore(facade): improve IoLoopV2 readability and simplify recv flow #7011
• chore(server): Move search serialization to own file by @dranikpg in chore(server): Move search serialization to own file #6996
• feat: add tls_handshakes_total labeled metric to /metrics by @romange in feat: add tls_handshakes_total labeled metric to /metrics #7015
• fix: coverage yml by @kostasrim in fix: coverage yml #6987
• fix: remove race condition in proxy during close by @BorysTheDev in fix: remove race condition in proxy during close #7019
• fix(cms): reject NaN in CMS.INITBYPROB to prevent RENAME crash by @vyavdoshenko in fix(cms): reject NaN in CMS.INITBYPROB to prevent RENAME crash #7026
• fix: remove set if lazy expiration by @BorysTheDev in fix: remove set if lazy expiration #7005
• Add --showlocals option to pytest configuration by @dranikpg in Add --showlocals option to pytest configuration #7029
• chore: remove dead code by @BorysTheDev in chore: remove dead code #7032
• fix: prevent master connection closing during await_synced_all by @BorysTheDev in fix: prevent master connection closing during await_synced_all #7035
• feat(rdb_saver): Flush each SBF chunk during serialization by @mkaruza in feat(rdb_saver): Flush each SBF chunk during serialization #7034
• fix: increase timeout for test_redis_replication_all by @BorysTheDev in fix: increase timeout for test_redis_replication_all #7037
• CI: Add script to handle pip install by @abhijat in CI: Add script to handle pip install #7038
• feat: upload logs for failed test on coverage by @BorysTheDev in feat: upload logs for failed test on coverage #7042
• chore: Increase kFiberStackBase by 8KB by @mkaruza in chore: Increase kFiberStackBase by 8KB #7039
• chore: Extend QList::Node to support tiered storage by @mkaruza in chore: Extend QList::Node to support tiered storage #7021
• fix: handle properly nan in geosearch by @kostasrim in fix: handle properly nan in geosearch #7043
• chore: use ranges and concepts by @kostasrim in chore: use ranges and concepts #7040
• fix: reject FT.CREATE with oversized VECTOR DIM to prevent search crash by @vyavdoshenko in fix: reject FT.CREATE with oversized VECTOR DIM to prevent search crash #7049
• fix: increase timeout for test_network_disconnect_during_migration by @BorysTheDev in fix: increase timeout for test_network_disconnect_during_migration #7050
• fix: prevent OOM crash in FT.CREATE with huge PREFIX/STOPWORDS count by @vyavdoshenko in fix: prevent OOM crash in FT.CREATE with huge PREFIX/STOPWORDS count #7046
• fix(search): emit complete field params in FT.INFO and BuildRestoreCommand by @vyavdoshenko in fix(search): emit complete field params in FT.INFO and BuildRestoreCommand #7044
• fix: dev container build failure on ARM runners by @vyavdoshenko in fix: dev container build failure on ARM runners #7051
• fix: support VECTOR_RANGE query without YIELD_DISTANCE_AS clause by @vyavdoshenko in fix: support VECTOR_RANGE query without YIELD_DISTANCE_AS clause #7053
• chore: Extend TieredStorage to support QList::Node by @mkaruza in chore: Extend TieredStorage to support QList::Node #7022
• chore(deps): bump the actions group with 3 updates by @dependabot[bot] in chore(deps): bump the actions group with 3 updates #7028
• fix: migrate glog references to helio wrappers by @romange in fix: migrate glog references to helio wrappers #7002
• fix(search): add missing FinalizeInitialization in numeric benchmarks by @vyavdoshenko in fix(search): add missing FinalizeInitialization in numeric benchmarks #7059
• chore: add fiber name assertion in SerializerBase::OnChange by @romange in chore: add fiber name assertion in SerializerBase::OnChange #7064
• feat(search): add DIALECT, APPLY, and KNN support to FT.AGGREGATE by @vyavdoshenko in feat(search): add DIALECT, APPLY, and KNN support to FT.AGGREGATE #7066
• chore: add WITH_AWS_CLOUD support for S3 snapshot storage by @romange in chore: add WITH_AWS_CLOUD support for S3 snapshot storage #6992
• fix(core): prevent MergeNodes from merging non-adjacent head and tail nodes by @vyavdoshenko in fix(core): prevent MergeNodes from merging non-adjacent head and tail nodes #7079
• fix(server): SIGSEGV in HRANDFIELD with expired hash fields by @vyavdoshenko in fix(server): SIGSEGV in HRANDFIELD with expired hash fields #7076
• docs(pubsub): add detailed Pub/Sub architecture guide by @glevkovich in docs(pubsub): add detailed Pub/Sub architecture guide #7075
• chore(core): extract HllEstimator class from dict_builder by @romange in chore(core): extract HllEstimator class from dict_builder #7067
• feat(server): Make parallel delayed entry processing safe by @dranikpg in feat(server): Make parallel delayed entry processing safe #7055
• feat(tiering): Experimental list node tiering by @mkaruza in feat(tiering): Experimental list node tiering #7023
• fix: proactively delete expired keys on replicas via replica_delete_expired flag by @shahyash2609 in fix: proactively delete expired keys on replicas via replica_delete_expired flag #6985
• tests: Add a test that verifies metrics for nested eval by @abhijat in tests: Add a test that verifies metrics for nested eval #7085
• Consolidate duplicate replication tests into parameterized variants by @Copilot in Consolidate duplicate replication tests into parameterized variants #7086
• chore: add replication stream tracing for double-apply bug by @romange in chore: add replication stream tracing for double-apply bug #7071
• fix: move FT.CREATE before data insertion in SearchSortByOptionNonSortableFieldJson test by @vyavdoshenko in fix: move FT.CREATE before data insertion in SearchSortByOptionNonSortableFieldJson test #7094
• fix(core): Update CVCUponInsert by @dranikpg in fix(core): Update CVCUponInsert #7068
• chore: Add pubsub-stress.py tool by @mkaruza in chore: Add pubsub-stress.py tool #7072
• fix(proactor_threads): fix that proactor_threads is not respected in k8s env by @starek4 in fix(proactor_threads): fix that proactor_threads is not respected in k8s env #7030
• chore: Disable clang format for src/redis directory by @mkaruza in chore: Disable clang format for src/redis directory #7097
• chore(snapshot): add version DCHECK invariant by @romange in chore(snapshot): add version DCHECK invariant #7096
• fix(hset): SIGABRT in HSETEX when called without field-value pairs by @vyavdoshenko in fix(hset): SIGABRT in HSETEX when called without field-value pairs #7100
• feat(search): add optional term frequency storage to CompressedSortedSet by @vyavdoshenko in feat(search): add optional term frequency storage to CompressedSortedSet #7093
• fix(server): Improve async eval command handling by @dranikpg in fix(server): Improve async eval command handling #7020
• fix: incorrect memory access for StringSet ttl entries by @BorysTheDev in fix: incorrect memory access for StringSet ttl entries #7102
• fix: handle STORE in unsorted path (SORT BY nosort STORE) by @vyavdoshenko in fix: handle STORE in unsorted path (SORT BY nosort STORE) #7114
• feat: preserve indexdata during serialization by @BorysTheDev in feat: preserve indexdata during serialization #7061
• fix(sort): delete destination key when SORT STORE produces empty result by @vyavdoshenko in fix(sort): delete destination key when SORT STORE produces empty result #7106
• Skip unstable test_hnsw_search_replication_with_network_disruptions by @Copilot in Skip unstable test_hnsw_search_replication_with_network_disruptions #7121
• fix: replica stuck during flushall by @BorysTheDev in fix: replica stuck during flushall #7120
• fix: prevent inline execution while shard lock is held by @vyavdoshenko in fix: prevent inline execution while shard lock is held #7116
• feat(fuzz): add optional snapshot serialization testing to AFL++ fuzzing by @vyavdoshenko in feat(fuzz): add optional snapshot serialization testing to AFL++ fuzzing #7119
• docs: Add GPG signing guidance to CONTRIBUTING.md by @romange in docs: Add GPG signing guidance to CONTRIBUTING.md #7122
• feat(memory): add MEMORY DECOMMIT COOL subcommand by @romange in feat(memory): add MEMORY DECOMMIT COOL subcommand #7124
• feat: search memory now more accurately reported by @BorysTheDev in feat: search memory now more accurately reported #7115
• fix(server): allow DFLY LOAD from cloud storage when --dir is local by @romange in fix(server): allow DFLY LOAD from cloud storage when --dir is local #7081
• refactor(rdb): use std::bit_cast (Fixes private chore(rdb): move object creation during loading to shard threads. #188) by @glevkovich in refactor(rdb): use std::bit_cast (Fixes private #188) #7130
• fix: use after free in HNSW by @BorysTheDev in fix: use after free in HNSW #7129

Huge thanks to all the contributors! ❤️

New Contributors

• @asherlaau made their first contribution in perf(server): optimize OpStrLen for tiered storage #6913
• @kanaevad made their first contribution in feat(aws): support EKS Pod Identity for S3 credentials #6917
• @shahyash2609 made their first contribution in fix: proactively delete expired keys on replicas via replica_delete_expired flag #6985
• @starek4 made their first contribution in fix(proactor_threads): fix that proactor_threads is not respected in k8s env #7030

Full Changelog: v1.37.0...v1.38.0

---

This discussion was created from the release v1.38.0.

[0x7s0lt1]

Docker update wen? frens? 🐸

[vyavdoshenko]

It works for me, please pull the latest tag.

➜  ~ docker run -d -p 6379:6379 --name df docker.dragonflydb.io/dragonflydb/dragonfly:latest
Unable to find image 'docker.dragonflydb.io/dragonflydb/dragonfly:latest' locally
latest: Pulling from dragonflydb/dragonfly
4e87bccff4c7: Pull complete 
5c978b9c33ee: Pull complete 
94afe42919e0: Pull complete 
488e09d6cc14: Pull complete 
4f4fb700ef54: Pull complete 
45f806a72dd5: Pull complete 
7d46fc055a26: Pull complete 
e495ca7a4c60: Pull complete 
Digest: sha256:243bf004df5e137d9432c35367d7c86e6d2b5bcb6700be4b6397fcb9306b794b
Status: Downloaded newer image for docker.dragonflydb.io/dragonflydb/dragonfly:latest
d42c74e1416b9b13856a570f095c6ef7fcb38358345ea260ef760d34a725eab6
➜  ~ redis-cli info server
# Server
redis_version:7.4.0
dragonfly_version:df-v1.38.0
redis_mode:standalone
arch_bits:64
process_id:7
os:Linux 6.12.54-linuxkit aarch64
thread_count:10
multiplexing_api:epoll
tcp_port:6379
uptime_in_seconds:12
uptime_in_days:0
hz:100
executable:dragonfly
config_file: