test: fuzz test for foyer-memory by xiaguan · Pull Request #337 · foyer-rs/foyer

xiaguan · 2024-04-14T11:07:50Z

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

As title

Checklist

I have written the necessary rustdoc comments
I have added the necessary unit tests and integration tests
I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

* chore: add license checker Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix license checker config Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…rs#15) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce FTL-like storage engine Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> * sort cargo file Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix memory leak Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: enable direct i/o on linux target - enable direct i/o on linux target - refine flusher and reclaimer Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix unit test Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…s#30) * chore: remove unused old storage engien and other components Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…-rs#31) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: impl storage recovery Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * export fifo fs store Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: export mods Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make cargo sort happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

This reverts commit a8a56cc.

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: deadlock fix deadlocks: 1. exclusive lock was not `Send` but unsafe impled. 2. submit flush task await blocks `set_region_evictable` 3. no new reclamation task created after `set_region_evctable` if all write process are waiting Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: gracefullly shutdown runners Use `Store::shutdown_runners()` to gracefully shutdown runners.` Signed-off-by: MrCroxx <mrcroxx@outlook.com> * log warn if send failed instead of panic Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce prometheus metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * calc metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix registry metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…oyer-rs#56) - support multiple admission or reinsertion policies - loose flush exclusive lock Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce rated random admission policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make clippy happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…er-rs#314) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…rs#315) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: some more Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…er-rs#317) * refactor: use crate allocator_api2 to make foyer build on stable Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: use stable pipeline for ci tests, use nightly for sanitizer Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: update ci prefix key Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: try fix ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: update ci nightly version Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: try resolve github issue Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* test: run CI on both linux and macos Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: fix build on macos Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: try fix ci run on targets Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* doc: update README and add a simple example Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: update Cargo.toml Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: fix ci coverage Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: fix check Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

) * refactor: remove Handle bound from Eviction Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: loose bound of in-memory eviction related traits Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* test: upload binaries when asan/lsan test fails Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: fix if condition Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: use different name for different artifacts Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: xiaguan <751080330@qq.com>

MrCroxx · 2024-04-16T08:57:42Z

Sorry for the delay. I've been working on some refactors these days.

What's the size of the fuzzy test dataset? Should it also be uploaded?

xiaguan · 2024-04-16T09:16:17Z

foyer-memory/fuzz/fuzz_targets/cache_fuzz.rs

+#[derive(Debug, Arbitrary)]
+struct Input {
+    capacity: usize,
+    cache_type: CacheType,
+    operations: Vec<Op>,
+}


The cargo-fuzz just random generate the Input to max the code coverage with sanitizer.
The afl fuzz seems better one, it need some input to start,and it is hard to use and maintain.

MrCroxx added 30 commits May 26, 2023 16:35

chore: add license checker (foyer-rs#14)

5c09e60

* chore: add license checker Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix license checker config Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: wrapper cache type, reorg utils, add metrics framework (foyer-…

e38ad6a

…rs#15) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: fix read only store recovery, impl random drop (foyer-rs#16)

f44aec1

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add metrics (foyer-rs#17)

8e80c6c

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add log support (foyer-rs#19)

79610aa

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add write stall config for read only file store (foyer-rs#20)

23e2612

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: introduce instrusive indexers and collections (foyer-rs#21)

506f3ef

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: remove unused deps (foyer-rs#28)

64b72b4

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: foyer storage bench support flexible writers and readers (foyer…

85d8341

…-rs#31) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: impl storage recovery (foyer-rs#32)

7c81272

* feat: impl storage recovery Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: introduce segment fifo eviction policy (foyer-rs#35)

ad3f125

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * export fifo fs store Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: export mods (foyer-rs#37)

98a0af7

* feat: export mods Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make cargo sort happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: export extern crate (foyer-rs#38)

a8a56cc

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Revert "fix: export extern crate (foyer-rs#38)" (foyer-rs#39)

09bf57a

This reverts commit a8a56cc.

fix: disable O_DIRECT on non-linux targets (foyer-rs#40)

691054a

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: ignore Cargo.lock and bump toolchain (foyer-rs#46)

7fc8542

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: export storage config (foyer-rs#50)

1018bca

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: AdmitAll & ReinsertNone no longer requires KV default (foyer-rs#51)

011a401

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: region advance & recovery & seal (foyer-rs#53)

e61eb5d

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: recovery test (foyer-rs#55)

3c0a0ca

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: support multiple admission or reinsertion policies, loose lock (f…

809ca83

…oyer-rs#56) - support multiple admission or reinsertion policies - loose flush exclusive lock Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: introduce rated random admission policy (foyer-rs#57)

32ec880

* feat: introduce rated random admission policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make clippy happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add rated random option for bench args (foyer-rs#58)

d037322

fix: recovery run flushers and reclaimers first (foyer-rs#59)

e502b30

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

MrCroxx and others added 20 commits April 9, 2024 14:31

fix: fix state() for s3fifo (foyer-rs#312)

b50813b

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer-memory to 0.1.4 (foyer-rs#313)

5a410c8

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: use generic type for cursor instead of associated type (foy…

823a879

…er-rs#314) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: remove unstable feature let_chains and lint_reasons (foyer-…

3aa689d

…rs#315) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: remove usage of unstable features (foyer-rs#316)

2693669

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: some more Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

doc: update README and add a simple example (foyer-rs#319)

475aee0

* doc: update README and add a simple example Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: update Cargo.toml Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: add msrv for foyer, update bager in readme (foyer-rs#321)

2abe3df

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: do not truncate when open wal (foyer-rs#322)

cb2616e

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer to 0.7.0 (foyer-rs#323)

48c3990

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: loose query key bound for in-memory cache (foyer-rs#324)

cc64fc9

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: impl contains for in-memory cache (foyer-rs#325)

e47c090

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

perf: add dynamic dispatch bench (foyer-rs#326)

818e3f9

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: fix ci coverage (foyer-rs#331)

3fda8cc

* chore: fix ci coverage Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: fix check Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: refine generic handle type (foyer-rs#330)

6b820fa

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

test: fuzz test for foyer-memory

541a329

more readable fuzz test

1f7f231

Signed-off-by: xiaguan <751080330@qq.com>

xiaguan force-pushed the memory_fuzz_test branch from 7943b4a to 1f7f231 Compare April 14, 2024 14:18

Merge branch 'main' into memory_fuzz_test

3f48bf4

xiaguan force-pushed the memory_fuzz_test branch from a33b1d0 to 348d135 Compare April 14, 2024 14:30

add license

75c8a51

Signed-off-by: xiaguan <751080330@qq.com>

xiaguan force-pushed the memory_fuzz_test branch from 348d135 to 75c8a51 Compare April 14, 2024 14:57

xiaguan commented Apr 16, 2024

View reviewed changes

MrCroxx force-pushed the main branch from 04d5ac3 to a2b91e3 Compare April 17, 2024 01:00

MrCroxx force-pushed the main branch 2 times, most recently from 3f7f53f to c009129 Compare September 12, 2025 05:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: fuzz test for foyer-memory#337

test: fuzz test for foyer-memory#337
xiaguan wants to merge 264 commits intofoyer-rs:mainfrom
xiaguan:memory_fuzz_test

xiaguan commented Apr 14, 2024 •

edited

Loading

Uh oh!

MrCroxx commented Apr 16, 2024

Uh oh!

xiaguan Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

xiaguan commented Apr 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Related issues or PRs (optional)

Uh oh!

MrCroxx commented Apr 16, 2024

Uh oh!

xiaguan Apr 16, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xiaguan commented Apr 14, 2024 •

edited

Loading