Skip to content

Agave-validator crashing periodically with "Failed to map the data file: Cannot allocate memory (os error 12)" #5774

Open
@akahand

Description

@akahand

Version: agave-validator 2.1.16 (src:5002c630; feat:3271415109, client:Agave)

[2025-04-11T13:20:47.084714298Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 736): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.
[2025-04-11T13:20:47.084724378Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 768): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.
[2025-04-11T13:20:47.084714948Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 544): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.
[2025-04-11T13:20:47.084725698Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 384): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.
[2025-04-11T13:20:47.084715668Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 608): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.
[2025-04-11T13:20:47.084727288Z ERROR solana_accounts_db::accounts_hash] Failed to map the data file (size: 384): Cannot allocate memory (os error 12). Please increase sysctl vm.max_map_count or equivalent for your platform.

Memory

               total        used        free      shared  buff/cache   available
Mem:           1.1Ti       395Gi        22Gi       1.0Mi       714Gi       730Gi
Swap:          4.0Gi        38Mi       4.0Gi

/etc/sysctl.d/21-solana-validator.conf

net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728
vm.max_map_count = 2000000
fs.nr_open = 1000000

First, vm.max_map_count was 1000000. Agave periodically crashed. An increase max_map_count to 200000 does not solve the problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions