Skip to content

sharding::strategy::fnv1a_shard — confirm the hashing scheme supports resharding without a full data migration #399

Description

@Christopherdominic

src/sharding/router.rs/rebalancer.rs use fnv1a_shard(&creator_username, num_shards) to deterministically map a creator to a shard — this reads like plain hash(key) % num_shards rather than consistent hashing (e.g. rendezvous hashing or a hash ring). With plain modulo sharding, changing num_shards (adding capacity) reassigns the shard for the majority of existing keys, not just a proportional fraction — meaning every capacity change requires migrating close to 100% of data rather than the ~1/new_num_shards a consistent-hashing scheme would require. Given this codebase already has a ShardRebalancer built specifically to move data between shards, it's worth confirming whether that machinery is actually sized for "rebalance an imbalanced fixed-N-shard cluster" (its apparent purpose, per RebalanceReport) or would also have to double as "migrate ~100% of data on any shard-count change," which is a much bigger and more disruptive operation that the current design may not obviously support online.

Task: Document whether num_shards is expected to ever change post-deployment. If yes, evaluate switching to a consistent-hashing scheme to bound the migration cost of capacity changes; if num_shards is meant to be fixed forever, document that constraint explicitly since it's a significant operational limitation that isn't otherwise stated anywhere obvious in the sharding module.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardThird CampaignCampaign: Third CampaignarchitectureStructural or architectural changedifficulty: very hardRequires deep expertise and substantial effortperformancePerformance, scalability, or resource usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions