Skip to content

Conversation

@hannahhoward
Copy link
Member

Goals

seperates out the process of adding nodes to shards to its own worker, by renaming and elevating nodes_in_shards to a node_uploads table with its own model -- and the ability to not be in a shard. This is useful cause the add-node-to-shard process is potentially error prone, so it's important to have a records of what shards need to be assigned still to nodes.

Implementation

  • new table node_uploads, instead of nodes_in_shards
  • pk is cid/space_did/upload_id (will only every assign to one shard_id)
  • migration to convert the existing nodes_in_shards to node_uploads
  • now, when we create a node, we create a node_upload record to record that it needs to be put in a shard
  • we have a query to check nodes not in shards
  • we have a step to add all outstanding nodes to shards
  • this does mean we have to read it off disk one more time to hash (when the shard is closed), but that's probably ok
  • we have a new worker to handle all this.

seperates out the process of adding nodes to shards to its own worker, by renaming and elevating
nodes_in_shards to a node_uploads table with its own model
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants