Tracking issue: resharding

[timvisee]

This issue tracks progress on the resharding feature. It allows to dynamically grow the number of shards in a Qdrant collection.

Tasks

• Add basic resharding types #4216
• Combine (re)sharding hash rings in a single type, update split-by-shard logic #4236
• Derive Clone on hash ring types #4255
• Add ReshardingOperation::Start consensus message #4238
• Add StartResharding operation to update_collection_cluster REST API #4239
• Add shard transfer method for resharding #4217
• Switch back to upstream hashring crate #4305
• Deduplicate points for shard reads #4301
• Prepare re-sharding #3499
• Add ReshardingOperation::Abort consensus message #4290
• Move resharding state into shard holder #4346
• Resharding code cleanup #4349
• Send updates to the new shard during resharding #4256
• Refactor and cleanup resharding consensus operations #4351
• Add ReshardingOperations::CommitHashRing consensus message #4344
• add resharding status to API #4319

Backlog

A rough outline of new tasks on this feature:

• deduplicate count during resharding
• implement resharding "driver"
  • TBD
  • decide which approach we would implement
    • either dedicated task running on receiver node is driving resharding (@timvisee proposal)
    • or resharding is driven through consensus without a dedicated "driver" (@ffuugoo proposal)
• abort/cancel resharding
  • when participating peer is removed
  • is it possible to remove the only/last replica of a shard? if so, then we also need to cancel resharding when we remove shard replicas
  • when resharding transfer fails
  • when new shard is marked as Dead
    • ensure new shard can be marked as Dead, even if it's the only replica of a shard
• implement APIs to control resharding (or extend existing)
  • expose HTTP API in the OpenAPI spec
  • implement gRPC API
• handle resharding when restoring consensus snapshot
• report resharding in cluster info
• expose resharding structs in OpenAPI spec
• report correct number of points transferred in resharding transfer
• fix exact counts while resharding
• change resharding state structure from option to list, prepare for multiple concurrent operations in the future
• ensure compatibility with 1.9.x patch versions that already persisted default resharding state on disk