[HDDS-15069] design doc to introduce and describe OM multiraft feature - #10103
[HDDS-15069] design doc to introduce and describe OM multiraft feature#10103vtutrinov wants to merge 5 commits into
Conversation
| #### Assignment Metadata | ||
|
|
||
| The bucket-to-group assignment is stored: | ||
| 1. **In Bucket Metadata** (RocksDB): Each bucket stores its assigned RAFT group ID |
There was a problem hiding this comment.
what if some poweruser grow one bucket to very large?
There was a problem hiding this comment.
The bucket (BucketInfo's representation in Rocks) stores only GUID of the related raft group, not a huge amount of data. But the current design doc does not address the problem of concentrating write requests within a single RAFT group when users of the ozone cluster use the same bucket, and millions of write requests would hit the same bucket state machine. As a Phase 2, I suggest rewriting the router that selects the raft group to handle the write requests and allow it to handle a huge amount of requests within multiple RAFT groups
3fd291f to
91644c1
Compare
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
review is in progress |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
review is in progress |
ivandika3
left a comment
ivandika3
left a comment
There was a problem hiding this comment.
@vtutrinov I had a incomplete review, but let me share it first.
| public class OmRaftGroupsLeadershipBalancer { | ||
|
|
||
| /** | ||
| * Ensures RAFT group leaders are distributed across OM nodes. | ||
| * Target: Each OM node should be leader for ~equal number of groups. | ||
| */ | ||
| public void balanceLeadership() { | ||
| Map<String, Integer> nodeToLeaderCount = getCurrentLeaderDistribution(); | ||
|
|
||
| // If imbalance detected (max - min > threshold) | ||
| if (isImbalanced(nodeToLeaderCount)) { | ||
| // Transfer leadership from overloaded to underloaded nodes | ||
| for (RaftGroupId group : getOverloadedGroups()) { | ||
| String targetNode = selectUnderloadedNode(); | ||
| transferLeadership(group, targetNode); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **Balancer Strategy**: | ||
| - Runs periodically (default: every 5 minutes) | ||
| - Transfers leadership via RAFT `transferLeadership()` API | ||
| - Considers node health and load | ||
| - Graceful transfers to avoid disruption |
There was a problem hiding this comment.
Which OM node is running this?
There was a problem hiding this comment.
a leader node of the main raft group
| The multi-raft architecture partitions buckets write request across a configurable number of RAFT groups (default: 6). Each RAFT group: | ||
| - Has its own RAFT leader, followers, and log | ||
| - Processes write requests independently and in parallel | ||
| - Uses the same OM nodes but with different leaders |
There was a problem hiding this comment.
The main concern is that although we shard the buckets namespace to multiple Raft groups, we are still using one underlying OM DB.
For example, what happens if the leader of Raft group 2 sends a notifyInstallSnapshot to follower? Currently, this requires the follower to download the OM DB and all AppendEntries will be rejected. Can I check what is the expected behavior for this?
There was a problem hiding this comment.
For now (in MVP), a snapshot installation for a specific state machine could corrupt the trxId<->raftLogIndex state of other raft groups. Suggest per-node (not per-group) sync of the state machines (pause all state machines, sync, unpause) as an initial step (Phase 1) and separation of per-group DB or column families state as a Phase 2. A new commit with the suggestion above is ready
…ible resolutions - per-group snapshot installation could corrupt other groups states
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
szetszwo
left a comment
szetszwo
left a comment
There was a problem hiding this comment.
@vtutrinov , Thanks for submitting the design! Have two questions inlined.
BTW, we should just have a high level design first. Details like confs and code can be added later. A shorter design doc is easier to be reviewed and get approved.
|
|
||
| Ratis snapshot and install-snapshot assume a **1:1 relationship between a Raft log | ||
| and the persisted state** it protects. The multi-raft design breaks that assumption: | ||
| there are `N` bucket RAFT groups plus the main RAFT group, but they all apply into a |
There was a problem hiding this comment.
The term "main RAFT group" appears the first time here. What is it and why it is needed? We should explain it in the High-Level Design section.
There was a problem hiding this comment.
The definition was added
| Ratis snapshot and install-snapshot assume a **1:1 relationship between a Raft log | ||
| and the persisted state** it protects. The multi-raft design breaks that assumption: | ||
| there are `N` bucket RAFT groups plus the main RAFT group, but they all apply into a | ||
| **single shared OM RocksDB**. This means a snapshot taken for one group, and an |
There was a problem hiding this comment.
Should we use multiple OM RocksDBs?
There was a problem hiding this comment.
Sounds like a good idea (at least it's worth comparing a number of implementations and choosing the best one based on performance and maintainability); it's already described and suggested in the following section (take a look at Future direction: per-group state isolation (Phase 2))
Done. Remove the code and conf examples |
5 participants
What changes were proposed in this pull request?
Design doc for OM multiraft feature
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15069
How was this patch tested?
NA