Network scalability: 500 parachain validators and 100 cores (async backing enabled) #26

Parachain consensus is restricted to a subset of the relay chain validators via themaxValidators configuration parameter. The plan is to remove the parameter, but until then we will gradually increase it's value as we improve the protocol and performance characteristics of the implementation. maxValidators is bumped to higher values only if key performance metrics, specifically parachain block times and finality lag do not degrade when more validators participate in parachain consensus.

The scalability project board is available here: Road to 1k paravalidators and 200 parachains

After improving our tooling for profiling, gathering metrics and debugging we identified bottlenecks which we are currently addressing.

Experiments and potential improvements are tested on Versi before enabling them on value bearing chains like Kusama and Polkadot. After a battery of tests successfully ran on Versi we have already bumped the number of paravalidators from 200 to 250 on Kusama and we are currently monitoring the key metrics.

Q2 Plan

• Test Versi Scaling Limit polkadot#5962
• approval-distribution: batched approval/assignment sending polkadot#6401
• availability-recovery: prefer fetching small PoVs from backing group polkadot#6832
• Network Bridge error: notification queue size > 100k messages polkadot#6904
• Subsystem channel tweaks polkadot#6905
• approval distribution: trigger assignment/votes resend based on approval checking lag  polkadot#7038
• Better approval voting paramters polkadot-sdk#640

Q2 milestones:

• Increase the number of parachain validators from 200 to 250 (supporting up to 70 parachains) on Kusama
• Increase the number of parachain validators from 250 to 300 (supporting up to 70 parachains) on Kusama
• Increase the number of parachain validators from 200 to 300 (supporting up 70 parachains) on Polkadot

We've learned from previous tests, that it is already possible for Versi to operate within parameters with 500 paravalidators and 70 parachains, but we've also identified that availability recovery is getting slower as the load on the network I/O increase. Supporting 100 parachains would create even more network load so this needs to be addressed for this milestone.

Q4 Plan

• Use Glutton to stress test parachain consensus on Versi polkadot-sdk#620
• Investigate any potential availability bottlenecks during load tests and identify potential mitigation using fastpaths
• Networking tasks CPU usage is too high polkadot-sdk#702
• approval-voting improvement: include all tranche0 assignments in one certificate  polkadot-sdk#1178
• Delay reputation updates polkadot#7214
• approval-voting: enable v2 assignments polkadot-sdk#628
• approval-voting: coalesce multiple approvals in a single signed message polkadot-sdk#701
• Better approval voting paramters polkadot-sdk#640
• availability-recovery: run CPU intensive work on separate thread polkadot#7411
• Process PeerViewChange messages with priority polkadot-sdk#577
• availability-recovery: fetch available data from approval checkers polkadot-sdk#575
• Add availability-recovery from systematic chunks polkadot-sdk#1644

Q1 2024 milestone:

• Increase the number of parachain validators from 300 to 500 (supporting up to 100 cores) on Kusama
• Maybe start increase of the active set size on Polkadot.

2024

• Idea: Core Groups polkadot-sdk#607
• Rewrite approval voting and approval distribution as a single subsystem polkadot-sdk#1617