| If the GPUs need to send messages to each other often, as in ZeRO-DP, then faster connectivity becomes super important to achieve faster training. | |
| NVlink | |
| NVLink is a wire-based serial multi-lane near-range communications link developed by Nvidia. | |
| Each new generation provides a faster bandwidth, e.g. |