Now that the dynamic master discovery is working, re-enable IB
with proper configuration for better throughput:
- NCCL_IB_DISABLE=0 (enable IB)
- Regex patterns for IB interfaces (vary by node: ibp6s0, ibp7s0)
- RoCE v2 configuration (GID_INDEX=3)
- GPUDirect RDMA enabled (NET_GDR_LEVEL=5)
Co-Authored-By: Claude <noreply@anthropic.com>
The fundamental issue was that MASTER_ADDR was hardcoded to
airflow-worker-gpu-0, but rank 0 might run on any worker due to
Celery's dynamic task scheduling.
Now rank 0 dynamically stores its FQDN as the master address,
and all other ranks connect to it.
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix MASTER_ADDR with full namespace suffix
- Use ethernet instead of IB (IB was causing hanging)
- Add better error handling and diagnostic logging
- Add NCCL environment variable logging
Co-Authored-By: Claude <noreply@anthropic.com>
