IB configuration is causing failures. Reverting to ethernet
which we confirmed works. IB can be enabled later once the
network setup is verified.
Co-Authored-By: Claude <noreply@anthropic.com>
The regex patterns for IB interfaces weren't working. Now letting
NCCL auto-detect the best available transport (IB or ethernet).
- Removed NCCL_IB_DISABLE (let NCCL decide)
- Removed NCCL_SOCKET_IFNAME (let NCCL auto-detect)
- Kept RoCE v2 configuration
Co-Authored-By: Claude <noreply@anthropic.com>
Now that the dynamic master discovery is working, re-enable IB
with proper configuration for better throughput:
- NCCL_IB_DISABLE=0 (enable IB)
- Regex patterns for IB interfaces (vary by node: ibp6s0, ibp7s0)
- RoCE v2 configuration (GID_INDEX=3)
- GPUDirect RDMA enabled (NET_GDR_LEVEL=5)
Co-Authored-By: Claude <noreply@anthropic.com>
The fundamental issue was that MASTER_ADDR was hardcoded to
airflow-worker-gpu-0, but rank 0 might run on any worker due to
Celery's dynamic task scheduling.
Now rank 0 dynamically stores its FQDN as the master address,
and all other ranks connect to it.
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix MASTER_ADDR with full namespace suffix
- Use ethernet instead of IB (IB was causing hanging)
- Add better error handling and diagnostic logging
- Add NCCL environment variable logging
Co-Authored-By: Claude <noreply@anthropic.com>
