TRACE was too verbose. INFO shows clean transport selection lines
(NET/IB, GPU Direct RDMA status) for both ranks via stderr redirect.
Per-batch logging removed; per-epoch summary kept with timing.
Co-Authored-By: Claude <noreply@anthropic.com>
- Redirect stderr to stdout via dup2 so Airflow captures NCCL TRACE
output for all ranks (not just rank 0)
- Log per-batch loss for all ranks
- Log per-epoch summary with local/global avg loss and timing
- Log total training time
Co-Authored-By: Claude <noreply@anthropic.com>
Set NCCL_IB_GPU_DIRECT_RDMA_MAX_DISTANCE=6 to allow GPU Direct RDMA
despite the 6-hop PCI distance between GPU (bus 0) and IB HCA (bus 6).
Restore NCCL_DEBUG=INFO since IB transport is confirmed working.
Co-Authored-By: Claude <noreply@anthropic.com>
Root cause: Subnet manager assigns different PKEYs to ports
- test-1 (LID 0x44): Full membership PKEYs (0xa001, 0x9001)
- test-2 (LID 0x7d): Limited membership PKEY 0x7fff only
This causes MTU mismatch (4092 vs 2044) and NCCL failure.
Cluster admin needs to configure opensm partitions to fix.
Ethernet is working and stable.
Co-Authored-By: Claude <noreply@anthropic.com>
- Set NCCL_IB_DISABLE=0 to enable IB
- Add IB-specific config (GID_INDEX, TC, TIMEOUT, RETRY_CNT, NET_GDR_LEVEL)
- Update deprecated NCCL_BLOCKING_WAIT to TORCH_NCCL_BLOCKING_WAIT
- Set NCCL environment variables before torch imports for proper initialization
Co-Authored-By: Claude <noreply@anthropic.com>
IB attempts are failing because /proc/drivers/infiniband is not
mounted in the container (container runtime security restriction).
Even though RDMA devices are exposed, NCCL needs /proc/drivers/infiniband
for proper IB initialization. This requires container image changes.
Co-Authored-By: Claude <noreply@anthropic.com>
The RDMA device plugin is running and devices are exposed
(/dev/infiniband/*). Try enabling IB with settings that
work with containerized IB where /proc/drivers/infiniband
may not be accessible.
Co-Authored-By: Claude <noreply@anthropic.com>
Two critical fixes:
1. Set device BEFORE initializing process group (PyTorch best practice)
2. Add NCCL environment variables to handle same GPU IDs across nodes:
- NCCL_LL_THRESHOLD=0
- NCCL_ALGO=Ring
- NCCL_PROTO=Simple
This fixes the "Duplicate GPU detected" error when both nodes
have GPUs with the same PCI bus ID.
Co-Authored-By: Claude <noreply@anthropic.com>
The sync_state variable was out of scope for non-zero ranks,
causing them to use an old sync_state value from an earlier
loop iteration. Now properly initialize and use master_addr
variable in both branches.
Co-Authored-By: Claude <noreply@anthropic.com>
IB configuration is causing failures. Reverting to ethernet
which we confirmed works. IB can be enabled later once the
network setup is verified.
Co-Authored-By: Claude <noreply@anthropic.com>
The regex patterns for IB interfaces weren't working. Now letting
NCCL auto-detect the best available transport (IB or ethernet).
- Removed NCCL_IB_DISABLE (let NCCL decide)
- Removed NCCL_SOCKET_IFNAME (let NCCL auto-detect)
- Kept RoCE v2 configuration
Co-Authored-By: Claude <noreply@anthropic.com>
Now that the dynamic master discovery is working, re-enable IB
with proper configuration for better throughput:
- NCCL_IB_DISABLE=0 (enable IB)
- Regex patterns for IB interfaces (vary by node: ibp6s0, ibp7s0)
- RoCE v2 configuration (GID_INDEX=3)
- GPUDirect RDMA enabled (NET_GDR_LEVEL=5)
Co-Authored-By: Claude <noreply@anthropic.com>
The fundamental issue was that MASTER_ADDR was hardcoded to
airflow-worker-gpu-0, but rank 0 might run on any worker due to
Celery's dynamic task scheduling.
Now rank 0 dynamically stores its FQDN as the master address,
and all other ranks connect to it.
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix MASTER_ADDR with full namespace suffix
- Use ethernet instead of IB (IB was causing hanging)
- Add better error handling and diagnostic logging
- Add NCCL environment variable logging
Co-Authored-By: Claude <noreply@anthropic.com>
