48abe40c15
Fix DDP master address for dynamic task scheduling
...
The fundamental issue was that MASTER_ADDR was hardcoded to
airflow-worker-gpu-0, but rank 0 might run on any worker due to
Celery's dynamic task scheduling.
Now rank 0 dynamically stores its FQDN as the master address,
and all other ranks connect to it.
Co-Authored-By: Claude <noreply@anthropic.com >
2026-04-15 12:23:15 +03:00
ef78f2a7b8
Fix PyTorch DDP initialization for K8s with IB
...
- Fix MASTER_ADDR with full namespace suffix
- Use ethernet instead of IB (IB was causing hanging)
- Add better error handling and diagnostic logging
- Add NCCL environment variable logging
Co-Authored-By: Claude <noreply@anthropic.com >
2026-04-15 12:20:52 +03:00
93d8b86105
update dags
2026-04-15 12:18:58 +03:00
f84c133b00
update dags
2026-04-15 12:12:11 +03:00
6dddc69d11
update dags
2026-04-15 12:10:19 +03:00
93ff7c40b8
update dags
2026-04-15 12:07:04 +03:00
0b46d00bb2
update dags
2026-04-15 12:04:00 +03:00
9b365092b5
update dags
2026-04-15 12:02:41 +03:00
5f7db605a6
update dags
2026-04-15 11:54:13 +03:00
54ead9f90a
update dags
2026-04-15 11:46:01 +03:00
09b2c0041f
update dags
2026-04-15 11:42:19 +03:00
6c0105d85e
change user to root for airflow images
2026-04-15 11:34:56 +03:00
276c1ba6c5
update dags
2026-04-15 11:07:47 +03:00
b3b527f568
update dags
2026-04-15 10:51:12 +03:00
be1c81817d
update dags
2026-04-15 10:37:43 +03:00
c0b03e9ab1
update dags
2026-04-15 10:35:29 +03:00
481b978b4c
update dags
2026-04-15 00:10:17 +03:00
1a79410ad7
update dags
2026-04-15 00:07:51 +03:00
e3c12ad31b
update dags
2026-04-15 00:04:34 +03:00
8f3e78c16a
update dags
2026-04-14 23:58:12 +03:00
a9b3696815
update dags
2026-04-14 23:56:26 +03:00
2d4eef5218
build airflow images to internal repo
2026-04-14 11:58:37 +03:00
2abf72c452
build airflow images to internal repo
2026-04-14 11:56:44 +03:00
22fd8900ac
update graphs
2026-01-26 16:13:01 +03:00
8976195a17
update dags
2026-01-26 15:52:57 +03:00
2edf537df6
update dags
2026-01-26 15:49:45 +03:00
4049bddbef
update webserver secret name
2026-01-26 15:31:53 +03:00
cb5ec12dda
update dags
2026-01-26 15:13:02 +03:00
32478f1116
update dags
2026-01-26 15:03:06 +03:00
4b0cd05547
update dags
2026-01-23 01:02:07 +03:00
e8d305207b
update dags
2026-01-23 00:54:50 +03:00
caafb2fd62
update dags
2026-01-23 00:45:03 +03:00
a90f57ee89
update dags
2026-01-23 00:39:05 +03:00
59cc9d7995
update dags
2026-01-23 00:37:37 +03:00
526f201f41
init commit
2026-01-12 16:25:19 +03:00
22e2b44551
Initial commit
2026-01-12 16:23:39 +03:00
