Loading...
Loading...
Compare original and translation side by side
TadaDeepSpeedPyTorchLlama-2-70BvLLMJIRA-12345PR #5751tadaLaunchPreparetada/prim.h::syncWaitPeerscratchBuf0e0a6bacscratchBuf == NULLTadaDeepSpeedPyTorchLlama-2-70BvLLMJIRA-12345PR #5751tadaLaunchPreparetada/prim.h::syncWaitPeerscratchBuf0e0a6bacscratchBuf == NULLJIRA-12345PR #5751JIRA-12345PR #5751JIRA-12345GET /rest/api/3/issue/<KEY>?fields=summary,status,priority,assignee,commentJIRA-12345GET /rest/api/3/issue/<KEY>?fields=summary,status,priority,assignee,commentPOST /rest/api/3/issue/<KEY>/commentPOST /rest/api/3/issue/<KEY>/commentMechanism: the single-stream fast-path in/tadaLaunchPrepare/tadaLaunchKernel(gated ontadaLaunchFinish) skipped the cross-stream event betweenscheduler->numStreams == 1 && !plan->persistentandlaunchStream. dumbModel hits this gate exactly. Kernel launched before deviceStream's IPC publish / scratch-buffer writes (the ones that populatehandle->shared->deviceStream) were visible to launchStream →scratchBufin the kernel → stray pointer dereference → ring ready-flag read from garbage → thread spins forever.scratchBuf == NULL
机制:/tadaLaunchPrepare/tadaLaunchKernel中的单流快速路径(由tadaLaunchFinish触发)跳过了scheduler->numStreams == 1 && !plan->persistent与launchStream之间的跨流事件。dumbModel恰好触发了这个条件。内核在deviceStream的IPC发布/临时缓冲区写入(用于填充handle->shared->deviceStream)对launchStream可见之前就已启动→ 内核中scratchBuf→ 野指针解引用→ 从垃圾数据中读取就绪标志→ 线程永久自旋。scratchBuf == NULL
Status: Fixed pending merge. Bug found, fix validated, PR up for review.Impact: LLM-7B fine-tuning on 8 GPUs would hang every time it tried to evaluate the model — blocking the entire workload. Affects customers using dumbModel (a popular framework for training large models that don't fit on a single GPU), which means most large-model fine-tuning runs on the platform were exposed.What broke: Our GPU communication library (Tada) skipped an internal synchronization step under a specific configuration that dumbModel happens to trigger. The GPUs ended up reading from an uninitialized buffer and got stuck waiting for a signal that would never arrive. The unsafe shortcut had been in the code for months but wasn't reached by any real workload until now.A previous fix attempt added a defensive check that hid the symptom in some paths but left the underlying race in place. This new fix removes the unsafe shortcut entirely and tightens the safety check on the device side.Owner: Alex (Tada team). PR org/platform#5751.Next steps: code review → merge. Customers hitting this today can disable IPC registration as a temporary workaround.
状态:修复完成,待合并。 已找到bug,修复方案已验证,PR已提交审核。影响范围: 8 GPU上的LLM-7B微调在每次评估模型时都会挂起——阻塞整个工作负载。影响使用dumbModel(一款流行的大模型训练框架,适用于无法在单GPU上运行的大模型)的客户,这意味着平台上大多数大模型微调任务都受影响。问题原因: 我们的GPU通信库(Tada)在特定配置下跳过了内部同步步骤,而dumbModel恰好触发了该配置。GPU最终读取了未初始化的缓冲区,并陷入永久等待信号的状态。这个不安全的捷径已存在于代码中数月,但从未被真实工作负载触发,直到现在。之前的修复尝试 添加了一个防御性检查,在某些路径上隐藏了症状,但未解决底层的竞态问题。本次新修复彻底移除了不安全的捷径,并收紧了设备端的安全检查。负责人: Alex(Tada团队)。PR org/platform#5751。下一步计划: 代码审核 → 合并。当前遇到该问题的客户可临时禁用IPC注册作为解决方案。
Tada hang affecting dumbModel LLM-7B fine-tuning is fixed pending merge. (JIRA-12345)
- Skipped synchronization in the comms fast-path → GPUs read uninitialized memory → hang. Latent for months; dumbModel was the first workload to hit it.
- Owner: Alex, PR #5751 in review.
- Workaround until merge: disable IPC registration.
影响dumbModel LLM-7B微调的Tada挂起问题已修复,待合并。(JIRA-12345)
- 通信快速路径中跳过同步步骤→ GPU读取未初始化内存→ 挂起。该问题已潜伏数月;dumbModel是首个触发它的工作负载。
- 负责人:Alex,PR #5751正在审核中。
- 合并前的临时解决方案:禁用IPC注册。
Fixed Tada hang on dumbModel LLM-7B (JIRA-12345). Alex's PR #5751 in review. Workaround posted in the ticket; backport to v7.2 next.
scratchBuftadaLaunchPrepare已修复dumbModel LLM-7B上的Tada挂起问题(JIRA-12345)。Alex的PR #5751正在审核中。工单中已发布临时解决方案;下一步回移植到v7.2版本。
scratchBuftadaLaunchPreparegit blamegit blame