16 posts tagged with "Training"

训练和推理抢卡、分布式作业因部分调度而死锁、GPU 利用率常年趴在低位——这些问题的根源往往是调度层设计。Ray 与 Kubernetes 生态各自给出了一套答案，关键是怎么组合。

两条路线的心智模型​

路线一：Ray 为中心。 把 Ray 当作计算编排层，Kubernetes 只负责运行 Ray Pod。KubeRay Operator 管理 RayCluster、RayJob、RayService 三类资源，Ray 自动伸缩器根据任务与 actor 的资源需求增删 worker 组。优点是 Python 世界的体验极好，任务级弹性、异构资源混用（不同代次 GPU 混部）都是内置能力；代价是调度决策发生在集群之外，Kubernetes 只看到结果。

路线二：Kubernetes 为中心。 用批调度器直接管理 GPU 作业，代表性组件是 Volcano 与 Kueue。Volcano 提供 Gang 调度、队列公平共享、抢占与网络拓扑感知调度；Kueue 做作业准入控制，排队等资源到位再放行。Ray 作业同样能被纳管：KubeRay 从早期版本就集成 Volcano 解决分布式训练的资源死锁，2025 年发布的 Volcano v1.13 进一步支持直接用原生 Volcano Job 创建和管理 Ray 集群。

能力对比​

能力	Ray 自动伸缩	Volcano	Kueue	NVIDIA KAI Scheduler
Gang 调度	弱（依赖 minReplicas）	原生支持	原生支持	原生支持
任务级弹性伸缩	强，秒级感知任务需求	作业级	队列准入级	支持工作负载伸缩
抢占与优先级	部分支持	支持	配合优先级类	高优推理可抢占低优训练
网络拓扑感知	无	v1.12 起 IB 拓扑，v1.13 支持 Label 自动发现	无	无
异构 GPU 混部	强	强	强	强

实践中的组合形态​

最常见的生产形态是"KubeRay 加批调度器"：RayCluster 的 Pod 标注调度器与队列，Gang 调度保证多机训练作业要么全起要么不起；启用了自动伸缩的集群按 minReplicas 做 Gang 约束，兼顾弹性与死锁防护。大型团队还会叠 NVIDIA KAI Scheduler，让在线推理在算力紧张时抢占离线训练。工程细节上，KubeRay 在大规模集群下要调大 reconcile 并发与客户端 QPS，社区压测显示默认并发配置扩容上万个 Pod 会明显变慢。Volcano 的基于节点标签的 HyperNode 自动发现功能则把"机架到交换机到节点"的拓扑维护简化为打标签，对要求亲和性调度的 IB 集群很实用。

实操建议​

纯 Python 研发团队、任务粒度碎、需要异构混部的场景，以 Ray 为中心最省心；作业规模大、需要队列治理与多团队配额的平台团队，以 Volcano 或 Kueue 为骨架、Ray 作为负载接入。两条路线不冲突，先想清楚"谁拥有资源审批权"这个组织问题，技术选型自然清晰。

训练选型的本质不是"买最强的"，而是在预算、负载规模、软件生态和供货确定性之间找到平衡点。本文数据均取自本站芯片卡与定价库，帮你按预算分档做出决策。

一、先分级：你的训练负载属于哪一档​

训练负载大致分为三档，档位不同，选型逻辑完全不同：

• 微调 / LoRA：单卡或 8 卡即可完成，7B–70B 模型为主。瓶颈在显存容量与单卡带宽，互联要求低，PCIe 机型甚至二手卡都能胜任。
• 继续预训练 / 领域适配：数百卡规模，需要稳定的多卡互联与成熟的软件栈，容错与断点续训能力开始重要。
• 从零预训练：千卡乃至万卡集群，互联带宽、机架方案（NVL72、Helios、CloudMatrix）和集群软件效率决定成败，预算通常以千万美元计。

二、按预算分档推荐​

预算档位	推荐方案	关键规格	参考单价（站内定价库）
入门微调（数十万元级）	H100 SXM / H200	BF16 1,979 TFLOPS（稀疏）、80GB / 141GB HBM3e、NVLink 900 GB/s、700W 风冷	H100 市场价约 $28,000；H200 约 $38,000
中等规模（数百万元级）	B200 / AMD MI355X	B200：192GB HBM3e、FP8 10 PFLOPS（稀疏）、NVLink 1.8 TB/s；MI355X：288GB HBM3E、FP8 dense 5 PFLOPS、UALink 600 GB/s	B200 约 $42,000；MI355X 官方指导价 $22,000
旗舰预训练（千万到亿级）	B300 Ultra / Rubin R200 / MI455X	B300：288GB HBM3e、FP8 dense 7 PFLOPS；Rubin：288GB HBM4、22 TB/s、NVLink 6 3.6 TB/s；MI455X：432GB HBM4、23.3 TB/s、MXFP4 40.3 PFLOPS	B300 约 $52,000；Rubin NVL72 机柜 350–400 万美元；MI455X 定价未公开（Helios 机柜约 500–550 万美元）
国产合规档	昇腾 910C / 昇腾 950DT / 寒武纪思元690	910C：BF16 800 TFLOPS、128GB HBM2e；950DT：FP8 约 1 PFLOPS、144GB 自研 HiZQ 4 TB/s；思元690：FP16 700+ TFLOPS、196GB HBM3	910C 渠道价超 ¥11 万；950DT 超 ¥25 万；思元690 约 ¥17.5 万

几个要点值得展开：

• 入门档首选 H100/H200 并不落伍。H100 的 BF16 稀疏算力 1,979 TFLOPS、700W TDP 风冷友好，生态最成熟；H200 显存加到 141GB、带宽 4.8 TB/s，微调 70B 级模型更从容。
• 中等规模档 B200 与 MI355X 各有侧重。MI355X 用 288GB HBM3E 和 3nm 制程在显存容量上压过 B200，官方口径 FP8 dense 5 PFLOPS 也略胜，价格还便宜约四成；但 B200 的 NVLink 1.8 TB/s 是 UALink 600 GB/s 的三倍，千卡扩展时差距会显现。
• 旗舰档看机架而不是单卡。B300 Ultra 必须液冷（TDP 1,400W），Rubin R200 的 Vera Rubin NVL72 机柜 FP4 算力约 3.6 EFLOPS、总显存 20.7TB；MI455X 的 Helios 机架 72 卡 31TB HBM4、机柜功耗 225–245kW。这一档的采购单位是"机柜"，还要配套液冷与供电改造。

三、关键参数怎么读​

• FP16/BF16 算力：注意 dense 与 sparse 口径差一倍。H100 的 1,979 TFLOPS 是稀疏值，dense 约 989 TFLOPS；比较时务必统一口径。
• 显存容量：从 H100 的 80GB 到 B300 的 288GB 再到 MI455X 的 432GB，容量决定了能训多大模型、能开多大 batch。训练 70B 模型 FP16 权重本身就要约 140GB，加上优化器状态后通常需要多卡切分。
• 互联带宽：NVLink 4.0（900 GB/s）→ NVLink 5（1.8 TB/s）→ NVLink 6（3.6 TB/s）；AMD 走开放路线的 UALink 600 GB/s；昇腾为 HCCS 784 GB/s 与灵衢 2 TB/s。张量并行通信密集，互联带宽直接决定扩展效率。
• 机架方案：GB200/GB300 NVL72、Helios、CloudMatrix 384 是三家给出的系统级答案。单卡强不等于集群强，采购前要确认整机柜方案、网络与液冷的完整交付能力。

四、软件生态权重：CUDA vs ROCm vs CANN​

• CUDA：十八年积累，算子覆盖最全，从 H100 到 Rubin 全线兼容，迁移成本最低。
• ROCm：AMD 官方口径承认成熟度仍落后 CUDA 约 2–3 年，但开放生态对云厂商和主权 AI 项目有吸引力，MI355X/MI455X 的客户名单里已有 Microsoft Azure、Meta、Oracle 等。
• CANN：昇腾 950 系列配套 CANN Next 提供 CUDA 兼容层，约 80% 的标准 PyTorch 代码仅需配置修改即可迁移；2026 年 10 月 DeepSeek 与华为还联合发布了面向昇腾 950 的开源编程工具与库。

经验法则是：微调档闭眼选 CUDA；中等规模档如果显存需求突出、团队有 ROCm 经验，MI355X 值得评估；合规档则要在选型时就把迁移人力写进预算。

五、供货与价格现实​

2026 年的选型绕不开 HBM：

• 涨价传导：HBM 成本攀升直接推高国产卡报价——昇腾 950DT 渠道指示价已超 ¥25 万（较两个月前上涨 20%–50%），昇腾 950PR 突破 ¥8 万，寒武纪等同期上调 20%–30%。
• 云租是一条现实路径：H100 云租约 $2.00/GPU·小时，B200 约 $5.87/GPU·小时，B300 的 AWS 8 卡实例约 $11.70/GPU·小时（Nebius 口径约 $9.50/GPU·小时）。训练周期短的项目，租往往比买划算。
• 旗舰卡要排队：Rubin R200 已于 2026 年 6 月全面量产，但自建机房用户要到 2027 年一季度才能采购；MI455X 也需到 2026 年下半年至 2027 年上半年才规模爬坡。急用算力，H100/H200/B200 的现货市场仍是主力。

想逐项对比更多型号（包括 Google TPU、Intel Gaudi 4、百度昆仑芯等），请查看站内完整对比表；每张卡的详细规格可进入对应芯片卡页面查看。

小结​

2026 年的训练 GPU 市场第一次出现了"四条路线"：CUDA 旗舰（Rubin/B300）、开放生态（MI355X/MI455X）、推理专用硬件和国产合规方案。选型的顺序应该是：先定负载档位，再按预算筛卡，然后核算互联与机架方案，最后把软件迁移和供货周期计入总成本。单卡峰值参数只讲了一半的故事，剩下的另一半在集群和生态里。

"推理占比什么时候超过训练"曾是个经典预测题，如今它已经有了答案：2026 年，推理算力占比正式过半。真正的悬念变成了——推理时代的算力基建，准备好了吗。

占比反转：数据给出的答案​

多家机构对训练与推理的算力结构给出了趋于一致的判断：

口径	关键数据	来源
全球 AI 负载占比	2024 年训练约六成、推理约四成；2025 年五五开；2026 年训练四成、推理六成	Epoch AI、AMD 公开材料
推理负载占比	从 2023 年约三分之一升至 2025 年约一半，2026 年预计达到三分之二	德勤估算
中国智能算力	2027 年推理占比预计突破七成；2028 年全球推理负载占比约七成三	IDC 预测
AI 云服务支出	2026 年全球 AI 优化型 IaaS 支出约 375 亿美元，推理占比 55%，首次超过训练	Gartner 预测

鼎帷咨询等机构进一步预计，训练与推理负载比将从早期的 7 比 3 演进到 2027 至 2030 年的 3 比 7 甚至 2 比 8。无论采用哪个口径，方向完全一致：算力的重心正在从"炼模型"迁移到"用模型"。

三股力量：服役、深思、智能体​

推理占比跃升由三重动力叠加驱动。第一，模型进入服役期：训练是一次性资本开支，推理是每天持续产生的运营账单——据业界估算，头部对话类产品的推理开销以每天数十万美元计，进入规模化生产后，推理可占模型全生命周期算力成本的八到九成。第二，推理时 Scaling：思维链模型用"内部深思"换取智能，用户看不见的推演 token 同样按价计费，单次请求的算力需求成倍膨胀——模型越会深思熟虑，推理阶段的需求通胀就越厉害。第三，智能体乘数：一次 Agent 任务消耗的 token 从数百跃升到数万，多轮链式思考叠加工具调用，使推理流量呈现碎片化与高波动特征。

中国市场的信号同样鲜明：据公开数据，全国日均 token 消耗从 2024 年初的千亿级跃升至 2026 年 3 月的约 140 万亿，一年半增长约三百倍——需求的斜率比任何预测模型都陡峭。

硬件逻辑的重新洗牌​

占比反转直接改写芯片设计与采购逻辑。训练硬件追求极致性能与集群可靠性，一次训练动辄连续运行数周，故障率必须压到极低；推理硬件的 KPI 则是每瓦 token 数与每百万 token 成本：FP4 与 INT8 级别的低精度成为默认态，显存容量与带宽决定长上下文的经济性，时延与吞吐的权衡成为核心竞争力。这解释了 2026 年推理专用芯片赛道的集体爆发——从英伟达面向推理优化的产品线，到云厂商自研 ASIC 与推理芯片创业公司的融资潮，都在响应同一个结构性信号。

结语：推理时代的真正考题​

推理时代不是某个季度财报里的占比数字，而是一整套产业逻辑的更替：计费从"卡时"转向"token 成本"，采购从"峰值算力"转向"单位经济学"，评测从训练基准转向真实业务负载的端到端表现。可以判断：2026 年是推理占比过半的符号性年份，而 2027 到 2028 年，推理将稳固占据七成以上份额。届时，所有今天围绕训练设计的集群、网络与商业模式，都要回答同一个问题——为推理做好准备了吗。

当算力的计量单位从"多少张卡"变成"多少吉瓦"，AI竞争的本质已经改变：大模型公司正在变成"能源公司+基建公司+模型公司"的三位一体。

5GW与10GW：新计量单位的诞生​

2026年9月的行业报道显示，AI算力竞赛进入了一个全新的量级：Anthropic的目标是在今年年底拥有约5GW的可用算力，而OpenAI明年的规划则是约10GW。

这两个数字意味着什么？5GW大致相当于数百万张高性能GPU同时满负荷运转所需的电力，而10GW已经接近一座大型核电站的装机容量。就在一年前，行业讨论的还是"十万卡集群"；如今，头部玩家谈论的是吉瓦级园区。算力竞赛的计量单位，已经从GPU的数量彻底转向了电力的功率。

从"买卡"到"买电厂"​

这种变迁背后，是AI产业价值链的整体上移：

• 能源成为第一约束。先进AI芯片供应在改善，但变电站、输电线路和发电容量的建设周期以年计。谁拿到电力指标，谁就拿到训练下一代模型的入场券。
• 基建能力成为核心竞争力。数据中心不再是租用IDC机房那么简单，而是涉及选址、并网、冷却系统乃至自有发电能力的系统工程。
• 模型公司与芯片公司深度绑定。OpenAI与博通合作开发Jalapeño自研芯片，并规划多吉瓦级部署，这已经不是"采购算力"，而是"定制算力供应链"。

20-30MW小数据中心为何成为抢手货​

一个耐人寻味的细节是：Anthropic和OpenAI两家还在争抢20-30MW级别的小型数据中心，Anthropic甚至向SpaceX租用算力。

原因很直接——大容量数据中心从规划到投运周期太长，而模型训练和推理需求等不起。20-30MW的小站点虽然容量有限，但胜在可快速拿容量：现有楼宇改造、电力接口现成、数月内即可上线。对于急需算力缺口的大模型公司，这批"小快灵"资源成了战略物资，甚至在二级市场上被反复转租、溢价争夺。向SpaceX租算力则更为激进——当地面电力不够快时，头部玩家已经开始评估包括星载算力在内的非常规选项。

算力军备竞赛的经济账​

这场竞赛的规模有多大？根据Bain 2026技术报告的口径，AI硬件市场2025年约为312.1亿美元，2026年预计达到384.9亿美元。而吉瓦级算力园区背后，是单笔动辄数十亿美元级别的芯片采购与多年期电力合约。

算力的本质正在从"IT设备"变成"重资产基础设施"——它的融资方式（项目融资、长期租约）、回报模型（算力利用率、每token成本）都越来越像电厂和通信网络，而不是服务器机房。

对AI芯片厂商意味着什么​

吉瓦级竞赛重塑了芯片厂商的客户结构，谁绑定大客户谁受益的格局愈发清晰：

厂商	核心绑定关系	算力绑定规模
NVIDIA	微软 / OpenAI	深度绑定OpenAI，吉瓦级部署主力供应商
AMD	OpenAI / Meta / Anthropic	OpenAI 6GW、Meta 6GW、Anthropic（多年期协议）
OpenAI（自研）	博通合作Jalapeño	多吉瓦级部署规划
云厂商自研	AWS Trainium、Google TPU等	服务自有平台负载

对芯片厂商而言，这带来两个深远影响：其一，订单变得极度集中，一张吉瓦级订单足以改变一家芯片公司的营收曲线，NVIDIA与AMD过去一年股价的分化某种程度上就是绑定关系的映射；其二，芯片设计开始围绕客户的电力预算优化，每瓦性能取代峰值算力成为核心指标——因为客户的稀缺资源不是芯片，而是那几吉瓦的电。

小结​

5GW和10GW不只是两个数字，它们标志着AI竞争进入了"基建时代"：电力取代芯片成为第一约束，20-30MW小数据中心成为快速补位的战略资源，大模型公司集能源、基建、模型三重身份于一身。对AI芯片产业来说，这意味着订单向深度绑定大客户的产品集中、设计向每瓦性能倾斜。算力军备竞赛的下半场，比的不是谁的芯片跑分更高，而是谁能先把电厂建起来、把电接进来。

On September 22, 2026, multiple media outlets reported that DeepSeek's CEO explicitly stated the company is betting heavily on Huawei chips, expects to receive a new batch of Huawei chips for large-model training, and stressed that "this choice must succeed." Following earlier reports that "DeepSeek planned to procure 160,000 Ascend 950DT units for inference," this is the most significant alignment signal yet in the domestic compute ecosystem — an upgrade from inference procurement to a training bet.

---

1. The Signal Chain: Four Steps to a "Training Bet"​

Piecing together the public information from the past six months, the DeepSeek–Ascend collaboration shows a clear escalation path:

Time	Signal	Nature
Mid-2026	DeepSeek V4 completes ecosystem migration from CUDA to CANN	Software adaptation
August 2026	Report: DeepSeek plans to procure 160,000 Ascend 950DT units for inference deployment	Inference procurement intent
2026-09-17	HC2026: 960DT ready three quarters ahead of schedule, 950DT ramping in Q4	Supply delivery
2026-09-22	DeepSeek CEO: betting on Huawei chips for LLM training, "must succeed"	Training bet

The key change is the workload tier: inference deployment means "cutting costs with off-the-shelf compute," while a training bet means "staking the existence of next-generation models on domestic chips" — training clusters demand an order of magnitude more in stability, interconnect efficiency, and software-stack maturity.

2. Huawei's Ability to Deliver​

DeepSeek's willingness to commit rests on a series of verifiable progress points on Huawei's supply side (all official figures):

• One generation per year, delivered: the 950PR is in mass production, the 950DT ramps in 2026 Q4, and the 960DT moved three quarters earlier than its original 2027 Q4 plan to ready in 2027 Q1 / launch in Q2 — the roadmap's credibility validated twice in a row;
• Deployment scale: Ascend supernodes have been commercially deployed at scale in over 1,000 sets, covering internet, finance, healthcare, and manufacturing;
• Ecosystem maturity: CANN has entered routine open-source operation, with external developers exceeding 61% for the first time and 5,200 monthly active developers; there are over 40 Ascend-native training models, making it the only domestic technology route supporting pretraining;
• Training evidence chain: China Telecom's Xing 4.0-29B-A4B agentic MoE model, open-sourced in September, was announced as trained end-to-end on Ascend (company claim) — "Ascend can train large models" is no longer just Huawei's self-attestation.

See the Ascend 950DT and Ascend 960 spec pages for details; for supernode analysis, see Ascend 960 Official Launch.

3. Why DeepSeek?​

DeepSeek's choice has strong structural drivers:

1. Supply certainty: under export-control constraints, NVIDIA's flagship supply to China keeps tightening; domestic compute is the only plannable large-scale training supply;
2. Cost structure: DeepSeek has always been known for extreme engineering efficiency (the V3 training cost set the industry benchmark), and domestic compute plus supernode system efficiency fits its approach;
3. Betting on ecosystem dividends: CANN open-sourcing plus deep binding with leading model vendors means model vendors can participate in shaping the toolchain's direction — something impossible within CUDA's closed system;
4. Self-fulfilling demonstration effect: a leading lab's public commitment pulls back on Huawei's production scheduling and upstream HBM and system investment, making "must succeed" a rational commitment rather than a slogan.

4. Risks and Open Questions​

Viewed coolly, this route still has three items that need time to verify:

• Actual training scale: the quantity, model type (950DT or 960DT), and delivery schedule of the new batch of chips are all undisclosed;
• MFU methodology: the MFU / latency gains Huawei cites all come from Markov-lab simulations, with no independent third-party measurements yet; the real effective compute of a training cluster depends on long-term data from large-scale production environments;
• Per-card generation gap: the 960DT's 4 PFLOPS (FP4) versus Rubin R200's 50 PFLOPS (FP4) — the per-card gap objectively exists, and training efficiency depends on whether supernode scale and software optimization can compensate; that is precisely the decisive battleground of "system-level competition."

5. Summary​

• DeepSeek confirms betting on Huawei chips for training large models — the first training-grade commitment from a leading lab in domestic compute;
• Signal chain: CUDA→CANN migration → 160,000-unit 950DT inference procurement rumor → 960 ready ahead of schedule → training bet;
• Supporting factors: one-generation-per-year delivery, 1,000+ supernode sets, and 61% external developers in the CANN open-source ecosystem;
• What to watch: actual arrival of the new chips and training-cluster scale, third-party MFU data, and training-compute disclosure in DeepSeek's next release.

---

Further Reading​

• Ascend 960 spec page / Ascend 950DT spec page (this site)
• Ascend 960 Official Launch: FP4 Compute Doubled, World's First NPO Supernode
• Domestic Top Three in H2 2026: Localization Rate Breaks 40%, Heading Toward 60%
• Alibaba Zhenwu V900 Unveiled at Apsara Conference

References​

• Toutiao Tech Morning Report: DeepSeek confirms betting on Huawei chips for large-model training (2026-09-22)
• HUAWEI CONNECT 2026 official announcements (2026-09-17)
• Earlier report: DeepSeek plans to procure 160,000 Ascend 950DT units (2026-08)

This article is compiled from public reports and vendors' official statements. Details such as chip quantities and training-cluster scale are subject to subsequent disclosures from DeepSeek and Huawei.

On June 5, 2026, Shenzhen announced a major piece of news: Shenzhen Hetao College, together with HIT (Shenzhen) and Huawei, used 1,000 Huawei Ascend 910C chips to successfully complete full-parameter post-training of the 1.6-trillion-parameter DeepSeek-V4-Pro large model.

This was no tentative attempt, but a milestone technological breakthrough. It proved with irrefutable engineering results that: domestic AI chips are fully capable of supporting world-class, super-large-parameter model training.

Why this matters​

The two thresholds of AI chips: "inference" and "training"​

• Inference: using an existing model to chat, write copy. Domestic chips could already do this
• Training: adjusting model parameters to learn new capabilities. Full-parameter training adjusts all 1.6 trillion parameters at once — maximum difficulty

Previously, full-parameter training of trillion-scale models was monopolized by NVIDIA H100/H200. Domestic chips could only do inference, not large-scale training.

The significance of this breakthrough: domestic compute leapt from "usable" to "useful", from "inference" to "training".

Technical details​

Training configuration​

Item	Parameter
Chips	Huawei Ascend 910C × 1,000
Model	DeepSeek-V4-Pro
Parameters	1.6 trillion (1600B)
Training type	Full-parameter post-training
Framework	MindSpore + torch_npu
Completed	Announced June 5, 2026

Performance metrics​

Metric	Value	Assessment
Compute utilization	>30%	Industrial grade (top overseas chips ~40%)
Key training operator efficiency	+14%	vs previous-gen 910B
Communication bandwidth utilization	>60% (est.)	MoE All-to-All communication
Stability	1,000 cards trained continuously with no failures	Cluster stability met standard

💡 About 30% compute utilization: many feel 30% is low, but in large-model training this is already a very respectable industrial-grade level. Even with the most advanced overseas chips, many teams' actual utilization is around 40%.

Ascend 910C detailed specs​

Ascend 910C is Huawei's AI training/inference chip announced at the Huawei Analyst Conference (April 24, 2024), with a theoretical peak of 800 TFLOPS (BF16), in the same class as NVIDIA H100.

Parameter	Ascend 910C	Ascend 910B	NVIDIA H100
Architecture	Ascend 910C	Ascend 910B	Hopper
Process	TSMC 7nm (est.)	TSMC 7nm	TSMC 4NP
BF16 compute	800 TFLOPS	256 TFLOPS	989 TFLOPS (sparse)
Memory	64GB HBM (est.)	64GB HBM2e (B1/B2)	80GB HBM3
Memory bandwidth	~2TB/s (est.)	600 GB/s (B1/B2)	3.35 TB/s
TDP	~400W (est.)	300-400W	700W
Mass production	April 2026 (full production)	Nov 2022	Mar 2022

Key upgrades:

• ✅ 3× compute: from 910B's 256 TFLOPS to 800 TFLOPS
• ✅ Mature software ecosystem: torch_npu adapts PyTorch, MindSpore framework mature
• ✅ Cluster stability: 1,000 cards trained continuously with no failures (the biggest breakthrough)

Technical challenges and solutions​

Challenge 1: Memory demand of trillion-scale models​

A 1.6-trillion-parameter model needs, just for model parameters:

• FP16: 1.6T × 2 bytes = 3.2 TB
• Plus gradients and optimizer states: at least 10 TB of memory

Huawei's solution:

• Model Parallel: distribute the model across 1,000 910C chips
• ZeRO optimizer: optimize memory footprint
• Gradient accumulation: update parameters in stages

Challenge 2: Communication efficiency of thousand-card clusters​

Training with 1,000 chips, inter-chip communication becomes the bottleneck. MoE models need All-to-All communication (each expert may need to communicate with all others).

Huawei's solution:

• HCCS (Huawei Collective Communication Scheduler): in-house high-speed interconnect protocol
• Layered communication: intra-node NVLink + inter-node HCCS
• Communication-compute overlap: data transfer concurrent with computation

Challenge 3: Training stability​

Trillion-scale model training takes weeks or months; any single card failure can interrupt the entire training.

Huawei's solution:

• Fault detection and auto-recovery: real-time monitoring of card status, auto-restart and recovery on failure
• Checkpoint optimization: high-frequency training-state saves (every N steps)
• Ascend cluster management software: designed specifically for enterprise training

Competitive comparison​

Vendor	Chip	1.6T-param training	Ecosystem maturity	Availability
Huawei	Ascend 910C	✅ Completed	⭐⭐⭐ (improving)	China-localized
NVIDIA	H100/H200	✅ Industry standard	⭐⭐⭐⭐⭐	Global (export-controlled)
AMD	MI300X	✅ Feasible	⭐⭐⭐⭐	Global
Google	TPU v5p/8t	✅ JAX-native	⭐⭐⭐⭐	Google Cloud

Conclusion: Ascend 910C has caught up to H100 in hardware performance, still lags in software ecosystem, but this training success proves engineering feasibility.

Industry impact​

1. The "Zunyi Conference" of domestic compute​

This breakthrough is called the "Zunyi Conference" of domestic compute — from passive defense to strategic counteroffensive.

Specific impact:

• ✅ Breaks the bias that "domestic chips can only do inference"
• ✅ Proves domestic chips can train frontier models
• ✅ Provides compute foundation for domestic large models (e.g., DeepSeek-V4, ERNIE 5.0)

2. Impact on NVIDIA​

Huawei Ascend 910C completing trillion-scale training means China's AI industry is less dependent on NVIDIA.

Scenario	Before	Now
Inference	Domestic chips usable	Domestic chips useful
Training	Must use H100/H200	Can use 910C
Large-scale training	Must use H100 clusters	Can use 910C clusters

3. Boost to the domestic chip industry​

This breakthrough will drive the entire domestic AI chip supply chain:

• Chip design: Cambricon, MetaX, Moore Threads accelerate iteration
• Wafer manufacturing: SMIC, Hua Hong get more orders
• Packaging/test: JCET, TFME benefit

Huawei Ascend roadmap (2025-2028)​

Time	Chip	Positioning
Q1 2025	Ascend 910C	Flagship training/inference (mass-produced)
Q1 2026	Ascend 950PR	Inference-optimized (~500 TFLOPS BF16)
Q4 2026	Ascend 950DT	Data-center training
Q4 2027	Ascend 960	Next-gen flagship
Q4 2028	Ascend 970	Next-next-gen

Training lessons shared​

The Shenzhen Hetao College team accumulated valuable experience:

✅ Successes​

1. Progressive training: start from small models (7B), gradually scale to 1.6T
2. Mixed-precision training: BF16 main + FP32 gradient accumulation
3. Communication optimization: All-to-All overlap with computation
4. Fault recovery: save checkpoint every 1,000 steps

⚠️ Challenges encountered​

1. Memory fragmentation: severe fragmentation over long training, needs periodic cleanup
2. Communication bottleneck: MoE All-to-All takes 30%+ of training time
3. Software bugs: torch_npu occasional memory leak, needs training process restart

Related chips​

• Huawei Ascend 910C - the chip used in this training
• Huawei Ascend 910B - previous-gen chip
• Huawei Ascend 950PR - next-gen inference chip
• NVIDIA H100 - benchmark product
• NVIDIA H200 - benchmark product (HBM3e upgrade)

References​

• Sina Finance: Huawei Ascend 910C completes DS-V4-Pro training
• Tencent News: Milestone! Ascend 910C completes 1.6-trillion-param model training
• 21IC: Huawei chip completes full-parameter post-training of 1.6T AI model

---

This article is compiled from public reports. Salute to the teams at Shenzhen Hetao College, HIT (Shenzhen), and Huawei — you proved the feasibility of China's AI compute with engineering results.