18 posts tagged with "Inference Chip"

当整个行业在"堆算力"上内卷时，SambaNova 选择回答另一个问题：推理效率取决于喂饱计算单元的能力，而不只是计算单元本身。

RDU：可重构数据流架构​

SN40L 的核心是可重构数据流单元（Reconfigurable Dataflow Unit，RDU）。它没有采用传统的"CPU 式取指执行"或"GPU 式 SIMT"，而是把计算组织成可编程的数据流图：每个 RDU 插槽包含 1040 个模式计算单元（PCU）与 1040 个模式内存单元（PMU），通过可编程的数据流网络互连。

芯片采用台积电 5nm 工艺与 2.5D CoWoS 封装，集成两个 RDU 计算裸片，单插槽峰值算力为 638 TFLOPS（BF16），片上聚合内存带宽达数百 TB/s 量级。这种架构最大的价值在于算子融合：编译器可以把数百个复杂操作合并为一次内核调用，把中间结果留在片上。以论文中给出的 Monarch FFT 为例，未融合时算术强度为每字节约 39.5 次操作，完全空间融合后提升至约 410 次——外部访存需求大幅下降。

三层内存：SRAM、HBM、DDR 各司其职​

SN40L 最具辨识度的设计是三层内存体系：

内存层级	容量（每芯片）	角色
片上 SRAM（PMU）	520 MB	最高带宽，紧贴计算单元
片上 HBM（封装内）	64 GB	存放正在运行的模型与热点数据
可插拔 DDR DRAM	最高 1.5 TB	存放更多模型与检查点

这个设计的巧思在于容量与成本的平衡：HBM 贵而小，DDR 便宜而大。模型从 DDR 加载到 HBM 的速度超过 1 TB/s，因此一套 16 芯片的 SambaRack 拥有 24TB DDR，可以在单系统中驻留 DeepSeek-R1 671B 这样的大模型，也能同时服务大量小模型。SambaNova 披露，在混合专家推理部署中，SN40L 相比 DGX H100 端到端提速 3.7 倍，模型切换时间提升 15 至 31 倍，机房占地最多缩减 19 倍；整rack采用风冷，功耗约 10kW，远低于晶圆级方案的 23kW 水冷系统。

企业推理的差异化路线​

SambaNova 与 Cerebras、Groq 的公开对比揭示了路线差异：Cerebras 擅长单用户极致 token 速度，但 SambaNova 指出其每片 LPU 仅约 230MB SRAM、无片外内存，服务 70B 模型需要数百颗芯片；而 SN40L 用 16 颗芯片即可完成同类部署。换句话说，SambaNova 押注的是"多用户、多模型、企业级"的推理经济学，而非单点跑分。

据公开报道，2026 年 2 月 SambaNova 发布了下一代 SN50 芯片，宣称面向智能体 AI 工作负载提供 3.2 PFLOPS FP8 算力，三层内存可支持高达 10 万亿参数模型，软银成为首个客户，公司同期完成超过 3.5 亿美元融资，并宣布与 Intel 展开多年期软硬件协同合作。

小结与建议​

SN40L 的启示在于：推理时代的竞争正在从峰值算力转向"单位 token 的系统经济学"。如果你的业务是多租户、多模型的企业 AI 平台，或需要频繁切换模型、在私有环境部署超大模型，三层内存架构提供了 GPU 集群难以复制的容量弹性；如果追求单用户最低延迟，晶圆级与 LPU 路线可能更合适。选型的关键，是先弄清自己的负载形状。

The Information 报道 NVIDIA 计划投资推理芯片对手 d-Matrix；同日，NVIDIA 参投的 Upscale AI 发布跨厂商互联平台 Token Fabric。从收购 Groq 到开放 NVLink Fusion，霸主的应对方式正在从对抗转向"收编"。

NVIDIA 计划投资 d-Matrix​

据 The Information 10 月 8 日报道，NVIDIA 计划投资 d-Matrix——一家 AI 推理芯片初创公司，同时也是 NVIDIA 产品线的直接竞争对手。交易条款和金额未披露。

值得注意的不是交易本身，而是报道给出的框架：d-Matrix 等"挑战者"正选择与 NVIDIA 合作，而非独立竞争。一家推理芯片公司拿到行业霸主的投资，这本身就是对"独立挑战"路线的一次重新定价——市场开始默认，与霸主同行可能比与其对抗更有前途。

Token Fabric：连接竞争对手的互联平台​

同日（10 月 8 日），NVIDIA 参投的 Upscale AI 发布 Token Fabric 平台。它的设计目标很特别：在同一个数据中心内连接多家竞争供应商的 AI 芯片，解决不同厂商芯片生态互不兼容所造成的网络孤岛问题。

在异构算力混布逐渐成为常态的背景下，多厂商芯片组网确实是真实痛点——数据中心里同时存在多家供应商的加速器时，网络层的不兼容会显著抬高部署成本。但一个由 NVIDIA 参投背书的多厂商互联标准，其"中立性"天然存疑：治理权在谁手里，标准就会向谁倾斜。

背景对照：从收购到开放的组合拳​

把近期动作放在一起看，NVIDIA 对推理层挑战者的策略是一条清晰的轨迹：

动作	内容	作用
收购 Groq	此前完成收购	直接消除一个推理挑战者并吸收其技术
开放 NVLink Fusion	允许伙伴集成自定义 CPU/加速器	把第三方设计拉进 NVIDIA 互联生态
投资 d-Matrix	推理芯片直接对手，条款和金额未披露	用资本绑定潜在对手
Token Fabric	参投 Upscale AI 发布，连接多家竞争供应商芯片	在互联标准层保持话语权

Inference 层是当前竞争最激烈的新战场：推理需求随应用爆发持续放量，而推理芯片的技术门槛与训练芯片不同，给了初创公司更多切入空间。NVIDIA 的回应是全套组合拳——能收购的收购，能投资的投资，能开放的开放，把每一个可能的威胁转化为生态的一部分。

市场涟漪：算力资产金融化​

同期两条市场消息进一步扩大了观察视野：

• 亚马逊拟向投资者转让约 80 亿美元的 NVIDIA 芯片，采用售后回租模式以缓解现金流压力；
• SpaceX 拟融资 400 亿美元采购英伟达芯片，由 Apollo 牵头，结构约为 100 亿美元银行贷款加 300 亿美元投资级债务。

AI 算力正在被金融工具深度改造：芯片从单纯的固定资产，变成可转让、可抵押、可回租的资产类别。售后回租和芯片抵押融资的出现，既反映了算力需求之热，也提示了部分买家的资金链已经绷得很紧。

分析：竞合逻辑的三层解读​

**对 NVIDIA：收编优于对抗。**面对推理挑战者，正面对抗意味着价格战和生态战，成本高且结果不确定；用资本和生态把挑战者拉进自己的轨道，既消解威胁又吸收创新。NVLink Fusion 开放、投资 d-Matrix、支持 Token Fabric，本质上是同一战略的不同侧面。

**对 Token Fabric：中立性是最大考题。**一个由 NVIDIA 参投的公司推出的"多厂商互联标准"，其他厂商接受它的前提是治理结构足够中立。如果标准最终被视作 NVIDIA 生态的延展工具，其多厂商价值将大打折扣，AMD、博通等厂商是否愿意接入将成为试金石。

**对推理芯片创业公司：独立生存空间收窄。**d-Matrix 选择与 NVIDIA 合作，说明独立路线的融资与商业化难度在上升。"借船出海"可以换来资本与生态的入场券，代价是战略自主性。对整个推理芯片赛道而言，站队正在加速，留给中立玩家的窗口期越来越短。

小结​

投资对手 d-Matrix、参投 Token Fabric、收购 Groq、开放 NVLink Fusion——NVIDIA 正在用资本与生态把推理层的挑战者逐个纳入轨道，"竞合"取代正面对抗成为新打法。与此同时，亚马逊 80 亿美元售后回租、SpaceX 400 亿美元债务融资采购芯片，显示算力资产金融化正在成为行业新变量。推理芯片这一新战场上的博弈，正从纯技术竞争转向资本与标准的综合较量。

Hot Chips 2026 上，OpenAI 与博通合作的首颗物理芯片 Jalapeño 完整规格曝光：700W 纯推理 ASIC、216GB HBM4、13.4 PFLOPS MXFP4，从初始设计到流片只用了约 9 个月。

Jalapeño 是什么​

综合 Data Center Dynamics、Tom's Hardware、ServeTheHome 在 Hot Chips 2026 上的报道，Jalapeño 是 OpenAI 与博通合作的首颗物理芯片，定位非常纯粹：一颗纯推理 ASIC，功耗 700W，专门针对 OpenAI 大规模推理服务（如 ChatGPT 等）的实际瓶颈设计。

与通用 GPU 不同，Jalapeño 不追求覆盖训练、微调、推理的全场景，而是把全部资源押在一件事上：以最低成本、最高密度跑好 OpenAI 自己的推理负载。这是自研芯片路线中最激进的一种——不做通用产品，只为自家业务画像画芯片。

完整规格​

项目	规格
定位	纯推理 ASIC（OpenAI × 博通首颗物理芯片）
功耗	700W
内存	216GB HBM4
算力（MXFP4）	13.4 PFLOPS
算力（MXFP8）	3.4 PFLOPS
架构	NUMA 式架构，64 个内存与计算切片
流片速度	从初始设计到流片约 9 个月
工艺	博通 3nm/2nm 混合设计，台积电代工
首批部署	目标 2026 年底

架构上，Jalapeño 采用 NUMA 式设计，共 64 个内存与计算切片。这种把内存与计算绑定为切片、再横向扩展的思路，本质上是围绕 OpenAI 自己的模型结构和流量特征做定制——博通在定制 ASIC 上最擅长的，正是这种"按客户负载画芯片"的方法论。

集群规模：单 pod 最高 27 EFLOPS​

单颗芯片只是起点，Jalapeño 的部署规划更能说明 OpenAI 的推理盘子有多大：

• 128 颗芯片组成的部署规模约 1.7 EFLOPS FP4，合计 27.5TB HBM4；
• 单个 pod 最多容纳 2048 颗加速器，最高 27 EFLOPS MXFP4。

作为背景，OpenAI 与博通的定制算力协议总规模约为 10GW，Jalapeño 只是这盘棋的第一颗子。如果 2026 年底首批部署如期落地，OpenAI 的推理成本结构将出现一个全新的变量。

背景与资金面​

围绕这颗芯片的产业与资金背景同样值得梳理：

• OpenAI 与博通签有约 10GW 算力的定制协议，芯片采用博通 3nm/2nm 混合设计、台积电代工，目标 2026 年底首批部署；
• 博通正为 OpenAI 定制芯片筹资超 500 亿美元，其中早期磋商约 300 亿美元为债务融资，接续此前为 Anthropic 安排的债务融资；
• FT 报道 OpenAI 年化营收约 500 亿美元，低于此前约 700 亿美元的口径；
• 与此同时，Meta 也在与博通合作开发下一代 MTIA 芯片。

定制芯片是资金密集型游戏：芯片要钱，HBM 要钱，数据中心还要钱。博通为 OpenAI 安排的数百亿美元债务融资，说明这类项目的资金结构已经接近基础设施建设，而非传统的芯片采购。

分析：专用化的赌注​

**纯推理专用 vs GPU 通用性。**已有分析师对 Jalapeño 的路线提出质疑：纯推理 ASIC 缺乏灵活性，一旦模型结构或推理范式发生变化，芯片可能无法跟上。OpenAI 的赌注是：自家推理需求足够大、足够可预测，专用化带来的成本与密度收益足以覆盖灵活性损失。这是一个用业务确定性换取硬件效率的交易——赌对了，推理成本大幅下降；赌错了，芯片就是沉没成本。

**9 个月流片的工程方法论。**reticle 级定制 ASIC 通常需要数年才能流片，Jalapeño 从初始设计到流片约 9 个月，是异常速度。这背后是三重支撑：博通成熟的 IP 库与设计流程、台积电先进工艺的稳定保障，以及 OpenAI 作为单一客户时的高度确定性需求——需求越明确，设计迭代越少，周期越短。

**博通成为定制 ASIC 时代最大赢家。**OpenAI 的 Jalapeño、Meta 的下一代 MTIA、Google 的 TPU，再加上为 Anthropic 安排的债务融资——几乎所有超大规模玩家的定制芯片路线，最终都汇入博通的订单簿。GPU 时代的赢家是英伟达，定制 ASIC 时代的赢家，很可能是那个卖铲子的博通。

小结​

Jalapeño 的完整规格披露，让外界第一次看清 OpenAI 自研芯片的成色：700W、216GB HBM4、13.4 PFLOPS MXFP4，9 个月流片，单 pod 最高 27 EFLOPS MXFP4。它不追求通用，只追求把 OpenAI 自己的推理账单打下来。这颗芯片能否成功，取决于推理需求是否如 OpenAI 预期的那样持续且可预测；而无论成败，博通都已经稳稳站在了赢家的位置上。

据IEEE Spectrum 2026年9月14日报道，OpenAI的Jalapeño AI加速器正逐步揭开面纱。这不仅是一颗芯片的诞生，更是一场芯片设计方法论的革命——由LLM深度参与设计、不足百人团队、从架构概念到首次流片不足20个月。

Jalapeño的核心规格​

根据OpenAI官方口径，Jalapeño AI加速器的关键规格如下：

规格项	Jalapeño参数
峰值算力	最高13.4 petaflops（4-bit精度）
内存容量	232GB
内存带宽	15.4TB/s
设计定位	专为LLM推理设计
合作伙伴	博通（Broadcom）
研发周期	从首个架构概念到首次流片不足20个月
团队规模	平均不足100人

其中最引人注目的是4-bit精度下13.4 petaflops的峰值算力。这一数字定位清晰：Jalapeño并非追求通用计算的"全能选手"，而是聚焦于大语言模型推理这一特定场景，在低精度计算上做到极致。232GB内存与15.4TB/s带宽的组合，则为KV Cache密集型的长上下文推理提供了吞吐保障。

性能对标：较GB300最高降低3.6倍延迟​

OpenAI引用的基准测试显示，在端到端延迟方面，Jalapeño较NVIDIA GB300最高降低3.6倍。对于推理业务而言，延迟直接决定用户体验——聊天机器人的首字响应、Agent系统的多轮迭代速度，都取决于每一毫秒。

需要指出的是，这一数据来自OpenAI自己的引用口径，测试条件与模型负载尚未完全公开。但即便打上折扣，一颗自研ASIC在特定推理负载上跑赢NVIDIA旗舰级GB300平台，本身就说明定制芯片在"场景足够聚焦"时确实具备结构性优势。

与博通合作：多代规划与吉瓦级部署​

Jalapeño由OpenAI与博通合作设计，规划了多代产品路线，目标直指吉瓦（GW）级部署。据公开报道，该芯片据称搭配AMD EPYC Turin主机CPU，而非完全依赖NVIDIA系统——这意味着OpenAI正在构建一条绕开NVIDIA生态的完整推理基础设施。

博通作为全球定制ASIC（XPUs）设计服务的头部玩家，此前已深度参与多家云巨头的自研芯片项目。OpenAI选择博通，等于站在了成熟的多项目晶圆与SerDes/IP积累之上，这也是20个月流片速度的重要支撑。

自定义芯片的适用条件​

Jalapeño的故事之所以成立，背后有一组苛刻的前提条件，并非所有公司都能复制：

1. 海量且可预测的推理需求。ChatGPT的token消耗量巨大且增长曲线稳定，只有这种量级才能摊薄流片与迭代成本。
2. 稳定的模型架构。Transformer架构多年未发生根本变化，才让针对特定计算模式的ASIC有了"押注"的价值。
3. 全栈软件掌控。OpenAI同时拥有模型、推理框架和基础设施，软硬件协同优化可以贯穿到底。
4. 足够的部署量。吉瓦级部署规划意味着单位芯片成本可以压到极低。

四个条件缺一不可。缺了需求，流片成本摊不掉；缺了架构稳定，芯片投产即过时；缺了软件栈，性能优势发挥不出来。

定制ASIC不会取代GPU​

一个常见的误读是"自研芯片将终结GPU"。但行业现实是：定制ASIC与GPU长期共存、各司其职。

• 训练仍是GPU的主场。训练负载多变、需要频繁调试与实验，通用性至关重要。
• 多模态与快速演进的架构同样属于GPU。新算子、新注意力变体往往先在GPU上验证。
• 推理中负载高度稳定、模型固化的部分，才是ASIC的收割区。

换句话说，NVIDIA的GPU生态不会因为Jalapeño而瓦解，但推理市场的价值分配确实开始被重新切分。对NVIDIA而言，最大的挑战或许不是失去训练市场，而是越来越多头部客户把"最大的一块推理蛋糕"留给自己。

LLM设计芯片：方法论的革命​

比规格更值得关注的，是Jalapeño的设计方式。据IEEE Spectrum报道，OpenAI的LLM深度参与了芯片设计——从首个架构概念到首次流片不足20个月，团队平均不足100人。传统芯片设计动辄数百人团队、三到五年的开发周期，Jalapeño把这个数字压缩到了令人难以置信的程度。

LLM辅助设计带来的是研发效率的量级变化：架构探索空间的快速评估、RTL代码生成与验证、设计文档与工具链自动化。如果这套方法论被验证可复制，那么芯片设计的门槛将被实质性改写——未来不仅是"大厂造芯"，而是"AI造芯"成为常态。

小结​

Jalapeño是2026年定制ASIC浪潮中最具标志性的产品之一：13.4 PFLOPS 4-bit算力、232GB内存、15.4TB/s带宽的规格专为LLM推理而生，与博通的合作、多代规划与吉瓦级部署展现了OpenAI摆脱NVIDIA依赖的决心。而LLM参与设计、不足百人团队、20个月流片的研发模式，其影响或许远超这颗芯片本身。定制ASIC不会取代GPU，但它正在重新定义推理经济的成本结构——对整个AI算力产业而言，这是一场刚刚开始的游戏。

The Tuesday-afternoon AI session at Hot Chips 2026 this August was the most historically significant of the conference — not because any single chip was so powerful, but because almost everything on stage was a "hyperscaler de-NVIDIA-ization" custom ASIC: Google's 8th-gen TPU, OpenAI's first self-designed chip, Microsoft Maia, Meta MTIA, and Cerebras wafer-scale racks, all on one stage. When the world's largest AI compute buyers start treating GPUs as "one of the options," the power structure of AI hardware is loosening.

---

1. OpenAI Jalapeno: Building a Chip in 9 Months​

On June 24, 2026, OpenAI, together with Broadcom, unveiled its first self-designed inference ASIC, Jalapeno — the fifth member of the "custom inference chip club."

Dimension	Jalapeno
Partner	Broadcom + TSMC manufacturing
Positioning	Inference-specific ASIC
Design cycle	9 months end-to-end (Greg Brockman says aided by OpenAI's own models)
Cost target	~50% lower token cost vs general-purpose GPU stack
Commercial timing	First deployments by end-2026; long-term goal 10GW of self-designed chips
Deal scale	Up to $10B strategic partnership with Broadcom (accelerators + networking by 2029)

The talk title "You Can Just Build Things … Chips" is itself a signal: the largest AI compute buyer no longer defaults to GPU as the only path.

---

2. Google TPU v8: The Biggest Architectural Pivot in a Decade — Train/Infer Split​

Google has the longest custom-chip history (2016 to now), and its 8th-gen TPU for the first time splits the product line in two:

Model	Codename	Partner	Positioning	Key Specs
TPU 8t	Sunfish	Broadcom	Training	9,600 cards per pod, 121 FP4 ExaFLOPS, 2PB shared HBM, 2× ICI bandwidth
TPU 8i	Zebrafish	MediaTek	Inference	288GB HBM, 384MB on-chip SRAM (3× prior gen), 19.2 Tb/s ICI

On capacity, Morgan Stanley estimates based on supply-chain interviews that Google TPU production in 2026 may exceed 3 million units (a brokerage estimate, not an official target). Google is also the only vendor to achieve large-scale custom-chip deployment and sell compute externally (Gemini runs on TPUs).

---

3. Meta MTIA: From Recommendation Systems to a GenAI Dual Mission​

Meta's custom journey has the clearest starting point — MTIA was originally built for recommendation ranking hardware and is being pulled toward a dual mission by generative AI.

• MTIA 300 is deployed; 400 / 450 / 500 are planned at roughly one new model every 6 months through 2027;
• Based on RISC-V, Meta claims up to 25× compute gain;
• Node evolves with industry cadence: 100 (7nm) → 200 (5nm) → 300 series (3nm + CoWoS);
• In partnership with Broadcom; another chip codenamed Iris reportedly passed testing in July 2026;
• Meta plans to start volume production of one of them in September 2026, doubling its overall compute.

---

4. Microsoft Maia 200/300: Most Advanced Deployment​

Microsoft's Maia 200, released January 26, 2026, is the most advanced in deployment among the four:

Dimension	Maia 200
Process	TSMC 3nm, 140B+ transistors
Compute	10+ PFLOPS FP4 / 5 PFLOPS FP8
Memory	216GB HBM3E, 7 TB/s
Power	750W
Deployment	Already running in Des Moines data center, serving OpenAI GPT-5.2 and Microsoft 365 Copilot

Microsoft claims roughly 3× the performance of Amazon's Trainium on specific benchmarks. The short-term strategy is a dual track of "self-designed Maia + purchased NVIDIA" in parallel — self-designed chips need time from design to mass production, and NVIDIA's mature ecosystem cannot be replaced in the short term.

---

5. Amazon Trainium 3 and Anthropic's In-House Team​

• Amazon: The Trainium series is already commercial, with 1.4 million units cumulatively deployed (officially disclosed) — a multi-billion-dollar business; its strength is the AWS customer base, letting enterprises choose between NVIDIA GPUs and self-designed chips. Trainium 3 continues this path.
• Anthropic: In August 2026 announced the formation of an in-house chip team, with no tape-out or mass-production timeline yet; initially positioned as a complement (not a replacement) to existing partnerships with NVIDIA/AMD/AWS/Google Cloud, aiming to tailor-build for the Claude architecture and shed reliance on a single GPU.

---

6. NVIDIA's Answer: Not a Faster GPU, But Full-Stack​

It's easy to simplify the narrative to "four companies build chips, NVIDIA defends GPU." But NVIDIA took 6 slots at Hot Chips: a RISC-V tutorial, the Vera CPU, the Rubin GPU, the BlueField-4 DPU, the Spectrum-X multi-plane network, and an LPU accelerator.

A hyperscaler ASIC replaces only one of those five pillars. If the CPU, NIC, switching fabric, and software all come from the same vendor, what you save by swapping out the accelerator is far less than the accelerator line item on the bill suggests. Rubin's play is a full-stack AI factory platform spanning seven chips and five racks — the competitive answer is "full-stack positioning," not "a faster single chip."

---

7. Trend Judgment: Inference De-GPU-izes, Training Still GPU-Led​

• Inference side: The CUDA moat visibly shallows. Inference is parallelizable and replaceable at the endpoint; custom ASICs trade away the generality tax (implementing only the operations LLMs actually execute) for lower cost/token. Groq LPU, Cerebras, and various TPU/ASIC players all compete on the same metric.
• Training side: Foundation models are still trained on GPUs, with no serious challenger in the short term. NVIDIA's three training moats (fastest silicon + NVLink + CUDA) remain firm.
• Conclusion: Custom chips are not "replacing NVIDIA," but giving buyers a credible external negotiation option in the largest and fastest-growing battlefield — inference. That alone is enough to reshape the economics of AI infrastructure.

Related Links​

• Inference Accelerator Market 2026
• Hot Chips 2026 Full Recap
• Google TPU 8t Spec Page
• Google TPU 8i Spec Page

References​

• Hot Chips 2026: 4 Hyperscalers Put Custom AI Silicon on One Afternoon - Sean Kim
• Custom AI inference silicon: why OpenAI built Jalapeno - nexi.fund
• Anthropic Builds Custom AI Chips for Claude - Studio Global AI
• 所有大厂都在做同一件事：自己造芯片，少买英伟达的 - Toutiao

---

This article is compiled from August 2026 Hot Chips on-site reports, corporate announcements, and industry analysis. Some capacity and performance figures are brokerage estimates or vendor-disclosed figures; actual results are subject to mass-produced products.

For the past three years, the entire AI hardware story was "training": who had the most H100s, who could connect a hundred thousand GPUs into a cluster. That race is essentially settled — NVIDIA won. But the next battlefield, "inference," is being fought under completely different rules: the measure is no longer peak FLOPS, but cost-per-token, latency, and power. In 2026, inference chips overtake training in scale for the first time, becoming the main battlefield of AI accelerators.

---

1. Inference Becomes the Main Battlefield: 80%–90% of Compute Spent on Inference​

Training a large model costs hundreds of millions of dollars — once. But once the model goes live, it must answer billions of queries day after day. A popular consumer model may need tens of thousands of accelerators running 7×24 to keep up with demand. Therefore:

• Inference accounts for roughly 80%–90% of a model's lifecycle compute;
• Inference chips will make up about 60%–70% of the ~$400B AI accelerator market in 2026, up from only ~40% in 2023;
• Inference chip growth (estimated +52.7% YoY) significantly outpaces training chips (+28.4%); the share of inference-side compute demand exceeded training-side for the first time in 2026, reaching 54% (~$1010B).

The economics of inference are straightforward: training cost is amortized to near-zero, while inference cost becomes the entire bill. Every 1% cut in inference cost flows directly to profit — for a company whose inference traffic reaches hyperscale like OpenAI, the half of the bill is a number followed by a string of zeros.

---

2. Market Size: Structural Growth Inflection Point Has Arrived​

Market	2026 Size	Growth	Notes
Global dedicated inference chips	$412.7B	+38.4%	14.2 pct higher growth than training chips
China dedicated inference chips	$118.6B (28.7% of global)	+44.1%	Strongest single market in APAC by growth
Global AI training/inference chips (incl. GPU/NPU)	exceeds $1850B	+40.2%	GPU ~62%

China's domestic substitution is accelerating, with domestic inference chips reaching 34.6% of shipments, up 9.8 pct from 2025.

---

3. Technology-Axis Share Inverts: GPU Slows, ASIC Soars​

Axis	2026 Shipment Share	Trend
GPU	52.6%	Still leads, but growth slows to 22.7%
ASIC custom chips	41.3%	Up sharply from 17.8% in 2022
FPGA	Stable	Specific low-latency scenarios

Thanks to ecosystem maturity, GPU remains the mainstay, but NPU/ASIC already holds a 1.8× advantage over same-generation GPUs in energy efficiency, driving rapid adoption at the edge and on-device. Shipments of inference-optimized ASICs are expected to reach 11.5 million units, with unit cost about 35% lower than GPUs.

---

4. Five Schools Clash​

School	Representative Products	Core Strength	Use Cases
General-purpose GPU	NVIDIA Rubin / B200 / H200	Mature ecosystem, train+infer unified	Frontier training + highly interactive inference
LPU (Language Processing Unit)	Groq LPU	Ultra-low latency, deterministic throughput	Real-time dialogue, high-concurrency inference
TPU (inference-specific)	Google TPU 8i (Zebrafish)	288GB HBM, 384MB on-chip SRAM, 19.2 Tb/s ICI	Google's scaled inference
Custom ASIC	OpenAI Jalapeno, Microsoft Maia 200, Meta MTIA	Strip generality tax for own models, ~50% lower cost/token	Hyperscaler's own workloads
Air-cooled inference card	Intel Crescent Island	350W air-cooled, 480GB LPDDR5X, tokens/watt	Cost-sensitive mid/long-tail inference

OpenAI's Jalapeno, co-developed with Broadcom, aims to cut inference token cost by roughly 50% versus a general-purpose GPU stack — the fifth member to join the "custom inference chip club" (after Google TPU, Amazon Inferentia/Trainium, Microsoft Maia, and Meta MTIA).

---

5. Core Metric Shifts: cost-per-token and tokens/watt​

The fundamental difference between the inference race and the training race is the low switching cost:

• Training requires a 100k-GPU cluster + NVLink + CUDA, with extremely high migration cost;
• Inference is "embarrassingly parallel" at the endpoint level — no million-GPU cluster needed; a node that produces tokens fast and cheaply suffices, and is replaceable per endpoint.

This means NVIDIA's three moats (fastest silicon, NVLink scale-out, CUDA) are no longer absolute on the inference side. When the largest AI buyer (OpenAI) starts treating GPUs as "one of the options," the GPU premium begins to erode — pricing power relies on scarcity, and custom chips attack that scarcity from two directions at once: both reducing merchant-chip demand and giving buyers a credible external negotiation option.

---

6. Edge and On-Device Explosion: Long-Tail Signal​

Demand shows significant long-tail and fragmentation:

Scenario	2026 Demand Size	Growth
Cloud inference	$198.2B (48%)	+24.5% (slowing)
Edge inference	$126.5B (30.7%)	+52.3%
On-device inference	$88.0B (21.3%)	+68.9%
Autonomous-driving inference	$67.3B	+58.2%
Industrial QA / robotics inference	$42.1B	+63.7%

The latency sensitivity and power constraints of inference workloads are reshaping chip architecture design priorities — which also explains why "air-cooled, large-memory" solutions like Crescent Island can find a niche.

Related Links​

• Hot Chips 2026 Full Recap
• Hyperscaler Custom Silicon Wave 2026
• NVIDIA Rubin Spec Page
• Intel Crescent Island Spec Page

References​

• Inference Chips vs Training Chips - ValueAdd VC
• Global and China Dedicated Inference Chip Industry In-Depth Research Report (2026) - IIM
• 2026 Global AI Training/Inference Chip Market and Trend Report - IIM
• Custom AI inference silicon: why OpenAI built Jalapeno - nexi.fund

---

This article is compiled from publicly available 2026 market research, brokerage reports, and industry analysis. Market sizes and shares are third-party estimates with inconsistent methodologies and are for reference only.

In the first half of 2026, the AI chip industry underwent a historic turning point — the center of gravity shifted from the "training race" to "inference efficiency," domestic chip market share broke 40% for the first time, NVIDIA built higher barriers with Blackwell Ultra, and the inference-specific chip track bloomed in diversity.

---

I. Compute Doubles Again: NVIDIA Blackwell Ultra Launch (June 1)​

On June 1, 2026, NVIDIA CEO Jensen Huang unveiled the new-generation AI chip Blackwell Ultra at Computex 2026 (Taipei), setting a new starting line for the AI infrastructure race over the next two years.

Key Specs​

Metric	Blackwell Ultra	B200	Improvement
FP8 compute	20 petaFLOPS	~10 petaFLOPS	100%
Architecture	Blackwell Ultra	Blackwell	Upgrade
Expected delivery	2027 Q1	2026 Q1	—
Positioning	Hyperscale training + inference	Training + inference	Flagship

Industry Significance​

1. Direct impact of doubled compute: 20 petaFLOPS FP8 means training time for hundred-billion-parameter models drops sharply; trillion-parameter model training moves from "scientific experiment" to "engineering routine"
2. System-level balance: Blackwell Ultra is not just a chip but a system-level engineering breakthrough across NVLink, HBM, cooling, and power delivery
3. Roadmap certainty: The Q1 2027 delivery timeline lets cloud vendors and AI labs plan infrastructure budgets 18 months ahead

Challenges​

• Energy crisis: Doubled performance comes with sharply higher power; datacenter power and cooling design face extreme challenges
• Accessibility: Top-tier compute goes first to top cloud vendors; how smaller developers and research institutes reach compute at reasonable cost via cloud services
• Software stack adaptation: New hardware needs matching CUDA versions and framework support; software ecosystem maturity becomes the key bottleneck for compute conversion

---

II. Domestic AI Chips: The Tipping Point from "Usable" to "Good"​

On June 16, 2026, Xinchuang World published "2026 China Domestic AI Chip Vendor Capability Quadrant", clearly outlining the overall domestic landscape.

2.1 Capability Quadrant Ranking​

Quadrant	Representative Vendors
Leader quadrant	Huawei Ascend, Hygon, Cambricon, Alibaba T-Head, Moore Threads
Visionary quadrant	Baidu Kunlunxin, Biren, Enflame, Iluvatar, HardyVision
Contender quadrant	TSINGMICRO, Black Sesame, SemiDrive, Lisuan, Houmo
Challenger quadrant	Denglin, Zhicun, VeriSilicon, Rockchip, Intellifusion

2.2 Huawei Ascend: The Anchor of Domestic Compute​

Market Position​

• In 2025, Ascend series shipped 812,000 units, capturing 49% of the domestic AI accelerator card share, firmly No.1 domestically
• Ascend 950PR single-card FP8 compute reaches 1P (PetaFLOPS), FP4 compute reaches 2P
• Inference performance is about 2.87x that of NVIDIA H20, priced at only 72,000-75,000 RMB, a significant price/performance advantage

Full-Stack Advantage​

Huawei's "device-network-cloud-chip" integrated strategy is Ascend's core moat:

• Chip design: Da Vinci 3.0 architecture iterating continuously
• OS: HarmonyOS/Euler OS deeply optimized
• Networking: Euler network protocol stack
• Cloud: Huawei Cloud ModelArts platform seamlessly integrated

Latest Progress​

• On June 5, 2026, Shenzhen Hetao College, together with HIT (Shenzhen) and Huawei, completed full-parameter post-training of a 1.6-trillion-parameter DeepSeek V4 Pro model on an Ascend 910C cluster
• This is the first time domestic AI chips completed trillion-parameter-level model training, marking "domestic substitution" moving from inference to training

2.3 Cambricon: The First Profitable Domestic AI Chip Benchmark​

Performance Explosion​

Metric	Full-year 2025	2026 Q1	YoY Growth
Revenue	6.497B RMB	2.885B RMB	+453% / +160%
Net profit	2.059B RMB (first annual profit)	1.013B RMB	— / +185%

Core Product: Siyuan 590​

• In DeepSeek R1 inference scenarios, TPS reaches 942, about 50% higher than H20
• Years of joint optimization with ByteDance; strongest short-term cloud inference deployment capability
• Of 2.885B RMB Q1 2026 revenue, Siyuan 590 contributed over 70%

Potential Risks​

Absent from the 2nd 2026 "Safe and Reliable Evaluation Results Announcement"; the reason is unclear and will affect its domestic government/enterprise market performance.

2.4 TSINGMICRO: The "Third Route" of Reconfigurable Chips​

Technical Route​

TSINGMICRO adopts a reconfigurable dataflow architecture同源 with Groq LPU, finding a balance between GPU generality and ASIC extreme efficiency.

Metric	TSINGMICRO TX81	Traditional GPU	Advantage
Inference cost	Baseline	+100%	Reduced 50%
Energy efficiency	Baseline	Baseline	3x improvement
Architecture	Reconfigurable dataflow	SIMT/SIMD	Better for inference

Deployment Progress​

• Cumulative shipments of reconfigurable chips exceed 30 million units
• Scaled deployment in a dozen-plus thousand-card-scale intelligent computing centers nationwide
• Has begun A-share IPO tutoring; likely to become the "first reconfigurable chip stock"

---

III. The Inference Chip Track: Core Signal of the Industry Shift​

On June 4, 2026, TrendForce published a deep report "The Era of Inference Economy: The Rules of AI Chips Are Being Rewritten," pointing out that the compute competition center of gravity is shifting from training to inference.

3.1 Why Now?​

Cost Structure Changed​

• Training is a one-time cost: Once a model is trained, marginal cost approaches zero
• Inference is a recurring cost: Every API call, every generated token represents compute consumption and gross-margin pressure
• Per-unit inference cost and energy efficiency directly affect gross margin and scale-expansion capability

Model Compression Tech Matured​

• 1.58-bit quantization and weight pruning let models maintain inference accuracy at extremely low memory footprint
• MoE (Mixture of Experts) architecture activates only a few expert sub-networks per inference via "partial wake-up," greatly reducing actual computation
• The rise of slimmed models provides commercial viability for hard-wired inference chips

3.2 NVIDIA's $20B Bet: Acquiring Groq (December 2025)​

On December 24, 2025, NVIDIA acquired Groq's inference technology license and core team for $20 billion, one of NVIDIA's largest M&A/tech acquisitions ever.

Strategic intent:

1. Fill the inference gap: NVIDIA GPU is unshakable in training, but inference efficiency was never its strongest suit
2. Counter specialized inference chips: Cerebras, Taalas, SambaNova and other startups are eroding the inference market
3. Position for Agentic AI: Agentic AI needs extremely low-latency, high-throughput inference

3.3 Taalas HC1: Proof of Concept for Hard-Wired Inference​

On February 20, 2026, Canadian AI chip startup Taalas launched inference chip Taalas HC1, directly etching Meta's open-source AI model Llama 3.1 8B into the chip.

Key Metrics​

Metric	Taalas HC1	NVIDIA B200 (throughput optimized)	Advantage
Inference rate	16,960 tokens/s/user	Baseline	~4-5x
Cost per million tokens	0.75 cents	3.79 cents	Reduced 80%
Power	~250W	~700W	Reduced 64%
Process	TSMC N6	TSMC 4nm	More mature
HBM	❌ Not used	✅ HBM3e	Lower cost

Technical Principle​

Taalas HC1 uses an aggressive Computing-in-Memory (CIM) implementation:

• Model weights directly固化 in Mask ROM (fully hardware-defined)
• On-chip SRAM handles dynamic data (KV cache and LoRA fine-tuning weights)
• Only 2 mask layers need modification to produce a dedicated chip for another AI model; turning an AI model into a physical chip takes only 2 months

Limitations​

• Lack of flexibility: Hard-wiring cannot cope with rapidly iterating model updates
• Ecosystem barrier: The current cloud market still relies on general-purpose platforms; customers may prefer flexible solutions that upgrade with models
• NRE cost: High one-time engineering cost, requiring sufficient deployment scale to amortize

3.4 Cerebras: The IPO Path of Wafer-Scale Integration​

On May 14, 2026, Cerebras Systems officially listed on NASDAQ, becoming the first wafer-scale AI chip company to go public.

Core Technology: Wafer-Scale Integration (WSI)​

• WSE-3 (third-gen wafer-scale engine): An entire 12-inch wafer as a single chip
• 44GB on-chip SRAM: No external HBM, eliminating the memory bandwidth bottleneck
• 21 PB/s bandwidth: On-chip communication bandwidth, thousands of times that of GPUs
• Partnership with OpenAI: Signed a 3-year, 750MW, $20B+ compute cooperation agreement

IPO Significance​

Cerebras's listing marks the maturation of the inference-specific chip track:

1. Capital markets begin pricing such companies
2. Proves "non-GPU" technical routes have commercial viability
3. Provides valuation references for other inference chip startups (Groq, SambaNova, Taalas, etc.)

3.5 Inference Chip Landscape: Multiple Technical Routes Coexist​

Company	Technical Route	Core Advantage	Representative Product
Taalas	Hard-wired (Mask ROM)	Extreme inference efficiency, low cost	HC1
Cerebras	Wafer-scale integration (WSI)	Ultra-high bandwidth, large-model inference	WSE-3
Groq	SRAM-first architecture	Deterministic latency, high throughput	LPU (acquired by NVIDIA)
d-Matrix	Digital in-memory compute (DIMC)	More flexible than hard-wiring	Corsair
Etched	Hard-wired Transformer	Works for all Transformer models	Sohu
Axelera AI	Digital in-memory compute (D-IMC) + RISC-V	High energy efficiency	Metis AIPU

TrendForce predicts:

• General-purpose GPUs still dominate training and multi-model environments
• But in mature, predictable scenarios, general-purpose GPU profit margins will be compressed
• The industry shifts from general compute monopoly to a dual-track structure of general + specialized coexistence

---

IV. Overall Domestic AI Chip Landscape in H1 2026​

4.1 Industry Enters Scale-Up Phase​

Metric	2025	2026 Q1	Trend
Domestic AI accelerator shipments	1.65M units (41% share)	—	Rising
Total China AI accelerator shipments	~4M units	—	—
Hygon revenue growth	—	Doubled	↑
Cambricon revenue growth	—	+160%	↑
Moore Threads revenue growth	—	Doubled	↑

Leading vendors collectively entered the revenue realization channel, moving from "technical validation" to "scale commercialization."

4.2 Three Core Development Trends​

Trend 1: Capitalization Wave Reshapes the Landscape​

• Late 2025 to early 2026: Moore Threads, Iluvatar listed on the STAR Market
• Biren listed on the Hong Kong stock exchange
• Enflame STAR Market IPO accepted
• Kunlunxin, T-Head initiated listing processes
• TSINGMICRO, HardyVision and others advancing IPOs

Capitalization brings dual effects:

• ✅ Positive: Supports R&D and ecosystem building
• ⚠️ Negative: Valuation bubbles and revenue realization pressure

Trend 2: Capacity Becomes the Biggest Constraint Variable​

The contradiction between explosive domestic AI chip demand and limited advanced-process capacity is sharpening:

Vendor	Advanced-process capacity need	Actually obtained
Huawei Ascend	15K wafers/month (7nm-class)	Priority guaranteed
SMIC total capacity	~20K wafers/month (7nm-class)	—
Other vendors	~5K wafers/month combined	Extremely tight

Whether stable wafer capacity can be secured directly determines vendor survival. Cambricon's 75.4% inventory-to-revenue ratio is essentially a lock on capacity.

Trend 3: Competition Shifts from "Usable" to "Good"​

Early competition focused on "can it run the model"; now it's about "runtime efficiency, deployment cost":

Dimension	"Usable" era	"Good" era
Hardware performance	Can it run the model	Runtime efficiency, energy efficiency
Software stack	Basic adaptation	Maturity, framework breadth
Ecosystem	Existence	Developer community activity
Deployment cost	Insensitive	Core competitive factor

---

V. H2 2026 Outlook​

5.1 Upcoming Key Events​

Time	Event	Impact
2026 Q3	NVIDIA Rubin architecture details revealed	Next-gen flagship specs unveiled
2026 Q3	Huawei Ascend 950PR/950DT formally launched	New benchmark for domestic inference chips
2026 Q4	AMD MI350X scaled delivery	NVIDIA Blackwell competitor
2026 Q4	Cambricon Siyuan 690 launch (est.)	New-gen training chip
2027 Q1	NVIDIA Blackwell Ultra delivery	New compute benchmark lands

5.2 Key Competitive Factors Over the Next Three Years​

1. Wafer capacity access: Advanced-process capacity is a scarce resource; vendors tied to SMIC and TSMC have inherent advantages
2. Capital operation efficiency: The IPO window is limited; raising enough capital on the market determines R&D sustainability
3. Software ecosystem depth: Hardware performance is only the entry ticket; software stack maturity, framework adaptation breadth, and developer community activity are the core moat

---

VI. Conclusion: A Diverse Ecosystem Will Eventually Form​

In H1 2026, the AI chip industry is undergoing a historic transition from "one dominant player" to "pluralistic coexistence."

• NVIDIA builds higher training barriers with Blackwell Ultra while laying out inference efficiency via the Groq acquisition
• Huawei Ascend holds the domestic compute baseline with full-stack capability; 950PR begins to surpass H20 in inference
• Cambricon proves the commercial viability of domestic AI chips by turning profitable first; Siyuan 590 surpasses international rivals in specific scenarios
• Cerebras, Taalas and other inference-specific chip companies opened a "non-GPU" third route
• TSINGMICRO's reconfigurable architecture provides a diversified technical route choice for China's AI chips

Over the next three years, the domestic AI chip endgame will form a pluralistic ecosystem where GPU, ASIC, and reconfigurable computing three technical routes coexist, with cloud and edge developing in coordination. "Domestic substitution" is no longer a slogan, but an industrial reality happening now.

---

Data sources:

• Xinchuang World "2026 China Domestic AI Chip Vendor Capability Quadrant" (2026-06-16)
• TrendForce "The Era of Inference Economy: The Rules of AI Chips Are Being Rewritten" (2026-06-04)
• RayByte "Compute Doubles! NVIDIA Blackwell Ultra Chip Launched" (2026-06-02)
• Official financial reports and announcements of each company

Related reading:

• Huawei Ascend 910C Specs Deep Dive
• Full Comparison Table

---

On June 24, 2026, OpenAI and Broadcom jointly announced their first in-house AI inference chip, Jalapeño. This ASIC designed specifically for large language model inference went from design to tape-out in just 9 months and cuts inference cost by roughly 50%, marking OpenAI's transformation from a pure model company into a full-stack AI infrastructure provider.

---

1. Core conclusions (read this first)​

Dimension	Jalapeño	Current GPU solution	Advantage
Inference cost	-50%	Baseline	✅ Half the cost
Performance per watt	Clearly superior	Most advanced accelerator	✅ Energy-efficiency lead
Design cycle	9 months	~18 months	✅ 2× faster
Positioning	Inference ASIC	Train+inference GPU	Dedicated optimization
Supply	Internal only	Market purchase	⚠️ Not for sale

One-line summary: Jalapeño is a key step in OpenAI's full-stack AI strategy, using in-house silicon to cut inference cost 50% while opening a new paradigm of "AI-assisted design of AI chips."

---

2. What is Jalapeño?​

2.1 Basic information​

Item	Detail
Name	Jalapeño (a chili pepper)
Type	Application-specific integrated circuit (ASIC)
Positioning	Large language model inference
Announced	2026-06-24
Taped out	Sep 2025 (est., 9-month rapid tape-out)
Deployment	End of 2026 (gigawatt-scale data centers)
Partners	Broadcom, TSMC, Celestica
Process	TSMC 3nm
Architecture	Systolic Array
HBM	8 stacks (est. HBM3E or HBM4)

2.2 Why "Jalapeño"?​

Jalapeño is a Mexican chili known for "medium heat, strong flavor." OpenAI's naming hints that the chip:

• ✅ Medium heat: not the most aggressive architecture (vs Cerebras WSE), but effective enough
• ✅ Strong flavor: strong presence in inference scenarios (50% cost reduction)
• ✅ Appetizer: just "the first step of a multi-generation roadmap" (Broadcom CEO Hock Tan)

---

3. Deep technical analysis​

3.1 9-month rapid tape-out: the new paradigm of AI-assisted chip design​

Normally, designing an ASIC from scratch takes 1.5 to 2 years. Jalapeño went from initial design to manufacturing tape-out in just 9 months.

Key reason: deep software-hardware co-development

Technique	Description
AI-assisted architecture exploration	OpenAI used its own frontier models (GPT-5.3-Codex-Spark) to explore chip architecture design space
AI power simulation	AI models for power simulation and optimization
RL optimization	RL to optimize chip placement and routing
Broadcom silicon implementation	Broadcom provides top-tier ASIC implementation (network, switch chip experience)

OpenAI President Greg Brockman said:

"We use the frontier models that serve our users to optimize the infrastructure that runs the models of the future."

3.2 Architecture optimized for inference​

Unlike general-purpose GPUs, Jalapeño is an ASIC built from scratch around OpenAI's deep understanding of LLM inference workloads:

Architecture feature	Description
Reduce data movement	Core principle is minimizing data movement (the main bottleneck in inference)
Balanced compute-memory-network	Resource allocation optimized for inference, bringing real utilization closer to theoretical peak
High throughput + low latency	Aims to combine the throughput of leading accelerators with the low latency of the fastest dedicated inference systems
Future model support	Supports not only current models (GPT-5, GPT-5.3) but adapts to next-gen inference needs

3.3 Full-stack platform: more than a chip​

Jalapeño is a multi-generation compute platform, not just a chip:

Component	Supplier	Description
Accelerator chip	OpenAI design, TSMC fab	TSMC 3nm, 8-stack HBM
Network switch chip	Broadcom Tomahawk	High-speed interconnect (competes with NVIDIA NVLink)
Board, rack, system	Celestica	Full-rack solution
Software stack	OpenAI	Deep adaptation for GPT, Codex, Agent products

Deployment target: gigawatt-scale data centers

Broadcom CEO Hock Tan said:

"Jalapeño will begin deployment this year in gigawatt-scale data centers with Microsoft and other partners."

---

4. Performance and cost analysis​

4.1 Inference cost cut 50%​

Although OpenAI's official release was conservative on Jalapeño's cost savings — only stating its "performance per watt is substantially better than today's state of the art" without a specific percentage — per Bloomberg, Broadcom CEO Hock Tan revealed:

Early internal tests show Jalapeño achieves roughly 50% inference cost savings versus today's mainstream AI GPUs.

Significance for OpenAI:

Item	Current (GPU)	Jalapeño	Savings
Daily API calls	Hundreds of millions	Hundreds of millions	—
Inference cost share	~60-70% of operating cost	~30-35%	-50%
Annual compute spend	Billions of dollars	Hundreds of millions	Saves billions

4.2 Performance per watt clearly better than state of the art​

OpenAI's announcement states:

"Jalapeño engineering samples have successfully run complex reinforcement-learning tasks such as GPT-5.3-Codex-Spark at target frequency and power; early tests show performance per watt substantially better than today's most advanced AI accelerators."

Comparison target: NVIDIA Blackwell (today's most advanced AI accelerator)

Metric	Jalapeño	NVIDIA Blackwell	Note
Performance per watt	Clearly superior	Baseline	OpenAI official statement
Inference latency	On par with fastest dedicated inference systems	Baseline	Target
Throughput	On par with leading accelerators	Baseline	Target
TDP	Not disclosed (est. 400-700W)	700-1000W	Jalapeño possibly lower

---

5. Impact on the AI chip market​

5.1 "De-NVIDIA-ification" accelerates​

Jalapeño's launch is another footnote in big-tech's collective challenge to NVIDIA's market dominance:

Vendor	In-house chip	Type	Status	Relation to OpenAI
Google	TPU v6e / Ironwood	Train+inference	✅ Commercial	Google Cloud supplies OpenAI
Amazon	Trainium 3	Training	✅ Launched	AWS supplies OpenAI
Microsoft	Maia 100	Train+inference	✅ Launched	OpenAI exclusive partner
Meta	MTIA	Train+inference	✅ Launched	—
Apple	Neural Engine	On-device inference	✅ Commercial	—
OpenAI	Jalapeño	Inference	🚧 Deploy end of 2026	Internal + possibly sold to third parties

5.2 OpenAI is not about to fully "abandon" NVIDIA​

Brockman admitted:

"We simply cannot get compute fast enough."

Currently OpenAI is simultaneously procuring chips from NVIDIA, AWS, AMD, and Cerebras; Jalapeño is a structural supplement to its explosive compute demand, not a replacement.

5.3 Possibly sold to third parties​

Broadcom CEO Hock Tan specifically emphasized:

"This is just 'the start of a multi-generation roadmap'; OpenAI and Broadcom aim to jointly build gigawatt-scale compute clusters."

This means OpenAI may sell its hardware to third parties, provided it can secure enough supply from Broadcom and TSMC.

---

6. Jalapeño vs other in-house chips​

Metric	Jalapeño (OpenAI)	TPU v6e (Google)	Trainium 3 (Amazon)	Maia 100 (Microsoft)
Announced	2026-06-24	2024	Q4 2025	2023
Type	Inference ASIC	Train+inference TPU	Training ASIC	Train+inference
Process	TSMC 3nm	TSMC 4nm	TSMC 5nm (est.)	TSMC 5nm (est.)
For sale	❌ Internal only (maybe later)	✅ GCP	✅ AWS	❌ Internal only
Design cycle	9 months	~18 months	~18 months	~18 months
AI-assisted design	✅ First	❌ No	❌ No	❌ No
Cost advantage	-50% inference cost	Optimized	Optimized	Optimized

Key differences:

• ✅ Jalapeño is the first AI chip designed with AI assistance
• ✅ Jalapeño design cycle only 9 months (industry average 18 months)
• ⚠️ Jalapeño not for sale (at least for now)

---

7. Future roadmap​

7.1 Multi-generation chip platform​

Jalapeño is just "the start of a multi-generation roadmap":

Time	Event
End of 2026	Jalapeño initial deployment (gigawatt-scale data centers)
2027	Jalapeño v2 (est., architecture optimization)
2027-2028	Jalapeño training version (est., challenging TPU/Trainium)
2028 and beyond	Gigawatt-scale compute cluster fully built

7.2 OpenAI full-stack AI infrastructure strategy​

Layer	OpenAI in-house	Outsourced/procured
Models	✅ GPT-5, GPT-5.3, Codex	—
Chips	✅ Jalapeño (inference)	NVIDIA GPU, AWS Trainium, AMD GPU
Systems	✅ With Celestica	Microsoft Azure data centers
Network	✅ Broadcom Tomahawk	Microsoft Azure network
Cloud platform	❌ None	Microsoft Azure (exclusive partner)

---

8. Industry reaction and expert views​

8.1 Supportive views​

Expert	View
Broadcom CEO Hock Tan	"Jalapeño is just the start of a multi-generation roadmap; the goal is to jointly build gigawatt-scale compute clusters."
OpenAI President Greg Brockman	"We use the frontier models that serve our users to optimize the infrastructure that runs the models of the future."
Industry insiders	"Jalapeño's launch is another footnote in big-tech's collective challenge to NVIDIA's market dominance."

8.2 Skeptical views​

Concern	Description
Not for sale	Currently internal only; third parties cannot purchase, limited impact on NVIDIA's market share
Software ecosystem	OpenAI must build its own software stack; competing with CUDA is hard
Supply capacity	TSMC capacity is limited; can it meet OpenAI + Broadcom + other customers' demand?
Opaque performance data	OpenAI has not released specs (compute, memory, bandwidth, TDP), hard to assess objectively

---

9. Significance for developers​

9.1 If OpenAI sells Jalapeño to developers in the future...​

Scenario	Current (NVIDIA GPU)	Future (Jalapeño)
Inference cost	Baseline	-50%
Inference latency	Baseline	Possibly lower
Software stack	CUDA + TensorRT	OpenAI API (possibly open-source stack)
Procurement difficulty	High (export controls, supply shortage)	Low (OpenAI direct supply)

9.2 Worth watching even if not sold​

• ✅ 50% inference cost cut forces NVIDIA, AMD, Intel to lower GPU prices
• ✅ The AI-assisted chip design paradigm will be rapidly copied by the industry
• ✅ The 9-month tape-out cycle becomes a new industry benchmark

---

10. Summary​

Dimension	Assessment
Technology innovation	⭐⭐⭐⭐⭐ First AI chip designed with AI assistance, 9-month tape-out
Cost advantage	⭐⭐⭐⭐⭐ 50% inference cost cut, billions saved annually
Strategic significance	⭐⭐⭐⭐⭐ OpenAI transforms from pure model company to full-stack AI infrastructure provider
Market impact	⭐⭐⭐⭐ "De-NVIDIA-ification" accelerates, big-tech in-house chip camp grows
Openness	⭐⭐ Internal only for now, possibly sold to third parties later

Final recommendations:

• 🇨🇳 China market: Keep watching Huawei Ascend, Cambricon MLU, Moore Threads MTT (Jalapeño not sold to China)
• 🌍 International market: Watch whether Jalapeño is eventually sold externally and its impact on NVIDIA's market share
• 💡 Developers: Watch for possible OpenAI API price cuts (50% inference cost cut may partially pass through)

---

References​

• OpenAI official announcement (technical white paper pending)
• Broadcom CEO Hock Tan interview
• Bloomberg: Jalapeño inference cost cut 50%
• SemiWiki: Jalapeño technical details
• Tom's Hardware: Jalapeño architecture analysis
• ESMChina: Jalapeño 9-month tape-out

---

Disclaimer: Some specs in this article are estimates, subject to OpenAI's official technical white paper. OpenAI will release a detailed performance white paper in the coming months.

Last updated: June 26, 2026