Inside Jalapeño: How OpenAI's Custom 3nm ASIC and 15.4 TB/s HBM4 Challenge NVIDIA's Datacenter Grip

For years, the generative AI frontier was constrained by training compute. Today, as agentic swarms, continuous test-time reasoning, and multi-step tool workflows dominate production workloads, the economic battleground has pivoted decisively to inference. At the Hot Chips 2026 symposium on August 25–26, OpenAI pulled back the curtain on Jalapeño — its first in-house custom Application-Specific Integrated Circuit (ASIC) engineered from a blank slate solely for large language model (LLM) inference.

Co-developed with Broadcom and fabricated on TSMC’s 3nm-class node, Jalapeño represents OpenAI’s boldest architectural move yet: bypassing general-purpose GPU overhead to achieve 216 GiB of next-generation HBM4 memory at a breathtaking 15.4 TB/s of bandwidth, packaged into a 700W thermal envelope.

What Happened

While OpenAI had initially confirmed its silicon ambitions earlier in the summer, the Hot Chips presentation delivered the first exhaustive microarchitectural disclosure, benchmark telemetry, and silicon roadmap:

Blank-Slate Inference Focus: Unlike NVIDIA’s Blackwell architectures (B200, GB200, GB300) which must balance general-purpose training primitives, graphics backwards compatibility, and complex FP64/FP32 pipelines, Jalapeño is strictly an inference engine. It strips out training-specific control logic and FP64 units in favor of pure matrix compute and spatial memory locality.
Massive Memory Subsystem: The silicon package integrates six HBM4 stacks yielding 216 GiB of memory capacity and 15.4 TB/s of aggregate memory bandwidth — directly addressing the memory-bandwidth bottleneck inherent to autoregressive token generation.
64-Slice Spatial Microarchitecture: The central reticle-sized compute die is divided into 64 core slices, with each compute slice mapped directly to an adjacent HBM channel slice to minimize cross-die interconnect latency.
Microscaling Numeric Engine: The matrix execution units utilize standardized Microscaling FP (MXFP) formats (including MXFP8 and MXFP6), optimizing precision-per-bit for KV cache compression and dense linear projections.
AI-Driven 9-Month Turnaround: The entire design journey — from initial register-transfer level (RTL) definitions to physical tapeout — was completed in just nine months, accelerated by OpenAI’s internal AI agent swarms handling automated floorplanning, formal timing verification, and routing optimization.

The Benchmark Data: Throughput, Latency, and Power

Independent evaluation data using SemiAnalysis’s InferenceX benchmark suite demonstrates where dedicated silicon diverges from general-purpose GPUs:

Metric / Benchmark	OpenAI Jalapeño (ASIC)	NVIDIA Blackwell GB200 / GB300	Improvement Delta
Process Node	TSMC 3nm class	TSMC 4NP (custom 5nm) / 3nm	Higher transistor density
Package Memory	216 GiB HBM4	192–288 GiB HBM3e / HBM4	Next-gen base die integration
Memory Bandwidth	15.4 TB/s	8.0–12.0 TB/s	+28% to +92% Bandwidth
Thermal Design Power (TDP)	700W	1,200W – 1,400W	~40–50% Power Reduction
Throughput / Kilowatt	Baseline (InferenceX)	Standard Reference	1.5× – 1.9× higher throughput/kW
Time-to-First-Token (TTFT)	Evaluated on GPT-OSS 120B	Standard Reference	1.7× – 2.4× lower latency
End-to-End Inter-Token Latency	Evaluated on DeepSeek R1 / Kimi	Standard Reference	Up to 3.6× lower latency

Crucially, benchmarks were not restricted to proprietary internal OpenAI weights. Testing validated generalized acceleration across third-party open architectures — including DeepSeek R1, GPT-OSS 120B, and Kimi K2.5 1T — confirming that the systolic array is generalizable across standard decoder-only transformer topologies.

Why It Matters

1. The Inference Economics Crisis

As reasoning models engage in extended chain-of-thought exploration and agents execute hundreds of tool calls per user query (as seen in Anthropic’s Claude Agent Stack and recent security evaluations), token generation volume has scaled super-linearly relative to user growth. Datacenter power limits — rather than silicon availability — have become the hard ceiling on AI growth. Delivering 1.5x–1.9x higher throughput per kilowatt directly translates into lower token serving costs and higher query capacity per megawatt of utility power.

2. The Death of the Memory Wall

Autoregressive decoding is notoriously memory-bound at low batch sizes (batch size = 1 to 4 in latency-critical interactive agent sessions). By coupling 15.4 TB/s HBM4 with 64 tightly coupled core slices, Jalapeño feeds weights to matrix cores fast enough to slash end-to-end token latency by up to 3.6x. For developers building real-time voice agents or interactive IDE assistants, sub-15ms inter-token latencies are transformative.

3. Supply Chain Sovereignty

NVIDIA’s near-monopoly on high-end AI compute allowed it to command 70%+ gross margins. By partnering directly with Broadcom for ASIC physical design, TSMC for 3nm fabrication, and Celestica for system assembly, OpenAI secures dedicated wafer allocations and captures substantial infrastructure margins in-house.

4. Silicon Designed by AI Agents

Perhaps the most notable meta-narrative from Hot Chips was the timeline: 9 months from RTL to tapeout is virtually unprecedented for a reticle-sized processor with advanced 2.5D packaging and HBM4. OpenAI attributed this speedup to internal coding agents and reinforcement learning systems that performed autonomous floorplanning exploration, timing closure closure, and formal verification in parallel.

Architecture Deep Dive: 64 Core Slices & MXFP Engine

Traditional GPU architectures rely on centralized crossbar switches or multi-stage network-on-chip (NoC) fabrics to route tensor traffic between memory controllers and streaming multiprocessors. At 700W, this data transport consumes a disproportionate amount of thermal energy.

Jalapeño solves this through a spatial core slice architecture:

1:1 Slice Pairing: The compute plane is segmented into 64 autonomous tiles. Each tile houses a dedicated vector unit, a dense weight-stationary systolic array, and a localized direct interface to an adjacent HBM channel slice.
Weight-Stationary Execution: During the generation phase of transformer decoding, weights are loaded once into high-speed local SRAM within each core slice while activation vectors stream through, minimizing expensive DRAM access cycles.
Native MXFP Precision: Rather than relying on traditional FP8 (E4M3/E5M2), Jalapeño natively implements Microscaling formats where 32-element vectors share an 8-bit scale factor alongside 4-bit, 6-bit, or 8-bit mantissas. This preserves numerical stability during long-context agent loops while halving KV cache memory footprint.

What Developers and Enterprises Should Expect

While Jalapeño is built exclusively for OpenAI’s internal cloud fleet and will not be sold as a PCIe card or standalone server to third parties, its deployment will directly influence the broader ecosystem:

Aggressive API Price Reductions: As Jalapeño servers begin deploying in late 2026 and scaling through 2027 across multi-gigawatt datacenters, OpenAI will have the margin headroom to reduce API token prices for flagship models like GPT-5.6 Sol and future reasoning models.
Lower Latency for Agent Loops: Complex multi-step agent workflows that previously felt sluggish due to cumulative token latency will experience noticeable speedups.
Increased Pressure on Merchant Silicon: Custom silicon efforts from Google (TPU v6), Meta (MTIA), Amazon (Trainium/Inferentia), and now OpenAI signal that the tier-1 hyperscaler market is diversifying rapidly beyond merchant GPUs.

Limitations and Open Questions

Despite the impressive Hot Chips demonstration, key execution challenges remain:

Software Stack Maturity: NVIDIA’s CUDA ecosystem has an 18-year head start. OpenAI must maintain compiler toolchains, kernel optimizations, and runtime dispatch layers that ensure seamless deployment of rapidly evolving model architectures without manual kernel rewrites.
Manufacturing Yields on 3nm & HBM4: Packaging reticle-sized dies with six HBM4 stacks requires advanced TSMC CoWoS (Chip-on-Wafer-on-Substrate) packaging, where industry-wide capacity constraints remain tight.
Deployment Ramp: Initial cluster deployment begins in late Q4 2026, with the bulk of multi-gigawatt capacity coming online in 2027.

Conclusion

Jalapeño represents the maturation of AI engineering from algorithmic experimentation to full-stack vertical integration. By co-designing silicon, software compilers, and neural architectures under one roof, OpenAI has built a formidable engine tailored specifically for the agentic inference era.