Microsoft Maia 300 AI Chip Targets Large-Scale AI Inference With 288 GB HBM4 Memory and 18 TB/s Bandwidth

Microsoft has introduced its third-generation Azure Maia 300 AI accelerator, designed to improve the efficiency and performance of artificial intelligence inference workloads. The system combines custom silicon, software, networking, rack infrastructure and datacenter technologies to support large-scale AI applications and agentic workloads.

Maia 300 is designed to meet the growing computational requirements of AI systems that perform multi-step tasks, process long contexts and interact with external tools. Microsoft said the accelerator aims to increase useful AI tokens delivered per watt of power and per dollar spent, helping improve the economics of operating AI services at cloud scale.

Microsoft Maia 300 Targets AI Inference Efficiency

The Maia 300 architecture focuses on optimizing data movement, memory utilization and communication between accelerators rather than relying solely on increases in computing power. The design builds on the Software Defined Locally Accessed Dataflow Architecture (SDLA) introduced with Maia 200.

The SDLA approach allows software to coordinate data placement, movement, conversion and synchronization across specialized memory, compute engines, direct memory access (DMA) engines and networking components. This approach is intended to reduce unnecessary data movement and improve the efficiency of inference workloads.

Maia 300 also introduces a non-uniform memory access (NUMA) organization across chiplets. This enables software to place data closer to the computing resources that use it, potentially reducing data transfer distances and associated energy consumption.

Maia 300 Introduces Multi-Chiplet Architecture and Low-Precision Computing

Microsoft has adopted a multi-chiplet design that separates computing functions from memory-controller and input/output functions. The compute chiplets use TSMC’s N2P process technology, while other components are designed around their specific bandwidth, connectivity and cost requirements.

According to Microsoft, these architectural changes increase the system’s computing performance by 36%, provide 2.5 times the high-bandwidth memory bandwidth and deliver 14% more scale-up networking bandwidth.

The accelerator also supports enhanced MXFP4 and sub-3-bit storage formats. These lower-precision representations reduce memory requirements and the volume of data that must be transferred during inference. The approach is intended to support larger AI models, longer context windows and more concurrent sessions within the available power envelope.

Additional instruction-set architecture enhancements optimize inference operations, including softmax calculations and low-precision data conversion.

Memory Capacity Reaches 288 GB Per Accelerator

Maia 300 incorporates eight HBM4 memory stacks, providing 288 GB of memory capacity and 18 TB/s of memory bandwidth per accelerator. The system also combines high-bandwidth memory with software-managed on-die static random-access memory (SRAM).

Compared with Maia 200, Microsoft said Maia 300 delivers 2.5 times the memory bandwidth, 33% more memory capacity and approximately twice the HBM bandwidth per watt.

These improvements are designed to keep more model data and key-value (KV) cache state close to the computing resources, supporting inference workloads that require substantial memory capacity and bandwidth.

Microsoft Maia 300 Scales to 12,288 Accelerators

The Maia 300 networking architecture is designed to connect up to 12,288 accelerators within a single superpod. The system uses a two-tier network structure that combines direct accelerator connections, rack-level switching and optical connectivity between racks.

Each rack contains 96 accelerators connected through a combination of direct electrical links and switched paths. A second networking tier connects up to 128 racks to form a superpod.

Microsoft said its integrated networking architecture enables accelerators to communicate directly with other accelerators across the superpod using a common communication model. The design avoids requiring host processors or separate scale-out network interface cards to manage every communication path.

The network architecture also supports flexible bandwidth allocation based on workload requirements. An accelerator can deliver up to 750 GB/s to a peer within the superpod when only a subset of accelerators uses cross-rack connectivity.

This configuration allows latency-sensitive tensor-parallel workloads to remain within a rack, while expert and pipeline parallelism can extend across racks. The approach is intended to improve resource utilization when workloads do not align with the physical configuration of individual racks.

Disaggregated Inference and Rack-Level Resilience

Maia 300 supports disaggregated inference, allowing prefill and decode operations to run on independently sized and placed groups of accelerators.

After the prefill stage, the KV cache state can be transferred directly to selected decode workers through the scale-up network. This reduces reliance on a separate host-mediated networking path for the transfer.

The superpod architecture also supports the use of spare accelerator capacity across racks. If an accelerator or tray becomes unavailable, scheduling systems can use available capacity in another rack, helping maintain workload availability and improve infrastructure utilization.

Maia 300 Rack Integrates 96 Accelerators Within 167 kW

Microsoft has designed the Maia 300 rack to accommodate 96 accelerators across 24 trays, with four accelerators in each tray. The rack operates within a 167-kW power envelope and integrates host processors, memory, storage and cooling into the accelerator trays.

Microsoft said the configuration doubles accelerator density compared with Maia 200. Consolidating the system components into a rack-level design also reduces the space required for separate host trays and allows power and thermal resources to be managed across the system.

The rack incorporates six high-radix switching application-specific integrated circuits (ASICs) distributed across three liquid-cooled switch trays. Its power infrastructure includes dual power feeds, high-capacity power shelves and a 50-volt busbar architecture.

An automatic transfer switch provides redundancy when one power feed is unavailable for servicing.

Liquid Cooling Supports High-Density AI Infrastructure

Maia 300 uses a combination of liquid and air cooling to manage the thermal requirements of its high-density configuration.

According to Microsoft, liquid cooling removes more than 80% of the rack’s thermal load from accelerators, voltage regulators and switch ASICs. Air cooling handles memory, storage and other supporting components.

The system can connect directly to facility water where available or use a heat-exchanger unit in datacenter environments that rely on air-based heat rejection. This provides flexibility for deployments across different datacenter infrastructure configurations.

Microsoft has also integrated rack- and row-level power telemetry with datacenter operations. The monitoring system is designed to improve reliability, fault handling and observability while helping operators allocate power and optimize workload placement according to available power and cooling capacity.

Microsoft Reports Up to 7 Times Higher Token Throughput

Microsoft said Maia 300 can deliver up to seven times greater rack-scale token throughput than Maia 200 when deployed in its full 96-accelerator configuration. The company also reported more than 2.5 times higher performance per watt compared with the previous generation.

The performance figures reflect Microsoft’s stated comparisons with Maia 200 and are based on the specified rack-scale configuration.

The combination of higher accelerator density, improved memory bandwidth, low-precision computing and integrated networking is intended to increase the amount of AI inference capacity that can be deployed within a datacenter’s power and cooling limits.

Microsoft Plans to Share More AI Infrastructure Updates

Maia 300 represents the next stage of Microsoft’s effort to develop AI infrastructure optimized for large-scale inference. The company is targeting workloads across Microsoft Copilot, GitHub Copilot, Microsoft Foundry and its broader AI services portfolio.

Microsoft also plans to discuss its AI strategy, technology adoption trends and infrastructure developments at Microsoft Ignite 2026, scheduled to take place in San Francisco and online the following month.

The Maia 300 design reflects a broader focus on improving the efficiency of AI infrastructure as demand increases for reasoning-intensive applications and autonomous AI agents. Its reported performance gains are intended to help Microsoft deliver greater inference capacity while managing energy consumption and datacenter operating constraints.