Maximizing TPU Performance: A Guide to Hardware-Aware AI Model Co-design

Google has released an open-source TPU codesign guide, providing developers with architectural mandates to maximize Model FLOP Utilization (MFU) on its custom Ironwood (TPU v7x) hardware.

By enforcing strict shape alignment and advanced parallelism strategies, the guide aims to minimize latency and improve total cost of ownership for LLMs and Mixture-of-Experts models.

Aligning Model Geometry with Systolic Arrays

Performance on Google’s TPUs is not merely about raw clock speed; it is about architectural symmetry. The hardware relies on wide 256×256 systolic arrays, which means that any tensor dimension failing to align with that 256-unit boundary leaves compute resources sitting idle. For developers, this requires a shift in how they define model layers.

The most frequent offender is the head_dim parameter. Standard practice often defaults to 64 or 128, but at those sizes, the MXU remains significantly underutilized during attention calculations. Reshaping to head_dim = 256—while adjusting head counts to maintain the same total projection width—effectively unlocks the hardware. Measured results from Qwen3-30B variants show this simple adjustment facilitates faster end-to-end training, particularly as sequence lengths scale into the 32K range.

Separate Buffer Shapes from Data Extents to Handle Dynamic Workloads

XLA compilation necessitates static tensor shapes to avoid the performance tax of recompilation and host stalls. However, modern dynamic workloads, such as MoE routing or paged attention, inherently resist static definitions. The solution lies in separating buffer shapes from data extents.

Rather than padding to the maximum sequence length, developers should utilize bucketed masking and scalar prefetching. By keeping buffers at fixed, precompiled shapes and using small integer arrays to track offsets and page tables, kernels can process ragged data structures without triggering expensive recompilations. This approach allows the TensorCore to treat data dependence as an addressable lookup, maintaining high FLOP counts even when batch composition changes.

Optimizing Throughput with LatentMoE

Standard Expert Parallelism (EP) often struggles with the communication overhead of moving full-width d_model tokens. LatentMoE solves this by projecting tokens into a compressed latent space before routing. This reduces the bandwidth pressure on all-to-all dispatch operations, allowing for larger nonlinearity budgets without sacrificing training throughput.

In benchmarks using a 30B MoE model, implementing an ℓ=768 LatentMoE configuration resulted in a 14% increase in tokens per second. This shift essentially swaps expensive communication for cheaper local projections. As Google noted, the strategy is particularly potent when the per-device batch size is insufficient to hide weight-gathering latency in traditional FSDP setups.

Optimization Metric TPU v7x Strategy Impact
Dimension Alignment Multiples of 256 Prevents MXU underutilization
Routing Traffic LatentMoE compression Reduces all-to-all dispatch overhead
Decode Latency Boardfly topology/CAE Aggregates HBM bandwidth

Bridging the Hardware-Software Gap

The release of these principles, including guidance for the newly announced TPU 8t and 8i series, moves hardware-aware model design forward. With native FP4 support in eighth-generation chips and increased on-chip SRAM, the gap between model architecture and physical substrate is closing. However, the onus remains on the developer to avoid “over-coarsening” their expert splits.

As noted in the documentation, over-fragmenting experts can starve the MXU tiles of work. Following the guidance of Yang et al. 2025 on Gated DeltaNet attention, developers are encouraged to prioritize compute-intensive layouts that amortize weight reads across the decode batch. This shift toward “shared-heavy” architectures is a direct response to the specific memory-compute ratios found in Google’s custom silicon.

Achieve Higher MFU by Treating TPUs as Specialized Compute Fabric

Modern AI development on TPUs requires moving beyond general-purpose training scripts. By treating the TPU as a specialized compute fabric—aligning shapes to 256, compressing routed tokens via LatentMoE, and leveraging speculative decoding to hide weight-read latency—teams can achieve significant gains in MFU. The provided repository serves as a baseline for teams looking to avoid the “performance ceiling” imposed by hardware-agnostic model design.

Photo of author

Sophie Lin - Technology Editor

Sophie is a tech innovator and acclaimed tech writer recognized by the Online News Association. She translates the fast-paced world of technology, AI, and digital trends into compelling stories for readers of all backgrounds.

Anne Hathaway’s the End of Oak Street Is HBO Max’s Biggest Hit

Ernst Criticizes Acupuncture Weight Loss Study Over Blinding Issues