Google splits TPU 8 to take on Nvidia’s inference economics

Google’s eighth-generation TPU arrives as two distinct chips for the first time: TPU 8t for training and TPU 8i for inference ^[1][3]. That is a notable break from the company’s decade-long habit of designing a single accelerator family to span both phases of the AI lifecycle. TPU 8t scales to 9,600 chips per superpod and Google says it can reach 121 exaflops, while also claiming 2.8x better price/performance than Ironwood ^[1][3].

The separation is not just architectural tidiness. Training still rewards huge throughput and cluster efficiency, but inference rewards predictable latency, memory locality, and cost per request ^[4][6]. Google says TPU 8i delivers 80% better performance per dollar than Ironwood and can effectively let customers serve nearly twice the user volume at the same cost ^[1][3]. In other words, Google is trying to make custom silicon look less like a lab asset and more like a cloud operating margin tool.

The strategic shift makes sense only if you follow the workload mix. Jeff Dean’s framing, echoed in later coverage, is that as AI usage moves from experimentation to production, it becomes sensible to specialize chips for training or inference rather than force one design to do both ^[2][4][6]. That shift matters because the fast-growing category is no longer only chatbots; it is persistent agents that reason, call tools, inspect context, and keep working around the clock.

Google’s own technical choices on TPU 8i reflect that agent reality. The chip pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM, which keeps active working sets closer to compute and reduces trips off chip ^[1][3]. Google also says its Boardfly topology and Collectives Acceleration Engine cut on-chip latency by up to 5x, while the upgraded interconnect reaches 19.2 Tb/s for Mixture-of-Experts models ^[1]. Those are the kinds of improvements that matter when a system has to answer quickly, not just train cheaply.

Google’s TPU announcement is also a map of how AI hardware is being industrialized. Reports say Broadcom designs the training chip, MediaTek handles the inference chip, Intel supplies Xeon CPUs and infrastructure processing units around the pod, and Marvell is in talks on additional memory and inference components ^[1]. TSMC fabricates the whole stack, with reporting that it is targeting 2nm for late 2027 ^[1].

That breadth suggests Google is no longer treating TPUs as a purely internal differentiator. The market reaction to MediaTek’s stock jumping on the TPU 8i news underscored how much second-order business now depends on AI accelerator demand ^[1]. It also points to a real buyer pipeline: Anthropic has reportedly lined up as much as a million TPUs in a separate Broadcom-Google arrangement, a commitment that could represent roughly 3.5 gigawatts of capacity beginning in 2027 ^[1]. In practical terms, Google is trying to turn custom silicon into a cloud platform with industrial-scale preorders, not just a benchmark win.

Google is not selling TPU 8t and 8i as isolated chips; it is bundling them into AI Hypercomputer and the broader Google Cloud stack. Both chips will be available through Google Cloud later this year, and Google says they support JAX, PyTorch, SGLang, and vLLM, which reduces the pain of moving existing workloads ^[3]. That compatibility matters because the fastest way to lose a chip race is to ask developers to rewrite their software before they can realize the hardware gains.

The companion story is Workspace Intelligence, which turns the launch into a full-stack product pitch ^[1]. Google says the layer sits beneath Workspace, learns company templates and user style, and powers Gemini features that can draft decks, review invoices, and generate work product from inbox context. The implication is that Google is using custom silicon to make everyday enterprise software more agentic, while also creating a closed loop between models, data, and the hardware they run on ^[1][3].

Nvidia still defines the high end of AI infrastructure, especially for flexible training, and Google knows that ^[4][6]. But Google has an unusual advantage: it designs chips at scale, runs its own frontier models, and can feed real production feedback from DeepMind and Gemini back into silicon decisions ^[4][6]. That makes TPU 8i a stronger competitive threat than a generic accelerator announcement, because Google can tune hardware, network, software, and model behavior together.

For enterprises, the lesson is not that TPUs replace GPUs everywhere. It is that the center of gravity is shifting toward inference economics, and the winning architecture may be the one that routes each task to the cheapest acceptable compute path ^[6]. Startups like Gimlet Labs already frame this as a routing problem, and analysts note that Google’s infrastructure advantage is growing right where the market is moving ^[6]. Buyers evaluating agent platforms should therefore benchmark real latency, memory behavior, and total cost of ownership, not just headline FLOPS or model quality.