PyTorch Without Nvidia: What Google's TorchTPU and Amazon's TorchNeuron Mean for Your Business

At the PyTorch Conference on 20 October, Google open-sourced TorchTPU, a native PyTorch backend for its TPU chips, built together with the PyTorch team at Meta. On the very same day Amazon released TorchNeuron, an Apache 2.0 licensed backend that does the same job for its Trainium chips. Two separate announcements, one message: you no longer need Nvidia hardware to run standard PyTorch.

That sentence would have sounded odd even a year ago. PyTorch is the framework behind most AI models in production today, and in practice it has always meant CUDA, Nvidia's proprietary software layer. Running PyTorch on anything else was possible, but it needed special tooling, patience and usually a dedicated engineer who understood all the quirks.

Now the two biggest cloud providers outside Nvidia's orbit say the switch is close to one line of code. The claim deserves some scepticism, and we will get to the fine print, but the direction is clear, and it affects anyone who pays for AI compute, directly or through their vendors. Let's sink in!

What Google and Amazon actually shipped

TorchTPU is a native PyTorch stack for Google's TPUs, now open source on GitHub. Native is the important word here. Until now, PyTorch reached TPUs through a translation layer called XLA, which worked but felt like a different framework: debugging was awkward, many libraries did not run, and teams mostly gave up and stayed on GPUs. TorchTPU makes the TPU a first-class PyTorch device instead. Code runs in eager mode for development, then goes through torch.compile for production performance. In Hugging Face Transformers v5, moving a model to a TPU is literally one argument: device_map="tpu". The serving tools most companies rely on, including vLLM, SGLang and Ray, got TPU backends as part of the launch.

Amazon's TorchNeuron follows the same recipe for its Trainium2 and Trainium3 chips. It entered public beta on 20 October under an Apache 2.0 license, installs with a simple pip command, and registers Trainium as a standard PyTorch device. Moving a model is a call to .to('neuron'), and Hugging Face models load directly with device_map="neuron". Amazon says it ran a sweep of 100,000 Hugging Face models on Trainium, covering about 99.9 percent of real-world usage by download count.

The timing was not an accident. Both companies picked the main PyTorch industry event to say, in public, that the framework no longer belongs to one chip vendor. Meta co-developing TorchTPU makes the statement even louder, since Meta is both the steward of PyTorch and one of the largest buyers of Nvidia hardware on the planet.

Why the CUDA lock-in mattered in the first place

CUDA is the software layer that lets code talk to Nvidia GPUs, and Nvidia has been polishing it for almost two decades. Every popular AI framework, every optimised kernel, every tutorial and Stack Overflow answer assumed CUDA underneath. That accumulated convenience is often called the CUDA moat, and it is the main reason Nvidia could sell accelerators at the margins it does.

For businesses the moat showed up as a line in the budget. GPU instances on every cloud carry premium prices, availability comes and goes, and the cost of AI features in the products you buy quietly includes it. Even companies that never touch a GPU directly pay for CUDA through their API bills and SaaS subscriptions.

Google and Amazon have been building their own AI chips for years precisely to get out of this arrangement. The chips were good; the problem was always software. Asking a team to rewrite a working PyTorch codebase for exotic hardware is usually a non-starter, no matter what the price sheet promises. That is the specific wall these two releases are trying to take down.

The numbers behind TorchTPU and TorchNeuron

Google's headline figures come from TorchTitan, the PyTorch reference stack for distributed training. Training a Qwen3 8B model reached up to 52 percent model FLOPs utilisation, with near-linear scaling across as many as 8,192 of its new Ironwood TPUs. On the inference side, the AI firm Zenteiq reported over four times higher aggregate throughput after moving to the new stack on Google Kubernetes Engine, raising its daily capacity from 1.47 billion to 5.97 billion generated tokens per thousand TPU v6e accelerators.

Amazon leans on customer stories. Deepgram, the speech recognition company, says Trainium processed about 60 percent more audio per dollar than its existing GPU setup. Cartesia moved its inference pipeline with minimal PyTorch changes and only wrote custom kernels for a handful of hot layers that drive latency.

All of these are vendor-reported numbers, published at launch, with workloads the vendors chose. Independent benchmarks will take months to arrive. Still, the shape of the claims matters: both companies are arguing in price-performance terms, not raw speed, because that is where the purchasing decision actually happens.

The fine print worth reading

TorchNeuron is a public beta, and Amazon is unusually open about the gaps. Some PyTorch operators have no native Trainium implementation yet and quietly fall back to the CPU, which can erase the cost advantage on the wrong workload. Dynamic shapes, sparse tensors and several attention mask variants are not supported in this first release. And the Deepgram migration that produced the 60 percent figure took a multi-month engagement with an AWS specialist team, which is rather far from changing one line of code.

There is also a trade worth naming honestly: leaving Nvidia lock-in for TPUs or Trainium means entering cloud lock-in. Nvidia GPUs can be rented from any provider or bought outright. TPUs exist only on Google Cloud, Trainium only on AWS. For some companies that swap makes financial sense, but it is a swap, not an escape.

None of this undoes the announcements. It just means the honest summary is that switching went from a rewrite to a project, and for many workloads a small one.

What this means for businesses building digital products

Most businesses will never train a model, but nearly all of them now pay for inference somewhere: through OpenAI or Anthropic bills, through AI features in their SaaS tools, or through their own products calling a model in the cloud. Serious competition at the hardware layer is the thing most likely to stop those costs from creeping up, and over time to pull them down. The effect will not show on next month's invoice, but it compounds.

If your company runs open models in production, the practical step is a benchmark. Spin up a TPU or Trainium instance, point your existing PyTorch code at it, and measure cost per thousand requests against your current GPU setup. A year ago this experiment cost weeks of engineering; now it costs days, and the potential saving on a busy inference workload justifies the effort many times over.

If you are commissioning development work, the lesson is about portability. Ask your development partner to keep model code in standard PyTorch, avoid CUDA-specific extensions where a standard API exists, and treat the choice of accelerator as configuration rather than architecture. Code written that way can chase cheaper hardware for years. Code welded to one vendor cannot.

And if you are simply buying AI-powered tools, this is a fair question for your vendors: what hardware does your inference run on, and have you tested alternatives? A vendor that has done the benchmark is a vendor thinking about your costs as well as their own.

Final notes

One conference day does not end a monopoly. Nvidia remains the default, CUDA remains the deepest toolbox in the industry, and plenty of workloads will stay exactly where they are. But this is the most serious, coordinated attempt yet to make PyTorch genuinely hardware-neutral, and the fact that it ships as open source with Meta's name attached makes it hard to dismiss as marketing.

For business owners the takeaway is simple. The price of AI compute is set by competition, and this week competition got real. Keep your code portable, ask your vendors the hardware question, and watch what the independent benchmarks say over the next two quarters. The companies that stay flexible will be the ones that benefit first.