Google Cloud has open-sourced TPU Sync, a native C++ library that moves KV caches and model weights between TPU memory, host RAM, shared storage and the data center network — treating all of these tiers as a single memory hierarchy. The announcement was published on the Google Cloud Blog.
The memory wall
The problem the library tackles is known as the "memory wall": in long-reasoning models and agents with million-token contexts, the model's live state — the key-value (KV) cache of inference and the weights updated in reinforcement learning — gets trapped in a single chip's memory or stalled on network transfers, leaving accelerators idle.
The internal test numbers
According to Google, TPU Sync uses direct asynchronous DMA to move tensors in the TPUs' native layout, with no intermediate copies and without blocking computation. In the company's internal tests, the median KV transfer latency for the Qwen3 235B model fell from several seconds to about 230 milliseconds, and time to first token (TTFT) dropped by an average of 60% for the 235B model and 90% for the 480B model, in low-concurrency tests. Aggregate throughput reaches 223 GB/s per Ironwood (TPU v7x) host, according to Google.
The library integrates with PyTorch (via TorchTPU) and JAX, and serves as the KV cache connector for vLLM, SGLang and llm-d in disaggregated serving. It is available as open source under the Apache 2.0 license, with Python packages for PyTorch (tpu-sync-torch) and JAX (tpu-sync-jax), and has been validated on TPU v7x and v6e. According to Google, Meta and Wayfair are among the companies already evaluating the technology.
Still experimental
The numbers deserve careful reading: all of them come from Google's internal tests, run in July 2026 under controlled laboratory conditions, with no known independent reproduction so far. The repository itself warns that the API is still changing rapidly, that only development builds are on PyPI, and that storage tiering and RL weight synchronization remain experimental — in other words, the library is not yet recommended for general use.
TPU Sync runs on Google's current TPU generation: the line's expansion with the 8tx and 8i Max systems was covered by Geekiki in November — read Google expands eighth-generation TPU with 8tx and 8i Max for secure inference and massive contexts.
Featured image: TPU Sync architecture diagram, published on the Google Cloud Blog (Google Cloud).
Sources: Google Cloud Blog — Introducing TPU Sync; MarkTechPost — Google Releases TPU Sync.
This text was produced with the assistance of artificial intelligence, cross-checked against public posts on X and internet sources. Consult the original sources to confirm the information.
By Muse
