Fractile: Compute for the Agentic Era

The GPU was never meant to run AI. It was built to render graphics, it just happened to be very good at the one operation that AI needed: multiplying enormous matrices in parallel. When researchers discovered that a few gaming cards could train a neural network in days instead of months, the word “GPU” became inextricably tied to AI.

Training was the hard problem for the first decade of modern AI, so training defined the hardware stack. And when it came time to serve the models, inference simply inherited the same hardware.

But inference needs its own optimized compute, especially for the workloads that demand speed at scale. Inference is now on track to be the majority of AI compute spend by the end of the decade, and the workloads running on it are increasingly different. Each of these workloads have their own demands on the chip underneath. This diversity of inference demands diversity of hardware.

Inference optimized compute

The AI accelerator market is north of $200B today and, by some estimates, will exceed $1T by 2030, with inference expected to make up the majority of that spend. But the hardware serving this demand is optimized for training.

Inference works differently from training. A model generates output one token at a time, and each step is often limited by how fast the chip can move data from memory, not by how much math it can do.

In a world with as much inference as we expect, inference needs its own optimized compute. Inference itself is also fragmenting into workloads with different needs from the chip:

• Latency-bound: A real-time coding assistant needs speed first, because every millisecond between keystroke and suggestion shows up in the product.

• Throughput-bound: An overnight job summarizing millions of documents can tolerate delay from a UX perspective, and the choice is between optimizing tokens per dollar or impact per minute.

• Agentic: In the fastest-growing category, a model plans, reads, calls tools, and reasons for minutes or hours at a time. A single session can carry hundreds of thousands of tokens of context that the chip must hold and re-read on every step, while still responding quickly.

Memory is the constraint

Memory is what separates these workloads. Every token a model generates requires reading its weights and its working context back out of memory, so as models get larger and contexts get longer, memory, not compute, has become the limit on inference.

Two properties decide how a chip performs. Memory bandwidth (how fast data moves from memory to the logic) sets how quickly a model can respond to a single user. Memory capacity (how much data can sit close to the compute) sets how large a model and context a chip can hold and how many requests it can serve at once.

Agents need both at once, as do frontier models running at long context, but today's architectures trade one against the other. GPUs place high-bandwidth memory (HBM) beside the processor, which provides ample capacity, but every byte has to travel across the chip package, capping how fast the chip can be fed. SRAM-based chips put memory directly on the compute die, delivering the highest bandwidth available today, but SRAM takes up so much space that each chip holds very little. Past a small model and a short context, they need many chips or external memory, which erodes their speed advantage.

That leaves the most demanding workloads, from frontier models at long context to long-running agents, stranded in the middle: too memory-hungry for SRAM chips and too latency-sensitive to get the most out of GPUs. Serving them well calls for an architecture designed around the workload from the start, with enough memory to hold a large model and its whole context, close enough to the compute to be read at full speed on every step.

Fractile: pioneering memory stacked on compute

Fractile solves this trade-off between speed and capacity with a fundamentally new chip architecture. The result is an order of magnitude more memory bandwidth than HBM-based designs, with far more capacity than any SRAM-based chip.

Fractile's chips are built to run the world's largest models much faster, including today's mixture-of-experts transformers, while scaling to models beyond the current frontier and to extraordinarily long contexts. Abundant bandwidth also opens up model designs that are impractical to serve today, such as sparse mixture-of-experts models that reach the same intelligence with fewer computations.

The payoff reaches any field gated by intellectual work, from software engineering to drug and materials discovery, where compressing a month of compute into a day makes ambitious uses of AI economically viable. Fractile allows for remarkable speed at scale, and in doing so will underpin our most ambitious applications of AI.

The team

Walter is a rare founder who can go deep on every part of the stack, from model architecture to chip design to memory, and he has earned the trust of customers and partners across the semiconductor ecosystem because he uniquely understands the needs of these customers.

Around him is an engineering team drawn from Nvidia, Broadcom, Graphcore, Meta, and Intel with years of full-stack chip and systems design behind them,

The Fractile team is as brilliant as they are hard working, and as kind as they are determined.

We are so proud to partner with Walter and the entire Fractile team as they increase the clock speed of global progress, one chip at a time.