As the artificial intelligence industry grapples with soaring energy demands and computational bottlenecks, startups like Subquadratic, Manifest AI, Liquid AI, Inception, and Pathway are pioneering alternatives to transformer-based architectures to build faster, more efficient large language models.
The Transformer Bottleneck
Back in the summer of 2017, AI researchers at Google published “Attention Is All You Need,” introducing the transformer neural network. Nine years later, transformers anchor every major large language model on the market.
Yet, the architecture is showing its age. According to Justin Dangel, cofounder and CEO of AI startup Subquadratic, “The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they’ve changed the world.”
The core mechanism inside every transformer is dense attention. This process encodes text meaning by comparing every single word, or token, against every other token through extensive multiplication. While accurate, the computational scale explodes quickly. A 10,000-word document forces a transformer to execute roughly 50 million multiplications.
This heavy compute load drives astronomical infrastructure costs. OpenAI is projected to spend $50 billion on computing this year, according to company president Greg Brockman. Meanwhile, the International Energy Agency estimates that total electricity consumption by global data centers will double by 2030.
Furthermore, transformers struggle with expansive context windows and modern reasoning models. Reasoning systems utilize a scratch pad known as a chain of thought to write and read notes, multiplying the data load. Transformers are hitting a structural wall.
Rethinking Attention and Retention Mechanisms
Startups are tackling this inefficiency head-on by modifying or replacing dense attention. Miami-based Subquadratic claims it has developed a sparse attention mechanism called SubQ. Instead of calculating every token pairing, sparse attention runs calculations on only a subset of word pairings. SubQ figures out dynamically which words matter for any given piece of text.
San Francisco-based Manifest AI takes a different route by swapping attention for power retention. Carles Gelada, cofounder and CTO of Manifest AI, explained in a company video last year that power retention drops less relevant data while keeping a rolling summary of the context window. Manifest AI adapted an open-source coding LLM named StarCoder into PowerCoder, and released Brumby to rival Alibaba’s open-source Qwen model.
Liquid Neural Networks and Smaller Footprints
Rather than ditching transformers completely, MIT spinout Liquid AI pairs them with liquid neural networks to build liquid foundation models, or LFMs. Ramin Hasani, cofounder and CEO of Liquid AI, notes that their models have racked up nearly 34 million downloads.
Inspired by worm brains, liquid neural networks evolve from convolutional networks. They feature a mechanism that allows the model to adapt its behavior to fresh data continuously. Standard transformers have fixed behavior post-training. Liquid AI’s recent LFMs use a hybrid architecture consisting of 20% transformers and 80% liquid neural networks, engineered by a proprietary designer AI.
Diffusion and State Spaces Beyond Words
In Palo Alto, Inception utilizes diffusion—the same technique powering image and video generators like stable diffusion—to generate entire blocks of text simultaneously. Stefano Ermon, cofounder and CEO of Inception alongside his role as a Stanford University researcher, helped figure out the math in 2024 to make diffusion work with text. Inception’s latest model, Mercury 2, matches OpenAI’s 2023 GPT-4 performance while running ten times faster.
Meanwhile, Pathway takes an even more radical approach by freeing LLMs from natural language constraints. Pathway built an LLM called Dragon Hatchling, named after Terry Pratchett’s novels. Instead of standard transformer math, Pathway uses a mathematical structure called a state space to compress information abstractly.
Zuzanna Stamirowska, cofounder and CEO of Pathway, points out that language remains an inefficient tool for complex reasoning tasks like mathematics or chess. Dragon Hatchling recently solved over 97% of a benchmark featuring 250,000 difficult sudoku puzzles where several leading mainstream LLMs failed completely.
“Transformers are an engineering convenience that we fell on,” Stamirowska notes. “It started a religion, but it’s silly to think that a breakthrough won’t happen again.”
