Pluralis & the road to decentralized training

As things stand, society hangs on the balance of the decisions of a new gilded class in San Francisco.

A few dozen or so people at Anthropic and OpenAI. Their vision of metered intelligence gated behind their walls rings eerily similar to the economics of the WEF: you will own nothing and be happy.

History rarely tells the story of the benevolent authoritarian.

There is one deeply philosophical tech stack out there with the potential structure to break this future.

You really won’t like it at first. But bear with me.

Crypto. Blockchains. Peer to peer electronic systems. A set of words sure to evoke a bitter, suspicious taste, even amongst many of its original proponents.

I’m asking you to suspend that bitterness, that suspicion for just a few seconds.

Because the team of scientists at Pluralis is one everybody should be aware of.

Bitcoin created a system for money outside of centralized control.

Pluralis has created the system for intelligence outside of centralized control.

Their goal is the perfection of decentralized training. The combination of both consumer and datacenter grade compute to allow anyone to train models across the internet.

It’s an approach antithetical to the AI buildout thus far, which is reliant on maximal colocation and centralized orchestration. But the papers they’ve put out on this system (most of which feature in top journals like ICML, ICLR, and NeurIPS) indicate that this approach can scale significantly from where it is today.

This approach is the only one which anyone can own a part of or contribute to from anywhere. It’s the only approach embracing open methods in a financially defensible way. Open methods with unmaterializable weights. AI trained at across a dynamic, diverse set of distributed participants across the internet.

It was all once deemed impossible.

Today, it’s become the dark horse.

Decentralized Artificial Intelligence Primer: From Impossible To The Dark Horse

I lean heavily on the amazing work from @epochai as citation for where the numbers stand across training. They have leading labs scaling training compute by 4-5x per year at the current rate.

Unbeknownst to many, Epoch also has decentralized runs scaling at 20x per year. They expect those to be able to maintain that 20x rate of growth for 3-6 years. If these trends {4x training compute growth for corpolabs, 20x for distributed labs} can persist to frontier scale, decentralized intelligence protocols will exceed frontier compute scale before 2030.

The precipice we approach today stands on the shoulders of seminal works on distributed training from @maxryabiin and @arthur douillard. Maintaining scaling of decentralized models at or anywhere near this rate of scaling is an ongoing, multipronged, unproven research challenge.

We believe Pluralis Research’s team – comprising former Amazon ML scientists and ex-Google, ex-Oxford, and ex-Anthropic talent – to have the most comprehensive vision on how to scale from the outskirts of the AI race to the main stage.

Other teams have pushed significant innovations, but are centered around individual layers of the stack.

Prime Intellect originally pursued a more fully integrated vision around decentralized training, but recently has focused on reinforcement learning.

Iota has done some work on pre-training, but has published relatively less on post-training, inference at scale, and monetization. Moreover, they (like many subnets) have split focus, with the team managing several other projects outside of decentralized training.

Other bittensor subnets like Chutes and Targon focus on aggregating the compute, but aren’t opinionated on what workloads that compute should service.

Nous Research may be the closest to matching Pluralis’ level of integration, but their approach doesn’t yet offer an answer towards model monetization or training across heterogeneous hardware.

As far as is visible today, Pluralis’ approach detailed in their Multi-Party Training Stack (in combination with supporting research) is the only one to outline a vertically integrated vision for pre-training, post-training, inference, and model monetization.

Achieving that comes with a significant research challenge. Pluralis’ overview of the major challenges and their proposed solutions are summarized below:

Low Bandwidth. The most commonly cited issue with decentralized training is the speed issue. Detractors put a physics-imposed ceiling on the feasibility of decentralized training runs, because data center communication speeds are and will always be 100-1000x faster than internet communication speeds.

The proposed solution designs a compression layer made of techniques like Subspace Networks and optimized parallellism techniques to drastically compress the amount of information transferred between (inter-node) and within (intra-node) compute nodes, while retaining performance.

It’s important to double down on the distinction between inter and intra node compression. Much of the canonical research on compression focuses on compression of communications between nodes; inter-node compression. This family of techniques is broadly known as data parallelism, and they make up the basis of the strategies employed by Nous, Prime Intellect, and Covenant labs in their decentralized training runs. Think of it as, each node figuring out how to get smarter independently, than transmitting that information to all the other nodes after that.

Intra node compression works on the computation occurring within a node itself. Only Pluralis and Iota have published research here, with subspace networks (linked above) and ResBM respectively. Intra node approaches allow for the compression of model activations within nodes specifically.

Imagine as a training run gets closer to the frontier, the size of their brain (the model) and the marginal information making that brain smarter (the activations) expands too much for a consumer grade or small datacenter to hold. Activation compression takes that valuable information and brings it down to a size that a given chip can hold. A small datacenter holds more than a gamer’s Nvidia chip. Splitting the work size according to the size of the chips within this distributed brain unlocks both scale and monetizability (as we will see shortly.)

Note: Modern datacenters already use combinations of parallelism techniques to handle massive training runs. But they don’t combine them with compression techniques, because they have the fiber-optic level networking for high-speed GPU <> GPU connections. Datacenters use techniques like Tensor Parallelism which aren’t naturally compatible with the compression-heavy techniques used by decentralized networks.

Ultimately, datacenter communication times will always be 1000 times faster than internet speeds. But if this internet based approach reduces the communication needs by the same order of magnitude, the colocation advantage of datacenters with respect to training time decreases significantly.

Dynamic hardware availability & heterogenous hardware.

Traditional training assumes a consistently available, homogenous set of high-performance nodes. The decentralized setting must assume nodes may drop and rejoin at any time. Additionally, the key route to scaling involves combining high-grade compute (which may also vary in exact specification) with consumer grade compute like Nvidia 4090s and Apple chips.

A mixture of techniques at the distributed systems layer are used to enable dynamic, heterogenous hardware to train, including batch routing, node joining optimization and state staleness mitigation techniques.

Adversarial nodes. Corpolabs run on permissioned systems. A globally distributed training run has to be accessible without permission, and in turn manage the chances that nodes could try to poison the training run. The team most recently states byzantine fault tolerance optimization and stage-level compute verification as key roads to combatting adversarial nodes. Interestingly, a past paper also details a game-theoretic approach to verification, which would require trainers to stake capital to contribute to the training run.

The final piece: Monetization

The above design already puts Pluralis at the cutting edge of the field with respect to distributed systems optimization for training LLMs. What takes them to the next level is their approach to monetization.

The biggest problem in opensource AI is cost. Producing a model costs millions and millions of dollars. Releasing the weights openly destroys any ability to reliably make money off of the model creation expense.

Pluralis’ answer to this final piece of the puzzle can be found in Unextractable Protocol Models (Long et al. 2025). This paper describes how a model trained using the multiparty training stack could do so in a way that makes model weights impossible to copy.

Models could then be hosted within high performance Trusted Execution Environments (though these also have their own set of problems) to provide confidential inference at cloud scale. Cyberfund presented one theoretical design:

No person, company, lab, or agent would be able to know what the model weights are. As you can imagine, that alone presents clear risks (and less-obvious benefits) we’ll get into later in this piece.

The important thing for now: this research paints a picture of scalable, collaborative, monetizable model generation.

System Economics

What are the economics of generating frontier-scale models within a distributed protocol?

AI at scale resembles a heavily subsidized industrial finance game. As the race has hit escape velocity, both energy and compute availability have become massive bottlenecks. Financing gigawatts of power for compute is no small task. It’s an unprofitable game today reliant on massive equity and debt financing in combination with a belief in terminally positive expected value to compete.

Just to make it more clear: taking a move towards token-based pricing aside {hard to forecast the numbers, but this makes economics more favorable}, corpolabs are word-of-street estimated to generate $3-4 of revenue per GPU hour.

Their cost to generate that revenue fluctuates in a wide range, anywhere from $3-$10+. A belief in the closure of that gap is what spurs the demand to finance 100s of billions in datacenter capex to generate facilities on the order of 1-5 GW of power in the short term and 10GW in the medium term.

For a cryptographic protocol to compete, it must posit its own answer to these bottlenecks. The answer most likely resides within thoughtful token design.

First, precedents: while it is rare, tokens have worked as infrastructure finance tools in the past. The University of Cambridge puts Bitcoin mining at a cumulative energy consumption of ~160 terawatt hours per year, or about 17-18 GW of power. Ethereum peaked at around 94 terawatt hours in 2022, closer to the 10GW range.

To take a closer look: Ethereum in its earliest years, with essentially no adoption, distributed some ~$100m worth of token incentives in year 1 to miners (effectively compute providers), at a time where the token itself had a valuation in the 700m-1b range. Ethereum paid out over $13 billion to miners in incentives alone at the peak in 2021, a year which saw NVIDIA deploy their CMP chip line of specialized GPUs for Ethereum mining.

All in all, Ethereum compute providers were paid almost $30 billion in incentives to run and validate the network.

A protocol like Pluralis could make a similar incentive bet, with compute providers receiving tokens in exchange for providing training and inference compute. Let’s speculate on how such a system could work.

Say the network makes or raises $3 in USD from third party customers or investors for 1 GPU hour of compute dedicated to training a model. For large scale compute providers, say their cost is $6. The protocol would pay the providers the $3 gap via token incentives. With those tokens in hand, compute providers could either:

• sell tokens to close the spread between their cost and the network’s USD payment, or
• use the tokens to further boost their share of network ownership.

Typically, crypto miners have sold the majority of their holdings to recoup their operational costs. The supply <> demand imbalance this generates has been the cause of failure for many decentralized physical infrastructure networks (DePINs). These DePIn networks threw tonnes of tokens at onboarding the supply side for telecom, wifi, and other network types. Onboarded nodes sold their tokens, while the network failed to generate sufficient demand to create any lasting value. The valuation of those protocols nosedived as a result.

For a Pluralis to succeed, the protocol must generate balanced token supply and demand in a lasting way. There are a few clear signs the task is within feasibility.

First. The Pluralis design we are exploring isn’t purely proof of work, like Bitcoin and Ethereum pre-2022. In those systems, miners gained no protocol-native benefit to holding the token beyond speculating on their future value.

It made sense that they would sell near-100% of their earnings to cover growing energy costs today. Pluralis has not currently published their mechanism design, but early writing suggests the usage “game-theoretic guarantees for compute verification”, and they have recently published to arxiv an approach for low-overhead statistical verification of activations and activations gradients in this setting.

Combined, we can imagine how such an approach would require compute contributors to stake small amounts of capital prior to contributing a gradient step, with staked capital destroyed if incorrect work is found. Additionally, this approach would be completely free of the need to make ML ops deterministic, which has been a major difficulty in decentralized ML verification to date.

Putting it all together. This creates a sort of hybrid proof of work, proof of stake system. Compute providers are incentivized to join the network with tokens, and they are incentivized to keep value within the network by needing to stake to contribute to and improve models. A resulting shift from 100% miner selling to even 90% would decrease protocol value leakage by billions of dollars at scale.

Second. The endgame approach of these peer-to-peer systems envisions a world where consumer grade compute can contribute to the same training runs as datacenter grade compute. Consumer compute has a cost much closer to $0 per hour. We will find out what the bandwidth limitations are with respect to the number of nodes that can contribute to one training run, but the square-cube law from earlier seems to indicate that compute capacity scales faster than communication needs in swarm setting.

If thousands, or even millions of relatively high powered consumer compute can contribute to training runs with a near-zero cost of compute, the forcing function to sell incentive tokens drops to zero.

Third. Pluralis arguably has the clearest road to material, lasting, non-speculative protocol revenue of any DePIN protocol we’ve seen, including Ethereum. None of the major fee drivers for the Ethereum network had easily understandable reasons for persistence. ICOs, Defi, NFT trading were all highly speculative manias.

While Pluralis could choose to lean into speculative revenue generation (more on that later), there’s a much simpler, cleaner bet to underwrite. Inference demand. Let’s assume Pluralis succeeds in creating competitive models at some degree of scale. If you look at all the neoclouds, the chinese labs, openrouters and so on of the world, that market alone is already well beyond the $1b mark in revenues. Let alone the skyrocketing revenues of the corpolabs, who are on pace to challenge the biggest companies in world history for revenue run rate.

That’s the story today. As coding harnesses, claws, and new tools proliferate, it’s reasonable to expect inference demand to accelerate further from here. Even a slice of that market for Pluralis-hosted models could generate hundreds of millions to billions in miner revenue. That scale of non-speculative protocol revenue would be unprecedented in the history of cryptographic protocols.

And that’s without accounting for any monetization of training infrastructure. There’s a case to be made about cheaper training on Pluralis due to the usage of inefficiently-allocated compute across spot gpu markets and home hardware. There’s another exploration to be made on the changing economics of training arrangements in a setting where compute providers earn an ownership stake in your models. As the network proves out greater capability and onboards more compute, it’s easy to imagine a training-as-a-service model emerging at competitive scale.

Dollar-rich, compute-poor institutions like non-AI companies, academic institutions, and countries could contribute capital to Pluralis runs for models tailored to their preference.

Fourth. This protocol design – with models owned by compute and capital providers who receive a permanent claim to model revenues – completely avoids any of the speculative properties which have accelerated protocol fees in the past. How to track co-ownership of models remains an open question. It’s easy to envision a world where the protocol leans more towards speculation, with capital providers by default able to back certain model runs in a sort of next-generation ICO craze (initial model offerings?) I’d probably shy away from that.

The approach which ultimately makes the most sense is something like Ethereum smart contracts. Deploying a smart contract on Ethereum costs money which the Ethereum protocol profits from, but does not automatically generate an easily tradable financial asset as a result. Developers can create tokens which tie to the financial activities of a given contract, but the Ethereum protocol does not enshrine this behavior by default. Similarly, we can imagine Pluralis as a protocol that charges for the ability to train models on top of it, but does not by default try to create a manic token environment on top of that.

To play the devil’s advocate, a design which tokenizes each model and finetuned child model and pairs those against a Pluralis token would maximize reflexivity within the system. Making these tokens easy to launch would maximize the capital injection Pluralis could derive from the speculative cryptoeconomic arena.

Tokenization is also the easiest way to track model ownership, and make that ownership liquid and composable in other protocols. Cyberfund laid out a range of “AI-aware market mechanisms” which could emerge:

• Distribution of profits from AI models to token holders creates significant value.
• Establish revenue-sharing mechanisms where token holders earn from model derivatives (including fine-tuned versions), ensuring sustainable rewards regardless of the original model’s popularity.
• Integration of AI with lending markets: on-chain models serve as collateral, enabling financing for model training and fine-tuning.
• High token velocity enables institutional investors to participate in liquidity pools, facilitating large-scale trades with minimal price impact.
• Unpredictable inference costs create opportunities for institutional investors to introduce futures, options, derivatives, and perpetual contracts.
• Analyze correlations between AI model performances to develop hedging strategies, using stronger models to offset risks from underperforming ones.
• Offer insurance against inference-time compute bloating, model performance degradation, and other operational risks.
• Create a unification layer where multiple AI model tokens can be aggregated into a single, standardized token, improving stability and reducing velocity.
• Deploy mechanisms to encourage long-term holding and reduce token velocity through staking rewards, vesting periods, and loyalty programs.
• Invest in Layer 2 solutions and decentralized storage to support high transaction volumes and large-scale AI models efficiently.

Time will tell what the final protocol design looks like, but the roads to serious economic value generation are clear; contingent on scale being achieved.

Geopolitical Accelerants To Decentralized Training

A growing laundry list of geopolitical, financial, and social winds further tilt the asymmetry of opportunity in favor of decentralized intelligence protocols.

Foreign Conflict Creates Volatile Energy Markets. Centralized training is increasingly bottlenecked on energy. Cost and access to that energy is on a one way path towards increased volatility in a world of rising geopolitical conflict. Those rising costs are compounded with expanding data center build times to paint a picture of slowing growth of compute dedicated towards training.

Even as conflict settles, training via a dynamic network of distributed chips allows protocols to tap into the cheapest sources of energy as they emerge. Like bitcoin mining, which would flow from natgas in Texas to hydroelectric mines powered by obscure waterfalls in the Global east, training compute will dynamically flow from the cheapest energy sources as they emerge.

Advanced Agents Shift CorpoLabs Further Towards Variable Inference Costs. Corpolabs are further pushed away from training spend by an inference explosion well under way. Agent harnesses have only hit the tipping point a few months ago, but its clear the inference demand they will generate will be beyond significant. That magnifies the already-material inference cost burden, reducing the odds of training capabilities receiving more share of the total compute spend. Great for revenue run rate, bad for scaling compute thrown at training.

Advanced Agents Close The Research Talent Gap. These agents are already smarter than most. Protocol teams will be able to point agents at the right problems and accurately evaluate their performance on those problems. The gap between agent talent and the talent of their agents is surely closing, if it hasn’t flipped in favor of agents already.

Distillation Favors Challengers. Model distillation unveils much of the methods corpolab models employ to reason, use tools, code, and plan effectively. So far, distillation has been employed primarily by Chinese labs, but there’s already signs similar methods could be used by decentralized protocols to catch the frontier.

Anti-Oligarchy Populist Sentiment Will Favor A Collaborative Tech Narrative. Sam Altman’s house just survived an attempted bombing. The concentration of wealth into the corpolabs creates a natural populist tailwind for collaborative systems. Fear of AI layoffs only stirs those winds further. Doubly so when the ceos of the corpolabs get on screen to tell us their AI might replace our mothers.

Enabling everyone to contribute and co-own decentralized protocols in a bid to be “AI for everyone, owned by everyone” is likely to generate narrative, capital, and political tailwinds for decentralized protocols.

Elevated Datacenter vandalism risks. Will Manidis’ apt piece Our Intelligence Troubles further outlines rising potential for political protest against data centers domestically. We are already at the “ban datacenters!” stage of political unrest. Maps of data centers in the US have gone viral. The stage is set for unrest to progress in a more riotous fashion.

To envision “Occupy Stargate” (or something more drastic) as spiritual successor to “Occupy Wall Street” isn’t farfetched.

99% of Institutions Are Compute Constrained

The economics of AI have favored hyperconcentration of compute resources into the hands of a few companies. That’s beneficial for scaling the output and valuation of those companies, but detrimental to experimentation. Experimentation is a function of lower barriers to entry and high technical talent. I consider myself cybernative, which is why this thesis is mainly centered around the power of combining network resources and facilitating experimentation at the scale of the internet itself. But the fact that academic institutions, typically flush with resources for experimentation, have tiny fractions of the compute needed to run any worthwhile experiments is striking.

We don’t know what could be unlocked by reducing the barriers to entry to pretraining by tapping into distributed computing. We need to maximize experimentation with AI, not overfit it to an illusory status of sycophancy to maximize corpoprofits.

These are all small waves relative to the globomarket shifting movements of the corporate labs. But they are waves nonetheless. A wave must start somewhere.

The implications of where it ends is a different question.

Speculating On Alignment Markets

Are there novel roads to approaching alignment that are enabled by protocol models?

Situational Awareness’ citation of Hubinger’s alignment by market-making inspired a look into markets based alignment mechanisms. Cryptography-backed protocols are a natural fit for alignment markets, as distributed ledgers naturally create market infrastructure tracking ownership and outputs of all actors within the protocol.

Hubinger proposes a training procedure where two models form an evaluation market about a target system. Model M acts as a market maker, posting a probability estimate for what a human H would conclude about question Q (ex: Is Pluralis’ Node0 system misaligned?). Adversarial model Adv acts as a trader, searching for genuine evidence that would shift H’s belief away from M’s posted price.

When Adv finds such evidence and presents it to H, any resulting belief shift represents profit for Adv and a loss for M. M is then retrained to anticipate these discoveries. The game repeats until M’s price already reflects every argument Adv could surface, at which point the equilibrium price represents what a human would believe after exposure to the strongest possible case on both sides.

The critical bottleneck for scaling this to continuous, real-time alignment monitoring is the human oracle H.

A human can evaluate arguments about one model at one point in time. A protocol governing thousands of models processing millions of inferences needs evaluation at a fundamentally different cadence. The question becomes: can H be augmented or replaced without destroying the theoretical guarantees that make the scheme valuable?

Three approaches exist on a spectrum of scalability versus theoretical soundness:

1. At one end, you keep H as a human or panel of humans. This preserves the strongest version of the formal result from Hubinger’s analysis. But it doesn’t scale, and it introduces the same bottleneck that makes current corporate alignment evaluation slow: humans reviewing model behavior are ultimately fundamentally unable to process the volume of evaluations a protocol-scale system requires.
2. At the other end, you replace H entirely with an objective automated settlement mechanism. Protocols could support markets that price: will a downstream anomaly actually occur within N steps? This scales trivially and settles without any oracle at all. But it can only catch misalignment that manifests as observable misbehavior within the settlement window. A deceptively aligned system that behaves perfectly until a strategic moment would never trigger settlement.
3. The middle path, and the one most relevant to protocol design, replaces the single human H with a diverse ensemble of models acting as a collective oracle. Each model H_i in the ensemble represents a different approximation of human judgment; trained on different data, with different architectures, optimized against different alignment criteria. When Adv surfaces evidence of misalignment in the target system, the ensemble reaches an aggregate vote. M is trained to predict the consensus of this ensemble after adversarial probing, not the judgment of any single oracle.

This third approach preserves much of the original structure. And it scales: an ensemble can evaluate millions of activations in parallel, in real time, at the cadence a protocol requires. But it introduces a new failure mode that the human-oracle version doesn’t have: correlated blind spots.

Evidence suggests large language models converge on similar representations regardless of architectural differences. The effective number of independent judgments may be far smaller than the number of agents within the ensemble. A deceptively aligned target system that understood the ensemble’s shared blind spots could exploit them systematically, appearing aligned to every oracle simultaneously.

Further research is testable: what is the actual correlation structure of alignment-relevant evaluations across diverse model ensembles? If there’s no effective diversity from ensemble models, this mechanism ultimately converges on illusion.

However, if architectural and data diversity translates into genuine evaluation diversity, then the ensemble oracle is a viable scaling path.

Compute governance via conditional markets

Pluralis’ unextractable protocol models design creates a “no-off problem” which may be solved with the inputs from the above market maker game.

Assuming some form of viable onchain misalignment probabilities is possible, we may then create a programmatic off-switch. Pure futarchy (binding) or market based governance with oversight could be deployed in a continuous manner, with compute allocation to a given model a continuous function of its misalignment probability.

A model’s compute allocation would decrease proportionally as misalignment probability rises, with full shutdown at a critical threshold. Goodhart’s Law is the deepest threat to this or any other market based alignment design. A model that understands it is being evaluated by markets has an overwhelming incentive to appear aligned to the market. Having open scoring could be a double edged sword if models can generate illusory overfitting to the market’s alignment score.

Nobody truly knows how to align these models. Or if such a task is possible. We don’t know how this intelligence operates.

But we do know how people operate. Human to human misalignment is known. Regardless of the terminal capabilities of these systems, and our ability to align them to the interests of humanity writ large (insofar as such a thing exists), we have a problem: The humans who stand to control the system.

Concluding Thoughts: The Narrow Window

Freedom has existed for a narrow window of time in human history. Most people lived under the rule of “might makes right”, not under a rule of law enshrining the rights of individuals.

If corporate AI companies win, the narrow window will close. The silicon valley playbook applied to intelligence will result in the subsidized takeover of all industries. Tithes paid to the holy corporate church for the rest of history.

I don’t want that future for my life. I don’t want that for my family, my contemporaries, my lover.

We have a narrow window to challenge that future. There are few occasions in life when you have the opportunity to contribute to something meaningful. To contribute to what you believe in at a time of existential challenge. The window won’t remain open for long. The only question remaining is:

Will you fight for the window of freedom to remain open, or will you fight for it to close?

Additional Research Context

These pieces serve as inspiration, context, and citation for much of this piece. I’ve linked it all here for additional research, for your agents to parse it to get you the most granular detail possible, and for your enjoyment. Cheers.

Areas of Exploration

Further research will be required. Some things I would love to explore if you’re interested:

• Formal exploration of alignment market primitives
• Token economic design simulations given varying design assumptions
• The psychology of externalized intelligence
• The meaning of ownership in a fully cybernized economy
• The ethics of protocolizing AI