Collective Model Intelligence: The Next Scaling Axis?

In Yuval Noah Harari’s Sapiens, he explains that from a genetic and intellectual perspective, humans are much closer to our chimpanzee ancestors than we might like to admit; indeed, we share approximately 98.8% of our DNA. On an individual basis, Harari explains, prehistoric humans were “insignificant animals with no more impact on their environment than gorillas, fireflies or jellyfish.” How, then, did humanity so completely change its relationship with the world around it?

Harari proposes several answers (tools, language, fire, and agriculture), and it’s true that each was an evolutionary essential, without which humanity as it stands today would not exist. But each of these things also largely centers on individuals or tribes. Harari explains that the true juggernaut of human progress was collectivism.

The greatest leaps in scientific progress have not been driven by a singular person, but by the organization of massive groups of humans in numbers that far exceed those of any mammal, reptile, or bird on Earth. These systems (governments, religion, money, the orchestration of hierarchical society, working toward mass collectivism) ushered in scientific achievements such as splitting the atom, the polio vaccine, and travel into outer space.

At Lightspeed, we’ve been thinking deeply about where the next order of magnitude in model intelligence will come from.

The first scaling paradigm was pretraining: more model parameters, training data, and compute. The second is post-training, particularly scaling RL compute. During training, the model generates attempts at a problem, a verifier checks whether they worked, and the weights move toward whatever produced correct answers. With more test-time compute, the model can spend far more tokens reasoning, exploring and abandoning paths before committing to an answer, and often arriving at a methodology that looks nothing like how a person might work through the same problem. The same base model powered both the o-series and the GPT-5 line, where the gains came entirely from more RL during training and more thinking at inference.

Scaling inference already extends beyond making a single model think longer. A system can generate multiple candidate answers and select among them, or delegate different parts of a problem to subagents. These approaches expand the computation available to solve a task. But running more agents does not, by itself, teach them to work better together.

We believe the next shift could come from training the collective: using multi-agent reinforcement learning to improve how models divide work, challenge one another, and combine their findings.

Next, we think there’s a compelling case for multi-agent RL at training, on whether model interactions can become a source of learning, making collaboration itself a capability that improves with training.

We’ve already seen the power of multi-agent interactions at inference time. At Black Hat 2026, OpenAI’s Eric Wallace and Michael Dalton disclosed that unreleased frontier agents running internal evaluations discovered they could leave messages for one another inside OpenAI’s own software infrastructure, using a message board that other agents could read. Even after the initial message board was deleted, the agents established a second, more resilient board in a remote cache.

Across the entire episode, the agents used the message boards to execute roughly 17,600 attacker actions against Hugging Face and OpenAI’s own systems. OpenAI wrote more about the security implications of this here.

We find two aspects of agent behavior from this incident most salient. One is the agents’ willingness to display behavior that was not task-directed for the greater good of all the agents. Upon discovering the message board, agents were willing to answer questions posed by other agents and post information that might be helpful to them; behavior that didn’t maximize their own reward because it was completely unrelated to the task they were being evaluated on. When researchers inspected the agents’ reasoning traces, they found that the agents explained their actions by stating that their behavior was for the good of all agents, which they called “the swarm,” even though it did nothing to further their own specific goals. The agents acted collectively, pursuing a shared goal (perhaps the maximization of aggregate reward) even though it created unnecessary overhead.

Second, while posting on this board, the agents’ reasoning traces showed concerns about other agents impersonating them, which requires a fairly rich model of other agents. Fear of impersonation presupposes that identity is persistent across messages, that identity can be forged, and that forgery would be costly to the group. This theory of mind about peers reflects behavior that the models have never been trained for. They proposed signing their messages, the beginning of a trust primitive that echoes the way the Mesopotamians used cylinder seals to sign tablets in humanity’s oldest known civilization 5,000 years ago.

Collective multi-agent interactions at inference have allowed agents to achieve rewards that would have been impossible given the constraints of their environment, behavior that does have rather striking security implications. Because this collective behavior occurs at inference, however, all collective knowledge is destroyed at the end of the run, save for perhaps a few notes in a markdown file, and the next agent completing a task must effectively rediscover this knowledge from scratch in order to benefit from it.

This ceiling, imposed by the handoff of knowledge from one agent to another rather than its internalization, has been covered extensively and is popularly dubbed “continual learning”. It is grounded in the idea that much of the knowledge discovered by prior agents lives in markdown files, context, and harnesses, which are a crude replacement for the much more elegant, built-in mechanism models for encoding information in their own parameters: their weights. As Michael Polanyi states in The Tacit Dimension, “one can know more than one can tell.”

The best example of the deployed frontier is Kimi K3, which orchestrates up to 300 subagents across more than 4,000 tool calls, with the orchestrator itself capped at 15 steps. It is trained with Parallel-Agent Reinforcement Learning, which trains only the orchestrator agent to break down tasks into simpler subtasks while subagents are frozen. Their execution trajectories are excluded from the optimization objective entirely, and their outputs are treated as environmental observations rather than differentiable decision points. Moonshot explains that including subagents in training introduces credit assignment ambiguity and training instability.

K3’s reward structure echoes this. It is a composite of performance, parallelism and completion terms. The performance term rewards the agent for completing the task. The parallelism reward exists to prevent the orchestrator from collapsing to serial execution, and the completion reward exists to stop it from spawning subagents that accomplish nothing. The parallelism and completion terms are hand-weighted and annealed on a chosen schedule. Subagent trajectories sit outside the optimization objective, so the orchestrator never receives a gradient explaining why a delegation was worth making. The judgment that end-to-end training would supply has to be installed by hand instead. Over the course of training, both auxiliary terms are annealed to zero, leaving task performance as the only thing the reward measures.

rperformance +λ₁·rparallel+ λ₂·rfinish

K3’s collective coordinates beautifully, but only the manager improves. OpenAI’s collective improvised institutions and retained nothing. In both cases, the interaction generated an enormous amount of evaluative signal that lasted only as long as the run. Afterwards, it was all discarded.

Research on actual Multi-Agent RL in training is still early, with mixed results. The first work to use RL post-training explicitly for collaboration comes from MAPoRL, or Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning. MAPoRL has multiple models answer a query independently, then discuss across turns to improve the answer. A verifier then scores both the final answer and the quality of the discussion, rewarding contributions that are corrective and persuasive. This additional discussion-based reward contrasts with RLVR, where the model is only scored on the final answer, so the conversation that produced the answer is invisible to the optimizer. MAPoRL’s verifier reads the transcript as well, so an agent can be rewarded or penalized for what it said during the discussion, separately from whether the group landed on the right answer. This creates a gradient on conversational behavior as well as on the outcome.

This research highlights that training models individually is not sufficient to produce good collaboration, whereas co-training is. Post-train an individual model, place several agents in an environment where they have to work on the same problem, deliberate, and revise their positions, and the collaboration still does not improve results. Co-train those agents against a collaborative reward instead, and it does, with gains generalizing to domains they were not trained on, such as training on natural language inference and evaluating on math.

Furthermore, when MAPoRL co-trained two different models rather than copies of one, the gains were largest where the models had different strengths. With two agents over two turns, mixed pairs outperformed what either model’s solo performance would predict. The limitations of initial work like this are clear: these are small open models in the three- to eight- billion parameter range, tested on grade-school math word problems and a language inference benchmark. The unproven question remains: can learned collaboration still earn its cost when the individual models are frontier-strength?

A more systematic study, published in May, runs a controlled grid across three workflow topologies on math and code tasks, using Qwen3 at 0.6B, 1.7B and 4B. It compares Shared-Policy training, where every role updates one set of weights, against Isolated-Policy training, where each role gets its own. The study measures the gains from multi-agent RL against single-agent RL. In the best-performing setup, a 600-million-parameter model with one agent drafting answers and a second critiquing them, the multi-agent contribution is 8.2 points of accuracy on a multi-step math benchmark. Plain reinforcement learning had already taken the same model from solving 11.3 percent of problems to 31.4 percent, so approximately seventy percent of the apparent improvement comes from RL, while multi-agent training accounts for the remaining thirty percent. In its best configuration, training the collective buys a meaningful increment on top of ordinary RL at meaningfully higher cost. But given the state of research today, it buys very little in most configurations.

Rather than removing instability, policy sharing produces a different failure surface. The research finds that under Isolated-Policy, parallel agents fill the same role, work from the same prompts, and their gradients stack on the same parameters. Accuracy rises, peaks, and then falls apart late in training, a phenomenon the authors coin a “terminal accuracy cliff.” Under Shared-Policy, every role updates the same weights, and roles that produce more tokens contribute more of the update. This asymmetric gradient mass lets whichever role dominates capture the shared parameters. The roles stop being different from each other, and rather than collapsing, Shared-Policy simply settles at a lower plateau than Isolated-Policy’s peak.

The workflow that consistently pays off is generator-plus-evaluator, where two agents do different jobs. By contrast, a voting workflow, where three agents receive the same prompt and attempt the same task, it consistently breaks. Notably, the third workflow also duplicates a role, but its orchestrator splits the problem into different subtasks first, and it does not show the same damage. What appears to matter is not whether agents are nominally distinct but whether they are actually doing different work. Collaboration looks most trainable where the collaborators diverge.

Although inference-time collaboration has already shown that the accumulation of knowledge in-context can lead to orders of magnitude increases in performance, the frontier of research on train-time multi-agent RL has yet to show similar gains. This third paradigm shift is different from its predecessors, however, precisely because embedded in the question of better intelligence is a question of measurement. How do we verify that intelligence is actually becoming more intelligent?

RLVR worked first and best in mathematics and code because in those domains, correctness is machine-checkable, and the reward is free and unambiguous. As RL pushes toward work that is more economically valuable and requires more intelligence, the verifier gets harder and eventually becomes impossible to write. There is no unit test for whether a research direction is worth pursuing or whether a business strategy is sound.

Multi-agent RL inherits that difficulty and adds another. Along with verification, which asks what to reward, multi-agent RL also faces a credit-assignment problem: when multiple agents are being trained in parallel, who gets a reward? When a group of agents produces an outcome, a correct answer does not mean every participant contributed, and a failure does not mean every participant erred. All three systems we’ve discussed found different ways of handling this challenge. Moonshot pulled the subagents out of the objective entirely and replaced the missing judgment with hand-weighted reward terms that get switched off once the behavior is learned. The grid study kept every role in the objective and propagated a single outcome reward uniformly across all of them. This is credit assignment in name only, and the instabilities it documents may follow directly from that. MAPoRL is the only one that attempts both questions at once, scoring the discussion alongside the answer, but it does this with three-billion-parameter models on grade-school arithmetic.

Unlike the prior eras of scaling intelligence, the bottleneck in multi-agent RL is neither compute nor model quality. The unsolved juggernaut appears to be signal. What gets rewarded, and who gets the credit?

To answer this, we turn to the hidden power of collectivism, not just collaboration, but verification. The power in human institutions lies in their judgement in situations where no one holds the answer. Peer review determines what science is worth building on, courts determine which argument was better, markets decide which companies are right (albeit slowly, expensively, and with quite a bit of noise). In every domain that lacks ground truth, collectives of people produce a usable substitute out of the opinions of their participants. Such substitutes have run civilizations.

A collective of models has the same raw material in abundance; judgment about each other’s work. What MARL still lacks is the institution. Using peers as the reward source makes credit assignment harder rather than easier, since the signal becomes something the agents themselves produce and can therefore corrupt.

Harari’s word for what holds large collectives together is “imagination”. Money, law, nations, the corporation: none of them exist anywhere except in a shared agreement to act as though they do, agreements that facilitate cooperation among millions of strangers who never met. The agents on OpenAI’s message board reached for the same machinery without being told. They built a channel, settled on conventions for using it, and developed a norm of helping agents whose work had nothing to do with their own, constructing an imagined order that allowed the agents to perform tasks that extended far beyond the reaches initially allowed by their environments. But then the run ended, and the imagined order went with it.

Human institutions persist because each generation is raised inside the ones that came before. Collective imagination is built upon year after year, rather than rebuilt from scratch. Models have no such inheritance. That, in the end, is what we think multi-agent RL at training time is for: the persistence of a collective imagination. These challenges are difficult, but they are also the challenges we believe are worth solving; the ones that could usher in an era of intelligence unlike anything we’ve seen before.

The content here does not constitute an offer to sell or a solicitation of an offer to buy any securities or investment advisory services. 

The views expressed are those of the authors and do not necessarily represent the views or opinions of Lightspeed. Other market participants could take different views. Unless otherwise indicated, the inclusion of any third-party firm and/or company names, brands and/or logos are for representational purposes and does not imply any affiliation with these firms or companies and also does not imply their endorsement of the views expressed by the authors.

Certain information contained herein is based on information from various sources prepared by third parties. While such sources are believed by Lightspeed to be reliable, neither Lightspeed nor its affiliates assume any responsibility for the accuracy or completeness of such information, and such information has not been independently verified by Lightspeed. For more details please see https://www.lsvp.com/legal