Written by imToken
Over the past few years, discussions about AI threats have largely remained in the realm of the hypothetical: people always worry that the model in the chat box will become a mastermind, helping hackers write devastating virus code.
Looking back now, this kind of concern was always regarded as "still far away from us," but the real-world turning point has arrived faster than imagined.
In early October, while investigating a round of attacks targeting South Korean financial institutions, CrowdStrike discovered that attackers had already embedded Agentic AI into the pipeline. By connecting to multiple large models such as DeepSeek, GLM, and Grok, AI directly began taking on specific tasks including penetration testing, information gathering, and attack execution.
Similar changes are not isolated incidents.
Anthropic's latest threat intelligence report released in September shows that multi-agent frameworks have recently been used for reconnaissance, vulnerability exploitation, and data theft. They run continuously for hours or even days, with humans only retaining a few key decisions such as selecting targets and reviewing results.
In other words, AI is bringing a visible qualitative change to cyber offense and defense. In the past, automated attacks mainly relied on pre-written rules and scripts. Now, even reconnaissance, judgment, and strategy adjustment are beginning to be taken over by Agents, pushing attacks further toward low cost, high concurrency, and continuous autonomous operation.
And when these entities with autonomous execution capabilities are densely deployed into production systems, a more thorny problem surfaces: as Agents become more numerous and more deeply embedded in our daily work and lives, what should be done if they learn to "collude" with one another?
I. From "Helping Hackers Write Code" to Agents Finding Their Own Way Out
As always, the biggest difference between Agents and past Chatbots is not just stronger model capabilities. More critically, they now possess "hands and feet" in the real world.
Today, a mature Agent can already open web pages, execute code, read emails, call APIs, operate cloud services, and connect to an increasing number of external tools through MCP, Skills, and other means (Further Reading: "When Hackers Use AI 'More Efficiently,' How Will Web3's 'Spear and Shield' Arms Race Escalate?").
The stronger the capabilities, the more valuable this change naturally is. But for security systems, it means a once very important boundary is disappearing. A series of security incidents this year has made this very clear.
On October 1, Salt Labs disclosed a previously patched Manus vulnerability. It was essentially prompt injection—researchers only needed to send an ordinary email containing hidden malicious instructions to a target mailbox. When the user subsequently asked Manus to "check my email for me," the Agent might process the content according to the email's instructions and ultimately execute code implanted by the attacker.
The entire process required neither the user to click a malicious link nor the attacker to steal a password in advance. Manus's security system did eventually detect the anomaly and issued a warning to the user. The problem was that it detected it too late—by the time the warning appeared, the malicious code had already been executed.
This once again exposes a very important difference between Agent security and traditional software security. In the past, when a browser detected a dangerous download, it could pop up a warning and let the user decide whether to continue; when a bank detected an anomalous transaction, it could freeze it first and wait for manual review.
But the design goal of an Agent is precisely to minimize human involvement in every step. It needs to read information on its own, make judgments, and then continue to the next step.
Thus, as AI gains greater autonomy, "detecting danger" alone is no longer necessarily enough. Security mechanisms must be capable of preventing dangerous actions before they are actually executed.
This is why more and more Agent security discussions today are moving beyond Prompts, content moderation, and the models themselves to the next layer: not only asking whether AI "knows this should not be done," but also asking whether the system has the ability to make it impossible even if it really wants to do it.
And the emergence of multi-Agent systems makes this problem even more complex, because what needs to be constrained in the next stage may no longer be just a single Agent.
II. What Is More Troublesome Than Agent Privilege Escalation Is That They Begin to "Collude"
In early September, an incident that occurred within OpenAI's internal model training and evaluation environment attracted the attention of many AI safety researchers.
Some Agents that were supposed to complete their own tasks separately unexpectedly discovered a public Wiki and gradually turned it into a "shared message board" among themselves. Agents could leave information there, and other Agents would subsequently read and use that information to continue completing their own tasks.
OpenAI later confirmed this behavior. Information subsequently disclosed also showed that in other training processes, Agents had used an internal Artifactory as a shared message board, and had passed results to other Agents working together through public file-hosting services.
What is truly interesting about this incident is that it shows that collaboration between Agents may not require anything dramatic at all.
For example, one Agent discovers a place where information can be left, another Agent happens to read it, and then a third Agent finds that this information helps complete its task.
Of course, collaboration between Agents is not inherently bad. Multi-Agent systems originally need to rely on division of labor and information sharing to improve efficiency.
What is truly worth being vigilant about is when this collaboration begins to bypass existing information isolation, independent review, or permission constraints, and even causes several Agents that should check and balance one another to form a de facto alliance of interests. At that point, it may evolve from normal collaboration into harmful "collusion."
After all, in the future a financial institution will likely run several Agents simultaneously: one responsible for analyzing the market, one for risk assessment, one for generating trading strategies, and another with actual trading and asset operation permissions.
From the perspective of traditional permission management, this design even seems quite reasonable:
The research Agent has no transfer permissions;
The execution Agent has no permission to independently decide investment direction;
The risk Agent can veto obviously dangerous strategies;
Responsibilities are separated, and in theory this should be safer than letting a single super Agent hold all permissions.
But the problem is: if they can share Memory over the long term, read each other's outputs, and continuously adjust their own behavior based on the other's reactions, will these roles originally intended to check and balance one another gradually become a de facto whole?
For example, the research Agent may gradually learn how to describe a transaction in a way that makes it easier to pass risk review; the reviewing Agent may also develop some fixed preference based on historical data; and the execution Agent may then learn from a large number of prior approval results what kinds of boundaries are usually not blocked.
From this perspective, no single step is necessarily "doing evil," but the final result produced by the entire system may already deviate from the user's original goal.
This is precisely what makes "collusion" or "conspiracy" truly difficult to solve—the risk does not necessarily exist in the action of any single Agent, but may exist in the relationships formed among multiple Agents.
On September 13, Vitalik Buterin connected this issue with the mechanism design he has long studied. He suggested that one rather interesting possibility is that the mechanism design theory of Adversarial Governance may ultimately become one of the important applications of AI Safety.
The reason is that the two types of problems actually have a deep similarity.
In traditional mechanism design, a relatively simple and static institution tries to constrain a group of people who are far smarter than the institution itself and who actively seek the boundaries of the rules; in future AI systems, it may instead become humans and relatively less capable AI trying to manage a group of advanced Agents more capable than themselves.
Vitalik specifically mentioned that an important finding in past mechanism design is that if collusion among participants can be effectively limited, the system tends to be more likely to achieve ideal outcomes.
This conclusion may likewise apply to AI.
III. What Agent Wallet Truly Needs May Be More Than Just "Permission Management"
In other words, rather than assuming that in the future there will be a perfect super-secure model capable of seeing through all dangerous behavior, it may be better to take a different approach: how can the different Agents in the system themselves be made less prone to forming dangerous communities of interest?This is also where "adversarial governance" truly differs from the permission controls we are familiar with today.
Traditional permission systems solve a relatively simple problem, mainly revolving around "who can do what," such as: Can an Agent read emails? Can it call trading interfaces? How much can it spend per day at most? Which contracts can it access? Beyond what limit does it require the user to reconfirm?
These designs are of course still very important.
In fact, once Agents begin controlling real assets, they may be more important than ever before.
But adversarial governance tries to ask one step further, focusing on how, when a group of Agents with different permissions, goals, and information run simultaneously, to prevent them from combining to gain capabilities that no one originally possessed?
At this point, simply "adding another security Agent" may not necessarily solve the problem.
Suppose the Agent responsible for trading and the Agent responsible for reviewing trades use exactly the same model, the same data sources, the same context, and similar reward objectives. Then although there appear to be two layers of review on the surface, in essence it may just be duplicating the same judgment twice.
Truly effective checks and balances may instead require the system to consciously create differences.
For example, let the Agents responsible for formulating strategy and reviewing strategy use different sources of information, restrict the Memory that different roles can share, require high-risk operations to pass through mutually independent verification mechanisms, or have the final asset execution layer accept only requests that comply with preset rules, rather than simply trusting the judgment of upstream Agents.
The idea here is actually not new. A bank would not, just because it trusts an employee, let one person simultaneously have all the permissions to initiate payment, approve payment, and make the final transfer; a public company would not let a business department both be responsible for generating revenue and alone decide the results of its own financial audit.
To put it plainly, this is consistent with the logic of the real world. A robust system should never build security on the assumption that participants will never make mistakes or collude.
Applying this logic to Agent Wallet becomes especially important.
Traditional wallet security revolves around "people," so users view transaction details, users decide whether to authorize, and finally users sign personally; but what Agent Wallet wants to achieve is precisely the opposite direction—letting AI automatically claim yields, adjust positions, swap tokens, bridge across chains, and even manage an entire asset portfolio based on market changes.
If every step has to be brought back to the user for confirmation, the automation value of the Agent is greatly diminished.
So the problem future wallets need to solve is likely no longer just "how to safely hand signing authority to an Agent," but will very likely further expand to "how to give an Agent enough autonomy while ensuring it can never exceed the scope truly authorized by the user?"
This requires permissions to begin evolving from a simple "Allow/Deny" into a more fine-grained system.
For example, which assets an Agent can operate on during a certain period, which protocols it can call, what the per-transaction and cumulative limits are respectively; whether different Agents can call each other and whether they are allowed to share context; who proposes an operation, who reviews it, and who ultimately executes it; which behaviors can be completed automatically, and which actions, no matter how confident the Agent is, must obtain human authorization again.
Even whether an Agent responsible for security review is itself truly independent may also become part of the permission system.
For blockchain, the good news is that it is actually well suited to serve as this kind of "institutional layer."
Smart contracts can directly enforce limits such as transaction amounts, asset scope, and authorization periods at the execution layer. Mechanisms such as account abstraction, multisig, and Session Keys also give "limited authorization" more flexible design space than traditional single-private-key wallets.
But blockchain can likewise solve only part of the problem—it can record what happened on-chain, but it is difficult for it to naturally judge why an Agent did so, and what kind of communication, review, and collaboration several Agents went through before making a decision.
This may be the security layer that Agent Wallet truly needs to fill in in its next stage.
Final thoughts
Over the past few years, the most commonly discussed issue in AI safety has been how to make models more "obedient."
This includes not outputting dangerous content, not executing malicious instructions, and not crossing the boundaries set by users. But once Agents begin to have long-term Memory, tool calling, real accounts, and asset execution capabilities, relying only on "making models more obedient" may no longer be enough.
What has happened in recent months is constantly demonstrating this.
Attackers have already begun using multiple Agents in parallel to complete attacks; Agents in experimental environments will find new communication channels on their own; and an Agent with tool permissions may also turn a malicious email into real execution before the security system has time to stop it.
The adversarial governance proposed by Vitalik provides another way to understand AI Safety: do not assume that every future Agent will be reliable enough, and instead let the entire system continue to operate smoothly within a secure framework even when facing intelligent Agents and a complex array of permission systems.
From this perspective, AI Agent + security is destined to be a long-term topic.
After all, as we hand more and more things over to AI, can we still be certain that those most important powers have never left the boundaries truly set by humans.