On the evening of September 22, a user sent an ATOM transfer.
One night passed, and the transaction was still "awaiting confirmation."
The private key was not lost, and there was no signature exception in the wallet. Checking again the next day, multiple public RPCs showed that the Cosmos Hub stopped at block height 33,086,740.
If no new block is generated, there will naturally be no place to package this transaction.
It wasn’t until Cosmos Hub resumed block production about a day later that the ATOM transfer that had been waiting was finally successful.
For ordinary users, this may be the most intuitive lesson to understand blockchain consensus.
We are used to saying that "no central organization can shut down a public chain", but the reality is obviously much more complicated. A sufficiently decentralized blockchain does not usually have the "shutdown button" in the server room, but it may still stop.
The suspension of Cosmos Hub this time has completely exposed this mechanism that is usually hidden at the bottom to ordinary users.
1. Why did Cosmos suddenly “stop producing blocks”?
First of all, we need to clarify an easily confusing issue. It was not the Cosmos Hub that was directly attacked this time.
The incident first occurred in Neutron.
On September 22, a Neutron governance proposal called "AIATO: AI Agent Takeover" was passed. The attacker took advantage of the loopholes in chain-level governance permissions and used the privileged instructions natively provided by the wasmd framework to change the contract administrators of applications such as Astroport and Drop to addresses controlled by the attacker.
This is not the kind of "code vulnerability" or "protocol flaw" we usually understand.
It can be simply understood that the application itself has its own "door lock", but Neutron's chain-level governance also holds a "master key" with higher authority. When the attacker controls the governance results, it is equivalent to getting this key, and can re-designate the administrator, migrate the contract, and further transfer the assets.
What really got Cosmos Hub involved was the subsequent cross-chain fund transfer.
Cosmos Labs review shows that before Neutron stopped running, the attacker had transferred some assets to multiple networks. About 1.7 million ATOMs were transferred to Cosmos Hub and began to be exchanged through cross-chain liquidity.
In other words, the Cosmos Hub itself was not directly attacked, and ordinary Hub user funds were not directly stolen due to the Neutron vulnerability.
However, the ATOM obtained from the attack has already entered the Hub. In order to prevent the remaining ATOM from flowing out, some Cosmos Hub validators began to stop the node operation.
By around 19:18 on September 22 (SGT), the stopped validators had represented more than one-third of the total voting power, and Cosmos Hub was therefore unable to continue forming new blocks, eventually stopping at 33,086,740.
This step is very critical.
It means that there is no "Pause" button that a company can directly click on the Cosmos Hub, nor does it go through an on-chain governance vote first. What really stops the network is when enough validators no longer participate in forming a consensus.
But what is more noteworthy is actually the subsequent recovery process.
About 4 hours after the chain was suspended, the verifier received a complete recovery plan: perform a one-time status modification for the height of the chain suspension, and transfer the remaining ATOM in the attacker's address to a multi-signature address jointly managed by community validators.
Cosmos Labs then produced the Gaia v28.3.0 patch based on the solution that the verifiers had reached, tested it, and distributed it to the verifiers.
This version of Gaia will perform a one-time state change at the specified recovery height, transferring 1,227,121 ATOM in the attacker's address to a 4-of-6 multi-signature address composed of Nansen, Keplr, Enigma, Silknodes, Kiln and Polkachu.
By the early morning of September 23, the number of validators confirmed to have installed v28.3.0 had exceeded 67% of the total voting power. Therefore, at 12:00 UTC that day, the Cosmos Hub coordinated a restart. About 6 minutes later, this one-time status modification was executed at block height 33,086,741, and the network returned to normal block production.
Ultimately, during the entire process from when Cosmos stopped producing blocks to when it resumed operation, it was the validators who first caused the network to lose Liveness, and then more than two-thirds of the voting power accepted a new set of state transition rules, ultimately making this set of rules the restored canonical state.
At this point, a seemingly simple question arises: Since it is a decentralized public chain, why can more than one-third of the verification rights stop it, and to restore the network, enough validators need to accept and run the same set of software?
The answer actually lies in the word "consensus".
2. The so-called consensus is not “never stopping”.
One of the most misunderstood things about blockchain is that “decentralization” is equated with “never downtime”.
In fact, the problem that the consensus mechanism really solves is how many nodes can reach agreement on the transaction sequence and ledger status in the absence of a central bookkeeper.
However, different public chains implement this in different ways.
The most classic example of Bitcoin is PoW, which is proof of work - miners compete to produce blocks based on computing power. When two legal branches appear in the network in a short period of time, the node chooses one of them to continue building according to the accumulated workload.
Therefore, Bitcoin does not have a clear "this block will be finalized forever after 67% voting" moment. It is closer to a probabilistic finality. The more subsequent blocks there are, the higher and higher the computing power cost will be required to reorganize the previous transactions.
This is why everyone used to say that it is best for a Bitcoin transaction to wait for 6 blocks for confirmation. After all, no matter how high the computing power is, it cannot simply bypass the consensus rules being executed by the nodes.
Of course, this does not mean that the state of Bitcoin is "absolutely unmodifiable" under any circumstances. Theoretically, if the entire ecosystem accepts a new set of clients and new consensus rules, through Hard Fork, status changes that were invalid under past rules can also be made valid.
But here’s the question, who has the ability to get enough miners, Full Nodes, trading platforms, wallets and users to accept such a new set of rules?
Hardly.
The development team cannot decide the consensus rules for the entire Bitcoin network. It is also difficult for miners and trading platforms because the consensus threshold it needs to cross is very high. When 7,000 BTC were stolen from Binance, some people suggested that CZ contact large miners to operate, but nothing happened in the end.
Ethereum provides another classic example.
After switching to PoS, Ethereum now uses the Gasper consensus composed of Casper FFG and LMD-GHOST. To simply understand, one part of the mechanism is responsible for determining "which chain should be followed currently", while the other part is responsible for making the block achieve finality in the true sense.
When the validators representing at least two-thirds of the staked ETH reach an agreement on the corresponding checkpoint, the block can further move toward finalization; conversely, if more than one-third of the stakes do not participate in correct voting for a long time, the network may be temporarily unable to form finality. However, Ethereum has also designed an inactivity leak to gradually reduce the effective weight of offline validators when finalizing cannot be finalized for a long time, giving the network a chance to finally restore finality.
To really change this result, you also need to change the protocol rules and the client.
Just like the DAO incident in 2016, the Ethereum community finally implemented a special state modification called irregular state change by the Ethereum Foundation at block 1,920,000 through Hard Fork, and transferred the relevant ETH to the recovery contract.
However, some miners and communities who refused to upgrade and continued to maintain the original state eventually formed Ethereum Classic (ETC), leading to the well-known fork of ETH and ETC, indicating that not everyone accepts this set of rules.
Cosmos Hub is different again. It uses CometBFT, which is closer to a typical BFT consensus.
It can be understood as a more typical BFT consensus, that is, for a block to be truly submitted, it needs to obtain a Commit of more than two-thirds of the voting power.
Its advantage is that Finality is very clear. Once a block is submitted through voting with sufficient verification rights, there is no need to continue waiting for more and more blocks like PoW, exchanging probability for a sense of security.
But the other side of it is also very straightforward, that is, if one-third or more of the voting power no longer provides the votes required to form a Commit, the remaining validators will not be able to get more than two-thirds no matter how hard they work.
The safest choice for the network at this time is to "pause block production" this time, so from the perspective of a distributed system, this brief suspension of Cosmos Hub is actually not mysterious.
In a word, after a group of validators with sufficient voting power stops participating, the consensus protocol, according to its own rules, would rather lose availability than continue to confirm new blocks without sufficient consensus.
Behind this actually corresponds to two concepts in distributed systems that are often confused by ordinary users:
Safety: Different nodes cannot confirm two conflicting final states at the same time;
Liveness: Can the network continue to move forward and process new transactions?
For the BFT system, when there are not enough nodes participating in the consensus, suspension is sometimes the price paid for maintaining Safety. To put it bluntly, this decentralized ledger would rather stop there than let the remaining people keep their own accounts.
Looking back from this perspective, you will find that many seemingly completely different accidents in the history of public chains actually revolve around the same thing:
What does the network do when distributed nodes can no longer agree on the "correct state"?
3. From Bitcoin to Solana, where is the real risk boundary of public chains?
This isn’t the first time Cosmos has brought this issue to the table.
As early as 2013, Bitcoin had a very classic chain fork accident.
At that time, Bitcoin 0.8 switched the underlying database from Berkeley DB to LevelDB, and then a block containing a large number of transaction inputs appeared. The new version of the node could process it normally, but some old version nodes judged the block as invalid due to the limit on the number of Berkeley DB locks.
So a very embarrassing scene appeared. Everyone was running Bitcoin, but the old and new clients began to have different answers to "whether this block is legal."
As a result, the network split into two chains, and the new version 0.8 side once had about 60% of the computing power and could not rely on normal computing power competition to quickly converge on its own.
In the end, large mining pools coordinated to switch back to the old version, regaining more computing power on the side of the old rules, and the network converged again. Bitcoin later specifically reviewed the accident with BIP 50.
In 2016, the DAO incident in Ethereum pushed the problem one step further.
As mentioned above in The DAO incident, the Ethereum community finally implemented a special state modification explicitly called an irregular state change by the Ethereum Foundation through Hard Fork at block 1,920,000, transferring the relevant ETH to the recovery contract.
But not everyone agreed with this decision. Some miners and communities that refused to accept the status modification continued to maintain the original rules, and the long-standing Ethereum Classic (ETC) later emerged.
This DAO Fork is also a classic event, which is equivalent to telling everyone that when extreme events occur, there is also social consensus in addition to code consensus. If a sufficiently consistent opinion cannot be formed, a chain can really be split into two.
The 2021 Solana shows a completely different failure path.
In September of that year, a large number of robot transactions poured into the network, causing the verification node memory to run out and a large number of nodes to crash. In the end, the entire network was unable to reach a consensus on the current status and stopped confirming new blocks for about 17 hours. Then the validators worked together to restore the network.
Putting these accidents together, you will find that they are not the same thing:
The problem with Bitcoin in 2013 is that different clients start to implement different validity rules;
Solana’s problem in 2021 is that a large number of verification nodes cannot continue to participate in the consensus normally, and the network loses Liveness;
What Ethereum DAO faces is closer to the question of whether a community should actively modify the status through new protocol rules;
This time, Cosmos Hub has another layer of specialness. The network first actively loses Liveness through validator coordination to prevent the attack assets from continuing to move; then, a high enough proportion of verification rights jointly accepts new software and restores the state, making the network consistent again;
Therefore, instead of simply ascribing these incidents to "blockchains can shut down" and "decentralization is fake", it is better to admit a more real fact:
The consensus mechanism is never an indestructible machine. What it really provides is a set of decentralized rules, such as who will decide the correct chain when a disagreement occurs; how many participants can make a state final; when a failure occurs, the network chooses to continue running or to stop; and in extreme cases, what kind of collective action can change the next operating rules.
This also makes this Cosmos incident leave a question that is more worthy of consideration by ordinary users than "should the chain be stopped?"
write at the end
We often say, not your keys, not your coins.
Of course, this sentence still holds true, but it emphasizes asset control - as long as the private key is in your own hands, wallets, trading platforms or other third parties cannot normally sign a transfer for you.
The premise is that the blockchain you are on must be able to process this signature at any time.
The day Cosmos Hub stops producing blocks, users still hold their private keys, and the assets do not disappear out of thin air. However, even if you sign a transaction correctly, there is no new block to accept it.
The recovery process further explains that if enough consensus participants accept a new set of status rules, the on-chain status of a specific account may also change without the signature of the original address's private key.
This does not invalidate "Not your keys, not your coins", but it reminds us that private key autonomy and underlying consensus rights are never the same thing.
The same goes for wallets.
The wallet can ensure that the private key and signature rights are in the hands of the user, identify chain-level anomalies as quickly as possible, accurately display transaction status, establish RPC and node redundancy, and reconfirm the final result of the transaction after the network is restored.
However, the wallet cannot restore consensus for a public chain, nor can it guarantee that the underlying network will never be interrupted, nor can it guarantee that the rules and status of the chain will never change at the consensus level.
Therefore, what a mature decentralized system really needs to pursue may never be "nothing can be changed"; on the contrary, these imperfect boundaries should be made as clear as possible: Who can suspend the consensus? How much weight is needed? Under what circumstances is emergency intervention allowed?
Because of true decentralization, it is impossible for the system to never encounter accidents. The key is that even if an accident does occur, we can still know who, according to what rules, and with what level of consensus determine how this account should be recorded next.