Nomad Bridge incident analysis

Tl;dr: Building a better crypto ecosystem means building a better, more equitable future for us all. That’s why we are investing in the larger community to make sure anyone who wants to participate in the crypto economy can do so in a secure way. In this blog post, we share lessons about the nature of the vulnerability, exploitation methodology, as well as on-chain analysis of attacker behavior during the Nomad Bridge incident.

While the Nomad bridge compromise does not directly affect Coinbase, we strongly believe that attacks on any crypto business are bad for the industry as a whole and hope the information in the blog will help strengthen and inform similar projects about threats and techniques used by malicious actors.

By: Peter Kacherginsky, Threat Intelligence and Heidi Wilder, Special Investigations

On August 1, 2022 Nomad Bridge suffered the fourth largest DeFi hack with more than $186M stolen in just a few hours. As we have described in our recent blog post, from the $540M Ronin Bridge compromise in March to the $250M Wormhole bridge hack in February of 2022, it is not a coincidence that DeFi bridges constitute some of the most costly incidents in our industry.

What makes the Nomad Bridge compromise unique is the simplicity of the exploit and the sheer number of individuals taking advantage of it to empty all stored assets piece by piece.

Vulnerability Analysis

Nomad is a bridging protocol supporting Ethereum, Moonbeam, and other chains. Nomad’s bridging protocol is built using both on-chain and off-chain components. On-chain smart contracts are used to collect and distribute bridged funds while off-chain agents relay and verify messages between different blockchains. Each blockchain deploys a Replica contract which validates and stores messages in a Merkle tree structure. Messages can be validated by either providing proof with the proveAndProcess() call or for already verified messages they can be simply submitted with the process() call. Verified messages are forwarded to a Bridge handler (e.g. ERC20 Router) which can distribute bridged assets.

On April 21, 2022 Nomad deployed a Replica proxy contract to handle processing and validation of users’ claims of bridged assets. This proxy would allow Nomad to easily change implementation logic while retaining storage across upgrades. As part of the proxy deployment, Nomad set initial contract parameters defined in the snippet below:

function initialize(

uint32 _remoteDomain,

address _updater,

bytes32 _committedRoot,

uint256 _optimisticSeconds

) public initializer {

__NomadBase_initialize(_updater);

// set storage variables

entered = 1;

remoteDomain = _remoteDomain;

committedRoot = _committedRoot;

confirmAt[_committedRoot] = 1;

optimisticSeconds = _optimisticSeconds;

emit SetOptimisticTimeout(_optimisticSeconds);

}

Notice the highlighted confirmAt map assignment which sets an initial entry for the trusted _committedRoot to the value of 1. The variable _committedRoot is provided as an initialization parameter by Nomad’s contract deployer. Let’s see what it was set to during the initialization:

$ cast run 0x99662dacfb4b963479b159fc43c2b4d048562104fe154a4d0c2519ada72e50bf --quick --rpc-url $MAINNET_RPC_URL

Traces:

[261514] → new UpgradeBeaconProxy@"0x5d94…aeba"

├─ [2160] UpgradeBeacon::fallback() [staticcall]

│   └─ ← 0x0000000000000000000000007f58bb8311db968ab110889f2dfa04ab7e8e831b

├─ [163890] Replica::initialize(1635148152, 0xb93d4dbb87b80f0869a5ce0839fb75acdbeb1b77, 0x0000000000000000000000000000000000000000000000000000000000000000, 1800) [delegatecall]

│   ├─ emit OwnershipTransferred(previousOwner: 0x0000000000000000000000000000000000000000, newOwner: 0xa5bd5c661f373256c0ccfbc628fd52de74f9bb55)

│   ├─ emit NewUpdater(oldUpdater: 0x0000000000000000000000000000000000000000, newUpdater: 0xb93d4dbb87b80f0869a5ce0839fb75acdbeb1b77)

│   ├─ emit SetOptimisticTimeout(timeout: 1800)

│   └─ ← ()

└─ ← 439 bytes of code

Interestingly the initialization parameter _committedRoot was set to 0. As a result the confirmAt map now has a value of 1 for a 0 entry that from April to this day:

$ cast call 0x5d94309e5a0090b165fa4181519701637b6daeba "confirmAt(bytes32)" 0x0 --rpc-url $MAINNET_RPC_URL

0x0000000000000000000000000000000000000000000000000000000000000001

On June 21, 2022, Nomad performed a series of upgrades to its bridging infrastructure including the Replica implementation. One of the changes included updates to the message verification logic in the process() function:

function process(bytes memory _message) public returns (bool _success) {

// ensure message was meant for this domain

bytes29 _m = _message.ref(0);

require(_m.destination() == localDomain, "!destination");

// ensure message has been proven

bytes32 _messageHash = _m.keccak();

require(acceptableRoot(messages[_messageHash]), "!proven");

// check re-entrancy guard

require(entered == 1, "!reentrant");

entered = 0;

// update message status as processed

messages[_messageHash] = LEGACY_STATUS_PROCESSED;

// call handle function

IMessageRecipient(_m.recipientAddress()).handle(

_m.origin(),

_m.nonce(),

_m.sender(),

_m.body().clone()

);

// emit process results

emit Process(_messageHash, true, "");

// reset re-entrancy guard

entered = 1;

// return true

return true;

}

The message verification flow now includes a call to the acceptableRoot() method which in turn references confirmAt map we mentioned above:

function acceptableRoot(bytes32 _root) public view returns (bool) {

// this is backwards-compatibility for messages proven/processed

// under previous versions

if (_root == LEGACY_STATUS_PROVEN) return true;

if (_root == LEGACY_STATUS_PROCESSED) return false;

uint256 _time = confirmAt[_root];

if (_time == 0) {

return false;

}

return block.timestamp >= _time;

}

The vulnerability appears in a scenario when fraudulent messages, not present in the trusted messages[] map, are sent directly to the process() method. In this scenario messages[_messageHash] returns a default null value for non-existent entries so the acceptableRoot() method is called as follows:

require(acceptableRoot(0), "!proven");

In turn, the acceptableRoot() method will perform a lookup against confirmAt[] map with a null value as follows:

uint256 _time = confirmAt[0];

if (_time == 0) {

return false;

}

return block.timestamp >= _time;

As we mentioned in the beginning of this section, confirmAt[] map has a null entry defined resulting in acceptableRoot() returning True and authorizing fraudulent messages.

Exploit Analysis

The exploit takes advantage of the above vulnerability by crafting a message which tricks Nomad bridge into sending stored tokens without proper authorization. Below is a sample process() payload in a transaction submitted by 0xb5c5…590e:

0x6265616d000000000000000000000000d3dfd3ede74e0dcebc1aa685e151332857efce2d000013d60065746800000000000000000000000088a69b4e698a4b090df6cf5bd7b2d47325ad30a3006574680000000000000000000000002260fac5e5542a773aa44fbcfedf7c193bc2c59903000000000000000000000000b5c55f76f90cc528b2609109ca14d8d84593590e00000000000000000000000000000000000000000000000000000002540be400e6e85ded018819209cfb948d074cb65de145734b5b0852e4a5db25cac2b8c39a

The Replica message has the following structure:

struct Message {

uint32 _originDomain,

bytes32 _sender,

uint32 _nonce,

uint32 _destinationDomain,

bytes32 _recipient,

bytes memory _messageBody

}

The recipient specific _messageBody contains transaction data to be processed by the _recipient. Nomad recipients accept several transaction and message types, but we will focus on the transfer type:

struct BridgeMessage {

uint32 domain;

bytes32 id;

uint8 type;

bytes32 recipient;

uint256 amount;

uint256 detailsHash;

}

Decoding the above payload illustrates how 0xb5c55f76f90cc528b2609109ca14d8d84593590e was able to steal 100 WBTC by submitting a specially crafted payload to bypass Nomad’s message checks.

In order to better understand the root cause of the exploit we developed a PoC to demonstrate it draining the entire token’s balance on the bridge in just a few transactions:

While writing a PoC we found it curious that attackers chose to extract funds in smaller increments when they could have drained the whole amount in a single transaction. This is likely due to the attackers not crafting bridge messages from scratch, but instead replaying existing transactions with patched receiving addresses.

On-Chain Analysis

Over $186M in ERC-20 tokens were stolen from the Nomad Bridge between August 1, 2022 at 21:32 UTC and August 2, 2022 at 05:49 UTC. The highest volume in stolen tokens were primarily USDC, followed by WETH, WBTC, and CQT. Within the first hour of the exploit, only WBTC and WETH were stolen, then followed by several other ERC-20s.

Source: Dune Dashboard

In analyzing the blockchain data, we see that there were various addresses piggybacking off of the original exploiters and using almost identical input data with modified recipient addresses in order to siphon off the same token for the same amount. Once the WBTC contract was mostly drained, the attackers then went on to drain the WETH contract, and so on.

Further analyzing the first attackers in block 15259101, we find that the initial two attacker addresses leveraged a helper contract to obfuscate the exact exploit. Unfortunately, within that same block, several indexes down another exploiter address seem to have struggled interacting with the helper contract and decided to bypass it — and publicly expose the exploit input data in the process. Other addresses in the same and latter blocks then followed suit and used almost identical payloads to conduct the exploit.

Following the initial exploitation, and due to the ease of triggering the exploit, hundreds of copycats joined a massive exploitation of a single contract. While analyzing the payloads of various future attackers, we found that there was not only the reuse of the same tokens being bridged over and the same amounts, but also that funds were consistently being “bridged” from Moonbeam just like the original exploit.

The attack happened in three stages:the vulnerability testing a day prior to the attack, the initial exploit targeting WBTC stored on the bridge, and the copycat stage involving hundreds of unique addresses. Let’s dive into each of these including partial return of stolen assets.

Vulnerability Testing

Throughout July 31, 2022, bitliq[.]eth was found to trigger the vulnerability using small amounts of WBTC and other tokens. For example, on Jul-31–2022 11:19:39 AM +UTC he sent a transaction to the process() method on Ethereum blockchain with the following payload:

0x617661780000000000000000000000005e5ea959686c73ed32c1bc71892f7f317d13a267000000390065746800000000000000000000000088a69b4e698a4b090df6cf5bd7b2d47325ad30a36176617800000000000000000000000050b7545627a5162f82a992c33b87adc75187b21803000000000000000000000000a8c83b1b30291a3a1a118058b5445cc83041cd9d000000000000000000000000000000000000000000000000000000000000f6088a36a47f8e81af64c44b079c42742190bbb402efb94e91c9515388af4c0669eb

The payload can be decoded as follows:

• Originating chain: “avax”
• Destination chain: “eth”
• Recipient: a8c83b1b30291a3a1a118058b5445cc83041cd9d (bitliq[.]eth)
• Token Address: 0x50b7545627a5162F82A992c33b87aDc75187B218 (WBTC.e on Avalanche)
• Amount: 0.00062984 BTC

This corresponds to 0.00062984 BTC transaction sent to the bridge on the Avalanche chain.

The payload was sent using the process() method as opposed to the more common proveAndProcess() and was not present in the messages[] map in the prior to execution in block 15249928 :

$ cast call 0x5d94309e5a0090b165fa4181519701637b6daeba "messages(bytes32)" "bc0f99a3ac1593c73dbbfe9e8dd29c749d8e1791cbe7f3e13d9ffd3ddea57284" --rpc-url $MAINNET_RPC_URL --block 15249928

0x0000000000000000000000000000000000000000000000000000000000000000

The transaction succeeded even without providing necessary proof by triggering the vulnerability in the acceptableRoot() method by supplying it with a 0x0 root hash value as illustrated in the debugger below:

Source: Tenderly Debugger

Messages not present in the messages[] storage can be validated using the proveAndProcess() method; however, since the address called process() directly they have triggered the vulnerability.

Interestingly enough, it seems that bitliq[.]eth was also likely testing the ERC-20 bridge contract an hour prior to the exploit and bridged over 0.01 WBTC over to Moonbeam. [Tx]

Initial Exploitation

Active exploitation started on August 1, 2022 all within the same block 15259101 and resulted in combined theft of 400 BTC.

All four transactions used identical exploit payloads with the exception of a recipient address as described in the Vulnerability section above:

0x6265616d000000000000000000000000d3dfd3ede74e0dcebc1aa685e151332857efce2d000013d60065746800000000000000000000000088a69b4e698a4b090df6cf5bd7b2d47325ad30a3006574680000000000000000000000002260fac5e5542a773aa44fbcfedf7c193bc2c59903000000000000000000000000f57113d8f6ff35747737f026fe0b37d4d7f4277700000000000000000000000000000000000000000000000000000002540be400e6e85ded018819209cfb948d074cb65de145734b5b0852e4a5db25cac2b8c39a

Some observations on the above:

• The first three addresses were funded by Tornado Cash and have been actively transacting with each other which indicates a single actor group.
• Unlike the first two exploit transactions, 0xb5c5…590e and bitliq[.]eth sent the exploit payload directly to the contract and without the use of flashbots to hide it from public mempool.
• bitliq[.]eth replayed an earlier exploit transaction in the same block 15259101 as 0xb5c5…590e indicating either prior knowledge of the exploit or learning about 0xb1fe…ae28 from the mempool.
• All four transactions used identical payloads, each stealing 100 WBTC at a time.

Copycats

In total, 88% of addresses conducting the exploits were identified as copycats and together they stole about $88M in tokens from the bridge.

The majority of copycats used a variation of the original exploit by simply modifying targeted tokens, amounts, and recipient addresses. We can classify unique payloads by grouping them based on contracts they call and unique method 4bytes invoked as illustrated below:

Based on our analysis, more than 88% of unique addresses called the vulnerable contract directly using the 928bc4b2 function identifier which corresponds to the process(bytes) method used in the original exploit. The remainder perform the same call using intermediary contracts such as 1cff79cd which is the execute(address,bytes) method, batching multiple process() transactions together, and other minor variations.

Following the initial compromise, the original exploiters had to compete against hundreds of copycats:

While the majority of valuable tokens were claimed by just two of the original exploiters’ addresses, hundreds of others were able to claim part of bridge’s holdings:

Below is a chart showing the tokens stolen over time in USD. It becomes apparent that the exploiters were going token by token as they were draining the bridge.

The Great Return

To date, 12% stolen from the Nomad Bridge contract has been returned — including partial returns. The majority of the returns took place in the hours following Nomad Bridge’s request to send funds to the recovery address on August 3, 2022. [Tweet, Tx]

Below is a breakdown of the funds returned, which includes ETH and various other tokens, some of which were never even on the bridge:

Funds continue to be sent back to the bridge’s recovery address, albeit more slowly in the recent days than when the address was initially posted:

The majority of returned funds appear to be in USDC, followed by DAI, CQT, WETH, and WBTC. This is notably different from the breakdown of the tokens exploited. The reason being that the initial original exploiters primarily drained the bridge of WBTC and WETH. Unlike later stage exploiters, these exploiters moved funds around with no intent to return them.

Interestingly, one of the original exploiters, bitliq[.]eth, has returned only 100 ETH to the bridge contract, but has begun cashing out the rest of their proceeds through renBTC and burning it in exchange for BTC.

Categorizing the “exploiters”

When assessing the Nomad Bridge exploiters, the attackers were categorized into the following buckets:

• Black hats: Those that don’t return funds and continue moving them onwards.
• White hats: Those that fully send funds back to the recovery addresses
• Please note that while we are using the term white hat for explanatory purposes here, the initial taking of the funds was not authorized and is not an activity we would endorse.
• Grey hats: Those that partially send funds back to the recovery addresses.
• Unknown unknowns: Those that have yet to move funds.

Approximately 24% of funds continue to sit untouched. We suspect these are either attackers waiting out the heat or shrewd degens holding out for a bounty from Nomad. However, the largest volume of funds has moved onwards. As of August 5, we estimate that ~64% has moved onwards.

To stay up to date with the latest in terms of the funds returned, check out this dashboard.

Delving Into the Blackhats

Of those funds that have moved onwards, we have identified several large rings of addresses that all commingle funds. In particular, one cluster of addresses seems to have amassed over $62M in volume. Interestingly, one address within this cluster was the first address to have conducted the exploit [tx hash].

To date, we primarily see these rings following one of the below patterns:

• MEV bot activity
• Commingle and hold on to wait out the heat
• Swapping funds and eventually returning a partial amount of funds to the recovery address
• Swapping funds and investing DeFi projects or cashing out at various CEXs
• Moving funds through Tornado Cash

Below is an example of how some addresses have begun moving funds through Tornado Cash, which as of August 8, 2022, is a sanctioned entity.

Beware of Scams:

Several white hats have already returned over 10% of funds to the bridge contract. However, this wasn’t without hiccups.

Originally, the Nomad team posted on both Twitter and on the blockchain the Ethereum address to send any exploited funds to

However, scammers cleverly followed suit and set up various fraudulent ENS domains to pose as the Nomad team and requested they have funds sent to vanity addresses with the same initial characters as the legitimate recovery address.

For example, below is a message sent by one of the scammers. Note the fraudulent recovery address, ENS domain, and also the 10% bounty off. Nomad has since offered that white hats claim 10% of exploited proceeds. [Tx]

Protecting Yourself

While most contracts are audited extensively by various blockchain auditors, contracts may still contain yet to be discovered vulnerabilities. While you may want to provide liquidity to a particular protocol or bridge over funds, here are some tips to keep in mind:

• When supplying liquidity, don’t keep all of your funds on one protocol or stored in the bridge.
• Make sure to regularly review and revoke any contract approvals you don’t actively need.
• Stay up to date with security intelligence feeds to track protocols you’ve invested in.

Coinbase is committed to improving our security and the wider industry’s security, as well as protecting our users. We believe that exploits like these can be mitigated and ultimately prevented. Besides making codebases open source for the public to review, we recommend frequent protocol audits, implement bug bounty programs, and actively work with security researchers. Although this exploit was a difficult learning experience, we believe that understanding how the exploit occurred can only help further mature our young industry.

References

• Exploit PoC by Peter Kacherginsky
• Dune Dashboard by Heidi Wilder.
• Initial Exploit Analysis by samczsun

Indicators

Initial exploiters:

Ethereum: 0x56d8b635a7c88fd1104d23d632af40c1c3aac4e3

Ethereum: 0xf57113d8f6ff35747737f026fe0b37d4d7f42777

Ethereum: 0xb88189cd5168c4676bd93e9768497155956f8445

Ethereum: 0x847e74d8cd0d4bc2716a6382736ae2870db94148

Ethereum: 0x000000000000660def84e69995117c0176ba446e

Ethereum: 0xb5c55f76f90cc528b2609109ca14d8d84593590e

Ethereum: 0xa8c83b1b30291a3a1a118058b5445cc83041cd9d

See Dune Dashboard for a complete listing of exploiter addresses, transactions, and live status of stolen assets.

---

Nomad Bridge incident analysis was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

9 August 2022 15:38