People who joined the crazy world of Bitcoin near the end of 2017, have been bombarded by the word SegWit as they peeked through the rabbit hole. The end half of 2017 was one the most important year in Bitcoin’s history. It concluded the biggest consensus battle ever played in the ecosystem, and SegWit was the product, along with top quality social drama. It was not just an event, it was a saga. A community-wide intense techno-political debate that raged for more than three years, and finally concluded with a split of the Bitcoin ecosystem. It not only divided the network, but it also ended with broken companies, relationships and is partly responsible for much of the reputed “toxicity” of Bitcoin twitter today. You can still find its aftershocks, and it all revolved around SegWit. If you missed that event in real-time you might find it difficult to recreate the impact of this episode. Few attempts have been made to archive the history and here’s a good one.
Because Bitcoin is “notoriously technical”, its often difficult to make up minds around the social drama, especially for tech-intensive episodes like the Block Size Debate. Thus taking a glimpse under the machine makes matters much more simpler. By understanding the upgrade we can better judge its implication, and your own exploration of these events can be more meaningful. In this piece we look under the hood and try to match SegWit the technical upgrade with it’s corresponding social drama.
Under The Hood
SegWit is an abbreviation for Segregated Witness, which was a protocol update first conceptualized in 2013 by Greg Maxwell and later developed into four BIP proposals (BIP 141, 142, 143 and 144) by Eric Lombrozo, Johnson Lau and Pieter Wuille, in 2015. the SegWit upgrade was motivated originally because of the malleability bug which was exploited on mainnet in October 2015. It was a substantial change compared to regular garden variety code update.
To understand the upgrade better, first we need a little peek into the generic bitcoin transaction structure, which is shown in the diagram below.
Think of a bitcoin transaction as one big abstract structure made up of different sub-structures. Each of these sub-structures can also hold smaller sub-structures or just raw data. When we say “structure”, think of them as imaginary mathematical objects that can contain formatted information. Kind of like regular excel table, formated to denote which cell contains what kind of data. Structures have semantic meaning which describes the usage of the data it contains. When these structures are sent through the internet, they are translated into binary (0/1) representation (aka Serialization). And after receiving such binary data from the internet, these structures can be recreated back in the local machine (aka Deserialization). In software development sometimes structures are also referred to as “objects”, and hence the term Object-Oriented Programming (OOP). It means programming with such kind of structured data objects. Though, depending upon programming language, there can be subtle semantic differences between “structures” and “objects”, for our purpose of understanding, they are the same things.
From Fig 1 we can see a transaction (Tx)structure is made up of four types of sub-structures.
- Transaction Version (raw data, 4 bytes)
- List of Inputs (a list of
TxInstructures, length varies) - List of Outputs (a list of
TxOutstructures, length varies) - Lock Time (raw data, 4 bytes)
of course, that’s a huge simplification, see the transaction structure in full details here, but for our purpose, it’s sufficient.
Each transaction (Tx structure) has one or more TxIns, which points to a TxOut structure from some earlier transaction, and one or more TxOuts of its own. These TxOut structures are abstracted in real life as bitcoins. What we all mean by “coins” are just these TxOut structures in a gigantic database called the “Blockchain”.
A successful spending of bitcoin means successfully adding a Tx into the blockchain (blockchain is a list of Blocks, Blocks are list of Tx structures), which has a TxIn such that, it correctly points to the previous TxOut (bitcoins being spent). If this Tx structure gets added “successfully” in the blockchain, then the bitcoins are considered “spent”.
To understand SegWit, we need to go one step deeper.
Peeling out further we can see TxIn and TxOut are made up of smaller sub-structures.
Input/TxIn —
The first two structures together (PrevTxId, TxIndex) does the work of pointing. PrevTxId is the ID of the past transaction in which our target TxOut lies. TxIndex is the index of the target TxOut in that past transaction.
The next structure is ScriptSig, which serves as a cryptographic proof that the creator of this Tx is the rightful owner of that target TxOut. This structure is associated with all the magical cryptography and smart contracting stuffs in the protocol.
Output/TxOut —
The Output structure is pretty simple, which has only two sub-structures, Value and ScriptPubkey. Value denotes a 64-bit integer, which is the amount of BTC (aka sats) locked into this TxOut. And you can think of ScriptPubkey as a mathematical lock that restricts who can spend this Txout. like ScriptSig, ScriptPubkey is also used for “crypto” stuffs.
Only the rightful owner (the person who possesses the private key, for the public key encoded in the ScriptPubkey) can create the correct ScriptSig. Only a Tx structure with the correct (ScriptSig, ScriptPubkey) pair will be considered as “valid” and will be added into the blockchain by the network, otherwise it will be rejected.
This is how a Bitcoin transaction works in a nutshell.
The only major change that SegWit made is to move the data from the ScriptSig structure into a separate Witness structure, added at the end of the TxIn. The new SegWit Txin structure looks as below.
The data inside this Witness (formerly inside ScriptSig) structure is used to verify the transaction, and canonically known in domains of cryptography as “witness data”, hence the name of the structure. The original position of this data, i.e the ScriptSig structure is simply left blank. As a result, non-upgraded nodes will read empty value from ScriptSig when the transaction follows new SegWit rules.
This might seem like a pretty trivial change at first glance, but from the “conservative” development perspective of Bitcoin, this is a massive change in the gears of the machine. There are of course a lot more nuances in the actual engineering and deployment of SegWit (refer the BIPs), but in a nutshell this is what segregation of witness means. In short, SegWit takes the data out of ScriptSig, and segregate it into a new Witness structure. Pretty simple.
Motivation
The only reason we are walking backward is (implementation without motivation), it’s easier to understand the motivation once we see the proposed change. Since the beginning of Bitcoin there was a very embarrassing bug in the protocol, known as the “Malleability Bug”. It was known since the early days, but wasn’t fixed because fixing it would require a very major protocol change (SegWit was one such proposed fix), and the developers didn’t find the risk/benefit ratio to be viable for deployment of such change.
The bug simply means that the ID of any transaction can be altered while keeping the transaction valid. This is because the signature data stored in the ScriptSig structure can be slightly modified without invalidating the signature itself. And the transaction ID is calculated by hashing everything inside the transaction, including ScriptSig. So if someone can alter the ScriptSig they can successfully alter the original transaction ID without invalidating the transaction (altered signature is still valid). And this is a major problem because consider the following scenario:
- Alice wants to pay Bob some BTC. Alice creates a transaction that has Alice’s signature in
ScriptSigand Bob’s public key inScriptPubkey. - Alice broadcasts the transaction. Bob receives the transaction in mempool and changes Alice’s signature slightly (the private key is not needed to make this change), which changes the transaction id, and publish the altered transaction to the network.
- If the altered transaction gets into the blockchain first, the original transaction will be rejected by the network as a “double-spend”. Bob successfully gets his money, but now can go back to Alice and say that the money never came.
- Alice, unaware of the alteration, looks into the blockchain; he cannot find his version of the transaction ID; he concludes that the payment didn’t happen, and pays Bob again.
This type of attack is very rare, and not very practical, but it did occur in 2015 in the mainnet. Following is a statistics from Bitcoin Magazine of the real-time attack as it was taking place.
But the biggest problem of this bug appears in the 2nd/offchain layer designs, namely Lightning Network, Sidechains, etc. It wasn’t possible to design these technologies without fixing the bug. A typical step of locking bitcoins into sidechains or in an LN channel involves creating a refund transaction that depends on a creation transaction. And the refund is created before the creation transaction goes into the blockchain. So a counterparty can easily malleate the creation transaction ID after mutually agreeing on the refund transaction, and publish the malleated version. If the malleated version is accepted in the blockchain, this will invalidate the refund transaction, and funds will be locked forever.
The concept of “segregation of witness” to solve this problem was first demonstrated by Blockstream engineers around 2015 in the Elements project. An implementation of Elements is available today as the Liquid Sidechain.
The Debate
The transaction malleability problem, even if not a major problem on its own, has a close tie with the scaling debate. It was the direct roadblock for implementing offchain solutions. And by the end of 2015, the scaling debate has become a public hot topic of discussion. It was understood from the very beginning that blockchains do not scale. In fact this issue was among one of the first public comments on the bitcoin white paper that Satoshi received back in 2008. And by the end of 2015, the community concluded that there was no other way around than to confront this issue. It’s either now or it’s too late. And thus the drama begins.
In terms of any blockchain, scaling is guided by two major parameters.
- Number of transactions per second (governs throughput)
- Size of the blockchain (governs decentralization)
Scaling up either means higher tx/sec or smaller blockchain size or both. And they have correlated tradeoffs between them, i.e, if you try higher tx/sec, that tends to bloat up the blockchain, and if you try reducing the size of the blockchain, that tends to reduce tx/sec, given everything else constant.
Even though the entire debate was very chaotic and messy, using the benefit of hindsight, we can summarise it into two major approaches to scaling.
Horizontal Scaling —
This means increasing the size of the blocks. If instead of X MB blocks we do n*X MB, tx/sec will naturally increase from Y to n*Y. But this straight forward approach of increasing tx/sec, will bloat up the blockchain and will affect the decentralization of the network. If a blockchain does millions of tx/sec but it is entirely centralized in terms of node operators, that’s not a valid Scale-Up solution. That’s the direct path towards “PayPal-fication”. Examples of these are all over the blockchain industry, and you don’t even have to look hard to find them. Invisible threat of centralized blockchains is slowly manifesting and the communities are realizing the ill effects from events like these.
Vertical Scaling —
This approach proposes to take the transaction traffic off the main network, and put them into auxiliary “higher-layer” networks like Lightning Network or Liquid (or any other sidechain) Network. The benefit is, this can provide arbitrary throughput in these auxiliary layers, while the base layer can be kept intact with its original property of decentralization. The tradeoff is in the trust model of these auxiliary networks. These networks are positioned along the spectrum of trust-minimization, and depending upon the specific design they can be anything from “adequately decentralized” to “completely centralized”. But the implication of these choices doesn’t percolate down the base chain. Even if one of these auxiliary networks fails, disruption is only limited at the higher layer. In a vertical approach, the system is envisioned as a technology stack architected in layers. Almost like the TCP/IP network stack.
The Dispute
These two approaches of scaling comes with their versions of tradeoffs. There can be good/bad judgment about these, but there is no right/wrong answer. It depends on the individual and their techno-political goal when it comes to ideas like Bitcoin. Moreover, these two approaches are antithetical to each other, i.e. if one plans to go horizontal, there’s no benefit in going vertical, and vice versa.
It is clear by now that a middle ground between these two approaches was impossible. SegWit was the upgrade that opened the door for vertical scaling. And it pissed the horizontal scaling guys, because it represents everything they don’t believe to be “Bitcoin”. The community couldn’t reach an agreement, which resulted in spectacular social drama in both the virtual and real world. It ultimately resulted in two parallel versions of the Bitcoin protocol, each taking their own approach. The horizontal group took the camp of Bitcoin Cash (BCH), who wholeheartedly rejected SegWit. Some of them even consider the current form of BTC with SegWit as “impure” (Muh Satoshi’s Vision!!).
The Climax
SegWit upgrade was being blocked by a conglomerate of major miners for well over a year. The code was lying dormant in Bitcoin codebase, and the miners refused to activate the upgrade (BIP9 requires 95% miner support to get any softfork locked in). The straw that broke the camel’s back was a private agreement between “industry leaders” in a meeting in New York in May 2017. Famously dubbed as the “New York Agreement”, it proposed a hard fork of Bitcoin on Dec 2017. Moreover the new protocol version proposed by NYA was incompatible with the dominant version in the market. And NYA had miner representation accumulating to more than 80% of the network hashrate. This was the first instance of major hashrate battle in the Bitcoin ecosystem. It was a clear attempt of coup by an Industry cartel and really became an existential threat for Bitcoin itself. If this could have been successfully done, it essentially meant the end of this apolitical, distributed consensus mechanism. In those summer months of 2017, The situation quickly went from chaotic to dire. A miracle was needed.
The rescue came from an anonymous entity shaolinfry in a form a two generic upgrade mechanism (BIP148 & 149) different from already existing BIP9. They were not even technical Bitcoin upgrades, they were a neat game theory trick, inspired by Nassim Nicolas Taleb’s concept of “Minority Rules”. Which states that a “determined economic minority” can shift an “indifferent economic majority”. With this idea shaolinfry called the community to upgrade their nodes and socially signal for BIP148. It quickly became a social meme, and was labeled “User Activated Soft Fork” #UASF. UASF allowed node operators in aggregate to force the miners to produce a certain kind of block. If the miners didn’t comply, the BIP148 nodes will reject those blocks and will not propagate them. If enough nodes start doing this, the non-compliant blocks will not make it to the blockchain and miners to lose mining fees. This forced the miners to upgrade for SegWit. UASF was the reaction by the rest of the ecosystem on the mining coup attack, and it was nothing short of a civil war.
UASF worked, and On July 20, 2017 SegWit reached locked-in status in the Bitcoin protocol, but 1st August 2017, the “judgment day” proposed in BIP148, is considered as the “Bitcoin Independence Day”. A more detailed timeline of this entire saga can be found here.
Beyond The Drama
Aside from all the socio-political mess this episode had created, there was a widescale consensus on SegWit from the technical community. SegWit was not only a fix, it was a brilliant upgrade. It can even be argued that every possible form of engineering efficiency was extracted in the design of SegWit. This was only possible because of the long time it took to flesh it out and get it activated.
SegWit not only solved the malleability problem, but it was also a blocksize increase. Because of the way transaction size and Blocksize calculation were modified, the theoretical max blocksize became 4 MB (from legacy 1MB blocks). At the same time each transaction consumed less percentage of the total block space, reducing their fees. And it was all done in a softfork compatible way.
There’s more. Because of the way the Witness structure was designed, it is now possible to make any future changes into Bitcoin protocol without a hard fork. This is probably the biggest upgrade SegWit made that changed the future trajectory of Bitcoin. Thinking about deep restructuring in the protocol became socially feasible for engineers now, as any such change can be deployed as a new Segwit Version. Currently we are on Segvit Version 0. The next proposed upgrade for Segwit Version 1 is a proposal package of Schnorr signature, Taproot and Tapscript. They will not require the entire network to upgrade simultaneously, and people can voluntarily opt in to the new system. The ecosystem can always reject a Segwit version, but that will not disrupt the whole network anymore. In a sense, a new door in the Bitcoin protocol development is opened because of SegWit, and we can now think of complex structural changes that were impossible earlier simply due to deployment disruption. Some even call it Bitcoin 2.0 after 1st August 2017, and we can safely assume that there will be very few hardforks in Bitcoin development from now on. A full list of technical benefits of SegWit can be found here.
But the real impact of “SegWit the saga”, goes far beyond than just technical upgrades. It gave us an important glimpse for the first time in the intriguing game-theoretic, social, political and economic moving parts of the Bitcoin machine, and many bitcoiners are still trying hard to sum up the “story in all of these”. For many, it forever changed Bitcoin’s outlook and possible future prospects.
“It’s not possible to deploy whatever you want on a consensus network, I could want to deploy a million features, but if other people don’t want to accept it, it’s not up to me. I can’t negotiate this on behalf of the network. It’s not up to me or anyone else.
.. It means that in the end, the people that run the validating nodes ultimately determine which transactions they want to consider valid. It demonstrated that, in the end, people can validate their own transactions, and there’s network incentive to join the network that’s going to give them the most value. I think we saw the dynamics play out that way,”
— Eric Lombrozo, 2017. source
Ultimately, this entire history showed us the resilience of this apparently chaotic and anarchic system. For many years of the scaling debate it seemed like a dead-end, and even after 2017, for many, “Bitcoin is captured/failed”. Yet the network exists, It produces blocks every 10 minutes, the supply is still capped at 21 million, and it will open its arm to anyone anywhere in need. It’s not up to you, me, or anyone else to decide on behalf of Bitcoin, its the network who decides. And this makes Bitcoin special from the rest of the blockchains. The network effect is strong enough that it can keep the system robust. And activation of SegWit gave us the first real-life test run of this decentralized governance philosophy.
Bitcoin 2.0
During my initial days of exploring the Bitcoin space, SegWit became almost a guide needle for self-verification. Almost every voices that were loud against SegWit deployment, ended up on the wrong side of the history. Bitmain, the biggest proponent against SegWit even got his leader thrown out of board and facing imminent bankruptcy. SegWit was simply the best possible middle ground and have incentives for every participant. Irrespective of either side of the scaling debate, if an entity was against SegWit, it could have been safe to assume that they had hidden incentives. And that generally ended up being the case. Bitmain was later caught with exploiting a vulnerability in the bitcoin mining algorithm which gave them almost 30% extra mining edge from the rest of the network in a hidden way, known as covert ASIC Boost, and that trick was incompatible with SegWit.
As of writing the average segwit transaction share per block is around 50%.
Even after two and a half years of SegWit activation many wallets do not provide SegWit support. Many big exchanges in the cryptocurrency industry lagged and are still lagging with SegWit adoption. Having legacy transactions are simply more expensive and creates more fees for users. Also future protocol upgrades like Taproot will require SegWit in wallets and exchanges for users to use it.
It’s very easy to identify your own SegWit usage. Segwit addresses start with bc1, whereas a legacy address starts with 1 or 3.
Legacy P2PKH Address : 1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2
Legacy P2SH Address : 3J98t1WpEZ73CNmQviecrnyiWrnqRhWNL
SegWit Bech32 Address : bc1qar0srrr7xfkvy5l643lydnw9re59gtzzwf5mdq
So are you still using legacy address? Demand your service provider to upgrade and hop on to Bitcoin 2.0.
