P2P (Peer-to-Peer) networks, also known as point-to-point networks, are internet systems that rely on user groups (peers) to exchange information without a central server (Figure 2-1). They are a type of distributed network. Generally speaking, the status of each node in this network is equal. Unlike centralized networks with a central server, such as Client-Server (C/S) systems (Figure 2-1), each node in a peer-to-peer network acts as both a client and a server. Nodes communicate information through their connections, sharing resources they possess (such as disk storage space, network bandwidth, processor usage, etc.) to provide services and content. Therefore, when new nodes join the network, the overall capacity of the system increases accordingly.
Compared to C/S network models, P2P networks are particularly suitable for file sharing: in a C/S structure, resources are stored on a central server, and as more users request downloads under fixed bandwidth, the average data transfer speed for each user slows down. In contrast, in a P2P network, many nodes store copies of the same file, allowing users to download it simultaneously from multiple nodes. Additionally, files that have already been downloaded can be uploaded to other nodes that are downloading, resulting in faster speeds as the network grows. P2P networks make full use of the bandwidth of other peer nodes in the network, rather than just relying on the bandwidth of the file source node.
The success of file sharing has made P2P networks very popular, but since most shared files are popular music and movies, issues of copyright infringement have also plagued P2P networks. In a typical P2P network, data can be copied freely, and copies can be stored arbitrarily. However, assets cannot be copied freely or exist in multiple copies. The Bitcoin project created by Satoshi Nakamoto retains the "distributed" characteristics of P2P networks and also addresses the issue of asset transfer within P2P networks: assets flow between different addresses rather than being simply "copied"; miners verify the destination of assets during the transaction process. The following will provide a detailed explanation of the P2P network in Bitcoin, which may inspire solutions to copyright protection and other issues using blockchain technology.
P2P Network in Bitcoin#
In the Bitcoin network, each node can randomly connect to other nodes, and nodes can join or leave the network at any time. The total number of nodes in the entire network is uncertain. When nodes update information, they are not in real-time sync but only need to reach consensus within a certain timeframe. In other words, the exit or crash of some nodes does not lead to the paralysis of the entire network; the joining and leaving of users do not significantly impact the overall network.
Information exchange between nodes is primarily reflected in transaction broadcasting. Transaction broadcasting is achieved through flooding. Specifically, when a node initiates a transaction, it informs all neighboring nodes of the transaction information. The neighboring nodes will verify whether the transaction can proceed based on the historical transaction information they store. If the verification passes, this transaction information will continue to be relayed to the next batch of neighboring nodes. Transaction information spreads and propagates through the nodes like ripples. When the transaction information received by a node overlaps with the information in that node's transaction pool, it indicates that this information has been propagated once, and the broadcast will then terminate. Since each transaction has a unique hash value, it is very convenient to check for duplicate transaction information using the hash value.
It is important to note that in a P2P network, due to bandwidth and other reasons, the transmission of information is often delayed. Therefore, the contents of transaction pools in different nodes may vary slightly. If someone initiates a double-spending attack and both payments reach the same node, the honest node will only keep one transaction and will not broadcast the other. Eventually, different nodes in the network may have discrepancies in recording different transactions for a short period, but this is not a problem. Over time, the consensus mechanism ensures that only one transaction will be recorded, making double-spending attacks impossible.
In brief, the Bitcoin project establishes a cash payment system within a P2P network, primarily relying on the following factors: the equal status of nodes; transaction information is propagated among nodes through flooding; nodes check whether transactions are valid; and the consensus mechanism determines the legitimacy of transaction information.
Limitations and Trade-offs of P2P Networks#
The advantages of P2P networks, such as fault tolerance, scalable transmission speed, and data security, come at the cost of low transaction processing capability in blockchain projects. Currently, in the fierce competition among public chains, many projects are showcasing their transaction processing capabilities (for example, claiming to handle over ten thousand transactions per second), which also indicates that this is a problem that existing blockchain technology still needs to solve. In fact, as more and more nodes are added to the network, the transmission delay of information between nodes gradually accumulates, and the time required for information to propagate throughout the entire network increases. Therefore, P2P network projects must balance low transaction throughput and centralization. When a small number of "super nodes" are set up to verify transaction information, the efficiency of processing transaction information can be improved, but this also makes the network more centralized. In a network where all nodes have equal status, the verification of transactions by all nodes will result in a certain degree of redundant labor and resource waste.
Blockchain technology is exciting and is closely related to its decentralized characteristics, which are largely based on P2P networks. P2P networks are a very balanced concept, but they also require a certain amount of resources as a cost. A trade-off must be made between a balanced network and improved work efficiency. Establishing a high-efficiency peer-to-peer network will require continued advancements in communication technology.
Blockchain technology proposes a distributed ledger architecture that eliminates third-party institutions from the system, allowing people to trade directly with each other. The solution offered by blockchain is to have all users possess a ledger, with all users participating in the accounting process. However, this also raises a question: how can it be ensured that all users have the same ledger? How can the consistency of ledger information be guaranteed?
In blockchain, transaction information is broadcast to the entire network, and each node can receive transaction information. Thus, the consistency issue of ledger information essentially becomes a "uniqueness" issue. As long as a rule is designed to ensure that only one unique piece of transaction information can be retained through filtering, it can guarantee that each user records the same information.
The "block" and "chain" are the data structures that achieve this uniqueness. A block stores transaction information over a period of time and is essentially a packaging of transaction information; in Bitcoin, a block can store about 3,000 transaction information entries. Once this block is confirmed, all 3,000 transactions are confirmed together. If transaction information is not packaged, confirming each transaction would require frequent confirmation operations, which would be inefficient.
Each block contains address information pointing to the previous block, forming a "chain" that links the latest block to the genesis block. Different consensus mechanisms provide different solutions for generating new blocks; sometimes, it is possible to generate two (or more) blocks with different contents within the same time frame, which is known as a "fork." After different blocks are created, new blocks will be generated for each, extending each chain. However, generally speaking, the speed of chain extension is different. The vast majority of blockchain projects follow the rule of "selecting the longest chain as the main chain," which ensures that even if a fork occurs, there will always be a recognized "main chain" after a certain period.
Since the longest chain is unique, all users will record the same chain in their local databases, which guarantees the uniqueness of the ledger and resolves the consistency issue of the ledger.
Additionally, the chain structure brings another benefit. All blocks are connected through the "chain," forming a tightly-knit whole. If a hacker wants to tamper with the content of a historical transaction, they would need to alter the block containing that transaction; hackers cannot create a new block and directly replace it but must replace the entire chain from that block to the latest block (in Bitcoin, this means starting to mine again from the block that needs to be modified, continuously mining all subsequent blocks, and ultimately surpassing all other miners in progress to create a new longest chain), which is costly. This helps to avoid attacks such as transaction tampering.
However, this data structure still has problems: low data processing capability. In blockchain, to avoid conflicting transaction information (no continuous forks allowed) and to ensure the consistency of the ledger (a unique chain must be selected), blockchain adopts the longest single chain structure. Since only one block can be added at a time, the propagation and confirmation of block information in the P2P network take time, and the capacity of blocks is limited, which results in an upper limit on the transaction information that can be recorded over a period. It is evident that "low data processing capability" is actually the price paid to meet consistency requirements. Currently, the Bitcoin blockchain can only process about 7 transactions per second on average, which is significantly lower than centralized electronic payment systems.
To solve the problem of low data processing capability (small transaction throughput), an important idea is to allow multiple transactions to be processed simultaneously in parallel. Sidechain technology can connect different blockchains by locking and unlocking funds on the main chain, expanding the space for transaction processing. The idea of sharding is to divide users into different shards, where transactions within each shard can be independently verified and processed in parallel, while transactions across shards require additional processing. In sidechain and sharding technologies, the main chain still exists, and both limit the flexibility of transactions (such as freezing funds, restricting transaction parties, etc.) to ensure consistency of the ledger while ensuring security.
On the other hand, DAG (Directed Acyclic Graph) is an exploration of another form of data structure. In general blockchain projects, all nodes save the same information; however, projects that adopt DAG technology allow each node to store different information. In DAG, blocks can be generated at any time, and a block connects to multiple parent blocks. This way, everyone can keep a ledger at any time, significantly increasing the speed of recording transaction information.
However, since multiple blocks can be generated simultaneously, and all are valid, DAG cannot guarantee consistency with a "unique longest chain." In this regard, some projects ensure the consistency of the ledger on DAG through "temporal" means. Specifically, in DAG, a new block randomly selects two recent blocks to connect to, while verifying the transaction information of all blocks connected to it. Blocks that have undergone multiple verifications have a very low probability of conflicting transaction content and can be considered confirmed transaction information. In this scheme, the verification of consistency relies on the extension and growth of the block network.
Other projects ensure ledger consistency through "full connectivity," meaning that each new block connects to all previous blocks and verifies all prior transaction information. Some projects ensure consistency through "ordering," confirming new blocks through recursive voting among blocks, etc.
DAG improves throughput, but "consistency" remains a complex problem that needs to be solved. Currently, solutions to this problem require some trade-offs: it may delay the exact verification time of transaction information; or it may require extensive network communication between nodes, making the actual transaction speed still to be observed.
Ultimately, the "consistency" problem of distributed ledgers is a balancing issue. It can be said that "consistency" is a system goal, and achieving this goal requires corresponding resources, so sacrificing transaction speed, limiting transaction flexibility, delaying confirmation times, or increasing transmission requirements across the entire network are all adaptive choices made under different system conditions. It is believed that the various technologies mentioned above will be further explored and validated in different application scenarios.
In a P2P network, all nodes have equal status, and information is transmitted between nodes; in a distributed ledger, it is necessary to ensure that the content stored among nodes is the same. The "consensus mechanism" is a solution that ensures that the content of each node is the same. In a P2P network, there may be situations where node performance declines or network congestion occurs, leading to the propagation of incorrect information in the system. Therefore, when designing consensus mechanisms, it should be assumed that unreliable nodes exist in the system. From an algorithmic perspective, the design of these mechanisms is essentially based on economic interest games, ensuring that the rewards for honest accounting exceed the rewards for malicious destruction, thus guaranteeing the cooperation of the vast majority.
The Boundaries of Technology#
When designing a new product, one must understand the current boundaries of technology: which technologies are fully usable and which still need to wait for a while. For technologies that need to wait, one must consider them later. Of course, science and technology are somewhat different; scientific research can provide theoretical limits, while engineering design focuses more on how to do the best within the approximate boundaries that are likely to occur. This is similar to an optimization problem, where one needs to know the given constraints to solve it correctly.
Regarding consensus mechanisms, previous research has provided two important boundaries:
● The Fischer-Lynch-Paterson theorem: It proves that in a multi-process asynchronous system, as long as there is one unreliable process, there cannot be a protocol that guarantees that all processes reach consensus within a finite time;
● The CAP principle: A distributed computing system cannot simultaneously ensure consistency, availability, and partition tolerance; designs often need to weaken the guarantee of one of these properties.
Consistency refers to the agreement among service nodes in the system regarding processing results; availability means that any non-failed node can respond to requests within a finite time; partition tolerance refers to the possibility of network partitioning, which makes communication between nodes unguaranteed.
Achieving complete consistency in a distributed scenario is impossible, but many engineering problems can be solved by making reasonable trade-offs. One can sacrifice some costs to achieve consistency in a distributed scenario. Currently, the differences in various consensus mechanisms based on blockchain mainly arise from the following two aspects:
First, different algorithmic assumptions. For example, algorithms like Paxos and Raft assume that nodes will not intentionally send incorrect messages, which is a relatively strong condition. The PoW consensus mechanism used by Bitcoin does not pre-know how many accounting nodes are in the system, while protocols like PBFT commonly used in consortium chains assume that nodes need permission.
Second, some costs are incurred to achieve a certain degree of consistency. For example, according to the CAP principle, availability is weakened, and services are refused in the event of system failures; algorithms like Paxos and Raft weaken availability to ensure the consistency of results. Similarly, Bitcoin sacrifices some fault tolerance (which may lead to forks) but guarantees consistency in the entire blockchain system after a certain time limit.
Algorithms are certainly not omnipotent; their boundaries dictate that some other incentive and constraint mechanisms must be introduced to ensure the normal operation of the entire system.
In blockchain projects based on PoS (Proof of Stake), creating new blocks does not require consuming computational power, and there are no penalties for nodes acting maliciously; for a node, the choice that maximizes benefits is to mine on multiple chains simultaneously, which can lead to serious forking phenomena. Generally, additional rules need to be introduced for such situations, such as adding penalty protocols.
Common Consensus Mechanisms in Public Chains#
The design of consensus mechanisms in public chains mainly revolves around decentralization and enhanced incentives. Many new blockchain systems currently support pluggable consensus mechanism modules, allowing different consensus mechanisms to be switched based on application scenarios and needs.
Maintaining the "uniqueness" of the main chain is crucial for public chains, as this is key to solving the "double spending" problem: to avoid the occurrence of double spending, one must be aware of all historical transaction information to ensure that this transaction does not conflict with previous history. How to ensure that transactions can proceed smoothly in an environment of asymmetric and uncertain information is the "Byzantine Generals Problem."
The PoW (Proof of Work) mechanism of Bitcoin solves the Byzantine Generals Problem through the following means:
● Maintaining periodic cycles to ensure nodes are synchronized: The difficulty is adjusted to ensure that the network always takes about 10 minutes to find a solution to a mathematical problem and generate a new block. During these 10 minutes, participants in the network send transaction information and complete transactions, and only then will the block information be broadcast, thus eliminating the state of nodes sending commands without limits or regularity.
● Ensuring that new blocks are generated by a single node through computational competition: Bitcoin ensures that only one (or a few, in the case of forks) node can mine a new block within a certain time period through timestamps and electronic signatures.
● Using a common ledger through blockchain: Each node in the Bitcoin network is synchronized in information during each cycle.
In fact, regardless of the method used, as long as it ensures synchronized time, consistent pace, single-point broadcasting, and a single chain, it can solve the Byzantine Generals Problem in distributed systems for cryptocurrencies.
PoS, as another consensus mechanism, allows miners to have a probability of generating new blocks equal to the proportion of cryptocurrency they hold. This can lead to greater power for wealthy accounts, potentially controlling the accounting rights, and may also result in increasing centralization of rights, but PoS does significantly reduce the energy costs of mining. In the long run, more cryptocurrencies may develop towards PoS.
In addition to these two common mainstream consensus mechanisms, the innovation of current public chain consensus mechanisms lies in the hybridization between the two, thereby improving data processing efficiency while retaining decentralized characteristics.
For example, Decred represents a PoW/PoS hybrid consensus:
- The mining process is similar to Bitcoin, requiring a certain amount of proof of work, but there are differences in the consensus phase; unlike Bitcoin, which requires all network nodes to verify blocks, the hybrid mechanism introduces PoS voting to determine whether the newly mined block is valid, greatly improving verification speed. Additionally, Hcash represents a PoW/PoS hybrid consensus, combining a dual-layer chain structure. It divides PoW difficulty into two levels, published on two chains, allowing both PoW miners and PoS miners to participate in the system's consensus and play a role.
Common Consensus Mechanisms in Consortium Chains#
Consortium chains place more emphasis on privacy, security, and regulation, so they incorporate more control elements and adopt consensus mechanisms similar to traditional Byzantine family consensus mechanisms.
Compared to public chains, consortium chains weaken the emphasis on decentralization, and due to the node admission system, they inherently grant a certain level of trust to nodes.
In the DPoS (Delegated Proof-of-Stake) mechanism, those with stock rights are elected and replaced, rather than generated based on the number of coins held as in PoS. It randomly selects a small group of nodes at irregular intervals through different strategies, allowing this small group of nodes to create, verify, sign, and supervise new blocks, significantly reducing the time and computational costs required for block creation and confirmation. DPoS does not require much trust; the selected delegates cannot change the details of transactions, and if nodes attempt to act maliciously, provide unstable computational power, or experience computer failures, the public community can quickly vote to expel them.
If PoW and PoS primarily solve consensus issues through economic models, then PBFT (Practical Byzantine Fault Tolerance) solves consensus through algorithmic models. It does not have a token distribution mechanism and has very low energy consumption. The process can be summarized as everyone voting to elect a leader, who records transactions, and others vote to approve. In the PBFT algorithm, it can be proven that as long as the number of faulty Byzantine nodes is less than one-third of the total number of nodes in the system, the entire system can operate normally. Current improvement directions for the algorithm roughly include using P2P networks, dynamically adjusting the number of nodes, and reducing the number of messages used in the protocol.
The innovation of consensus mechanism algorithms in consortium chains also includes the hybridization of DPoS and PBFT, applying the authorization mechanism of DPoS to PBFT to achieve dynamic authorization. Existing research has shown that such algorithms can achieve TPS of 10,000-12,000 with a best block time interval of 20 seconds, and latency controlled between 100-200ms. It is precisely because consortium chains retain some degree of "centralization" that they achieve faster transaction speeds and significantly lower transaction costs.
The Economic Cost of Consensus#
Consensus comes at a cost. Public chains like PoW incur substantial computational costs, hardware wear, and natural resource consumption to perform calculations and solve a mathematically meaningless problem to compete for accounting rights; in consortium chains, achieving consensus is akin to democratic voting, requiring rounds of negotiation and exchange of opinions to reach agreement. How to reduce the cost of democracy, how to achieve consensus with the fewest negotiation rounds and the least communication cost, is the goal pursued by algorithms and is also an important factor determining whether the blockchain machine runs fast enough.
Ultimately, we should focus on the balance between the cost and effect of consensus mechanisms. After all, blockchain technology must eventually be implemented. For enterprises, they should carefully consider their input-output ratio to decide whether to use blockchain technology or if there are lower-cost alternative solutions.
For example, using distributed databases to solve information asymmetry between enterprises, setting viewing permissions and encryption levels for data to achieve immutability, combined with a series of management measures, along with the fact that in most scenarios, leading enterprises may have little motivation to manipulate data and have sufficient motivation to maintain the database, in such cases, even the most complex consensus mechanisms may not be as effective as a good business model.
Characteristics of Hash Functions#
A hash function refers to a type of mathematical operation that accepts input values of any size and produces a fixed-length output value after computation. This output value can serve as a digital fingerprint for the input value. Just as twins have unique fingerprints, the design of hash functions ensures that they possess the same characteristic: even a very small difference in input values will result in a very large difference in the output of the hash function. Additionally, hash functions have no heuristic algorithms; the relationship between input and output appears completely random. For example, given a specific output result, asking what the corresponding input value should be, or asking for an input value that results in an output less than a certain value, these problems have no tricks or methods to follow and can only be solved through continuous trial and error; the more attempts made, the more likely one is to find the answer.
People can utilize these characteristics of hash functions to achieve many functions. For example, data protection: sending the content of the data along with the hash value of the data allows the receiver to perform a hash operation on the received data and compare it to determine whether the data has been tampered with. Additionally, when users log into a website, the hash value of the user's password can be stored in the database, and the hash value of the password entered by the user can be compared to verify identity. The benefit is that if the database is leaked, hackers cannot reverse-engineer the user's password from these hash values, making it relatively secure.
It is worth noting that the input set of hash functions is infinite, while the output length is fixed, meaning that the set of all possible outputs is finite. According to the pigeonhole principle: if there are n+1 elements placed into n sets, at least one set must contain at least two elements. Thus, it is theoretically certain that two different input values will have the same hash value, but the probability of this happening is very small, and hash functions are continuously being improved. The SHA1 function, for example, was found to have effective attack methods by cryptanalysts, leading systems like Bitcoin to adopt more advanced SHA2 series algorithms. The years of good operation of Bitcoin indicate that the SHA256 algorithm has stood the test of time. Additionally, using hash functions multiple times is a safer choice.
Hash functions are used in Bitcoin in several key roles.
- Purpose One: Compression and Verification of Transaction Information
Since the blockchain must handle a large amount of transaction information, directly storing all data in each block in a sequential manner would be very inefficient and time-consuming. However, using hash functions allows for compression and verification of information. In Merkle trees (a binary tree structure that can be understood as a topological structure for storing data), combined with hash function technology, one can quickly verify whether a transaction belongs to a certain block. For all transactions packed into a block, they are first divided into parts such as transaction information 1, transaction information 2, etc., and their corresponding hash values are calculated. Then, pairs are combined for hashing, ultimately yielding the root hash value of this Merkle tree. If the data recorded for a particular transaction changes, the final calculated Merkle root hash value will also differ.
So why use such an algorithm instead of simply stringing all transaction information together into a large block and calculating its hash value? The reason is that this binary tree structure allows for verification of only a small amount of data, and if there is an error in the transaction data, it can quickly locate the source of the error.
- Purpose Two: Proof of Work
Why is blockchain considered immutable? First, consider a simple hash chain: each time a block is packed, it includes the hash value of the previous block and the relevant information of this block. If the information of a certain block is tampered with, all subsequent blocks' hash values will change, and others will notice this change. However, the problem with this design is that anyone can modify the information on a certain block, recalculate all the information of the remaining chain, and claim that this is the correct chain.
The brilliance of Bitcoin's design lies in the fact that it makes achieving this process require expensive costs. It adopts a proof-of-work consensus mechanism, where everyone competes to prove that they have completed a certain amount of work, and the first to complete it gains the right to account. The work refers to finding a random number such that when added to a given string, the resulting hash value is less than a certain value. In Bitcoin, this given string includes version number, hash value of the previous block, transaction information stored as the Merkle root hash value, timestamp, and difficulty value. When miners find a random number that meets the requirements, they both "legitimately" declare their accounting rights and encode the transaction information through the hash function, storing it in an immutable manner. If someone attempts to change transaction information, they must be exceptionally lucky to quickly and successfully find the correct random number for each block in the subsequent chain, making their tampered chain the current longest chain. While this situation is theoretically possible, it is unlikely under limited computational power.
- Purpose Three: Bitcoin Wallet Address
In Bitcoin transactions, visible information includes the transaction number in the upper left corner, and the string of letters and numbers connected by arrows represents the Bitcoin address, indicating that Bitcoin has transferred between two addresses. The generation of this address is derived from the wallet's public key through a hash function. The public key is formed by asymmetrically encrypting a randomly generated private key. During transactions, both the public key and the Bitcoin address need to be publicly released to allow the blockchain system to verify the validity of payment transactions.
In this context, the role of the hash function is quite clever: quantum computers can easily derive the private key from the public key, but when faced with hash algorithms, quantum computers find it difficult to identify two different input values that have the same hash value. Satoshi Nakamoto's design allows Bitcoin to potentially resist threats from quantum computers: for example, each Bitcoin address is used only once, and each payment transfer is made to another person's address and the user's change address.
Satoshi Nakamoto cleverly utilized the characteristics of hash functions and ultimately formed a well-functioning system, involving various interdisciplinary fields, and reminding people to abstract the essence of things and pay attention to integration with other fields during technological innovation. With technological advancements, new hash functions are continuously being designed and tested.
Zero-Knowledge Proof#
Zero-knowledge proof is a probabilistic verification method that includes "factual statements" and "statements about personal knowledge." The verifier poses questions to the prover based on certain randomness, and if the prover can provide correct answers, it indicates that the prover likely possesses the "knowledge" they claim. Zerocoin (Zero-Coin Protocol) applies zero-knowledge verification in the process of minting and redeeming zero coins to hide the sender and receiver information corresponding to a transaction. Zerocash (Zero-Cash Protocol) employs the more novel zkSNARKs technology to convert the transaction content that needs to be verified into proof that two polynomial products are equal, combining techniques like homomorphic encryption to protect hidden transaction amounts while verifying transactions.
The downside is that if the network is attacked and an excessive amount of zero cash appears on the network, people may not be able to detect this situation or take measures; both Zerocoin and Zerocash require a pre-trust setup, failing to achieve true "trustlessness." New technologies like Intel SGX and zkSTARKs may solve these problems, but they still require practical testing.
Zero-knowledge proof is a cryptographic scheme first proposed in the 1980s by MIT researchers in a paper: "A zero-knowledge protocol is a method by which one party (the prover) can prove to another party (the verifier) that something is true without revealing any additional information beyond the fact that this specific statement is true. For example, in logging into a website, the server stores the hash value of the customer's password. To verify that the customer actually knows the password, most websites currently use a method where the server hashes the password entered by the customer and compares it to the stored result. However, this method has the flaw that the server can know the customer's original password during the calculation, and if the server is attacked, the user's password will be leaked. If zero-knowledge proof could be implemented, it would allow verification of the customer's login without knowing the customer's password, ensuring that even if the server is attacked, the user's account remains secure since the plaintext password is not stored.
The basic zero-knowledge proof protocol is interactive, requiring the verifier to continuously ask the prover a series of questions about their "knowledge." If the prover can provide correct answers, it is highly likely that they possess the claimed "knowledge."
For example, if someone claims to know the answer to a Sudoku puzzle, one way to prove this is for the verifier to randomly specify whether to check by row, column, or box. Each time, the verifier does not need to see the specific positions of the numbers but only needs to check whether the numbers 1-9 are included. As long as the number of verifications is sufficient, it can be believed with high probability that the prover knows the solution to the Sudoku puzzle. However, this simple method does not ensure that neither the prover nor the verifier is cheating; in the Sudoku case, both could have colluded beforehand, allowing the prover to pass verification without knowing the answer.
If they want to convince a third party, the verifier must also prove that their detection scheme is random and that they have not colluded with the prover.
Since it is difficult for third-party observers to verify the results of interactive zero-knowledge proofs, extra effort and costs are required when proving certain content to multiple people. Non-interactive zero-knowledge proofs, as the name suggests, do not require an interactive process, avoiding the possibility of collusion, but may require additional machines and programs to determine the sequence of experiments: for example, in the Sudoku case, the sequence of checks must be kept secret; otherwise, if the verifier knows the sequence in advance, they could use this information to prepare ahead of time and pass verification without knowing the actual "knowledge."
The content of zero-knowledge proofs can be summarized into two categories: "factual" statements:
-
For example, proving "a specific graph can be three-colored" or "a number N is composite";
-
Statements about personal knowledge: for example, "I know the specific coloring scheme for this graph" or "I know the factorization of N."
However, not all problems have zero-knowledge proof cryptographic schemes. Goldreich, Micali, and Wigderson provided the theoretical range of problems that can have zero-knowledge proof solutions. They found that for decision problems of polynomial complexity (where the answer is simply yes or no), known zero-knowledge proof schemes exist. One only needs to find the statement to be proven within such NP problems and convert it into an instance of the three-color problem to utilize existing protocols to achieve zero-knowledge proof. Since the three-color problem is an NPC problem, any other NP problem can be converted into an instance of this problem.
In blockchain transactions, such as those on Bitcoin and Ethereum networks, in addition to using addresses to replace the real identities of the transaction parties, the sending and receiving addresses and amounts are known. Others may be able to correlate Bitcoin addresses with real identities through various information on the network and interactions occurring in the real world, thus exposing privacy risks. Zerocoin designs a completely new approach that cannot obtain the user's real identity through transaction history analysis. In Zerocoin, a certain value of the currency to be traded is consumed to generate a zero coin with a unique serial number. Zero-knowledge proof can verify that you indeed spent this money without revealing which specific currency was spent. To transfer this money to someone else, logically, the zero coin must not be spent again. The method for zero coins is to maintain a list of invalidated serial numbers of all zero coins that have already been spent. When miners verify this spending transaction, they use zero-knowledge proof methods to check whether the serial number of the zero coin is on the invalidation list without needing to know which specific zero coin was spent. Since the spending transaction does not include the address and signature information, miners also do not know the source of this zero coin, making it difficult to analyze transaction history to obtain the user's identity.
In Zerocoin, the transaction amount can be known, while in Zerocash, which uses zkSNARKs technology, even the transaction amount can be hidden, with the only publicly recorded content in the ledger being the existence of the transaction. It can be proven that zkSNARKs exist for all problems in NP. It introduces many innovative technologies that allow them to be used in blockchain. Most importantly, zkSNARKs reduce the size of the proof and the computational effort required to verify them. The process can be summarized as follows:
(1) Decomposing the program to be verified into logical verification steps, breaking these logical steps down into arithmetic circuits composed of addition, subtraction, multiplication, and division.
(2) Through a series of transformations, the program to be verified is converted into a verification polynomial product that is equal, such as proving t(x)h(x)=w(x)v(x).
(3) To make the proof more concise, the verifier randomly selects several checkpoints s in advance to check whether the equations hold at these points.
(4) Using homomorphic encoding/encryption, the verifier can compute the equations without knowing the actual input values but can still perform verification.
(5) If a non-zero secret value k is multiplied on both sides of the equation, then when verifying t(s)h(s)k equals w(s)v(s)k, it becomes impossible to know the specific t(s), h(s), w(s), and v(s), thus protecting information security.
Unlike the cryptographic primitive RSA accumulator used in Zerocoin, zkSNARKs technology is newer and has not been widely validated, posing risks. Additionally, due to stronger anonymity, vulnerabilities in Zerocash are harder to detect. Compared to Zerocoin, Zerocash, with unknown transaction amount information, makes it impossible to detect if an attacker issues unlimited zero cash.
Furthermore, both Zerocoin and Zerocash require pre-built parameters, and users must trust that these parameters have not been leaked. However, once these parameters are leaked, the entire network faces catastrophic consequences. The complex trust setup makes Zerocash controversial, even though they have designed a "ceremony" (such as recording the process of destroying the computer containing the keys) to prove their security.
Possible solutions include utilizing modern "trusted execution environments" like Intel SGX and ARM TrustZone. In the case of Intel's SGX technology, even if the application, operating system, BIOS, or VMM is compromised, the private key remains secure. Additionally, the recently proposed zkSTARKs technology does not require a trust setup.
According to the zkSTARKs white paper, zkSTARKs is the first to achieve blockchain verification without relying on any trust setup while providing a system that accelerates computation exponentially as the amount of computational data increases. It does not rely on public key cryptography systems, and its simpler assumptions theoretically make it more secure, as its only cryptographic assumption is that hash functions (like SHA2) are unpredictable (this assumption is also the basis for the stability of Bitcoin mining), thus providing quantum resistance. As a novel technology, like zkSTARKs, it also needs to be tested over time.
If a problem can be solved by an algorithm in polynomial time, it is called a P-class problem. NP problems are those for which a solution can be verified in polynomial time. If a problem is an NP problem and all NP problems can be reduced to it, it is called an NPC problem.
Hash Time Lock Agreements#
Hash Time Lock Agreements (HTLAs) are a technology that allows token transactions and exchanges between different blockchain projects. In traditional exchanges, traders often need to pledge tokens in advance to the exchange, which brings certain transaction risks and requires high transaction fees. In the hash time lock agreement, only the sender, connector, and receiver are needed to achieve token transactions without any exchange platform, and if the transaction fails, the tokens have not actually transferred, incurring no additional transaction fees. Compared to exchanges, the hash time lock agreement provides a "flea market" where no third-party custody is required, and the role of the exchange is decentralized to individuals within the community, allowing people to safely trade tokens with each other.
The idea of hash time lock protocol technology originated from a discussion on the BitcoinTalk forum in 2013; its practical implementation is also related to Bitcoin's Lightning Network. In the Lightning Network, to establish a small payment channel between two users, users need to lock a portion of their funds in advance, and transactions involving that portion of funds occur off-chain. After a period, the final distribution of funds is determined, and this distribution plan is then uploaded to the main chain (Figure 7-1). This allows for a large number of small transactions to occur off-chain, increasing the transaction throughput of the Bitcoin network.
The hash lock contract (Hashed Timelock Contracts, HTLC) technology used to lock user funds in the Lightning Network inspired later developers. The transaction between tokens requires conversion through an intermediary, and the key is to obtain trust from all parties. The process of locking tokens is precisely a pledge process that can generate trust.
For a fictional transaction, suppose 1 Bitcoin is equivalent to 10 Ether. The sender (Sender) has 1.1 Bitcoin and wishes to purchase a service worth 10 Ether from the receiver (Receiver). The sender can contact a connector who has both a Bitcoin address and an Ethereum address and negotiate a token conversion fee of 0.1 Bitcoin. The transaction process is illustrated in the figure.
In this process, the step of transferring 1.1 Bitcoin carries a higher risk: the connector may exit the transaction immediately after receiving 1.1 Bitcoin, causing losses for the sender. A reasonable solution is to delay the delivery of Bitcoin. After the Ether is delivered, the Bitcoin can then be delivered, transferring the transaction risk to the connector (Plan Two).
To simultaneously protect the interests of the connector, the problem that needs to be solved is to ensure that when the receiver receives 10 Ether, the sender's 1.1 Bitcoin is also sent to the connector, and both events need to occur simultaneously. This actually reflects the "atomicity" of the transaction: either the funds are fully transferred, or they are not transferred at all, with no intermediate state. This problem is solved by the Pre-Shared Key (PSK) technology.
If we consider 1.1 Bitcoin as one transaction package and 10 Ether as another transaction package, in PSK technology, both transaction packages are initiated by the same key, thus achieving "simultaneous occurrence."
The sender pre-generates a key through encryption algorithms, sends the key to the receiver, and sends relevant information to the connector. At the same time, the sender locks their 1.1 Bitcoin in transaction package 1, which requires the key to transfer the funds.
The connector uses the information provided by the sender to create a transaction package 2 containing 10 Ether and sends it to the receiver. When the receiver uses the key to unlock transaction package 2, they receive 10 Ether, and the key is also sent to the connector, who can use that key to obtain the 1.1 Bitcoin in transaction package 1. This achieves the exchange between tokens.
To avoid locking funds for too long, both transaction packages 1 and 2 need to agree on a time limit; after the time limit, the funds will be unlocked and returned to the original address. This is the time lock function (Timelock). The aforementioned pre-shared key uses hash encryption, thus this technology solution is called the Hash Time Lock Agreement (Hashed-Timelock Agreements).
Limitations#
This technology still has certain limitations:
(1) The connector must bear some risk. In PSK technology, the connector must inject the key into transaction package 1 to obtain Bitcoin, meaning that the delivery of Bitcoin and Ether does not occur simultaneously. Since both currencies have agreed-upon time limits, if the time limit of transaction package 2 is greater than that of transaction package 1, it is possible for the receiver to obtain 10 Ether and for the connector to be unable to reclaim the 1.1 Bitcoin, resulting in a loss. This risk can be avoided by setting the time limit of transaction package 1 to be greater than that of transaction package 2.
(2) For blockchain projects that do not support hash time lock technology, the above process can only be conducted through another ledger platform. An additional accounting platform can save the transaction records of token transfers. However, since the accounting platform itself does not facilitate the transfer of tokens, it essentially records information about credit and borrowing, requiring sufficient trust between the two parties for the transaction to proceed. Through the accounting platform, as long as both parties have a foundation of trust, non-blockchain assets can also be exchanged through this accounting method.
In essence, transactions and flows between different assets only require a foundation of trust (to establish a connection) and ensure the atomicity of transactions (asset delivery) to proceed. In the hash time lock agreement, the locking of tokens achieves asset pledging, providing a foundation of trust for transactions. The transfer of keys ensures the atomicity of transactions. The introduction of time locks avoids disputes or accidents caused by excessively long transaction times. Beyond blockchain projects, this model can be applied to the flow of different asset classes.
The hash time lock protocol technology has been basically implemented by the Ripple Interledger project, and in blockchain projects capable of running smart contracts, token exchange technology may gradually become commonplace. This technology provides a possibility for the ecosystem of blockchain projects: mainstream blockchain projects like BTC serve as the main settlement system, while other application projects specifically address different user needs. When users enjoy services and settle payments, they use token exchange technology for payment. This way, mainstream projects convey value; application projects target segmented needs while sharing service pressure with mainstream projects; users get what they need. Ultimately, this will build a globally shared and trusted settlement network, running all decentralized applications on this foundation, truly realizing a rich blockchain application ecosystem.
References:
https://bitcointalk.org/index.php?topic=193281.msg2224949#msg2224
Sharding Technology#
The traditional concept of sharding technology involves splitting a database into multiple fragments and placing them on different servers. In modern cloud services, data is often hosted at different sites and partitioned. This practice aims to balance the load between multiple computers, thereby improving scalability; it also enhances availability by storing data across multiple sites.
Blockchain sharding technology is an expansion technology based on the concept of database sharding.
Whether in the blockchain field or the database field, the first step in sharding is to extract key feature values of the data and partition these key feature values into different fragments for processing according to certain rules. The selection of key feature values is crucial, as it relates to the guarantee of data uniqueness and the effectiveness of sharding. A concise standard for selecting feature values is: "Use what you consider to be the basic data pattern as the standard." Therefore, in blockchain projects, it is common to see sharding based on user private keys/account addresses, as these values are unique and do not change over time, making the logic of sharding clear.
Traditional Database Sharding Methods#
Traditional database sharding mainly has three methods:
(1) Hashing and Modulus: For example, if the entire network is divided into 3 shards, data is hashed and then taken modulus 3 to allocate it to a specific fragment. The purpose of this strategy is to reduce the occurrence of uneven shard loads, as the results of hash function calculations are completely random, breaking the correlation between certain key feature values and load amounts, thus making data more likely to be evenly distributed across shards. A counterexample is if the key feature value is the order of registration time, newly registered data being more active could lead to them all being allocated to one shard. However, this method has the drawback that if new shards are added, rebalancing shards becomes difficult; its advantage is that it does not require additional maintenance of state information.
(2) Consistent Hashing: The consistent hashing method without virtual nodes means that data is mapped to a circular hash ring based on feature values, while nodes are also mapped according to certain rules. The first node found in a clockwise direction is where the data is stored. The consistent hashing method with virtual nodes is similar but maps virtual nodes to the hash ring, allowing a physical node to occupy multiple ranges on the hash ring. This method requires maintaining state information, which indicates which data has been allocated to which node, but its advantage is that if the number of shards needs to increase, rebalancing shards becomes easier. However, maintaining shard state information requires considering consistency issues, which can be complex.
(3) Manual Partitioning: This involves dividing based on key feature values into different intervals, with each node corresponding to one or more intervals, similar to consistent hashing, and also requiring maintenance of state information.
- Consistency Challenges in Sharding
In blockchain technology, there needs to be a mechanism to know which node implements which shard. In traditional database systems, shard information (i.e., metadata, indicating which data has been allocated to which fragment) is generally stored on a dedicated server. Sometimes, to alleviate the pressure on the metadata server, distributed systems cache metadata on other nodes. The approach in blockchain is generally similar, requiring consistency of cached metadata between nodes or introducing a similar master server to ensure performance, but these solutions face challenges of data consistency.
The consistency and availability of multiple replicas fall within the scope of CAP theory, with two main available solutions.
The first is master-slave synchronization, where a master server is selected, and only the master server provides external services. The master server logs update information to a shared storage space, and slave servers read the logs from the shared storage space and apply them to achieve a state consistent with the master server. If the master server is detected to have failed, a new master server will be re-elected.
In the case of network partitioning, it is possible for everyone to believe that the original master server has crashed and elect a new master server, while the original master server continues to provide services, resulting in a "dual-master" phenomenon. To solve this problem, it is necessary to isolate the old master server so that it cannot provide services normally. To ensure strong consistency of metadata, the new master server must confirm that metadata is fully synchronized before continuing to provide external services. One way to achieve this is to immediately notify all cached servers when metadata changes and lock the data. If the system needs to complete tasks that require updating the state across multiple shards, access will be denied until the updates are completed. Another method frequently implemented in highly scalable NoSQL databases to maintain high consistency between replicated data is to use read-write arbitration and version control. This method avoids locking data, but at the cost of added complexity during read and write operations.
The second method is to achieve consistency among multiple replicas through distributed consistency protocols, such as Paxos and Raft, which can ensure that all backups provide external services while guaranteeing strong consistency.
Sharding Methods in Blockchain Technology#
In blockchain networks, based on the different objects, technology can be divided into state sharding, transaction sharding, and network sharding. Among them, network sharding employs more technical solutions.
State sharding in blockchain refers to each node only storing a portion of the blockchain state information, and the consistency issues that need to be resolved are similar to those mentioned above.
The implementation of transaction sharding is simpler. In account-based blockchain systems, each transaction will have a sender's address, and the system can allocate a shard based on the sender's address. This ensures that two double-spending transactions will be verified within the same shard, allowing the system to easily detect double-spending transactions without requiring any cross-shard communication. If the nodes are determined, there are almost no issues arising from the updating of metadata discussed above. However, if transaction verification involves communication across shards, the cost is usually high, impacting network throughput and economic benefits.
Network sharding in blockchain refers to dividing miners into several groups to verify transactions simultaneously, enhancing the system's ability to process transactions in parallel. Typically, this can be achieved by periodically generating random numbers to determine which nodes will be selected to reach consensus, mapping them to pre-numbered shards. However, if a node crashes, when redistributing nodes, consensus must be formed between shards.
It is important to note that when using network sharding technology in blockchain, which divides miners into several sub-networks responsible for verifying transactions on that shard, it is necessary to ensure that the number of malicious nodes is sufficiently small. Therefore, attention should be paid to ensuring randomness in the rules for allocating miners.
The key to sharding technology is that since the data in each shard is updated separately, when designing application logic, it must ensure successful updates of information while balancing efficiency, and it also needs to reserve a certain degree of robustness to address potential issues that may arise during the process of achieving final consistency. In applying sharding technology in blockchain, it is also necessary to consider issues such as defense against various attacks like Sybil attacks, DDoS attacks, and double-spending attacks, ensuring that the total number of nodes within each shard is sufficiently large and that honest nodes constitute the majority. Sharding technology requires extremely high security, and the number of nodes in blockchain systems may exceed that in traditional databases, facing bandwidth limitations, necessitating full consideration of performance and security issues caused by inconsistencies due to latency. Therefore, there are very few related projects that have been implemented. Long-term testing and verification in large-scale networks, combined with rigorous theoretical schemes, are needed to be convincing.
Space Proof Consensus Algorithm#
Since the first Bitcoin was mined in 2009, the blockchain industry has gradually expanded into a massive global market. Besides BTC, various blockchain projects such as LTC, ETH, and EOS have emerged. Currently, there are over 110,000 ERC20 token projects on Ethereum alone, and countless companies have published project white papers.
Bitcoin realized a peer-to-peer electronic payment system, and the birth of this distributed system relies on the PoW (Proof of Work) consensus algorithm it adopts. Currently, the vast majority of blockchain projects with main chains still use PoW or modified PoW consensus algorithms, with only a small number adopting PoS (Proof of Stake) or DPoS (Delegated Proof of Stake) algorithms. PoW provides a clear and effective consensus generation mechanism for distributed ledgers, but it also creates some problems: a large amount of energy is wasted in the process of calculating hash functions. With the emergence of ASIC and other chips, Bitcoin also faces increasing challenges of centralization. The current state of Bitcoin is far from Satoshi Nakamoto's original design.
However, PoS and DPoS mechanisms also have centralization issues, and the voting process is often cumbersome, making them clearly not the best solutions. It is worth mentioning that some blockchain projects have emerged that adopt schemes like "transaction mining," "staking mining," "insurance mining," and "mining mining." However, essentially, these projects only issue ERC20 tokens on Ethereum. Since they do not have a main chain, these projects do not require a consensus mechanism. The so-called mining schemes essentially belong to airdrop schemes, serving as an incentive mechanism with no inherent connection to the core technology of blockchain.
To truly solve the problems of energy waste and centralization arising from PoW, it is necessary to develop diversified mining schemes that address a series of technical issues:
(1) What can be used as proof of user expenditure without consuming work?
(2) How is this proof verified?
(3) How is the winner of the mining competition determined?
(4) How to avoid main chain forks, etc.?
Principles of Space Proof#
In the process of technological advancement, the PoSpace scheme has made significant explorations. PoSpace, or Proof of Space, aims to replace the PoW mechanism in Bitcoin and become a new consensus mechanism solution. This scheme has already been implemented in some blockchain projects. It uses the hard disk space paid by users as proof of expenditure, occupying hard disk space by downloading files; the larger the occupied space, the greater the user's expenditure.
PoSpace can bring the following benefits: greatly reducing resource waste; once users pay for hard disk space, subsequent mining does not require additional expenditure, etc. According to some teams' calculations, user behavior in PoSpace can be viewed as an extensible game model, and over time, more and more users will join.
To address the issue of hard disk space fraud, PoSpace divides nodes into two roles: provers and verifiers. Provers are ordinary nodes that need to store large amounts of information data (e.g., 100GB), while verifiers store databases and a small portion of the prover's stored information for verification.
When a user/prover first joins the network, they need to store a portion of data with a specific sequence based on the size of the chosen storage space (the data stored is determined by the user's public key, so the data of each user is different). This data is stored in a Directed Acyclic Graph (DAG) structure, and the relationship between each data block is sent to the verifier in the form of a Merkle tree. Thus, the verifier can know what data the prover has stored through the public key and understand how this data is structured through the sent Merkle tree.
In the verification phase, the verifier sends a "challenge" to the prover. This challenge is a random combination of the data blocks stored by the prover. The prover needs to generate the hash value of the corresponding combination of data based on the challenge information and return it to the verifier, who verifies whether the hash value is correct.
Since the challenge is a random combination of data, even slightly different data will yield completely different hash values. Therefore, the prover must indeed store the data blocks indicated by the "challenge" to generate the correct hash value. The verifier, having stored the complete database, can also verify the hash value returned by the prover.
The prover may only store a small portion of the data but still generate correct feedback through the verifier's challenge (the small portion of data stored by the prover happens to encompass the data combination included in the challenge). However, as the "challenge" process is repeated multiple times, the probability of the prover generating correct feedback by storing a small amount of data significantly decreases. Thus, multiple verifications can be used to prevent cheating by the prover. This is the space confirmation process in PoSpace.
Once a method for verifying user storage space is established, a way to determine the winner of the mining competition is still needed. A reasonable approach is that the miner with larger storage space is more likely to win the mining competition. PoSpace achieves this goal by designing a "quality function."
Quality Function#
The "quality function" needs to maintain a certain degree of randomness while distinguishing the probability of winning for each miner based on the size of their contributed space. A simplified approach is to respond to the verifier's challenge, where the hash value returned by the miner (a string of numbers) directly serves as a random quantity, and this string is adjusted based on the space occupied by the miner. For example, if the total size of the miner's stored space is N, the hash value can be squared N times to obtain the quality function. This way, the larger the miner's stored space, the smaller the quality function's value. We can stipulate that the miner with the smallest quality function in a single mining competition wins.
However, this still presents a problem: since miners do not need to incur subsequent costs after one-time payment for hard disk space, there is almost no cost to participate in the mining competition, making it easy to fork the main chain. To prevent miners from forking freely and causing confusion such as double spending, a rule is still needed to determine which chain is the unique chain, and all users must only record this unique chain to achieve true consensus.
Since each block is mined by the miner with the smallest "quality function," a natural idea arises: the unique main chain can be determined by the quality function. A quantity i can be set, stipulating that the total quality function of the last i blocks is summed to obtain the total quality function of the chain. The chain with the smallest total quality function can be designated as the main chain. To emphasize that earlier blocks carry more weight, a discount function can be added to reduce the importance of earlier blocks.
Thus, when a fork occurs in the main chain, the total quality function of the two (or more) forked chains can be calculated to determine the unique chain, ensuring that there is only one main chain, thereby establishing a distributed and unified ledger system among users.
PoSpace uses physical hard disk space as proof of expenditure, solving the problem of PoW continuously wasting a large amount of resources while establishing an electronic payment system that functions similarly to Bitcoin. PoSpace can be seen as a significant advancement in consensus mechanisms based on PoW. However, PoSpace still has some issues: for example, the introduction of verifier roles increases the risk of the system; how to design and arrange verifiers remains a question; using hard disk space as proof carries centralization risks, as a small number of individuals can monopolize mining by purchasing large amounts of hard disk space, leading to a "51% attack." Satoshi Nakamoto's vision of "one CPU chip representing one individual, with each individual having an equal opportunity to mine" remains difficult to achieve.
Nonetheless, the ideas behind PoSpace provide many insights, such as verifying the costs incurred by users through random methods; designing block quality functions to determine the winners of mining competitions; and designing chain quality functions to avoid main chain forks. Following this line of thought, it is entirely possible to develop consensus mechanisms that adapt to different usage scenarios, such as "proof of attention" and "proof of time." Additionally, if the hard disk space stored in PoSpace is transformed from meaningless bytes to meaningful content (such as videos and other materials), PoSpace may naturally be suitable for establishing a network resource-sharing community. It is believed that in the near future, consensus mechanisms will see more development and application.
Mimble-Wimble Technology#
In Bitcoin, full nodes need to download approximately 232GB of historical transaction data to verify new transactions. They must check all transactions submitted to the blockchain (which increase by about 300,000 daily) to obtain nearly 66 million UTXOs (Unspent Transaction Outputs). It is precisely because of this step of checking all historical transaction data that the threshold for becoming a Bitcoin full node has greatly increased, and lowering the threshold for full nodes is crucial for ensuring the decentralization of the blockchain. Moreover, the historical ledger will continue to grow over time, making it difficult for ordinary personal computers to support the operation of Bitcoin full nodes in the near future.
Is there a way to allow full nodes to verify transactions without downloading the entire historical ledger data while still ensuring the security of the blockchain system?
At the same time, the existing privacy and anonymity of Bitcoin are not as good as imagined. Since transaction addresses and amounts are public, big data analysis techniques can potentially analyze the identities of the transaction parties based on these transaction histories, thus threatening user privacy.
In fact, there are indeed solutions to the aforementioned problems in Bitcoin, which can not only reduce the historical ledger that originally needed to be downloaded from 232GB to 50GB, significantly reducing storage space and bandwidth usage, but also provide stronger privacy. This powerful technology is called Mimble-Wimble.
Its initial white paper was published in 2016, and the name and author of the technology are filled with magical colors; Mimble-Wimble is the "disappearing curse" from "Harry Potter." Like Satoshi Nakamoto, the author used the pseudonym Tom Elvis Jedusor (the French name for Voldemort in "Harry Potter") and disappeared after throwing down the white paper.
Implementation Principles#
In Mimble-Wimble, the reasons for ensuring privacy and scalability come from the following three points:
1) There are no addresses in the blockchain; each time a transfer occurs, the receiver must construct a new transaction witness.
(2) The transaction amount is also hidden.
(3) Intermediate state transactions can be merged. Merging means that if money is transferred from A to B and then from B to C, there is no need to record both transactions; it suffices to record how much A transferred to C, along with B's signature, ensuring that the transaction is valid while significantly reducing the storage space required for the block.
Mimble-Wimble relies on ECC (Elliptic Curve Cryptography). In ECC, a very large number k is typically chosen as the private key, and if H is a point on the elliptic curve, then k×H serves as the corresponding public key. The properties of the elliptic curve ensure that it is difficult to derive the private key k from the public key, as division of curve points is very challenging. Based on this property, the actual transaction amount can be hidden. The specific method is as follows:
Assuming the transaction amount is v, the node verifies that the output equals the input, which is equivalent to verifying v1 + v2 = v3. This equation is equivalent to multiplying both sides by a point H on the elliptic curve, meaning that it needs to verify:
v1×H + v2×H = v3×H
While this makes it difficult to reverse-engineer the actual transaction amount, since the set of possible attempts is limited, an attacker may still be able to deduce the value of v1. Therefore, a second elliptic curve point G and private key r are introduced, allowing any input or output value in the transaction to be expressed as r×G + v×H. Since both r and v cannot be deduced, the equation that needs to be verified becomes:
r1G + v1H + r2G + v2H = r3G + v3H
And it requires r1 + r2 = r3, thus effectively hiding the actual transaction amount. In actual transactions, only the parties involved know the transaction amount, while the nodes on the blockchain see encrypted numbers, and only the sender knows the private key r. To verify that the input equals the output while protecting the sender's private key from being cracked by the receiver, the sender chooses an additional value to add to their private key, so that from the receiver's perspective, they only see the sum of the two, while only they know the actual private key value. During verification, it is only necessary to verify that the sum of the transaction outputs equals the inputs, and that the transactor knows the additional value, which can be proven by using it to construct an ECDSA signature. Thus, the additional value acts as a private key for a transaction, preventing double spending.
During the merging of transactions, the verification process is similar because the content that needs to be verified for a single transaction is that the output equals the input. Thus, the merged (including mixing and reducing intermediate states) transactions essentially need to verify that the final output equals the input, and the intermediate state transactions only need to verify their signatures. If double spending occurs, just like in Bitcoin, nodes can easily verify that the total transaction output does not equal the input. In Mimble-Wimble, nodes can compare the total sum of all funds generated through mining with the total amount held to check the correctness of the total currency supply. Range proofs can be used to ensure that no currency is over-issued in transactions.
In summary, the main advantages of Mimble-Wimble are that it provides strong privacy while requiring very little storage space, resulting in high scalability. Since it does not require storing the entire historical transaction blockchain, by merging intermediate state transactions, only the source and current state of a coin need to be stored, with each historical transaction requiring only about 100 bytes of information, saving a significant amount of space compared to other blockchains and making the amount of information that new nodes need to synchronize and transmit very small. However, it has removed Bitcoin's scripting capabilities, and the encryption process requires some time, resulting in a block time of about 1 minute. Additionally, during transaction verification, both parties need to exchange some information, which may limit certain transaction functionalities.
As a newer and promising technology, its security needs to be tested over time, and a large number of developers need to participate in testing and verification. Bitcoin technology has undergone years of development and indeed requires significant optimization. Mimble-Wimble retains the advantages of PoW while optimizing the UTXO set and theoretically offers significant storage space optimization, bringing new ideas for improving scalability while preserving decentralization, pointing to new prospects.
Since the publication of Bitcoin in 2008, the application of blockchain technology has mainly gone through three stages:
(1) The blockchain 1.0 model, characterized by a programmable digital currency system, represented by Bitcoin, primarily solves the decentralization of currency payment methods.
(2) The blockchain 2.0 model, characterized by a programmable financial system, introduces the concept of "smart contracts," attempting to digitize any asset for trading, expanding blockchain from the initial currency system to the financial field, such as securities trading and equity transfer.
(3) The blockchain 3.0 model, characterized by a programmable society, extends the application scenarios beyond currency and finance to all aspects of social life.
From an industry perspective, fields such as finance, copyright, healthcare, notarization, supply chain, etc., have begun to recognize the importance of blockchain and are attempting to connect technology with real society. Blockchain technology can become a new tool, helping society reduce platform costs and making intermediary institutions a thing of the past, potentially shifting the focus of existing business models and accelerating company development. At the same time, the data on the blockchain is expected to promote the transformation of traditional data recording, dissemination, and storage management methods. From a societal perspective, blockchain technology is expected to integrate law and economy, fundamentally overturning the original regulatory model of society, and organizational forms will change because of it. Blockchain may ultimately lead people towards a society of distributed autonomy.
● The products or services exchanged by both parties can be digitized;
● Services/products are standardized, and the evaluation system is clear and traceable;
● Services are provided by individuals, and individuals consume services;
● As individuals gradually join, the value of the network increases.
A simple example is the shared mobility industry, where the service provider is primarily the driver (car owner), and the service consumer is the passenger. The driver takes the passenger from point A to point B, and the passenger pays the driver a certain fee, completing the transaction. The driver can calculate the driving distance using GPS navigation and record the driving time with a timestamp, automatically generating the fare according to a publicly disclosed calculation method. After the driver delivers the passenger to the destination and confirms, the passenger's account automatically transfers the money to the driver's account. As more passengers and drivers join the blockchain network, drivers are more likely to receive more suitable orders, and passengers find it easier to get rides.
Many other industries, such as P2P insurance, credit, gambling, prediction, and gaming, can achieve decentralized transactions to solve trust issues by "blockchainizing" value.
Based on the underlying technology of blockchain, it is evident that the application of blockchain needs to face the "intractable triangle" problem.
● Decentralization: Everyone has the entire network's ledger data and can participate in verification and recording;
● Consistency: Ensuring real-time consistency of transaction records among all participating nodes;
● Scalability: Increasing transaction throughput and scale while reducing transaction latency.
Existing blockchain applications, such as Bitcoin blockchain, ensure decentralization and consistency, but have weak scalability, processing about 7 transactions per second; for example, POS/DPoS verification sacrifices a certain degree of decentralization, not requiring everyone to participate in verification, thus improving scalability performance by several orders of magnitude to maintain ledger consistency.
When thinking of blockchain applications, the first thing that comes to mind is the field of digital currencies like Bitcoin. However, with the explosion of ICOs, the speculative nature of digital currencies has been wildly amplified, and some have even become tools for fraud. The application of blockchain is still in its infancy, and concrete landing scenarios have not yet been found. Currently, applications are more common in finance, copyright protection, supply chain, etc. However, the technical rules are still not very compatible with existing commercial structures. For example, the current blockchain has shortcomings in transforming financial infrastructure, including post-event irretrievability, inability to settle net positions, inability to lend securities, insufficient matching of transaction tokens with physical assets, and the potential for self-reinforcing feedback loops in the automatic execution of smart contracts, leading to financial instability.
In addition, the application of blockchain needs to control the accuracy of input data and ensure accurate interaction with physical devices. For instance, the price assessment of assets on the blockchain may still require consensus among multiple parties to ensure the authenticity, accuracy, and completeness of the original data. In application scenarios combining blockchain and the Internet of Things, the chaotic nature of the real physical world must also be considered, as there may be many uncertain human or accidental interference factors. For example, the Juicero problem: a computer ecosystem can be established and linked to bagged fruits to squeeze the fruit bags, but not everything that happens in the fruit bags can be controlled by the computer; a person can squeeze the bag with their hand. How to ensure that the data input into the blockchain accurately reflects the real world and that the blockchain's output is correctly implemented and operated is a critical issue for the successful application of blockchain. Thus, the slow pace of industry landing is also normal, caused by multiple factors, including the need for further maturity of underlying technology, lack of smart contract public chain platforms, insufficient compatibility of various token ecosystems, and unclear regulatory policies.
Finance creates value through the operation of funds. Robert Merton, a renowned finance professor at Harvard University, believes that the financial system has six basic functions:
(1) Clearing and payment functions: The financial system provides clearing and payment means for the trading of goods and services. One of the highlights of the rise of internet finance is the significant change in payment methods.
(2) Financing and equity refinement functions: The financial system can legally merge funds, allowing funds to flow from one party to another, achieving fund integration during the flow process, which is crucial for the construction of large projects and enterprises with development potential. Equity financing divides large investment projects into small shares for participation by small and medium investors, achieving equity refinement.
(3) Resource allocation function: Individual investors often find it difficult to make reasonable judgments about the market investment environment and company investment expectations. Financial intermediaries help investors make investment decisions, dispersing investment risks and optimizing the allocation of social capital investments.
(4) Risk management function: Due to trading costs and information asymmetry in financial markets, financial investments carry varying degrees of risk. The financial system can price, trade, disperse, and transfer risks, ensuring that financial market risks are reasonably allocated.
(5) Incentive function: This incentive mainly refers to equity issues. For enterprises, if their employees hold a certain amount of the company's stock or stock options, it can incentivize employees.
(6) Information function: In the financial market, fundraisers can obtain cost information for various financing methods, while investors can gather information about the prices of investment products and the factors influencing those prices.
The essence of finance is credit, as it enables the circulation and allocation of funds between different times, places, and participants. Therefore, there are many intermediary institutions in the financial industry, including banks, third-party payments, asset management institutions, etc. However, this centralized model has many problems. Centralization means high intercommunication costs between centers, time-consuming and labor-intensive communication, low operational efficiency, and centralized nodes are vulnerable to attacks and tampering, leading to data insecurity.
The financial system has developed to form a stable scale and structure, yet there are still some issues that need improvement in practical operations.
● Cross-border payment cycles are long and costs are high: The financial industry bears the payment function. Depending on the situation, there may be various payment methods for both parties in a transaction of goods and services. Based on the transaction parties, they can be categorized into direct and indirect payments. Direct payments occur simultaneously without the need for third-party guarantees or turnover. Indirect payments require both parties to rely on a third-party payment company, which must perform accounting calculations before completing the payment. Cross-border payments require the involvement of third-party companies due to geographical and trust issues. The actual cross-border payment arrival cycle is generally around one week, and the costs are high. For example, PayPal charges a standard cross-border payment transaction fee of 4.4% + $0.3.
● Financing cycles are long and costs are high: For small and medium enterprises, third-party credit agencies need to spend a lot of time reviewing various credentials and account records of the enterprise, and the financing costs they provide are generally high.
● Centralized risks in the bill market: The bill market relies on central commercial banks, and if the central server fails, the bill market will collapse. Additionally, issues such as multiple sales of a single bill and asynchronous endorsements of electronic bills often arise.
● Difficulty in verifying the authenticity of underlying assets: Funds, asset securitization, and various financial products are all packaging of underlying assets for direct purchase by investors. The sources of underlying assets are diverse, ranging from standardized bonds, stocks, public funds to non-standardized accounts receivable and income rights. The transformation of underlying assets into products for direct purchase by investors requires the participation of multiple parties, and there are information asymmetries in the transaction process, making it difficult for trading institutions to grasp the authenticity and accuracy of underlying assets.
Finance is a form of credit transaction, and credit is the foundation of finance. Therefore, the financial industry has many intermediary institutions, including banks, government, etc. To some extent, such institutions have value, as they can reduce the credit cost of transactions and ensure that transactions proceed normally, but we must pay some costs for this, commonly seen as various fees. Blockchain, as a distributed ledger technology, has publicly transparent and immutable ledger information, which can serve as a means of credit investigation and granting, reducing trust costs, shifting trust from traditional centralized credit institutions to the data trust of blockchain ledgers.