0G 백서

bitpro가 원문에서 글자만 모은 내용이에요. 표·그림·수식은 빠지거나 깨질 수 있어요. 전체 3쪽 중 2쪽

process of PoRA. The default data flow is called the “main flow” of0G, and it incorporates all new log entries (unless otherwise specified) in an append-only manner. There are also specialized flows that only accept some category of log entries, e.g. data related to a specific application. The most significant advantage of specialized flows is a consecutive addressing space, which may be crucial in some use cases. Furthermore, a specialized flow can apply customized storage price, which is typically significantly higher than the 10 floor price of the default flow, and hence achieves better data availability and reliability. 5 Incentive Mechanism This section describes the incentive mechanism design of the 0G Storage, which consists of two types of actors: users and miners (a.k.a. ”storage nodes”). Users pay 0G tokens to create data entries in the log and add data to the network, while miners provide data services and receive 0G tokens as a reward from the network. The payment from users to miners is mediated by the 0G consensus network, as the service is sustained by the whole network rather than any specific miner. 0G Storage implements storage service in a “pay once for a fixed period of time” manner. Users pay a storage endowment for each created data entry for a certain period of time, e.g., 3 years, and thereafter the endow- ment is used to incentivize miners who maintain that data entry by linear distribution. Users can recharge to extend the storage period. The storage endowment is maintained per data entry, and a miner is only eligible for storage rewards from data entries that he has access to. The total storage reward paid for a data entry is independent of the popularity of that data entry. For instance, a popular data entry stored by many miners will be frequently mined, but the reward is amortized amongst those miners. On the other hand, a less popular data entry that is rarely mined would have its storage reward accumulate and hence provide a higher payoff to miners who store this rare data entry. 5.1 Storage Request Pricing The cost of each 0G Storage request is composed of two parts: a) fee and b) storage endowment. The fee part is paid to host chain miners/validators for invoking the 0G contract to process storage requests and add new data entries into the log, which is priced like any other smart contract invoca- tion transaction. In what follows we focus on the storage endowment part, which supports the perpetual reward to 0G Storage miners who serve the corresponding data. Given a data storage request SR with specific amount of endowment 11 SRendowment and size of committed data SRdata size (measured in number of 256 B sectors), the unit price of SR is calculated as follows: SRunit price = SRendowment/SRdata size (1) This unit price SRunit price must exceed a globally specified lower bound to be added to the log, otherwise the request will be pending until when the lower bound decreases below SRunit price (in the meantime miners will most likely not store this unpaid data). Users are free to set a higher unit price SRunit price, which would motivate more storage nodes mining on that data entry and hence lead to better data availability. 5.2 Proof of Random Access The 0G network adopts a Proof of Random Access (PoRA) mechanism to incentivize miners to store data. By requiring miners to answer randomly produced queries to archived data chunks, the PoRA mechanism establishes the relation between mining proof generation power and data storage. Min- ers answer the queries repeatly and computes an output digest for each loaded chunk util find a digest satisfies the mining difficulty (i.e., has enough leading zeros). PoRA will stress the miners’ disk I/O and reduce their capa- bility in responding user queries. So 0G Storage adopts intermittent mining, in which a mining epoch starts with a block generation at a specific block height on the host chain and stops when a valid PoRA is submitted to the 0G Storage contract. In a strawman design, a PoRA iteration consists of a computing stage and a loading stage. In the computing stage, a miner computes a random recall position (the universal offset in the flow) based on an arbitrary picked random nonce and a mining status read from the host chain. In the loading stage, a miner loads the archived data chunks at the given recall position, and computes output digest by hashing the tuple of mining status and the data chunks. If the output digest satisfies the target difficulty, the miner can construct a legitimate PoRA consists of the chosen random nonce, the loaded data chunk and the proof for the correctness of data chunk to the mining contract. In the following, we introduce some major considerations in improving the fairness in PoRA mining and formalize the PoRA mining mechanism. 12 5.2.1 F airness for Small Miners When the storage size of the 0G Storage network significantly exceeds the storage capacity of a single machine, a single machine will take almost all the time in finding an available recall position. This makes the PoRA becomes proof of work. To make the PoRA mining process be friendly to small miners with a single machine, the mining range is limited to a threshold 8 TB. When the size of archived data chunks exceeds 8 TB, a miner must specify a mining range over the data flow sequence in size of 8 TB. For large miners have enough machines to store all the data, it can mine on different data ranges concurrently. 5.2.2 Disincentivize Storage Outsourcing To promote the whole network storing enough replicas of data chunks, 0G Storage limits the reward share of a single miner with a single storage replica. PoRA mechanism seals the data for each miner in different ways, challenges the accessing of sealed data, and prevents mining with data chunks from others. If a miner outsources the data storage and queries the archived data chunks in PoRA mining, it must seal the answering data to compute the output digest, which is a large cost. A miner can seal and unseal tens of MB data chunks by a single thread, so it can still serving the user queries efficiently. As reading data from SSDs can reach up to 7 GB/s, the data sealing is a heavy task for PoRA mining. As long as the cost imposed by data sealing exceeds the cost of purchasing disks and synchronizing data, the miners intend to store a replica of data chunks instead of sealing data on PoRA mining. More specifically, every 4KB data chunk will be sealed by the miner ID and other context data. Suppose seal seed is a 32-byte digest of the miner ID and the context data, figure 5 defines the seal process. 5.2.3 Disincentivize Distributed Mining As a mining mechanism for incentivizing data storage, a single storage replica should produce a similar hashrate for fairness. In PoRA mining process, the hashrate is bottlenecked by the storage Input/Output (I/O). So we expected all the miners to have similar storage I/O per unit of stor- age. However, a mining farm can significantly increase its I/O compared 13 Input : seal seed, unsealed data Output: sealed data 1 Regard unsealed data in length of 4 KB as an array of 32-byte elements ⃗d 2 h ← seal seed 3 for i from 0 to 127 do 4 ⃗d[i] ← ⃗d[i] XOR h 5 h ← Keccak256( ⃗d[i]) 6 Regard ⃗d as a 4-KB data chunk sealed data Figure 5: Seal 4 KB data chunks to normal miners by storing data chunks in a distributed system with in- memory filesystem. As the DDR memory has a significantly larger band- width compared to SSD, the mining farm can produce much more hashrate while contributing negligible to the data storage. 0G Storage prevents this behavior by imposing a large amount of data transfer from the computing stage to the loading stage to encourage miners to complete two stages on the same machine. So if the computing stage and loading stage are separated by different machines, the hashrate is bounded by the network bandwidth, which is usually smaller than the storage I/O bandwidth. In the computing stage, a random scratchpad will be generated coupled with the recall position. It has the same length as the data chunk to be loaded. Before computing the hash of the loaded data chunk, the data chunk should be mixed with the scratchpad by an XOR operation. However, there is a dilemma between the read bandwidth and the trans- action fee for submitting PoRA to the host chain. Since loading data chunks from local disks should be more efficient than downloading them via the net- work, the loading stage should leverage the fast SSD read speeds in sequen- tial access to maximize the data chunk loading bandwidth. According to our benchmarks, randomly loading 256-KB data chunks reaches 80% read speed of sequential access. However, when submitting the data chunk to the host chain, a 256-KB transaction is too large and will consume a large amount of transaction fee. Reducing the size of the data chunk could mitigate this issue, but would impair the read speed. PoRA mechanism adopts “batched data loading” to resolve this dilemma. In each iteration, a miner loads a 256-KB data chunk from storage, divides it into 4-KB data chunks, and computes the hash for each 14 4-KB data chunk individually. If one of the 64 outputs matches the target quality, the miner finds a valid answer. So only the 4-KB data chunk contributing to this answer is required to be submitted to the host chain. 5.2.4 F ormal Definition for PoRA Mechanism Precisely, the mining process has the following steps: 1. Register the miner id on the mining contract; 2. For each mining epoch, repeat the following steps: (a) Wait for the layer-1 blockchain to release a block at a given epoch height; (b) Get the block hash block hash of this block and the relevant context (including merkle root, data length, context digest) at this time; (c) Compute the number of minable entries n = ⌊data length/256KB⌋. (d) For each iteration, repeat the following steps: i. Pick a random 32-byte nonce; ii. Decide the mining range parameters start position and mine length; mine length should be equal to min(8TB , n· 256KB); iii. Compute the recall position τ and the scratchpad ⃗ sby the algorithm in figure 6; iv. Load the 256-kilobyte sealed data chunk ⃗d started from the position of h · 256KB; v. Compute ⃗ w= ⃗d XOR ⃗ sand divide ⃗ winto 64 4-kilobyte pieces; vi. For each piece ⃗ v, compute the Blake2b hash of the tuple (miner id, nonce, context digest, start position, mine length, ⃗ v); vii. If one of the Blake2b hash outputs is smaller than a target value, the miner found a legitimate PoRA solution. 15 Input : miner id, nonce, context digest, start position, mine length Output: Recall position τ and the scratchpad ⃗ s 1 h ← Blake2b(miner id, nonce, context digest, start position, mine length) 2 Initialize an empty array ⃗ swith 4096 64-byte elements 3 for i ∈ {x ∈ Z | 0 ≤ x <4095} do 4 h ← Blake2b(h) 5 ⃗ s[i] ← h 6 r ← Keccak256(h) 7 n ← mine length/256KB 8 τ ← start position + r mod n Figure 6: Computing the recall position and the scratchpad 5.3 How to Balance the Number of Replicas? Every miner in 0G Storage is required to explicitly claim the range of data entries that he maintains, and then he may only use the claimed data in his mining process. That is, a miner cannot receive mining rewards from unclaimed data, and on the other hand, the mining output is decreased if he fails to answer a challenge within his claimed range. As a result, a miner has no incentive to lie about the range of data used for mining. Since miners truthfully expose the list of data entries that they maintain, it is easy to observe and estimate the number of replicas for every specific piece of data. Furthermore, recalling that the storage endowment is paid to miners who maintain corresponding pieces of data, miners are incentivized to store rare data and hence balance the number of replicas. 6 Key-Value Runtime 0G Storage provides a Key-Value runtime upon the log layer. Each key-value node can access the key-value store state through the runtime interface. The key-value runtime provides the standard interface likePut() and Get(), and accepts serialized key-value pair from any application-specific structure. During the normal execution of the key-value store node, it maintains the latest key-value state locally. It updates the value of a key through Put() API which composes a log entry containing the updated key-value pair and appends it to the log. The runtime constantly monitors the new log entries in the log and fetches them back to the key-value node, updating the local key- 16 value state according to the log entry contents. In this sense, multiple key- value store nodes essentially synchronize with each other through the shared decentralized log. A user-defined function will be used to deserialize the raw content in the log entry to the application-specific key-value structure. Applications can use Get() API to access the latest value of any given key. To improve the efficiency of the updates for small key-value pairs, the Put() allows batched updates with multiple key-value pairs at once. Figure 7 illustrates the architecture of the decentralized key-value store. To manage the access control, the ownership information of each key can also be maintained in the log entries. All the honest key-value nodes follow the same update rule for the keys based on the ownership to achieve state consistency. KA Ionian KV Runtime Put(KA, VA) Get(KB) App Server 0 App Server 1 A B A’ C B’ A’’ Storage Network KB KC Ionian Log Runtime IA IB IC KA Ionian KV Runtime KB KC Ionian Log Runtime IA IB IC Blockchain Data in the log are structured as key- based objects Indices are constructed by playing Ionian log and can be consistent across app severs. Log Layer Figure 7: Decentralized Key-Value Store When a new key-value node just joins the network, it connects to the log layer and plays the log entries from head to tail to construct the latest state of the key-value store. During the log entry playing, an application-specific key-value node can skip irrelevant log entries that do not contain the stream IDs that it cares. 7 Transactional Processing on Key-Value Store 0G Storage employs concurrency control in the key-value runtime to support transactional processing for concurrent operations on multiple keys. The to- tal order of the log entries guaranteed by the underlying log system provides 17 the foundation for this concurrency control of the transactional executions on top of the key-value store. This mechanism is optimistic and illustrated in Figure 8. When an application server linking with the 0G Storage key-value run- time starts a transaction using BeginTx() interface, it notifies the runtime that the transaction will work on the current state snapshot constructed by playing the log to the current tail. The further key-value operations before the invocation of EndTx() update the key-values locally in the server with- out exposing the updates to the log. When EndTx() is invoked, the runtime composes a commit record containing the log position the transaction starts from and the read-write set of the transaction. This commit record is then appended to the log. Blockchain 1 2 3 4 5 App Server TX start pos = #1 T->BeginTx(); buffer = T->Get(fileKey); // update buffer; T->Put(fileKey, buffer) status = T->EndTx(); updates by other clients Commit record: reads: fileKey writes: fileKey TX commits if read set has not changed in conflict window conflict window Transaction Model: 1. Assume the transaction participants are collaborative and will honestly add read/write set in commit record. Or: 2. Use ZKP or secure computation hardware for verifiable transaction computation. Figure 8: Transactional Processing on 0G Storage KV Store When an application server with the key-value runtime encounters the commit record while playing the log, it identifies aconflict window consisting of all the log entries between the start log position of the transaction and the position of the commit record. The log entries in the conflict window therefore contain the key-value operations concurrent with the transaction submitting the commit record. The runtime further detects whether these concurrent operations contain the updates on the keys that belong to the read set of the transaction. If yes, the transaction is aborted. Otherwise, it is committed successfully. 18 This transaction model assumes that the transaction participants are collaborative and will honestly compose the commit record with the correct content. Although this assumption in a decentralized environment is too strong, it is still achievable for specific applications. For example, for an application like Google Docs, a user normally shares access with others who are trusted. In case this assumption cannot hold, the code of the transaction can be stored in 0G log and some mechanism of verifiable computation like zero-knowledge proof or hardware with trust execution environment (TEE) can be employed by the transaction executors to detect the validity of the commit record. 8 Service Marketplace over 0G Compute Network As a decentralized AI operating system, 0G also provides a separate plat- form managing the decentralized computing resources as services. It orga- nizes the computation resources in a compute network and allows services deployed and registered with verifiability so that users can choose trusted services in the decentralized environment. 0G compute network is designed as a free marketplace where users and service providers decide the prices of the services through a peer-to-peer fashion. The service provider is free to quote their services, and then users are free to choose the services that are priced appropriately for them to use. The system assumes that any service result can be verified by the user who submits the request. In the trust model, it treats the service providers as more trustworthy than users and less powerful to apply sybil attack. Therefore, a user may face the risk of losing the cost of a single request. In other words, when a user submits a request and sends fee to smart contract, she may spend the fee but fail to get the service result if the service provider is malicious. In this case, the user can choose other service providers in the network to retry the request. The service provider should first register into the smart contract with information about the service types that it provides, the price of each type, and the specific verification mechanism. When a user wants to access the service, she pre-charges some amount of fee to the smart contract for this service provider. The user then can issue requests to the service provider and the service provider decides to respond with the serving results based 19 on whether the remaining fee is enough. Any request and response have the signatures of the user and the service provider, respectively. The user can verify each response and decide to stop sending further request if the verification is failed. The service provider can decide at any point to send the request traces with user’s signature to the smart contract for settlement. Once settlement is done, the corresponding portion of the pre-charged fee can be sent to service provider’s account. The performance trade-off is how often the service provider submits the settlement request to the smart contract. For the high-latency services, the provider can submit settlement per user request. But for the low-latency ser- vices, the provider may need to batch more user requests for one settlement. Another consideration is how the user request trace should be organized as the smart contract input for settlement. Plainly submitting all the re- quests in the trace may involve too much on-chain gas consumption. 0G therefore employs zero-knowledge proof mechanism to optimize the on-chain settlement cost. 0G Storage Network OpML Providers TEE Providers 4. Verification 0G Consensus smart contract 1. Load models
프로젝트로 돌아가기