bitpro가 원문에서 글자만 모은 내용이에요. 표·그림·수식은 빠지거나 깨질 수 있어요. 전체 3쪽 중 2쪽
process of PoRA.
The default data flow is called the “main flow” of0G, and it incorporates
all new log entries (unless otherwise specified) in an append-only manner.
There are also specialized flows that only accept some category of log
entries, e.g. data related to a specific application. The most significant
advantage of specialized flows is a consecutive addressing space, which may
be crucial in some use cases. Furthermore, a specialized flow can apply
customized storage price, which is typically significantly higher than the
10
floor price of the default flow, and hence achieves better data availability
and reliability.
5 Incentive Mechanism
This section describes the incentive mechanism design of the 0G Storage,
which consists of two types of actors: users and miners (a.k.a. ”storage
nodes”). Users pay 0G tokens to create data entries in the log and add
data to the network, while miners provide data services and receive 0G
tokens as a reward from the network. The payment from users to miners is
mediated by the 0G consensus network, as the service is sustained by the
whole network rather than any specific miner.
0G Storage implements storage service in a “pay once for a fixed period
of time” manner. Users pay a storage endowment for each created data
entry for a certain period of time, e.g., 3 years, and thereafter the endow-
ment is used to incentivize miners who maintain that data entry by linear
distribution. Users can recharge to extend the storage period.
The storage endowment is maintained per data entry, and a miner is only
eligible for storage rewards from data entries that he has access to. The total
storage reward paid for a data entry is independent of the popularity of that
data entry. For instance, a popular data entry stored by many miners will
be frequently mined, but the reward is amortized amongst those miners. On
the other hand, a less popular data entry that is rarely mined would have
its storage reward accumulate and hence provide a higher payoff to miners
who store this rare data entry.
5.1 Storage Request Pricing
The cost of each 0G Storage request is composed of two parts: a) fee and
b) storage endowment. The fee part is paid to host chain miners/validators
for invoking the 0G contract to process storage requests and add new data
entries into the log, which is priced like any other smart contract invoca-
tion transaction. In what follows we focus on the storage endowment part,
which supports the perpetual reward to 0G Storage miners who serve the
corresponding data.
Given a data storage request SR with specific amount of endowment
11
SRendowment and size of committed data SRdata size (measured in number of
256 B sectors), the unit price of SR is calculated as follows:
SRunit price = SRendowment/SRdata size (1)
This unit price SRunit price must exceed a globally specified lower bound
to be added to the log, otherwise the request will be pending until when the
lower bound decreases below SRunit price (in the meantime miners will most
likely not store this unpaid data). Users are free to set a higher unit price
SRunit price, which would motivate more storage nodes mining on that data
entry and hence lead to better data availability.
5.2 Proof of Random Access
The 0G network adopts a Proof of Random Access (PoRA) mechanism to
incentivize miners to store data. By requiring miners to answer randomly
produced queries to archived data chunks, the PoRA mechanism establishes
the relation between mining proof generation power and data storage. Min-
ers answer the queries repeatly and computes an output digest for each
loaded chunk util find a digest satisfies the mining difficulty (i.e., has enough
leading zeros). PoRA will stress the miners’ disk I/O and reduce their capa-
bility in responding user queries. So 0G Storage adopts intermittent mining,
in which a mining epoch starts with a block generation at a specific block
height on the host chain and stops when a valid PoRA is submitted to the
0G Storage contract.
In a strawman design, a PoRA iteration consists of a computing stage
and a loading stage. In the computing stage, a miner computes a random
recall position (the universal offset in the flow) based on an arbitrary picked
random nonce and a mining status read from the host chain. In the loading
stage, a miner loads the archived data chunks at the given recall position,
and computes output digest by hashing the tuple of mining status and the
data chunks. If the output digest satisfies the target difficulty, the miner
can construct a legitimate PoRA consists of the chosen random nonce, the
loaded data chunk and the proof for the correctness of data chunk to the
mining contract.
In the following, we introduce some major considerations in improving
the fairness in PoRA mining and formalize the PoRA mining mechanism.
12
5.2.1 F airness for Small Miners
When the storage size of the 0G Storage network significantly exceeds the
storage capacity of a single machine, a single machine will take almost all the
time in finding an available recall position. This makes the PoRA becomes
proof of work. To make the PoRA mining process be friendly to small
miners with a single machine, the mining range is limited to a threshold
8 TB. When the size of archived data chunks exceeds 8 TB, a miner must
specify a mining range over the data flow sequence in size of 8 TB. For large
miners have enough machines to store all the data, it can mine on different
data ranges concurrently.
5.2.2 Disincentivize Storage Outsourcing
To promote the whole network storing enough replicas of data chunks, 0G
Storage limits the reward share of a single miner with a single storage replica.
PoRA mechanism seals the data for each miner in different ways, challenges
the accessing of sealed data, and prevents mining with data chunks from
others. If a miner outsources the data storage and queries the archived data
chunks in PoRA mining, it must seal the answering data to compute the
output digest, which is a large cost. A miner can seal and unseal tens of
MB data chunks by a single thread, so it can still serving the user queries
efficiently. As reading data from SSDs can reach up to 7 GB/s, the data
sealing is a heavy task for PoRA mining. As long as the cost imposed by
data sealing exceeds the cost of purchasing disks and synchronizing data,
the miners intend to store a replica of data chunks instead of sealing data
on PoRA mining.
More specifically, every 4KB data chunk will be sealed by the miner ID
and other context data. Suppose seal seed is a 32-byte digest of the miner
ID and the context data, figure 5 defines the seal process.
5.2.3 Disincentivize Distributed Mining
As a mining mechanism for incentivizing data storage, a single storage
replica should produce a similar hashrate for fairness. In PoRA mining
process, the hashrate is bottlenecked by the storage Input/Output (I/O).
So we expected all the miners to have similar storage I/O per unit of stor-
age. However, a mining farm can significantly increase its I/O compared
13
Input : seal seed, unsealed data
Output: sealed data
1 Regard unsealed data in length of 4 KB as an array of 32-byte elements ⃗d
2 h ← seal seed
3 for i from 0 to 127 do
4 ⃗d[i] ← ⃗d[i] XOR h
5 h ← Keccak256( ⃗d[i])
6 Regard ⃗d as a 4-KB data chunk sealed data
Figure 5: Seal 4 KB data chunks
to normal miners by storing data chunks in a distributed system with in-
memory filesystem. As the DDR memory has a significantly larger band-
width compared to SSD, the mining farm can produce much more hashrate
while contributing negligible to the data storage.
0G Storage prevents this behavior by imposing a large amount of data
transfer from the computing stage to the loading stage to encourage miners
to complete two stages on the same machine. So if the computing stage and
loading stage are separated by different machines, the hashrate is bounded
by the network bandwidth, which is usually smaller than the storage I/O
bandwidth. In the computing stage, a random scratchpad will be generated
coupled with the recall position. It has the same length as the data chunk
to be loaded. Before computing the hash of the loaded data chunk, the data
chunk should be mixed with the scratchpad by an XOR operation.
However, there is a dilemma between the read bandwidth and the trans-
action fee for submitting PoRA to the host chain. Since loading data chunks
from local disks should be more efficient than downloading them via the net-
work, the loading stage should leverage the fast SSD read speeds in sequen-
tial access to maximize the data chunk loading bandwidth. According to our
benchmarks, randomly loading 256-KB data chunks reaches 80% read speed
of sequential access. However, when submitting the data chunk to the host
chain, a 256-KB transaction is too large and will consume a large amount
of transaction fee. Reducing the size of the data chunk could mitigate this
issue, but would impair the read speed.
PoRA mechanism adopts “batched data loading” to resolve this
dilemma. In each iteration, a miner loads a 256-KB data chunk from
storage, divides it into 4-KB data chunks, and computes the hash for each
14
4-KB data chunk individually. If one of the 64 outputs matches the target
quality, the miner finds a valid answer. So only the 4-KB data chunk
contributing to this answer is required to be submitted to the host chain.
5.2.4 F ormal Definition for PoRA Mechanism
Precisely, the mining process has the following steps:
1. Register the miner id on the mining contract;
2. For each mining epoch, repeat the following steps:
(a) Wait for the layer-1 blockchain to release a block at a given epoch
height;
(b) Get the block hash block hash of this block and the relevant
context (including merkle root, data length, context digest) at this
time;
(c) Compute the number of minable entries n = ⌊data length/256KB⌋.
(d) For each iteration, repeat the following steps:
i. Pick a random 32-byte nonce;
ii. Decide the mining range parameters start position and
mine length; mine length should be equal to min(8TB , n·
256KB);
iii. Compute the recall position τ and the scratchpad ⃗ sby the
algorithm in figure 6;
iv. Load the 256-kilobyte sealed data chunk ⃗d started from the
position of h · 256KB;
v. Compute ⃗ w= ⃗d XOR ⃗ sand divide ⃗ winto 64 4-kilobyte pieces;
vi. For each piece ⃗ v, compute the Blake2b hash of the tuple
(miner id, nonce, context digest, start position, mine length,
⃗ v);
vii. If one of the Blake2b hash outputs is smaller than a target
value, the miner found a legitimate PoRA solution.
15
Input : miner id, nonce, context digest, start position, mine length
Output: Recall position τ and the scratchpad ⃗ s
1 h ← Blake2b(miner id, nonce, context digest, start position, mine length)
2 Initialize an empty array ⃗ swith 4096 64-byte elements
3 for i ∈ {x ∈ Z | 0 ≤ x <4095} do
4 h ← Blake2b(h)
5 ⃗ s[i] ← h
6 r ← Keccak256(h)
7 n ← mine length/256KB
8 τ ← start position + r mod n
Figure 6: Computing the recall position and the scratchpad
5.3 How to Balance the Number of Replicas?
Every miner in 0G Storage is required to explicitly claim the range of data
entries that he maintains, and then he may only use the claimed data in
his mining process. That is, a miner cannot receive mining rewards from
unclaimed data, and on the other hand, the mining output is decreased if
he fails to answer a challenge within his claimed range. As a result, a miner
has no incentive to lie about the range of data used for mining.
Since miners truthfully expose the list of data entries that they maintain,
it is easy to observe and estimate the number of replicas for every specific
piece of data. Furthermore, recalling that the storage endowment is paid to
miners who maintain corresponding pieces of data, miners are incentivized
to store rare data and hence balance the number of replicas.
6 Key-Value Runtime
0G Storage provides a Key-Value runtime upon the log layer. Each key-value
node can access the key-value store state through the runtime interface. The
key-value runtime provides the standard interface likePut() and Get(), and
accepts serialized key-value pair from any application-specific structure.
During the normal execution of the key-value store node, it maintains the
latest key-value state locally. It updates the value of a key through Put()
API which composes a log entry containing the updated key-value pair and
appends it to the log. The runtime constantly monitors the new log entries in
the log and fetches them back to the key-value node, updating the local key-
16
value state according to the log entry contents. In this sense, multiple key-
value store nodes essentially synchronize with each other through the shared
decentralized log. A user-defined function will be used to deserialize the
raw content in the log entry to the application-specific key-value structure.
Applications can use Get() API to access the latest value of any given
key. To improve the efficiency of the updates for small key-value pairs,
the Put() allows batched updates with multiple key-value pairs at once.
Figure 7 illustrates the architecture of the decentralized key-value store. To
manage the access control, the ownership information of each key can also be
maintained in the log entries. All the honest key-value nodes follow the same
update rule for the keys based on the ownership to achieve state consistency.
KA
Ionian KV
Runtime
Put(KA, VA) Get(KB)
App Server 0 App Server 1
A B A’ C B’ A’’
Storage Network
KB KC
Ionian Log
Runtime IA IB IC
KA
Ionian KV
Runtime KB KC
Ionian Log
Runtime IA IB IC
Blockchain
Data in the log are
structured as key-
based objects
Indices are constructed
by playing Ionian log
and can be consistent
across app severs.
Log Layer
Figure 7: Decentralized Key-Value Store
When a new key-value node just joins the network, it connects to the log
layer and plays the log entries from head to tail to construct the latest state
of the key-value store. During the log entry playing, an application-specific
key-value node can skip irrelevant log entries that do not contain the stream
IDs that it cares.
7 Transactional Processing on Key-Value Store
0G Storage employs concurrency control in the key-value runtime to support
transactional processing for concurrent operations on multiple keys. The to-
tal order of the log entries guaranteed by the underlying log system provides
17
the foundation for this concurrency control of the transactional executions
on top of the key-value store. This mechanism is optimistic and illustrated
in Figure 8.
When an application server linking with the 0G Storage key-value run-
time starts a transaction using BeginTx() interface, it notifies the runtime
that the transaction will work on the current state snapshot constructed by
playing the log to the current tail. The further key-value operations before
the invocation of EndTx() update the key-values locally in the server with-
out exposing the updates to the log. When EndTx() is invoked, the runtime
composes a commit record containing the log position the transaction starts
from and the read-write set of the transaction. This commit record is then
appended to the log.
Blockchain
1 2 3 4 5
App Server
TX start pos = #1
T->BeginTx();
buffer = T->Get(fileKey);
// update buffer;
T->Put(fileKey, buffer)
status = T->EndTx();
updates by
other clients
Commit record:
reads: fileKey
writes: fileKey
TX commits if read set has not
changed in conflict window
conflict window
Transaction Model:
1. Assume the transaction
participants are collaborative
and will honestly add read/write
set in commit record. Or:
2. Use ZKP or secure computation
hardware for verifiable
transaction computation.
Figure 8: Transactional Processing on 0G Storage KV Store
When an application server with the key-value runtime encounters the
commit record while playing the log, it identifies aconflict window consisting
of all the log entries between the start log position of the transaction and
the position of the commit record. The log entries in the conflict window
therefore contain the key-value operations concurrent with the transaction
submitting the commit record. The runtime further detects whether these
concurrent operations contain the updates on the keys that belong to the
read set of the transaction. If yes, the transaction is aborted. Otherwise, it
is committed successfully.
18
This transaction model assumes that the transaction participants are
collaborative and will honestly compose the commit record with the correct
content. Although this assumption in a decentralized environment is too
strong, it is still achievable for specific applications. For example, for an
application like Google Docs, a user normally shares access with others who
are trusted. In case this assumption cannot hold, the code of the transaction
can be stored in 0G log and some mechanism of verifiable computation like
zero-knowledge proof or hardware with trust execution environment (TEE)
can be employed by the transaction executors to detect the validity of the
commit record.
8 Service Marketplace over 0G Compute Network
As a decentralized AI operating system, 0G also provides a separate plat-
form managing the decentralized computing resources as services. It orga-
nizes the computation resources in a compute network and allows services
deployed and registered with verifiability so that users can choose trusted
services in the decentralized environment.
0G compute network is designed as a free marketplace where users and
service providers decide the prices of the services through a peer-to-peer
fashion. The service provider is free to quote their services, and then users
are free to choose the services that are priced appropriately for them to
use. The system assumes that any service result can be verified by the user
who submits the request. In the trust model, it treats the service providers
as more trustworthy than users and less powerful to apply sybil attack.
Therefore, a user may face the risk of losing the cost of a single request. In
other words, when a user submits a request and sends fee to smart contract,
she may spend the fee but fail to get the service result if the service provider
is malicious. In this case, the user can choose other service providers in the
network to retry the request.
The service provider should first register into the smart contract with
information about the service types that it provides, the price of each type,
and the specific verification mechanism. When a user wants to access the
service, she pre-charges some amount of fee to the smart contract for this
service provider. The user then can issue requests to the service provider
and the service provider decides to respond with the serving results based
19
on whether the remaining fee is enough. Any request and response have
the signatures of the user and the service provider, respectively. The user
can verify each response and decide to stop sending further request if the
verification is failed. The service provider can decide at any point to send
the request traces with user’s signature to the smart contract for settlement.
Once settlement is done, the corresponding portion of the pre-charged fee
can be sent to service provider’s account.
The performance trade-off is how often the service provider submits the
settlement request to the smart contract. For the high-latency services, the
provider can submit settlement per user request. But for the low-latency ser-
vices, the provider may need to batch more user requests for one settlement.
Another consideration is how the user request trace should be organized
as the smart contract input for settlement. Plainly submitting all the re-
quests in the trace may involve too much on-chain gas consumption. 0G
therefore employs zero-knowledge proof mechanism to optimize the on-chain
settlement cost.
0G Storage Network
OpML
Providers
TEE
Providers
4. Verification
0G Consensus
smart
contract
1. Load
models 프로젝트로 돌아가기