Towns 백서 전문

s protocol. The data structure representing direct messaging is defined by the `DmChannelPayload` protobuf. Each DM is created with the pair of users privy to the DM conversation. Note that just like with channel messages, DM's are `EncryptedData` messages from the node's vantage point. ### GDM Channel messages Group direct message messages are supported by the Towns protocol as their own streams as well. Each Group DM is created with the list of members that forms the initial Membership roster. Though outside the specification of the protocol, the membership roster is meant to convey entitlement to encryption keys, such that clients can implement logic to share keys only with members of the same group dm. > DM's, and GDM's are all created as new streams in the Towns Node using the `createStream` rpc client method. Unlike Spaces and Channels though, there exists no on-chain record of these streams. ### Media messages Media files are supported as separate streams in Towns protocol associated with a parent `channel_id`. Messages added to a Towns node containing media must pass validation criteria that proves the `creator` is a member of that `channel_id` to prevent man-in-the-middle type attacks. Media files can be quite large in size and as such are chunked in byte arrays on nodes and therefore support pagination by clients. # Chunk Management Source: https://docs.towns.com/concepts/chunk-management In distributed systems, managing resource allocation efficiently is crucial to prevent certain nodes from becoming overburdened. This is particularly important in the context of stream processing. To mitigate the issue of hotspots, where some nodes are overloaded with highly active streams while others handle less active streams, a strategy of breaking streams into chunks is proposed. The idea is that once a chunk of a stream reaches a predetermined size, it is then handed over to a new set of nodes. This approach, which is not part of the initial implementation but is on the immediate roadmap, ensures a more balanced distribution of processing load across the nodes. Additionally, these chunks would maintain the same replication factor, which is currently set at five, as is the case with entire streams in the current system. This method aims to optimize resource utilization and enhance the overall efficiency of the distributed system. # Cross Chain Entitlements Source: https://docs.towns.com/concepts/cross-chain-entitlement-checking ## Overview Cross Chain Entitlement Checking is a key feature in Towns Protocol, enabling the verification of entitlements that are stored on blockchains other than Base. Initially this set is restricted to other EVM chains Ethereum Mainnet, Optimism, Arbitrum, and Polygon. Other chains are planned to be added in the future. ## Use Cases 1. **Asset Requirements Across Chains:** For instance, a Space might necessitate members to hold a Crypto Punk on Ethereum Mainnet and a Degod on Polygon. 2. **Asset Quantity Requirements:** Another example could be a Space that allows entry only to users who possess over 5 ETH across their linked wallets. ## Entitlement Checks for Permissions Cross Chain Entitlement Checks can be utilized for specific permissions such as Read, Write, or Mint. However, it's important to note that permissions requiring direct contract transactions, like channel creation or role assignment, must rely solely on entitlements based on assets stored on the Base Chain. ### Process for Minting a Member NFT 1. **Join Request:** A user initiates a request to join a Space through the smart contract. 2. **Membership Fee Verification:** The contract verifies whether the user has paid the necessary membership fee, if applicable. 3. **Cross-Chain Requirement Assessment:** The contract determines if joining the Space requires a cross-chain entitlement check. 4. **Broadcasting the Request:** If a cross-chain check is needed, the request is broadcast as an event from the smart contract, targeting a randomly selected NodeID from the active nodes in the node registry. 5. **Node Processing:** The designated node picks up the request and retrieves the entry requirements for the Space, which are defined in the contract. 6. **Wallet Link and Evaluation:** The node accesses all linked wallets of the user and assesses if the user fulfills the Space's entry criteria. 7. **Minting Member NFT:** Upon successful verification, the node sends a positive response back to the smart contract, triggering the automatic minting of the Member NFT for that Space to the user's address. # Encryption Source: https://docs.towns.com/concepts/encryption End-to-End Message Encryption All messages in the context of the communication modalities that Towns Protocol supports are encrypted. Nodes are only privy to ciphertext and clients manage their device encryption keys. Two related cryptographic ratcheting algorithms are used by Towns to encrypt messages. Towns employs an implementation of the [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/) cryptographic ratchet described by Signal to encrypt peer to peer messages. Towns also employs an AES-based cryptographic ratched optimized for group messaging. ## Devices To facilitate end to end encryption in Towns, each `(user, client instance)` tuple is associated with a `deviceId`, which identifies that device and is used to establish peer-to-peer encrypted sessions for the purpose of sharing group message encryption keys. Devices are objects storing key material created on the client and stored in the Towns Node on the user's `UserDeviceKey` stream containing the following pair of keys: 1. **Curve25519 peer-to-peer encryption key** - A long-lived Curve25519 asymmetric key pair is created with a new device of a user. The private portion of this key never leaves the device, while the public portion is stored in the user's `UserDeviceKey` stream. This key along with the following key are used by other user's to establish secure and ephemeral p2p sessions. 2. **Curve25519 fallback key** - A second Curve25519 key pair is created using the encryption key and published to a user's UserDeviceKey stream. For Alice to establish a new secure p2p encrypted session with Bob, Alice would use her encryption key along with Bob's public key portion of his encryption key and fallback key. > Device lifecycle is outside of the purview of the Towns protocol and managed entirely by client implementations. However, given it is expected under the protocol that there exists a 1-1 relation between `(user, client instance)` tuples and `devices`, the Towns Node performs periodic compaction criteria to stem the uncontrolled growth of user's device key stream in storage. ## Encryption Data Schemas Encrypted data originating from messages or metadata, such as usernames, is described in the protocol with a protobuf message as follows. ```protobuf theme={null} message EncryptedData { /** * Ciphertext of the encryption envelope. */ string ciphertext = 1; /** * Encryption algorithm used to encrypt this event. */ string algorithm = 2; /** * Sender device public key identifying the sender's device. */ string sender_key = 3; /** * The ID of the session used to encrypt the message. */ string session_id = 4; /** * Optional checksum of the cleartext data. */ optional string checksum = 5; } ``` The `session_id` is used to identify the keys associated with the ciphertext, which can be used to decrypt the same message multiple times. This is particularly useful in a group messaging application as it avoids the need to re-establish peer-to-peer encrypted sessions for each message. Peer to peer encryption sessions are only used to transmit session keys corresponding to message events and are described by the following protobuf in the protocol. ```protobuf theme={null} message GroupEncryptionSessions { string stream_id = 1; string sender_key = 2; repeated string session_ids = 3; // deviceKey: per device ciphertext of encrypted session keys that match session_ids map<string, string> ciphertexts = 4; } ``` A map is used to index ciphertext by the intended user's deviceKey since peer-to-peer encrypted payloads are only able to be decrypted by the deviceKey that the `outbound` sender's session was created for. In general, peer-to-peer encrypted messages are encrypted in a per device basis. ## Encryption Lifecycle Below is an example of the encryption lifecycle between Alice and Bob who are co-members of a channel within a space. 1. Alice logs in to her client, creates a new `device`, and joins a Space. 2. Bob who is already logged in and a member of the Space sees a `KeySolicitation` message with a `device_key` corresponding to Alice's device. 3. Bob validates that Alice `isEntitled` to decryption keys for the channel stream that the solicitation event appeared in and creates a new p2p encrypted `outbound` session using Alice's device key and fallback key to transmit the keys requested from his local cache. 4. Bob sends an `ack` to the stream to notify other co-members of the channel that Alice's request is being worked on. 5. Alice sees Bob's message on her `UserToDevice` key stream and created a new `inbound` session with Bob's device key and fallback key obtained from his `UserDeviceKey` stream. 6. Alice decrypts Bob's message and extracts the key material storing it in her local cache. 7. Alice attempts re-decrypting the channel messages that share the `session_id` of the keys she now has in her possession. > Peer-to-peer encryption in Towns requires distinct sessions `outbound`, `inbound` for encryption and decryption, respectively. Moreover, each message can only be decrypted once per established session. ## Key Sharing ### Active Sharing Session keys to encrypt message events and metadata in Towns are created on an as-needed basis by clients. If a user is joining a channel and decides to subsequently send a message for the first time, they will create a new outbound session key to encrypt the message and send it to the member roster of the channel along with the message. The same key can be used as an inbound session key by other members to decrypt the message encrypted with that session. ### Passive Sharing The above contrasts with passive key sharing, which is used to transmit keys that users are entitled to but do not have locally. When joining a channel for the first time, a user will sync stream events, but will need session keys to decrypt message events. The protocol supports an efficient key sharing scheme that has users place an `KeySolicitation` event on the stream, which any online member of the stream will see and conditionally service if the solicitator is an entitled member of the stream. ### Data The protocol allows for any stream to support key sharing by way of `KeySolicitation`, and `KeyFulfillment` messages. Since fulfilling a solicitation requires creating a peer-to-peer encrypted session with the solicitator, the `device_key` and `fallback_key` are added to the payload to save a lookup against the `UserDeviceKey` stream. Fulfillments are synced by members of the same stream to avoid the worst case behavior of every member fulfilling every request in a duplicative manner. ```protobuf theme={null} message KeySolicitation { string device_key = 1; // requesters device_key string fallback_key = 2; // requesters fallback_key bool is_new_device = 3; // true if this is a new device, session_ids will be empty repeated string session_ids = 4; } message KeyFulfillment { string user_id = 1; string device_key = 2; repeated string session_ids = 3; } ``` # Mini Blocks Source: https://docs.towns.com/concepts/mini-blocks # Mini Block Production Similar to traditional blockchain building, mini blocks are used to ensure data integrity and achieve consensus for the ordering of valid events in a given stream. The following is a detailed description of each step in the process: 1. **Election of Responsible Node**: A responsible node is elected to construct the mini block. This election is based on the block number and incorporates an element of randomness, utilizing `randdao`. 2. **Initiation of Voting Process**: The elected leader initiates a voting process. In this stage, each participating node contributes by sending hashes of the events present in their mini-pool. 3. **Combination of Votes**: The leader combines these votes and selects events that are present in the majority of the mini-pools. 4. **Authority of the Leader**: The leader has the discretion to determine the order of events within the mini block. 5. **Formation of Mini Block**: The events chosen are then combined to form the mini block. 6. **Mini Block Header Creation**: A header for the mini block is generated and subsequently signed by the responsible node. 7. **Integration into Towns Chain**: This mini block header is then integrated into the Towns chain and disseminated to all participating nodes. 8. **Request for Missing Events**: If a node does not possess all the events in the newly received mini block, it must proactively request these missing events from other participating nodes. 9. **Local Storage Update**: Participating nodes update their local storage to reflect the new mini block. They also remove old mini-pools that were included in that mini block. 10. **Formation of New Mini-Pool**: Events not included in the current mini block form the basis of a new mini-pool for the next generation. 11. **Event Effectuation at Block Boundaries**: Certain events, such as joins and leaves, are only effectuated at block boundaries. This is to maintain consistent ordering across the network. This procedure ensures a transparent and efficient process for mini block production, critical to the stability and reliability of the protocol. ## Active Minipool Reconciliation The system is designed to actively propagate events across stream nodes. Once a valid event reaches a majority of nodes, it is virtually guaranteed to end up in the block. Therefore, during an StreamService.AddEvent operation, the event is actively forwarded to other stream nodes. If a stream node notices that it lacks an event referenced in a NodeToNode.NewEventInPool message, it proactively attempts to fetch it from the node that sent this message. A node actively fetches missing events if one of the following conditions occurs: The hash of an event is received through the NodeToNode.NewEventInPool RPC, but the NodeToNode.NewEventReceived RPC for the same hash is not received after a short delay. A miniblock header is received that contains hashes of events missing from the local minipool. Once such an event is fetched from the node containing it, the NodeToNode.NewEventReceived server-side workflow is executed. # Network Partitioning Source: https://docs.towns.com/concepts/network-partitioning In the event of complete network partitioning, where the majority of nodes reside in a single partition, the stream will continue to grow and blocks will be produced. Nodes in the minority partition will stall. As a potential future upgrade, a feature could be developed to avoid stalling nodes. In the case of partial partitioning, where some stream nodes do not see other nodes in the group serving the same stream, but a connected path among all nodes still exists, the stream will continue to progress as normal due to active minipool reconciliation. That is, nodes will actively fetch events when they observe a `NodeToNode.NewEventInPool` operation with an unknown hash. # Node to Node communication Source: https://docs.towns.com/concepts/node-node-communication ## How it works Nodes communicate with each other using the `NodeToNode` service defined in the [protocol](https://github.com/towns-protocol/towns/blob/main/protocol/internode.proto). The envelope is received through the `NodeToNode.NewEventReceived` RPC from the node that is processing the `StreamService.AddEvent` client RPC request. The node verifies if the action is permitted, by validating the signature of the event. The event is then added to the minipool of the stream and committed to local storage. A success response is sent to the `NodeToNode.NewEventReceived` RPC. A `NodeToNode.NewEventInPool` message, containing the event hash, is broadcast to the stream node, excluding the original `NodeToNode.NewEventReceived` RPC caller. Once a majority of minipools contain the event, the event is sent to any client monitoring the given stream through a Sync RPC call. * The number of minipools is calculated by counting the local minipool and the number of `NodeToNode.NewEventInPool` messages received. # Rebalancing Source: https://docs.towns.com/concepts/rebalancing In distributed systems, rebalancing is a critical process required to prevent specific nodes from being overloaded. This need arises particularly when nodes join or leave the network. It's crucial to differentiate between a node crash exiting and a node cleanly exiting, as these scenarios necessitate distinct responses. When a node exits a distributed system, it is essential to have a signal and corresponding logic to determine how to rebalance the network. The rebalance logic can be incorporated into a smart contract. However, generating a signal in a decentralized system is more challenging. Currently, there's ongoing research exploring methods for the protocol to autonomously detect crash exits through communication with other nodes. This approach aims to create a fully decentralized solution. In the interim, a hybrid process is being utilized. This process involves 'watchers' that are part of the Towns DAO. These watchers are responsible for monitoring the system and providing signals to initiate rebalancing when a node crashes or exits unexpectedly. This hybrid approach serves as a transitional solution while more autonomous and decentralized methods are being developed and refined. # Snapshotting Source: https://docs.towns.com/concepts/snapshotting ### Event Ordering Towns nodes are responsible for assembling events from the stream's `minipool` into `miniblocks` that are sealed blocks of event hashes pointing to the previous block hash. Coarse ordering of events in a given stream is achieved due to the sequential nature of block production. However, for certain events, such as a stream's membership roster, strict ordering is required. As a concrete example, take the case where Alice `joins` a space at local time `t1` then `leaves` the space at time `t2`. We should expect a read of the membership roster after the `leave` event to exclude Alice. We should also expect Alice to have been a member in the coarse grained time interval `t2 - t1` between when the `join` and `leave` events were included by a node each in a mini-block. Since different Towns Nodes may have seen the `join` and `leave` events, there is no way to guarrantee that they would be included in mini-blocks with a temporal sequence equivalent to the sequence in which the actions were taken. Therefore, to achieve strict ordering under the Towns protocol a technique called **Snapshotting** is undertaken at the block boundary for each stream. A positive side effect of Snapshotting is that disk utilisation is reduced for nodes as snapshots are updated in-place. ### Snapshot Production Snapshots are taken every N events at the block boundary, which occurs when a node that is elected by `RANDDAO` to propose the next block creates and appends a new signed `miniblock` to the stream's `minichain`. > Though miniblocks are comprised of event hashes, snapshot data is stored unhashed in miniblock to allow for fast client syncs of `state` events from miniblock which is rolled up onto Towns Chain. <img alt="Snapshotting" /> ### Data Schema Each payload type described in the protocol may be subject to snapshotting. Nodes stand ready to snapshot events for streams they are assigned t