DISC
DISC (Distributed Immutable Store for Chunks) is a storage solution developed by Swarm based on a modified implementation of a Kademlia DHT which has been specialized for data storage. Swarm's implementation of a DHT differs significantly in that it stores the content in the DHT directly, rather than just storing a list of seeders who are able to serve the content. This approach allows for much faster and more efficient retrieval of data.
Kademlia Topology and Routing
Kademlia is a distributed hash table (DHT) widely used in peer-to-peer networks such as Ethereum and Bittorent. It serves as the routing and topology foundation for communication between nodes in the Swarm network. It organizes nodes based on their overlay addresses and ensures that messages are relayed efficiently, even in a dynamic, decentralized environment.
One of the advantages of using Kademlia as a model for network topology is that both the number of forwarding "hops" required to route a chunk to its destination and the number of peer connections required to maintain Kademlia topology are logarithmic to the size of the network (a minimum of two connections is required in order to maintain Kademlia topology in case of network churn - nodes dropping in and out of the network). This makes Swarm a highly scalable system which is efficient even at very large scales.
Neighborhoods
Neighborhoods are groups of nodes which are responsible for sharing the same chunks. The chunks which each neighborhood is responsible for storing are defined by the proximity order of the nodes and the chunks. In other words, each node is responsible for storing chunks with which their overlay addresses share a certain number of prefix bits, and together with other nodes which share the same prefix bits, make up neighborhoods which share the responsibility for storing the same chunks.
Neighborhoods play a key role in providing data redundancy for chunks stored on Swarm since each node in a neighborhood will keep copies of the same chunks. The optional erasure coding feature can also be enabled for added redundancy and greater data protection.
Chunks
In the DISC model, chunks are the basic storage unit of the network layer. When a file is uploaded to Swarm, it gets broken down into chunks - pieces of at most 4KB, each with a small metadata header. Every chunk gets its own address, and each chunk is stored by the nodes whose overlay addresses are closest to that chunk address. Chunk addressing is deterministic, collision-free, and uniformly distributed, which is what gives Swarm local validity, integrity guarantee, and load balancing across nodes. There are two fundamental chunk types: content-addressed chunks and single-owner chunks.
Content-Addressed Chunks and Single-Owner Chunks
Content-addressed chunks are chunks whose address is based on the hash digest of their data. Using a hash as the chunk address makes it possible to verify the integrity of chunk data. Swarm uses the BMT hash function, a binary Merkle tree (BMT) built with Keccak256 over the 32-byte segments of the chunk data; payloads shorter than 4KB are hashed as if zero-padded up to 4KB. A content-addressed chunk has an at most 4KB payload, and its address is calculated as the hash of the 8-byte span and the BMT root of the payload. Because the address is derived from the content, a content-addressed chunk cannot be changed in place. Different content yields a different address.
For single-owner chunks on the other hand, the address is calculated as the hash of an identifier and an owner's Ethereum address. The content consists of an arbitrary data payload along with required headers: a 32-byte identifier, a 65-byte signature, and the same 8-byte span used by content-addressed chunks. The signature signs off on the identifier and the BMT hash of the span and payload, so integrity comes from the owner's signature rather than from the content itself. Validating a single-owner chunk means recovering the owner's address from the signature and checking the hash of the identifier and that address against the chunk address, which is in effect, an authentication that the owner has write access to that address.
Single-owner chunks form the basis for feeds. A feed chunk is a single-owner chunk whose identifier is the hash of a feed topic and an index, so each update is published as a new chunk at a new, deterministically derivable address. A reader can therefore find the latest update from the owner's address and the topic alone, even though no individual chunk is ever overwritten, meaning the chunk store itself remains immutable.
| Content-addressed chunk (CAC) | Single-owner chunk (SOC) | |
|---|---|---|
| Address derived from | Hash of the 8-byte span and the BMT root of the payload | Hash of the 32-byte identifier and the owner's Ethereum address |
| Integrity attested by | The content itself — the address is the hash of the content | The owner's signature over the identifier and the BMT hash of span and payload |
| Chunk mutable? | No — different content yields a different address | No — signing a second payload for the same identifier makes network behaviour unpredictable; mutability is achieved at the feed layer |
| Max payload | 4 KB + 8-byte span header | 4 KB + 105 bytes of headers (identifier, signature, span) |
| Basis for | File and manifest data | Feeds (a mutable resource resolved from owner address + topic) |
Push-Sync, Pull-Sync, and Retrieval Protocols
When a file is first uploaded to Swarm, it gets broken down by the uploading Bee node chunks which are then distributed amongst other Bee nodes in the Swarm network. Chunks get distributed to the target neighborhood by the push-sync protocol. Once a chunk reaches its destination, it will then be duplicated and synced to other nodes in order to achieve data redundancy through the pull-sync protocol. The pull-sync protocol operates continuously as nodes enter or exit the network – ensuring that data redundancy is always maintained. When a client node requests a file for download, its request gets forwarded by the retrieval-protocol to all the nodes storing the relevant chunks, and then those chunks get returned to the requesting node and the file gets reconstructed from its constituent chunks.