💾 Storage: What Lives Where

This doc explains how ethlambda saves data. Especially, the split between the fork choice Store and the StorageBackend trait, what each of the eight tables holds, and which data is in-memory only.

Overview

All chain data flows through a single high-level type, the Store (crates/storage/src/store.rs), which persists it through a small pluggable key-value abstraction, the StorageBackend trait (crates/storage/src/api/traits.rs). Two backends implement the trait: RocksDB for production and an in-memory backend for tests. Everything persisted is SSZ-encoded bytes.

                        LAYERED ARCHITECTURE
                        ────────────────────

   ┌──────────────────┐          ┌──────────────────┐
   │ BlockChain actor │          │    P2P actor     │
   └────────┬─────────┘          └────────┬─────────┘
            │      (cloned Store: shared  │
            │       backend + buffers)    │
            ▼                             ▼
   ┌─────────────────────────────────────────────────┐
   │                     Store                       │
   │        crates/storage/src/store.rs              │
   │                                                 │
   │  • table selection, key encoding, SSZ codec     │
   │  • snapshot-vs-diff decisions, pruning          │
   │  • in-memory attestation buffers + state cache  │
   └────────────────────────┬────────────────────────┘
                            │ begin_read() / begin_write()
                            ▼
   ┌─────────────────────────────────────────────────┐
   │              StorageBackend trait               │
   │        crates/storage/src/api/traits.rs         │
   │                                                 │
   │        raw bytes in, raw bytes out              │
   └───────────┬─────────────────────────┬───────────┘
               │ (production)            │ (test)
               ▼                         ▼
   ┌───────────────────────┐   ┌───────────────────────┐
   │    RocksDBBackend     │   │    InMemoryBackend    │
   │  (production, one     │   │  (tests, HashMap per  │
   │   column family per   │   │   table, lost on      │
   │   table)              │   │   drop)               │
   └───────────────────────┘   └───────────────────────┘

The Store / StorageBackend Split

StorageBackend: dumb bytes

The StorageBackend trait knows nothing about consensus types. It moves raw bytes in and out of named tables:

• begin_read() returns a StorageReadView with get(table, key) and prefix_iterator(table, prefix).
• begin_write() returns a StorageWriteBatch with put_batch, delete_batch, and commit(). A batch stages puts and deletes across multiple tables and applies them atomically on commit.

The two implementations live in crates/storage/src/backend/:

Store: all the semantics

The Store owns everything the backend doesn’t: which table each datum goes to, how keys are built, SSZ encoding/decoding, when to write a full state snapshot versus a diff, and when to prune. It is the only writer to the backend.

A naming subtlety: the Store struct lives in the storage crate (crates/storage/src/store.rs), while the fork choice logic that drives it (on_block, on_tick, update_head, …) lives in crates/blockchain/src/store.rs as free functions taking &mut Store.

Store is Clone, and every field is an Arc, so clones are cheap and all clones share the same backend and the same in-memory pools. At startup (bin/ethlambda/src/main.rs) one Arc<RocksDBBackend> is opened, one Store is built from it, and clones are handed to the BlockChain and P2P actors.

                        INSIDE THE STORE
                        ────────────────

   ┌──────────────────────────── Store ────────────────────────────┐
   │                                                               │
   │   PERSISTED (via backend)         IN-MEMORY ONLY              │
   │   ──────────────────────          ──────────────────────      │
   │   ┌─────────────────────┐         ┌──────────────────────┐    │
   │   │ BlockHeaders        │         │ new_payloads         │    │
   │   │ BlockBodies         │         │  (pending aggregated │    │
   │   │ BlockProof          │         │   attestations)      │    │
   │   │ BlockRoots          │         │ known_payloads       │    │
   │   │ States              │         │  (fork-choice-active │    │
   │   │ StateDiffs          │         │   attestations)      │    │
   │   │ Metadata            │         │ gossip_signatures    │    │
   │   │ LiveChain           │         │  (raw XMSS sigs      │    │
   │   └─────────────────────┘         │   awaiting           │    │
   │                                   │   aggregation)       │    │
   │   Survives restarts.              │ state_cache (LRU)    │    │
   │                                   └──────────────────────┘    │
   │                                                               │
   │                                   Lost on restart.            │
   └───────────────────────────────────────────────────────────────┘

The Tables

The eight variants of the Table enum (crates/storage/src/api/tables.rs):

Key encoding

Three key layouts are used:

• Root-keyed tables use the 32-byte SSZ encoding of the block root (root.to_ssz()).
• Slot-prefixed tables (BlockProof, LiveChain) use encode_slot_root_key: an 8-byte big-endian slot followed by the 32-byte root. Big-endian means lexicographic key order equals numeric slot order, so pruning can iterate from the start of the table and stop at the first key past its cutoff instead of scanning everything.
• Slot-only (BlockRoots) uses encode_block_root_key: just the 8-byte big-endian slot, since the value already holds the root. This table is never pruned, so the ordering buys nothing here; it is kept only for consistency with the other slot-prefixed keys.

BlockHeaders

root → BlockHeader. Written for every block, including the genesis/anchor block, and never pruned: headers are the permanent record of the chain. Headers are also read back during state reconstruction (see State Storage).

BlockBodies

root → BlockBody. Written for every block except those with an empty body: if header.body_root == EMPTY_BODY_ROOT (the hash tree root of BlockBody::default()), nothing is stored and reads synthesize BlockBody::default(). This covers the genesis block and checkpoint sync anchors, whose bodies are either empty or unavailable. Never pruned.

BlockProof

slot ‖ root → MultiMessageAggregate. This table stores the block’s merged aggregate proof blob. It is keyed by slot ‖ root so that pruning can scan in slot order and stop early.

Stored separately from headers/bodies because the genesis block has no proof. get_signed_block synthesizes an empty proof for the slot-0 anchor only; for any other block a missing entry (a pruned finalized block) surfaces as None rather than a fabricated block.

This is the one block table that is pruned; see Pruning.

BlockRoots

slot → H256, the canonical block root at each slot. Rewritten on every head update inside update_checkpoints: block_root_index_changes walks the old and new head’s branches back to their common ancestor, deleting the slots that leave the canonical chain and writing the ones that join it. A reorg therefore touches only the affected slot range, not the whole table. Never pruned.

Backs get_signed_blocks_by_slot_range, which serves BlocksByRange requests over req/resp (crates/net/p2p/src/req_resp/handlers.rs). It does not back the RPC GET /lean/v0/blocks/:slot endpoint: that handler resolves a slot through the head state’s historical_block_hashes instead (resolve_slot in crates/net/rpc/src/blocks.rs), so a block on a side fork is reachable there only by root, never by slot.

States

root → State (full SSZ snapshot). Holds full-state snapshots only: the bootstrap anchor written at initialization, plus one anchor whenever a block crosses a SNAPSHOT_ANCHOR_INTERVAL-slot boundary. Never pruned — these anchors are the base every diff chain resolves against, so reconstruction always terminates.

The genesis validator registry is constant for the life of the chain (validators is fixed at genesis; the lean STF never mutates it), but it has no table of its own: it rides inside every States snapshot alongside config, which StateDiff reconstruction relies on (see State Storage).

StateDiffs

root → StateDiff. A parent-linked diff written for every non-genesis state. Never pruned, so together with the snapshots this preserves the full state history. See State Storage for what a diff contains and how states are rebuilt.

Metadata

String keys mapping to SSZ-encoded scalars — the Store’s own persistent fields:

config is the odd one out: init_store writes it once at bootstrap and nothing ever rewrites it afterward (it has a getter, Store::config, but no setter). Because it never changes, the Store keeps a copy in memory and reads of it never reach the backend. It is also part of the DB’s fingerprint: from_db_state refuses to resume a data directory belonging to another network (see Startup and Restore). Every other Metadata key is mutated in place as the chain progresses.

LiveChain

slot ‖ root → parent_root. A pure index for fork choice: it lets get_live_chain() build the root → (slot, parent_root) block tree without deserializing a single block. It contains the finalized anchor plus all non-finalized blocks, and is pruned as finalization advances (the finalized block itself is kept).

Presence in LiveChain is what makes a block visible to fork choice: insert_pending_block deliberately writes a block’s header/body/proof without a LiveChain entry, persisting the heavy proof data (~3 KB+) while the block waits for its parent. When the block is later processed, insert_signed_block overwrites the same keys (idempotent) and adds the LiveChain entry.

State Storage: Snapshots + Diffs

Storing a full State per block would be wasteful: most fields never change or change predictably. Instead, insert_state writes:

1. Always a StateDiff keyed by the block root, linked to its parent via base_root (the block’s parent_root).
2. Only at anchors a full snapshot into States. A block is an anchor when it crosses a SNAPSHOT_ANCHOR_INTERVAL slot boundary relative to its parent (~68 minutes at 4-second slots). This bounds any reconstruction walk to at most SNAPSHOT_ANCHOR_INTERVAL diff applications.

A StateDiff stores only what cannot be recovered elsewhere: the target slot, justified/finalized checkpoints, and the justification fields (justified_slots, justifications_roots, justifications_validators, stored in full — they are bounded by the non-finalized window, so they stay small under healthy finality). The rest is deliberately omitted:

The historical_block_hashes append is checked rather than trusted blindly: validate_history_append (crates/storage/src/state_diff.rs) rejects a diff whose appended hashes don’t match the expected slot gap or aren’t zero-filled for skipped slots, so a broken append surfaces at diff-creation time instead of corrupting a later reconstruction.

Reads go through get_state, which tries three levels:

1. An in-memory LRU cache (STATE_CACHE_CAPACITY = 32 states, keyed by block root). States are content-addressed and immutable, so the cache never needs invalidation. The common case — reading the parent state right after importing its block — is a cache hit.
2. A full snapshot in States.
3. Reconstruction: walk base_root pointers back through StateDiffs until a snapshot is found, then replay the diffs forward.

                    STATE RECONSTRUCTION
                    ────────────────────

   get_state(D): not in the cache and no snapshot → rebuild in two passes.

   Pass 1: walk backward from D, following each diff's base_root pointer
           and collecting diffs, until a block with a snapshot is found:

   ┌────────┐  base=C   ┌────────┐  base=B   ┌────────┐  base=A   ┌──────────┐
   │ diff D │ ───────▶  │ diff C │ ───────▶  │ diff B │ ───────▶  │ snapshot │
   └────────┘           └────────┘           └────────┘           │   at A   │
    (target)           (StateDiffs table)                         └──────────┘
                                                                 (States table)

   Pass 2: starting from the snapshot, apply the diffs oldest-first:

   state A ──apply B──▶ state B ──apply C──▶ state C ──apply D──▶ state D ✓

   The rebuilt state D gets its latest_block_header from the BlockHeaders
   table and is memoized in the LRU cache before being returned.

If the diff chain is broken or the target’s header is missing, get_state returns None rather than a partial state.

Write Paths: What a Block Import Persists

Block import (on_block in crates/blockchain/src/store.rs) commits a sequence of independent write batches:

                 BLOCK IMPORT WRITE SEQUENCE
                 ───────────────────────────

  on_block(signed_block)
   │
   ├─ 1. update_checkpoints()          Metadata: head,
   │      (only if the post-state      latest_justified
   │       justified a higher slot)    (+ triggers pruning)
   │
   ├─ 2. insert_signed_block()  ┐            BlockHeaders[root]
   │                            │            BlockBodies[root]    (if non-empty)
   │                            ├─one batch─ BlockProof[slot‖root]
   │                            │            LiveChain[slot‖root]
   │                            ┘
   │
   ├─ 3. insert_state()         ┐            StateDiffs[root]
   │                            ├─one batch─ States[root]         (anchors only)
   │                            ┘            (+ LRU cache insert)
   │
   └─ 4. update_head()                 Metadata: head
          (re-runs fork choice)        (+ justified/finalized if advanced,
                                          + BlockRoots diff (canonical index),
                                          + pruning on finalization)

Each numbered step is atomic on its own, but the import as a whole is not one transaction. The commit order keeps the on-disk store consistent after any prefix of these steps: the justified checkpoint written in step 1 always names an already-persisted ancestor of the imported block (the state transition only counts attestations whose roots match the state’s own historical_block_hashes, and every ancestor was fully persisted when it was imported), and the head only advances in step 4, after the block and state are durable. A crash mid-import can therefore lose the tail of the import — e.g. a persisted block and state the head does not point to yet — but never leave metadata referencing missing data. Re-importing the block is idempotent (a duplicate is skipped via has_state).

Pruning

Pruning is driven by finalization and splits into a cheap immediate phase and a deferred heavy phase.

Immediately, when finalization advances (inside update_checkpoints):

• prune_live_chain: deletes LiveChain entries below the finalized slot, keeping the finalized block itself. This keeps the fork choice working set bounded to the non-finalized chain.
• prune_gossip_signatures: drops buffered in-memory gossip signatures at or below the finalized slot.
• prune_stale_aggregated_payloads: drops in-memory aggregated payloads (both pending and known) whose target slot is at or below the finalized slot.

Deferred (prune_old_data, called after a batch of blocks has been processed):

• prune_old_block_proofs: deletes BlockProof entries below cutoff = tip_slot − BLOCK_PROOF_PRUNING_RANGE (21,600 slots, ~1 day at 4-second slots) — but only when cutoff ≤ finalized_slot, i.e. the entire pruned range lies within finalized history. Non-finalized proofs are never touched. Finalized blocks can never revert, so their proofs are not needed for fork choice, reorg safety, or re-aggregation once outside the window.

Never pruned: BlockHeaders, BlockBodies, BlockRoots, States, StateDiffs, and Metadata. Headers, bodies, the canonical slot index, and the snapshot+diff chain are the full historical record; only the proof blobs and the (non-finalized) fork choice index are disposable.

In-Memory Only (Lost on Restart)

Four Store fields never touch the backend. All are bounded buffers shared across Store clones:

The payload buffers evict FIFO when full, and redundant proofs (whose participants are a subset of an existing proof for the same attestation data) are skipped on insert.

Note that the per-validator “latest attestation” maps used by fork choice are not stored anywhere — they are derived on demand from these buffers via extract_latest_known_attestations and friends. See the attestation pipeline section of lmd_ghost.md for how attestations move between the pools.

After a restart these buffers start empty: pending attestations and un-aggregated gossip signatures are lost and must be re-collected from the network. Everything persisted in the eight tables survives.

Startup and Restore

A Store is created through one of three constructors in crates/storage/src/store.rs:

The first two funnel into init_store, which writes the anchor in one atomic batch: all six Metadata keys (time = 0, config, head = safe_target = anchor root, justified = finalized = anchor checkpoint), the anchor header, its BlockRoots entry, the body if non-empty, a full snapshot into States (the base of every future diff chain), and the anchor’s LiveChain entry.

from_db_state is the restore path: it reads config and latest_finalized from Metadata, returning None for an empty DB. A populated DB from another network is fatal instead: the finalized state’s genesis time and validator registry are compared against the genesis config, and a mismatch fails with Error::GenesisMismatch rather than being treated as empty, since writing a fresh anchor would leave the foreign chain’s rows in place to be served to peers. At startup the node prefers this path but only accepts the on-disk store if its head is at most MAX_RESUMABLE_DB_STATE_AGE = 450 slots (~30 minutes) behind the current slot; a staler DB falls through to checkpoint sync, which writes a fresh anchor on top of the existing data.

Key Files