Introduction

ethlambda is a minimalist, fast and modular implementation of the Lean Ethereum consensus client, written in Rust.

This book collects the design notes and operator-facing references for ethlambda. It is split into two parts:

• Consensus explains how the chain advances: the slot and interval structure that schedules every validator duty, the 3SF-mini justification and finalization rules, and the LMD-GHOST fork choice algorithm. These documents are implementation-agnostic; ethlambda-specific behaviour is called out in blockquotes.
• Operations documents observable surfaces of a running node: Prometheus metrics, checkpoint sync, and the fork choice visualization served by the API.

For build and contribution instructions, see the README and CONTRIBUTING.md in the repository.

Visual references

Two standalone HTML infographics ship alongside this book and are copied verbatim into the rendered output:

• 3SF-mini infographic
• ethlambda architecture infographic

Community

• Telegram: ethlambda group, where we post daily updates; drop by to ask questions or chat about anything Lean-related.
• X (Twitter): @ethlambda_lean for occasional updates.
• Weekly community call: every Friday, streamed live on @class_lambda; the call link is posted on Telegram beforehand.
• Ecosystem coordination: the PQ Interop calls on ethereum/pm cover cross-client Lean Ethereum work and related updates; the meeting links are posted on each issue.

Related projects

ethlambda is one of several Lean Ethereum consensus clients under active development. For comparison and cross-client testing:

• zeam (Zig)
• ream (Rust)
• qlean (C++)
• grandine (Rust)
• gean (Go)
• Lantern (C)

Slots and Intervals

A Lean Chain slot has a duration of 4 seconds and is divided in 5 intervals of 800 ms. Every duty a validator owes the chain is due in one of them:

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │   block    │    vote    │    vote    │safe target │    head    │
    │  proposal  │propagation │aggregation │computation │   update   │
    └────────────┴────────────┴────────────┴────────────┴────────────┘
     ◄───────────── gossiped ─────────────▶ ◄───── local only ───────▶

The grid comes from a genesis timestamp every node shares, so the schedule needs no coordination messages: a node reads its clock, works out which interval it is in, and knows which duty is due. The order is a dependency chain, since each interval consumes what the previous one produced. A duty that overruns its interval is not rescheduled: it lands late, and the slot moves on without it.

In ethlambda: the intervals are the SlotInterval variants in crates/blockchain/src/lib.rs, and their length comes from MILLISECONDS_PER_INTERVAL and INTERVALS_PER_SLOT in crates/common/types/src/constants.rs.

Interval 0: Block proposal

A block proposer, selected in a round-robin fashion (slot % num_validators), proposes a new block and gossips it to the network. Right before building the block, the proposer merges their “new attestations buffer” into their fork-choice view. They then include attestations that the proposer has recently seen into their block. Other validators verify the block and its contents, and merge the votes it includes into their fork-choice view. After importing a block, all validators recompute their head, and update the latest finalized and justified checkpoints according to the block’s post-state.

A block body carries at most MAX_ATTESTATIONS_DATA aggregated attestations: distinct (slot, head, target, source) tuples, each paired with a bitfield naming the validators bound to it. Genesis occupies slot 0, so proposals start at slot 1, and nothing forces a slot to be filled: a proposer that is offline or too slow leaves an empty slot, and the next block simply points its parent root at an older block.

In ethlambda: block proposal is merged into the previous slot’s head-update interval: the proposer advances its store to the next slot, builds the block there, and holds publication until the slot boundary. That buys the build one extra interval of headroom and leaves no actor work at the block-proposal tick itself.

Interval 1: Vote propagation

Validators gossip their votes for the block they consider to be the head of the chain, and append to it a (source, target) finality vote. These votes are in aggregation subnets and are imported by aggregators. Aggregators verify the votes in their subnet and store them for later aggregation.

In ethlambda: a validator’s subnet is validator_index % attestation_committee_count, and a node only aggregates for subnets it subscribed to at startup. Aggregation is also gated on the aggregator role, seeded by --is-aggregator and flippable at runtime through the admin API. A chain whose validators all decline the role still gossips votes and logs them as processed, but no aggregate is ever produced, so every block is empty and the chain never justifies.

Interval 2: Vote aggregation

Aggregators aggregate the votes they have received and gossip the resulting aggregated attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a “new attestations buffer”.

Aggregation earns its own interval because collapsing a subnet’s worth of XMSS signatures into one proof is the heaviest recurring computation in the client. It is also what makes a block affordable, since a block carrying raw votes would need one full XMSS signature per voter, quickly going over the network bandwidth limit.

In ethlambda: the proofs run on an off-thread worker so the blockchain actor’s message loop stays responsive, and a session may start up to EARLY_AGGREGATION_WINDOW before the interval boundary once two thirds of the signatures are in. At the session’s soft deadline the actor stops handing out new jobs, but a proof already in flight finishes and publishes late rather than being discarded.

Interval 3: Safe target computation

Validators compute the safe target they’ll use when deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot.

It is LMD-GHOST again, but run over just the votes that arrived this slot and with a two-thirds weight threshold, where head selection applies none. The safe target therefore sits at or behind the head and advances only once a branch is backed by a supermajority. Deriving targets from it is what stops 3SF-mini from justifying a branch the network has not visibly converged on.

Interval 4: Head update

Validators merge the aggregated attestations they have in their “new attestations buffer” into their fork-choice view, and recompute their head.

This is the slot’s second and last promotion point; the first is the proposer’s, just before it builds. Until a vote is promoted it carries no weight in head selection, which is what keeps a validator’s fork-choice view from shifting under it mid-slot. The safe target is the exception: it reads the unpromoted buffer directly, which is how it stays a view of this slot alone. See why staged promotion for the reasoning.

3SF-mini: Justification & Finalization

ethlambda uses 3SF-mini (Three-Stage Finality, minimal version) for justification and finalization. Unlike the Ethereum Beacon Chain’s epoch-based Casper FFG, 3SF-mini operates at the slot level: any slot can be justified, not just epoch boundaries.

Quick Example: Three Slots to Finality

4 validators, slot N already finalized and justified.

                             source  target
                                │       │
                                ▼       ▼
    Slot N        ──[ N-2 ]──[ N-1 ]──[ N ]
                       F        J       H

                                     source    target
                                        │         │
                                        ▼         ▼
    Slot N+1      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]
                       F        F       J         H

                                               source     target
                                                  │          │
                                                  ▼          ▼
    Slot N+2      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]────[ N+2 ]
                       F        F       F         J          H

    H = head    J = justified    F = finalized

At each slot, validators vote for the newest block as their target, citing the latest justified checkpoint as their source:

• Slot N+1: Votes source=N, target=N+1. Three of four vote (3×3=9 >= 2×4=8), so N+1 is justified.
• Slot N+2: Votes source=N+1, target=N+2. Three of four vote, so N+2 justified. N+1 and N+2 are consecutive justifiable slots and both are justified, so N+1 is finalized.

In the ideal case, each block carries attestations that justify the parent slot and finalize the one before it. In practice, forks, missed slots, and delayed votes can break this cadence. The rest of this document explains the rules that make this work, and what happens when things go wrong.

Concepts

Justification via Supermajority

A checkpoint becomes justified when at least two-thirds of validators attest to it as a target:

                   JUSTIFICATION
                   ─────────────

    Validators:  V0  V1  V2  V3  V4  V5  V6  V7  V8
                  │   │   │   │   │       │   │
                  └───┴───┴───┴───┴───────┴───┘
                              │
                    7 out of 9 votes
                  (3×7=21 >= 2×9=18) ✓
                              │
                              ▼
                     ┌──────────────┐
                     │ Checkpoint C │
                     │ JUSTIFIED ✓  │
                     └──────────────┘

The threshold is computed as: 3 × vote_count >= 2 × validator_count

In ethlambda: Justification and finalization are processed inside process_attestations() in crates/blockchain/state_transition/src/lib.rs, called from process_block(). The supermajority check is 3 * vote_count >= 2 * validator_count.

Attestations must also pass validity checks before they count:

• Source checkpoint must already be justified
• Target must not already be justified
• Neither source nor target may have a zero-hash root
• Source slot < Target slot (time flows forward)
• Both checkpoints must reference known blocks
• Target slot must be justifiable per the 3SF-mini schedule (see below)

The Justifiability Schedule

Not every slot can be justified, only slots at specific distances from the last finalized slot. This is the novel part of 3SF-mini.

A slot is justifiable if delta = slot - finalized_slot matches any rule:

In ethlambda: The function slot_is_justifiable_after(slot, finalized_slot) in crates/blockchain/state_transition/src/lib.rs implements this check. It uses isqrt() for perfect square detection and the identity 4n(n+1) + 1 = (2n+1)² for pronic number detection.

    ┌───────────────────────────────────────────────────────┐
    │             JUSTIFIABILITY RULES                      │
    │                                                       │
    │  Rule 1:  delta ≤ 5          (always justifiable)     │
    │                                                       │
    │  Rule 2:  delta = n²         (perfect squares)        │
    │           1, 4, 9, 16, 25, 36, 49, 64, 81, 100, ...   │
    │                                                       │
    │  Rule 3:  delta = n(n+1)     (pronic numbers)         │
    │           2, 6, 12, 20, 30, 42, 56, 72, 90, 110, ...  │
    │                                                       │
    └───────────────────────────────────────────────────────┘

Visualizing the first 40 slots after finalization (✓ = justifiable):

    delta: 0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20
           ✓  ✓  ✓  ✓  ✓  ✓  ✓  ·  ·  ✓  ·  ·  ✓  ·  ·  ·  ✓  ·  ·  ·  ✓
           ╰─ delta ≤ 5 ──╯  2×3      3²       3×4         4²          4×5

    delta: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
           ·  ·  ·  ·  ✓  ·  ·  ·  ·  ✓  ·  ·  ·  ·  ·  ✓  ·  ·  ·  ·
                       5²             5×6               6²

Key property: Gaps between justifiable slots grow, but never become infinite. As more time passes since finalization, the network gets progressively wider windows to accumulate votes. This creates a natural backpressure: if the network is struggling to reach a two-thirds majority (e.g., due to partitions or validator dropouts), the increasing gaps give more time for the supermajority to form.

Finalization

A justified checkpoint becomes finalized when it is the source of a justification whose target is the next justifiable slot. In other words, there must be no justifiable slots between source and target: the two must be consecutive entries in the justifiability schedule.

In ethlambda: The try_finalize() function iterates over slots between source and target and calls slot_is_justifiable_after on each. If any slot is justifiable, finalization fails (source and target aren’t consecutive). The check uses original_finalized_slot (the finalized slot at the start of block processing), not the current one, since finalization can advance mid-processing.

                    FINALIZATION CHECK
                    ──────────────────

    Example 1: Finalization FAILS

    Finalized=10   Source=13 (justified)   Target=16 (justified)

    [ 10 ] · · · [ 13 ]  14   15  [ 16 ]
                          ▲    ▲
                          │    └── delta=5 ≤ 5 → justifiable!
                          └────── delta=4 ≤ 5 → justifiable!

    Justifiable slots exist between S and T → NOT FINALIZED ✗
    (13 and 16 are not consecutive justifiable slots)


    Example 2: Finalization SUCCEEDS

    Finalized=10   Source=16 (justified)   Target=19 (justified)

    [ 10 ] · · · [ 16 ]  17   18  [ 19 ]
                          ▲    ▲
                          │    └── delta=8 → not justifiable ✓
                          └────── delta=7 → not justifiable ✓

    No justifiable slots between S and T → S is FINALIZED ✓
    (16 and 19 are consecutive: delta=6=2×3, then delta=9=3²)

The reasoning: if a justifiable slot exists between source and target, validators could have directed their votes to that intermediate slot instead, potentially on a different fork. By requiring source and target to be consecutive justifiable slots, the protocol ensures that no alternative justification path can exist between them.

Justifiable Slot Backoff

The justifiability schedule acts as a backoff mechanism to increase finalization rate during periods of asynchrony. By “diluting” the possible targets of a justification vote (via the slot_is_justifiable_after function), the protocol increases the window during which votes for a given slot can be included, improving the chances of achieving the required two-thirds majority.

Since finalization requires two consecutively justifiable slots to both be justified, this backoff isn’t immediately reset after finalization occurs; it only lowers over time when synchrony is restored.

Example: Extended asynchrony with gradual recovery.

    F=0. Justifiable slots grow sparser as delta increases:

    delta ≤ 5:     0   1   2   3   4   5                  (gap = 1)
    delta 6–20:    6       9       12          16          20   (gap = 3–4)
    delta 20–36:   20          25          30              36   (gap = 5–6)
    ...
    delta ~1000:   900     930     961     992      1024        (gap = 30–32)
                   30²    30×31   31²    31×32      32²

Phase 1: Long asynchrony, slow progress.

    Validators vote, but with many justifiable targets, votes scatter
    and no single slot reaches >=2/3. As gaps widen, votes concentrate.

    Near slot 1000, the 32-slot gap between 992 and 1024 means
    no competing justifiable target exists for 32 slots after 992.
    All votes funnel toward 1024 once it is built.

Phase 2: Slot 992 finalized.

    Slot 992 justified (source = earlier justified slot).
    Slot 1024 justified (source = 992).

    slot:  0  ...  992              1024
           F        J    ·······     J
                    ▲                ▲
                 source ──────────▶ target

    Slots 993–1023: any justifiable from F=0?
      Perfect squares? 31²=961 (before), 32²=1024 (boundary). None.
      Pronic? 31×32=992 (boundary), 32×33=1056 (after). None.
    No justifiable slots between them → slot 992 FINALIZED ✓

Phase 3: Partial reset. Backoff shrinks but doesn’t vanish.

    New F=992. Justifiable slots shift:

    slot: 992 993 994 995 996 997 998 ··· 1001 ··· 1004 ··· 1008 ··· 1022 ··· 1028
           F   ✓   ✓   ✓   ✓   ✓   ✓      ✓       ✓       ✓       ✓       ✓
              ╰── delta ≤ 5 ──╯  2×3     3²      3×4     4²      5×6     6²

    Dense slots 993–998 are already in the past!
    Near the current slot (~1024), justifiable slots are ~6 apart:

    ...  1022     1028     1034     1041  ...
         δ=30     δ=36     δ=42     δ=49
         5×6      6²       6×7      7²
         └──6──┘  └──6──┘  └──7──┘

    Gaps shrank from 32 → 6, but didn't reset to 1.

Phase 4: Further finalization closes the gap.

    Justify 1022 and 1028, finalize 1022. New F=1022.

    From F=1022, at slot ~1028 (delta = 6):

    slot:  1022  1023  1024  1025  1026  1027  1028
            F     ✓     ✓     ✓     ✓     ✓     ✓
                  ╰────── delta ≤ 5 ──────╯    2×3

    Gaps are back to 1. Fast finalization resumes.

    Summary of gradual recovery:

    ┌───────────────────┬──────┬───────┬───────┬──────────────┐
    │ Finalization step │  F   │ Head  │ Delta │ Nearby gaps  │
    ├───────────────────┼──────┼───────┼───────┼──────────────┤
    │ Before any        │    0 │ ~1000 │ ~1000 │ 31–32        │
    │ After 1st (992)   │  992 │ ~1024 │   ~32 │ 6–7          │
    │ After 2nd (1022)  │ 1022 │ ~1028 │    ~6 │ 1            │
    └───────────────────┴──────┴───────┴───────┴──────────────┘

    Each finalization step reduces the delta between the finalized
    slot and the chain head, progressively tightening the gaps.

When finalization advances, the following cleanup occurs:

• justified_slots window shifts forward (old slots pruned)
• LiveChain entries for finalized slots are pruned
• Gossip signatures and aggregation proofs for finalized blocks are cleaned up
• Future fork choice runs start from the finalized slot’s successor

In ethlambda: The justified_slots bitlist uses relative indexing (index 0 = finalized_slot + 1). When finalization advances, shift_window() in crates/blockchain/state_transition/src/justified_slots_ops.rs drops the now-finalized prefix. The attestation target is also walked back to the nearest justifiable slot via slot_is_justifiable_after in crates/blockchain/src/store.rs.

End-to-End: From Head Selection to Finalization

This section connects LMD-GHOST fork choice with 3SF-mini. The quick example above showed the happy path; here we focus on what happens when things go wrong.

Recap: Attestation Anatomy

Each attestation carries three checkpoints, each determined by a different mechanism:

    ┌────────────────────────────────────────────────────────────────┐
    │                       ATTESTATION                              │
    │                                                                │
    │  head    Newest block the validator sees                       │
    │          ← LMD-GHOST with min_score = 0                        │
    │                                                                │
    │  target  Block the validator wants justified next              │
    │          ← Derived from safe target, walked back to nearest    │
    │            justifiable slot (feeds into 3SF-mini)              │
    │                                                                │
    │  source  Latest justified checkpoint                           │
    │          ← Read from store state                               │
    └────────────────────────────────────────────────────────────────┘

The safe target is computed by running LMD-GHOST with a two-thirds vote threshold. Only blocks backed by a supermajority qualify, so the safe target is always at or behind the head. The attestation target is derived by walking back from the head toward the safe target (max 3 steps), then to the nearest justifiable slot. See Safe Target Selection for details.

In ethlambda: get_attestation_target() in crates/blockchain/src/store.rs implements this walk-back. JUSTIFICATION_LOOKBACK_SLOTS = 3 provides a liveness guarantee: even if the safe target is stuck, the target eventually advances once the head moves far enough ahead.

Lagging Safe Target (Fork with Delayed Convergence)

When validators disagree about the head, the safe target lags behind: no single branch has two-thirds support. This delays justification until the fork resolves.

    Setup: 9 validators, finalized=100, justified=101
    Safe target threshold: >=6 votes (2/3 of 9)

Slots 102–103: Fork splits votes. No progress.

                         ┌──[ B102a ]──[ B103a ]     V0–V4 (5)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]     V5–V8 (4)

Neither branch clears two-thirds → safe target stuck at B101. Walk-back from head always lands on source (B101). No attestation can advance justification.

Slot 104: V7 and V8 switch sides. Fork resolves.

V7 and V8 receive B102a (delayed by the partition) and switch to the a-branch.

                         ┌──[ B102a ]──[ B103a ]──[ B104a ]     V0–V4, V7, V8 (7)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]──[ B104b ]     V5–V6  (2)

B102a subtree now has 7 votes >= 6 → safe target = B102a. Walk-back from B104a lands on B102a (2 steps). Slot 102 is justifiable (delta=2 ≤ 5).

    source=101 ──▶ target=102    7/9 votes → 3×7=21 >= 2×9=18 → JUSTIFIED ✓
    Finalization: no slots between 101 and 102 → 101 FINALIZED ✓

After slot 104: finalized=101, justified=102.

Slots 105–106: Full convergence and recovery.

All 9 validators on the a-branch. Slot 105: target=B104a → B104a JUSTIFIED. But finalization fails: slot 103 (between source=102 and target=104) is justifiable but was never justified (lost in the fork).

Slot 106: target=B105a → B105a JUSTIFIED. No justifiable slots between 104 and 105 → 104 FINALIZED. Finalization jumped from 101 to 104, skipping 102 and 103.

    FORK WITH DELAYED CONVERGENCE
    ═════════════════════════════

    Slot:     100   101   102   103   104   105   106
    Status:    F     J     ·     ·     ·     ·     ·
                                fork ──────┤
                                          resolves
    Head:      ·    B101  B102a B103a B104a B105a B106a
    Safe:      ·    B100  B101  B101  B102a B104a B105a
                          stuck ─────┘  ▲
                                        │
                           V7+V8 switch, safe target unsticks

    Justified:  ·    101   ─     ─    102   104   105
    Finalized:  ·     ·    ─     ─    101   ─     104
                                                   ▲
                              finalization jumps ──┘
                               (102,103 skipped; 103 was never justified)

Comparison with Casper FFG

Both 3SF-mini and Casper FFG are finality gadgets built on the same foundation: supermajority links between checkpoints. They differ fundamentally in their unit of time and what that implies for validator participation. For a thorough treatment of Casper FFG as used in Ethereum, see the eth2book chapter on Casper FFG.

Slots vs Epochs: The Core Architectural Split

3SF-mini: Every Validator, Every Slot

In 3SF-mini, all validators vote in every slot. A checkpoint can be justified at any slot (subject to the justifiability schedule), and finalization can happen as soon as two consecutive justifiable slots are both justified.

    3SF-mini  (4-second slots, 4 validators)

    Slot 100        Slot 101        Slot 102        Slot 103
    ┌───────┐       ┌───────┐       ┌───────┐       ┌───────┐
    │V0 V1  │       │V0 V1  │       │V0 V1  │       │V0 V1  │
    │V2 V3  │       │V2 V3  │       │V2 V3  │       │V2 V3  │
    └───┬───┘       └───┬───┘       └───┬───┘       └───┬───┘
        │               │               │               │
     4 votes         4 votes         4 votes         4 votes
     per slot        per slot        per slot        per slot

    Every validator participates in every slot.
    >=2/3 threshold checked per-slot → can justify any slot.

This is simple and fast, but it means every validator must produce and verify a vote every slot. The total message load scales as validators × slots.

Casper FFG: Validators Split Across an Epoch

Ethereum’s beacon chain has ~1,000,000 active validators. Having all of them vote every 12-second slot would be unmanageable. Instead, Casper FFG groups 32 slots into an epoch, and splits the validator set across the slots within it:

    Casper FFG  (12-second slots, 32 per epoch, ~900k validators)

    Epoch N
    ┌─────────────────────────────────────────────────────────────┐
    │ Slot 0     Slot 1     Slot 2    ...    Slot 30    Slot 31   │
    │ ┌──────┐   ┌──────┐   ┌──────┐        ┌──────┐   ┌──────┐   │
    │ │~28125│   │~28125│   │~28125│  ...   │~28125│   │~28125│   │
    │ │valids│   │valids│   │valids│        │valids│   │valids│   │
    │ └──┬───┘   └──┬───┘   └──┬───┘        └──┬───┘   └──┬───┘   │
    │    │           │           │               │           │    │
    └────┼───────────┼───────────┼───────────────┼───────────┼────┘
         └───────────┴───────────┴───┬───────────┴───────────┘
                                     │
                            All ~900k votes
                          collected over 32 slots
                                     │
                                     ▼
                              Epoch checkpoint
                              (first slot of epoch)

    Each validator attests exactly ONCE per epoch.
    The full >=2/3 tally is only meaningful at epoch boundaries.

Each validator is shuffled into a committee assigned to one specific slot. Within that slot, the committee may be further split (up to 64 sub-committees) for parallel aggregation. The result: each validator only attests once per epoch, and the network processes ~28,000 attestations per slot instead of ~900,000.

The trade-off:

Epochs exist because of a scalability constraint, not a protocol-theory preference. If you could process a million votes per slot, Casper FFG wouldn’t need epochs at all. 3SF-mini sidesteps this by targeting a smaller validator set, which lets it operate at slot granularity.

Finalization Logic

Both require a chain of justified checkpoints, but the rules differ in what they check.

Casper FFG uses k-finality. The original rule (k=1) requires a direct supermajority link from a checkpoint to its immediate successor: justify epoch N+1 with source=N, and N is finalized. Ethereum generalizes this to k=2, which handles the case where the network falls slightly behind:

    Casper FFG — 1-finality (ideal case):

    Epoch N       Epoch N+1
    ┌─────┐       ┌─────┐
    │ CP  │══════▶│ CP  │       Supermajority link N → N+1
    │ J ✓ │       │     │
    └─────┘       └─────┘

    Processing this link:
      1. Epoch N+1 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (direct successor justified)


    Casper FFG — 2-finality (one epoch behind):

    Epoch N       Epoch N+1     Epoch N+2
    ┌─────┐       ┌─────┐       ┌─────┐
    │ CP  │       │ CP  │       │ CP  │
    │ J ✓ │       │ J ✓ │       │     │
    └─────┘       └─────┘       └─────┘
       │                           │
       └══════ supermajority ══════┘
               link N → N+2

    The direct link N→N+1 didn't form in time.
    Instead, a link forms from N→N+2. Processing this link:
      1. Epoch N+2 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (all intermediates are justified)

The 2-finality rule is a recovery mechanism: even if the network missed the ideal one-epoch finalization window, it gets a second chance. Ethereum tracks the justification status of the last 4 epoch boundaries to detect both cases. In practice, most finalization happens via 1-finality during normal operation; 2-finality kicks in during brief network hiccups.

3SF-mini takes a different approach entirely:

    Slot S        Slot T
    ┌─────┐       ┌─────┐
    │ CP  │──────▶│ CP  │       No justifiable slots exist
    │ J ✓ │       │ J ✓ │       between S and T
    └─────┘       └─────┘
    ∴ Slot S is FINALIZED

    Rule: Finalized when NO intermediate checkpoints could exist

Instead of checking that intermediate checkpoints are justified, 3SF-mini checks that no intermediate checkpoints could exist at all. This is a stronger guarantee: validators’ votes between source and target could only have gone to the target, since there’s nowhere else to direct them. This structural property is also why 3SF-mini doesn’t need Casper’s surround-vote slashing condition.

Casper’s k-finality is essentially a tolerance parameter: “how many epochs behind can we be and still finalize?” Ethereum chose k=2, meaning it tolerates one missed epoch. 3SF-mini doesn’t need this concept because the justifiability schedule itself adapts. Instead of tolerating missed windows, it makes the windows wider when the network is struggling.

Adaptive Backoff (unique to 3SF-mini)

Casper FFG has a fixed checkpoint every epoch, regardless of network conditions. 3SF-mini’s justifiability schedule adapts: gaps between justifiable slots grow under prolonged asynchrony (via the perfect square and pronic number rules), creating natural vote concentration when the network is struggling to reach a two-thirds majority. Casper FFG has no equivalent; its epoch spacing is the same whether the network is healthy or partitioned. See Justifiable Slot Backoff for a detailed walkthrough.

👻 LMD-GHOST fork choice algorithm

A deep dive into how the LMD-GHOST (Latest Message Driven, Greedy Heaviest Observed SubTree) fork choice algorithm works. LMD-GHOST is the fork choice rule used by Ethereum’s consensus layer and its derivatives. Each validator’s latest attestation is their single active vote, and the algorithm follows the heaviest branch at every fork.

This document is implementation-agnostic, with ethlambda-specific details called out in blockquotes marked “In ethlambda”.

Much of the conceptual framing in this document is inspired by Ben Edgington’s Eth2 Book, particularly the LMD GHOST chapter. Highly recommended reading for anyone interested in Ethereum consensus.

Background & History

The GHOST protocol was introduced by Sompolinsky and Zohar in a 2013 paper. Its core idea: instead of choosing the heaviest chain, we choose the heaviest subtree, counting orphaned blocks as evidence of support for their ancestors.

The “LMD” in LMD-GHOST stands for Latest Message Driven: only each validator’s most recent attestation counts, preventing vote amplification. LMD-GHOST is the fork choice rule used by the Ethereum Beacon Chain and Lean Ethereum.

Why Fork Choice?

In a distributed system where validators propose blocks concurrently, the blockchain can fork: two valid blocks may appear at the same slot, creating competing chains. The fork choice rule answers a critical question:

Which chain tip should I follow?

                   ┌──────────┐
             ┌────▶│ Block C  │  ← Chain tip 1
             │     │ slot 5   │
┌──────────┐ │     └──────────┘
│ Block A  │─┤
│ slot 3   │ │     ┌──────────┐
└──────────┘ └────▶│ Block D  │  ← Chain tip 2
                   │ slot 5   │
                   └──────────┘

                    Which tip should validators follow?

Every node in the network must be able to independently arrive at the same answer using only its local view of blocks and attestations. The fork choice rule is what makes this possible. It is a deterministic function from a node’s observed state to a single chain tip.

From Heaviest Chain to Heaviest Subtree

The simplest fork choice rule is heaviest chain: follow the chain tip with the most accumulated weight. This works when fork rates are low, but breaks down when honest validators fork within a common branch:

              HEAVIEST CHAIN vs HEAVIEST SUBTREE
              ──────────────────────────────────

    An attacker with 40% of stake forks at A.
    The honest majority (60%) builds on B but forks into C and D:

                    ┌───B──┬──C     V0, V1, V2 vote for C (30%)
              A ────┤      └──D     V3, V4, V5 vote for D (30%)
                    │
                    └───X──Y──Z     V6, V7, V8, V9 vote for Z (40%)

    Heaviest chain:
      Z has 40% of votes, C and D each have 30%.
      Attacker wins! ✗

    Heaviest subtree (LMD-GHOST):
      At A: B subtree has 60% (C + D), X subtree has 40%.
      Pick B. Then at B: C has 30%, D has 30% (tiebreaker).
      Honest majority wins. ✓

LMD-GHOST is strictly better when honest validators fork within a common subtree. Instead of requiring all honest validators to agree on a single chain tip (which is impossible under network delay), it aggregates their support at each level of the tree.

How Subtree Weight Works (the “GHOST” Part)

The key insight behind the “Heaviest Observed SubTree” part of LMD-GHOST: a vote for a block is implicitly a vote for all its ancestors.

When a validator attests to block F as their head, they are also expressing support for every block on the path from the root to F:

    Validator attests: head = F

    A ── B ── C ── D ── E ── F
    ▲    ▲    ▲    ▲    ▲    ▲
    │    │    │    │    │    │
    └────┴────┴────┴────┴────┘
    All ancestors implicitly supported

This is why LMD-GHOST counts the subtree weight: a block’s weight includes every attestation for any of its descendants, because those attestations implicitly endorse the ancestor too. The algorithm exploits this by walking backward from each attested head and incrementing every block along the path.

LMD: Why Only the Latest Message?

The “LMD” in LMD-GHOST stands for Latest Message Driven. Each validator’s most recent attestation is their only vote. All previous attestations are discarded.

    Validator 7's attestation history:

    Slot 10: attests to head = B     ← discarded
    Slot 11: attests to head = C     ← discarded
    Slot 12: attests to head = E     ← THIS is the active vote

    Only the slot 12 attestation counts for fork choice.

Why only the latest? Two reasons:

1. Prevents double-voting. If all messages counted, a validator could cast many attestations and amplify their influence. With LMD, each validator gets exactly one active vote regardless of how many attestations they’ve broadcast.

2. Reflects current knowledge. A validator’s latest attestation reflects their most recent view of the chain. Older attestations may reference blocks that are no longer on the best chain. Keeping only the latest ensures fork choice uses the most up-to-date information.

The fork choice store maintains a mapping of validator_index → latest attestation. When a new attestation arrives from a validator, it replaces their previous entry:

    Fork choice store (latest messages):

    ┌──────────────┬──────────────────────────────┐
    │ Validator    │ Latest Attestation           │
    ├──────────────┼──────────────────────────────┤
    │ 0            │ head=E, target=C, source=A   │
    │ 1            │ head=D, target=C, source=A   │
    │ 2            │ head=E, target=C, source=A   │
    │ 3            │ head=F, target=D, source=A   │
    │ ...          │ ...                          │
    └──────────────┴──────────────────────────────┘

    One row per validator. New attestation → overwrite row.

LMD-GHOST Step by Step

The algorithm takes a set of inputs and produces a single block root: the head of the chain.

Inputs

In ethlambda: The function is compute_lmd_ghost_head() in crates/blockchain/fork_choice/src/lib.rs. The block tree comes from the LiveChain storage index, and min_score is 0 for head selection or ⌈2V/3⌉ for safe target computation.

The Algorithm

First, accumulate weights. Each attestation “paints” the path from its head back to the start root. In the simplest form (equal-weight validators), this adds +1 to every block on the path. In systems with balance-weighted voting, the validator’s effective balance is added instead.

    Validator 0 attests to head = F

      J ─ A ─ B ─ C ─ D ─ E ─ F       (J = justified root)
          +1  +1  +1  +1  +1  +1       J is at start_slot, not counted

    Validator 1 attests to head = D

      J ─ A ─ B ─ C ─ D
          +1  +1  +1  +1

    Accumulated weights:

      Block:    J    A    B    C    D    E    F
      Weight:   ─    2    2    2    2    1    1
                │
                └ start_root (not weighted, used as the descent origin)

In ethlambda: All validators have equal weight (+1 per vote). The Ethereum Beacon Chain instead weights votes by effective balance (up to 2048 ETH).

Then, greedily descend. Starting from the start root, at each node pick the child with the most weight. Repeat until reaching a leaf:

    J ──┬── B (5)   ← pick B (higher weight)
        └── G (2)

    B ──┬── C (3)   ← pick C (higher weight)
        └── H (2)

    C ──── D (3)    ← only child, continue

    D ── (no children) → HEAD = D!

Children below min_score are ignored during the descent. With min_score = 0 (normal head selection) all children are visible. With a higher threshold, only branches with strong support are followed. This is used for safe target selection.

The Tiebreaker

When two children have exactly equal weight, a deterministic tiebreaker is needed. Without one, different nodes could pick different heads from the same data, breaking consensus. The tiebreaker is lexicographically higher block root hash, i.e., higher hash value wins.

    Equal weight scenario:

        Parent
        │
    ┌───┴───┐
    B (3)   C (3)         ← Equal weight!
    root:   root:
    0x3a..  0x7f..        ← 0x7f > 0x3a, so pick C

The choice of “higher hash wins” is a convention. Any deterministic rule would work; what matters is that all nodes apply the same one.

Worked Example: Head Selection

Consider a network with 5 validators (indices 0–4) and the following block tree rooted at the justified checkpoint J at slot 10:

                          BLOCK TREE
                          ──────────

Slot 10     ┌──────┐
(justified) │  J   │ ← Justified checkpoint (start_root)
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │
         └──────┘    └──────┘

Latest attestations (one per validator):

Accumulate weights by walking backward from each attested head, adding +1 per block (stopping at J’s slot):

    V0 (head=D):  D+1  B+1  A+1
    V1 (head=D):  D+1  B+1  A+1
    V2 (head=E):  E+1  C+1  A+1
    V3 (head=E):  E+1  C+1  A+1
    V4 (head=E):  E+1  C+1  A+1

Greedily descend from J, always picking the heaviest child:

    Start at J
      └─▶ A (only child, weight 5)
           ├── B (weight 2)
           └── C (weight 3)  ← Pick C (3 > 2)
                └─▶ E (only child, weight 3)
                     └─▶ No children → HEAD = E ✓

Result: The canonical head is Block E. Even though both branches have the same depth, the C→E branch has 3 votes vs B→D’s 2 votes.

                          RESOLVED HEAD
                          ─────────────

Slot 10     ┌──────┐
            │  J   │
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │ ✓ canonical
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │ ✓ canonical (weight 3 > 2)
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │ ★ HEAD
         └──────┘    └──────┘

What If a Vote Changes?

Suppose validator 1 now sees block E and switches their attestation from D to E:

    Before:  V0=D, V1=D, V2=E, V3=E, V4=E   → Head = E (3 vs 2)
    After:   V0=D, V1=E, V2=E, V3=E, V4=E   → Head = E (4 vs 1)

    The head didn't change, but the margin increased from 1 to 3.
    If instead V2 and V3 had switched to D:

    After:   V0=D, V1=D, V2=D, V3=D, V4=E   → Head = D (4 vs 1)

    The head reorgs from E to D.

Fork Choice vs Finality

An important conceptual distinction: LMD-GHOST provides fork choice, not finality.

LMD-GHOST gives the network a way to agree on the current head of the chain at any moment, but the head can change. A block selected by fork choice today could be reorged away tomorrow if attestations shift. LMD-GHOST alone provides no guarantee that any block is permanent.

Finality, the guarantee that a block can never be reverted, comes from a separate mechanism called a finality gadget. LMD-GHOST is designed to compose with any finality gadget (e.g., Casper FFG in the Ethereum Beacon Chain, or 3SF-mini in Lean Ethereum).

    ┌────────────────────────────────────────────────────┐
    │                 CONSENSUS = TWO LAYERS             │
    │                                                    │
    │  ┌─────────────┐        ┌──────────────────────┐   │
    │  │  LMD-GHOST  │        │  Finality Gadget     │   │
    │  │             │        │                      │   │
    │  │ "Which tip  │        │ "Which blocks are    │   │
    │  │  is best    │        │  permanent and can   │   │
    │  │  right now?"│        │  never be reverted?" │   │
    │  │             │        │                      │   │
    │  │ Dynamic,    │        │ Monotonic, only      │   │
    │  │ can reorg   │        │ moves forward        │   │
    │  └──────┬──────┘        └──────────┬───────────┘   │
    │         │                          │               │
    │         └──────────┬───────────────┘               │
    │                    ▼                               │
    │         ┌──────────────────┐                       │
    │         │  Full Consensus  │                       │
    │         └──────────────────┘                       │
    └────────────────────────────────────────────────────┘

In ethlambda: The finality gadget is 3SF-mini, which operates at the slot level rather than epoch boundaries.

The two layers interact: LMD-GHOST runs its greedy descent starting from the latest justified checkpoint (not genesis). This means finality constrains fork choice: once a checkpoint is finalized, no fork choice run will ever consider blocks before it.

    ┌─────────┐         ┌─────────┐         ┌──── ...
    │FINALIZED│────────▶│JUSTIFIED│────────▶│  fork choice
    │ slot 50 │         │ slot 55 │         │  runs here
    └─────────┘         └─────────┘         └──── ...
         │                   │
         │                   └── start_root for LMD-GHOST
         │
         └── everything before this is permanent

This has a major practical benefit: finality allows aggressive pruning of the block tree. Without finality, fork choice would need to consider every block since genesis, and the tree would grow without bound. With finality, all blocks at or before the finalized checkpoint can be discarded from the fork choice’s working set.

In ethlambda: The LiveChain index (the in-memory block tree used by fork choice) is pruned every time finalization advances, keeping it bounded to only the non-finalized portion of the chain.

Attestation Pipeline

In a naive implementation, every attestation would influence fork choice the instant it arrives. This creates problems: validators with faster network connections see different heads than slower ones, and the proposer’s view of the chain could shift mid-block-construction.

Lean Ethereum solves this with a staged promotion pipeline: attestations are collected into a pending set and only promoted to the active fork choice set at designated moments. This ensures all validators operate on a consistent view.

                       ATTESTATION LIFECYCLE
                       ─────────────────────

  ┌──────────────┐       ┌──────────────────┐       ┌──────────────────┐
  │   Network    │       │    Pending       │       │    Active        │
  │  (gossip)    │──────▶│  Attestations    │──────▶│  Attestations    │
  │              │       │                  │       │                  │
  └──────────────┘       └──────────────────┘       └──────────────────┘
                                 │                          │
                          NOT used for               Used for fork choice
                          fork choice                weight calculations
                                 │                          │
                          Promoted at ─────────────▶ designated intervals
                          fixed points

In ethlambda: The two stages are called “new” and “known” attestations, held in the in-memory new_payloads and known_payloads buffers of the Store respectively. Promotion happens at tick intervals 0 (if proposing) and 4 (end of slot).

Why Staged Promotion?

The staged design serves two purposes:

1. Consistency: All validators promote attestations at the same moments, reducing divergence in head selection. Without batching, validators with faster network connections would see different heads than slower ones.

2. Proposer fairness: The proposer computes the block against a known, fixed set of attestations. If new attestations could influence fork choice mid-computation, different validators might disagree on the head.

On-Chain vs Off-Chain Attestations

Attestations arrive from two sources, and how they enter the pipeline matters:

The proposer’s own attestation enters as pending (not active) deliberately. If it were immediately active, the proposer would gain an unfair weight advantage for their own block, a circular dependency where proposing a block gives you an extra vote toward making that block canonical.

Safe Target Selection

The safe target is a conservative head computed with a high weight threshold. It constrains the target field in attestations, which feeds into 3SF-mini for justification and finalization decisions. Validators still vote for the newest head they see (regular LMD-GHOST with min_score = 0) in the head field. The safe target only affects which blocks can progress toward finality. It is computed by running the same LMD-GHOST algorithm but with a non-zero min_score in the filtering phase.

                    SAFE TARGET vs HEAD
                    ────────────────────

    Regular head (min_score = 0):
    Follow heaviest branch, even with a slim margin

             ┌── B (3 votes) ← HEAD (3 > 2)
    J ── A ──┤
             └── C (2 votes)


    Safe target (min_score = ⌈2V/3⌉):
    Only follow branches with supermajority support

    V = 5 validators, threshold = ⌈10/3⌉ = 4

             ┌── B (3 votes) ← Below threshold (3 < 4), pruned
    J ── A ──┤
             └── C (2 votes) ← Below threshold (2 < 4), pruned

    Safe target = A (no children pass threshold)

This means the safe target lags behind the head. It only advances when a branch accumulates overwhelming support, making it resistant to temporary fluctuations:

    Timeline of safe target vs head:

    Slot:    10    11    12    13    14    15    16
    Head:    J     A     B     D     D     E     F
    Safe:    J     J     J     A     A     A     D
                                                  │
                        Safe target is always ────┘
                        at or behind the head

The safe target prevents 3SF-mini from finalizing unstable branches: without it, a slim-majority fork could reach justification and finalization before the network converges. By requiring supermajority support for the target, only branches with strong consensus can progress toward finality, even though validators’ head votes freely follow the newest chain tip.

Reorgs

A reorg (reorganization) occurs when the fork choice head switches from one branch to another. This happens when a competing branch accumulates more attestation weight than the current head’s branch.

                    REORG SCENARIO
                    ──────────────

    Before (head = D):

              ┌── B ── D   ★ HEAD (weight 4)
    J ── A ──┤
              └── C ── E      (weight 3)


    New attestations arrive, 3 validators switch to E:

              ┌── B ── D      (weight 2)
    J ── A ──┤
              └── C ── E   ★ HEAD (weight 5)    ← REORG!


    The canonical chain changed from  J─A─B─D  to  J─A─C─E
    Blocks B and D are no longer canonical (but remain in the block tree).

Reorgs are normal during transient network conditions but should be rare in stable operation. They cannot cross a finalization boundary: once a block is finalized, it is permanently part of the canonical chain.

In ethlambda: Reorgs are detected by checking whether the old and new heads share a common prefix, and tracked via Prometheus metrics (lean_fork_choice_reorgs_total).

LMD-GHOST Variants

LMD-GHOST is one of several variants that have been proposed and studied. Understanding the design space helps explain why LMD was chosen.

The Ethereum consensus mini-spec originally used IMD-GHOST but switched to LMD in November 2018 due to superior stability properties.

    IMD: All attestations count         LMD: Only latest counts

    V0: slot 5 → head B                V0: slot 5 → head B  (overwritten)
    V0: slot 8 → head C                V0: slot 8 → head C  ← active
    V0: slot 11 → head E               V0: slot 11 → head E ← active

    V0 contributes 3 votes!            V0 contributes 1 vote.
    Validators who attest more          Equal influence regardless
    often have outsized influence.      of attestation frequency.

ethlambda Implementation Reference

This section covers ethlambda-specific details: scheduling, Beacon Chain differences, source code locations, and performance.

Tick-Based Scheduling

ethlambda divides time into 4-second slots, each split into 5 intervals (800 ms each), as described in Slots and Intervals. Fork choice operations are scheduled at specific intervals:

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │            │            │            │            │            │
    │IF PROPOSER:│ ALL        │ aggregators│update_safe │accept_new_ │
    │ accept new │ VALIDATORS:│ publish    │_target()   │attestations│
    │ attestation│  produce   │ aggregated │            │()          │
    │ + propose  │ attestation│ attestation│ (2/3 vote  │            │
    │ block      │            │            │ threshold) │update_head │
    │            │            │            │            │()          │
    │update_head │            │            │            │            │
    │()          │            │            │            │            │
    └────────────┴────────────┴────────────┴────────────┴────────────┘

    ◄─────────────── Slot N ──────────────────────────────────────────►

Detailed sequence:

    Interval 0 ─ Slot boundary
    │
    ├── Am I the proposer for this slot?
    │   ├── YES: promote new → known attestations
    │   │        run fork choice → update_head()
    │   │        build block using known attestations
    │   │        publish block to network
    │   └── NO:  (wait for block from proposer)
    │
    Interval 1 ─ Attestation production
    │
    ├── All validators, proposer included:
    │   └── Create attestation with:
    │       • head   = current fork choice head (newest head)
    │       • target = derived from safe_target (for 3SF-mini)
    │       • source = latest_justified checkpoint
    │       Publish attestation to gossipsub
    │
    Interval 2 ─ Aggregation
    │
    ├── Aggregators: aggregate their subnet's gossip signatures
    │   └── Publish the aggregated attestation to gossipsub
    │
    Interval 3 ─ Safe target update
    │
    ├── Recalculate safe_target using 2/3 supermajority threshold
    │   └── Only blocks with ≥ ⌈2V/3⌉ attestation weight qualify
    │       (V = total validators)
    │
    Interval 4 ─ End of slot
    │
    ├── Promote new → known attestations
    └── Run fork choice → update_head()

Differences from the Ethereum Beacon Chain

ethlambda is a lean consensus client with several simplifications compared to the Ethereum Beacon Chain:

No proposer boost. The Beacon Chain adds a “proposer boost”, a temporary weight bonus given to newly proposed blocks to prevent balancing attacks. ethlambda does not implement this. Instead, proposer fairness is handled through the two-stage attestation pipeline (the proposer’s own attestation enters as “new”, not “known”).

No balance weighting. In the Beacon Chain, a validator with 32 ETH of effective balance has more fork choice weight than one with 16 ETH. In ethlambda, every validator has exactly equal weight (1 vote = 1 unit of weight), simplifying the algorithm and analysis.

No equivocation discounting. The Beacon Chain’s fork choice detects validators who equivocate (attest to conflicting blocks in the same slot) and excludes their weight. This addresses the “nothing at stake” problem where validators can costlessly vote for multiple forks. ethlambda does not implement this in its fork choice.

Key Files

Data Flow Summary

     ┌───────────┐         ┌──────────────┐             ┌───────────────┐
     │ Gossipsub │────────▶│ New          │──(promote)─▶│ Known         │
     │ (network) │         │ Attestations │             │ Attestations  │
     └───────────┘         └──────────────┘             └───────┬───────┘
                                                                │
     ┌───────────┐                                              │
     │ LiveChain │──── { root → (slot, parent) } ───────────────┤
     │  (index)  │                                              │
     └───────────┘                                              │
                                                                ▼
                                                  ┌─────────────────┐
     ┌───────────┐                                │ compute_lmd_    │
     │ Justified │──── start_root ───────────────▶│ ghost_head()    │
     │Checkpoint │                                │                 │
     └───────────┘                                └────────┬────────┘
                                                           │
                                                    ┌──────┴──────┐
                                                    │             │
                                                    ▼             ▼
                                              ┌──────────┐ ┌───────────┐
                                              │   HEAD   │ │   SAFE    │
                                              │ (min=0)  │ │  TARGET   │
                                              └──────────┘ │ (min=2V/3)│
                                                           └───────────┘

Performance Characteristics

In practice with a small validator set and bounded non-finalized chain length, all operations complete in sub-millisecond time. The // TODO: add proto-array implementation comment in the source indicates a future optimization path: proto-array is an O(1) amortized fork choice algorithm used by most Beacon Chain clients.

HTTP API

ethlambda exposes HTTP over two independent Axum servers on separate ports, so the API and the metrics/debug surface can have different network policies:

• API server — consensus data (blocks, states, checkpoints, fork choice) and admin controls.
• Metrics & debug server — Prometheus metrics and heap-profiling endpoints. No store access.

All consensus API paths are versioned under the /lean/v0 prefix. Roots are serialized as 0x-prefixed hex strings.

Servers & Ports

If --api-port and --metrics-port are equal, all routers are merged onto a single port.

API Server (:5052)

GET /lean/v0/health

The handler emits a fixed, compact body (no whitespace):

{"status":"healthy","service":"lean-rpc-api"}

GET /lean/v0/config/spec

Protocol constants the node was built with. Keys mirror the leanSpec constant names:

{
  "MILLISECONDS_PER_SLOT": 4000,
  "INTERVALS_PER_SLOT": 5,
  "MILLISECONDS_PER_INTERVAL": 800,
  "HISTORICAL_ROOTS_LIMIT": 262144,
  "FORK_DIGEST": "12345678"
}

FORK_DIGEST is the 4-byte hex string (no 0x prefix) embedded in gossipsub topic names.

GET /lean/v0/genesis

{ "genesis_time": 1770407233, "validator_count": 16 }

validator_count is read from the head state’s validator registry. Lean validators are fixed at genesis (no churn), so it always equals the size of the genesis registry.

GET /lean/v0/states/finalized

SSZ-encoded State at the latest finalized checkpoint (Content-Type: application/octet-stream). The served state has its latest_block_header.state_root zeroed to match the canonical post-state representation the state transition produces, so checkpoint-sync peers reconstruct an identical state root. See Checkpoint Sync.

GET /lean/v0/blocks/finalized

SSZ-encoded SignedBlock at the latest finalized checkpoint. The genesis/anchor block has no stored signature, so a placeholder blank proof is synthesized and the endpoint still returns 200. Returns 404 only in the rare case where a non-genesis finalized block’s signature has been pruned below the finalized boundary and can no longer be served.

GET /lean/v0/checkpoints/justified

{ "slot": 128, "root": "0x1a2b…" }

GET /lean/v0/events

Server-Sent Events stream (Content-Type: text/event-stream) of live chain events published by the blockchain actor. Seven event types:

Payload fields mirror the Ethereum beacon-API eventstream where an analog exists: block is the block root, state the state root, and slot stands in for the beacon epoch. justified_checkpoint and aggregate are ethlambda extensions with no beacon topic.

The topic name travels only on the SSE event: line; the data: line carries the flat JSON payload. Example frame:

event: head
data: {"slot":128,"block":"0x1a2b…","state":"0x3c4d…"}

Filtering with ?topics=

A required comma-separated list of event names selects which events to stream:

curl -N 'http://127.0.0.1:5052/lean/v0/events?topics=head,finalized_checkpoint'

Valid values are exactly the event names above: head, block, justified_checkpoint, finalized_checkpoint, block_gossip, attestation, aggregate. As in the Beacon API eventstream endpoint, topics is mandatory: there is no “subscribe to everything” default; list the topics you want.

Events are fanned out over a single bounded broadcast channel shared by all topics. A client that reads too slowly skips past the events it missed: they are dropped for that subscriber rather than back-pressured onto the actor, so treat the stream as best-effort and re-sync via the blocks endpoints after a gap. A client that falls behind receives an SSE comment line : error - dropped N messages marking the gap (wire-compatible with Lighthouse) before the stream continues; re-sync via the blocks endpoints rather than trusting the skipped range. Keep-alive comments are sent periodically to hold idle connections open.

Because the ring buffer is shared, the high-rate attestation events (roughly one per validator per slot) dominate its occupancy: a subscriber’s tolerable stall is capacity / total_event_rate, not per-topic, so filtering with ?topics= narrows what you receive but does not widen the lag window against an attestation flood. Subscribers that only need low-rate topics (head, finalized_checkpoint, …) are still evicted at the aggregate rate. If real usage shows this biting, the fix is a per-topic channel split behind the event bus (the subscribe(TopicSet) API is unaffected).

GET /lean/v0/blocks/{block_id} and /header

block_id is either:

• a 0x-prefixed 32-byte hex root, or
• a decimal slot.

Slot lookups resolve through the head state’s historical_block_hashes, so only canonical blocks are reachable by slot; blocks on side forks must be addressed by their root. The /header variant returns just the BlockHeader.

Error bodies are JSON: { "error": "invalid block_id" } / { "error": "block not found" }.

GET /lean/v0/fork_choice

The fork-choice tree from the finalized root, with LMD-GHOST weights computed over the live chain and currently known attestations.

{
  "nodes": [
    { "root": "0x…", "slot": 128, "parent_root": "0x…", "proposer_index": 3, "weight": 12 }
  ],
  "head": "0x…",
  "justified": { "slot": 128, "root": "0x…" },
  "finalized": { "slot": 96,  "root": "0x…" },
  "safe_target": "0x…",
  "validator_count": 16
}

/lean/v0/fork_choice/ui serves an interactive D3.js page rendering this data. See Fork Choice Visualization.

GET /lean/v0/node/identity

{
  "version": "ethlambda/v0.1.0-main-892ad575/x86_64-unknown-linux-gnu/rustc-v1.97.1",
  "peer_id": "16Uiu2HAm7v1x…"
}

version is the full client version string, identical to what ethlambda --version prints: crate semver, git branch and short SHA, target triple, and rustc version. Baked in at compile time from CARGO_PKG_VERSION plus the vergen-git2 build metadata.

peer_id is the node’s libp2p peer ID (base58), derived from the node key and fixed for the lifetime of the process; it matches the identity the node presents to peers on the wire.

GET /lean/v0/node/syncing

{ "is_syncing": false, "head_slot": 1024, "sync_distance": 1, "finalized_slot": 986 }

is_syncing is the node’s own stateful sync decision: head-vs-wall-clock lag with hysteresis and a network-stall override, updated each tick. It is the same signal that gates validator duties and drives the lean_node_sync_status metric, so the endpoint, the gate, and the metric always agree.

sync_distance is the raw number of slots between the node’s current head and the current wall-clock slot, computed per request. Because is_syncing carries hysteresis and stall handling and is not recomputed from sync_distance, the two can point different ways near the threshold or during a network-wide stall.

GET / POST /lean/v0/admin/aggregator

Toggle the aggregator role at runtime without restarting the node (hot-standby model, ported from leanSpec PR #636).

# Read current role
curl http://127.0.0.1:5052/lean/v0/admin/aggregator
# → {"is_aggregator": true}

# Toggle role; body must be a JSON boolean
curl -X POST http://127.0.0.1:5052/lean/v0/admin/aggregator \
  -H 'content-type: application/json' -d '{"enabled": false}'
# → {"is_aggregator": false, "previous": true}

Note: Runtime toggles do not resubscribe gossip subnets, which are frozen at startup. A standby aggregator should boot with --is-aggregator=true (so subscriptions are in place), then use this endpoint to rotate duties. See the CLAUDE.md “Runtime Aggregator Toggle” notes for the operational model.

Metrics & Debug Server (:5054)

The metrics endpoint reads from the global Prometheus registry and needs no store access. See Metrics for the full list of exposed series.

Heap-profiling endpoints are backed by jemalloc’s built-in profiler and are only functional on Linux; other platforms return 501 Not Implemented. On Linux they return 500 if profiling was not enabled at startup.

Test-Driver Endpoints (Hive)

When the binary boots with HIVE_LEAN_TEST_DRIVER=1 (any of 1/true/yes), it runs in test-driver mode instead of the normal API server. The ethereum/hive lean simulator drives these endpoints to replay leanSpec fixtures over HTTP. The driver swaps its in-process Store on every fork_choice/init, so one container can replay many fixtures without restart.

Content Types

Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

A ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The exposed metrics follow the leanMetrics specification, with some metrics not yet implemented. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

PQ Signature Metrics

Block Production Metrics

lean_block_building_time_seconds intentionally deviates from the leanMetrics bucket set, which tops out at 1s. Real builds on our devnets routinely run past that, so every sample landed in +Inf and histogram_quantile reported a flat 1s ceiling. The range now covers the same span as the lean_block_proposal_attestation_build_phase_seconds phases it contains.

Fork-Choice Metrics

State Transition Metrics

Validator Metrics

Network Metrics

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Network Sizes

Gossip Arrival Timing

These histograms record the absolute distance between a gossip message’s arrival and the start of the interval it was due in, so an arrival that is early by some amount and one that is late by the same amount land in the same bucket; the counters’ position label is what tells them apart. inside means the message arrived within the interval it was due in, not merely somewhere in the right slot: an attestation for slot 10 that lands during slot 10’s interval 2 is after, not inside, since it missed the AttestationProduction interval it was actually due in.

Blocks anchor to interval 0 of their own slot and attestations to interval 1 of their data slot; both are unbounded above, so a message that never arrives close to real time can be arbitrarily late. Aggregates anchor instead to the most recent aggregation-interval boundary rather than their own data slot, since a stale-group catch-up aggregate can carry a data.slot several slots in the past; anchoring to the latest boundary bounds the delay to one slot and rules out before entirely.

Only gossip-received blocks are sampled here: blocks fetched via req/resp during sync are excluded, since sync backfill delivers blocks long after they were due and would swamp these histograms with catch-up noise rather than gossip-health signal.

The aggregate metrics do include an aggregator’s own freshly produced aggregates, which never come back over gossip; without them an aggregator would report an empty aggregate profile. The two populations are not quite the same measurement: delivery of a locally produced aggregate is held until the interval-2 boundary, so it lands near zero unless proving overran the interval, whereas a received one adds propagation on top of whenever the producer managed to publish it.

In practice the distribution is bimodal and dominated by production rather than propagation: a mode in the lowest bucket for aggregates that made their interval, plus a tail for those whose proving overran it. A late aggregate is late for every node at once, so that tail shows up on receivers too and is not evidence of a slow network. Read a rising tail as aggregation cost, and cross-check lean_pq_sig_aggregated_signatures_building_time_seconds and lean_committee_signatures_aggregation_time_seconds to confirm.

Storage

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

• timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
• agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
• proposal_combined is emitted only when this node proposes a block.

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Troubleshooting

Docker Desktop on MacOS

lean-quickstart uses the host network mode for Docker containers, which is a problem on MacOS. To work around this, enable the “Enable host networking” option in Docker Desktop settings under Resources > Network.

Checkpoint Sync

Overview

Checkpoint sync allows a new consensus node to skip replaying the entire chain from genesis. Instead, it downloads a recent finalized state from a running peer and starts from there. This mitigates long-range attacks by starting from a recent trusted checkpoint.

Usage

Checkpoint sync still requires the network config files (genesis, validators, bootnodes, etc.). The genesis config is needed to verify the downloaded state: checkpoint sync only replaces the starting state, not node configuration.

Pass the --checkpoint-sync-url flag when starting ethlambda:

ethlambda \
  --checkpoint-sync-url <URL> \
  --genesis ./network-config/config.yaml \
  --validators ./network-config/annotated_validators.yaml \
  --bootnodes ./network-config/nodes.yaml \
  --validator-config ./network-config/validator-config.yaml \
  --hash-sig-keys-dir ./network-config/hash-sig-keys \
  --node-key ./node.key \
  --node-id ethlambda_0

Where <URL> is the address of a checkpoint source (see Checkpoint Sources below).

State already on disk takes precedence over both checkpoint sync and genesis: if the data directory holds a previous run’s chain state for this network, the node resumes from it. --checkpoint-sync-url is the fallback for when there is nothing resumable on disk, or when what is there has fallen too far behind (see Restarts and Existing State). With no resumable state and no URL, the node initializes from genesis.

Checkpoint Sources

Direct peer

Any running node that serves the finalized state as SSZ can be used as a checkpoint source, not just ethlambda. For ethlambda nodes, the endpoint is /lean/v0/states/finalized.

This is the simplest option, with no additional infrastructure needed. The trade-off is that you trust a single peer to provide a correct finalized state.

Leanpoint

Leanpoint is a dedicated checkpoint sync provider. It polls multiple nodes and only serves state when 50%+ agree on finality, adding a layer of consensus validation.

This is the recommended option for production deployments since it reduces trust in any single peer.

How It Works

1. Fetch and verify: The node sends an HTTP GET to the provided URL requesting the SSZ-encoded finalized state. Once downloaded, the state is decoded and verified against the local genesis config (see Verification Checks below).

   Timeouts:

   • Connect: 15 seconds (fail fast if peer is unreachable)
   • Read: 15 seconds of inactivity that resets on each successful read, so large states can download as long as data keeps flowing

2. Initialize: The node stores the block header and the full state from the checkpoint. No block body is stored since it isn’t available from the checkpoint. The node does not need the anchor block body to participate from this point forward.

Failure and success

If any step fails (network error, decoding error, verification failure), the node logs the error and exits. There is no automatic retry; restart the node to try again. The database is not modified until verification succeeds, so a failed checkpoint sync leaves the data directory clean.

After successful initialization, the node starts normally: it connects to the P2P network and begins participating from the checkpoint slot.

Restarts and Existing State

A node restarted against a populated data directory resumes from disk rather than re-initializing, so no flag is needed to preserve the chain across a redeploy. The decision is made before any download:

The resume window is MAX_RESUMABLE_DB_STATE_AGE (450 slots, ~30 minutes at 4-second slots) measured as current_slot - head_slot. Staleness is measured against the head, not the finalized checkpoint, so a node whose head is current still resumes during a finality stall.

Beyond that window the node prefers a checkpoint when one is offered, since catching up over P2P costs more than downloading a recent state. With no URL configured there is no anchor to switch to, so the node simply runs against the data directory it was given: that is the setup that was asked for. The warning is there because range sync may not be able to close a gap this large. Peers prune block signatures past SIGNATURE_PRUNING_RANGE (21600 slots, ~1 day), so beyond that horizon they cannot serve the history the node is missing and it needs a checkpoint URL to catch up at all. The warning logs the gap so this is visible in the boot log.

When a checkpoint URL is set and every URL fails, the node exits rather than falling back to the stale state on disk. This is intentional: configuring the flag asks for a specific anchor, so an unreachable source is a misconfiguration worth surfacing at boot instead of quietly starting a node that is hours behind. Omitting the flag is how you ask for “resume whatever is on disk”; that path never exits.

To deliberately discard existing state and start over from genesis or from a checkpoint, remove the data directory first. Checkpoint sync itself writes its anchor state on top without clearing existing data.

Foreign State

Persisted state is accepted only after it is verified against the local genesis config: same GENESIS_TIME and the same validator registry (count, sequential indices, and both pubkeys per validator). The validator set is fixed at genesis, so any state of this chain must carry exactly that registry. These are the same identity checks checkpoint sync applies to a downloaded state, sharing one implementation.

If the data directory belongs to a different network, startup aborts with persisted state does not match the configured genesis: …. It is not treated as an empty directory, because initializing a new anchor on top would leave the foreign chain’s rows in place, and the slot-indexed reads behind BlocksByRange would then serve those blocks to peers. Point --data-dir at the right directory, or remove it.

Note that a genesis time comparison alone would not catch a network that was regenerated with the same GENESIS_TIME but a different validator set, which is why the whole registry is compared.

Verification Checks

All checks are performed before a downloaded checkpoint state is accepted. The genesis-identity subset (marked below) is shared with the resume-from-disk path:

HTTP errors and SSZ decoding failures are caught before verification runs.

Security Considerations

Trust model

Checkpoint sync operates under a weak subjectivity assumption. In proof of work, any node can objectively determine the canonical chain by verifying the most cumulative work. Proof of stake doesn’t have this property: validators can costlessly sign multiple forks, so a node that wasn’t online to observe the chain in real time cannot distinguish the real chain from a fabricated one using protocol rules alone.

Weak subjectivity resolves this: a new node obtains a recent trusted state through a social channel (a peer, a checkpoint provider, a block explorer) and starts from there. Nodes that are always online are unaffected because they continuously track the chain and don’t need external trust.

What you are trusting:

• The checkpoint source is honest about which state is finalized
• The state hasn’t been crafted to put you on a fork that diverged within the weak subjectivity period

What verification does protect against:

• Wrong network (genesis time mismatch)
• Wrong validator set (pubkey or count mismatch)
• Structurally invalid states (impossible slot orderings, inconsistent checkpoints)
• Corrupted data (SSZ decode failures)

What verification does not protect against:

• A checkpoint source that serves a structurally valid state on a minority fork. It will pass all checks but put you on the wrong chain. This is why the choice of checkpoint source matters.

Fork Choice Visualization

A browser-based real-time visualization of the LMD GHOST fork choice tree, served from the existing RPC server with no additional dependencies.

Endpoints

Both endpoints are served on the API port (--api-port, default 5052).

Quick Start

Local devnet

make run-devnet

The local devnet runs 3 ethlambda nodes with metrics ports 8085, 8086, and 8087. Open any of them:

• http://localhost:8085/lean/v0/fork_choice/ui
• http://localhost:8086/lean/v0/fork_choice/ui
• http://localhost:8087/lean/v0/fork_choice/ui

Standalone node

cargo run --release -- \
  --genesis ./config/config.yaml \
  --validators ./config/annotated_validators.yaml \
  --bootnodes ./config/nodes.yaml \
  --validator-config ./config/validator-config.yaml \
  --hash-sig-keys-dir ./config/hash-sig-keys \
  --node-key ./keys/node.key \
  --node-id 0 \
  --api-port 5052

Then open http://localhost:5052/lean/v0/fork_choice/ui.

Visualization Guide

Color coding

Layout

• Y axis: slot number (time flows downward)
• X axis: fork spreading — branches appear when competing chains exist
• Circle size: scaled by weight / validator_count — larger circles have more attestation support

Interactive features

• Tooltips: hover any block to see root hash, slot, proposer index, and weight
• Auto-polling: the page fetches fresh data every 2 seconds
• Auto-scroll: the view follows the head as the chain progresses

What to look for

• Single vertical chain: healthy consensus, no forks
• Horizontal branching: competing chains — check attestation weights to see which branch validators prefer
• Color transitions: blocks turning green as finalization advances
• Stalled finalization: if justified/finalized slots stop advancing, check validator attestation activity

JSON API

curl -s http://localhost:5052/lean/v0/fork_choice | jq .

Response schema:

{
  "nodes": [
    {
      "root": "0x...",
      "slot": 42,
      "parent_root": "0x...",
      "proposer_index": 3,
      "weight": 5
    }
  ],
  "head": "0x...",
  "justified": { "root": "0x...", "slot": 10 },
  "finalized": { "root": "0x...", "slot": 5 },
  "safe_target": "0x...",
  "validator_count": 8
}

💾 Storage: What Lives Where

This doc explains how ethlambda saves data. Especially, the split between the fork choice Store and the StorageBackend trait, what each of the eight tables holds, and which data is in-memory only.

Overview

All chain data flows through a single high-level type, the Store (crates/storage/src/store.rs), which persists it through a small pluggable key-value abstraction, the StorageBackend trait (crates/storage/src/api/traits.rs). Two backends implement the trait: RocksDB for production and an in-memory backend for tests. Everything persisted is SSZ-encoded bytes.

                        LAYERED ARCHITECTURE
                        ────────────────────

   ┌──────────────────┐          ┌──────────────────┐
   │ BlockChain actor │          │    P2P actor     │
   └────────┬─────────┘          └────────┬─────────┘
            │      (cloned Store: shared  │
            │       backend + buffers)    │
            ▼                             ▼
   ┌─────────────────────────────────────────────────┐
   │                     Store                       │
   │        crates/storage/src/store.rs              │
   │                                                 │
   │  • table selection, key encoding, SSZ codec     │
   │  • snapshot-vs-diff decisions, pruning          │
   │  • in-memory attestation buffers + state cache  │
   └────────────────────────┬────────────────────────┘
                            │ begin_read() / begin_write()
                            ▼
   ┌─────────────────────────────────────────────────┐
   │              StorageBackend trait               │
   │        crates/storage/src/api/traits.rs         │
   │                                                 │
   │        raw bytes in, raw bytes out              │
   └───────────┬─────────────────────────┬───────────┘
               │ (production)            │ (test)
               ▼                         ▼
   ┌───────────────────────┐   ┌───────────────────────┐
   │    RocksDBBackend     │   │    InMemoryBackend    │
   │  (production, one     │   │  (tests, HashMap per  │
   │   column family per   │   │   table, lost on      │
   │   table)              │   │   drop)               │
   └───────────────────────┘   └───────────────────────┘

The Store / StorageBackend Split

StorageBackend: dumb bytes

The StorageBackend trait knows nothing about consensus types. It moves raw bytes in and out of named tables:

• begin_read() returns a StorageReadView with get(table, key) and prefix_iterator(table, prefix).
• begin_write() returns a StorageWriteBatch with put_batch, delete_batch, and commit(). A batch stages puts and deletes across multiple tables and applies them atomically on commit.

The two implementations live in crates/storage/src/backend/:

Store: all the semantics

The Store owns everything the backend doesn’t: which table each datum goes to, how keys are built, SSZ encoding/decoding, when to write a full state snapshot versus a diff, and when to prune. It is the only writer to the backend.

A naming subtlety: the Store struct lives in the storage crate (crates/storage/src/store.rs), while the fork choice logic that drives it (on_block, on_tick, update_head, …) lives in crates/blockchain/src/store.rs as free functions taking &mut Store.

Store is Clone, and every field is an Arc, so clones are cheap and all clones share the same backend and the same in-memory pools. At startup (bin/ethlambda/src/main.rs) one Arc<RocksDBBackend> is opened, one Store is built from it, and clones are handed to the BlockChain and P2P actors.

                        INSIDE THE STORE
                        ────────────────

   ┌──────────────────────────── Store ────────────────────────────┐
   │                                                               │
   │   PERSISTED (via backend)         IN-MEMORY ONLY              │
   │   ──────────────────────          ──────────────────────      │
   │   ┌─────────────────────┐         ┌──────────────────────┐    │
   │   │ BlockHeaders        │         │ new_payloads         │    │
   │   │ BlockBodies         │         │  (pending aggregated │    │
   │   │ BlockProof          │         │   attestations)      │    │
   │   │ BlockRoots          │         │ known_payloads       │    │
   │   │ States              │         │  (fork-choice-active │    │
   │   │ StateDiffs          │         │   attestations)      │    │
   │   │ Metadata            │         │ gossip_signatures    │    │
   │   │ LiveChain           │         │  (raw XMSS sigs      │    │
   │   └─────────────────────┘         │   awaiting           │    │
   │                                   │   aggregation)       │    │
   │   Survives restarts.              │ state_cache (LRU)    │    │
   │                                   └──────────────────────┘    │
   │                                                               │
   │                                   Lost on restart.            │
   └───────────────────────────────────────────────────────────────┘

The Tables

The eight variants of the Table enum (crates/storage/src/api/tables.rs):

Key encoding

Three key layouts are used:

• Root-keyed tables use the 32-byte SSZ encoding of the block root (root.to_ssz()).
• Slot-prefixed tables (BlockProof, LiveChain) use encode_slot_root_key: an 8-byte big-endian slot followed by the 32-byte root. Big-endian means lexicographic key order equals numeric slot order, so pruning can iterate from the start of the table and stop at the first key past its cutoff instead of scanning everything.
• Slot-only (BlockRoots) uses encode_block_root_key: just the 8-byte big-endian slot, since the value already holds the root. This table is never pruned, so the ordering buys nothing here; it is kept only for consistency with the other slot-prefixed keys.

BlockHeaders

root → BlockHeader. Written for every block, including the genesis/anchor block, and never pruned: headers are the permanent record of the chain. Headers are also read back during state reconstruction (see State Storage).

BlockBodies

root → BlockBody. Written for every block except those with an empty body: if header.body_root == EMPTY_BODY_ROOT (the hash tree root of BlockBody::default()), nothing is stored and reads synthesize BlockBody::default(). This covers the genesis block and checkpoint sync anchors, whose bodies are either empty or unavailable. Never pruned.

BlockProof

slot ‖ root → MultiMessageAggregate. This table stores the block’s merged aggregate proof blob. It is keyed by slot ‖ root so that pruning can scan in slot order and stop early.

Stored separately from headers/bodies because the genesis block has no proof. get_signed_block synthesizes an empty proof for the slot-0 anchor only; for any other block a missing entry (a pruned finalized block) surfaces as None rather than a fabricated block.

This is the one block table that is pruned; see Pruning.

BlockRoots

slot → H256, the canonical block root at each slot. Rewritten on every head update inside update_checkpoints: block_root_index_changes walks the old and new head’s branches back to their common ancestor, deleting the slots that leave the canonical chain and writing the ones that join it. A reorg therefore touches only the affected slot range, not the whole table. Never pruned.

Backs get_signed_blocks_by_slot_range, which serves BlocksByRange requests over req/resp (crates/net/p2p/src/req_resp/handlers.rs). It does not back the RPC GET /lean/v0/blocks/:slot endpoint: that handler resolves a slot through the head state’s historical_block_hashes instead (resolve_slot in crates/net/rpc/src/blocks.rs), so a block on a side fork is reachable there only by root, never by slot.

States

root → State (full SSZ snapshot). Holds full-state snapshots only: the bootstrap anchor written at initialization, plus one anchor whenever a block crosses a SNAPSHOT_ANCHOR_INTERVAL-slot boundary. Never pruned — these anchors are the base every diff chain resolves against, so reconstruction always terminates.

The genesis validator registry is constant for the life of the chain (validators is fixed at genesis; the lean STF never mutates it), but it has no table of its own: it rides inside every States snapshot alongside config, which StateDiff reconstruction relies on (see State Storage).

StateDiffs

root → StateDiff. A parent-linked diff written for every non-genesis state. Never pruned, so together with the snapshots this preserves the full state history. See State Storage for what a diff contains and how states are rebuilt.

Metadata

String keys mapping to SSZ-encoded scalars — the Store’s own persistent fields:

config is the odd one out: init_store writes it once at bootstrap and nothing ever rewrites it afterward (it has a getter, Store::config, but no setter). Because it never changes, the Store keeps a copy in memory and reads of it never reach the backend. It is also part of the DB’s fingerprint: from_db_state refuses to resume a data directory belonging to another network (see Startup and Restore). Every other Metadata key is mutated in place as the chain progresses.

LiveChain

slot ‖ root → parent_root. A pure index for fork choice: it lets get_live_chain() build the root → (slot, parent_root) block tree without deserializing a single block. It contains the finalized anchor plus all non-finalized blocks, and is pruned as finalization advances (the finalized block itself is kept).

Presence in LiveChain is what makes a block visible to fork choice: insert_pending_block deliberately writes a block’s header/body/proof without a LiveChain entry, persisting the heavy proof data (~3 KB+) while the block waits for its parent. When the block is later processed, insert_signed_block overwrites the same keys (idempotent) and adds the LiveChain entry.

State Storage: Snapshots + Diffs

Storing a full State per block would be wasteful: most fields never change or change predictably. Instead, insert_state writes:

1. Always a StateDiff keyed by the block root, linked to its parent via base_root (the block’s parent_root).
2. Only at anchors a full snapshot into States. A block is an anchor when it crosses a SNAPSHOT_ANCHOR_INTERVAL slot boundary relative to its parent (~68 minutes at 4-second slots). This bounds any reconstruction walk to at most SNAPSHOT_ANCHOR_INTERVAL diff applications.

A StateDiff stores only what cannot be recovered elsewhere: the target slot, justified/finalized checkpoints, and the justification fields (justified_slots, justifications_roots, justifications_validators, stored in full — they are bounded by the non-finalized window, so they stay small under healthy finality). The rest is deliberately omitted:

The historical_block_hashes append is checked rather than trusted blindly: validate_history_append (crates/storage/src/state_diff.rs) rejects a diff whose appended hashes don’t match the expected slot gap or aren’t zero-filled for skipped slots, so a broken append surfaces at diff-creation time instead of corrupting a later reconstruction.

Reads go through get_state, which tries three levels:

1. An in-memory LRU cache (STATE_CACHE_CAPACITY = 32 states, keyed by block root). States are content-addressed and immutable, so the cache never needs invalidation. The common case — reading the parent state right after importing its block — is a cache hit.
2. A full snapshot in States.
3. Reconstruction: walk base_root pointers back through StateDiffs until a snapshot is found, then replay the diffs forward.

                    STATE RECONSTRUCTION
                    ────────────────────

   get_state(D): not in the cache and no snapshot → rebuild in two passes.

   Pass 1: walk backward from D, following each diff's base_root pointer
           and collecting diffs, until a block with a snapshot is found:

   ┌────────┐  base=C   ┌────────┐  base=B   ┌────────┐  base=A   ┌──────────┐
   │ diff D │ ───────▶  │ diff C │ ───────▶  │ diff B │ ───────▶  │ snapshot │
   └────────┘           └────────┘           └────────┘           │   at A   │
    (target)           (StateDiffs table)                         └──────────┘
                                                                 (States table)

   Pass 2: starting from the snapshot, apply the diffs oldest-first:

   state A ──apply B──▶ state B ──apply C──▶ state C ──apply D──▶ state D ✓

   The rebuilt state D gets its latest_block_header from the BlockHeaders
   table and is memoized in the LRU cache before being returned.

If the diff chain is broken or the target’s header is missing, get_state returns None rather than a partial state.

Write Paths: What a Block Import Persists

Block import (on_block in crates/blockchain/src/store.rs) commits a sequence of independent write batches:

                 BLOCK IMPORT WRITE SEQUENCE
                 ───────────────────────────

  on_block(signed_block)
   │
   ├─ 1. update_checkpoints()          Metadata: head,
   │      (only if the post-state      latest_justified
   │       justified a higher slot)    (+ triggers pruning)
   │
   ├─ 2. insert_signed_block()  ┐            BlockHeaders[root]
   │                            │            BlockBodies[root]    (if non-empty)
   │                            ├─one batch─ BlockProof[slot‖root]
   │                            │            LiveChain[slot‖root]
   │                            ┘
   │
   ├─ 3. insert_state()         ┐            StateDiffs[root]
   │                            ├─one batch─ States[root]         (anchors only)
   │                            ┘            (+ LRU cache insert)
   │
   └─ 4. update_head()                 Metadata: head
          (re-runs fork choice)        (+ justified/finalized if advanced,
                                          + BlockRoots diff (canonical index),
                                          + pruning on finalization)

Each numbered step is atomic on its own, but the import as a whole is not one transaction. The commit order keeps the on-disk store consistent after any prefix of these steps: the justified checkpoint written in step 1 always names an already-persisted ancestor of the imported block (the state transition only counts attestations whose roots match the state’s own historical_block_hashes, and every ancestor was fully persisted when it was imported), and the head only advances in step 4, after the block and state are durable. A crash mid-import can therefore lose the tail of the import — e.g. a persisted block and state the head does not point to yet — but never leave metadata referencing missing data. Re-importing the block is idempotent (a duplicate is skipped via has_state).

Pruning

Pruning is driven by finalization and splits into a cheap immediate phase and a deferred heavy phase.

Immediately, when finalization advances (inside update_checkpoints):

• prune_live_chain: deletes LiveChain entries below the finalized slot, keeping the finalized block itself. This keeps the fork choice working set bounded to the non-finalized chain.
• prune_gossip_signatures: drops buffered in-memory gossip signatures at or below the finalized slot.
• prune_stale_aggregated_payloads: drops in-memory aggregated payloads (both pending and known) whose target slot is at or below the finalized slot.

Deferred (prune_old_data, called after a batch of blocks has been processed):

• prune_old_block_proofs: deletes BlockProof entries below cutoff = tip_slot − BLOCK_PROOF_PRUNING_RANGE (21,600 slots, ~1 day at 4-second slots) — but only when cutoff ≤ finalized_slot, i.e. the entire pruned range lies within finalized history. Non-finalized proofs are never touched. Finalized blocks can never revert, so their proofs are not needed for fork choice, reorg safety, or re-aggregation once outside the window.

Never pruned: BlockHeaders, BlockBodies, BlockRoots, States, StateDiffs, and Metadata. Headers, bodies, the canonical slot index, and the snapshot+diff chain are the full historical record; only the proof blobs and the (non-finalized) fork choice index are disposable.

In-Memory Only (Lost on Restart)

Four Store fields never touch the backend. All are bounded buffers shared across Store clones:

The payload buffers evict FIFO when full, and redundant proofs (whose participants are a subset of an existing proof for the same attestation data) are skipped on insert.

Note that the per-validator “latest attestation” maps used by fork choice are not stored anywhere — they are derived on demand from these buffers via extract_latest_known_attestations and friends. See the attestation pipeline section of lmd_ghost.md for how attestations move between the pools.

After a restart these buffers start empty: pending attestations and un-aggregated gossip signatures are lost and must be re-collected from the network. Everything persisted in the eight tables survives.

Startup and Restore

A Store is created through one of three constructors in crates/storage/src/store.rs:

The first two funnel into init_store, which writes the anchor in one atomic batch: all six Metadata keys (time = 0, config, head = safe_target = anchor root, justified = finalized = anchor checkpoint), the anchor header, its BlockRoots entry, the body if non-empty, a full snapshot into States (the base of every future diff chain), and the anchor’s LiveChain entry.

from_db_state is the restore path: it reads config and latest_finalized from Metadata, returning None for an empty DB. A populated DB from another network is fatal instead: the finalized state’s genesis time and validator registry are compared against the genesis config, and a mismatch fails with Error::GenesisMismatch rather than being treated as empty, since writing a fresh anchor would leave the foreign chain’s rows in place to be served to peers. At startup the node prefers this path but only accepts the on-disk store if its head is at most MAX_RESUMABLE_DB_STATE_AGE = 450 slots (~30 minutes) behind the current slot; a staler DB falls through to checkpoint sync, which writes a fresh anchor on top of the existing data.

Key Files