Introduction

ethlambda is a minimalist, fast and modular implementation of the Lean Ethereum consensus client, written in Rust.

This book collects the design notes and operator-facing references for ethlambda. It is split into four parts:

• Design describes the shape of a running node: the architecture of its actors, workers and shared storage, and how they are wired together at startup.
• Consensus explains how the chain advances: the slot and interval structure that schedules every validator duty, the 3SF-mini justification and finalization rules, and the LMD-GHOST fork choice algorithm. These documents are implementation-agnostic; ethlambda-specific behaviour is called out in blockquotes.
• Operations documents observable surfaces of a running node: Prometheus metrics, checkpoint sync, and the fork choice visualization served by the API.
• Development collects notes for contributors, starting with the spec deviations where ethlambda departs from leanSpec.

For build and contribution instructions, see the README and CONTRIBUTING.md in the repository.

Visual references

Two standalone HTML infographics ship alongside this book and are copied verbatim into the rendered output:

• 3SF-mini infographic
• ethlambda architecture infographic

Community

• Telegram: ethlambda group, where we post daily updates; drop by to ask questions or chat about anything Lean-related.
• X (Twitter): @ethlambda_lean for occasional updates.
• Weekly community call: every Friday, streamed live on @class_lambda; the call link is posted on Telegram beforehand.
• Ecosystem coordination: the PQ Interop calls on ethereum/pm cover cross-client Lean Ethereum work and related updates; the meeting links are posted on each issue.

Related projects

ethlambda is one of several Lean Ethereum consensus clients under active development. For comparison and cross-client testing:

• zeam (Zig)
• ream (Rust)
• qlean (C++)
• grandine (Rust)
• gean (Go)
• Lantern (C)

Architecture

A running node is two actors, a few helper tasks, and one shared Store. The actors are genservers: each owns its state, and the only way in is a message.

• BlockChainServer (crates/blockchain/src/lib.rs) drives consensus. It runs the slot clock, performs validator duties, imports blocks and attestations, and is the sole writer of consensus state.
• P2PServer (crates/net/p2p/src/lib.rs) owns the network side: gossip publication, request/response, peer bookkeeping and long-range sync.

Everything else hangs off those two: an aggregation worker on a blocking thread, the libp2p swarm loop on its own task, and the axum servers that expose the node to the outside world.

Note: for what a genserver is, read this blogpost on the spawned crate.

                        Tick (self-message, one per interval)
                      ┌──────────────────────────────┐
                      ▼                              │
       ┌─────────────────────────────┐               │      ┌────────────────────┐
       │      BlockChainServer       │───────────────┘      │  aggregation       │
  ┌───▶│  actor, sole state writer   │─────── jobs ────────▶│  worker            │
  │    │                             │◀───── aggregates ────│  (blocking thread) │
  │    └──────────────┬──────────────┘                      └────────────────────┘
  │                   │
  │ P2PToBlockChain   │ BlockChainToP2P
  │ new block/vote    │ publish, fetch block
  │                   ▼
  │    ┌─────────────────────────────┐  SwarmCommand   ┌────────────────────┐
  └────│          P2PServer          │────────────────▶│   swarm adapter    │
       │  actor, gossip + req/resp   │◀────────────────│   (libp2p task)    │◀══▶ peers
       └─────────────────────────────┘   SwarmEvent    └────────────────────┘

       ┌────────────────────────────────────────────────────────────────────┐
       │ Store: pluggable key-value backend + in-memory fork-choice buffers │
       │ written by BlockChainServer, read by P2PServer and the API servers │
       └────────────────────────────────────────────────────────────────────┘

Actor protocols

The two actors never share memory: they talk over the typed protocols in crates/net/api/src/lib.rs.

Both refs start as None. The InitP2P and InitBlockChain messages fill them in right after spawn, so neither actor needs the other to exist at construction time.

The BlockChainServer

The tick loop

The actor schedules its first Tick for genesis time, and every handler re-arms the next one at the following interval boundary. A handler that overruns that boundary re-arms with a zero delay instead, so the interval it just missed still gets its duty. The handler derives (slot, interval) from the wall clock, compares it against the store’s own interval counter, and skips ticks the store already passed.

store::on_tick then walks the store clock forward one interval at a time, fast-forwarding if it fell more than a slot behind, so a late tick still runs each interval’s bookkeeping in order. That walk and the actor split the duties:

See Slots and Intervals for what each duty means at the protocol level, and why the proposer builds one interval early.

The actor also advances its XMSS signing keys on every tick, and catches them up to the current slot once at spawn. The keys are one-time and slot-bound, so a node that skipped this would have nothing left to sign with.

Block import

Blocks take the same path whether they arrived on gossip or came back from a BlocksByRange sync request. The actor verifies the signature, then runs the state transition (crates/blockchain/state_transition): process_slots advances the pre-state through empty slots, process_block validates the header and applies the block’s attestations, and the result is rejected unless the recomputed state root matches the one the proposer committed to. Justification and finalization move as part of that transition, following the 3SF-mini rules. The actor then writes the block and its post-state, and recomputes the head with LMD GHOST (crates/blockchain/fork_choice).

Aggregation off the message loop

XMSS proving costs hundreds of milliseconds, so it cannot run on the actor loop: a blocked actor stops importing blocks. The actor instead snapshots aggregation inputs from the store, ranks candidates by consensus value, and hands a job list to a spawn_blocking worker (crates/blockchain/src/aggregation.rs). The worker holds no store access, streams one AggregateProduced message back per finished job, and ends with AggregationDone. The actor publishes each result on gossip when it arrives.

A soft deadline cancels the worker through a CancellationToken, so an overrunning slot cannot eat the next one. The session can also start up to EARLY_AGGREGATION_WINDOW before interval 2, once two thirds of the expected signatures are in. Either entry point counts as the slot’s one session, so a slot aggregates once. Starting early buys proving time, not an earlier publication: the worker holds each finished aggregate until the interval-2 boundary before delivering it to the actor.

Block import runs a second, smaller aggregation path. reaggregate.rs splits an imported block’s merged proof back into per-attestation aggregates and folds them into the local pool, which is how a node that only saw a vote inside a block gets its fork-choice weight. Aggregators go one step further and republish those aggregates on gossip. Each split runs a fresh SNARK, so reaggregate.rs caps how many it does per block, and the actor skips the whole path while the node is catching up.

Subnet-windowed aggregation

Two aggregators handed the same pool of existing proofs would otherwise pick the same two children every session, since the greedy selection in aggregation.rs is deterministic: all that duplicated leanVM proving buys nothing once one of them publishes. Each aggregator instead scores that pool through a window: a contiguous run of subnets starting at its duty subnet, the first value of --aggregate-subnet-ids (or the lowest subnet it subscribes to, if that flag is unset). A proof outside the window still counts if it partly overlaps, but earns credit only for its in-window share, so aggregators with different windows tend to land on different children without anyone being excluded from merging.

The width is derived, not chosen: wide enough to hold two proofs at the reach of the aggregator’s anchor, capped at the committee count, so it only widens once a data root’s proof has actually climbed. The anchor is the largest-coverage proof in the candidate’s pool that touches the aggregator’s own duty subnet. Picking it by coverage rather than by reach keeps a sparse proof, one validator in each of many subnets, from setting the width for everybody; requiring it to touch the duty subnet means “no anchor” says “no peer has covered my subnet”, which is exactly when this node’s raw signatures are irreplaceable, and the narrowest window then leaves it aggregating those instead of merging other aggregators’ proofs. The price is that two aggregators reading one lopsided pool can derive different widths, so their windows nest rather than tile; that costs a round of climbing, not correctness.

A window can still decline a merge the unwindowed pool would have allowed, when the proof pool is sparse relative to the window’s contiguous span (a strided aggregator placement is the common cause); selection then retries once with the full committee set, so the feature can only improve on the pre-window selection, never regress below it.

--skip-redundant-aggregation trades that safety net away on purpose. With it set, an aggregator sits out any candidate whose derived width it does not own in the current slot (duty_subnet % width == slot % width), and the freed job goes to the next-best attestation data rather than to a narrower merge of the same one. Ownership rotates with the slot, so every duty subnet gets a turn, and the narrowest width is owned by everyone, so a candidate with no anchor on this node’s subnet is never skipped. The full-width fallback is disabled under the flag: every width below the committee count has several owners, so retrying there would rebuild exactly the duplication the flag buys away.

The rotation guarantees an owner at every width only when every subnet below the committee count has an aggregator holding it as its duty subnet, so treat that as a precondition for the flag. On a sparser placement a width can have no owner in a given slot even with every configured node healthy: with duty subnets {0, 2} at committee count 4, nothing owns width 4 in an odd slot, and with the fallback off that merge level is dropped for the slot. Leave the flag unset on a placement that does not cover every subnet.

Sync gate

sync_status.rs tracks how far the local head lags the slot clock. Past the threshold the node reports itself syncing and stops attesting and proposing, since a head derived from a partial view is not worth voting for. A hysteresis band stops the state from flapping at the boundary, and a network-wide stall (nobody else is ahead either) leaves the node synced so its validators can help the chain recover. The same status feeds the lean_node_sync_status metric and, through a shared controller, the /lean/v0/node/syncing endpoint. --disable-duty-sync-gate reduces the gate to observe-only.

Missing parents

The actor cannot import a block whose parent is unknown, so it parks the block in pending_blocks under its parent root and records the deepest missing ancestor it can find, walking back through already-stored pending blocks, in pending_block_parents. That ancestor is what it asks the P2PServer to fetch. Once the ancestor lands, the actor cascades down the parent index and re-imports every block that was waiting.

Chain events

The actor is the sole publisher on an EventBus (crates/blockchain/src/events.rs), which carries seven topics: head moves, imports and gossip sightings of blocks, single votes and aggregates, plus justification and finalization updates. The bus is best-effort: emission never blocks the actor, and a slow subscriber loses events instead of back-pressuring consensus. The API server subscribes one receiver per SSE client.

The P2PServer

Only the swarm adapter task touches the libp2p::Swarm. The actor sends it SwarmCommands (publish, dial, send request, send response) and receives SwarmEvents back as actor messages. That split keeps non-Clone swarm types (response channels, for one) out of the typed protocol, and keeps the adapter polling network I/O while the actor is busy with a message.

What the actor does with those events:

• Gossip. It decodes blocks, aggregates and per-subnet attestations, then forwards them to the BlockChainServer. The node computes its subscriptions once at startup from its validator set and aggregator role, and never revisits them.
• Status. Sent on the first connection to a peer, not on every redundant one. When a peer reports a head ahead of ours, the actor opens a long-range sync range or extends the one it has, then requests BlocksByRange batches one at a time, dropping peers that fall behind the range.
• BlocksByRoot. Backs the fetch_block requests above. Retries use exponential backoff and prefer a peer that has not already failed for that root, falling back to the full connected set once every peer has failed.

The P2PServer holds its own Store clone, which it only ever reads, so it answers Status and BlocksBy* requests without a round trip through the consensus actor.

The Store

The Store follows a similar approach as ethrex’s: a safe, easy to use interface over a pluggable key-value backend (StorageBackend, RocksDB for a normal node and in-memory for tests and the Hive test driver). Cloning one shares the backend, the state LRU cache, and the in-memory buffers that fork choice runs on: the new and known attestation payload buffers, the latest votes, and the gossip signatures awaiting aggregation. The node never persists those buffers, since they only matter for the slot they belong to.

Every component gets a clone, but only the BlockChainServer writes consensus state. A reader can therefore hold a handle without interleaving with a state transition.

See Data Storage for the table layout, the snapshot/diff scheme used for states, and pruning rules.

HTTP API

We use axum as our API router, with requests served in tokio tasks. Handlers read node state without messaging the actors: the Store is router state, and the EventBus and the runtime controllers arrive as extension layers. Metrics and debug endpoints live in their own routers, on a port you configure separately: distinct ports bind two independent servers, equal ports merge all three routers onto a single listener. See HTTP API for the endpoint reference.

Startup

bin/ethlambda/src/main.rs wires the node in a fixed order. Everything before the actors spawn is fail-fast, so a misconfigured node stops at boot instead of hours later.

1. Install the tracing subscriber, parse CLI options, register metrics, raise the file-descriptor limit for RocksDB’s unbounded table cache.
2. Load the node key, then genesis config, validator config, bootnode ENRs and validator keys.
3. Open the database, then pick an anchor: resume from disk if the on-disk head is recent enough, otherwise checkpoint sync from the configured URLs, and otherwise build the genesis state. A stale database with no checkpoint URL configured is resumed anyway, with a warning, since that is the setup the node was given.
4. Build the shared handles: aggregator controller, sync-status controller, event bus, and the subnet set that both the swarm and the actor need to agree on.
5. Spawn BlockChainServer (which schedules its first tick for genesis time), build the swarm, spawn P2PServer, and wire the two together with InitP2P and InitBlockChain.
6. Spawn the API and metrics servers.

Shutdown runs in reverse: the first ctrl+c stops both actors and cancels the servers’ shutdown token, and three more force the process to exit if a graceful stop hangs.

Note: booting with HIVE_LEAN_TEST_DRIVER=1 short-circuits everything from step 2 on and exposes only the Hive test-driver endpoints, so a driver run never touches the node key, the genesis config or any other consensus prerequisite.

Slots and Intervals

A Lean Chain slot is divided in 5 intervals of equal length. Every duty a validator owes the chain is due in one of them. The slot lasts 4 seconds by default, so the offsets below are the 800 ms grid; a network that sets MILLISECONDS_PER_SLOT in its config file scales every offset by the same factor. That key can only stretch the grid: 4 seconds is also the floor, since a few client timings are fixed in milliseconds rather than expressed as a fraction of the slot.

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │   block    │    vote    │    vote    │safe target │    head    │
    │  proposal  │propagation │aggregation │computation │   update   │
    └────────────┴────────────┴────────────┴────────────┴────────────┘
     ◄───────────── gossiped ─────────────▶ ◄───── local only ───────▶

The grid comes from a genesis timestamp every node shares, so the schedule needs no coordination messages: a node reads its clock, works out which interval it is in, and knows which duty is due. The order is a dependency chain, since each interval consumes what the previous one produced. A duty that overruns its interval is not rescheduled: it lands late, and the slot moves on without it.

In ethlambda: the intervals are the SlotInterval variants in crates/blockchain/src/lib.rs. INTERVALS_PER_SLOT is fixed in crates/common/types/src/constants.rs (each interval carries a distinct duty, so the count is not a knob), while the slot duration is read from the network’s config file into ChainConfig, which derives the interval length from it. Every node on a network must agree on the value, and other clients still hold it at compile time: setting it only has an effect where every node reads the key.

Interval 0: Block proposal

A block proposer, selected in a round-robin fashion (slot % num_validators), proposes a new block and gossips it to the network. Right before building the block, the proposer merges their “new attestations buffer” into their fork-choice view. They then include attestations that the proposer has recently seen into their block. Other validators verify the block and its contents, and merge the votes it includes into their fork-choice view. After importing a block, all validators recompute their head, and update the latest finalized and justified checkpoints according to the block’s post-state.

A block body carries at most MAX_ATTESTATIONS_DATA aggregated attestations: distinct (slot, head, target, source) tuples, each paired with a bitfield naming the validators bound to it. Genesis occupies slot 0, so proposals start at slot 1, and nothing forces a slot to be filled: a proposer that is offline or too slow leaves an empty slot, and the next block simply points its parent root at an older block.

In ethlambda: block proposal is merged into the previous slot’s head-update interval: the proposer advances its store to the next slot, builds the block there, and holds publication until the slot boundary. That buys the build one extra interval of headroom and leaves no actor work at the block-proposal tick itself.

Interval 1: Vote propagation

Validators gossip their votes for the block they consider to be the head of the chain, and append to it a (source, target) finality vote. These votes are in aggregation subnets and are imported by aggregators. Aggregators verify the votes in their subnet and store them for later aggregation.

In ethlambda: a validator’s subnet is validator_index % attestation_committee_count, and a node only aggregates for subnets it subscribed to at startup. Aggregation is also gated on the aggregator role, seeded by --is-aggregator and flippable at runtime through the admin API. A chain whose validators all decline the role still gossips votes and logs them as processed, but no aggregate is ever produced, so every block is empty and the chain never justifies.

Interval 2: Vote aggregation

Aggregators aggregate the votes they have received and gossip the resulting aggregated attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a “new attestations buffer”.

Aggregation earns its own interval because collapsing a subnet’s worth of XMSS signatures into one proof is the heaviest recurring computation in the client. It is also what makes a block affordable, since a block carrying raw votes would need one full XMSS signature per voter, quickly going over the network bandwidth limit.

In ethlambda: the proofs run on an off-thread worker so the blockchain actor’s message loop stays responsive, and a session may start up to EARLY_AGGREGATION_WINDOW before the interval boundary once two thirds of the signatures are in. At the session’s soft deadline the actor stops handing out new jobs, but a proof already in flight finishes and publishes late rather than being discarded.

Interval 3: Safe target computation

Validators compute the safe target they’ll use when deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot.

It is LMD-GHOST again, but run over just the votes that arrived this slot and with a two-thirds weight threshold, where head selection applies none. The safe target therefore sits at or behind the head and advances only once a branch is backed by a supermajority. Deriving targets from it is what stops 3SF-mini from justifying a branch the network has not visibly converged on.

Interval 4: Head update

Validators merge the aggregated attestations they have in their “new attestations buffer” into their fork-choice view, and recompute their head.

This is the slot’s second and last promotion point; the first is the proposer’s, just before it builds. Until a vote is promoted it carries no weight in head selection, which is what keeps a validator’s fork-choice view from shifting under it mid-slot. The safe target is the exception: it reads the unpromoted buffer directly, which is how it stays a view of this slot alone. See why staged promotion for the reasoning.

3SF-mini: Justification & Finalization

ethlambda uses 3SF-mini (Three-Stage Finality, minimal version) for justification and finalization. Unlike the Ethereum Beacon Chain’s epoch-based Casper FFG, 3SF-mini operates at the slot level: any slot can be justified, not just epoch boundaries.

Quick Example: Three Slots to Finality

4 validators, slot N already finalized and justified.

                             source  target
                                │       │
                                ▼       ▼
    Slot N        ──[ N-2 ]──[ N-1 ]──[ N ]
                       F        J       H

                                     source    target
                                        │         │
                                        ▼         ▼
    Slot N+1      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]
                       F        F       J         H

                                               source     target
                                                  │          │
                                                  ▼          ▼
    Slot N+2      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]────[ N+2 ]
                       F        F       F         J          H

    H = head    J = justified    F = finalized

At each slot, validators vote for the newest block as their target, citing the latest justified checkpoint as their source:

• Slot N+1: Votes source=N, target=N+1. Three of four vote (3×3=9 >= 2×4=8), so N+1 is justified.
• Slot N+2: Votes source=N+1, target=N+2. Three of four vote, so N+2 justified. N+1 and N+2 are consecutive justifiable slots and both are justified, so N+1 is finalized.

In the ideal case, each block carries attestations that justify the parent slot and finalize the one before it. In practice, forks, missed slots, and delayed votes can break this cadence. The rest of this document explains the rules that make this work, and what happens when things go wrong.

Concepts

Justification via Supermajority

A checkpoint becomes justified when at least two-thirds of validators attest to it as a target:

                   JUSTIFICATION
                   ─────────────

    Validators:  V0  V1  V2  V3  V4  V5  V6  V7  V8
                  │   │   │   │   │       │   │
                  └───┴───┴───┴───┴───────┴───┘
                              │
                    7 out of 9 votes
                  (3×7=21 >= 2×9=18) ✓
                              │
                              ▼
                     ┌──────────────┐
                     │ Checkpoint C │
                     │ JUSTIFIED ✓  │
                     └──────────────┘

The threshold is computed as: 3 × vote_count >= 2 × validator_count

In ethlambda: Justification and finalization are processed inside process_attestations() in crates/blockchain/state_transition/src/lib.rs, called from process_block(). The supermajority check is 3 * vote_count >= 2 * validator_count.

Attestations must also pass validity checks before they count:

• Source checkpoint must already be justified
• Target must not already be justified
• Neither source nor target may have a zero-hash root
• Source slot < Target slot (time flows forward)
• Both checkpoints must reference known blocks
• Target slot must be justifiable per the 3SF-mini schedule (see below)

The Justifiability Schedule

Not every slot can be justified, only slots at specific distances from the last finalized slot. This is the novel part of 3SF-mini.

A slot is justifiable if delta = slot - finalized_slot matches any rule:

In ethlambda: The function slot_is_justifiable_after(slot, finalized_slot) in crates/blockchain/state_transition/src/lib.rs implements this check. It uses isqrt() for perfect square detection and the identity 4n(n+1) + 1 = (2n+1)² for pronic number detection.

    ┌───────────────────────────────────────────────────────┐
    │             JUSTIFIABILITY RULES                      │
    │                                                       │
    │  Rule 1:  delta ≤ 5          (always justifiable)     │
    │                                                       │
    │  Rule 2:  delta = n²         (perfect squares)        │
    │           1, 4, 9, 16, 25, 36, 49, 64, 81, 100, ...   │
    │                                                       │
    │  Rule 3:  delta = n(n+1)     (pronic numbers)         │
    │           2, 6, 12, 20, 30, 42, 56, 72, 90, 110, ...  │
    │                                                       │
    └───────────────────────────────────────────────────────┘

Visualizing the first 40 slots after finalization (✓ = justifiable):

    delta: 0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20
           ✓  ✓  ✓  ✓  ✓  ✓  ✓  ·  ·  ✓  ·  ·  ✓  ·  ·  ·  ✓  ·  ·  ·  ✓
           ╰─ delta ≤ 5 ──╯  2×3      3²       3×4         4²          4×5

    delta: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
           ·  ·  ·  ·  ✓  ·  ·  ·  ·  ✓  ·  ·  ·  ·  ·  ✓  ·  ·  ·  ·
                       5²             5×6               6²

Key property: Gaps between justifiable slots grow, but never become infinite. As more time passes since finalization, the network gets progressively wider windows to accumulate votes. This creates a natural backpressure: if the network is struggling to reach a two-thirds majority (e.g., due to partitions or validator dropouts), the increasing gaps give more time for the supermajority to form.

Finalization

A justified checkpoint becomes finalized when it is the source of a justification whose target is the next justifiable slot. In other words, there must be no justifiable slots between source and target: the two must be consecutive entries in the justifiability schedule.

In ethlambda: The try_finalize() function iterates over slots between source and target and calls slot_is_justifiable_after on each. If any slot is justifiable, finalization fails (source and target aren’t consecutive). The check uses original_finalized_slot (the finalized slot at the start of block processing), not the current one, since finalization can advance mid-processing.

                    FINALIZATION CHECK
                    ──────────────────

    Example 1: Finalization FAILS

    Finalized=10   Source=13 (justified)   Target=16 (justified)

    [ 10 ] · · · [ 13 ]  14   15  [ 16 ]
                          ▲    ▲
                          │    └── delta=5 ≤ 5 → justifiable!
                          └────── delta=4 ≤ 5 → justifiable!

    Justifiable slots exist between S and T → NOT FINALIZED ✗
    (13 and 16 are not consecutive justifiable slots)


    Example 2: Finalization SUCCEEDS

    Finalized=10   Source=16 (justified)   Target=19 (justified)

    [ 10 ] · · · [ 16 ]  17   18  [ 19 ]
                          ▲    ▲
                          │    └── delta=8 → not justifiable ✓
                          └────── delta=7 → not justifiable ✓

    No justifiable slots between S and T → S is FINALIZED ✓
    (16 and 19 are consecutive: delta=6=2×3, then delta=9=3²)

The reasoning: if a justifiable slot exists between source and target, validators could have directed their votes to that intermediate slot instead, potentially on a different fork. By requiring source and target to be consecutive justifiable slots, the protocol ensures that no alternative justification path can exist between them.

Justifiable Slot Backoff

The justifiability schedule acts as a backoff mechanism to increase finalization rate during periods of asynchrony. By “diluting” the possible targets of a justification vote (via the slot_is_justifiable_after function), the protocol increases the window during which votes for a given slot can be included, improving the chances of achieving the required two-thirds majority.

Since finalization requires two consecutively justifiable slots to both be justified, this backoff isn’t immediately reset after finalization occurs; it only lowers over time when synchrony is restored.

Example: Extended asynchrony with gradual recovery.

    F=0. Justifiable slots grow sparser as delta increases:

    delta ≤ 5:     0   1   2   3   4   5                  (gap = 1)
    delta 6–20:    6       9       12          16          20   (gap = 3–4)
    delta 20–36:   20          25          30              36   (gap = 5–6)
    ...
    delta ~1000:   900     930     961     992      1024        (gap = 30–32)
                   30²    30×31   31²    31×32      32²

Phase 1: Long asynchrony, slow progress.

    Validators vote, but with many justifiable targets, votes scatter
    and no single slot reaches >=2/3. As gaps widen, votes concentrate.

    Near slot 1000, the 32-slot gap between 992 and 1024 means
    no competing justifiable target exists for 32 slots after 992.
    All votes funnel toward 1024 once it is built.

Phase 2: Slot 992 finalized.

    Slot 992 justified (source = earlier justified slot).
    Slot 1024 justified (source = 992).

    slot:  0  ...  992              1024
           F        J    ·······     J
                    ▲                ▲
                 source ──────────▶ target

    Slots 993–1023: any justifiable from F=0?
      Perfect squares? 31²=961 (before), 32²=1024 (boundary). None.
      Pronic? 31×32=992 (boundary), 32×33=1056 (after). None.
    No justifiable slots between them → slot 992 FINALIZED ✓

Phase 3: Partial reset. Backoff shrinks but doesn’t vanish.

    New F=992. Justifiable slots shift:

    slot: 992 993 994 995 996 997 998 ··· 1001 ··· 1004 ··· 1008 ··· 1022 ··· 1028
           F   ✓   ✓   ✓   ✓   ✓   ✓      ✓       ✓       ✓       ✓       ✓
              ╰── delta ≤ 5 ──╯  2×3     3²      3×4     4²      5×6     6²

    Dense slots 993–998 are already in the past!
    Near the current slot (~1024), justifiable slots are ~6 apart:

    ...  1022     1028     1034     1041  ...
         δ=30     δ=36     δ=42     δ=49
         5×6      6²       6×7      7²
         └──6──┘  └──6──┘  └──7──┘

    Gaps shrank from 32 → 6, but didn't reset to 1.

Phase 4: Further finalization closes the gap.

    Justify 1022 and 1028, finalize 1022. New F=1022.

    From F=1022, at slot ~1028 (delta = 6):

    slot:  1022  1023  1024  1025  1026  1027  1028
            F     ✓     ✓     ✓     ✓     ✓     ✓
                  ╰────── delta ≤ 5 ──────╯    2×3

    Gaps are back to 1. Fast finalization resumes.

    Summary of gradual recovery:

    ┌───────────────────┬──────┬───────┬───────┬──────────────┐
    │ Finalization step │  F   │ Head  │ Delta │ Nearby gaps  │
    ├───────────────────┼──────┼───────┼───────┼──────────────┤
    │ Before any        │    0 │ ~1000 │ ~1000 │ 31–32        │
    │ After 1st (992)   │  992 │ ~1024 │   ~32 │ 6–7          │
    │ After 2nd (1022)  │ 1022 │ ~1028 │    ~6 │ 1            │
    └───────────────────┴──────┴───────┴───────┴──────────────┘

    Each finalization step reduces the delta between the finalized
    slot and the chain head, progressively tightening the gaps.

When finalization advances, the following cleanup occurs:

• justified_slots window shifts forward (old slots pruned)
• LiveChain entries for finalized slots are pruned
• Gossip signatures and aggregation proofs for finalized blocks are cleaned up
• Future fork choice runs start from the finalized slot’s successor

In ethlambda: The justified_slots bitlist uses relative indexing (index 0 = finalized_slot + 1). When finalization advances, shift_window() in crates/blockchain/state_transition/src/justified_slots_ops.rs drops the now-finalized prefix. The attestation target is also walked back to the nearest justifiable slot via slot_is_justifiable_after in crates/blockchain/src/store.rs.

End-to-End: From Head Selection to Finalization

This section connects LMD-GHOST fork choice with 3SF-mini. The quick example above showed the happy path; here we focus on what happens when things go wrong.

Recap: Attestation Anatomy

Each attestation carries three checkpoints, each determined by a different mechanism:

    ┌────────────────────────────────────────────────────────────────┐
    │                       ATTESTATION                              │
    │                                                                │
    │  head    Newest block the validator sees                       │
    │          ← LMD-GHOST with min_score = 0                        │
    │                                                                │
    │  target  Block the validator wants justified next              │
    │          ← Derived from safe target, walked back to nearest    │
    │            justifiable slot (feeds into 3SF-mini)              │
    │                                                                │
    │  source  Latest justified checkpoint                           │
    │          ← Read from store state                               │
    └────────────────────────────────────────────────────────────────┘

The safe target is computed by running LMD-GHOST with a two-thirds vote threshold. Only blocks backed by a supermajority qualify, so the safe target is always at or behind the head. The attestation target is derived by walking back from the head toward the safe target (max 3 steps), then to the nearest justifiable slot. See Safe Target Selection for details.

In ethlambda: get_attestation_target() in crates/blockchain/src/store.rs implements this walk-back. JUSTIFICATION_LOOKBACK_SLOTS = 3 provides a liveness guarantee: even if the safe target is stuck, the target eventually advances once the head moves far enough ahead.

Lagging Safe Target (Fork with Delayed Convergence)

When validators disagree about the head, the safe target lags behind: no single branch has two-thirds support. This delays justification until the fork resolves.

    Setup: 9 validators, finalized=100, justified=101
    Safe target threshold: >=6 votes (2/3 of 9)

Slots 102–103: Fork splits votes. No progress.

                         ┌──[ B102a ]──[ B103a ]     V0–V4 (5)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]     V5–V8 (4)

Neither branch clears two-thirds → safe target stuck at B101. Walk-back from head always lands on source (B101). No attestation can advance justification.

Slot 104: V7 and V8 switch sides. Fork resolves.

V7 and V8 receive B102a (delayed by the partition) and switch to the a-branch.

                         ┌──[ B102a ]──[ B103a ]──[ B104a ]     V0–V4, V7, V8 (7)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]──[ B104b ]     V5–V6  (2)

B102a subtree now has 7 votes >= 6 → safe target = B102a. Walk-back from B104a lands on B102a (2 steps). Slot 102 is justifiable (delta=2 ≤ 5).

    source=101 ──▶ target=102    7/9 votes → 3×7=21 >= 2×9=18 → JUSTIFIED ✓
    Finalization: no slots between 101 and 102 → 101 FINALIZED ✓

After slot 104: finalized=101, justified=102.

Slots 105–106: Full convergence and recovery.

All 9 validators on the a-branch. Slot 105: target=B104a → B104a JUSTIFIED. But finalization fails: slot 103 (between source=102 and target=104) is justifiable but was never justified (lost in the fork).

Slot 106: target=B105a → B105a JUSTIFIED. No justifiable slots between 104 and 105 → 104 FINALIZED. Finalization jumped from 101 to 104, skipping 102 and 103.

    FORK WITH DELAYED CONVERGENCE
    ═════════════════════════════

    Slot:     100   101   102   103   104   105   106
    Status:    F     J     ·     ·     ·     ·     ·
                                fork ──────┤
                                          resolves
    Head:      ·    B101  B102a B103a B104a B105a B106a
    Safe:      ·    B100  B101  B101  B102a B104a B105a
                          stuck ─────┘  ▲
                                        │
                           V7+V8 switch, safe target unsticks

    Justified:  ·    101   ─     ─    102   104   105
    Finalized:  ·     ·    ─     ─    101   ─     104
                                                   ▲
                              finalization jumps ──┘
                               (102,103 skipped; 103 was never justified)

Comparison with Casper FFG

Both 3SF-mini and Casper FFG are finality gadgets built on the same foundation: supermajority links between checkpoints. They differ fundamentally in their unit of time and what that implies for validator participation. For a thorough treatment of Casper FFG as used in Ethereum, see the eth2book chapter on Casper FFG.

Slots vs Epochs: The Core Architectural Split

3SF-mini: Every Validator, Every Slot

In 3SF-mini, all validators vote in every slot. A checkpoint can be justified at any slot (subject to the justifiability schedule), and finalization can happen as soon as two consecutive justifiable slots are both justified.

    3SF-mini  (4-second slots, 4 validators)

    Slot 100        Slot 101        Slot 102        Slot 103
    ┌───────┐       ┌───────┐       ┌───────┐       ┌───────┐
    │V0 V1  │       │V0 V1  │       │V0 V1  │       │V0 V1  │
    │V2 V3  │       │V2 V3  │       │V2 V3  │       │V2 V3  │
    └───┬───┘       └───┬───┘       └───┬───┘       └───┬───┘
        │               │               │               │
     4 votes         4 votes         4 votes         4 votes
     per slot        per slot        per slot        per slot

    Every validator participates in every slot.
    >=2/3 threshold checked per-slot → can justify any slot.

This is simple and fast, but it means every validator must produce and verify a vote every slot. The total message load scales as validators × slots.

Casper FFG: Validators Split Across an Epoch

Ethereum’s beacon chain has ~1,000,000 active validators. Having all of them vote every 12-second slot would be unmanageable. Instead, Casper FFG groups 32 slots into an epoch, and splits the validator set across the slots within it:

    Casper FFG  (12-second slots, 32 per epoch, ~900k validators)

    Epoch N
    ┌─────────────────────────────────────────────────────────────┐
    │ Slot 0     Slot 1     Slot 2    ...    Slot 30    Slot 31   │
    │ ┌──────┐   ┌──────┐   ┌──────┐        ┌──────┐   ┌──────┐   │
    │ │~28125│   │~28125│   │~28125│  ...   │~28125│   │~28125│   │
    │ │valids│   │valids│   │valids│        │valids│   │valids│   │
    │ └──┬───┘   └──┬───┘   └──┬───┘        └──┬───┘   └──┬───┘   │
    │    │           │           │               │           │    │
    └────┼───────────┼───────────┼───────────────┼───────────┼────┘
         └───────────┴───────────┴───┬───────────┴───────────┘
                                     │
                            All ~900k votes
                          collected over 32 slots
                                     │
                                     ▼
                              Epoch checkpoint
                              (first slot of epoch)

    Each validator attests exactly ONCE per epoch.
    The full >=2/3 tally is only meaningful at epoch boundaries.

Each validator is shuffled into a committee assigned to one specific slot. Within that slot, the committee may be further split (up to 64 sub-committees) for parallel aggregation. The result: each validator only attests once per epoch, and the network processes ~28,000 attestations per slot instead of ~900,000.

The trade-off:

Epochs exist because of a scalability constraint, not a protocol-theory preference. If you could process a million votes per slot, Casper FFG wouldn’t need epochs at all. 3SF-mini sidesteps this by targeting a smaller validator set, which lets it operate at slot granularity.

Finalization Logic

Both require a chain of justified checkpoints, but the rules differ in what they check.

Casper FFG uses k-finality. The original rule (k=1) requires a direct supermajority link from a checkpoint to its immediate successor: justify epoch N+1 with source=N, and N is finalized. Ethereum generalizes this to k=2, which handles the case where the network falls slightly behind:

    Casper FFG — 1-finality (ideal case):

    Epoch N       Epoch N+1
    ┌─────┐       ┌─────┐
    │ CP  │══════▶│ CP  │       Supermajority link N → N+1
    │ J ✓ │       │     │
    └─────┘       └─────┘

    Processing this link:
      1. Epoch N+1 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (direct successor justified)


    Casper FFG — 2-finality (one epoch behind):

    Epoch N       Epoch N+1     Epoch N+2
    ┌─────┐       ┌─────┐       ┌─────┐
    │ CP  │       │ CP  │       │ CP  │
    │ J ✓ │       │ J ✓ │       │     │
    └─────┘       └─────┘       └─────┘
       │                           │
       └══════ supermajority ══════┘
               link N → N+2

    The direct link N→N+1 didn't form in time.
    Instead, a link forms from N→N+2. Processing this link:
      1. Epoch N+2 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (all intermediates are justified)

The 2-finality rule is a recovery mechanism: even if the network missed the ideal one-epoch finalization window, it gets a second chance. Ethereum tracks the justification status of the last 4 epoch boundaries to detect both cases. In practice, most finalization happens via 1-finality during normal operation; 2-finality kicks in during brief network hiccups.

3SF-mini takes a different approach entirely:

    Slot S        Slot T
    ┌─────┐       ┌─────┐
    │ CP  │──────▶│ CP  │       No justifiable slots exist
    │ J ✓ │       │ J ✓ │       between S and T
    └─────┘       └─────┘
    ∴ Slot S is FINALIZED

    Rule: Finalized when NO intermediate checkpoints could exist

Instead of checking that intermediate checkpoints are justified, 3SF-mini checks that no intermediate checkpoints could exist at all. This is a stronger guarantee: validators’ votes between source and target could only have gone to the target, since there’s nowhere else to direct them. This structural property is also why 3SF-mini doesn’t need Casper’s surround-vote slashing condition.

Casper’s k-finality is essentially a tolerance parameter: “how many epochs behind can we be and still finalize?” Ethereum chose k=2, meaning it tolerates one missed epoch. 3SF-mini doesn’t need this concept because the justifiability schedule itself adapts. Instead of tolerating missed windows, it makes the windows wider when the network is struggling.

Adaptive Backoff (unique to 3SF-mini)

Casper FFG has a fixed checkpoint every epoch, regardless of network conditions. 3SF-mini’s justifiability schedule adapts: gaps between justifiable slots grow under prolonged asynchrony (via the perfect square and pronic number rules), creating natural vote concentration when the network is struggling to reach a two-thirds majority. Casper FFG has no equivalent; its epoch spacing is the same whether the network is healthy or partitioned. See Justifiable Slot Backoff for a detailed walkthrough.

👻 LMD-GHOST fork choice algorithm

A deep dive into how the LMD-GHOST (Latest Message Driven, Greedy Heaviest Observed SubTree) fork choice algorithm works. LMD-GHOST is the fork choice rule used by Ethereum’s consensus layer and its derivatives. Each validator’s latest attestation is their single active vote, and the algorithm follows the heaviest branch at every fork.

This document is implementation-agnostic, with ethlambda-specific details called out in blockquotes marked “In ethlambda”.

Much of the conceptual framing in this document is inspired by Ben Edgington’s Eth2 Book, particularly the LMD GHOST chapter. Highly recommended reading for anyone interested in Ethereum consensus.

Background & History

The GHOST protocol was introduced by Sompolinsky and Zohar in a 2013 paper. Its core idea: instead of choosing the heaviest chain, we choose the heaviest subtree, counting orphaned blocks as evidence of support for their ancestors.

The “LMD” in LMD-GHOST stands for Latest Message Driven: only each validator’s most recent attestation counts, preventing vote amplification. LMD-GHOST is the fork choice rule used by the Ethereum Beacon Chain and Lean Ethereum.

Why Fork Choice?

In a distributed system where validators propose blocks concurrently, the blockchain can fork: two valid blocks may appear at the same slot, creating competing chains. The fork choice rule answers a critical question:

Which chain tip should I follow?

                   ┌──────────┐
             ┌────▶│ Block C  │  ← Chain tip 1
             │     │ slot 5   │
┌──────────┐ │     └──────────┘
│ Block A  │─┤
│ slot 3   │ │     ┌──────────┐
└──────────┘ └────▶│ Block D  │  ← Chain tip 2
                   │ slot 5   │
                   └──────────┘

                    Which tip should validators follow?

Every node in the network must be able to independently arrive at the same answer using only its local view of blocks and attestations. The fork choice rule is what makes this possible. It is a deterministic function from a node’s observed state to a single chain tip.

From Heaviest Chain to Heaviest Subtree

The simplest fork choice rule is heaviest chain: follow the chain tip with the most accumulated weight. This works when fork rates are low, but breaks down when honest validators fork within a common branch:

              HEAVIEST CHAIN vs HEAVIEST SUBTREE
              ──────────────────────────────────

    An attacker with 40% of stake forks at A.
    The honest majority (60%) builds on B but forks into C and D:

                    ┌───B──┬──C     V0, V1, V2 vote for C (30%)
              A ────┤      └──D     V3, V4, V5 vote for D (30%)
                    │
                    └───X──Y──Z     V6, V7, V8, V9 vote for Z (40%)

    Heaviest chain:
      Z has 40% of votes, C and D each have 30%.
      Attacker wins! ✗

    Heaviest subtree (LMD-GHOST):
      At A: B subtree has 60% (C + D), X subtree has 40%.
      Pick B. Then at B: C has 30%, D has 30% (tiebreaker).
      Honest majority wins. ✓

LMD-GHOST is strictly better when honest validators fork within a common subtree. Instead of requiring all honest validators to agree on a single chain tip (which is impossible under network delay), it aggregates their support at each level of the tree.

How Subtree Weight Works (the “GHOST” Part)

The key insight behind the “Heaviest Observed SubTree” part of LMD-GHOST: a vote for a block is implicitly a vote for all its ancestors.

When a validator attests to block F as their head, they are also expressing support for every block on the path from the root to F:

    Validator attests: head = F

    A ── B ── C ── D ── E ── F
    ▲    ▲    ▲    ▲    ▲    ▲
    │    │    │    │    │    │
    └────┴────┴────┴────┴────┘
    All ancestors implicitly supported

This is why LMD-GHOST counts the subtree weight: a block’s weight includes every attestation for any of its descendants, because those attestations implicitly endorse the ancestor too. The algorithm exploits this by walking backward from each attested head and incrementing every block along the path.

LMD: Why Only the Latest Message?

The “LMD” in LMD-GHOST stands for Latest Message Driven. Each validator’s most recent attestation is their only vote. All previous attestations are discarded.

    Validator 7's attestation history:

    Slot 10: attests to head = B     ← discarded
    Slot 11: attests to head = C     ← discarded
    Slot 12: attests to head = E     ← THIS is the active vote

    Only the slot 12 attestation counts for fork choice.

Why only the latest? Two reasons:

1. Prevents double-voting. If all messages counted, a validator could cast many attestations and amplify their influence. With LMD, each validator gets exactly one active vote regardless of how many attestations they’ve broadcast.

2. Reflects current knowledge. A validator’s latest attestation reflects their most recent view of the chain. Older attestations may reference blocks that are no longer on the best chain. Keeping only the latest ensures fork choice uses the most up-to-date information.

The fork choice store maintains a mapping of validator_index → latest attestation. When a new attestation arrives from a validator, it replaces their previous entry:

    Fork choice store (latest messages):

    ┌──────────────┬──────────────────────────────┐
    │ Validator    │ Latest Attestation           │
    ├──────────────┼──────────────────────────────┤
    │ 0            │ head=E, target=C, source=A   │
    │ 1            │ head=D, target=C, source=A   │
    │ 2            │ head=E, target=C, source=A   │
    │ 3            │ head=F, target=D, source=A   │
    │ ...          │ ...                          │
    └──────────────┴──────────────────────────────┘

    One row per validator. New attestation → overwrite row.

LMD-GHOST Step by Step

The algorithm takes a set of inputs and produces a single block root: the head of the chain.

Inputs

In ethlambda: The function is compute_lmd_ghost_head() in crates/blockchain/fork_choice/src/lib.rs. The block tree comes from the LiveChain storage index, and min_score is 0 for head selection or ⌈2V/3⌉ for safe target computation.

The Algorithm

First, accumulate weights. Each attestation “paints” the path from its head back to the start root. In the simplest form (equal-weight validators), this adds +1 to every block on the path. In systems with balance-weighted voting, the validator’s effective balance is added instead.

    Validator 0 attests to head = F

      J ─ A ─ B ─ C ─ D ─ E ─ F       (J = justified root)
          +1  +1  +1  +1  +1  +1       J is at start_slot, not counted

    Validator 1 attests to head = D

      J ─ A ─ B ─ C ─ D
          +1  +1  +1  +1

    Accumulated weights:

      Block:    J    A    B    C    D    E    F
      Weight:   ─    2    2    2    2    1    1
                │
                └ start_root (not weighted, used as the descent origin)

In ethlambda: All validators have equal weight (+1 per vote). The Ethereum Beacon Chain instead weights votes by effective balance (up to 2048 ETH).

Then, greedily descend. Starting from the start root, at each node pick the child with the most weight. Repeat until reaching a leaf:

    J ──┬── B (5)   ← pick B (higher weight)
        └── G (2)

    B ──┬── C (3)   ← pick C (higher weight)
        └── H (2)

    C ──── D (3)    ← only child, continue

    D ── (no children) → HEAD = D!

Children below min_score are ignored during the descent. With min_score = 0 (normal head selection) all children are visible. With a higher threshold, only branches with strong support are followed. This is used for safe target selection.

The Tiebreaker

When two children have exactly equal weight, a deterministic tiebreaker is needed. Without one, different nodes could pick different heads from the same data, breaking consensus. The tiebreaker is lexicographically higher block root hash, i.e., higher hash value wins.

    Equal weight scenario:

        Parent
        │
    ┌───┴───┐
    B (3)   C (3)         ← Equal weight!
    root:   root:
    0x3a..  0x7f..        ← 0x7f > 0x3a, so pick C

The choice of “higher hash wins” is a convention. Any deterministic rule would work; what matters is that all nodes apply the same one.

Worked Example: Head Selection

Consider a network with 5 validators (indices 0–4) and the following block tree rooted at the justified checkpoint J at slot 10:

                          BLOCK TREE
                          ──────────

Slot 10     ┌──────┐
(justified) │  J   │ ← Justified checkpoint (start_root)
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │
         └──────┘    └──────┘

Latest attestations (one per validator):

Accumulate weights by walking backward from each attested head, adding +1 per block (stopping at J’s slot):

    V0 (head=D):  D+1  B+1  A+1
    V1 (head=D):  D+1  B+1  A+1
    V2 (head=E):  E+1  C+1  A+1
    V3 (head=E):  E+1  C+1  A+1
    V4 (head=E):  E+1  C+1  A+1

Greedily descend from J, always picking the heaviest child:

    Start at J
      └─▶ A (only child, weight 5)
           ├── B (weight 2)
           └── C (weight 3)  ← Pick C (3 > 2)
                └─▶ E (only child, weight 3)
                     └─▶ No children → HEAD = E ✓

Result: The canonical head is Block E. Even though both branches have the same depth, the C→E branch has 3 votes vs B→D’s 2 votes.

                          RESOLVED HEAD
                          ─────────────

Slot 10     ┌──────┐
            │  J   │
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │ ✓ canonical
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │ ✓ canonical (weight 3 > 2)
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │ ★ HEAD
         └──────┘    └──────┘

What If a Vote Changes?

Suppose validator 1 now sees block E and switches their attestation from D to E:

    Before:  V0=D, V1=D, V2=E, V3=E, V4=E   → Head = E (3 vs 2)
    After:   V0=D, V1=E, V2=E, V3=E, V4=E   → Head = E (4 vs 1)

    The head didn't change, but the margin increased from 1 to 3.
    If instead V2 and V3 had switched to D:

    After:   V0=D, V1=D, V2=D, V3=D, V4=E   → Head = D (4 vs 1)

    The head reorgs from E to D.

Fork Choice vs Finality

An important conceptual distinction: LMD-GHOST provides fork choice, not finality.

LMD-GHOST gives the network a way to agree on the current head of the chain at any moment, but the head can change. A block selected by fork choice today could be reorged away tomorrow if attestations shift. LMD-GHOST alone provides no guarantee that any block is permanent.

Finality, the guarantee that a block can never be reverted, comes from a separate mechanism called a finality gadget. LMD-GHOST is designed to compose with any finality gadget (e.g., Casper FFG in the Ethereum Beacon Chain, or 3SF-mini in Lean Ethereum).

    ┌────────────────────────────────────────────────────┐
    │                 CONSENSUS = TWO LAYERS             │
    │                                                    │
    │  ┌─────────────┐        ┌──────────────────────┐   │
    │  │  LMD-GHOST  │        │  Finality Gadget     │   │
    │  │             │        │                      │   │
    │  │ "Which tip  │        │ "Which blocks are    │   │
    │  │  is best    │        │  permanent and can   │   │
    │  │  right now?"│        │  never be reverted?" │   │
    │  │             │        │                      │   │
    │  │ Dynamic,    │        │ Monotonic, only      │   │
    │  │ can reorg   │        │ moves forward        │   │
    │  └──────┬──────┘        └──────────┬───────────┘   │
    │         │                          │               │
    │         └──────────┬───────────────┘               │
    │                    ▼                               │
    │         ┌──────────────────┐                       │
    │         │  Full Consensus  │                       │
    │         └──────────────────┘                       │
    └────────────────────────────────────────────────────┘

In ethlambda: The finality gadget is 3SF-mini, which operates at the slot level rather than epoch boundaries.

The two layers interact: LMD-GHOST runs its greedy descent starting from the latest justified checkpoint (not genesis). This means finality constrains fork choice: once a checkpoint is finalized, no fork choice run will ever consider blocks before it.

    ┌─────────┐         ┌─────────┐         ┌──── ...
    │FINALIZED│────────▶│JUSTIFIED│────────▶│  fork choice
    │ slot 50 │         │ slot 55 │         │  runs here
    └─────────┘         └─────────┘         └──── ...
         │                   │
         │                   └── start_root for LMD-GHOST
         │
         └── everything before this is permanent

This has a major practical benefit: finality allows aggressive pruning of the block tree. Without finality, fork choice would need to consider every block since genesis, and the tree would grow without bound. With finality, all blocks at or before the finalized checkpoint can be discarded from the fork choice’s working set.

In ethlambda: The LiveChain index (the in-memory block tree used by fork choice) is pruned every time finalization advances, keeping it bounded to only the non-finalized portion of the chain.

Attestation Pipeline

In a naive implementation, every attestation would influence fork choice the instant it arrives. This creates problems: validators with faster network connections see different heads than slower ones, and the proposer’s view of the chain could shift mid-block-construction.

Lean Ethereum solves this with a staged promotion pipeline: attestations are collected into a pending set and only promoted to the active fork choice set at designated moments. This ensures all validators operate on a consistent view.

                       ATTESTATION LIFECYCLE
                       ─────────────────────

  ┌──────────────┐       ┌──────────────────┐       ┌──────────────────┐
  │   Network    │       │    Pending       │       │    Active        │
  │  (gossip)    │──────▶│  Attestations    │──────▶│  Attestations    │
  │              │       │                  │       │                  │
  └──────────────┘       └──────────────────┘       └──────────────────┘
                                 │                          │
                          NOT used for               Used for fork choice
                          fork choice                weight calculations
                                 │                          │
                          Promoted at ─────────────▶ designated intervals
                          fixed points

In ethlambda: The two stages are called “new” and “known” attestations, held in the in-memory new_payloads and known_payloads buffers of the Store respectively. Promotion happens at tick intervals 0 (if proposing) and 4 (end of slot).

Why Staged Promotion?

The staged design serves two purposes:

1. Consistency: All validators promote attestations at the same moments, reducing divergence in head selection. Without batching, validators with faster network connections would see different heads than slower ones.

2. Proposer fairness: The proposer computes the block against a known, fixed set of attestations. If new attestations could influence fork choice mid-computation, different validators might disagree on the head.

On-Chain vs Off-Chain Attestations

Attestations arrive from two sources, and how they enter the pipeline matters:

The proposer’s own attestation enters as pending (not active) deliberately. If it were immediately active, the proposer would gain an unfair weight advantage for their own block, a circular dependency where proposing a block gives you an extra vote toward making that block canonical.

Safe Target Selection

The safe target is a conservative head computed with a high weight threshold. It constrains the target field in attestations, which feeds into 3SF-mini for justification and finalization decisions. Validators still vote for the newest head they see (regular LMD-GHOST with min_score = 0) in the head field. The safe target only affects which blocks can progress toward finality. It is computed by running the same LMD-GHOST algorithm but with a non-zero min_score in the filtering phase.

                    SAFE TARGET vs HEAD
                    ────────────────────

    Regular head (min_score = 0):
    Follow heaviest branch, even with a slim margin

             ┌── B (3 votes) ← HEAD (3 > 2)
    J ── A ──┤
             └── C (2 votes)


    Safe target (min_score = ⌈2V/3⌉):
    Only follow branches with supermajority support

    V = 5 validators, threshold = ⌈10/3⌉ = 4

             ┌── B (3 votes) ← Below threshold (3 < 4), pruned
    J ── A ──┤
             └── C (2 votes) ← Below threshold (2 < 4), pruned

    Safe target = A (no children pass threshold)

This means the safe target lags behind the head. It only advances when a branch accumulates overwhelming support, making it resistant to temporary fluctuations:

    Timeline of safe target vs head:

    Slot:    10    11    12    13    14    15    16
    Head:    J     A     B     D     D     E     F
    Safe:    J     J     J     A     A     A     D
                                                  │
                        Safe target is always ────┘
                        at or behind the head

The safe target prevents 3SF-mini from finalizing unstable branches: without it, a slim-majority fork could reach justification and finalization before the network converges. By requiring supermajority support for the target, only branches with strong consensus can progress toward finality, even though validators’ head votes freely follow the newest chain tip.

Reorgs

A reorg (reorganization) occurs when the fork choice head switches from one branch to another. This happens when a competing branch accumulates more attestation weight than the current head’s branch.

                    REORG SCENARIO
                    ──────────────

    Before (head = D):

              ┌── B ── D   ★ HEAD (weight 4)
    J ── A ──┤
              └── C ── E      (weight 3)


    New attestations arrive, 3 validators switch to E:

              ┌── B ── D      (weight 2)
    J ── A ──┤
              └── C ── E   ★ HEAD (weight 5)    ← REORG!


    The canonical chain changed from  J─A─B─D  to  J─A─C─E
    Blocks B and D are no longer canonical (but remain in the block tree).

Reorgs are normal during transient network conditions but should be rare in stable operation. They cannot cross a finalization boundary: once a block is finalized, it is permanently part of the canonical chain.

In ethlambda: Reorgs are detected by checking whether the old and new heads share a common prefix, and tracked via Prometheus metrics (lean_fork_choice_reorgs_total).

LMD-GHOST Variants

LMD-GHOST is one of several variants that have been proposed and studied. Understanding the design space helps explain why LMD was chosen.

The Ethereum consensus mini-spec originally used IMD-GHOST but switched to LMD in November 2018 due to superior stability properties.

    IMD: All attestations count         LMD: Only latest counts

    V0: slot 5 → head B                V0: slot 5 → head B  (overwritten)
    V0: slot 8 → head C                V0: slot 8 → head C  ← active
    V0: slot 11 → head E               V0: slot 11 → head E ← active

    V0 contributes 3 votes!            V0 contributes 1 vote.
    Validators who attest more          Equal influence regardless
    often have outsized influence.      of attestation frequency.

ethlambda Implementation Reference

This section covers ethlambda-specific details: scheduling, Beacon Chain differences, source code locations, and performance.

Tick-Based Scheduling

ethlambda divides time into slots, each split into 5 intervals. The slot lasts 4 seconds unless the network’s config file sets MILLISECONDS_PER_SLOT, so the 800 ms intervals below are the default grid, as described in Slots and Intervals. Fork choice operations are scheduled at specific intervals:

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │            │            │            │            │            │
    │IF PROPOSER:│ ALL        │ aggregators│update_safe │accept_new_ │
    │ accept new │ VALIDATORS:│ publish    │_target()   │attestations│
    │ attestation│  produce   │ aggregated │            │()          │
    │ + propose  │ attestation│ attestation│ (2/3 vote  │            │
    │ block      │            │            │ threshold) │update_head │
    │            │            │            │            │()          │
    │update_head │            │            │            │            │
    │()          │            │            │            │            │
    └────────────┴────────────┴────────────┴────────────┴────────────┘

    ◄─────────────── Slot N ──────────────────────────────────────────►

Detailed sequence:

    Interval 0 ─ Slot boundary
    │
    ├── Am I the proposer for this slot?
    │   ├── YES: promote new → known attestations
    │   │        run fork choice → update_head()
    │   │        build block using known attestations
    │   │        publish block to network
    │   └── NO:  (wait for block from proposer)
    │
    Interval 1 ─ Attestation production
    │
    ├── All validators, proposer included:
    │   └── Create attestation with:
    │       • head   = current fork choice head (newest head)
    │       • target = derived from safe_target (for 3SF-mini)
    │       • source = latest_justified checkpoint
    │       Publish attestation to gossipsub
    │
    Interval 2 ─ Aggregation
    │
    ├── Aggregators: aggregate their subnet's gossip signatures
    │   └── Publish the aggregated attestation to gossipsub
    │
    Interval 3 ─ Safe target update
    │
    ├── Recalculate safe_target using 2/3 supermajority threshold
    │   └── Only blocks with ≥ ⌈2V/3⌉ attestation weight qualify
    │       (V = total validators)
    │
    Interval 4 ─ End of slot
    │
    ├── Promote new → known attestations
    └── Run fork choice → update_head()

Differences from the Ethereum Beacon Chain

ethlambda is a lean consensus client with several simplifications compared to the Ethereum Beacon Chain:

No proposer boost. The Beacon Chain adds a “proposer boost”, a temporary weight bonus given to newly proposed blocks to prevent balancing attacks. ethlambda does not implement this. Instead, proposer fairness is handled through the two-stage attestation pipeline (the proposer’s own attestation enters as “new”, not “known”).

No balance weighting. In the Beacon Chain, a validator with 32 ETH of effective balance has more fork choice weight than one with 16 ETH. In ethlambda, every validator has exactly equal weight (1 vote = 1 unit of weight), simplifying the algorithm and analysis.

No equivocation discounting. The Beacon Chain’s fork choice detects validators who equivocate (attest to conflicting blocks in the same slot) and excludes their weight. This addresses the “nothing at stake” problem where validators can costlessly vote for multiple forks. ethlambda does not implement this in its fork choice.

Key Files

Data Flow Summary

     ┌───────────┐         ┌──────────────┐             ┌───────────────┐
     │ Gossipsub │────────▶│ New          │──(promote)─▶│ Known         │
     │ (network) │         │ Attestations │             │ Attestations  │
     └───────────┘         └──────────────┘             └───────┬───────┘
                                                                │
     ┌───────────┐                                              │
     │ LiveChain │──── { root → (slot, parent) } ───────────────┤
     │  (index)  │                                              │
     └───────────┘                                              │
                                                                ▼
                                                  ┌─────────────────┐
     ┌───────────┐                                │ compute_lmd_    │
     │ Justified │──── start_root ───────────────▶│ ghost_head()    │
     │Checkpoint │                                │                 │
     └───────────┘                                └────────┬────────┘
                                                           │
                                                    ┌──────┴──────┐
                                                    │             │
                                                    ▼             ▼
                                              ┌──────────┐ ┌───────────┐
                                              │   HEAD   │ │   SAFE    │
                                              │ (min=0)  │ │  TARGET   │
                                              └──────────┘ │ (min=2V/3)│
                                                           └───────────┘

Performance Characteristics

In practice with a small validator set and bounded non-finalized chain length, all operations complete in sub-millisecond time. The // TODO: add proto-array implementation comment in the source indicates a future optimization path: proto-array is an O(1) amortized fork choice algorithm used by most Beacon Chain clients.

HTTP API

ethlambda exposes HTTP over two independent Axum servers on separate ports, so the API and the metrics/debug surface can have different network policies:

• API server — consensus data (blocks, states, checkpoints, fork choice) and admin controls.
• Metrics & debug server — Prometheus metrics and heap-profiling endpoints. No store access.

All consensus API paths are versioned under the /lean/v0 prefix. Roots are serialized as 0x-prefixed hex strings.

Servers & Ports

If --api-port and --metrics-port are equal, all routers are merged onto a single port.

API Server (:5052)

GET /lean/v0/health

The handler emits a fixed, compact body (no whitespace):

{"status":"healthy","service":"lean-rpc-api"}

GET /lean/v0/config/spec

Protocol parameters the node is running on. Keys mirror the leanSpec constant names. MILLISECONDS_PER_SLOT and MILLISECONDS_PER_INTERVAL reflect the network’s config file rather than a compile-time constant, so a node on an 8-second network reports 8000 and 1600 here:

{
  "MILLISECONDS_PER_SLOT": 4000,
  "INTERVALS_PER_SLOT": 5,
  "MILLISECONDS_PER_INTERVAL": 800,
  "HISTORICAL_ROOTS_LIMIT": 262144,
  "FORK_DIGEST": "12345678"
}

FORK_DIGEST is the 4-byte hex string (no 0x prefix) embedded in gossipsub topic names.

GET /lean/v0/genesis

{ "genesis_time": 1770407233, "validator_count": 16 }

validator_count is read from the head state’s validator registry. Lean validators are fixed at genesis (no churn), so it always equals the size of the genesis registry.

GET /lean/v0/states/finalized

SSZ-encoded State at the latest finalized checkpoint (Content-Type: application/octet-stream). The served state has its latest_block_header.state_root zeroed to match the canonical post-state representation the state transition produces, so checkpoint-sync peers reconstruct an identical state root. See Checkpoint Sync.

GET /lean/v0/blocks/finalized

SSZ-encoded SignedBlock at the latest finalized checkpoint. The genesis/anchor block has no stored signature, so a placeholder blank proof is synthesized and the endpoint still returns 200. Returns 404 only in the rare case where a non-genesis finalized block’s signature has been pruned below the finalized boundary and can no longer be served.

GET /lean/v0/checkpoints/justified

{ "slot": 128, "root": "0x1a2b…" }

GET /lean/v0/events

Server-Sent Events stream (Content-Type: text/event-stream) of live chain events published by the blockchain actor. Seven event types:

Payload fields mirror the Ethereum beacon-API eventstream where an analog exists: block is the block root, state the state root, and slot stands in for the beacon epoch. justified_checkpoint and aggregate are ethlambda extensions with no beacon topic.

The topic name travels only on the SSE event: line; the data: line carries the flat JSON payload. Example frame:

event: head
data: {"slot":128,"block":"0x1a2b…","state":"0x3c4d…"}

Filtering with ?topics=

A required comma-separated list of event names selects which events to stream:

curl -N 'http://127.0.0.1:5052/lean/v0/events?topics=head,finalized_checkpoint'

Valid values are exactly the event names above: head, block, justified_checkpoint, finalized_checkpoint, block_gossip, attestation, aggregate. As in the Beacon API eventstream endpoint, topics is mandatory: there is no “subscribe to everything” default; list the topics you want.

Events are fanned out over a single bounded broadcast channel shared by all topics. A client that reads too slowly skips past the events it missed: they are dropped for that subscriber rather than back-pressured onto the actor, so treat the stream as best-effort and re-sync via the blocks endpoints after a gap. A client that falls behind receives an SSE comment line : error - dropped N messages marking the gap (wire-compatible with Lighthouse) before the stream continues; re-sync via the blocks endpoints rather than trusting the skipped range. Keep-alive comments are sent periodically to hold idle connections open.

Because the ring buffer is shared, the high-rate attestation events (roughly one per validator per slot) dominate its occupancy: a subscriber’s tolerable stall is capacity / total_event_rate, not per-topic, so filtering with ?topics= narrows what you receive but does not widen the lag window against an attestation flood. Subscribers that only need low-rate topics (head, finalized_checkpoint, …) are still evicted at the aggregate rate. If real usage shows this biting, the fix is a per-topic channel split behind the event bus (the subscribe(TopicSet) API is unaffected).

GET /lean/v0/blocks/{block_id} and /header

block_id is either:

• a 0x-prefixed 32-byte hex root, or
• a decimal slot.

Slot lookups resolve through the head state’s historical_block_hashes, so only canonical blocks are reachable by slot; blocks on side forks must be addressed by their root. The /header variant returns just the BlockHeader.

Error bodies are JSON: { "error": "invalid block_id" } / { "error": "block not found" }.

GET /lean/v0/fork_choice

The fork-choice tree from the finalized root, with LMD-GHOST weights computed over the live chain and currently known attestations.

{
  "nodes": [
    { "root": "0x…", "slot": 128, "parent_root": "0x…", "proposer_index": 3, "weight": 12 }
  ],
  "head": "0x…",
  "justified": { "slot": 128, "root": "0x…" },
  "finalized": { "slot": 96,  "root": "0x…" },
  "safe_target": "0x…",
  "validator_count": 16
}

/lean/v0/fork_choice/ui serves an interactive D3.js page rendering this data. See Fork Choice Visualization.

GET /lean/v0/node/identity

{
  "version": "ethlambda/v0.1.0-main-892ad575/x86_64-unknown-linux-gnu/rustc-v1.97.1",
  "peer_id": "16Uiu2HAm7v1x…"
}

version is the full client version string, identical to what ethlambda --version prints: crate semver, git branch and short SHA, target triple, and rustc version. Baked in at compile time from CARGO_PKG_VERSION plus the vergen-git2 build metadata.

peer_id is the node’s libp2p peer ID (base58), derived from the node key and fixed for the lifetime of the process; it matches the identity the node presents to peers on the wire.

GET /lean/v0/node/syncing

{ "is_syncing": false, "head_slot": 1024, "sync_distance": 1, "finalized_slot": 986 }

is_syncing is the node’s own stateful sync decision: head-vs-wall-clock lag with hysteresis and a network-stall override, updated each tick. It is the same signal that gates validator duties and drives the lean_node_sync_status metric, so the endpoint, the gate, and the metric always agree.

sync_distance is the raw number of slots between the node’s current head and the current wall-clock slot, computed per request. Because is_syncing carries hysteresis and stall handling and is not recomputed from sync_distance, the two can point different ways near the threshold or during a network-wide stall.

GET / POST /lean/v0/admin/aggregator

Toggle the aggregator role at runtime without restarting the node (hot-standby model, ported from leanSpec PR #636).

# Read current role
curl http://127.0.0.1:5052/lean/v0/admin/aggregator
# → {"is_aggregator": true}

# Toggle role; body must be a JSON boolean
curl -X POST http://127.0.0.1:5052/lean/v0/admin/aggregator \
  -H 'content-type: application/json' -d '{"enabled": false}'
# → {"is_aggregator": false, "previous": true}

Note: Runtime toggles do not resubscribe gossip subnets, which are frozen at startup. A standby aggregator should boot with --is-aggregator=true (so subscriptions are in place), then use this endpoint to rotate duties. See the CLAUDE.md “Runtime Aggregator Toggle” notes for the operational model.

Metrics & Debug Server (:5054)

The metrics endpoint reads from the global Prometheus registry and needs no store access. See Metrics for the full list of exposed series.

Heap-profiling endpoints are backed by jemalloc’s built-in profiler and are only functional on Linux; other platforms return 501 Not Implemented. On Linux they return 500 if profiling was not enabled at startup.

Test-Driver Endpoints (Hive)

When the binary boots with HIVE_LEAN_TEST_DRIVER=1 (any of 1/true/yes), it runs in test-driver mode instead of the normal API server. The ethereum/hive lean simulator drives these endpoints to replay leanSpec fixtures over HTTP. The driver swaps its in-process Store on every fork_choice/init, so one container can replay many fixtures without restart.

Content Types

Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

A ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The exposed metrics follow the leanMetrics specification, with some metrics not yet implemented. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

PQ Signature Metrics

Block Production Metrics

lean_block_building_time_seconds intentionally deviates from the leanMetrics bucket set, which tops out at 1s. Real builds on our devnets routinely run past that, so every sample landed in +Inf and histogram_quantile reported a flat 1s ceiling. The range now covers the same span as the lean_block_proposal_attestation_build_phase_seconds phases it contains.

Fork-Choice Metrics

State Transition Metrics

Validator Metrics

Network Metrics

All three are emitted only by aggregators, once per candidate AttestationData per interval-2 session. lean_aggregation_window_width has buckets 1, 2, 4, 8, 16, 32, 64 and climbs from 1 as the aggregator’s anchor proof climbs the reduction tree; it is capped at lean_attestation_committee_count, so samples pinned there mean the window no longer restricts selection. Compare against that gauge rather than reading the buckets alone: at a committee count that is not a power of two, two different widths can share a bucket. A width stuck at 1 while the network is aggregating means the pool holds nothing on this node’s duty subnet, so it is only aggregating its own raw signatures; check the duty subnet against the aggregator placement. lean_aggregation_skipped_redundant_total only increments with --skip-redundant-aggregation, once per candidate handed to another duty subnet; read it against lean_aggregation_window_width_count for the share of candidates sat out. lean_aggregation_window_fallback_total counts recoveries, not attempts: it increments only when a windowed selection produced no viable job and the full-committee-width retry did. Candidates that are non-viable whatever the window, a lone raw signature or a group with a single proof, never reach the retry and never increment it, so a sparse (strided) aggregator placement across subnets is the only expected cause and the counter should stay at or near zero on a well-tiled deployment. It stays flat entirely under --skip-redundant-aggregation, which disables the fallback.

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Network Sizes

Peer Discovery

Only emitted when discv5 discovery is enabled (--discovery.enable); see Peer discovery. Counts dials discovery initiated, as opposed to the static bootnode dials every node makes. Connection outcomes are not repeated here: a discovery dial that succeeds or fails shows up in lean_peer_connection_events_total like any other.

Transport Mix

Which transport actually carried each established connection, read off the connection’s own multiaddr rather than off the address we dialed: libp2p races a peer’s QUIC and TCP addresses within one dial, so the answer is not knowable before the connection exists. tcp counts are what say the fallback in Peer discovery is doing work rather than merely being advertised.

Counts connections rather than peers, so it can exceed lean_peer_connection_events_total{result="success"}, which fires only on a peer’s first connection. unknown covers a multiaddr naming neither transport, which nothing ethlambda binds produces.

Gossip Arrival Timing

These histograms record the absolute distance between a gossip message’s arrival and the start of the interval it was due in, so an arrival that is early by some amount and one that is late by the same amount land in the same bucket; the counters’ position label is what tells them apart. inside means the message arrived within the interval it was due in, not merely somewhere in the right slot: an attestation for slot 10 that lands during slot 10’s interval 2 is after, not inside, since it missed the AttestationProduction interval it was actually due in.

The bucket boundaries are the interval and slot edges of the default 4-second cadence. Prometheus fixes buckets when a histogram is registered, so a network that sets MILLISECONDS_PER_SLOT reads these histograms against the default grid rather than its own; the position label still follows the configured interval width.

Blocks anchor to interval 0 of their own slot and attestations to interval 1 of their data slot; both are unbounded above, so a message that never arrives close to real time can be arbitrarily late. Aggregates anchor instead to the most recent aggregation-interval boundary rather than their own data slot, since a stale-group catch-up aggregate can carry a data.slot several slots in the past; anchoring to the latest boundary bounds the delay to one slot and rules out before entirely.

Only gossip-received blocks are sampled here: blocks fetched via req/resp during sync are excluded, since sync backfill delivers blocks long after they were due and would swamp these histograms with catch-up noise rather than gossip-health signal.

The aggregate metrics do include an aggregator’s own freshly produced aggregates, which never come back over gossip; without them an aggregator would report an empty aggregate profile. The two populations are not quite the same measurement: delivery of a locally produced aggregate is held until the interval-2 boundary, so it lands near zero unless proving overran the interval, whereas a received one adds propagation on top of whenever the producer managed to publish it.

In practice the distribution is bimodal and dominated by production rather than propagation: a mode in the lowest bucket for aggregates that made their interval, plus a tail for those whose proving overran it. A late aggregate is late for every node at once, so that tail shows up on receivers too and is not evidence of a slow network. Read a rising tail as aggregation cost, and cross-check lean_pq_sig_aggregated_signatures_building_time_seconds and lean_committee_signatures_aggregation_time_seconds to confirm.

Storage

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

• timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
• agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
• proposal_combined is emitted only when this node proposes a block.

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Troubleshooting

Docker Desktop on MacOS

lean-quickstart uses the host network mode for Docker containers, which is a problem on MacOS. To work around this, enable the “Enable host networking” option in Docker Desktop settings under Resources > Network.

Checkpoint Sync

Overview

Checkpoint sync allows a new consensus node to skip replaying the entire chain from genesis. Instead, it downloads a recent finalized state from a running peer and starts from there. This mitigates long-range attacks by starting from a recent trusted checkpoint.

Usage

Checkpoint sync still requires the network config files (genesis, validators, bootnodes, etc.). The genesis config is needed to verify the downloaded state: checkpoint sync only replaces the starting state, not node configuration.

Pass the --checkpoint-sync-url flag when starting ethlambda:

ethlambda \
  --checkpoint-sync-url <URL> \
  --genesis ./network-config/config.yaml \
  --validators ./network-config/annotated_validators.yaml \
  --bootnodes ./network-config/nodes.yaml \
  --validator-config ./network-config/validator-config.yaml \
  --hash-sig-keys-dir ./network-config/hash-sig-keys \
  --node-key ./node.key \
  --node-id ethlambda_0

Where <URL> is the address of a checkpoint source (see Checkpoint Sources below).

State already on disk takes precedence over both checkpoint sync and genesis: if the data directory holds a previous run’s chain state for this network, the node resumes from it. --checkpoint-sync-url is the fallback for when there is nothing resumable on disk, or when what is there has fallen too far behind (see Restarts and Existing State). With no resumable state and no URL, the node initializes from genesis.

Checkpoint Sources

Direct peer

Any running node that serves the finalized state as SSZ can be used as a checkpoint source, not just ethlambda. For ethlambda nodes, the endpoint is /lean/v0/states/finalized.

This is the simplest option, with no additional infrastructure needed. The trade-off is that you trust a single peer to provide a correct finalized state.

Leanpoint

Leanpoint is a dedicated checkpoint sync provider. It polls multiple nodes and only serves state when 50%+ agree on finality, adding a layer of consensus validation.

This is the recommended option for production deployments since it reduces trust in any single peer.

How It Works

1. Fetch and verify: The node sends an HTTP GET to the provided URL requesting the SSZ-encoded finalized state. Once downloaded, the state is decoded and verified against the local genesis config (see Verification Checks below).

   Timeouts:

   • Connect: 15 seconds (fail fast if peer is unreachable)
   • Read: 15 seconds of inactivity that resets on each successful read, so large states can download as long as data keeps flowing

2. Initialize: The node stores the block header and the full state from the checkpoint. No block body is stored since it isn’t available from the checkpoint. The node does not need the anchor block body to participate from this point forward.

Failure and success

If any step fails (network error, decoding error, verification failure), the node logs the error and exits. There is no automatic retry; restart the node to try again. The database is not modified until verification succeeds, so a failed checkpoint sync leaves the data directory clean.

After successful initialization, the node starts normally: it connects to the P2P network and begins participating from the checkpoint slot.

Restarts and Existing State

A node restarted against a populated data directory resumes from disk rather than re-initializing, so no flag is needed to preserve the chain across a redeploy. The decision is made before any download:

The resume window is MAX_RESUMABLE_DB_STATE_AGE (450 slots, ~30 minutes at 4-second slots) measured as current_slot - head_slot. Staleness is measured against the head, not the finalized checkpoint, so a node whose head is current still resumes during a finality stall.

Beyond that window the node prefers a checkpoint when one is offered, since catching up over P2P costs more than downloading a recent state. With no URL configured there is no anchor to switch to, so the node simply runs against the data directory it was given: that is the setup that was asked for. The warning is there because range sync may not be able to close a gap this large. Peers prune block signatures past SIGNATURE_PRUNING_RANGE (21600 slots, ~1 day), so beyond that horizon they cannot serve the history the node is missing and it needs a checkpoint URL to catch up at all. The warning logs the gap so this is visible in the boot log.

When a checkpoint URL is set and every URL fails, the node exits rather than falling back to the stale state on disk. This is intentional: configuring the flag asks for a specific anchor, so an unreachable source is a misconfiguration worth surfacing at boot instead of quietly starting a node that is hours behind. Omitting the flag is how you ask for “resume whatever is on disk”; that path never exits.

To deliberately discard existing state and start over from genesis or from a checkpoint, remove the data directory first. Checkpoint sync itself writes its anchor state on top without clearing existing data.

Foreign State

Persisted state is accepted only after it is verified against the local genesis config: same GENESIS_TIME, same MILLISECONDS_PER_SLOT, and the same validator registry (count, sequential indices, and both pubkeys per validator). The validator set is fixed at genesis, so any state of this chain must carry exactly that registry. These are the same identity checks checkpoint sync applies to a downloaded state, sharing one implementation.

If the data directory belongs to a different network, startup aborts with persisted state does not match the configured genesis: …. It is not treated as an empty directory, because initializing a new anchor on top would leave the foreign chain’s rows in place, and the slot-indexed reads behind BlocksByRange would then serve those blocks to peers. Point --data-dir at the right directory, or remove it.

Note that a genesis time comparison alone would not catch a network that was regenerated with the same GENESIS_TIME but a different validator set, which is why the whole registry is compared.

Verification Checks

All checks are performed before a downloaded checkpoint state is accepted. The genesis-identity subset (marked below) is shared with the resume-from-disk path:

HTTP errors and SSZ decoding failures are caught before verification runs.

Security Considerations

Trust model

Checkpoint sync operates under a weak subjectivity assumption. In proof of work, any node can objectively determine the canonical chain by verifying the most cumulative work. Proof of stake doesn’t have this property: validators can costlessly sign multiple forks, so a node that wasn’t online to observe the chain in real time cannot distinguish the real chain from a fabricated one using protocol rules alone.

Weak subjectivity resolves this: a new node obtains a recent trusted state through a social channel (a peer, a checkpoint provider, a block explorer) and starts from there. Nodes that are always online are unaffected because they continuously track the chain and don’t need external trust.

What you are trusting:

• The checkpoint source is honest about which state is finalized
• The state hasn’t been crafted to put you on a fork that diverged within the weak subjectivity period

What verification does protect against:

• Wrong network (genesis time mismatch)
• Wrong validator set (pubkey or count mismatch)
• Structurally invalid states (impossible slot orderings, inconsistent checkpoints)
• Corrupted data (SSZ decode failures)

What verification does not protect against:

• A checkpoint source that serves a structurally valid state on a minority fork. It will pass all checks but put you on the wrong chain. This is why the choice of checkpoint source matters.

Fork Choice Visualization

A browser-based real-time visualization of the LMD GHOST fork choice tree, served from the existing RPC server with no additional dependencies.

Endpoints

Both endpoints are served on the API port (--api-port, default 5052).

Quick Start

Local devnet

make run-devnet

The local devnet runs 3 ethlambda nodes with metrics ports 8085, 8086, and 8087. Open any of them:

• http://localhost:8085/lean/v0/fork_choice/ui
• http://localhost:8086/lean/v0/fork_choice/ui
• http://localhost:8087/lean/v0/fork_choice/ui

Standalone node

cargo run --release -- \
  --genesis ./config/config.yaml \
  --validators ./config/annotated_validators.yaml \
  --bootnodes ./config/nodes.yaml \
  --validator-config ./config/validator-config.yaml \
  --hash-sig-keys-dir ./config/hash-sig-keys \
  --node-key ./keys/node.key \
  --node-id 0 \
  --api-port 5052

Then open http://localhost:5052/lean/v0/fork_choice/ui.

Visualization Guide

Color coding

Layout

• Y axis: slot number (time flows downward)
• X axis: fork spreading — branches appear when competing chains exist
• Circle size: scaled by weight / validator_count — larger circles have more attestation support

Interactive features

• Tooltips: hover any block to see root hash, slot, proposer index, and weight
• Auto-polling: the page fetches fresh data every 2 seconds
• Auto-scroll: the view follows the head as the chain progresses

What to look for

• Single vertical chain: healthy consensus, no forks
• Horizontal branching: competing chains — check attestation weights to see which branch validators prefer
• Color transitions: blocks turning green as finalization advances
• Stalled finalization: if justified/finalized slots stop advancing, check validator attestation activity

JSON API

curl -s http://localhost:5052/lean/v0/fork_choice | jq .

Response schema:

{
  "nodes": [
    {
      "root": "0x...",
      "slot": 42,
      "parent_root": "0x...",
      "proposer_index": 3,
      "weight": 5
    }
  ],
  "head": "0x...",
  "justified": { "root": "0x...", "slot": 10 },
  "finalized": { "root": "0x...", "slot": 5 },
  "safe_target": "0x...",
  "validator_count": 8
}

💾 Storage: What Lives Where

This doc explains how ethlambda saves data. Especially, the split between the fork choice Store and the StorageBackend trait, what each of the eight tables holds, and which data is in-memory only.

Overview

All chain data flows through a single high-level type, the Store (crates/storage/src/store.rs), which persists it through a small pluggable key-value abstraction, the StorageBackend trait (crates/storage/src/api/traits.rs). Two backends implement the trait: RocksDB for production and an in-memory backend for tests. Everything persisted is SSZ-encoded bytes.

                        LAYERED ARCHITECTURE
                        ────────────────────

   ┌──────────────────┐          ┌──────────────────┐
   │ BlockChain actor │          │    P2P actor     │
   └────────┬─────────┘          └────────┬─────────┘
            │      (cloned Store: shared  │
            │       backend + buffers)    │
            ▼                             ▼
   ┌─────────────────────────────────────────────────┐
   │                     Store                       │
   │        crates/storage/src/store.rs              │
   │                                                 │
   │  • table selection, key encoding, SSZ codec     │
   │  • snapshot-vs-diff decisions, pruning          │
   │  • in-memory attestation buffers + state cache  │
   └────────────────────────┬────────────────────────┘
                            │ begin_read() / begin_write()
                            ▼
   ┌─────────────────────────────────────────────────┐
   │              StorageBackend trait               │
   │        crates/storage/src/api/traits.rs         │
   │                                                 │
   │        raw bytes in, raw bytes out              │
   └───────────┬─────────────────────────┬───────────┘
               │ (production)            │ (test)
               ▼                         ▼
   ┌───────────────────────┐   ┌───────────────────────┐
   │    RocksDBBackend     │   │    InMemoryBackend    │
   │  (production, one     │   │  (tests, HashMap per  │
   │   column family per   │   │   table, lost on      │
   │   table)              │   │   drop)               │
   └───────────────────────┘   └───────────────────────┘

The Store / StorageBackend Split

StorageBackend: dumb bytes

The StorageBackend trait knows nothing about consensus types. It moves raw bytes in and out of named tables:

• begin_read() returns a StorageReadView with get(table, key) and prefix_iterator(table, prefix).
• begin_write() returns a StorageWriteBatch with put_batch, delete_batch, and commit(). A batch stages puts and deletes across multiple tables and applies them atomically on commit.

The two implementations live in crates/storage/src/backend/:

Store: all the semantics

The Store owns everything the backend doesn’t: which table each datum goes to, how keys are built, SSZ encoding/decoding, when to write a full state snapshot versus a diff, and when to prune. It is the only writer to the backend.

A naming subtlety: the Store struct lives in the storage crate (crates/storage/src/store.rs), while the fork choice logic that drives it (on_block, on_tick, update_head, …) lives in crates/blockchain/src/store.rs as free functions taking &mut Store.

Store is Clone, and every field is an Arc, so clones are cheap and all clones share the same backend and the same in-memory pools. At startup (bin/ethlambda/src/main.rs) one Arc<RocksDBBackend> is opened, one Store is built from it, and clones are handed to the BlockChain and P2P actors.

                        INSIDE THE STORE
                        ────────────────

   ┌──────────────────────────── Store ────────────────────────────┐
   │                                                               │
   │   PERSISTED (via backend)         IN-MEMORY ONLY              │
   │   ──────────────────────          ──────────────────────      │
   │   ┌─────────────────────┐         ┌──────────────────────┐    │
   │   │ BlockHeaders        │         │ new_payloads         │    │
   │   │ BlockBodies         │         │  (pending aggregated │    │
   │   │ BlockProof          │         │   attestations)      │    │
   │   │ BlockRoots          │         │ known_payloads       │    │
   │   │ States              │         │  (fork-choice-active │    │
   │   │ StateDiffs          │         │   attestations)      │    │
   │   │ Metadata            │         │ gossip_signatures    │    │
   │   │ LiveChain           │         │  (raw XMSS sigs      │    │
   │   └─────────────────────┘         │   awaiting           │    │
   │                                   │   aggregation)       │    │
   │   Survives restarts.              │ state_cache (LRU)    │    │
   │                                   └──────────────────────┘    │
   │                                                               │
   │                                   Lost on restart.            │
   └───────────────────────────────────────────────────────────────┘

The Tables

The eight variants of the Table enum (crates/storage/src/api/tables.rs):

Key encoding

Three key layouts are used:

• Root-keyed tables use the 32-byte SSZ encoding of the block root (root.to_ssz()).
• Slot-prefixed tables (BlockProof, LiveChain) use encode_slot_root_key: an 8-byte big-endian slot followed by the 32-byte root. Big-endian means lexicographic key order equals numeric slot order, so pruning can iterate from the start of the table and stop at the first key past its cutoff instead of scanning everything.
• Slot-only (BlockRoots) uses encode_block_root_key: just the 8-byte big-endian slot, since the value already holds the root. This table is never pruned, so the ordering buys nothing here; it is kept only for consistency with the other slot-prefixed keys.

BlockHeaders

root → BlockHeader. Written for every block, including the genesis/anchor block, and never pruned: headers are the permanent record of the chain. Headers are also read back during state reconstruction (see State Storage).

BlockBodies

root → BlockBody. Written for every block except those with an empty body: if header.body_root == EMPTY_BODY_ROOT (the hash tree root of BlockBody::default()), nothing is stored and reads synthesize BlockBody::default(). This covers the genesis block and checkpoint sync anchors, whose bodies are either empty or unavailable. Never pruned.

BlockProof

slot ‖ root → MultiMessageAggregate. This table stores the block’s merged aggregate proof blob. It is keyed by slot ‖ root so that pruning can scan in slot order and stop early.

Stored separately from headers/bodies because the genesis block has no proof. get_signed_block synthesizes an empty proof for the slot-0 anchor only; for any other block a missing entry (a pruned finalized block) surfaces as None rather than a fabricated block.

This is the one block table that is pruned; see Pruning.

BlockRoots

slot → H256, the canonical block root at each slot. Rewritten on every head update inside update_checkpoints: block_root_index_changes walks the old and new head’s branches back to their common ancestor, deleting the slots that leave the canonical chain and writing the ones that join it. A reorg therefore touches only the affected slot range, not the whole table. Never pruned.

Backs get_signed_blocks_by_slot_range, which serves BlocksByRange requests over req/resp (crates/net/p2p/src/req_resp/handlers.rs). It does not back the RPC GET /lean/v0/blocks/:slot endpoint: that handler resolves a slot through the head state’s historical_block_hashes instead (resolve_slot in crates/net/rpc/src/blocks.rs), so a block on a side fork is reachable there only by root, never by slot.

States

root → State (full SSZ snapshot). Holds full-state snapshots only: the bootstrap anchor written at initialization, plus one anchor whenever a block crosses a SNAPSHOT_ANCHOR_INTERVAL-slot boundary. Never pruned — these anchors are the base every diff chain resolves against, so reconstruction always terminates.

The genesis validator registry is constant for the life of the chain (validators is fixed at genesis; the lean STF never mutates it), but it has no table of its own: it rides inside every States snapshot alongside config, which StateDiff reconstruction relies on (see State Storage).

StateDiffs

root → StateDiff. A parent-linked diff written for every non-genesis state. Never pruned, so together with the snapshots this preserves the full state history. See State Storage for what a diff contains and how states are rebuilt.

Metadata

String keys mapping to SSZ-encoded scalars — the Store’s own persistent fields:

config is the odd one out: init_store writes it once at bootstrap and nothing ever rewrites it afterward (it has a getter, Store::config, but no setter). Because it never changes, the Store keeps a copy in memory and reads of it never reach the backend. It is also part of the DB’s fingerprint: from_db_state refuses to resume a data directory belonging to another network (see Startup and Restore). Every other Metadata key is mutated in place as the chain progresses.

Note that this is not the SSZ StateConfig carried inside State. That one is merkleized into the state root, so its layout is fixed by the spec and holds only genesis_time; ChainConfig adds the slot duration, which the node needs to schedule duties but which never enters a state root. A blob written before the slot duration existed still decodes, filling in the 4-second default that chain ran on.

LiveChain

slot ‖ root → parent_root. A pure index for fork choice: it lets get_live_chain() build the root → (slot, parent_root) block tree without deserializing a single block. It contains the finalized anchor plus all non-finalized blocks, and is pruned as finalization advances (the finalized block itself is kept).

Presence in LiveChain is what makes a block visible to fork choice: insert_pending_block deliberately writes a block’s header/body/proof without a LiveChain entry, persisting the heavy proof data (~3 KB+) while the block waits for its parent. When the block is later processed, insert_signed_block overwrites the same keys (idempotent) and adds the LiveChain entry.

State Storage: Snapshots + Diffs

Storing a full State per block would be wasteful: most fields never change or change predictably. Instead, insert_state writes:

1. Always a StateDiff keyed by the block root, linked to its parent via base_root (the block’s parent_root).
2. Only at anchors a full snapshot into States. A block is an anchor when it crosses a SNAPSHOT_ANCHOR_INTERVAL slot boundary relative to its parent (~68 minutes at 4-second slots). This bounds any reconstruction walk to at most SNAPSHOT_ANCHOR_INTERVAL diff applications.

A StateDiff stores only what cannot be recovered elsewhere: the target slot, justified/finalized checkpoints, and the justification fields (justified_slots, justifications_roots, justifications_validators, stored in full — they are bounded by the non-finalized window, so they stay small under healthy finality). The rest is deliberately omitted:

The historical_block_hashes append is checked rather than trusted blindly: validate_history_append (crates/storage/src/state_diff.rs) rejects a diff whose appended hashes don’t match the expected slot gap or aren’t zero-filled for skipped slots, so a broken append surfaces at diff-creation time instead of corrupting a later reconstruction.

Reads go through get_state, which tries three levels:

1. An in-memory LRU cache (STATE_CACHE_CAPACITY = 32 states, keyed by block root). States are content-addressed and immutable, so the cache never needs invalidation. The common case — reading the parent state right after importing its block — is a cache hit.
2. A full snapshot in States.
3. Reconstruction: walk base_root pointers back through StateDiffs until a snapshot is found, then replay the diffs forward.

                    STATE RECONSTRUCTION
                    ────────────────────

   get_state(D): not in the cache and no snapshot → rebuild in two passes.

   Pass 1: walk backward from D, following each diff's base_root pointer
           and collecting diffs, until a block with a snapshot is found:

   ┌────────┐  base=C   ┌────────┐  base=B   ┌────────┐  base=A   ┌──────────┐
   │ diff D │ ───────▶  │ diff C │ ───────▶  │ diff B │ ───────▶  │ snapshot │
   └────────┘           └────────┘           └────────┘           │   at A   │
    (target)           (StateDiffs table)                         └──────────┘
                                                                 (States table)

   Pass 2: starting from the snapshot, apply the diffs oldest-first:

   state A ──apply B──▶ state B ──apply C──▶ state C ──apply D──▶ state D ✓

   The rebuilt state D gets its latest_block_header from the BlockHeaders
   table and is memoized in the LRU cache before being returned.

If the diff chain is broken or the target’s header is missing, get_state returns None rather than a partial state.

Write Paths: What a Block Import Persists

Block import (on_block in crates/blockchain/src/store.rs) commits a sequence of independent write batches:

                 BLOCK IMPORT WRITE SEQUENCE
                 ───────────────────────────

  on_block(signed_block)
   │
   ├─ 1. update_checkpoints()          Metadata: head,
   │      (only if the post-state      latest_justified
   │       justified a higher slot)    (+ triggers pruning)
   │
   ├─ 2. insert_signed_block()  ┐            BlockHeaders[root]
   │                            │            BlockBodies[root]    (if non-empty)
   │                            ├─one batch─ BlockProof[slot‖root]
   │                            │            LiveChain[slot‖root]
   │                            ┘
   │
   ├─ 3. insert_state()         ┐            StateDiffs[root]
   │                            ├─one batch─ States[root]         (anchors only)
   │                            ┘            (+ LRU cache insert)
   │
   └─ 4. update_head()                 Metadata: head
          (re-runs fork choice)        (+ justified/finalized if advanced,
                                          + BlockRoots diff (canonical index),
                                          + pruning on finalization)

Each numbered step is atomic on its own, but the import as a whole is not one transaction. The commit order keeps the on-disk store consistent after any prefix of these steps: the justified checkpoint written in step 1 always names an already-persisted ancestor of the imported block (the state transition only counts attestations whose roots match the state’s own historical_block_hashes, and every ancestor was fully persisted when it was imported), and the head only advances in step 4, after the block and state are durable. A crash mid-import can therefore lose the tail of the import — e.g. a persisted block and state the head does not point to yet — but never leave metadata referencing missing data. Re-importing the block is idempotent (a duplicate is skipped via has_state).

Pruning

Pruning is driven by finalization and splits into a cheap immediate phase and a deferred heavy phase.

Immediately, when finalization advances (inside update_checkpoints):

• prune_live_chain: deletes LiveChain entries below the finalized slot, keeping the finalized block itself. This keeps the fork choice working set bounded to the non-finalized chain.
• prune_gossip_signatures: drops buffered in-memory gossip signatures at or below the finalized slot.
• prune_stale_aggregated_payloads: drops in-memory aggregated payloads (both pending and known) whose target slot is at or below the finalized slot.

Deferred (prune_old_data, called after a batch of blocks has been processed):

• prune_old_block_proofs: deletes BlockProof entries below cutoff = tip_slot − BLOCK_PROOF_PRUNING_RANGE (21,600 slots, ~1 day at 4-second slots) — but only when cutoff ≤ finalized_slot, i.e. the entire pruned range lies within finalized history. Non-finalized proofs are never touched. Finalized blocks can never revert, so their proofs are not needed for fork choice, reorg safety, or re-aggregation once outside the window.

Never pruned: BlockHeaders, BlockBodies, BlockRoots, States, StateDiffs, and Metadata. Headers, bodies, the canonical slot index, and the snapshot+diff chain are the full historical record; only the proof blobs and the (non-finalized) fork choice index are disposable.

In-Memory Only (Lost on Restart)

Four Store fields never touch the backend. All are bounded buffers shared across Store clones:

The payload buffers evict FIFO when full, and redundant proofs (whose participants are a subset of an existing proof for the same attestation data) are skipped on insert.

Note that the per-validator “latest attestation” maps used by fork choice are not stored anywhere — they are derived on demand from these buffers via extract_latest_known_attestations and friends. See the attestation pipeline section of lmd_ghost.md for how attestations move between the pools.

After a restart these buffers start empty: pending attestations and un-aggregated gossip signatures are lost and must be re-collected from the network. Everything persisted in the eight tables survives.

Startup and Restore

A Store is created through one of three constructors in crates/storage/src/store.rs:

The first two funnel into init_store, which writes the anchor in one atomic batch: all six Metadata keys (time = 0, config, head = safe_target = anchor root, justified = finalized = anchor checkpoint), the anchor header, its BlockRoots entry, the body if non-empty, a full snapshot into States (the base of every future diff chain), and the anchor’s LiveChain entry.

from_db_state is the restore path: it reads config and latest_finalized from Metadata, returning None for an empty DB. A populated DB from another network is fatal instead: the persisted config’s genesis time and slot duration, plus the finalized state’s genesis time and validator registry, are compared against the genesis config, and a mismatch fails with Error::GenesisMismatch rather than being treated as empty, since writing a fresh anchor would leave the foreign chain’s rows in place to be served to peers. The slot duration has to be checked against the persisted config because it is absent from the state by design. At startup the node prefers this path but only accepts the on-disk store if its head is at most MAX_RESUMABLE_DB_STATE_AGE = 450 slots (~30 minutes) behind the current slot; a staler DB falls through to checkpoint sync, which writes a fresh anchor on top of the existing data.

Key Files

Peer discovery (discv5)

ethlambda can find peers over discv5 instead of relying only on the static bootnode list. The implementation reuses ethrex’s discovery stack, with discv4 disabled.

Discovery is off by default. Nothing else on the lean network speaks discv5 today: not leanSpec, not ream’s lean network, not zeam. Enabling it currently only finds other ethlambda nodes.

Enabling it

ethlambda --discovery.enable

--discovery.port and --gossipsub-port (default 9001, libp2p QUIC) are both UDP and so cannot share a port. --gossipsub-port also binds a libp2p TCP listener on the same number, which collides with neither: TCP and UDP are separate namespaces. The defaults are one apart, so --discovery.enable works on its own; overriding either onto the other is rejected at startup.

The discv5 socket always binds the wildcard 0.0.0.0, since that is where we listen, not where peers should dial us. Without --discovery.advertise-ip the published ENR inherits that same 0.0.0.0, which is not a dialable address: set the flag to 127.0.0.1 for a local devnet or to the host’s public address so the ENR is usable as soon as it is published. discv5’s PONG-based IP voting may still replace the advertised address later, once a peer’s response tells the node what its external address looks like.

The ENR

The layout follows the discovery domain of the beacon-chain phase0 p2p interface spec.

tcp and quic share the same port number: TCP and UDP are separate namespaces, so build_swarm binds both without a collision. Advertising both is what lets a peer whose quic port does not answer still reach this node over TCP. It also gets us past lighthouse’s discovery predicate, which requires enr.tcp4().is_some() || enr.tcp6().is_some() on top of the spec’s fork_digest comparison; the lean fork digest is still the cross-client dummy 0x12345678, so a beacon-chain client rejects us on that instead.

Both ports come from configuration rather than from the bound listeners, so a --gossipsub-port 0 would name neither of the two real OS-assigned ports. Startup rejects that combination when discovery is enabled, and the writer omits a 0 either way, matching every reader’s rule that 0 means absent.

The local ENR is logged once at startup.

This same record is handed to ethrex’s DiscoveryServer, so it is what answers discv5 queries: what we report and what peers see are the same bytes. If IP voting later changes our external address, ethrex edits and re-signs that record rather than rebuilding one, so the consensus entries survive the bump; only the sequence number and ip move, which the reported ENR then lags.

Which peers get dialed

A discovered peer is admitted only if:

• its ENR carries a decodable eth2 entry, and
• that entry’s fork_digest equals ours, and
• it advertises a quic port, a tcp port, or both.

A differing next_fork_version or next_fork_epoch is not grounds for rejection: the spec permits connecting to a peer that is incompatible with an upcoming fork but compatible now.

These checks are handed to ethrex’s peer table as a PeerFilter, so each record is judged the moment it arrives and a peer that fails is not offered for dialing. No rejection is final: the peer table runs the filter again as soon as the peer publishes a higher-seq ENR, so a node that adds a quic entry, or gains an address through discv5’s IP voting, is reconsidered without a restart.

A peer’s dial list carries every address it advertises, quic and tcp both, in one dial attempt. libp2p races them: it starts up to dial_concurrency_factor handshakes at once and keeps whichever completes first, dropping the other. So a peer whose quic port does not answer still connects over tcp with no separate retry and no connect timeout waited out first.

The list order is not a preference, and nothing should be read into it: the default concurrency factor exceeds the two addresses a lean peer can offer, so both are always attempted. The cost of that is the thing to know, since it is paid on every dial rather than only on a failure: two sockets and two handshakes per peer, on both ends, until one wins.

Admitted peers are ranked by how many attestation subnets they advertise that no currently connected peer covers, so discovery preferentially fills gaps in subnet coverage. A peer advertising no attnets is ranked last but never dropped.

Dialing stops once --discovery.target-peers peers are connected, and resumes if that count drops. That is all the flag does: it is the dial loop’s cutoff, and nothing in ethrex’s peer table or discv5’s own pacing enforces it (see below).

Bootnodes

The three entries a bootnode ENR can carry are read independently, because they answer different questions:

A bootnode is dropped only when it has none of the three: neither transport to dial nor a udp port to seed discv5 from. Any other combination is kept, including one with only quic, only tcp, only udp, or any pair. The ENRs lean-quickstart generates today carry ip/quic/secp256k1 and no udp, so they stay reachable but contribute nothing to discovery. A beacon-chain bootnode is close to the mirror image, udp and tcp but no quic, and the tcp entry is what now makes it statically dialable rather than a discv5 seed only. A record missing an ip or a secp256k1 key is dropped regardless of its transports.

The ENR a node logs at startup is only useful to a peer if that node was started with a real --discovery.advertise-ip. Copying an ENR built from the default 0.0.0.0 into another node’s bootnode list produces a udp/quic/tcp target that cannot be dialed, since 0.0.0.0 names no reachable host. Set --discovery.advertise-ip before pointing other nodes at this one’s ENR: 127.0.0.1 on a local devnet, or the host’s public address otherwise.

Known limitations

Static bootnodes bypass admission and are redialed indefinitely

Everything under Which peers get dialed applies to discovered peers. A static bootnode reaches the swarm by a different path: parse_enr reads ip, secp256k1 and the three port entries and never looks at eth2, so a --bootnodes list is dialed as given. Now that a tcp-only record is dialable, a beacon-chain ENR is a valid static target, and the noise+yamux handshake to a lighthouse node succeeds: the peer occupies a --discovery.target-peers slot, contributes no attestation subnets, and is eventually dropped by the remote for sharing no protocols. Each drop re-arms the redial timer, which runs at a flat interval with no backoff and no cap for the life of the process.

Accepted rather than fixed. Bootnodes are operator-supplied, so a list naming another network’s infrastructure is a configuration mistake, and the unbounded redial is what keeps a devnet’s own bootnode reachable across its restarts. Splitting the flag into initial peers, dialed once, and bootnodes, seeded into discv5, is the real fix and is left to a follow-up.

An upgrade’s new ENR entries are invisible to peers that stayed up

The record is signed at ethrex’s INITIAL_ENR_SEQ, a constant. A peer identifies a record by (node id, seq) and accepts a replacement only at a strictly higher seq, and ethrex’s WHOAREYOU responder does not even send the record when the requester’s enr_seq already matches. So when an ethlambda release changes which entries it publishes, as adding tcp did, a peer holding the previous record under the same seq keeps it: the new entries reach only peers that meet this node for the first time.

A fixed local floor above INITIAL_ENR_SEQ does not close this. ethrex re-signs the record at seq + 1 whenever discv5’s IP voting moves the advertised address, so a node behind NAT may already be serving a seq above any constant the code could pick, which is exactly the case the floor was meant to cover. Closing it needs a seq that grows without bound across restarts: a persisted counter bumped on every content change, or one derived from the wall clock at startup, which is what several beacon clients do.

One lean devnet is not separated from another

The spec’s fork_digest is derived from genesis, so it separates one chain from another. ethlambda’s is the hardcoded cross-client dummy 0x12345678, and lean defines no fork schedule, so every ENRForkID field is a constant. The eth2 check therefore separates lean from non-lean but not one lean devnet from another: two devnets running this code will peer with each other. Closing that gap requires lean adopting a genesis-derived fork digest, which is a cross-client change to gossip topic names.

discv5 lookups run at the startup rate

ethrex paces its discv5 iterative lookups by how full its own peer table is, easing from one lookup every 500ms at startup to one every 10s once the table reaches its target. That table only counts peers registered through NewConnectedPeer, which carries an RLPx connection; ethlambda connects over libp2p and registers nothing, so the count is permanently zero and the pacing never eases off the startup rate. A lean node therefore keeps looking up every 500ms rather than settling at 10s, roughly 20x the intended steady-state FindNode traffic, for the life of the process.

--discovery.target-peers deliberately does not feed that computation, since a target of 0 would make it divide by zero and re-fire the lookup timer with no delay at all. Closing the gap properly means ethrex learning about non-RLPx connections, which is an upstream change.

attnets is not a fixed-width SSZ Bitvector

The spec’s attnets is Bitvector[ATTESTATION_SUBNET_COUNT], a constant every conformant client shares, which is what makes an undelimited bitfield decodable. ethlambda derives the width from attestation_committee_count, which is runtime configuration, so two nodes can legitimately exchange bitfields of different lengths. The bit-packing convention is identical to the spec’s; only the width is negotiable. Readers tolerate a foreign length by treating bits past the end as unset, and a peer’s advertised subnets are clamped to the local committee count before they influence anything.

Validator Key Generation

ethlambda keygen writes the validator XMSS key set and manifest that --hash-sig-keys-dir reads, in the layout hash-sig-cli generate produces.

# One validator's attester and proposer pair
ethlambda keygen --output-dir keys

# A genesis for a three-node devnet
ethlambda keygen --num-validators 3 --output-dir local-devnet/genesis/hash-sig-keys

--output-dir is the only required flag.

Why the client generates its own keys

An XMSS key file is only usable by a client built against the same signature scheme, and the scheme lives in leanVM, which ethlambda pins to one revision.

The trap is that nothing in the file layout changes when the scheme does. leanVM’s move from Poseidon over KoalaBear to BLAKE2s over binary fields kept the public key at 32 SSZ bytes and the secret key in postcard, so a key set from the wrong revision satisfies every check a genesis generator makes: the file names match, the sizes match, and the manifest parses. It fails only later, when a signature is verified, and then it looks like a consensus bug rather than a provisioning one.

Generating here removes the second pin. These keys come from the same ethlambda-crypto types the node loads them with, so the generator and the loader cannot disagree about the format.

Flags

Each validator gets two independent keys, an attester and a proposer, so it can sign an attestation and a block in the same slot without spending one slot’s one-time leaf twice.

--log-num-active-epochs is the network’s lifetime, not a tuning knob: a key cannot sign past its range, and the range is fixed at generation. At the default cadence 2^18 slots is about 12 days, after which every validator holding such a key stops signing.

--force is off by default because a key is one-time-use material. Replacing a set that validators are still signing with makes each of them sign twice at the same slot, under a key someone else now holds.

Output

hash-sig-keys/
├── validator_0_attester_key_pk.ssz   32 bytes, SSZ
├── validator_0_attester_key_sk.ssz   postcard
├── validator_0_proposer_key_pk.ssz
├── validator_0_proposer_key_sk.ssz
├── validator_1_...
└── validator-keys-manifest.yaml

The .ssz extension on a secret key is a misnomer kept for the tooling’s sake: SSZ has no encoding for one, so it is postcard.

hash-sig-cli had an --export-format both that additionally dumped each key as serde JSON. Its own help called that legacy and nothing reads it, so this writes only the form above and takes no format flag. A caller carrying --export-format ssz from the old invocation has to drop it.

The manifest

key_scheme: XmssTargetSumLifetime32Dim42Base8
hash_function: BLAKE2s
encoding: TargetSum
pubkey_bytes: 32
lifetime: 4294967296
leanvm_rev: 48a904208d682848dac0e18ef8b01ebfc40df9ad
log_num_active_epochs: 18
num_active_epochs: 262144
num_validators: 3

validators:
  - index: 0
    proposer_key_pubkey_hex: 0x...
    proposer_key_privkey_file: validator_0_proposer_key_sk.ssz
    attester_key_pubkey_hex: 0x...
    attester_key_privkey_file: validator_0_attester_key_sk.ssz

leanvm_rev is ethlambda’s addition, and it is the field to trust. Read against the scheme parameters alone, two key sets from either side of the BLAKE2s rewrite are indistinguishable: key_scheme is built from the lifetime, V and CHAIN_LENGTH, and all three came through that change unaltered. hash_function does separate them but is a hardcoded label, since leanVM’s facade exports no name for its hash. The revision is resolved from Cargo.lock at build time and cannot drift from what the binary actually links.

generate-genesis.sh reads key_scheme and cross-checks pubkey_bytes against the pubkeys the manifest holds; the rest is informational.

Cost

Key generation is O(sqrt(range)) in memory and fans out over the bottom Merkle subtrees, so a large range is cheaper than it looks: about 3 seconds per validator at 2^18 epochs on an M4 Max, generated serially. A 1024-validator genesis is therefore tens of minutes, and worth doing once and keeping.

Building on aarch64 without the crypto extensions makes this, and every other leanVM operation, far slower. See the aarch64 entry in .cargo/config.toml.

Benchmarking block building

ethlambda benchmark measures block building the way the node performs it when it proposes, against a reproducible synthetic workload, with no devnet running.

Block building is otherwise only observable through the Prometheus histograms a live node exports. Those are noisy, depend on whatever the network happened to be doing, and cannot be diffed against a baseline — which makes them a poor instrument for tracking performance. The benchmark trades network realism for repeatability: the same parameters produce the same blocks every run, so two reports differ only where the code differs.

Running it

make bench                                  # defaults, mock crypto
BENCH_ARGS="synthetic" make bench           # real XMSS/leanVM crypto
BENCH_ARGS="synthetic --iterations 50" make bench

make bench is a thin wrapper. The binary takes the same arguments directly:

ethlambda benchmark synthetic --num-validators 8 --iterations 10 --key-cache ~/.cache/ethlambda-bench-keys
ethlambda benchmark synthetic --mock-crypto --num-validators 8 --iterations 10

Without --mock-crypto the run uses real cryptography end to end: seed-derived XMSS keys, real attestation signatures aggregated into leanVM type-1 proofs, and the proposer’s real seal (block-root signature, singleton type-1 wrap, type-2 merge), with every built block imported through the verifying on_block path. A default real run takes a few minutes; --key-cache saves the seed-derived keys so reruns skip key generation. A default mock run finishes in well under a second, which is why CI can afford to run one on every pull request.

Logs go to stderr and the report to stdout, so --format json pipes straight into jq.

What it measures

Each iteration enters produce_block_with_signatures and then seal_block — the same functions BlockChainServer::propose_block calls — and the harness reports the phases inside them:

overhead is wall minus the sum of the phases, so the columns add up by construction. In mock mode there is nothing to sign with, so the seal is skipped and its three phases are absent.

Deliberately outside the measured span, matching the boundary of the node’s own lean_block_building_time_seconds metric: gossip publish, the slot-alignment sleep, and importing the block that was just built. The import still happens between iterations — otherwise every iteration would build on the same head and process_slots would get more expensive as the run went on. Two such costs are reported anyway, because they are real crypto worth watching:

Phase times come from the sample sums of the existing lean_block_proposal_attestation_build_phase_seconds histogram, read before and after each build. Histogram sums accumulate raw f64 seconds, so the difference between two readings is the elapsed phase time and bucket boundaries play no part. Nothing is added to the hot path for the benchmark’s benefit. The harness asserts each phase was observed exactly once per build and fails the run otherwise, because a mis-attributed report is worse than no report.

Reading a report

Block-building benchmark — synthetic workload (real crypto)
  validators=2 warmup_slots=1 iterations=2 proofs_per_data=1 seed=42
  enable_proposer_aggregation=false max_attestations_per_block=3
  ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 leanvm=48a90420 os=macos arch=aarch64 threads=14

  iter           compact      merge_type2  select_payloads    sign_proposer     stf_simulate    wrap_proposer   overhead       wall  aggregate     import         root
  1              0.001ms        550.641ms          0.007ms          0.461ms          0.011ms         65.127ms    0.103ms  616.350ms  103.548ms   19.205ms   0x77465b33
  2              0.001ms       1175.326ms          0.015ms          2.024ms          0.015ms         73.124ms    0.119ms 1250.623ms   94.691ms   20.472ms   0xf7e48c73

  phase              count        min       mean        p50        p90        max
  compact                2    0.001ms    0.001ms    0.001ms    0.001ms    0.001ms
  merge_type2            2  550.641ms  862.983ms 1175.326ms 1175.326ms 1175.326ms
  ...
  wall                   2  616.350ms  933.487ms 1250.623ms 1250.623ms 1250.623ms

  outside the measured span:
  aggregate              2   94.691ms   99.119ms  103.548ms  103.548ms  103.548ms
  import                 2   19.205ms   19.838ms   20.472ms   20.472ms   20.472ms

Every measured iteration gets its own row, and the summary follows below it. Outliers are never discarded: XMSS signing and its Merkle-subtree cache misses produce legitimate heavy tails, and hiding them would misrepresent the thing being measured. A coefficient of variation above 10% is flagged so a noisy run is not mistaken for a result.

Percentiles are nearest-rank, without interpolation. Sample counts here are small, so an actual observed value is more informative than a blend of two neighbours.

The root column is the block root of each built block. It is what makes a before/after comparison trustworthy: if an optimization leaves the root sequence unchanged, it changed only speed and not which attestations were selected. If the roots move, the change altered block contents and the timing comparison means something different than intended.

Comparing two runs

Same seed and same parameters produce identical root sequences, so a baseline and a candidate can be diffed directly. The header line exists to tell you when they cannot be compared:

• leanvm is the resolved revision the binary was built against, read from Cargo.lock at build time. leanVM owns the whole signature stack (XMSS and aggregation), so a rev bump changes the measured crypto. Real-mode roots also depend on the seed-derived keys, so the same seed on the same leanVM revision reproduces the same signatures and the same roots.
• os, arch and threads change results across machines.

Two reports that disagree on any of those are not measuring the same thing.

Limitations

• Synthetic workloads only. Replaying a real datadir is not implemented, so results reflect a synthetic chain rather than a deep production state.
• Short-lived keys. Real-mode XMSS keys are generated for exactly the slots the run signs, so key generation is cheap but the OTS window advancement a long-lived validator key performs every 65,536 slots is never exercised.
• Mock mode skips the seal. Without keys there is nothing to sign, so the three seal phases only appear in real runs.

In CI

The Test job runs a short mock benchmark and asserts the JSON report’s shape (schema_version, one sample per iteration). It costs seconds, and it means a change to the report contract cannot land unnoticed.

Spec Deviations

ethlambda diverges from the leanSpec reference in a few places, mainly for performance reasons. This page lists those deviations; each will be fleshed out with rationale, implementation notes, and trade-offs over time.

Asynchronous signature aggregation with an early start and an early stop

Aggregation runs off the main BlockChainServer actor loop, may start before its interval, and stops early once it runs out of time.

• ethlambda: the actor snapshots everything aggregation needs (snapshot_aggregation_inputs, crates/blockchain/src/aggregation.rs) and spawns a tokio::task::spawn_blocking worker (run_aggregation_worker, aggregation.rs). Candidates are the store’s gossip-signature groups plus payload-only groups (new_payload_keys, which need at least two existing proofs to merge). A tiered greedy selector orders them by consensus value (current-slot before stale, then Finalize > Justify > Build, mirroring the block builder) and emits at most MAX_AGGREGATION_JOBS jobs, dropping to a single job in the slot before one of our validators proposes. The worker streams each finished group back as an AggregateProduced message; the actor loop is never blocked on XMSS work.
• Early start: a session normally fires at interval 2, but may start up to EARLY_AGGREGATION_WINDOW earlier once the 2/3 signature threshold is already met (maybe_start_early_aggregation, crates/blockchain/src/lib.rs), so the proof lands earlier in the slot.
• Early stop: a send_after(AGGREGATION_DEADLINE, ...) timer cancels the session that long after session start, so a session that started early also ends early (AGGREGATION_DEADLINE, aggregation.rs). The worker checks cancel.is_cancelled() before each job (aggregation.rs); in-flight jobs finish, remaining jobs are dropped.
• leanSpec: aggregate() is called inline and synchronously from tick_interval, at interval 2 only. It walks every attestation data with fresh evidence, with no job cap, no time budget, no worker, and no cancellation.
• Equivalence: on cancellation the worker emits only the groups that finished, so a slot may pack fewer aggregates than the synchronous path would; any such subset still yields a valid block, affecting how many votes are included rather than signature validity. The job cap has the same character: it bounds prover work per slot, not what a block may carry.

Attestation scoring on block building

Attestations are scored and selected when packing a block, rather than taken in target-slot order as they are scanned.

• ethlambda: select_attestations (crates/blockchain/src/block_builder.rs) ranks candidate AttestationData entries by tier Finalize > Justify > Build (enum Tier, block_builder.rs). The within-tier order is tier-dependent (EntryScore::ordering_key, block_builder.rs): Finalize/Justify entries already cross 2/3, so newer chain progress leads (target slot, attestation slot, then new-voter count); Build entries only add marginal voters, so coverage leads (new-voter count, target slot, then attestation slot). data_root is the final deterministic tiebreak in both tiers. Each round picks the best candidate against a projected post-state.
• Proposer budget: rounds stop at max_attestations_per_block distinct AttestationData entries (--max-attestations-per-block, default 3), clamped to MAX_ATTESTATIONS_DATA. The consensus cap is MAX_ATTESTATIONS_DATA, the same value leanSpec enforces in its state transition; only the proposer-side budget differs, and it is configurable.
• Collapsing duplicate data: a winning entry may carry several proofs, which must collapse to one proof per AttestationData before the block is valid. By default ethlambda keeps only the best-coverage proof and drops the rest (keep_best_proof_per_data, block_builder.rs), skipping the leanVM merge at the cost of the voters those proofs carried. With --enable-proposer-aggregation, compact_attestations (block_builder.rs) instead merges them through recursive proof aggregation, which is what leanSpec always does.
• leanSpec: build_block scans candidates sorted by (target.slot, data_root), oldest target first, and includes the first ones that pass its filters (greedy, no scoring), re-running the scan as a fixed point when justification/finalization advances. Its proposer budget is MAX_ATTESTATIONS_DATA itself.
• Equivalence: both produce a valid block. ethlambda front-loads the attestations that advance justification and finality, and within those tiers prefers the newest target where leanSpec takes the oldest; combined with the smaller default budget, an older entry can be outranked by newer ones round after round, so which votes reach peers through blocks differs even though every block stays valid. The smaller budget yields smaller blocks and lower build times.
• Upstream status: the tiered strategy is proposed upstream as leanSpec PR #1149 (open at the time of writing), so this deviation may converge; the recursive-merge collapse follows leanSpec #510.