Introduction

ethlambda is a minimalist, fast and modular implementation of a Beacon Chain and Lean Chain client, written in Rust.

This book collects the design notes and operator-facing references for ethlambda. It is split into five parts:

• Design describes the shape of a running node: the architecture of its actors, workers and shared storage, and how they are wired together at startup.
• Consensus explains how the Lean Chain advances: the slot and interval structure that schedules every validator duty, the 3SF-mini justification and finalization rules, and the LMD-GHOST fork choice algorithm. These documents are implementation-agnostic; ethlambda-specific behaviour is called out in blockquotes.
• Beacon Chain covers the Ethereum Beacon Chain side: its state transition, the mainnet wire, and the execution layer pairing.
• Operations documents observable surfaces of a running node: Prometheus metrics, checkpoint sync, and the fork choice visualization served by the API.
• Development collects notes for contributors, starting with the spec deviations where ethlambda departs from leanSpec (Lean Chain) or consensus-specs (Beacon Chain).

For build and contribution instructions, see the README and CONTRIBUTING.md in the repository.

Visual references

Two standalone HTML infographics ship alongside this book and are copied verbatim into the rendered output:

• 3SF-mini infographic
• ethlambda architecture infographic

Community

• Telegram: ethlambda group, where we post daily updates; drop by to ask questions or chat about anything Lean-related.
• X (Twitter): @ethlambda_lean for occasional updates.
• Weekly community call: every Friday, streamed live on @class_lambda; the call link is posted on Telegram beforehand.
• Ecosystem coordination: the PQ Interop calls on ethereum/pm cover cross-client Lean Ethereum work and related updates; the meeting links are posted on each issue.

Related projects

On the Lean Chain, ethlambda is one of several clients under active development. For comparison and cross-client testing:

• zeam (Zig)
• ream (Rust)
• qlean (C++)
• grandine (Rust)
• gean (Go)
• Lantern (C)

Architecture

A running node is two actors, a few helper tasks, and one shared Store. The actors are genservers: each owns its state, and the only way in is a message.

• BlockChainServer (crates/blockchain/src/lib.rs) drives consensus. It runs the slot clock, performs validator duties, imports blocks and attestations, and is the sole writer of consensus state.
• P2PServer (crates/net/p2p/src/lib.rs) owns the network side: gossip publication, request/response, peer bookkeeping and long-range sync.

Everything else hangs off those two: an aggregation worker on a blocking thread, the libp2p swarm loop on its own task, and the axum servers that expose the node to the outside world.

This page describes the Lean Chain node, ethlambda node. ethlambda beacon runs the same two actors over its own Store, but its chain actor (BlockChain::spawn_beacon) has no validator keys, no aggregator role and no duties for the sync gate to hold back: it imports blocks and runs the Beacon Chain’s fork choice. The mainnet wire and Beacon Chain state transition cover that side.

Note: for what a genserver is, read this blogpost on the spawned crate.

                        Tick (self-message, one per interval)
                      ┌──────────────────────────────┐
                      ▼                              │
       ┌─────────────────────────────┐               │      ┌────────────────────┐
       │      BlockChainServer       │───────────────┘      │  aggregation       │
  ┌───▶│  actor, sole state writer   │─────── jobs ────────▶│  worker            │
  │    │                             │◀───── aggregates ────│  (blocking thread) │
  │    └──────────────┬──────────────┘                      └────────────────────┘
  │                   │
  │ P2PToBlockChain   │ BlockChainToP2P
  │ new block/vote    │ publish, fetch block
  │                   ▼
  │    ┌─────────────────────────────┐  SwarmCommand   ┌────────────────────┐
  └────│          P2PServer          │────────────────▶│   swarm adapter    │
       │  actor, gossip + req/resp   │◀────────────────│   (libp2p task)    │◀══▶ peers
       └─────────────────────────────┘   SwarmEvent    └────────────────────┘

       ┌────────────────────────────────────────────────────────────────────┐
       │ Store: pluggable key-value backend + in-memory fork-choice buffers │
       │ written by BlockChainServer, read by P2PServer and the API servers │
       └────────────────────────────────────────────────────────────────────┘

Actor protocols

The two actors never share memory: they talk over the typed protocols in crates/net/api/src/lib.rs.

Both refs start as None. The InitP2P and InitBlockChain messages fill them in right after spawn, so neither actor needs the other to exist at construction time.

The BlockChainServer

The tick loop

The actor schedules its first Tick for genesis time, and every handler re-arms the next one at the following interval boundary. A handler that overruns that boundary re-arms with a zero delay instead, so the interval it just missed still gets its duty. The handler derives (slot, interval) from the wall clock, compares it against the store’s own interval counter, and skips ticks the store already passed.

store::on_tick then walks the store clock forward one interval at a time, fast-forwarding if it fell more than a slot behind, so a late tick still runs each interval’s bookkeeping in order. That walk and the actor split the duties:

See Slots and Intervals for what each duty means at the protocol level, and why the proposer builds one interval early.

The actor also advances its XMSS signing keys on every tick, and catches them up to the current slot once at spawn. The keys are one-time and slot-bound, so a node that skipped this would have nothing left to sign with.

Block import

Blocks take the same path whether they arrived on gossip or came back from a BlocksByRange sync request. The actor verifies the signature, then runs the state transition (crates/blockchain/state_transition): process_slots advances the pre-state through empty slots, process_block validates the header and applies the block’s attestations, and the result is rejected unless the recomputed state root matches the one the proposer committed to. Justification and finalization move as part of that transition, following the 3SF-mini rules. The actor then writes the block and its post-state, and recomputes the head with LMD GHOST (crates/blockchain/fork_choice).

Aggregation off the message loop

XMSS proving costs hundreds of milliseconds, so it cannot run on the actor loop: a blocked actor stops importing blocks. The actor instead snapshots aggregation inputs from the store, ranks candidates by consensus value, and hands a job list to a spawn_blocking worker (crates/blockchain/src/aggregation.rs). The worker holds no store access, streams one AggregateProduced message back per finished job, and ends with AggregationDone. The actor publishes each result on gossip when it arrives.

A soft deadline cancels the worker through a CancellationToken, so an overrunning slot cannot eat the next one. The session can also start up to EARLY_AGGREGATION_WINDOW before interval 2, once two thirds of the expected signatures are in. Either entry point counts as the slot’s one session, so a slot aggregates once. Starting early buys proving time, not an earlier publication: the worker holds each finished aggregate until the interval-2 boundary before delivering it to the actor.

Block import runs a second, smaller aggregation path. reaggregate.rs splits an imported block’s merged proof back into per-attestation aggregates and folds them into the local pool, which is how a node that only saw a vote inside a block gets its fork-choice weight. Aggregators go one step further and republish those aggregates on gossip. Each split runs a fresh SNARK, so reaggregate.rs caps how many it does per block, and the actor skips the whole path while the node is catching up.

Subnet-windowed aggregation

Two aggregators handed the same pool of existing proofs would otherwise pick the same two children every session, since the greedy selection in aggregation.rs is deterministic: all that duplicated leanVM proving buys nothing once one of them publishes. Each aggregator instead scores that pool through a window: a contiguous run of subnets starting at its duty subnet, the first value of --aggregate-subnet-ids (or the lowest subnet it subscribes to, if that flag is unset). A proof outside the window still counts if it partly overlaps, but earns credit only for its in-window share, so aggregators with different windows tend to land on different children without anyone being excluded from merging.

The width is derived, not chosen: wide enough to hold two proofs at the reach of the aggregator’s anchor, capped at the committee count, so it only widens once a data root’s proof has actually climbed. The anchor is the largest-coverage proof in the candidate’s pool that touches the aggregator’s own duty subnet. Picking it by coverage rather than by reach keeps a sparse proof, one validator in each of many subnets, from setting the width for everybody; requiring it to touch the duty subnet means “no anchor” says “no peer has covered my subnet”, which is exactly when this node’s raw signatures are irreplaceable, and the narrowest window then leaves it aggregating those instead of merging other aggregators’ proofs. The price is that two aggregators reading one lopsided pool can derive different widths, so their windows nest rather than tile; that costs a round of climbing, not correctness.

A window can still decline a merge the unwindowed pool would have allowed, when the proof pool is sparse relative to the window’s contiguous span (a strided aggregator placement is the common cause); selection then retries once with the full committee set, so the feature can only improve on the pre-window selection, never regress below it.

--skip-redundant-aggregation trades that safety net away on purpose. With it set, an aggregator sits out any candidate whose derived width it does not own in the current slot (duty_subnet % width == slot % width), and the freed job goes to the next-best attestation data rather than to a narrower merge of the same one. Ownership rotates with the slot, so every duty subnet gets a turn, and the narrowest width is owned by everyone, so a candidate with no anchor on this node’s subnet is never skipped. The full-width fallback is disabled under the flag: every width below the committee count has several owners, so retrying there would rebuild exactly the duplication the flag buys away.

The rotation guarantees an owner at every width only when every subnet below the committee count has an aggregator holding it as its duty subnet, so treat that as a precondition for the flag. On a sparser placement a width can have no owner in a given slot even with every configured node healthy: with duty subnets {0, 2} at committee count 4, nothing owns width 4 in an odd slot, and with the fallback off that merge level is dropped for the slot. Leave the flag unset on a placement that does not cover every subnet.

Sync gate

sync_status.rs tracks how far the local head lags the slot clock. Past the threshold the node reports itself syncing and stops attesting and proposing, since a head derived from a partial view is not worth voting for. A hysteresis band stops the state from flapping at the boundary, and a network-wide stall (nobody else is ahead either) leaves the node synced so its validators can help the chain recover. The same status feeds the lean_node_sync_status metric and, through a shared controller, the /lean/v0/node/syncing endpoint. --disable-duty-sync-gate reduces the gate to observe-only.

Missing parents

The actor cannot import a block whose parent is unknown, so it parks the block in pending_blocks under its parent root and records the deepest missing ancestor it can find, walking back through already-stored pending blocks, in pending_block_parents. That ancestor is what it asks the P2PServer to fetch. Once the ancestor lands, the actor cascades down the parent index and re-imports every block that was waiting.

Chain events

The actor is the sole publisher on an EventBus (crates/blockchain/src/events.rs), which carries seven topics: head moves, imports and gossip sightings of blocks, single votes and aggregates, plus justification and finalization updates. The bus is best-effort: emission never blocks the actor, and a slow subscriber loses events instead of back-pressuring consensus. The API server subscribes one receiver per SSE client.

The P2PServer

Only the swarm adapter task touches the libp2p::Swarm. The actor sends it SwarmCommands (publish, dial, send request, send response) and receives SwarmEvents back as actor messages. That split keeps non-Clone swarm types (response channels, for one) out of the typed protocol, and keeps the adapter polling network I/O while the actor is busy with a message.

What the actor does with those events:

• Gossip. It decodes blocks, aggregates and per-subnet attestations, then forwards them to the BlockChainServer. The node computes its subscriptions once at startup from its validator set and aggregator role, and never revisits them.
• Status. Sent on the first connection to a peer, not on every redundant one. When a peer reports a head ahead of ours, the actor opens a long-range sync range or extends the one it has, then requests BlocksByRange batches one at a time, dropping peers that fall behind the range.
• BlocksByRoot. Backs the fetch_block requests above. Retries use exponential backoff and prefer a peer that has not already failed for that root, falling back to the full connected set once every peer has failed.

The P2PServer holds its own Store clone, which it only ever reads, so it answers Status and BlocksBy* requests without a round trip through the consensus actor.

The Store

The Store follows a similar approach as ethrex’s: a safe, easy to use interface over a pluggable key-value backend (StorageBackend, RocksDB for a normal node and in-memory for tests and the Hive test driver). Cloning one shares the backend, the state LRU cache, and the in-memory buffers that fork choice runs on: the new and known attestation payload buffers, the latest votes, and the gossip signatures awaiting aggregation. The node never persists those buffers, since they only matter for the slot they belong to.

Every component gets a clone, but only the BlockChainServer writes consensus state. A reader can therefore hold a handle without interleaving with a state transition.

See Data Storage for the table layout, the snapshot/diff scheme used for states, and pruning rules.

HTTP API

We use axum as our API router, with requests served in tokio tasks. Handlers read node state without messaging the actors: the Store is router state, and the EventBus and the runtime controllers arrive as extension layers. Metrics and debug endpoints live in their own routers, on a port you configure separately: distinct ports bind two independent servers, equal ports merge all three routers onto a single listener. See HTTP API for the endpoint reference.

Startup

bin/ethlambda/src/main.rs wires the node in a fixed order. Everything before the actors spawn is fail-fast, so a misconfigured node stops at boot instead of hours later.

1. Install the tracing subscriber, parse CLI options, register metrics, raise the file-descriptor limit for RocksDB’s unbounded table cache.
2. Load the node key, then genesis config, validator config, bootnode ENRs and validator keys.
3. Open the database, then pick an anchor: resume from disk if the on-disk head is recent enough, otherwise checkpoint sync from the configured URLs, and otherwise build the genesis state. A stale database with no checkpoint URL configured is resumed anyway, with a warning, since that is the setup the node was given.
4. Build the shared handles: aggregator controller, sync-status controller, event bus, and the subnet set that both the swarm and the actor need to agree on.
5. Spawn BlockChainServer (which schedules its first tick for genesis time), build the swarm, spawn P2PServer, and wire the two together with InitP2P and InitBlockChain.
6. Spawn the API and metrics servers.

Shutdown runs in reverse: the first ctrl+c stops both actors and cancels the servers’ shutdown token, and three more force the process to exit if a graceful stop hangs.

Note: booting with HIVE_LEAN_TEST_DRIVER=1 short-circuits everything from step 2 on and exposes only the Hive test-driver endpoints, so a driver run never touches the node key, the genesis config or any other consensus prerequisite.

Slots and Intervals

A Lean Chain slot is divided in 5 intervals of equal length. Every duty a validator owes the chain is due in one of them. The slot lasts 4 seconds by default, so the offsets below are the 800 ms grid; a network that sets MILLISECONDS_PER_SLOT in its config file scales every offset by the same factor. That key can only stretch the grid: 4 seconds is also the floor, since a few client timings are fixed in milliseconds rather than expressed as a fraction of the slot.

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │   block    │    vote    │    vote    │safe target │    head    │
    │  proposal  │propagation │aggregation │computation │   update   │
    └────────────┴────────────┴────────────┴────────────┴────────────┘
     ◄───────────── gossiped ─────────────▶ ◄───── local only ───────▶

The grid comes from a genesis timestamp every node shares, so the schedule needs no coordination messages: a node reads its clock, works out which interval it is in, and knows which duty is due. The order is a dependency chain, since each interval consumes what the previous one produced. A duty that overruns its interval is not rescheduled: it lands late, and the slot moves on without it.

In ethlambda’s Lean Chain: the intervals are the SlotInterval variants in crates/blockchain/src/lib.rs. INTERVALS_PER_SLOT is fixed in crates/common/types/src/constants.rs (each interval carries a distinct duty, so the count is not a knob), while the slot duration is read from the network’s config file into ChainConfig, which derives the interval length from it. Every node on a network must agree on the value, and other clients still hold it at compile time: setting it only has an effect where every node reads the key.

Interval 0: Block proposal

A block proposer, selected in a round-robin fashion (slot % num_validators), proposes a new block and gossips it to the network. Right before building the block, the proposer merges their “new attestations buffer” into their fork-choice view. They then include attestations that the proposer has recently seen into their block. Other validators verify the block and its contents, and merge the votes it includes into their fork-choice view. After importing a block, all validators recompute their head, and update the latest finalized and justified checkpoints according to the block’s post-state.

A block body carries at most MAX_ATTESTATIONS_DATA aggregated attestations: distinct (slot, head, target, source) tuples, each paired with a bitfield naming the validators bound to it. Genesis occupies slot 0, so proposals start at slot 1, and nothing forces a slot to be filled: a proposer that is offline or too slow leaves an empty slot, and the next block simply points its parent root at an older block.

In ethlambda’s Lean Chain: block proposal is merged into the previous slot’s head-update interval: the proposer advances its store to the next slot, builds the block there, and holds publication until the slot boundary. That buys the build one extra interval of headroom and leaves no actor work at the block-proposal tick itself.

Interval 1: Vote propagation

Validators gossip their votes for the block they consider to be the head of the chain, and append to it a (source, target) finality vote. These votes are in aggregation subnets and are imported by aggregators. Aggregators verify the votes in their subnet and store them for later aggregation.

In ethlambda’s Lean Chain: a validator’s subnet is validator_index % attestation_committee_count, and a node only aggregates for subnets it subscribed to at startup. Aggregation is also gated on the aggregator role, seeded by --is-aggregator and flippable at runtime through the admin API. A chain whose validators all decline the role still gossips votes and logs them as processed, but no aggregate is ever produced, so every block is empty and the chain never justifies.

Interval 2: Vote aggregation

Aggregators aggregate the votes they have received and gossip the resulting aggregated attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a “new attestations buffer”.

Aggregation earns its own interval because collapsing a subnet’s worth of XMSS signatures into one proof is the heaviest recurring computation in the client. It is also what makes a block affordable, since a block carrying raw votes would need one full XMSS signature per voter, quickly going over the network bandwidth limit.

In ethlambda’s Lean Chain: the proofs run on an off-thread worker so the blockchain actor’s message loop stays responsive, and a session may start up to EARLY_AGGREGATION_WINDOW before the interval boundary once two thirds of the signatures are in. At the session’s soft deadline the actor stops handing out new jobs, but a proof already in flight finishes and publishes late rather than being discarded.

Interval 3: Safe target computation

Validators compute the safe target they’ll use when deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot.

It is LMD-GHOST again, but run over just the votes that arrived this slot and with a two-thirds weight threshold, where head selection applies none. The safe target therefore sits at or behind the head and advances only once a branch is backed by a supermajority. Deriving targets from it is what stops 3SF-mini from justifying a branch the network has not visibly converged on.

Interval 4: Head update

Validators merge the aggregated attestations they have in their “new attestations buffer” into their fork-choice view, and recompute their head.

This is the slot’s second and last promotion point; the first is the proposer’s, just before it builds. Until a vote is promoted it carries no weight in head selection, which is what keeps a validator’s fork-choice view from shifting under it mid-slot. The safe target is the exception: it reads the unpromoted buffer directly, which is how it stays a view of this slot alone. See why staged promotion for the reasoning.

3SF-mini: Justification & Finalization

ethlambda’s Lean Chain (ethlambda node) uses 3SF-mini (Three-Stage Finality, minimal version) for justification and finalization. Unlike the Ethereum Beacon Chain’s epoch-based Casper FFG, which ethlambda beacon follows, 3SF-mini operates at the slot level: any slot can be justified, not just epoch boundaries.

Quick Example: Three Slots to Finality

4 validators, slot N already finalized and justified.

                             source  target
                                │       │
                                ▼       ▼
    Slot N        ──[ N-2 ]──[ N-1 ]──[ N ]
                       F        J       H

                                     source    target
                                        │         │
                                        ▼         ▼
    Slot N+1      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]
                       F        F       J         H

                                               source     target
                                                  │          │
                                                  ▼          ▼
    Slot N+2      ──[ N-2 ]──[ N-1 ]──[ N ]────[ N+1 ]────[ N+2 ]
                       F        F       F         J          H

    H = head    J = justified    F = finalized

At each slot, validators vote for the newest block as their target, citing the latest justified checkpoint as their source:

• Slot N+1: Votes source=N, target=N+1. Three of four vote (3×3=9 >= 2×4=8), so N+1 is justified.
• Slot N+2: Votes source=N+1, target=N+2. Three of four vote, so N+2 justified. N+1 and N+2 are consecutive justifiable slots and both are justified, so N+1 is finalized.

In the ideal case, each block carries attestations that justify the parent slot and finalize the one before it. In practice, forks, missed slots, and delayed votes can break this cadence. The rest of this document explains the rules that make this work, and what happens when things go wrong.

Concepts

Justification via Supermajority

A checkpoint becomes justified when at least two-thirds of validators attest to it as a target:

                   JUSTIFICATION
                   ─────────────

    Validators:  V0  V1  V2  V3  V4  V5  V6  V7  V8
                  │   │   │   │   │       │   │
                  └───┴───┴───┴───┴───────┴───┘
                              │
                    7 out of 9 votes
                  (3×7=21 >= 2×9=18) ✓
                              │
                              ▼
                     ┌──────────────┐
                     │ Checkpoint C │
                     │ JUSTIFIED ✓  │
                     └──────────────┘

The threshold is computed as: 3 × vote_count >= 2 × validator_count

In ethlambda’s Lean Chain: Justification and finalization are processed inside process_attestations() in crates/blockchain/state_transition/src/lib.rs, called from process_block(). The supermajority check is 3 * vote_count >= 2 * validator_count.

Attestations must also pass validity checks before they count:

• Source checkpoint must already be justified
• Target must not already be justified
• Neither source nor target may have a zero-hash root
• Source slot < Target slot (time flows forward)
• Both checkpoints must reference known blocks
• Target slot must be justifiable per the 3SF-mini schedule (see below)

The Justifiability Schedule

Not every slot can be justified, only slots at specific distances from the last finalized slot. This is the novel part of 3SF-mini.

A slot is justifiable if delta = slot - finalized_slot matches any rule:

In ethlambda’s Lean Chain: The function slot_is_justifiable_after(slot, finalized_slot) in crates/blockchain/state_transition/src/lib.rs implements this check. It uses isqrt() for perfect square detection and the identity 4n(n+1) + 1 = (2n+1)² for pronic number detection.

    ┌───────────────────────────────────────────────────────┐
    │             JUSTIFIABILITY RULES                      │
    │                                                       │
    │  Rule 1:  delta ≤ 5          (always justifiable)     │
    │                                                       │
    │  Rule 2:  delta = n²         (perfect squares)        │
    │           1, 4, 9, 16, 25, 36, 49, 64, 81, 100, ...   │
    │                                                       │
    │  Rule 3:  delta = n(n+1)     (pronic numbers)         │
    │           2, 6, 12, 20, 30, 42, 56, 72, 90, 110, ...  │
    │                                                       │
    └───────────────────────────────────────────────────────┘

Visualizing the first 40 slots after finalization (✓ = justifiable):

    delta: 0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20
           ✓  ✓  ✓  ✓  ✓  ✓  ✓  ·  ·  ✓  ·  ·  ✓  ·  ·  ·  ✓  ·  ·  ·  ✓
           ╰─ delta ≤ 5 ──╯  2×3      3²       3×4         4²          4×5

    delta: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
           ·  ·  ·  ·  ✓  ·  ·  ·  ·  ✓  ·  ·  ·  ·  ·  ✓  ·  ·  ·  ·
                       5²             5×6               6²

Key property: Gaps between justifiable slots grow, but never become infinite. As more time passes since finalization, the network gets progressively wider windows to accumulate votes. This creates a natural backpressure: if the network is struggling to reach a two-thirds majority (e.g., due to partitions or validator dropouts), the increasing gaps give more time for the supermajority to form.

Finalization

A justified checkpoint becomes finalized when it is the source of a justification whose target is the next justifiable slot. In other words, there must be no justifiable slots between source and target: the two must be consecutive entries in the justifiability schedule.

In ethlambda’s Lean Chain: The try_finalize() function iterates over slots between source and target and calls slot_is_justifiable_after on each. If any slot is justifiable, finalization fails (source and target aren’t consecutive). The check uses original_finalized_slot (the finalized slot at the start of block processing), not the current one, since finalization can advance mid-processing.

                    FINALIZATION CHECK
                    ──────────────────

    Example 1: Finalization FAILS

    Finalized=10   Source=13 (justified)   Target=16 (justified)

    [ 10 ] · · · [ 13 ]  14   15  [ 16 ]
                          ▲    ▲
                          │    └── delta=5 ≤ 5 → justifiable!
                          └────── delta=4 ≤ 5 → justifiable!

    Justifiable slots exist between S and T → NOT FINALIZED ✗
    (13 and 16 are not consecutive justifiable slots)


    Example 2: Finalization SUCCEEDS

    Finalized=10   Source=16 (justified)   Target=19 (justified)

    [ 10 ] · · · [ 16 ]  17   18  [ 19 ]
                          ▲    ▲
                          │    └── delta=8 → not justifiable ✓
                          └────── delta=7 → not justifiable ✓

    No justifiable slots between S and T → S is FINALIZED ✓
    (16 and 19 are consecutive: delta=6=2×3, then delta=9=3²)

The reasoning: if a justifiable slot exists between source and target, validators could have directed their votes to that intermediate slot instead, potentially on a different fork. By requiring source and target to be consecutive justifiable slots, the protocol ensures that no alternative justification path can exist between them.

Justifiable Slot Backoff

The justifiability schedule acts as a backoff mechanism to increase finalization rate during periods of asynchrony. By “diluting” the possible targets of a justification vote (via the slot_is_justifiable_after function), the protocol increases the window during which votes for a given slot can be included, improving the chances of achieving the required two-thirds majority.

Since finalization requires two consecutively justifiable slots to both be justified, this backoff isn’t immediately reset after finalization occurs; it only lowers over time when synchrony is restored.

Example: Extended asynchrony with gradual recovery.

    F=0. Justifiable slots grow sparser as delta increases:

    delta ≤ 5:     0   1   2   3   4   5                  (gap = 1)
    delta 6–20:    6       9       12          16          20   (gap = 3–4)
    delta 20–36:   20          25          30              36   (gap = 5–6)
    ...
    delta ~1000:   900     930     961     992      1024        (gap = 30–32)
                   30²    30×31   31²    31×32      32²

Phase 1: Long asynchrony, slow progress.

    Validators vote, but with many justifiable targets, votes scatter
    and no single slot reaches >=2/3. As gaps widen, votes concentrate.

    Near slot 1000, the 32-slot gap between 992 and 1024 means
    no competing justifiable target exists for 32 slots after 992.
    All votes funnel toward 1024 once it is built.

Phase 2: Slot 992 finalized.

    Slot 992 justified (source = earlier justified slot).
    Slot 1024 justified (source = 992).

    slot:  0  ...  992              1024
           F        J    ·······     J
                    ▲                ▲
                 source ──────────▶ target

    Slots 993–1023: any justifiable from F=0?
      Perfect squares? 31²=961 (before), 32²=1024 (boundary). None.
      Pronic? 31×32=992 (boundary), 32×33=1056 (after). None.
    No justifiable slots between them → slot 992 FINALIZED ✓

Phase 3: Partial reset. Backoff shrinks but doesn’t vanish.

    New F=992. Justifiable slots shift:

    slot: 992 993 994 995 996 997 998 ··· 1001 ··· 1004 ··· 1008 ··· 1022 ··· 1028
           F   ✓   ✓   ✓   ✓   ✓   ✓      ✓       ✓       ✓       ✓       ✓
              ╰── delta ≤ 5 ──╯  2×3     3²      3×4     4²      5×6     6²

    Dense slots 993–998 are already in the past!
    Near the current slot (~1024), justifiable slots are ~6 apart:

    ...  1022     1028     1034     1041  ...
         δ=30     δ=36     δ=42     δ=49
         5×6      6²       6×7      7²
         └──6──┘  └──6──┘  └──7──┘

    Gaps shrank from 32 → 6, but didn't reset to 1.

Phase 4: Further finalization closes the gap.

    Justify 1022 and 1028, finalize 1022. New F=1022.

    From F=1022, at slot ~1028 (delta = 6):

    slot:  1022  1023  1024  1025  1026  1027  1028
            F     ✓     ✓     ✓     ✓     ✓     ✓
                  ╰────── delta ≤ 5 ──────╯    2×3

    Gaps are back to 1. Fast finalization resumes.

    Summary of gradual recovery:

    ┌───────────────────┬──────┬───────┬───────┬──────────────┐
    │ Finalization step │  F   │ Head  │ Delta │ Nearby gaps  │
    ├───────────────────┼──────┼───────┼───────┼──────────────┤
    │ Before any        │    0 │ ~1000 │ ~1000 │ 31–32        │
    │ After 1st (992)   │  992 │ ~1024 │   ~32 │ 6–7          │
    │ After 2nd (1022)  │ 1022 │ ~1028 │    ~6 │ 1            │
    └───────────────────┴──────┴───────┴───────┴──────────────┘

    Each finalization step reduces the delta between the finalized
    slot and the chain head, progressively tightening the gaps.

When finalization advances, the following cleanup occurs:

• justified_slots window shifts forward (old slots pruned)
• LiveChain entries for finalized slots are pruned
• Gossip signatures and aggregation proofs for finalized blocks are cleaned up
• Future fork choice runs start from the finalized slot’s successor

In ethlambda’s Lean Chain: The justified_slots bitlist uses relative indexing (index 0 = finalized_slot + 1). When finalization advances, shift_window() in crates/blockchain/state_transition/src/justified_slots_ops.rs drops the now-finalized prefix. The attestation target is also walked back to the nearest justifiable slot via slot_is_justifiable_after in crates/blockchain/src/store.rs.

End-to-End: From Head Selection to Finalization

This section connects LMD-GHOST fork choice with 3SF-mini. The quick example above showed the happy path; here we focus on what happens when things go wrong.

Recap: Attestation Anatomy

Each attestation carries three checkpoints, each determined by a different mechanism:

    ┌────────────────────────────────────────────────────────────────┐
    │                       ATTESTATION                              │
    │                                                                │
    │  head    Newest block the validator sees                       │
    │          ← LMD-GHOST with min_score = 0                        │
    │                                                                │
    │  target  Block the validator wants justified next              │
    │          ← Derived from safe target, walked back to nearest    │
    │            justifiable slot (feeds into 3SF-mini)              │
    │                                                                │
    │  source  Latest justified checkpoint                           │
    │          ← Read from store state                               │
    └────────────────────────────────────────────────────────────────┘

The safe target is computed by running LMD-GHOST with a two-thirds vote threshold. Only blocks backed by a supermajority qualify, so the safe target is always at or behind the head. The attestation target is derived by walking back from the head toward the safe target (max 3 steps), then to the nearest justifiable slot. See Safe Target Selection for details.

In ethlambda’s Lean Chain: get_attestation_target() in crates/blockchain/src/store.rs implements this walk-back. JUSTIFICATION_LOOKBACK_SLOTS = 3 provides a liveness guarantee: even if the safe target is stuck, the target eventually advances once the head moves far enough ahead.

Lagging Safe Target (Fork with Delayed Convergence)

When validators disagree about the head, the safe target lags behind: no single branch has two-thirds support. This delays justification until the fork resolves.

    Setup: 9 validators, finalized=100, justified=101
    Safe target threshold: >=6 votes (2/3 of 9)

Slots 102–103: Fork splits votes. No progress.

                         ┌──[ B102a ]──[ B103a ]     V0–V4 (5)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]     V5–V8 (4)

Neither branch clears two-thirds → safe target stuck at B101. Walk-back from head always lands on source (B101). No attestation can advance justification.

Slot 104: V7 and V8 switch sides. Fork resolves.

V7 and V8 receive B102a (delayed by the partition) and switch to the a-branch.

                         ┌──[ B102a ]──[ B103a ]──[ B104a ]     V0–V4, V7, V8 (7)
    [ F=100 ]──[ J=101 ]─┤
                         └──[ B102b ]──[ B103b ]──[ B104b ]     V5–V6  (2)

B102a subtree now has 7 votes >= 6 → safe target = B102a. Walk-back from B104a lands on B102a (2 steps). Slot 102 is justifiable (delta=2 ≤ 5).

    source=101 ──▶ target=102    7/9 votes → 3×7=21 >= 2×9=18 → JUSTIFIED ✓
    Finalization: no slots between 101 and 102 → 101 FINALIZED ✓

After slot 104: finalized=101, justified=102.

Slots 105–106: Full convergence and recovery.

All 9 validators on the a-branch. Slot 105: target=B104a → B104a JUSTIFIED. But finalization fails: slot 103 (between source=102 and target=104) is justifiable but was never justified (lost in the fork).

Slot 106: target=B105a → B105a JUSTIFIED. No justifiable slots between 104 and 105 → 104 FINALIZED. Finalization jumped from 101 to 104, skipping 102 and 103.

    FORK WITH DELAYED CONVERGENCE
    ═════════════════════════════

    Slot:     100   101   102   103   104   105   106
    Status:    F     J     ·     ·     ·     ·     ·
                                fork ──────┤
                                          resolves
    Head:      ·    B101  B102a B103a B104a B105a B106a
    Safe:      ·    B100  B101  B101  B102a B104a B105a
                          stuck ─────┘  ▲
                                        │
                           V7+V8 switch, safe target unsticks

    Justified:  ·    101   ─     ─    102   104   105
    Finalized:  ·     ·    ─     ─    101   ─     104
                                                   ▲
                              finalization jumps ──┘
                               (102,103 skipped; 103 was never justified)

Comparison with Casper FFG

Both 3SF-mini and Casper FFG are finality gadgets built on the same foundation: supermajority links between checkpoints. They differ fundamentally in their unit of time and what that implies for validator participation. For a thorough treatment of Casper FFG as used in Ethereum, see the eth2book chapter on Casper FFG.

Slots vs Epochs: The Core Architectural Split

3SF-mini: Every Validator, Every Slot

In 3SF-mini, all validators vote in every slot. A checkpoint can be justified at any slot (subject to the justifiability schedule), and finalization can happen as soon as two consecutive justifiable slots are both justified.

    3SF-mini  (4-second slots, 4 validators)

    Slot 100        Slot 101        Slot 102        Slot 103
    ┌───────┐       ┌───────┐       ┌───────┐       ┌───────┐
    │V0 V1  │       │V0 V1  │       │V0 V1  │       │V0 V1  │
    │V2 V3  │       │V2 V3  │       │V2 V3  │       │V2 V3  │
    └───┬───┘       └───┬───┘       └───┬───┘       └───┬───┘
        │               │               │               │
     4 votes         4 votes         4 votes         4 votes
     per slot        per slot        per slot        per slot

    Every validator participates in every slot.
    >=2/3 threshold checked per-slot → can justify any slot.

This is simple and fast, but it means every validator must produce and verify a vote every slot. The total message load scales as validators × slots.

Casper FFG: Validators Split Across an Epoch

Ethereum’s beacon chain has ~1,000,000 active validators. Having all of them vote every 12-second slot would be unmanageable. Instead, Casper FFG groups 32 slots into an epoch, and splits the validator set across the slots within it:

    Casper FFG  (12-second slots, 32 per epoch, ~900k validators)

    Epoch N
    ┌─────────────────────────────────────────────────────────────┐
    │ Slot 0     Slot 1     Slot 2    ...    Slot 30    Slot 31   │
    │ ┌──────┐   ┌──────┐   ┌──────┐        ┌──────┐   ┌──────┐   │
    │ │~28125│   │~28125│   │~28125│  ...   │~28125│   │~28125│   │
    │ │valids│   │valids│   │valids│        │valids│   │valids│   │
    │ └──┬───┘   └──┬───┘   └──┬───┘        └──┬───┘   └──┬───┘   │
    │    │           │           │               │           │    │
    └────┼───────────┼───────────┼───────────────┼───────────┼────┘
         └───────────┴───────────┴───┬───────────┴───────────┘
                                     │
                            All ~900k votes
                          collected over 32 slots
                                     │
                                     ▼
                              Epoch checkpoint
                              (first slot of epoch)

    Each validator attests exactly ONCE per epoch.
    The full >=2/3 tally is only meaningful at epoch boundaries.

Each validator is shuffled into a committee assigned to one specific slot. Within that slot, the committee may be further split (up to 64 sub-committees) for parallel aggregation. The result: each validator only attests once per epoch, and the network processes ~28,000 attestations per slot instead of ~900,000.

The trade-off:

Epochs exist because of a scalability constraint, not a protocol-theory preference. If you could process a million votes per slot, Casper FFG wouldn’t need epochs at all. 3SF-mini sidesteps this by targeting a smaller validator set, which lets it operate at slot granularity.

Finalization Logic

Both require a chain of justified checkpoints, but the rules differ in what they check.

Casper FFG uses k-finality. The original rule (k=1) requires a direct supermajority link from a checkpoint to its immediate successor: justify epoch N+1 with source=N, and N is finalized. Ethereum generalizes this to k=2, which handles the case where the network falls slightly behind:

    Casper FFG — 1-finality (ideal case):

    Epoch N       Epoch N+1
    ┌─────┐       ┌─────┐
    │ CP  │══════▶│ CP  │       Supermajority link N → N+1
    │ J ✓ │       │     │
    └─────┘       └─────┘

    Processing this link:
      1. Epoch N+1 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (direct successor justified)


    Casper FFG — 2-finality (one epoch behind):

    Epoch N       Epoch N+1     Epoch N+2
    ┌─────┐       ┌─────┐       ┌─────┐
    │ CP  │       │ CP  │       │ CP  │
    │ J ✓ │       │ J ✓ │       │     │
    └─────┘       └─────┘       └─────┘
       │                           │
       └══════ supermajority ══════┘
               link N → N+2

    The direct link N→N+1 didn't form in time.
    Instead, a link forms from N→N+2. Processing this link:
      1. Epoch N+2 becomes JUSTIFIED (target of a supermajority link)
      2. Epoch N becomes FINALIZED (all intermediates are justified)

The 2-finality rule is a recovery mechanism: even if the network missed the ideal one-epoch finalization window, it gets a second chance. Ethereum tracks the justification status of the last 4 epoch boundaries to detect both cases. In practice, most finalization happens via 1-finality during normal operation; 2-finality kicks in during brief network hiccups.

3SF-mini takes a different approach entirely:

    Slot S        Slot T
    ┌─────┐       ┌─────┐
    │ CP  │──────▶│ CP  │       No justifiable slots exist
    │ J ✓ │       │ J ✓ │       between S and T
    └─────┘       └─────┘
    ∴ Slot S is FINALIZED

    Rule: Finalized when NO intermediate checkpoints could exist

Instead of checking that intermediate checkpoints are justified, 3SF-mini checks that no intermediate checkpoints could exist at all. This is a stronger guarantee: validators’ votes between source and target could only have gone to the target, since there’s nowhere else to direct them. This structural property is also why 3SF-mini doesn’t need Casper’s surround-vote slashing condition.

Casper’s k-finality is essentially a tolerance parameter: “how many epochs behind can we be and still finalize?” Ethereum chose k=2, meaning it tolerates one missed epoch. 3SF-mini doesn’t need this concept because the justifiability schedule itself adapts. Instead of tolerating missed windows, it makes the windows wider when the network is struggling.

Adaptive Backoff (unique to 3SF-mini)

Casper FFG has a fixed checkpoint every epoch, regardless of network conditions. 3SF-mini’s justifiability schedule adapts: gaps between justifiable slots grow under prolonged asynchrony (via the perfect square and pronic number rules), creating natural vote concentration when the network is struggling to reach a two-thirds majority. Casper FFG has no equivalent; its epoch spacing is the same whether the network is healthy or partitioned. See Justifiable Slot Backoff for a detailed walkthrough.

👻 LMD-GHOST fork choice algorithm

A deep dive into how the LMD-GHOST (Latest Message Driven, Greedy Heaviest Observed SubTree) fork choice algorithm works. LMD-GHOST is the fork choice rule used by Ethereum’s consensus layer and its derivatives. Each validator’s latest attestation is their single active vote, and the algorithm follows the heaviest branch at every fork.

This document is implementation-agnostic, with details of ethlambda’s Lean Chain (ethlambda node) called out in blockquotes marked “In ethlambda’s Lean Chain”. ethlambda’s Beacon Chain follower (ethlambda beacon) runs the consensus-specs fork choice instead; see its section of the Beacon Chain state transition.

Much of the conceptual framing in this document is inspired by Ben Edgington’s Eth2 Book, particularly the LMD GHOST chapter. Highly recommended reading for anyone interested in Ethereum consensus.

Background & History

The GHOST protocol was introduced by Sompolinsky and Zohar in a 2013 paper. Its core idea: instead of choosing the heaviest chain, we choose the heaviest subtree, counting orphaned blocks as evidence of support for their ancestors.

The “LMD” in LMD-GHOST stands for Latest Message Driven: only each validator’s most recent attestation counts, preventing vote amplification. LMD-GHOST is the fork choice rule used by the Ethereum Beacon Chain and Lean Ethereum.

Why Fork Choice?

In a distributed system where validators propose blocks concurrently, the blockchain can fork: two valid blocks may appear at the same slot, creating competing chains. The fork choice rule answers a critical question:

Which chain tip should I follow?

                   ┌──────────┐
             ┌────▶│ Block C  │  ← Chain tip 1
             │     │ slot 5   │
┌──────────┐ │     └──────────┘
│ Block A  │─┤
│ slot 3   │ │     ┌──────────┐
└──────────┘ └────▶│ Block D  │  ← Chain tip 2
                   │ slot 5   │
                   └──────────┘

                    Which tip should validators follow?

Every node in the network must be able to independently arrive at the same answer using only its local view of blocks and attestations. The fork choice rule is what makes this possible. It is a deterministic function from a node’s observed state to a single chain tip.

From Heaviest Chain to Heaviest Subtree

The simplest fork choice rule is heaviest chain: follow the chain tip with the most accumulated weight. This works when fork rates are low, but breaks down when honest validators fork within a common branch:

              HEAVIEST CHAIN vs HEAVIEST SUBTREE
              ──────────────────────────────────

    An attacker with 40% of stake forks at A.
    The honest majority (60%) builds on B but forks into C and D:

                    ┌───B──┬──C     V0, V1, V2 vote for C (30%)
              A ────┤      └──D     V3, V4, V5 vote for D (30%)
                    │
                    └───X──Y──Z     V6, V7, V8, V9 vote for Z (40%)

    Heaviest chain:
      Z has 40% of votes, C and D each have 30%.
      Attacker wins! ✗

    Heaviest subtree (LMD-GHOST):
      At A: B subtree has 60% (C + D), X subtree has 40%.
      Pick B. Then at B: C has 30%, D has 30% (tiebreaker).
      Honest majority wins. ✓

LMD-GHOST is strictly better when honest validators fork within a common subtree. Instead of requiring all honest validators to agree on a single chain tip (which is impossible under network delay), it aggregates their support at each level of the tree.

How Subtree Weight Works (the “GHOST” Part)

The key insight behind the “Heaviest Observed SubTree” part of LMD-GHOST: a vote for a block is implicitly a vote for all its ancestors.

When a validator attests to block F as their head, they are also expressing support for every block on the path from the root to F:

    Validator attests: head = F

    A ── B ── C ── D ── E ── F
    ▲    ▲    ▲    ▲    ▲    ▲
    │    │    │    │    │    │
    └────┴────┴────┴────┴────┘
    All ancestors implicitly supported

This is why LMD-GHOST counts the subtree weight: a block’s weight includes every attestation for any of its descendants, because those attestations implicitly endorse the ancestor too. The algorithm exploits this by walking backward from each attested head and incrementing every block along the path.

LMD: Why Only the Latest Message?

The “LMD” in LMD-GHOST stands for Latest Message Driven. Each validator’s most recent attestation is their only vote. All previous attestations are discarded.

    Validator 7's attestation history:

    Slot 10: attests to head = B     ← discarded
    Slot 11: attests to head = C     ← discarded
    Slot 12: attests to head = E     ← THIS is the active vote

    Only the slot 12 attestation counts for fork choice.

Why only the latest? Two reasons:

1. Prevents double-voting. If all messages counted, a validator could cast many attestations and amplify their influence. With LMD, each validator gets exactly one active vote regardless of how many attestations they’ve broadcast.

2. Reflects current knowledge. A validator’s latest attestation reflects their most recent view of the chain. Older attestations may reference blocks that are no longer on the best chain. Keeping only the latest ensures fork choice uses the most up-to-date information.

The fork choice store maintains a mapping of validator_index → latest attestation. When a new attestation arrives from a validator, it replaces their previous entry:

    Fork choice store (latest messages):

    ┌──────────────┬──────────────────────────────┐
    │ Validator    │ Latest Attestation           │
    ├──────────────┼──────────────────────────────┤
    │ 0            │ head=E, target=C, source=A   │
    │ 1            │ head=D, target=C, source=A   │
    │ 2            │ head=E, target=C, source=A   │
    │ 3            │ head=F, target=D, source=A   │
    │ ...          │ ...                          │
    └──────────────┴──────────────────────────────┘

    One row per validator. New attestation → overwrite row.

LMD-GHOST Step by Step

The algorithm takes a set of inputs and produces a single block root: the head of the chain.

Inputs

In ethlambda’s Lean Chain: The function is compute_lmd_ghost_head() in crates/blockchain/fork_choice/src/lib.rs. The block tree comes from the LiveChain storage index, and min_score is 0 for head selection or ⌈2V/3⌉ for safe target computation.

The Algorithm

First, accumulate weights. Each attestation “paints” the path from its head back to the start root. In the simplest form (equal-weight validators), this adds +1 to every block on the path. In systems with balance-weighted voting, the validator’s effective balance is added instead.

    Validator 0 attests to head = F

      J ─ A ─ B ─ C ─ D ─ E ─ F       (J = justified root)
          +1  +1  +1  +1  +1  +1       J is at start_slot, not counted

    Validator 1 attests to head = D

      J ─ A ─ B ─ C ─ D
          +1  +1  +1  +1

    Accumulated weights:

      Block:    J    A    B    C    D    E    F
      Weight:   ─    2    2    2    2    1    1
                │
                └ start_root (not weighted, used as the descent origin)

In ethlambda’s Lean Chain: All validators have equal weight (+1 per vote). The Ethereum Beacon Chain instead weights votes by effective balance (up to 2048 ETH).

Then, greedily descend. Starting from the start root, at each node pick the child with the most weight. Repeat until reaching a leaf:

    J ──┬── B (5)   ← pick B (higher weight)
        └── G (2)

    B ──┬── C (3)   ← pick C (higher weight)
        └── H (2)

    C ──── D (3)    ← only child, continue

    D ── (no children) → HEAD = D!

Children below min_score are ignored during the descent. With min_score = 0 (normal head selection) all children are visible. With a higher threshold, only branches with strong support are followed. This is used for safe target selection.

The Tiebreaker

When two children have exactly equal weight, a deterministic tiebreaker is needed. Without one, different nodes could pick different heads from the same data, breaking consensus. The tiebreaker is lexicographically higher block root hash, i.e., higher hash value wins.

    Equal weight scenario:

        Parent
        │
    ┌───┴───┐
    B (3)   C (3)         ← Equal weight!
    root:   root:
    0x3a..  0x7f..        ← 0x7f > 0x3a, so pick C

The choice of “higher hash wins” is a convention. Any deterministic rule would work; what matters is that all nodes apply the same one.

Worked Example: Head Selection

Consider a network with 5 validators (indices 0–4) and the following block tree rooted at the justified checkpoint J at slot 10:

                          BLOCK TREE
                          ──────────

Slot 10     ┌──────┐
(justified) │  J   │ ← Justified checkpoint (start_root)
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │
         └──────┘    └──────┘

Latest attestations (one per validator):

Accumulate weights by walking backward from each attested head, adding +1 per block (stopping at J’s slot):

    V0 (head=D):  D+1  B+1  A+1
    V1 (head=D):  D+1  B+1  A+1
    V2 (head=E):  E+1  C+1  A+1
    V3 (head=E):  E+1  C+1  A+1
    V4 (head=E):  E+1  C+1  A+1

Greedily descend from J, always picking the heaviest child:

    Start at J
      └─▶ A (only child, weight 5)
           ├── B (weight 2)
           └── C (weight 3)  ← Pick C (3 > 2)
                └─▶ E (only child, weight 3)
                     └─▶ No children → HEAD = E ✓

Result: The canonical head is Block E. Even though both branches have the same depth, the C→E branch has 3 votes vs B→D’s 2 votes.

                          RESOLVED HEAD
                          ─────────────

Slot 10     ┌──────┐
            │  J   │
            └──┬───┘
               │
Slot 11     ┌──┴───┐
            │  A   │ ✓ canonical
            └──┬───┘
            ┌──┴────────┐
            │           │
Slot 12  ┌──┴───┐    ┌──┴───┐
         │  B   │    │  C   │ ✓ canonical (weight 3 > 2)
         └──┬───┘    └──┬───┘
            │           │
Slot 13  ┌──┴───┐    ┌──┴───┐
         │  D   │    │  E   │ ★ HEAD
         └──────┘    └──────┘

What If a Vote Changes?

Suppose validator 1 now sees block E and switches their attestation from D to E:

    Before:  V0=D, V1=D, V2=E, V3=E, V4=E   → Head = E (3 vs 2)
    After:   V0=D, V1=E, V2=E, V3=E, V4=E   → Head = E (4 vs 1)

    The head didn't change, but the margin increased from 1 to 3.
    If instead V2 and V3 had switched to D:

    After:   V0=D, V1=D, V2=D, V3=D, V4=E   → Head = D (4 vs 1)

    The head reorgs from E to D.

Fork Choice vs Finality

An important conceptual distinction: LMD-GHOST provides fork choice, not finality.

LMD-GHOST gives the network a way to agree on the current head of the chain at any moment, but the head can change. A block selected by fork choice today could be reorged away tomorrow if attestations shift. LMD-GHOST alone provides no guarantee that any block is permanent.

Finality, the guarantee that a block can never be reverted, comes from a separate mechanism called a finality gadget. LMD-GHOST is designed to compose with any finality gadget (e.g., Casper FFG in the Ethereum Beacon Chain, or 3SF-mini in Lean Ethereum).

    ┌────────────────────────────────────────────────────┐
    │                 CONSENSUS = TWO LAYERS             │
    │                                                    │
    │  ┌─────────────┐        ┌──────────────────────┐   │
    │  │  LMD-GHOST  │        │  Finality Gadget     │   │
    │  │             │        │                      │   │
    │  │ "Which tip  │        │ "Which blocks are    │   │
    │  │  is best    │        │  permanent and can   │   │
    │  │  right now?"│        │  never be reverted?" │   │
    │  │             │        │                      │   │
    │  │ Dynamic,    │        │ Monotonic, only      │   │
    │  │ can reorg   │        │ moves forward        │   │
    │  └──────┬──────┘        └──────────┬───────────┘   │
    │         │                          │               │
    │         └──────────┬───────────────┘               │
    │                    ▼                               │
    │         ┌──────────────────┐                       │
    │         │  Full Consensus  │                       │
    │         └──────────────────┘                       │
    └────────────────────────────────────────────────────┘

In ethlambda’s Lean Chain: The finality gadget is 3SF-mini, which operates at the slot level rather than epoch boundaries.

The two layers interact: LMD-GHOST runs its greedy descent starting from the latest justified checkpoint (not genesis). This means finality constrains fork choice: once a checkpoint is finalized, no fork choice run will ever consider blocks before it.

    ┌─────────┐         ┌─────────┐         ┌──── ...
    │FINALIZED│────────▶│JUSTIFIED│────────▶│  fork choice
    │ slot 50 │         │ slot 55 │         │  runs here
    └─────────┘         └─────────┘         └──── ...
         │                   │
         │                   └── start_root for LMD-GHOST
         │
         └── everything before this is permanent

This has a major practical benefit: finality allows aggressive pruning of the block tree. Without finality, fork choice would need to consider every block since genesis, and the tree would grow without bound. With finality, all blocks at or before the finalized checkpoint can be discarded from the fork choice’s working set.

In ethlambda’s Lean Chain: The LiveChain index (the in-memory block tree used by fork choice) is pruned every time finalization advances, keeping it bounded to only the non-finalized portion of the chain.

Attestation Pipeline

In a naive implementation, every attestation would influence fork choice the instant it arrives. This creates problems: validators with faster network connections see different heads than slower ones, and the proposer’s view of the chain could shift mid-block-construction.

Lean Ethereum solves this with a staged promotion pipeline: attestations are collected into a pending set and only promoted to the active fork choice set at designated moments. This ensures all validators operate on a consistent view.

                       ATTESTATION LIFECYCLE
                       ─────────────────────

  ┌──────────────┐       ┌──────────────────┐       ┌──────────────────┐
  │   Network    │       │    Pending       │       │    Active        │
  │  (gossip)    │──────▶│  Attestations    │──────▶│  Attestations    │
  │              │       │                  │       │                  │
  └──────────────┘       └──────────────────┘       └──────────────────┘
                                 │                          │
                          NOT used for               Used for fork choice
                          fork choice                weight calculations
                                 │                          │
                          Promoted at ─────────────▶ designated intervals
                          fixed points

In ethlambda’s Lean Chain: The two stages are called “new” and “known” attestations, held in the in-memory new_payloads and known_payloads buffers of the Store respectively. Promotion happens at tick intervals 0 (if proposing) and 4 (end of slot).

Why Staged Promotion?

The staged design serves two purposes:

1. Consistency: All validators promote attestations at the same moments, reducing divergence in head selection. Without batching, validators with faster network connections would see different heads than slower ones.

2. Proposer fairness: The proposer computes the block against a known, fixed set of attestations. If new attestations could influence fork choice mid-computation, different validators might disagree on the head.

On-Chain vs Off-Chain Attestations

Attestations arrive from two sources, and how they enter the pipeline matters:

The proposer’s own attestation enters as pending (not active) deliberately. If it were immediately active, the proposer would gain an unfair weight advantage for their own block, a circular dependency where proposing a block gives you an extra vote toward making that block canonical.

Safe Target Selection

The safe target is a conservative head computed with a high weight threshold. It constrains the target field in attestations, which feeds into 3SF-mini for justification and finalization decisions. Validators still vote for the newest head they see (regular LMD-GHOST with min_score = 0) in the head field. The safe target only affects which blocks can progress toward finality. It is computed by running the same LMD-GHOST algorithm but with a non-zero min_score in the filtering phase.

                    SAFE TARGET vs HEAD
                    ────────────────────

    Regular head (min_score = 0):
    Follow heaviest branch, even with a slim margin

             ┌── B (3 votes) ← HEAD (3 > 2)
    J ── A ──┤
             └── C (2 votes)


    Safe target (min_score = ⌈2V/3⌉):
    Only follow branches with supermajority support

    V = 5 validators, threshold = ⌈10/3⌉ = 4

             ┌── B (3 votes) ← Below threshold (3 < 4), pruned
    J ── A ──┤
             └── C (2 votes) ← Below threshold (2 < 4), pruned

    Safe target = A (no children pass threshold)

This means the safe target lags behind the head. It only advances when a branch accumulates overwhelming support, making it resistant to temporary fluctuations:

    Timeline of safe target vs head:

    Slot:    10    11    12    13    14    15    16
    Head:    J     A     B     D     D     E     F
    Safe:    J     J     J     A     A     A     D
                                                  │
                        Safe target is always ────┘
                        at or behind the head

The safe target prevents 3SF-mini from finalizing unstable branches: without it, a slim-majority fork could reach justification and finalization before the network converges. By requiring supermajority support for the target, only branches with strong consensus can progress toward finality, even though validators’ head votes freely follow the newest chain tip.

Reorgs

A reorg (reorganization) occurs when the fork choice head switches from one branch to another. This happens when a competing branch accumulates more attestation weight than the current head’s branch.

                    REORG SCENARIO
                    ──────────────

    Before (head = D):

              ┌── B ── D   ★ HEAD (weight 4)
    J ── A ──┤
              └── C ── E      (weight 3)


    New attestations arrive, 3 validators switch to E:

              ┌── B ── D      (weight 2)
    J ── A ──┤
              └── C ── E   ★ HEAD (weight 5)    ← REORG!


    The canonical chain changed from  J─A─B─D  to  J─A─C─E
    Blocks B and D are no longer canonical (but remain in the block tree).

Reorgs are normal during transient network conditions but should be rare in stable operation. They cannot cross a finalization boundary: once a block is finalized, it is permanently part of the canonical chain.

In ethlambda’s Lean Chain: Reorgs are detected by checking whether the old and new heads share a common prefix, and tracked via Prometheus metrics (lean_fork_choice_reorgs_total).

LMD-GHOST Variants

LMD-GHOST is one of several variants that have been proposed and studied. Understanding the design space helps explain why LMD was chosen.

The Ethereum consensus mini-spec originally used IMD-GHOST but switched to LMD in November 2018 due to superior stability properties.

    IMD: All attestations count         LMD: Only latest counts

    V0: slot 5 → head B                V0: slot 5 → head B  (overwritten)
    V0: slot 8 → head C                V0: slot 8 → head C  ← active
    V0: slot 11 → head E               V0: slot 11 → head E ← active

    V0 contributes 3 votes!            V0 contributes 1 vote.
    Validators who attest more          Equal influence regardless
    often have outsized influence.      of attestation frequency.

ethlambda Implementation Reference

This section covers ethlambda-specific details: scheduling, Beacon Chain differences, source code locations, and performance.

Tick-Based Scheduling

ethlambda divides time into slots, each split into 5 intervals. The slot lasts 4 seconds unless the network’s config file sets MILLISECONDS_PER_SLOT, so the 800 ms intervals below are the default grid, as described in Slots and Intervals. Fork choice operations are scheduled at specific intervals:

                             ONE SLOT (4000 ms)
    ┌────────────┬────────────┬────────────┬────────────┬────────────┐
    │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │
    │  t+0 ms    │  t+800 ms  │ t+1600 ms  │ t+2400 ms  │ t+3200 ms  │
    ├────────────┼────────────┼────────────┼────────────┼────────────┤
    │            │            │            │            │            │
    │IF PROPOSER:│ ALL        │ aggregators│update_safe │accept_new_ │
    │ accept new │ VALIDATORS:│ publish    │_target()   │attestations│
    │ attestation│  produce   │ aggregated │            │()          │
    │ + propose  │ attestation│ attestation│ (2/3 vote  │            │
    │ block      │            │            │ threshold) │update_head │
    │            │            │            │            │()          │
    │update_head │            │            │            │            │
    │()          │            │            │            │            │
    └────────────┴────────────┴────────────┴────────────┴────────────┘

    ◄─────────────── Slot N ──────────────────────────────────────────►

Detailed sequence:

    Interval 0 ─ Slot boundary
    │
    ├── Am I the proposer for this slot?
    │   ├── YES: promote new → known attestations
    │   │        run fork choice → update_head()
    │   │        build block using known attestations
    │   │        publish block to network
    │   └── NO:  (wait for block from proposer)
    │
    Interval 1 ─ Attestation production
    │
    ├── All validators, proposer included:
    │   └── Create attestation with:
    │       • head   = current fork choice head (newest head)
    │       • target = derived from safe_target (for 3SF-mini)
    │       • source = latest_justified checkpoint
    │       Publish attestation to gossipsub
    │
    Interval 2 ─ Aggregation
    │
    ├── Aggregators: aggregate their subnet's gossip signatures
    │   └── Publish the aggregated attestation to gossipsub
    │
    Interval 3 ─ Safe target update
    │
    ├── Recalculate safe_target using 2/3 supermajority threshold
    │   └── Only blocks with ≥ ⌈2V/3⌉ attestation weight qualify
    │       (V = total validators)
    │
    Interval 4 ─ End of slot
    │
    ├── Promote new → known attestations
    └── Run fork choice → update_head()

Differences from the Ethereum Beacon Chain

Lean consensus simplifies several parts of the Ethereum Beacon Chain’s fork choice. This section describes ethlambda’s Lean Chain (ethlambda node); its Beacon Chain follower (ethlambda beacon) runs the Beacon Chain’s own fork choice, proposer boost and equivocation discounting included.

No proposer boost. The Beacon Chain adds a “proposer boost”, a temporary weight bonus given to newly proposed blocks to prevent balancing attacks. The Lean Chain has no such boost. Instead, proposer fairness is handled through the two-stage attestation pipeline (the proposer’s own attestation enters as “new”, not “known”).

No balance weighting. In the Beacon Chain, a validator with 32 ETH of effective balance has more fork choice weight than one with 16 ETH. On the Lean Chain, every validator has exactly equal weight (1 vote = 1 unit of weight), simplifying the algorithm and analysis.

No equivocation discounting. The Beacon Chain’s fork choice detects validators who equivocate (attest to conflicting blocks in the same slot) and excludes their weight. This addresses the “nothing at stake” problem where validators can costlessly vote for multiple forks. The Lean Chain’s fork choice does not do this.

Key Files

Data Flow Summary

     ┌───────────┐         ┌──────────────┐             ┌───────────────┐
     │ Gossipsub │────────▶│ New          │──(promote)─▶│ Known         │
     │ (network) │         │ Attestations │             │ Attestations  │
     └───────────┘         └──────────────┘             └───────┬───────┘
                                                                │
     ┌───────────┐                                              │
     │ LiveChain │──── { root → (slot, parent) } ───────────────┤
     │  (index)  │                                              │
     └───────────┘                                              │
                                                                ▼
                                                  ┌─────────────────┐
     ┌───────────┐                                │ compute_lmd_    │
     │ Justified │──── start_root ───────────────▶│ ghost_head()    │
     │Checkpoint │                                │                 │
     └───────────┘                                └────────┬────────┘
                                                           │
                                                    ┌──────┴──────┐
                                                    │             │
                                                    ▼             ▼
                                              ┌──────────┐ ┌───────────┐
                                              │   HEAD   │ │   SAFE    │
                                              │ (min=0)  │ │  TARGET   │
                                              └──────────┘ │ (min=2V/3)│
                                                           └───────────┘

Performance Characteristics

In practice with a small validator set and bounded non-finalized chain length, all operations complete in sub-millisecond time. The // TODO: add proto-array implementation comment in the source indicates a future optimization path: proto-array is an O(1) amortized fork choice algorithm used by most Beacon Chain clients.

Beacon Chain state transition

The beacon module of ethlambda-state-transition (crates/blockchain/state_transition/src/beacon/) implements the Ethereum Beacon Chain consensus specification, ethereum/consensus-specs, phase0 through gloas.

This is not the Lean consensus protocol the rest of the crate implements. The two sit in one crate for the reason their types sit in one crate: a caller dispatching on a state’s fork can then reach either chain’s rules without the two living in separate dependency trees. They share nothing else. Nothing above beacon in this crate reads anything inside it, and nothing inside it reads lean’s own modules.

The containers, presets, configuration and primitives this module transitions are in ethlambda-types, under its own beacon namespace, because ethlambda-storage and the networking crates need those types and must not depend on blst and c-kzg. What lives here is the behavior: the state transition, the fork choice store, the helpers, and the two cryptography modules.

blst, c-kzg and num-bigint are therefore on the lean binary’s dependency path, since ethlambda-blockchain, ethlambda-rpc and ethlambda-test-fixtures all depend on this crate. The module is not feature-gated; that is the accepted cost of holding both chains’ rules in one place.

The types are re-exported at the paths they had when they were defined here, so crate::beacon::containers, crate::beacon::preset and crate::beacon::config still resolve, and ethlambda_state_transition::beacon::containers::X and ethlambda_types::beacon::containers::X are one type by one name. Two definitions travel in the other direction, for the same reason they moved: fork_choice re-exports LatestMessage and PowBlock, which ethlambda-storage persists, and helpers::misc re-exports compute_fork_data_root, which the networking crate’s fork digest is built on.

One consequence reaches every match in the module. ethlambda-types gives BeaconState and ForkName a Lean variant, so that one state type can carry either chain, and every match on either needs an arm for it. No lean value can reach this module: a fixture case’s fork is parsed from a directory name and ForkName::ALL has no lean entry, and a beacon block or a deposit set produces a beacon state. So those arms panic and name themselves, through lean_state_unreachable/lean_fork_unreachable (src/beacon/lean_boundary.rs) and lean_is_not_a_fixture_fork in the spec tests, rather than widening signatures to a Result no correct caller could ever see. They are named arms rather than a catch-all _, so a real fork added to the enum still breaks every match that has to grow an arm for it.

Correctness is defined by the released spec test fixtures, pinned in the Makefile.

Running the tests

make consensus-spec-tests   # download the fixture tarballs (mainnet and minimal)
make cryptography-specs     # download the BLS/KZG test vectors
make test-beacon            # build and test once per preset

make test-beacon is the two per-preset targets, test-beacon-mainnet and test-beacon-minimal, run one after the other. Either can be run on its own, which is what CI does: the presets get a job each, so the two build and run concurrently instead of end to end.

A run reads its own preset’s tree and opens nothing else. CONSENSUS_SPEC_TESTS_CONFIGS narrows the download to that, and CI sets it. Locally the default fetches both, so both presets can be run without re-downloading. The BLS and KZG vectors are a separate, preset-independent download: consensus-specs shipped them itself, under a general config, through v1.7.0-alpha.12, but v1.7.0-alpha.13 (consensus-specs #5398) moved them to ethereum/cryptography-specs, which make cryptography-specs fetches into its own cryptography-specs/ tree.

The download is stamped with the release it came from and the configs it holds (consensus-spec-tests/.version-<version>-<configs>), so changing either wipes the tree and fetches afresh. Without the version in that name a bump would be a silent no-op: the extracted directories already exist, so make would consider them current and both presets would report green against the old fixtures. Without the configs, a narrowed download would mark a tree missing one preset as complete, and that preset’s next run would fail on a missing directory rather than fetching what it needs. The wipe matters as much as the re-download, since the tarballs unpack side by side into one directory, and layering a new release over an old one would keep cases the new one deleted.

make test covers the whole workspace, in two halves, and still needs no fixture download. The halves are all that test-node’s --exclude flags do; nothing is dropped to keep the beacon suite out. Two gates keep it that way, both tied to the beacon-spec-tests feature that make test-beacon turns on:

• The beacon_spec_tests target declares required-features = ["beacon-spec-tests"], so cargo test skips building it entirely.
• The BLS and KZG fixture vectors are unit tests inside the module, which no target gate reaches, so each carries #[cfg_attr(not(feature = "beacon-spec-tests"), ignore = ...)]. They report as ignored rather than disappearing, which is why make test-beacon also passes --lib: 15 tests would otherwise never run.

Neither gate touches the module itself, which always compiles. What they stand for is the fixture tree, not the code.

Layout

The first five come from ethlambda_types::beacon and are re-exported by beacon/mod.rs; the rest are defined here.

Three kinds of parameter

The specification distinguishes constants, presets, and configuration, and so does this module, because they have genuinely different lifetimes.

Presets (preset) set container sizes, so they must be compile-time constants: SszVector<Root, { preset::SLOTS_PER_HISTORICAL_ROOT }>. The preset therefore cannot be a runtime value or a type parameter. Stable Rust cannot take a const-generic argument from a trait’s associated const, so the preset is selected by Cargo feature: mainnet by default, minimal with preset-minimal. The test target builds the crate twice and each run walks only its own fixture tree.

Configuration (config) is runtime, because the transition fixture suite moves a fork’s activation epoch per test case. That is the concrete reason fork scheduling is not a preset.

Constants (constants) are fixed by the specification and vary by neither.

Retuning a value without redefining a function

The specification expresses a retuned constant as a fresh name per fork (MIN_SLASHING_PENALTY_QUOTIENT, then ..._ALTAIR, then ..._BELLATRIX) and redefines the function that reads it, so each fork carries its own copy of that function differing in one identifier. preset::retuned (crates/common/types/src/beacon/preset.rs, reached here as crate::preset) selects the value by fork instead, in four functions, so slash_validator and its neighbors stay a single copy each rather than one per fork.

This is the one place in the crate where a preset value is chosen at runtime. It is sound because none of these values bound a container: they are divisors and multipliers in balance arithmetic, not container shape.

The minimal preset is not a uniformly scaled mainnet: it overrides only phase0’s retuned values and inherits every later fork’s from mainnet unchanged. So a value can move one way across a fork boundary under mainnet and the other way under minimal: INACTIVITY_PENALTY_QUOTIENT falls from phase0 to altair under mainnet and rises under minimal. A test asserting that slashing gets uniformly harsher across the forks holds under mainnet and is false under minimal, so preset::retuned’s own tests pin the fork-to-constant mapping directly, which holds under both presets, rather than any numeric relationship, which does not.

How forks are represented

Containers that change between forks are defined once per fork as plain structs that derive their SSZ encoding, decoding, and merkleization, and an enum wraps them:

#![allow(unused)]
fn main() {
pub enum BeaconState {
    Phase0(phase0::BeaconState),
    // one variant per fork that changes the state's shape
}
}

Deriving the SSZ traits is the whole reason for that shape. Container serialization and merkleization is the highest-risk code in the crate, and the per-fork field lists are not a growing tail:

• previous_epoch_attestations and current_epoch_attestations exist only in phase0. From altair on they are absent from both the encoding and the merkle tree, replaced in position by previous_epoch_participation and current_epoch_participation, which have a different type.
• latest_execution_payload_header keeps its name from bellatrix on, but is a different container only in bellatrix, capella, and deneb; electra and fulu reuse deneb’s shape unchanged, and gloas drops it for a block hash and the latest bid.
• The field count crosses a power of two at electra, so the state’s merkle tree is five levels deep through deneb and six from electra on. The same logical field has a different generalized index in different forks. Gloas’s state is a progressive container (EIP-7495), whose tree has no fixed depth at all.
• SignedBeaconBlock::Fulu wraps electra::SignedBeaconBlock rather than a fulu type of its own, since fulu changes no field of a block. It still needs to be its own variant: fulu changes how a block is processed, since the blob commitment limit becomes epoch-dependent, so code that dispatches on fork still has to tell a fulu block from an electra one even though both carry the identical payload. Gloas is a different case: its block body changes (a bid in place of a payload, payload attestations, the parent’s execution requests), so SignedBeaconBlock::Gloas has a type of its own.

A single container with fork-conditional serialization would have to reproduce all of that by hand. Derived codecs get it from the struct definition, which is checked field by field against the spec text and then verified by ssz_static.

Since SSZ carries no type tag, the fork cannot be recovered from the bytes, so decoding takes it from context: BeaconState::from_ssz(fork, bytes).

Not duplicating the state transition once per fork

The cost of per-fork structs is that a naive implementation would copy every function once per fork. Two things prevent that:

1. Accessors for the fork-invariant fields. About twenty of the state’s fields are identical in every fork. One macro_rules! in containers/mod.rs generates their read and write accessors from a single list, and that list doubles as the crate’s statement of which fields are fork-invariant: a fork that changes one moves it out of the list and gains an explicit match at each use site.

2. Matching only where the spec diverges. Functions take the enum and use accessors, matching on the fork only where the specification itself changes behavior, so a match arm can be reviewed against the spec’s own diff.

No block-body enum

BeaconState and SignedBeaconBlock are the only enums; there is no BeaconBlockBody enum or a body trait. Such a type would have to grow a method or match arm per fork-specific field or operation list, which defeats the point of dispatching once: a caller of body.attester_slashings() would still have to know which fork it is dealing with to make sense of what comes back, since electra’s attester slashings are not phase0’s. What actually lets one function serve every fork is narrower and cheaper: process_block_header, process_randao, and process_eth1_data each take the handful of fields they read directly, rather than a whole body, so validating a header does not care whether the body it came from also carries a sync aggregate or an execution payload. A shared step earns its genericity by needing less, not by being handed a bigger abstraction to see through. See stf/mod.rs’s module doc for the full reasoning.

Crossing a fork boundary

process_slots performs the fork-boundary state upgrade itself, inside its slot loop, at the first slot of the activation epoch, looping again in case a single configuration activates two forks at the same epoch. state_transition checks the block’s fork against the state’s own only after calling process_slots, not before: a block at the first slot of an activation epoch is legitimately post-fork shaped while the state arriving there is still pre-fork, which is exactly what a fork transition consists of. Checking first would reject every legitimate fork-boundary block and make crossing a fork impossible.

Registry and balances

validators and balances hold one entry per validator, about 2.4M on mainnet, so they dominate both the cost of a state root and the memory of every cached state. They are ethlambda_ssz_tree::Lists rather than SszLists (from gloas on, ethlambda_ssz_tree::ProgressiveLists, described in the Gloas section below): persistent Merkle trees in the shape of the SSZ one, modeled on the milhouse lists lighthouse keeps its state in.

• Nodes cache their hash, and states share nodes. A state derived from another shares every subtree the block did not touch through Arc, and its root rehashes only the touched paths. A state decoded from storage is rebased onto a cached relative, so it shares memory with it too.
• Leaves and inner nodes are page-sized. A leaf holds a contiguous run of elements (32 validators, 512 balances), and an inner node up to 512 child pointers standing for nine binary levels at once. A lookup therefore crosses a handful of nodes rather than one per level of the registry’s depth, and iteration walks each leaf as a slice. A composite leaf also keeps each element’s root once hashed, and a rebuilt leaf carries over the roots of the elements it did not change, so one changed validator costs one validator hash plus a fold of the cached roots.
• Writes are buffered. get_mut, IndexMut and push record the new value; BeaconState::apply_pending_mutations folds everything buffered into the trees in one pass. The state transition calls it at the top of process_slot, after process_block, and at the end of process_slots (epoch processing runs after the loop’s last process_slot). A root taken with writes pending is still correct but computed on a throwaway copy, and the store flushes a state before caching it, since a shared Arc cannot be flushed later.
• A pass that writes most of a list uses the write cursor instead. List::iter_cow (and try_update_each, its closure form) walks the list in order and copies a leaf only when a write in it changes something, comparing against the original at the leaf’s end and keeping the original leaf, its Arc and its hash if nothing differs. Nothing goes through the buffer, so a full sweep avoids a map entry and a sorted copy per element. The changed leaves are swapped into the tree when the cursor is dropped, which rebuilds only the paths to them; a changed composite element’s root is computed then, on the state-transition thread. Writes pending when the pass starts are applied first. BeaconState::registry_mut hands out validators and balances as disjoint borrows, so a pass can write one while reading the other. The balance application of process_rewards_and_penalties and both process_effective_balance_updates run this way: they move together, since once the rewards write bypasses the buffer, the effective balance pass would otherwise pay a tree descent per balances()[i] instead of a buffer hit.

The access pattern matters. state.validator(i) and state.balance(i) are tree descents, cheap next to a hash but far from an array index, and they add up when a helper calls them once per validator: get_total_active_balance used to build the active-index Vec and then read every index back, and runs several times per block (once per attestation through get_base_reward_per_increment, twice per sync aggregate, once per execution request through the churn limits). In the 2026-09-28 import profile, those per-index reads and the repeated whole-registry scans were the largest cost left after hashing. A loop over the registry should walk iter_validators(), zipped with iter_balances() where it needs both; get_total_active_balance now does, in one pass.

Block processing does not even pay that pass more than once per epoch. The total is cached in ActiveBalanceCache (crates/storage/src/active_balance_cache.rs), held by the Store beside the committee cache and consulted through ActiveBalanceCacheExt::total_active_balance. The key is the epoch plus the block root at the last slot of the epoch before it: effective balances are written only by that epoch’s process_effective_balance_updates, and every activation or exit lands at least MAX_SEED_LOOKAHEAD epochs ahead, so no block inside the epoch can move the total, and two states agreeing on that root agree on it. The cache lives outside the state, so it adds no field to BeaconState and no SSZ concern; callers with no store (spec runners, block production, tests) hold a fresh ActiveBalanceCache::default(). Epoch processing, churn helpers and fork choice keep calling the uncached one-pass get_total_active_balance. In debug builds every hit is cross-checked against a fresh computation.

Epoch steps 1-3 (justification, inactivity updates, rewards) follow that rule through helpers::participation. One walk of iter_validators(), zipped with the flat participation slices, produces each validator’s flags, its effective balance and the balance totals (an EpochSummary); the three steps then run over flat data and write back only the balances that changed. The summary lives for those three steps only, since registry updates change what it read. The per-block pulled-up tip needs just the totals, which ParticipationTotals computes without allocating. The fixtures still call each step alone, so the public step functions stay and each builds what it needs. The specification-shaped implementation is kept in participation_reference and stf::epoch::altair_reference for tests and debug builds: the driver runs it on a clone for registries of up to 4096 validators and asserts the outcomes agree. The leak flag is read once per step rather than once per validator, which fails some states the specification accepts; see Spec Deviations.

Electra and fulu go further (stf::epoch::single_pass): steps 2-9 (inactivity, rewards, registry updates, slashings, pending deposits, effective-balance updates) run in one loop over the registry, each validator on a local copy, with the changes written back afterwards. What depends on more than one validator is handled outside the loop: the exit-churn cursor is advanced locally in index order, the pending-deposit queue is planned before the loop (top-ups of existing validators inside it, deposits that create validators after it), and pending consolidations run after it, with the effective-balance update of every validator they name deferred until they are done. The genesis epoch and states whose registry-sized lists differ in length fall back to the step-by-step path. The debug oracle runs the specification-shaped steps on a clone and asserts the two full states hash equal.

Gloas

Five of gloas’s EIPs shape this module: ePBS (EIP-7732), progressive lists and containers (EIP-7688), proposer selection that skips slashed validators (EIP-8045), increased exit and consolidation churn (EIP-8061), and builder deposits and exits (EIP-8282). EIP-7843 and EIP-7928 only add payload fields (slot_number, which verify_execution_payload_envelope checks, and block_access_list). It is the one fork whose containers change shape wholesale, so its code sits beside the pre-gloas code rather than inside it: stf/gloas.rs and stf/epoch/gloas.rs (transition), helpers/gloas.rs (helpers), the gloas section of fork_choice.rs (fork choice), and containers/gloas.rs in ethlambda-types. Every gloas case that is not ignored passes on both presets.

The live follower follows gloas (ForkName::Gloas.is_followed()): the chain actor imports gloas blocks and their payload envelopes, a gloas checkpoint anchor is accepted (it starts EMPTY, with no payload known), and the clock passing the fork epoch no longer forces the sync status to syncing. Fork choice records the head node’s payload status next to the head root (Store::head_payload_status, in memory, recomputed by the first head computation after a restart), which the envelope by-range server reads to withhold the head’s envelope while the head is the EMPTY node. The node is still only a follower: the validator-client endpoints refuse a gloas epoch.

The envelope pipeline in the chain actor

Order of arrival is not order of validity: an envelope can beat its block, a block can name a parent payload that has not arrived, and the columns an envelope’s commitments need can lag both. beacon_envelope.rs holds each case in a queue with its own bound, and beacon_columns.rs bounds the parked sidecars of the same fork.

Data availability moves with the payload: a gloas block’s commitments are in its bid, so the columns gate the envelope that reveals the payload, not the block’s own import. A held block’s parent envelope is requested through FetchRequest.needs_envelope the moment the block is held, whatever its source: BlockSource::Sync also marks by-root answers and pending children a cascade released, so no range batch can be assumed to bring the envelope, and p2p dedups in-flight roots. The tick re-asks once a slot for every parent still missing one, which is only what recovers a fetch that gave up; catch-up does not wait for it.

The two envelope-before-block queues differ in who has vouched for the root. A range batch’s envelopes arrive when all but its first block are stored and unimported, so a stored block’s own bid is the authority: an envelope matching it (envelope_matches_bid) goes to awaiting_import outside the per-slot cap, and one contradicting it is dropped. Only a root with no stored block, which nobody has authenticated, is held under the per-slot cap and the TTL. Everything is swept at finality, except the finalized block’s own envelope.

Fork-choice events are timed at their arrival: the store clock is advanced to the moment an envelope, block or payload vote reached the node before its handler runs, because the tick fires once per slot and the timeliness deadlines would otherwise be judged at the slot’s start. Payload votes already verified in p2p are applied without a second signature check (apply_verified_payload_attestation). The PTC vote vectors are not persisted and are reseeded empty on resume; see data_storage.md.

The deferred payload

A gloas block has no execution payload. Its body commits to a builder’s bid (process_execution_payload_bid), and the builder reveals the payload later in a SignedExecutionPayloadEnvelope. The payload is applied by the next block: process_parent_execution_payload compares that block’s bid against the parent’s committed one to tell whether the parent was full or empty, and for a full parent apply_parent_execution_payload runs the execution-layer requests carried in parent_execution_requests and settles the builder’s payment. An empty parent must carry no requests.

verify_execution_payload_envelope is the checking half that used to sit in process_execution_payload, and it mutates nothing: fork choice calls it on a stored state when an envelope arrives, and only the next block changes state. Block processing therefore takes no ExecutionEngine, and process_withdrawals takes no payload, since the expected withdrawals are a function of the state alone.

Execution-layer requests moved with the payload. Deposit, withdrawal and consolidation requests, and the two builder requests, are reached only through apply_parent_execution_payload; gloas’s process_operations no longer calls the first three. Their behavior did not change, so apply_parent_execution_payload calls the fulu and electra processors instead of keeping copies.

Progressive lists and the registry

EIP-7688 makes most lists in the gloas state and body unbounded, merkleized as a chain of growing subtrees (EIP-7916), and makes the big containers progressive (EIP-7495); a progressive container has no fixed depth. The other progressive lists are libssz’s ProgressiveList. The registry is the exception: validators and balances stay tree-backed, through ethlambda_ssz_tree::ProgressiveList, which has the same persistent nodes, update buffer and rebase as the bounded List and the same method set, so a gloas state shares subtrees with its parent the way earlier ones do.

The registry therefore has two list types across the forks, and an accessor cannot return either one. The whole-registry accessors were replaced by element-level ones on BeaconState: validator_count, validator, validator_mut, balance, balance_mut, push_validator, iter_validators, iter_balances and validators_root. Each dispatches once and hands back an element, a count or an iterator. Only the few that need the list itself (iter_validators, iter_balances, rebasing onto a cached state, pointer equality) match on the private Registry enum.

One function for both list types

The spec modifies a small part of what gloas touches, and the rest is electra’s text run against a differently typed list. Copying it would fork every fix, so the shared functions take narrow views instead:

Both list types deref to a slice of the same element type, so a view can expose slices and element-level methods and let electra’s process_withdrawal_request, process_consolidation_request, exit and slashing paths serve a gloas state unchanged. The churn limit is the one input that differs: electra’s cursor functions read gloas’s own limits on a gloas state, which is a fork dispatch inside them rather than a second copy. A function gets a gloas copy only where the spec marks it modified. The rule for choosing between the views and a projection is the one in “A recurring bug” below: project only when the return type must be gloas’s own.

Fork choice

Gloas gives each block two branches to weigh: empty, where no payload was revealed or attested as timely and available, and full, where the builder’s payload was. A gloas fork-choice node is therefore (root, PayloadStatus) with Empty, Full and Pending, where every earlier fork’s node is a bare root. Nodes are derived on demand from the block tree, each block’s bid and the set of verified payloads (get_node_children builds them), not stored, which mirrors the spec’s Store.

ForkRules (PreGloas or Gloas) is the other new dimension. It is derived from an exhaustive match on a fork or a container, and it is what validate_on_attestation and update_latest_messages read, since gloas turns data.index into the payload flag and orders votes by slot instead of target epoch. A vote’s rules follow its own container, not the clock.

One head computation serves every fork: get_head_node builds a weight table for the node tree in one bottom-up pass and descends over it. The current slot’s fork selects the two rules that differ: from GLOAS_FORK_EPOCH on, blocks carry the payload dimension their bids give them and the proposer boost is gated by should_apply_proposer_boost; before it, every block is a single full node and the boost applies whenever it is set. A block’s fork and parent payload status are recorded at import (BlockPayloadLink), so the walk decodes no block in the ordinary case; the two exceptions are in spec_deviations.md. compute_head and the spec-literal gloas_get_head are kept as the references it is tested against; the cost argument is in the same page. The spec never describes a tree that crosses the boundary, so that page also records the rule this module chose. Two new handlers, on_execution_payload_envelope and on_payload_attestation_message, join the spec’s list of ways to change the store.

Fork choice has the same problem on a longer loop: get_head weighs every validator’s latest vote by the voter’s balance at the justified checkpoint, and the boost needs the total balance of that state’s active, unslashed validators. Both read one JustifiedBalances snapshot (ethlambda-types, held in the store’s BeaconScratch) instead of a validator(i) descent per vote and a registry scan per boost. It is built from checkpoint_state(justified) only, keyed by the checkpoint it came from, so a moved justified checkpoint is a miss with no hook on the writers; it holds a zero for validators that are inactive or slashed, and the sum of what it holds, floored at one increment. That sum leaves slashed-but-active validators out, as calculate_committee_fraction does since consensus-specs #5679 (v1.7.0-beta.4), so it is not get_total_active_balance. Equivocations stay out of it: they are store-level and can change at any time, so they are filtered per vote. get_weight stays as the specification’s per-root definition, and the fork-choice fixture runner checks compute_weights against it at every checks step.

The latest votes are stored the same way, as a dense table indexed by validator index rather than a hash map, so the vote loop reads votes and snapshot balances in index order. The table grows to the highest voting index, which is why set_latest_message only takes indices from validated attestations (a debug assertion bounds it).

Macros and traits

Two macro_rules! in the whole crate, both local, both replacing boilerplate that would otherwise run to hundreds of near-identical lines: the fixed-length byte strings in primitives, and the state accessors in containers. No procedural macros beyond the SSZ derives, and no trait abstracting over BLS or KZG backends.

Cryptography

bls wraps blst, and kzg wraps c-kzg with the eip-7594 cell and column functions fulu needs, both matching the versions ethrex uses.

Two details worth knowing:

• Every BLS function re-validates its inputs on each call rather than trusting a wrapper validated once. BlsPubkey is deliberately unvalidated on construction, because deposit processing has to be able to hold a key that never validates, so nothing upstream guarantees a key is a subgroup-correct point.
• compute_challenge and compute_verify_cell_kzg_proof_batch_challenge are implemented directly from the spec rather than called, because c-kzg keeps them as private steps of its own proof routines yet both have their own fixture handlers.

Fixture suites

consensus-spec-tests/tests/<config>/<fork>/<runner>/<handler>/<suite>/<case>/, where <config> is the preset name. Container files are .ssz_snappy: SSZ compressed with raw snappy, not the framed format. The BLS and KZG vectors live in a separate tree, cryptography-specs/tests/<kind>/<handler>/<case>/, flatter still: no fork and no suite level, since neither is preset- or fork-dependent.

Runners discover their cases from disk rather than listing them, so a fixture release that adds cases needs no code change. Two properties are deliberate: a suite that matches no case fails rather than reporting green, and a container or fork pair with no implementation yet is counted and printed rather than passed over silently, so the output never implies more coverage than exists.

value.yaml goes unread in ssz_static. Using it would need a serde implementation for every container and would pin down nothing that the serialized bytes and expected root do not already.

Performance

The mainnet spec suite went from 1576s to about 106s. Two causes:

1. sha2’s asm feature selects the CPU’s SHA-256 instructions. Without it, sha2 compiles the portable scalar backend on aarch64 regardless of what the CPU supports, and merkleization is almost entirely SHA-256 compressions, so a spec fixture case running two whole-state merkleizations feels the difference more than anything else does.
2. Every fixture case is its own test, so the harness runs them concurrently at one case per work item. That is finer-grained than a suite-per-test layout could balance, where the slowest suite alone set the wall clock.

CPU time and wall clock separate the two cleanly, since running work in parallel cannot reduce the total CPU time it takes:

So parallelism accounts for the remaining 4.9x, taking the run from two of eleven cores busy to about eight. The 3.3x understates the hashing change, because the later run does strictly more work: transition went from failing immediately to running every case.

Epoch-transition precompute

The first block of an epoch used to run process_epoch inline on the import path (fork_choice::on_block, then process_slots), followed by the rehash the state-root check needs. The beacon chain actor now does that work ahead of time, on a blocking worker, and the import resumes from the result.

The code lives in crates/blockchain/src/epoch_precompute.rs (triggers, worker, actor handlers) and fork_choice::advance_to_epoch_start / fork_choice::transition_block in the state-transition crate. The upgrade to a new fork at the boundary happens inside process_slots, so the precompute needs no fork-specific code. Gloas is covered by the same key: a gloas block’s post-state never contains its own payload (the next block applies it inside process_block, after process_slots), and the store keeps exactly one state per root, so the entry for (E+1, root) is advanced from the same state an import of a child of root would clone. Hits and misses, worker time and trigger counts are exported as lean_beacon_epoch_precompute_*; see metrics.

One test per fixture case

The suites were once one test apiece, each looping over its own cases and aggregating outcomes. That made every failure a failure of the whole suite: the name in the output was the suite’s, and a single bad case marked thousands of passing ones as part of one failed test.

A fixture case is not known until the fixture tree is walked, and #[test] needs its tests at compile time, so the spec binary supplies its own harness (harness = false, with libtest_mimic) and builds its test list at run time. Each case is then named, counted, filtered, and attributed on its own, and --test-threads, --ignored, --list, and substring filters all keep working. A filter selects a whole suite as readily as one case, since a test’s name is its runner followed by Case::id:

operations/electra/attester_slashing/pyspec_tests/basic_double

Two things that arrangement has to be careful about:

• A case whose fork this module does not implement becomes an ignored test rather than a missing one. The harness counts and names ignored tests, which says more than the tally the old aggregate printed.
• A suite that matches no case at all would otherwise contribute no tests, and a run of nothing passes. The aggregate used to assert it had matched something; that check survives as one <runner>/matched_fixture_cases test per suite, so a stale runner or handler name still fails loudly.

Status

All eight forks, phase0 through gloas, have containers, fork upgrades, state transitions, and epoch processing. At the pinned fixture release every case that is not ignored passes on both presets:

The lib counts are with beacon-spec-tests on, and include lean’s own unit tests, since the two chains share one lib target. The one ignored lib test is a timing measurement, not a fixture.

Minimal runs more cases because the release ships more fixtures for it, and it runs two runners mainnet does not: genesis’s initialization and validity. The release ships no mainnet genesis fixtures, so that whole runner is gated behind the preset-minimal feature (tests/beacon_spec/genesis.rs).

The gossip validation vectors this crate’s gossip.rs runner covers come from the same tree as every other suite. The node validates four topics under fulu’s rules; every other case is ignored by name rather than left unmatched.

Nothing is ignored for being unimplemented in the state transition or fork choice: HIGHEST_IMPLEMENTED_FORK is gloas, and every gloas case that is not ignored runs. The ignored cases are deliberate exclusions:

• LightClient* containers under ssz_static, 5 container types across altair through gloas. The light-client sync protocol is a different layer from the state transition and fork choice, and is not in this module’s scope.
• networking/gossip_* cases outside what the node validates: every fork’s cases for topics it has no validator for (IGNORED_HANDLERS), non-fulu cases of the aggregate and attestation topics (block and data column validate fulu and gloas), and the vectors in SKIPPED, which assume a bad-block cache.
• The heze fixture tree, one ignored entry. See “Accounting for every fork directory” below.

Accounting for every fork directory

collect identifies a fork by parsing the directory name into a ForkName, and a name that does not parse is skipped. That skip is silent in a way the HIGHEST_IMPLEMENTED_FORK gate is not: the cases never become tests, so they are not counted as ignored either, and nothing in the output says they exist.

gloas does not take this silent path: ForkName::parse("gloas") succeeds, so its cases become ordinary tests, individually named and run. heze, the one fork after gloas, is still unparseable and would be silently skipped without this section’s own accounting.

So UNMODELED_FORKS names heze alone, it reports as one ignored test, and fixture_forks/every_directory_is_accounted_for fails if the tree holds a fork directory that is neither parseable nor listed. A release that adds a fork forces a decision instead of quietly widening the gap.

Fork choice is fixture-verified

205 mainnet fork_choice cases pass, covering bellatrix’s on_merge_block/ terminal-PoW validation, the v1.7.0 proposer-boost dependent-root gate and get_proposer_head’s proposer-equivocation branch, deneb’s blob data availability, fulu’s column data availability, and gloas’s payload-aware suites: get_head, get_parent_payload_status, on_block, on_attestation, on_execution_payload_envelope, on_payload_attestation_message, payload_data_availability, payload_timeliness and ex_ante. Gloas accounts for 54 of the 205.

The release still ships no phase0 fork_choice suite: the earliest is altair’s, built from altair-shaped states even though altair changes nothing about fork choice itself (Store accepts a block from any fork this module implements; see fork_choice.rs’s own module doc). Landing altair’s state transition is what let this suite start running at all.

A note on earlier case counts

Counts reported before the runners landed were too high by roughly sevenfold. collect_all_handlers walked the fork directories and then delegated to collect, which walks them again, so every case was emitted once per fork shipping that runner. The cases were always being run; they were counted many times over. Both collectors now share one suite walk.

A recurring bug: projecting when an accessor was needed

Reach for a concrete per-fork projection, such as altair_state(state), only when the return type must be that fork’s own. Reach through a BeaconState accessor when the fields involved are shared. The trap is that a projection like altair_state(state) matches only BeaconState::Altair, so it compiles cleanly, passes the fork that introduced the field, and returns UnsupportedForFork for every later fork that shares the exact same field through a different variant.

This caused five separate bugs during implementation:

• Capella’s withdrawal sweep, projected to capella’s own state.
• Deneb reusing that same sweep, still projected to capella’s state.
• Altair’s participation fields, projected to altair’s state, so every later fork that also carries participation failed.
• Deneb’s process_attestation, projected to deneb’s state.
• slash_validator calling phase0’s initiate_validator_exit at electra, which also left electra’s EIP-7251 churn cursor unadvanced, mispricing every later exit processed in the same epoch.

Deliberate simplifications

• Light client containers and suites are out of scope.
• ssz_generic exercises the SSZ library rather than the beacon containers, so it is not a gate.
• The state transition mutates in place, as the specification does, so a state passed to state_transition is left partly modified when a block turns out to be invalid. Callers that need the pre-state clone it first, which is what the fixture runners do.
• ExecutionEngine (stf::mod) stands in for a real execution client: just execution_valid: bool, read straight from a fixture’s execution.yaml rather than a payload actually validated.

What the fixture format asserts by omission

A case with a post state must succeed and land exactly on it. A case without one must be rejected. The second half is what keeps the suites honest: an implementation that accepted everything would otherwise pass every case that ships a post-state. That rule lives in one place, check_transition, and every state-comparing runner goes through it.

Two consequences worth knowing:

• Roughly a quarter of the phase0 operations cases are rejection cases, so the invalid path gets as much coverage as the valid one.
• The rewards suite is the exception to state comparison: it compares the five per-component delta vectors directly. That is sharper, because the components are summed into balances, so a sign error in one component and a compensating error in another would produce correct balances from incorrect deltas.

Where the specification’s Python does more than it appears to

Two places where a faithful-looking transcription is wrong, both found by the fixtures rather than by reading:

• get_matching_target_attestations evaluates get_block_root inside a list comprehension, so it never runs when there are no attestations. That call has its own range assertion, which fails for the epoch a state sits at the start of. Hoisting it out of the loop rejects states the specification accepts.
• get_attesting_indices returns an unordered set, and get_indexed_attestation sorts it. A committee is a shuffled slice of the registry, so filtering it in position order yields attesters in shuffle order, which is almost never ascending. is_valid_indexed_attestation requires sorted indices, so skipping the sort rejects every valid attestation.

Both are cases where the Python reads as if order or evaluation point does not matter, and both change the result.

The mainnet wire

ethlambda beacon follows the Ethereum Beacon Chain’s gossip. This page describes what it puts on the wire; discovery.md covers the discv5 stack it shares with lean, and cli.md the flags and the startup order.

It follows, from its checkpoint anchor to the tip: a chain actor imports what gossip announces and what range sync fetches, and the two block protocols serve other peers from the same store. Fork choice learns its votes from block bodies and from the aggregate topic, which is how it sees votes for the current head rather than only ones at least a block old; see Aggregate attestations. It publishes only what its Beacon API clients submit (attestations, aggregates, blocks with their columns, operations) and relays validated gossip. It also custodies and serves a slice of the fulu data column matrix, and backbones a slice of the attestation subnets, both sized and selected by its own node id; see Data column sidecars.

Running it

ethlambda beacon \
  --node-key ./node-key \
  --gossipsub-port 9001

No flag is required, but that is only true for a built-in network: ethlambda beacon on its own follows mainnet, since --network defaults to mainnet. Sepolia, Hoodi and Platåberget are built in too, and any other network is a --network naming a directory of published files (config.yaml and genesis.ssz, see cli.md):

ethlambda beacon --network hoodi --checkpoint-sync-url https://checkpoint-sync.hoodi.ethpandaops.io
ethlambda beacon --network ./my-network --node-key ./node-key

genesis_time and genesis_validators_root, and therefore the fork digest, are derived from the resolved network: a built-in network (mainnet, sepolia, hoodi, plataberget) carries the two values as constants beside its published config.yaml and bootnode list, and no genesis state; for a loaded network they are read off that directory’s own genesis.ssz. Nothing about startup touches the network to get there. discv5 is forced on and needs no flag either: published mainnet bootnodes are largely seed-only, so a crawl is how a peer is reached.

The fork digest

Computed from the resolved network’s genesis state and fork schedule, never hardcoded as a digest. Startup derives the first one and a running node follows the schedule from there (see below):

epoch        = (now - genesis_time) / (SECONDS_PER_SLOT * SLOTS_PER_EPOCH)
fork_version = the resolved network's fork schedule at epoch
base         = compute_fork_data_root(fork_version, genesis_validators_root)
digest       = base[..4]                                    if epoch <  FULU_FORK_EPOCH
             = xor(base, sha256(le64(bp.epoch) ++
                                le64(bp.max_blobs)))[..4]   if epoch >= FULU_FORK_EPOCH

bp is the latest blob-schedule entry at or before epoch, falling back to (ELECTRA_FORK_EPOCH, MAX_BLOBS_PER_BLOCK_ELECTRA). The fulu branch is EIP-7892’s, which is why mainnet’s digest is 8c9f62fe rather than fulu’s bare 82fae541.

Startup logs the next boundary’s epoch, wall-clock time, fork and digest. A running node crosses it without a restart (beacon::transition in ethlambda-p2p, driven by beacon::fork_schedule). Both fork activations and blob-parameter-only forks move the digest, so both are boundaries. Following the spec’s “Transitioning the gossip” rules, the next digest’s topics are joined SUBSCRIBE_LEAD_EPOCHS before its boundary and the previous digest’s topics are left UNSUBSCRIBE_LAG_EPOCHS after it. At the boundary the node publishes and advertises the new digest, bumps its metadata sequence number and moves its discv5 admission filter. While the window is open a gossip message’s fork comes from its own topic’s digest, and a Status on either digest is answered, but range sync starts only from a peer on the current digest (or the next one, within clock skew of the boundary).

One gap: the ENR this node serves over discv5 keeps the eth2 entry it started with, because ethrex’s DiscoveryServer offers no way to replace its record at runtime. Peers already connected are unaffected; peers discovering the node after a boundary read the stale digest and reject the record.

Gossip

Seven global topics, /eth2/{digest}/{name}/ssz_snappy, plus two subnet families this node’s own node id selects a narrow slice of: the data column subnets it custodies and the attestation subnets it backbones, both described below. A gloas digest adds four more, execution_payload, payload_attestation_message, execution_payload_bid and proposer_preferences, which BeaconTopics::for_fork subscribes under gloas digests and not under earlier ones (so one epoch before the fork, when the digest is joined). An envelope is validated like a block (its stateful half on the blocking pool) and goes to the chain actor on Accept and on Queue, since the actor holds an envelope whose block is not imported yet. A payload attestation is validated on the attestation permit pool and goes to the actor on Accept only: a vote for a block not imported here is dropped, since it is valid only within its own slot. Bids and preferences are the builder market: they never reach the chain actor. Their rules live in state_transition::beacon::gossip::{execution_payload_bid, proposer_preferences}, the cheap half runs in p2p’s triage (p2p/src/beacon/builder_market.rs) and the stateful half on a permit pool of their own (builder_validation_permits), so a bid burst cannot starve blocks, columns or attestations. Whatever is accepted goes into the node’s one shared BuilderMarket, which the Beacon API and block production read. Neither is ever queued: a message whose parent or dependent block is unknown is ignored. A bid is capped at 196,932 decompressed bytes (Reject(Malformed) above that). The node publishes both: a bid on the digest of its slot, preferences on the digest of the proposal slot, which in the epoch before gloas is the gloas digest already joined. Envelopes the node publishes itself are recorded as known payloads, since gossip never echoes them.

beacon_attestation_{0..63} is no longer wholly unsubscribed. This node holds SUBNETS_PER_NODE (2 on mainnet) long-lived subscriptions from that family, chosen by compute_subscribed_subnets(node_id, epoch), a public function of this node’s own discv5 node id in the same way custody_columns is, so any peer can compute the set without asking. p2p-interface.md asks every beacon node to hold them whether or not it runs validators: phase 0 has no shard committees, so nothing else gives these subnets a stable membership for validators to publish into. The subscription is therefore owed to the network rather than to this node’s head: this node verifies and relays what arrives on it (see Gossip validation below) but never applies it to fork choice. A lighthouse node with no validators behaves the same way, subscribing to its own node-id backbone and verifying what arrives on it while should_process_attestation keeps it out of fork choice unless a local aggregator duty or --import-all-attestations says otherwise.

One deliberate shortfall: the set is computed once at startup and kept for the process’s lifetime rather than rotating every EPOCHS_PER_SUBNET_SUBSCRIPTION epochs. Lighthouse does the same, and reads that constant nowhere.

sync_committee_{0..3} is not in the startup set either: those subnets are joined on demand, see Sync committee subnets. blob_sidecar_{subnet_id} stays absent permanently: it is deneb’s format for blobs, deprecated at fulu in favor of the column matrix below.

data_column_sidecar_{0..127} is no longer in that absent list. This node subscribes to sampling_size(CUSTODY_REQUIREMENT) of them — the sampling size floored by SAMPLES_PER_SLOT above CUSTODY_REQUIREMENT itself — chosen by custody_columns(node_id, …), a public function of this node’s own discv5 node id (das-core.md), so any peer can compute the same set without asking. NUMBER_OF_CUSTODY_GROUPS and DATA_COLUMN_SIDECAR_SUBNET_COUNT are equal today, so a column is its own subnet with no reduction. That makes this node’s total subscription count the seven global topics plus its sampling size plus SUBNETS_PER_NODE, still far short of a full subscription to every attestation, sync-committee and data-column subnet, and narrower still than NUMBER_OF_CUSTODY_GROUPS columns of custody, which is what a supernode would carry alone. A sidecar decodes as the fork-agnostic DataColumnSidecar enum, its fork taken from the topic’s digest (the bytes carry no tag): fulu’s names its block by a signed header and inclusion proof, gloas’s by a slot and block root, with the commitments it is checked against living in that block’s bid. The checks it passes before this node keeps or forwards it are described just below. How a kept sidecar is later served back out over req/resp is under Data column sidecars.

Every beacon message is held by gossipsub until it has a verdict (validate_messages() is on for this wire only). Blocks, data column sidecars, aggregates, subnet attestations and the four operation topics are all validated by fulu’s gossip rules, or (operations aside) gloas’s modified ones from the fork on (ethlambda_state_transition::beacon::gossip, one module per topic family): the checks that need no state run inline in the p2p actor, the rest on a bounded spawn_blocking task whose verdict comes back to the actor (crate::beacon::verdict). Two permit pools bound how many of these run at once, so a burst on one family cannot starve another: gossip_validation_permits for blocks and columns, attestation_validation_permits for aggregates, subnet attestations and the operation topics. A mainnet slot carries up to MAX_COMMITTEES_PER_SLOT * TARGET_AGGREGATORS_PER_COMMITTEE aggregates alone, arriving every slot rather than only during a range sync, which is why that traffic needs a pool of its own rather than sharing the block and column one. Accept propagates the message; a message whose dependency is not ready yet is IGNOREd. See Aggregate attestations for the two topics with their own section.

Blocks and data column sidecars are, either way, handed to the chain actor, which parks what it cannot import yet and imports the rest immediately, such as a sidecar whose slot merely falls outside its parent state’s proposer lookahead. An aggregate reaches the chain actor only on Accept, carrying the attesting indices gossip validation resolved. A subnet attestation never reaches it at all, on any outcome: verifying and relaying it is the whole of what this node owes the topic (see above), so there is nothing further for the chain actor to do with one.

The rules above are fulu’s, except that blocks, data columns, aggregates and subnet attestations also have gloas rules. A gloas aggregate has electra’s bytes but its own progressive Attestation, so it decodes to its own SignedAggregateAndProof::Gloas (the signed root differs); a gloas subnet attestation is electra’s SingleAttestation unchanged. In both, data.index is the payload flag (0 or 1), and a vote for the full payload (1) is IGNOREd until the block’s envelope has been seen and its payload validated (verify_attestation_payload_status). Gloas is a followed fork, so a block at a gloas slot that fails to decode is REJECTed like one at any other followed fork; unsupported_fork is kept for a fork the node does not follow (ForkName::is_followed), none today. A gloas aggregate’s aggregation bits are an unbounded progressive bitlist, so one longer than electra’s type bound is REJECTed first, reading its length without expanding it, since expanding is what the seen-set check does before any other.

Three deliberate departures from the gloas gossip rules, each for want of state this node does not keep. Where the specification rejects a message whose block failed validation, this node (having no bad-block cache) cannot tell that from a block it has not finished importing, so it queues or ignores instead; the matching reject_block_failed_validation vectors are in SKIPPED. An execution_payload whose block state is not in the cache p2p reads (a parent imported moments ago, say) is queued and forwarded to the chain actor, which verifies every envelope itself, without being propagated. And a payload_attestation_message is applied to fork choice after p2p verified its signature, with the actor re-checking only the store-state conditions (known block, current slot, committee seat).

voluntary_exit, proposer_slashing, attester_slashing and bls_to_execution_change are validated in p2p like aggregates, by gossip::operations: the specification’s validate_*_gossip functions (consensus-specs v1.7.0-beta.1), run against the fork-choice head’s post-state as it is, with no state advance and no process_*. The seen records (the first valid exit or credential change per validator, proposer slashing per proposer, and the validators an attester slashing slashes) live in SeenOperations, are recorded only on Accept and never pruned, so an operation a block already included and a peer re-gossips is IGNOREd rather than REJECTed. An attester slashing’s range and slashability checks run before its BLS checks. What Accepts goes into Store::operation_pool, which block production packs from (see rpc.md). The rules are electra’s shapes against an electra or fulu head: under a gloas head every operation is IGNOREd (state_unavailable), and a gloas-shaped attester slashing is IGNOREd (no_consumer) at decode, like a phase0-shaped one.

sync_committee_contribution_and_proof is decoded, logged at debug, and IGNOREd, since nothing consumes it; an undecodable payload on any topic is REJECTed.

Publication runs on the Beacon API’s POST /eth/v2/beacon/blocks, the only path that puts a block of this node’s own on the wire. It gossips the block on beacon_block, then, if it carries blobs, all 128 data column sidecars, each on data_column_sidecar_{compute_subnet_for_data_column_sidecar(index)} whether or not this node custodies that subnet (publishing to a subnet one has not joined goes through fanout), all under the digest the block’s slot falls in. Operations posted to the pool/* routes are published on their topics the same way, under the digest of the wall-clock slot.

Peer scoring

Gossipsub peer scoring is on for this wire only (crate::beacon::scoring). The parameters are lighthouse’s (gossipsub_scoring_parameters.rs), ported formula for formula; Grandine, Teku and Lodestar use the same constants, so thresholds mean the same thing here as on most of mainnet.

Columns, sync committee contributions and BLS changes carry no topic parameters. Only Prysm scores columns. This node Ignores every contribution and change (no consumer), and a delivery is only credited once accepted, so scoring P3 there would penalize every mesh peer for this node’s own gap. All 64 attestation subnets get parameters, not just the backbone ones, because aggregator duties join others at runtime.

Three differences from lighthouse:

• mesh_n is this node’s D (8), not lighthouse’s 5. It only enters the first-message-delivery cap.
• P3 waits for this node to keep up. Lighthouse joins these topics only once synced, so it has no mesh while it catches up. This node subscribes at startup, and while catching up it Ignores every aggregate and attestation voting for a block it has not imported. A mesh peer credited with no aggregates scores below the graylist, so a restart would graylist the whole aggregate mesh because of this node’s lag. MeshDeliveryGate turns P3 off while the head lags the wall clock by more than MESH_DELIVERY_MAX_HEAD_LAG slots, and back on only after an epoch within that lag, so the counters have refilled. While P3 is off its threshold and weight are zero but its cap is not, so the counters keep counting to their real ceiling. SyncStatus cannot serve as this gate: on the mainnet follower it reported synced through a 10-minute catch-up up to 98 slots behind, since its network-stall rule reads a lagging freshest-known block as a stalled network.
• Leaving a topic waits out the peers’ backoff. Gossipsub’s default backoff after unsubscribing (10 s) is shorter than its prune backoff (60 s), and a rust-libp2p peer that sees a mesh peer unsubscribe backs off from it for the longer one, which the leaver’s PRUNE cannot shorten. An aggregator subnet left when its duty expires and rejoined within that window for the next one (a duty in the last slots of epoch N, none in N+1, one in N+2) re-grafts into peers that reject the graft and charge a behaviour penalty. PRUNE_BACKOFF sets both backoffs to the prune value, so the node waits out the same window and is not penalized. Neither consensus-specs nor lighthouse sets these: the spec defers gossipsub v1.1’s extra parameters, and lighthouse keeps the defaults.

The block, aggregate and attestation parameters depend on the active validator count, so the p2p actor rebuilds them every slot from the head’s pinned current-epoch shuffling (CommitteeCache::head_current_committees) and hands them to the swarm. The swarm starts on lighthouse’s placeholder (32 validators, P3 off) until the first refresh, which runs at startup. Every 10 s the swarm task disconnects peers below the graylist. There is no ban list: a peer’s negative score is kept for retain_score (100 epochs) after it leaves, so a peer that reconnects comes back graylisted and is dropped again.

Around a fork-digest boundary the node holds two digests’ topics, and every held digest gets the whole parameter set (exits and slashings included), scored by its role:

• Parameters before joins. transition::apply hands the swarm every held digest’s parameters before it subscribes, on the same command channel. Gossipsub tracks a mesh peer on a topic only if the topic had parameters when the peer was grafted; parameters installed later never reach that mesh.
• The previous digest weighs nothing. Its traffic moves to the new digest at the boundary, so its delivery counters drain. Still scored, the aggregate topic alone would graylist an honest mesh peer 53 slots after the boundary (at 1M validators), before the topic is left, and leaving it would charge every mesh peer the sticky P3b. Lighthouse zeroes the same weights (remove_topic_weight_except).
• A switch restarts the warmup. The new digest’s counters start empty, but its mesh, formed before the boundary, can already be past the deliveries’ activation (one epoch for blocks and aggregates). P3 on the new digest therefore waits MESH_DELIVERY_WARMUP_SLOTS, as after a catch-up. Lighthouse joins two slots ahead, so activation alone covers it there.

Aggregate attestations

beacon_aggregate_and_proof reaches fork choice. It is how a follower learns votes for the current head rather than only the votes a block body carries, which are always at least one block old. beacon_attestation_{subnet_id}, covered in the same section below, never does: this node relays its backbone subnets without ever applying what arrives on them.

Both topics are validated the same way blocks and columns are, in ethlambda_state_transition::beacon::gossip::{aggregate,attestation}: cheap conditions (seen cache, propagation window, data.index == 0, exactly one committee named) run inline in the p2p actor; the rest run on a blocking thread, in this order:

1. The vote’s block is known (Store::has_block); if not, IGNORE.
2. Committees, signatures and ancestry all resolve against the vote block’s own cached post-state (store.cached_state(CacheKey::BlockState(beacon_block_root))), not the specification’s head state and not the target checkpoint’s state either. Three reasons converge: it is the attested chain’s own state, so its shuffling is the one the attesters were actually assigned, where the head (or a checkpoint reached by replaying a different branch) can name the wrong one; it is an O(1) cache read rather than a lookup or a replay; and its own block_roots answers both ancestry questions without a Store::block_index / LiveChain scan.
3. The signatures that need only a pubkey, before any committee derivation: an aggregate’s selection proof and aggregator signature, an attestation’s own signature. An unknown validator index REJECTs here rather than paying for a committee lookup first, since a forged message must not be able to reach a shuffling derivation.
4. The committees, through the Store-held cache both actors share; then is_aggregator (rewritten to take a committee length rather than a pre-fetched cache), committee membership, and, for a subnet attestation, the subnet match (compute_subnet_for_attestation).
5. An aggregate’s own signature, over the indexed attestation built from that committee. The indices it verifies travel to the chain actor; it never rebuilds them.
6. Ancestry against the vote state’s block_roots: the target is the vote block’s ancestor at the target epoch (REJECT), and the finalized checkpoint is an ancestor of the vote block (IGNORE).

The seen caches (SeenAggregates, keyed both by (target_epoch, aggregator_index) and by (hash_tree_root(data), committee_index); SeenAttestations, by (target_epoch, attester_index)) live in p2p now, recorded only on Accept by verdict::settle, and are bounded by capacity (an LRU, like the block and column seen caches) rather than pruned on finality: a peer must not get to grow either one just by outlasting finalization. Recording only after the signatures verify is still what keeps a garbage aggregate from being able to censor a genuine one for the rest of the epoch by claiming its (epoch, aggregator) pair first; lighthouse splits the same way, reading its observed-sets in verify_early_checks and writing them in verify_late_checks.

Only an aggregate that gossip Accepts reaches the chain actor, carrying the attesting indices already resolved; the actor never re-derives a committee or checks a signature for this topic. What is left for it:

• A lighter, actor-local applied-bits gate, a running union of aggregation bits already applied per (target_epoch, hash_tree_root(data), committee_index). Not a spec seen-set (p2p owns that one now); it exists so a committee’s other aggregators, each individually accepted by gossip because each is a first-seen, valid message, do not all pay for apply_verified_aggregate when the first one already covered their bits. Pruned by the store’s own clock to the current and previous epoch, not by finality, so a stalled chain cannot make this grow without bound either.
• The deferral queue. validate_on_attestation requires get_current_slot(store) >= data.slot + 1, and aggregates are published two thirds of the way through the slot they vote for, so every one arrives too early. Without the queue this topic would apply approximately nothing. It drains once per beacon tick, between the clock advancing and the head being recomputed, so released votes are in fork choice before that tick’s head is chosen. The specification licenses this directly (“consider scheduling it for later processing in such case”) and lighthouse has the same queue. Held entries now carry their gossip-resolved attesting indices alongside them, so a drain applies them without recomputing anything.

A subnet attestation is fully verified by this same pipeline and then simply dropped: nothing forwards it to the chain actor, on any outcome, matching a lighthouse follower with no validators, which verifies and relays its own backbone subnets while should_process_attestation keeps them out of its fork choice.

Sync committee subnets

Sync committee gossip is joined on demand, never at startup. A validator client’s POST /eth/v1/validator/sync_committee_subscriptions reaches P2PServer as subscribe_sync_committee_subnets, which subscribes each subnet (sync_committee_{0..3}) immediately under every digest the node holds, until the request’s until_epoch (exclusive). Validating all four subnets permanently would cost every follower up to a committee’s worth of BLS verifications per slot, so nothing is joined without a request.

Request/response

The two data column sidecar protocols are registered because this node custodies the columns its node id selects and can answer for them out of Table::DataColumns (see data_storage.md); see Data column sidecars for what each serves and asks for. The blob sidecar protocols stay absent: nothing here custodies a whole blob, only the erasure-coded columns fulu derives it into, and a registered protocol with no implementation behind it is an untested encoder peers can reach.

Only version 2 of the two block protocols is registered. Version 1 is deprecated by the spec, which lets a client answer it with an empty list, and its chunks carry no <context-bytes>, so serving it would mean a second response encoder for a shape no mainnet peer needs.

The Status this node sends is derived from its anchored store, so a peer reading it has a reason to ask for the blocks the two block protocols serve. There is one window where every checkpoint is still zero, before fork choice has inserted the block KEY_HEAD names; lighthouse’s relevance check exempts a zero finalized_root, so that reads as “peer is syncing” rather than as a conflicting chain. Either way it is answered in the version the stream asked for: a v1 body on a v2 stream is eight bytes short and the codec refuses to write it, which drops the connection of every peer that opens the handshake on status/2.

The two block protocols

Both request bodies have a shape that cannot be guessed from the response they produce:

step is deprecated and must be 1, but it is still on the wire: altair says the v2 request is unchanged from phase0’s, and lighthouse pins this protocol’s request length to min == max == 24 bytes, so a two-field body is refused before it is ever decoded. A body naming any other step is answered with INVALID_REQUEST, not dropped: the field is carried up to the handler precisely so the peer learns which rule it broke, where refusing at decode would close the stream with nothing on it. Phase0 does permit answering a larger step with a single block, but that leniency is for a transition that finished years ago, and the spec’s requirement on the requester is a MUST.

The root request is a bare list with no container around it, because the spec says this body “MUST be encoded as an SSZ-field” where the range body is “an SSZ-container”. The difference shows up on the wire: a container holding one variable-length field prefixes it with a four-byte offset.

Neither of those differences reaches above the codec. What a peer is asking for is the same question on either chain, so req_resp::messages owns the shared BlocksByRangeRequest, and each chain’s encoder converts to and from its own wire container: lean’s has no step and fills it with 1 on the way in, beacon’s carries all three fields. The shared struct is deliberately not SSZ-derived, so it cannot be written to either wire by accident. The root list needs no conversion at all, since beacon::primitives::Root is H256 and both lists are SszList<H256, 1024>; only lean’s container comes off.

Which chain a request arrived on is not on the message either: a node speaks one wire for its whole life, so the dispatch reads Wire::is_beacon instead. The same holds for the answer. ResponsePayload::Blocks is one variant for all four block protocols, because SignedBeaconBlock already carries a Lean variant and both stores hand blocks back in exactly that type. The narrowing to lean’s own SignedBlock happens once, where a fetched block is handed to the chain actor, and lean::encoding::write_blocks_response refuses to put a block of any other fork on a lean stream.

Two ceilings apply, and they are not the same number. MAX_REQUEST_BLOCKS (1024) is what an inbound request is judged against, because it is the widest a peer may ever legitimately have been built to ask for; a count above it is refused as INVALID_REQUEST. MAX_REQUEST_BLOCKS_DENEB (128) is what an outbound request is built to and what an answer is truncated to, since the spec allows “Clients MAY limit the number of blocks in the response” and a peer asking for more than 128 is running older logic rather than misbehaving.

A range answer comes off the canonical branch in ascending slot order, with empty slots skipped: BlockRoots holds one root per slot on the branch ending at the current head, so a sibling block at a slot the head does not descend from is never read. A root answer follows the order the roots were asked in, and leaves out any root this node does not hold.

Context bytes

Every successful block chunk carries a four-byte ForkDigest between the result byte and the payload:

response_chunk ::= <result> | <context-bytes> | <encoding-dependent-header> | <encoded-payload>

The digest is the one that block’s own epoch computes to, not the one this node is running on, so a backfill labels each chunk with its own fork. On an error chunk the field is empty, which is why the reader only reads it after a SUCCESS byte. Serving history therefore needs genesis_validators_root as well as the current digest, which is why BeaconWire and the codec both carry it.

Going the other way, the fork a chunk decodes under comes from the slot inside the payload, through the same decode path gossip uses, and the context bytes are then checked against the digest that slot implies. That is a stronger test than using them as the decoder key: it catches a peer whose genesis_validators_root or fork schedule differs from ours, which is exactly what the digest exists to say and is not otherwise visible until a signature fails. A mismatch ends the stream, logged at warn with both digests: the one way to reach it in good faith is a blob schedule of ours that has fallen behind the network’s.

Execution payload envelopes

Both gloas envelope protocols are registered Full: the server answers, and the client (req_resp/envelope_client.rs) asks. A chunk carries one SignedExecutionPayloadEnvelope under the fork digest of its block’s epoch. The envelope has no slot of its own, so the epoch comes from payload.slot_number, which verify_execution_payload_envelope pins to the block’s slot. A chunk whose digest names no gloas-or-later scheduled fork, or not the digest its slot implies, ends the stream.

Both are bounded by MAX_REQUEST_PAYLOADS: a by-root list over it fails to decode, and a by-range count over it is truncated, which the specification’s “Clients MAY limit the number of payload envelopes in the response” allows. A by-range request starting below Store::anchor_slot gets RESOURCE_UNAVAILABLE, and a window before gloas is simply empty.

The client asks in two situations. By root: the chain actor’s FetchRequest.needs_envelope (a block whose child waits on its payload) starts a lookup with the block fetch’s dedup, peer choice and retry ladder; a give-up is recovered by the actor’s per-slot re-ask. By range: after a beacon_blocks_by_range/2 answer for a span that reaches gloas has been handed to the actor, the same peer is asked for that span’s envelopes, and range sync holds the next block request until that answer is checked and forwarded (or has failed). The pause exists because the actor keeps an envelope whose block has not imported only briefly and in small numbers, so a batch’s envelopes must arrive before the next batch’s blocks do. Everything fetched goes through the same rules as an execution_payload gossip message (beacon/envelope_checks.rs) before the actor sees it; a Queue verdict is forwarded too.

Data column sidecars

Both protocols are registered Full, since with the columns on disk this node can serve every slot it has custodied:

Restricting the range answer to the canonical root matters because Table::DataColumns is never pruned and gossip only asks that a sidecar’s block name a known, finalized-descendant parent, not a canonical one: a live fork can leave both siblings’ columns stored at the same slot, and an unscoped scan would leak the losing side into every future range answer covering it.

Both per-item lookups are lenient the way the block protocols are: a column this node never custodied, or a root it holds no header for, is left out of the answer rather than turned into an error, since the spec’s own words are “Clients MUST respond with at least one sidecar, if they have it.” A data_column_sidecars_by_range/1 request starting before Store::anchor_slot gets RESOURCE_UNAVAILABLE instead of a merely-empty answer, the same distinction the block-range handler draws and for the same reason: an empty window this node’s canonical chain skipped is normal, but a window below where this node’s chain begins cannot be served from any point onward. That floor is the same value Status advertises as earliest_available_slot, so the refusal and the advertisement cannot disagree. max_request_data_column_sidecars() (MAX_REQUEST_BLOCKS_DENEB * NUMBER_OF_COLUMNS) bounds what one request may ask for; an answer over that is truncated, not refused, matching how the block protocols treat a peer built to an older, wider ceiling.

Asking happens on two paths, bulk and per block.

The bulk one rides with range sync: every BeaconBlocksByRange batch sends a data_column_sidecars_by_range/1 for the same slot span, covering this node’s whole custody set. The spec names this protocol for exactly that — “DataColumnSidecarsByRange is primarily used to sync data columns that may have been missed on gossip and to sync within the MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS window” — and it is what keeps a follower backfilling from a checkpoint anchor from needing the per-block path at all: the columns are normally already stored by the time their block reaches the availability gate. Nothing waits on the answer, so a short, empty or refused one costs nothing and is not retried; the per-block path is the backstop.

Since the column request is aimed by custody, a batch whose last slot is fulu or later is held back, blocks included, until every custody column has a known custodian among the connected peers. Lighthouse’s range sync holds its batches the same way. Right after startup, custody is known for almost no peer, since a peer’s custody arrives with its metadata/3 answer after it connects. A mainnet follower’s first batch after a fresh checkpoint sync went out 5 s after startup with two peers and at most one known custodian per column, and 121 of the 122 holds it caused were missing every custody column. The batch is re-checked after every metadata answer and every Status answer, and it goes regardless once RANGE_BATCH_CUSTODY_WAIT has passed, because nothing here searches for a custodian of a specific column; past that deadline the uncovered columns go to a couple of peers whose custody is unknown, as before.

The per-block one starts at hold_block_for_columns, called when a fulu block carrying commitments arrives short of the columns this node custodies for it. Import holds the block — it stays out of fork choice, but its header, body and proof are already written, the same way a block missing its parent is held. What happens next depends on the block’s age. A block still at or ahead of the current slot asks for nothing yet: its columns are published alongside it, so the rest are normally already in flight on gossip, and a peer asked at that moment usually does not have them either; on mainnet followers at the tip, gossip completed a held block within 0.3 s at p99. A block already older than the current slot has no such gossip left to race, since whatever delivered it did so long before this node held it, so it sends data_column_sidecars_by_root/1 immediately instead: a range-synced catch-up hits this on every held block, and without it such a follower imported roughly one block per slot, waiting out the redrive below for a request gossip was never going to answer. Either way, redrive_held_blocks runs on every slot tick: for each block still held it sends the same request for exactly the columns still missing, with MAX_FETCH_RETRIES attempts and backoff doubling from INITIAL_BACKOFF_MS, a peer that has already failed this lookup excluded until the whole pool is exhausted. A lookup that runs out of peers or retries stops asking until the next tick asks again; see lean_data_column_fetch_failures_total in metrics.md. The block is released the moment its last missing column arrives, and dropped along with any pending descendants once finality passes its slot, whichever comes first — nothing else times out a hold, so a peer that claims commitments and never answers pins the block only until finality clears it, not indefinitely.

Both asking paths aim per column rather than per peer. A peer’s custody set is a public function of its node id and its advertised custody group count, so this node computes it and sends each column to someone who actually holds it — the spec’s own observation that “due to the deterministic custody functions, a node knows exactly what a peer should be able to respond to”. The count comes from metadata/3, requested once per connection right behind Status, and is seeded from the ENR cgc for a peer this node dialed; a peer that has supplied neither is not assumed to custody anything, and the by-root path falls back to asking one at random for whatever no known custodian covers. At mainnet’s CUSTODY_REQUIREMENT a peer holds 8 of 128 columns, so this is the difference between a request that can be answered and one that usually cannot.

What this node deliberately does not do with its own slice of the matrix: it does not run compute_matrix or recover_matrix to reconstruct the rest of a block’s data from it, since reconstruction needs half of NUMBER_OF_CUSTODY_GROUPS and this node never holds more than its own sampling size; and, over req/resp, it does not cross-seed a verified column to a peer that never asked for it: every sidecar this node sends over data_column_sidecars_by_{root,range} leaves in direct answer to that peer’s own request. Gossip is the one path where the same column is an unsolicited push by design: an Accept verdict (see Gossip above) both hands the sidecar to the chain actor and re-propagates it to every mesh peer, whether or not any of them asked for it.

What is not wired yet

Serving both protocols is live, off the checkpoint-anchored store, and so is asking on both. A request that reaches a node whose data directory is not a beacon one is refused with RESOURCE_UNAVAILABLE, the spec’s own code for a peer “unable to reply to block requests”, where INVALID_REQUEST would blame the asker for a request that was fine. That guard is for a lean directory, not for the ordinary case.

Asking is now driven from two places. A peer’s Status starts a range session, which sends through request_beacon_blocks_by_range; and Handler<FetchBlock> reaches request_beacon_block_by_root through fetch_block_from_peer, which picks its protocol from the wire so a beacon node cannot put a lean-framed BlocksByRoot on its beacon streams.

Both paths now import. A fetched block reaches the chain actor as BlockSource::Sync, keeping every per-block range and root check on the way in, and the by-root path is what resolves a gossiped block’s missing parent.

Range sync is paced by P2PServer::beacon_fetched_through, the highest slot handed to the actor, not by the store’s head. Delivery is a message and import is work, so the store trails a delivered batch by the whole actor mailbox. Driven off the head, every resync tick re-requested the part still draining, which on the live follower meant 11,213 blocks off the wire to import 100, each duplicate paying a hash_tree_root before the store could reject it. Paced off the watermark, the ratio was 1.7:1.

The watermark alone misses gossip, though. Only a range answer advances it, so a follower that had caught up and then imported from gossip kept it at its last range batch, and every peer that connected later was asked again for every block since: 748 blocks in 66 bursts on one follower. A Status answer is therefore measured against the higher of the watermark and the store’s head (beacon_sync_target). The head is a safe floor, since a head block’s whole ancestry is already imported.

Status is store-derived: head from Store::beacon_head, the finalized checkpoint from Store::beacon_finalized_checkpoint, and earliest_available_slot from the anchor’s own slot rather than finalized_epoch * SLOTS_PER_EPOCH, since this node keeps only the unfinalized window above its anchor and the epoch’s first slot is not necessarily one it holds. Naming that slot tells a peer not to ask for anything older; naming zero would claim genesis is in reach. There is one window where beacon_head answers None, between init_beacon seeding KEY_HEAD and fork choice inserting the block that root names, and every field but the fork digest falls back to zero there. That stays honest: lighthouse’s relevance check exempts a zero finalized_root from its finalized-root comparison, reading it as “this peer is syncing” rather than as a conflicting chain.

The ENR

Two of these advertise less, or more, than they look like:

• attnets names exactly the subnets this node subscribed to, which is the only honest value: claiming one it does not serve earns peer-score penalties for silence there, and claiming none while serving two loses the peers looking for precisely that. It used to be all-unset, which was honest while this node held no subscription at all.
• syncnets names the sync committee subnets joined on a validator client’s request, so it is all-unset until one asks. It is advertised in MetaData only; the ENR has no syncnets entry.
• cgc advertises CUSTODY_REQUIREMENT, the floor below which peers may reject a record outright, not sampling_size(CUSTODY_REQUIREMENT), the larger number of columns this node actually custodies, stores and serves (see Data column sidecars). That undersells rather than oversells: a request for more than cgc’s worth of columns still succeeds, since this node holds every column its own node id assigned it at the sampling size, not merely cgc’s worth.

The tcp entry is what makes us discoverable in return: lighthouse’s discovery predicate requires enr.tcp4().is_some() || enr.tcp6().is_some() and applies it as a query filter, so a quic-only record is invisible to it.

Metrics

The four aggregate histograms no longer cover what they used to: gossip validation (committees, all three signatures, the seen caches) runs in p2p now and is folded into lean_beacon_gossip_validation_seconds{kind="beacon_aggregate_and_proof"} instead. decode is p2p’s own decode step; the other three, all in ethlambda-blockchain, now measure only apply_verified_aggregate and the actor’s applied-bits gate, which is why processing reads far lower than it used to. mailbox_wait is still the one that cannot be derived any other way, and it is still the failure mode this path introduces: roughly a thousand aggregates a slot (verified ones only, now) queueing behind block imports arrive too late to move the head while every other timing still looks healthy.

The lean_ prefix is the repo-wide convention and applies here too. Data column sidecar metrics (lean_data_columns_stored_total, lean_data_columns_rejected_total, lean_data_column_kzg_verify_seconds, lean_data_column_fetch_failures_total, lean_blocks_held_for_columns) and the disk-growth gauge for Table::DataColumns (lean_table_bytes{table="data_columns"}) are documented in full in metrics.md rather than repeated here.

ethlambda beacon serves these on --metrics-port, alongside /health and the /debug/pprof heap-profiling routes, through the same start_rpc_server entry point ethlambda node uses: one HTTP call site in run_node serves both chains. It also binds --api-port and mounts the /lean/v0 API, off the empty in-memory Store and the default AggregatorController, SyncStatusController and EventBus this startup path hands it. Those routes answer for a chain that is not running; treat /metrics as the only meaningful HTTP surface of a beacon run until the follower gets an API of its own.

Checking it against the live network

No unit test can assert “peers with mainnet”, so this is the procedure.

openssl rand -hex 32 > /tmp/beacon-node-key
RUST_LOG=info,ethlambda_p2p=debug \
cargo run --profile release-fast -p ethlambda --bin ethlambda -- beacon \
  --node-key /tmp/beacon-node-key \
  --gossipsub-port 9001

gossipsub’s own warnings, chiefly the send-queue-full line a slow peer earns once per heartbeat, are capped at ERROR by default; add libp2p_gossipsub=warn to RUST_LOG to see them.

Within 5 seconds:

Derived the mainnet wire parameters  genesis_time=1606824023 genesis_validators_root=0x4b363db9… epoch=… fork=fulu fork_digest=8c9f62fe
No fork or blob-schedule boundary is scheduled
Custodying data columns  columns=[…]
Backboning attestation subnets  subnets_per_node=2 subnets=[…]
Advertising cgc=4; sync committee subnets are joined only on a validator client's request
Beacon P2P node started  socket=0.0.0.0:9001 fork_digest=8c9f62fe topics=17 columns=8 attestation_subnets=[…]
HTTP server listening  addr=127.0.0.1:5054
Starting discv5 discovery  discovery_addr=0.0.0.0:9002 seeds=17 total_bootnodes=17
Local ENR  enr=enr:-…

The Advertising cgc=… line names what is true of sync committees: this node subscribes to no sync-committee subnet at startup, and joins one only when a validator client asks (see Sync committee subnets).

It publishes only what its Beacon API clients submit (attestations, aggregates, sync committee messages, blocks with their columns, operations), plus relayed validated gossip.

seeds=17 proves the built-in list parsed; a lower number means a bootnode ENR was skipped with a warning. topics=17 proves the subscription set: the 7 global topics, the 8 column subnets this run’s (randomly generated) node id selected, and the 2 attestation subnets the same id selected; columns=8 and attestation_subnets=[…] confirm the two families directly.

Within 30 seconds:

External IP detected via PONG voting, updating local ENR  old_ip=0.0.0.0 new_ip=…
Beacon block decoded  slot=… proposer=… fork=fulu block_root=… bytes=…
Beacon aggregate attestation decoded  slot=… aggregator=… attesters=… target_epoch=… target_root=… bytes=…

A measured run on 2026-08-31, from a laptop behind NAT with no port forwarding, reached its first decoded block 16 seconds after start and its first aggregate at 24 seconds, then took a block every slot. bytes on mainnet is typically 100 KB to 310 KB for a block and about 500 for an aggregate, and attesters sits in the low hundreds. Peer lifecycle lines (Peer connected, Beacon handshake complete) are at trace: at mainnet peer counts they are what drowned the log, so raise the level to see them.

What each failure looks like:

Both transports, not just QUIC

ethlambda used to dial QUIC only, against the quic port a peer’s ENR advertises. Most mainnet beacon nodes are reachable over TCP and advertise a quic entry that does not answer, either because the node is behind a NAT that forwards only TCP or because QUIC is disabled behind an advertised port. The result was a long run of Handshake with the remote timed out against otherwise valid, correctly admitted peers.

The node now listens on and dials both transports, and admission accepts a peer advertising either, so a dead quic entry falls back to TCP within the same dial rather than ending it. The TCP transport offers mplex alongside yamux because mainnet peers answer na to a yamux-only proposal; see the muxers module in crates/net/p2p/src/lib.rs for that measurement.

A node behind NAT with no inbound UDP can only ever be the dialer, and peers fill up: expect to wait for a peer that stays, and expect a first connection to be refused with a Goodbye.

A run where discv5 finds contacts but every one is rejected for a missing eth2 entry means the records reaching the dial loop are not the ones the peers published. discv5 hands back a full NodeRecord per NODES response, so an eth2 entry that was there on the wire and absent here is a record-plumbing problem in the discovery layer rather than anything in admit, which is covered by unit tests against records carrying every entry. Bootnode records are a poor control: mainnet’s advertise the phase0 digest and are supposed to be rejected, just as fork digest mismatch rather than as missing.

The execution layer pairing

ethlambda beacon follows the Ethereum Beacon Chain. A beacon block carries an execution payload that consensus cannot validate on its own: only an execution client can say whether the transactions in it are valid and whether the state root they produce matches. This page describes how the follower asks, what it does with each answer, and what it does not yet do.

The consensus rules implemented here are specs/bellatrix/optimistic-sync.md and the verify_and_notify_new_payload half of each fork’s process_execution_payload. The wire is execution-apis/src/engine.

Which methods, and why so few

Six:

Osaka introduces no new newPayload. Its own document adds only engine_getPayloadV5 and engine_getBlobsV2/V3; engine_newPayloadV5 belongs to Amsterdam, where gloas’s envelope import uses it. So the Osaka-current call for a payload is prague’s V4, and the Osaka-current fork choice notification is cancun’s V3. Amsterdam’s pair (newPayload V5, forkchoiceUpdated V4) is used from gloas on; the startup handshake warns when the execution client does not advertise either, but only on a network whose schedule includes gloas.

Three method families a full client would have are deliberately absent:

• engine_getPayload* outside the gloas path. The follower itself never asks an execution client to start building a block, so its own forkchoiceUpdated calls carry null payload attributes. Block production for a validator client (produceBlockV3 for fulu, produceBlockV4 for gloas) does: forkchoiceUpdatedV3 with PayloadAttributesV3 then engine_getPayloadV5 before gloas, and forkchoiceUpdatedV4 with PayloadAttributesV4 (V3’s fields plus slotNumber and targetGasLimit, the parent bid’s gas limit) then engine_getPayloadV6 (ExecutionPayloadV4, with blockAccessList and slotNumber) from gloas on. The handshake warns when a gloas network’s execution client does not advertise engine_getPayloadV6.
• engine_getBlobs*. There is no blob-pool fetch path here; data columns come from peers over the network.
• engine_getPayloadBodies*. Nothing consumes them.

Blocks before electra are not asked about at all. This node checkpoint-syncs onto a chain far past bellatrix, capella and deneb and never imports one of their blocks, and engine_newPayloadV4 would refuse their payloads as an unsupported fork anyway. Supporting them would mean carrying V1 through V3 as well, for chains this follower cannot reach.

Where the call sits

process_block (beacon arm)
  │
  ├─ data availability gate ────────────► held, if a custody column is missing
  │
  ├─ engine_newPayloadV4  ◄── this page
  │
  └─ fork_choice::on_block(…, payload_validity)
        │
        ├─ state_transition, with the engine derived from the verdict
        └─ record the block's execution hash and its optimistic status

This diagram is the pre-gloas path. A gloas block carries no payload, only a builder’s bid, so nothing is asked about it at import; the question moves to the envelope that reveals the payload (see Gloas below).

The order is the specification’s own: is_data_available runs before state_transition, so a block about to be held for its custody columns is never one this node asks an execution client about.

Every field of the request is a pure function of the block: parent_beacon_block_root is state.latest_block_header.parent_root, which after process_block_header is the block’s own parent_root, and the other three are body fields. That is what lets the round trip happen outside the state transition, so the state transition itself stays synchronous while the import cascade above it is async.

The verdict then reaches state_transition as an ExecutionEngine: an INVALIDATED answer makes verify_and_notify_new_payload return false and the transition fail from inside process_execution_payload, which is where the specification puts that failure. The transition is run even when the verdict is already INVALIDATED, which costs one merkleization on a path that should never run and buys the failure arriving where a reviewer checks this code against the specification.

The verdicts

optimistic-sync.md groups the five wire statuses into three outcomes:

A block in optimistic_roots carries normal fork-choice weight and can be the head. That is the point of optimistic sync, and the sync/optimistic fixture suite proves it: its one case, from_syncing_to_invalid, has a SYNCING branch take the head from a VALID one on attestation weight, and then requires the head to fall back when the SYNCING branch turns out to be invalid.

An invalidated block leaves fork choice by having its LiveChain index row deleted. That is sufficient because Store::block_index() is the only source filter_block_tree, compute_node_weights and walk_head read: a root with no row contributes no weight to any ancestor and can never be walked to. The block and its state stay in their own tables, so an operator can still inspect what was rejected.

That an INVALIDATED block is never imported is enforced on the verdict itself, in on_block, not on the state transition having failed. The transition still runs first, so the failure arrives from inside process_execution_payload where the specification puts it and where the sync/optimistic fixture exercises it. But it only fails there for forks that consult the ExecutionEngine at all: bellatrix gates that step on is_execution_enabled, and phase0 and altair have no such step, so on those a condemned block would otherwise transition cleanly.

Optimistic roots

A block imported on NOT_VALIDATED is recorded in optimistic_roots against its slot. An entry leaves on a later VALID (which clears the whole optimistic prefix, per optimistic-sync.md’s “all ancestors of the block MUST also transition”) or INVALIDATED verdict, and, for the ones that get neither, on finality: an execution client doing a long state sync answers NOT_VALIDATED to every block, so without that bound the set would take one root per import for the life of the process.

Besides mark_validated’s own ancestor walk, the Beacon API reads the set: every execution_optimistic response field (see rpc.md for a gloas block’s), is_optimistic on /eth/v1/node/syncing and the 206 on /eth/v1/node/health (both about the head only), and attestation_data and aggregate_attestation, which answer 503 rather than hand a validator a vote for a block the execution client has not validated (see rpc.md).

is_optimistic_candidate_block

A NOT_VALIDATED verdict is not enough on its own. The block must also either

1. have a parent this node has already recorded an execution hash for, or
2. be at least --safe-slots-to-import-optimistically behind the wall clock.

Condition 1 is store.beacon_el_block_hash(parent_root).is_some(): the specification’s is_execution_block(parent), read off a cache of the parent’s payload hash rather than by decoding the block. The cache holds a hash for every post-merge block the store has, and none for a pre-merge block (a zero payload hash is not an execution block). It is filled at import, for the checkpoint anchor when the store is built, and on resume for the unfinalized window.

The horizon guards against a poisoned merge transition block, whose parent hash names an execution block nobody can produce. Once a node is on a post-merge chain every block’s parent has a recorded hash, so condition 1 carries each one and condition 2 stops mattering.

The anchor is never imported and never asked about (see The anchor is assumed valid), so its hash is cached when the store is built; only the hash, not a verdict. That is what lets the first block after a checkpoint-sync anchor take condition 1 even while the execution client is still syncing and answers SYNCING to every newPayload. A restart empties the cache, so resume rebuilds it from the stored blocks.

latestValidHash

An INVALID answer names the last execution block that was valid. Resolving which beacon block that condemns is a walk up the rejected block’s own ancestry, not a lookup in a global index, because the specification scopes it that way: “the child of a block with body.execution_payload.block_hash == latestValidHash in the chain containing the block with payload in question”. Two branches can share a parent whose payload is the last valid one, and only the branch that was rejected may die.

The last row is the specification’s own instruction, not a convenience: “when latestValidHash is a meaningful execution block hash but consensus engine cannot find a block satisfying body.execution_payload.block_hash == latestValidHash, consensus engine SHOULD behave the same as if latestValidHash was null”. A checkpoint-synced follower meets this whenever the named block is below its anchor.

The finality floor

A condemned root at or below the finalized checkpoint is refused: the log says so and nothing is removed. Obeying it would delete every LiveChain row from finality upward, after which get_head fails its “block_root in store.blocks” check on every call and the node can only report that it cannot compute a head until its database is rebuilt.

The all-zeroes row above is what reaches this without anyone naming a finalized block: the walk it describes stops where the execution hash cache does, and that cache is bounded by finality, so its deepest entry is the finalized block itself. An execution client condemning finalized history means it and this node disagree about what is final, which is an operator emergency rather than something to resolve by emptying fork choice.

Gloas

A gloas block commits to a builder’s bid and the payload arrives later in a SignedExecutionPayloadEnvelope, so the execution client judges the envelope, not the block. The order differs from the specification’s, which asks the engine last inside one boolean function: the follower runs the pure consensus checks first (signature, bid, state, data availability), and only an envelope that passed them is sent to engine_newPayloadV5. That way an unsigned copy with the honest block hash and a different body cannot earn an INVALID that lands on the honest payload, and the engine never spends time on an envelope this node would refuse anyway.

The null-latestValidHash row exists because ethrex has no INVALID_BLOCK_HASH status: it answers a mismatch with a null hash and an executed-and-failed payload with its last valid ancestor. latestValidHash and VALID follow the execution chain (each block’s bid.parent_block_hash), not every verified ancestor, since a block built past an ancestor’s EMPTY node skips that ancestor’s payload. The details, and why each departs from the specification, are in spec_deviations.md.

With the engine down, the per-slot retry asks about the oldest held envelope only and releases the rest once that ask is answered, so an unreachable client costs one retry ladder per slot however many envelopes wait. This is the gloas counterpart of the limitation below, and unlike a block the envelope is not dropped.

forkchoiceUpdated

Sent once after every head recompute, which is once per import cascade and once per tick, not once per block. It is sent even when nothing moved: an execution client doing state sync needs to keep being fed a recent head or its sync cannot converge.

Its three hashes come from the cached execution hash of the head, the justified checkpoint’s root and the finalized checkpoint’s root. A root with no cached hash contributes the zero hash, which is a meaningful value here rather than an absence: EIP-3675 requires finalized_block_hash to be zero before a post-transition block is finalized, and the specification’s own get_safe_execution_block_hash returns zero when no payload is justified yet. A follower whose head has no cached hash is still sitting on its checkpoint anchor and sends nothing at all.

The cache is pruned to the unfinalized window on every tick and every import, with the finalized checkpoint’s own root exempted by name. The slot bound alone does not reach it: a checkpoint is stored as its epoch’s start slot, while its root is the last block at or before that boundary, so a missed proposal at an epoch boundary leaves the finalized block below the bound. Dropping its hash would make every later call carry finalized_block_hash = 0x00..0 and stop the execution client advancing its own finalized block for the life of the process.

For a gloas head the hashes come from bids rather than from a cached payload hash: the head node’s bid.block_hash when it is FULL and its bid.parent_block_hash when it is EMPTY, the finalized block’s bid.parent_block_hash, and the justified block’s bid.parent_block_hash as the safe hash. The last is a fallback, since the specification builds the safe hash on fast confirmation, which the follower does not run. The call is V4 with the node’s custody columns. A head whose payload status has not been computed yet (the first moments after a restart) sends nothing.

The response carries a PayloadStatusV1 of its own, and that is the channel by which a block imported on SYNCING later becomes VALID or is found to be INVALID. An invalidation reaching the follower this way can remove the head itself, unlike one reaching it through newPayload, because the block in question has already been imported and is in the index. When it does, fork choice is re-run and the head rewritten before the response handler returns: KEY_HEAD and Table::BlockRoots still name the removed roots otherwise, and the req/resp handlers read both, so a peer would be advertised and served a block this node has just refused. Only the head is rewritten there, not announced: telling the execution client from inside its own response would re-enter the call being answered, so the new head goes out on the next cascade or tick.

Configuration

--execution-endpoint <URL>                   e.g. http://127.0.0.1:8551
--execution-jwt-secret <PATH>                file holding 32 bytes of hex
--safe-slots-to-import-optimistically <N>    default: the spec's own value

The first two must be given together: an Engine API endpoint always requires authentication, so an endpoint without a secret is refused at startup, before anything binds or connects. Clap cannot express “both or neither” across two optional flags, so the pairing is checked once, in From<BeaconOptions>.

With neither flag, the follower contacts no execution client and every block gets PayloadValidity::NotRequired, which is exactly what it did before any of this existed.

Authentication is HMAC-SHA256 over a JOSE header and a claim set whose only member is iat. Execution clients accept about a minute of skew either way; a fresh token is minted per request rather than cached, because minting is two hashes and a cache would need a clock of its own.

The startup handshake warns rather than refuses when the execution client does not advertise a method this node needs: an execution client that under-reports its capabilities still works, and refusing to start over a handshake would turn a cosmetic mismatch into an outage.

Limitations

The retry ladder gives up, and nothing re-drives it

Each call is attempted ENGINE_MAX_ATTEMPTS times, with a per-attempt timeout of ENGINE_TIMEOUT and a backoff starting at ENGINE_INITIAL_BACKOFF and doubling. An RPC error is not retried: that is the execution client answering, with a refusal but an answer, so asking again would get the same refusal and burn the ladder for nothing.

After the last attempt the block is dropped. optimistic-sync.md requires exactly that much (“a consensus engine MUST NOT import the block and MUST NOT apply it to the fork choice store”), but nothing here re-drives the call afterwards, so a persistently unreachable execution client parks the follower’s head behind the first block it could not ask about, permanently, until a restart.

This was accepted knowingly for this change. The intended fix is a tick-driven re-drive of the set of blocks that got no verdict. lean_engine_no_verdict_total is the metric that says it is happening: a non-zero rate means the follower has stopped following rather than merely slowed down.

optimistic_roots does not survive a restart

The optimistic set lives in memory alongside the rest of the beacon fork-choice scratch and is not persisted. After a restart, blocks that were imported optimistically are no longer marked as such.

This is safe because the set is advisory. It records which blocks have not yet been vouched for, so that a later VALID can clear them and a later INVALID can be attributed; it never gates whether a block stays in fork choice. On restart the execution client is asked again about everything the follower imports from that point on, and forkchoiceUpdated re-establishes the head’s status on the first tick.

The anchor is assumed valid

A checkpoint-synced follower takes its anchor state and block on trust, without asking an execution client about the anchor’s own payload. That is inherent to checkpoint sync, not specific to this change.

The merge transition is untouched

validate_merge_block and the terminal-PoW checks are bellatrix’s, are reached from one place, and were removed outright by capella. Nothing here changes them.

Command line

ethlambda follows one of two chains, selected by a subcommand:

Four subcommands follow no chain. ethlambda replay-era replays real chain history from .era files and checks it against the states stored in them; see era_replay.md for its flags and checks. ethlambda benchmark is the offline benchmarking harness, covering block building (benchmark synthetic, run through make bench) and block import (benchmark import fetch / benchmark import replay). See benchmarking.md for what each workload measures; benchmark import’s flags are in their own section below. ethlambda validator is the validator client: a separate process that holds keys and performs duties against a beacon node over the standard REST API, documented in its own section below. ethlambda slashing-protection moves that client’s EIP-3076 history in and out of its directory, offline; see its section below.

node is the default

clap has no native default subcommand, so the binary rewrites its own argv before parsing. If the first argument is not node, beacon, benchmark, replay-era, validator, help, -h, --help, -V, or --version, then node is inserted ahead of it. The function is inject_default_subcommand in bin/ethlambda/src/command.rs. Every subcommand has to be listed there: a missing one would have node inserted ahead of it and become unreachable.

ethlambda --genesis c.yaml ...              ->  ethlambda node --genesis c.yaml ...
ethlambda node --genesis c.yaml ...         ->  unchanged
ethlambda beacon --gossipsub-port 9001 ...  ->  unchanged
ethlambda benchmark synthetic ...           ->  unchanged
ethlambda --help | --version | -h | -V      ->  unchanged, no injection

This is what keeps every existing caller working with no edit: the Docker ENTRYPOINT, lean-quickstart’s client-cmds/ethlambda-cmd.sh, the Hive lean client shim, preview-config.nix, and the docker run blocks in the devnet-runner skill all pass bare flags.

ethlambda with no arguments at all is left alone, so it prints the subcommand listing rather than a missing-flag error for node.

Common flags

Taken by both subcommands, with the same meaning and the same defaults, from one CommonOptions struct flattened into each.

Three of those have no clap default, because the two chains answer them differently. A single default_value can only say one thing, so the flag carries none and each chain resolves an absent value itself:

Whether discv5 runs is not a common flag. beacon always runs it, since published mainnet bootnode ENRs carry no quic entry and so are not statically dialable; node runs it only with --discovery.enable (see below). Where it runs, it binds DEFAULT_DISCOVERY_PORT (9000) unless --discovery.port says otherwise. The two defaults are one apart because both sockets are UDP; pointing either flag at the other’s port is rejected at startup, before the node touches its data directory. Without discovery neither that check nor the ban on --gossipsub-port 0 applies, since nothing binds the discovery port and no ENR is published.

node flags

On top of the common flags above.

A shadow-integration build adds the --shadow-xmss-* flags; they are absent from a normal build.

beacon flags

Accepted range is CUSTODY_REQUIREMENT to NUMBER_OF_CUSTODY_GROUPS (4 to 128), enforced at parse time rather than clamped: serving a different set than the operator asked for is the failure hardest to notice.

It is not the number of columns custodied. That is sampling_size, the larger of this and SAMPLES_PER_SLOT, so the default still custodies 8 columns and the two only converge once the flag is raised past 8. A node at 128 is a supernode, custodying every column; raising the value costs storage and bandwidth in proportion and makes this node useful to more peers, which on a network where a cgc=4 peer holds 4 of 128 columns is what decides whether a lookup finds a custodian at all.

Changing it changes which columns this node custodies, since the custody set is a function of the node id and the count. Sidecars already on disk belong to the old set: nothing is corrupted, but the node advertises a set it has not finished filling until it backfills the difference.

With neither execution flag, the follower contacts no execution client and imports blocks without validating their payloads, which is what it did before those flags existed. It then answers attestation_data, aggregate_attestation and block production with 503, so a validator client cannot use it. Supplying one without the other is refused at startup: an Engine API endpoint always requires authentication. See the execution layer pairing for what each verdict does and for the limitations that go with the retry ladder.

beacon takes one more flag of its own, and two common flags also mean something specific here.

genesis_validators_root and genesis_time, which the fork digest that keys every gossip topic, the ENR eth2 entry and discv5 admission is computed from, come from the resolved network:

The genesis constants are checked twice at runtime: checkpoint sync verifies the downloaded anchor state against both, and a resumed data directory is checked the same way, so a wrong constant fails at startup rather than following the wrong chain.

beacon therefore takes no genesis config, validator registry, or bootnode file of its own the way node does: those are read off the resolved network instead of being separate operator input. A config.yaml (a directory’s, or a built-in network’s embedded one) is read permissively: absent keys fall back to mainnet’s values, numbers are accepted quoted or bare, and unrecognised keys (on a current config, the heze schedule this build does not claim) are dropped with one warning line naming each. Its PRESET_BASE is checked against the compiled preset; a mismatch is a hard startup error naming the cargo feature that would fix it. Sepolia’s config schedules gloas, which this build follows, so it parses and runs like any other fork. A scheduled fork the build claims but does not follow gets a separate, explicit warning at startup, since the chain follower would stop tracking the chain there regardless of what parsed cleanly; none exists today. The keys this build runs on compile-time constants for (the custody and subnet counts, the MAX_REQUEST_* limits, MAX_PAYLOAD_SIZE, the snappy message domains and MAXIMUM_GOSSIP_CLOCK_DISPARITY) must equal those constants, or startup fails naming each key that differs: the node cannot follow a network that sets them otherwise, and /eth/v1/config/spec would report values it does not use. For a directory, both checks run before its genesis.ssz is decoded, so a directory built for the other preset fails with the preset error rather than an SSZ one.

Resuming a data directory under a config.yaml whose CONFIG_NAME differs from the stored one only logs a warning; the stored name is kept. Any changed chain value (a fork epoch or version, the slot time, PRESET_BASE) is still a hard error.

--checkpoint-sync-url now supplies the beacon anchor: a finalized BeaconState and its anchor block, fetched from a standard Beacon API server and verified against the genesis identity above plus the anchor’s own internal consistency (see checkpoint_sync.md). The full anchor precedence is: a resumable data directory, then --checkpoint-sync-url, then, for a loaded network only, that directory’s own genesis.ssz, then abort. The URL is therefore required on a fresh data directory only for a built-in network: no built-in network carries a genesis state, and anchoring one at genesis would leave the follower replaying a chain that has been live for years. A loaded network’s own genesis state is a legitimate anchor instead, since a freshly started devnet has no checkpoint provider at slot 0 and this is the only way to join one. A directory already anchored from a previous run resumes without the flag either way, the same way node’s does.

--api-port is bound here too, off beacon’s own anchored store: one HTTP call site serves both chains, but it picks the router by the store’s chain, so a beacon run serves the Beacon API under /eth/v1 and /eth/v2 and answers a /lean/v0/... path with a 404. See rpc.md.

What ethlambda beacon does today

It follows the resolved network as a non-validating follower. It anchors a RocksDB-backed store at a checkpoint-synced (or resumed) finalized state, joins the network’s gossip and req/resp, and the chain actor imports blocks from there (phase0 through gloas), runs fork choice, and, with --execution-endpoint, asks an execution client about payloads. It never proposes or attests. See beacon_wire.md for what goes on the wire and how to check a run against the live network.

One entry point, two chains

Both sub-commands run through run_node in bin/ethlambda/src/main.rs. Each parses into a single cli::Options, whose network field is Network::Lean or Network::Mainnet and carries that chain’s own flags. run_node does what is not chain-specific, branches once, and shares the shutdown:

ChainSetup is what the match produces: the wire configuration, the ENR entries that describe it, the store the req/resp handlers answer from, the node-name roster, and a ChainActor saying what to spawn: lean’s validator keys and BlockChainConfig, or beacon’s marker. The operator-supplied half of the discv5 configuration (node key, ports, bootnodes, peer target) is the same on either chain, so it is filled in once below the match; everything after it, including the chain actor spawn and the shutdown, is shared.

beacon::wire_params reads genesis_time and genesis_validators_root off the resolved network (a built-in network’s two constants, or a loaded network’s own genesis.ssz, decoded at whatever fork its own schedule names for epoch 0), derives the wall-clock epoch and fork digest from them, and logs the next boundary that would move the digest. It builds nothing: one build_swarm serves both chains, dispatching on the WireConfig variant for the topics, the req/resp protocol set, the gossipsub seen_ttl, the identify version and the connection limits.

Both chains run a chain actor, so RunningNode carries a plain BlockChain.

validator flags

ethlambda validator is not a node. It binds no libp2p port, opens no chain database, runs no discovery and follows no chain: it holds validator keys and performs duties against a beacon node’s standard REST API. It works against any conformant beacon node, this repository’s beacon subcommand or another implementation’s, so none of the common flags apply to it.

Slashing protection lives in --validators-dir/slashing_protection.json, locked to one process, so --validators-dir must be writable: a read-only mount of the keys (a Kubernetes Secret, a :ro volume) fails at startup, as does a record that is a symlink (say, onto another volume) rather than a regular file, since a dangling one would read as “no record”; and the directory should be a local disk, since every signature waits for the record to be flushed to it, and the lock that keeps a second process off it is an flock, which a network filesystem may not honour. Unix only. A key with no record is refused at startup unless --init-slashing-protection is passed. A key moving here from another client brings its history with slashing-protection import, or in the same keymanager API request that imports the key. See Spec Deviations.

What it does today

Attestations, block proposals and attestation aggregation, and from gloas on the self-built payload envelope and the payload timeliness committee vote.

Each epoch it resolves its validators’ indices, fetches their attester duties for this epoch and the next, fetches this epoch’s proposer duties, subscribes to the committee subnets the attester duties need, and registers each validator’s fee recipient with the beacon node. The subscription also tells the node which committees this client will aggregate for, which it has to know in advance so it can collect the votes.

Each slot it wakes at the boundary. If one of its validators proposes that slot, it signs the RANDAO reveal, asks the beacon node for a block, checks that the block is for the slot and proposer it asked about, signs it and publishes it. One third into the slot it fetches the attestation data, signs for every validator due that slot, and submits the batch. Two thirds in, for any duty it was selected to aggregate, it asks the node for the aggregate covering that committee’s votes on the data it just signed, wraps it with the selection proof and publishes it.

Aggregation is not a choice. A validator signs the slot under a dedicated domain, and whether the hash of that signature divides evenly by a modulus from the committee’s size decides it. Signatures are deterministic, so a validator gets one answer per slot, cannot search for a better one, and cannot decline: the beacon node checks the same thing when the aggregate arrives.

Proposals are bounded by that one-third mark rather than by the end of the slot. The specification defines no block-production deadline; the nearest thing it defines is the point attesters stop waiting for a block, which is the same instant. A proposal still running then has already lost most of the block’s value, and every second past it comes out of this client’s own attestations, so it is abandoned and the slot’s attesters still vote.

Electra is the earliest fork it will propose under. A deneb block body has one field fewer than an electra one, so it would need a container pair of its own, and it would buy nothing: the attestations this client submits are electra’s SingleAttestation, which has no earlier form, so a pre-electra chain is one it cannot serve whatever it does with blocks. A block from any earlier fork is refused by name rather than failing to decode.

Blocks are fetched and published as SSZ rather than JSON. Signing a block means computing its hash_tree_root, which only the typed container can give; a hand-written JSON mapping of an execution payload would be a large surface on which a single wrong field silently produces a signature over the wrong block.

From gloas on a proposal is two messages. The client asks for the block with include_payload=true (produceBlockV4), publishes the block bare, then signs the envelope with the proposer’s own key under the builder domain and publishes it with its blobs and cell proofs (Eth-Blob-Data-Included: true). Both go out as SSZ. The envelope shares the proposal’s budget (the attester offset, below), so a proposal that overruns it is abandoned wherever it is. A failure to publish the envelope is logged at error and counted, and does not fail the proposal. A block that commits to a builder’s bid is left alone.

Each epoch it also fetches the payload timeliness committee duties for the validators it holds, for this epoch and the next when they are gloas ones. A member sleeps until PAYLOAD_ATTESTATION_DUE_BPS of the slot, asks the node for the payload attestation data, signs it under the PTC domain and submits it. A 204 (the node knows no block for the slot) means no vote. So a gloas slot runs in this order: propose at the boundary, attest at ATTESTATION_DUE_BPS_GLOAS, aggregate at AGGREGATE_DUE_BPS_GLOAS, then the committee vote. The attestation and the committee vote are each bounded by the end of the slot.

Sync committee duties run beside the attester and committee work, not after it. Each epoch the client fetches its sync duties for the current period and the next (POST /eth/v1/validator/duties/sync/{epoch}, asked at the period’s first epoch) and posts sync_committee_subscriptions for the current period’s seats, until the period’s end, so the node joins the subnets. The next period’s are added once its boundary is within four epochs. A member sleeps until SYNC_MESSAGE_DUE_BPS (SYNC_MESSAGE_DUE_BPS_GLOAS from gloas), reads /eth/v1/beacon/blocks/head/root, signs it and submits one message per validator. It refuses to sign when the node reports the head optimistic. A member selected by the selection-proof hash for a subnet then waits for CONTRIBUTION_DUE_BPS (CONTRIBUTION_DUE_BPS_GLOAS), fetches the node’s contribution for that subnet and publishes it wrapped and signed. A subnet whose contribution cannot be fetched is skipped without stopping the others. Duties for wall slot S are those of the period containing S + 1, since members assigned to slot S sign for S - 1. Both halves are bounded by the end of the slot.

Not implemented: the builder flow and blinded blocks (the client asks for an unblinded block and refuses a blinded one), voluntary exits, doppelganger protection and remote signing.

Duty offsets come from the network

The slot length and the duty offsets are read from the beacon node’s /eth/v1/config/spec, not divided out of a compiled-in constant. The specification states them as SLOT_DURATION_MS plus basis points of it, ATTESTATION_DUE_BPS and AGGREGATE_DUE_BPS, which on mainnet’s 12-second slot work out at 3999 ms and 8000 ms. Gloas moves both earlier (ATTESTATION_DUE_BPS_GLOAS, AGGREGATE_DUE_BPS_GLOAS) and adds PAYLOAD_ATTESTATION_DUE_BPS (and PAYLOAD_DUE_BPS, which the node uses), so the clock picks the pair by the fork of each slot’s own epoch. The sync committee offsets (SYNC_MESSAGE_DUE_BPS, CONTRIBUTION_DUE_BPS and their _GLOAS variants) are read and picked the same way.

SECONDS_PER_SLOT no longer exists in the specification and is accepted only as a fallback, since deployed nodes still send it. A node sending both is required to agree with itself.

Proposer duties are less durable than attester ones

Worth knowing if a proposal is ever missed after a reorg. An attester schedule depends on the block two epochs back and survives anything shallower; a proposer schedule depends on the block one epoch back, so a reorg that leaves attester duties untouched can still move a proposer between slots. This client fetches proposer duties once per epoch, at the boundary, so a reorg inside the epoch is not picked up until the next one.

The keymanager API

Off unless --enable-keymanager is passed. It serves GET, POST and DELETE /eth/v1/keystores, authenticated with a bearer token read from api-token.txt in the validators directory and generated there on first start if absent. The token file is written 0600 and a file too short to be a real token is refused rather than accepted.

It also serves GET, POST and DELETE on /eth/v1/validator/{pubkey}/feerecipient, /graffiti and /gas_limit, which give one validator its own value in place of the command line’s default:

These are held in memory only. A restart puts every validator back on the defaults, so an operator driving these endpoints must set them again after one. Deleting a keystore drops its values too.

A few answers worth knowing: GET feerecipient is a 500 when the validator has no address and there is no --suggested-fee-recipient default, as Lighthouse answers it; POST feerecipient refuses the zero address, as the specification requires; POST graffiti refuses text over 32 bytes rather than truncating it; and every route answers 404 for a key this client does not hold.

The keystore routes carry slashing-protection history. POST refuses every key in the request if its slashing_protection does not parse or is for another chain, and otherwise writes that history before any key is saved, so a key never loads ahead of it; a validator the history names without a keystore in the request gets a record too. DELETE returns the deleted keys’ records as a minimal EIP-3076 interchange, answering not_active for a key that is gone but still has a record.

The specification requires TLS. This binds to --http-address, loopback by default; exposing it further means putting a TLS terminator in front.

Metrics

Served on --metrics-port, prefixed ethlambda_validator_ rather than the lean_ used elsewhere in this repository, since this process follows the beacon chain. Every series is registered at startup so it reads zero rather than being absent before its first event, which is what lets an alert fire on “attestations stopped”. See Metrics.

slashing-protection

Moves the validator client’s EIP-3076 history in and out of its --validators-dir, offline. It takes the lock the client holds while running, so it refuses while a client runs on that directory, and it needs no beacon node: the record names its own chain.

# Bring a key's history from another client before its first start here.
ethlambda slashing-protection import --validators-dir ./validators interchange.json

# Hand the history over before running the keys elsewhere.
ethlambda slashing-protection export --validators-dir ./validators out.json
ethlambda slashing-protection export --validators-dir ./validators --pubkeys 0xab..,0xcd.. out.json

benchmark import flags

Neither sub-command follows a chain; see benchmarking.md for what the workload measures and why it exists.

benchmark import fetch

benchmark import replay

HTTP API

ethlambda exposes HTTP over two independent Axum servers on separate ports, so the API and the metrics/debug surface can have different network policies:

• API server — consensus data (blocks, states, checkpoints, fork choice) and admin controls.
• Metrics & debug server — Prometheus metrics and heap-profiling endpoints. No store access.

Which API server a node serves follows from its store’s chain tag, not from the sub-command: ethlambda node serves the lean surface under /lean/v0, ethlambda beacon serves the Beacon API under /eth/v1 and /eth/v2. The two are alternatives, never merged, because the lean handlers read state variants and metadata keys a beacon directory does not carry. Roots are serialized as 0x-prefixed hex strings on both.

Servers & Ports

If --api-port and --metrics-port are equal, all routers are merged onto a single port.

API Server (:5052)

GET /lean/v0/health

The handler emits a fixed, compact body (no whitespace):

{"status":"healthy","service":"lean-rpc-api"}

GET /lean/v0/config/spec

Protocol parameters the node is running on. Keys mirror the leanSpec constant names. MILLISECONDS_PER_SLOT and MILLISECONDS_PER_INTERVAL reflect the network’s config file rather than a compile-time constant, so a node on an 8-second network reports 8000 and 1600 here:

{
  "MILLISECONDS_PER_SLOT": 4000,
  "INTERVALS_PER_SLOT": 5,
  "MILLISECONDS_PER_INTERVAL": 800,
  "HISTORICAL_ROOTS_LIMIT": 262144,
  "FORK_DIGEST": "12345678"
}

FORK_DIGEST is the 4-byte hex string (no 0x prefix) embedded in gossipsub topic names.

GET /lean/v0/genesis

{ "genesis_time": 1770407233, "validator_count": 16 }

validator_count is read from the head state’s validator registry. Lean validators are fixed at genesis (no churn), so it always equals the size of the genesis registry.

GET /lean/v0/states/finalized

SSZ-encoded State at the latest finalized checkpoint (Content-Type: application/octet-stream). The served state has its latest_block_header.state_root zeroed to match the canonical post-state representation the state transition produces, so checkpoint-sync peers reconstruct an identical state root. See Checkpoint Sync.

GET /lean/v0/blocks/finalized

SSZ-encoded SignedBlock at the latest finalized checkpoint. The genesis/anchor block has no stored signature, so a placeholder blank proof is synthesized and the endpoint still returns 200. Returns 404 only in the rare case where a non-genesis finalized block’s signature has been pruned below the finalized boundary and can no longer be served.

GET /lean/v0/checkpoints/justified

{ "slot": 128, "root": "0x1a2b…" }

GET /lean/v0/events

Server-Sent Events stream (Content-Type: text/event-stream) of live chain events published by the blockchain actor. Eight event types:

Payload fields mirror the Ethereum beacon-API eventstream where an analog exists: block is the block root, state the state root, and slot stands in for the beacon epoch. justified_checkpoint and aggregate are ethlambda extensions with no beacon topic.

The topic name travels only on the SSE event: line; the data: line carries the flat JSON payload. Example frame:

event: head
data: {"slot":128,"block":"0x1a2b…","state":"0x3c4d…"}

Filtering with ?topics=

A required comma-separated list of event names selects which events to stream:

curl -N 'http://127.0.0.1:5052/lean/v0/events?topics=head,finalized_checkpoint'

Valid values are exactly the event names above: head, block, justified_checkpoint, finalized_checkpoint, block_gossip, attestation, aggregate, chain_reorg. A Beacon API topic this surface does not serve (data_column_sidecar, say) is refused as unknown. As in the Beacon API eventstream endpoint, topics is mandatory: there is no “subscribe to everything” default; list the topics you want.

Events are fanned out over a single bounded broadcast channel shared by all topics. A client that reads too slowly skips past the events it missed: they are dropped for that subscriber rather than back-pressured onto the actor, so treat the stream as best-effort and re-sync via the blocks endpoints after a gap. A client that falls behind receives an SSE comment line : error - dropped N messages marking the gap (wire-compatible with Lighthouse) before the stream continues; re-sync via the blocks endpoints rather than trusting the skipped range. Keep-alive comments are sent periodically to hold idle connections open.

Because the ring buffer is shared, the high-rate attestation events (roughly one per validator per slot) dominate its occupancy: a subscriber’s tolerable stall is capacity / total_event_rate, not per-topic, so filtering with ?topics= narrows what you receive but does not widen the lag window against an attestation flood. Subscribers that only need low-rate topics (head, finalized_checkpoint, …) are still evicted at the aggregate rate. If real usage shows this biting, the fix is a per-topic channel split behind the event bus (the subscribe(TopicSet) API is unaffected).

GET /lean/v0/blocks/{block_id} and /header

block_id is either:

• a 0x-prefixed 32-byte hex root, or
• a decimal slot.

Slot lookups resolve through the head state’s historical_block_hashes, so only canonical blocks are reachable by slot; blocks on side forks must be addressed by their root. The /header variant returns just the BlockHeader.

Error bodies are JSON: { "error": "invalid block_id" } / { "error": "block not found" }.

GET /lean/v0/fork_choice

The fork-choice tree from the finalized root, with LMD-GHOST weights computed over the live chain and currently known attestations.

{
  "nodes": [
    { "root": "0x…", "slot": 128, "parent_root": "0x…", "proposer_index": 3, "weight": 12 }
  ],
  "head": "0x…",
  "justified": { "slot": 128, "root": "0x…" },
  "finalized": { "slot": 96,  "root": "0x…" },
  "safe_target": "0x…",
  "validator_count": 16
}

/lean/v0/fork_choice/ui serves an interactive D3.js page rendering this data. See Fork Choice Visualization.

GET /lean/v0/node/identity

{
  "version": "ethlambda/v0.1.0-main-892ad575/x86_64-unknown-linux-gnu/rustc-v1.97.1",
  "peer_id": "16Uiu2HAm7v1x…"
}

version is the full client version string, identical to what ethlambda --version prints: crate semver, git branch and short SHA, target triple, and rustc version. Baked in at compile time from CARGO_PKG_VERSION plus the vergen-git2 build metadata.

peer_id is the node’s libp2p peer ID (base58), derived from the node key and fixed for the lifetime of the process; it matches the identity the node presents to peers on the wire.

GET /lean/v0/node/syncing

{ "is_syncing": false, "head_slot": 1024, "sync_distance": 1, "finalized_slot": 986 }

is_syncing is the node’s own stateful sync decision: head-vs-wall-clock lag with hysteresis and a network-stall override, updated each tick. It is the same signal that gates validator duties and drives the lean_node_sync_status metric, so the endpoint, the gate, and the metric always agree.

sync_distance is the raw number of slots between the node’s current head and the current wall-clock slot, computed per request. Because is_syncing carries hysteresis and stall handling and is not recomputed from sync_distance, the two can point different ways near the threshold or during a network-wide stall.

GET / POST /lean/v0/admin/aggregator

Toggle the aggregator role at runtime without restarting the node (hot-standby model, ported from leanSpec PR #636).

# Read current role
curl http://127.0.0.1:5052/lean/v0/admin/aggregator
# → {"is_aggregator": true}

# Toggle role; body must be a JSON boolean
curl -X POST http://127.0.0.1:5052/lean/v0/admin/aggregator \
  -H 'content-type: application/json' -d '{"enabled": false}'
# → {"is_aggregator": false, "previous": true}

Note: Runtime toggles do not resubscribe gossip subnets, which are frozen at startup. A standby aggregator should boot with --is-aggregator=true (so subscriptions are in place), then use this endpoint to rotate duties. See the CLAUDE.md “Runtime Aggregator Toggle” notes for the operational model.

Beacon API Server (:5052, on ethlambda beacon)

The subset of the Ethereum Beacon API that this node can answer from its own store. It replaces the /lean/v0 surface rather than sitting beside it; a /lean/v0 path on a beacon node is a 404.

Validator endpoints

These are what ethlambda validator needs to attest through this node. Every answer is computed from the fork-choice head’s post-state, read off the store the chain actor writes, so no request waits on the actor.

• Duties are bounded by the wall clock, as the Beacon API defines it: any epoch up to one past the current one is served (or one past the head’s, if the head is ahead of a lagging clock), so a validator client’s next-epoch lookahead at an epoch boundary, while the head is still in the previous epoch, is answered. Proposer duties (v1 and v2) read fulu’s proposer_lookahead, which covers the head’s epoch and the next; attester duties read the head state as it is for its previous, current and next epoch. A later epoch is computed from a copy of the head advanced with process_slots to the first epoch that can derive it, taken from fork choice’s checkpoint_state cache and run on a blocking thread. An epoch before the head’s (attester: more than one before) or past the bound is a 400. dependent_root follows each endpoint’s own definition: the last block before the epoch for v1 proposer duties, and the last block before the previous epoch for v2 proposer duties and attester duties. v2’s is the block fulu’s lookahead depends on, so it changes only on reorgs that can change the duties. Proposer duties v2 is a 503 while syncing. Attester duties walk every committee of the epoch, a full shuffle per request on mainnet. Gloas epochs are served like fulu ones: gloas keeps fulu’s proposer_lookahead and upgrade_to_gloas carries it over, so a fulu head already answers proposer duties (v1 and v2) for the first gloas epoch, and a gloas head answers from its own lookahead.
• Sync duties read the head state’s current_sync_committee for an epoch in the head’s own sync committee period and next_sync_committee for the one after. The upper bound is the wall clock’s, as the Beacon API defines it: the period after the clock’s current one. When the head lags a period boundary (its block is late or missing), the later period is read from a copy of the head advanced to the first epoch of the period before it, through fork choice’s checkpoint-state cache on a blocking thread. A period past that bound or before the head’s is a 400 (an earlier one would need a historical state; see docs/spec_deviations.md). A validator is matched by pubkey and gets every seat it holds, since the committee is drawn with replacement; one with no seat is left out. An unknown index is a 400, and the endpoint is a 503 while the node is syncing. What a duty asks for is served by the four sync committee endpoints in Sync committee below.
• Liveness is this node’s own view, which the Beacon API allows. A validator is live in an epoch if the head state credits it for that epoch (a non-zero participation byte, so anything a block already included), or the node observed it act: an accepted gossip aggregate (its aggregator and every attester its signature verified) or subnet attestation, an imported block’s proposer, or a submission through pool/attestations or aggregate_and_proofs. The observed half covers what no block has included yet, most of the current epoch; it keeps the newest three epochs recorded, as one bitset each. The window is the store clock’s previous, current and next epoch (the next is always false); anything else, and an unknown index, is a 400, and the endpoint is a 503 while the node is syncing.
• attestation_data follows phase0’s validator.md: the head block, the epoch’s boundary block as target, and as source the current justified checkpoint of the head state advanced to the slot’s epoch (through fork choice’s cached checkpoint_state, and only when the head is in an earlier epoch). A slot before the head, or past the wall clock, is a 400. An optimistic head is a 503: the Beacon API requires one whenever the answer’s beacon_block_root has not been validated by the execution client, and the head is always that root here. So is every request on a node run without an execution client, whose blocks are never marked optimistic because nothing validates them. committee_index is optional and ignored: from electra on every committee of a slot attests to the same data. data.index is 0 before gloas. At a gloas slot it is the payload-present flag: 0 when the head block is from the requested slot itself (its payload cannot be revealed yet), otherwise 1 exactly when fork choice holds the head block’s payload as FULL (Store::head_payload_status, or one fresh get_head_node walk when none is recorded for this head). A pre-gloas head counts as FULL (see Spec Deviations).
• states/{state_id}/validators/{validator_id} is one entry of the list endpoint, in the same shape and with the same state_id handling, execution_optimistic and finalized. The id is an index or a 0x pubkey; one naming no validator is a 404 (the list form omits it), a malformed one a 400. Lighthouse’s validator client resolves each key to its index this way, and stays inactive without it.
• pool/attestations checks each attestation against the electra beacon_attestation_{subnet_id} gossip conditions it can evaluate (clock window, data.index == 0, target epoch, the voted block known and the target its checkpoint block, committee membership, BLS signature), then gossips it on its subnet through fanout, without subscribing. A rejected attestation comes back in an IndexedErrorMessage with its position; the valid ones in the same batch are still published. There is no seen-attestation cache. Each accepted attestation also goes into the node’s attestation pool, since gossip never delivers a node its own messages. Eth-Consensus-Version must name electra, fulu or gloas here and on aggregate_and_proofs, and must match the fork of each item’s slot: a gloas header on a pre-gloas slot, or an older one on a gloas slot, fails that item (not the batch) with Eth-Consensus-Version does not match the attestation slot's fork. At a gloas slot data.index may be 0 or 1, checked with gloas’s payload-status gossip rule (a vote in the voted block’s own slot must be 0; 1 needs the node to have verified the payload, otherwise the item fails). A gloas SingleAttestation is electra’s own container.
• beacon_committee_subscriptions: each aggregator’s entry makes the node join its committee’s attestation subnet until the end of that slot, so the committee’s votes from other validators reach the pool too: every attestation this node relays has passed the beacon_attestation_{subnet_id} checks first, and the ones accepted on a joined subnet are pooled. Joined subnets are left once their slot has passed, and never appear in attnets.
• aggregate_attestation answers from the pool: every vote held for the data root and committee, as electra’s Attestation with the BLS aggregate of their signatures; at a gloas slot the answer is {version: "gloas", data} with gloas’s Attestation (the same JSON shape). JSON only. 404 when nothing is held, and 503 when the block the votes name is optimistic, whether or not it is the head, or when no execution client is configured.
• aggregate_and_proofs checks each aggregate with the same beacon_aggregate_and_proof gossip conditions this node applies to its peers’ aggregates (gossip::aggregate), signatures included, against a fresh seen-cache (a node never receives its own messages, so P2P’s says nothing about them). What passes is gossiped on the topic, goes into the pool, and is handed to the chain actor, which applies it to fork choice and announces it on /eth/v1/events’s attestation. The Eth-Consensus-Version header picks the decoder: gloas takes gloas’s SignedAggregateAndProof, the others electra’s.
• Request body encodings. pool/attestations, pool/payload_attestations and aggregate_and_proofs take either JSON (application/json, also the reading of a request with no Content-Type) or the SSZ List[...] of the same items (application/octet-stream): the offset-table encoding every variable-size list uses, which is what prysm sends for aggregates and payload votes, with a JSON retry only on a 415, and what nimbus can send for attestations. Any other content type is a 415, and a body that does not decode in the type’s encoding a 400. POST beacon/blocks and POST beacon/execution_payload_envelopes choose the same way (below); the other validator-client submissions are JSON.
• The attestation pool holds, the best-covered per data root and committee: votes from pool/attestations and the aggregator subnets, aggregates from aggregate_and_proofs, and every electra gossip aggregate P2P accepts (gloas aggregates included, in gloas’s container), once all three of its signatures have verified. Gossip aggregates are pooled on arrival, so a slot’s aggregates are there when the next slot’s block is asked for. Entries more than an epoch old are dropped once a slot.
• prepare_beacon_proposer records each validator’s fee recipient, in memory (a validator client repeats the call every epoch).
• blocks/{slot} advances the head state to the slot and asks the node’s own execution client to build on the head (forkchoiceUpdatedV3 with payload attributes, then getPayloadV5), with the proposer’s fee recipient. The body packs the pool’s best aggregates (committees voting alike merged into one EIP-7549 attestation, up to MAX_ATTESTATIONS_ELECTRA). A candidate is packed only if its target root is the advanced state’s own block root for that epoch and its aggregate signature verifies against that state, so an aggregate made on another branch cannot fail the whole block. The body also votes the state’s own eth1_data, and carries the operation pool’s slashings, exits and BLS changes, packed by pack_operations (each kept only if its process_* succeeds on a scratch copy of the state with everything packed before it applied, up to the preset’s maximums), and the sync aggregate described under Sync committee (empty when nothing verifiable is pooled). The payload bundle’s KZG commitments go in the body. The state root comes from running the block through process_block. The answer is fulu BlockContents: the block, the bundle’s cell proofs and its blobs, with Eth-Execution-Payload-Blinded: false; there is no builder flow. It is a 503 without a configured execution client, or when the bundle is malformed (its commitments, proofs and blobs disagree in count). A slot the schedule does not place at fulu (gloas included) is a 400, checked before the execution client; produceBlockV3 is the fulu endpoint, and a gloas slot goes to produceBlockV4.
• Graffiti is the validator client’s, with both clients’ codes and commits appended as far as they fit: hello RH1a2bLA3c4d for reth at 1a2b… under ethlambda at 3c4d…. The execution client’s half comes from engine_getClientVersionV1, asked alongside the payload build and given up on after EL_VERSION_TIMEOUT, and is left out when it cannot be had (hello LA3c4d). The suffix shrinks rather than cutting the validator’s text short, in Lighthouse’s tiers, and is appended to every block with no opt-out; the graffiti_policy parameter Lighthouse’s validator client sends is not read. LA is ethlambda’s own code: identification.md reserves none for it and lets an unlisted client pick any two letters no listed client uses.
• POST beacon/blocks takes JSON or SSZ, by Content-Type as above, and the Eth-Consensus-Version header (fulu or gloas, else 400; any other content type is a 415). The JSON is the specification’s: lodestar’s and nimbus’s validator clients post it, teku’s falls back to it, and lighthouse and prysm post SSZ. Both reach the same checks after decoding. Fulu: a SignedBlockContents. A slot the schedule does not place at fulu is a 400, checked before any state is advanced. The block is then checked against its parent’s state advanced to its slot: an unknown parent, a slot not after the parent’s, the wrong proposer or a bad proposer signature is a 400. Then every blob’s cells are computed and all cell proofs batch-verified on a blocking thread (400 on any failure, nothing gossiped); the time that takes is lean_beacon_publish_data_columns_seconds. Then the block is gossiped on beacon_block, then, if it carries blobs, all 128 data column sidecars, each on data_column_sidecar_{compute_subnet_for_data_column_sidecar(index)}, and last the chain actor gets this node’s custody columns, then the block, to import. Gloas: a bare gloas::SignedBeaconBlock (a gloas block carries no payload or blobs, so no sidecars go with it; they follow in the envelope). The slot must be scheduled at gloas, the parent must be held, the slot must be after the parent’s, and the proposer and signature are checked against the parent’s state advanced to the slot (all 400 otherwise), before the block is gossiped and handed to the chain actor.
• The operation pool holds, beside the attestation pool and in memory in the same Store (Store::operation_pool): proposer slashings (one per proposer), attester slashings (one per hash_tree_root), voluntary exits and BLS changes (one per validator), first one wins. Entries come from the four pool/* POST routes and from gossip P2P accepts, so each has passed the topic’s gossip rules; the pool never checks a signature. Entries the head state has made pointless are pruned.
• pool/proposer_slashings, pool/attester_slashings, pool/voluntary_exits and pool/bls_to_execution_changes: GET lists the pool. POST runs the specification’s validate_*_gossip rule for the topic against the head’s post-state as it is (consensus-specs v1.7.0-beta.1; no process_* and no state advance, and an attester slashing’s range and slashability checks run before its BLS checks), pools the operation and gossips it on the topic. A rejected operation is a 400. bls_to_execution_changes takes an array: on partial failure the response is a 400 listing the failures by index, and the valid entries are still pooled and published.

Gloas block production

A gloas proposer signs two things on their own, the block and the envelope that reveals its payload, and this node serves both halves for a block it builds itself. It may instead build on a builder’s bid it holds in the shared builder market (see Builder market); then the builder, not this node, reveals the payload. The builder-API path (a bid requested from a builder’s URL through the BuilderConfig’s builders entries) is not implemented: those entries are decoded and not consulted.

POST /eth/v4/validator/blocks/{slot} (produceBlockV4):

The node advances the head state to the slot, and decides which parent payload to build on with should_build_on_full over the payload status fork choice recorded for the head. It then asks its execution client to build (forkchoiceUpdatedV4 with PayloadAttributesV4, then getPayloadV6). The fee recipient and gas target of that build come from the proposer’s signed ProposerPreferences for the slot when the market holds them under the slot’s dependent root, else from prepare_beacon_proposer (fee recipient, or the zero address with a warning) and the parent bid’s gas limit. The body packs the attestation pool’s best aggregates and the payload attestation pool’s votes for the parent block (an aggregate that does not verify against the advanced state is dropped, since one bad operation fails the block), and the state root comes from running the block through process_block. The bid is a zero-value self-build bid read off the built payload. What was built is cached by (slot, block root), for the current and previous slot only.

Choosing between the local payload and a bid. The bids considered are the market’s for (slot, head root, parent payload hash) with this build’s prev_randao, packable into a block on the advanced state, and paying the preferences’ fee recipient when those are held. The best bid is the highest whose value + execution_payment is at least the config’s min_bid. It wins iff builder_boost_factor * bid_gwei > floor(local_value_wei / 10^7) (the specification’s weighting of the local value by 100, in integers), and the local payload wins a tie, so a factor of 0 prefers it and 2^64 - 1 prefers the bid. An engine that sets shouldOverrideBuilder keeps the local payload. Without a local payload (no engine, or the build failed) the best bid is taken whatever its weight. A bid-won block caches nothing, so the envelope GET below answers 404 for it, and POST /eth/v1/beacon/execution_payload_envelopes refuses a proposer-signed (self-build) envelope for it because the builder index differs from the bid’s. The builder’s own envelope goes through that endpoint as for any block.

GET /eth/v1/validator/execution_payload_envelopes/{slot}/{beacon_block_root} serves the cached unsigned envelope, for a client that asked for the block with include_payload=false: {version: "gloas", data} as JSON, or the SSZ ExecutionPayloadEnvelope on Accept: application/octet-stream, with Eth-Consensus-Version: gloas. 404 when nothing is cached for the pair (unknown, or older than the previous slot); 400 for an unparseable slot or root.

POST /eth/v1/beacon/execution_payload_envelopes takes the signed envelope, as JSON or SSZ by Content-Type (any other is a 415; teku’s validator client posts JSON, nimbus posts the bare envelope as JSON and the contents as SSZ, lighthouse posts SSZ, prysm posts SSZ and retries as JSON on a 415), with Eth-Consensus-Version: gloas and Eth-Blob-Data-Included (both required, else 400):

• true: the body is SignedExecutionPayloadEnvelopeContents (the signed envelope, kzg_proofs, blobs).
• false: the body is the bare signed envelope, and the blobs and proofs come from the production cache. 400 when none is cached and the block’s bid commits to blobs.

The envelope’s block is waited for, polling every 50 ms for up to 4 seconds, since a validator client publishes the envelope right after the block returns, which is before the chain actor has imported it; a block still unknown after that is a 400 (unknown block). The envelope must fulfill the block’s bid (builder index and block hash), the blobs must match the bid’s commitments, the signature must verify under DOMAIN_BEACON_BUILDER against the block’s post-state, and every data column built from the blobs must verify against the commitments and cell proofs; any failure is a 400. A success is 200 once the envelope and all NUMBER_OF_COLUMNS gloas data column sidecars are handed to P2P, which gossips them together. A node that does not subscribe to a column’s subnet publishes through gossipsub fanout.

Builder market

Three endpoints serve the gloas builder market. Bids and preferences go through the same rules as the execution_payload_bid and proposer_preferences gossip topics and are held in one shared builder market that p2p also fills, so a message accepted from either side is seen by both.

• POST /eth/v1/beacon/execution_payload_bids takes one SignedExecutionPayloadBid, as JSON (or no Content-Type) or SSZ (application/octet-stream, at most 196,932 bytes; any other type is 415). Eth-Consensus-Version is optional and must be gloas when present. An identical bid already pooled is 200 and is not republished. Otherwise the gossip rules run (a known parent, the preferences for the slot and their fee recipient and gas limit, a known parent payload, an active funded builder, the signature), and a bid that is anything but accepted is a 400 whose message is "{outcome}: {reason}" (for instance reject: bad_signature or ignore: preferences_unseen), because there is no status for a valid bid this node would not relay. An accepted bid is pooled for block production and gossiped. 500 when the network actor is down.
• POST /eth/v1/validator/proposer_preferences takes a list of SignedProposerPreferences (at most (MIN_SEED_LOOKAHEAD + 1) * SLOTS_PER_EPOCH, 64 on mainnet), as a JSON array or an SSZ list. Eth-Consensus-Version is optional and may be fulu or gloas, since a validator client submits during the epoch before the fork. Each entry runs the gossip rules in order; an entry already cached, identical, succeeds without a republish. Accepted entries are cached (the first per (proposal slot, dependent root) wins) and gossiped. If any entry fails the answer is 400 {code, message, failures: [{index, message}]} and the others were still published. The endpoint is not gated on the sync status. A node whose dependent block is unknown answers ignore: unknown_block for that entry.
• POST /eth/v1/beacon/states/{state_id}/builders takes an optional JSON body {ids?: [index | 0x pubkey], statuses?: [pending | active | exited]} (empty or absent selects everything) and answers {execution_optimistic, finalized, data: [{index, status, builder}]} in registry order, integers quoted. An id naming no builder is omitted. exited is a set withdrawable_epoch, else active per is_active_builder, else pending. 400 for a malformed body, id or status, or a pre-gloas state; 404 for an unknown state. The Beacon API defines no GET form.

POST /eth/v1/validator/builder_preferences is not served (404).

Sync committee

All four endpoints are JSON only (beacon-APIs lists no SSZ body for them), and read integers quoted or bare. All share one SharedSyncCommitteePool, which P2P fills from accepted gossip as well. None of it reaches the chain actor: sync committee votes have no fork-choice effect.

• POST /eth/v1/beacon/pool/sync_committees takes a JSON array of SyncCommitteeMessages and needs no Eth-Consensus-Version. Each is checked against the head state with the rules sync_committee_{subnet_id} gossip applies (the current slot; the validator holds a seat in the committee its slot names, which is the next committee at a period’s last slot; the signature), then pooled once per seat and published on every distinct subnet its seats sit on. One BLS verification per message, on a blocking thread. Not gated on syncing, like pool/attestations. A refused message comes back in an IndexedErrorMessage (400) as {outcome}: {reason} with its position, and the rest still go out. A body that is not a list of messages is a 400 with no failures; a dead network actor is a per-item failure.
• GET /eth/v1/validator/sync_committee_contribution?slot=&subcommittee_index=&beacon_block_root= answers {data: SyncCommitteeContribution} (no version): the best pooled contribution for the key, extended by every pooled direct message at a position it does not cover. 404 when the pool holds nothing for it, 400 when subcommittee_index is not below SYNC_COMMITTEE_SUBNET_COUNT, and 503 while the node is syncing or when the block is optimistic or unknown (an aggregator must not sign over a root this node cannot vouch for).
• POST /eth/v1/validator/contribution_and_proofs takes a JSON array of SignedContributionAndProofs. Each passes the checks sync_committee_contribution_and_proof gossip applies (current slot, subcommittee in range, a participant, the selection proof selects the aggregator, aggregator is in the subcommittee, and the selection proof, envelope and aggregate signatures) against a fresh seen cache, since P2P’s holds what peers sent. A valid one is pooled and gossiped; the failure shape is the same as above.
• POST /eth/v1/validator/sync_committee_subscriptions takes [{validator_index, sync_committee_indices, until_epoch}]. The subnet of a position is index / SYNC_SUBCOMMITTEE_SIZE; entries are grouped by subnet keeping the latest until_epoch (exclusive), clamped to the end of the next sync committee period, and handed to P2P, which joins the subnets immediately. A position at or above SYNC_COMMITTEE_SIZE is a 400 and nothing is joined. A node forgets these on restart, so a client repeats them every epoch. sync_committee_selections (distributed validators) is not served; beacon-APIs lets a node leave it out.

Block production. The block at slot N packs only what is pooled for (N - 1, parent_root): per subcommittee the best contribution, extended by pooled direct messages at uncovered positions. The result is verified exactly as process_sync_aggregate will check it, against the state the block is built on; if it fails, or nothing is pooled, the block carries the empty aggregate. Fulu (blocks/{slot}) and gloas (produceBlockV4) do the same, and the “retry without operations” fallback of each also drops the sync aggregate.

Payload timeliness committee

• POST /eth/v1/validator/duties/ptc/{epoch} takes a JSON array of quoted validator indices (an empty array or a bad index is a 400) and answers {dependent_root, execution_optimistic, data: [{pubkey, validator_index, slot}]}: one duty per validator, the first slot of the epoch whose committee seats it. An index with no seat, or unknown, gets none. The epoch may be at most one past the later of the head state’s and the wall clock’s epoch (else 400). An epoch before the gloas fork answers 200 with an empty data, so a client can ask every epoch. A gloas head state answers from its ptc_window; an epoch the head has not caught up to, or the first gloas epoch under a fulu head, is answered from a copy of the head state advanced to the epoch’s first slot, crossing the gloas upgrade where it falls, on a blocking thread. A gloas epoch more than one before the head state’s is a 400. dependent_root is defined as for attester duties. There is no 503.
• GET /eth/v1/validator/payload_attestation_data?slot= answers {version: "gloas", data: {beacon_block_root, slot, payload_present, blob_data_available}} as JSON, or the SSZ PayloadAttestationData on Accept: application/octet-stream, with Eth-Consensus-Version: gloas. The data is for the block of that very slot on the head’s chain; a slot with no such block (or whose block is not a gloas block) answers 204 with no body, which a client reads as “cast no vote”. A slot before the gloas fork is a 400, and a node that is syncing answers 503, as does one run without an execution client or whose block has an unvalidated execution payload behind it (the same rule as attestation data). payload_present is whether the block’s envelope reached this node before the slot’s start plus PAYLOAD_DUE_BPS of it, judged by arrival time (see Spec Deviations). blob_data_available is true when the bid commits to no blobs, the payload is verified, or every column this node custodies for the block is stored.
• POST /eth/v1/beacon/pool/payload_attestations takes a JSON array of PayloadAttestationMessages; Eth-Consensus-Version is optional and must be gloas when present. Each message passes the checks gossip applies to a peer’s (the current slot, a known block at the slot, committee membership, signature) against a fresh seen-cache. A valid one is pooled and gossiped; one already pooled is neither pooled nor gossiped again and still counts as accepted. A rejected one comes back in an IndexedErrorMessage (400) as {outcome}: {reason} with its position, and the rest still go out; a head-state cache miss is a per-item ignore: state_unavailable.
• GET /eth/v1/beacon/pool/payload_attestations?slot= answers {version: "gloas", data} with the pool’s votes as the specification’s PayloadAttestations, those of slot only when given: one aggregate per slot and PayloadAttestationData, a bit per seat of that slot’s committee (get_ptc on the head state) and the seats’ signatures aggregated, the way block production packs them. The pool itself keeps the individual messages. JSON only; a vote whose slot falls outside the head state’s committee window is left out.

The payload attestation pool is shared with P2P, which fills it from accepted gossip, so a block this node builds carries the votes its peers sent too.

A cl_type: ethlambda participant in our ethereum-package fork (lambdaclass/ethereum-package, branch add-ethlambda) runs an ethlambda validator client that, by default, talks to this node alone, so every block its validators propose is one this node built.

Encoding

JSON is the default; SSZ is served on Accept: application/octet-stream, which is the order lighthouse serves. An Accept listing both is ranked by its q weights. Every response carrying a fork-versioned container also sets Eth-Consensus-Version to the lowercase fork name, in both encodings, since SSZ carries no type tag.

JSON here follows the Beacon API’s own encoding, which is not the lean surface’s: every integer is a quoted decimal string, byte strings are 0x-prefixed hex, and Uint256 is quoted decimal rather than hex. Errors are {"code": ..., "message": ...}, where the lean surface uses {"error": ...}.

Serving /eth/v2/debug/beacon/states/finalized as SSZ is what makes this client checkpoint-syncable from itself: it is the exact path checkpoint_sync.rs fetches from other clients.

GET /eth/v1/beacon/execution_payload_envelopes/{block_id}

Gloas moves a block’s execution payload into a separate signed envelope. This serves the one the node has verified, as {version: "gloas", execution_optimistic, finalized, data} JSON or as SSZ on Accept: application/octet-stream, with Eth-Consensus-Version: gloas either way. It is also what a checkpoint-syncing node asks for the payload of its anchor (see checkpoint_sync.md).

The second 404 is deliberately one status for several situations: the store keeps verified envelopes only, so a withheld payload, a payload not yet received and a pre-gloas block (which has no envelope; the specification’s error list names no other status for it) are indistinguishable here. A caller must read 404 as “unknown to this node”, not as “the payload is empty”.

execution_optimistic under gloas

Before gloas a block’s payload ran inside the block, so the flag is whether the block is in the store’s optimistic set. From gloas on the payload is a separate object that may arrive late or never, and two rules share the flag:

• an envelope response is optimistic iff that payload’s verdict is not VALID (an unrecorded verdict reads as NOT_VALIDATED);
• a block, header, state or finality-checkpoints response for a gloas block is optimistic iff the latest FULL payload the block builds on is not VALID. The walk starts at the block’s parent: if the block’s bid names the parent’s payload as its parent block hash it reads the parent’s verdict, and otherwise it steps back to the parent and repeats. A pre-gloas block on the way reads the optimistic set. The block’s own payload does not count, so a withheld payload does not make the chain look unverified.

Pre-gloas responses are unchanged. One helper implements both rules (shared/optimistic.rs) and every response that carries the flag goes through it.

GET /eth/v1/config/fork_schedule

The Fork objects of the Config’s schedule, oldest first, as {previous_version, current_version, epoch} with the epoch quoted. Phase0 is first, at epoch 0 and its own predecessor; every later fork whose epoch is not FAR_FUTURE_EPOCH follows, and an unscheduled fork is skipped without breaking the chain, so each entry’s previous_version is the one before it in the list. Nimbus’s validator client reads this every epoch, requires that linked-list shape, and marks a node it cannot decode as incompatible. Nimbus also compares /eth/v1/config/spec against its own constants before use: the 13 preset values, the 7 domain types, SECONDS_PER_SLOT (or SLOT_DURATION_MS, which this node does not report) and every fork’s *_FORK_VERSION and *_FORK_EPOCH, with altair scheduled. All are served (a unit test pins the list).

GET /eth/v1/config/spec

One flat object holding the network’s configuration, the compiled preset, and the specification’s constants, as the Beacon API asks. Validator clients depend on it: lighthouse’s refuses a beacon node whose PRESET_BASE does not match its own, and treats an absent key as a mismatch.

The key set is lighthouse’s plus gloas’s preset and constant keys, read from beacon::preset and beacon::constants rather than typed here: PTC_SIZE, MAX_PAYLOAD_ATTESTATIONS, MAX_BUILDER_DEPOSIT_REQUESTS_PER_PAYLOAD, MAX_BUILDER_EXIT_REQUESTS_PER_PAYLOAD, MAX_BUILDERS_PER_WITHDRAWALS_SWEEP, the four gloas domains (DOMAIN_BEACON_BUILDER, DOMAIN_PTC_ATTESTER, DOMAIN_PROPOSER_PREFERENCES, DOMAIN_BUILDER_DEPOSIT), the builder withdrawal prefix and request types, and the BUILDER_INDEX_*, BUILDER_PAYMENT_THRESHOLD_* and PAYLOAD_BUILDER_VERSION constants. Gloas’s schedule, timing and churn keys are Config fields, so they were already reported. Three keys are left out because the specification does not define them (GAS_LIMIT_ADJUSTMENT_FACTOR, RESP_TIMEOUT, TTFB_TIMEOUT), and so are gloas’s MAX_*_SIZE preset keys, which bound gossip message sizes the node does not run on. Domain types and withdrawal prefixes are 0x-prefixed hex; VERSIONED_HASH_VERSION_KZG is a decimal, as lighthouse reports it. GENESIS_TIME is absent: /eth/v1/beacon/genesis reports it.

The configuration keys come from the Config the data directory was initialized with. Some of them (the custody and subnet counts, the MAX_REQUEST_* limits, MAX_PAYLOAD_SIZE, the snappy message domains, MAXIMUM_GOSSIP_CLOCK_DISPARITY) the node runs on compile-time constants for, rather than reading them from Config. Startup refuses a network whose config.yaml sets any of them to a different value (see cli.md), so what this endpoint reports is what the node uses. CONFIG_NAME is the stored name: a resume under a renamed config.yaml warns and keeps it.

Accepted ids, and three that are refused

block_id and state_id accept head, finalized, justified, a slot number, and a 0x-prefixed 32-byte root. Two cases are deliberate refusals, both recorded in Spec Deviations:

• genesis is a 404 on either id. Table::BlockRoots indexes the canonical branch above the store’s anchor, and the anchor’s own slot is never written to it; a checkpoint-synced directory has no genesis block either way.
• A state_id given as a 0x… root is a 404: states are keyed by block root here, with no reverse index. The refusal names the ids that do work.

The anchor’s own slot does resolve, despite being absent from that index: the lookup falls back to the roots the store can name and accepts one only when the block under it really sits at that slot. This matters because it is the slot a checkpoint-syncing peer asks for right after reading the finalized state. A slot the store holds nothing at is still a 404.

/eth/v1/node/identity reports the peer id, the ENR discv5 publishes, and the node’s QUIC, TCP and discv5 multiaddrs. The multiaddrs need --discovery.advertise-ip, since a node bound to 0.0.0.0 does not know the address peers reach it on; without the flag they are empty and the ENR carries no IP. metadata is the MetaData the node answers peers with over req/resp, read at request time, so syncnets and seq_number follow the sync committee subnets a validator client asks for. That seq_number versions MetaData alone: the p2p specification counts it separately from the ENR’s seq.

GET /eth/v1/events

The Beacon API eventstream: the same Server-Sent Events stream as /lean/v0/events (topic on the event: line, payload on data:, one shared best-effort ring, : error - dropped N messages on a gap), with the Beacon API’s own topics and payloads. The response also sets X-Accel-Buffering: no, so a reverse proxy does not hold events back.

curl -N 'http://127.0.0.1:5052/eth/v1/events?topics=head&topics=block,finalized_checkpoint'

topics is required, and may be repeated (?topics=head&topics=block, the specification’s form) or comma-separated (?topics=head,block, which lighthouse also accepts). Duplicates collapse. A missing or empty topics, or a name that is not a Beacon API topic, is a 400 in the Beacon API error shape: {"code":400,"message":"Invalid topic: weather_forecast"}.

Accepted is not emitted. Every topic in the specification is accepted, so a validator client subscribing to several at once is not turned away, but only these are ever sent:

Never emitted: single_attestation (subnet votes are verified and relayed but never reach the chain actor), the operation topics (voluntary_exit, proposer_slashing, attester_slashing, bls_to_execution_change, contribution_and_proof: no operation pools), payload_attributes, the light client topics, blob_sidecar, and head_v2 with every other gloas topic (this build stops at fulu).

How head and finality events behave:

• One head per head recompute. The chain actor recomputes the head once per tick and once per import cascade, and diffs it against the head it last reported. A cascade importing several blocks announces each block, then a single head for where it ended.
• Any head change counts, including one no block caused (a tick’s votes moving the head to a sibling, or an execution client’s INVALID verdict moving it back to an ancestor). Lighthouse reports only the chain_reorg for those; Prysm reports both, as this node does.
• Stale heads are not reported. A head more than 32 slots behind the wall clock (catch-up) sends no head; block events still show progress. chain_reorg and finalized_checkpoint are not gated.
• Order: each block as it imports, then chain_reorg, head, finalized_checkpoint.
• epoch_transition is whether the head’s epoch is later than the previous head’s. The dependent roots follow the specification, with the genesis block root on underflow (epochs 0 and 1).
• chain_reorg.depth is how many slots the previous head sat above the two heads’ common ancestor (lighthouse’s definition).
• execution_optimistic is reported as it is, optimistic heads included (lighthouse suppresses those head events; Prysm sends them flagged).

An aggregate this node’s own validator client submits reaches the chain actor too: gossip never delivers a node its own messages, so POST /eth/v2/validator/aggregate_and_proofs hands each accepted one to the actor after gossiping it. That is also what puts this node’s own aggregates into its fork choice.

Event payloads are built only while at least one client is connected, whatever topics that client asked for; with none, the stream costs nothing.

Metrics & Debug Server (:5054)

The metrics endpoint reads from the global Prometheus registry and needs no store access. See Metrics for the full list of exposed series.

Heap-profiling endpoints are backed by jemalloc’s built-in profiler and are only functional on Linux; other platforms return 501 Not Implemented. On Linux they return 500 if profiling was not enabled at startup.

Test-Driver Endpoints (Hive)

When the binary boots with HIVE_LEAN_TEST_DRIVER=1 (any of 1/true/yes), it runs in test-driver mode instead of the normal API server. The ethereum/hive lean simulator drives these endpoints to replay leanSpec fixtures over HTTP. The driver swaps its in-process Store on every fork_choice/init, so one container can replay many fixtures without restart.

Content Types

Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

For the Lean Chain, a ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The Lean Chain’s metrics follow the leanMetrics specification, with some metrics not yet implemented. The Beacon Chain side has no such specification: its own metrics are among the Custom Metrics (the “Beacon” sections and Data Column Sidecars), and the validator client’s are under Validator client. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

On Linux the endpoint also serves the prometheus crate’s process collector, read from /proc/self at each scrape: process_resident_memory_bytes, process_virtual_memory_bytes, process_cpu_seconds_total, process_open_fds, process_max_fds, process_threads and process_start_time_seconds. They are not part of the leanMetrics spec, and other platforms do not export them.

PQ Signature Metrics

Block Production Metrics

lean_block_building_time_seconds intentionally deviates from the leanMetrics bucket set, which tops out at 1s. Real builds on our devnets routinely run past that, so every sample landed in +Inf and histogram_quantile reported a flat 1s ceiling. The range now covers the same span as the lean_block_proposal_attestation_build_phase_seconds phases it contains.

Fork-Choice Metrics

State Transition Metrics

Validator Metrics

Network Metrics

All three are emitted only by aggregators, once per candidate AttestationData per interval-2 session. lean_aggregation_window_width has buckets 1, 2, 4, 8, 16, 32, 64 and climbs from 1 as the aggregator’s anchor proof climbs the reduction tree; it is capped at lean_attestation_committee_count, so samples pinned there mean the window no longer restricts selection. Compare against that gauge rather than reading the buckets alone: at a committee count that is not a power of two, two different widths can share a bucket. A width stuck at 1 while the network is aggregating means the pool holds nothing on this node’s duty subnet, so it is only aggregating its own raw signatures; check the duty subnet against the aggregator placement. lean_aggregation_skipped_redundant_total only increments with --skip-redundant-aggregation, once per candidate handed to another duty subnet; read it against lean_aggregation_window_width_count for the share of candidates sat out. lean_aggregation_window_fallback_total counts recoveries, not attempts: it increments only when a windowed selection produced no viable job and the full-committee-width retry did. Candidates that are non-viable whatever the window, a lone raw signature or a group with a single proof, never reach the retry and never increment it, so a sparse (strided) aggregator placement across subnets is the only expected cause and the counter should stay at or near zero on a well-tiled deployment. It stays flat entirely under --skip-redundant-aggregation, which disables the fallback.

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Network Sizes

Peer Discovery

See Peer discovery, which is always on for beacon and opt-in (--discovery.enable) for node. Counts dials discovery initiated, as opposed to the static bootnode dials every node makes. Connection outcomes are not repeated here: a discovery dial that succeeds or fails shows up in lean_peer_connection_events_total like any other.

Transport Mix

Which transport actually carried each established connection, read off the connection’s own multiaddr rather than off the address we dialed: libp2p races a peer’s QUIC and TCP addresses within one dial, so the answer is not knowable before the connection exists. tcp counts are what say the fallback in Peer discovery is doing work rather than merely being advertised.

Counts connections rather than peers, so it can exceed lean_peer_connection_events_total{result="success"}, which fires only on a peer’s first connection. unknown covers a multiaddr naming neither transport, which nothing ethlambda binds produces.

Why Peers Leave

lean_peer_disconnection_events_total above is leanMetrics-specified down to its reason values, and those four cannot carry this: measured on the mainnet follower, 92% of outbound closes land in error, which says only that libp2p handed back a cause. These two split that bucket without widening the specified metric, the same way the transport counter above sits beside the specified connect counter rather than inside it.

lean_peer_goodbye_total is the only one of the three that is not an inference. A goodbye is the peer stating why it is dropping us, and the two readings that matter are indistinguishable from the socket alone: too_many_peers means the peer had no room, while bad_score, banned and banned_ip mean it decided against this node, and those call for opposite responses. It has no direction label because goodbye/1 is registered inbound-only; ethlambda never sends one. The reason is a u64 off the wire, so the labels are a fixed set with other as the residue: a remote must not be able to choose this node’s metric cardinality.

lean_peer_disconnect_cause_total covers the closes that carry no goodbye, read off the ConnectionError variant and the error types inside it. It is charged on the same event as the specified counter, once per peer fully disconnecting rather than once per connection, so the two total to the same number and can be read against each other directly. clean_close is libp2p reporting no error at all, which for a beacon peer is the ordinary shape of a deliberate disconnect and should be read beside lean_peer_goodbye_total.

An I/O close almost always arrives as ErrorKind::Other with the muxer’s own error inside, so the label comes from that inner error. The quic_* values are QUIC’s close reasons: quic_application_close is the peer’s application closing (libp2p’s normal close after a goodbye, and go-libp2p’s connection gater), quic_transport_close a transport-level close. TCP closes resolve to the socket error beneath the muxer where there is one (unexpected_eof, connection_reset), otherwise yamux_closed for a clean yamux shutdown. io_other is the residue and should stay near empty; the Peer connection closed line at DEBUG prints the full cause for whatever lands there.

Reason codes 1, 2 and 3 are the only ones the spec names; it reserves [4, 127] and leaves 128 and up to each client. The four above 127 follow lighthouse’s GoodbyeReason, which is what mainnet peers actually send.

Peer Supply

Who this node is connected to, and whether those peers can serve what it needs. All three are set from a full re-count of the state that decides them rather than incremented and decremented per event: a gauge meant to reveal a leak must not be able to leak itself.

lean_peers_by_direction read against lean_swarm_established_connections is that leak check. The first counts peers this node believes it holds; the second libp2p’s own counters, which the connection limits in discovery.md are enforced against. A persistent gap is a connection charged to the cap that no live peer is using.

lean_custody_column_peers is the supply side of the data-availability gate: a fulu block is held until every column this node custodies arrives, so a column sitting at zero connected custodians is a stall waiting to happen, and it is invisible in a total peer count. Only the columns this node samples get a series, since publishing all 128 would bury the ones that can actually block an import. Read it beside lean_blocks_held_for_columns. Beacon-only; a lean node samples nothing and publishes no series here.

Block Import Timing

These record where a block’s time goes between coming off the wire and having a post-state, which is the question lean_fork_choice_block_processing_time_seconds cannot answer: it starts inside store::on_block, so the mailbox hop, the holds, the head update and the per-table size estimates every import pays for all fall outside it. The sections here are the same ones the Block import timing log prints as a tree, and a test asserts the two lists stay identical.

One histogram carries every section, including total for a whole import and the sections an arrival is charged once for. They share a unit and a bucket set, and the query that matters is rate(..._sum[5m]), seconds spent per second, which does not divide by an event count and so does not care that some sections are counted per block and others per arrival. lean_block_import_cascade_blocks is what relates the two counts when you do need them.

total is written only when the block actually imported. A held block publishes every section it crossed and no total, because its import has not finished: that is what keeps a block that waited two slots for its parent out of the import-cost percentiles, and why there is no outcome label to filter on.

parent_wait and columns_wait are separate sections rather than one “pending”, so a block held for its parent and a block held for its custody columns never collapse into the same number. An absent columns_wait still has two readings, and only the log separates them: its da_complete_on_arrival field says whether the columns were never missing, or landed while the block was held for its parent.

The bookkeeping an import triggers (chain-event emission, the finality eviction sweep, the gauge refresh including a RocksDB size estimate per table) has no section of its own. It is charged to the section it follows, which is fc_head on lean and block_atts on beacon. Those three were sections once: across 5248 imports on a mainnet follower none of them reached a millisecond, so they were three rows that never moved in every tree and three label values that never said anything. The work is still counted, just where it happens.

decode is a gossip-only section, so queue is the only one a fetched block crosses before the chain actor. The req/resp codec has already turned the bytes into a block before any handler sees one, leaving no decode boundary to take; the path reports nothing rather than a zero, since a zero reads as free work rather than as unmeasured work and would drag the decode histogram down with samples that measured nothing. Two consequences: decode is a gossip population even though the source label allows sync, and a fetched block’s total starts later in its life than a gossiped block’s, having never counted the request round trip at all.

On the beacon wire, decode for source="gossip" spans more than its name says. BlockArrival::decode_start is still the wire arrival, but handed_off is stamped only once the block has a gossip verdict, so the section also covers the cheap, stateless checks, the stateful check’s own spawn_blocking task, and the verdict’s trip back through the p2p actor’s mailbox. There is no wait for a free validation slot to attribute here either: try_acquire_owned never blocks, and a message arriving with none free is reported Ignore(Overloaded) (and still forwarded to the chain actor) rather than queued. A rising decode on the beacon wire alone therefore does not mean decoding got slower; check lean_beacon_gossip_validation_seconds before assuming so.

engine and fcu are execution-client round trips. They are I/O waits rather than work, so a node whose import time is dominated by them is waiting on its execution client, not spending CPU.

The source label has two values, gossip and sync (req/resp backfill). Two populations are deliberately absent. A block re-delivered to itself because its slot had not started reports under the source it first arrived on, since the hold is already visible as its defer section and a third label value would take the block out of the population it belongs to for every section it has left. A block this node built itself is not measured at all: it crossed no wire, so it has no decode and an empty queue taken when the import began. Both still appear in the log, which names them deferred and local.

Series appear as blocks arrive rather than being seeded, since a third of the phases are beacon-only and a lean node can never write to them.

Per-block phases, in the order a block crosses them, plus total for a completed import:

Per-arrival phases, charged once per arriving message however many blocks its cascade imported:

Gossip Arrival Timing

These histograms record the absolute distance between a gossip message’s arrival and the start of the interval it was due in, so an arrival that is early by some amount and one that is late by the same amount land in the same bucket; the counters’ position label is what tells them apart. inside means the message arrived within the interval it was due in, not merely somewhere in the right slot: an attestation for slot 10 that lands during slot 10’s interval 2 is after, not inside, since it missed the AttestationProduction interval it was actually due in.

The bucket boundaries are the interval and slot edges of the default 4-second cadence. Prometheus fixes buckets when a histogram is registered, so a network that sets MILLISECONDS_PER_SLOT reads these histograms against the default grid rather than its own; the position label still follows the configured interval width.

Blocks anchor to interval 0 of their own slot and attestations to interval 1 of their data slot; both are unbounded above, so a message that never arrives close to real time can be arbitrarily late. Aggregates anchor instead to the most recent aggregation-interval boundary rather than their own data slot, since a stale-group catch-up aggregate can carry a data.slot several slots in the past; anchoring to the latest boundary bounds the delay to one slot and rules out before entirely.

Only gossip-received blocks are sampled here: blocks fetched via req/resp during sync are excluded, since sync backfill delivers blocks long after they were due and would swamp these histograms with catch-up noise rather than gossip-health signal.

The aggregate metrics do include an aggregator’s own freshly produced aggregates, which never come back over gossip; without them an aggregator would report an empty aggregate profile. The two populations are not quite the same measurement: delivery of a locally produced aggregate is held until the interval-2 boundary, so it lands near zero unless proving overran the interval, whereas a received one adds propagation on top of whenever the producer managed to publish it.

In practice the distribution is bimodal and dominated by production rather than propagation: a mode in the lowest bucket for aggregates that made their interval, plus a tail for those whose proving overran it. A late aggregate is late for every node at once, so that tail shows up on receivers too and is not evidence of a slow network. Read a rising tail as aggregation cost, and cross-check lean_pq_sig_aggregated_signatures_building_time_seconds and lean_committee_signatures_aggregation_time_seconds to confirm.

Storage

On a beacon follower, watch lean_table_bytes{table="data_columns"}. Every other table’s series either stays flat or is bounded by pruning; data_columns backs Table::DataColumns, the one table with no pruning rule (see data_storage.md), so this is the disk-growth trajectory for the whole node. Budget roughly 360 KB per slot across the columns this node custodies at the blob cap, nearer 1 GB per day at current mainnet blob counts, and size the disk against however long the node is meant to run before a pruner exists.

lean_state_write_queue_depth is the early warning for a slow disk. Up to three states can be in flight without the importer ever blocking — two queued plus the one the writer thread is currently encoding, diffing and committing — so a depth resting at or below three is healthy overlap. A depth climbing past three means insert_state blocked on the full channel waiting for the writer, and lean_state_write_seconds says whether that time went into the encode or the commit. A depth that stays at zero means the writer has nothing outstanding.

Read lean_state_cache_lookups_total per method, not summed. cached_state with kind="block" is gossip validation’s lookup of the voted or parent block’s state, made once per attestation and aggregate, so it runs far more often than the other two and a summed rate is mostly its own. A get_state or has_state miss means the state was not resident and came from the writer’s buffer or RocksDB. A cached_state miss with kind="checkpoint" means fork choice’s checkpoint_state rebuilt the state, running process_slots up to the epoch boundary when the checkpoint’s block precedes it.

Beacon Gossip Validation

Every beacon gossip message gets a verdict before gossipsub forwards it (validate_messages() is on for the beacon wire only), beacon_aggregate_and_proof and beacon_attestation_{subnet_id} included: both are validated the same way blocks and columns are, in ethlambda_state_transition::beacon::gossip::{aggregate,attestation}. See beacon_wire.md for the flow and beacon_wire.md for the two topics’ own section. These are ethlambda-specific, not part of the leanMetrics spec.

kind is the topic kind, with every data_column_sidecar_{subnet} sharing the label data_column_sidecar, every beacon_attestation_{subnet_id} sharing beacon_attestation and every sync_committee_{subnet_id} sharing sync_committee. The gloas topics execution_payload, payload_attestation_message, execution_payload_bid and proposer_preferences are labelled by their own name. queue means IGNORE to gossipsub while the chain actor still receives the object and parks it; of the gloas topics only execution_payload answers it, for an envelope whose block is not known yet (block_unknown), has no post-state yet (block_not_ready), or has one that is not in the cache the checks read (state_not_cached, as when the block was imported moments ago; the actor verifies the signature itself, so the envelope is forwarded rather than dropped). The aggregate, attestation, payload_attestation_message, execution_payload_bid and proposer_preferences topics never answer queue, since the vote block’s post-state is either cached or it is not (IgnoreReason::UnknownBlock/StateUnavailable), with nothing to hold the message for. verdict_expired_total should stay at zero: a rising count means validation is too slow for gossipsub’s message cache.

The two topics add reasons the others do not, all reason label values on lean_beacon_gossip_validation_total: outside_epoch_window, covered_bits (aggregate only), unknown_block, state_unavailable, finalized_not_ancestor, ancestry_unknown, payload_envelope_unseen and payload_optimistic (gloas only) on the ignore side; epoch_mismatch, no_participants (aggregate only), non_zero_data_index, committee_bits (aggregate only), committee_index, bits_length (aggregate only), not_aggregator (aggregate only), not_in_committee, unknown_validator, selection_proof (aggregate only), aggregator_signature (aggregate only), aggregate_signature (aggregate only), target_not_ancestor, wrong_subnet (attestation only), and gloas’s data_index_out_of_range, same_slot_payload_flag and payload_invalid on the reject side. The builder market topics add, on the ignore side, pre_gloas_slot, not_current_or_next_slot, not_highest_bid, beyond_lookahead, preferences_unseen, fee_recipient_mismatch, parent_payload_unknown, gas_limit_incompatible, not_on_head_branch, builder_cannot_cover, builder_may_exit, slot_started and impossible_dependent_root, and on the reject side execution_payment_nonzero, block_hash_equals_parent, prev_randao, unknown_builder, not_payload_builder, inactive_builder and dependent_root_too_late; a bid over its size cap is malformed. The sync committee topics add sync_committee_unavailable (the head state’s committees cannot answer for the message’s period) on the ignore side and subcommittee_index (contribution only) on the reject side; they reuse not_current_slot, already_seen, covered_bits, no_participants, not_aggregator, not_in_committee, unknown_validator, selection_proof, aggregator_signature, aggregate_signature, wrong_subnet and bad_signature. already_seen, overloaded and unsupported_fork are shared with the other topics: unsupported_fork is an ignore reason for a message of a fork this build has no gossip rules for, so an honest peer past the fork epoch is not scored as a bad decoder. Gloas blocks, data columns, aggregates and attestations are validated, so they carry their own verdict reasons instead.

Three permit pools bound the blocking-thread half of validation: gossip_validation_permits for blocks, columns and envelopes, attestation_validation_permits for aggregates, subnet attestations and payload votes, and builder_validation_permits for bids and proposer preferences. They are deliberately separate: a mainnet slot’s worth of aggregates and backbone attestations arrives every slot, not only during a range sync, and sharing one pool would let that burst answer Ignore(Overloaded) for a block or a column instead. Neither pool has a metric of its own yet; a permit exhausted on either shows up as outcome="ignore",reason="overloaded" on lean_beacon_gossip_validation_total, for the topics that draw from it.

Beacon Peer Scoring

Gossipsub peer scoring runs on the beacon wire only; see beacon_wire.md for the parameters and the mesh-delivery gate. These are ethlambda-specific, not part of the leanMetrics spec.

The bands are mutually exclusive and split at the gossip (-4000), publish (-8000) and graylist (-16000) thresholds, so they add up to the peers gossipsub knows. below_graylist counts the peers that same pass is about to disconnect, so it should not hold a value across passes. A persistent below_gossip/below_publish population points at slow peers first: a full send queue costs a peer 10 points per dropped message.

Beacon Block Publication

POST /eth/v2/beacon/blocks computes every blob’s cells and batch-verifies all its cell proofs on a blocking thread before it gossips anything, then builds the block’s data column sidecars; at the blob cap that is thousands of cells on the proposal’s critical path. See rpc.md. This is ethlambda-specific, not part of the leanMetrics spec.

Beacon Committee Cache

ethlambda beacon derives an epoch’s attester committees with one whole-epoch shuffle and keeps the result in CommitteeCache, keyed by epoch and the block that decided the shuffling. The cache itself lives in the Store (crates/storage/src/committee_cache.rs), shared by both actors: the chain actor’s own attestation processing and p2p’s gossip validation for beacon_aggregate_and_proof/beacon_attestation_{subnet_id} read the same entries rather than each keeping a copy, which is what lets a validation task and the actor race a shuffle at an epoch boundary and have only the first one pay for it. CommitteeCacheExt in crates/blockchain/state_transition/src/beacon/helpers/accessors.rs is the consensus-logic wrapper (ShufflingKey derivation, the shuffle itself) around the storage-side container. This is ethlambda-specific, not part of the leanMetrics spec.

Read the miss rate against the epoch rate, not the hit rate. A follower on one chain misses about once per epoch, when the first block of a new epoch asks for that epoch’s shuffling, and hits on everything else. Misses that track imports instead mean entries are being evicted and rebuilt, which happens when more branches are being imported at once than the cache has room for; each miss is a registry scan plus a whole-epoch shuffle on the import thread. unkeyable is a lookup the cache could not key at all, mostly the genesis state asking about its own first epochs, and should be zero on a checkpoint-synced follower.

Beacon Total Active Balance Cache

The total active balance (the divisor of every attestation’s base reward and of the sync aggregate’s rewards) is one registry pass, and an ordinary block asks for it about nine times. ActiveBalanceCache in the Store keeps it per epoch, keyed by the epoch and the block root that fixes it. This is ethlambda-specific, not part of the leanMetrics spec.

Expect one miss per epoch per chain, and hits otherwise. unkeyable is the genesis state asking about its own epoch, and should be zero on a checkpoint-synced follower.

Beacon Justified Balances

Beacon fork choice weighs every vote by the voter’s effective balance at the justified checkpoint. justified_balances (in crates/blockchain/state_transition/src/beacon/fork_choice.rs) flattens that state into one array, keyed by the justified checkpoint and rebuilt on the first get_head after the checkpoint moves. This is ethlambda-specific, not part of the leanMetrics spec.

Read the miss rate against the justified-checkpoint rate: about one miss per justified checkpoint change, so a handful per hour on a healthy chain. Misses that track get_head calls mean the justified checkpoint is flapping between branches, or the snapshot is being replaced between two readers.

Beacon Epoch Precompute

The beacon chain actor advances the head state across the next epoch boundary ahead of time, so the first block of an epoch does not run process_epoch on the import path. See “Epoch-transition precompute” in beacon_stf.md. This is ethlambda-specific, not part of the leanMetrics spec.

Read the hit ratio against epochs. On a synced follower nearly every epoch-crossing import should hit. Misses mean the worker had not finished (the block raced it, see lean_beacon_epoch_precompute_seconds against the slot duration), the node was syncing, or the entry was evicted from the state cache. A timer start with no matching head start means the last slot was skipped.

Beacon Pubkey Cache

Every BLS signature check ethlambda beacon runs (block import, fork choice, gossip validation) resolves its signers’ compressed public keys to validated curve points through one process-wide cache, keyed by the compressed bytes. See PubkeyCache in crates/blockchain/state_transition/src/beacon/bls.rs. This is ethlambda-specific, not part of the leanMetrics spec.

Misses should fall to near zero within an epoch of a start, once every active validator has signed something the node checked. A steady miss rate after that means keys the cache is refusing to hold: either the entry bound is reached (the gauge stops rising), or the keys are invalid, which are never cached and pay the full check every time. entries is also the cache’s memory footprint, roughly a decompressed point plus its compressed key per entry.

Data Column Sidecars (Fulu DAS)

ethlambda beacon is a fulu data-availability-sampling custodian: it derives a slice of the column matrix from its own discv5 node id (see beacon_wire.md), verifies what gossip and req/resp deliver for that slice, stores what verifies, serves it back to peers, and refuses to import a fulu block until every column it owes for that block is on hand. These are ethlambda-specific, not part of the leanMetrics spec.

lean_data_columns_rejected_total counts the chain checks, which run in the p2p layer on every sidecar gossip did not accept: every fetched sidecar, a gossiped one reported Queue or Ignore(Overloaded), and a parked one sent back once its parent imported. A gossiped sidecar is judged first by the gossip rules and counted in lean_beacon_gossip_validation_total{kind="data_column_sidecar"}; the reasons here are that counter’s reason labels. The chain actor stores what reaches it without checking it again.

Watch lean_blocks_held_for_columns. It is the first symptom of a stalled availability gate, and a healthy node returns it to zero within a slot or two of a hold: fetch or gossip should complete well inside the retry ladder lean_data_column_fetch_failures_total counts down. A gauge that sits above zero for several slots means either the fetch is failing — check whether lean_data_column_fetch_failures_total’s no_peers or max_retries reason is climbing — or no reachable peer actually serves the missing columns; short of that, only finality passing the held block’s slot clears it, which can be minutes away.

Read lean_sidecars_awaiting_parent beside it. A sidecar whose block’s parent has no post-state yet is parked rather than dropped, and a held block is precisely a block with no post-state, so the two gauges rise together while the gate waits: held blocks on one, their children’s columns on the other. Both returning to zero is a recovered hold. lean_sidecars_awaiting_parent climbing without coming back down while lean_blocks_held_for_columns stays above zero is the signature of a gate that is not recovering. The parked queue is uncapped, so that gauge is also the only warning that a peer is parking sidecars under parents it never means to supply: each one holds a Table::PendingDataColumns row until finality passes its slot.

Beacon Aggregate Attestations

ethlambda beacon applies beacon_aggregate_and_proof to fork choice, which is how it sees votes for the current head rather than only the votes a block body carries (see beacon_wire.md). These are ethlambda-specific, not part of the leanMetrics spec, and lean nodes never emit them.

decode is the only one of the four histograms still timing what it always did. The other three no longer include gossip validation: committees, all three signature checks and the specification’s own seen sets moved to p2p (lean_beacon_gossip_validation_seconds{kind="beacon_aggregate_and_proof"} covers that half now), so processing and end_to_end read a great deal lower than before this change, and a comparison against an older deployment’s values is comparing two different things. What is left for these three to measure is real, though: validate_on_attestation_indexed and the applied-bits gate still cost a Table::LiveChain scan and a hash-map lookup per aggregate, so a slow chain-actor number still points at the store rather than at cryptography.

Watch lean_beacon_aggregate_mailbox_wait_seconds. It is the failure mode this path introduces and the one no other timing can show: a mainnet slot carries up to MAX_COMMITTEES_PER_SLOT * TARGET_AGGREGATORS_PER_COMMITTEE aggregates, and if they queue behind block imports their votes arrive too late to move the head while every per-aggregate timing still looks healthy.

Read lean_beacon_aggregate_total{outcome} as a ratio, not a rate. known_subset is the actor’s own applied-bits gate now, not the specification’s seen set (that one is p2p’s, and a duplicate or already-covered aggregate is refused there, under lean_beacon_gossip_validation_total, before it ever reaches this counter). known_subset dominating is still the design working: a committee’s aggregators mostly converge on the same votes, and each one the running union already covers is dropped before another apply_verified_aggregate call. applied falling toward zero while known_subset stays high means the node is seeing only aggregates it has already covered, which is normal; invalid climbing means aggregates gossip already accepted are failing apply_verified_aggregate regardless, most often because this node has not imported the block being voted for, or has since finalized past its target.

queue_full should be zero. The deferral queue holds two slots’ worth, and reaching its cap means aggregates are arriving faster than the once-per-slot drain clears them. Read it beside lean_beacon_aggregates_deferred, which sits near one slot’s worth in the steady state and near the cap when the drain is falling behind.

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

• timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
• agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
• proposal_combined is emitted only when this node proposes a block.

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Validator client

Served by ethlambda validator on its own --metrics-port, separate from the node’s. Prefixed ethlambda_validator_ rather than lean_, because this process follows the beacon chain and a lean_ series here would be misleading on a shared dashboard.

Every series is registered at startup rather than on first use, so each reads zero from the moment the process is up. That matters for alerting: a rule on “attestations stopped” cannot fire against a series that does not exist yet, and absent is not the same as zero.