Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

For the Lean Chain, a ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The Lean Chain’s metrics follow the leanMetrics specification, with some metrics not yet implemented. The Beacon Chain side has no such specification: its own metrics are among the Custom Metrics (the “Beacon” sections and Data Column Sidecars), and the validator client’s are under Validator client. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

On Linux the endpoint also serves the prometheus crate’s process collector, read from /proc/self at each scrape: process_resident_memory_bytes, process_virtual_memory_bytes, process_cpu_seconds_total, process_open_fds, process_max_fds, process_threads and process_start_time_seconds. They are not part of the leanMetrics spec, and other platforms do not export them.

PQ Signature Metrics

Block Production Metrics

lean_block_building_time_seconds intentionally deviates from the leanMetrics bucket set, which tops out at 1s. Real builds on our devnets routinely run past that, so every sample landed in +Inf and histogram_quantile reported a flat 1s ceiling. The range now covers the same span as the lean_block_proposal_attestation_build_phase_seconds phases it contains.

Fork-Choice Metrics

State Transition Metrics

Validator Metrics

Network Metrics

All three are emitted only by aggregators, once per candidate AttestationData per interval-2 session. lean_aggregation_window_width has buckets 1, 2, 4, 8, 16, 32, 64 and climbs from 1 as the aggregator’s anchor proof climbs the reduction tree; it is capped at lean_attestation_committee_count, so samples pinned there mean the window no longer restricts selection. Compare against that gauge rather than reading the buckets alone: at a committee count that is not a power of two, two different widths can share a bucket. A width stuck at 1 while the network is aggregating means the pool holds nothing on this node’s duty subnet, so it is only aggregating its own raw signatures; check the duty subnet against the aggregator placement. lean_aggregation_skipped_redundant_total only increments with --skip-redundant-aggregation, once per candidate handed to another duty subnet; read it against lean_aggregation_window_width_count for the share of candidates sat out. lean_aggregation_window_fallback_total counts recoveries, not attempts: it increments only when a windowed selection produced no viable job and the full-committee-width retry did. Candidates that are non-viable whatever the window, a lone raw signature or a group with a single proof, never reach the retry and never increment it, so a sparse (strided) aggregator placement across subnets is the only expected cause and the counter should stay at or near zero on a well-tiled deployment. It stays flat entirely under --skip-redundant-aggregation, which disables the fallback.

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Network Sizes

Peer Discovery

See Peer discovery, which is always on for beacon and opt-in (--discovery.enable) for node. Counts dials discovery initiated, as opposed to the static bootnode dials every node makes. Connection outcomes are not repeated here: a discovery dial that succeeds or fails shows up in lean_peer_connection_events_total like any other.

Transport Mix

Which transport actually carried each established connection, read off the connection’s own multiaddr rather than off the address we dialed: libp2p races a peer’s QUIC and TCP addresses within one dial, so the answer is not knowable before the connection exists. tcp counts are what say the fallback in Peer discovery is doing work rather than merely being advertised.

Counts connections rather than peers, so it can exceed lean_peer_connection_events_total{result="success"}, which fires only on a peer’s first connection. unknown covers a multiaddr naming neither transport, which nothing ethlambda binds produces.

Why Peers Leave

lean_peer_disconnection_events_total above is leanMetrics-specified down to its reason values, and those four cannot carry this: measured on the mainnet follower, 92% of outbound closes land in error, which says only that libp2p handed back a cause. These two split that bucket without widening the specified metric, the same way the transport counter above sits beside the specified connect counter rather than inside it.

lean_peer_goodbye_total is the only one of the three that is not an inference. A goodbye is the peer stating why it is dropping us, and the two readings that matter are indistinguishable from the socket alone: too_many_peers means the peer had no room, while bad_score, banned and banned_ip mean it decided against this node, and those call for opposite responses. It has no direction label because goodbye/1 is registered inbound-only; ethlambda never sends one. The reason is a u64 off the wire, so the labels are a fixed set with other as the residue: a remote must not be able to choose this node’s metric cardinality.

lean_peer_disconnect_cause_total covers the closes that carry no goodbye, read off the ConnectionError variant and the error types inside it. It is charged on the same event as the specified counter, once per peer fully disconnecting rather than once per connection, so the two total to the same number and can be read against each other directly. clean_close is libp2p reporting no error at all, which for a beacon peer is the ordinary shape of a deliberate disconnect and should be read beside lean_peer_goodbye_total.

An I/O close almost always arrives as ErrorKind::Other with the muxer’s own error inside, so the label comes from that inner error. The quic_* values are QUIC’s close reasons: quic_application_close is the peer’s application closing (libp2p’s normal close after a goodbye, and go-libp2p’s connection gater), quic_transport_close a transport-level close. TCP closes resolve to the socket error beneath the muxer where there is one (unexpected_eof, connection_reset), otherwise yamux_closed for a clean yamux shutdown. io_other is the residue and should stay near empty; the Peer connection closed line at DEBUG prints the full cause for whatever lands there.

Reason codes 1, 2 and 3 are the only ones the spec names; it reserves [4, 127] and leaves 128 and up to each client. The four above 127 follow lighthouse’s GoodbyeReason, which is what mainnet peers actually send.

Peer Supply

Who this node is connected to, and whether those peers can serve what it needs. All three are set from a full re-count of the state that decides them rather than incremented and decremented per event: a gauge meant to reveal a leak must not be able to leak itself.

lean_peers_by_direction read against lean_swarm_established_connections is that leak check. The first counts peers this node believes it holds; the second libp2p’s own counters, which the connection limits in discovery.md are enforced against. A persistent gap is a connection charged to the cap that no live peer is using.

lean_custody_column_peers is the supply side of the data-availability gate: a fulu block is held until every column this node custodies arrives, so a column sitting at zero connected custodians is a stall waiting to happen, and it is invisible in a total peer count. Only the columns this node samples get a series, since publishing all 128 would bury the ones that can actually block an import. Read it beside lean_blocks_held_for_columns. Beacon-only; a lean node samples nothing and publishes no series here.

Block Import Timing

These record where a block’s time goes between coming off the wire and having a post-state, which is the question lean_fork_choice_block_processing_time_seconds cannot answer: it starts inside store::on_block, so the mailbox hop, the holds, the head update and the per-table size estimates every import pays for all fall outside it. The sections here are the same ones the Block import timing log prints as a tree, and a test asserts the two lists stay identical.

One histogram carries every section, including total for a whole import and the sections an arrival is charged once for. They share a unit and a bucket set, and the query that matters is rate(..._sum[5m]), seconds spent per second, which does not divide by an event count and so does not care that some sections are counted per block and others per arrival. lean_block_import_cascade_blocks is what relates the two counts when you do need them.

total is written only when the block actually imported. A held block publishes every section it crossed and no total, because its import has not finished: that is what keeps a block that waited two slots for its parent out of the import-cost percentiles, and why there is no outcome label to filter on.

parent_wait and columns_wait are separate sections rather than one “pending”, so a block held for its parent and a block held for its custody columns never collapse into the same number. An absent columns_wait still has two readings, and only the log separates them: its da_complete_on_arrival field says whether the columns were never missing, or landed while the block was held for its parent.

The bookkeeping an import triggers (chain-event emission, the finality eviction sweep, the gauge refresh including a RocksDB size estimate per table) has no section of its own. It is charged to the section it follows, which is fc_head on lean and block_atts on beacon. Those three were sections once: across 5248 imports on a mainnet follower none of them reached a millisecond, so they were three rows that never moved in every tree and three label values that never said anything. The work is still counted, just where it happens.

decode is a gossip-only section, so queue is the only one a fetched block crosses before the chain actor. The req/resp codec has already turned the bytes into a block before any handler sees one, leaving no decode boundary to take; the path reports nothing rather than a zero, since a zero reads as free work rather than as unmeasured work and would drag the decode histogram down with samples that measured nothing. Two consequences: decode is a gossip population even though the source label allows sync, and a fetched block’s total starts later in its life than a gossiped block’s, having never counted the request round trip at all.

On the beacon wire, decode for source="gossip" spans more than its name says. BlockArrival::decode_start is still the wire arrival, but handed_off is stamped only once the block has a gossip verdict, so the section also covers the cheap, stateless checks, the stateful check’s own spawn_blocking task, and the verdict’s trip back through the p2p actor’s mailbox. There is no wait for a free validation slot to attribute here either: try_acquire_owned never blocks, and a message arriving with none free is reported Ignore(Overloaded) (and still forwarded to the chain actor) rather than queued. A rising decode on the beacon wire alone therefore does not mean decoding got slower; check lean_beacon_gossip_validation_seconds before assuming so.

engine and fcu are execution-client round trips. They are I/O waits rather than work, so a node whose import time is dominated by them is waiting on its execution client, not spending CPU.

The source label has two values, gossip and sync (req/resp backfill). Two populations are deliberately absent. A block re-delivered to itself because its slot had not started reports under the source it first arrived on, since the hold is already visible as its defer section and a third label value would take the block out of the population it belongs to for every section it has left. A block this node built itself is not measured at all: it crossed no wire, so it has no decode and an empty queue taken when the import began. Both still appear in the log, which names them deferred and local.

Series appear as blocks arrive rather than being seeded, since a third of the phases are beacon-only and a lean node can never write to them.

Per-block phases, in the order a block crosses them, plus total for a completed import:

Per-arrival phases, charged once per arriving message however many blocks its cascade imported:

Gossip Arrival Timing

These histograms record the absolute distance between a gossip message’s arrival and the start of the interval it was due in, so an arrival that is early by some amount and one that is late by the same amount land in the same bucket; the counters’ position label is what tells them apart. inside means the message arrived within the interval it was due in, not merely somewhere in the right slot: an attestation for slot 10 that lands during slot 10’s interval 2 is after, not inside, since it missed the AttestationProduction interval it was actually due in.

The bucket boundaries are the interval and slot edges of the default 4-second cadence. Prometheus fixes buckets when a histogram is registered, so a network that sets MILLISECONDS_PER_SLOT reads these histograms against the default grid rather than its own; the position label still follows the configured interval width.

Blocks anchor to interval 0 of their own slot and attestations to interval 1 of their data slot; both are unbounded above, so a message that never arrives close to real time can be arbitrarily late. Aggregates anchor instead to the most recent aggregation-interval boundary rather than their own data slot, since a stale-group catch-up aggregate can carry a data.slot several slots in the past; anchoring to the latest boundary bounds the delay to one slot and rules out before entirely.

Only gossip-received blocks are sampled here: blocks fetched via req/resp during sync are excluded, since sync backfill delivers blocks long after they were due and would swamp these histograms with catch-up noise rather than gossip-health signal.

The aggregate metrics do include an aggregator’s own freshly produced aggregates, which never come back over gossip; without them an aggregator would report an empty aggregate profile. The two populations are not quite the same measurement: delivery of a locally produced aggregate is held until the interval-2 boundary, so it lands near zero unless proving overran the interval, whereas a received one adds propagation on top of whenever the producer managed to publish it.

In practice the distribution is bimodal and dominated by production rather than propagation: a mode in the lowest bucket for aggregates that made their interval, plus a tail for those whose proving overran it. A late aggregate is late for every node at once, so that tail shows up on receivers too and is not evidence of a slow network. Read a rising tail as aggregation cost, and cross-check lean_pq_sig_aggregated_signatures_building_time_seconds and lean_committee_signatures_aggregation_time_seconds to confirm.

Storage

On a beacon follower, watch lean_table_bytes{table="data_columns"}. Every other table’s series either stays flat or is bounded by pruning; data_columns backs Table::DataColumns, the one table with no pruning rule (see data_storage.md), so this is the disk-growth trajectory for the whole node. Budget roughly 360 KB per slot across the columns this node custodies at the blob cap, nearer 1 GB per day at current mainnet blob counts, and size the disk against however long the node is meant to run before a pruner exists.

lean_state_write_queue_depth is the early warning for a slow disk. Up to three states can be in flight without the importer ever blocking — two queued plus the one the writer thread is currently encoding, diffing and committing — so a depth resting at or below three is healthy overlap. A depth climbing past three means insert_state blocked on the full channel waiting for the writer, and lean_state_write_seconds says whether that time went into the encode or the commit. A depth that stays at zero means the writer has nothing outstanding.

Read lean_state_cache_lookups_total per method, not summed. cached_state with kind="block" is gossip validation’s lookup of the voted or parent block’s state, made once per attestation and aggregate, so it runs far more often than the other two and a summed rate is mostly its own. A get_state or has_state miss means the state was not resident and came from the writer’s buffer or RocksDB. A cached_state miss with kind="checkpoint" means fork choice’s checkpoint_state rebuilt the state, running process_slots up to the epoch boundary when the checkpoint’s block precedes it.

Beacon Gossip Validation

Every beacon gossip message gets a verdict before gossipsub forwards it (validate_messages() is on for the beacon wire only), beacon_aggregate_and_proof and beacon_attestation_{subnet_id} included: both are validated the same way blocks and columns are, in ethlambda_state_transition::beacon::gossip::{aggregate,attestation}. See beacon_wire.md for the flow and beacon_wire.md for the two topics’ own section. These are ethlambda-specific, not part of the leanMetrics spec.

kind is the topic kind, with every data_column_sidecar_{subnet} sharing the label data_column_sidecar, every beacon_attestation_{subnet_id} sharing beacon_attestation and every sync_committee_{subnet_id} sharing sync_committee. The gloas topics execution_payload, payload_attestation_message, execution_payload_bid and proposer_preferences are labelled by their own name. queue means IGNORE to gossipsub while the chain actor still receives the object and parks it; of the gloas topics only execution_payload answers it, for an envelope whose block is not known yet (block_unknown), has no post-state yet (block_not_ready), or has one that is not in the cache the checks read (state_not_cached, as when the block was imported moments ago; the actor verifies the signature itself, so the envelope is forwarded rather than dropped). The aggregate, attestation, payload_attestation_message, execution_payload_bid and proposer_preferences topics never answer queue, since the vote block’s post-state is either cached or it is not (IgnoreReason::UnknownBlock/StateUnavailable), with nothing to hold the message for. verdict_expired_total should stay at zero: a rising count means validation is too slow for gossipsub’s message cache.

The two topics add reasons the others do not, all reason label values on lean_beacon_gossip_validation_total: outside_epoch_window, covered_bits (aggregate only), unknown_block, state_unavailable, finalized_not_ancestor, ancestry_unknown, payload_envelope_unseen and payload_optimistic (gloas only) on the ignore side; epoch_mismatch, no_participants (aggregate only), non_zero_data_index, committee_bits (aggregate only), committee_index, bits_length (aggregate only), not_aggregator (aggregate only), not_in_committee, unknown_validator, selection_proof (aggregate only), aggregator_signature (aggregate only), aggregate_signature (aggregate only), target_not_ancestor, wrong_subnet (attestation only), and gloas’s data_index_out_of_range, same_slot_payload_flag and payload_invalid on the reject side. The builder market topics add, on the ignore side, pre_gloas_slot, not_current_or_next_slot, not_highest_bid, beyond_lookahead, preferences_unseen, fee_recipient_mismatch, parent_payload_unknown, gas_limit_incompatible, not_on_head_branch, builder_cannot_cover, builder_may_exit, slot_started and impossible_dependent_root, and on the reject side execution_payment_nonzero, block_hash_equals_parent, prev_randao, unknown_builder, not_payload_builder, inactive_builder and dependent_root_too_late; a bid over its size cap is malformed. The sync committee topics add sync_committee_unavailable (the head state’s committees cannot answer for the message’s period) on the ignore side and subcommittee_index (contribution only) on the reject side; they reuse not_current_slot, already_seen, covered_bits, no_participants, not_aggregator, not_in_committee, unknown_validator, selection_proof, aggregator_signature, aggregate_signature, wrong_subnet and bad_signature. already_seen, overloaded and unsupported_fork are shared with the other topics: unsupported_fork is an ignore reason for a message of a fork this build has no gossip rules for, so an honest peer past the fork epoch is not scored as a bad decoder. Gloas blocks, data columns, aggregates and attestations are validated, so they carry their own verdict reasons instead.

Three permit pools bound the blocking-thread half of validation: gossip_validation_permits for blocks, columns and envelopes, attestation_validation_permits for aggregates, subnet attestations and payload votes, and builder_validation_permits for bids and proposer preferences. They are deliberately separate: a mainnet slot’s worth of aggregates and backbone attestations arrives every slot, not only during a range sync, and sharing one pool would let that burst answer Ignore(Overloaded) for a block or a column instead. Neither pool has a metric of its own yet; a permit exhausted on either shows up as outcome="ignore",reason="overloaded" on lean_beacon_gossip_validation_total, for the topics that draw from it.

Beacon Peer Scoring

Gossipsub peer scoring runs on the beacon wire only; see beacon_wire.md for the parameters and the mesh-delivery gate. These are ethlambda-specific, not part of the leanMetrics spec.

The bands are mutually exclusive and split at the gossip (-4000), publish (-8000) and graylist (-16000) thresholds, so they add up to the peers gossipsub knows. below_graylist counts the peers that same pass is about to disconnect, so it should not hold a value across passes. A persistent below_gossip/below_publish population points at slow peers first: a full send queue costs a peer 10 points per dropped message.

Beacon Block Publication

POST /eth/v2/beacon/blocks computes every blob’s cells and batch-verifies all its cell proofs on a blocking thread before it gossips anything, then builds the block’s data column sidecars; at the blob cap that is thousands of cells on the proposal’s critical path. See rpc.md. This is ethlambda-specific, not part of the leanMetrics spec.

Beacon Committee Cache

ethlambda beacon derives an epoch’s attester committees with one whole-epoch shuffle and keeps the result in CommitteeCache, keyed by epoch and the block that decided the shuffling. The cache itself lives in the Store (crates/storage/src/committee_cache.rs), shared by both actors: the chain actor’s own attestation processing and p2p’s gossip validation for beacon_aggregate_and_proof/beacon_attestation_{subnet_id} read the same entries rather than each keeping a copy, which is what lets a validation task and the actor race a shuffle at an epoch boundary and have only the first one pay for it. CommitteeCacheExt in crates/blockchain/state_transition/src/beacon/helpers/accessors.rs is the consensus-logic wrapper (ShufflingKey derivation, the shuffle itself) around the storage-side container. This is ethlambda-specific, not part of the leanMetrics spec.

Read the miss rate against the epoch rate, not the hit rate. A follower on one chain misses about once per epoch, when the first block of a new epoch asks for that epoch’s shuffling, and hits on everything else. Misses that track imports instead mean entries are being evicted and rebuilt, which happens when more branches are being imported at once than the cache has room for; each miss is a registry scan plus a whole-epoch shuffle on the import thread. unkeyable is a lookup the cache could not key at all, mostly the genesis state asking about its own first epochs, and should be zero on a checkpoint-synced follower.

Beacon Total Active Balance Cache

The total active balance (the divisor of every attestation’s base reward and of the sync aggregate’s rewards) is one registry pass, and an ordinary block asks for it about nine times. ActiveBalanceCache in the Store keeps it per epoch, keyed by the epoch and the block root that fixes it. This is ethlambda-specific, not part of the leanMetrics spec.

Expect one miss per epoch per chain, and hits otherwise. unkeyable is the genesis state asking about its own epoch, and should be zero on a checkpoint-synced follower.

Beacon Justified Balances

Beacon fork choice weighs every vote by the voter’s effective balance at the justified checkpoint. justified_balances (in crates/blockchain/state_transition/src/beacon/fork_choice.rs) flattens that state into one array, keyed by the justified checkpoint and rebuilt on the first get_head after the checkpoint moves. This is ethlambda-specific, not part of the leanMetrics spec.

Read the miss rate against the justified-checkpoint rate: about one miss per justified checkpoint change, so a handful per hour on a healthy chain. Misses that track get_head calls mean the justified checkpoint is flapping between branches, or the snapshot is being replaced between two readers.

Beacon Epoch Precompute

The beacon chain actor advances the head state across the next epoch boundary ahead of time, so the first block of an epoch does not run process_epoch on the import path. See “Epoch-transition precompute” in beacon_stf.md. This is ethlambda-specific, not part of the leanMetrics spec.

Read the hit ratio against epochs. On a synced follower nearly every epoch-crossing import should hit. Misses mean the worker had not finished (the block raced it, see lean_beacon_epoch_precompute_seconds against the slot duration), the node was syncing, or the entry was evicted from the state cache. A timer start with no matching head start means the last slot was skipped.

Beacon Pubkey Cache

Every BLS signature check ethlambda beacon runs (block import, fork choice, gossip validation) resolves its signers’ compressed public keys to validated curve points through one process-wide cache, keyed by the compressed bytes. See PubkeyCache in crates/blockchain/state_transition/src/beacon/bls.rs. This is ethlambda-specific, not part of the leanMetrics spec.

Misses should fall to near zero within an epoch of a start, once every active validator has signed something the node checked. A steady miss rate after that means keys the cache is refusing to hold: either the entry bound is reached (the gauge stops rising), or the keys are invalid, which are never cached and pay the full check every time. entries is also the cache’s memory footprint, roughly a decompressed point plus its compressed key per entry.

Data Column Sidecars (Fulu DAS)

ethlambda beacon is a fulu data-availability-sampling custodian: it derives a slice of the column matrix from its own discv5 node id (see beacon_wire.md), verifies what gossip and req/resp deliver for that slice, stores what verifies, serves it back to peers, and refuses to import a fulu block until every column it owes for that block is on hand. These are ethlambda-specific, not part of the leanMetrics spec.

lean_data_columns_rejected_total counts the chain checks, which run in the p2p layer on every sidecar gossip did not accept: every fetched sidecar, a gossiped one reported Queue or Ignore(Overloaded), and a parked one sent back once its parent imported. A gossiped sidecar is judged first by the gossip rules and counted in lean_beacon_gossip_validation_total{kind="data_column_sidecar"}; the reasons here are that counter’s reason labels. The chain actor stores what reaches it without checking it again.

Watch lean_blocks_held_for_columns. It is the first symptom of a stalled availability gate, and a healthy node returns it to zero within a slot or two of a hold: fetch or gossip should complete well inside the retry ladder lean_data_column_fetch_failures_total counts down. A gauge that sits above zero for several slots means either the fetch is failing — check whether lean_data_column_fetch_failures_total’s no_peers or max_retries reason is climbing — or no reachable peer actually serves the missing columns; short of that, only finality passing the held block’s slot clears it, which can be minutes away.

Read lean_sidecars_awaiting_parent beside it. A sidecar whose block’s parent has no post-state yet is parked rather than dropped, and a held block is precisely a block with no post-state, so the two gauges rise together while the gate waits: held blocks on one, their children’s columns on the other. Both returning to zero is a recovered hold. lean_sidecars_awaiting_parent climbing without coming back down while lean_blocks_held_for_columns stays above zero is the signature of a gate that is not recovering. The parked queue is uncapped, so that gauge is also the only warning that a peer is parking sidecars under parents it never means to supply: each one holds a Table::PendingDataColumns row until finality passes its slot.

Beacon Aggregate Attestations

ethlambda beacon applies beacon_aggregate_and_proof to fork choice, which is how it sees votes for the current head rather than only the votes a block body carries (see beacon_wire.md). These are ethlambda-specific, not part of the leanMetrics spec, and lean nodes never emit them.

decode is the only one of the four histograms still timing what it always did. The other three no longer include gossip validation: committees, all three signature checks and the specification’s own seen sets moved to p2p (lean_beacon_gossip_validation_seconds{kind="beacon_aggregate_and_proof"} covers that half now), so processing and end_to_end read a great deal lower than before this change, and a comparison against an older deployment’s values is comparing two different things. What is left for these three to measure is real, though: validate_on_attestation_indexed and the applied-bits gate still cost a Table::LiveChain scan and a hash-map lookup per aggregate, so a slow chain-actor number still points at the store rather than at cryptography.

Watch lean_beacon_aggregate_mailbox_wait_seconds. It is the failure mode this path introduces and the one no other timing can show: a mainnet slot carries up to MAX_COMMITTEES_PER_SLOT * TARGET_AGGREGATORS_PER_COMMITTEE aggregates, and if they queue behind block imports their votes arrive too late to move the head while every per-aggregate timing still looks healthy.

Read lean_beacon_aggregate_total{outcome} as a ratio, not a rate. known_subset is the actor’s own applied-bits gate now, not the specification’s seen set (that one is p2p’s, and a duplicate or already-covered aggregate is refused there, under lean_beacon_gossip_validation_total, before it ever reaches this counter). known_subset dominating is still the design working: a committee’s aggregators mostly converge on the same votes, and each one the running union already covers is dropped before another apply_verified_aggregate call. applied falling toward zero while known_subset stays high means the node is seeing only aggregates it has already covered, which is normal; invalid climbing means aggregates gossip already accepted are failing apply_verified_aggregate regardless, most often because this node has not imported the block being voted for, or has since finalized past its target.

queue_full should be zero. The deferral queue holds two slots’ worth, and reaching its cap means aggregates are arriving faster than the once-per-slot drain clears them. Read it beside lean_beacon_aggregates_deferred, which sits near one slot’s worth in the steady state and near the cap when the drain is falling behind.

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

• timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
• agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
• proposal_combined is emitted only when this node proposes a block.

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Validator client

Served by ethlambda validator on its own --metrics-port, separate from the node’s. Prefixed ethlambda_validator_ rather than lean_, because this process follows the beacon chain and a lean_ series here would be misleading on a shared dashboard.

Every series is registered at startup rather than on first use, so each reads zero from the moment the process is up. That matters for alerting: a rule on “attestations stopped” cannot fire against a series that does not exist yet, and absent is not the same as zero.

Three of these distinguish failures that look alike on a dashboard but have different causes, and the distinction is the reason they are separate series: attestation_failures_total is a duty that ran and failed, attestation_deadline_missed_total is one that never finished, and attestations_refused_total is one deliberately not attempted. A rise in the second points at the beacon nodes; a rise in the third points at the clock or the duty schedule and is worth investigating even though the attestation was correctly suppressed.

The proposal series split the same way, plus one that is neither a success nor a failure. blocks_proposed_total counts blocks a node accepted; block_proposal_failures_total counts duties that produced none; blocks_refused_total counts blocks deliberately not signed. Between them, blocks_broadcast_not_imported_total counts blocks that did reach the network but that the node answering could not import, which is a beacon-node fault rather than a validator one and is why it is not folded into either.

Gloas adds two pairs. A gloas proposal ends in a block and then an envelope, and the envelope’s failure does not fail the proposal: blocks_proposed_total counts the block, and envelope_failures_total the reveal that did not follow. The committee vote has the same split as the attestation’s: payload_attestations_published_total counts votes accepted and payload_attestation_failures_total the slots whose duty produced none. A slot where the node knows no block for the committee to vote on is neither, since there was nothing to sign.

Two series should read zero for the life of a healthy deployment, and are the ones worth alerting on at any non-zero value rather than on a rate: attestations_refused_total and blocks_refused_total mean this client’s own guards caught a duty they judged unsafe, and fee_recipient_mismatches_total means blocks are being proposed that pay someone else. envelope_failures_total belongs with them on a gloas network.

Troubleshooting

Docker Desktop on MacOS

lean-quickstart uses the host network mode for Docker containers, which is a problem on MacOS. To work around this, enable the “Enable host networking” option in Docker Desktop settings under Resources > Network.