Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

A ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The exposed metrics follow the leanMetrics specification, with some metrics not yet implemented. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

PQ Signature Metrics

Block Production Metrics

lean_block_building_time_seconds intentionally deviates from the leanMetrics bucket set, which tops out at 1s. Real builds on our devnets routinely run past that, so every sample landed in +Inf and histogram_quantile reported a flat 1s ceiling. The range now covers the same span as the lean_block_proposal_attestation_build_phase_seconds phases it contains.

Fork-Choice Metrics

State Transition Metrics

Validator Metrics

Network Metrics

All three are emitted only by aggregators, once per candidate AttestationData per interval-2 session. lean_aggregation_window_width has buckets 1, 2, 4, 8, 16, 32, 64 and climbs from 1 as the aggregator’s anchor proof climbs the reduction tree; it is capped at lean_attestation_committee_count, so samples pinned there mean the window no longer restricts selection. Compare against that gauge rather than reading the buckets alone: at a committee count that is not a power of two, two different widths can share a bucket. A width stuck at 1 while the network is aggregating means the pool holds nothing on this node’s duty subnet, so it is only aggregating its own raw signatures; check the duty subnet against the aggregator placement. lean_aggregation_skipped_redundant_total only increments with --skip-redundant-aggregation, once per candidate handed to another duty subnet; read it against lean_aggregation_window_width_count for the share of candidates sat out. lean_aggregation_window_fallback_total counts recoveries, not attempts: it increments only when a windowed selection produced no viable job and the full-committee-width retry did. Candidates that are non-viable whatever the window, a lone raw signature or a group with a single proof, never reach the retry and never increment it, so a sparse (strided) aggregator placement across subnets is the only expected cause and the counter should stay at or near zero on a well-tiled deployment. It stays flat entirely under --skip-redundant-aggregation, which disables the fallback.

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Network Sizes

Peer Discovery

Only emitted when discv5 discovery is enabled (--discovery.enable); see Peer discovery. Counts dials discovery initiated, as opposed to the static bootnode dials every node makes. Connection outcomes are not repeated here: a discovery dial that succeeds or fails shows up in lean_peer_connection_events_total like any other.

Transport Mix

Which transport actually carried each established connection, read off the connection’s own multiaddr rather than off the address we dialed: libp2p races a peer’s QUIC and TCP addresses within one dial, so the answer is not knowable before the connection exists. tcp counts are what say the fallback in Peer discovery is doing work rather than merely being advertised.

Counts connections rather than peers, so it can exceed lean_peer_connection_events_total{result="success"}, which fires only on a peer’s first connection. unknown covers a multiaddr naming neither transport, which nothing ethlambda binds produces.

Gossip Arrival Timing

These histograms record the absolute distance between a gossip message’s arrival and the start of the interval it was due in, so an arrival that is early by some amount and one that is late by the same amount land in the same bucket; the counters’ position label is what tells them apart. inside means the message arrived within the interval it was due in, not merely somewhere in the right slot: an attestation for slot 10 that lands during slot 10’s interval 2 is after, not inside, since it missed the AttestationProduction interval it was actually due in.

The bucket boundaries are the interval and slot edges of the default 4-second cadence. Prometheus fixes buckets when a histogram is registered, so a network that sets MILLISECONDS_PER_SLOT reads these histograms against the default grid rather than its own; the position label still follows the configured interval width.

Blocks anchor to interval 0 of their own slot and attestations to interval 1 of their data slot; both are unbounded above, so a message that never arrives close to real time can be arbitrarily late. Aggregates anchor instead to the most recent aggregation-interval boundary rather than their own data slot, since a stale-group catch-up aggregate can carry a data.slot several slots in the past; anchoring to the latest boundary bounds the delay to one slot and rules out before entirely.

Only gossip-received blocks are sampled here: blocks fetched via req/resp during sync are excluded, since sync backfill delivers blocks long after they were due and would swamp these histograms with catch-up noise rather than gossip-health signal.

The aggregate metrics do include an aggregator’s own freshly produced aggregates, which never come back over gossip; without them an aggregator would report an empty aggregate profile. The two populations are not quite the same measurement: delivery of a locally produced aggregate is held until the interval-2 boundary, so it lands near zero unless proving overran the interval, whereas a received one adds propagation on top of whenever the producer managed to publish it.

In practice the distribution is bimodal and dominated by production rather than propagation: a mode in the lowest bucket for aggregates that made their interval, plus a tail for those whose proving overran it. A late aggregate is late for every node at once, so that tail shows up on receivers too and is not evidence of a slow network. Read a rising tail as aggregation cost, and cross-check lean_pq_sig_aggregated_signatures_building_time_seconds and lean_committee_signatures_aggregation_time_seconds to confirm.

Storage

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

• timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
• agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
• proposal_combined is emitted only when this node proposes a block.

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Troubleshooting

Docker Desktop on MacOS

lean-quickstart uses the host network mode for Docker containers, which is a problem on MacOS. To work around this, enable the “Enable host networking” option in Docker Desktop settings under Resources > Network.