Metrics

We collect various metrics and serve them via a Prometheus-compatible HTTP endpoint at http://<http_address>:<metrics_port>/metrics (default: http://127.0.0.1:5054/metrics).

A ready-to-use Grafana + Prometheus monitoring stack with pre-configured leanMetrics dashboards is available in lean-quickstart.

The exposed metrics follow the leanMetrics specification, with some metrics not yet implemented. We have a full list of implemented metrics below, with a checkbox indicating whether each metric is currently supported or not.

Node Info Metrics

Name	Type	Usage	Sample collection event	Labels	Supported
`lean_node_info`	Gauge	Node information (always 1)	On node start	name, version	✅
`lean_node_start_time_seconds`	Gauge	Start timestamp	On node start		✅

PQ Signature Metrics

Name	Type	Usage	Sample collection event	Buckets	Supported
`lean_pq_sig_attestation_signatures_total`	Counter	Total number of individual attestation signatures	On each attestation signing		✅
`lean_pq_sig_attestation_signatures_valid_total`	Counter	Total number of valid individual attestation signatures	On each attestation signature verification		✅
`lean_pq_sig_attestation_signatures_invalid_total`	Counter	Total number of invalid individual attestation signatures	On each attestation signature verification		✅
`lean_pq_sig_attestation_signing_time_seconds`	Histogram	Time taken to sign an attestation	On each attestation signing	0.005, 0.01, 0.025, 0.05, 0.1, 1	✅
`lean_pq_sig_attestation_verification_time_seconds`	Histogram	Time taken to verify an attestation signature	On each attestation signature verification	0.005, 0.01, 0.025, 0.05, 0.1, 1	✅
`lean_pq_sig_aggregated_signatures_total`	Counter	Total number of aggregated signatures	On aggregated signature production		✅
`lean_pq_sig_aggregated_signatures_valid_total`	Counter	Total number of valid aggregated signatures	On aggregated signature verification		✅
`lean_pq_sig_aggregated_signatures_invalid_total`	Counter	Total number of invalid aggregated signatures	On aggregated signature verification		✅
`lean_pq_sig_attestations_in_aggregated_signatures_total`	Counter	Total number of attestations included into aggregated signatures	On aggregated signature production		✅
`lean_pq_sig_aggregated_signatures_building_time_seconds`	Histogram	Time taken to build an aggregated attestation signature	On aggregated signature production	0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 4	✅
`lean_pq_sig_aggregated_signatures_verification_time_seconds`	Histogram	Time taken to verify an aggregated attestation signature	On aggregated signature verification	0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 4	✅

Block Production Metrics

Name	Type	Usage	Sample collection event	Labels	Buckets	Supported
`lean_block_aggregated_payloads`	Histogram	Number of `aggregated_payloads` in a block	On block production		1, 2, 4, 8, 16, 32, 64, 128	✅
`lean_block_building_payload_aggregation_time_seconds`	Histogram	Time taken to build `aggregated_payloads` during block building	On block production		0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4	✅
`lean_block_building_time_seconds`	Histogram	Time taken to build a block	On block production		0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1	✅
`lean_block_building_success_total`	Counter	Successful block builds	On block production			✅
`lean_block_building_failures_total`	Counter	Failed block builds (error building the block, signing the block root, or processing it locally)	On block production failure			✅
`lean_block_proposal_attestation_build_phase_seconds`	Histogram	Phase-level time in block-proposal attestation selection	On block production	phase=select_payloads,compact,stf_simulate	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8	✅
`lean_block_proposal_attestation_builds_total`	Counter	Attestations selected during block-proposal selection (one per selection-loop round that picks an `AttestationData`)	On each attestation selection			✅
`lean_block_proposal_child_payloads_consumed_total`	Counter	Child aggregated payloads selected during greedy proof picking (before compaction)	On block production			✅
`lean_block_proposal_attestation_data_selected`	Histogram	Distinct `AttestationData` entries in the proposal block body	On block production		0, 1, 2, 4, 8, 16, 32	✅
`lean_block_proposal_aggregates_selected`	Histogram	Aggregated signature proofs in the proposal result after compaction	On block production		0, 1, 2, 4, 8, 16, 32, 64, 128	✅

Fork-Choice Metrics

Name	Type	Usage	Sample collection event	Labels	Buckets	Supported
`lean_head_slot`	Gauge	Latest slot of the lean chain	On get fork choice head			✅
`lean_current_slot`	Gauge	Current slot of the lean chain	On scrape			✅(*)
`lean_safe_target_slot`	Gauge	Safe target slot	On safe target update			✅
`lean_fork_choice_block_processing_time_seconds`	Histogram	Time taken to process block	On fork choice process block		0.005, 0.01, 0.025, 0.05, 0.1, 1, 1.25, 1.5, 2, 4	✅
`lean_attestations_valid_total`	Counter	Total number of valid attestations	On validate attestation			✅
`lean_attestations_invalid_total`	Counter	Total number of invalid attestations	On validate attestation			✅
`lean_attestation_validation_time_seconds`	Histogram	Time taken to validate attestation	On validate attestation		0.005, 0.01, 0.025, 0.05, 0.1, 1	✅
`lean_fork_choice_reorgs_total`	Counter	Total number of fork choice reorgs	On fork choice reorg			✅
`lean_fork_choice_reorg_depth`	Histogram	Depth of fork choice reorgs (in blocks)	On fork choice reorg		1, 2, 3, 5, 7, 10, 20, 30, 50, 100	✅
`lean_tick_interval_duration_seconds`	Histogram	Elapsed time between clock ticks in seconds	At the start of each tick interval		0.4, 0.6, 0.75, 0.8, 0.805, 0.81, 0.815, 0.82, 0.825, 0.85, 0.9, 1.0, 1.2, 1.6	✅
`lean_gossip_signatures`	Gauge	Number of gossip signatures in fork-choice store	On gossip signatures update			✅
`lean_latest_new_aggregated_payloads`	Gauge	Number of new aggregated payload items	On `latest_new_aggregated_payloads` update			✅
`lean_latest_known_aggregated_payloads`	Gauge	Number of known aggregated payload items	On `latest_known_aggregated_payloads` update			✅
`lean_committee_signatures_aggregation_time_seconds`	Histogram	Time taken to aggregate committee signatures	On committee signatures aggregation		0.05, 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4	✅
`lean_node_sync_status`	Gauge	Node sync status	On node sync status change	status=idle,syncing,synced		✅

State Transition Metrics

Name	Type	Usage	Sample collection event	Labels	Buckets	Supported
`lean_latest_justified_slot`	Gauge	Latest justified slot	On state transition			✅
`lean_latest_finalized_slot`	Gauge	Latest finalized slot	On state transition			✅
`lean_justified_slot`	Gauge	Current justified slot	On state transition			❌
`lean_finalized_slot`	Gauge	Current finalized slot	On state transition			❌
`lean_finalizations_total`	Counter	Total number of finalization attempts	On finalization attempt	result=success,error		✅
`lean_state_transition_time_seconds`	Histogram	Time to process state transition	On state transition		0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 2.5, 3, 4	✅
`lean_state_transition_slots_processed_total`	Counter	Total number of processed slots	On state transition process slots			✅
`lean_state_transition_slots_processing_time_seconds`	Histogram	Time taken to process slots	On state transition process slots		0.005, 0.01, 0.025, 0.05, 0.1, 1	✅
`lean_state_transition_block_processing_time_seconds`	Histogram	Time taken to process block	On state transition process block		0.005, 0.01, 0.025, 0.05, 0.1, 1	✅
`lean_state_transition_attestations_processed_total`	Counter	Total number of processed attestations	On state transition process attestations			✅
`lean_state_transition_attestations_processing_time_seconds`	Histogram	Time taken to process attestations	On state transition process attestations		0.005, 0.01, 0.025, 0.05, 0.1, 1	✅

Validator Metrics

Name	Type	Usage	Sample collection event	Buckets	Supported
`lean_validators_count`	Gauge	Number of validators managed by a node	On scrape		✅(*)
`lean_is_aggregator`	Gauge	Validator’s `is_aggregator` status. True=1, False=0	On node start		✅
`lean_attestations_production_time_seconds`	Histogram	Time taken to produce attestation	On attestation production	0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1	✅

Network Metrics

Name	Type	Usage	Sample collection event	Labels	Supported
`lean_attestation_committee_count`	Gauge	Number of attestation committees	On node start		✅
`lean_attestation_committee_subnet`	Gauge	Node’s attestation committee subnet	On node start		✅
`lean_connected_peers`	Gauge	Number of connected peers	On scrape	client=ethlambda,grandine,lantern,lighthouse,qlean,ream,zeam	✅(*)
`lean_gossip_mesh_peers`	Gauge	Number of peers in the gossipsub mesh	On scrape	client=`<name>_<N>`,unknown (ex. zeam_0)	✅(*)
`lean_peer_connection_events_total`	Counter	Total number of peer connection events	On peer connection	direction=inbound,outbound result=success,timeout,error	✅
`lean_peer_disconnection_events_total`	Counter	Total number of peer disconnection events	On peer disconnection	direction=inbound,outbound reason=timeout,remote_close,local_close,error	✅

Custom Metrics (non-leanMetrics)

The metrics below are not part of the leanMetrics specification. They are ethlambda-specific observability around on-wire message sizes and post-quantum aggregated proof sizes.

PQ Signature Sizes

Name	Type	Usage	Sample collection event	Labels	Buckets
`lean_aggregated_proof_size_bytes`	Histogram	Bytes size of an aggregated signature proof’s `proof_data` field	On aggregated signature production		1024, 4096, 16384, 65536, 131072, 262144, 524288, 1048576

Network Sizes

Name	Type	Usage	Sample collection event	Labels	Buckets
`lean_gossip_block_size_bytes`	Histogram	Bytes size of a gossip block message (raw SSZ or snappy on-wire)	On gossip block send/receive	compression=raw,snappy	10000, 50000, 100000, 250000, 500000, 1000000, 2000000, 5000000
`lean_gossip_attestation_size_bytes`	Histogram	Bytes size of a gossip attestation message (raw SSZ or snappy on-wire)	On gossip attestation send/receive	compression=raw,snappy	512, 1024, 2048, 4096, 8192, 16384
`lean_gossip_aggregation_size_bytes`	Histogram	Bytes size of a gossip aggregated attestation message (raw SSZ or snappy on-wire)	On gossip aggregation send/receive	compression=raw,snappy	1024, 4096, 16384, 65536, 131072, 262144, 524288, 1048576
`lean_reqresp_request_size_bytes`	Histogram	Bytes size of a req/resp request (raw SSZ or snappy on-wire)	On req/resp request send/receive	protocol=status,blocks_by_root compression=raw,snappy	64, 128, 256, 512, 1024, 4096, 16384, 65536
`lean_reqresp_response_chunk_size_bytes`	Histogram	Bytes size of a single req/resp response chunk (raw SSZ or snappy on-wire)	On req/resp response chunk send/receive	protocol=status,blocks_by_root compression=raw,snappy	128, 1024, 10000, 100000, 500000, 1000000, 5000000, 10000000

Storage

Name	Type	Usage	Sample collection event	Labels
`lean_table_bytes`	Gauge	Estimated byte size of a storage table (key + value bytes)	After each processed block (one update per table); retains its previous value on empty slots	table=`<table_name>`

Attestation Aggregate Coverage

Observability into how many validators/subnets are covered by the attestations the node has aggregated, broken down by pipeline section (the section label). The slot is the X-axis. These are sampled roughly once per slot, but emission is gated by the section’s source data, so a gauge can retain its previous value:

timely, late, block, combined and the diff_validators directions are emitted on block import, and only when the canonical head block carries that round’s votes (otherwise the round is skipped and prior values are kept).
agg_start_new is emitted at interval 2, right before fork-choice aggregation runs.
proposal_combined is emitted only when this node proposes a block.

Name	Type	Usage	Sample collection event	Labels
`lean_attestation_aggregate_coverage_validators`	Gauge	Validator coverage in attestation aggregate reports	Per round, per section (see note above)	section=timely,late,block,combined,agg_start_new,proposal_combined subnet=combined,subnet_0,subnet_1,…,subnet_N-1
`lean_attestation_aggregate_coverage_subnets`	Gauge	Number of covered subnets in attestation aggregate reports	Per round, per section (see note above)	section=timely,late,block,combined,agg_start_new,proposal_combined
`lean_attestation_aggregate_coverage_diff_validators`	Gauge	Validators in the symmetric difference between block-included aggregates and locally-aggregated timely aggregates for the same slot	On block import, when the head carries the round’s votes (see note above)	direction=block_only,timely_only

✅(*) Partial support: These metrics are implemented but not collected “on scrape” as the spec requires. They are updated on specific events (e.g., on tick, on block processing) rather than being computed fresh on each Prometheus scrape.

Troubleshooting

Docker Desktop on MacOS

lean-quickstart uses the host network mode for Docker containers, which is a problem on MacOS. To work around this, enable the “Enable host networking” option in Docker Desktop settings under Resources > Network.

Keyboard shortcuts

ethlambda