Periodic sync lag on Sei full nodes

[victor-lambda256]

Seid version
6.5

Chain ID
pacific

Periodic sync lag on Sei full nodes despite high-spec NVMe infrastructure

Summary

We operate multiple Sei full nodes and continuously monitor their block heights in real time to ensure that all nodes remain fully synced with the latest chain height.

As many node operators are aware, Sei nodes are highly I/O intensive, so we are already running our nodes on high-spec hardware with NVMe storage. However, we are observing frequent and periodic sync delays where full nodes fall behind the latest block height and then later catch up again.

From the perspective of an RPC provider, this behavior is problematic because periodic sync lag can directly impact RPC reliability and user experience.

Issue observed

Even with high-performance hardware and NVMe storage, Sei full nodes periodically experience the following pattern:

1. The node is fully synced.

2. The node starts falling behind the latest block height.

3. After some time, it catches up again.

4. The same pattern repeats periodically.

This does not appear to be a one-time sync issue. It looks like a recurring stability issue during normal full node operation.

Why this matters

For RPC providers, full nodes are expected to remain close to the latest block height at all times. If nodes periodically fall behind, RPC responses may become stale or unreliable, especially for users and services that depend on up-to-date chain data.

Since we are already operating Sei nodes on high-spec machines with NVMe storage, we would like to better understand whether this behavior is expected, known, or related to a node/client-level performance bottleneck.

Questions

• Is the Sei team aware of periodic sync lag issues on full nodes?

• Are there any known bottlenecks related to IOPS, state processing, consensus, mempool, or block execution that could cause this behavior?

• Are there recommended configuration changes for full nodes serving RPC traffic?

• Are there any planned improvements to node stability or sync performance?

• What metrics or logs would be useful for us to provide to help investigate this issue?

Expected behavior

A Sei full node running on high-spec NVMe infrastructure should be able to consistently remain close to the latest block height without periodically falling behind during normal operation.

Actual behavior

The node frequently and periodically falls behind the latest block height, then catches up again. This repeating sync lag pattern makes it difficult to provide stable RPC service.

[github-actions]

Thank you for opening this issue and taking the time to share your thoughts.
We’d love to keep moving this forward, but we need a bit more information from you.
Please add any additional details within the next 2 day(s) so we can continue collaborating on a solution together.
If we don’t hear back, the issue will close automatically — but you’re always welcome to reopen it when you’re ready.