Troubleshooting

Understanding common errors and their solutions helps maintain a healthy node operation.

Common Error Codes

Here are the most frequent errors you might encounter and their solutions:

Consensus Errors

When you encounter consensus errors, quick and appropriate action is essential:

Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status

Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json

Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines

Network Errors

Network errors can prevent your node from participating in consensus:

Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status

Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info

Database Errors

Database corruption can require immediate attention:

Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/

Diagnostic Commands

These commands help you investigate issues and monitor your node:

# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)

AppHash Mismatch Errors

If you encounter an AppHash mismatch, you’ll need to capture the state for comparison with a known good version:

# For SeiDB (all supported nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid

As with FlatKV, the memIAVL store has two possible locations: nodes created before the layout change keep the legacy $HOME/.sei/data/committer.db shown above (it takes precedence when present), while new nodes use $HOME/.sei/data/state_commit/memiavl. Point -d at whichever exists on your node.

On Giga Storage nodes, EVM state lives in a FlatKV store rather than in the memIAVL trees, so an AppHash comparison there requires dumping the FlatKV state as well. Use the dump-flatkv command to iterate and dump the physical (key, value) pairs into per-bucket files (formatted to match dump-iavl so the same diff tooling works on both):

# For FlatKV (Giga Storage nodes hold EVM state in FlatKV):
systemctl stop seid
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump
systemctl restart seid

Nodes created before the storage layout change keep the FlatKV store at the legacy path $HOME/.sei/data/flatkv instead of $HOME/.sei/data/state_commit/flatkv. Pass whichever directory exists on your node to --db-dir.

The dump-flatkv command accepts the following flags:

--db-dir / -d: FlatKV database directory.
--output-dir / -o: output directory; one file is written per bucket.
--height: FlatKV target version; 0 (the default) selects the latest available version.
--bucket / -b: restrict the dump to a single bucket (account, code, storage, or legacy). Defaults to all buckets.

For example, to dump only the storage bucket at a specific version:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage

Comparing EVM State Between memIAVL and FlatKV

When debugging an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends even when the underlying state is identical. This is because every FlatKV value embeds a per-key block-height stamp (the height the key was last written or migrated), which differs from the memIAVL leaf versions on a freshly migrated node. The evm-logical-digest command works around this by computing a backend-independent digest of the EVM logical state — it strips the serialization-version and block-height header on both sides and digests only the logical payload (account balance/nonce/code hash, bytecode, and storage words). This lets a memIAVL node and a FlatKV node be compared at the same chain height:

# FlatKV digest at a height (WAL-replays to it):
seidb evm-logical-digest --backend flatkv \
    --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000

# memIAVL digest at the same height (0 = current symlink):
seidb evm-logical-digest --backend memiavl \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000

Each run prints per-bucket bucket_digest values and a single FINAL_DIGEST line covering the account, code, storage, and legacy buckets. Compare the FINAL_DIGEST lines from both backends at the same height — they should match. Because FlatKV can contain a FlatKV-only migration-version marker that a memIAVL-only node never owns, that row is automatically omitted from the FlatKV final result so the comparison is apples-to-apples. For targeted debugging, the command also supports inspecting a single normalized bucket instead of printing the global digest. The full flag reference (inspect mode, sharding, --find-hash) lives in the seidb tooling section of the technical reference. For example, to list the first 50 account rows with version metadata, or to shard the storage bucket under a key prefix by the next 2 bytes:

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket account --list --list-limit 50 --details

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2

The legacy IAVL backend has been fully removed and SeiDB State Commit (SC) is now mandatory. SC must be enabled via sc-enable = true in the [state-commit] section of app.toml. If SC is not enabled, the node no longer falls back to IAVL — it panics at startup with:

SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated

The seid debug dump-iavl command has also been removed along with the IAVL backend. To inspect state, use the seidb dump-iavl tool shown above.

Always include the app hash, commit hash, and block height from your logs when reporting issues.

Identifying AppHash Errors

AppHash errors typically appear in logs as:

ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]

Common Causes:

Using incorrect node version during sync (ensure you’re on the latest version)
Corrupted or incorrectly applied snapshots
Database inconsistencies from improper shutdowns
Syncing with outdated or incompatible peers

Resolution Steps:

Stop the node immediately.
Try a node rollback first:, see here
If rollback fails, restore from a fresh snapshot:
- Download a recent snapshot from trusted providers (Polkachu, PublicNode)
- Ensure you’re using the correct node version
- Verify peer configurations are up to date
Restart the node and monitor logs for continued errors

Peer Connection Issues as AppHash Red Herrings

Important Note: Peer connection failures are often symptoms of underlying AppHash errors, not the root cause. When you see extensive peer connection errors like:

ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF

Don’t focus solely on fixing peer connections first. Instead:

Scan your logs carefully for AppHash errors that may appear intermittently

Look for the actual error pattern:

ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]

Check if your node is stuck at a specific height despite peer connection attempts

Why This Happens:

AppHash mismatches prevent proper block validation
Node cannot advance to new blocks due to state inconsistency
Peers may reject connections from nodes with corrupted state
Network appears to be the problem when it’s actually a local state issue

Debugging Approach:

First, check for AppHash errors in your logs (search for “wrong Block.Header.AppHash”)
If AppHash errors are found, treat this as the primary issue
Only focus on peer connection fixes if no AppHash errors exist

This approach can save hours of debugging time by addressing the root cause rather than symptoms.

Peer Connection and Handshake Issues

Identifying Peer Issues: Look for these error patterns in your logs:

ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p

Common Causes:

Outdated peer configurations with mismatched node IDs
Network infrastructure changes on peer side
Firewall blocking connections on port 26656
DNS resolution issues

Resolution Steps:

Update peer configurations with current node IDs:

Verify network connectivity:

# Test connection to peer endpoints
nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656

# Check if port 26656 is open for inbound connections
netstat -tulpn | grep :26656

Check current peer status:

curl http://localhost:26657/net_info | jq '.result.peers | length'
curl http://localhost:26657/lag_status | jq .

Sync Performance Issues

Identifying Sync Problems: Monitor these indicators:

# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'

Common Solutions:

Increase packet payload size for large block processing:

# In config.toml [p2p] section
max-packet-msg-payload-size = 1024000  # Increase from default 102400

Optimize mempool settings in config.toml:

# In [mempool] section
keep-invalid-txs-in-cache = true
ttl-duration = "5s"
ttl-num-blocks = 5

If node gets stuck at specific height:
- Try restarting the node
- If restart doesn’t help, perform rollback
- Consider taking a fresh snapshot

Warning Signs to Watch For:

Current height not increasing over time
Increasing lag between current height and max peer height
Repeated timeout errors in logs
Mempool size consistently reaching limits

Crash and Panic Debugging

For crashes, panics, or nil pointer exceptions:

Capture at least 1,000 lines of logs leading up to the crash
Or collect 15 minutes of log data, whichever provides more context
Include the full stack trace if available

Logging Configuration

Proper logging configuration is essential for debugging and monitoring:

# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs

Configure log rotation to manage storage effectively:

# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF

Enable core dumps for crash analysis:

# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern

Other common Issues and Fixes

Sync Problems
- Check available disk space (df -h)
- Ensure proper peer connections (curl http://localhost:26657/net_info)
- Verify firewall settings (port 26656 open)
Performance Issues
- Monitor system resources (htop or iotop)
- Check disk I/O performance (iostat)
- Analyze network traffic (iftop)
Database Issues
- Run database integrity checks using:
  seid debug dump-db | grep -i error
  If errors are detected, consider restoring from a recent backup.
- Consider pruning excessive historical data by adjusting ss-keep-recent in app.toml or running:
  seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
  Alternatively, manually remove old state snapshots to free up space:
  rm -rf $HOME/.sei/data/snapshots/*

Node Rollback

To rollback a node from an AppHashed state, you need to stop the node first. Do this in your preferred way. Next, do a rollback with:

seid rollback

Then, restart the node. In case you see the following error while trying to rollback:

failed to initialize database: resource temporarily unavailable

This means that you did not shutdown the node properly. Try to shutdown or kill the seid process directly in that case. If this doesn’t help, restart your machine. Then try the rollback steps again.

Overview

Node Operations

Advanced Operations

Common Error Codes

Consensus Errors

Network Errors

Database Errors

Diagnostic Commands

AppHash Mismatch Errors

Comparing EVM State Between memIAVL and FlatKV

Identifying AppHash Errors

Peer Connection Issues as AppHash Red Herrings

Peer Connection and Handshake Issues

Sync Performance Issues

Crash and Panic Debugging

Logging Configuration

Other common Issues and Fixes

Node Rollback

​Common Error Codes

​Consensus Errors

​Network Errors

​Database Errors

​Diagnostic Commands

​AppHash Mismatch Errors

​Comparing EVM State Between memIAVL and FlatKV

​Identifying AppHash Errors

​Peer Connection Issues as AppHash Red Herrings

​Peer Connection and Handshake Issues

​Sync Performance Issues

​Crash and Panic Debugging

​Logging Configuration

​Other common Issues and Fixes

​Node Rollback

Common Error Codes

Consensus Errors

Network Errors

Database Errors

Diagnostic Commands

AppHash Mismatch Errors

Comparing EVM State Between memIAVL and FlatKV

Identifying AppHash Errors

Peer Connection Issues as AppHash Red Herrings

Peer Connection and Handshake Issues

Sync Performance Issues

Crash and Panic Debugging

Logging Configuration

Other common Issues and Fixes

Node Rollback