Validator Operations

​

Purpose and Scope

This document provides a guide for operating validator nodes in the StableNet network.
It covers validator key management, node configuration, block production (mining), governance participation, monitoring practices, and common failure-handling scenarios. Validators are the core operators of the StableNet blockchain, responsible for block production, consensus participation, and network governance. Related documents:

• Network Deployment
• Node Monitoring and Maintenance
• Anzeon WBFT Consensus Protocol
• Validator Governance (GovValidator)

​

Validator Architecture Overview

​

Validator Key Structure

StableNet validators use three types of keys, each serving a distinct purpose.

• The Operator Key can be a standard EOA (Externally Owned Account) or a multisig contract address for enhanced security.
• The Validator Key and BLS Key must always be accessible to the node process for consensus participation.
• StableNet nodes automatically derive the validator address from the nodekey and use it consistently across the Anzeon chain.

​

Genesis Configuration for Validators

Validators are defined in two locations within the genesis file.

1. anzeon.init section
   • Used to construct EpochInfo in the genesis block (block 0)
   • Defines the active validator set for the first epoch (from block 0 up to the first epoch block)
2. anzeon.systemContracts.govValidator section
   • Defines the initial state of the GovValidator contract (storage initialized at genesis)
   • From the second epoch onward, validator BLS keys are read from this contract when creating new EpochInfo at epoch blocks
   • After genesis, validator additions and removals are only possible via governance (GovValidator proposals, voting, and execution)

The govValidator.params object contains three parallel lists:

• members
• validators
• blsPublicKeys

These lists must have identical lengths and ordering.
Entries at the same index across all lists together represent a single validator identity.

• Operator address (members): Used for governance voting and validator management
• Validator address (validators): Used for consensus message signing and as the coinbase address; derived from the node’s nodekey
• BLS public key (blsPublicKeys): Derived from the validator key; used for WBFT PREPARE/COMMIT aggregated signatures

​

Node Configuration

​

Initial Setup

​

1. Validator Key Generation

The validator address and BLS key are derived from the node’s nodekey.
The nodekey is generated using bootnode -genkey as described in Network Deployment. The BLS public key can be prepared in one of the following ways:

• Automatically derived from the nodekey by genesis_generator
• Generated via bootnode tooling and registered in genesis and GovValidator

​

2. Configuration File Setup

Each validator node uses a config.toml file to configure data directories, RPC, P2P, static nodes, and bootnode settings.
For multi-node deployments, you may directly use the config.toml generated by genesis_generator or customize it for the operational environment.

​

3. Network Configuration

Mainnet and testnet validators can use network presets (-mainnet, -testnet) or specify bootnodes using -bootnodes.
For private networks, follow the static node and bootnode configuration described in
11.1 Network Deployment.

​

Important Configuration Parameters

​

Mining Configuration

In Anzeon/WBFT chains, the etherbase is automatically set to the address derived from the node’s nodekey, regardless of -miner.etherbase.
This guarantees that the validator key and block coinbase always match.
The code (e.g., eth/backend.go) consistently derives the validator address from the P2P node key.

​

Starting Validator Operations

​

Node Startup

Run the node using a data directory initialized with the genesis file.
To enable mining (block production), specify the -mine flag. Example: gstable -datadir <path> -networkid <chainId> -mine -config config.toml If required, add the -bootnodes option to specify bootnodes.

​

Mining Lifecycle

Mining begins once the node is synchronized with the network.
Under WBFT consensus, only the proposer selected for the current round proposes a block, while other validators participate via prepare and commit messages.

​

WBFT-Specific Mining Loop

In WBFT, mining is event-driven rather than timer-based.
When the consensus engine starts a new round, it signals the worker to generate and submit a block. Execution flow summary:

• Trigger
  When WBFT core starts a new round (startNewRound), NotifyNewRound is invoked in the backend and forwarded to the worker’s readyToCommit channel.
  An optional delay until the next block time may also be included.
• Worker
  Upon receiving a signal on readyToCommitCh, the worker creates new sealing work.
  Any in-progress block build is aborted and retried using the new round context.
  When WBFT is enabled, block creation is triggered only by this channel signal; simple chain head updates do not trigger new work.
• Proposer
  A proposer is selected for each round.
  Although all validators may build blocks, only the proposer’s PRE-PREPARE block is used for consensus.

This integration is achieved by registering a readyToCommit callback with the WBFT engine during backend startup, linking the consensus engine and execution backend (worker).

​

Governance Participation

​

Proposal Voting

Governance members participate via the GovValidator contract.
Using the Operator Key, they can create proposals (propose), vote (vote), and execute (execute).
Refer to Validator Governance (GovValidator) for details.

​

Gas Tip Governance

Governance members determine the gas tip (priority fee) for non-authorized accounts via GovValidator governance.
The selected gas tip is queried from GovValidator on every block and automatically applied to both the worker and transaction pool.

​

Key Management Best Practices

​

Security Considerations

​

Protecting the Operator Key

Because the operator key controls governance actions, it carries the highest privilege. Recommended practices:

1. Use multisig contracts
   Use a multisig contract address instead of a plain EOA
2. Cold storage
   Keep the operator key offline except during voting
3. Hardware wallets
   Use hardware wallets for operator key signing
4. Isolated signing services
   Operate a dedicated signing service for operator actions

​

BLS Key Management

BLS keys are mandatory for WBFT participation. The following must be strictly observed.

1. Registration consistency
   The BLS public key registered in GovValidator (or genesis init) must match the BLS public key derived from the node’s nodekey.
   The consensus engine verifies signatures using the registered key; mismatches prevent participation.
2. Derivation during initialization
   During initialization, the WBFT backend derives the BLS secret key by calling bls.DeriveFromECDSA on the node’s ECDSA private key (nodekey).
   There is no separate BLS key file loading path.
3. Secure backups
   Losing the nodekey makes it impossible to recover the same BLS key pair.
   In such cases, governance must be used to register a new validator with a new nodekey and BLS key.

To obtain the BLS public key for genesis or GovValidator registration:

• Derivation logic: genesis_generator.deriveAccount()
• Reference: Data derived from a Nodekey in
  Network Deployment

In production, either generate validator and BLS keys via genesis_generator or use a BLS12-381–capable utility, ensuring the same key pair is used by the node.

​

Validator Rotation

Active validators and BLS keys can be changed only via governance, allowing zero-downtime rotation.

1. Preparation
   Generate a new nodekey and BLS key pair and obtain the validator address and BLS public key.
2. Propose addition
   Submit a proposal to add the new validator using the operator key.
3. Propose removal
   After approval, submit a proposal to remove the old validator.
4. Zero-downtime transition
   Run both validator nodes concurrently during the transition period.

Governance process:

• Call GovValidator.propose() with the operator key
• Members vote via GovValidator.vote()
• Execute via GovValidator.execute() once quorum is reached
• Changes take effect at the next epoch block

​

Key Storage

Restrict file permissions so sensitive key files (e.g., nodekey) are readable only by the validator node.
Example: chmod 600 nodekey
Ensure the directory containing key files also has restricted permissions.

​

Validator Status Monitoring

​

Key Metrics to Monitor

Regularly monitor block production rate, peer count, sync lag, disk and memory usage, and BLS/consensus-related errors.
See Node Monitoring and Maintenance for details.

​

Monitoring Commands

After attaching to the node console using gstable attach, use:

• eth.mining : Mining (block production) status
• eth.blockNumber : Current block height
• admin.peers.length : Number of connected peers
• admin.peers : Detailed peer information

Validator lists can be queried via GovValidator contract calls.

​

Log Monitoring

Inspect logs for:

• Block proposal and commit flow
• BLS signature failures and consensus errors
• Peer connections and disconnections
• Gas tip updates
• Governance-related events

Investigate and address warnings or errors promptly.

​

Log Levels

The default -verbosity is often sufficient.
For deeper inspection of consensus, P2P, or worker behavior, increase -verbosity or use -vmodule to enable detailed logs for specific packages.

​

Metrics Export

Enable Prometheus-compatible metrics using the -metrics flag.
Default endpoint:

• http://localhost:6060/debug/metrics

The metrics port and path can be customized via flags.

​

Common Troubleshooting

​

Issue: Mining Does Not Start

Symptoms: eth.mining returns false and no blocks are produced. Diagnostics:

1. Attach to the node and check eth.mining and eth.coinbase
2. Verify the node was started with the -mine flag
3. Check eth.syncing (mining may be delayed while syncing)

Resolution:

• Restart the node with -mine
• Ensure the nodekey file exists in -datadir (or default data directory)
• Recheck mining status after synchronization completes

​

Issue: Block Production Stops

Symptoms
The validator no longer produces blocks. Diagnostics:

1. Query GovValidator to confirm the node’s address is still in the validator set
2. Inspect logs for WBFT proposal/commit messages or errors
3. Check peer connectivity via admin.peers.length and admin.peers

Resolution:

• Verify validator address and BLS public key registration
• Ensure the node’s BLS key matches the registered public key
• Increase peer connectivity (20–30 peers recommended for stability)
• Check firewall and network access for the P2P port (default 30303)

​

Issue: Gas Tip Mismatch

Symptoms
Transactions fail with "gas tip too low" errors. Diagnostics:

1. Check the current required gas tip via GovValidator.gasTip() or eth.maxPriorityFeePerGas
2. Verify the transaction’s maxPriorityFeePerGas meets or exceeds the required value
3. Confirm the node is fully synchronized and recent governance changes are applied

Resolution:

• Ensure the node is synced (gas tip is fetched automatically by the worker)
• Allow time for block propagation after governance changes
• Submit transactions with maxPriorityFeePerGas ≥ required network gas tip

​

Issue: Node Crash or Restart

Symptoms
The node process exits unexpectedly. Diagnostics:

1. Check available disk space on -datadir
2. Inspect logs for database errors after restart
3. Review system and kernel logs for OOM kills or I/O errors

Resolution:

• Free disk space (≥500GB recommended for production)
  Optionally configure -datadir.minfreedisk
• Increase RAM or adjust -cache if memory pressure occurs
• If DB corruption is suspected, back up data, clear -datadir, and resync
• Use SIGINT or SIGTERM for graceful shutdown and verify shutdown logs

​

Issue: Validator Removed from Set

Symptoms
The node stops producing blocks and is no longer in the validator set. Diagnostics:

1. Query GovValidator.getValidators() to confirm removal
2. Check recent governance proposals, votes, and executions
3. Investigate potential operator key compromise or misuse

Resolution:

• If key compromise is suspected, immediately prepare key rotation
• Coordinate with governance members to propose re-adding the validator
• Review proposal history and logs to determine the cause
• Protect operator keys using multisig and cold storage

​

Performance Optimization

​

Resource Requirements

​

Cache Configuration

Use the -cache option to allocate memory (MB) to the node.
Validators are recommended to use at least 4096MB, considering full sync and block production load.
Increasing the value reduces disk I/O if memory is available.

​

Network Optimization

Set an appropriate -maxpeers value and maintain connections to trusted peers using -bootnodes and static nodes.
Ensure the P2P port (default 30303) is open and the network environment has low latency and packet loss.

​

Epoch Transitions and Validator Set Updates

​

Epoch Structure

WBFT manages blocks in epochs, and validator set changes apply only at epoch blocks. The epoch length is defined by config.anzeon.epochLength in genesis
(e.g., epochLength = 10). At the last block of each epoch (blockNumber % epochLength == 0), the WBFT engine:

1. Reads the current validator list from GovValidator
2. Reads the BLS public key list from GovValidator
3. Updates WBFTExtra.EpochInfo in the next block header
4. Activates the updated validator set starting from the next block

​

Validator Set Synchronization

New or restarting nodes read the validator set from the latest block’s WBFTExtra.EpochInfo during chain synchronization. Because GovValidator updates are applied at epoch blocks, once synchronization completes, the node’s active validator set automatically matches the network’s current state.

​

Validator Rewards and Economics

StableNet validators are rewarded through the following structure.

​

Validator Reward Model

Transaction Fees

• Both Base Fee and Gas Tip are paid to validators
• Base Fee is not burned
  (Unlike Ethereum, no burn mechanism is used to preserve 1:1 fiat backing)
• Gas Tip is paid directly to the block producer
  (credited via Header.Coinbase)
• No block rewards
  There is no inflationary block subsidy by design

​

Gas Fee Policy

StableNet uses a gas fee policy optimized for a stablecoin environment.

​

Gas Tip Policy

Gas Tip application depends on account type.

1. Authorized Accounts

• Same as Ethereum: user-specified Gas Tip per transaction
• maxPriorityFeePerGas is applied as provided

2. Non-Authorized Accounts

• Fixed Gas Tip value set by governance
• Gas Tip is adjustable via GovValidator governance
• Even if a higher maxPriorityFeePerGas is specified, the governance-defined fixed Gas Tip is enforced

​

Base Fee Policy

Unlike Ethereum EIP-1559, StableNet uses a dual-threshold fixed-rate adjustment model. Base Fee Parameters Base Fee Adjustment Rules Base Fee is adjusted based on the parent block’s gas usage.

1. Gas usage > 20%
   • Increase Base Fee by 2%
   • Roughly corresponds to >1,000 simple transfers per block
2. 6% ≤ Gas usage ≤ 20%
   • No change to Base Fee
3. Gas usage < 6%
   • Decrease Base Fee by 2%
   • Roughly corresponds to < 300 simple transfers per block

Base Fee is always clamped within MinBaseFee and MaxBaseFee.

​

Validator Summary Checklist

Setup

• Generate and securely store operator, validator, and BLS keys
• Configure config.toml for the operational environment
• Register the validator via governance
• Set up monitoring and alerting

Operations

• Run validator nodes with -mine
• Verify mining and block production
• Monitor peer count and sync status
• Track gas tip governance changes
• Participate in governance proposals and voting

Maintenance

• Regularly back up configuration files and keys
• Monitor disk space and system resources
• Review logs for errors and warnings
• Test upgrades on testnet before production rollout
• Coordinate changes with other validators and governance members

Security

• Protect operator keys with multisig or cold storage
• Encrypt validator and BLS keys
• Enforce correct file permissions (chmod 600)
• Restrict unnecessary network access via firewall rules
• Conduct regular security reviews of access control and key management