smom-dbis-138/runbooks/validator-transitions.md

Validator Transitions Runbook

Overview

This runbook provides procedures for adding and removing validators from the network.

Adding a Validator

Prerequisites

• New validator keys generated
• Validator infrastructure ready
• Coordination with existing validators

Procedure

1. Generate Validator Keys

   ./scripts/key-management/generate-validator-keys.sh 1
   

2. Store Keys in Key Vault

   az keyvault secret set --vault-name defi-oracle-kv --name validator-key-new --value <private-key>
   

3. Update Genesis File

   • Add validator address to extraData
   • Regenerate genesis file using Besu operator

4. Update Static Nodes

   • Add validator enode to static-nodes.json
   • Update all node configurations

5. Deploy Validator

   # Update StatefulSet replica count
   kubectl scale statefulset/besu-validator --replicas=5 -n besu-network
   

6. Verify Validator

   • Check validator is producing blocks
   • Verify validator is in validator set
   • Monitor validator performance

Removing a Validator

Prerequisites

• Coordination with validator operator
• Backup of validator data
• Maintenance window scheduled

Procedure

1. Plan Removal

   • Notify all validators
   • Schedule maintenance window
   • Plan rollback procedure

2. Update Genesis File

   • Remove validator address from extraData
   • Regenerate genesis file

3. Update Static Nodes

   • Remove validator enode from static-nodes.json
   • Update all node configurations

4. Remove Validator Pod

   kubectl scale statefulset/besu-validator --replicas=3 -n besu-network
   

5. Verify Removal

   • Check validator set
   • Verify network is still producing blocks
   • Monitor network performance

Validator Key Rotation

See Key Rotation Script for detailed procedures.

Troubleshooting

Validator Not Producing Blocks

1. Check validator status
2. Verify validator keys
3. Check network connectivity
4. Verify QBFT configuration
5. Check logs

Validator Not in Validator Set

1. Verify validator address in extraData
2. Check static nodes configuration
3. Verify network connectivity
4. Check validator keys

Contacts

• Validators: validators@d-bis.org
• Network Operators: operators@d-bis.org

References

• Node Add/Remove Runbook
• Parameter Change Runbook
• Key Management Scripts