# Incident Response Runbook

## Overview

This runbook provides procedures for responding to incidents in the DeFi Oracle Meta Mainnet (ChainID 138) network.

## Incident Classification

### Severity Levels

- **P0 - Critical**: Network down, data loss, security breach
- **P1 - High**: Service degradation, validator failures
- **P2 - Medium**: Performance issues, non-critical service failures
- **P3 - Low**: Minor issues, informational alerts

## Incident Response Process

### 1. Detection

- Monitor alerts from Prometheus/Alertmanager
- Check Grafana dashboards
- Review logs in Loki
- Monitor external reports

### 2. Triage

- Classify severity
- Identify affected components
- Assess impact
- Assign incident owner

### 3. Response

- Follow runbook procedures
- Document actions taken
- Communicate with stakeholders
- Escalate if needed

### 4. Resolution

- Verify resolution
- Document root cause
- Update runbooks if needed
- Conduct post-incident review

## Common Incidents

### Network Outage

**Symptoms**:
- No blocks being produced
- Validators not responding
- RPC endpoints unavailable

**Response**:
1. Check validator status: `kubectl get pods -n besu-network -l component=validator`
2. Check logs: `kubectl logs -n besu-network <validator-pod>`
3. Check network connectivity
4. Restart validators if needed: `kubectl rollout restart statefulset/besu-validator -n besu-network`
5. Verify block production: `curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://<rpc-endpoint>`

### Validator Failure

**Symptoms**:
- Validator pod not running
- Validator not producing blocks
- High error rate in logs

**Response**:
1. Check pod status: `kubectl describe pod <validator-pod> -n besu-network`
2. Check logs: `kubectl logs <validator-pod> -n besu-network`
3. Check resource usage: `kubectl top pod <validator-pod> -n besu-network`
4. Restart validator if needed
5. Check validator keys in Key Vault
6. Verify network connectivity

### RPC Endpoint Issues

**Symptoms**:
- RPC endpoints not responding
- High latency
- Error rates increasing

**Response**:
1. Check RPC pod status: `kubectl get pods -n besu-network -l component=rpc`
2. Check Application Gateway status
3. Check rate limiting
4. Scale RPC nodes if needed: `kubectl scale statefulset/besu-rpc --replicas=5 -n besu-network`
5. Check network policies
6. Verify backend connectivity

### Oracle Update Failures

**Symptoms**:
- Oracle not updating
- High error rate in oracle publisher
- Circuit breaker open

**Response**:
1. Check oracle publisher status: `kubectl get pods -n besu-network -l app=oracle-publisher`
2. Check logs: `kubectl logs <oracle-pod> -n besu-network`
3. Check circuit breaker state
4. Verify data sources
5. Check RPC connectivity
6. Verify private key access
7. Restart oracle publisher if needed

### Security Incident

**Symptoms**:
- Unauthorized access attempts
- Unusual network traffic
- Suspicious transactions

**Response**:
1. Isolate affected components
2. Preserve logs and evidence
3. Notify security team
4. Review access logs
5. Check for compromised keys
6. Rotate keys if needed
7. Update security policies

## Escalation

### Escalation Path

1. **On-Call Engineer**: Initial response
2. **Team Lead**: For P1/P0 incidents
3. **Engineering Manager**: For critical incidents
4. **CTO**: For security incidents

### Communication

- Update incident status in Slack/PagerDuty
- Notify stakeholders via email
- Post updates to status page
- Conduct post-incident review

## Post-Incident Review

### Review Process

1. Document incident timeline
2. Identify root cause
3. Document lessons learned
4. Update runbooks
5. Implement improvements
6. Share findings with team

### Review Template

- **Incident**: Brief description
- **Timeline**: Key events and timestamps
- **Root Cause**: What caused the incident
- **Impact**: What was affected
- **Resolution**: How it was resolved
- **Lessons Learned**: What we learned
- **Action Items**: What needs to be done

## Contacts

- **On-Call**: Check PagerDuty
- **Security Team**: security@d-bis.org
- **Engineering Lead**: engineering@d-bis.org
- **Emergency**: +1-XXX-XXX-XXXX

## Resources

- [Monitoring Dashboards](https://grafana.d-bis.org)
- [Logs](https://loki.d-bis.org)
- [Alerts](https://alertmanager.d-bis.org)
- [Status Page](https://status.d-bis.org)