defi-arbitrage/docs/settlement/as4/DETAILED_NEXT_STEPS.md

# AS4 Settlement - Detailed Next Steps

**Date**: 2026-01-19  
**Version**: 1.0.0

---

## Table of Contents

1. [Database Setup](#1-database-setup)
2. [Environment Configuration](#2-environment-configuration)
3. [Certificate Setup](#3-certificate-setup)
4. [Database Migration](#4-database-migration)
5. [Marketplace Seeding](#5-marketplace-seeding)
6. [Service Configuration](#6-service-configuration)
7. [Testing](#7-testing)
8. [Server Startup](#8-server-startup)
9. [API Verification](#9-api-verification)
10. [Member Onboarding](#10-member-onboarding)
11. [Production Hardening](#11-production-hardening)
12. [Monitoring Setup](#12-monitoring-setup)
13. [Security Audit](#13-security-audit)
14. [Documentation Review](#14-documentation-review)

---

## 1. Database Setup

### 1.1 Verify Database Connectivity

```bash
# Check PostgreSQL is running
psql -h 192.168.11.105 -U dbis_user -d dbis_core -c "SELECT version();"

# Verify connection string in .env
grep DATABASE_URL .env
```

**Expected Output**: PostgreSQL version and connection success

**If Failed**:
- Check PostgreSQL service status
- Verify network connectivity to database server
- Verify credentials in `.env` file

### 1.2 Verify Database Schema

```bash
# Check current schema version
psql -h 192.168.11.105 -U dbis_user -d dbis_core -c "SELECT * FROM _prisma_migrations ORDER BY finished_at DESC LIMIT 5;"
```

**Action**: Note the latest migration version

### 1.3 Backup Database (Production Only)

```bash
# Create backup before migration
pg_dump -h 192.168.11.105 -U dbis_user -d dbis_core > backup_$(date +%Y%m%d_%H%M%S).sql
```

**Action**: Store backup in secure location

---

## 2. Environment Configuration

### 2.1 Create/Update `.env` File

Add the following environment variables to `dbis_core/.env`:

```env
# AS4 Gateway Configuration
AS4_BASE_URL=https://as4.dbis.org
AS4_GATEWAY_PORT=8443
AS4_GATEWAY_HOST=0.0.0.0

# Certificate Paths
AS4_TLS_CERT_PATH=/etc/dbis/certs/as4-tls-cert.pem
AS4_TLS_KEY_PATH=/etc/dbis/certs/as4-tls-key.pem
AS4_SIGNING_CERT_PATH=/etc/dbis/certs/as4-signing-cert.pem
AS4_SIGNING_KEY_PATH=/etc/dbis/certs/as4-signing-key.pem
AS4_ENCRYPTION_CERT_PATH=/etc/dbis/certs/as4-encryption-cert.pem
AS4_ENCRYPTION_KEY_PATH=/etc/dbis/certs/as4-encryption-key.pem

# HSM Configuration (if using HSM)
HSM_ENABLED=true
HSM_PROVIDER=softhsm
HSM_SLOT=0
HSM_PIN=your-hsm-pin
HSM_LIBRARY_PATH=/usr/lib/softhsm/libsofthsm2.so

# Redis Configuration (for nonce tracking)
REDIS_URL=redis://localhost:6379
REDIS_PASSWORD=
AS4_NONCE_TTL=300
AS4_NONCE_CLEANUP_INTERVAL=3600

# ChainID 138 Configuration
CHAIN138_RPC_URL=http://192.168.11.250:8545
CHAIN138_WS_URL=ws://192.168.11.250:8546
CHAIN138_ANCHOR_INTERVAL=3600
CHAIN138_CONTRACT_ADDRESS=0x...

# Compliance Configuration
SANCTIONS_SCREENING_ENABLED=true
SANCTIONS_SCREENING_PROVIDER=internal
AML_CHECKS_ENABLED=true
AML_CHECKS_PROVIDER=internal

# Message Processing
AS4_MESSAGE_TIMEOUT=30000
AS4_MAX_MESSAGE_SIZE=10485760
AS4_RATE_LIMIT_PER_MEMBER=1000
AS4_RATE_LIMIT_WINDOW=3600

# Security
AS4_REPLAY_WINDOW_MINUTES=5
AS4_CERTIFICATE_VALIDATION_STRICT=true
AS4_REQUIRE_MESSAGE_SIGNATURE=true
AS4_REQUIRE_MESSAGE_ENCRYPTION=false

# Logging
AS4_LOG_LEVEL=info
AS4_AUDIT_LOG_ENABLED=true
AS4_PAYLOAD_VAULT_ENABLED=true
```

### 2.2 Verify Environment Variables

```bash
# Check all AS4 variables are set
cd dbis_core
node -e "require('dotenv').config(); console.log('AS4_BASE_URL:', process.env.AS4_BASE_URL);"
```

**Action**: Verify all required variables are set

---

## 3. Certificate Setup

### 3.1 Generate TLS Certificate (for DBIS)

```bash
# Create certificate directory
sudo mkdir -p /etc/dbis/certs
sudo chmod 700 /etc/dbis/certs

# Generate TLS certificate
openssl req -x509 -newkey rsa:2048 \
  -keyout /etc/dbis/certs/as4-tls-key.pem \
  -out /etc/dbis/certs/as4-tls-cert.pem \
  -days 365 -nodes \
  -subj "/CN=as4.dbis.org/O=DBIS/C=US"

# Set permissions
sudo chmod 600 /etc/dbis/certs/as4-tls-key.pem
sudo chmod 644 /etc/dbis/certs/as4-tls-cert.pem
```

### 3.2 Generate Signing Certificate

```bash
# Generate signing certificate
openssl req -x509 -newkey rsa:2048 \
  -keyout /etc/dbis/certs/as4-signing-key.pem \
  -out /etc/dbis/certs/as4-signing-cert.pem \
  -days 365 -nodes \
  -subj "/CN=DBIS AS4 Signing/O=DBIS/C=US"

# Set permissions
sudo chmod 600 /etc/dbis/certs/as4-signing-key.pem
sudo chmod 644 /etc/dbis/certs/as4-signing-cert.pem
```

### 3.3 Generate Encryption Certificate

```bash
# Generate encryption certificate
openssl req -x509 -newkey rsa:2048 \
  -keyout /etc/dbis/certs/as4-encryption-key.pem \
  -out /etc/dbis/certs/as4-encryption-cert.pem \
  -days 365 -nodes \
  -subj "/CN=DBIS AS4 Encryption/O=DBIS/C=US"

# Set permissions
sudo chmod 600 /etc/dbis/certs/as4-encryption-key.pem
sudo chmod 644 /etc/dbis/certs/as4-encryption-cert.pem
```

### 3.4 Calculate Certificate Fingerprints

```bash
# Calculate TLS fingerprint
openssl x509 -fingerprint -sha256 -noout -in /etc/dbis/certs/as4-tls-cert.pem

# Calculate signing fingerprint
openssl x509 -fingerprint -sha256 -noout -in /etc/dbis/certs/as4-signing-cert.pem

# Calculate encryption fingerprint
openssl x509 -fingerprint -sha256 -noout -in /etc/dbis/certs/as4-encryption-cert.pem
```

**Action**: Store fingerprints securely for Member Directory registration

### 3.5 HSM Setup (Production Only)

```bash
# Initialize HSM slot (if using SoftHSM)
softhsm2-util --init-token --slot 0 --label "DBIS-AS4" --pin your-pin --so-pin your-so-pin

# Import certificates to HSM
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --slot 0 --pin your-pin \
  --write-object /etc/dbis/certs/as4-signing-cert.pem \
  --type cert --id 01 --label "AS4-Signing"
```

**Action**: Configure HSM paths in environment variables

---

## 4. Database Migration

### 4.1 Review Migration File

```bash
# Review migration SQL
cat prisma/migrations/20260119000000_add_as4_settlement_models/migration.sql
```

**Action**: Verify migration SQL is correct

### 4.2 Run Migration (Development)

```bash
cd dbis_core

# Generate Prisma client
npx prisma generate

# Run migration
npx prisma migrate dev --name add_as4_settlement_models
```

**Expected Output**: 
- Migration applied successfully
- 6 new tables created

### 4.3 Run Migration (Production)

```bash
cd dbis_core

# Generate Prisma client
npx prisma generate

# Deploy migration (no prompt)
npx prisma migrate deploy
```

**Expected Output**: Migration applied successfully

### 4.4 Verify Tables Created

```bash
# Check tables exist
psql -h 192.168.11.105 -U dbis_user -d dbis_core -c "
SELECT table_name 
FROM information_schema.tables 
WHERE table_schema = 'public' 
AND table_name LIKE 'as4_%'
ORDER BY table_name;
"
```

**Expected Output**: 6 tables listed:
- as4_member
- as4_member_certificate
- as4_settlement_instruction
- as4_advice
- as4_payload_vault
- as4_replay_nonce

### 4.5 Verify Indexes

```bash
# Check indexes
psql -h 192.168.11.105 -U dbis_user -d dbis_core -c "
SELECT indexname, tablename 
FROM pg_indexes 
WHERE tablename LIKE 'as4_%'
ORDER BY tablename, indexname;
"
```

**Action**: Verify all indexes are created

---

## 5. Marketplace Seeding

### 5.1 Review Seed Script

```bash
# Review seed script
cat scripts/seed-as4-settlement-marketplace-offering.ts
```

**Action**: Verify offering details are correct

### 5.2 Run Seed Script

```bash
cd dbis_core
npx ts-node scripts/seed-as4-settlement-marketplace-offering.ts
```

**Expected Output**: 
```
Seeding AS4 Settlement Marketplace Offering...
AS4 Settlement Marketplace Offering created: AS4-SETTLEMENT-MASTER
```

### 5.3 Verify Offering in Database

```bash
# Check offering exists
psql -h 192.168.11.105 -U dbis_user -d dbis_core -c "
SELECT offeringId, name, status, capacityTier 
FROM \"IruOffering\" 
WHERE offeringId = 'AS4-SETTLEMENT-MASTER';
"
```

**Expected Output**: Offering record with status 'active'

### 5.4 Verify in Marketplace UI

**Action**: 
1. Access Sankofa Phoenix Marketplace
2. Navigate to offerings
3. Verify "AS4 Settlement Master Service" is visible
4. Verify capacity tier is 1
5. Verify pricing is displayed

---

## 6. Service Configuration

### 6.1 Redis Setup (for Nonce Tracking)

```bash
# Check Redis is running
redis-cli ping

# If not running, start Redis
sudo systemctl start redis
sudo systemctl enable redis

# Test connection
redis-cli -h localhost -p 6379 ping
```

**Expected Output**: `PONG`

### 6.2 Configure Redis (if needed)

```bash
# Edit Redis config
sudo nano /etc/redis/redis.conf

# Set maxmemory and eviction policy
maxmemory 256mb
maxmemory-policy allkeys-lru

# Restart Redis
sudo systemctl restart redis
```

### 6.3 ChainID 138 RPC Verification

```bash
# Test RPC connection
curl -X POST http://192.168.11.250:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'
```

**Expected Output**: `{"jsonrpc":"2.0","id":1,"result":"0x8a"}` (138 in hex)

**If Failed**:
- Verify ChainID 138 node is running
- Check network connectivity
- Verify RPC endpoint URL

---

## 7. Testing

### 7.1 Unit Tests

```bash
cd dbis_core

# Run all tests
npm test

# Run only AS4 tests
npm test -- as4-settlement.test.ts

# Run with coverage
npm test -- --coverage as4-settlement.test.ts
```

**Expected Output**: All tests pass

**If Tests Fail**:
- Review error messages
- Check database connectivity
- Verify test data setup

### 7.2 Integration Tests

```bash
# Run integration tests
npm test -- --testPathPattern=integration

# Run specific integration test
npm test -- src/__tests__/integration/settlement/as4-settlement.test.ts
```

**Action**: Verify all integration tests pass

### 7.3 Manual API Testing

See [Section 9: API Verification](#9-api-verification) for detailed API tests

---

## 8. Server Startup

### 8.1 Build Application

```bash
cd dbis_core

# Build TypeScript
npm run build

# Verify build succeeded
ls -la dist/
```

**Action**: Verify `dist/` directory contains compiled files

### 8.2 Start Development Server

```bash
cd dbis_core

# Start dev server
npm run dev
```

**Expected Output**: 
```
Server running on port 3000
AS4 Gateway initialized
Member Directory initialized
Settlement Core initialized
```

### 8.3 Start Production Server

```bash
cd dbis_core

# Start production server
NODE_ENV=production npm start
```

**Action**: Verify server starts without errors

### 8.4 Verify Server Health

```bash
# Check health endpoint
curl http://localhost:3000/health
```

**Expected Output**: 
```json
{
  "status": "healthy",
  "timestamp": "2026-01-19T...",
  "version": "1.0.0",
  "database": "connected",
  "hsm": "available"
}
```

---

## 9. API Verification

### 9.1 Health Check

```bash
curl -X GET http://localhost:3000/health
```

**Expected**: HTTP 200 with health status

### 9.2 Register Test Member

```bash
curl -X POST http://localhost:3000/api/v1/as4/directory/members \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "memberId": "TEST-MEMBER-001",
    "organizationName": "Test Bank",
    "as4EndpointUrl": "https://test-bank.example.com/as4",
    "tlsCertFingerprint": "AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99",
    "allowedMessageTypes": ["DBIS.SI.202", "DBIS.SI.202COV"],
    "routingGroups": ["DEFAULT"],
    "capacityTier": 3
  }'
```

**Expected**: HTTP 201 with member record

### 9.3 Get Member

```bash
curl -X GET http://localhost:3000/api/v1/as4/directory/members/TEST-MEMBER-001 \
  -H "Authorization: Bearer YOUR_TOKEN"
```

**Expected**: HTTP 200 with member details

### 9.4 Submit Test Instruction

```bash
curl -X POST http://localhost:3000/api/v1/as4/settlement/instructions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "fromMemberId": "TEST-MEMBER-001",
    "payloadHash": "abc123def456",
    "signatureEvidence": {},
    "as4ReceiptEvidence": {},
    "message": {
      "MessageId": "MSG-TEST-001",
      "BusinessType": "DBIS.SI.202",
      "CreatedAt": "2026-01-19T12:00:00Z",
      "FromMemberId": "TEST-MEMBER-001",
      "ToMemberId": "DBIS",
      "CorrelationId": "CORR-001",
      "ReplayNonce": "nonce-123",
      "SchemaVersion": "1.0",
      "Instr": {
        "InstrId": "INSTR-TEST-001",
        "ValueDate": "2026-01-20",
        "Currency": "USD",
        "Amount": "1000.00",
        "DebtorAccount": "MSA:TEST-MEMBER-001:USD",
        "CreditorAccount": "MSA:TEST-MEMBER-002:USD",
        "Charges": "SHA",
        "PurposeCode": "SETT"
      }
    }
  }'
```

**Expected**: HTTP 202 with instruction acceptance

### 9.5 Get Instruction Status

```bash
curl -X GET "http://localhost:3000/api/v1/as4/settlement/instructions/INSTR-TEST-001?fromMemberId=TEST-MEMBER-001" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

**Expected**: HTTP 200 with instruction status

### 9.6 Generate Statement

```bash
curl -X GET "http://localhost:3000/api/v1/as4/settlement/statements?memberId=TEST-MEMBER-001&accountId=MSA:TEST-MEMBER-001:USD&startDate=2026-01-01&endDate=2026-01-31" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

**Expected**: HTTP 200 with statement data

### 9.7 Export Audit Trail

```bash
curl -X GET "http://localhost:3000/api/v1/as4/settlement/audit/INSTR-TEST-001?fromMemberId=TEST-MEMBER-001" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

**Expected**: HTTP 200 with audit trail

---

## 10. Member Onboarding

### 10.1 Marketplace Subscription

**Action**:
1. Member accesses Sankofa Phoenix Marketplace
2. Member submits inquiry for AS4 Settlement offering
3. Member completes qualification process
4. Member subscribes to offering

### 10.2 Automated Provisioning

**Action**: 
1. Deployment orchestrator detects AS4 Settlement subscription
2. Provisioning service creates member record
3. Member receives credentials and endpoint information

### 10.3 Certificate Registration

```bash
# Member submits certificate via API
curl -X POST http://localhost:3000/api/v1/as4/directory/members/MEMBER-XXX/certificates \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer MEMBER_TOKEN" \
  -d '{
    "certificateType": "TLS",
    "fingerprint": "MEMBER_CERT_FINGERPRINT",
    "certificateData": "-----BEGIN CERTIFICATE-----\n...",
    "validFrom": "2026-01-19T00:00:00Z",
    "validTo": "2027-01-19T23:59:59Z"
  }'
```

**Action**: Verify certificate is registered and active

### 10.4 Test Connectivity

**Action**:
1. Member tests AS4 endpoint connectivity
2. Member sends test message
3. Verify receipt is received
4. Verify message is processed

### 10.5 Production Activation

**Action**:
1. Complete test transactions
2. Verify all compliance checks pass
3. Activate member for production
4. Monitor first production transactions

---

## 11. Production Hardening

### 11.1 High Availability Setup

```bash
# Deploy multiple AS4 gateway instances
# Configure load balancer
# Set up health checks
# Configure auto-scaling
```

**Action**: 
- Deploy 3+ gateway instances
- Configure load balancer with health checks
- Set up auto-scaling rules
- Configure session affinity if needed

### 11.2 Database Replication

```bash
# Set up PostgreSQL replication
# Configure read replicas
# Set up failover
```

**Action**:
- Configure primary-replica setup
- Test failover procedures
- Monitor replication lag

### 11.3 Redis Cluster

```bash
# Set up Redis cluster
# Configure replication
# Set up failover
```

**Action**:
- Deploy Redis cluster (3+ nodes)
- Configure replication
- Test failover
- Monitor cluster health

### 11.4 Backup Configuration

```bash
# Configure automated backups
# Set up backup retention
# Test restore procedures
```

**Action**:
- Daily full backups
- Hourly incremental backups
- 30-day retention
- Test restore monthly

### 11.5 Security Hardening

```bash
# Review security configuration
# Enable all security features
# Configure firewall rules
# Set up DDoS protection
```

**Action**:
- Enable HSM for production
- Configure strict certificate validation
- Enable message encryption
- Set up firewall rules
- Configure DDoS protection (CloudFlare/AWS Shield)

---

## 12. Monitoring Setup

### 12.1 Prometheus Configuration

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'as4-settlement'
    static_configs:
      - targets: ['localhost:3000']
    metrics_path: '/api/v1/as4/metrics'
```

**Action**: Configure Prometheus to scrape AS4 metrics

### 12.2 Key Metrics to Monitor

- Message processing latency (P99)
- Instruction success rate
- Failed instruction rate
- Certificate expiration warnings
- System availability
- Database connection pool usage
- Redis connection status
- ChainID 138 anchoring status

### 12.3 Alerting Rules

```yaml
# alerts.yml
groups:
  - name: as4_settlement
    rules:
      - alert: AS4HighLatency
        expr: as4_message_latency_p99 > 5
        for: 5m
        
      - alert: AS4HighFailureRate
        expr: rate(as4_instructions_failed[5m]) > 0.01
        
      - alert: AS4CertificateExpiring
        expr: as4_certificate_days_until_expiry < 30
```

**Action**: Configure alerting rules in Prometheus

### 12.4 Log Aggregation

```bash
# Configure log aggregation (ELK/Loki)
# Set up log retention
# Configure log parsing
```

**Action**:
- Set up ELK stack or Loki
- Configure log shipping
- Set up log retention (7 years for audit logs)
- Configure log parsing for AS4 messages

### 12.5 Dashboard Creation

**Action**: Create Grafana dashboards for:
- AS4 Gateway metrics
- Settlement processing metrics
- Member activity metrics
- System health metrics
- Compliance metrics

---

## 13. Security Audit

### 13.1 Code Security Review

```bash
# Run security scanning
npm audit
npm audit fix

# Run SAST tools
# Configure Snyk, SonarQube, etc.
```

**Action**: 
- Review all security vulnerabilities
- Fix high/critical issues
- Document accepted risks

### 13.2 Penetration Testing

**Action**:
- Engage security team for pen testing
- Test AS4 endpoint security
- Test certificate validation
- Test message signing/encryption
- Test replay protection
- Test rate limiting

### 13.3 Compliance Review

**Action**:
- Review compliance with rulebook
- Verify audit trail completeness
- Verify evidence storage
- Review sanctions screening integration
- Review AML/CTF checks

### 13.4 Access Control Review

**Action**:
- Review RBAC configuration
- Verify HSM access controls
- Review certificate management access
- Review audit log access

---

## 14. Documentation Review

### 14.1 Technical Documentation

**Action**: Review and update:
- [ ] API documentation (Swagger/OpenAPI)
- [ ] Message schema documentation
- [ ] Integration guides
- [ ] Architecture diagrams

### 14.2 Operational Documentation

**Action**: Review and update:
- [ ] Operational runbooks
- [ ] Incident response procedures
- [ ] Deployment procedures
- [ ] Troubleshooting guides

### 14.3 User Documentation

**Action**: Create:
- [ ] Member onboarding guide
- [ ] API integration guide
- [ ] Certificate management guide
- [ ] FAQ document

### 14.4 Compliance Documentation

**Action**: Review:
- [ ] Member rulebook
- [ ] PKI/CA model documentation
- [ ] Security controls documentation
- [ ] Audit procedures

---

## 15. Performance Testing

### 15.1 Load Testing

```bash
# Use k6, JMeter, or similar
# Test message processing throughput
# Test concurrent member connections
# Test database load
```

**Action**:
- Test with 100 concurrent members
- Test 1000 messages/second throughput
- Test P99 latency under load
- Identify bottlenecks

### 15.2 Stress Testing

**Action**:
- Test system behavior under extreme load
- Test failover scenarios
- Test recovery procedures
- Document limits and thresholds

### 15.3 Endurance Testing

**Action**:
- Run system for 24+ hours
- Monitor memory leaks
- Monitor database growth
- Monitor log file sizes

---

## 16. Disaster Recovery

### 16.1 DR Plan Documentation

**Action**: Document:
- Recovery time objectives (RTO)
- Recovery point objectives (RPO)
- Backup procedures
- Restore procedures
- Failover procedures

### 16.2 DR Testing

**Action**:
- Test database restore
- Test service failover
- Test certificate recovery
- Test audit log recovery
- Document test results

---

## 17. Go-Live Checklist

### 17.1 Pre-Go-Live

- [ ] All tests passing
- [ ] Security audit complete
- [ ] Performance testing complete
- [ ] Documentation complete
- [ ] Team training complete
- [ ] Monitoring configured
- [ ] Alerting configured
- [ ] Backup procedures tested
- [ ] DR procedures tested

### 17.2 Go-Live

- [ ] Database migration applied
- [ ] Marketplace offering active
- [ ] Services running
- [ ] Monitoring active
- [ ] Support team on standby
- [ ] Communication sent to stakeholders

### 17.3 Post-Go-Live

- [ ] Monitor first 24 hours closely
- [ ] Review all alerts
- [ ] Verify all transactions processed
- [ ] Collect feedback from members
- [ ] Document any issues
- [ ] Schedule post-mortem

---

## 18. Ongoing Operations

### 18.1 Daily Tasks

- [ ] Review health checks
- [ ] Review error logs
- [ ] Check certificate expiration warnings
- [ ] Review member activity
- [ ] Monitor system metrics

### 18.2 Weekly Tasks

- [ ] Review performance metrics
- [ ] Review security alerts
- [ ] Review compliance reports
- [ ] Update documentation if needed

### 18.3 Monthly Tasks

- [ ] Review audit logs
- [ ] Review member onboarding
- [ ] Review system capacity
- [ ] Update runbooks
- [ ] Security review

### 18.4 Quarterly Tasks

- [ ] Disaster recovery testing
- [ ] Security audit
- [ ] Performance review
- [ ] Documentation review
- [ ] Capacity planning

---

## 19. Troubleshooting Common Issues

### 19.1 Database Connection Issues

**Symptoms**: Migration fails, services can't connect

**Actions**:
1. Check PostgreSQL service status
2. Verify network connectivity
3. Check credentials in `.env`
4. Verify database exists
5. Check firewall rules

### 19.2 Certificate Issues

**Symptoms**: TLS handshake fails, signature validation fails

**Actions**:
1. Verify certificate paths in `.env`
2. Check certificate permissions
3. Verify certificate validity
4. Check certificate fingerprints match
5. Review certificate expiration

### 19.3 Redis Connection Issues

**Symptoms**: Nonce validation fails, replay protection not working

**Actions**:
1. Check Redis service status
2. Verify Redis URL in `.env`
3. Test Redis connectivity
4. Check Redis memory usage
5. Review Redis logs

### 19.4 Message Processing Failures

**Symptoms**: Instructions rejected, errors in logs

**Actions**:
1. Check instruction logs
2. Verify member status
3. Check compliance gates
4. Review liquidity/limits
5. Check posting engine status

---

## 20. Support and Maintenance

### 20.1 Support Channels

- **Email**: as4-support@dbis.org
- **Slack**: #as4-settlement
- **On-call**: PagerDuty rotation
- **Documentation**: `/docs/settlement/as4/`

### 20.2 Maintenance Windows

- **Scheduled**: Monthly, 2-hour window
- **Emergency**: As needed
- **Notification**: 7 days advance notice

### 20.3 Version Updates

- **Process**: Follow semantic versioning
- **Testing**: Test in staging first
- **Deployment**: Blue-green deployment
- **Rollback**: Automated rollback on failure

---

## Summary Checklist

### Immediate (Before Go-Live)
- [ ] Database migration applied
- [ ] Marketplace offering seeded
- [ ] Environment variables configured
- [ ] Certificates generated and installed
- [ ] Redis configured
- [ ] ChainID 138 RPC verified
- [ ] All tests passing
- [ ] Server starts successfully
- [ ] API endpoints verified
- [ ] Monitoring configured

### Short-term (First Week)
- [ ] Member onboarding tested
- [ ] Production transactions monitored
- [ ] Performance metrics reviewed
- [ ] Security audit completed
- [ ] Documentation finalized

### Long-term (Ongoing)
- [ ] Regular security audits
- [ ] Performance optimization
- [ ] Capacity planning
- [ ] Feature enhancements
- [ ] Member feedback integration

---

**End of Detailed Next Steps**