Apply Composer changes: comprehensive API updates, migrations, middleware, and infrastructure improvements
- Add comprehensive database migrations (001-024) for schema evolution - Enhance API schema with expanded type definitions and resolvers - Add new middleware: audit logging, rate limiting, MFA enforcement, security, tenant auth - Implement new services: AI optimization, billing, blockchain, compliance, marketplace - Add adapter layer for cloud integrations (Cloudflare, Kubernetes, Proxmox, storage) - Update Crossplane provider with enhanced VM management capabilities - Add comprehensive test suite for API endpoints and services - Update frontend components with improved GraphQL subscriptions and real-time updates - Enhance security configurations and headers (CSP, CORS, etc.) - Update documentation and configuration files - Add new CI/CD workflows and validation scripts - Implement design system improvements and UI enhancements
This commit is contained in:
defiQUG
224
docs/runbooks/DATA_RETENTION_POLICY.md
Normal file
224
docs/runbooks/DATA_RETENTION_POLICY.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# Data Retention Policy
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines data retention policies for the Sankofa Phoenix platform to ensure compliance with regulatory requirements and optimize storage costs.
|
||||
|
||||
## Retention Periods
|
||||
|
||||
### Application Data
|
||||
|
||||
#### User Data
|
||||
- **Active Users**: Retained indefinitely while account is active
|
||||
- **Inactive Users**: Retained for 7 years after last login
|
||||
- **Deleted Users**: Soft delete for 90 days, then permanent deletion
|
||||
- **User Activity Logs**: 2 years
|
||||
|
||||
#### Tenant Data
|
||||
- **Active Tenants**: Retained indefinitely while tenant is active
|
||||
- **Suspended Tenants**: Retained for 1 year after suspension
|
||||
- **Deleted Tenants**: Soft delete for 90 days, then permanent deletion
|
||||
|
||||
#### Resource Data
|
||||
- **Active Resources**: Retained indefinitely
|
||||
- **Deleted Resources**: Retained for 90 days for recovery purposes
|
||||
- **Resource History**: 1 year
|
||||
|
||||
### Audit and Compliance Data
|
||||
|
||||
#### Audit Logs
|
||||
- **Security Events**: 7 years (compliance requirement)
|
||||
- **Authentication Logs**: 2 years
|
||||
- **Authorization Logs**: 2 years
|
||||
- **Data Access Logs**: 2 years
|
||||
- **Administrative Actions**: 7 years
|
||||
|
||||
#### Compliance Data
|
||||
- **STIG Compliance Reports**: 7 years
|
||||
- **RMF Documentation**: 7 years
|
||||
- **Incident Reports**: 7 years
|
||||
- **Risk Assessments**: 7 years
|
||||
|
||||
### Operational Data
|
||||
|
||||
#### Application Logs
|
||||
- **Application Logs (Loki)**: 30 days
|
||||
- **Access Logs**: 90 days
|
||||
- **Error Logs**: 90 days
|
||||
- **Performance Logs**: 30 days
|
||||
|
||||
#### Metrics
|
||||
- **Prometheus Metrics**: 30 days (raw)
|
||||
- **Aggregated Metrics**: 1 year
|
||||
- **Custom Metrics**: 90 days
|
||||
|
||||
#### Backups
|
||||
- **Database Backups**: 7 days (daily), 4 weeks (weekly), 12 months (monthly)
|
||||
- **Configuration Backups**: 90 days
|
||||
- **Disaster Recovery Backups**: 7 years
|
||||
|
||||
### Blockchain Data
|
||||
|
||||
#### Transaction History
|
||||
- **All Transactions**: Retained indefinitely (immutable)
|
||||
- **Transaction Logs**: 7 years
|
||||
|
||||
#### Smart Contract Data
|
||||
- **Contract State**: Retained indefinitely
|
||||
- **Contract Events**: 7 years
|
||||
|
||||
## Data Deletion Procedures
|
||||
|
||||
### Automated Deletion
|
||||
|
||||
#### Scheduled Cleanup Jobs
|
||||
```bash
|
||||
# Run daily cleanup job
|
||||
kubectl create cronjob cleanup-old-data \
|
||||
--image=postgres:14-alpine \
|
||||
--schedule="0 3 * * *" \
|
||||
--restart=OnFailure \
|
||||
-- /bin/bash -c "psql $DATABASE_URL -f /scripts/cleanup-old-data.sql"
|
||||
```
|
||||
|
||||
#### Cleanup Scripts
|
||||
- **User Data Cleanup**: Runs monthly, deletes users inactive > 7 years
|
||||
- **Log Cleanup**: Runs daily, deletes logs older than retention period
|
||||
- **Backup Cleanup**: Runs daily, deletes backups older than retention period
|
||||
|
||||
### Manual Deletion
|
||||
|
||||
#### User-Requested Deletion
|
||||
1. User submits deletion request
|
||||
2. Account marked for deletion
|
||||
3. 30-day grace period for account recovery
|
||||
4. Data anonymized after grace period
|
||||
5. Permanent deletion after 90 days
|
||||
|
||||
#### Administrative Deletion
|
||||
1. Admin initiates deletion
|
||||
2. Approval required for sensitive data
|
||||
3. Data exported for compliance (if required)
|
||||
4. Data deleted according to retention policy
|
||||
|
||||
## Compliance Requirements
|
||||
|
||||
### GDPR (General Data Protection Regulation)
|
||||
- **Right to Erasure**: Users can request data deletion
|
||||
- **Data Portability**: Users can export their data
|
||||
- **Retention Limitation**: Data retained only as long as necessary
|
||||
|
||||
### SOX (Sarbanes-Oxley Act)
|
||||
- **Financial Records**: 7 years retention
|
||||
- **Audit Trails**: 7 years retention
|
||||
|
||||
### HIPAA (Health Insurance Portability and Accountability Act)
|
||||
- **PHI Data**: 6 years minimum retention
|
||||
- **Access Logs**: 6 years minimum retention
|
||||
|
||||
### DoD/MilSpec Compliance
|
||||
- **Security Logs**: 7 years retention
|
||||
- **Audit Trails**: 7 years retention
|
||||
- **Compliance Reports**: 7 years retention
|
||||
|
||||
## Implementation
|
||||
|
||||
### Database Retention
|
||||
|
||||
#### Automated Cleanup Queries
|
||||
```sql
|
||||
-- Delete inactive users (7 years)
|
||||
DELETE FROM users
|
||||
WHERE last_login < NOW() - INTERVAL '7 years'
|
||||
AND status = 'INACTIVE';
|
||||
|
||||
-- Delete old audit logs (after 2 years, archive first)
|
||||
INSERT INTO audit_logs_archive
|
||||
SELECT * FROM audit_logs
|
||||
WHERE created_at < NOW() - INTERVAL '2 years';
|
||||
|
||||
DELETE FROM audit_logs
|
||||
WHERE created_at < NOW() - INTERVAL '2 years';
|
||||
```
|
||||
|
||||
### Log Retention
|
||||
|
||||
#### Loki Retention Configuration
|
||||
```yaml
|
||||
# gitops/apps/monitoring/loki-config.yaml
|
||||
retention_period: 30d
|
||||
retention_stream:
|
||||
- selector: '{job="api"}'
|
||||
period: 90d
|
||||
- selector: '{job="portal"}'
|
||||
period: 90d
|
||||
```
|
||||
|
||||
#### Prometheus Retention Configuration
|
||||
```yaml
|
||||
# gitops/apps/monitoring/prometheus-config.yaml
|
||||
retention: 30d
|
||||
retentionSize: 50GB
|
||||
```
|
||||
|
||||
### Backup Retention
|
||||
|
||||
#### Backup Cleanup Script
|
||||
```bash
|
||||
# Delete backups older than retention period
|
||||
find /backups/postgres -name "*.sql.gz" -mtime +7 -delete
|
||||
find /backups/postgres -name "*.sql.gz" -mtime +30 -delete # Weekly backups
|
||||
find /backups/postgres -name "*.sql.gz" -mtime +365 -delete # Monthly backups
|
||||
```
|
||||
|
||||
## Data Archival
|
||||
|
||||
### Long-Term Storage
|
||||
|
||||
#### Archived Data Storage
|
||||
- **Location**: S3 Glacier or equivalent
|
||||
- **Format**: Compressed, encrypted archives
|
||||
- **Retention**: Per compliance requirements
|
||||
- **Access**: On-demand restoration
|
||||
|
||||
#### Archive Process
|
||||
1. Data identified for archival
|
||||
2. Data compressed and encrypted
|
||||
3. Data uploaded to archival storage
|
||||
4. Index updated with archive location
|
||||
5. Original data deleted after verification
|
||||
|
||||
## Monitoring and Compliance
|
||||
|
||||
### Retention Policy Compliance
|
||||
|
||||
#### Automated Checks
|
||||
- Daily verification of retention policies
|
||||
- Alert on data older than retention period
|
||||
- Report on data deletion activities
|
||||
|
||||
#### Compliance Reporting
|
||||
- Monthly retention compliance report
|
||||
- Quarterly audit of data retention
|
||||
- Annual compliance review
|
||||
|
||||
## Exceptions and Extensions
|
||||
|
||||
### Legal Hold
|
||||
- Data subject to legal hold cannot be deleted
|
||||
- Legal hold overrides retention policy
|
||||
- Legal hold must be documented
|
||||
- Data released after legal hold lifted
|
||||
|
||||
### Business Requirements
|
||||
- Extended retention for business-critical data
|
||||
- Approval required for extensions
|
||||
- Extensions documented and reviewed annually
|
||||
|
||||
## Contact
|
||||
|
||||
For questions about data retention:
|
||||
- **Data Protection Officer**: dpo@sankofa.nexus
|
||||
- **Compliance Team**: compliance@sankofa.nexus
|
||||
- **Legal Team**: legal@sankofa.nexus
|
||||
|
||||
Reference in New Issue
Block a user