Initial commit: loc_az_hci (smom-dbis-138 excluded via .gitignore)
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
321
docs/operations/runbooks/gitops-workflow.md
Normal file
321
docs/operations/runbooks/gitops-workflow.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# GitOps Workflow Runbook
|
||||
|
||||
## Overview
|
||||
|
||||
This runbook describes the GitOps workflow using Flux for managing Kubernetes deployments.
|
||||
|
||||
## GitOps Architecture
|
||||
|
||||
```
|
||||
Git Repository (Gitea/GitLab)
|
||||
│
|
||||
│ (Poll/Sync)
|
||||
│
|
||||
▼
|
||||
Flux Controller (Kubernetes)
|
||||
│
|
||||
│ (Apply)
|
||||
│
|
||||
▼
|
||||
Kubernetes Cluster
|
||||
│
|
||||
│ (Deploy)
|
||||
│
|
||||
▼
|
||||
Application Pods
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Making Changes
|
||||
|
||||
#### Update Application Configuration
|
||||
|
||||
1. Clone Git repository:
|
||||
```bash
|
||||
git clone http://git.local:3000/user/gitops-repo.git
|
||||
cd gitops-repo
|
||||
```
|
||||
|
||||
2. Edit Helm chart values:
|
||||
```bash
|
||||
# Edit values.yaml
|
||||
vim gitops/apps/besu/values.yaml
|
||||
```
|
||||
|
||||
3. Commit and push:
|
||||
```bash
|
||||
git add gitops/apps/besu/values.yaml
|
||||
git commit -m "Update Besu configuration"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
#### Add New Application
|
||||
|
||||
1. Add Helm chart to repository:
|
||||
```bash
|
||||
cp -r /path/to/new-chart gitops/apps/new-app/
|
||||
```
|
||||
|
||||
2. Create Flux Kustomization:
|
||||
```bash
|
||||
# Create gitops/apps/new-app/kustomization.yaml
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: new-app
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
path: ./apps/new-app
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
```
|
||||
|
||||
3. Commit and push:
|
||||
```bash
|
||||
git add gitops/apps/new-app/
|
||||
git commit -m "Add new application"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
### 2. Monitoring Sync Status
|
||||
|
||||
#### Check Flux Status
|
||||
|
||||
```bash
|
||||
# Check Flux pods
|
||||
kubectl get pods -n flux-system
|
||||
|
||||
# Check Git repository status
|
||||
kubectl get gitrepository -n flux-system
|
||||
kubectl describe gitrepository flux-system -n flux-system
|
||||
|
||||
# Check Kustomization status
|
||||
kubectl get kustomization -n flux-system
|
||||
kubectl describe kustomization <app-name> -n flux-system
|
||||
```
|
||||
|
||||
#### View Sync Events
|
||||
|
||||
```bash
|
||||
# Watch Flux events
|
||||
kubectl get events -n flux-system --sort-by='.lastTimestamp'
|
||||
|
||||
# View Flux logs
|
||||
kubectl logs -n flux-system -l app=flux -f
|
||||
```
|
||||
|
||||
### 3. Troubleshooting
|
||||
|
||||
#### Sync Not Happening
|
||||
|
||||
**Check Git repository access**:
|
||||
```bash
|
||||
kubectl get gitrepository flux-system -n flux-system -o yaml
|
||||
kubectl describe gitrepository flux-system -n flux-system
|
||||
```
|
||||
|
||||
**Check authentication**:
|
||||
```bash
|
||||
# For HTTPS with token
|
||||
kubectl get secret -n flux-system
|
||||
|
||||
# For SSH
|
||||
kubectl get secret flux-system -n flux-system -o yaml
|
||||
```
|
||||
|
||||
#### Application Not Deploying
|
||||
|
||||
**Check Kustomization**:
|
||||
```bash
|
||||
kubectl get kustomization <app-name> -n flux-system
|
||||
kubectl describe kustomization <app-name> -n flux-system
|
||||
```
|
||||
|
||||
**Check Helm release**:
|
||||
```bash
|
||||
kubectl get helmrelease -n <namespace>
|
||||
kubectl describe helmrelease <app-name> -n <namespace>
|
||||
```
|
||||
|
||||
#### Manual Sync Trigger
|
||||
|
||||
```bash
|
||||
# Trigger immediate sync
|
||||
flux reconcile source git flux-system
|
||||
flux reconcile kustomization <app-name>
|
||||
```
|
||||
|
||||
### 4. Best Practices
|
||||
|
||||
#### Repository Structure
|
||||
|
||||
```
|
||||
gitops-repo/
|
||||
├── infrastructure/
|
||||
│ ├── namespace.yaml
|
||||
│ ├── ingress-controller.yaml
|
||||
│ └── cert-manager.yaml
|
||||
└── apps/
|
||||
├── besu/
|
||||
│ ├── Chart.yaml
|
||||
│ ├── values.yaml
|
||||
│ └── templates/
|
||||
├── firefly/
|
||||
└── ...
|
||||
```
|
||||
|
||||
#### Branch Strategy
|
||||
|
||||
- **main**: Production deployments
|
||||
- **staging**: Staging environment
|
||||
- **develop**: Development environment
|
||||
|
||||
#### Change Management
|
||||
|
||||
1. Create feature branch
|
||||
2. Make changes
|
||||
3. Test in development
|
||||
4. Merge to staging
|
||||
5. Promote to production
|
||||
|
||||
### 5. Common Operations
|
||||
|
||||
#### Suspend Sync
|
||||
|
||||
```bash
|
||||
# Suspend specific application
|
||||
flux suspend kustomization <app-name>
|
||||
|
||||
# Resume
|
||||
flux resume kustomization <app-name>
|
||||
```
|
||||
|
||||
#### Rollback Changes
|
||||
|
||||
```bash
|
||||
# Revert Git commit
|
||||
git revert <commit-hash>
|
||||
git push origin main
|
||||
|
||||
# Or manually edit and push
|
||||
```
|
||||
|
||||
#### Update Helm Chart
|
||||
|
||||
```bash
|
||||
# Update chart version in values.yaml
|
||||
# Commit and push
|
||||
git add gitops/apps/<app>/values.yaml
|
||||
git commit -m "Update <app> to version X.Y.Z"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
### 6. Azure Arc GitOps Integration
|
||||
|
||||
#### Configure GitOps in Azure Portal
|
||||
|
||||
1. Navigate to: Azure Arc → Kubernetes → Your cluster
|
||||
2. Go to "GitOps" section
|
||||
3. Add configuration:
|
||||
- Repository URL
|
||||
- Branch
|
||||
- Path
|
||||
- Authentication
|
||||
|
||||
#### View GitOps Status in Azure
|
||||
|
||||
```bash
|
||||
az k8s-extension show \
|
||||
--resource-group HC-Stack \
|
||||
--cluster-name proxmox-k3s-cluster \
|
||||
--cluster-type connectedClusters \
|
||||
--name flux
|
||||
```
|
||||
|
||||
### 7. Security
|
||||
|
||||
#### Secret Management
|
||||
|
||||
**Option 1: Kubernetes Secrets** (not recommended for production):
|
||||
```bash
|
||||
kubectl create secret generic app-secret \
|
||||
--from-literal=password=secret-value \
|
||||
-n <namespace>
|
||||
```
|
||||
|
||||
**Option 2: Sealed Secrets**:
|
||||
```bash
|
||||
# Install Sealed Secrets controller
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml
|
||||
|
||||
# Create sealed secret
|
||||
kubeseal < secret.yaml > sealed-secret.yaml
|
||||
```
|
||||
|
||||
**Option 3: External Secrets Operator**:
|
||||
- Integrate with Azure Key Vault
|
||||
- Use External Secrets Operator
|
||||
|
||||
#### RBAC
|
||||
|
||||
Configure Flux RBAC:
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: flux-<namespace>
|
||||
namespace: <namespace>
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["*"]
|
||||
verbs: ["*"]
|
||||
```
|
||||
|
||||
### 8. Monitoring
|
||||
|
||||
#### Set Up Alerts
|
||||
|
||||
```bash
|
||||
# Create alert for sync failures
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: flux-sync-alerts
|
||||
spec:
|
||||
groups:
|
||||
- name: flux
|
||||
rules:
|
||||
- alert: FluxSyncFailed
|
||||
expr: flux_kustomization_status_condition{status="False"} == 1
|
||||
annotations:
|
||||
summary: "Flux sync failed"
|
||||
EOF
|
||||
```
|
||||
|
||||
### 9. Disaster Recovery
|
||||
|
||||
#### Backup Git Repository
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone --mirror http://git.local:3000/user/gitops-repo.git
|
||||
|
||||
# Backup to external location
|
||||
tar -czf gitops-backup-$(date +%Y%m%d).tar.gz gitops-repo.git
|
||||
```
|
||||
|
||||
#### Restore from Backup
|
||||
|
||||
```bash
|
||||
# Restore repository
|
||||
tar -xzf gitops-backup-YYYYMMDD.tar.gz
|
||||
cd gitops-repo.git
|
||||
git remote set-url origin http://git.local:3000/user/gitops-repo.git
|
||||
git push --mirror
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user