scripts/NEXT_STEPS_DETAILED.md

# Detailed Next Steps - Scripts Modularization

**Date**: 2025-01-27
**Status**: Structure Complete - Ready for Implementation
**Priority**: High

---

## 📋 Complete Implementation Checklist

### Phase 1: Script Migration (Move Files)

#### 1.1 Migration Scripts → `migration/`

- [ ] **Move `migrate-readme.sh`**
  ```bash
  mv scripts/migrate-readme.sh scripts/migration/migrate-readme.sh
  ```

- [ ] **Move `migrate-terraform.sh`**
  ```bash
  mv scripts/migrate-terraform.sh scripts/migration/migrate-terraform.sh
  ```

- [ ] **Move `migrate-to-api-gateway.sh`**
  ```bash
  mv scripts/migrate-to-api-gateway.sh scripts/migration/migrate-to-api-gateway.sh
  ```

- [ ] **Move `migrate-to-k8s.sh`**
  ```bash
  mv scripts/migrate-to-k8s.sh scripts/migration/migrate-to-k8s.sh
  ```

- [ ] **Move `migrate-to-monitoring.sh`**
  ```bash
  mv scripts/migrate-to-monitoring.sh scripts/migration/migrate-to-monitoring.sh
  ```

- [ ] **Move `migrate-to-shared-packages.sh`**
  ```bash
  mv scripts/migrate-to-shared-packages.sh scripts/migration/migrate-to-shared-packages.sh
  ```

#### 1.2 Metrics Scripts → `metrics/` and `metrics/collect/`

- [ ] **Move collection scripts to `metrics/collect/`**
  ```bash
  mv scripts/collect-code-metrics.sh scripts/metrics/collect/collect-code-metrics.sh
  mv scripts/collect-deployment-metrics.sh scripts/metrics/collect/collect-deployment-metrics.sh
  mv scripts/collect-developer-metrics.sh scripts/metrics/collect/collect-developer-metrics.sh
  mv scripts/collect-infrastructure-metrics.sh scripts/metrics/collect/collect-infrastructure-metrics.sh
  mv scripts/collect-operational-metrics.sh scripts/metrics/collect/collect-operational-metrics.sh
  mv scripts/collect-service-metrics.sh scripts/metrics/collect/collect-service-metrics.sh
  ```

- [ ] **Move tracking scripts to `metrics/`**
  ```bash
  mv scripts/track-all-metrics.sh scripts/metrics/track-all-metrics.sh
  mv scripts/track-success-metrics.sh scripts/metrics/track-success-metrics.sh
  mv scripts/update-metrics.sh scripts/metrics/update-metrics.sh
  mv scripts/generate-metrics-report.sh scripts/metrics/generate-metrics-report.sh
  ```

#### 1.3 DBIS Scripts → `dbis/`

- [ ] **Move DBIS scripts**
  ```bash
  mv scripts/automate-dbis-migration.sh scripts/dbis/automate-dbis-migration.sh
  mv scripts/migrate-all-dbis-projects.sh scripts/dbis/migrate-all-dbis-projects.sh
  mv scripts/migrate-dbis-project.sh scripts/dbis/migrate-dbis-project.sh
  mv scripts/test-dbis-migration.sh scripts/dbis/test-dbis-migration.sh
  ```

#### 1.4 Infrastructure Scripts → `infrastructure/`

- [ ] **Move infrastructure scripts**
  ```bash
  mv scripts/setup-shared-infrastructure.sh scripts/infrastructure/setup-shared-infrastructure.sh
  mv scripts/setup.sh scripts/infrastructure/setup.sh
  ```

#### 1.5 Utility Scripts → `utils/`

- [ ] **Move utility scripts**
  ```bash
  mv scripts/analyze-costs.sh scripts/utils/analyze-costs.sh
  mv scripts/optimize-builds.sh scripts/utils/optimize-builds.sh
  mv scripts/deps-analyze.sh scripts/utils/deps-analyze.sh
  mv scripts/deps-audit.sh scripts/utils/deps-audit.sh
  mv scripts/build-all.sh scripts/utils/build-all.sh
  mv scripts/test-all.sh scripts/utils/test-all.sh
  mv scripts/verify-all.sh scripts/utils/verify-all.sh
  mv scripts/cleanup.sh scripts/utils/cleanup.sh
  ```

---

### Phase 2: Update Script Paths and Add Libraries

#### 2.1 Update Migration Scripts

For each script in `migration/`, add library initialization at the top:

- [ ] **Update `migration/migrate-readme.sh`**
  ```bash
  # Add after shebang:
  SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
  source "$SCRIPT_DIR/../lib/init.sh"
  ```

- [ ] **Update `migration/migrate-terraform.sh`**
  - Add library initialization
  - Replace `echo` with `log_info`, `log_error`, etc.
  - Add input validation using `validate_project_name`

- [ ] **Update `migration/migrate-to-api-gateway.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add validation for project name and URL

- [ ] **Update `migration/migrate-to-k8s.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add `validate_project_name` for input
  - Use `require_command kubectl`

- [ ] **Update `migration/migrate-to-monitoring.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add validation for project name and environment

- [ ] **Update `migration/migrate-to-shared-packages.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add input validation

#### 2.2 Update Metrics Scripts

- [ ] **Update all scripts in `metrics/collect/`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Standardize output format

- [ ] **Update `metrics/track-all-metrics.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Use `require_command jq` if needed

- [ ] **Update `metrics/track-success-metrics.sh`**
  - Add library initialization
  - Replace `echo` with logging functions

- [ ] **Update `metrics/update-metrics.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add validation for metric category parameter

- [ ] **Update `metrics/generate-metrics-report.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Use `ensure_dir` for output directories

#### 2.3 Update DBIS Scripts

- [ ] **Update `dbis/automate-dbis-migration.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add validation for project name

- [ ] **Update `dbis/migrate-all-dbis-projects.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add error handling with `set_error_trap`

- [ ] **Update `dbis/migrate-dbis-project.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add `validate_project_name`

- [ ] **Update `dbis/test-dbis-migration.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add validation

#### 2.4 Update Infrastructure Scripts

- [ ] **Update `infrastructure/setup-shared-infrastructure.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Use `require_command` for required tools
  - Add `set_error_trap` for error handling

- [ ] **Update `infrastructure/setup.sh`**
  - Add library initialization
  - Replace `check_tool` function with `require_command`
  - Replace `echo` with logging functions
  - Use `log_success` and `log_failure` for status

#### 2.5 Update Utility Scripts

- [ ] **Update `utils/analyze-costs.sh`**
  - Add library initialization
  - Replace `echo` with logging functions

- [ ] **Update `utils/optimize-builds.sh`**
  - Add library initialization
  - Replace `echo` with logging functions

- [ ] **Update `utils/deps-analyze.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Use `require_command` for required tools

- [ ] **Update `utils/deps-audit.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Use `require_command` for required tools

- [ ] **Update `utils/build-all.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add error handling

- [ ] **Update `utils/test-all.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add error handling

- [ ] **Update `utils/verify-all.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add error handling

- [ ] **Update `utils/cleanup.sh`**
  - Add library initialization
  - Replace `echo` with logging functions
  - Add confirmation prompt using `confirm`

---

### Phase 3: Create Backward Compatibility

#### 3.1 Create Wrapper Scripts (Optional - for backward compatibility)

Create wrapper scripts in root `scripts/` directory that call the new locations:

- [ ] **Create `scripts/migrate-to-k8s.sh` wrapper**
  ```bash
  #!/bin/bash
  # Wrapper for backward compatibility
  # This script has moved to scripts/migration/migrate-to-k8s.sh
  exec "$(dirname "$0")/migration/migrate-to-k8s.sh" "$@"
  ```

- [ ] **Create wrapper for each migrated script**
  - `migrate-readme.sh` → `migration/migrate-readme.sh`
  - `migrate-terraform.sh` → `migration/migrate-terraform.sh`
  - `migrate-to-api-gateway.sh` → `migration/migrate-to-api-gateway.sh`
  - `migrate-to-monitoring.sh` → `migration/migrate-to-monitoring.sh`
  - `migrate-to-shared-packages.sh` → `migration/migrate-to-shared-packages.sh`
  - All metrics scripts
  - All DBIS scripts
  - All infrastructure scripts
  - All utility scripts

**Alternative**: Update all references instead of creating wrappers (recommended)

---

### Phase 4: Update Script References

#### 4.1 Update Documentation

- [ ] **Update main `README.md`**
  - Update script paths in examples
  - Update quick start guide
  - Update script references

- [ ] **Update `QUICK_START_GUIDE.md`**
  - Update all script paths
  - Update examples

- [ ] **Update `INTEGRATION_TASKS_LIST.md`**
  - Update script references if any

- [ ] **Update all migration guides in `docs/`**
  - Search for script references
  - Update paths

#### 4.2 Update CI/CD Workflows

- [ ] **Update `.github/workflows/ci-pilot-template.yml`**
  - Update script paths if referenced

- [ ] **Update `.github/workflows/publish-shared-packages.yml`**
  - Update script paths if referenced

- [ ] **Update `.github/workflows/infrastructure-deploy.yml`**
  - Update script paths if referenced

- [ ] **Update `.github/workflows/metrics-collection.yml`**
  - Update script paths for metrics scripts

#### 4.3 Update Other Scripts

- [ ] **Search for script references in other scripts**
  ```bash
  grep -r "scripts/migrate-to-k8s.sh" . --exclude-dir=node_modules
  grep -r "scripts/collect-" . --exclude-dir=node_modules
  grep -r "scripts/track-" . --exclude-dir=node_modules
  ```

- [ ] **Update found references**
  - Update paths to new locations
  - Test after updates

---

### Phase 5: Testing

#### 5.1 Test Library Loading

- [ ] **Test `lib/init.sh`**
  ```bash
  source scripts/lib/init.sh
  log_info "Test message"
  log_error "Test error"
  ```

- [ ] **Test each library individually**
  - Test `colors.sh`
  - Test `logging.sh`
  - Test `utils.sh`
  - Test `validation.sh`
  - Test `error-handling.sh`
  - Test `env.sh`

#### 5.2 Test Migration Scripts

- [ ] **Test `migration/migrate-to-k8s.sh`**
  ```bash
  ./scripts/migration/migrate-to-k8s.sh test-project
  ```

- [ ] **Test `migration/migrate-to-api-gateway.sh`**
  ```bash
  ./scripts/migration/migrate-to-api-gateway.sh test-service http://test-service:8080
  ```

- [ ] **Test all migration scripts with help flags**
  - Verify help text displays correctly
  - Verify error messages are clear

#### 5.3 Test Metrics Scripts

- [ ] **Test `metrics/update-metrics.sh`**
  ```bash
  ./scripts/metrics/update-metrics.sh all
  ```

- [ ] **Test `metrics/generate-metrics-report.sh`**
  ```bash
  ./scripts/metrics/generate-metrics-report.sh
  ```

- [ ] **Test each collection script**
  - Verify they create output files
  - Verify output format is correct

#### 5.4 Test DBIS Scripts

- [ ] **Test `dbis/test-dbis-migration.sh`**
  ```bash
  ./scripts/dbis/test-dbis-migration.sh dbis_monorepo
  ```

- [ ] **Test all DBIS scripts with help flags**

#### 5.5 Test Infrastructure Scripts

- [ ] **Test `infrastructure/setup.sh`**
  ```bash
  ./scripts/infrastructure/setup.sh
  ```

- [ ] **Test `infrastructure/setup-shared-infrastructure.sh`**
  - Verify it checks for required tools
  - Verify error handling works

#### 5.6 Test Utility Scripts

- [ ] **Test `utils/build-all.sh`**
- [ ] **Test `utils/test-all.sh`**
- [ ] **Test `utils/verify-all.sh`**
- [ ] **Test `utils/cleanup.sh`** (with confirmation)

#### 5.7 Integration Testing

- [ ] **Test script calling other scripts**
  - Verify paths work correctly
  - Verify error propagation

- [ ] **Test from project root**
  - All scripts should work from project root
  - Paths should resolve correctly

---

### Phase 6: Documentation Updates

#### 6.1 Update Scripts README

- [ ] **Add usage examples for each category**
  - Migration examples
  - Metrics examples
  - DBIS examples
  - Infrastructure examples
  - Utility examples

- [ ] **Add library usage examples**
  - Show how to use each library
  - Show common patterns

- [ ] **Add troubleshooting section**
  - Common issues
  - Solutions

#### 6.2 Create Migration Guide

- [ ] **Create `docs/SCRIPTS_MIGRATION_GUIDE.md`**
  - Document the migration process
  - Document breaking changes
  - Document new paths
  - Document library usage

#### 6.3 Update Main Documentation

- [ ] **Update `README.md` (project root)**
  - Update script references
  - Update quick start examples

- [ ] **Update `QUICK_START_GUIDE.md`**
  - Update all script paths
  - Update examples

- [ ] **Update `PROJECT_INDEX.md`**
  - Update script counts
  - Update structure description

---

### Phase 7: Quality Assurance

#### 7.1 Code Quality

- [ ] **Lint all scripts**
  ```bash
  find scripts -name "*.sh" -exec shellcheck {} \;
  ```

- [ ] **Fix any linting errors**
  - Address shellcheck warnings
  - Fix syntax errors

- [ ] **Verify shebang lines**
  - All scripts should have `#!/bin/bash` or `#!/usr/bin/env bash`
  - Make all scripts executable: `chmod +x scripts/**/*.sh`

#### 7.2 Consistency Check

- [ ] **Verify all scripts use libraries**
  - Check for scripts still using `echo` instead of logging
  - Check for scripts not using validation
  - Check for scripts not using error handling

- [ ] **Verify consistent patterns**
  - All scripts should initialize libraries the same way
  - All scripts should have help text
  - All scripts should validate inputs

#### 7.3 Performance Check

- [ ] **Test script execution time**
  - Compare before/after library loading
  - Verify no significant performance impact

---

### Phase 8: Final Verification

#### 8.1 Complete Verification Checklist

- [ ] **All scripts moved to correct directories**
- [ ] **All scripts updated to use libraries**
- [ ] **All scripts tested and working**
- [ ] **All documentation updated**
- [ ] **All references updated**
- [ ] **No broken links or paths**
- [ ] **All scripts executable**
- [ ] **No linting errors**
- [ ] **Backward compatibility verified** (if using wrappers)

#### 8.2 Create Verification Script

- [ ] **Create `scripts/verify-structure.sh`**
  ```bash
  #!/bin/bash
  # Verify scripts directory structure

  # Check all directories exist
  # Check all scripts are in correct locations
  # Check all scripts are executable
  # Check all scripts can load libraries
  ```

#### 8.3 Run Final Tests

- [ ] **Run all scripts with `--help` or no args**
  - Verify help text displays
  - Verify no errors

- [ ] **Run integration tests**
  - Test complete workflows
  - Verify end-to-end functionality

---

## 📊 Progress Tracking

### Current Status
- ✅ Directory structure: **Complete**
- ✅ Shared libraries: **Complete**
- ✅ Documentation (structure): **Complete**
- ⏳ Script migration: **0/30** (0%)
- ⏳ Script updates: **0/30** (0%)
- ⏳ Testing: **0%**
- ⏳ Documentation updates: **0%**

### Estimated Time
- **Phase 1 (Migration)**: 1-2 hours
- **Phase 2 (Updates)**: 4-6 hours
- **Phase 3 (Compatibility)**: 1 hour (optional)
- **Phase 4 (References)**: 2-3 hours
- **Phase 5 (Testing)**: 3-4 hours
- **Phase 6 (Documentation)**: 2-3 hours
- **Phase 7 (QA)**: 1-2 hours
- **Phase 8 (Verification)**: 1 hour

**Total Estimated Time**: 15-22 hours

---

## 🎯 Priority Order

### High Priority (Do First)
1. Phase 1: Script Migration
2. Phase 2: Update Scripts (at least library initialization)
3. Phase 5: Basic Testing

### Medium Priority
4. Phase 4: Update References
5. Phase 6: Documentation Updates

### Low Priority (Can be done later)
6. Phase 3: Backward Compatibility (if needed)
7. Phase 7: Quality Assurance
8. Phase 8: Final Verification

---

## 🚀 Quick Start Commands

### Move All Scripts at Once
```bash
cd /home/intlc/projects/scripts

# Migration scripts
mv migrate-readme.sh migrate-terraform.sh migrate-to-*.sh migration/

# Metrics scripts
mv collect-*.sh metrics/collect/
mv track-*.sh update-metrics.sh generate-metrics-report.sh metrics/

# DBIS scripts
mv *dbis*.sh dbis/

# Infrastructure scripts
mv setup*.sh infrastructure/

# Utility scripts
mv analyze-costs.sh optimize-builds.sh deps-*.sh build-all.sh test-all.sh verify-all.sh cleanup.sh utils/
```

### Update All Scripts to Use Libraries
```bash
# Add to top of each script (after shebang):
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../lib/init.sh"
```

### Make All Scripts Executable
```bash
find scripts -name "*.sh" -exec chmod +x {} \;
```

---

**Status**: Ready for Implementation
**Next Action**: Begin Phase 1 - Script Migration