Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 11:32:49 -08:00
commit b4753cef7e
81 changed files with 9255 additions and 0 deletions
--- a/NEXT_STEPS.md
+++ b/NEXT_STEPS.md
@@ -0,0 +1,162 @@
+# Next Steps for Virtual Banker
+
+## ✅ Completed Integration Steps
+
+1. **Service Integration**: All services are now connected:
+   - Orchestrator → LLM Gateway
+   - Orchestrator → RAG Service
+   - Orchestrator → Tool Executor
+   - Banking tools → Banking API client
+
+2. **WebSocket Support**: Realtime gateway integrated into API routes
+
+3. **Startup Scripts**: Created scripts for database setup and backend startup
+
+4. **Banking Integration**: Tools now connect to backend banking services with fallback
+
+## 🔄 Remaining Integration Tasks
+
+### 1. Replace Mock Services with Real APIs
+
+**ASR Service** (Speech-to-Text):
+- [ ] Integrate Deepgram API
+  - Get API key from Deepgram
+  - Update `backend/asr/service.go` to use Deepgram streaming API
+  - Test with real audio streams
+
+- [ ] Or integrate Google Speech-to-Text
+  - Set up Google Cloud credentials
+  - Implement streaming transcription
+
+**TTS Service** (Text-to-Speech):
+- [ ] Integrate ElevenLabs API
+  - Get API key from ElevenLabs
+  - Update `backend/tts/service.go` to use ElevenLabs API
+  - Configure voice selection per tenant
+
+- [ ] Or integrate Azure TTS
+  - Set up Azure credentials
+  - Implement SSML support
+
+**LLM Gateway**:
+- [ ] Integrate OpenAI API
+  - Get API key
+  - Update `backend/llm/gateway.go` to use OpenAI
+  - Implement function calling
+  - Add streaming support
+
+- [ ] Or integrate Anthropic Claude
+  - Get API key
+  - Implement tool use
+
+### 2. Complete WebRTC Implementation
+
+- [ ] Implement full WebRTC signaling
+  - SDP offer/answer exchange
+  - ICE candidate handling
+  - TURN server configuration
+
+- [ ] Add media streaming
+  - Audio stream from client → ASR
+  - Audio stream from TTS → client
+  - Video stream from avatar → client
+
+### 3. Connect to Existing Banking Services
+
+Update banking tool integrations to match actual API endpoints:
+
+```go
+// Check actual endpoints in backend/banking/
+// Update integration.go with correct paths
+```
+
+### 4. Unreal Engine Avatar Setup
+
+- [ ] Install Unreal Engine 5.3+
+- [ ] Create new project
+- [ ] Enable PixelStreaming plugin
+- [ ] Import digital human character
+- [ ] Set up blendshapes for visemes
+- [ ] Configure animation blueprints
+- [ ] Package for Linux deployment
+
+### 5. Testing
+
+- [ ] Unit tests for all services
+- [ ] Integration tests for API endpoints
+- [ ] E2E tests for widget
+- [ ] Load testing for concurrent sessions
+
+### 6. Production Deployment
+
+- [ ] Set up secrets management
+- [ ] Configure monitoring (Prometheus/Grafana)
+- [ ] Set up logging aggregation
+- [ ] Configure auto-scaling
+- [ ] Security audit
+- [ ] Performance optimization
+
+## Quick Start Commands
+
+```bash
+# Setup database
+cd virtual-banker
+./scripts/setup-database.sh
+
+# Start backend
+./scripts/start-backend.sh
+
+# Build widget
+cd widget
+npm install
+npm run build
+```
+
+## Environment Variables
+
+Create `.env` file:
+
+```bash
+DATABASE_URL=postgres://explorer:changeme@localhost:5432/explorer?sslmode=disable
+REDIS_URL=redis://localhost:6379
+PORT=8081
+
+# For real services (when ready):
+DEEPGRAM_API_KEY=your_key_here
+ELEVENLABS_API_KEY=your_key_here
+OPENAI_API_KEY=your_key_here
+BANKING_API_URL=http://localhost:8080
+```
+
+## Testing the Integration
+
+1. **Start services**:
+   ```bash
+   docker-compose up -d postgres redis
+   ./scripts/start-backend.sh
+   ```
+
+2. **Create a session**:
+   ```bash
+   curl -X POST http://localhost:8081/v1/sessions \
+     -H "Content-Type: application/json" \
+     -d '{
+       "tenant_id": "default",
+       "user_id": "test-user",
+       "auth_assertion": "test-token"
+     }'
+   ```
+
+3. **Test WebSocket**:
+   ```bash
+   # Use wscat or similar tool
+   wscat -c ws://localhost:8081/v1/realtime/{session_id}
+   ```
+
+## Notes
+
+- All mock services are functional and can be tested independently
+- Banking tools have fallback to mock data if service unavailable
+- WebRTC gateway is ready but needs full signaling implementation
+- Widget is fully functional for text chat (voice requires WebRTC completion)
+