Initial commit: add .gitignore and README
This commit is contained in:
defiQUG
587
docs/STRATEGY_TESTING.md
Normal file
587
docs/STRATEGY_TESTING.md
Normal file
@@ -0,0 +1,587 @@
|
||||
# 🧪 DeFi Strategy Testing Framework
|
||||
|
||||
> A comprehensive CLI tool for testing DeFi strategies against local mainnet forks with support for success paths and controlled failure scenarios.
|
||||
|
||||
---
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
The DeFi Strategy Testing Framework allows you to:
|
||||
|
||||
- ✅ Run **repeatable, deterministic simulations** of DeFi strategies on local mainnet forks
|
||||
- 💥 Test both **success** and **failure** cases: liquidations, oracle shocks, cap limits, slippage, approvals, paused assets, etc.
|
||||
- ✅ Provide **clear pass/fail assertions** (e.g., Aave Health Factor >= 1 after each step; exact token deltas; gas ceilings)
|
||||
- 📊 Produce **auditable reports** (JSON + HTML) suitable for CI
|
||||
- 🎲 **Fuzz test** strategies with parameterized inputs
|
||||
- 🐋 **Automatically fund** test accounts via whale impersonation
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
/defi-strat-cli
|
||||
/src/strat
|
||||
/core # 🔧 Engine: fork control, scenario runner, assertions, reporting
|
||||
- fork-orchestrator.ts # 🍴 Fork management (Anvil/Hardhat)
|
||||
- scenario-runner.ts # ▶️ Executes scenarios step by step
|
||||
- assertion-evaluator.ts # ✅ Evaluates assertions
|
||||
- failure-injector.ts # 💥 Injects failure scenarios
|
||||
- fuzzer.ts # 🎲 Fuzz testing with parameterized inputs
|
||||
- whale-registry.ts # 🐋 Whale addresses for token funding
|
||||
/adapters # 🔌 Protocol adapters
|
||||
/aave-v3-adapter.ts # 🏦 Aave v3 operations
|
||||
/uniswap-v3-adapter.ts # 🔄 Uniswap v3 swaps
|
||||
/compound-v3-adapter.ts # 🏛️ Compound v3 operations
|
||||
/erc20-adapter.ts # 💰 ERC20 token operations
|
||||
/dsl # 📝 Strategy/Scenario schema + loader
|
||||
- scenario-loader.ts # 📄 YAML/JSON parser
|
||||
/reporters # 📊 Report generators
|
||||
- json-reporter.ts # 📄 JSON reports
|
||||
- html-reporter.ts # 🌐 HTML reports
|
||||
- junit-reporter.ts # 🔧 JUnit XML for CI
|
||||
/config # ⚙️ Configuration
|
||||
- networks.ts # 🌐 Network configurations
|
||||
- oracle-feeds.ts # 🔮 Oracle feed addresses
|
||||
/scenarios # 📚 Example strategies
|
||||
/aave
|
||||
- leveraged-long.yml
|
||||
- liquidation-drill.yml
|
||||
/compound3
|
||||
- supply-borrow.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 📦 Installation
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pnpm install
|
||||
```
|
||||
|
||||
### ▶️ Run a Scenario
|
||||
|
||||
```bash
|
||||
# Run a scenario
|
||||
pnpm run strat run scenarios/aave/leveraged-long.yml
|
||||
|
||||
# Run with custom network
|
||||
pnpm run strat run scenarios/aave/leveraged-long.yml --network base
|
||||
|
||||
# Generate reports
|
||||
pnpm run strat run scenarios/aave/leveraged-long.yml \
|
||||
--report out/run.json \
|
||||
--html out/report.html \
|
||||
--junit out/junit.xml
|
||||
```
|
||||
|
||||
### 🧪 Test Script
|
||||
|
||||
For comprehensive testing with a real fork:
|
||||
|
||||
```bash
|
||||
# Set your RPC URL
|
||||
export MAINNET_RPC_URL=https://mainnet.infura.io/v3/YOUR_KEY
|
||||
|
||||
# Run test script
|
||||
pnpm run strat:test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ CLI Commands
|
||||
|
||||
### 🍴 `fork up`
|
||||
|
||||
Start or attach to a fork instance.
|
||||
|
||||
```bash
|
||||
pnpm run strat fork up --network mainnet --block 18500000
|
||||
```
|
||||
|
||||
### ▶️ `run`
|
||||
|
||||
Run a scenario file.
|
||||
|
||||
```bash
|
||||
pnpm run strat run <scenario-file> [options]
|
||||
```
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--network <network>` | Network name or chain ID | `mainnet` |
|
||||
| `--report <file>` | Output JSON report path | - |
|
||||
| `--html <file>` | Output HTML report path | - |
|
||||
| `--junit <file>` | Output JUnit XML report path | - |
|
||||
| `--rpc <url>` | Custom RPC URL | - |
|
||||
|
||||
### 🎲 `fuzz`
|
||||
|
||||
Fuzz test a scenario with parameterized inputs.
|
||||
|
||||
```bash
|
||||
pnpm run strat fuzz scenarios/aave/leveraged-long.yml --iters 100 --seed 42
|
||||
```
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--iters <number>` | Number of iterations | `100` |
|
||||
| `--seed <number>` | Random seed for reproducibility | - |
|
||||
| `--report <file>` | Output JSON report path | - |
|
||||
|
||||
### 💥 `failures`
|
||||
|
||||
List available failure injection methods.
|
||||
|
||||
```bash
|
||||
pnpm run strat failures [protocol]
|
||||
```
|
||||
|
||||
### 📊 `compare`
|
||||
|
||||
Compare two run reports.
|
||||
|
||||
```bash
|
||||
pnpm run strat compare out/run1.json out/run2.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Writing Scenarios
|
||||
|
||||
Scenarios are defined in YAML or JSON format:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
network: mainnet
|
||||
protocols: [aave-v3, uniswap-v3]
|
||||
|
||||
assumptions:
|
||||
baseCurrency: USD
|
||||
slippageBps: 30
|
||||
minHealthFactor: 1.05
|
||||
|
||||
accounts:
|
||||
trader:
|
||||
funded:
|
||||
- token: WETH
|
||||
amount: "5"
|
||||
|
||||
steps:
|
||||
- name: Approve WETH to Aave Pool
|
||||
action: erc20.approve
|
||||
args:
|
||||
token: WETH
|
||||
spender: aave-v3:Pool
|
||||
amount: "max"
|
||||
|
||||
- name: Supply WETH
|
||||
action: aave-v3.supply
|
||||
args:
|
||||
asset: WETH
|
||||
amount: "5"
|
||||
onBehalfOf: $accounts.trader
|
||||
assert:
|
||||
- aave-v3.healthFactor >= 1.5
|
||||
|
||||
- name: Borrow USDC
|
||||
action: aave-v3.borrow
|
||||
args:
|
||||
asset: USDC
|
||||
amount: "6000"
|
||||
rateMode: variable
|
||||
|
||||
- name: Swap USDC->WETH
|
||||
action: uniswap-v3.exactInputSingle
|
||||
args:
|
||||
tokenIn: USDC
|
||||
tokenOut: WETH
|
||||
fee: 500
|
||||
amountIn: "3000"
|
||||
|
||||
- name: Oracle shock (-12% WETH)
|
||||
action: failure.oracleShock
|
||||
args:
|
||||
feed: CHAINLINK_WETH_USD
|
||||
pctDelta: -12
|
||||
|
||||
- name: Check HF still safe
|
||||
action: assert
|
||||
args:
|
||||
expression: "aave-v3.healthFactor >= 1.05"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Supported Actions
|
||||
|
||||
### 🏦 Aave v3
|
||||
|
||||
| Action | Description | Status |
|
||||
|--------|-------------|--------|
|
||||
| `aave-v3.supply` | Supply assets to Aave | ✅ |
|
||||
| `aave-v3.withdraw` | Withdraw assets from Aave | ✅ |
|
||||
| `aave-v3.borrow` | Borrow assets from Aave | ✅ |
|
||||
| `aave-v3.repay` | Repay borrowed assets | ✅ |
|
||||
| `aave-v3.flashLoanSimple` | Execute a flash loan | ✅ |
|
||||
|
||||
**Views:**
|
||||
- `aave-v3.healthFactor`: Get user health factor
|
||||
- `aave-v3.userAccountData`: Get full user account data
|
||||
|
||||
### 🏛️ Compound v3
|
||||
|
||||
| Action | Description | Status |
|
||||
|--------|-------------|--------|
|
||||
| `compound-v3.supply` | Supply collateral to Compound v3 | ✅ |
|
||||
| `compound-v3.withdraw` | Withdraw collateral or base asset | ✅ |
|
||||
| `compound-v3.borrow` | Borrow base asset (withdraws base asset) | ✅ |
|
||||
| `compound-v3.repay` | Repay debt (supplies base asset) | ✅ |
|
||||
|
||||
**Views:**
|
||||
- `compound-v3.borrowBalance`: Get borrow balance
|
||||
- `compound-v3.collateralBalance`: Get collateral balance for an asset
|
||||
|
||||
### 🔄 Uniswap v3
|
||||
|
||||
| Action | Description | Status |
|
||||
|--------|-------------|--------|
|
||||
| `uniswap-v3.exactInputSingle` | Execute an exact input swap | ✅ |
|
||||
| `uniswap-v3.exactOutputSingle` | Execute an exact output swap | ✅ |
|
||||
|
||||
### 💰 ERC20
|
||||
|
||||
| Action | Description | Status |
|
||||
|--------|-------------|--------|
|
||||
| `erc20.approve` | Approve token spending | ✅ |
|
||||
|
||||
**Views:**
|
||||
- `erc20.balanceOf`: Get token balance
|
||||
|
||||
### 💥 Failure Injection
|
||||
|
||||
| Action | Description | Status |
|
||||
|--------|-------------|--------|
|
||||
| `failure.oracleShock` | Inject an oracle price shock (attempts storage manipulation) | ✅ |
|
||||
| `failure.timeTravel` | Advance time | ✅ |
|
||||
| `failure.setTimestamp` | Set block timestamp | ✅ |
|
||||
| `failure.liquidityShock` | Move liquidity | ✅ |
|
||||
| `failure.setBaseFee` | Set gas price | ✅ |
|
||||
| `failure.pauseReserve` | Pause a reserve (Aave) | ✅ |
|
||||
| `failure.capExhaustion` | Simulate cap exhaustion | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## ✅ Assertions
|
||||
|
||||
Assertions can be added to any step:
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- name: Check health factor
|
||||
action: assert
|
||||
args:
|
||||
expression: "aave-v3.healthFactor >= 1.05"
|
||||
```
|
||||
|
||||
### Supported Operators
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `>=` | Greater than or equal | `aave-v3.healthFactor >= 1.05` |
|
||||
| `<=` | Less than or equal | `amount <= 1000` |
|
||||
| `>` | Greater than | `balance > 0` |
|
||||
| `<` | Less than | `gasUsed < 1000000` |
|
||||
| `==` | Equal to | `status == "success"` |
|
||||
| `!=` | Not equal to | `error != null` |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Reports
|
||||
|
||||
### 📄 JSON Report
|
||||
|
||||
Machine-readable JSON format with full run details.
|
||||
|
||||
**Features:**
|
||||
- ✅ Complete step-by-step execution log
|
||||
- ✅ Assertion results
|
||||
- ✅ Gas usage metrics
|
||||
- ✅ Error messages and stack traces
|
||||
- ✅ State deltas
|
||||
|
||||
### 🌐 HTML Report
|
||||
|
||||
Human-readable HTML report with:
|
||||
|
||||
- ✅ Run summary (pass/fail status, duration, gas)
|
||||
- ✅ Step-by-step execution details
|
||||
- ✅ Assertion results with visual indicators
|
||||
- ✅ Gas usage charts
|
||||
- ✅ Error messages with syntax highlighting
|
||||
|
||||
### 🔧 JUnit XML
|
||||
|
||||
CI-friendly XML format for integration with test runners.
|
||||
|
||||
**Features:**
|
||||
- ✅ Compatible with Jenkins, GitLab CI, GitHub Actions
|
||||
- ✅ Test suite and case structure
|
||||
- ✅ Pass/fail status
|
||||
- ✅ Error messages and stack traces
|
||||
|
||||
---
|
||||
|
||||
## 🍴 Fork Orchestration
|
||||
|
||||
The framework supports:
|
||||
|
||||
| Backend | Status | Features |
|
||||
|---------|--------|----------|
|
||||
| **Anvil** (Foundry) | ✅ | Fast, rich custom RPC methods |
|
||||
| **Hardhat** | ✅ | Wider familiarity |
|
||||
| **Tenderly** | 🚧 Coming soon | Optional remote simulation backend |
|
||||
|
||||
### 🎯 Fork Features
|
||||
|
||||
- ✅ **Snapshot/revert** - Fast test loops
|
||||
- 🐋 **Account impersonation** - Fund/borrow from whales
|
||||
- ⏰ **Time travel** - Advance time, set timestamp
|
||||
- 💾 **Storage manipulation** - Oracle overrides
|
||||
- ⛽ **Gas price control** - Test gas scenarios
|
||||
|
||||
---
|
||||
|
||||
## 🐋 Token Funding
|
||||
|
||||
The framework automatically funds test accounts via whale impersonation. Known whale addresses are maintained in the whale registry for common tokens.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. 📋 Look up whale address from registry
|
||||
2. 🎭 Impersonate whale on the fork
|
||||
3. 💸 Transfer tokens to test account
|
||||
4. ✅ Verify balance
|
||||
|
||||
### Adding New Whales
|
||||
|
||||
```typescript
|
||||
// src/strat/core/whale-registry.ts
|
||||
export const WHALE_REGISTRY: Record<number, Record<string, Address>> = {
|
||||
1: {
|
||||
YOUR_TOKEN: '0x...' as Address,
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Protocol Adapters
|
||||
|
||||
### Adding a New Adapter
|
||||
|
||||
Implement the `ProtocolAdapter` interface:
|
||||
|
||||
```typescript
|
||||
export interface ProtocolAdapter {
|
||||
name: string;
|
||||
discover(network: Network): Promise<RuntimeAddresses>;
|
||||
actions: Record<string, (ctx: StepContext, args: any) => Promise<StepResult>>;
|
||||
invariants?: Array<(ctx: StepContext) => Promise<void>>;
|
||||
views?: Record<string, (ctx: ViewContext, args?: any) => Promise<any>>;
|
||||
}
|
||||
```
|
||||
|
||||
### Example Implementation
|
||||
|
||||
```typescript
|
||||
export class MyProtocolAdapter implements ProtocolAdapter {
|
||||
name = 'my-protocol';
|
||||
|
||||
async discover(network: Network): Promise<RuntimeAddresses> {
|
||||
return {
|
||||
contract: '0x...',
|
||||
};
|
||||
}
|
||||
|
||||
actions = {
|
||||
myAction: async (ctx: StepContext, args: any): Promise<StepResult> => {
|
||||
// Implement action
|
||||
return { success: true };
|
||||
},
|
||||
};
|
||||
|
||||
views = {
|
||||
myView: async (ctx: ViewContext): Promise<any> => {
|
||||
// Implement view
|
||||
return value;
|
||||
},
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💥 Failure Injection
|
||||
|
||||
### 🔮 Oracle Shocks
|
||||
|
||||
Inject price changes to test liquidation scenarios. The framework attempts to modify Chainlink aggregator storage:
|
||||
|
||||
```yaml
|
||||
- name: Oracle shock
|
||||
action: failure.oracleShock
|
||||
args:
|
||||
feed: CHAINLINK_WETH_USD
|
||||
pctDelta: -12 # -12% price drop
|
||||
# aggregatorAddress: 0x... # Optional, auto-resolved if not provided
|
||||
```
|
||||
|
||||
> ⚠️ **Note:** Oracle storage manipulation requires precise slot calculation and may not work on all forks. The framework will attempt the manipulation and log warnings if it fails.
|
||||
|
||||
### ⏰ Time Travel
|
||||
|
||||
Advance time for interest accrual, maturity, etc.:
|
||||
|
||||
```yaml
|
||||
- name: Advance time
|
||||
action: failure.timeTravel
|
||||
args:
|
||||
seconds: 86400 # 1 day
|
||||
```
|
||||
|
||||
### 💧 Liquidity Shocks
|
||||
|
||||
Move liquidity to test pool utilization:
|
||||
|
||||
```yaml
|
||||
- name: Liquidity shock
|
||||
action: failure.liquidityShock
|
||||
args:
|
||||
token: WETH
|
||||
whale: 0x...
|
||||
amount: "1000"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎲 Fuzzing
|
||||
|
||||
Fuzz testing runs scenarios with parameterized inputs:
|
||||
|
||||
```bash
|
||||
pnpm run strat fuzz scenarios/aave/leveraged-long.yml --iters 100 --seed 42
|
||||
```
|
||||
|
||||
### What Gets Fuzzed
|
||||
|
||||
| Parameter | Variation | Description |
|
||||
|-----------|-----------|-------------|
|
||||
| Amounts | ±20% | Randomly vary token amounts |
|
||||
| Oracle shocks | Within range | Vary oracle shock percentages |
|
||||
| Fee tiers | Random selection | Test different fee tiers |
|
||||
| Slippage | Variable | Vary slippage parameters |
|
||||
|
||||
### Features
|
||||
|
||||
- ✅ Each iteration runs on a fresh snapshot
|
||||
- ✅ Failures don't affect subsequent runs
|
||||
- ✅ Reproducible with seed parameter
|
||||
- ✅ Detailed report for all iterations
|
||||
|
||||
---
|
||||
|
||||
## 🌐 Network Support
|
||||
|
||||
| Network | Chain ID | Status |
|
||||
|---------|----------|--------|
|
||||
| Ethereum Mainnet | 1 | ✅ |
|
||||
| Base | 8453 | ✅ |
|
||||
| Arbitrum One | 42161 | ✅ |
|
||||
| Optimism | 10 | ✅ |
|
||||
| Polygon | 137 | ✅ |
|
||||
|
||||
> 💡 Or use chain IDs directly: `--network 1` for mainnet.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security & Safety
|
||||
|
||||
> ⚠️ **IMPORTANT**: This tool is for **local forks and simulations only**. Do **not** use real keys or send transactions on mainnet from this tool.
|
||||
|
||||
Testing "oracle shocks", liquidations, and admin toggles are **defensive simulations** to validate strategy resilience, **not** instructions for real-world exploitation.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Examples
|
||||
|
||||
See the `scenarios/` directory for example scenarios:
|
||||
|
||||
| Scenario | Description | Path |
|
||||
|----------|-------------|------|
|
||||
| **Leveraged Long** | Leveraged long strategy with Aave and Uniswap | `aave/leveraged-long.yml` |
|
||||
| **Liquidation Drill** | Test liquidation scenarios with oracle shocks | `aave/liquidation-drill.yml` |
|
||||
| **Supply & Borrow** | Compound v3 supply and borrow example | `compound3/supply-borrow.yml` |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### ❌ Token Funding Fails
|
||||
|
||||
If token funding fails, check:
|
||||
|
||||
1. ✅ Whale address has sufficient balance on the fork
|
||||
2. ✅ Fork supports account impersonation (Anvil)
|
||||
3. ✅ RPC endpoint allows custom methods
|
||||
|
||||
### ❌ Oracle Shocks Don't Work
|
||||
|
||||
Oracle storage manipulation is complex and may fail if:
|
||||
|
||||
1. ❌ Storage slot calculation is incorrect
|
||||
2. ❌ Fork doesn't support storage manipulation
|
||||
3. ❌ Aggregator uses a different storage layout
|
||||
|
||||
> 💡 The framework will log warnings and continue - verify price changes manually if needed.
|
||||
|
||||
### ❌ Fork Connection Issues
|
||||
|
||||
If the fork fails to start:
|
||||
|
||||
1. ✅ Check RPC URL is correct and accessible
|
||||
2. ✅ Verify network configuration
|
||||
3. ✅ Check if fork block number is valid
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Future Enhancements
|
||||
|
||||
- [ ] 🎯 Tenderly backend integration
|
||||
- [ ] ⛽ Gas profiling & diffing
|
||||
- [ ] 📊 Risk margin calculators
|
||||
- [ ] 📈 HTML charts for HF over time
|
||||
- [ ] 🔌 More protocol adapters (Maker, Curve, Balancer, etc.)
|
||||
- [ ] ⚡ Parallel execution of scenarios
|
||||
- [ ] 📝 Scenario templates and generators
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions welcome! Please:
|
||||
|
||||
1. 🍴 Fork the repository
|
||||
2. 🌿 Create a feature branch
|
||||
3. ✏️ Make your changes
|
||||
4. 🧪 Add tests
|
||||
5. 📤 Submit a pull request
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user