76aa419320
- CCIP/trustless bridge contracts, GRU tokens, DEX/PMM tests, reserve vault. - Token-aggregation service routes, planner, chain config, relay env templates. - Config snapshots and multi-chain deployment markdown updates. - gitignore services/btc-intake/dist/ (tsc output); do not track dist. Run forge build && forge test before deploy (large solc graph). Made-with: Cursor
252 lines
11 KiB
Markdown
252 lines
11 KiB
Markdown
# Token Aggregation Service
|
||
|
||
A comprehensive token aggregation service that indexes token info, volume, liquidity, and market signals from on-chain data and enriches with CoinGecko, CoinMarketCap, and DexScreener APIs for ChainID 138 (DeFi Oracle Meta Mainnet) and ChainID 651940 (ALL Mainnet).
|
||
|
||
**REST API reference:** [docs/REST_API_REFERENCE.md](docs/REST_API_REFERENCE.md) — tokens, pools, prices, volume, OHLCV for dApps and MetaMask Snap discovery.
|
||
|
||
**Chain 138 Snap:** The MetaMask Chain 138 Snap (companion site at e.g. https://explorer.d-bis.org/snap/) calls this service for market data, swap quotes, and bridge routes. If the Snap is built with `GATSBY_SNAP_API_BASE_URL=https://explorer.d-bis.org`, then explorer.d-bis.org must serve this API (e.g. proxy `/api/v1/*` to this service). Otherwise build the Snap site with `GATSBY_SNAP_API_BASE_URL` set to this service’s public URL. See [metamask-integration/chain138-snap/docs/CHAIN138_SNAP_TROUBLESHOOTING.md](../../../metamask-integration/chain138-snap/docs/CHAIN138_SNAP_TROUBLESHOOTING.md). **CORS:** The service uses `cors()` (all origins allowed by default) so MetaMask Snap and browser clients can fetch token list and networks.
|
||
|
||
## Features
|
||
|
||
- **Chain-Native Indexing**: Indexes tokens, DEX pools, and swap events directly from blockchain
|
||
- **External API Enrichment**: Enriches data with CoinGecko, CoinMarketCap, and DexScreener
|
||
- **Multi-DEX Support**: Supports UniswapV2, UniswapV3, and DODO PMM protocols
|
||
- **Chain 138 PMM quotes**: `GET /api/v1/quote` runs on-chain `querySellBase` / `querySellQuote` when RPC is configured (`RPC_URL_138` or `TOKEN_AGGREGATION_*` in `.env.example`); JSON includes `quoteEngine` (`pmm-onchain` vs `constant-product`)
|
||
- **OHLCV Data**: Generates Open, High, Low, Close, Volume data for price charts
|
||
- **Volume Analytics**: Calculates 5m, 1h, 24h, 7d, 30d volume metrics
|
||
- **REST API**: Unified REST API for all token data
|
||
|
||
## Architecture
|
||
|
||
### Three-Layer Design
|
||
|
||
1. **Layer 1: Chain-Native Indexer** - Source of truth from on-chain data
|
||
2. **Layer 2: External Enrichment Adapters** - Best-effort enrichment from external APIs
|
||
3. **Layer 3: Unified REST API** - Single API endpoint for consumption
|
||
|
||
## Prerequisites
|
||
|
||
- Node.js 20+
|
||
- PostgreSQL 14+ with TimescaleDB extension
|
||
- Access to RPC endpoints for ChainID 138 and 651940
|
||
- (Optional) API keys for CoinGecko, CoinMarketCap, DexScreener
|
||
|
||
## Installation
|
||
|
||
1. Clone the repository and navigate to the service directory:
|
||
```bash
|
||
cd smom-dbis-138/services/token-aggregation
|
||
```
|
||
|
||
2. Install dependencies:
|
||
```bash
|
||
npm install
|
||
```
|
||
|
||
3. Copy environment file:
|
||
```bash
|
||
cp .env.example .env
|
||
```
|
||
|
||
4. Configure environment variables in `.env` (see `.env.example` for full list):
|
||
```bash
|
||
# Chain RPCs
|
||
# Explorer-side/LAN deployment: use the public RPC node directly.
|
||
CHAIN_138_RPC_URL=http://192.168.11.221:8545
|
||
# External-only deployments may use: https://rpc-http-pub.d-bis.org
|
||
CHAIN_651940_RPC_URL=https://mainnet-rpc.alltra.global
|
||
|
||
# Database
|
||
DATABASE_URL=postgresql://user:password@localhost:5432/explorer_db
|
||
|
||
# External APIs (optional) — use placeholders in .env.example; get keys from provider dashboards
|
||
COINGECKO_API_KEY=your_key_here
|
||
COINMARKETCAP_API_KEY=your_key_here
|
||
DEXSCREENER_API_KEY=your_key_here
|
||
```
|
||
|
||
For Chain 138 there is an important split:
|
||
- Explorer, Blockscout, token-aggregation, and other read-mostly services should use the public RPC node on `192.168.11.221:8545` when they run on the LAN.
|
||
- External wallets and dApps should use `https://rpc-http-pub.d-bis.org`.
|
||
- Operator/deploy workflows use the core RPC `http://192.168.11.211:8545`, but that core endpoint is not for explorer/read traffic.
|
||
|
||
**Canonical token addresses (report API):** Set per-chain env vars for tokens you want in `/api/v1/report/*`. Required/minimal for report: `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138` (Chain 138); optionally `CUSDC_ADDRESS_651940`, `CUSDT_ADDRESS_651940` for Chain 651940. **Chain 138 compliant fiat** (cEURC, cEURT, cGBPC, cGBPT, cAUDC, cJPYC, cCHFC, cCADC, cXAUC, cXAUT) have **fallback addresses** in `src/config/canonical-tokens.ts` (DeployCompliantFiatTokens 2026-02-27); they are included in the report without env. The live **ALL Mainnet AUSDT corridor** is also surfaced there as `cAUSDT` on Chain 138 plus `cWAUSDT` mirrors on BSC, Polygon, Avalanche, and Celo. The planned **ALL Mainnet gold corridor** is env-gated under `CAXAUC_ADDRESS_651940`, `CAXAUT_ADDRESS_651940`, `CWAXAUC_ADDRESS_651940`, and `CWAXAUT_ADDRESS_651940`. Other symbols (USDW, acUSDC, vdcUSDC, sdcUSDC, etc.) — see `.env.example`. Unset tokens (with no fallback) are omitted from the report.
|
||
|
||
### Required environment variables (canonical tokens — Blitzkrieg Step 1/9)
|
||
|
||
The canonical token list is defined in `src/config/canonical-tokens.ts` and is the single source of truth for the Token Aggregation API and report endpoints. Addresses are read from environment variables; unset tokens are omitted from `getCanonicalTokensByChain` and report output.
|
||
|
||
| Purpose | Env var pattern | Example |
|
||
|--------|------------------|--------|
|
||
| Chain 138 | `{SYMBOL}_ADDRESS_138` (symbol with `-` → `_`, uppercase) | `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138`, `CAUSDT_ADDRESS_138`, `USDW_ADDRESS_138`, `ACUSDC_ADDRESS_138`, `VDCUSDC_ADDRESS_138`, `SDCUSDC_ADDRESS_138` |
|
||
| Chain 651940 (ALL Mainnet) | `{SYMBOL}_ADDRESS_651940` | `CUSDC_ADDRESS_651940`, `CUSDT_ADDRESS_651940`, `AUSDT_ADDRESS_651940`, `CAXAUC_ADDRESS_651940`, `CWAXAUC_ADDRESS_651940` |
|
||
|
||
**Minimum for report API:** `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138`. For full GRU M1 + W-tokens + ac*/vdc*/sdc* coverage, set the corresponding `*_ADDRESS_138` and `*_ADDRESS_651940` vars. See `.env.example` for the full commented list. Refs: [BLITZKRIEG_SUPER_PRO_MAX_MASTER_PLAN](../../../docs/00-meta/BLITZKRIEG_SUPER_PRO_MAX_MASTER_PLAN.md) §2–§3, [PLACEHOLDERS_AND_COMPLETION_MASTER_LIST](../../../docs/00-meta/PLACEHOLDERS_AND_COMPLETION_MASTER_LIST.md).
|
||
|
||
**CoinGecko/CMC chain support:** ChainID 138 and 651940 are not yet supported by CoinGecko/CMC; external price/volume for these chains will be empty in the report until the platforms add support or you use another source. The report API still returns chain-native token/pool data.
|
||
|
||
**Bridge routes (CCIP + Trustless):** `GET /api/v1/bridge/routes` returns CCIP (WETH9/WETH10) and **Trustless** bridge data. Trustless is **enabled in production by default**: the deployed Lockbox138 address on Chain 138 is included when `LOCKBOX_138` is not set. Optional env (restart after change):
|
||
|
||
- `LOCKBOX_138` — Override Lockbox contract address on Chain 138 (default: deployed Lockbox138).
|
||
- `INBOX_ETH` — (Optional) InboxETH contract on Ethereum Mainnet; when set, the Snap shows "Trustless → Ethereum Mainnet" with this address.
|
||
|
||
The MetaMask Snap bridge dialog always shows the Trustless (Lockbox) route when using this API. **Full Trustless operations** (user lock/claim) also require the Trustless stack on Ethereum (InboxETH, BondManager, LiquidityPool, etc.) to be deployed and operational; see `smom-dbis-138/docs/bridge/trustless`.
|
||
|
||
**Token mapping (multichain):** When run from the monorepo (proxmox), `GET /api/v1/token-mapping?fromChain=138&toChain=651940` (and `?fromChain=&toChain=` for any pair), `GET /api/v1/token-mapping/pairs`, and `GET /api/v1/token-mapping/resolve?fromChain=&toChain=&address=` expose `config/token-mapping-multichain.json` for bridge UIs and cross-chain address resolution.
|
||
|
||
5. Run database migrations:
|
||
```bash
|
||
# Ensure the migration has been run in the explorer database
|
||
# Migration file: explorer-monorepo/backend/database/migrations/0011_token_aggregation_schema.up.sql
|
||
```
|
||
|
||
6. Build the project:
|
||
```bash
|
||
npm run build
|
||
```
|
||
|
||
7. Start the service:
|
||
```bash
|
||
npm start
|
||
```
|
||
|
||
## Development
|
||
|
||
```bash
|
||
# Run in development mode with hot reload
|
||
npm run dev
|
||
|
||
# Run tests
|
||
npm test
|
||
|
||
# Lint code
|
||
npm run lint
|
||
```
|
||
|
||
## Docker Deployment
|
||
|
||
### Build and run with Docker Compose
|
||
|
||
```bash
|
||
docker-compose up -d
|
||
```
|
||
|
||
### Build Docker image
|
||
|
||
```bash
|
||
docker build -t token-aggregation-service .
|
||
docker run -p 3000:3000 --env-file .env token-aggregation-service
|
||
```
|
||
|
||
## API Endpoints
|
||
|
||
### Health Check
|
||
```
|
||
GET /health
|
||
```
|
||
|
||
### List Chains
|
||
```
|
||
GET /api/v1/chains
|
||
```
|
||
|
||
### List Tokens
|
||
```
|
||
GET /api/v1/tokens?chainId=138&limit=50&offset=0
|
||
```
|
||
|
||
### Get Token Details
|
||
```
|
||
GET /api/v1/tokens/:address?chainId=138
|
||
```
|
||
|
||
### Get Token Pools
|
||
```
|
||
GET /api/v1/tokens/:address/pools?chainId=138
|
||
```
|
||
|
||
### Get OHLCV Data
|
||
```
|
||
GET /api/v1/tokens/:address/ohlcv?chainId=138&interval=1h&from=timestamp&to=timestamp
|
||
```
|
||
|
||
### Get Token Signals
|
||
```
|
||
GET /api/v1/tokens/:address/signals?chainId=138
|
||
```
|
||
|
||
### Search Tokens
|
||
```
|
||
GET /api/v1/search?q=USDT&chainId=138
|
||
```
|
||
|
||
### Get Pool Details
|
||
```
|
||
GET /api/v1/pools/:poolAddress?chainId=138
|
||
```
|
||
|
||
### Route Decision Tree
|
||
```
|
||
GET /api/v1/routes/tree?chainId=138&tokenIn=0x...&tokenOut=0x...&amountIn=1000000&destinationChainId=1
|
||
```
|
||
Live route tree with current pool depth, freshness, bridge fallback, and destination-swap legs.
|
||
|
||
### Route Depth Summary
|
||
```
|
||
GET /api/v1/routes/depth?chainId=138&tokenIn=0x...&tokenOut=0x...&amountIn=1000000&destinationChainId=1
|
||
```
|
||
Compact summary of the same route tree for routing UIs.
|
||
|
||
### CMC and CoinGecko Reporting (all tokens, liquidity, volume)
|
||
|
||
Full API for all coins, tokens, liquidity, and reportable data in CMC/CoinGecko-friendly formats:
|
||
|
||
| Endpoint | Description |
|
||
|----------|-------------|
|
||
| `GET /api/v1/report/all` | All tokens, pools, liquidity, volume, and summary by chain |
|
||
| `GET /api/v1/report/coingecko?chainId=138` | Token list in CoinGecko submission format |
|
||
| `GET /api/v1/report/cmc?chainId=138` | Token list and DEX pairs in CoinMarketCap format |
|
||
| `GET /api/v1/report/token-list` | Flat canonical token list, including explicit Chain 138 V1/V2 GRU deployments where available |
|
||
| `GET /api/v1/report/canonical` | Raw canonical token spec with version/family metadata and addresses |
|
||
|
||
See [docs/CMC_COINGECKO_REPORTING.md](docs/CMC_COINGECKO_REPORTING.md) for usage and listing submission.
|
||
|
||
### Token mapping (multichain)
|
||
|
||
When the service runs from the monorepo (with `config/token-mapping-multichain.json` available):
|
||
|
||
| Endpoint | Description |
|
||
|----------|-------------|
|
||
| `GET /api/v1/token-mapping?fromChain=138&toChain=651940` | Token mapping for a chain pair (tokens, addressMapFromTo, addressMapToFrom) |
|
||
| `GET /api/v1/token-mapping/pairs` | All defined chain pairs |
|
||
| `GET /api/v1/token-mapping/resolve?fromChain=&toChain=&address=` | Resolve token address on target chain |
|
||
|
||
## Configuration
|
||
|
||
### DEX Factory Configuration
|
||
|
||
Configure DEX factory addresses in `src/config/dex-factories.ts` or via environment variables:
|
||
|
||
```bash
|
||
# ChainID 138
|
||
CHAIN_138_DODO_POOL_MANAGER=0x...
|
||
CHAIN_138_UNISWAP_V2_FACTORY=0x...
|
||
CHAIN_138_UNISWAP_V3_FACTORY=0x...
|
||
|
||
# ChainID 651940
|
||
CHAIN_651940_UNISWAP_V2_FACTORY=0x...
|
||
CHAIN_651940_UNISWAP_V3_FACTORY=0x...
|
||
```
|
||
|
||
## Monitoring
|
||
|
||
The service includes:
|
||
- Health check endpoint at `/health`
|
||
- Structured logging with Winston
|
||
- Request rate limiting
|
||
- Response caching
|
||
|
||
## License
|
||
|
||
MIT
|