c39465c2bd
Some checks failed
Test / test (push) Has been cancelled
Co-authored-by: Cursor <cursoragent@cursor.com>
234 lines
8.3 KiB
Markdown
234 lines
8.3 KiB
Markdown
# PCIe Slot Allocation Map
|
||
|
||
## Router/Switch/Storage Controller Server
|
||
|
||
This document provides the PCIe slot allocation map for the Router/Switch/Storage Controller server, ensuring optimal lane distribution and avoiding conflicts.
|
||
|
||
## Slot Allocation
|
||
|
||
### Visual Slot Map
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ Router Server PCIe Slots │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [x16_1] Intel QAT 8970 (PCIe 3.0 x16) │
|
||
│ └─ Crypto acceleration (TLS/IPsec/compression) │
|
||
│ │
|
||
│ [x8_1] Intel X550-T2 (2× 10GbE RJ45) │
|
||
│ └─ Future uplinks or direct server links │
|
||
│ │
|
||
│ [x8_2] LSI 9207-8e (SAS HBA #1) │
|
||
│ └─ External storage shelves (2 shelves) │
|
||
│ │
|
||
│ [x8_3] LSI 9207-8e (SAS HBA #2) │
|
||
│ └─ External storage shelves (2 shelves) │
|
||
│ │
|
||
│ [x4_1] Intel i350-T4 (4× 1GbE WAN) │
|
||
│ └─ 4× Spectrum WAN connections │
|
||
│ │
|
||
│ [x4_2] Intel i350-T8 (8× 1GbE LAN) │
|
||
│ └─ Remaining servers and appliances │
|
||
│ │
|
||
│ [x4_3] Intel i225 Quad-Port (4× 2.5GbE LAN) │
|
||
│ └─ Direct to ML110, R630, key services │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Detailed Slot Configuration
|
||
|
||
### Slot x16_1: Intel QAT 8970
|
||
|
||
- **Card:** Intel QuickAssist Technology 8970
|
||
- **Interface:** PCIe 3.0 x16
|
||
- **Lane Usage:** x16 (full bandwidth)
|
||
- **Purpose:** Crypto acceleration for TLS/IPsec/compression
|
||
- **Priority:** High (ensures bandwidth and stability)
|
||
- **Thermal:** High power consumption, ensure adequate cooling
|
||
- **Software:** qatlib drivers, OpenSSL QAT engine
|
||
|
||
**Rationale:** QAT card requires maximum bandwidth for crypto operations. x16 slot ensures no bottlenecks.
|
||
|
||
### Slot x8_1: Intel X550-T2
|
||
|
||
- **Card:** Intel X550-T2 (2× 10GbE RJ45)
|
||
- **Interface:** PCIe 3.0 x8
|
||
- **Lane Usage:** x8 (full bandwidth)
|
||
- **Purpose:** Future 10GbE uplinks or direct server links
|
||
- **Priority:** High (future expansion)
|
||
- **Thermal:** Moderate
|
||
- **Software:** Intel PROSet drivers
|
||
|
||
**Rationale:** 10GbE requires x8 lanes for full bandwidth. CPU-connected slot preferred.
|
||
|
||
### Slot x8_2: LSI 9207-8e (SAS HBA #1)
|
||
|
||
- **Card:** LSI 9207-8e (SAS2308, IT mode)
|
||
- **Interface:** PCIe 3.0 x8
|
||
- **Lane Usage:** x8 (full bandwidth)
|
||
- **Purpose:** External storage shelves (2 shelves)
|
||
- **Priority:** High (storage performance)
|
||
- **Thermal:** Moderate
|
||
- **Software:** mpt3sas driver, IT mode firmware
|
||
- **Cables:** 2× SFF-8644 Mini-SAS HD cables
|
||
|
||
**Rationale:** Storage HBAs require x8 lanes for optimal performance. CPU-connected slot preferred.
|
||
|
||
### Slot x8_3: LSI 9207-8e (SAS HBA #2)
|
||
|
||
- **Card:** LSI 9207-8e (SAS2308, IT mode)
|
||
- **Interface:** PCIe 3.0 x8
|
||
- **Lane Usage:** x8 (full bandwidth)
|
||
- **Purpose:** External storage shelves (2 shelves)
|
||
- **Priority:** High (storage performance)
|
||
- **Thermal:** Moderate
|
||
- **Software:** mpt3sas driver, IT mode firmware
|
||
- **Cables:** 2× SFF-8644 Mini-SAS HD cables
|
||
|
||
**Rationale:** Second HBA for redundancy and additional storage capacity.
|
||
|
||
### Slot x4_1: Intel i350-T4
|
||
|
||
- **Card:** Intel i350-T4 (4× 1GbE ports)
|
||
- **Interface:** PCIe 3.0 x4
|
||
- **Lane Usage:** x4 (full bandwidth)
|
||
- **Purpose:** 4× Spectrum WAN connections
|
||
- **Priority:** High (WAN connectivity)
|
||
- **Thermal:** Low
|
||
- **Software:** Intel PROSet drivers, OpenWrt mwan3
|
||
- **Cables:** 4× Cat6 Ethernet cables
|
||
|
||
**Rationale:** 4× 1GbE requires x4 lanes. WAN connectivity is critical.
|
||
|
||
### Slot x4_2: Intel i350-T8
|
||
|
||
- **Card:** Intel i350-T8 (8× 1GbE ports)
|
||
- **Interface:** PCIe 3.0 x4
|
||
- **Lane Usage:** x4 (full bandwidth)
|
||
- **Purpose:** Remaining servers and appliances
|
||
- **Priority:** Medium
|
||
- **Thermal:** Low
|
||
- **Software:** Intel PROSet drivers, OpenWrt firewall zones
|
||
- **Cables:** 8× Cat6 Ethernet cables
|
||
|
||
**Rationale:** 8× 1GbE can operate on x4 lanes (2 Gbps per lane is sufficient).
|
||
|
||
### Slot x4_3: Intel i225 Quad-Port
|
||
|
||
- **Card:** Intel i225 Quad-Port (4× 2.5GbE ports)
|
||
- **Interface:** PCIe 3.0 x4
|
||
- **Lane Usage:** x4 (full bandwidth)
|
||
- **Purpose:** Direct to ML110, R630, and two key services
|
||
- **Priority:** High (key server connectivity)
|
||
- **Thermal:** Low
|
||
- **Software:** Intel PROSet drivers, OpenWrt firewall zones
|
||
- **Cables:** 4× Cat6 Ethernet cables
|
||
|
||
**Rationale:** 4× 2.5GbE requires x4 lanes for full bandwidth.
|
||
|
||
## Lane Budget Analysis
|
||
|
||
### Total Lane Requirements
|
||
|
||
| Slot | Lanes | Component | Bandwidth |
|
||
|------|-------|-----------|-----------|
|
||
| x16_1 | 16 | Intel QAT 8970 | ~16 GB/s |
|
||
| x8_1 | 8 | Intel X550-T2 | ~8 GB/s |
|
||
| x8_2 | 8 | LSI 9207-8e #1 | ~8 GB/s |
|
||
| x8_3 | 8 | LSI 9207-8e #2 | ~8 GB/s |
|
||
| x4_1 | 4 | Intel i350-T4 | ~4 GB/s |
|
||
| x4_2 | 4 | Intel i350-T8 | ~4 GB/s |
|
||
| x4_3 | 4 | Intel i225 Quad | ~4 GB/s |
|
||
| **Total** | **52** | | **~52 GB/s** |
|
||
|
||
### CPU Lane Availability
|
||
|
||
- **Typical Xeon E-2100:** 16 PCIe 3.0 lanes from CPU
|
||
- **Chipset lanes:** Additional lanes from PCH (varies by chipset)
|
||
- **Total available:** Typically 24-40 lanes depending on chipset
|
||
|
||
**Note:** Ensure motherboard supports sufficient PCIe lanes. Most server motherboards provide adequate lane budget through CPU + chipset combination.
|
||
|
||
## Thermal Considerations
|
||
|
||
### High-Power Components
|
||
|
||
1. **Intel QAT 8970 (x16_1):**
|
||
- Power consumption: ~25-30W
|
||
- Ensure adequate airflow
|
||
- Consider slot spacing if possible
|
||
|
||
2. **LSI 9207-8e HBAs (x8_2, x8_3):**
|
||
- Power consumption: ~10-15W each
|
||
- Moderate thermal load
|
||
- Ensure proper cooling
|
||
|
||
### Cooling Recommendations
|
||
|
||
- Ensure adequate case airflow
|
||
- Consider slot spacing for high-power cards
|
||
- Monitor temperatures during operation
|
||
- Use server-grade case with proper ventilation
|
||
|
||
## Slot Priority and Conflict Resolution
|
||
|
||
### Priority Order
|
||
|
||
1. **Critical (Must have):**
|
||
- x16_1: QAT 8970 (crypto acceleration)
|
||
- x4_1: i350-T4 (WAN connectivity)
|
||
- x8_2/x8_3: LSI HBAs (storage)
|
||
|
||
2. **High Priority:**
|
||
- x8_1: X550-T2 (future expansion)
|
||
- x4_3: i225 Quad (key server connectivity)
|
||
|
||
3. **Medium Priority:**
|
||
- x4_2: i350-T8 (remaining servers)
|
||
|
||
### Conflict Resolution
|
||
|
||
If lane budget is insufficient:
|
||
|
||
1. **Option 1:** Use chipset-connected slots for lower-priority NICs
|
||
2. **Option 2:** Reduce some x8 slots to x4 if card supports it
|
||
3. **Option 3:** Use onboard NICs for some connections
|
||
4. **Option 4:** Upgrade to CPU with more PCIe lanes
|
||
|
||
## Physical Installation Notes
|
||
|
||
### Installation Order
|
||
|
||
1. Install QAT card first (x16_1) - highest priority
|
||
2. Install storage HBAs (x8_2, x8_3) - critical for storage
|
||
3. Install WAN NIC (x4_1) - critical for connectivity
|
||
4. Install LAN NICs (x4_2, x4_3) - complete network setup
|
||
5. Install 10GbE NIC (x8_1) - future expansion
|
||
|
||
### Cable Management
|
||
|
||
- Label all cables at both ends
|
||
- Use cable management accessories
|
||
- Document cable routing
|
||
- Ensure cables don't obstruct airflow
|
||
|
||
## Verification Checklist
|
||
|
||
- [ ] All cards physically installed in correct slots
|
||
- [ ] All cards detected in BIOS/UEFI
|
||
- [ ] All cards detected in OS
|
||
- [ ] Drivers installed and verified
|
||
- [ ] All ports functional
|
||
- [ ] Thermal monitoring active
|
||
- [ ] Cable labeling complete
|
||
- [ ] Documentation updated
|
||
|
||
## Related Documentation
|
||
|
||
- [Hardware BOM](hardware-bom.md) - Complete bill of materials
|
||
- [Complete Architecture](complete-architecture.md) - Full architecture overview
|
||
- [Network Topology](network-topology.md) - Network configuration
|
||
|