docs: ADR-020 E2E 驗證框架規範
Phase 18.3 配套決策文檔: - E2E 驗證腳本架構 (5 步驟標準) - Safe Label 防護機制 - Daily Health Check 排程規範 - 目標資源驗證要求 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
98
docs/adr/ADR-020-e2e-verification-framework.md
Normal file
98
docs/adr/ADR-020-e2e-verification-framework.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# ADR-020: E2E 驗證框架規範
|
||||
|
||||
| 欄位 | 值 |
|
||||
|------|-----|
|
||||
| **狀態** | Accepted |
|
||||
| **建立日期** | 2026-03-26 |
|
||||
| **決策者** | 首席架構師 (Claude Code) |
|
||||
| **相關 ADR** | ADR-016 (K8s 資源命名), ADR-018 (LLM 測試) |
|
||||
|
||||
---
|
||||
|
||||
## 背景
|
||||
|
||||
Phase 18 建立了完整的 E2E Tool Call 驗證流程:
|
||||
- Alert → AI 分析 → Approval 建立 → 執行
|
||||
- 需要標準化腳本規範,確保持續維護與擴展
|
||||
|
||||
## 決策
|
||||
|
||||
### 1. E2E 驗證腳本架構
|
||||
|
||||
```
|
||||
apps/api/scripts/
|
||||
├── e2e_tool_call_verification.py # 主驗證腳本 (v2.0)
|
||||
└── __init__.py # 模組入口
|
||||
```
|
||||
|
||||
### 2. 驗證步驟標準
|
||||
|
||||
| Step | 名稱 | 驗證內容 | 超時 |
|
||||
|------|------|---------|------|
|
||||
| 1 | 發送測試告警 | Webhook 回應成功 | 120s |
|
||||
| 2 | 驗證 AI 分析 | 目標資源正規化、風險評估 | 60s |
|
||||
| 3 | 檢查 Approval 建立 | ID 存在、狀態 pending | 30s |
|
||||
| 4 | 動態簽署驗證 | 根據風險等級決定簽署數 | 60s |
|
||||
| 5 | 執行結果檢查 | 狀態更新、無錯誤 | 30s |
|
||||
|
||||
### 3. Safe Label 防護
|
||||
|
||||
所有 E2E 測試告警必須包含:
|
||||
|
||||
```json
|
||||
{
|
||||
"labels": {
|
||||
"env": "e2e-test",
|
||||
"safe_mode": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
執行層應檢測 `safe_mode=true` 並跳過實際 K8s 操作。
|
||||
|
||||
### 4. Daily Health Check 排程
|
||||
|
||||
| 欄位 | 值 |
|
||||
|------|-----|
|
||||
| **Workflow** | `.github/workflows/daily-e2e-health.yaml` |
|
||||
| **排程** | 每日 00:30 UTC (08:30 台北) |
|
||||
| **Runner** | self-hosted (harbor, k8s) |
|
||||
| **失敗通知** | Telegram (OPENCLAW_TG_*) |
|
||||
|
||||
### 5. 目標資源驗證 (Phase 18.1)
|
||||
|
||||
E2E 腳本必須驗證 AI 產生的 kubectl 指令:
|
||||
|
||||
```python
|
||||
def verify_action_target(action: str, expected_resource: str) -> bool:
|
||||
"""確保 AI 沒有殺錯人"""
|
||||
return expected_resource in action
|
||||
```
|
||||
|
||||
## 後果
|
||||
|
||||
### 正面
|
||||
- 自動化驗證完整 Tool Call 鏈路
|
||||
- 早期發現 AI 產生無效指令
|
||||
- 持續監控系統健康
|
||||
|
||||
### 負面
|
||||
- 需維護測試腳本與 Daily Health Check workflow
|
||||
- Telegram 通知依賴外部服務
|
||||
|
||||
## 相關檔案
|
||||
|
||||
| 檔案 | 功能 |
|
||||
|------|------|
|
||||
| `scripts/e2e_tool_call_verification.py` | E2E 驗證腳本 v2.0 |
|
||||
| `.github/workflows/daily-e2e-health.yaml` | Daily Health Check |
|
||||
| `src/utils/k8s_naming.py` | 資源名稱正規化 |
|
||||
| `ADR-016` | K8s 資源命名契約 |
|
||||
|
||||
---
|
||||
|
||||
## 審查紀錄
|
||||
|
||||
| 日期 | 審查者 | 結果 |
|
||||
|------|--------|------|
|
||||
| 2026-03-26 | Claude Code | Accepted |
|
||||
Reference in New Issue
Block a user