fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:
A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
· success/timeout/error outcome label
· histogram 寫入 aiops_agent_step_duration_seconds
A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout
新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality
新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序
新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
· 三 Agent histogram alert rules(p99 > timeout 80% → warning)
驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)
INC-20260425 雙修總工作量(595629c0 + 此 commit):
· 5 個 service/agent 檔修改
· 1 個新 observability 模組
· 4 個新測試/配套檔
· 1372+187 = 1559 行新增
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
This commit is contained in:
@@ -11,13 +11,14 @@ AWOOOI AIOps Phase 2 — 多 Agent 協作訊息協定
|
||||
|
||||
ADR-082: 多 Agent 協作架構(Phase 2)
|
||||
2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 2 初始建立
|
||||
2026-04-27 Claude Sonnet 4.6: B1 — 新增 RecommendedAction schema(北極星 §1.1 修復多樣性 ≥ 40%)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
@@ -102,6 +103,34 @@ class CandidateAction:
|
||||
rationale: str = "" # 為什麼選此方案
|
||||
|
||||
|
||||
# 2026-04-27 Claude Sonnet 4.6: B1 — Solver 結構化動作 (北極星 §1.1 修復多樣性 ≥ 40%)
|
||||
# RecommendedAction 是 ActionPlan.recommended_actions 的元素,供 B3 Telegram 按鈕動態生成用。
|
||||
# 與 CandidateAction(kubectl 命令字串)不同:RecommendedAction 指向 MCP tool(可被 B2 allowlist 審核)。
|
||||
@dataclass
|
||||
class RecommendedAction:
|
||||
"""
|
||||
結構化推薦修復動作(B1 新增,供 Telegram 按鈕動態生成)
|
||||
|
||||
與 CandidateAction 的差異:
|
||||
- CandidateAction:kubectl 命令字串(供 Coordinator 判斷)
|
||||
- RecommendedAction:MCP tool 呼叫規格(供 B3 Telegram 按鈕動態渲染)
|
||||
|
||||
mcp_provider 必須在 callback_action_spec.yaml 的 provider 清單內。
|
||||
mcp_tool 必須在 B2 allowlist(待 B2 任務建立)。
|
||||
params 支援模板替換:{labels.xxx} / {incident_id}。
|
||||
"""
|
||||
name: str # action 識別(如 check_pod_logs)
|
||||
label: str # UI 顯示文字(如「查 Pod 日誌」)
|
||||
emoji: str # UI 圖示(如「📋」)
|
||||
mcp_provider: Literal[ # MCP provider 限制在已知清單
|
||||
"k8s", "ssh", "prometheus", "signoz", "database", "internal"
|
||||
]
|
||||
mcp_tool: str # MCP tool 名(必須在 B2 allowlist)
|
||||
params: dict[str, str] # 參數模板(支援 {labels.xxx} / {incident_id})
|
||||
risk: Literal["low", "medium", "high", "critical"] # 風險等級
|
||||
reasoning: str # 為何推薦此動作(讓 critic 能審)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActionPlan:
|
||||
"""
|
||||
@@ -109,12 +138,18 @@ class ActionPlan:
|
||||
|
||||
對每個根因假設提出 ≥1 個候選方案(含 blast_radius / rollback_cost)。
|
||||
blast_radius > 50 → Reviewer 必須標 `request_revision`。
|
||||
|
||||
2026-04-27 Claude Sonnet 4.6: B1 新增 recommended_actions(結構化動作清單)
|
||||
- recommended_actions 為空 list 代表降級(degraded=True)或 LLM 無法輸出合法動作
|
||||
- Coordinator 舊邏輯只讀 candidates,不受影響
|
||||
"""
|
||||
candidates: list[CandidateAction]
|
||||
diagnosis_report: DiagnosisReport
|
||||
latency_ms: int
|
||||
vote: AgentVote = AgentVote.APPROVE
|
||||
degraded: bool = False
|
||||
# 2026-04-27 Claude Sonnet 4.6: B1 — 結構化推薦動作(0-3 個,降級時為 [])
|
||||
recommended_actions: list[RecommendedAction] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def top_candidate(self) -> CandidateAction | None:
|
||||
|
||||
Reference in New Issue
Block a user