wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
1,312 Commits 190 Branches 2 Tags
9c8dde095115bf63ae91bbeff73dcab5da9bd7a0
Commit Graph

605 Commits

Author SHA1 Message Date
OG T
9c8dde0951 fix(telegram): 修復 Incident 無 title 欄位導致所有 Telegram 推送失敗
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m3s
Details 
根因: _push_decision_to_telegram() 有兩處引用 incident.title，
但 Incident model 從來沒有此欄位，導致所有告警卡片推送都
拋 AttributeError，事件在 telegram_decision_push_failed 靜默失敗。

修法:
- line 188: message 改用 signal annotation summary/description/alert_name
- line 249: TYPE-1 title 改用 alertname label / signal.alert_name

影響: 自從 decision_manager 加入這兩行以來，所有 Telegram 通知都沒發出
（包含 TYPE-1 資訊通知和 TYPE-3 審批卡）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 21:02:55 +08:00
OG T
3d8b0e4f90 fix(adr075): TYPE-3 格式改用 spec 模板 — ACTION REQUIRED + AI深度診斷 + 建議修復動作
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m15s
Details 
- 標頭改為 "{emoji} ACTION REQUIRED | {severity_zh}"
- 新增 "🧠 AI 深度診斷" 區塊 (分析/責任/AI來源)
- 新增 "⚡ 建議修復動作" 區塊 (<code> 格式)
- confidence=0 顯示 "📋 規則分析" 取代誤導性 "🔴 0%"
- SignOz 指標區塊補回 Trace 連結

2026-04-12 ogt: ADR-075 TYPE-3 格式標準化

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 21:00:28 +08:00
OG T
a7f2b9c0f5 fix(display): 規則匹配改顯示 ✅ 取代 🔴 0% + 修復 LLM 字串 confidence 解析
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- telegram_gateway.py: confidence==0 (規則匹配/Expert fallback) 不再顯示
  「🔴 0%」，改顯示「⚙️ 規則匹配 ✅」，兩個 card 類型都修正
- openclaw.py: NIM/Ollama 有時回傳字串 "0.85" 而非 float，導致
  isinstance(str, int|float)=False → confidence 被強制設 0.0。
  現在先嘗試 float() 解析，解析失敗才 fallback 0.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 20:50:53 +08:00
OG T
eda0cfd034 fix(adr075): drift 通知改用 send_drift_card，補齊所有呼叫點
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 14m13s
Details 
- drift.py: 移除死碼 send_text()，改由 narrate_and_notify() 統一發卡片
- drift_narrator_service: _send_telegram() 改呼 send_drift_card() 帶四顆按鈕
- webhooks.py /alerts 路徑: 補傳 alert_category 啟用動態按鈕

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 20:20:47 +08:00
OG T
c3fea26222 fix(adr075): webhooks send_approval_card 補傳 alert_category+notification_type
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
斷點真正根因：_push_to_telegram_background 呼叫 send_approval_card()
時沒有傳入 alert_category 和 notification_type，導致動態按鈕永遠
fallback 到通用 [批准][拒絕][靜默]。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 20:07:12 +08:00
OG T
0a4b7e9609 fix(classify): HostBackupFailed 精確補入 backup/TYPE-1（測試通過）
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
前次修法用 'backup' in alertname_lower 太寬，導致 BackupJobFailed warning
被分到 TYPE-1，破壞 test_backup_keyword_warning_not_type1。

改為精確白名單：
  _BACKUP_TYPE1_NAMES = {HostBackupFailed, HostBackupStale, HostBackupMissing,
                         BackupRestoreTestFailed, BackupRestoreTestStale}
  + alertname.startswith('HostBackup') 兜底

結果：664 passed, 0 failed

2026-04-12 ogt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 20:03:46 +08:00
OG T
f25d82a88a fix(adr075): 修補斷點E — _push_to_telegram_background 補 TYPE-8M routing
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
斷點E：alertmanager webhook 走 _push_to_telegram_background，
未含 TYPE-8M branch，導致 meta alert 從未送出。

- webhooks.py: 新增 alert_category 參數 + TYPE-8M branch
- incident_service.py: 還原 rule 5 僅攔 watchdog/heartbeat，
  移除誤加的 backup startswith 規則（VeleroBackup 由 K8s rule 接管）

Tests: 52/52 passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 20:01:51 +08:00
OG T
1f7975170a fix(classify): HostBackupFailed 補入 backup/TYPE-1 規則
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m51s
Details 
classify_alert_early() 的 backup 規則只攔 watchdog/Heartbeat，
HostBackupFailed 先被 Host prefix 規則攔走 → host_resource/TYPE-3 → 跑 LLM → 審批卡。

修法：在 Host prefix 前新增 backup 關鍵字/前綴攔截：
  - HostBackup* / Backup* / VeleroBackup* / BackupRestore*
  - alertname 含 "backup"（大小寫不敏感）

影響：所有備份相關告警直接走 TYPE-1 info 通知，不進 LLM。
HostHighCpu / HostDown 等非備份的 Host 告警不受影響。

2026-04-12 ogt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:52:05 +08:00
OG T
a5f17cea79 fix(notification): TYPE-1 backup/info 告警不再發審批卡
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
classify_notification() 不知道 alert_category，對 backup 告警
（confidence=0, auto_executed=False）返回 TYPE-3，覆蓋掉
classify_alert_early() 已設好的 notification_type=TYPE-1。

修法：在路由分支前，讓 incident.notification_type 明確值
（TYPE-1 / TYPE-4D / TYPE-8M）覆蓋 classify_notification()。

影響：backup/info/watchdog 告警只發 send_info_notification()，
不再噴帶按鈕的審批卡到 Telegram。

2026-04-12 ogt (ADR-075 bugfix)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:49:31 +08:00
OG T
7f3e585d6d fix(webhooks): alertmanager handler — alert_type 超範圍改為 custom
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
AlertPayload.alert_type 只接受 8 個 Literal 值
ALERTNAME_TO_TYPE 映射回傳 host_cpu/backup_failure 等不在白名單 → ValidationError
修法：凡不在 Literal 白名單的 alert_type 一律 fallback 為 "custom"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:22:35 +08:00
OG T
bc2665ef6b feat(adr075): Step-5 decision_manager TYPE-5S/TYPE-6B 路由分支
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 新增 secops elif：alert_category=secops → send_secops_card()
  (resource, threat_behavior 從 incident.signals labels 提取)
- 新增 business elif：alert_category=business → send_business_alert()
  (metric_name/current_value/threshold 從 Prometheus labels 提取)
- TYPE-7E escalation_monitor 標記 out-of-scope (ADR-075 範疇外)
- 兩分支均加 2026-04-12 ogt (ADR-075 Step-5) 變更標記

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:12:35 +08:00
OG T
24c1b5677b feat(adr075): Step1-3 classify補丁+新按鈕+TYPE-5S/6B/7E格式函數 
Step-1 incident_service.py classify_alert_early():
  - 新增 secops (TYPE-5S): UnauthorizedSSH/KubeAudit/CVE/WAFAttack/PodAbnormal
  - 新增 business (TYPE-6B): AITokenCost/GeminiAPIError/SLOBurn/MomoScraper
  - 新增 flywheel_health MCPProvider/OllamaDown/NemotronDown 前綴
  - ssl_cert: 依 days_remaining 決定 TYPE-1(≥14d) vs TYPE-3(<14d)

Step-2 telegram_gateway.py _build_inline_keyboard():
  - 新增 secops: [隔離] [封鎖IP] [驅逐] [確認授權]
  - 新增 business: [暫停1h] [查SignOz] [忽略]
  - 新增 flywheel_health: [觸發診斷] [飛輪面板] [靜默]

Step-3 telegram_gateway.py 新增格式化函數 (Tier 2):
  - send_secops_card() — TYPE-5S 防禦按鈕+nonce
  - send_business_alert() — TYPE-6B 業務損失速率
  - send_escalation_card() — TYPE-7E P0/P1 升級，發 DM+群組

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:50:37 +08:00
OG T
65a5220e16 feat(flywheel-c2-c3): C2 hasType4接真實API + C3 WebSocket指數退避重連
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 3m41s
Details 
C2: flywheel_stats_service 加 type4_count query → API 回傳
    flywheel-diagram.tsx hasType4 改由 type4Count prop 驅動（非 false）
    flywheel-kpi-card.tsx 傳入 type4Count={flowData?.type4_count}

C3: WebSocket onclose 加指數退避重連 (1s→2s→4s→最大30s)
    cancelled 旗標確保 unmount 後不重連
    wsRetryTimer 加入 cleanup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:45:40 +08:00
OG T
1cb654cf59 fix(adr-075): CR P0/P1 修補 — TYPE_8M enum + 死碼清理 + docstring 更新
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P0-2: NotificationType 新增 TYPE_8M = "TYPE-8M"
      classify_notification 早期回傳 TYPE-8M
      decision_manager 改用 NotificationType.TYPE_8M enum 比較（移除字串字面量）
P1-1: 移除 _CATEGORY_BUTTONS 中不可達的 alertchain_health/flywheel_health 條目
P1-4: test_classify_alert_early.py docstring 更新為 13 條規則/10 分類

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:44:12 +08:00
OG T
561c1d806b feat(adr-075): Phase 2 — TYPE-8M 飛輪/告警鏈路健康通知格式與路由
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 4m0s
Details 
新增 send_meta_alert() — ⚙️ META SYSTEM 卡片（觸發診斷/查看面板/靜默）
decision_manager 新增 TYPE-8M elif 分支（在 TYPE-4D 後）
_alert_category 提取提前至 if 鏈前，三個分支共用

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:39:04 +08:00
OG T
2cef2098d3 feat(adr-075): 修復 Telegram 動態按鈕 4 個斷點 + 新增 7 種告警分類
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
斷點 A: decision_manager 提取 alert_category/notification_type 傳入 send_approval_card
斷點 B: send_approval_card 新增參數並傳遞至 _build_inline_keyboard
斷點 C: 互動型通知 (TYPE-3/4/4D/8M) 禁止發 SRE 群組，防 nonce 洩漏
斷點 D: _CATEGORY_BUTTONS k8s_workload → kubernetes + 新增 6 類按鈕組

classify_alert_early 新增: alertchain_health, flywheel_health, storage,
devops_tool, external_site, ssl_cert, host_resource (從 infrastructure 分離)
Test: 52 classify + 664 total passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:35:56 +08:00
OG T
587d745a50 fix(km): 修補 KMConversionService 兩個屬性錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 28s
Details 
- incident.title → getattr(incident, 'title', None) or alertname
  (Incident model 無 title 欄位)
- km_entry.entry_id → km_entry.id
  (KnowledgeEntry model 主鍵為 id 非 entry_id)
- 補跑後 KM entries 714 → 821 (+107), incidents.vectorized 全部歸零

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 17:52:57 +08:00
OG T
38dddcc7a2 fix(heartbeat): KM向量化改用raw SQL + 格式優化去除空格對齊
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 29s
Details 
- KM vectorized 改用 raw SQL (ORM 無 embedding 欄位)
- 移除 {display:<18} 空格對齊（非等寬字體Telegram會錯位）
- 格式: Name: value 每行一項，清楚易讀
- KM向量化加狀態icon (✅ ≥90% / ⚠️ <90%)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 17:36:01 +08:00
OG T
a1691c41d5 fix(flywheel-stats): 修補 FlywheelStatsService 三個欄位錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 30s
Details 
- KnowledgeEntryRecord.vectorized → embedding.is_(None) (欄位不存在)
- IncidentRecord.id → IncidentRecord.incident_id (主鍵名稱)
- 修復後 /api/v1/stats/flywheel nodes 不再全部回傳 unknown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 17:27:35 +08:00
OG T
99b489ca63 fix(flywheel): 修補剩餘 P0/P1 缺陷
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- CRITICAL-1: TYPE-1 path approval_id=str(alert_id) → uuid.uuid4()，
  避免 UUID(approval_id) 拋 ValueError 導致所有 Heartbeat/Info 告警崩潰
- CRITICAL-2: asyncio.create_task() 結果存入 _exec_task 並加 done_callback，
  防止 GC 在執行中途回收任務
- FORMAT: _push_to_telegram_background 新增 notification_type + diff_summary 參數，
  TYPE-4D → send_drift_card()，其他 → send_approval_card()（修正 ConfigDrift 顯示錯誤卡片）
- 傳遞 notification_type 至 Alertmanager 兩個呼叫點

ADR-073 四斷點修補最終收尾

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 17:14:57 +08:00
OG T
f0e14136ca fix(flywheel): 修補飛輪四個核心斷點，讓完整流程真正串接起來
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
1. incident_service.py: save_to_episodic_memory() 補寫 alertname/notification_type/alert_category
   → 之前這3欄在DB永遠NULL，LLM無alertname，Playbook匹配全失敗

2. telegram_gateway.py: Telegram批准後呼叫 execute_approved_action()
   → 之前sign_approval()只改DB狀態，380筆批准0筆真正執行kubectl指令

3. approval_execution.py: 執行成功後呼叫 resolve_incident()
   webhooks.py: auto-repair成功後呼叫 resolve_incident()
   → 之前Incident永遠停在INVESTIGATING，KM轉換永遠不觸發，Playbook=0

4. webhooks.py: TYPE-1告警短路，不進LLM
   → 之前Heartbeat/Backup/Info仍燒LLM token，產生垃圾修復建議

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 17:01:10 +08:00
OG T
93f9522d5a fix(heartbeat): 對齊整點發送避免多replica各自發 + KM向量化改查embedding欄位
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 14m10s
Details 
- _heartbeat_loop: 先 sleep 到下一個整點倍數再開始循環
  避免 3 個 replica 啟動時間不同導致短時間內收到多條心跳
- heartbeat_report_service: km_vectorized 改查 KnowledgeEntryRecord.embedding IS NOT NULL
  原本錯誤查 IncidentRecord.vectorized 導致顯示 0/714 (0%)

2026-04-12 ogt (ADR-073 heartbeat fix)
 2026-04-12 16:33:15 +08:00
OG T
effd78807e fix(heartbeat): blocking_timeout 5→0，多 replica 不排隊等鎖避免重複發送
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 14m0s
Details 
3 個 replica 各自跑 loop，blocking_timeout=5.0 導致鎖釋放後
其他 replica 依序拿鎖，每次心跳最多發 3 條。
改為 blocking_timeout=0：拿不到鎖立刻跳過，同週期只發一條。

2026-04-12 ogt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 16:13:41 +08:00
OG T
a28625f088 fix(cr): 首席架構師 CR P0/P1/P2 全修補
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P0-1: incident_service.py — 刪除 classify_alert_early 死碼 L131-132
P0-2: cron_backup_restore_test.sh — date +%s%3N→+%s，修正毫秒時間戳
P1-2: gitea_webhook.py — fingerprint 移除 sha_short，收斂同 branch 失敗
heartbeat: 還原原始空格對齊格式（統帥要求原本怎樣就怎樣）

P1-1(積木化)/P1-3(TYPE-4)/P2-1(timeZone)/P2-2(IP)/P2-3(WS重連) 待後續處理

2026-04-12 ogt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 16:10:46 +08:00
OG T
d72c7d5ac4 fix(P0): classify_alert_early 參數名稱修正 _labels→labels
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
webhooks.py 呼叫傳 labels= 但函數定義用 _labels，導致所有
Alertmanager webhook 500，告警鏈路完全中斷。

2026-04-12 ogt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 16:02:25 +08:00
OG T
36f285fb85 fix(heartbeat): 移除空格對齊，改用直接排版避免 Telegram 跑版 
Telegram HTML 模式不渲染等寬字型，空格對齊無效。
改成不對齊但清晰的格式，每行直接顯示 label + value。

2026-04-12 ogt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 16:01:47 +08:00
OG T
0c2892ac19 feat(c3): ADR-073-C C3 — WebSocket 飛輪即時推送 
後端：
- stats.py 新增 @router.websocket('/flywheel/ws')
- 每 10 秒推送 flywheel_summary JSON

前端 FlywheelKPICard：
- WebSocket 優先，WS 斷線自動降級到 30s HTTP 輪詢
- onopen 時停止 HTTP polling，onclose 時恢復

2026-04-12 ogt (ADR-073-C C3)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:40:20 +08:00
OG T
3489e05c84 feat(m3): ADR-074 M3 — Gitea CI/CD 管線失敗 webhook 
新增 workflow_run 事件處理：
- GiteaWorkflowRun Pydantic model
- handle_workflow_run() — status/conclusion=failure → TYPE-1 Incident
- 透過 get_incident_service().create_incident_from_signal() 建立告警
- 純通知路徑，不觸發自動修復

2026-04-12 ogt (ADR-074 M3)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:35:25 +08:00
OG T
00a31abb85 feat(heartbeat): ADR-073 P2 心跳整合重構 — HeartbeatReportService + RedisLock
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 新增 HeartbeatReportService：11 個並行探針（Ollama/Nemotron/Gemini/Claude/MCP×4/ArgoCD/Velero）
- 重寫 send_heartbeat()：RedisLock 防重發 + 統一發送 SRE_GROUP_CHAT_ID
- 簡化 _heartbeat_loop()：移除散落的 silence 多次發送
- config.py：新增 OLLAMA_REQUIRED_MODELS 欄位
- 03-secrets.example.yaml：補 SRE_GROUP_CHAT_ID 確保 CD Inject 不遺漏

2026-04-12 ogt (ADR-073 Phase 2-3/4)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:35:13 +08:00
OG T
16d682346a feat(adr-074): M1 飛輪健康度 Exporter + M2 主機網路監控
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-074 M1:
  - FlywheelStatsService: 計算6項飛輪指標（Playbook數/成功率/KM向量化/alertname NULL/卡住數）
  - GET /api/v1/stats/flywheel — 六節點即時狀態（C1 前端用）
  - GET /api/v1/stats/summary — KPI 面板數據（C1 前端用）
  - GET /api/v1/stats/flywheel/metrics — Prometheus text format
  - flywheel-alerts.yaml: 5條告警規則（FlywheelPlaybookZero/ExecutionSuccessLow/KMVectorizationLow/AlertnameNullHigh/IncidentsStuck）
  - prometheus.yml: awoooi-flywheel scrape job（5分鐘間隔）

ADR-074 M2:
  - prometheus.yml: host-connectivity Blackbox TCP probe（110:22/188:22/120:6443/121:6443）
  - flywheel-alerts.yaml: HostNetworkPartition 告警規則

597 unit tests passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:31:01 +08:00
OG T
1074936e54 fix(classify): backup/heartbeat severity=warning/critical 告警恢復告警卡片格式
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m38s
Details 
根因：classify_alert_early() backup 規則無 severity 條件，導致
VeleroBackupFailed / HostBackupFailed (warning/critical) 被分為 TYPE-1
（純資訊無按鈕），告警卡片格式遺失。

修復：
- backup/heartbeat 關鍵字只在 severity=info/none 才命中 TYPE-1
- severity=warning/critical 的 backup 告警走正確 prefix 規則
  (Velero→kubernetes TYPE-3, HostBackup→infrastructure TYPE-3)
- Watchdog (severity=none) 由 severity 規則先命中，維持 TYPE-1
- 補強測試：25 cases，含 VeleroBackupFailed critical → kubernetes TYPE-3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:24:00 +08:00
OG T
0d239838b4 fix(cr): Code Review P2 — 測試覆蓋 + CronJob 腳本重構
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P2-1: CronJob inline Python 抽成 scripts/cron_km_vectorize.py
      Dockerfile 加入 COPY scripts/，CronJob YAML 改用腳本路徑
P2-2: 新增 test_classify_alert_early.py — 23 tests 覆蓋 7 條分類規則
      含邊界情況：VeleroBackupFailed(backup優先於k8s)、優先順序驗證

595 unit tests passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:14:44 +08:00
OG T
c09521a1c6 fix(cr): Code Review P0/P1 全修補 — 積木化+SSH路由+安全守衛順序
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m30s
Details 
P0-1: classify_alert_early 移至 incident_service (Service層)，webhooks.py import 修正
P0-2: _ssh_execute() 改用 self._ssh，移除冗餘 SSHProvider() 實例化
P1-1: infrastructure SSH routing 移至 kubectl safety guard 之前，docker指令不再被攔截
P1-2: alert_rule_engine 新增 get_risk_for_alertname() public API
P1-3: classify_notification() docstring 修正 ORM→Pydantic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:51:12 +08:00
OG T
f2fc4712ad feat(flywheel): Phase 4 — KM conversion hook + daily vectorize cron
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-073 Phase 4-2: incident_service.resolve_incident() KM conversion hook
- resolve 時 fire-and-forget KMConversionService.convert(incident)
- 已解決的 Incident 自動轉換為結構化 KM 條目，完成飛輪「學習固化」節點
- KMConversionService (Phase 4-1) 已存在 (km_conversion_service.py, 336 lines)

ADR-073 Phase 4-3: 15-cronjob-km-vectorize.yaml
- 每日 03:00 台北時間呼叫 /api/v1/knowledge/embed-all
- 自動向量化當日新增 KM 條目，確保 RAG 查詢不遺漏新知識
- 加入 kustomization.yaml resources

Phase 4-4: handle_callback log_manual_fix 已存在 (telegram_gateway.py:2468)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:40:33 +08:00
OG T
dbc77c5e62 feat(flywheel): Phase 3 — decision_manager Tier 3 七大修復 (首席架構師授權)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-073 Phase 3 全部完成:

3-1: TYPE-1 triage guard
- get_or_create_decision() 入口: notification_type=TYPE-1 直接 bypass LLM 分析
- classify_notification() 優先讀 incident.notification_type (早期分診結果)
- ConfigurationDrift/KubeConfigDrift 補入 TYPE-4D 匹配清單

3-2: infrastructure → SSH MCP routing
- _auto_execute() 中 alert_category=infrastructure + 非 kubectl action → _ssh_execute()
- _ssh_execute(): docker_restart / service_restart tool 路由
- 取 instance label 對應 SSH_MCP_ALLOWED_HOSTS 白名單主機

3-3: send_info_notification() TYPE-1 已存在，classify_notification 修復確保正確呼叫

3-4: Dynamic button builder 已存在 _build_inline_keyboard + _CATEGORY_BUTTONS

3-5: action | parse fix
- _auto_execute() 開頭: action 含 | 時取第一段 (LLM 有時輸出 "kubectl X | kubectl get")

3-6: risk_level YAML priority override LLM
- dual_engine_analyze() LLM 結果返回後，用 alert_rules.yaml 對應 rule.risk 覆蓋

3-7: send_drift_card() TYPE-4D 已存在，classify_notification 修復確保正確觸發

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:39:19 +08:00
OG T
5b956a9a47 feat(flywheel): Phase 2-3/2-5 — auto_repair outcome 寫入 + 134 筆 alertname 回填腳本
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-073 Phase 2-3: _try_auto_repair_background() 修復執行後寫入 Incident.outcome
- effectiveness_score: 5(成功) / 2(失敗)
- human_feedback: auto_repair:<playbook_id>:success|failed
- should_remember: True(成功) → KMConversionService 飛輪入口
- 讓 KMConversionService 可依 outcome 判斷 EXECUTION_SUCCESS

ADR-073 Phase 2-5: scripts/backfill_alertname.py
- UPDATE incidents SET alertname = COALESCE(signals->0->>'alertname', signals->0->>'alert_name')
- 已在 Pod 執行：134 筆 NULL → 0 筆 (2026-04-12 ogt)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:33:11 +08:00
OG T
d4b8b1588b feat(flywheel): Phase 2-2/2-4 — classify_alert_early + alertname/notification_type/alert_category 寫入 
ADR-073 Phase 2-2: 早期分診，在 LLM 分析前決定 alert_category + notification_type
- webhooks.py: 新增 classify_alert_early() — 6 條規則覆蓋 config_drift/info/backup/infra/k8s/db/general
- webhooks.py: alertmanager_webhook 呼叫 classify_alert_early() 並傳入兩個 create_incident_for_approval() 呼叫點
- incident_service.py: create_incident_for_approval() 新增 notification_type/alert_category 參數，寫入 Incident model
- incident_repository.py: _incident_to_record_data() 新增 alertname/notification_type/alert_category 序列化
- db/models.py: IncidentRecord ORM 新增 alertname/notification_type/alert_category 三個 mapped_column

防止 HostBackupFailed 等告警被誤路由到 K8s executor (ADR-073 Phase 2-4 同步完成)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:33:11 +08:00
OG T
7c4b36c2cd fix(flywheel): Phase 1 — 部署 8be87b0 + debounce 30min + alertname NULL 修復
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m29s
Details 
ADR-073 Phase 1 三項修復:

1. k8s/kustomization.yaml: newTag a86ecf3 → 8be87b0
   - 解封 _collect_mcp_context + auto_approve + DESTRUCTIVE_PATTERNS
   - 這是飛輪解封的關鍵

2. webhooks.py: DEBOUNCE_WINDOW_MINUTES 5 → 30
   - 防止同一問題每 5 分鐘重建 Incident，改為 30 分鐘收斂窗口

3. incident_repository.py: signals JSONB 補充 alertname key
   - signal.model_dump() 只有 alert_name，DB query 用 signals->0->>'alertname'
   - 補充 alertname alias，修復 132 筆 incidents.alertname = NULL 根因

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 13:41:22 +08:00
OG T
a67a27f780 fix(test): test_model_regression 加 @pytest.mark.integration（需 Ollama 服務） 
與 global_repair_cooldown / anomaly_counter 一致，
Ollama 測試預設排除，需真實服務時用 pytest -m integration 執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 13:32:42 +08:00
OG T
184b37a8b1 refactor(decision_manager): I2 DI 化 MCP Providers + fix config list type bug 
- DecisionManager.__init__ 注入 SSHProvider/K8sProvider，移除函數內 import+實例化
- config.get_tg_user_whitelist() 支援 list 輸入（monkeypatch/直接傳入），修復 AttributeError
- LOGBOOK 更新（test fix 6e0ee8b）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 13:04:46 +08:00
OG T
6e0ee8b413 fix(test): 排除 integration 測試防止 Redis 未初始化錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 34s
Details 
pytest 預設排除 @pytest.mark.integration 標記的測試（需真實 Redis）。
如需執行整合測試：pytest -m integration

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 12:37:54 +08:00
OG T
8be87b0f32 fix(review): 首席架構師 Code Review — c439277 Tier 3 紅區修補
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m39s
Details 
Critical:
- C1: decision_manager _collect_mcp_context container 變數 Python ternary 優先度 bug 修正
  原: `A or B or C[0] if list else ""` (ternary 控制全式)
  修: `A or B or (C[0] if list else "")` (明確括號)
- C2: 所有 MCP 呼叫加 asyncio.wait_for timeout=5s，防止阻塞決策主路徑
  同時加 unknown host warning log (C4)
- C3+M1: _DESTRUCTIVE_PATTERNS 補全移至模組頂層常量
  新增: delete pods(複數)/kubectl drain/kubectl cordon/kubectl rollout undo/
        docker rm/docker stop/docker kill/rm -rf/"replicas": 0(JSON patch)

Important:
- I1: webhooks.py IP 排除改用 is_internal_ip() 支援全 RFC-1918 (10.x/172.16-31.x/192.168.x)
- I4: 新增 test_destructive_patterns.py — 25 測試全過
  涵蓋: 常量存在、攔截、誤攔迴歸、critical 永遠攔截

🔴 Tier 3 紅區 — 首席架構師 Code Review 通過後 push

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 22:05:52 +08:00
OG T
c439277fc3 feat(aiops): ADR-070 全自動化方向 — 三大修復
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
1. auto_approve.py: 允許 high risk 自動執行 (low/medium/high 全開放)
   - min_confidence 0.65→0.50 (信心門檻降低)
   - 新增 DESTRUCTIVE_PATTERNS 攔截真正危險指令
     (scale=0, delete deployment/pvc/namespace, drop table)
   - 核心: critical + 破壞性操作 → 人工; 其他 → 全自動

2. decision_manager.py: 新增 _collect_mcp_context()
   - LLM 分析前先收集真實環境狀態 (SSH/K8s MCP)
   - Host/Docker 告警 → ssh_get_container_status + ssh_get_top_processes
   - K8s 告警 → k8s_get_events
   - 注入 diagnosis_context "當前環境狀態 (MCP 實時查詢)" 區段

3. webhooks.py: 修復 target_resource 提取
   - 新增 name/container/job label 提取
   - DockerContainerUnhealthy 不再 target=alertname
   - IP 位址自動排除 (192.x 開頭不作為 target)

🔴 Tier 3 紅區 — 需首席架構師批准
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:39:52 +08:00
OG T
d77b2add73 fix(review): 首席架構師 Code Review 修補 — I1 get_incident_type 邏輯修正 + 測試補全
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m13s
Details 
Code Review 發現 2 個 Critical + 2 個 Important 問題：

Critical：
- rule.id 語意為「規則識別符」，與 incident_type 命名空間不同，不可混用
  移除 rule_id fallback 路徑，YAML 匹配無 incident_type 時 fall through 靜態 dict
- get_incident_type() 關鍵路徑無測試覆蓋
  新增 test_get_incident_type.py：11 測試、4 類別（靜態/YAML優先/YAML錯誤/custom）全過

Important：
- ALERTNAME_TO_TYPE deferred import 移至模組頂層（無 circular 風險）
- alert_types.py TODO 過期 → 更新為 I1 整合後正確說明

技術債記錄：NetworkPolicy ArgoCD egress ClusterIP 10.43.16.201/32 需 ArgoCD 重裝後更新

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:33:19 +08:00
OG T
b2dfcf9b0d fix(telegram): safety guard 攔截改發人工審核卡片，不再發 ❌ 失敗訊息 
問題：AI 無法確認 deployment name 時，每次告警都發一條
「❌ 自動修復失敗 kubectl scale deployment unknown」的垃圾訊息

修復：
- safety guard 攔截 → token.state 回 READY（非 ERROR）
- 改呼叫 _push_decision_to_telegram，發 TYPE-4 人工審核卡片
- mcp_all_failed=True 讓 classify_notification 選 TYPE-4
- K8s 找不到 target 的路徑同樣處理

效果：統帥看到的是「需要人工介入的審核卡片」而非「修復失敗」錯誤訊息

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:33:19 +08:00
OG T
615822dcf3 feat(I1): ADR-064 Rule Engine 整合 — 動態推斷 incident_type
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 11m28s
Details 
- alert_rule_engine.py: 新增 get_incident_type(alertname)
  優先從 YAML 規則 match.alertname 查找 incident_type/rule_id
  Fallback: ALERTNAME_TO_TYPE 靜態 dict → "custom"
- webhooks.py: alert_type 改用 get_incident_type(alertname)
  取代 ALERTNAME_TO_TYPE.get() 靜態查找
- YAML 規則 19 條 alertname 覆蓋自動生效（無需手改 dict）
- 新 alertname 觸發 generic_fallback → auto_generate_rule() 後自動加入 YAML

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:21:41 +08:00
OG T
1ede9f933f refactor(M3): alertname_to_type 抽至 src/constants/alert_types.py
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 新增 src/constants/__init__.py + alert_types.py
- ALERTNAME_TO_TYPE 常數（56 筆）從 webhooks.py 內聯 dict 遷移至模組
- webhooks.py 改用 ALERTNAME_TO_TYPE.get(alertname, "custom")
- TODO I1: 下 Sprint 整合 ADR-064 Rule Engine 動態推斷（此為中間狀態）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:19:52 +08:00
OG T
f2c18c4e63 feat(D1): models.json 集中化 — ADR-067 五大 Ollama 應用 hardcode 消除
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m56s
Details 
- models.json v1.3.0: providers.ollama.models 新增 9 個 purpose keys
  (drift_summary/drift_intent/log_anomaly/nemoclaw/playbook_draft/
   code_review/embedding/rag_generate/image_analysis)
- drift_narrator_service: NARRATOR_MODEL → get_model("ollama","drift_summary")
- drift_interpreter: MODEL → get_model("ollama","drift_intent")
- log_summary_service: SUMMARY_MODEL → get_model("ollama","log_anomaly")
- local_code_review_service: _MODEL_OLLAMA → get_model("ollama","code_review")
- image_analysis_service: _MODEL → get_model("ollama","image_analysis")
- decision_manager: nemoclaw + playbook_draft 兩處 → get_model()
- embedding_service: get_embedding_service() factory → get_model("ollama","embedding")
- knowledge_service: OllamaEmbeddingService(model=...) → get_model()

所有模型名稱現在統一由 models.json 管理，修改模型只需改一個檔案。
LOGBOOK 更新：D1 完成 + B2 已完成確認

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:45:53 +08:00
OG T
82e1c05df8 fix(review): Code Review C1/C2/I2/M2 修補
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
C1 drift_interpreter: 寫死 192.168.0.111 → settings.OLLAMA_URL
  違反 feedback_frontend_internal_ip_ban 鐵律（後端 service 層同樣禁止寫死內網 IP）

C2 km_conversion_service: BUG-004 補同步 Redis Working Memory vectorized 欄位
  原修復只更新 DB，Redis incident:{id} JSON 的 vectorized 未同步
  → 審計查 Redis 仍顯示 False，fly-wheel 閉環指標仍不準
  修復：DB 更新後 GET → JSON patch vectorized=True → SET（保留原 TTL）

I2 decision_manager: _ALERTNAME_KEYWORDS HostHighDiskUsage→HostOutOfDiskSpace
  + 補 DockerContainerExited
  + fallback 路徑加 debug log

M2 decision_manager: import json as _json 從 for 迴圈移至方法頂部

docs: ADR-072 新增 Code Review 發現與技術債記錄

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:36:59 +08:00
OG T
e447f97616 fix(telegram): 接通 classify_notification + 修復 HostBackupFailed 亂送按鈕 
三個問題同時修復：

1. classify_notification() 死程式碼接通
   - _push_decision_to_telegram() 現在先呼叫 classify_notification()
   - TYPE-1 (純資訊) → send_info_notification()，無按鈕
   - TYPE-4D (Config Drift) → send_drift_card()
   - 其餘 TYPE-2/3/4 → send_approval_card()（原有按鈕）
   - decision_state + auto_executed 從呼叫端注入 proposal_data

2. alert_rules.yaml 補 host_backup_failed 規則
   - HostBackupFailed / VeleroBackupFailed / VeleroBackupNotRun → NO_ACTION
   - 不再走 generic_fallback → 不再產生 kubectl rollout restart deployment/backup

3. _verify_k8s_deployment_exists() 主機層告警不再保守放行
   - Host*/Docker*/Backup*/Velero*/SSH* 前綴告警 → K8s MCP 不可用時 return False
   - _auto_execute() 收到 NO_ACTION 或空 kubectl_command → 早退，不執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:35:48 +08:00