ogt
|
77d3a1da48
|
feat(ai-ops): ADR-013 AIOps 自動修復閉環完整實作
CD Pipeline / deploy (push) Failing after 3m24s
架構(Exception → Incident → PlayBook → Heal → KM → Telegram):
新增元件:
- database/autoheal_models.py: Incident/Playbook/HealLog 三張表 + 7 條種子 PlayBook
- migrations/013_autoheal.sql: 建表 DDL + 種子資料(冪等 INSERT)
- services/auto_heal_service.py: 核心引擎 7 步閉環
- _classify_error: 8 類錯誤自動分類 (DNS_FAIL/DB_UNREACHABLE/OOM/...)
- _match_playbook: error_type + keyword + 冷卻 + max_retries 保護
- _execute_playbook: DOCKER_RESTART/SSH_CMD/ALERT_ONLY/WAIT_RETRY
- _sink_to_km: 修復知識寫入 ai_insights (auto_heal_playbook)
- SSH 白名單:僅允許 docker restart / compose restart / docker start
修改元件:
- database/manager.py: _init_autoheal_tables() 啟動時建表+種子 PlayBook
- scheduler.py: 3 個核心任務植入 handle_exception
(run_auto_import_task / run_icaim_analysis_task / run_weekly_strategy_task)
- requirements.txt: paramiko(SSH 跳板;不可用時降級 subprocess+CLI ssh)
安全設計: CMD 白名單 + cooldown + max_retries escalation + DB 冪等 migration
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-19 16:03:49 +08:00 |
|