Your Name
|
fe618960a8
|
fix(ops): monitor systemd runners in host baseline
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
Code Review / ai-code-review (push) Has been cancelled
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
|
2026-05-05 14:08:43 +08:00 |
|
Your Name
|
e8e6748f70
|
fix(ops): add docker host resource baseline guardrails
CD Pipeline / tests (push) Failing after 1m50s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
Code Review / ai-code-review (push) Successful in 25s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 38s
|
2026-05-05 13:45:09 +08:00 |
|
Your Name
|
75b404379b
|
fix(critic-h2-h4): proactive_inspector metric 改名 + probe_success fallback
CD Pipeline / build-and-deploy (push) Failing after 2m7s
H2 — metric semantic 切換污染 baseline:
- cpu_usage_awoooi_api → cpu_usage_node_188
- memory_usage_awoooi_api → memory_usage_node_188
原 metric_name 對應 container working set，新 PromQL 改為 node-level ratio
（cadvisor 停止後的替代）。語意完全不同但保留同名 → 既有 DynamicBaseline
模型用舊單位訓練的 σ 對新值失真，5 分鐘 inspector 週期會狂報假 anomaly。
改名後 baseline 從零學習，初期 sample 數不足會被 _has_enough_samples 守門
跳過告警，安全度過 30 個週期暖機期。
H4 — probe_success 全部不可達假觸發:
- 1 - avg(probe_success)
+ 1 - avg(probe_success or on() vector(1))
原 expr 在 Blackbox 全部 target 失聯時 avg 回空 vector → _fetch_current_value
若把空當 0 → 1-0=1 遠超 0.05 threshold → 5min 一次假告警。
fallback 視為全部成功（值=1，1-1=0），真實 probe down 由獨立的
BlackboxProbeFailure rule 偵測，責任分離。
部署後驗證:
- baseline 表新增 metric_name='memory_usage_node_188' / 'cpu_usage_node_188' 的 row
- 舊 metric_name='memory_usage_awoooi_api' / 'cpu_usage_awoooi_api' 的 row 30 天後可清理
- proactive_inspection_logs 30 個週期內看 _baseline_warmup_skipped 條目而非假 anomaly
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-26 20:40:57 +08:00 |
|
Your Name
|
fd40b79db4
|
feat(p0.6+p1.3+p1.4): 飛輪閉環最後一哩 + ProactiveInspector PromQL 三修
run-migration / migrate (push) Failing after 17s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 47s
CD Pipeline / build-and-deploy (push) Failing after 1m50s
P0.6 ProactiveInspector PromQL labels 修正 (Engineer-B):
- http_error_rate: blackbox_probe_success → probe_success（實測 metric 名稱）
- cpu_usage_awoooi_api: cadvisor up=0（停止）→ 改 node-exporter node_cpu_seconds_total
- memory_usage_awoooi_api: cadvisor 停止 → node-exporter 記憶體使用率比例
P1.3+P1.4 飛輪閉環最後一哩 (Engineer-B2):
- webhooks.py:_try_auto_repair_background 補 PostExecutionVerifier 接線
- feature flag AIOPS_P1_POST_EXECUTION_VERIFIER 守住（default off，可漸進啟用）
- 60s timeout + try/except 三重防護（timeout / 一般 exception / outer exception）
- asyncio.wait_for + EvidenceSnapshot.get_latest_snapshot
- 補 learning_service.record_verification_result 呼叫
- matched_playbook_id 從 result.playbook_id 帶入
- 觸發 EWMA trust_score 演化（飛輪閉環）
- 對稱於人工審核路徑 approval_execution._run_post_execution_verify
ADR 對應: ADR-081 Phase 1 (Verifier) + ADR-083 Phase 3 (Learning)
plan_complete_v3.md L5/L6 階段：⚠️ → ✅（飛輪自主化分數預估 +12 分）
Note: feature flag default off → 不會立即影響 production 行為；
啟用前需 critic 審查 + production E2E 驗證。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-26 20:20:11 +08:00 |
|
Your Name
|
7f4088bcd0
|
fix(aiops-p0): 六大病根 P0 全面修復（ADR-092 B4）
【P0.1】knowledge_extractor_service.py:210 — AttributeError 修復
- Signal.description 欄位不存在（100% 失敗，KM 每天+5 根因）
- 改用 alert_name + annotations.summary 拼接文字
【P0.2+P0.3】Gate 9+11 唯讀指令鬆綁
- blast_radius_calculator: kubectl get/top/describe/logs/version → score=1（非 50）
- operation_parser: 增加 INVESTIGATE 類型識別（唯讀 kubectl 不回 None）
- executor.py: OperationType 新增 INVESTIGATE enum
- approval_execution.py: INVESTIGATE 路徑直接呼叫 execute_kubectl_command
【P0.4】MCP SSH/K8s Provider 修復
- decision_manager: params= → parameters=（符合 MCPToolProvider.execute 簽名）
- decision_manager: MCPToolResult .get() → .success/.output（dataclass 用法）
- decision_manager + ssh_provider: 補入 hosts 120/121（原 default 缺失）
- auto_approve: phase2_agent_debate source bypass confidence 閾值
【P0.5】告警規則語義矛盾修復
- alert_rules.yaml: 8 條 kubectl 查詢規則 RESTART_DEPLOYMENT → NO_ACTION
(CrashLoopBackOff/PostgreSQL 連線/慢查詢/MinIO 磁碟/K3s 節點/告警鏈路/SSL/CoreDNS 等)
- incident_service.py: cAdvisor/CoreDNS 從 general 拆出獨立分類
【P0.6】proactive_inspector 動態基線 PromQL 全修
- 5 個 MONITORED_METRICS PromQL 全部修正（cadvisor label/datname/blackbox）
- db_connection_pool: datname="awoooi" → "awoooi_prod"
- http_error_rate: 無效 http_requests_total → blackbox probe_success
- cpu/memory: namespace label → name=~"k8s_api_awoooi-api.*"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-24 15:32:23 +08:00 |
|
OG T
|
14a02263ae
|
feat(Phase 4): 主動巡檢 + 趨勢預測 + 8D 感官升級 全部完成
CD Pipeline / build-and-deploy (push) Failing after 12m32s
## Phase 4 完整交付（ADR-084）
### 新增服務
- trend_predictor.py: numpy 線性回歸，4h 閾值突破預警，R² 信心評分
- proactive_inspector.py: 每 5 分鐘主動巡檢協調器
- DynamicBaselineService（3σ 偏離）
- LogAnomalyDetector（新 Drain3 pattern）
- TrendPredictor（斜率外推 4h 預測）
- Shadow Mode + 30 分鐘去重 + Holt-Winters 背景重訓
### 8D 感官升級（EvidenceSnapshot Phase 4 增強）
- PreDecisionInvestigator._collect_phase4_anomalies(): 決策前讀取
ProactiveInspector 最近巡檢快取 + LogAnomalyDetector 新 pattern
- EvidenceSnapshot.anomaly_context: 新欄位，Phase 4 動態異常上下文
- DiagnosticianAgent._build_prompt(): prompt 包含 anomaly_context，
LLM RCA 可參考動態基線偏差與趨勢預警
### 資料庫遷移
- incident_evidence: ADD COLUMN anomaly_context JSONB（冪等）
### main.py
- 啟動 run_proactive_inspector_loop() asyncio task
2026-04-15 ogt + Claude Sonnet 4.6（亞太）: Phase 4 全部完成
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-15 15:47:05 +08:00 |
|