94 Commits

Author	SHA1	Message	Date
ogt	5e4887d15c	fix(ops): gate reboot recovery on product freshness [skip ci]	2026-06-25 19:39:42 +08:00
ogt	6f5e22ba69	fix(ops): classify momo source absence in cold-start gate [skip ci]	2026-06-24 23:05:42 +08:00
Your Name	2ec7f6f440	fix(ops): harden heartbeat and momo alert noise	2026-06-24 19:38:33 +08:00
Your Name	95f442adab	fix(ops): harden 188 backup exporter recovery [skip ci]	2026-06-24 06:37:44 +08:00
Your Name	93ac6030cf	fix(ops): 同步 source provider freshness 告警規則	2026-06-18 14:23:13 +08:00
Your Name	ff18872a23	feat(ops): 新增 host runaway process aiops guard	2026-06-18 14:17:03 +08:00
Your Name	b997016991	docs(ops): 鎖定重啟 Plan B 機制檢查 [skip ci]	2026-06-18 11:50:53 +08:00
Your Name	6efd186750	docs(security): 建立高價值配置控管清冊 [skip ci]	2026-06-11 11:29:58 +08:00
Your Name	ae7b39d96a	fix(ops): harden reboot recovery and backup alerts	2026-05-29 12:41:34 +08:00
Your Name	6d2b0ed4cd	ops(runner): add isolation readiness gate [skip ci]	2026-05-24 09:56:47 +08:00
Your Name	4407b46bb6	ops(runner): inventory workflow labels [skip ci]	2026-05-24 09:52:04 +08:00
Your Name	22b45006b7	ops(runner): add pool inventory audit [skip ci]	2026-05-24 09:47:02 +08:00
Your Name	9b465ee140	ci(runner): drain legacy docker act runner safely	2026-05-21 18:53:45 +08:00
Your Name	b3ab4da03b	ci(cd): wait for host web build pressure	2026-05-21 15:51:36 +08:00
Your Name	ae9d0b7385	feat(monitoring): alert on stale source provider ingestion	2026-05-20 19:19:21 +08:00
Your Name	598f33ae8b	fix(monitoring): clarify alert chain smoke evidence	2026-05-20 13:11:44 +08:00
Your Name	21dcfbd991	fix(governance): collapse km slo fallback series	2026-05-14 19:37:15 +08:00
Your Name	d2a4a17969	fix(governance): stabilize adr100 km growth slo	2026-05-14 19:33:52 +08:00
Your Name	4111ea4f9f	fix(ai): remove 188 ollama provider	2026-05-06 14:34:48 +08:00
OG T	c4f40235f4	fix(alertmanager): gate direct telegram to alertchain emergencies	2026-05-06 13:45:33 +08:00
OG T	4753099155	fix(alertmanager): send direct alerts to sre group	2026-05-06 13:38:47 +08:00
Your Name	587551c1f1	fix(ops): monitor full-stack cold-start gates	2026-05-06 00:48:05 +08:00
Your Name	6e96623884	fix(ops): harden momo scheduler cold start gate	2026-05-06 00:15:14 +08:00
Your Name	0315c2b510	docs(ops): codify full stack cold start recovery	2026-05-06 00:07:57 +08:00
Your Name	23932773ef	fix(monitoring): route docker baseline alerts to ssh	2026-05-06 00:00:12 +08:00
Your Name	2f50c67f5c	fix(monitoring): keep host alert ssh diagnostics canonical	2026-05-05 23:57:53 +08:00
Your Name	2221fd3256	fix(ops): persist host resource guardrails	2026-05-05 16:13:19 +08:00
Your Name	1cc9de5722	fix(ops): point runner guardrail alerts to host script	2026-05-05 15:25:37 +08:00
Your Name	d08d1e4951	fix(ops): alert on missing docker resource limits	2026-05-05 15:01:31 +08:00
Your Name	72d66e4ae6	fix(ops): align stale job cleanup thresholds	2026-05-05 14:54:17 +08:00
Your Name	5e625f777d	fix(ops): add stale gitea job cleanup guard	2026-05-05 14:50:47 +08:00
Your Name	7d45f0cb58	fix(ops): alert on stale gitea actions jobs	2026-05-05 14:42:09 +08:00
Your Name	fe618960a8	fix(ops): monitor systemd runners in host baseline	2026-05-05 14:08:43 +08:00
Your Name	e8e6748f70	fix(ops): add docker host resource baseline guardrails	2026-05-05 13:45:09 +08:00
Your Name	ec013f662d	fix(watchdog): 修复 Trust Drift 重复告警 + 建立 GCP Ollama nginx proxy ... - ai_slo_watchdog_job: 改用 trust_drift_detector 纯统计 lib 避免与 governance_agent 每小时自检查重复触发 Telegram - infra/ansible: 建立 110 nginx proxy 转发到 GCP-A/B 端口 11435 -> 34.143.170.20:11434 (GCP-A) 端口 11436 -> 34.21.145.224:11434 (GCP-B) - docs/runbooks: DEPLOY-GCP-OLLAMA-PROXY.md 完整部署指南 - ops/nginx: 手动部署脚本供 110 直接执行 ADR-110 三层容灾启用前提：先部署 proxy，再改 ConfigMap	2026-05-04 23:12:35 +08:00
Your Name	b1ef05fa8c	feat(ollama): ADR-110 GCP 三層容災架構（GCP-A → GCP-B → Local → Gemini） ... ## 變更摘要 - Primary: http://34.143.170.20:11434 (GCP-A SSD, 9x 載速 + 2x 推理) - Secondary: http://34.21.145.224:11434 (GCP-B SSD) - Fallback: http://192.168.0.111:11434 (M1 Pro Local HDD，最後防線) - 廢止 ADR-105「111 唯一鐵律」，新建 ADR-110 ## 核心改動 - config.py: 新增 OLLAMA_SECONDARY_URL；validator 加 GCP IP 白名單（34.143.170.20, 34.21.145.224） - ollama_failover_manager.py: 三層 Ollama 決策矩陣；並行健康檢查三台；health_111 → health_gcp_a - ollama_health_monitor.py: host label 萃取改為通用版（支援 GCP 公網 IP） - failover_alerter.py: 故障/恢復主機動態顯示，不再硬編碼「Ollama 111 (GPU)」 - ollama_auto_recovery.py: notify_recovery 改為 ollama_gcp_a；recovered_host 動態 - k8s/awoooi-prod: configmap + deployment + network-policy 同步更新（egress 加 GCP /32） - 服務層: 10 個服務檔案硬編碼 192.168.0.111 改為讀 settings.OLLAMA_URL - 測試: URL 常數更新，新增三層容災場景，GCP IP 白名單驗證測試 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 22:49:23 +08:00
Your Name	577250a678	fix(governance): 修反消音化 W-3/W-4 守衛 + Prometheus 補資料缺失告警 ... 【統帥怒訓 — 違反 feedback_full_chain_first_then_fix.md 鐵律】 前次 commit f1362fcc 用 skip 條件把告警吞掉，是消音化解法： - W-3：total_exec<10 永遠 skip → Redis 永遠空也不會告警 - W-4：playbooks total==0 永遠 skip → 表被清空也不會告警 - Prometheus NaN sentinel + 既有 < 0.1 規則疊加後沒任何路徑會告警 統帥怒訓「又把告警給消失了」「已經這樣做幾次了」。本 commit 救回告警可見性。 【修法 — 啟動 30 分鐘寬限 + 過期改打資料管線斷新告警】 - ai_slo_watchdog_job.py 新增模組層 _PROCESS_START 與 _grace_active() 守衛： - W-3a：metric 有資料 + rate<0.30 → 既有「飛輪成功率過低」 - W-3b：rate=None 且 uptime>30min → 新告警「飛輪資料管線無流量」 - W-4a：playbooks total>0 + approved=0 → 既有「自動修復鏈路斷裂」 - W-4b：playbooks total=0 且 uptime>30min → 新告警「Playbook 表初始化失敗」 - 3 份 Prometheus rule（k8s/monitoring/flywheel-alerts.yaml、 ops/monitoring/alerts.yml、ops/monitoring/alerts-unified.yml）新增 FlywheelExecutionRateMissing：absent() 或 NaN 持續 30 分鐘 → 告警， 與 watchdog W-3b 雙保險 【已加入 memory】 feedback_silencing_alerts_recurring_violation.md 鎖入紅線鐵律： 「fresh deploy / init guard 用 skip 吞告警 = 結構性失職，必須分流寬限期 + 過期改打資料管線斷新告警」 【驗證】 106 個治理相關 unit test 全過： test_trust_drift_watchdog / test_governance_agent / test_failover_alerter / test_check_trust_drift_commit_outside_context_poc / test_governance_remediation_dispatch / test_ai_governance_endpoints / test_governance_dispatcher 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 12:39:46 +08:00
Your Name	b371edb70c	fix host alert auto-repair routing and backup false positives	2026-05-02 23:44:12 +08:00
Your Name	da772a1605	fix(decision): block kubectl actions on bare_metal host alerts ... When HostHighCpuLoad / HostOutOfMemory fire on a bare-metal host (192.168.0.110 et al, where Sentry / ClickHouse / Snuba are eating CPU), the LLM kept proposing "kubectl rollout restart awoooi-api", which is a wrong-domain action — restarting awoooi cannot fix a third-party process's CPU usage on the host. Auto-execute would then either run the no-op kubectl restart (wasted) or escalate after ssh_diagnose because no safe action was found, producing the "AI 自動修復失敗" Telegram noise the user just complained about. Adds a guard at the top of DecisionManager._auto_execute: if the incident's primary signal carries host_type=bare_metal AND the proposed action starts with "kubectl", refuse to execute. The incident is marked READY with a clear blocked_reason so human operators see why automation declined, and emergency_escalation records the event in AOL for audit. Also patches /home/wooo/monitoring/alerts.yml on 110 (and the new ops/monitoring/alerts.yml in repo) to add an explicit auto_repair_action annotation on HostHighCpuLoad / HostOutOfMemory that hints LLM toward `ssh ... ps aux` rather than kubectl restart. Prometheus reload returned 200. Tests: tests/test_decision_manager_bare_metal_kubectl_guard.py covers (1) bare_metal+kubectl blocked, (2) kubectl get also blocked, (3) bare_metal+ssh NOT blocked, (4) k8s host_type+kubectl NOT blocked, (5) missing host_type label NOT blocked. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 17:41:28 +08:00
Your Name	3156ff1c69	feat(aiops): add ssh_docker_prune to auto-repair flywheel for disk-full alerts ... Adds Group B SSH MCP tool ssh_docker_prune (image+volume+builder prune with ≥75% disk usage gate) and routes "docker prune" actions through it. Flips HostDiskUsageHigh from auto_repair=false to true with mcp_provider routing labels so the flywheel can self-heal next disk-full event without hitting the emergency_channel Telegram path. Trigger: 2026-05-01 → 05-02 Telegram alert storm (peak 53/hr) caused by empty ssh-mcp-key/known_hosts secret rejecting all SSH and forcing every disk-full alert through "Host key is not trusted → escalate" loop. known_hosts patched live; this commit closes the playbook gap so the next occurrence resolves without manual intervention. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 12:31:37 +08:00
Your Name	3650fc727a	docs(ci): record runner user service takeover state	2026-05-01 16:30:54 +08:00
Your Name	e7991b8e6c	fix(ci): keep runner installer idempotent without restart	2026-05-01 16:27:37 +08:00
Your Name	bc295eaec2	fix(ci): allow user service for gitea host runner	2026-05-01 16:24:45 +08:00
Your Name	cb5ab900c4	fix(ci): preserve gitea runner jobs on shutdown	2026-05-01 16:16:27 +08:00
Your Name	ca22ec2fd2	fix(aiops): route backup failures rule-first	2026-05-01 10:11:10 +08:00
Your Name	f0d14ab6c4	fix(aiops): escalate blocked auto repair	2026-04-30 23:49:17 +08:00
Your Name	e27b462bef	fix(ops): keep disabled gitea runner stopped	2026-04-30 10:59:46 +08:00
Your Name	0f7e9d3467	fix(cd): run docker builds on host runner	2026-04-30 10:43:33 +08:00
Your Name	7cc10b2599	fix(cd): serialize gitea docker builds	2026-04-30 10:11:50 +08:00
Your Name	c5753e1c57	fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化 ... critic PR review 揭示已 push commits 的 7 個 blocker，本 commit 全部修復。 ## C1 + C2 + M1 + M2 + M3 — KMWriter 真正統一契約（critic 最嚴重 5 條） ### C1 km_writer.py:194 — backfill 自打臉修 - 裸 asyncio.create_task(_backfill_path_a_approval) → await _backfill_path_a_approval_safe() - 同步 await + 獨立 DLQ km:backfill:dlq + try/except 不阻塞主寫入 - 新增 km_backfill_reconciler_job.py（每 5 分鐘掃 DLQ）+ ENABLE_KM_BACKFILL_RECONCILER flag - 防 Path B 比 Path A 先完成 → related_approval_id 永遠 NULL 的 race ### C2 km_writer.py:391 — KM_WRITE_AWAIT=false 路徑收緊 - 從 ensure_future（fire-and-forget 比舊版同步寫更糟） - 改 await writer.write(retry=1, timeout=2.0)（仍 await 但只試一次、超時短） - docstring 明確標註「緊急回滾用，不保證可靠性」 ### M1 decision_manager.py:2178/2203 — 移除 _fire_and_forget 旁路 - 兩處 _fire_and_forget(executor.write_execution_result_to_km(...)) - 改 await asyncio.shield(...) + BaseException 保護（防上層 cancel 中斷） - KM_WRITE_AWAIT=true 在這條路徑終於真正 await ### M2 incident_service.py:1099 — 自製 path 加 retry+DLQ - 原本 if settings.KM_WRITE_AWAIT: await asyncio.wait_for else create_task - 改 3 次指數退避 retry + DLQ 保護（呼叫 km_writer 私有 helper） ### M3 km_writer.py:166 — 冪等聲明對齊實作 - knowledge_repository.create() 加 UPSERT 路徑（pg_insert ON CONFLICT DO UPDATE） - KnowledgeEntryCreate / KnowledgeEntryRecord 加 path_type 欄位 - migration: ADD COLUMN path_type + partial unique index uix_knowledge_incident_path ## M4 alertmanager.yml — equal: [] 收緊（critic 防爆炸抑制） - OllamaInstanceDown / KMConverterDown 抑制加 equal: ['cluster'] 約束 - 防多 cluster 場景下任一 Ollama down 誤抑全 AI/SLO 告警 ## M5 Alertmanager 版本驗證（已確認 v0.31.1，遠超 v0.22+） ## M6 governance_agent.py — health score 區分 skipped vs ok vs violated - check_slo_compliance 加 _meta {violated_count, skipped_count, ok_count, all_skipped, status} - run_self_check: SLO 全 skipped 時獨立發 governance_slo_data_gap 告警 （不污染 self_failure 計數，因為 no_data 是 emitter 未實作不是治理機制故障） ## M7 scripts/check_config_drift.py — 改 AST 解析 - regex 改 ast.parse 找 Settings ClassDef AnnAssign Field(default=...) - 避免多行 list / default_factory= / 含跳行字串的 false negative - 4 欄位（AI_FALLBACK_ORDER / ARGOCD_URL / PROMETHEUS_URL / OLLAMA_URL）全對齊 ## 新增測試 - test_km_writer_backfill_reconciler.py: 7 cases（C1 reconciler + safe helper） - test_km_writer_idempotent.py: 5 cases（M3 path_type 注入 + UPSERT 分支） ## 驗證 - 1585 unit tests 全綠（+13 從 1572） - amtool check-config SUCCESS（8 inhibit_rules / 2 receivers） - drift checker AST-based 4 欄位全對齊 - Alertmanager v0.31.1 確認支援新語法 ## 期望影響 - KMWriter 名實統一：飛輪閉環 KM 寫入路徑 100% 可靠 - M4 抑制爆炸風險解除 - 治理層不再對 SLO no_data 靜默 - drift checker false negative 風險解除 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-29 10:44:39 +08:00