wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 4 Packages Projects Releases Wiki Activity
1,867 Commits 179 Branches 2 Tags
bc295eaec24c07e613a005cb73c966db7ec2aa21
Commit Graph

52 Commits

Author SHA1 Message Date
Your Name
bc295eaec2 fix(ci): allow user service for gitea host runner
Some checks failed
Code Review / ai-code-review (push) Has been cancelled
Details
2026-05-01 16:24:45 +08:00
Your Name
cb5ab900c4 fix(ci): preserve gitea runner jobs on shutdown
All checks were successful
Code Review / ai-code-review (push) Successful in 46s
Details
2026-05-01 16:16:27 +08:00
Your Name
ca22ec2fd2 fix(aiops): route backup failures rule-first
All checks were successful
CD Pipeline / tests (push) Successful in 1m51s
Details
Code Review / ai-code-review (push) Successful in 30s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 42s
Details
CD Pipeline / build-and-deploy (push) Successful in 8m21s
Details
CD Pipeline / post-deploy-checks (push) Successful in 4m18s
Details
2026-05-01 10:11:10 +08:00
Your Name
f0d14ab6c4 fix(aiops): escalate blocked auto repair
Some checks failed
CD Pipeline / tests (push) Successful in 1m33s
Details
Code Review / ai-code-review (push) Successful in 28s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 40s
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
2026-04-30 23:49:17 +08:00
Your Name
e27b462bef fix(ops): keep disabled gitea runner stopped
All checks were successful
Code Review / ai-code-review (push) Successful in 27s
Details
2026-04-30 10:59:46 +08:00
Your Name
0f7e9d3467 fix(cd): run docker builds on host runner
All checks were successful
CD Pipeline / tests (push) Successful in 1m33s
Details
Code Review / ai-code-review (push) Successful in 25s
Details
CD Pipeline / build-and-deploy (push) Successful in 9m20s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m33s
Details
2026-04-30 10:43:33 +08:00
Your Name
7cc10b2599 fix(cd): serialize gitea docker builds
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 40s
Details
Code Review / ai-code-review (push) Successful in 24s
Details
2026-04-30 10:11:50 +08:00
Your Name
c5753e1c57 fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化 
critic PR review 揭示已 push commits 的 7 個 blocker，本 commit 全部修復。

## C1 + C2 + M1 + M2 + M3 — KMWriter 真正統一契約（critic 最嚴重 5 條）

### C1 km_writer.py:194 — backfill 自打臉修
- 裸 asyncio.create_task(_backfill_path_a_approval) → await _backfill_path_a_approval_safe()
- 同步 await + 獨立 DLQ km:backfill:dlq + try/except 不阻塞主寫入
- 新增 km_backfill_reconciler_job.py（每 5 分鐘掃 DLQ）+ ENABLE_KM_BACKFILL_RECONCILER flag
- 防 Path B 比 Path A 先完成 → related_approval_id 永遠 NULL 的 race

### C2 km_writer.py:391 — KM_WRITE_AWAIT=false 路徑收緊
- 從 ensure_future（fire-and-forget 比舊版同步寫更糟）
- 改 await writer.write(retry=1, timeout=2.0)（仍 await 但只試一次、超時短）
- docstring 明確標註「緊急回滾用，不保證可靠性」

### M1 decision_manager.py:2178/2203 — 移除 _fire_and_forget 旁路
- 兩處 _fire_and_forget(executor.write_execution_result_to_km(...))
- 改 await asyncio.shield(...) + BaseException 保護（防上層 cancel 中斷）
- KM_WRITE_AWAIT=true 在這條路徑終於真正 await

### M2 incident_service.py:1099 — 自製 path 加 retry+DLQ
- 原本 if settings.KM_WRITE_AWAIT: await asyncio.wait_for else create_task
- 改 3 次指數退避 retry + DLQ 保護（呼叫 km_writer 私有 helper）

### M3 km_writer.py:166 — 冪等聲明對齊實作
- knowledge_repository.create() 加 UPSERT 路徑（pg_insert ON CONFLICT DO UPDATE）
- KnowledgeEntryCreate / KnowledgeEntryRecord 加 path_type 欄位
- migration: ADD COLUMN path_type + partial unique index uix_knowledge_incident_path

## M4 alertmanager.yml — equal: [] 收緊（critic 防爆炸抑制）
- OllamaInstanceDown / KMConverterDown 抑制加 equal: ['cluster'] 約束
- 防多 cluster 場景下任一 Ollama down 誤抑全 AI/SLO 告警

## M5 Alertmanager 版本驗證（已確認 v0.31.1，遠超 v0.22+）

## M6 governance_agent.py — health score 區分 skipped vs ok vs violated
- check_slo_compliance 加 _meta {violated_count, skipped_count, ok_count, all_skipped, status}
- run_self_check: SLO 全 skipped 時獨立發 governance_slo_data_gap 告警
  （不污染 self_failure 計數，因為 no_data 是 emitter 未實作不是治理機制故障）

## M7 scripts/check_config_drift.py — 改 AST 解析
- regex 改 ast.parse 找 Settings ClassDef AnnAssign Field(default=...)
- 避免多行 list / default_factory= / 含跳行字串的 false negative
- 4 欄位（AI_FALLBACK_ORDER / ARGOCD_URL / PROMETHEUS_URL / OLLAMA_URL）全對齊

## 新增測試
- test_km_writer_backfill_reconciler.py: 7 cases（C1 reconciler + safe helper）
- test_km_writer_idempotent.py: 5 cases（M3 path_type 注入 + UPSERT 分支）

## 驗證
- 1585 unit tests 全綠（+13 從 1572）
- amtool check-config SUCCESS（8 inhibit_rules / 2 receivers）
- drift checker AST-based 4 欄位全對齊
- Alertmanager v0.31.1 確認支援新語法

## 期望影響
- KMWriter 名實統一：飛輪閉環 KM 寫入路徑 100% 可靠
- M4 抑制爆炸風險解除
- 治理層不再對 SLO no_data 靜默
- drift checker false negative 風險解除

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-29 10:44:39 +08:00
Your Name
715dc3cb91 fix(observability): P0 假警報止血 + ConfigMap drift 對齊 + 治理工具 
12-Agent 全景診斷觸發的 P0/P1 觀測層修復。

## P0 假警報止血（4 SLO 雪崩根因）
- governance_agent.py:306 — 空 result 不再 fallback 0.0，改 continue + log warning
  根因：Prometheus 查無資料（emitter 未實作 / rule 未部署）被誤判為 SLO=0
  必觸發 violated=True 噴 4 條假告警

## P0 鬼魂按鈕守門
- telegram_gateway.py:1654 — LLM 動態按鈕 Redis 失敗時 btn_list.clear()
  first_row（批准/拒絕，HMAC nonce 無狀態）由 caller 1488 永遠保留
  feedback_no_ghost_buttons.md 三缺一鐵律對齊

## ConfigMap drift 修復（3 處）
- config.py:683 PROMETHEUS_URL: 188→110（drift checker 揪出 = SPF-4 部分根因）
- config.py:705 ARGOCD_URL: 125→121（T0 G3 已知）
- config.py:375 AI_FALLBACK_ORDER: 補 nvidia 對齊 ConfigMap

## P1 Alertmanager 升級（amtool SUCCESS）
- ops/alertmanager/alertmanager.yml: deprecated → v0.27+ 新語法
  - match/match_re → matchers
  - source_match/target_match → source_matchers/target_matchers
  - group_by 加 team label（防 SLO 雪崩 4 條同秒推）
  - PostgreSQL/Redis inhibit 補 equal: ['instance']（防爆炸抑制）
- 新增 3 組因果抑制：
  - OllamaInstanceDown → SLO_*/AI_*（30 分鐘）
  - KMConverterDown → SLO_KMGrowthRate*
  - SLO_*_FastBurn → SLO_*_(Medium|Slow)Burn

## 治理工具落地
- scripts/check_config_drift.py: ConfigMap vs code default drift 檢測
  揪出 PROMETHEUS_URL drift 是 SPF-4 根因（governance_agent 連 188 而非 110）
- scripts/health_check_session.sh: 11 服務 + 4 SSH + drift + git 全景驗證

## 驗證
- 1552 unit tests 全綠
- amtool check-config SUCCESS（8 inhibit_rules / 2 receivers）
- drift checker 4 欄位全對齊
- health check 11 服務全可達

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-29 10:44:39 +08:00
Your Name
ed205489c1 feat(p3.2-tests+ci-schema): model_version 測試 + CI test_schema 對齊 + Grafana SLO Dashboard
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m20s
Details 
P3.2 配套測試 + CI 環境同步 + ADR-100 Grafana 視覺化：

CI test_schema 補齊（解 1162-1172 阻塞之延伸）:
- setup_test_schema.sql 加 ai_provider_version_history 表
- 對齊 production p3_2_provider_version_history.sql（已 K8s exec 上線）

新增測試 (636 行):
- test_model_version_probe.py (387) — Provider 探測單元測試
- test_model_version_tracker.py (249) — Tracker 整合測試
  · 4 個 DB-dependent tests 標 @pytest.mark.integration
  · 15 unit + 4 integration（unit step 跳過 integration class）

新增配套:
- ai-slo-dashboard.json (496 行) — Grafana 儀表板
  · 對應 ADR-100 SLO 規則的 4 大面板：
    自主修復成功率 / 飛輪閉環延遲 / 治理事件 / Provider 健康度

修改:
- governance_agent.py +122 行 — SLO 指標暴露 + retrieve metric 整合

Tests: 15 passed (probe + tracker unit), 4 deselected (integration class)

Production 部署狀態:
- p2_decision_fusion_columns.sql ✅ K8s exec 完成（commit c58bdd0c）
- p3_2_provider_version_history.sql ✅ K8s exec 完成（this commit）
- 兩個 production migration 都已上線，CI test_schema 同步補齊

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 14:57:16 +08:00
Your Name
025a493f06 feat(p3.2+adr-100): Model Version Tracker + SLO 自治 + KB rot cleaner
Some checks failed
run-migration / migrate (push) Failing after 12s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
Wave 8 P3.2 模型版本追蹤 + ADR-100 SLO 自我治理 + 配套：

P3.2 — Model Version Tracking:
- model_version_probe.py (268 行) — 探測 Ollama / OpenRouter 等 provider 的 model version
- model_version_tracker.py (101 行) — 對齊 PG provider_version_history 表
- migrations/p3_2_provider_version_history.sql + rollback — 25 行 schema
- db/models.py +32 行 — ProviderVersionHistory ORM

ADR-100 — AI 自主化 SLO:
- docs/adr/ADR-100-ai-autonomous-slo.md (167 行) — 飛輪 SLO 設計與閾值
- ops/monitoring/slo-rules.yml (254 行) — Prometheus SLO recording rules + alerts
- ops/monitoring/tests/test_slo_rules.yaml (242 行) — promtool unit tests

整合修改:
- main.py +72 行 — Lifespan 啟動 model_version_probe + KB rot cleaner schedule
- gitea_webhook.py +45 行 — webhook 接收 model 版本變化通知
- ci_auto_repair.py / evidence_snapshot.py / pre_decision_investigator.py — 配合接線

新測試:
- test_kb_rot_cleaner_schedule.py (120 行) — 9 tests pass
- test_slo_rules.yaml — promtool 驗收

Tests: 9 passed (test_kb_rot_cleaner_schedule)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Multiple Engineers (P3.2 + ADR-100) <noreply@anthropic.com>
 2026-04-27 14:54:19 +08:00
Your Name
fb130c9a28 feat(p3.1-t2): DiagnosisAggregator stub tests + sanitization 補強 + metrics 補欄
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m16s
Details 
Wave 8 P3.1-T2 後續補測 + 配套：

新增測試:
- test_diagnosis_aggregator_stub.py (238 行) — 15 tests
  · stub fixture 驗證 _collect_diagnosis_aggregator 接線
  · feature flag default off 不呼叫
  · timeout 邊界 / exception fail-soft

修改:
- core/metrics.py +23 — 新增 DiagnosisAggregator 相關 Prometheus 指標
- sanitization_service.py +24 — 補強 prompt sanitize 邊界（vuln #4 配套）
- RUNBOOK-AGENT-STEP-LATENCY.md / agent_step_latency_rules.yaml — 微調

Tests: 15 passed

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 08:30:26 +08:00
Your Name
fefe4c21cd fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
延續 595629c0 INC-20260425 修復，補三段 Agent + 全鏈路觀測：

A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0（env override）
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0（env override）
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
  · success/timeout/error outcome label
  · histogram 寫入 aiops_agent_step_duration_seconds

A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain（NEMO→GEMINI→CLAUDE）
- 完全避開 Ollama CPU 238s 二次 timeout

新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality

新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序

新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
  · 三 Agent histogram alert rules（p99 > timeout 80% → warning）

驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)

INC-20260425 雙修總工作量（595629c0 + 此 commit）:
  · 5 個 service/agent 檔修改
  · 1 個新 observability 模組
  · 4 個新測試/配套檔
  · 1372+187 = 1559 行新增

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
 2026-04-27 08:15:53 +08:00
Your Name
1ab6786ce3 feat(ops): Ollama 容災 Runbook + Grafana 儀表板 + Consensus K8s ConfigMap patch
Some checks failed
run-migration / migrate (push) Failing after 13s
Details
CD Pipeline / build-and-deploy (push) Failing after 2m1s
Details 
Wave 6 P2.3 ops 配套 + tool-expert 部署文件：

新增:
- docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md (240 行)
  · 三大鐵律驗證步驟（自動切 Gemini / 自動切回 / quota 熔斷）
  · failover/recovery 完整 SOP
  · 故障排查清單（Ollama 111/188 不通、Gemini quota 超發等）
- ops/monitoring/grafana/dashboards/ollama_failover.json (295 行)
  · 4 panel：current primary / failover events / quota usage / health status
  · 對應 P2.3 metrics: OLLAMA_FAILOVER_TRIGGERED_TOTAL / GEMINI_DAILY_CALL_COUNT
- k8s/awoooi-prod/04-configmap.yaml.patch-consensus
  · ENABLE_12AGENT_CONSENSUS / ENABLE_AIOPS_P2_FUSION feature flag patch

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tool-expert agent (Wave 6) <noreply@anthropic.com>
 2026-04-27 08:11:40 +08:00
Your Name
2c57b71db9 feat(wave5-p2): GovernanceAgent 4 項自檢 + Ollama 健康告警規則 + Prometheus metrics 整合
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 10m45s
Details 
MASTER plan_complete_v3.md Wave 5 P2.2 + P2.3 完成（multiple engineers 在限額前完成代碼，補 commit）：

P2.2 — GovernanceAgent 4 項自檢:
- governance_agent.py (342 行) — 每 1 小時自檢循環:
  · trust_drift（信任度漂移檢測）
  · knowledge_degradation（知識退化檢測）
  · llm_hallucination（LLM 幻覺檢測）
  · execution_blast_radius（執行爆炸半徑檢測）
- main.py lifespan: asyncio.create_task(run_governance_loop()) 啟動
  try/except 包裹，schedule 失敗不阻斷主流程
- failover_alerter.py: alert_governance(event_type, payload) 1h dedup
  四類事件 → Telegram MarkdownV2 告警

P2.3 — Ollama 健康規則 + Prometheus Metrics:
- ops/monitoring/ollama_health_rules.yaml (148 行):
  · OllamaHealthDegraded / OllamaPrimaryDown
  · OllamaFailoverTriggered / GeminiQuotaExceeded
  · 補 Prometheus 取資料的 alert rules
- core/metrics.py (57 行):
  · GEMINI_DAILY_CALL_COUNT / GEMINI_DAILY_QUOTA Gauge
  · OLLAMA_FAILOVER_TRIGGERED_TOTAL Counter
  · OLLAMA_CURRENT_PRIMARY_IS_OLLAMA Gauge
- ollama_failover_manager.py:
  · _check_gemini_quota: 每次 check 同步更新 Gauge（讓 Prometheus 取最新值）
  · select_provider: failover 時 inc Counter + 切 Primary Gauge
  · try/except 包裹，metric 失敗不阻斷主路由

E2E 測試:
- test_failover_e2e_dispatch.py (365 行)
  完整 dispatch 路徑：health check → failover decide → alerter → metrics

Tests: 54 passed (e2e_dispatch + failover_manager + failover_alerter)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Multiple Engineers (上 session Wave 5) <noreply@anthropic.com>
 2026-04-26 20:56:19 +08:00
Your Name
7cd53c0228 fix(monitoring): 記憶體告警改用 working_set，停止 page cache 假告警 
- alerts-unified.yml:
  - SentryClickHouseMemoryPressure: usage_bytes → working_set_bytes，0.8 → 0.85
  - GiteaMemoryPressure: 同步修正（同樣 page cache 虛高根因）
- ops/monitoring/tests/clickhouse_memory_test.yml: promtool 4 cases
- 04-awoooi-devops-commander.md v2.8: Prometheus 指標選擇規範 + Gitea HMAC Webhook 規範
- LOGBOOK: 記錄 T0 五大並行任務（A 按鈕 / B ClickHouse / C Gitea webhook / D ElephantAlpha / F Code review）

鐵證: 2026-04-23 23:13 sentry-clickhouse usage_bytes=88.5% vs working_set=7.8%
根因: container_memory_usage_bytes 含 OS page cache，OOM killer 不視為壓力
修法: 改用 K8s/cadvisor 認可的 working_set_bytes (RSS + active cache)，閾值 0.85

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 20:16:12 +08:00
OG T
ba18ad2ef8 feat(hermes+rules): LLM 升級 Hermes + 統帥決策 deprecate PostgreSQLDiskGrowthRate
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 40s
Details
CD Pipeline / build-and-deploy (push) Successful in 8m37s
Details 
統帥 2026-04-19 決策:
  - Rule 1 PostgreSQLDiskGrowthRate → 選項 C: deprecate + 替代新規則
  - Rule 2 NoAlertsReceived2Hours → 保留 (真實告警鏈路守護)
  - noise_rate 算法先修正 (NO_ACTION 不算 fp),觀察後動態調整

1. rule_stats_updater v2 noise 算法:
  原: 任何 EXPIRED approval 都算 fp
  問題: NO_ACTION/OBSERVE/INVESTIGATE 是 AI 純觀察,不該算假報
  修: WHERE ar.action NOT ILIKE '%NO_ACTION%' AND NOT ILIKE '%OBSERVE%' AND ...

2. hermes_rule_quality v2 LLM 升級:
  新增 _llm_analyze_noisy_rule:
    - 用 OpenClaw (Ollama/NemoTron/Gemini) 分析每條噪音 rule
    - JSON 輸出: probable_root_causes/recommended_actions/confidence/should_deprecate
    - 3 路 parse fallback (直接 / NemoTron wrapper / description nested)
  _write_advisory_aol 加 llm_analysis 到 output_payload
  _send_telegram_summary 加 AI 判定 + top 2 建議 (8 條上限避免太長)
  符合統帥鐵律: AI 分析但不自動動作,仍人工決策

3. ops/monitoring/alerts-unified.yml 替換 Rule 1:
  刪 PostgreSQLDiskGrowthRate (500MB/h 增長 → 觸發 WAL 正常行為誤報)
  加 HostDiskUsageHigh (>80% for 10m, warning)
  加 HostDiskUsageCritical (>90% for 5m, critical)
  兩者 labels.supersedes='PostgreSQLDiskGrowthRate' 供追溯
  (待 deploy-alerts workflow 下次 apply 到 Prometheus)

4. DB 即時 mark deprecated (避免等 alerts yaml 部署前 Hermes 又推):
  UPDATE alert_rule_catalog SET review_status='deprecated' WHERE rule_name='PostgreSQLDiskGrowthRate'

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 19:39:05 +08:00
OG T
eab3f527cd feat(monitoring): Phase 7 盲區治理 — L2 配額 + 自監控告警 (ADR-090)
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 1m21s
Details
CD Pipeline / build-and-deploy (push) Failing after 9m24s
Details 
戰場：110 load=17 持續 13 天 + 188 cadvisor 321% CPU 重啟無效
統帥鐵律：不要只降低，要長期解決 → 結構性治理而非補丁

本 commit 涵蓋：
1. k8s/monitoring/docker-compose-110.yml
   - cadvisor 加 mem_limit 512M + cpus 1.0（L2 防爆網）
   - 備註 110 live 與本檔 drift（下一 session 納入 CD）

2. ops/monitoring/alerts-unified.yml 新增 infra_self_monitoring 群組：
   - CadvisorDown / MemoryPressure / CPUThrottled
   - NodeExporterDown / CPUThrottled
   - SentryClickHouseMemoryPressure / CPUThrottled
   - GiteaMemoryPressure / CPUThrottled
   - PrometheusDown（監控自監控元層）
   → 全部用 (memory usage / spec_memory_limit) 動態判斷，
     不寫死 80% 或 MB 數，配額改閾值自動跟著變

其他配套（非本 repo，已 SSH patch 到 110/188）：
- /home/ollama/wooo-aiops/docker-compose.yml：188 cadvisor 加 --disable_metrics / --docker_only / --housekeeping_interval + 1g/1.5c
- /home/wooo/monitoring/docker-compose.yml：110 cadvisor + node-exporter 納管 + 降維 flags + 配額
- /opt/sentry/docker-compose.override.yml：Sentry L2 配額（clickhouse 8g/4c, kafka 3g/2c 等）
- /home/wooo/gitea/docker-compose.yml：Gitea 3g/3c
- /home/wooo/act-runner/docker-compose.yml：Actions Runner 2g/2c

對映：
- feedback_monitor_self_monitoring.md 🔴🔴🔴 監控工具必須被監控
- feedback_ai_autonomous_direction.md 動態閾值 ≠ 寫死規則
- ADR-090 Layer 2 資源配額強制

驗收（48h）：
- 188 cadvisor CPU 從 321% → <50%（配額強制）
- 110 load5 從 18 → <10（Sentry/Gitea 釋壓後）
- 自監控告警無誤報

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 01:50:41 +08:00
OG T
5e4dbbbb41 fix(alertmanager): webhook URL 改指向 VIP 192.168.0.125:32334 
根因: Alertmanager 打 120:32334 → Connection Refused
      120/121 NodePort 直接訪問不通，只有 VIP 125:32334 可通
影響: 告警完全無法送達 AWOOOI API，鏈路靜默失效 (自 2026-04-12 起)
修復: url → http://192.168.0.125:32334/api/v1/webhooks/alertmanager
驗證: 手動 inject 測試告警，API 端收到並觸發完整 LLM 分析流程

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-16 03:19:58 +08:00
OG T
946fe1fa7c fix(monitoring): 合併重複飛輪告警 group + 補 notification_type: TYPE-8M
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 44s
Details 
awoooi_flywheel_health (重複) 合入 awoooi_flywheel_meta_alerts:
- 所有 5 條規則加 notification_type: TYPE-8M
- 新增 FlywheelAlertnameNullHigh（原僅在舊 group）
- 刪除重複 group，消除 Prometheus 同名告警衝突

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 22:43:02 +08:00
OG T
bd75aca727 feat(adr-075): 補全 2 個欠缺的 Prometheus 告警規則
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 49s
Details 
- MomoScraperSuccessLow: 業務爬蟲成功率 <90% (business group)
- CoreDNSResolutionFailed: CoreDNS SERVFAIL 率 >5% (kubernetes group)

ADR-075 Phase 3 完成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 21:59:18 +08:00
OG T
edb97fd29b fix(monitoring): 補回 4 個僅存於主機的 Prometheus 規則群組
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 41s
Details 
deploy-alerts.sh 部署時覆寫了這 4 個從未進 repo 的群組：
- awoooi_flywheel_health (5條：Playbook/Success/Vectorization/NullRate/Stuck)
- awoooi_backup_restore (2條：RestoreTestFailed/TestStale)
- awoooi_infrastructure_detailed (3條：Container/RedisStream/DiskGrowth)
- awoooi_host_connectivity (1條：NetworkPartition)

從 /home/wooo/monitoring/alerts.yml.bak_20260412_183835 還原。
offset PromQL 已修正為各個 selector 上，而非整個表達式。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:14:39 +08:00
OG T
f52dc459e6 feat(adr075): Step4 新增4組Prometheus規則 secops/business/flywheel_meta
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 41s
Details 
新增規則群組:
- awoooi_secops_alerts: UnauthorizedSSHLogin (5min>10次失敗)
- awoooi_business_alerts: AITokenCostSpike + GeminiAPIErrorRateHigh
- awoooi_flywheel_meta_alerts:
    FlywheelPlaybookZero / FlywheelExecutionSuccessLow
    FlywheelKMVectorizationLow / FlywheelIncidentsStuck

飛輪 meta 規則依賴 ADR-074 Exporter 指標
secops/business 規則依賴 node_exporter/awoooi custom metrics

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:51:23 +08:00
OG T
43edff184d feat(dr): Sprint C — Host rsync 備份 + DR SOP 文件 
C-1 Velero: 已確認運作中（daily-awoooi-prod schedule, 13d, MinIO Available）

C-2 Host rsync 備份:
  scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
    - Harbor registry data（最高優先）
    - Gitea repos
    - bitan-pharmacy.git（若存在）
    - 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
    - 失敗時 Telegram 告警
  ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則

C-3 DR SOP 文件:
  docs/runbooks/disaster-recovery/DR-K8s-awoooi.md  (<15分鐘)
  docs/runbooks/disaster-recovery/DR-Nginx.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Harbor.md       (<30分鐘)
  docs/runbooks/disaster-recovery/DR-Bitan.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Stock.md        (<5分鐘)

部署備份腳本說明 (需手動執行):
  scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
  ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
  ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:04:18 +08:00
OG T
6351e9a0e9 feat(mcp-phase2): MCP Phase 2 — Prometheus MCP + SSH MCP + alert labels
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m37s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 35s
Details 
MCP-2b: prometheus_provider.py
  - prometheus_query (PromQL 即時查詢)
  - prometheus_query_range (歷史趨勢，預設 15 分鐘)
  - prometheus_get_alert_history (告警觸發歷史)
  - config: PROMETHEUS_URL + PROMETHEUS_MCP_ENABLED

MCP-2a: ssh_provider.py
  - 群組A 9 個只讀診斷工具 (top/disk/memory/logs/status/port/nginx/swap)
  - 群組B 6 個安全操作工具 (restart/compose/systemctl/clear-log/ssl/nginx-reload)
  - 四層安全守衛 (白名單/allowed_hosts/forbidden_patterns/trust_score)
  - config: SSH_MCP_ENABLED + SSH_MCP_ALLOWED_HOSTS

K8s: 04-ssh-mcp-secret.example.yaml (ssh-mcp-key Secret 範本 + 建立步驟)

Alert labels: alerts-unified.yml 補充 mcp_provider/host_type/alert_category
  覆蓋: HostHighCpuLoad/HostOutOfMemory/HostOutOfDiskSpace/DockerContainer*
        SignOzDown/SentryDown/HarborDown/GiteaDown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:35:35 +08:00
OG T
e1dfbedf0e fix(alerts): HostHighCpuLoad auto_repair: false → true
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
飛輪一直 GUARDRAIL_BLOCKED 的根本原因：
Prometheus rule 標籤 auto_repair=false 強制 HITL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 13:33:23 +08:00
OG T
ab3e266a23 fix(monitoring): Phase O-6.2 service-registry 補齊 9 個缺失 K8s 部署 
新增:
- argocd 5個元件 (applicationset/dex/notifications/redis/repo-server)
- awoooi-dev/awoooi-api
- kube-state-metrics
- observability/event-exporter
- velero/velero

結果: prometheus 覆蓋率 94%→96%, errors 9→0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 10:44:36 +08:00
OG T
07a097c259 fix(infra): Sprint 3 自動修復全鏈路修復 — docker-188 SSH egress + service registry 擴充
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
NetworkPolicy: 新增 192.168.0.188:22 egress — repair-bot-188.sh 執行路徑
service-registry.yaml: 新增 signoz/bitan-app (AUTO, 188主機)

修復覆蓋: Bug #11 補完 (188 SSH) + 188 服務分級覆蓋
E2E 驗證: MoWoooWorkDown → SSH → REPAIR_OK:momo-app (3791ms)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 18:04:19 +08:00
OG T
1fb0c0ca90 fix(auto-repair): Bug #5+#6 — SSH binary + affected_services 匹配修正
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
Bug #5 (webhooks.py): target_resource 現在優先用 component label
  - SentryDown alert 有 labels.component="sentry"
  - 舊邏輯: labels.instance="192.168.0.110:9000" → Playbook affected_services 不匹配
  - 新邏輯: component → pod → instance → alertname

Bug #6 (Dockerfile): python:3.11-slim 無 openssh-client
  - SSH_COMMAND Playbook 執行路徑調用 asyncio.create_subprocess_exec("ssh", ...)
  - image 沒有 ssh binary → 所有 SSH 修復必然失敗
  - 修正: 在 production stage 安裝 openssh-client

服務清單: 補 sentry 主服務到 service-registry.yaml (AUTO 級別)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 14:11:50 +08:00
OG T
85d4857d1b fix(monitoring): RedisMemoryHigh 誤報 — max_bytes=0 除以零修正
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 37s
Details 
- 加入 redis_memory_max_bytes > 0 前置條件
- 防止 Redis 未設 maxmemory 時除以零產生 +Inf 永遠觸發
- 影響: alerts-unified.yml + database-alerts.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:41:10 +08:00
OG T
7e327c806e feat(alertmanager): Telegram Fallback 直送路徑 (ADR-035) 
新增 telegram-direct receiver，critical 告警同時走：
1. awoooi-webhook (主路徑: AI 分析 + 去重)
2. telegram-direct (fallback: AWOOOI API 掛時直接通知)

continue:true 讓 critical route 同時匹配兩個 receiver。
warning 僅走 awoooi-webhook，避免雙重通知。

已在 110 熱重載驗證 (receivers: awoooi-webhook + telegram-direct)。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 09:04:46 +08:00
OG T
9799a14f54 feat(monitoring): Plan C 外部網站告警 — 4個網站 + SSL憑證預警
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 34s
Details 
新增 external_website_alerts 群組：
- MoWoooWorkDown (mo.wooo.work, 188, momo-app)
- TsenyangWebsiteDown (tsenyang.com, 188, tsenyang-website)
- StockWoooWorkDown (stock.wooo.work, 110, stock-platform)
- BitanWoooWorkDown (bitan.wooo.work, 188, bitan-app)
- ExternalSiteSSLExpiringSoon (14天預警, auto_repair:false)

blackbox-http 已涵蓋全部目標，此為結構化告警規則。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 08:53:08 +08:00
OG T
3c6807d79c ops(monitoring): 觸發 deploy-alerts — database_detail_alerts 6條規則補部署
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
d9e0fab 新增了 6 條 DB 詳細告警規則但 deploy-alerts 因 pyyaml 未安裝失敗
0f86c5c 已修復 workflow，此 commit 觸發重新部署

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 21:17:26 +08:00
OG T
d9e0fab3fe feat(monitoring): Sprint 5.2 Plan B — 資料庫詳細告警規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 17s
Details 
新增 database_detail_alerts 規則群組:
  PostgreSQL:
    - PostgreSQLSlowQueries: 慢查詢 >60s
    - PostgreSQLDeadlocks: 死鎖發生
    - PostgreSQLTooManyConnections: 連接數 >50
  Redis:
    - RedisKeyEviction: Key 驅逐
    - RedisConnectionsHigh: 連接數 >100
    - RedisCommandLatencyHigh: 命令延遲 >10ms

前置: postgres-exporter:9187 + redis-exporter:9121 已在 188 部署 ✅
Prometheus scrape 已更新 ✅

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:19:03 +08:00
OG T
170ce2f11d fix(ci): 修正測試與 Sprint 5.2 部署腳本
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
Details 
tests/test_auto_repair_service.py:
  - 更新 3個測試符合 2026-04-07 統帥指令移除門檻
  - APPROVED Playbook 直接通過 (低相似度/低品質/高風險均通過)

tests/test_phase22_nemotron_collab.py:
  - 更新 log key: nemotron_collaboration_failed → exhausted

ops/monitoring/docker-compose.exporters.yaml:
  - 修正 postgres DSN: awoooi:awoooi_prod_2026@localhost:5432/awoooi_prod

Sprint 5.2 新增腳本:
  - scripts/sprint51_e2e_validation.py: L7 E2E 驗收腳本 (T1-T5)
  - scripts/ops/deploy-docker-health-monitor.sh: Plan A 一鍵部署腳本

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:17:48 +08:00
OG T
0847fa3a60 feat(sprint5.1): L2-2 — alerts-unified.yml 補 DockerContainerUnhealthy/Exited 規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 19s
Details 
新增 docker_health_alerts group：
  - DockerContainerUnhealthy: container_health_status==0, for 2m, auto_repair=true
  - DockerContainerExited:    container_running_status==0, for 1m, auto_repair=true

標籤 auto_repair=true 讓 AWOOOI API 進入 Guardrail 決策鏈路，
實際修復動作由 Service Registry 分級（ADR-062）決定，
docker-health-monitor.sh（純感知層）送 webhook 後由此規則補充 Prometheus 路徑。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 16:40:44 +08:00
OG T
88696dba9b feat(sprint5.1): Data Safety Guardrails 全鏈路整合 (L1-L5)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m33s
Details
Type Sync Check / check-type-sync (push) Failing after 58s
Details 
Layer 0 - K8s RBAC:
  - k8s/rbac/api-velero-reader.yaml: awoooi-executor SA Velero backup reader

Layer 1 - DB Migration (已在 188 執行):
  - M-002: approval_records 新增 approval_level/votes/required_votes
  - M-003: alert_event_type ENUM 新增 8 個值

Layer 2 - IaC:
  - ops/config/service-registry.yaml: 全服務 Stateful 分級清單 (BLOCK/CRITICAL_HITL/STANDARD_HITL/AUTO)

Layer 3 - Python Services:
  - service_registry.py: 讀取 YAML，提供 is_blocked/requires_multisig/get_required_votes
  - velero_client.py: kubectl 查詢 Velero 備份年齡，失敗 fallback 999h
  - preflight_service.py: Pre-flight 安全檢查 (Q2/Q4 決策)

Layer 1-M001 - Playbook model:
  - playbook.py: 新增 requires_approval_level/stateful_targets/requires_pre_backup

Layer 4 - 業務邏輯:
  - alert_operation_log_repository.py: 新增 8 個 event_type (Guardrail/Pre-flight/MultiSig/備份)
  - auto_repair_service.py: 注入 Service Registry Guardrail 檢查 (BLOCK → 直接拒絕)
  - webhooks.py: ALERT_RECEIVED 溯源記錄 + auto_repair flag Q9 + Langfuse trace_id Q10
  - db/models.py: ApprovalRecord 同步 approval_level/votes/required_votes 欄位
  - docker-health-monitor.sh: 純感知層改造（移除所有 docker restart 邏輯）

Layer 5 - Telegram 通知:
  - telegram_gateway.py: T1-T6 六個新通知方法 (Guardrail/Pre-flight/Backup/MultiSig/ChangeApplied)

參考: ADR-062 Data Safety Guardrails, ADR-063 Service Registry IaC

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 16:24:09 +08:00
OG T
e70ceaba61 ops(signoz): 建立 log-based alert rules 文檔 (Sprint 2) 
5 條規則: APIHighErrorLogRate/WorkerTaskFailed/PodOOMKilled/
         TelegramPollingFailed/NemotronAllTimeout
含 SigNoz UI 設定步驟 + webhook 驗證指令
標籤與 Prometheus 統一規範對齊 (layer/component/team)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 11:10:02 +08:00
OG T
dc27f8f811 ops(monitoring): 統一 Prometheus 告警規則 — 40+條含統一 layer 標籤 
修正:
- ClawBotDown → OpenClawDown (舊命名廢棄)
- 加入 SentryDown/HarborDown/GiteaDown/AlertmanagerDown
- 所有規則補齊 layer/component/host/auto_repair 統一標籤
- 整合 k8s/monitoring/*.yaml → ops/monitoring/alerts-unified.yml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 02:26:18 +08:00
OG T
30b7b10f01 feat(grafana): Wave D — AI監控 + 基礎設施 Dashboard (Grafana 188:3002) 
新增 2 個 Dashboard，匯入既有 Nemotron Dashboard：

1. ai-monitoring.json — LLM + NVIDIA AI 監控
   - LLM 呼叫速率 (req/min)
   - LLM P99/P50 延遲
   - Nemotron Tool Calling P99/P50 延遲
   - LLM Cache 命中率 %
   - LLM Fallback 次數
   - Alert Chain 健康/最後成功時間

2. infra-monitoring.json — Node + K3s 基礎設施
   - CPU/Memory 使用率
   - K3s Pod 數量 (by namespace)
   - K3s Pod 重啟次數
   - Prometheus Targets UP/DOWN
   - API 請求速率

3. nvidia-nemotron.json — 既有 18-panel Nemotron Dashboard (版控)

部署: 192.168.0.188:3002 (Grafana 12.4.1)
Provisioning: monitoring/grafana/provisioning/dashboards/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 00:18:00 +08:00
OG T
3f339110dd fix(observability): 同步 .188 實際部署調整至 repo
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
與原始計畫的差異:

1. MinIO Bearer Token 認證
   - 原計畫: MINIO_PROMETHEUS_AUTH_TYPE=public (此版本不支援)
   - 實際: mc admin prometheus generate 產生 Bearer Token
   - 更新: prometheus-config-phase-o.yaml 加入 bearer_token

2. remote_write 廢棄 → OTEL Collector Prometheus scrape
   - 原計畫: Prometheus remote_write → SigNoz OTEL /api/v1/write
   - 實際: SigNoz OTEL Collector 不支援 Prometheus remote_write 格式 (404)
   - 改用: OTEL Collector prometheus receiver 直接 scrape node-exporter + kube-state-metrics
   - 新增: ops/signoz/otel-collector-config-phase-o.yaml (版本控管副本)

3. ADR-053 驗收清單更新為實際結果

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-04-02 21:23:47 +08:00
OG T
3e4612f259 docs(observability): ADR-053 SigNoz 統一架構 + Phase O 驗收
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 36s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- 新增 ADR-053: 可觀測性統一架構決策記錄
- 更新 service-registry.yaml: 補齊 MinIO/Kali 監控入口
- 更新 LOGBOOK: Phase O 完成狀態

Phase O 驗收清單:
✅ kubectl Mac 本機免密碼
✅ OTEL Collector 2 Pod Running
✅ Event Exporter 1 Pod Running
✅ Descheduler CronJob Completed
✅ MinIO + Kali 告警規則
✅ Alert Chain Smoke Test
✅ CD Pipeline 整合
⏳ ClickHouse TTL / remote_write / SigNoz rules (待 .188 手動)

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-04-02 18:26:57 +08:00
OG T
a5a6bd3408 feat(monitoring): K8s alert rules + Grafana dashboards + ops 腳本 
- k8s/monitoring/alert-chain-monitor.yaml
- k8s/monitoring/database-alerts.yaml
- ops/grafana/ Grafana dashboards
- ops/signoz/ SignOz 配置
- ops/scripts/ 維運腳本

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:04:14 +08:00
OG T
c7f9c119e7 fix(cd): 補提交 ops/monitoring 腳本 
遺漏文件導致 CD Monitoring Coverage 步驟失敗

新增:
- generate_monitoring.py - 監控覆蓋率檢查
- coverage_report.py - 覆蓋率報告
- discover_docker.py - Docker 服務發現
- deploy-exporters.sh - Exporter 部署腳本
- postgres-exporter-queries.yaml - PostgreSQL 查詢配置

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:45:42 +08:00
OG T
12e49d844a feat(monitoring): ADR-037 Wave B - Database Exporters + Prometheus 整合 
- 部署 PostgreSQL Exporter (192.168.0.188:9187)
- 部署 Redis Exporter (192.168.0.188:9121)
- 更新 Prometheus scrape config
- 首席架構師審查: 97% OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:18:54 +08:00
OG T
d15fb7d9f4 fix(cd): 序列建構修復 Runner _runner_file_commands 衝突 
根因: 並行 Job 的 Set up job 階段會同時寫入 RUNNER_TEMP
解法: build-api needs build-web，確保序列執行
移除: Job-level concurrency groups (不再需要)
更新: ops/runner/README.md v1.0→v2.0

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 10:29:11 +08:00
OG T
07114f9181 fix(runner): v4 - 啟用 cancel-in-progress 防止並行衝突 
根因確認:
- _diag/pages 衝突發生在 "Set up job" 階段
- 這是在任何自定義步驟執行之前
- Runner 內部 bug，workflow 層清理無法解決

永久解決方案:
- cancel-in-progress: true (確保同一時間只有一個 workflow)
- 不再嘗試清理 RUNNER_TEMP (會破壞其他 Job)
- 保留 _diag/pages 清理作為輔助措施

更新 ops/runner/README.md:
- 完整根因分析
- v3 最終解決方案說明
- 警告: 不要清理 RUNNER_TEMP

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:10:17 +08:00
OG T
93c3280481 feat(monitoring): Phase 20 Nemotron 完整監控整合 
服務註冊表:
- 新增 nvidia-nemotron AI 服務
- 3 個 Prometheus metrics 定義
- 4 個告警規則 (circuit_breaker, timeout, error_rate, rate_limit)
- fallback 策略 (nvidia → gemini → ollama)

Alertmanager 規則:
- NvidiaCircuitBreakerOpen (P1)
- NvidiaToolCallingHighLatency (P2)
- NvidiaToolCallingHighErrorRate (P0)
- NvidiaCircuitBreakerHalfOpen (Info)
- NvidiaCircuitBreakerClosed (Info)
- NvidiaNoRequests (P3)

自動修復:
- fallback_to_gemini
- fallback_to_ollama
- switch_model

ADR: ADR-036

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:05:59 +08:00
OG T
183776a34f fix(runner): 永久修復 _diag/pages 檔案衝突問題 
問題: Runner 並行執行時 "file already exists" 導致 CD 失敗

解決方案:
1. CD Workflow: 刪除整個 _diag/pages 目錄再重建 (非 rm -rf /*)
2. Systemd Timer: 每 5 分鐘自動清理過期檔案
3. flock 鎖定: 防止清理程序競爭

新增檔案:
- ops/runner/cleanup-runner-diag.sh - 清理腳本
- ops/runner/runner-diag-cleanup.service - Systemd service
- ops/runner/runner-diag-cleanup.timer - 定時器
- ops/runner/deploy-runner-cleanup.sh - 部署腳本
- ops/runner/README.md - 文檔

部署指令:
  ssh wooo@192.168.0.110
  bash awoooi/ops/runner/deploy-runner-cleanup.sh

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:04:35 +08:00
OG T
40163a51b5 feat(monitoring): 完整監控策略與自動整合架構 
新增:
1. MONITORING_COMPLETE_STRATEGY.md - 完整監控策略
   - 5 主機 × 60+ 服務監控矩陣
   - P0/P1/P2 告警規則清單
   - AI 自動修復閉環流程
   - 安全護欄配置

2. MONITORING_INTEGRATION_ARCHITECTURE.md - 自動整合架構
   - 服務註冊表 (Single Source of Truth)
   - CI/CD 自動驗證監控覆蓋率
   - 新服務自動獲得監控

3. ops/monitoring/service-registry.yaml - 服務清單
   - K8s 工作負載 (API/Web/Worker/ArgoCD)
   - Docker 容器 (Ollama/OpenClaw/Redis/Postgres)
   - 前端頁面 SLO
   - API 端點 SLO
   - 告警模板與自動修復動作

4. ops/monitoring/validate_coverage.py - 覆蓋率驗證
   - CI 階段執行
   - 檢測未監控服務
   - 生成覆蓋率報告

設計原則:
- 監控即代碼 (Monitoring as Code)
- 新服務必須在 registry 註冊才能部署
- 自動發現機制防止遺漏

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:52:08 +08:00