wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity
4,108 Commits 195 Branches 2 Tags
44ea892e4fb3e5847eaef30c8e9564d0acdafcea
Commit Graph

17 Commits

Author SHA1 Message Date
Your Name
ee2cc2bfc3 fix(alerts): 收斂 Telegram 告警到 SRE 戰情室
Some checks failed
CD Pipeline / tests (push) Failing after 1m23s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Successful in 15s
Details
2026-06-12 11:06:16 +08:00
Your Name
4452a006bf feat(governance): show knowledge degradation ownership
All checks were successful
Code Review / ai-code-review (push) Successful in 13s
Details
CD Pipeline / tests (push) Successful in 6m20s
Details
CD Pipeline / build-and-deploy (push) Successful in 4m54s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m48s
Details
2026-05-19 20:38:29 +08:00
Your Name
bf8974be03 fix(governance): normalize knowledge degradation payloads
All checks were successful
Code Review / ai-code-review (push) Successful in 11s
Details
CD Pipeline / tests (push) Successful in 5m55s
Details
CD Pipeline / build-and-deploy (push) Successful in 4m22s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m39s
Details
2026-05-19 15:35:59 +08:00
Your Name
795c9a4e93 fix(governance): clarify knowledge degradation alerts
All checks were successful
Code Review / ai-code-review (push) Successful in 14s
Details
CD Pipeline / tests (push) Successful in 6m1s
Details
CD Pipeline / build-and-deploy (push) Successful in 4m32s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m34s
Details
2026-05-19 15:14:47 +08:00
Your Name
341c3b6523 fix(telegram): format governance and runbook alerts
All checks were successful
Code Review / ai-code-review (push) Successful in 10s
Details
CD Pipeline / tests (push) Successful in 1m13s
Details
CD Pipeline / build-and-deploy (push) Successful in 3m22s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m28s
Details
2026-05-07 00:58:20 +08:00
Your Name
8f715fd3f2 fix(telegram): sanitize failover alert errors
All checks were successful
Code Review / ai-code-review (push) Successful in 10s
Details
CD Pipeline / tests (push) Successful in 1m5s
Details
CD Pipeline / build-and-deploy (push) Successful in 3m25s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m16s
Details
2026-05-06 16:45:47 +08:00
Your Name
3f853accf2 fix(alerter): Ollama 恢復告警去重修復 — per-host key + 1h TTL 
根因：
1. dedup_key 固定為 "alert:recovery"，GCP-A 每 10min 健康閃爍就觸發重發
2. 三層容災下不同主機恢復共用同一個 key，互相污染

修法：
- dedup key 改為 "alert:recovery:{safe_host}"，各主機獨立 dedup
- RECOVERY_DEDUP_TTL_SEC = 3600（1h），GCP 持續閃爍只報一次

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-05 01:22:01 +08:00
Your Name
b1ef05fa8c feat(ollama): ADR-110 GCP 三層容災架構（GCP-A → GCP-B → Local → Gemini）
Some checks failed
Code Review / ai-code-review (push) Successful in 50s
Details
CD Pipeline / tests (push) Failing after 1m14s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details 
## 變更摘要
- Primary: http://34.143.170.20:11434 (GCP-A SSD, 9x 載速 + 2x 推理)
- Secondary: http://34.21.145.224:11434 (GCP-B SSD)
- Fallback: http://192.168.0.111:11434 (M1 Pro Local HDD，最後防線)
- 廢止 ADR-105「111 唯一鐵律」，新建 ADR-110

## 核心改動
- config.py: 新增 OLLAMA_SECONDARY_URL；validator 加 GCP IP 白名單（34.143.170.20, 34.21.145.224）
- ollama_failover_manager.py: 三層 Ollama 決策矩陣；並行健康檢查三台；health_111 → health_gcp_a
- ollama_health_monitor.py: host label 萃取改為通用版（支援 GCP 公網 IP）
- failover_alerter.py: 故障/恢復主機動態顯示，不再硬編碼「Ollama 111 (GPU)」
- ollama_auto_recovery.py: notify_recovery 改為 ollama_gcp_a；recovered_host 動態
- k8s/awoooi-prod: configmap + deployment + network-policy 同步更新（egress 加 GCP /32）
- 服務層: 10 個服務檔案硬編碼 192.168.0.111 改為讀 settings.OLLAMA_URL
- 測試: URL 常數更新，新增三層容災場景，GCP IP 白名單驗證測試

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 22:49:23 +08:00
Your Name
f1362fcc8d fix(governance): 修治理告警 4 個 silent failure + Prom sentinel 連鎖
Some checks failed
Code Review / ai-code-review (push) Successful in 49s
Details
CD Pipeline / tests (push) Successful in 2m9s
Details
CD Pipeline / build-and-deploy (push) Failing after 31m11s
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details 
【全景檢測：12-agent 並行掃描定位 4 大 bug 與 1 個 P0 連鎖回歸】

Bug 1（P0 silent failure）— governance_agent.check_trust_drift
  原 `await db.commit()` 縮排錯在 async with 區塊外（8 空格 vs 12），
  session 已 auto-commit 關閉，二次 commit 拋 InvalidRequestError 被吞，
  governance_trust_drift_auto_deprecated log 從不出現。修：commit/log 移回 with 內。
  附 AST regression guard test 擋退化。

Bug 2 — flywheel_stats_service / W-3 fresh deploy 假告警
  Redis 空時 total_exec=0 → rate=0.0 → watchdog `< 0.30` 立即觸發
  「飛輪成功率 0%」假告警。修：total_exec < FLYWHEEL_MIN_SAMPLE(10) 回 None，
  watchdog 判 None 跳過 W-3。Prometheus sentinel 用 NaN（非 -1.0）
  避免觸發 ops/monitoring/alerts.yml:775 等 3 份 prom rule 的 `< 0.1`
  條件造成 2h 後假告警連鎖。前端 type 同步 number | null。

Bug 3 — failover_alerter dedup key
  原 key 只看 event_type 不看 payload，trust_drift 4→25 IDs 變動全被
  1h dedup 吞掉。修：dedup key 加 sha256(impact subdict)[:8]，event_type
  sanitize 防特殊字元污染 Redis key。

Bug 4 — ai_slo_watchdog_job W-4 evolver 全封存初始化誤報
  原邏輯 approved==0 即告警，未排除「playbooks 表初始化中」場景。
  修：_count_approved_playbooks 回 (approved, total)，total==0 → skip。

【執行結果】
- 39 個相關 unit test 全過（test_failover_alerter / test_governance_agent /
  test_trust_drift_watchdog / test_check_trust_drift_commit_outside_context_poc）
- 6 個關鍵路徑實測：NaN sentinel / float 渲染 / hash 區分性 / dedup 同 impact
  相同 hash / datetime 容錯 / 4 檔 py_compile 全過

【調度教訓 — 留作未來改進】
- 12-agent 並行調度時，vuln-verifier 與 fullstack-engineer 競態
  導致 vuln-verifier 讀到已修代碼誤判 NOT REPRODUCIBLE。
  未來：vuln-verifier 應在 fullstack 之前執行，或用 git show HEAD~1 對比修復前。
- fullstack-engineer 引入 P0 regression（f-string 內嵌 ternary 非法 format spec），
  critic 抓到 + Prom sentinel 連鎖 — 證明 critic 審查必要不可省。

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 00:18:57 +08:00
Your Name
b710f3f38f feat(governance): normalize AI治理告警輸出與元告警解析度
Some checks failed
CD Pipeline / tests (push) Failing after 25s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Successful in 46s
Details
2026-05-02 23:49:59 +08:00
Your Name
61f5a6a419 fix(telegram): route alerts to SRE war room
Some checks failed
CD Pipeline / tests (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
Code Review / ai-code-review (push) Has been cancelled
Details
2026-04-30 15:01:23 +08:00
Your Name
80defbed7c fix(aiops): fallback and escalate automation blockers
Some checks failed
CD Pipeline / tests (push) Successful in 2m41s
Details
Code Review / ai-code-review (push) Successful in 24s
Details
CD Pipeline / build-and-deploy (push) Successful in 7m51s
Details
CD Pipeline / post-deploy-checks (push) Failing after 2m15s
Details
2026-04-30 14:13:57 +08:00
Your Name
ae5e33d254 feat(failover+dispatcher): 補齊 unstaged 服務變更 
- callback_dispatcher: params 型別放寬支援 numeric
- failover_alerter: alert TTL 修正
- model_version_tracker: 小調整

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 19:56:51 +08:00
Your Name
cc547736ab feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復 
承接 Wave 6/7/8 多 engineer 在 agent 限額前完成的代碼，補 commit 解 production
HEAD 隱性 import error（decision_fusion 已被 decision_manager 引用但檔案 untracked）。

新增（後端核心）:
- decision_fusion.py (562 行) — P2.1 方法 III（OpenClaw + Hermes + Elephant 三 LLM 融合）
- aiops_timeline.py + aiops_timeline_service.py — critic B4 修復
  /api/v1/aiops/timeline endpoint，DB 存取抽到 service 層遵守 leWOOOgo 積木化
- migrations/p2_decision_fusion_columns.sql + rollback — approval_records fusion 欄位

修改（後端整合）:
- decision_manager.py — fusion 三斷鏈修補（critic B1+B2+B3）：
  · B1: 寫 _evidence_snapshot_ref 到 token.proposal_data
  · B2: fusion 前計算 complexity_score 並寫 token
  · B3: fusion composite 寫 token.proposal_data["decision_fusion"]
- auto_approve.py — fusion + consensus 認識（critic B3+B5）：
  · composite > 0.7 → auto_execute_eligible bypass min_confidence
  · source=consensus_engine + score>=0.6 → 規則可信路徑
- consensus_engine.py — db-fix _save_consensus 重用 agent_sessions
- governance_agent.py — db-fix _alert PG 寫入 ai_governance_events
- approval_db.py — fusion 3 欄位 + 2 partial index + CheckConstraint
- db/models.py — schema 對齊 migration
- core/config.py — vuln #1 修復：OLLAMA_URL/_FALLBACK_URL field_validator
  拒絕公網 IP + 外部域名，僅允許私網/loopback/K8s SVC 白名單
- core/feature_flags.py — P2 fusion + consensus flags
- main.py — governance_agent lifespan 啟動
- failover_alerter.py — Wave8-X2: in-memory dedup fallback（Redis 拒絕後不 fail-open）
- ollama_*.py — metrics 整合 + recovery 改善
- auto_repair_service.py — verifier 接線

新增（測試 2438 行）:
- test_decision_fusion.py / test_governance_agent.py / test_consensus_integration.py
- test_p2_db_fixes.py / test_wave8_fusion_fixes.py
- test_config_url_validation.py（vuln #1 12 tests）
- test_failover_alerter.py +Wave8-X2 in-memory dedup 補測

驗收: 116 tests pass (decision_fusion + wave8_fusion + config_url + consensus +
                      governance + p2_db_fixes + failover_alerter)

Conflict resolution:
- 3 檔（config.py + auto_approve.py + decision_manager.py）git stash pop 衝突
  保留 stashed (engineer 最終版)，補回 ValueError 「公網 IP」字樣對齊 test

Note: 此 commit 解 production HEAD 隱性 import error
仍未修: vuln #4 prompt injection / debugger B14 quota fail-closed
       / B25-B26 drain_pending_tasks / B8 governance fail alert

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Multiple Engineers (Wave 6/7/8) <noreply@anthropic.com>
 2026-04-27 08:11:40 +08:00
Your Name
2c57b71db9 feat(wave5-p2): GovernanceAgent 4 項自檢 + Ollama 健康告警規則 + Prometheus metrics 整合
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 10m45s
Details 
MASTER plan_complete_v3.md Wave 5 P2.2 + P2.3 完成（multiple engineers 在限額前完成代碼，補 commit）：

P2.2 — GovernanceAgent 4 項自檢:
- governance_agent.py (342 行) — 每 1 小時自檢循環:
  · trust_drift（信任度漂移檢測）
  · knowledge_degradation（知識退化檢測）
  · llm_hallucination（LLM 幻覺檢測）
  · execution_blast_radius（執行爆炸半徑檢測）
- main.py lifespan: asyncio.create_task(run_governance_loop()) 啟動
  try/except 包裹，schedule 失敗不阻斷主流程
- failover_alerter.py: alert_governance(event_type, payload) 1h dedup
  四類事件 → Telegram MarkdownV2 告警

P2.3 — Ollama 健康規則 + Prometheus Metrics:
- ops/monitoring/ollama_health_rules.yaml (148 行):
  · OllamaHealthDegraded / OllamaPrimaryDown
  · OllamaFailoverTriggered / GeminiQuotaExceeded
  · 補 Prometheus 取資料的 alert rules
- core/metrics.py (57 行):
  · GEMINI_DAILY_CALL_COUNT / GEMINI_DAILY_QUOTA Gauge
  · OLLAMA_FAILOVER_TRIGGERED_TOTAL Counter
  · OLLAMA_CURRENT_PRIMARY_IS_OLLAMA Gauge
- ollama_failover_manager.py:
  · _check_gemini_quota: 每次 check 同步更新 Gauge（讓 Prometheus 取最新值）
  · select_provider: failover 時 inc Counter + 切 Primary Gauge
  · try/except 包裹，metric 失敗不阻斷主路由

E2E 測試:
- test_failover_e2e_dispatch.py (365 行)
  完整 dispatch 路徑：health check → failover decide → alerter → metrics

Tests: 54 passed (e2e_dispatch + failover_manager + failover_alerter)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Multiple Engineers (上 session Wave 5) <noreply@anthropic.com>
 2026-04-26 20:56:19 +08:00
Your Name
32affaffeb fix(critic-hotfix): 4 修補 critic BLOCKER + HIGH（CD 阻塞 + 飛輪空轉）
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
Critic 全面審查 6 個 commit 後抓出：

CD 阻塞修復:
- test_ai_router_failover_integration.py: 3 個 test 改用 patch.object 直接
  mock _select_provider_and_model 強制初始 OLLAMA。原 IntentType.UNKNOWN mock
  在 router 內仍被 reclassify 成 DIAGNOSE → openclaw_nemo，failover 不觸發。
  → 5/5 PASSED

BLOCKER B1 — Gitea Telegram 通知永遠發不出去:
- apps/api/src/api/v1/gitea_webhook.py:399
  redis = await get_redis()  →  redis = get_redis()
  原 await 會 raise TypeError 被外層 except 吞 → Task C PR merged + workflow_run
  failure 通知全部失效（CI 綠燈是假象，test 只驗 HTTP 202 不驗實際送達）

BLOCKER B2 — P1.3+P1.4 學習鏈閉環空轉（兩處同 bug）:
- apps/api/src/api/v1/webhooks.py:261
- apps/api/src/services/approval_execution.py:771（pre-existing）
  EvidenceSnapshot.get_latest_snapshot(...) 是 module-level async function
  不是 classmethod → AttributeError 被 except 吞成 warning
  → 飛輪閉環假性接通實際空跑（feature flag default off 暫時免爆）

HIGH H3 — main.py lifespan 順序競爭:
- apps/api/src/main.py: configure_alerter() 移到 _recovery_svc.start() 之前
  原順序：start() 觸發 immediate-check → 可能呼叫 alert_recovery，但 alerter
  尚未注入 Redis → dedup fail-open，重複告警風險。

HIGH H1 — Gemini quota dedup 跨日吞告警:
- apps/api/src/services/failover_alerter.py:89
  dedup key 加 :{YYYY-MM-DD} 後綴，每日獨立 dedup window
  原昨 22:00 觸發，今 21:30 再觸發時 dedup 還沒過期會被吞掉

Tests: 14 passed (failover_alerter + ai_router_failover_integration + lifespan_wiring)

延後 follow-up:
- H2: proactive_inspector memory metric 改名 + baseline 清理
- H4: probe_success NaN fallback
- M1-M4 / S1-S2: 見 critic 報告

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 20:39:53 +08:00
Your Name
55c6b4e2d9 feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警 
ADR-092 P1 飛輪閉環的 Ollama 失敗轉移子系統，全部 Engineer-A2/C/C2 補上。

新服務 (1581 行)：
- ollama_health_monitor.py (356)：3 層健康檢測（TCP/HTTP/推理）
- ollama_failover_manager.py (571)：111→188 自動切換 + Redis 持久化 + recovery callback
- ollama_auto_recovery.py (436)：30s 背景監控 + 連續 3 次 HEALTHY → 切回 + clear_cache
- failover_alerter.py (218)：P1.5 Telegram 容災告警

服務整合：
- ai_router.py: AIProviderEnum.OLLAMA_188 + 120s budget + failover fallback chain
- main.py lifespan: 啟動時 wire callback + start recovery，關閉時優雅 stop
- config.py: OLLAMA_FALLBACK_URL / OLLAMA_HEALTH_CHECK_MODEL / GEMINI_DAILY_QUOTA（帳單熔斷）

K8s 配置：
- 04-configmap.yaml.patch-188-fallback：注入 OLLAMA_FALLBACK_URL=http://192.168.0.188:11434

測試 (2082 行)：
- test_ollama_health_monitor.py (402)
- test_ollama_failover_manager.py (707)
- test_ollama_auto_recovery.py (580)
- test_ai_router_failover_integration.py (257)
- test_lifespan_failover_wiring.py (136)

依賴鏈：service 三件套 + ai_router + main.py 一起 commit，缺一就 ImportError。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 20:18:33 +08:00