wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
1,166 Commits 190 Branches 2 Tags
a4d655ea7f050ce22ffc4e893e8fb6da4a5e609f
Commit Graph

14 Commits

Author SHA1 Message Date
OG T
e1dfbedf0e fix(alerts): HostHighCpuLoad auto_repair: false → true
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
飛輪一直 GUARDRAIL_BLOCKED 的根本原因：
Prometheus rule 標籤 auto_repair=false 強制 HITL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 13:33:23 +08:00
OG T
ab3e266a23 fix(monitoring): Phase O-6.2 service-registry 補齊 9 個缺失 K8s 部署 
新增:
- argocd 5個元件 (applicationset/dex/notifications/redis/repo-server)
- awoooi-dev/awoooi-api
- kube-state-metrics
- observability/event-exporter
- velero/velero

結果: prometheus 覆蓋率 94%→96%, errors 9→0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 10:44:36 +08:00
OG T
85d4857d1b fix(monitoring): RedisMemoryHigh 誤報 — max_bytes=0 除以零修正
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 37s
Details 
- 加入 redis_memory_max_bytes > 0 前置條件
- 防止 Redis 未設 maxmemory 時除以零產生 +Inf 永遠觸發
- 影響: alerts-unified.yml + database-alerts.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:41:10 +08:00
OG T
9799a14f54 feat(monitoring): Plan C 外部網站告警 — 4個網站 + SSL憑證預警
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 34s
Details 
新增 external_website_alerts 群組：
- MoWoooWorkDown (mo.wooo.work, 188, momo-app)
- TsenyangWebsiteDown (tsenyang.com, 188, tsenyang-website)
- StockWoooWorkDown (stock.wooo.work, 110, stock-platform)
- BitanWoooWorkDown (bitan.wooo.work, 188, bitan-app)
- ExternalSiteSSLExpiringSoon (14天預警, auto_repair:false)

blackbox-http 已涵蓋全部目標，此為結構化告警規則。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 08:53:08 +08:00
OG T
3c6807d79c ops(monitoring): 觸發 deploy-alerts — database_detail_alerts 6條規則補部署
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
d9e0fab 新增了 6 條 DB 詳細告警規則但 deploy-alerts 因 pyyaml 未安裝失敗
0f86c5c 已修復 workflow，此 commit 觸發重新部署

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 21:17:26 +08:00
OG T
d9e0fab3fe feat(monitoring): Sprint 5.2 Plan B — 資料庫詳細告警規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 17s
Details 
新增 database_detail_alerts 規則群組:
  PostgreSQL:
    - PostgreSQLSlowQueries: 慢查詢 >60s
    - PostgreSQLDeadlocks: 死鎖發生
    - PostgreSQLTooManyConnections: 連接數 >50
  Redis:
    - RedisKeyEviction: Key 驅逐
    - RedisConnectionsHigh: 連接數 >100
    - RedisCommandLatencyHigh: 命令延遲 >10ms

前置: postgres-exporter:9187 + redis-exporter:9121 已在 188 部署 ✅
Prometheus scrape 已更新 ✅

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:19:03 +08:00
OG T
170ce2f11d fix(ci): 修正測試與 Sprint 5.2 部署腳本
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
Details 
tests/test_auto_repair_service.py:
  - 更新 3個測試符合 2026-04-07 統帥指令移除門檻
  - APPROVED Playbook 直接通過 (低相似度/低品質/高風險均通過)

tests/test_phase22_nemotron_collab.py:
  - 更新 log key: nemotron_collaboration_failed → exhausted

ops/monitoring/docker-compose.exporters.yaml:
  - 修正 postgres DSN: awoooi:awoooi_prod_2026@localhost:5432/awoooi_prod

Sprint 5.2 新增腳本:
  - scripts/sprint51_e2e_validation.py: L7 E2E 驗收腳本 (T1-T5)
  - scripts/ops/deploy-docker-health-monitor.sh: Plan A 一鍵部署腳本

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:17:48 +08:00
OG T
0847fa3a60 feat(sprint5.1): L2-2 — alerts-unified.yml 補 DockerContainerUnhealthy/Exited 規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 19s
Details 
新增 docker_health_alerts group：
  - DockerContainerUnhealthy: container_health_status==0, for 2m, auto_repair=true
  - DockerContainerExited:    container_running_status==0, for 1m, auto_repair=true

標籤 auto_repair=true 讓 AWOOOI API 進入 Guardrail 決策鏈路，
實際修復動作由 Service Registry 分級（ADR-062）決定，
docker-health-monitor.sh（純感知層）送 webhook 後由此規則補充 Prometheus 路徑。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 16:40:44 +08:00
OG T
dc27f8f811 ops(monitoring): 統一 Prometheus 告警規則 — 40+條含統一 layer 標籤 
修正:
- ClawBotDown → OpenClawDown (舊命名廢棄)
- 加入 SentryDown/HarborDown/GiteaDown/AlertmanagerDown
- 所有規則補齊 layer/component/host/auto_repair 統一標籤
- 整合 k8s/monitoring/*.yaml → ops/monitoring/alerts-unified.yml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 02:26:18 +08:00
OG T
3e4612f259 docs(observability): ADR-053 SigNoz 統一架構 + Phase O 驗收
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 36s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- 新增 ADR-053: 可觀測性統一架構決策記錄
- 更新 service-registry.yaml: 補齊 MinIO/Kali 監控入口
- 更新 LOGBOOK: Phase O 完成狀態

Phase O 驗收清單:
✅ kubectl Mac 本機免密碼
✅ OTEL Collector 2 Pod Running
✅ Event Exporter 1 Pod Running
✅ Descheduler CronJob Completed
✅ MinIO + Kali 告警規則
✅ Alert Chain Smoke Test
✅ CD Pipeline 整合
⏳ ClickHouse TTL / remote_write / SigNoz rules (待 .188 手動)

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-04-02 18:26:57 +08:00
OG T
c7f9c119e7 fix(cd): 補提交 ops/monitoring 腳本 
遺漏文件導致 CD Monitoring Coverage 步驟失敗

新增:
- generate_monitoring.py - 監控覆蓋率檢查
- coverage_report.py - 覆蓋率報告
- discover_docker.py - Docker 服務發現
- deploy-exporters.sh - Exporter 部署腳本
- postgres-exporter-queries.yaml - PostgreSQL 查詢配置

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:45:42 +08:00
OG T
12e49d844a feat(monitoring): ADR-037 Wave B - Database Exporters + Prometheus 整合 
- 部署 PostgreSQL Exporter (192.168.0.188:9187)
- 部署 Redis Exporter (192.168.0.188:9121)
- 更新 Prometheus scrape config
- 首席架構師審查: 97% OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:18:54 +08:00
OG T
93c3280481 feat(monitoring): Phase 20 Nemotron 完整監控整合 
服務註冊表:
- 新增 nvidia-nemotron AI 服務
- 3 個 Prometheus metrics 定義
- 4 個告警規則 (circuit_breaker, timeout, error_rate, rate_limit)
- fallback 策略 (nvidia → gemini → ollama)

Alertmanager 規則:
- NvidiaCircuitBreakerOpen (P1)
- NvidiaToolCallingHighLatency (P2)
- NvidiaToolCallingHighErrorRate (P0)
- NvidiaCircuitBreakerHalfOpen (Info)
- NvidiaCircuitBreakerClosed (Info)
- NvidiaNoRequests (P3)

自動修復:
- fallback_to_gemini
- fallback_to_ollama
- switch_model

ADR: ADR-036

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:05:59 +08:00
OG T
40163a51b5 feat(monitoring): 完整監控策略與自動整合架構 
新增:
1. MONITORING_COMPLETE_STRATEGY.md - 完整監控策略
   - 5 主機 × 60+ 服務監控矩陣
   - P0/P1/P2 告警規則清單
   - AI 自動修復閉環流程
   - 安全護欄配置

2. MONITORING_INTEGRATION_ARCHITECTURE.md - 自動整合架構
   - 服務註冊表 (Single Source of Truth)
   - CI/CD 自動驗證監控覆蓋率
   - 新服務自動獲得監控

3. ops/monitoring/service-registry.yaml - 服務清單
   - K8s 工作負載 (API/Web/Worker/ArgoCD)
   - Docker 容器 (Ollama/OpenClaw/Redis/Postgres)
   - 前端頁面 SLO
   - API 端點 SLO
   - 告警模板與自動修復動作

4. ops/monitoring/validate_coverage.py - 覆蓋率驗證
   - CI 階段執行
   - 檢測未監控服務
   - 生成覆蓋率報告

設計原則:
- 監控即代碼 (Monitoring as Code)
- 新服務必須在 registry 註冊才能部署
- 自動發現機制防止遺漏

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:52:08 +08:00