wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 6 Packages Projects Releases Wiki Activity
1,337 Commits 195 Branches 2 Tags
2c6ed4e9cf798cc37e2e6df331f339d952fa82d7
Commit Graph

20 Commits

Author SHA1 Message Date
OG T
946fe1fa7c fix(monitoring): 合併重複飛輪告警 group + 補 notification_type: TYPE-8M
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 44s
Details 
awoooi_flywheel_health (重複) 合入 awoooi_flywheel_meta_alerts:
- 所有 5 條規則加 notification_type: TYPE-8M
- 新增 FlywheelAlertnameNullHigh（原僅在舊 group）
- 刪除重複 group，消除 Prometheus 同名告警衝突

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 22:43:02 +08:00
OG T
bd75aca727 feat(adr-075): 補全 2 個欠缺的 Prometheus 告警規則
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 49s
Details 
- MomoScraperSuccessLow: 業務爬蟲成功率 <90% (business group)
- CoreDNSResolutionFailed: CoreDNS SERVFAIL 率 >5% (kubernetes group)

ADR-075 Phase 3 完成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 21:59:18 +08:00
OG T
edb97fd29b fix(monitoring): 補回 4 個僅存於主機的 Prometheus 規則群組
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 41s
Details 
deploy-alerts.sh 部署時覆寫了這 4 個從未進 repo 的群組：
- awoooi_flywheel_health (5條：Playbook/Success/Vectorization/NullRate/Stuck)
- awoooi_backup_restore (2條：RestoreTestFailed/TestStale)
- awoooi_infrastructure_detailed (3條：Container/RedisStream/DiskGrowth)
- awoooi_host_connectivity (1條：NetworkPartition)

從 /home/wooo/monitoring/alerts.yml.bak_20260412_183835 還原。
offset PromQL 已修正為各個 selector 上，而非整個表達式。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:14:39 +08:00
OG T
f52dc459e6 feat(adr075): Step4 新增4組Prometheus規則 secops/business/flywheel_meta
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 41s
Details 
新增規則群組:
- awoooi_secops_alerts: UnauthorizedSSHLogin (5min>10次失敗)
- awoooi_business_alerts: AITokenCostSpike + GeminiAPIErrorRateHigh
- awoooi_flywheel_meta_alerts:
    FlywheelPlaybookZero / FlywheelExecutionSuccessLow
    FlywheelKMVectorizationLow / FlywheelIncidentsStuck

飛輪 meta 規則依賴 ADR-074 Exporter 指標
secops/business 規則依賴 node_exporter/awoooi custom metrics

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:51:23 +08:00
OG T
43edff184d feat(dr): Sprint C — Host rsync 備份 + DR SOP 文件 
C-1 Velero: 已確認運作中（daily-awoooi-prod schedule, 13d, MinIO Available）

C-2 Host rsync 備份:
  scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
    - Harbor registry data（最高優先）
    - Gitea repos
    - bitan-pharmacy.git（若存在）
    - 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
    - 失敗時 Telegram 告警
  ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則

C-3 DR SOP 文件:
  docs/runbooks/disaster-recovery/DR-K8s-awoooi.md  (<15分鐘)
  docs/runbooks/disaster-recovery/DR-Nginx.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Harbor.md       (<30分鐘)
  docs/runbooks/disaster-recovery/DR-Bitan.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Stock.md        (<5分鐘)

部署備份腳本說明 (需手動執行):
  scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
  ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
  ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:04:18 +08:00
OG T
6351e9a0e9 feat(mcp-phase2): MCP Phase 2 — Prometheus MCP + SSH MCP + alert labels
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m37s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 35s
Details 
MCP-2b: prometheus_provider.py
  - prometheus_query (PromQL 即時查詢)
  - prometheus_query_range (歷史趨勢，預設 15 分鐘)
  - prometheus_get_alert_history (告警觸發歷史)
  - config: PROMETHEUS_URL + PROMETHEUS_MCP_ENABLED

MCP-2a: ssh_provider.py
  - 群組A 9 個只讀診斷工具 (top/disk/memory/logs/status/port/nginx/swap)
  - 群組B 6 個安全操作工具 (restart/compose/systemctl/clear-log/ssl/nginx-reload)
  - 四層安全守衛 (白名單/allowed_hosts/forbidden_patterns/trust_score)
  - config: SSH_MCP_ENABLED + SSH_MCP_ALLOWED_HOSTS

K8s: 04-ssh-mcp-secret.example.yaml (ssh-mcp-key Secret 範本 + 建立步驟)

Alert labels: alerts-unified.yml 補充 mcp_provider/host_type/alert_category
  覆蓋: HostHighCpuLoad/HostOutOfMemory/HostOutOfDiskSpace/DockerContainer*
        SignOzDown/SentryDown/HarborDown/GiteaDown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:35:35 +08:00
OG T
e1dfbedf0e fix(alerts): HostHighCpuLoad auto_repair: false → true
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
飛輪一直 GUARDRAIL_BLOCKED 的根本原因：
Prometheus rule 標籤 auto_repair=false 強制 HITL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 13:33:23 +08:00
OG T
ab3e266a23 fix(monitoring): Phase O-6.2 service-registry 補齊 9 個缺失 K8s 部署 
新增:
- argocd 5個元件 (applicationset/dex/notifications/redis/repo-server)
- awoooi-dev/awoooi-api
- kube-state-metrics
- observability/event-exporter
- velero/velero

結果: prometheus 覆蓋率 94%→96%, errors 9→0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 10:44:36 +08:00
OG T
85d4857d1b fix(monitoring): RedisMemoryHigh 誤報 — max_bytes=0 除以零修正
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 37s
Details 
- 加入 redis_memory_max_bytes > 0 前置條件
- 防止 Redis 未設 maxmemory 時除以零產生 +Inf 永遠觸發
- 影響: alerts-unified.yml + database-alerts.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:41:10 +08:00
OG T
9799a14f54 feat(monitoring): Plan C 外部網站告警 — 4個網站 + SSL憑證預警
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 34s
Details 
新增 external_website_alerts 群組：
- MoWoooWorkDown (mo.wooo.work, 188, momo-app)
- TsenyangWebsiteDown (tsenyang.com, 188, tsenyang-website)
- StockWoooWorkDown (stock.wooo.work, 110, stock-platform)
- BitanWoooWorkDown (bitan.wooo.work, 188, bitan-app)
- ExternalSiteSSLExpiringSoon (14天預警, auto_repair:false)

blackbox-http 已涵蓋全部目標，此為結構化告警規則。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 08:53:08 +08:00
OG T
3c6807d79c ops(monitoring): 觸發 deploy-alerts — database_detail_alerts 6條規則補部署
All checks were successful
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 39s
Details 
d9e0fab 新增了 6 條 DB 詳細告警規則但 deploy-alerts 因 pyyaml 未安裝失敗
0f86c5c 已修復 workflow，此 commit 觸發重新部署

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 21:17:26 +08:00
OG T
d9e0fab3fe feat(monitoring): Sprint 5.2 Plan B — 資料庫詳細告警規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 17s
Details 
新增 database_detail_alerts 規則群組:
  PostgreSQL:
    - PostgreSQLSlowQueries: 慢查詢 >60s
    - PostgreSQLDeadlocks: 死鎖發生
    - PostgreSQLTooManyConnections: 連接數 >50
  Redis:
    - RedisKeyEviction: Key 驅逐
    - RedisConnectionsHigh: 連接數 >100
    - RedisCommandLatencyHigh: 命令延遲 >10ms

前置: postgres-exporter:9187 + redis-exporter:9121 已在 188 部署 ✅
Prometheus scrape 已更新 ✅

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:19:03 +08:00
OG T
170ce2f11d fix(ci): 修正測試與 Sprint 5.2 部署腳本
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
Details 
tests/test_auto_repair_service.py:
  - 更新 3個測試符合 2026-04-07 統帥指令移除門檻
  - APPROVED Playbook 直接通過 (低相似度/低品質/高風險均通過)

tests/test_phase22_nemotron_collab.py:
  - 更新 log key: nemotron_collaboration_failed → exhausted

ops/monitoring/docker-compose.exporters.yaml:
  - 修正 postgres DSN: awoooi:awoooi_prod_2026@localhost:5432/awoooi_prod

Sprint 5.2 新增腳本:
  - scripts/sprint51_e2e_validation.py: L7 E2E 驗收腳本 (T1-T5)
  - scripts/ops/deploy-docker-health-monitor.sh: Plan A 一鍵部署腳本

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 18:17:48 +08:00
OG T
0847fa3a60 feat(sprint5.1): L2-2 — alerts-unified.yml 補 DockerContainerUnhealthy/Exited 規則
Some checks failed
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Failing after 19s
Details 
新增 docker_health_alerts group：
  - DockerContainerUnhealthy: container_health_status==0, for 2m, auto_repair=true
  - DockerContainerExited:    container_running_status==0, for 1m, auto_repair=true

標籤 auto_repair=true 讓 AWOOOI API 進入 Guardrail 決策鏈路，
實際修復動作由 Service Registry 分級（ADR-062）決定，
docker-health-monitor.sh（純感知層）送 webhook 後由此規則補充 Prometheus 路徑。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 16:40:44 +08:00
OG T
dc27f8f811 ops(monitoring): 統一 Prometheus 告警規則 — 40+條含統一 layer 標籤 
修正:
- ClawBotDown → OpenClawDown (舊命名廢棄)
- 加入 SentryDown/HarborDown/GiteaDown/AlertmanagerDown
- 所有規則補齊 layer/component/host/auto_repair 統一標籤
- 整合 k8s/monitoring/*.yaml → ops/monitoring/alerts-unified.yml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 02:26:18 +08:00
OG T
3e4612f259 docs(observability): ADR-053 SigNoz 統一架構 + Phase O 驗收
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 36s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- 新增 ADR-053: 可觀測性統一架構決策記錄
- 更新 service-registry.yaml: 補齊 MinIO/Kali 監控入口
- 更新 LOGBOOK: Phase O 完成狀態

Phase O 驗收清單:
✅ kubectl Mac 本機免密碼
✅ OTEL Collector 2 Pod Running
✅ Event Exporter 1 Pod Running
✅ Descheduler CronJob Completed
✅ MinIO + Kali 告警規則
✅ Alert Chain Smoke Test
✅ CD Pipeline 整合
⏳ ClickHouse TTL / remote_write / SigNoz rules (待 .188 手動)

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-04-02 18:26:57 +08:00
OG T
c7f9c119e7 fix(cd): 補提交 ops/monitoring 腳本 
遺漏文件導致 CD Monitoring Coverage 步驟失敗

新增:
- generate_monitoring.py - 監控覆蓋率檢查
- coverage_report.py - 覆蓋率報告
- discover_docker.py - Docker 服務發現
- deploy-exporters.sh - Exporter 部署腳本
- postgres-exporter-queries.yaml - PostgreSQL 查詢配置

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:45:42 +08:00
OG T
12e49d844a feat(monitoring): ADR-037 Wave B - Database Exporters + Prometheus 整合 
- 部署 PostgreSQL Exporter (192.168.0.188:9187)
- 部署 Redis Exporter (192.168.0.188:9121)
- 更新 Prometheus scrape config
- 首席架構師審查: 97% OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:18:54 +08:00
OG T
93c3280481 feat(monitoring): Phase 20 Nemotron 完整監控整合 
服務註冊表:
- 新增 nvidia-nemotron AI 服務
- 3 個 Prometheus metrics 定義
- 4 個告警規則 (circuit_breaker, timeout, error_rate, rate_limit)
- fallback 策略 (nvidia → gemini → ollama)

Alertmanager 規則:
- NvidiaCircuitBreakerOpen (P1)
- NvidiaToolCallingHighLatency (P2)
- NvidiaToolCallingHighErrorRate (P0)
- NvidiaCircuitBreakerHalfOpen (Info)
- NvidiaCircuitBreakerClosed (Info)
- NvidiaNoRequests (P3)

自動修復:
- fallback_to_gemini
- fallback_to_ollama
- switch_model

ADR: ADR-036

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:05:59 +08:00
OG T
40163a51b5 feat(monitoring): 完整監控策略與自動整合架構 
新增:
1. MONITORING_COMPLETE_STRATEGY.md - 完整監控策略
   - 5 主機 × 60+ 服務監控矩陣
   - P0/P1/P2 告警規則清單
   - AI 自動修復閉環流程
   - 安全護欄配置

2. MONITORING_INTEGRATION_ARCHITECTURE.md - 自動整合架構
   - 服務註冊表 (Single Source of Truth)
   - CI/CD 自動驗證監控覆蓋率
   - 新服務自動獲得監控

3. ops/monitoring/service-registry.yaml - 服務清單
   - K8s 工作負載 (API/Web/Worker/ArgoCD)
   - Docker 容器 (Ollama/OpenClaw/Redis/Postgres)
   - 前端頁面 SLO
   - API 端點 SLO
   - 告警模板與自動修復動作

4. ops/monitoring/validate_coverage.py - 覆蓋率驗證
   - CI 階段執行
   - 檢測未監控服務
   - 生成覆蓋率報告

設計原則:
- 監控即代碼 (Monitoring as Code)
- 新服務必須在 registry 註冊才能部署
- 自動發現機制防止遺漏

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:52:08 +08:00