wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
656 Commits 194 Branches 2 Tags
dev
Commit Graph

70 Commits

Author SHA1 Message Date
OG T
3ecfe7b3f5 chore: 清理 NemoNodeAnimation 殘留 + 修復 Migration YAML
Some checks failed
E2E Health Check / e2e-health (push) Successful in 19s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 移除 nemo-node-animation.tsx (無人引用，已被 NemoClaw 取代)
- Migration YAML: 修復 $$ 在 YAML heredoc 被 shell 解析問題
  改用單引號字串 DO '' 語法

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 09:09:25 +08:00
OG T
d8be78b135 feat(api): Knowledge Base Phase 1 後端四層架構
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 7m0s
Details
E2E Health Check / e2e-health (push) Successful in 17s
Details
Type Sync Check / check-type-sync (push) Failing after 30s
Details 
- models/knowledge.py: Pydantic Schema (EntryType/Source/Status/CRUD)
- db/models.py: KnowledgeEntryRecord ORM (PostgreSQL)
- repositories/interfaces.py: IKnowledgeRepository Protocol
- repositories/knowledge_repository.py: PostgreSQL CRUD 實作
- services/knowledge_service.py: 業務邏輯 (get_db_context 內部管理 session)
- api/v1/knowledge.py: REST Router (get_knowledge_service，無直接 DB 存取)
- main.py: 掛載 Knowledge Base Router
- k8s/jobs/migrate-knowledge-entries.yaml: DB Migration Job

API 端點: GET/POST / | GET/PATCH/DELETE /{id} | POST /{id}/approve
         GET /search | GET /categories

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 00:55:56 +08:00
OG T
9cf73bda4f feat(llmops): 啟用 Langfuse LLMOps 追蹤 + CD 自動注入 Keys
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m6s
Details
E2E Health Check / e2e-health (push) Successful in 18s
Details 
- 04-configmap.yaml: LANGFUSE_ENABLED=true (Phase 15.1 Key 已在 K8s Secret)
- cd.yaml: 補齊 Langfuse keys CD 自動注入 (LANGFUSE_PUBLIC/SECRET_KEY)
- LOGBOOK.md: ClawBot → OpenClaw 命名修正
- .gitignore: 加入 tsconfig.tsbuildinfo + .superpowers/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 22:19:22 +08:00
OG T
dc9a07bf20 fix(k8s): NetworkPolicy 修正 OpenClaw port 8089→8088 (clawbot-v5 使用 8088)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
2026-04-01 ogt: OpenClaw 從 openclaw container (8089) 遷移到 clawbot (8088)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 22:17:09 +08:00
OG T
5809d3e336 feat(ai): 委派 Incident RCA 給 OpenClaw (Nemo) — 架構鐵律修正
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
架構鐵律: OpenClaw = AI 大腦，AWOOOI API 透過 HTTP 委派仲裁
修改:
- openclaw.py: 加入 _call_openclaw_analyze()，在 LLM fallback 前先呼叫 OpenClaw
- 04-configmap.yaml: OPENCLAW_URL 修正為 :8088 (新容器 port)
- AI_FALLBACK_ORDER 改為 ["ollama","claude"] (移除 Gemini 付費 API)

OpenClaw /api/v1/analyze/incident → qwen2.5:7b 本機 Ollama (Nemo)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 21:11:30 +08:00
OG T
bfee94bb6f fix(ai): ADR-044 修正 AI_FALLBACK_ORDER — Gemini 優先做 RCA 仲裁
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m49s
Details
E2E Health Check / e2e-health (push) Successful in 21s
Details 
問題連鎖:
1. NVIDIA nemotron-mini-4b JSON 截斷 → proposal_parse_failed
2. Ollama K8s 並發請求 → GPU 排隊逾時 (90s)
3. Expert System fallback → 信心 0%

ADR-044: OpenClaw RCA 應用 Ollama/Gemini，非 NVIDIA Tool Calling 模型
修正: gemini > ollama > nvidia > claude

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 20:35:42 +08:00
OG T
555e808f39 fix(ai): ollama 優先於 nvidia — 修復 nemotron-mini-4b JSON 截斷導致 0% 信心
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m42s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
根本原因: nemotron-mini-4b-instruct 輸出 JSON 被截斷 (raw_response={"confidence": )
→ proposal_parse_failed → fallback Expert System → AI 仲裁 0% 信心

修復: AI_FALLBACK_ORDER 改為 ollama 優先，NVIDIA 降為第二
(Ollama qwen2.5:7b-instruct 在 192.168.0.188:11434 輸出品質穩定)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 19:22:15 +08:00
OG T
5b938887c0 fix(telegram): 關閉 K8s prod TELEGRAM_ENABLE_POLLING 解決 409 Conflict
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m48s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
AWOOOI API 與 OpenClaw(192.168.0.188) 同時 Long Polling 造成 409 Conflict，
導致 AI 仲裁降級為規則匹配(0%信心)。

架構原則: OpenClaw 是唯一 Telegram Gateway，K8s 只發送訊息。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 18:09:47 +08:00
OG T
71a4e0f8c8 fix(k8s): 修復 dev RBAC RoleBinding 欄位名稱錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 6m54s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details
CD Pipeline (Dev) / build-and-deploy-dev (push) Failing after 3m53s
Details 
apiRef → name (正確 Kubernetes 欄位名稱)
防止 RoleBinding 建立失敗

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:27:12 +08:00
OG T
9913f5dc6d feat(infra): 開發環境分離 + BuildKit cache 修復 + circuit breaker 優化
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 6m52s
Details
E2E Health Check / e2e-health (push) Successful in 17s
Details
CD Pipeline (Dev) / build-and-deploy-dev (push) Failing after 9s
Details 
1. k8s/awoooi-dev/: 新建 dev namespace (01-05 配置)
   - Namespace + ResourceQuota (cpu 2/4, mem 4Gi/8Gi)
   - ConfigMap: ENVIRONMENT=dev, LOG_LEVEL=DEBUG, SHADOW_MODE=false
   - Deployment: 1 replica, NodePort 32344, image dev-latest
   - RBAC: awoooi-executor-dev ServiceAccount

2. .gitea/workflows/cd-dev.yaml: dev branch CD pipeline
   - 觸發: dev branch push
   - Build: --no-cache (防 cache poisoning)
   - Tag: dev-{sha} / dev-latest
   - Deploy: awoooi-dev namespace, health check 32344
   - Telegram: [DEV] 前綴通知

3. apps/api/Dockerfile: ARG CACHE_BUST=none (防 BuildKit cache 毒化)
   - deps 層 (pip install) 仍可 cache
   - src/ 和 models.json 層每次重建

4. .gitea/workflows/cd.yaml: 正式環境 API build 加入 CACHE_BUST=git_sha
   - 確保 models.json 等配置變更正確進入 image

5. apps/api/src/services/nvidia_provider.py: timeout 不計入 circuit breaker
   - TimeoutException → 只 log，不 record_failure()
   - 只有硬性錯誤 (auth/rate limit/exception) 才斷路

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:22:21 +08:00
OG T
419dc2f8e0 fix(nvidia): timeout 60s→30s，NVIDIA 第一保免費，失敗轉 Gemini
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 5m46s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- nvidia_provider.py: NVIDIA_TIMEOUT 60→30s
- models.json: timeout_seconds 60→30s
- configmap: NEMOTRON_TIMEOUT_SECONDS 45→30s, fallback 恢復 nvidia 第一
目標: Nemo 有足夠時間回應(free)，失敗快速轉 Gemini(備援)，整體機制可運作

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:05:19 +08:00
OG T
4c622813af fix(auto-repair): 實際可用的自動修復門檻 (Phase 22 P1)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
問題: 四道鎖全卡死導致自動修復永遠不觸發
1. configmap: Gemini 排第一 (100ms vs NVIDIA 60s timeout)
2. auto_approve: confidence 0.90→0.65, trust 5→1, playbook 3→1
3. auto_approve: 開放 medium 風險, require_playbook=False

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:02:16 +08:00
OG T
eccf61fbc9 fix(ai): 修復假信心度 + 解除 Shadow Mode (Phase 22 P1)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
1. openclaw.py: LLM 截斷時 confidence 0.82→0.0 (禁止偽造信心度)
2. prompts.py: NEMOTRON schema 範例值改用佔位符，防模型照抄 0.75
3. configmap: SHADOW_MODE_ENABLED=false，開放 low 風險自動執行
   條件門檻: confidence≥90% + trust_score≥5 + playbook_success≥95%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 15:59:42 +08:00
OG T
cc6b18e3bc fix(phase22): 修復 Telegram 對話三個 Bug (ADR-044)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 18s
Details 
P0: security_interceptor.py 新增 intercept_telegram() 方法
- 修復 _handle_chat_message 的 AttributeError (致命 Bug)
- 白名單驗證，不需要 Nonce (對話訊息 vs 按鈕回調)

P1: nvidia_provider.py chat() 新增 use_json_mode 參數
- 對話場景預設 False (自然語言回應)
- RCA/分析場景傳入 True (結構化 JSON 輸出)
- openclaw.py RCA 呼叫加上 use_json_mode=True

P2: K8s ConfigMap 啟用 TELEGRAM_ENABLE_POLLING=true
- K8s AWOOOI API 接管 @tsenyangbot Long Polling
- OpenClaw (188) 停止 Telegram，改為純 REST 服務

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-31 21:53:09 +08:00
OG T
6da240219d feat(k8s): Phase 22 啟用 OpenClaw + Nemotron 協作 (ADR-044)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 17s
Details 
- 新增 ENABLE_NEMOTRON_COLLABORATION=true
- 新增 NEMOTRON_TIMEOUT_SECONDS=45
- 新增 NEMOTRON_ASYNC_UPDATE=true
- 統帥需求: @tsenyangbot 同時顯示 OpenClaw 仲裁 + Nemotron 執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-31 21:39:14 +08:00
OG T
219525f64f fix(k8s): #33 Sentry + OTEL 修復
All checks were successful
E2E Health Check / e2e-health (push) Successful in 17s
Details 
修復項目:
1. SENTRY_DSN: K8s secret 已修補 (空值 → 正確 DSN)
2. OTEL_EXPORTER_OTLP_ENDPOINT: 24318 → 24317 (gRPC 端口)

驗證結果:
- sentry_initialized ✅
- OTEL export 無錯誤 ✅
- Session Replay 配置 ✅

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 12:18:06 +08:00
OG T
723e8ef251 feat(api): Phase 21.3 Weekly Report (ADR-041)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- 新增 WeeklyReportMessage dataclass (telegram_gateway.py)
- 新增 WeeklyReportService (整合 StatsService + K3sMonitor)
- 新增 CronJob (每週五 18:00 台北)
- 新增 API 端點 (/stats/weekly/preview, /stats/weekly/report)

Phase 21 定期報告機制全部完成！
- 21.1 Daily E2E Schedule ✅
- 21.2 K3s Telegram Report ✅
- 21.3 Weekly Report ✅

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 11:28:46 +08:00
OG T
ce6b1b1c64 docs: 更新 LOGBOOK - #17 i18n Hydration 完成 
前端 P1 改進全部完成:
- #15 SSE + 樂觀更新 (8c8664c)
- #16 DOM Bypass (0b87018)
- #17 i18n Hydration (f25e94e)

首席架構師審查: 96/100 OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 11:23:38 +08:00
OG T
fbc670036d fix(ai): NVIDIA Nemotron 優先仲裁 
2026-03-30 ogt: AI Fallback 順序修正
- ["nvidia","gemini","ollama","claude"]
- Nemotron Tool Calling 83.3% 精準度
- 修復 Gemini 仍為首位問題

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-30 01:12:28 +08:00
OG T
89f0bae3f2 feat(safety-net): complete wave 1 atomicity (adr-038, adr-039, debounce, graceful degrade, xclaim)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
2026-03-29 23:55:38 +08:00
OG T
79134fb019 feat(ai): 新增 NVIDIA Nemotron 到告警 Fallback Chain 
- 新增 _call_nvidia() 一般告警支援 (非 Tool Calling)
- Fallback 順序: Gemini → Nvidia → Ollama → Claude
- Nvidia 免費 tier ($0)，含 Token 追蹤

解決: Gemini 超限 (500/500) 後無法 fallback 問題

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 20:28:24 +08:00
OG T
b97f9364fb feat(k8s): add Worker HPA + fix non-AI confidence values 
Wave 2 Deployment:
- Worker HPA: min:1 max:3, CPU 70%, Memory 80%
- 前置條件: XCLAIM + terminationGracePeriodSeconds:90 (Wave 1 ✅)
- 比 API/Web 更保守的擴縮策略 (120s up, 600s down)

Confidence Fix:
- 非 AI 分析來源 (fallback/playbook/historical/consensus) 設 confidence=0.0
- 避免混淆 AI 信心度與其他指標 (成功率/相似度)
- 涉及: github_webhook, decision_manager, intent_classifier, learning_service

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:09:37 +08:00
OG T
a5a6bd3408 feat(monitoring): K8s alert rules + Grafana dashboards + ops 腳本 
- k8s/monitoring/alert-chain-monitor.yaml
- k8s/monitoring/database-alerts.yaml
- ops/grafana/ Grafana dashboards
- ops/signoz/ SignOz 配置
- ops/scripts/ 維運腳本

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:04:14 +08:00
OG T
237fb64a81 chore(k8s): secrets template + web deployment 更新 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:03:47 +08:00
OG T
39396dc57a feat(worker): Wave 1 Signal Worker XCLAIM + Graceful Shutdown 
ADR-038/039 Wave 1 強化:
- 新增 Active Sweeper: XPENDING + XCLAIM 回收閒置訊息
- PENDING_IDLE_MS: 60秒無ACK則可被回收
- SWEEP_INTERVAL_S: 每30秒掃描一次
- Graceful Shutdown: 75秒超時 (搭配 K8s 90秒)
- 超過 MAX_RETRIES 的訊息強制 ACK

K8s Worker Deployment:
- 新增 terminationGracePeriodSeconds: 90

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:53:05 +08:00
OG T
12e49d844a feat(monitoring): ADR-037 Wave B - Database Exporters + Prometheus 整合 
- 部署 PostgreSQL Exporter (192.168.0.188:9187)
- 部署 Redis Exporter (192.168.0.188:9121)
- 更新 Prometheus scrape config
- 首席架構師審查: 97% OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:18:54 +08:00
OG T
93c3280481 feat(monitoring): Phase 20 Nemotron 完整監控整合 
服務註冊表:
- 新增 nvidia-nemotron AI 服務
- 3 個 Prometheus metrics 定義
- 4 個告警規則 (circuit_breaker, timeout, error_rate, rate_limit)
- fallback 策略 (nvidia → gemini → ollama)

Alertmanager 規則:
- NvidiaCircuitBreakerOpen (P1)
- NvidiaToolCallingHighLatency (P2)
- NvidiaToolCallingHighErrorRate (P0)
- NvidiaCircuitBreakerHalfOpen (Info)
- NvidiaCircuitBreakerClosed (Info)
- NvidiaNoRequests (P3)

自動修復:
- fallback_to_gemini
- fallback_to_ollama
- switch_model

ADR: ADR-036

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:05:59 +08:00
OG T
179e659f14 chore: 清理 Playwright 產物 + kube-state-metrics 告警擴充 
清理工作:
- .gitignore 新增 playwright-report/ 和 test-results/ 排除
- 保留 phase19/ 參考截圖目錄

kube-state-metrics 告警擴充 (P3):
- CronJobLastRunFailed: Job 執行失敗
- DaemonSetMissingPods: DaemonSet 缺少 Pod
- StatefulSetReplicasMismatch: StatefulSet 副本不足
- ContainerWaiting: ImagePullBackOff/CrashLoopBackOff 偵測
- PDBViolation: PDB 健康 Pod 數不足
- NodeUnschedulable: 節點標記為不可排程

新增:
- apps/api/scripts/test_nemotron_tool_calling.py (E2E 比較測試)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:28:35 +08:00
OG T
725392b578 fix(k8s): NetworkPolicy 繞過 kustomize commonLabels 
問題: kustomize commonLabels 會加到 NetworkPolicy egress[].to[].podSelector
      導致 DNS rule 要求 CoreDNS pods 有 system:awoooi + environment:prod
      但 CoreDNS 只有 k8s-app:kube-dns，造成 DNS 解析失敗

修復:
- kustomization.yaml: 移除 02-network-policy.yaml
- cd.yaml: 新增 Apply NetworkPolicy step 單獨套用

2026-03-29 ogt: 根本原因修復

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:27:29 +08:00
OG T
ee2bceefff feat(monitoring): Phase 19.6 測試文檔 + P1-P3 改進 + 首席架構師審查 
Phase 19.6 測試文檔收尾:
- E2E 測試擴充至 18 項 (Terminal/GenUI 驗證)
- 新增 PHASE19-VERIFICATION-CHECKLIST.md (完整驗證清單)

P1 驗證:
- ArgoCD Metrics NodePort 監控 (30883/30884)
- TLS 證書監控 (Blackbox Exporter 9115)

P2 改進:
- waitForTimeout → waitForLoadState('networkidle')
- 跨平台快捷鍵 (Meta+J / Control+J)
- SKIP_MULTISIG_TESTS 環境變數控制
- Prometheus GitOps 部署腳本

P3 改進:
- HPA maxReplicas 4 → 6 (API/Web)

首席架構師審查: 47/50 OUTSTANDING (94%)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:19:26 +08:00
OG T
dc7daf5d81 docs(monitoring): 更新 ArgoCD Metrics 端點文檔 
- ArgoCD Server Pod 運行在 mon1 (192.168.0.121)
- 更新 Prometheus target 為 192.168.0.121:30883
- 標記配置已部署並驗證

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:59:46 +08:00
OG T
e75e578547 feat(monitoring): P1/P2 改進 - ArgoCD Metrics + TLS 證書告警 
## P1: ArgoCD Metrics
- 新增 ArgoCD Metrics NodePort (30882, 30883)
- 更新 NetworkPolicy 允許 Prometheus (188) 抓取
- 提供 Prometheus scrape config 範本

## P1: NetworkPolicy AI API
- 文檔標註 K8s NetworkPolicy 不支援 FQDN 限制
- 維持現有配置避免 AI 功能中斷

## P2: TLS 證書告警
- 新增 TLSCertExpiringIn30Days (30天預警)
- 新增 TLSCertExpiringIn7Days (7天緊急)
- 新增 TLSCertExpired (已過期)
- 新增 TLSProbeFailure (探測失敗)

## P2: Multi-Sig E2E 測試
- 標記為條件式執行 (API 不可用時自動跳過)
- 避免 CI/CD 因無法連接生產 API 而失敗

首席架構師審查: 2026-03-29 (台北時間)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:48:57 +08:00
OG T
a30f766eb1 feat(monitoring): 首席架構師完整審查 + 補充告警規則 
## 首席架構師審查結果: 198/200 (99%) EXCEPTIONAL

### 審查範圍
- 架構設計: 50/50 ⭐
- 安全性: 49/50
- 模組化合規: 50/50 ⭐
- 監控告警: 49/50
- E2E 測試: 49/50

### 新增補充告警 (12 條)
- RedisDown, PostgreSQLDown, OllamaDown, OpenClawDown
- HarborDown, LangfuseDown
- HPAMaxedOut, HPAScalingDisabled
- WorkerUnavailable
- NodeHighCPU, NodeHighMemory, ContainerOOMKilled

### 檔案
- k8s/monitoring/k3s-alerts-supplemental.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:30:44 +08:00
OG T
f0572ae906 feat(k4.3): Pod Security Standards + Grafana Dashboard 
K4.3 Pod Security Standards:
- awoooi-prod: baseline
- kube-state-metrics: baseline
- kured: privileged (hostPID required)
- descheduler: restricted
- velero: baseline
- argocd: baseline

Grafana Dashboard:
- K3s Cluster Overview (9 panels)
- Nodes, Pods, HPA, Velero, Alerts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:16:54 +08:00
OG T
bcbb386ee4 fix(kured): 修復 CrashLoopBackOff - 新增 ds-namespace/ds-name 參數 
問題: Kured 預設在 kube-system 尋找 DaemonSet
修復: 新增 --ds-namespace=kured --ds-name=kured

驗證: 2/2 pods Running

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 22:53:21 +08:00
OG T
0b68352fc2 feat(k3s): P2/P3 改進 - kube-state-metrics + Kured 時區修復 + Descheduler 調整 
P2 改進:
- 新增 kube-state-metrics v2.10.1 (NodePort:30888)
- 新增 7 條 kube-state-metrics 告警規則 (NPD 整合)

P3 改進:
- 修復 Kured 維護窗口時區 (18:00→02:00 台北時間)
- Descheduler threshold 20%→30% (避免過度遷移)

首席架構師審查建議執行項目

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 22:23:42 +08:00
OG T
984d31de0c feat(ai): Gemini 優先 + Token/Cost 追蹤 (2026-03-29 ogt) 
變更:
1. ConfigMap: Gemini 優先 ["gemini","ollama","claude"]
2. openclaw.py: 捕獲 Gemini usageMetadata (tokens/cost)
3. webhooks.py: 傳遞 ai_tokens/ai_cost 到 Telegram
4. telegram_gateway.py: 顯示 💰 Tokens: X / $Y.YYYY

Gemini 1.5 Flash 定價:
- Input: $0.075/1M tokens
- Output: $0.30/1M tokens

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 22:18:24 +08:00
OG T
541565de48 feat(k4.2): Descheduler for pod rebalancing 
- Deploy Descheduler v0.30.1 as CronJob
- Schedule: Every 2 hours
- Policies enabled:
  - LowNodeUtilization: rebalance when node < 20% usage
  - RemoveDuplicates: spread replicas across nodes
  - RemovePodsViolatingNodeAffinity: enforce affinity rules

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 22:03:54 +08:00
OG T
c6bef20a97 feat(k4.1): Kured automatic node reboot daemon 
- Deploy Kured v1.15.1 as DaemonSet
- Maintenance window: 02:00-04:00 Taipei time
- Reboot period: 1 hour between node reboots
- PDB-aware: checks AWOOOI pods before draining
- Prometheus integration for metrics

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 22:03:05 +08:00
OG T
f010f42795 feat(k3): HPA for AWOOOI API/Web (Phase K3.2) 
- Add HPA for awoooi-api: 2-4 replicas, 70% CPU / 80% memory target
- Add HPA for awoooi-web: 2-4 replicas, 70% CPU / 80% memory target
- Scale-up stabilization: 60s
- Scale-down stabilization: 300s (prevent flapping)

Based on VPA recommendations:
- API target: 100m CPU (current: 16% utilization)
- Web target: 63m CPU (current: 29% utilization)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 21:59:52 +08:00
OG T
1a4be7b18a feat(k-mon): K3s monitoring integration (Phase K-MON) 
- Add Velero metrics NodePort service (30885)
- Add K3s infrastructure alert rules:
  - VIP 6443 availability
  - Node ICMP checks
  - AWOOOI API/Web TCP checks
  - SignOz/Sentry availability
- Add Velero backup alerts (failed/missing)
- Add ADR-034 for ArgoCD GitOps adoption

Deployed to:
- K3s: velero-metrics service
- 188: Prometheus + Alertmanager configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 21:57:57 +08:00
OG T
66fb56c691 feat(k8s): Phase K2 自動化維運完成 
- K2.4 NPD: Node Problem Detector (DaemonSet)
- K2.3 VPA: 3 Vertical Pod Autoscaler (Off 模式)
- K2.1 ArgoCD: v3.3.6 @ :30443 (GitOps)
- K2.2 Sealed Secrets: v0.26.0 (加密 Secrets)

新增檔案:
- k8s/npd/node-problem-detector.yaml
- k8s/awoooi-prod/11-vpa.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 21:27:05 +08:00
OG T
eea6e3acc3 feat(k8s): 新增 Velero 備份系統 (K1.1) 
Phase K1 災難恢復:
- MinIO 部署在 192.168.0.188:9000/9001
- Velero v1.13.0 完整安裝 manifests
- velero-backups bucket 已建立
- README 含部署與使用指南

部署方式:
  ssh wooo@192.168.0.120
  sudo kubectl apply -f k8s/velero/velero-install-full.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 20:53:02 +08:00
OG T
269c81bdbb fix(k8s): OpenClaw 端口統一 8088→8089 
- ConfigMap: OPENCLAW_URL 更新為 8089
- NetworkPolicy: 允許 8089 出站
- SERVICE-ENDPOINTS.md: 移除 legacy 8088 引用

2026-03-28 清理舊配置，統一使用正式端口

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 20:32:30 +08:00
OG T
7b9b0c490b feat(phase19): Omni-Terminal 100% 完成 + 首席架構師審查 47/50 
## Phase 19 Omni-Terminal (Wave 0-6 全部完成)

### 核心功能
- SSE 狀態機 (7-State 設計，10/10 分)
- GenUI 動態渲染 (6 張卡片 + Zod Schema 驗證)
- 核鑰 UX (長按授權 + 風險分級)
- Terminal Telemetry (Sentry 整合)

### P0-P2 修復
- P0: Singleton → FastAPI Depends 依賴注入
- P1: Zod Schema 升級 (7 個驗證 Schema)
- P1: 錯誤分類碼聚合 (Sentry fingerprint)
- P2: Slow Query 監控 (5s 警告 / 10s 嚴重)

### 測試
- test_terminal_service.py: 54 項測試全通過
- 意圖分類: 42 個測試案例 (9 種 IntentType)

### 文檔
- ADR-031: SSE 架構實作紀錄
- ADR-032: GenUI 渲染實作紀錄
- Skills: v1.9 (後端 Terminal 章節)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 18:04:12 +08:00
OG T
3e5315aaf8 docs(k3s): 首席架構師審查完成 46/50 (92%) 
K3s 優化工作審查完成:

- ADR-033: Phase K0 + K-NET 標記為已完成
- 09-pdb.yaml: Worker PDB 設計說明註釋
- DevOps Skill: 新增 keepalived 快速操作參考

審查結果:
- 架構合規性: 9/10
- Runbook 完整性: 10/10 ⭐
- 模組化合規: 9/10
- 風險控制: 9/10
- 文檔完整性: 9/10

P2 問題已修復，無 P0/P1 阻擋項

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 18:00:07 +08:00
OG T
efb80b403e feat(k8s): Phase K0.5 Startup Probe + PDB + revisionHistoryLimit 
K3s 生產級優化 Phase K0 變更:

- 新增 startupProbe 到 API/Web/Worker Deployment (60s 啟動時間)
- 新增 revisionHistoryLimit: 3 (減少孤立 ReplicaSet)
- 新增 09-pdb.yaml (PodDisruptionBudget 保護)
- 新增 K3S-OPTIMIZATION-RUNBOOK.md (執行手冊)
- 修正 selector 對齊現有 Deployment (app+environment+system)

首席架構師審查: 9.0/10 ✅

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 11:13:44 +08:00
OG T
a579710982 fix(k8s): 補齊 Sentry DSN 配置 (首席架構師審查) 
- 03-secrets.example.yaml: 新增 SENTRY_DSN
- 04-configmap.yaml: 新增 Sentry 元數據
- LOGBOOK: 新增 CD Lint 修復記錄

Phase 10 Sentry 整合 - DSN 配置補齊

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-27 14:51:41 +08:00
OG T
190cfda65c revert(k8s): 恢復 commonLabels (Deployment selector immutable) 
還原到 commonLabels，因為:
1. Deployment selector 是 immutable，不能移除 environment/system labels
2. commonLabels 只影響 spec.podSelector，不影響 egress[].to[].podSelector
3. DNS 規則 (k8s-app=kube-dns) 不會被 commonLabels 破壞

DNS 問題的根因是之前的錯誤配置，NetworkPolicy YAML 已修復

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 19:56:41 +08:00
OG T
42bf6a8729 fix(k8s): 修復 NetworkPolicy DNS 被 kustomize commonLabels 破壞問題 
根因: commonLabels 會自動加到 NetworkPolicy 的所有 selector，
導致 DNS egress 規則要求 CoreDNS 有 system/environment labels (它沒有)

修復: 改用 labels + includeSelectors=false，只加 metadata labels
不會影響 NetworkPolicy 的 podSelector/namespaceSelector

- 2026-03-27 (台北時間) DNS 解析失敗 RCA
- Telegram Bot 無法連線是因為這個問題

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 19:44:22 +08:00