wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
1,212 Commits 195 Branches 2 Tags
f2c18c4e63b7e130949e2b153e33b28bb75c082b
Commit Graph

1212 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
OG T
f2c18c4e63 feat(D1): models.json 集中化 — ADR-067 五大 Ollama 應用 hardcode 消除
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m56s
Details 
- models.json v1.3.0: providers.ollama.models 新增 9 個 purpose keys
  (drift_summary/drift_intent/log_anomaly/nemoclaw/playbook_draft/
   code_review/embedding/rag_generate/image_analysis)
- drift_narrator_service: NARRATOR_MODEL → get_model("ollama","drift_summary")
- drift_interpreter: MODEL → get_model("ollama","drift_intent")
- log_summary_service: SUMMARY_MODEL → get_model("ollama","log_anomaly")
- local_code_review_service: _MODEL_OLLAMA → get_model("ollama","code_review")
- image_analysis_service: _MODEL → get_model("ollama","image_analysis")
- decision_manager: nemoclaw + playbook_draft 兩處 → get_model()
- embedding_service: get_embedding_service() factory → get_model("ollama","embedding")
- knowledge_service: OllamaEmbeddingService(model=...) → get_model()

所有模型名稱現在統一由 models.json 管理，修改模型只需改一個檔案。
LOGBOOK 更新：D1 完成 + B2 已完成確認

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:45:53 +08:00
AWOOOI CD
694471891f chore(cd): deploy 82e1c05 [skip ci] 2026-04-11 12:45:05 +00:00
OG T
82e1c05df8 fix(review): Code Review C1/C2/I2/M2 修補
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
C1 drift_interpreter: 寫死 192.168.0.111 → settings.OLLAMA_URL
  違反 feedback_frontend_internal_ip_ban 鐵律（後端 service 層同樣禁止寫死內網 IP）

C2 km_conversion_service: BUG-004 補同步 Redis Working Memory vectorized 欄位
  原修復只更新 DB，Redis incident:{id} JSON 的 vectorized 未同步
  → 審計查 Redis 仍顯示 False，fly-wheel 閉環指標仍不準
  修復：DB 更新後 GET → JSON patch vectorized=True → SET（保留原 TTL）

I2 decision_manager: _ALERTNAME_KEYWORDS HostHighDiskUsage→HostOutOfDiskSpace
  + 補 DockerContainerExited
  + fallback 路徑加 debug log

M2 decision_manager: import json as _json 從 for 迴圈移至方法頂部

docs: ADR-072 新增 Code Review 發現與技術債記錄

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:36:59 +08:00
OG T
e447f97616 fix(telegram): 接通 classify_notification + 修復 HostBackupFailed 亂送按鈕 
三個問題同時修復：

1. classify_notification() 死程式碼接通
   - _push_decision_to_telegram() 現在先呼叫 classify_notification()
   - TYPE-1 (純資訊) → send_info_notification()，無按鈕
   - TYPE-4D (Config Drift) → send_drift_card()
   - 其餘 TYPE-2/3/4 → send_approval_card()（原有按鈕）
   - decision_state + auto_executed 從呼叫端注入 proposal_data

2. alert_rules.yaml 補 host_backup_failed 規則
   - HostBackupFailed / VeleroBackupFailed / VeleroBackupNotRun → NO_ACTION
   - 不再走 generic_fallback → 不再產生 kubectl rollout restart deployment/backup

3. _verify_k8s_deployment_exists() 主機層告警不再保守放行
   - Host*/Docker*/Backup*/Velero*/SSH* 前綴告警 → K8s MCP 不可用時 return False
   - _auto_execute() 收到 NO_ACTION 或空 kubectl_command → 早退，不執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:35:48 +08:00
OG T
9382814d14 docs(adr-072): 全部完成 BUG-001~008 
ADR-072 狀態更新為「全部修復完成」
BUG-007 確認不需修（alerts-unified.yml 全 42 規則均有 severity）
BUG-008 已修復（f34fe19）
LOGBOOK 新增 P2 完成條目 + 下一步說明

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:30:29 +08:00
OG T
f34fe19134 fix(aiops): ADR-072 BUG-008 alertname_to_type 9→56 筆
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
從 9 筆靜態 map 擴充至完整涵蓋 alerts-unified.yml 全 42 個 alertname：
- host_alerts: HostDown/HostHighCpuLoad/HostOutOfMemory/HostOutOfDiskSpace/HostBackupFailed
- k8s: K3sNodeNotReady/KubePodCrashLooping/KubeDeploymentReplicasMismatch/Velero* (8筆)
- database: PostgreSQL*/Redis* (10 筆)
- service_alerts: *Down (8 筆)
- external: *Down/SSLExpiring (5 筆)
- alert_chain: AlertChainBroken*/NoAlerts/Unhealthy (4 筆)
- docker_health: DockerContainerUnhealthy/Exited (2 筆)
- auto_repair: AutoRepairLowSuccessRate/PermanentFixRequired (2 筆)
- 舊版相容: HighCPUUsage/HighMemoryUsage/DiskSpaceLow/SSLCertExpiringSoon/TargetDown

預期效果: 69/112 incidents "custom" → 大幅降低，HostHighCpuLoad → "host_cpu"

BUG-007 確認不需修: alerts-unified.yml 全 42 規則均已有 severity label

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:29:34 +08:00
OG T
85c71bf73c docs(adr-072): 更新 Bug 修復狀態 + LOGBOOK 
ADR-072: BUG-001~006 標記已修復 (P0 commit 88e3197, P1 commit 5aa0244)
LOGBOOK: 新增 ADR-072 P0+P1 全修復條目
P2 待修: BUG-007 severity labels + BUG-008 alertname_to_type

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:26:27 +08:00
OG T
5aa0244c9a fix(aiops): ADR-072 P1 Bug 修復 — BUG-004/005/006
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
BUG-004 KM vectorization 108/112 = False:
  km_conversion_service: KM entry 建立後（embedding 已背景觸發），
  補寫 incidents.vectorized = True，飛輪閉環（ADR-068）學習指標正常

BUG-005 15 ready decisions 無人審核:
  decision_manager: 新增 resend_stale_ready_tokens()，
  掃描 Redis decision:* key，找出 state=ready 且 dedup_key 過期的 token，
  重新推送 Telegram 審核卡片
  main.py: lifespan startup 排程 resend_stale_ready_tokens()（asyncio.create_task 非阻塞）

BUG-006 outcome/verification_result 全 null:
  _push_auto_repair_result: Telegram 推送前先寫入
  incidents.outcome + incidents.verification_result 到 DB

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:24:41 +08:00
OG T
2185e1755c fix(aiops): ADR-072 P0 Bug 修復 — BUG-001/002/003
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
BUG-001 drift_interpreter: nvidia_provider 已重構為 NvidiaProviderResult 物件（非 4-tuple）
  → 改用 Ollama httpx 直接呼叫 qwen2.5:7b-instruct，繞過 nvidia_provider
  → 消除所有 K8s config drift 告警的 "too many values to unpack" 永久失敗

BUG-002 deployment_name="unknown": 主機層告警(HostHighCpuLoad 等)無 component/job/pod label
  → _auto_execute() 新增 _resolve_target_from_k8s() 補救
  → K8s MCP kubectl get pods 動態查詢受影響 Pod，去掉 hash suffix 得到 deployment name

BUG-003 無效 deployment 通過 safety guard:
  → _auto_execute() safety guard 通過後加入 _verify_k8s_deployment_exists() 存在性確認
  → K8s 中找不到 deployment/pod → 拒絕執行，寫入 DecisionToken.error
  → K8s MCP 不可用時保守放行（不阻塞主流程）

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:20:39 +08:00
AWOOOI CD
2ad2a7ba45 chore(cd): deploy f323633 [skip ci] 2026-04-11 12:18:44 +00:00
OG T
f3236338a5 fix(security): Code Review P0+P1+P2 全修補 — MCP Phase 2b-3 + decision_manager
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P0: decision_manager _fetch_metrics_snapshot 參數型別錯誤
  - prom._instant_query(str) → prom._instant_query({"query": str})
  - 結果解析 r.get("status")=="success" → r.get("result", [])

P1: prometheus_provider — alertname PromQL injection 防範
  - 新增 _RE_SAFE_ALERTNAME 白名單正則

P1: decision_manager — kubectl action 危險字元注入防範
  - 新增 _ALLOWED_KUBECTL_PATTERN 白名單，非法指令格式直接拒絕

P1: decision_manager — 6 個 asyncio.create_task() GC 風險
  - 新增 _background_tasks: set + _fire_and_forget() helper
  - 所有 bare create_task 改用 _fire_and_forget

P1: ssh_provider — Group B 寫入工具強制需要 known_hosts
  - known_hosts 未設定或檔案不存在時拒絕執行，防 MITM

P2: sentry_provider — query 語意白名單驗證
  - 新增 _RE_SAFE_SENTRY_QUERY，拒絕含特殊字元的 query

P2: argocd_provider — verify=False 改為 ARGOCD_VERIFY_TLS 環境變數開關
  - 新增 _tls_verify() helper，預設 false（self-signed cert）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:10:33 +08:00
OG T
083b1a5449 fix(cd): 修復 gitea remote 設定邏輯 — remove+add 取代 add||set-url
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
原始 `add 2>/dev/null || set-url` 邏輯：當 remote 不存在時 set-url 也失敗
新邏輯：先強制 remove（允許失敗），再 add，確保 remote 一定存在

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:07:54 +08:00
OG T
09982fdfaa docs(session6): Telegram 全面審計 + ADR-072 Bug 清單 + 規格整合 
- LOGBOOK: Session 6 Redis DB10 審計結果（8個系統性問題，P0-P2分級）
- ADR-072: AIOps 閉環 Bug 修復清單（drift_interpreter/deployment_name/KM vectorization等）
- 規格文件 v2.2: 確認 Sprint A/B/C + MCP 1-4 + ADR-071 全部完成，標記下一步為 ADR-072

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:04:50 +08:00
OG T
a1432c03ed docs: ADR-070/071 + ssh-mcp-setup runbook + Skill-04 v2.7 
- ADR-070: 全自動 AIOps 閉環 MCP Phase 1-4 決策文件
- ADR-071: 告警通知四類型 + KM 三段資料閉環決策文件
- docs/runbooks/ssh-mcp-setup.md: SSH MCP 建立/驗證/輪換 SOP
- Skill-04: v2.7 新增 Sprint C DR + ADR-070 MCP 10 providers 完整記錄

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:04:47 +08:00
OG T
0f46799d56 docs(logbook): MCP 全驗收完成 + Sentry/Prometheus bug 修復記錄 2026-04-11 19:54:05 +08:00
OG T
b5aa607a30 fix(mcp): 修正 Prometheus URL (110:9090) + Sentry DSN 改 HTTP 內網
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m45s
Details 
- PROMETHEUS_URL: 188:9090 → 110:9090 (Prometheus server 正確位置)
- SENTRY_DSN: https://sentry.wooo.work → http://192.168.0.110:9000 (消除 SSL hostname mismatch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 19:51:20 +08:00
OG T
a6e6f389e2 chore: 清理觸發 CD 的臨時注釋
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m9s
Details
2026-04-11 19:15:04 +08:00
OG T
40d6536b62 ci: 觸發 CD — MCP Phase 3/4 + SSH MCP 完整啟用 (providers注釋更新)
Some checks are pending
CD Pipeline / build-and-deploy (push) Waiting to run
Details
2026-04-11 19:14:17 +08:00
OG T
a0d0d66809 ci: 觸發 CD 2026-04-11 19:14:17 +08:00
OG T
5c2cdff37f ci: 觸發 CD — MCP Phase 3/4 + SSH MCP 啟用 2026-04-11 19:13:46 +08:00
OG T
95b61802be fix(mcp): ssh-mcp-key volumeMount 路徑修正 — subPath 對齊 ssh_provider.py
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 7m45s
Details 
- ssh_mcp_key → /run/secrets/ssh_mcp_key (SSH_KEY_PATH)
- known_hosts → /etc/ssh-mcp/known_hosts (SSH_MCP_KNOWN_HOSTS_FILE)

同步: K8s Secret 重建（含 ssh_mcp_key + known_hosts）
      188/110 authorized_keys 已加入公鑰
      SSH 連線驗證: 188 OK / 110 OK

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:59:29 +08:00
OG T
9f5120bde1 docs(logbook): Session 結尾 — MCP Phase 2a SSH volume + 全啟用完成 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:36:11 +08:00
OG T
b1c1091787 feat(mcp): MCP Phase 2a — SSH MCP key volume + SSH/ArgoCD/Sentry MCP 啟用
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 7m58s
Details 
- 06-deployment-api.yaml: ssh-mcp-key volume 定義（optional: true, 0400）
- 04-configmap.yaml: SSH_MCP_ENABLED/KNOWN_HOSTS_FILE + ARGOCD_MCP_ENABLED + SENTRY_MCP_ENABLED

MCP Phase 1-4 全部實作完成，10 providers 全部已啟用（ArgoCD/Sentry/SSH 需人工 Secret）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:35:52 +08:00
OG T
5d78c5492b feat(argocd-mcp): 啟用 ArgoCD MCP Provider + token 注入流程
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- config.py: ARGOCD_URL → https://192.168.0.125:30443（實際 HTTPS NodePort）
- config.py: ARGOCD_MCP_ENABLED=True + SENTRY_MCP_ENABLED=True（預設啟用）
- cd.yaml: 新增 ARGOCD_API_TOKEN Gitea Secret → K8s Secret 注入步驟
- K8s: ARGOCD_API_TOKEN 已手動注入 awoooi-secrets + API pods 已 rollout restart
- ArgoCD: 已開啟 admin account apiKey capability

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:32:28 +08:00
OG T
f14ca4b117 docs(logbook): Session 4 結尾更新 — MCP Phase 3/4 全完成 + ADR-070 閉環 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:17:21 +08:00
OG T
7eb49f9c20 feat(mcp-phase4c): AI 動態規則生成 — 新 alertname 自動產 Playbook 草稿
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m29s
Details 
_generate_playbook_draft_if_new():
  - Playbook 無命中時非同步觸發（不阻塞決策主流程）
  - 先用 semantic_search(threshold=0.92) 確認 KM 無同名 Playbook
  - 呼叫 qwen2.5:7b-instruct (Ollama 188) 生成五段結構化草稿
    (症狀/根因/診斷步驟/修復動作/驗收條件)
  - 寫入 KnowledgeEntry(type=PLAYBOOK, status=DRAFT, source=AI_EXTRACTED)
  - 寫入 AlertOperationLog PLAYBOOK_DRAFT_CREATED 事件
  - 失敗靜默 debug log

完成 MCP Phase 4 全三項:
  4a NemoClaw second opinion (信心 < 0.7)
  4b K8s 狀態快照 k8s_state_after
  4c AI 動態 Playbook 草稿生成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:16:39 +08:00
OG T
0fa3b35a1c feat(mcp-phase4b): 自動修復後抓 K8s Pod 狀態寫入 k8s_state_after
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 24s
Details 
_push_auto_repair_result() 成功後:
  - 呼叫 K8sProvider.kubectl_get(pods, label=app=<service>)
  - 結果截斷 500 字寫入 incidents.k8s_state_after
  - km_conversion_service._build_content() 已支援顯示此欄位
  - 失敗靜默 debug log，不阻塞主流程

完成 KM 三段資料閉環: 症狀(labels) + 情境(metrics_before) + 動作(action) + 效果(metrics_after + k8s_state_after)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:15:31 +08:00
OG T
f3ee577f9d feat(mcp-phase4a): NemoClaw second opinion — 信心 < 0.7 觸發 deepseek-r1:14b 複審
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- _nemoclaw_second_opinion(): 呼叫 Ollama 188 deepseek-r1:14b 做獨立推理
  - 解析 <think>...</think> CoT 格式，只取正文
  - 30s timeout，失敗靜默降級
  - 輸出截斷 300 字
- _dual_engine_analyze(): LLM 信心 < 0.7 時非同步觸發 second opinion
  - 結果附加到 proposal_data["advisory_note"]
- _push_decision_to_telegram(): advisory_note 以 NemoClaw bot 身分追加訊息
  - 格式: "NemoClaw 第二意見 (信心=0.xx)"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:14:54 +08:00
OG T
a2cc985f60 feat(mcp-phase3): ArgoCD MCP + Sentry MCP + 完整 Provider 註冊
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ArgoCDProvider (3 工具):
  - argocd_list_apps: 列出所有 App + sync/health 狀態
  - argocd_get_app_status: 詳細狀態 + 問題資源清單
  - argocd_get_sync_history: 最近 N 筆部署記錄
  - 輸入驗證: app_name 白名單 regex
  - 需 ARGOCD_API_TOKEN + ARGOCD_MCP_ENABLED=true

SentryProvider (3 工具):
  - sentry_list_issues: 列出最近 Issues（狀態過濾）
  - sentry_get_issue: 詳情 + stacktrace 最後 5 frames
  - sentry_search_issues: PromQL 風格搜尋
  - issue_id 白名單驗證（只允許純數字）
  - 需 SENTRY_AUTH_TOKEN + SENTRY_MCP_ENABLED=true

providers/__init__.py: 補上 Prometheus + SSH + ArgoCD + Sentry 全部 10 個 providers
config.py: 新增 ARGOCD_URL / ARGOCD_API_TOKEN / ARGOCD_MCP_ENABLED / SENTRY_MCP_ENABLED

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:11:53 +08:00
OG T
3b896d0fbd docs(logbook): Session 3 結尾更新 — ADR-071-I/J + Backlog 清零 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:08:28 +08:00
OG T
de055778b3 fix(cd): CD_PUSH_TOKEN + backup 路徑使用 BACKUP_ROOT 環境變數
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- cd.yaml: GITEA_CD_TOKEN → CD_PUSH_TOKEN（Gitea 保留 GITEA_ 前綴）
- ADR-069: 同步更新 token 名稱說明
- backup-from-110.sh: 改用 BACKUP_ROOT 環境變數（預設 /home/ollama/backup/110）
  避免 /var/log /var/run 需要 root 權限
- 已部署到 188 + cron 0 1 * * * 設定完成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:07:47 +08:00
OG T
1ec19656b5 feat(adr071-ij): TYPE-2 指標快照卡片 + KM 三段資料整合
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m17s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 36s
Details
Ansible Lint / lint (push) Has been cancelled
Details 
ADR-071-I: decision_manager 執行前後各抓一次 Prometheus metrics
  - _fetch_metrics_snapshot(): 依 alertname 選擇 CPU/Mem/Disk/Restart 查詢
  - _format_metrics_delta(): 輸出 "CPU 92%→23% | Mem 78%→45%" 格式
  - _push_auto_repair_result(): metrics_after 寫 DB + TYPE-2 卡片顯示 delta
  - _auto_execute(): metrics_before 在執行前寫 DB（完成閉環）

ADR-071-J: km_conversion_service._build_content() 使用精簡 delta 格式
  - 從 metrics_before/after 產生人讀 delta（CPU/Mem/Disk/重啟次數）
  - 附加 k8s_state_after（若有）
  - 格式: 症狀 + 根因 + 動作 + 效果數字（症狀→情境→動作→效果）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:09:35 +08:00
OG T
43edff184d feat(dr): Sprint C — Host rsync 備份 + DR SOP 文件 
C-1 Velero: 已確認運作中（daily-awoooi-prod schedule, 13d, MinIO Available）

C-2 Host rsync 備份:
  scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
    - Harbor registry data（最高優先）
    - Gitea repos
    - bitan-pharmacy.git（若存在）
    - 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
    - 失敗時 Telegram 告警
  ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則

C-3 DR SOP 文件:
  docs/runbooks/disaster-recovery/DR-K8s-awoooi.md  (<15分鐘)
  docs/runbooks/disaster-recovery/DR-Nginx.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Harbor.md       (<30分鐘)
  docs/runbooks/disaster-recovery/DR-Bitan.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Stock.md        (<5分鐘)

部署備份腳本說明 (需手動執行):
  scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
  ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
  ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:04:18 +08:00
OG T
a29e5e1de2 feat(mcp-phase1): K8s MCP 強化 — 6 個新工具 + namespace 白名單 
MCP Phase 1 (ADR-069 Sprint B 後驗收):
  k8s_get_pod_logs    — Pod log 取得 (tail 1-500，支援 previous)
  k8s_watch_rollout   — rollout 狀態監控直到完成 (timeout 10-300s)
  k8s_get_events      — K8s events (可過濾 resource_name / event_type)
  k8s_describe_pod    — 完整 Pod describe (Conditions/Volumes/Env)
  k8s_get_hpa_status  — HPA 副本數/CPU utilization
  k8s_get_node_conditions — Node Ready/MemoryPressure/DiskPressure

安全強化:
  - ALLOWED_NAMESPACES = {"awoooi-prod"} 硬編碼白名單
  - _validate_namespace() + _validate_name() 參數白名單
  - 數值參數上下限夾緊 (tail 1-500, timeout 10-300s)
  - event_type 只允許 Warning / Normal

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:01:38 +08:00
OG T
b783f71b97 docs: LOGBOOK + Memory 更新 — Sprint B 全完成 
Sprint B-1/B-2/B-3 全部完成，後置動作：
- 建立 Gitea Secret GITEA_CD_TOKEN
- 首席架構師確認 2af4dff 後 push gitea main

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:58:04 +08:00
OG T
7f4ec717ef feat(gitops): Sprint B-2/B-3 — ArgoCD Application + CD GitOps 模式 
B-2: k8s/argocd/awoooi-prod-app.yaml
  - ArgoCD Application awoooi-prod 建立（已 apply 到 K8s）
  - automated sync: prune + selfHeal
  - ignoreDifferences: Deployment image + Secret data
  - 全部 17 個 K8s 資源已確認 Synced

B-3: .gitea/workflows/cd.yaml — Deploy step 重寫
  - 舊: kubectl set image（與 ArgoCD selfHeal 衝突）
  - 新: kustomize edit set image → git commit [skip ci] → push → ArgoCD sync
  - 新增等待 ArgoCD Synced + Healthy（最多 120s）
  - 需建立 Gitea Secret: GITEA_CD_TOKEN（見 ADR-069）

docs/adr/ADR-069-infra-gitops-sprint-b.md
  - 決策記錄：循環觸發防護 + ignoreDifferences 設計

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:57:42 +08:00
OG T
a63c586d9a docs: LOGBOOK + Skill04 更新 — Sprint B-1 + Architecture Review 記錄 
- LOGBOOK: 新增 Sprint B-1 完成條目 + 架構Review修復清單
- Skill04 v2.6: 加入 Ansible IaC 目錄結構 + SSH MCP 安全規則

記錄首席架構師 2026-04-11 架構Review指令執行結果

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:52:29 +08:00
OG T
2af4dffcc6 fix(security): Architecture Review 修復 5 項高信心問題 
安全修復 (P0):
1. ssh_provider: 新增 _validate_param() 白名單驗證，防止 command injection
   - container_name/service/filter_name: [a-zA-Z0-9._-]{1,128}
   - compose_dir: 必須以 /opt/ 或 /srv/ 開頭，禁止 ..
   - domain: FQDN 白名單
   - tail/port/lines: int() 轉換 + 上下限夾緊
2. ssh_provider: known_hosts=None 改為讀 SSH_MCP_KNOWN_HOSTS_FILE 環境變數
   - 預設仍 None（內網快速啟動），但啟動時寫入 warning log
   - 設定文件：ops/runbooks/ssh-mcp-setup.md (待補)

模組化修復 (P1):
3. km_conversion_service: 移除 import 時的 ALERT_EVENT_TYPES.update() 副作用
   - ADR-071 event types 移入 alert_operation_log_repository.py 靜態集合
4. telegram_gateway: create_task() 改為 await + try/except
   - 避免 DB session 關閉後的競爭條件
   - KM 轉換失敗記錄 warning log，不中斷主流程
5. km_conversion_service: 新增頂層 try/except，錯誤一律 error log 後 re-raise

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:50:26 +08:00
OG T
0139aa79e7 feat(infra): B-1 Ansible Host IaC 骨架完整版 
- roles/nginx/templates/188-all-sites.conf.j2: 8 個服務 Jinja2 模板
- roles/docker-compose-service/tasks/main.yml: 通用 Docker Compose role
- roles/swap/tasks/main.yml: swap2.img 管理 role (110 專用)
- roles/pm2-service/tasks/main.yml: PM2 process 狀態確認 role
- .gitea/workflows/ansible-lint.yml: infra/ansible/** 異動自動 lint

Sprint B-1 完成: Git = 唯一真相 (Host IaC 骨架)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:47:10 +08:00
OG T
44e8b22585 docs(logbook): Session 結尾更新 — ADR-071 第一批 + MCP Phase 2 全完成 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:36:07 +08:00
OG T
6351e9a0e9 feat(mcp-phase2): MCP Phase 2 — Prometheus MCP + SSH MCP + alert labels
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m37s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 35s
Details 
MCP-2b: prometheus_provider.py
  - prometheus_query (PromQL 即時查詢)
  - prometheus_query_range (歷史趨勢，預設 15 分鐘)
  - prometheus_get_alert_history (告警觸發歷史)
  - config: PROMETHEUS_URL + PROMETHEUS_MCP_ENABLED

MCP-2a: ssh_provider.py
  - 群組A 9 個只讀診斷工具 (top/disk/memory/logs/status/port/nginx/swap)
  - 群組B 6 個安全操作工具 (restart/compose/systemctl/clear-log/ssl/nginx-reload)
  - 四層安全守衛 (白名單/allowed_hosts/forbidden_patterns/trust_score)
  - config: SSH_MCP_ENABLED + SSH_MCP_ALLOWED_HOSTS

K8s: 04-ssh-mcp-secret.example.yaml (ssh-mcp-key Secret 範本 + 建立步驟)

Alert labels: alerts-unified.yml 補充 mcp_provider/host_type/alert_category
  覆蓋: HostHighCpuLoad/HostOutOfMemory/HostOutOfDiskSpace/DockerContainer*
        SignOzDown/SentryDown/HarborDown/GiteaDown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:35:35 +08:00
OG T
325b3851b5 feat(adr-071): 告警通知四類型第一批 B/C/E/F/G/H 全實作
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
Type Sync Check / check-type-sync (push) Failing after 1m7s
Details 
ADR-071-B: classify_notification() — 五型分類器 (TYPE-1/2/3/4/4D)
ADR-071-C: send_info_notification() — TYPE-1 純資訊無按鈕卡片
ADR-071-E: _build_inline_keyboard() — 依 alert_category 動態組合 TYPE-3 按鈕
ADR-071-F: send_drift_card() — TYPE-4D Config Drift 卡片 + Diff 截斷
ADR-071-G: km_conversion_service.py — Incident RESOLVED 自動轉 KM
ADR-071-H: handle_manual_fix_done() — TYPE-4 手動修復 Bot 對話閉環

前批已完成: ADR-071-A (DB Migration) + ADR-071-D (狀態機守衛)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:24:20 +08:00
OG T
45b13f1d7c fix(k8s): 更新 03-secrets.example.yaml — Sentry DSN 改 HTTPS 公網域名 
ADR-069 Sprint A A-0-5:
- SENTRY_DSN: http://...@192.168.0.110:9000/3 → https://...@sentry.wooo.work/3
- 同步 Web DSN 範例（NEXT_PUBLIC_SENTRY_DSN）
- 加入取得 DSN 的步驟說明
- system.url-prefix 已設定為 https://sentry.wooo.work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:05:11 +08:00
OG T
68a3858ae4 fix(auto_execute): 守衛加入 target==alertname 檢查，防止 LLM 把告警名稱當 deployment 名稱
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m33s
Details 
HostHighCpuLoad 等主機告警，NemoTron Tool Calling 可能把
alertname 填入 deployment_name，導致執行
'kubectl rollout restart deployment HostHighCpuLoad'。

新增守衛: _target == _alertname 時拒絕執行並通知人工介入。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 01:13:24 +08:00
OG T
8a8c6a4eb1 docs(logbook): ADR-069 Sprint A 主幹完成 — SSL/HTTPS/nginx 全站收斂 
- A-0-1~A-0-2: Swap擴充 + snuba/Harbor修復
- A-1~A-4: GitLab移除 + n8n/open-webui啟動 + Harbor port修正
- A-5: SSL申請 sentry/gitea/langfuse/signoz/stock.wooo.work
- A-6: 188 nginx HTTPS blocks 全部上線
- A-7: 110 all-sites-from-188.conf 封存，188單一控制點
- A-8/A-9: stock NodePort + keepalived VIP:200 確認
- 全域驗收：商業服務全通 + 新9個域名HTTPS全通

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 01:07:05 +08:00
OG T
fa7b763689 docs(infra): ADR-069 基礎設施重建計畫規格 v1.3 — Sprint A/B/C 完整設計 
新增 Sprint A（清廢棄修錯誤）+ Sprint B（Ansible+ArgoCD GitOps）+ Sprint C（Velero+rsync DR）
完整技術調查：Sentry snuba DNS根因、Harbor port錯誤、bitan Docker化需求、volumes盤點
加入第十二節（與現有專案整合）+ 第十三節（文件更新時間表）
LOGBOOK 更新、project_master_workplan 加入 ADR-069 Sprint A/B/C

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 00:01:07 +08:00
OG T
a4d655ea7f fix(auto_execute): 安全守衛 — 拒絕執行含 unknown 或未解析 placeholder 的 action
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 19m7s
Details
E2E Health Check / e2e-health (push) Successful in 43s
Details 
主機層告警（HostHighCpuLoad、DockerContainerUnhealthy 等）沒有對應
K8s deployment 名稱，affected_services=[]，導致 _target='unknown'，
執行 'kubectl rollout restart deployment unknown' 這種無意義命令。

修復: 替換後若 action 仍含 'unknown' 或 <...>/{...} 格式，
直接拒絕執行並通知人工介入，不允許帶 placeholder 的命令上線。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 23:57:17 +08:00
OG T
dabc62e0f8 fix(telegram): append_incident_update — 儲存告警卡片 message_id 到 Redis
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 14m31s
Details 
_send_approval_card_to_group 發出告警卡片後，將 Telegram message_id
存入 Redis tg_msg:{incident_id}（TTL 24h），供後續 append_incident_update
換掉批准按鈕 + reply 狀態。

修復前：tg_msg key 從未被寫入，append 永遠 fallback 發新訊息，
批准按鈕永遠無法被移除。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 22:41:30 +08:00
OG T
797c7c749e fix(nemotron): deepseek-r1 num_predict 400→1200，避免 <think> block 截斷後空回覆
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 28s
Details 
deepseek-r1:14b 思考 token 超過 400 會在 </think> 前截斷，導致
清理後 body 為空，Telegram 顯示空訊息。
- chat_manager: num_predict 400 → 1200
- telegram_gateway: _clean_ai_reply 空值加 fallback 錯誤提示

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 22:35:37 +08:00
OG T
de6dcd181a docs(logbook): Session 結尾更新 — Backlog 清零 + 全站真實數據驗收 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 22:19:38 +08:00