# AWOOOI 重開機恢復 SOP

> **版本**: v5.0  
> **最後更新**: 2026-04-05 下午 (台北時間)  
> **更新者**: Claude Code (首席架構師)  
> **觸發事件**: Harbor Exited(128) Race Condition 根治 + harbor-watchdog 常駐自愈

---

## 目錄

1. [架構概覽與依賴圖](#架構概覽與依賴圖)
2. [自動化腳本狀態](#自動化腳本狀態)
3. [正常重啟流程 (計劃性維護)](#正常重啟流程)
4. [異常重啟流程 (緊急恢復)](#異常重啟流程)
5. [各主機詳細啟動序列](#各主機詳細啟動序列)
6. [常見故障排查手冊](#常見故障排查手冊)
7. [E2E 驗證腳本](#e2e-驗證腳本)

---

## 架構概覽與依賴圖

### 五主機全貌

```
192.168.0.188  (OLLAMA / 主服務主機)
├── containerd              ← 基礎，必須最先啟動
├── Docker                  ← 依賴 containerd
├── PostgreSQL :5432        ← K3s kine 後端，API DB
├── Redis :6380             ← Working Memory (重啟後清空，需 warm-up)
├── Ollama :11434           ← LLM 推論引擎
├── Nginx :80/:443          ← 反向代理
├── OpenClaw (clawbot) :8088 ← AI 核心 (Docker Compose)
├── MinIO                   ← Velero 備份存儲 (Docker Compose)
└── SignOz                  ← 可觀測性 (Docker Compose)

192.168.0.110  (DevOps 主機)
├── Docker
├── Harbor :5000            ← Container Registry (K3s 拉映像用)
├── Gitea :3001             ← 主要 Git / CI 管理介面
├── Gitea Act Runner        ← CD pipeline 執行者 (docker: gitea-runner)
├── Langfuse :3100          ← LLMOps 追蹤
├── Prometheus :9090        ← 指標收集
├── Alertmanager :9093      ← 告警路由
├── Grafana :3002           ← 監控儀表板
├── Sentry :9000            ← Error Tracking (2026-03-24，2026-04-05 加入 startup)
└── SignOz                  ← 可觀測性

192.168.0.120  (K3s Master - mon)
├── K3s server (control plane)
├── kube-proxy              ← NodePort 轉發 32334/32335
└── keepalived              ← VIP 192.168.0.125 (secondary)

192.168.0.121  (K3s Worker - mon1)
├── K3s agent (worker)
└── kube-proxy

192.168.0.125  (VIP, keepalived 管理)
├── → :32334  AWOOOI API
└── → :32335  AWOOOI Web
```

### 服務依賴關係

```
嚴格啟動順序:

【188 層】
containerd
  └── Docker
        ├── PostgreSQL (kine DB, API DB)
        │     └── Redis (Working Memory)
        │           └── Ollama (LLM)
        │                 └── Nginx
        │                       ├── SignOz
        │                       ├── MinIO
        │                       └── OpenClaw (依賴 aiops-network)

【110 層】
Docker
  └── harbor-log (等 healthy)
        └── Harbor 全組件 (:5000)
              └── Gitea (:3001)
                    ├── Langfuse
                    ├── Monitoring (Prometheus + Alertmanager + Grafana)
                    │         [Alertmanager → AWOOOI API 告警鏈路]
                    ├── SignOz
                    └── Gitea Act Runner (等 Gitea 就緒後才啟動)

【120/121 層】
K3s (依賴 PostgreSQL@188)
  └── kube-proxy (NodePort 規則)
        └── Pods: API / Web / Worker
              └── keepalived (VIP)

【告警鏈路】
Prometheus → Alertmanager(110)
  → AWOOOI API(VIP 192.168.0.125:32334/api/v1/webhooks/alertmanager)  ← 直接，不走 OpenClaw
  → TelegramGateway → Telegram

【CD 鏈路】
Git push → Gitea(110:3001)
  → Act Runner (gitea-runner container)
  → Build Docker image → Harbor(:5000) → kubectl → K3s pods
```

### 關鍵依賴說明

| 服務 | 關鍵依賴 | 若依賴失敗 |
|------|---------|-----------|
| K3s | PostgreSQL@188 ready | K3s 無法啟動，所有 pod 無法調度 |
| OpenClaw | Docker + aiops-network | Telegram AI 對話失效 |
| Alertmanager → API | NetworkPolicy 允許 110/32 | 告警全面沉默 |
| CD pipeline | Gitea Act Runner online | 所有部署失效 |
| Harbor | harbor-log healthy | 所有其他 Harbor 容器 Exited 128 |
| K8s pods | Harbor (映像倉庫) | ImagePullBackOff |

---

## 自動化腳本狀態

### 已部署 (2026-04-05)

| 主機 | 腳本 | 部署位置 | systemd service | 狀態 | 類型 |
|------|------|---------|----------------|------|------|
| **188** | `awoooi-startup.sh` | `/usr/local/bin/` | `awoooi-startup.service` | ✅ enabled | 重開機 oneshot |
| **110** | `awoooi-startup-110.sh` | `/usr/local/bin/` | `awoooi-startup-110.service` | ✅ enabled | 重開機 oneshot |
| **110** | `harbor-watchdog.sh` | `/usr/local/bin/` | `harbor-watchdog.service` | ✅ active | **常駐 watchdog** |
| **120** | K3s 原生 | 系統內建 | `k3s.service` | ✅ enabled | 系統服務 |
| **121** | K3s 原生 | 系統內建 | `k3s-agent.service` | ✅ enabled | 系統服務 |

**本地原始碼**: `scripts/reboot-recovery/`

### harbor-watchdog（2026-04-05 新增）

**問題背景**: Harbor 的 Exited (128) Race Condition
- `restart: always` 在 harbor-log 未就緒時不斷重試，最終進入 backoff 放棄
- 即使 harbor-log 後來 healthy，其他容器不會自動重試
- 只靠 startup 腳本無法處理**運行中崩潰**的情況

**設計**:
- `Type=simple` 常駐進程（不是 oneshot），systemd 永久監控
- 每 60 秒輪詢 `http://127.0.0.1:5000/v2/`（401 = healthy）
- 偵測到不健康 → 等 5 秒再確認（避免誤報）→ 執行完整時序修復
- 修復邏輯：清除 Exited 容器 → 只啟動 harbor-log → 等 healthy → 啟動全部

**查看狀態**:
```bash
journalctl -u harbor-watchdog.service -f          # 即時 log
systemctl status harbor-watchdog.service          # 服務狀態
cat /var/log/harbor-watchdog.log                  # 持久化 log
```

### 各腳本覆蓋清單

**188 (7 步驟)**:
- containerd BoltDB 損壞偵測與修復
- Docker BoltDB 損壞偵測與修復
- PostgreSQL WAL 損壞偵測 + pg_resetwal + kine VACUUM
- Redis 啟動
- Ollama 啟動
- Nginx 啟動
- SignOz + MinIO + aiops-network + OpenClaw

**110 (7 步驟 + harbor-watchdog 常駐)**:
- Docker BoltDB 損壞偵測與修復
- 孤兒容器清除 (Exited 128)
- Harbor（harbor-log first 策略：Phase 1 清除 → Phase 2 只啟 harbor-log → Phase 3 等 healthy → Phase 4 啟全部）
- Gitea + Langfuse + Monitoring (含 Alertmanager 健康驗證)
- SignOz
- **Gitea Act Runner** (自動清除過期 .runner 配置)
- **Sentry** (/opt/sentry，含 PostgreSQL WAL + Redis RDB 損壞自動修復)
- **harbor-watchdog.service** (常駐，每 60s 自動修復)

---

## 正常重啟流程

> 適用於：計劃性維護、安全更新、OS 升級等**有預期**的重啟。

### 重啟前準備 (T-30 分鐘)

```bash
# 1. 確認沒有 INVESTIGATING/MITIGATING 的 incident
curl -s http://192.168.0.121:32334/api/v1/incidents | python3 -c 'import sys,json; d=json.load(sys.stdin); print("Active incidents:", d.get("count",0))'

# 2. 確認 K3s 健康
ssh wooo@192.168.0.120 "kubectl get nodes && kubectl get pods -n awoooi-prod"

# 3. 備份確認 (MinIO Velero)
ssh wooo@192.168.0.120 "kubectl get backup -n velero 2>/dev/null | head -5"
```

### 建議重啟順序

```
第一步: 188 + 110 同時重啟
        (兩者獨立，可並行)

等待 5 分鐘確認基礎服務就緒:
  - PostgreSQL accepting connections
  - Harbor returning HTTP 200
  - Gitea accessible

第二步: 120 + 121 同時重啟
        (K3s cluster)

等待 10 分鐘:
  - Nodes Ready
  - Pods Running
```

> **重要**: 120/121 必須在 188 PostgreSQL 完全就緒後才重啟，
> 否則 K3s kine 連不到 DB 無法啟動。

### 重啟後驗證 (T+15 分鐘)

執行底部 E2E 驗證腳本。所有項目必須 ✅。

---

## 異常重啟流程

> 適用於：電源中斷、OOM、系統崩潰、強制重啟等**非預期**的重啟。

### 第一步：快速診斷 (T+2 分鐘)

```bash
# 188 狀態
ssh ollama@192.168.0.188 "
systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx
docker ps --format '{{.Names}}: {{.Status}}' | grep -E 'clawbot|minio|signoz'
"

# 110 狀態
ssh wooo@192.168.0.110 "
docker ps --format '{{.Names}}: {{.Status}}' | grep -E 'harbor-log|gitea$|alertmanager|gitea-runner'
"

# K3s 狀態
ssh wooo@192.168.0.120 "
kubectl get nodes 2>/dev/null || echo 'K3s not ready'
kubectl get pods -n awoooi-prod 2>/dev/null | grep -v Running | head -10
"

# API 健康
curl -s --max-time 5 http://192.168.0.121:32334/api/v1/health 2>/dev/null | python3 -c 'import sys,json; d=json.load(sys.stdin); print("API:", d["status"])' 2>/dev/null || echo 'API unreachable'
```

### 第二步：按優先級恢復

**P0 (T+0~5 分鐘): 基礎設施**
```bash
# 自動化腳本通常已處理，但若沒有自動啟動:
ssh ollama@192.168.0.188 "sudo /usr/local/bin/awoooi-startup.sh"
ssh wooo@192.168.0.110 "echo '0936223270' | sudo -S /usr/local/bin/awoooi-startup-110.sh"
```

**P1 (T+5~15 分鐘): K3s 叢集**
```bash
# 等 PostgreSQL@188 就緒後
ssh ollama@192.168.0.188 "pg_isready -h localhost -p 5432"

# K3s 若未自動恢復
ssh wooo@192.168.0.120 "sudo systemctl restart k3s"
ssh wooo@192.168.0.121 "sudo systemctl restart k3s-agent"

# 等待 Nodes Ready
ssh wooo@192.168.0.120 "kubectl wait --for=condition=Ready nodes --all --timeout=120s"
```

**P2 (T+15~30 分鐘): 業務驗證**
```bash
# Pods 全部 Running?
ssh wooo@192.168.0.120 "kubectl get pods -n awoooi-prod"

# API 健康 (含所有 components)?
curl -s http://192.168.0.121:32334/api/v1/health | python3 -m json.tool

# 告警鏈路 E2E 測試
curl -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"RebootRecoveryTest","severity":"info"},"annotations":{"summary":"重開機恢復測試，請忽略"},"startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"test"}'
# 預期: {"success":true,...} 且 Telegram 收到測試告警
```

---

## 各主機詳細啟動序列

### 192.168.0.188 — 主服務主機

| 步驟 | 服務 | 驗證方式 | 常見失敗原因 |
|------|------|---------|------------|
| 1 | containerd | `systemctl is-active containerd` | BoltDB meta.db 損壞 |
| 2 | Docker | `systemctl is-active docker` | BoltDB local-kv.db 損壞 |
| 3 | PostgreSQL | `pg_isready -h localhost -p 5432` | WAL checkpoint 損壞 |
| 3a | kine VACUUM | `psql -d k3s_datastore -c "VACUUM ANALYZE kine;"` | 慢查詢導致 K3s 超時 |
| 4 | Redis | `redis-cli -p 6380 ping` | bind 配置錯誤 (必須 0.0.0.0:6380) |
| 5 | Ollama | `systemctl is-active ollama` | GPU 記憶體不足 |
| 6 | Nginx | `systemctl is-active nginx` | port 衝突 |
| 7a | aiops-network | `docker network ls \| grep aiops-network` | 重啟後 external network 消失 |
| 7b | OpenClaw | `curl http://localhost:8088/health` | pip 依賴損壞需 rebuild |
| 7c | MinIO | `docker ps \| grep minio` | 無狀態服務，重啟即可 |
| 7d | SignOz | `curl http://localhost:3301/-/healthy` | ClickHouse 起慢 |

**BoltDB 損壞修復**:
```bash
# containerd meta.db
BOLT=/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart containerd

# Docker network BoltDB
BOLT=/var/lib/docker/network/files/local-kv.db
cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart docker
```

**PostgreSQL WAL 修復**:
```bash
# 確認是 WAL 損壞
journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint"

# 修復
systemctl stop postgresql@14-main
/usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
systemctl start postgresql@14-main
```

### 192.168.0.110 — DevOps 主機

| 步驟 | 服務 | 驗證方式 | 常見失敗原因 |
|------|------|---------|------------|
| 1 | Docker | `systemctl is-active docker` | BoltDB 損壞 |
| 2 | 孤兒容器清除 | `docker ps -a \| grep "Exited (128)"` | 舊容器引用已不存在的 network |
| 3 | harbor-log | `docker inspect --format='{{.State.Health.Status}}' harbor-log` | syslog :1514 未就緒 |
| 3 | Harbor 全組件 | `curl http://localhost:5000/v2/` | harbor-log 還沒 healthy |
| 4 | Gitea | `curl http://localhost:3001` | 無特殊問題 |
| 5 | Langfuse | `curl http://localhost:3100` | DB 未就緒 |
| 6 | Alertmanager | `curl http://localhost:9093/-/healthy` | docker compose 未啟動 |
| 6a | 告警鏈路驗證 | 發送測試 webhook | webhook URL 錯誤 / NetworkPolicy |
| 7 | SignOz | `curl http://localhost:3301/-/healthy` | ClickHouse 起慢 |
| 8 | Gitea Runner | `docker logs gitea-runner \| grep SUCCESS` | .runner 配置過期 |
| 9 | Sentry | `curl -o /dev/null -w '%{http_code}' http://localhost:9000/` | PostgreSQL WAL/Redis RDB 損壞 (見下方) |

**Harbor Exited 128 修復**:
```bash
# 等 harbor-log healthy
until [ "$(docker inspect --format='{{.State.Health.Status}}' harbor-log)" = "healthy" ]; do
  echo "waiting..."; sleep 5
done

# 清除 Exited 128 容器
docker rm -f $(docker ps -a --filter status=exited --format "{{.Names}}" | grep -v harbor-log)

# 重新啟動
cd /home/wooo/harbor/harbor && docker compose up -d
```

**Gitea Runner 重新註冊**:
```bash
# 清除過期配置 (指向錯誤 hostname 的)
sudo rm /home/wooo/act-runner/data/.runner

# 獲取新的 registration token
curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners/registration-token'

# 更新 docker-compose.yml 的 GITEA_RUNNER_REGISTRATION_TOKEN
# 然後啟動
cd /home/wooo/act-runner && docker compose up -d
```

### 192.168.0.120/121 — K3s 叢集

| 步驟 | 項目 | 驗證方式 |
|------|------|---------|
| 前提 | PostgreSQL@188 ready | `pg_isready -h 192.168.0.188 -p 5432` |
| 1 | K3s service | `systemctl is-active k3s` (120) |
| 2 | K3s agent | `systemctl is-active k3s-agent` (121) |
| 3 | Nodes Ready | `kubectl get nodes` |
| 4 | Pods Running | `kubectl get pods -n awoooi-prod` |
| 5 | keepalived VIP | `ip addr show \| grep 192.168.0.125` (120) |
| 6 | NodePort | `nc -zv 192.168.0.121 32334` |

---

## 常見故障排查手冊

### 1. 告警沉默 (Telegram 沒收到告警)

```
診斷樹:
  Alertmanager healthy? (http://192.168.0.110:9093/-/healthy)
  ├── NO → docker compose up -d (在 110 /home/wooo/monitoring/)
  └── YES
        ↓
  Webhook URL 正確?
  grep 'url:' /home/wooo/monitoring/alertmanager.yml
  必須是: http://192.168.0.125:32334/api/v1/webhooks/alertmanager
  ├── NO → bash scripts/ops/deploy-alertmanager-config.sh
  └── YES
        ↓
  從 110 curl POST webhook 成功?
  curl -X POST http://192.168.0.125:32334/api/v1/webhooks/alertmanager ...
  ├── timeout → NetworkPolicy 未允許 110
  │            kubectl apply -f k8s/awoooi-prod/02-network-policy.yaml
  └── {"success":true} → 檢查 Telegram Bot Token
```

Alertmanager config drift 修復：

```bash
# 從專案根目錄執行；會從 K8s Secret 注入 Telegram bot token 與 SRE_GROUP_CHAT_ID，
# 先用 amtool 驗證，再備份 110 live config、原 inode 覆寫、修正 0644 權限並 HUP reload。
bash scripts/ops/deploy-alertmanager-config.sh
```

**補充診斷 — 特定服務異常但無告警（Alertmanager 正常）**:
```bash
# 確認 Prometheus 規則數量和關鍵規則是否存在
ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"
import sys,json
r=json.load(sys.stdin)
names=[x['name'] for g in r['data']['groups'] for x in g['rules']]
total=len(names)
key=['SentryDown','HarborDown','GiteaDown','OpenClawDown','AlertmanagerDown','AlertChainUnhealthy']
for k in key:
    print(f'{'OK' if k in names else 'MISSING'}: {k}')
print(f'Total rules: {total}')
\""

# 若規則不存在或 total < 25 → 規則未部署
# 修復: 從專案根目錄執行
bash scripts/ops/deploy-alerts.sh

# 若規則存在但 inactive → 確認 blackbox probe target
curl http://192.168.0.110:9090/api/v1/query?query=probe_success
```

**根本教訓 (2026-04-05)**:
- 舊架構: Alertmanager → OpenClaw 8088 (中間層，重開機後 OpenClaw 不健康就沉默)
- 新架構: Alertmanager → AWOOOI API 直通 (更穩定，單一故障點少)
- NetworkPolicy 必須允許 110/32 進入 pod

### 2. Gitea CD 無法部署

```
診斷樹:
  Gitea runner 數量?
  curl -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners'
  ├── total_count: 0 → Runner 離線
  │     docker ps | grep gitea-runner
  │     ├── 不存在 → cd /home/wooo/act-runner && docker compose up -d
  │     └── 存在但 Restarting → docker logs gitea-runner
  │           "lookup gitea" → 清除 data/.runner → docker compose up -d
  └── total_count: 1 → Runner 在線但 CD 失敗
        查看 Gitea Actions 日誌: http://192.168.0.110:3001/wooo/awoooi/actions
```

### 3. 網站數據顯示 0 / incidents 消失

```
原因: Redis Working Memory 重啟後清空
     新版 API (f4f454f+) 啟動時自動 warm-up
     舊版 API 沒有此邏輯

診斷:
  curl http://192.168.0.121:32334/api/v1/incidents
  → count: 0 但 Telegram 有歷史告警?
  → 確認 API pod image 版本

修復 (若 image 版本正確):
  kubectl rollout restart deployment/awoooi-api -n awoooi-prod

修復 (若 image 版本過舊):
  等待 CD 完成部署最新版本
```

### 4. NodePort 32334 從外部無法訪問

```
診斷:
  nc -zv 192.168.0.121 32334  ← 這個通嗎?
  ├── 通 → 用 192.168.0.121:32334 直連
  └── 不通
        kubectl get pods -n awoooi-prod
        ├── 沒有 Running pods → K3s pods 問題
        └── 有 Running pods
              iptables -t nat -L KUBE-NODEPORTS -n | grep 32334
              ├── 沒有規則 → K3s kube-proxy 問題，重啟 k3s
              └── 有規則 → NetworkPolicy 或防火牆
```

### 5. Sentry 重開機後損壞修復 (2026-04-05 事故記錄)

**症狀**: sentry-self-hosted-postgres-1 / redis-1 / clickhouse-1 持續 Restarting

**原因**: 重開機時各 DB 未正常關閉，導致資料損壞：
- PostgreSQL: WAL 損壞 (`could not locate a valid checkpoint record`)
- Redis: dump.rdb 損壞 (`Wrong signature trying to load DB from file`)
- ClickHouse: system table store parts 損壞 (`broken and needs manual correction`)

**修復步驟**:

```bash
# Step 1: 修復 PostgreSQL WAL
docker run --rm -u 999 -v sentry-postgres:/var/lib/postgresql/data \
  postgres:14 pg_resetwal -f /var/lib/postgresql/data
echo "PostgreSQL WAL reset 完成"

# Step 2: 修復 Redis (session cache，可丟失)
docker run --rm --user root -v sentry-redis:/data alpine \
  sh -c 'rm -f /data/dump.rdb && echo cleared'
echo "Redis dump.rdb 已清除"

# Step 3: 找出損壞的 ClickHouse store 目錄
docker logs sentry-self-hosted-clickhouse-1 2>&1 \
  | grep -oE 'store/[a-f0-9]{3}/[a-f0-9-]+' \
  | sort -u

# Step 4: 依據上方輸出刪除損壞目錄 (以 UUID 為準，以下為範例)
for uuid in "469/469957c9-..." "57d/57d1d567-..."; do
  docker run --rm --user root -v sentry-clickhouse:/data alpine \
    sh -c "rm -rf /data/store/$uuid && echo OK:$uuid"
done

# Step 5: 重啟 Sentry
cd /opt/sentry && docker compose up -d

# Step 6: 等待 2-3 分鐘後驗證
sleep 120
curl -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/
# 預期: 200 / 302 / 400 (需登入)
```

**startup-110.sh 已自動化修復**: PostgreSQL WAL 和 Redis RDB 損壞會自動偵測並修復。ClickHouse parts 損壞需手動識別 UUID（因 UUID 每次不同）。

---

### 6. Harbor ImagePullBackOff

```
診斷:
  kubectl describe pod <pod> -n awoooi-prod | grep -A 5 Events
  → "ImagePullBackOff" or "ErrImagePull"
  
確認 Harbor 健康:
  curl http://192.168.0.110:5000/v2/ → 應該回 {}

若 Harbor 不健康:
  - 等 harbor-log healthy 後重啟: 見 Harbor Exited 128 修復
  - 確認 Docker daemon 在 120/121 有設定 insecure registry:
    cat /etc/docker/daemon.json | grep insecure
```

---

## E2E 驗證腳本

執行此腳本確認重開機後全系統正常。

```bash
#!/bin/bash
# 重開機後 E2E 驗證
# 使用: bash docs/runbooks/scripts/e2e-verify.sh
# 執行位置: 任意可 SSH 的機器

API="http://192.168.0.121:32334"
GREEN='\033[0;32m'; RED='\033[0;31m'; NC='\033[0m'
PASS=0; FAIL=0

check() {
    local name="$1"; local cmd="$2"; local expect="$3"
    result=$(eval "$cmd" 2>/dev/null)
    if echo "$result" | grep -q "$expect"; then
        echo -e "${GREEN}✅ $name${NC}"
        ((PASS++))
    else
        echo -e "${RED}❌ $name${NC} (got: ${result:0:80})"
        ((FAIL++))
    fi
}

echo "=== AWOOOI 重開機 E2E 驗證 $(date '+%Y-%m-%d %H:%M:%S') ==="

# 188 服務
check "188 PostgreSQL" "ssh ollama@192.168.0.188 'pg_isready -h localhost -p 5432'" "accepting"
check "188 Redis" "ssh ollama@192.168.0.188 'redis-cli -p 6380 ping'" "PONG"
check "188 OpenClaw" "ssh ollama@192.168.0.188 'curl -s http://localhost:8088/health'" "healthy"

# 110 服務
check "110 Harbor" "curl -s -o /dev/null -w '%{http_code}' http://192.168.0.110:5000/v2/" "200"
check "110 Gitea" "curl -s -o /dev/null -w '%{http_code}' http://192.168.0.110:3001" "200"
check "110 Alertmanager" "curl -s http://192.168.0.110:9093/-/healthy" "OK"
check "110 Gitea Runner" "curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get(\"total_count\",0))'" "1"
check "110 Sentry" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "[23][0-9][0-9]\|400"
check "Prometheus rules ≥25" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules' | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g['rules']) for g in r['data']['groups']))\"" "[2-9][5-9]\|[3-9][0-9]"

# K3s
check "K3s Nodes Ready" "ssh wooo@192.168.0.120 'kubectl get nodes --no-headers' | awk '{print \$2}'" "Ready"
check "K3s Pods Running" "ssh wooo@192.168.0.120 'kubectl get pods -n awoooi-prod --no-headers' | grep -c Running" "3"

# AWOOOI API
check "API Health" "curl -s $API/api/v1/health" '"status":"healthy"'
check "API openclaw" "curl -s $API/api/v1/health" '"openclaw":{"status":"up"'

# 告警鏈路 E2E
PAYLOAD='{"receiver":"e2e-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"RebootE2ETest","severity":"info"},"annotations":{"summary":"重開機 E2E 驗證，請忽略"},"startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"e2e-test"}'
check "告警鏈路 E2E" "curl -s -X POST $API/api/v1/webhooks/alertmanager -H 'Content-Type: application/json' -d '$PAYLOAD'" '"success":true'

echo ""
echo "=== 結果: ${PASS} 通過, ${FAIL} 失敗 ==="
[ $FAIL -eq 0 ] && echo -e "${GREEN}🎉 全部通過！系統正常${NC}" || echo -e "${RED}⚠️  有 ${FAIL} 項失敗，請排查${NC}"
```

---

## 版本歷史

| 版本 | 日期 | 說明 |
|------|------|------|
| v1.0 | 2026-04-04 | 初版：containerd + Docker + PostgreSQL WAL 修復流程 |
| v2.0 | 2026-04-05 上午 | 188 啟動腳本 + 110 啟動腳本 + Harbor race condition 修復 |
| v3.0 | 2026-04-05 下午 | 完整架構重盤 + Gitea Runner 自動化 + 告警鏈路根因修復 + NetworkPolicy 修正 |
| v4.0 | 2026-04-05 下午 | Prometheus 規則統一部署 (28條) + Sentry startup (Step 7) + Sentry損壞修復SOP + 規則未部署診斷樹 + E2E腳本加入Sentry/Prometheus驗證 |
| v5.0 | 2026-04-05 下午 | Harbor Exited(128) Race Condition 根治：startup-110.sh 改用 harbor-log first 策略 + 新增 harbor-watchdog.service 常駐自愈（curl -f bug 修正）|
