awoooi/docs/adr/ADR-010-secrets-management.md

# ADR-010: 集中式 Secrets 管理架構

**狀態**: 已批准
**日期**: 2026-03-23
**決策者**: 統帥

## 背景

### 當前痛點

```
問題：密碼散落各處，變更一處 → 全線崩潰

Harbor 密碼存在位置：
├── GitHub Secrets (wooo-aiops)
├── GitHub Secrets (awoooi)      ← 今天才補上
├── K8s Secrets (awoooi-prod)
├── Self-hosted Runner 環境變數
├── 開發者本地 ~/.docker/config.json
└── 文檔/記憶體 MD 檔案

結果：
- 變更密碼需同步 5+ 個地方
- 經常遺漏 → CI/CD 崩潰
- 無審計日誌
- 無法追蹤誰何時改了什麼
```

### 需要管理的 Secrets 清單

| 類別 | Secret 名稱 | 使用位置 |
|------|------------|----------|
| **Registry** | HARBOR_USER / HARBOR_PASSWORD | CI/CD, K8s |
| **Database** | DATABASE_URL (PostgreSQL) | API, Worker |
| **Cache** | REDIS_URL | API, Worker, OpenClaw |
| **AI** | ANTHROPIC_API_KEY | API (Agent Teams) |
| **AI** | GEMINI_API_KEY | API (Fallback) |
| **AI** | OLLAMA_URL | API (Local LLM) |
| **Notification** | OPENCLAW_TG_BOT_TOKEN | OpenClaw |
| **Notification** | OPENCLAW_TG_CHAT_ID | OpenClaw |
| **K8s** | KUBECONFIG | CI/CD |
| **Webhook** | WEBHOOK_HMAC_SECRET | API |

---

## 決策

採用 **混合式 Secrets 管理架構**：

1. **Bitwarden (Self-hosted)** - Single Source of Truth
2. **Sealed Secrets** - K8s 內部密碼加密
3. **GitHub Secrets** - CI/CD 專用 (從 Bitwarden 同步)

---

## 架構設計

```
┌─────────────────────────────────────────────────────────────────────┐
│                    AWOOOI Secrets 管理架構                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │  Bitwarden (Self-hosted on 192.168.0.188)                     │  │
│  │  ══════════════════════════════════════                       │  │
│  │  🔐 Single Source of Truth                                    │  │
│  │                                                               │  │
│  │  Organizations:                                               │  │
│  │  └── AWOOOI/                                                  │  │
│  │      ├── Infrastructure/                                      │  │
│  │      │   ├── Harbor (admin / Wooo_Harbor_2026!)              │  │
│  │      │   ├── PostgreSQL (awoooi / ******)                    │  │
│  │      │   └── Redis (password / ******)                       │  │
│  │      ├── AI-Services/                                         │  │
│  │      │   ├── Anthropic API Key                               │  │
│  │      │   ├── Gemini API Key                                  │  │
│  │      │   └── Ollama URL                                      │  │
│  │      └── Notifications/                                       │  │
│  │          ├── Telegram Bot Token                              │  │
│  │          └── Telegram Chat ID                                │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│              ┌───────────────┼───────────────┐                      │
│              │               │               │                      │
│              ▼               ▼               ▼                      │
│  ┌─────────────────┐ ┌─────────────┐ ┌─────────────────┐           │
│  │ GitHub Secrets  │ │ Sealed      │ │ Local Dev       │           │
│  │ (CI/CD)         │ │ Secrets     │ │ (.env.local)    │           │
│  │                 │ │ (K8s)       │ │                 │           │
│  │ HARBOR_USER     │ │             │ │ bw get password │           │
│  │ HARBOR_PASSWORD │ │ 加密存 Git  │ │ "Harbor"        │           │
│  │ OP_TOKEN (sync) │ │ 自動解密    │ │                 │           │
│  └────────┬────────┘ └──────┬──────┘ └─────────────────┘           │
│           │                 │                                       │
│           ▼                 ▼                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  K3s Cluster (192.168.0.120/121)                            │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │  awoooi-prod namespace                              │    │   │
│  │  │  ├── awoooi-secrets (from Sealed Secrets)           │    │   │
│  │  │  ├── awoooi-config (ConfigMap, non-sensitive)       │    │   │
│  │  │  └── Pods (consume secrets as env vars)             │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 實施計畫

### Phase 1: Bitwarden Self-hosted (Day 1-2)

```bash
# 在 192.168.0.188 部署 Vaultwarden (輕量版 Bitwarden)
docker run -d --name vaultwarden \
  -v /data/vaultwarden:/data \
  -e ADMIN_TOKEN='your-admin-token' \
  -e DOMAIN='https://vault.wooo.work' \
  -p 8088:80 \
  vaultwarden/server:latest

# Nginx 設定
server {
    listen 443 ssl;
    server_name vault.wooo.work;

    location / {
        proxy_pass http://localhost:8088;
    }
}
```

**初始化步驟：**
1. 訪問 https://vault.wooo.work
2. 建立 Organization: AWOOOI
3. 建立 Collections: Infrastructure, AI-Services, Notifications
4. 匯入現有密碼

### Phase 2: Sealed Secrets (Day 2-3)

```bash
# Step 1: 安裝 Sealed Secrets Controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.26.0/controller.yaml

# Step 2: 取得 Public Key
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > /Users/ogt/awoooi/k8s/sealed-secrets-cert.pem

# Step 3: 加密現有 Secrets
cat <<EOF > /tmp/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: awoooi-secrets
  namespace: awoooi-prod
type: Opaque
stringData:
  DATABASE_URL: "postgresql+asyncpg://awoooi:xxx@192.168.0.188:5432/awoooi_prod"
  REDIS_URL: "redis://:xxx@192.168.0.188:6380/10"
  HARBOR_USER: "admin"
  HARBOR_PASSWORD: "Wooo_Harbor_2026!"
  ANTHROPIC_API_KEY: "sk-ant-xxx"
EOF

kubeseal --cert=sealed-secrets-cert.pem \
  --format=yaml \
  < /tmp/secrets.yaml \
  > k8s/awoooi-prod/03-sealed-secrets.yaml

# 清理明文
rm /tmp/secrets.yaml

# Step 4: 提交加密版本
git add k8s/awoooi-prod/03-sealed-secrets.yaml
git commit -m "feat(security): migrate to Sealed Secrets"
```

### Phase 3: GitHub Actions 整合 (Day 3-4)

```yaml
# .github/workflows/sync-secrets.yml
name: Sync Secrets from Bitwarden

on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * 0'  # 每週日同步

jobs:
  sync:
    runs-on: [self-hosted, harbor, k8s]
    steps:
      - name: Install Bitwarden CLI
        run: |
          npm install -g @bitwarden/cli
          bw config server https://vault.wooo.work

      - name: Login & Sync
        env:
          BW_PASSWORD: ${{ secrets.BW_MASTER_PASSWORD }}
        run: |
          export BW_SESSION=$(bw login --raw)

          # 取得 Harbor 認證
          HARBOR_USER=$(bw get username "Harbor")
          HARBOR_PASSWORD=$(bw get password "Harbor")

          # 更新 GitHub Secrets
          gh secret set HARBOR_USER --body "$HARBOR_USER"
          gh secret set HARBOR_PASSWORD --body "$HARBOR_PASSWORD"

          echo "✅ Secrets synced from Bitwarden"
```

### Phase 4: 開發者本地環境 (Day 4)

```bash
# 安裝 Bitwarden CLI
brew install bitwarden-cli

# 登入
bw config server https://vault.wooo.work
bw login

# 產生 .env.local
cat > generate-env.sh << 'EOF'
#!/bin/bash
export BW_SESSION=$(bw unlock --raw)

echo "DATABASE_URL=$(bw get password 'PostgreSQL')" > .env.local
echo "REDIS_URL=$(bw get password 'Redis')" >> .env.local
echo "ANTHROPIC_API_KEY=$(bw get password 'Anthropic')" >> .env.local

echo "✅ .env.local generated from Bitwarden"
EOF

chmod +x generate-env.sh
```

---

## 密碼 Rotation SOP

### 變更密碼流程

```
┌─────────────────────────────────────────────────────────────┐
│  密碼變更 SOP (Single Source of Truth)                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Step 1: 在 Bitwarden 更新密碼                              │
│          └── vault.wooo.work → Edit → Save                  │
│                                                             │
│  Step 2: 觸發同步 Workflow                                  │
│          └── gh workflow run sync-secrets.yml               │
│                                                             │
│  Step 3: 重新 Seal K8s Secrets                              │
│          └── ./scripts/reseal-secrets.sh                    │
│                                                             │
│  Step 4: 部署更新                                           │
│          └── kubectl apply -f k8s/awoooi-prod/              │
│                                                             │
│  Step 5: 驗證                                               │
│          └── curl https://awoooi.wooo.work/api/v1/health   │
│                                                             │
└─────────────────────────────────────────────────────────────┘
```

### 自動化 Rotation Script

```bash
#!/bin/bash
# scripts/rotate-secret.sh

SECRET_NAME=$1
NEW_VALUE=$2

if [ -z "$SECRET_NAME" ] || [ -z "$NEW_VALUE" ]; then
  echo "Usage: ./rotate-secret.sh <secret-name> <new-value>"
  exit 1
fi

echo "🔄 Rotating secret: $SECRET_NAME"

# Step 1: Update Bitwarden
export BW_SESSION=$(bw unlock --raw)
bw edit item "$SECRET_NAME" --value "$NEW_VALUE"

# Step 2: Sync to GitHub
gh workflow run sync-secrets.yml

# Step 3: Re-seal K8s secrets
./scripts/reseal-secrets.sh

# Step 4: Apply to cluster
kubectl apply -f k8s/awoooi-prod/03-sealed-secrets.yaml

# Step 5: Rolling restart
kubectl rollout restart deployment -n awoooi-prod

echo "✅ Secret rotated successfully"
```

---

## 緊急撤銷流程

```bash
# 如果密碼洩漏，立即執行：

# 1. 在 Bitwarden 產生新密碼
bw generate -ulns --length 32

# 2. 更新所有相關服務
./scripts/rotate-secret.sh "Harbor" "$(bw generate -ulns --length 32)"

# 3. 撤銷舊 Token (如適用)
# Harbor: Admin UI → Users → Reset Password
# GitHub: Settings → Secrets → Delete old, Add new

# 4. 審計日誌檢查
bw list items --search "Harbor" --pretty
```

---

## 監控與告警

```yaml
# 新增到 Prometheus 告警規則
groups:
  - name: secrets-monitoring
    rules:
      - alert: SecretRotationOverdue
        expr: time() - secret_last_rotation_timestamp > 86400 * 90
        for: 1d
        labels:
          severity: warning
        annotations:
          summary: "Secret {{ $labels.secret_name }} 超過 90 天未輪換"

      - alert: SecretAccessAnomaly
        expr: rate(secret_access_total[1h]) > 100
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "異常的 Secret 存取頻率"
```

---

## 成本分析

| 項目 | 方案 | 成本 |
|------|------|------|
| Secrets 儲存 | Vaultwarden (Self-hosted) | $0 |
| K8s 加密 | Sealed Secrets | $0 |
| 同步工具 | Bitwarden CLI | $0 |
| **總計** | | **$0/年** |

---

## 驗收標準

- [ ] Vaultwarden 部署在 192.168.0.188
- [ ] 所有 Secrets 已匯入 Bitwarden
- [ ] Sealed Secrets Controller 運行中
- [ ] K8s Secrets 已加密存入 Git
- [ ] GitHub Actions 可從 Bitwarden 同步
- [ ] 開發者可用 CLI 產生 .env.local
- [ ] Rotation SOP 文件完成
- [ ] 監控告警已設定

---

## 附錄：Secrets 清單 (機密)

> ⚠️ 此清單僅供內部參考，實際密碼存於 Bitwarden

| 名稱 | 類型 | 最後更新 | 下次 Rotation |
|------|------|----------|---------------|
| Harbor | Registry | 2026-03-23 | 2026-06-23 |
| PostgreSQL | Database | TBD | TBD |
| Redis | Cache | TBD | TBD |
| Anthropic | API Key | TBD | TBD |
| Gemini | API Key | TBD | TBD |
| Telegram Bot | Token | 2026-03-22 | 2026-06-22 |