awoooi/docs/adr/ADR-010-secrets-management.md

ADR-010: 集中式 Secrets 管理架構

狀態: 已批准 日期: 2026-03-23 決策者: 統帥

背景

當前痛點

問題：密碼散落各處，變更一處 → 全線崩潰

Harbor 密碼存在位置：
├── GitHub Secrets (wooo-aiops)
├── GitHub Secrets (awoooi)      ← 今天才補上
├── K8s Secrets (awoooi-prod)
├── Self-hosted Runner 環境變數
├── 開發者本地 ~/.docker/config.json
└── 文檔/記憶體 MD 檔案

結果：
- 變更密碼需同步 5+ 個地方
- 經常遺漏 → CI/CD 崩潰
- 無審計日誌
- 無法追蹤誰何時改了什麼

需要管理的 Secrets 清單

類別	Secret 名稱	使用位置
Registry	HARBOR_USER / HARBOR_PASSWORD	CI/CD, K8s
Database	DATABASE_URL (PostgreSQL)	API, Worker
Cache	REDIS_URL	API, Worker, OpenClaw
AI	ANTHROPIC_API_KEY	API (Agent Teams)
AI	GEMINI_API_KEY	API (Fallback)
AI	OLLAMA_URL	API (Local LLM)
Notification	OPENCLAW_TG_BOT_TOKEN	OpenClaw
Notification	OPENCLAW_TG_CHAT_ID	OpenClaw
K8s	KUBECONFIG	CI/CD
Webhook	WEBHOOK_HMAC_SECRET	API

---

決策

採用 混合式 Secrets 管理架構：

1. Bitwarden (Self-hosted) - Single Source of Truth
2. Sealed Secrets - K8s 內部密碼加密
3. GitHub Secrets - CI/CD 專用 (從 Bitwarden 同步)

---

架構設計

┌─────────────────────────────────────────────────────────────────────┐
│                    AWOOOI Secrets 管理架構                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │  Bitwarden (Self-hosted on 192.168.0.188)                     │  │
│  │  ══════════════════════════════════════                       │  │
│  │  🔐 Single Source of Truth                                    │  │
│  │                                                               │  │
│  │  Organizations:                                               │  │
│  │  └── AWOOOI/                                                  │  │
│  │      ├── Infrastructure/                                      │  │
│  │      │   ├── Harbor (admin / Wooo_Harbor_2026!)              │  │
│  │      │   ├── PostgreSQL (awoooi / ******)                    │  │
│  │      │   └── Redis (password / ******)                       │  │
│  │      ├── AI-Services/                                         │  │
│  │      │   ├── Anthropic API Key                               │  │
│  │      │   ├── Gemini API Key                                  │  │
│  │      │   └── Ollama URL                                      │  │
│  │      └── Notifications/                                       │  │
│  │          ├── Telegram Bot Token                              │  │
│  │          └── Telegram Chat ID                                │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│              ┌───────────────┼───────────────┐                      │
│              │               │               │                      │
│              ▼               ▼               ▼                      │
│  ┌─────────────────┐ ┌─────────────┐ ┌─────────────────┐           │
│  │ GitHub Secrets  │ │ Sealed      │ │ Local Dev       │           │
│  │ (CI/CD)         │ │ Secrets     │ │ (.env.local)    │           │
│  │                 │ │ (K8s)       │ │                 │           │
│  │ HARBOR_USER     │ │             │ │ bw get password │           │
│  │ HARBOR_PASSWORD │ │ 加密存 Git  │ │ "Harbor"        │           │
│  │ OP_TOKEN (sync) │ │ 自動解密    │ │                 │           │
│  └────────┬────────┘ └──────┬──────┘ └─────────────────┘           │
│           │                 │                                       │
│           ▼                 ▼                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  K3s Cluster (192.168.0.120/121)                            │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │  awoooi-prod namespace                              │    │   │
│  │  │  ├── awoooi-secrets (from Sealed Secrets)           │    │   │
│  │  │  ├── awoooi-config (ConfigMap, non-sensitive)       │    │   │
│  │  │  └── Pods (consume secrets as env vars)             │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

實施計畫

Phase 1: Bitwarden Self-hosted (Day 1-2)

# 在 192.168.0.188 部署 Vaultwarden (輕量版 Bitwarden)
docker run -d --name vaultwarden \
  -v /data/vaultwarden:/data \
  -e ADMIN_TOKEN='your-admin-token' \
  -e DOMAIN='https://vault.wooo.work' \
  -p 8088:80 \
  vaultwarden/server:latest

# Nginx 設定
server {
    listen 443 ssl;
    server_name vault.wooo.work;

    location / {
        proxy_pass http://localhost:8088;
    }
}

初始化步驟：

1. 訪問 https://vault.wooo.work
2. 建立 Organization: AWOOOI
3. 建立 Collections: Infrastructure, AI-Services, Notifications
4. 匯入現有密碼

Phase 2: Sealed Secrets (Day 2-3)

# Step 1: 安裝 Sealed Secrets Controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.26.0/controller.yaml

# Step 2: 取得 Public Key
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > /Users/ogt/awoooi/k8s/sealed-secrets-cert.pem

# Step 3: 加密現有 Secrets
cat <<EOF > /tmp/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: awoooi-secrets
  namespace: awoooi-prod
type: Opaque
stringData:
  DATABASE_URL: "postgresql+asyncpg://awoooi:xxx@192.168.0.188:5432/awoooi_prod"
  REDIS_URL: "redis://:xxx@192.168.0.188:6380/10"
  HARBOR_USER: "admin"
  HARBOR_PASSWORD: "Wooo_Harbor_2026!"
  ANTHROPIC_API_KEY: "sk-ant-xxx"
EOF

kubeseal --cert=sealed-secrets-cert.pem \
  --format=yaml \
  < /tmp/secrets.yaml \
  > k8s/awoooi-prod/03-sealed-secrets.yaml

# 清理明文
rm /tmp/secrets.yaml

# Step 4: 提交加密版本
git add k8s/awoooi-prod/03-sealed-secrets.yaml
git commit -m "feat(security): migrate to Sealed Secrets"

Phase 3: GitHub Actions 整合 (Day 3-4)

# .github/workflows/sync-secrets.yml
name: Sync Secrets from Bitwarden

on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * 0'  # 每週日同步

jobs:
  sync:
    runs-on: [self-hosted, harbor, k8s]
    steps:
      - name: Install Bitwarden CLI
        run: |
          npm install -g @bitwarden/cli
          bw config server https://vault.wooo.work

      - name: Login & Sync
        env:
          BW_PASSWORD: ${{ secrets.BW_MASTER_PASSWORD }}
        run: |
          export BW_SESSION=$(bw login --raw)

          # 取得 Harbor 認證
          HARBOR_USER=$(bw get username "Harbor")
          HARBOR_PASSWORD=$(bw get password "Harbor")

          # 更新 GitHub Secrets
          gh secret set HARBOR_USER --body "$HARBOR_USER"
          gh secret set HARBOR_PASSWORD --body "$HARBOR_PASSWORD"

          echo "✅ Secrets synced from Bitwarden"

Phase 4: 開發者本地環境 (Day 4)

# 安裝 Bitwarden CLI
brew install bitwarden-cli

# 登入
bw config server https://vault.wooo.work
bw login

# 產生 .env.local
cat > generate-env.sh << 'EOF'
#!/bin/bash
export BW_SESSION=$(bw unlock --raw)

echo "DATABASE_URL=$(bw get password 'PostgreSQL')" > .env.local
echo "REDIS_URL=$(bw get password 'Redis')" >> .env.local
echo "ANTHROPIC_API_KEY=$(bw get password 'Anthropic')" >> .env.local

echo "✅ .env.local generated from Bitwarden"
EOF

chmod +x generate-env.sh

---

密碼 Rotation SOP

變更密碼流程

┌─────────────────────────────────────────────────────────────┐
│  密碼變更 SOP (Single Source of Truth)                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Step 1: 在 Bitwarden 更新密碼                              │
│          └── vault.wooo.work → Edit → Save                  │
│                                                             │
│  Step 2: 觸發同步 Workflow                                  │
│          └── gh workflow run sync-secrets.yml               │
│                                                             │
│  Step 3: 重新 Seal K8s Secrets                              │
│          └── ./scripts/reseal-secrets.sh                    │
│                                                             │
│  Step 4: 部署更新                                           │
│          └── kubectl apply -f k8s/awoooi-prod/              │
│                                                             │
│  Step 5: 驗證                                               │
│          └── curl https://awoooi.wooo.work/api/v1/health   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

自動化 Rotation Script

#!/bin/bash
# scripts/rotate-secret.sh

SECRET_NAME=$1
NEW_VALUE=$2

if [ -z "$SECRET_NAME" ] || [ -z "$NEW_VALUE" ]; then
  echo "Usage: ./rotate-secret.sh <secret-name> <new-value>"
  exit 1
fi

echo "🔄 Rotating secret: $SECRET_NAME"

# Step 1: Update Bitwarden
export BW_SESSION=$(bw unlock --raw)
bw edit item "$SECRET_NAME" --value "$NEW_VALUE"

# Step 2: Sync to GitHub
gh workflow run sync-secrets.yml

# Step 3: Re-seal K8s secrets
./scripts/reseal-secrets.sh

# Step 4: Apply to cluster
kubectl apply -f k8s/awoooi-prod/03-sealed-secrets.yaml

# Step 5: Rolling restart
kubectl rollout restart deployment -n awoooi-prod

echo "✅ Secret rotated successfully"

---

緊急撤銷流程

# 如果密碼洩漏，立即執行：

# 1. 在 Bitwarden 產生新密碼
bw generate -ulns --length 32

# 2. 更新所有相關服務
./scripts/rotate-secret.sh "Harbor" "$(bw generate -ulns --length 32)"

# 3. 撤銷舊 Token (如適用)
# Harbor: Admin UI → Users → Reset Password
# GitHub: Settings → Secrets → Delete old, Add new

# 4. 審計日誌檢查
bw list items --search "Harbor" --pretty

---

監控與告警

# 新增到 Prometheus 告警規則
groups:
  - name: secrets-monitoring
    rules:
      - alert: SecretRotationOverdue
        expr: time() - secret_last_rotation_timestamp > 86400 * 90
        for: 1d
        labels:
          severity: warning
        annotations:
          summary: "Secret {{ $labels.secret_name }} 超過 90 天未輪換"

      - alert: SecretAccessAnomaly
        expr: rate(secret_access_total[1h]) > 100
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "異常的 Secret 存取頻率"

---

成本分析

項目	方案	成本
Secrets 儲存	Vaultwarden (Self-hosted)	$0
K8s 加密	Sealed Secrets	$0
同步工具	Bitwarden CLI	$0
總計		$0/年

---

驗收標準

• Vaultwarden 部署在 192.168.0.188
• 所有 Secrets 已匯入 Bitwarden
• Sealed Secrets Controller 運行中
• K8s Secrets 已加密存入 Git
• GitHub Actions 可從 Bitwarden 同步
• 開發者可用 CLI 產生 .env.local
• Rotation SOP 文件完成
• 監控告警已設定

---

附錄：Secrets 清單 (機密)

⚠️ 此清單僅供內部參考，實際密碼存於 Bitwarden

名稱	類型	最後更新	下次 Rotation
Harbor	Registry	2026-03-23	2026-06-23
PostgreSQL	Database	TBD	TBD
Redis	Cache	TBD	TBD
Anthropic	API Key	TBD	TBD
Gemini	API Key	TBD	TBD
Telegram Bot	Token	2026-03-22	2026-06-22