feat(gitops): Sprint B-2/B-3 — ArgoCD Application + CD GitOps 模式

B-2: k8s/argocd/awoooi-prod-app.yaml
  - ArgoCD Application awoooi-prod 建立(已 apply 到 K8s)
  - automated sync: prune + selfHeal
  - ignoreDifferences: Deployment image + Secret data
  - 全部 17 個 K8s 資源已確認 Synced

B-3: .gitea/workflows/cd.yaml — Deploy step 重寫
  - 舊: kubectl set image(與 ArgoCD selfHeal 衝突)
  - 新: kustomize edit set image → git commit [skip ci] → push → ArgoCD sync
  - 新增等待 ArgoCD Synced + Healthy(最多 120s)
  - 需建立 Gitea Secret: GITEA_CD_TOKEN(見 ADR-069)

docs/adr/ADR-069-infra-gitops-sprint-b.md
  - 決策記錄:循環觸發防護 + ignoreDifferences 設計

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-11 02:57:42 +08:00
parent a63c586d9a
commit 7f4ec717ef
3 changed files with 197 additions and 35 deletions

View File

@@ -320,68 +320,92 @@ jobs:
echo "✅ 所有 Secrets 注入完成"
SECRETS
# 2026-04-01 ogt: 合併 ConfigMap + Deploy + Health Check 為單一 SSH step
# 原本 3 次獨立 SSH 連線 → 節省 ~30s 握手開銷
- name: Deploy to K8s
# 2026-04-11 Claude Sonnet 4.6 (Sprint B-3 ADR-069):
# Deploy 改為 ArgoCD GitOps 模式:更新 kustomization.yaml → git push [skip ci] → ArgoCD sync
# 舊做法 (kubectl set image) 與 ArgoCD selfHeal 衝突 — ArgoCD 會 revert 任何直接 kubectl 操作
# 新做法流程:
# 1. 更新 kustomization.yaml image tag用 kustomize edit set image
# 2. Apply ConfigMap/ServiceRegistry不含 Deployment由 ArgoCD 管)
# 3. git commit [skip ci] + push → 觸發 ArgoCD automated sync
# 4. 等待 ArgoCD sync + rollout 完成
# 5. Health Check
- name: Deploy to K8s (ArgoCD GitOps)
env:
SSH_PRIVATE_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
GITEA_TOKEN: ${{ secrets.GITEA_CD_TOKEN }}
run: |
# 首席架構師 Review C4: Deploy 步驟自行設定 SSH key不依賴 Inject Secrets 的副作用
# S1/S2: 統一命名為 deploy_key改用 ssh-keyscan比 StrictHostKeyChecking=no 更安全)
mkdir -p ~/.ssh
echo "$SSH_PRIVATE_KEY" > ~/.ssh/deploy_key
chmod 600 ~/.ssh/deploy_key
ssh-keyscan 192.168.0.121 >> ~/.ssh/known_hosts 2>/dev/null
# Step 1: Apply ConfigMap (stdin pipe必須獨立)
IMAGE_TAG="${{ github.sha }}"
HARBOR=192.168.0.110:5000
# ─── Step 1: Apply ConfigMap + ServiceRegistry (ArgoCD 管的是 DeploymentConfigMap 仍直接 apply) ───
cat k8s/awoooi-prod/04-configmap.yaml | \
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
echo "✅ ConfigMap 已更新"
# Step 1c: Apply Service Registry ConfigMap (Sprint 5.1 Guardrail)
# 2026-04-08 Claude Sonnet 4.6: 掛載 service-registry.yaml 到容器 /app/ops/config/
cat k8s/awoooi-prod/15-service-registry-configmap.yaml | \
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
echo "✅ Service Registry ConfigMap 已更新"
# Step 1b: Apply Deployment yamls (套用 volumes/resources/probe 等非 image 設定)
# 2026-04-05 Claude Code: 確保 deployment 結構變更(如 SSH key mount持久化到 K8s
# C3 修正 2026-04-05: 先 sed 替換 IMAGE_TAG_PLACEHOLDER 為正確 sha
# 避免 Step 1b 和 Step 2 之間若中斷導致 K8s desired state 含無效 image tag
IMAGE_TAG="${{ github.sha }}"
for f in k8s/awoooi-prod/06-deployment-api.yaml k8s/awoooi-prod/05-deployment-web.yaml k8s/awoooi-prod/08-deployment-worker.yaml; do
sed "s/IMAGE_TAG_PLACEHOLDER/${IMAGE_TAG}/g" "$f" | \
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
done
echo "✅ Deployment yamls 已套用"
# ─── Step 2: 更新 kustomization.yaml image tag ───
# 安裝 kustomize若未安裝
if ! command -v kustomize &>/dev/null; then
curl -sL https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.3.0/kustomize_v5.3.0_linux_amd64.tar.gz | tar xz -C /usr/local/bin
fi
# Step 2: Set images + Rollout + Health Check (合併一次 SSH)
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 << 'DEPLOY'
cd k8s/awoooi-prod
# kustomize edit set image 更新 tag
kustomize edit set image \
192.168.0.110:5000/library/api:IMAGE_TAG_PLACEHOLDER=${HARBOR}/awoooi/api:${IMAGE_TAG}
kustomize edit set image \
192.168.0.110:5000/library/web:IMAGE_TAG_PLACEHOLDER=${HARBOR}/awoooi/web:${IMAGE_TAG}
cd ../..
# ─── Step 3: git commit [skip ci] + push → 觸發 ArgoCD sync ───
git config user.email "cd@awoooi.internal"
git config user.name "AWOOOI CD"
git add k8s/awoooi-prod/kustomization.yaml
git diff --cached --quiet && echo "⚡ kustomization.yaml 無變化,跳過 push" || {
git commit -m "chore(cd): deploy ${IMAGE_TAG::7} [skip ci]"
# 用 token 推送(避免 SSH key 需要額外設定 push 權限)
git remote set-url gitea http://wooo:${GITEA_TOKEN}@192.168.0.110:3001/wooo/awoooi.git
git push gitea main
echo "✅ kustomization.yaml 已 push等待 ArgoCD sync..."
}
# ─── Step 4: 等待 ArgoCD sync + rollout ───
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 << 'ARGOCD_WAIT'
set -e
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
# 2026-03-30 ogt: sudoers NOPASSWD 已設定,無需密碼
sudo kubectl set image deployment/awoooi-api \
api=192.168.0.110:5000/awoooi/api:${{ github.sha }} \
-n awoooi-prod
sudo kubectl set image deployment/awoooi-web \
web=192.168.0.110:5000/awoooi/web:${{ github.sha }} \
-n awoooi-prod
sudo kubectl set image deployment/awoooi-worker \
worker=192.168.0.110:5000/awoooi/api:${{ github.sha }} \
-n awoooi-prod
# 等待 ArgoCD Application Synced最多 120s
echo "⏳ 等待 ArgoCD sync..."
for i in $(seq 1 24); do
SYNC=$(sudo kubectl get application awoooi-prod -n argocd \
-o jsonpath='{.status.sync.status}' 2>/dev/null || echo "Unknown")
HEALTH=$(sudo kubectl get application awoooi-prod -n argocd \
-o jsonpath='{.status.health.status}' 2>/dev/null || echo "Unknown")
echo " ArgoCD: sync=$SYNC health=$HEALTH"
if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then
echo "✅ ArgoCD Synced + Healthy"
break
fi
sleep 5
done
# 確認 rollout 完成
sudo kubectl rollout status deployment/awoooi-api -n awoooi-prod --timeout=120s
sudo kubectl rollout status deployment/awoooi-web -n awoooi-prod --timeout=120s
sudo kubectl rollout status deployment/awoooi-worker -n awoooi-prod --timeout=120s
echo "✅ 部署完成"
# Health Check (同一 SSH session省去再次握手)
# 2026-04-01 Claude Code: 改用 break+flag避免 exit 0 在 heredoc 引發 SIGPIPE
sleep 10
# Health Check
HEALTH_PASS=0
for i in 1 2 3; do
HTTP_CODE=$(curl -s -w "%{http_code}" -o /dev/null --connect-timeout 10 "http://localhost:32334/api/v1/health")
@@ -397,7 +421,7 @@ jobs:
echo "❌ API 健康檢查失敗"
exit 1
fi
DEPLOY
ARGOCD_WAIT
# 2026-04-09 Claude Sonnet 4.6: Sprint 5.2 — 同步 ops 腳本到 188 (ollama user)
# DEPLOY_SSH_KEY_188 = gitea-cd-deploy-188 (ed25519只有 188 authorized_keys)

View File

@@ -0,0 +1,74 @@
# ADR-069: 基礎設施重建 Sprint B — ArgoCD GitOps 部署模式
**狀態**: Accepted
**日期**: 2026-04-11 (台北時間)
**作者**: Claude Sonnet 4.6 + 首席架構師審查
**關聯**: Sprint B (ADR-069), `docs/superpowers/specs/2026-04-10-infra-rebuild-sprint-abc-design.md`
---
## 問題
CD pipeline 直接使用 `kubectl set image` 更新 K8s Deployment與 ArgoCD 的 `selfHeal` 機制衝突:
- ArgoCD 偵測到 Git 狀態 ≠ K8s 狀態時自動 revert
- 無法追蹤部署歷史(誰在何時部署了什麼 image tag
- kustomization.yaml 中的 `IMAGE_TAG_PLACEHOLDER` 從未真正更新到 Git
## 決定
採用 **ArgoCD GitOps 模式**
```
CD Pipeline:
Build → Push to Harbor
→ kustomize edit set image更新 kustomization.yaml
→ git commit [skip ci] + push to Gitea
→ ArgoCD 偵測 Git 變更,自動 sync
→ 等待 ArgoCD Synced + Healthy
→ kubectl rollout status 確認
→ Health Check
```
## 架構
### ArgoCD Application (`k8s/argocd/awoooi-prod-app.yaml`)
| 設定 | 值 | 說明 |
|------|-----|------|
| `automated.prune` | true | 刪除 Git 已移除的資源 |
| `automated.selfHeal` | true | 防止直接 kubectl 操作污染 |
| `syncOptions.ServerSideApply` | true | 支援大型 CRD |
| `ignoreDifferences` | Deployment/Secret | image tag 和 secrets 由 CD 管理 |
### 循環觸發防護
CD pipeline 的 kustomization.yaml 更新 commit 使用 `[skip ci]` 標記,
Gitea Actions 原生支援此標記,不會觸發新的 CD run。
### 新增 Secret
`GITEA_CD_TOKEN`: Gitea Personal Access Token用於 CD pipeline push kustomization.yaml
- 建立Gitea → Settings → Applications → Generate Token
- 權限:`write:repository`
- 加到 Gitea Repository Secrets: Settings → Secrets → Add Secret
## 影響
### 正面
- **完整 GitOps 閉環**:所有部署變更都有 Git commit 記錄
- **防止 Configuration Drift**ArgoCD selfHeal 確保 K8s = Git
- **可回滾**`git revert` kustomization.yaml commit 即可回滾
### 注意事項
- `secrets.GITEA_CD_TOKEN` 需手動在 Gitea 建立並設定
- CD pipeline 現在有 `git push` 操作,需確保 runner 有網路存取 Gitea (192.168.0.110:3001)
- ArgoCD `ignoreDifferences` 排除 Deployment image 欄位,否則 ArgoCD 會顯示 OutOfSync
## 設定清單
- [ ] 在 Gitea 建立 Personal Access Tokenwrite:repository 權限)
- [ ] 加到 Gitea Repository Secrets: `GITEA_CD_TOKEN`
- [ ] 確認 ArgoCD Application 已建立:`kubectl get app awoooi-prod -n argocd`
- [ ] 確認 ArgoCD 可存取 Gitea檢查 ArgoCD Repo Server 網路策略

View File

@@ -0,0 +1,64 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: awoooi-prod
namespace: argocd
# ADR-069 Sprint B-2: ArgoCD 管控 awoooi-prod namespace
# 建立時間: 2026-04-11 (台北時間)
# 建立者: Claude Sonnet 4.6
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: default
source:
repoURL: http://192.168.0.110:3001/wooo/awoooi.git
targetRevision: main
path: k8s/awoooi-prod
# kustomize 模式
kustomize:
# image tag 由 CD pipeline 寫入 kustomization.yaml 後觸發 ArgoCD sync
# 不在此設定 images避免覆蓋 CD 注入的 tag
{}
destination:
server: https://kubernetes.default.svc
namespace: awoooi-prod
syncPolicy:
automated:
# prune: 刪除 Git 中已移除的資源
prune: true
# selfHeal: Git 與 K8s 狀態偏離時自動修復(防止手動 kubectl 污染)
selfHeal: true
# 允許在尚未同步時也自動同步
allowEmpty: false
syncOptions:
# 命名空間不存在時自動建立
- CreateNamespace=true
# 使用 server-side apply支援 CRD 大型資源)
- ServerSideApply=true
# 不強制替換 secrets避免誤刪敏感資料
- RespectIgnoreDifferences=true
retry:
limit: 3
backoff:
duration: 10s
factor: 2
maxDuration: 3m
# 忽略差異項目
ignoreDifferences:
# kustomization.yaml 中的 IMAGE_TAG_PLACEHOLDER 由 CD 動態注入,
# 不計入 ArgoCD drift 偵測(否則每次 push 前都會顯示 OutOfSync
- group: apps
kind: Deployment
jsonPointers:
- /spec/template/spec/containers/0/image
- /spec/template/spec/containers/1/image
# secrets 由 CD 管理,不進入 ArgoCD drift 偵測
- group: ""
kind: Secret
jsonPointers:
- /data