feat(gitops): Sprint B-2/B-3 — ArgoCD Application + CD GitOps 模式
B-2: k8s/argocd/awoooi-prod-app.yaml - ArgoCD Application awoooi-prod 建立(已 apply 到 K8s) - automated sync: prune + selfHeal - ignoreDifferences: Deployment image + Secret data - 全部 17 個 K8s 資源已確認 Synced B-3: .gitea/workflows/cd.yaml — Deploy step 重寫 - 舊: kubectl set image(與 ArgoCD selfHeal 衝突) - 新: kustomize edit set image → git commit [skip ci] → push → ArgoCD sync - 新增等待 ArgoCD Synced + Healthy(最多 120s) - 需建立 Gitea Secret: GITEA_CD_TOKEN(見 ADR-069) docs/adr/ADR-069-infra-gitops-sprint-b.md - 決策記錄:循環觸發防護 + ignoreDifferences 設計 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -320,68 +320,92 @@ jobs:
|
||||
echo "✅ 所有 Secrets 注入完成"
|
||||
SECRETS
|
||||
|
||||
# 2026-04-01 ogt: 合併 ConfigMap + Deploy + Health Check 為單一 SSH step
|
||||
# 原本 3 次獨立 SSH 連線 → 節省 ~30s 握手開銷
|
||||
- name: Deploy to K8s
|
||||
# 2026-04-11 Claude Sonnet 4.6 (Sprint B-3 ADR-069):
|
||||
# Deploy 改為 ArgoCD GitOps 模式:更新 kustomization.yaml → git push [skip ci] → ArgoCD sync
|
||||
# 舊做法 (kubectl set image) 與 ArgoCD selfHeal 衝突 — ArgoCD 會 revert 任何直接 kubectl 操作
|
||||
# 新做法流程:
|
||||
# 1. 更新 kustomization.yaml image tag(用 kustomize edit set image)
|
||||
# 2. Apply ConfigMap/ServiceRegistry(不含 Deployment,由 ArgoCD 管)
|
||||
# 3. git commit [skip ci] + push → 觸發 ArgoCD automated sync
|
||||
# 4. 等待 ArgoCD sync + rollout 完成
|
||||
# 5. Health Check
|
||||
- name: Deploy to K8s (ArgoCD GitOps)
|
||||
env:
|
||||
SSH_PRIVATE_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_CD_TOKEN }}
|
||||
run: |
|
||||
# 首席架構師 Review C4: Deploy 步驟自行設定 SSH key,不依賴 Inject Secrets 的副作用
|
||||
# S1/S2: 統一命名為 deploy_key,改用 ssh-keyscan(比 StrictHostKeyChecking=no 更安全)
|
||||
mkdir -p ~/.ssh
|
||||
echo "$SSH_PRIVATE_KEY" > ~/.ssh/deploy_key
|
||||
chmod 600 ~/.ssh/deploy_key
|
||||
ssh-keyscan 192.168.0.121 >> ~/.ssh/known_hosts 2>/dev/null
|
||||
|
||||
# Step 1: Apply ConfigMap (stdin pipe,必須獨立)
|
||||
IMAGE_TAG="${{ github.sha }}"
|
||||
HARBOR=192.168.0.110:5000
|
||||
|
||||
# ─── Step 1: Apply ConfigMap + ServiceRegistry (ArgoCD 管的是 Deployment,ConfigMap 仍直接 apply) ───
|
||||
cat k8s/awoooi-prod/04-configmap.yaml | \
|
||||
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
|
||||
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
|
||||
echo "✅ ConfigMap 已更新"
|
||||
|
||||
# Step 1c: Apply Service Registry ConfigMap (Sprint 5.1 Guardrail)
|
||||
# 2026-04-08 Claude Sonnet 4.6: 掛載 service-registry.yaml 到容器 /app/ops/config/
|
||||
cat k8s/awoooi-prod/15-service-registry-configmap.yaml | \
|
||||
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
|
||||
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
|
||||
echo "✅ Service Registry ConfigMap 已更新"
|
||||
|
||||
# Step 1b: Apply Deployment yamls (套用 volumes/resources/probe 等非 image 設定)
|
||||
# 2026-04-05 Claude Code: 確保 deployment 結構變更(如 SSH key mount)持久化到 K8s
|
||||
# C3 修正 2026-04-05: 先 sed 替換 IMAGE_TAG_PLACEHOLDER 為正確 sha,
|
||||
# 避免 Step 1b 和 Step 2 之間若中斷導致 K8s desired state 含無效 image tag
|
||||
IMAGE_TAG="${{ github.sha }}"
|
||||
for f in k8s/awoooi-prod/06-deployment-api.yaml k8s/awoooi-prod/05-deployment-web.yaml k8s/awoooi-prod/08-deployment-worker.yaml; do
|
||||
sed "s/IMAGE_TAG_PLACEHOLDER/${IMAGE_TAG}/g" "$f" | \
|
||||
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 \
|
||||
"export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && sudo kubectl apply -f -"
|
||||
done
|
||||
echo "✅ Deployment yamls 已套用"
|
||||
# ─── Step 2: 更新 kustomization.yaml image tag ───
|
||||
# 安裝 kustomize(若未安裝)
|
||||
if ! command -v kustomize &>/dev/null; then
|
||||
curl -sL https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.3.0/kustomize_v5.3.0_linux_amd64.tar.gz | tar xz -C /usr/local/bin
|
||||
fi
|
||||
|
||||
# Step 2: Set images + Rollout + Health Check (合併一次 SSH)
|
||||
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 << 'DEPLOY'
|
||||
cd k8s/awoooi-prod
|
||||
# kustomize edit set image 更新 tag
|
||||
kustomize edit set image \
|
||||
192.168.0.110:5000/library/api:IMAGE_TAG_PLACEHOLDER=${HARBOR}/awoooi/api:${IMAGE_TAG}
|
||||
kustomize edit set image \
|
||||
192.168.0.110:5000/library/web:IMAGE_TAG_PLACEHOLDER=${HARBOR}/awoooi/web:${IMAGE_TAG}
|
||||
cd ../..
|
||||
|
||||
# ─── Step 3: git commit [skip ci] + push → 觸發 ArgoCD sync ───
|
||||
git config user.email "cd@awoooi.internal"
|
||||
git config user.name "AWOOOI CD"
|
||||
git add k8s/awoooi-prod/kustomization.yaml
|
||||
git diff --cached --quiet && echo "⚡ kustomization.yaml 無變化,跳過 push" || {
|
||||
git commit -m "chore(cd): deploy ${IMAGE_TAG::7} [skip ci]"
|
||||
# 用 token 推送(避免 SSH key 需要額外設定 push 權限)
|
||||
git remote set-url gitea http://wooo:${GITEA_TOKEN}@192.168.0.110:3001/wooo/awoooi.git
|
||||
git push gitea main
|
||||
echo "✅ kustomization.yaml 已 push,等待 ArgoCD sync..."
|
||||
}
|
||||
|
||||
# ─── Step 4: 等待 ArgoCD sync + rollout ───
|
||||
ssh -i ~/.ssh/deploy_key wooo@192.168.0.121 << 'ARGOCD_WAIT'
|
||||
set -e
|
||||
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
||||
|
||||
# 2026-03-30 ogt: sudoers NOPASSWD 已設定,無需密碼
|
||||
sudo kubectl set image deployment/awoooi-api \
|
||||
api=192.168.0.110:5000/awoooi/api:${{ github.sha }} \
|
||||
-n awoooi-prod
|
||||
sudo kubectl set image deployment/awoooi-web \
|
||||
web=192.168.0.110:5000/awoooi/web:${{ github.sha }} \
|
||||
-n awoooi-prod
|
||||
sudo kubectl set image deployment/awoooi-worker \
|
||||
worker=192.168.0.110:5000/awoooi/api:${{ github.sha }} \
|
||||
-n awoooi-prod
|
||||
# 等待 ArgoCD Application Synced(最多 120s)
|
||||
echo "⏳ 等待 ArgoCD sync..."
|
||||
for i in $(seq 1 24); do
|
||||
SYNC=$(sudo kubectl get application awoooi-prod -n argocd \
|
||||
-o jsonpath='{.status.sync.status}' 2>/dev/null || echo "Unknown")
|
||||
HEALTH=$(sudo kubectl get application awoooi-prod -n argocd \
|
||||
-o jsonpath='{.status.health.status}' 2>/dev/null || echo "Unknown")
|
||||
echo " ArgoCD: sync=$SYNC health=$HEALTH"
|
||||
if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then
|
||||
echo "✅ ArgoCD Synced + Healthy"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# 確認 rollout 完成
|
||||
sudo kubectl rollout status deployment/awoooi-api -n awoooi-prod --timeout=120s
|
||||
sudo kubectl rollout status deployment/awoooi-web -n awoooi-prod --timeout=120s
|
||||
sudo kubectl rollout status deployment/awoooi-worker -n awoooi-prod --timeout=120s
|
||||
echo "✅ 部署完成"
|
||||
|
||||
# Health Check (同一 SSH session,省去再次握手)
|
||||
# 2026-04-01 Claude Code: 改用 break+flag,避免 exit 0 在 heredoc 引發 SIGPIPE
|
||||
sleep 10
|
||||
# Health Check
|
||||
HEALTH_PASS=0
|
||||
for i in 1 2 3; do
|
||||
HTTP_CODE=$(curl -s -w "%{http_code}" -o /dev/null --connect-timeout 10 "http://localhost:32334/api/v1/health")
|
||||
@@ -397,7 +421,7 @@ jobs:
|
||||
echo "❌ API 健康檢查失敗"
|
||||
exit 1
|
||||
fi
|
||||
DEPLOY
|
||||
ARGOCD_WAIT
|
||||
|
||||
# 2026-04-09 Claude Sonnet 4.6: Sprint 5.2 — 同步 ops 腳本到 188 (ollama user)
|
||||
# DEPLOY_SSH_KEY_188 = gitea-cd-deploy-188 (ed25519,只有 188 authorized_keys)
|
||||
|
||||
74
docs/adr/ADR-069-infra-gitops-sprint-b.md
Normal file
74
docs/adr/ADR-069-infra-gitops-sprint-b.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# ADR-069: 基礎設施重建 Sprint B — ArgoCD GitOps 部署模式
|
||||
|
||||
**狀態**: Accepted
|
||||
**日期**: 2026-04-11 (台北時間)
|
||||
**作者**: Claude Sonnet 4.6 + 首席架構師審查
|
||||
**關聯**: Sprint B (ADR-069), `docs/superpowers/specs/2026-04-10-infra-rebuild-sprint-abc-design.md`
|
||||
|
||||
---
|
||||
|
||||
## 問題
|
||||
|
||||
CD pipeline 直接使用 `kubectl set image` 更新 K8s Deployment,與 ArgoCD 的 `selfHeal` 機制衝突:
|
||||
- ArgoCD 偵測到 Git 狀態 ≠ K8s 狀態時自動 revert
|
||||
- 無法追蹤部署歷史(誰在何時部署了什麼 image tag)
|
||||
- kustomization.yaml 中的 `IMAGE_TAG_PLACEHOLDER` 從未真正更新到 Git
|
||||
|
||||
## 決定
|
||||
|
||||
採用 **ArgoCD GitOps 模式**:
|
||||
|
||||
```
|
||||
CD Pipeline:
|
||||
Build → Push to Harbor
|
||||
→ kustomize edit set image(更新 kustomization.yaml)
|
||||
→ git commit [skip ci] + push to Gitea
|
||||
→ ArgoCD 偵測 Git 變更,自動 sync
|
||||
→ 等待 ArgoCD Synced + Healthy
|
||||
→ kubectl rollout status 確認
|
||||
→ Health Check
|
||||
```
|
||||
|
||||
## 架構
|
||||
|
||||
### ArgoCD Application (`k8s/argocd/awoooi-prod-app.yaml`)
|
||||
|
||||
| 設定 | 值 | 說明 |
|
||||
|------|-----|------|
|
||||
| `automated.prune` | true | 刪除 Git 已移除的資源 |
|
||||
| `automated.selfHeal` | true | 防止直接 kubectl 操作污染 |
|
||||
| `syncOptions.ServerSideApply` | true | 支援大型 CRD |
|
||||
| `ignoreDifferences` | Deployment/Secret | image tag 和 secrets 由 CD 管理 |
|
||||
|
||||
### 循環觸發防護
|
||||
|
||||
CD pipeline 的 kustomization.yaml 更新 commit 使用 `[skip ci]` 標記,
|
||||
Gitea Actions 原生支援此標記,不會觸發新的 CD run。
|
||||
|
||||
### 新增 Secret
|
||||
|
||||
`GITEA_CD_TOKEN`: Gitea Personal Access Token,用於 CD pipeline push kustomization.yaml
|
||||
- 建立:Gitea → Settings → Applications → Generate Token
|
||||
- 權限:`write:repository`
|
||||
- 加到 Gitea Repository Secrets: Settings → Secrets → Add Secret
|
||||
|
||||
## 影響
|
||||
|
||||
### 正面
|
||||
|
||||
- **完整 GitOps 閉環**:所有部署變更都有 Git commit 記錄
|
||||
- **防止 Configuration Drift**:ArgoCD selfHeal 確保 K8s = Git
|
||||
- **可回滾**:`git revert` kustomization.yaml commit 即可回滾
|
||||
|
||||
### 注意事項
|
||||
|
||||
- `secrets.GITEA_CD_TOKEN` 需手動在 Gitea 建立並設定
|
||||
- CD pipeline 現在有 `git push` 操作,需確保 runner 有網路存取 Gitea (192.168.0.110:3001)
|
||||
- ArgoCD `ignoreDifferences` 排除 Deployment image 欄位,否則 ArgoCD 會顯示 OutOfSync
|
||||
|
||||
## 設定清單
|
||||
|
||||
- [ ] 在 Gitea 建立 Personal Access Token(write:repository 權限)
|
||||
- [ ] 加到 Gitea Repository Secrets: `GITEA_CD_TOKEN`
|
||||
- [ ] 確認 ArgoCD Application 已建立:`kubectl get app awoooi-prod -n argocd`
|
||||
- [ ] 確認 ArgoCD 可存取 Gitea:檢查 ArgoCD Repo Server 網路策略
|
||||
64
k8s/argocd/awoooi-prod-app.yaml
Normal file
64
k8s/argocd/awoooi-prod-app.yaml
Normal file
@@ -0,0 +1,64 @@
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: awoooi-prod
|
||||
namespace: argocd
|
||||
# ADR-069 Sprint B-2: ArgoCD 管控 awoooi-prod namespace
|
||||
# 建立時間: 2026-04-11 (台北時間)
|
||||
# 建立者: Claude Sonnet 4.6
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "0"
|
||||
spec:
|
||||
project: default
|
||||
|
||||
source:
|
||||
repoURL: http://192.168.0.110:3001/wooo/awoooi.git
|
||||
targetRevision: main
|
||||
path: k8s/awoooi-prod
|
||||
|
||||
# kustomize 模式
|
||||
kustomize:
|
||||
# image tag 由 CD pipeline 寫入 kustomization.yaml 後觸發 ArgoCD sync
|
||||
# 不在此設定 images,避免覆蓋 CD 注入的 tag
|
||||
{}
|
||||
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: awoooi-prod
|
||||
|
||||
syncPolicy:
|
||||
automated:
|
||||
# prune: 刪除 Git 中已移除的資源
|
||||
prune: true
|
||||
# selfHeal: Git 與 K8s 狀態偏離時自動修復(防止手動 kubectl 污染)
|
||||
selfHeal: true
|
||||
# 允許在尚未同步時也自動同步
|
||||
allowEmpty: false
|
||||
syncOptions:
|
||||
# 命名空間不存在時自動建立
|
||||
- CreateNamespace=true
|
||||
# 使用 server-side apply(支援 CRD 大型資源)
|
||||
- ServerSideApply=true
|
||||
# 不強制替換 secrets(避免誤刪敏感資料)
|
||||
- RespectIgnoreDifferences=true
|
||||
retry:
|
||||
limit: 3
|
||||
backoff:
|
||||
duration: 10s
|
||||
factor: 2
|
||||
maxDuration: 3m
|
||||
|
||||
# 忽略差異項目
|
||||
ignoreDifferences:
|
||||
# kustomization.yaml 中的 IMAGE_TAG_PLACEHOLDER 由 CD 動態注入,
|
||||
# 不計入 ArgoCD drift 偵測(否則每次 push 前都會顯示 OutOfSync)
|
||||
- group: apps
|
||||
kind: Deployment
|
||||
jsonPointers:
|
||||
- /spec/template/spec/containers/0/image
|
||||
- /spec/template/spec/containers/1/image
|
||||
# secrets 由 CD 管理,不進入 ArgoCD drift 偵測
|
||||
- group: ""
|
||||
kind: Secret
|
||||
jsonPointers:
|
||||
- /data
|
||||
Reference in New Issue
Block a user