ewoooc/DRIFT_SCANNER_FIX_GUIDE.md

Drift Scanner Pods Fix Guide

Problem Analysis

You have multiple drift-scanner pods that are failing or completed:

• 6 Failed pods: drift-scanner-29602260-sns8t, drift-scanner-29602320-vf6dj, etc.
• 3 Succeeded pods: drift-scanner-29613600-x67jn, etc.

Root Cause

These drift-scanner pods appear to be external Kubernetes Jobs (not part of your codebase) that are created by an external system or cron job. The numeric suffixes suggest time-based job scheduling.

Immediate Fix Actions

Option 1: Manual Cleanup (Quick Fix)

# SSH to your K8s server
ssh wooo@192.168.0.110

# Delete failed pods
sudo kubectl delete pod drift-scanner-29602260-sns8t -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29602320-vf6dj -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29602380-862vh -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29602440-mwd7m -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29602500-gpr27 -n momo --force --grace-period=0

# Delete old succeeded pods (optional)
sudo kubectl delete pod drift-scanner-29613600-x67jn -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29613660-7tk4d -n momo --force --grace-period=0
sudo kubectl delete pod drift-scanner-29613720-c7zcp -n momo --force --grace-period=0

Option 2: Use Cleanup Script

# On K8s server
cd /home/wooo/scripts
./cleanup_drift_scanner_pods.sh cleanup-failed

Option 3: Batch Cleanup

# Delete all drift-scanner pods at once
sudo kubectl delete pods -l app=drift-scanner -n momo --force --grace-period=0

Prevention Strategies

1. Identify the Source

Find what's creating these drift-scanner jobs:

# Check for CronJobs
sudo kubectl get cronjobs -n momo
sudo kubectl get cronjobs --all-namespaces | grep drift

# Check for scheduled jobs
sudo kubectl get jobs -n momo | grep drift
sudo kubectl get jobs --all-namespaces | grep drift

# Check events
sudo kubectl get events -n momo --sort-by='.lastTimestamp' | grep drift

2. Monitor and Auto-Cleanup

Add to your existing health monitoring script (/home/wooo/scripts/k8s_health_monitor.sh):

# Add this function to the script
check_drift_scanner_pods() {
    local drift_pods=$(sudo kubectl get pods -n ${MOMO_NAMESPACE} --no-headers | grep drift-scanner || echo "")
    
    if [[ -n "$drift_pods" ]]; then
        local failed_count=$(echo "$drift_pods" | grep "Failed" | wc -l)
        local succeeded_count=$(echo "$drift_pods" | grep "Succeeded" | wc -l)
        
        if [[ $failed_count -gt 5 ]] || [[ $succeeded_count -gt 10 ]]; then
            log "WARNING: Too many drift-scanner pods (Failed: $failed_count, Succeeded: $succeeded_count)"
            
            # Auto-cleanup failed pods
            echo "$drift_pods" | grep "Failed" | awk '{print $1}' | xargs -r sudo kubectl delete pod -n ${MOMO_NAMESPACE} --force --grace-period=0
            
            # Auto-cleanup old succeeded pods (older than 24h)
            echo "$drift_pods" | grep "Succeeded" | awk '{print $1}' | xargs -r sudo kubectl delete pod -n ${MOMO_NAMESPACE} --force --grace-period=0
        fi
    fi
}

3. Resource Limits

If these are legitimate jobs, consider setting resource limits and TTL:

# Example Job template with TTL
apiVersion: batch/v1
kind: Job
metadata:
  name: drift-scanner
spec:
  ttlSecondsAfterFinished: 3600  # Clean up after 1 hour
  backoffLimit: 3               # Limit retries
  template:
    spec:
      containers:
      - name: drift-scanner
        image: your-image
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
      restartPolicy: OnFailure

Monitoring Setup

Add to Prometheus Monitoring

Create alert rule in /home/wooo/monitoring/prometheus.yml:

- alert: TooManyFailedDriftScannerPods
  expr: kube_pod_status_phase{phase="Failed", pod=~"drift-scanner-.*"} > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Too many failed drift-scanner pods"
    description: "More than 5 drift-scanner pods have failed"

Telegram Alert Integration

The cleanup script already includes Telegram notifications when pods are cleaned up.

Long-term Solutions

1. Identify the Owner: Find which system or team is creating these drift-scanner jobs
2. Fix the Root Cause: Address why these jobs are failing
3. Implement TTL: Add ttlSecondsAfterFinished to job specifications
4. Resource Quotas: Set limits to prevent resource exhaustion
5. Regular Cleanup: Schedule the cleanup script to run periodically

Emergency Commands

# Quick check of drift-scanner status
sudo kubectl get pods -n momo | grep drift-scanner

# Force delete all drift-scanner pods
sudo kubectl delete pods -n momo --all --force --grace-period=0 --selector=app=drift-scanner

# Check what's creating them
sudo kubectl get events -n momo --sort-by='.lastTimestamp' | tail -20

Files Created

• scripts/cleanup_drift_scanner_pods.sh - Comprehensive cleanup script
• DRIFT_SCANNER_FIX_GUIDE.md - This guide

Next Steps

1. Run the immediate cleanup commands
2. Identify the source of drift-scanner jobs
3. Implement prevention measures
4. Set up monitoring and auto-cleanup