Building a Secure Kubernetes CI/CD Pipeline with Jenkins, Argo CD, Trivy, and SonarQube

Most Kubernetes deployment pipelines are optimized for speed.

Very few are optimized for security.

The result is predictable:

• →Containers with critical CVEs reaching production
• →Secrets leaking into Git repositories
• →Developers bypassing quality gates to speed up releases
• →Misconfigured RBAC policies exposing cluster access
• →Helm charts deploying insecure defaults
• →No rollback visibility when deployments fail

In one environment, we discovered workloads running with:

securityContext:
  privileged: true

inside production namespaces.

Nobody noticed because the pipeline validated deployment success — not workload security.

This article covers how we designed a secure Kubernetes CI/CD pipeline using:

• →Jenkins
• →Argo CD
• →Trivy
• →SonarQube
• →GitHub
• →EKS
• →Helm
• →Kubernetes RBAC

The goal was simple:

Prevent insecure workloads from ever reaching production.

---

The Problem with Traditional CI/CD Pipelines

A typical Kubernetes deployment flow looks like this:

Developer → Git Push → Jenkins Build → Docker Image → kubectl apply

The pipeline works.

But it usually misses:

• →Static code analysis
• →Container vulnerability scanning
• →Secret detection
• →Image signing
• →Deployment policy enforcement
• →Runtime validation
• →RBAC auditing
• →Drift detection

The biggest issue is that most validations happen too late.

Security reviews become manual bottlenecks instead of automated controls.

---

Secure CI/CD Architecture

Our production workflow looked like this:

Developer Commit
      ↓
GitHub Pull Request
      ↓
SonarQube Static Analysis
      ↓
Jenkins CI Pipeline
      ↓
Unit Tests + Build Validation
      ↓
Docker Image Build
      ↓
Trivy Vulnerability Scan
      ↓
Image Push to ECR
      ↓
GitOps Manifest Update
      ↓
Argo CD Sync
      ↓
Kubernetes Admission Policies
      ↓
Production Deployment

The important design decision:

Deployments were only allowed through GitOps.

Direct kubectl-based production access was removed for application teams.

This reduced:

• →configuration drift
• →unauthorized deployments
• →inconsistent rollbacks
• →manual production changes

---

CI Pipeline Design with Jenkins

The Jenkins pipeline handled:

• →code validation
• →artifact creation
• →security scanning
• →GitOps updates

Example Jenkins pipeline:

pipeline {
    agent any

    environment {
        IMAGE = "backend-service:${BUILD_NUMBER}"
    }

    stages {
        stage('Code Analysis') {
            steps {
                sh 'sonar-scanner'
            }
        }

        stage('Build Image') {
            steps {
                sh 'docker build -t $IMAGE .'
            }
        }

        stage('Trivy Scan') {
            steps {
                sh 'trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE'
            }
        }

        stage('Push Image') {
            steps {
                sh 'docker push $IMAGE'
            }
        }
    }
}

The critical configuration:

--exit-code 1

This forced the build to fail if critical vulnerabilities were detected.

Without this, security scans become informational dashboards instead of enforcement mechanisms.

---

Static Code Analysis with SonarQube

SonarQube was integrated early in the pipeline.

We used it for:

• →code quality checks
• →duplicated code detection
• →security hotspot analysis
• →maintainability scoring
• →technical debt tracking

Example quality gate:

- Coverage > 80%
- No blocker vulnerabilities
- No critical code smells
- Maintainability rating: A

This immediately reduced:

• →broken deployments
• →risky merges
• →unstable releases

One major improvement came from blocking hardcoded credentials during pull request validation.

Developers occasionally committed:

AWS_SECRET_ACCESS_KEY="example-key"

before pre-commit scanning was enforced.

---

Container Security Scanning with Trivy

Trivy became one of the most important components in the pipeline.

We scanned:

• →container images
• →filesystem packages
• →IaC manifests
• →Kubernetes configurations

Example:

trivy image backend-service:v1.4.2

Common findings included:

• →outdated Alpine packages
• →vulnerable OpenSSL libraries
• →deprecated Python dependencies
• →exposed package managers inside runtime containers

One production image contained:

CRITICAL: openssl CVE-2025-xxxx

The vulnerability existed because the base image used:

FROM ubuntu:latest

The image had not been rebuilt for several months.

We fixed this by:

• →pinning image versions
• →enabling scheduled rebuilds
• →enforcing automated patch pipelines

Updated example:

FROM python:3.12-alpine

---

Kubernetes Deployment Security

CI/CD security does not stop after image scanning.

Most Kubernetes compromises happen through:

• →excessive privileges
• →weak RBAC
• →insecure pod configurations
• →unrestricted network traffic

We enforced baseline security policies across all workloads.

---

Secure Pod Configuration

Example secure deployment:

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

This prevented:

• →root container execution
• →privilege escalation
• →unnecessary Linux capabilities
• →writable filesystem abuse

We also blocked:

• →privileged containers
• →host networking
• →hostPath mounts

for standard application namespaces.

---

Admission Control Policies

We implemented policy enforcement using Kyverno.

Example policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: enforce
  rules:
    - name: validate-image-tag
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "latest tag is not allowed"
        pattern:
          spec:
            containers:
              - image: "!*:latest"

This eliminated one of the most common deployment problems:

image: backend:latest

Using latest tags caused:

• →rollback confusion
• →inconsistent deployments
• →cache mismatch issues
• →untracked production versions

---

GitOps Security with Argo CD

Argo CD became the deployment control layer.

Only Git repositories were allowed to modify cluster state.

This created:

• →auditable deployments
• →rollback visibility
• →deployment history
• →environment consistency

Example Argo CD application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backend-prod
spec:
  destination:
    namespace: production
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/company/gitops-repo
    path: environments/prod
    targetRevision: main

---

Secret Management Strategy

Hardcoded secrets were completely removed.

We used:

• →AWS Secrets Manager
• →External Secrets Operator
• →Kubernetes RBAC
• →IAM Roles for Service Accounts (IRSA)

Example External Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-secret
spec:
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-secret

This eliminated:

• →plaintext secrets in Git
• →manually rotated credentials
• →shared environment secrets

---

RBAC Hardening

One of the biggest risks in Kubernetes environments is excessive cluster access.

We discovered service accounts with:

verbs:
  - "*"
resources:
  - "*"

This effectively granted cluster-admin permissions.

We replaced this with namespace-scoped RBAC.

Example:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: production
  name: app-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

---

Runtime Security Monitoring

Scanning images before deployment is not enough.

Containers can still:

• →spawn unexpected processes
• →execute shell access
• →create outbound connections
• →attempt privilege escalation

We integrated runtime monitoring with:

• →Falco
• →CloudWatch
• →Datadog
• →Prometheus alerts

Example Falco rule:

- rule: Terminal shell in container
  desc: Detect shell execution inside containers
  condition: container and shell_procs
  output: Shell spawned in container
  priority: WARNING

---

Supply Chain Security Enhancements

Modern Kubernetes security is not only about scanning containers.

Software supply chain attacks increasingly target:

• →package registries
• →CI runners
• →build artifacts
• →compromised dependencies
• →unsigned images

To reduce these risks, we added image provenance validation.

---

Image Signing with Cosign

Every production image was cryptographically signed before deployment.

Example:

cosign sign backend-service:v1.4.2

During deployment, admission policies validated signatures before allowing workloads into the cluster.

This prevented:

• →tampered images
• →untrusted registries
• →unauthorized artifact replacement

Without image signing, a compromised registry can silently distribute malicious containers.

---

Network Segmentation with Kubernetes Network Policies

By default, Kubernetes allows unrestricted pod-to-pod communication.

This creates a major lateral movement risk.

We enforced namespace-level isolation using Network Policies.

Example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
    - Egress

This restricted:

• →unauthorized east-west traffic
• →cross-namespace communication
• →unrestricted outbound internet access

One misconfigured workload previously exposed internal services across environments because no network restrictions existed.

---

Infrastructure-as-Code Security Validation

Application security alone is not sufficient.

Infrastructure misconfigurations can expose entire environments.

We integrated:

• →Checkov
• →tfsec
• →Kubernetes manifest scanning

into pull request validation.

Example:

checkov -d terraform/

Common findings included:

• →public S3 buckets
• →unrestricted security groups
• →missing encryption settings
• →overly permissive IAM policies

This prevented risky infrastructure changes from being merged into production repositories.

---

Multi-Environment Deployment Strategy

We separated deployments into:

- development
- staging
- production

Each environment had:

• →separate namespaces
• →isolated secrets
• →dedicated RBAC policies
• →independent Argo CD applications
• →separate approval workflows

Production deployments required:

• →pull request approval
• →successful security scans
• →deployment validation
• →observability checks

This significantly reduced accidental production releases.

---

Observability Integration

Security incidents are difficult to investigate without centralized observability.

We integrated:

• →Datadog
• →Prometheus
• →Grafana
• →ELK Stack
• →CloudWatch

for:

• →deployment monitoring
• →audit visibility
• →runtime alerting
• →failed rollout detection
• →suspicious activity tracking

Example alerts included:

• →spike in container restarts
• →unauthorized API access attempts
• →repeated failed deployments
• →abnormal outbound traffic

One deployment issue was detected because Prometheus identified a sudden crash-loop increase immediately after a release.

Without monitoring integration, the issue would likely have reached customers before detection.

---

Deployment Rollback Strategy

Secure deployments must also support fast recovery.

Argo CD allowed controlled rollback workflows.

Example:

argocd app rollback backend-prod

Rollback procedures included:

• →reverting Git commits
• →restoring known-good manifests
• →validating health checks
• →monitoring post-rollback metrics

One failed deployment caused elevated memory consumption across production pods.

GitOps-based rollback reduced recovery time significantly because deployment history already existed inside the repository.

---

Production Problems We Faced

The hardest problems were operational — not tooling.

---

1. Vulnerability Scan Noise

Initial Trivy scans produced hundreds of findings.

Many were:

• →low severity
• →non-exploitable
• →dependency-related
• →irrelevant runtime packages

Developers started ignoring reports.

We fixed this by:

• →prioritizing HIGH and CRITICAL vulnerabilities
• →creating approved exception workflows
• →defining remediation SLAs

---

2. Pipeline Slowdowns

Adding security checks increased build times significantly.

Average pipeline duration:

Before security: 6 minutes
After security: 18 minutes

We optimized this using:

• →layer caching
• →parallel scan stages
• →incremental builds
• →dedicated build runners

Final pipeline duration stabilized around:

9-11 minutes

---

3. Argo CD Sync Loops

Mutating admission webhooks occasionally modified manifests after deployment.

Argo CD detected this as drift and continuously resynced applications.

This caused:

• →deployment loops
• →noisy alerts
• →unnecessary pod restarts

We resolved this by configuring:

ignoreDifferences:

for dynamically injected fields.

---

4. Developers Bypassing Security Gates

Some teams attempted to bypass security checks by:

• →manually pushing images
• →reusing old tags
• →skipping GitOps workflows

We solved this by:

• →restricting ECR push permissions
• →enforcing immutable image tags
• →disabling direct production access
• →using signed deployment workflows

---

Security Improvements Achieved

The biggest improvement was operational confidence.

Releases became:

• →repeatable
• →auditable
• →predictable
• →significantly safer

---

Lessons Learned

The most important lesson was that:

Security tooling alone does not secure deployments.

The real challenge is enforcing consistent engineering practices.

The biggest improvements came from:

• →eliminating manual production changes
• →standardizing deployment workflows
• →reducing privilege escalation paths
• →improving deployment visibility
• →integrating security directly into CI/CD

Another important realization:

Developers adopt security faster when pipelines provide actionable feedback instead of generic failures.

Clear remediation guidance reduced friction dramatically.

---

Final Recommendations

If you are building Kubernetes CI/CD pipelines in production:

Do:

• →enforce security checks early
• →block critical vulnerabilities automatically
• →use GitOps for deployments
• →remove direct cluster access
• →enforce RBAC least privilege
• →scan infrastructure manifests
• →pin image versions
• →monitor runtime activity

Avoid:

• →latest image tags
• →shared admin accounts
• →plaintext secrets
• →manual kubectl deployments
• →privileged containers
• →unrestricted network policies

Security works best when it becomes part of the delivery system — not a manual review step.

The goal is not to make deployments slower.

The goal is to make insecure deployments impossible.