GitOps in Practice: Managing k8s with FluxCD

5 min read
#GitOps #FluxCD #Kubernetes #CI/CD #Automation
Technical DevOps
Table of Contents

After running the Fako Cluster for several months, I can confidently say that adopting GitOps with FluxCD has been one of the best decisions for managing my Kubernetes infrastructure. In this post, I’ll share practical insights and patterns that have proven invaluable.

What is GitOps?

GitOps is a way of implementing Continuous Deployment for cloud native applications. FluxCD is designed to simplify that, making it easier than ever to manage your Kubernetes environments. Here’s why i leverage it in my workflow:

  • Declarative Infrastructure: You define your desired state of your Kubernetes infrastructure as manifest and version control in a Git repository
  • Automated Reconciliation: FluxCD automatically checks your cluster to see if it matches the state defined in Git, ensuring consistency.
  • Version Control: Every change is tracked in Git, providing a complete audit trail and allowing for easy rollbacks.
  • Pull-Based Deployment: No need to directly interact with kubectl – FluxCD handles the updates automatically.

Why FluxCD?

Among GitOps tools like ArgoCD and co, I chose FluxCD because:

  1. Lightweight: Minimal resource footprint
  2. Native Kubernetes: Uses standard Kubernetes controllers
  3. Multi-tenancy: Built-in support for multiple environments
  4. Extensible: Works with Helm, Kustomize, and raw manifests

My Implementation

Monorepo Approach

I use a monorepo approach where all Kubernetes manifests are stored in a single Git repository. The environment-specific configs are all stored in the same branch (e.g. main), which provides several benefits:

  • Single source of truth: Everything in one place
  • Atomic changes: Update multiple environments in one commit
  • Simplified CI/CD: One repository to manage
  • Better visibility: See all environments at once

Repository Structure

My GitOps repository follows this structure using Kustomize overlays:

fakocluster-apps/
├── apps
│   ├── base
│   ├── production 
│   └── staging
├── infrastructure
│   ├── base
│   ├── production 
│   └── staging
└── clusters
    ├── production
    └── staging

The separation between apps and infrastructure is crucial - it allows me to define the reconciliation order:

  1. First: Cluster addons and Kubernetes controllers (infrastructure)
  2. Then: Applications that depend on that infrastructure

This approach ensures that dependencies are satisfied before applications are deployed.

For a complete example of this monorepo approach, check out the flux2-kustomize-helm-example repository, which demonstrates best practices for structuring a GitOps repository with FluxCD.

Bootstrapping FluxCD

The initial setup is straightforward:

flux bootstrap github \
  --owner=lyzetam \
  --repository=fakocluster-apps \
  --branch=main \
  --path=./clusters/production \
  --personal

Managing Secrets Safely

One of the biggest challenges with GitOps is handling secrets securely. Base64 encoding is NOT encryption - anyone can decode it. That’s why I’ve implemented a comprehensive multi-layered approach using SOPS and External Secrets.

For a deep dive into secrets management, including SOPS configuration, AWS Secrets Manager integration, and best practices, check out my dedicated post: Secrets Management with SOPS, Flux & GitOps.

The key takeaway: Never store plain secrets in Git, even with base64 encoding. Use proper encryption tools like SOPS for versioned configuration and External Secrets Operator for dynamic secrets from AWS Secrets Manager.

Practical Patterns

1. Environment Promotion

I use Kustomize overlays for environment-specific configurations:

# apps/base/myapp/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# apps/production/myapp/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
resources:
  - ../../base/myapp
patchesStrategicMerge:
  - deployment-patch.yaml

2. Automated Image Updates

FluxCD’s image automation controllers handle container updates:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    commit:
      author:
        name: fluxcdbot
        email: [email protected]
      messageTemplate: |
        Automated image update
        
        [ci skip]
    push:
      branch: main

3. Health Checks and Notifications

Monitoring FluxCD reconciliation:

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
  name: on-call-webapp
  namespace: flux-system
spec:
  providerRef:
    name: discord
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: Kustomization
      name: '*'

Lessons Learned

1. Start Simple

Don’t try to automate everything at once. I started with:

  • Basic deployments
  • Gradually added Helm charts
  • Then implemented secret management
  • Finally added image automation

2. Namespace Organization

Keeping services in separate namespaces with FluxCD has benefits:

  • Better resource isolation
  • Easier RBAC management
  • Cleaner GitOps structure

3. Rollback Strategy

Git makes rollbacks trivial:

# Revert to previous state
git revert HEAD
git push

# FluxCD automatically reconciles

4. Testing Changes

I use a staging branch for testing:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system-staging
spec:
  ref:
    branch: staging  # Test changes here first

Common Pitfalls to Avoid

  1. Don’t commit secrets: Use External Secrets or Sealed Secrets
  2. Watch resource limits: FluxCD controllers need adequate resources
  3. Plan for Git outages: Consider local caching strategies
  4. Version your CRDs: Include them in your GitOps flow

Performance Optimization

For larger clusters:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
spec:
  interval: 10m  # Adjust based on change frequency
  retryInterval: 2m
  timeout: 5m
  prune: true
  wait: true
  force: false  # Avoid unless necessary

Debugging FluxCD

Useful commands for troubleshooting:

# Check FluxCD status
flux get all

# View reconciliation errors
flux logs --follow

# Suspend reconciliation for debugging
flux suspend kustomization apps

# Force reconciliation
flux reconcile kustomization apps --with-source

The GitOps Advantage

After months of using GitOps:

  • Zero manual deployments: Everything through Git
  • Complete audit trail: Every change tracked
  • Disaster recovery: Rebuild cluster from Git
  • Team collaboration: PR reviews for infrastructure

What’s Next?

GitOps with FluxCD has transformed how I manage the Fako Cluster. Future posts will cover:

  • Multi-cluster GitOps patterns
  • Progressive delivery with Flagger
  • GitOps for stateful applications
  • Integrating GitOps with CI pipelines

The journey from manual kubectl commands to fully automated GitOps has been rewarding. If you’re managing Kubernetes, I highly recommend exploring this approach!

Want to implement GitOps in your cluster? Check out the FluxCD documentation or explore my GitOps repository for real examples.