Securing GitOps: SOPS Secret Management in k8s with FluxCD
Table of Contents
The Problem: Secrets in GitOps
When I started my Kubernetes homelab journey, I quickly hit a wall that every GitOps practitioner faces: How do you store secrets in Git without compromising security?
The GitOps philosophy is beautifully simple - everything should be in Git. Your entire cluster state, all your applications, configurations, and yes, even secrets. But committing passwords, API keys, and certificates to a public (or even private) repository feels fundamentally wrong. It is wrong.
It’s like writing your WiFi password on your front door - sure, it’s convenient for guests, but so is it for everyone else walking by!
After experimenting with various approaches - from separate private repos to manual secret creation (defeating the purpose of GitOps) - I discovered the elegant solution that production teams use: Mozilla SOPS with age encryption.
Key Concepts in Plain English
Before we dive in, let’s demystify the encryption jargon with some everyday analogies:
Public vs Private Keys: The Mailbox Analogy
Public Key = Your mailbox address
- Anyone can use it to send you mail (encrypt data for you)
- You can share it freely - put it on business cards, websites, anywhere
- Knowing your address doesn’t let someone read your mail
Private Key = The key to your mailbox
- Only you have it
- It’s the only way to open the mailbox and read the mail
- Lose it, and all your mail is locked forever
- Someone steals it, they can read all your mail
Encryption: The Secret Recipe Analogy
Encryption = Grandma’s secret cookie recipe in a locked recipe box
- The recipe (your data) goes into the box
- The box is locked with a special padlock (public key)
- Once locked, even Grandma can’t peek without the key
- Only the person with the matching key (private key) can open it
SOPS Encryption = A smart recipe box that only locks the secret ingredients
- The recipe card stays readable: “Cookies: Flour, Sugar, [ENCRYPTED: Secret Ingredient]”
- You can see what the recipe is for, but not the secret parts
- Perfect for code reviews - you know what’s changing without seeing the secrets
The Bootstrap Paradox: The Chicken and Egg Problem
Here’s the mind-bender that stumped me for weeks: If everything should be encrypted in Git, how do you store the key that does the decrypting?
It’s like trying to mail yourself your own mailbox key - you need the key to get the key! Or putting your car keys in the trunk, then locking it. This is why we need ONE manual step (the bootstrap) to break the cycle.
In this post, I’ll share the complete, detailed process of how I implemented enterprise-grade secret management in my homelab, including every command, every file location, and every decision point that tripped me up initially.
Table of Contents
- Why SOPS + age + FluxCD?
- Understanding the Complete Architecture
- Setting Up Your Local Environment
- The Critical Bridge: Getting Keys into Kubernetes
- Integrating with FluxCD
- Real-World Example: Securing PostgreSQL
- Common Pitfalls and Solutions
- Security Best Practices
- Conclusion
Why SOPS + age + FluxCD?
Before diving into the implementation, let me explain why this stack won me over:
SOPS (Secrets OPerationS)
Think of SOPS as a smart encryption wrapper - like a gift wrapper that only covers the valuable parts:
- Partial Encryption: Only encrypts values, not keys - like a restaurant menu where prices are hidden but you can still read the dishes
- Version Control Friendly: Encrypted files can be diffed, merged, and reviewed in PRs - you can see what changed without seeing the actual secrets
- Git-Native: Encrypted secrets live right next to the apps that use them - no more “where did I put that password?” moments
age
age is like the iPhone of encryption tools - it just works:
- Modern Simplicity: No key servers, no web of trust, no “cryptography PhD required” - just keys that work like your house keys
- Human-Readable: Keys are ASCII strings you can actually type (try doing that with GPG!)
- Lightweight: Single binary, no dependencies - if it were any simpler, it would encrypt itself
FluxCD
FluxCD is your cluster’s faithful butler - always keeping things tidy:
- Native Integration: Built-in SOPS support - it speaks encryption fluently
- GitOps Native: Like having a very pedantic assistant who ensures reality matches the plan
- Production Ready: The same tool Netflix uses (if it’s good enough for Tiger King streams, it’s good enough for your homelab)
Understanding the Complete Architecture
Here’s the complete flow of how secrets move from your local machine to running applications:
┌─────────────────────────────────────────────────────────────┐
│ YOUR MAC (Local Machine) │
│ │
│ ~/keys/ (Private key storage) │
│ ├── age.agekey (Your private key file) │
│ └── age.agekey.backup (Backup copy) │
│ │
│ ~/repos/fako-cluster/ (Your Git repository) │
│ ├── clusters/staging/ │
│ │ └── .sops.yaml (SOPS config with public key) │
│ └── apps/staging/postgres/ │
│ └── secret.yaml (Encrypted with SOPS) │
│ │
└─────────────┬────────────────────────┬──────────────────────┘
│ │
│ kubectl apply │ git push
│ (One-time setup) │ (GitOps flow)
▼ ▼
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ flux-system namespace: │
│ ├── sops-age secret (Private key from kubectl) │
│ └── flux controllers (Pull from Git) │
│ │
│ Decryption Flow: │
│ 1. Flux pulls encrypted secret.yaml from Git │
│ 2. Detects SOPS encryption metadata │
│ 3. Loads private key from sops-age secret │
│ 4. Decrypts the file in memory │
│ 5. Creates actual Kubernetes secret │
│ │
└─────────────────────────────────────────────────────────────┘
Critical Understanding: The private key takes TWO different paths:
- Imperative Path (One-time): Local → kubectl → Kubernetes Secret
- GitOps Path (Continuous): Encrypted files → Git → Flux → Decryption using the key from step 1
Setting Up Your Local Environment
Let’s start from absolute zero on a fresh Mac.
Step 1: Install Required Tools
On macOS, using Homebrew:
# Install age encryption tool
brew install age
# Install SOPS
brew install sops
# Install Flux CLI
brew install fluxcd/tap/flux
# Verify installations
age --version # age 1.1.1
sops --version # sops 3.8.1
flux --version # flux version 2.1.0
Step 2: Create a Secure Local Key Storage
This is crucial - you need a dedicated, backed-up location for your keys:
# Create a dedicated directory for your keys
# NOT in your git repository!
mkdir -p ~/keys
cd ~/keys
# Set restrictive permissions
chmod 700 ~/keys
Step 3: Generate Your Encryption Keypair
Now generate the actual age keypair - this is like the birth of your cluster’s security:
# Generate the keypair
age-keygen -o ~/keys/age.agekey
# This creates a file with content like:
# # created: 2024-01-15T10:30:45Z
# # public key: age1publickeythisisusedtoencryptyoursecrets
# AGE-SECRET-KEY-1THISISYOURPRIVATEKEYUSEDTODECRYPTYOURSECRETS
Let’s look at what was actually created:
# View the generated file
cat ~/keys/age.agekey
# You'll see:
# # created: 2024-01-15T10:30:45Z
# # public key: age1publickeythisisusedtoencryptyoursecrets
# AGE-SECRET-KEY-1THISISYOURPRIVATEKEYUSEDTODECRYPTYOURSECRETS
Understanding the keys (in human terms):
- Public key (age1…): This is like your email address - share it freely, put it in your .sops.yaml, tattoo it on your forehead (okay, maybe not that last one). It can only ENCRYPT things, like a padlock anyone can click shut
- Private key (AGE-SECRET-KEY-1…): This is like the password to your email - guard it with your life! It’s the only thing that can DECRYPT your secrets. Lose it = lose everything. Share it = game over
Step 4: Immediately Back Up Your Private Key
This is NOT optional - lose this key and you’ll be explaining to your family why all the home automation stopped working:
# Create local backup
cp ~/keys/age.agekey ~/keys/age.agekey.backup
# Display the key to save in password manager
echo "=== SAVE THIS IN YOUR PASSWORD MANAGER ==="
cat ~/keys/age.agekey
echo "=== END OF KEY ==="
The Paranoid’s Guide to Key Backup (aka my actual backup strategy):
# Backup Strategy 1: The "I trust Apple/1Password/Bitwarden" approach
# Copy the key to your password manager (you ARE using one, right?)
# Backup Strategy 2: The "USB in a Fireproof Safe" method
cp ~/keys/age.agekey /Volumes/SecureUSB/age.agekey.backup
# Then unplug that USB and lock it away like it contains nuclear codes
# Backup Strategy 3: The "Encrypted Disk Image" technique
hdiutil create -size 10m -fs HFS+ -volname "AgeBackup" \
-encryption AES-256 -stdinpass ~/keys/age-backup.dmg << EOF
your-secure-password-here
your-secure-password-here
EOF
# Mount and copy
hdiutil attach ~/keys/age-backup.dmg
cp ~/keys/age.agekey /Volumes/AgeBackup/
hdiutil detach /Volumes/AgeBackup/
# Backup Strategy 4: The "Old School" approach
# Print it out, laminate it, put it in a safe deposit box
# Your bank teller won't understand, but your future self will thank you
Remember: Your private key is like your Bitcoin wallet seed phrase, except you actually use this one. No backup = eventual tears.
Step 5: Configure Your Repository for SOPS
Now set up your Git repository to use SOPS:
# Navigate to your repository
cd ~/repos/fako-cluster # Or wherever you cloned it
# Extract your public key
PUBLIC_KEY=$(grep "public key:" ~/keys/age.agekey | cut -d' ' -f4)
echo "Your public key is: $PUBLIC_KEY"
# Create SOPS configuration
cat > clusters/staging/.sops.yaml << EOF
creation_rules:
- path_regex: .*\.yaml$
encrypted_regex: ^(data|stringData)$
age: $PUBLIC_KEY
EOF
# Verify the file was created correctly
cat clusters/staging/.sops.yaml
Step 6: Configure Your Shell Environment
Make SOPS aware of your private key location:
# Add to ~/.zshrc (macOS default shell)
echo 'export SOPS_AGE_KEY_FILE=~/keys/age.agekey' >> ~/.zshrc
# Reload your shell configuration
source ~/.zshrc
# Verify it's set
echo $SOPS_AGE_KEY_FILE
# Output: /Users/yourusername/keys/age.agekey
The Critical Bridge: Getting Keys into Kubernetes
This is the most important part that many guides skip - how does your private key get from your Mac into the Kubernetes cluster?
Understanding the Two Paths
Here’s the crucial concept that took me weeks to fully grasp: there are TWO completely different paths in play, and understanding their relationship is key to everything.
Think of it like a fancy hotel:
- Path 1 is giving the concierge the master key to the safe (one-time, in person)
- Path 2 is guests dropping encrypted valuables that only the concierge can open (continuous, automated)
Path 1: The Private Key (One-time, Imperative)
~/keys/age.agekey → kubectl create secret → Kubernetes Secret (sops-age)
This path is:
- One-time setup: Like getting your house keys cut - you do it once, not every time you come home
- Imperative: Uses
kubectl createdirectly, NOT GitOps - this is you physically handing over the key - Manual: You physically run this command yourself - no automation here, this is a trust ceremony
- Creates the decryption capability: Without this, Flux is like a locksmith without any tools
- Never stored in Git: This secret exists only in the cluster - like keeping your spare key with a trusted neighbor, not under a fake rock
- NOT encrypted with SOPS: The private key itself is NOT encrypted - it’s the key that opens all the locks!
Why isn’t the private key encrypted with SOPS? This is the bootstrap paradox - you can’t encrypt the key that does the decrypting! It’s like trying to lock your keys inside the safe they open. Or mailing yourself an envelope with the key to your mailbox inside. Or password-protecting the password to your password manager. (We’ve all been there, right?)
The private key must be readable by Flux to decrypt everything else. Instead of SOPS encryption, we rely on:
- Kubernetes RBAC to protect access (like a security guard for the key)
- Base64 encoding (not encryption, just encoding - like putting the key in an envelope)
- Never committing this to Git (the key never leaves the building)
- Manual, one-time installation (the sacred handoff ritual)
Think of this as installing the “master key” into your cluster’s vault. Just like giving a trusted person the key to your house - you do it once, in person, not by leaving it under the doormat (or worse, posting it on Instagram with #MyHouseKey).
Path 2: Encrypted Secrets (Continuous, GitOps)
secret.yaml → sops encrypt → Git → Flux → Decrypt using sops-age secret → App
This path is:
- Continuous: Happens every time you push a new secret to Git
- Declarative/GitOps: Everything is defined in Git
- Automated: Flux handles everything automatically
- Uses the key from Path 1: Flux uses the sops-age secret to decrypt
- Fully auditable: Every secret change is tracked in Git history
Think of this as the normal flow of secrets - like sending encrypted messages that only someone with the master key (from Path 1) can read.
The Critical Relationship:
- Path 1 must happen BEFORE Path 2 can work
- Path 1 enables Path 2
- If you delete the result of Path 1 (the sops-age secret), Path 2 breaks immediately
- Path 1 is your “bootstrap trust” - the one manual step that makes everything else automatic
Here’s what happens in practice:
- Initial Setup (Path 1): You manually create the sops-age secret with kubectl
- Daily Operations (Path 2): You encrypt secrets locally, push to Git, Flux decrypts using the key from step 1
- If something breaks: 99% of the time, it’s because the sops-age secret is missing or wrong
Let me visualize this with a more detailed diagram:
┌──────────────────────────────────────────────────────────────┐
│ PATH 1: ONE-TIME SETUP │
│ (The Foundation - Do This First!) │
├──────────────────────────────────────────────────────────────┤
│ │
│ Your Mac Kubernetes Cluster │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ ~/keys/ │ kubectl │ flux-system ns │ │
│ │ age.agekey │──create────▶ │ sops-age │ │
│ │ (private key) │ secret │ (Secret) │ │
│ └─────────────────┘ └──────────────────┘ │
│ │
│ This happens ONCE, MANUALLY, IMPERATIVELY │
│ Never stored in Git, never automated │
└──────────────────────────────────────────────────────────────┘
↓ Enables ↓
┌──────────────────────────────────────────────────────────────┐
│ PATH 2: CONTINUOUS OPERATIONS │
│ (Daily GitOps Workflow) │
├──────────────────────────────────────────────────────────────┤
│ │
│ Your Mac GitHub Flux in Cluster │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │secret. │ push │Encrypted │ pull │ Flux │ │
│ │yaml │─────▶│secret. │─────▶│ reads │ │
│ │(encrypt) │ │yaml │ │ sops-age │ │
│ └──────────┘ └──────────┘ │ secret │ │
│ │ ↓ │ │
│ │ Decrypts │ │
│ │ ↓ │ │
│ │ Creates │ │
│ │ K8s Secret │ │
│ └──────────────┘ │
│ │
│ This happens CONTINUOUSLY, AUTOMATICALLY, via GitOps │
└──────────────────────────────────────────────────────────────┘
Step 1: Create the flux-system Namespace
# Create namespace (idempotent - safe to run multiple times)
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
# Verify it exists
kubectl get namespace flux-system
# Output: NAME STATUS AGE
# flux-system Active 10s
Step 2: Install Your Private Key as a Kubernetes Secret
This is THE critical step - your private key must be in the cluster BEFORE Flux tries to decrypt anything:
# Create the secret containing your private key
# This is an IMPERATIVE command (not GitOps)
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=$HOME/keys/age.agekey
# Output: secret/sops-age created
Let’s break down EXACTLY what this command does:
kubectl create secret generic sops-age \
# └─────┬─────┘ └──┬──┘ └───┬───┘
# │ │ │
# │ │ └── Name of the secret (must be "sops-age"
# │ │ for Flux to find it)
# │ │
# │ └── Type of secret (generic = Opaque type,
# │ just key-value pairs)
# │
# └── Creates a NEW Kubernetes Secret object
--namespace=flux-system \
# └─────────┬────────────┘
# │
# └── MUST be in flux-system namespace where Flux controllers run
# This is where Flux will look for it
--from-file=age.agekey=$HOME/keys/age.agekey
# └─────┬────┘└───┬────┘ └──────────┬──────────┘
# │ │ │
# │ │ └── Path to your actual private key file
# │ │ on your Mac
# │ │
# │ └── The KEY name inside the Kubernetes secret
# │ (Flux expects exactly "age.agekey")
# │
# └── Read the content from a file (not typing it directly)
What Actually Happens:
- Reads your private key file from
~/keys/age.agekey - Base64 encodes it (NOT encrypts - just encodes for storage)
- Creates a Kubernetes Secret that looks like this:
apiVersion: v1
kind: Secret
metadata:
name: sops-age
namespace: flux-system
type: Opaque
data:
age.agekey: QUdFLVNFQ1JFVC1LRVktMVRISVNJU1lPVVJQUklWQVRFS0VZLi4u...
# ↑ This is your private key, just base64 encoded
Critical Points:
- The private key is NOT encrypted with SOPS (bootstrap paradox!)
- It’s only base64 encoded (Kubernetes standard for all Secrets)
- This is the ONLY secret you create manually
- The name MUST be
sops-ageand MUST be influx-systemnamespace - The key inside MUST be named
age.agekey - If ANY of these are wrong, Flux cannot decrypt your secrets
Let’s verify this worked:
# Check the secret exists
kubectl -n flux-system get secret sops-age
# Output: NAME TYPE DATA AGE
# sops-age Opaque 1 30s
# View the secret structure (BE CAREFUL - contains your private key!)
kubectl -n flux-system get secret sops-age -o yaml
# Output will show:
# apiVersion: v1
# kind: Secret
# metadata:
# name: sops-age
# namespace: flux-system
# type: Opaque
# data:
# age.agekey: QUdFLVNFQ1JFVC1LRVktMVRISVNJU1lPVVJQUklWQVRFS0VZLi4u...
# # ^^^ This is your private key, base64 encoded
Is Base64 Encoding Secure?
Haha, no! Base64 is NOT encryption - it’s like writing your password backwards and calling it “secure.” Anyone can decode it:
echo "QUdFLVNFQ1JFVC1LRVktMVRISVNJU1lPVVJQUklWQVRFS0VZLi4u..." | base64 -d
# Would show: AGE-SECRET-KEY-1THISISYOURPRIVATEKEYUSEDTODECRYPT...
Base64 encoding is just changing the alphabet - like writing “HELLO” as “48 45 4C 4C 4F” in hex. It’s obfuscation, not encryption. It’s the digital equivalent of speaking Pig Latin to hide secrets from your kids (spoiler: they figured it out).
So why is this secure enough?
Because we’re not relying on the encoding for security! It’s like keeping your house key in a locked box at the bank:
- RBAC Protection: Only Flux service account can read this secret (the bank vault)
- Not in Git: Never committed to version control (not left under the doormat)
- Namespace Isolation: Lives in flux-system namespace (locked room in the vault)
- Audit Logging: Access is logged and can be monitored (security cameras everywhere)
- It’s the bootstrap: This one manual step enables everything else to be properly encrypted
Think of it like your house key - it’s not encrypted itself (it’s just a piece of metal with ridges), but you carefully control who has access to it. You don’t need to encrypt your key if it’s in a secure location with restricted access.
The real security comes from WHERE the key is stored and WHO can access it, not from the key being encrypted. After all, at some point, something needs to be readable to start the decryption chain - otherwise, you’d have turtles all the way down!
Step 3: Verify the Key is Accessible
# Decode and verify it matches your local key
# BE CAREFUL - only do this in a secure environment
kubectl -n flux-system get secret sops-age \
-o jsonpath="{.data.age\.agekey}" | base64 -d
# Compare with your local key
cat ~/keys/age.agekey
# They should be IDENTICAL
Important Notes:
- This secret is NOT stored in Git (it’s created imperatively with kubectl)
- This is the ONLY secret you create manually
- All other secrets will be encrypted in Git and decrypted by Flux using this key
- If you delete this secret, Flux cannot decrypt anything
Integrating with FluxCD
Now that your private key is in the cluster, let’s set up Flux to use it.
Step 1: Bootstrap FluxCD
# Set your GitHub credentials
export GITHUB_USER="yourusername"
export GITHUB_TOKEN="ghp_yourGitHubPersonalAccessToken"
# Bootstrap Flux
flux bootstrap github \
--owner=$GITHUB_USER \
--repository=fako-cluster \
--branch=main \
--path=clusters/staging \
--personal
# This will:
# 1. Create Flux components in flux-system namespace
# 2. Create a deploy key in your GitHub repo
# 3. Configure Flux to sync from clusters/staging/
Step 2: Configure Flux Kustomizations for Decryption
Create a Kustomization that tells Flux to decrypt SOPS files:
# Create the apps Kustomization
cat > clusters/staging/apps.yaml << 'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 1m0s
retryInterval: 1m0s
timeout: 5m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/staging
prune: true
decryption: # THIS SECTION IS CRITICAL!
provider: sops # Tell Flux to use SOPS
secretRef:
name: sops-age # The secret we created with kubectl
EOF
# Commit and push
git add clusters/staging/apps.yaml
git commit -m "Enable SOPS decryption for apps"
git push
Step 3: Verify Flux Can Decrypt
# Check if the Kustomization was created
flux get kustomizations
# Output:
# NAME READY MESSAGE
# flux-system True Applied revision: main/abc123
# apps True Applied revision: main/abc123
# Check for any decryption errors
flux logs --level=error --tail=20
Real-World Example: Securing PostgreSQL
Let’s walk through a complete example of creating and using an encrypted secret.
Step 1: Create the PostgreSQL Secret
# Create directory structure
mkdir -p apps/staging/postgres
# Create the secret file (PLAINTEXT initially)
cat > apps/staging/postgres/postgres-secret.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
name: postgres-admin
namespace: postgres
type: kubernetes.io/basic-auth
stringData:
username: postgres
password: "MyActualSuperSecurePassword123!"
postgres-password: "MyActualSuperSecurePassword123!"
replication-password: "ReplicationPassword456!"
EOF
Step 2: Encrypt the Secret with SOPS
# Make sure you're in the repository root
cd ~/repos/fako-cluster
# Verify SOPS can find your key
echo $SOPS_AGE_KEY_FILE
# Output: /Users/yourusername/keys/age.agekey
# Encrypt the file IN PLACE
sops -e -i apps/staging/postgres/postgres-secret.yaml
# Verify encryption worked
cat apps/staging/postgres/postgres-secret.yaml
After encryption, your file will look like:
apiVersion: v1
kind: Secret
metadata:
name: postgres-admin
namespace: postgres
type: kubernetes.io/basic-auth
stringData:
username: ENC[AES256_GCM,data:Lkx9...,iv:abc...,tag:xyz...,type:str]
password: ENC[AES256_GCM,data:8kN2...,iv:def...,tag:uvw...,type:str]
postgres-password: ENC[AES256_GCM,data:pQ9x...,iv:ghi...,tag:rst...,type:str]
replication-password: ENC[AES256_GCM,data:mN3p...,iv:jkl...,tag:opq...,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1publickeythisisusedtoencryptyoursecrets
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkdUpOc3ZnbTNRQ0IzRjBF
[... more encrypted content ...]
-----END AGE ENCRYPTED FILE-----
lastmodified: "2024-01-15T10:45:00Z"
mac: ENC[AES256_GCM,data:abc123...,iv:xyz789...,tag:qrs456...,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.8.1
Step 3: Create Kustomization for PostgreSQL
# Create kustomization file
cat > apps/staging/postgres/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: postgres
resources:
- namespace.yaml
- postgres-secret.yaml
EOF
# Create namespace definition
cat > apps/staging/postgres/namespace.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: postgres
EOF
Step 4: Commit and Push
# Add all files
git add apps/staging/postgres/
# Verify you're committing ENCRYPTED secrets
git diff --staged apps/staging/postgres/postgres-secret.yaml
# Should show ENC[AES256_GCM...] values, NOT plaintext
# Commit and push
git commit -m "Add encrypted PostgreSQL secrets"
git push
Step 5: Watch Flux Deploy It
# Watch Flux sync
flux get kustomizations --watch
# Check if namespace was created
kubectl get namespace postgres
# Check if secret was created (and decrypted!)
kubectl -n postgres get secrets
# Verify the secret contains decrypted values
kubectl -n postgres get secret postgres-admin \
-o jsonpath='{.data.password}' | base64 -d
# Output: MyActualSuperSecurePassword123!
Step 6: Use the Secret in Your PostgreSQL Deployment
# apps/staging/postgres/postgres-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: postgres
spec:
instances: 3
primaryUpdateStrategy: unsupervised
postgresql:
parameters:
max_connections: "200"
shared_buffers: "256MB"
bootstrap:
initdb:
database: app
owner: postgres
secret:
name: postgres-admin # References our decrypted secret!
storage:
size: 10Gi
storageClass: local-path
Common Pitfalls and Solutions
Pitfall 1: Forgetting to Create the sops-age Secret First
Symptom:
flux logs --level=error
# Error: decryption failed: no secret found
This is like showing up to a party with a locked gift box but forgetting to give anyone the key.
Solution:
# Always create the secret BEFORE bootstrapping Flux
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=$HOME/keys/age.agekey
Pitfall 2: SOPS_AGE_KEY_FILE Not Set
Symptom:
sops -e -i secret.yaml
# Error: no age identity found
This is SOPS saying “I’d love to encrypt this for you, but you haven’t told me where the key is!” It’s like trying to lock a door while your keys are still in yesterday’s pants.
Solution:
# Set the environment variable
export SOPS_AGE_KEY_FILE=~/keys/age.agekey
# Add to ~/.zshrc to make permanent
echo 'export SOPS_AGE_KEY_FILE=~/keys/age.agekey' >> ~/.zshrc
source ~/.zshrc
Pitfall 3: Committing Unencrypted Secrets
Symptom: Your passwords are visible in GitHub (disaster! catastrophe! time to update your resume!)
This is the equivalent of accidentally hitting “Reply All” with your salary details. Once it’s out there, it’s OUT THERE.
Prevention:
# Add a pre-commit hook - your safety net
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
# Check for unencrypted secrets
for file in $(git diff --cached --name-only | grep -E "secret.*\.yaml$"); do
if ! grep -q "sops:" "$file" 2>/dev/null; then
echo "ERROR: $file appears to be an unencrypted secret!"
echo "Run: sops -e -i $file"
echo "Crisis averted! You're welcome."
exit 1
fi
done
EOF
chmod +x .git/hooks/pre-commit
Pitfall 4: Wrong Public Key in .sops.yaml
Symptom:
sops -d secret.yaml
# Error: age: no identity matched any recipient
This is like trying to open a lock with a key from a different lock - no amount of jiggling will help.
Solution:
# Check your public key
grep "public key:" ~/keys/age.agekey
# Update .sops.yaml with the correct key
vi clusters/staging/.sops.yaml
# Re-encrypt all files with correct key
find . -name "*.yaml" -exec grep -l "sops:" {} \; | while read file; do
sops updatekeys -y "$file"
done
Pitfall 5: Lost Private Key
Symptom: Cannot decrypt any secrets, cluster is broken, considering new career in farming
This is the “locked myself out of the house” scenario, except the house contains all your infrastructure.
Prevention (aka “Future You Will Thank Present You”):
# Multiple backup strategies - because paranoia is just good planning
# 1. Password manager (the "I trust the cloud" approach)
# 2. Encrypted USB drive (the "physical security" method)
# 3. Printed paper in safe (the "apocalypse-proof" solution)
# 4. Encrypted cloud backup (the "belt and suspenders" technique)
# Recovery test (do this NOW, not when you need it)
# 1. Move your key temporarily
mv ~/keys/age.agekey ~/keys/age.agekey.temp
# 2. Restore from backup
cp ~/keys/age.agekey.backup ~/keys/age.agekey
# 3. Test decryption
sops -d apps/staging/postgres/postgres-secret.yaml
# 4. If it works, you're good! If not, panic responsibly
rm ~/keys/age.agekey.temp
Remember: The universe has a sense of humor - you’ll lose your key exactly 5 minutes after your backup drive fails. Plan accordingly.
Security Best Practices
1. Protect Your Private Key
# Set restrictive permissions on key file
chmod 600 ~/keys/age.agekey
# Set restrictive permissions on keys directory
chmod 700 ~/keys
# Never store in:
# - Git repositories
# - Unencrypted cloud storage
# - Slack/Email/Discord
# - Unencrypted local files
2. Use Different Keys for Different Environments
# Generate environment-specific keys
age-keygen -o ~/keys/age-dev.agekey
age-keygen -o ~/keys/age-staging.agekey
age-keygen -o ~/keys/age-prod.agekey
# Use different .sops.yaml for each environment
# clusters/dev/.sops.yaml → uses age-dev public key
# clusters/staging/.sops.yaml → uses age-staging public key
# clusters/prod/.sops.yaml → uses age-prod public key
3. Implement Key Rotation
# Quarterly key rotation process
# 1. Generate new keypair
age-keygen -o ~/keys/age-new.agekey
# 2. Update .sops.yaml with new public key
NEW_PUBLIC_KEY=$(grep "public key:" ~/keys/age-new.agekey | cut -d' ' -f4)
sed -i '' "s/age: .*/age: $NEW_PUBLIC_KEY/" clusters/staging/.sops.yaml
# 3. Re-encrypt all secrets
find . -name "*.yaml" -exec grep -l "sops:" {} \; | while read file; do
# Decrypt with old key
SOPS_AGE_KEY_FILE=~/keys/age.agekey sops -d -i "$file"
# Encrypt with new key
SOPS_AGE_KEY_FILE=~/keys/age-new.agekey sops -e -i "$file"
done
# 4. Update cluster secret
kubectl delete secret sops-age -n flux-system
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=$HOME/keys/age-new.agekey
# 5. Archive old key (don't delete immediately)
mv ~/keys/age.agekey ~/keys/age.agekey.old-$(date +%Y%m%d)
mv ~/keys/age-new.agekey ~/keys/age.agekey
4. Audit Secret Access
# Enable Kubernetes audit logging for secret access
# Add to your cluster configuration:
cat > audit-policy.yaml << 'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources: ["secrets"]
namespaces: ["flux-system"]
EOF
# Check who accessed the sops-age secret
kubectl logs -n kube-system kube-apiserver-* | grep sops-age
5. Use RBAC to Protect the Private Key
# Create strict RBAC for the sops-age secret
cat > sops-rbac.yaml << 'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: sops-age-reader
namespace: flux-system
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["sops-age"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: flux-sops-age-reader
namespace: flux-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: sops-age-reader
subjects:
- kind: ServiceAccount
name: kustomize-controller
namespace: flux-system
EOF
kubectl apply -f sops-rbac.yaml
Monitoring and Alerting
Monitor Decryption Failures
# Create a simple monitoring script
cat > ~/monitor-sops.sh << 'EOF'
#!/bin/bash
# Check for SOPS decryption failures
ERRORS=$(flux logs --level=error --since=1h | grep -c "decryption failed")
if [ $ERRORS -gt 0 ]; then
echo "ALERT: Found $ERRORS SOPS decryption failures in the last hour"
# Send alert to Slack/PagerDuty/Email
fi
EOF
chmod +x ~/monitor-sops.sh
# Add to crontab for hourly checks
crontab -e
# Add: 0 * * * * /Users/yourusername/monitor-sops.sh
Track Secret Changes
# View history of secret changes
git log --follow apps/staging/postgres/postgres-secret.yaml
# See who changed what
git blame apps/staging/postgres/postgres-secret.yaml
# Diff encrypted secrets (yes, this works!)
git diff HEAD~1 apps/staging/postgres/postgres-secret.yaml
Testing Your Setup
Complete Test Workflow
Let’s make sure everything works before you trust it with real secrets (because testing in production is how legends are made… and resumes updated):
# 1. Create a test secret
cat > test-secret.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
name: test
namespace: default
stringData:
message: "If you can read this, SOPS is working!"
celebration: "Time to encrypt all the things!"
warning: "With great encryption comes great responsibility"
EOF
# 2. Encrypt it
sops -e -i test-secret.yaml
# 3. Verify encryption (the "did it actually work?" check)
grep "ENC\[AES256_GCM" test-secret.yaml && echo "✓ Encrypted" || echo "✗ Still plaintext - try again!"
# 4. Test local decryption (the "can I still read it?" test)
sops -d test-secret.yaml | grep "SOPS is working" && echo "✓ Local decryption works" || echo "✗ Houston, we have a problem"
# 5. Push to Git (the "moment of truth")
git add test-secret.yaml
git commit -m "Test SOPS encryption - if you can read the secrets in this commit, something went wrong"
git push
# 6. Watch Flux process it (the "popcorn moment")
flux reconcile kustomization flux-system --with-source
kubectl get secret test -n default
# 7. Verify cluster decryption (the "victory lap")
kubectl get secret test -n default -o jsonpath='{.data.message}' | base64 -d
# Output: If you can read this, SOPS is working!
echo "🎉 Success! Your secrets are safer than your browser history!"
# 8. Clean up (leave no trace)
kubectl delete secret test -n default
git rm test-secret.yaml
git commit -m "Remove test secret - it served us well"
git push
If all these tests pass, congratulations! You’ve just implemented the same secret management system that protects production workloads worldwide. Your homelab is now more secure than most corporate environments I’ve audited (that’s both impressive and slightly concerning).
Conclusion
Implementing SOPS with FluxCD transformed my homelab from a learning playground into a production-grade platform. The key insights that made it click for me:
The private key must be manually installed in the cluster - This is the one imperative step in an otherwise declarative setup. It’s like the first domino that makes all the others fall perfectly.
Keep your keys in a dedicated local directory - Never in your Git repo, always backed up multiple times. Treat it like your passport - you know exactly where it is, and you have copies in secure places.
The flow is actually simple once you understand it: Local encryption → Git → Flux decryption → Application use. It’s just like sending a locked suitcase on a flight - you lock it at home, it travels securely, and opens at the destination.
Test your backups - Actually restore from your backup before you need to. It’s like a fire drill - boring when you don’t need it, lifesaving when you do.
What started as a solution to “how do I avoid committing passwords to Git” became a deep dive into production secret management patterns. The same setup that secures my homelab PostgreSQL passwords could scale to protect enterprise credentials.
The beauty of this approach is its simplicity once configured. I now think of secrets like any other Kubernetes resource - defined in Git, versioned, reviewed, and automatically deployed. The encryption is transparent to my workflow but provides rock-solid security.
Remember: Your private key is everything. Guard it like your house keys, back it up like your family photos, and test your recovery process like your home automation depends on it (because it does).
The journey from “passwords in plaintext” to “enterprise-grade secret management” is just a few commands and one paranoid backup strategy away. Once you understand that it’s really just about:
- One key to rule them all (private key)
- One place to store it (the cluster)
- One system to encrypt them (SOPS)
- And in the GitOps bind them (Flux)
…you realize it’s not magic, it’s just good engineering.
Quick Reference Commands
# Generate age keypair (the birth of security)
age-keygen -o ~/keys/age.agekey
# Install private key in cluster (the sacred handoff)
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=$HOME/keys/age.agekey
# Encrypt a file (the lockdown)
export SOPS_AGE_KEY_FILE=~/keys/age.agekey
sops -e -i path/to/secret.yaml
# Decrypt a file (the peek inside)
sops -d path/to/secret.yaml
# Edit encrypted file (the secure editor)
sops path/to/secret.yaml
# Force Flux to reconcile (the "do it now" button)
flux reconcile kustomization apps --with-source
# Check for errors (the "what went wrong?" command)
flux logs --level=error --follow
Resources and Next Steps
Coming Next
In my next post, I’ll cover how I layer External Secrets Operator on top of SOPS for dynamic secret management with AWS Secrets Manager, creating a dual-layer secret architecture that combines the best of both GitOps and centralized secret management.
For more on implementing GitOps, check out my post on GitOps with FluxCD. If this helped you secure your own cluster, consider starring the repo or sharing this post!