GitOps Framework: How the Fako Cluster Repository is Organized

Table of Contents

Overview

The Fako Cluster uses a GitOps approach where Git is the single source of truth for all cluster state. Every deployment, configuration change, and infrastructure update flows through Git commits. FluxCD watches the repository and automatically reconciles the cluster to match the desired state defined in code.

This post explains the repository structure, the reasoning behind each directory, and how the pieces work together.

Repository Structure

fako-cluster/
├── apps/                    # Application deployments
│   ├── base/               # Base Kustomize configurations
│   └── staging/            # Environment overlay (what actually deploys)
├── clusters/                # Cluster bootstrap and Flux config
│   └── staging/            # Flux Kustomizations that orchestrate everything
├── infrastructure/          # Platform components
│   ├── controllers/        # Operators (Cert-Manager, CloudNative-PG, etc.)
│   └── configs/            # Configuration for controllers
├── monitoring/              # Observability stack
│   ├── controllers/        # Prometheus, Loki, Grafana, etc.
│   └── configs/            # Dashboards, alerting rules
├── scripts/                 # Deployment and maintenance scripts
└── notes/                   # Documentation and troubleshooting

The Base/Staging Pattern

The most important concept to understand is the base/staging pattern used throughout the repository.

What is Base?

The base/ directory contains the actual Kubernetes manifests for each application:

apps/base/n8n/
├── namespace.yaml          # Namespace definition
├── kustomization.yaml      # Kustomize configuration
├── deployment.yaml         # The actual deployment
├── service.yaml            # Service definition
├── ingress.yaml            # External access
├── storage.yaml            # Persistent volume claims
├── external-secret.yaml    # Secret references
└── ...

Each application in base/ is a complete, self-contained unit with all resources it needs to run.

What is Staging?

The staging/ directory is an overlay that selects which applications from base/ should be deployed:

# apps/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  # AI/ML stack
  - ollama
  - open-webui
  - whisper
  - piper

  # Workflow Automation
  - n8n

  # Disabled apps (commented out)
  # - linkding
  # - audiobookshelf

Why This Pattern?

Easy enable/disable: Comment out a line to disable an app, uncomment to enable
Clean separation: Base contains the “what”, staging contains the “which”
Future multi-environment: Could add production/ overlay with different selections
No duplication: All manifests live in one place (base), overlays just reference them

The Four Pillars

The repository is organized into four main sections, each with a specific purpose:

1. Infrastructure Controllers

infrastructure/controllers/base/
├── cert-manager/           # TLS certificate automation
├── cloudnative-pg/         # PostgreSQL operator
├── external-secrets/       # AWS Secrets Manager integration
├── gpu-operator/           # NVIDIA GPU support
├── nfs-storage/            # NFS CSI driver
├── smb-csi/                # SMB/CIFS storage driver
├── renovate/               # Automated dependency updates
└── k8s-backup/             # ETCD and PVC backups

Purpose: Install operators and controllers that provide platform capabilities. These must be installed before applications can use features like TLS certificates, external secrets, or GPU acceleration.

2. Infrastructure Configs

infrastructure/configs/base/
├── cert-manager/           # ClusterIssuers, certificates
├── nfs-config/             # NFS mount configurations
└── control-plane-taint/    # Node configuration

Purpose: Configure the controllers installed above. For example, after Cert-Manager is installed, this creates the ClusterIssuer that applications use for TLS.

3. Monitoring

monitoring/controllers/base/
├── kube-prometheus-stack/  # Prometheus, Grafana, AlertManager
├── loki-stack/             # Log aggregation
├── alloy/                  # Unified telemetry collection
├── tempo/                  # Distributed tracing
├── gpu-monitoring/         # NVIDIA GPU metrics
└── metrics-server/         # Resource metrics

Purpose: Observability stack. Separated from infrastructure because monitoring has its own lifecycle and can be updated independently.

4. Applications

apps/base/
├── ollama/                 # LLM inference server
├── n8n/                    # Workflow automation
├── paperless-ngx/          # Document management
├── open-webui/             # AI chat interface
├── whisper/                # Speech-to-text
├── piper/                  # Text-to-speech
└── ... (30+ more)

Purpose: The actual workloads that provide business value. These depend on infrastructure being in place.

Flux Orchestration

How Flux Knows What to Deploy

The clusters/staging/ directory contains Flux Kustomization resources that point to the other directories:

# clusters/staging/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/staging      # Points to apps/staging/kustomization.yaml
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The Dependency Chain

Flux deploys resources in a specific order to ensure dependencies are met:

flux-system (bootstrap)
    │
    ▼
infrastructure-controllers
    │ (Cert-Manager, External Secrets, GPU Operator installed)
    ▼
infrastructure-configs
    │ (Certificates created, NFS configured)
    ▼
apps
    │ (Applications can now use TLS, secrets, GPUs)
    ▼
monitoring-controllers → monitoring-configs
    (Observability watches everything)

This ordering is critical:

Apps can’t get TLS certificates until Cert-Manager is installed and configured
Apps can’t pull secrets until External Secrets Operator is running
GPU workloads can’t schedule until GPU Operator has configured the nodes

Controllers vs Configs Pattern

Within infrastructure and monitoring, you’ll notice a controllers/configs split:

Controllers

Install the actual operators/software:

# infrastructure/controllers/base/cert-manager/
- Helm release for cert-manager
- CRDs for Certificate, ClusterIssuer, etc.

Configs

Configure the installed software:

# infrastructure/configs/base/cert-manager/
- ClusterIssuer for Let's Encrypt
- Wildcard certificate for *.landryzetam.net

Why separate them?

Different update cycles: Controllers update when new versions release, configs change based on business needs
Dependency clarity: Configs depend on controllers being ready
Troubleshooting: Easy to identify if the operator or the configuration is the problem

Deployment Flow

Adding a New Application

Create base manifests:

mkdir apps/base/myapp/
# Create namespace.yaml, deployment.yaml, service.yaml, etc.
# Create kustomization.yaml listing all resources

Enable in staging:

# apps/staging/kustomization.yaml
resources:
  - myapp  # Add this line

Commit and push:

git add . && git commit -m "feat: add myapp" && git push

Flux deploys automatically:

# Watch it happen
flux get kustomization apps
kubectl get pods -n myapp

Disabling an Application

Simply comment out the line in apps/staging/kustomization.yaml:

resources:
  # - myapp  # Disabled

With prune: true, Flux will automatically delete the resources.

Secret Management

No secrets are stored in Git. Instead:

Secrets live in AWS Secrets Manager:

aws secretsmanager create-secret \
  --name "/fako/myapp/database" \
  --secret-string '{"password":"xxx"}'

ExternalSecret resources reference them:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  secretStoreRef:
    name: aws-secrets-store
  target:
    name: database-credentials
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: /fako/myapp/database
        property: password

SOPS for encrypted files (when AWS isn’t suitable):

sops --encrypt --age $AGE_PUBLIC_KEY secret.yaml > secret.enc.yaml

Directory Reference

Directory	Purpose	Updates When
`apps/base/`	Application manifests	Adding/modifying apps
`apps/staging/`	Which apps to deploy	Enabling/disabling apps
`clusters/staging/`	Flux orchestration	Rarely (bootstrap config)
`infrastructure/controllers/`	Platform operators	New operator versions
`infrastructure/configs/`	Platform configuration	Certificates, mounts change
`monitoring/controllers/`	Observability tools	Prometheus/Loki updates
`monitoring/configs/`	Dashboards, alerts	New dashboards, alert rules

Benefits of This Structure

1. Self-Documenting

The repository structure tells you what’s deployed. apps/staging/kustomization.yaml is the definitive list of running applications.

2. Git History = Audit Trail

Every change has a commit with author, timestamp, and reason. “Who deployed X and when?” is a git log away.

3. Easy Rollback

Deployment broke something? git revert and push. Flux rolls back automatically.

4. Pull Request Workflow

Changes can be reviewed before deployment:

git checkout -b feature/add-myapp
# Make changes
git push && gh pr create
# Review, approve, merge
# Flux deploys

5. Disaster Recovery

Cluster dies completely? Bootstrap Flux on a new cluster, point it at the same repo, everything rebuilds automatically.

Common Operations

Force Reconciliation

flux reconcile source git flux-system
flux reconcile kustomization apps

Check Status

flux get all -A
flux get kustomization apps

View Logs

flux logs --follow
flux logs --kind=Kustomization --name=apps

Debug Failed Deployment

# See what Flux sees
flux get kustomization apps -o yaml

# Check events
kubectl get events -n flux-system --sort-by='.lastTimestamp'

Lessons Learned

Start with this structure: Retrofitting GitOps onto an existing cluster is painful. Start clean.
Resist the temptation to kubectl apply: Every manual change creates drift. Always go through Git.
Keep base manifests simple: Don’t over-engineer. Start with deployment, service, ingress. Add complexity only when needed.
Use meaningful commit messages: “fix: stuff” doesn’t help when debugging at 2 AM.
Test in a branch first: For risky changes, create a branch and temporarily point Flux at it.

Kubernetes (K8s) - The Fako Cluster - Overview of the cluster itself
Local AI with Ollama - AI services running on this framework
Home Assistant Voice Pipeline - Multi-service deployment example

The beauty of GitOps is that your infrastructure becomes as reviewable, testable, and reversible as your application code. The repository structure described here has evolved over months of running production workloads, and represents a balance between simplicity and capability.