Enabling DNS-01 TLS for Internal Kubernetes Services

11 min read
#Kubernetes #TLS #cert-manager #Cloudflare #DNS-01 #Homelab #Security #Let's Encrypt #External Secrets
Kubernetes
Table of Contents

Ever visited a website and seen that little padlock icon in your browser? That’s TLS at work, encrypting the connection between you and the server. But what happens when you want that same security for services running on your home network that aren’t accessible from the internet? Today I solved this exact problem, and I want to walk you through not just how I did it, but why each piece matters.

Part 1: Understanding the Basics

What is TLS and Why Do We Need It?

TLS (Transport Layer Security) is a protocol that encrypts data traveling between two points—like your browser and a web server. Without TLS:

  • Anyone on the network can read your traffic (passwords, data, everything)
  • Anyone can pretend to be the server you’re connecting to
  • Anyone can modify data in transit without you knowing

When you see https:// instead of http://, that’s TLS in action. The “S” literally stands for “Secure.”

What is a Certificate?

A TLS certificate is like a digital ID card for a website. It contains:

  1. The domain name it’s valid for (e.g., paperless.landryzetam.net)
  2. A public key used for encryption
  3. A digital signature from a trusted authority proving it’s legitimate
  4. An expiration date (certificates don’t last forever)

When your browser connects to a site, it checks the certificate to verify:

  • “Am I really talking to who I think I’m talking to?”
  • “Is this certificate signed by someone I trust?”

Who Signs Certificates? Certificate Authorities

A Certificate Authority (CA) is a trusted organization that verifies domain ownership and signs certificates. Your browser comes pre-loaded with a list of trusted CAs.

Let’s Encrypt is a free, automated CA that has revolutionized TLS adoption. Before Let’s Encrypt, certificates cost money and required manual renewal. Now, anyone can get valid certificates for free.

The Challenge: Proving You Own a Domain

Before Let’s Encrypt signs a certificate, you must prove you control the domain. They offer two main methods:

MethodHow It WorksBest For
HTTP-01Place a file at http://yourdomain/.well-known/acme-challenge/Public websites
DNS-01Create a TXT record in your DNSInternal services, wildcards

Part 2: The Problem with Internal Services

My Situation

I deployed Paperless-ngx (a document management system) to my Kubernetes homelab. I wanted to access it at https://paperless.landryzetam.net, but:

  • The DNS record points to an internal IP: 10.85.30.214
  • This IP is only reachable from my home network
  • Let’s Encrypt servers on the internet cannot reach it

Why HTTP-01 Fails

The HTTP-01 challenge works like this:

  1. You request a certificate for paperless.landryzetam.net
  2. Let’s Encrypt says: “Prove you own it by putting this file at this URL”
  3. Let’s Encrypt tries to fetch http://paperless.landryzetam.net/.well-known/acme-challenge/xyz
  4. Problem: Let’s Encrypt can’t reach 10.85.30.214—it’s a private IP!

The error message tells the story:

authorization error for paperless.landryzetam.net:
no valid A records found for paperless.landryzetam.net

The Solution: DNS-01

DNS-01 proves domain ownership differently:

  1. You request a certificate for paperless.landryzetam.net
  2. Let’s Encrypt says: “Create a DNS TXT record with this value”
  3. You create _acme-challenge.paperless.landryzetam.net with the specified value
  4. Let’s Encrypt queries DNS (which is public) and verifies the record
  5. Certificate issued!

The key insight: DNS is always publicly queryable, even if the actual service is internal. Let’s Encrypt doesn’t need to reach your server—it just needs to see the DNS record.

Part 3: The Components We’ll Use

Before diving into implementation, let’s understand each tool and technology in the data flow:

Traefik (Ingress Controller)

What it is: An ingress controller—software that actually handles incoming HTTP/HTTPS traffic in Kubernetes.

What it does:

  • Listens on ports 80 (HTTP) and 443 (HTTPS)
  • Reads Ingress resources to know how to route traffic
  • Terminates TLS (decrypts HTTPS using certificates)
  • Forwards decrypted requests to backend services

Why we need it: Kubernetes doesn’t handle external traffic by itself. An Ingress resource is just a configuration—you need an ingress controller like Traefik, Nginx, or HAProxy to actually process the traffic.

Analogy: If Ingress is the GPS directions, Traefik is the actual car that follows them.

Kubernetes Service

What it is: An abstraction that defines how to access a set of Pods.

What it does:

  • Provides a stable IP address and DNS name for your application
  • Load balances traffic across multiple Pod replicas
  • Decouples “how to reach the app” from “which Pods are running”

Why we need it: Pods are ephemeral—they come and go, changing IP addresses each time. A Service provides a consistent endpoint (paperless-ngx:8000) that always routes to healthy Pods.

Types:

  • ClusterIP: Only accessible within the cluster (default)
  • NodePort: Exposes on each node’s IP
  • LoadBalancer: Provisions an external load balancer

Kubernetes Secret

What it is: A Kubernetes resource for storing sensitive data.

What it does:

  • Stores data like passwords, tokens, and certificates
  • Base64 encodes values (not encryption, just encoding!)
  • Can be mounted as files or exposed as environment variables

For TLS certificates: cert-manager stores the certificate and private key in a Secret. Traefik reads this Secret to serve HTTPS.

Important: Secrets in Kubernetes are only base64 encoded by default—not encrypted. That’s why we use External Secrets Operator to sync from a proper secret store.

Cloudflare

What it is: A DNS provider and CDN (Content Delivery Network) service.

What it does for us:

  • Hosts DNS records for landryzetam.net
  • Provides an API to programmatically manage DNS records
  • Allows cert-manager to create _acme-challenge TXT records automatically

DNS basics: DNS (Domain Name System) translates human-readable names like paperless.landryzetam.net into IP addresses like 10.85.30.214. When you type a URL, your computer asks DNS servers “what’s the IP for this domain?”

Why Cloudflare?: Any DNS provider with an API works. Cloudflare is popular because it’s free, fast, and has excellent API support.

AWS Secrets Manager

What it is: A managed service from Amazon Web Services for storing and retrieving secrets.

What it does:

  • Stores secrets encrypted at rest
  • Controls access via IAM policies (who can read what)
  • Provides audit logging (who accessed what, when)
  • Supports automatic rotation for some secret types

Why we need it: We need to store the Cloudflare API token somewhere secure. Options include:

  • Git (❌ Never commit secrets!)
  • Kubernetes Secrets directly (⚠️ Works, but secrets live only in cluster)
  • AWS Secrets Manager (✅ Centralized, encrypted, audited)

How it works with ESO: External Secrets Operator connects to AWS, reads the secret, and creates a Kubernetes Secret with the same value.

ACME Protocol

What it is: Automatic Certificate Management Environment—the protocol used to communicate with Let’s Encrypt.

What it does:

  • Standardizes how clients request certificates
  • Defines challenge types (HTTP-01, DNS-01, etc.)
  • Handles certificate issuance and renewal

The flow:

  1. cert-manager sends a Certificate Signing Request (CSR) to Let’s Encrypt
  2. Let’s Encrypt responds with a challenge (“prove you own this domain”)
  3. cert-manager completes the challenge (creates DNS record)
  4. Let’s Encrypt verifies and issues the certificate
  5. cert-manager downloads and stores the certificate

Why it matters: ACME made free, automated certificates possible. Before ACME, getting a certificate meant emails, manual verification, and payment.

cert-manager

What it is: A Kubernetes add-on that automates certificate management.

What it does:

  • Requests certificates from CAs like Let’s Encrypt
  • Handles the challenge process (HTTP-01 or DNS-01)
  • Stores certificates as Kubernetes Secrets
  • Automatically renews certificates before they expire

Why we need it: Without cert-manager, you’d manually request certificates, copy them into your cluster, and set calendar reminders to renew them. cert-manager does all this automatically.

ClusterIssuer

What it is: A Kubernetes resource that tells cert-manager how to get certificates.

What it does:

  • Defines which CA to use (Let’s Encrypt production, staging, etc.)
  • Specifies which challenge method to use (HTTP-01 or DNS-01)
  • Contains credentials for DNS providers (if using DNS-01)

Cluster vs Namespaced: A ClusterIssuer works across all namespaces. A regular Issuer only works in one namespace. I use ClusterIssuer so any application can request certificates.

External Secrets Operator (ESO)

What it is: A Kubernetes operator that syncs secrets from external sources.

What it does:

  • Connects to secret stores like AWS Secrets Manager, HashiCorp Vault, etc.
  • Creates Kubernetes Secrets from external secrets
  • Keeps them in sync (refreshes periodically)

Why we need it: I need to give cert-manager a Cloudflare API token for DNS-01 challenges. But I can’t commit secrets to Git (that’s a security disaster). ESO lets me:

  1. Store the token in AWS Secrets Manager (secure, outside Git)
  2. Have ESO sync it into Kubernetes automatically

SecretStore

What it is: Tells External Secrets Operator where to find secrets.

What it does:

  • Defines the connection to AWS Secrets Manager (or other providers)
  • Contains authentication details
  • Scoped to a namespace

ExternalSecret

What it is: Tells External Secrets Operator which secret to sync.

What it does:

  • References a SecretStore
  • Specifies which secret to fetch from AWS
  • Creates a Kubernetes Secret with the fetched data

Ingress

What it is: A Kubernetes resource that manages external access to services.

What it does:

  • Routes HTTP/HTTPS traffic to your services
  • Handles TLS termination (decrypts HTTPS)
  • Uses certificates from Kubernetes Secrets

With cert-manager: When you add the annotation cert-manager.io/cluster-issuer: "letsencrypt-dns01", cert-manager automatically requests a certificate for that ingress.

Part 4: The Implementation

Step 1: Create a Cloudflare API Token

Cloudflare manages my DNS. cert-manager needs API access to create TXT records.

  1. Go to https://dash.cloudflare.com/profile/api-tokens
  2. Click “Create Token”
  3. Choose “Custom token”
  4. Configure:
    • Token name: cert-manager-dns01
    • Permissions: Zone → DNS → Edit
    • Zone Resources: Include → Specific zone → landryzetam.net
  5. Create and copy the token

Why these specific permissions? We’re following the principle of least privilege. This token can only edit DNS records for one specific zone—it can’t access other Cloudflare features or other domains.

Step 2: Store the Token in AWS Secrets Manager

aws secretsmanager create-secret \
  --name "cert-manager/cloudflare-api-token" \
  --secret-string '{"api-token":"YOUR_TOKEN_HERE"}'

Why AWS Secrets Manager?

  • Secrets never touch Git
  • Access is controlled by IAM policies
  • Audit trail of who accessed what
  • Encryption at rest

Step 3: Update IAM Policy

External Secrets Operator runs with an IAM user that needs permission to read the new secret:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue",
    "secretsmanager:DescribeSecret"
  ],
  "Resource": [
    "arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:cert-manager/*"
  ]
}

Step 4: Create the SecretStore

This tells ESO how to connect to AWS:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secret-store
  namespace: cert-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: AWS_ACCESS_KEY_ID
          secretAccessKeySecretRef:
            name: aws-credentials
            key: AWS_SECRET_ACCESS_KEY

What’s happening here?

  • We’re creating a SecretStore in the cert-manager namespace
  • It connects to AWS Secrets Manager in us-east-1
  • Authentication uses AWS credentials stored in another Kubernetes secret

Step 5: Create the ExternalSecret

This tells ESO which secret to sync:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore
  target:
    name: cloudflare-api-token-secret
  data:
    - secretKey: api-token
      remoteRef:
        key: cert-manager/cloudflare-api-token
        property: api-token

What’s happening here?

  • Every hour (refreshInterval: 1h), ESO checks AWS
  • It fetches cert-manager/cloudflare-api-token from AWS
  • It creates a Kubernetes Secret called cloudflare-api-token-secret
  • The secret has a key api-token with the token value

Step 6: Create the DNS-01 ClusterIssuer

Now we tell cert-manager how to use DNS-01 with Cloudflare:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-dns01
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

What’s happening here?

  • server: Points to Let’s Encrypt’s production API
  • email: Let’s Encrypt sends expiry warnings here
  • privateKeySecretRef: Where to store the ACME account private key
  • solvers: Use DNS-01 with Cloudflare, using our synced API token

Step 7: Update the Ingress

Finally, tell your ingress to use the new issuer:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: paperless-ngx
  namespace: paperless-ngx
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-dns01"
spec:
  ingressClassName: traefik
  rules:
  - host: paperless.landryzetam.net
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: paperless-ngx
            port:
              number: 8000
  tls:
  - hosts:
    - paperless.landryzetam.net
    secretName: paperless-tls-cert

The magic annotation: cert-manager.io/cluster-issuer: "letsencrypt-dns01" triggers cert-manager to:

  1. See the ingress wants TLS for paperless.landryzetam.net
  2. Check if paperless-tls-cert secret exists
  3. If not, request a certificate using the letsencrypt-dns01 issuer
  4. Store the certificate in the secret
  5. Traefik picks up the secret and starts serving HTTPS

Part 5: Watching It Work

After applying the manifests, watch the certificate lifecycle:

# Check certificate status
$ kubectl get certificate -n paperless-ngx
NAME                 READY   SECRET               AGE
paperless-tls-cert   False   paperless-tls-cert   30s

# A challenge is created
$ kubectl get challenge -n paperless-ngx
NAME                           STATE     DOMAIN                      AGE
paperless-tls-cert-1-xxx       pending   paperless.landryzetam.net   10s

# cert-manager created the DNS record!
$ dig +short TXT _acme-challenge.paperless.landryzetam.net
"-BiwNVBzCuCPW3_IPC1sZ2fZbNkRUB1SDQnYMFdq5FA"

# After ~60 seconds, certificate is ready
$ kubectl get certificate -n paperless-ngx
NAME                 READY   SECRET               AGE
paperless-tls-cert   True    paperless-tls-cert   90s

# HTTPS works!
$ curl -I https://paperless.landryzetam.net
HTTP/2 302

Part 6: The Complete Data Flow

Let’s trace what happens when you access https://paperless.landryzetam.net:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Browser   │────▶│   Traefik    │────▶│ Paperless   │
│             │◀────│  (Ingress)   │◀────│   Service   │
└─────────────┘     └──────────────┘     └─────────────┘
                           │
                           │ Uses certificate from
                           ▼
                    ┌──────────────┐
                    │ K8s Secret   │
                    │ (TLS cert)   │
                    └──────────────┘
                           ▲
                           │ Created by
                           │
                    ┌──────────────┐     ┌──────────────┐
                    │ cert-manager │────▶│ Let's        │
                    │              │◀────│ Encrypt      │
                    └──────────────┘     └──────────────┘
                           │
                           │ Creates DNS record via
                           ▼
                    ┌──────────────┐
                    │ Cloudflare   │
                    │ API          │
                    └──────────────┘
                           ▲
                           │ Token from
                           │
                    ┌──────────────┐     ┌──────────────┐
                    │ External     │────▶│ AWS Secrets  │
                    │ Secrets Op   │◀────│ Manager      │
                    └──────────────┘     └──────────────┘

Key Takeaways

  1. TLS certificates prove identity and enable encryption - Without them, anyone can eavesdrop or impersonate your services

  2. DNS-01 is the answer for internal services - It proves domain ownership without requiring your service to be publicly accessible

  3. cert-manager automates the entire certificate lifecycle - Request, challenge, issuance, renewal—all hands-free

  4. External Secrets keeps credentials out of Git - Store sensitive data in AWS Secrets Manager, sync it into Kubernetes automatically

  5. One ClusterIssuer serves all namespaces - Set it up once, use it everywhere with a simple annotation

  6. Certificates auto-renew - cert-manager handles renewal before expiry, no manual intervention needed