Complete Kubernetes Security Pipeline - Kube-bench and Kubescape to Obsidian

What if your Kubernetes cluster could write comprehensive security reports directly into your second brain every night? Not just one perspective, but two complementary views: infrastructure security and workload security. I built exactly that—a pipeline combining kube-bench (CIS benchmarks) and kubescape (NSA/MITRE frameworks) that publishes human-readable reports to Obsidian automatically. Here’s how the complete system works.

Part 1: Why You Need Both Tools

The Security Visibility Gap

Running a Kubernetes homelab means wearing many hats. Security scanning tools exist, but most people either:

• Run them once, look at the output, and forget about it
• Set up dashboards they never check
• Parse JSON files manually when something breaks

I wanted security data that’s impossible to ignore—automatically appearing in my daily knowledge system.

Two Tools, Complete Coverage

Here’s the key insight: kube-bench and kubescape answer different questions.

Kube-bench checks if your kubelet has the right flags, if etcd is encrypted, if the API server has proper authentication. Infrastructure stuff.

Kubescape checks if your pods run as root, if containers have resource limits, if RBAC permissions are too broad, if your images have known CVEs. Application stuff.

You need both for complete visibility.

Part 2: Architecture Overview

The Combined Pipeline

┌─────────────────────────────────────────────────────────────────────────┐
│                         SECURITY SCANNING                                │
├─────────────────────────────────┬───────────────────────────────────────┤
│         KUBE-BENCH              │           KUBESCAPE                    │
│                                 │                                        │
│  ┌─────────────────┐            │    ┌─────────────────┐                 │
│  │ CronJob 2:00 AM │            │    │    Operator     │                 │
│  │ (CIS Benchmark) │            │    │  (Continuous)   │                 │
│  └────────┬────────┘            │    └────────┬────────┘                 │
│           │                     │             │                          │
│           ▼                     │             ▼                          │
│  ┌─────────────────┐            │    ┌─────────────────┐                 │
│  │   JSON Files    │            │    │ Kubernetes CRDs │                 │
│  │   (NFS Volume)  │            │    │ - ConfigScans   │                 │
│  └────────┬────────┘            │    │ - VulnSummaries │                 │
│           │                     │    │ - WorkloadScans │                 │
│           │                     │    └────────┬────────┘                 │
├───────────┼─────────────────────┼─────────────┼──────────────────────────┤
│           │      REPORTERS      │             │                          │
│           ▼                     │             ▼                          │
│  ┌─────────────────┐            │    ┌─────────────────┐                 │
│  │ kube-bench      │            │    │ kubescape       │                 │
│  │ reporter        │            │    │ reporter        │                 │
│  │ CronJob 2:30 AM │            │    │ CronJob 3:00 AM │                 │
│  └────────┬────────┘            │    └────────┬────────┘                 │
│           │                     │             │                          │
│           └──────────┬──────────┴─────────────┘                          │
│                      │                                                   │
│                      ▼                                                   │
│             ┌─────────────────┐                                          │
│             │    Obsidian     │                                          │
│             │  Local REST API │                                          │
│             └─────────────────┘                                          │
└─────────────────────────────────────────────────────────────────────────┘

Timing Strategy

The CronJobs are staggered to avoid overlap and ensure fresh data:

By morning, four fresh reports await in Cluster/Security/.

Part 3: The Kube-bench Pipeline

How Kube-bench Works

Kube-bench runs the CIS Kubernetes Benchmark—a checklist of 100+ security controls for cluster infrastructure. It’s a simple model:

1. CronJob triggers at 2:00 AM
2. kube-bench container runs with host access (needed to check node config)
3. Results written to JSON file on NFS volume
4. Container exits

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench
  namespace: kube-bench
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true  # Required for node checks
          containers:
          - name: kube-bench
            image: aquasec/kube-bench:latest
            command: ["kube-bench", "--json", "--outputfile", "/results/results.json"]
            volumeMounts:
            - name: results
              mountPath: /results
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true

The Kube-bench Reporter

The reporter is ~200 lines of Python that:

1. Finds the latest JSON results file
2. Parses and counts checks by status (PASS, FAIL, WARN, INFO)
3. Generates summary and detailed markdown reports
4. Publishes to Obsidian via REST API

#!/usr/bin/env python3
"""Kube-bench report publisher to Obsidian."""
import os
import json
import requests
from pathlib import Path
from datetime import datetime

RESULTS_DIR = os.getenv("RESULTS_DIR", "/results")
OBSIDIAN_API_URL = os.getenv("OBSIDIAN_API_URL")
OBSIDIAN_API_KEY = os.getenv("OBSIDIAN_API_KEY")
OBSIDIAN_FOLDER = os.getenv("OBSIDIAN_FOLDER", "Cluster/Security")

def find_latest_results():
    """Find the most recent results file."""
    results_path = Path(RESULTS_DIR)
    json_files = list(results_path.glob("*.json"))
    if not json_files:
        raise FileNotFoundError("No results files found")
    return max(json_files, key=lambda f: f.stat().st_mtime)

def count_by_status(data):
    """Count checks by status."""
    counts = {"PASS": 0, "FAIL": 0, "WARN": 0, "INFO": 0}
    for control in data.get("Controls", []):
        for test in control.get("tests", []):
            for result in test.get("results", []):
                status = result.get("status", "INFO")
                counts[status] = counts.get(status, 0) + 1
    return counts

def publish_to_obsidian(note_path, content):
    """Publish markdown to Obsidian."""
    url = f"{OBSIDIAN_API_URL}/vault/{note_path}"
    headers = {
        "Authorization": f"Bearer {OBSIDIAN_API_KEY}",
        "Content-Type": "text/markdown"
    }
    response = requests.put(url, headers=headers, data=content.encode('utf-8'),
                           timeout=30, verify=False)
    response.raise_for_status()

Part 4: The Kubescape Pipeline

How Kubescape Differs

Kubescape runs as an operator—continuously scanning your cluster and storing results in Custom Resource Definitions:

$ kubectl get crd | grep kubescape
configurationscansummaries.spdx.softwarecomposition.kubescape.io
vulnerabilitymanifestsummaries.spdx.softwarecomposition.kubescape.io
workloadconfigurationscans.spdx.softwarecomposition.kubescape.io

Instead of reading files, we query the Kubernetes API directly.

The Kubescape Reporter

This reporter uses the kubernetes Python client to query CRDs:

from kubernetes import client, config

def get_configuration_scan_summaries(api):
    """Fetch all configuration scan summaries."""
    return api.list_cluster_custom_object(
        group="spdx.softwarecomposition.kubescape.io",
        version="v1beta1",
        plural="configurationscansummaries"
    )

def get_vulnerability_summaries(api):
    """Fetch all vulnerability manifest summaries."""
    return api.list_cluster_custom_object(
        group="spdx.softwarecomposition.kubescape.io",
        version="v1beta1",
        plural="vulnerabilitymanifestsummaries"
    )

def process_config_scans(items):
    """Process configuration scan summaries into metrics."""
    results = {
        "total_scans": len(items),
        "by_severity": {"critical": 0, "high": 0, "medium": 0, "low": 0},
        "by_namespace": {}
    }

    for item in items:
        spec = item.get("spec", {})
        severities = spec.get("severities", {})

        for sev in ["critical", "high", "medium", "low"]:
            results["by_severity"][sev] += severities.get(sev, 0)

        ns = item["metadata"].get("namespace", "cluster-wide")
        if ns not in results["by_namespace"]:
            results["by_namespace"][ns] = {"critical": 0, "high": 0}
        results["by_namespace"][ns]["critical"] += severities.get("critical", 0)
        results["by_namespace"][ns]["high"] += severities.get("high", 0)

    return results

RBAC for CRD Access

The kubescape reporter needs a ClusterRole to read CRDs:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubescape-reporter
rules:
  - apiGroups: ["spdx.softwarecomposition.kubescape.io"]
    resources:
      - configurationscansummaries
      - vulnerabilitymanifestsummaries
      - workloadconfigurationscans
      - workloadconfigurationscansummaries
    verbs: ["get", "list"]

Part 5: Shared Infrastructure

Secret Management with External Secrets

Both reporters need credentials for the Obsidian REST API. I store these in AWS Secrets Manager and sync via External Secrets Operator:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: obsidian-api-secret
  namespace: kube-bench  # Same pattern in kubescape namespace
spec:
  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore
  target:
    name: obsidian-api
  data:
    - secretKey: api_url
      remoteRef:
        key: obsidian/api
        property: api_url
    - secretKey: api_key
      remoteRef:
        key: obsidian/api
        property: api_key

Multi-Architecture Docker Builds

If you develop on Apple Silicon but deploy to Intel nodes:

# Create multi-arch builder
docker buildx create --name multiarch --driver docker-container --use

# Build for both architectures
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t lzetam/kube-bench-reporter:latest \
  --push .

The Dockerfile Pattern

Both reporters use the same secure pattern:

FROM python:3.12-slim
WORKDIR /app
RUN useradd -m -u 1000 appuser
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=appuser:appuser reporter.py .
USER appuser
ENTRYPOINT ["python", "reporter.py"]

Part 6: The Result

Daily Security Reports

Every morning, my Obsidian vault has fresh reports in Cluster/Security/:

Cluster/Security/
├── Kube-Bench-Summary-2025-12-19.md    # CIS benchmarks (infrastructure)
├── Kube-Bench-Details-2025-12-19.md
├── Kubescape-Summary-2025-12-19.md     # NSA/MITRE/CIS (workloads)
└── Kubescape-Details-2025-12-19.md

Sample Kube-bench Summary

# Kube-Bench Security Summary - 2025-12-19

## Results Overview

| Status | Count | Percentage |
|--------|-------|------------|
| PASS | 45 | 52.3% |
| FAIL | 18 | 20.9% |
| WARN | 12 | 14.0% |
| INFO | 11 | 12.8% |

## Failed Checks

- 1.2.5: Ensure that the --kubelet-certificate-authority argument is set
- 1.2.6: Ensure that the --authorization-mode argument includes Node
...

Sample Kubescape Summary

# Kubescape Security Summary - 2025-12-19

## Overview

| Metric | Count |
|--------|-------|
| Configuration Scans | 34 |
| Vulnerability Summaries | 158 |
| Workload Scans | 500 |

## Severity Distribution

| Severity | Configuration | Vulnerabilities |
|----------|---------------|-----------------|
| Critical | 3 | 12 |
| High | 15 | 47 |
| Medium | 28 | 89 |
| Low | 41 | 203 |

## Namespaces with Critical Issues

| Namespace | Critical | High |
|-----------|----------|------|
| default | 2 | 5 |
| monitoring | 0 | 3 |
| gpu-operator | 1 | 2 |

Part 7: Lessons Learned

1. Different Data Sources Need Different Approaches

Kube-bench outputs files. Kubescape stores CRDs. Don’t try to force them into the same pattern—embrace the differences.

2. Stagger Your Jobs

Running everything at once causes resource spikes and race conditions. Space out your CronJobs.

3. Network Assumptions Break

My Obsidian REST API runs on my Mac. When DHCP changed the IP, both reporters failed. Consider:

• Static IP assignment for the Obsidian host
• DNS name resolution
• Updating secrets when IPs change

4. RBAC Scoping Matters

The kubescape reporter only needs read access to specific CRDs. Define minimal ClusterRoles.

5. Non-Root Containers Need Care

File permissions on NFS volumes tripped me up. Always test with the same UID your container runs as.

6. GitOps Everything

Both reporters, their CronJobs, RBAC, ConfigMaps—all in Git. When something breaks, git log tells you what changed.

Key Takeaways

1. Layer your security tools - Kube-bench for infrastructure, kubescape for workloads. Together they give complete visibility.

2. Automate the boring stuff - Security scanning only works if you look at the results. Publishing to your knowledge system makes it visible.

3. CRDs are first-class data sources - Modern operators store results in CRDs. Query them via the Kubernetes API instead of parsing files.

4. Bridge the gap - Security tools produce machine-readable output. Simple Python scripts transform that into human-readable notes.

5. Make security impossible to ignore - Reports in your daily vault are harder to forget than dashboards you never check.