Building an Automated Audio Transcription Pipeline on Kubernetes
Table of Contents
Every day, raw audio recordings land on an SFTP server. By morning, they’re compressed, transcribed, and stored as searchable JSON - all without human intervention. Here’s how I built this pipeline on my homelab Kubernetes cluster.
The Problem
I had a growing collection of audio recordings that needed transcription. Manual processing wasn’t sustainable: download files, compress them, run through a transcription service, organize the output. This needed automation.
The requirements were straightforward:
- Pull new audio files automatically from an SFTP source
- Compress WAV files to a format optimized for speech recognition
- Transcribe using Whisper (the best open-source STT model)
- Store transcripts in a searchable format
- Run daily without manual intervention
The Architecture
The pipeline starts with physical hardware and flows through several automated stages:
┌─────────────────────────┐
│ Rode Recorder │
│ (physical device) │
└─────────────────────────┘
↓
Auto-upload WAV
↓
┌─────────────────────────┐
│ NAS Storage │
└─────────────────────────┘
↓
NFS mount
↓
┌─────────────────────────┐
│ SFTP Server (K8s) │
│ exposes NAS via SFTP │
└─────────────────────────┘
↓
2:00 AM
↓
┌─────────────────────────┐
│ audio-compressor │
│ WAV → MP3 (32kbps) │
│ 16kHz mono │
└─────────────────────────┘
↓
NFS Storage
↓
4:00 AM
↓
┌─────────────────────────┐
│ audio-transcriber │
│ Whisper API │
│ → JSON transcripts │
└─────────────────────────┘
↓
NFS Storage
The two-hour gap between jobs ensures compression completes before transcription begins.
The Recording Setup
The pipeline begins with a Rode recording device. When I record, the device automatically uploads WAV files to a designated folder on my NAS. Since the NAS doesn’t provide SFTP natively, I deployed an SFTP server in Kubernetes that mounts the NAS storage via NFS and exposes it over SFTP.
This setup means recordings flow into the pipeline with zero manual intervention - I press record, and hours later the transcript appears.
Component 1: The Audio Compressor
The compressor CronJob runs at 2 AM daily. It connects to the SFTP server, identifies new recordings, and compresses them with settings optimized for speech recognition:
apiVersion: batch/v1
kind: CronJob
metadata:
name: audio-compressor
namespace: audio-compressor
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: audio-compressor
image: lzetam/audio-compressor:latest
env:
- name: SFTP_HOST
value: "sftp-server.sftp-server.svc.cluster.local"
- name: SAMPLE_RATE
value: "16000" # 16kHz - Whisper's native rate
- name: CHANNELS
value: "1" # Mono
- name: BITRATE
value: "32k" # 32 kbps - sufficient for speech
- name: AUDIO_FORMAT
value: "mp3"
Why these settings?
Whisper processes audio at 16kHz internally. Feeding it higher sample rates wastes bandwidth and processing time. Mono audio is sufficient for speech (stereo provides no benefit). And 32kbps MP3 preserves speech clarity while dramatically reducing file sizes - a 100MB WAV becomes roughly 3MB.
The compressor also uploads the compressed files back to the SFTP server’s /compressed directory as a backup.
Component 2: The Transcriber
Two hours later, the transcriber CronJob wakes up:
apiVersion: batch/v1
kind: CronJob
metadata:
name: audio-transcriber
namespace: audio-compressor
spec:
schedule: "0 4 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: audio-transcriber
image: lzetam/audio-transcriber:latest
env:
- name: WHISPER_MODEL
value: "faster-distil-whisper-large-v3-en"
- name: OUTPUT_FORMAT
value: "json"
- name: SKIP_PROCESSED
value: "true"
volumeMounts:
- name: compressed-audio
mountPath: /data/compressed
readOnly: true
- name: transcriptions
mountPath: /data/transcriptions
The SKIP_PROCESSED: true flag is crucial - it prevents re-transcribing files that have already been processed, making the job idempotent.
Component 3: The Whisper Backend
I run a dedicated Whisper GPU deployment using the faster-distil-whisper-large-v3-en model. This distilled version offers near-identical accuracy to the full model at significantly faster inference speeds:
apiVersion: apps/v1
kind: Deployment
metadata:
name: whisper-gpu
namespace: whisper
spec:
template:
spec:
containers:
- name: whisper
resources:
limits:
nvidia.com/gpu: 1
The Whisper service runs on a node with an RTX 3050, providing hardware acceleration for transcription. The transcriber CronJob connects via an internal Kubernetes service endpoint.
Storage Architecture
Both CronJobs share storage through NFS-backed PersistentVolumeClaims:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: audio-compressed-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-csi-v2
resources:
requests:
storage: 100Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: audio-transcript-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-csi-v2
resources:
requests:
storage: 100Gi
ReadWriteMany access mode allows multiple pods to mount the same volume - essential when CronJob pods might overlap or when you need to inspect the data from other workloads.
Secrets Management
SFTP credentials and Whisper API keys are managed through External Secrets Operator, pulling from AWS Secrets Manager:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: sftp-credentials-secret
spec:
secretStoreRef:
name: aws-secret-store
kind: SecretStore
data:
- secretKey: username
remoteRef:
key: sftp-server/credentials
property: username
- secretKey: password
remoteRef:
key: sftp-server/credentials
property: password
No credentials in Git. Ever.
The GitOps Workflow
All changes deploy through FluxCD. When I push a manifest change:
- Flux detects the Git change
- Kustomize renders the manifests
- Resources apply to the cluster
git add . && git commit -m "feat: update transcriber config"
git push
flux reconcile source git flux-system
flux reconcile kustomization apps
This means I can tune compression settings, change the Whisper model, or adjust schedules - all through Git commits with full audit history.
Results
The pipeline has been running for weeks, processing recordings daily:
$ ls -la /data/transcriptions/ | head -10
-rw-rw-r-- 1 1000 1001 58681 Dec 12 10:07 10-16-25-03_compressed_20251206.json
-rw-rw-r-- 1 1000 1001 90041 Dec 12 08:32 10-17-25-01_compressed_20251206.json
...
Each JSON transcript includes timestamps, allowing me to search and navigate the audio content programmatically.
Key Takeaways
Separate concerns: Compression and transcription as independent CronJobs means I can tune, debug, and scale each independently.
Optimize for your model: Whisper works at 16kHz mono. Don’t waste resources on higher fidelity the model can’t use.
Make jobs idempotent:
SKIP_PROCESSEDflags prevent duplicate work and allow safe re-runs.GitOps everything: Infrastructure as code means changes are reviewable, reversible, and reproducible.
Use shared storage wisely: NFS with
ReadWriteManyenables flexible data sharing between jobs.
What’s Next
The transcripts are just data until they’re useful. Next steps include:
- Building a search interface over the JSON transcripts
- Adding speaker diarization to identify who’s speaking
- Integrating with an LLM for summarization and Q&A
Related Posts
- Kubernetes (K8s) - The Fako Cluster - The infrastructure running this pipeline
- Home Assistant Voice Pipeline - Another Whisper-powered project
- GitOps Framework - How deployments work