ChatBuddy - Voice AI Companion for Children

Overview

ChatBuddy is an AI-powered voice companion designed specifically for children aged 3-8. When a child asks “Why is the sky blue?”, they get a fun, age-appropriate answer—not a Wikipedia article.

The system runs entirely on-device with 5 containerized services on hardware with just 8GB RAM, ensuring complete privacy while delivering natural voice interactions with educational content grounded in verified facts.

Why Build This?

Kids don’t want to type. They want to talk. And they need answers that:

• Use simple vocabulary they understand
• Stay short (2 sentences max)
• Are factually accurate (no AI hallucinations about cheese moons)
• Come back fast (5-8 seconds, before they lose interest)

Key Features

• Dual Voice Modes: Auto Speech (hands-free) and Push to Talk for noisy environments
• Smart Speech Recognition: faster-whisper with WebRTC VAD filtering
• Local AI: Ollama running gemma3 entirely on-device
• Natural Voice: Kokoro-ONNX text-to-speech with kid-friendly voice
• Educational Content: RAG-powered facts from curated children’s literature
• Resource Efficient: Runs on 8GB RAM with careful memory management
• Privacy First: No cloud dependencies, all processing local

Architecture

The Voice Pipeline

Browser Mic → WebRTC VAD → STT (Whisper) → LLM (Ollama) → TTS (Kokoro) → Audio Playback

Service Components

ChatBuddy runs 5 containers simultaneously:

Total: ~7.25GB allocated on an 8GB system. It works because not all services peak simultaneously.

Data Flow

Voice Path (Socket.IO)

Browser Microphone
    ↓
16kHz WAV chunks
    ↓
WebRTC VAD (Voice Activity Detection)
    ↓
Auto-send after ~0.6s silence
    ↓
faster-whisper transcription
    ↓
RAG context enrichment
    ↓
Ollama (gemma3:4b) response
    ↓
Sentence-by-sentence TTS
    ↓
Streaming audio playback

Text Path (SSE) React useChat → POST /api/chat/stream → Ollama → SSE tokens → Frontend

Performance Metrics

On Apple Silicon (Mac), total response time is 5-8 seconds:

Voice Modes

Auto Speech Mode (Hands-free)

Continuous listening with automatic speech detection. Browser VAD detects speech and sends automatically after ~0.6s of silence. Perfect for focused one-on-one time with kids.

Push to Talk Mode

Press and hold button to record, release to send. More reliable in noisy environments—essential for homes with multiple kids or background TV.

Voice Activity Detection (VAD)

WebRTC VAD handles the tricky problem of knowing when a child has finished speaking. Kids pause, think, and start over.

# WebRTC VAD configuration
VAD_AGGRESSIVENESS = 1  # 0-3, lower = less aggressive
VAD_FRAME_DURATION_MS = 30
VAD_MIN_SPEECH_RATIO = 0.15

For children’s voices (higher pitch, more pauses), aggressiveness of 0 or 1 works best.

RAG Educational System

The Problem with Pure LLM

Small language models are great for conversation but terrible for factual accuracy. They’ll confidently tell kids that Pluto is the biggest planet or dinosaurs lived with humans.

The RAG Solution

RAG grounds LLM responses in verified source material:

Child's Question
       ↓
Topic Extraction
       ↓
Vector Search / Keyword Match
       ↓
Retrieved Facts (from curated corpus)
       ↓
LLM + Facts → Kid-Friendly Response

Educational Content Sources

Topics covered: solar system, animals, dinosaurs, colors, shapes, numbers, letters, and nature.

SQLite Backend

All educational content lives in a single SQLite database, distributed via S3:

CREATE TABLE topics (
    id INTEGER PRIMARY KEY,
    name TEXT UNIQUE,
    age_min INTEGER,
    age_max INTEGER
);

CREATE TABLE facts (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER REFERENCES topics(id),
    subject TEXT,
    fact TEXT,
    source TEXT
);

CREATE TABLE dictionary_cache (
    word TEXT PRIMARY KEY,
    definition TEXT,
    pronunciation_url TEXT
);

S3 Distribution: chatbuddy-rag-data.s3.amazonaws.com/data.tar.gz downloads during Docker build.

RAG Server API

Before vs After RAG

Before RAG:

“Tell me about Pluto” “Pluto is a faraway world of ice and mystery, swirling through the cosmic dance…”

After RAG:

“Tell me about Pluto” “Pluto is a tiny dwarf planet - smaller than our Moon! It’s super far away and freezing cold. Brrrr!”

The Kid-Friendly System Prompt

def get_educational_system_prompt() -> str:
    return f"""You are {COMPANION_NAME}, a fun AI friend for kids ages 3-8.

⚠️ CRITICAL RULES - FOLLOW EXACTLY:
1. MAX 2 SENTENCES. Never more than 40 words total.
2. NEVER ask questions. No "Cool, right?" or "Want to learn more?"
3. Just answer and stop. The kid will ask more if they want.
4. Use the VERIFIED FACTS below as your answer.

✨ YOUR STYLE:
- Playful, geeky, excited: "Ooh!", "Fun fact!", "So cool!"
- Simple words for little kids
- Sound effects welcome: "Boom!", "Whoosh!", "ROARRR!"
"""

Constraints: 2 sentences maximum, no follow-up questions, sound effects encouraged.

Configuration

Environment Variables

Key settings in .env.mac:

Docker Compose Configurations

Development (5 containers)

# docker-compose.dev.yml - includes containerized Ollama
services:
  ollama:
    image: ollama/ollama:latest
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
    deploy:
      resources:
        limits:
          memory: 2.5G

Production (4 containers)

# docker-compose.prod.yml - uses native Ollama on host
services:
  client:
    environment:
      - OLLAMA_URL=http://host.docker.internal:11434
      - OLLAMA_MODEL=gemma3:4b
    extra_hosts:
      - "host.docker.internal:host-gateway"

Why host Ollama in production?

• Metal acceleration (GPU/Neural Engine on Apple Silicon)
• Better memory management
• Easier model switching

Quick Start

Prerequisites

• Mac (8GB+ RAM) or Jetson Nano Super (8GB RAM)
• Docker and Docker Compose installed

Local Development

# Clone and navigate
git clone https://github.com/lyzetam/3xl.git
cd 3xl/chatbuddy

# Start all services (5 containers including Ollama)
docker compose -f docker-compose.dev.yml up -d --build

# Wait for model download (first time only)
docker logs chatbuddy-ollama -f

# Access the app
open http://localhost:3000

Production (Mac Studio)

# Requires native Ollama running on host
brew install ollama && ollama serve
ollama pull gemma3:4b

# Start services (4 containers, uses host Ollama)
docker compose -f docker-compose.prod.yml up -d --build

Operations

Health Checks

curl http://localhost:3000/health | jq     # Client
curl http://localhost:5002/health | jq     # STT
curl http://localhost:5003/health | jq     # TTS
curl http://localhost:8000/health | jq     # RAG
curl http://localhost:11434/api/tags | jq  # Ollama

Resource Monitoring

docker stats

Typical output during conversation:

CONTAINER            CPU %   MEM USAGE / LIMIT
chatbuddy-ollama     45%     1.8GiB / 2.5GiB
chatbuddy-stt-api    12%     980MiB / 1.5GiB
chatbuddy-tts-api    8%      1.2GiB / 2GiB
chatbuddy-rag        0.5%    45MiB / 256MiB
chatbuddy-client     3%      180MiB / 1GiB

Model Management

# Download models
docker exec chatbuddy-ollama ollama pull gemma3:1b
docker exec chatbuddy-ollama ollama pull gemma3:4b

# List models
docker exec chatbuddy-ollama ollama list

# Remove old model
docker exec chatbuddy-ollama ollama rm gemma2:2b

Hybrid Development

# Terminal 1: Backend services only
docker compose -f docker-compose.dev.yml up -d ollama stt-api tts-api rag-server

# Terminal 2: Frontend with hot reload
cd frontend && npm run dev

# Terminal 3: Backend locally (if modifying client_app.py)
cd backend && python3 client_app.py

Troubleshooting

Ollama Not Responding

# Check status
curl http://localhost:11434/api/tags
docker exec chatbuddy-ollama ollama list

# Pull model if missing
docker exec chatbuddy-ollama ollama pull gemma3:1b
docker restart chatbuddy-ollama

Security Considerations

• Local network only - No external API calls
• No authentication by default (add if exposing externally)
• In-memory history - Cleared on restart
• All processing on-device - Complete privacy

Future Improvements

• Child profiles for personalized responses
• Conversation memory across sessions
• Jetson Nano deployment for dedicated hardware
• Multi-language support
• Emotion detection in voice
• Proactive learning suggestions

Lessons Learned

1. Latency matters more than accuracy - Kids won’t wait. A fast, good-enough response beats a slow, perfect one.

2. Sentence-by-sentence TTS - Don’t wait for the full LLM response. Stream sentences to TTS as they complete.

3. Resource limits are your friend - Docker memory limits prevent any single service from starving others.

4. Two voice modes solve different problems - Auto Speech for focused time, Push to Talk for chaotic households.

5. RAG grounds hallucinations - Small models need verified facts. The corpus is as important as the model.

6. Name things for future-you - Clear names (RAG vs MCP) reduce cognitive load.

ChatBuddy is open source at github.com/lyzetam/3xl/chatbuddy. Building AI for kids forces you to think about what truly matters: simplicity, speed, and safety.

📚 Related Projects & Posts

AI & Machine Learning

• Ollama AI Model Server
• Home Assistant Voice Pipeline
• Open WebUI - AI Chat Interface

Infrastructure

• Kubernetes (K8s) - The Fako Cluster
• From GitHub Notes to Production Blog