DarkNebula — Production-Grade Homelab Infrastructure

Table of Contents

• The Challenge
• Architecture & Solution
• Tech Stack
• Key Engineering Decisions
• Service Inventory
• Automation & Scripts
• Bot Systems
• Self-Healing Agent
• Security Hardening
• Deployment
• Roadmap

The Challenge

Managing a homelab infrastructure with over 40 Docker containers, multiple network access methods, hybrid storage (SSD and HDD), and diverse service categories (media, development, monitoring, security) presents significant operational challenges that traditional manual management approaches cannot address effectively.

Before this system, service management required executing individual docker compose commands for each container, with no unified state tracking or mode-based resource optimization. Monitoring required manually checking multiple dashboards, with no consolidated view of system health. Backup operations were executed manually with no automated verification. Security monitoring relied on separate tools with no unified threat response. The lack of remote access capabilities limited administration to local network connections or complex SSH tunnel setups.

The fundamental challenge was building an infrastructure that could operate autonomously with minimal manual intervention while providing comprehensive remote control through multiple interfaces (CLI, Telegram, Discord). The system needed to be intelligent enough to detect failures and recover automatically, resource-efficient enough to operate on consumer hardware, and secure enough to expose via Cloudflare Tunnel without compromising the host system.

The requirement: Build a production-grade, self-healing homelab infrastructure with centralized configuration management, multi-interface remote control (CLI + Telegram + Discord), automated resource optimization through mode switching (Work/Media/Full), hardware-level power monitoring via Intel RAPL, and enterprise-grade security hardening with CrowdSec integration.

Architecture & Solution

The architecture follows a modular, metadata-driven design where every service contains a .homelab-meta file defining its tier classification (core, work, media, extra). This metadata drives all automation logic including mode switching, self-healing scope, and status reporting.

The configuration architecture implements a single-source-of-truth model where all service configurations reference a centralized global.env file through symlinks. The update-env.sh script recursively verifies and recreates these symlinks, ensuring configuration changes propagate consistently across all services without manual intervention.

Tech Stack

Key Engineering Decisions

1. Metadata-Driven Service Classification

Every service directory contains a .homelab-meta file with tier classification. This metadata drives all automation logic without hardcoding service lists.

# ~/docker/applications/media/jellyfin/.homelab-meta
NAME=Jellyfin
TIER=media
CATEGORY=media

The homelab-ctl.sh script reads these metadata files to determine which services should run in each mode. This approach eliminates maintenance overhead when adding or removing services.

2. Centralized Configuration with Symlink Propagation

Rather than maintaining individual .env files for each service, all configurations reference a single global.env through symlinks maintained by update-env.sh.

# update-env.sh excerpt
SERVICES=(
    "base/caddy"
    "base/portainer"
    "base/cloudflared"
    "applications/media/jellyfin"
    "applications/monitoring/telemetry"
    ...
)

for service in "${SERVICES[@]}"; do
    ln -sf "$GLOBAL_ENV" "$DOCKER_DIR/$service/.env"
done

This design ensures that updating a password or API key in one place automatically propagates to all services, eliminating configuration drift between containers.

3. Mode-Based Resource Management

The system implements three operational modes that control which service tiers are active, enabling resource optimization based on usage patterns:

The homelab-ctl.sh script implements mode switching by stopping and archiving services that should not run in the target mode, moving their compose files to ~/docker/archived/ to free memory and CPU resources.

# homelab-ctl.sh mode switching
case "$MODE" in
    work)
        archive_tier "media"
        archive_tier "extra"
        ;;
    media)
        restore_tier "media"
        archive_tier "extra"
        ;;
    full)
        restore_all
        ;;
esac

4. Intel RAPL Power Monitoring

The system uses Scaphandre to read CPU power data directly from Intel RAPL (Running Average Power Limit) hardware counters, providing per-second power attribution at the process level.

# power-monitor.sh calculation
CPU_POWER=$(cat /sys/class/powercap/intel-rapl:0/energy_uj)
GPU_POWER=$(cat /sys/class/hwmon/hwmon*/power1_average 2>/dev/null || echo 0)
OVERHEAD=15  # Motherboard, RAM, SSD, NIC
PSU_LOSS=0.85  # 85% efficiency

TOTAL_POWER=$(( (CPU_POWER + GPU_POWER + OVERHEAD) / PSU_LOSS ))

This data is scraped by Prometheus every 15 seconds and visualized in Grafana with electricity cost calculations based on ₹8/kWh (Assam APDCL rates).

5. Exponential Backoff with Chronic Failure Suppression

The self-healing agent tracks consecutive failures per container and implements progressive backoff to prevent restart loops while still allowing recovery attempts.

# Failure tracking
FAILURE_COUNT=$(cat ~/docker/status/heal-tracker/${CONTAINER}.count 2>/dev/null || echo 0)
((FAILURE_COUNT++))

if [ $FAILURE_COUNT -ge 7 ]; then
    echo "⛔ CHRONIC: Skipping $CONTAINER (suppressed)"
    exit 0
fi

# Backoff intervals
if [ $FAILURE_COUNT -le 2 ]; then
    RESTART_NOW
elif [ $FAILURE_COUNT -le 4 ]; then
    sleep 900  # 15 minutes
elif [ $FAILURE_COUNT -le 6 ]; then
    sleep 3600  # 1 hour
fi

Service Inventory

Infrastructure (7 containers)

Media (16 containers)

Development (3 containers)

Telemetry (9 containers)

Automation & Scripts

The system includes 64 automation scripts organized by function:

The dark CLI provides unified access to all automation:

dark status         # Full system health check
dark mode work      # Switch to work mode
dark power          # Live power and thermal data
dark audit          # Run modular system audit
dark heal           # Trigger self-healing
dark backup         # Configuration backup
dark hdd mount      # Mount media HDD
dark update         # Update all containers

Shell aliases provide quick access to common operations:

Bot Systems

Telegram Bot (44 Commands)

The Telegram bot provides remote control through pure bash processing of long-polling updates. All commands are processed through dark-bot.sh with responses sent via the Telegram Bot API.

Discord Bot (49 Commands)

The Discord bot runs as a Docker container using Discord.js v14, providing richer interactive features including modals for config editing, trend charts via QuickChart API, and a live Pulse embed that updates every 30 seconds.

// Discord bot power command
const { execSync } = require('child_process');
const powerData = execSync('docker exec scaphandre wget -qO- http://localhost:8086/metrics')
    .toString()
    .split('\n')
    .filter(line => line.startsWith('scaphandre_power_socket_watts'));

const embed = new EmbedBuilder()
    .setTitle('⚡ Power Draw')
    .addFields(
        { name: 'CPU', value: `${cpuWatts}W`, inline: true },
        { name: 'GPU', value: `${gpuWatts}W`, inline: true },
        { name: 'Cost/hr', value: `₹${costPerHour}`, inline: true }
    );

Discord-exclusive features include:

• The Pulse: Auto-updating pinned embed with live metrics
• Trend Charts: 24-hour historical data via QuickChart
• Remote Studio: Interactive .env editor via Discord Modals

Self-Healing Agent

The dark heal command implements an 11-phase autonomous recovery system that respects the current operational mode:

The agent is mode-aware, meaning it only attempts to heal services whose tier matches the current operational mode. It will not try to start media services when in work mode, avoiding unnecessary resource consumption and error logs.

Security Hardening

Network Perimeter

The system implements a defense-in-depth approach with multiple security layers:

Host Hardening

Audit Schedule

The automated audit runs 20 modular checks via cron:

Deployment

Quick Start

# Clone and configure
git clone https://github.com/bhargav-pratim-sarma/Darknebula-Homelab.git
cd Darknebula-Homelab/homelab-installer

# Run installer
chmod +x install.sh
sudo ./install.sh

# Access services
dark status              # System health
dark power              # Live power data

Automated Installer

The install.sh script automates the complete setup in ~10 minutes:

1. Network configuration (static IP)
2. Docker engine installation
3. Modern CLI tools (Starship, Lazygit, Lazydocker)
4. Tailscale VPN with zero-touch auth
5. 40+ Docker containers deployment
6. Samba file shares configuration
7. Systemd services (power monitor, bot, boot notification)
8. ZSH with Oh-My-Zsh and plugins

Configuration

The installer supports non-interactive mode via install.conf:

# install.conf
INSTALL_ANTIGRAVITY=true
INSTALL_DESKTOP=false
INITIAL_MODE=full
TAILSCALE_AUTH_KEY=your_key_here
GITHUB_PAT=your_pat_here

Roadmap

• Phase 1 — Core Infrastructure (Complete) - Docker, Caddy, Portainer, basic monitoring
• Phase 2 — Service Catalog (Complete) - 40+ containers across all categories
• Phase 3 — Automation Scripts (Complete) - 64 scripts for all operations
• Phase 4 — Telegram Bot (Complete) - 44 commands for remote control
• Phase 5 — Discord Bot (Complete) - 49 slash commands with Pulse, Trends
• Phase 6 — Power Monitoring (Complete) - Intel RAPL via Scaphandre, INR costing
• Phase 7 — Self-Healing (Complete) - 11-phase autonomous recovery
• Phase 8 — Mode Switching (Complete) - Work/Media/Full resource optimization
• Phase 9 — IPv6 Support (In Progress) - Full IPv6 dual-stack
• Phase 10 — Kubernetes Migration (Planned) - K3s for container orchestration

Conclusion

DarkNebula represents a production-grade homelab infrastructure that achieves near-autonomous operation through comprehensive automation. The 40+ services are managed through a metadata-driven system that eliminates maintenance overhead while providing multiple control interfaces (CLI, Telegram, Discord) for remote administration.

The Intel RAPL power monitoring provides hardware-level accuracy at 15-second resolution, enabling precise electricity cost tracking in Grafana. The self-healing agent automatically detects and recovers from 11 different failure modes, with exponential backoff preventing restart loops on chronically failing services.

The mode-based resource management enables running the infrastructure on consumer hardware by selectively activating service tiers based on current needs. The work mode consumes ~800MB RAM while full mode enables the complete automation stack at ~5GB.

Security hardening through CrowdSec, Fail2ban, UFW, and Cloudflare Tunnel provides enterprise-grade protection while maintaining accessibility through the public domain. The centralized configuration through global.env with symlink propagation ensures consistency across all services without manual synchronization.