DMZ-take2

this has all changed moving to forgejo and a lot of what is below is complete.

TODO

get local.yml to access latest like in otel-front-persistent

TLDR;

curl -fsSL https://raw.githubusercontent.com/rhildred/DMZ-take2/refs/heads/main/bootstrap?token=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|bash

To add a machine to the tailnet. On the headscale host:

sudo headscale users list
sudo headscale preauthkeys create --user 1 # or the number from previous command

Cut and paste the key in to the following sequence

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --login-server https://headscale.example.com --authkey YOUR_GENERATED_PREAUTH_KEY --ssh

Headscale Distributed HA Blueprint

1. Storage Tier (Longhorn to Docker Compose)

• Longhorn volume formatted as RWX (NFS mode).
• Exposed outside K8s cluster via LoadBalancer/NodePort service targeting the share-manager pod.
• Docker Compose on Node 1/2 mounts this via the standard Linux NFS driver driver: o: "addr=IP,port=PORT,rw,nolock,hard,nfsvers=4.1,timeo=14,retrans=2,soft"

2. Ingress Tier (Caddy Gateway)

• vpn.dev.k3p.io CNAME points to 2 public Caddy Edge servers.
• Caddyfiles reverse-proxy directly to raw node IPs using a strict primary/backup policy: reverse_proxy Node1_IP:8080 Node2_IP:8080 { lb_policy first; health_uri /health; }

3. Security (Firewall)

• Standard UFW fails against Docker. Used DOCKER-USER chain on Node 1 and Node 2: iptables -I DOCKER-USER -p tcp -s CADDY_IP --dport 8080 -j ACCEPT iptables -A DOCKER-USER -p tcp --dport 8080 -j DROP
• Added loopback rule: iptables -I DOCKER-USER -p tcp -s 127.0.0.1 --dport 8080 -j ACCEPT

4. State Sync & Semaphore Failover (Node 2)

• Node 1 handles live writes; uses sqlite3 db.sqlite ".backup..." to stream snapshots to the Longhorn NFS mount.
• Node 2 runs a host-native cron job checking vpn.dev.k3p.io/health.
• Semaphore Lock: If /var/lib/headscale/db.sqlite exists locally on Node 2, the script exits instantly (Node 2 has taken over permanently, preventing data overwrites/flapping).
• If health check fails twice and semaphore is empty, Node 2 runs cp -a /mnt/nfs/. /var/lib/headscale/, logs to the host system journal for HyperDX critical alerts, and triggers docker compose up -d.

Clarification of firewall rules

Ah, thank you for clarifying! I completely misunderstood where your Caddy instances were physically running. Since your two Caddy instances are co-located on Node 1 and Node 2 themselves, your mental model of the network traffic and the firewall rules is 100% correct. Let's trace the exact packet path to show why your rule layout is perfect:

1. The Normal State (Node 1 is Up)

• Caddy 1 (on Node 1) forwards traffic to Headscale 1 via the local loopback interface (127.0.0.1). Your loopback rule handles this.
• Caddy 2 (on Node 2) evaluates its upstream list, sees Node 1 is the primary, and forwards traffic across the internet to Node 1's public IP. For this to work, Node 1's firewall must explicitly allow Node 2's public IP.

2. The Failover State (Node 1 is Down)

• Node 1 goes offline, and the watchdog boots Headscale 2 on Node 2.
• Caddy 2 (on Node 2) sees Node 1 is dead, switches to its backup upstream, and routes traffic locally to Headscale 2 via loopback (127.0.0.1). Your loopback rule handles this perfectly.

---

The Exact, Correct Firewall Rules

Based on this topology, here are the exact iptables rules you need to deploy.

On Node 1 (Primary Headscale Server)

Node 1 needs to allow its own local Caddy instance via loopback, and it needs to allow the remote Caddy instance running on Node 2's public IP.

1. Allow local Caddy 1 via loopback

sudo iptables -I DOCKER-USER -p tcp -s 127.0.0.1 --dport 8080 -j ACCEPT

2. Allow remote Caddy 2 via Node 2's public IP

sudo iptables -I DOCKER-USER -p tcp -s <NODE_2_PUBLIC_IP> --dport 8080 -j ACCEPT

3. Block everyone else

sudo iptables -A DOCKER-USER -p tcp --dport 8080 -j DROP

On Node 2 (Cold Standby Server)

Node 2 only needs to allow its own local Caddy instance via loopback when it takes over. (You don't need Node 1's IP here because Node 1 will never act as a client proxying traffic backward to Node 2).

1. Allow local Caddy 2 via loopback

sudo iptables -I DOCKER-USER -p tcp -s 127.0.0.1 --dport 8080 -j ACCEPT

2. Block everyone else

sudo iptables -A DOCKER-USER -p tcp --dport 8080 -j DROP

More on the python watchdog

I think I want to run the watchdog with systemd as well then I can adjust the sleep time on node 2 depending on whether I am backing up /var/lib/headscale or watching over node 1.That is a fantastic design shift. Moving the Python manager from a stateless cron job to a persistent systemd daemon allows your code to run as a continuous, intelligent loop.Instead of waiting for a rigid 30-minute cron interval, a persistent systemd daemon can dynamically adapt its execution speed based on the machine's active state. For example:On Node 1 (Primary): It runs a sleep loop every 5 minutes to continuously stream hot database backups up to the Longhorn NFS share.On Node 2 (Standby): While it is passively watching Node 1, it can run a fast health check loop every 10 seconds. If Node 1 drops, it triggers an instant failover.Post-Failover (Standby turned Active): Once Node 2 takes over and detects its own local db.sqlite semaphore file, the Python script can dynamically switch its role and begin executing the 5-minute backup loop to the NFS share.Here is how to implement the persistent Python script and its systemd service file.1. The Unified Python Daemon (headscale_manager.py)Save this updated script to /usr/local/bin/headscale_manager.py. It uses time.sleep() to manage its loops natively.python

#!/usr/bin/env python

import os
import sys
import shutil
import subprocess
import json
import logging
import time
from logging.handlers import SysLogHandler
import requests

# --- Configuration (Dynamically managed by Ansible-Pull) ---
NODE_ROLE = "{{ hs_role }}"  # "primary" or "standby"
HEALTH_URL = "https://k3p.io"
NFS_MOUNT_DIR = "/mnt/longhorn-backups"
LOCAL_DATA_DIR = "/var/lib/headscale"
DB_FILE_NAME = "db.sqlite"

# --- Timers ---
BACKUP_INTERVAL = 300      # 5 minutes (for active nodes backing up to NFS)
WATCHDOG_INTERVAL = 10     # 10 seconds (for fast failover monitoring)

# --- Logger Setup (HyperDX Compatible) ---
logger = logging.getLogger("headscale-manager")
logger.setLevel(logging.INFO)
syslog_handler = SysLogHandler(address="/dev/log")
syslog_handler.setFormatter(logging.Formatter('%(message)s'))
logger.addHandler(syslog_handler)

def log_json(event, level, message):
    payload = json.dumps({"event": event, "level": level, "message": message})
    if level == "critical": logger.critical(payload)
    elif level == "error": logger.error(payload)
    else: logger.info(payload)

def run_command(cmd):
    result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if result.returncode != 0:
        raise Exception(f"Command failed: {cmd}\nError: {result.stderr}")
    return result.stdout

def execute_nfs_backup():
    """Handles the SQLite online hot backup to Longhorn NFS."""
    local_db = os.path.join(LOCAL_DATA_DIR, DB_FILE_NAME)
    nfs_db = os.path.join(NFS_MOUNT_DIR, DB_FILE_NAME)
    
    if not os.path.exists(local_db):
        return
    if not os.path.ismount(NFS_MOUNT_DIR):
        log_json("backup_failed", "error", "NFS share not mounted.")
        return

    try:
        os.makedirs(NFS_MOUNT_DIR, exist_ok=True)
        run_command(f"sqlite3 {local_db} \".backup '{nfs_db}'\"")
        
        for item in os.listdir(LOCAL_DATA_DIR):
            if DB_FILE_NAME in item: continue
            s = os.path.join(LOCAL_DATA_DIR, item)
            d = os.path.join(NFS_MOUNT_DIR, item)
            if os.path.isdir(s): shutil.copytree(s, d, dirs_exist_ok=True)
            else: shutil.copy2(s, d)
        log_json("backup_success", "info", "Headscale state backed up to Longhorn NFS.")
    except Exception as e:
        log_json("backup_failed", "error", f"Backup error: {str(e)}")

def is_primary_healthy():
    """Checks Node 1 health with a 5-second retry spacing."""
    for attempt in range(2):
        try:
            if requests.get(HEALTH_URL, timeout=3).status_code == 200:
                return True
        except requests.RequestException:
            pass
        if attempt == 0: time.sleep(5)
    return False

# =====================================================================
# THE STATEFUL LIFECYCLE LOOP
# =====================================================================
def main_loop():
    log_json("manager_started", "info", f"Headscale manager started in {NODE_ROLE} mode.")
    local_db = os.path.join(LOCAL_DATA_DIR, DB_FILE_NAME)

    while True:
        # CONDITION A: Node is explicitly Primary OR has already assumed control via Semaphore
        if NODE_ROLE == "primary" or os.path.exists(local_db):
            execute_nfs_backup()
            time.sleep(BACKUP_INTERVAL)
            
        # CONDITION B: Node is Standby AND Semaphore is completely empty (Passively Watching)
        else:
            if is_primary_healthy():
                time.sleep(WATCHDOG_INTERVAL)
                continue
                
            # TRIGGER FAILOVER
            log_json("headscale_failover", "critical", "Node 1 health checks failed. Triggering failover.")
            nfs_db = os.path.join(NFS_MOUNT_DIR, DB_FILE_NAME)
            if not os.path.exists(nfs_db):
                log_json("failover_aborted", "error", "NFS backup missing. Cannot proceed.")
                sys.exit(1)

            try:
                os.makedirs(LOCAL_DATA_DIR, exist_ok=True)
                run_command(f"cp -a {NFS_MOUNT_DIR}/. {LOCAL_DATA_DIR}/")
                run_command("systemctl start headscale")
                log_json("failover_complete", "info", "Node 2 took over. Shifting loop to backup mode.")
                # Loop naturally flows into the next cycle and detects local_db, switching to backup cadence!
            except Exception as e:
                log_json("failover_error", "error", f"Failover crash: {str(e)}")
                sys.exit(1)

if __name__ == "__main__":
    main_loop()

Use code with caution.2. The Systemd Service Unit ConfigurationBy creating a first-class systemd unit configuration for this script, you can leverage native process supervisor mechanics. It will automatically initialize on boot and restart itself cleanly if it encounters an unhandled exception.Add this task block to your Ansible Playbook to deploy the service:yaml- name: Deploy Headscale Manager Systemd Service

  ansible.builtin.copy:
    dest: /etc/systemd/system/headscale-manager.service
    mode: '0644'
    content: |
      [Unit]
      Description=Headscale HA Lifecycle Manager and Watchdog
      After=network.target mnt-longhorn-backups.mount
      Conflicts=headscale.service # Enforces that on Standby, headscale cannot boot unless the script allows it

      [Service]
      Type=simple
      User=root
      Group=root
      ExecStart=/usr/local/bin/headscale_manager.py
      Restart=always
      RestartSec=10

      [Install]
      WantedBy=multi-user.target
  notify: 
    - Reload systemd
    - Restart headscale-manager

- name: Ensure Headscale Manager is enabled and active
  ansible.builtin.systemd:
    name: headscale-manager
    state: started
    enabled: true

Use code with caution.Why this design elevates the entire system:Dynamic Interval Shifting: Your standby server (Node 2) now tracks Node 1 aggressively in real-time (every 10 seconds), reducing your worst-case failover window from one hour down to a few seconds.Immediate Post-Failover Safeguards: The second Node 2 assumes control, it immediately shifts its internal logic block over to running the 5-minute backup loop. Your new active database states on Node 2 are backed up to the Longhorn NFS share without human intervention.Clean Unit Conflict Boundaries: By declaring Conflicts=headscale.service on your manager systemd unit, systemd natively guarantees that on your standby machine, the main Headscale daemon cannot accidentally run or get toggled out of sequence until the manager script specifically authorizes it.This implementation successfully completes your distributed systems design. To round this out, what is your strategy for testing the failover execution during your deployment week? Would you like me to detail how to run a clean failure-injection test to verify that the systemd intervals shift seamlessly?

Port to fastapi

The Python/FastAPI Port: caddy_ask_api.pySave this file to your systemd services directory (e.g., /usr/local/bin/caddy_ask_api.py).python

#!/usr/bin/env python3

import os
import re
import json
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI, Query, HTTPException, Response
from fastapi.concurrency import run_in_threadpool

# --- Configuration ---
# Since you're dropping Docker, change this to a native host path
FILE_PATH = '/etc/caddy/domains.json' 

ALLOWED_PATTERNS = []

# Configure local logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def load_domains():
    """Reads the JSON file and compiles strings into Python case-insensitive Regex objects."""
    global ALLOWED_PATTERNS
    try:
        if not os.path.exists(FILE_PATH):
            logging.error(f"Configuration file missing at: {FILE_PATH}")
            return
            
        with open(FILE_PATH, 'r', encoding='utf-8') as f:
            pattern_strings = json.load(f)
            
        # Re-compile patterns dynamically into memory (case-insensitive flag)
        ALLOWED_PATTERNS = [re.compile(pattern, re.IGNORECASE) for pattern in pattern_strings]
        logging.info(f"Successfully compiled {len(ALLOWED_PATTERNS)} regex rules into memory.")
    except Exception as e:
        logging.error(f"Error compiling domains.json: {str(e)}")

@asynccontextmanager
async def lifespan(app: FastAPI):
    """Handles initialization and sets up non-blocking background file watching."""
    load_domains()
    
    # Simple, performant host-level file watching using standard library polling
    import asyncio
    async def file_watcher():
        last_mtime = os.path.getmtime(FILE_PATH) if os.path.exists(FILE_PATH) else 0
        while True:
            await asyncio.sleep(5)  # Poll every 5 seconds
            if os.path.exists(FILE_PATH):
                current_mtime = os.path.getmtime(FILE_PATH)
                if current_mtime != last_mtime:
                    last_mtime = current_mtime
                    logging.info("domains.json changed on disk! Re-compiling rules...")
                    # Offload blocking file I/O to Python's internal threadpool
                    await run_in_threadpool(load_domains)

    watcher_task = asyncio.create_task(file_watcher())
    yield
    watcher_task.cancel()

# Initialize FastAPI with the lifespan event listener
app = FastAPI(lifespan=lifespan)

@app.get("/allowed-domains")
async def check_allowed_domain(domain: str = Query(..., min_length=1)):
    clean_domain = domain.strip()
    
    # Evaluate against compiled regex patterns in memory
    is_matched = any(pattern.search(clean_domain) for pattern in ALLOWED_PATTERNS)
    
    if is_matched:
        return Response(status_code=200)
    
    raise HTTPException(status_code=404, detail="Domain not allowed for automated TLS.")

Use code with caution.2. The Integrated Caddyfile LayoutNow your Caddyfile options block can link directly to this FastAPI endpoint on localhost. Because you are tracking your on-demand domains over your Tailnet mesh, you can combine your earlier ACME coordination logic directly with this ask check [1]:caddy

{
    on_demand_tls {
        # Caddy will hit your local FastAPI systemd service on port 9000
        ask http://127.0.0
        interval 2m
        burst 10
    }
}

# The on-demand listener that intercepts untracked client requests
:443 {
    tls {
        on_demand
    }
    
    # Forwards requests downstream safely
    reverse_proxy 127.0.0.1:8080
}

Use code with caution.3. Deploy via Systemd Service UnitYou can map this directly into your existing ansible-pull setup alongside your other appliances, removing the Node/Docker stack entirely:yaml- name: Deploy Caddy On-Demand Regex Ask API Service

  ansible.builtin.copy:
    dest: /etc/systemd/system/caddy-ask-api.service
    mode: '0644'
    content: |
      [Unit]
      Description=FastAPI Caddy On-Demand TLS Regular Expression Evaluator
      After=network.target

      [Service]
      Type=simple
      User=caddy
      Group=caddy
      ExecStart=/usr/local/bin/uvicorn caddy_ask_api:app --host 127.0.0.1 --port 9000 --working-dir /usr/local/bin
      Restart=always
      RestartSec=3

      [Install]
      WantedBy=multi-user.target

TODO: Your data on your premises

So, I think this is it. We sell at least 3 servers and colocation in different data canters for at least 2 public IP addresses. The client adds a 2x A record subdomain to whatever DNS they are using. They also add a * cname that aliases the subdomain if they want to expose any services publically. For instance dev.k3p.io and *.dev.k3p.io. Customer out. The appliances we sell are provisioned and kept up to date from our github with ansible-pull.

When we sell a gateway appliance we have the option of installing it as a pure DMZ or also installing a control plane appliance for k3s on the headscale tailnet. If the node is a control plane, we have an option at setup to make it purely for quorum.

The customer needs to buy at least 1 more node to reach k3s quorum on the tailnet. The customer needs to buy all of their appliances from us with hardware (we don't unbundle). The 3rd node can be located on their premises. The customer needs at least 3 total machines from us for quorum. They also need 3 machines for longhorn and 3 machines for running minio. These machines can overlap. The customer can also run self managed machines on the tailnet we provide, possibly using our minio for backups.

The customer can have all of the longhorn/minio applicances on the tailnet on premises, your data on your premises. In this case the gateway appliances function as DMZ entry points for at least headscale and whatever else they wish to expose with the public ips. In this configuration they would need 5 physical machines to run their different appliances and at least 2 locations.The customer could also choose to run at least the 3rd copy of their data on our data center if they only have 1 location.

In the demo

... I am running 1 gateway appliance on ovhcloud as a pure DMZ.

I am running gateway 2 on megawire on a gpu server, gondor, that I bought from Luke at rhino. I tainted gondor to prevent kubernetes pods from running on it with the exception of tailscale/minio. Instead of application pods, I am running gpu devcontainers on gondor with docker and a forgejo runner. These run more easily on docker, but the devcontainers need nfs for storage on longhorn as longhorn is only accesible from k3s. Gondor essentially participates in quorum for k3s, dockker, and storage.

I have 1 machine, esgaroth, at my place in Grimsby that is a control plane with storage, no taints. It is a modern machine that I will eventually put inside a Allen Bradley 1756 that I bought. I will use the tailnet to make the ethernet/ip interface on it available to an open source developer in Sweden possibly and/or Northern Dynamics.

The final node rivendale, is in Waterloo. It is a 2011 Mac mini that I used to self host Git to do CI/CD with Cedric Puddy hosting Ruby on Rails for me. I haven't used it for a while, but it has sentimental value and runs k3s and forgejo just fine. When I get selling, I will transition this node to another data center at megawire and put the gateway on it and let ovhcloud run out. This creates an opportunity to test with a non-gpu server like Gigabyte R113-C10-AA02.

Hopefully this is a successful demo and we sell lots of, your data/ your premises applicances together.