On this page

• Root Cause: Docker’s Default Port Binding
• Why This Went Undetected
• What We’d Do Differently Next Time
  • Development practices
  • Infrastructure
  • Security
• The One Thing That Almost Broke Everything
• Container Restart Surprises

Lessons Learned

Root Cause: Docker’s Default Port Binding

The single biggest issue was that Docker, by default, publishes ports on 0.0.0.0 when no host IP is specified:

ports:
  - "6379:6379"     # Binds to 0.0.0.0 — publicly accessible!

vs.

ports:
  - "127.0.0.1:6379:6379"  # Only accessible from localhost

Lesson: Always use 127.0.0.1: prefix for internal services. The one container that had it correct (orcta-valkey-pss) was the template to follow — we just didn’t follow it everywhere.

Why This Went Undetected

1. No regular port audit — ss -tlnp had never been run to check what was listening.
2. Assumed safety in numbers — “It’s a VPS behind a firewall” (but Contabo doesn’t have a cloud firewall by default).
3. Caddy configuration grew organically — each new subdomain was added manually, security headers were copy-pasted inconsistently, and no one noticed the sites without headers.
4. SSH socket activation was invisible — changing ListenAddress in sshd_config had no effect because systemd’s ssh.socket was managing the actual bind. This wasted time debugging “why is SSH still on 0.0.0.0?”

What We’d Do Differently Next Time

Development practices

• Use a compose linter or template that forces 127.0.0.1: on all non-public ports.
• Add a pre-deploy check script that runs ss -tlnp and alerts on unexpected 0.0.0.0 listeners.
• Use Caddy snippets from day one — the (security) snippet eliminated 50+ lines of duplication and made it easy to add headers to new sites.

Infrastructure

• Tailscale first — the VPS should have been on the org tailnet from the start, with SSH bound to the Tailscale IP only. This would have made the port exposure less critical (though still not OK).
• Infrastructure documentation — an up-to-date inventory of services, ports, and compose files would have made the audit much faster.
• iptables baseline — a DROP rule for inbound ports that aren’t explicitly allowed should be part of the initial provisioning.

Security

The One Thing That Almost Broke Everything

When we locked down SSH, the systemd socket activation meant that editing sshd_config.d/*.conf had zero effect on what address the SSH daemon listened on. The socket was managed by ssh.socket, not by sshd itself. This is a common Ubuntu 24.04 default that’s easy to miss.

Fix: Always check systemctl status ssh.socket before debugging SSH binding issues. And use either:

• Socket drop-in: /etc/systemd/system/ssh.socket.d/override.conf
• Or disable the socket: systemctl disable --now ssh.socket (then sshd manages its own socket)

Container Restart Surprises

When we changed compose files, docker compose up -d tried to recreate dependency containers (like Valkey in ORCTA-BACKEND-PSS) even though only the app service’s port binding changed. This caused a name conflict error. The fix was docker compose up -d --no-deps <service>.