Add reusable Debian and Ubuntu hardening playbooks

docs/hardening.md

@@ -0,0 +1,70 @@
+# Host hardening
+
+The playbooks target Debian or Ubuntu hosts using systemd. They create a separate admin account with password-required sudo and SSH public key access, configure Fail2ban and UFW, enable unattended security updates, disable SSH root and password login, and move SSH to a configurable port. The default replacement port is `2222`; changing it reduces scan noise but is not a security boundary.
+
+## Configure inventory
+
+Copy the tracked inventory examples as shown in [README.md](../README.md). Add each server under the `debian_servers` group in `inventory/hosts.yml`. Set `ansible_user` to an existing account with sudo access and `ansible_port` to its current SSH port. Put the admin public key in `inventory/host_vars/<alias>.yml`. Set `admin_user`, `ssh_port`, and `ssh_legacy_port` in the shared settings or override them per host. Every active inventory `.yml` file is ignored by Git.
+
+Create a SHA-512 crypt password hash with `openssl passwd -6`, which prompts for the password. Copy `inventory/group_vars/all/vault.yml.example` to `inventory/group_vars/all/vault.yml`, insert the hash, and encrypt the file:
+
+```sh
+ansible-vault encrypt inventory/group_vars/all/vault.yml
+```
+
+Keep the Vault password outside the repository. The password hash is encrypted in Vault and its Ansible task output is suppressed.
+
+## Configure ingress
+
+Add only required service ports to `ufw_allowed_rules` in a host's inventory file:
+
+```yaml
+ufw_allowed_rules:
+  - port: "80"
+    proto: tcp
+  - port: "443"
+    proto: tcp
+  - port: "51820"
+    proto: udp
+```
+
+SSH ports are handled by `ssh_port`, `ssh_legacy_port`, and `ssh_keep_legacy_port`. A rule may set `from_ip` to restrict its source. To remove an existing rule, move its port, protocol, and source into `ufw_remove_rules`, run the playbook once, then remove that cleanup entry. UFW defaults to denying inbound and routed traffic and allowing outbound traffic.
+
+Docker-published ports can bypass UFW's normal host INPUT filtering. Publish only intended host ports and keep application and database ports on Docker networks or bind them locally.
+
+## Roll out one host
+
+Replace `server1` with the host alias from `inventory/hosts.yml` in these commands. Keep a working SSH session open during the port change.
+
+1. Inspect the host and its current listeners:
+
+   ```sh
+   ansible-playbook playbooks/discover.yml -l server1
+   ```
+
+2. Bootstrap the admin account through the existing SSH account:
+
+   ```sh
+   ansible-playbook playbooks/bootstrap-admin.yml -l server1 --ask-become-pass --ask-vault-pass
+   ```
+
+   Omit `--ask-become-pass` if the existing account already has passwordless sudo. Verify a fresh SSH login as `admin_user` on the old port and check that `sudo` asks for that account's password.
+
+3. Change `ansible_user` in the host inventory to `admin_user`. Keep `ansible_port` on the old port, and leave `ssh_keep_legacy_port: true`. Apply hardening:
+
+   ```sh
+   ansible-playbook playbooks/harden.yml -l server1 -K --ask-vault-pass
+   ```
+
+   UFW allows both SSH ports before it is enabled. SSH then listens on both ports. On hosts using `ssh.socket`, the playbook updates the systemd socket listeners as well.
+
+4. Verify a fresh SSH login as the admin on `ssh_port`, plus UFW and Fail2ban status. Change `ansible_port` to `ssh_port`, set `ssh_keep_legacy_port: false`, and rerun the hardening command. The old SSH listener and its UFW rule are then removed.
+
+5. Add or retire future service ports through that host's `ufw_allowed_rules` and `ufw_remove_rules`, then rerun the hardening playbook with `-l <alias>`.
+
+## Ongoing work
+
+- Review native listeners before closing a service port.
+- Review container port publishing separately from UFW rules.
+- Keep secrets in Vault and runtime secret files, and verify a backup restore.
+- Review container updates after DIUN alerts; they are not applied automatically.

docs/plan.md

@@ -0,0 +1,47 @@
+# Pi-first rollout plan
+
+The Pi is the first inventory target for validation. The Ansible playbooks target the generic `debian_servers` group and can be used for any Debian or Ubuntu host with per-host inventory values.
+
+## Agreed design
+
+- Ansible is the host configuration and deployment controller.
+- Docker Compose defines each service in its own `stacks/<service>/` directory.
+- The local Git checkout bootstraps Gitea; after Gitea is available, it is added as a remote.
+- Traefik is the only public HTTP(S) edge. It uses file-provider route fragments supplied alongside each service, rather than Docker labels requiring access to the Docker API.
+- All proxied services join the externally created `traefik` network. Databases and internal workers remain on per-stack private networks.
+- DIUN alerts about registry image changes. It uses its file provider and per-service image watch files, so it needs no Docker socket. Alerts prompt a reviewed update; they do not auto-update services.
+- restic backs up important persistent data to a separate destination with storage-aware retention and a tested restore procedure.
+- Portainer is out of scope.
+
+## Pi facts collected
+
+- Raspberry Pi 4 Model B Rev 1.1, 4 GB RAM, 64-bit ARM.
+- Debian GNU/Linux 13 (Trixie), kernel 6.12.47+rpt-rpi-v8.
+- Docker Engine 28.5.1 and Compose v2.40.0 are installed.
+- Root filesystem: 115 GB, about 101 GB available at discovery.
+- SSH is listening on all IPv4 interfaces at port 22. Root key login and password login are currently permitted. `ufw` is not installed and Fail2ban is inactive. The provided Windows public key was not in `/home/pi/.ssh/authorized_keys`; the `pi` password login still works.
+- Containers `egress-proxy` and `uptime-kuma` were stopped at the owner's request. Their files and data were left untouched.
+- Native Nginx (80/tcp) and sing-box (2080/tcp, 2081/tcp) are still running. The Pi inventory temporarily includes these ports in the UFW allow-list so the firewall rollout does not silently cut them off. Remove them from inventory when the services are retired or migrated.
+
+## Steps
+
+1. Resolve the Windows-to-Pi SSH route conflict: Cisco AnyConnect installs a route for the Pi's LAN subnet through its VPN interface. With Cisco disconnected, Windows SSH works on the direct Ethernet route. Separately, the installed Windows Ansible launcher still fails at startup (`WinError 87`); Ansible documents Windows as unsupported as a control node. WSL is excluded, so the playbooks are authored here but cannot be executed from this Windows Ansible install until a supported non-WSL controller is available.
+2. Create the `ops` admin account with the provided key and a Vault-protected password hash; verify key login and password-required sudo while the existing `pi` session/port remains available.
+3. Apply the baseline, Fail2ban, UFW allow-list, root-login denial, and key-only SSH on both port 22 and the configured replacement port.
+4. Verify a fresh connection to the replacement SSH port. Only after that succeeds, update `ansible_port`, set `ssh_keep_legacy_port: false`, and redeploy to remove port 22 from sshd and UFW.
+5. Add future service ingress by editing that host's `ufw_allowed_rules` list. Retire a port through `ufw_remove_rules` for one run. Keep Docker host-port behavior separate from UFW INPUT rules.
+6. Create the external Traefik network and deploy Traefik, DIUN, and Uptime Kuma from their separate Compose directories. Initially use test ports that avoid the Pi's native Nginx port 80. Decide whether to reuse or replace existing Uptime Kuma data before mounting it.
+7. Configure DIUN image notifications (notification channel/credentials still need choosing). Notifications prompt reviewed image updates; DIUN does not update containers automatically.
+8. Verify routing, notification delivery, reboot persistence, and a storage-aware restic backup/restore on the Pi.
+9. Deploy Gitea and push the local repository to it. Add 3x-ui, password manager, Logseq sync, portfolio, and pet projects individually, each in its own stack directory with an optional Traefik route fragment and DIUN watch file.
+10. Once the Pi runbook is stable, add the Ubuntu VPS and planned nearby VPS as inventory hosts, with per-host variables as needed.
+
+## Safety gates
+
+- No automated reboot during the first pilot.
+- No deletion or migration of existing application data before a backup and an explicit service cutover plan.
+- Bootstrap and verify the new admin account before disabling SSH password authentication.
+- Keep the old SSH port listening and allowed through the first hardening run. Change inventory to the new port only after a second SSH login succeeds; remove the old port in a later run.
+- Keep an existing SSH session open throughout the port migration.
+- Review UFW allow-list changes against required native and Docker-published services before applying.
+- Container updates remain manual and reviewable after DIUN alerts.