machine-config/bert
Chandler Swift 3c8150a5aa
Fix stage2 IPv6 networking
When `ip=dhcp` is passed as a kernel param (which I assume, but don't
know for sure, just gets passed to systemd and probably then to
systemd-networkd or NetworkManager or whatever does the networking in
the initrd?), IPv6 doesn't come up. With the param, bad IPv6:

```text
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 3c:52:82:00:46:d6 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 192.168.1.20/24 brd 192.168.1.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 2601:441:8301:8af7:94e1:4d4c:73c2:ac31/64 scope global temporary dynamic
       valid_lft 322763sec preferred_lft 86330sec
    inet6 fd0a:8f9f:6f83:0:3e52:82ff:fe00:46d6/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft forever preferred_lft forever
    inet6 2601:441:8301:8af7:3e52:82ff:fe00:46d6/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft 322763sec preferred_lft 322763sec
    inet6 fe80::3e52:82ff:fe00:46d6/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
```

whereas without the param I get good (whatever that means) addresses:

```text
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 3c:52:82:00:46:d6 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 192.168.1.20/24 brd 192.168.1.255 scope global dynamic noprefixroute eno1
       valid_lft 43188sec preferred_lft 43188sec
    inet6 fd0a:8f9f:6f83::33b/128 scope global dynamic noprefixroute
       valid_lft 43190sec preferred_lft 43190sec
    inet6 2601:441:8301:8af7::33b/128 scope global dynamic noprefixroute
       valid_lft 43190sec preferred_lft 43190sec
    inet6 2601:441:8301:8af7:b572:fc6c:ba27:dcdb/64 scope global temporary dynamic
       valid_lft 322965sec preferred_lft 86235sec
    inet6 2601:441:8301:8af7:f3f0:df23:a6bb:7d25/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 322965sec preferred_lft 322965sec
    inet6 fd0a:8f9f:6f83:0:485e:f55d:fa1d:cef4/64 scope global temporary dynamic
       valid_lft 604790sec preferred_lft 86235sec
    inet6 fd0a:8f9f:6f83:0:7851:1417:807:f017/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::eb0c:1983:24a5:3e42/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
```

I'm honestly not entirely sure what this _should_ look like, but what I
_do_ know is that in the "bad" configuration, SSH takes an extra ~10s to
connect as it has to wait for two ~5s `No route to host` timeouts:

```console
[chandler@oscar:~]$ ssh -v root@bert
OpenSSH_9.8p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/chandler/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: Connecting to bert [fd0a:8f9f:6f83::33b] port 22.
debug1: connect to address fd0a:8f9f:6f83::33b port 22: No route to host
debug1: Connecting to bert [2601:441:8301:8af7::33b] port 22.
debug1: connect to address 2601:441:8301:8af7::33b port 22: No route to host
debug1: Connecting to bert [192.168.1.20] port 22.
debug1: Connection established.
```

Future work: What _should_ my IPv6 config look like? Is there any way I
can keep network config stable between the initrd and the booted system?
Why is this recommended against?

> The default is false when systemd is enabled in initrd, because the
> systemd-networkd documentation suggests it.
2024-11-02 00:26:21 -05:00
..
services Add Factorio Server 2024-10-31 02:04:15 -05:00
configuration.nix Fix stage2 IPv6 networking 2024-11-02 00:26:21 -05:00
hardware-configuration.nix Start setting up bert 2024-10-31 01:06:38 -05:00
Makefile Start setting up bert 2024-10-31 01:06:38 -05:00
README.md Start setting up bert 2024-10-31 01:06:38 -05:00

Installation

  1. Install NixOS minimal
  2. ssh-keygen -N "" -f /etc/secrets/initrd/ssh_host_ed25519_key
  3. Enable SSH server and add root SSH key
  4. Deploy updated config with make
  5. Set up Remote Disk Unlocking
    1. mkdir -p /etc/secrets/initrd && ssh-keygen -N "" -f /etc/secrets/initrd/ssh_host_25519_key
  6. Deploy content to web services

Notes on Caddy

Until 2.8 is released with 24.11, Caddy has a pretty limited sense of what content-types should be compressed:

https://github.com/caddyserver/caddy/blob/v2.7.6/modules/caddyhttp/encode/encode.go#L85-L101

Specifically, this doesn't include GeoJSON, which is a bit of a shame for maps.chandlerswift.com. That said, I'll probably be upgrading to 24.11 as soon as it comes out, so in the intervening time I'm just not going to worry about it.

The list was expanded in this PR:

https://github.com/caddyserver/caddy/pull/6081