Cool minor @Tailscale moment: I'm recommissioning a server that got moved from a different network, so all its network config was wrong, and generally I couldn't get at it over the network, only IPKVM console. But then my `ping` over Tailscale started working?!
Like, this machine has no working DNS, `ping 8.8.8.8` from the local console doesn't work, but my desktop and monitoring server both went "oh hey, you're back" and were able to reach it? Including over the internet?!
Pause here if you want to figure out what happened for yourself :)
Turns out, IPv6 autoconfiguration is what happened. Sure, v4 configuration was entirely wrong (it was trying to connect to wifi, via a wifi dongle that was no longer installed, and wanted to talk to a DNS server that doesn't exist any more), but eno1 had a cable plugged in!
The server noticed IPv6 router advertisements, went "I'll have some of that", and got global IPv6 connectivity automagically. IPv4 and DNS were still down though, so all it had at this point is the ability to send/receive IPv6 packets.
So, how did Tailscale get from there to a working setup? It still needs to contact https://t.co/hEs4S8qvTw to get a network map, and still needs to talk to DERP servers to get p2p tunnels working outside the LAN. Enter bootstrap DNS!
controlplane.tailscale.com
The Tailscale daemon has the IPv4 and IPv6 addresses for all our DERP servers burned into the binary. The list is frequently out of date and drifts over time... But with very high likelihood, one of those IPs will still be a DERP server.
So, when DNS lookups for
controlplane.tailscale.com
If it finds one, it can send it a query for /bootstrap-dns?q=
controlplane.tailscale.com
Our DERP servers are only willing to resolve a very small number of hostnames critical to Tailscale's own bootstrapping, all in *.tailscale.com. But for those few hostnames, the Tailscale binary has a list of, currently, 34 IPs at which it *might* be able to get DNS help.
So, what happened to my server is that the Tailscale daemon, which was running but unhappy (couldn't find
controlplane.tailscale.com
This took a bit of time, since the IPv4 internet was unreachable, and the tailscale version was a bit old so the bootstrap list had drifted a bit... But after a minute or two, it found a working IPv6 address for one of our DERPs!
From there, it got a DNS resolution for
controlplane.tailscale.com
Its connectivity was still pretty poor... but along with a netmap, it got a fresh DERP map from the control plane, registered itself at its nearest DERP, and now it's fully reachable, either p2p over IPv6, or relayed via DERP for v4-only peers.
DERP being a protocol that routes based on Curve25519 public keys, rather than IP addresses, DERP servers naturally function as a v4/v6 translation gateway. My server can only talk IPv6 to DERP, some of my other machines only have IPv4 internet connectivity... But that's fine!
Both can talk to DERP, because DERP servers are dual-stacked, and because the packet routing is based on crypto keys rather than IPs, it's totally fine and normal for a packet to bounce through DERP from an IPV4-only client to an IPv6-only client.
Anyway, that's the story of how I was able to ssh into my server over Tailscale, to fix its network configuration. I'm writing this down partially because I was *extremely* confused when this magic occurred, and it took me a while to work out what had happened.
But that's the sort of lengths we at Tailscale go to in order to make connectivity work. If there's *any* way for us to get your machines talking, and we fail to find it, that's a bug :)
(Addendum for security-minded people: we of course verify TLS certs at every step of this bootstrap process, so even if our bootstrap list drifted and we end up talking to some rando other server, we only trust bootstrap resolutions from authentic DERP servers)
Further reading: my magic moment was an instance of the general idea expressed in
apenwarr.ca/log/20200708
