Over the past year I’ve been moving a lot of my personal infrastructure to connect via Tailscale. This has let me reduce the Internet-accessible attack surface on some of these hosts, in some cases removing all public presence. Many of these hosts serve internal websites (like FreshRSS), for which I want a TLS certificate. Previously I would use the ACME HTTP-01 challenge type, where you simply host a file on a web server to assert control over the domain. But now, with no public Internet endpoint, that is not an option. Tailscale can automatically provision certificates, but only for the Tailnet domain. However, I prefer to use the DNS names of my choosing (“vanity domains”), but also I consider the Tailnet names to be control-plane details rather than the data-plane interface.

An alternative ACME challenge type is DNS-01, in which you modify a DNS record to assert control over the domain. Many domain registrars and hosting providers provide API access for updating DNS records. But I found that most do not offer a sufficient level of access control: the API key is account-wide, rather than being able to create individual API keys scoped to specific DNS zones.

Luckily I came across the awesome acme-dns project. This is a limited-purpose DNS server for hosting the DNS-01 challenges and providing a small HTTP API to update the records. Both Lego and Traefik have native support for acme-dns, which made it very easy to switch from the HTTP-01 challenge.

Initial Setup

I deployed acme-dns in a container using this Podman Quadlet:

[Unit]
Description=acme-dns

[Container]
Image=registry.hub.docker.com/joohoi/acme-dns:v1.0
ContainerName=acme-dns
PublishPort=53:53
PublishPort=53:53/udp
PublishPort=127.0.0.1:9701:80
Volume=/srv/containers/acme-dns/config:/etc/acme-dns:ro
Volume=acme-dns-data.volume:/var/lib/acme-dns

[Install]
WantedBy=default.target

I published acme-dns’s HTTP API on port 9701 behind a nginx reverse proxy. This lets me restrict access to the /register API using an allow rule to only the IPs of my servers. I only IP-restricted the register action because the /update API is used by a machine on my home network, and I did not want to have to deal with my residential IP address changing. Additionally, I put the endpoints behind a random string subpath to keep the access log quiet:

  location /G1p91lMo66X2Vnw9RqiEg/register {
    allow A.B.C.D;
    allow 0.1.2.3;
    deny all;
    proxy_pass http://127.0.0.1:9701/register;
  }
  location /G1p91lMo66X2Vnw9RqiEg/update {
    proxy_pass http://127.0.0.1:9701/update;
  }
  location /G1p91lMo66X2Vnw9RqiEg/health {
    proxy_pass http://127.0.0.1:9701/health;
  }

The remainder of the setup was just updating DNS records and reconfiguring servers to use the DNS-01 challenge. In the below steps infra.net is a host for public-facing infrastructure, and vanity.host.net is the vanity domain I want to use to access a Tailnet resource.

These records are needed to establish acme-dns as the authoritative DNS server for the ACME challenges:

  1. DNS A record for dnstls.infra.net pointing to the server hosting acme-dns
  2. DNS NS record for dnstls.infra.net pointing to dnstls.infra.net

Because dnstls.infra.net is an authoratative nameserver for itself, it will also need to publish the records that are set up on infra.net. This can be configured by the general.records option of acme-dns’s config.cfg.

Vanity Domain Setup

To get the vanity domains working, each requires three additional DNS records:

  1. DNS A and AAAA records for vanity.host.net pointing to the Tailnet IPs
  2. DNS CNAME record for _acme-challenge.vanity.host.net pointing to <uuid>.dnstls.infra.net

The <uuid> is the registration entry in acme-dns for the specific domain being served. I’ve found the best way to get the UUID is to first use the LetsEncrypt Staging Environment as the ACME provider. Upon requesting a certiciate, the first verification attempt will fail, but it will print the UUID registered by acme-dns when it does. At that point, record (2) can be created, and another certificate request can be issued to the Staging Environment. Once it works successfully on Staging, point it to the production ACME server to request a real certificate.

Lego and other ACME tools may manually verify the DNS records prior to making the certificate request to the ACME server. The host’s local caching DNS resolver may break this verification, so manually overriding the Lego DNS resolvers to either 8.8.8.8:53 (Google) or 1.1.1.1:53 (Cloudflare) can remedy the issue.

A sample lego command:

export ACME_DNS_STORAGE_PATH=/var/acme/dnstls.json
export ACME_DNS_API_BASE=https://dnstls.infra.net/G1p91lMo66X2Vnw9RqiEg
lego --path /var/acme/certs --dns acme-dns --dns.resolvers 8.8.8.8:53 \
    --domains vanity.host.net --email webmaster@host.net \
    renew

I would have preferred to use a CNAME record to the Tailnet name rather than an A/AAAA record to the Tailnet IP, but not all OSes will recursively resolve CNAMEs through Tailscale MagicDNS (#7650).

How it Works

When the ACME tool needs a certificate, it submits a POST request to acme-dns to update a DNS TXT record hosted on <uuid>.dnstls.infra.net. The DNS NS record is critical because it delegates authority for resolving subdomains on dnstls.infra.net to that same host, i.e. the acme-dns server.

When the certificate authority attempts to validate the domain using DNS-01, it will look up the _acme-challenge.vanity.host.net CNAME record, which will return a uuid.dnstls.infra.net DNS name. That will be resolved by another DNS query that will first hit infra.net, which will respond with the record saying that domains under dnstls.infra.net are delegated to dnstls.infra.net. The resolver will then query for dnstls.infra.net to get the IP of the host, where it will resend the query for uuid.dnstls.infra.net, which will finally be handled by acme-dns.

---
config:
  fontFamily: "Source Serif Pro"
---
sequenceDiagram
    Lego->>acme-dns: Register/renew<br>"vanity.host.net"
    acme-dns->>acme-dns: Publish DNS TXT
    Lego->>ACME CA: Certificate request via DNS-01
    ACME CA->>infra.net registrar: query: "_acme-challenge.vanity.host.net"
    infra.net registrar->>ACME CA: reply: "CNAME <uuid>.dnstls.infra.net"
    ACME CA->>infra.net registrar: query: "<uuid>.dnstls.infra.net"
    infra.net registrar->>ACME CA: reply: "NS dnstls.infra.net"
    ACME CA->>infra.net registrar: query: "dnstls.infra.net"
    infra.net registrar->>ACME CA: reply: "A <ip@acme-dns>"
    ACME CA->>acme-dns: query: "<uuid>.dnstls.infra.net"
    acme-dns->>ACME CA: reply: "TXT <ACME verification>"
    ACME CA->>ACME CA: Verify records
    ACME CA->>Lego: Issue certificate

(In reality, some of these queries may be internally handled and simplified to reduce the total number of round trips).