PowerDNS High Availability with Patroni: How We Built Automatic PostgreSQL Failover

Running several authoritative nameservers solved the data-plane side of HYE DNS: a query could reach another location if one server or route failed. It did not solve the control plane. Our PowerDNS Generic PostgreSQL backend still needed one safe writer, and a fixed writer turns record changes, DNSSEC operations and zone metadata into a single point of failure.

We rebuilt that path around PostgreSQL 17, Patroni and etcd. Every location now reads DNS data from its own PostgreSQL instance. Patroni controls database leadership and physical streaming replication; etcd provides the distributed lock and consensus state; the HYEHOST Panel discovers the current writer before every mutation. The important outcome is separation: an election may pause changes, but it does not force ordinary authoritative answers across a WAN.

This is the design we operate, with addresses and credentials replaced by documentation examples. It is a pattern to adapt and test, not a drop-in configuration.

Why PowerDNS Database High Availability Matters

The PowerDNS Generic PostgreSQL backend contains more than exported resource records. It stores zones, records, domain metadata, comments, TSIG material and DNSSEC keys. Copying a zone file therefore does not reproduce the complete operational state of an API-managed PowerDNS service.

Three mechanisms solve three different problems:

• AXFR and IXFR distribute published zone contents between DNS primaries and secondaries. They remain valuable protocol-level tools.
• PostgreSQL physical replication keeps the whole PowerDNS backend consistent, including metadata the DNS transfer format does not carry.
• Independent backups provide historical recovery after deletion, corruption, a bad migration or operator error.

A PowerDNS HA cluster needs all of the relevant layers. Four healthy pdns_server processes can still share one fragile database writer. Likewise, a healthy database cluster says nothing about UDP/53, TCP/53, IPv6 routing, DNSSEC validation or anycast reachability.

The Four-Region HYEHOST Architecture

HYE DNS runs an authoritative PowerDNS and PostgreSQL instance in each location. London is the preferred leader, Wolverhampton is the nearby synchronous standby, and Ashburn and Auckland normally remain asynchronous. London, Wolverhampton and Ashburn vote in etcd. Auckland holds a database copy but does not add intercontinental latency to the consensus quorum.

                         HYEHOST Panel
                   discovers exactly one writer
                              |
                private Patroni /primary checks
                              |
        +---------------------+---------------------+
        |                     |                     |
 London: leader  ====  Wolverhampton: sync     Ashburn: async
 PDNS -> local PG       PDNS -> local PG        PDNS -> local PG
 etcd voter             etcd voter              etcd voter
        |                                               |
        + - - physical streaming replication - - - - - +
                              |
                       Auckland: async
                       PDNS -> local PG
                       etcd non-voter

Each PowerDNS process connects to the PostgreSQL server on the same node. A normal authoritative query never waits for London, the Panel or the current leader. This keeps PostgreSQL DNS failover in the control plane: hot standbys continue serving read-only backend queries while write-oriented PowerDNS duties stay disabled.

Design Decisions That Matter

Use real failure domains

Several virtual machines on one hypervisor do not provide the resilience implied by a multi-node diagram. We placed nodes across separate sites, networks and geographic regions. The same thinking applies to power, upstream connectivity and management access.

Keep the consensus group small and odd

etcd normally wants three or five voters. Three voters tolerate one unavailable member; five tolerate two, at a higher coordination cost. A fourth voter does not increase tolerated failures and can make quorum harder to maintain. Auckland is useful as a PostgreSQL replica, but making it vote would put every consensus operation across a much longer path.

Patroni synchronous replication: choose durability deliberately

Our deployed Patroni policy uses synchronous_mode: true, synchronous_mode_strict: true and synchronous_node_count: 1. PostgreSQL uses synchronous_commit: remote_apply. A write is acknowledged only after the selected synchronous standby has replayed it.

This favours durability over control-plane write availability. If no eligible synchronous standby is available, record changes can intentionally stop. Existing authoritative answers continue from local copies. That is preferable to acknowledging a change we cannot meet our durability policy for—but it is still downtime for mutations, and the extra commit latency must be measured.

Keep management traffic private

PostgreSQL, etcd peer/client ports, the Patroni REST API and PowerDNS HTTP API are management surfaces. We carry them on private networking with explicit firewall rules, TLS where supported and narrowly scoped identities. Our private networking guide covers the underlying isolation pattern; no production addresses, keys or certificates appear below.

Building the etcd Quorum for Patroni

In a Patroni etcd setup, etcd is the distributed configuration store used for leadership state. Its correctness depends on stable member identity, authenticated peer communication and disciplined membership changes. A sanitised three-member bootstrap might use documentation addresses like these:

# London example only — use private addresses and real certificates
name: pdns-lon
initial-advertise-peer-urls: https://192.0.2.10:2380
listen-peer-urls: https://192.0.2.10:2380
advertise-client-urls: https://192.0.2.10:2379
listen-client-urls: https://192.0.2.10:2379,https://127.0.0.1:2379
initial-cluster: pdns-lon=https://192.0.2.10:2380,pdns-wlv=https://192.0.2.20:2380,pdns-iad=https://192.0.2.30:2380
initial-cluster-token: hye-dns-example-2026
initial-cluster-state: new
client-transport-security:
  cert-file: /etc/etcd/pki/server.crt
  key-file: /etc/etcd/pki/server.key
  trusted-ca-file: /etc/etcd/pki/ca.crt
  client-cert-auth: true

Use a unique member name and cluster token. Bootstrap the cluster once; do not restart an established cluster with initial-cluster-state: new. Add, replace or remove members through the supported membership procedure, one controlled change at a time. Regularly take authenticated etcd snapshots and test restoring them away from production. The official etcd clustering guide is the source of truth for the version you deploy.

Configuring Patroni Synchronous Replication for PostgreSQL 17

Patroni owns the PostgreSQL lifecycle. Disable the distribution PostgreSQL service for this data directory; two supervisors trying to start, stop or promote the same cluster is an avoidable failure mode.

The following shortened Patroni example shows intent, not complete production configuration:

scope: hye-dns-example
namespace: /service/
name: pdns-lon

restapi:
  listen: 192.0.2.10:8008
  connect_address: 192.0.2.10:8008
  certfile: /etc/patroni/pki/server.crt
  keyfile: /etc/patroni/pki/server.key
  cafile: /etc/patroni/pki/ca.crt
  verify_client: required

etcd3:
  hosts: 192.0.2.10:2379,192.0.2.20:2379,192.0.2.30:2379
  protocol: https
  cacert: /etc/patroni/pki/etcd-ca.crt
  cert: /etc/patroni/pki/client.crt
  key: /etc/patroni/pki/client.key

bootstrap:
  dcs:
    synchronous_mode: true
    synchronous_mode_strict: true
    synchronous_node_count: 1
    maximum_lag_on_failover: 16777216
    maximum_lag_on_syncnode: 8388608
    check_timeline: true
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_level: replica
        hot_standby: "on"
        synchronous_commit: remote_apply
        password_encryption: scram-sha-256

Set per-node nofailover, nosync and failover priority according to the intended topology. The nearby Wolverhampton node is eligible and preferred for synchronous duty; distant nodes stay asynchronous in normal operation. Lag limits and timeline checks reduce the chance of promoting a stale or divergent replica. Replication slots protect required WAL, while retention and disk alerts prevent an absent replica from filling the leader.

Use SCRAM credentials and tightly scoped pg_hba.conf rules for the replication and PowerDNS roles. Store passwords and certificates in protected secret files or a secret manager, not the repository. Consult the current Patroni replication-mode documentation, Patroni REST API reference and PostgreSQL synchronous replication documentation before selecting values.

Connecting PowerDNS Locally

Create the PowerDNS role and database on the current leader, then import the official PostgreSQL schema supplied for your PowerDNS version. Replication carries that database to the other nodes. We use the local Unix socket so DNS reads do not leave the server:

launch=gpgsql
gpgsql-host=/run/postgresql
gpgsql-port=
gpgsql-dbname=powerdns
gpgsql-user=powerdns
gpgsql-password=${FROM_A_PROTECTED_SECRET_FILE}
gpgsql-dnssec=yes

The placeholder above is deliberately not a literal environment expansion recommendation for every init system. Use the secret-loading mechanism supported by your deployment and confirm file permissions. Never expose the database or PowerDNS API to the public internet.

gpgsql-dnssec=yes enables the backend features needed for DNSSEC data. After restore or failover, verify domain metadata and cryptographic key rows as well as ordinary records. If you are building PowerDNS itself for the first time, begin with our PowerDNS Authoritative VPS guide.

Following Patroni Role Changes

Database promotion must also change application duties. Our Patroni role callback applies a conservative mapping:

• Primary: enable the required PowerDNS primary and transfer duties.
• Replica: disable write-oriented primary/secondary duties but keep authoritative reads running.
• Unknown or stopped: default to duties disabled.

case "${PATRONI_ROLE:-unknown}" in
  primary) mode="primary" ;;
  replica) mode="read-only" ;;
  *)       mode="disabled" ;;
esac

/usr/local/sbin/render-pdns-role-config "$mode"
pdns_server --config=check
sudo -n /usr/bin/systemctl restart pdns

In real code, obtain the role from Patroni's documented callback arguments, write configuration atomically and log every transition. The service account receives a narrowly scoped sudoers entry for the exact validation/restart action—not unrestricted root access. If validation fails, duties remain in the safer disabled state.

Making the Application Leader-Aware

A working Patroni cluster did not make the HYEHOST Panel safe automatically. A fixed PowerDNS API endpoint could target yesterday's leader after promotion. We changed the mutation path to fail closed:

1. Before every DNS mutation, the Panel queries the private Patroni /primary endpoint on every enabled node.
2. A candidate must report primary status and hold Patroni's leader lock.
3. If zero candidates qualify, the Panel rejects the write.
4. If more than one qualifies, the Panel also rejects the write.
5. Only with exactly one leader does the Panel send the PowerDNS API mutation to that node.
6. Afterward it checks the zone serial and DNSSEC state across the authoritative fleet.

Temporarily rejecting a record change is safer than writing to a stale leader or proceeding during a possible split-brain condition. This exactly-one-primary rule turns uncertainty into an explicit, observable failure instead of corrupted state.

A private HAProxy endpoint driven by Patroni health checks is a reasonable alternative. The proxy then becomes part of the critical path: deploy it highly available, restrict access, test health-check semantics and ensure it cannot route writes to replicas. In our case direct discovery fitted the Panel's existing multi-location health model. The broader PostgreSQL and private-network pattern is described in our PostgreSQL, PgBouncer and WireGuard Panel case study.

Monitoring the Complete DNS Service

“Patroni has a leader” is one check, not a service-health conclusion. Our HYE DNS admin and health systems observe separate layers:

Alert these independently. A promoted database can coexist with a blocked IPv6 firewall, broken TCP/53, stale anycast route or invalid DNSSEC chain. Our monitoring guide explains the baseline metrics, external probes and log handling we extend here.

Testing Failover Without Risking Customer Zones

We use a dedicated canary zone and documentation addresses. A controlled exercise follows the same path a real change would use:

1. Confirm etcd quorum and membership, Patroni roles, timelines and replication lag.
2. Query the canary directly from every location over IPv4 and IPv6, UDP and TCP.
3. Write a unique record such as failover-20260816.canary.example A 192.0.2.44 through the normal Panel path.
4. Perform a controlled Patroni switchover—not an improvised process kill for the first test.
5. Confirm the Panel discovers exactly one new writer and mutations reach it.
6. Check the new record and SOA serial on every remaining authoritative node.
7. Validate DNSSEC, public recursive answers and anycast endpoints independently.
8. Rejoin the former leader, verify pg_rewind or rebuild behaviour, and confirm its timeline and convergence.

Measure the write-rejection window and time to authoritative convergence. Repeat with a synchronous standby unavailable, one etcd voter unavailable and a distant replica lagging. Stop if the pre-flight state is not clean. Canary tests should be routine enough that failover is an exercised procedure rather than a diagram.

Backups and Disaster Recovery

Replication is not a backup: destructive SQL, accidental zone deletion and corruption can reach every replica. We keep off-cluster PostgreSQL backups with retention and checksums, etcd snapshots, and copies of the PowerDNS and Patroni configuration. Restores run in isolation so a test cannot join production or overwrite live state.

A useful restore test proves more than “PostgreSQL started”. Query restored zones; compare SOA serials; inspect domain metadata, TSIG material and DNSSEC keys; validate a signed canary; and document how applications will discover the restored writer. Recovery time and recovery point targets should come from measured restores.

Common PowerDNS and Patroni Mistakes

• Making every remote region an etcd voter: more members do not automatically mean more availability, and WAN latency affects consensus.
• Letting a distant replica become synchronous unexpectedly: commit latency can jump across an ocean. Encode and test eligibility.
• Running PostgreSQL under Patroni and the distribution systemd unit: one data directory must have one supervisor.
• Sending writes to a fixed PowerDNS node: promotion changes the writer; the application must discover it safely.
• Exposing management APIs publicly: PostgreSQL, etcd, Patroni and PowerDNS APIs belong on restricted private networks.
• Monitoring Patroni but not DNS: query actual authoritative service over both protocols and address families.
• Treating replication as backup: replication distributes bad changes too.
• Claiming guaranteed zero data loss: state the synchronous policy, limitations and tested failure cases precisely.

Production Checklist

• Nodes occupy genuine compute, power, network and geographic failure domains.
• etcd has three or five authenticated voters; distant non-voters do not slow quorum.
• PostgreSQL, etcd, Patroni and PowerDNS APIs use private, filtered management paths.
• Sync eligibility, node priority, lag limits, timeline checks, slots and rewind behaviour are explicit.
• The latency and write-availability effect of strict synchronous replication is measured.
• Patroni exclusively supervises the PostgreSQL data directory.
• PowerDNS reads locally and write-oriented duties follow Patroni roles, failing disabled.
• The application requires exactly one verified primary before mutation.
• Monitoring covers database, DNS, IPv4, IPv6, UDP, TCP, DNSSEC, anycast and public recursion.
• A canary zone exercises controlled switchover and rejoin regularly.
• Off-cluster backups and etcd snapshots pass isolated restoration tests.

PowerDNS Patroni High Availability FAQ

Can PowerDNS use Patroni?

Yes. Patroni manages the PostgreSQL cluster used by PowerDNS; PowerDNS still uses its Generic PostgreSQL backend. The application and write-oriented PowerDNS duties must follow the current Patroni leader.

Does Patroni replace PowerDNS primary and secondary replication?

No. Patroni replicates the complete PostgreSQL backend. PowerDNS AXFR and IXFR remain DNS distribution mechanisms with different operational purposes.

How many etcd members should a Patroni cluster use?

Use an odd number of voters, normally three or five, in meaningful failure domains. Extra distant voters can increase latency without improving useful fault tolerance.

Should cross-region Patroni use synchronous replication?

Only after measuring latency and accepting the availability trade-off. HYEHOST uses one nearby preferred synchronous standby and keeps distant replicas asynchronous during normal operation.

Can PowerDNS answer from a PostgreSQL hot standby?

Yes. Read-only authoritative queries can be served from a local hot standby when the backend and schema are compatible. Write-oriented PowerDNS duties must remain disabled there.

Does high availability remove the need for backups?

No. Deletion, corruption and mistakes can be replicated. Keep independent off-cluster backups and test isolated restoration.

What happens to DNS answers while Patroni elects a new leader?

Existing authoritative answers continue from each region's local PostgreSQL copy. Control-plane changes may pause until the Panel discovers exactly one safe writer.

Official Resources

• PowerDNS modes of operation
• PowerDNS Generic PostgreSQL backend
• Patroni replication modes
• Patroni REST API
• PostgreSQL synchronous replication
• etcd clustering guide