A stalled replica piles up connections in use, so reads go to the replica with the fewest
connections in use instead of round-robin, and to the primary when every replica's pool is
full. ntfy_db_ping_seconds shows how long each database took to answer its health check.
A node cut off from its database only showed in the logs. The health check loop now always
runs and pings the primary too; ntfy_db_up (1/0, per pool, next to ntfy_db_pool_*) is the
result of the last check, so an alert can fire on ntfy_db_up{role="primary"} == 0.
ntfy_db_pool_connections_{max,open,in_use,idle}, ntfy_db_pool_waits_total and
ntfy_db_pool_wait_seconds_total, one series per pool (role primary or
replica, host). Read from database/sql's pool stats at scrape time, so a
burst that exhausts a pool between two scrapes still shows up as waits.
Needed to size pool_max_conns for several active cluster nodes: the
managed database allows 97 connections, and the open count alone says
nothing, since the pool rotates through all its connections.
A link that only works one way (a firewall dropping the cluster port in one direction) left
both nodes healthy: the sender only logged lost batches and bumped an unlabeled error counter,
and the receiver had no way to see it never heard from that peer. Every node now counts the
message batches it sends to and receives from each peer, and shows them:
- GET /v1/cluster/status on the cluster listener (secret-authenticated): role, leadership,
health, and per peer the outbound and inbound counters, last success and last error
- ntfy_cluster_peer_batches_sent_total{peer,result}, ntfy_cluster_peer_batches_received_total{peer}
and ntfy_cluster_peer_last_send_success_timestamp_seconds{peer}
- ntfy cluster status [--json], which reads the listener and secret from the server config and
lists the link problems
/v1/health and /v1/ready are unchanged.
A read takes up to 10 pages of 1,000 rows until it is caught up, so a
burst of up to 10,000 messages a second no longer leaves it behind. Pages
read before a failed one are returned; the failed one is retried.
ntfy_tail_lag_seconds is the age of the newest message of the last read:
a value well above the write delay means the tail is falling behind.
Every node reads the rows written to the message cache once a second, in
write order (by row id), and sends each message to the local subscribers
that did not get it live: all of them if its fan-out to this node was
lost, or those that subscribed after it went out (they were reconnecting,
and their database read ran before the message was written).
Row ids are handed out on insert but become visible on commit, so a row
id the tail had to skip is re-checked until its row shows up or a minute
passes. In a cluster, since=<id> is read as a time a few seconds before
that message, since nodes write their batches independently.
The cluster package now does the thing the seam described. Nothing constructs
it: the server still holds no cluster, and cmd still refuses to start with the
options set, so this is inert like the pieces before it.
A publish marshals once and drops the line into one bounded queue per peer. A
worker batches what it finds, up to the batch linger and flushed early at 100
messages or 256KB, and POSTs NDJSON to that peer's /v1/cluster/message. Fire
and forget: a wedged peer backs up only its own queue, and a full queue drops
rather than blocking the caller. Inbound batches are authenticated by shared
secret, carry an origin header for loop prevention, and are decoded line by
line into the server's delivery callback.
The heartbeat loop refreshes this node's registry row, reads the live set back
and reconciles the per-peer queues, so a departing peer's queue is retired and
a new one is reachable on the next publish. Members reports this node plus its
peers, with peer health being the freshness of this node's own registry view
(registry.Fresh, added here), because an isolated node answering from a frozen
view must not be mistaken for current membership. A node that lost the
database while a peer is still healthy calls back to close its subscribers, and
keeps them when no peer is healthy, since that is an outage rather than
isolation.
Delivery gaps are not here: a failed batch is logged and counted. They come
with the recovery mechanism that uses them.
binwiederhier