A weight like 1e16 overflowed to a negative token count, which the limiter always allows, so the code never banned. Weights above 1,000,000 and thresholds above 1,000,000,000 are now rejected at startup.
Earlier cmd tests leave servers running whose sync workers keep logging into the global log output; the test read a plain bytes.Buffer while they wrote to it.
Rename listen-admin-unix to experimental-listen-admin-unix and drop the
/run/ntfy/admin.sock default: the CLI side (#2089, #2135) is not merged,
so nothing uses the socket yet. The startup warning no longer points at a
CLI fallback, and the docs say the commands do not use it yet.
Like the Postgres pool: user.OpenSQLite opens the auth-file (case-sensitive LIKE),
NewSQLiteManager takes the handle and migrates it, and the usage store gets the same handle
instead of reaching into the manager (Manager.DB is gone). The usage table stays part of the
user schema.
Move the base-url check out of the else-if chain instead of into a
helper, and test both cases with an invalid auth-default-access, which
fails right after the checks, instead of starting a server.
Other systems keep two things apart: draining before a shutdown (ephemeral, cleared by the restart: RabbitMQ drain, CockroachDB node drain) and an operator's maintenance flag (persistent cluster state: Kubernetes cordon, Nomad eligibility, Consul maint). This PR is the former, so it is named that way: Server.Drain (synchronous again; the on/off API is gone until the persistent maintenance flag exists), Cluster.SetDraining/Draining, the read-only phase flag readOnly, /v1/ready reports "draining": true, error 50301 says the node is shutting down, and docs/config.md has a "Draining on shutdown" section. A drain duration of 0 now closes all streams at once instead of disabling the drain.
Maintenance mode now has two phases. Entering (experimental-cluster-drain-delay, default 15s): /v1/ready fails and the node acts as passive in the cluster, but it still serves everything, so the load balancers can stop sending it requests without any request failing. Draining (experimental-cluster-drain-duration, default 1m): read-only, i.e. new subscriptions and all publishes get 503 (email gets SMTP 451, so mail servers retry instead of bouncing), and the streams are closed evenly. The defaults together fit systemd's 90s stop timeout. docs/config.md gets a maintenance mode section with the behavior table.
A node in maintenance mode now registers as passive (registry.SetRole at runtime), so peers stop counting it as a place for clients and leadership moves to the next active node; it is not ready, does not lead (at once, before its row says passive), and never fences. Its configured role is unchanged. It also claims no due delayed messages, which the next leader sends. Also tests that a drain duration of 0 is accepted.
A node draining before a shutdown is in maintenance mode: /v1/ready reports "maintenance": true (was "draining"), and the 503 says the node is in maintenance mode. Maintenance is a runtime state next to the configured role, not a role: a later admin command can enter and leave it without a shutdown, with the same semantics. server_drain.go is now server_maintenance.go.
On SIGTERM, an active cluster node with another healthy active node first drains: new subscriptions (HTTP streams and WebSockets) get 503, which a load balancer retries on another node, /v1/ready fails with "draining": true, and the open streams are closed evenly over experimental-cluster-drain-duration (default 1m, fits systemd's 90s stop timeout; 0 disables). Clients reconnect a few at a time and resume with since=, instead of all at once. Publishes and polls are still served, since a load balancer does not retry a POST. A second signal skips the rest of the drain.
A stream registry on the server tracks open subscriber connections once each, however many topics they subscribe to.
A load balancer connects to each upstream address from one source IP, so one address caps it at ~63k connections to a node. listen-http now takes a comma-separated list (whitespace trimmed, empty and duplicate entries rejected, "-" still unsets it); Config.ListenHTTP becomes a slice, Run starts one HTTP server per address on the same handler, and stop closes all of them.
The server listens on admin-socket (default /run/ntfy/admin.sock, none
on Windows) when auth is configured. Requests on it run as a synthetic
admin, skip rate limiting and the ban feed, and may do what the network
API refuses (roles, changing or deleting admins). File mode 0660 in a
0750 directory is the authorization; peer credentials go into the audit
log on Linux and the BSDs.
A stale socket file is replaced, a socket another server answers on is
left alone, and failing to listen only logs a warning so the CLI can
fall back to the database. The systemd unit gets RuntimeDirectory=ntfy.
The client calls in the mesh, the peer sort and the test closures from #2074 and this PR were one-line func literals. Tests that ignore deliveries now pass a named discardMessage instead of an empty literal.
From the review of #2074: a peer whose sends all failed had no last-success series, so an alert on its age had no sample; a forgotten peer's series stayed exported until restart; and a failure after a success in the same Unix second hid the warning, since the CLI compared second-resolution times. New peers' series now start at zero, forgotten peers' series are deleted, and PeerOutbound.LastFailed records the latest outcome. The docs suggest an alert on failures instead of the last-success age, which also fires for a node with nothing to send.
Every outbound peer call now goes through one client in the cluster package (like s3/client.go):
it builds the request, sets the secret and origin headers, and checks and decodes the answer. The
mesh keeps its decisions (what a failure means, counters, metrics, logs) and sends message batches,
state and health probes through it; the CLI reads node statuses with it. FetchStatus is replaced by
Client.Status, and the URL helpers move into client.go.
Status kept a departed peer's counters with live: false, but not when it left, and until this node restarted. Each peer now has a last-seen time (refreshed while it is in the registry); one gone for 24 hours is dropped. The CLI shows "last seen ... ago" for it, and only a peer that left within the last 10 minutes is a problem; one that left earlier was likely removed on purpose.
It fetches every live peer's status from its advertised cluster URL and prints each link once: what the sender sent next to what the receiver received. A one-way link, like a firewall dropping the cluster port in one direction, then shows from a single command on either node; a peer whose status cannot be fetched is reported instead of failing the command.
A link that only works one way (a firewall dropping the cluster port in one direction) left
both nodes healthy: the sender only logged lost batches and bumped an unlabeled error counter,
and the receiver had no way to see it never heard from that peer. Every node now counts the
message batches it sends to and receives from each peer, and shows them:
- GET /v1/cluster/status on the cluster listener (secret-authenticated): role, leadership,
health, and per peer the outbound and inbound counters, last success and last error
- ntfy_cluster_peer_batches_sent_total{peer,result}, ntfy_cluster_peer_batches_received_total{peer}
and ntfy_cluster_peer_last_send_success_timestamp_seconds{peer}
- ntfy cluster status [--json], which reads the listener and secret from the server config and
lists the link problems
/v1/health and /v1/ready are unchanged.
Attachments never outlive their message, so a longer attachment expiry is
silently capped at cache-duration. Warn at startup and document the cap.
Fixes#1053
Makes clustering usable: experimental-cluster-listen is accepted now and the
startup guard is gone.
The server holds a cluster, constructs it with the callbacks it supplies (so
the cluster package never imports the server), and serves the peer API on a
dedicated private listener that the public listeners never expose. dispatch,
the one choke point every publish path goes through, hands each message to the
cluster before delivering locally. deliverFromBus publishes an inbound peer
message to the local topic and nothing else: the origin already ran Firebase,
email, web push and upstream, so a peer never re-fires them or re-relays.
/v1/health answers 503 once this node's registration has gone stale, so a load
balancer can pull it, and the cluster calls closeLocalSubscribers when this
node is isolated but a peer is healthy, so those clients reconnect elsewhere.
Two jobs become leader-only, checked per tick to survive failover. The Firebase
keepaliver has to be: one keepalive wakes every subscribed phone. The delayed
sender does not, and the comment says so, since the claim in the message cache
is what makes that safe; the gate only keeps the work on one node.
What a cluster does not do yet, each waiting for its own piece: recover a
dropped fan-out batch, enforce rate limits and quotas across nodes, share
per-topic state (UnifiedPush billing, the Matrix pushkey decision), replay
unflushed messages to a reconnecting subscriber, or relay subscriber cancels.
Hence experimental-.
Five option names and their validation, ahead of the implementation, so the
config surface is settled and the cluster change is only code. The
experimental- prefix is there because the names may still change.
The options parse into server.Config and the cross-field rules are enforced:
a listen address needs database-url, a secret and a node id; a secret without
a listen address is an error rather than clustering being silently off; a
wildcard bind needs an explicit advertise URL; a negative linger is rejected.
Setting experimental-cluster-listen then refuses to start the server, since
accepting the options and running single-node would look like a working
cluster. The cluster change deletes that one line.
With cache-batch-size/-timeout set (ntfy.sh: 100 / 1500ms), accepted messages
wait in a BatchingQueue before they are written. Nothing drained that queue on
shutdown, and `ntfy serve` did not handle SIGTERM at all (only SIGHUP), so
every restart dropped up to one batch timeout's worth of accepted messages
from the cache: live subscribers got them, pollers and since= replays never
did.
- util.BatchingQueue.Close emits the pending elements and closes the output
channel (Enqueue after Close is a no-op)
- message.Cache.Close closes the queue and waits for the batch writer before
closing the database
- serve stops the server on SIGTERM/SIGINT and waits for Stop to finish (Run
returns as soon as the listeners close); Stop is idempotent now, since the
signal handler and serve both call it
binwiederhier
Philipp C. Heckel
Philipp C. Heckel