Canceling a subscriber logs the topic's context, which takes the topic lock. The cancels held that lock while they logged: CancelSubscribersExceptUser held the write lock (deadlock at once), the per-user cancels a read lock (deadlock as soon as a canceled stream's Unsubscribe waited for the write lock). The context is only evaluated with debug logging or log-level-overrides on. In the lab this froze the access cache reload loop after its first re-check. The cancels now collect the matching subscribers under the lock and cancel them after releasing it.
globalLevelWithOverride iterated the overrides map without the log lock while SetLevelOverride changes it in place; it now holds the read lock while iterating.
A burst of "access changed" signals no longer queues one full reload per signal on each peer: RequestAccessCacheReload returns at once and hands the request to the reload loop through a channel with room for one, so requests that arrive while a reload runs collapse into one more reload, which still sees every change. The node making the change keeps its synchronous reload (read-after-write).
Access changes are rare and a full reload is cheap (about 15 ms on ntfy.sh's 5,500 entries), so the access cache no longer reloads per user. Every reload is full and serialized (which replaces the sequence-number race guard), and the diff after each reload re-checks the live subscribers on the changed topics.
An access write (reservation, allow, reset, user removal, role change) reloads the local cache and tells the other cluster nodes "access changed" (cluster.State.AccessChanged), which reload theirs; their diff ends the streams that lost access there. This replaces the relayed per-topic cancels and the targeted reloads. Only a deleted user's cancel is still relayed: a user without access entries (e.g. an admin) leaves nothing in the cache to compare. The periodic reload, the backstop for a lost signal and for writes by other processes (ntfy access CLI), now runs every 31 s instead of 87 s.
Other systems keep two things apart: draining before a shutdown (ephemeral, cleared by the restart: RabbitMQ drain, CockroachDB node drain) and an operator's maintenance flag (persistent cluster state: Kubernetes cordon, Nomad eligibility, Consul maint). This PR is the former, so it is named that way: Server.Drain (synchronous again; the on/off API is gone until the persistent maintenance flag exists), Cluster.SetDraining/Draining, the read-only phase flag readOnly, /v1/ready reports "draining": true, error 50301 says the node is shutting down, and docs/config.md has a "Draining on shutdown" section. A drain duration of 0 now closes all streams at once instead of disabling the drain.
Maintenance mode is now a state that can be entered and left without a restart: Server.SetMaintenance(on) runs the phases in the background and returns at once; off stops them wherever they are and takes the node back into service (ready, writable, active in the cluster). WaitDrained waits for the drain. On SIGTERM, Drain is SetMaintenance(true) plus WaitDrained, as before. SetMaintenance(true) is idempotent and refuses with drain duration 0, on a passive node, or without another active node. This is what a later 'ntfy cluster maintenance on|off' needs.
registry.Peers is now a defined type with HasActive (cluster.Peer and cluster.Peers alias the registry types, like Role), and the Cluster interface exposes the live peers instead of a single yes/no; the server asks s.cluster.Peers().HasActive(). Replaces the mesh's hasActivePeer helper.
SetMaintenance stores the flag and then sets the registered role; two concurrent calls could interleave and leave them disagreeing. Writers now hold c.mu; reads stay lock-free (atomic), since Role is called on paths that already hold c.mu.
The cluster now owns the maintenance state (Cluster.Maintenance) instead of the server keeping its own copy; the server keeps only the draining phase. Role() returns the role the node acts in (passive in maintenance mode), so /v1/ready says "role":"passive","maintenance":true. Cluster.HasActivePeer replaces the server's own walk over Members. A single-node server no longer logs about maintenance mode on every shutdown. The stream registry moves to its own file, with unit tests.
GitHub's ubuntu-24.04 runners point ubuntu.sources at mirror+file:/etc/apt/apt-mirrors.txt, which lists the Azure mirror first, so rewriting ubuntu.sources alone changed nothing (PR #2095's build still fetched from azure.archive.ubuntu.com). build-deps-ubuntu now rewrites both files.
Maintenance mode now has two phases. Entering (experimental-cluster-drain-delay, default 15s): /v1/ready fails and the node acts as passive in the cluster, but it still serves everything, so the load balancers can stop sending it requests without any request failing. Draining (experimental-cluster-drain-duration, default 1m): read-only, i.e. new subscriptions and all publishes get 503 (email gets SMTP 451, so mail servers retry instead of bouncing), and the streams are closed evenly. The defaults together fit systemd's 90s stop timeout. docs/config.md gets a maintenance mode section with the behavior table.
The Azure apt mirror on GitHub runners trickles for minutes without stalling, which apt's timeouts do not catch: PR #2095's build hit the new 10 minute step timeout with one package taking 4.5 minutes. build-deps-ubuntu now points the runner's ubuntu.sources at archive.ubuntu.com before apt-get update; a no-op without the file or without Azure entries.
A node in maintenance mode now registers as passive (registry.SetRole at runtime), so peers stop counting it as a place for clients and leadership moves to the next active node; it is not ready, does not lead (at once, before its row says passive), and never fences. Its configured role is unchanged. It also claims no due delayed messages, which the next leader sends. Also tests that a drain duration of 0 is accepted.
A node draining before a shutdown is in maintenance mode: /v1/ready reports "maintenance": true (was "draining"), and the 503 says the node is in maintenance mode. Maintenance is a runtime state next to the configured role, not a role: a later admin command can enter and leave it without a shutdown, with the same semantics. server_drain.go is now server_maintenance.go.
pg_indexes spans every schema, so TestPostgresStore_CreatesTopicPublishedIDIndex also matched the message tables of parallel tests, and failed with 'could not open relation with OID' when one of them dropped its schema at that moment (seen once in CI on this branch).
The Azure apt mirrors on GitHub runners sometimes stall mid-download, and apt waits forever (three hung runs on 2026-10-23, two of them releases). build-deps-ubuntu now passes Acquire::Retries=3 and a 30s http/https timeout, and the Install dependencies step of the build, test and release workflows fails after 10 minutes instead of hanging.
On SIGTERM, an active cluster node with another healthy active node first drains: new subscriptions (HTTP streams and WebSockets) get 503, which a load balancer retries on another node, /v1/ready fails with "draining": true, and the open streams are closed evenly over experimental-cluster-drain-duration (default 1m, fits systemd's 90s stop timeout; 0 disables). Clients reconnect a few at a time and resume with since=, instead of all at once. Publishes and polls are still served, since a load balancer does not retry a POST. A second signal skips the rest of the drain.
A stream registry on the server tracks open subscriber connections once each, however many topics they subscribe to.
A load balancer connects to each upstream address from one source IP, so one address caps it at ~63k connections to a node. listen-http now takes a comma-separated list (whitespace trimmed, empty and duplicate entries rejected, "-" still unsets it); Config.ListenHTTP becomes a slice, Run starts one HTTP server per address on the same handler, and stop closes all of them.
A goreleaser post-build hook runs the amd64 binary under qemu-x86_64 -cpu qemu64 (no AVX2, BMI2, MOVBE), so x86-64-v3 code in the statically linked C parts (#2092) fails make release before anything is published, and fails CI builds. Without qemu the check fails in CI and is skipped with a warning locally. build-deps-ubuntu installs qemu-user.
v2.33.0 was built on the ubuntu-26.04 runner image (via ubuntu-latest), whose toolchain made the statically linked C code (glibc, SQLite) use x86-64-v3 instructions, crashing on CPUs without AVX2/BMI2 (#2092). Pin the release, build and test workflows to ubuntu-24.04, add v2.33.1 release notes and bump install.md.
The server listens on admin-socket (default /run/ntfy/admin.sock, none
on Windows) when auth is configured. Requests on it run as a synthetic
admin, skip rate limiting and the ban feed, and may do what the network
API refuses (roles, changing or deleting admins). File mode 0660 in a
0750 directory is the authorization; peer credentials go into the audit
log on Linux and the BSDs.
A stale socket file is replaced, a socket another server answers on is
left alone, and failing to listen only logs a warning so the CLI can
fall back to the database. The systemd unit gets RuntimeDirectory=ntfy.
Fill the gaps between the admin HTTP API and what the ntfy CLI does
directly against the database: list/add/remove other users' tokens,
manage tiers (ported from the user-management branch), create password
reset links (optionally emailed), remove a user's tier, reset a user's
or the whole ACL, and return provisioned flags, tier names and the
default access in GET /v1/users.
All admin routes go through adminHandler, which the admin socket will
share. Requests marked as coming over the socket may set roles and
change or delete admins; the network API still refuses that. User
manager errors now map to 4xx errors instead of 500s.
Go 1.27.2 and 1.27.3 are security releases; 1.27.3 makes crypto/x509
reject chains through the eight compromised eMudhra intermediate CAs.
ntfy's outbound TLS (Firebase, web push, Stripe, Twilio, S3, upstream)
only gets that by being rebuilt.
The bump was held at 1.27.1 because staticcheck v0.8.1 could not read
Go 1.27.2's export data; staticcheck 2026.2.2 (v0.8.2) can, and is
clean. text/template is unchanged, so only GENERATED_FROM moves.
ntfy_db_pool_connections_{max,open,in_use,idle}, ntfy_db_pool_waits_total and
ntfy_db_pool_wait_seconds_total, one series per pool (role primary or
replica, host). Read from database/sql's pool stats at scrape time, so a
burst that exhausts a pool between two scrapes still shows up as waits.
Needed to size pool_max_conns for several active cluster nodes: the
managed database allows 97 connections, and the open count alone says
nothing, since the pool rotates through all its connections.
Reserving a topic or revoking access only canceled the subscribers on the
node that made the change; on other cluster nodes, an anonymous stream kept
receiving a topic that had just become private.
- Relay reservation takeovers and access revocations to peers (cluster
CancelFunc). The peer reloads its access cache for everyone and the user
concerned before canceling, so an immediate reconnect is denied.
- Access cache reloads now report the entries they found changed, and the
server re-checks only the subscribers on matching topics. This catches a
lost relay and changes made with the ntfy access CLI within one reload
interval (auth-access-cache-reload-interval).
- CancelSubscriberUser ends all of a user's streams on a topic, not just
the first one.
Alvaro
binwiederhier
roypadina
Philipp C. Heckel
dependabot[bot]
cangen185-sudo