Requests from one IP (anonymous and untiered users) shared a single visitor
object with a mutable user, so overlapping requests could be authorized as
each other's user. The visitor is now built per request and never changed;
the limit service, keyed by visitor key, still shares the buckets and counts.
The Firebase penalty becomes a limiter kind, open subscriptions move to a
node-local gauge per key, and tier users' stats are enqueued per admitted
unit.
Fixes#1051
The Matrix gateway built its publish request from the push key without the
router's topic path check. Publishing sanitizes the topic after authorization
(it strips NUL bytes and invalid UTF-8), so an anonymous push key like
/sec%00ret was authorized against an unmatched topic and stored under the
protected topic "secret".
A tier upgrade used to keep the old bucket's tokens, capped at the new\nburst, so a bigger tier did not add its difference. The bucket now shifts\nby the burst difference both ways, down to a debt of one burst.\n\nemails_remaining is now capped by what the email bucket holds, so it no\nlonger shows emails the visitor cannot send (e.g. after the daily reset).\n\nClaude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
The limit service reads a zero cap as no cap, so a tier with no calls
(or no messages, or no attachment bandwidth on upload) became unlimited
where main's FixedLimiter(0) and bytes limiter admitted nothing. The
visitor now answers zero limits itself before asking the service, like
it already did for requests, emails and bandwidth downloads.
Allow(key, rule, n), Refund, Tokens and Usage on a service that keeps a
token bucket and a daily count per key and kind, sharded by key, with the
rule passed in on every call. The server's visitor keeps its identity and
the resolved limits and asks the service; rate.Limiter, util.RateLimiter
and util.FixedLimiter are gone from it (subscriptions stay a per-node
gauge). util.RateLimiter, NewBytesLimiter and BurnTokens had no users
left and are removed.
Behavior changes: refunds work for every kind (a revert on the bandwidth
limiter hands the bytes back), a tier change applies on the next request
without handing back full buckets, a changed tier limit applies without
a tier ID change, and the daily email limit is a real daily cap on top
of the bucket. Step 2 of plans/261025-shared-rate-limits.md.
The visitor key (ip:<prefix> with the configured prefix bits, or user:<id>
for a user with a tier) and the resolution of a visitor's limits from the
server defaults or the user's tier move out of server/visitor.go into the
new package visitor, unchanged. The server keeps thin wrappers, so its
tests are untouched. No behavior change. First step of the shared rate
limits design (plans/261025-shared-rate-limits.md).
meshCluster.Ready is already false while draining, since the node acts as passive; the extra check in handleReady was redundant. The fake cluster now follows the same contract.
accessCache now owns everything about reloading: Reload (synchronous, reports changes to the callback after releasing reloadMu), RequestReload (coalesced, at most one pending), the periodic loop, and Close. The Manager only tells the peers and forwards.
The coalescing test made its write before the burst, so it passed even if requests arriving during a reload were dropped. It now holds the first reload after its scan, writes and bursts while it is held, and requires the follow-up reload to see the write. Found by a Codex review.
topicsFromPattern held Server.mu while regex-matching every topic (~313k on ntfy.sh, ~89 ms), blocking every request that creates a visitor; allTopics copies the pointers under the lock and the match runs outside it (also used for deleted users). The access diff (~2.3 ms) no longer runs under the cache's write lock: only reloads replace the maps, and they are serialized. Found by a Codex review.
Account self-deletion (MarkUserRemoved) sent no deleted-user signal and did not end the user's local streams; an admin without access entries kept them, here and on peers. Both deletes now go through userDeleted. Found by a Codex review.
A revoke whose re-check ran between a request's authorization and its stream subscribing missed the stream for good: later reloads find no difference. With the access cache on, the subscribe handlers now authorize once more right after subscribing (an in-memory lookup); the re-check runs after the cache swap, so one of the two always sees the other. Found by a Codex review.
SubscriberUsers kept one user per user ID, so streams of one user with different users as of subscribing (e.g. before and after a demotion) were all judged by whichever came last. topic.CancelSubscribersWhere checks each subscription's own user, outside the topic lock; SubscriberUsers and CancelAnonymousSubscribers are gone. Found by a Codex review.
Added, removed and changed entries, exact and wildcard, per user, and the * form of wildcard topics. The loadCache test helper now builds entries with toACLEntry, as Reload does, so wildcard entries carry their stored topic.
The only cancel still relayed was a deleted user's, so the cluster now signals the event, like AccessChanged: State.DeletedUserIDs (deleted_user_ids on the wire) and Config.UsersDeletedFunc replace SubscriberCancels, the SubscriberCancel type and CancelFunc. The server's handleUsersDeleted ends those users' streams on every topic, for a local delete and a peer's.
The reload diff flattens both snapshots into one map keyed by user and stored topic, replacing four helpers. ReloadAccessCache was only used by tests (the server requests coalesced reloads), so they call maybeReloadAccessCache. killUserSubscriberEverywhere builds the cancel once, applies it locally and broadcasts it.
The only cancel still relayed is a deleted user's, on every topic; access changes go through the access cache reload. Topic and ExceptUserID are gone, and cancelUserSubscribersLocal ends a user's streams on all local topics for both the local delete and the peer cancel. No released version sends these cancels, so the wire format can change.
topicSubscriber no longer stores the user ID next to the user (userID() derives it), which also brings the struct below its size before the user was added. The cancel loops assign the matching subscribers to a variable instead of calling subscribersMatching inside the for header. The periodic access cache reload runs every 63 s: access changes are rare and peers are told directly, so the interval only bounds a missed signal.
topic.Subscribe now takes the subscriber's *user.User (nil if anonymous) and keeps it on the subscription. reauthorizeSubscribers authorizes that user against the access cache, so an access change no longer costs one UserByID query per distinct subscriber user on every matching topic, on every node (a wildcard change could mean thousands). A deleted user's streams end through the relayed cancel (killUserSubscriberEverywhere), as before.
It fires when an access cache reload found a difference, whoever made the change; the cache's content changed, not necessarily because of this node. The cluster's AccessChangedFunc (a peer changed access) keeps its name.
accessWritten now calls AccessWrittenFunc first, so the other nodes start their reloads without waiting for this node's (about one reload, ~20 ms on prod), then reloads the local cache before the write returns.
newAccessCache takes the database and the reload query, so Reload needs no arguments. reloadRequests is now accessCacheReloadRequests, above quit. AccessChangedFunc and AccessWrittenFunc sit below AccessCacheReloadInterval in user.Config, with comments.
Canceling a subscriber logs the topic's context, which takes the topic lock. The cancels held that lock while they logged: CancelSubscribersExceptUser held the write lock (deadlock at once), the per-user cancels a read lock (deadlock as soon as a canceled stream's Unsubscribe waited for the write lock). The context is only evaluated with debug logging or log-level-overrides on. In the lab this froze the access cache reload loop after its first re-check. The cancels now collect the matching subscribers under the lock and cancel them after releasing it.
globalLevelWithOverride iterated the overrides map without the log lock while SetLevelOverride changes it in place; it now holds the read lock while iterating.
A burst of "access changed" signals no longer queues one full reload per signal on each peer: RequestAccessCacheReload returns at once and hands the request to the reload loop through a channel with room for one, so requests that arrive while a reload runs collapse into one more reload, which still sees every change. The node making the change keeps its synchronous reload (read-after-write).
Access changes are rare and a full reload is cheap (about 15 ms on ntfy.sh's 5,500 entries), so the access cache no longer reloads per user. Every reload is full and serialized (which replaces the sequence-number race guard), and the diff after each reload re-checks the live subscribers on the changed topics.
An access write (reservation, allow, reset, user removal, role change) reloads the local cache and tells the other cluster nodes "access changed" (cluster.State.AccessChanged), which reload theirs; their diff ends the streams that lost access there. This replaces the relayed per-topic cancels and the targeted reloads. Only a deleted user's cancel is still relayed: a user without access entries (e.g. an admin) leaves nothing in the cache to compare. The periodic reload, the backstop for a lost signal and for writes by other processes (ntfy access CLI), now runs every 31 s instead of 87 s.
Other systems keep two things apart: draining before a shutdown (ephemeral, cleared by the restart: RabbitMQ drain, CockroachDB node drain) and an operator's maintenance flag (persistent cluster state: Kubernetes cordon, Nomad eligibility, Consul maint). This PR is the former, so it is named that way: Server.Drain (synchronous again; the on/off API is gone until the persistent maintenance flag exists), Cluster.SetDraining/Draining, the read-only phase flag readOnly, /v1/ready reports "draining": true, error 50301 says the node is shutting down, and docs/config.md has a "Draining on shutdown" section. A drain duration of 0 now closes all streams at once instead of disabling the drain.
Maintenance mode is now a state that can be entered and left without a restart: Server.SetMaintenance(on) runs the phases in the background and returns at once; off stops them wherever they are and takes the node back into service (ready, writable, active in the cluster). WaitDrained waits for the drain. On SIGTERM, Drain is SetMaintenance(true) plus WaitDrained, as before. SetMaintenance(true) is idempotent and refuses with drain duration 0, on a passive node, or without another active node. This is what a later 'ntfy cluster maintenance on|off' needs.
binwiederhier
Philipp C. Heckel
binwiederhier
Alvaro