The server builds the usage store on the user database (the shared pool on
Postgres, the auth-file on SQLite) and starts the limiter before serving. A
visitor without a user writes its message count from a tenth of the cap on,
everyone else from the first message; anonymous emails get the daily cap the
replenish interval implies. The account API reports reset_at. The user stats
columns are no longer read or written.
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
Allow(key, rule, n), Refund, Tokens and Usage on a service that keeps a
token bucket and a daily count per key and kind, sharded by key, with the
rule passed in on every call. The server's visitor keeps its identity and
the resolved limits and asks the service; rate.Limiter, util.RateLimiter
and util.FixedLimiter are gone from it (subscriptions stay a per-node
gauge). util.RateLimiter, NewBytesLimiter and BurnTokens had no users
left and are removed.
Behavior changes: refunds work for every kind (a revert on the bandwidth
limiter hands the bytes back), a tier change applies on the next request
without handing back full buckets, a changed tier limit applies without
a tier ID change, and the daily email limit is a real daily cap on top
of the bucket. Step 2 of plans/261025-shared-rate-limits.md.
A link that only works one way (a firewall dropping the cluster port in one direction) left
both nodes healthy: the sender only logged lost batches and bumped an unlabeled error counter,
and the receiver had no way to see it never heard from that peer. Every node now counts the
message batches it sends to and receives from each peer, and shows them:
- GET /v1/cluster/status on the cluster listener (secret-authenticated): role, leadership,
health, and per peer the outbound and inbound counters, last success and last error
- ntfy_cluster_peer_batches_sent_total{peer,result}, ntfy_cluster_peer_batches_received_total{peer}
and ntfy_cluster_peer_last_send_success_timestamp_seconds{peer}
- ntfy cluster status [--json], which reads the listener and secret from the server config and
lists the link problems
/v1/health and /v1/ready are unchanged.
The message cache decides how many batches may wait for its writer (queueBufferedBatches in
message/cache.go, next to the shutdown flush timeout it has to fit in); util stays policy-free.
On cron bursts (top of the hour), the single batch writer fell behind: it wrote one INSERT round
trip per message, so against a busy remote Postgres it managed ~100 rows/s while ~300/s arrived.
New messages took up to ~40s to reach the database (polls missed them), and publishes blocked on
the unbuffered queue for up to ~25s.
- Postgres: write each batch with one multi-row INSERT (up to 1000 rows per statement) instead of
one statement per row. SQLite keeps the row-by-row path.
- Skip DeleteScheduledBySequenceID when the client did not set a sequence ID: the sequence ID is
then the message's own fresh ID, so no scheduled message can match. This saves a transaction on
the primary for almost every publish.
- Buffer up to 100 batches in the BatchingQueue, so a briefly slow writer does not stall publishes.
- dbtest: add a latency proxy (CreateTestPostgresWithLatency, WithLatency) to test round trips.
A review of this PR pointed out that the cap was computed from a separate
read of the available tokens: a read and a reservation are two operations, so
anything consuming the limiter in between makes the read stale and the burn
lands deeper than one burst. The cap now comes from each reservation's own
delay, which the limiter computes atomically with the consumption, and the
chunk that would run past one burst of replenishment is cancelled.
That holds the cap against any amount of ordinary concurrent consumption. It
does not make concurrent BURNS of one limiter safe: two burners can each
reserve a chunk that looked acceptable, and the cap is then exceeded by about
a chunk per racing burner (roughly three bursts with 32 of them, measured).
Both the old and the new code behave that way, so this is a documented
contract rather than a fix: callers burning one limiter from several
goroutines serialize, which RateLimiter.Burn does with its own mutex and the
cluster does by burning peer usage from a single flush loop.
Tests cover the two invariants that hold: the cap survives ordinary traffic
racing the burn, and serialized burners share one burst of debt instead of
each taking one.
A bounded non-blocking batching queue, a token-burn helper for consumption
that happened elsewhere, and MaxTime. Nothing uses them yet; they land on
their own so they can be reviewed as primitives rather than inside the
cluster diff.
LingerQueue differs from BatchingQueue in that producers never block: a full
queue drops and says so, and Close flushes the remainder before closing the
consumer channel, so per-peer queues can come and go with the nodes.
BurnTokens caps the debt it creates at one burst beyond what is available, so
usage reported from another node can drain a bucket but cannot lock a visitor
out for an unbounded time.
The final batch write has no deadline (no statement_timeout, and database/sql
is used without a context), so an unresponsive database would hold Stop until
systemd's stop timeout fires and SIGKILLs us. Cache.Close now gives up after
5 seconds, both on handing the pending messages to the batch writer and on
waiting for it, and logs that it did.
With cache-batch-size/-timeout set (ntfy.sh: 100 / 1500ms), accepted messages
wait in a BatchingQueue before they are written. Nothing drained that queue on
shutdown, and `ntfy serve` did not handle SIGTERM at all (only SIGHUP), so
every restart dropped up to one batch timeout's worth of accepted messages
from the cache: live subscribers got them, pollers and since= replays never
did.
- util.BatchingQueue.Close emits the pending elements and closes the output
channel (Enqueue after Close is a no-op)
- message.Cache.Close closes the queue and waits for the batch writer before
closing the database
- serve stops the server on SIGTERM/SIGINT and waits for Stop to finish (Run
returns as soon as the listeners close); Stop is idempotent now, since the
signal handler and serve both call it
binwiederhier
Philipp C. Heckel
binwiederhier
cangen185-sudo
Philipp C. Heckel
Martin Brehmer
Bee mines
tanhuaan
Hunter Kehoe