Release notes: main moved the v2.32.0 notes to the released section, so this PR's bullet goes under v2.33.0's Bug fixes + maintenance. The test's one-line t.Cleanup literals are spread over lines.
Migrate took the target version as a separate argument, kept in step with the steps by hand:
adding a step without bumping the version constant silently skipped it on existing installs,
while fresh installs got the new tables. The target is now the version after the last step (1
without steps), and the steps must be contiguous, so a forgotten one fails at startup. The
per-store version constants are gone.
The stored total was overwritten with one node's in-memory count, so only the leader could
write it (#2066) and publishes on other nodes were never counted. Every node now adds the
publishes since its last flush to the stored total, reads the total back, and moves its flush
marker to it, so peer counts are never added twice. The total loaded at startup is the first
marker, so a restart does not count it again; a single node counts exactly as before.
Instead of on every start: the schema is defined in one place again. Postgres gets
idx_message_topic_published_id (topic, published, id), SQLite idx_topic_published_time (topic,
published, time), both with IF NOT EXISTS (ntfy.sh got the Postgres one by hand). A downgrade to
v2.31.0 or earlier now needs the schema version set back by hand, as described in the release notes;
from v2.31.1 on, a newer schema only logs a warning (#2059).
On ntfy.sh, v2.30.0's own replay (topic = $1 AND id > $2) was expensive: on (topic, published,
time, id), id > marker is checked inside the index but walks every entry of the topic, or the plan
scans the primary key from the marker across all topics. Replica-1 spent about one vCPU on it.
idx_message_topic_published_id (topic, published, id), created by hand on ntfy.sh on 2026-10-16,
made it ~50x cheaper in the lab. It is now created on every start (IF NOT EXISTS, no schema
version, so rollbacks to 2.30.0 still work).
With that index, the late-written half of the replay (time >= marker AND id < marker) picked it
and walked the topic's history (6 ms -> 104 ms in the lab). The replay is now split by time
instead: rows published in the marker's second or later (time index, from the marker), and rows
written after the marker but published before it (the new index, from the marker; time + 0 keeps
it off the time index). Same result as before, and both parts start at the marker.
The plan test now runs on prod-shaped data (200k rows, a few busy topics) with statistics, and
judges each scan by its index: a scan must start at the marker on that index, and replays must
stay within the topic. It fails on v2.31.0's query and on v2.30.0's without the index.
The replay query of v2.31.0 only read whole topics in the plan Postgres makes without the
parameter values, which it switches to after a prepared statement has run five times (pgx
prepares statements); with the values, it had found a cheaper plan. The plan tests now check that
generic plan. They found the same problem in the tail read (id > cursor OR id = ANY(holes)): its
generic plan scans the whole message table through the primary key, 700 ms on 3.3M rows in the
lab, on the primary every second. SQLite, which always plans without the values, scanned the whole
table for it too. Both are now two parts joined with UNION ALL.
The replay query of #2040 matched "id > marker OR (time >= marker time AND id <> marker)". The index
can check neither side of the OR, so Postgres read every row of the topic from the table and filtered
it afterwards; after the v2.31.0 upgrade, the reconnects of ntfy.sh's subscribers saturated a read
replica within a minute.
The two halves are now separate queries joined with UNION ALL (they cannot overlap: one has
id > marker, the other id < marker), and the index checks every condition of each: rows after
the marker as in v2.30.0, and late-written rows through a time range. SQLite gets an index on
(topic, published, time) for the time half, created on every start instead of in a migration, so
older versions still open the cache file. On ntfy.sh's busiest topic the replay now takes ~60 ms
(rows after the marker) and 0.2 ms (late-written rows), both from the index alone.
Cluster nodes write their batches independently, and late when one is
cut off from the database, so a message can be written after one that
was published later. A client whose marker is the late-written one never
got the later message: its row id is lower, and the node it reconnects
to delivered it before the client arrived, so the tail does not resend
it. since=<id> now also returns the rows published in the marker's
second or later with a lower row id. The marker's own second is replayed
as a consequence, which clients drop by id.
A marker that is not written yet (the client reconnected right after
receiving it) used to resolve to row 0 and replay the topic's entire
history. It is now looked up again for up to the batch timeout plus a
second first.
A subscriber's missed messages from one read now wait for those of the
previous read, so a later read cannot overtake an earlier one.
A read that fails part way no longer moves the cursor or clears holes, so
it is retried whole instead of losing the rows read before the failure.
A new tail records the gaps among the last rows as holes, so a row whose
id was handed out before startup but committed after it is not skipped.
Every node reads the rows written to the message cache once a second, in
write order (by row id), and sends each message to the local subscribers
that did not get it live: all of them if its fan-out to this node was
lost, or those that subscribed after it went out (they were reconnecting,
and their database read ran before the message was written).
Row ids are handed out on insert but become visible on commit, so a row
id the tail had to skip is re-checked until its row shows up or a minute
passes. In a cluster, since=<id> is read as a time a few seconds before
that message, since nodes write their batches independently.
The shared insert helper only returns the values of a message in a fixed
order. Each backend writes its own INSERT text as a constant: PostgreSQL
the unnest statement with the array casts, SQLite the multi-row INSERT
head and row placeholder. No column names or types travel with the values
and no renaming between backends.
One insertMessageValues function now yields column name, PostgreSQL type
and value side by side per row. The backends build their statements from
the row, so the separate column lists (and the order they had to share
with the values) are gone. SQLite keeps a one-entry rename for its user
column.
The batch writer built a different multi-row INSERT text for every batch
size, and pgx prepares each distinct text server-side and keeps up to 512
per connection. With pool connections now long-lived, every backend on
ntfy.sh accumulated hundreds of multi-MB plans: primary memory crept from
50% to 65% in three hours after 2.29.0. Locally, one backend grew from
20 MB to 1.15 GB after 1000 batch sizes and kept growing once the cache
was full.
The INSERT now takes one array parameter per column and unnests them, so
the statement text is the same for any batch size (32 MB after the same
workload). SQLite keeps the multi-row INSERT, chunked at 1000 rows. The
docs now recommend pool settings, including a connection lifetime.
SQLite gets a nop 15 -> 16 so the two backends keep the same version numbers;
the claim column is Postgres-only because SQLite is single-node by definition.
Also fixes a comment that no longer matched the code: claimMessagesDue stopped
marking rows published two commits ago.
Marking a claimed message published was wrong in the way a reviewer would
notice first: published is read by four other queries (since= replays, polls,
scheduled-message replacement, expiry), so a claimed-but-undelivered message
looked delivered to all of them, and a node dying after the claim lost the
push entirely.
The claim is now its own column. claimed_at holds the row off from the other
nodes, the claim statement is a single CTE that locks what it picks and skips
what another node is already claiming, and MarkPublished stays where it was,
after dispatch. So published still means delivered, and delivery is still at
least once.
The claim expires after two minutes, which is what makes a dead claimer
harmless: nothing would ever deliver its rows again if the claim were
permanent. Two minutes is far more than dispatching a claimed batch takes
(side effects are goroutine handoffs), and the claim also takes at most 1000
rows, so a backlog drains over several ticks instead of one node holding all
of it.
Schema 15 -> 16 adds the column; the migration test asserts the migrated
database is structurally identical to a fresh one, so the ALTER and the CREATE
TABLE cannot drift.
Tests: concurrent callers get disjoint sets, an abandoned claim is invisible
until it expires and claimable afterwards, and a published message never comes
back. Both halves of the reclaim test fail if the claim filter is dropped.
Every node's sender loop runs the same MessagesDue query, so once there is
more than one node, a scheduled message is delivered once per node. On
PostgreSQL the query is now SELECT ... FOR UPDATE SKIP LOCKED inside a
transaction, and the claimed rows are marked published in that transaction:
each row goes to exactly one caller and the others skip it instead of
blocking. SQLite and the in-memory store keep the old path.
This does change PostgreSQL semantics. A row used to be marked published
after dispatch returned, so a node dying mid-delivery re-sent on the next
tick; now it is marked before, so a node dying between claim and delivery
loses that message. The alternative is holding the row lock across Firebase,
web push, email and upstream delivery, where one stuck delivery would block
the claim path for every node. A narrow at-most-once window buys a
short-lived lock, and only for delayed messages.
The test runs six concurrent callers against 40 due messages and asserts each
was claimed exactly once; without the claim it reports "claimed 6 times".
One write path for both backends; the magic column count is gone. A column added to the list
but not to insertMessageArgs (or vice versa) now fails every message write on every backend.
On cron bursts (top of the hour), the single batch writer fell behind: it wrote one INSERT round
trip per message, so against a busy remote Postgres it managed ~100 rows/s while ~300/s arrived.
New messages took up to ~40s to reach the database (polls missed them), and publishes blocked on
the unbuffered queue for up to ~25s.
- Postgres: write each batch with one multi-row INSERT (up to 1000 rows per statement) instead of
one statement per row. SQLite keeps the row-by-row path.
- Skip DeleteScheduledBySequenceID when the client did not set a sequence ID: the sequence ID is
then the message's own fresh ID, so no scheduled message can match. This saves a transaction on
the primary for almost every publish.
- Buffer up to 100 batches in the BatchingQueue, so a briefly slow writer does not stall publishes.
- dbtest: add a latency proxy (CreateTestPostgresWithLatency, WithLatency) to test round trips.
The since marker was resolved by an inline subquery on the replica. A message
ID the replica had not seen yet (a client reconnecting right after receiving
a message, ahead of replication) resolved to 'id > 0' and replayed the topic's
whole retained history to that client.
Resolution is now its own lookup: on the replica, and only if the ID is
unknown there, on the primary. At most two point lookups before the replay,
and none on the primary unless a replica is configured. An ID unknown to the
primary as well (expired) keeps the full-replay fallback.
binwiederhier
Philipp C. Heckel