Release notes: main moved the v2.32.0 notes to the released section, so this PR's bullet goes under v2.33.0's Bug fixes + maintenance. The test's one-line t.Cleanup literals are spread over lines.
Migrate took the target version as a separate argument, kept in step with the steps by hand:
adding a step without bumping the version constant silently skipped it on existing installs,
while fresh installs got the new tables. The target is now the version after the last step (1
without steps), and the steps must be contiguous, so a forgotten one fails at startup. The
per-store version constants are gone.
The stored total was overwritten with one node's in-memory count, so only the leader could
write it (#2066) and publishes on other nodes were never counted. Every node now adds the
publishes since its last flush to the stored total, reads the total back, and moves its flush
marker to it, so peer counts are never added twice. The total loaded at startup is the first
marker, so a restart does not count it again; a single node counts exactly as before.
Instead of on every start: the schema is defined in one place again. Postgres gets
idx_message_topic_published_id (topic, published, id), SQLite idx_topic_published_time (topic,
published, time), both with IF NOT EXISTS (ntfy.sh got the Postgres one by hand). A downgrade to
v2.31.0 or earlier now needs the schema version set back by hand, as described in the release notes;
from v2.31.1 on, a newer schema only logs a warning (#2059).
On ntfy.sh, v2.30.0's own replay (topic = $1 AND id > $2) was expensive: on (topic, published,
time, id), id > marker is checked inside the index but walks every entry of the topic, or the plan
scans the primary key from the marker across all topics. Replica-1 spent about one vCPU on it.
idx_message_topic_published_id (topic, published, id), created by hand on ntfy.sh on 2026-10-16,
made it ~50x cheaper in the lab. It is now created on every start (IF NOT EXISTS, no schema
version, so rollbacks to 2.30.0 still work).
With that index, the late-written half of the replay (time >= marker AND id < marker) picked it
and walked the topic's history (6 ms -> 104 ms in the lab). The replay is now split by time
instead: rows published in the marker's second or later (time index, from the marker), and rows
written after the marker but published before it (the new index, from the marker; time + 0 keeps
it off the time index). Same result as before, and both parts start at the marker.
The plan test now runs on prod-shaped data (200k rows, a few busy topics) with statistics, and
judges each scan by its index: a scan must start at the marker on that index, and replays must
stay within the topic. It fails on v2.31.0's query and on v2.30.0's without the index.
The replay query of v2.31.0 only read whole topics in the plan Postgres makes without the
parameter values, which it switches to after a prepared statement has run five times (pgx
prepares statements); with the values, it had found a cheaper plan. The plan tests now check that
generic plan. They found the same problem in the tail read (id > cursor OR id = ANY(holes)): its
generic plan scans the whole message table through the primary key, 700 ms on 3.3M rows in the
lab, on the primary every second. SQLite, which always plans without the values, scanned the whole
table for it too. Both are now two parts joined with UNION ALL.
The replay query of #2040 matched "id > marker OR (time >= marker time AND id <> marker)". The index
can check neither side of the OR, so Postgres read every row of the topic from the table and filtered
it afterwards; after the v2.31.0 upgrade, the reconnects of ntfy.sh's subscribers saturated a read
replica within a minute.
The two halves are now separate queries joined with UNION ALL (they cannot overlap: one has
id > marker, the other id < marker), and the index checks every condition of each: rows after
the marker as in v2.30.0, and late-written rows through a time range. SQLite gets an index on
(topic, published, time) for the time half, created on every start instead of in a migration, so
older versions still open the cache file. On ntfy.sh's busiest topic the replay now takes ~60 ms
(rows after the marker) and 0.2 ms (late-written rows), both from the index alone.
Cluster nodes write their batches independently, and late when one is
cut off from the database, so a message can be written after one that
was published later. A client whose marker is the late-written one never
got the later message: its row id is lower, and the node it reconnects
to delivered it before the client arrived, so the tail does not resend
it. since=<id> now also returns the rows published in the marker's
second or later with a lower row id. The marker's own second is replayed
as a consequence, which clients drop by id.
A marker that is not written yet (the client reconnected right after
receiving it) used to resolve to row 0 and replay the topic's entire
history. It is now looked up again for up to the batch timeout plus a
second first.
A subscriber's missed messages from one read now wait for those of the
previous read, so a later read cannot overtake an earlier one.
A read that fails part way no longer moves the cursor or clears holes, so
it is retried whole instead of losing the rows read before the failure.
A new tail records the gaps among the last rows as holes, so a row whose
id was handed out before startup but committed after it is not skipped.
Every node reads the rows written to the message cache once a second, in
write order (by row id), and sends each message to the local subscribers
that did not get it live: all of them if its fan-out to this node was
lost, or those that subscribed after it went out (they were reconnecting,
and their database read ran before the message was written).
Row ids are handed out on insert but become visible on commit, so a row
id the tail had to skip is re-checked until its row shows up or a minute
passes. In a cluster, since=<id> is read as a time a few seconds before
that message, since nodes write their batches independently.
The shared insert helper only returns the values of a message in a fixed
order. Each backend writes its own INSERT text as a constant: PostgreSQL
the unnest statement with the array casts, SQLite the multi-row INSERT
head and row placeholder. No column names or types travel with the values
and no renaming between backends.
One insertMessageValues function now yields column name, PostgreSQL type
and value side by side per row. The backends build their statements from
the row, so the separate column lists (and the order they had to share
with the values) are gone. SQLite keeps a one-entry rename for its user
column.
The batch writer built a different multi-row INSERT text for every batch
size, and pgx prepares each distinct text server-side and keeps up to 512
per connection. With pool connections now long-lived, every backend on
ntfy.sh accumulated hundreds of multi-MB plans: primary memory crept from
50% to 65% in three hours after 2.29.0. Locally, one backend grew from
20 MB to 1.15 GB after 1000 batch sizes and kept growing once the cache
was full.
The INSERT now takes one array parameter per column and unnests them, so
the statement text is the same for any batch size (32 MB after the same
workload). SQLite keeps the multi-row INSERT, chunked at 1000 rows. The
docs now recommend pool settings, including a connection lifetime.
One write path for both backends; the magic column count is gone. A column added to the list
but not to insertMessageArgs (or vice versa) now fails every message write on every backend.
The since marker was resolved by an inline subquery on the replica. A message
ID the replica had not seen yet (a client reconnecting right after receiving
a message, ahead of replication) resolved to 'id > 0' and replayed the topic's
whole retained history to that client.
Resolution is now its own lookup: on the replica, and only if the ID is
unknown there, on the primary. At most two point lookups before the replay,
and none on the primary unless a replica is configured. An ID unknown to the
primary as well (expired) keeps the full-replay fallback.
binwiederhier