4690 Commits
Author SHA1 Message Date
binwiederhier beb3266db1 Document the Retry-After header on 429 responses 2026-11-01 11:47:56 +00:00
binwiederhier 5306dfaa97 Send a Retry-After header with rate limit 429 responses 2026-11-01 11:47:56 +00:00
binwiederhier b916f216fc Add limit.Service.Check, a bucket check that uses nothing but reports the wait 2026-11-01 11:47:51 +00:00
binwiederhier 6dc9abb3f7 Prune topic rows two weeks after their last write instead of 30 days
Half the memory per node; an abandoned UnifiedPush topic still has its row when a push 7+ days later decides on the 404, as long as it is pushed to at least every two weeks.
2026-11-01 11:25:02 +00:00
binwiederhier b65e642d05 Merge remote-tracking branch 'origin/main' into up-payer
# Conflicts:
#	docs/releases.md
2026-11-01 11:13:01 +00:00
Philipp C. Heckel e04b01af6d Merge pull request #2140 from binwiederhier/cluster-wasted
Count forwarded messages for topics without local subscribers as wasted
2026-10-31 15:44:59 -04:00
binwiederhier 859e6443c7 Count forwarded messages for topics without local subscribers as wasted, and do not keep those topics alive
A cluster message whose topic is known on the receiving node but has no subscriber there was published to nobody and uncounted, and its Keepalive kept the topic from ever expiring: a passive node held every topic the active one published to. It is now counted in ntfy_cluster_messages_wasted_total and not published, so idle topics expire after topicExpungeAfter.
2026-10-31 19:43:29 +00:00
binwiederhier 25f595f8e7 Drop a node's leftover usage of a finished period instead of adding it to the new one
With several nodes, counts one node admitted in the last seconds of a key's period could be written after another node moved the row to the new period, and were added to the new day. They are the old day's usage, so they are dropped: an undercount of at most one sync interval, never a double count.
2026-10-31 19:27:12 +00:00
binwiederhier abcfd88cd8 Rename topic.Topic to topic.State, so the live topic can take the name later
The package comment now says the server's live topics are meant to move into the topic package.
2026-10-31 18:29:29 +00:00
Philipp C. Heckel 2d55160690 Merge pull request #2139 from binwiederhier/release-v2360
Drop the admin socket directory from the Docker images
v2.36.0
2026-10-31 14:06:14 -04:00
binwiederhier cc590ca391 Drop the admin socket directory from the Docker images
The RUN step broke the armv7 release image (no QEMU on the release
runner), and the admin socket is off by default now, so the directory
is unused.
2026-10-31 17:54:47 +00:00
Philipp C. Heckel 0472c5055f Merge pull request #2138 from binwiederhier/release-v2360
Make the admin socket experimental, release notes for v2.36.0
2026-10-31 13:38:55 -04:00
binwiederhier 3938bc6886 Shorten the v2.36.0 release notes, install docs for v2.36.0 2026-10-31 15:02:51 +00:00
binwiederhier ea71722ba7 Release notes for v2.36.0 2026-10-31 15:00:21 +00:00
binwiederhier 2e8506f827 Make the admin socket experimental and off by default
Rename listen-admin-unix to experimental-listen-admin-unix and drop the
/run/ntfy/admin.sock default: the CLI side (#2089, #2135) is not merged,
so nothing uses the socket yet. The startup warning no longer points at a
CLI fallback, and the docs say the commands do not use it yet.
2026-10-31 15:00:21 +00:00
binwiederhier ef3514e42c Add the topic package to the Docker build 2026-10-31 14:37:16 +00:00
binwiederhier 31f2143a3a Document the rate visitor key, the 7-day Matrix rule and the tier refresh 2026-10-31 14:05:37 +00:00
binwiederhier ac9c6b2bac Keep the UnifiedPush rate visitors in the topic service, derive the limiter key at push time
The server uses the topic package instead of the limit service for the rate
visitors: a subscribe sets the rate visitor (user ID and masked IP), a poll,
a stream close and the manager's keepalive record subscriber activity, a
refused push notes an unknown topic (insert-only). The limiter key is derived
on every push from the rate visitor's current tier (refreshed in the
background every minute, never on the publish path), so a downgrade or a
deleted account charges the IP. Before a 404 for an abandoned topic the row
is read from the database. The Matrix gateway rejects the pushkey under the
same 7-day rule; its 12 h per-node rule is gone.
2026-10-31 14:05:37 +00:00
binwiederhier 14f631da8f Add Manager.UsersByIDs, a bulk user lookup for the rate visitor refresh 2026-10-31 13:51:38 +00:00
binwiederhier 6ae42e7590 Add the topic package: per-topic cluster-wide state synced through the topic table
One row per topic (first_seen_at, subscribed_at, rate visitor user and IP,
updated_at), merged by rule (earliest, latest, last written) so the nodes can
write in any order, synced by the package's own 5 s worker from a dirty set.
User schema 10 -> 11 now creates this table.
2026-10-31 13:51:38 +00:00
Philipp Heckel a11f090df3 Rename the UnifiedPush payer to rate visitor, and its table to topic
"Rate visitor" is the term the docs already use. The table is called topic so it can hold more per-topic, cluster-wide state later. seen_at keeps its name, since a noted row's seen_at is a refused push, not a subscription.
2026-10-31 13:05:33 +00:00
binwiederhier de3e51838c Forget a settled limiter key a minute after it is full again and unused
Prune now forgets a key's bucket entry once forgetting it changes nothing:
every bucket is full again, no bandwidth is tallied in the key's current
period, and it has not been used for a minute. Daily counts live in the
counters and are unaffected. Prune loses its idle parameter, since a bucket
without refill never holds a key.

Note when a bucket is full again after every refill, including denied calls
and Tokens, so a slower rule (a tier downgrade) cannot get the key forgotten
early. Tally only bandwidth outside the counters, and keep the entry while
that tally is nonzero, so the usage it reports survives; tallying requests
would keep every key for the rest of its period.
2026-10-31 11:37:13 +00:00
binwiederhier 1c72a97736 Document shared UnifiedPush payers and the 404 for abandoned topics 2026-10-31 11:31:32 +00:00
binwiederhier b552856219 Merge remote-tracking branch 'origin/main' into up-payer 2026-10-31 11:31:18 +00:00
binwiederhier 9c6733d06c Bill UnifiedPush pushes to the topic's payer on every node, answer 404 for topics abandoned for 7 days 2026-10-31 11:31:09 +00:00
binwiederhier ffd16e16b7 Credit Jonathan Kats for reporting the same-IP visitor race 2026-10-31 11:28:49 +00:00
Philipp C. Heckel cf20b764a9 Merge pull request #2134 from binwiederhier/adminsock-server
Admin socket: single-user lookup, safer socket file handling, Docker socket directory
2026-10-31 07:21:55 -04:00
binwiederhier 2b726f4ea7 Split readUsersAndGrants into readUserAndGrants and readAllUsersAndGrants 2026-10-31 11:19:48 +00:00
binwiederhier d3ca4b1126 Keep the payers of UnifiedPush topics in the limit service, synced through the topic_payer table 2026-10-31 11:17:02 +00:00
binwiederhier a9583d1d52 Merge origin/main, take main's limiter rework and drop the old settled-key pruning 2026-10-31 11:15:59 +00:00
binwiederhier 633d592095 Keep the admin socket in the startup 'Listening on' line 2026-10-31 11:13:01 +00:00
binwiederhier d4b5f383ed Add the topic_payer table to the user schema, migration 10 to 11 2026-10-31 11:12:46 +00:00
Philipp C. Heckel d934f53ffd Merge pull request #2131 from binwiederhier/shared-daily-limits
Shared daily limits: usage table, rolling 24h periods
2026-10-31 06:59:09 -04:00
binwiederhier b4d9df9807 Move the shard types to types.go, trim the daily limit docs to what admins need 2026-10-31 10:56:28 +00:00
binwiederhier fdc88a3163 Decide in the Postgres writer whether a failed chunk may have been written, drop the dialect hook 2026-10-31 10:34:52 +00:00
binwiederhier bfa0bfa333 Admin socket: single-user lookup, safer socket file handling, Docker socket directory
- GET /v1/users?username= returns one user, so the CLI does not download all users
- Never replace a non-socket file at listen-admin-unix, and say so when the socket belongs to another user's server
- Log when the admin socket is listening; default to /var/run/ntfy on the BSDs too
- Deleting an admin user over HTTP returns 40302 (admin socket only), like the other admin-only operations
- Docker images ship a /run/ntfy directory for the socket
2026-10-31 03:31:01 +00:00
binwiederhier a40af7e627 Count a Postgres write that timed out after it was sent as written, not twice
A chunk whose statement was sent but whose answer came after the deadline may have committed
(lab: 60 messages counted as 86-89 at 2.2-2.4 s to Postgres). It is now retired like a written
chunk; if it did not commit, its counts are lost, which fails open like an outage. Errors from
the server, errors before sending, and every SQLite error still retry.
2026-10-31 03:15:45 +00:00
binwiederhier 0f3a57a2a2 Merge remote-tracking branch 'origin/main' into shared-daily-limits
# Conflicts:
#	docs/releases.md
#	server/server.go
2026-10-31 03:14:26 +00:00
binwiederhier 398cc69b52 Test that a daily denial keeps the bucket's tokens and that read-back keys are not used here, add benchmarks
The benchmarks (400k request-only keys, 50k publishers) measure a publish, the sync scan and
memory per key; they were used to compare the split counter/bucket shards with one merged map
(publish ~10% faster merged, sync scan 50x slower, memory the same), so the shards stay split.
2026-10-31 03:07:26 +00:00
Philipp C. Heckel 14fd3de7bc Merge pull request #2133 from binwiederhier/db-up
Export ntfy_db_up and health-check the primary
2026-10-30 22:09:36 -04:00
binwiederhier 8e16b880c8 Close the cluster when the server stops
Server.stop never closed the cluster, so a stopped node stayed registered and its
heartbeat kept running against the closed database, logging from leaked goroutines.
In CI that raced with captureLog's unsynchronized buffer in a later test. captureLog
now uses syncBuffer as well.
2026-10-31 01:55:42 +00:00
binwiederhier 521ca530b5 Open the SQLite user database in the server and pass it to the user manager and the usage store
Like the Postgres pool: user.OpenSQLite opens the auth-file (case-sensitive LIKE),
NewSQLiteManager takes the handle and migrates it, and the usage store gets the same handle
instead of reaching into the manager (Manager.DB is gone). The usage table stays part of the
user schema.
2026-10-31 01:50:10 +00:00
binwiederhier 9ecbf4dc16 Export ntfy_db_up and health-check the primary like the replicas
A node cut off from its database only showed in the logs. The health check loop now always
runs and pings the primary too; ntfy_db_up (1/0, per pool, next to ntfy_db_pool_*) is the
result of the last check, so an alert can fire on ntfy_db_up{role="primary"} == 0.
2026-10-31 01:31:13 +00:00
binwiederhier 6c32bee44e Use the node's clock for periods and updated_at, require synced clocks
The service no longer reads the database clock: no clock offset, no clock query per sync
round. Periods, the read-back watermark and the rows' updated_at all use the node's clock, so
the database's clock does not matter; the app nodes must run NTP (documented in config.md).
2026-10-31 01:16:50 +00:00
Philipp C. Heckel c62439f8fa Merge pull request #2088 from binwiederhier/adminsock-socket
Admin socket: serve the admin API on a local unix socket
2026-10-30 20:46:49 -04:00
binwiederhier 84861a015c Docs: show the admin socket startup warning instead of a CLI message that does not exist 2026-10-31 00:44:26 +00:00
binwiederhier a2656890d1 Drop the date from the admin socket startup warning 2026-10-31 00:42:49 +00:00
binwiederhier 6b4eca0654 Warn if the admin socket directory lets other users use or replace the socket 2026-10-31 00:37:42 +00:00
Philipp C. Heckel e7c534c06c Merge pull request #1710 from bekoeppel/fix/wait-pid-eperm-and-pidof-docs
Fix --wait-pid for processes owned by other users
2026-10-30 19:53:22 -04:00
binwiederhier 6cb87c31b4 Rename adminSocketListener to adminUnixListener 2026-10-30 23:51:28 +00:00