Commit Graph
4652 Commits
Author SHA1 Message Date
binwiederhier 858af8c6ad Write a topic's subscriber activity every one to two hours instead of every 30 to 60 minutes
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
2026-11-02 17:59:00 +00:00
binwiederhier 3000245ad6 Link the PR in the v2.37.1 release note
Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
2026-11-02 17:52:02 +00:00
binwiederhier 799d13f6d8 Write a UnifiedPush topic's subscriber activity about once an hour, spread over time
v2.37.0 wrote every topic with an open stream every 5 minutes, and the topics touched together after a restart stayed in step, so ntfy.sh wrote ~45k topic rows in the same minute every 5 minutes (~145 rows/s on average, more than the message inserts), stalling the primary. Each topic now has its own interval between 30 and 60 minutes (by hash), and a subscribe or poll that only repeats the same rate visitor is a touch.

Claude-Session: https://claude.ai/code/session_01DeXuhSjq5Gta5qPJgSceFe
2026-11-02 17:51:41 +00:00
Philipp C. Heckel ad162eea0a Merge pull request #2148 from binwiederhier/release-v2370
Release notes and install docs for v2.37.0
v2.37.0
2026-11-02 08:13:53 -05:00
binwiederhier 51d23183d8 Link the PRs in the v2.37.0 release notes 2026-11-02 01:45:36 +00:00
binwiederhier 400f39505e Release notes for v2.37.0, install docs for v2.37.0 2026-11-02 01:31:13 +00:00
Philipp C. Heckel 5a52d476f7 Merge pull request #2136 from binwiederhier/up-payer
UnifiedPush across nodes: shared rate visitors in a new topic table, 410 Gone for abandoned topics
2026-11-01 20:30:15 -05:00
binwiederhier c795ba4fd7 server.yml: rewrap the clustering note 2026-11-02 01:26:27 +00:00
binwiederhier 666c63700c server.yml: the tail recovers messages a node misses 2026-11-02 01:26:16 +00:00
binwiederhier 750e5f49c6 Bring the example server.yml up to date: ban defaults, database-replica-urls, shared daily limits
ban-window defaults to 10m and ban-weights to 42909:2, 50701:0.05 and 41002:0.05 (weights may be fractional); database-replica-urls was missing; the cluster note now says daily limits are shared.
2026-11-02 01:25:18 +00:00
binwiederhier 4f1f62b0b5 Rename RecordRefused to Refused 2026-11-02 01:23:04 +00:00
binwiederhier 7250c3d474 Weigh the abandoned-topic 410 like the UnifiedPush 507 in the ban feed (0.05) 2026-11-02 01:15:26 +00:00
binwiederhier bfb257b3eb Merge remote-tracking branch 'origin/main' into up-payer
# Conflicts:
#	cmd/serve_test.go
#	docs/releases.md
2026-11-02 01:15:09 +00:00
Philipp C. Heckel 32e0a992e4 Merge pull request #2147 from binwiederhier/ban-weights
Fractional ban weights; weigh auth failures 2 and UnifiedPush 507s 0.05 by default
2026-11-01 20:14:34 -05:00
binwiederhier 9ccddc7b10 Cap ban weights and the threshold, so counting in thousandths cannot overflow
A weight like 1e16 overflowed to a negative token count, which the limiter always allows, so the code never banned. Weights above 1,000,000 and thresholds above 1,000,000,000 are now rejected at startup.
2026-11-02 01:13:18 +00:00
binwiederhier 093012dc8d Count ban strikes in thousandths, so weights go down to 0.001 2026-11-02 01:11:46 +00:00
binwiederhier b9f9c562a5 Allow fractional ban weights, weigh auth failures 2 and UnifiedPush 507s 0.05 by default
Strikes are counted in hundredths, so a weight like 0.05 makes 20 rejections one strike. An app server pushing to many offline phones gets many 507s without being an attacker; auth-failure floods still ban at half the threshold.
2026-11-02 01:05:56 +00:00
binwiederhier 15c924fd3b Rename Note to RecordRefused, and its counter and cap to match 2026-11-02 00:52:26 +00:00
binwiederhier 01047d68b7 Keep the topic service's pending rows and the store's row slices as pointers 2026-11-02 00:47:09 +00:00
Philipp C. Heckel 8f29076f63 Merge pull request #2146 from binwiederhier/ci-speed
Run CI's checks and Go tests as parallel jobs, without coverage
2026-11-01 19:02:34 -05:00
binwiederhier d6b510dc46 Run CI's checks, the server tests and the other Go tests as three parallel jobs, without coverage
The server package's race run takes about as long as all other packages together, so it gets its own job. Coverage was only printed to the log, and its instrumentation slowed the race run.
2026-11-01 23:54:10 +00:00
Philipp C. Heckel c3e6b542b5 Merge pull request #2145 from binwiederhier/ci-speed
Run the Go tests once in CI, with the race detector
2026-11-01 18:50:47 -05:00
binwiederhier feead71416 Run the Go tests once in CI, with the race detector, instead of twice
make checkv ran every Go test without -race, then make coverage ran them all again with -race and coverage. The check step now runs only the web tests, formatting, vetting and linting.
2026-11-01 23:39:16 +00:00
binwiederhier 1a330f72e0 Capture the serve warning into a mutex-guarded buffer, so leaked servers' logs cannot race it
Earlier cmd tests leave servers running whose sync workers keep logging into the global log output; the test read a plain bytes.Buffer while they wrote to it.
2026-11-01 22:58:55 +00:00
binwiederhier 5146f3d929 Merge remote-tracking branch 'origin/main' into up-payer
# Conflicts:
#	docs/releases.md
#	server/server_test.go
#	server/visitor.go
#	server/visitor_test.go
2026-11-01 22:42:26 +00:00
Philipp C. Heckel daf15641ed Merge pull request #2126 from binwiederhier/limit-prune-full
Forget settled limiter keys after a minute instead of a day
2026-11-01 17:32:10 -05:00
binwiederhier 2c91effa30 Remove the per-period bandwidth tally from the limit service
Nothing outside the limit package read it: enforcement is the bandwidth
token bucket, and stats only report messages, emails and calls. Without
the tally, a key is forgotten once all of its buckets are full again and
it was idle for settledIdle, like a key that used no bandwidth. Usage now
holds only the daily counts.
2026-11-01 22:05:26 +00:00
binwiederhier 5bd36bdcd9 Merge remote-tracking branch 'origin/main' into up-payer
# Conflicts:
#	server/errors.go
2026-11-01 20:30:27 +00:00
binwiederhier 6c4663cc0a Note when a bucket is full again on Check too, so a slower rule there holds the key 2026-11-01 20:30:22 +00:00
binwiederhier 7aa6908147 Merge origin/main, keep the Retry-After and RateLimit decisions with settled key forgetting 2026-11-01 20:29:52 +00:00
Philipp C. Heckel 9555ca6206 Merge pull request #2143 from binwiederhier/ratelimit-headers
Send IETF draft RateLimit-Policy and RateLimit headers on publish responses
2026-11-01 15:17:00 -05:00
binwiederhier 3a3e4217b9 Merge Keepalive into Touch, throttled to once per 5 minutes
The manager touches the topics of open streams every minute; subscribes, polls and closing streams use the same throttle, which shifts subscriber activity by at most 5 minutes.
2026-11-01 20:07:17 +00:00
binwiederhier bc53ed8ec9 Name the header each Decision field is sent as 2026-11-01 20:03:19 +00:00
binwiederhier 4a2f5e91bd Comment the rate visitor loop 2026-11-01 19:44:46 +00:00
binwiederhier 4a5bb66f37 Inline setRateVisitors into its only caller 2026-11-01 19:43:17 +00:00
binwiederhier c3ceac411e Set the rate visitors in one place, with or without access control 2026-11-01 19:39:53 +00:00
binwiederhier 372c167ce7 Export MaskedIP like the visitor's other accessors 2026-11-01 19:35:34 +00:00
binwiederhier e6b97f524a Pass the store to limit.New and topic.New instead of a one-field Config 2026-11-01 18:29:46 +00:00
binwiederhier 805537ea2e Expunge idle topics from memory after 6 hours instead of 16 2026-11-01 18:24:29 +00:00
binwiederhier 0ff3e571b1 Move maskedIP to visitor.go 2026-11-01 17:51:37 +00:00
binwiederhier d6bc42c6f6 Rename rate_visitor.go to server_rate_visitor.go, like the other server files 2026-11-01 17:49:15 +00:00
binwiederhier c8bbddb647 Rewrap the subscriber-based rate limiting paragraph in the docs 2026-11-01 17:39:18 +00:00
binwiederhier c56f4fb529 Answer 410 after 4 days without a subscriber, and prune topic rows after 7 days 2026-11-01 17:14:11 +00:00
binwiederhier 70a992aba5 Clarify that the cap on topics known from refused pushes is cluster-wide 2026-11-01 17:11:46 +00:00
binwiederhier 943dec84f1 Serve if the topics cannot be loaded at startup, and leave the load to the sync worker 2026-11-01 17:11:46 +00:00
binwiederhier c56a9f9f73 Do not write a stale rate visitor with a touch or keepalive, so a newer one from another node wins 2026-11-01 17:11:07 +00:00
binwiederhier fd9a0747cb Shorten the topic table comments 2026-11-01 15:16:30 +00:00
binwiederhier 582cae9f4c Simplify the topic service: one pending-rows map, one flat State row, a synchronous startup load
A refused push's Note returns the topic's state for the publish path, prune forgets unwritten
changes along with the topic, and SetForTest is gone: tests age rows in the database.
2026-11-01 15:16:30 +00:00
binwiederhier 312e036ec9 Do not mark topics to be written without a store, so the dirty set cannot grow forever 2026-11-01 14:59:24 +00:00
binwiederhier 3fb5b53086 Look up the rate visitor's user on each push, like an authenticated publish does
Drops the background refresh, its cache and Manager.UsersByIDs: one primary key read per UnifiedPush push to a topic whose subscriber has an account (none for anonymous subscribers), and a tier change applies to the next message.
2026-11-01 14:35:10 +00:00