near_dup_demote compares each candidate with every earlier
non-duplicate by Jaccard, quadratic when candidates differ. Its pool
grows with limit (about max(3*limit, 30) plus keyword hits), so a
limit-200 page could cost ~180k comparisons. It now runs only when the
caller's limit is below MMR_MAX_TOP_K, like MMR; large pages want every
relevant item, similar ones included. The read reports
near_dup_applied next to mmr_applied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
alvinfeiandClaude Opus 5.5
e540386db1