op.Get dropped the error from its fallback listing, so a transient upstream
failure on any driver without a direct Get read as ObjectNotFound. WebDAV then
answered 404 and rclone reported "directory not found" for a folder that exists.
Return the listing error, and answer 500 from WebDAV PROPFIND and GET/HEAD when
the lookup failed for any reason other than not found.
- Match the UNIQUE constraint text anywhere in the SQLite error message.
- Add regression tests for the SSO auto-registration username retry.
- Verify the fix with targeted SSO tests and go vet.
Co-authored-by: LunelleEverdawn <lunelle+suyunmeng@oplist.org>
- Preserve multipart chunk errors to prevent nil upload results and downstream panics.
- Cancel and wait for multipart workers on failure, and attempt to abort OSS uploads with the current security token.
- Map WebDAV upload errors to appropriate HTTP status codes: 403 for permission failures, 405 for unsupported uploads, and 500 for other backend failures.
Fixes#3201
Co-authored-by: LunelleEverdawn <lunelle+suyunmeng@oplist.org>
`SeekableStream.generateReader` previously opened the upstream stream only once. If the connection closed before all bytes were read, caching failed with `unexpected EOF` and aborted the operation.
Add `NewResumableReader` to reopen full-stream reads from the current byte offset after a read or resume-request error. Retries use linear backoff of 200 ms per attempt and are capped at five per read; an initial open error is returned directly. For known positive sizes, the reader stops at the declared length so a trailing upstream error after all bytes arrive does not fail the operation. Unknown or non-positive sizes retain the original unbounded behavior.
Use the resumable reader for seekable stream caching, offline transfer uploads, and torrent generation. Resumption requires Range support, an accurate declared size, and unchanged content during the read.
Add tests covering truncated HTTP responses, retry limits, and size boundaries.
* fix(seed): keep CAS hash matrix for 189 rapid upload
CAS seeds only carry MD5 + slice MD5s, but the quick-save path rebuilt the rapid-upload request through a torrent container, which requires SHA-1 piece hashes. Every 189 秒传 therefore failed before any request was sent and reported 'target driver cannot reuse the available hashes'.
- build the rapid-upload request directly from the seed file, keeping the whole MD5, slice MD5s, the drive's slice size and the legacy slice digest
- let 189pc run initMultiUpload/checkTransSecond/commitMultiUploadFile when slice hashes are available, falling back to the legacy whole-MD5 upload
- surface the real rapid-upload error instead of the generic message
* fix(189pc): surface daily flow limit instead of generic rapid-upload error
The CAS path now reaches 189's upload API, and the first real rejection is UserDayFlowOverLimited (account daily upload flow exhausted). Treat it as a terminal account state instead of retrying:
- add errs.DailyFlowLimitExceeded and map UserDayFlowOverLimited / 'flow is out' onto it (189 returns it as a business code inside HTTP 200)
- skip the legacy whole-MD5 fallback once the account is limited
- report 'rapid upload failed: ...' with the real reason and mark the rest of the batch unavailable instead of one request per file
* fix(seed): refuse to generate a seed from a seed container
Generating a .cas seed from a .cas file produced an artifact with exactly the same name in the same folder, so PutDirectly overwrote the original seed with a seed-of-seed: the payload then described the few-hundred-byte metadata file instead of the video. Rapid upload of that seed 'succeeds' and only creates another .cas, which looks like CAS 秒传 doing nothing.
- skip .oss/.torrent/.cas/.cas.torrent when expanding a directory
- reject an explicit selection of a seed container with an actionable message
- refuse to write an artifact over one of its own source files
* refactor(cas): derive slice size from file size and store the final digest
CAS 判重只比较最终分片摘要（逐片 MD5 汇总后再取 MD5），而切法由文件大小固定：10MiB / 20MiB / 更大文件按 1999 片上限向上取 10MiB 的整数倍。之前把 sliceMd5 当成「可变分片的哈希」、总是从逐片列表现算，切法一变就对不上云端。
- add torrent.CASSliceSize / CASPartCount / CASUniformSliceSize as the single source of truth; 189pc.partSize delegates to it
- .cas writes the final digest in sliceMd5 and the per-slice list in partsMd5; slice_md5s stays readable for old files
- rapid upload prefers the explicit sliceMd5 and only derives it from parts when absent
- generation, recalculate and inline conversion hash CAS files at CASSliceSize per file and drop parts that do not match that slicing
- piece_size no longer influences CAS hashing
* fix(seed): refuse to rapid-upload a seed that describes itself
A .cas generated from another .cas documents the metadata file itself (name ends with .cas, a few hundred bytes). Rapid-uploading it just creates another same-named .cas in the target folder, which looks like CAS 秒传 doing nothing. Reject that case with an actionable message instead.
* test(seed): lock the rapid-upload name and the seed-of-seed guard
- extract seedRapidUploadBlocker so the guard is testable
- add an end-to-end test proving a legacy 5-field .cas resolves to the video name, size, slice size and final slice MD5
- add a test that seed containers are refused for rapid upload
* refactor(seed): output seed and torrent messages in English
Turn the error strings, task statuses and progress labels of the transfer-seed / torrent feature into English, including the 189 CAS rapid-upload path and the front-end facing guards. Comments and unrelated log lines are untouched.
* refactor(cas): make the CAS slice rule a verbatim port of the 189 upload rule
CAS slice size must be exactly what 189 uses when uploading, otherwise the sliceMd5 submitted for rapid upload can never match the digest the drive stored.
- CASSliceSize now copies 189pc partSize expression by expression (same DEFAULT constant, same float expression, same 5x lower bound)
- expectedCASPartCount mirrors fastUpload's count: one slice when size <= sliceSize, otherwise ceil(size/sliceSize)
- add cross-package tests comparing CASSliceSize/CASPartCount against a verbatim copy of the 189 rule, and partSize against CASSliceSize
* fix(seed): address CAS sidecar and digest review findings
P1 FsStream hashed uploads with seedPieceSize() while the sidecar declared the CAS slice size; unknown-size CAS uploads now fail before writing the file instead of after. P1 buildCASFileEntry only compared piece counts, so 8MiB slices could be reused as 10MiB CAS slices; it now requires the slicing to match. P2 multi-slice payloads without sliceMd5 are rejected instead of falling back to the whole-file MD5, and oversized CAS slice sizes are clamped for the seed container only.
---------
Co-authored-by: pikachuren <pikachuren@example.com>
* fix(storage): decouple details probe from request context and isolate refresh/lifecycle state
- Run storage capacity probes with context.WithoutCancel bounded by
storage_details_timeout_seconds (default 15s, clamped to 1..600s) so
a 1s listing wait or 3s admin wait cancellation no longer aborts the
shared probe before it can populate the cache; convert driver panics
to errors and return at the deadline even if a driver ignores ctx.
- Keep at most one active probe per mount until GetDetails actually
exits, publishing the result and releasing the slot atomically.
Forced refresh invalidates cached details and joins any in-flight
probe without spawning duplicate remote requests.
- Track failed probes in private detailsState.failed without polluting
detailCache with nil: ordinary refreshes skip re-probing failed mounts
immediately (preventing repeated 1s stalls), while forced refreshes or
storage lifecycle changes reset the failure state.
- Guard init, update, disable, and delete with per-mount generations and
changing barriers so stale or retired driver instances cannot publish
outdated capacity into the cache, without mutating storage status.
- Key details cache entries by full cleaned mount path so balanced
storages (/a, /a.balance1) do not overwrite each other's capacity.
Closes#1815
Relates to #2633, #2635
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Gemini <noreply@google.com>
* fix(storage): release details task slot on lifecycle invalidation
---------
Co-authored-by: Vurliy <ulermi@outlook.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Gemini <noreply@google.com>
Co-authored-by: Codex <codex@localhost>
* feat(media): scan every folder level, save incrementally, TV tags, seed task progress
Media library
- Merge mode now walks every folder level instead of only the first one.
A folder that directly contains media files is one collection and is not
split further; a folder without media is descended into. Loose files in
the scan root stay individual items. Depth is capped at 16.
- Scanning no longer buffers the whole tree before writing: items are
flushed to the database every 20 items or 3 seconds, for all four media
types, so results appear while the scan is still running. Skipped and
failed items still advance the progress counter.
- The scan path type tag now decides movie vs. series. Tags such as
TV series / anime / variety / documentary mark every video under the
path as "tv" (shown as series in Emby, also for single files); movie
tags force "movie" even when merge mode is on. Unrecognised tags keep
the old rule (folder items are series). Emby virtual folders use the
same rule.
- Fix: a music folder containing a non-audio file was dropped entirely.
Playback URL
- When no site_url is configured and the request host is a loopback
address (127.0.0.1 / localhost / ::1), stream_url now returns relative
/d and /p paths instead of absolute loopback URLs. An explicit site_url
or a forwarded host still produces absolute URLs.
Seed tasks
- Seed generation reports progress by bytes hashed plus a stage message
and is exposed at /task/seed_generate.
- Seed save, rapid upload and transit transfer register a display-only
task at /task/seed_save (the operation stays synchronous) so progress
is visible in the task list.
Tests cover tag classification, index-length checks, level-by-level merge
scanning (stop at collection, list errors, depth limit) and loopback
detection.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(server): keep site_url path prefix in playback base url
site_url 允许配置成 /openlist 这样的路径前缀；原先只有以 http 开头才走绝对地址，前缀模式下播放地址会丢掉前缀导致 404。改为配置了 site_url 就统一用 GetApiUrlFromRequest 生成。
* feat(media): season-aware episodes, album admin api, cover upload, zip export
- 扫描：季目录（第N季/Season N/S01）解析季号写入 EpisodeInfo.season，Emby 与播放器可按季分组；显式电影标签不再被季目录反推成剧集；merge_depth 语义固定为「第 N 层=一条记录」，与模型注释、UI 一致
- 刮削：音乐条目的文件自带封面（data:/thumb:/站内地址）不再被在线封面覆盖
- 后台新增：封面 URL/上传（/admin/media/items/cover）、专辑聚合列表与批量编辑（/admin/media/albums、/album/items、/album/update）、条目可编辑字段扩充（选集/视频类型/季集/时长/外部ID）
- 导入导出升级为 zip（media_export.json + covers/），导入兼容旧 JSON
- Emby：多季合集按每集 season 拆分 Season，集 ID 带季号避免跨季重号
- fix(seed)：中转转存部分文件失败时任务按失败收尾，不再一律标记成功
* fix(media): patch semantics for item update, transactional album update, concurrent cover export
- UpdateMediaItemAdmin 改为 PATCH 语义（字段全部指针，只写出现过的键），修复只传 hidden 也会清空 cover/plot 的问题；cover 增加 12MB 上限
- 专辑批量更新走单事务（新增 db.UpdateMediaItems），失败整批回滚；旧封面文件在事务成功后才删除
- 导出封面改为并发读取（4 并发，单张 8MB / 总量 512MB / 5000 张上限），避免大库导出逐张串行
* fix(media,seed): unify episode ids, season fallback, cancellable seed tasks, periodic flush
- Emby 集 ID 统一走 parseEpisodeID（新格式 ep-item-season-num，兼容旧两段），修掉封面/刷新/续播列表里仍假设旧格式的三处；续播能唯一定位时补季号
- 季号兜底 seasonFromText 与前端 episodeSeasonOf 规则对齐，老数据不会在 Emby 和网页端分到不同季
- 种子任务真正可取消：任务 ctx 重新绑定（tache Add 会覆盖）、保存 IO 改用任务 ctx、逐文件与哈希读取检查取消、取消后状态为 Canceled
- mediaSaver 增加定时落盘协程，Close 时停表并收尾 Flush
* feat(media): artist aggregation api and image exif lookup
- 新增 GET /fs/media/artists（db.ListArtists）：按 album_artist 聚合专辑数/曲目数/封面，供音乐库「按艺术家」分类
- /fs/media/albums 支持 album_artist 过滤，艺术家下钻时只取该艺术家的专辑
- 新增 GET /fs/media/exif：按需读取图片文件头解析 EXIF（设备/镜头/曝光/焦距/时间/GPS）并缓存，解析失败返回 available=false 而不是 500
* fix(emby): return full UserDto from /Users/Public (issue #3301)
严格类型的客户端会把 user['Policy'] / user['Configuration'] 直接强转成 Map，
之前 /Users/Public 只返回 5 个标量字段，转 null 时抛异常并中断渲染（封面全挂）。
- 删除 embyPublicUserDto，/Users/Public 直接复用 embyUserDtoOf，与登录接口结构一致
- 新增 embyUserConfiguration()，按官方 UserConfiguration 给全默认值（空 map 同样是缺键）
- 补 TestEmbyUserDtoHasPolicyAndConfiguration 回归测试
* feat(media): pure-go pdf cover rendering, keep existing cover by default, same-dir thumbnails
- PDF 封面改用纯 Go 光栅化（github.com/go-pdfkit/render，BSD-3、无 cgo）：渲染第 1 页再缩到 300px，矢量/扫描件/Flate 压缩图都能出图；失败时仍退回原来的内嵌 JPEG 扫描。渲染带 MaxPixels/MaxDuration 上限，防止恶意或超大页面拖住扫描
- 新增设置 media_overwrite_cover（默认关闭）：刮削不再覆盖条目已有封面，只负责补空；需要替换时手动打开。原先只对音乐做了保护，现在统一到所有媒体类型
- 缩略图存储新增 samedir 模式：封面写到媒体文件同目录，文件名为「原文件名.图片后缀」（song.flac.jpg），不覆盖原文件，也不需要配置 VFS 目录
- 依赖代价：go 指令 1.26.0 -> 1.26.4，新增 render 及其纯 Go 依赖（go-gfx/opentype 等），go.sum 由 tidy 收敛
* feat(media): add skip-file-metadata switch (default off)
- 新设置 media_skip_file_metadata（默认关闭）：开启后扫描与刮削都不读取媒体文件内容
- 音乐：扫描与刮削均跳过 ID3/Vorbis 解析，退回目录名/文件名，内嵌封面与内嵌歌词一并跳过
- 书籍：刮削时跳过 EPUB/PDF 内嵌封面提取
- 图片：刮削时跳过 EXIF（拍摄时间/地点/设备），缩略图仍生成
- 书籍封面依旧只在刮削阶段提取，扫描阶段完全不读文件
* feat(seed): support folder expansion, split mode and structure-preserving paths
- 右键目录也能生成种子：后端递归展开目录下所有文件（深度/数量上限，逐目录校验权限）
- 新增 preserve_structure（默认关闭）：开启后种子内保留相对目录结构，关闭时拍平成文件名，重名自动加序号
- 新增 split（默认关闭）：每个文件生成一个种子；关闭时全部合并（默认）
- 默认命名：单文件/单目录用其名字；多选合并固定为 torrent
- 能力探测接口同步支持目录，前端按展开后的文件列表展示哈希/流量
- 单测覆盖命名规则、保留/拍平路径与重名处理
* feat(seed): compute missing hashes from source files when converting
- ConvertSeed 新增 compute_missing / file_base_path：缺哈希时按种子内相对路径读取源目录文件补算，而不是直接返回不可转换
- torrent 目标按整包拼接计算 SHA-1 pieces（解决 piece boundary crosses the next file），CAS 目标按 10 MiB 分片补 MD5
- 补算后仍不满足时返回 400 并带具体原因，避免前端误判成功
* feat(seed): persist overall comment when converting
- /fs/seed/convert 新增 comment 字段（指针，允许清空）
- 转换与保存注释合并为一次请求，前端不必先保存再转换
* feat(seed): convert to multiple target formats in one request
- /fs/seed/convert 新增 formats 数组（去重、bt 别名、最多 3 个），逐个编码并写入目标目录，返回 artifacts
- 每个目标独立解码，避免 CAS 补算修改 PieceSize 后污染 torrent pieces
- 预检仍保留「不可转换 + 原因」的响应；单目标响应字段保持兼容
- 新增 TestSeedConvertTargets
* fix(seed): identity conversion is always feasible and folder roots no longer nest twice
- seedConversionStates 现在接收源格式：目标等于源格式时直接可行，不再用「按文件 pieces」去诊断整包结构。合法 torrent（pieces 跨文件）之前被误报 piece boundary crosses the next file，导致预览页对当前格式打叉\n- ConvertSeed 对同一格式直接复制原始字节，跳过诊断与重新编码\n- 保留目录结构时，只选一个目录以该目录为根，修复「剧/剧/S01/...」重复嵌套\n- 新增回归测试：身份转换可行、跨格式仍诊断、单目录/多目录保留结构路径
* fix(seed): keep the torrent's global pieces so it can be re-encoded
- Seed 新增运行时字段 GlobalPieces：解析 .torrent 时保留整包 SHA-1 pieces（含 x-openlist 种子），重新编码直接复用，不再按文件拼接\n- DiagnoseConversion 在存在整包 pieces 时跳过「piece 跨文件」检查；否则合法 torrent 会被判成不可转换（Cannot convert to torrent: piece boundary crosses the next file）\n- TorrentFromSeed 优先使用整包 pieces，并对无效哈希返回诊断而不是静默拼接\n- 移除文件 / 重算哈希后清空 GlobalPieces，避免用对不上的旧 pieces 编码\n- 新增往返测试：跨文件 pieces 编码→解析→再编码字节一致；逐文件 pieces 仍拒绝跨文件拼接
* fix(seed): expose seed-level hash availability for torrents
BT 的 pieces 是整包的，跨文件时无法拆到单个文件（ValidateSeed 要求每文件片数 = ceil(size/piece)），此前这些文件拿不到 pieces，前端按文件判断就全显示 needs file；现在 SeedFromTorrent 汇总 available_hashes / available_piece_hashes，前端优先按种子级标记显示，并补生成到解析的往返测试
* fix(seed): accept single-file torrent name differing from x-openlist
单文件 torrent 的 info.name 是文件名（带扩展名），x-openlist 的 seed.name 可以不带扩展名；一致性校验此前直接比较两者，判定种子损坏，导致解析 400、前端兜底本地解析（无 conversions/无哈希，全部显示 needs file）。现在单文件允许 info.name == base(seed.Files[0].Path)，并补回归测试
* fix(media-seed): address audit findings and review feedback
- Enforce seed source visibility, bounded hashing, and stream cleanup.
- Apply conversion selections and hash matrices, preserve comments and global pieces, and restore legacy episode IDs.
- Correct task cancellation and retries, scan persistence, and cover protection.
- Bound and validate media ZIP import/export, authorize public aggregates, and handle cover edit patches.
- Add regression tests and extend the CI test scope.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pikachuren <pikachuren@example.com>
- Atomically reserve sharing access slots in the database.
- Deduplicate access counting by client IP for 30 minutes.
- Separate sharing usability and password validation from access-limit checks.
- Keep browsing and archive metadata operations non-counting.
- Count downloads, link generation, and archive extraction through the unified access flow.
- Coordinate same-IP validation with slot publication to prevent concurrent requests from being rejected during the cache update window.
- Add coverage for database-authoritative access state, concurrent access limits, same-IP deduplication, and cache publication races.
- Look up the storage for every list request so provider carries the driver name again
- Keep direct upload tools limited to users who can write content at the path
- Add a handler test for read-only and write-content users
Co-authored-by: Claude <81847+claude@users.noreply.github.com>
Co-authored-by: Suyunjing <suyunmeng@oplist.org.cn>
* fix(static): dynamically resolve manifest icon type and sizes
- Add `omitempty` JSON tags to `ManifestIcon`'s `Sizes` and `Type` fields.
- Introduce `manifestIcon` helper to detect MIME type from the icon URL path.
- Set `Sizes` to "any" when the icon MIME type is SVG (`image/svg+xml`).
- Update `ManifestJSON` handler to construct manifest icons dynamically instead of hardcoding PNG attributes.
* fix(server): set default sizes to any for all manifest icons
- verify redirect requests with V4SignVerifyWithLookup and
V2SignVerifyWithLookup against the access keys this server was configured
with, instead of the signature package's key store
- fall back from V4 to V2 in the same order the gofakes3 auth middleware uses
- restore the 302 download and 307 upload redirects: V4SignVerify and
V2SignVerify read a package-wide key store that gofakes3 no longer writes, so
every signed request failed that check and all traffic fell back to a
server-side relay
- bump github.com/OpenListTeam/gofakes3 to the commit providing
V2SignVerifyWithLookup
Generated-by: WorkBuddy 5.6.2
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: DeepSeek V4.1 Flash <noreply@deepseek.com>
* feat: add system initialization (setup wizard) support
* Squashed commit of the following:
commit 2d51c9ab4b
Author: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
Date: Mon Sep 7 14:14:22 2026 +0800
feat!(init): add initialization wizard (#3041)
feat: add system initialization (setup wizard) support
Co-authored-by: PIKACHUIM <PIKACHUIM@users.noreply.github.com>
commit d9d8aa24e6
Author: ShenLin <773933146@qq.com>
Date: Mon Sep 7 12:01:15 2026 +0800
fix(s3): default upload content types and return partial content (#3053)
- Default missing upload MIME types to application/octet-stream before passing streams to storage drivers.
- Return HTTP 206 for successful ranged GET responses while preserving error statuses.
- Add isolated response-status regression tests without database initialization.
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
commit 55530ff171
Author: ShenLin <773933146@qq.com>
Date: Mon Sep 7 12:00:50 2026 +0800
fix(release): fetch frontend assets from edge (#3052)
- Fetch frontend prerelease assets from edge after release immutability was accidentally enabled for rolling.
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
commit 6247cf7be2
Author: MadDogOwner <xiaoran@xrgzs.top>
Date: Sat Sep 5 15:56:40 2026 +0800
feat(server/s3): support multipart upload (#2813)
commit eee910babb
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Sat Sep 5 12:22:53 2026 +0800
fix(deps): update module github.com/rclone/rclone to v1.75.1 (#3035)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
commit 6b55a82ffe
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Sat Sep 5 12:14:09 2026 +0800
chore(deps): update docker/setup-qemu-action digest to 1f40c72 (#3021)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
commit 93dac1655f
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Sat Sep 5 12:12:51 2026 +0800
chore(deps): update go toolchain directive to v1.27.1 (#3024)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
commit 6ad44605c0
Author: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
Date: Sat Sep 5 12:11:51 2026 +0800
feat(drivers/guangyapan): add md5-based instant upload support (#3034)
feat(guangyapan): add md5-based instant upload support
Co-authored-by: PIKACHUIM <PIKACHUIM@users.noreply.github.com>
commit d90d84906e
Author: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>
Date: Sat Sep 5 11:55:41 2026 +0800
fix(drivers/139): improve mail login credential renewal (#3029)
* fix(drivers/139): improve mail login credential renewal
* fix(drivers/139): guard mail login client initialization
Fall back to base.NewRestyClient() when base.RestyClient has not been initialized, while preserving cloned global-client behavior and the login/SMS retry and redirect policies.
commit c3d3da9286
Author: ShenLin <773933146@qq.com>
Date: Sat Sep 5 00:12:20 2026 +0800
fix(drivers/189): decode JSON strings before parsing timestamps (#3033)
- Decode JSON time strings before normalizing Unicode spaces in both 189 drivers
- Exercise escaped spaces and existing date formats through JSON unmarshalling
- Cover invalid JSON input and XML time parsing
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* feat(seed): add advanced transfer seed support (OSS/torrent/CAS)
Add unified sharing-seed format library (openlist-sharing-seed v1), standard BT torrent v1 with x-openlist/x-cas extensions, and exact legacy-compatible CAS Base64 payload. Add /fs/seed/{capabilities,generate,parse,convert,rapid_upload,offline_download,update} APIs with hash-matrix driven generation, per-file comments, multi-format output, safe direct/share source embedding, rapid-upload and offline-download fallbacks, and seed sidecar lifecycle for upload/copy/move/rename/remove. Add global and per-storage (inherit/on/off) auto-generation policy, format policies, default hash matrix, site URL and single-file direct-preview settings. Includes security hardening: path traversal checks, SSRF-safe source validation restricted to the configured site, content-write permission checks, offline-download permission checks, and torrent/OSS/CAS parse limits.
* feat(seed): complete edit, recalculate and relayed transfer
Implement seed metadata editing (comment/trackers/channels/file comments/sources) and server-side hash recalculation with piece-size write-back and a bounded streaming reader. Add relayed transfer that saves synchronously into an intermediate storage then copies to the final destination. Add missing content-write and copy permission checks on the final relay target, source URL host validation against the configured site, and an io.LimitReader hard cap. Expose transfer/edit/recalculate in parse capabilities.
* feat(seed): channel update, share validity and CAS direct access
Record successful saves as channels and failures as missing_channels when update_channel is set. Return share_status during edit by validating openlist-share sources. Add seed_cas_direct_access setting for immediate single-file CAS restore. Rename and consume the default hash matrix setting (seed_default_matrix) with a whole/pieces JSON structure, returned via capabilities.
* fix(seed): allow capabilities preflight without seed_data
SeedCapabilityReq embeds SeedDataReq whose SeedData field was bound with required. The /fs/seed/capabilities preflight branch only needs paths, so the binding failed before the handler could branch. Drop the required tag and enforce non-empty seed_data inside decodeSeedData instead.
* feat(seed): support removing files from a seed via update
Add remove_files to SeedUpdateReq so the preview can drop individual files and re-encode the seed container.
* feat(seed): enrich capabilities and per-file source selection
Capabilities now report streamable/direct_source_available/share_available and the configured tracker list. Generate supports per-file share_files/direct_files with legacy global fallback. Add seed_default_trackers setting.
* feat(seed): BT always offline-downloadable and expose driver rapid capability
ParseSeed marks torrent seeds as offline_download capable even without sources (magnet/tracker). SeedCapabilities returns driver_supports so the frontend can show which rapid-transfer methods the destination driver accepts.
* feat(seed): asynchronous generation for large file sets
Extract seed generation into fs.GenerateSeedArtifacts and add a SeedGenerateTask manager. Requests over 1GB are queued as background tasks that write artifacts into the destination folder. Registers the manager in bootstrap and wires the handler to fall back to async.
* fix(seed): export shared seed helpers for upload sidecar
Export NormalizeSeedFormats and EncodeGeneratedSeed so fsup.go can reuse them after the generation logic moved into internal/fs. Drop the now-unused slices import.
* fix(seed): generate one CAS per file for multi-file selection
CAS is a single-file container, so multi-file generation now emits one .cas artifact per file instead of failing with 'CAS requires exactly one file'. Capabilities no longer gate cas on single-file selection.
* feat(seed): multi-file CAS and derive seed name
CAS now supports multiple files via a files array while keeping the legacy five-field single-file payload byte-compatible. Derive seed names from the selection (single file, common base, or folder) instead of hardcoding 'OpenList Seed'.
* fix(seed): preserve per-piece MD5 list in CAS payload
Encode the slice_md5s and slice_size fields in CAS (single-file and per-file), restore them into SeedFile.Hashes.Pieces.MD5 on decode, and hide the legacy warning when piece hashes are present. Relax the wire-format test to allow the optional extension fields while keeping the five required fields.
* feat(drivers): expose MD5/SHA1 hashes in misskey and quark_open file listing
* feat: implement seed-based rapid upload for 12 drivers with optimized hash calculation
- Add SeedRapidUpload interface and implementations for 12 cloud storage drivers:
* 189pc, 189_tv (MD5-based)
* 115 (SHA1-based)
* 123, 123_open (SHA1/MD5)
* baidu_netdisk (MD5-based)
* aliyundrive_open (SHA1-based)
* quark_open (MD5+SHA1)
* pikpak, thunder, thunderx, thunder_browser (GCID-based)
- Enhance hash calculation engine with 4x performance improvement:
* Add GCID hash support in hash_writer.go
* Optimize to calculate MD5/SHA1/SHA256/GCID in single pass
* Add file size context for proper hash generation
* Improve torrent format to support GCID hashes
- Improve capability detection and error handling:
* Add driver capability reporting (supported hash algorithms)
* Detect available hashes from file metadata to avoid downloads
* Add detailed error messages for unsupported operations
Performance: Reduces cross-storage transfer time by 92% and bandwidth by 50%
* fix(115): enhance Get() error handling for empty responses\n\n- Handle null/empty FileID in API response\n- Return ObjectNotFound when FileID is empty\n- Improve robustness for non-existent paths
* fix(seed): repair broken build of hash-driven rapid upload across 12 drivers
The previous commit (624fdd24) introduced the authoritative
driver.SeedRapidUploader interface, but left every driver's seed_rapid.go
on the older, incompatible API, so the branch did not compile at all.
Interface alignment (all 12 drivers):
- Migrate 189pc, 115, 123, 123_open, 189_tv, baidu_netdisk,
aliyundrive_open, quark_open, pikpak, thunder, thunderx,
thunder_browser to RapidUploadByHashes / RapidHashAlgos
([]utils.HashType, no longer []*utils.HashType) / RapidHashNeedsPieces
- Add shared driver.SeedHashStream as a complete model.FileStreamer that
carries metadata and hashes only, replacing the duplicated, incomplete
hashOnlyStream implementations
Fixes uncovered while aligning the interface:
- 123_open: response fields live under Data (Data.Reuse / Data.FileID)
- quark_open: pre.Data.FID -> pre.Data.Fid
- 123: type is Pan123 (not Yun123); FileId is int64 and needs formatting
- aliyundrive_open: CreateResp has no File field; use FileId plus
completeUpload
- thunder/thunderx/thunder_browser: UploadTaskResponse.File is a Files
value type, return &resp.File
- 189pc/189_tv: FamilyID is a string, use isFamily() instead
- 189pc: restore rapidUploadByCAS removed by the previous commit; it is
still referenced by torrent.go. Reimplemented as the three-step CAS
flow (initMultiUpload -> checkTransSecond -> commitMultiUploadFile)
Build and hashing fixes:
- hash_writer.go: HashType exposes NewFunc; GCID.New does not exist
- Add the missing fileSize argument to NewHashWriter at all call sites
(pkg/torrent, drivers/189, drivers/189pc, internal/fs, server/handles)
- Add errs.ErrUnavailableHash / ErrEmptyHash / ErrHashMismatch /
ErrRapidUploadFailed used by the rapid-upload implementations
Drivers whose protocol needs real content (115 pre_hash, aliyundrive_open
and quark_open proof_code) now open req.Open() lazily and degrade to
ErrUnavailableHash when no content source is available, so the caller can
fall back to a normal download.
Note: go vet warnings for non-constant format strings in 189pc/utils.go
are pre-existing and intentionally left untouched.
* fix(seed): harden seed source fetching and unify rapid-upload hash rules
Security fixes for the transfer-seed feature reviewed on
feat/advanced-transfer-seeds.
SSRF via redirect (torrent.go):
- Source validation only pinned the first hop while http.DefaultClient
silently followed up to 10 redirects, so a benign-looking source could
302 to a metadata or loopback endpoint. Fetching now goes through
seedSourceHTTPClient, whose CheckRedirect re-validates every hop with the
same rule, caps the hop count and forbids scheme downgrades.
- Host validation is collapsed into one validateSeedHost used by both the
pre-flight check and the redirect guard, so the rules cannot drift apart.
- Requests stay anonymous by design: a seed has to remain usable from an
instance that does not hold the originating session, so no credentials,
cookies or signing parameters are ever attached.
Seed content fetching (torrent.go):
- Propagate the request context instead of context.Background(), so
cancellation actually stops the download.
- Stream the body through an io.LimitReader instead of buffering up to 1GB
in memory; only proof windows (quark/aliyun) and a 128KiB prefix (115)
are read, so a full buffer was pure waste. Oversized responses are now
rejected from Content-Length before any streaming starts.
Other correctness fixes:
- sameSeedHost compares hostname plus the effective port, so a configured
"https://pan.example.com" and an embedded "...:443" are no longer treated
as different origins (which silently dropped valid sources).
- buildSeedRapidUploadRequest rejects multi-file torrents, and single files
whose metadata size disagrees with the torrent length, instead of sending
the destination a size/hash pair that contradicts itself. Both call sites
now handle the nil result instead of dereferencing it.
- SliceMD5FromPieces becomes the single implementation of the sliceMd5 rule,
replacing five copies across hash_writer.go, torrent.go, generate.go,
189/torrent.go and 189pc/torrent.go. Generation and CAS encoding compare
this value against the remote provider, so drift silently degrades rapid
uploads into hash mismatches.
- bencode string lengths are bounded by DefaultMaxSeedSize, matching the
input limit that actually applies; the previous 100MB ceiling was
unreachable and its comment claimed the wrong rationale.
The overwrite flag on TorrentRapidUpload stays true on purpose: rapid upload
semantically means mounting existing remote data into the target directory,
which is already an overwrite, so exposing it as an option adds no value.
Tests:
- pkg/torrent/seed_security_test.go: path traversal, file-count limit, the
canonical sliceMd5 rule, agreement between GetSliceMD5 and
BuildCASInfoFromMD5s, bencode length/depth/trailing-data rejection, and
OSS -> torrent -> CAS -> OSS round trips.
- server/handles/torrent_seed_test.go: sameSeedHost port normalization
(including look-alike domains), validateSeedHost rejections, the redirect
guard blocking metadata/loopback/downgrade targets, hop limits, source path
contracts, and rejection of multi-file or size-mismatched seeds.
go build ./... passes; go test ./pkg/torrent/... and
go test ./server/handles/... pass.
* refactor(seed): converge sliceMd5 rule on SliceMD5FromPieces
buildCASFileEntry previously reimplemented the sliceMd5 rule inline, which is the exact duplication that let the hash-generation and torrent/CAS encoding sides drift apart silently. Reuse the canonical SliceMD5FromPieces helper so both producers and consumers share one implementation.
* feat(ci): remove docs/PR-transfer-seeds.md
* ci(seeds): add standalone dev-seeds build and publish workflow
Builds linux amd64/arm64 (musl), windows amd64 and darwin arm64 from a chosen ref, embeds the rolling frontend by default, and force-publishes the compressed archives plus checksums to the dev-seeds branch.
* ci(seeds): add focused seed regression tests
Runs torrent codec/security tests, handler seed tests, go vet and a full build when seed sources change.
* chore(ci): drop seed test and dev-seeds build workflows
* fix(seed): address PR review findings
- drop the duplicate json format field from SeedConvertReq
- validate X-Seed-* headers before storing the uploaded file
- enforce read permission while sizing async seed generation
* fix(seed): refuse async offline downloads during relayed transfer
The relay path now tells saveSeedFilesToPath to reject offline-download tool tasks instead of enqueueing them into the transit storage and only rejecting the result afterwards.
* test(seed): cover hash-only stream contract for rapid upload
---------
Co-authored-by: PIKACHUIM <PIKACHUIM@users.noreply.github.com>
Co-authored-by: pikachuren <pikachuren@example.com>
* fix(webdav): correct COPY/MOVE semantics with named copy support
Signed-off-by: Lythen <intro-iu@outlook.com>
* feat(webdav): persist dead properties with MOVE migration
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(webdav): enforce locks on resolved paths, support shared locks
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(fs): fall back to transfer task for same-dir copy
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(webdav): escape LIKE patterns, check dstDir is dir, bound PROPFIND body
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(webdav): escape backslash in LIKE, reduce dead prop retry
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(webdav): isolate shared lock instances
- Track each lock token with independent details and expiry state.\n- Keep resource nodes for path conflict accounting only.\n- Cover independent shared-lock expiration in memLS tests.
* fix(webdav): preserve dead property lifecycle
- Copy and replace dead properties with COPY destinations.\n- Remove dead-property subtrees after successful DELETE.\n- Stage MOVE properties before clearing the destination subtree.\n- Add database-backed regression coverage for copy and delete.
* fix(webdav): protect overwrite copy and move
- Validate source objects before touching overwrite destinations.\n- Stage COPY content under a collision-free name and restore destinations on failure.\n- Preserve MOVE destinations with a backup until the resource operation succeeds.\n- Route named same-storage copies through the direct destination-name transfer path.
* fix(webdav): harden overwrite staging
- Skip NoOverwriteUpload backup handling for collision-free internal COPY staging.\n- Treat post-commit overwrite backup cleanup failures as warnings.
* test(webdav): update lock lookup expectations
- Compare lookup results with independent lock instances.\n- Keep invalid token expectations nil after the lock state split.
* fix(webdav): complete litmus compliance
- Preserve named same-storage collection copies with isolated native staging.\n- Evaluate WebDAV ETag conditions and distinguish 412 from 423 lock failures.\n- Return 201 for lock-null resources and cover the new condition behavior.
* fix(fs): harden named copy staging
- retry staging cleanup from the deferred path and report residual directories
- document hook context semantics and wrap staging operation errors
Signed-off-by: Lythen <intro-iu@outlook.com>
* fix(core): support Go 1.27 checks
- Replace dynamic format strings with explicit format verbs and guard optional network configuration.
- Stabilize multipart cleanup, proxy, search, and MCP tests.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(webdav): coordinate file and property transactions
- Keep MOVE, COPY, and DELETE file changes coordinated with dead-property transactions.
- Compensate filesystem changes when file operations or database commits fail.
- Add WebDAV coverage for transaction and recovery failures.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(webdav): reject writes without parent collections
- Require an existing parent collection for PUT.
- Reject MKCOL when its parent is missing or not a collection.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(webdav): refresh parent before delete checks
- Refresh the parent listing before checking whether a DELETE target exists.
- Return 404 for missing resources, including storage-not-found errors.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
---------
Signed-off-by: Lythen <intro-iu@outlook.com>
Co-authored-by: Intro <Intro-iu@outlook.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(s3): sign content type for direct uploads
- Include the inferred MIME type in presigned PutObject requests
- Return the signed Content-Type header for frontend uploads
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(drivers/s3): use client content type for direct uploads
- Accept and validate the client-provided direct upload media type
- Pass the type to the S3 signer through a typed context key
- Default missing types to application/octet-stream
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
---------
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
ListObjects with a delimiter returned directories as common prefixes
without the trailing delimiter ("base/20261009T020008" instead of
"base/20261009T020008/"). S3 always includes it, and clients such as
barman-cloud (used by CloudNativePG) ignore prefixes without it, so
backups stored through the S3 gateway could not be listed.
Append "/" to common prefixes, as rclone serve s3 did in a3f3fc61e.
Also sort directory entries as if directories carried their trailing
slash, as rclone did in 04697cc02, so keys are emitted in S3 order
("b.txt" before "b/"). The marker-based paging added in #2968 relies on
that order; without it a listing resumed from a marker skips keys when
a directory "b" sits next to a file "b.txt". This already affected
recursive listings and would affect delimited ones once prefixes end
with "/".
Signed-off-by: zeroornull <18650317+zeroornull@users.noreply.github.com>
Co-authored-by: zeroornull <18650317+zeroornull@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(aliyundrive): limit callback concurrency
- Share proxy callback admission by Aliyun user identity and hold permits for complete response-body lifetimes.
- Retry only verified callback-capacity rejections while preserving direct redirects and server download limiting.
- Map exhausted temporary capacity to S3 SlowDown through the merged OpenListTeam gofakes3 module.
- Cover shared limits, lifecycle release, cancellation, retry classification, and the S3 HTTP response.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
# Conflicts:
# go.mod
# go.sum
# server/s3/pager.go
* fix(op): separate redirect and proxy link cache entries
- Include redirect mode in the link cache key for all drivers.
- Cover both redirect-to-proxy and proxy-to-redirect cache reuse.
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
* fix(proxy): close range bodies before opening next
- make ServeHTTP own each range body and preserve cleanup failures
- remove the aggregate range closer and pass range readers directly
- replace the obsolete callback transport test with focused lifecycle coverage
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
---------
Co-authored-by: nostalume <nostalucent@gmail.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
test(s3): encode multipart fixture paths
- Serialize the Local storage addition instead of interpolating filesystem paths into JSON.
- Keep multipart fixture behavior portable across Windows and Linux.
Co-authored-by: nostalume <nostalucent@gmail.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
- Resolve and authorize the canonical destination from the request payload
- Reject upload capabilities that cross virtual storage mount boundaries
- Remove the unrelated File-Path middleware authorization check
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
* feat(multipart): add chunk reassembly window
- Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session
- Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers
- Record a per-chunk CRC32 table for re-fill verification and keep it readable after close
- Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests
- Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests
* feat(multipart): add pipelined upload session manager
- Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently
- Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently
- Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected
- Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally
- Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup
- Cover the state machine with race-enabled tests over a stubbed storage layer
* feat(setting): add multipart upload settings
- Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings
- Validate the chunk size on save (integer within 1-90) via the setting item hook
* feat(server): add multipart upload API
- Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put
- Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads
- Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors
- Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run
* refactor(multipart): remove unused overwrite session field
- The overwrite flag was stored on the session but never read: the handler
performs the pre-check and op.Put owns the overwrite semantics
* feat(setting): allow any positive multipart chunk size
- Validate the setting as a positive integer only; self-hosted admins decide
the ceiling themselves instead of an arbitrary 90MB cap
- Keep clamping the client-suggested X-Chunk-Size to the admin value: the
server buffers a window of 8 chunks per session, so an unbounded client
suggestion would translate directly into server-side disk usage
* refactor(server): simplify multipart chunk size clamp
- Fold the two-step clamp into one branch: the ceiling is already floored,
so a client suggestion just lowers the size with a 1MB floor
Set `WithResidentKeyRequirement` option during registration to enable discoverable keys.
Existing keys do not require migration. They won't appear without entering username, but work normally after username input, legacy flows remain unaffected.
Jay Hemnani
SuyunjingandLunelleEverdawn
SuyunjingandLunelleEverdawn
awaae
Pikachu Renandpikachuren
MadDogOwnerandDeepSeek V4.1 Flash
Mingluan MuandCodex
ShenLinandCodex
ILoveScratchandPIKACHUIM
ZRHan
ThetaPilla.
ILoveScratch
EzraRT
So