From 25e4eb663ffb73991a68d85df4f690d48ad137a6 Mon Sep 17 00:00:00 2001 From: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com> Date: Sat, 24 Oct 2026 17:34:29 +0800 Subject: [PATCH] feat(func): media library (#2210) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(func): media library * feat(func): media library * feat(func): media library * feat(func): media library build test * feat(func): media library build test * feat(func): media library fix remote fetch error * feat(func): media library fix remote fetch error * feat(func): media library fix remote fetch error * feat(func): media library support select * feat(func): media library support select * feat(func): media library support select * feat(func): media library support select * feat(func): media library support change api url * feat(func): media library support change api url * feat(func): media library support change api url * feat(media): 并发刮削（默认 5）+ 单条落库 增强 TMDB 搜索：模糊匹配 + 多候选关键词 管理页面新增导入/导出媒体库（全部 / 按扫描路径） 缩略图路径默认值改为 imgs * feat(media): 清空刮削、删除已失效 * fix(media): fix api proxy path * fix(media): fix lrc file display & music id3 * fix(media): fix media scan sub path error * feat(media): support cloud codec via ffmpeg * feat(media): support cloud codec via ffmpeg * feat(media): support cloud codec via ffmpeg * feat(media): support cloud codec via ffmpeg * feat(media): 提升刮削精准度，支持模糊刮削 * fix(media): clean up imports and improve FetchFileReader logic * feat(media): 提升刮削精准度，支持模糊刮削 * feat(media): 刮削支持自定义刮削数量，避免太占性能 * feat(media): 刮削支持自定义刮削数量，避免太占性能 * feat(media): 刮削修复写入作品名问题 * fix(media): return 404 on missing item_id and fix unscraped item query - Validate GetMediaItemByID result before spawning goroutine in StartMediaScrape; return 404 when item_id > 0 but item is not found - Replace `description` check with `plot` check in GetUnscrappedItems since scrapers populate Plot, not Description, to avoid re-selecting already-scraped items on every run * Apply remaining changes * feat(media): add master switch, app keys, STRM, Emby and OpenSubsonic APIs * feat(media): allow any logged-in user to manage their own media app keys * feat(media): allow signing arbitrary media path for local player * fix(media): always return an array for media status types * fix(media): persist per-type scraper config and apply it to thumbnails * feat(media): add local thumbnail endpoint and cover proxy with fallback gateway * fix(media): allow guest media access, per-file local thumbnails and optional cover proxy * feat(media): thumbnail storage modes and STRM same-directory output * feat(media): Google Books/OpenLibrary sources and remote cover download * chore(media): rename transcode cache dir to ctmp and log thumbnail storage dir * feat(media): progress and favorites API shared with Emby/Subsonic * chore: ignore runtime thumbnail directory * feat(media): add preferred scrape language and namespace protocol endpoints under @ * fix(media): harden anonymous emby images, pagination and cover proxy SSRF * fix(emby): allow anonymous user discovery and login before auth * chore: drop redundant media build workflow and protocol doc * fix(media): ID3v2.4 syncsafe frame size and preserve Hidden flag on rescan * fix(media): address security review (path traversal, guest bypass, cover XSS, auth rate limit, SSRF, quotas, migration) * fix(media): 清理失效条目时只删除确认不存在的记录 此前 fs.Get 返回任何错误都视为失效并硬删除，驱动故障/超时/鉴权失败时会误删有效媒体记录。现在仅 errs.IsObjectNotFound 才删除，其余错误跳过并在响应中返回 skipped 计数。 * fix(transcode): 播放凭据与 Worker 回写凭据分离，并收紧 profile 与鉴权校验 - 新增 Job.PlayToken（json 标签为 -），master/playlist/切片只认播放凭据，播放器不再拿到可上传、可结束任务的 callback_token - Worker 上传切片时校验 profile 必须属于任务声明的档位 - TCMaster 与 WorkerJobFinish 改用恒定时间比较，避免时序侧信道 * fix(transcode): 含冒号的本地文件名可转码，补基址缺失与重新激活配额 - ValidSourceInput 不再按冒号位置判协议：本地输入只要求以 / 开头或 Windows 盘符路径，concat:/subfile: 等前缀仍被拒且由 -protocol_whitelist 兜底（修复 /mnt/x/Star Wars: IV.mkv 被误判） - trustedInternalBaseURL 返回空串时直接 503 并给出明确原因，不再拼出 /d/xxx 让 ffmpeg 报文件不存在 - 远程输入白名单收窄为 http,https,tcp,tls（去掉 file 与 crypto） - Reactivate 复用同一套单用户配额，避免「先让任务闲置取消、再批量激活」绕过 SubmitLimited；配额不足返回 429 - OwnerID 改为配额键：普通用户 user:，访客 guest:，避免单个匿名访客占满全部匿名配额；补充对应单测 * fix(media): 收紧缩略图解码内存、跳过超长条目可见化，并补 CI 单测 - MaxImagePixels 从 64MP 降到 32MP，宽高改 int64 相乘防 32 位溢出，解码阶段加 12 路全局信号量 - 封面解码失败退回原始字节时限制 512KB，避免十几 MB 的 base64 写进 cover 列（豆瓣/本地书籍三处） - 扫描进度新增 Skipped 计数，入库被跳过的超长路径会在完成消息与日志中体现 - 数据库迁移失败日志补充 OPENLIST_MEDIA_MIGRATE_WIPE=1 的处置提示 - Subsonic 认证只对「确实带了凭据却校验失败」计数，裸探测不再刷新 IP 锁定 - goexif 从 indirect 调整为直接依赖（internal/media/scraper/image.go 直接 import） - CI 新增 go test ./internal/db/... ./internal/transcode/... ./internal/media/... * fix(transcode): 配额/队列满时仍放行已缓存切片，并同步 worker 协议白名单 - TCSegment/TCPlaylist: Reactivate 返回 ErrUserQuota/ErrQueueFull 时不再直接 429，改为标记 quotaBlocked，缓存命中的切片与 playlist 仍可读，只有缓存缺失才返回 429 - Reactivate: 补充全局排队上限 (MaxPendingJobs) 检查，避免批量激活顶爆队列，并补单测 - cmd/worker: 远程输入协议白名单与 master 同步为 http,https,tcp,tls（去掉 file/crypto） - scraper: decodeSem 容量 12 -> 3，避免信号量大于默认刮削并发而形同失效 * feat(emby): 完善客户端兼容层并修复认证锁定，Subsonic 复用统一密钥校验 - Emby: 用户 ID 改用 GUID 并兼容数字/旧 ID；补齐 /Users/{id}/Authenticate、Sessions 与 WebSocket、首页分区、合集/播放列表、字幕流与歌词、Series/Season 结构 - Emby: 版本号返回客户端可接受的兼容版本，/Users/Public 返回可登录用户以渲染用户选择 - Subsonic: 支持 enc: 十六进制密码编码；app key 校验与 Emby 共用 LookUpAppKeyUser - 限流: 认证被拒时不再刷新锁定过期时间，避免客户端持续轮询把 IP 永久锁死 * fix(media): 认证失败计数增加用户名维度，防止伪造 X-Forwarded-For 绕过锁定 - media_ratelimit: 新增 mediaAuthLockedFor / mediaAuthFailedFor / mediaAuthSucceededFor，IP 与用户名双维度计数；达到上限同样锁定 5 分钟且被拒时不续期 - emby: 按用户名登录与按 id 登录都检查用户名维度；失败计入两个维度，成功清除两个维度 - subsonic: subAuth 与 429 分支改用双维度判定；成功时用真实用户名清理计数（apiKey 认证时 u 参数为空） - 仍保留主干 5 次/5 分钟的策略，代理信任范围（SetTrustedProxies）留给项目层面单独处理 * fix(media): 匿名封面按访客身份过滤，认证日志脱敏并收紧限流计数 - embyImage: 匿名封面不再走 loadEmbyLibraryAll（全库可见），改为 loadEmbyLibrary(guest)： hidden / base_path 之外 / meta 受保护的条目都不可见，访客被禁用时直接 401 - embyUnauthorized: 只记录 Path、UA 与是否带 token，避免 query 里的 Pw/Password/api_key 和 X-Emby-Authorization 里的设备令牌写进日志 - media_ratelimit: 失败计数 Set 时带 5min TTL，避免 IP/用户名 key 无限增长（用户名来自请求参数） - auth.loginHash: 锁定期间不再续期，失败计数同样带 TTL，与媒体侧语义一致 - /System/Endpoint: 按 RemoteAddr 判断 IsLocal/IsInNetwork；/System/Info 不再返回真实监听端口 * feat(media): PDF 批注持久化 - 新增 MediaAnnotation 模型：按 (user_id, item_id, remote_id) 唯一，anchor 原样保存查看器批注对象，不重复建模页码/坐标 - GET/POST /fs/media/annotations：读取当前用户批注、按 remote_id 增量 upsert/delete（单条 512KB、单次 8MB、最多 500 条） - 批注为用户私有数据：未登录返回空列表，访客写入 403，并校验条目可见性与访问权限 - 条目删除 / 清空类型 / 清空扫描路径 / 批量删除时，在同一事务内清理对应批注 * fix(media): 选集签名前校验最终文件权限；转码在 ffprobe 前预检配额 - GetMediaStreamURL / GetMediaStrm: 解析选集后对最终 rawPath 再执行 CanUserAccessPath，避免「目录可见但 meta 隐藏规则禁止其中某个文件」时仍能拿到有效签名（签名之后下载链路不会重跑隐藏规则） - TranscodePlay: ProbeSource 之前调用 Scheduler.Capacity 预检配额，已满直接 429，不再先起最长 30s 的 ffprobe - 新增探测并发闸门（全局 8 / 每用户 3），覆盖预检与 SubmitLimited 之间的并发窗口；用 defer 释放，探测 panic 也不会占死槽位 - 补 Capacity 单测 --------- Co-authored-by: j2rong4cn Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pikachuren --- .github/workflows/build.yml | 16 + .gitignore | 1 + cmd/worker/main.go | 580 +++++++ go.mod | 1 + go.sum | 2 + internal/bootstrap/config.go | 1 + internal/bootstrap/data/setting.go | 108 ++ internal/bootstrap/run.go | 5 + internal/conf/const.go | 63 +- internal/conf/var.go | 2 + internal/db/db.go | 98 +- internal/db/db_media_migrate_test.go | 123 ++ internal/db/media.go | 626 ++++++++ internal/db/media_extra.go | 223 +++ internal/media/appkey.go | 120 ++ internal/media/id3.go | 684 ++++++++ internal/media/imgmime/imgmime.go | 36 + internal/media/imgmime/imgmime_test.go | 41 + internal/media/local_fill.go | 130 ++ internal/media/scanner.go | 758 +++++++++ internal/media/scraper/book_local.go | 389 +++++ internal/media/scraper/book_online.go | 278 ++++ internal/media/scraper/discogs.go | 317 ++++ internal/media/scraper/douban.go | 497 ++++++ internal/media/scraper/image.go | 196 +++ internal/media/scraper/language.go | 27 + internal/media/scraper/limits.go | 65 + internal/media/scraper/thumb_store.go | 160 ++ internal/media/scraper/tmdb.go | 1448 +++++++++++++++++ internal/media/strm.go | 336 ++++ internal/model/media.go | 129 ++ internal/model/media_extra.go | 91 ++ internal/model/setting.go | 2 + internal/transcode/cache.go | 489 ++++++ internal/transcode/cache_security_test.go | 70 + internal/transcode/chunk_manager.go | 520 ++++++ internal/transcode/decide.go | 131 ++ internal/transcode/local_runner.go | 772 +++++++++ internal/transcode/manager.go | 177 +++ internal/transcode/protocol.go | 75 + internal/transcode/protocol_guard.go | 56 + internal/transcode/registry.go | 122 ++ internal/transcode/scheduler.go | 460 ++++++ internal/transcode/scheduler_quota_test.go | 266 ++++ internal/transcode/types.go | 178 +++ server/handles/auth.go | 10 +- server/handles/down.go | 4 +- server/handles/emby.go | 1246 +++++++++++++++ server/handles/emby_browse.go | 1654 ++++++++++++++++++++ server/handles/emby_play.go | 681 ++++++++ server/handles/media.go | 1101 +++++++++++++ server/handles/media_extra.go | 277 ++++ server/handles/media_gate.go | 222 +++ server/handles/media_image.go | 201 +++ server/handles/media_ratelimit.go | 92 ++ server/handles/media_reader.go | 140 ++ server/handles/media_user.go | 268 ++++ server/handles/subsonic.go | 772 +++++++++ server/handles/subsonic_handlers.go | 667 ++++++++ server/handles/transcode.go | 656 ++++++++ server/router.go | 87 + 61 files changed, 18937 insertions(+), 10 deletions(-) create mode 100644 cmd/worker/main.go create mode 100644 internal/db/db_media_migrate_test.go create mode 100644 internal/db/media.go create mode 100644 internal/db/media_extra.go create mode 100644 internal/media/appkey.go create mode 100644 internal/media/id3.go create mode 100644 internal/media/imgmime/imgmime.go create mode 100644 internal/media/imgmime/imgmime_test.go create mode 100644 internal/media/local_fill.go create mode 100644 internal/media/scanner.go create mode 100644 internal/media/scraper/book_local.go create mode 100644 internal/media/scraper/book_online.go create mode 100644 internal/media/scraper/discogs.go create mode 100644 internal/media/scraper/douban.go create mode 100644 internal/media/scraper/image.go create mode 100644 internal/media/scraper/language.go create mode 100644 internal/media/scraper/limits.go create mode 100644 internal/media/scraper/thumb_store.go create mode 100644 internal/media/scraper/tmdb.go create mode 100644 internal/media/strm.go create mode 100644 internal/model/media.go create mode 100644 internal/model/media_extra.go create mode 100644 internal/transcode/cache.go create mode 100644 internal/transcode/cache_security_test.go create mode 100644 internal/transcode/chunk_manager.go create mode 100644 internal/transcode/decide.go create mode 100644 internal/transcode/local_runner.go create mode 100644 internal/transcode/manager.go create mode 100644 internal/transcode/protocol.go create mode 100644 internal/transcode/protocol_guard.go create mode 100644 internal/transcode/registry.go create mode 100644 internal/transcode/scheduler.go create mode 100644 internal/transcode/scheduler_quota_test.go create mode 100644 internal/transcode/types.go create mode 100644 server/handles/emby.go create mode 100644 server/handles/emby_browse.go create mode 100644 server/handles/emby_play.go create mode 100644 server/handles/media.go create mode 100644 server/handles/media_extra.go create mode 100644 server/handles/media_gate.go create mode 100644 server/handles/media_image.go create mode 100644 server/handles/media_ratelimit.go create mode 100644 server/handles/media_reader.go create mode 100644 server/handles/media_user.go create mode 100644 server/handles/subsonic.go create mode 100644 server/handles/subsonic_handlers.go create mode 100644 server/handles/transcode.go diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 1ff88a09..8c2b4977 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -10,6 +10,22 @@ concurrency: cancel-in-progress: true jobs: + # 媒体库相关的安全/配额/迁移测试单独跑一次，不必跟随 7 个构建目标重复执行 + test: + name: Unit tests + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 + with: + go-version: "1.27.3" + + - name: Go test + run: go test ./internal/db/... ./internal/transcode/... ./internal/media/... + build: strategy: matrix: diff --git a/.gitignore b/.gitignore index 06129e0a..6f951cd4 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,7 @@ output/ /tmp/ /log/ /lang/ +/imgs/ /daemon/ /public/dist/* !/public/dist/README.md diff --git a/cmd/worker/main.go b/cmd/worker/main.go new file mode 100644 index 00000000..e9686da6 --- /dev/null +++ b/cmd/worker/main.go @@ -0,0 +1,580 @@ +// Command openlist-transcode-worker 是独立的远程转码 Worker 节点。 +// 它通过 OpenList 自研协议（HTTP/JSON）注册到 Master，长轮询领任务，调用 +// 本机 ffmpeg 转码，并把生成的 HLS 切片实时上传回 Master。 +package main + +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "os/signal" + "path/filepath" + "runtime" + "strconv" + "strings" + "sync" + "syscall" + "time" +) + +var ( + masterURL = flag.String("master", "http://127.0.0.1:5244", "OpenList Master URL") + secret = flag.String("secret", "", "Worker 共享密钥（与 Master 配置一致）") + name = flag.String("name", hostname(), "Worker 名称") + capacity = flag.Int("capacity", 2, "并发任务数") + hwaccel = flag.String("hwaccel", "none", "硬件加速：none/auto/nvenc/qsv/vaapi/amf/videotoolbox") + workdir = flag.String("workdir", "./worker_tmp", "临时工作目录") + ffmpeg = flag.String("ffmpeg", "ffmpeg", "ffmpeg 路径") +) + +func hostname() string { + h, _ := os.Hostname() + if h == "" { + h = "worker" + } + return h +} + +// ---- 协议结构 ----（与 Master 端 internal/transcode/protocol.go 一致） + +type RegisterReq struct { + Name string `json:"name"` + Version string `json:"version"` + Capacity int `json:"capacity"` + HWAccel []string `json:"hwaccel"` + CodecsDecode []string `json:"codecs_decode"` + CodecsEncode []string `json:"codecs_encode"` + MaxResolution string `json:"max_resolution"` + Tags []string `json:"tags"` +} +type RegisterResp struct { + Data struct { + WorkerID string `json:"worker_id"` + HeartbeatInterval int `json:"heartbeat_interval"` + } `json:"data"` +} + +type HeartbeatReq struct { + WorkerID string `json:"worker_id"` + Load float64 `json:"load"` + Running []string `json:"running"` + FreeSlots int `json:"free_slots"` +} +type HeartbeatResp struct { + Data struct { + OK bool `json:"ok"` + Kick bool `json:"kick"` + Cancel []string `json:"cancel"` + } `json:"data"` +} + +type ClaimReq struct { + WorkerID string `json:"worker_id"` + Slots int `json:"slots"` + Wait int `json:"wait"` +} + +type Profile struct { + Name string `json:"name"` + VideoCodec string `json:"video_codec"` + VideoBitrate string `json:"video_bitrate"` + Scale string `json:"scale"` + AudioCodec string `json:"audio_codec"` + AudioBitrate string `json:"audio_bitrate"` + HWAccel string `json:"hwaccel"` +} +type OutputSpec struct { + Format string `json:"format"` + SegmentDuration int `json:"segment_duration"` +} +type Job struct { + ID string `json:"id"` + Path string `json:"path"` + SourceURL string `json:"source_url"` + Profiles []Profile `json:"profiles"` + Output OutputSpec `json:"output"` + CallbackToken string `json:"callback_token"` + Deadline int64 `json:"deadline"` +} +type ClaimResp struct { + Data struct { + Jobs []Job `json:"jobs"` + } `json:"data"` +} + +type FinishReq struct { + JobID string `json:"job_id"` + Status string `json:"status"` + Error string `json:"error,omitempty"` + Stats struct { + Elapsed float64 `json:"elapsed"` + BytesOut int64 `json:"bytes_out"` + } `json:"stats"` +} + +// ---- HTTP helper ---- + +var httpClient = &http.Client{Timeout: 60 * time.Second} + +func postJSON(path string, body any, out any, bearer string) error { + buf, _ := json.Marshal(body) + req, _ := http.NewRequest(http.MethodPost, *masterURL+path, bytes.NewReader(buf)) + req.Header.Set("Content-Type", "application/json") + if bearer != "" { + req.Header.Set("Authorization", "Bearer "+bearer) + } + resp, err := httpClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode >= 400 { + raw, _ := io.ReadAll(resp.Body) + return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw)) + } + if out != nil { + return json.NewDecoder(resp.Body).Decode(out) + } + return nil +} + +// ---- main ---- + +func main() { + flag.Parse() + if *secret == "" { + fmt.Fprintln(os.Stderr, "missing -secret") + os.Exit(1) + } + if err := os.MkdirAll(*workdir, 0o755); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + + workerID, hbInterval, err := registerOnce() + if err != nil { + fmt.Fprintln(os.Stderr, "register failed:", err) + os.Exit(1) + } + fmt.Printf("[worker] registered: id=%s heartbeat=%ds\n", workerID, hbInterval) + + ctx, cancel := context.WithCancel(context.Background()) + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + go func() { + <-sigCh + fmt.Println("[worker] shutting down") + cancel() + }() + + state := &workerState{ + id: workerID, + runningMu: sync.Mutex{}, + running: map[string]context.CancelFunc{}, + } + + // 心跳协程 + go func() { + t := time.NewTicker(time.Duration(hbInterval) * time.Second) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + state.heartbeat() + } + } + }() + // 工作协程 + for i := 0; i < *capacity; i++ { + go state.workLoop(ctx) + } + <-ctx.Done() + time.Sleep(500 * time.Millisecond) +} + +func registerOnce() (string, int, error) { + req := RegisterReq{ + Name: *name, + Version: "1.0.0", + Capacity: *capacity, + HWAccel: []string{*hwaccel}, + CodecsDecode: []string{"h264", "hevc", "av1", "vp9", "mpeg4", "mpeg2video"}, + CodecsEncode: []string{"h264", "hevc"}, + MaxResolution: "3840x2160", + Tags: []string{runtime.GOOS, runtime.GOARCH}, + } + var resp RegisterResp + if err := postJSON("/api/transcode/worker/register", req, &resp, *secret); err != nil { + return "", 0, err + } + if resp.Data.WorkerID == "" { + return "", 0, fmt.Errorf("master returned empty worker_id") + } + if resp.Data.HeartbeatInterval == 0 { + resp.Data.HeartbeatInterval = 10 + } + return resp.Data.WorkerID, resp.Data.HeartbeatInterval, nil +} + +// ---- 状态机 ---- + +type workerState struct { + id string + runningMu sync.Mutex + running map[string]context.CancelFunc +} + +func (s *workerState) heartbeat() { + s.runningMu.Lock() + ids := make([]string, 0, len(s.running)) + for id := range s.running { + ids = append(ids, id) + } + free := *capacity - len(ids) + s.runningMu.Unlock() + var resp HeartbeatResp + if err := postJSON("/api/transcode/worker/heartbeat", HeartbeatReq{ + WorkerID: s.id, + Running: ids, + FreeSlots: free, + }, &resp, *secret); err != nil { + fmt.Println("[worker] heartbeat error:", err) + return + } + for _, jid := range resp.Data.Cancel { + s.runningMu.Lock() + if cancel, ok := s.running[jid]; ok { + cancel() + } + s.runningMu.Unlock() + } +} + +func (s *workerState) workLoop(ctx context.Context) { + for { + if ctx.Err() != nil { + return + } + var resp ClaimResp + err := postJSON("/api/transcode/worker/claim", ClaimReq{ + WorkerID: s.id, Slots: 1, Wait: 20, + }, &resp, *secret) + if err != nil { + fmt.Println("[worker] claim error:", err) + time.Sleep(5 * time.Second) + continue + } + if len(resp.Data.Jobs) == 0 { + continue + } + for _, j := range resp.Data.Jobs { + s.runJob(ctx, j) + } + } +} + +func (s *workerState) runJob(parent context.Context, job Job) { + jobCtx, cancel := context.WithCancel(parent) + s.runningMu.Lock() + s.running[job.ID] = cancel + s.runningMu.Unlock() + defer func() { + s.runningMu.Lock() + delete(s.running, job.ID) + s.runningMu.Unlock() + cancel() + }() + + startedAt := time.Now() + tmpDir := filepath.Join(*workdir, job.ID) + _ = os.MkdirAll(tmpDir, 0o755) + defer os.RemoveAll(tmpDir) + + if len(job.Profiles) == 0 { + s.finish(job, "failed", "no profiles", 0, 0) + return + } + profile := job.Profiles[0] + playlist := filepath.Join(tmpDir, "playlist.m3u8") + segPattern := filepath.Join(tmpDir, "seg-%d.ts") + args := buildFFmpegArgs(job, profile, segPattern, playlist) + + cmd := exec.CommandContext(jobCtx, *ffmpeg, args...) + stderr := &bytes.Buffer{} + cmd.Stderr = stderr + fmt.Printf("[worker] running job %s, ffmpeg %s\n", job.ID, strings.Join(args, " ")) + if err := cmd.Start(); err != nil { + s.finish(job, "failed", err.Error(), 0, 0) + return + } + + // 上传协程 + stopUpload := make(chan struct{}) + uploadDone := make(chan struct{}) + go func() { + defer close(uploadDone) + s.watchAndUpload(jobCtx, job, profile.Name, tmpDir, stopUpload) + }() + + err := cmd.Wait() + close(stopUpload) + <-uploadDone + // 最后一次扫描：把还未上传的尾片传上去 + s.uploadOnce(job, profile.Name, tmpDir, true) + + if err != nil { + if jobCtx.Err() == context.Canceled { + s.finish(job, "cancelled", "cancelled by master", 0, 0) + return + } + s.finish(job, "failed", fmt.Sprintf("%v: %s", err, truncate(stderr.String(), 1024)), 0, 0) + return + } + s.finish(job, "finished", "", time.Since(startedAt).Seconds(), 0) +} + +func (s *workerState) finish(job Job, status, errStr string, elapsed float64, bytesOut int64) { + req := FinishReq{ + JobID: job.ID, Status: status, Error: errStr, + } + req.Stats.Elapsed = elapsed + req.Stats.BytesOut = bytesOut + if err := postJSON("/api/transcode/worker/job/finish", req, nil, job.CallbackToken); err != nil { + fmt.Printf("[worker] finish report failed: %v\n", err) + } +} + +func (s *workerState) watchAndUpload(_ context.Context, job Job, profile, dir string, stop chan struct{}) { + t := time.NewTicker(800 * time.Millisecond) + defer t.Stop() + uploaded := map[int]bool{} + flush := func(final bool) { + s.uploadDelta(job, profile, dir, uploaded, final) + } + for { + select { + case <-stop: + flush(false) + return + case <-t.C: + flush(false) + } + } +} + +func (s *workerState) uploadOnce(job Job, profile, dir string, final bool) { + uploaded := map[int]bool{} + s.uploadDelta(job, profile, dir, uploaded, final) +} + +// uploadDelta 扫描目录，上传新切片（uploaded 跟踪已传过的 seq；最后一片到 final 时才上传） +func (s *workerState) uploadDelta(job Job, profile, dir string, uploaded map[int]bool, final bool) { + entries, err := os.ReadDir(dir) + if err != nil { + return + } + type segFile struct { + seq int + path string + } + var segs []segFile + for _, e := range entries { + if e.IsDir() { + continue + } + n := e.Name() + if !strings.HasPrefix(n, "seg-") || !strings.HasSuffix(n, ".ts") { + continue + } + num, err := strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(n, "seg-"), ".ts")) + if err != nil { + continue + } + segs = append(segs, segFile{num, filepath.Join(dir, n)}) + } + // 排序 + for i := 0; i < len(segs); i++ { + for j := i + 1; j < len(segs); j++ { + if segs[j].seq < segs[i].seq { + segs[i], segs[j] = segs[j], segs[i] + } + } + } + limit := len(segs) + if !final && limit > 0 { + limit-- // 最后一片可能还在写，留到下一轮 + } + segDur := job.Output.SegmentDuration + if segDur == 0 { + segDur = 6 + } + for i := 0; i < limit; i++ { + seg := segs[i] + if uploaded[seg.seq] { + continue + } + isLast := final && i == limit-1 + if err := s.uploadSegment(job, profile, seg.seq, float64(segDur), seg.path, isLast); err != nil { + fmt.Printf("[worker] upload seg-%d failed: %v\n", seg.seq, err) + return + } + uploaded[seg.seq] = true + } +} + +func (s *workerState) uploadSegment(job Job, profile string, seq int, dur float64, file string, final bool) error { + f, err := os.Open(file) + if err != nil { + return err + } + defer f.Close() + q := fmt.Sprintf("?job=%s&profile=%s&seq=%d&duration=%f&final=%t", + job.ID, profile, seq, dur, final) + req, _ := http.NewRequest(http.MethodPut, *masterURL+"/api/transcode/worker/segment"+q, f) + req.Header.Set("Content-Type", "video/mp2t") + req.Header.Set("Authorization", "Bearer "+job.CallbackToken) + resp, err := httpClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode >= 400 { + raw, _ := io.ReadAll(resp.Body) + return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw)) + } + return nil +} + +// ---- ffmpeg args（与 master 端 local_runner 保持一致）---- + +// protocolWhitelistFor 与 internal/transcode.ProtocolWhitelistFor 保持一致： +// 远程输入只放行 HTTP(S) 及其传输层，本地路径只放行 file。 +func protocolWhitelistFor(source string) string { + s := strings.ToLower(strings.TrimSpace(source)) + if strings.HasPrefix(s, "http://") || strings.HasPrefix(s, "https://") { + // 与 master 的 protocolWhitelistRemote 保持一致：HTTP 源用不到 file/crypto， + // 留着等于给被构造过的播放列表留一条读本地文件的通道。 + return "http,https,tcp,tls" + } + return "file" +} + +func buildFFmpegArgs(job Job, profile Profile, segPattern, playlist string) []string { + hw := strings.ToLower(profile.HWAccel) + args := []string{"-y", "-loglevel", "warning"} + switch hw { + case "nvenc": + args = append(args, "-hwaccel", "cuda") + case "qsv": + args = append(args, "-hwaccel", "qsv") + case "vaapi": + args = append(args, "-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", "/dev/dri/renderD128") + case "amf": + args = append(args, "-hwaccel", "d3d11va") + case "videotoolbox": + args = append(args, "-hwaccel", "videotoolbox") + case "auto": + args = append(args, "-hwaccel", "auto") + } + // 限制 ffmpeg 可用协议：Master 下发的 source_url 不应被解释成 concat:/subfile: 等特殊协议。 + // -protocol_whitelist 是输入选项，必须位于 -i 之前。 + args = append(args, "-protocol_whitelist", protocolWhitelistFor(job.SourceURL)) + args = append(args, "-i", job.SourceURL) + args = append(args, "-c:v", pickEncoder(profile.VideoCodec, hw)) + if profile.VideoBitrate != "" { + args = append(args, "-b:v", profile.VideoBitrate, "-maxrate", profile.VideoBitrate, "-bufsize", profile.VideoBitrate) + } + if profile.Scale != "" { + switch hw { + case "vaapi": + args = append(args, "-vf", "scale_vaapi="+profile.Scale) + case "qsv": + args = append(args, "-vf", "scale_qsv="+profile.Scale) + default: + args = append(args, "-vf", "scale="+profile.Scale) + } + } + if profile.AudioCodec == "copy" { + args = append(args, "-c:a", "copy") + } else { + ac := profile.AudioCodec + if ac == "" { + ac = "aac" + } + args = append(args, "-c:a", ac) + if profile.AudioBitrate != "" { + args = append(args, "-b:a", profile.AudioBitrate) + } + } + segDur := job.Output.SegmentDuration + if segDur == 0 { + segDur = 6 + } + args = append(args, + "-f", "hls", + "-hls_time", strconv.Itoa(segDur), + "-hls_playlist_type", "vod", + "-hls_segment_filename", segPattern, + "-hls_flags", "independent_segments", + playlist, + ) + return args +} + +func pickEncoder(codec, hw string) string { + codec = strings.ToLower(codec) + switch codec { + case "h264", "": + switch hw { + case "nvenc": + return "h264_nvenc" + case "qsv": + return "h264_qsv" + case "vaapi": + return "h264_vaapi" + case "amf": + return "h264_amf" + case "videotoolbox": + return "h264_videotoolbox" + } + return "libx264" + case "hevc", "h265": + switch hw { + case "nvenc": + return "hevc_nvenc" + case "qsv": + return "hevc_qsv" + case "vaapi": + return "hevc_vaapi" + case "amf": + return "hevc_amf" + case "videotoolbox": + return "hevc_videotoolbox" + } + return "libx265" + case "av1": + switch hw { + case "nvenc": + return "av1_nvenc" + case "qsv": + return "av1_qsv" + } + return "libsvtav1" + } + return "libx264" +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] +} diff --git a/go.mod b/go.mod index 5a304186..a225df35 100644 --- a/go.mod +++ b/go.mod @@ -65,6 +65,7 @@ require ( github.com/pquerna/otp v1.5.0 github.com/quic-go/quic-go v0.61.0 github.com/rclone/rclone v1.75.1 + github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd github.com/shirou/gopsutil/v4 v4.26.8 github.com/sirupsen/logrus v1.10.2 github.com/spf13/afero v1.15.0 diff --git a/go.sum b/go.sum index dae5e47a..a437bd1f 100644 --- a/go.sum +++ b/go.sum @@ -791,6 +791,8 @@ github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUc github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc= +github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk= github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 h1:GHRpF1pTW19a8tTFrMLUcfWwyC0pnifVo2ClaLq+hP8= github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46/go.mod h1:uAQ5PCi+MFsC7HjREoAz1BU+Mq60+05gifQSsHSDG/8= github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4 h1:PT+ElG/UUFMfqy5HrxJxNzj3QBOf7dZwupeVC+mG1Lo= diff --git a/internal/bootstrap/config.go b/internal/bootstrap/config.go index 83044680..80d513bb 100644 --- a/internal/bootstrap/config.go +++ b/internal/bootstrap/config.go @@ -40,6 +40,7 @@ func InitConfig() { if !filepath.IsAbs(dataDir) { flags.DataDir = filepath.Join(pwd, flags.DataDir) } + conf.DataDir = flags.DataDir // Determine config file path: use flags.ConfigPath if provided, otherwise default to /config.json configPath := flags.ConfigPath if configPath == "" { diff --git a/internal/bootstrap/data/setting.go b/internal/bootstrap/data/setting.go index b301f84d..b34df84c 100644 --- a/internal/bootstrap/data/setting.go +++ b/internal/bootstrap/data/setting.go @@ -259,6 +259,114 @@ func InitialSettings() []model.SettingItem { {Key: conf.StreamMaxClientUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE}, {Key: conf.StreamMaxServerDownloadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE}, {Key: conf.StreamMaxServerUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE}, + + // media settings + {Key: conf.MediaTMDBKey, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaTMDBAPIURL, Value: "api.themoviedb.org", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaDiscogsToken, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaDiscogsAPIURL, Value: "api.discogs.com", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaStoreThumbnail, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaThumbnailMode, Value: "runtime", Type: conf.TypeSelect, Options: "base64,runtime,local", Group: model.MEDIA, Flag: model.PRIVATE, + Help: `缩略图存储方式：base64 存入数据库；runtime 存入程序数据目录（data/imgs）；local 写入挂载目录下的 VFS 路径`}, + {Key: conf.MediaThumbnailPath, Value: "/imgs", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaScrapeConcurrency, Value: "5", Type: conf.TypeNumber, Group: model.MEDIA, Flag: model.PRIVATE}, + {Key: conf.MediaScrapeLanguage, Value: "zh-CN", Type: conf.TypeSelect, Options: "zh-CN,zh-TW,en-US,ja-JP,ko-KR,fr-FR,de-DE,es-ES,ru-RU,pt-BR", Group: model.MEDIA, Flag: model.PRIVATE, + Help: `首选刮削语言：影响 TMDB 的标题/简介/海报文字与 Google Books 的检索语言；豆瓣固定为中文`}, + {Key: conf.MediaBookSource, Value: "douban", Type: conf.TypeSelect, Options: "douban,google,openlibrary", Group: model.MEDIA, Flag: model.PRIVATE, + Help: `书籍刮削数据源：douban（豆瓣）、google（Google Books）、openlibrary（OpenLibrary）`}, + {Key: conf.MediaGoogleBooksKey, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `Google Books API Key，可留空（无 Key 时配额较低）；申请地址 https://developers.google.com/books/docs/v1/using#APIKey`}, + {Key: conf.MediaDownloadCover, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `开启后把 TMDB / Google Books / OpenLibrary 等返回的封面链接下载到下方缩略图存储路径，而不是直接保存图片链接`}, + {Key: conf.MediaEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PUBLIC, + Help: `媒体库总开关。关闭时媒体库相关页面、API 与协议端点全部不可用（返回 404），与未安装媒体库时表现一致`}, + {Key: conf.MediaEmbyEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `开启后提供 Emby 兼容 API（/@emby），可供 Infuse、VidHub、MoviePilot 等客户端直接接入`}, + {Key: conf.MediaSubsonicEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `开启后提供 OpenSubsonic API（/@rest），可供 Symfonium、Feishin、Amperfy 等音乐客户端接入`}, + {Key: conf.MediaSubsonicAllowPwd, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `允许 Subsonic 客户端用 OpenList 登录密码（u+p 参数）认证。该方式会把主账号密码暴露给第三方客户端，默认关闭，推荐使用应用密钥（apiKey 或 t+s）`}, + {Key: conf.MediaSubsonicProxyStream, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `Subsonic 播放默认返回 302 直链；若客户端不跟随重定向，可开启此项改为服务端代理转发（占用服务器带宽）`}, + {Key: conf.MediaProxyCover, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `开启后由服务端代理 TMDB/豆瓣等外部封面图片，适合直连图片域名受限的网络；默认关闭，直接使用原始图片地址`}, + {Key: conf.MediaCoverGateway, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `封面代理直连失败时，回退到第三方图片网关 wsrv.nl 重试。开启后封面地址会发送给该第三方服务；默认关闭`}, + {Key: conf.MediaStrmOnScan, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `扫描完成后自动把媒体条目生成为 .strm 文件并写入下方目录`}, + {Key: conf.MediaStrmOverwrite, Value: "true", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `生成 STRM 时覆盖已存在的同名文件`}, + {Key: conf.MediaStrmSameDir, Value: "true", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `开启时 .strm 与媒体文件放在同一目录且同名（仅扩展名不同）；关闭时统一输出到下方指定目录`}, + {Key: conf.MediaStrmPath, Value: "/strm", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `STRM 输出目录（VFS 路径，需位于可写存储中），仅在关闭「与媒体文件同目录」时使用`}, + {Key: conf.MediaStrmTemplate, Value: "{site_url}/d{path}?sign={sign}", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE, + Help: `STRM 内容模板，可用变量：{site_url} 站点地址、{path} 文件虚拟路径、{sign} 签名、{name} 文件名`}, + + // transcode settings (FFmpeg 云端/本地转码) - 默认全部关闭 + {Key: conf.TranscodeEnabled, Value: "false", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `开启后，超过阈值的媒体文件将通过 FFmpeg 转码后再播放；默认关闭`}, + {Key: conf.TranscodeRunMode, Value: "local", Type: conf.TypeSelect, Options: "local,remote,hybrid", Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `local=仅使用本机内置 worker；remote=只使用远程 Worker 节点；hybrid=本地优先，超载后派发到远程`}, + {Key: conf.TranscodeMinSizeGB, Value: "5", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `文件大于该 GB 数才走转码（小于则直链播放），0=任意大小都转码`}, + {Key: conf.TranscodeMinBitrateMbps, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `视频码率超过该 Mbps 才转码，0=不限制`}, + {Key: conf.TranscodeSourceCodecs, Value: "hevc,h265,av1,vvc,vp9", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `仅对这些源视频编码进行转码（逗号分隔）。常见高码率/兼容性差的编码：hevc,av1,vvc,vp9`}, + {Key: conf.TranscodeSourceExtensions, Value: "mkv,ts,m2ts,mov,avi,wmv,flv,rmvb,webm", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `仅对这些后缀进行转码（逗号分隔，不带点），mp4 默认不转码可直接播放`}, + {Key: conf.TranscodeOutputFormat, Value: "hls", Type: conf.TypeSelect, Options: "hls,dash,mp4", Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `输出封装格式，HLS 兼容性最好`}, + {Key: conf.TranscodeOutputCodec, Value: "h264", Type: conf.TypeSelect, Options: "h264,hevc,av1", Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `重新编码后的视频编码，推荐 h264（最广兼容、互联网分发免授权费）`}, + {Key: conf.TranscodeOutputBitrate, Value: "4000k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `输出视频码率，例如 4000k / 6M。建议 1080p:4000k、720p:2500k`}, + {Key: conf.TranscodeOutputAudioCodec, Value: "aac", Type: conf.TypeSelect, Options: "aac,mp3,opus,copy", Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `输出音频编码，aac 兼容性最好；copy=直接复制源音频流`}, + {Key: conf.TranscodeOutputAudioBitrate, Value: "160k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `输出音频码率`}, + {Key: conf.TranscodeOutputResolution, Value: "1920x1080", Type: conf.TypeSelect, Options: "source,3840x2160,2560x1440,1920x1080,1280x720,854x480", Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `输出分辨率上限，超过该分辨率会下采样；source=保持源分辨率`}, + {Key: conf.TranscodeSegmentDuration, Value: "6", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `HLS/DASH 切片时长(秒)，越小首帧越快但请求数变多，推荐 4-10`}, + {Key: conf.TranscodeHWAccel, Value: "none", Type: conf.TypeSelect, + Options: "none,auto,nvenc,qsv,vaapi,amf,videotoolbox", + Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `GPU 硬件加速： +none = 纯 CPU(libx264) +auto = 自动探测可用加速器 +nvenc = NVIDIA GPU(GeForce/Tesla/Quadro/RTX,需 NVIDIA 驱动) +qsv = Intel 集显/独显 QuickSync(免费,功耗低) +vaapi = Linux 通用 VA-API(支持 Intel/AMD) +amf = AMD GPU(Windows AMF) +videotoolbox = macOS 硬件加速`}, + {Key: conf.TranscodeFFmpegPath, Value: "ffmpeg", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `FFmpeg 可执行路径，留空则使用 PATH 中的 ffmpeg`}, + {Key: conf.TranscodeFFprobePath, Value: "ffprobe", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `FFprobe 可执行路径`}, + {Key: conf.TranscodeWorkerSecret, Value: "", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `远程 Worker 注册时使用的共享密钥，留空则禁用远程 Worker 注册`}, + {Key: conf.TranscodeCachePath, Value: "data/ctmp", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `转码切片缓存目录`}, + {Key: conf.TranscodeCacheMaxGB, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `切片缓存最大容量(GB)，超过后按 LRU 清理`}, + {Key: conf.TranscodeJobTimeoutMin, Value: "120", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `单个转码任务超时分钟数，超时自动失败`}, + {Key: conf.TranscodeLocalConcurrency, Value: "1", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `本机内置 worker 同时执行的转码任务数(local/hybrid 模式生效)`}, + {Key: conf.TranscodeIdleTimeoutSec, Value: "90", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `播放端无请求多少秒后自动停止转码（默认90秒，0=禁用）`}, + {Key: conf.TranscodeChunkDurationSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `每个 chunk 时长(秒)，越大每段缓冲越长但 ffmpeg 进程切换越少（默认60秒）`}, + {Key: conf.TranscodeMaxChunkConcurrency, Value: "2", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `单个视频任务最多同时运行的 chunk 数（默认2，越大并发越高但 CPU/GPU 占用越多）`}, + {Key: conf.TranscodeChunkIdleSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `chunk 多久没被访问就 kill ffmpeg 释放资源（默认60秒）`}, + {Key: conf.TranscodeChunkPrefetch, Value: "true", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE, + Help: `当前 chunk 转码进度过半时，是否预启动下一个 chunk（默认开启，可让顺序播放无缝衔接）`}, + + // multipart upload settings (from main) {Key: conf.MultipartEnabled, Value: "true", Type: conf.TypeBool, Group: model.TRAFFIC, Flag: model.PUBLIC}, {Key: conf.MultipartChunkSize, Value: "10", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: `chunk size of multipart upload in MB (positive integer), keep it under your CDN's request body limit; each active session buffers up to 8 chunks on the server's disk`}, } diff --git a/internal/bootstrap/run.go b/internal/bootstrap/run.go index 6740dba6..558e297c 100644 --- a/internal/bootstrap/run.go +++ b/internal/bootstrap/run.go @@ -15,6 +15,7 @@ import ( "github.com/OpenListTeam/OpenList/v4/internal/conf" "github.com/OpenListTeam/OpenList/v4/internal/db" "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/transcode" "github.com/OpenListTeam/OpenList/v4/pkg/utils" "github.com/OpenListTeam/OpenList/v4/server" "github.com/OpenListTeam/OpenList/v4/server/middlewares" @@ -92,6 +93,10 @@ func Start() { InitOfflineDownloadTools() LoadStorages() InitTaskManager() + // 初始化转码模块（总开关关时也会启动 manager，但本地 worker 仅在 enabled 时由 Manager.Start 内部按 run_mode 启动） + if transcode.IsEnabled() { + transcode.Default().Start() + } if !flags.Debug && !flags.Dev { gin.SetMode(gin.ReleaseMode) } diff --git a/internal/conf/const.go b/internal/conf/const.go index 3284b4db..436c3805 100644 --- a/internal/conf/const.go +++ b/internal/conf/const.go @@ -175,8 +175,67 @@ const ( StreamMaxClientUploadSpeed = "max_client_upload_speed" StreamMaxServerDownloadSpeed = "max_server_download_speed" StreamMaxServerUploadSpeed = "max_server_upload_speed" - MultipartEnabled = "multipart_enabled" - MultipartChunkSize = "multipart_chunk_size" + + // media + MediaTMDBKey = "media_tmdb_key" + MediaTMDBAPIURL = "media_tmdb_api_url" + MediaDiscogsToken = "media_discogs_token" + MediaDiscogsAPIURL = "media_discogs_api_url" + MediaThumbnailMode = "media_thumbnail_mode" + MediaThumbnailPath = "media_thumbnail_path" + MediaStoreThumbnail = "media_store_thumbnail" + MediaScrapeConcurrency = "media_scrape_concurrency" + MediaScrapeLanguage = "media_scrape_language" // 首选刮削语言（TMDB / Google Books） + MediaBookSource = "media_book_source" // 书籍刮削源：douban/google/openlibrary + MediaGoogleBooksKey = "media_google_books_key" // Google Books API Key（可选） + MediaDownloadCover = "media_download_cover" // 是否把远程封面下载到本地存储 + + // media library master switch & protocol switches + MediaEnabled = "media_enabled" // 媒体库总开关（PUBLIC，前端据此显示/隐藏） + MediaEmbyEnabled = "media_emby_enabled" // Emby 兼容 API 开关 + MediaSubsonicEnabled = "media_subsonic_enabled" // OpenSubsonic API 开关 + MediaSubsonicProxyStream = "media_subsonic_proxy_stream" // Subsonic 播放是否强制代理（兼容不跟随 302 的客户端） + MediaSubsonicAllowPwd = "media_subsonic_allow_password_auth" // Subsonic 是否允许 u+p 使用 OpenList 登录密码认证（默认关闭） + MediaProxyCover = "media_proxy_cover" // 是否通过服务端代理外部封面（TMDB/豆瓣等） + MediaCoverGateway = "media_cover_gateway" // 封面代理直连失败时是否回退第三方图片网关（wsrv.nl，默认关闭） + MediaStrmOnScan = "media_strm_on_scan" // 扫描完成后自动生成 STRM + MediaStrmSameDir = "media_strm_same_dir" // STRM 是否与媒体文件同目录（同名 .strm） + MediaStrmPath = "media_strm_path" // STRM 输出目录（VFS 路径，仅在指定目录模式下使用） + MediaStrmOverwrite = "media_strm_overwrite" // 生成 STRM 时是否覆盖已存在文件 + MediaStrmTemplate = "media_strm_template" // STRM 内容模板 + + // transcode (云端/本地 FFmpeg 转码) + TranscodeEnabled = "transcode_enabled" // 总开关，默认关 + TranscodeMinSizeGB = "transcode_min_size_gb" // 文件大于多少 GB 才转码 + TranscodeMinBitrateMbps = "transcode_min_bitrate_mbps" // 视频码率高于多少 Mbps 才转码（0=不限） + TranscodeSourceCodecs = "transcode_source_codecs" // 仅对这些源编码进行转码 + TranscodeSourceExtensions = "transcode_source_extensions" // 仅对这些后缀进行转码 + TranscodeOutputFormat = "transcode_output_format" // 输出封装格式 hls/dash/mp4 + TranscodeOutputCodec = "transcode_output_codec" // 重新编码的视频编码 + TranscodeOutputBitrate = "transcode_output_bitrate" // 输出视频码率，例如 4000k + TranscodeOutputAudioCodec = "transcode_output_audio_codec" // 输出音频编码 + TranscodeOutputAudioBitrate = "transcode_output_audio_bitrate" // 输出音频码率，例如 160k + TranscodeOutputResolution = "transcode_output_resolution" // 输出分辨率上限 + TranscodeSegmentDuration = "transcode_segment_duration" // HLS 切片时长(秒) + TranscodeHWAccel = "transcode_hwaccel" // 硬件加速类型 none/auto/nvenc/qsv/vaapi/amf/videotoolbox + TranscodeFFmpegPath = "transcode_ffmpeg_path" // 自定义 ffmpeg 可执行路径 + TranscodeFFprobePath = "transcode_ffprobe_path" // 自定义 ffprobe 可执行路径 + TranscodeRunMode = "transcode_run_mode" // local / remote / hybrid + TranscodeWorkerSecret = "transcode_worker_secret" // 远程 Worker 共享密钥 + TranscodeCachePath = "transcode_cache_path" // 切片缓存目录 + TranscodeCacheMaxGB = "transcode_cache_max_gb" // 缓存最大占用，超出 LRU 淘汰 + TranscodeJobTimeoutMin = "transcode_job_timeout_min" // 单个任务超时分钟 + TranscodeLocalConcurrency = "transcode_local_concurrency" // 本地内置 worker 并发数(run_mode=local/hybrid 时生效) + TranscodeIdleTimeoutSec = "transcode_idle_timeout_sec" // 播放端无请求多少秒后自动停止转码（默认90秒） + // === 智能 chunk 调度相关 === + TranscodeChunkDurationSec = "transcode_chunk_duration_sec" // 每个 chunk 时长(秒)，将整个视频分成多个 chunk 按需转码（默认60秒） + TranscodeMaxChunkConcurrency = "transcode_max_chunk_concurrency" // 单个 Job 最多同时运行的 chunk 数（默认2） + TranscodeChunkIdleSec = "transcode_chunk_idle_sec" // chunk 多久没被访问就 kill ffmpeg 释放资源（默认60秒） + TranscodeChunkPrefetch = "transcode_chunk_prefetch" // 当前 chunk 转码进度过半时，是否预启动下一个 chunk（默认true） + + // multipart upload (from main) + MultipartEnabled = "multipart_enabled" + MultipartChunkSize = "multipart_chunk_size" ) const ( diff --git a/internal/conf/var.go b/internal/conf/var.go index 6b25bcfb..c7fed954 100644 --- a/internal/conf/var.go +++ b/internal/conf/var.go @@ -18,6 +18,8 @@ var ( Conf *Config URL *url.URL ConfigPath string + // DataDir 程序数据目录（绝对路径），用于运行时文件（如本地缩略图） + DataDir string = "data" ) var SlicesMap = make(map[string][]string) diff --git a/internal/db/db.go b/internal/db/db.go index 59e99f39..85864800 100644 --- a/internal/db/db.go +++ b/internal/db/db.go @@ -1,6 +1,9 @@ package db import ( + "os" + "strings" + log "github.com/sirupsen/logrus" "github.com/OpenListTeam/OpenList/v4/internal/conf" @@ -10,12 +13,101 @@ import ( var db *gorm.DB +// MediaMigrateWipeEnv 设为 1/true 时，允许升级流程清空旧版 media_items 数据后重建。 +// 旧版本的 folder_path 语义不同，无法原地迁移；但清库是破坏性操作，必须由管理员显式确认。 +const MediaMigrateWipeEnv = "OPENLIST_MEDIA_MIGRATE_WIPE" + func Init(d *gorm.DB) { db = d - err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB), new(model.WebDAVProperty)) - if err != nil { - log.Fatalf("failed migrate database: %s", err.Error()) + if err := migrateMediaItems(); err != nil { + log.Fatalf("failed migrate media_items: %s (如果旧数据无法原地迁移，请先备份数据库，再设置 %s=1 重建媒体表)", err.Error(), MediaMigrateWipeEnv) } + err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB), new(model.WebDAVProperty), new(model.MediaItem), new(model.MediaConfig), new(model.MediaScanPath), new(model.MediaAppKey), new(model.MediaUserData), new(model.MediaPlaylist), new(model.MediaPlaylistItem), new(model.EmbyDevice), new(model.MediaAnnotation)) + if err != nil { + // 典型场景：未确认清库时，旧 media_items 数据违反新建的唯一索引。 + // 这里把解决办法直接写进日志，避免用户只看到一句 failed migrate database。 + log.Fatalf( + "failed migrate database: %s (若由媒体表唯一索引与旧数据冲突导致，请先备份数据库，再设置 %s=1 重建媒体表)", + err.Error(), + MediaMigrateWipeEnv, + ) + } +} + +func mediaWipeConfirmed() bool { + switch os.Getenv(MediaMigrateWipeEnv) { + case "1", "true", "TRUE", "True", "yes": + return true + } + return false +} + +// mediaItemsTable 返回 MediaItem 的真实表名。 +// 表名带有 conf.Database.TablePrefix（可配置，默认 x_），不能硬编码。 +func mediaItemsTable() (string, error) { + stmt := &gorm.Statement{DB: db} + if err := stmt.Parse(&model.MediaItem{}); err != nil { + return "", err + } + return stmt.Schema.Table, nil +} + +// migrateMediaItems 处理 media_items 表的迁移兼容性 +// 存储语义历史变更： +// +// v1: folder_path 为文件完整路径 +// v2: folder_path 恒定为扫描根路径 +// v3（当前）: folder_path 为文件/文件夹的实际父目录，供前端拼接播放地址 +// +// 唯一性由 folder_path + file_name + album_name 组合索引保证。 +// +// 检测到 v1/v2 的旧表（缺少新组合索引）时： +// - 默认不动数据，只输出醒目的告警，并让 AutoMigrate 在旧表上补建新索引； +// - 管理员设置环境变量 OPENLIST_MEDIA_MIGRATE_WIPE=1 后才会清空旧数据并重扫。 +func migrateMediaItems() error { + table, err := mediaItemsTable() + if err != nil { + return err + } + m := db.Migrator() + if !m.HasTable(table) { + return nil + } + const idx = "idx_media_folder_file_album" + if m.HasIndex(table, idx) { + // 已是新索引：仅清理软删除残留数据，避免与唯一索引冲突 + if err := db.Exec("DELETE FROM "+quoteIdent(table)+" WHERE deleted_at IS NOT NULL").Error; err != nil { + log.Warnf("media_items: 清理软删除残留数据失败: %v", err) + } + return nil + } + + // 旧表存在但没有新组合索引：旧数据的 folder_path 语义已变更，无法复用。 + if !mediaWipeConfirmed() { + log.Warnf("media_items: 检测到旧版本数据（表 %s 缺少索引 %s）。", table, idx) + log.Warnf("media_items: 旧数据的路径语义已变更，建议清空后重新扫描；为避免误删，本次启动不会自动清库。") + log.Warnf("media_items: 如确认清空，请设置环境变量 %s=1 后重启；也可在管理后台对各媒体库点击「清空数据库」再重扫。", MediaMigrateWipeEnv) + return nil + } + log.Warnf("media_items: 已确认 %s，清空旧版本数据（表 %s）后重新迁移", MediaMigrateWipeEnv, table) + // 先尝试删除旧的单字段唯一索引（如果存在），避免 AutoMigrate 冲突 + oldIdx := "idx_" + table + "_folder_path" + if m.HasIndex(table, oldIdx) { + if err := m.DropIndex(table, oldIdx); err != nil { + log.Warnf("media_items: 删除旧唯一索引失败: %v", err) + } + } + if err := db.Exec("DELETE FROM " + quoteIdent(table)).Error; err != nil { + return err + } + return nil +} + +// quoteIdent 按当前数据库方言给标识符加引号，表名来自 gorm schema 解析而非用户输入。 +func quoteIdent(name string) string { + var sb strings.Builder + db.Dialector.QuoteTo(&sb, name) + return sb.String() } func AutoMigrate(dst ...interface{}) error { diff --git a/internal/db/db_media_migrate_test.go b/internal/db/db_media_migrate_test.go new file mode 100644 index 00000000..a270f344 --- /dev/null +++ b/internal/db/db_media_migrate_test.go @@ -0,0 +1,123 @@ +package db + +import ( + "testing" + + "github.com/glebarez/sqlite" + "gorm.io/gorm" + "gorm.io/gorm/schema" + + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +func openTestDB(t *testing.T, prefix string) *gorm.DB { + t.Helper() + d, err := gorm.Open(sqlite.Open("file::memory:"), &gorm.Config{ + NamingStrategy: schema.NamingStrategy{TablePrefix: prefix}, + }) + if err != nil { + t.Fatal(err) + } + return d +} + +// 旧版表：只有 media_items 的基础结构，没有新的组合唯一索引。 +func createLegacyMediaTable(t *testing.T, d *gorm.DB, table string) { + t.Helper() + if err := d.Exec("CREATE TABLE " + table + " (id integer primary key autoincrement, deleted_at datetime, folder_path text, file_name text)").Error; err != nil { + t.Fatal(err) + } + if err := d.Exec("INSERT INTO " + table + " (folder_path, file_name) VALUES ('/a/b.mp3', 'b.mp3')").Error; err != nil { + t.Fatal(err) + } +} + +func countRows(t *testing.T, d *gorm.DB, table string) int64 { + t.Helper() + var n int64 + if err := d.Table(table).Count(&n).Error; err != nil { + t.Fatal(err) + } + return n +} + +func TestMediaItemsTableHonorsPrefix(t *testing.T) { + for _, prefix := range []string{"x_", "ol_", ""} { + db = openTestDB(t, prefix) + got, err := mediaItemsTable() + if err != nil { + t.Fatal(err) + } + if want := prefix + "media_items"; got != want { + t.Errorf("prefix %q: table = %q, want %q", prefix, got, want) + } + } +} + +func TestMigrateKeepsLegacyDataWithoutConfirmation(t *testing.T) { + t.Setenv(MediaMigrateWipeEnv, "") + db = openTestDB(t, "ol_") // 非默认前缀：旧实现硬编码 x_ 会直接跳过迁移 + createLegacyMediaTable(t, db, "ol_media_items") + + if err := migrateMediaItems(); err != nil { + t.Fatal(err) + } + if n := countRows(t, db, "ol_media_items"); n != 1 { + t.Fatalf("legacy rows must be preserved without confirmation, got %d", n) + } +} + +func TestMigrateWipesOnlyWhenConfirmed(t *testing.T) { + t.Setenv(MediaMigrateWipeEnv, "1") + db = openTestDB(t, "ol_") + createLegacyMediaTable(t, db, "ol_media_items") + + if err := migrateMediaItems(); err != nil { + t.Fatal(err) + } + if n := countRows(t, db, "ol_media_items"); n != 0 { + t.Fatalf("legacy rows should be wiped after confirmation, got %d", n) + } +} + +func TestAutoMigrateCreatesCompositeUniqueIndex(t *testing.T) { + db = openTestDB(t, "ol_") + if err := db.AutoMigrate(&model.MediaItem{}); err != nil { + t.Fatal(err) + } + if !db.Migrator().HasIndex("ol_media_items", "idx_media_folder_file_album") { + t.Fatal("composite unique index missing after AutoMigrate") + } + // 已有新索引时迁移不应触碰有效数据 + if err := db.Create(&model.MediaItem{FolderPath: "/m", FileName: "a.mp3"}).Error; err != nil { + t.Fatal(err) + } + if err := migrateMediaItems(); err != nil { + t.Fatal(err) + } + if n := countRows(t, db, "ol_media_items"); n != 1 { + t.Fatalf("valid rows must survive migration, got %d", n) + } +} + +func TestCreateOrUpdateRejectsOverlongIndexColumns(t *testing.T) { + db = openTestDB(t, "ol_") + if err := db.AutoMigrate(&model.MediaItem{}); err != nil { + t.Fatal(err) + } + long := make([]rune, model.MediaIndexColumnMax+1) + for i := range long { + long[i] = '文' + } + err := CreateOrUpdateMediaItem(&model.MediaItem{FolderPath: "/m", FileName: string(long)}) + if err == nil { + t.Fatal("overlong file_name must be rejected") + } + ok := make([]rune, model.MediaIndexColumnMax) + for i := range ok { + ok[i] = '文' + } + if err := CreateOrUpdateMediaItem(&model.MediaItem{FolderPath: "/m", FileName: string(ok)}); err != nil { + t.Fatalf("boundary length must be accepted: %v", err) + } +} diff --git a/internal/db/media.go b/internal/db/media.go new file mode 100644 index 00000000..9f7e92dc --- /dev/null +++ b/internal/db/media.go @@ -0,0 +1,626 @@ +package db + +import ( + "encoding/json" + "fmt" + "strings" + "time" + "unicode/utf8" + + "github.com/OpenListTeam/OpenList/v4/internal/model" + "gorm.io/gorm" +) + +// ==================== MediaConfig ==================== + +// GetMediaConfig 获取指定类型的媒体库配置，不存在则返回默认值 +func GetMediaConfig(mediaType model.MediaType) (*model.MediaConfig, error) { + var cfg model.MediaConfig + result := db.Where("media_type = ?", mediaType).First(&cfg) + if result.Error == gorm.ErrRecordNotFound { + return &model.MediaConfig{ + MediaType: mediaType, + Enabled: false, + }, nil + } + return &cfg, result.Error +} + +// GetAllMediaConfigs 获取所有媒体库配置 +func GetAllMediaConfigs() ([]model.MediaConfig, error) { + var cfgs []model.MediaConfig + err := db.Find(&cfgs).Error + return cfgs, err +} + +// SaveMediaConfig 保存媒体库配置（upsert） +func SaveMediaConfig(cfg *model.MediaConfig) error { + var existing model.MediaConfig + result := db.Where("media_type = ?", cfg.MediaType).First(&existing) + if result.Error == gorm.ErrRecordNotFound { + return db.Create(cfg).Error + } + cfg.ID = existing.ID + return db.Save(cfg).Error +} + +// ==================== MediaScanPath ==================== + +// ListMediaScanPaths 获取指定媒体类型的所有扫描路径 +func ListMediaScanPaths(mediaType model.MediaType) ([]model.MediaScanPath, error) { + var paths []model.MediaScanPath + tx := db.Model(&model.MediaScanPath{}) + if mediaType != "" { + tx = tx.Where("media_type = ?", mediaType) + } + err := tx.Order("id asc").Find(&paths).Error + return paths, err +} + +// GetMediaScanPath 按ID获取扫描路径 +func GetMediaScanPath(id uint) (*model.MediaScanPath, error) { + var p model.MediaScanPath + err := db.First(&p, id).Error + return &p, err +} + +// CreateMediaScanPath 创建扫描路径 +func CreateMediaScanPath(p *model.MediaScanPath) error { + return db.Create(p).Error +} + +// UpdateMediaScanPath 更新扫描路径 +func UpdateMediaScanPath(p *model.MediaScanPath) error { + return db.Save(p).Error +} + +// DeleteMediaScanPath 删除扫描路径（硬删除） +func DeleteMediaScanPath(id uint) error { + return db.Unscoped().Delete(&model.MediaScanPath{}, id).Error +} + +// ==================== MediaItem ==================== + +// escapeLike 转义 LIKE 通配符，避免路径中的 % _ 影响前缀匹配 +func escapeLike(s string) string { + s = strings.ReplaceAll(s, `\`, `\\`) + s = strings.ReplaceAll(s, "%", `\%`) + s = strings.ReplaceAll(s, "_", `\_`) + return s +} + +// MediaItemQuery 媒体条目查询参数 +type MediaItemQuery struct { + MediaType model.MediaType + ScanPathID uint // 按扫描路径ID筛选 + FolderPath string // 按文件夹路径筛选 + TypeTag string // 按类型标签筛选（电影、电视剧等） + ContentTag string // 按内容标签筛选（喜剧、惊悚等） + PathPrefix string // 按文件夹路径前缀筛选（用于用户 base_path 隔离） + Hidden *bool + Keyword string + OrderBy string // "name", "date", "size" + OrderDir string // "asc", "desc" + Page int + PageSize int +} + +// ListMediaItems 分页查询媒体条目 +func ListMediaItems(q MediaItemQuery) ([]model.MediaItem, int64, error) { + tx := db.Model(&model.MediaItem{}) + if q.MediaType != "" { + tx = tx.Where("media_type = ?", q.MediaType) + } + if q.ScanPathID > 0 { + tx = tx.Where("scan_path_id = ?", q.ScanPathID) + } + if q.FolderPath != "" { + tx = tx.Where("folder_path = ?", q.FolderPath) + } + if q.PathPrefix != "" { + prefix := strings.TrimSuffix(q.PathPrefix, "/") + "/" + tx = tx.Where(`folder_path LIKE ? ESCAPE '\'`, escapeLike(prefix)+"%") + } + if q.Hidden != nil { + tx = tx.Where("hidden = ?", *q.Hidden) + } + if q.Keyword != "" { + like := "%" + q.Keyword + "%" + tx = tx.Where("file_name LIKE ? OR scraped_name LIKE ?", like, like) + } + // 按类型标签筛选（通过关联扫描路径的type_tag） + if q.TypeTag != "" { + tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id"). + Where("media_scan_paths.type_tag = ?", q.TypeTag) + } + // 按内容标签筛选（通过关联扫描路径的content_tags） + if q.ContentTag != "" { + tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id"). + Where("media_scan_paths.content_tags LIKE ?", "%"+q.ContentTag+"%") + } + + var total int64 + if err := tx.Count(&total).Error; err != nil { + return nil, 0, err + } + + // 排序 + orderCol := "created_at" + switch q.OrderBy { + case "name": + orderCol = "COALESCE(NULLIF(scraped_name,''), file_name)" + case "date": + orderCol = "release_date" + case "size": + orderCol = "file_size" + } + dir := "asc" + if q.OrderDir == "desc" { + dir = "desc" + } + tx = tx.Order(orderCol + " " + dir) + + // 分页 + if q.PageSize <= 0 { + q.PageSize = 20 + } + if q.Page <= 0 { + q.Page = 1 + } + tx = tx.Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize) + + var items []model.MediaItem + err := tx.Find(&items).Error + return items, total, err +} + +// GetMediaItemByID 按ID获取媒体条目 +func GetMediaItemByID(id uint) (*model.MediaItem, error) { + var item model.MediaItem + err := db.First(&item, id).Error + return &item, err +} + +// GetMediaItemByFolderPath 按文件夹路径获取媒体条目（用于合并模式） +func GetMediaItemByFolderPath(folderPath string) (*model.MediaItem, error) { + var item model.MediaItem + result := db.Where("folder_path = ?", folderPath).First(&item) + return &item, result.Error +} + +// CreateOrUpdateMediaItem 创建或更新媒体条目（按 folder_path + file_name + album_name 组合唯一） +// 更新时保留已有的刮削数据，避免重新扫描时把已刮削的字段清空 +// +// 注意：使用 Unscoped() 查询是为了把软删除残留记录也包含进来， +// 否则唯一索引 idx_media_folder_file_album 会与软删除行冲突， +// 导致新建条目时报 UNIQUE constraint failed。 +func CreateOrUpdateMediaItem(item *model.MediaItem) error { + // 复合唯一索引三列在 MySQL 下限定为 191 字符（见 model.MediaItem）。 + // 超长值在严格模式下会写入失败、非严格模式下会被静默截断并造成不同文件撞同一条目， + // 因此显式拒绝，由调用方记录告警并跳过。 + if utf8.RuneCountInString(item.FolderPath) > model.MediaIndexColumnMax || + utf8.RuneCountInString(item.FileName) > model.MediaIndexColumnMax || + utf8.RuneCountInString(item.AlbumName) > model.MediaIndexColumnMax { + return fmt.Errorf("media item path/name exceeds %d characters, skipped", model.MediaIndexColumnMax) + } + var existing model.MediaItem + result := db.Unscoped().Where("folder_path = ? AND file_name = ? AND album_name = ?", item.FolderPath, item.FileName, item.AlbumName).First(&existing) + if result.Error == gorm.ErrRecordNotFound { + return db.Create(item).Error + } + if result.Error != nil { + return result.Error + } + item.ID = existing.ID + item.CreatedAt = existing.CreatedAt + // 复用已有记录时清除软删除标记，确保该条目“恢复”为正常记录 + item.DeletedAt = gorm.DeletedAt{} + // 如果已有刮削数据，保留刮削字段，防止重新扫描时覆盖刮削结果 + if existing.ScrapedAt != nil { + item.ScrapedAt = existing.ScrapedAt + item.ScrapedName = existing.ScrapedName + item.Cover = existing.Cover + item.AlbumName = existing.AlbumName + item.AlbumArtist = existing.AlbumArtist + item.TrackNumber = existing.TrackNumber + item.Duration = existing.Duration + item.Genre = existing.Genre + item.ReleaseDate = existing.ReleaseDate + item.Rating = existing.Rating + item.Plot = existing.Plot + item.Authors = existing.Authors + item.Description = existing.Description + item.Publisher = existing.Publisher + item.ISBN = existing.ISBN + item.ExternalID = existing.ExternalID + } + // 扫描不覆盖用户手动隐藏的状态 + item.Hidden = existing.Hidden + return db.Unscoped().Save(item).Error +} + +// UpdateMediaItem 更新媒体条目 +// 使用 Unscoped 确保即使存在软删除标记也能正确更新（避免命中唯一索引但行不可见的诡异情况）。 +func UpdateMediaItem(item *model.MediaItem) error { + return db.Unscoped().Save(item).Error +} + +// DeleteMediaItem 硬删除媒体条目（真正从数据库删除），并清理该条目的批注 +func DeleteMediaItem(id uint) error { + return db.Transaction(func(tx *gorm.DB) error { + if err := tx.Unscoped().Delete(&model.MediaItem{}, id).Error; err != nil { + return err + } + return tx.Where("item_id = ?", id).Delete(&model.MediaAnnotation{}).Error + }) +} + +// ClearMediaItems 硬删除指定类型的所有媒体条目（真正从数据库删除），并清理对应批注 +func ClearMediaItems(mediaType model.MediaType) error { + return db.Transaction(func(tx *gorm.DB) error { + ids := []uint{} + if err := tx.Unscoped().Model(&model.MediaItem{}). + Where("media_type = ?", mediaType).Pluck("id", &ids).Error; err != nil { + return err + } + if err := tx.Unscoped().Where("media_type = ?", mediaType). + Delete(&model.MediaItem{}).Error; err != nil { + return err + } + if len(ids) == 0 { + return nil + } + return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error + }) +} + +// ClearMediaItemsByScanPath 硬删除指定扫描路径的所有媒体条目，并清理对应批注 +func ClearMediaItemsByScanPath(scanPathID uint) error { + return db.Transaction(func(tx *gorm.DB) error { + ids := []uint{} + if err := tx.Unscoped().Model(&model.MediaItem{}). + Where("scan_path_id = ?", scanPathID).Pluck("id", &ids).Error; err != nil { + return err + } + if err := tx.Unscoped().Where("scan_path_id = ?", scanPathID). + Delete(&model.MediaItem{}).Error; err != nil { + return err + } + if len(ids) == 0 { + return nil + } + return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error + }) +} + +// ClearMediaScrapedData 清空指定类型的所有刮削数据（保留扫描出的文件记录本身） +// 仅清空刮削结果相关字段，不删除条目，便于重新刮削。 +// mediaType 为空时表示对所有类型生效。 +func ClearMediaScrapedData(mediaType model.MediaType) (int64, error) { + tx := db.Unscoped().Model(&model.MediaItem{}) + if mediaType != "" { + tx = tx.Where("media_type = ?", mediaType) + } + updates := map[string]interface{}{ + "scraped_name": "", + "description": "", + "cover": "", + "release_date": "", + "rating": 0, + "genre": "", + "authors": "", + "plot": "", + "reviews": "", + "external_id": "", + "album_artist": "", + "publisher": "", + "isbn": "", + "lyrics": "", + "scraped_at": nil, + } + result := tx.Updates(updates) + return result.RowsAffected, result.Error +} + +// ListAllValidMediaItemsForCheck 列出指定类型下所有未删除的媒体条目（仅取必要字段，用于失效检查） +// mediaType 为空时返回所有类型条目。 +func ListAllValidMediaItemsForCheck(mediaType model.MediaType) ([]model.MediaItem, error) { + var items []model.MediaItem + tx := db.Model(&model.MediaItem{}). + Select("id, media_type, scan_path_id, file_name, folder_path, is_folder") + if mediaType != "" { + tx = tx.Where("media_type = ?", mediaType) + } + err := tx.Find(&items).Error + return items, err +} + +// DeleteMediaItemsByIDs 按ID列表硬删除媒体条目，并清理对应批注 +func DeleteMediaItemsByIDs(ids []uint) error { + if len(ids) == 0 { + return nil + } + return db.Transaction(func(tx *gorm.DB) error { + if err := tx.Unscoped().Where("id IN ?", ids).Delete(&model.MediaItem{}).Error; err != nil { + return err + } + return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error + }) +} + +// ListAlbums 列出所有专辑（音乐专用） +func ListAlbums(q MediaItemQuery) ([]AlbumInfo, int64, error) { + type albumRow struct { + AlbumName string + AlbumArtist string + Cover string + ReleaseDate string + TrackCount int + ScanPathID uint + } + + // 构建基础查询 + baseQuery := db.Model(&model.MediaItem{}). + Where("media_type = ?", model.MediaTypeMusic) + if q.Hidden != nil { + baseQuery = baseQuery.Where("hidden = ?", *q.Hidden) + } + if q.ScanPathID > 0 { + baseQuery = baseQuery.Where("scan_path_id = ?", q.ScanPathID) + } + if q.Keyword != "" { + like := "%" + q.Keyword + "%" + baseQuery = baseQuery.Where("album_name LIKE ? OR album_artist LIKE ? OR scraped_name LIKE ?", like, like, like) + } + + // 统计分组数（用子查询） + var total int64 + if err := db.Table("(?) as sub", baseQuery. + Select("album_name, album_artist"). + Group("album_name, album_artist")). + Count(&total).Error; err != nil { + return nil, 0, err + } + + if q.PageSize <= 0 { + q.PageSize = 20 + } + if q.Page <= 0 { + q.Page = 1 + } + + tx := baseQuery. + Select("album_name, album_artist, MAX(cover) as cover, MAX(release_date) as release_date, COUNT(*) as track_count, MAX(scan_path_id) as scan_path_id"). + Group("album_name, album_artist"). + Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize) + + var rows []albumRow + if err := tx.Scan(&rows).Error; err != nil { + return nil, 0, err + } + + albums := make([]AlbumInfo, len(rows)) + for i, r := range rows { + albums[i] = AlbumInfo{ + AlbumName: r.AlbumName, + AlbumArtist: r.AlbumArtist, + Cover: r.Cover, + ReleaseDate: r.ReleaseDate, + TrackCount: r.TrackCount, + ScanPathID: r.ScanPathID, + } + } + return albums, total, nil +} + +// AlbumInfo 专辑信息 +type AlbumInfo struct { + AlbumName string `json:"album_name"` + AlbumArtist string `json:"album_artist"` + Cover string `json:"cover"` + ReleaseDate string `json:"release_date"` + TrackCount int `json:"track_count"` + ScanPathID uint `json:"scan_path_id"` +} + +// GetAlbumTracks 获取专辑曲目列表 +// 支持两种模式： +// 1. 普通模式（is_folder=false）：直接返回独立文件记录 +// 2. 合并文件夹模式（is_folder=true）：把 episodes 展开成虚拟 MediaItem 列表 +// 展开后每条记录的 folder_path = 原folder_path/file_name（文件夹实际路径），file_name = episode.FileName +func GetAlbumTracks(albumName, albumArtist string) ([]model.MediaItem, error) { + var items []model.MediaItem + tx := db.Where("media_type = ?", model.MediaTypeMusic) + if albumName != "" { + tx = tx.Where("album_name = ?", albumName) + } else { + tx = tx.Where("(album_name = '' OR album_name IS NULL)") + } + if albumArtist != "" { + tx = tx.Where("album_artist = ?", albumArtist) + } + err := tx.Order("track_number asc").Find(&items).Error + if err != nil { + return nil, err + } + + // 展开合并文件夹条目的 episodes + var result []model.MediaItem + for _, item := range items { + if !item.IsFolder || item.Episodes == "" { + result = append(result, item) + continue + } + // 解析 episodes + type EpisodeInfo struct { + FileName string `json:"file_name"` + Index int `json:"index"` + Title string `json:"title"` + } + var eps []EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil || len(eps) == 0 { + // 解析失败则跳过该条目（不返回文件夹本身，避免播放路径错误） + continue + } + // 文件夹实际路径 = folder_path + "/" + file_name + actualDir := strings.TrimRight(item.FolderPath, "/") + "/" + item.FileName + for _, ep := range eps { + track := item // 复制基础信息（封面、专辑名、艺术家等） + track.ID = 0 + track.IsFolder = false + track.FolderPath = actualDir + track.FileName = ep.FileName + track.TrackNumber = ep.Index + track.ScrapedName = ep.Title + track.Episodes = "" + result = append(result, track) + } + } + return result, nil +} + +// ListFolderPaths 列出指定媒体类型下的所有文件夹路径（目录浏览模式） +func ListFolderPaths(mediaType model.MediaType) ([]string, error) { + var paths []string + err := db.Model(&model.MediaItem{}). + Where("media_type = ?", mediaType). + Distinct("folder_path"). + Pluck("folder_path", &paths).Error + return paths, err +} + +// GetUnscrappedItems 获取未刮削或刮削不完整的媒体条目 +// 只要 scraped_at 为空，或 cover/scraped_name/plot 任一为空，就需要重新刮削 +// limit <= 0 表示不限制数量，返回所有未刮削条目 +func GetUnscrappedItems(mediaType model.MediaType, limit int) ([]model.MediaItem, error) { + var items []model.MediaItem + tx := db.Where( + "media_type = ? AND (scraped_at IS NULL OR cover = '' OR cover IS NULL OR scraped_name = '' OR scraped_name IS NULL OR plot = '' OR plot IS NULL)", + mediaType, + ) + if limit > 0 { + tx = tx.Limit(limit) + } + err := tx.Find(&items).Error + return items, err +} + +// ==================== 导入/导出 ==================== + +// ListAllMediaItems 列出指定媒体类型下的所有条目（不分页，用于导出） +// mediaType 为空时返回全部类型的所有条目 +func ListAllMediaItems(mediaType model.MediaType) ([]model.MediaItem, error) { + var items []model.MediaItem + tx := db.Model(&model.MediaItem{}) + if mediaType != "" { + tx = tx.Where("media_type = ?", mediaType) + } + err := tx.Order("id asc").Find(&items).Error + return items, err +} + +// ListMediaItemsByScanPath 列出指定扫描路径下的所有条目（不分页，用于导出） +func ListMediaItemsByScanPath(scanPathID uint) ([]model.MediaItem, error) { + var items []model.MediaItem + err := db.Where("scan_path_id = ?", scanPathID).Order("id asc").Find(&items).Error + return items, err +} + +// ListAllMediaScanPaths 列出所有扫描路径（不区分类型，用于全量导出） +func ListAllMediaScanPaths() ([]model.MediaScanPath, error) { + var paths []model.MediaScanPath + err := db.Order("id asc").Find(&paths).Error + return paths, err +} + +// ImportMediaItems 批量导入媒体条目 +// 策略：按 (folder_path, file_name, album_name) 唯一键 upsert： +// - 已存在则覆盖刮削字段（导入是用户主动行为，覆盖优先于保留） +// - 不存在则新建 +// +// 注意：导入数据中的 ID/CreatedAt/UpdatedAt 会被忽略，由数据库重新分配， +// 避免和现有记录的主键冲突。 +func ImportMediaItems(items []model.MediaItem, scanPathIDOverride *uint) (created, updated int, err error) { + for i := range items { + it := items[i] + // 清空主键和时间戳，让数据库重新生成 + it.ID = 0 + it.CreatedAt = time.Time{} + it.UpdatedAt = time.Time{} + it.DeletedAt = gorm.DeletedAt{} + // 如果指定了覆盖的 scan_path_id（按扫描路径导入场景），则强制覆盖 + if scanPathIDOverride != nil { + it.ScanPathID = *scanPathIDOverride + } + + var existing model.MediaItem + result := db.Unscoped().Where( + "folder_path = ? AND file_name = ? AND album_name = ?", + it.FolderPath, it.FileName, it.AlbumName, + ).First(&existing) + if result.Error == gorm.ErrRecordNotFound { + if e := db.Create(&it).Error; e != nil { + return created, updated, e + } + created++ + continue + } + if result.Error != nil { + return created, updated, result.Error + } + // 覆盖更新 + it.ID = existing.ID + it.CreatedAt = existing.CreatedAt + it.DeletedAt = gorm.DeletedAt{} + if e := db.Unscoped().Save(&it).Error; e != nil { + return created, updated, e + } + updated++ + } + return created, updated, nil +} + +// ImportMediaScanPaths 批量导入扫描路径 +// 策略：按 (media_type, path) 唯一对去重： +// - 已存在则更新名称/标签等可编辑字段 +// - 不存在则新建并返回新ID +// +// 返回值 idMap：导入数据中的原始ID -> 数据库实际ID 的映射， +// 供随后的 MediaItem 导入用于把 scan_path_id 重新指向新ID。 +func ImportMediaScanPaths(paths []model.MediaScanPath) (idMap map[uint]uint, created, updated int, err error) { + idMap = make(map[uint]uint, len(paths)) + for i := range paths { + p := paths[i] + originalID := p.ID + p.ID = 0 + p.CreatedAt = time.Time{} + p.UpdatedAt = time.Time{} + p.DeletedAt = gorm.DeletedAt{} + + var existing model.MediaScanPath + result := db.Where("media_type = ? AND path = ?", p.MediaType, p.Path).First(&existing) + if result.Error == gorm.ErrRecordNotFound { + if e := db.Create(&p).Error; e != nil { + return idMap, created, updated, e + } + idMap[originalID] = p.ID + created++ + continue + } + if result.Error != nil { + return idMap, created, updated, result.Error + } + // 已存在：更新可编辑字段 + existing.Name = p.Name + existing.PathMerge = p.PathMerge + existing.TypeTag = p.TypeTag + existing.ContentTags = p.ContentTags + existing.EnableScrape = p.EnableScrape + if e := db.Save(&existing).Error; e != nil { + return idMap, created, updated, e + } + idMap[originalID] = existing.ID + updated++ + } + return idMap, created, updated, nil +} diff --git a/internal/db/media_extra.go b/internal/db/media_extra.go new file mode 100644 index 00000000..b9e43497 --- /dev/null +++ b/internal/db/media_extra.go @@ -0,0 +1,223 @@ +package db + +import ( + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/model" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +// ==================== MediaAppKey ==================== + +func CreateMediaAppKey(k *model.MediaAppKey) error { + return db.Create(k).Error +} + +func ListMediaAppKeys(userID uint) ([]model.MediaAppKey, error) { + var keys []model.MediaAppKey + err := db.Where("user_id = ?", userID).Order("id asc").Find(&keys).Error + return keys, err +} + +func GetMediaAppKey(id uint) (*model.MediaAppKey, error) { + var k model.MediaAppKey + err := db.First(&k, id).Error + return &k, err +} + +func DeleteMediaAppKey(userID, id uint) error { + return db.Where("id = ? AND user_id = ?", id, userID).Delete(&model.MediaAppKey{}).Error +} + +func TouchMediaAppKey(id uint) { + now := time.Now() + _ = db.Model(&model.MediaAppKey{}).Where("id = ?", id).Update("last_used_at", now).Error +} + +// ListMediaAppKeysByPrefix 认证时按明文前缀预筛选候选密钥 +func ListMediaAppKeysByPrefix(prefix string) ([]model.MediaAppKey, error) { + var keys []model.MediaAppKey + err := db.Where("prefix = ?", prefix).Find(&keys).Error + return keys, err +} + +// ==================== MediaUserData ==================== + +func GetMediaUserData(userID, itemID uint, episodeIndex int) (*model.MediaUserData, error) { + var d model.MediaUserData + err := db.Where("user_id = ? AND item_id = ? AND episode_index = ?", userID, itemID, episodeIndex).First(&d).Error + if err != nil { + return nil, err + } + return &d, nil +} + +func SaveMediaUserData(d *model.MediaUserData) error { + var existing model.MediaUserData + err := db.Where("user_id = ? AND item_id = ? AND episode_index = ?", d.UserID, d.ItemID, d.EpisodeIndex).First(&existing).Error + if err == gorm.ErrRecordNotFound { + return db.Create(d).Error + } + if err != nil { + return err + } + d.ID = existing.ID + d.CreatedAt = existing.CreatedAt + return db.Save(d).Error +} + +// ListMediaUserDataByItems 批量取用户对一组条目的播放状态 +func ListMediaUserDataByItems(userID uint, itemIDs []uint) ([]model.MediaUserData, error) { + var list []model.MediaUserData + if len(itemIDs) == 0 { + return list, nil + } + err := db.Where("user_id = ? AND item_id IN ?", userID, itemIDs).Find(&list).Error + return list, err +} + +// ListMediaFavorites 用户收藏的条目 +func ListMediaFavorites(userID uint) ([]model.MediaUserData, error) { + var list []model.MediaUserData + err := db.Where("user_id = ? AND is_favorite = ?", userID, true).Order("updated_at desc").Find(&list).Error + return list, err +} + +// ListMediaResume 继续观看：未看完且有进度 +func ListMediaResume(userID uint, limit int) ([]model.MediaUserData, error) { + var list []model.MediaUserData + if limit <= 0 { + limit = 50 + } + err := db.Where("user_id = ? AND played = ? AND position_ticks > 0", userID, false). + Order("last_played_at desc").Limit(limit).Find(&list).Error + return list, err +} + +// ClearMediaUserDataByUser 删除用户全部播放状态 +func ClearMediaUserDataByUser(userID uint) error { + return db.Where("user_id = ?", userID).Delete(&model.MediaUserData{}).Error +} + +// ==================== MediaPlaylist ==================== + +func CreateMediaPlaylist(p *model.MediaPlaylist) error { + return db.Create(p).Error +} + +func GetMediaPlaylist(id uint) (*model.MediaPlaylist, error) { + var p model.MediaPlaylist + err := db.First(&p, id).Error + return &p, err +} + +func ListMediaPlaylists(userID uint) ([]model.MediaPlaylist, error) { + var list []model.MediaPlaylist + err := db.Where("user_id = ?", userID).Order("id asc").Find(&list).Error + return list, err +} + +func UpdateMediaPlaylist(p *model.MediaPlaylist) error { + return db.Save(p).Error +} + +func DeleteMediaPlaylist(id uint) error { + if err := db.Where("playlist_id = ?", id).Delete(&model.MediaPlaylistItem{}).Error; err != nil { + return err + } + return db.Delete(&model.MediaPlaylist{}, id).Error +} + +// ReplaceMediaPlaylistItems 覆盖式写入播放列表条目 +func ReplaceMediaPlaylistItems(playlistID uint, items []model.MediaPlaylistItem) error { + if err := db.Where("playlist_id = ?", playlistID).Delete(&model.MediaPlaylistItem{}).Error; err != nil { + return err + } + if len(items) == 0 { + return nil + } + for i := range items { + items[i].ID = 0 + items[i].PlaylistID = playlistID + items[i].Order = i + } + return db.Create(&items).Error +} + +func ListMediaPlaylistItems(playlistID uint) ([]model.MediaPlaylistItem, error) { + var items []model.MediaPlaylistItem + err := db.Where("playlist_id = ?", playlistID).Order("sort_order asc").Find(&items).Error + return items, err +} + +// CountMediaPlaylistItems 统计条目数 +func CountMediaPlaylistItems(playlistID uint) (int64, error) { + var n int64 + err := db.Model(&model.MediaPlaylistItem{}).Where("playlist_id = ?", playlistID).Count(&n).Error + return n, err +} + +// ==================== EmbyDevice ==================== + +func CreateEmbyDevice(d *model.EmbyDevice) error { + return db.Create(d).Error +} + +func GetEmbyDeviceByTokenHash(hash string) (*model.EmbyDevice, error) { + var d model.EmbyDevice + err := db.Where("token_hash = ?", hash).First(&d).Error + if err != nil { + return nil, err + } + return &d, nil +} + +func DeleteEmbyDevice(userID uint, deviceID string) error { + tx := db.Where("user_id = ?", userID) + if deviceID != "" { + tx = tx.Where("device_id = ?", deviceID) + } + return tx.Delete(&model.EmbyDevice{}).Error +} + +func TouchEmbyDevice(id uint) { + _ = db.Model(&model.EmbyDevice{}).Where("id = ?", id).Update("last_seen_at", time.Now()).Error +} + +// ==================== MediaAnnotation ==================== + +// ListMediaAnnotations 列出某用户对某个条目的全部批注 +func ListMediaAnnotations(userID, itemID uint) ([]model.MediaAnnotation, error) { + list := []model.MediaAnnotation{} + err := db.Where("user_id = ? AND item_id = ?", userID, itemID). + Order("id asc").Find(&list).Error + return list, err +} + +// SyncMediaAnnotations 增量保存批注：upserts 覆盖同一 remote_id 的记录，deletes 按 remote_id 删除。 +// 放在一个事务里，避免多设备同时保存时出现半新半旧的状态。 +func SyncMediaAnnotations(userID, itemID uint, upserts []model.MediaAnnotation, deletes []string) error { + return db.Transaction(func(tx *gorm.DB) error { + for i := range upserts { + row := upserts[i] + row.ID = 0 + row.UserID, row.ItemID = userID, itemID + if err := tx.Clauses(clause.OnConflict{ + Columns: []clause.Column{ + {Name: "user_id"}, {Name: "item_id"}, {Name: "remote_id"}, + }, + DoUpdates: clause.AssignmentColumns([]string{"anchor", "updated_at"}), + }).Create(&row).Error; err != nil { + return err + } + } + if len(deletes) > 0 { + if err := tx.Where("user_id = ? AND item_id = ? AND remote_id IN ?", userID, itemID, deletes). + Delete(&model.MediaAnnotation{}).Error; err != nil { + return err + } + } + return nil + }) +} diff --git a/internal/media/appkey.go b/internal/media/appkey.go new file mode 100644 index 00000000..6e4edad4 --- /dev/null +++ b/internal/media/appkey.go @@ -0,0 +1,120 @@ +package media + +import ( + "crypto/aes" + "crypto/cipher" + "crypto/md5" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "errors" + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" +) + +const appKeyPrefixLen = 8 + +// AppKeySecretLen 明文密钥长度（不含前缀分隔符） +const appKeySecretLen = 32 + +// NewAppKeyPlain 生成一个新的应用密钥明文，格式：olmp_<40 位随机串> +func NewAppKeyPlain() (string, error) { + buf := make([]byte, appKeySecretLen) + if _, err := rand.Read(buf); err != nil { + return "", err + } + body := strings.TrimRight(base64.RawURLEncoding.EncodeToString(buf), "=") + if len(body) > 40 { + body = body[:40] + } + return "olmp_" + body, nil +} + +// AppKeyPrefix 返回用于检索/展示的明文前缀 +func AppKeyPrefix(plain string) string { + if len(plain) <= appKeyPrefixLen { + return plain + } + return plain[:appKeyPrefixLen] +} + +func appKeyCipherKey() []byte { + seed := conf.Conf.JwtSecret + if seed == "" { + seed = setting.GetStr(conf.Token) + } + sum := sha256.Sum256([]byte("openlist-media-appkey:" + seed)) + return sum[:] +} + +func newGCM() (cipher.AEAD, error) { + block, err := aes.NewCipher(appKeyCipherKey()) + if err != nil { + return nil, err + } + return cipher.NewGCM(block) +} + +// EncryptAppKey 加密应用密钥明文 +func EncryptAppKey(plain string) (string, error) { + gcm, err := newGCM() + if err != nil { + return "", err + } + nonce := make([]byte, gcm.NonceSize()) + if _, err := rand.Read(nonce); err != nil { + return "", err + } + out := gcm.Seal(nonce, nonce, []byte(plain), nil) + return base64.RawURLEncoding.EncodeToString(out), nil +} + +// DecryptAppKey 解密应用密钥明文 +func DecryptAppKey(secret string) (string, error) { + raw, err := base64.RawURLEncoding.DecodeString(secret) + if err != nil { + return "", err + } + gcm, err := newGCM() + if err != nil { + return "", err + } + if len(raw) < gcm.NonceSize() { + return "", errors.New("invalid app key secret") + } + nonce, data := raw[:gcm.NonceSize()], raw[gcm.NonceSize():] + plain, err := gcm.Open(nil, nonce, data, nil) + if err != nil { + return "", err + } + return string(plain), nil +} + +// HashToken 计算设备令牌的存储哈希 +func HashToken(token string) string { + sum := sha256.Sum256([]byte("openlist-emby-token:" + token)) + return hex.EncodeToString(sum[:]) +} + +// NewDeviceToken 生成 Emby 设备令牌 +func NewDeviceToken() (string, error) { + buf := make([]byte, 24) + if _, err := rand.Read(buf); err != nil { + return "", err + } + return hex.EncodeToString(buf), nil +} + +// SubsonicToken 计算 Subsonic 的 t = md5(password + salt) +func SubsonicToken(password, salt string) string { + sum := md5.Sum([]byte(password + salt)) + return hex.EncodeToString(sum[:]) +} + +// MatchSubsonicToken 校验客户端提交的 t 是否匹配 +func MatchSubsonicToken(password, salt, token string) bool { + return strings.EqualFold(SubsonicToken(password, salt), token) +} diff --git a/internal/media/id3.go b/internal/media/id3.go new file mode 100644 index 00000000..08acd080 --- /dev/null +++ b/internal/media/id3.go @@ -0,0 +1,684 @@ +package media + +import ( + "encoding/binary" + "io" + "strconv" + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/media/imgmime" +) + +// maxEmbeddedPictureBytes 内嵌封面图片最大字节数，防止畸形文件触发超大内存分配 +const maxEmbeddedPictureBytes = 16 << 20 + +// MusicTag 音频文件标签信息 +type MusicTag struct { + Title string // TIT2 + Artist string // TPE1 + Album string // TALB + AlbumArtist string // TPE2 + TrackNumber int // TRCK + Year string // TYER / TDRC + Genre string // TCON + CoverData []byte // 封面图片原始字节（APIC / PICTURE） + CoverMIME string // 封面图片 MIME 类型（如 image/jpeg） + Lyrics string // 歌词内容（LRC 文本或纯文本）：来自 USLT / SYLT / Vorbis LYRICS +} + +// ParseID3v2 从 io.Reader 中解析 ID3v2 标签（只读取文件头部，不需要 Seek） +// 支持 ID3v2.3 和 ID3v2.4 +func ParseID3v2(r io.Reader) (*MusicTag, error) { + // 读取 ID3v2 头部（10 字节） + header := make([]byte, 10) + if _, err := io.ReadFull(r, header); err != nil { + return nil, err + } + + // 检查魔数 + if string(header[0:3]) != "ID3" { + return nil, nil // 不是 ID3v2 文件，不报错 + } + + version := header[3] // 主版本号：3 = ID3v2.3, 4 = ID3v2.4 + if version < 3 || version > 4 { + return nil, nil // 只支持 v2.3 和 v2.4 + } + + // 解析标签总大小（syncsafe integer，4 字节，每字节最高位为 0） + tagSize := syncsafeToInt(header[6:10]) + if tagSize <= 0 || tagSize > 10*1024*1024 { // 最大 10MB + return nil, nil + } + + // 读取所有帧数据 + data := make([]byte, tagSize) + if _, err := io.ReadFull(r, data); err != nil { + return nil, nil // 读取失败时静默跳过 + } + + tag := &MusicTag{} + pos := 0 + + // 跳过扩展头部（如果有） + flags := header[5] + if flags&0x40 != 0 { // 有扩展头部 + if pos+4 > len(data) { + return tag, nil + } + var extSize int + if version == 4 { + extSize = syncsafeToInt(data[pos : pos+4]) + } else { + extSize = int(binary.BigEndian.Uint32(data[pos : pos+4])) + } + pos += extSize + } + + // 解析帧 + for pos+10 <= len(data) { + frameID := string(data[pos : pos+4]) + if frameID == "\x00\x00\x00\x00" { + break // 填充区域，结束 + } + + // ID3v2.4 帧大小使用 syncsafe integer（与标签头一致）， + // ID3v2.3 使用普通 32 位大端整数。 + var frameSize int + if version == 4 { + frameSize = syncsafeToInt(data[pos+4 : pos+8]) + } else { + frameSize = int(binary.BigEndian.Uint32(data[pos+4 : pos+8])) + } + pos += 10 // 跳过帧头（4+4+2） + + if frameSize <= 0 || pos+frameSize > len(data) { + break + } + + frameData := data[pos : pos+frameSize] + pos += frameSize + + // 解析文本帧（T 开头的帧） + if len(frameID) == 4 && frameID[0] == 'T' && len(frameData) > 0 { + text := decodeID3Text(frameData) + switch frameID { + case "TIT2": + tag.Title = text + case "TPE1": + tag.Artist = text + case "TALB": + tag.Album = text + case "TPE2": + tag.AlbumArtist = text + case "TRCK": + // 格式可能是 "1" 或 "1/12" + parts := strings.SplitN(text, "/", 2) + if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil { + tag.TrackNumber = n + } + case "TYER", "TDRC": + if tag.Year == "" { + tag.Year = text + } + case "TCON": + tag.Genre = parseID3Genre(text) + } + } + + // 解析 APIC 帧（内嵌封面图片），只取第一张 + if frameID == "APIC" && tag.CoverData == nil && len(frameData) > 1 { + tag.CoverData, tag.CoverMIME = parseAPICFrame(frameData) + } + + // 解析 USLT 帧（非同步歌词） + // 格式：编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本 + if frameID == "USLT" && tag.Lyrics == "" && len(frameData) > 5 { + tag.Lyrics = parseUSLTFrame(frameData) + } + + // 解析 SYLT 帧（同步歌词，二进制时间戳格式），仅在没有 USLT 时尝试转 LRC + if frameID == "SYLT" && tag.Lyrics == "" && len(frameData) > 6 { + if lrc := parseSYLTFrame(frameData); lrc != "" { + tag.Lyrics = lrc + } + } + } + + return tag, nil +} + +// parseUSLTFrame 解析 ID3v2 USLT 帧（非同步歌词） +// 格式：编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本 +func parseUSLTFrame(data []byte) string { + if len(data) < 5 { + return "" + } + encoding := data[0] + // 跳过编码(1) + 语言(3) = 4 字节 + pos := 4 + + // 跳过描述字符串 + if encoding == 0x01 || encoding == 0x02 { + // UTF-16：找 \x00\x00 结尾 + for pos+1 < len(data) { + if data[pos] == 0 && data[pos+1] == 0 { + pos += 2 + break + } + pos += 2 + } + } else { + // ISO-8859-1 / UTF-8：找单个 \x00 结尾 + for pos < len(data) { + if data[pos] == 0 { + pos++ + break + } + pos++ + } + } + + if pos >= len(data) { + return "" + } + + // 剩余字节就是歌词文本，按编码解码（复用 decodeID3Text 但需要把 encoding 字节加回头部） + lyricsRaw := make([]byte, 1+len(data)-pos) + lyricsRaw[0] = encoding + copy(lyricsRaw[1:], data[pos:]) + return strings.TrimSpace(decodeID3Text(lyricsRaw)) +} + +// parseSYLTFrame 解析 ID3v2 SYLT 帧（同步歌词），转换为 LRC 格式 +// 格式：编码(1) + 语言(3) + 时间戳格式(1) + 内容类型(1) + 描述(null 结尾) + 多组 (歌词文本 + null + 4字节时间戳) +// 时间戳格式：1=MPEG帧数（不支持），2=毫秒 +func parseSYLTFrame(data []byte) string { + if len(data) < 6 { + return "" + } + encoding := data[0] + timestampFormat := data[4] // 1=MPEG帧, 2=ms + if timestampFormat != 2 { + return "" // 只支持毫秒时间戳 + } + pos := 6 // 跳过 encoding(1) + lang(3) + timestamp_fmt(1) + content_type(1) + + // 跳过描述字符串 + if encoding == 0x01 || encoding == 0x02 { + for pos+1 < len(data) { + if data[pos] == 0 && data[pos+1] == 0 { + pos += 2 + break + } + pos += 2 + } + } else { + for pos < len(data) { + if data[pos] == 0 { + pos++ + break + } + pos++ + } + } + + var sb strings.Builder + wide := encoding == 0x01 || encoding == 0x02 + for pos < len(data) { + // 找文本终止符 + textStart := pos + textEnd := -1 + if wide { + for i := pos; i+1 < len(data); i += 2 { + if data[i] == 0 && data[i+1] == 0 { + textEnd = i + break + } + } + } else { + for i := pos; i < len(data); i++ { + if data[i] == 0 { + textEnd = i + break + } + } + } + if textEnd < 0 { + break + } + // 解码歌词文本 + lineRaw := make([]byte, 1+textEnd-textStart) + lineRaw[0] = encoding + copy(lineRaw[1:], data[textStart:textEnd]) + lineText := decodeID3Text(lineRaw) + + // 跳过文本终止符 + if wide { + pos = textEnd + 2 + } else { + pos = textEnd + 1 + } + // 4 字节时间戳（毫秒） + if pos+4 > len(data) { + break + } + ms := int(binary.BigEndian.Uint32(data[pos : pos+4])) + pos += 4 + + minutes := ms / 60000 + seconds := (ms % 60000) / 1000 + cs := (ms % 1000) / 10 + // LRC 标准格式 [mm:ss.xx] + sb.WriteString("[") + if minutes < 10 { + sb.WriteString("0") + } + sb.WriteString(strconv.Itoa(minutes)) + sb.WriteString(":") + if seconds < 10 { + sb.WriteString("0") + } + sb.WriteString(strconv.Itoa(seconds)) + sb.WriteString(".") + if cs < 10 { + sb.WriteString("0") + } + sb.WriteString(strconv.Itoa(cs)) + sb.WriteString("]") + sb.WriteString(lineText) + sb.WriteString("\n") + } + return sb.String() +} + +// parseAPICFrame 解析 ID3v2 APIC 帧，返回图片字节和 MIME 类型 +// APIC 格式：编码字节(1) + MIME字符串(null结尾) + 图片类型(1) + 描述字符串(null结尾) + 图片数据 +func parseAPICFrame(data []byte) ([]byte, string) { + if len(data) < 4 { + return nil, "" + } + // encoding := data[0] // 文本编码（只影响描述字段，MIME 始终是 ASCII） + pos := 1 + + // 读取 MIME 类型（null 结尾的 ASCII 字符串） + nullIdx := -1 + for i := pos; i < len(data); i++ { + if data[i] == 0 { + nullIdx = i + break + } + } + if nullIdx < 0 { + return nil, "" + } + mimeType := string(data[pos:nullIdx]) + pos = nullIdx + 1 + + if pos >= len(data) { + return nil, "" + } + // 图片类型（1字节，3 = Cover (front)，但我们取任意第一张） + pos++ // 跳过图片类型字节 + + // 跳过描述字符串（null 结尾，编码由第一字节决定） + encoding := data[0] + if encoding == 0x01 || encoding == 0x02 { + // UTF-16：找 \x00\x00 结尾 + for pos+1 < len(data) { + if data[pos] == 0 && data[pos+1] == 0 { + pos += 2 + break + } + pos += 2 + } + } else { + // ISO-8859-1 / UTF-8：找单个 \x00 结尾 + for pos < len(data) { + if data[pos] == 0 { + pos++ + break + } + pos++ + } + } + + if pos >= len(data) { + return nil, "" + } + + imgData := make([]byte, len(data)-pos) + copy(imgData, data[pos:]) + // 声明的 MIME 来自被扫描文件本身，不可信（可能是 image/svg+xml）。 + // 只按魔数判定并限定为四种栅格格式，无法识别则丢弃封面。 + _ = mimeType + safe := imgmime.Sniff(imgData) + if safe == "" { + return nil, "" + } + return imgData, safe +} + +// syncsafeToInt 将 syncsafe integer 转换为普通整数 +func syncsafeToInt(b []byte) int { + result := 0 + for _, v := range b { + result = (result << 7) | int(v&0x7F) + } + return result +} + +// decodeID3Text 解码 ID3 文本帧（第一字节是编码标识） +func decodeID3Text(data []byte) string { + if len(data) == 0 { + return "" + } + encoding := data[0] + content := data[1:] + + // 去掉末尾的 null 字节 + switch encoding { + case 0x00: // ISO-8859-1 + // 去掉末尾 null + content = trimNull(content, false) + // 尝试 UTF-8 解码，如果失败则按 Latin-1 处理 + return latin1ToUTF8(content) + case 0x01: // UTF-16 with BOM + content = trimNull(content, true) + return utf16ToUTF8(content) + case 0x02: // UTF-16 BE without BOM + content = trimNull(content, true) + return utf16BEToUTF8(content) + case 0x03: // UTF-8 + content = trimNull(content, false) + return string(content) + default: + return string(content) + } +} + +// trimNull 去掉末尾的 null 字节 +func trimNull(b []byte, wide bool) []byte { + if wide { + // UTF-16: 去掉末尾的 \x00\x00 + for len(b) >= 2 && b[len(b)-2] == 0 && b[len(b)-1] == 0 { + b = b[:len(b)-2] + } + } else { + for len(b) > 0 && b[len(b)-1] == 0 { + b = b[:len(b)-1] + } + } + return b +} + +// latin1ToUTF8 将 Latin-1 编码转换为 UTF-8 +func latin1ToUTF8(b []byte) string { + runes := make([]rune, len(b)) + for i, v := range b { + runes[i] = rune(v) + } + return string(runes) +} + +// utf16ToUTF8 将 UTF-16（带 BOM）转换为 UTF-8 +func utf16ToUTF8(b []byte) string { + if len(b) < 2 { + return "" + } + var bigEndian bool + if b[0] == 0xFF && b[1] == 0xFE { + bigEndian = false + b = b[2:] + } else if b[0] == 0xFE && b[1] == 0xFF { + bigEndian = true + b = b[2:] + } + return decodeUTF16(b, bigEndian) +} + +// utf16BEToUTF8 将 UTF-16 BE（无 BOM）转换为 UTF-8 +func utf16BEToUTF8(b []byte) string { + return decodeUTF16(b, true) +} + +// decodeUTF16 解码 UTF-16 字节序列 +func decodeUTF16(b []byte, bigEndian bool) string { + if len(b)%2 != 0 { + b = b[:len(b)-1] + } + runes := make([]rune, 0, len(b)/2) + for i := 0; i+1 < len(b); i += 2 { + var code uint16 + if bigEndian { + code = uint16(b[i])<<8 | uint16(b[i+1]) + } else { + code = uint16(b[i+1])<<8 | uint16(b[i]) + } + runes = append(runes, rune(code)) + } + return string(runes) +} + +// parseID3Genre 解析 ID3 流派字段（可能是 "(17)" 格式的数字引用） +func parseID3Genre(s string) string { + s = strings.TrimSpace(s) + if len(s) > 2 && s[0] == '(' && s[len(s)-1] == ')' { + // 数字引用格式，直接返回原始字符串 + return s + } + return s +} + +// ─── FLAC Vorbis Comment 解析 ──────────────────────────────────────────────── + +// ParseFLACVorbisComment 从 io.Reader 中解析 FLAC 文件的 Vorbis Comment 元数据 +// FLAC 格式：4字节魔数 "fLaC" + 若干 METADATA_BLOCK +// METADATA_BLOCK：1字节(最高位=是否最后块, 低7位=块类型) + 3字节长度 + 数据 +// 块类型 4 = VORBIS_COMMENT +func ParseFLACVorbisComment(r io.Reader) (*MusicTag, error) { + // 读取魔数（4字节） + magic := make([]byte, 4) + if _, err := io.ReadFull(r, magic); err != nil { + return nil, err + } + if string(magic) != "fLaC" { + return nil, nil // 不是 FLAC 文件 + } + + // 遍历 METADATA_BLOCK，找到 VORBIS_COMMENT（类型4）和 PICTURE（类型6） + var tag *MusicTag + for { + // 读取块头（4字节：1字节标志+类型 + 3字节长度） + blockHeader := make([]byte, 4) + if _, err := io.ReadFull(r, blockHeader); err != nil { + return nil, nil // 读取失败，静默跳过 + } + + isLast := blockHeader[0]&0x80 != 0 + blockType := blockHeader[0] & 0x7F + blockLen := int(blockHeader[1])<<16 | int(blockHeader[2])<<8 | int(blockHeader[3]) + + if blockLen < 0 || blockLen > 16*1024*1024 { // 最大 16MB + return nil, nil + } + + if blockType == 4 { + // VORBIS_COMMENT 块 + data := make([]byte, blockLen) + if _, err := io.ReadFull(r, data); err != nil { + return nil, nil + } + tag = parseVorbisCommentData(data) + if isLast { + return tag, nil + } + // 继续读取后续块，寻找 PICTURE 块（类型6） + continue + } + + if blockType == 6 { + // PICTURE 块（FLAC 内嵌封面） + data := make([]byte, blockLen) + if _, err := io.ReadFull(r, data); err == nil && tag != nil && tag.CoverData == nil { + tag.CoverData, tag.CoverMIME = parseFLACPictureBlock(data) + } else if err != nil { + // 读取失败，跳过 + _ = err + } + if isLast { + return tag, nil + } + continue + } + + // 跳过此块 + if _, err := io.CopyN(io.Discard, r, int64(blockLen)); err != nil { + return nil, nil + } + + if isLast { + break + } + } + + return tag, nil +} + +// parseFLACPictureBlock 解析 FLAC PICTURE 元数据块，返回图片字节和 MIME 类型 +// 格式（大端序）： +// 4字节 picture_type +// 4字节 mime_length + mime_string +// 4字节 description_length + description_string +// 4字节 width, 4字节 height, 4字节 color_depth, 4字节 color_count +// 4字节 data_length + data +func parseFLACPictureBlock(data []byte) ([]byte, string) { + if len(data) < 8 { + return nil, "" + } + pos := 4 // 跳过 picture_type + + // 读取 MIME 类型 + if pos+4 > len(data) { + return nil, "" + } + mimeLen := int(binary.BigEndian.Uint32(data[pos : pos+4])) + pos += 4 + if pos+mimeLen > len(data) { + return nil, "" + } + mimeType := string(data[pos : pos+mimeLen]) + pos += mimeLen + + // 跳过描述字符串 + if pos+4 > len(data) { + return nil, "" + } + descLen := int(binary.BigEndian.Uint32(data[pos : pos+4])) + pos += 4 + descLen + + // 跳过 width(4) + height(4) + color_depth(4) + color_count(4) + pos += 16 + + // 读取图片数据 + if pos+4 > len(data) { + return nil, "" + } + dataLen := int(binary.BigEndian.Uint32(data[pos : pos+4])) + pos += 4 + if dataLen <= 0 || dataLen > maxEmbeddedPictureBytes || pos+dataLen > len(data) { + return nil, "" + } + + imgData := make([]byte, dataLen) + copy(imgData, data[pos:pos+dataLen]) + // 同 parseAPICFrame：不信任声明的 MIME，只按魔数判定 + _ = mimeType + safe := imgmime.Sniff(imgData) + if safe == "" { + return nil, "" + } + return imgData, safe +} + +// parseVorbisCommentData 解析 Vorbis Comment 数据块 +// 格式（小端序）： +// 4字节 vendor_length + vendor_string +// 4字节 comment_count +// 每条注释：4字节长度 + UTF-8字符串（格式 "KEY=VALUE"） +func parseVorbisCommentData(data []byte) *MusicTag { + tag := &MusicTag{} + pos := 0 + + // 跳过 vendor string + if pos+4 > len(data) { + return tag + } + vendorLen := int(binary.LittleEndian.Uint32(data[pos : pos+4])) + pos += 4 + vendorLen + + // 读取注释数量 + if pos+4 > len(data) { + return tag + } + commentCount := int(binary.LittleEndian.Uint32(data[pos : pos+4])) + pos += 4 + + for i := 0; i < commentCount; i++ { + if pos+4 > len(data) { + break + } + commentLen := int(binary.LittleEndian.Uint32(data[pos : pos+4])) + pos += 4 + + if commentLen <= 0 || pos+commentLen > len(data) { + break + } + + comment := string(data[pos : pos+commentLen]) + pos += commentLen + + // 解析 "KEY=VALUE" 格式（大小写不敏感） + eqIdx := strings.IndexByte(comment, '=') + if eqIdx < 0 { + continue + } + key := strings.ToUpper(comment[:eqIdx]) + value := strings.TrimSpace(comment[eqIdx+1:]) + if value == "" { + continue + } + + switch key { + case "TITLE": + tag.Title = value + case "ARTIST": + if tag.Artist == "" { + tag.Artist = value + } + case "ALBUM": + tag.Album = value + case "ALBUMARTIST", "ALBUM ARTIST": + tag.AlbumArtist = value + case "TRACKNUMBER", "TRACK": + // 格式可能是 "1" 或 "1/12" + parts := strings.SplitN(value, "/", 2) + if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil { + tag.TrackNumber = n + } + case "DATE", "YEAR": + if tag.Year == "" && len(value) >= 4 { + tag.Year = value[:4] + } + case "GENRE": + tag.Genre = value + case "LYRICS", "UNSYNCEDLYRICS", "UNSYNCED LYRICS", "SYNCEDLYRICS", "SYNCED LYRICS": + // FLAC/Vorbis 内嵌歌词字段（不同打标软件使用不同 key，全部兼容） + if tag.Lyrics == "" { + tag.Lyrics = value + } + } + } + + return tag +} diff --git a/internal/media/imgmime/imgmime.go b/internal/media/imgmime/imgmime.go new file mode 100644 index 00000000..a2534b84 --- /dev/null +++ b/internal/media/imgmime/imgmime.go @@ -0,0 +1,36 @@ +// Package imgmime 提供封面图片 MIME 的安全判定。 +// +// 封面字节与 MIME 往往来自不可信来源（音频文件内嵌 APIC/PICTURE、远端站点的 +// Content-Type）。如果把声明的 MIME 原样写进响应头，就可能把 image/svg+xml 或 +// text/html 以站点同源方式交付给浏览器，造成存储型 XSS。 +// 这里只信任文件头魔数，且只允许四种栅格格式。 +package imgmime + +import "bytes" + +// Sniff 根据文件头魔数返回 image/jpeg、image/png、image/gif、image/webp 之一； +// 无法识别（包括 SVG、HTML、未知格式）时返回空串，调用方应丢弃该封面。 +func Sniff(data []byte) string { + switch { + case len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF: + return "image/jpeg" + case len(data) >= 8 && bytes.Equal(data[:8], []byte{0x89, 'P', 'N', 'G', 0x0D, 0x0A, 0x1A, 0x0A}): + return "image/png" + case len(data) >= 6 && (bytes.Equal(data[:6], []byte("GIF87a")) || bytes.Equal(data[:6], []byte("GIF89a"))): + return "image/gif" + case len(data) >= 12 && bytes.Equal(data[:4], []byte("RIFF")) && bytes.Equal(data[8:12], []byte("WEBP")): + return "image/webp" + default: + return "" + } +} + +// Allowed 判断 MIME 是否属于允许对外输出的栅格图片类型。 +func Allowed(mime string) bool { + switch mime { + case "image/jpeg", "image/png", "image/gif", "image/webp": + return true + default: + return false + } +} diff --git a/internal/media/imgmime/imgmime_test.go b/internal/media/imgmime/imgmime_test.go new file mode 100644 index 00000000..70b36d02 --- /dev/null +++ b/internal/media/imgmime/imgmime_test.go @@ -0,0 +1,41 @@ +package imgmime + +import "testing" + +func TestSniff(t *testing.T) { + cases := []struct { + name string + data []byte + want string + }{ + {"jpeg", []byte{0xFF, 0xD8, 0xFF, 0xE0, 0, 0}, "image/jpeg"}, + {"png", []byte{0x89, 'P', 'N', 'G', 0x0D, 0x0A, 0x1A, 0x0A, 0}, "image/png"}, + {"gif89", []byte("GIF89a....."), "image/gif"}, + {"webp", append(append([]byte("RIFF\x00\x00\x00\x00"), []byte("WEBP")...), 0), "image/webp"}, + {"svg", []byte(``), ""}, + {"svg with xml prolog", []byte(``), ""}, + {"html", []byte(""), ""}, + {"empty", nil, ""}, + {"short", []byte{0xFF, 0xD8}, ""}, + // RIFF 但不是 WEBP（例如 WAV）不能被当作图片 + {"riff wav", append(append([]byte("RIFF\x00\x00\x00\x00"), []byte("WAVE")...), 0), ""}, + } + for _, tc := range cases { + if got := Sniff(tc.data); got != tc.want { + t.Errorf("%s: Sniff = %q, want %q", tc.name, got, tc.want) + } + } +} + +func TestAllowed(t *testing.T) { + for _, m := range []string{"image/jpeg", "image/png", "image/gif", "image/webp"} { + if !Allowed(m) { + t.Errorf("%s should be allowed", m) + } + } + for _, m := range []string{"image/svg+xml", "text/html", "", "image/"} { + if Allowed(m) { + t.Errorf("%q should not be allowed", m) + } + } +} diff --git a/internal/media/local_fill.go b/internal/media/local_fill.go new file mode 100644 index 00000000..9c9a4737 --- /dev/null +++ b/internal/media/local_fill.go @@ -0,0 +1,130 @@ +package media + +import ( + "context" + "encoding/base64" + "encoding/json" + stdpath "path" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +// FillMusicFromLocal 在刮削前/失败时，使用音乐文件本身的元数据（ID3 / Vorbis Comment） +// 以及同目录 .lrc 歌词，**只填充 item 中为空的字段**。 +// +// 适用场景： +// 1. 用户清空刮削后重新刮削，原本由扫描阶段填入的本地数据（封面/歌词等）已丢失 +// 2. 在线刮削失败，退而求其次用文件本地信息补全 +// +// 行为： +// - is_folder=true 的合并文件夹模式：使用 episodes 中第一首音乐文件读取 +// - is_folder=false：使用 folder_path/file_name 直接读取 +// +// 错误以静默忽略为主——本地兜底失败不应阻塞主流程。 +func FillMusicFromLocal(ctx context.Context, item *model.MediaItem) { + if item == nil || item.MediaType != model.MediaTypeMusic { + return + } + + musicPath := pickMusicFilePath(item) + if musicPath == "" { + return + } + + // 解析 tag（mp3 / flac / 其它） + ext := strings.ToLower(stdpath.Ext(musicPath)) + var tag *MusicTag + readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second) + if reader := FetchFileReader(readCtx, musicPath); reader != nil { + switch ext { + case ".flac": + tag, _ = ParseFLACVorbisComment(reader) + default: + tag, _ = ParseID3v2(reader) + } + _ = reader.Close() + } + readCancel() + + // 应用 tag 到空字段 + if tag != nil { + if item.AlbumName == "" && tag.Album != "" { + item.AlbumName = tag.Album + } + if item.AlbumArtist == "" { + if tag.AlbumArtist != "" { + item.AlbumArtist = tag.AlbumArtist + } else if tag.Artist != "" { + item.AlbumArtist = tag.Artist + } + } + if item.ScrapedName == "" { + if tag.Title != "" { + item.ScrapedName = tag.Title + } else if item.AlbumName != "" { + item.ScrapedName = item.AlbumName + } + } + if item.TrackNumber == 0 && tag.TrackNumber > 0 { + item.TrackNumber = tag.TrackNumber + } + if item.ReleaseDate == "" && len(tag.Year) >= 4 { + item.ReleaseDate = tag.Year[:4] + "-01-01" + } + if item.Genre == "" && tag.Genre != "" { + item.Genre = tag.Genre + } + if item.Authors == "" && tag.Artist != "" { + if b, err := json.Marshal([]string{tag.Artist}); err == nil { + item.Authors = string(b) + } + } + if item.Cover == "" && len(tag.CoverData) > 0 { + mimeType := tag.CoverMIME + if mimeType == "" { + mimeType = "image/jpeg" + } + item.Cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData) + } + } + + // 歌词：若为空则尝试 .lrc 同名文件 / 内嵌歌词 + if item.Lyrics == "" { + if lrc := loadLyricsForMusic(ctx, musicPath, tag); lrc != "" { + item.Lyrics = lrc + } + } +} + +// pickMusicFilePath 从 MediaItem 推断出可读取的音乐文件 VFS 路径 +// 合并模式 (is_folder=true): 取 episodes 中第一首；解析失败回退到 folder_path/file_name +// 普通模式: folder_path/file_name +func pickMusicFilePath(item *model.MediaItem) string { + if item == nil { + return "" + } + if item.IsFolder { + // episodes 是 JSON 数组：[{file_name,index,title,...}] + if item.Episodes != "" { + var eps []EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil && len(eps) > 0 { + // 文件夹模式下 folder_path 是扫描根，file_name 是文件夹名 + return stdpath.Join(item.FolderPath, item.FileName, eps[0].FileName) + } + } + // 回退：尝试列出文件夹内第一首音乐 + folder := stdpath.Join(item.FolderPath, item.FileName) + if entries, err := fs.List(context.Background(), folder, &fs.ListArgs{NoLog: true}); err == nil { + for _, e := range entries { + if !e.IsDir() && isMediaFile(e.GetName(), model.MediaTypeMusic) { + return stdpath.Join(folder, e.GetName()) + } + } + } + return "" + } + return stdpath.Join(item.FolderPath, item.FileName) +} diff --git a/internal/media/scanner.go b/internal/media/scanner.go new file mode 100644 index 00000000..2baa7a95 --- /dev/null +++ b/internal/media/scanner.go @@ -0,0 +1,758 @@ +package media + +import ( + "context" + "encoding/base64" + "encoding/json" + "fmt" + "io" + "mime" + stdpath "path" + "regexp" + "strings" + "sync" + "time" + "unicode" + + log "github.com/sirupsen/logrus" + + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/stream" + "github.com/OpenListTeam/OpenList/v4/pkg/http_range" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" +) + +// 支持的文件扩展名 +var ( + videoExts = map[string]bool{ + ".mp4": true, ".mkv": true, ".avi": true, ".mov": true, + ".wmv": true, ".flv": true, ".webm": true, ".m4v": true, + ".ts": true, ".rmvb": true, ".rm": true, ".3gp": true, + } + musicExts = map[string]bool{ + ".mp3": true, ".flac": true, ".aac": true, ".ogg": true, + ".wav": true, ".wma": true, ".m4a": true, ".ape": true, + ".opus": true, ".aiff": true, + } + imageExts = map[string]bool{ + ".jpg": true, ".jpeg": true, ".png": true, ".gif": true, + ".webp": true, ".bmp": true, ".tiff": true, ".svg": true, + ".heic": true, ".avif": true, + } + bookExts = map[string]bool{ + ".epub": true, ".pdf": true, ".mobi": true, ".azw3": true, + ".txt": true, ".djvu": true, ".cbz": true, ".cbr": true, + } +) + +// ScanProgress 扫描进度（全局，按媒体类型维护） +type ScanProgress struct { + mu sync.RWMutex + Running bool + Total int + Done int + // Skipped 入库被跳过的条目数（典型原因：目录/文件名超过唯一索引列长度上限）。 + // 以前只在日志里打一条 Warn，用户完全看不到漏扫了多少条。 + Skipped int + Message string + Error string +} + +var progressMap = map[model.MediaType]*ScanProgress{ + model.MediaTypeVideo: {}, + model.MediaTypeMusic: {}, + model.MediaTypeImage: {}, + model.MediaTypeBook: {}, +} + +// GetProgress 获取扫描进度 +func GetProgress(mediaType model.MediaType) model.MediaScanProgress { + p, ok := progressMap[mediaType] + if !ok { + return model.MediaScanProgress{MediaType: mediaType} + } + p.mu.RLock() + defer p.mu.RUnlock() + return model.MediaScanProgress{ + MediaType: mediaType, + Running: p.Running, + Total: p.Total, + Done: p.Done, + Message: p.Message, + Error: p.Error, + } +} + +// ScanMedia 扫描媒体文件（异步），扫描指定媒体类型下的所有启用扫描路径 +func ScanMedia(cfg *model.MediaConfig) { + p, ok := progressMap[cfg.MediaType] + if !ok { + return + } + p.mu.Lock() + if p.Running { + p.mu.Unlock() + return + } + p.Running = true + p.Total = 0 + p.Done = 0 + p.Skipped = 0 + p.Error = "" + p.Message = "正在扫描..." + p.mu.Unlock() + + go func() { + defer func() { + p.mu.Lock() + p.Running = false + p.mu.Unlock() + }() + + // 获取该媒体类型下所有扫描路径 + scanPaths, err := db.ListMediaScanPaths(cfg.MediaType) + if err != nil { + p.mu.Lock() + p.Error = err.Error() + p.Message = "获取扫描路径失败" + p.mu.Unlock() + return + } + if len(scanPaths) == 0 { + p.mu.Lock() + p.Message = "没有配置扫描路径" + p.mu.Unlock() + return + } + + for i := range scanPaths { + sp := &scanPaths[i] + p.mu.Lock() + p.Message = "正在扫描路径: " + sp.Name + p.mu.Unlock() + + if err := doScanPath(sp, p); err != nil { + log.Errorf("media scan error [%s] path[%s]: %v", cfg.MediaType, sp.Path, err) + } else { + // 更新最后扫描时间 + now := time.Now() + sp.LastScanAt = &now + _ = db.UpdateMediaScanPath(sp) + } + } + + p.mu.Lock() + if p.Skipped > 0 { + // 让管理端能直接看到「有 N 条因为路径过长没入库」 + p.Message = fmt.Sprintf("扫描完成（跳过 %d 条）", p.Skipped) + log.Warnf("media scan skipped %d items (path/name too long for unique index)", p.Skipped) + } else { + p.Message = "扫描完成" + } + p.mu.Unlock() + // 更新媒体库最后扫描时间 + now := time.Now() + cfg.LastScanAt = &now + _ = db.SaveMediaConfig(cfg) + + // 按需生成 STRM + TriggerStrmOnScan(cfg.MediaType, GetAPIRoot()) + }() +} + +// ScanMediaPath 扫描单个扫描路径（异步） +func ScanMediaPath(sp *model.MediaScanPath) { + p, ok := progressMap[sp.MediaType] + if !ok { + return + } + p.mu.Lock() + if p.Running { + p.mu.Unlock() + return + } + p.Running = true + p.Total = 0 + p.Done = 0 + p.Skipped = 0 + p.Error = "" + p.Message = "正在扫描路径: " + sp.Name + p.mu.Unlock() + + go func() { + defer func() { + p.mu.Lock() + p.Running = false + p.mu.Unlock() + }() + + if err := doScanPath(sp, p); err != nil { + p.mu.Lock() + p.Error = err.Error() + p.Message = "扫描失败" + p.mu.Unlock() + log.Errorf("media scan error [%s] path[%s]: %v", sp.MediaType, sp.Path, err) + } else { + p.mu.Lock() + if p.Skipped > 0 { + p.Message = fmt.Sprintf("扫描完成（跳过 %d 条）", p.Skipped) + log.Warnf("media scan skipped %d items (path/name too long for unique index)", p.Skipped) + } else { + p.Message = "扫描完成" + } + p.mu.Unlock() + now := time.Now() + sp.LastScanAt = &now + _ = db.UpdateMediaScanPath(sp) + } + }() +} + +func doScanPath(sp *model.MediaScanPath, p *ScanProgress) error { + scanRoot := sp.Path + if scanRoot == "" { + scanRoot = "/" + } + + // 收集所有待处理路径（VFS 虚拟路径） + var targets []string + + ctx := context.Background() + + if sp.PathMerge { + // 路径合并模式： + // - 子文件夹 → 作为一个合并条目（带 Episodes 选集信息） + // - 直接放在根目录下的单个媒体文件 → 正常作为独立条目扫描 + entries, err := fs.List(ctx, scanRoot, &fs.ListArgs{NoLog: true, Refresh: true}) + if err != nil { + return err + } + for _, e := range entries { + childPath := stdpath.Join(scanRoot, e.GetName()) + if e.IsDir() { + targets = append(targets, childPath) + } else if isMediaFile(e.GetName(), sp.MediaType) { + targets = append(targets, childPath) + } + } + } else { + // 普通模式：递归扫描所有匹配文件（每个目录都刷新缓存） + if err := walkVFS(ctx, scanRoot, sp.MediaType, &targets); err != nil { + return err + } + } + + p.mu.Lock() + p.Total += len(targets) + p.mu.Unlock() + + // 音乐类型：按专辑合并处理 + if sp.MediaType == model.MediaTypeMusic { + return doScanMusicByAlbum(ctx, targets, sp, p) + } + + for _, target := range targets { + item, err := buildMediaItemFromVFS(ctx, target, sp) + if err != nil { + log.Warnf("build media item error [%s]: %v", target, err) + continue + } + + // 路径合并模式下，扫描文件夹内的文件，填充选集信息 + if sp.PathMerge && item.IsFolder { + if episodes, err := buildEpisodesFromFolder(ctx, target, sp.MediaType); err == nil { + item.Episodes = episodes + } else { + log.Warnf("build episodes error [%s]: %v", target, err) + } + } + + if err := db.CreateOrUpdateMediaItem(item); err != nil { + log.Warnf("save media item error [%s]: %v", target, err) + p.mu.Lock() + p.Skipped++ + p.mu.Unlock() + } + p.mu.Lock() + p.Done++ + p.Message = "已扫描: " + stdpath.Base(target) + p.mu.Unlock() + } + return nil +} + +// doScanMusicByAlbum 音乐扫描 +// 存储规则： +// - 普通文件（is_folder=false）： +// folder_path=文件所在真实目录，file_name=音乐文件名，episodes=空 +// - 合并文件夹（is_folder=true）： +// folder_path=文件夹所在真实父目录，file_name=文件夹名， +// episodes=文件夹内所有音乐文件列表，封面/标签取第一首 +func doScanMusicByAlbum(ctx context.Context, targets []string, sp *model.MediaScanPath, p *ScanProgress) error { + for _, target := range targets { + obj, err := fs.Get(ctx, target, &fs.GetArgs{NoLog: true}) + if err != nil { + log.Warnf("get vfs object error [%s]: %v", target, err) + p.mu.Lock() + p.Done++ + p.mu.Unlock() + continue + } + + name := obj.GetName() + + if obj.IsDir() { + // ---- 合并文件夹模式：文件夹条目 ---- + // 列出文件夹内所有音乐文件 + entries, err := fs.List(ctx, target, &fs.ListArgs{NoLog: true}) + if err != nil { + log.Warnf("list music folder error [%s]: %v", target, err) + p.mu.Lock() + p.Done++ + p.mu.Unlock() + continue + } + + var musicFiles []string + var episodes []EpisodeInfo + for idx, e := range entries { + if e.IsDir() || !isMediaFile(e.GetName(), sp.MediaType) { + continue + } + musicFiles = append(musicFiles, stdpath.Join(target, e.GetName())) + ep := EpisodeInfo{ + FileName: e.GetName(), + Index: idx + 1, + Title: strings.TrimSuffix(e.GetName(), stdpath.Ext(e.GetName())), + } + episodes = append(episodes, ep) + } + + if len(musicFiles) == 0 { + p.mu.Lock() + p.Done++ + p.mu.Unlock() + continue + } + + // 取第一首音乐文件读取标签 + firstFile := musicFiles[0] + firstExt := strings.ToLower(stdpath.Ext(firstFile)) + var tag *MusicTag + readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second) + if reader := FetchFileReader(readCtx, firstFile); reader != nil { + switch firstExt { + case ".flac": + tag, _ = ParseFLACVorbisComment(reader) + default: + tag, _ = ParseID3v2(reader) + } + _ = reader.Close() + } + readCancel() + + // 解析标签（用文件夹名作为默认专辑名） + albumName := name + albumArtist := "" + cover := "" + releaseDate := "" + genre := "" + authors := "" + if tag != nil { + if tag.Album != "" { + albumName = tag.Album + } + albumArtist = tag.AlbumArtist + if albumArtist == "" { + albumArtist = tag.Artist + } + if tag.Year != "" && len(tag.Year) >= 4 { + releaseDate = tag.Year[:4] + "-01-01" + } + if tag.Genre != "" { + genre = tag.Genre + } + if tag.Artist != "" { + if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil { + authors = string(authorsJSON) + } + } + if len(tag.CoverData) > 0 { + mimeType := tag.CoverMIME + if mimeType == "" { + mimeType = "image/jpeg" + } + cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData) + } + } + + episodesJSON := "" + if b, err := json.Marshal(episodes); err == nil { + episodesJSON = string(b) + } + + // 加载歌词：同目录同名 .lrc 优先，其次内嵌歌词 + lyrics := loadLyricsForMusic(ctx, firstFile, tag) + + item := &model.MediaItem{ + MediaType: sp.MediaType, + ScanPathID: sp.ID, + FileName: name, // 文件夹名 + FolderPath: stdpath.Dir(target), // 文件夹所在的真实父目录 + IsFolder: true, + AlbumName: albumName, + AlbumArtist: albumArtist, + ScrapedName: albumName, + Cover: cover, + ReleaseDate: releaseDate, + Genre: genre, + Authors: authors, + Lyrics: lyrics, + Episodes: episodesJSON, + } + + if err := db.CreateOrUpdateMediaItem(item); err != nil { + log.Warnf("save music folder item error [%s]: %v", target, err) + } + } else { + // ---- 普通文件：每首歌独立一条记录 ---- + folderPath := stdpath.Dir(target) + ext := strings.ToLower(stdpath.Ext(name)) + + var tag *MusicTag + readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second) + if reader := FetchFileReader(readCtx, target); reader != nil { + switch ext { + case ".flac": + tag, _ = ParseFLACVorbisComment(reader) + default: + tag, _ = ParseID3v2(reader) + } + _ = reader.Close() + } + readCancel() + + albumName := "" + albumArtist := "" + trackTitle := strings.TrimSuffix(name, stdpath.Ext(name)) + trackNumber := 0 + cover := "" + releaseDate := "" + genre := "" + authors := "" + + if tag != nil { + albumName = tag.Album + albumArtist = tag.AlbumArtist + if albumArtist == "" { + albumArtist = tag.Artist + } + if tag.Title != "" { + trackTitle = tag.Title + } + trackNumber = tag.TrackNumber + if tag.Year != "" && len(tag.Year) >= 4 { + releaseDate = tag.Year[:4] + "-01-01" + } + if tag.Genre != "" { + genre = tag.Genre + } + if tag.Artist != "" { + if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil { + authors = string(authorsJSON) + } + } + if len(tag.CoverData) > 0 { + mimeType := tag.CoverMIME + if mimeType == "" { + mimeType = "image/jpeg" + } + cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData) + } + } + + // 加载歌词：同目录同名 .lrc 优先，其次内嵌歌词 + lyrics := loadLyricsForMusic(ctx, target, tag) + + item := &model.MediaItem{ + MediaType: sp.MediaType, + ScanPathID: sp.ID, + FileName: name, + FolderPath: folderPath, + IsFolder: false, + FileSize: obj.GetSize(), + MimeType: mime.TypeByExtension(ext), + AlbumName: albumName, + AlbumArtist: albumArtist, + ScrapedName: trackTitle, + Cover: cover, + ReleaseDate: releaseDate, + Genre: genre, + Authors: authors, + TrackNumber: trackNumber, + Lyrics: lyrics, + } + + if err := db.CreateOrUpdateMediaItem(item); err != nil { + log.Warnf("save music item error [%s]: %v", target, err) + p.mu.Lock() + p.Skipped++ + p.mu.Unlock() + } + } + + p.mu.Lock() + p.Done++ + p.Message = "已扫描: " + name + p.mu.Unlock() + } + return nil +} + +// loadLyricsForMusic 为音乐文件加载歌词： +// 1. 优先读取同目录同名 .lrc 文件（如 song.mp3 → song.lrc） +// 2. 如果不存在，使用 tag 中的内嵌歌词（USLT / Vorbis LYRICS） +// +// musicVfsPath 为音乐文件的 VFS 全路径；tag 可为 nil +func loadLyricsForMusic(ctx context.Context, musicVfsPath string, tag *MusicTag) string { + // 尝试同目录同名 .lrc + ext := stdpath.Ext(musicVfsPath) + if ext != "" { + lrcPath := strings.TrimSuffix(musicVfsPath, ext) + ".lrc" + lrcCtx, lrcCancel := context.WithTimeout(ctx, 8*time.Second) + reader := FetchFileReader(lrcCtx, lrcPath) + if reader != nil { + // 限制读取上限 1MB，避免异常文件压垮内存 + data, err := io.ReadAll(io.LimitReader(reader, 1024*1024)) + _ = reader.Close() + lrcCancel() + if err == nil && len(data) > 0 { + // 去掉可能的 UTF-8 BOM + if len(data) >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF { + data = data[3:] + } + s := strings.TrimSpace(string(data)) + if s != "" { + return s + } + } + } else { + lrcCancel() + } + } + // 回退到内嵌歌词 + if tag != nil && tag.Lyrics != "" { + return tag.Lyrics + } + return "" +} + +// FetchFileReader 通过 VFS 路径获取文件内容流（用于刮削器读取文件内容） +// 优先使用 RangeReader 直接读取（本地存储无需 HTTP），失败时回退到 HTTP URL +// 返回 nil 表示无法获取（不影响主流程） +func FetchFileReader(ctx context.Context, vfsPath string) io.ReadCloser { + link, file, err := fs.Link(ctx, vfsPath, model.LinkArgs{}) + if err != nil || link == nil { + return nil + } + size := link.ContentLength + if size <= 0 { + size = file.GetSize() + } + rr, err := stream.GetRangeReaderFromLink(size, link) + if err != nil { + _ = link.Close() + return nil + } + rc, err := rr.RangeRead(ctx, http_range.Range{Length: size}) + if err != nil { + _ = link.Close() + return nil + } + link.SyncClosers.Add(rc) + return utils.ReadCloser{Reader: rc, Closer: link} +} + +// walkVFS 递归遍历 VFS 路径，收集匹配的媒体文件路径（每个目录都刷新缓存） +func walkVFS(ctx context.Context, dirPath string, mediaType model.MediaType, targets *[]string) error { + entries, err := fs.List(ctx, dirPath, &fs.ListArgs{NoLog: true, Refresh: true}) + if err != nil { + log.Warnf("media scan: list vfs path [%s] error: %v", dirPath, err) + return nil // 跳过无权限目录，不中断整体扫描 + } + for _, e := range entries { + childPath := stdpath.Join(dirPath, e.GetName()) + if e.IsDir() { + if err := walkVFS(ctx, childPath, mediaType, targets); err != nil { + return err + } + } else if isMediaFile(e.GetName(), mediaType) { + *targets = append(*targets, childPath) + } + } + return nil +} + +// buildMediaItemFromVFS 根据 VFS 路径构建 MediaItem（非音乐类型使用） +// 存储规则： +// - folder_path：文件/文件夹所在的真实父目录（VFS 父路径），用于前端按 folder_path + file_name 反查真实文件 +// - file_name：文件夹就是文件夹名，文件就是文件名 +// - episodes：文件夹时存里面每个文件的信息，文件时为空 +func buildMediaItemFromVFS(ctx context.Context, vfsPath string, sp *model.MediaScanPath) (*model.MediaItem, error) { + obj, err := fs.Get(ctx, vfsPath, &fs.GetArgs{NoLog: true}) + if err != nil { + return nil, err + } + + name := obj.GetName() + ext := strings.ToLower(stdpath.Ext(name)) + + item := &model.MediaItem{ + MediaType: sp.MediaType, + ScanPathID: sp.ID, + FileName: name, // 文件夹名 或 文件名 + FolderPath: stdpath.Dir(vfsPath), // 真实父目录 + IsFolder: obj.IsDir(), + ScrapedName: strings.TrimSuffix(name, stdpath.Ext(name)), // 去掉扩展名作为默认名称 + } + + if !obj.IsDir() { + // 普通文件：记录大小和 MIME 类型，episodes 为空 + item.FileSize = obj.GetSize() + item.MimeType = mime.TypeByExtension(ext) + } + + return item, nil +} + +// episodeNumRe 匹配文件名开头的数字序号，支持 "1、" "2." "3-" "4 " 等分隔符 +var episodeNumRe = regexp.MustCompile(`^(\d+)[、.\-\s_]+(.*)`) + +// EpisodeInfo 选集信息（定义在 model 包中，供协议层复用） +type EpisodeInfo = model.EpisodeInfo + +// buildEpisodesFromFolder 扫描文件夹内的媒体文件，构建选集信息 JSON 字符串 +func buildEpisodesFromFolder(ctx context.Context, folderPath string, mediaType model.MediaType) (string, error) { + entries, err := fs.List(ctx, folderPath, &fs.ListArgs{NoLog: true}) + if err != nil { + return "", err + } + + var episodes []EpisodeInfo + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.GetName() + if !isMediaFile(name, mediaType) { + continue + } + + // 去掉扩展名得到裸文件名 + ext := stdpath.Ext(name) + baseName := strings.TrimSuffix(name, ext) + + ep := EpisodeInfo{ + FileName: name, + Index: 0, + Title: baseName, + } + + // 尝试从文件名开头提取数字序号 + if m := episodeNumRe.FindStringSubmatch(baseName); len(m) == 3 { + if idx := parseLeadingInt(m[1]); idx > 0 { + ep.Index = idx + ep.Title = strings.TrimSpace(m[2]) + } + } else { + // 文件名直接以纯数字开头（无分隔符），也尝试提取 + if idx, rest := splitLeadingNumber(baseName); idx > 0 { + ep.Index = idx + ep.Title = strings.TrimSpace(rest) + } + } + + episodes = append(episodes, ep) + } + + if len(episodes) == 0 { + return "", nil + } + + b, err := json.Marshal(episodes) + if err != nil { + return "", err + } + return string(b), nil +} + +// parseLeadingInt 将纯数字字符串解析为 int，失败返回 0 +func parseLeadingInt(s string) int { + v := 0 + for _, c := range s { + if c < '0' || c > '9' { + return 0 + } + v = v*10 + int(c-'0') + } + return v +} + +// splitLeadingNumber 从字符串开头提取连续数字，返回 (数字值, 剩余字符串) +// 仅当开头确实有数字时才返回非零值 +func splitLeadingNumber(s string) (int, string) { + i := 0 + for i < len(s) && s[i] >= '0' && s[i] <= '9' { + i++ + } + if i == 0 || i == len(s) { + // 没有数字，或者整个字符串都是数字（没有标题部分） + return 0, s + } + // 剩余部分必须以非字母数字字符开头，避免把 "1080p" 之类的误识别 + if unicode.IsLetter(rune(s[i])) || unicode.IsDigit(rune(s[i])) { + return 0, s + } + v := parseLeadingInt(s[:i]) + return v, s[i:] +} + +// isMediaFile 判断文件名是否为指定媒体类型（按扩展名判断） +func isMediaFile(name string, mediaType model.MediaType) bool { + ext := strings.ToLower(stdpath.Ext(name)) + switch mediaType { + case model.MediaTypeVideo: + return videoExts[ext] + case model.MediaTypeMusic: + return musicExts[ext] + case model.MediaTypeImage: + return imageExts[ext] + case model.MediaTypeBook: + return bookExts[ext] + } + return false +} + +// GetSupportedExts 获取指定媒体类型支持的扩展名列表 +func GetSupportedExts(mediaType model.MediaType) []string { + var extMap map[string]bool + switch mediaType { + case model.MediaTypeVideo: + extMap = videoExts + case model.MediaTypeMusic: + extMap = musicExts + case model.MediaTypeImage: + extMap = imageExts + case model.MediaTypeBook: + extMap = bookExts + default: + return nil + } + exts := make([]string, 0, len(extMap)) + for ext := range extMap { + exts = append(exts, ext) + } + return exts +} diff --git a/internal/media/scraper/book_local.go b/internal/media/scraper/book_local.go new file mode 100644 index 00000000..ca57580c --- /dev/null +++ b/internal/media/scraper/book_local.go @@ -0,0 +1,389 @@ +package scraper + +import ( + "archive/zip" + "bytes" + "context" + "encoding/base64" + "encoding/xml" + "errors" + "image/jpeg" + "io" + "strings" + "time" + + "github.com/disintegration/imaging" + "github.com/OpenListTeam/OpenList/v4/internal/media/imgmime" + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +// BookLocalScraper 书籍本地刮削器 +// 尝试从书籍文件中提取封面图片并以 base64 存入 Cover 字段，或保存到本地文件。 +// - EPUB：解压 zip，提取 OPF 声明的封面或 cover.jpg/cover.png +// - PDF：尝试提取内嵌 JPEG 图片流作为封面 +// - 其他/失败：降级为文件路径 +type BookLocalScraper struct { + // ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件） + ThumbnailMode string + // ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail" + ThumbnailPath string +} + +// NewBookLocalScraper 创建书籍本地刮削器 +func NewBookLocalScraper() *BookLocalScraper { + return &BookLocalScraper{ + ThumbnailMode: "base64", + ThumbnailPath: "/.thumbnail", + } +} + +// NewBookLocalScraperWithConfig 创建带完整配置的书籍本地刮削器 +func NewBookLocalScraperWithConfig(thumbnailMode, thumbnailPath string) *BookLocalScraper { + if thumbnailMode == "" { + thumbnailMode = "base64" + } + if thumbnailPath == "" { + thumbnailPath = "/.thumbnail" + } + return &BookLocalScraper{ + ThumbnailMode: thumbnailMode, + ThumbnailPath: thumbnailPath, + } +} + +// ScrapeBookLocal 从书籍文件流中提取封面图片，填充基本信息。 +// reader 为书籍文件内容流，不能为 nil。 +// 返回提取到的封面 base64 字符串（data URI），提取失败返回空字符串。 +// 注意：本函数不会将文件路径作为 cover 降级，cover 为空表示提取失败。 +func (s *BookLocalScraper) ScrapeBookLocal(item *model.MediaItem, reader io.Reader) error { + ext := strings.ToLower(getFileExt(item.FileName)) + + // 根据文件类型补充 MimeType + if item.MimeType == "" { + item.MimeType = mimeTypeByBookExt(ext) + } + // 去掉扩展名作为展示名 + if item.ScrapedName == "" { + item.ScrapedName = trimExt(item.FileName) + } + + // 从文件流提取封面，提取失败则 cover 保持为空 + if reader != nil { + data, err := readAllLimited(reader, MaxBookFileBytes) + if err == nil && len(data) > 0 { + var coverData []byte + switch ext { + case ".epub": + coverData = extractEpubCoverData(data) + case ".pdf": + coverData = extractPDFCoverData(data) + } + if len(coverData) > 0 { + cover := s.storeCover(strings.TrimSuffix(item.FolderPath, "/")+"/"+item.FileName, coverData) + if cover != "" { + item.Cover = cover + } + } + } + } + + now := time.Now() + item.ScrapedAt = &now + return nil +} + +// ExtractLocalCover 从书籍文件流中提取封面并根据配置存储（供外部调用）。 +// 提取失败返回空字符串，不降级为文件路径。 +func (s *BookLocalScraper) ExtractLocalCover(fileName string, filePath string, reader io.Reader) string { + if reader == nil { + return "" + } + ext := strings.ToLower(getFileExt(fileName)) + data, err := readAllLimited(reader, MaxBookFileBytes) + if err != nil || len(data) == 0 { + return "" + } + var coverData []byte + switch ext { + case ".epub": + coverData = extractEpubCoverData(data) + case ".pdf": + coverData = extractPDFCoverData(data) + } + if len(coverData) == 0 { + return "" + } + return s.storeCover(filePath, coverData) +} + +// storeCover 根据 ThumbnailMode 存储封面数据，返回 Cover 字段值 +func (s *BookLocalScraper) storeCover(filePath string, coverData []byte) string { + if NormalizeThumbMode(s.ThumbnailMode) == ThumbModeBase64 { + return imageDataToBase64Thumb(coverData) + } + thumb := resizeCoverBytes(coverData) + if len(thumb) == 0 { + thumb = coverData + } + return SaveThumb(context.Background(), s.ThumbnailMode, s.ThumbnailPath, filePath, "image/jpeg", thumb) +} + +// resizeCoverBytes 将封面缩放为 300px 宽，返回 JPEG 字节 +func resizeCoverBytes(data []byte) []byte { + img, err := decodeBounded(data) + if err != nil { + return nil + } + thumb := imaging.Resize(img, 300, 0, imaging.Lanczos) + var buf bytes.Buffer + if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil { + return nil + } + return buf.Bytes() +} + +// ── EPUB 封面提取 ───────────────────────────────────────────────────────────── + +// epubContainer 解析 META-INF/container.xml +type epubContainer struct { + Rootfiles []struct { + FullPath string `xml:"full-path,attr"` + } `xml:"rootfiles>rootfile"` +} + +// epubOPF 解析 OPF 文件，找封面 item +type epubOPF struct { + Metadata struct { + Metas []struct { + Name string `xml:"name,attr"` + Content string `xml:"content,attr"` + } `xml:"meta"` + } `xml:"metadata"` + Manifest struct { + Items []struct { + ID string `xml:"id,attr"` + Href string `xml:"href,attr"` + MediaType string `xml:"media-type,attr"` + } `xml:"item"` + } `xml:"manifest"` +} + +// extractEpubCoverData 从 EPUB（zip）数据中提取封面图片原始字节 +func extractEpubCoverData(data []byte) []byte { + r, err := zip.NewReader(bytes.NewReader(data), int64(len(data))) + if err != nil { + return nil + } + + // 建立文件名→ZipFile 映射（忽略大小写） + fileMap := make(map[string]*zip.File, len(r.File)) + for _, f := range r.File { + fileMap[strings.ToLower(f.Name)] = f + } + + // 1. 尝试读取 META-INF/container.xml 找 OPF 路径 + opfPath := "" + if cf, ok := fileMap["meta-inf/container.xml"]; ok { + if rc, err := cf.Open(); err == nil { + var container epubContainer + if xml.NewDecoder(rc).Decode(&container) == nil && len(container.Rootfiles) > 0 { + opfPath = container.Rootfiles[0].FullPath + } + rc.Close() + } + } + + // OPF 所在目录（用于拼接相对路径） + opfDir := "" + if idx := strings.LastIndex(opfPath, "/"); idx >= 0 { + opfDir = opfPath[:idx+1] + } + + // 2. 从 OPF 中找封面 item ID + coverHref := "" + if opfPath != "" { + if of, ok := fileMap[strings.ToLower(opfPath)]; ok { + if rc, err := of.Open(); err == nil { + var opf epubOPF + if xml.NewDecoder(rc).Decode(&opf) == nil { + // 先找 指向的 item id + coverItemID := "" + for _, m := range opf.Metadata.Metas { + if strings.EqualFold(m.Name, "cover") && m.Content != "" { + coverItemID = m.Content + break + } + } + // 在 manifest 中找对应 item + for _, item := range opf.Manifest.Items { + if coverItemID != "" && item.ID == coverItemID { + coverHref = opfDir + item.Href + break + } + // 如果没有指定 cover id，找第一个图片类型的 item + if coverItemID == "" && strings.Contains(strings.ToLower(item.MediaType), "image") { + // 优先选择 id 或 href 中包含 "cover" 的 + if strings.Contains(strings.ToLower(item.ID), "cover") || + strings.Contains(strings.ToLower(item.Href), "cover") { + coverHref = opfDir + item.Href + break + } + } + } + // 如果还没找到，取第一个图片 + if coverHref == "" && coverItemID == "" { + for _, item := range opf.Manifest.Items { + if strings.Contains(strings.ToLower(item.MediaType), "image") { + coverHref = opfDir + item.Href + break + } + } + } + } + rc.Close() + } + } + } + + // 3. 常见封面文件名候选（按优先级） + candidates := []string{} + if coverHref != "" { + candidates = append(candidates, strings.ToLower(coverHref)) + } + candidates = append(candidates, + "cover.jpg", "cover.jpeg", "cover.png", + "images/cover.jpg", "images/cover.jpeg", "images/cover.png", + "oebps/cover.jpg", "oebps/images/cover.jpg", + "oebps/cover.jpeg", "oebps/images/cover.jpeg", + ) + + for _, name := range candidates { + if f, ok := fileMap[name]; ok { + if rc, err := f.Open(); err == nil { + imgData, readErr := readAllLimited(rc, MaxEmbeddedEntryBytes) + rc.Close() + if readErr == nil && len(imgData) > 0 { + return imgData + } + } + } + } + return nil +} + +// extractEpubCoverBase64 从 EPUB（zip）数据中提取封面图片，返回 data URI base64（向后兼容） +func extractEpubCoverBase64(data []byte) string { + coverData := extractEpubCoverData(data) + if len(coverData) == 0 { + return "" + } + return imageDataToBase64Thumb(coverData) +} + +// ── PDF 封面提取 ────────────────────────────────────────────────────────────── + +// extractPDFCoverData 从 PDF 数据中提取第一个内嵌 JPEG 图片流，返回原始字节 +// PDF 内嵌图片通常以 JPEG 流存储（SOI marker: 0xFF 0xD8），直接扫描字节流提取 +func extractPDFCoverData(data []byte) []byte { + // 扫描 JPEG SOI (0xFF 0xD8 0xFF) 标记 + for i := 0; i < len(data)-2; i++ { + if data[i] == 0xFF && data[i+1] == 0xD8 && data[i+2] == 0xFF { + // 找对应的 EOI (0xFF 0xD9) + for j := i + 2; j < len(data)-1; j++ { + if data[j] == 0xFF && data[j+1] == 0xD9 { + jpegData := data[i : j+2] + // 验证是否为有效图片（至少 1KB，避免误匹配小缩略图） + if len(jpegData) > 1024 { + return jpegData + } + break + } + } + } + } + return nil +} + +// extractPDFCoverBase64 从 PDF 数据中提取第一个内嵌 JPEG 图片流，返回 data URI base64（向后兼容） +func extractPDFCoverBase64(data []byte) string { + coverData := extractPDFCoverData(data) + if len(coverData) == 0 { + return "" + } + return imageDataToBase64Thumb(coverData) +} + +// ── 公共辅助 ────────────────────────────────────────────────────────────────── + +// imageDataToBase64Thumb 将图片字节解码并缩放为 300px 宽缩略图，返回 data URI base64 +func imageDataToBase64Thumb(data []byte) string { + img, err := decodeBounded(data) + if err != nil { + // 超限图片不做任何保留；解码失败但魔数是合法栅格格式时，原样 base64（可能是解码器不支持的变体） + if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes { + return "" + } + mimeType := imgmime.Sniff(data) + if mimeType == "" { + return "" + } + return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data) + } + thumb := imaging.Resize(img, 300, 0, imaging.Lanczos) + var buf bytes.Buffer + if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil { + return "" + } + return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes()) +} + +// detectImageMime 根据文件头检测图片 MIME 类型 +func detectImageMime(data []byte) string { + if len(data) < 4 { + return "image/jpeg" + } + switch { + case data[0] == 0xFF && data[1] == 0xD8: + return "image/jpeg" + case data[0] == 0x89 && data[1] == 0x50 && data[2] == 0x4E && data[3] == 0x47: + return "image/png" + case data[0] == 0x47 && data[1] == 0x49 && data[2] == 0x46: + return "image/gif" + case data[0] == 0x52 && data[1] == 0x49 && data[2] == 0x46 && data[3] == 0x46: + return "image/webp" + default: + return "image/jpeg" + } +} + +// getFileExt 获取文件扩展名（含点号，小写） +func getFileExt(fileName string) string { + if idx := strings.LastIndex(fileName, "."); idx >= 0 { + return fileName[idx:] + } + return "" +} + +// mimeTypeByBookExt 根据书籍文件扩展名返回 MIME 类型 +func mimeTypeByBookExt(ext string) string { + switch ext { + case ".pdf": + return "application/pdf" + case ".epub": + return "application/epub+zip" + case ".mobi": + return "application/x-mobipocket-ebook" + case ".azw3": + return "application/vnd.amazon.ebook" + case ".txt": + return "text/plain" + case ".djvu": + return "image/vnd.djvu" + case ".cbz": + return "application/vnd.comicbook+zip" + case ".cbr": + return "application/vnd.comicbook-rar" + default: + return "application/octet-stream" + } +} diff --git a/internal/media/scraper/book_online.go b/internal/media/scraper/book_online.go new file mode 100644 index 00000000..30caff32 --- /dev/null +++ b/internal/media/scraper/book_online.go @@ -0,0 +1,278 @@ +package scraper + +import ( + "encoding/json" + "fmt" + "net/http" + "net/url" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +// ==================== Google Books ==================== + +// GoogleBooksScraper 通过 Google Books API 刮削书籍 +type GoogleBooksScraper struct { + APIKey string + ThumbnailMode string + ThumbnailPath string + client *http.Client +} + +func NewGoogleBooksScraper(apiKey, thumbnailMode, thumbnailPath string) *GoogleBooksScraper { + if thumbnailMode == "" { + thumbnailMode = "base64" + } + if thumbnailPath == "" { + thumbnailPath = "imgs" + } + return &GoogleBooksScraper{ + APIKey: apiKey, + ThumbnailMode: thumbnailMode, + ThumbnailPath: thumbnailPath, + client: &http.Client{Timeout: 20 * time.Second}, + } +} + +type googleBooksResponse struct { + Items []struct { + ID string `json:"id"` + VolumeInfo struct { + Title string `json:"title"` + Authors []string `json:"authors"` + Description string `json:"description"` + PublishedDate string `json:"publishedDate"` + Publisher string `json:"publisher"` + Categories []string `json:"categories"` + AverageRating float64 `json:"averageRating"` + IndustryIDs []struct { + Type string `json:"type"` + Identifier string `json:"identifier"` + } `json:"industryIdentifiers"` + ImageLinks struct { + Thumbnail string `json:"thumbnail"` + SmallThumbnail string `json:"smallThumbnail"` + } `json:"imageLinks"` + } `json:"volumeInfo"` + } `json:"items"` +} + +// ScrapeBook 用 Google Books 填充书籍信息，封面保存为远程链接（由调用方决定是否落盘） +func (s *GoogleBooksScraper) ScrapeBook(item *model.MediaItem) error { + query := strings.TrimSpace(item.ISBN) + if query == "" { + query = strings.TrimSpace(item.ScrapedName) + } + if query == "" && item.FileName != "" { + query = trimExt(item.FileName) + } + if query == "" { + return fmt.Errorf("empty query") + } + api := "https://www.googleapis.com/books/v1/volumes?maxResults=1&printType=books&langRestrict=" + + url.QueryEscape(ScrapeLanguagePrimary()) + "&q=" + url.QueryEscape(query) + if s.APIKey != "" { + api += "&key=" + url.QueryEscape(s.APIKey) + } + var resp googleBooksResponse + if err := s.getJSON(api, &resp); err != nil { + return err + } + if len(resp.Items) == 0 { + return fmt.Errorf("google books: no result for %q", query) + } + it := resp.Items[0] + v := it.VolumeInfo + + if item.ScrapedName == "" { + item.ScrapedName = v.Title + } + if item.Plot == "" { + item.Plot = v.Description + } + if item.Description == "" { + item.Description = v.Description + } + if item.Publisher == "" { + item.Publisher = v.Publisher + } + if item.ReleaseDate == "" { + item.ReleaseDate = v.PublishedDate + } + if item.Authors == "" && len(v.Authors) > 0 { + item.Authors = toAuthorsJSON(v.Authors) + } + if item.Genre == "" && len(v.Categories) > 0 { + item.Genre = strings.Join(v.Categories, ",") + } + if item.ISBN == "" { + for _, id := range v.IndustryIDs { + if id.Type == "ISBN_13" || id.Type == "ISBN_10" { + item.ISBN = id.Identifier + break + } + } + } + if item.Rating == 0 && v.AverageRating > 0 { + item.Rating = float32(v.AverageRating * 2) + } + if item.ExternalID == "" { + item.ExternalID = "google:" + it.ID + } + if item.Cover == "" { + img := v.ImageLinks.Thumbnail + if img == "" { + img = v.ImageLinks.SmallThumbnail + } + item.Cover = normalizeGoogleImage(img) + } + return nil +} + +func (s *GoogleBooksScraper) getJSON(api string, out any) error { + req, err := http.NewRequest(http.MethodGet, api, nil) + if err != nil { + return err + } + req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)") + resp, err := s.client.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("google books: unexpected status %d", resp.StatusCode) + } + return json.NewDecoder(resp.Body).Decode(out) +} + +func normalizeGoogleImage(img string) string { + if img == "" { + return "" + } + img = strings.Replace(img, "http://", "https://", 1) + if idx := strings.Index(img, "&edge="); idx > 0 { + img = img[:idx] + } + return img +} + +// ==================== OpenLibrary ==================== + +// OpenLibraryScraper 通过 OpenLibrary Search API 刮削书籍（无需凭据） +type OpenLibraryScraper struct { + ThumbnailMode string + ThumbnailPath string + client *http.Client +} + +func NewOpenLibraryScraper(thumbnailMode, thumbnailPath string) *OpenLibraryScraper { + if thumbnailMode == "" { + thumbnailMode = "base64" + } + if thumbnailPath == "" { + thumbnailPath = "imgs" + } + return &OpenLibraryScraper{ + ThumbnailMode: thumbnailMode, + ThumbnailPath: thumbnailPath, + client: &http.Client{Timeout: 20 * time.Second}, + } +} + +type openLibraryResponse struct { + Docs []struct { + Key string `json:"key"` + Title string `json:"title"` + AuthorName []string `json:"author_name"` + FirstPublish int `json:"first_publish_year"` + Publisher []string `json:"publisher"` + ISBN []string `json:"isbn"` + CoverID int `json:"cover_i"` + Subject []string `json:"subject"` + RatingsAverage float64 `json:"ratings_average"` + } `json:"docs"` +} + +// ScrapeBook 用 OpenLibrary 填充书籍信息，封面保存为远程链接（由调用方决定是否落盘） +func (s *OpenLibraryScraper) ScrapeBook(item *model.MediaItem) error { + query := strings.TrimSpace(item.ISBN) + if query == "" { + query = strings.TrimSpace(item.ScrapedName) + } + if query == "" && item.FileName != "" { + query = trimExt(item.FileName) + } + if query == "" { + return fmt.Errorf("empty query") + } + api := "https://openlibrary.org/search.json?limit=1&fields=key,title,author_name,first_publish_year,publisher,isbn,cover_i,subject,ratings_average&q=" + url.QueryEscape(query) + var resp openLibraryResponse + if err := s.getJSON(api, &resp); err != nil { + return err + } + if len(resp.Docs) == 0 { + return fmt.Errorf("openlibrary: no result for %q", query) + } + doc := resp.Docs[0] + + if item.ScrapedName == "" { + item.ScrapedName = doc.Title + } + if item.ReleaseDate == "" && doc.FirstPublish > 0 { + item.ReleaseDate = fmt.Sprintf("%d", doc.FirstPublish) + } + if item.Publisher == "" && len(doc.Publisher) > 0 { + item.Publisher = doc.Publisher[0] + } + if item.Authors == "" && len(doc.AuthorName) > 0 { + item.Authors = toAuthorsJSON(doc.AuthorName) + } + if item.Genre == "" && len(doc.Subject) > 0 { + limit := len(doc.Subject) + if limit > 8 { + limit = 8 + } + item.Genre = strings.Join(doc.Subject[:limit], ",") + } + if item.ISBN == "" && len(doc.ISBN) > 0 { + item.ISBN = doc.ISBN[0] + } + if item.Rating == 0 && doc.RatingsAverage > 0 { + item.Rating = float32(doc.RatingsAverage * 2) + } + if item.ExternalID == "" { + item.ExternalID = "openlibrary:" + doc.Key + } + if item.Cover == "" && doc.CoverID > 0 { + item.Cover = fmt.Sprintf("https://covers.openlibrary.org/b/id/%d-L.jpg", doc.CoverID) + } + return nil +} + +func (s *OpenLibraryScraper) getJSON(api string, out any) error { + req, err := http.NewRequest(http.MethodGet, api, nil) + if err != nil { + return err + } + req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)") + resp, err := s.client.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("openlibrary: unexpected status %d", resp.StatusCode) + } + return json.NewDecoder(resp.Body).Decode(out) +} + +func toAuthorsJSON(authors []string) string { + b, err := json.Marshal(authors) + if err != nil { + return "" + } + return string(b) +} diff --git a/internal/media/scraper/discogs.go b/internal/media/scraper/discogs.go new file mode 100644 index 00000000..7e72c043 --- /dev/null +++ b/internal/media/scraper/discogs.go @@ -0,0 +1,317 @@ +package scraper + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +const discogsDefaultBaseURL = "https://api.discogs.com" + +// DiscogsScraper Discogs音乐刮削器 +type DiscogsScraper struct { + Token string + BaseURL string // 可自定义的 Discogs API 地址 + client *http.Client +} + +// NewDiscogsScraper 创建Discogs刮削器 +// +// 用户只需填写「域名 + API 前缀」，例如： +// 1. 留空：使用默认 https://api.discogs.com +// 2. 官方：api.discogs.com -> https://api.discogs.com +// 3. 反代：example.edgeone.run/api/discogs -> https://example.edgeone.run/api/discogs +// 4. 反代：example.edgeone.run/api/discogs/ -> https://example.edgeone.run/api/discogs（自动去尾斜杠） +// +// 后续会基于 BaseURL 直接拼接 /database/search、/releases/{id} 等具体路径。 +func NewDiscogsScraper(token string, baseURL string) *DiscogsScraper { + base := strings.TrimRight(strings.TrimSpace(baseURL), "/") + if base == "" { + base = discogsDefaultBaseURL + } + // 协议头容错：用户没填 http/https 时自动补 https:// + if !strings.HasPrefix(base, "http://") && !strings.HasPrefix(base, "https://") { + base = "https://" + base + } + // 再次去掉末尾多余的 /，避免后续拼接出 //database/search 这类双斜杠 + base = strings.TrimRight(base, "/") + return &DiscogsScraper{ + Token: token, + BaseURL: base, + client: &http.Client{Timeout: 15 * time.Second}, + } +} + +type discogsSearchResult struct { + Results []struct { + ID int `json:"id"` + Title string `json:"title"` + Type string `json:"type"` + Year string `json:"year"` + Thumb string `json:"thumb"` + CoverImage string `json:"cover_image"` + Genre []string `json:"genre"` + Style []string `json:"style"` + ResourceURL string `json:"resource_url"` + } `json:"results"` +} + +type discogsReleaseDetail struct { + ID int `json:"id"` + Title string `json:"title"` + Year int `json:"year"` + CoverImage string `json:"cover_image"` + Notes string `json:"notes"` + Artists []struct { + Name string `json:"name"` + } `json:"artists"` + Genres []string `json:"genres"` + Styles []string `json:"styles"` + Tracklist []struct { + Position string `json:"position"` + Title string `json:"title"` + Duration string `json:"duration"` + } `json:"tracklist"` + Community struct { + Rating struct { + Average float32 `json:"average"` + } `json:"rating"` + } `json:"community"` +} + +// parseMusicFileName 从音乐文件名中提取搜索关键词列表 +// 规则：去掉扩展名后，按" - "或"-"分割，返回各部分作为候选搜索词 +// 例如： +// +// "周杰伦 - 七里香.mp3" -> ["周杰伦 - 七里香", "周杰伦", "七里香"] +// "Coldplay-Yellow.flac" -> ["Coldplay-Yellow", "Coldplay", "Yellow"] +// "七里香.mp3" -> ["七里香"] +func parseMusicFileName(fileName string) []string { + // 去掉扩展名 + if idx := strings.LastIndex(fileName, "."); idx > 0 { + ext := strings.ToLower(fileName[idx:]) + if len(ext) <= 5 { + fileName = fileName[:idx] + } + } + fileName = strings.TrimSpace(fileName) + if fileName == "" { + return nil + } + + var queries []string + // 完整文件名作为第一候选 + queries = append(queries, fileName) + + // 按" - "（带空格）分割 + parts := strings.SplitN(fileName, " - ", 2) + if len(parts) == 2 { + before := strings.TrimSpace(parts[0]) + after := strings.TrimSpace(parts[1]) + if before != "" { + queries = append(queries, before) + } + if after != "" { + queries = append(queries, after) + } + } else { + // 按"-"（不带空格）分割 + parts = strings.SplitN(fileName, "-", 2) + if len(parts) == 2 { + before := strings.TrimSpace(parts[0]) + after := strings.TrimSpace(parts[1]) + if before != "" { + queries = append(queries, before) + } + if after != "" { + queries = append(queries, after) + } + } + } + + return queries +} + +// doDiscogsSearch 执行单次Discogs搜索 +func (s *DiscogsScraper) doDiscogsSearch(query string) (*discogsSearchResult, error) { + searchURL := fmt.Sprintf("%s/database/search?q=%s&type=release&token=%s", + s.BaseURL, url.QueryEscape(query), s.Token) + + req, _ := http.NewRequest("GET", searchURL, nil) + req.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList") + + resp, err := s.client.Do(req) + if err != nil { + return nil, fmt.Errorf("Discogs搜索请求失败: %w", err) + } + defer resp.Body.Close() + + body, _ := io.ReadAll(resp.Body) + var result discogsSearchResult + if err := json.Unmarshal(body, &result); err != nil { + return nil, fmt.Errorf("Discogs搜索结果解析失败: %w", err) + } + return &result, nil +} + +// buildMusicQuery 根据 MediaItem 已有字段拼接 Discogs 搜索词 +// 优先级：专辑名 > ScrapedName > 文件名解析 +// 若有歌手或年份信息，则附加到搜索词中以提高精准度 +func buildMusicQuery(base string, item *model.MediaItem) string { + q := base + // 附加歌手 + if item.AlbumArtist != "" { + q = item.AlbumArtist + " " + q + } + // 附加年份（取 ReleaseDate 前4位） + if len(item.ReleaseDate) >= 4 { + q = q + " " + item.ReleaseDate[:4] + } + return strings.TrimSpace(q) +} + +// ScrapeMusic 刮削音乐/专辑信息 +func (s *DiscogsScraper) ScrapeMusic(item *model.MediaItem) error { + if s.Token == "" { + return fmt.Errorf("Discogs Token 未配置") + } + + // 构建候选搜索词列表 + var queries []string + if item.AlbumName != "" { + queries = append(queries, buildMusicQuery(item.AlbumName, item)) + // 同时保留不带附加信息的纯专辑名作为降级候选 + if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 { + queries = append(queries, item.AlbumName) + } + } else if item.ScrapedName != "" { + queries = append(queries, buildMusicQuery(item.ScrapedName, item)) + if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 { + queries = append(queries, item.ScrapedName) + } + } else { + // 从文件名解析，按"-"分割分别搜索 + baseQueries := parseMusicFileName(item.FileName) + for _, bq := range baseQueries { + enhanced := buildMusicQuery(bq, item) + if enhanced != bq { + // 先放带附加信息的精准词，再放原始词作为降级 + queries = append(queries, enhanced) + } + queries = append(queries, bq) + } + } + + if len(queries) == 0 { + return fmt.Errorf("无法从文件名中提取有效搜索词") + } + + // 依次尝试各候选词，找到结果即停止（模糊匹配降级） + var searchResult *discogsSearchResult + var lastQuery string + for _, q := range queries { + result, err := s.doDiscogsSearch(q) + if err != nil { + return err + } + lastQuery = q + if len(result.Results) > 0 { + searchResult = result + break + } + } + + if searchResult == nil || len(searchResult.Results) == 0 { + return fmt.Errorf("Discogs未找到匹配结果: %s", lastQuery) + } + + first := searchResult.Results[0] + + // 获取详情 + detailURL := fmt.Sprintf("%s/releases/%d?token=%s", s.BaseURL, first.ID, s.Token) + detailReq, _ := http.NewRequest("GET", detailURL, nil) + detailReq.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList") + + detailResp, err := s.client.Do(detailReq) + if err != nil { + return fmt.Errorf("Discogs详情请求失败: %w", err) + } + defer detailResp.Body.Close() + + detailBody, _ := io.ReadAll(detailResp.Body) + var detail discogsReleaseDetail + if err := json.Unmarshal(detailBody, &detail); err != nil { + return fmt.Errorf("Discogs详情解析失败: %w", err) + } + + // 填充字段：已有值的字段不覆盖，仅补充空缺 + // Discogs 的 title 就是专辑名（不是 "Artist - Album" 格式） + if detail.Title != "" { + if item.AlbumName == "" { + item.AlbumName = detail.Title + } + if item.ScrapedName == "" { + item.ScrapedName = detail.Title + } + } + + if item.ReleaseDate == "" { + if detail.Year > 0 { + item.ReleaseDate = fmt.Sprintf("%d-01-01", detail.Year) + } else if first.Year != "" { + item.ReleaseDate = first.Year + "-01-01" + } + } + + if item.Rating == 0 { + item.Rating = detail.Community.Rating.Average + } + if item.Plot == "" { + item.Plot = detail.Notes + } + if item.ExternalID == "" { + item.ExternalID = fmt.Sprintf("discogs:%d", detail.ID) + } + + // 封面（存储 URL，前端直接展示） + if item.Cover == "" { + if detail.CoverImage != "" { + item.Cover = detail.CoverImage + } else if first.CoverImage != "" { + item.Cover = first.CoverImage + } else if first.Thumb != "" { + item.Cover = first.Thumb + } + } + + // 类型 + if item.Genre == "" { + genres := append(detail.Genres, detail.Styles...) + item.Genre = strings.Join(genres, ",") + } + + // 艺术家：仅在 Authors 和 AlbumArtist 均为空时才填充（ID3已有值则保留） + artists := make([]string, 0, len(detail.Artists)) + for _, a := range detail.Artists { + artists = append(artists, a.Name) + } + if len(artists) > 0 { + if item.Authors == "" { + authorsJSON, _ := json.Marshal(artists) + item.Authors = string(authorsJSON) + } + if item.AlbumArtist == "" { + item.AlbumArtist = artists[0] + } + } + + now := time.Now() + item.ScrapedAt = &now + return nil +} diff --git a/internal/media/scraper/douban.go b/internal/media/scraper/douban.go new file mode 100644 index 00000000..dd446a8a --- /dev/null +++ b/internal/media/scraper/douban.go @@ -0,0 +1,497 @@ +package scraper + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "image/jpeg" + "io" + "net/http" + "net/url" + "regexp" + "strings" + "time" + + "github.com/disintegration/imaging" + "github.com/OpenListTeam/OpenList/v4/internal/media/imgmime" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "golang.org/x/net/html" +) + +const ( + doubanSearchURL = "https://www.douban.com/search" + doubanBookCat = "1001" + doubanBase = "https://book.douban.com/" + + // maxCoverDownloadBytes 单张远端封面最大读取字节数 + maxCoverDownloadBytes int64 = 16 << 20 +) + +var ( + doubanBookURLPattern = regexp.MustCompile(`.*/subject/(\d+)/?`) + doubanDatePattern = regexp.MustCompile(`(\d{4})-(\d+)`) + doubanDefaultHeaders = map[string]string{ + "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", + "Referer": doubanBase, + } +) + +// DoubanScraper 豆瓣书籍刮削器（移植自 NewDouban.py） +type DoubanScraper struct { + client *http.Client + // ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件） + ThumbnailMode string + // ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail" + ThumbnailPath string +} + +// NewDoubanScraper 创建豆瓣刮削器 +func NewDoubanScraper() *DoubanScraper { + return &DoubanScraper{ + client: &http.Client{Timeout: 20 * time.Second}, + ThumbnailMode: "base64", + ThumbnailPath: "/.thumbnail", + } +} + +// NewDoubanScraperWithConfig 创建带完整配置的豆瓣刮削器 +func NewDoubanScraperWithConfig(thumbnailMode, thumbnailPath string) *DoubanScraper { + if thumbnailMode == "" { + thumbnailMode = "base64" + } + if thumbnailPath == "" { + thumbnailPath = "/.thumbnail" + } + return &DoubanScraper{ + client: &http.Client{Timeout: 20 * time.Second}, + ThumbnailMode: thumbnailMode, + ThumbnailPath: thumbnailPath, + } +} + +// ScrapeBook 刮削书籍信息 +func (s *DoubanScraper) ScrapeBook(item *model.MediaItem) error { + query := item.ScrapedName + if query == "" { + query = strings.TrimSuffix(item.FileName, strings.ToLower(item.FileName[strings.LastIndex(item.FileName, "."):])) + } + + // 搜索书籍URL列表 + bookURLs, err := s.searchBookURLs(query) + if err != nil { + return fmt.Errorf("豆瓣搜索失败: %w", err) + } + if len(bookURLs) == 0 { + return fmt.Errorf("豆瓣未找到匹配书籍: %s", query) + } + + // 取第一个结果 + bookDetail, err := s.loadBookDetail(bookURLs[0]) + if err != nil { + return fmt.Errorf("豆瓣获取书籍详情失败: %w", err) + } + + // 填充字段（仅填空字段，已有值不覆盖） + if item.ScrapedName == "" && bookDetail.Title != "" { + item.ScrapedName = bookDetail.Title + } + if item.Plot == "" { + item.Plot = bookDetail.Description + } + // 仅在豆瓣返回了有效封面图片URL且当前 item 还没有封面时才下载/覆盖 + if item.Cover == "" && bookDetail.Cover != "" { + if cover := s.downloadAndStoreCover(item.FolderPath, bookDetail.Cover); cover != "" { + item.Cover = cover + } else { + // 下载失败时保留原 URL（降级） + item.Cover = bookDetail.Cover + } + } + if item.Rating == 0 { + item.Rating = bookDetail.Rating + } + if item.ReleaseDate == "" { + item.ReleaseDate = bookDetail.PublishedDate + } + if item.Publisher == "" { + item.Publisher = bookDetail.Publisher + } + if item.ISBN == "" { + item.ISBN = bookDetail.ISBN + } + if item.ExternalID == "" { + item.ExternalID = "douban:" + bookDetail.ID + } + + if item.Authors == "" && len(bookDetail.Authors) > 0 { + authorsJSON, _ := json.Marshal(bookDetail.Authors) + item.Authors = string(authorsJSON) + } + if item.Genre == "" && len(bookDetail.Tags) > 0 { + item.Genre = strings.Join(bookDetail.Tags, ",") + } + + now := time.Now() + item.ScrapedAt = &now + return nil +} + +type doubanBookDetail struct { + ID string + Title string + Authors []string + Publisher string + PublishedDate string + Cover string + Rating float32 + Description string + Tags []string + ISBN string +} + +// searchBookURLs 搜索豆瓣书籍URL列表 +func (s *DoubanScraper) searchBookURLs(query string) ([]string, error) { + searchURL := fmt.Sprintf("%s?cat=%s&q=%s", doubanSearchURL, doubanBookCat, url.QueryEscape(query)) + body, err := s.doGet(searchURL) + if err != nil { + return nil, err + } + + doc, err := html.Parse(strings.NewReader(string(body))) + if err != nil { + return nil, err + } + + var bookURLs []string + var traverse func(*html.Node) + traverse = func(n *html.Node) { + if n.Type == html.ElementNode && n.Data == "a" { + for _, attr := range n.Attr { + if attr.Key == "class" && attr.Val == "nbg" { + for _, a := range n.Attr { + if a.Key == "href" { + parsed := s.calcURL(a.Val) + if parsed != "" && len(bookURLs) < 5 { + bookURLs = append(bookURLs, parsed) + } + } + } + } + } + } + for c := n.FirstChild; c != nil; c = c.NextSibling { + traverse(c) + } + } + traverse(doc) + return bookURLs, nil +} + +// calcURL 解析豆瓣搜索结果中的真实URL +func (s *DoubanScraper) calcURL(href string) string { + parsed, err := url.Parse(href) + if err != nil { + return "" + } + query := parsed.Query() + rawURL := query.Get("url") + if rawURL == "" { + return "" + } + decoded, err := url.QueryUnescape(rawURL) + if err != nil { + return "" + } + if doubanBookURLPattern.MatchString(decoded) { + return decoded + } + return "" +} + +// loadBookDetail 加载书籍详情页 +func (s *DoubanScraper) loadBookDetail(bookURL string) (*doubanBookDetail, error) { + body, err := s.doGet(bookURL) + if err != nil { + return nil, err + } + return s.parseBookHTML(bookURL, string(body)) +} + +// parseBookHTML 解析书籍HTML（移植自 DoubanBookHtmlParser.parse_book） +func (s *DoubanScraper) parseBookHTML(bookURL, content string) (*doubanBookDetail, error) { + doc, err := html.Parse(strings.NewReader(content)) + if err != nil { + return nil, err + } + + detail := &doubanBookDetail{} + + // 提取豆瓣ID + if m := doubanBookURLPattern.FindStringSubmatch(bookURL); len(m) > 1 { + detail.ID = m[1] + } + + var traverse func(*html.Node) + traverse = func(n *html.Node) { + if n.Type == html.ElementNode { + switch n.Data { + case "span": + // 标题 + if getAttr(n, "property") == "v:itemreviewed" { + detail.Title = getTextContent(n) + } + // 元数据字段 + if getAttr(n, "class") == "pl" { + text := getTextContent(n) + tail := getNextText(n) + switch { + case strings.HasPrefix(text, "作者") || strings.HasPrefix(text, "译者"): + // 作者通过链接提取 + collectAuthors(n, &detail.Authors) + case strings.HasPrefix(text, "出版社"): + detail.Publisher = strings.TrimSpace(tail) + case strings.HasPrefix(text, "出版年"): + detail.PublishedDate = parseDoubanDate(strings.TrimSpace(tail)) + case strings.HasPrefix(text, "ISBN"): + detail.ISBN = strings.TrimSpace(tail) + case strings.HasPrefix(text, "副标题"): + if detail.Title != "" { + detail.Title += ":" + strings.TrimSpace(tail) + } + } + } + // 评分 + if getAttr(n, "property") == "v:average" { + ratingStr := getTextContent(n) + var rating float32 + fmt.Sscanf(ratingStr, "%f", &rating) + detail.Rating = rating / 2 + } + case "img": + // 封面图片： + if getAttr(n, "class") == "nbg" { + src := getAttr(n, "src") + if src != "" && !strings.HasSuffix(src, "update_image") { + detail.Cover = src + } + } + // 备用：#mainpic 下的 img + if getAttr(n, "id") == "mainpic" || (n.Parent != nil && getAttr(n.Parent, "id") == "mainpic") { + src := getAttr(n, "src") + if src != "" && detail.Cover == "" { + detail.Cover = src + } + } + case "a": + // 标签（原封面逻辑移除，改为从img获取） + cls := getAttr(n, "class") + if strings.Contains(cls, "tag") { + tag := getTextContent(n) + if tag != "" { + detail.Tags = append(detail.Tags, tag) + } + } + case "div": + // 简介 + if getAttr(n, "id") == "link-report" { + detail.Description = extractIntroText(n) + } + } + } + for c := n.FirstChild; c != nil; c = c.NextSibling { + traverse(c) + } + } + traverse(doc) + return detail, nil +} + +// HTML辅助函数 + +func getAttr(n *html.Node, key string) string { + for _, attr := range n.Attr { + if attr.Key == key { + return attr.Val + } + } + return "" +} + +func getTextContent(n *html.Node) string { + var sb strings.Builder + var walk func(*html.Node) + walk = func(node *html.Node) { + if node.Type == html.TextNode { + sb.WriteString(node.Data) + } + for c := node.FirstChild; c != nil; c = c.NextSibling { + walk(c) + } + } + walk(n) + return strings.TrimSpace(sb.String()) +} + +func getNextText(n *html.Node) string { + if n.NextSibling != nil { + if n.NextSibling.Type == html.TextNode { + return strings.TrimSpace(n.NextSibling.Data) + } + return getTextContent(n.NextSibling) + } + return "" +} + +func collectAuthors(n *html.Node, authors *[]string) { + var walk func(*html.Node) + walk = func(node *html.Node) { + if node.Type == html.ElementNode && node.Data == "a" { + href := getAttr(node, "href") + if strings.Contains(href, "/author") || strings.Contains(href, "/search") { + name := getTextContent(node) + if name != "" { + *authors = append(*authors, name) + } + } + } + for c := node.FirstChild; c != nil; c = c.NextSibling { + walk(c) + } + } + // 从父节点开始找 + if n.Parent != nil { + walk(n.Parent) + } +} + +func extractIntroText(n *html.Node) string { + var sb strings.Builder + var walk func(*html.Node) + walk = func(node *html.Node) { + if node.Type == html.ElementNode && node.Data == "div" { + if getAttr(node, "class") == "intro" { + sb.WriteString(getTextContent(node)) + return + } + } + for c := node.FirstChild; c != nil; c = c.NextSibling { + walk(c) + } + } + walk(n) + return strings.TrimSpace(sb.String()) +} + +func parseDoubanDate(dateStr string) string { + if dateStr == "" { + return "" + } + if m := doubanDatePattern.FindStringSubmatch(dateStr); len(m) > 2 { + return fmt.Sprintf("%s-%s-01", m[1], m[2]) + } + return dateStr +} + +// downloadAndStoreCover 下载封面图片并根据 ThumbnailMode 存储 +// 失败时返回空字符串 +func (s *DoubanScraper) downloadAndStoreCover(filePath, imgURL string) string { + data, mimeType := s.downloadImage(imgURL) + if len(data) == 0 { + return "" + } + if NormalizeThumbMode(s.ThumbnailMode) != ThumbModeBase64 { + // 运行目录 / 挂载目录模式 + localScraper := NewBookLocalScraperWithConfig(s.ThumbnailMode, s.ThumbnailPath) + return localScraper.storeCover(filePath, data) + } + // Base64 模式（默认） + img, err := decodeBounded(data) + if err != nil { + // 解码失败时退回原始字节；超过阈值直接放弃，避免十几 MB 的图被 base64 写进 cover 列 + if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes { + return "" + } + return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data) + } + thumb := imaging.Resize(img, 300, 0, imaging.Lanczos) + var buf bytes.Buffer + if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil { + return "" + } + return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes()) +} + +// downloadCoverAsBase64 下载封面图片并转为 data URI Base64 字符串（向后兼容） +// 失败时返回空字符串 +func (s *DoubanScraper) downloadCoverAsBase64(imgURL string) string { + data, mimeType := s.downloadImage(imgURL) + if len(data) == 0 { + return "" + } + img, err := decodeBounded(data) + if err != nil { + // 解码失败时退回原始字节；超过阈值直接放弃，避免十几 MB 的图被 base64 写进 cover 列 + if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes { + return "" + } + return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data) + } + thumb := imaging.Resize(img, 300, 0, imaging.Lanczos) + var buf bytes.Buffer + if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil { + return "" + } + return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes()) +} + +// downloadImage 下载图片，返回原始字节和 MIME 类型 +func (s *DoubanScraper) downloadImage(imgURL string) ([]byte, string) { + req, err := http.NewRequest("GET", imgURL, nil) + if err != nil { + return nil, "" + } + for k, v := range doubanDefaultHeaders { + req.Header.Set(k, v) + } + resp, err := s.client.Do(req) + if err != nil { + return nil, "" + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return nil, "" + } + // 远端图片大小不可信，限制读取上限 + data, err := io.ReadAll(io.LimitReader(resp.Body, maxCoverDownloadBytes)) + if err != nil || len(data) == 0 { + return nil, "" + } + // 不信任远端 Content-Type：只接受魔数可识别的栅格图片，防止 SVG/HTML 被当作封面 + mimeType := imgmime.Sniff(data) + if mimeType == "" { + return nil, "" + } + return data, mimeType +} + +// doGet 发送GET请求 +func (s *DoubanScraper) doGet(reqURL string) ([]byte, error) { + req, err := http.NewRequest("GET", reqURL, nil) + if err != nil { + return nil, err + } + for k, v := range doubanDefaultHeaders { + req.Header.Set(k, v) + } + resp, err := s.client.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != 200 && resp.StatusCode != 201 { + return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, reqURL) + } + return io.ReadAll(resp.Body) +} diff --git a/internal/media/scraper/image.go b/internal/media/scraper/image.go new file mode 100644 index 00000000..dba75915 --- /dev/null +++ b/internal/media/scraper/image.go @@ -0,0 +1,196 @@ +package scraper + +import ( + "bytes" + "context" + "fmt" + "image/jpeg" + "io" + "strings" + "time" + + "github.com/disintegration/imaging" + "github.com/rwcarlsen/goexif/exif" + + "github.com/OpenListTeam/OpenList/v4/internal/model" +) + +// ImageScraper 图片刮削器 +// 从图片 EXIF 中提取元数据，并生成缩略图写入 Cover 字段。 +type ImageScraper struct { + // StoreThumbnail 为 true 时生成缩略图写入 Cover； + // 为 false 时直接用文件路径作为 Cover，节省数据库空间。 + StoreThumbnail bool + // ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件） + ThumbnailMode string + // ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail" + ThumbnailPath string +} + +// NewImageScraper 创建图片刮削器 +// storeThumbnail 为 true 时生成缩略图写入 Cover。 +func NewImageScraper(storeThumbnail bool) *ImageScraper { + return &ImageScraper{ + StoreThumbnail: storeThumbnail, + ThumbnailMode: "base64", + ThumbnailPath: "/.thumbnail", + } +} + +// NewImageScraperWithConfig 创建带完整配置的图片刮削器 +func NewImageScraperWithConfig(storeThumbnail bool, thumbnailMode, thumbnailPath string) *ImageScraper { + if thumbnailMode == "" { + thumbnailMode = "base64" + } + if thumbnailPath == "" { + thumbnailPath = "/.thumbnail" + } + return &ImageScraper{ + StoreThumbnail: storeThumbnail, + ThumbnailMode: thumbnailMode, + ThumbnailPath: thumbnailPath, + } +} + +// ScrapeImage 刮削图片信息 +// reader 为图片文件内容流，必须传入有效流才能生成缩略图。 +// 当 reader 不为 nil 时： +// - 从 EXIF 读取拍摄时间、GPS 地点、相机型号+参数、评分 +// - 生成 300px 宽缩略图，根据 ThumbnailMode 存储为 Base64 或本地文件 +// +// 注意：绝不将文件路径作为 cover，reader 为 nil 或缩略图生成失败时 cover 保持为空。 +func (s *ImageScraper) ScrapeImage(item *model.MediaItem, reader io.Reader) error { + // 去掉扩展名作为展示名（如果尚未设置） + if item.ScrapedName == "" { + item.ScrapedName = trimExt(item.FileName) + } + + if reader == nil { + // 无文件流时无法生成缩略图，cover 保持为空，等待下次重试 + now := time.Now() + item.ScrapedAt = &now + return nil + } + + // 将流读入内存（EXIF 解析和图像解码都需要完整数据） + data, err := readAllLimited(reader, MaxImageFileBytes) + if err != nil || len(data) == 0 { + // 读取失败时 cover 保持为空 + now := time.Now() + item.ScrapedAt = &now + return nil + } + + // ── 1. 解析 EXIF ────────────────────────────────────────────── + s.parseEXIF(item, bytes.NewReader(data)) + + // ── 2. 生成缩略图并按配置存储 ──────────────────────────────── + if thumb := s.generateThumbBytes(data); len(thumb) > 0 { + fullPath := strings.TrimSuffix(item.FolderPath, "/") + "/" + item.FileName + item.Cover = SaveThumb(context.Background(), s.ThumbnailMode, s.ThumbnailPath, fullPath, "image/jpeg", thumb) + } + + now := time.Now() + item.ScrapedAt = &now + return nil +} + +// parseEXIF 从图片数据中解析 EXIF 元数据并写入 item +func (s *ImageScraper) parseEXIF(item *model.MediaItem, r io.Reader) { + x, err := exif.Decode(r) + if err != nil { + return + } + + // ── 拍摄时间 → ReleaseDate ──────────────────────────────────── + if item.ReleaseDate == "" { + if t, err := x.DateTime(); err == nil { + item.ReleaseDate = t.Format("2006-01-02") + } + } + + // ── GPS 地点 → Authors（借用作者字段存储地点信息）──────────── + if item.Authors == "" { + if lat, lon, err := x.LatLong(); err == nil { + item.Authors = fmt.Sprintf(`["GPS: %.6f, %.6f"]`, lat, lon) + } + } + + // ── 相机型号 + 参数 → Genre ─────────────────────────────────── + if item.Genre == "" { + var parts []string + if make, err := x.Get(exif.Make); err == nil { + if v, err := make.StringVal(); err == nil && v != "" { + parts = append(parts, strings.TrimSpace(v)) + } + } + if model_, err := x.Get(exif.Model); err == nil { + if v, err := model_.StringVal(); err == nil && v != "" { + parts = append(parts, strings.TrimSpace(v)) + } + } + if fnum, err := x.Get(exif.FNumber); err == nil { + if num, den, err := fnum.Rat2(0); err == nil && den != 0 { + parts = append(parts, fmt.Sprintf("f/%.1f", float64(num)/float64(den))) + } + } + if exp, err := x.Get(exif.ExposureTime); err == nil { + if num, den, err := exp.Rat2(0); err == nil && den != 0 { + if num == 1 { + parts = append(parts, fmt.Sprintf("1/%ds", den)) + } else { + parts = append(parts, fmt.Sprintf("%d/%ds", num, den)) + } + } + } + if iso, err := x.Get(exif.ISOSpeedRatings); err == nil { + if v, err := iso.Int(0); err == nil { + parts = append(parts, fmt.Sprintf("ISO%d", v)) + } + } + if focal, err := x.Get(exif.FocalLength); err == nil { + if num, den, err := focal.Rat2(0); err == nil && den != 0 { + parts = append(parts, fmt.Sprintf("%.0fmm", float64(num)/float64(den))) + } + } + if len(parts) > 0 { + item.Genre = strings.Join(parts, ",") + } + } + + // ── EXIF Rating（0-5 星）→ Rating（0-10 分）────────────────── + // Windows 资源管理器、Lightroom、Darktable 等软件会写入此标签 + if item.Rating == 0 { + if ratingTag, err := x.Get(exif.FieldName("Rating")); err == nil { + if v, err := ratingTag.Int(0); err == nil && v > 0 { + // 1-5 星映射到 2-10 分 + item.Rating = float32(v) * 2 + } + } + } +} + +// generateThumbBytes 将图片数据缩放为 300px 宽缩略图，返回 JPEG 字节 +func (s *ImageScraper) generateThumbBytes(data []byte) []byte { + img, err := decodeBounded(data) + if err != nil { + return nil + } + + // 缩放到宽度 300px，高度等比 + thumb := imaging.Resize(img, 300, 0, imaging.Lanczos) + + var buf bytes.Buffer + if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 75}); err != nil { + return nil + } + return buf.Bytes() +} + +// trimExt 去掉文件名的扩展名 +func trimExt(fileName string) string { + if idx := strings.LastIndex(fileName, "."); idx > 0 { + return fileName[:idx] + } + return fileName +} diff --git a/internal/media/scraper/language.go b/internal/media/scraper/language.go new file mode 100644 index 00000000..df8c0745 --- /dev/null +++ b/internal/media/scraper/language.go @@ -0,0 +1,27 @@ +package scraper + +import ( + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" +) + +// ScrapeLanguage 返回首选刮削语言（BCP-47，如 zh-CN / en-US），未配置时默认中文。 +func ScrapeLanguage() string { + lang := strings.TrimSpace(setting.GetStr(conf.MediaScrapeLanguage, "zh-CN")) + if lang == "" { + return "zh-CN" + } + return lang +} + +// ScrapeLanguagePrimary 返回 ISO 639-1 主语言代码（如 zh / en）， +// 供只接受两位语言码的接口使用（Google Books 的 langRestrict）。 +func ScrapeLanguagePrimary() string { + lang := ScrapeLanguage() + if idx := strings.IndexAny(lang, "-_"); idx > 0 { + lang = lang[:idx] + } + return strings.ToLower(lang) +} diff --git a/internal/media/scraper/limits.go b/internal/media/scraper/limits.go new file mode 100644 index 00000000..a1883cc5 --- /dev/null +++ b/internal/media/scraper/limits.go @@ -0,0 +1,65 @@ +package scraper + +import ( + "bytes" + "errors" + "image" + "io" +) + +// 刮削阶段会把文件整体读入内存再解码，必须限制单文件大小与像素总量， +// 否则几路并发刮削一张超大图片/压缩炸弹就能把进程内存打满。 +const ( + // MaxImageFileBytes 单张待刮削图片的最大字节数 + MaxImageFileBytes int64 = 32 << 20 + // MaxBookFileBytes 单本待提取封面的书籍（epub/pdf）最大字节数 + MaxBookFileBytes int64 = 128 << 20 + // MaxEmbeddedEntryBytes epub(zip) 内单个封面条目的最大解压字节数，防止 zip 炸弹 + MaxEmbeddedEntryBytes int64 = 16 << 20 + // MaxImagePixels 解码前允许的最大像素总量（约 5657x5657）。 + // 解码后按 RGBA 计算约 128MB，默认 5 路并发刮削最坏约 640MB； + // 原来的 64MP 单张就要 256MB，并发时会接近 1.3GB。 + MaxImagePixels int64 = 32_000_000 + // MaxRawCoverFallbackBytes 解码失败时退回「原始字节 base64 写库」的上限。 + // 超过就直接放弃封面——否则单条记录可能写入 16MB 以上的 base64。 + MaxRawCoverFallbackBytes = 512 << 10 +) + +// ErrTooLarge 内容超过读取上限 +var ErrTooLarge = errors.New("scraper: content exceeds size limit") + +// decodeSem 限制同时解码的图片数：解码阶段的内存峰值 ≈ 像素数 × 4， +// 只靠单张上限无法约束多路并发叠加后的总量。 +// 容量必须小于默认刮削并发（5），否则信号量等于不起作用： +// 每路最坏 32MP × 4B ≈ 128MB，12 路就是 1.5GB。 +// 解码是 CPU 密集型操作，并发超过核数也不会更快，取 3 兼顾内存与吞吐。 +var decodeSem = make(chan struct{}, 3) + +// readAllLimited 最多读取 max 字节；超过上限返回 ErrTooLarge（不会继续读取剩余内容）。 +func readAllLimited(r io.Reader, max int64) ([]byte, error) { + data, err := io.ReadAll(io.LimitReader(r, max+1)) + if err != nil { + return nil, err + } + if int64(len(data)) > max { + return nil, ErrTooLarge + } + return data, nil +} + +// decodeBounded 先只读取图片头部获取尺寸，超过像素上限直接拒绝，再做完整解码。 +func decodeBounded(data []byte) (image.Image, error) { + cfg, _, err := image.DecodeConfig(bytes.NewReader(data)) + if err != nil { + return nil, err + } + // 用 int64 相乘：32 位平台上 int 溢出会让超限图片绕过检查 + if cfg.Width <= 0 || cfg.Height <= 0 || + int64(cfg.Width)*int64(cfg.Height) > MaxImagePixels { + return nil, ErrTooLarge + } + decodeSem <- struct{}{} + defer func() { <-decodeSem }() + img, _, err := image.Decode(bytes.NewReader(data)) + return img, err +} diff --git a/internal/media/scraper/thumb_store.go b/internal/media/scraper/thumb_store.go new file mode 100644 index 00000000..54db8610 --- /dev/null +++ b/internal/media/scraper/thumb_store.go @@ -0,0 +1,160 @@ +package scraper + +import ( + "bytes" + "context" + "crypto/sha1" + "encoding/base64" + "encoding/hex" + "fmt" + "io" + "net/http" + "net/url" + "os" + stdpath "path" + "path/filepath" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/sign" + "github.com/OpenListTeam/OpenList/v4/internal/stream" +) + +// 缩略图存储模式 +const ( + // ThumbModeBase64 缩略图以 base64 存入数据库 + ThumbModeBase64 = "base64" + // ThumbModeRuntime 缩略图存入程序数据目录（/） + ThumbModeRuntime = "runtime" + // ThumbModeVFS 缩略图写入挂载目录（VFS 路径） + ThumbModeVFS = "local" +) + +// NormalizeThumbMode 归一化缩略图模式，未知/空值按运行目录处理 +func NormalizeThumbMode(mode string) string { + switch strings.ToLower(strings.TrimSpace(mode)) { + case ThumbModeBase64: + return ThumbModeBase64 + case ThumbModeVFS, "vfs": + return ThumbModeVFS + default: + return ThumbModeRuntime + } +} + +// RuntimeThumbRoot 返回运行目录模式下缩略图的本地根目录 +func RuntimeThumbRoot(thumbPath string) string { + dir := strings.Trim(strings.TrimSpace(thumbPath), "/") + if dir == "" { + dir = "imgs" + } + return filepath.Join(conf.DataDir, filepath.FromSlash(dir)) +} + +// SaveThumb 按模式保存缩略图数据，返回 cover 字段值。 +// rawPath 为媒体文件的完整 VFS 路径，用于生成稳定且唯一的文件名。 +func SaveThumb(ctx context.Context, mode, thumbPath, rawPath, mimeType string, data []byte) string { + if len(data) == 0 { + return "" + } + mode = NormalizeThumbMode(mode) + if mimeType == "" { + mimeType = "image/jpeg" + } + if mode == ThumbModeBase64 { + return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data) + } + + dir := strings.Trim(strings.TrimSpace(thumbPath), "/") + if dir == "" { + dir = "imgs" + } + name := thumbName(rawPath) + + if mode == ThumbModeVFS { + vfsDir := "/" + dir + vfsFile := stdpath.Join(vfsDir, name) + file := &stream.FileStream{ + Ctx: ctx, + Obj: &model.Object{ + Name: name, + Size: int64(len(data)), + Modified: time.Now(), + }, + Reader: bytes.NewReader(data), + Mimetype: mimeType, + } + if err := fs.PutDirectly(ctx, vfsDir, file, true); err != nil { + return "" + } + // 带上签名，兼容开启「签名所有」的站点 + return "/d" + vfsFile + "?sign=" + sign.Sign(vfsFile) + } + + // 运行目录模式：// + root := RuntimeThumbRoot(dir) + if err := os.MkdirAll(root, 0755); err != nil { + return "" + } + if err := os.WriteFile(filepath.Join(root, name), data, 0644); err != nil { + return "" + } + return "thumb:/" + stdpath.Join(dir, name) +} + +// thumbName 根据媒体文件路径生成缩略图文件名 +func thumbName(rawPath string) string { + sum := sha1.Sum([]byte(rawPath)) + return hex.EncodeToString(sum[:]) + ".jpg" +} + +// DownloadCoverToStorage 下载远程封面并按缩略图配置保存，失败返回空字符串 +func DownloadCoverToStorage(ctx context.Context, imgURL, rawPath, thumbMode, thumbPath string) string { + if imgURL == "" || + !(strings.HasPrefix(imgURL, "http://") || strings.HasPrefix(imgURL, "https://")) { + return "" + } + data, contentType, err := fetchImageBytes(ctx, imgURL) + if err != nil { + // 部分网络环境无法直连图片域名，回退到公共图片网关 + if u, parseErr := url.Parse(imgURL); parseErr == nil { + gateway := "https://wsrv.nl/?url=" + url.QueryEscape( + strings.TrimPrefix(strings.TrimPrefix(u.String(), "https://"), "http://"), + ) + data, contentType, err = fetchImageBytes(ctx, gateway) + } + } + if err != nil || len(data) == 0 { + return "" + } + return SaveThumb(ctx, thumbMode, thumbPath, rawPath, contentType, data) +} + +func fetchImageBytes(ctx context.Context, raw string) ([]byte, string, error) { + client := &http.Client{Timeout: 20 * time.Second} + req, err := http.NewRequestWithContext(ctx, http.MethodGet, raw, nil) + if err != nil { + return nil, "", err + } + req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)") + resp, err := client.Do(req) + if err != nil { + return nil, "", err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, "", fmt.Errorf("unexpected status %d", resp.StatusCode) + } + contentType := resp.Header.Get("Content-Type") + if contentType == "" { + contentType = "image/jpeg" + } + if !strings.HasPrefix(contentType, "image/") { + return nil, "", fmt.Errorf("unexpected content type %s", contentType) + } + data, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20)) + return data, contentType, err +} diff --git a/internal/media/scraper/tmdb.go b/internal/media/scraper/tmdb.go new file mode 100644 index 00000000..7b69eb44 --- /dev/null +++ b/internal/media/scraper/tmdb.go @@ -0,0 +1,1448 @@ +package scraper + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + stdpath "path" + "regexp" + "sort" + "strings" + "sync/atomic" + "time" + "unicode/utf8" + + "github.com/OpenListTeam/OpenList/v4/internal/model" + log "github.com/sirupsen/logrus" +) + +// 年份正则（匹配 1900-2099） +var yearRegexp = regexp.MustCompile(`\b((?:19|20)\d{2})\b`) + +// chineseRegexp 匹配包含中文字符的片段 +var chineseRegexp = regexp.MustCompile(`[\p{Han}]`) + +// 噪声词正则：发布组、编码、分辨率、音轨等无意义片段 +// 清洗中文标题尾部常见的污染词 +// 注意：这里只用于「子串匹配」清洗，不要求整段命中 +var noiseTokenRegexp = regexp.MustCompile(`(?i)(双语字幕|双语|中字|中英字幕|中英双字|中英双语|国英双语|国英双轨|国粤双语|粤语中字|国语中字|英语中字|日语中字|韩语中字|国英|国粤|粤语|国语|英语|日语|韩语|内封字幕|外挂字幕|HDTV|HR-HDTV|BluRay|BDRip|WEB-?DL|HDRip|DVDRip|REMUX|x264|x265|h264|h265|HEVC|AVC|AAC|AC3|DTS|FLAC|10bit|8bit|HDR|SDR|4K|2160P|1080P|720P|480P|完整版|未删减版|加长版|导演剪辑版|蓝光版|高清版|正版上译公映|正版上译|上译公映|公映译制|译制公映|公映版|上译版|京译版|长译版|国配版|台配版|港配版|国语配音|粤语配音|国粤配音|原声版|原盘|配音版|译制版)`) + +// chineseReleaseGroupRe 中文字幕组/压制组/影视站名的整段匹配正则 +// 这些词出现在文件名中通常是发布信息，绝不应该被当成片名候选 +// 例：「-人人影视制作」「.YYeTs.」「-飞鸟影视」 +var chineseReleaseGroupRe = regexp.MustCompile(`^(?:` + + `人人影视制作|人人影视|人人字幕|人人字幕组|YYeTs|YYETs|FRDS|` + + `飞鸟影视|飞鸟影院|风行网|风行影视|远鉴字幕组|远鉴|深影字幕组|深影|` + + `破烂熊|圣城家园|TLF|HDS|MNHD|WiKi|CMCT|RARBG|YIFY|YTS|` + + `蓝色狂想|蓝色狂想字幕组|肥羊字幕组|衣柜字幕组|喵萌奶茶屋|` + + `众乐字幕组|猪猪乐园|猪猪字幕组|流鸣字幕|动漫国字幕组|诸神字幕组|` + + `字幕组|压制组|发布组|官方版本|高清影视|高清电影|蓝光原盘` + + `)$`) + +// chineseGarbageContainsRe 包含即视为发布组信息的中文片段（用于片段中嵌有这些字眼时整体丢弃） +// 例：「人人影视」、「天天美剧」即使附带其他字符，整段都不太可能是片名 +var chineseGarbageContainsRe = regexp.MustCompile(`人人影视|人人字幕|YYeTs|字幕组|压制组|发布组|正版上译|上译公映|公映译制|译制公映|公映版|上译版|国配版|台配版|港配版|配音版|译制版`) + +// 中文数字到阿拉伯数字的简单映射，用于「钢铁侠三」=>「钢铁侠3」类的归一化 +var cnNumMap = map[string]string{ + "〇": "0", "零": "0", "一": "1", "二": "2", "三": "3", "四": "4", + "五": "5", "六": "6", "七": "7", "八": "8", "九": "9", "十": "10", +} + +// normalizeTitle 对标题做模糊匹配前的归一化处理 +// - 去除括号及其中内容 +// - 去除版本/编码等噪声词 +// - 合并多余空白 +func normalizeTitle(s string) string { + if s == "" { + return s + } + // 去掉中英文括号包裹的内容 + bracketRe := regexp.MustCompile(`[\(（\[【][^\)）\]】]*[\)）\]】]`) + s = bracketRe.ReplaceAllString(s, " ") + // 去掉常见噪声词 + s = noiseTokenRegexp.ReplaceAllString(s, " ") + // 替换分隔符为空格 + s = strings.NewReplacer(".", " ", "_", " ", "-", " ", "+", " ").Replace(s) + // 合并多余空白 + s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ") + return strings.TrimSpace(s) +} + +// cnNumToArabic 将标题中的中文数字归一化为阿拉伯数字（仅做轻量处理） +func cnNumToArabic(s string) string { + if s == "" { + return s + } + var b strings.Builder + for _, r := range s { + if v, ok := cnNumMap[string(r)]; ok { + b.WriteString(v) + } else { + b.WriteRune(r) + } + } + return b.String() +} + +// parsedVideoTitle 解析后的视频标题信息 +// +// 解析时除了主中文标题/英文标题之外，可能还会从文件名中识别到其他中文片段 +// （比如同时存在「[钢铁侠]」和「Iron Man」，或多段中文），这些都作为兜底候选保留。 +// 主中文/英文标题用于「常规」搜索路径；额外候选会在主候选搜不到时再尝试。 +type parsedVideoTitle struct { + EnglishTitle string // 英文标题（第一个中文片段之前、年份之前的部分） + ChineseTitle string // 中文标题（最优中文片段） + ExtraChineseTitles []string // 其他中文片段，作为兜底候选（如方括号外又出现的中文标题、副标题等） + Year string // 年份 +} + +// leadingSerialSpaceRe 匹配开头是 "数字 + 空格"（如 "01 [钢铁侠]"、"19 [复仇者联盟3：无限战争]"） +// 数字 + 空格 几乎可以确定是序号（合法片名极少以"数字 + 空格"开头） +// 只匹配 1-3 位数字，避免把年份(2008) 或 4 位以上数字当成序号 +var leadingSerialSpaceRe = regexp.MustCompile(`^\s*\d{1,3}\s+`) + +// leadingSerialDotChineseRe 匹配开头是 "数字 + 点/-/_/、 + 中文" 的形式 +// 这种形式既可能是"序号 + 中文片名"，也可能是"数字片名 + 中文别名"，比如： +// +// "169.谍影重重3" -> 序号 + 中文片名 +// "1.漫威短片.47号物品" -> 序号 + 中文片名 +// "300.斯巴达勇士..." -> 数字片名 + 中文别名（300 是片名核心） +// "36总局" 这种中文里就含数字的也算 +// +// 启发式判定： +// - 1-2 位数字（1-99）：序号概率高，删掉数字 +// - 3 位数字（100-999）：片名概率高，保留数字 +// - 4 位以上：在外层正则就被 \d{1,3} 排除 +var leadingSerialDotChineseRe = regexp.MustCompile(`^\s*(\d{1,3})\s*[.\-_、]\s*(\p{Han})`) + +// stripLeadingSerial 去除文件名开头的序号前缀，返回剥离后的文件名 +// +// 处理两种序号情况： +// 1. "数字 + 空格"：如 "01 [钢铁侠]Iron.Man..." -> "[钢铁侠]Iron.Man..." +// 2. "1-2 位数字 + 点/-/_/、 + 中文"：如 "1.漫威短片..." -> "漫威短片..."、"169.谍影重重3" -> "谍影重重3" +// +// 不视为序号（保留原样）： +// - "30.Days.Of.Night..."（数字 + 点 + 英文，数字是片名一部分） +// - "300.Rise.Of.An.Empire..."（同上） +// - "300.斯巴达勇士..."（3 位数字 + 点 + 中文，可能是片名 "300"） +// - "3096.Days.2013..."（4 位数字直接被排除） +// +// 被剥离的序号本身不是片名关键词，不需要回填。 +func stripLeadingSerial(s string) string { + // 1) "数字 + 空格"：稳定删除（序号信号最强） + if loc := leadingSerialSpaceRe.FindStringIndex(s); loc != nil && loc[0] == 0 { + return strings.TrimSpace(s[loc[1]:]) + } + // 2) "1-2 位数字 + 分隔符 + 中文"：视为序号删除；3 位数字保守保留 + if m := leadingSerialDotChineseRe.FindStringSubmatchIndex(s); m != nil && m[0] == 0 { + // m[2]:m[3] 是数字捕获组的索引；m[4]:m[5] 是中文字符 + numStr := s[m[2]:m[3]] + // 3 位数字保守保留（避免删除 "300.斯巴达勇士" 中的 "300"） + if len(numStr) >= 3 { + return s + } + // 1-2 位数字视为序号删除，从中文字符位置开始保留 + return strings.TrimSpace(s[m[4]:]) + } + return s +} + +// noiseTokenWholeRe 判断一个完整片段是否为“纯噪声词”（完全匹配，不包含其他字符） +// 覆盖常见发布组/字幕组/编码/音轨/分辨率/语言标记等。被识别为噪声的片段不会进入 EnglishTitle。 +// 例："HR-HDTV"、"AC3"、"x264"、"X264"、"1024x576"、"2160p"、"BOBO"(发布组) 等 +var noiseTokenWholeRe = regexp.MustCompile(`(?i)^(?:` + + // 字幕/语言标记 + `双语字幕|中字|国英|国粤|粤语|国语|英语|日语|韩语|国英双轨|国粤英双轨|中英双字|` + + `English|Chinese|Cantonese|Mandarin|Japanese|Korean|CHS|CHT|ENG|JPN|KOR|CHS-ENG|CHS-JPN|CHT-ENG|` + + // 片源/容器 + `HDTV|HR-HDTV|HR\.HDTV|BluRay|Blu-Ray|BDRip|BDMV|WEB-?DL|WEBRip|HDRip|DVDRip|DVD|REMUX|UHD|RAW|TS|TC|CAM|HC|TVRip|` + + // 视频编码 + `x264|x265|h264|h265|HEVC|AVC|XviD|DivX|VP9|AV1|10bit|8bit|HDR10\+?|HDR|SDR|DV|DolbyVision|` + + // 音频编码 + `AAC|AAC2\.0|AC3|EAC3|DD5\.1|DD7\.1|DD\+|DDP|DTS|DTS-HD|DTS-X|DTSX|DTSHD|FLAC|MP3|TrueHD|Atmos|MA|` + + // 分辨率 / 画质 + `4K|2K|8K|2160p|2160P|1080p|1080P|720p|720P|480p|480P|\d{3,4}[xX×]\d{3,4}|` + + // 中文描述词 + `完整版|未删减版|院线版|导演剪辑版|加长版|年度佳作|` + + // 常见发布组名（可能不完全，但覆盖示例中出现过的） + `BOBO|SWTYBLZ|CMCT|HDS|FRDS|MNHD|WiKi|TLF|RARBG|YIFY|YTS|EVO|SPARKS` + + `)$`) + +// isNoiseToken 判断一个片段是否是完全的噪声词。 +// 同时覆盖一些“数字.数字”组合被 "." 拆出后的碎片，如“5”、“1”、“7”（来自 5.1/7.1 音轨拆分） +func isNoiseToken(p string) bool { + if p == "" { + return true + } + // 微小碎片（单个或两个数字）但不是年份，一般是被"."拆分开的 5.1/7.1/2.0 碎片。 + // 为避免误伤「4 Months 3 Weeks And 2 Days」这种中间字随意出现的数字，不在这里过滤， + // 仅依靠 noiseTokenWholeRe 覆盖。 + return noiseTokenWholeRe.MatchString(p) +} + +// extractBracketChinese 从原始文件名中提取出方括号/中文括号里的中文片段 +// +// "01 [钢铁侠]Iron.Man.2008..." -> ["钢铁侠"] +// "19 [复仇者联盟3：无限战争]Avengers..." -> ["复仇者联盟3：无限战争"] +// "[Pixar][玩具总动员]Toy.Story.1995..." -> ["玩具总动员"] +// +// 仅返回包含中文的括号内容，避免把发布组、字幕组等英文括号信息当成标题。 +func extractBracketChinese(fileName string) []string { + bracketRe := regexp.MustCompile(`[\(（\[【]([^\)）\]】]*)[\)）\]】]`) + matches := bracketRe.FindAllStringSubmatch(fileName, -1) + var out []string + for _, m := range matches { + if len(m) < 2 { + continue + } + inner := strings.TrimSpace(m[1]) + if inner == "" { + continue + } + // 必须包含中文字符才视为中文标题候选 + if chineseRegexp.MatchString(inner) { + out = append(out, inner) + } + } + return out +} + +// parseVideoFileName 从视频文件名中提取标题和年份 +// 规则： +// - 自动去除文件名开头的序号（如 "01 "、"1."、"169."），避免污染英文标题 +// - 优先把方括号 [中文] 内的中文片段作为中文标题候选 +// - 文件名中允许使用 . / 空格 / _ / 的分隔多个字段 +// - 第一个含中文的片段即为中文标题（若上一步括号内未找到中文） +// - 中文标题之前的非中文、非年份片段拼接为英文标题 +// - 第一个 1900-2099 之间的 4 位数字识别为年份 +// +// 例如： +// +// "Inception.2010.盗梦空间.双语字幕.HR-HDTV.AC3.1024X576.X264-" -> {English:"Inception", Chinese:"盗梦空间", Year:"2010"} +// "Iron.Man.3.2013.钢铁侠3.国英音轨.双语字幕.HR-HDTV.AC3.x264-" -> {English:"Iron Man 3", Chinese:"钢铁侠3", Year:"2013"} +// "The.Dark.Knight.2008.1080p.BluRay" -> {English:"The Dark Knight", Chinese:"", Year:"2008"} +// "盗梦空间.2010.1080p.BluRay" -> {English:"", Chinese:"盗梦空间", Year:"2010"} +// "钢铁侠3 (2013).mkv" -> {English:"", Chinese:"钢铁侠3", Year:"2013"} +// "群星-演唱会现场.mp4" -> {English:"", Chinese:"群星", Year:""} +// "01 [钢铁侠]Iron.Man.2008.2160p.HDR.BluRay..." -> {English:"Iron Man", Chinese:"钢铁侠", Year:"2008"} +// "19 [复仇者联盟3：无限战争]Avengers.Infinity.War.2018.2160p..." -> {English:"Avengers Infinity War", Chinese:"复仇者联盟3：无限战争", Year:"2018"} +// "169.谍影重重3" -> {English:"", Chinese:"谍影重重3", Year:""} +// "1.漫威短片.47号物品" -> {English:"", Chinese:"漫威短片", Year:""}（额外候选含 "47号物品"） +func parseVideoFileName(fileName string) parsedVideoTitle { + // 去掉扩展名（.mkv .mp4 .avi 等，扩展名长度 <= 5） + if idx := strings.LastIndex(fileName, "."); idx > 0 { + ext := strings.ToLower(fileName[idx:]) + if len(ext) <= 5 { + fileName = fileName[:idx] + } + } + + // 1) 先把开头的序号 "01 "、"1."、"169." 去掉，避免它们污染英文标题 + // 仅在能明确判定为序号的场景才剥离，避免误伤 "30.Days..."、"300.斯巴达勇士" 这类数字作为片名一部分的情况 + fileName = stripLeadingSerial(fileName) + + // 2) 在剥离括号之前，先尝试从方括号 [中文] 中抽取中文标题候选 + // 很多发布组习惯是 "01 [中文标题]English.Title.Year..."，这里要保留 "中文标题" + bracketChinese := extractBracketChinese(fileName) + + // 3) 提取括号中的年份（如 "钢铁侠3 (2013).mkv"），把年份替换到括号外 + bracketYearRe := regexp.MustCompile(`[\(（\[【]\s*((?:19|20)\d{2})\s*[\)）\]】]`) + fileName = bracketYearRe.ReplaceAllString(fileName, " $1 ") + // 4) 再把剩余括号块替换成空格（其中的中文标题已在第 2 步保存到 bracketChinese） + bracketRe := regexp.MustCompile(`[\(（\[【][^\)）\]】]*[\)）\]】]`) + fileName = bracketRe.ReplaceAllString(fileName, " ") + + // 把多种分隔符统一成 "."，方便后续按 "." 解析 + // 注意：中文之间通常没分隔符，应保留；这里只把 "." 之外的分隔符换成 "." + replacer := strings.NewReplacer( + "_", ".", + "+", ".", + ) + fileName = replacer.Replace(fileName) + + // 把空格、" - " 也作为分隔符（但只在 ASCII 段之间使用，避免破坏中文）， + // 简化处理：直接把它们也变成 "." + fileName = regexp.MustCompile(`[\s]+`).ReplaceAllString(fileName, ".") + fileName = strings.ReplaceAll(fileName, "-", ".") + + // 按"."分割各字段 + parts := strings.Split(fileName, ".") + + var result parsedVideoTitle + var englishParts []string + var chineseParts []string // 收集所有中文片段，便于兜底 + foundYear := false + foundChinese := false // 一旦出现中文片段，后面的非中文片段都不再加入英文标题 + + // pureNoise 判断字符串经过噪声词清洗后是否为空（即整体都是噪声词） + // 用于过滤掉"双语字幕"、"国英音轨" 这种纯噪声片段，避免被当成中文标题 + pureNoise := func(s string) bool { + cleaned := noiseTokenRegexp.ReplaceAllString(s, "") + cleaned = strings.TrimSpace(cleaned) + return cleaned == "" + } + + // isChineseReleaseGroup 判断中文片段是否为字幕组/压制组名 + // 这类片段绝不应该被当成片名候选（如 "人人影视"、"YYeTs"、"飞鸟影视"） + isChineseReleaseGroup := func(s string) bool { + s = strings.TrimSpace(s) + if s == "" { + return false + } + if chineseReleaseGroupRe.MatchString(s) { + return true + } + if chineseGarbageContainsRe.MatchString(s) { + return true + } + return false + } + + for _, p := range parts { + p = strings.TrimSpace(p) + if p == "" { + continue + } + + // 检测是否含中文 + if chineseRegexp.MatchString(p) { + // 整体被识别为噪声词的中文片段（如 "双语字幕"、"国英音轨"）直接跳过 + if pureNoise(p) { + continue + } + // 字幕组/压制组中文名（如 "人人影视"、"飞鸟影院"）也直接跳过 + if isChineseReleaseGroup(p) { + continue + } + chineseParts = append(chineseParts, p) + foundChinese = true + continue + } + + // 检测是否为年份（1900-2099） + if yearRegexp.MatchString(p) && !foundYear { + result.Year = yearRegexp.FindString(p) + foundYear = true + // 年份本身不加入英文标题 + continue + } + + // 后续非中文片段加入英文标题的判定： + // - 必须在年份出现之前（年份后的 ASCII 全是发布信息噪声） + // - 必须在中文出现之前（中文后的 ASCII 也是发布信息噪声，避免 "斯巴达勇士 HR HDTV AC3 X264" 这种污染） + // - 不能是完全的噪声词（如 HR-HDTV、AC3、x264、分辨率） + if foundYear || foundChinese { + continue + } + if isNoiseToken(p) { + continue + } + englishParts = append(englishParts, p) + } + + result.EnglishTitle = strings.Join(englishParts, " ") + + // 选定中文标题： + // - 优先使用方括号内的中文（最可靠的发布组标记） + // - 否则使用解析出的中文片段中第一个 + // - 同时把所有中文候选去重保留到 ExtraChineseTitles 里，留作兜底 + allChinese := append([]string{}, bracketChinese...) + allChinese = append(allChinese, chineseParts...) + allChinese = dedupStrings(allChinese) + + if len(allChinese) > 0 { + result.ChineseTitle = allChinese[0] + if len(allChinese) > 1 { + result.ExtraChineseTitles = allChinese[1:] + } + } + + // 当解析出多个连续中文片段时，它们很可能是「主标题.副标题」结构（如 "300勇士.帝国崛起" + // 实际是「300勇士：帝国崛起」），把合并后的候选也加入 ExtraChineseTitles 用于兜底搜索。 + // 注意：合并最多保留前两段（主标题 + 副标题），避免「主标题：副标题：发布版本」这类 + // 把发布信息（正版上译公映等）当成第二副标题的脏候选；电影/剧名极少有真正的三段式官方片名。 + if len(chineseParts) > 1 { + mergeParts := chineseParts + if len(mergeParts) > 2 { + mergeParts = mergeParts[:2] + } + merged := strings.Join(mergeParts, "：") + // 避免与已有候选重复 + exists := merged == result.ChineseTitle + for _, t := range result.ExtraChineseTitles { + if t == merged { + exists = true + break + } + } + if !exists { + result.ExtraChineseTitles = append(result.ExtraChineseTitles, merged) + } + } + + // 对中文标题做尾部噪声清洗（去除「钢铁侠3 国英音轨」末尾的污染词，但保留主标题） + if result.ChineseTitle != "" { + result.ChineseTitle = stripTrailingNoise(result.ChineseTitle) + } + for i, t := range result.ExtraChineseTitles { + result.ExtraChineseTitles[i] = stripTrailingNoise(t) + } + return result +} + +// dedupStrings 去除字符串切片中的空白与重复项，保留首次出现顺序 +func dedupStrings(in []string) []string { + seen := make(map[string]bool) + out := make([]string, 0, len(in)) + for _, s := range in { + s = strings.TrimSpace(s) + if s == "" || seen[s] { + continue + } + seen[s] = true + out = append(out, s) + } + return out +} + +// stripTrailingNoise 去除中文标题中嵌入的噪声词，保留主标题部分 +// 如「钢铁侠3 国英音轨」-> 「钢铁侠3」 +// 同时剖除末尾的 (1) (2) V2 这类重复/版本后缀、字幕组名等 +func stripTrailingNoise(s string) string { + cleaned := noiseTokenRegexp.ReplaceAllString(s, " ") + // 剖除「-人人影视」「.YYeTs」这种跟在中文后面的字幕组名 + cleaned = regexp.MustCompile(`[\s\-・]*`+ + `(?:人人影视制作|人人影视|人人字幕|YYeTs|YYETs|FRDS|飞鸟影视|飞鸟影院|风行网|深影|远鉴|字幕组|压制组)[s]?`+ + `(?:、|。|\.|,)?\s*$`).ReplaceAllString(cleaned, " ") + // 剖除末尾的 (1)（重复文件标记）、V2、v3 这类版本后缀 + cleaned = regexp.MustCompile(`(?i)\s*[\(（]\s*\d+\s*[\)）]\s*$`).ReplaceAllString(cleaned, " ") + cleaned = regexp.MustCompile(`(?i)\s*[Vv]\s*\d+\s*$`).ReplaceAllString(cleaned, " ") + cleaned = regexp.MustCompile(`\s+`).ReplaceAllString(cleaned, " ") + cleaned = strings.TrimSpace(cleaned) + if cleaned == "" { + return strings.TrimSpace(s) + } + return cleaned +} + +const tmdbDefaultBaseURL = "https://api.themoviedb.org" +const tmdbImageBase = "https://image.tmdb.org/t/p/w500" + +// TMDBStats TMDB 刮削统计（线程安全：原子计数） +// 仅记录一次进程级别的累积值；如需更精细可按 ScraperID 分桶 +type TMDBStats struct { + TotalScraped int64 // 总刮削请求次数（每次 ScrapeVideo 调用 +1） + HitCount int64 // 成功刮到结果的次数（搜索阶段命中即停） + NoMatchCount int64 // 完全 0 命中的次数 + LowConfidenceCount int64 // 命中但低于阈值，使用了「最佳兜底」结果的次数 + TotalAttempts int64 // 累计 TMDB 搜索请求次数（不含详情） + TotalSearchMillis int64 // 累计搜索耗时（毫秒） +} + +// 全局统计实例 +var globalTMDBStats = &TMDBStats{} + +// GetTMDBStats 返回当前进程的 TMDB 刮削统计快照 +func GetTMDBStats() TMDBStats { + return *globalTMDBStats +} + +// ResetTMDBStats 重置统计（仅用于测试或运维触发） +func ResetTMDBStats() { + *globalTMDBStats = TMDBStats{} +} + +// matchConfidenceThreshold 命中可信度阈值。 +// scoreMatch >= 该阈值视为「足够可信」，立即返回； +// 介于 fallbackThreshold 和该阈值之间则记为「最佳兜底」； +// 低于 fallbackThreshold 进入「降级重试」阶段；都失败才报错。 +const ( + matchConfidenceThreshold float32 = 0.60 + fallbackThreshold float32 = 0.30 + // desperateThreshold 「降级重试」阶段的阈值：当主流程完全无任何 score>=0.30 的命中时， + // 采用更宽的阈值（0.20）接受 score 在 [0.20, 0.30) 之间的结果作为最后的兜底。 + desperateThreshold float32 = 0.20 +) + +// TMDBScraper TMDB视频刮削器 +type TMDBScraper struct { + APIKey string + BaseURL string // 含 /3 版本路径的最终请求地址，例如 https://api.themoviedb.org/3 + client *http.Client +} + +// NewTMDBScraper 创建TMDB刮削器 +// baseURL 为可自定义的 TMDB API 服务地址，为空时使用默认 https://api.themoviedb.org/3 +// +// 支持的填写形式： +// 1. 留空：使用默认 https://api.themoviedb.org/3 +// 2. 官方主域名（不带路径）：如 api.themoviedb.org / https://api.themoviedb.org，自动补全协议和 /3 版本路径 +// 3. 反代/自定义完整路径：如 https://eo-tmd.example.com/api/tmdb/3，会原样使用，不再追加 /3 +func NewTMDBScraper(apiKey string, baseURL string) *TMDBScraper { + base := strings.TrimRight(strings.TrimSpace(baseURL), "/") + if base == "" { + base = tmdbDefaultBaseURL + } + // 1. 协议头容错：用户没填 http/https 时自动补 https:// + if !strings.HasPrefix(base, "http://") && !strings.HasPrefix(base, "https://") { + base = "https://" + base + } + // 2. 版本号容错：约定前端要求填写「域名 + API 前缀」 + // - 官方：api.themoviedb.org -> https://api.themoviedb.org/3 + // - 反代：example.edgeone.run/api/tmdb/ -> https://example.edgeone.run/api/tmdb/3 + // - 反代：example.edgeone.run/api/tmdb -> https://example.edgeone.run/api/tmdb/3 + // - 用户已填到 /3：原样保留，不重复追加 + if !strings.HasSuffix(base, "/3") { + base = base + "/3" + } + return &TMDBScraper{ + APIKey: apiKey, + BaseURL: base, + client: &http.Client{Timeout: 15 * time.Second}, + } +} + +// tmdbSearchResult TMDB搜索结果 +type tmdbSearchResult struct { + Results []struct { + ID int `json:"id"` + Title string `json:"title"` + Name string `json:"name"` // 电视剧用name + Overview string `json:"overview"` + PosterPath string `json:"poster_path"` + ReleaseDate string `json:"release_date"` + FirstAirDate string `json:"first_air_date"` // 电视剧 + VoteAverage float32 `json:"vote_average"` + MediaType string `json:"media_type"` + GenreIDs []int `json:"genre_ids"` + } `json:"results"` +} + +// tmdbMovieDetail TMDB电影详情 +type tmdbMovieDetail struct { + ID int `json:"id"` + Title string `json:"title"` + Overview string `json:"overview"` + PosterPath string `json:"poster_path"` + ReleaseDate string `json:"release_date"` + VoteAverage float32 `json:"vote_average"` + Genres []struct { + ID int `json:"id"` + Name string `json:"name"` + } `json:"genres"` + Credits struct { + Cast []struct { + Name string `json:"name"` + } `json:"cast"` + } `json:"credits"` +} + +// doTMDBSearch 执行一次TMDB搜索请求 +// +// endpoint： +// - movie：/search/movie，参数参考 https://developer.themoviedb.org/reference/search-movie +// 支持 query / language / primary_release_year / year / region / include_adult / page +// - tv：/search/tv，参数参考 https://developer.themoviedb.org/reference/search-tv +// 支持 query / language / first_air_date_year / year / include_adult / page +// - multi：/search/multi，参数参考 https://developer.themoviedb.org/reference/search-multi +// 支持 query / language / include_adult / page / region（不支持按年份过滤） +// +// language 取值：zh-CN / en-US / 空字符串（不限定语言，让 TMDB 用所有别名匹配，模糊兜底） +// 注意：search_type=ngram 是早期废弃参数；TMDB 当前默认即支持子串/模糊匹配 +func (s *TMDBScraper) doTMDBSearch(endpoint, query, year, language string) (*tmdbSearchResult, error) { + if endpoint == "" { + endpoint = "movie" + } + if strings.TrimSpace(query) == "" { + return &tmdbSearchResult{}, nil + } + + params := url.Values{} + params.Set("api_key", s.APIKey) + params.Set("query", query) + params.Set("include_adult", "true") + params.Set("page", "1") + if language != "" { + params.Set("language", language) + } + if year != "" { + switch endpoint { + case "movie": + // movie 同时设置 year 和 primary_release_year，TMDB 文档建议优先使用 primary_release_year + params.Set("year", year) + params.Set("primary_release_year", year) + case "tv": + // tv 接口使用 first_air_date_year + params.Set("first_air_date_year", year) + params.Set("year", year) + case "multi": + // /search/multi 不支持按年份过滤；忽略 year 参数避免被反代/官方判为非法参数 + } + } + + searchURL := fmt.Sprintf("%s/search/%s?%s", s.BaseURL, endpoint, params.Encode()) + + // 输出脱敏后的请求 URL，便于排查刮削失败问题 + // （把 api_key 替换为 ***，避免日志泄漏密钥） + log.Debugf("[TMDB] 请求: %s", maskAPIKey(searchURL)) + + resp, err := s.client.Get(searchURL) + if err != nil { + log.Warnf("[TMDB] 请求失败 url=%s err=%v", maskAPIKey(searchURL), err) + return nil, fmt.Errorf("TMDB搜索请求失败: %w", err) + } + defer resp.Body.Close() + + body, _ := io.ReadAll(resp.Body) + var result tmdbSearchResult + if err := json.Unmarshal(body, &result); err != nil { + // 附带响应状态码和响应体片段，便于排查 API 地址错误、被代理拦截等问题 + snippet := string(body) + if len(snippet) > 200 { + snippet = snippet[:200] + } + log.Warnf("[TMDB] 解析失败 status=%d url=%s body=%s", resp.StatusCode, maskAPIKey(searchURL), snippet) + return nil, fmt.Errorf("TMDB搜索结果解析失败(status=%d, url=%s): %w, body=%s", + resp.StatusCode, searchURL, err, snippet) + } + // 输出响应概要 + log.Debugf("[TMDB] 响应 status=%d hits=%d url=%s", resp.StatusCode, len(result.Results), maskAPIKey(searchURL)) + + // /search/movie /search/tv 返回的结果不带 media_type，需要根据 endpoint 补齐 + if endpoint == "movie" || endpoint == "tv" { + for i := range result.Results { + if result.Results[i].MediaType == "" { + result.Results[i].MediaType = endpoint + } + } + } + return &result, nil +} + +// maskAPIKey 把 URL 中的 api_key 参数遮蔽为 ***，避免日志输出真实密钥 +var apiKeyMaskRe = regexp.MustCompile(`(api_key=)[^&]*`) + +func maskAPIKey(rawURL string) string { + return apiKeyMaskRe.ReplaceAllString(rawURL, "${1}***") +} + +// searchAttempt 单次搜索尝试参数 +type searchAttempt struct { + endpoint string // multi / movie / tv + query string + year string + language string +} + +// subtitleSepRe 副标题分隔符：「：」「:」「-」前后空白 +// 用于把「复仇者联盟3：无限战争」拆成「复仇者联盟3」和「无限战争」 +var subtitleSepRe = regexp.MustCompile(`\s*[：:\-—]\s*`) + +// TitleCandidate 单个搜索候选 +// 用于把文件名解析出来的多组标题/语言/年份/置信度统一成一条「线索」， +// 然后按 Confidence 排序后，逐条尝试 TMDB 搜索。 +type TitleCandidate struct { + Name string // 待搜索的标题字符串 + Lang string // "zh-CN" / "en-US" / ""（不指定） + Year string // 候选年份（可空） + Confidence float32 // 0-1，越高越优先；用于排序与「命中可信度」校验 + Source string // 来源标签：bracket-cn / main-cn / merged-cn / sub-cn / extra-cn / main-en / degenerate-cn / degenerate-en +} + +// tvHintRe 检测文件名是否带电视剧特征（SxxExx、季、集、Episode） +// 命中则只走 /search/tv 端点，否则只走 /search/movie，节约一半请求量 +var tvHintRe = regexp.MustCompile(`(?i)(s\d{1,2}e\d{1,3}|season\s*\d+|episode\s*\d+|第\s*\d+\s*季|第\s*\d+\s*集)`) + +// pickEndpoint 根据原始文件名启发式判断走 movie 还是 tv +func pickEndpoint(rawFileName string) string { + if tvHintRe.MatchString(rawFileName) { + return "tv" + } + return "movie" +} + +// englishStopwords 英文冠词/介词等无信息量小词，提取关键词时可丢弃 +// 注意：仅丢弃整段完全等于这些词的部分，不做子串匹配，避免误伤 "and" 在 "Andromeda" 里 +var englishStopwords = map[string]bool{ + "a": true, "an": true, "the": true, + "of": true, "in": true, "on": true, "at": true, "by": true, "to": true, "for": true, + "and": true, "or": true, "but": true, + "is": true, "are": true, "was": true, "were": true, + "with": true, "from": true, "into": true, "onto": true, + "vs": true, "v": true, +} + +// extractEnglishKeywords 从英文标题中提取最有信息量的关键词 +// +// 规则： +// 1. 按空格分词，去掉 stopwords 与长度 < 3 的词 +// 2. 取剩余词的前 3 个，用空格拼接（更长的标题往往是 TMDB 上没有的精确组合） +// 3. 如果剔除后没有剩余词，返回空字符串（让调用方跳过） +// +// 例： +// +// "And Soon The Darkness" -> "Soon Darkness" +// "Alvin and the Chipmunks" -> "Alvin Chipmunks" +// "30 Days Of Night Dark Days" -> "30 Days Night" +// "Iron Man" -> "Iron Man"（原样保留，关键词都够长） +// "The Dark Knight" -> "Dark Knight" +func extractEnglishKeywords(s string) string { + if s == "" { + return "" + } + words := strings.Fields(s) + keepers := make([]string, 0, len(words)) + for _, w := range words { + lw := strings.ToLower(strings.Trim(w, ".,;:!?'\"")) + if englishStopwords[lw] { + continue + } + if utf8.RuneCountInString(lw) < 3 && !isAllDigits(lw) { + // 长度 < 3 且不是纯数字的词丢弃（数字往往是片名一部分如"300"） + continue + } + keepers = append(keepers, w) + } + if len(keepers) == 0 { + return "" + } + if len(keepers) > 3 { + keepers = keepers[:3] + } + return strings.Join(keepers, " ") +} + +func isAllDigits(s string) bool { + if s == "" { + return false + } + for _, r := range s { + if r < '0' || r > '9' { + return false + } + } + return true +} + +// titleSimilarity 简易标题相似度（0-1） +// 规则： +// - 完全相等 -> 1.0 +// - 一个字符串包含另一个 -> 0.85 +// - 归一化（去空格、大小写）后相等 -> 0.95 +// - 归一化后一个包含另一个 -> 0.7 +// - 否则用「公共字符占比」粗算（避免引入完整编辑距离的开销） +func titleSimilarity(a, b string) float32 { + if a == "" || b == "" { + return 0 + } + if a == b { + return 1.0 + } + la := strings.ToLower(strings.TrimSpace(a)) + lb := strings.ToLower(strings.TrimSpace(b)) + if la == lb { + return 0.95 + } + if strings.Contains(la, lb) || strings.Contains(lb, la) { + return 0.85 + } + // 进一步归一化：去掉所有空白与标点，再比一次包含关系 + normA := normalizeForCompare(la) + normB := normalizeForCompare(lb) + if normA == normB && normA != "" { + return 0.92 + } + if normA != "" && normB != "" && (strings.Contains(normA, normB) || strings.Contains(normB, normA)) { + return 0.7 + } + // 兜底：公共字符比例（仅取较短字符串里的字符在较长字符串中出现的比例） + short, long := normA, normB + if len(long) < len(short) { + short, long = long, short + } + if short == "" { + return 0 + } + hit := 0 + for _, r := range short { + if strings.ContainsRune(long, r) { + hit++ + } + } + ratio := float32(hit) / float32(utf8.RuneCountInString(short)) + // 公共字符比例最高只贡献 0.5，避免冷门误判 + if ratio > 0.5 { + ratio = 0.5 + } + return ratio +} + +// normalizeForCompare 归一化字符串以做相似度比较： +// 去掉所有空白、标点（含中文标点），转小写。 +func normalizeForCompare(s string) string { + var b strings.Builder + for _, r := range s { + switch { + case r == ' ' || r == '\t' || r == '\n': + case r == '.' || r == ',' || r == '-' || r == '_' || r == ':' || r == ';' || r == '!' || r == '?': + case r == '：' || r == '，' || r == '。' || r == '、' || r == '！' || r == '？': + case r == '(' || r == ')' || r == '[' || r == ']' || r == '（' || r == '）' || r == '【' || r == '】': + default: + b.WriteRune(r) + } + } + return strings.ToLower(b.String()) +} + +// tmdbHit 从 tmdbSearchResult.Results 提取出来的单个命中项（仅用于内部传递） +type tmdbHit = struct { + ID int `json:"id"` + Title string `json:"title"` + Name string `json:"name"` + Overview string `json:"overview"` + PosterPath string `json:"poster_path"` + ReleaseDate string `json:"release_date"` + FirstAirDate string `json:"first_air_date"` + VoteAverage float32 `json:"vote_average"` + MediaType string `json:"media_type"` + GenreIDs []int `json:"genre_ids"` +} + +// scoreMatch 给一个 TMDB 搜索结果打分，作为「这个 hit 是否可信」的依据。 +// score = 0.5*titleSim + 0.3*yearScore + 0.2*voteScore (范围 0-1) +func scoreMatch(query string, year string, hit tmdbHit) float32 { + hitTitle := hit.Title + if hitTitle == "" { + hitTitle = hit.Name + } + hitYear := "" + if len(hit.ReleaseDate) >= 4 { + hitYear = hit.ReleaseDate[:4] + } else if len(hit.FirstAirDate) >= 4 { + hitYear = hit.FirstAirDate[:4] + } + + titleSim := titleSimilarity(query, hitTitle) + + var yearScore float32 + if year == "" { + // 无年份信号时给一个中性分（不奖励、不惩罚） + yearScore = 0.4 + } else if hitYear == "" { + // 文件名有年份但 TMDB 没填，给较低分 + yearScore = 0.2 + } else if hitYear == year { + yearScore = 1.0 + } else { + // 容忍 ±1 年（部分电影上映日期记录差一年） + diff := absInt(parseIntSafe(hitYear) - parseIntSafe(year)) + switch { + case diff == 1: + yearScore = 0.5 + case diff == 2: + yearScore = 0.2 + default: + yearScore = 0.0 + } + } + + // vote_average 0-10，用作冷门片名的最弱信号 + voteScore := hit.VoteAverage / 10.0 + if voteScore > 1 { + voteScore = 1 + } + + return 0.5*titleSim + 0.3*yearScore + 0.2*voteScore +} + +func absInt(x int) int { + if x < 0 { + return -x + } + return x +} + +func parseIntSafe(s string) int { + n := 0 + for _, r := range s { + if r < '0' || r > '9' { + return 0 + } + n = n*10 + int(r-'0') + } + return n +} + +// buildTitleCandidates 根据原始标题构造一组候选搜索词（按优先级返回） +// 候选生成策略，从精确到模糊： +// 1. 原始标题（保留所有信息） +// 2. 归一化标题（去括号/噪声词/分隔符 -> 空格合并） +// 3. 中文数字 -> 阿拉伯数字（钢铁侠三 -> 钢铁侠3） +// 4. 阿拉伯数字 -> 中文数字（钢铁侠3 -> 钢铁侠三，少数 TMDB 别名用中文数字） +// 5. 去除尾部数字/罗马数字（钢铁侠3 -> 钢铁侠，盗梦空间2 -> 盗梦空间） +// 6. 拆分多个词，每个非短词作为独立候选 +// 7. 副标题拆分（「复仇者联盟3：无限战争」 -> 「复仇者联盟3」、「无限战争」） +func buildTitleCandidates(title string) []string { + if title == "" { + return nil + } + seen := make(map[string]bool) + var out []string + add := func(s string) { + s = strings.TrimSpace(s) + if s == "" || seen[s] { + return + } + seen[s] = true + out = append(out, s) + } + + add(title) + norm := normalizeTitle(title) + add(norm) + + // 副标题拆分：以「：」「:」「-」分隔后得到多个候选 + // 例：「复仇者联盟3：无限战争」-> 主标题 "复仇者联盟3"、副标题 "无限战争" + // 「美国队长2：冬日战士」 -> 主标题 "美国队长2"、副标题 "冬日战士" + // 「雷神2：黑暗世界」 -> 主标题 "雷神2"、副标题 "黑暗世界" + subtitleParts := subtitleSepRe.Split(title, -1) + if len(subtitleParts) > 1 { + for _, p := range subtitleParts { + add(p) + } + } + + // 中文数字 <-> 阿拉伯数字 双向归一化 + arabic := cnNumToArabic(norm) + add(arabic) + add(arabicToCnNum(norm)) + + // 去除尾部数字（系列编号），帮助匹配主作品 + // 例：「钢铁侠3」-> 「钢铁侠」、「Iron Man 3」-> 「Iron Man」 + // 「异形1」-> 「异形」、「三十极夜2」-> 「三十极夜」 + // 这条对 title / norm / arabic 三者都做，提升 TMDB 中文别名不带后缀作品的命中率 + trailingNumRe := regexp.MustCompile(`[\s]*[0-9０-９]+\s*$`) + add(trailingNumRe.ReplaceAllString(title, "")) + add(trailingNumRe.ReplaceAllString(arabic, "")) + add(trailingNumRe.ReplaceAllString(norm, "")) + + // 去除尾部罗马数字（II / III / IV 等） + trailingRomanRe := regexp.MustCompile(`(?i)\s+(?:I{1,3}|IV|V|VI{0,3}|IX|X)$`) + add(trailingRomanRe.ReplaceAllString(norm, "")) + + // 若中文标题里夹杂了空格分隔的多个词，把每个非短词单独作为候选 + for _, w := range strings.Fields(norm) { + if utf8.RuneCountInString(w) >= 2 { + add(w) + } + } + // 阿拉伯数字归一化版本同样拆词 + for _, w := range strings.Fields(arabic) { + if utf8.RuneCountInString(w) >= 2 { + add(w) + } + } + + // 数字 + 中文 / 中文 + 数字 的组合标题，把中文部分单独提取作为候选 + // 例：「300勇士」 -> 候选包含 "勇士" + // 「3096天」 -> 候选包含 "天"（虽然过短） + // 「36总局」 -> 候选包含 "总局" + // 「47号物品」 -> 候选包含 "号物品" 较弱，跳过 + // 同时把开头数字单独作为候选，匹配「300」「3096」这类纯数字片名 + digitChineseRe := regexp.MustCompile(`^(\d+)([\p{Han}].*)$`) + chineseDigitRe := regexp.MustCompile(`^([\p{Han}].*?)(\d+)$`) + for _, s := range []string{title, norm} { + if m := digitChineseRe.FindStringSubmatch(s); m != nil { + // 数字部分（如 "300"） + add(m[1]) + // 中文部分（如 "勇士"），仅当中文长度 >= 2 才有用 + if utf8.RuneCountInString(m[2]) >= 2 { + add(m[2]) + } + } + if m := chineseDigitRe.FindStringSubmatch(s); m != nil { + // 中文部分（去掉尾部数字） + if utf8.RuneCountInString(m[1]) >= 2 { + add(m[1]) + } + // 数字部分单独作为候选 + add(m[2]) + } + } + + return out +} + +// arabicToCnNum 将标题中的单个阿拉伯数字归一化为中文数字（仅 0-9 的简单替换，10 以上不处理） +// 用于「钢铁侠3」-> 「钢铁侠三」，提升 TMDB 中文别名命中率 +func arabicToCnNum(s string) string { + if s == "" { + return s + } + rev := map[rune]string{ + '0': "零", '1': "一", '2': "二", '3': "三", '4': "四", + '5': "五", '6': "六", '7': "七", '8': "八", '9': "九", + } + // 仅当字符串包含中文且数字是单个字符（前后非数字）时才转换，避免 "2010" 年份被错误处理 + if !chineseRegexp.MatchString(s) { + return s + } + runes := []rune(s) + var b strings.Builder + for i, r := range runes { + if r >= '0' && r <= '9' { + // 检查是否是孤立的单个数字（前后不是数字） + prevDigit := i > 0 && runes[i-1] >= '0' && runes[i-1] <= '9' + nextDigit := i+1 < len(runes) && runes[i+1] >= '0' && runes[i+1] <= '9' + if !prevDigit && !nextDigit { + b.WriteString(rev[r]) + continue + } + } + b.WriteRune(r) + } + return b.String() +} + +// extractCandidates 把 parsedVideoTitle 转换成有序的 TitleCandidate 列表 +// +// 顺序原则：中文优先 -> 英文兜底；高置信度优先 -> 低置信度退化候选 +// 来源标签 (Source)： +// - bracket-cn : 来自方括号内的中文，置信度最高（发布组明确标注） +// - main-cn : 文件名中第一个中文片段 +// - main-en : 主英文标题 +// - merged-cn : 多个中文片段合并（300勇士：帝国崛起） +// - sub-cn : 中文标题的副标题拆分（无限战争） +// - extra-cn : 其它中文片段 +// - degenerate-cn / degenerate-en : 去尾数/去括号等退化形式 +func extractCandidates(parsed parsedVideoTitle, bracketChinese []string) []TitleCandidate { + var list []TitleCandidate + seen := make(map[string]bool) + add := func(name, lang, source string, conf float32) { + name = strings.TrimSpace(name) + if name == "" || utf8.RuneCountInString(name) < 1 { + return + } + key := lang + "|" + name + if seen[key] { + return + } + seen[key] = true + list = append(list, TitleCandidate{ + Name: name, + Lang: lang, + Year: parsed.Year, + Confidence: conf, + Source: source, + }) + } + + // 1) 方括号中文：发布组明确标注的片名，最可信 + for _, b := range bracketChinese { + add(b, "zh-CN", "bracket-cn", 0.95) + } + + // 2) 主中文标题 + if parsed.ChineseTitle != "" { + add(parsed.ChineseTitle, "zh-CN", "main-cn", 0.90) + // 副标题拆分：「复仇者联盟3：无限战争」-> 「复仇者联盟3」+「无限战争」 + parts := subtitleSepRe.Split(parsed.ChineseTitle, -1) + if len(parts) > 1 { + for _, p := range parts { + add(p, "zh-CN", "sub-cn", 0.55) + } + } + } + + // 3) 主英文标题 + if parsed.EnglishTitle != "" { + add(parsed.EnglishTitle, "en-US", "main-en", 0.80) + } + + // 4) 额外中文候选（解析出多个中文片段时；包括 merged 合并版本） + for _, t := range parsed.ExtraChineseTitles { + source := "extra-cn" + conf := float32(0.50) + if strings.Contains(t, "：") { + source = "merged-cn" + conf = 0.65 + } + add(t, "zh-CN", source, conf) + } + + // 5) 退化候选：对中文/英文都做「去尾部数字」「归一化」 + trailingNumRe := regexp.MustCompile(`[\s]*[0-9０-９]+\s*$`) + addDegenerate := func(src, lang, sourceTag string) { + if src == "" { + return + } + norm := normalizeTitle(src) + if norm != src { + add(norm, lang, sourceTag, 0.35) + } + if stripped := trailingNumRe.ReplaceAllString(src, ""); stripped != src { + add(stripped, lang, sourceTag, 0.30) + } + if stripped := trailingNumRe.ReplaceAllString(norm, ""); stripped != norm { + add(stripped, lang, sourceTag, 0.30) + } + // 中文数字 ⇄ 阿拉伯数字 + if arabic := cnNumToArabic(src); arabic != src { + add(arabic, lang, sourceTag, 0.35) + } + } + addDegenerate(parsed.ChineseTitle, "zh-CN", "degenerate-cn") + addDegenerate(parsed.EnglishTitle, "en-US", "degenerate-en") + + return list +} + +// searchWithFallback 带命中校验的多候选搜索 +// +// 流程： +// 1. 把候选按 Confidence 排序（中文高 > 中文低 > 英文 > 退化） +// 2. 对每个候选，最多发 2 次请求：带年份精搜 + 不带年份模糊搜 +// 3. 命中即用 scoreMatch 评分： +// - score >= matchConfidenceThreshold (0.60) → 立即返回 +// - score >= fallbackThreshold (0.30) → 暂存为「最佳兜底」继续尝试 +// - 其它 → 丢弃 +// 4. 全部尝试完仍无强命中 → 返回最佳兜底；都没有则报错 +// +// 端点选择：根据原始文件名启发式（含 SxxExx/季/集 -> tv，否则 movie），不再 movie/tv 互打 +func (s *TMDBScraper) searchWithFallback(parsed parsedVideoTitle, rawFileName string, bracketChinese []string) (*tmdbSearchResult, error) { + candidates := extractCandidates(parsed, bracketChinese) + // 按 Confidence 倒序，置信度高的先尝试 + sort.SliceStable(candidates, func(i, j int) bool { + return candidates[i].Confidence > candidates[j].Confidence + }) + + endpoint := pickEndpoint(rawFileName) + + // Debug 输出所有待尝试的候选 + log.Debugf("[TMDB] 候选生成 endpoint=%s year=%q candidates(%d):", endpoint, parsed.Year, len(candidates)) + for i, c := range candidates { + log.Debugf("[TMDB] #%d source=%-13s lang=%-5s conf=%.2f name=%q", i+1, c.Source, c.Lang, c.Confidence, c.Name) + } + + if len(candidates) == 0 { + return nil, fmt.Errorf("无法从文件名中提取有效标题") + } + + // 去重 + type attemptKey struct{ ep, q, y, l string } + done := make(map[attemptKey]bool) + attempts := int64(0) + + var bestFallback *tmdbSearchResult // 暂存的「分数过得去但不够强」的结果 + var bestScore float32 + var bestQuery string + var lastErr error + + tryOne := func(ep, q, year, lang, source string, candConf float32) (*tmdbSearchResult, float32, bool) { + k := attemptKey{ep, q, year, lang} + if done[k] { + return nil, 0, false + } + done[k] = true + attempts++ + + result, err := s.doTMDBSearch(ep, q, year, lang) + if err != nil { + lastErr = err + log.Debugf("[TMDB] try [%s] q=%q year=%q lang=%q -> 请求失败: %v", source, q, year, lang, err) + return nil, 0, false + } + if len(result.Results) == 0 { + log.Debugf("[TMDB] try [%s] q=%q year=%q lang=%q -> 0 hits", source, q, year, lang) + return nil, 0, false + } + first := result.Results[0] + score := scoreMatch(q, year, first) + hitTitle := first.Title + if hitTitle == "" { + hitTitle = first.Name + } + hitYear := "" + if len(first.ReleaseDate) >= 4 { + hitYear = first.ReleaseDate[:4] + } else if len(first.FirstAirDate) >= 4 { + hitYear = first.FirstAirDate[:4] + } + log.Debugf("[TMDB] try [%s] q=%q year=%q lang=%q -> hit{id=%d, title=%q, year=%s, vote=%.1f} score=%.2f", + source, q, year, lang, first.ID, hitTitle, hitYear, first.VoteAverage, score) + return result, score, true + } + + startTs := time.Now() + + for _, c := range candidates { + // 5.1 带年份精搜（如果有年份） + if c.Year != "" { + if res, score, ok := tryOne(endpoint, c.Name, c.Year, c.Lang, c.Source+"+y", c.Confidence); ok { + if score >= matchConfidenceThreshold { + recordHit(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ✅ 强命中 (%.2f >= %.2f) source=%s q=%q", score, matchConfidenceThreshold, c.Source, c.Name) + return res, nil + } + if score >= fallbackThreshold && score > bestScore { + bestFallback = res + bestScore = score + bestQuery = c.Name + } + } + } + // 5.2 不带年份模糊搜（兜底） + if res, score, ok := tryOne(endpoint, c.Name, "", c.Lang, c.Source, c.Confidence); ok { + if score >= matchConfidenceThreshold { + recordHit(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ✅ 强命中 (%.2f >= %.2f) source=%s q=%q", score, matchConfidenceThreshold, c.Source, c.Name) + return res, nil + } + if score >= fallbackThreshold && score > bestScore { + bestFallback = res + bestScore = score + bestQuery = c.Name + } + } + } + + // 没有强命中：使用 best fallback（可能仍然是错的，但至少有数据） + if bestFallback != nil { + recordLowConfidence(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ⚠️ 使用低置信度兜底命中 score=%.2f q=%q (low confidence < %.2f)", + bestScore, bestQuery, matchConfidenceThreshold) + return bestFallback, nil + } + + // 主流程完全无命中 → 进入「降级重试」阶段 + // 这一阶段主动放宽限制，尝试更“街头魔法”的查询方式： + // L2: 主候选换 multi 端点 + 不指定语言 → 匹配 TMDB 全语言别名 + // L3: 英文标题去冠词/取关键词重新搜 + // L4: 阈值降到 desperateThreshold (0.20)，换 multi 端点接纳「还有点像」的结果 + log.Debugf("[TMDB] 主流程无命中，进入降级重试阶段 attempts=%d", attempts) + + // L2: 主候选换 multi 端点 + 不指定语言 + for i, c := range candidates { + if i >= 3 { // 只对前 3 个高置信度候选走 multi，避免请求爆炸 + break + } + if res, score, ok := tryOne("multi", c.Name, "", "", c.Source+"@multi", c.Confidence); ok { + if score >= matchConfidenceThreshold { + recordHit(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ✅ L2 multi 端点强命中 (%.2f >= %.2f) q=%q", score, matchConfidenceThreshold, c.Name) + return res, nil + } + if score >= fallbackThreshold && score > bestScore { + bestFallback = res + bestScore = score + bestQuery = c.Name + } + } + } + if bestFallback != nil { + recordLowConfidence(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ⚠️ L2 低置信度兜底命中 score=%.2f q=%q", bestScore, bestQuery) + return bestFallback, nil + } + + // L3: 英文标题提取关键词（去冠词及过短词）重新搜索 + if kw := extractEnglishKeywords(parsed.EnglishTitle); kw != "" && kw != parsed.EnglishTitle { + log.Debugf("[TMDB] L3 英文关键词提取: %q -> %q", parsed.EnglishTitle, kw) + for _, ep := range []string{endpoint, "multi"} { + if res, score, ok := tryOne(ep, kw, "", "", "keywords-en", 0.4); ok { + if score >= matchConfidenceThreshold { + recordHit(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ✅ L3 关键词强命中 (%.2f >= %.2f) q=%q", score, matchConfidenceThreshold, kw) + return res, nil + } + if score >= fallbackThreshold && score > bestScore { + bestFallback = res + bestScore = score + bestQuery = kw + } + } + } + if bestFallback != nil { + recordLowConfidence(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ⚠️ L3 低置信度兜底命中 score=%.2f q=%q", bestScore, bestQuery) + return bestFallback, nil + } + } + + // L4: 「绝望式」阈值——陈障足以阁，重跳主候选但阈值降到 0.20 + // 说明：部分冷门片 vote_average=0、不同语言别名不及时补录，会被 0.30 阈值误杀， + // 这里使用更宽的 0.20 阈值接纳（阅读者可手动修正错误结果，总比空手好）。 + log.Debugf("[TMDB] L4 启用绝望式阈值 %.2f 重试主候选", desperateThreshold) + for i, c := range candidates { + if i >= 3 { + break + } + for _, ep := range []string{endpoint, "multi"} { + if res, score, ok := tryOne(ep, c.Name, "", "", c.Source+"@desperate", c.Confidence); ok { + if score >= desperateThreshold && score > bestScore { + bestFallback = res + bestScore = score + bestQuery = c.Name + } + } + } + } + if bestFallback != nil { + recordLowConfidence(attempts, time.Since(startTs)) + log.Debugf("[TMDB] ⚠️ L4 绝望式兜底命中 score=%.2f q=%q (阈值仅 %.2f，结果可能不准)", + bestScore, bestQuery, desperateThreshold) + return bestFallback, nil + } + + recordNoMatch(attempts, time.Since(startTs)) + titleInfo := parsed.ChineseTitle + if titleInfo == "" { + titleInfo = parsed.EnglishTitle + } + if lastErr != nil { + return nil, fmt.Errorf("TMDB未找到匹配结果: %s (last err: %v)", titleInfo, lastErr) + } + return nil, fmt.Errorf("TMDB未找到匹配结果: %s", titleInfo) +} + +// 统计辅助函数（使用 atomic 保证并发安全） +func recordHit(attempts int64, elapsed time.Duration) { + atomic.AddInt64(&globalTMDBStats.HitCount, 1) + atomic.AddInt64(&globalTMDBStats.TotalAttempts, attempts) + atomic.AddInt64(&globalTMDBStats.TotalSearchMillis, elapsed.Milliseconds()) +} + +func recordLowConfidence(attempts int64, elapsed time.Duration) { + atomic.AddInt64(&globalTMDBStats.LowConfidenceCount, 1) + atomic.AddInt64(&globalTMDBStats.TotalAttempts, attempts) + atomic.AddInt64(&globalTMDBStats.TotalSearchMillis, elapsed.Milliseconds()) +} + +func recordNoMatch(attempts int64, elapsed time.Duration) { + atomic.AddInt64(&globalTMDBStats.NoMatchCount, 1) + atomic.AddInt64(&globalTMDBStats.TotalAttempts, attempts) + atomic.AddInt64(&globalTMDBStats.TotalSearchMillis, elapsed.Milliseconds()) +} + +// ScrapeVideo 刮削视频信息 +func (s *TMDBScraper) ScrapeVideo(item *model.MediaItem) error { + if s.APIKey == "" { + return fmt.Errorf("TMDB API Key 未配置") + } + + atomic.AddInt64(&globalTMDBStats.TotalScraped, 1) + + // 始终从文件名中解析出标题和年份（ScrapedName 是刮削结果字段，不作为搜索输入） + parsed := parseVideoFileName(item.FileName) + // 在剥离括号前再次抽一次方括号中文，作为最高置信度候选源 + bracketChinese := extractBracketChinese(item.FileName) + log.Debugf("[TMDB] 开始刮削 file=%q parsed={chinese=%q english=%q year=%q extras=%v bracket=%v} baseURL=%s", + item.FileName, parsed.ChineseTitle, parsed.EnglishTitle, parsed.Year, parsed.ExtraChineseTitles, bracketChinese, s.BaseURL) + + // 搜索策略：中文标题优先，英文标题兜底，都搜不到才失败 + searchResult, err := s.searchWithFallback(parsed, item.FileName, bracketChinese) + if err != nil { + log.Debugf("[TMDB] ❌ 刮削失败 file=%q err=%v", item.FileName, err) + return err + } + + // 取第一个结果 + first := searchResult.Results[0] + mediaType := first.MediaType + if mediaType == "" { + mediaType = "movie" + } + + // 获取详情（语言跟随「首选刮削语言」设置） + detailURL := fmt.Sprintf("%s/%s/%d?api_key=%s&language=%s&append_to_response=credits", + s.BaseURL, mediaType, first.ID, s.APIKey, url.QueryEscape(ScrapeLanguage())) + log.Debugf("[TMDB] 详情请求: %s", maskAPIKey(detailURL)) + + detailResp, err := s.client.Get(detailURL) + if err != nil { + return fmt.Errorf("TMDB详情请求失败: %w", err) + } + defer detailResp.Body.Close() + + detailBody, _ := io.ReadAll(detailResp.Body) + var detail tmdbMovieDetail + if err := json.Unmarshal(detailBody, &detail); err != nil { + return fmt.Errorf("TMDB详情解析失败: %w", err) + } + + // 填充字段（仅填空字段，已有值不覆盖；ScrapedAt 由调用方统一设置） + title := detail.Title + if title == "" { + title = first.Name + } + // ScrapedName 处理： + // - 扫描阶段会把「去扩展名的文件名」作为默认占位写入 ScrapedName， + // 这种占位值应该被刮削得到的真实标题覆盖； + // - 用户手动编辑过的 ScrapedName 不会等于该占位值，保留不覆盖。 + if title != "" { + defaultName := strings.TrimSuffix(item.FileName, stdpath.Ext(item.FileName)) + if item.ScrapedName == "" || item.ScrapedName == defaultName { + item.ScrapedName = title + } + } + if item.Plot == "" { + item.Plot = detail.Overview + } + if item.Rating == 0 { + item.Rating = detail.VoteAverage + } + if item.ExternalID == "" { + item.ExternalID = fmt.Sprintf("tmdb:%d", detail.ID) + } + if item.VideoType == "" { + item.VideoType = mediaType + } + + releaseDate := detail.ReleaseDate + if releaseDate == "" { + releaseDate = first.FirstAirDate + } + if item.ReleaseDate == "" { + item.ReleaseDate = releaseDate + } + + if item.Cover == "" && detail.PosterPath != "" { + item.Cover = tmdbImageBase + detail.PosterPath + } + + // 类型 + if item.Genre == "" { + genres := make([]string, 0, len(detail.Genres)) + for _, g := range detail.Genres { + genres = append(genres, g.Name) + } + item.Genre = strings.Join(genres, ",") + } + + // 演员（取前10个） + if item.Authors == "" { + actors := make([]string, 0) + for i, cast := range detail.Credits.Cast { + if i >= 10 { + break + } + actors = append(actors, cast.Name) + } + authorsJSON, _ := json.Marshal(actors) + item.Authors = string(authorsJSON) + } + + now := time.Now() + item.ScrapedAt = &now + hitTitle := title + hitYear := "" + if len(detail.ReleaseDate) >= 4 { + hitYear = detail.ReleaseDate[:4] + } + log.Debugf("[TMDB] ✅ 刮削完成 file=%q -> tmdb_id=%d, title=%q, year=%s, mediaType=%s", + item.FileName, detail.ID, hitTitle, hitYear, mediaType) + return nil +} diff --git a/internal/media/strm.go b/internal/media/strm.go new file mode 100644 index 00000000..9b606f59 --- /dev/null +++ b/internal/media/strm.go @@ -0,0 +1,336 @@ +package media + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + stdpath "path" + "strings" + "sync/atomic" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/internal/sign" + "github.com/OpenListTeam/OpenList/v4/internal/stream" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" + log "github.com/sirupsen/logrus" +) + +var apiRootHolder atomic.Value + +// SetAPIRoot 记录本次运行使用的站点地址（由 HTTP 层注入，供异步任务复用） +func SetAPIRoot(u string) { + if u == "" { + return + } + apiRootHolder.Store(strings.TrimSuffix(u, "/")) +} + +// GetAPIRoot 返回当前站点地址 +func GetAPIRoot() string { + if v, ok := apiRootHolder.Load().(string); ok && v != "" { + return v + } + return strings.TrimSuffix(conf.Conf.SiteURL, "/") +} + +// StrmOptions STRM 生成参数 +type StrmOptions struct { + APIRoot string // 站点地址前缀，例如 https://example.com + Template string // 内容模板 + Overwrite bool // 是否覆盖已存在文件 + SameDir bool // 是否与媒体文件同目录（同名 .strm） + OutputPath string // 输出目录（VFS 路径，仅 SameDir=false 时使用） +} + +// StrmTarget 一个待生成的 .strm 目标 +type StrmTarget struct { + RelPath string // 相对输出目录的路径 + RawPath string // 对应媒体文件的 VFS 路径 + SameDirPath string // 与媒体文件同目录时的 .strm 路径 +} + +// strmSibling 计算与媒体文件同目录、同名的 .strm 路径 +func strmSibling(rawPath string, isDir bool) string { + if isDir { + return stdpath.Join(rawPath, stdpath.Base(rawPath)+".strm") + } + ext := stdpath.Ext(rawPath) + if ext == "" { + return rawPath + ".strm" + } + return strings.TrimSuffix(rawPath, ext) + ".strm" +} + +// DefaultStrmOptions 从设置中读取 STRM 参数 +func DefaultStrmOptions(apiRoot string) StrmOptions { + return StrmOptions{ + APIRoot: strings.TrimSuffix(apiRoot, "/"), + Template: setting.GetStr(conf.MediaStrmTemplate, "{site_url}/d{path}?sign={sign}"), + Overwrite: setting.GetBool(conf.MediaStrmOverwrite), + SameDir: setting.GetBool(conf.MediaStrmSameDir), + OutputPath: setting.GetStr(conf.MediaStrmPath, "/strm"), + } +} + +// RenderStrmContent 渲染 .strm 内容（单行 URL） +func RenderStrmContent(tpl, apiRoot, rawPath string) string { + if tpl == "" { + tpl = "{site_url}/d{path}?sign={sign}" + } + apiRoot = strings.TrimSuffix(apiRoot, "/") + p := rawPath + if !strings.HasPrefix(p, "/") { + p = "/" + p + } + replaced := strings.NewReplacer( + "{site_url}", apiRoot, + "{path}", utils.EncodePath(p), + "{sign}", sign.Sign(rawPath), + "{raw_path}", p, + "{name}", stdpath.Base(p), + ).Replace(tpl) + return strings.TrimSpace(replaced) +} + +// sanitizeName 清洗文件/目录名中的非法字符 +func sanitizeName(s string) string { + s = strings.TrimSpace(s) + if s == "" { + return "Unknown" + } + replacer := strings.NewReplacer( + "/", "_", "\\", "_", ":", "_", "*", "_", "?", "_", + "\"", "_", "<", "_", ">", "_", "|", "_", + "\n", " ", "\r", " ", + ) + s = replacer.Replace(s) + if runes := []rune(s); len(runes) > 120 { + s = string(runes[:120]) + } + return strings.TrimSpace(s) +} + +func itemTitle(item *model.MediaItem) string { + if item.ScrapedName != "" { + return item.ScrapedName + } + name := item.FileName + if !item.IsFolder { + name = strings.TrimSuffix(name, stdpath.Ext(name)) + } + return name +} + +func itemYear(item *model.MediaItem) string { + if len(item.ReleaseDate) >= 4 { + return item.ReleaseDate[:4] + } + return "" +} + +func parseEpisodes(item *model.MediaItem) []model.EpisodeInfo { + if item.Episodes == "" { + return nil + } + var eps []model.EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil { + return nil + } + return eps +} + +// MediaItemFolder 条目所在目录 +func MediaItemFolder(item *model.MediaItem) string { + if item == nil { + return "/" + } + folder := strings.TrimSuffix(item.FolderPath, "/") + if folder == "" { + folder = "/" + } + return folder +} + +// StrmTargets 计算条目对应的 STRM 目标列表。 +// 图片/书籍返回空列表（不生成 STRM）。 +func StrmTargets(item *model.MediaItem) []StrmTarget { + if item == nil { + return nil + } + title := sanitizeName(itemTitle(item)) + year := itemYear(item) + folder := MediaItemFolder(item) + + switch item.MediaType { + case model.MediaTypeVideo: + if item.VideoType == "tv" { + series := title + season := item.Season + if season <= 0 { + season = 1 + } + var out []StrmTarget + if item.IsFolder { + for i, ep := range parseEpisodes(item) { + num := ep.Index + if num <= 0 { + num = i + 1 + } + epTitle := sanitizeName(ep.Title) + epRaw := stdpath.Join(folder, item.FileName, ep.FileName) + out = append(out, StrmTarget{ + RelPath: stdpath.Join(series, fmt.Sprintf("Season %02d", season), + fmt.Sprintf("%s S%02dE%02d - %s.strm", series, season, num, epTitle)), + RawPath: epRaw, + SameDirPath: strmSibling(epRaw, false), + }) + } + return out + } + num := item.Episode + if num <= 0 { + num = 1 + } + raw := stdpath.Join(folder, item.FileName) + return []StrmTarget{{ + RelPath: stdpath.Join(series, fmt.Sprintf("Season %02d", season), + fmt.Sprintf("%s S%02dE%02d.strm", series, season, num)), + RawPath: raw, + SameDirPath: strmSibling(raw, false), + }} + } + name := title + if year != "" { + name = fmt.Sprintf("%s (%s)", title, year) + } + raw := stdpath.Join(folder, item.FileName) + return []StrmTarget{{ + RelPath: stdpath.Join(name, name+".strm"), + RawPath: raw, + SameDirPath: strmSibling(raw, item.IsFolder), + }} + case model.MediaTypeMusic: + artist := sanitizeName(item.AlbumArtist) + if artist == "" { + artist = "Unknown Artist" + } + album := sanitizeName(item.AlbumName) + if album == "" { + album = "Unknown Album" + } + track := item.TrackNumber + if track <= 0 { + track = 1 + } + raw := stdpath.Join(folder, item.FileName) + return []StrmTarget{{ + RelPath: stdpath.Join(artist, album, fmt.Sprintf("%02d - %s.strm", track, title)), + RawPath: raw, + SameDirPath: strmSibling(raw, false), + }} + default: + return nil + } +} + +// GenerateStrmForItems 为一组条目生成 STRM 文件，返回写入数量 +func GenerateStrmForItems(ctx context.Context, items []model.MediaItem, opts StrmOptions) (int, error) { + if !opts.SameDir && opts.OutputPath == "" { + return 0, fmt.Errorf("strm output path is empty") + } + // 归一输出根目录，后续用它校验每个目标文件都在根目录之内 + outRoot := "" + if !opts.SameDir { + outRoot = stdpath.Clean("/" + strings.TrimSpace(opts.OutputPath)) + } + written := 0 + for i := range items { + item := &items[i] + if item.Hidden { + continue + } + for _, target := range StrmTargets(item) { + dst := target.SameDirPath + if !opts.SameDir { + dst = stdpath.Join(outRoot, target.RelPath) + // RelPath 含刮削得到的艺术家/专辑名等外部数据，必须保证不会借 ../ 逃出输出目录 + if dst != outRoot && !strings.HasPrefix(dst, strings.TrimSuffix(outRoot, "/")+"/") { + continue + } + } + if dst == "" { + continue + } + if !opts.Overwrite { + if _, err := fs.Get(ctx, dst, &fs.GetArgs{NoLog: true}); err == nil { + continue + } + } + data := []byte(RenderStrmContent(opts.Template, opts.APIRoot, target.RawPath)) + file := &stream.FileStream{ + Ctx: ctx, + Obj: &model.Object{ + Name: stdpath.Base(dst), + Size: int64(len(data)), + Modified: time.Now(), + }, + Reader: bytes.NewReader(data), + Mimetype: "text/plain", + } + if err := fs.PutDirectly(ctx, stdpath.Dir(dst), file, true); err != nil { + return written, fmt.Errorf("write %s: %w", dst, err) + } + written++ + } + } + return written, nil +} + +// GenerateStrmForType 按媒体类型（可选扫描路径）批量生成 STRM +func GenerateStrmForType(ctx context.Context, mediaType model.MediaType, scanPathID uint, opts StrmOptions) (int, error) { + var items []model.MediaItem + var err error + if scanPathID > 0 { + items, err = db.ListMediaItemsByScanPath(scanPathID) + } else { + items, err = db.ListAllMediaItems(mediaType) + } + if err != nil { + return 0, err + } + if mediaType != "" { + filtered := make([]model.MediaItem, 0, len(items)) + for _, it := range items { + if it.MediaType == mediaType { + filtered = append(filtered, it) + } + } + items = filtered + } + return GenerateStrmForItems(ctx, items, opts) +} + +// TriggerStrmOnScan 扫描结束后按需异步生成 STRM +func TriggerStrmOnScan(mediaType model.MediaType, apiRoot string) { + if !setting.GetBool(conf.MediaStrmOnScan) { + return + } + opts := DefaultStrmOptions(apiRoot) + if !opts.SameDir && opts.OutputPath == "" { + return + } + go func() { + n, err := GenerateStrmForType(context.Background(), mediaType, 0, opts) + if err != nil { + log.Errorf("[media] generate strm for %s failed: %+v", mediaType, err) + return + } + log.Infof("[media] generated %d strm files for %s", n, mediaType) + }() +} diff --git a/internal/model/media.go b/internal/model/media.go new file mode 100644 index 00000000..e7713e8e --- /dev/null +++ b/internal/model/media.go @@ -0,0 +1,129 @@ +package model + +import ( + "time" + + "gorm.io/gorm" +) + +// MediaType 媒体类型，使用字符串便于后期扩展 +type MediaType string + +const ( + MediaTypeVideo MediaType = "video" + MediaTypeMusic MediaType = "music" + MediaTypeImage MediaType = "image" + MediaTypeBook MediaType = "book" +) + +// MediaScanPath 媒体库扫描路径（每个媒体库可配置多个扫描路径） +type MediaScanPath struct { + gorm.Model + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + DeletedAt gorm.DeletedAt `gorm:"index" json:"-"` + + MediaType MediaType `gorm:"index;not null" json:"media_type"` // 所属媒体类型 + Name string `json:"name"` // 路径名称（用于前端筛选显示） + Path string `gorm:"not null" json:"path"` // VFS 扫描路径 + PathMerge bool `gorm:"default:false" json:"path_merge"` // 路径合并模式（子文件夹作为一个条目） + TypeTag string `json:"type_tag"` // 类型标签：电影、电视剧等 + ContentTags string `json:"content_tags"` // 内容标签：喜剧、惊悚等（逗号分隔） + EnableScrape bool `gorm:"default:true" json:"enable_scrape"` // 是否启用刮削 + LastScanAt *time.Time `json:"last_scan_at"` // 最后扫描时间 +} + +// MediaIndexColumnMax 复合唯一索引列（folder_path / file_name / album_name）的最大字符数 +const MediaIndexColumnMax = 191 + +// MediaItem 媒体条目（统一表，通过 media_type 区分类型，便于后期扩展新类型） +type MediaItem struct { + gorm.Model + // 覆盖 gorm.Model 的 ID 字段，使 JSON 序列化为小写 "id"，与前端保持一致 + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + DeletedAt gorm.DeletedAt `gorm:"index" json:"-"` + // 基础信息 + MediaType MediaType `gorm:"index;not null" json:"media_type"` + ScanPathID uint `gorm:"index" json:"scan_path_id"` // 关联的扫描路径ID + // 三个字段共同组成复合唯一索引。MySQL 对「有索引且未指定长度」的 string 会生成 longtext， + // 而 TEXT 列不能直接建索引（AutoMigrate 会失败）；因此必须显式限定 size。 + // utf8mb4 下 InnoDB 索引总长上限 3072 字节：191*3*4 = 2292，留足余量。 + FileName string `gorm:"size:191;uniqueIndex:idx_media_folder_file_album" json:"file_name"` + FileSize int64 `json:"file_size"` + MimeType string `json:"mime_type"` + Hidden bool `gorm:"default:false" json:"hidden"` + + // 刮削/编辑信息 + ScrapedName string `json:"scraped_name"` + Description string `gorm:"type:text" json:"description"` + Cover string `json:"cover"` // 封面URL或本地路径 + ReleaseDate string `json:"release_date"` // 发布时间，格式 YYYY-MM-DD + Rating float32 `json:"rating"` // 评分 0-10 + Genre string `json:"genre"` // 类型，逗号分隔，如 "动作,科幻" + Authors string `gorm:"type:text" json:"authors"` // 作者/演员，JSON数组字符串 + Plot string `gorm:"type:text" json:"plot"` // 剧情/内容介绍 + Reviews string `gorm:"type:text" json:"reviews"` // 用户评价，JSON数组字符串 + + // 外部ID（用于刮削关联） + ExternalID string `json:"external_id"` // TMDB ID / Discogs ID / 豆瓣ID + + // 音乐专属字段 + AlbumName string `gorm:"size:191;uniqueIndex:idx_media_folder_file_album" json:"album_name"` // 所属专辑名 + AlbumArtist string `json:"album_artist"` // 专辑艺术家 + TrackNumber int `json:"track_number"` // 曲目编号 + Duration int `json:"duration"` // 时长（秒） + Lyrics string `gorm:"type:text" json:"lyrics"` // LRC格式歌词 + + // 视频专属字段 + VideoType string `json:"video_type"` // "movie" 或 "tv" + Season int `json:"season"` // 季（电视剧） + Episode int `json:"episode"` // 集（电视剧） + + // 书籍专属字段 + Publisher string `json:"publisher"` // 出版社 + ISBN string `json:"isbn"` // ISBN + + // 目录合并模式 + IsFolder bool `gorm:"default:false" json:"is_folder"` // 是否为文件夹模式条目 + FolderPath string `gorm:"size:191;index;uniqueIndex:idx_media_folder_file_album" json:"folder_path"` // 扫描根路径（恒定为扫描路径，与file_name+album_name组合唯一） + Episodes string `gorm:"type:text" json:"episodes"` // 选集信息，JSON数组，格式：[{file_name,index,title},...] + + ScrapedAt *time.Time `json:"scraped_at"` +} + +// MediaConfig 媒体库配置（每种类型一条记录） +type MediaConfig struct { + gorm.Model + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + DeletedAt gorm.DeletedAt `gorm:"index" json:"-"` + MediaType MediaType `gorm:"uniqueIndex;not null" json:"media_type"` + Enabled bool `gorm:"default:false" json:"enabled"` + LastScanAt *time.Time `json:"last_scan_at"` + LastScrapeAt *time.Time `json:"last_scrape_at"` + // ScraperConfig 该媒体类型独立的刮削配置（JSON）： + // store_thumbnail / thumbnail_mode / thumbnail_path，未设置时回落到全局设置 + ScraperConfig string `gorm:"type:text" json:"scraper_config"` +} + +// EpisodeInfo 选集信息（文件夹型媒体的单集描述） +type EpisodeInfo struct { + FileName string `json:"file_name"` // 原始文件名（含扩展名） + Index int `json:"index"` // 序号，默认0，文件名开头有数字则取该数字 + Title string `json:"title"` // 选集标题（去掉序号后的文件名，不含扩展名） +} + +// MediaScanProgress 扫描进度（内存中维护，不持久化） +type MediaScanProgress struct { + MediaType MediaType `json:"media_type"` + ScanPathID uint `json:"scan_path_id,omitempty"` // 0 表示全部路径 + Running bool `json:"running"` + Total int `json:"total"` + Done int `json:"done"` + Message string `json:"message"` + Error string `json:"error,omitempty"` +} \ No newline at end of file diff --git a/internal/model/media_extra.go b/internal/model/media_extra.go new file mode 100644 index 00000000..658cfa4f --- /dev/null +++ b/internal/model/media_extra.go @@ -0,0 +1,91 @@ +package model + +import "time" + +// MediaAppKey 媒体应用密钥。 +// 用于无法走交互式登录的客户端（Subsonic 的 token 认证、Emby 的 api_key、 +// MoviePilot 对接等）。密钥本身以 AES-GCM 加密存储，数据库中不保留明文。 +type MediaAppKey struct { + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + + UserID uint `gorm:"index;not null" json:"user_id"` + Name string `json:"name"` // 备注名，例如 "Symfonium"、"MoviePilot" + Prefix string `gorm:"index" json:"prefix"` // 明文前缀（便于识别/排查） + Secret string `gorm:"not null" json:"-"` // AES-GCM 密文（base64） + + LastUsedAt *time.Time `json:"last_used_at"` +} + +// MediaUserData 用户对媒体条目的播放状态，Emby 与 Subsonic 共用。 +type MediaUserData struct { + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + + UserID uint `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"user_id"` + ItemID uint `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"item_id"` + EpisodeIndex int `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"episode_index"` // 0 表示条目本身，>0 表示选集序号 + + Played bool `json:"played"` + PlayCount int `json:"play_count"` + IsFavorite bool `gorm:"index" json:"is_favorite"` + Rating int `json:"rating"` // 0-5，Subsonic 语义 + PositionTicks int64 `json:"position_ticks"` // 100ns tick，与 Emby 一致 + LastPlayedAt *time.Time `json:"last_played_at"` +} + +// MediaPlaylist 用户播放列表（Subsonic 需要，Emby 的 Playlist 复用同一份数据） +type MediaPlaylist struct { + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + + UserID uint `gorm:"index;not null" json:"user_id"` + Name string `json:"name"` + Comment string `json:"comment"` + Public bool `json:"public"` + ItemCount int `json:"item_count"` +} + +// MediaPlaylistItem 播放列表条目（按 Order 排序） +type MediaPlaylistItem struct { + ID uint `gorm:"primarykey" json:"id"` + PlaylistID uint `gorm:"index;not null" json:"playlist_id"` + ItemID uint `gorm:"index;not null" json:"item_id"` + EpisodeIndex int `json:"episode_index"` + Order int `gorm:"column:sort_order" json:"order"` +} + +// MediaAnnotation 用户在 PDF 阅读器里添加的批注。 +// +// 只存查看器原生的批注对象（Anchor，JSON）：页码、rect/quadPoints、颜色、 +// 文字、绘制点集都在里面，拆成独立列会出现两份可能不一致的数据。 +// 重新打开时原样回灌给 PDF 引擎，因此这里不重复建模。 +type MediaAnnotation struct { + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + + UserID uint `gorm:"uniqueIndex:idx_media_annotation_key;not null" json:"-"` + ItemID uint `gorm:"uniqueIndex:idx_media_annotation_key;not null;index" json:"item_id"` + // RemoteID 是查看器分配的批注 id：多设备打开同一份 PDF 时靠它做去重与更新 + RemoteID string `gorm:"uniqueIndex:idx_media_annotation_key;size:128;not null" json:"remote_id"` + Anchor string `gorm:"type:text;not null" json:"anchor"` +} + +// EmbyDevice Emby 客户端会话（由 AuthenticateByName 签发） +type EmbyDevice struct { + ID uint `gorm:"primarykey" json:"id"` + CreatedAt time.Time `json:"created_at"` + + UserID uint `gorm:"index;not null" json:"user_id"` + DeviceID string `json:"device_id"` // 客户端上报的 DeviceId + DeviceName string `json:"device_name"` + Client string `json:"client"` + Version string `json:"version"` + TokenHash string `gorm:"uniqueIndex;not null" json:"-"` + AppKeyID uint `json:"app_key_id"` // 通过应用密钥认证时为对应 key，否则为 0 + LastSeenAt time.Time `json:"last_seen_at"` +} diff --git a/internal/model/setting.go b/internal/model/setting.go index 1e0fda0b..d367aa50 100644 --- a/internal/model/setting.go +++ b/internal/model/setting.go @@ -13,6 +13,8 @@ const ( S3 FTP TRAFFIC + MEDIA + TRANSCODE ) const ( diff --git a/internal/transcode/cache.go b/internal/transcode/cache.go new file mode 100644 index 00000000..918874ce --- /dev/null +++ b/internal/transcode/cache.go @@ -0,0 +1,489 @@ +package transcode + +import ( + "errors" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +// SegmentInfo 单个切片元数据 +type SegmentInfo struct { + Seq int `json:"seq"` + Duration float64 `json:"duration"` + Size int64 `json:"size"` + Path string `json:"-"` // 本地实际文件路径 +} + +// JobCache 单个 Job 的切片缓存 +type JobCache struct { + JobID string + Profile string + BaseDir string + + mu sync.RWMutex + segments map[int]*SegmentInfo + final bool // 是否所有切片都到齐 + waiters map[int][]chan struct{} // seq -> 等待这一片的 channel 列表 +} + +// Cache 全局切片缓存 +type Cache struct { + rootDir string + maxBytes int64 + mu sync.RWMutex + jobs map[string]map[string]*JobCache // jobID -> profile -> cache +} + +func NewCache(rootDir string, maxGB int64) *Cache { + if maxGB <= 0 { + maxGB = 20 + } + return &Cache{ + rootDir: rootDir, + maxBytes: maxGB * 1024 * 1024 * 1024, + jobs: make(map[string]map[string]*JobCache), + } +} + +// Init 创建根目录 +func (c *Cache) Init() error { + return os.MkdirAll(c.rootDir, 0o755) +} + +const ( + // MaxSegmentBytes 单个切片允许写入的最大字节数，防止 callback_token 持有者打满磁盘 + MaxSegmentBytes int64 = 32 << 20 + // maxSegmentSeq 切片序号上限（按 2 秒/片约 6 天），拒绝负数与异常大值 + maxSegmentSeq = 200000 +) + +var ( + profileNameRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,32}$`) + jobIDRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`) + + // ErrInvalidProfile profile 名称不合法（可能是路径穿越尝试） + ErrInvalidProfile = errors.New("invalid profile name") + // ErrInvalidJobID job id 不合法 + ErrInvalidJobID = errors.New("invalid job id") + // ErrInvalidSegment 切片序号越界 + ErrInvalidSegment = errors.New("invalid segment seq") + // ErrSegmentTooLarge 切片超过大小上限 + ErrSegmentTooLarge = errors.New("segment too large") +) + +// ValidProfileName 判断 profile 是否为安全的目录名（与 BuildProfile 产出的 source/1080p/720p 对齐）。 +func ValidProfileName(profile string) bool { + return profileNameRe.MatchString(profile) +} + +// safeJoin 在 rootDir 下拼接子路径，并保证结果不会越出 rootDir。 +func (c *Cache) safeJoin(jobID, profile string) (string, error) { + if !jobIDRe.MatchString(jobID) { + return "", ErrInvalidJobID + } + if !ValidProfileName(profile) { + return "", ErrInvalidProfile + } + full := filepath.Join(c.rootDir, jobID, profile) + rel, err := filepath.Rel(c.rootDir, full) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return "", ErrInvalidProfile + } + return full, nil +} + +// getOrCreate 获取或创建 (job, profile) 的 cache +func (c *Cache) getOrCreate(jobID, profile string) (*JobCache, error) { + baseDir, err := c.safeJoin(jobID, profile) + if err != nil { + return nil, err + } + c.mu.Lock() + defer c.mu.Unlock() + pmap, ok := c.jobs[jobID] + if !ok { + pmap = make(map[string]*JobCache) + c.jobs[jobID] = pmap + } + jc, ok := pmap[profile] + if !ok { + jc = &JobCache{ + JobID: jobID, + Profile: profile, + BaseDir: baseDir, + segments: make(map[int]*SegmentInfo), + waiters: make(map[int][]chan struct{}), + } + if err := os.MkdirAll(jc.BaseDir, 0o755); err != nil { + return nil, err + } + pmap[profile] = jc + } + return jc, nil +} + +// PutSegment 写入一个切片，并通知所有等待者 +func (c *Cache) PutSegment(jobID, profile string, seq int, duration float64, body io.Reader, isFinal bool) (*SegmentInfo, error) { + if seq < 0 || seq > maxSegmentSeq { + return nil, ErrInvalidSegment + } + jc, err := c.getOrCreate(jobID, profile) + if err != nil { + return nil, err + } + fname := fmt.Sprintf("seg-%d.ts", seq) + fpath := filepath.Join(jc.BaseDir, fname) + tmpPath := fpath + ".tmp" + f, err := os.Create(tmpPath) + if err != nil { + return nil, err + } + // 多读 1 字节用于判断是否超限 + n, err := io.Copy(f, io.LimitReader(body, MaxSegmentBytes+1)) + if err == nil && n > MaxSegmentBytes { + err = ErrSegmentTooLarge + } + cerr := f.Close() + if err != nil { + os.Remove(tmpPath) + return nil, err + } + if cerr != nil { + os.Remove(tmpPath) + return nil, cerr + } + if err := os.Rename(tmpPath, fpath); err != nil { + os.Remove(tmpPath) + return nil, err + } + info := &SegmentInfo{Seq: seq, Duration: duration, Size: n, Path: fpath} + + jc.mu.Lock() + jc.segments[seq] = info + if isFinal { + jc.final = true + } + // 通知所有等待 <= seq 的 waiter + var toClose []chan struct{} + for s, chs := range jc.waiters { + if s <= seq { + toClose = append(toClose, chs...) + delete(jc.waiters, s) + } + } + jc.mu.Unlock() + for _, ch := range toClose { + close(ch) + } + return info, nil +} + +// GetSegment 读取一个切片（不等待） +func (c *Cache) GetSegment(jobID, profile string, seq int) (*SegmentInfo, error) { + c.mu.RLock() + pmap, ok := c.jobs[jobID] + c.mu.RUnlock() + if !ok { + return nil, errors.New("job not found") + } + jc, ok := pmap[profile] + if !ok { + return nil, errors.New("profile not found") + } + jc.mu.RLock() + defer jc.mu.RUnlock() + info, ok := jc.segments[seq] + if !ok { + return nil, errors.New("segment not found") + } + return info, nil +} + +// WaitForSegment 等待某个 seq 切片就绪，timeout 后返回 nil +func (c *Cache) WaitForSegment(jobID, profile string, seq int, timeout time.Duration) (*SegmentInfo, error) { + jc, err := c.getOrCreate(jobID, profile) + if err != nil { + return nil, err + } + jc.mu.Lock() + if info, ok := jc.segments[seq]; ok { + jc.mu.Unlock() + return info, nil + } + if jc.final { + jc.mu.Unlock() + return nil, errors.New("segment never produced") + } + ch := make(chan struct{}) + jc.waiters[seq] = append(jc.waiters[seq], ch) + jc.mu.Unlock() + select { + case <-ch: + jc.mu.RLock() + info := jc.segments[seq] + jc.mu.RUnlock() + if info == nil { + return nil, errors.New("segment not produced") + } + return info, nil + case <-time.After(timeout): + // 【内存泄漏修复】超时后必须从 waiters map 中移除自己的 channel， + // 否则 channel 会永远留在 map 中无法被 GC + jc.mu.Lock() + if chs, ok := jc.waiters[seq]; ok { + for i, w := range chs { + if w == ch { + jc.waiters[seq] = append(chs[:i], chs[i+1:]...) + break + } + } + if len(jc.waiters[seq]) == 0 { + delete(jc.waiters, seq) + } + } + jc.mu.Unlock() + return nil, errors.New("timeout waiting segment") + } +} + +// SegmentCount 返回 (jobID, profile) 当前已生成的切片数量。 +// 用于 TCPlaylist 在首次返回前等待至少若干切片就绪，避免 playlist +// 中只有 seg-0 导致 HLS.js 反复请求填充缓冲。 +func (c *Cache) SegmentCount(jobID, profile string) int { + c.mu.RLock() + pmap, ok := c.jobs[jobID] + c.mu.RUnlock() + if !ok { + return 0 + } + jc, ok := pmap[profile] + if !ok { + return 0 + } + jc.mu.RLock() + defer jc.mu.RUnlock() + return len(jc.segments) +} + +// BuildPlaylist 生成 HLS playlist (m3u8) +// 【重要】当 totalDuration > 0 时，会根据总时长预先生成完整切片列表（VOD 模式）， +// 即使切片还未生成，URL 也会出现在 playlist 中。HLS.js 请求未生成切片时 +// 会阻塞等待（见 TCSegment 的 WaitForSegment）。这样可以让 HLS.js 知道 +// 视频总时长，避免： +// 1. 时长显示错误（只显示已转码部分） +// 2. 拖动跳跃（playlist 增长导致切片序号重新对齐） +func (c *Cache) BuildPlaylist(jobID, profile string, segDur int, completed bool, baseURL string, totalDuration float64) string { + c.mu.RLock() + pmap, ok := c.jobs[jobID] + c.mu.RUnlock() + if !ok { + return "" + } + jc, ok := pmap[profile] + if !ok { + return "" + } + jc.mu.RLock() + defer jc.mu.RUnlock() + + var b strings.Builder + b.WriteString("#EXTM3U\n") + b.WriteString("#EXT-X-VERSION:3\n") + b.WriteString("#EXT-X-TARGETDURATION:" + strconv.Itoa(segDur) + "\n") + b.WriteString("#EXT-X-MEDIA-SEQUENCE:0\n") + + // 如果有总时长，按总时长生成完整切片列表 + if totalDuration > 0 { + // VOD 模式：完整列表，HLS.js 知道总时长 + b.WriteString("#EXT-X-PLAYLIST-TYPE:VOD\n") + fSegDur := float64(segDur) + // 总切片数 = ceil(总时长 / 每片时长) + totalSegs := int(totalDuration / fSegDur) + lastDur := totalDuration - float64(totalSegs)*fSegDur + if lastDur > 0.01 { + totalSegs++ + } + for i := 0; i < totalSegs; i++ { + dur := fSegDur + if i == totalSegs-1 && lastDur > 0.01 { + dur = lastDur + } + // 优先使用真实时长（如果已知） + if seg, ok := jc.segments[i]; ok && seg.Duration > 0 { + dur = seg.Duration + } + b.WriteString(fmt.Sprintf("#EXTINF:%.3f,\n", dur)) + b.WriteString(fmt.Sprintf("%sseg-%d.ts\n", baseURL, i)) + } + b.WriteString("#EXT-X-ENDLIST\n") + return b.String() + } + + // fallback: 没有总时长时退回到原 EVENT 模式（保留旧行为） + seqs := make([]int, 0, len(jc.segments)) + for s := range jc.segments { + seqs = append(seqs, s) + } + sort.Ints(seqs) + if jc.final || completed { + b.WriteString("#EXT-X-PLAYLIST-TYPE:VOD\n") + } else { + b.WriteString("#EXT-X-PLAYLIST-TYPE:EVENT\n") + } + for _, s := range seqs { + seg := jc.segments[s] + b.WriteString(fmt.Sprintf("#EXTINF:%.3f,\n", seg.Duration)) + b.WriteString(fmt.Sprintf("%sseg-%d.ts\n", baseURL, s)) + } + if jc.final || completed { + b.WriteString("#EXT-X-ENDLIST\n") + } + return b.String() +} + +// MasterPlaylist 多档位主 m3u8 +func (c *Cache) MasterPlaylist(profiles []Profile, baseURL string) string { + var b strings.Builder + b.WriteString("#EXTM3U\n#EXT-X-VERSION:3\n") + for _, p := range profiles { + bw, _ := parseBitrate(p.VideoBitrate) + ab, _ := parseBitrate(p.AudioBitrate) + // 【关键修复】根据实际编码动态生成 codec 字符串 + // 错误的 codec 声明会导致 HLS.js 用错误参数初始化 SourceBuffer + // 然后 appendBuffer 时数据格式不匹配 → bufferAppendError 死循环 + codecs := buildCodecString(p) + b.WriteString(fmt.Sprintf("#EXT-X-STREAM-INF:BANDWIDTH=%d,CODECS=\"%s\"\n", bw+ab, codecs)) + b.WriteString(fmt.Sprintf("%s%s/playlist.m3u8\n", baseURL, p.Name)) + } + return b.String() +} + +// buildCodecString 根据 Profile 生成 HLS CODECS 属性字符串 +// avc1.42E01E = H.264 Baseline Profile Level 3.0（兼容性最好） +// avc1.4D401E = H.264 Main Profile Level 3.0 +// avc1.640028 = H.264 High Profile Level 4.0 +// mp4a.40.2 = AAC-LC（HE-AAC v1 用 mp4a.40.5，HE-AAC v2 用 mp4a.40.29） +func buildCodecString(p Profile) string { + // 视频部分：默认 H.264 Main Profile Level 4.0，平衡兼容性与质量 + video := "avc1.4D4028" + switch strings.ToLower(p.VideoCodec) { + case "hevc", "h265": + // HEVC Main Profile Level 4.1，部分 Safari/Edge 支持 + video = "hvc1.1.6.L120.90" + case "av1": + video = "av01.0.05M.08" + } + // 音频部分：固定 AAC-LC（后端已强制转 AAC） + audio := "mp4a.40.2" + return video + "," + audio +} + +func parseBitrate(s string) (int64, error) { + s = strings.TrimSpace(strings.ToLower(s)) + if s == "" { + return 0, nil + } + mult := int64(1) + if strings.HasSuffix(s, "k") { + mult = 1000 + s = strings.TrimSuffix(s, "k") + } else if strings.HasSuffix(s, "m") { + mult = 1_000_000 + s = strings.TrimSuffix(s, "m") + } + v, err := strconv.ParseInt(s, 10, 64) + if err != nil { + return 0, err + } + return v * mult, nil +} + +// Cleanup 删除某个 job 的所有切片，并关闭所有阻塞的 waiter goroutine +func (c *Cache) Cleanup(jobID string) { + c.mu.Lock() + pmap, exists := c.jobs[jobID] + delete(c.jobs, jobID) + c.mu.Unlock() + // 【内存泄漏修复】关闭所有 profile 下阻塞等待的 waiter channel， + // 避免 goroutine 永远挂起（每个 goroutine 至少 8KB 栈空间） + if exists { + for _, jc := range pmap { + jc.mu.Lock() + jc.final = true // 标记为 final，防止新 waiter 进入 + for seq, chs := range jc.waiters { + for _, ch := range chs { + close(ch) + } + delete(jc.waiters, seq) + } + jc.mu.Unlock() + } + } + _ = os.RemoveAll(filepath.Join(c.rootDir, jobID)) +} + +// EnforceLRU 简易 LRU 容量控制：超过 maxBytes 时按目录修改时间从早到晚删除 +func (c *Cache) EnforceLRU() error { + entries, err := os.ReadDir(c.rootDir) + if err != nil { + return err + } + type item struct { + dir string + size int64 + mtime time.Time + } + var items []item + var total int64 + for _, e := range entries { + // 只管理本模块创建的 job 目录，避免误删缓存根目录下的其它内容 + if !e.IsDir() || !strings.HasPrefix(e.Name(), "job_") { + continue + } + full := filepath.Join(c.rootDir, e.Name()) + size := dirSize(full) + fi, _ := os.Stat(full) + mt := time.Now() + if fi != nil { + mt = fi.ModTime() + } + items = append(items, item{full, size, mt}) + total += size + } + if total <= c.maxBytes { + return nil + } + sort.Slice(items, func(i, j int) bool { return items[i].mtime.Before(items[j].mtime) }) + for _, it := range items { + if total <= c.maxBytes { + break + } + _ = os.RemoveAll(it.dir) + total -= it.size + // 同步从内存里删 + jobID := filepath.Base(it.dir) + c.mu.Lock() + delete(c.jobs, jobID) + c.mu.Unlock() + } + return nil +} + +func dirSize(dir string) int64 { + var size int64 + _ = filepath.Walk(dir, func(_ string, info os.FileInfo, err error) error { + if err == nil && info != nil && !info.IsDir() { + size += info.Size() + } + return nil + }) + return size +} diff --git a/internal/transcode/cache_security_test.go b/internal/transcode/cache_security_test.go new file mode 100644 index 00000000..79db6d96 --- /dev/null +++ b/internal/transcode/cache_security_test.go @@ -0,0 +1,70 @@ +package transcode + +import ( + "bytes" + "errors" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestValidProfileName(t *testing.T) { + ok := []string{"source", "1080p", "720p", "a_b-c", "A1"} + for _, p := range ok { + if !ValidProfileName(p) { + t.Errorf("%q should be valid", p) + } + } + bad := []string{"", "..", "../x", "a/b", `a\b`, "a b", "a.b", strings.Repeat("a", 33), "%2e%2e"} + for _, p := range bad { + if ValidProfileName(p) { + t.Errorf("%q should be rejected", p) + } + } +} + +func TestPutSegmentRejectsPathTraversal(t *testing.T) { + root := filepath.Join(t.TempDir(), "cache") + c := NewCache(root, 1) + if err := c.Init(); err != nil { + t.Fatal(err) + } + _, err := c.PutSegment("job_x", "../../../outside", 0, 1, bytes.NewReader([]byte("x")), false) + if !errors.Is(err, ErrInvalidProfile) { + t.Fatalf("want ErrInvalidProfile, got %v", err) + } + // 目录之外不应出现任何文件 + parent := filepath.Dir(root) + entries, _ := os.ReadDir(parent) + for _, e := range entries { + if e.Name() != "cache" { + t.Fatalf("unexpected path created outside cache root: %s", e.Name()) + } + } + if _, err := c.PutSegment("../job", "source", 0, 1, bytes.NewReader([]byte("x")), false); !errors.Is(err, ErrInvalidJobID) { + t.Fatalf("want ErrInvalidJobID, got %v", err) + } +} + +func TestPutSegmentLimits(t *testing.T) { + c := NewCache(filepath.Join(t.TempDir(), "cache"), 1) + if err := c.Init(); err != nil { + t.Fatal(err) + } + if _, err := c.PutSegment("job_x", "source", -1, 1, bytes.NewReader([]byte("x")), false); !errors.Is(err, ErrInvalidSegment) { + t.Fatalf("negative seq: want ErrInvalidSegment, got %v", err) + } + big := bytes.NewReader(make([]byte, MaxSegmentBytes+10)) + if _, err := c.PutSegment("job_x", "source", 0, 1, big, false); !errors.Is(err, ErrSegmentTooLarge) { + t.Fatalf("oversize: want ErrSegmentTooLarge, got %v", err) + } + // 超限后不应残留 tmp 文件 + matches, _ := filepath.Glob(filepath.Join(c.rootDir, "job_x", "source", "*")) + if len(matches) != 0 { + t.Fatalf("leftover files after rejected upload: %v", matches) + } + if _, err := c.PutSegment("job_x", "source", 0, 1, bytes.NewReader([]byte("ok")), false); err != nil { + t.Fatalf("normal upload failed: %v", err) + } +} diff --git a/internal/transcode/chunk_manager.go b/internal/transcode/chunk_manager.go new file mode 100644 index 00000000..cc989ea1 --- /dev/null +++ b/internal/transcode/chunk_manager.go @@ -0,0 +1,520 @@ +// Package transcode chunk_manager.go：智能分段转码调度器 +// +// 设计目标： +// 1. 把视频按时间切成多个 chunk（默认每 60 秒一段），每个 chunk 独立用 ffmpeg 转码 +// 2. 用户拖动到任意位置时，按需启动对应 chunk 的 ffmpeg，不再线性等待 +// 3. 控制单 Job 最多并发 N 个 chunk ffmpeg，超出时 LRU 淘汰最久未访问的 chunk +// 4. 当前 chunk 接近末尾时自动预启动下一个 chunk，保证顺序播放无缝衔接 +// 5. chunk 空闲超过阈值时 kill ffmpeg 释放 CPU，但保留已转出的切片文件 +// +// 与 LocalRunner 的关系： +// - LocalRunner.runJob 不再直接启动一个完整的 ffmpeg，而是创建 ChunkSession 并阻塞等待 +// - ChunkManager 是 ChunkSession 的"大脑"，负责调度 +// - TCSegment 收到切片请求时，调用 ChunkManager.EnsureChunkRunningForSeg(seq) 触发按需调度 +package transcode + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "sync" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" +) + +// ChunkState chunk 当前状态 +type ChunkState int + +const ( + ChunkIdle ChunkState = iota // 未启动 + ChunkRunning // ffmpeg 正在转码 + ChunkDone // ffmpeg 自然退出（chunk 内所有切片已转完） + ChunkFailed // ffmpeg 异常失败 + ChunkKilled // 被 LRU 淘汰或空闲超时 kill +) + +// Chunk 表示视频中的一个时间段，对应一段连续的切片序号 +type Chunk struct { + Index int // chunk 序号（0 开始） + StartTime float64 // 起始秒 + Duration float64 // 持续时长（最后一个 chunk 可能不足 chunkDur） + StartSeq int // 起始切片序号（包含） + EndSeq int // 结束切片序号（不包含） + State ChunkState + StartedAt time.Time + LastAccess time.Time + Error string + + cancel context.CancelFunc // ffmpeg 进程的 cancel + doneCh chan struct{} // ffmpeg 进程结束时关闭 + doneOnce sync.Once +} + +// chunkDoneClose 关闭 doneCh（幂等） +func (c *Chunk) chunkDoneClose() { + c.doneOnce.Do(func() { + if c.doneCh != nil { + close(c.doneCh) + } + }) +} + +// ChunkSession 单个 (jobID, profile) 下的 chunk 调度上下文 +type ChunkSession struct { + JobID string + Profile string + Job *Job + ProfDef Profile + + // 静态参数 + TotalDuration float64 // 视频总时长 + SegDur int // 每片时长（秒） + ChunkDur int // 每个 chunk 时长（秒） + MaxConcurrent int // 最大并发 chunk 数 + IdleSec int // chunk 空闲多久就 kill + Prefetch bool // 是否启用预读 + + mu sync.Mutex + chunks map[int]*Chunk // chunkIndex -> Chunk + + // 给 LocalRunner 用：所有切片转完时关闭，让 runJob 能退出 + allDoneCh chan struct{} + allDoneOnce sync.Once + + // outDir / segPattern (供 ffmpeg 使用) + OutDir string +} + +// allDoneClose 关闭 allDoneCh（幂等） +func (s *ChunkSession) allDoneClose() { + s.allDoneOnce.Do(func() { + close(s.allDoneCh) + }) +} + +// ChunkManager 全局 chunk 管理器（一个 Manager 一个） +type ChunkManager struct { + mgr *Manager + + mu sync.Mutex + sessions map[string]*ChunkSession // key = jobID + ":" + profile + + stopCh chan struct{} +} + +// NewChunkManager 创建实例 +func NewChunkManager(mgr *Manager) *ChunkManager { + return &ChunkManager{ + mgr: mgr, + sessions: make(map[string]*ChunkSession), + stopCh: make(chan struct{}), + } +} + +func sessionKey(jobID, profile string) string { return jobID + ":" + profile } + +// StartSession 创建并启动一个 chunk session（由 LocalRunner.runJob 调用） +// 返回的 ChunkSession.allDoneCh 在所有切片都转完时关闭 +func (cm *ChunkManager) StartSession(job *Job, profile Profile, outDir string) (*ChunkSession, error) { + totalDur := job.Probe.Duration + if totalDur <= 0 { + return nil, fmt.Errorf("chunk session requires probe.duration > 0") + } + segDur := setting.GetInt(conf.TranscodeSegmentDuration, 6) + if segDur <= 0 { + segDur = 6 + } + chunkDur := setting.GetInt(conf.TranscodeChunkDurationSec, 60) + if chunkDur < segDur { + chunkDur = segDur * 10 + } + // 让 chunkDur 是 segDur 的整数倍，保证切片对齐 + if chunkDur%segDur != 0 { + chunkDur = (chunkDur / segDur) * segDur + if chunkDur < segDur { + chunkDur = segDur + } + } + maxConc := setting.GetInt(conf.TranscodeMaxChunkConcurrency, 2) + if maxConc <= 0 { + maxConc = 2 + } + idleSec := setting.GetInt(conf.TranscodeChunkIdleSec, 60) + if idleSec <= 0 { + idleSec = 60 + } + prefetch := setting.GetBool(conf.TranscodeChunkPrefetch) + + s := &ChunkSession{ + JobID: job.ID, + Profile: profile.Name, + Job: job, + ProfDef: profile, + TotalDuration: totalDur, + SegDur: segDur, + ChunkDur: chunkDur, + MaxConcurrent: maxConc, + IdleSec: idleSec, + Prefetch: prefetch, + chunks: make(map[int]*Chunk), + allDoneCh: make(chan struct{}), + OutDir: outDir, + } + + // 预先按总时长生成所有 chunk 元数据（不启动） + segsPerChunk := chunkDur / segDur + totalSegs := int(totalDur) / segDur + if int(totalDur)%segDur > 0 { + totalSegs++ + } + chunkCount := totalSegs / segsPerChunk + if totalSegs%segsPerChunk > 0 { + chunkCount++ + } + for i := 0; i < chunkCount; i++ { + startSeq := i * segsPerChunk + endSeq := startSeq + segsPerChunk + if endSeq > totalSegs { + endSeq = totalSegs + } + startT := float64(startSeq * segDur) + dur := float64((endSeq - startSeq) * segDur) + // 最后一个 chunk 可能不足 + if i == chunkCount-1 { + realDur := totalDur - startT + if realDur > 0 && realDur < dur { + dur = realDur + } + } + s.chunks[i] = &Chunk{ + Index: i, + StartTime: startT, + Duration: dur, + StartSeq: startSeq, + EndSeq: endSeq, + State: ChunkIdle, + } + } + + cm.mu.Lock() + cm.sessions[sessionKey(job.ID, profile.Name)] = s + cm.mu.Unlock() + + // 立刻启动第一个 chunk（保证 playlist 首切片可用） + cm.ensureChunk(s, 0) + + // 启动后台维护协程：空闲清理 + 完成检测 + go cm.sessionLoop(s) + + utils.Log.Infof("[transcode][chunk] session started job=%s profile=%s totalDur=%.1fs chunkDur=%ds chunks=%d maxConc=%d", + job.ID, profile.Name, totalDur, chunkDur, chunkCount, maxConc) + return s, nil +} + +// EnsureChunkRunningForSeg 确保覆盖 seq 的 chunk 正在转码（由 TCSegment 触发） +// 返回所属 chunk index；如果 seq 越界返回 -1 +func (cm *ChunkManager) EnsureChunkRunningForSeg(jobID, profile string, seq int) int { + cm.mu.Lock() + s, ok := cm.sessions[sessionKey(jobID, profile)] + cm.mu.Unlock() + if !ok { + utils.Log.Warnf("[transcode][chunk] EnsureChunk: session not found job=%s profile=%s seq=%d", jobID, profile, seq) + return -1 + } + idx := cm.findChunkBySeq(s, seq) + if idx < 0 { + utils.Log.Warnf("[transcode][chunk] EnsureChunk: seq=%d out of range job=%s", seq, jobID) + return -1 + } + cm.ensureChunk(s, idx) + return idx +} + +// findChunkBySeq 按切片序号找到所属 chunk +func (cm *ChunkManager) findChunkBySeq(s *ChunkSession, seq int) int { + s.mu.Lock() + defer s.mu.Unlock() + for idx, c := range s.chunks { + if seq >= c.StartSeq && seq < c.EndSeq { + return idx + } + } + return -1 +} + +// ensureChunk 确保某个 chunk 处于 Running 状态（已运行/已完成则跳过） +// 同时受 MaxConcurrent 限制，超出时 LRU 淘汰最久未访问的 chunk +func (cm *ChunkManager) ensureChunk(s *ChunkSession, idx int) { + s.mu.Lock() + c, ok := s.chunks[idx] + if !ok { + s.mu.Unlock() + utils.Log.Warnf("[transcode][chunk] ensureChunk: chunk %d not in session (job=%s)", idx, s.JobID) + return + } + now := time.Now() + c.LastAccess = now + prevState := c.State + switch c.State { + case ChunkRunning: + s.mu.Unlock() + utils.Log.Debugf("[transcode][chunk] ensureChunk: chunk %d already running (job=%s)", idx, s.JobID) + return + case ChunkDone: + s.mu.Unlock() + utils.Log.Debugf("[transcode][chunk] ensureChunk: chunk %d already done (job=%s)", idx, s.JobID) + return + } + // 【关键修复】ChunkFailed 也允许重启：因为可能是网络抖动、source URL 临时失效等 + // 而不是永久性故障；如果连续失败，runChunk 内部不会陷入死循环（每次都会重新尝试源） + // ChunkIdle / ChunkKilled / ChunkFailed → 需要启动 + running := 0 + var lruIdx = -1 + var lruTime = now + for i, ck := range s.chunks { + if ck.State == ChunkRunning { + running++ + if i != idx && ck.LastAccess.Before(lruTime) { + lruTime = ck.LastAccess + lruIdx = i + } + } + } + if running >= s.MaxConcurrent && lruIdx >= 0 { + victim := s.chunks[lruIdx] + utils.Log.Infof("[transcode][chunk] LRU evict chunk %d (job=%s) to make room for chunk %d", + lruIdx, s.JobID, idx) + if victim.cancel != nil { + victim.cancel() + } + victim.State = ChunkKilled + } + // 启动 chunk ffmpeg + c.State = ChunkRunning + c.StartedAt = now + c.doneCh = make(chan struct{}) + c.doneOnce = sync.Once{} + s.mu.Unlock() + + utils.Log.Infof("[transcode][chunk] ensureChunk: launching chunk %d (job=%s prevState=%v running=%d)", + idx, s.JobID, prevState, running) + go cm.runChunk(s, idx) +} + +// runChunk 实际启动 ffmpeg 转码该 chunk +func (cm *ChunkManager) runChunk(s *ChunkSession, idx int) { + s.mu.Lock() + c := s.chunks[idx] + s.mu.Unlock() + if c == nil { + return + } + + timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120) + ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMin)*time.Minute) + s.mu.Lock() + c.cancel = cancel + s.mu.Unlock() + defer cancel() + + segPattern := filepath.Join(s.OutDir, "seg-%d.ts") + // 每个 chunk 写自己的 playlist 文件，防止互相覆盖；后端不读这个 playlist 用于响应， + // 真正的总 playlist 由 cache.BuildPlaylist 根据总时长生成。 + chunkPlaylist := filepath.Join(s.OutDir, fmt.Sprintf("_chunk_%d.m3u8", idx)) + + args := buildFFmpegChunkArgs(s.Job, s.ProfDef, s.SegDur, c.StartTime, c.Duration, c.StartSeq, segPattern, chunkPlaylist) + ffmpegPath := setting.GetStr(conf.TranscodeFFmpegPath, "ffmpeg") + cmd := exec.CommandContext(ctx, ffmpegPath, args...) + stderr := &limitedBuffer{maxSize: 32 * 1024} + cmd.Stderr = stderr + + utils.Log.Infof("[transcode][chunk] start ffmpeg job=%s chunk=%d ss=%.1f t=%.1f startSeq=%d", + s.JobID, idx, c.StartTime, c.Duration, c.StartSeq) + + startedAt := time.Now() + err := cmd.Run() + elapsed := time.Since(startedAt) + + s.mu.Lock() + switch { + case ctx.Err() != nil && c.State == ChunkKilled: + // 被 LRU 主动 kill，状态保持 Killed + utils.Log.Infof("[transcode][chunk] chunk %d killed (job=%s elapsed=%.1fs)", idx, s.JobID, elapsed.Seconds()) + case err != nil: + c.State = ChunkFailed + c.Error = truncate(stderr.String(), 512) + utils.Log.Errorf("[transcode][chunk] chunk %d failed (job=%s elapsed=%.1fs): %v stderr=%s", + idx, s.JobID, elapsed.Seconds(), err, c.Error) + default: + c.State = ChunkDone + utils.Log.Infof("[transcode][chunk] chunk %d done (job=%s elapsed=%.1fs)", idx, s.JobID, elapsed.Seconds()) + } + c.chunkDoneClose() + s.mu.Unlock() + + // 删除临时 playlist 文件 + _ = os.Remove(chunkPlaylist) + + // 【关键修复】chunk 自然完成时，必须用 finalScan-style 扫描，否则边界切片（最大 seq） + // 永远不会被登记到 cache，导致下次播放/拖动到该位置时 60 秒超时 + if cm.mgr != nil && cm.mgr.localRunner != nil { + cm.mgr.localRunner.publishChunkSegments(s.Job, s.ProfDef, s.OutDir, c.StartSeq, c.EndSeq) + } +} + +// sessionLoop 每个 session 的后台维护：空闲清理 + 全部完成检测 + 预读 +func (cm *ChunkManager) sessionLoop(s *ChunkSession) { + t := time.NewTicker(5 * time.Second) + defer t.Stop() + for { + select { + case <-cm.stopCh: + cm.killAllChunks(s) + return + case <-s.allDoneCh: + return + case <-t.C: + cm.maintainSession(s) + } + } +} + +// maintainSession 一次维护：检查空闲、预读下一 chunk、判断是否全部完成 +func (cm *ChunkManager) maintainSession(s *ChunkSession) { + now := time.Now() + idleThreshold := time.Duration(s.IdleSec) * time.Second + + s.mu.Lock() + // 1. 空闲清理：Running 状态但长时间未访问 → kill + for _, c := range s.chunks { + if c.State == ChunkRunning && now.Sub(c.LastAccess) > idleThreshold { + utils.Log.Infof("[transcode][chunk] chunk %d idle for %.0fs, killing (job=%s)", + c.Index, now.Sub(c.LastAccess).Seconds(), s.JobID) + if c.cancel != nil { + c.cancel() + } + c.State = ChunkKilled + } + } + // 2. 预读：找出最近被访问的 Running chunk，如果它的进度过半，启动下一个 chunk + var prefetchTarget = -1 + if s.Prefetch { + var hottestIdx = -1 + var hottestAccess time.Time + for i, c := range s.chunks { + if c.State == ChunkRunning && c.LastAccess.After(hottestAccess) { + hottestAccess = c.LastAccess + hottestIdx = i + } + } + if hottestIdx >= 0 { + hot := s.chunks[hottestIdx] + // 估算进度：用 ffmpeg 已经写出的切片数 / 该 chunk 总切片数 + // 由于 cache.segments 是全局的，我们检查范围 [StartSeq, EndSeq) 中已就绪数量 + done := cm.countReadySegments(s, hot.StartSeq, hot.EndSeq) + expected := hot.EndSeq - hot.StartSeq + if expected > 0 && done*2 >= expected { // 进度 >= 50% + next := hottestIdx + 1 + if nc, ok := s.chunks[next]; ok && nc.State == ChunkIdle { + prefetchTarget = next + } + } + } + } + // 3. 全部完成检测：所有 chunk 都 Done + allDone := true + for _, c := range s.chunks { + if c.State != ChunkDone { + allDone = false + break + } + } + s.mu.Unlock() + + if prefetchTarget >= 0 { + utils.Log.Infof("[transcode][chunk] prefetch chunk %d (job=%s)", prefetchTarget, s.JobID) + cm.ensureChunk(s, prefetchTarget) + } + + if allDone { + s.allDoneClose() + } +} + +// countReadySegments 统计 [start, end) 范围内已就绪的切片数 +func (cm *ChunkManager) countReadySegments(s *ChunkSession, start, end int) int { + if cm.mgr == nil || cm.mgr.Cache == nil { + return 0 + } + cnt := 0 + for seq := start; seq < end; seq++ { + if _, err := cm.mgr.Cache.GetSegment(s.JobID, s.Profile, seq); err == nil { + cnt++ + } + } + return cnt +} + +// killAllChunks 关闭某 session 下所有运行中的 ffmpeg +func (cm *ChunkManager) killAllChunks(s *ChunkSession) { + s.mu.Lock() + defer s.mu.Unlock() + for _, c := range s.chunks { + if c.State == ChunkRunning && c.cancel != nil { + c.cancel() + c.State = ChunkKilled + } + } +} + +// CancelSession 外部取消（job 被 cancel 或 idle 超时） +func (cm *ChunkManager) CancelSession(jobID, profile string) { + cm.mu.Lock() + s, ok := cm.sessions[sessionKey(jobID, profile)] + if ok { + delete(cm.sessions, sessionKey(jobID, profile)) + } + cm.mu.Unlock() + if !ok { + return + } + cm.killAllChunks(s) + s.allDoneClose() +} + +// CancelAllForJob 取消该 job 下所有 profile 的 session（CancelJob 时调用） +func (cm *ChunkManager) CancelAllForJob(jobID string) { + cm.mu.Lock() + var targets []*ChunkSession + for k, s := range cm.sessions { + if s.JobID == jobID { + targets = append(targets, s) + delete(cm.sessions, k) + } + } + cm.mu.Unlock() + for _, s := range targets { + cm.killAllChunks(s) + s.allDoneClose() + } +} + +// WaitDone 阻塞等待所有 chunk 完成（LocalRunner.runJob 用） +func (s *ChunkSession) WaitDone() <-chan struct{} { return s.allDoneCh } + +// Stop 停止 ChunkManager（Manager.Stop 调用） +func (cm *ChunkManager) Stop() { + close(cm.stopCh) + cm.mu.Lock() + defer cm.mu.Unlock() + for _, s := range cm.sessions { + cm.killAllChunks(s) + s.allDoneClose() + } + cm.sessions = nil +} diff --git a/internal/transcode/decide.go b/internal/transcode/decide.go new file mode 100644 index 00000000..456a32f4 --- /dev/null +++ b/internal/transcode/decide.go @@ -0,0 +1,131 @@ +package transcode + +import ( + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" +) + +// DecideRequest 决策器输入 +type DecideRequest struct { + FileName string + FileSize int64 // 字节 + Probe SourceProbe // 可选；为空则只用大小+后缀判断 +} + +// DecideResult 决策结果 +type DecideResult struct { + NeedTranscode bool + Reason string +} + +// Decide 根据全局设置判断是否需要走转码 +func Decide(r DecideRequest) DecideResult { + if !setting.GetBool(conf.TranscodeEnabled) { + return DecideResult{false, "transcode disabled"} + } + // 1. 大小阈值 + minGB := setting.GetInt(conf.TranscodeMinSizeGB, 5) + if minGB > 0 { + minBytes := int64(minGB) * 1024 * 1024 * 1024 + if r.FileSize > 0 && r.FileSize < minBytes { + return DecideResult{false, "size below threshold"} + } + } + // 2. 后缀过滤 + exts := splitCSV(setting.GetStr(conf.TranscodeSourceExtensions)) + if len(exts) > 0 { + ext := strings.ToLower(strings.TrimPrefix(getExt(r.FileName), ".")) + if !containsFold(exts, ext) { + return DecideResult{false, "extension not in list"} + } + } + // 3. 编码过滤（仅当 probe 提供时） + codecs := splitCSV(setting.GetStr(conf.TranscodeSourceCodecs)) + if r.Probe.VideoCodec != "" && len(codecs) > 0 { + if !containsFold(codecs, r.Probe.VideoCodec) { + return DecideResult{false, "codec not in list"} + } + } + // 4. 码率过滤（仅当 probe 提供时） + minBR := setting.GetInt(conf.TranscodeMinBitrateMbps, 0) + if minBR > 0 && r.Probe.VideoBitrate > 0 { + if r.Probe.VideoBitrate < int64(minBR)*1_000_000 { + return DecideResult{false, "bitrate below threshold"} + } + } + return DecideResult{true, "ok"} +} + +func splitCSV(s string) []string { + var out []string + for _, p := range strings.Split(s, ",") { + p = strings.TrimSpace(p) + if p != "" { + out = append(out, p) + } + } + return out +} + +func containsFold(arr []string, v string) bool { + for _, a := range arr { + if strings.EqualFold(a, v) { + return true + } + } + return false +} + +func getExt(name string) string { + i := strings.LastIndex(name, ".") + if i < 0 { + return "" + } + return name[i:] +} + +// BuildProfile 根据当前全局设置构建默认 Profile +func BuildProfile() Profile { + res := setting.GetStr(conf.TranscodeOutputResolution) + scale := "" + name := "default" + switch res { + case "source": + scale = "" + name = "source" + case "3840x2160": + scale = "3840:-2" + name = "2160p" + case "2560x1440": + scale = "2560:-2" + name = "1440p" + case "1920x1080": + scale = "1920:-2" + name = "1080p" + case "1280x720": + scale = "1280:-2" + name = "720p" + case "854x480": + scale = "854:-2" + name = "480p" + } + return Profile{ + Name: name, + VideoCodec: setting.GetStr(conf.TranscodeOutputCodec), + VideoBitrate: setting.GetStr(conf.TranscodeOutputBitrate), + Scale: scale, + AudioCodec: setting.GetStr(conf.TranscodeOutputAudioCodec), + AudioBitrate: setting.GetStr(conf.TranscodeOutputAudioBitrate), + HWAccel: setting.GetStr(conf.TranscodeHWAccel), + } +} + +// BuildOutputSpec 根据当前全局设置构建输出规格 +func BuildOutputSpec() OutputSpec { + return OutputSpec{ + Format: OutputFormat(setting.GetStr(conf.TranscodeOutputFormat)), + SegmentDuration: setting.GetInt(conf.TranscodeSegmentDuration, 6), + } +} diff --git a/internal/transcode/local_runner.go b/internal/transcode/local_runner.go new file mode 100644 index 00000000..7768280a --- /dev/null +++ b/internal/transcode/local_runner.go @@ -0,0 +1,772 @@ +package transcode + +import ( + "context" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" +) + +// limitedBuffer 是一个有容量上限的 io.Writer，超出 maxSize 后静默丢弃。 +// 用于捕获 ffmpeg stderr，防止异常文件导致无限内存增长。 +type limitedBuffer struct { + data []byte + maxSize int +} + +func (b *limitedBuffer) Write(p []byte) (int, error) { + remain := b.maxSize - len(b.data) + if remain <= 0 { + return len(p), nil // 静默丢弃 + } + if len(p) > remain { + p = p[:remain] + } + b.data = append(b.data, p...) + return len(p), nil +} + +func (b *limitedBuffer) String() string { + return string(b.data) +} + +// LocalRunner 本地内置 worker，直接调用本机 ffmpeg +type LocalRunner struct { + mgr *Manager + workerID string + + stopCh chan struct{} + wg sync.WaitGroup + + // 运行中 job 的 cancel 管理：jobID -> cancelFunc + runMu sync.Mutex + runningJobs map[string]context.CancelFunc +} + +func NewLocalRunner(mgr *Manager) *LocalRunner { + return &LocalRunner{ + mgr: mgr, + stopCh: make(chan struct{}), + runningJobs: make(map[string]context.CancelFunc), + } +} + +// Start 注册本地 worker 并启动消费协程 +func (r *LocalRunner) Start() error { + concurrency := setting.GetInt(conf.TranscodeLocalConcurrency, 1) + if concurrency <= 0 { + concurrency = 1 + } + hwaccel := setting.GetStr(conf.TranscodeHWAccel, "none") + hostname, _ := os.Hostname() + w := r.mgr.Registry.Register(&RegisterRequest{ + Name: "local-" + hostname, + Version: "embedded", + Capacity: concurrency, + HWAccel: []string{hwaccel}, + CodecsDecode: []string{"h264", "hevc", "av1", "vp9", "mpeg4", "mpeg2video"}, + CodecsEncode: []string{"h264", "hevc"}, + MaxResolution: "3840x2160", + Tags: []string{"local", "embedded"}, + }, true) + r.workerID = w.ID + utils.Log.Infof("[transcode] local worker started: id=%s capacity=%d hwaccel=%s", w.ID, concurrency, hwaccel) + + for i := 0; i < concurrency; i++ { + r.wg.Add(1) + go r.loop() + } + return nil +} + +func (r *LocalRunner) Stop() { + close(r.stopCh) + // 取消所有运行中的 FFmpeg 进程 + r.runMu.Lock() + for id, cancel := range r.runningJobs { + utils.Log.Infof("[transcode] stopping ffmpeg for job %s (runner shutdown)", id) + cancel() + } + r.runMu.Unlock() + r.wg.Wait() + r.mgr.Registry.Unregister(r.workerID) +} + +// CancelJob 外部取消指定 job 的 FFmpeg 进程（由 Scheduler.Cancel / 空闲超时调用） +func (r *LocalRunner) CancelJob(jobID string) bool { + r.runMu.Lock() + cancel, ok := r.runningJobs[jobID] + r.runMu.Unlock() + if ok { + utils.Log.Infof("[transcode] killing ffmpeg for job %s (external cancel)", jobID) + cancel() + } + return ok +} + +// RunningJobIDs 返回当前正在运行的所有 job ID 列表 +func (r *LocalRunner) RunningJobIDs() []string { + r.runMu.Lock() + defer r.runMu.Unlock() + ids := make([]string, 0, len(r.runningJobs)) + for id := range r.runningJobs { + ids = append(ids, id) + } + return ids +} + +func (r *LocalRunner) loop() { + defer r.wg.Done() + for { + select { + case <-r.stopCh: + return + default: + } + jobs := r.mgr.Scheduler.Claim(r.workerID, 1, 5*time.Second) + if len(jobs) == 0 { + continue + } + for _, j := range jobs { + r.runJob(j) + } + } +} + +// runJob 执行一个转码任务（单 profile，HLS 输出，边转边写入 cache） +func (r *LocalRunner) runJob(job *Job) { + utils.Log.Infof("[transcode] local runner picked job %s (%s)", job.ID, job.Path) + if len(job.Profiles) == 0 { + _ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: "no profiles"}) + return + } + profile := job.Profiles[0] + timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120) + ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMin)*time.Minute) + defer cancel() + + // 注册到 runningJobs，支持外部取消 + r.runMu.Lock() + r.runningJobs[job.ID] = cancel + r.runMu.Unlock() + defer func() { + r.runMu.Lock() + delete(r.runningJobs, job.ID) + r.runMu.Unlock() + }() + + // 输出目录 + jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name) + if err != nil { + _ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()}) + return + } + outDir := jc.BaseDir + + // 【智能调度】如果探测到了视频总时长，启用 chunk 模式（按需分段并发转码） + // 否则回退到旧的"一次性整段"模式（兼容 ffprobe 失败的场景） + if job.Probe.Duration > 0 && r.mgr.Chunks != nil { + r.runJobByChunks(ctx, job, profile, outDir) + return + } + + // === 旧路径：一次性整段转码 === + r.runJobLegacy(ctx, job, profile, outDir) +} + +// runJobByChunks 智能 chunk 调度模式：按需启动 ffmpeg 进程，支持随机访问 +func (r *LocalRunner) runJobByChunks(ctx context.Context, job *Job, profile Profile, outDir string) { + startedAt := time.Now() + + // 启动 watchSegments：周期扫描目录把切片登记到 cache + stopWatcher := make(chan struct{}) + var watcherWG sync.WaitGroup + watcherWG.Add(1) + go func() { + defer watcherWG.Done() + r.watchSegments(job, profile, outDir, stopWatcher) + }() + + // 创建 chunk session（会立刻启动 chunk-0 + 后台维护协程） + session, err := r.mgr.Chunks.StartSession(job, profile, outDir) + if err != nil { + close(stopWatcher) + watcherWG.Wait() + _ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()}) + return + } + + // 阻塞等待全部 chunk 完成 / 任务被取消 + select { + case <-session.WaitDone(): + // 所有 chunk 都 Done，正常完成 + case <-ctx.Done(): + // 外部取消：通知 chunk session 杀掉所有 ffmpeg + r.mgr.Chunks.CancelSession(job.ID, profile.Name) + } + + close(stopWatcher) + watcherWG.Wait() + r.scanAndPublish(job, profile, outDir, true) + + if ctx.Err() != nil { + utils.Log.Infof("[transcode] job %s chunk session cancelled", job.ID) + _ = r.mgr.Scheduler.Finish(&FinishRequest{ + JobID: job.ID, + Status: JobCancelled, + Error: "cancelled: " + ctx.Err().Error(), + }) + return + } + utils.Log.Infof("[transcode] job %s completed in %.1fs (chunk mode)", job.ID, time.Since(startedAt).Seconds()) + _ = r.mgr.Scheduler.Finish(&FinishRequest{ + JobID: job.ID, + Status: JobFinished, + Stats: JobStats{Elapsed: time.Since(startedAt).Seconds()}, + }) +} + +// runJobLegacy 旧路径：一次性整段转码（不知道总时长时回退使用） +func (r *LocalRunner) runJobLegacy(ctx context.Context, job *Job, profile Profile, outDir string) { + playlistPath := filepath.Join(outDir, "playlist.m3u8") + segPattern := filepath.Join(outDir, "seg-%d.ts") + + args := buildFFmpegArgs(job, profile, segPattern, playlistPath) + ffmpegPath := setting.GetStr(conf.TranscodeFFmpegPath, "ffmpeg") + + cmd := exec.CommandContext(ctx, ffmpegPath, args...) + stderr := &limitedBuffer{maxSize: 64 * 1024} + cmd.Stderr = stderr + + startedAt := time.Now() + if err := cmd.Start(); err != nil { + _ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()}) + return + } + + stopWatcher := make(chan struct{}) + var watcherWG sync.WaitGroup + watcherWG.Add(1) + go func() { + defer watcherWG.Done() + r.watchSegments(job, profile, outDir, stopWatcher) + }() + + err := cmd.Wait() + close(stopWatcher) + watcherWG.Wait() + r.scanAndPublish(job, profile, outDir, true) + + if err != nil { + if ctx.Err() != nil { + utils.Log.Infof("[transcode] job %s ffmpeg killed (context cancelled)", job.ID) + _ = r.mgr.Scheduler.Finish(&FinishRequest{ + JobID: job.ID, + Status: JobCancelled, + Error: "cancelled: " + ctx.Err().Error(), + }) + } else { + _ = r.mgr.Scheduler.Finish(&FinishRequest{ + JobID: job.ID, + Status: JobFailed, + Error: fmt.Sprintf("ffmpeg failed: %v, stderr=%s", err, truncate(stderr.String(), 1024)), + }) + } + return + } + utils.Log.Infof("[transcode] job %s completed in %.1fs (legacy mode)", job.ID, time.Since(startedAt).Seconds()) + _ = r.mgr.Scheduler.Finish(&FinishRequest{ + JobID: job.ID, + Status: JobFinished, + Stats: JobStats{Elapsed: time.Since(startedAt).Seconds()}, + }) +} + +// watchSegments 周期性扫描输出目录，把新切片登记到 Cache +// 【内存优化】把扫描频率从 500ms 降到 2s，减少每次 os.ReadDir 和 parseFFmpegPlaylistDurations +// 触发的字符串/字节切片分配。chunk 模式下每个 chunk 完成时还会主动调用 publishChunkSegments， +// 所以这里只需要兜底登记中间切片，2s 频率完全够用。 +func (r *LocalRunner) watchSegments(job *Job, profile Profile, dir string, stop chan struct{}) { + t := time.NewTicker(2 * time.Second) + defer t.Stop() + for { + select { + case <-stop: + return + case <-t.C: + r.scanAndPublish(job, profile, dir, false) + } + } +} + +func (r *LocalRunner) scanAndPublish(job *Job, profile Profile, dir string, finalScan bool) { + jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name) + if err != nil { + return + } + entries, err := os.ReadDir(dir) + if err != nil { + return + } + type segFile struct { + seq int + path string + size int64 + } + var segs []segFile + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.Name() + if !strings.HasPrefix(name, "seg-") || !strings.HasSuffix(name, ".ts") { + continue + } + numStr := strings.TrimSuffix(strings.TrimPrefix(name, "seg-"), ".ts") + seq, err := strconv.Atoi(numStr) + if err != nil { + continue + } + fi, err := e.Info() + if err != nil { + continue + } + segs = append(segs, segFile{seq: seq, path: filepath.Join(dir, name), size: fi.Size()}) + } + sort.Slice(segs, func(i, j int) bool { return segs[i].seq < segs[j].seq }) + + // 【优化】解析 ffmpeg 写出的 playlist.m3u8 获取每个切片的真实时长 + // 避免硬编码 segDur 导致 HLS.js 显示总时长不准确 + realDurations := parseFFmpegPlaylistDurations(filepath.Join(dir, "playlist.m3u8")) + + // ffmpeg 在写入 .ts 时是边写边写,稳妥起见仅登记除最后一片以外的切片(最后一片可能未写完) + limit := len(segs) + if !finalScan && limit > 0 { + limit-- + } + defaultDur := float64(setting.GetInt(conf.TranscodeSegmentDuration, 6)) + for i := 0; i < limit; i++ { + s := segs[i] + jc.mu.RLock() + _, exists := jc.segments[s.seq] + jc.mu.RUnlock() + if exists { + continue + } + // 优先用 ffmpeg playlist 中的真实时长 + dur := defaultDur + if d, ok := realDurations[s.seq]; ok && d > 0 { + dur = d + } + // 直接登记，不复制（已经在 cache 目录里了） + jc.mu.Lock() + jc.segments[s.seq] = &SegmentInfo{Seq: s.seq, Duration: dur, Size: s.size, Path: s.path} + var toClose []chan struct{} + for sk, chs := range jc.waiters { + if sk <= s.seq { + toClose = append(toClose, chs...) + delete(jc.waiters, sk) + } + } + jc.mu.Unlock() + for _, ch := range toClose { + close(ch) + } + // 第一片就绪 → 通知 job ready + if s.seq == 0 { + job.MarkReady() + } + } + if finalScan { + jc.mu.Lock() + jc.final = true + jc.mu.Unlock() + } +} + +// publishChunkSegments 把 [startSeq, endSeq) 范围内已经写完的切片登记到 cache。 +// 与 scanAndPublish 不同，这个方法专门用于"某个 chunk 的 ffmpeg 已经退出"的场景， +// 此时该范围的所有 .ts 文件都是完整的，不能再丢掉最大序号的那片（避免边界切片永远丢失）。 +func (r *LocalRunner) publishChunkSegments(job *Job, profile Profile, dir string, startSeq, endSeq int) { + jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name) + if err != nil { + return + } + defaultDur := float64(setting.GetInt(conf.TranscodeSegmentDuration, 6)) + registered := 0 + for seq := startSeq; seq < endSeq; seq++ { + jc.mu.RLock() + _, exists := jc.segments[seq] + jc.mu.RUnlock() + if exists { + continue + } + fpath := filepath.Join(dir, fmt.Sprintf("seg-%d.ts", seq)) + fi, err := os.Stat(fpath) + if err != nil || fi.IsDir() || fi.Size() == 0 { + continue + } + jc.mu.Lock() + jc.segments[seq] = &SegmentInfo{Seq: seq, Duration: defaultDur, Size: fi.Size(), Path: fpath} + var toClose []chan struct{} + for sk, chs := range jc.waiters { + if sk <= seq { + toClose = append(toClose, chs...) + delete(jc.waiters, sk) + } + } + jc.mu.Unlock() + for _, ch := range toClose { + close(ch) + } + if seq == 0 { + job.MarkReady() + } + registered++ + } + if registered > 0 { + utils.Log.Infof("[transcode][chunk] published %d segments [%d, %d) for job=%s", + registered, startSeq, endSeq, job.ID) + } +} + +// playlistDurCache 缓存 m3u8 解析结果，按文件 mtime+size 失效。 +// 【内存优化】scanAndPublish 每 2 秒一次，每次都会解析同一个 m3u8（长视频几百 KB）， +// strings.Split 会产生大量临时字符串切片造成 GC 压力。用缓存减少 99% 的重复解析。 +type playlistDurCacheEntry struct { + mtime time.Time + size int64 + durs map[int]float64 +} + +var ( + playlistDurCache = make(map[string]*playlistDurCacheEntry) + playlistDurCacheMu sync.Mutex +) + +// parseFFmpegPlaylistDurations 解析 ffmpeg 写出的 m3u8，返回 seq -> duration 的映射 +// 用于获取每个切片的真实时长，避免后端用硬编码 segDur 导致总时长显示偏差 +func parseFFmpegPlaylistDurations(path string) map[int]float64 { + fi, err := os.Stat(path) + if err != nil { + return map[int]float64{} + } + // 命中缓存：mtime + size 都没变就直接复用 + playlistDurCacheMu.Lock() + if e, ok := playlistDurCache[path]; ok && e.mtime.Equal(fi.ModTime()) && e.size == fi.Size() { + playlistDurCacheMu.Unlock() + return e.durs + } + playlistDurCacheMu.Unlock() + + out := make(map[int]float64) + data, err := os.ReadFile(path) + if err != nil { + return out + } + lines := strings.Split(string(data), "\n") + var pendingDur float64 + for _, line := range lines { + line = strings.TrimSpace(line) + if strings.HasPrefix(line, "#EXTINF:") { + // #EXTINF:6.000000, + body := strings.TrimPrefix(line, "#EXTINF:") + body = strings.TrimSuffix(body, ",") + d, err := strconv.ParseFloat(strings.TrimSpace(body), 64) + if err == nil { + pendingDur = d + } + continue + } + if strings.HasSuffix(line, ".ts") && strings.HasPrefix(line, "seg-") { + numStr := strings.TrimSuffix(strings.TrimPrefix(line, "seg-"), ".ts") + if seq, err := strconv.Atoi(numStr); err == nil && pendingDur > 0 { + out[seq] = pendingDur + } + pendingDur = 0 + } + } + // 写入缓存 + playlistDurCacheMu.Lock() + playlistDurCache[path] = &playlistDurCacheEntry{ + mtime: fi.ModTime(), + size: fi.Size(), + durs: out, + } + playlistDurCacheMu.Unlock() + return out +} + +// buildFFmpegArgs 构造 ffmpeg 命令行（一次性整段转码，兼容旧路径） +// 当 Probe.Duration 不可用时，LocalRunner.runJob 会回退到这条路径。 +func buildFFmpegArgs(job *Job, profile Profile, segPattern, playlistPath string) []string { + segDur := setting.GetInt(conf.TranscodeSegmentDuration, 6) + // startTime=0, duration=0(=不限) 表示整段转码；startSeq=0 + return buildFFmpegArgsCommon(job, profile, segDur, 0, 0, 0, segPattern, playlistPath) +} + +// buildFFmpegChunkArgs 构造 chunk 模式的 ffmpeg 命令行 +// startTime: 起始时间（秒）；duration: 持续时长（秒，0=不限）；startSeq: 切片起始编号 +// chunk 模式与一次性整段共用底层参数构造，差异仅在 -ss/-t/-start_number。 +func buildFFmpegChunkArgs(job *Job, profile Profile, segDur int, startTime, duration float64, startSeq int, segPattern, playlistPath string) []string { + return buildFFmpegArgsCommon(job, profile, segDur, startTime, duration, startSeq, segPattern, playlistPath) +} + +// buildFFmpegArgsCommon 通用 ffmpeg 命令行构造 +// 当 startTime > 0 或 duration > 0 时，启用"chunk 模式"：使用 -ss / -t / -start_number 限定范围 +// 关键：-ss 必须放在 -i 之前（input seek，快），且需配合 force_key_frames 让 chunk 起始处生成 IDR 帧 +func buildFFmpegArgsCommon(job *Job, profile Profile, segDur int, startTime, duration float64, startSeq int, segPattern, playlistPath string) []string { + hw := strings.ToLower(profile.HWAccel) + args := []string{"-y", "-loglevel", "warning"} + + // 输入侧硬件加速 + switch hw { + case "nvenc": + args = append(args, "-hwaccel", "cuda") + case "qsv": + args = append(args, "-hwaccel", "qsv") + case "vaapi": + args = append(args, "-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", "/dev/dri/renderD128") + case "amf": + args = append(args, "-hwaccel", "d3d11va") + case "videotoolbox": + args = append(args, "-hwaccel", "videotoolbox") + case "auto": + args = append(args, "-hwaccel", "auto") + } + + // 【chunk 模式】input seek：放在 -i 之前，速度快，会按关键帧对齐 + // 注意 -ss 在 -i 之前是 input seek（用容器索引快速跳转），但落点是关键帧； + // 我们通过 force_key_frames 保证每个 chunk 边界都有关键帧，所以这里精度足够。 + if startTime > 0.001 { + args = append(args, "-ss", fmt.Sprintf("%.3f", startTime)) + } + + // 输入：先限定协议白名单（输入选项，必须在 -i 之前），再指定输入 + args = append(args, "-protocol_whitelist", ProtocolWhitelistFor(job.SourceURL)) + args = append(args, "-i", job.SourceURL) + + // 【chunk 模式】限定时长（output side 更精确） + if duration > 0.001 { + args = append(args, "-t", fmt.Sprintf("%.3f", duration)) + } + + // 视频编码器 + vcodec := pickEncoder(profile.VideoCodec, hw) + args = append(args, "-c:v", vcodec) + // 【关键】强制浏览器兼容的 profile/level/pix_fmt，必须与 master.m3u8 声明的 + // CODECS="avc1.640028" (H.264 High Profile Level 4.0) 一致，否则 HLS.js 会触发 + // bufferAppendError 死循环。 + if strings.Contains(vcodec, "264") { + args = append(args, "-profile:v", "high", "-level:v", "4.0", "-pix_fmt", "yuv420p") + } + if profile.VideoBitrate != "" { + args = append(args, "-b:v", profile.VideoBitrate) + args = append(args, "-maxrate", profile.VideoBitrate, "-bufsize", profile.VideoBitrate) + } + if profile.Scale != "" { + // 软件 / 硬件缩放 + switch hw { + case "vaapi": + args = append(args, "-vf", "scale_vaapi="+profile.Scale) + case "qsv": + args = append(args, "-vf", "scale_qsv="+profile.Scale) + default: + args = append(args, "-vf", "scale="+profile.Scale) + } + } + args = append(args, "-preset", defaultPresetFor(vcodec)) + + // 音频 + // 【关键修复】不能使用 -c:a copy，因为浏览器 MSE 不支持 AC3/DTS/EAC3/TrueHD 等 + // 常见于 mkv 的音频编码。直接 copy 会导致 SourceBuffer.appendBuffer 失败 + // 触发 bufferAppendError 死循环。统一强制转 AAC 兼容所有浏览器。 + ac := profile.AudioCodec + if ac == "" || ac == "copy" { + ac = "aac" + } + args = append(args, "-c:a", ac) + ab := profile.AudioBitrate + if ab == "" { + ab = "160k" + } + args = append(args, "-b:a", ab) + // 强制立体声 + 48kHz，避免 5.1 声道 / 高采样率导致的兼容问题 + args = append(args, "-ac", "2", "-ar", "48000") + + // HLS 输出 + if segDur <= 0 { + segDur = setting.GetInt(conf.TranscodeSegmentDuration, 6) + } + // 【关键】强制关键帧对齐切片边界，确保每个切片以 IDR 帧开始可独立解码。 + // 没有这些参数时，第一个切片可能不含 IDR 帧，导致 HLS.js bufferAppendError。 + // fps 假设 25 (帧率未知时的保守值)，gop = segDur * fps；force_key_frames 让 + // ffmpeg 在每个切片边界强制插入关键帧。 + gop := segDur * 25 + args = append(args, + "-g", strconv.Itoa(gop), + "-keyint_min", strconv.Itoa(gop), + "-sc_threshold", "0", + "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", segDur), + ) + // 【chunk 模式 - 关键修复】input seek 后 ffmpeg 会把第一帧 PTS 重置为 0， + // 这会导致 chunk-30 的切片在播放器看来是 PTS=0~60s（而不是 1800~1860s）， + // 进而触发 HLS.js bufferAppendError 或时间轴错位。 + // 解决方法：用 -output_ts_offset 让生成的切片时间戳偏移到原始位置。 + if startTime > 0.001 { + args = append(args, "-output_ts_offset", fmt.Sprintf("%.3f", startTime)) + } + args = append(args, + "-f", "hls", + "-hls_time", strconv.Itoa(segDur), + "-hls_playlist_type", "vod", + "-hls_segment_filename", segPattern, + "-hls_flags", "independent_segments+temp_file", + ) + // 【chunk 模式】指定起始切片编号，让全局切片编号连续 + if startSeq > 0 { + args = append(args, "-start_number", strconv.Itoa(startSeq)) + } + args = append(args, playlistPath) + return args +} + +// pickEncoder 根据视频编码 + HW 类型选择 ffmpeg 编码器 +func pickEncoder(codec, hw string) string { + codec = strings.ToLower(codec) + switch codec { + case "h264", "": + switch hw { + case "nvenc": + return "h264_nvenc" + case "qsv": + return "h264_qsv" + case "vaapi": + return "h264_vaapi" + case "amf": + return "h264_amf" + case "videotoolbox": + return "h264_videotoolbox" + default: + return "libx264" + } + case "hevc", "h265": + switch hw { + case "nvenc": + return "hevc_nvenc" + case "qsv": + return "hevc_qsv" + case "vaapi": + return "hevc_vaapi" + case "amf": + return "hevc_amf" + case "videotoolbox": + return "hevc_videotoolbox" + default: + return "libx265" + } + case "av1": + switch hw { + case "nvenc": + return "av1_nvenc" + case "qsv": + return "av1_qsv" + default: + return "libsvtav1" + } + } + return "libx264" +} + +func defaultPresetFor(encoder string) string { + if strings.Contains(encoder, "nvenc") { + return "p5" + } + if strings.Contains(encoder, "qsv") { + return "medium" + } + if strings.Contains(encoder, "amf") { + return "balanced" + } + if strings.Contains(encoder, "videotoolbox") { + return "medium" + } + return "veryfast" +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "...(truncated)" +} + +// ProbeSource 调用 ffprobe 探测源文件 +func ProbeSource(url string) (SourceProbe, error) { + if !ValidSourceInput(url) { + return SourceProbe{}, fmt.Errorf("unsupported probe source") + } + probePath := setting.GetStr(conf.TranscodeFFprobePath, "ffprobe") + args := []string{ + "-v", "error", + "-protocol_whitelist", ProtocolWhitelistFor(url), + "-show_entries", "format=duration,size,bit_rate:stream=codec_type,codec_name,bit_rate,width,height", + "-of", "json", + url, + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, probePath, args...) + if runtime.GOOS == "windows" { + // no special handling + } + out, err := cmd.Output() + if err != nil { + return SourceProbe{}, err + } + var raw struct { + Format struct { + Duration string `json:"duration"` + Size string `json:"size"` + BitRate string `json:"bit_rate"` + } `json:"format"` + Streams []struct { + CodecType string `json:"codec_type"` + CodecName string `json:"codec_name"` + BitRate string `json:"bit_rate"` + Width int `json:"width"` + Height int `json:"height"` + } `json:"streams"` + } + if err := json.Unmarshal(out, &raw); err != nil { + return SourceProbe{}, err + } + p := SourceProbe{} + p.Duration, _ = strconv.ParseFloat(raw.Format.Duration, 64) + p.Size, _ = strconv.ParseInt(raw.Format.Size, 10, 64) + for _, s := range raw.Streams { + if s.CodecType == "video" && p.VideoCodec == "" { + p.VideoCodec = s.CodecName + p.Width = s.Width + p.Height = s.Height + p.VideoBitrate, _ = strconv.ParseInt(s.BitRate, 10, 64) + } + if s.CodecType == "audio" && p.AudioCodec == "" { + p.AudioCodec = s.CodecName + p.AudioBitrate, _ = strconv.ParseInt(s.BitRate, 10, 64) + } + } + return p, nil +} diff --git a/internal/transcode/manager.go b/internal/transcode/manager.go new file mode 100644 index 00000000..9436c825 --- /dev/null +++ b/internal/transcode/manager.go @@ -0,0 +1,177 @@ +package transcode + +import ( + "sync" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" +) + +// Manager 全局转码管理器单例 +type Manager struct { + Registry *WorkerRegistry + Scheduler *Scheduler + Cache *Cache + Chunks *ChunkManager // 智能 chunk 调度器（按需启动 ffmpeg，支持随机访问） + + localRunner *LocalRunner + + stopCh chan struct{} + once sync.Once +} + +var ( + defaultManager *Manager + managerOnce sync.Once +) + +// Default 返回全局 Manager（懒加载） +func Default() *Manager { + managerOnce.Do(func() { + defaultManager = newManager() + }) + return defaultManager +} + +func newManager() *Manager { + reg := NewWorkerRegistry() + sch := NewScheduler(reg) + cachePath := setting.GetStr(conf.TranscodeCachePath, "data/ctmp") + // 兼容旧默认值：统一迁移到 data/ctmp + if cachePath == "data/transcode_cache" || cachePath == "./data/transcode_cache" { + cachePath = "data/ctmp" + } + cacheMaxGB := int64(setting.GetInt(conf.TranscodeCacheMaxGB, 20)) + cache := NewCache(cachePath, cacheMaxGB) + _ = cache.Init() + m := &Manager{ + Registry: reg, + Scheduler: sch, + Cache: cache, + stopCh: make(chan struct{}), + } + m.Chunks = NewChunkManager(m) + return m +} + +// Start 启动后台维护协程：心跳清理 + 缓存 LRU + 完成任务清理 +func (m *Manager) Start() { + m.once.Do(func() { + go m.maintenance() + // 根据 run_mode 决定是否启动本地 worker + mode := setting.GetStr(conf.TranscodeRunMode, "local") + if mode == "local" || mode == "hybrid" { + m.startLocalWorker() + } + }) +} + +// Stop 停止 Manager +func (m *Manager) Stop() { + close(m.stopCh) + if m.Chunks != nil { + m.Chunks.Stop() + } + if m.localRunner != nil { + m.localRunner.Stop() + } +} + +func (m *Manager) maintenance() { + t1 := time.NewTicker(15 * time.Second) // 心跳清理 + t2 := time.NewTicker(5 * time.Minute) // 缓存 LRU + t3 := time.NewTicker(30 * time.Minute) // 历史 job 清理 + t4 := time.NewTicker(10 * time.Second) // 空闲超时检测 + defer t1.Stop() + defer t2.Stop() + defer t3.Stop() + defer t4.Stop() + for { + select { + case <-m.stopCh: + return + case <-t1.C: + removed := m.Registry.Sweep() + for _, id := range removed { + utils.Log.Infof("[transcode] worker %s removed (heartbeat timeout)", id) + } + case <-t2.C: + _ = m.Cache.EnforceLRU() + case <-t3.C: + purged := m.Scheduler.PurgeBefore(time.Now().Add(-2 * time.Hour)) + if len(purged) > 0 { + // 【内存泄漏修复】联动清理 Cache 中对应的 JobCache + segments map + for _, jobID := range purged { + m.Cache.Cleanup(jobID) + } + utils.Log.Infof("[transcode] purged %d finished jobs (cache cleaned)", len(purged)) + } + case <-t4.C: + m.sweepIdleJobs() + } + } +} + +// sweepIdleJobs 扫描所有活跃任务，超过空闲超时时间无播放端请求则自动取消并 kill FFmpeg +// 注意：这里只 cancel job + kill ffmpeg 释放 CPU，**不删除已转好的切片缓存**。 +// 当用户继续播放时，TCSegment / TCPlaylist 会通过 Scheduler.Reactivate 复用同一 jobID/token +// 重新启动转码；已存在的切片可以直接命中，不需要再次转码。 +func (m *Manager) sweepIdleJobs() { + idleTimeoutSec := setting.GetInt(conf.TranscodeIdleTimeoutSec, 90) + if idleTimeoutSec <= 0 { + return // 配置为 0 表示禁用空闲超时 + } + timeout := time.Duration(idleTimeoutSec) * time.Second + now := time.Now() + + activeJobs := m.Scheduler.ListActiveJobs() + for _, j := range activeJobs { + lastAccess := j.GetLastAccess() + idle := now.Sub(lastAccess) + if idle > timeout { + utils.Log.Infof("[transcode] job %s idle for %.0fs (threshold %ds), pausing ffmpeg (cache kept for resume)", + j.ID, idle.Seconds(), idleTimeoutSec) + // 先取消所有 chunk ffmpeg，再调度器层面取消 + if m.Chunks != nil { + m.Chunks.CancelAllForJob(j.ID) + } + m.Scheduler.Cancel(j.ID) + // 【重要】不再调用 m.Cache.Cleanup —— 已转好的切片必须保留， + // 这样浏览器恢复播放时旧的 seg-N.ts 仍然能命中，避免 404 和重新转码。 + // 真正的磁盘清理由更长周期的 PurgeBefore + Cache.EnforceLRU 负责。 + } + } +} + +func (m *Manager) startLocalWorker() { + r := NewLocalRunner(m) + if err := r.Start(); err != nil { + utils.Log.Errorf("[transcode] start local worker failed: %+v", err) + return + } + m.localRunner = r + // 注入取消回调：Scheduler.Cancel 时直接 kill 本地 FFmpeg 进程 + m.Scheduler.SetOnCancel(func(jobID string) { + // 先清理 chunk session（kill 所有 chunk ffmpeg） + if m.Chunks != nil { + m.Chunks.CancelAllForJob(jobID) + } + if m.localRunner != nil { + m.localRunner.CancelJob(jobID) + } + }) +} + +// IsEnabled 总开关 +func IsEnabled() bool { return setting.GetBool(conf.TranscodeEnabled) } + +// VerifyWorkerSecret 校验远程 worker 注册凭据 +func VerifyWorkerSecret(token string) bool { + expected := setting.GetStr(conf.TranscodeWorkerSecret) + if expected == "" { + return false // 未配置则禁用远程注册 + } + return token == expected +} diff --git a/internal/transcode/protocol.go b/internal/transcode/protocol.go new file mode 100644 index 00000000..7c771882 --- /dev/null +++ b/internal/transcode/protocol.go @@ -0,0 +1,75 @@ +package transcode + +// 协议版本 +const ProtocolVersion = "v1" + +// ---- 注册 ---- + +type RegisterRequest struct { + Name string `json:"name"` + Version string `json:"version"` + Capacity int `json:"capacity"` + HWAccel []string `json:"hwaccel"` + CodecsDecode []string `json:"codecs_decode"` + CodecsEncode []string `json:"codecs_encode"` + MaxResolution string `json:"max_resolution"` + Tags []string `json:"tags"` + Endpoint string `json:"endpoint,omitempty"` +} + +type RegisterResponse struct { + WorkerID string `json:"worker_id"` + HeartbeatInterval int `json:"heartbeat_interval"` + ClaimStrategy string `json:"claim_strategy"` // pull + ProtocolVersion string `json:"protocol_version"` +} + +// ---- 心跳 ---- + +type HeartbeatRequest struct { + WorkerID string `json:"worker_id"` + Load float64 `json:"load"` + Running []string `json:"running"` + FreeSlots int `json:"free_slots"` +} + +type HeartbeatResponse struct { + OK bool `json:"ok"` + Kick bool `json:"kick"` // 服务端要求 worker 退出 + Cancel []string `json:"cancel,omitempty"` // 需要取消的 job ids +} + +// ---- 领任务 ---- + +type ClaimRequest struct { + WorkerID string `json:"worker_id"` + Slots int `json:"slots"` + Wait int `json:"wait"` // 长轮询超时秒，0=不等 +} + +type ClaimResponse struct { + Jobs []*Job `json:"jobs"` +} + +// ---- 完成 ---- + +type FinishRequest struct { + JobID string `json:"job_id"` + Status JobStatus `json:"status"` + Error string `json:"error,omitempty"` + Stats JobStats `json:"stats"` +} + +type JobStats struct { + Elapsed float64 `json:"elapsed"` + FPS float64 `json:"fps"` + Speed float64 `json:"speed"` + BytesOut int64 `json:"bytes_out"` +} + +// ---- 通用响应 ---- + +type ErrorResponse struct { + Code int `json:"code"` + Message string `json:"message"` +} diff --git a/internal/transcode/protocol_guard.go b/internal/transcode/protocol_guard.go new file mode 100644 index 00000000..339c9d4d --- /dev/null +++ b/internal/transcode/protocol_guard.go @@ -0,0 +1,56 @@ +package transcode + +import "strings" + +// ffmpeg / ffprobe 支持 concat:、subfile:、data: 等大量协议。 +// SourceURL 最终来自请求中的路径或 Host 头，若不限制协议，攻击者就可能借 ffmpeg +// 读取服务器本地文件或访问内网地址。这里按输入类型收敛到最小协议集。 + +const ( + // protocolWhitelistLocal 本地文件输入只允许 file 协议 + protocolWhitelistLocal = "file" + // protocolWhitelistRemote 远程输入允许 HTTP(S) 及其底层传输；禁止 concat/subfile/data/pipe， + // 同时去掉 file 与 crypto：HTTP 源用不到它们，留着等于给被构造的播放列表留了读本地文件的通道。 + protocolWhitelistRemote = "http,https,tcp,tls" +) + +// ProtocolWhitelistFor 根据输入地址返回应传给 ffmpeg/ffprobe 的 -protocol_whitelist 值。 +func ProtocolWhitelistFor(source string) string { + s := strings.ToLower(strings.TrimSpace(source)) + if strings.HasPrefix(s, "http://") || strings.HasPrefix(s, "https://") { + return protocolWhitelistRemote + } + return protocolWhitelistLocal +} + +// ValidSourceInput 判断输入是否可以安全交给 ffmpeg： +// 只接受绝对本地路径或 http(s) URL，拒绝 concat:/subfile:/data:/pipe: 等协议前缀， +// 也拒绝以 - 开头（会被当作 ffmpeg 选项）的值。 +func ValidSourceInput(source string) bool { + s := strings.TrimSpace(source) + if s == "" || strings.HasPrefix(s, "-") { + return false + } + low := strings.ToLower(s) + if strings.HasPrefix(low, "http://") || strings.HasPrefix(low, "https://") { + return true + } + // 本地输入只接受绝对路径。 + // 不能按「冒号位置」判 scheme：文件名里带冒号非常常见 + // （/mnt/media/Star Wars: Episode IV.mkv），那样会被误判成协议前缀。 + // concat:/subfile:/data: 这些前缀都不是绝对路径，且还会被 -protocol_whitelist 挡住。 + // 注意不能用 filepath.IsAbs：它是平台相关的，Windows 上会拒绝 "/data/a.mkv" + // 这类 VFS 绝对路径（OpenList 的路径始终是正斜杠开头）。 + return strings.HasPrefix(s, "/") || isWindowsDrivePath(s) +} + +// isWindowsDrivePath 判断 C:\ 或 C:/ 形式的盘符绝对路径。 +// 跨平台使用：在 Linux 上构建时会话也可能处理 Windows 风格路径。 +func isWindowsDrivePath(s string) bool { + if len(s) < 3 || s[1] != ':' { + return false + } + c := s[0] + isLetter := (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') + return isLetter && (s[2] == '\\' || s[2] == '/') +} diff --git a/internal/transcode/registry.go b/internal/transcode/registry.go new file mode 100644 index 00000000..2888af2e --- /dev/null +++ b/internal/transcode/registry.go @@ -0,0 +1,122 @@ +package transcode + +import ( + "sync" + "time" + + "github.com/google/uuid" +) + +// WorkerRegistry 维护当前在线的 Worker 池 +type WorkerRegistry struct { + mu sync.RWMutex + workers map[string]*Worker + + heartbeatTimeout time.Duration +} + +func NewWorkerRegistry() *WorkerRegistry { + return &WorkerRegistry{ + workers: make(map[string]*Worker), + heartbeatTimeout: 30 * time.Second, + } +} + +// Register 注册一个新的 worker，返回分配的 ID +func (r *WorkerRegistry) Register(req *RegisterRequest, local bool) *Worker { + w := &Worker{ + ID: "wk_" + uuid.NewString(), + Name: req.Name, + Version: req.Version, + Capacity: req.Capacity, + HWAccel: req.HWAccel, + CodecsDecode: req.CodecsDecode, + CodecsEncode: req.CodecsEncode, + MaxResolution: req.MaxResolution, + Tags: req.Tags, + Endpoint: req.Endpoint, + FreeSlots: req.Capacity, + LastSeen: time.Now(), + RegisteredAt: time.Now(), + Local: local, + } + if w.Capacity <= 0 { + w.Capacity = 1 + w.FreeSlots = 1 + } + r.mu.Lock() + r.workers[w.ID] = w + r.mu.Unlock() + return w +} + +// Heartbeat 更新 worker 心跳信息，未注册返回 false +func (r *WorkerRegistry) Heartbeat(req *HeartbeatRequest) (*Worker, bool) { + r.mu.Lock() + defer r.mu.Unlock() + w, ok := r.workers[req.WorkerID] + if !ok { + return nil, false + } + w.LastSeen = time.Now() + w.Load = req.Load + w.Running = req.Running + w.FreeSlots = req.FreeSlots + return w, true +} + +// Get 获取 worker +func (r *WorkerRegistry) Get(id string) (*Worker, bool) { + r.mu.RLock() + defer r.mu.RUnlock() + w, ok := r.workers[id] + return w, ok +} + +// Unregister 主动移除 worker +func (r *WorkerRegistry) Unregister(id string) { + r.mu.Lock() + delete(r.workers, id) + r.mu.Unlock() +} + +// List 列出所有 worker（拷贝） +func (r *WorkerRegistry) List() []*Worker { + r.mu.RLock() + defer r.mu.RUnlock() + out := make([]*Worker, 0, len(r.workers)) + for _, w := range r.workers { + out = append(out, w) + } + return out +} + +// Sweep 清理超时未心跳的 worker；返回被移除的 worker_id 列表 +func (r *WorkerRegistry) Sweep() []string { + r.mu.Lock() + defer r.mu.Unlock() + now := time.Now() + var removed []string + for id, w := range r.workers { + if w.Local { + continue // 本地 Worker 不通过心跳判活 + } + if now.Sub(w.LastSeen) > r.heartbeatTimeout { + removed = append(removed, id) + delete(r.workers, id) + } + } + return removed +} + +// HasAvailable 是否存在有空闲槽位的 worker +func (r *WorkerRegistry) HasAvailable() bool { + r.mu.RLock() + defer r.mu.RUnlock() + for _, w := range r.workers { + if w.FreeSlots > 0 { + return true + } + } + return false +} diff --git a/internal/transcode/scheduler.go b/internal/transcode/scheduler.go new file mode 100644 index 00000000..6c14054f --- /dev/null +++ b/internal/transcode/scheduler.go @@ -0,0 +1,460 @@ +package transcode + +import ( + "errors" + "sort" + "strings" + "sync" + "time" + + "github.com/google/uuid" +) + +// Scheduler 任务调度器：维护 pending 队列并支持 worker pull 领取 +type Scheduler struct { + mu sync.Mutex + jobs map[string]*Job // 全部任务（含完成） + pending []*Job // 待领取队列 + + // 等待新任务的 worker 通知 + waiters []chan struct{} + + // cancellation 待派发给 worker 的取消列表（worker_id -> job_ids） + pendingCancels map[string][]string + + // onCancel 回调：Cancel 时同步通知本地 runner kill FFmpeg 进程 + // 由 Manager 在启动时注入 + onCancel func(jobID string) + + registry *WorkerRegistry +} + +func NewScheduler(reg *WorkerRegistry) *Scheduler { + return &Scheduler{ + jobs: make(map[string]*Job), + pendingCancels: make(map[string][]string), + registry: reg, + } +} + +// SetOnCancel 注入取消回调（由 Manager 在启动本地 worker 后调用） +func (s *Scheduler) SetOnCancel(fn func(jobID string)) { + s.mu.Lock() + s.onCancel = fn + s.mu.Unlock() +} + +// 提交配额。转码是 CPU/磁盘密集型操作，必须限制单用户并发与全局排队长度， +// 否则任何能访问播放接口的用户都能无限堆积任务（每个任务还会触发 30s 的 ffprobe）。 +const ( + // MaxPendingJobs 全局排队（未被 worker 领取）任务上限 + MaxPendingJobs = 200 + // MaxActiveJobsPerUser 单用户同时存在的未结束任务上限 + MaxActiveJobsPerUser = 3 +) + +var ( + // ErrQueueFull 全局队列已满 + ErrQueueFull = errors.New("transcode queue is full, try again later") + // ErrUserQuota 单用户并发任务已达上限 + ErrUserQuota = errors.New("too many active transcode jobs for this user") + // ErrJobNotFound 任务不存在或已被清理 + ErrJobNotFound = errors.New("transcode job not found") +) + +// isActiveStatus 任务是否仍占用资源（未结束） +func isActiveStatus(st JobStatus) bool { + return st == JobPending || st == JobRunning || st == JobReady +} + +// FindReusable 查找同一用户对同一路径、仍在进行中的任务，用于去重复用。 +func (s *Scheduler) FindReusable(ownerID string, path string) (*Job, bool) { + s.mu.Lock() + defer s.mu.Unlock() + return s.findReusableLocked(ownerID, path) +} + +func (s *Scheduler) findReusableLocked(ownerID string, path string) (*Job, bool) { + for _, j := range s.jobs { + if j.OwnerID == ownerID && j.Path == path && isActiveStatus(j.GetStatus()) { + return j, true + } + } + return nil, false +} + +// SubmitLimited 带去重与配额的提交： +// 1. 同一用户对同一路径已有进行中的任务时直接复用，不重复入队； +// 2. 全局排队过长或用户任务过多时拒绝。 +// +// 返回 (job, reused, err)。 +func (s *Scheduler) SubmitLimited(j *Job) (*Job, bool, error) { + s.mu.Lock() + defer s.mu.Unlock() + if exist, ok := s.findReusableLocked(j.OwnerID, j.Path); ok { + return exist, true, nil + } + if len(s.pending) >= MaxPendingJobs { + return nil, false, ErrQueueFull + } + active := 0 + for _, x := range s.jobs { + if x.OwnerID == j.OwnerID && isActiveStatus(x.GetStatus()) { + active++ + } + } + if active >= MaxActiveJobsPerUser { + return nil, false, ErrUserQuota + } + // 检查与入队在同一把锁内完成，避免并发请求同时越过配额 + return s.enqueueLocked(j), false, nil +} + +// Capacity 只读预检：在启动 ffprobe 这类昂贵操作之前先看配额是否已满。 +// +// 真正的配额判定仍由 SubmitLimited 在同一把锁内完成；这里的目的有两个： +// 1. 不让用户先等一次最长 30s 的 ffprobe 才收到 429； +// 2. 把「并发请求同时通过配额判定」这个窗口内的探测数量压下来。 +func (s *Scheduler) Capacity(ownerID string) error { + s.mu.Lock() + defer s.mu.Unlock() + if len(s.pending) >= MaxPendingJobs { + return ErrQueueFull + } + active := 0 + for _, x := range s.jobs { + if x.OwnerID == ownerID && isActiveStatus(x.GetStatus()) { + active++ + } + } + if active >= MaxActiveJobsPerUser { + return ErrUserQuota + } + return nil +} + +// Submit 提交一个任务并返回（带 ID + token）。不做配额限制，仅供内部流程（如重新激活）使用。 +func (s *Scheduler) Submit(j *Job) *Job { + s.mu.Lock() + defer s.mu.Unlock() + return s.enqueueLocked(j) +} + +// enqueueLocked 补全任务字段并入队，调用方必须已持有 s.mu。 +func (s *Scheduler) enqueueLocked(j *Job) *Job { + if j.ID == "" { + j.ID = "job_" + uuid.NewString() + } + if j.CallbackToken == "" { + j.CallbackToken = "tk_" + uuid.NewString() + } + // 播放凭据与 Worker 回写凭据分离：播放端只能读，不能上传切片或结束任务 + if j.PlayToken == "" { + j.PlayToken = "pt_" + uuid.NewString() + } + if j.readyCh == nil { + j.readyCh = make(chan struct{}) + } + if j.CreatedAt.IsZero() { + j.CreatedAt = time.Now() + } + j.Status = JobPending + + s.jobs[j.ID] = j + s.pending = append(s.pending, j) + // 唤醒等待的 worker + for _, ch := range s.waiters { + close(ch) + } + s.waiters = nil + return j +} + +// Get 通过 id 获取任务 +func (s *Scheduler) Get(id string) (*Job, bool) { + s.mu.Lock() + defer s.mu.Unlock() + j, ok := s.jobs[id] + return j, ok +} + +// Claim 由 worker 调用领取任务（最多 slots 个）；如果没有任务则等待 wait 秒 +func (s *Scheduler) Claim(workerID string, slots int, wait time.Duration) []*Job { + worker, ok := s.registry.Get(workerID) + if !ok { + return nil + } + taken := s.claimNow(worker, slots) + if len(taken) > 0 || wait <= 0 { + return taken + } + // 长轮询等待 + s.mu.Lock() + notify := make(chan struct{}) + s.waiters = append(s.waiters, notify) + s.mu.Unlock() + select { + case <-notify: + case <-time.After(wait): + } + return s.claimNow(worker, slots) +} + +func (s *Scheduler) claimNow(worker *Worker, slots int) []*Job { + s.mu.Lock() + defer s.mu.Unlock() + if slots <= 0 || len(s.pending) == 0 { + return nil + } + // 按打分排序候选任务，选最匹配的前 slots 个 + candidates := append([]*Job(nil), s.pending...) + sort.SliceStable(candidates, func(i, j int) bool { + return scoreJobForWorker(candidates[i], worker) > scoreJobForWorker(candidates[j], worker) + }) + var taken []*Job + for _, j := range candidates { + if len(taken) >= slots { + break + } + if !workerCanRun(worker, j) { + continue + } + j.mu.Lock() + j.Status = JobRunning + j.WorkerID = worker.ID + j.StartedAt = time.Now() + j.mu.Unlock() + taken = append(taken, j) + // 从 pending 中移除 + for idx, pj := range s.pending { + if pj == j { + s.pending = append(s.pending[:idx], s.pending[idx+1:]...) + break + } + } + } + return taken +} + +// scoreJobForWorker 给 (worker, job) 打分（数值越大越匹配） +func scoreJobForWorker(j *Job, w *Worker) float64 { + score := 1.0 + // 硬件加速匹配 + 50 + if len(j.Profiles) > 0 { + req := strings.ToLower(j.Profiles[0].HWAccel) + if req != "" && req != "none" && req != "auto" { + for _, hw := range w.HWAccel { + if strings.EqualFold(hw, req) { + score += 50 + break + } + } + } + } + // 空闲槽位多 + + score += float64(w.FreeSlots) * 2 + // 负载低 + + score += (1 - w.Load) * 10 + return score +} + +// workerCanRun 检查 worker 是否能运行该 job +func workerCanRun(w *Worker, j *Job) bool { + if w.FreeSlots <= 0 { + return false + } + if j.Probe.VideoCodec != "" && len(w.CodecsDecode) > 0 { + ok := false + for _, c := range w.CodecsDecode { + if strings.EqualFold(c, j.Probe.VideoCodec) { + ok = true + break + } + } + if !ok { + return false + } + } + return true +} + +// Finish 由 worker 调用，标记任务完成/失败 +func (s *Scheduler) Finish(req *FinishRequest) error { + s.mu.Lock() + j, ok := s.jobs[req.JobID] + s.mu.Unlock() + if !ok { + return errors.New("job not found") + } + j.mu.Lock() + defer j.mu.Unlock() + j.Status = req.Status + j.Error = req.Error + j.FinishedAt = time.Now() + // 任务一旦完成或失败，确保 readyCh 一定会被关闭，避免播放端死等 + j.readyOnce.Do(func() { close(j.readyCh) }) + return nil +} + +// Cancel 主动取消任务（同时通知对应 worker 并 kill 本地 FFmpeg 进程） +func (s *Scheduler) Cancel(jobID string) { + s.mu.Lock() + j, ok := s.jobs[jobID] + if !ok { + s.mu.Unlock() + return + } + j.mu.Lock() + if j.Status == JobPending || j.Status == JobRunning || j.Status == JobReady { + j.Status = JobCancelled + } + j.readyOnce.Do(func() { close(j.readyCh) }) + wid := j.WorkerID + j.mu.Unlock() + if wid != "" { + s.pendingCancels[wid] = append(s.pendingCancels[wid], jobID) + } + // 从 pending 中剔除 + for idx, pj := range s.pending { + if pj.ID == jobID { + s.pending = append(s.pending[:idx], s.pending[idx+1:]...) + break + } + } + cancelFn := s.onCancel + s.mu.Unlock() + // 在锁外调用回调，避免死锁；回调会 kill 本地 FFmpeg 进程 + if cancelFn != nil { + cancelFn(jobID) + } +} + +// ListActiveJobs 列出所有活跃任务（pending/running/ready），用于空闲超时检测 +func (s *Scheduler) ListActiveJobs() []*Job { + s.mu.Lock() + defer s.mu.Unlock() + var out []*Job + for _, j := range s.jobs { + st := j.GetStatus() + if st == JobRunning || st == JobReady || st == JobPending { + out = append(out, j) + } + } + return out +} + +// PopCancellations 取出某个 worker 待处理的取消列表（心跳响应中下发） +func (s *Scheduler) PopCancellations(workerID string) []string { + s.mu.Lock() + defer s.mu.Unlock() + ids := s.pendingCancels[workerID] + delete(s.pendingCancels, workerID) + return ids +} + +// ListJobs 列出全部任务（仅用于调试/管理） +func (s *Scheduler) ListJobs() []*Job { + s.mu.Lock() + defer s.mu.Unlock() + out := make([]*Job, 0, len(s.jobs)) + for _, j := range s.jobs { + out = append(out, j) + } + return out +} + +// Reactivate 把一个已 Cancelled / Failed / Finished 的 job 重新激活回 Pending 状态， +// 让 worker 能再次领取并跑转码。用于"播放端重新请求时按需重启转码"的场景。 +// +// 设计点： +// 1. 复用原 jobID 和两套 token（播放用 PlayToken / Worker 回写用 CallbackToken）， +// 前端持有的 master.m3u8 中的 URL 不需要变更 +// 2. 重置 readyCh / Status / WorkerID 等运行时字段，但保留 Path / SourceURL / Profiles / Probe +// 3. 重新加入 pending 队列并唤醒等待的 worker +// +// 返回被激活的 Job；job 不存在返回 ErrJobNotFound，单用户配额不足返回 ErrUserQuota， +// 全局排队已满返回 ErrQueueFull。 +func (s *Scheduler) Reactivate(jobID string) (*Job, error) { + s.mu.Lock() + defer s.mu.Unlock() + j, ok := s.jobs[jobID] + if !ok { + return nil, ErrJobNotFound + } + // 已经活跃：无需重新入队 + if isActiveStatus(j.GetStatus()) { + return j, nil + } + + // 重新激活同样受全局排队上限约束：否则可以靠「先让任务 idle 取消、再批量激活」 + // 把 pending 队列顶到 MaxPendingJobs 之外。 + if len(s.pending) >= MaxPendingJobs { + return nil, ErrQueueFull + } + + // 重新激活同样占用配额。否则持有 PlayToken 的用户可以先让任务因 idle 被取消， + // 再统一请求 playlist / seg-N.ts 把它们一起重新激活，绕过 SubmitLimited 的单用户上限。 + active := 0 + for _, x := range s.jobs { + if x == j { + continue + } + if x.OwnerID == j.OwnerID && isActiveStatus(x.GetStatus()) { + active++ + } + } + if active >= MaxActiveJobsPerUser { + return nil, ErrUserQuota + } + + j.mu.Lock() + // 期间可能已被其它请求激活 + if isActiveStatus(j.Status) { + j.mu.Unlock() + return j, nil + } + // 重置运行时状态 + j.Status = JobPending + j.WorkerID = "" + j.Error = "" + j.FinishedAt = time.Time{} + j.LastAccessAt = time.Now() + // 重置 readyCh：原 channel 已 closed，需要全新一份给后续 WaitReady + j.readyCh = make(chan struct{}) + j.readyOnce = sync.Once{} + j.mu.Unlock() + + // 重新入 pending 队列（先确认不在队列中以免重复） + inQueue := false + for _, pj := range s.pending { + if pj.ID == jobID { + inQueue = true + break + } + } + if !inQueue { + s.pending = append(s.pending, j) + } + // 唤醒等待的 worker + for _, ch := range s.waiters { + close(ch) + } + s.waiters = nil + return j, nil +} + +// PurgeBefore 删除 finishedAt 早于某时刻的任务，避免 jobs map 无限增长 +// 返回被清理的 job ID 列表，调用方需联动清理 Cache +func (s *Scheduler) PurgeBefore(t time.Time) []string { + s.mu.Lock() + defer s.mu.Unlock() + var purged []string + for id, j := range s.jobs { + if (j.Status == JobFinished || j.Status == JobFailed || j.Status == JobCancelled) && + !j.FinishedAt.IsZero() && j.FinishedAt.Before(t) { + delete(s.jobs, id) + purged = append(purged, id) + } + } + return purged +} diff --git a/internal/transcode/scheduler_quota_test.go b/internal/transcode/scheduler_quota_test.go new file mode 100644 index 00000000..cf8d6866 --- /dev/null +++ b/internal/transcode/scheduler_quota_test.go @@ -0,0 +1,266 @@ +package transcode + +import ( + "errors" + "fmt" + "sync" + "testing" +) + +func newTestScheduler() *Scheduler { + return NewScheduler(NewWorkerRegistry()) +} + +func TestSubmitLimitedReusesSameUserPath(t *testing.T) { + s := newTestScheduler() + a := NewJob() + a.OwnerID, a.Path = "u1", "/movie.mkv" + first, reused, err := s.SubmitLimited(a) + if err != nil || reused { + t.Fatalf("first submit: reused=%v err=%v", reused, err) + } + + b := NewJob() + b.OwnerID, b.Path = "u1", "/movie.mkv" + second, reused, err := s.SubmitLimited(b) + if err != nil || !reused { + t.Fatalf("second submit should reuse: reused=%v err=%v", reused, err) + } + if second.ID != first.ID { + t.Fatalf("reuse returned a different job: %s vs %s", second.ID, first.ID) + } + if got := len(s.ListJobs()); got != 1 { + t.Fatalf("duplicate job queued, jobs=%d", got) + } + + // 其他用户不应复用别人的任务（callback_token 属于发起者） + c := NewJob() + c.OwnerID, c.Path = "u2", "/movie.mkv" + third, reused, err := s.SubmitLimited(c) + if err != nil || reused || third.ID == first.ID { + t.Fatalf("other user must get own job: reused=%v err=%v", reused, err) + } +} + +func TestSubmitLimitedPerUserQuota(t *testing.T) { + s := newTestScheduler() + for i := 0; i < MaxActiveJobsPerUser; i++ { + j := NewJob() + j.OwnerID, j.Path = "u7", fmt.Sprintf("/f%d.mkv", i) + if _, _, err := s.SubmitLimited(j); err != nil { + t.Fatalf("submit %d: %v", i, err) + } + } + over := NewJob() + over.OwnerID, over.Path = "u7", "/over.mkv" + if _, _, err := s.SubmitLimited(over); !errors.Is(err, ErrUserQuota) { + t.Fatalf("want ErrUserQuota, got %v", err) + } + // 其它用户不受影响 + other := NewJob() + other.OwnerID, other.Path = "u8", "/other.mkv" + if _, _, err := s.SubmitLimited(other); err != nil { + t.Fatalf("other user blocked: %v", err) + } +} + +// Capacity 的判定必须与 SubmitLimited 一致：handler 靠它在启动 ffprobe 前预检配额。 +func TestCapacityMatchesSubmitLimitedQuota(t *testing.T) { + s := newTestScheduler() + if err := s.Capacity("cap-user"); err != nil { + t.Fatalf("empty scheduler should have capacity: %v", err) + } + for i := 0; i < MaxActiveJobsPerUser; i++ { + j := NewJob() + j.OwnerID, j.Path = "cap-user", fmt.Sprintf("/cap%d.mkv", i) + if _, _, err := s.SubmitLimited(j); err != nil { + t.Fatalf("submit %d: %v", i, err) + } + } + if err := s.Capacity("cap-user"); !errors.Is(err, ErrUserQuota) { + t.Fatalf("want ErrUserQuota, got %v", err) + } + // 其他用户不受影响 + if err := s.Capacity("cap-other"); err != nil { + t.Fatalf("other user should have capacity: %v", err) + } +} + +func TestSubmitLimitedQueueCap(t *testing.T) { + s := newTestScheduler() + for i := 0; i < MaxPendingJobs; i++ { + j := NewJob() + // 每个任务不同用户，避免触发单用户上限 + j.OwnerID, j.Path = fmt.Sprintf("u%d", 1000+i), "/x.mkv" + if _, _, err := s.SubmitLimited(j); err != nil { + t.Fatalf("submit %d: %v", i, err) + } + } + j := NewJob() + j.OwnerID, j.Path = "u99999", "/y.mkv" + if _, _, err := s.SubmitLimited(j); !errors.Is(err, ErrQueueFull) { + t.Fatalf("want ErrQueueFull, got %v", err) + } +} + +func TestSubmitLimitedConcurrentQuota(t *testing.T) { + s := newTestScheduler() + var wg sync.WaitGroup + var mu sync.Mutex + accepted := 0 + for i := 0; i < 50; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + j := NewJob() + j.OwnerID, j.Path = "u5", fmt.Sprintf("/p%d.mkv", i) + if _, _, err := s.SubmitLimited(j); err == nil { + mu.Lock() + accepted++ + mu.Unlock() + } + }(i) + } + wg.Wait() + if accepted != MaxActiveJobsPerUser { + t.Fatalf("concurrent submits accepted=%d, want %d", accepted, MaxActiveJobsPerUser) + } +} + +// 重新激活（播放端请求 playlist / 切片时触发）也必须占配额， +// 否则可以先让任务闲置被取消，再统一激活来绕过单用户上限。 +func TestReactivateHonorsUserQuota(t *testing.T) { + s := newTestScheduler() + for i := 0; i < MaxActiveJobsPerUser; i++ { + j := NewJob() + j.OwnerID, j.Path = "u9", fmt.Sprintf("/r%d.mkv", i) + if _, _, err := s.SubmitLimited(j); err != nil { + t.Fatalf("submit %d: %v", i, err) + } + } + + // 构造一个"该用户此前创建、随后因闲置被取消"的任务 + idle := NewJob() + idle.OwnerID, idle.Path = "u10", "/idle.mkv" + sub, _, err := s.SubmitLimited(idle) + if err != nil { + t.Fatalf("submit idle: %v", err) + } + sub.OwnerID = "u9" + s.Cancel(sub.ID) + + if _, rerr := s.Reactivate(sub.ID); !errors.Is(rerr, ErrUserQuota) { + t.Fatalf("want ErrUserQuota, got %v", rerr) + } + + // 释放一个活跃任务后应能重新激活 + for _, j := range s.ListJobs() { + if j.Path != "/idle.mkv" && j.OwnerID == "u9" { + s.Cancel(j.ID) + break + } + } + if _, rerr := s.Reactivate(sub.ID); rerr != nil { + t.Fatalf("reactivate after freeing quota: %v", rerr) + } + if st := sub.GetStatus(); st != JobPending { + t.Fatalf("status=%v, want pending", st) + } +} + +// 重新激活同样不能突破全局排队上限：否则可以先让任务闲置取消，再批量激活把队列顶爆。 +func TestReactivateHonorsQueueLimit(t *testing.T) { + s := newTestScheduler() + + // 先造一个随后被取消的任务（Cancel 会把它从 pending 中移除） + idle := NewJob() + idle.OwnerID, idle.Path = "q1", "/idle.mkv" + sub, _, err := s.SubmitLimited(idle) + if err != nil { + t.Fatalf("submit idle: %v", err) + } + s.Cancel(sub.ID) + + // 用不同用户把全局队列填满（避开单用户配额） + for i := 0; i < MaxPendingJobs; i++ { + j := NewJob() + j.OwnerID, j.Path = fmt.Sprintf("q%d", i+2), fmt.Sprintf("/q%d.mkv", i) + if _, _, err := s.SubmitLimited(j); err != nil { + t.Fatalf("fill %d: %v", i, err) + } + } + + if _, rerr := s.Reactivate(sub.ID); !errors.Is(rerr, ErrQueueFull) { + t.Fatalf("want ErrQueueFull, got %v", rerr) + } + if st := sub.GetStatus(); st != JobCancelled { + t.Fatalf("status=%v, want cancelled (must not be requeued)", st) + } +} + +func TestValidSourceInput(t *testing.T) { + ok := []string{ + "/data/movies/a.mkv", + "http://127.0.0.1:5244/d/a.mkv?sign=x", + "https://example.com/d/a.mkv", + `C:\media\a.mkv`, + "C:/media/a.mkv", + // 文件名里的冒号很常见，不能被当成协议前缀 + "/mnt/media/Star Wars: Episode IV.mkv", + "/data/Spider-Man: Homecoming (2017).mkv", + `D:\Movies\Star Wars: IV.mkv`, + } + for _, s := range ok { + if !ValidSourceInput(s) { + t.Errorf("%q should be accepted", s) + } + } + bad := []string{ + "", "-i", "-f", "concat:/etc/passwd|/etc/hosts", "subfile,,start,0,end,0,,:/etc/passwd", + "data:text/plain;base64,AAAA", "pipe:0", "rtmp://x/y", "ftp://x/y", "file:///etc/passwd", + // 盘符后必须紧跟路径分隔符，否则不是合法的 Windows 绝对路径 + "C:foo", + } + for _, s := range bad { + if ValidSourceInput(s) { + t.Errorf("%q should be rejected", s) + } + } +} + +func TestProtocolWhitelistFor(t *testing.T) { + if got := ProtocolWhitelistFor("/data/a.mkv"); got != "file" { + t.Errorf("local = %q", got) + } + if got := ProtocolWhitelistFor("HTTPS://x/y"); got == "file" || got == "" { + t.Errorf("remote = %q", got) + } + // 远程输入不应包含 file / crypto：HTTP 源用不到，留着等于给构造过的播放列表 + // 留了一条读本地文件的通道 + for _, banned := range []string{"concat", "subfile", "data", "pipe", "rtmp", "ftp", "file", "crypto"} { + if contains := func(s string) bool { + for _, p := range splitComma(s) { + if p == banned { + return true + } + } + return false + }(ProtocolWhitelistFor("http://x/y")); contains { + t.Errorf("remote whitelist must not include %s", banned) + } + } +} + +func splitComma(s string) []string { + var out []string + cur := "" + for _, r := range s { + if r == ',' { + out = append(out, cur) + cur = "" + continue + } + cur += string(r) + } + return append(out, cur) +} diff --git a/internal/transcode/types.go b/internal/transcode/types.go new file mode 100644 index 00000000..4df99320 --- /dev/null +++ b/internal/transcode/types.go @@ -0,0 +1,178 @@ +// Package transcode 实现 OpenList 的 FFmpeg 云端/本地转码功能。 +// 设计目标： +// 1. 总开关默认关，开启后按文件大小、源码率、源编码、文件后缀过滤决定是否转码。 +// 2. 协议自研，纯 HTTP/JSON，可挂接本地内置 Worker 或远程 Worker 节点池。 +// 3. 切片流式回吐，支持边转边播。 +package transcode + +import ( + "sync" + "time" +) + +// JobStatus 转码任务状态 +type JobStatus string + +const ( + JobPending JobStatus = "pending" // 等待 Worker 领取 + JobRunning JobStatus = "running" // Worker 已领取并在转码 + JobReady JobStatus = "ready" // 至少首切片可用 + JobFinished JobStatus = "finished" // 全部切片完成 + JobFailed JobStatus = "failed" + JobCancelled JobStatus = "cancelled" +) + +// OutputFormat 输出封装格式 +type OutputFormat string + +const ( + FormatHLS OutputFormat = "hls" + FormatDASH OutputFormat = "dash" + FormatMP4 OutputFormat = "mp4" +) + +// HWAccel 硬件加速类型 +type HWAccel string + +const ( + HWNone HWAccel = "none" + HWAuto HWAccel = "auto" + HWNVENC HWAccel = "nvenc" // NVIDIA + HWQSV HWAccel = "qsv" // Intel QuickSync + HWVAAPI HWAccel = "vaapi" // Linux VA-API + HWAMF HWAccel = "amf" // AMD AMF + HWVideoToolbox HWAccel = "videotoolbox" // macOS +) + +// SourceProbe 源文件探测信息（由 ffprobe 得出） +type SourceProbe struct { + Duration float64 `json:"duration"` // 秒 + Size int64 `json:"size"` // 字节 + VideoCodec string `json:"video_codec"` // h264 / hevc / av1 / vp9 ... + VideoBitrate int64 `json:"video_bitrate"` // bps + Width int `json:"width"` + Height int `json:"height"` + AudioCodec string `json:"audio_codec"` + AudioBitrate int64 `json:"audio_bitrate"` +} + +// Profile 输出转码档位 +type Profile struct { + Name string `json:"name"` // 例如 1080p + VideoCodec string `json:"video_codec"` // h264 / hevc / av1 + VideoBitrate string `json:"video_bitrate"` // 例如 4000k + Scale string `json:"scale"` // 例如 1920:-2 + AudioCodec string `json:"audio_codec"` // aac / mp3 / opus / copy + AudioBitrate string `json:"audio_bitrate"` // 例如 160k + HWAccel string `json:"hwaccel"` // worker 端如何选择硬件加速 +} + +// OutputSpec 输出参数 +type OutputSpec struct { + Format OutputFormat `json:"format"` + SegmentDuration int `json:"segment_duration"` // 秒 +} + +// Job 一次转码任务 +type Job struct { + ID string `json:"id"` + Path string `json:"path"` // OpenList 文件路径 + SourceURL string `json:"source_url"` // 提供给 Worker 的签名 URL + Probe SourceProbe `json:"probe"` + Profiles []Profile `json:"profiles"` + Output OutputSpec `json:"output"` + CallbackToken string `json:"callback_token"` // Worker 回吐切片用（同时具备写权限，绝不外泄给播放端） + // PlayToken 播放凭据：只用于 master/variant playlist 与切片下发， + // 与 Worker 回写凭据（CallbackToken）分离，避免播放器拿到可写权限。 + // json:"-" 保证下发给 Worker 的任务描述里不会包含它。 + PlayToken string `json:"-"` + Deadline int64 `json:"deadline"` // unix 秒 + + // OwnerID 发起转码者的配额键，仅服务端用于去重与配额，不下发给 Worker。 + // 用字符串而不是用户 ID：访客共用一个账号，需要按 "guest:" 拆分配额， + // 否则单个匿名访客就能占满所有匿名转码配额。 + OwnerID string `json:"-"` + + // 运行时 + Status JobStatus `json:"status"` + WorkerID string `json:"worker_id"` + CreatedAt time.Time `json:"created_at"` + StartedAt time.Time `json:"started_at"` + FinishedAt time.Time `json:"finished_at"` + LastAccessAt time.Time `json:"last_access_at"` // 播放端最后一次请求 segment/playlist 的时间 + Error string `json:"error,omitempty"` + + // 内部：用于"边转边播"——首切片就绪后关闭该 channel + readyOnce sync.Once + readyCh chan struct{} + mu sync.RWMutex +} + +// NewJob 构造任务（自动初始化内部 channel） +func NewJob() *Job { + now := time.Now() + return &Job{ + Status: JobPending, + CreatedAt: now, + LastAccessAt: now, + readyCh: make(chan struct{}), + } +} + +// Touch 更新最后访问时间（播放端每次请求 segment/playlist 时调用） +func (j *Job) Touch() { + j.mu.Lock() + j.LastAccessAt = time.Now() + j.mu.Unlock() +} + +// GetLastAccess 获取最后访问时间（线程安全） +func (j *Job) GetLastAccess() time.Time { + j.mu.RLock() + defer j.mu.RUnlock() + return j.LastAccessAt +} + +// GetStatus 获取当前状态（线程安全） +func (j *Job) GetStatus() JobStatus { + j.mu.RLock() + defer j.mu.RUnlock() + return j.Status +} + +// MarkReady 标记任务已可播放（首切片就绪） +func (j *Job) MarkReady() { + j.readyOnce.Do(func() { + j.mu.Lock() + if j.Status == JobRunning || j.Status == JobPending { + j.Status = JobReady + } + j.mu.Unlock() + close(j.readyCh) + }) +} + +// WaitReady 阻塞等待首切片就绪 +func (j *Job) WaitReady() <-chan struct{} { return j.readyCh } + +// Worker 节点信息 +type Worker struct { + ID string `json:"id"` + Name string `json:"name"` + Version string `json:"version"` + Capacity int `json:"capacity"` // 最大并发任务数 + HWAccel []string `json:"hwaccel"` // 支持的硬件加速 + CodecsDecode []string `json:"codecs_decode"` // 可解码 + CodecsEncode []string `json:"codecs_encode"` // 可编码 + MaxResolution string `json:"max_resolution"` // 最大分辨率 + Tags []string `json:"tags"` + Endpoint string `json:"endpoint,omitempty"` // 反向连接地址（可选） + + // 运行时 + Load float64 `json:"load"` // 0~1 + Running []string `json:"running"` // 当前 job_ids + FreeSlots int `json:"free_slots"` + LastSeen time.Time `json:"last_seen"` + RegisteredAt time.Time `json:"registered_at"` + Local bool `json:"local"` // 是否本地内置 Worker +} diff --git a/server/handles/auth.go b/server/handles/auth.go index 2013a5c2..75d3b4d7 100644 --- a/server/handles/auth.go +++ b/server/handles/auth.go @@ -11,6 +11,7 @@ import ( "github.com/OpenListTeam/OpenList/v4/internal/model" "github.com/OpenListTeam/OpenList/v4/internal/op" "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/OpenListTeam/go-cache" "github.com/gin-gonic/gin" "github.com/pquerna/otp/totp" "gorm.io/gorm" @@ -48,21 +49,22 @@ func loginHash(c *gin.Context, req *LoginReq) { ip := c.ClientIP() count, ok := model.LoginCache.Get(ip) if ok && count >= model.DefaultMaxAuthRetries { + // 被拒时不能再续期：自动重试的客户端（如凭据失效的 Subsonic/Web 轮询） + // 会每来一次就刷新锁定窗口，把自己永久锁死。锁定窗口只在计数时设定。 common.ErrorStrResp(c, model.TooManyAttempts, 429) - model.LoginCache.Expire(ip, model.DefaultLockDuration) return } // check username user, err := op.GetUserByName(req.Username) if err != nil { common.ErrorStrResp(c, model.InvalidUsernameOrPassword, 401) - model.LoginCache.Set(ip, count+1) + model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration)) return } // validate password hash if err := user.ValidatePwdStaticHash(req.Password); err != nil { common.ErrorStrResp(c, model.InvalidUsernameOrPassword, 401) - model.LoginCache.Set(ip, count+1) + model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration)) return } // check 2FA @@ -70,7 +72,7 @@ func loginHash(c *gin.Context, req *LoginReq) { if !totp.Validate(req.OtpCode, user.OtpSecret) { // 402 - need opt common.ErrorStrResp(c, model.Invalid2FACode, 402) - model.LoginCache.Set(ip, count+1) + model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration)) return } } diff --git a/server/handles/down.go b/server/handles/down.go index 50025a0b..82c44304 100644 --- a/server/handles/down.go +++ b/server/handles/down.go @@ -51,7 +51,9 @@ func Proxy(c *gin.Context) { common.ErrorPage(c, err, 500) return } - if canProxy(storage, filename) { + // 支持 ?force 参数强制代理（用于媒体库等需要 JS 加载文件内容的场景，避免 CORS 问题） + _, forceProxy := c.GetQuery("force") + if forceProxy || canProxy(storage, filename) { if _, ok := c.GetQuery("d"); !ok { if url := common.GenerateDownProxyURL(storage.GetStorage(), rawPath); url != "" { c.Redirect(302, url) diff --git a/server/handles/emby.go b/server/handles/emby.go new file mode 100644 index 00000000..85c0389d --- /dev/null +++ b/server/handles/emby.go @@ -0,0 +1,1246 @@ +package handles + +import ( + "crypto/md5" + "crypto/sha1" + "encoding/hex" + "encoding/json" + "hash/fnv" + "net" + "net/http" + stdpath "path" + "regexp" + "strconv" + "strings" + "sync" + "time" + + "github.com/gorilla/websocket" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/op" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +// ==================== 数据结构（Emby 使用 PascalCase 字段名） ==================== + +type embyNameID struct { + Name string + // Id 在官方 DTO 中是 Guid?，空字符串必须省略 + Id string `json:",omitempty"` +} + +type embyUserData struct { + PlaybackPositionTicks int64 + PlayCount int + IsFavorite bool + Played bool + UnplayedItemCount int + PlayedPercentage float64 + Rating float64 `json:",omitempty"` + Likes bool + // LastPlayedDate 是 DateTime?，空字符串必须省略 + LastPlayedDate string `json:",omitempty"` +} + +// embyMediaSource 对应官方 MediaSourceInfo。 +// 客户端按 SupportsDirectPlay -> SupportsDirectStream -> TranscodingUrl 的顺序判定播放方式， +// 同时读取 MediaStreams（视频/音频轨）和 RequiresOpening 等字段，缺失会被判定为"不可播放"。 +type embyMediaSource struct { + Protocol string + Id string + Path string `json:",omitempty"` + Type string + Container string `json:",omitempty"` + Name string `json:",omitempty"` + Size int64 `json:",omitempty"` + IsRemote bool + ETag string `json:",omitempty"` + RunTimeTicks int64 `json:",omitempty"` + ReadAtNativeFramerate bool + IgnoreDts bool + IgnoreIndex bool + GenPtsInput bool + SupportsTranscoding bool + SupportsDirectStream bool + SupportsDirectPlay bool + IsInfiniteStream bool + RequiresOpening bool + RequiresClosing bool + RequiresLooping bool + SupportsProbing bool + VideoType string `json:",omitempty"` + MediaStreams []embyMediaStream + MediaAttachments []any + Formats []string + Bitrate int `json:",omitempty"` + RequiredHttpHeaders map[string]string + DirectStreamUrl string `json:",omitempty"` + AddApiKeyToDirectStreamUrl bool + DefaultAudioStreamIndex int `json:",omitempty"` + // DefaultSubtitleStreamIndex 指向歌词字幕轨：客户端歌词面板只认 + // Type=="Subtitle" 且 Index 等于该值的流，缺失就不请求歌词 + DefaultSubtitleStreamIndex *int `json:",omitempty"` + ItemId string `json:",omitempty"` +} + +// embyMediaStream 对应官方 MediaStream，只填写客户端判定播放所需的最小字段。 +type embyMediaStream struct { + Codec string `json:",omitempty"` + Type string + Index int + IsDefault bool + IsForced bool + IsExternal bool + IsInterlaced bool + IsTextSubtitleStream bool + SupportsExternalStream bool + Channels int `json:",omitempty"` + SampleRate int `json:",omitempty"` + BitRate int `json:",omitempty"` + DisplayTitle string `json:",omitempty"` + Language string `json:",omitempty"` + DeliveryMethod string `json:",omitempty"` + DeliveryUrl string `json:",omitempty"` +} + +// embyItem 字段名与类型严格对应 Emby/Jellyfin 的 BaseItemDto： +// Id/ParentId/SeriesId/SeasonId/AlbumId 是 Guid，DateCreated/PremiereDate/EndDate 是 DateTime?。 +// 空值必须省略（omitempty），否则客户端会把 "" 当作 Guid/DateTime 解析并抛异常。 +type embyItem struct { + Name string `json:",omitempty"` + ServerId string `json:",omitempty"` + Id string `json:",omitempty"` + DateCreated string `json:",omitempty"` + Container string `json:",omitempty"` + SortName string `json:",omitempty"` + PremiereDate string `json:",omitempty"` + Overview string `json:",omitempty"` + ProductionYear int `json:",omitempty"` + CommunityRating float32 `json:",omitempty"` + OfficialRating string `json:",omitempty"` + RunTimeTicks int64 `json:",omitempty"` + IndexNumber int `json:",omitempty"` + ParentIndexNumber int `json:",omitempty"` + Type string `json:",omitempty"` + MediaType string `json:",omitempty"` + IsFolder bool + ParentId string `json:",omitempty"` + SeriesId string `json:",omitempty"` + SeriesName string `json:",omitempty"` + SeasonId string `json:",omitempty"` + SeasonName string `json:",omitempty"` + Album string `json:",omitempty"` + AlbumId string `json:",omitempty"` + AlbumArtist string `json:",omitempty"` + AlbumArtists []embyNameID + Artists []string + ArtistItems []embyNameID + Genres []string + GenreItems []embyNameID + Studios []embyNameID + People []any + MediaStreams []any + Taglines []string + ProductionLocations []string + Keywords []string + RemoteTrailers []any + ExternalUrls []any + LockedFields []string + Path string `json:",omitempty"` + ImageTags map[string]string + BackdropImageTags []string + ProviderIds map[string]string + UserData *embyUserData + MediaSources []embyMediaSource + ChildCount int `json:",omitempty"` + LocationType string `json:",omitempty"` + CollectionType string `json:",omitempty"` + ViewType string `json:",omitempty"` + // Subviews 是电影库页面的必需字段：客户端 videos.js 直接读取 + // item.Subviews 并调用 .includes()，缺失会抛 undefined.includes 导致整页报错 + Subviews []string `json:",omitempty"` + EndDate string `json:",omitempty"` + Status string `json:",omitempty"` + PlayAccess string `json:",omitempty"` + Video3DFormat string `json:",omitempty"` + PreferredMetadataLanguage string `json:",omitempty"` + PreferredMetadataCountryCode string `json:",omitempty"` + CleanName string `json:",omitempty"` + PresentationUniqueKey string `json:",omitempty"` + ShortOverview string `json:",omitempty"` + HomePageUrl string `json:",omitempty"` + SeriesPrimaryImageTag string `json:",omitempty"` + PrimaryImageTag string `json:",omitempty"` + ParentThumbItemId string `json:",omitempty"` + ParentThumbImageTag string `json:",omitempty"` + ParentPrimaryImageItemId string `json:",omitempty"` + ParentPrimaryImageTag string `json:",omitempty"` + AlbumPrimaryImageTag string `json:",omitempty"` + ParentBackdropItemId string `json:",omitempty"` + AirTime string `json:",omitempty"` + PrimaryImageAspectRatio float64 `json:",omitempty"` + CriticRating float64 `json:",omitempty"` + MediaSourceCount int `json:",omitempty"` + LocalTrailerCount int `json:",omitempty"` + RecursiveItemCount int `json:",omitempty"` + SpecialFeatureCount int `json:",omitempty"` + PartCount int `json:",omitempty"` + IsHD bool + CanDelete bool + CanDownload bool + SupportsSync bool + IsPlaceHolder bool + LockData bool + ImageBlurHashes map[string]any + TagItems []embyNameID + AirDays []string + ParentBackdropImageTags []string + OriginalTitle string `json:",omitempty"` + ForcedSortName string `json:",omitempty"` + Etag string `json:",omitempty"` + DateLastMediaAdded string `json:",omitempty"` + CustomRating string `json:",omitempty"` + ChannelId string `json:",omitempty"` + ChannelName string `json:",omitempty"` + FileName string `json:",omitempty"` + Number string `json:",omitempty"` + SeriesStudio string `json:",omitempty"` + AspectRatio string `json:",omitempty"` + DisplayPreferencesId string `json:",omitempty"` + ParentLogoItemId string `json:",omitempty"` + ParentLogoImageTag string `json:",omitempty"` + IsoType string `json:",omitempty"` + Software string `json:",omitempty"` + CameraMake string `json:",omitempty"` + CameraModel string `json:",omitempty"` + Size int64 `json:",omitempty"` + Bitrate int `json:",omitempty"` + IndexNumberEnd int `json:",omitempty"` + DisplayOrder string `json:",omitempty"` + Width int `json:",omitempty"` + Height int `json:",omitempty"` + Chapters []any + Tags []string +} + +// MarshalJSON 把 nil 切片/映射输出为空数组/空对象。 +// Emby 客户端会对部分字段直接调用 .includes()， +// JSON 里出现 null 或字段缺失都会让客户端抛异常。 +func (e embyItem) MarshalJSON() ([]byte, error) { + type alias embyItem + a := alias(e) + if a.AlbumArtists == nil { + a.AlbumArtists = []embyNameID{} + } + if a.Artists == nil { + a.Artists = []string{} + } + if a.ArtistItems == nil { + a.ArtistItems = []embyNameID{} + } + if a.Genres == nil { + a.Genres = []string{} + } + if a.GenreItems == nil { + a.GenreItems = []embyNameID{} + } + if a.Studios == nil { + a.Studios = []embyNameID{} + } + if a.People == nil { + a.People = []any{} + } + if a.MediaStreams == nil { + a.MediaStreams = []any{} + } + if a.Taglines == nil { + a.Taglines = []string{} + } + if a.ProductionLocations == nil { + a.ProductionLocations = []string{} + } + if a.Keywords == nil { + a.Keywords = []string{} + } + if a.RemoteTrailers == nil { + a.RemoteTrailers = []any{} + } + if a.ExternalUrls == nil { + a.ExternalUrls = []any{} + } + if a.LockedFields == nil { + a.LockedFields = []string{} + } + if a.ImageTags == nil { + a.ImageTags = map[string]string{} + } + // 没有封面时不输出 Primary：空 tag 会让 Theater 拼出 ?tag= 的无效图片请求， + // openemby_tv 等客户端也只在 tag 非空时才请求图片 + if v, ok := a.ImageTags["Primary"]; ok && v == "" { + delete(a.ImageTags, "Primary") + } + // 剧集/歌曲等子项没有自己的封面时，让客户端回退到父级封面，保证列表不出现空白图 + if a.ImageTags["Primary"] == "" && a.ParentPrimaryImageTag != "" { + if a.ParentPrimaryImageItemId == "" { + a.ParentPrimaryImageItemId = a.ParentId + } + } + if a.BackdropImageTags == nil { + a.BackdropImageTags = []string{} + } + if a.ProviderIds == nil { + a.ProviderIds = map[string]string{} + } + if a.MediaSources == nil { + a.MediaSources = []embyMediaSource{} + } + if a.ImageBlurHashes == nil { + a.ImageBlurHashes = map[string]any{} + } + if a.TagItems == nil { + a.TagItems = []embyNameID{} + } + if a.AirDays == nil { + a.AirDays = []string{} + } + if a.ParentBackdropImageTags == nil { + a.ParentBackdropImageTags = []string{} + } + if a.Chapters == nil { + a.Chapters = []any{} + } + if a.Tags == nil { + a.Tags = []string{} + } + if a.UserData == nil { + a.UserData = &embyUserData{} + } + // PlayAccess 为空会让客户端判定"播放受限" + if a.PlayAccess == "" { + a.PlayAccess = "Full" + } + if !a.IsFolder { + a.CanDownload = true + } + // 客户端会直接展示 CleanName，并把它当作排序/查找键 + if a.CleanName == "" { + a.CleanName = a.Name + } + if a.PresentationUniqueKey == "" { + a.PresentationUniqueKey = a.Id + } + return json.Marshal(a) +} + +type embyQueryResult struct { + Items []*embyItem + TotalRecordCount int + StartIndex int +} + +type embyUserDto struct { + Name string + ServerId string + Id string + HasPassword bool + HasConfiguredPassword bool + Configuration map[string]any + Policy map[string]any + PrimaryImageTag string `json:",omitempty"` + // DateTime? 字段空值必须省略 + LastLoginDate string `json:",omitempty"` + LastActivityDate string `json:",omitempty"` + IsAdministrator bool + IsDisabled bool +} + +type embyAuthResult struct { + User *model.User + AccessToken string + ServerId string +} + +// embyPublicUserDto 登录前展示的用户信息。 +// Emby 客户端依赖 /Users/Public 渲染用户选择列表，返回空数组会跳过登录界面。 +type embyPublicUserDto struct { + Name string + ServerId string + Id string + HasPassword bool + PrimaryImageTag string +} + +// ==================== 基础工具 ==================== + +func embyServerID() string { + seed := conf.Conf.SiteURL + conf.Conf.JwtSecret + strconv.Itoa(conf.Conf.Scheme.HttpPort) + sum := sha1.Sum([]byte("openlist-emby:" + seed)) + // Emby 的 ServerId 是 32 位 hex GUID，客户端本地库会用 Guid 解析 + return hex.EncodeToString(sum[:16]) +} + +// embyCompatVersion 是返回给 Emby 客户端的兼容版本号。 +// Emby 客户端会把该值与其要求的最低服务器版本比较，直接返回 OpenList 版本 +// （如 v4.1.0 / dev）会被判定为"服务器需要升级"而拒绝连接。 +// 客户端在 4.10.0.4 起改用服务端 HomeSections API； +// 低于该版本会走本地生成逻辑，其 section 缺少 ListOptions.fields 会直接报错。 +const embyCompatVersion = "4.10.0.5" + +func embyVersion() string { + return embyCompatVersion +} + +func embyTicks(seconds int) int64 { + if seconds <= 0 { + return 0 + } + return int64(seconds) * 10_000_000 +} + +func embyTimeStr(t time.Time) string { + return t.UTC().Format("2006-01-02T15:04:05.0000000Z") +} + +func embyExt(name string) string { + ext := strings.TrimPrefix(strings.ToLower(stdpath.Ext(name)), ".") + if ext == "" { + return "mkv" + } + return ext +} + +var embyAuthHeaderRe = regexp.MustCompile(`(?i)([A-Za-z]+)\s*=\s*"([^"]*)"`) + +func embyParseAuthHeader(v string) map[string]string { + out := map[string]string{} + for _, m := range embyAuthHeaderRe.FindAllStringSubmatch(v, -1) { + out[strings.ToLower(m[1])] = m[2] + } + return out +} + +func embyRequestToken(c *gin.Context) string { + // Emby Windows 等客户端会把 X-Emby-Token 直接放在 URL query 里， + // 必须与 api_key / ApiKey 一起解析，否则登录成功后每个请求仍是 401 + for key, values := range c.Request.URL.Query() { + if len(values) == 0 || values[0] == "" { + continue + } + switch { + case strings.EqualFold(key, "api_key"), + strings.EqualFold(key, "ApiKey"), + strings.EqualFold(key, "X-Emby-Token"), + strings.EqualFold(key, "X-MediaBrowser-Token"): + return strings.TrimSpace(values[0]) + } + } + for _, h := range []string{"X-Emby-Token", "X-MediaBrowser-Token"} { + if t := strings.TrimSpace(c.GetHeader(h)); t != "" { + return t + } + } + if v := c.GetHeader("X-Emby-Authorization"); v != "" { + if t, ok := embyParseAuthHeader(v)["token"]; ok && t != "" { + return t + } + } + if v := c.GetHeader("Authorization"); strings.HasPrefix(strings.ToLower(v), "mediabrowser ") { + if t, ok := embyParseAuthHeader(v)["token"]; ok && t != "" { + return t + } + } + return "" +} + +// embyAuth 解析请求身份，token 可以是设备会话令牌或媒体应用密钥 +func embyAuth(c *gin.Context) (*model.User, bool) { + token := embyRequestToken(c) + if token == "" { + return nil, false + } + if dev, err := db.GetEmbyDeviceByTokenHash(media.HashToken(token)); err == nil { + db.TouchEmbyDevice(dev.ID) + user, err := op.GetUserById(dev.UserID) + if err != nil || user.Disabled { + return nil, false + } + return user, true + } + user, _, ok := LookUpAppKeyUser(token) + if !ok { + return nil, false + } + return user, true +} + +// embyUserGUID 返回给客户端的用户 ID（Emby 规范为 32 位 hex GUID） +func embyUserGUID(id uint) string { + return embyGUID("user-" + strconv.FormatUint(uint64(id), 10)) +} + +// embyResolveUserID 兼容数字 ID 与 GUID 两种形式 +func embyResolveUserID(idStr string) (uint, bool) { + if n, err := strconv.ParseUint(idStr, 10, 64); err == nil { + return uint(n), true + } + users, _, err := op.GetUsers(1, -1) + if err != nil { + return 0, false + } + for i := range users { + if embyUserGUID(users[i].ID) == idStr { + return users[i].ID, true + } + } + return 0, false +} + +func embyUserDtoOf(u *model.User) *embyUserDto { + // Emby 客户端的 PlayAccessValidation 会直接读取 Policy.EnableMediaPlayback， + // 缺失即判定为禁止播放，并弹出 "Playback of this content is currently restricted"。 + // 这里按 Emby 的 UserPolicy 结构给全默认值，避免任何字段被读成 undefined。 + policy := map[string]any{ + "IsAdministrator": u.IsAdmin(), + "IsDisabled": u.Disabled, + "IsHidden": false, + "EnableMediaPlayback": true, + "EnableAudioPlaybackTranscoding": true, + "EnableVideoPlaybackTranscoding": true, + "EnablePlaybackRemuxing": true, + "EnableContentDownloading": true, + "EnableSyncTranscoding": true, + "EnableSubtitleDownloading": true, + "EnableSubtitleManagement": true, + "EnableRemoteControlOfOtherUsers": false, + "EnableSharedDeviceControl": true, + "EnableRemoteAccess": true, + "EnableLiveTvManagement": false, + "EnableLiveTvAccess": false, + "EnableAllChannels": true, + "EnableAllFolders": true, + "EnableAllDevices": true, + "ForceRemoteSourceTranscoding": false, + "EnabledFolders": []string{}, + "EnabledChannels": []string{}, + "EnabledDevices": []string{}, + "BlockedChannels": []string{}, + "BlockedMediaFolders": []string{}, + "BlockedTags": []string{}, + "AllowedTags": []string{}, + "MaxParentalRating": nil, + "AuthenticationProviderId": "Default", + "PasswordResetProviderId": "Default", + } + return &embyUserDto{ + Name: u.Username, + ServerId: embyServerID(), + Id: embyUserGUID(u.ID), + HasPassword: u.PwdHash != "", + HasConfiguredPassword: u.PwdHash != "", + Configuration: map[string]any{}, + Policy: policy, + IsAdministrator: u.IsAdmin(), + IsDisabled: u.Disabled, + } +} + +// embyWebNoticeHTML 浏览器直接打开 @emby 时显示的提示页。 +// OpenList 只实现 Emby API，没有 Emby Web 客户端静态资源。 +const embyWebNoticeHTML = ` + + +OpenList - Emby API + +

OpenList 仅提供 Emby 兼容 API

+

当前地址没有 Emby Web 客户端界面，请使用支持 Emby 协议的客户端添加服务器。

+ + + + +
服务器地址http://你的地址:端口/@emby
用户名OpenList 用户名
密码媒体应用密钥（管理 - 媒体库 - 应用密钥）
+

推荐客户端：Infuse、VidHub、SenPlayer、Emby 官方 App 等。

+
+

This endpoint only serves the Emby-compatible API. Please use an Emby client instead of a browser.

+` + +// ==================== 请求分发 ==================== + +// EmbyAPI Emby 兼容 API 入口，兼容 /@emby 与 /@mediabrowser 前缀，路径大小写不敏感 +func EmbyAPI(c *gin.Context) { + if !MediaEmbyEnabled() { + c.AbortWithStatus(404) + return + } + raw := strings.Trim(c.Param("path"), "/") + if raw == "" { + c.JSON(200, gin.H{"Service": "OpenList", "Version": embyVersion()}) + return + } + // 兼容把接入地址填成 .../@emby 后客户端又自动追加 /emby（或 /mediabrowser）的情况 + if first, rest, found := strings.Cut(raw, "/"); found { + switch strings.ToLower(first) { + case "emby", "mediabrowser": + raw = rest + } + } + segs := strings.Split(raw, "/") + lower := make([]string, len(segs)) + for i, s := range segs { + lower[i] = strings.ToLower(s) + } + method := c.Request.Method + + // 公共端点：无需认证 + switch { + case len(lower) == 3 && lower[0] == "system" && lower[1] == "info" && lower[2] == "public": + embyPublicSystemInfo(c) + return + case len(lower) == 2 && lower[0] == "system" && lower[1] == "ping": + c.String(200, "Emby Server") + return + // 客户端连接阶段还会拉取公共配置，返回空对象即可，否则同样会 401 + case len(lower) == 3 && lower[0] == "system" && lower[1] == "configuration" && lower[2] == "public": + c.JSON(200, gin.H{}) + return + case len(lower) == 2 && lower[0] == "branding" && lower[1] == "configuration": + c.JSON(200, gin.H{}) + return + // 离线同步接口：OpenList 不支持客户端下载同步，返回空结果而不是 401 + case len(lower) >= 1 && lower[0] == "sync": + if method == "GET" { + c.JSON(200, []any{}) + } else { + c.Status(204) + } + return + // 浏览器直接打开 @emby 时会拉取 Emby Web 静态资源，这里返回说明页而不是 401 + case lower[0] == "web": + if strings.HasSuffix(strings.ToLower(c.Request.URL.Path), ".json") { + c.JSON(200, gin.H{}) + return + } + c.Header("Content-Type", "text/html; charset=utf-8") + c.String(200, embyWebNoticeHTML) + return + // 没有 token 的连接探测：部分客户端直接请求 /System/Info（非 Public）。 + // 必须返回完整结构（含 WebPath），精简版会让客户端在处理 sections 时崩溃 + case len(lower) == 2 && lower[0] == "system" && lower[1] == "info": + embySystemInfo(c) + return + case len(lower) == 2 && lower[0] == "system" && lower[1] == "endpoint": + // 按实际来源判断，不要固定返回 true：硬编码「内网直连」会让客户端 + // 误以为可直连并跳过中转，也是对匿名访问者宣称内网可达。 + local := embyIsLocalRequest(c) + c.JSON(200, gin.H{"IsLocal": local, "IsInNetwork": local}) + return + case len(lower) == 2 && lower[0] == "localization" && lower[1] == "options": + c.JSON(200, []any{}) + return + case len(lower) == 2 && lower[0] == "sessions" && lower[1] == "capabilities": + c.Status(204) + return + case len(lower) == 3 && lower[0] == "sessions" && lower[1] == "capabilities" && lower[2] == "full": + c.Status(204) + return + // 播放进度上报：部分客户端的原生播放器不带 token， + // 这类请求不能 401，否则客户端会中断播放状态跟踪 + case len(lower) >= 2 && lower[0] == "sessions" && lower[1] == "playing": + if user, ok := embyAuth(c); ok { + embySessionPlaying(c, user, strings.Join(lower[2:], "/")) + } else { + c.Status(204) + } + return + // 电影库页面可能请求的推荐/字母索引接口，返回空结果 + case method == "GET" && len(lower) == 2 && lower[0] == "movies" && lower[1] == "recommendations": + c.JSON(200, []any{}) + return + case method == "GET" && len(lower) == 2 && lower[0] == "items" && lower[1] == "prefixes": + c.JSON(200, []any{}) + return + // 添加服务器 / 登录流程必须在鉴权前放行，否则客户端只会看到 Unauthorized + case len(lower) == 2 && lower[0] == "users" && lower[1] == "public": + embyPublicUsers(c) + return + case len(lower) == 2 && lower[0] == "users" && lower[1] == "authenticatebyname": + embyAuthenticateByName(c) + return + // 旧版客户端登录：POST /Users/{userId}/Authenticate，密码在 body 或 query + case len(lower) == 3 && lower[0] == "users" && lower[2] == "authenticate": + embyAuthenticateByID(c, segs[1]) + return + case len(lower) >= 3 && lower[0] == "items" && strings.EqualFold(lower[2], "images"): + // /Items/{id}/Images/Primary 等：图片 URL 不带 token，必须匿名可访问 + embyImage(c, segs[1], segs, true) + return + } + + user, ok := embyAuth(c) + if !ok { + embyUnauthorized(c) + return + } + + switch { + // Emby 客户端用它做会话与实时通知，缺失时会持续重连 + case len(lower) == 1 && lower[0] == "embywebsocket": + embyWebSocket(c) + // 客户端启动/仪表盘探测接口，返回空数据避免 404 弹通用错误 + case method == "GET" && len(lower) == 1 && lower[0] == "sessions": + c.JSON(200, []any{}) + case method == "GET" && len(lower) == 1 && lower[0] == "scheduledtasks": + c.JSON(200, []any{}) + case len(lower) == 3 && lower[0] == "system" && lower[1] == "activitylog" && lower[2] == "entries": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + // LiveTv：会话页/剧集页会查询录像、节目单等，全部返回空结果 + case len(lower) >= 2 && lower[0] == "livetv": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + // Emby 4.10 客户端首页读取用户设置：GET /usersettings/{userId} + case len(lower) == 2 && lower[0] == "usersettings": + if method == "GET" { + c.JSON(200, gin.H{ + "Screensaver": gin.H{}, + "HomeSections": []any{}, + "Playback": gin.H{}, + }) + } else { + c.Status(204) + } + // 客户端登录后会读取/保存界面偏好，字段缺失会让 sectionscontroller 抛异常 + case len(lower) >= 1 && lower[0] == "displaypreferences": + if method == "GET" { + c.JSON(200, gin.H{ + "Id": "usersettings", + "ViewType": "", + "SortBy": "SortName", + "IndexBy": "", + "RememberIndexing": false, + "PrimaryImageHeight": 250, + "PrimaryImageWidth": 250, + "CustomPrefs": gin.H{ + "homeSections": "", + }, + "ScrollDirection": "Horizontal", + "ShowBackdrop": true, + "RememberSorting": false, + "SortOrder": "Ascending", + "ShowSidebar": false, + "Client": "emby", + }) + } else { + c.Status(204) + } + case len(lower) == 2 && lower[0] == "system" && lower[1] == "info": + embySystemInfo(c) + case len(lower) == 2 && lower[0] == "system" && lower[1] == "endpoint": + c.JSON(200, gin.H{"IsLocal": true, "IsInNetwork": true}) + case len(lower) == 1 && lower[0] == "users" && method == "GET": + embyListUsers(c, user) + case len(lower) == 2 && lower[0] == "users" && lower[1] == "query": + embyQueryUsers(c) + case len(lower) == 2 && lower[0] == "users" && method == "GET": + embyGetUser(c, segs[1], user) + case len(lower) == 3 && lower[0] == "users" && lower[2] == "views": + embyViews(c, user) + case len(lower) == 2 && lower[0] == "items" && lower[1] == "counts": + embyItemCounts(c, user) + case len(lower) == 2 && lower[0] == "library" && lower[1] == "refresh" && method == "POST": + embyLibraryRefresh(c) + case len(lower) == 2 && lower[0] == "library" && lower[1] == "selectablemediafolders": + embySelectableFolders(c, user) + case len(lower) == 3 && lower[0] == "library" && lower[1] == "virtualfolders" && lower[2] == "query": + embyVirtualFolders(c, user) + case len(lower) == 2 && lower[0] == "search" && lower[1] == "hints": + embySearchHints(c, user) + // 首页「收藏的艺术家 / 人物」分区查询，无对应数据时返回空结果 + case method == "GET" && len(lower) == 1 && lower[0] == "artists": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + case method == "GET" && len(lower) == 1 && lower[0] == "persons": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + case len(lower) == 2 && lower[0] == "artists": + embyArtists(c, user, lower[1] == "albumartists") + case len(lower) == 3 && lower[0] == "sessions" && lower[1] == "playing": + embySessionPlaying(c, user, lower[2]) + case len(lower) == 2 && lower[0] == "sessions" && lower[1] == "playing" && method == "POST": + embySessionPlaying(c, user, "") + case len(lower) == 2 && lower[0] == "sessions" && lower[1] == "logout" && method == "POST": + embyLogout(c, user) + case len(lower) == 2 && lower[0] == "sessions" && lower[1] == "capabilities": + c.Status(204) + case len(lower) == 3 && lower[0] == "sessions" && lower[1] == "capabilities" && lower[2] == "full": + c.Status(204) + // 媒体库页（电影/剧集/音乐）筛选栏依赖的分面接口，缺失返回 404 会让整页报错 + case method == "GET" && len(lower) == 2 && lower[0] == "items" && (lower[1] == "filters" || lower[1] == "filters2"): + c.JSON(200, gin.H{"Genres": []any{}, "Tags": []any{}, "OfficialRatings": []any{}, "Years": []any{}}) + case method == "GET" && len(lower) == 1 && embyEmptyListRoot[lower[0]]: + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + // 用户态条目 + case len(lower) >= 3 && lower[0] == "users": + embyUserScoped(c, user, segs, lower, method) + // 全局条目 + case len(lower) >= 2 && lower[0] == "items": + embyItemsScoped(c, user, segs, lower, method) + // 播放流 + case len(lower) >= 3 && (lower[0] == "videos" || lower[0] == "audio") && + (lower[2] == "stream" || strings.HasPrefix(lower[2], "stream.") || lower[2] == "universal"): + embyStream(c, user, segs[1], lower[0] == "audio") + case len(lower) == 3 && lower[0] == "audio" && lower[2] == "lyrics": + embyLyrics(c, user, segs[1]) + // 剧集：/Shows/NextUp 只有两段，/Shows/{id}/Seasons 为三段 + case len(lower) >= 2 && lower[0] == "shows": + embyShows(c, user, segs, lower, method) + default: + embyNotFound(c) + } +} + +func embyUnauthorized(c *gin.Context) { + // 记录客户端实际请求，便于排查"连接后被判未认证"的兼容问题。 + // 只能记 Path 不能记 RequestURI：query 里可能带 Pw/Password/api_key， + // 而 X-Emby-Authorization 里是可直接使用的设备令牌，都不能落到日志里。 + utils.Log.Warnf("emby unauthorized: %s %s client=%q ua=%q hasToken=%v", + c.Request.Method, c.Request.URL.Path, c.ClientIP(), + c.GetHeader("User-Agent"), embyRequestToken(c) != "") + c.JSON(401, gin.H{"Error": "Unauthorized", "ResponseStatus": gin.H{"ErrorCode": "Unauthorized"}}) +} + +func embyNotFound(c *gin.Context) { + c.JSON(404, gin.H{"Error": "Not Found"}) +} + +// embyEmptyListRoot 本服务无对应数据、但客户端页面一定会请求的顶层集合接口 +var embyEmptyListRoot = map[string]bool{ + "genres": true, "musicgenres": true, "studios": true, "tags": true, + "officialratings": true, "years": true, "collections": true, + "playlists": true, "trailers": true, "channels": true, "livetv": true, +} + +// embyAuxItem 处理详情页附带请求（相似、预告片、特辑、片头等）。 +// 这类接口缺失时客户端详情页和列表页都会弹"error processing the request"。 +// 返回 true 表示已响应。 +func embyAuxItem(c *gin.Context, name string) bool { + switch name { + case "similar", "suggestions", "instantmix", "intros": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0}) + case "localtrailers", "specialfeatures", "additionalparts", "remoteimages", "thememedia": + c.JSON(200, []any{}) + case "criticreviews": + c.JSON(200, gin.H{"Items": []any{}, "TotalRecordCount": 0}) + default: + return false + } + return true +} + +var embyWSUpgrader = websocket.Upgrader{ + ReadBufferSize: 1024, + WriteBufferSize: 1024, + CheckOrigin: func(r *http.Request) bool { return true }, +} + +// embyWebSocket 提供最小可用的 Emby WebSocket 通道。 +// 客户端用它做会话/实时通知，缺失时会持续重连并产生大量 404。 +func embyWebSocket(c *gin.Context) { + conn, err := embyWSUpgrader.Upgrade(c.Writer, c.Request, nil) + if err != nil { + return + } + defer conn.Close() + conn.SetReadLimit(4096) + _ = conn.SetReadDeadline(time.Now().Add(90 * time.Second)) + conn.SetPongHandler(func(string) error { + return conn.SetReadDeadline(time.Now().Add(90 * time.Second)) + }) + stop := make(chan struct{}) + defer close(stop) + go func() { + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + for { + select { + case <-ticker.C: + if err := conn.WriteMessage(websocket.TextMessage, []byte(`{"MessageType":"KeepAlive"}`)); err != nil { + return + } + case <-stop: + return + } + } + }() + for { + if _, _, err := conn.ReadMessage(); err != nil { + return + } + } +} + +// ==================== 系统 / 用户 ==================== + +func embyPublicSystemInfo(c *gin.Context) { + api := common.GetApiUrl(c.Request.Context()) + c.JSON(200, gin.H{ + "ServerName": setting.GetStr(conf.SiteTitle, "OpenList"), + "Version": embyVersion(), + "Id": embyServerID(), + "OperatingSystem": "Linux", + "LocalAddress": api, + "WebPath": "", + "StartupWizardCompleted": true, + }) +} + +func embySystemInfo(c *gin.Context) { + api := common.GetApiUrl(c.Request.Context()) + c.JSON(200, gin.H{ + "ServerName": setting.GetStr(conf.SiteTitle, "OpenList"), + "Version": embyVersion(), + "Id": embyServerID(), + "OperatingSystem": "Linux", + "OperatingSystemDisplayName": "Linux", + "LocalAddress": api, + "WebPath": "", + "StartupWizardCompleted": true, + "HasPendingRestart": false, + "IsShuttingDown": false, + "SupportsLibraryMonitor": false, + "CanSelfRestart": false, + "CanLaunchWebBrowser": false, + "SupportsHttps": false, + "HasUpdateAvailable": false, + "EncoderLocation": "System", + "SystemArchitecture": "X64", + "SystemUpdateLevel": "Release", + // 不返回真实监听端口：这个端点匿名可访问，客户端本来就是从已知端口连进来的， + // 回了真实端口只是白给一份端口探测信息。 + "HttpServerPortNumber": 0, + "HttpsPortNumber": 0, + "ProgramDataPath": "", + "ItemsByNamePath": "", + "CachePath": "", + "LogPath": "", + "InternalMetadataPath": "", + "TranscodingTempPath": "", + }) +} + +// embyIsLocalRequest 判断请求是否来自本机 / 内网，用于 /System/Endpoint。 +func embyIsLocalRequest(c *gin.Context) bool { + ip := net.ParseIP(c.ClientIP()) + if ip == nil { + return false + } + return ip.IsLoopback() || ip.IsPrivate() +} + +// embyReadAuthBody 读取登录凭据，兼容 JSON / 表单 / query 三种提交方式 +func embyReadAuthBody(c *gin.Context) (username, pw string) { + if strings.Contains(strings.ToLower(c.ContentType()), "json") { + var req struct { + Username string `json:"Username"` + Pw string `json:"Pw"` + Password string `json:"Password"` + } + _ = c.ShouldBindJSON(&req) + username, pw = req.Username, req.Pw + if pw == "" { + pw = req.Password + } + } else { + _ = c.Request.ParseForm() + username = c.PostForm("Username") + pw = c.PostForm("Pw") + if pw == "" { + pw = c.PostForm("Password") + } + } + if username == "" { + username = c.Query("Username") + } + if pw == "" { + pw = c.Query("Pw") + } + if pw == "" { + pw = c.Query("Password") + } + return strings.TrimSpace(username), strings.TrimSpace(pw) +} + +// embyAuthenticateByName 标准登录接口 POST /Users/AuthenticateByName +func embyAuthenticateByName(c *gin.Context) { + username, pw := embyReadAuthBody(c) + // 除 IP 外还要检查用户名维度：IP 可被 X-Forwarded-For 伪造 + if mediaAuthLockedFor(c, username) { + c.JSON(429, gin.H{"Error": model.TooManyAttempts}) + return + } + if username == "" || pw == "" { + embyUnauthorized(c) + return + } + user, err := op.GetUserByName(username) + if err != nil || user.Disabled || user.IsGuest() { + mediaAuthFailedFor(c, username) + embyUnauthorized(c) + return + } + embyCompleteAuth(c, user, pw) +} + +// embyAuthenticateByID 旧版登录接口 POST /Users/{userId}/Authenticate +func embyAuthenticateByID(c *gin.Context, idStr string) { + if mediaAuthLocked(c) { + c.JSON(429, gin.H{"Error": model.TooManyAttempts}) + return + } + id, ok := embyResolveUserID(idStr) + if !ok { + embyUnauthorized(c) + return + } + user, err := op.GetUserById(id) + if err != nil || user.Disabled || user.IsGuest() { + mediaAuthFailed(c) + embyUnauthorized(c) + return + } + // 按 id 登录同样要受用户名维度的锁定约束，否则可以换 id 绕过按用户名的计数 + if mediaAuthLockedFor(c, user.Username) { + c.JSON(429, gin.H{"Error": model.TooManyAttempts}) + return + } + _, pw := embyReadAuthBody(c) + if pw == "" { + embyUnauthorized(c) + return + } + embyCompleteAuth(c, user, pw) +} + +// embyCompleteAuth 校验密码并签发设备令牌。 +// 密码位置既接受 OpenList 登录密码（无 2FA 时），也接受属于该用户的应用密钥。 +func embyCompleteAuth(c *gin.Context, user *model.User, pw string) { + keyUser, _, keyOK := LookUpAppKeyUser(pw) + switch { + case keyOK && keyUser.ID == user.ID: + // 应用密钥认证通过，2FA 账号也允许（密钥本身即第二因素） + case keyOK: + mediaAuthFailedFor(c, user.Username) + embyUnauthorized(c) + return + case user.OtpSecret != "": + // 2FA 账号不允许用登录密码通过 Emby 认证 + mediaAuthFailedFor(c, user.Username) + c.JSON(403, gin.H{"Error": "Accounts with two-factor authentication must use an app key"}) + return + case user.ValidateRawPassword(pw) != nil: + mediaAuthFailedFor(c, user.Username) + embyUnauthorized(c) + return + } + mediaAuthSucceededFor(c, user.Username) + authHeader := embyParseAuthHeader(c.GetHeader("X-Emby-Authorization")) + token, err := media.NewDeviceToken() + if err != nil { + c.JSON(500, gin.H{"Error": err.Error()}) + return + } + dev := &model.EmbyDevice{ + UserID: user.ID, + DeviceID: authHeader["deviceid"], + DeviceName: authHeader["device"], + Client: authHeader["client"], + Version: authHeader["version"], + TokenHash: media.HashToken(token), + LastSeenAt: time.Now(), + } + _ = db.CreateEmbyDevice(dev) + c.JSON(200, gin.H{ + "User": embyUserDtoOf(user), + "AccessToken": token, + "ServerId": embyServerID(), + }) +} + +// embyPublicUsers 匿名可访问，只返回登录界面需要的公开字段， +// 不包含权限、策略、是否管理员等敏感信息。 +func embyPublicUsers(c *gin.Context) { + out := make([]*embyPublicUserDto, 0) + users, _, err := op.GetUsers(1, -1) + if err == nil { + for i := range users { + if users[i].IsGuest() || users[i].Disabled { + continue + } + out = append(out, &embyPublicUserDto{ + Name: users[i].Username, + ServerId: embyServerID(), + Id: embyUserGUID(users[i].ID), + HasPassword: users[i].PwdHash != "", + }) + } + } + c.JSON(200, out) +} + +func embyListUsers(c *gin.Context, current *model.User) { + users, _, err := op.GetUsers(1, -1) + if err != nil { + c.JSON(500, gin.H{"Error": err.Error()}) + return + } + out := make([]*embyUserDto, 0, len(users)) + for i := range users { + if users[i].IsGuest() { + continue + } + if !current.IsAdmin() && users[i].ID != current.ID { + continue + } + out = append(out, embyUserDtoOf(&users[i])) + } + c.JSON(200, out) +} + +func embyQueryUsers(c *gin.Context) { + users, _, err := op.GetUsers(1, -1) + if err != nil { + c.JSON(500, gin.H{"Error": err.Error()}) + return + } + out := make([]*embyUserDto, 0, len(users)) + for i := range users { + if users[i].IsGuest() { + continue + } + out = append(out, embyUserDtoOf(&users[i])) + } + c.JSON(200, gin.H{"Items": out, "TotalRecordCount": len(out), "StartIndex": 0}) +} + +func embyGetUser(c *gin.Context, idStr string, current *model.User) { + // Jellyfin/部分 Emby 客户端使用 /Users/Me 获取当前用户 + if strings.EqualFold(idStr, "me") { + c.JSON(200, embyUserDtoOf(current)) + return + } + id, ok := embyResolveUserID(idStr) + if !ok { + embyNotFound(c) + return + } + user, err := op.GetUserById(id) + if err != nil { + embyNotFound(c) + return + } + c.JSON(200, embyUserDtoOf(user)) +} + +func embyLogout(c *gin.Context, user *model.User) { + token := embyRequestToken(c) + if token != "" { + if dev, err := db.GetEmbyDeviceByTokenHash(media.HashToken(token)); err == nil { + _ = db.DeleteEmbyDevice(user.ID, dev.DeviceID) + } + } + c.Status(204) +} + +// ==================== 媒体库刷新 ==================== + +var ( + embyRefreshMu sync.Mutex + embyRefreshTimer *time.Timer +) + +func scheduleLibraryRefresh() { + embyRefreshMu.Lock() + defer embyRefreshMu.Unlock() + if embyRefreshTimer != nil { + embyRefreshTimer.Stop() + } + embyRefreshTimer = time.AfterFunc(10*time.Second, func() { + for _, mt := range MediaTypeEnabledList() { + cfg, err := db.GetMediaConfig(mt) + if err != nil || cfg == nil { + continue + } + media.ScanMedia(cfg) + } + }) +} + +func embyLibraryRefresh(c *gin.Context) { + scheduleLibraryRefresh() + c.Status(204) +} + +// ==================== 内部工具 ==================== + +func embyPaging(c *gin.Context) (start, limit int) { + start, _ = strconv.Atoi(c.DefaultQuery("StartIndex", "0")) + limit, _ = strconv.Atoi(c.DefaultQuery("Limit", "100")) + if start < 0 { + start = 0 + } + if limit <= 0 || limit > 1000 { + limit = 200 + } + return +} + +func embyBoolQuery(c *gin.Context, key string) bool { + v := strings.ToLower(c.Query(key)) + return v == "true" || v == "1" +} + +func embyIntQuery(c *gin.Context, key string) int { + v, _ := strconv.Atoi(c.Query(key)) + return v +} + +func embyJson(v any) string { + b, err := json.Marshal(v) + if err != nil { + return "" + } + return string(b) +} + +func embyHash(s string) string { + sum := sha1.Sum([]byte(s)) + return hex.EncodeToString(sum[:6]) +} + +// embyGUID 把内部可读 ID 映射为 32 位 hex GUID。 +// embyGUID 用于用户 ID（Emby 规范为 32 位 hex GUID）。 +// 注意：条目的 Item Id 不能用 GUID，见 embyNumericID。 +func embyGUID(s string) string { + sum := md5.Sum([]byte("openlist-emby-id:" + s)) + return hex.EncodeToString(sum[:]) +} + +// embyNumericID 生成稳定的十进制数字 ID，供条目（影片/剧集/专辑等）使用。 +// Emby 真实服务器的 Item Id 就是数字字符串；Emby Theater 的原生 +// LocalItemRepository 会把它当 long 解析（System.Number.ThrowFormatException）， +// 传 GUID 或 mv-1 这类可读 ID 都会让客户端播放/取图直接抛异常。 +func embyNumericID(s string) string { + h := fnv.New64a() + _, _ = h.Write([]byte("openlist-emby-item:" + s)) + // 取 47 位：既在 .NET long / JS 安全整数范围内，又足够大避免与真实 ID 混淆 + return strconv.FormatUint(h.Sum64()&0x7FFFFFFFFFFF, 10) +} + +func embyImageTag(item *model.MediaItem) string { + if item.Cover == "" { + return "" + } + return embyHash(item.Cover + item.UpdatedAt.String()) +} diff --git a/server/handles/emby_browse.go b/server/handles/emby_browse.go new file mode 100644 index 00000000..6473a329 --- /dev/null +++ b/server/handles/emby_browse.go @@ -0,0 +1,1654 @@ +package handles + +import ( + "encoding/json" + "fmt" + stdpath "path" + "sort" + "strconv" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/gin-gonic/gin" +) + +// ==================== 媒体库索引 ==================== + +type embyLibrary struct { + user *model.User + videos []model.MediaItem + music []model.MediaItem + + userData map[[2]uint]*model.MediaUserData + // ids 保存 GUID -> 内部可读 ID 的映射，客户端只会回传 GUID + ids map[string]string +} + +func loadEmbyLibrary(user *model.User) *embyLibrary { + l := &embyLibrary{ + user: user, + userData: map[[2]uint]*model.MediaUserData{}, + ids: map[string]string{}, + } + if MediaTypeEnabled(model.MediaTypeVideo) { + items, err := db.ListAllMediaItems(model.MediaTypeVideo) + if err == nil { + for i := range items { + if CanUserAccessItem(user, &items[i]) { + l.videos = append(l.videos, items[i]) + } + } + } + } + if MediaTypeEnabled(model.MediaTypeMusic) { + items, err := db.ListAllMediaItems(model.MediaTypeMusic) + if err == nil { + for i := range items { + if CanUserAccessItem(user, &items[i]) { + l.music = append(l.music, items[i]) + } + } + } + } + ids := make([]uint, 0, len(l.videos)+len(l.music)) + for _, it := range l.videos { + ids = append(ids, it.ID) + } + for _, it := range l.music { + ids = append(ids, it.ID) + } + if user != nil && len(ids) > 0 { + if list, err := db.ListMediaUserDataByItems(user.ID, ids); err == nil { + for i := range list { + d := list[i] + l.userData[[2]uint{d.ItemID, uint(d.EpisodeIndex)}] = &d + } + } + } + l.registerIDs() + return l +} + +// guid 返回内部 ID 对应的对外数字 ID，并登记反查映射。 +// 名称保留 guid 是为了少改调用点，实际输出的是纯数字（Emby Item Id 规范）。 +func (l *embyLibrary) guid(orig string) string { + g := embyNumericID(orig) + if l.ids != nil { + l.ids[g] = orig + // 兼容旧版本缓存下来的 GUID 形式 ID：客户端没清缓存时仍能解析， + // 否则会一直拿旧 ID 请求并得到 {"Error":"Not Found"} + l.ids[embyGUID(orig)] = orig + } + return g +} + +// origID 把客户端回传的数字 ID 还原为内部 ID；已是内部 ID 时原样返回 +func (l *embyLibrary) origID(id string) string { + id = strings.TrimSpace(id) + if l.ids != nil { + if orig, ok := l.ids[id]; ok { + return orig + } + } + return id +} + +// registerIDs 预先登记全部条目的对外 ID，保证任意请求都能反查出内部 ID +func (l *embyLibrary) registerIDs() { + register := func(orig string) { + l.ids[embyNumericID(orig)] = orig + l.ids[embyGUID(orig)] = orig + } + register("root") + register("lib-movie") + register("lib-tv") + register("lib-music") + for i := range l.videos { + register(fmt.Sprintf("mv-%d", l.videos[i].ID)) + } + for _, s := range l.seriesList() { + register("sr-" + s.Key) + for _, season := range l.seasonList(s) { + register(fmt.Sprintf("sn-%s-%d", s.Key, season)) + } + for _, ep := range l.episodesOf(s.Key, 0) { + register(ep.ID) + } + } + for i := range l.music { + it := &l.music[i] + register(fmt.Sprintf("so-%d", it.ID)) + register("al-" + albumKeyOf(it)) + register("ar-" + artistKeyOf(it)) + } +} + +func (l *embyLibrary) userDataOf(itemID uint, episodeIndex int) *embyUserData { + if l == nil || l.user == nil { + return &embyUserData{} + } + d, ok := l.userData[[2]uint{itemID, uint(episodeIndex)}] + if !ok { + return &embyUserData{} + } + return &embyUserData{ + PlaybackPositionTicks: d.PositionTicks, + PlayCount: d.PlayCount, + IsFavorite: d.IsFavorite, + Played: d.Played, + } +} + +type embySeries struct { + Key string + Name string + Items []model.MediaItem + Year int + Overview string + Cover string +} + +// isEmbySeriesItem 判断视频条目是否属于剧集。 +// 电视剧按目录扫描（IsFolder=true，带 Episodes 选集），或被刮削为 tv； +// 其余单文件视频一律按电影处理。仅看 video_type 会把目录型剧集误判为电影。 +func isEmbySeriesItem(it *model.MediaItem) bool { + return it.IsFolder || it.VideoType == "tv" +} + +func seriesKeyOf(item *model.MediaItem) string { + name := item.ScrapedName + if name == "" { + name = strings.TrimSuffix(stdpath.Base(mediaFolderOf(item)), "/") + } + return embyHash("series:" + strings.ToLower(name)) +} + +func seriesNameOf(item *model.MediaItem) string { + if item.ScrapedName != "" { + return item.ScrapedName + } + if item.IsFolder { + return strings.TrimSuffix(item.FileName, stdpath.Ext(item.FileName)) + } + return strings.TrimSuffix(stdpath.Base(mediaFolderOf(item)), "/") +} + +// seriesList 汇总剧集（按剧名去重） +func (l *embyLibrary) seriesList() []*embySeries { + index := map[string]*embySeries{} + var order []string + for i := range l.videos { + it := &l.videos[i] + if !isEmbySeriesItem(it) { + continue + } + key := seriesKeyOf(it) + s, ok := index[key] + if !ok { + s = &embySeries{Key: key, Name: seriesNameOf(it), Cover: it.Cover} + if len(it.ReleaseDate) >= 4 { + s.Year, _ = strconv.Atoi(it.ReleaseDate[:4]) + } + s.Overview = it.Plot + index[key] = s + order = append(order, key) + } + if s.Cover == "" { + s.Cover = it.Cover + } + if s.Overview == "" { + s.Overview = it.Plot + } + s.Items = append(s.Items, *it) + } + out := make([]*embySeries, 0, len(order)) + for _, k := range order { + out = append(out, index[k]) + } + return out +} + +func (l *embyLibrary) seriesByKey(key string) *embySeries { + for _, s := range l.seriesList() { + if s.Key == key { + return s + } + } + return nil +} + +type embyEpisode struct { + ID string + Item *model.MediaItem + Episode model.EpisodeInfo + Index int + Season int + SeriesKey string + SeriesName string +} + +func parseEpisodeList(item *model.MediaItem) []model.EpisodeInfo { + if item.Episodes == "" { + return nil + } + var eps []model.EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil { + return nil + } + return eps +} + +// episodesOf 返回某部剧（可按季过滤）的集列表 +func (l *embyLibrary) episodesOf(seriesKey string, season int) []*embyEpisode { + var out []*embyEpisode + for i := range l.videos { + it := &l.videos[i] + if !isEmbySeriesItem(it) || seriesKeyOf(it) != seriesKey { + continue + } + s := it.Season + if s <= 0 { + s = 1 + } + if season > 0 && s != season { + continue + } + if it.IsFolder { + for idx, ep := range parseEpisodeList(it) { + num := ep.Index + if num <= 0 { + num = idx + 1 + } + out = append(out, &embyEpisode{ + ID: fmt.Sprintf("ep-%d-%d", it.ID, num), + Item: it, + Episode: ep, + Index: num, + Season: s, + SeriesKey: seriesKey, + SeriesName: seriesNameOf(it), + }) + } + continue + } + num := it.Episode + if num <= 0 { + num = 1 + } + out = append(out, &embyEpisode{ + ID: fmt.Sprintf("ep-%d-%d", it.ID, num), + Item: it, + Index: num, + Season: s, + SeriesKey: seriesKey, + SeriesName: seriesNameOf(it), + }) + } + sort.Slice(out, func(i, j int) bool { + if out[i].Season != out[j].Season { + return out[i].Season < out[j].Season + } + return out[i].Index < out[j].Index + }) + return out +} + +func (l *embyLibrary) findEpisode(id string) *embyEpisode { + // id: ep-- + parts := strings.Split(id, "-") + if len(parts) != 3 { + return nil + } + itemID, err := strconv.ParseUint(parts[1], 10, 64) + if err != nil { + return nil + } + num, _ := strconv.Atoi(parts[2]) + item, err := db.GetMediaItemByID(uint(itemID)) + if err != nil || !CanUserAccessItem(l.user, item) { + return nil + } + season := item.Season + if season <= 0 { + season = 1 + } + if item.IsFolder { + for i, ep := range parseEpisodeList(item) { + n := ep.Index + if n <= 0 { + n = i + 1 + } + if n == num { + return &embyEpisode{ID: id, Item: item, Episode: ep, Index: n, Season: season, + SeriesKey: seriesKeyOf(item), SeriesName: seriesNameOf(item)} + } + } + return nil + } + return &embyEpisode{ID: id, Item: item, Index: num, Season: season, + SeriesKey: seriesKeyOf(item), SeriesName: seriesNameOf(item)} +} + +// music 索引 + +type embyAlbum struct { + Key string + Name string + Artist string + Cover string + Year int + Items []model.MediaItem +} + +func albumArtistOf(item *model.MediaItem) string { + if item.AlbumArtist != "" { + return item.AlbumArtist + } + if item.Authors != "" { + return item.Authors + } + return "Unknown Artist" +} + +func albumKeyOf(item *model.MediaItem) string { + album := item.AlbumName + if album == "" { + album = "Unknown Album" + } + return embyHash("album:" + strings.ToLower(albumArtistOf(item)) + "\x00" + strings.ToLower(album)) +} + +func artistKeyOf(item *model.MediaItem) string { + return embyHash("artist:" + strings.ToLower(albumArtistOf(item))) +} + +func (l *embyLibrary) albums() []*embyAlbum { + index := map[string]*embyAlbum{} + var order []string + for i := range l.music { + it := &l.music[i] + key := albumKeyOf(it) + al, ok := index[key] + if !ok { + name := it.AlbumName + if name == "" { + name = "Unknown Album" + } + al = &embyAlbum{Key: key, Name: name, Artist: albumArtistOf(it), Cover: it.Cover} + if len(it.ReleaseDate) >= 4 { + al.Year, _ = strconv.Atoi(it.ReleaseDate[:4]) + } + index[key] = al + order = append(order, key) + } + if al.Cover == "" { + al.Cover = it.Cover + } + al.Items = append(al.Items, *it) + } + out := make([]*embyAlbum, 0, len(order)) + for _, k := range order { + out = append(out, index[k]) + } + sort.Slice(out, func(i, j int) bool { return strings.ToLower(out[i].Name) < strings.ToLower(out[j].Name) }) + return out +} + +func (l *embyLibrary) albumByKey(key string) *embyAlbum { + for _, a := range l.albums() { + if a.Key == key { + return a + } + } + return nil +} + +// ==================== DTO 构造 ==================== + +func (l *embyLibrary) movieItem(it *model.MediaItem) *embyItem { + year := 0 + if len(it.ReleaseDate) >= 4 { + year, _ = strconv.Atoi(it.ReleaseDate[:4]) + } + name := it.ScrapedName + if name == "" { + name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName)) + } + e := &embyItem{ + Name: name, + SortName: name, + ServerId: embyServerID(), + Id: l.guid(fmt.Sprintf("mv-%d", it.ID)), + ParentId: l.guid("lib-movie"), + Type: "Movie", + MediaType: "Video", + IsFolder: false, + Path: MediaItemFilePath(it), + Container: embyExt(it.FileName), + RunTimeTicks: embyTicks(it.Duration), + ProductionYear: year, + Overview: it.Plot, + CommunityRating: it.Rating, + PremiereDate: it.ReleaseDate, + DateCreated: embyTimeStr(it.CreatedAt), + UserData: l.userDataOf(it.ID, 0), + ImageTags: map[string]string{}, + ProviderIds: map[string]string{}, + LocationType: "FileSystem", + } + if tag := embyImageTag(it); tag != "" { + e.ImageTags["Primary"] = tag + } + if it.ExternalID != "" { + e.ProviderIds["Tmdb"] = it.ExternalID + } + return e +} + +func (l *embyLibrary) seriesItem(s *embySeries) *embyItem { + e := &embyItem{ + Name: s.Name, + SortName: s.Name, + ServerId: embyServerID(), + Id: l.guid("sr-" + s.Key), + ParentId: l.guid("lib-tv"), + Type: "Series", + IsFolder: true, + ProductionYear: s.Year, + Overview: s.Overview, + DateCreated: embyTimeStr(s.Items[0].CreatedAt), + ChildCount: l.seasonCount(s), + LocationType: "FileSystem", + ImageTags: map[string]string{}, + ProviderIds: map[string]string{}, + UserData: &embyUserData{}, + } + if tag := embyImageTag(&s.Items[0]); tag != "" { + e.ImageTags["Primary"] = tag + } + if s.Items[0].ExternalID != "" { + e.ProviderIds["Tmdb"] = s.Items[0].ExternalID + } + return e +} + +func (l *embyLibrary) seasonCount(s *embySeries) int { + seasons := map[int]bool{} + for i := range s.Items { + season := s.Items[i].Season + if season <= 0 { + season = 1 + } + seasons[season] = true + } + return len(seasons) +} + +func (l *embyLibrary) seasonList(s *embySeries) []int { + set := map[int]bool{} + for i := range s.Items { + season := s.Items[i].Season + if season <= 0 { + season = 1 + } + set[season] = true + } + out := make([]int, 0, len(set)) + for n := range set { + out = append(out, n) + } + sort.Ints(out) + return out +} + +func (l *embyLibrary) seasonItem(s *embySeries, season int) *embyItem { + eps := l.episodesOf(s.Key, season) + unplayed := 0 + for _, ep := range eps { + if !l.userDataOf(ep.Item.ID, ep.Index).Played { + unplayed++ + } + } + e := &embyItem{ + Name: fmt.Sprintf("Season %d", season), + SortName: fmt.Sprintf("Season %02d", season), + ServerId: embyServerID(), + Id: l.guid(fmt.Sprintf("sn-%s-%d", s.Key, season)), + Type: "Season", + IsFolder: true, + ParentId: l.guid("sr-" + s.Key), + SeriesId: l.guid("sr-" + s.Key), + SeriesName: s.Name, + IndexNumber: season, + ChildCount: len(eps), + LocationType: "FileSystem", + ImageTags: map[string]string{}, + UserData: &embyUserData{UnplayedItemCount: unplayed}, + } + if len(s.Items) > 0 { + if tag := embyImageTag(&s.Items[0]); tag != "" { + e.ImageTags["Primary"] = tag + } + } + return e +} + +func (l *embyLibrary) episodeItem(ep *embyEpisode) *embyItem { + title := ep.Episode.Title + if title == "" { + title = fmt.Sprintf("Episode %d", ep.Index) + } + seriesID := "sr-" + ep.SeriesKey + seasonID := fmt.Sprintf("sn-%s-%d", ep.SeriesKey, ep.Season) + rawPath := MediaItemFilePath(ep.Item) + if ep.Episode.FileName != "" { + rawPath = stdpath.Join(mediaFolderOf(ep.Item), ep.Item.FileName, ep.Episode.FileName) + } + e := &embyItem{ + Name: title, + SortName: fmt.Sprintf("%02d-%03d", ep.Season, ep.Index), + ServerId: embyServerID(), + Id: l.guid(ep.ID), + Type: "Episode", + MediaType: "Video", + IsFolder: false, + ParentId: l.guid(seasonID), + SeriesId: l.guid(seriesID), + SeriesName: ep.SeriesName, + SeasonId: l.guid(seasonID), + SeasonName: fmt.Sprintf("Season %d", ep.Season), + IndexNumber: ep.Index, + ParentIndexNumber: ep.Season, + Path: rawPath, + Container: embyExt(rawPath), + RunTimeTicks: embyTicks(ep.Item.Duration), + Overview: ep.Item.Plot, + DateCreated: embyTimeStr(ep.Item.CreatedAt), + UserData: l.userDataOf(ep.Item.ID, ep.Index), + ImageTags: map[string]string{}, + ProviderIds: map[string]string{}, + LocationType: "FileSystem", + } + if tag := embyImageTag(ep.Item); tag != "" { + e.ImageTags["Primary"] = tag + } + return e +} + +func (l *embyLibrary) albumItem(al *embyAlbum) *embyItem { + e := &embyItem{ + Name: al.Name, + SortName: al.Name, + ServerId: embyServerID(), + Id: l.guid("al-" + al.Key), + ParentId: l.guid("lib-music"), + Type: "MusicAlbum", + IsFolder: true, + AlbumArtist: al.Artist, + AlbumArtists: []embyNameID{{Name: al.Artist, Id: l.guid("ar-" + artistKeyOf(&al.Items[0]))}}, + ProductionYear: al.Year, + ChildCount: len(al.Items), + DateCreated: embyTimeStr(al.Items[0].CreatedAt), + LocationType: "FileSystem", + ImageTags: map[string]string{}, + UserData: &embyUserData{}, + } + if tag := embyImageTag(&al.Items[0]); tag != "" { + e.ImageTags["Primary"] = tag + } + return e +} + +func (l *embyLibrary) songItem(it *model.MediaItem) *embyItem { + name := it.ScrapedName + if name == "" { + name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName)) + } + albumName := it.AlbumName + if albumName == "" { + albumName = "Unknown Album" + } + artist := albumArtistOf(it) + albumKey := albumKeyOf(it) + artistKey := artistKeyOf(it) + e := &embyItem{ + Name: name, + SortName: name, + ServerId: embyServerID(), + Id: l.guid(fmt.Sprintf("so-%d", it.ID)), + Type: "Audio", + MediaType: "Audio", + IsFolder: false, + ParentId: l.guid("al-" + albumKey), + Album: albumName, + AlbumId: l.guid("al-" + albumKey), + AlbumArtist: artist, + AlbumArtists: []embyNameID{{Name: artist, Id: l.guid("ar-" + artistKey)}}, + Artists: []string{artist}, + ArtistItems: []embyNameID{{Name: artist, Id: l.guid("ar-" + artistKey)}}, + IndexNumber: it.TrackNumber, + RunTimeTicks: embyTicks(it.Duration), + Path: MediaItemFilePath(it), + Container: embyExt(it.FileName), + DateCreated: embyTimeStr(it.CreatedAt), + CommunityRating: it.Rating, + UserData: l.userDataOf(it.ID, 0), + ImageTags: map[string]string{}, + ProviderIds: map[string]string{}, + LocationType: "FileSystem", + } + if tag := embyImageTag(it); tag != "" { + e.ImageTags["Primary"] = tag + } + return e +} + +func (l *embyLibrary) songByID(id string) (*model.MediaItem, bool) { + id = l.origID(id) + n, err := strconv.ParseUint(strings.TrimPrefix(id, "so-"), 10, 64) + if err != nil { + return nil, false + } + for i := range l.music { + if l.music[i].ID == uint(n) { + return &l.music[i], true + } + } + item, err := db.GetMediaItemByID(uint(n)) + if err != nil || !CanUserAccessItem(l.user, item) { + return nil, false + } + return item, true +} + +// ==================== 视图 / 列表 ==================== + +func (l *embyLibrary) viewItems() []*embyItem { + movieCount, seriesCount := 0, 0 + for i := range l.videos { + if isEmbySeriesItem(&l.videos[i]) { + continue + } + movieCount++ + } + seriesCount = len(l.seriesList()) + albumCount := len(l.albums()) + + var out []*embyItem + if MediaTypeEnabled(model.MediaTypeVideo) { + out = append(out, &embyItem{ + Name: "Movies", SortName: "Movies", ServerId: embyServerID(), + Id: l.guid("lib-movie"), ParentId: l.guid("root"), Type: "CollectionFolder", IsFolder: true, + CollectionType: "movies", ViewType: "movies", LocationType: "FileSystem", + ChildCount: movieCount, RecursiveItemCount: movieCount, + // 只声明客户端能正常处理的子视图，避免生成 OpenList 没有实现的 tab + Subviews: []string{"movies"}, + ImageTags: map[string]string{}, UserData: &embyUserData{}, + }) + out = append(out, &embyItem{ + Name: "TV Shows", SortName: "TV Shows", ServerId: embyServerID(), + Id: l.guid("lib-tv"), ParentId: l.guid("root"), Type: "CollectionFolder", IsFolder: true, + CollectionType: "tvshows", ViewType: "tvshows", LocationType: "FileSystem", + ChildCount: seriesCount, RecursiveItemCount: seriesCount, + ImageTags: map[string]string{}, UserData: &embyUserData{}, + }) + } + if MediaTypeEnabled(model.MediaTypeMusic) { + out = append(out, &embyItem{ + Name: "Music", SortName: "Music", ServerId: embyServerID(), + Id: l.guid("lib-music"), ParentId: l.guid("root"), Type: "CollectionFolder", IsFolder: true, + CollectionType: "music", ViewType: "music", LocationType: "FileSystem", + ChildCount: albumCount, RecursiveItemCount: albumCount, + ImageTags: map[string]string{}, UserData: &embyUserData{}, + }) + } + return out +} + +func embyViews(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + items := lib.viewItems() + c.JSON(200, gin.H{"Items": items, "TotalRecordCount": len(items), "StartIndex": 0}) +} + +// embyHomeSections 返回服务端首页分区配置（Emby 4.10+ 客户端直接使用）。 +// ListOptions.fields 必须存在：客户端会无条件调用 fields.includes()。 +func embyHomeSections() []gin.H { + return []gin.H{ + { + "Id": "userviews", + "SectionType": "userviews", + "Name": "Media", + "CollectionType": "", + "ViewType": "cards", + "ScrollDirection": "Horizontal", + "LimitItems": false, + "ListOptions": gin.H{ + "fields": []string{}, + "shape": "auto", + "context": "home", + "preferThumb": false, + "preferSeriesImage": false, + "showChildCountIndicator": false, + "image": nil, + "sideFooter": nil, + "smallSideFooter": nil, + "showCurrentProgramImage": false, + "showAirDateTime": false, + "showAirEndTime": false, + "defaultShape": nil, + "action": nil, + "multiSelect": false, + "albumFirst": false, + "lines": 1, + "focusTransformTitleAdjust": true, + "contextMenu": true, + "hoverMenu": true, + "hoverPlayButton": false, + }, + "QueryOptions": gin.H{}, + }, + } +} + +// embySectionItems 返回首页某个分区的条目 +func embySectionItems(user *model.User, sectionID string) gin.H { + if sectionID == "userviews" { + lib := loadEmbyLibrary(user) + items := lib.viewItems() + return gin.H{"Items": items, "TotalRecordCount": len(items), "StartIndex": 0} + } + return gin.H{"Items": []any{}, "TotalRecordCount": 0, "StartIndex": 0} +} + +func (l *embyLibrary) rootItem() *embyItem { + return &embyItem{ + Name: "Media", SortName: "Media", ServerId: embyServerID(), + Id: l.guid("root"), Type: "CollectionFolder", IsFolder: true, + LocationType: "FileSystem", ImageTags: map[string]string{}, UserData: &embyUserData{}, + } +} + +// childrenOf 返回指定父级下的子项 +func (l *embyLibrary) childrenOf(parentID string) []*embyItem { + parentID = l.origID(parentID) + switch { + case parentID == "" || parentID == "root": + return l.viewItems() + case parentID == "lib-movie": + var out []*embyItem + for i := range l.videos { + if !isEmbySeriesItem(&l.videos[i]) { + out = append(out, l.movieItem(&l.videos[i])) + } + } + return out + case parentID == "lib-tv": + var out []*embyItem + for _, s := range l.seriesList() { + out = append(out, l.seriesItem(s)) + } + return out + case parentID == "lib-music": + var out []*embyItem + for _, a := range l.albums() { + out = append(out, l.albumItem(a)) + } + return out + } + if strings.HasPrefix(parentID, "sr-") { + key := strings.TrimPrefix(parentID, "sr-") + s := l.seriesByKey(key) + if s == nil { + return nil + } + var out []*embyItem + for _, season := range l.seasonList(s) { + out = append(out, l.seasonItem(s, season)) + } + return out + } + if strings.HasPrefix(parentID, "sn-") { + // sn-- + rest := strings.TrimPrefix(parentID, "sn-") + idx := strings.LastIndex(rest, "-") + if idx < 0 { + return nil + } + key := rest[:idx] + season, _ := strconv.Atoi(rest[idx+1:]) + var out []*embyItem + for _, ep := range l.episodesOf(key, season) { + out = append(out, l.episodeItem(ep)) + } + return out + } + if strings.HasPrefix(parentID, "al-") { + al := l.albumByKey(strings.TrimPrefix(parentID, "al-")) + if al == nil { + return nil + } + var out []*embyItem + for i := range al.Items { + out = append(out, l.songItem(&al.Items[i])) + } + return out + } + if strings.HasPrefix(parentID, "ar-") { + key := strings.TrimPrefix(parentID, "ar-") + var out []*embyItem + for _, al := range l.albums() { + if artistKeyOf(&al.Items[0]) == key { + out = append(out, l.albumItem(al)) + } + } + return out + } + return nil +} + +// allItems 返回媒体库全部条目（用于搜索/递归） +func (l *embyLibrary) allItems() []*embyItem { + var out []*embyItem + for i := range l.videos { + if isEmbySeriesItem(&l.videos[i]) { + continue + } + out = append(out, l.movieItem(&l.videos[i])) + } + for _, s := range l.seriesList() { + out = append(out, l.seriesItem(s)) + } + for i := range l.music { + out = append(out, l.songItem(&l.music[i])) + } + return out +} + +// embySplitCSV 解析 Emby 的逗号分隔参数，同时兼容 | 分隔 +func embySplitCSV(v string) []string { + var out []string + for _, p := range strings.FieldsFunc(v, func(r rune) bool { return r == ',' || r == '|' }) { + if p = strings.TrimSpace(p); p != "" { + out = append(out, p) + } + } + return out +} + +func embyInList(list []string, v string) bool { + for _, s := range list { + if strings.EqualFold(s, v) { + return true + } + } + return false +} + +// filterEmbyItems 实现 /Items 查询的通用过滤，覆盖 openemby_tv / Theater / Tsukimi 实际会发送的参数： +// IncludeItemTypes、ExcludeItemTypes、MediaTypes、Ids、IsFolder、SearchTerm、NameStartsWith、Filters +// allItemsFor 按请求的 IncludeItemTypes 只构造需要的类型。 +// 客户端无 ParentId 的递归查询（如 Type=Episode / MusicAlbum）必须能返回对应类型， +// 且不应把其他媒体类型混进来。 +func (l *embyLibrary) allItemsFor(include []string) []*embyItem { + if len(include) == 0 { + return l.allItems() + } + want := func(t string) bool { return embyInList(include, t) } + var out []*embyItem + if want("Movie") { + for i := range l.videos { + if !isEmbySeriesItem(&l.videos[i]) { + out = append(out, l.movieItem(&l.videos[i])) + } + } + } + if want("Series") || want("Season") || want("Episode") { + for _, s := range l.seriesList() { + if want("Series") { + out = append(out, l.seriesItem(s)) + } + if want("Season") { + for _, season := range l.seasonList(s) { + out = append(out, l.seasonItem(s, season)) + } + } + if want("Episode") { + for _, ep := range l.episodesOf(s.Key, 0) { + out = append(out, l.episodeItem(ep)) + } + } + } + } + if want("Audio") { + for i := range l.music { + out = append(out, l.songItem(&l.music[i])) + } + } + if want("MusicAlbum") { + for _, a := range l.albums() { + out = append(out, l.albumItem(a)) + } + } + return out +} + +func filterEmbyItems(items []*embyItem, c *gin.Context) []*embyItem { + include := embySplitCSV(c.Query("IncludeItemTypes")) + exclude := embySplitCSV(c.Query("ExcludeItemTypes")) + mediaTypes := embySplitCSV(c.Query("MediaTypes")) + searchTerm := strings.ToLower(strings.TrimSpace(c.Query("SearchTerm"))) + namePrefix := strings.ToLower(strings.TrimSpace(c.Query("NameStartsWith"))) + filters := strings.ToLower(c.Query("Filters")) + isFolder := c.Query("IsFolder") + idSet := map[string]bool{} + for _, id := range embySplitCSV(c.Query("Ids")) { + idSet[strings.ToLower(id)] = true + } + + var out []*embyItem + for _, it := range items { + if len(include) > 0 && !embyInList(include, it.Type) { + continue + } + if len(exclude) > 0 && embyInList(exclude, it.Type) { + continue + } + if len(mediaTypes) > 0 && !embyInList(mediaTypes, it.MediaType) { + continue + } + if len(idSet) > 0 && !idSet[strings.ToLower(it.Id)] { + continue + } + lname := strings.ToLower(it.Name) + if searchTerm != "" && !strings.Contains(lname, searchTerm) { + continue + } + if namePrefix != "" && !strings.HasPrefix(lname, namePrefix) { + continue + } + if isFolder != "" { + want := strings.EqualFold(isFolder, "true") + if it.IsFolder != want { + continue + } + } + if strings.Contains(filters, "isfavorite") { + if it.UserData == nil || !it.UserData.IsFavorite { + continue + } + } + if strings.Contains(filters, "isunplayed") { + if it.UserData != nil && it.UserData.Played { + continue + } + } else if strings.Contains(filters, "isplayed") { + if it.UserData == nil || !it.UserData.Played { + continue + } + } + out = append(out, it) + } + return out +} + +func embyTypeMatches(itemType, wanted string) bool { + return embyInList(embySplitCSV(wanted), itemType) +} + +// sortEmbyItems 支持多个排序键（SortBy=SeriesSortName,ParentIndexNumber,IndexNumber,SortName） +func sortEmbyItems(items []*embyItem, sortBy, order string) { + if sortBy == "" { + return + } + keys := embySplitCSV(sortBy) + desc := strings.EqualFold(order, "Descending") + if len(keys) > 0 && strings.EqualFold(keys[0], "random") { + for i := len(items) - 1; i > 0; i-- { + j := int(timeNowUnixNano() % int64(i+1)) + items[i], items[j] = items[j], items[i] + } + return + } + cmp := func(a, b *embyItem, key string) int { + switch strings.ToLower(key) { + case "sortname", "name", "seriessortname": + la, lb := strings.ToLower(a.SortName), strings.ToLower(b.SortName) + if strings.EqualFold(key, "seriessortname") { + la, lb = strings.ToLower(a.SeriesName), strings.ToLower(b.SeriesName) + } + switch { + case la < lb: + return -1 + case la > lb: + return 1 + } + case "datecreated", "dateplayed": + switch { + case a.DateCreated < b.DateCreated: + return -1 + case a.DateCreated > b.DateCreated: + return 1 + } + case "productionyear": + return a.ProductionYear - b.ProductionYear + case "parentindexnumber": + return a.ParentIndexNumber - b.ParentIndexNumber + case "indexnumber": + return a.IndexNumber - b.IndexNumber + case "communityrating": + switch { + case a.CommunityRating < b.CommunityRating: + return -1 + case a.CommunityRating > b.CommunityRating: + return 1 + } + } + return 0 + } + sort.SliceStable(items, func(i, j int) bool { + for _, k := range keys { + if r := cmp(items[i], items[j], k); r != 0 { + if desc { + return r > 0 + } + return r < 0 + } + } + return false + }) +} + +func embyPaginate(items []*embyItem, start, limit int) []*embyItem { + if start >= len(items) { + return []*embyItem{} + } + end := start + limit + if end > len(items) { + end = len(items) + } + return items[start:end] +} + +func timeNowUnixNano() int64 { + return time.Now().UnixNano() +} + +// ==================== 条目分发 ==================== + +// embyUserScoped 处理 /Users/{uid}/... 下的条目相关请求 +func embyUserScoped(c *gin.Context, user *model.User, segs, lower []string, method string) { + // 服务端首页分区（Emby 4.10+ 客户端） + if len(lower) >= 3 && lower[2] == "homesections" { + if method == "GET" && len(lower) == 3 { + c.JSON(200, embyHomeSections()) + } else { + c.Status(204) + } + return + } + // 首页分区条目：/Users/{uid}/Sections/{sectionId}/Items + if len(lower) == 5 && lower[2] == "sections" && lower[4] == "items" { + c.JSON(200, embySectionItems(user, lower[3])) + return + } + // /users/{uid}/items... + if len(lower) >= 3 && lower[2] == "items" { + lib := loadEmbyLibrary(user) + switch { + case len(lower) == 3 && (method == "GET" || method == "POST"): + // /Users/{uid}/Items?ParentId=...&Recursive=...&IncludeItemTypes=... + // 必须按查询条件返回真实数据；Playlist/BoxSet 等无对应类型会被过滤为空 + embyItemList(c, lib, c.Query("ParentId")) + case len(lower) == 4 && lower[3] == "root": + c.JSON(200, lib.rootItem()) + case len(lower) == 4 && lower[3] == "latest": + embyLatest(c, lib) + case len(lower) == 4 && lower[3] == "resume": + embyResume(c, lib) + case len(lower) == 4 && method == "GET": + // 单条目 + embyGetItem(c, lib, segs[3]) + case len(lower) == 5 && lower[4] == "children": + embyChildren(c, lib, segs[3]) + case len(lower) == 5 && lower[4] == "playbackinfo": + embyPlaybackInfo(c, lib, segs[3]) + case len(lower) == 5 && lower[4] == "userdata": + embyUpdateUserData(c, user, segs[3]) + case len(lower) == 4: + embyItemList(c, lib, segs[3]) + case len(lower) == 5 && embyAuxItem(c, lower[4]): + default: + embyNotFound(c) + } + return + } + if len(lower) == 3 && embyAuxItem(c, lower[2]) { + return + } + // /users/{uid}/settings：客户端保存/读取首页与播放设置 + if len(lower) == 3 && lower[2] == "settings" { + if method == "GET" { + c.JSON(200, gin.H{"HomeSections": []any{}}) + } else { + c.Status(204) + } + return + } + // /users/{uid}/views 已在外部处理 + if len(lower) >= 3 && (lower[2] == "playeditems" || lower[2] == "favoriteitems" || lower[2] == "playingitems") { + if len(lower) < 4 { + embyNotFound(c) + return + } + kind := lower[2] + id := segs[3] + action := "" + if len(lower) == 5 { + action = lower[4] + } + switch kind { + case "playeditems": + played := method == "POST" && action != "delete" + embySetPlayed(c, user, id, played) + case "favoriteitems": + fav := method == "POST" && action != "delete" + embySetFavorite(c, user, id, fav) + case "playingitems": + if len(lower) == 5 && lower[4] == "progress" { + embySaveProgress(c, user, id, false) + return + } + embySaveProgress(c, user, id, true) + } + return + } + embyNotFound(c) +} + +// embyItemsScoped 处理 /Items/... 全局请求 +func embyItemsScoped(c *gin.Context, user *model.User, segs, lower []string, method string) { + lib := loadEmbyLibrary(user) + switch { + case len(lower) == 2 && lower[1] == "counts": + embyItemCounts(c, user) + case len(lower) == 2 && method == "GET": + embyGetItem(c, lib, segs[1]) + case len(lower) == 3 && lower[2] == "children": + embyChildren(c, lib, segs[1]) + case len(lower) == 3 && lower[2] == "playbackinfo": + embyPlaybackInfo(c, lib, segs[1]) + case len(lower) == 3 && lower[2] == "refresh" && method == "POST": + embyRefreshItem(c, lib.origID(segs[1])) + // 歌词：/Items/{id}/{mediaSourceId}/Subtitles/{index}/Stream.js + case len(lower) >= 4 && lower[3] == "subtitles": + embySubtitleStream(c, user, segs[1]) + case len(lower) >= 3 && lower[2] == "subtitles": + embySubtitleStream(c, user, segs[1]) + case len(lower) >= 3 && lower[2] == "images": + embyImage(c, segs[1], segs, true) + case len(lower) == 3 && embyAuxItem(c, lower[2]): + default: + embyNotFound(c) + } +} + +func embyGetItem(c *gin.Context, lib *embyLibrary, id string) { + item, ok := lib.resolveItem(id) + if !ok { + embyNotFound(c) + return + } + // 有歌词的歌曲在详情里也带上媒体源：客户端歌词面板读取 + // item.MediaSources[0].DefaultSubtitleStreamIndex 来定位歌词轨 + if item.Type == "Audio" { + if song, ok := lib.songByID(lib.origID(id)); ok && song.Lyrics != "" { + idx := 1 + item.MediaSources = []embyMediaSource{{ + Protocol: "Http", Id: item.Id, ItemId: item.Id, Type: "Default", + SupportsDirectPlay: true, SupportsDirectStream: true, + MediaStreams: []embyMediaStream{ + {Type: "Audio", Index: 0, IsDefault: true}, + embyLyricsMediaStream(idx), + }, + MediaAttachments: []any{}, Formats: []string{}, + RequiredHttpHeaders: map[string]string{}, + DefaultSubtitleStreamIndex: &idx, + }} + } + } + c.JSON(200, item) +} + +func (l *embyLibrary) resolveItem(id string) (*embyItem, bool) { + id = l.origID(id) + switch { + case strings.HasPrefix(id, "mv-"): + n, err := strconv.ParseUint(strings.TrimPrefix(id, "mv-"), 10, 64) + if err != nil { + return nil, false + } + for i := range l.videos { + if l.videos[i].ID == uint(n) { + return l.movieItem(&l.videos[i]), true + } + } + case strings.HasPrefix(id, "so-"): + if it, ok := l.songByID(id); ok { + return l.songItem(it), true + } + case strings.HasPrefix(id, "sr-"): + if s := l.seriesByKey(strings.TrimPrefix(id, "sr-")); s != nil { + return l.seriesItem(s), true + } + case strings.HasPrefix(id, "sn-"): + rest := strings.TrimPrefix(id, "sn-") + idx := strings.LastIndex(rest, "-") + if idx > 0 { + key := rest[:idx] + season, _ := strconv.Atoi(rest[idx+1:]) + if s := l.seriesByKey(key); s != nil { + return l.seasonItem(s, season), true + } + } + case strings.HasPrefix(id, "al-"): + if al := l.albumByKey(strings.TrimPrefix(id, "al-")); al != nil { + return l.albumItem(al), true + } + case strings.HasPrefix(id, "ep-"): + if ep := l.findEpisode(id); ep != nil { + return l.episodeItem(ep), true + } + case strings.HasPrefix(id, "ar-"): + key := strings.TrimPrefix(id, "ar-") + for i := range l.music { + if artistKeyOf(&l.music[i]) != key { + continue + } + name := albumArtistOf(&l.music[i]) + e := &embyItem{ + Name: name, SortName: name, ServerId: embyServerID(), + Id: l.guid("ar-" + key), ParentId: l.guid("lib-music"), + Type: "MusicArtist", IsFolder: true, + ImageTags: map[string]string{}, UserData: &embyUserData{}, + LocationType: "FileSystem", + } + if tag := embyImageTag(&l.music[i]); tag != "" { + e.ImageTags["Primary"] = tag + } + return e, true + } + case strings.HasPrefix(id, "lib-") || id == "root": + for _, v := range l.viewItems() { + if strings.EqualFold(v.Id, l.guid(id)) { + return v, true + } + } + if id == "root" { + return l.rootItem(), true + } + } + return nil, false +} + +// descendantsOf 递归展开 parentID 下的全部后代条目。 +// 递归必须以 ParentId 为范围：此前直接返回全库条目，导致电影/剧集/音乐库互相串内容。 +func (l *embyLibrary) descendantsOf(parentID string) []*embyItem { + var out []*embyItem + var walk func(id string, depth int) + walk = func(id string, depth int) { + if depth > 6 { + return + } + for _, ch := range l.childrenOf(id) { + out = append(out, ch) + if ch.IsFolder { + walk(ch.Id, depth+1) + } + } + } + walk(parentID, 0) + return out +} + +func embyItemList(c *gin.Context, lib *embyLibrary, parentID string) { + if q := strings.TrimSpace(c.Query("ParentId")); q != "" { + parentID = q + } + var items []*embyItem + if embyBoolQuery(c, "Recursive") { + if lib.origID(parentID) == "" || lib.origID(parentID) == "root" { + items = lib.allItemsFor(embySplitCSV(c.Query("IncludeItemTypes"))) + } else { + items = lib.descendantsOf(parentID) + } + } else { + items = lib.childrenOf(parentID) + } + items = filterEmbyItems(items, c) + sortEmbyItems(items, c.Query("SortBy"), c.Query("SortOrder")) + total := len(items) + start, limit := embyPaging(c) + items = embyPaginate(items, start, limit) + c.JSON(200, embyQueryResult{Items: items, TotalRecordCount: total, StartIndex: start}) +} + +func embyChildren(c *gin.Context, lib *embyLibrary, parentID string) { + items := lib.childrenOf(parentID) + items = filterEmbyItems(items, c) + start, limit := embyPaging(c) + total := len(items) + c.JSON(200, embyQueryResult{Items: embyPaginate(items, start, limit), TotalRecordCount: total, StartIndex: start}) +} + +func embyLatest(c *gin.Context, lib *embyLibrary) { + // 客户端按媒体库分别请求"最近添加"：带 ParentId 时只返回该库内容，避免电影/电视剧/音乐互相串 + if pid := lib.origID(c.Query("ParentId")); pid == "lib-movie" || pid == "lib-tv" || pid == "lib-music" { + items := filterEmbyItems(lib.childrenOf(pid), c) + sort.SliceStable(items, func(i, j int) bool { return items[i].DateCreated > items[j].DateCreated }) + _, limit := embyPaging(c) + c.JSON(200, embyPaginate(items, 0, limit)) + return + } + var items []*embyItem + if MediaTypeEnabled(model.MediaTypeVideo) { + var movies []model.MediaItem + for i := range lib.videos { + if !isEmbySeriesItem(&lib.videos[i]) { + movies = append(movies, lib.videos[i]) + } + } + sort.Slice(movies, func(i, j int) bool { return movies[i].CreatedAt.After(movies[j].CreatedAt) }) + for i := range movies { + items = append(items, lib.movieItem(&movies[i])) + if len(items) >= 50 { + break + } + } + } + if MediaTypeEnabled(model.MediaTypeMusic) { + sort.Slice(lib.music, func(i, j int) bool { return lib.music[i].CreatedAt.After(lib.music[j].CreatedAt) }) + for i := range lib.music { + items = append(items, lib.songItem(&lib.music[i])) + if len(items) >= 100 { + break + } + } + } + _, limit := embyPaging(c) + c.JSON(200, embyPaginate(items, 0, limit)) +} + +func embyResume(c *gin.Context, lib *embyLibrary) { + if lib.user == nil { + c.JSON(200, []*embyItem{}) + return + } + rows, err := db.ListMediaResume(lib.user.ID, 100) + if err != nil { + c.JSON(200, []*embyItem{}) + return + } + var items []*embyItem + for _, row := range rows { + id := fmt.Sprintf("mv-%d", row.ItemID) + if row.EpisodeIndex > 0 { + id = fmt.Sprintf("ep-%d-%d", row.ItemID, row.EpisodeIndex) + } + if it, ok := lib.resolveItem(id); ok { + if it.UserData == nil { + it.UserData = &embyUserData{} + } + it.UserData.PlaybackPositionTicks = row.PositionTicks + items = append(items, it) + } + } + c.JSON(200, items) +} + +func embyItemCounts(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + movies, series, episodes, songs, albums := 0, 0, 0, 0, 0 + for i := range lib.videos { + if isEmbySeriesItem(&lib.videos[i]) { + continue + } + movies++ + } + for range lib.seriesList() { + series++ + } + for _, s := range lib.seriesList() { + episodes += len(lib.episodesOf(s.Key, 0)) + } + songs = len(lib.music) + albums = len(lib.albums()) + c.JSON(200, gin.H{ + "MovieCount": movies, "SeriesCount": series, "EpisodeCount": episodes, + "SongCount": songs, "AlbumCount": albums, + "ArtistCount": len(embyArtistKeys(lib)), "BookCount": 0, "TrailerCount": 0, + }) +} + +// ==================== 剧集接口 ==================== + +func embyShows(c *gin.Context, user *model.User, segs, lower []string, method string) { + lib := loadEmbyLibrary(user) + // /Shows/NextUp 只有两段路径 + if len(lower) == 2 && lower[1] == "nextup" { + embyNextUp(c, lib) + return + } + // /Shows/{id} 返回剧集详情 + if len(lower) == 2 { + if item, ok := lib.resolveItem(segs[1]); ok { + c.JSON(200, item) + return + } + embyNotFound(c) + return + } + seriesID := lib.origID(segs[1]) + key := strings.TrimPrefix(seriesID, "sr-") + s := lib.seriesByKey(key) + if s == nil { + embyNotFound(c) + return + } + switch lower[2] { + case "seasons": + var items []*embyItem + for _, season := range lib.seasonList(s) { + items = append(items, lib.seasonItem(s, season)) + } + c.JSON(200, gin.H{"Items": items, "TotalRecordCount": len(items), "StartIndex": 0}) + case "episodes": + season := embyIntQuery(c, "Season") + // 客户端也会传 SeasonId（GUID）按季过滤 + if sid := lib.origID(c.Query("SeasonId")); strings.HasPrefix(sid, "sn-") { + if idx := strings.LastIndex(sid, "-"); idx > 0 { + season, _ = strconv.Atoi(sid[idx+1:]) + } + } + var items []*embyItem + for _, ep := range lib.episodesOf(key, season) { + items = append(items, lib.episodeItem(ep)) + } + c.JSON(200, gin.H{"Items": items, "TotalRecordCount": len(items), "StartIndex": 0}) + default: + embyNotFound(c) + } +} + +func embyNextUp(c *gin.Context, lib *embyLibrary) { + var items []*embyItem + seriesFilter := strings.TrimPrefix(lib.origID(c.Query("SeriesId")), "sr-") + for _, s := range lib.seriesList() { + if seriesFilter != "" && s.Key != seriesFilter { + continue + } + if lib.user != nil { + if rows, err := db.ListMediaResume(lib.user.ID, 200); err == nil { + nextIdx := -1 + for _, row := range rows { + if seriesKeyOf(&model.MediaItem{ScrapedName: s.Name}) == s.Key && + row.ItemID == s.Items[0].ID && row.EpisodeIndex > 0 { + nextIdx = row.EpisodeIndex + 1 + break + } + } + eps := lib.episodesOf(s.Key, 0) + if nextIdx > 0 { + for _, ep := range eps { + if ep.Index == nextIdx { + items = append(items, lib.episodeItem(ep)) + break + } + } + } else if len(eps) > 0 { + items = append(items, lib.episodeItem(eps[0])) + } + continue + } + } + eps := lib.episodesOf(s.Key, 0) + if len(eps) > 0 { + items = append(items, lib.episodeItem(eps[0])) + } + } + c.JSON(200, gin.H{"Items": items, "TotalRecordCount": len(items)}) +} + +// ==================== 音乐 / 搜索 ==================== + +func embyArtistKeys(lib *embyLibrary) []string { + set := map[string]string{} + for i := range lib.music { + key := artistKeyOf(&lib.music[i]) + set[key] = albumArtistOf(&lib.music[i]) + } + out := make([]string, 0, len(set)) + for k := range set { + out = append(out, k) + } + sort.Strings(out) + return out +} + +func embyArtists(c *gin.Context, user *model.User, albumOnly bool) { + lib := loadEmbyLibrary(user) + seen := map[string]bool{} + var items []*embyItem + for i := range lib.music { + it := &lib.music[i] + key := artistKeyOf(it) + if seen[key] { + continue + } + seen[key] = true + name := albumArtistOf(it) + e := &embyItem{ + Name: name, SortName: name, ServerId: embyServerID(), + Id: lib.guid("ar-" + key), ParentId: lib.guid("lib-music"), Type: "MusicArtist", IsFolder: true, + ImageTags: map[string]string{}, UserData: &embyUserData{}, + LocationType: "FileSystem", + } + if tag := embyImageTag(it); tag != "" { + e.ImageTags["Primary"] = tag + } + items = append(items, e) + } + start, limit := embyPaging(c) + c.JSON(200, embyQueryResult{Items: embyPaginate(items, start, limit), TotalRecordCount: len(items), StartIndex: start}) +} + +func embySearchHints(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + term := strings.ToLower(strings.TrimSpace(c.Query("SearchTerm"))) + items := lib.allItems() + items = filterEmbyItems(items, c) + var hints []gin.H + for _, it := range items { + if term != "" && !strings.Contains(strings.ToLower(it.Name), term) { + continue + } + hints = append(hints, gin.H{ + "ItemId": it.Id, + "Id": it.Id, + "Name": it.Name, + "Type": it.Type, + "MediaType": it.MediaType, + "RunTimeTicks": it.RunTimeTicks, + "ProductionYear": it.ProductionYear, + "ImageTags": it.ImageTags, + }) + if len(hints) >= 50 { + break + } + } + c.JSON(200, gin.H{"SearchHints": hints, "TotalRecordCount": len(hints)}) +} + +// ==================== 媒体库目录信息（MoviePilot 需要） ==================== + +func embySelectableFolders(c *gin.Context, user *model.User) { + paths, err := db.ListMediaScanPaths("") + if err != nil { + c.JSON(500, gin.H{"Error": err.Error()}) + return + } + var out []gin.H + for _, p := range paths { + if !CanUserAccessPath(user, p.Path) { + continue + } + out = append(out, gin.H{ + "Name": p.Name, + "SubFolders": []gin.H{{ + "Name": p.Name, + "Path": p.Path, + "Id": strconv.FormatUint(uint64(p.ID), 10), + }}, + }) + } + c.JSON(200, out) +} + +func embyVirtualFolders(c *gin.Context, user *model.User) { + paths, err := db.ListMediaScanPaths("") + if err != nil { + c.JSON(500, gin.H{"Error": err.Error()}) + return + } + var out []gin.H + for _, p := range paths { + if !CanUserAccessPath(user, p.Path) { + continue + } + collectionType := "movies" + switch p.MediaType { + case model.MediaTypeVideo: + if p.TypeTag == "电视剧" || p.TypeTag == "tv" { + collectionType = "tvshows" + } + case model.MediaTypeMusic: + collectionType = "music" + case model.MediaTypeImage: + collectionType = "homevideos" + case model.MediaTypeBook: + collectionType = "books" + } + out = append(out, gin.H{ + "Name": p.Name, + "ItemId": "libpath-" + strconv.FormatUint(uint64(p.ID), 10), + "CollectionType": collectionType, + "Locations": []string{p.Path}, + "LibraryOptions": gin.H{ + "PathInfos": []gin.H{{"Path": p.Path}}, + }, + }) + } + c.JSON(200, out) +} diff --git a/server/handles/emby_play.go b/server/handles/emby_play.go new file mode 100644 index 00000000..eef8d916 --- /dev/null +++ b/server/handles/emby_play.go @@ -0,0 +1,681 @@ +package handles + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "net/http" + "net/url" + stdpath "path" + "regexp" + "strconv" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/media/imgmime" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/op" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +// embyPlayTarget 播放目标（条目或某一集） +type embyPlayTarget struct { + Item *model.MediaItem + Episode *model.EpisodeInfo + RawPath string + Container string + Size int64 + Name string + Duration int + ItemID uint + EpisodeNo int +} + +func (l *embyLibrary) playTarget(id string) (*embyPlayTarget, bool) { + id = l.origID(id) + switch { + case strings.HasPrefix(id, "mv-"): + n, err := strconv.ParseUint(strings.TrimPrefix(id, "mv-"), 10, 64) + if err != nil { + return nil, false + } + for i := range l.videos { + if l.videos[i].ID == uint(n) { + it := &l.videos[i] + return &embyPlayTarget{ + Item: it, RawPath: MediaItemFilePath(it), Container: embyExt(it.FileName), + Size: it.FileSize, Name: it.FileName, Duration: it.Duration, ItemID: it.ID, + }, true + } + } + case strings.HasPrefix(id, "so-"): + it, ok := l.songByID(id) + if !ok { + return nil, false + } + return &embyPlayTarget{ + Item: it, RawPath: MediaItemFilePath(it), Container: embyExt(it.FileName), + Size: it.FileSize, Name: it.FileName, Duration: it.Duration, ItemID: it.ID, + }, true + case strings.HasPrefix(id, "ep-"): + ep := l.findEpisode(id) + if ep == nil { + return nil, false + } + raw := MediaItemFilePath(ep.Item) + if ep.Episode.FileName != "" { + raw = stdpath.Join(mediaFolderOf(ep.Item), ep.Item.FileName, ep.Episode.FileName) + } + return &embyPlayTarget{ + Item: ep.Item, Episode: &ep.Episode, RawPath: raw, Container: embyExt(raw), + Size: ep.Item.FileSize, Name: ep.Episode.FileName, Duration: ep.Item.Duration, + ItemID: ep.Item.ID, EpisodeNo: ep.Index, + }, true + } + return nil, false +} + +// embyPlaybackInfo 返回直连播放信息 +func embyPlaybackInfo(c *gin.Context, lib *embyLibrary, id string) { + target, ok := lib.playTarget(id) + if !ok { + embyNotFound(c) + return + } + api := common_ApiBase(c) + isAudio := target.Item.MediaType == model.MediaTypeMusic + kind, streamType := "Videos", "Video" + if isAudio { + kind, streamType = "Audio", "Audio" + } + // 流地址必须自带 api_key：播放器取流时不会再附加请求头， + // 缺失会得到 401，客户端表现为"Playback of this content is currently restricted" + q := url.Values{} + q.Set("Static", "true") + q.Set("MediaSourceId", id) + if tok := embyRequestToken(c); tok != "" { + q.Set("api_key", tok) + } + streamURL := fmt.Sprintf("%s/@emby/%s/%s/stream.%s?%s", api, kind, id, target.Container, q.Encode()) + if target.Container == "" { + streamURL = fmt.Sprintf("%s/@emby/%s/%s/stream?%s", api, kind, id, q.Encode()) + } + // 必须是 Http：客户端看到 File 会当成"服务器本地文件"， + // 转而调用原生 LocalItemRepository 解析本地路径，必然失败（Playback Error） + streams := []embyMediaStream{{Type: streamType, Index: 0, IsDefault: true}} + var subtitleIndex *int + if isAudio && target.Item.Lyrics != "" { + // 歌词以"外挂字幕轨"的形式提供给客户端（Emby Theater 的歌词面板 + // 只遍历 Subtitle 流，并请求 Subtitles/{index}/Stream.js） + lyricsIndex := len(streams) + streams = append(streams, embyLyricsMediaStream(lyricsIndex)) + subtitleIndex = &lyricsIndex + } + source := embyMediaSource{ + Protocol: "Http", + Id: id, + ItemId: id, + // Emby Theater 在本地播放时会把 Path 直接交给 mpv； + // 填文件系统路径会让 mpv 去开本机不存在的文件（一直转圈/Playback Error）， + // 这里填可播放的 HTTP 流地址。 + Path: streamURL, + Type: "Default", + Container: target.Container, + Name: target.Name, + Size: target.Size, + ETag: embyHash(fmt.Sprintf("%s|%d", target.RawPath, target.Size)), + RunTimeTicks: embyTicks(target.Duration), + SupportsDirectPlay: true, + SupportsDirectStream: true, + SupportsTranscoding: false, + SupportsProbing: false, + MediaStreams: streams, + MediaAttachments: []any{}, + Formats: []string{}, + RequiredHttpHeaders: map[string]string{}, + DirectStreamUrl: streamURL, + AddApiKeyToDirectStreamUrl: false, + DefaultSubtitleStreamIndex: subtitleIndex, + } + c.JSON(200, gin.H{ + "MediaSources": []embyMediaSource{source}, + "PlaySessionId": fmt.Sprintf("openlist-%d-%d", target.ItemID, target.EpisodeNo), + }) +} + +// embyStream 302 到签名直链 +func embyStream(c *gin.Context, user *model.User, id string, isAudio bool) { + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + embyNotFound(c) + return + } + url := SignedStreamURL(c, target.RawPath) + c.Redirect(http.StatusFound, url) +} + +// ==================== 图片 ==================== + +var dataURIPrefix = "data:image/" + +// embyImage 返回条目封面 +func embyImage(c *gin.Context, id string, segs []string, allowAnon bool) { + var lib *embyLibrary + if user, ok := embyAuth(c); ok { + lib = loadEmbyLibrary(user) + } else if allowAnon { + // Emby 客户端的封面 URL 不带 token，只能允许匿名访问； + // 但必须按访客身份做权限过滤，否则等于把整个媒体库（含 hidden、 + // base_path 之外、meta 受保护的条目）的封面暴露给未登录用户。 + // 访客被禁用时直接拒绝，不能退化成「全库可见」。 + guest, err := op.GetGuest() + if err != nil || guest.Disabled { + embyUnauthorized(c) + return + } + lib = loadEmbyLibrary(guest) + } else { + embyUnauthorized(c) + return + } + // 客户端回传的是 GUID，这里先还原成内部 ID 再做前缀判断。 + // 注意不能先走 resolveItem：匿名库的 user 为 nil，剧集条目会被权限校验判为不可见。 + id = lib.origID(id) + var mediaItem *model.MediaItem + switch { + case strings.HasPrefix(id, "mv-"): + n, _ := strconv.ParseUint(strings.TrimPrefix(id, "mv-"), 10, 64) + for i := range lib.videos { + if lib.videos[i].ID == uint(n) { + mediaItem = &lib.videos[i] + } + } + case strings.HasPrefix(id, "so-"): + mediaItem, _ = lib.songByID(id) + case strings.HasPrefix(id, "ep-"): + // 图片 URL 不带 token，匿名库的 user 为 nil； + // findEpisode 会做用户权限校验并直接失败，这里只按库内条目定位封面来源 + if it, ok := lib.episodeSourceItem(id); ok { + mediaItem = it + } + case strings.HasPrefix(id, "sr-"): + if s := lib.seriesByKey(strings.TrimPrefix(id, "sr-")); s != nil && len(s.Items) > 0 { + mediaItem = &s.Items[0] + } + case strings.HasPrefix(id, "sn-"): + rest := strings.TrimPrefix(id, "sn-") + if idx := strings.LastIndex(rest, "-"); idx > 0 { + if s := lib.seriesByKey(rest[:idx]); s != nil && len(s.Items) > 0 { + mediaItem = &s.Items[0] + } + } + case strings.HasPrefix(id, "al-"): + if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 { + mediaItem = &al.Items[0] + } + case strings.HasPrefix(id, "ar-"): + key := strings.TrimPrefix(id, "ar-") + for i := range lib.music { + if artistKeyOf(&lib.music[i]) == key { + mediaItem = &lib.music[i] + break + } + } + } + if mediaItem == nil { + c.Status(404) + return + } + serveEmbyCover(c, mediaItem) +} + +// episodeSourceItem 按 ep-- 在库内定位剧集所属条目，不做权限校验。 +// 仅用于封面这类匿名可访问的资源，播放与条目详情仍走 findEpisode 的权限校验。 +func (l *embyLibrary) episodeSourceItem(id string) (*model.MediaItem, bool) { + parts := strings.Split(strings.TrimPrefix(id, "ep-"), "-") + if len(parts) != 2 { + return nil, false + } + n, err := strconv.ParseUint(parts[0], 10, 64) + if err != nil { + return nil, false + } + for i := range l.videos { + if l.videos[i].ID == uint(n) { + return &l.videos[i], true + } + } + return nil, false +} + +func serveEmbyCover(c *gin.Context, item *model.MediaItem) { + cover := strings.TrimSpace(item.Cover) + if cover == "" { + c.Status(404) + return + } + if strings.HasPrefix(cover, dataURIPrefix) { + idx := strings.Index(cover, ";base64,") + if idx < 0 { + c.Status(404) + return + } + data, err := base64.StdEncoding.DecodeString(cover[idx+8:]) + if err != nil { + c.Status(404) + return + } + // 数据库里的 data URI 声明的 MIME 不可信（历史数据可能来自内嵌标签/远端响应）， + // 响应头只使用按魔数嗅探出的四种栅格类型，SVG/HTML 一律拒绝。 + mime := imgmime.Sniff(data) + if mime == "" { + c.Status(404) + return + } + c.Header("X-Content-Type-Options", "nosniff") + c.Header("Content-Disposition", `inline; filename="cover"`) + c.Header("Content-Security-Policy", "default-src 'none'; sandbox") + c.Header("Cache-Control", "private, max-age=86400") + c.Data(200, mime, data) + return + } + if strings.HasPrefix(cover, "http://") || strings.HasPrefix(cover, "https://") { + c.Redirect(302, cover) + return + } + // 本地缩略图路径 + c.Redirect(302, SignedStreamURL(c, cover)) +} + +// ==================== 歌词 ==================== + +var lrcLineRe = regexp.MustCompile(`\[(\d{1,3}):(\d{1,2})(?:[.:](\d{1,3}))?\](.*)`) + +func parseLrc(content string) []gin.H { + var out []gin.H + for _, line := range strings.Split(content, "\n") { + m := lrcLineRe.FindStringSubmatch(strings.TrimSpace(line)) + if len(m) == 0 { + continue + } + min, _ := strconv.Atoi(m[1]) + sec, _ := strconv.Atoi(m[2]) + ms := 0 + if m[3] != "" { + frac := m[3] + for len(frac) < 3 { + frac += "0" + } + ms, _ = strconv.Atoi(frac[:3]) + } + text := strings.TrimSpace(m[4]) + out = append(out, gin.H{ + "Text": text, + "Start": int64(min*60000 + sec*1000 + ms), + }) + } + return out +} + +func embyLyrics(c *gin.Context, user *model.User, id string) { + lib := loadEmbyLibrary(user) + item, ok := lib.songByID(id) + if !ok || item.Lyrics == "" { + c.Status(404) + return + } + lines := parseLrc(item.Lyrics) + c.JSON(200, gin.H{"Lyrics": lines, "IsSynced": len(lines) > 0}) +} + +// embyLyricsMediaStream 构造歌词字幕轨（Emby 用外挂字幕承载歌词）。 +func embyLyricsMediaStream(index int) embyMediaStream { + return embyMediaStream{ + Type: "Subtitle", + Index: index, + Codec: "lrc", + Language: "und", + DisplayTitle: "Lyrics", + IsExternal: true, + IsTextSubtitleStream: true, + SupportsExternalStream: true, + DeliveryMethod: "External", + DeliveryUrl: "", + } +} + +// embyLyricsTrackEvents 把 LRC 转成客户端歌词面板使用的 TrackEvent 列表。 +// 客户端按 StartPositionTicks 与当前播放位置比较（1ms = 10000 ticks）。 +// 没有时间轴的纯文本歌词会按行均分到歌曲时长，至少能顺序展示。 +func embyLyricsTrackEvents(content string, durationSec int) []gin.H { + lines := parseLrc(content) + if len(lines) == 0 { + texts := make([]string, 0, 32) + for _, l := range strings.Split(strings.ReplaceAll(content, "\r\n", "\n"), "\n") { + if t := strings.TrimSpace(l); t != "" { + texts = append(texts, t) + } + } + if len(texts) > 0 { + total := int64(durationSec) * 1000 + if total <= 0 { + total = int64(len(texts)) * 5000 + } + step := total / int64(len(texts)) + for i, t := range texts { + lines = append(lines, gin.H{"Text": t, "Start": int64(i) * step}) + } + } + } + events := make([]gin.H, 0, len(lines)) + for _, l := range lines { + ms, _ := l["Start"].(int64) + events = append(events, gin.H{ + "StartPositionTicks": ms * 10000, + "Text": l["Text"], + }) + } + return events +} + +// embySubtitleStream 处理 /Items/{id}/{mediaSourceId}/Subtitles/{index}/Stream.js +// Emby Theater 用它拉取歌词行（JSON 形式的 TrackEvents） +func embySubtitleStream(c *gin.Context, user *model.User, id string) { + lib := loadEmbyLibrary(user) + item, ok := lib.songByID(lib.origID(id)) + if !ok || item.Lyrics == "" { + c.JSON(200, gin.H{"TrackEvents": []any{}}) + return + } + events := embyLyricsTrackEvents(item.Lyrics, item.Duration) + c.JSON(200, gin.H{"TrackEvents": events, "TotalRecordCount": len(events)}) +} + +// ==================== 播放会话 / 用户状态 ==================== + +type embyPlaybackBody struct { + ItemId string `json:"ItemId"` + ItemID string `json:"ItemID"` + PositionTicks int64 `json:"PositionTicks"` + PlayMethod string `json:"PlayMethod"` + EventName string `json:"EventName"` + MediaSourceId string `json:"MediaSourceId"` +} + +func (b embyPlaybackBody) itemID() string { + if b.ItemId != "" { + return b.ItemId + } + return b.ItemID +} + +func embySessionPlaying(c *gin.Context, user *model.User, action string) { + var body embyPlaybackBody + _ = c.ShouldBindJSON(&body) + if action == "ping" { + c.Status(204) + return + } + id := body.itemID() + if id == "" { + id = c.Query("ItemId") + } + if id == "" { + c.Status(204) + return + } + stopped := action == "stopped" || action == "stop" + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + c.Status(204) + return + } + position := body.PositionTicks + if position == 0 { + if v, err := strconv.ParseInt(c.DefaultQuery("PositionTicks", "0"), 10, 64); err == nil { + position = v + } + } + played := false + if stopped && target.Duration > 0 { + ratio := float64(position) / float64(embyTicks(target.Duration)) + played = ratio >= 0.9 + } + applyProgress(user, target, position, played, stopped) + c.Status(204) +} + +func embySaveProgress(c *gin.Context, user *model.User, id string, start bool) { + var body embyPlaybackBody + _ = c.ShouldBindJSON(&body) + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + c.Status(204) + return + } + position := body.PositionTicks + if position == 0 { + position, _ = strconv.ParseInt(c.DefaultQuery("PositionTicks", "0"), 10, 64) + } + played := false + stopped := c.Request.Method == "DELETE" + if stopped && target.Duration > 0 { + played = float64(position)/float64(embyTicks(target.Duration)) >= 0.9 + } + applyProgress(user, target, position, played, stopped) + c.Status(204) +} + +func applyProgress(user *model.User, target *embyPlayTarget, position int64, played bool, stopped bool) { + if user == nil || target == nil { + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + d.PositionTicks = position + if played { + d.Played = true + d.PositionTicks = 0 + d.PlayCount++ + } + now := time.Now() + d.LastPlayedAt = &now + _ = db.SaveMediaUserData(d) +} + +func embySetPlayed(c *gin.Context, user *model.User, id string, played bool) { + lib := loadEmbyLibrary(user) + id = lib.origID(id) + target, ok := lib.playTarget(id) + if !ok { + // 剧集/季整体标记 + if strings.HasPrefix(id, "sr-") || strings.HasPrefix(id, "sn-") { + embyMarkContainerPlayed(c, user, lib, id, played) + return + } + embyNotFound(c) + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + d.Played = played + if played { + d.PositionTicks = 0 + d.PlayCount++ + } else { + d.PlayCount = 0 + } + now := time.Now() + d.LastPlayedAt = &now + _ = db.SaveMediaUserData(d) + c.JSON(200, gin.H{ + "PlaybackPositionTicks": d.PositionTicks, + "PlayCount": d.PlayCount, + "Played": d.Played, + "IsFavorite": d.IsFavorite, + }) +} + +func embyMarkContainerPlayed(c *gin.Context, user *model.User, lib *embyLibrary, id string, played bool) { + var targets []*embyPlayTarget + if strings.HasPrefix(id, "sr-") { + key := strings.TrimPrefix(id, "sr-") + for _, ep := range lib.episodesOf(key, 0) { + if t, ok := lib.playTarget(ep.ID); ok { + targets = append(targets, t) + } + } + } else { + rest := strings.TrimPrefix(id, "sn-") + idx := strings.LastIndex(rest, "-") + if idx > 0 { + key := rest[:idx] + season, _ := strconv.Atoi(rest[idx+1:]) + for _, ep := range lib.episodesOf(key, season) { + if t, ok := lib.playTarget(ep.ID); ok { + targets = append(targets, t) + } + } + } + } + for _, t := range targets { + d, err := db.GetMediaUserData(user.ID, t.ItemID, t.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: t.ItemID, EpisodeIndex: t.EpisodeNo} + } + d.Played = played + d.PositionTicks = 0 + if !played { + d.PlayCount = 0 + } + _ = db.SaveMediaUserData(d) + } + c.Status(204) +} + +func embySetFavorite(c *gin.Context, user *model.User, id string, fav bool) { + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + embyNotFound(c) + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + d.IsFavorite = fav + _ = db.SaveMediaUserData(d) + c.JSON(200, gin.H{ + "PlaybackPositionTicks": d.PositionTicks, + "PlayCount": d.PlayCount, + "Played": d.Played, + "IsFavorite": d.IsFavorite, + }) +} + +// embyUpdateUserData 部分更新用户状态 +func embyUpdateUserData(c *gin.Context, user *model.User, id string) { + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + embyNotFound(c) + return + } + var body map[string]any + _ = c.ShouldBindJSON(&body) + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + if v, ok := body["Played"].(bool); ok { + d.Played = v + if v { + d.PositionTicks = 0 + } + } + if v, ok := body["IsFavorite"].(bool); ok { + d.IsFavorite = v + } + if v, ok := body["PlaybackPositionTicks"].(float64); ok { + d.PositionTicks = int64(v) + } + if v, ok := body["PlayCount"].(float64); ok { + d.PlayCount = int(v) + } + _ = db.SaveMediaUserData(d) + c.JSON(200, gin.H{ + "PlaybackPositionTicks": d.PositionTicks, + "PlayCount": d.PlayCount, + "Played": d.Played, + "IsFavorite": d.IsFavorite, + }) +} + +// embyRefreshItem 触发条目所属扫描路径的增量扫描 +func embyRefreshItem(c *gin.Context, id string) { + lib := loadEmbyLibrary(nil) + var itemID uint + switch { + case strings.HasPrefix(id, "mv-"), strings.HasPrefix(id, "so-"): + n, _ := strconv.ParseUint(id[3:], 10, 64) + itemID = uint(n) + case strings.HasPrefix(id, "ep-"): + parts := strings.Split(id, "-") + if len(parts) == 3 { + n, _ := strconv.ParseUint(parts[1], 10, 64) + itemID = uint(n) + } + case strings.HasPrefix(id, "sr-"): + if s := lib.seriesByKey(strings.TrimPrefix(id, "sr-")); s != nil && len(s.Items) > 0 { + itemID = s.Items[0].ID + } + case strings.HasPrefix(id, "al-"): + if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 { + itemID = al.Items[0].ID + } + } + if itemID == 0 { + c.Status(204) + return + } + item, err := db.GetMediaItemByID(itemID) + if err != nil { + c.Status(204) + return + } + if sp, err := db.GetMediaScanPath(item.ScanPathID); err == nil && sp != nil { + media.ScanMediaPath(sp) + } + c.Status(204) +} + +// ==================== 内部工具 ==================== + +func common_ApiBase(c *gin.Context) string { + return strings.TrimSuffix(common.GetApiUrl(c.Request.Context()), "/") +} + +// embyBodyJSON 便于调试时输出请求体 +func embyBodyJSON(c *gin.Context) string { + raw, _ := c.GetRawData() + if len(raw) == 0 { + return "" + } + var v any + if err := json.Unmarshal(raw, &v); err != nil { + return string(raw) + } + b, _ := json.Marshal(v) + return string(b) +} diff --git a/server/handles/media.go b/server/handles/media.go new file mode 100644 index 00000000..b6f981b5 --- /dev/null +++ b/server/handles/media.go @@ -0,0 +1,1101 @@ +package handles + +import ( + "context" + "encoding/json" + "strconv" + "strings" + "sync" + "time" + + stdpath "path" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/errs" + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/media/scraper" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" + log "github.com/sirupsen/logrus" +) + +// scrapeConcurrency 刮削并发度（默认 5） +// 走 setting 读取，未配置时使用 5；远端 API 调用密集型任务，并发 5 在限流与吞吐间较平衡。 +const defaultScrapeConcurrency = 5 + +// ==================== 配置管理 ==================== + +// ListMediaConfigs 获取所有媒体库配置 +func ListMediaConfigs(c *gin.Context) { + types := []model.MediaType{ + model.MediaTypeVideo, + model.MediaTypeMusic, + model.MediaTypeImage, + model.MediaTypeBook, + } + cfgs := make([]*model.MediaConfig, 0, len(types)) + for _, t := range types { + cfg, err := db.GetMediaConfig(t) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + cfgs = append(cfgs, cfg) + } + common.SuccessResp(c, cfgs) +} + +// SaveMediaConfigReq 保存配置请求 +type SaveMediaConfigReq struct { + MediaType model.MediaType `json:"media_type" binding:"required"` + Enabled bool `json:"enabled"` + ScraperConfig *string `json:"scraper_config"` +} + +// SaveMediaConfig 保存媒体库配置 +func SaveMediaConfig(c *gin.Context) { + var req SaveMediaConfigReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + // 读取已有配置，避免用零值覆盖 last_scan_at / last_scrape_at + cfg, err := db.GetMediaConfig(req.MediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + cfg.Enabled = req.Enabled + if req.ScraperConfig != nil { + cfg.ScraperConfig = *req.ScraperConfig + } + if err := db.SaveMediaConfig(cfg); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ==================== 扫描路径管理 ==================== + +// ListMediaScanPaths 获取指定媒体类型的扫描路径列表 +func ListMediaScanPaths(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + paths, err := db.ListMediaScanPaths(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, paths) +} + +// CreateMediaScanPathReq 创建扫描路径请求 +type CreateMediaScanPathReq struct { + MediaType model.MediaType `json:"media_type" binding:"required"` + Name string `json:"name"` + Path string `json:"path" binding:"required"` + PathMerge bool `json:"path_merge"` + TypeTag string `json:"type_tag"` + ContentTags string `json:"content_tags"` + EnableScrape bool `json:"enable_scrape"` +} + +// CreateMediaScanPath 创建扫描路径 +func CreateMediaScanPath(c *gin.Context) { + var req CreateMediaScanPathReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + if req.Path == "" { + req.Path = "/" + } + name := req.Name + if name == "" { + name = req.Path + } + sp := &model.MediaScanPath{ + MediaType: req.MediaType, + Name: name, + Path: req.Path, + PathMerge: req.PathMerge, + TypeTag: req.TypeTag, + ContentTags: req.ContentTags, + EnableScrape: req.EnableScrape, + } + if err := db.CreateMediaScanPath(sp); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, sp) +} + +// UpdateMediaScanPathReq 更新扫描路径请求 +type UpdateMediaScanPathReq struct { + ID uint `json:"id" binding:"required"` + Name string `json:"name"` + Path string `json:"path"` + PathMerge bool `json:"path_merge"` + TypeTag string `json:"type_tag"` + ContentTags string `json:"content_tags"` + EnableScrape bool `json:"enable_scrape"` +} + +// UpdateMediaScanPath 更新扫描路径 +func UpdateMediaScanPath(c *gin.Context) { + var req UpdateMediaScanPathReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + sp, err := db.GetMediaScanPath(req.ID) + if err != nil { + common.ErrorResp(c, err, 404) + return + } + sp.Name = req.Name + sp.Path = req.Path + sp.PathMerge = req.PathMerge + sp.TypeTag = req.TypeTag + sp.ContentTags = req.ContentTags + sp.EnableScrape = req.EnableScrape + if err := db.UpdateMediaScanPath(sp); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// DeleteMediaScanPath 删除扫描路径 +func DeleteMediaScanPath(c *gin.Context) { + idStr := c.Query("id") + id, err := strconv.ParseUint(idStr, 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + if err := db.DeleteMediaScanPath(uint(id)); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ClearMediaScanPathDB 清空指定扫描路径的媒体数据 +func ClearMediaScanPathDB(c *gin.Context) { + idStr := c.Query("id") + id, err := strconv.ParseUint(idStr, 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + if err := db.ClearMediaItemsByScanPath(uint(id)); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ==================== 媒体条目管理（后台） ==================== + +// clampMediaPage 限制分页参数，避免 page_size 过大或为负导致整表扫描 / 响应体爆炸 +func clampMediaPage(page, pageSize, defaultSize, maxSize int) (int, int) { + if page < 1 { + page = 1 + } + if pageSize <= 0 { + pageSize = defaultSize + } + if pageSize > maxSize { + pageSize = maxSize + } + return page, pageSize +} + +// ListMediaItemsAdmin 后台分页查询媒体条目 +func ListMediaItemsAdmin(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + page, _ := strconv.Atoi(c.DefaultQuery("page", "1")) + pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "20")) + page, pageSize = clampMediaPage(page, pageSize, 20, 500) + keyword := c.Query("keyword") + orderBy := c.DefaultQuery("order_by", "name") + orderDir := c.DefaultQuery("order_dir", "asc") + scanPathIDStr := c.Query("scan_path_id") + var scanPathID uint + if scanPathIDStr != "" { + if id, err := strconv.ParseUint(scanPathIDStr, 10, 64); err == nil { + scanPathID = uint(id) + } + } + + q := db.MediaItemQuery{ + MediaType: mediaType, + ScanPathID: scanPathID, + Keyword: keyword, + OrderBy: orderBy, + OrderDir: orderDir, + Page: page, + PageSize: pageSize, + } + items, total, err := db.ListMediaItems(q) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, common.PageResp{Content: items, Total: total}) +} + +// UpdateMediaItemReq 更新媒体条目请求 +type UpdateMediaItemReq struct { + ID uint `json:"id" binding:"required"` + ScrapedName string `json:"scraped_name"` + Description string `json:"description"` + Cover string `json:"cover"` + ReleaseDate string `json:"release_date"` + Rating float32 `json:"rating"` + Genre string `json:"genre"` + Authors string `json:"authors"` + Plot string `json:"plot"` + Reviews string `json:"reviews"` + AlbumName string `json:"album_name"` + AlbumArtist string `json:"album_artist"` + Publisher string `json:"publisher"` + ISBN string `json:"isbn"` + Hidden bool `json:"hidden"` +} + +// UpdateMediaItemAdmin 后台更新媒体条目 +func UpdateMediaItemAdmin(c *gin.Context) { + var req UpdateMediaItemReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + item, err := db.GetMediaItemByID(req.ID) + if err != nil { + common.ErrorResp(c, err, 404) + return + } + item.ScrapedName = req.ScrapedName + item.Description = req.Description + item.Cover = req.Cover + item.ReleaseDate = req.ReleaseDate + item.Rating = req.Rating + item.Genre = req.Genre + item.Authors = req.Authors + item.Plot = req.Plot + item.Reviews = req.Reviews + item.AlbumName = req.AlbumName + item.AlbumArtist = req.AlbumArtist + item.Publisher = req.Publisher + item.ISBN = req.ISBN + item.Hidden = req.Hidden + + if err := db.UpdateMediaItem(item); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// DeleteMediaItemAdmin 后台删除媒体条目 +func DeleteMediaItemAdmin(c *gin.Context) { + idStr := c.Query("id") + id, err := strconv.ParseUint(idStr, 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + if err := db.DeleteMediaItem(uint(id)); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ClearMediaDB 清空指定类型媒体数据库 +func ClearMediaDB(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + if mediaType == "" { + common.ErrorStrResp(c, "media_type 不能为空", 400) + return + } + if err := db.ClearMediaItems(mediaType); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ClearMediaScrape 清空刮削数据（保留扫描记录，但清空所有刮削结果字段） +// 参数：media_type 可选，留空表示所有类型；item_id 暂不支持（统一为类型/全部） +func ClearMediaScrape(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + affected, err := db.ClearMediaScrapedData(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + // 同步清掉对应媒体配置中的最近刮削时间 + if mediaType != "" { + if cfg, e := db.GetMediaConfig(mediaType); e == nil && cfg != nil { + cfg.LastScrapeAt = nil + _ = db.SaveMediaConfig(cfg) + } + } else { + if cfgs, e := db.GetAllMediaConfigs(); e == nil { + for i := range cfgs { + cfgs[i].LastScrapeAt = nil + _ = db.SaveMediaConfig(&cfgs[i]) + } + } + } + common.SuccessResp(c, gin.H{"affected": affected}) +} + +// DeleteInvalidMedia 删除已失效的媒体条目（即对应文件 / 文件夹在存储中已不存在） +// 参数：media_type 可选，留空表示扫描所有类型 +// 返回：检测总数、被删除条目数 +func DeleteInvalidMedia(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + + items, err := db.ListAllValidMediaItemsForCheck(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + var invalidIDs []uint + var skipped int + for i := range items { + it := &items[i] + // 拼接文件 / 文件夹的完整 VFS 路径 + // - 文件夹模式 (is_folder=true): folder_path 即文件夹自身（扫描根 + 文件夹名 视情况而定）；这里直接使用 folder_path/file_name + // - 普通文件: folder_path 是所在目录，file_name 是文件名 + fullPath := stdpath.Join(it.FolderPath, it.FileName) + if fullPath == "" || fullPath == "/" { + continue + } + if _, gerr := fs.Get(ctx, fullPath, &fs.GetArgs{NoLog: true}); gerr != nil { + // 只有「确认不存在」才算失效条目；驱动故障、超时、鉴权失败等可能是临时问题， + // 一律跳过（保留记录），否则一次网络抖动就会把有效的媒体记录硬删除。 + if errs.IsObjectNotFound(gerr) { + invalidIDs = append(invalidIDs, it.ID) + continue + } + skipped++ + log.Warnf("跳过无法确认状态的媒体条目 %s（%s）: %v", fullPath, it.FileName, gerr) + } + } + + if len(invalidIDs) > 0 { + if err := db.DeleteMediaItemsByIDs(invalidIDs); err != nil { + common.ErrorResp(c, err, 500) + return + } + } + log.Infof("删除已失效媒体条目：检测 %d 条，删除 %d 条，跳过 %d 条 (media_type=%q)", len(items), len(invalidIDs), skipped, mediaType) + common.SuccessResp(c, gin.H{ + "checked": len(items), + "deleted": len(invalidIDs), + "skipped": skipped, + }) +} + +// ==================== 扫描与刮削 ==================== + +// ScanMediaReq 扫描请求 +type ScanMediaReq struct { + MediaType model.MediaType `json:"media_type" binding:"required"` + ScanPathID uint `json:"scan_path_id"` // 0 表示扫描全部路径 +} + +// StartMediaScan 开始扫描 +func StartMediaScan(c *gin.Context) { + var req ScanMediaReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + cfg, err := db.GetMediaConfig(req.MediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + if !cfg.Enabled { + common.ErrorStrResp(c, "该媒体库未启用", 400) + return + } + + if req.ScanPathID > 0 { + // 扫描单个路径 + sp, err := db.GetMediaScanPath(req.ScanPathID) + if err != nil { + common.ErrorResp(c, err, 404) + return + } + media.ScanMediaPath(sp) + } else { + // 扫描全部路径 + media.ScanMedia(cfg) + } + common.SuccessResp(c) +} + +// GetMediaScanProgress 获取扫描进度 +func GetMediaScanProgress(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + if mediaType == "" { + common.ErrorStrResp(c, "media_type 不能为空", 400) + return + } + progress := media.GetProgress(mediaType) + common.SuccessResp(c, progress) +} + +// MediaScrapeStatsResp 刮削统计响应 +// 包含 TMDB 视频刮削的累计指标，用于运维观测与命中率评估 +type MediaScrapeStatsResp struct { + TMDB scraper.TMDBStats `json:"tmdb"` +} + +// GetMediaScrapeStats 获取刮削统计（进程级累计值，重启清零） +// 返回字段： +// - total_scraped : 累计触发刮削次数 +// - hit_count : 命中（含强命中）次数 +// - low_confidence_count : 仅低置信度兜底命中次数 +// - no_match_count : 完全未命中次数 +// - total_attempts : 累计 TMDB 搜索请求次数 +// - total_search_millis : 累计搜索耗时（毫秒） +func GetMediaScrapeStats(c *gin.Context) { + common.SuccessResp(c, MediaScrapeStatsResp{ + TMDB: scraper.GetTMDBStats(), + }) +} + +// ScrapeMediaReq 刮削请求 +type ScrapeMediaReq struct { + MediaType model.MediaType `json:"media_type" binding:"required"` + ItemID uint `json:"item_id"` // 0 表示刮削全部未刮削的 +} + +// StartMediaScrape 开始刮削 +func StartMediaScrape(c *gin.Context) { + var req ScrapeMediaReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + + cfg, err := db.GetMediaConfig(req.MediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + + // 从系统设置中读取刮削配置 + tmdbKey := setting.GetStr(conf.MediaTMDBKey) + tmdbAPIURL := setting.GetStr(conf.MediaTMDBAPIURL, "https://api.themoviedb.org") + discogsToken := setting.GetStr(conf.MediaDiscogsToken) + discogsAPIURL := setting.GetStr(conf.MediaDiscogsAPIURL, "https://api.discogs.com") + thumbnailMode := setting.GetStr(conf.MediaThumbnailMode, "runtime") + thumbnailPath := setting.GetStr(conf.MediaThumbnailPath, "/imgs") + storeThumbnail := setting.GetBool(conf.MediaStoreThumbnail) + bookSource := strings.ToLower(setting.GetStr(conf.MediaBookSource, "douban")) + googleBooksKey := setting.GetStr(conf.MediaGoogleBooksKey) + downloadCover := setting.GetBool(conf.MediaDownloadCover) + + // 该媒体类型如果单独配置过缩略图，覆盖全局设置 + if cfg.ScraperConfig != "" { + var sc struct { + StoreThumbnail string `json:"store_thumbnail"` + ThumbnailMode string `json:"thumbnail_mode"` + ThumbnailPath string `json:"thumbnail_path"` + } + if json.Unmarshal([]byte(cfg.ScraperConfig), &sc) == nil { + if sc.ThumbnailMode != "" { + thumbnailMode = sc.ThumbnailMode + } + if sc.ThumbnailPath != "" { + thumbnailPath = sc.ThumbnailPath + } + if sc.StoreThumbnail != "" { + storeThumbnail = sc.StoreThumbnail == "true" + } + } + } + + log.Infof("[media] scrape %s: thumbnail mode=%s, path=%s, runtime dir=%s", + req.MediaType, thumbnailMode, thumbnailPath, scraper.RuntimeThumbRoot(thumbnailPath)) + + var preloadedItems []model.MediaItem + if req.ItemID > 0 { + item, e := db.GetMediaItemByID(req.ItemID) + if e != nil { + common.ErrorResp(c, e, 404) + return + } + preloadedItems = []model.MediaItem{*item} + } + + go func() { + var items []model.MediaItem + var err error + + if req.ItemID > 0 { + items = preloadedItems + } else { + // limit 传 -1，GORM 不会拼接 LIMIT 子句，从而一次性取出全部未刮削条目 + items, err = db.GetUnscrappedItems(req.MediaType, -1) + if err != nil { + log.Errorf("获取未刮削条目失败: %v", err) + return + } + } + + // 读取并发度（默认5），并启动 worker pool + concurrency := setting.GetInt(conf.MediaScrapeConcurrency, defaultScrapeConcurrency) + if concurrency <= 0 { + concurrency = defaultScrapeConcurrency + } + if concurrency > len(items) && len(items) > 0 { + concurrency = len(items) + } + + var wg sync.WaitGroup + jobs := make(chan int, len(items)) + // 落库串行化：所有 worker 把刮削成功的 item 推到 saveCh，由单协程负责落库， + // 这样既能并发抓 API、又避免多 goroutine 同时写 sqlite/mysql 引发锁竞争。 + saveCh := make(chan *model.MediaItem, concurrency*2) + var saveWg sync.WaitGroup + + // 单写入协程：刮一个写一个 + saveWg.Add(1) + go func() { + defer saveWg.Done() + for it := range saveCh { + if err := db.UpdateMediaItem(it); err != nil { + log.Warnf("保存刮削结果失败 [id=%d, %s/%s]: %v", it.ID, it.FolderPath, it.FileName, err) + } + } + }() + + worker := func() { + defer wg.Done() + for idx := range jobs { + item := &items[idx] + var scrapeErr error + + switch req.MediaType { + case model.MediaTypeVideo: + s := scraper.NewTMDBScraper(tmdbKey, tmdbAPIURL) + scrapeErr = s.ScrapeVideo(item) + case model.MediaTypeMusic: + // 音乐：刮削前先用文件本身的 ID3/Vorbis tag + .lrc 做本地兜底， + // 仅填空字段；这样用户清空刮削后重新刮削也能恢复封面/歌词等。 + localCtx, localCancel := context.WithTimeout(context.Background(), 30*time.Second) + media.FillMusicFromLocal(localCtx, item) + localCancel() + + s := scraper.NewDiscogsScraper(discogsToken, discogsAPIURL) + scrapeErr = s.ScrapeMusic(item) + case model.MediaTypeBook: + var bookErr error + switch bookSource { + case "google": + bookErr = scraper.NewGoogleBooksScraper( + googleBooksKey, + thumbnailMode, + thumbnailPath, + ).ScrapeBook(item) + case "openlibrary": + bookErr = scraper.NewOpenLibraryScraper( + thumbnailMode, + thumbnailPath, + ).ScrapeBook(item) + default: + bookErr = scraper.NewDoubanScraperWithConfig( + thumbnailMode, + thumbnailPath, + ).ScrapeBook(item) + } + if bookErr != nil { + log.Debugf("书籍刮削失败 [%s/%s]: %v，将尝试本地提取封面", item.FolderPath, item.FileName, bookErr) + } + + if item.Cover == "" { + bookCtx, bookCancel := context.WithTimeout(context.Background(), 60*time.Second) + bookFilePath := item.FolderPath + "/" + item.FileName + bookReader := media.FetchFileReader(bookCtx, bookFilePath) + if bookReader != nil { + localScraper := scraper.NewBookLocalScraperWithConfig( + thumbnailMode, + thumbnailPath, + ) + if localCover := localScraper.ExtractLocalCover(item.FileName, bookFilePath, bookReader); localCover != "" { + item.Cover = localCover + } + _ = bookReader.Close() + } + bookCancel() + } + + if bookErr != nil && item.Cover == "" { + scrapeErr = bookErr + } + case model.MediaTypeImage: + imgCtx, imgCancel := context.WithTimeout(context.Background(), 30*time.Second) + imgFilePath := item.FolderPath + "/" + item.FileName + imgReader := media.FetchFileReader(imgCtx, imgFilePath) + s := scraper.NewImageScraperWithConfig( + storeThumbnail, + thumbnailMode, + thumbnailPath, + ) + scrapeErr = s.ScrapeImage(item, imgReader) + if imgReader != nil { + _ = imgReader.Close() + } + imgCancel() + } + + // 远程封面按需下载到本地存储（或 base64），避免直接保存图片链接 + if downloadCover && strings.HasPrefix(item.Cover, "http") { + rawPath := strings.TrimSuffix(item.FolderPath, "/") + "/" + item.FileName + if cover := scraper.DownloadCoverToStorage( + context.Background(), + item.Cover, + rawPath, + thumbnailMode, + thumbnailPath, + ); cover != "" { + item.Cover = cover + } + } + + if scrapeErr != nil { + log.Warnf("刮削失败 [%s] %s: %v", req.MediaType, item.FolderPath, scrapeErr) + // 即使远端刮削失败，只要本地兜底已经写入了任意有效字段， + // 也保存一次（不标记 ScrapedAt，便于下次还能重试）， + // 避免 cover/lyrics 等被丢弃。 + if hasAnyScrapedField(item) { + saveCh <- item + } + continue + } + now := time.Now() + item.ScrapedAt = &now + saveCh <- item + } + } + + for i := 0; i < concurrency; i++ { + wg.Add(1) + go worker() + } + + for i := range items { + jobs <- i + } + close(jobs) + wg.Wait() + close(saveCh) + saveWg.Wait() + + log.Infof("刮削完成 [%s]，共处理 %d 条（并发=%d）", req.MediaType, len(items), concurrency) + }() + + _ = cfg + common.SuccessResp(c) +} + +// hasAnyScrapedField 判断 item 是否已经有任意一个有意义的刮削字段被填充 +// 用于：远端刮削失败但本地兜底已经写入了内容时，仍然落库保留这些信息 +func hasAnyScrapedField(item *model.MediaItem) bool { + if item == nil { + return false + } + return item.Cover != "" || + item.Lyrics != "" || + item.Description != "" || + item.Plot != "" || + item.Genre != "" || + item.Authors != "" || + item.AlbumArtist != "" || + item.ReleaseDate != "" || + item.Rating > 0 || + item.Publisher != "" || + item.ISBN != "" || + item.ExternalID != "" +} + +// ==================== 公开API（前端媒体库浏览） ==================== + +// PublicListMedia 公开媒体列表（前端浏览用） +func PublicListMedia(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + page, _ := strconv.Atoi(c.DefaultQuery("page", "1")) + pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "40")) + page, pageSize = clampMediaPage(page, pageSize, 40, 200) + orderBy := c.DefaultQuery("order_by", "name") + orderDir := c.DefaultQuery("order_dir", "asc") + folderPath := c.Query("folder_path") + keyword := c.Query("keyword") + typeTag := c.Query("type_tag") + contentTag := c.Query("content_tag") + scanPathIDStr := c.Query("scan_path_id") + var scanPathID uint + if scanPathIDStr != "" { + if id, err := strconv.ParseUint(scanPathIDStr, 10, 64); err == nil { + scanPathID = uint(id) + } + } + + hidden := false + user := mediaUser(c) + q := db.MediaItemQuery{ + MediaType: mediaType, + ScanPathID: scanPathID, + FolderPath: folderPath, + TypeTag: typeTag, + ContentTag: contentTag, + PathPrefix: userPathPrefix(user), + Hidden: &hidden, + Keyword: keyword, + OrderBy: orderBy, + OrderDir: orderDir, + Page: page, + PageSize: pageSize, + } + items, total, err := db.ListMediaItems(q) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + // 元信息(meta) 级别的读权限无法下推到 SQL，这里对当前页做二次过滤 + filtered := make([]model.MediaItem, 0, len(items)) + for i := range items { + if CanUserAccessItem(user, &items[i]) { + filtered = append(filtered, items[i]) + } + } + common.SuccessResp(c, common.PageResp{Content: filtered, Total: total}) +} + +// PublicGetMedia 公开获取媒体详情 +func PublicGetMedia(c *gin.Context) { + idStr := c.Param("id") + id, err := strconv.ParseUint(idStr, 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + item, err := db.GetMediaItemByID(uint(id)) + if err != nil { + common.ErrorResp(c, err, 404) + return + } + if item.Hidden || !CanUserAccessItem(mediaUser(c), item) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + common.SuccessResp(c, item) +} + +// PublicListAlbums 公开专辑列表（音乐） +func PublicListAlbums(c *gin.Context) { + page, _ := strconv.Atoi(c.DefaultQuery("page", "1")) + pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "40")) + page, pageSize = clampMediaPage(page, pageSize, 40, 200) + keyword := c.Query("keyword") + scanPathIDStr := c.Query("scan_path_id") + var scanPathID uint + if scanPathIDStr != "" { + if id, err := strconv.ParseUint(scanPathIDStr, 10, 64); err == nil { + scanPathID = uint(id) + } + } + + hidden := false + q := db.MediaItemQuery{ + MediaType: model.MediaTypeMusic, + ScanPathID: scanPathID, + PathPrefix: userPathPrefix(mediaUser(c)), + Hidden: &hidden, + Keyword: keyword, + Page: page, + PageSize: pageSize, + } + albums, total, err := db.ListAlbums(q) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, common.PageResp{Content: albums, Total: total}) +} + +// PublicGetAlbum 公开获取专辑详情及曲目 +func PublicGetAlbum(c *gin.Context) { + albumName := c.Query("album_name") + albumArtist := c.Query("album_artist") + if albumName == "" && albumArtist == "" { + common.ErrorStrResp(c, "album_name 和 album_artist 不能同时为空", 400) + return + } + tracks, err := db.GetAlbumTracks(albumName, albumArtist) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + user := mediaUser(c) + filtered := make([]model.MediaItem, 0, len(tracks)) + for i := range tracks { + if CanUserAccessItem(user, &tracks[i]) { + filtered = append(filtered, tracks[i]) + } + } + common.SuccessResp(c, filtered) +} + +// PublicListFolders 公开文件夹列表（目录浏览模式） +func PublicListFolders(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + if mediaType == "" { + common.ErrorStrResp(c, "media_type 不能为空", 400) + return + } + paths, err := db.ListFolderPaths(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + user := mediaUser(c) + filtered := make([]string, 0, len(paths)) + for _, p := range paths { + if CanUserAccessPath(user, p) { + filtered = append(filtered, p) + } + } + common.SuccessResp(c, filtered) +} + +// PublicListScanPaths 公开获取扫描路径列表（前端筛选用） +func PublicListScanPaths(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + paths, err := db.ListMediaScanPaths(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + user := mediaUser(c) + filtered := make([]model.MediaScanPath, 0, len(paths)) + for _, p := range paths { + if CanUserAccessPath(user, p.Path) { + filtered = append(filtered, p) + } + } + common.SuccessResp(c, filtered) +} + +// ==================== 导入 / 导出 ==================== + +// MediaExportData 导出数据结构 +// 包含版本号便于后续兼容性升级；导出粒度由 scope 字段说明： +// - "all" ：所有类型的全部数据 +// - "media_type" ：单个媒体类型（video/music/image/book）的全部数据 +// - "scan_path" ：单个扫描路径下的数据（仅 1 个路径 + 该路径下的条目） +type MediaExportData struct { + Version int `json:"version"` + ExportAt time.Time `json:"export_at"` + Scope string `json:"scope"` + MediaType model.MediaType `json:"media_type,omitempty"` + ScanPaths []model.MediaScanPath `json:"scan_paths"` + Items []model.MediaItem `json:"items"` +} + +const mediaExportVersion = 1 + +// ExportMediaDB 导出媒体数据 +// query: media_type（可选，为空表示全部类型）, scan_path_id（可选，单路径导出） +// 返回 JSON 文件下载流 +func ExportMediaDB(c *gin.Context) { + mediaType := model.MediaType(c.Query("media_type")) + scanPathIDStr := c.Query("scan_path_id") + + export := MediaExportData{ + Version: mediaExportVersion, + ExportAt: time.Now(), + } + + if scanPathIDStr != "" { + // 按扫描路径导出 + id, err := strconv.ParseUint(scanPathIDStr, 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的 scan_path_id", 400) + return + } + sp, err := db.GetMediaScanPath(uint(id)) + if err != nil { + common.ErrorResp(c, err, 404) + return + } + items, err := db.ListMediaItemsByScanPath(uint(id)) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + export.Scope = "scan_path" + export.MediaType = sp.MediaType + export.ScanPaths = []model.MediaScanPath{*sp} + export.Items = items + } else if mediaType != "" { + // 按媒体类型导出 + paths, err := db.ListMediaScanPaths(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + items, err := db.ListAllMediaItems(mediaType) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + export.Scope = "media_type" + export.MediaType = mediaType + export.ScanPaths = paths + export.Items = items + } else { + // 全部导出 + paths, err := db.ListAllMediaScanPaths() + if err != nil { + common.ErrorResp(c, err, 500) + return + } + items, err := db.ListAllMediaItems("") + if err != nil { + common.ErrorResp(c, err, 500) + return + } + export.Scope = "all" + export.ScanPaths = paths + export.Items = items + } + + // 直接以 JSON 流下载 + filename := buildExportFilename(export) + c.Header("Content-Disposition", `attachment; filename="`+filename+`"`) + c.Header("Content-Type", "application/json; charset=utf-8") + if err := json.NewEncoder(c.Writer).Encode(export); err != nil { + log.Errorf("导出媒体数据失败: %v", err) + } +} + +// buildExportFilename 根据导出范围生成下载文件名 +func buildExportFilename(d MediaExportData) string { + ts := d.ExportAt.Format("20060102_150405") + switch d.Scope { + case "scan_path": + spName := "" + if len(d.ScanPaths) > 0 { + spName = d.ScanPaths[0].Name + if spName == "" { + spName = d.ScanPaths[0].Path + } + } + // 去掉路径中的斜杠等字符，避免文件名非法 + safe := strings.NewReplacer("/", "_", "\\", "_", ":", "_", "?", "_", "*", "_", "\"", "_", "<", "_", ">", "_", "|", "_").Replace(spName) + return "media_export_" + string(d.MediaType) + "_" + safe + "_" + ts + ".json" + case "media_type": + return "media_export_" + string(d.MediaType) + "_" + ts + ".json" + default: + return "media_export_all_" + ts + ".json" + } +} + +// ImportMediaDB 导入媒体数据 +// 支持两种调用方式： +// 1. multipart/form-data：file 字段携带导出生成的 JSON 文件 +// 2. application/json：直接 POST 整个 MediaExportData 结构 +// +// 可选 query 参数 scan_path_id：当导入"scan_path"范围数据时， +// 把所有条目重定向到该扫描路径ID（覆盖文件中原有的 scan_path_id）。 +func ImportMediaDB(c *gin.Context) { + var data MediaExportData + + contentType := c.GetHeader("Content-Type") + if strings.HasPrefix(contentType, "multipart/form-data") { + fileHeader, err := c.FormFile("file") + if err != nil { + common.ErrorStrResp(c, "缺少 file 文件参数", 400) + return + } + f, err := fileHeader.Open() + if err != nil { + common.ErrorResp(c, err, 400) + return + } + defer f.Close() + if err := json.NewDecoder(f).Decode(&data); err != nil { + common.ErrorStrResp(c, "导入文件解析失败: "+err.Error(), 400) + return + } + } else { + if err := c.ShouldBindJSON(&data); err != nil { + common.ErrorResp(c, err, 400) + return + } + } + + if data.Version <= 0 || data.Version > mediaExportVersion { + common.ErrorStrResp(c, "不支持的导入文件版本", 400) + return + } + + // 1) 先导入扫描路径，建立 oldID -> newID 的映射 + idMap, spCreated, spUpdated, err := db.ImportMediaScanPaths(data.ScanPaths) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + + // 2) 再导入条目，根据 idMap 把 scan_path_id 重定向到新ID + // 修正每个条目的 scan_path_id： + // - 如果 query 指定了 scan_path_id，全部覆盖为该值（按扫描路径重定向导入） + // - 否则按 idMap 自动重定向；找不到映射的保留原值（通常发生在该路径未一并导入时） + scanPathIDStr := c.Query("scan_path_id") + var override *uint + if scanPathIDStr != "" { + if id, err := strconv.ParseUint(scanPathIDStr, 10, 64); err == nil { + v := uint(id) + override = &v + } + } + if override == nil { + for i := range data.Items { + if newID, ok := idMap[data.Items[i].ScanPathID]; ok { + data.Items[i].ScanPathID = newID + } + } + } + + itemCreated, itemUpdated, err := db.ImportMediaItems(data.Items, override) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + + common.SuccessResp(c, gin.H{ + "scan_paths_created": spCreated, + "scan_paths_updated": spUpdated, + "items_created": itemCreated, + "items_updated": itemUpdated, + }) +} diff --git a/server/handles/media_extra.go b/server/handles/media_extra.go new file mode 100644 index 00000000..54e0974d --- /dev/null +++ b/server/handles/media_extra.go @@ -0,0 +1,277 @@ +package handles + +import ( + "encoding/json" + "fmt" + "net/url" + stdpath "path" + "strconv" + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +// MediaStatus 媒体库状态（前端据此决定是否渲染媒体库入口/侧栏） +type MediaStatus struct { + Enabled bool `json:"enabled"` + Types []string `json:"types"` + EmbyEnabled bool `json:"emby_enabled"` + SubsonicEnabled bool `json:"subsonic_enabled"` + EmbyBaseURL string `json:"emby_base_url"` + SubsonicBaseURL string `json:"subsonic_base_url"` + StrmPath string `json:"strm_path"` + StrmOnScan bool `json:"strm_on_scan"` + StrmSameDir bool `json:"strm_same_dir"` + ThumbPath string `json:"thumb_path"` + Version string `json:"version"` + ProxyCover bool `json:"proxy_cover"` +} + +// PublicMediaStatus 公开的媒体库状态 +func PublicMediaStatus(c *gin.Context) { + // Types 必须是非 nil 切片：前端会直接调用 .includes()，nil 序列化成 null 会抛异常 + status := MediaStatus{Enabled: MediaEnabled(), Types: []string{}} + apiBase := common.GetApiUrl(c.Request.Context()) + status.StrmPath = media.DefaultStrmOptions(apiBase).OutputPath + status.StrmOnScan = setting.GetBool(conf.MediaStrmOnScan) + status.StrmSameDir = setting.GetBool(conf.MediaStrmSameDir) + status.ThumbPath = setting.GetStr(conf.MediaThumbnailPath, "/imgs") + status.Version = conf.Version + status.ProxyCover = setting.GetBool(conf.MediaProxyCover) + if !status.Enabled { + common.SuccessResp(c, status) + return + } + for _, mt := range MediaTypeEnabledList() { + status.Types = append(status.Types, string(mt)) + } + status.EmbyEnabled = MediaEmbyEnabled() + status.SubsonicEnabled = MediaSubsonicEnabled() + status.EmbyBaseURL = strings.TrimSuffix(apiBase, "/") + "/@emby" + status.SubsonicBaseURL = strings.TrimSuffix(apiBase, "/") + "/@rest" + common.SuccessResp(c, status) +} + +// ==================== 应用密钥 ==================== + +// ListMediaAppKeys 列出当前管理员的媒体应用密钥 +func ListMediaAppKeys(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "未登录", 401) + return + } + keys, err := db.ListMediaAppKeys(user.ID) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, keys) +} + +// CreateMediaAppKey 创建应用密钥（明文只在创建时返回一次） +func CreateMediaAppKey(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "未登录", 401) + return + } + var req struct { + Name string `json:"name"` + } + _ = c.ShouldBindJSON(&req) + name := strings.TrimSpace(req.Name) + if name == "" { + name = "Media Key" + } + plain, err := media.NewAppKeyPlain() + if err != nil { + common.ErrorResp(c, err, 500) + return + } + secret, err := media.EncryptAppKey(plain) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + key := &model.MediaAppKey{ + UserID: user.ID, + Name: name, + Prefix: media.AppKeyPrefix(plain), + Secret: secret, + } + if err := db.CreateMediaAppKey(key); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, gin.H{ + "id": key.ID, + "name": key.Name, + "prefix": key.Prefix, + "key": plain, + }) +} + +// DeleteMediaAppKey 删除应用密钥 +func DeleteMediaAppKey(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "未登录", 401) + return + } + id, err := strconv.ParseUint(c.Query("id"), 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + if err := db.DeleteMediaAppKey(user.ID, uint(id)); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, nil) +} + +// ==================== STRM ==================== + +// GenerateMediaStrm 手动批量生成 STRM +func GenerateMediaStrm(c *gin.Context) { + var req struct { + MediaType model.MediaType `json:"media_type"` + ScanPathID uint `json:"scan_path_id"` + } + _ = c.ShouldBindJSON(&req) + apiRoot := common.GetApiUrl(c.Request.Context()) + media.SetAPIRoot(apiRoot) + opts := media.DefaultStrmOptions(apiRoot) + if !opts.SameDir && opts.OutputPath == "" { + common.ErrorStrResp(c, "未配置 STRM 输出目录", 400) + return + } + n, err := media.GenerateStrmForType(c.Request.Context(), req.MediaType, req.ScanPathID, opts) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + outPath := opts.OutputPath + if opts.SameDir { + outPath = "与媒体文件同目录" + } + common.SuccessResp(c, gin.H{"written": n, "path": outPath}) +} + +// GetMediaStreamURL 返回条目（或其中一集）的签名直链，供本地播放器/外链使用 +func GetMediaStreamURL(c *gin.Context) { + // 直接按路径取签名链接（用于选集等场景） + if rawPath := c.Query("path"); rawPath != "" { + if !CanUserAccessPath(mediaUser(c), rawPath) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + common.SuccessResp(c, gin.H{ + "url": SignedStreamURL(c, rawPath), + "path": rawPath, + "name": stdpath.Base(rawPath), + "proxied": SignedProxyURL(c, rawPath), + }) + return + } + id, err := strconv.ParseUint(c.Query("id"), 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + item, err := db.GetMediaItemByID(uint(id)) + if err != nil || !CanUserAccessItem(mediaUser(c), item) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + episodeIndex, _ := strconv.Atoi(c.DefaultQuery("episode", "0")) + rawPath := MediaItemFilePath(item) + if episodeIndex > 0 { + var eps []model.EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil { + for i := range eps { + if eps[i].Index == episodeIndex || i+1 == episodeIndex { + rawPath = MediaEpisodeFilePath(item, &eps[i]) + break + } + } + } + } + // 选集是目录里的另一个文件，必须对最终文件再校验一次： + // 目录可见不代表其中每个文件都可见（例如 meta 隐藏规则单独禁止某个文件）， + // 而签名之后的下载链路只验签名，不会重新执行用户的隐藏规则。 + if !CanUserAccessPath(mediaUser(c), rawPath) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + common.SuccessResp(c, gin.H{ + "url": SignedStreamURL(c, rawPath), + "path": rawPath, + "name": stdpath.Base(rawPath), + "proxied": SignedProxyURL(c, rawPath), + }) +} + +// GetMediaStrm 动态生成单个条目的 .strm 内容（详情页下载） +func GetMediaStrm(c *gin.Context) { + id, err := strconv.ParseUint(c.Query("id"), 10, 64) + if err != nil { + common.ErrorStrResp(c, "无效的ID", 400) + return + } + item, err := db.GetMediaItemByID(uint(id)) + if err != nil { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + if !CanUserAccessItem(mediaUser(c), item) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + episodeIndex, _ := strconv.Atoi(c.DefaultQuery("episode", "0")) + + var rawPath string + var filename string + if episodeIndex > 0 { + var eps []model.EpisodeInfo + if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil { + for i := range eps { + if eps[i].Index == episodeIndex || i+1 == episodeIndex { + rawPath = MediaEpisodeFilePath(item, &eps[i]) + filename = fmt.Sprintf("%s S%02dE%02d.strm", item.ScrapedName, item.Season, episodeIndex) + break + } + } + } + } + if rawPath == "" { + rawPath = MediaItemFilePath(item) + if item.IsFolder { + filename = item.ScrapedName + ".strm" + } else { + filename = strings.TrimSuffix(item.FileName, stdpath.Ext(item.FileName)) + ".strm" + } + } + // 与 GetMediaStreamURL 同理：选集解析出来的是另一个文件，需要按最终路径再校验一次 + if !CanUserAccessPath(mediaUser(c), rawPath) { + common.ErrorStrResp(c, "资源不存在", 404) + return + } + if filename == "" { + filename = "media.strm" + } + + apiRoot := common.GetApiUrl(c.Request.Context()) + opts := media.DefaultStrmOptions(apiRoot) + content := media.RenderStrmContent(opts.Template, apiRoot, rawPath) + c.Header("Content-Type", "text/plain; charset=utf-8") + c.Header("Content-Disposition", fmt.Sprintf("attachment; filename*=UTF-8''%s", url.PathEscape(filename))) + c.String(200, content) +} diff --git a/server/handles/media_gate.go b/server/handles/media_gate.go new file mode 100644 index 00000000..c6fccbef --- /dev/null +++ b/server/handles/media_gate.go @@ -0,0 +1,222 @@ +package handles + +import ( + "crypto/subtle" + "fmt" + "net/url" + stdpath "path" + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/errs" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/op" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/internal/sign" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" + "github.com/pkg/errors" +) + +// MediaEnabled 媒体库总开关是否开启 +func MediaEnabled() bool { + return setting.GetBool(conf.MediaEnabled) +} + +// MediaEmbyEnabled Emby 兼容 API 是否可用（依赖总开关） +func MediaEmbyEnabled() bool { + return MediaEnabled() && setting.GetBool(conf.MediaEmbyEnabled) +} + +// MediaSubsonicEnabled OpenSubsonic API 是否可用（依赖总开关） +func MediaSubsonicEnabled() bool { + return MediaEnabled() && setting.GetBool(conf.MediaSubsonicEnabled) +} + +// RequireMediaEnabled 媒体库关闭时统一返回 404，使其表现为"功能不存在" +func RequireMediaEnabled(c *gin.Context) { + if !MediaEnabled() { + c.AbortWithStatus(404) + } +} + +// RequireMediaProtocol 校验指定协议开关 +func RequireMediaProtocol(enabled func() bool) gin.HandlerFunc { + return func(c *gin.Context) { + if !enabled() { + c.AbortWithStatus(404) + } + } +} + +// mediaUser 返回当前请求的登录用户，未登录时为 nil +func mediaUser(c *gin.Context) *model.User { + user, _ := c.Request.Context().Value(conf.UserKey).(*model.User) + return user +} + +// CanUserAccessPath 判断用户是否有权访问指定 VFS 路径 +// 规则与文件浏览保持一致：base_path 前缀 + 元信息(meta) 读权限 + 隐藏文件规则。 +// +// user 为 nil 一律拒绝（fail-closed）：所有调用点都应经过 Auth / embyAuth 拿到用户， +// 缺失用户说明鉴权链路出了问题，不能默认放行；被禁用的访客同样拒绝。 +func CanUserAccessPath(user *model.User, reqPath string) bool { + if user == nil || (user.IsGuest() && user.Disabled) { + return false + } + reqPath = utils.FixAndCleanPath(reqPath) + base := utils.FixAndCleanPath(user.BasePath) + if base == "" { + base = "/" + } + if base != "/" && !utils.IsSubPath(base, reqPath) && !utils.PathEqual(base, reqPath) { + return false + } + meta, err := op.GetNearestMeta(reqPath) + if err != nil && !errors.Is(errors.Cause(err), errs.MetaNotFound) { + return false + } + // media 播放链路无法携带目录密码，因此受密码保护的目录视为不可访问 + return common.CanAccess(user, meta, reqPath, "") +} + +// userPathPrefix 返回用于 SQL 前缀过滤的用户目录，全盘用户返回空串 +func userPathPrefix(user *model.User) string { + if user == nil { + return "" + } + base := utils.FixAndCleanPath(user.BasePath) + if base == "" || base == "/" { + return "" + } + return base +} + +// mediaFolderOf 返回条目所在目录 +func mediaFolderOf(item *model.MediaItem) string { + if item == nil { + return "/" + } + folder := strings.TrimSuffix(item.FolderPath, "/") + if folder == "" { + folder = "/" + } + return folder +} + +// MediaItemFilePath 拼接媒体条目对应的实际文件路径 +func MediaItemFilePath(item *model.MediaItem) string { + if item == nil { + return "" + } + folder := strings.TrimSuffix(item.FolderPath, "/") + if folder == "" { + folder = "/" + } + return stdpath.Join(folder, item.FileName) +} + +// MediaEpisodeFilePath 拼接文件夹型条目的单集文件路径 +func MediaEpisodeFilePath(item *model.MediaItem, ep *model.EpisodeInfo) string { + if item == nil || ep == nil { + return "" + } + return stdpath.Join(MediaItemFilePath(item), ep.FileName) +} + +// CanUserAccessItem 判断用户是否可访问某个媒体条目（含所属媒体类型开关） +func CanUserAccessItem(user *model.User, item *model.MediaItem) bool { + if item == nil || item.Hidden { + return false + } + if !MediaTypeEnabled(item.MediaType) { + return false + } + return CanUserAccessPath(user, MediaItemFilePath(item)) +} + +// MediaTypeEnabled 判断某媒体类型是否启用扫描 +func MediaTypeEnabled(mediaType model.MediaType) bool { + cfg, err := db.GetMediaConfig(mediaType) + if err != nil || cfg == nil { + return false + } + return cfg.Enabled +} + +// MediaTypeEnabledList 返回已启用的媒体类型 +func MediaTypeEnabledList() []model.MediaType { + all := []model.MediaType{model.MediaTypeVideo, model.MediaTypeMusic, model.MediaTypeImage, model.MediaTypeBook} + enabled := make([]model.MediaType, 0, len(all)) + for _, mt := range all { + if MediaTypeEnabled(mt) { + enabled = append(enabled, mt) + } + } + return enabled +} + +// LookUpAppKeyUser 按明文应用密钥查找对应用户与密钥记录。 +// Emby 与 Subsonic 的所有密钥认证都走这里，避免两套实现产生行为差异。 +func LookUpAppKeyUser(key string) (*model.User, *model.MediaAppKey, bool) { + key = strings.TrimSpace(key) + if !strings.HasPrefix(key, "olmp_") { + return nil, nil, false + } + keys, err := db.ListMediaAppKeysByPrefix(media.AppKeyPrefix(key)) + if err != nil { + return nil, nil, false + } + for i := range keys { + plain, err := media.DecryptAppKey(keys[i].Secret) + if err != nil { + continue + } + if subtle.ConstantTimeCompare([]byte(plain), []byte(key)) != 1 { + continue + } + user, err := op.GetUserById(keys[i].UserID) + if err != nil || user.Disabled || user.IsGuest() { + return nil, nil, false + } + db.TouchMediaAppKey(keys[i].ID) + return user, &keys[i], true + } + return nil, nil, false +} + +// SignedStreamURL 构造带签名的 /d 直链（绝对地址，供 Emby/Subsonic/STRM 使用） +func SignedStreamURL(c *gin.Context, rawPath string) string { + return SignedStreamURLWithBase(common.GetApiUrl(c.Request.Context()), rawPath) +} + +// SignedStreamURLWithBase 使用指定站点前缀构造签名直链 +func SignedStreamURLWithBase(apiBase, rawPath string) string { + apiBase = strings.TrimSuffix(apiBase, "/") + p := rawPath + if !strings.HasPrefix(p, "/") { + p = "/" + p + } + return fmt.Sprintf("%s/d%s?sign=%s", apiBase, utils.EncodePath(p), url.QueryEscape(sign.Sign(rawPath))) +} + +// SignedProxyURL 构造经服务端代理的签名链接（/p） +func SignedProxyURL(c *gin.Context, rawPath string) string { + apiBase := strings.TrimSuffix(common.GetApiUrl(c.Request.Context()), "/") + p := rawPath + if !strings.HasPrefix(p, "/") { + p = "/" + p + } + return fmt.Sprintf("%s/p%s?sign=%s", apiBase, utils.EncodePath(p), url.QueryEscape(sign.Sign(rawPath))) +} + +// MediaStreamURL 返回媒体条目（或其中一集）的签名直链 +func MediaStreamURL(c *gin.Context, item *model.MediaItem, ep *model.EpisodeInfo) string { + if ep != nil { + return SignedStreamURL(c, MediaEpisodeFilePath(item, ep)) + } + return SignedStreamURL(c, MediaItemFilePath(item)) +} diff --git a/server/handles/media_image.go b/server/handles/media_image.go new file mode 100644 index 00000000..87d50273 --- /dev/null +++ b/server/handles/media_image.go @@ -0,0 +1,201 @@ +package handles + +import ( + "context" + "fmt" + "io" + "net" + "net/http" + "net/url" + stdpath "path" + "path/filepath" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/media/imgmime" + "github.com/OpenListTeam/OpenList/v4/internal/media/scraper" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/pkg/utils" + "github.com/gin-gonic/gin" +) + +// 允许代理的封面来源域名（TMDB / 豆瓣 / Discogs）。 +// 仅代理图片，避免该接口被当作通用反向代理滥用。 +var coverAllowedHosts = []string{ + "image.tmdb.org", + "tmdb.org", + "doubanio.com", + "douban.com", + "discogs.com", +} + +func hostAllowed(host string) bool { + host = strings.ToLower(strings.TrimSuffix(host, ".")) + if host == "" || host == "localhost" || strings.HasSuffix(host, ".local") { + return false + } + // 显式拒绝 IP 字面量，避免白名单域名被解析/重定向到内网 + if ip := net.ParseIP(host); ip != nil { + return false + } + for _, h := range coverAllowedHosts { + if host == h || strings.HasSuffix(host, "."+h) { + return true + } + } + return false +} + +// safeCoverURL 校验封面地址的协议与域名（含重定向后的每一跳） +func safeCoverURL(u *url.URL) error { + if u == nil { + return fmt.Errorf("empty url") + } + if u.Scheme != "http" && u.Scheme != "https" { + return fmt.Errorf("unsupported scheme") + } + if !hostAllowed(u.Hostname()) { + return fmt.Errorf("host not allowed") + } + return nil +} + +// PublicMediaProxy 代理外部封面图（TMDB/豆瓣/Discogs），解决客户端直连被墙的问题。 +// 用法：/api/public/media/proxy?url=https%3A%2F%2Fimage.tmdb.org%2F... +func PublicMediaProxy(c *gin.Context) { + if !MediaEnabled() { + c.Status(http.StatusNotFound) + return + } + raw := c.Query("url") + if raw == "" { + c.Status(http.StatusBadRequest) + return + } + u, err := url.Parse(raw) + if err != nil || safeCoverURL(u) != nil { + c.Status(http.StatusForbidden) + return + } + resp, err := fetchCover(c.Request.Context(), u.String()) + if err != nil && setting.GetBool(conf.MediaCoverGateway) { + // 直连失败时回退到公共图片网关（部分网络环境下 TMDB 图片域名不可达）。 + // 这会把封面 URL 交给第三方 wsrv.nl，因此默认关闭，需管理员显式开启。 + gateway := "https://wsrv.nl/?url=" + url.QueryEscape(strings.TrimPrefix(strings.TrimPrefix(u.String(), "https://"), "http://")) + resp, err = fetchCover(c.Request.Context(), gateway) + } + if err != nil { + utils.Log.Warnf("media cover proxy failed: %s: %v", u.String(), err) + c.Status(http.StatusBadGateway) + return + } + defer resp.Body.Close() + // 不信任上游 Content-Type：读取有限字节后按魔数判定，只输出四种栅格图片， + // 避免 image/svg+xml 等可执行脚本的类型以本站源被浏览器渲染。 + data, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil || len(data) == 0 { + c.Status(http.StatusBadGateway) + return + } + ct := imgmime.Sniff(data) + if ct == "" { + c.Status(http.StatusUnsupportedMediaType) + return + } + setSafeImageHeaders(c) + c.Header("Cache-Control", "public, max-age=604800") + c.Data(http.StatusOK, ct, data) +} + +// setSafeImageHeaders 为用户可影响内容的图片响应加上防 XSS 头。 +func setSafeImageHeaders(c *gin.Context) { + c.Header("X-Content-Type-Options", "nosniff") + c.Header("Content-Disposition", `inline; filename="cover"`) + c.Header("Content-Security-Policy", "default-src 'none'; sandbox") +} + +// fetchCover 拉取图片，仅接受 200 且 Content-Type 为 image/* +// 重定向的每一跳都重新做域名校验，避免白名单域名 302 到内网造成 SSRF。 +func fetchCover(ctx context.Context, raw string) (*http.Response, error) { + client := &http.Client{ + Timeout: 20 * time.Second, + CheckRedirect: func(req *http.Request, via []*http.Request) error { + if len(via) >= 3 { + return fmt.Errorf("too many redirects") + } + return safeCoverURL(req.URL) + }, + } + req, err := http.NewRequestWithContext(ctx, http.MethodGet, raw, nil) + if err != nil { + return nil, err + } + req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)") + resp, err := client.Do(req) + if err != nil { + return nil, err + } + if resp.StatusCode != http.StatusOK { + _ = resp.Body.Close() + return nil, fmt.Errorf("unexpected status %d", resp.StatusCode) + } + return resp, nil +} + +// thumbnailRoot 返回运行目录模式下缩略图的本地根目录（/） +func thumbnailRoot() string { + dir := setting.GetStr(conf.MediaThumbnailPath, "imgs") + return scraper.RuntimeThumbRoot(dir) +} + +// PublicMediaThumb 读取本地缩略图目录中的文件。 +// 缩略图由刮削器写入本地磁盘（非 VFS），因此不能用 /d 读取。 +// 用法：/api/public/media/thumb?path=/imgs/xxx.jpg +func PublicMediaThumb(c *gin.Context) { + if !MediaEnabled() { + c.Status(http.StatusNotFound) + return + } + reqPath := strings.TrimPrefix(stdpath.Clean("/"+c.Query("path")), "/") + // stdpath.Clean 已消解 ../ 段；这里不能再用 Contains("..") 粗暴过滤，会误杀 a..b.jpg 这类合法文件名。 + // 越界由下方 filepath.Rel 校验兜底。 + if reqPath == "" || reqPath == "." { + c.Status(http.StatusBadRequest) + return + } + base := strings.Trim(setting.GetStr(conf.MediaThumbnailPath, "/imgs"), "/") + if base != "" { + if !strings.HasPrefix(reqPath, base+"/") { + c.Status(http.StatusNotFound) + return + } + reqPath = strings.TrimPrefix(reqPath, base+"/") + } + root := thumbnailRoot() + full := filepath.Join(root, filepath.FromSlash(reqPath)) + rel, err := filepath.Rel(root, full) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + c.Status(http.StatusForbidden) + return + } + // 只按扩展名白名单输出栅格图片，拒绝 svg / html 等可执行类型 + var ct string + switch strings.ToLower(filepath.Ext(full)) { + case ".jpg", ".jpeg": + ct = "image/jpeg" + case ".png": + ct = "image/png" + case ".gif": + ct = "image/gif" + case ".webp": + ct = "image/webp" + default: + c.Status(http.StatusUnsupportedMediaType) + return + } + setSafeImageHeaders(c) + c.Header("Content-Type", ct) + c.Header("Cache-Control", "public, max-age=604800") + c.File(full) +} diff --git a/server/handles/media_ratelimit.go b/server/handles/media_ratelimit.go new file mode 100644 index 00000000..b8a37d70 --- /dev/null +++ b/server/handles/media_ratelimit.go @@ -0,0 +1,92 @@ +package handles + +import ( + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/go-cache" + "github.com/gin-gonic/gin" +) + +// 媒体协议（Emby / Subsonic）的认证入口不经过 /api/auth/login， +// 因此必须自行复用 LoginCache 的失败计数与锁定策略，否则等于附送一个无限次数的爆破接口。 +// 使用独立的 key 前缀，避免 Subsonic 客户端的失败重试把 Web 登录一并锁死。 + +func mediaAuthKey(c *gin.Context) string { + return "media:" + c.ClientIP() +} + +// mediaAuthLocked 当前 IP 是否已达到失败上限（需要拒绝认证尝试） +// 注意：这里绝不能刷新过期时间。客户端（尤其是 Subsonic）会持续轮询重试， +// 每次被拒都续期会把 IP 永久锁死，连之后正确的密钥也无法登录。 +func mediaAuthLocked(c *gin.Context) bool { + count, ok := model.LoginCache.Get(mediaAuthKey(c)) + return ok && count >= model.DefaultMaxAuthRetries +} + +// bumpMediaAuthCount 增加一次失败计数。 +// 计数必须带 TTL：key 里包含 IP / 用户名（后者来自请求参数，可被随意伪造）， +// 不带过期时间会让缓存随请求无限增长；固定窗口也让锁定到期后计数自动归零。 +// 达到上限后请求会被 mediaAuthLocked / mediaAuthLockedFor 拦截，不会再走到这里， +// 因此锁定窗口不会被持续刷新。 +func bumpMediaAuthCount(key string) { + count, _ := model.LoginCache.Get(key) + count++ + model.LoginCache.Set(key, count, cache.WithEx[int](model.DefaultLockDuration)) +} + +// mediaAuthFailed 记录一次认证失败（IP 维度） +func mediaAuthFailed(c *gin.Context) { + bumpMediaAuthCount(mediaAuthKey(c)) +} + +// mediaAuthSucceeded 认证成功后清除失败计数 +func mediaAuthSucceeded(c *gin.Context) { + model.LoginCache.Del(mediaAuthKey(c)) +} + +// ==================== 用户名维度 ==================== +// 只按 IP 计数是可以被绕过的：gin 默认信任所有代理（仓库里没有设置 +// SetTrustedProxies），客户端自带 X-Forwarded-For 就能改写 c.ClientIP()， +// 轮换这个头即可无限次尝试。因此再按用户名记一份失败次数：它与 IP 无关， +// 即使 IP 被伪造，对同一账号的爆破依然会被锁住。 +// （代理信任范围属于项目层面的问题，主干 /api/auth/login 现状相同。） + +func mediaAuthUserKey(username string) string { + return "media:user:" + strings.ToLower(strings.TrimSpace(username)) +} + +// mediaAuthLockedFor 本次请求是否应被限流：IP 或用户名任一维度达到上限即拒绝。 +func mediaAuthLockedFor(c *gin.Context, username string) bool { + if mediaAuthLocked(c) { + return true + } + if strings.TrimSpace(username) == "" { + return false + } + count, ok := model.LoginCache.Get(mediaAuthUserKey(username)) + return ok && count >= model.DefaultMaxAuthRetries +} + +// bumpMediaAuthUser 给用户名维度记一次失败 +func bumpMediaAuthUser(username string) { + name := strings.TrimSpace(username) + if name == "" { + return + } + bumpMediaAuthCount(mediaAuthUserKey(name)) +} + +// mediaAuthFailedFor 记录一次认证失败：同时计入 IP 与用户名两个维度。 +func mediaAuthFailedFor(c *gin.Context, username string) { + mediaAuthFailed(c) + bumpMediaAuthUser(username) +} + +// mediaAuthSucceededFor 认证成功后清除 IP 与用户名两个维度的计数。 +func mediaAuthSucceededFor(c *gin.Context, username string) { + mediaAuthSucceeded(c) + if name := strings.TrimSpace(username); name != "" { + model.LoginCache.Del(mediaAuthUserKey(name)) + } +} diff --git a/server/handles/media_reader.go b/server/handles/media_reader.go new file mode 100644 index 00000000..b5b6f1b2 --- /dev/null +++ b/server/handles/media_reader.go @@ -0,0 +1,140 @@ +package handles + +import ( + "strings" + + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +const ( + // 单个条目最多保存的批注数，防止恶意请求把表撑大 + maxMediaAnnotations = 500 + // 单条批注对象（含绘制点集）的 JSON 上限 + maxMediaAnnotationAnchor = 512 << 10 + // 单次请求的批注对象总量上限，避免一次塞进几百个大体积批注 + maxMediaAnnotationTotal = 8 << 20 +) + +// mediaAnnotationTarget 校验条目存在且当前用户可以写入批注。 +// 批注是用户私有数据，不允许访客：访客是公共账号，多设备之间的批注会互相覆盖。 +func mediaAnnotationTarget(c *gin.Context, itemID uint) (*model.MediaItem, *model.User, bool) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "unauthorized", 401) + return nil, nil, false + } + if user.IsGuest() { + common.ErrorStrResp(c, "guest cannot save annotations", 403) + return nil, nil, false + } + if itemID == 0 { + common.ErrorStrResp(c, "invalid request", 400) + return nil, nil, false + } + item, err := db.GetMediaItemByID(itemID) + if err != nil || item.Hidden || item.DeletedAt.Valid || !CanUserAccessItem(user, item) { + common.ErrorStrResp(c, "not found", 404) + return nil, nil, false + } + return item, user, true +} + +// ListMediaAnnotationsAPI 列出当前用户对某个条目保存的批注 +// GET /fs/media/annotations?item_id=1 +func ListMediaAnnotationsAPI(c *gin.Context) { + user := mediaUser(c) + itemID := uint(atoiDefault(c.Query("item_id"), 0)) + // 未登录 / 访客 / 无权访问一律返回空列表：这是私有数据，不对访客暴露 + if user == nil || user.IsGuest() || itemID == 0 { + common.SuccessResp(c, []model.MediaAnnotation{}) + return + } + item, err := db.GetMediaItemByID(itemID) + if err != nil || item.Hidden || item.DeletedAt.Valid || !CanUserAccessItem(user, item) { + common.SuccessResp(c, []model.MediaAnnotation{}) + return + } + list, err := db.ListMediaAnnotations(user.ID, item.ID) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, list) +} + +type mediaAnnotationPayload struct { + RemoteID string `json:"remote_id"` + Anchor string `json:"anchor"` +} + +type mediaAnnotationSyncReq struct { + ItemID uint `json:"item_id"` + Upserts []mediaAnnotationPayload `json:"upserts"` + Deletes []string `json:"deletes"` +} + +// SyncMediaAnnotationsAPI 增量保存批注：upserts 覆盖同一 remote_id，deletes 按 remote_id 删除。 +// 查看器每次改动只会带上变化的那几条，不需要整份文档重传。 +// POST /fs/media/annotations +func SyncMediaAnnotationsAPI(c *gin.Context) { + var req mediaAnnotationSyncReq + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorStrResp(c, "invalid request", 400) + return + } + item, user, ok := mediaAnnotationTarget(c, req.ItemID) + if !ok { + return + } + if len(req.Upserts) > maxMediaAnnotations || len(req.Deletes) > maxMediaAnnotations { + common.ErrorStrResp(c, "too many annotations", 400) + return + } + + upserts := make([]model.MediaAnnotation, 0, len(req.Upserts)) + seen := make(map[string]struct{}, len(req.Upserts)) + anchorBytes := 0 + for _, p := range req.Upserts { + remoteID := strings.TrimSpace(p.RemoteID) + // remote_id 是查看器分配的批注主键，多设备同步时靠它去重，必须存在且唯一 + if remoteID == "" || p.Anchor == "" || len(remoteID) > 128 { + common.ErrorStrResp(c, "invalid annotation", 400) + return + } + if _, dup := seen[remoteID]; dup { + continue + } + seen[remoteID] = struct{}{} + if len(p.Anchor) > maxMediaAnnotationAnchor { + common.ErrorStrResp(c, "annotation too large", 400) + return + } + anchorBytes += len(p.Anchor) + if anchorBytes > maxMediaAnnotationTotal { + common.ErrorStrResp(c, "annotations too large", 400) + return + } + upserts = append(upserts, model.MediaAnnotation{ + RemoteID: remoteID, + Anchor: p.Anchor, + }) + } + + deletes := make([]string, 0, len(req.Deletes)) + for _, id := range req.Deletes { + id = strings.TrimSpace(id) + if id == "" || len(id) > 128 { + continue + } + deletes = append(deletes, id) + } + + if err := db.SyncMediaAnnotations(user.ID, item.ID, upserts, deletes); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, gin.H{"saved": len(upserts), "deleted": len(deletes)}) +} diff --git a/server/handles/media_user.go b/server/handles/media_user.go new file mode 100644 index 00000000..87ba65ad --- /dev/null +++ b/server/handles/media_user.go @@ -0,0 +1,268 @@ +package handles + +import ( + "strconv" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +const mediaTicksPerSecond = 10_000_000 + +// GetMediaUserDataBatch 批量查询当前用户对条目的播放状态（收藏/进度/已看） +// GET /fs/media/userdata?ids=1,2,3 +func GetMediaUserDataBatch(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.SuccessResp(c, []model.MediaUserData{}) + return + } + raw := c.Query("ids") + ids := make([]uint, 0) + for _, part := range strings.Split(raw, ",") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + if id, err := strconv.ParseUint(part, 10, 64); err == nil { + ids = append(ids, uint(id)) + } + } + if len(ids) == 0 { + common.SuccessResp(c, []model.MediaUserData{}) + return + } + // 限制批量查询数量，避免超长 IN 子句造成数据库压力 + if len(ids) > 500 { + ids = ids[:500] + } + list, err := db.ListMediaUserDataByItems(user.ID, ids) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, list) +} + +// GetMediaProgress 查询单个条目（或选集）的播放进度 +func GetMediaProgress(c *gin.Context) { + user := mediaUser(c) + itemID := uint(atoiDefault(c.Query("item_id"), 0)) + episode := atoiDefault(c.Query("episode_index"), 0) + if user == nil || itemID == 0 { + common.SuccessResp(c, nil) + return + } + item, err := db.GetMediaItemByID(itemID) + if err != nil || item.Hidden || !CanUserAccessItem(user, item) { + common.SuccessResp(c, nil) + return + } + data, err := db.GetMediaUserData(user.ID, itemID, episode) + if err != nil { + common.SuccessResp(c, nil) + return + } + common.SuccessResp(c, data) +} + +// SaveMediaProgress 上报播放进度 +type saveProgressReq struct { + ItemID uint `json:"item_id"` + EpisodeIndex int `json:"episode_index"` + PositionSec float64 `json:"position"` + DurationSec float64 `json:"duration"` + Played bool `json:"played"` +} + +func SaveMediaProgress(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + var req saveProgressReq + if err := c.ShouldBindJSON(&req); err != nil || req.ItemID == 0 { + common.ErrorStrResp(c, "invalid request", 400) + return + } + item, err := db.GetMediaItemByID(req.ItemID) + if err != nil || item.Hidden || !CanUserAccessItem(user, item) { + common.ErrorStrResp(c, "not found", 404) + return + } + data, err := db.GetMediaUserData(user.ID, req.ItemID, req.EpisodeIndex) + if err != nil { + data = &model.MediaUserData{ + UserID: user.ID, + ItemID: req.ItemID, + EpisodeIndex: req.EpisodeIndex, + } + } + now := time.Now() + data.PositionTicks = int64(req.PositionSec * mediaTicksPerSecond) + if req.DurationSec > 0 { + data.PositionTicks = int64(minFloat(req.PositionSec, req.DurationSec) * mediaTicksPerSecond) + } + data.LastPlayedAt = &now + finished := req.Played + if req.DurationSec > 0 && req.PositionSec/req.DurationSec >= 0.9 { + finished = true + } + if finished && !data.Played { + data.Played = true + data.PlayCount++ + } + // 重新播放时清空已看标记 + if !finished && req.PositionSec < 5 && data.Played { + data.Played = false + } + if err := db.SaveMediaUserData(data); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, data) +} + +// SetMediaFavorite 收藏 / 取消收藏 +type favoriteReq struct { + ItemID uint `json:"item_id"` + EpisodeIndex int `json:"episode_index"` + Favorite bool `json:"favorite"` +} + +func SetMediaFavorite(c *gin.Context) { + user := mediaUser(c) + if user == nil { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + var req favoriteReq + if err := c.ShouldBindJSON(&req); err != nil || req.ItemID == 0 { + common.ErrorStrResp(c, "invalid request", 400) + return + } + item, err := db.GetMediaItemByID(req.ItemID) + if err != nil || item.Hidden || !CanUserAccessItem(user, item) { + common.ErrorStrResp(c, "not found", 404) + return + } + data, err := db.GetMediaUserData(user.ID, req.ItemID, req.EpisodeIndex) + if err != nil { + data = &model.MediaUserData{ + UserID: user.ID, + ItemID: req.ItemID, + EpisodeIndex: req.EpisodeIndex, + } + } + data.IsFavorite = req.Favorite + if err := db.SaveMediaUserData(data); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c, gin.H{"favorite": data.IsFavorite}) +} + +// ListMediaFavoritesAPI 列出当前用户收藏的媒体条目 +// GET /fs/media/favorites?media_type=video&page=1&page_size=20 +func ListMediaFavoritesAPI(c *gin.Context) { + user := mediaUser(c) + mediaType := model.MediaType(c.Query("media_type")) + page := atoiDefault(c.DefaultQuery("page", "1"), 1) + pageSize := atoiDefault(c.DefaultQuery("page_size", "20"), 20) + if page < 1 { + page = 1 + } + if pageSize <= 0 || pageSize > 100 { + pageSize = 20 + } + items := []model.MediaItem{} + if user != nil { + data, err := db.ListMediaFavorites(user.ID) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + for i := range data { + row := data[i] + if row.EpisodeIndex != 0 { + continue + } + item, err := db.GetMediaItemByID(row.ItemID) + if err != nil || item.Hidden || item.DeletedAt.Valid { + continue + } + if mediaType != "" && item.MediaType != mediaType { + continue + } + if !CanUserAccessItem(user, item) { + continue + } + items = append(items, *item) + } + } + total := len(items) + start := (page - 1) * pageSize + if start > total { + start = total + } + end := start + pageSize + if end > total { + end = total + } + common.SuccessResp(c, common.PageResp{Content: items[start:end], Total: int64(total)}) +} + +// ListMediaResumeAPI 继续观看列表 +func ListMediaResumeAPI(c *gin.Context) { + user := mediaUser(c) + mediaType := model.MediaType(c.Query("media_type")) + limit := atoiDefault(c.DefaultQuery("limit", "20"), 20) + items := []model.MediaItem{} + if user != nil { + data, err := db.ListMediaResume(user.ID, 200) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + for i := range data { + row := data[i] + if row.PositionTicks <= 0 { + continue + } + item, err := db.GetMediaItemByID(row.ItemID) + if err != nil || item.Hidden || item.DeletedAt.Valid { + continue + } + if mediaType != "" && item.MediaType != mediaType { + continue + } + if !CanUserAccessItem(user, item) { + continue + } + items = append(items, *item) + if len(items) >= limit { + break + } + } + } + common.SuccessResp(c, items) +} + +func atoiDefault(s string, def int) int { + if v, err := strconv.Atoi(s); err == nil { + return v + } + return def +} + +func minFloat(a, b float64) float64 { + if a < b { + return a + } + return b +} diff --git a/server/handles/subsonic.go b/server/handles/subsonic.go new file mode 100644 index 00000000..01c0f131 --- /dev/null +++ b/server/handles/subsonic.go @@ -0,0 +1,772 @@ +package handles + +import ( + "encoding/hex" + "encoding/json" + "encoding/xml" + "fmt" + stdpath "path" + "sort" + "strconv" + "strings" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/op" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/gin-gonic/gin" +) + +const subsonicAPIVersion = "1.16.1" + +// ==================== 响应结构 ==================== + +type subsonicError struct { + Code int `json:"code" xml:"code,attr"` + Message string `json:"message" xml:"message,attr"` +} + +type subsonicResponse struct { + XMLName xml.Name `json:"-" xml:"subsonic-response"` + + Status string `json:"status" xml:"status,attr"` + Version string `json:"version" xml:"version,attr"` + Type string `json:"type" xml:"type,attr"` + ServerVersion string `json:"serverVersion" xml:"serverVersion,attr"` + OpenSubsonic bool `json:"openSubsonic" xml:"openSubsonic,attr"` + + Error *subsonicError `json:"error,omitempty" xml:"error,omitempty"` + + License *subsonicLicense `json:"license,omitempty" xml:"license,omitempty"` + MusicFolders *subsonicMusicFolders `json:"musicFolders,omitempty" xml:"musicFolders,omitempty"` + Indexes *subsonicIndexes `json:"indexes,omitempty" xml:"indexes,omitempty"` + Artists *subsonicArtists `json:"artists,omitempty" xml:"artists,omitempty"` + Artist *subsonicArtist `json:"artist,omitempty" xml:"artist,omitempty"` + Album *subsonicAlbum `json:"album,omitempty" xml:"album,omitempty"` + Song *subsonicChild `json:"song,omitempty" xml:"song,omitempty"` + AlbumList *subsonicAlbumList `json:"albumList,omitempty" xml:"albumList,omitempty"` + AlbumList2 *subsonicAlbumList `json:"albumList2,omitempty" xml:"albumList2,omitempty"` + SearchResult2 *subsonicSearchResult `json:"searchResult2,omitempty" xml:"searchResult2,omitempty"` + SearchResult3 *subsonicSearchResult `json:"searchResult3,omitempty" xml:"searchResult3,omitempty"` + Playlists *subsonicPlaylists `json:"playlists,omitempty" xml:"playlists,omitempty"` + Playlist *subsonicPlaylist `json:"playlist,omitempty" xml:"playlist,omitempty"` + Lyrics *subsonicLyrics `json:"lyrics,omitempty" xml:"lyrics,omitempty"` + LyricsList *subsonicLyricsList `json:"lyricsList,omitempty" xml:"lyricsList,omitempty"` + Starred *subsonicStarred `json:"starred,omitempty" xml:"starred,omitempty"` + Starred2 *subsonicStarred `json:"starred2,omitempty" xml:"starred2,omitempty"` + Genres *subsonicGenres `json:"genres,omitempty" xml:"genres,omitempty"` + RandomSongs *subsonicSongs `json:"randomSongs,omitempty" xml:"randomSongs,omitempty"` + SongsByGenre *subsonicSongs `json:"songsByGenre,omitempty" xml:"songsByGenre,omitempty"` + ScanStatus *subsonicScanStatus `json:"scanStatus,omitempty" xml:"scanStatus,omitempty"` + User *subsonicUser `json:"user,omitempty" xml:"user,omitempty"` + Users *subsonicUsers `json:"users,omitempty" xml:"users,omitempty"` + OpenSubsonicExtensions *subsonicExtensions `json:"openSubsonicExtensions,omitempty" xml:"openSubsonicExtensions,omitempty"` + NowPlaying *subsonicNowPlaying `json:"nowPlaying,omitempty" xml:"nowPlaying,omitempty"` + PlayQueue *subsonicPlayQueue `json:"playQueue,omitempty" xml:"playQueue,omitempty"` + ArtistInfo2 *subsonicArtistInfo `json:"artistInfo2,omitempty" xml:"artistInfo2,omitempty"` + AlbumInfo *subsonicAlbumInfo `json:"albumInfo,omitempty" xml:"albumInfo,omitempty"` + TopSongs *subsonicSongs `json:"topSongs,omitempty" xml:"topSongs,omitempty"` + SimilarSongs2 *subsonicSongs `json:"similarSongs2,omitempty" xml:"similarSongs2,omitempty"` + MusicDirectory *subsonicDirectory `json:"directory,omitempty" xml:"directory,omitempty"` + Bookmarks *subsonicBookmarks `json:"bookmarks,omitempty" xml:"bookmarks,omitempty"` +} + +type subsonicLicense struct { + Valid bool `json:"valid" xml:"valid,attr"` +} + +type subsonicMusicFolder struct { + Id string `json:"id" xml:"id,attr"` + Name string `json:"name" xml:"name,attr"` +} + +type subsonicMusicFolders struct { + MusicFolder []subsonicMusicFolder `json:"musicFolder" xml:"musicFolder"` +} + +type subsonicIndex struct { + Name string `json:"name" xml:"name,attr"` + Artist []subsonicChild `json:"artist" xml:"artist"` +} + +type subsonicIndexes struct { + LastModified int64 `json:"lastModified" xml:"lastModified,attr"` + IgnoredArticles string `json:"ignoredArticles" xml:"ignoredArticles,attr"` + Index []subsonicIndex `json:"index" xml:"index"` +} + +type subsonicArtist struct { + Id string `json:"id" xml:"id,attr"` + Name string `json:"name" xml:"name,attr"` + AlbumCount int `json:"albumCount,omitempty" xml:"albumCount,attr,omitempty"` + CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"` + Album []subsonicAlbum `json:"album" xml:"album"` +} + +type subsonicArtists struct { + IgnoredArticles string `json:"ignoredArticles" xml:"ignoredArticles,attr"` + Index []subsonicIndex2 `json:"index" xml:"index"` +} + +type subsonicIndex2 struct { + Name string `json:"name" xml:"name,attr"` + Artist []subsonicArtist `json:"artist" xml:"artist"` +} + +type subsonicAlbum struct { + Id string `json:"id" xml:"id,attr"` + Name string `json:"name" xml:"name,attr"` + Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"` + ArtistId string `json:"artistId,omitempty" xml:"artistId,attr,omitempty"` + CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"` + SongCount int `json:"songCount" xml:"songCount,attr"` + Duration int `json:"duration" xml:"duration,attr"` + Created string `json:"created" xml:"created,attr"` + Year int `json:"year,omitempty" xml:"year,attr,omitempty"` + Song []subsonicChild `json:"song" xml:"song"` +} + +type subsonicChild struct { + Id string `json:"id" xml:"id,attr"` + Parent string `json:"parent,omitempty" xml:"parent,attr,omitempty"` + IsDir bool `json:"isDir" xml:"isDir,attr"` + Title string `json:"title" xml:"title,attr"` + Album string `json:"album,omitempty" xml:"album,attr,omitempty"` + Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"` + Track int `json:"track,omitempty" xml:"track,attr,omitempty"` + Year int `json:"year,omitempty" xml:"year,attr,omitempty"` + Genre string `json:"genre,omitempty" xml:"genre,attr,omitempty"` + CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"` + Size int64 `json:"size,omitempty" xml:"size,attr,omitempty"` + ContentType string `json:"contentType,omitempty" xml:"contentType,attr,omitempty"` + Suffix string `json:"suffix,omitempty" xml:"suffix,attr,omitempty"` + Duration int `json:"duration,omitempty" xml:"duration,attr,omitempty"` + BitRate int `json:"bitRate,omitempty" xml:"bitRate,attr,omitempty"` + Path string `json:"path,omitempty" xml:"path,attr,omitempty"` + AlbumId string `json:"albumId,omitempty" xml:"albumId,attr,omitempty"` + ArtistId string `json:"artistId,omitempty" xml:"artistId,attr,omitempty"` + Type string `json:"type,omitempty" xml:"type,attr,omitempty"` + IsVideo bool `json:"isVideo,omitempty" xml:"isVideo,attr,omitempty"` + PlayCount int `json:"playCount,omitempty" xml:"playCount,attr,omitempty"` + DiscNumber int `json:"discNumber,omitempty" xml:"discNumber,attr,omitempty"` + Created string `json:"created,omitempty" xml:"created,attr,omitempty"` + Starred string `json:"starred,omitempty" xml:"starred,attr,omitempty"` + UserRating int `json:"userRating,omitempty" xml:"userRating,attr,omitempty"` +} + +type subsonicAlbumList struct { + Album []subsonicAlbum `json:"album" xml:"album"` +} + +type subsonicSearchResult struct { + Offset int `json:"offset" xml:"offset,attr"` + Total int `json:"total" xml:"total,attr"` + Artist []subsonicArtist `json:"artist" xml:"artist"` + Album []subsonicAlbum `json:"album" xml:"album"` + Song []subsonicChild `json:"song" xml:"song"` +} + +type subsonicSongs struct { + Song []subsonicChild `json:"song" xml:"song"` +} + +type subsonicDirectory struct { + Id string `json:"id" xml:"id,attr"` + Parent string `json:"parent,omitempty" xml:"parent,attr,omitempty"` + Name string `json:"name" xml:"name,attr"` + Starred string `json:"starred,omitempty" xml:"starred,attr,omitempty"` + Child []subsonicChild `json:"child" xml:"child"` +} + +type subsonicPlaylists struct { + Playlist []subsonicPlaylist `json:"playlist" xml:"playlist"` +} + +type subsonicPlaylist struct { + Id string `json:"id" xml:"id,attr"` + Name string `json:"name" xml:"name,attr"` + Comment string `json:"comment,omitempty" xml:"comment,attr,omitempty"` + Owner string `json:"owner,omitempty" xml:"owner,attr,omitempty"` + Public bool `json:"public" xml:"public,attr"` + SongCount int `json:"songCount" xml:"songCount,attr"` + Duration int `json:"duration" xml:"duration,attr"` + Created string `json:"created" xml:"created,attr"` + Entry []subsonicChild `json:"entry" xml:"entry"` +} + +type subsonicLyrics struct { + Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"` + Title string `json:"title,omitempty" xml:"title,attr,omitempty"` + Value string `json:"value" xml:",chardata"` +} + +type subsonicLyricsList struct { + StructuredLyrics []subsonicStructuredLyrics `json:"structuredLyrics" xml:"structuredLyrics"` +} + +type subsonicStructuredLyrics struct { + DisplayArtist string `json:"displayArtist,omitempty" xml:"displayArtist,attr,omitempty"` + DisplayTitle string `json:"displayTitle,omitempty" xml:"displayTitle,attr,omitempty"` + Lang string `json:"lang" xml:"lang,attr"` + Synced bool `json:"synced" xml:"synced,attr"` + Line []subsonicLyricLine `json:"line" xml:"line"` +} + +type subsonicLyricLine struct { + Start int64 `json:"start" xml:"start,attr"` + Value string `json:"value" xml:"value,attr"` +} + +type subsonicStarred struct { + Artist []subsonicArtist `json:"artist" xml:"artist"` + Album []subsonicAlbum `json:"album" xml:"album"` + Song []subsonicChild `json:"song" xml:"song"` +} + +type subsonicGenre struct { + Value string `json:"value" xml:",chardata"` + SongCount int `json:"songCount" xml:"songCount,attr"` + AlbumCount int `json:"albumCount" xml:"albumCount,attr"` +} + +type subsonicGenres struct { + Genre []subsonicGenre `json:"genre" xml:"genre"` +} + +type subsonicScanStatus struct { + Scanning bool `json:"scanning" xml:"scanning,attr"` + Count int64 `json:"count" xml:"count,attr"` +} + +type subsonicUser struct { + Username string `json:"username" xml:"username,attr"` + Email string `json:"email,omitempty" xml:"email,attr,omitempty"` + ScrobblingEnabled bool `json:"scrobblingEnabled" xml:"scrobblingEnabled,attr"` + AdminRole bool `json:"adminRole" xml:"adminRole,attr"` + DownloadRole bool `json:"downloadRole" xml:"downloadRole,attr"` + PlaylistRole bool `json:"playlistRole" xml:"playlistRole,attr"` + StreamRole bool `json:"streamRole" xml:"streamRole,attr"` + CoverArtRole bool `json:"coverArtRole" xml:"coverArtRole,attr"` +} + +type subsonicUsers struct { + User []subsonicUser `json:"user" xml:"user"` +} + +type subsonicExtension struct { + Name string `json:"name" xml:"name,attr"` + Versions []int `json:"versions" xml:"versions"` +} + +type subsonicExtensions struct { + Extension []subsonicExtension `json:"openSubsonicExtension" xml:"openSubsonicExtension"` +} + +type subsonicNowPlaying struct { + Entry []subsonicChild `json:"entry" xml:"entry"` +} + +type subsonicPlayQueue struct { + Current string `json:"current,omitempty" xml:"current,attr,omitempty"` + Position int64 `json:"position,omitempty" xml:"position,attr,omitempty"` + Username string `json:"username,omitempty" xml:"username,attr,omitempty"` + Changed string `json:"changed,omitempty" xml:"changed,attr,omitempty"` + Entry []subsonicChild `json:"entry" xml:"entry"` +} + +type subsonicArtistInfo struct { + Biography string `json:"biography,omitempty" xml:"biography,omitempty"` + MusicBrainzId string `json:"musicBrainzId,omitempty" xml:"musicBrainzId,omitempty"` +} + +type subsonicAlbumInfo struct { + Notes string `json:"notes,omitempty" xml:"notes,omitempty"` +} + +type subsonicBookmarks struct { + Bookmark []any `json:"bookmark" xml:"bookmark"` +} + +// ==================== 分发 ==================== + +// SubsonicAPI OpenSubsonic API 入口（/@rest/*） +func SubsonicAPI(c *gin.Context) { + if !MediaSubsonicEnabled() { + c.AbortWithStatus(404) + return + } + name := strings.Trim(c.Param("path"), "/") + // 兼容接入地址已带 /@rest、客户端又自动追加 /rest 的情况 + if first, rest, found := strings.Cut(name, "/"); found && strings.EqualFold(first, "rest") { + name = rest + } + name = strings.TrimSuffix(name, ".view") + name = strings.ToLower(name) + if name == "" { + subError(c, 0, "missing method") + return + } + + // 免认证端点 + if name == "getopensubsonicextensions" { + subRespond(c, &subsonicResponse{OpenSubsonicExtensions: &subsonicExtensions{ + Extension: []subsonicExtension{ + {Name: "apiKeyAuthentication", Versions: []int{1}}, + {Name: "formPost", Versions: []int{1}}, + {Name: "songLyrics", Versions: []int{1}}, + }, + }}) + return + } + + user, _, ok := subAuth(c) + if !ok { + // 锁定判定同样要看用户名维度，否则被锁的账号只会收到「密码错误」 + if mediaAuthLockedFor(c, subParam(c, "u")) { + subError(c, 40, "Too many failed attempts, please try again later") + return + } + subError(c, 40, "Wrong username or password; use an app key as the password or enable password auth") + return + } + + switch name { + case "ping": + subRespond(c, &subsonicResponse{}) + case "getlicense": + subRespond(c, &subsonicResponse{License: &subsonicLicense{Valid: true}}) + case "tokeninfo": + subRespond(c, &subsonicResponse{User: subUserOf(user)}) + case "getuser": + subRespond(c, &subsonicResponse{User: subUserOf(user)}) + case "getusers": + if !user.IsAdmin() { + subError(c, 50, "not authorized") + return + } + users, _, err := op.GetUsers(1, -1) + if err != nil { + subError(c, 0, err.Error()) + return + } + out := &subsonicUsers{} + for i := range users { + if users[i].IsGuest() { + continue + } + out.User = append(out.User, *subUserOf(&users[i])) + } + subRespond(c, &subsonicResponse{Users: out}) + case "getmusicfolders": + subRespond(c, &subsonicResponse{MusicFolders: &subsonicMusicFolders{ + MusicFolder: []subsonicMusicFolder{{Id: "1", Name: "Music"}}, + }}) + case "getindexes": + subGetIndexes(c, user) + case "getartists": + subGetArtists(c, user) + case "getartist": + subGetArtist(c, user) + case "getalbum": + subGetAlbum(c, user) + case "getsong": + subGetSong(c, user) + case "getmusicdirectory": + subGetMusicDirectory(c, user) + case "getalbumlist", "getalbumlist2": + subGetAlbumList(c, user, name == "getalbumlist") + case "getrandomsongs": + subGetRandomSongs(c, user) + case "getsongsbygenre": + subGetSongsByGenre(c, user) + case "search2", "search3": + subSearch(c, user, name == "search2") + case "getstarred", "getstarred2": + subGetStarred(c, user, name == "getstarred") + case "stream", "download": + subStream(c, user) + case "getcoverart": + subGetCoverArt(c, user) + case "getlyrics": + subGetLyrics(c, user) + case "getlyricsbysongid": + subGetLyricsBySongID(c, user) + case "star", "unstar": + subStar(c, user, name == "star") + case "setrating": + subSetRating(c, user) + case "scrobble": + subScrobble(c, user) + case "getnowplaying": + subRespond(c, &subsonicResponse{NowPlaying: &subsonicNowPlaying{Entry: subNowPlayingEntries()}}) + case "getplaylists": + subGetPlaylists(c, user) + case "getplaylist": + subGetPlaylist(c, user) + case "createplaylist": + subCreatePlaylist(c, user) + case "updateplaylist": + subUpdatePlaylist(c, user) + case "deleteplaylist": + subDeletePlaylist(c, user) + case "getscanstatus": + subRespond(c, &subsonicResponse{ScanStatus: &subsonicScanStatus{Scanning: false, Count: subSongCount(user)}}) + case "startscan": + if !user.IsAdmin() { + subError(c, 50, "not authorized") + return + } + subStartScan(c) + case "saveplayqueue": + subSavePlayQueue(c, user) + case "getplayqueue": + subRespond(c, &subsonicResponse{PlayQueue: subLoadPlayQueue(user)}) + case "getartistinfo2": + subRespond(c, &subsonicResponse{ArtistInfo2: &subsonicArtistInfo{}}) + case "getalbuminfo", "getalbuminfo2": + subRespond(c, &subsonicResponse{AlbumInfo: &subsonicAlbumInfo{}}) + case "gettopsongs", "getsimilarsongs2": + subRespond(c, &subsonicResponse{TopSongs: &subsonicSongs{}}) + case "getbookmarks": + subRespond(c, &subsonicResponse{Bookmarks: &subsonicBookmarks{}}) + case "getgenres": + subGetGenres(c, user) + case "getavatars": + c.Status(404) + // 明确不支持 + case "getpodcasts", "getnewestpodcasts", "refreshpodcasts", "createpodcastchannel", + "deletepodcastchannel", "deletepodcastepisode", "downloadpodcastepisode", + "createuser", "updateuser", "deleteuser", "changepassword", + "getshares", "createshare", "updateshare", "deleteshare", "jukeboxcontrol": + c.String(501, "This endpoint is not implemented, but may be in future releases") + case "search", "getchatmessages", "addchatmessage", "getvideos", "getvideoinfo", "getcaptions", "hls": + c.String(410, "This endpoint will not be implemented") + default: + subError(c, 0, "unknown method: "+name) + } +} + +// ==================== 参数 / 认证 / 响应 ==================== + +func subParam(c *gin.Context, key string) string { + if v := c.Query(key); v != "" { + return v + } + if c.Request.Method == "POST" { + _ = c.Request.ParseForm() + if v := c.Request.PostForm.Get(key); v != "" { + return v + } + } + return "" +} + +func subIntParam(c *gin.Context, key string, def int) int { + v := subParam(c, key) + if v == "" { + return def + } + n, err := strconv.Atoi(v) + if err != nil { + return def + } + return n +} + +// subAuth 支持 apiKey / u+t+s / u+p（默认关闭）认证，并对失败次数做 IP + 用户名双维度限流 +func subAuth(c *gin.Context) (*model.User, *model.MediaAppKey, bool) { + username := strings.TrimSpace(subParam(c, "u")) + if mediaAuthLockedFor(c, username) { + return nil, nil, false + } + u, k, ok := subAuthInner(c) + if ok { + // 用真实用户名清除计数：apiKey 认证时 u 参数是空的 + mediaAuthSucceededFor(c, u.Username) + } else if subHasCredential(c) { + // 只有「确实带了凭据但校验失败」才计数； + // 完全裸请求（部分客户端启动时会先发一次不带参数的探测）不计入失败次数。 + mediaAuthFailedFor(c, username) + } + return u, k, ok +} + +// subHasCredential 判断请求是否真的携带了认证凭据 +func subHasCredential(c *gin.Context) bool { + return subParam(c, "apiKey") != "" || subParam(c, "u") != "" || + subParam(c, "t") != "" || subParam(c, "p") != "" +} + +func subAuthInner(c *gin.Context) (*model.User, *model.MediaAppKey, bool) { + // OpenSubsonic apiKey 扩展：直接把应用密钥放在 apiKey 参数里 + if key := strings.TrimSpace(subParam(c, "apiKey")); key != "" { + if u, k, ok := subAuthByAppKey(key); ok { + return u, k, true + } + return nil, nil, false + } + username := strings.TrimSpace(subParam(c, "u")) + password := decodeSubsonicPassword(subParam(c, "p")) + // 用户名留空、只填了应用密钥（部分客户端只允许填一个"密码"字段） + if username == "" { + if password != "" { + if u, k, ok := subAuthByAppKey(password); ok { + return u, k, true + } + } + return nil, nil, false + } + user, err := op.GetUserByName(username) + if err != nil || user.Disabled || user.IsGuest() { + return nil, nil, false + } + // t+s：用应用密钥的明文做 md5(密码+salt) 校验（Subsonic token 认证） + if salt := subParam(c, "s"); salt != "" { + if token := subParam(c, "t"); token != "" { + keys, _ := db.ListMediaAppKeys(user.ID) + for i := range keys { + plain, err := media.DecryptAppKey(keys[i].Secret) + if err != nil { + continue + } + if media.MatchSubsonicToken(plain, salt, token) { + db.TouchMediaAppKey(keys[i].ID) + return user, &keys[i], true + } + } + // 不直接返回：个别客户端会同时带上 p，继续按明文校验 + } + } + // p：OpenList 密码或应用密钥明文 + if password != "" { + // 使用主账号密码认证会把登录密码暴露给第三方客户端，必须由管理员显式开启 + if setting.GetBool(conf.MediaSubsonicAllowPwd) && user.OtpSecret == "" && + user.ValidateRawPassword(password) == nil { + return user, nil, true + } + if u, k, ok := subAuthByAppKey(password); ok && u.ID == user.ID { + return u, k, true + } + } + return nil, nil, false +} + +// decodeSubsonicPassword 解析 p 参数，兼容 Subsonic 的 enc: 编码 +func decodeSubsonicPassword(p string) string { + p = strings.TrimSpace(p) + if strings.HasPrefix(p, "enc:") { + if raw, err := hex.DecodeString(strings.TrimPrefix(p, "enc:")); err == nil { + return string(raw) + } + } + return p +} + +func subAuthByAppKey(key string) (*model.User, *model.MediaAppKey, bool) { + return LookUpAppKeyUser(key) +} + +// subsonicServerVersion 返回给 Subsonic 客户端的服务器版本（仅用于展示） +func subsonicServerVersion() string { + if conf.Version == "" { + return "OpenList" + } + return conf.Version +} + +func subRespond(c *gin.Context, resp *subsonicResponse) { + resp.Status = "ok" + resp.Version = subsonicAPIVersion + resp.Type = "OpenList" + resp.ServerVersion = subsonicServerVersion() + resp.OpenSubsonic = true + format := strings.ToLower(subParam(c, "f")) + switch format { + case "json": + c.JSON(200, gin.H{"subsonic-response": resp}) + case "jsonp": + callback := subParam(c, "callback") + if !validJsonpCallback(callback) { + c.JSON(200, gin.H{"subsonic-response": &subsonicResponse{ + Status: "failed", Version: subsonicAPIVersion, Type: "OpenList", + ServerVersion: subsonicServerVersion(), OpenSubsonic: true, + Error: &subsonicError{Code: 0, Message: "invalid callback parameter"}, + }}) + return + } + body, _ := json.Marshal(gin.H{"subsonic-response": resp}) + c.Header("Content-Type", "application/javascript; charset=utf-8") + c.String(200, "%s(%s)", callback, string(body)) + default: + c.XML(200, resp) + } +} + +func subError(c *gin.Context, code int, message string) { + resp := &subsonicResponse{ + Status: "failed", + Version: subsonicAPIVersion, + Type: "OpenList", + ServerVersion: subsonicServerVersion(), + OpenSubsonic: true, + Error: &subsonicError{Code: code, Message: message}, + } + format := strings.ToLower(subParam(c, "f")) + switch format { + case "json": + c.JSON(200, gin.H{"subsonic-response": resp}) + case "jsonp": + body, _ := json.Marshal(gin.H{"subsonic-response": resp}) + c.String(200, "%s(%s)", subParam(c, "callback"), string(body)) + default: + c.XML(200, resp) + } +} + +func validJsonpCallback(s string) bool { + if s == "" { + return false + } + for i, r := range s { + if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r == '_' || r == '$' { + continue + } + if i > 0 && (r >= '0' && r <= '9' || r == '.') { + continue + } + return false + } + return true +} + +// ==================== 数据映射 ==================== + +func subUserOf(u *model.User) *subsonicUser { + return &subsonicUser{ + Username: u.Username, + ScrobblingEnabled: true, + AdminRole: u.IsAdmin(), + DownloadRole: true, + PlaylistRole: true, + StreamRole: true, + CoverArtRole: true, + } +} + +func (l *embyLibrary) subChild(it *model.MediaItem) subsonicChild { + title := it.ScrapedName + if title == "" { + title = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName)) + } + album := it.AlbumName + if album == "" { + album = "Unknown Album" + } + artist := albumArtistOf(it) + image := fmt.Sprintf("al-%s", albumKeyOf(it)) + year := 0 + if len(it.ReleaseDate) >= 4 { + year, _ = strconv.Atoi(it.ReleaseDate[:4]) + } + userData := l.userDataOf(it.ID, 0) + c := subsonicChild{ + Id: fmt.Sprintf("so-%d", it.ID), + Parent: image, + IsDir: false, + Title: title, + Album: album, + Artist: artist, + Track: it.TrackNumber, + Year: year, + Genre: it.Genre, + CoverArt: image, + Size: it.FileSize, + ContentType: it.MimeType, + Suffix: strings.TrimPrefix(strings.ToLower(stdpath.Ext(it.FileName)), "."), + Duration: it.Duration, + Path: MediaItemFilePath(it), + AlbumId: image, + ArtistId: "ar-" + artistKeyOf(it), + Type: "music", + PlayCount: userData.PlayCount, + Created: it.CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z"), + UserRating: userDataRating(it.ID, l), + } + if userData.IsFavorite { + c.Starred = it.UpdatedAt.UTC().Format("2006-01-02T15:04:05.000Z") + } + return c +} + +func userDataRating(itemID uint, l *embyLibrary) int { + if l == nil { + return 0 + } + if d, ok := l.userData[[2]uint{itemID, 0}]; ok { + return d.Rating + } + return 0 +} + +func (l *embyLibrary) subAlbum(al *embyAlbum) subsonicAlbum { + duration := 0 + for i := range al.Items { + duration += al.Items[i].Duration + } + created := time.Now().UTC().Format("2006-01-02T15:04:05.000Z") + if len(al.Items) > 0 { + created = al.Items[0].CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z") + } + return subsonicAlbum{ + Id: "al-" + al.Key, + Name: al.Name, + Artist: al.Artist, + ArtistId: "ar-" + artistKeyOf(&al.Items[0]), + CoverArt: "al-" + al.Key, + SongCount: len(al.Items), + Duration: duration, + Created: created, + Year: al.Year, + } +} + +func (l *embyLibrary) subAlbumWithSongs(al *embyAlbum) subsonicAlbum { + out := l.subAlbum(al) + for i := range al.Items { + out.Song = append(out.Song, l.subChild(&al.Items[i])) + } + return out +} + +func (l *embyLibrary) subArtist(key, name string) subsonicArtist { + out := subsonicArtist{Id: "ar-" + key, Name: name} + for _, al := range l.albums() { + if artistKeyOf(&al.Items[0]) != key { + continue + } + out.Album = append(out.Album, l.subAlbum(al)) + } + out.AlbumCount = len(out.Album) + if len(out.Album) > 0 { + out.CoverArt = out.Album[0].CoverArt + } + return out +} + +func (l *embyLibrary) subArtists() []subsonicArtist { + seen := map[string]bool{} + var out []subsonicArtist + for i := range l.music { + key := artistKeyOf(&l.music[i]) + if seen[key] { + continue + } + seen[key] = true + out = append(out, l.subArtist(key, albumArtistOf(&l.music[i]))) + } + sort.Slice(out, func(i, j int) bool { return strings.ToLower(out[i].Name) < strings.ToLower(out[j].Name) }) + return out +} diff --git a/server/handles/subsonic_handlers.go b/server/handles/subsonic_handlers.go new file mode 100644 index 00000000..81ba06e8 --- /dev/null +++ b/server/handles/subsonic_handlers.go @@ -0,0 +1,667 @@ +package handles + +import ( + "fmt" + "math/rand" + "net/http" + stdpath "path" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/db" + "github.com/OpenListTeam/OpenList/v4/internal/media" + "github.com/OpenListTeam/OpenList/v4/internal/model" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/gin-gonic/gin" +) + +// ==================== 浏览 ==================== + +func subInitial(name string) string { + if name == "" { + return "#" + } + r := strings.ToUpper(string([]rune(name)[0])) + if r < "A" || r > "Z" { + return "#" + } + return r +} + +func subGetIndexes(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + artists := lib.subArtists() + groups := map[string][]subsonicChild{} + var keys []string + for _, a := range artists { + idx := subInitial(a.Name) + if _, ok := groups[idx]; !ok { + keys = append(keys, idx) + } + groups[idx] = append(groups[idx], subsonicChild{ + Id: a.Id, + Title: a.Name, + IsDir: true, + Album: a.Name, + }) + } + sort.Strings(keys) + out := &subsonicIndexes{IgnoredArticles: "The El La Los Las Le Les", LastModified: time.Now().UnixMilli()} + for _, k := range keys { + out.Index = append(out.Index, subsonicIndex{Name: k, Artist: groups[k]}) + } + subRespond(c, &subsonicResponse{Indexes: out}) +} + +func subGetArtists(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + artists := lib.subArtists() + groups := map[string][]subsonicArtist{} + var keys []string + for _, a := range artists { + idx := subInitial(a.Name) + if _, ok := groups[idx]; !ok { + keys = append(keys, idx) + } + groups[idx] = append(groups[idx], a) + } + sort.Strings(keys) + out := &subsonicArtists{IgnoredArticles: "The El La Los Las Le Les"} + for _, k := range keys { + out.Index = append(out.Index, subsonicIndex2{Name: k, Artist: groups[k]}) + } + subRespond(c, &subsonicResponse{Artists: out}) +} + +func subGetArtist(c *gin.Context, user *model.User) { + id := subParam(c, "id") + if !strings.HasPrefix(id, "ar-") { + subError(c, 70, "artist not found") + return + } + lib := loadEmbyLibrary(user) + key := strings.TrimPrefix(id, "ar-") + for _, a := range lib.subArtists() { + if a.Id == id { + _ = key + subRespond(c, &subsonicResponse{Artist: &a}) + return + } + } + subError(c, 70, "artist not found") +} + +func subGetAlbum(c *gin.Context, user *model.User) { + id := subParam(c, "id") + if !strings.HasPrefix(id, "al-") { + subError(c, 70, "album not found") + return + } + lib := loadEmbyLibrary(user) + al := lib.albumByKey(strings.TrimPrefix(id, "al-")) + if al == nil { + subError(c, 70, "album not found") + return + } + out := lib.subAlbumWithSongs(al) + subRespond(c, &subsonicResponse{Album: &out}) +} + +func subGetSong(c *gin.Context, user *model.User) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + it, ok := lib.songByID(id) + if !ok { + subError(c, 70, "song not found") + return + } + child := lib.subChild(it) + subRespond(c, &subsonicResponse{Song: &child}) +} + +func subGetMusicDirectory(c *gin.Context, user *model.User) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + switch { + case strings.HasPrefix(id, "ar-"): + key := strings.TrimPrefix(id, "ar-") + dir := &subsonicDirectory{Id: id, Name: id} + for _, al := range lib.albums() { + if artistKeyOf(&al.Items[0]) != key { + continue + } + a := lib.subAlbum(al) + dir.Child = append(dir.Child, subsonicChild{ + Id: a.Id, IsDir: true, Title: a.Name, Album: a.Name, + Artist: a.Artist, CoverArt: a.CoverArt, Year: a.Year, + }) + if dir.Name == id { + dir.Name = a.Artist + } + } + subRespond(c, &subsonicResponse{MusicDirectory: dir}) + case strings.HasPrefix(id, "al-"): + al := lib.albumByKey(strings.TrimPrefix(id, "al-")) + if al == nil { + subError(c, 70, "directory not found") + return + } + dir := &subsonicDirectory{Id: id, Name: al.Name, Parent: "ar-" + artistKeyOf(&al.Items[0])} + for i := range al.Items { + dir.Child = append(dir.Child, lib.subChild(&al.Items[i])) + } + subRespond(c, &subsonicResponse{MusicDirectory: dir}) + default: + subError(c, 70, "directory not found") + } +} + +func subGetAlbumList(c *gin.Context, user *model.User, legacy bool) { + lib := loadEmbyLibrary(user) + listType := strings.ToLower(subParam(c, "type")) + albums := lib.albums() + switch listType { + case "newest": + sort.SliceStable(albums, func(i, j int) bool { return albums[i].Items[0].CreatedAt.After(albums[j].Items[0].CreatedAt) }) + case "recent": + sort.SliceStable(albums, func(i, j int) bool { return albums[i].Items[0].UpdatedAt.After(albums[j].Items[0].UpdatedAt) }) + case "alphabeticalbyartist": + sort.SliceStable(albums, func(i, j int) bool { + if albums[i].Artist == albums[j].Artist { + return strings.ToLower(albums[i].Name) < strings.ToLower(albums[j].Name) + } + return strings.ToLower(albums[i].Artist) < strings.ToLower(albums[j].Artist) + }) + case "random": + rand.Shuffle(len(albums), func(i, j int) { albums[i], albums[j] = albums[j], albums[i] }) + case "byyear": + from := subIntParam(c, "fromYear", 0) + to := subIntParam(c, "toYear", 0) + var filtered []*embyAlbum + for _, al := range albums { + if from > 0 && al.Year < from { + continue + } + if to > 0 && al.Year > to { + continue + } + filtered = append(filtered, al) + } + albums = filtered + default: + sort.SliceStable(albums, func(i, j int) bool { return strings.ToLower(albums[i].Name) < strings.ToLower(albums[j].Name) }) + } + size := subIntParam(c, "size", 50) + offset := subIntParam(c, "offset", 0) + if offset > len(albums) { + offset = len(albums) + } + end := offset + size + if end > len(albums) { + end = len(albums) + } + out := &subsonicAlbumList{} + for _, al := range albums[offset:end] { + out.Album = append(out.Album, lib.subAlbum(al)) + } + if legacy { + subRespond(c, &subsonicResponse{AlbumList: out}) + return + } + subRespond(c, &subsonicResponse{AlbumList2: out}) +} + +func (l *embyLibrary) allSongs() []subsonicChild { + out := make([]subsonicChild, 0, len(l.music)) + for i := range l.music { + out = append(out, l.subChild(&l.music[i])) + } + return out +} + +func subGetRandomSongs(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + songs := lib.allSongs() + rand.Shuffle(len(songs), func(i, j int) { songs[i], songs[j] = songs[j], songs[i] }) + size := subIntParam(c, "size", 10) + if size > len(songs) { + size = len(songs) + } + subRespond(c, &subsonicResponse{RandomSongs: &subsonicSongs{Song: songs[:size]}}) +} + +func subGetSongsByGenre(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + genre := strings.ToLower(subParam(c, "genre")) + var songs []subsonicChild + for i := range lib.music { + if genre != "" && !strings.Contains(strings.ToLower(lib.music[i].Genre), genre) { + continue + } + songs = append(songs, lib.subChild(&lib.music[i])) + } + subRespond(c, &subsonicResponse{SongsByGenre: &subsonicSongs{Song: songs}}) +} + +func subSearch(c *gin.Context, user *model.User, legacy bool) { + lib := loadEmbyLibrary(user) + term := strings.ToLower(strings.TrimSpace(subParam(c, "query"))) + if term == "" { + term = strings.ToLower(strings.TrimSpace(subParam(c, "any"))) + } + artistCount := subIntParam(c, "artistCount", 20) + albumCount := subIntParam(c, "albumCount", 20) + songCount := subIntParam(c, "songCount", 20) + + out := &subsonicSearchResult{} + for _, a := range lib.subArtists() { + if len(out.Artist) >= artistCount { + break + } + if term == "" || strings.Contains(strings.ToLower(a.Name), term) { + out.Artist = append(out.Artist, a) + } + } + for _, al := range lib.albums() { + if len(out.Album) >= albumCount { + break + } + if term == "" || strings.Contains(strings.ToLower(al.Name), term) || + strings.Contains(strings.ToLower(al.Artist), term) { + out.Album = append(out.Album, lib.subAlbum(al)) + } + } + for i := range lib.music { + if len(out.Song) >= songCount { + break + } + name := lib.music[i].ScrapedName + if name == "" { + name = lib.music[i].FileName + } + if term == "" || strings.Contains(strings.ToLower(name), term) { + out.Song = append(out.Song, lib.subChild(&lib.music[i])) + } + } + out.Total = len(out.Artist) + len(out.Album) + len(out.Song) + if legacy { + subRespond(c, &subsonicResponse{SearchResult2: out}) + return + } + subRespond(c, &subsonicResponse{SearchResult3: out}) +} + +func subGetStarred(c *gin.Context, user *model.User, legacy bool) { + lib := loadEmbyLibrary(user) + out := &subsonicStarred{} + for i := range lib.music { + if d, ok := lib.userData[[2]uint{lib.music[i].ID, 0}]; ok && d.IsFavorite { + out.Song = append(out.Song, lib.subChild(&lib.music[i])) + } + } + if legacy { + subRespond(c, &subsonicResponse{Starred: out}) + return + } + subRespond(c, &subsonicResponse{Starred2: out}) +} + +func subGetGenres(c *gin.Context, user *model.User) { + lib := loadEmbyLibrary(user) + counts := map[string]int{} + for i := range lib.music { + for _, g := range strings.Split(lib.music[i].Genre, ",") { + g = strings.TrimSpace(g) + if g == "" { + continue + } + counts[g]++ + } + } + out := &subsonicGenres{} + for g, n := range counts { + out.Genre = append(out.Genre, subsonicGenre{Value: g, SongCount: n}) + } + sort.Slice(out.Genre, func(i, j int) bool { return out.Genre[i].Value < out.Genre[j].Value }) + subRespond(c, &subsonicResponse{Genres: out}) +} + +// ==================== 播放 ==================== + +func subStream(c *gin.Context, user *model.User) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + subError(c, 70, "song not found") + return + } + if setting.GetBool(conf.MediaSubsonicProxyStream) { + c.Redirect(http.StatusFound, SignedProxyURL(c, target.RawPath)) + return + } + c.Redirect(http.StatusFound, SignedStreamURL(c, target.RawPath)) +} + +func subGetCoverArt(c *gin.Context, user *model.User) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + var item *model.MediaItem + switch { + case strings.HasPrefix(id, "so-"): + item, _ = lib.songByID(id) + case strings.HasPrefix(id, "al-"): + if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 { + item = &al.Items[0] + } + case strings.HasPrefix(id, "ar-"): + key := strings.TrimPrefix(id, "ar-") + for i := range lib.music { + if artistKeyOf(&lib.music[i]) == key { + item = &lib.music[i] + break + } + } + case strings.HasPrefix(id, "mv-"), strings.HasPrefix(id, "ep-"): + if it, ok := lib.resolveItem(id); ok { + _ = it + if p, ok := lib.playTarget(id); ok { + item = p.Item + } + } + } + if item == nil { + c.Status(404) + return + } + serveEmbyCover(c, item) +} + +func subGetLyrics(c *gin.Context, user *model.User) { + artist := subParam(c, "artist") + title := subParam(c, "title") + lib := loadEmbyLibrary(user) + for i := range lib.music { + it := &lib.music[i] + name := it.ScrapedName + if name == "" { + name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName)) + } + if title != "" && !strings.EqualFold(title, name) { + continue + } + if artist != "" && !strings.EqualFold(artist, albumArtistOf(it)) { + continue + } + subRespond(c, &subsonicResponse{Lyrics: &subsonicLyrics{ + Artist: albumArtistOf(it), Title: name, Value: it.Lyrics, + }}) + return + } + subError(c, 70, "lyrics not found") +} + +func subGetLyricsBySongID(c *gin.Context, user *model.User) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + it, ok := lib.songByID(id) + if !ok || it.Lyrics == "" { + subError(c, 70, "lyrics not found") + return + } + name := it.ScrapedName + if name == "" { + name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName)) + } + lines := parseLrc(it.Lyrics) + structured := subsonicStructuredLyrics{ + DisplayArtist: albumArtistOf(it), + DisplayTitle: name, + Lang: "und", + Synced: len(lines) > 0, + } + for _, l := range lines { + structured.Line = append(structured.Line, subsonicLyricLine{ + Start: l["Start"].(int64), + Value: l["Text"].(string), + }) + } + subRespond(c, &subsonicResponse{LyricsList: &subsonicLyricsList{ + StructuredLyrics: []subsonicStructuredLyrics{structured}, + }}) +} + +// ==================== 状态 ==================== + +func subStar(c *gin.Context, user *model.User, star bool) { + id := subParam(c, "id") + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + if album := strings.HasPrefix(id, "al-"); album { + subError(c, 70, "not supported") + return + } + subError(c, 70, "song not found") + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + d.IsFavorite = star + if err := db.SaveMediaUserData(d); err != nil { + subError(c, 0, err.Error()) + return + } + subRespond(c, &subsonicResponse{}) +} + +func subSetRating(c *gin.Context, user *model.User) { + id := subParam(c, "id") + rating := subIntParam(c, "rating", 0) + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + subError(c, 70, "song not found") + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + d.Rating = rating + _ = db.SaveMediaUserData(d) + subRespond(c, &subsonicResponse{}) +} + +func subScrobble(c *gin.Context, user *model.User) { + id := subParam(c, "id") + submission := strings.EqualFold(subParam(c, "submission"), "true") + lib := loadEmbyLibrary(user) + target, ok := lib.playTarget(id) + if !ok { + subRespond(c, &subsonicResponse{}) + return + } + d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo) + if err != nil || d == nil { + d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo} + } + now := time.Now() + d.LastPlayedAt = &now + if submission { + d.Played = true + d.PlayCount++ + } + _ = db.SaveMediaUserData(d) + subRespond(c, &subsonicResponse{}) +} + +// ==================== 播放列表 ==================== + +func subGetPlaylists(c *gin.Context, user *model.User) { + list, err := db.ListMediaPlaylists(user.ID) + if err != nil { + subError(c, 0, err.Error()) + return + } + out := &subsonicPlaylists{} + for i := range list { + out.Playlist = append(out.Playlist, subPlaylistOf(&list[i])) + } + subRespond(c, &subsonicResponse{Playlists: out}) +} + +func subPlaylistOf(p *model.MediaPlaylist) subsonicPlaylist { + return subsonicPlaylist{ + Id: strconv.FormatUint(uint64(p.ID), 10), + Name: p.Name, + Comment: p.Comment, + Public: p.Public, + SongCount: p.ItemCount, + Created: p.CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z"), + } +} + +func subGetPlaylist(c *gin.Context, user *model.User) { + id, err := strconv.ParseUint(subParam(c, "id"), 10, 64) + if err != nil { + subError(c, 70, "playlist not found") + return + } + p, err := db.GetMediaPlaylist(uint(id)) + if err != nil || (p.UserID != user.ID && !user.IsAdmin()) { + subError(c, 70, "playlist not found") + return + } + out := subPlaylistOf(p) + items, _ := db.ListMediaPlaylistItems(p.ID) + lib := loadEmbyLibrary(user) + for _, it := range items { + if song, ok := lib.songByID(fmt.Sprintf("so-%d", it.ItemID)); ok { + out.Entry = append(out.Entry, lib.subChild(song)) + } + } + out.SongCount = len(out.Entry) + subRespond(c, &subsonicResponse{Playlist: &out}) +} + +func subCreatePlaylist(c *gin.Context, user *model.User) { + name := subParam(c, "name") + if name == "" { + subError(c, 10, "missing name") + return + } + p := &model.MediaPlaylist{UserID: user.ID, Name: name} + if err := db.CreateMediaPlaylist(p); err != nil { + subError(c, 0, err.Error()) + return + } + subError(c, 0, "created:"+strconv.FormatUint(uint64(p.ID), 10)) +} + +func subUpdatePlaylist(c *gin.Context, user *model.User) { + id, err := strconv.ParseUint(subParam(c, "playlistId"), 10, 64) + if err != nil { + subError(c, 10, "missing playlistId") + return + } + p, err := db.GetMediaPlaylist(uint(id)) + if err != nil || (p.UserID != user.ID && !user.IsAdmin()) { + subError(c, 70, "playlist not found") + return + } + if name := subParam(c, "name"); name != "" { + p.Name = name + } + if comment := subParam(c, "comment"); comment != "" { + p.Comment = comment + } + if pub := subParam(c, "public"); pub != "" { + p.Public = strings.EqualFold(pub, "true") + } + _ = db.UpdateMediaPlaylist(p) + subRespond(c, &subsonicResponse{}) +} + +func subDeletePlaylist(c *gin.Context, user *model.User) { + id, err := strconv.ParseUint(subParam(c, "id"), 10, 64) + if err != nil { + subError(c, 10, "missing id") + return + } + p, err := db.GetMediaPlaylist(uint(id)) + if err != nil || (p.UserID != user.ID && !user.IsAdmin()) { + subError(c, 70, "playlist not found") + return + } + _ = db.DeleteMediaPlaylist(p.ID) + subRespond(c, &subsonicResponse{}) +} + +// ==================== 扫描 ==================== + +func subSongCount(user *model.User) int64 { + lib := loadEmbyLibrary(user) + return int64(len(lib.music)) +} + +func subStartScan(c *gin.Context) { + for _, mt := range MediaTypeEnabledList() { + cfg, err := db.GetMediaConfig(mt) + if err != nil || cfg == nil { + continue + } + media.ScanMedia(cfg) + } + subRespond(c, &subsonicResponse{ScanStatus: &subsonicScanStatus{Scanning: true}}) +} + +// ==================== 播放队列（内存） ==================== + +var ( + subQueueMu sync.Mutex + subQueues = map[uint]*subsonicPlayQueue{} +) + +func subSavePlayQueue(c *gin.Context, user *model.User) { + ids := c.Query("id") + if ids == "" { + _ = c.Request.ParseForm() + ids = c.Request.PostForm.Get("id") + } + q := &subsonicPlayQueue{ + Current: subParam(c, "current"), + Position: int64(subIntParam(c, "position", 0)), + Username: user.Username, + Changed: time.Now().UTC().Format("2006-01-02T15:04:05.000Z"), + } + if ids != "" { + lib := loadEmbyLibrary(user) + for _, id := range strings.Split(ids, ",") { + if song, ok := lib.songByID(strings.TrimSpace(id)); ok { + q.Entry = append(q.Entry, lib.subChild(song)) + } + } + } + subQueueMu.Lock() + subQueues[user.ID] = q + subQueueMu.Unlock() + subRespond(c, &subsonicResponse{}) +} + +func subLoadPlayQueue(user *model.User) *subsonicPlayQueue { + subQueueMu.Lock() + defer subQueueMu.Unlock() + if q, ok := subQueues[user.ID]; ok { + return q + } + return &subsonicPlayQueue{Username: user.Username} +} + +func subNowPlayingEntries() []subsonicChild { return []subsonicChild{} } diff --git a/server/handles/transcode.go b/server/handles/transcode.go new file mode 100644 index 00000000..dfe8f258 --- /dev/null +++ b/server/handles/transcode.go @@ -0,0 +1,656 @@ +package handles + +import ( + "crypto/subtle" + "errors" + "fmt" + "net/http" + "net/url" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "time" + + "github.com/OpenListTeam/OpenList/v4/internal/conf" + "github.com/OpenListTeam/OpenList/v4/internal/driver" + "github.com/OpenListTeam/OpenList/v4/internal/fs" + "github.com/OpenListTeam/OpenList/v4/internal/op" + "github.com/OpenListTeam/OpenList/v4/internal/setting" + "github.com/OpenListTeam/OpenList/v4/internal/sign" + "github.com/OpenListTeam/OpenList/v4/internal/transcode" + "github.com/OpenListTeam/OpenList/v4/server/common" + "github.com/gin-gonic/gin" +) + +// 探测阶段的独立并发上限。 +// +// 转码配额（每用户 3 个活跃任务）只约束「入队」，而配额判定发生在探测之后： +// 并发请求可能同时通过配额预检、各起一个最长 30s 的 ffprobe。这里给探测单独 +// 设一道闸，保证这段窗口内的进程数有界。每用户上限取任务配额值，正常并发播放 +// 不会被误伤（用户有 3 个任务时预检已经先拒绝了）。 +const ( + maxConcurrentProbes = 8 + maxProbesPerUser = transcode.MaxActiveJobsPerUser +) + +type probeLimiter struct { + mu sync.Mutex + active int + users map[string]int +} + +var probeSlots = &probeLimiter{users: make(map[string]int)} + +func (p *probeLimiter) acquire(owner string) bool { + p.mu.Lock() + defer p.mu.Unlock() + if p.active >= maxConcurrentProbes || p.users[owner] >= maxProbesPerUser { + return false + } + p.active++ + p.users[owner]++ + return true +} + +func (p *probeLimiter) release(owner string) { + p.mu.Lock() + defer p.mu.Unlock() + if p.active > 0 { + p.active-- + } + if n := p.users[owner]; n <= 1 { + delete(p.users, owner) + } else { + p.users[owner] = n - 1 + } +} + +// ============================================================ +// 播放端：/api/transcode/play 发起或复用一个转码任务 +// ============================================================ + +// TranscodePlay 入口：POST /api/fs/transcode body: {path} +// +// 行为： +// 1. 检查 transcode_enabled +// 2. fs.Get 获取文件并跑 Decide +// 3. 不需要转码 → 返回 {transcode:false, direct: true} +// 4. 需要转码 → 创建 Job 入队，返回 {transcode:true, job_id, master_url} +func TranscodePlay(c *gin.Context) { + if !transcode.IsEnabled() { + common.SuccessResp(c, gin.H{"transcode": false, "reason": "disabled"}) + return + } + var req struct { + Path string `json:"path" binding:"required"` + } + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + // 该路由挂在 Auth(true)（允许被禁用的访客通过）之下，必须在这里自行拒绝， + // 否则关闭访客后匿名请求仍能触发 ffprobe/ffmpeg 并占用转码缓存。 + user := mediaUser(c) + if user == nil || (user.IsGuest() && user.Disabled) { + common.ErrorStrResp(c, "Guest user is disabled, login please", 401) + return + } + if !CanUserAccessPath(user, req.Path) { + common.ErrorStrResp(c, "path is not accessible", 403) + return + } + obj, err := fs.Get(c.Request.Context(), req.Path, &fs.GetArgs{NoLog: true}) + if err != nil { + common.ErrorResp(c, err, 500) + return + } + if obj.IsDir() { + common.ErrorStrResp(c, "path is a directory", 400) + return + } + dec := transcode.Decide(transcode.DecideRequest{ + FileName: obj.GetName(), + FileSize: obj.GetSize(), + }) + if !dec.NeedTranscode { + common.SuccessResp(c, gin.H{ + "transcode": false, + "reason": dec.Reason, + }) + return + } + + mgr := transcode.Default() + mgr.Start() + + // apiURL 仅用于返回给浏览器的 master_url（需要跟随请求 Host 才能被客户端访问）。 + apiURL := common.GetApiUrlFromRequest(c.Request) + + // 配额键：普通用户按 ID，访客按 IP——访客共用一个账号， + // 否则一个匿名访客就能占满全部匿名转码配额。 + ownerID := "user:" + strconv.FormatUint(uint64(user.ID), 10) + if user.IsGuest() { + ownerID = "guest:" + c.ClientIP() + } + + // 同一用户重复请求同一文件：直接复用进行中的任务，避免重复 ffprobe 与重复转码。 + if exist, ok := mgr.Scheduler.FindReusable(ownerID, req.Path); ok { + respondTranscodeJob(c, apiURL, exist) + return + } + + // 【内存优化】优先尝试解析为本地文件路径，让 ffmpeg 直接读取本地文件， + // 完全绕过 HTTP /d 代理路径——避免 net.Downloader 给每个 range 请求分配 + // 高达 MaxBufferLimit (默认系统内存 5%) 的大缓冲。这是降低 Go 进程内存 + // 占用最有效的方法。如果不是本地驱动则退回到 HTTP 签名 URL。 + var sourceURL string + if localFile, ok := resolveLocalFilePath(req.Path); ok { + // ffmpeg 接受本地路径作为输入，无需 file:// 前缀 + sourceURL = localFile + fmt.Printf("[transcode] using local file path for ffmpeg: %s\n", localFile) + } else { + // 远程驱动：构造签名 HTTP URL。 + // 这个 URL 会被服务端 ffmpeg 直接访问，绝不能使用 Host / X-Forwarded-Host： + // 未配置 site_url 时攻击者可借此让服务器去拉取任意地址（SSRF）。 + base := trustedInternalBaseURL() + if base == "" { + // 仅 HTTPS 监听且未配置 site_url 时无可用基址； + // 若继续拼出 "/d/..." 会被当成绝对本地路径，ffmpeg 只会报文件不存在。 + common.ErrorStrResp( + c, + "transcode needs site_url (or an HTTP port) to fetch the source", + 503, + ) + return + } + signedPath := sign.Sign(req.Path) + sourceURL = fmt.Sprintf("%s/d%s?sign=%s", base, encodePath(req.Path), signedPath) + } + if !transcode.ValidSourceInput(sourceURL) { + common.ErrorStrResp(c, "unsupported transcode source", 400) + return + } + + // 创建 Job + job := transcode.NewJob() + job.Path = req.Path + job.OwnerID = ownerID + job.SourceURL = sourceURL + // 配额预检：ProbeSource 会起一个最长 30s 的 ffprobe 子进程， + // 不能等探测完才告诉用户「已有 3 个活跃任务」。最终判定仍由 SubmitLimited 原子完成。 + if err := mgr.Scheduler.Capacity(ownerID); err != nil { + common.ErrorResp(c, err, 429) + return + } + if !probeSlots.acquire(ownerID) { + common.ErrorStrResp(c, "too many concurrent transcode requests", 429) + return + } + // 【关键】探测视频总时长（异步会让首响应变慢，但只有一次，必须等待） + // 总时长用于生成完整的 playlist，避免 HLS.js 时长显示错误和拖动跳跃问题 + probe, perr := func() (transcode.SourceProbe, error) { + // 用 defer 释放：探测内部 panic 时也不能把槽位永久占住 + defer probeSlots.release(ownerID) + return transcode.ProbeSource(sourceURL) + }() + if perr != nil { + // 探测失败不阻塞流程，但记录日志；后续 playlist 会缺少总时长信息 + fmt.Printf("[transcode] probe source failed: %v\n", perr) + probe = transcode.SourceProbe{Size: obj.GetSize()} + } else if probe.Size == 0 { + probe.Size = obj.GetSize() + } + job.Probe = probe + profile := transcode.BuildProfile() + job.Profiles = []transcode.Profile{profile} + job.Output = transcode.BuildOutputSpec() + timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120) + job.Deadline = time.Now().Add(time.Duration(timeoutMin) * time.Minute).Unix() + submitted, _, err := mgr.Scheduler.SubmitLimited(job) + if err != nil { + // 队列已满 / 用户配额超限：返回 429，前端可降级为直链播放 + common.ErrorResp(c, err, 429) + return + } + respondTranscodeJob(c, apiURL, submitted) +} + +// respondTranscodeJob 返回转码任务的播放信息 +func respondTranscodeJob(c *gin.Context, apiURL string, job *transcode.Job) { + profileName := "" + if len(job.Profiles) > 0 { + profileName = job.Profiles[0].Name + } + masterURL := fmt.Sprintf("%s/tc/%s/%s/master.m3u8", apiURL, job.ID, job.PlayToken) + common.SuccessResp(c, gin.H{ + "transcode": true, + "job_id": job.ID, + "master_url": masterURL, + "profile": profileName, + }) +} + +func encodePath(p string) string { + // 与 down.go 保持一致：保留 '/' 不转义 + parts := strings.Split(p, "/") + for i, s := range parts { + parts[i] = url.PathEscape(s) + } + return strings.Join(parts, "/") +} + +// trustedInternalBaseURL 返回供服务端 ffmpeg 访问本站 /d 接口的可信基址，不读取任何请求头。 +// 优先使用管理员配置的 site_url；未配置时回环到本机实际监听的 HTTP 端口。 +// 两者都不可用（例如仅监听 Unix socket）时返回空串，调用方应拒绝转码。 +func trustedInternalBaseURL() string { + if site := strings.TrimSuffix(conf.Conf.SiteURL, "/"); strings.HasPrefix(site, "http") { + return site + } + if p := conf.Conf.Scheme.HttpPort; p > 0 { + return fmt.Sprintf("http://127.0.0.1:%d", p) + } + return "" +} + +// resolveLocalFilePath 尝试把 OpenList 路径解析为宿主机文件系统的实际路径。 +// 仅对本地驱动 (Local) 有效，其他驱动（云盘等）返回 false。 +// +// 这是内存优化的关键路径：本地文件场景下让 ffmpeg 直接读文件，可以完全绕过 +// HTTP /d 代理路径上的 net.Downloader 大块缓冲（默认每个 range 请求最高 +// 占用 MaxBufferLimit = 系统内存 5%，多 chunk 并发 + 多 range 累计可达数 GB）。 +func resolveLocalFilePath(rawPath string) (string, bool) { + storage, actualPath, err := op.GetStorageAndActualPath(rawPath) + if err != nil || storage == nil { + return "", false + } + if storage.Config().Name != "Local" { + return "", false + } + rooter, ok := storage.(driver.IRootPath) + if !ok { + return "", false + } + root := rooter.GetRootPath() + if root == "" { + return "", false + } + full := filepath.Join(root, actualPath) + // 只接受真实存在的文件，避免传给 ffmpeg 一个无效路径 + fi, err := os.Stat(full) + if err != nil || fi.IsDir() { + return "", false + } + return full, true +} + +// ============================================================ +// 播放端：/tc/:job/:token/... Master/Variant playlist & 切片 +// ============================================================ + +// TCMaster GET /tc/:job/:token/master.m3u8 +func TCMaster(c *gin.Context) { + jobID := c.Param("job") + token := c.Param("token") + job, ok := transcode.Default().Scheduler.Get(jobID) + if !ok { + c.String(http.StatusNotFound, "job not found") + return + } + if !tokenMatches(job.PlayToken, token) { + c.String(http.StatusForbidden, "bad token") + return + } + if len(job.Profiles) == 0 { + c.String(http.StatusInternalServerError, "no profiles") + return + } + apiURL := common.GetApiUrlFromRequest(c.Request) + baseURL := fmt.Sprintf("%s/tc/%s/%s/", apiURL, jobID, token) + body := transcode.Default().Cache.MasterPlaylist(job.Profiles, baseURL) + job.Touch() + c.Header("Content-Type", "application/vnd.apple.mpegurl") + // Master playlist 内容在 job 生命周期内不变，设置较长缓存 + c.Header("Cache-Control", "public, max-age=3600") + c.String(http.StatusOK, body) +} + +// TCPlaylist GET /tc/:job/:token/:profile/playlist.m3u8 +// 等待首切片就绪后返回；阻塞最多 30s +func TCPlaylist(c *gin.Context) { + jobID := c.Param("job") + token := c.Param("token") + profile := c.Param("profile") + mgr := transcode.Default() + job, ok := mgr.Scheduler.Get(jobID) + if !ok { + c.String(http.StatusNotFound, "job not found") + return + } + if !tokenMatches(job.PlayToken, token) { + c.String(http.StatusForbidden, "bad token") + return + } + if !transcode.ValidProfileName(profile) { + c.String(http.StatusBadRequest, "bad profile") + return + } + // 【关键】如果 job 之前因 idle 被 cancel，这里重新激活：复用 jobID/token 重新入队 + // 让 worker 再次启动 ffmpeg 转码，避免播放器看到 404 后无法恢复。 + // 配额/队列已满时不再起新的 ffmpeg，但 playlist 与已缓存的切片仍应可读： + // 这里只记下 quotaBlocked，交给切片请求在「缓存也没有」时才返回 429。 + quotaBlocked := false + if st := job.GetStatus(); st == transcode.JobCancelled || st == transcode.JobFinished || st == transcode.JobFailed { + if _, rerr := mgr.Scheduler.Reactivate(jobID); rerr == nil { + fmt.Printf("[transcode] reactivate job %s on playlist request (was %s)\n", jobID, st) + } else if errors.Is(rerr, transcode.ErrUserQuota) || errors.Is(rerr, transcode.ErrQueueFull) { + quotaBlocked = true + fmt.Printf("[transcode] reactivate job %s on playlist request rejected (%v), serving from cache only\n", jobID, rerr) + } + } + job.Touch() + // 等首切片；配额已满时无法启动新的转码，没必要干等 30s + if !quotaBlocked { + select { + case <-job.WaitReady(): + case <-time.After(30 * time.Second): + c.String(http.StatusGatewayTimeout, "timeout waiting first segment") + return + } + } + if job.Status == transcode.JobFailed { + c.String(http.StatusInternalServerError, "transcode failed: "+job.Error) + return + } + apiURL := common.GetApiUrlFromRequest(c.Request) + baseURL := fmt.Sprintf("%s/tc/%s/%s/%s/", apiURL, jobID, token, profile) + isFinished := job.Status == transcode.JobFinished + // 【关键】传入总时长，让 BuildPlaylist 能够生成完整切片列表（VOD 模式）， + // 这样 HLS.js 就知道视频真实总时长，避免时长显示错误和拖动跳跃问题。 + body := transcode.Default().Cache.BuildPlaylist(jobID, profile, job.Output.SegmentDuration, + isFinished, baseURL, job.Probe.Duration) + c.Header("Content-Type", "application/vnd.apple.mpegurl") + if isFinished || job.Probe.Duration > 0 { + // VOD 模式：playlist 完整不变，长缓存 + c.Header("Cache-Control", "public, max-age=3600") + } else { + // 没有总时长时退回 EVENT 模式：短缓存 + c.Header("Cache-Control", "public, max-age=2") + } + c.String(http.StatusOK, body) +} + +// TCSegment GET /tc/:job/:token/:profile/seg-:seq.ts +// 切片就绪则直接返回；否则等待最多 60s +func TCSegment(c *gin.Context) { + jobID := c.Param("job") + token := c.Param("token") + profile := c.Param("profile") + segName := c.Param("seg") + mgr := transcode.Default() + job, ok := mgr.Scheduler.Get(jobID) + if !ok { + c.String(http.StatusNotFound, "job not found") + return + } + if !tokenMatches(job.PlayToken, token) { + c.String(http.StatusForbidden, "bad token") + return + } + if !transcode.ValidProfileName(profile) { + c.String(http.StatusBadRequest, "bad profile") + return + } + // 【关键】job 之前因 idle 被 cancel 时，复用同一 jobID/token 重新入队启动转码。 + // 这样浏览器持有的旧 master.m3u8 + 切片 URL 仍然有效，暂停再继续也不会 404。 + // 配额/队列已满时不重新起 ffmpeg，但已缓存的切片依然要能命中（回看已转完的视频）。 + quotaBlocked := false + if st := job.GetStatus(); st == transcode.JobCancelled || st == transcode.JobFinished || st == transcode.JobFailed { + if _, rerr := mgr.Scheduler.Reactivate(jobID); rerr == nil { + fmt.Printf("[transcode] reactivate job %s on segment request (was %s)\n", jobID, st) + } else if errors.Is(rerr, transcode.ErrUserQuota) || errors.Is(rerr, transcode.ErrQueueFull) { + quotaBlocked = true + fmt.Printf("[transcode] reactivate job %s on segment request rejected (%v), serving from cache only\n", jobID, rerr) + } + } + // seg-N.ts + name := strings.TrimSuffix(strings.TrimPrefix(segName, "seg-"), ".ts") + seq, err := strconv.Atoi(name) + if err != nil { + c.String(http.StatusBadRequest, "bad segment name") + return + } + cache := mgr.Cache + // 【智能调度】触发对应 chunk 的 ffmpeg 启动（如已运行则只更新 LastAccess） + // 这是用户拖动进度条到任意位置时能快速响应的关键。 + // 配额已满时不能触发新的 ffmpeg，否则「限制并发」就没有意义了。 + if mgr.Chunks != nil && !quotaBlocked { + chunkIdx := mgr.Chunks.EnsureChunkRunningForSeg(jobID, profile, seq) + fmt.Printf("[tc-segment] req seq=%d chunk=%d job=%s\n", seq, chunkIdx, jobID) + } + info, err := cache.GetSegment(jobID, profile, seq) + if err != nil { + if quotaBlocked { + // 配额已满且该切片不在缓存里：只能拒绝，让前端降级为直链播放 + c.String(http.StatusTooManyRequests, "too many active transcode jobs") + return + } + // 等待 + waitStart := time.Now() + info, err = cache.WaitForSegment(jobID, profile, seq, 60*time.Second) + if err != nil { + fmt.Printf("[tc-segment] req seq=%d FAILED after %.1fs: %v\n", seq, time.Since(waitStart).Seconds(), err) + c.String(http.StatusNotFound, "segment not available: "+err.Error()) + return + } + fmt.Printf("[tc-segment] req seq=%d ready after %.1fs\n", seq, time.Since(waitStart).Seconds()) + } + job.Touch() + // 切片内容不可变，设置强缓存避免浏览器/HLS.js 反复请求同一切片 + c.Header("Cache-Control", "public, max-age=86400, immutable") + // 【内存优化】流式发送切片文件，避免 gin c.File() 将整个切片读入内存 + // 注意：不能使用 http.ServeFile，它会自动嗅探 Content-Type 并处理 Range 请求， + // 与 gin 已写入的响应头冲突，导致 HLS.js 收到异常数据触发 bufferAppendError。 + f, err := os.Open(info.Path) + if err != nil { + c.String(http.StatusInternalServerError, "open segment: "+err.Error()) + return + } + defer f.Close() + fi, err := f.Stat() + if err != nil { + c.String(http.StatusInternalServerError, "stat segment: "+err.Error()) + return + } + c.DataFromReader(http.StatusOK, fi.Size(), "video/mp2t", f, nil) +} + +// ============================================================ +// Worker 协议端 +// ============================================================ + +// WorkerRegister POST /api/transcode/worker/register +func WorkerRegister(c *gin.Context) { + if !transcode.IsEnabled() { + common.ErrorStrResp(c, "transcode disabled", 403) + return + } + if !checkWorkerSecret(c) { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + var req transcode.RegisterRequest + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + w := transcode.Default().Registry.Register(&req, false) + common.SuccessResp(c, transcode.RegisterResponse{ + WorkerID: w.ID, + HeartbeatInterval: 10, + ClaimStrategy: "pull", + ProtocolVersion: transcode.ProtocolVersion, + }) +} + +// WorkerHeartbeat POST /api/transcode/worker/heartbeat +func WorkerHeartbeat(c *gin.Context) { + if !checkWorkerSecret(c) { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + var req transcode.HeartbeatRequest + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + mgr := transcode.Default() + if _, ok := mgr.Registry.Heartbeat(&req); !ok { + common.SuccessResp(c, transcode.HeartbeatResponse{OK: false, Kick: true}) + return + } + cancellations := mgr.Scheduler.PopCancellations(req.WorkerID) + common.SuccessResp(c, transcode.HeartbeatResponse{OK: true, Cancel: cancellations}) +} + +// WorkerClaim POST /api/transcode/worker/claim +func WorkerClaim(c *gin.Context) { + if !checkWorkerSecret(c) { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + var req transcode.ClaimRequest + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + wait := time.Duration(req.Wait) * time.Second + if wait > 30*time.Second { + wait = 30 * time.Second + } + jobs := transcode.Default().Scheduler.Claim(req.WorkerID, req.Slots, wait) + common.SuccessResp(c, transcode.ClaimResponse{Jobs: jobs}) +} + +// WorkerSegmentUpload PUT /api/transcode/worker/segment?job=&profile=&seq=&duration=&final= +func WorkerSegmentUpload(c *gin.Context) { + jobID := c.Query("job") + profile := c.Query("profile") + seq, _ := strconv.Atoi(c.Query("seq")) + dur, _ := strconv.ParseFloat(c.Query("duration"), 64) + isFinal, _ := strconv.ParseBool(c.Query("final")) + job, ok := transcode.Default().Scheduler.Get(jobID) + if !ok { + common.ErrorStrResp(c, "job not found", 404) + return + } + // 校验 callback_token (Bearer)，使用恒定时间比较避免时序侧信道 + auth := c.GetHeader("Authorization") + expect := "Bearer " + job.CallbackToken + if subtle.ConstantTimeCompare([]byte(auth), []byte(expect)) != 1 { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + // profile 会被拼进缓存目录路径，必须是白名单字符，防止 ../ 路径穿越 + if !transcode.ValidProfileName(profile) { + common.ErrorStrResp(c, "invalid profile", 400) + return + } + // 只接受任务声明过的档位，避免持有 callback_token 的一方往任意档位目录写数据 + if !jobHasProfile(job, profile) { + common.ErrorStrResp(c, "profile not declared for this job", 400) + return + } + defer c.Request.Body.Close() + _, err := transcode.Default().Cache.PutSegment(jobID, profile, seq, dur, c.Request.Body, isFinal) + if err != nil { + switch err { + case transcode.ErrInvalidProfile, transcode.ErrInvalidJobID, transcode.ErrInvalidSegment: + common.ErrorResp(c, err, 400) + case transcode.ErrSegmentTooLarge: + common.ErrorResp(c, err, 413) + default: + common.ErrorResp(c, err, 500) + } + return + } + if seq == 0 { + job.MarkReady() + } + c.Status(http.StatusNoContent) +} + +// WorkerJobFinish POST /api/transcode/worker/job/finish +func WorkerJobFinish(c *gin.Context) { + var req transcode.FinishRequest + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + job, ok := transcode.Default().Scheduler.Get(req.JobID) + if !ok { + common.ErrorStrResp(c, "job not found", 404) + return + } + auth := c.GetHeader("Authorization") + if !tokenMatches("Bearer "+job.CallbackToken, auth) { + common.ErrorStrResp(c, "unauthorized", 401) + return + } + if err := transcode.Default().Scheduler.Finish(&req); err != nil { + common.ErrorResp(c, err, 500) + return + } + common.SuccessResp(c) +} + +// ListTranscodeWorkers GET /api/admin/transcode/workers (admin) +func ListTranscodeWorkers(c *gin.Context) { + common.SuccessResp(c, transcode.Default().Registry.List()) +} + +// ListTranscodeJobs GET /api/admin/transcode/jobs (admin) +func ListTranscodeJobs(c *gin.Context) { + common.SuccessResp(c, transcode.Default().Scheduler.ListJobs()) +} + +// CancelTranscodeJob POST /api/admin/transcode/cancel +func CancelTranscodeJob(c *gin.Context) { + var req struct { + JobID string `json:"job_id" binding:"required"` + } + if err := c.ShouldBindJSON(&req); err != nil { + common.ErrorResp(c, err, 400) + return + } + transcode.Default().Scheduler.Cancel(req.JobID) + common.SuccessResp(c) +} + +// jobHasProfile 判断 profile 是否属于任务声明的输出档位 +func jobHasProfile(job *transcode.Job, profile string) bool { + if job == nil { + return false + } + for _, p := range job.Profiles { + if p.Name == profile { + return true + } + } + return false +} + +// tokenMatches 恒定时间比较凭据，避免按字节比较造成的时序侧信道 +func tokenMatches(expected, got string) bool { + if expected == "" || got == "" { + return false + } + return subtle.ConstantTimeCompare([]byte(expected), []byte(got)) == 1 +} + +func checkWorkerSecret(c *gin.Context) bool { + auth := c.GetHeader("Authorization") + auth = strings.TrimPrefix(auth, "Bearer ") + return transcode.VerifyWorkerSecret(auth) +} diff --git a/server/router.go b/server/router.go index 5fae18c0..fe210e6e 100644 --- a/server/router.go +++ b/server/router.go @@ -67,6 +67,11 @@ func Init(e *gin.Engine) { g.HEAD("/sad/:sid", middlewares.EmptyPathParse, middlewares.SharingIdParse, handles.SharingArchiveExtract) g.HEAD("/sad/:sid/*path", middlewares.PathParse, middlewares.SharingIdParse, handles.SharingArchiveExtract) + // 转码播放：master/variant playlist 与切片下发；token 已包含在路径中作为鉴权 + g.GET("/tc/:job/:token/master.m3u8", handles.TCMaster) + g.GET("/tc/:job/:token/:profile/playlist.m3u8", handles.TCPlaylist) + g.GET("/tc/:job/:token/:profile/:seg", handles.TCSegment) + api := g.Group("/api") auth := api.Group("", middlewares.Auth(false)) webauthn := api.Group("/authn", middlewares.Authn) @@ -76,6 +81,10 @@ func Init(e *gin.Engine) { api.POST("/auth/login/ldap", handles.LoginLdap) auth.GET("/me", handles.CurrentUser) auth.POST("/me/update", handles.UpdateCurrent) + // 媒体应用密钥（每个登录用户都可以为自己的客户端签发密钥） + auth.GET("/me/appkey/list", handles.ListMediaAppKeys) + auth.POST("/me/appkey/create", handles.CreateMediaAppKey) + auth.POST("/me/appkey/delete", handles.DeleteMediaAppKey) auth.GET("/me/sshkey/list", handles.ListMyPublicKey) auth.POST("/me/sshkey/add", handles.AddMyPublicKey) auth.POST("/me/sshkey/delete", handles.DeleteMyPublicKey) @@ -100,12 +109,51 @@ func Init(e *gin.Engine) { // no need auth public := api.Group("/public") public.Any("/settings", handles.PublicSettings) + public.GET("/media_status", handles.PublicMediaStatus) + // 封面代理与本地缩略图（媒体库页面使用，免鉴权） + public.GET("/media/proxy", handles.PublicMediaProxy) + public.GET("/media/thumb", handles.PublicMediaThumb) public.Any("/offline_download_tools", handles.OfflineDownloadTools) public.Any("/archive_extensions", handles.ArchiveExtensions) public.Any("/init_status", handles.InitStatus) public.POST("/init/setup", handles.InitSetup) + // 转码 Worker 协议端点（远程 Worker 节点对接，由共享密钥/token 鉴权，无需登录） + transcodeWorker := api.Group("/transcode/worker") + transcodeWorker.POST("/register", handles.WorkerRegister) + transcodeWorker.POST("/heartbeat", handles.WorkerHeartbeat) + transcodeWorker.POST("/claim", handles.WorkerClaim) + transcodeWorker.PUT("/segment", handles.WorkerSegmentUpload) + transcodeWorker.POST("/job/finish", handles.WorkerJobFinish) + + // Emby 兼容 API（自带鉴权与开关判断，路径大小写由 handler 归一化） + // 使用 @ 前缀，避免与用户自己创建的 /emby、/rest 目录冲突 + g.Any("/@emby/*path", handles.EmbyAPI) + g.Any("/@mediabrowser/*path", handles.EmbyAPI) + // OpenSubsonic API（自带鉴权与开关判断） + g.Any("/@rest/*path", handles.SubsonicAPI) + _fs(auth.Group("/fs")) + // 媒体库公开 API：与文件浏览一致，允许访客访问（访客仍受自身 base_path/权限限制） + mediaPublic := auth.Group("/fs/media", handles.RequireMediaEnabled) + mediaPublic.GET("/list", handles.PublicListMedia) + mediaPublic.GET("/item/:id", handles.PublicGetMedia) + mediaPublic.GET("/albums", handles.PublicListAlbums) + mediaPublic.GET("/album", handles.PublicGetAlbum) + mediaPublic.GET("/folders", handles.PublicListFolders) + mediaPublic.GET("/scan_paths", handles.PublicListScanPaths) + mediaPublic.GET("/strm", handles.GetMediaStrm) + mediaPublic.GET("/stream_url", handles.GetMediaStreamURL) + // 播放进度与收藏（与 Emby / Subsonic 共用同一份用户数据） + mediaPublic.GET("/userdata", handles.GetMediaUserDataBatch) + mediaPublic.GET("/progress", handles.GetMediaProgress) + mediaPublic.POST("/progress", handles.SaveMediaProgress) + mediaPublic.POST("/favorite", handles.SetMediaFavorite) + mediaPublic.GET("/favorites", handles.ListMediaFavoritesAPI) + mediaPublic.GET("/resume", handles.ListMediaResumeAPI) + // 阅读器批注（登录用户私有数据；访客只读返回空列表，写入直接 403） + mediaPublic.GET("/annotations", handles.ListMediaAnnotationsAPI) + mediaPublic.POST("/annotations", handles.SyncMediaAnnotationsAPI) fsAndShare(api.Group("/fs", middlewares.Auth(true))) _task(auth.Group("/task", middlewares.AuthNotGuest)) _sharing(auth.Group("/share", middlewares.AuthNotGuest)) @@ -190,6 +238,42 @@ func admin(g *gin.RouterGroup) { scan.POST("/start", handles.StartManualScan) scan.POST("/stop", handles.StopManualScan) scan.GET("/progress", handles.GetManualScanProgress) + + // 媒体库管理 + mediaAdmin := g.Group("/media") + mediaAdmin.GET("/config/list", handles.ListMediaConfigs) + mediaAdmin.POST("/config/save", handles.SaveMediaConfig) + mediaAdmin.GET("/items", handles.ListMediaItemsAdmin) + mediaAdmin.POST("/items/update", handles.UpdateMediaItemAdmin) + mediaAdmin.POST("/items/delete", handles.DeleteMediaItemAdmin) + mediaAdmin.POST("/scan/start", handles.StartMediaScan) + mediaAdmin.GET("/scan/progress", handles.GetMediaScanProgress) + mediaAdmin.POST("/scrape/start", handles.StartMediaScrape) + mediaAdmin.GET("/scrape/stats", handles.GetMediaScrapeStats) + mediaAdmin.POST("/clear", handles.ClearMediaDB) + mediaAdmin.POST("/clear_scrape", handles.ClearMediaScrape) + mediaAdmin.POST("/delete_invalid", handles.DeleteInvalidMedia) + // 扫描路径管理 + mediaAdmin.GET("/scan_paths", handles.ListMediaScanPaths) + mediaAdmin.POST("/scan_paths/create", handles.CreateMediaScanPath) + mediaAdmin.POST("/scan_paths/update", handles.UpdateMediaScanPath) + mediaAdmin.POST("/scan_paths/delete", handles.DeleteMediaScanPath) + mediaAdmin.POST("/scan_paths/clear", handles.ClearMediaScanPathDB) + // 导入 / 导出 + mediaAdmin.GET("/export", handles.ExportMediaDB) + mediaAdmin.POST("/import", handles.ImportMediaDB) + // 媒体应用密钥（Subsonic/Emby/MoviePilot 等无交互客户端） + mediaAdmin.GET("/appkey/list", handles.ListMediaAppKeys) + mediaAdmin.POST("/appkey/create", handles.CreateMediaAppKey) + mediaAdmin.POST("/appkey/delete", handles.DeleteMediaAppKey) + // STRM 批量生成 + mediaAdmin.POST("/strm/generate", handles.GenerateMediaStrm) + + // 转码管理 + transcodeAdmin := g.Group("/transcode") + transcodeAdmin.GET("/workers", handles.ListTranscodeWorkers) + transcodeAdmin.GET("/jobs", handles.ListTranscodeJobs) + transcodeAdmin.POST("/cancel", handles.CancelTranscodeJob) } func fsAndShare(g *gin.RouterGroup) { @@ -198,6 +282,9 @@ func fsAndShare(g *gin.RouterGroup) { a := g.Group("/archive") a.Any("/meta", handles.FsArchiveMetaSplit) a.Any("/list", handles.FsArchiveListSplit) + + // 转码 - 用户态：申请播放 + g.POST("/transcode/play", handles.RequireMediaEnabled, handles.TranscodePlay) } func _fs(g *gin.RouterGroup) {