mirror of
https://github.com/OpenListTeam/OpenList.git
synced 2026-11-10 20:50:50 +08:00
feat(func): media library (#2210)
* feat(func): media library
* feat(func): media library
* feat(func): media library
* feat(func): media library build test
* feat(func): media library build test
* feat(func): media library fix remote fetch error
* feat(func): media library fix remote fetch error
* feat(func): media library fix remote fetch error
* feat(func): media library support select
* feat(func): media library support select
* feat(func): media library support select
* feat(func): media library support select
* feat(func): media library support change api url
* feat(func): media library support change api url
* feat(func): media library support change api url
* feat(media): 并发刮削（默认 5）+ 单条落库
增强 TMDB 搜索：模糊匹配 + 多候选关键词
管理页面新增导入/导出媒体库（全部 / 按扫描路径）
缩略图路径默认值改为 imgs
* feat(media): 清空刮削、删除已失效
* fix(media): fix api proxy path
* fix(media): fix lrc file display & music id3
* fix(media): fix media scan sub path error
* feat(media): support cloud codec via ffmpeg
* feat(media): support cloud codec via ffmpeg
* feat(media): support cloud codec via ffmpeg
* feat(media): support cloud codec via ffmpeg
* feat(media): 提升刮削精准度，支持模糊刮削
* fix(media): clean up imports and improve FetchFileReader logic
* feat(media): 提升刮削精准度，支持模糊刮削
* feat(media): 刮削支持自定义刮削数量，避免太占性能
* feat(media): 刮削支持自定义刮削数量，避免太占性能
* feat(media): 刮削修复写入作品名问题
* fix(media): return 404 on missing item_id and fix unscraped item query
- Validate GetMediaItemByID result before spawning goroutine in
StartMediaScrape; return 404 when item_id > 0 but item is not found
- Replace `description` check with `plot` check in GetUnscrappedItems
since scrapers populate Plot, not Description, to avoid re-selecting
already-scraped items on every run
* Apply remaining changes
* feat(media): add master switch, app keys, STRM, Emby and OpenSubsonic APIs
* feat(media): allow any logged-in user to manage their own media app keys
* feat(media): allow signing arbitrary media path for local player
* fix(media): always return an array for media status types
* fix(media): persist per-type scraper config and apply it to thumbnails
* feat(media): add local thumbnail endpoint and cover proxy with fallback gateway
* fix(media): allow guest media access, per-file local thumbnails and optional cover proxy
* feat(media): thumbnail storage modes and STRM same-directory output
* feat(media): Google Books/OpenLibrary sources and remote cover download
* chore(media): rename transcode cache dir to ctmp and log thumbnail storage dir
* feat(media): progress and favorites API shared with Emby/Subsonic
* chore: ignore runtime thumbnail directory
* feat(media): add preferred scrape language and namespace protocol endpoints under @
* fix(media): harden anonymous emby images, pagination and cover proxy SSRF
* fix(emby): allow anonymous user discovery and login before auth
* chore: drop redundant media build workflow and protocol doc
* fix(media): ID3v2.4 syncsafe frame size and preserve Hidden flag on rescan
* fix(media): address security review (path traversal, guest bypass, cover XSS, auth rate limit, SSRF, quotas, migration)
* fix(media): 清理失效条目时只删除确认不存在的记录
此前 fs.Get 返回任何错误都视为失效并硬删除，驱动故障/超时/鉴权失败时会误删有效媒体记录。现在仅 errs.IsObjectNotFound 才删除，其余错误跳过并在响应中返回 skipped 计数。
* fix(transcode): 播放凭据与 Worker 回写凭据分离，并收紧 profile 与鉴权校验
- 新增 Job.PlayToken（json 标签为 -），master/playlist/切片只认播放凭据，播放器不再拿到可上传、可结束任务的 callback_token
- Worker 上传切片时校验 profile 必须属于任务声明的档位
- TCMaster 与 WorkerJobFinish 改用恒定时间比较，避免时序侧信道
* fix(transcode): 含冒号的本地文件名可转码，补基址缺失与重新激活配额
- ValidSourceInput 不再按冒号位置判协议：本地输入只要求以 / 开头或 Windows 盘符路径，concat:/subfile: 等前缀仍被拒且由 -protocol_whitelist 兜底（修复 /mnt/x/Star Wars: IV.mkv 被误判）
- trustedInternalBaseURL 返回空串时直接 503 并给出明确原因，不再拼出 /d/xxx 让 ffmpeg 报文件不存在
- 远程输入白名单收窄为 http,https,tcp,tls（去掉 file 与 crypto）
- Reactivate 复用同一套单用户配额，避免「先让任务闲置取消、再批量激活」绕过 SubmitLimited；配额不足返回 429
- OwnerID 改为配额键：普通用户 user:<id>，访客 guest:<clientIP>，避免单个匿名访客占满全部匿名配额；补充对应单测
* fix(media): 收紧缩略图解码内存、跳过超长条目可见化，并补 CI 单测
- MaxImagePixels 从 64MP 降到 32MP，宽高改 int64 相乘防 32 位溢出，解码阶段加 12 路全局信号量
- 封面解码失败退回原始字节时限制 512KB，避免十几 MB 的 base64 写进 cover 列（豆瓣/本地书籍三处）
- 扫描进度新增 Skipped 计数，入库被跳过的超长路径会在完成消息与日志中体现
- 数据库迁移失败日志补充 OPENLIST_MEDIA_MIGRATE_WIPE=1 的处置提示
- Subsonic 认证只对「确实带了凭据却校验失败」计数，裸探测不再刷新 IP 锁定
- goexif 从 indirect 调整为直接依赖（internal/media/scraper/image.go 直接 import）
- CI 新增 go test ./internal/db/... ./internal/transcode/... ./internal/media/...
* fix(transcode): 配额/队列满时仍放行已缓存切片，并同步 worker 协议白名单
- TCSegment/TCPlaylist: Reactivate 返回 ErrUserQuota/ErrQueueFull 时不再直接 429，改为标记 quotaBlocked，缓存命中的切片与 playlist 仍可读，只有缓存缺失才返回 429
- Reactivate: 补充全局排队上限 (MaxPendingJobs) 检查，避免批量激活顶爆队列，并补单测
- cmd/worker: 远程输入协议白名单与 master 同步为 http,https,tcp,tls（去掉 file/crypto）
- scraper: decodeSem 容量 12 -> 3，避免信号量大于默认刮削并发而形同失效
* feat(emby): 完善客户端兼容层并修复认证锁定，Subsonic 复用统一密钥校验
- Emby: 用户 ID 改用 GUID 并兼容数字/旧 ID；补齐 /Users/{id}/Authenticate、Sessions 与 WebSocket、首页分区、合集/播放列表、字幕流与歌词、Series/Season 结构
- Emby: 版本号返回客户端可接受的兼容版本，/Users/Public 返回可登录用户以渲染用户选择
- Subsonic: 支持 enc: 十六进制密码编码；app key 校验与 Emby 共用 LookUpAppKeyUser
- 限流: 认证被拒时不再刷新锁定过期时间，避免客户端持续轮询把 IP 永久锁死
* fix(media): 认证失败计数增加用户名维度，防止伪造 X-Forwarded-For 绕过锁定
- media_ratelimit: 新增 mediaAuthLockedFor / mediaAuthFailedFor / mediaAuthSucceededFor，IP 与用户名双维度计数；达到上限同样锁定 5 分钟且被拒时不续期
- emby: 按用户名登录与按 id 登录都检查用户名维度；失败计入两个维度，成功清除两个维度
- subsonic: subAuth 与 429 分支改用双维度判定；成功时用真实用户名清理计数（apiKey 认证时 u 参数为空）
- 仍保留主干 5 次/5 分钟的策略，代理信任范围（SetTrustedProxies）留给项目层面单独处理
* fix(media): 匿名封面按访客身份过滤，认证日志脱敏并收紧限流计数
- embyImage: 匿名封面不再走 loadEmbyLibraryAll（全库可见），改为 loadEmbyLibrary(guest)：
hidden / base_path 之外 / meta 受保护的条目都不可见，访客被禁用时直接 401
- embyUnauthorized: 只记录 Path、UA 与是否带 token，避免 query 里的 Pw/Password/api_key
和 X-Emby-Authorization 里的设备令牌写进日志
- media_ratelimit: 失败计数 Set 时带 5min TTL，避免 IP/用户名 key 无限增长（用户名来自请求参数）
- auth.loginHash: 锁定期间不再续期，失败计数同样带 TTL，与媒体侧语义一致
- /System/Endpoint: 按 RemoteAddr 判断 IsLocal/IsInNetwork；/System/Info 不再返回真实监听端口
* feat(media): PDF 批注持久化
- 新增 MediaAnnotation 模型：按 (user_id, item_id, remote_id) 唯一，anchor 原样保存查看器批注对象，不重复建模页码/坐标
- GET/POST /fs/media/annotations：读取当前用户批注、按 remote_id 增量 upsert/delete（单条 512KB、单次 8MB、最多 500 条）
- 批注为用户私有数据：未登录返回空列表，访客写入 403，并校验条目可见性与访问权限
- 条目删除 / 清空类型 / 清空扫描路径 / 批量删除时，在同一事务内清理对应批注
* fix(media): 选集签名前校验最终文件权限；转码在 ffprobe 前预检配额
- GetMediaStreamURL / GetMediaStrm: 解析选集后对最终 rawPath 再执行 CanUserAccessPath，避免「目录可见但 meta 隐藏规则禁止其中某个文件」时仍能拿到有效签名（签名之后下载链路不会重跑隐藏规则）
- TranscodePlay: ProbeSource 之前调用 Scheduler.Capacity 预检配额，已满直接 429，不再先起最长 30s 的 ffprobe
- 新增探测并发闸门（全局 8 / 每用户 3），覆盖预检与 SubmitLimited 之间的并发窗口；用 defer 释放，探测 panic 也不会占死槽位
- 补 Capacity 单测
---------
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: pikachuren <pikachuren@example.com>
This commit is contained in:
61 files changed
+18937
-10
No files matched your search
@@ -10,6 +10,22 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# 媒体库相关的安全/配额/迁移测试单独跑一次，不必跟随 7 个构建目标重复执行
|
||||
test:
|
||||
name: Unit tests
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
|
||||
- name: Setup Go
|
||||
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
|
||||
with:
|
||||
go-version: "1.27.3"
|
||||
|
||||
- name: Go test
|
||||
run: go test ./internal/db/... ./internal/transcode/... ./internal/media/...
|
||||
|
||||
build:
|
||||
strategy:
|
||||
matrix:
|
||||
|
||||
@@ -28,6 +28,7 @@ output/
|
||||
/tmp/
|
||||
/log/
|
||||
/lang/
|
||||
/imgs/
|
||||
/daemon/
|
||||
/public/dist/*
|
||||
!/public/dist/README.md
|
||||
|
||||
@@ -0,0 +1,580 @@
|
||||
// Command openlist-transcode-worker 是独立的远程转码 Worker 节点。
|
||||
// 它通过 OpenList 自研协议（HTTP/JSON）注册到 Master，长轮询领任务，调用
|
||||
// 本机 ffmpeg 转码，并把生成的 HLS 切片实时上传回 Master。
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
masterURL = flag.String("master", "http://127.0.0.1:5244", "OpenList Master URL")
|
||||
secret = flag.String("secret", "", "Worker 共享密钥（与 Master 配置一致）")
|
||||
name = flag.String("name", hostname(), "Worker 名称")
|
||||
capacity = flag.Int("capacity", 2, "并发任务数")
|
||||
hwaccel = flag.String("hwaccel", "none", "硬件加速：none/auto/nvenc/qsv/vaapi/amf/videotoolbox")
|
||||
workdir = flag.String("workdir", "./worker_tmp", "临时工作目录")
|
||||
ffmpeg = flag.String("ffmpeg", "ffmpeg", "ffmpeg 路径")
|
||||
)
|
||||
|
||||
func hostname() string {
|
||||
h, _ := os.Hostname()
|
||||
if h == "" {
|
||||
h = "worker"
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// ---- 协议结构 ----（与 Master 端 internal/transcode/protocol.go 一致）
|
||||
|
||||
type RegisterReq struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Capacity int `json:"capacity"`
|
||||
HWAccel []string `json:"hwaccel"`
|
||||
CodecsDecode []string `json:"codecs_decode"`
|
||||
CodecsEncode []string `json:"codecs_encode"`
|
||||
MaxResolution string `json:"max_resolution"`
|
||||
Tags []string `json:"tags"`
|
||||
}
|
||||
type RegisterResp struct {
|
||||
Data struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
HeartbeatInterval int `json:"heartbeat_interval"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
type HeartbeatReq struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
Load float64 `json:"load"`
|
||||
Running []string `json:"running"`
|
||||
FreeSlots int `json:"free_slots"`
|
||||
}
|
||||
type HeartbeatResp struct {
|
||||
Data struct {
|
||||
OK bool `json:"ok"`
|
||||
Kick bool `json:"kick"`
|
||||
Cancel []string `json:"cancel"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
type ClaimReq struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
Slots int `json:"slots"`
|
||||
Wait int `json:"wait"`
|
||||
}
|
||||
|
||||
type Profile struct {
|
||||
Name string `json:"name"`
|
||||
VideoCodec string `json:"video_codec"`
|
||||
VideoBitrate string `json:"video_bitrate"`
|
||||
Scale string `json:"scale"`
|
||||
AudioCodec string `json:"audio_codec"`
|
||||
AudioBitrate string `json:"audio_bitrate"`
|
||||
HWAccel string `json:"hwaccel"`
|
||||
}
|
||||
type OutputSpec struct {
|
||||
Format string `json:"format"`
|
||||
SegmentDuration int `json:"segment_duration"`
|
||||
}
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
Path string `json:"path"`
|
||||
SourceURL string `json:"source_url"`
|
||||
Profiles []Profile `json:"profiles"`
|
||||
Output OutputSpec `json:"output"`
|
||||
CallbackToken string `json:"callback_token"`
|
||||
Deadline int64 `json:"deadline"`
|
||||
}
|
||||
type ClaimResp struct {
|
||||
Data struct {
|
||||
Jobs []Job `json:"jobs"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
type FinishReq struct {
|
||||
JobID string `json:"job_id"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Stats struct {
|
||||
Elapsed float64 `json:"elapsed"`
|
||||
BytesOut int64 `json:"bytes_out"`
|
||||
} `json:"stats"`
|
||||
}
|
||||
|
||||
// ---- HTTP helper ----
|
||||
|
||||
var httpClient = &http.Client{Timeout: 60 * time.Second}
|
||||
|
||||
func postJSON(path string, body any, out any, bearer string) error {
|
||||
buf, _ := json.Marshal(body)
|
||||
req, _ := http.NewRequest(http.MethodPost, *masterURL+path, bytes.NewReader(buf))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if bearer != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+bearer)
|
||||
}
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode >= 400 {
|
||||
raw, _ := io.ReadAll(resp.Body)
|
||||
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw))
|
||||
}
|
||||
if out != nil {
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---- main ----
|
||||
|
||||
func main() {
|
||||
flag.Parse()
|
||||
if *secret == "" {
|
||||
fmt.Fprintln(os.Stderr, "missing -secret")
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := os.MkdirAll(*workdir, 0o755); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
workerID, hbInterval, err := registerOnce()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "register failed:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("[worker] registered: id=%s heartbeat=%ds\n", workerID, hbInterval)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
sigCh := make(chan os.Signal, 1)
|
||||
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-sigCh
|
||||
fmt.Println("[worker] shutting down")
|
||||
cancel()
|
||||
}()
|
||||
|
||||
state := &workerState{
|
||||
id: workerID,
|
||||
runningMu: sync.Mutex{},
|
||||
running: map[string]context.CancelFunc{},
|
||||
}
|
||||
|
||||
// 心跳协程
|
||||
go func() {
|
||||
t := time.NewTicker(time.Duration(hbInterval) * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
state.heartbeat()
|
||||
}
|
||||
}
|
||||
}()
|
||||
// 工作协程
|
||||
for i := 0; i < *capacity; i++ {
|
||||
go state.workLoop(ctx)
|
||||
}
|
||||
<-ctx.Done()
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
|
||||
func registerOnce() (string, int, error) {
|
||||
req := RegisterReq{
|
||||
Name: *name,
|
||||
Version: "1.0.0",
|
||||
Capacity: *capacity,
|
||||
HWAccel: []string{*hwaccel},
|
||||
CodecsDecode: []string{"h264", "hevc", "av1", "vp9", "mpeg4", "mpeg2video"},
|
||||
CodecsEncode: []string{"h264", "hevc"},
|
||||
MaxResolution: "3840x2160",
|
||||
Tags: []string{runtime.GOOS, runtime.GOARCH},
|
||||
}
|
||||
var resp RegisterResp
|
||||
if err := postJSON("/api/transcode/worker/register", req, &resp, *secret); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
if resp.Data.WorkerID == "" {
|
||||
return "", 0, fmt.Errorf("master returned empty worker_id")
|
||||
}
|
||||
if resp.Data.HeartbeatInterval == 0 {
|
||||
resp.Data.HeartbeatInterval = 10
|
||||
}
|
||||
return resp.Data.WorkerID, resp.Data.HeartbeatInterval, nil
|
||||
}
|
||||
|
||||
// ---- 状态机 ----
|
||||
|
||||
type workerState struct {
|
||||
id string
|
||||
runningMu sync.Mutex
|
||||
running map[string]context.CancelFunc
|
||||
}
|
||||
|
||||
func (s *workerState) heartbeat() {
|
||||
s.runningMu.Lock()
|
||||
ids := make([]string, 0, len(s.running))
|
||||
for id := range s.running {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
free := *capacity - len(ids)
|
||||
s.runningMu.Unlock()
|
||||
var resp HeartbeatResp
|
||||
if err := postJSON("/api/transcode/worker/heartbeat", HeartbeatReq{
|
||||
WorkerID: s.id,
|
||||
Running: ids,
|
||||
FreeSlots: free,
|
||||
}, &resp, *secret); err != nil {
|
||||
fmt.Println("[worker] heartbeat error:", err)
|
||||
return
|
||||
}
|
||||
for _, jid := range resp.Data.Cancel {
|
||||
s.runningMu.Lock()
|
||||
if cancel, ok := s.running[jid]; ok {
|
||||
cancel()
|
||||
}
|
||||
s.runningMu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *workerState) workLoop(ctx context.Context) {
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
var resp ClaimResp
|
||||
err := postJSON("/api/transcode/worker/claim", ClaimReq{
|
||||
WorkerID: s.id, Slots: 1, Wait: 20,
|
||||
}, &resp, *secret)
|
||||
if err != nil {
|
||||
fmt.Println("[worker] claim error:", err)
|
||||
time.Sleep(5 * time.Second)
|
||||
continue
|
||||
}
|
||||
if len(resp.Data.Jobs) == 0 {
|
||||
continue
|
||||
}
|
||||
for _, j := range resp.Data.Jobs {
|
||||
s.runJob(ctx, j)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *workerState) runJob(parent context.Context, job Job) {
|
||||
jobCtx, cancel := context.WithCancel(parent)
|
||||
s.runningMu.Lock()
|
||||
s.running[job.ID] = cancel
|
||||
s.runningMu.Unlock()
|
||||
defer func() {
|
||||
s.runningMu.Lock()
|
||||
delete(s.running, job.ID)
|
||||
s.runningMu.Unlock()
|
||||
cancel()
|
||||
}()
|
||||
|
||||
startedAt := time.Now()
|
||||
tmpDir := filepath.Join(*workdir, job.ID)
|
||||
_ = os.MkdirAll(tmpDir, 0o755)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
if len(job.Profiles) == 0 {
|
||||
s.finish(job, "failed", "no profiles", 0, 0)
|
||||
return
|
||||
}
|
||||
profile := job.Profiles[0]
|
||||
playlist := filepath.Join(tmpDir, "playlist.m3u8")
|
||||
segPattern := filepath.Join(tmpDir, "seg-%d.ts")
|
||||
args := buildFFmpegArgs(job, profile, segPattern, playlist)
|
||||
|
||||
cmd := exec.CommandContext(jobCtx, *ffmpeg, args...)
|
||||
stderr := &bytes.Buffer{}
|
||||
cmd.Stderr = stderr
|
||||
fmt.Printf("[worker] running job %s, ffmpeg %s\n", job.ID, strings.Join(args, " "))
|
||||
if err := cmd.Start(); err != nil {
|
||||
s.finish(job, "failed", err.Error(), 0, 0)
|
||||
return
|
||||
}
|
||||
|
||||
// 上传协程
|
||||
stopUpload := make(chan struct{})
|
||||
uploadDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(uploadDone)
|
||||
s.watchAndUpload(jobCtx, job, profile.Name, tmpDir, stopUpload)
|
||||
}()
|
||||
|
||||
err := cmd.Wait()
|
||||
close(stopUpload)
|
||||
<-uploadDone
|
||||
// 最后一次扫描：把还未上传的尾片传上去
|
||||
s.uploadOnce(job, profile.Name, tmpDir, true)
|
||||
|
||||
if err != nil {
|
||||
if jobCtx.Err() == context.Canceled {
|
||||
s.finish(job, "cancelled", "cancelled by master", 0, 0)
|
||||
return
|
||||
}
|
||||
s.finish(job, "failed", fmt.Sprintf("%v: %s", err, truncate(stderr.String(), 1024)), 0, 0)
|
||||
return
|
||||
}
|
||||
s.finish(job, "finished", "", time.Since(startedAt).Seconds(), 0)
|
||||
}
|
||||
|
||||
func (s *workerState) finish(job Job, status, errStr string, elapsed float64, bytesOut int64) {
|
||||
req := FinishReq{
|
||||
JobID: job.ID, Status: status, Error: errStr,
|
||||
}
|
||||
req.Stats.Elapsed = elapsed
|
||||
req.Stats.BytesOut = bytesOut
|
||||
if err := postJSON("/api/transcode/worker/job/finish", req, nil, job.CallbackToken); err != nil {
|
||||
fmt.Printf("[worker] finish report failed: %v\n", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *workerState) watchAndUpload(_ context.Context, job Job, profile, dir string, stop chan struct{}) {
|
||||
t := time.NewTicker(800 * time.Millisecond)
|
||||
defer t.Stop()
|
||||
uploaded := map[int]bool{}
|
||||
flush := func(final bool) {
|
||||
s.uploadDelta(job, profile, dir, uploaded, final)
|
||||
}
|
||||
for {
|
||||
select {
|
||||
case <-stop:
|
||||
flush(false)
|
||||
return
|
||||
case <-t.C:
|
||||
flush(false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *workerState) uploadOnce(job Job, profile, dir string, final bool) {
|
||||
uploaded := map[int]bool{}
|
||||
s.uploadDelta(job, profile, dir, uploaded, final)
|
||||
}
|
||||
|
||||
// uploadDelta 扫描目录，上传新切片（uploaded 跟踪已传过的 seq；最后一片到 final 时才上传）
|
||||
func (s *workerState) uploadDelta(job Job, profile, dir string, uploaded map[int]bool, final bool) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
type segFile struct {
|
||||
seq int
|
||||
path string
|
||||
}
|
||||
var segs []segFile
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "seg-") || !strings.HasSuffix(n, ".ts") {
|
||||
continue
|
||||
}
|
||||
num, err := strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(n, "seg-"), ".ts"))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
segs = append(segs, segFile{num, filepath.Join(dir, n)})
|
||||
}
|
||||
// 排序
|
||||
for i := 0; i < len(segs); i++ {
|
||||
for j := i + 1; j < len(segs); j++ {
|
||||
if segs[j].seq < segs[i].seq {
|
||||
segs[i], segs[j] = segs[j], segs[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
limit := len(segs)
|
||||
if !final && limit > 0 {
|
||||
limit-- // 最后一片可能还在写，留到下一轮
|
||||
}
|
||||
segDur := job.Output.SegmentDuration
|
||||
if segDur == 0 {
|
||||
segDur = 6
|
||||
}
|
||||
for i := 0; i < limit; i++ {
|
||||
seg := segs[i]
|
||||
if uploaded[seg.seq] {
|
||||
continue
|
||||
}
|
||||
isLast := final && i == limit-1
|
||||
if err := s.uploadSegment(job, profile, seg.seq, float64(segDur), seg.path, isLast); err != nil {
|
||||
fmt.Printf("[worker] upload seg-%d failed: %v\n", seg.seq, err)
|
||||
return
|
||||
}
|
||||
uploaded[seg.seq] = true
|
||||
}
|
||||
}
|
||||
|
||||
func (s *workerState) uploadSegment(job Job, profile string, seq int, dur float64, file string, final bool) error {
|
||||
f, err := os.Open(file)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
q := fmt.Sprintf("?job=%s&profile=%s&seq=%d&duration=%f&final=%t",
|
||||
job.ID, profile, seq, dur, final)
|
||||
req, _ := http.NewRequest(http.MethodPut, *masterURL+"/api/transcode/worker/segment"+q, f)
|
||||
req.Header.Set("Content-Type", "video/mp2t")
|
||||
req.Header.Set("Authorization", "Bearer "+job.CallbackToken)
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode >= 400 {
|
||||
raw, _ := io.ReadAll(resp.Body)
|
||||
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---- ffmpeg args（与 master 端 local_runner 保持一致）----
|
||||
|
||||
// protocolWhitelistFor 与 internal/transcode.ProtocolWhitelistFor 保持一致：
|
||||
// 远程输入只放行 HTTP(S) 及其传输层，本地路径只放行 file。
|
||||
func protocolWhitelistFor(source string) string {
|
||||
s := strings.ToLower(strings.TrimSpace(source))
|
||||
if strings.HasPrefix(s, "http://") || strings.HasPrefix(s, "https://") {
|
||||
// 与 master 的 protocolWhitelistRemote 保持一致：HTTP 源用不到 file/crypto，
|
||||
// 留着等于给被构造过的播放列表留一条读本地文件的通道。
|
||||
return "http,https,tcp,tls"
|
||||
}
|
||||
return "file"
|
||||
}
|
||||
|
||||
func buildFFmpegArgs(job Job, profile Profile, segPattern, playlist string) []string {
|
||||
hw := strings.ToLower(profile.HWAccel)
|
||||
args := []string{"-y", "-loglevel", "warning"}
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
args = append(args, "-hwaccel", "cuda")
|
||||
case "qsv":
|
||||
args = append(args, "-hwaccel", "qsv")
|
||||
case "vaapi":
|
||||
args = append(args, "-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", "/dev/dri/renderD128")
|
||||
case "amf":
|
||||
args = append(args, "-hwaccel", "d3d11va")
|
||||
case "videotoolbox":
|
||||
args = append(args, "-hwaccel", "videotoolbox")
|
||||
case "auto":
|
||||
args = append(args, "-hwaccel", "auto")
|
||||
}
|
||||
// 限制 ffmpeg 可用协议：Master 下发的 source_url 不应被解释成 concat:/subfile: 等特殊协议。
|
||||
// -protocol_whitelist 是输入选项，必须位于 -i 之前。
|
||||
args = append(args, "-protocol_whitelist", protocolWhitelistFor(job.SourceURL))
|
||||
args = append(args, "-i", job.SourceURL)
|
||||
args = append(args, "-c:v", pickEncoder(profile.VideoCodec, hw))
|
||||
if profile.VideoBitrate != "" {
|
||||
args = append(args, "-b:v", profile.VideoBitrate, "-maxrate", profile.VideoBitrate, "-bufsize", profile.VideoBitrate)
|
||||
}
|
||||
if profile.Scale != "" {
|
||||
switch hw {
|
||||
case "vaapi":
|
||||
args = append(args, "-vf", "scale_vaapi="+profile.Scale)
|
||||
case "qsv":
|
||||
args = append(args, "-vf", "scale_qsv="+profile.Scale)
|
||||
default:
|
||||
args = append(args, "-vf", "scale="+profile.Scale)
|
||||
}
|
||||
}
|
||||
if profile.AudioCodec == "copy" {
|
||||
args = append(args, "-c:a", "copy")
|
||||
} else {
|
||||
ac := profile.AudioCodec
|
||||
if ac == "" {
|
||||
ac = "aac"
|
||||
}
|
||||
args = append(args, "-c:a", ac)
|
||||
if profile.AudioBitrate != "" {
|
||||
args = append(args, "-b:a", profile.AudioBitrate)
|
||||
}
|
||||
}
|
||||
segDur := job.Output.SegmentDuration
|
||||
if segDur == 0 {
|
||||
segDur = 6
|
||||
}
|
||||
args = append(args,
|
||||
"-f", "hls",
|
||||
"-hls_time", strconv.Itoa(segDur),
|
||||
"-hls_playlist_type", "vod",
|
||||
"-hls_segment_filename", segPattern,
|
||||
"-hls_flags", "independent_segments",
|
||||
playlist,
|
||||
)
|
||||
return args
|
||||
}
|
||||
|
||||
func pickEncoder(codec, hw string) string {
|
||||
codec = strings.ToLower(codec)
|
||||
switch codec {
|
||||
case "h264", "":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "h264_nvenc"
|
||||
case "qsv":
|
||||
return "h264_qsv"
|
||||
case "vaapi":
|
||||
return "h264_vaapi"
|
||||
case "amf":
|
||||
return "h264_amf"
|
||||
case "videotoolbox":
|
||||
return "h264_videotoolbox"
|
||||
}
|
||||
return "libx264"
|
||||
case "hevc", "h265":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "hevc_nvenc"
|
||||
case "qsv":
|
||||
return "hevc_qsv"
|
||||
case "vaapi":
|
||||
return "hevc_vaapi"
|
||||
case "amf":
|
||||
return "hevc_amf"
|
||||
case "videotoolbox":
|
||||
return "hevc_videotoolbox"
|
||||
}
|
||||
return "libx265"
|
||||
case "av1":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "av1_nvenc"
|
||||
case "qsv":
|
||||
return "av1_qsv"
|
||||
}
|
||||
return "libsvtav1"
|
||||
}
|
||||
return "libx264"
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
@@ -65,6 +65,7 @@ require (
|
||||
github.com/pquerna/otp v1.5.0
|
||||
github.com/quic-go/quic-go v0.61.0
|
||||
github.com/rclone/rclone v1.75.1
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd
|
||||
github.com/shirou/gopsutil/v4 v4.26.8
|
||||
github.com/sirupsen/logrus v1.10.2
|
||||
github.com/spf13/afero v1.15.0
|
||||
|
||||
@@ -791,6 +791,8 @@ github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUc
|
||||
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
|
||||
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
|
||||
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 h1:GHRpF1pTW19a8tTFrMLUcfWwyC0pnifVo2ClaLq+hP8=
|
||||
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46/go.mod h1:uAQ5PCi+MFsC7HjREoAz1BU+Mq60+05gifQSsHSDG/8=
|
||||
github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4 h1:PT+ElG/UUFMfqy5HrxJxNzj3QBOf7dZwupeVC+mG1Lo=
|
||||
|
||||
@@ -40,6 +40,7 @@ func InitConfig() {
|
||||
if !filepath.IsAbs(dataDir) {
|
||||
flags.DataDir = filepath.Join(pwd, flags.DataDir)
|
||||
}
|
||||
conf.DataDir = flags.DataDir
|
||||
// Determine config file path: use flags.ConfigPath if provided, otherwise default to <dataDir>/config.json
|
||||
configPath := flags.ConfigPath
|
||||
if configPath == "" {
|
||||
|
||||
@@ -259,6 +259,114 @@ func InitialSettings() []model.SettingItem {
|
||||
{Key: conf.StreamMaxClientUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
|
||||
{Key: conf.StreamMaxServerDownloadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
|
||||
{Key: conf.StreamMaxServerUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
|
||||
|
||||
// media settings
|
||||
{Key: conf.MediaTMDBKey, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaTMDBAPIURL, Value: "api.themoviedb.org", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaDiscogsToken, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaDiscogsAPIURL, Value: "api.discogs.com", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaStoreThumbnail, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaThumbnailMode, Value: "runtime", Type: conf.TypeSelect, Options: "base64,runtime,local", Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `缩略图存储方式：base64 存入数据库；runtime 存入程序数据目录（data/imgs）；local 写入挂载目录下的 VFS 路径`},
|
||||
{Key: conf.MediaThumbnailPath, Value: "/imgs", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaScrapeConcurrency, Value: "5", Type: conf.TypeNumber, Group: model.MEDIA, Flag: model.PRIVATE},
|
||||
{Key: conf.MediaScrapeLanguage, Value: "zh-CN", Type: conf.TypeSelect, Options: "zh-CN,zh-TW,en-US,ja-JP,ko-KR,fr-FR,de-DE,es-ES,ru-RU,pt-BR", Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `首选刮削语言：影响 TMDB 的标题/简介/海报文字与 Google Books 的检索语言；豆瓣固定为中文`},
|
||||
{Key: conf.MediaBookSource, Value: "douban", Type: conf.TypeSelect, Options: "douban,google,openlibrary", Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `书籍刮削数据源：douban（豆瓣）、google（Google Books）、openlibrary（OpenLibrary）`},
|
||||
{Key: conf.MediaGoogleBooksKey, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `Google Books API Key，可留空（无 Key 时配额较低）；申请地址 https://developers.google.com/books/docs/v1/using#APIKey`},
|
||||
{Key: conf.MediaDownloadCover, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `开启后把 TMDB / Google Books / OpenLibrary 等返回的封面链接下载到下方缩略图存储路径，而不是直接保存图片链接`},
|
||||
{Key: conf.MediaEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PUBLIC,
|
||||
Help: `媒体库总开关。关闭时媒体库相关页面、API 与协议端点全部不可用（返回 404），与未安装媒体库时表现一致`},
|
||||
{Key: conf.MediaEmbyEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `开启后提供 Emby 兼容 API（/@emby），可供 Infuse、VidHub、MoviePilot 等客户端直接接入`},
|
||||
{Key: conf.MediaSubsonicEnabled, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `开启后提供 OpenSubsonic API（/@rest），可供 Symfonium、Feishin、Amperfy 等音乐客户端接入`},
|
||||
{Key: conf.MediaSubsonicAllowPwd, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `允许 Subsonic 客户端用 OpenList 登录密码（u+p 参数）认证。该方式会把主账号密码暴露给第三方客户端，默认关闭，推荐使用应用密钥（apiKey 或 t+s）`},
|
||||
{Key: conf.MediaSubsonicProxyStream, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `Subsonic 播放默认返回 302 直链；若客户端不跟随重定向，可开启此项改为服务端代理转发（占用服务器带宽）`},
|
||||
{Key: conf.MediaProxyCover, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `开启后由服务端代理 TMDB/豆瓣等外部封面图片，适合直连图片域名受限的网络；默认关闭，直接使用原始图片地址`},
|
||||
{Key: conf.MediaCoverGateway, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `封面代理直连失败时，回退到第三方图片网关 wsrv.nl 重试。开启后封面地址会发送给该第三方服务；默认关闭`},
|
||||
{Key: conf.MediaStrmOnScan, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `扫描完成后自动把媒体条目生成为 .strm 文件并写入下方目录`},
|
||||
{Key: conf.MediaStrmOverwrite, Value: "true", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `生成 STRM 时覆盖已存在的同名文件`},
|
||||
{Key: conf.MediaStrmSameDir, Value: "true", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `开启时 .strm 与媒体文件放在同一目录且同名（仅扩展名不同）；关闭时统一输出到下方指定目录`},
|
||||
{Key: conf.MediaStrmPath, Value: "/strm", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `STRM 输出目录（VFS 路径，需位于可写存储中），仅在关闭「与媒体文件同目录」时使用`},
|
||||
{Key: conf.MediaStrmTemplate, Value: "{site_url}/d{path}?sign={sign}", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE,
|
||||
Help: `STRM 内容模板，可用变量：{site_url} 站点地址、{path} 文件虚拟路径、{sign} 签名、{name} 文件名`},
|
||||
|
||||
// transcode settings (FFmpeg 云端/本地转码) - 默认全部关闭
|
||||
{Key: conf.TranscodeEnabled, Value: "false", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `开启后，超过阈值的媒体文件将通过 FFmpeg 转码后再播放；默认关闭`},
|
||||
{Key: conf.TranscodeRunMode, Value: "local", Type: conf.TypeSelect, Options: "local,remote,hybrid", Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `local=仅使用本机内置 worker；remote=只使用远程 Worker 节点；hybrid=本地优先，超载后派发到远程`},
|
||||
{Key: conf.TranscodeMinSizeGB, Value: "5", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `文件大于该 GB 数才走转码（小于则直链播放），0=任意大小都转码`},
|
||||
{Key: conf.TranscodeMinBitrateMbps, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `视频码率超过该 Mbps 才转码，0=不限制`},
|
||||
{Key: conf.TranscodeSourceCodecs, Value: "hevc,h265,av1,vvc,vp9", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `仅对这些源视频编码进行转码（逗号分隔）。常见高码率/兼容性差的编码：hevc,av1,vvc,vp9`},
|
||||
{Key: conf.TranscodeSourceExtensions, Value: "mkv,ts,m2ts,mov,avi,wmv,flv,rmvb,webm", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `仅对这些后缀进行转码（逗号分隔，不带点），mp4 默认不转码可直接播放`},
|
||||
{Key: conf.TranscodeOutputFormat, Value: "hls", Type: conf.TypeSelect, Options: "hls,dash,mp4", Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `输出封装格式，HLS 兼容性最好`},
|
||||
{Key: conf.TranscodeOutputCodec, Value: "h264", Type: conf.TypeSelect, Options: "h264,hevc,av1", Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `重新编码后的视频编码，推荐 h264（最广兼容、互联网分发免授权费）`},
|
||||
{Key: conf.TranscodeOutputBitrate, Value: "4000k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `输出视频码率，例如 4000k / 6M。建议 1080p:4000k、720p:2500k`},
|
||||
{Key: conf.TranscodeOutputAudioCodec, Value: "aac", Type: conf.TypeSelect, Options: "aac,mp3,opus,copy", Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `输出音频编码，aac 兼容性最好；copy=直接复制源音频流`},
|
||||
{Key: conf.TranscodeOutputAudioBitrate, Value: "160k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `输出音频码率`},
|
||||
{Key: conf.TranscodeOutputResolution, Value: "1920x1080", Type: conf.TypeSelect, Options: "source,3840x2160,2560x1440,1920x1080,1280x720,854x480", Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `输出分辨率上限，超过该分辨率会下采样；source=保持源分辨率`},
|
||||
{Key: conf.TranscodeSegmentDuration, Value: "6", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `HLS/DASH 切片时长(秒)，越小首帧越快但请求数变多，推荐 4-10`},
|
||||
{Key: conf.TranscodeHWAccel, Value: "none", Type: conf.TypeSelect,
|
||||
Options: "none,auto,nvenc,qsv,vaapi,amf,videotoolbox",
|
||||
Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `GPU 硬件加速：
|
||||
none = 纯 CPU(libx264)
|
||||
auto = 自动探测可用加速器
|
||||
nvenc = NVIDIA GPU(GeForce/Tesla/Quadro/RTX,需 NVIDIA 驱动)
|
||||
qsv = Intel 集显/独显 QuickSync(免费,功耗低)
|
||||
vaapi = Linux 通用 VA-API(支持 Intel/AMD)
|
||||
amf = AMD GPU(Windows AMF)
|
||||
videotoolbox = macOS 硬件加速`},
|
||||
{Key: conf.TranscodeFFmpegPath, Value: "ffmpeg", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `FFmpeg 可执行路径，留空则使用 PATH 中的 ffmpeg`},
|
||||
{Key: conf.TranscodeFFprobePath, Value: "ffprobe", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `FFprobe 可执行路径`},
|
||||
{Key: conf.TranscodeWorkerSecret, Value: "", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `远程 Worker 注册时使用的共享密钥，留空则禁用远程 Worker 注册`},
|
||||
{Key: conf.TranscodeCachePath, Value: "data/ctmp", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `转码切片缓存目录`},
|
||||
{Key: conf.TranscodeCacheMaxGB, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `切片缓存最大容量(GB)，超过后按 LRU 清理`},
|
||||
{Key: conf.TranscodeJobTimeoutMin, Value: "120", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `单个转码任务超时分钟数，超时自动失败`},
|
||||
{Key: conf.TranscodeLocalConcurrency, Value: "1", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `本机内置 worker 同时执行的转码任务数(local/hybrid 模式生效)`},
|
||||
{Key: conf.TranscodeIdleTimeoutSec, Value: "90", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `播放端无请求多少秒后自动停止转码（默认90秒，0=禁用）`},
|
||||
{Key: conf.TranscodeChunkDurationSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `每个 chunk 时长(秒)，越大每段缓冲越长但 ffmpeg 进程切换越少（默认60秒）`},
|
||||
{Key: conf.TranscodeMaxChunkConcurrency, Value: "2", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `单个视频任务最多同时运行的 chunk 数（默认2，越大并发越高但 CPU/GPU 占用越多）`},
|
||||
{Key: conf.TranscodeChunkIdleSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `chunk 多久没被访问就 kill ffmpeg 释放资源（默认60秒）`},
|
||||
{Key: conf.TranscodeChunkPrefetch, Value: "true", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE,
|
||||
Help: `当前 chunk 转码进度过半时，是否预启动下一个 chunk（默认开启，可让顺序播放无缝衔接）`},
|
||||
|
||||
// multipart upload settings (from main)
|
||||
{Key: conf.MultipartEnabled, Value: "true", Type: conf.TypeBool, Group: model.TRAFFIC, Flag: model.PUBLIC},
|
||||
{Key: conf.MultipartChunkSize, Value: "10", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: `chunk size of multipart upload in MB (positive integer), keep it under your CDN's request body limit; each active session buffers up to 8 chunks on the server's disk`},
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/transcode"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
"github.com/OpenListTeam/OpenList/v4/server"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/middlewares"
|
||||
@@ -92,6 +93,10 @@ func Start() {
|
||||
InitOfflineDownloadTools()
|
||||
LoadStorages()
|
||||
InitTaskManager()
|
||||
// 初始化转码模块（总开关关时也会启动 manager，但本地 worker 仅在 enabled 时由 Manager.Start 内部按 run_mode 启动）
|
||||
if transcode.IsEnabled() {
|
||||
transcode.Default().Start()
|
||||
}
|
||||
if !flags.Debug && !flags.Dev {
|
||||
gin.SetMode(gin.ReleaseMode)
|
||||
}
|
||||
|
||||
+61
-2
@@ -175,8 +175,67 @@ const (
|
||||
StreamMaxClientUploadSpeed = "max_client_upload_speed"
|
||||
StreamMaxServerDownloadSpeed = "max_server_download_speed"
|
||||
StreamMaxServerUploadSpeed = "max_server_upload_speed"
|
||||
MultipartEnabled = "multipart_enabled"
|
||||
MultipartChunkSize = "multipart_chunk_size"
|
||||
|
||||
// media
|
||||
MediaTMDBKey = "media_tmdb_key"
|
||||
MediaTMDBAPIURL = "media_tmdb_api_url"
|
||||
MediaDiscogsToken = "media_discogs_token"
|
||||
MediaDiscogsAPIURL = "media_discogs_api_url"
|
||||
MediaThumbnailMode = "media_thumbnail_mode"
|
||||
MediaThumbnailPath = "media_thumbnail_path"
|
||||
MediaStoreThumbnail = "media_store_thumbnail"
|
||||
MediaScrapeConcurrency = "media_scrape_concurrency"
|
||||
MediaScrapeLanguage = "media_scrape_language" // 首选刮削语言（TMDB / Google Books）
|
||||
MediaBookSource = "media_book_source" // 书籍刮削源：douban/google/openlibrary
|
||||
MediaGoogleBooksKey = "media_google_books_key" // Google Books API Key（可选）
|
||||
MediaDownloadCover = "media_download_cover" // 是否把远程封面下载到本地存储
|
||||
|
||||
// media library master switch & protocol switches
|
||||
MediaEnabled = "media_enabled" // 媒体库总开关（PUBLIC，前端据此显示/隐藏）
|
||||
MediaEmbyEnabled = "media_emby_enabled" // Emby 兼容 API 开关
|
||||
MediaSubsonicEnabled = "media_subsonic_enabled" // OpenSubsonic API 开关
|
||||
MediaSubsonicProxyStream = "media_subsonic_proxy_stream" // Subsonic 播放是否强制代理（兼容不跟随 302 的客户端）
|
||||
MediaSubsonicAllowPwd = "media_subsonic_allow_password_auth" // Subsonic 是否允许 u+p 使用 OpenList 登录密码认证（默认关闭）
|
||||
MediaProxyCover = "media_proxy_cover" // 是否通过服务端代理外部封面（TMDB/豆瓣等）
|
||||
MediaCoverGateway = "media_cover_gateway" // 封面代理直连失败时是否回退第三方图片网关（wsrv.nl，默认关闭）
|
||||
MediaStrmOnScan = "media_strm_on_scan" // 扫描完成后自动生成 STRM
|
||||
MediaStrmSameDir = "media_strm_same_dir" // STRM 是否与媒体文件同目录（同名 .strm）
|
||||
MediaStrmPath = "media_strm_path" // STRM 输出目录（VFS 路径，仅在指定目录模式下使用）
|
||||
MediaStrmOverwrite = "media_strm_overwrite" // 生成 STRM 时是否覆盖已存在文件
|
||||
MediaStrmTemplate = "media_strm_template" // STRM 内容模板
|
||||
|
||||
// transcode (云端/本地 FFmpeg 转码)
|
||||
TranscodeEnabled = "transcode_enabled" // 总开关，默认关
|
||||
TranscodeMinSizeGB = "transcode_min_size_gb" // 文件大于多少 GB 才转码
|
||||
TranscodeMinBitrateMbps = "transcode_min_bitrate_mbps" // 视频码率高于多少 Mbps 才转码（0=不限）
|
||||
TranscodeSourceCodecs = "transcode_source_codecs" // 仅对这些源编码进行转码
|
||||
TranscodeSourceExtensions = "transcode_source_extensions" // 仅对这些后缀进行转码
|
||||
TranscodeOutputFormat = "transcode_output_format" // 输出封装格式 hls/dash/mp4
|
||||
TranscodeOutputCodec = "transcode_output_codec" // 重新编码的视频编码
|
||||
TranscodeOutputBitrate = "transcode_output_bitrate" // 输出视频码率，例如 4000k
|
||||
TranscodeOutputAudioCodec = "transcode_output_audio_codec" // 输出音频编码
|
||||
TranscodeOutputAudioBitrate = "transcode_output_audio_bitrate" // 输出音频码率，例如 160k
|
||||
TranscodeOutputResolution = "transcode_output_resolution" // 输出分辨率上限
|
||||
TranscodeSegmentDuration = "transcode_segment_duration" // HLS 切片时长(秒)
|
||||
TranscodeHWAccel = "transcode_hwaccel" // 硬件加速类型 none/auto/nvenc/qsv/vaapi/amf/videotoolbox
|
||||
TranscodeFFmpegPath = "transcode_ffmpeg_path" // 自定义 ffmpeg 可执行路径
|
||||
TranscodeFFprobePath = "transcode_ffprobe_path" // 自定义 ffprobe 可执行路径
|
||||
TranscodeRunMode = "transcode_run_mode" // local / remote / hybrid
|
||||
TranscodeWorkerSecret = "transcode_worker_secret" // 远程 Worker 共享密钥
|
||||
TranscodeCachePath = "transcode_cache_path" // 切片缓存目录
|
||||
TranscodeCacheMaxGB = "transcode_cache_max_gb" // 缓存最大占用，超出 LRU 淘汰
|
||||
TranscodeJobTimeoutMin = "transcode_job_timeout_min" // 单个任务超时分钟
|
||||
TranscodeLocalConcurrency = "transcode_local_concurrency" // 本地内置 worker 并发数(run_mode=local/hybrid 时生效)
|
||||
TranscodeIdleTimeoutSec = "transcode_idle_timeout_sec" // 播放端无请求多少秒后自动停止转码（默认90秒）
|
||||
// === 智能 chunk 调度相关 ===
|
||||
TranscodeChunkDurationSec = "transcode_chunk_duration_sec" // 每个 chunk 时长(秒)，将整个视频分成多个 chunk 按需转码（默认60秒）
|
||||
TranscodeMaxChunkConcurrency = "transcode_max_chunk_concurrency" // 单个 Job 最多同时运行的 chunk 数（默认2）
|
||||
TranscodeChunkIdleSec = "transcode_chunk_idle_sec" // chunk 多久没被访问就 kill ffmpeg 释放资源（默认60秒）
|
||||
TranscodeChunkPrefetch = "transcode_chunk_prefetch" // 当前 chunk 转码进度过半时，是否预启动下一个 chunk（默认true）
|
||||
|
||||
// multipart upload (from main)
|
||||
MultipartEnabled = "multipart_enabled"
|
||||
MultipartChunkSize = "multipart_chunk_size"
|
||||
)
|
||||
|
||||
const (
|
||||
|
||||
@@ -18,6 +18,8 @@ var (
|
||||
Conf *Config
|
||||
URL *url.URL
|
||||
ConfigPath string
|
||||
// DataDir 程序数据目录（绝对路径），用于运行时文件（如本地缩略图）
|
||||
DataDir string = "data"
|
||||
)
|
||||
|
||||
var SlicesMap = make(map[string][]string)
|
||||
|
||||
+95
-3
@@ -1,6 +1,9 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
log "github.com/sirupsen/logrus"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
@@ -10,12 +13,101 @@ import (
|
||||
|
||||
var db *gorm.DB
|
||||
|
||||
// MediaMigrateWipeEnv 设为 1/true 时，允许升级流程清空旧版 media_items 数据后重建。
|
||||
// 旧版本的 folder_path 语义不同，无法原地迁移；但清库是破坏性操作，必须由管理员显式确认。
|
||||
const MediaMigrateWipeEnv = "OPENLIST_MEDIA_MIGRATE_WIPE"
|
||||
|
||||
func Init(d *gorm.DB) {
|
||||
db = d
|
||||
err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB), new(model.WebDAVProperty))
|
||||
if err != nil {
|
||||
log.Fatalf("failed migrate database: %s", err.Error())
|
||||
if err := migrateMediaItems(); err != nil {
|
||||
log.Fatalf("failed migrate media_items: %s (如果旧数据无法原地迁移，请先备份数据库，再设置 %s=1 重建媒体表)", err.Error(), MediaMigrateWipeEnv)
|
||||
}
|
||||
err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB), new(model.WebDAVProperty), new(model.MediaItem), new(model.MediaConfig), new(model.MediaScanPath), new(model.MediaAppKey), new(model.MediaUserData), new(model.MediaPlaylist), new(model.MediaPlaylistItem), new(model.EmbyDevice), new(model.MediaAnnotation))
|
||||
if err != nil {
|
||||
// 典型场景：未确认清库时，旧 media_items 数据违反新建的唯一索引。
|
||||
// 这里把解决办法直接写进日志，避免用户只看到一句 failed migrate database。
|
||||
log.Fatalf(
|
||||
"failed migrate database: %s (若由媒体表唯一索引与旧数据冲突导致，请先备份数据库，再设置 %s=1 重建媒体表)",
|
||||
err.Error(),
|
||||
MediaMigrateWipeEnv,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func mediaWipeConfirmed() bool {
|
||||
switch os.Getenv(MediaMigrateWipeEnv) {
|
||||
case "1", "true", "TRUE", "True", "yes":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// mediaItemsTable 返回 MediaItem 的真实表名。
|
||||
// 表名带有 conf.Database.TablePrefix（可配置，默认 x_），不能硬编码。
|
||||
func mediaItemsTable() (string, error) {
|
||||
stmt := &gorm.Statement{DB: db}
|
||||
if err := stmt.Parse(&model.MediaItem{}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return stmt.Schema.Table, nil
|
||||
}
|
||||
|
||||
// migrateMediaItems 处理 media_items 表的迁移兼容性
|
||||
// 存储语义历史变更：
|
||||
//
|
||||
// v1: folder_path 为文件完整路径
|
||||
// v2: folder_path 恒定为扫描根路径
|
||||
// v3（当前）: folder_path 为文件/文件夹的实际父目录，供前端拼接播放地址
|
||||
//
|
||||
// 唯一性由 folder_path + file_name + album_name 组合索引保证。
|
||||
//
|
||||
// 检测到 v1/v2 的旧表（缺少新组合索引）时：
|
||||
// - 默认不动数据，只输出醒目的告警，并让 AutoMigrate 在旧表上补建新索引；
|
||||
// - 管理员设置环境变量 OPENLIST_MEDIA_MIGRATE_WIPE=1 后才会清空旧数据并重扫。
|
||||
func migrateMediaItems() error {
|
||||
table, err := mediaItemsTable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m := db.Migrator()
|
||||
if !m.HasTable(table) {
|
||||
return nil
|
||||
}
|
||||
const idx = "idx_media_folder_file_album"
|
||||
if m.HasIndex(table, idx) {
|
||||
// 已是新索引：仅清理软删除残留数据，避免与唯一索引冲突
|
||||
if err := db.Exec("DELETE FROM "+quoteIdent(table)+" WHERE deleted_at IS NOT NULL").Error; err != nil {
|
||||
log.Warnf("media_items: 清理软删除残留数据失败: %v", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// 旧表存在但没有新组合索引：旧数据的 folder_path 语义已变更，无法复用。
|
||||
if !mediaWipeConfirmed() {
|
||||
log.Warnf("media_items: 检测到旧版本数据（表 %s 缺少索引 %s）。", table, idx)
|
||||
log.Warnf("media_items: 旧数据的路径语义已变更，建议清空后重新扫描；为避免误删，本次启动不会自动清库。")
|
||||
log.Warnf("media_items: 如确认清空，请设置环境变量 %s=1 后重启；也可在管理后台对各媒体库点击「清空数据库」再重扫。", MediaMigrateWipeEnv)
|
||||
return nil
|
||||
}
|
||||
log.Warnf("media_items: 已确认 %s，清空旧版本数据（表 %s）后重新迁移", MediaMigrateWipeEnv, table)
|
||||
// 先尝试删除旧的单字段唯一索引（如果存在），避免 AutoMigrate 冲突
|
||||
oldIdx := "idx_" + table + "_folder_path"
|
||||
if m.HasIndex(table, oldIdx) {
|
||||
if err := m.DropIndex(table, oldIdx); err != nil {
|
||||
log.Warnf("media_items: 删除旧唯一索引失败: %v", err)
|
||||
}
|
||||
}
|
||||
if err := db.Exec("DELETE FROM " + quoteIdent(table)).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// quoteIdent 按当前数据库方言给标识符加引号，表名来自 gorm schema 解析而非用户输入。
|
||||
func quoteIdent(name string) string {
|
||||
var sb strings.Builder
|
||||
db.Dialector.QuoteTo(&sb, name)
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
func AutoMigrate(dst ...interface{}) error {
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/glebarez/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/schema"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
func openTestDB(t *testing.T, prefix string) *gorm.DB {
|
||||
t.Helper()
|
||||
d, err := gorm.Open(sqlite.Open("file::memory:"), &gorm.Config{
|
||||
NamingStrategy: schema.NamingStrategy{TablePrefix: prefix},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// 旧版表：只有 media_items 的基础结构，没有新的组合唯一索引。
|
||||
func createLegacyMediaTable(t *testing.T, d *gorm.DB, table string) {
|
||||
t.Helper()
|
||||
if err := d.Exec("CREATE TABLE " + table + " (id integer primary key autoincrement, deleted_at datetime, folder_path text, file_name text)").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := d.Exec("INSERT INTO " + table + " (folder_path, file_name) VALUES ('/a/b.mp3', 'b.mp3')").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func countRows(t *testing.T, d *gorm.DB, table string) int64 {
|
||||
t.Helper()
|
||||
var n int64
|
||||
if err := d.Table(table).Count(&n).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestMediaItemsTableHonorsPrefix(t *testing.T) {
|
||||
for _, prefix := range []string{"x_", "ol_", ""} {
|
||||
db = openTestDB(t, prefix)
|
||||
got, err := mediaItemsTable()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if want := prefix + "media_items"; got != want {
|
||||
t.Errorf("prefix %q: table = %q, want %q", prefix, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateKeepsLegacyDataWithoutConfirmation(t *testing.T) {
|
||||
t.Setenv(MediaMigrateWipeEnv, "")
|
||||
db = openTestDB(t, "ol_") // 非默认前缀：旧实现硬编码 x_ 会直接跳过迁移
|
||||
createLegacyMediaTable(t, db, "ol_media_items")
|
||||
|
||||
if err := migrateMediaItems(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := countRows(t, db, "ol_media_items"); n != 1 {
|
||||
t.Fatalf("legacy rows must be preserved without confirmation, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateWipesOnlyWhenConfirmed(t *testing.T) {
|
||||
t.Setenv(MediaMigrateWipeEnv, "1")
|
||||
db = openTestDB(t, "ol_")
|
||||
createLegacyMediaTable(t, db, "ol_media_items")
|
||||
|
||||
if err := migrateMediaItems(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := countRows(t, db, "ol_media_items"); n != 0 {
|
||||
t.Fatalf("legacy rows should be wiped after confirmation, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAutoMigrateCreatesCompositeUniqueIndex(t *testing.T) {
|
||||
db = openTestDB(t, "ol_")
|
||||
if err := db.AutoMigrate(&model.MediaItem{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !db.Migrator().HasIndex("ol_media_items", "idx_media_folder_file_album") {
|
||||
t.Fatal("composite unique index missing after AutoMigrate")
|
||||
}
|
||||
// 已有新索引时迁移不应触碰有效数据
|
||||
if err := db.Create(&model.MediaItem{FolderPath: "/m", FileName: "a.mp3"}).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := migrateMediaItems(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := countRows(t, db, "ol_media_items"); n != 1 {
|
||||
t.Fatalf("valid rows must survive migration, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateOrUpdateRejectsOverlongIndexColumns(t *testing.T) {
|
||||
db = openTestDB(t, "ol_")
|
||||
if err := db.AutoMigrate(&model.MediaItem{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
long := make([]rune, model.MediaIndexColumnMax+1)
|
||||
for i := range long {
|
||||
long[i] = '文'
|
||||
}
|
||||
err := CreateOrUpdateMediaItem(&model.MediaItem{FolderPath: "/m", FileName: string(long)})
|
||||
if err == nil {
|
||||
t.Fatal("overlong file_name must be rejected")
|
||||
}
|
||||
ok := make([]rune, model.MediaIndexColumnMax)
|
||||
for i := range ok {
|
||||
ok[i] = '文'
|
||||
}
|
||||
if err := CreateOrUpdateMediaItem(&model.MediaItem{FolderPath: "/m", FileName: string(ok)}); err != nil {
|
||||
t.Fatalf("boundary length must be accepted: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,626 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// ==================== MediaConfig ====================
|
||||
|
||||
// GetMediaConfig 获取指定类型的媒体库配置，不存在则返回默认值
|
||||
func GetMediaConfig(mediaType model.MediaType) (*model.MediaConfig, error) {
|
||||
var cfg model.MediaConfig
|
||||
result := db.Where("media_type = ?", mediaType).First(&cfg)
|
||||
if result.Error == gorm.ErrRecordNotFound {
|
||||
return &model.MediaConfig{
|
||||
MediaType: mediaType,
|
||||
Enabled: false,
|
||||
}, nil
|
||||
}
|
||||
return &cfg, result.Error
|
||||
}
|
||||
|
||||
// GetAllMediaConfigs 获取所有媒体库配置
|
||||
func GetAllMediaConfigs() ([]model.MediaConfig, error) {
|
||||
var cfgs []model.MediaConfig
|
||||
err := db.Find(&cfgs).Error
|
||||
return cfgs, err
|
||||
}
|
||||
|
||||
// SaveMediaConfig 保存媒体库配置（upsert）
|
||||
func SaveMediaConfig(cfg *model.MediaConfig) error {
|
||||
var existing model.MediaConfig
|
||||
result := db.Where("media_type = ?", cfg.MediaType).First(&existing)
|
||||
if result.Error == gorm.ErrRecordNotFound {
|
||||
return db.Create(cfg).Error
|
||||
}
|
||||
cfg.ID = existing.ID
|
||||
return db.Save(cfg).Error
|
||||
}
|
||||
|
||||
// ==================== MediaScanPath ====================
|
||||
|
||||
// ListMediaScanPaths 获取指定媒体类型的所有扫描路径
|
||||
func ListMediaScanPaths(mediaType model.MediaType) ([]model.MediaScanPath, error) {
|
||||
var paths []model.MediaScanPath
|
||||
tx := db.Model(&model.MediaScanPath{})
|
||||
if mediaType != "" {
|
||||
tx = tx.Where("media_type = ?", mediaType)
|
||||
}
|
||||
err := tx.Order("id asc").Find(&paths).Error
|
||||
return paths, err
|
||||
}
|
||||
|
||||
// GetMediaScanPath 按ID获取扫描路径
|
||||
func GetMediaScanPath(id uint) (*model.MediaScanPath, error) {
|
||||
var p model.MediaScanPath
|
||||
err := db.First(&p, id).Error
|
||||
return &p, err
|
||||
}
|
||||
|
||||
// CreateMediaScanPath 创建扫描路径
|
||||
func CreateMediaScanPath(p *model.MediaScanPath) error {
|
||||
return db.Create(p).Error
|
||||
}
|
||||
|
||||
// UpdateMediaScanPath 更新扫描路径
|
||||
func UpdateMediaScanPath(p *model.MediaScanPath) error {
|
||||
return db.Save(p).Error
|
||||
}
|
||||
|
||||
// DeleteMediaScanPath 删除扫描路径（硬删除）
|
||||
func DeleteMediaScanPath(id uint) error {
|
||||
return db.Unscoped().Delete(&model.MediaScanPath{}, id).Error
|
||||
}
|
||||
|
||||
// ==================== MediaItem ====================
|
||||
|
||||
// escapeLike 转义 LIKE 通配符，避免路径中的 % _ 影响前缀匹配
|
||||
func escapeLike(s string) string {
|
||||
s = strings.ReplaceAll(s, `\`, `\\`)
|
||||
s = strings.ReplaceAll(s, "%", `\%`)
|
||||
s = strings.ReplaceAll(s, "_", `\_`)
|
||||
return s
|
||||
}
|
||||
|
||||
// MediaItemQuery 媒体条目查询参数
|
||||
type MediaItemQuery struct {
|
||||
MediaType model.MediaType
|
||||
ScanPathID uint // 按扫描路径ID筛选
|
||||
FolderPath string // 按文件夹路径筛选
|
||||
TypeTag string // 按类型标签筛选（电影、电视剧等）
|
||||
ContentTag string // 按内容标签筛选（喜剧、惊悚等）
|
||||
PathPrefix string // 按文件夹路径前缀筛选（用于用户 base_path 隔离）
|
||||
Hidden *bool
|
||||
Keyword string
|
||||
OrderBy string // "name", "date", "size"
|
||||
OrderDir string // "asc", "desc"
|
||||
Page int
|
||||
PageSize int
|
||||
}
|
||||
|
||||
// ListMediaItems 分页查询媒体条目
|
||||
func ListMediaItems(q MediaItemQuery) ([]model.MediaItem, int64, error) {
|
||||
tx := db.Model(&model.MediaItem{})
|
||||
if q.MediaType != "" {
|
||||
tx = tx.Where("media_type = ?", q.MediaType)
|
||||
}
|
||||
if q.ScanPathID > 0 {
|
||||
tx = tx.Where("scan_path_id = ?", q.ScanPathID)
|
||||
}
|
||||
if q.FolderPath != "" {
|
||||
tx = tx.Where("folder_path = ?", q.FolderPath)
|
||||
}
|
||||
if q.PathPrefix != "" {
|
||||
prefix := strings.TrimSuffix(q.PathPrefix, "/") + "/"
|
||||
tx = tx.Where(`folder_path LIKE ? ESCAPE '\'`, escapeLike(prefix)+"%")
|
||||
}
|
||||
if q.Hidden != nil {
|
||||
tx = tx.Where("hidden = ?", *q.Hidden)
|
||||
}
|
||||
if q.Keyword != "" {
|
||||
like := "%" + q.Keyword + "%"
|
||||
tx = tx.Where("file_name LIKE ? OR scraped_name LIKE ?", like, like)
|
||||
}
|
||||
// 按类型标签筛选（通过关联扫描路径的type_tag）
|
||||
if q.TypeTag != "" {
|
||||
tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id").
|
||||
Where("media_scan_paths.type_tag = ?", q.TypeTag)
|
||||
}
|
||||
// 按内容标签筛选（通过关联扫描路径的content_tags）
|
||||
if q.ContentTag != "" {
|
||||
tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id").
|
||||
Where("media_scan_paths.content_tags LIKE ?", "%"+q.ContentTag+"%")
|
||||
}
|
||||
|
||||
var total int64
|
||||
if err := tx.Count(&total).Error; err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
// 排序
|
||||
orderCol := "created_at"
|
||||
switch q.OrderBy {
|
||||
case "name":
|
||||
orderCol = "COALESCE(NULLIF(scraped_name,''), file_name)"
|
||||
case "date":
|
||||
orderCol = "release_date"
|
||||
case "size":
|
||||
orderCol = "file_size"
|
||||
}
|
||||
dir := "asc"
|
||||
if q.OrderDir == "desc" {
|
||||
dir = "desc"
|
||||
}
|
||||
tx = tx.Order(orderCol + " " + dir)
|
||||
|
||||
// 分页
|
||||
if q.PageSize <= 0 {
|
||||
q.PageSize = 20
|
||||
}
|
||||
if q.Page <= 0 {
|
||||
q.Page = 1
|
||||
}
|
||||
tx = tx.Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize)
|
||||
|
||||
var items []model.MediaItem
|
||||
err := tx.Find(&items).Error
|
||||
return items, total, err
|
||||
}
|
||||
|
||||
// GetMediaItemByID 按ID获取媒体条目
|
||||
func GetMediaItemByID(id uint) (*model.MediaItem, error) {
|
||||
var item model.MediaItem
|
||||
err := db.First(&item, id).Error
|
||||
return &item, err
|
||||
}
|
||||
|
||||
// GetMediaItemByFolderPath 按文件夹路径获取媒体条目（用于合并模式）
|
||||
func GetMediaItemByFolderPath(folderPath string) (*model.MediaItem, error) {
|
||||
var item model.MediaItem
|
||||
result := db.Where("folder_path = ?", folderPath).First(&item)
|
||||
return &item, result.Error
|
||||
}
|
||||
|
||||
// CreateOrUpdateMediaItem 创建或更新媒体条目（按 folder_path + file_name + album_name 组合唯一）
|
||||
// 更新时保留已有的刮削数据，避免重新扫描时把已刮削的字段清空
|
||||
//
|
||||
// 注意：使用 Unscoped() 查询是为了把软删除残留记录也包含进来，
|
||||
// 否则唯一索引 idx_media_folder_file_album 会与软删除行冲突，
|
||||
// 导致新建条目时报 UNIQUE constraint failed。
|
||||
func CreateOrUpdateMediaItem(item *model.MediaItem) error {
|
||||
// 复合唯一索引三列在 MySQL 下限定为 191 字符（见 model.MediaItem）。
|
||||
// 超长值在严格模式下会写入失败、非严格模式下会被静默截断并造成不同文件撞同一条目，
|
||||
// 因此显式拒绝，由调用方记录告警并跳过。
|
||||
if utf8.RuneCountInString(item.FolderPath) > model.MediaIndexColumnMax ||
|
||||
utf8.RuneCountInString(item.FileName) > model.MediaIndexColumnMax ||
|
||||
utf8.RuneCountInString(item.AlbumName) > model.MediaIndexColumnMax {
|
||||
return fmt.Errorf("media item path/name exceeds %d characters, skipped", model.MediaIndexColumnMax)
|
||||
}
|
||||
var existing model.MediaItem
|
||||
result := db.Unscoped().Where("folder_path = ? AND file_name = ? AND album_name = ?", item.FolderPath, item.FileName, item.AlbumName).First(&existing)
|
||||
if result.Error == gorm.ErrRecordNotFound {
|
||||
return db.Create(item).Error
|
||||
}
|
||||
if result.Error != nil {
|
||||
return result.Error
|
||||
}
|
||||
item.ID = existing.ID
|
||||
item.CreatedAt = existing.CreatedAt
|
||||
// 复用已有记录时清除软删除标记，确保该条目“恢复”为正常记录
|
||||
item.DeletedAt = gorm.DeletedAt{}
|
||||
// 如果已有刮削数据，保留刮削字段，防止重新扫描时覆盖刮削结果
|
||||
if existing.ScrapedAt != nil {
|
||||
item.ScrapedAt = existing.ScrapedAt
|
||||
item.ScrapedName = existing.ScrapedName
|
||||
item.Cover = existing.Cover
|
||||
item.AlbumName = existing.AlbumName
|
||||
item.AlbumArtist = existing.AlbumArtist
|
||||
item.TrackNumber = existing.TrackNumber
|
||||
item.Duration = existing.Duration
|
||||
item.Genre = existing.Genre
|
||||
item.ReleaseDate = existing.ReleaseDate
|
||||
item.Rating = existing.Rating
|
||||
item.Plot = existing.Plot
|
||||
item.Authors = existing.Authors
|
||||
item.Description = existing.Description
|
||||
item.Publisher = existing.Publisher
|
||||
item.ISBN = existing.ISBN
|
||||
item.ExternalID = existing.ExternalID
|
||||
}
|
||||
// 扫描不覆盖用户手动隐藏的状态
|
||||
item.Hidden = existing.Hidden
|
||||
return db.Unscoped().Save(item).Error
|
||||
}
|
||||
|
||||
// UpdateMediaItem 更新媒体条目
|
||||
// 使用 Unscoped 确保即使存在软删除标记也能正确更新（避免命中唯一索引但行不可见的诡异情况）。
|
||||
func UpdateMediaItem(item *model.MediaItem) error {
|
||||
return db.Unscoped().Save(item).Error
|
||||
}
|
||||
|
||||
// DeleteMediaItem 硬删除媒体条目（真正从数据库删除），并清理该条目的批注
|
||||
func DeleteMediaItem(id uint) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Unscoped().Delete(&model.MediaItem{}, id).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Where("item_id = ?", id).Delete(&model.MediaAnnotation{}).Error
|
||||
})
|
||||
}
|
||||
|
||||
// ClearMediaItems 硬删除指定类型的所有媒体条目（真正从数据库删除），并清理对应批注
|
||||
func ClearMediaItems(mediaType model.MediaType) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
ids := []uint{}
|
||||
if err := tx.Unscoped().Model(&model.MediaItem{}).
|
||||
Where("media_type = ?", mediaType).Pluck("id", &ids).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Unscoped().Where("media_type = ?", mediaType).
|
||||
Delete(&model.MediaItem{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error
|
||||
})
|
||||
}
|
||||
|
||||
// ClearMediaItemsByScanPath 硬删除指定扫描路径的所有媒体条目，并清理对应批注
|
||||
func ClearMediaItemsByScanPath(scanPathID uint) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
ids := []uint{}
|
||||
if err := tx.Unscoped().Model(&model.MediaItem{}).
|
||||
Where("scan_path_id = ?", scanPathID).Pluck("id", &ids).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Unscoped().Where("scan_path_id = ?", scanPathID).
|
||||
Delete(&model.MediaItem{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error
|
||||
})
|
||||
}
|
||||
|
||||
// ClearMediaScrapedData 清空指定类型的所有刮削数据（保留扫描出的文件记录本身）
|
||||
// 仅清空刮削结果相关字段，不删除条目，便于重新刮削。
|
||||
// mediaType 为空时表示对所有类型生效。
|
||||
func ClearMediaScrapedData(mediaType model.MediaType) (int64, error) {
|
||||
tx := db.Unscoped().Model(&model.MediaItem{})
|
||||
if mediaType != "" {
|
||||
tx = tx.Where("media_type = ?", mediaType)
|
||||
}
|
||||
updates := map[string]interface{}{
|
||||
"scraped_name": "",
|
||||
"description": "",
|
||||
"cover": "",
|
||||
"release_date": "",
|
||||
"rating": 0,
|
||||
"genre": "",
|
||||
"authors": "",
|
||||
"plot": "",
|
||||
"reviews": "",
|
||||
"external_id": "",
|
||||
"album_artist": "",
|
||||
"publisher": "",
|
||||
"isbn": "",
|
||||
"lyrics": "",
|
||||
"scraped_at": nil,
|
||||
}
|
||||
result := tx.Updates(updates)
|
||||
return result.RowsAffected, result.Error
|
||||
}
|
||||
|
||||
// ListAllValidMediaItemsForCheck 列出指定类型下所有未删除的媒体条目（仅取必要字段，用于失效检查）
|
||||
// mediaType 为空时返回所有类型条目。
|
||||
func ListAllValidMediaItemsForCheck(mediaType model.MediaType) ([]model.MediaItem, error) {
|
||||
var items []model.MediaItem
|
||||
tx := db.Model(&model.MediaItem{}).
|
||||
Select("id, media_type, scan_path_id, file_name, folder_path, is_folder")
|
||||
if mediaType != "" {
|
||||
tx = tx.Where("media_type = ?", mediaType)
|
||||
}
|
||||
err := tx.Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
// DeleteMediaItemsByIDs 按ID列表硬删除媒体条目，并清理对应批注
|
||||
func DeleteMediaItemsByIDs(ids []uint) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Unscoped().Where("id IN ?", ids).Delete(&model.MediaItem{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Where("item_id IN ?", ids).Delete(&model.MediaAnnotation{}).Error
|
||||
})
|
||||
}
|
||||
|
||||
// ListAlbums 列出所有专辑（音乐专用）
|
||||
func ListAlbums(q MediaItemQuery) ([]AlbumInfo, int64, error) {
|
||||
type albumRow struct {
|
||||
AlbumName string
|
||||
AlbumArtist string
|
||||
Cover string
|
||||
ReleaseDate string
|
||||
TrackCount int
|
||||
ScanPathID uint
|
||||
}
|
||||
|
||||
// 构建基础查询
|
||||
baseQuery := db.Model(&model.MediaItem{}).
|
||||
Where("media_type = ?", model.MediaTypeMusic)
|
||||
if q.Hidden != nil {
|
||||
baseQuery = baseQuery.Where("hidden = ?", *q.Hidden)
|
||||
}
|
||||
if q.ScanPathID > 0 {
|
||||
baseQuery = baseQuery.Where("scan_path_id = ?", q.ScanPathID)
|
||||
}
|
||||
if q.Keyword != "" {
|
||||
like := "%" + q.Keyword + "%"
|
||||
baseQuery = baseQuery.Where("album_name LIKE ? OR album_artist LIKE ? OR scraped_name LIKE ?", like, like, like)
|
||||
}
|
||||
|
||||
// 统计分组数（用子查询）
|
||||
var total int64
|
||||
if err := db.Table("(?) as sub", baseQuery.
|
||||
Select("album_name, album_artist").
|
||||
Group("album_name, album_artist")).
|
||||
Count(&total).Error; err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
if q.PageSize <= 0 {
|
||||
q.PageSize = 20
|
||||
}
|
||||
if q.Page <= 0 {
|
||||
q.Page = 1
|
||||
}
|
||||
|
||||
tx := baseQuery.
|
||||
Select("album_name, album_artist, MAX(cover) as cover, MAX(release_date) as release_date, COUNT(*) as track_count, MAX(scan_path_id) as scan_path_id").
|
||||
Group("album_name, album_artist").
|
||||
Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize)
|
||||
|
||||
var rows []albumRow
|
||||
if err := tx.Scan(&rows).Error; err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
albums := make([]AlbumInfo, len(rows))
|
||||
for i, r := range rows {
|
||||
albums[i] = AlbumInfo{
|
||||
AlbumName: r.AlbumName,
|
||||
AlbumArtist: r.AlbumArtist,
|
||||
Cover: r.Cover,
|
||||
ReleaseDate: r.ReleaseDate,
|
||||
TrackCount: r.TrackCount,
|
||||
ScanPathID: r.ScanPathID,
|
||||
}
|
||||
}
|
||||
return albums, total, nil
|
||||
}
|
||||
|
||||
// AlbumInfo 专辑信息
|
||||
type AlbumInfo struct {
|
||||
AlbumName string `json:"album_name"`
|
||||
AlbumArtist string `json:"album_artist"`
|
||||
Cover string `json:"cover"`
|
||||
ReleaseDate string `json:"release_date"`
|
||||
TrackCount int `json:"track_count"`
|
||||
ScanPathID uint `json:"scan_path_id"`
|
||||
}
|
||||
|
||||
// GetAlbumTracks 获取专辑曲目列表
|
||||
// 支持两种模式：
|
||||
// 1. 普通模式（is_folder=false）：直接返回独立文件记录
|
||||
// 2. 合并文件夹模式（is_folder=true）：把 episodes 展开成虚拟 MediaItem 列表
|
||||
// 展开后每条记录的 folder_path = 原folder_path/file_name（文件夹实际路径），file_name = episode.FileName
|
||||
func GetAlbumTracks(albumName, albumArtist string) ([]model.MediaItem, error) {
|
||||
var items []model.MediaItem
|
||||
tx := db.Where("media_type = ?", model.MediaTypeMusic)
|
||||
if albumName != "" {
|
||||
tx = tx.Where("album_name = ?", albumName)
|
||||
} else {
|
||||
tx = tx.Where("(album_name = '' OR album_name IS NULL)")
|
||||
}
|
||||
if albumArtist != "" {
|
||||
tx = tx.Where("album_artist = ?", albumArtist)
|
||||
}
|
||||
err := tx.Order("track_number asc").Find(&items).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 展开合并文件夹条目的 episodes
|
||||
var result []model.MediaItem
|
||||
for _, item := range items {
|
||||
if !item.IsFolder || item.Episodes == "" {
|
||||
result = append(result, item)
|
||||
continue
|
||||
}
|
||||
// 解析 episodes
|
||||
type EpisodeInfo struct {
|
||||
FileName string `json:"file_name"`
|
||||
Index int `json:"index"`
|
||||
Title string `json:"title"`
|
||||
}
|
||||
var eps []EpisodeInfo
|
||||
if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil || len(eps) == 0 {
|
||||
// 解析失败则跳过该条目（不返回文件夹本身，避免播放路径错误）
|
||||
continue
|
||||
}
|
||||
// 文件夹实际路径 = folder_path + "/" + file_name
|
||||
actualDir := strings.TrimRight(item.FolderPath, "/") + "/" + item.FileName
|
||||
for _, ep := range eps {
|
||||
track := item // 复制基础信息（封面、专辑名、艺术家等）
|
||||
track.ID = 0
|
||||
track.IsFolder = false
|
||||
track.FolderPath = actualDir
|
||||
track.FileName = ep.FileName
|
||||
track.TrackNumber = ep.Index
|
||||
track.ScrapedName = ep.Title
|
||||
track.Episodes = ""
|
||||
result = append(result, track)
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// ListFolderPaths 列出指定媒体类型下的所有文件夹路径（目录浏览模式）
|
||||
func ListFolderPaths(mediaType model.MediaType) ([]string, error) {
|
||||
var paths []string
|
||||
err := db.Model(&model.MediaItem{}).
|
||||
Where("media_type = ?", mediaType).
|
||||
Distinct("folder_path").
|
||||
Pluck("folder_path", &paths).Error
|
||||
return paths, err
|
||||
}
|
||||
|
||||
// GetUnscrappedItems 获取未刮削或刮削不完整的媒体条目
|
||||
// 只要 scraped_at 为空，或 cover/scraped_name/plot 任一为空，就需要重新刮削
|
||||
// limit <= 0 表示不限制数量，返回所有未刮削条目
|
||||
func GetUnscrappedItems(mediaType model.MediaType, limit int) ([]model.MediaItem, error) {
|
||||
var items []model.MediaItem
|
||||
tx := db.Where(
|
||||
"media_type = ? AND (scraped_at IS NULL OR cover = '' OR cover IS NULL OR scraped_name = '' OR scraped_name IS NULL OR plot = '' OR plot IS NULL)",
|
||||
mediaType,
|
||||
)
|
||||
if limit > 0 {
|
||||
tx = tx.Limit(limit)
|
||||
}
|
||||
err := tx.Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
// ==================== 导入/导出 ====================
|
||||
|
||||
// ListAllMediaItems 列出指定媒体类型下的所有条目（不分页，用于导出）
|
||||
// mediaType 为空时返回全部类型的所有条目
|
||||
func ListAllMediaItems(mediaType model.MediaType) ([]model.MediaItem, error) {
|
||||
var items []model.MediaItem
|
||||
tx := db.Model(&model.MediaItem{})
|
||||
if mediaType != "" {
|
||||
tx = tx.Where("media_type = ?", mediaType)
|
||||
}
|
||||
err := tx.Order("id asc").Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
// ListMediaItemsByScanPath 列出指定扫描路径下的所有条目（不分页，用于导出）
|
||||
func ListMediaItemsByScanPath(scanPathID uint) ([]model.MediaItem, error) {
|
||||
var items []model.MediaItem
|
||||
err := db.Where("scan_path_id = ?", scanPathID).Order("id asc").Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
// ListAllMediaScanPaths 列出所有扫描路径（不区分类型，用于全量导出）
|
||||
func ListAllMediaScanPaths() ([]model.MediaScanPath, error) {
|
||||
var paths []model.MediaScanPath
|
||||
err := db.Order("id asc").Find(&paths).Error
|
||||
return paths, err
|
||||
}
|
||||
|
||||
// ImportMediaItems 批量导入媒体条目
|
||||
// 策略：按 (folder_path, file_name, album_name) 唯一键 upsert：
|
||||
// - 已存在则覆盖刮削字段（导入是用户主动行为，覆盖优先于保留）
|
||||
// - 不存在则新建
|
||||
//
|
||||
// 注意：导入数据中的 ID/CreatedAt/UpdatedAt 会被忽略，由数据库重新分配，
|
||||
// 避免和现有记录的主键冲突。
|
||||
func ImportMediaItems(items []model.MediaItem, scanPathIDOverride *uint) (created, updated int, err error) {
|
||||
for i := range items {
|
||||
it := items[i]
|
||||
// 清空主键和时间戳，让数据库重新生成
|
||||
it.ID = 0
|
||||
it.CreatedAt = time.Time{}
|
||||
it.UpdatedAt = time.Time{}
|
||||
it.DeletedAt = gorm.DeletedAt{}
|
||||
// 如果指定了覆盖的 scan_path_id（按扫描路径导入场景），则强制覆盖
|
||||
if scanPathIDOverride != nil {
|
||||
it.ScanPathID = *scanPathIDOverride
|
||||
}
|
||||
|
||||
var existing model.MediaItem
|
||||
result := db.Unscoped().Where(
|
||||
"folder_path = ? AND file_name = ? AND album_name = ?",
|
||||
it.FolderPath, it.FileName, it.AlbumName,
|
||||
).First(&existing)
|
||||
if result.Error == gorm.ErrRecordNotFound {
|
||||
if e := db.Create(&it).Error; e != nil {
|
||||
return created, updated, e
|
||||
}
|
||||
created++
|
||||
continue
|
||||
}
|
||||
if result.Error != nil {
|
||||
return created, updated, result.Error
|
||||
}
|
||||
// 覆盖更新
|
||||
it.ID = existing.ID
|
||||
it.CreatedAt = existing.CreatedAt
|
||||
it.DeletedAt = gorm.DeletedAt{}
|
||||
if e := db.Unscoped().Save(&it).Error; e != nil {
|
||||
return created, updated, e
|
||||
}
|
||||
updated++
|
||||
}
|
||||
return created, updated, nil
|
||||
}
|
||||
|
||||
// ImportMediaScanPaths 批量导入扫描路径
|
||||
// 策略：按 (media_type, path) 唯一对去重：
|
||||
// - 已存在则更新名称/标签等可编辑字段
|
||||
// - 不存在则新建并返回新ID
|
||||
//
|
||||
// 返回值 idMap：导入数据中的原始ID -> 数据库实际ID 的映射，
|
||||
// 供随后的 MediaItem 导入用于把 scan_path_id 重新指向新ID。
|
||||
func ImportMediaScanPaths(paths []model.MediaScanPath) (idMap map[uint]uint, created, updated int, err error) {
|
||||
idMap = make(map[uint]uint, len(paths))
|
||||
for i := range paths {
|
||||
p := paths[i]
|
||||
originalID := p.ID
|
||||
p.ID = 0
|
||||
p.CreatedAt = time.Time{}
|
||||
p.UpdatedAt = time.Time{}
|
||||
p.DeletedAt = gorm.DeletedAt{}
|
||||
|
||||
var existing model.MediaScanPath
|
||||
result := db.Where("media_type = ? AND path = ?", p.MediaType, p.Path).First(&existing)
|
||||
if result.Error == gorm.ErrRecordNotFound {
|
||||
if e := db.Create(&p).Error; e != nil {
|
||||
return idMap, created, updated, e
|
||||
}
|
||||
idMap[originalID] = p.ID
|
||||
created++
|
||||
continue
|
||||
}
|
||||
if result.Error != nil {
|
||||
return idMap, created, updated, result.Error
|
||||
}
|
||||
// 已存在：更新可编辑字段
|
||||
existing.Name = p.Name
|
||||
existing.PathMerge = p.PathMerge
|
||||
existing.TypeTag = p.TypeTag
|
||||
existing.ContentTags = p.ContentTags
|
||||
existing.EnableScrape = p.EnableScrape
|
||||
if e := db.Save(&existing).Error; e != nil {
|
||||
return idMap, created, updated, e
|
||||
}
|
||||
idMap[originalID] = existing.ID
|
||||
updated++
|
||||
}
|
||||
return idMap, created, updated, nil
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// ==================== MediaAppKey ====================
|
||||
|
||||
func CreateMediaAppKey(k *model.MediaAppKey) error {
|
||||
return db.Create(k).Error
|
||||
}
|
||||
|
||||
func ListMediaAppKeys(userID uint) ([]model.MediaAppKey, error) {
|
||||
var keys []model.MediaAppKey
|
||||
err := db.Where("user_id = ?", userID).Order("id asc").Find(&keys).Error
|
||||
return keys, err
|
||||
}
|
||||
|
||||
func GetMediaAppKey(id uint) (*model.MediaAppKey, error) {
|
||||
var k model.MediaAppKey
|
||||
err := db.First(&k, id).Error
|
||||
return &k, err
|
||||
}
|
||||
|
||||
func DeleteMediaAppKey(userID, id uint) error {
|
||||
return db.Where("id = ? AND user_id = ?", id, userID).Delete(&model.MediaAppKey{}).Error
|
||||
}
|
||||
|
||||
func TouchMediaAppKey(id uint) {
|
||||
now := time.Now()
|
||||
_ = db.Model(&model.MediaAppKey{}).Where("id = ?", id).Update("last_used_at", now).Error
|
||||
}
|
||||
|
||||
// ListMediaAppKeysByPrefix 认证时按明文前缀预筛选候选密钥
|
||||
func ListMediaAppKeysByPrefix(prefix string) ([]model.MediaAppKey, error) {
|
||||
var keys []model.MediaAppKey
|
||||
err := db.Where("prefix = ?", prefix).Find(&keys).Error
|
||||
return keys, err
|
||||
}
|
||||
|
||||
// ==================== MediaUserData ====================
|
||||
|
||||
func GetMediaUserData(userID, itemID uint, episodeIndex int) (*model.MediaUserData, error) {
|
||||
var d model.MediaUserData
|
||||
err := db.Where("user_id = ? AND item_id = ? AND episode_index = ?", userID, itemID, episodeIndex).First(&d).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &d, nil
|
||||
}
|
||||
|
||||
func SaveMediaUserData(d *model.MediaUserData) error {
|
||||
var existing model.MediaUserData
|
||||
err := db.Where("user_id = ? AND item_id = ? AND episode_index = ?", d.UserID, d.ItemID, d.EpisodeIndex).First(&existing).Error
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return db.Create(d).Error
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.ID = existing.ID
|
||||
d.CreatedAt = existing.CreatedAt
|
||||
return db.Save(d).Error
|
||||
}
|
||||
|
||||
// ListMediaUserDataByItems 批量取用户对一组条目的播放状态
|
||||
func ListMediaUserDataByItems(userID uint, itemIDs []uint) ([]model.MediaUserData, error) {
|
||||
var list []model.MediaUserData
|
||||
if len(itemIDs) == 0 {
|
||||
return list, nil
|
||||
}
|
||||
err := db.Where("user_id = ? AND item_id IN ?", userID, itemIDs).Find(&list).Error
|
||||
return list, err
|
||||
}
|
||||
|
||||
// ListMediaFavorites 用户收藏的条目
|
||||
func ListMediaFavorites(userID uint) ([]model.MediaUserData, error) {
|
||||
var list []model.MediaUserData
|
||||
err := db.Where("user_id = ? AND is_favorite = ?", userID, true).Order("updated_at desc").Find(&list).Error
|
||||
return list, err
|
||||
}
|
||||
|
||||
// ListMediaResume 继续观看：未看完且有进度
|
||||
func ListMediaResume(userID uint, limit int) ([]model.MediaUserData, error) {
|
||||
var list []model.MediaUserData
|
||||
if limit <= 0 {
|
||||
limit = 50
|
||||
}
|
||||
err := db.Where("user_id = ? AND played = ? AND position_ticks > 0", userID, false).
|
||||
Order("last_played_at desc").Limit(limit).Find(&list).Error
|
||||
return list, err
|
||||
}
|
||||
|
||||
// ClearMediaUserDataByUser 删除用户全部播放状态
|
||||
func ClearMediaUserDataByUser(userID uint) error {
|
||||
return db.Where("user_id = ?", userID).Delete(&model.MediaUserData{}).Error
|
||||
}
|
||||
|
||||
// ==================== MediaPlaylist ====================
|
||||
|
||||
func CreateMediaPlaylist(p *model.MediaPlaylist) error {
|
||||
return db.Create(p).Error
|
||||
}
|
||||
|
||||
func GetMediaPlaylist(id uint) (*model.MediaPlaylist, error) {
|
||||
var p model.MediaPlaylist
|
||||
err := db.First(&p, id).Error
|
||||
return &p, err
|
||||
}
|
||||
|
||||
func ListMediaPlaylists(userID uint) ([]model.MediaPlaylist, error) {
|
||||
var list []model.MediaPlaylist
|
||||
err := db.Where("user_id = ?", userID).Order("id asc").Find(&list).Error
|
||||
return list, err
|
||||
}
|
||||
|
||||
func UpdateMediaPlaylist(p *model.MediaPlaylist) error {
|
||||
return db.Save(p).Error
|
||||
}
|
||||
|
||||
func DeleteMediaPlaylist(id uint) error {
|
||||
if err := db.Where("playlist_id = ?", id).Delete(&model.MediaPlaylistItem{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return db.Delete(&model.MediaPlaylist{}, id).Error
|
||||
}
|
||||
|
||||
// ReplaceMediaPlaylistItems 覆盖式写入播放列表条目
|
||||
func ReplaceMediaPlaylistItems(playlistID uint, items []model.MediaPlaylistItem) error {
|
||||
if err := db.Where("playlist_id = ?", playlistID).Delete(&model.MediaPlaylistItem{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(items) == 0 {
|
||||
return nil
|
||||
}
|
||||
for i := range items {
|
||||
items[i].ID = 0
|
||||
items[i].PlaylistID = playlistID
|
||||
items[i].Order = i
|
||||
}
|
||||
return db.Create(&items).Error
|
||||
}
|
||||
|
||||
func ListMediaPlaylistItems(playlistID uint) ([]model.MediaPlaylistItem, error) {
|
||||
var items []model.MediaPlaylistItem
|
||||
err := db.Where("playlist_id = ?", playlistID).Order("sort_order asc").Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
// CountMediaPlaylistItems 统计条目数
|
||||
func CountMediaPlaylistItems(playlistID uint) (int64, error) {
|
||||
var n int64
|
||||
err := db.Model(&model.MediaPlaylistItem{}).Where("playlist_id = ?", playlistID).Count(&n).Error
|
||||
return n, err
|
||||
}
|
||||
|
||||
// ==================== EmbyDevice ====================
|
||||
|
||||
func CreateEmbyDevice(d *model.EmbyDevice) error {
|
||||
return db.Create(d).Error
|
||||
}
|
||||
|
||||
func GetEmbyDeviceByTokenHash(hash string) (*model.EmbyDevice, error) {
|
||||
var d model.EmbyDevice
|
||||
err := db.Where("token_hash = ?", hash).First(&d).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &d, nil
|
||||
}
|
||||
|
||||
func DeleteEmbyDevice(userID uint, deviceID string) error {
|
||||
tx := db.Where("user_id = ?", userID)
|
||||
if deviceID != "" {
|
||||
tx = tx.Where("device_id = ?", deviceID)
|
||||
}
|
||||
return tx.Delete(&model.EmbyDevice{}).Error
|
||||
}
|
||||
|
||||
func TouchEmbyDevice(id uint) {
|
||||
_ = db.Model(&model.EmbyDevice{}).Where("id = ?", id).Update("last_seen_at", time.Now()).Error
|
||||
}
|
||||
|
||||
// ==================== MediaAnnotation ====================
|
||||
|
||||
// ListMediaAnnotations 列出某用户对某个条目的全部批注
|
||||
func ListMediaAnnotations(userID, itemID uint) ([]model.MediaAnnotation, error) {
|
||||
list := []model.MediaAnnotation{}
|
||||
err := db.Where("user_id = ? AND item_id = ?", userID, itemID).
|
||||
Order("id asc").Find(&list).Error
|
||||
return list, err
|
||||
}
|
||||
|
||||
// SyncMediaAnnotations 增量保存批注：upserts 覆盖同一 remote_id 的记录，deletes 按 remote_id 删除。
|
||||
// 放在一个事务里，避免多设备同时保存时出现半新半旧的状态。
|
||||
func SyncMediaAnnotations(userID, itemID uint, upserts []model.MediaAnnotation, deletes []string) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
for i := range upserts {
|
||||
row := upserts[i]
|
||||
row.ID = 0
|
||||
row.UserID, row.ItemID = userID, itemID
|
||||
if err := tx.Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{
|
||||
{Name: "user_id"}, {Name: "item_id"}, {Name: "remote_id"},
|
||||
},
|
||||
DoUpdates: clause.AssignmentColumns([]string{"anchor", "updated_at"}),
|
||||
}).Create(&row).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if len(deletes) > 0 {
|
||||
if err := tx.Where("user_id = ? AND item_id = ? AND remote_id IN ?", userID, itemID, deletes).
|
||||
Delete(&model.MediaAnnotation{}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/md5"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
)
|
||||
|
||||
const appKeyPrefixLen = 8
|
||||
|
||||
// AppKeySecretLen 明文密钥长度（不含前缀分隔符）
|
||||
const appKeySecretLen = 32
|
||||
|
||||
// NewAppKeyPlain 生成一个新的应用密钥明文，格式：olmp_<40 位随机串>
|
||||
func NewAppKeyPlain() (string, error) {
|
||||
buf := make([]byte, appKeySecretLen)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
body := strings.TrimRight(base64.RawURLEncoding.EncodeToString(buf), "=")
|
||||
if len(body) > 40 {
|
||||
body = body[:40]
|
||||
}
|
||||
return "olmp_" + body, nil
|
||||
}
|
||||
|
||||
// AppKeyPrefix 返回用于检索/展示的明文前缀
|
||||
func AppKeyPrefix(plain string) string {
|
||||
if len(plain) <= appKeyPrefixLen {
|
||||
return plain
|
||||
}
|
||||
return plain[:appKeyPrefixLen]
|
||||
}
|
||||
|
||||
func appKeyCipherKey() []byte {
|
||||
seed := conf.Conf.JwtSecret
|
||||
if seed == "" {
|
||||
seed = setting.GetStr(conf.Token)
|
||||
}
|
||||
sum := sha256.Sum256([]byte("openlist-media-appkey:" + seed))
|
||||
return sum[:]
|
||||
}
|
||||
|
||||
func newGCM() (cipher.AEAD, error) {
|
||||
block, err := aes.NewCipher(appKeyCipherKey())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return cipher.NewGCM(block)
|
||||
}
|
||||
|
||||
// EncryptAppKey 加密应用密钥明文
|
||||
func EncryptAppKey(plain string) (string, error) {
|
||||
gcm, err := newGCM()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
nonce := make([]byte, gcm.NonceSize())
|
||||
if _, err := rand.Read(nonce); err != nil {
|
||||
return "", err
|
||||
}
|
||||
out := gcm.Seal(nonce, nonce, []byte(plain), nil)
|
||||
return base64.RawURLEncoding.EncodeToString(out), nil
|
||||
}
|
||||
|
||||
// DecryptAppKey 解密应用密钥明文
|
||||
func DecryptAppKey(secret string) (string, error) {
|
||||
raw, err := base64.RawURLEncoding.DecodeString(secret)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
gcm, err := newGCM()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(raw) < gcm.NonceSize() {
|
||||
return "", errors.New("invalid app key secret")
|
||||
}
|
||||
nonce, data := raw[:gcm.NonceSize()], raw[gcm.NonceSize():]
|
||||
plain, err := gcm.Open(nil, nonce, data, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(plain), nil
|
||||
}
|
||||
|
||||
// HashToken 计算设备令牌的存储哈希
|
||||
func HashToken(token string) string {
|
||||
sum := sha256.Sum256([]byte("openlist-emby-token:" + token))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// NewDeviceToken 生成 Emby 设备令牌
|
||||
func NewDeviceToken() (string, error) {
|
||||
buf := make([]byte, 24)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(buf), nil
|
||||
}
|
||||
|
||||
// SubsonicToken 计算 Subsonic 的 t = md5(password + salt)
|
||||
func SubsonicToken(password, salt string) string {
|
||||
sum := md5.Sum([]byte(password + salt))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// MatchSubsonicToken 校验客户端提交的 t 是否匹配
|
||||
func MatchSubsonicToken(password, salt, token string) bool {
|
||||
return strings.EqualFold(SubsonicToken(password, salt), token)
|
||||
}
|
||||
@@ -0,0 +1,684 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"io"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/imgmime"
|
||||
)
|
||||
|
||||
// maxEmbeddedPictureBytes 内嵌封面图片最大字节数，防止畸形文件触发超大内存分配
|
||||
const maxEmbeddedPictureBytes = 16 << 20
|
||||
|
||||
// MusicTag 音频文件标签信息
|
||||
type MusicTag struct {
|
||||
Title string // TIT2
|
||||
Artist string // TPE1
|
||||
Album string // TALB
|
||||
AlbumArtist string // TPE2
|
||||
TrackNumber int // TRCK
|
||||
Year string // TYER / TDRC
|
||||
Genre string // TCON
|
||||
CoverData []byte // 封面图片原始字节（APIC / PICTURE）
|
||||
CoverMIME string // 封面图片 MIME 类型（如 image/jpeg）
|
||||
Lyrics string // 歌词内容（LRC 文本或纯文本）：来自 USLT / SYLT / Vorbis LYRICS
|
||||
}
|
||||
|
||||
// ParseID3v2 从 io.Reader 中解析 ID3v2 标签（只读取文件头部，不需要 Seek）
|
||||
// 支持 ID3v2.3 和 ID3v2.4
|
||||
func ParseID3v2(r io.Reader) (*MusicTag, error) {
|
||||
// 读取 ID3v2 头部（10 字节）
|
||||
header := make([]byte, 10)
|
||||
if _, err := io.ReadFull(r, header); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 检查魔数
|
||||
if string(header[0:3]) != "ID3" {
|
||||
return nil, nil // 不是 ID3v2 文件，不报错
|
||||
}
|
||||
|
||||
version := header[3] // 主版本号：3 = ID3v2.3, 4 = ID3v2.4
|
||||
if version < 3 || version > 4 {
|
||||
return nil, nil // 只支持 v2.3 和 v2.4
|
||||
}
|
||||
|
||||
// 解析标签总大小（syncsafe integer，4 字节，每字节最高位为 0）
|
||||
tagSize := syncsafeToInt(header[6:10])
|
||||
if tagSize <= 0 || tagSize > 10*1024*1024 { // 最大 10MB
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// 读取所有帧数据
|
||||
data := make([]byte, tagSize)
|
||||
if _, err := io.ReadFull(r, data); err != nil {
|
||||
return nil, nil // 读取失败时静默跳过
|
||||
}
|
||||
|
||||
tag := &MusicTag{}
|
||||
pos := 0
|
||||
|
||||
// 跳过扩展头部（如果有）
|
||||
flags := header[5]
|
||||
if flags&0x40 != 0 { // 有扩展头部
|
||||
if pos+4 > len(data) {
|
||||
return tag, nil
|
||||
}
|
||||
var extSize int
|
||||
if version == 4 {
|
||||
extSize = syncsafeToInt(data[pos : pos+4])
|
||||
} else {
|
||||
extSize = int(binary.BigEndian.Uint32(data[pos : pos+4]))
|
||||
}
|
||||
pos += extSize
|
||||
}
|
||||
|
||||
// 解析帧
|
||||
for pos+10 <= len(data) {
|
||||
frameID := string(data[pos : pos+4])
|
||||
if frameID == "\x00\x00\x00\x00" {
|
||||
break // 填充区域，结束
|
||||
}
|
||||
|
||||
// ID3v2.4 帧大小使用 syncsafe integer（与标签头一致），
|
||||
// ID3v2.3 使用普通 32 位大端整数。
|
||||
var frameSize int
|
||||
if version == 4 {
|
||||
frameSize = syncsafeToInt(data[pos+4 : pos+8])
|
||||
} else {
|
||||
frameSize = int(binary.BigEndian.Uint32(data[pos+4 : pos+8]))
|
||||
}
|
||||
pos += 10 // 跳过帧头（4+4+2）
|
||||
|
||||
if frameSize <= 0 || pos+frameSize > len(data) {
|
||||
break
|
||||
}
|
||||
|
||||
frameData := data[pos : pos+frameSize]
|
||||
pos += frameSize
|
||||
|
||||
// 解析文本帧（T 开头的帧）
|
||||
if len(frameID) == 4 && frameID[0] == 'T' && len(frameData) > 0 {
|
||||
text := decodeID3Text(frameData)
|
||||
switch frameID {
|
||||
case "TIT2":
|
||||
tag.Title = text
|
||||
case "TPE1":
|
||||
tag.Artist = text
|
||||
case "TALB":
|
||||
tag.Album = text
|
||||
case "TPE2":
|
||||
tag.AlbumArtist = text
|
||||
case "TRCK":
|
||||
// 格式可能是 "1" 或 "1/12"
|
||||
parts := strings.SplitN(text, "/", 2)
|
||||
if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil {
|
||||
tag.TrackNumber = n
|
||||
}
|
||||
case "TYER", "TDRC":
|
||||
if tag.Year == "" {
|
||||
tag.Year = text
|
||||
}
|
||||
case "TCON":
|
||||
tag.Genre = parseID3Genre(text)
|
||||
}
|
||||
}
|
||||
|
||||
// 解析 APIC 帧（内嵌封面图片），只取第一张
|
||||
if frameID == "APIC" && tag.CoverData == nil && len(frameData) > 1 {
|
||||
tag.CoverData, tag.CoverMIME = parseAPICFrame(frameData)
|
||||
}
|
||||
|
||||
// 解析 USLT 帧（非同步歌词）
|
||||
// 格式：编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本
|
||||
if frameID == "USLT" && tag.Lyrics == "" && len(frameData) > 5 {
|
||||
tag.Lyrics = parseUSLTFrame(frameData)
|
||||
}
|
||||
|
||||
// 解析 SYLT 帧（同步歌词，二进制时间戳格式），仅在没有 USLT 时尝试转 LRC
|
||||
if frameID == "SYLT" && tag.Lyrics == "" && len(frameData) > 6 {
|
||||
if lrc := parseSYLTFrame(frameData); lrc != "" {
|
||||
tag.Lyrics = lrc
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return tag, nil
|
||||
}
|
||||
|
||||
// parseUSLTFrame 解析 ID3v2 USLT 帧（非同步歌词）
|
||||
// 格式：编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本
|
||||
func parseUSLTFrame(data []byte) string {
|
||||
if len(data) < 5 {
|
||||
return ""
|
||||
}
|
||||
encoding := data[0]
|
||||
// 跳过编码(1) + 语言(3) = 4 字节
|
||||
pos := 4
|
||||
|
||||
// 跳过描述字符串
|
||||
if encoding == 0x01 || encoding == 0x02 {
|
||||
// UTF-16：找 \x00\x00 结尾
|
||||
for pos+1 < len(data) {
|
||||
if data[pos] == 0 && data[pos+1] == 0 {
|
||||
pos += 2
|
||||
break
|
||||
}
|
||||
pos += 2
|
||||
}
|
||||
} else {
|
||||
// ISO-8859-1 / UTF-8：找单个 \x00 结尾
|
||||
for pos < len(data) {
|
||||
if data[pos] == 0 {
|
||||
pos++
|
||||
break
|
||||
}
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
if pos >= len(data) {
|
||||
return ""
|
||||
}
|
||||
|
||||
// 剩余字节就是歌词文本，按编码解码（复用 decodeID3Text 但需要把 encoding 字节加回头部）
|
||||
lyricsRaw := make([]byte, 1+len(data)-pos)
|
||||
lyricsRaw[0] = encoding
|
||||
copy(lyricsRaw[1:], data[pos:])
|
||||
return strings.TrimSpace(decodeID3Text(lyricsRaw))
|
||||
}
|
||||
|
||||
// parseSYLTFrame 解析 ID3v2 SYLT 帧（同步歌词），转换为 LRC 格式
|
||||
// 格式：编码(1) + 语言(3) + 时间戳格式(1) + 内容类型(1) + 描述(null 结尾) + 多组 (歌词文本 + null + 4字节时间戳)
|
||||
// 时间戳格式：1=MPEG帧数（不支持），2=毫秒
|
||||
func parseSYLTFrame(data []byte) string {
|
||||
if len(data) < 6 {
|
||||
return ""
|
||||
}
|
||||
encoding := data[0]
|
||||
timestampFormat := data[4] // 1=MPEG帧, 2=ms
|
||||
if timestampFormat != 2 {
|
||||
return "" // 只支持毫秒时间戳
|
||||
}
|
||||
pos := 6 // 跳过 encoding(1) + lang(3) + timestamp_fmt(1) + content_type(1)
|
||||
|
||||
// 跳过描述字符串
|
||||
if encoding == 0x01 || encoding == 0x02 {
|
||||
for pos+1 < len(data) {
|
||||
if data[pos] == 0 && data[pos+1] == 0 {
|
||||
pos += 2
|
||||
break
|
||||
}
|
||||
pos += 2
|
||||
}
|
||||
} else {
|
||||
for pos < len(data) {
|
||||
if data[pos] == 0 {
|
||||
pos++
|
||||
break
|
||||
}
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
var sb strings.Builder
|
||||
wide := encoding == 0x01 || encoding == 0x02
|
||||
for pos < len(data) {
|
||||
// 找文本终止符
|
||||
textStart := pos
|
||||
textEnd := -1
|
||||
if wide {
|
||||
for i := pos; i+1 < len(data); i += 2 {
|
||||
if data[i] == 0 && data[i+1] == 0 {
|
||||
textEnd = i
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for i := pos; i < len(data); i++ {
|
||||
if data[i] == 0 {
|
||||
textEnd = i
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if textEnd < 0 {
|
||||
break
|
||||
}
|
||||
// 解码歌词文本
|
||||
lineRaw := make([]byte, 1+textEnd-textStart)
|
||||
lineRaw[0] = encoding
|
||||
copy(lineRaw[1:], data[textStart:textEnd])
|
||||
lineText := decodeID3Text(lineRaw)
|
||||
|
||||
// 跳过文本终止符
|
||||
if wide {
|
||||
pos = textEnd + 2
|
||||
} else {
|
||||
pos = textEnd + 1
|
||||
}
|
||||
// 4 字节时间戳（毫秒）
|
||||
if pos+4 > len(data) {
|
||||
break
|
||||
}
|
||||
ms := int(binary.BigEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4
|
||||
|
||||
minutes := ms / 60000
|
||||
seconds := (ms % 60000) / 1000
|
||||
cs := (ms % 1000) / 10
|
||||
// LRC 标准格式 [mm:ss.xx]
|
||||
sb.WriteString("[")
|
||||
if minutes < 10 {
|
||||
sb.WriteString("0")
|
||||
}
|
||||
sb.WriteString(strconv.Itoa(minutes))
|
||||
sb.WriteString(":")
|
||||
if seconds < 10 {
|
||||
sb.WriteString("0")
|
||||
}
|
||||
sb.WriteString(strconv.Itoa(seconds))
|
||||
sb.WriteString(".")
|
||||
if cs < 10 {
|
||||
sb.WriteString("0")
|
||||
}
|
||||
sb.WriteString(strconv.Itoa(cs))
|
||||
sb.WriteString("]")
|
||||
sb.WriteString(lineText)
|
||||
sb.WriteString("\n")
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// parseAPICFrame 解析 ID3v2 APIC 帧，返回图片字节和 MIME 类型
|
||||
// APIC 格式：编码字节(1) + MIME字符串(null结尾) + 图片类型(1) + 描述字符串(null结尾) + 图片数据
|
||||
func parseAPICFrame(data []byte) ([]byte, string) {
|
||||
if len(data) < 4 {
|
||||
return nil, ""
|
||||
}
|
||||
// encoding := data[0] // 文本编码（只影响描述字段，MIME 始终是 ASCII）
|
||||
pos := 1
|
||||
|
||||
// 读取 MIME 类型（null 结尾的 ASCII 字符串）
|
||||
nullIdx := -1
|
||||
for i := pos; i < len(data); i++ {
|
||||
if data[i] == 0 {
|
||||
nullIdx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if nullIdx < 0 {
|
||||
return nil, ""
|
||||
}
|
||||
mimeType := string(data[pos:nullIdx])
|
||||
pos = nullIdx + 1
|
||||
|
||||
if pos >= len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
// 图片类型（1字节，3 = Cover (front)，但我们取任意第一张）
|
||||
pos++ // 跳过图片类型字节
|
||||
|
||||
// 跳过描述字符串（null 结尾，编码由第一字节决定）
|
||||
encoding := data[0]
|
||||
if encoding == 0x01 || encoding == 0x02 {
|
||||
// UTF-16：找 \x00\x00 结尾
|
||||
for pos+1 < len(data) {
|
||||
if data[pos] == 0 && data[pos+1] == 0 {
|
||||
pos += 2
|
||||
break
|
||||
}
|
||||
pos += 2
|
||||
}
|
||||
} else {
|
||||
// ISO-8859-1 / UTF-8：找单个 \x00 结尾
|
||||
for pos < len(data) {
|
||||
if data[pos] == 0 {
|
||||
pos++
|
||||
break
|
||||
}
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
if pos >= len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
|
||||
imgData := make([]byte, len(data)-pos)
|
||||
copy(imgData, data[pos:])
|
||||
// 声明的 MIME 来自被扫描文件本身，不可信（可能是 image/svg+xml）。
|
||||
// 只按魔数判定并限定为四种栅格格式，无法识别则丢弃封面。
|
||||
_ = mimeType
|
||||
safe := imgmime.Sniff(imgData)
|
||||
if safe == "" {
|
||||
return nil, ""
|
||||
}
|
||||
return imgData, safe
|
||||
}
|
||||
|
||||
// syncsafeToInt 将 syncsafe integer 转换为普通整数
|
||||
func syncsafeToInt(b []byte) int {
|
||||
result := 0
|
||||
for _, v := range b {
|
||||
result = (result << 7) | int(v&0x7F)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// decodeID3Text 解码 ID3 文本帧（第一字节是编码标识）
|
||||
func decodeID3Text(data []byte) string {
|
||||
if len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
encoding := data[0]
|
||||
content := data[1:]
|
||||
|
||||
// 去掉末尾的 null 字节
|
||||
switch encoding {
|
||||
case 0x00: // ISO-8859-1
|
||||
// 去掉末尾 null
|
||||
content = trimNull(content, false)
|
||||
// 尝试 UTF-8 解码，如果失败则按 Latin-1 处理
|
||||
return latin1ToUTF8(content)
|
||||
case 0x01: // UTF-16 with BOM
|
||||
content = trimNull(content, true)
|
||||
return utf16ToUTF8(content)
|
||||
case 0x02: // UTF-16 BE without BOM
|
||||
content = trimNull(content, true)
|
||||
return utf16BEToUTF8(content)
|
||||
case 0x03: // UTF-8
|
||||
content = trimNull(content, false)
|
||||
return string(content)
|
||||
default:
|
||||
return string(content)
|
||||
}
|
||||
}
|
||||
|
||||
// trimNull 去掉末尾的 null 字节
|
||||
func trimNull(b []byte, wide bool) []byte {
|
||||
if wide {
|
||||
// UTF-16: 去掉末尾的 \x00\x00
|
||||
for len(b) >= 2 && b[len(b)-2] == 0 && b[len(b)-1] == 0 {
|
||||
b = b[:len(b)-2]
|
||||
}
|
||||
} else {
|
||||
for len(b) > 0 && b[len(b)-1] == 0 {
|
||||
b = b[:len(b)-1]
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// latin1ToUTF8 将 Latin-1 编码转换为 UTF-8
|
||||
func latin1ToUTF8(b []byte) string {
|
||||
runes := make([]rune, len(b))
|
||||
for i, v := range b {
|
||||
runes[i] = rune(v)
|
||||
}
|
||||
return string(runes)
|
||||
}
|
||||
|
||||
// utf16ToUTF8 将 UTF-16（带 BOM）转换为 UTF-8
|
||||
func utf16ToUTF8(b []byte) string {
|
||||
if len(b) < 2 {
|
||||
return ""
|
||||
}
|
||||
var bigEndian bool
|
||||
if b[0] == 0xFF && b[1] == 0xFE {
|
||||
bigEndian = false
|
||||
b = b[2:]
|
||||
} else if b[0] == 0xFE && b[1] == 0xFF {
|
||||
bigEndian = true
|
||||
b = b[2:]
|
||||
}
|
||||
return decodeUTF16(b, bigEndian)
|
||||
}
|
||||
|
||||
// utf16BEToUTF8 将 UTF-16 BE（无 BOM）转换为 UTF-8
|
||||
func utf16BEToUTF8(b []byte) string {
|
||||
return decodeUTF16(b, true)
|
||||
}
|
||||
|
||||
// decodeUTF16 解码 UTF-16 字节序列
|
||||
func decodeUTF16(b []byte, bigEndian bool) string {
|
||||
if len(b)%2 != 0 {
|
||||
b = b[:len(b)-1]
|
||||
}
|
||||
runes := make([]rune, 0, len(b)/2)
|
||||
for i := 0; i+1 < len(b); i += 2 {
|
||||
var code uint16
|
||||
if bigEndian {
|
||||
code = uint16(b[i])<<8 | uint16(b[i+1])
|
||||
} else {
|
||||
code = uint16(b[i+1])<<8 | uint16(b[i])
|
||||
}
|
||||
runes = append(runes, rune(code))
|
||||
}
|
||||
return string(runes)
|
||||
}
|
||||
|
||||
// parseID3Genre 解析 ID3 流派字段（可能是 "(17)" 格式的数字引用）
|
||||
func parseID3Genre(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if len(s) > 2 && s[0] == '(' && s[len(s)-1] == ')' {
|
||||
// 数字引用格式，直接返回原始字符串
|
||||
return s
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// ─── FLAC Vorbis Comment 解析 ────────────────────────────────────────────────
|
||||
|
||||
// ParseFLACVorbisComment 从 io.Reader 中解析 FLAC 文件的 Vorbis Comment 元数据
|
||||
// FLAC 格式：4字节魔数 "fLaC" + 若干 METADATA_BLOCK
|
||||
// METADATA_BLOCK：1字节(最高位=是否最后块, 低7位=块类型) + 3字节长度 + 数据
|
||||
// 块类型 4 = VORBIS_COMMENT
|
||||
func ParseFLACVorbisComment(r io.Reader) (*MusicTag, error) {
|
||||
// 读取魔数（4字节）
|
||||
magic := make([]byte, 4)
|
||||
if _, err := io.ReadFull(r, magic); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if string(magic) != "fLaC" {
|
||||
return nil, nil // 不是 FLAC 文件
|
||||
}
|
||||
|
||||
// 遍历 METADATA_BLOCK，找到 VORBIS_COMMENT（类型4）和 PICTURE（类型6）
|
||||
var tag *MusicTag
|
||||
for {
|
||||
// 读取块头（4字节：1字节标志+类型 + 3字节长度）
|
||||
blockHeader := make([]byte, 4)
|
||||
if _, err := io.ReadFull(r, blockHeader); err != nil {
|
||||
return nil, nil // 读取失败，静默跳过
|
||||
}
|
||||
|
||||
isLast := blockHeader[0]&0x80 != 0
|
||||
blockType := blockHeader[0] & 0x7F
|
||||
blockLen := int(blockHeader[1])<<16 | int(blockHeader[2])<<8 | int(blockHeader[3])
|
||||
|
||||
if blockLen < 0 || blockLen > 16*1024*1024 { // 最大 16MB
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
if blockType == 4 {
|
||||
// VORBIS_COMMENT 块
|
||||
data := make([]byte, blockLen)
|
||||
if _, err := io.ReadFull(r, data); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
tag = parseVorbisCommentData(data)
|
||||
if isLast {
|
||||
return tag, nil
|
||||
}
|
||||
// 继续读取后续块，寻找 PICTURE 块（类型6）
|
||||
continue
|
||||
}
|
||||
|
||||
if blockType == 6 {
|
||||
// PICTURE 块（FLAC 内嵌封面）
|
||||
data := make([]byte, blockLen)
|
||||
if _, err := io.ReadFull(r, data); err == nil && tag != nil && tag.CoverData == nil {
|
||||
tag.CoverData, tag.CoverMIME = parseFLACPictureBlock(data)
|
||||
} else if err != nil {
|
||||
// 读取失败，跳过
|
||||
_ = err
|
||||
}
|
||||
if isLast {
|
||||
return tag, nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// 跳过此块
|
||||
if _, err := io.CopyN(io.Discard, r, int64(blockLen)); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
if isLast {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return tag, nil
|
||||
}
|
||||
|
||||
// parseFLACPictureBlock 解析 FLAC PICTURE 元数据块，返回图片字节和 MIME 类型
|
||||
// 格式（大端序）：
|
||||
// 4字节 picture_type
|
||||
// 4字节 mime_length + mime_string
|
||||
// 4字节 description_length + description_string
|
||||
// 4字节 width, 4字节 height, 4字节 color_depth, 4字节 color_count
|
||||
// 4字节 data_length + data
|
||||
func parseFLACPictureBlock(data []byte) ([]byte, string) {
|
||||
if len(data) < 8 {
|
||||
return nil, ""
|
||||
}
|
||||
pos := 4 // 跳过 picture_type
|
||||
|
||||
// 读取 MIME 类型
|
||||
if pos+4 > len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
mimeLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4
|
||||
if pos+mimeLen > len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
mimeType := string(data[pos : pos+mimeLen])
|
||||
pos += mimeLen
|
||||
|
||||
// 跳过描述字符串
|
||||
if pos+4 > len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
descLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4 + descLen
|
||||
|
||||
// 跳过 width(4) + height(4) + color_depth(4) + color_count(4)
|
||||
pos += 16
|
||||
|
||||
// 读取图片数据
|
||||
if pos+4 > len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
dataLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4
|
||||
if dataLen <= 0 || dataLen > maxEmbeddedPictureBytes || pos+dataLen > len(data) {
|
||||
return nil, ""
|
||||
}
|
||||
|
||||
imgData := make([]byte, dataLen)
|
||||
copy(imgData, data[pos:pos+dataLen])
|
||||
// 同 parseAPICFrame：不信任声明的 MIME，只按魔数判定
|
||||
_ = mimeType
|
||||
safe := imgmime.Sniff(imgData)
|
||||
if safe == "" {
|
||||
return nil, ""
|
||||
}
|
||||
return imgData, safe
|
||||
}
|
||||
|
||||
// parseVorbisCommentData 解析 Vorbis Comment 数据块
|
||||
// 格式（小端序）：
|
||||
// 4字节 vendor_length + vendor_string
|
||||
// 4字节 comment_count
|
||||
// 每条注释：4字节长度 + UTF-8字符串（格式 "KEY=VALUE"）
|
||||
func parseVorbisCommentData(data []byte) *MusicTag {
|
||||
tag := &MusicTag{}
|
||||
pos := 0
|
||||
|
||||
// 跳过 vendor string
|
||||
if pos+4 > len(data) {
|
||||
return tag
|
||||
}
|
||||
vendorLen := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4 + vendorLen
|
||||
|
||||
// 读取注释数量
|
||||
if pos+4 > len(data) {
|
||||
return tag
|
||||
}
|
||||
commentCount := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4
|
||||
|
||||
for i := 0; i < commentCount; i++ {
|
||||
if pos+4 > len(data) {
|
||||
break
|
||||
}
|
||||
commentLen := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
|
||||
pos += 4
|
||||
|
||||
if commentLen <= 0 || pos+commentLen > len(data) {
|
||||
break
|
||||
}
|
||||
|
||||
comment := string(data[pos : pos+commentLen])
|
||||
pos += commentLen
|
||||
|
||||
// 解析 "KEY=VALUE" 格式（大小写不敏感）
|
||||
eqIdx := strings.IndexByte(comment, '=')
|
||||
if eqIdx < 0 {
|
||||
continue
|
||||
}
|
||||
key := strings.ToUpper(comment[:eqIdx])
|
||||
value := strings.TrimSpace(comment[eqIdx+1:])
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
switch key {
|
||||
case "TITLE":
|
||||
tag.Title = value
|
||||
case "ARTIST":
|
||||
if tag.Artist == "" {
|
||||
tag.Artist = value
|
||||
}
|
||||
case "ALBUM":
|
||||
tag.Album = value
|
||||
case "ALBUMARTIST", "ALBUM ARTIST":
|
||||
tag.AlbumArtist = value
|
||||
case "TRACKNUMBER", "TRACK":
|
||||
// 格式可能是 "1" 或 "1/12"
|
||||
parts := strings.SplitN(value, "/", 2)
|
||||
if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil {
|
||||
tag.TrackNumber = n
|
||||
}
|
||||
case "DATE", "YEAR":
|
||||
if tag.Year == "" && len(value) >= 4 {
|
||||
tag.Year = value[:4]
|
||||
}
|
||||
case "GENRE":
|
||||
tag.Genre = value
|
||||
case "LYRICS", "UNSYNCEDLYRICS", "UNSYNCED LYRICS", "SYNCEDLYRICS", "SYNCED LYRICS":
|
||||
// FLAC/Vorbis 内嵌歌词字段（不同打标软件使用不同 key，全部兼容）
|
||||
if tag.Lyrics == "" {
|
||||
tag.Lyrics = value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return tag
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
// Package imgmime 提供封面图片 MIME 的安全判定。
|
||||
//
|
||||
// 封面字节与 MIME 往往来自不可信来源（音频文件内嵌 APIC/PICTURE、远端站点的
|
||||
// Content-Type）。如果把声明的 MIME 原样写进响应头，就可能把 image/svg+xml 或
|
||||
// text/html 以站点同源方式交付给浏览器，造成存储型 XSS。
|
||||
// 这里只信任文件头魔数，且只允许四种栅格格式。
|
||||
package imgmime
|
||||
|
||||
import "bytes"
|
||||
|
||||
// Sniff 根据文件头魔数返回 image/jpeg、image/png、image/gif、image/webp 之一；
|
||||
// 无法识别（包括 SVG、HTML、未知格式）时返回空串，调用方应丢弃该封面。
|
||||
func Sniff(data []byte) string {
|
||||
switch {
|
||||
case len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF:
|
||||
return "image/jpeg"
|
||||
case len(data) >= 8 && bytes.Equal(data[:8], []byte{0x89, 'P', 'N', 'G', 0x0D, 0x0A, 0x1A, 0x0A}):
|
||||
return "image/png"
|
||||
case len(data) >= 6 && (bytes.Equal(data[:6], []byte("GIF87a")) || bytes.Equal(data[:6], []byte("GIF89a"))):
|
||||
return "image/gif"
|
||||
case len(data) >= 12 && bytes.Equal(data[:4], []byte("RIFF")) && bytes.Equal(data[8:12], []byte("WEBP")):
|
||||
return "image/webp"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Allowed 判断 MIME 是否属于允许对外输出的栅格图片类型。
|
||||
func Allowed(mime string) bool {
|
||||
switch mime {
|
||||
case "image/jpeg", "image/png", "image/gif", "image/webp":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package imgmime
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestSniff(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
data []byte
|
||||
want string
|
||||
}{
|
||||
{"jpeg", []byte{0xFF, 0xD8, 0xFF, 0xE0, 0, 0}, "image/jpeg"},
|
||||
{"png", []byte{0x89, 'P', 'N', 'G', 0x0D, 0x0A, 0x1A, 0x0A, 0}, "image/png"},
|
||||
{"gif89", []byte("GIF89a....."), "image/gif"},
|
||||
{"webp", append(append([]byte("RIFF\x00\x00\x00\x00"), []byte("WEBP")...), 0), "image/webp"},
|
||||
{"svg", []byte(`<svg xmlns="http://www.w3.org/2000/svg" onload="alert(1)"></svg>`), ""},
|
||||
{"svg with xml prolog", []byte(`<?xml version="1.0"?><svg/>`), ""},
|
||||
{"html", []byte("<html><script>alert(1)</script></html>"), ""},
|
||||
{"empty", nil, ""},
|
||||
{"short", []byte{0xFF, 0xD8}, ""},
|
||||
// RIFF 但不是 WEBP（例如 WAV）不能被当作图片
|
||||
{"riff wav", append(append([]byte("RIFF\x00\x00\x00\x00"), []byte("WAVE")...), 0), ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := Sniff(tc.data); got != tc.want {
|
||||
t.Errorf("%s: Sniff = %q, want %q", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllowed(t *testing.T) {
|
||||
for _, m := range []string{"image/jpeg", "image/png", "image/gif", "image/webp"} {
|
||||
if !Allowed(m) {
|
||||
t.Errorf("%s should be allowed", m)
|
||||
}
|
||||
}
|
||||
for _, m := range []string{"image/svg+xml", "text/html", "", "image/"} {
|
||||
if Allowed(m) {
|
||||
t.Errorf("%q should not be allowed", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
stdpath "path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
// FillMusicFromLocal 在刮削前/失败时，使用音乐文件本身的元数据（ID3 / Vorbis Comment）
|
||||
// 以及同目录 .lrc 歌词，**只填充 item 中为空的字段**。
|
||||
//
|
||||
// 适用场景：
|
||||
// 1. 用户清空刮削后重新刮削，原本由扫描阶段填入的本地数据（封面/歌词等）已丢失
|
||||
// 2. 在线刮削失败，退而求其次用文件本地信息补全
|
||||
//
|
||||
// 行为：
|
||||
// - is_folder=true 的合并文件夹模式：使用 episodes 中第一首音乐文件读取
|
||||
// - is_folder=false：使用 folder_path/file_name 直接读取
|
||||
//
|
||||
// 错误以静默忽略为主——本地兜底失败不应阻塞主流程。
|
||||
func FillMusicFromLocal(ctx context.Context, item *model.MediaItem) {
|
||||
if item == nil || item.MediaType != model.MediaTypeMusic {
|
||||
return
|
||||
}
|
||||
|
||||
musicPath := pickMusicFilePath(item)
|
||||
if musicPath == "" {
|
||||
return
|
||||
}
|
||||
|
||||
// 解析 tag（mp3 / flac / 其它）
|
||||
ext := strings.ToLower(stdpath.Ext(musicPath))
|
||||
var tag *MusicTag
|
||||
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
if reader := FetchFileReader(readCtx, musicPath); reader != nil {
|
||||
switch ext {
|
||||
case ".flac":
|
||||
tag, _ = ParseFLACVorbisComment(reader)
|
||||
default:
|
||||
tag, _ = ParseID3v2(reader)
|
||||
}
|
||||
_ = reader.Close()
|
||||
}
|
||||
readCancel()
|
||||
|
||||
// 应用 tag 到空字段
|
||||
if tag != nil {
|
||||
if item.AlbumName == "" && tag.Album != "" {
|
||||
item.AlbumName = tag.Album
|
||||
}
|
||||
if item.AlbumArtist == "" {
|
||||
if tag.AlbumArtist != "" {
|
||||
item.AlbumArtist = tag.AlbumArtist
|
||||
} else if tag.Artist != "" {
|
||||
item.AlbumArtist = tag.Artist
|
||||
}
|
||||
}
|
||||
if item.ScrapedName == "" {
|
||||
if tag.Title != "" {
|
||||
item.ScrapedName = tag.Title
|
||||
} else if item.AlbumName != "" {
|
||||
item.ScrapedName = item.AlbumName
|
||||
}
|
||||
}
|
||||
if item.TrackNumber == 0 && tag.TrackNumber > 0 {
|
||||
item.TrackNumber = tag.TrackNumber
|
||||
}
|
||||
if item.ReleaseDate == "" && len(tag.Year) >= 4 {
|
||||
item.ReleaseDate = tag.Year[:4] + "-01-01"
|
||||
}
|
||||
if item.Genre == "" && tag.Genre != "" {
|
||||
item.Genre = tag.Genre
|
||||
}
|
||||
if item.Authors == "" && tag.Artist != "" {
|
||||
if b, err := json.Marshal([]string{tag.Artist}); err == nil {
|
||||
item.Authors = string(b)
|
||||
}
|
||||
}
|
||||
if item.Cover == "" && len(tag.CoverData) > 0 {
|
||||
mimeType := tag.CoverMIME
|
||||
if mimeType == "" {
|
||||
mimeType = "image/jpeg"
|
||||
}
|
||||
item.Cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
|
||||
}
|
||||
}
|
||||
|
||||
// 歌词：若为空则尝试 .lrc 同名文件 / 内嵌歌词
|
||||
if item.Lyrics == "" {
|
||||
if lrc := loadLyricsForMusic(ctx, musicPath, tag); lrc != "" {
|
||||
item.Lyrics = lrc
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pickMusicFilePath 从 MediaItem 推断出可读取的音乐文件 VFS 路径
|
||||
// 合并模式 (is_folder=true): 取 episodes 中第一首；解析失败回退到 folder_path/file_name
|
||||
// 普通模式: folder_path/file_name
|
||||
func pickMusicFilePath(item *model.MediaItem) string {
|
||||
if item == nil {
|
||||
return ""
|
||||
}
|
||||
if item.IsFolder {
|
||||
// episodes 是 JSON 数组：[{file_name,index,title,...}]
|
||||
if item.Episodes != "" {
|
||||
var eps []EpisodeInfo
|
||||
if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil && len(eps) > 0 {
|
||||
// 文件夹模式下 folder_path 是扫描根，file_name 是文件夹名
|
||||
return stdpath.Join(item.FolderPath, item.FileName, eps[0].FileName)
|
||||
}
|
||||
}
|
||||
// 回退：尝试列出文件夹内第一首音乐
|
||||
folder := stdpath.Join(item.FolderPath, item.FileName)
|
||||
if entries, err := fs.List(context.Background(), folder, &fs.ListArgs{NoLog: true}); err == nil {
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && isMediaFile(e.GetName(), model.MediaTypeMusic) {
|
||||
return stdpath.Join(folder, e.GetName())
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
return stdpath.Join(item.FolderPath, item.FileName)
|
||||
}
|
||||
@@ -0,0 +1,758 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime"
|
||||
stdpath "path"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
log "github.com/sirupsen/logrus"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/stream"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/http_range"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
)
|
||||
|
||||
// 支持的文件扩展名
|
||||
var (
|
||||
videoExts = map[string]bool{
|
||||
".mp4": true, ".mkv": true, ".avi": true, ".mov": true,
|
||||
".wmv": true, ".flv": true, ".webm": true, ".m4v": true,
|
||||
".ts": true, ".rmvb": true, ".rm": true, ".3gp": true,
|
||||
}
|
||||
musicExts = map[string]bool{
|
||||
".mp3": true, ".flac": true, ".aac": true, ".ogg": true,
|
||||
".wav": true, ".wma": true, ".m4a": true, ".ape": true,
|
||||
".opus": true, ".aiff": true,
|
||||
}
|
||||
imageExts = map[string]bool{
|
||||
".jpg": true, ".jpeg": true, ".png": true, ".gif": true,
|
||||
".webp": true, ".bmp": true, ".tiff": true, ".svg": true,
|
||||
".heic": true, ".avif": true,
|
||||
}
|
||||
bookExts = map[string]bool{
|
||||
".epub": true, ".pdf": true, ".mobi": true, ".azw3": true,
|
||||
".txt": true, ".djvu": true, ".cbz": true, ".cbr": true,
|
||||
}
|
||||
)
|
||||
|
||||
// ScanProgress 扫描进度（全局，按媒体类型维护）
|
||||
type ScanProgress struct {
|
||||
mu sync.RWMutex
|
||||
Running bool
|
||||
Total int
|
||||
Done int
|
||||
// Skipped 入库被跳过的条目数（典型原因：目录/文件名超过唯一索引列长度上限）。
|
||||
// 以前只在日志里打一条 Warn，用户完全看不到漏扫了多少条。
|
||||
Skipped int
|
||||
Message string
|
||||
Error string
|
||||
}
|
||||
|
||||
var progressMap = map[model.MediaType]*ScanProgress{
|
||||
model.MediaTypeVideo: {},
|
||||
model.MediaTypeMusic: {},
|
||||
model.MediaTypeImage: {},
|
||||
model.MediaTypeBook: {},
|
||||
}
|
||||
|
||||
// GetProgress 获取扫描进度
|
||||
func GetProgress(mediaType model.MediaType) model.MediaScanProgress {
|
||||
p, ok := progressMap[mediaType]
|
||||
if !ok {
|
||||
return model.MediaScanProgress{MediaType: mediaType}
|
||||
}
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
return model.MediaScanProgress{
|
||||
MediaType: mediaType,
|
||||
Running: p.Running,
|
||||
Total: p.Total,
|
||||
Done: p.Done,
|
||||
Message: p.Message,
|
||||
Error: p.Error,
|
||||
}
|
||||
}
|
||||
|
||||
// ScanMedia 扫描媒体文件（异步），扫描指定媒体类型下的所有启用扫描路径
|
||||
func ScanMedia(cfg *model.MediaConfig) {
|
||||
p, ok := progressMap[cfg.MediaType]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p.mu.Lock()
|
||||
if p.Running {
|
||||
p.mu.Unlock()
|
||||
return
|
||||
}
|
||||
p.Running = true
|
||||
p.Total = 0
|
||||
p.Done = 0
|
||||
p.Skipped = 0
|
||||
p.Error = ""
|
||||
p.Message = "正在扫描..."
|
||||
p.mu.Unlock()
|
||||
|
||||
go func() {
|
||||
defer func() {
|
||||
p.mu.Lock()
|
||||
p.Running = false
|
||||
p.mu.Unlock()
|
||||
}()
|
||||
|
||||
// 获取该媒体类型下所有扫描路径
|
||||
scanPaths, err := db.ListMediaScanPaths(cfg.MediaType)
|
||||
if err != nil {
|
||||
p.mu.Lock()
|
||||
p.Error = err.Error()
|
||||
p.Message = "获取扫描路径失败"
|
||||
p.mu.Unlock()
|
||||
return
|
||||
}
|
||||
if len(scanPaths) == 0 {
|
||||
p.mu.Lock()
|
||||
p.Message = "没有配置扫描路径"
|
||||
p.mu.Unlock()
|
||||
return
|
||||
}
|
||||
|
||||
for i := range scanPaths {
|
||||
sp := &scanPaths[i]
|
||||
p.mu.Lock()
|
||||
p.Message = "正在扫描路径: " + sp.Name
|
||||
p.mu.Unlock()
|
||||
|
||||
if err := doScanPath(sp, p); err != nil {
|
||||
log.Errorf("media scan error [%s] path[%s]: %v", cfg.MediaType, sp.Path, err)
|
||||
} else {
|
||||
// 更新最后扫描时间
|
||||
now := time.Now()
|
||||
sp.LastScanAt = &now
|
||||
_ = db.UpdateMediaScanPath(sp)
|
||||
}
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
if p.Skipped > 0 {
|
||||
// 让管理端能直接看到「有 N 条因为路径过长没入库」
|
||||
p.Message = fmt.Sprintf("扫描完成（跳过 %d 条）", p.Skipped)
|
||||
log.Warnf("media scan skipped %d items (path/name too long for unique index)", p.Skipped)
|
||||
} else {
|
||||
p.Message = "扫描完成"
|
||||
}
|
||||
p.mu.Unlock()
|
||||
// 更新媒体库最后扫描时间
|
||||
now := time.Now()
|
||||
cfg.LastScanAt = &now
|
||||
_ = db.SaveMediaConfig(cfg)
|
||||
|
||||
// 按需生成 STRM
|
||||
TriggerStrmOnScan(cfg.MediaType, GetAPIRoot())
|
||||
}()
|
||||
}
|
||||
|
||||
// ScanMediaPath 扫描单个扫描路径（异步）
|
||||
func ScanMediaPath(sp *model.MediaScanPath) {
|
||||
p, ok := progressMap[sp.MediaType]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p.mu.Lock()
|
||||
if p.Running {
|
||||
p.mu.Unlock()
|
||||
return
|
||||
}
|
||||
p.Running = true
|
||||
p.Total = 0
|
||||
p.Done = 0
|
||||
p.Skipped = 0
|
||||
p.Error = ""
|
||||
p.Message = "正在扫描路径: " + sp.Name
|
||||
p.mu.Unlock()
|
||||
|
||||
go func() {
|
||||
defer func() {
|
||||
p.mu.Lock()
|
||||
p.Running = false
|
||||
p.mu.Unlock()
|
||||
}()
|
||||
|
||||
if err := doScanPath(sp, p); err != nil {
|
||||
p.mu.Lock()
|
||||
p.Error = err.Error()
|
||||
p.Message = "扫描失败"
|
||||
p.mu.Unlock()
|
||||
log.Errorf("media scan error [%s] path[%s]: %v", sp.MediaType, sp.Path, err)
|
||||
} else {
|
||||
p.mu.Lock()
|
||||
if p.Skipped > 0 {
|
||||
p.Message = fmt.Sprintf("扫描完成（跳过 %d 条）", p.Skipped)
|
||||
log.Warnf("media scan skipped %d items (path/name too long for unique index)", p.Skipped)
|
||||
} else {
|
||||
p.Message = "扫描完成"
|
||||
}
|
||||
p.mu.Unlock()
|
||||
now := time.Now()
|
||||
sp.LastScanAt = &now
|
||||
_ = db.UpdateMediaScanPath(sp)
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func doScanPath(sp *model.MediaScanPath, p *ScanProgress) error {
|
||||
scanRoot := sp.Path
|
||||
if scanRoot == "" {
|
||||
scanRoot = "/"
|
||||
}
|
||||
|
||||
// 收集所有待处理路径（VFS 虚拟路径）
|
||||
var targets []string
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
if sp.PathMerge {
|
||||
// 路径合并模式：
|
||||
// - 子文件夹 → 作为一个合并条目（带 Episodes 选集信息）
|
||||
// - 直接放在根目录下的单个媒体文件 → 正常作为独立条目扫描
|
||||
entries, err := fs.List(ctx, scanRoot, &fs.ListArgs{NoLog: true, Refresh: true})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range entries {
|
||||
childPath := stdpath.Join(scanRoot, e.GetName())
|
||||
if e.IsDir() {
|
||||
targets = append(targets, childPath)
|
||||
} else if isMediaFile(e.GetName(), sp.MediaType) {
|
||||
targets = append(targets, childPath)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// 普通模式：递归扫描所有匹配文件（每个目录都刷新缓存）
|
||||
if err := walkVFS(ctx, scanRoot, sp.MediaType, &targets); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
p.Total += len(targets)
|
||||
p.mu.Unlock()
|
||||
|
||||
// 音乐类型：按专辑合并处理
|
||||
if sp.MediaType == model.MediaTypeMusic {
|
||||
return doScanMusicByAlbum(ctx, targets, sp, p)
|
||||
}
|
||||
|
||||
for _, target := range targets {
|
||||
item, err := buildMediaItemFromVFS(ctx, target, sp)
|
||||
if err != nil {
|
||||
log.Warnf("build media item error [%s]: %v", target, err)
|
||||
continue
|
||||
}
|
||||
|
||||
// 路径合并模式下，扫描文件夹内的文件，填充选集信息
|
||||
if sp.PathMerge && item.IsFolder {
|
||||
if episodes, err := buildEpisodesFromFolder(ctx, target, sp.MediaType); err == nil {
|
||||
item.Episodes = episodes
|
||||
} else {
|
||||
log.Warnf("build episodes error [%s]: %v", target, err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := db.CreateOrUpdateMediaItem(item); err != nil {
|
||||
log.Warnf("save media item error [%s]: %v", target, err)
|
||||
p.mu.Lock()
|
||||
p.Skipped++
|
||||
p.mu.Unlock()
|
||||
}
|
||||
p.mu.Lock()
|
||||
p.Done++
|
||||
p.Message = "已扫描: " + stdpath.Base(target)
|
||||
p.mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// doScanMusicByAlbum 音乐扫描
|
||||
// 存储规则：
|
||||
// - 普通文件（is_folder=false）：
|
||||
// folder_path=文件所在真实目录，file_name=音乐文件名，episodes=空
|
||||
// - 合并文件夹（is_folder=true）：
|
||||
// folder_path=文件夹所在真实父目录，file_name=文件夹名，
|
||||
// episodes=文件夹内所有音乐文件列表，封面/标签取第一首
|
||||
func doScanMusicByAlbum(ctx context.Context, targets []string, sp *model.MediaScanPath, p *ScanProgress) error {
|
||||
for _, target := range targets {
|
||||
obj, err := fs.Get(ctx, target, &fs.GetArgs{NoLog: true})
|
||||
if err != nil {
|
||||
log.Warnf("get vfs object error [%s]: %v", target, err)
|
||||
p.mu.Lock()
|
||||
p.Done++
|
||||
p.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
|
||||
name := obj.GetName()
|
||||
|
||||
if obj.IsDir() {
|
||||
// ---- 合并文件夹模式：文件夹条目 ----
|
||||
// 列出文件夹内所有音乐文件
|
||||
entries, err := fs.List(ctx, target, &fs.ListArgs{NoLog: true})
|
||||
if err != nil {
|
||||
log.Warnf("list music folder error [%s]: %v", target, err)
|
||||
p.mu.Lock()
|
||||
p.Done++
|
||||
p.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
|
||||
var musicFiles []string
|
||||
var episodes []EpisodeInfo
|
||||
for idx, e := range entries {
|
||||
if e.IsDir() || !isMediaFile(e.GetName(), sp.MediaType) {
|
||||
continue
|
||||
}
|
||||
musicFiles = append(musicFiles, stdpath.Join(target, e.GetName()))
|
||||
ep := EpisodeInfo{
|
||||
FileName: e.GetName(),
|
||||
Index: idx + 1,
|
||||
Title: strings.TrimSuffix(e.GetName(), stdpath.Ext(e.GetName())),
|
||||
}
|
||||
episodes = append(episodes, ep)
|
||||
}
|
||||
|
||||
if len(musicFiles) == 0 {
|
||||
p.mu.Lock()
|
||||
p.Done++
|
||||
p.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
|
||||
// 取第一首音乐文件读取标签
|
||||
firstFile := musicFiles[0]
|
||||
firstExt := strings.ToLower(stdpath.Ext(firstFile))
|
||||
var tag *MusicTag
|
||||
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
if reader := FetchFileReader(readCtx, firstFile); reader != nil {
|
||||
switch firstExt {
|
||||
case ".flac":
|
||||
tag, _ = ParseFLACVorbisComment(reader)
|
||||
default:
|
||||
tag, _ = ParseID3v2(reader)
|
||||
}
|
||||
_ = reader.Close()
|
||||
}
|
||||
readCancel()
|
||||
|
||||
// 解析标签（用文件夹名作为默认专辑名）
|
||||
albumName := name
|
||||
albumArtist := ""
|
||||
cover := ""
|
||||
releaseDate := ""
|
||||
genre := ""
|
||||
authors := ""
|
||||
if tag != nil {
|
||||
if tag.Album != "" {
|
||||
albumName = tag.Album
|
||||
}
|
||||
albumArtist = tag.AlbumArtist
|
||||
if albumArtist == "" {
|
||||
albumArtist = tag.Artist
|
||||
}
|
||||
if tag.Year != "" && len(tag.Year) >= 4 {
|
||||
releaseDate = tag.Year[:4] + "-01-01"
|
||||
}
|
||||
if tag.Genre != "" {
|
||||
genre = tag.Genre
|
||||
}
|
||||
if tag.Artist != "" {
|
||||
if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil {
|
||||
authors = string(authorsJSON)
|
||||
}
|
||||
}
|
||||
if len(tag.CoverData) > 0 {
|
||||
mimeType := tag.CoverMIME
|
||||
if mimeType == "" {
|
||||
mimeType = "image/jpeg"
|
||||
}
|
||||
cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
|
||||
}
|
||||
}
|
||||
|
||||
episodesJSON := ""
|
||||
if b, err := json.Marshal(episodes); err == nil {
|
||||
episodesJSON = string(b)
|
||||
}
|
||||
|
||||
// 加载歌词：同目录同名 .lrc 优先，其次内嵌歌词
|
||||
lyrics := loadLyricsForMusic(ctx, firstFile, tag)
|
||||
|
||||
item := &model.MediaItem{
|
||||
MediaType: sp.MediaType,
|
||||
ScanPathID: sp.ID,
|
||||
FileName: name, // 文件夹名
|
||||
FolderPath: stdpath.Dir(target), // 文件夹所在的真实父目录
|
||||
IsFolder: true,
|
||||
AlbumName: albumName,
|
||||
AlbumArtist: albumArtist,
|
||||
ScrapedName: albumName,
|
||||
Cover: cover,
|
||||
ReleaseDate: releaseDate,
|
||||
Genre: genre,
|
||||
Authors: authors,
|
||||
Lyrics: lyrics,
|
||||
Episodes: episodesJSON,
|
||||
}
|
||||
|
||||
if err := db.CreateOrUpdateMediaItem(item); err != nil {
|
||||
log.Warnf("save music folder item error [%s]: %v", target, err)
|
||||
}
|
||||
} else {
|
||||
// ---- 普通文件：每首歌独立一条记录 ----
|
||||
folderPath := stdpath.Dir(target)
|
||||
ext := strings.ToLower(stdpath.Ext(name))
|
||||
|
||||
var tag *MusicTag
|
||||
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
if reader := FetchFileReader(readCtx, target); reader != nil {
|
||||
switch ext {
|
||||
case ".flac":
|
||||
tag, _ = ParseFLACVorbisComment(reader)
|
||||
default:
|
||||
tag, _ = ParseID3v2(reader)
|
||||
}
|
||||
_ = reader.Close()
|
||||
}
|
||||
readCancel()
|
||||
|
||||
albumName := ""
|
||||
albumArtist := ""
|
||||
trackTitle := strings.TrimSuffix(name, stdpath.Ext(name))
|
||||
trackNumber := 0
|
||||
cover := ""
|
||||
releaseDate := ""
|
||||
genre := ""
|
||||
authors := ""
|
||||
|
||||
if tag != nil {
|
||||
albumName = tag.Album
|
||||
albumArtist = tag.AlbumArtist
|
||||
if albumArtist == "" {
|
||||
albumArtist = tag.Artist
|
||||
}
|
||||
if tag.Title != "" {
|
||||
trackTitle = tag.Title
|
||||
}
|
||||
trackNumber = tag.TrackNumber
|
||||
if tag.Year != "" && len(tag.Year) >= 4 {
|
||||
releaseDate = tag.Year[:4] + "-01-01"
|
||||
}
|
||||
if tag.Genre != "" {
|
||||
genre = tag.Genre
|
||||
}
|
||||
if tag.Artist != "" {
|
||||
if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil {
|
||||
authors = string(authorsJSON)
|
||||
}
|
||||
}
|
||||
if len(tag.CoverData) > 0 {
|
||||
mimeType := tag.CoverMIME
|
||||
if mimeType == "" {
|
||||
mimeType = "image/jpeg"
|
||||
}
|
||||
cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
|
||||
}
|
||||
}
|
||||
|
||||
// 加载歌词：同目录同名 .lrc 优先，其次内嵌歌词
|
||||
lyrics := loadLyricsForMusic(ctx, target, tag)
|
||||
|
||||
item := &model.MediaItem{
|
||||
MediaType: sp.MediaType,
|
||||
ScanPathID: sp.ID,
|
||||
FileName: name,
|
||||
FolderPath: folderPath,
|
||||
IsFolder: false,
|
||||
FileSize: obj.GetSize(),
|
||||
MimeType: mime.TypeByExtension(ext),
|
||||
AlbumName: albumName,
|
||||
AlbumArtist: albumArtist,
|
||||
ScrapedName: trackTitle,
|
||||
Cover: cover,
|
||||
ReleaseDate: releaseDate,
|
||||
Genre: genre,
|
||||
Authors: authors,
|
||||
TrackNumber: trackNumber,
|
||||
Lyrics: lyrics,
|
||||
}
|
||||
|
||||
if err := db.CreateOrUpdateMediaItem(item); err != nil {
|
||||
log.Warnf("save music item error [%s]: %v", target, err)
|
||||
p.mu.Lock()
|
||||
p.Skipped++
|
||||
p.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
p.Done++
|
||||
p.Message = "已扫描: " + name
|
||||
p.mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// loadLyricsForMusic 为音乐文件加载歌词：
|
||||
// 1. 优先读取同目录同名 .lrc 文件（如 song.mp3 → song.lrc）
|
||||
// 2. 如果不存在，使用 tag 中的内嵌歌词（USLT / Vorbis LYRICS）
|
||||
//
|
||||
// musicVfsPath 为音乐文件的 VFS 全路径；tag 可为 nil
|
||||
func loadLyricsForMusic(ctx context.Context, musicVfsPath string, tag *MusicTag) string {
|
||||
// 尝试同目录同名 .lrc
|
||||
ext := stdpath.Ext(musicVfsPath)
|
||||
if ext != "" {
|
||||
lrcPath := strings.TrimSuffix(musicVfsPath, ext) + ".lrc"
|
||||
lrcCtx, lrcCancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
reader := FetchFileReader(lrcCtx, lrcPath)
|
||||
if reader != nil {
|
||||
// 限制读取上限 1MB，避免异常文件压垮内存
|
||||
data, err := io.ReadAll(io.LimitReader(reader, 1024*1024))
|
||||
_ = reader.Close()
|
||||
lrcCancel()
|
||||
if err == nil && len(data) > 0 {
|
||||
// 去掉可能的 UTF-8 BOM
|
||||
if len(data) >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
|
||||
data = data[3:]
|
||||
}
|
||||
s := strings.TrimSpace(string(data))
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
} else {
|
||||
lrcCancel()
|
||||
}
|
||||
}
|
||||
// 回退到内嵌歌词
|
||||
if tag != nil && tag.Lyrics != "" {
|
||||
return tag.Lyrics
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// FetchFileReader 通过 VFS 路径获取文件内容流（用于刮削器读取文件内容）
|
||||
// 优先使用 RangeReader 直接读取（本地存储无需 HTTP），失败时回退到 HTTP URL
|
||||
// 返回 nil 表示无法获取（不影响主流程）
|
||||
func FetchFileReader(ctx context.Context, vfsPath string) io.ReadCloser {
|
||||
link, file, err := fs.Link(ctx, vfsPath, model.LinkArgs{})
|
||||
if err != nil || link == nil {
|
||||
return nil
|
||||
}
|
||||
size := link.ContentLength
|
||||
if size <= 0 {
|
||||
size = file.GetSize()
|
||||
}
|
||||
rr, err := stream.GetRangeReaderFromLink(size, link)
|
||||
if err != nil {
|
||||
_ = link.Close()
|
||||
return nil
|
||||
}
|
||||
rc, err := rr.RangeRead(ctx, http_range.Range{Length: size})
|
||||
if err != nil {
|
||||
_ = link.Close()
|
||||
return nil
|
||||
}
|
||||
link.SyncClosers.Add(rc)
|
||||
return utils.ReadCloser{Reader: rc, Closer: link}
|
||||
}
|
||||
|
||||
// walkVFS 递归遍历 VFS 路径，收集匹配的媒体文件路径（每个目录都刷新缓存）
|
||||
func walkVFS(ctx context.Context, dirPath string, mediaType model.MediaType, targets *[]string) error {
|
||||
entries, err := fs.List(ctx, dirPath, &fs.ListArgs{NoLog: true, Refresh: true})
|
||||
if err != nil {
|
||||
log.Warnf("media scan: list vfs path [%s] error: %v", dirPath, err)
|
||||
return nil // 跳过无权限目录，不中断整体扫描
|
||||
}
|
||||
for _, e := range entries {
|
||||
childPath := stdpath.Join(dirPath, e.GetName())
|
||||
if e.IsDir() {
|
||||
if err := walkVFS(ctx, childPath, mediaType, targets); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if isMediaFile(e.GetName(), mediaType) {
|
||||
*targets = append(*targets, childPath)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// buildMediaItemFromVFS 根据 VFS 路径构建 MediaItem（非音乐类型使用）
|
||||
// 存储规则：
|
||||
// - folder_path：文件/文件夹所在的真实父目录（VFS 父路径），用于前端按 folder_path + file_name 反查真实文件
|
||||
// - file_name：文件夹就是文件夹名，文件就是文件名
|
||||
// - episodes：文件夹时存里面每个文件的信息，文件时为空
|
||||
func buildMediaItemFromVFS(ctx context.Context, vfsPath string, sp *model.MediaScanPath) (*model.MediaItem, error) {
|
||||
obj, err := fs.Get(ctx, vfsPath, &fs.GetArgs{NoLog: true})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
name := obj.GetName()
|
||||
ext := strings.ToLower(stdpath.Ext(name))
|
||||
|
||||
item := &model.MediaItem{
|
||||
MediaType: sp.MediaType,
|
||||
ScanPathID: sp.ID,
|
||||
FileName: name, // 文件夹名 或 文件名
|
||||
FolderPath: stdpath.Dir(vfsPath), // 真实父目录
|
||||
IsFolder: obj.IsDir(),
|
||||
ScrapedName: strings.TrimSuffix(name, stdpath.Ext(name)), // 去掉扩展名作为默认名称
|
||||
}
|
||||
|
||||
if !obj.IsDir() {
|
||||
// 普通文件：记录大小和 MIME 类型，episodes 为空
|
||||
item.FileSize = obj.GetSize()
|
||||
item.MimeType = mime.TypeByExtension(ext)
|
||||
}
|
||||
|
||||
return item, nil
|
||||
}
|
||||
|
||||
// episodeNumRe 匹配文件名开头的数字序号，支持 "1、" "2." "3-" "4 " 等分隔符
|
||||
var episodeNumRe = regexp.MustCompile(`^(\d+)[、.\-\s_]+(.*)`)
|
||||
|
||||
// EpisodeInfo 选集信息（定义在 model 包中，供协议层复用）
|
||||
type EpisodeInfo = model.EpisodeInfo
|
||||
|
||||
// buildEpisodesFromFolder 扫描文件夹内的媒体文件，构建选集信息 JSON 字符串
|
||||
func buildEpisodesFromFolder(ctx context.Context, folderPath string, mediaType model.MediaType) (string, error) {
|
||||
entries, err := fs.List(ctx, folderPath, &fs.ListArgs{NoLog: true})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var episodes []EpisodeInfo
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.GetName()
|
||||
if !isMediaFile(name, mediaType) {
|
||||
continue
|
||||
}
|
||||
|
||||
// 去掉扩展名得到裸文件名
|
||||
ext := stdpath.Ext(name)
|
||||
baseName := strings.TrimSuffix(name, ext)
|
||||
|
||||
ep := EpisodeInfo{
|
||||
FileName: name,
|
||||
Index: 0,
|
||||
Title: baseName,
|
||||
}
|
||||
|
||||
// 尝试从文件名开头提取数字序号
|
||||
if m := episodeNumRe.FindStringSubmatch(baseName); len(m) == 3 {
|
||||
if idx := parseLeadingInt(m[1]); idx > 0 {
|
||||
ep.Index = idx
|
||||
ep.Title = strings.TrimSpace(m[2])
|
||||
}
|
||||
} else {
|
||||
// 文件名直接以纯数字开头（无分隔符），也尝试提取
|
||||
if idx, rest := splitLeadingNumber(baseName); idx > 0 {
|
||||
ep.Index = idx
|
||||
ep.Title = strings.TrimSpace(rest)
|
||||
}
|
||||
}
|
||||
|
||||
episodes = append(episodes, ep)
|
||||
}
|
||||
|
||||
if len(episodes) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
b, err := json.Marshal(episodes)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
// parseLeadingInt 将纯数字字符串解析为 int，失败返回 0
|
||||
func parseLeadingInt(s string) int {
|
||||
v := 0
|
||||
for _, c := range s {
|
||||
if c < '0' || c > '9' {
|
||||
return 0
|
||||
}
|
||||
v = v*10 + int(c-'0')
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// splitLeadingNumber 从字符串开头提取连续数字，返回 (数字值, 剩余字符串)
|
||||
// 仅当开头确实有数字时才返回非零值
|
||||
func splitLeadingNumber(s string) (int, string) {
|
||||
i := 0
|
||||
for i < len(s) && s[i] >= '0' && s[i] <= '9' {
|
||||
i++
|
||||
}
|
||||
if i == 0 || i == len(s) {
|
||||
// 没有数字，或者整个字符串都是数字（没有标题部分）
|
||||
return 0, s
|
||||
}
|
||||
// 剩余部分必须以非字母数字字符开头，避免把 "1080p" 之类的误识别
|
||||
if unicode.IsLetter(rune(s[i])) || unicode.IsDigit(rune(s[i])) {
|
||||
return 0, s
|
||||
}
|
||||
v := parseLeadingInt(s[:i])
|
||||
return v, s[i:]
|
||||
}
|
||||
|
||||
// isMediaFile 判断文件名是否为指定媒体类型（按扩展名判断）
|
||||
func isMediaFile(name string, mediaType model.MediaType) bool {
|
||||
ext := strings.ToLower(stdpath.Ext(name))
|
||||
switch mediaType {
|
||||
case model.MediaTypeVideo:
|
||||
return videoExts[ext]
|
||||
case model.MediaTypeMusic:
|
||||
return musicExts[ext]
|
||||
case model.MediaTypeImage:
|
||||
return imageExts[ext]
|
||||
case model.MediaTypeBook:
|
||||
return bookExts[ext]
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// GetSupportedExts 获取指定媒体类型支持的扩展名列表
|
||||
func GetSupportedExts(mediaType model.MediaType) []string {
|
||||
var extMap map[string]bool
|
||||
switch mediaType {
|
||||
case model.MediaTypeVideo:
|
||||
extMap = videoExts
|
||||
case model.MediaTypeMusic:
|
||||
extMap = musicExts
|
||||
case model.MediaTypeImage:
|
||||
extMap = imageExts
|
||||
case model.MediaTypeBook:
|
||||
extMap = bookExts
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
exts := make([]string, 0, len(extMap))
|
||||
for ext := range extMap {
|
||||
exts = append(exts, ext)
|
||||
}
|
||||
return exts
|
||||
}
|
||||
@@ -0,0 +1,389 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"image/jpeg"
|
||||
"io"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/disintegration/imaging"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/imgmime"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
// BookLocalScraper 书籍本地刮削器
|
||||
// 尝试从书籍文件中提取封面图片并以 base64 存入 Cover 字段，或保存到本地文件。
|
||||
// - EPUB：解压 zip，提取 OPF 声明的封面或 cover.jpg/cover.png
|
||||
// - PDF：尝试提取内嵌 JPEG 图片流作为封面
|
||||
// - 其他/失败：降级为文件路径
|
||||
type BookLocalScraper struct {
|
||||
// ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件）
|
||||
ThumbnailMode string
|
||||
// ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail"
|
||||
ThumbnailPath string
|
||||
}
|
||||
|
||||
// NewBookLocalScraper 创建书籍本地刮削器
|
||||
func NewBookLocalScraper() *BookLocalScraper {
|
||||
return &BookLocalScraper{
|
||||
ThumbnailMode: "base64",
|
||||
ThumbnailPath: "/.thumbnail",
|
||||
}
|
||||
}
|
||||
|
||||
// NewBookLocalScraperWithConfig 创建带完整配置的书籍本地刮削器
|
||||
func NewBookLocalScraperWithConfig(thumbnailMode, thumbnailPath string) *BookLocalScraper {
|
||||
if thumbnailMode == "" {
|
||||
thumbnailMode = "base64"
|
||||
}
|
||||
if thumbnailPath == "" {
|
||||
thumbnailPath = "/.thumbnail"
|
||||
}
|
||||
return &BookLocalScraper{
|
||||
ThumbnailMode: thumbnailMode,
|
||||
ThumbnailPath: thumbnailPath,
|
||||
}
|
||||
}
|
||||
|
||||
// ScrapeBookLocal 从书籍文件流中提取封面图片，填充基本信息。
|
||||
// reader 为书籍文件内容流，不能为 nil。
|
||||
// 返回提取到的封面 base64 字符串（data URI），提取失败返回空字符串。
|
||||
// 注意：本函数不会将文件路径作为 cover 降级，cover 为空表示提取失败。
|
||||
func (s *BookLocalScraper) ScrapeBookLocal(item *model.MediaItem, reader io.Reader) error {
|
||||
ext := strings.ToLower(getFileExt(item.FileName))
|
||||
|
||||
// 根据文件类型补充 MimeType
|
||||
if item.MimeType == "" {
|
||||
item.MimeType = mimeTypeByBookExt(ext)
|
||||
}
|
||||
// 去掉扩展名作为展示名
|
||||
if item.ScrapedName == "" {
|
||||
item.ScrapedName = trimExt(item.FileName)
|
||||
}
|
||||
|
||||
// 从文件流提取封面，提取失败则 cover 保持为空
|
||||
if reader != nil {
|
||||
data, err := readAllLimited(reader, MaxBookFileBytes)
|
||||
if err == nil && len(data) > 0 {
|
||||
var coverData []byte
|
||||
switch ext {
|
||||
case ".epub":
|
||||
coverData = extractEpubCoverData(data)
|
||||
case ".pdf":
|
||||
coverData = extractPDFCoverData(data)
|
||||
}
|
||||
if len(coverData) > 0 {
|
||||
cover := s.storeCover(strings.TrimSuffix(item.FolderPath, "/")+"/"+item.FileName, coverData)
|
||||
if cover != "" {
|
||||
item.Cover = cover
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
|
||||
// ExtractLocalCover 从书籍文件流中提取封面并根据配置存储（供外部调用）。
|
||||
// 提取失败返回空字符串，不降级为文件路径。
|
||||
func (s *BookLocalScraper) ExtractLocalCover(fileName string, filePath string, reader io.Reader) string {
|
||||
if reader == nil {
|
||||
return ""
|
||||
}
|
||||
ext := strings.ToLower(getFileExt(fileName))
|
||||
data, err := readAllLimited(reader, MaxBookFileBytes)
|
||||
if err != nil || len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
var coverData []byte
|
||||
switch ext {
|
||||
case ".epub":
|
||||
coverData = extractEpubCoverData(data)
|
||||
case ".pdf":
|
||||
coverData = extractPDFCoverData(data)
|
||||
}
|
||||
if len(coverData) == 0 {
|
||||
return ""
|
||||
}
|
||||
return s.storeCover(filePath, coverData)
|
||||
}
|
||||
|
||||
// storeCover 根据 ThumbnailMode 存储封面数据，返回 Cover 字段值
|
||||
func (s *BookLocalScraper) storeCover(filePath string, coverData []byte) string {
|
||||
if NormalizeThumbMode(s.ThumbnailMode) == ThumbModeBase64 {
|
||||
return imageDataToBase64Thumb(coverData)
|
||||
}
|
||||
thumb := resizeCoverBytes(coverData)
|
||||
if len(thumb) == 0 {
|
||||
thumb = coverData
|
||||
}
|
||||
return SaveThumb(context.Background(), s.ThumbnailMode, s.ThumbnailPath, filePath, "image/jpeg", thumb)
|
||||
}
|
||||
|
||||
// resizeCoverBytes 将封面缩放为 300px 宽，返回 JPEG 字节
|
||||
func resizeCoverBytes(data []byte) []byte {
|
||||
img, err := decodeBounded(data)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
|
||||
return nil
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// ── EPUB 封面提取 ─────────────────────────────────────────────────────────────
|
||||
|
||||
// epubContainer 解析 META-INF/container.xml
|
||||
type epubContainer struct {
|
||||
Rootfiles []struct {
|
||||
FullPath string `xml:"full-path,attr"`
|
||||
} `xml:"rootfiles>rootfile"`
|
||||
}
|
||||
|
||||
// epubOPF 解析 OPF 文件，找封面 item
|
||||
type epubOPF struct {
|
||||
Metadata struct {
|
||||
Metas []struct {
|
||||
Name string `xml:"name,attr"`
|
||||
Content string `xml:"content,attr"`
|
||||
} `xml:"meta"`
|
||||
} `xml:"metadata"`
|
||||
Manifest struct {
|
||||
Items []struct {
|
||||
ID string `xml:"id,attr"`
|
||||
Href string `xml:"href,attr"`
|
||||
MediaType string `xml:"media-type,attr"`
|
||||
} `xml:"item"`
|
||||
} `xml:"manifest"`
|
||||
}
|
||||
|
||||
// extractEpubCoverData 从 EPUB（zip）数据中提取封面图片原始字节
|
||||
func extractEpubCoverData(data []byte) []byte {
|
||||
r, err := zip.NewReader(bytes.NewReader(data), int64(len(data)))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 建立文件名→ZipFile 映射（忽略大小写）
|
||||
fileMap := make(map[string]*zip.File, len(r.File))
|
||||
for _, f := range r.File {
|
||||
fileMap[strings.ToLower(f.Name)] = f
|
||||
}
|
||||
|
||||
// 1. 尝试读取 META-INF/container.xml 找 OPF 路径
|
||||
opfPath := ""
|
||||
if cf, ok := fileMap["meta-inf/container.xml"]; ok {
|
||||
if rc, err := cf.Open(); err == nil {
|
||||
var container epubContainer
|
||||
if xml.NewDecoder(rc).Decode(&container) == nil && len(container.Rootfiles) > 0 {
|
||||
opfPath = container.Rootfiles[0].FullPath
|
||||
}
|
||||
rc.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// OPF 所在目录（用于拼接相对路径）
|
||||
opfDir := ""
|
||||
if idx := strings.LastIndex(opfPath, "/"); idx >= 0 {
|
||||
opfDir = opfPath[:idx+1]
|
||||
}
|
||||
|
||||
// 2. 从 OPF 中找封面 item ID
|
||||
coverHref := ""
|
||||
if opfPath != "" {
|
||||
if of, ok := fileMap[strings.ToLower(opfPath)]; ok {
|
||||
if rc, err := of.Open(); err == nil {
|
||||
var opf epubOPF
|
||||
if xml.NewDecoder(rc).Decode(&opf) == nil {
|
||||
// 先找 <meta name="cover" content="..."/> 指向的 item id
|
||||
coverItemID := ""
|
||||
for _, m := range opf.Metadata.Metas {
|
||||
if strings.EqualFold(m.Name, "cover") && m.Content != "" {
|
||||
coverItemID = m.Content
|
||||
break
|
||||
}
|
||||
}
|
||||
// 在 manifest 中找对应 item
|
||||
for _, item := range opf.Manifest.Items {
|
||||
if coverItemID != "" && item.ID == coverItemID {
|
||||
coverHref = opfDir + item.Href
|
||||
break
|
||||
}
|
||||
// 如果没有指定 cover id，找第一个图片类型的 item
|
||||
if coverItemID == "" && strings.Contains(strings.ToLower(item.MediaType), "image") {
|
||||
// 优先选择 id 或 href 中包含 "cover" 的
|
||||
if strings.Contains(strings.ToLower(item.ID), "cover") ||
|
||||
strings.Contains(strings.ToLower(item.Href), "cover") {
|
||||
coverHref = opfDir + item.Href
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
// 如果还没找到，取第一个图片
|
||||
if coverHref == "" && coverItemID == "" {
|
||||
for _, item := range opf.Manifest.Items {
|
||||
if strings.Contains(strings.ToLower(item.MediaType), "image") {
|
||||
coverHref = opfDir + item.Href
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
rc.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 常见封面文件名候选（按优先级）
|
||||
candidates := []string{}
|
||||
if coverHref != "" {
|
||||
candidates = append(candidates, strings.ToLower(coverHref))
|
||||
}
|
||||
candidates = append(candidates,
|
||||
"cover.jpg", "cover.jpeg", "cover.png",
|
||||
"images/cover.jpg", "images/cover.jpeg", "images/cover.png",
|
||||
"oebps/cover.jpg", "oebps/images/cover.jpg",
|
||||
"oebps/cover.jpeg", "oebps/images/cover.jpeg",
|
||||
)
|
||||
|
||||
for _, name := range candidates {
|
||||
if f, ok := fileMap[name]; ok {
|
||||
if rc, err := f.Open(); err == nil {
|
||||
imgData, readErr := readAllLimited(rc, MaxEmbeddedEntryBytes)
|
||||
rc.Close()
|
||||
if readErr == nil && len(imgData) > 0 {
|
||||
return imgData
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractEpubCoverBase64 从 EPUB（zip）数据中提取封面图片，返回 data URI base64（向后兼容）
|
||||
func extractEpubCoverBase64(data []byte) string {
|
||||
coverData := extractEpubCoverData(data)
|
||||
if len(coverData) == 0 {
|
||||
return ""
|
||||
}
|
||||
return imageDataToBase64Thumb(coverData)
|
||||
}
|
||||
|
||||
// ── PDF 封面提取 ──────────────────────────────────────────────────────────────
|
||||
|
||||
// extractPDFCoverData 从 PDF 数据中提取第一个内嵌 JPEG 图片流，返回原始字节
|
||||
// PDF 内嵌图片通常以 JPEG 流存储（SOI marker: 0xFF 0xD8），直接扫描字节流提取
|
||||
func extractPDFCoverData(data []byte) []byte {
|
||||
// 扫描 JPEG SOI (0xFF 0xD8 0xFF) 标记
|
||||
for i := 0; i < len(data)-2; i++ {
|
||||
if data[i] == 0xFF && data[i+1] == 0xD8 && data[i+2] == 0xFF {
|
||||
// 找对应的 EOI (0xFF 0xD9)
|
||||
for j := i + 2; j < len(data)-1; j++ {
|
||||
if data[j] == 0xFF && data[j+1] == 0xD9 {
|
||||
jpegData := data[i : j+2]
|
||||
// 验证是否为有效图片（至少 1KB，避免误匹配小缩略图）
|
||||
if len(jpegData) > 1024 {
|
||||
return jpegData
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractPDFCoverBase64 从 PDF 数据中提取第一个内嵌 JPEG 图片流，返回 data URI base64（向后兼容）
|
||||
func extractPDFCoverBase64(data []byte) string {
|
||||
coverData := extractPDFCoverData(data)
|
||||
if len(coverData) == 0 {
|
||||
return ""
|
||||
}
|
||||
return imageDataToBase64Thumb(coverData)
|
||||
}
|
||||
|
||||
// ── 公共辅助 ──────────────────────────────────────────────────────────────────
|
||||
|
||||
// imageDataToBase64Thumb 将图片字节解码并缩放为 300px 宽缩略图，返回 data URI base64
|
||||
func imageDataToBase64Thumb(data []byte) string {
|
||||
img, err := decodeBounded(data)
|
||||
if err != nil {
|
||||
// 超限图片不做任何保留；解码失败但魔数是合法栅格格式时，原样 base64（可能是解码器不支持的变体）
|
||||
if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes {
|
||||
return ""
|
||||
}
|
||||
mimeType := imgmime.Sniff(data)
|
||||
if mimeType == "" {
|
||||
return ""
|
||||
}
|
||||
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
|
||||
}
|
||||
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
|
||||
return ""
|
||||
}
|
||||
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
|
||||
}
|
||||
|
||||
// detectImageMime 根据文件头检测图片 MIME 类型
|
||||
func detectImageMime(data []byte) string {
|
||||
if len(data) < 4 {
|
||||
return "image/jpeg"
|
||||
}
|
||||
switch {
|
||||
case data[0] == 0xFF && data[1] == 0xD8:
|
||||
return "image/jpeg"
|
||||
case data[0] == 0x89 && data[1] == 0x50 && data[2] == 0x4E && data[3] == 0x47:
|
||||
return "image/png"
|
||||
case data[0] == 0x47 && data[1] == 0x49 && data[2] == 0x46:
|
||||
return "image/gif"
|
||||
case data[0] == 0x52 && data[1] == 0x49 && data[2] == 0x46 && data[3] == 0x46:
|
||||
return "image/webp"
|
||||
default:
|
||||
return "image/jpeg"
|
||||
}
|
||||
}
|
||||
|
||||
// getFileExt 获取文件扩展名（含点号，小写）
|
||||
func getFileExt(fileName string) string {
|
||||
if idx := strings.LastIndex(fileName, "."); idx >= 0 {
|
||||
return fileName[idx:]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// mimeTypeByBookExt 根据书籍文件扩展名返回 MIME 类型
|
||||
func mimeTypeByBookExt(ext string) string {
|
||||
switch ext {
|
||||
case ".pdf":
|
||||
return "application/pdf"
|
||||
case ".epub":
|
||||
return "application/epub+zip"
|
||||
case ".mobi":
|
||||
return "application/x-mobipocket-ebook"
|
||||
case ".azw3":
|
||||
return "application/vnd.amazon.ebook"
|
||||
case ".txt":
|
||||
return "text/plain"
|
||||
case ".djvu":
|
||||
return "image/vnd.djvu"
|
||||
case ".cbz":
|
||||
return "application/vnd.comicbook+zip"
|
||||
case ".cbr":
|
||||
return "application/vnd.comicbook-rar"
|
||||
default:
|
||||
return "application/octet-stream"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,278 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
// ==================== Google Books ====================
|
||||
|
||||
// GoogleBooksScraper 通过 Google Books API 刮削书籍
|
||||
type GoogleBooksScraper struct {
|
||||
APIKey string
|
||||
ThumbnailMode string
|
||||
ThumbnailPath string
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
func NewGoogleBooksScraper(apiKey, thumbnailMode, thumbnailPath string) *GoogleBooksScraper {
|
||||
if thumbnailMode == "" {
|
||||
thumbnailMode = "base64"
|
||||
}
|
||||
if thumbnailPath == "" {
|
||||
thumbnailPath = "imgs"
|
||||
}
|
||||
return &GoogleBooksScraper{
|
||||
APIKey: apiKey,
|
||||
ThumbnailMode: thumbnailMode,
|
||||
ThumbnailPath: thumbnailPath,
|
||||
client: &http.Client{Timeout: 20 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
type googleBooksResponse struct {
|
||||
Items []struct {
|
||||
ID string `json:"id"`
|
||||
VolumeInfo struct {
|
||||
Title string `json:"title"`
|
||||
Authors []string `json:"authors"`
|
||||
Description string `json:"description"`
|
||||
PublishedDate string `json:"publishedDate"`
|
||||
Publisher string `json:"publisher"`
|
||||
Categories []string `json:"categories"`
|
||||
AverageRating float64 `json:"averageRating"`
|
||||
IndustryIDs []struct {
|
||||
Type string `json:"type"`
|
||||
Identifier string `json:"identifier"`
|
||||
} `json:"industryIdentifiers"`
|
||||
ImageLinks struct {
|
||||
Thumbnail string `json:"thumbnail"`
|
||||
SmallThumbnail string `json:"smallThumbnail"`
|
||||
} `json:"imageLinks"`
|
||||
} `json:"volumeInfo"`
|
||||
} `json:"items"`
|
||||
}
|
||||
|
||||
// ScrapeBook 用 Google Books 填充书籍信息，封面保存为远程链接（由调用方决定是否落盘）
|
||||
func (s *GoogleBooksScraper) ScrapeBook(item *model.MediaItem) error {
|
||||
query := strings.TrimSpace(item.ISBN)
|
||||
if query == "" {
|
||||
query = strings.TrimSpace(item.ScrapedName)
|
||||
}
|
||||
if query == "" && item.FileName != "" {
|
||||
query = trimExt(item.FileName)
|
||||
}
|
||||
if query == "" {
|
||||
return fmt.Errorf("empty query")
|
||||
}
|
||||
api := "https://www.googleapis.com/books/v1/volumes?maxResults=1&printType=books&langRestrict=" +
|
||||
url.QueryEscape(ScrapeLanguagePrimary()) + "&q=" + url.QueryEscape(query)
|
||||
if s.APIKey != "" {
|
||||
api += "&key=" + url.QueryEscape(s.APIKey)
|
||||
}
|
||||
var resp googleBooksResponse
|
||||
if err := s.getJSON(api, &resp); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(resp.Items) == 0 {
|
||||
return fmt.Errorf("google books: no result for %q", query)
|
||||
}
|
||||
it := resp.Items[0]
|
||||
v := it.VolumeInfo
|
||||
|
||||
if item.ScrapedName == "" {
|
||||
item.ScrapedName = v.Title
|
||||
}
|
||||
if item.Plot == "" {
|
||||
item.Plot = v.Description
|
||||
}
|
||||
if item.Description == "" {
|
||||
item.Description = v.Description
|
||||
}
|
||||
if item.Publisher == "" {
|
||||
item.Publisher = v.Publisher
|
||||
}
|
||||
if item.ReleaseDate == "" {
|
||||
item.ReleaseDate = v.PublishedDate
|
||||
}
|
||||
if item.Authors == "" && len(v.Authors) > 0 {
|
||||
item.Authors = toAuthorsJSON(v.Authors)
|
||||
}
|
||||
if item.Genre == "" && len(v.Categories) > 0 {
|
||||
item.Genre = strings.Join(v.Categories, ",")
|
||||
}
|
||||
if item.ISBN == "" {
|
||||
for _, id := range v.IndustryIDs {
|
||||
if id.Type == "ISBN_13" || id.Type == "ISBN_10" {
|
||||
item.ISBN = id.Identifier
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if item.Rating == 0 && v.AverageRating > 0 {
|
||||
item.Rating = float32(v.AverageRating * 2)
|
||||
}
|
||||
if item.ExternalID == "" {
|
||||
item.ExternalID = "google:" + it.ID
|
||||
}
|
||||
if item.Cover == "" {
|
||||
img := v.ImageLinks.Thumbnail
|
||||
if img == "" {
|
||||
img = v.ImageLinks.SmallThumbnail
|
||||
}
|
||||
item.Cover = normalizeGoogleImage(img)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *GoogleBooksScraper) getJSON(api string, out any) error {
|
||||
req, err := http.NewRequest(http.MethodGet, api, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)")
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("google books: unexpected status %d", resp.StatusCode)
|
||||
}
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
|
||||
func normalizeGoogleImage(img string) string {
|
||||
if img == "" {
|
||||
return ""
|
||||
}
|
||||
img = strings.Replace(img, "http://", "https://", 1)
|
||||
if idx := strings.Index(img, "&edge="); idx > 0 {
|
||||
img = img[:idx]
|
||||
}
|
||||
return img
|
||||
}
|
||||
|
||||
// ==================== OpenLibrary ====================
|
||||
|
||||
// OpenLibraryScraper 通过 OpenLibrary Search API 刮削书籍（无需凭据）
|
||||
type OpenLibraryScraper struct {
|
||||
ThumbnailMode string
|
||||
ThumbnailPath string
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
func NewOpenLibraryScraper(thumbnailMode, thumbnailPath string) *OpenLibraryScraper {
|
||||
if thumbnailMode == "" {
|
||||
thumbnailMode = "base64"
|
||||
}
|
||||
if thumbnailPath == "" {
|
||||
thumbnailPath = "imgs"
|
||||
}
|
||||
return &OpenLibraryScraper{
|
||||
ThumbnailMode: thumbnailMode,
|
||||
ThumbnailPath: thumbnailPath,
|
||||
client: &http.Client{Timeout: 20 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
type openLibraryResponse struct {
|
||||
Docs []struct {
|
||||
Key string `json:"key"`
|
||||
Title string `json:"title"`
|
||||
AuthorName []string `json:"author_name"`
|
||||
FirstPublish int `json:"first_publish_year"`
|
||||
Publisher []string `json:"publisher"`
|
||||
ISBN []string `json:"isbn"`
|
||||
CoverID int `json:"cover_i"`
|
||||
Subject []string `json:"subject"`
|
||||
RatingsAverage float64 `json:"ratings_average"`
|
||||
} `json:"docs"`
|
||||
}
|
||||
|
||||
// ScrapeBook 用 OpenLibrary 填充书籍信息，封面保存为远程链接（由调用方决定是否落盘）
|
||||
func (s *OpenLibraryScraper) ScrapeBook(item *model.MediaItem) error {
|
||||
query := strings.TrimSpace(item.ISBN)
|
||||
if query == "" {
|
||||
query = strings.TrimSpace(item.ScrapedName)
|
||||
}
|
||||
if query == "" && item.FileName != "" {
|
||||
query = trimExt(item.FileName)
|
||||
}
|
||||
if query == "" {
|
||||
return fmt.Errorf("empty query")
|
||||
}
|
||||
api := "https://openlibrary.org/search.json?limit=1&fields=key,title,author_name,first_publish_year,publisher,isbn,cover_i,subject,ratings_average&q=" + url.QueryEscape(query)
|
||||
var resp openLibraryResponse
|
||||
if err := s.getJSON(api, &resp); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(resp.Docs) == 0 {
|
||||
return fmt.Errorf("openlibrary: no result for %q", query)
|
||||
}
|
||||
doc := resp.Docs[0]
|
||||
|
||||
if item.ScrapedName == "" {
|
||||
item.ScrapedName = doc.Title
|
||||
}
|
||||
if item.ReleaseDate == "" && doc.FirstPublish > 0 {
|
||||
item.ReleaseDate = fmt.Sprintf("%d", doc.FirstPublish)
|
||||
}
|
||||
if item.Publisher == "" && len(doc.Publisher) > 0 {
|
||||
item.Publisher = doc.Publisher[0]
|
||||
}
|
||||
if item.Authors == "" && len(doc.AuthorName) > 0 {
|
||||
item.Authors = toAuthorsJSON(doc.AuthorName)
|
||||
}
|
||||
if item.Genre == "" && len(doc.Subject) > 0 {
|
||||
limit := len(doc.Subject)
|
||||
if limit > 8 {
|
||||
limit = 8
|
||||
}
|
||||
item.Genre = strings.Join(doc.Subject[:limit], ",")
|
||||
}
|
||||
if item.ISBN == "" && len(doc.ISBN) > 0 {
|
||||
item.ISBN = doc.ISBN[0]
|
||||
}
|
||||
if item.Rating == 0 && doc.RatingsAverage > 0 {
|
||||
item.Rating = float32(doc.RatingsAverage * 2)
|
||||
}
|
||||
if item.ExternalID == "" {
|
||||
item.ExternalID = "openlibrary:" + doc.Key
|
||||
}
|
||||
if item.Cover == "" && doc.CoverID > 0 {
|
||||
item.Cover = fmt.Sprintf("https://covers.openlibrary.org/b/id/%d-L.jpg", doc.CoverID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *OpenLibraryScraper) getJSON(api string, out any) error {
|
||||
req, err := http.NewRequest(http.MethodGet, api, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)")
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("openlibrary: unexpected status %d", resp.StatusCode)
|
||||
}
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
|
||||
func toAuthorsJSON(authors []string) string {
|
||||
b, err := json.Marshal(authors)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
const discogsDefaultBaseURL = "https://api.discogs.com"
|
||||
|
||||
// DiscogsScraper Discogs音乐刮削器
|
||||
type DiscogsScraper struct {
|
||||
Token string
|
||||
BaseURL string // 可自定义的 Discogs API 地址
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
// NewDiscogsScraper 创建Discogs刮削器
|
||||
//
|
||||
// 用户只需填写「域名 + API 前缀」，例如：
|
||||
// 1. 留空：使用默认 https://api.discogs.com
|
||||
// 2. 官方：api.discogs.com -> https://api.discogs.com
|
||||
// 3. 反代：example.edgeone.run/api/discogs -> https://example.edgeone.run/api/discogs
|
||||
// 4. 反代：example.edgeone.run/api/discogs/ -> https://example.edgeone.run/api/discogs（自动去尾斜杠）
|
||||
//
|
||||
// 后续会基于 BaseURL 直接拼接 /database/search、/releases/{id} 等具体路径。
|
||||
func NewDiscogsScraper(token string, baseURL string) *DiscogsScraper {
|
||||
base := strings.TrimRight(strings.TrimSpace(baseURL), "/")
|
||||
if base == "" {
|
||||
base = discogsDefaultBaseURL
|
||||
}
|
||||
// 协议头容错：用户没填 http/https 时自动补 https://
|
||||
if !strings.HasPrefix(base, "http://") && !strings.HasPrefix(base, "https://") {
|
||||
base = "https://" + base
|
||||
}
|
||||
// 再次去掉末尾多余的 /，避免后续拼接出 //database/search 这类双斜杠
|
||||
base = strings.TrimRight(base, "/")
|
||||
return &DiscogsScraper{
|
||||
Token: token,
|
||||
BaseURL: base,
|
||||
client: &http.Client{Timeout: 15 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
type discogsSearchResult struct {
|
||||
Results []struct {
|
||||
ID int `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Type string `json:"type"`
|
||||
Year string `json:"year"`
|
||||
Thumb string `json:"thumb"`
|
||||
CoverImage string `json:"cover_image"`
|
||||
Genre []string `json:"genre"`
|
||||
Style []string `json:"style"`
|
||||
ResourceURL string `json:"resource_url"`
|
||||
} `json:"results"`
|
||||
}
|
||||
|
||||
type discogsReleaseDetail struct {
|
||||
ID int `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year"`
|
||||
CoverImage string `json:"cover_image"`
|
||||
Notes string `json:"notes"`
|
||||
Artists []struct {
|
||||
Name string `json:"name"`
|
||||
} `json:"artists"`
|
||||
Genres []string `json:"genres"`
|
||||
Styles []string `json:"styles"`
|
||||
Tracklist []struct {
|
||||
Position string `json:"position"`
|
||||
Title string `json:"title"`
|
||||
Duration string `json:"duration"`
|
||||
} `json:"tracklist"`
|
||||
Community struct {
|
||||
Rating struct {
|
||||
Average float32 `json:"average"`
|
||||
} `json:"rating"`
|
||||
} `json:"community"`
|
||||
}
|
||||
|
||||
// parseMusicFileName 从音乐文件名中提取搜索关键词列表
|
||||
// 规则：去掉扩展名后，按" - "或"-"分割，返回各部分作为候选搜索词
|
||||
// 例如：
|
||||
//
|
||||
// "周杰伦 - 七里香.mp3" -> ["周杰伦 - 七里香", "周杰伦", "七里香"]
|
||||
// "Coldplay-Yellow.flac" -> ["Coldplay-Yellow", "Coldplay", "Yellow"]
|
||||
// "七里香.mp3" -> ["七里香"]
|
||||
func parseMusicFileName(fileName string) []string {
|
||||
// 去掉扩展名
|
||||
if idx := strings.LastIndex(fileName, "."); idx > 0 {
|
||||
ext := strings.ToLower(fileName[idx:])
|
||||
if len(ext) <= 5 {
|
||||
fileName = fileName[:idx]
|
||||
}
|
||||
}
|
||||
fileName = strings.TrimSpace(fileName)
|
||||
if fileName == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
var queries []string
|
||||
// 完整文件名作为第一候选
|
||||
queries = append(queries, fileName)
|
||||
|
||||
// 按" - "（带空格）分割
|
||||
parts := strings.SplitN(fileName, " - ", 2)
|
||||
if len(parts) == 2 {
|
||||
before := strings.TrimSpace(parts[0])
|
||||
after := strings.TrimSpace(parts[1])
|
||||
if before != "" {
|
||||
queries = append(queries, before)
|
||||
}
|
||||
if after != "" {
|
||||
queries = append(queries, after)
|
||||
}
|
||||
} else {
|
||||
// 按"-"（不带空格）分割
|
||||
parts = strings.SplitN(fileName, "-", 2)
|
||||
if len(parts) == 2 {
|
||||
before := strings.TrimSpace(parts[0])
|
||||
after := strings.TrimSpace(parts[1])
|
||||
if before != "" {
|
||||
queries = append(queries, before)
|
||||
}
|
||||
if after != "" {
|
||||
queries = append(queries, after)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return queries
|
||||
}
|
||||
|
||||
// doDiscogsSearch 执行单次Discogs搜索
|
||||
func (s *DiscogsScraper) doDiscogsSearch(query string) (*discogsSearchResult, error) {
|
||||
searchURL := fmt.Sprintf("%s/database/search?q=%s&type=release&token=%s",
|
||||
s.BaseURL, url.QueryEscape(query), s.Token)
|
||||
|
||||
req, _ := http.NewRequest("GET", searchURL, nil)
|
||||
req.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList")
|
||||
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("Discogs搜索请求失败: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
var result discogsSearchResult
|
||||
if err := json.Unmarshal(body, &result); err != nil {
|
||||
return nil, fmt.Errorf("Discogs搜索结果解析失败: %w", err)
|
||||
}
|
||||
return &result, nil
|
||||
}
|
||||
|
||||
// buildMusicQuery 根据 MediaItem 已有字段拼接 Discogs 搜索词
|
||||
// 优先级：专辑名 > ScrapedName > 文件名解析
|
||||
// 若有歌手或年份信息，则附加到搜索词中以提高精准度
|
||||
func buildMusicQuery(base string, item *model.MediaItem) string {
|
||||
q := base
|
||||
// 附加歌手
|
||||
if item.AlbumArtist != "" {
|
||||
q = item.AlbumArtist + " " + q
|
||||
}
|
||||
// 附加年份（取 ReleaseDate 前4位）
|
||||
if len(item.ReleaseDate) >= 4 {
|
||||
q = q + " " + item.ReleaseDate[:4]
|
||||
}
|
||||
return strings.TrimSpace(q)
|
||||
}
|
||||
|
||||
// ScrapeMusic 刮削音乐/专辑信息
|
||||
func (s *DiscogsScraper) ScrapeMusic(item *model.MediaItem) error {
|
||||
if s.Token == "" {
|
||||
return fmt.Errorf("Discogs Token 未配置")
|
||||
}
|
||||
|
||||
// 构建候选搜索词列表
|
||||
var queries []string
|
||||
if item.AlbumName != "" {
|
||||
queries = append(queries, buildMusicQuery(item.AlbumName, item))
|
||||
// 同时保留不带附加信息的纯专辑名作为降级候选
|
||||
if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 {
|
||||
queries = append(queries, item.AlbumName)
|
||||
}
|
||||
} else if item.ScrapedName != "" {
|
||||
queries = append(queries, buildMusicQuery(item.ScrapedName, item))
|
||||
if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 {
|
||||
queries = append(queries, item.ScrapedName)
|
||||
}
|
||||
} else {
|
||||
// 从文件名解析，按"-"分割分别搜索
|
||||
baseQueries := parseMusicFileName(item.FileName)
|
||||
for _, bq := range baseQueries {
|
||||
enhanced := buildMusicQuery(bq, item)
|
||||
if enhanced != bq {
|
||||
// 先放带附加信息的精准词，再放原始词作为降级
|
||||
queries = append(queries, enhanced)
|
||||
}
|
||||
queries = append(queries, bq)
|
||||
}
|
||||
}
|
||||
|
||||
if len(queries) == 0 {
|
||||
return fmt.Errorf("无法从文件名中提取有效搜索词")
|
||||
}
|
||||
|
||||
// 依次尝试各候选词，找到结果即停止（模糊匹配降级）
|
||||
var searchResult *discogsSearchResult
|
||||
var lastQuery string
|
||||
for _, q := range queries {
|
||||
result, err := s.doDiscogsSearch(q)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lastQuery = q
|
||||
if len(result.Results) > 0 {
|
||||
searchResult = result
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if searchResult == nil || len(searchResult.Results) == 0 {
|
||||
return fmt.Errorf("Discogs未找到匹配结果: %s", lastQuery)
|
||||
}
|
||||
|
||||
first := searchResult.Results[0]
|
||||
|
||||
// 获取详情
|
||||
detailURL := fmt.Sprintf("%s/releases/%d?token=%s", s.BaseURL, first.ID, s.Token)
|
||||
detailReq, _ := http.NewRequest("GET", detailURL, nil)
|
||||
detailReq.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList")
|
||||
|
||||
detailResp, err := s.client.Do(detailReq)
|
||||
if err != nil {
|
||||
return fmt.Errorf("Discogs详情请求失败: %w", err)
|
||||
}
|
||||
defer detailResp.Body.Close()
|
||||
|
||||
detailBody, _ := io.ReadAll(detailResp.Body)
|
||||
var detail discogsReleaseDetail
|
||||
if err := json.Unmarshal(detailBody, &detail); err != nil {
|
||||
return fmt.Errorf("Discogs详情解析失败: %w", err)
|
||||
}
|
||||
|
||||
// 填充字段：已有值的字段不覆盖，仅补充空缺
|
||||
// Discogs 的 title 就是专辑名（不是 "Artist - Album" 格式）
|
||||
if detail.Title != "" {
|
||||
if item.AlbumName == "" {
|
||||
item.AlbumName = detail.Title
|
||||
}
|
||||
if item.ScrapedName == "" {
|
||||
item.ScrapedName = detail.Title
|
||||
}
|
||||
}
|
||||
|
||||
if item.ReleaseDate == "" {
|
||||
if detail.Year > 0 {
|
||||
item.ReleaseDate = fmt.Sprintf("%d-01-01", detail.Year)
|
||||
} else if first.Year != "" {
|
||||
item.ReleaseDate = first.Year + "-01-01"
|
||||
}
|
||||
}
|
||||
|
||||
if item.Rating == 0 {
|
||||
item.Rating = detail.Community.Rating.Average
|
||||
}
|
||||
if item.Plot == "" {
|
||||
item.Plot = detail.Notes
|
||||
}
|
||||
if item.ExternalID == "" {
|
||||
item.ExternalID = fmt.Sprintf("discogs:%d", detail.ID)
|
||||
}
|
||||
|
||||
// 封面（存储 URL，前端直接展示）
|
||||
if item.Cover == "" {
|
||||
if detail.CoverImage != "" {
|
||||
item.Cover = detail.CoverImage
|
||||
} else if first.CoverImage != "" {
|
||||
item.Cover = first.CoverImage
|
||||
} else if first.Thumb != "" {
|
||||
item.Cover = first.Thumb
|
||||
}
|
||||
}
|
||||
|
||||
// 类型
|
||||
if item.Genre == "" {
|
||||
genres := append(detail.Genres, detail.Styles...)
|
||||
item.Genre = strings.Join(genres, ",")
|
||||
}
|
||||
|
||||
// 艺术家：仅在 Authors 和 AlbumArtist 均为空时才填充（ID3已有值则保留）
|
||||
artists := make([]string, 0, len(detail.Artists))
|
||||
for _, a := range detail.Artists {
|
||||
artists = append(artists, a.Name)
|
||||
}
|
||||
if len(artists) > 0 {
|
||||
if item.Authors == "" {
|
||||
authorsJSON, _ := json.Marshal(artists)
|
||||
item.Authors = string(authorsJSON)
|
||||
}
|
||||
if item.AlbumArtist == "" {
|
||||
item.AlbumArtist = artists[0]
|
||||
}
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,497 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image/jpeg"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/disintegration/imaging"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/imgmime"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"golang.org/x/net/html"
|
||||
)
|
||||
|
||||
const (
|
||||
doubanSearchURL = "https://www.douban.com/search"
|
||||
doubanBookCat = "1001"
|
||||
doubanBase = "https://book.douban.com/"
|
||||
|
||||
// maxCoverDownloadBytes 单张远端封面最大读取字节数
|
||||
maxCoverDownloadBytes int64 = 16 << 20
|
||||
)
|
||||
|
||||
var (
|
||||
doubanBookURLPattern = regexp.MustCompile(`.*/subject/(\d+)/?`)
|
||||
doubanDatePattern = regexp.MustCompile(`(\d{4})-(\d+)`)
|
||||
doubanDefaultHeaders = map[string]string{
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Referer": doubanBase,
|
||||
}
|
||||
)
|
||||
|
||||
// DoubanScraper 豆瓣书籍刮削器（移植自 NewDouban.py）
|
||||
type DoubanScraper struct {
|
||||
client *http.Client
|
||||
// ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件）
|
||||
ThumbnailMode string
|
||||
// ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail"
|
||||
ThumbnailPath string
|
||||
}
|
||||
|
||||
// NewDoubanScraper 创建豆瓣刮削器
|
||||
func NewDoubanScraper() *DoubanScraper {
|
||||
return &DoubanScraper{
|
||||
client: &http.Client{Timeout: 20 * time.Second},
|
||||
ThumbnailMode: "base64",
|
||||
ThumbnailPath: "/.thumbnail",
|
||||
}
|
||||
}
|
||||
|
||||
// NewDoubanScraperWithConfig 创建带完整配置的豆瓣刮削器
|
||||
func NewDoubanScraperWithConfig(thumbnailMode, thumbnailPath string) *DoubanScraper {
|
||||
if thumbnailMode == "" {
|
||||
thumbnailMode = "base64"
|
||||
}
|
||||
if thumbnailPath == "" {
|
||||
thumbnailPath = "/.thumbnail"
|
||||
}
|
||||
return &DoubanScraper{
|
||||
client: &http.Client{Timeout: 20 * time.Second},
|
||||
ThumbnailMode: thumbnailMode,
|
||||
ThumbnailPath: thumbnailPath,
|
||||
}
|
||||
}
|
||||
|
||||
// ScrapeBook 刮削书籍信息
|
||||
func (s *DoubanScraper) ScrapeBook(item *model.MediaItem) error {
|
||||
query := item.ScrapedName
|
||||
if query == "" {
|
||||
query = strings.TrimSuffix(item.FileName, strings.ToLower(item.FileName[strings.LastIndex(item.FileName, "."):]))
|
||||
}
|
||||
|
||||
// 搜索书籍URL列表
|
||||
bookURLs, err := s.searchBookURLs(query)
|
||||
if err != nil {
|
||||
return fmt.Errorf("豆瓣搜索失败: %w", err)
|
||||
}
|
||||
if len(bookURLs) == 0 {
|
||||
return fmt.Errorf("豆瓣未找到匹配书籍: %s", query)
|
||||
}
|
||||
|
||||
// 取第一个结果
|
||||
bookDetail, err := s.loadBookDetail(bookURLs[0])
|
||||
if err != nil {
|
||||
return fmt.Errorf("豆瓣获取书籍详情失败: %w", err)
|
||||
}
|
||||
|
||||
// 填充字段（仅填空字段，已有值不覆盖）
|
||||
if item.ScrapedName == "" && bookDetail.Title != "" {
|
||||
item.ScrapedName = bookDetail.Title
|
||||
}
|
||||
if item.Plot == "" {
|
||||
item.Plot = bookDetail.Description
|
||||
}
|
||||
// 仅在豆瓣返回了有效封面图片URL且当前 item 还没有封面时才下载/覆盖
|
||||
if item.Cover == "" && bookDetail.Cover != "" {
|
||||
if cover := s.downloadAndStoreCover(item.FolderPath, bookDetail.Cover); cover != "" {
|
||||
item.Cover = cover
|
||||
} else {
|
||||
// 下载失败时保留原 URL（降级）
|
||||
item.Cover = bookDetail.Cover
|
||||
}
|
||||
}
|
||||
if item.Rating == 0 {
|
||||
item.Rating = bookDetail.Rating
|
||||
}
|
||||
if item.ReleaseDate == "" {
|
||||
item.ReleaseDate = bookDetail.PublishedDate
|
||||
}
|
||||
if item.Publisher == "" {
|
||||
item.Publisher = bookDetail.Publisher
|
||||
}
|
||||
if item.ISBN == "" {
|
||||
item.ISBN = bookDetail.ISBN
|
||||
}
|
||||
if item.ExternalID == "" {
|
||||
item.ExternalID = "douban:" + bookDetail.ID
|
||||
}
|
||||
|
||||
if item.Authors == "" && len(bookDetail.Authors) > 0 {
|
||||
authorsJSON, _ := json.Marshal(bookDetail.Authors)
|
||||
item.Authors = string(authorsJSON)
|
||||
}
|
||||
if item.Genre == "" && len(bookDetail.Tags) > 0 {
|
||||
item.Genre = strings.Join(bookDetail.Tags, ",")
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
|
||||
type doubanBookDetail struct {
|
||||
ID string
|
||||
Title string
|
||||
Authors []string
|
||||
Publisher string
|
||||
PublishedDate string
|
||||
Cover string
|
||||
Rating float32
|
||||
Description string
|
||||
Tags []string
|
||||
ISBN string
|
||||
}
|
||||
|
||||
// searchBookURLs 搜索豆瓣书籍URL列表
|
||||
func (s *DoubanScraper) searchBookURLs(query string) ([]string, error) {
|
||||
searchURL := fmt.Sprintf("%s?cat=%s&q=%s", doubanSearchURL, doubanBookCat, url.QueryEscape(query))
|
||||
body, err := s.doGet(searchURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
doc, err := html.Parse(strings.NewReader(string(body)))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var bookURLs []string
|
||||
var traverse func(*html.Node)
|
||||
traverse = func(n *html.Node) {
|
||||
if n.Type == html.ElementNode && n.Data == "a" {
|
||||
for _, attr := range n.Attr {
|
||||
if attr.Key == "class" && attr.Val == "nbg" {
|
||||
for _, a := range n.Attr {
|
||||
if a.Key == "href" {
|
||||
parsed := s.calcURL(a.Val)
|
||||
if parsed != "" && len(bookURLs) < 5 {
|
||||
bookURLs = append(bookURLs, parsed)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
traverse(c)
|
||||
}
|
||||
}
|
||||
traverse(doc)
|
||||
return bookURLs, nil
|
||||
}
|
||||
|
||||
// calcURL 解析豆瓣搜索结果中的真实URL
|
||||
func (s *DoubanScraper) calcURL(href string) string {
|
||||
parsed, err := url.Parse(href)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
query := parsed.Query()
|
||||
rawURL := query.Get("url")
|
||||
if rawURL == "" {
|
||||
return ""
|
||||
}
|
||||
decoded, err := url.QueryUnescape(rawURL)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
if doubanBookURLPattern.MatchString(decoded) {
|
||||
return decoded
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// loadBookDetail 加载书籍详情页
|
||||
func (s *DoubanScraper) loadBookDetail(bookURL string) (*doubanBookDetail, error) {
|
||||
body, err := s.doGet(bookURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.parseBookHTML(bookURL, string(body))
|
||||
}
|
||||
|
||||
// parseBookHTML 解析书籍HTML（移植自 DoubanBookHtmlParser.parse_book）
|
||||
func (s *DoubanScraper) parseBookHTML(bookURL, content string) (*doubanBookDetail, error) {
|
||||
doc, err := html.Parse(strings.NewReader(content))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
detail := &doubanBookDetail{}
|
||||
|
||||
// 提取豆瓣ID
|
||||
if m := doubanBookURLPattern.FindStringSubmatch(bookURL); len(m) > 1 {
|
||||
detail.ID = m[1]
|
||||
}
|
||||
|
||||
var traverse func(*html.Node)
|
||||
traverse = func(n *html.Node) {
|
||||
if n.Type == html.ElementNode {
|
||||
switch n.Data {
|
||||
case "span":
|
||||
// 标题
|
||||
if getAttr(n, "property") == "v:itemreviewed" {
|
||||
detail.Title = getTextContent(n)
|
||||
}
|
||||
// 元数据字段
|
||||
if getAttr(n, "class") == "pl" {
|
||||
text := getTextContent(n)
|
||||
tail := getNextText(n)
|
||||
switch {
|
||||
case strings.HasPrefix(text, "作者") || strings.HasPrefix(text, "译者"):
|
||||
// 作者通过链接提取
|
||||
collectAuthors(n, &detail.Authors)
|
||||
case strings.HasPrefix(text, "出版社"):
|
||||
detail.Publisher = strings.TrimSpace(tail)
|
||||
case strings.HasPrefix(text, "出版年"):
|
||||
detail.PublishedDate = parseDoubanDate(strings.TrimSpace(tail))
|
||||
case strings.HasPrefix(text, "ISBN"):
|
||||
detail.ISBN = strings.TrimSpace(tail)
|
||||
case strings.HasPrefix(text, "副标题"):
|
||||
if detail.Title != "" {
|
||||
detail.Title += ":" + strings.TrimSpace(tail)
|
||||
}
|
||||
}
|
||||
}
|
||||
// 评分
|
||||
if getAttr(n, "property") == "v:average" {
|
||||
ratingStr := getTextContent(n)
|
||||
var rating float32
|
||||
fmt.Sscanf(ratingStr, "%f", &rating)
|
||||
detail.Rating = rating / 2
|
||||
}
|
||||
case "img":
|
||||
// 封面图片：<img class="nbg" src="..." />
|
||||
if getAttr(n, "class") == "nbg" {
|
||||
src := getAttr(n, "src")
|
||||
if src != "" && !strings.HasSuffix(src, "update_image") {
|
||||
detail.Cover = src
|
||||
}
|
||||
}
|
||||
// 备用：#mainpic 下的 img
|
||||
if getAttr(n, "id") == "mainpic" || (n.Parent != nil && getAttr(n.Parent, "id") == "mainpic") {
|
||||
src := getAttr(n, "src")
|
||||
if src != "" && detail.Cover == "" {
|
||||
detail.Cover = src
|
||||
}
|
||||
}
|
||||
case "a":
|
||||
// 标签（原封面逻辑移除，改为从img获取）
|
||||
cls := getAttr(n, "class")
|
||||
if strings.Contains(cls, "tag") {
|
||||
tag := getTextContent(n)
|
||||
if tag != "" {
|
||||
detail.Tags = append(detail.Tags, tag)
|
||||
}
|
||||
}
|
||||
case "div":
|
||||
// 简介
|
||||
if getAttr(n, "id") == "link-report" {
|
||||
detail.Description = extractIntroText(n)
|
||||
}
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
traverse(c)
|
||||
}
|
||||
}
|
||||
traverse(doc)
|
||||
return detail, nil
|
||||
}
|
||||
|
||||
// HTML辅助函数
|
||||
|
||||
func getAttr(n *html.Node, key string) string {
|
||||
for _, attr := range n.Attr {
|
||||
if attr.Key == key {
|
||||
return attr.Val
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func getTextContent(n *html.Node) string {
|
||||
var sb strings.Builder
|
||||
var walk func(*html.Node)
|
||||
walk = func(node *html.Node) {
|
||||
if node.Type == html.TextNode {
|
||||
sb.WriteString(node.Data)
|
||||
}
|
||||
for c := node.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(n)
|
||||
return strings.TrimSpace(sb.String())
|
||||
}
|
||||
|
||||
func getNextText(n *html.Node) string {
|
||||
if n.NextSibling != nil {
|
||||
if n.NextSibling.Type == html.TextNode {
|
||||
return strings.TrimSpace(n.NextSibling.Data)
|
||||
}
|
||||
return getTextContent(n.NextSibling)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func collectAuthors(n *html.Node, authors *[]string) {
|
||||
var walk func(*html.Node)
|
||||
walk = func(node *html.Node) {
|
||||
if node.Type == html.ElementNode && node.Data == "a" {
|
||||
href := getAttr(node, "href")
|
||||
if strings.Contains(href, "/author") || strings.Contains(href, "/search") {
|
||||
name := getTextContent(node)
|
||||
if name != "" {
|
||||
*authors = append(*authors, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
for c := node.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
// 从父节点开始找
|
||||
if n.Parent != nil {
|
||||
walk(n.Parent)
|
||||
}
|
||||
}
|
||||
|
||||
func extractIntroText(n *html.Node) string {
|
||||
var sb strings.Builder
|
||||
var walk func(*html.Node)
|
||||
walk = func(node *html.Node) {
|
||||
if node.Type == html.ElementNode && node.Data == "div" {
|
||||
if getAttr(node, "class") == "intro" {
|
||||
sb.WriteString(getTextContent(node))
|
||||
return
|
||||
}
|
||||
}
|
||||
for c := node.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(n)
|
||||
return strings.TrimSpace(sb.String())
|
||||
}
|
||||
|
||||
func parseDoubanDate(dateStr string) string {
|
||||
if dateStr == "" {
|
||||
return ""
|
||||
}
|
||||
if m := doubanDatePattern.FindStringSubmatch(dateStr); len(m) > 2 {
|
||||
return fmt.Sprintf("%s-%s-01", m[1], m[2])
|
||||
}
|
||||
return dateStr
|
||||
}
|
||||
|
||||
// downloadAndStoreCover 下载封面图片并根据 ThumbnailMode 存储
|
||||
// 失败时返回空字符串
|
||||
func (s *DoubanScraper) downloadAndStoreCover(filePath, imgURL string) string {
|
||||
data, mimeType := s.downloadImage(imgURL)
|
||||
if len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
if NormalizeThumbMode(s.ThumbnailMode) != ThumbModeBase64 {
|
||||
// 运行目录 / 挂载目录模式
|
||||
localScraper := NewBookLocalScraperWithConfig(s.ThumbnailMode, s.ThumbnailPath)
|
||||
return localScraper.storeCover(filePath, data)
|
||||
}
|
||||
// Base64 模式（默认）
|
||||
img, err := decodeBounded(data)
|
||||
if err != nil {
|
||||
// 解码失败时退回原始字节；超过阈值直接放弃，避免十几 MB 的图被 base64 写进 cover 列
|
||||
if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes {
|
||||
return ""
|
||||
}
|
||||
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
|
||||
}
|
||||
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
|
||||
return ""
|
||||
}
|
||||
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
|
||||
}
|
||||
|
||||
// downloadCoverAsBase64 下载封面图片并转为 data URI Base64 字符串（向后兼容）
|
||||
// 失败时返回空字符串
|
||||
func (s *DoubanScraper) downloadCoverAsBase64(imgURL string) string {
|
||||
data, mimeType := s.downloadImage(imgURL)
|
||||
if len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
img, err := decodeBounded(data)
|
||||
if err != nil {
|
||||
// 解码失败时退回原始字节；超过阈值直接放弃，避免十几 MB 的图被 base64 写进 cover 列
|
||||
if errors.Is(err, ErrTooLarge) || len(data) > MaxRawCoverFallbackBytes {
|
||||
return ""
|
||||
}
|
||||
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
|
||||
}
|
||||
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
|
||||
return ""
|
||||
}
|
||||
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
|
||||
}
|
||||
|
||||
// downloadImage 下载图片，返回原始字节和 MIME 类型
|
||||
func (s *DoubanScraper) downloadImage(imgURL string) ([]byte, string) {
|
||||
req, err := http.NewRequest("GET", imgURL, nil)
|
||||
if err != nil {
|
||||
return nil, ""
|
||||
}
|
||||
for k, v := range doubanDefaultHeaders {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, ""
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return nil, ""
|
||||
}
|
||||
// 远端图片大小不可信，限制读取上限
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, maxCoverDownloadBytes))
|
||||
if err != nil || len(data) == 0 {
|
||||
return nil, ""
|
||||
}
|
||||
// 不信任远端 Content-Type：只接受魔数可识别的栅格图片，防止 SVG/HTML 被当作封面
|
||||
mimeType := imgmime.Sniff(data)
|
||||
if mimeType == "" {
|
||||
return nil, ""
|
||||
}
|
||||
return data, mimeType
|
||||
}
|
||||
|
||||
// doGet 发送GET请求
|
||||
func (s *DoubanScraper) doGet(reqURL string) ([]byte, error) {
|
||||
req, err := http.NewRequest("GET", reqURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for k, v := range doubanDefaultHeaders {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 && resp.StatusCode != 201 {
|
||||
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, reqURL)
|
||||
}
|
||||
return io.ReadAll(resp.Body)
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"image/jpeg"
|
||||
"io"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/disintegration/imaging"
|
||||
"github.com/rwcarlsen/goexif/exif"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
)
|
||||
|
||||
// ImageScraper 图片刮削器
|
||||
// 从图片 EXIF 中提取元数据，并生成缩略图写入 Cover 字段。
|
||||
type ImageScraper struct {
|
||||
// StoreThumbnail 为 true 时生成缩略图写入 Cover；
|
||||
// 为 false 时直接用文件路径作为 Cover，节省数据库空间。
|
||||
StoreThumbnail bool
|
||||
// ThumbnailMode 缩略图存储方式："base64"（默认，存入数据库）或 "local"（存到本地文件）
|
||||
ThumbnailMode string
|
||||
// ThumbnailPath 本地存储路径，ThumbnailMode 为 "local" 时有效，默认 "/.thumbnail"
|
||||
ThumbnailPath string
|
||||
}
|
||||
|
||||
// NewImageScraper 创建图片刮削器
|
||||
// storeThumbnail 为 true 时生成缩略图写入 Cover。
|
||||
func NewImageScraper(storeThumbnail bool) *ImageScraper {
|
||||
return &ImageScraper{
|
||||
StoreThumbnail: storeThumbnail,
|
||||
ThumbnailMode: "base64",
|
||||
ThumbnailPath: "/.thumbnail",
|
||||
}
|
||||
}
|
||||
|
||||
// NewImageScraperWithConfig 创建带完整配置的图片刮削器
|
||||
func NewImageScraperWithConfig(storeThumbnail bool, thumbnailMode, thumbnailPath string) *ImageScraper {
|
||||
if thumbnailMode == "" {
|
||||
thumbnailMode = "base64"
|
||||
}
|
||||
if thumbnailPath == "" {
|
||||
thumbnailPath = "/.thumbnail"
|
||||
}
|
||||
return &ImageScraper{
|
||||
StoreThumbnail: storeThumbnail,
|
||||
ThumbnailMode: thumbnailMode,
|
||||
ThumbnailPath: thumbnailPath,
|
||||
}
|
||||
}
|
||||
|
||||
// ScrapeImage 刮削图片信息
|
||||
// reader 为图片文件内容流，必须传入有效流才能生成缩略图。
|
||||
// 当 reader 不为 nil 时：
|
||||
// - 从 EXIF 读取拍摄时间、GPS 地点、相机型号+参数、评分
|
||||
// - 生成 300px 宽缩略图，根据 ThumbnailMode 存储为 Base64 或本地文件
|
||||
//
|
||||
// 注意：绝不将文件路径作为 cover，reader 为 nil 或缩略图生成失败时 cover 保持为空。
|
||||
func (s *ImageScraper) ScrapeImage(item *model.MediaItem, reader io.Reader) error {
|
||||
// 去掉扩展名作为展示名（如果尚未设置）
|
||||
if item.ScrapedName == "" {
|
||||
item.ScrapedName = trimExt(item.FileName)
|
||||
}
|
||||
|
||||
if reader == nil {
|
||||
// 无文件流时无法生成缩略图，cover 保持为空，等待下次重试
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
|
||||
// 将流读入内存（EXIF 解析和图像解码都需要完整数据）
|
||||
data, err := readAllLimited(reader, MaxImageFileBytes)
|
||||
if err != nil || len(data) == 0 {
|
||||
// 读取失败时 cover 保持为空
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
|
||||
// ── 1. 解析 EXIF ──────────────────────────────────────────────
|
||||
s.parseEXIF(item, bytes.NewReader(data))
|
||||
|
||||
// ── 2. 生成缩略图并按配置存储 ────────────────────────────────
|
||||
if thumb := s.generateThumbBytes(data); len(thumb) > 0 {
|
||||
fullPath := strings.TrimSuffix(item.FolderPath, "/") + "/" + item.FileName
|
||||
item.Cover = SaveThumb(context.Background(), s.ThumbnailMode, s.ThumbnailPath, fullPath, "image/jpeg", thumb)
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
item.ScrapedAt = &now
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseEXIF 从图片数据中解析 EXIF 元数据并写入 item
|
||||
func (s *ImageScraper) parseEXIF(item *model.MediaItem, r io.Reader) {
|
||||
x, err := exif.Decode(r)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
// ── 拍摄时间 → ReleaseDate ────────────────────────────────────
|
||||
if item.ReleaseDate == "" {
|
||||
if t, err := x.DateTime(); err == nil {
|
||||
item.ReleaseDate = t.Format("2006-01-02")
|
||||
}
|
||||
}
|
||||
|
||||
// ── GPS 地点 → Authors（借用作者字段存储地点信息）────────────
|
||||
if item.Authors == "" {
|
||||
if lat, lon, err := x.LatLong(); err == nil {
|
||||
item.Authors = fmt.Sprintf(`["GPS: %.6f, %.6f"]`, lat, lon)
|
||||
}
|
||||
}
|
||||
|
||||
// ── 相机型号 + 参数 → Genre ───────────────────────────────────
|
||||
if item.Genre == "" {
|
||||
var parts []string
|
||||
if make, err := x.Get(exif.Make); err == nil {
|
||||
if v, err := make.StringVal(); err == nil && v != "" {
|
||||
parts = append(parts, strings.TrimSpace(v))
|
||||
}
|
||||
}
|
||||
if model_, err := x.Get(exif.Model); err == nil {
|
||||
if v, err := model_.StringVal(); err == nil && v != "" {
|
||||
parts = append(parts, strings.TrimSpace(v))
|
||||
}
|
||||
}
|
||||
if fnum, err := x.Get(exif.FNumber); err == nil {
|
||||
if num, den, err := fnum.Rat2(0); err == nil && den != 0 {
|
||||
parts = append(parts, fmt.Sprintf("f/%.1f", float64(num)/float64(den)))
|
||||
}
|
||||
}
|
||||
if exp, err := x.Get(exif.ExposureTime); err == nil {
|
||||
if num, den, err := exp.Rat2(0); err == nil && den != 0 {
|
||||
if num == 1 {
|
||||
parts = append(parts, fmt.Sprintf("1/%ds", den))
|
||||
} else {
|
||||
parts = append(parts, fmt.Sprintf("%d/%ds", num, den))
|
||||
}
|
||||
}
|
||||
}
|
||||
if iso, err := x.Get(exif.ISOSpeedRatings); err == nil {
|
||||
if v, err := iso.Int(0); err == nil {
|
||||
parts = append(parts, fmt.Sprintf("ISO%d", v))
|
||||
}
|
||||
}
|
||||
if focal, err := x.Get(exif.FocalLength); err == nil {
|
||||
if num, den, err := focal.Rat2(0); err == nil && den != 0 {
|
||||
parts = append(parts, fmt.Sprintf("%.0fmm", float64(num)/float64(den)))
|
||||
}
|
||||
}
|
||||
if len(parts) > 0 {
|
||||
item.Genre = strings.Join(parts, ",")
|
||||
}
|
||||
}
|
||||
|
||||
// ── EXIF Rating（0-5 星）→ Rating（0-10 分）──────────────────
|
||||
// Windows 资源管理器、Lightroom、Darktable 等软件会写入此标签
|
||||
if item.Rating == 0 {
|
||||
if ratingTag, err := x.Get(exif.FieldName("Rating")); err == nil {
|
||||
if v, err := ratingTag.Int(0); err == nil && v > 0 {
|
||||
// 1-5 星映射到 2-10 分
|
||||
item.Rating = float32(v) * 2
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// generateThumbBytes 将图片数据缩放为 300px 宽缩略图，返回 JPEG 字节
|
||||
func (s *ImageScraper) generateThumbBytes(data []byte) []byte {
|
||||
img, err := decodeBounded(data)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 缩放到宽度 300px，高度等比
|
||||
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 75}); err != nil {
|
||||
return nil
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// trimExt 去掉文件名的扩展名
|
||||
func trimExt(fileName string) string {
|
||||
if idx := strings.LastIndex(fileName, "."); idx > 0 {
|
||||
return fileName[:idx]
|
||||
}
|
||||
return fileName
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
)
|
||||
|
||||
// ScrapeLanguage 返回首选刮削语言（BCP-47，如 zh-CN / en-US），未配置时默认中文。
|
||||
func ScrapeLanguage() string {
|
||||
lang := strings.TrimSpace(setting.GetStr(conf.MediaScrapeLanguage, "zh-CN"))
|
||||
if lang == "" {
|
||||
return "zh-CN"
|
||||
}
|
||||
return lang
|
||||
}
|
||||
|
||||
// ScrapeLanguagePrimary 返回 ISO 639-1 主语言代码（如 zh / en），
|
||||
// 供只接受两位语言码的接口使用（Google Books 的 langRestrict）。
|
||||
func ScrapeLanguagePrimary() string {
|
||||
lang := ScrapeLanguage()
|
||||
if idx := strings.IndexAny(lang, "-_"); idx > 0 {
|
||||
lang = lang[:idx]
|
||||
}
|
||||
return strings.ToLower(lang)
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"image"
|
||||
"io"
|
||||
)
|
||||
|
||||
// 刮削阶段会把文件整体读入内存再解码，必须限制单文件大小与像素总量，
|
||||
// 否则几路并发刮削一张超大图片/压缩炸弹就能把进程内存打满。
|
||||
const (
|
||||
// MaxImageFileBytes 单张待刮削图片的最大字节数
|
||||
MaxImageFileBytes int64 = 32 << 20
|
||||
// MaxBookFileBytes 单本待提取封面的书籍（epub/pdf）最大字节数
|
||||
MaxBookFileBytes int64 = 128 << 20
|
||||
// MaxEmbeddedEntryBytes epub(zip) 内单个封面条目的最大解压字节数，防止 zip 炸弹
|
||||
MaxEmbeddedEntryBytes int64 = 16 << 20
|
||||
// MaxImagePixels 解码前允许的最大像素总量（约 5657x5657）。
|
||||
// 解码后按 RGBA 计算约 128MB，默认 5 路并发刮削最坏约 640MB；
|
||||
// 原来的 64MP 单张就要 256MB，并发时会接近 1.3GB。
|
||||
MaxImagePixels int64 = 32_000_000
|
||||
// MaxRawCoverFallbackBytes 解码失败时退回「原始字节 base64 写库」的上限。
|
||||
// 超过就直接放弃封面——否则单条记录可能写入 16MB 以上的 base64。
|
||||
MaxRawCoverFallbackBytes = 512 << 10
|
||||
)
|
||||
|
||||
// ErrTooLarge 内容超过读取上限
|
||||
var ErrTooLarge = errors.New("scraper: content exceeds size limit")
|
||||
|
||||
// decodeSem 限制同时解码的图片数：解码阶段的内存峰值 ≈ 像素数 × 4，
|
||||
// 只靠单张上限无法约束多路并发叠加后的总量。
|
||||
// 容量必须小于默认刮削并发（5），否则信号量等于不起作用：
|
||||
// 每路最坏 32MP × 4B ≈ 128MB，12 路就是 1.5GB。
|
||||
// 解码是 CPU 密集型操作，并发超过核数也不会更快，取 3 兼顾内存与吞吐。
|
||||
var decodeSem = make(chan struct{}, 3)
|
||||
|
||||
// readAllLimited 最多读取 max 字节；超过上限返回 ErrTooLarge（不会继续读取剩余内容）。
|
||||
func readAllLimited(r io.Reader, max int64) ([]byte, error) {
|
||||
data, err := io.ReadAll(io.LimitReader(r, max+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(data)) > max {
|
||||
return nil, ErrTooLarge
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// decodeBounded 先只读取图片头部获取尺寸，超过像素上限直接拒绝，再做完整解码。
|
||||
func decodeBounded(data []byte) (image.Image, error) {
|
||||
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 用 int64 相乘：32 位平台上 int 溢出会让超限图片绕过检查
|
||||
if cfg.Width <= 0 || cfg.Height <= 0 ||
|
||||
int64(cfg.Width)*int64(cfg.Height) > MaxImagePixels {
|
||||
return nil, ErrTooLarge
|
||||
}
|
||||
decodeSem <- struct{}{}
|
||||
defer func() { <-decodeSem }()
|
||||
img, _, err := image.Decode(bytes.NewReader(data))
|
||||
return img, err
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package scraper
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha1"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
stdpath "path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/sign"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/stream"
|
||||
)
|
||||
|
||||
// 缩略图存储模式
|
||||
const (
|
||||
// ThumbModeBase64 缩略图以 base64 存入数据库
|
||||
ThumbModeBase64 = "base64"
|
||||
// ThumbModeRuntime 缩略图存入程序数据目录（<data>/<thumbnail_path>）
|
||||
ThumbModeRuntime = "runtime"
|
||||
// ThumbModeVFS 缩略图写入挂载目录（VFS 路径）
|
||||
ThumbModeVFS = "local"
|
||||
)
|
||||
|
||||
// NormalizeThumbMode 归一化缩略图模式，未知/空值按运行目录处理
|
||||
func NormalizeThumbMode(mode string) string {
|
||||
switch strings.ToLower(strings.TrimSpace(mode)) {
|
||||
case ThumbModeBase64:
|
||||
return ThumbModeBase64
|
||||
case ThumbModeVFS, "vfs":
|
||||
return ThumbModeVFS
|
||||
default:
|
||||
return ThumbModeRuntime
|
||||
}
|
||||
}
|
||||
|
||||
// RuntimeThumbRoot 返回运行目录模式下缩略图的本地根目录
|
||||
func RuntimeThumbRoot(thumbPath string) string {
|
||||
dir := strings.Trim(strings.TrimSpace(thumbPath), "/")
|
||||
if dir == "" {
|
||||
dir = "imgs"
|
||||
}
|
||||
return filepath.Join(conf.DataDir, filepath.FromSlash(dir))
|
||||
}
|
||||
|
||||
// SaveThumb 按模式保存缩略图数据，返回 cover 字段值。
|
||||
// rawPath 为媒体文件的完整 VFS 路径，用于生成稳定且唯一的文件名。
|
||||
func SaveThumb(ctx context.Context, mode, thumbPath, rawPath, mimeType string, data []byte) string {
|
||||
if len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
mode = NormalizeThumbMode(mode)
|
||||
if mimeType == "" {
|
||||
mimeType = "image/jpeg"
|
||||
}
|
||||
if mode == ThumbModeBase64 {
|
||||
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
|
||||
}
|
||||
|
||||
dir := strings.Trim(strings.TrimSpace(thumbPath), "/")
|
||||
if dir == "" {
|
||||
dir = "imgs"
|
||||
}
|
||||
name := thumbName(rawPath)
|
||||
|
||||
if mode == ThumbModeVFS {
|
||||
vfsDir := "/" + dir
|
||||
vfsFile := stdpath.Join(vfsDir, name)
|
||||
file := &stream.FileStream{
|
||||
Ctx: ctx,
|
||||
Obj: &model.Object{
|
||||
Name: name,
|
||||
Size: int64(len(data)),
|
||||
Modified: time.Now(),
|
||||
},
|
||||
Reader: bytes.NewReader(data),
|
||||
Mimetype: mimeType,
|
||||
}
|
||||
if err := fs.PutDirectly(ctx, vfsDir, file, true); err != nil {
|
||||
return ""
|
||||
}
|
||||
// 带上签名，兼容开启「签名所有」的站点
|
||||
return "/d" + vfsFile + "?sign=" + sign.Sign(vfsFile)
|
||||
}
|
||||
|
||||
// 运行目录模式：<data>/<dir>/<name>
|
||||
root := RuntimeThumbRoot(dir)
|
||||
if err := os.MkdirAll(root, 0755); err != nil {
|
||||
return ""
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, name), data, 0644); err != nil {
|
||||
return ""
|
||||
}
|
||||
return "thumb:/" + stdpath.Join(dir, name)
|
||||
}
|
||||
|
||||
// thumbName 根据媒体文件路径生成缩略图文件名
|
||||
func thumbName(rawPath string) string {
|
||||
sum := sha1.Sum([]byte(rawPath))
|
||||
return hex.EncodeToString(sum[:]) + ".jpg"
|
||||
}
|
||||
|
||||
// DownloadCoverToStorage 下载远程封面并按缩略图配置保存，失败返回空字符串
|
||||
func DownloadCoverToStorage(ctx context.Context, imgURL, rawPath, thumbMode, thumbPath string) string {
|
||||
if imgURL == "" ||
|
||||
!(strings.HasPrefix(imgURL, "http://") || strings.HasPrefix(imgURL, "https://")) {
|
||||
return ""
|
||||
}
|
||||
data, contentType, err := fetchImageBytes(ctx, imgURL)
|
||||
if err != nil {
|
||||
// 部分网络环境无法直连图片域名，回退到公共图片网关
|
||||
if u, parseErr := url.Parse(imgURL); parseErr == nil {
|
||||
gateway := "https://wsrv.nl/?url=" + url.QueryEscape(
|
||||
strings.TrimPrefix(strings.TrimPrefix(u.String(), "https://"), "http://"),
|
||||
)
|
||||
data, contentType, err = fetchImageBytes(ctx, gateway)
|
||||
}
|
||||
}
|
||||
if err != nil || len(data) == 0 {
|
||||
return ""
|
||||
}
|
||||
return SaveThumb(ctx, thumbMode, thumbPath, rawPath, contentType, data)
|
||||
}
|
||||
|
||||
func fetchImageBytes(ctx context.Context, raw string) ([]byte, string, error) {
|
||||
client := &http.Client{Timeout: 20 * time.Second}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, raw, nil)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, "", fmt.Errorf("unexpected status %d", resp.StatusCode)
|
||||
}
|
||||
contentType := resp.Header.Get("Content-Type")
|
||||
if contentType == "" {
|
||||
contentType = "image/jpeg"
|
||||
}
|
||||
if !strings.HasPrefix(contentType, "image/") {
|
||||
return nil, "", fmt.Errorf("unexpected content type %s", contentType)
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20))
|
||||
return data, contentType, err
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,336 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
stdpath "path"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/sign"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/stream"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
log "github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
var apiRootHolder atomic.Value
|
||||
|
||||
// SetAPIRoot 记录本次运行使用的站点地址（由 HTTP 层注入，供异步任务复用）
|
||||
func SetAPIRoot(u string) {
|
||||
if u == "" {
|
||||
return
|
||||
}
|
||||
apiRootHolder.Store(strings.TrimSuffix(u, "/"))
|
||||
}
|
||||
|
||||
// GetAPIRoot 返回当前站点地址
|
||||
func GetAPIRoot() string {
|
||||
if v, ok := apiRootHolder.Load().(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
return strings.TrimSuffix(conf.Conf.SiteURL, "/")
|
||||
}
|
||||
|
||||
// StrmOptions STRM 生成参数
|
||||
type StrmOptions struct {
|
||||
APIRoot string // 站点地址前缀，例如 https://example.com
|
||||
Template string // 内容模板
|
||||
Overwrite bool // 是否覆盖已存在文件
|
||||
SameDir bool // 是否与媒体文件同目录（同名 .strm）
|
||||
OutputPath string // 输出目录（VFS 路径，仅 SameDir=false 时使用）
|
||||
}
|
||||
|
||||
// StrmTarget 一个待生成的 .strm 目标
|
||||
type StrmTarget struct {
|
||||
RelPath string // 相对输出目录的路径
|
||||
RawPath string // 对应媒体文件的 VFS 路径
|
||||
SameDirPath string // 与媒体文件同目录时的 .strm 路径
|
||||
}
|
||||
|
||||
// strmSibling 计算与媒体文件同目录、同名的 .strm 路径
|
||||
func strmSibling(rawPath string, isDir bool) string {
|
||||
if isDir {
|
||||
return stdpath.Join(rawPath, stdpath.Base(rawPath)+".strm")
|
||||
}
|
||||
ext := stdpath.Ext(rawPath)
|
||||
if ext == "" {
|
||||
return rawPath + ".strm"
|
||||
}
|
||||
return strings.TrimSuffix(rawPath, ext) + ".strm"
|
||||
}
|
||||
|
||||
// DefaultStrmOptions 从设置中读取 STRM 参数
|
||||
func DefaultStrmOptions(apiRoot string) StrmOptions {
|
||||
return StrmOptions{
|
||||
APIRoot: strings.TrimSuffix(apiRoot, "/"),
|
||||
Template: setting.GetStr(conf.MediaStrmTemplate, "{site_url}/d{path}?sign={sign}"),
|
||||
Overwrite: setting.GetBool(conf.MediaStrmOverwrite),
|
||||
SameDir: setting.GetBool(conf.MediaStrmSameDir),
|
||||
OutputPath: setting.GetStr(conf.MediaStrmPath, "/strm"),
|
||||
}
|
||||
}
|
||||
|
||||
// RenderStrmContent 渲染 .strm 内容（单行 URL）
|
||||
func RenderStrmContent(tpl, apiRoot, rawPath string) string {
|
||||
if tpl == "" {
|
||||
tpl = "{site_url}/d{path}?sign={sign}"
|
||||
}
|
||||
apiRoot = strings.TrimSuffix(apiRoot, "/")
|
||||
p := rawPath
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
p = "/" + p
|
||||
}
|
||||
replaced := strings.NewReplacer(
|
||||
"{site_url}", apiRoot,
|
||||
"{path}", utils.EncodePath(p),
|
||||
"{sign}", sign.Sign(rawPath),
|
||||
"{raw_path}", p,
|
||||
"{name}", stdpath.Base(p),
|
||||
).Replace(tpl)
|
||||
return strings.TrimSpace(replaced)
|
||||
}
|
||||
|
||||
// sanitizeName 清洗文件/目录名中的非法字符
|
||||
func sanitizeName(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return "Unknown"
|
||||
}
|
||||
replacer := strings.NewReplacer(
|
||||
"/", "_", "\\", "_", ":", "_", "*", "_", "?", "_",
|
||||
"\"", "_", "<", "_", ">", "_", "|", "_",
|
||||
"\n", " ", "\r", " ",
|
||||
)
|
||||
s = replacer.Replace(s)
|
||||
if runes := []rune(s); len(runes) > 120 {
|
||||
s = string(runes[:120])
|
||||
}
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
func itemTitle(item *model.MediaItem) string {
|
||||
if item.ScrapedName != "" {
|
||||
return item.ScrapedName
|
||||
}
|
||||
name := item.FileName
|
||||
if !item.IsFolder {
|
||||
name = strings.TrimSuffix(name, stdpath.Ext(name))
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
func itemYear(item *model.MediaItem) string {
|
||||
if len(item.ReleaseDate) >= 4 {
|
||||
return item.ReleaseDate[:4]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func parseEpisodes(item *model.MediaItem) []model.EpisodeInfo {
|
||||
if item.Episodes == "" {
|
||||
return nil
|
||||
}
|
||||
var eps []model.EpisodeInfo
|
||||
if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil {
|
||||
return nil
|
||||
}
|
||||
return eps
|
||||
}
|
||||
|
||||
// MediaItemFolder 条目所在目录
|
||||
func MediaItemFolder(item *model.MediaItem) string {
|
||||
if item == nil {
|
||||
return "/"
|
||||
}
|
||||
folder := strings.TrimSuffix(item.FolderPath, "/")
|
||||
if folder == "" {
|
||||
folder = "/"
|
||||
}
|
||||
return folder
|
||||
}
|
||||
|
||||
// StrmTargets 计算条目对应的 STRM 目标列表。
|
||||
// 图片/书籍返回空列表（不生成 STRM）。
|
||||
func StrmTargets(item *model.MediaItem) []StrmTarget {
|
||||
if item == nil {
|
||||
return nil
|
||||
}
|
||||
title := sanitizeName(itemTitle(item))
|
||||
year := itemYear(item)
|
||||
folder := MediaItemFolder(item)
|
||||
|
||||
switch item.MediaType {
|
||||
case model.MediaTypeVideo:
|
||||
if item.VideoType == "tv" {
|
||||
series := title
|
||||
season := item.Season
|
||||
if season <= 0 {
|
||||
season = 1
|
||||
}
|
||||
var out []StrmTarget
|
||||
if item.IsFolder {
|
||||
for i, ep := range parseEpisodes(item) {
|
||||
num := ep.Index
|
||||
if num <= 0 {
|
||||
num = i + 1
|
||||
}
|
||||
epTitle := sanitizeName(ep.Title)
|
||||
epRaw := stdpath.Join(folder, item.FileName, ep.FileName)
|
||||
out = append(out, StrmTarget{
|
||||
RelPath: stdpath.Join(series, fmt.Sprintf("Season %02d", season),
|
||||
fmt.Sprintf("%s S%02dE%02d - %s.strm", series, season, num, epTitle)),
|
||||
RawPath: epRaw,
|
||||
SameDirPath: strmSibling(epRaw, false),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
num := item.Episode
|
||||
if num <= 0 {
|
||||
num = 1
|
||||
}
|
||||
raw := stdpath.Join(folder, item.FileName)
|
||||
return []StrmTarget{{
|
||||
RelPath: stdpath.Join(series, fmt.Sprintf("Season %02d", season),
|
||||
fmt.Sprintf("%s S%02dE%02d.strm", series, season, num)),
|
||||
RawPath: raw,
|
||||
SameDirPath: strmSibling(raw, false),
|
||||
}}
|
||||
}
|
||||
name := title
|
||||
if year != "" {
|
||||
name = fmt.Sprintf("%s (%s)", title, year)
|
||||
}
|
||||
raw := stdpath.Join(folder, item.FileName)
|
||||
return []StrmTarget{{
|
||||
RelPath: stdpath.Join(name, name+".strm"),
|
||||
RawPath: raw,
|
||||
SameDirPath: strmSibling(raw, item.IsFolder),
|
||||
}}
|
||||
case model.MediaTypeMusic:
|
||||
artist := sanitizeName(item.AlbumArtist)
|
||||
if artist == "" {
|
||||
artist = "Unknown Artist"
|
||||
}
|
||||
album := sanitizeName(item.AlbumName)
|
||||
if album == "" {
|
||||
album = "Unknown Album"
|
||||
}
|
||||
track := item.TrackNumber
|
||||
if track <= 0 {
|
||||
track = 1
|
||||
}
|
||||
raw := stdpath.Join(folder, item.FileName)
|
||||
return []StrmTarget{{
|
||||
RelPath: stdpath.Join(artist, album, fmt.Sprintf("%02d - %s.strm", track, title)),
|
||||
RawPath: raw,
|
||||
SameDirPath: strmSibling(raw, false),
|
||||
}}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// GenerateStrmForItems 为一组条目生成 STRM 文件，返回写入数量
|
||||
func GenerateStrmForItems(ctx context.Context, items []model.MediaItem, opts StrmOptions) (int, error) {
|
||||
if !opts.SameDir && opts.OutputPath == "" {
|
||||
return 0, fmt.Errorf("strm output path is empty")
|
||||
}
|
||||
// 归一输出根目录，后续用它校验每个目标文件都在根目录之内
|
||||
outRoot := ""
|
||||
if !opts.SameDir {
|
||||
outRoot = stdpath.Clean("/" + strings.TrimSpace(opts.OutputPath))
|
||||
}
|
||||
written := 0
|
||||
for i := range items {
|
||||
item := &items[i]
|
||||
if item.Hidden {
|
||||
continue
|
||||
}
|
||||
for _, target := range StrmTargets(item) {
|
||||
dst := target.SameDirPath
|
||||
if !opts.SameDir {
|
||||
dst = stdpath.Join(outRoot, target.RelPath)
|
||||
// RelPath 含刮削得到的艺术家/专辑名等外部数据，必须保证不会借 ../ 逃出输出目录
|
||||
if dst != outRoot && !strings.HasPrefix(dst, strings.TrimSuffix(outRoot, "/")+"/") {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if dst == "" {
|
||||
continue
|
||||
}
|
||||
if !opts.Overwrite {
|
||||
if _, err := fs.Get(ctx, dst, &fs.GetArgs{NoLog: true}); err == nil {
|
||||
continue
|
||||
}
|
||||
}
|
||||
data := []byte(RenderStrmContent(opts.Template, opts.APIRoot, target.RawPath))
|
||||
file := &stream.FileStream{
|
||||
Ctx: ctx,
|
||||
Obj: &model.Object{
|
||||
Name: stdpath.Base(dst),
|
||||
Size: int64(len(data)),
|
||||
Modified: time.Now(),
|
||||
},
|
||||
Reader: bytes.NewReader(data),
|
||||
Mimetype: "text/plain",
|
||||
}
|
||||
if err := fs.PutDirectly(ctx, stdpath.Dir(dst), file, true); err != nil {
|
||||
return written, fmt.Errorf("write %s: %w", dst, err)
|
||||
}
|
||||
written++
|
||||
}
|
||||
}
|
||||
return written, nil
|
||||
}
|
||||
|
||||
// GenerateStrmForType 按媒体类型（可选扫描路径）批量生成 STRM
|
||||
func GenerateStrmForType(ctx context.Context, mediaType model.MediaType, scanPathID uint, opts StrmOptions) (int, error) {
|
||||
var items []model.MediaItem
|
||||
var err error
|
||||
if scanPathID > 0 {
|
||||
items, err = db.ListMediaItemsByScanPath(scanPathID)
|
||||
} else {
|
||||
items, err = db.ListAllMediaItems(mediaType)
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if mediaType != "" {
|
||||
filtered := make([]model.MediaItem, 0, len(items))
|
||||
for _, it := range items {
|
||||
if it.MediaType == mediaType {
|
||||
filtered = append(filtered, it)
|
||||
}
|
||||
}
|
||||
items = filtered
|
||||
}
|
||||
return GenerateStrmForItems(ctx, items, opts)
|
||||
}
|
||||
|
||||
// TriggerStrmOnScan 扫描结束后按需异步生成 STRM
|
||||
func TriggerStrmOnScan(mediaType model.MediaType, apiRoot string) {
|
||||
if !setting.GetBool(conf.MediaStrmOnScan) {
|
||||
return
|
||||
}
|
||||
opts := DefaultStrmOptions(apiRoot)
|
||||
if !opts.SameDir && opts.OutputPath == "" {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
n, err := GenerateStrmForType(context.Background(), mediaType, 0, opts)
|
||||
if err != nil {
|
||||
log.Errorf("[media] generate strm for %s failed: %+v", mediaType, err)
|
||||
return
|
||||
}
|
||||
log.Infof("[media] generated %d strm files for %s", n, mediaType)
|
||||
}()
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// MediaType 媒体类型，使用字符串便于后期扩展
|
||||
type MediaType string
|
||||
|
||||
const (
|
||||
MediaTypeVideo MediaType = "video"
|
||||
MediaTypeMusic MediaType = "music"
|
||||
MediaTypeImage MediaType = "image"
|
||||
MediaTypeBook MediaType = "book"
|
||||
)
|
||||
|
||||
// MediaScanPath 媒体库扫描路径（每个媒体库可配置多个扫描路径）
|
||||
type MediaScanPath struct {
|
||||
gorm.Model
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
|
||||
|
||||
MediaType MediaType `gorm:"index;not null" json:"media_type"` // 所属媒体类型
|
||||
Name string `json:"name"` // 路径名称（用于前端筛选显示）
|
||||
Path string `gorm:"not null" json:"path"` // VFS 扫描路径
|
||||
PathMerge bool `gorm:"default:false" json:"path_merge"` // 路径合并模式（子文件夹作为一个条目）
|
||||
TypeTag string `json:"type_tag"` // 类型标签：电影、电视剧等
|
||||
ContentTags string `json:"content_tags"` // 内容标签：喜剧、惊悚等（逗号分隔）
|
||||
EnableScrape bool `gorm:"default:true" json:"enable_scrape"` // 是否启用刮削
|
||||
LastScanAt *time.Time `json:"last_scan_at"` // 最后扫描时间
|
||||
}
|
||||
|
||||
// MediaIndexColumnMax 复合唯一索引列（folder_path / file_name / album_name）的最大字符数
|
||||
const MediaIndexColumnMax = 191
|
||||
|
||||
// MediaItem 媒体条目（统一表，通过 media_type 区分类型，便于后期扩展新类型）
|
||||
type MediaItem struct {
|
||||
gorm.Model
|
||||
// 覆盖 gorm.Model 的 ID 字段，使 JSON 序列化为小写 "id"，与前端保持一致
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
|
||||
// 基础信息
|
||||
MediaType MediaType `gorm:"index;not null" json:"media_type"`
|
||||
ScanPathID uint `gorm:"index" json:"scan_path_id"` // 关联的扫描路径ID
|
||||
// 三个字段共同组成复合唯一索引。MySQL 对「有索引且未指定长度」的 string 会生成 longtext，
|
||||
// 而 TEXT 列不能直接建索引（AutoMigrate 会失败）；因此必须显式限定 size。
|
||||
// utf8mb4 下 InnoDB 索引总长上限 3072 字节：191*3*4 = 2292，留足余量。
|
||||
FileName string `gorm:"size:191;uniqueIndex:idx_media_folder_file_album" json:"file_name"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
MimeType string `json:"mime_type"`
|
||||
Hidden bool `gorm:"default:false" json:"hidden"`
|
||||
|
||||
// 刮削/编辑信息
|
||||
ScrapedName string `json:"scraped_name"`
|
||||
Description string `gorm:"type:text" json:"description"`
|
||||
Cover string `json:"cover"` // 封面URL或本地路径
|
||||
ReleaseDate string `json:"release_date"` // 发布时间，格式 YYYY-MM-DD
|
||||
Rating float32 `json:"rating"` // 评分 0-10
|
||||
Genre string `json:"genre"` // 类型，逗号分隔，如 "动作,科幻"
|
||||
Authors string `gorm:"type:text" json:"authors"` // 作者/演员，JSON数组字符串
|
||||
Plot string `gorm:"type:text" json:"plot"` // 剧情/内容介绍
|
||||
Reviews string `gorm:"type:text" json:"reviews"` // 用户评价，JSON数组字符串
|
||||
|
||||
// 外部ID（用于刮削关联）
|
||||
ExternalID string `json:"external_id"` // TMDB ID / Discogs ID / 豆瓣ID
|
||||
|
||||
// 音乐专属字段
|
||||
AlbumName string `gorm:"size:191;uniqueIndex:idx_media_folder_file_album" json:"album_name"` // 所属专辑名
|
||||
AlbumArtist string `json:"album_artist"` // 专辑艺术家
|
||||
TrackNumber int `json:"track_number"` // 曲目编号
|
||||
Duration int `json:"duration"` // 时长（秒）
|
||||
Lyrics string `gorm:"type:text" json:"lyrics"` // LRC格式歌词
|
||||
|
||||
// 视频专属字段
|
||||
VideoType string `json:"video_type"` // "movie" 或 "tv"
|
||||
Season int `json:"season"` // 季（电视剧）
|
||||
Episode int `json:"episode"` // 集（电视剧）
|
||||
|
||||
// 书籍专属字段
|
||||
Publisher string `json:"publisher"` // 出版社
|
||||
ISBN string `json:"isbn"` // ISBN
|
||||
|
||||
// 目录合并模式
|
||||
IsFolder bool `gorm:"default:false" json:"is_folder"` // 是否为文件夹模式条目
|
||||
FolderPath string `gorm:"size:191;index;uniqueIndex:idx_media_folder_file_album" json:"folder_path"` // 扫描根路径（恒定为扫描路径，与file_name+album_name组合唯一）
|
||||
Episodes string `gorm:"type:text" json:"episodes"` // 选集信息，JSON数组，格式：[{file_name,index,title},...]
|
||||
|
||||
ScrapedAt *time.Time `json:"scraped_at"`
|
||||
}
|
||||
|
||||
// MediaConfig 媒体库配置（每种类型一条记录）
|
||||
type MediaConfig struct {
|
||||
gorm.Model
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
|
||||
MediaType MediaType `gorm:"uniqueIndex;not null" json:"media_type"`
|
||||
Enabled bool `gorm:"default:false" json:"enabled"`
|
||||
LastScanAt *time.Time `json:"last_scan_at"`
|
||||
LastScrapeAt *time.Time `json:"last_scrape_at"`
|
||||
// ScraperConfig 该媒体类型独立的刮削配置（JSON）：
|
||||
// store_thumbnail / thumbnail_mode / thumbnail_path，未设置时回落到全局设置
|
||||
ScraperConfig string `gorm:"type:text" json:"scraper_config"`
|
||||
}
|
||||
|
||||
// EpisodeInfo 选集信息（文件夹型媒体的单集描述）
|
||||
type EpisodeInfo struct {
|
||||
FileName string `json:"file_name"` // 原始文件名（含扩展名）
|
||||
Index int `json:"index"` // 序号，默认0，文件名开头有数字则取该数字
|
||||
Title string `json:"title"` // 选集标题（去掉序号后的文件名，不含扩展名）
|
||||
}
|
||||
|
||||
// MediaScanProgress 扫描进度（内存中维护，不持久化）
|
||||
type MediaScanProgress struct {
|
||||
MediaType MediaType `json:"media_type"`
|
||||
ScanPathID uint `json:"scan_path_id,omitempty"` // 0 表示全部路径
|
||||
Running bool `json:"running"`
|
||||
Total int `json:"total"`
|
||||
Done int `json:"done"`
|
||||
Message string `json:"message"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package model
|
||||
|
||||
import "time"
|
||||
|
||||
// MediaAppKey 媒体应用密钥。
|
||||
// 用于无法走交互式登录的客户端（Subsonic 的 token 认证、Emby 的 api_key、
|
||||
// MoviePilot 对接等）。密钥本身以 AES-GCM 加密存储，数据库中不保留明文。
|
||||
type MediaAppKey struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
|
||||
UserID uint `gorm:"index;not null" json:"user_id"`
|
||||
Name string `json:"name"` // 备注名，例如 "Symfonium"、"MoviePilot"
|
||||
Prefix string `gorm:"index" json:"prefix"` // 明文前缀（便于识别/排查）
|
||||
Secret string `gorm:"not null" json:"-"` // AES-GCM 密文（base64）
|
||||
|
||||
LastUsedAt *time.Time `json:"last_used_at"`
|
||||
}
|
||||
|
||||
// MediaUserData 用户对媒体条目的播放状态，Emby 与 Subsonic 共用。
|
||||
type MediaUserData struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
|
||||
UserID uint `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"user_id"`
|
||||
ItemID uint `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"item_id"`
|
||||
EpisodeIndex int `gorm:"uniqueIndex:idx_media_user_item_ep;not null" json:"episode_index"` // 0 表示条目本身，>0 表示选集序号
|
||||
|
||||
Played bool `json:"played"`
|
||||
PlayCount int `json:"play_count"`
|
||||
IsFavorite bool `gorm:"index" json:"is_favorite"`
|
||||
Rating int `json:"rating"` // 0-5，Subsonic 语义
|
||||
PositionTicks int64 `json:"position_ticks"` // 100ns tick，与 Emby 一致
|
||||
LastPlayedAt *time.Time `json:"last_played_at"`
|
||||
}
|
||||
|
||||
// MediaPlaylist 用户播放列表（Subsonic 需要，Emby 的 Playlist 复用同一份数据）
|
||||
type MediaPlaylist struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
|
||||
UserID uint `gorm:"index;not null" json:"user_id"`
|
||||
Name string `json:"name"`
|
||||
Comment string `json:"comment"`
|
||||
Public bool `json:"public"`
|
||||
ItemCount int `json:"item_count"`
|
||||
}
|
||||
|
||||
// MediaPlaylistItem 播放列表条目（按 Order 排序）
|
||||
type MediaPlaylistItem struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
PlaylistID uint `gorm:"index;not null" json:"playlist_id"`
|
||||
ItemID uint `gorm:"index;not null" json:"item_id"`
|
||||
EpisodeIndex int `json:"episode_index"`
|
||||
Order int `gorm:"column:sort_order" json:"order"`
|
||||
}
|
||||
|
||||
// MediaAnnotation 用户在 PDF 阅读器里添加的批注。
|
||||
//
|
||||
// 只存查看器原生的批注对象（Anchor，JSON）：页码、rect/quadPoints、颜色、
|
||||
// 文字、绘制点集都在里面，拆成独立列会出现两份可能不一致的数据。
|
||||
// 重新打开时原样回灌给 PDF 引擎，因此这里不重复建模。
|
||||
type MediaAnnotation struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
|
||||
UserID uint `gorm:"uniqueIndex:idx_media_annotation_key;not null" json:"-"`
|
||||
ItemID uint `gorm:"uniqueIndex:idx_media_annotation_key;not null;index" json:"item_id"`
|
||||
// RemoteID 是查看器分配的批注 id：多设备打开同一份 PDF 时靠它做去重与更新
|
||||
RemoteID string `gorm:"uniqueIndex:idx_media_annotation_key;size:128;not null" json:"remote_id"`
|
||||
Anchor string `gorm:"type:text;not null" json:"anchor"`
|
||||
}
|
||||
|
||||
// EmbyDevice Emby 客户端会话（由 AuthenticateByName 签发）
|
||||
type EmbyDevice struct {
|
||||
ID uint `gorm:"primarykey" json:"id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
|
||||
UserID uint `gorm:"index;not null" json:"user_id"`
|
||||
DeviceID string `json:"device_id"` // 客户端上报的 DeviceId
|
||||
DeviceName string `json:"device_name"`
|
||||
Client string `json:"client"`
|
||||
Version string `json:"version"`
|
||||
TokenHash string `gorm:"uniqueIndex;not null" json:"-"`
|
||||
AppKeyID uint `json:"app_key_id"` // 通过应用密钥认证时为对应 key，否则为 0
|
||||
LastSeenAt time.Time `json:"last_seen_at"`
|
||||
}
|
||||
@@ -13,6 +13,8 @@ const (
|
||||
S3
|
||||
FTP
|
||||
TRAFFIC
|
||||
MEDIA
|
||||
TRANSCODE
|
||||
)
|
||||
|
||||
const (
|
||||
|
||||
@@ -0,0 +1,489 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// SegmentInfo 单个切片元数据
|
||||
type SegmentInfo struct {
|
||||
Seq int `json:"seq"`
|
||||
Duration float64 `json:"duration"`
|
||||
Size int64 `json:"size"`
|
||||
Path string `json:"-"` // 本地实际文件路径
|
||||
}
|
||||
|
||||
// JobCache 单个 Job 的切片缓存
|
||||
type JobCache struct {
|
||||
JobID string
|
||||
Profile string
|
||||
BaseDir string
|
||||
|
||||
mu sync.RWMutex
|
||||
segments map[int]*SegmentInfo
|
||||
final bool // 是否所有切片都到齐
|
||||
waiters map[int][]chan struct{} // seq -> 等待这一片的 channel 列表
|
||||
}
|
||||
|
||||
// Cache 全局切片缓存
|
||||
type Cache struct {
|
||||
rootDir string
|
||||
maxBytes int64
|
||||
mu sync.RWMutex
|
||||
jobs map[string]map[string]*JobCache // jobID -> profile -> cache
|
||||
}
|
||||
|
||||
func NewCache(rootDir string, maxGB int64) *Cache {
|
||||
if maxGB <= 0 {
|
||||
maxGB = 20
|
||||
}
|
||||
return &Cache{
|
||||
rootDir: rootDir,
|
||||
maxBytes: maxGB * 1024 * 1024 * 1024,
|
||||
jobs: make(map[string]map[string]*JobCache),
|
||||
}
|
||||
}
|
||||
|
||||
// Init 创建根目录
|
||||
func (c *Cache) Init() error {
|
||||
return os.MkdirAll(c.rootDir, 0o755)
|
||||
}
|
||||
|
||||
const (
|
||||
// MaxSegmentBytes 单个切片允许写入的最大字节数，防止 callback_token 持有者打满磁盘
|
||||
MaxSegmentBytes int64 = 32 << 20
|
||||
// maxSegmentSeq 切片序号上限（按 2 秒/片约 6 天），拒绝负数与异常大值
|
||||
maxSegmentSeq = 200000
|
||||
)
|
||||
|
||||
var (
|
||||
profileNameRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,32}$`)
|
||||
jobIDRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
|
||||
|
||||
// ErrInvalidProfile profile 名称不合法（可能是路径穿越尝试）
|
||||
ErrInvalidProfile = errors.New("invalid profile name")
|
||||
// ErrInvalidJobID job id 不合法
|
||||
ErrInvalidJobID = errors.New("invalid job id")
|
||||
// ErrInvalidSegment 切片序号越界
|
||||
ErrInvalidSegment = errors.New("invalid segment seq")
|
||||
// ErrSegmentTooLarge 切片超过大小上限
|
||||
ErrSegmentTooLarge = errors.New("segment too large")
|
||||
)
|
||||
|
||||
// ValidProfileName 判断 profile 是否为安全的目录名（与 BuildProfile 产出的 source/1080p/720p 对齐）。
|
||||
func ValidProfileName(profile string) bool {
|
||||
return profileNameRe.MatchString(profile)
|
||||
}
|
||||
|
||||
// safeJoin 在 rootDir 下拼接子路径，并保证结果不会越出 rootDir。
|
||||
func (c *Cache) safeJoin(jobID, profile string) (string, error) {
|
||||
if !jobIDRe.MatchString(jobID) {
|
||||
return "", ErrInvalidJobID
|
||||
}
|
||||
if !ValidProfileName(profile) {
|
||||
return "", ErrInvalidProfile
|
||||
}
|
||||
full := filepath.Join(c.rootDir, jobID, profile)
|
||||
rel, err := filepath.Rel(c.rootDir, full)
|
||||
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return "", ErrInvalidProfile
|
||||
}
|
||||
return full, nil
|
||||
}
|
||||
|
||||
// getOrCreate 获取或创建 (job, profile) 的 cache
|
||||
func (c *Cache) getOrCreate(jobID, profile string) (*JobCache, error) {
|
||||
baseDir, err := c.safeJoin(jobID, profile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
pmap, ok := c.jobs[jobID]
|
||||
if !ok {
|
||||
pmap = make(map[string]*JobCache)
|
||||
c.jobs[jobID] = pmap
|
||||
}
|
||||
jc, ok := pmap[profile]
|
||||
if !ok {
|
||||
jc = &JobCache{
|
||||
JobID: jobID,
|
||||
Profile: profile,
|
||||
BaseDir: baseDir,
|
||||
segments: make(map[int]*SegmentInfo),
|
||||
waiters: make(map[int][]chan struct{}),
|
||||
}
|
||||
if err := os.MkdirAll(jc.BaseDir, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pmap[profile] = jc
|
||||
}
|
||||
return jc, nil
|
||||
}
|
||||
|
||||
// PutSegment 写入一个切片，并通知所有等待者
|
||||
func (c *Cache) PutSegment(jobID, profile string, seq int, duration float64, body io.Reader, isFinal bool) (*SegmentInfo, error) {
|
||||
if seq < 0 || seq > maxSegmentSeq {
|
||||
return nil, ErrInvalidSegment
|
||||
}
|
||||
jc, err := c.getOrCreate(jobID, profile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fname := fmt.Sprintf("seg-%d.ts", seq)
|
||||
fpath := filepath.Join(jc.BaseDir, fname)
|
||||
tmpPath := fpath + ".tmp"
|
||||
f, err := os.Create(tmpPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 多读 1 字节用于判断是否超限
|
||||
n, err := io.Copy(f, io.LimitReader(body, MaxSegmentBytes+1))
|
||||
if err == nil && n > MaxSegmentBytes {
|
||||
err = ErrSegmentTooLarge
|
||||
}
|
||||
cerr := f.Close()
|
||||
if err != nil {
|
||||
os.Remove(tmpPath)
|
||||
return nil, err
|
||||
}
|
||||
if cerr != nil {
|
||||
os.Remove(tmpPath)
|
||||
return nil, cerr
|
||||
}
|
||||
if err := os.Rename(tmpPath, fpath); err != nil {
|
||||
os.Remove(tmpPath)
|
||||
return nil, err
|
||||
}
|
||||
info := &SegmentInfo{Seq: seq, Duration: duration, Size: n, Path: fpath}
|
||||
|
||||
jc.mu.Lock()
|
||||
jc.segments[seq] = info
|
||||
if isFinal {
|
||||
jc.final = true
|
||||
}
|
||||
// 通知所有等待 <= seq 的 waiter
|
||||
var toClose []chan struct{}
|
||||
for s, chs := range jc.waiters {
|
||||
if s <= seq {
|
||||
toClose = append(toClose, chs...)
|
||||
delete(jc.waiters, s)
|
||||
}
|
||||
}
|
||||
jc.mu.Unlock()
|
||||
for _, ch := range toClose {
|
||||
close(ch)
|
||||
}
|
||||
return info, nil
|
||||
}
|
||||
|
||||
// GetSegment 读取一个切片（不等待）
|
||||
func (c *Cache) GetSegment(jobID, profile string, seq int) (*SegmentInfo, error) {
|
||||
c.mu.RLock()
|
||||
pmap, ok := c.jobs[jobID]
|
||||
c.mu.RUnlock()
|
||||
if !ok {
|
||||
return nil, errors.New("job not found")
|
||||
}
|
||||
jc, ok := pmap[profile]
|
||||
if !ok {
|
||||
return nil, errors.New("profile not found")
|
||||
}
|
||||
jc.mu.RLock()
|
||||
defer jc.mu.RUnlock()
|
||||
info, ok := jc.segments[seq]
|
||||
if !ok {
|
||||
return nil, errors.New("segment not found")
|
||||
}
|
||||
return info, nil
|
||||
}
|
||||
|
||||
// WaitForSegment 等待某个 seq 切片就绪，timeout 后返回 nil
|
||||
func (c *Cache) WaitForSegment(jobID, profile string, seq int, timeout time.Duration) (*SegmentInfo, error) {
|
||||
jc, err := c.getOrCreate(jobID, profile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
jc.mu.Lock()
|
||||
if info, ok := jc.segments[seq]; ok {
|
||||
jc.mu.Unlock()
|
||||
return info, nil
|
||||
}
|
||||
if jc.final {
|
||||
jc.mu.Unlock()
|
||||
return nil, errors.New("segment never produced")
|
||||
}
|
||||
ch := make(chan struct{})
|
||||
jc.waiters[seq] = append(jc.waiters[seq], ch)
|
||||
jc.mu.Unlock()
|
||||
select {
|
||||
case <-ch:
|
||||
jc.mu.RLock()
|
||||
info := jc.segments[seq]
|
||||
jc.mu.RUnlock()
|
||||
if info == nil {
|
||||
return nil, errors.New("segment not produced")
|
||||
}
|
||||
return info, nil
|
||||
case <-time.After(timeout):
|
||||
// 【内存泄漏修复】超时后必须从 waiters map 中移除自己的 channel，
|
||||
// 否则 channel 会永远留在 map 中无法被 GC
|
||||
jc.mu.Lock()
|
||||
if chs, ok := jc.waiters[seq]; ok {
|
||||
for i, w := range chs {
|
||||
if w == ch {
|
||||
jc.waiters[seq] = append(chs[:i], chs[i+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(jc.waiters[seq]) == 0 {
|
||||
delete(jc.waiters, seq)
|
||||
}
|
||||
}
|
||||
jc.mu.Unlock()
|
||||
return nil, errors.New("timeout waiting segment")
|
||||
}
|
||||
}
|
||||
|
||||
// SegmentCount 返回 (jobID, profile) 当前已生成的切片数量。
|
||||
// 用于 TCPlaylist 在首次返回前等待至少若干切片就绪，避免 playlist
|
||||
// 中只有 seg-0 导致 HLS.js 反复请求填充缓冲。
|
||||
func (c *Cache) SegmentCount(jobID, profile string) int {
|
||||
c.mu.RLock()
|
||||
pmap, ok := c.jobs[jobID]
|
||||
c.mu.RUnlock()
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
jc, ok := pmap[profile]
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
jc.mu.RLock()
|
||||
defer jc.mu.RUnlock()
|
||||
return len(jc.segments)
|
||||
}
|
||||
|
||||
// BuildPlaylist 生成 HLS playlist (m3u8)
|
||||
// 【重要】当 totalDuration > 0 时，会根据总时长预先生成完整切片列表（VOD 模式），
|
||||
// 即使切片还未生成，URL 也会出现在 playlist 中。HLS.js 请求未生成切片时
|
||||
// 会阻塞等待（见 TCSegment 的 WaitForSegment）。这样可以让 HLS.js 知道
|
||||
// 视频总时长，避免：
|
||||
// 1. 时长显示错误（只显示已转码部分）
|
||||
// 2. 拖动跳跃（playlist 增长导致切片序号重新对齐）
|
||||
func (c *Cache) BuildPlaylist(jobID, profile string, segDur int, completed bool, baseURL string, totalDuration float64) string {
|
||||
c.mu.RLock()
|
||||
pmap, ok := c.jobs[jobID]
|
||||
c.mu.RUnlock()
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
jc, ok := pmap[profile]
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
jc.mu.RLock()
|
||||
defer jc.mu.RUnlock()
|
||||
|
||||
var b strings.Builder
|
||||
b.WriteString("#EXTM3U\n")
|
||||
b.WriteString("#EXT-X-VERSION:3\n")
|
||||
b.WriteString("#EXT-X-TARGETDURATION:" + strconv.Itoa(segDur) + "\n")
|
||||
b.WriteString("#EXT-X-MEDIA-SEQUENCE:0\n")
|
||||
|
||||
// 如果有总时长，按总时长生成完整切片列表
|
||||
if totalDuration > 0 {
|
||||
// VOD 模式：完整列表，HLS.js 知道总时长
|
||||
b.WriteString("#EXT-X-PLAYLIST-TYPE:VOD\n")
|
||||
fSegDur := float64(segDur)
|
||||
// 总切片数 = ceil(总时长 / 每片时长)
|
||||
totalSegs := int(totalDuration / fSegDur)
|
||||
lastDur := totalDuration - float64(totalSegs)*fSegDur
|
||||
if lastDur > 0.01 {
|
||||
totalSegs++
|
||||
}
|
||||
for i := 0; i < totalSegs; i++ {
|
||||
dur := fSegDur
|
||||
if i == totalSegs-1 && lastDur > 0.01 {
|
||||
dur = lastDur
|
||||
}
|
||||
// 优先使用真实时长（如果已知）
|
||||
if seg, ok := jc.segments[i]; ok && seg.Duration > 0 {
|
||||
dur = seg.Duration
|
||||
}
|
||||
b.WriteString(fmt.Sprintf("#EXTINF:%.3f,\n", dur))
|
||||
b.WriteString(fmt.Sprintf("%sseg-%d.ts\n", baseURL, i))
|
||||
}
|
||||
b.WriteString("#EXT-X-ENDLIST\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// fallback: 没有总时长时退回到原 EVENT 模式（保留旧行为）
|
||||
seqs := make([]int, 0, len(jc.segments))
|
||||
for s := range jc.segments {
|
||||
seqs = append(seqs, s)
|
||||
}
|
||||
sort.Ints(seqs)
|
||||
if jc.final || completed {
|
||||
b.WriteString("#EXT-X-PLAYLIST-TYPE:VOD\n")
|
||||
} else {
|
||||
b.WriteString("#EXT-X-PLAYLIST-TYPE:EVENT\n")
|
||||
}
|
||||
for _, s := range seqs {
|
||||
seg := jc.segments[s]
|
||||
b.WriteString(fmt.Sprintf("#EXTINF:%.3f,\n", seg.Duration))
|
||||
b.WriteString(fmt.Sprintf("%sseg-%d.ts\n", baseURL, s))
|
||||
}
|
||||
if jc.final || completed {
|
||||
b.WriteString("#EXT-X-ENDLIST\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// MasterPlaylist 多档位主 m3u8
|
||||
func (c *Cache) MasterPlaylist(profiles []Profile, baseURL string) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("#EXTM3U\n#EXT-X-VERSION:3\n")
|
||||
for _, p := range profiles {
|
||||
bw, _ := parseBitrate(p.VideoBitrate)
|
||||
ab, _ := parseBitrate(p.AudioBitrate)
|
||||
// 【关键修复】根据实际编码动态生成 codec 字符串
|
||||
// 错误的 codec 声明会导致 HLS.js 用错误参数初始化 SourceBuffer
|
||||
// 然后 appendBuffer 时数据格式不匹配 → bufferAppendError 死循环
|
||||
codecs := buildCodecString(p)
|
||||
b.WriteString(fmt.Sprintf("#EXT-X-STREAM-INF:BANDWIDTH=%d,CODECS=\"%s\"\n", bw+ab, codecs))
|
||||
b.WriteString(fmt.Sprintf("%s%s/playlist.m3u8\n", baseURL, p.Name))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// buildCodecString 根据 Profile 生成 HLS CODECS 属性字符串
|
||||
// avc1.42E01E = H.264 Baseline Profile Level 3.0（兼容性最好）
|
||||
// avc1.4D401E = H.264 Main Profile Level 3.0
|
||||
// avc1.640028 = H.264 High Profile Level 4.0
|
||||
// mp4a.40.2 = AAC-LC（HE-AAC v1 用 mp4a.40.5，HE-AAC v2 用 mp4a.40.29）
|
||||
func buildCodecString(p Profile) string {
|
||||
// 视频部分：默认 H.264 Main Profile Level 4.0，平衡兼容性与质量
|
||||
video := "avc1.4D4028"
|
||||
switch strings.ToLower(p.VideoCodec) {
|
||||
case "hevc", "h265":
|
||||
// HEVC Main Profile Level 4.1，部分 Safari/Edge 支持
|
||||
video = "hvc1.1.6.L120.90"
|
||||
case "av1":
|
||||
video = "av01.0.05M.08"
|
||||
}
|
||||
// 音频部分：固定 AAC-LC（后端已强制转 AAC）
|
||||
audio := "mp4a.40.2"
|
||||
return video + "," + audio
|
||||
}
|
||||
|
||||
func parseBitrate(s string) (int64, error) {
|
||||
s = strings.TrimSpace(strings.ToLower(s))
|
||||
if s == "" {
|
||||
return 0, nil
|
||||
}
|
||||
mult := int64(1)
|
||||
if strings.HasSuffix(s, "k") {
|
||||
mult = 1000
|
||||
s = strings.TrimSuffix(s, "k")
|
||||
} else if strings.HasSuffix(s, "m") {
|
||||
mult = 1_000_000
|
||||
s = strings.TrimSuffix(s, "m")
|
||||
}
|
||||
v, err := strconv.ParseInt(s, 10, 64)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return v * mult, nil
|
||||
}
|
||||
|
||||
// Cleanup 删除某个 job 的所有切片，并关闭所有阻塞的 waiter goroutine
|
||||
func (c *Cache) Cleanup(jobID string) {
|
||||
c.mu.Lock()
|
||||
pmap, exists := c.jobs[jobID]
|
||||
delete(c.jobs, jobID)
|
||||
c.mu.Unlock()
|
||||
// 【内存泄漏修复】关闭所有 profile 下阻塞等待的 waiter channel，
|
||||
// 避免 goroutine 永远挂起（每个 goroutine 至少 8KB 栈空间）
|
||||
if exists {
|
||||
for _, jc := range pmap {
|
||||
jc.mu.Lock()
|
||||
jc.final = true // 标记为 final，防止新 waiter 进入
|
||||
for seq, chs := range jc.waiters {
|
||||
for _, ch := range chs {
|
||||
close(ch)
|
||||
}
|
||||
delete(jc.waiters, seq)
|
||||
}
|
||||
jc.mu.Unlock()
|
||||
}
|
||||
}
|
||||
_ = os.RemoveAll(filepath.Join(c.rootDir, jobID))
|
||||
}
|
||||
|
||||
// EnforceLRU 简易 LRU 容量控制：超过 maxBytes 时按目录修改时间从早到晚删除
|
||||
func (c *Cache) EnforceLRU() error {
|
||||
entries, err := os.ReadDir(c.rootDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
type item struct {
|
||||
dir string
|
||||
size int64
|
||||
mtime time.Time
|
||||
}
|
||||
var items []item
|
||||
var total int64
|
||||
for _, e := range entries {
|
||||
// 只管理本模块创建的 job 目录，避免误删缓存根目录下的其它内容
|
||||
if !e.IsDir() || !strings.HasPrefix(e.Name(), "job_") {
|
||||
continue
|
||||
}
|
||||
full := filepath.Join(c.rootDir, e.Name())
|
||||
size := dirSize(full)
|
||||
fi, _ := os.Stat(full)
|
||||
mt := time.Now()
|
||||
if fi != nil {
|
||||
mt = fi.ModTime()
|
||||
}
|
||||
items = append(items, item{full, size, mt})
|
||||
total += size
|
||||
}
|
||||
if total <= c.maxBytes {
|
||||
return nil
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].mtime.Before(items[j].mtime) })
|
||||
for _, it := range items {
|
||||
if total <= c.maxBytes {
|
||||
break
|
||||
}
|
||||
_ = os.RemoveAll(it.dir)
|
||||
total -= it.size
|
||||
// 同步从内存里删
|
||||
jobID := filepath.Base(it.dir)
|
||||
c.mu.Lock()
|
||||
delete(c.jobs, jobID)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func dirSize(dir string) int64 {
|
||||
var size int64
|
||||
_ = filepath.Walk(dir, func(_ string, info os.FileInfo, err error) error {
|
||||
if err == nil && info != nil && !info.IsDir() {
|
||||
size += info.Size()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return size
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestValidProfileName(t *testing.T) {
|
||||
ok := []string{"source", "1080p", "720p", "a_b-c", "A1"}
|
||||
for _, p := range ok {
|
||||
if !ValidProfileName(p) {
|
||||
t.Errorf("%q should be valid", p)
|
||||
}
|
||||
}
|
||||
bad := []string{"", "..", "../x", "a/b", `a\b`, "a b", "a.b", strings.Repeat("a", 33), "%2e%2e"}
|
||||
for _, p := range bad {
|
||||
if ValidProfileName(p) {
|
||||
t.Errorf("%q should be rejected", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPutSegmentRejectsPathTraversal(t *testing.T) {
|
||||
root := filepath.Join(t.TempDir(), "cache")
|
||||
c := NewCache(root, 1)
|
||||
if err := c.Init(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := c.PutSegment("job_x", "../../../outside", 0, 1, bytes.NewReader([]byte("x")), false)
|
||||
if !errors.Is(err, ErrInvalidProfile) {
|
||||
t.Fatalf("want ErrInvalidProfile, got %v", err)
|
||||
}
|
||||
// 目录之外不应出现任何文件
|
||||
parent := filepath.Dir(root)
|
||||
entries, _ := os.ReadDir(parent)
|
||||
for _, e := range entries {
|
||||
if e.Name() != "cache" {
|
||||
t.Fatalf("unexpected path created outside cache root: %s", e.Name())
|
||||
}
|
||||
}
|
||||
if _, err := c.PutSegment("../job", "source", 0, 1, bytes.NewReader([]byte("x")), false); !errors.Is(err, ErrInvalidJobID) {
|
||||
t.Fatalf("want ErrInvalidJobID, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPutSegmentLimits(t *testing.T) {
|
||||
c := NewCache(filepath.Join(t.TempDir(), "cache"), 1)
|
||||
if err := c.Init(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := c.PutSegment("job_x", "source", -1, 1, bytes.NewReader([]byte("x")), false); !errors.Is(err, ErrInvalidSegment) {
|
||||
t.Fatalf("negative seq: want ErrInvalidSegment, got %v", err)
|
||||
}
|
||||
big := bytes.NewReader(make([]byte, MaxSegmentBytes+10))
|
||||
if _, err := c.PutSegment("job_x", "source", 0, 1, big, false); !errors.Is(err, ErrSegmentTooLarge) {
|
||||
t.Fatalf("oversize: want ErrSegmentTooLarge, got %v", err)
|
||||
}
|
||||
// 超限后不应残留 tmp 文件
|
||||
matches, _ := filepath.Glob(filepath.Join(c.rootDir, "job_x", "source", "*"))
|
||||
if len(matches) != 0 {
|
||||
t.Fatalf("leftover files after rejected upload: %v", matches)
|
||||
}
|
||||
if _, err := c.PutSegment("job_x", "source", 0, 1, bytes.NewReader([]byte("ok")), false); err != nil {
|
||||
t.Fatalf("normal upload failed: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,520 @@
|
||||
// Package transcode chunk_manager.go：智能分段转码调度器
|
||||
//
|
||||
// 设计目标：
|
||||
// 1. 把视频按时间切成多个 chunk（默认每 60 秒一段），每个 chunk 独立用 ffmpeg 转码
|
||||
// 2. 用户拖动到任意位置时，按需启动对应 chunk 的 ffmpeg，不再线性等待
|
||||
// 3. 控制单 Job 最多并发 N 个 chunk ffmpeg，超出时 LRU 淘汰最久未访问的 chunk
|
||||
// 4. 当前 chunk 接近末尾时自动预启动下一个 chunk，保证顺序播放无缝衔接
|
||||
// 5. chunk 空闲超过阈值时 kill ffmpeg 释放 CPU，但保留已转出的切片文件
|
||||
//
|
||||
// 与 LocalRunner 的关系：
|
||||
// - LocalRunner.runJob 不再直接启动一个完整的 ffmpeg，而是创建 ChunkSession 并阻塞等待
|
||||
// - ChunkManager 是 ChunkSession 的"大脑"，负责调度
|
||||
// - TCSegment 收到切片请求时，调用 ChunkManager.EnsureChunkRunningForSeg(seq) 触发按需调度
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
)
|
||||
|
||||
// ChunkState chunk 当前状态
|
||||
type ChunkState int
|
||||
|
||||
const (
|
||||
ChunkIdle ChunkState = iota // 未启动
|
||||
ChunkRunning // ffmpeg 正在转码
|
||||
ChunkDone // ffmpeg 自然退出（chunk 内所有切片已转完）
|
||||
ChunkFailed // ffmpeg 异常失败
|
||||
ChunkKilled // 被 LRU 淘汰或空闲超时 kill
|
||||
)
|
||||
|
||||
// Chunk 表示视频中的一个时间段，对应一段连续的切片序号
|
||||
type Chunk struct {
|
||||
Index int // chunk 序号（0 开始）
|
||||
StartTime float64 // 起始秒
|
||||
Duration float64 // 持续时长（最后一个 chunk 可能不足 chunkDur）
|
||||
StartSeq int // 起始切片序号（包含）
|
||||
EndSeq int // 结束切片序号（不包含）
|
||||
State ChunkState
|
||||
StartedAt time.Time
|
||||
LastAccess time.Time
|
||||
Error string
|
||||
|
||||
cancel context.CancelFunc // ffmpeg 进程的 cancel
|
||||
doneCh chan struct{} // ffmpeg 进程结束时关闭
|
||||
doneOnce sync.Once
|
||||
}
|
||||
|
||||
// chunkDoneClose 关闭 doneCh（幂等）
|
||||
func (c *Chunk) chunkDoneClose() {
|
||||
c.doneOnce.Do(func() {
|
||||
if c.doneCh != nil {
|
||||
close(c.doneCh)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ChunkSession 单个 (jobID, profile) 下的 chunk 调度上下文
|
||||
type ChunkSession struct {
|
||||
JobID string
|
||||
Profile string
|
||||
Job *Job
|
||||
ProfDef Profile
|
||||
|
||||
// 静态参数
|
||||
TotalDuration float64 // 视频总时长
|
||||
SegDur int // 每片时长（秒）
|
||||
ChunkDur int // 每个 chunk 时长（秒）
|
||||
MaxConcurrent int // 最大并发 chunk 数
|
||||
IdleSec int // chunk 空闲多久就 kill
|
||||
Prefetch bool // 是否启用预读
|
||||
|
||||
mu sync.Mutex
|
||||
chunks map[int]*Chunk // chunkIndex -> Chunk
|
||||
|
||||
// 给 LocalRunner 用：所有切片转完时关闭，让 runJob 能退出
|
||||
allDoneCh chan struct{}
|
||||
allDoneOnce sync.Once
|
||||
|
||||
// outDir / segPattern (供 ffmpeg 使用)
|
||||
OutDir string
|
||||
}
|
||||
|
||||
// allDoneClose 关闭 allDoneCh（幂等）
|
||||
func (s *ChunkSession) allDoneClose() {
|
||||
s.allDoneOnce.Do(func() {
|
||||
close(s.allDoneCh)
|
||||
})
|
||||
}
|
||||
|
||||
// ChunkManager 全局 chunk 管理器（一个 Manager 一个）
|
||||
type ChunkManager struct {
|
||||
mgr *Manager
|
||||
|
||||
mu sync.Mutex
|
||||
sessions map[string]*ChunkSession // key = jobID + ":" + profile
|
||||
|
||||
stopCh chan struct{}
|
||||
}
|
||||
|
||||
// NewChunkManager 创建实例
|
||||
func NewChunkManager(mgr *Manager) *ChunkManager {
|
||||
return &ChunkManager{
|
||||
mgr: mgr,
|
||||
sessions: make(map[string]*ChunkSession),
|
||||
stopCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
func sessionKey(jobID, profile string) string { return jobID + ":" + profile }
|
||||
|
||||
// StartSession 创建并启动一个 chunk session（由 LocalRunner.runJob 调用）
|
||||
// 返回的 ChunkSession.allDoneCh 在所有切片都转完时关闭
|
||||
func (cm *ChunkManager) StartSession(job *Job, profile Profile, outDir string) (*ChunkSession, error) {
|
||||
totalDur := job.Probe.Duration
|
||||
if totalDur <= 0 {
|
||||
return nil, fmt.Errorf("chunk session requires probe.duration > 0")
|
||||
}
|
||||
segDur := setting.GetInt(conf.TranscodeSegmentDuration, 6)
|
||||
if segDur <= 0 {
|
||||
segDur = 6
|
||||
}
|
||||
chunkDur := setting.GetInt(conf.TranscodeChunkDurationSec, 60)
|
||||
if chunkDur < segDur {
|
||||
chunkDur = segDur * 10
|
||||
}
|
||||
// 让 chunkDur 是 segDur 的整数倍，保证切片对齐
|
||||
if chunkDur%segDur != 0 {
|
||||
chunkDur = (chunkDur / segDur) * segDur
|
||||
if chunkDur < segDur {
|
||||
chunkDur = segDur
|
||||
}
|
||||
}
|
||||
maxConc := setting.GetInt(conf.TranscodeMaxChunkConcurrency, 2)
|
||||
if maxConc <= 0 {
|
||||
maxConc = 2
|
||||
}
|
||||
idleSec := setting.GetInt(conf.TranscodeChunkIdleSec, 60)
|
||||
if idleSec <= 0 {
|
||||
idleSec = 60
|
||||
}
|
||||
prefetch := setting.GetBool(conf.TranscodeChunkPrefetch)
|
||||
|
||||
s := &ChunkSession{
|
||||
JobID: job.ID,
|
||||
Profile: profile.Name,
|
||||
Job: job,
|
||||
ProfDef: profile,
|
||||
TotalDuration: totalDur,
|
||||
SegDur: segDur,
|
||||
ChunkDur: chunkDur,
|
||||
MaxConcurrent: maxConc,
|
||||
IdleSec: idleSec,
|
||||
Prefetch: prefetch,
|
||||
chunks: make(map[int]*Chunk),
|
||||
allDoneCh: make(chan struct{}),
|
||||
OutDir: outDir,
|
||||
}
|
||||
|
||||
// 预先按总时长生成所有 chunk 元数据（不启动）
|
||||
segsPerChunk := chunkDur / segDur
|
||||
totalSegs := int(totalDur) / segDur
|
||||
if int(totalDur)%segDur > 0 {
|
||||
totalSegs++
|
||||
}
|
||||
chunkCount := totalSegs / segsPerChunk
|
||||
if totalSegs%segsPerChunk > 0 {
|
||||
chunkCount++
|
||||
}
|
||||
for i := 0; i < chunkCount; i++ {
|
||||
startSeq := i * segsPerChunk
|
||||
endSeq := startSeq + segsPerChunk
|
||||
if endSeq > totalSegs {
|
||||
endSeq = totalSegs
|
||||
}
|
||||
startT := float64(startSeq * segDur)
|
||||
dur := float64((endSeq - startSeq) * segDur)
|
||||
// 最后一个 chunk 可能不足
|
||||
if i == chunkCount-1 {
|
||||
realDur := totalDur - startT
|
||||
if realDur > 0 && realDur < dur {
|
||||
dur = realDur
|
||||
}
|
||||
}
|
||||
s.chunks[i] = &Chunk{
|
||||
Index: i,
|
||||
StartTime: startT,
|
||||
Duration: dur,
|
||||
StartSeq: startSeq,
|
||||
EndSeq: endSeq,
|
||||
State: ChunkIdle,
|
||||
}
|
||||
}
|
||||
|
||||
cm.mu.Lock()
|
||||
cm.sessions[sessionKey(job.ID, profile.Name)] = s
|
||||
cm.mu.Unlock()
|
||||
|
||||
// 立刻启动第一个 chunk（保证 playlist 首切片可用）
|
||||
cm.ensureChunk(s, 0)
|
||||
|
||||
// 启动后台维护协程：空闲清理 + 完成检测
|
||||
go cm.sessionLoop(s)
|
||||
|
||||
utils.Log.Infof("[transcode][chunk] session started job=%s profile=%s totalDur=%.1fs chunkDur=%ds chunks=%d maxConc=%d",
|
||||
job.ID, profile.Name, totalDur, chunkDur, chunkCount, maxConc)
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// EnsureChunkRunningForSeg 确保覆盖 seq 的 chunk 正在转码（由 TCSegment 触发）
|
||||
// 返回所属 chunk index；如果 seq 越界返回 -1
|
||||
func (cm *ChunkManager) EnsureChunkRunningForSeg(jobID, profile string, seq int) int {
|
||||
cm.mu.Lock()
|
||||
s, ok := cm.sessions[sessionKey(jobID, profile)]
|
||||
cm.mu.Unlock()
|
||||
if !ok {
|
||||
utils.Log.Warnf("[transcode][chunk] EnsureChunk: session not found job=%s profile=%s seq=%d", jobID, profile, seq)
|
||||
return -1
|
||||
}
|
||||
idx := cm.findChunkBySeq(s, seq)
|
||||
if idx < 0 {
|
||||
utils.Log.Warnf("[transcode][chunk] EnsureChunk: seq=%d out of range job=%s", seq, jobID)
|
||||
return -1
|
||||
}
|
||||
cm.ensureChunk(s, idx)
|
||||
return idx
|
||||
}
|
||||
|
||||
// findChunkBySeq 按切片序号找到所属 chunk
|
||||
func (cm *ChunkManager) findChunkBySeq(s *ChunkSession, seq int) int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
for idx, c := range s.chunks {
|
||||
if seq >= c.StartSeq && seq < c.EndSeq {
|
||||
return idx
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// ensureChunk 确保某个 chunk 处于 Running 状态（已运行/已完成则跳过）
|
||||
// 同时受 MaxConcurrent 限制，超出时 LRU 淘汰最久未访问的 chunk
|
||||
func (cm *ChunkManager) ensureChunk(s *ChunkSession, idx int) {
|
||||
s.mu.Lock()
|
||||
c, ok := s.chunks[idx]
|
||||
if !ok {
|
||||
s.mu.Unlock()
|
||||
utils.Log.Warnf("[transcode][chunk] ensureChunk: chunk %d not in session (job=%s)", idx, s.JobID)
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
c.LastAccess = now
|
||||
prevState := c.State
|
||||
switch c.State {
|
||||
case ChunkRunning:
|
||||
s.mu.Unlock()
|
||||
utils.Log.Debugf("[transcode][chunk] ensureChunk: chunk %d already running (job=%s)", idx, s.JobID)
|
||||
return
|
||||
case ChunkDone:
|
||||
s.mu.Unlock()
|
||||
utils.Log.Debugf("[transcode][chunk] ensureChunk: chunk %d already done (job=%s)", idx, s.JobID)
|
||||
return
|
||||
}
|
||||
// 【关键修复】ChunkFailed 也允许重启：因为可能是网络抖动、source URL 临时失效等
|
||||
// 而不是永久性故障；如果连续失败，runChunk 内部不会陷入死循环（每次都会重新尝试源）
|
||||
// ChunkIdle / ChunkKilled / ChunkFailed → 需要启动
|
||||
running := 0
|
||||
var lruIdx = -1
|
||||
var lruTime = now
|
||||
for i, ck := range s.chunks {
|
||||
if ck.State == ChunkRunning {
|
||||
running++
|
||||
if i != idx && ck.LastAccess.Before(lruTime) {
|
||||
lruTime = ck.LastAccess
|
||||
lruIdx = i
|
||||
}
|
||||
}
|
||||
}
|
||||
if running >= s.MaxConcurrent && lruIdx >= 0 {
|
||||
victim := s.chunks[lruIdx]
|
||||
utils.Log.Infof("[transcode][chunk] LRU evict chunk %d (job=%s) to make room for chunk %d",
|
||||
lruIdx, s.JobID, idx)
|
||||
if victim.cancel != nil {
|
||||
victim.cancel()
|
||||
}
|
||||
victim.State = ChunkKilled
|
||||
}
|
||||
// 启动 chunk ffmpeg
|
||||
c.State = ChunkRunning
|
||||
c.StartedAt = now
|
||||
c.doneCh = make(chan struct{})
|
||||
c.doneOnce = sync.Once{}
|
||||
s.mu.Unlock()
|
||||
|
||||
utils.Log.Infof("[transcode][chunk] ensureChunk: launching chunk %d (job=%s prevState=%v running=%d)",
|
||||
idx, s.JobID, prevState, running)
|
||||
go cm.runChunk(s, idx)
|
||||
}
|
||||
|
||||
// runChunk 实际启动 ffmpeg 转码该 chunk
|
||||
func (cm *ChunkManager) runChunk(s *ChunkSession, idx int) {
|
||||
s.mu.Lock()
|
||||
c := s.chunks[idx]
|
||||
s.mu.Unlock()
|
||||
if c == nil {
|
||||
return
|
||||
}
|
||||
|
||||
timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMin)*time.Minute)
|
||||
s.mu.Lock()
|
||||
c.cancel = cancel
|
||||
s.mu.Unlock()
|
||||
defer cancel()
|
||||
|
||||
segPattern := filepath.Join(s.OutDir, "seg-%d.ts")
|
||||
// 每个 chunk 写自己的 playlist 文件，防止互相覆盖；后端不读这个 playlist 用于响应，
|
||||
// 真正的总 playlist 由 cache.BuildPlaylist 根据总时长生成。
|
||||
chunkPlaylist := filepath.Join(s.OutDir, fmt.Sprintf("_chunk_%d.m3u8", idx))
|
||||
|
||||
args := buildFFmpegChunkArgs(s.Job, s.ProfDef, s.SegDur, c.StartTime, c.Duration, c.StartSeq, segPattern, chunkPlaylist)
|
||||
ffmpegPath := setting.GetStr(conf.TranscodeFFmpegPath, "ffmpeg")
|
||||
cmd := exec.CommandContext(ctx, ffmpegPath, args...)
|
||||
stderr := &limitedBuffer{maxSize: 32 * 1024}
|
||||
cmd.Stderr = stderr
|
||||
|
||||
utils.Log.Infof("[transcode][chunk] start ffmpeg job=%s chunk=%d ss=%.1f t=%.1f startSeq=%d",
|
||||
s.JobID, idx, c.StartTime, c.Duration, c.StartSeq)
|
||||
|
||||
startedAt := time.Now()
|
||||
err := cmd.Run()
|
||||
elapsed := time.Since(startedAt)
|
||||
|
||||
s.mu.Lock()
|
||||
switch {
|
||||
case ctx.Err() != nil && c.State == ChunkKilled:
|
||||
// 被 LRU 主动 kill，状态保持 Killed
|
||||
utils.Log.Infof("[transcode][chunk] chunk %d killed (job=%s elapsed=%.1fs)", idx, s.JobID, elapsed.Seconds())
|
||||
case err != nil:
|
||||
c.State = ChunkFailed
|
||||
c.Error = truncate(stderr.String(), 512)
|
||||
utils.Log.Errorf("[transcode][chunk] chunk %d failed (job=%s elapsed=%.1fs): %v stderr=%s",
|
||||
idx, s.JobID, elapsed.Seconds(), err, c.Error)
|
||||
default:
|
||||
c.State = ChunkDone
|
||||
utils.Log.Infof("[transcode][chunk] chunk %d done (job=%s elapsed=%.1fs)", idx, s.JobID, elapsed.Seconds())
|
||||
}
|
||||
c.chunkDoneClose()
|
||||
s.mu.Unlock()
|
||||
|
||||
// 删除临时 playlist 文件
|
||||
_ = os.Remove(chunkPlaylist)
|
||||
|
||||
// 【关键修复】chunk 自然完成时，必须用 finalScan-style 扫描，否则边界切片（最大 seq）
|
||||
// 永远不会被登记到 cache，导致下次播放/拖动到该位置时 60 秒超时
|
||||
if cm.mgr != nil && cm.mgr.localRunner != nil {
|
||||
cm.mgr.localRunner.publishChunkSegments(s.Job, s.ProfDef, s.OutDir, c.StartSeq, c.EndSeq)
|
||||
}
|
||||
}
|
||||
|
||||
// sessionLoop 每个 session 的后台维护：空闲清理 + 全部完成检测 + 预读
|
||||
func (cm *ChunkManager) sessionLoop(s *ChunkSession) {
|
||||
t := time.NewTicker(5 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-cm.stopCh:
|
||||
cm.killAllChunks(s)
|
||||
return
|
||||
case <-s.allDoneCh:
|
||||
return
|
||||
case <-t.C:
|
||||
cm.maintainSession(s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// maintainSession 一次维护：检查空闲、预读下一 chunk、判断是否全部完成
|
||||
func (cm *ChunkManager) maintainSession(s *ChunkSession) {
|
||||
now := time.Now()
|
||||
idleThreshold := time.Duration(s.IdleSec) * time.Second
|
||||
|
||||
s.mu.Lock()
|
||||
// 1. 空闲清理：Running 状态但长时间未访问 → kill
|
||||
for _, c := range s.chunks {
|
||||
if c.State == ChunkRunning && now.Sub(c.LastAccess) > idleThreshold {
|
||||
utils.Log.Infof("[transcode][chunk] chunk %d idle for %.0fs, killing (job=%s)",
|
||||
c.Index, now.Sub(c.LastAccess).Seconds(), s.JobID)
|
||||
if c.cancel != nil {
|
||||
c.cancel()
|
||||
}
|
||||
c.State = ChunkKilled
|
||||
}
|
||||
}
|
||||
// 2. 预读：找出最近被访问的 Running chunk，如果它的进度过半，启动下一个 chunk
|
||||
var prefetchTarget = -1
|
||||
if s.Prefetch {
|
||||
var hottestIdx = -1
|
||||
var hottestAccess time.Time
|
||||
for i, c := range s.chunks {
|
||||
if c.State == ChunkRunning && c.LastAccess.After(hottestAccess) {
|
||||
hottestAccess = c.LastAccess
|
||||
hottestIdx = i
|
||||
}
|
||||
}
|
||||
if hottestIdx >= 0 {
|
||||
hot := s.chunks[hottestIdx]
|
||||
// 估算进度：用 ffmpeg 已经写出的切片数 / 该 chunk 总切片数
|
||||
// 由于 cache.segments 是全局的，我们检查范围 [StartSeq, EndSeq) 中已就绪数量
|
||||
done := cm.countReadySegments(s, hot.StartSeq, hot.EndSeq)
|
||||
expected := hot.EndSeq - hot.StartSeq
|
||||
if expected > 0 && done*2 >= expected { // 进度 >= 50%
|
||||
next := hottestIdx + 1
|
||||
if nc, ok := s.chunks[next]; ok && nc.State == ChunkIdle {
|
||||
prefetchTarget = next
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 3. 全部完成检测：所有 chunk 都 Done
|
||||
allDone := true
|
||||
for _, c := range s.chunks {
|
||||
if c.State != ChunkDone {
|
||||
allDone = false
|
||||
break
|
||||
}
|
||||
}
|
||||
s.mu.Unlock()
|
||||
|
||||
if prefetchTarget >= 0 {
|
||||
utils.Log.Infof("[transcode][chunk] prefetch chunk %d (job=%s)", prefetchTarget, s.JobID)
|
||||
cm.ensureChunk(s, prefetchTarget)
|
||||
}
|
||||
|
||||
if allDone {
|
||||
s.allDoneClose()
|
||||
}
|
||||
}
|
||||
|
||||
// countReadySegments 统计 [start, end) 范围内已就绪的切片数
|
||||
func (cm *ChunkManager) countReadySegments(s *ChunkSession, start, end int) int {
|
||||
if cm.mgr == nil || cm.mgr.Cache == nil {
|
||||
return 0
|
||||
}
|
||||
cnt := 0
|
||||
for seq := start; seq < end; seq++ {
|
||||
if _, err := cm.mgr.Cache.GetSegment(s.JobID, s.Profile, seq); err == nil {
|
||||
cnt++
|
||||
}
|
||||
}
|
||||
return cnt
|
||||
}
|
||||
|
||||
// killAllChunks 关闭某 session 下所有运行中的 ffmpeg
|
||||
func (cm *ChunkManager) killAllChunks(s *ChunkSession) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
for _, c := range s.chunks {
|
||||
if c.State == ChunkRunning && c.cancel != nil {
|
||||
c.cancel()
|
||||
c.State = ChunkKilled
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// CancelSession 外部取消（job 被 cancel 或 idle 超时）
|
||||
func (cm *ChunkManager) CancelSession(jobID, profile string) {
|
||||
cm.mu.Lock()
|
||||
s, ok := cm.sessions[sessionKey(jobID, profile)]
|
||||
if ok {
|
||||
delete(cm.sessions, sessionKey(jobID, profile))
|
||||
}
|
||||
cm.mu.Unlock()
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
cm.killAllChunks(s)
|
||||
s.allDoneClose()
|
||||
}
|
||||
|
||||
// CancelAllForJob 取消该 job 下所有 profile 的 session（CancelJob 时调用）
|
||||
func (cm *ChunkManager) CancelAllForJob(jobID string) {
|
||||
cm.mu.Lock()
|
||||
var targets []*ChunkSession
|
||||
for k, s := range cm.sessions {
|
||||
if s.JobID == jobID {
|
||||
targets = append(targets, s)
|
||||
delete(cm.sessions, k)
|
||||
}
|
||||
}
|
||||
cm.mu.Unlock()
|
||||
for _, s := range targets {
|
||||
cm.killAllChunks(s)
|
||||
s.allDoneClose()
|
||||
}
|
||||
}
|
||||
|
||||
// WaitDone 阻塞等待所有 chunk 完成（LocalRunner.runJob 用）
|
||||
func (s *ChunkSession) WaitDone() <-chan struct{} { return s.allDoneCh }
|
||||
|
||||
// Stop 停止 ChunkManager（Manager.Stop 调用）
|
||||
func (cm *ChunkManager) Stop() {
|
||||
close(cm.stopCh)
|
||||
cm.mu.Lock()
|
||||
defer cm.mu.Unlock()
|
||||
for _, s := range cm.sessions {
|
||||
cm.killAllChunks(s)
|
||||
s.allDoneClose()
|
||||
}
|
||||
cm.sessions = nil
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
)
|
||||
|
||||
// DecideRequest 决策器输入
|
||||
type DecideRequest struct {
|
||||
FileName string
|
||||
FileSize int64 // 字节
|
||||
Probe SourceProbe // 可选；为空则只用大小+后缀判断
|
||||
}
|
||||
|
||||
// DecideResult 决策结果
|
||||
type DecideResult struct {
|
||||
NeedTranscode bool
|
||||
Reason string
|
||||
}
|
||||
|
||||
// Decide 根据全局设置判断是否需要走转码
|
||||
func Decide(r DecideRequest) DecideResult {
|
||||
if !setting.GetBool(conf.TranscodeEnabled) {
|
||||
return DecideResult{false, "transcode disabled"}
|
||||
}
|
||||
// 1. 大小阈值
|
||||
minGB := setting.GetInt(conf.TranscodeMinSizeGB, 5)
|
||||
if minGB > 0 {
|
||||
minBytes := int64(minGB) * 1024 * 1024 * 1024
|
||||
if r.FileSize > 0 && r.FileSize < minBytes {
|
||||
return DecideResult{false, "size below threshold"}
|
||||
}
|
||||
}
|
||||
// 2. 后缀过滤
|
||||
exts := splitCSV(setting.GetStr(conf.TranscodeSourceExtensions))
|
||||
if len(exts) > 0 {
|
||||
ext := strings.ToLower(strings.TrimPrefix(getExt(r.FileName), "."))
|
||||
if !containsFold(exts, ext) {
|
||||
return DecideResult{false, "extension not in list"}
|
||||
}
|
||||
}
|
||||
// 3. 编码过滤（仅当 probe 提供时）
|
||||
codecs := splitCSV(setting.GetStr(conf.TranscodeSourceCodecs))
|
||||
if r.Probe.VideoCodec != "" && len(codecs) > 0 {
|
||||
if !containsFold(codecs, r.Probe.VideoCodec) {
|
||||
return DecideResult{false, "codec not in list"}
|
||||
}
|
||||
}
|
||||
// 4. 码率过滤（仅当 probe 提供时）
|
||||
minBR := setting.GetInt(conf.TranscodeMinBitrateMbps, 0)
|
||||
if minBR > 0 && r.Probe.VideoBitrate > 0 {
|
||||
if r.Probe.VideoBitrate < int64(minBR)*1_000_000 {
|
||||
return DecideResult{false, "bitrate below threshold"}
|
||||
}
|
||||
}
|
||||
return DecideResult{true, "ok"}
|
||||
}
|
||||
|
||||
func splitCSV(s string) []string {
|
||||
var out []string
|
||||
for _, p := range strings.Split(s, ",") {
|
||||
p = strings.TrimSpace(p)
|
||||
if p != "" {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func containsFold(arr []string, v string) bool {
|
||||
for _, a := range arr {
|
||||
if strings.EqualFold(a, v) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func getExt(name string) string {
|
||||
i := strings.LastIndex(name, ".")
|
||||
if i < 0 {
|
||||
return ""
|
||||
}
|
||||
return name[i:]
|
||||
}
|
||||
|
||||
// BuildProfile 根据当前全局设置构建默认 Profile
|
||||
func BuildProfile() Profile {
|
||||
res := setting.GetStr(conf.TranscodeOutputResolution)
|
||||
scale := ""
|
||||
name := "default"
|
||||
switch res {
|
||||
case "source":
|
||||
scale = ""
|
||||
name = "source"
|
||||
case "3840x2160":
|
||||
scale = "3840:-2"
|
||||
name = "2160p"
|
||||
case "2560x1440":
|
||||
scale = "2560:-2"
|
||||
name = "1440p"
|
||||
case "1920x1080":
|
||||
scale = "1920:-2"
|
||||
name = "1080p"
|
||||
case "1280x720":
|
||||
scale = "1280:-2"
|
||||
name = "720p"
|
||||
case "854x480":
|
||||
scale = "854:-2"
|
||||
name = "480p"
|
||||
}
|
||||
return Profile{
|
||||
Name: name,
|
||||
VideoCodec: setting.GetStr(conf.TranscodeOutputCodec),
|
||||
VideoBitrate: setting.GetStr(conf.TranscodeOutputBitrate),
|
||||
Scale: scale,
|
||||
AudioCodec: setting.GetStr(conf.TranscodeOutputAudioCodec),
|
||||
AudioBitrate: setting.GetStr(conf.TranscodeOutputAudioBitrate),
|
||||
HWAccel: setting.GetStr(conf.TranscodeHWAccel),
|
||||
}
|
||||
}
|
||||
|
||||
// BuildOutputSpec 根据当前全局设置构建输出规格
|
||||
func BuildOutputSpec() OutputSpec {
|
||||
return OutputSpec{
|
||||
Format: OutputFormat(setting.GetStr(conf.TranscodeOutputFormat)),
|
||||
SegmentDuration: setting.GetInt(conf.TranscodeSegmentDuration, 6),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,772 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
)
|
||||
|
||||
// limitedBuffer 是一个有容量上限的 io.Writer，超出 maxSize 后静默丢弃。
|
||||
// 用于捕获 ffmpeg stderr，防止异常文件导致无限内存增长。
|
||||
type limitedBuffer struct {
|
||||
data []byte
|
||||
maxSize int
|
||||
}
|
||||
|
||||
func (b *limitedBuffer) Write(p []byte) (int, error) {
|
||||
remain := b.maxSize - len(b.data)
|
||||
if remain <= 0 {
|
||||
return len(p), nil // 静默丢弃
|
||||
}
|
||||
if len(p) > remain {
|
||||
p = p[:remain]
|
||||
}
|
||||
b.data = append(b.data, p...)
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (b *limitedBuffer) String() string {
|
||||
return string(b.data)
|
||||
}
|
||||
|
||||
// LocalRunner 本地内置 worker，直接调用本机 ffmpeg
|
||||
type LocalRunner struct {
|
||||
mgr *Manager
|
||||
workerID string
|
||||
|
||||
stopCh chan struct{}
|
||||
wg sync.WaitGroup
|
||||
|
||||
// 运行中 job 的 cancel 管理：jobID -> cancelFunc
|
||||
runMu sync.Mutex
|
||||
runningJobs map[string]context.CancelFunc
|
||||
}
|
||||
|
||||
func NewLocalRunner(mgr *Manager) *LocalRunner {
|
||||
return &LocalRunner{
|
||||
mgr: mgr,
|
||||
stopCh: make(chan struct{}),
|
||||
runningJobs: make(map[string]context.CancelFunc),
|
||||
}
|
||||
}
|
||||
|
||||
// Start 注册本地 worker 并启动消费协程
|
||||
func (r *LocalRunner) Start() error {
|
||||
concurrency := setting.GetInt(conf.TranscodeLocalConcurrency, 1)
|
||||
if concurrency <= 0 {
|
||||
concurrency = 1
|
||||
}
|
||||
hwaccel := setting.GetStr(conf.TranscodeHWAccel, "none")
|
||||
hostname, _ := os.Hostname()
|
||||
w := r.mgr.Registry.Register(&RegisterRequest{
|
||||
Name: "local-" + hostname,
|
||||
Version: "embedded",
|
||||
Capacity: concurrency,
|
||||
HWAccel: []string{hwaccel},
|
||||
CodecsDecode: []string{"h264", "hevc", "av1", "vp9", "mpeg4", "mpeg2video"},
|
||||
CodecsEncode: []string{"h264", "hevc"},
|
||||
MaxResolution: "3840x2160",
|
||||
Tags: []string{"local", "embedded"},
|
||||
}, true)
|
||||
r.workerID = w.ID
|
||||
utils.Log.Infof("[transcode] local worker started: id=%s capacity=%d hwaccel=%s", w.ID, concurrency, hwaccel)
|
||||
|
||||
for i := 0; i < concurrency; i++ {
|
||||
r.wg.Add(1)
|
||||
go r.loop()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *LocalRunner) Stop() {
|
||||
close(r.stopCh)
|
||||
// 取消所有运行中的 FFmpeg 进程
|
||||
r.runMu.Lock()
|
||||
for id, cancel := range r.runningJobs {
|
||||
utils.Log.Infof("[transcode] stopping ffmpeg for job %s (runner shutdown)", id)
|
||||
cancel()
|
||||
}
|
||||
r.runMu.Unlock()
|
||||
r.wg.Wait()
|
||||
r.mgr.Registry.Unregister(r.workerID)
|
||||
}
|
||||
|
||||
// CancelJob 外部取消指定 job 的 FFmpeg 进程（由 Scheduler.Cancel / 空闲超时调用）
|
||||
func (r *LocalRunner) CancelJob(jobID string) bool {
|
||||
r.runMu.Lock()
|
||||
cancel, ok := r.runningJobs[jobID]
|
||||
r.runMu.Unlock()
|
||||
if ok {
|
||||
utils.Log.Infof("[transcode] killing ffmpeg for job %s (external cancel)", jobID)
|
||||
cancel()
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
// RunningJobIDs 返回当前正在运行的所有 job ID 列表
|
||||
func (r *LocalRunner) RunningJobIDs() []string {
|
||||
r.runMu.Lock()
|
||||
defer r.runMu.Unlock()
|
||||
ids := make([]string, 0, len(r.runningJobs))
|
||||
for id := range r.runningJobs {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func (r *LocalRunner) loop() {
|
||||
defer r.wg.Done()
|
||||
for {
|
||||
select {
|
||||
case <-r.stopCh:
|
||||
return
|
||||
default:
|
||||
}
|
||||
jobs := r.mgr.Scheduler.Claim(r.workerID, 1, 5*time.Second)
|
||||
if len(jobs) == 0 {
|
||||
continue
|
||||
}
|
||||
for _, j := range jobs {
|
||||
r.runJob(j)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runJob 执行一个转码任务（单 profile，HLS 输出，边转边写入 cache）
|
||||
func (r *LocalRunner) runJob(job *Job) {
|
||||
utils.Log.Infof("[transcode] local runner picked job %s (%s)", job.ID, job.Path)
|
||||
if len(job.Profiles) == 0 {
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: "no profiles"})
|
||||
return
|
||||
}
|
||||
profile := job.Profiles[0]
|
||||
timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMin)*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
// 注册到 runningJobs，支持外部取消
|
||||
r.runMu.Lock()
|
||||
r.runningJobs[job.ID] = cancel
|
||||
r.runMu.Unlock()
|
||||
defer func() {
|
||||
r.runMu.Lock()
|
||||
delete(r.runningJobs, job.ID)
|
||||
r.runMu.Unlock()
|
||||
}()
|
||||
|
||||
// 输出目录
|
||||
jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name)
|
||||
if err != nil {
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()})
|
||||
return
|
||||
}
|
||||
outDir := jc.BaseDir
|
||||
|
||||
// 【智能调度】如果探测到了视频总时长，启用 chunk 模式（按需分段并发转码）
|
||||
// 否则回退到旧的"一次性整段"模式（兼容 ffprobe 失败的场景）
|
||||
if job.Probe.Duration > 0 && r.mgr.Chunks != nil {
|
||||
r.runJobByChunks(ctx, job, profile, outDir)
|
||||
return
|
||||
}
|
||||
|
||||
// === 旧路径：一次性整段转码 ===
|
||||
r.runJobLegacy(ctx, job, profile, outDir)
|
||||
}
|
||||
|
||||
// runJobByChunks 智能 chunk 调度模式：按需启动 ffmpeg 进程，支持随机访问
|
||||
func (r *LocalRunner) runJobByChunks(ctx context.Context, job *Job, profile Profile, outDir string) {
|
||||
startedAt := time.Now()
|
||||
|
||||
// 启动 watchSegments：周期扫描目录把切片登记到 cache
|
||||
stopWatcher := make(chan struct{})
|
||||
var watcherWG sync.WaitGroup
|
||||
watcherWG.Add(1)
|
||||
go func() {
|
||||
defer watcherWG.Done()
|
||||
r.watchSegments(job, profile, outDir, stopWatcher)
|
||||
}()
|
||||
|
||||
// 创建 chunk session（会立刻启动 chunk-0 + 后台维护协程）
|
||||
session, err := r.mgr.Chunks.StartSession(job, profile, outDir)
|
||||
if err != nil {
|
||||
close(stopWatcher)
|
||||
watcherWG.Wait()
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 阻塞等待全部 chunk 完成 / 任务被取消
|
||||
select {
|
||||
case <-session.WaitDone():
|
||||
// 所有 chunk 都 Done，正常完成
|
||||
case <-ctx.Done():
|
||||
// 外部取消：通知 chunk session 杀掉所有 ffmpeg
|
||||
r.mgr.Chunks.CancelSession(job.ID, profile.Name)
|
||||
}
|
||||
|
||||
close(stopWatcher)
|
||||
watcherWG.Wait()
|
||||
r.scanAndPublish(job, profile, outDir, true)
|
||||
|
||||
if ctx.Err() != nil {
|
||||
utils.Log.Infof("[transcode] job %s chunk session cancelled", job.ID)
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{
|
||||
JobID: job.ID,
|
||||
Status: JobCancelled,
|
||||
Error: "cancelled: " + ctx.Err().Error(),
|
||||
})
|
||||
return
|
||||
}
|
||||
utils.Log.Infof("[transcode] job %s completed in %.1fs (chunk mode)", job.ID, time.Since(startedAt).Seconds())
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{
|
||||
JobID: job.ID,
|
||||
Status: JobFinished,
|
||||
Stats: JobStats{Elapsed: time.Since(startedAt).Seconds()},
|
||||
})
|
||||
}
|
||||
|
||||
// runJobLegacy 旧路径：一次性整段转码（不知道总时长时回退使用）
|
||||
func (r *LocalRunner) runJobLegacy(ctx context.Context, job *Job, profile Profile, outDir string) {
|
||||
playlistPath := filepath.Join(outDir, "playlist.m3u8")
|
||||
segPattern := filepath.Join(outDir, "seg-%d.ts")
|
||||
|
||||
args := buildFFmpegArgs(job, profile, segPattern, playlistPath)
|
||||
ffmpegPath := setting.GetStr(conf.TranscodeFFmpegPath, "ffmpeg")
|
||||
|
||||
cmd := exec.CommandContext(ctx, ffmpegPath, args...)
|
||||
stderr := &limitedBuffer{maxSize: 64 * 1024}
|
||||
cmd.Stderr = stderr
|
||||
|
||||
startedAt := time.Now()
|
||||
if err := cmd.Start(); err != nil {
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{JobID: job.ID, Status: JobFailed, Error: err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
stopWatcher := make(chan struct{})
|
||||
var watcherWG sync.WaitGroup
|
||||
watcherWG.Add(1)
|
||||
go func() {
|
||||
defer watcherWG.Done()
|
||||
r.watchSegments(job, profile, outDir, stopWatcher)
|
||||
}()
|
||||
|
||||
err := cmd.Wait()
|
||||
close(stopWatcher)
|
||||
watcherWG.Wait()
|
||||
r.scanAndPublish(job, profile, outDir, true)
|
||||
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
utils.Log.Infof("[transcode] job %s ffmpeg killed (context cancelled)", job.ID)
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{
|
||||
JobID: job.ID,
|
||||
Status: JobCancelled,
|
||||
Error: "cancelled: " + ctx.Err().Error(),
|
||||
})
|
||||
} else {
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{
|
||||
JobID: job.ID,
|
||||
Status: JobFailed,
|
||||
Error: fmt.Sprintf("ffmpeg failed: %v, stderr=%s", err, truncate(stderr.String(), 1024)),
|
||||
})
|
||||
}
|
||||
return
|
||||
}
|
||||
utils.Log.Infof("[transcode] job %s completed in %.1fs (legacy mode)", job.ID, time.Since(startedAt).Seconds())
|
||||
_ = r.mgr.Scheduler.Finish(&FinishRequest{
|
||||
JobID: job.ID,
|
||||
Status: JobFinished,
|
||||
Stats: JobStats{Elapsed: time.Since(startedAt).Seconds()},
|
||||
})
|
||||
}
|
||||
|
||||
// watchSegments 周期性扫描输出目录，把新切片登记到 Cache
|
||||
// 【内存优化】把扫描频率从 500ms 降到 2s，减少每次 os.ReadDir 和 parseFFmpegPlaylistDurations
|
||||
// 触发的字符串/字节切片分配。chunk 模式下每个 chunk 完成时还会主动调用 publishChunkSegments，
|
||||
// 所以这里只需要兜底登记中间切片，2s 频率完全够用。
|
||||
func (r *LocalRunner) watchSegments(job *Job, profile Profile, dir string, stop chan struct{}) {
|
||||
t := time.NewTicker(2 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
case <-t.C:
|
||||
r.scanAndPublish(job, profile, dir, false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *LocalRunner) scanAndPublish(job *Job, profile Profile, dir string, finalScan bool) {
|
||||
jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
type segFile struct {
|
||||
seq int
|
||||
path string
|
||||
size int64
|
||||
}
|
||||
var segs []segFile
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.Name()
|
||||
if !strings.HasPrefix(name, "seg-") || !strings.HasSuffix(name, ".ts") {
|
||||
continue
|
||||
}
|
||||
numStr := strings.TrimSuffix(strings.TrimPrefix(name, "seg-"), ".ts")
|
||||
seq, err := strconv.Atoi(numStr)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
fi, err := e.Info()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
segs = append(segs, segFile{seq: seq, path: filepath.Join(dir, name), size: fi.Size()})
|
||||
}
|
||||
sort.Slice(segs, func(i, j int) bool { return segs[i].seq < segs[j].seq })
|
||||
|
||||
// 【优化】解析 ffmpeg 写出的 playlist.m3u8 获取每个切片的真实时长
|
||||
// 避免硬编码 segDur 导致 HLS.js 显示总时长不准确
|
||||
realDurations := parseFFmpegPlaylistDurations(filepath.Join(dir, "playlist.m3u8"))
|
||||
|
||||
// ffmpeg 在写入 .ts 时是边写边写,稳妥起见仅登记除最后一片以外的切片(最后一片可能未写完)
|
||||
limit := len(segs)
|
||||
if !finalScan && limit > 0 {
|
||||
limit--
|
||||
}
|
||||
defaultDur := float64(setting.GetInt(conf.TranscodeSegmentDuration, 6))
|
||||
for i := 0; i < limit; i++ {
|
||||
s := segs[i]
|
||||
jc.mu.RLock()
|
||||
_, exists := jc.segments[s.seq]
|
||||
jc.mu.RUnlock()
|
||||
if exists {
|
||||
continue
|
||||
}
|
||||
// 优先用 ffmpeg playlist 中的真实时长
|
||||
dur := defaultDur
|
||||
if d, ok := realDurations[s.seq]; ok && d > 0 {
|
||||
dur = d
|
||||
}
|
||||
// 直接登记，不复制（已经在 cache 目录里了）
|
||||
jc.mu.Lock()
|
||||
jc.segments[s.seq] = &SegmentInfo{Seq: s.seq, Duration: dur, Size: s.size, Path: s.path}
|
||||
var toClose []chan struct{}
|
||||
for sk, chs := range jc.waiters {
|
||||
if sk <= s.seq {
|
||||
toClose = append(toClose, chs...)
|
||||
delete(jc.waiters, sk)
|
||||
}
|
||||
}
|
||||
jc.mu.Unlock()
|
||||
for _, ch := range toClose {
|
||||
close(ch)
|
||||
}
|
||||
// 第一片就绪 → 通知 job ready
|
||||
if s.seq == 0 {
|
||||
job.MarkReady()
|
||||
}
|
||||
}
|
||||
if finalScan {
|
||||
jc.mu.Lock()
|
||||
jc.final = true
|
||||
jc.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// publishChunkSegments 把 [startSeq, endSeq) 范围内已经写完的切片登记到 cache。
|
||||
// 与 scanAndPublish 不同，这个方法专门用于"某个 chunk 的 ffmpeg 已经退出"的场景，
|
||||
// 此时该范围的所有 .ts 文件都是完整的，不能再丢掉最大序号的那片（避免边界切片永远丢失）。
|
||||
func (r *LocalRunner) publishChunkSegments(job *Job, profile Profile, dir string, startSeq, endSeq int) {
|
||||
jc, err := r.mgr.Cache.getOrCreate(job.ID, profile.Name)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defaultDur := float64(setting.GetInt(conf.TranscodeSegmentDuration, 6))
|
||||
registered := 0
|
||||
for seq := startSeq; seq < endSeq; seq++ {
|
||||
jc.mu.RLock()
|
||||
_, exists := jc.segments[seq]
|
||||
jc.mu.RUnlock()
|
||||
if exists {
|
||||
continue
|
||||
}
|
||||
fpath := filepath.Join(dir, fmt.Sprintf("seg-%d.ts", seq))
|
||||
fi, err := os.Stat(fpath)
|
||||
if err != nil || fi.IsDir() || fi.Size() == 0 {
|
||||
continue
|
||||
}
|
||||
jc.mu.Lock()
|
||||
jc.segments[seq] = &SegmentInfo{Seq: seq, Duration: defaultDur, Size: fi.Size(), Path: fpath}
|
||||
var toClose []chan struct{}
|
||||
for sk, chs := range jc.waiters {
|
||||
if sk <= seq {
|
||||
toClose = append(toClose, chs...)
|
||||
delete(jc.waiters, sk)
|
||||
}
|
||||
}
|
||||
jc.mu.Unlock()
|
||||
for _, ch := range toClose {
|
||||
close(ch)
|
||||
}
|
||||
if seq == 0 {
|
||||
job.MarkReady()
|
||||
}
|
||||
registered++
|
||||
}
|
||||
if registered > 0 {
|
||||
utils.Log.Infof("[transcode][chunk] published %d segments [%d, %d) for job=%s",
|
||||
registered, startSeq, endSeq, job.ID)
|
||||
}
|
||||
}
|
||||
|
||||
// playlistDurCache 缓存 m3u8 解析结果，按文件 mtime+size 失效。
|
||||
// 【内存优化】scanAndPublish 每 2 秒一次，每次都会解析同一个 m3u8（长视频几百 KB），
|
||||
// strings.Split 会产生大量临时字符串切片造成 GC 压力。用缓存减少 99% 的重复解析。
|
||||
type playlistDurCacheEntry struct {
|
||||
mtime time.Time
|
||||
size int64
|
||||
durs map[int]float64
|
||||
}
|
||||
|
||||
var (
|
||||
playlistDurCache = make(map[string]*playlistDurCacheEntry)
|
||||
playlistDurCacheMu sync.Mutex
|
||||
)
|
||||
|
||||
// parseFFmpegPlaylistDurations 解析 ffmpeg 写出的 m3u8，返回 seq -> duration 的映射
|
||||
// 用于获取每个切片的真实时长，避免后端用硬编码 segDur 导致总时长显示偏差
|
||||
func parseFFmpegPlaylistDurations(path string) map[int]float64 {
|
||||
fi, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return map[int]float64{}
|
||||
}
|
||||
// 命中缓存：mtime + size 都没变就直接复用
|
||||
playlistDurCacheMu.Lock()
|
||||
if e, ok := playlistDurCache[path]; ok && e.mtime.Equal(fi.ModTime()) && e.size == fi.Size() {
|
||||
playlistDurCacheMu.Unlock()
|
||||
return e.durs
|
||||
}
|
||||
playlistDurCacheMu.Unlock()
|
||||
|
||||
out := make(map[int]float64)
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
lines := strings.Split(string(data), "\n")
|
||||
var pendingDur float64
|
||||
for _, line := range lines {
|
||||
line = strings.TrimSpace(line)
|
||||
if strings.HasPrefix(line, "#EXTINF:") {
|
||||
// #EXTINF:6.000000,
|
||||
body := strings.TrimPrefix(line, "#EXTINF:")
|
||||
body = strings.TrimSuffix(body, ",")
|
||||
d, err := strconv.ParseFloat(strings.TrimSpace(body), 64)
|
||||
if err == nil {
|
||||
pendingDur = d
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.HasSuffix(line, ".ts") && strings.HasPrefix(line, "seg-") {
|
||||
numStr := strings.TrimSuffix(strings.TrimPrefix(line, "seg-"), ".ts")
|
||||
if seq, err := strconv.Atoi(numStr); err == nil && pendingDur > 0 {
|
||||
out[seq] = pendingDur
|
||||
}
|
||||
pendingDur = 0
|
||||
}
|
||||
}
|
||||
// 写入缓存
|
||||
playlistDurCacheMu.Lock()
|
||||
playlistDurCache[path] = &playlistDurCacheEntry{
|
||||
mtime: fi.ModTime(),
|
||||
size: fi.Size(),
|
||||
durs: out,
|
||||
}
|
||||
playlistDurCacheMu.Unlock()
|
||||
return out
|
||||
}
|
||||
|
||||
// buildFFmpegArgs 构造 ffmpeg 命令行（一次性整段转码，兼容旧路径）
|
||||
// 当 Probe.Duration 不可用时，LocalRunner.runJob 会回退到这条路径。
|
||||
func buildFFmpegArgs(job *Job, profile Profile, segPattern, playlistPath string) []string {
|
||||
segDur := setting.GetInt(conf.TranscodeSegmentDuration, 6)
|
||||
// startTime=0, duration=0(=不限) 表示整段转码；startSeq=0
|
||||
return buildFFmpegArgsCommon(job, profile, segDur, 0, 0, 0, segPattern, playlistPath)
|
||||
}
|
||||
|
||||
// buildFFmpegChunkArgs 构造 chunk 模式的 ffmpeg 命令行
|
||||
// startTime: 起始时间（秒）；duration: 持续时长（秒，0=不限）；startSeq: 切片起始编号
|
||||
// chunk 模式与一次性整段共用底层参数构造，差异仅在 -ss/-t/-start_number。
|
||||
func buildFFmpegChunkArgs(job *Job, profile Profile, segDur int, startTime, duration float64, startSeq int, segPattern, playlistPath string) []string {
|
||||
return buildFFmpegArgsCommon(job, profile, segDur, startTime, duration, startSeq, segPattern, playlistPath)
|
||||
}
|
||||
|
||||
// buildFFmpegArgsCommon 通用 ffmpeg 命令行构造
|
||||
// 当 startTime > 0 或 duration > 0 时，启用"chunk 模式"：使用 -ss / -t / -start_number 限定范围
|
||||
// 关键：-ss 必须放在 -i 之前（input seek，快），且需配合 force_key_frames 让 chunk 起始处生成 IDR 帧
|
||||
func buildFFmpegArgsCommon(job *Job, profile Profile, segDur int, startTime, duration float64, startSeq int, segPattern, playlistPath string) []string {
|
||||
hw := strings.ToLower(profile.HWAccel)
|
||||
args := []string{"-y", "-loglevel", "warning"}
|
||||
|
||||
// 输入侧硬件加速
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
args = append(args, "-hwaccel", "cuda")
|
||||
case "qsv":
|
||||
args = append(args, "-hwaccel", "qsv")
|
||||
case "vaapi":
|
||||
args = append(args, "-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", "/dev/dri/renderD128")
|
||||
case "amf":
|
||||
args = append(args, "-hwaccel", "d3d11va")
|
||||
case "videotoolbox":
|
||||
args = append(args, "-hwaccel", "videotoolbox")
|
||||
case "auto":
|
||||
args = append(args, "-hwaccel", "auto")
|
||||
}
|
||||
|
||||
// 【chunk 模式】input seek：放在 -i 之前，速度快，会按关键帧对齐
|
||||
// 注意 -ss 在 -i 之前是 input seek（用容器索引快速跳转），但落点是关键帧；
|
||||
// 我们通过 force_key_frames 保证每个 chunk 边界都有关键帧，所以这里精度足够。
|
||||
if startTime > 0.001 {
|
||||
args = append(args, "-ss", fmt.Sprintf("%.3f", startTime))
|
||||
}
|
||||
|
||||
// 输入：先限定协议白名单（输入选项，必须在 -i 之前），再指定输入
|
||||
args = append(args, "-protocol_whitelist", ProtocolWhitelistFor(job.SourceURL))
|
||||
args = append(args, "-i", job.SourceURL)
|
||||
|
||||
// 【chunk 模式】限定时长（output side 更精确）
|
||||
if duration > 0.001 {
|
||||
args = append(args, "-t", fmt.Sprintf("%.3f", duration))
|
||||
}
|
||||
|
||||
// 视频编码器
|
||||
vcodec := pickEncoder(profile.VideoCodec, hw)
|
||||
args = append(args, "-c:v", vcodec)
|
||||
// 【关键】强制浏览器兼容的 profile/level/pix_fmt，必须与 master.m3u8 声明的
|
||||
// CODECS="avc1.640028" (H.264 High Profile Level 4.0) 一致，否则 HLS.js 会触发
|
||||
// bufferAppendError 死循环。
|
||||
if strings.Contains(vcodec, "264") {
|
||||
args = append(args, "-profile:v", "high", "-level:v", "4.0", "-pix_fmt", "yuv420p")
|
||||
}
|
||||
if profile.VideoBitrate != "" {
|
||||
args = append(args, "-b:v", profile.VideoBitrate)
|
||||
args = append(args, "-maxrate", profile.VideoBitrate, "-bufsize", profile.VideoBitrate)
|
||||
}
|
||||
if profile.Scale != "" {
|
||||
// 软件 / 硬件缩放
|
||||
switch hw {
|
||||
case "vaapi":
|
||||
args = append(args, "-vf", "scale_vaapi="+profile.Scale)
|
||||
case "qsv":
|
||||
args = append(args, "-vf", "scale_qsv="+profile.Scale)
|
||||
default:
|
||||
args = append(args, "-vf", "scale="+profile.Scale)
|
||||
}
|
||||
}
|
||||
args = append(args, "-preset", defaultPresetFor(vcodec))
|
||||
|
||||
// 音频
|
||||
// 【关键修复】不能使用 -c:a copy，因为浏览器 MSE 不支持 AC3/DTS/EAC3/TrueHD 等
|
||||
// 常见于 mkv 的音频编码。直接 copy 会导致 SourceBuffer.appendBuffer 失败
|
||||
// 触发 bufferAppendError 死循环。统一强制转 AAC 兼容所有浏览器。
|
||||
ac := profile.AudioCodec
|
||||
if ac == "" || ac == "copy" {
|
||||
ac = "aac"
|
||||
}
|
||||
args = append(args, "-c:a", ac)
|
||||
ab := profile.AudioBitrate
|
||||
if ab == "" {
|
||||
ab = "160k"
|
||||
}
|
||||
args = append(args, "-b:a", ab)
|
||||
// 强制立体声 + 48kHz，避免 5.1 声道 / 高采样率导致的兼容问题
|
||||
args = append(args, "-ac", "2", "-ar", "48000")
|
||||
|
||||
// HLS 输出
|
||||
if segDur <= 0 {
|
||||
segDur = setting.GetInt(conf.TranscodeSegmentDuration, 6)
|
||||
}
|
||||
// 【关键】强制关键帧对齐切片边界，确保每个切片以 IDR 帧开始可独立解码。
|
||||
// 没有这些参数时，第一个切片可能不含 IDR 帧，导致 HLS.js bufferAppendError。
|
||||
// fps 假设 25 (帧率未知时的保守值)，gop = segDur * fps；force_key_frames 让
|
||||
// ffmpeg 在每个切片边界强制插入关键帧。
|
||||
gop := segDur * 25
|
||||
args = append(args,
|
||||
"-g", strconv.Itoa(gop),
|
||||
"-keyint_min", strconv.Itoa(gop),
|
||||
"-sc_threshold", "0",
|
||||
"-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", segDur),
|
||||
)
|
||||
// 【chunk 模式 - 关键修复】input seek 后 ffmpeg 会把第一帧 PTS 重置为 0，
|
||||
// 这会导致 chunk-30 的切片在播放器看来是 PTS=0~60s（而不是 1800~1860s），
|
||||
// 进而触发 HLS.js bufferAppendError 或时间轴错位。
|
||||
// 解决方法：用 -output_ts_offset 让生成的切片时间戳偏移到原始位置。
|
||||
if startTime > 0.001 {
|
||||
args = append(args, "-output_ts_offset", fmt.Sprintf("%.3f", startTime))
|
||||
}
|
||||
args = append(args,
|
||||
"-f", "hls",
|
||||
"-hls_time", strconv.Itoa(segDur),
|
||||
"-hls_playlist_type", "vod",
|
||||
"-hls_segment_filename", segPattern,
|
||||
"-hls_flags", "independent_segments+temp_file",
|
||||
)
|
||||
// 【chunk 模式】指定起始切片编号，让全局切片编号连续
|
||||
if startSeq > 0 {
|
||||
args = append(args, "-start_number", strconv.Itoa(startSeq))
|
||||
}
|
||||
args = append(args, playlistPath)
|
||||
return args
|
||||
}
|
||||
|
||||
// pickEncoder 根据视频编码 + HW 类型选择 ffmpeg 编码器
|
||||
func pickEncoder(codec, hw string) string {
|
||||
codec = strings.ToLower(codec)
|
||||
switch codec {
|
||||
case "h264", "":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "h264_nvenc"
|
||||
case "qsv":
|
||||
return "h264_qsv"
|
||||
case "vaapi":
|
||||
return "h264_vaapi"
|
||||
case "amf":
|
||||
return "h264_amf"
|
||||
case "videotoolbox":
|
||||
return "h264_videotoolbox"
|
||||
default:
|
||||
return "libx264"
|
||||
}
|
||||
case "hevc", "h265":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "hevc_nvenc"
|
||||
case "qsv":
|
||||
return "hevc_qsv"
|
||||
case "vaapi":
|
||||
return "hevc_vaapi"
|
||||
case "amf":
|
||||
return "hevc_amf"
|
||||
case "videotoolbox":
|
||||
return "hevc_videotoolbox"
|
||||
default:
|
||||
return "libx265"
|
||||
}
|
||||
case "av1":
|
||||
switch hw {
|
||||
case "nvenc":
|
||||
return "av1_nvenc"
|
||||
case "qsv":
|
||||
return "av1_qsv"
|
||||
default:
|
||||
return "libsvtav1"
|
||||
}
|
||||
}
|
||||
return "libx264"
|
||||
}
|
||||
|
||||
func defaultPresetFor(encoder string) string {
|
||||
if strings.Contains(encoder, "nvenc") {
|
||||
return "p5"
|
||||
}
|
||||
if strings.Contains(encoder, "qsv") {
|
||||
return "medium"
|
||||
}
|
||||
if strings.Contains(encoder, "amf") {
|
||||
return "balanced"
|
||||
}
|
||||
if strings.Contains(encoder, "videotoolbox") {
|
||||
return "medium"
|
||||
}
|
||||
return "veryfast"
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "...(truncated)"
|
||||
}
|
||||
|
||||
// ProbeSource 调用 ffprobe 探测源文件
|
||||
func ProbeSource(url string) (SourceProbe, error) {
|
||||
if !ValidSourceInput(url) {
|
||||
return SourceProbe{}, fmt.Errorf("unsupported probe source")
|
||||
}
|
||||
probePath := setting.GetStr(conf.TranscodeFFprobePath, "ffprobe")
|
||||
args := []string{
|
||||
"-v", "error",
|
||||
"-protocol_whitelist", ProtocolWhitelistFor(url),
|
||||
"-show_entries", "format=duration,size,bit_rate:stream=codec_type,codec_name,bit_rate,width,height",
|
||||
"-of", "json",
|
||||
url,
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, probePath, args...)
|
||||
if runtime.GOOS == "windows" {
|
||||
// no special handling
|
||||
}
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return SourceProbe{}, err
|
||||
}
|
||||
var raw struct {
|
||||
Format struct {
|
||||
Duration string `json:"duration"`
|
||||
Size string `json:"size"`
|
||||
BitRate string `json:"bit_rate"`
|
||||
} `json:"format"`
|
||||
Streams []struct {
|
||||
CodecType string `json:"codec_type"`
|
||||
CodecName string `json:"codec_name"`
|
||||
BitRate string `json:"bit_rate"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
} `json:"streams"`
|
||||
}
|
||||
if err := json.Unmarshal(out, &raw); err != nil {
|
||||
return SourceProbe{}, err
|
||||
}
|
||||
p := SourceProbe{}
|
||||
p.Duration, _ = strconv.ParseFloat(raw.Format.Duration, 64)
|
||||
p.Size, _ = strconv.ParseInt(raw.Format.Size, 10, 64)
|
||||
for _, s := range raw.Streams {
|
||||
if s.CodecType == "video" && p.VideoCodec == "" {
|
||||
p.VideoCodec = s.CodecName
|
||||
p.Width = s.Width
|
||||
p.Height = s.Height
|
||||
p.VideoBitrate, _ = strconv.ParseInt(s.BitRate, 10, 64)
|
||||
}
|
||||
if s.CodecType == "audio" && p.AudioCodec == "" {
|
||||
p.AudioCodec = s.CodecName
|
||||
p.AudioBitrate, _ = strconv.ParseInt(s.BitRate, 10, 64)
|
||||
}
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
)
|
||||
|
||||
// Manager 全局转码管理器单例
|
||||
type Manager struct {
|
||||
Registry *WorkerRegistry
|
||||
Scheduler *Scheduler
|
||||
Cache *Cache
|
||||
Chunks *ChunkManager // 智能 chunk 调度器（按需启动 ffmpeg，支持随机访问）
|
||||
|
||||
localRunner *LocalRunner
|
||||
|
||||
stopCh chan struct{}
|
||||
once sync.Once
|
||||
}
|
||||
|
||||
var (
|
||||
defaultManager *Manager
|
||||
managerOnce sync.Once
|
||||
)
|
||||
|
||||
// Default 返回全局 Manager（懒加载）
|
||||
func Default() *Manager {
|
||||
managerOnce.Do(func() {
|
||||
defaultManager = newManager()
|
||||
})
|
||||
return defaultManager
|
||||
}
|
||||
|
||||
func newManager() *Manager {
|
||||
reg := NewWorkerRegistry()
|
||||
sch := NewScheduler(reg)
|
||||
cachePath := setting.GetStr(conf.TranscodeCachePath, "data/ctmp")
|
||||
// 兼容旧默认值：统一迁移到 data/ctmp
|
||||
if cachePath == "data/transcode_cache" || cachePath == "./data/transcode_cache" {
|
||||
cachePath = "data/ctmp"
|
||||
}
|
||||
cacheMaxGB := int64(setting.GetInt(conf.TranscodeCacheMaxGB, 20))
|
||||
cache := NewCache(cachePath, cacheMaxGB)
|
||||
_ = cache.Init()
|
||||
m := &Manager{
|
||||
Registry: reg,
|
||||
Scheduler: sch,
|
||||
Cache: cache,
|
||||
stopCh: make(chan struct{}),
|
||||
}
|
||||
m.Chunks = NewChunkManager(m)
|
||||
return m
|
||||
}
|
||||
|
||||
// Start 启动后台维护协程：心跳清理 + 缓存 LRU + 完成任务清理
|
||||
func (m *Manager) Start() {
|
||||
m.once.Do(func() {
|
||||
go m.maintenance()
|
||||
// 根据 run_mode 决定是否启动本地 worker
|
||||
mode := setting.GetStr(conf.TranscodeRunMode, "local")
|
||||
if mode == "local" || mode == "hybrid" {
|
||||
m.startLocalWorker()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Stop 停止 Manager
|
||||
func (m *Manager) Stop() {
|
||||
close(m.stopCh)
|
||||
if m.Chunks != nil {
|
||||
m.Chunks.Stop()
|
||||
}
|
||||
if m.localRunner != nil {
|
||||
m.localRunner.Stop()
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) maintenance() {
|
||||
t1 := time.NewTicker(15 * time.Second) // 心跳清理
|
||||
t2 := time.NewTicker(5 * time.Minute) // 缓存 LRU
|
||||
t3 := time.NewTicker(30 * time.Minute) // 历史 job 清理
|
||||
t4 := time.NewTicker(10 * time.Second) // 空闲超时检测
|
||||
defer t1.Stop()
|
||||
defer t2.Stop()
|
||||
defer t3.Stop()
|
||||
defer t4.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-m.stopCh:
|
||||
return
|
||||
case <-t1.C:
|
||||
removed := m.Registry.Sweep()
|
||||
for _, id := range removed {
|
||||
utils.Log.Infof("[transcode] worker %s removed (heartbeat timeout)", id)
|
||||
}
|
||||
case <-t2.C:
|
||||
_ = m.Cache.EnforceLRU()
|
||||
case <-t3.C:
|
||||
purged := m.Scheduler.PurgeBefore(time.Now().Add(-2 * time.Hour))
|
||||
if len(purged) > 0 {
|
||||
// 【内存泄漏修复】联动清理 Cache 中对应的 JobCache + segments map
|
||||
for _, jobID := range purged {
|
||||
m.Cache.Cleanup(jobID)
|
||||
}
|
||||
utils.Log.Infof("[transcode] purged %d finished jobs (cache cleaned)", len(purged))
|
||||
}
|
||||
case <-t4.C:
|
||||
m.sweepIdleJobs()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sweepIdleJobs 扫描所有活跃任务，超过空闲超时时间无播放端请求则自动取消并 kill FFmpeg
|
||||
// 注意：这里只 cancel job + kill ffmpeg 释放 CPU，**不删除已转好的切片缓存**。
|
||||
// 当用户继续播放时，TCSegment / TCPlaylist 会通过 Scheduler.Reactivate 复用同一 jobID/token
|
||||
// 重新启动转码；已存在的切片可以直接命中，不需要再次转码。
|
||||
func (m *Manager) sweepIdleJobs() {
|
||||
idleTimeoutSec := setting.GetInt(conf.TranscodeIdleTimeoutSec, 90)
|
||||
if idleTimeoutSec <= 0 {
|
||||
return // 配置为 0 表示禁用空闲超时
|
||||
}
|
||||
timeout := time.Duration(idleTimeoutSec) * time.Second
|
||||
now := time.Now()
|
||||
|
||||
activeJobs := m.Scheduler.ListActiveJobs()
|
||||
for _, j := range activeJobs {
|
||||
lastAccess := j.GetLastAccess()
|
||||
idle := now.Sub(lastAccess)
|
||||
if idle > timeout {
|
||||
utils.Log.Infof("[transcode] job %s idle for %.0fs (threshold %ds), pausing ffmpeg (cache kept for resume)",
|
||||
j.ID, idle.Seconds(), idleTimeoutSec)
|
||||
// 先取消所有 chunk ffmpeg，再调度器层面取消
|
||||
if m.Chunks != nil {
|
||||
m.Chunks.CancelAllForJob(j.ID)
|
||||
}
|
||||
m.Scheduler.Cancel(j.ID)
|
||||
// 【重要】不再调用 m.Cache.Cleanup —— 已转好的切片必须保留，
|
||||
// 这样浏览器恢复播放时旧的 seg-N.ts 仍然能命中，避免 404 和重新转码。
|
||||
// 真正的磁盘清理由更长周期的 PurgeBefore + Cache.EnforceLRU 负责。
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) startLocalWorker() {
|
||||
r := NewLocalRunner(m)
|
||||
if err := r.Start(); err != nil {
|
||||
utils.Log.Errorf("[transcode] start local worker failed: %+v", err)
|
||||
return
|
||||
}
|
||||
m.localRunner = r
|
||||
// 注入取消回调：Scheduler.Cancel 时直接 kill 本地 FFmpeg 进程
|
||||
m.Scheduler.SetOnCancel(func(jobID string) {
|
||||
// 先清理 chunk session（kill 所有 chunk ffmpeg）
|
||||
if m.Chunks != nil {
|
||||
m.Chunks.CancelAllForJob(jobID)
|
||||
}
|
||||
if m.localRunner != nil {
|
||||
m.localRunner.CancelJob(jobID)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// IsEnabled 总开关
|
||||
func IsEnabled() bool { return setting.GetBool(conf.TranscodeEnabled) }
|
||||
|
||||
// VerifyWorkerSecret 校验远程 worker 注册凭据
|
||||
func VerifyWorkerSecret(token string) bool {
|
||||
expected := setting.GetStr(conf.TranscodeWorkerSecret)
|
||||
if expected == "" {
|
||||
return false // 未配置则禁用远程注册
|
||||
}
|
||||
return token == expected
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package transcode
|
||||
|
||||
// 协议版本
|
||||
const ProtocolVersion = "v1"
|
||||
|
||||
// ---- 注册 ----
|
||||
|
||||
type RegisterRequest struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Capacity int `json:"capacity"`
|
||||
HWAccel []string `json:"hwaccel"`
|
||||
CodecsDecode []string `json:"codecs_decode"`
|
||||
CodecsEncode []string `json:"codecs_encode"`
|
||||
MaxResolution string `json:"max_resolution"`
|
||||
Tags []string `json:"tags"`
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
}
|
||||
|
||||
type RegisterResponse struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
HeartbeatInterval int `json:"heartbeat_interval"`
|
||||
ClaimStrategy string `json:"claim_strategy"` // pull
|
||||
ProtocolVersion string `json:"protocol_version"`
|
||||
}
|
||||
|
||||
// ---- 心跳 ----
|
||||
|
||||
type HeartbeatRequest struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
Load float64 `json:"load"`
|
||||
Running []string `json:"running"`
|
||||
FreeSlots int `json:"free_slots"`
|
||||
}
|
||||
|
||||
type HeartbeatResponse struct {
|
||||
OK bool `json:"ok"`
|
||||
Kick bool `json:"kick"` // 服务端要求 worker 退出
|
||||
Cancel []string `json:"cancel,omitempty"` // 需要取消的 job ids
|
||||
}
|
||||
|
||||
// ---- 领任务 ----
|
||||
|
||||
type ClaimRequest struct {
|
||||
WorkerID string `json:"worker_id"`
|
||||
Slots int `json:"slots"`
|
||||
Wait int `json:"wait"` // 长轮询超时秒，0=不等
|
||||
}
|
||||
|
||||
type ClaimResponse struct {
|
||||
Jobs []*Job `json:"jobs"`
|
||||
}
|
||||
|
||||
// ---- 完成 ----
|
||||
|
||||
type FinishRequest struct {
|
||||
JobID string `json:"job_id"`
|
||||
Status JobStatus `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Stats JobStats `json:"stats"`
|
||||
}
|
||||
|
||||
type JobStats struct {
|
||||
Elapsed float64 `json:"elapsed"`
|
||||
FPS float64 `json:"fps"`
|
||||
Speed float64 `json:"speed"`
|
||||
BytesOut int64 `json:"bytes_out"`
|
||||
}
|
||||
|
||||
// ---- 通用响应 ----
|
||||
|
||||
type ErrorResponse struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package transcode
|
||||
|
||||
import "strings"
|
||||
|
||||
// ffmpeg / ffprobe 支持 concat:、subfile:、data: 等大量协议。
|
||||
// SourceURL 最终来自请求中的路径或 Host 头，若不限制协议，攻击者就可能借 ffmpeg
|
||||
// 读取服务器本地文件或访问内网地址。这里按输入类型收敛到最小协议集。
|
||||
|
||||
const (
|
||||
// protocolWhitelistLocal 本地文件输入只允许 file 协议
|
||||
protocolWhitelistLocal = "file"
|
||||
// protocolWhitelistRemote 远程输入允许 HTTP(S) 及其底层传输；禁止 concat/subfile/data/pipe，
|
||||
// 同时去掉 file 与 crypto：HTTP 源用不到它们，留着等于给被构造的播放列表留了读本地文件的通道。
|
||||
protocolWhitelistRemote = "http,https,tcp,tls"
|
||||
)
|
||||
|
||||
// ProtocolWhitelistFor 根据输入地址返回应传给 ffmpeg/ffprobe 的 -protocol_whitelist 值。
|
||||
func ProtocolWhitelistFor(source string) string {
|
||||
s := strings.ToLower(strings.TrimSpace(source))
|
||||
if strings.HasPrefix(s, "http://") || strings.HasPrefix(s, "https://") {
|
||||
return protocolWhitelistRemote
|
||||
}
|
||||
return protocolWhitelistLocal
|
||||
}
|
||||
|
||||
// ValidSourceInput 判断输入是否可以安全交给 ffmpeg：
|
||||
// 只接受绝对本地路径或 http(s) URL，拒绝 concat:/subfile:/data:/pipe: 等协议前缀，
|
||||
// 也拒绝以 - 开头（会被当作 ffmpeg 选项）的值。
|
||||
func ValidSourceInput(source string) bool {
|
||||
s := strings.TrimSpace(source)
|
||||
if s == "" || strings.HasPrefix(s, "-") {
|
||||
return false
|
||||
}
|
||||
low := strings.ToLower(s)
|
||||
if strings.HasPrefix(low, "http://") || strings.HasPrefix(low, "https://") {
|
||||
return true
|
||||
}
|
||||
// 本地输入只接受绝对路径。
|
||||
// 不能按「冒号位置」判 scheme：文件名里带冒号非常常见
|
||||
// （/mnt/media/Star Wars: Episode IV.mkv），那样会被误判成协议前缀。
|
||||
// concat:/subfile:/data: 这些前缀都不是绝对路径，且还会被 -protocol_whitelist 挡住。
|
||||
// 注意不能用 filepath.IsAbs：它是平台相关的，Windows 上会拒绝 "/data/a.mkv"
|
||||
// 这类 VFS 绝对路径（OpenList 的路径始终是正斜杠开头）。
|
||||
return strings.HasPrefix(s, "/") || isWindowsDrivePath(s)
|
||||
}
|
||||
|
||||
// isWindowsDrivePath 判断 C:\ 或 C:/ 形式的盘符绝对路径。
|
||||
// 跨平台使用：在 Linux 上构建时会话也可能处理 Windows 风格路径。
|
||||
func isWindowsDrivePath(s string) bool {
|
||||
if len(s) < 3 || s[1] != ':' {
|
||||
return false
|
||||
}
|
||||
c := s[0]
|
||||
isLetter := (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|
||||
return isLetter && (s[2] == '\\' || s[2] == '/')
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// WorkerRegistry 维护当前在线的 Worker 池
|
||||
type WorkerRegistry struct {
|
||||
mu sync.RWMutex
|
||||
workers map[string]*Worker
|
||||
|
||||
heartbeatTimeout time.Duration
|
||||
}
|
||||
|
||||
func NewWorkerRegistry() *WorkerRegistry {
|
||||
return &WorkerRegistry{
|
||||
workers: make(map[string]*Worker),
|
||||
heartbeatTimeout: 30 * time.Second,
|
||||
}
|
||||
}
|
||||
|
||||
// Register 注册一个新的 worker，返回分配的 ID
|
||||
func (r *WorkerRegistry) Register(req *RegisterRequest, local bool) *Worker {
|
||||
w := &Worker{
|
||||
ID: "wk_" + uuid.NewString(),
|
||||
Name: req.Name,
|
||||
Version: req.Version,
|
||||
Capacity: req.Capacity,
|
||||
HWAccel: req.HWAccel,
|
||||
CodecsDecode: req.CodecsDecode,
|
||||
CodecsEncode: req.CodecsEncode,
|
||||
MaxResolution: req.MaxResolution,
|
||||
Tags: req.Tags,
|
||||
Endpoint: req.Endpoint,
|
||||
FreeSlots: req.Capacity,
|
||||
LastSeen: time.Now(),
|
||||
RegisteredAt: time.Now(),
|
||||
Local: local,
|
||||
}
|
||||
if w.Capacity <= 0 {
|
||||
w.Capacity = 1
|
||||
w.FreeSlots = 1
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.workers[w.ID] = w
|
||||
r.mu.Unlock()
|
||||
return w
|
||||
}
|
||||
|
||||
// Heartbeat 更新 worker 心跳信息，未注册返回 false
|
||||
func (r *WorkerRegistry) Heartbeat(req *HeartbeatRequest) (*Worker, bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
w, ok := r.workers[req.WorkerID]
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
w.LastSeen = time.Now()
|
||||
w.Load = req.Load
|
||||
w.Running = req.Running
|
||||
w.FreeSlots = req.FreeSlots
|
||||
return w, true
|
||||
}
|
||||
|
||||
// Get 获取 worker
|
||||
func (r *WorkerRegistry) Get(id string) (*Worker, bool) {
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
w, ok := r.workers[id]
|
||||
return w, ok
|
||||
}
|
||||
|
||||
// Unregister 主动移除 worker
|
||||
func (r *WorkerRegistry) Unregister(id string) {
|
||||
r.mu.Lock()
|
||||
delete(r.workers, id)
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// List 列出所有 worker（拷贝）
|
||||
func (r *WorkerRegistry) List() []*Worker {
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
out := make([]*Worker, 0, len(r.workers))
|
||||
for _, w := range r.workers {
|
||||
out = append(out, w)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Sweep 清理超时未心跳的 worker；返回被移除的 worker_id 列表
|
||||
func (r *WorkerRegistry) Sweep() []string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
now := time.Now()
|
||||
var removed []string
|
||||
for id, w := range r.workers {
|
||||
if w.Local {
|
||||
continue // 本地 Worker 不通过心跳判活
|
||||
}
|
||||
if now.Sub(w.LastSeen) > r.heartbeatTimeout {
|
||||
removed = append(removed, id)
|
||||
delete(r.workers, id)
|
||||
}
|
||||
}
|
||||
return removed
|
||||
}
|
||||
|
||||
// HasAvailable 是否存在有空闲槽位的 worker
|
||||
func (r *WorkerRegistry) HasAvailable() bool {
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
for _, w := range r.workers {
|
||||
if w.FreeSlots > 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,460 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Scheduler 任务调度器：维护 pending 队列并支持 worker pull 领取
|
||||
type Scheduler struct {
|
||||
mu sync.Mutex
|
||||
jobs map[string]*Job // 全部任务（含完成）
|
||||
pending []*Job // 待领取队列
|
||||
|
||||
// 等待新任务的 worker 通知
|
||||
waiters []chan struct{}
|
||||
|
||||
// cancellation 待派发给 worker 的取消列表（worker_id -> job_ids）
|
||||
pendingCancels map[string][]string
|
||||
|
||||
// onCancel 回调：Cancel 时同步通知本地 runner kill FFmpeg 进程
|
||||
// 由 Manager 在启动时注入
|
||||
onCancel func(jobID string)
|
||||
|
||||
registry *WorkerRegistry
|
||||
}
|
||||
|
||||
func NewScheduler(reg *WorkerRegistry) *Scheduler {
|
||||
return &Scheduler{
|
||||
jobs: make(map[string]*Job),
|
||||
pendingCancels: make(map[string][]string),
|
||||
registry: reg,
|
||||
}
|
||||
}
|
||||
|
||||
// SetOnCancel 注入取消回调（由 Manager 在启动本地 worker 后调用）
|
||||
func (s *Scheduler) SetOnCancel(fn func(jobID string)) {
|
||||
s.mu.Lock()
|
||||
s.onCancel = fn
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// 提交配额。转码是 CPU/磁盘密集型操作，必须限制单用户并发与全局排队长度，
|
||||
// 否则任何能访问播放接口的用户都能无限堆积任务（每个任务还会触发 30s 的 ffprobe）。
|
||||
const (
|
||||
// MaxPendingJobs 全局排队（未被 worker 领取）任务上限
|
||||
MaxPendingJobs = 200
|
||||
// MaxActiveJobsPerUser 单用户同时存在的未结束任务上限
|
||||
MaxActiveJobsPerUser = 3
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrQueueFull 全局队列已满
|
||||
ErrQueueFull = errors.New("transcode queue is full, try again later")
|
||||
// ErrUserQuota 单用户并发任务已达上限
|
||||
ErrUserQuota = errors.New("too many active transcode jobs for this user")
|
||||
// ErrJobNotFound 任务不存在或已被清理
|
||||
ErrJobNotFound = errors.New("transcode job not found")
|
||||
)
|
||||
|
||||
// isActiveStatus 任务是否仍占用资源（未结束）
|
||||
func isActiveStatus(st JobStatus) bool {
|
||||
return st == JobPending || st == JobRunning || st == JobReady
|
||||
}
|
||||
|
||||
// FindReusable 查找同一用户对同一路径、仍在进行中的任务，用于去重复用。
|
||||
func (s *Scheduler) FindReusable(ownerID string, path string) (*Job, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.findReusableLocked(ownerID, path)
|
||||
}
|
||||
|
||||
func (s *Scheduler) findReusableLocked(ownerID string, path string) (*Job, bool) {
|
||||
for _, j := range s.jobs {
|
||||
if j.OwnerID == ownerID && j.Path == path && isActiveStatus(j.GetStatus()) {
|
||||
return j, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// SubmitLimited 带去重与配额的提交：
|
||||
// 1. 同一用户对同一路径已有进行中的任务时直接复用，不重复入队；
|
||||
// 2. 全局排队过长或用户任务过多时拒绝。
|
||||
//
|
||||
// 返回 (job, reused, err)。
|
||||
func (s *Scheduler) SubmitLimited(j *Job) (*Job, bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if exist, ok := s.findReusableLocked(j.OwnerID, j.Path); ok {
|
||||
return exist, true, nil
|
||||
}
|
||||
if len(s.pending) >= MaxPendingJobs {
|
||||
return nil, false, ErrQueueFull
|
||||
}
|
||||
active := 0
|
||||
for _, x := range s.jobs {
|
||||
if x.OwnerID == j.OwnerID && isActiveStatus(x.GetStatus()) {
|
||||
active++
|
||||
}
|
||||
}
|
||||
if active >= MaxActiveJobsPerUser {
|
||||
return nil, false, ErrUserQuota
|
||||
}
|
||||
// 检查与入队在同一把锁内完成，避免并发请求同时越过配额
|
||||
return s.enqueueLocked(j), false, nil
|
||||
}
|
||||
|
||||
// Capacity 只读预检：在启动 ffprobe 这类昂贵操作之前先看配额是否已满。
|
||||
//
|
||||
// 真正的配额判定仍由 SubmitLimited 在同一把锁内完成；这里的目的有两个：
|
||||
// 1. 不让用户先等一次最长 30s 的 ffprobe 才收到 429；
|
||||
// 2. 把「并发请求同时通过配额判定」这个窗口内的探测数量压下来。
|
||||
func (s *Scheduler) Capacity(ownerID string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(s.pending) >= MaxPendingJobs {
|
||||
return ErrQueueFull
|
||||
}
|
||||
active := 0
|
||||
for _, x := range s.jobs {
|
||||
if x.OwnerID == ownerID && isActiveStatus(x.GetStatus()) {
|
||||
active++
|
||||
}
|
||||
}
|
||||
if active >= MaxActiveJobsPerUser {
|
||||
return ErrUserQuota
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Submit 提交一个任务并返回（带 ID + token）。不做配额限制，仅供内部流程（如重新激活）使用。
|
||||
func (s *Scheduler) Submit(j *Job) *Job {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.enqueueLocked(j)
|
||||
}
|
||||
|
||||
// enqueueLocked 补全任务字段并入队，调用方必须已持有 s.mu。
|
||||
func (s *Scheduler) enqueueLocked(j *Job) *Job {
|
||||
if j.ID == "" {
|
||||
j.ID = "job_" + uuid.NewString()
|
||||
}
|
||||
if j.CallbackToken == "" {
|
||||
j.CallbackToken = "tk_" + uuid.NewString()
|
||||
}
|
||||
// 播放凭据与 Worker 回写凭据分离：播放端只能读，不能上传切片或结束任务
|
||||
if j.PlayToken == "" {
|
||||
j.PlayToken = "pt_" + uuid.NewString()
|
||||
}
|
||||
if j.readyCh == nil {
|
||||
j.readyCh = make(chan struct{})
|
||||
}
|
||||
if j.CreatedAt.IsZero() {
|
||||
j.CreatedAt = time.Now()
|
||||
}
|
||||
j.Status = JobPending
|
||||
|
||||
s.jobs[j.ID] = j
|
||||
s.pending = append(s.pending, j)
|
||||
// 唤醒等待的 worker
|
||||
for _, ch := range s.waiters {
|
||||
close(ch)
|
||||
}
|
||||
s.waiters = nil
|
||||
return j
|
||||
}
|
||||
|
||||
// Get 通过 id 获取任务
|
||||
func (s *Scheduler) Get(id string) (*Job, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
j, ok := s.jobs[id]
|
||||
return j, ok
|
||||
}
|
||||
|
||||
// Claim 由 worker 调用领取任务（最多 slots 个）；如果没有任务则等待 wait 秒
|
||||
func (s *Scheduler) Claim(workerID string, slots int, wait time.Duration) []*Job {
|
||||
worker, ok := s.registry.Get(workerID)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
taken := s.claimNow(worker, slots)
|
||||
if len(taken) > 0 || wait <= 0 {
|
||||
return taken
|
||||
}
|
||||
// 长轮询等待
|
||||
s.mu.Lock()
|
||||
notify := make(chan struct{})
|
||||
s.waiters = append(s.waiters, notify)
|
||||
s.mu.Unlock()
|
||||
select {
|
||||
case <-notify:
|
||||
case <-time.After(wait):
|
||||
}
|
||||
return s.claimNow(worker, slots)
|
||||
}
|
||||
|
||||
func (s *Scheduler) claimNow(worker *Worker, slots int) []*Job {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if slots <= 0 || len(s.pending) == 0 {
|
||||
return nil
|
||||
}
|
||||
// 按打分排序候选任务，选最匹配的前 slots 个
|
||||
candidates := append([]*Job(nil), s.pending...)
|
||||
sort.SliceStable(candidates, func(i, j int) bool {
|
||||
return scoreJobForWorker(candidates[i], worker) > scoreJobForWorker(candidates[j], worker)
|
||||
})
|
||||
var taken []*Job
|
||||
for _, j := range candidates {
|
||||
if len(taken) >= slots {
|
||||
break
|
||||
}
|
||||
if !workerCanRun(worker, j) {
|
||||
continue
|
||||
}
|
||||
j.mu.Lock()
|
||||
j.Status = JobRunning
|
||||
j.WorkerID = worker.ID
|
||||
j.StartedAt = time.Now()
|
||||
j.mu.Unlock()
|
||||
taken = append(taken, j)
|
||||
// 从 pending 中移除
|
||||
for idx, pj := range s.pending {
|
||||
if pj == j {
|
||||
s.pending = append(s.pending[:idx], s.pending[idx+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return taken
|
||||
}
|
||||
|
||||
// scoreJobForWorker 给 (worker, job) 打分（数值越大越匹配）
|
||||
func scoreJobForWorker(j *Job, w *Worker) float64 {
|
||||
score := 1.0
|
||||
// 硬件加速匹配 + 50
|
||||
if len(j.Profiles) > 0 {
|
||||
req := strings.ToLower(j.Profiles[0].HWAccel)
|
||||
if req != "" && req != "none" && req != "auto" {
|
||||
for _, hw := range w.HWAccel {
|
||||
if strings.EqualFold(hw, req) {
|
||||
score += 50
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 空闲槽位多 +
|
||||
score += float64(w.FreeSlots) * 2
|
||||
// 负载低 +
|
||||
score += (1 - w.Load) * 10
|
||||
return score
|
||||
}
|
||||
|
||||
// workerCanRun 检查 worker 是否能运行该 job
|
||||
func workerCanRun(w *Worker, j *Job) bool {
|
||||
if w.FreeSlots <= 0 {
|
||||
return false
|
||||
}
|
||||
if j.Probe.VideoCodec != "" && len(w.CodecsDecode) > 0 {
|
||||
ok := false
|
||||
for _, c := range w.CodecsDecode {
|
||||
if strings.EqualFold(c, j.Probe.VideoCodec) {
|
||||
ok = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Finish 由 worker 调用，标记任务完成/失败
|
||||
func (s *Scheduler) Finish(req *FinishRequest) error {
|
||||
s.mu.Lock()
|
||||
j, ok := s.jobs[req.JobID]
|
||||
s.mu.Unlock()
|
||||
if !ok {
|
||||
return errors.New("job not found")
|
||||
}
|
||||
j.mu.Lock()
|
||||
defer j.mu.Unlock()
|
||||
j.Status = req.Status
|
||||
j.Error = req.Error
|
||||
j.FinishedAt = time.Now()
|
||||
// 任务一旦完成或失败，确保 readyCh 一定会被关闭，避免播放端死等
|
||||
j.readyOnce.Do(func() { close(j.readyCh) })
|
||||
return nil
|
||||
}
|
||||
|
||||
// Cancel 主动取消任务（同时通知对应 worker 并 kill 本地 FFmpeg 进程）
|
||||
func (s *Scheduler) Cancel(jobID string) {
|
||||
s.mu.Lock()
|
||||
j, ok := s.jobs[jobID]
|
||||
if !ok {
|
||||
s.mu.Unlock()
|
||||
return
|
||||
}
|
||||
j.mu.Lock()
|
||||
if j.Status == JobPending || j.Status == JobRunning || j.Status == JobReady {
|
||||
j.Status = JobCancelled
|
||||
}
|
||||
j.readyOnce.Do(func() { close(j.readyCh) })
|
||||
wid := j.WorkerID
|
||||
j.mu.Unlock()
|
||||
if wid != "" {
|
||||
s.pendingCancels[wid] = append(s.pendingCancels[wid], jobID)
|
||||
}
|
||||
// 从 pending 中剔除
|
||||
for idx, pj := range s.pending {
|
||||
if pj.ID == jobID {
|
||||
s.pending = append(s.pending[:idx], s.pending[idx+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
cancelFn := s.onCancel
|
||||
s.mu.Unlock()
|
||||
// 在锁外调用回调，避免死锁；回调会 kill 本地 FFmpeg 进程
|
||||
if cancelFn != nil {
|
||||
cancelFn(jobID)
|
||||
}
|
||||
}
|
||||
|
||||
// ListActiveJobs 列出所有活跃任务（pending/running/ready），用于空闲超时检测
|
||||
func (s *Scheduler) ListActiveJobs() []*Job {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []*Job
|
||||
for _, j := range s.jobs {
|
||||
st := j.GetStatus()
|
||||
if st == JobRunning || st == JobReady || st == JobPending {
|
||||
out = append(out, j)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// PopCancellations 取出某个 worker 待处理的取消列表（心跳响应中下发）
|
||||
func (s *Scheduler) PopCancellations(workerID string) []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
ids := s.pendingCancels[workerID]
|
||||
delete(s.pendingCancels, workerID)
|
||||
return ids
|
||||
}
|
||||
|
||||
// ListJobs 列出全部任务（仅用于调试/管理）
|
||||
func (s *Scheduler) ListJobs() []*Job {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]*Job, 0, len(s.jobs))
|
||||
for _, j := range s.jobs {
|
||||
out = append(out, j)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Reactivate 把一个已 Cancelled / Failed / Finished 的 job 重新激活回 Pending 状态，
|
||||
// 让 worker 能再次领取并跑转码。用于"播放端重新请求时按需重启转码"的场景。
|
||||
//
|
||||
// 设计点：
|
||||
// 1. 复用原 jobID 和两套 token（播放用 PlayToken / Worker 回写用 CallbackToken），
|
||||
// 前端持有的 master.m3u8 中的 URL 不需要变更
|
||||
// 2. 重置 readyCh / Status / WorkerID 等运行时字段，但保留 Path / SourceURL / Profiles / Probe
|
||||
// 3. 重新加入 pending 队列并唤醒等待的 worker
|
||||
//
|
||||
// 返回被激活的 Job；job 不存在返回 ErrJobNotFound，单用户配额不足返回 ErrUserQuota，
|
||||
// 全局排队已满返回 ErrQueueFull。
|
||||
func (s *Scheduler) Reactivate(jobID string) (*Job, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
j, ok := s.jobs[jobID]
|
||||
if !ok {
|
||||
return nil, ErrJobNotFound
|
||||
}
|
||||
// 已经活跃：无需重新入队
|
||||
if isActiveStatus(j.GetStatus()) {
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// 重新激活同样受全局排队上限约束：否则可以靠「先让任务 idle 取消、再批量激活」
|
||||
// 把 pending 队列顶到 MaxPendingJobs 之外。
|
||||
if len(s.pending) >= MaxPendingJobs {
|
||||
return nil, ErrQueueFull
|
||||
}
|
||||
|
||||
// 重新激活同样占用配额。否则持有 PlayToken 的用户可以先让任务因 idle 被取消，
|
||||
// 再统一请求 playlist / seg-N.ts 把它们一起重新激活，绕过 SubmitLimited 的单用户上限。
|
||||
active := 0
|
||||
for _, x := range s.jobs {
|
||||
if x == j {
|
||||
continue
|
||||
}
|
||||
if x.OwnerID == j.OwnerID && isActiveStatus(x.GetStatus()) {
|
||||
active++
|
||||
}
|
||||
}
|
||||
if active >= MaxActiveJobsPerUser {
|
||||
return nil, ErrUserQuota
|
||||
}
|
||||
|
||||
j.mu.Lock()
|
||||
// 期间可能已被其它请求激活
|
||||
if isActiveStatus(j.Status) {
|
||||
j.mu.Unlock()
|
||||
return j, nil
|
||||
}
|
||||
// 重置运行时状态
|
||||
j.Status = JobPending
|
||||
j.WorkerID = ""
|
||||
j.Error = ""
|
||||
j.FinishedAt = time.Time{}
|
||||
j.LastAccessAt = time.Now()
|
||||
// 重置 readyCh：原 channel 已 closed，需要全新一份给后续 WaitReady
|
||||
j.readyCh = make(chan struct{})
|
||||
j.readyOnce = sync.Once{}
|
||||
j.mu.Unlock()
|
||||
|
||||
// 重新入 pending 队列（先确认不在队列中以免重复）
|
||||
inQueue := false
|
||||
for _, pj := range s.pending {
|
||||
if pj.ID == jobID {
|
||||
inQueue = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !inQueue {
|
||||
s.pending = append(s.pending, j)
|
||||
}
|
||||
// 唤醒等待的 worker
|
||||
for _, ch := range s.waiters {
|
||||
close(ch)
|
||||
}
|
||||
s.waiters = nil
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// PurgeBefore 删除 finishedAt 早于某时刻的任务，避免 jobs map 无限增长
|
||||
// 返回被清理的 job ID 列表，调用方需联动清理 Cache
|
||||
func (s *Scheduler) PurgeBefore(t time.Time) []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var purged []string
|
||||
for id, j := range s.jobs {
|
||||
if (j.Status == JobFinished || j.Status == JobFailed || j.Status == JobCancelled) &&
|
||||
!j.FinishedAt.IsZero() && j.FinishedAt.Before(t) {
|
||||
delete(s.jobs, id)
|
||||
purged = append(purged, id)
|
||||
}
|
||||
}
|
||||
return purged
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func newTestScheduler() *Scheduler {
|
||||
return NewScheduler(NewWorkerRegistry())
|
||||
}
|
||||
|
||||
func TestSubmitLimitedReusesSameUserPath(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
a := NewJob()
|
||||
a.OwnerID, a.Path = "u1", "/movie.mkv"
|
||||
first, reused, err := s.SubmitLimited(a)
|
||||
if err != nil || reused {
|
||||
t.Fatalf("first submit: reused=%v err=%v", reused, err)
|
||||
}
|
||||
|
||||
b := NewJob()
|
||||
b.OwnerID, b.Path = "u1", "/movie.mkv"
|
||||
second, reused, err := s.SubmitLimited(b)
|
||||
if err != nil || !reused {
|
||||
t.Fatalf("second submit should reuse: reused=%v err=%v", reused, err)
|
||||
}
|
||||
if second.ID != first.ID {
|
||||
t.Fatalf("reuse returned a different job: %s vs %s", second.ID, first.ID)
|
||||
}
|
||||
if got := len(s.ListJobs()); got != 1 {
|
||||
t.Fatalf("duplicate job queued, jobs=%d", got)
|
||||
}
|
||||
|
||||
// 其他用户不应复用别人的任务（callback_token 属于发起者）
|
||||
c := NewJob()
|
||||
c.OwnerID, c.Path = "u2", "/movie.mkv"
|
||||
third, reused, err := s.SubmitLimited(c)
|
||||
if err != nil || reused || third.ID == first.ID {
|
||||
t.Fatalf("other user must get own job: reused=%v err=%v", reused, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubmitLimitedPerUserQuota(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
for i := 0; i < MaxActiveJobsPerUser; i++ {
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = "u7", fmt.Sprintf("/f%d.mkv", i)
|
||||
if _, _, err := s.SubmitLimited(j); err != nil {
|
||||
t.Fatalf("submit %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
over := NewJob()
|
||||
over.OwnerID, over.Path = "u7", "/over.mkv"
|
||||
if _, _, err := s.SubmitLimited(over); !errors.Is(err, ErrUserQuota) {
|
||||
t.Fatalf("want ErrUserQuota, got %v", err)
|
||||
}
|
||||
// 其它用户不受影响
|
||||
other := NewJob()
|
||||
other.OwnerID, other.Path = "u8", "/other.mkv"
|
||||
if _, _, err := s.SubmitLimited(other); err != nil {
|
||||
t.Fatalf("other user blocked: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Capacity 的判定必须与 SubmitLimited 一致：handler 靠它在启动 ffprobe 前预检配额。
|
||||
func TestCapacityMatchesSubmitLimitedQuota(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
if err := s.Capacity("cap-user"); err != nil {
|
||||
t.Fatalf("empty scheduler should have capacity: %v", err)
|
||||
}
|
||||
for i := 0; i < MaxActiveJobsPerUser; i++ {
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = "cap-user", fmt.Sprintf("/cap%d.mkv", i)
|
||||
if _, _, err := s.SubmitLimited(j); err != nil {
|
||||
t.Fatalf("submit %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if err := s.Capacity("cap-user"); !errors.Is(err, ErrUserQuota) {
|
||||
t.Fatalf("want ErrUserQuota, got %v", err)
|
||||
}
|
||||
// 其他用户不受影响
|
||||
if err := s.Capacity("cap-other"); err != nil {
|
||||
t.Fatalf("other user should have capacity: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubmitLimitedQueueCap(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
for i := 0; i < MaxPendingJobs; i++ {
|
||||
j := NewJob()
|
||||
// 每个任务不同用户，避免触发单用户上限
|
||||
j.OwnerID, j.Path = fmt.Sprintf("u%d", 1000+i), "/x.mkv"
|
||||
if _, _, err := s.SubmitLimited(j); err != nil {
|
||||
t.Fatalf("submit %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = "u99999", "/y.mkv"
|
||||
if _, _, err := s.SubmitLimited(j); !errors.Is(err, ErrQueueFull) {
|
||||
t.Fatalf("want ErrQueueFull, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubmitLimitedConcurrentQuota(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
accepted := 0
|
||||
for i := 0; i < 50; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = "u5", fmt.Sprintf("/p%d.mkv", i)
|
||||
if _, _, err := s.SubmitLimited(j); err == nil {
|
||||
mu.Lock()
|
||||
accepted++
|
||||
mu.Unlock()
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if accepted != MaxActiveJobsPerUser {
|
||||
t.Fatalf("concurrent submits accepted=%d, want %d", accepted, MaxActiveJobsPerUser)
|
||||
}
|
||||
}
|
||||
|
||||
// 重新激活（播放端请求 playlist / 切片时触发）也必须占配额，
|
||||
// 否则可以先让任务闲置被取消，再统一激活来绕过单用户上限。
|
||||
func TestReactivateHonorsUserQuota(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
for i := 0; i < MaxActiveJobsPerUser; i++ {
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = "u9", fmt.Sprintf("/r%d.mkv", i)
|
||||
if _, _, err := s.SubmitLimited(j); err != nil {
|
||||
t.Fatalf("submit %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
// 构造一个"该用户此前创建、随后因闲置被取消"的任务
|
||||
idle := NewJob()
|
||||
idle.OwnerID, idle.Path = "u10", "/idle.mkv"
|
||||
sub, _, err := s.SubmitLimited(idle)
|
||||
if err != nil {
|
||||
t.Fatalf("submit idle: %v", err)
|
||||
}
|
||||
sub.OwnerID = "u9"
|
||||
s.Cancel(sub.ID)
|
||||
|
||||
if _, rerr := s.Reactivate(sub.ID); !errors.Is(rerr, ErrUserQuota) {
|
||||
t.Fatalf("want ErrUserQuota, got %v", rerr)
|
||||
}
|
||||
|
||||
// 释放一个活跃任务后应能重新激活
|
||||
for _, j := range s.ListJobs() {
|
||||
if j.Path != "/idle.mkv" && j.OwnerID == "u9" {
|
||||
s.Cancel(j.ID)
|
||||
break
|
||||
}
|
||||
}
|
||||
if _, rerr := s.Reactivate(sub.ID); rerr != nil {
|
||||
t.Fatalf("reactivate after freeing quota: %v", rerr)
|
||||
}
|
||||
if st := sub.GetStatus(); st != JobPending {
|
||||
t.Fatalf("status=%v, want pending", st)
|
||||
}
|
||||
}
|
||||
|
||||
// 重新激活同样不能突破全局排队上限：否则可以先让任务闲置取消，再批量激活把队列顶爆。
|
||||
func TestReactivateHonorsQueueLimit(t *testing.T) {
|
||||
s := newTestScheduler()
|
||||
|
||||
// 先造一个随后被取消的任务（Cancel 会把它从 pending 中移除）
|
||||
idle := NewJob()
|
||||
idle.OwnerID, idle.Path = "q1", "/idle.mkv"
|
||||
sub, _, err := s.SubmitLimited(idle)
|
||||
if err != nil {
|
||||
t.Fatalf("submit idle: %v", err)
|
||||
}
|
||||
s.Cancel(sub.ID)
|
||||
|
||||
// 用不同用户把全局队列填满（避开单用户配额）
|
||||
for i := 0; i < MaxPendingJobs; i++ {
|
||||
j := NewJob()
|
||||
j.OwnerID, j.Path = fmt.Sprintf("q%d", i+2), fmt.Sprintf("/q%d.mkv", i)
|
||||
if _, _, err := s.SubmitLimited(j); err != nil {
|
||||
t.Fatalf("fill %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, rerr := s.Reactivate(sub.ID); !errors.Is(rerr, ErrQueueFull) {
|
||||
t.Fatalf("want ErrQueueFull, got %v", rerr)
|
||||
}
|
||||
if st := sub.GetStatus(); st != JobCancelled {
|
||||
t.Fatalf("status=%v, want cancelled (must not be requeued)", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidSourceInput(t *testing.T) {
|
||||
ok := []string{
|
||||
"/data/movies/a.mkv",
|
||||
"http://127.0.0.1:5244/d/a.mkv?sign=x",
|
||||
"https://example.com/d/a.mkv",
|
||||
`C:\media\a.mkv`,
|
||||
"C:/media/a.mkv",
|
||||
// 文件名里的冒号很常见，不能被当成协议前缀
|
||||
"/mnt/media/Star Wars: Episode IV.mkv",
|
||||
"/data/Spider-Man: Homecoming (2017).mkv",
|
||||
`D:\Movies\Star Wars: IV.mkv`,
|
||||
}
|
||||
for _, s := range ok {
|
||||
if !ValidSourceInput(s) {
|
||||
t.Errorf("%q should be accepted", s)
|
||||
}
|
||||
}
|
||||
bad := []string{
|
||||
"", "-i", "-f", "concat:/etc/passwd|/etc/hosts", "subfile,,start,0,end,0,,:/etc/passwd",
|
||||
"data:text/plain;base64,AAAA", "pipe:0", "rtmp://x/y", "ftp://x/y", "file:///etc/passwd",
|
||||
// 盘符后必须紧跟路径分隔符，否则不是合法的 Windows 绝对路径
|
||||
"C:foo",
|
||||
}
|
||||
for _, s := range bad {
|
||||
if ValidSourceInput(s) {
|
||||
t.Errorf("%q should be rejected", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProtocolWhitelistFor(t *testing.T) {
|
||||
if got := ProtocolWhitelistFor("/data/a.mkv"); got != "file" {
|
||||
t.Errorf("local = %q", got)
|
||||
}
|
||||
if got := ProtocolWhitelistFor("HTTPS://x/y"); got == "file" || got == "" {
|
||||
t.Errorf("remote = %q", got)
|
||||
}
|
||||
// 远程输入不应包含 file / crypto：HTTP 源用不到，留着等于给构造过的播放列表
|
||||
// 留了一条读本地文件的通道
|
||||
for _, banned := range []string{"concat", "subfile", "data", "pipe", "rtmp", "ftp", "file", "crypto"} {
|
||||
if contains := func(s string) bool {
|
||||
for _, p := range splitComma(s) {
|
||||
if p == banned {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}(ProtocolWhitelistFor("http://x/y")); contains {
|
||||
t.Errorf("remote whitelist must not include %s", banned)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func splitComma(s string) []string {
|
||||
var out []string
|
||||
cur := ""
|
||||
for _, r := range s {
|
||||
if r == ',' {
|
||||
out = append(out, cur)
|
||||
cur = ""
|
||||
continue
|
||||
}
|
||||
cur += string(r)
|
||||
}
|
||||
return append(out, cur)
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
// Package transcode 实现 OpenList 的 FFmpeg 云端/本地转码功能。
|
||||
// 设计目标：
|
||||
// 1. 总开关默认关，开启后按文件大小、源码率、源编码、文件后缀过滤决定是否转码。
|
||||
// 2. 协议自研，纯 HTTP/JSON，可挂接本地内置 Worker 或远程 Worker 节点池。
|
||||
// 3. 切片流式回吐，支持边转边播。
|
||||
package transcode
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// JobStatus 转码任务状态
|
||||
type JobStatus string
|
||||
|
||||
const (
|
||||
JobPending JobStatus = "pending" // 等待 Worker 领取
|
||||
JobRunning JobStatus = "running" // Worker 已领取并在转码
|
||||
JobReady JobStatus = "ready" // 至少首切片可用
|
||||
JobFinished JobStatus = "finished" // 全部切片完成
|
||||
JobFailed JobStatus = "failed"
|
||||
JobCancelled JobStatus = "cancelled"
|
||||
)
|
||||
|
||||
// OutputFormat 输出封装格式
|
||||
type OutputFormat string
|
||||
|
||||
const (
|
||||
FormatHLS OutputFormat = "hls"
|
||||
FormatDASH OutputFormat = "dash"
|
||||
FormatMP4 OutputFormat = "mp4"
|
||||
)
|
||||
|
||||
// HWAccel 硬件加速类型
|
||||
type HWAccel string
|
||||
|
||||
const (
|
||||
HWNone HWAccel = "none"
|
||||
HWAuto HWAccel = "auto"
|
||||
HWNVENC HWAccel = "nvenc" // NVIDIA
|
||||
HWQSV HWAccel = "qsv" // Intel QuickSync
|
||||
HWVAAPI HWAccel = "vaapi" // Linux VA-API
|
||||
HWAMF HWAccel = "amf" // AMD AMF
|
||||
HWVideoToolbox HWAccel = "videotoolbox" // macOS
|
||||
)
|
||||
|
||||
// SourceProbe 源文件探测信息（由 ffprobe 得出）
|
||||
type SourceProbe struct {
|
||||
Duration float64 `json:"duration"` // 秒
|
||||
Size int64 `json:"size"` // 字节
|
||||
VideoCodec string `json:"video_codec"` // h264 / hevc / av1 / vp9 ...
|
||||
VideoBitrate int64 `json:"video_bitrate"` // bps
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
AudioCodec string `json:"audio_codec"`
|
||||
AudioBitrate int64 `json:"audio_bitrate"`
|
||||
}
|
||||
|
||||
// Profile 输出转码档位
|
||||
type Profile struct {
|
||||
Name string `json:"name"` // 例如 1080p
|
||||
VideoCodec string `json:"video_codec"` // h264 / hevc / av1
|
||||
VideoBitrate string `json:"video_bitrate"` // 例如 4000k
|
||||
Scale string `json:"scale"` // 例如 1920:-2
|
||||
AudioCodec string `json:"audio_codec"` // aac / mp3 / opus / copy
|
||||
AudioBitrate string `json:"audio_bitrate"` // 例如 160k
|
||||
HWAccel string `json:"hwaccel"` // worker 端如何选择硬件加速
|
||||
}
|
||||
|
||||
// OutputSpec 输出参数
|
||||
type OutputSpec struct {
|
||||
Format OutputFormat `json:"format"`
|
||||
SegmentDuration int `json:"segment_duration"` // 秒
|
||||
}
|
||||
|
||||
// Job 一次转码任务
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
Path string `json:"path"` // OpenList 文件路径
|
||||
SourceURL string `json:"source_url"` // 提供给 Worker 的签名 URL
|
||||
Probe SourceProbe `json:"probe"`
|
||||
Profiles []Profile `json:"profiles"`
|
||||
Output OutputSpec `json:"output"`
|
||||
CallbackToken string `json:"callback_token"` // Worker 回吐切片用（同时具备写权限，绝不外泄给播放端）
|
||||
// PlayToken 播放凭据：只用于 master/variant playlist 与切片下发，
|
||||
// 与 Worker 回写凭据（CallbackToken）分离，避免播放器拿到可写权限。
|
||||
// json:"-" 保证下发给 Worker 的任务描述里不会包含它。
|
||||
PlayToken string `json:"-"`
|
||||
Deadline int64 `json:"deadline"` // unix 秒
|
||||
|
||||
// OwnerID 发起转码者的配额键，仅服务端用于去重与配额，不下发给 Worker。
|
||||
// 用字符串而不是用户 ID：访客共用一个账号，需要按 "guest:<clientIP>" 拆分配额，
|
||||
// 否则单个匿名访客就能占满所有匿名转码配额。
|
||||
OwnerID string `json:"-"`
|
||||
|
||||
// 运行时
|
||||
Status JobStatus `json:"status"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FinishedAt time.Time `json:"finished_at"`
|
||||
LastAccessAt time.Time `json:"last_access_at"` // 播放端最后一次请求 segment/playlist 的时间
|
||||
Error string `json:"error,omitempty"`
|
||||
|
||||
// 内部：用于"边转边播"——首切片就绪后关闭该 channel
|
||||
readyOnce sync.Once
|
||||
readyCh chan struct{}
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewJob 构造任务（自动初始化内部 channel）
|
||||
func NewJob() *Job {
|
||||
now := time.Now()
|
||||
return &Job{
|
||||
Status: JobPending,
|
||||
CreatedAt: now,
|
||||
LastAccessAt: now,
|
||||
readyCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// Touch 更新最后访问时间（播放端每次请求 segment/playlist 时调用）
|
||||
func (j *Job) Touch() {
|
||||
j.mu.Lock()
|
||||
j.LastAccessAt = time.Now()
|
||||
j.mu.Unlock()
|
||||
}
|
||||
|
||||
// GetLastAccess 获取最后访问时间（线程安全）
|
||||
func (j *Job) GetLastAccess() time.Time {
|
||||
j.mu.RLock()
|
||||
defer j.mu.RUnlock()
|
||||
return j.LastAccessAt
|
||||
}
|
||||
|
||||
// GetStatus 获取当前状态（线程安全）
|
||||
func (j *Job) GetStatus() JobStatus {
|
||||
j.mu.RLock()
|
||||
defer j.mu.RUnlock()
|
||||
return j.Status
|
||||
}
|
||||
|
||||
// MarkReady 标记任务已可播放（首切片就绪）
|
||||
func (j *Job) MarkReady() {
|
||||
j.readyOnce.Do(func() {
|
||||
j.mu.Lock()
|
||||
if j.Status == JobRunning || j.Status == JobPending {
|
||||
j.Status = JobReady
|
||||
}
|
||||
j.mu.Unlock()
|
||||
close(j.readyCh)
|
||||
})
|
||||
}
|
||||
|
||||
// WaitReady 阻塞等待首切片就绪
|
||||
func (j *Job) WaitReady() <-chan struct{} { return j.readyCh }
|
||||
|
||||
// Worker 节点信息
|
||||
type Worker struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Capacity int `json:"capacity"` // 最大并发任务数
|
||||
HWAccel []string `json:"hwaccel"` // 支持的硬件加速
|
||||
CodecsDecode []string `json:"codecs_decode"` // 可解码
|
||||
CodecsEncode []string `json:"codecs_encode"` // 可编码
|
||||
MaxResolution string `json:"max_resolution"` // 最大分辨率
|
||||
Tags []string `json:"tags"`
|
||||
Endpoint string `json:"endpoint,omitempty"` // 反向连接地址（可选）
|
||||
|
||||
// 运行时
|
||||
Load float64 `json:"load"` // 0~1
|
||||
Running []string `json:"running"` // 当前 job_ids
|
||||
FreeSlots int `json:"free_slots"`
|
||||
LastSeen time.Time `json:"last_seen"`
|
||||
RegisteredAt time.Time `json:"registered_at"`
|
||||
Local bool `json:"local"` // 是否本地内置 Worker
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/op"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/OpenListTeam/go-cache"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/pquerna/otp/totp"
|
||||
"gorm.io/gorm"
|
||||
@@ -48,21 +49,22 @@ func loginHash(c *gin.Context, req *LoginReq) {
|
||||
ip := c.ClientIP()
|
||||
count, ok := model.LoginCache.Get(ip)
|
||||
if ok && count >= model.DefaultMaxAuthRetries {
|
||||
// 被拒时不能再续期：自动重试的客户端（如凭据失效的 Subsonic/Web 轮询）
|
||||
// 会每来一次就刷新锁定窗口，把自己永久锁死。锁定窗口只在计数时设定。
|
||||
common.ErrorStrResp(c, model.TooManyAttempts, 429)
|
||||
model.LoginCache.Expire(ip, model.DefaultLockDuration)
|
||||
return
|
||||
}
|
||||
// check username
|
||||
user, err := op.GetUserByName(req.Username)
|
||||
if err != nil {
|
||||
common.ErrorStrResp(c, model.InvalidUsernameOrPassword, 401)
|
||||
model.LoginCache.Set(ip, count+1)
|
||||
model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration))
|
||||
return
|
||||
}
|
||||
// validate password hash
|
||||
if err := user.ValidatePwdStaticHash(req.Password); err != nil {
|
||||
common.ErrorStrResp(c, model.InvalidUsernameOrPassword, 401)
|
||||
model.LoginCache.Set(ip, count+1)
|
||||
model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration))
|
||||
return
|
||||
}
|
||||
// check 2FA
|
||||
@@ -70,7 +72,7 @@ func loginHash(c *gin.Context, req *LoginReq) {
|
||||
if !totp.Validate(req.OtpCode, user.OtpSecret) {
|
||||
// 402 - need opt
|
||||
common.ErrorStrResp(c, model.Invalid2FACode, 402)
|
||||
model.LoginCache.Set(ip, count+1)
|
||||
model.LoginCache.Set(ip, count+1, cache.WithEx[int](model.DefaultLockDuration))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
@@ -51,7 +51,9 @@ func Proxy(c *gin.Context) {
|
||||
common.ErrorPage(c, err, 500)
|
||||
return
|
||||
}
|
||||
if canProxy(storage, filename) {
|
||||
// 支持 ?force 参数强制代理（用于媒体库等需要 JS 加载文件内容的场景，避免 CORS 问题）
|
||||
_, forceProxy := c.GetQuery("force")
|
||||
if forceProxy || canProxy(storage, filename) {
|
||||
if _, ok := c.GetQuery("d"); !ok {
|
||||
if url := common.GenerateDownProxyURL(storage.GetStorage(), rawPath); url != "" {
|
||||
c.Redirect(302, url)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,681 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
stdpath "path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/imgmime"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/op"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// embyPlayTarget 播放目标（条目或某一集）
|
||||
type embyPlayTarget struct {
|
||||
Item *model.MediaItem
|
||||
Episode *model.EpisodeInfo
|
||||
RawPath string
|
||||
Container string
|
||||
Size int64
|
||||
Name string
|
||||
Duration int
|
||||
ItemID uint
|
||||
EpisodeNo int
|
||||
}
|
||||
|
||||
func (l *embyLibrary) playTarget(id string) (*embyPlayTarget, bool) {
|
||||
id = l.origID(id)
|
||||
switch {
|
||||
case strings.HasPrefix(id, "mv-"):
|
||||
n, err := strconv.ParseUint(strings.TrimPrefix(id, "mv-"), 10, 64)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
for i := range l.videos {
|
||||
if l.videos[i].ID == uint(n) {
|
||||
it := &l.videos[i]
|
||||
return &embyPlayTarget{
|
||||
Item: it, RawPath: MediaItemFilePath(it), Container: embyExt(it.FileName),
|
||||
Size: it.FileSize, Name: it.FileName, Duration: it.Duration, ItemID: it.ID,
|
||||
}, true
|
||||
}
|
||||
}
|
||||
case strings.HasPrefix(id, "so-"):
|
||||
it, ok := l.songByID(id)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
return &embyPlayTarget{
|
||||
Item: it, RawPath: MediaItemFilePath(it), Container: embyExt(it.FileName),
|
||||
Size: it.FileSize, Name: it.FileName, Duration: it.Duration, ItemID: it.ID,
|
||||
}, true
|
||||
case strings.HasPrefix(id, "ep-"):
|
||||
ep := l.findEpisode(id)
|
||||
if ep == nil {
|
||||
return nil, false
|
||||
}
|
||||
raw := MediaItemFilePath(ep.Item)
|
||||
if ep.Episode.FileName != "" {
|
||||
raw = stdpath.Join(mediaFolderOf(ep.Item), ep.Item.FileName, ep.Episode.FileName)
|
||||
}
|
||||
return &embyPlayTarget{
|
||||
Item: ep.Item, Episode: &ep.Episode, RawPath: raw, Container: embyExt(raw),
|
||||
Size: ep.Item.FileSize, Name: ep.Episode.FileName, Duration: ep.Item.Duration,
|
||||
ItemID: ep.Item.ID, EpisodeNo: ep.Index,
|
||||
}, true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// embyPlaybackInfo 返回直连播放信息
|
||||
func embyPlaybackInfo(c *gin.Context, lib *embyLibrary, id string) {
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
embyNotFound(c)
|
||||
return
|
||||
}
|
||||
api := common_ApiBase(c)
|
||||
isAudio := target.Item.MediaType == model.MediaTypeMusic
|
||||
kind, streamType := "Videos", "Video"
|
||||
if isAudio {
|
||||
kind, streamType = "Audio", "Audio"
|
||||
}
|
||||
// 流地址必须自带 api_key：播放器取流时不会再附加请求头，
|
||||
// 缺失会得到 401，客户端表现为"Playback of this content is currently restricted"
|
||||
q := url.Values{}
|
||||
q.Set("Static", "true")
|
||||
q.Set("MediaSourceId", id)
|
||||
if tok := embyRequestToken(c); tok != "" {
|
||||
q.Set("api_key", tok)
|
||||
}
|
||||
streamURL := fmt.Sprintf("%s/@emby/%s/%s/stream.%s?%s", api, kind, id, target.Container, q.Encode())
|
||||
if target.Container == "" {
|
||||
streamURL = fmt.Sprintf("%s/@emby/%s/%s/stream?%s", api, kind, id, q.Encode())
|
||||
}
|
||||
// 必须是 Http：客户端看到 File 会当成"服务器本地文件"，
|
||||
// 转而调用原生 LocalItemRepository 解析本地路径，必然失败（Playback Error）
|
||||
streams := []embyMediaStream{{Type: streamType, Index: 0, IsDefault: true}}
|
||||
var subtitleIndex *int
|
||||
if isAudio && target.Item.Lyrics != "" {
|
||||
// 歌词以"外挂字幕轨"的形式提供给客户端（Emby Theater 的歌词面板
|
||||
// 只遍历 Subtitle 流，并请求 Subtitles/{index}/Stream.js）
|
||||
lyricsIndex := len(streams)
|
||||
streams = append(streams, embyLyricsMediaStream(lyricsIndex))
|
||||
subtitleIndex = &lyricsIndex
|
||||
}
|
||||
source := embyMediaSource{
|
||||
Protocol: "Http",
|
||||
Id: id,
|
||||
ItemId: id,
|
||||
// Emby Theater 在本地播放时会把 Path 直接交给 mpv；
|
||||
// 填文件系统路径会让 mpv 去开本机不存在的文件（一直转圈/Playback Error），
|
||||
// 这里填可播放的 HTTP 流地址。
|
||||
Path: streamURL,
|
||||
Type: "Default",
|
||||
Container: target.Container,
|
||||
Name: target.Name,
|
||||
Size: target.Size,
|
||||
ETag: embyHash(fmt.Sprintf("%s|%d", target.RawPath, target.Size)),
|
||||
RunTimeTicks: embyTicks(target.Duration),
|
||||
SupportsDirectPlay: true,
|
||||
SupportsDirectStream: true,
|
||||
SupportsTranscoding: false,
|
||||
SupportsProbing: false,
|
||||
MediaStreams: streams,
|
||||
MediaAttachments: []any{},
|
||||
Formats: []string{},
|
||||
RequiredHttpHeaders: map[string]string{},
|
||||
DirectStreamUrl: streamURL,
|
||||
AddApiKeyToDirectStreamUrl: false,
|
||||
DefaultSubtitleStreamIndex: subtitleIndex,
|
||||
}
|
||||
c.JSON(200, gin.H{
|
||||
"MediaSources": []embyMediaSource{source},
|
||||
"PlaySessionId": fmt.Sprintf("openlist-%d-%d", target.ItemID, target.EpisodeNo),
|
||||
})
|
||||
}
|
||||
|
||||
// embyStream 302 到签名直链
|
||||
func embyStream(c *gin.Context, user *model.User, id string, isAudio bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
embyNotFound(c)
|
||||
return
|
||||
}
|
||||
url := SignedStreamURL(c, target.RawPath)
|
||||
c.Redirect(http.StatusFound, url)
|
||||
}
|
||||
|
||||
// ==================== 图片 ====================
|
||||
|
||||
var dataURIPrefix = "data:image/"
|
||||
|
||||
// embyImage 返回条目封面
|
||||
func embyImage(c *gin.Context, id string, segs []string, allowAnon bool) {
|
||||
var lib *embyLibrary
|
||||
if user, ok := embyAuth(c); ok {
|
||||
lib = loadEmbyLibrary(user)
|
||||
} else if allowAnon {
|
||||
// Emby 客户端的封面 URL 不带 token，只能允许匿名访问；
|
||||
// 但必须按访客身份做权限过滤，否则等于把整个媒体库（含 hidden、
|
||||
// base_path 之外、meta 受保护的条目）的封面暴露给未登录用户。
|
||||
// 访客被禁用时直接拒绝，不能退化成「全库可见」。
|
||||
guest, err := op.GetGuest()
|
||||
if err != nil || guest.Disabled {
|
||||
embyUnauthorized(c)
|
||||
return
|
||||
}
|
||||
lib = loadEmbyLibrary(guest)
|
||||
} else {
|
||||
embyUnauthorized(c)
|
||||
return
|
||||
}
|
||||
// 客户端回传的是 GUID，这里先还原成内部 ID 再做前缀判断。
|
||||
// 注意不能先走 resolveItem：匿名库的 user 为 nil，剧集条目会被权限校验判为不可见。
|
||||
id = lib.origID(id)
|
||||
var mediaItem *model.MediaItem
|
||||
switch {
|
||||
case strings.HasPrefix(id, "mv-"):
|
||||
n, _ := strconv.ParseUint(strings.TrimPrefix(id, "mv-"), 10, 64)
|
||||
for i := range lib.videos {
|
||||
if lib.videos[i].ID == uint(n) {
|
||||
mediaItem = &lib.videos[i]
|
||||
}
|
||||
}
|
||||
case strings.HasPrefix(id, "so-"):
|
||||
mediaItem, _ = lib.songByID(id)
|
||||
case strings.HasPrefix(id, "ep-"):
|
||||
// 图片 URL 不带 token，匿名库的 user 为 nil；
|
||||
// findEpisode 会做用户权限校验并直接失败，这里只按库内条目定位封面来源
|
||||
if it, ok := lib.episodeSourceItem(id); ok {
|
||||
mediaItem = it
|
||||
}
|
||||
case strings.HasPrefix(id, "sr-"):
|
||||
if s := lib.seriesByKey(strings.TrimPrefix(id, "sr-")); s != nil && len(s.Items) > 0 {
|
||||
mediaItem = &s.Items[0]
|
||||
}
|
||||
case strings.HasPrefix(id, "sn-"):
|
||||
rest := strings.TrimPrefix(id, "sn-")
|
||||
if idx := strings.LastIndex(rest, "-"); idx > 0 {
|
||||
if s := lib.seriesByKey(rest[:idx]); s != nil && len(s.Items) > 0 {
|
||||
mediaItem = &s.Items[0]
|
||||
}
|
||||
}
|
||||
case strings.HasPrefix(id, "al-"):
|
||||
if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 {
|
||||
mediaItem = &al.Items[0]
|
||||
}
|
||||
case strings.HasPrefix(id, "ar-"):
|
||||
key := strings.TrimPrefix(id, "ar-")
|
||||
for i := range lib.music {
|
||||
if artistKeyOf(&lib.music[i]) == key {
|
||||
mediaItem = &lib.music[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if mediaItem == nil {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
serveEmbyCover(c, mediaItem)
|
||||
}
|
||||
|
||||
// episodeSourceItem 按 ep-<itemID>-<num> 在库内定位剧集所属条目，不做权限校验。
|
||||
// 仅用于封面这类匿名可访问的资源，播放与条目详情仍走 findEpisode 的权限校验。
|
||||
func (l *embyLibrary) episodeSourceItem(id string) (*model.MediaItem, bool) {
|
||||
parts := strings.Split(strings.TrimPrefix(id, "ep-"), "-")
|
||||
if len(parts) != 2 {
|
||||
return nil, false
|
||||
}
|
||||
n, err := strconv.ParseUint(parts[0], 10, 64)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
for i := range l.videos {
|
||||
if l.videos[i].ID == uint(n) {
|
||||
return &l.videos[i], true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func serveEmbyCover(c *gin.Context, item *model.MediaItem) {
|
||||
cover := strings.TrimSpace(item.Cover)
|
||||
if cover == "" {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(cover, dataURIPrefix) {
|
||||
idx := strings.Index(cover, ";base64,")
|
||||
if idx < 0 {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
data, err := base64.StdEncoding.DecodeString(cover[idx+8:])
|
||||
if err != nil {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
// 数据库里的 data URI 声明的 MIME 不可信（历史数据可能来自内嵌标签/远端响应），
|
||||
// 响应头只使用按魔数嗅探出的四种栅格类型，SVG/HTML 一律拒绝。
|
||||
mime := imgmime.Sniff(data)
|
||||
if mime == "" {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
c.Header("X-Content-Type-Options", "nosniff")
|
||||
c.Header("Content-Disposition", `inline; filename="cover"`)
|
||||
c.Header("Content-Security-Policy", "default-src 'none'; sandbox")
|
||||
c.Header("Cache-Control", "private, max-age=86400")
|
||||
c.Data(200, mime, data)
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(cover, "http://") || strings.HasPrefix(cover, "https://") {
|
||||
c.Redirect(302, cover)
|
||||
return
|
||||
}
|
||||
// 本地缩略图路径
|
||||
c.Redirect(302, SignedStreamURL(c, cover))
|
||||
}
|
||||
|
||||
// ==================== 歌词 ====================
|
||||
|
||||
var lrcLineRe = regexp.MustCompile(`\[(\d{1,3}):(\d{1,2})(?:[.:](\d{1,3}))?\](.*)`)
|
||||
|
||||
func parseLrc(content string) []gin.H {
|
||||
var out []gin.H
|
||||
for _, line := range strings.Split(content, "\n") {
|
||||
m := lrcLineRe.FindStringSubmatch(strings.TrimSpace(line))
|
||||
if len(m) == 0 {
|
||||
continue
|
||||
}
|
||||
min, _ := strconv.Atoi(m[1])
|
||||
sec, _ := strconv.Atoi(m[2])
|
||||
ms := 0
|
||||
if m[3] != "" {
|
||||
frac := m[3]
|
||||
for len(frac) < 3 {
|
||||
frac += "0"
|
||||
}
|
||||
ms, _ = strconv.Atoi(frac[:3])
|
||||
}
|
||||
text := strings.TrimSpace(m[4])
|
||||
out = append(out, gin.H{
|
||||
"Text": text,
|
||||
"Start": int64(min*60000 + sec*1000 + ms),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func embyLyrics(c *gin.Context, user *model.User, id string) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
item, ok := lib.songByID(id)
|
||||
if !ok || item.Lyrics == "" {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
lines := parseLrc(item.Lyrics)
|
||||
c.JSON(200, gin.H{"Lyrics": lines, "IsSynced": len(lines) > 0})
|
||||
}
|
||||
|
||||
// embyLyricsMediaStream 构造歌词字幕轨（Emby 用外挂字幕承载歌词）。
|
||||
func embyLyricsMediaStream(index int) embyMediaStream {
|
||||
return embyMediaStream{
|
||||
Type: "Subtitle",
|
||||
Index: index,
|
||||
Codec: "lrc",
|
||||
Language: "und",
|
||||
DisplayTitle: "Lyrics",
|
||||
IsExternal: true,
|
||||
IsTextSubtitleStream: true,
|
||||
SupportsExternalStream: true,
|
||||
DeliveryMethod: "External",
|
||||
DeliveryUrl: "",
|
||||
}
|
||||
}
|
||||
|
||||
// embyLyricsTrackEvents 把 LRC 转成客户端歌词面板使用的 TrackEvent 列表。
|
||||
// 客户端按 StartPositionTicks 与当前播放位置比较（1ms = 10000 ticks）。
|
||||
// 没有时间轴的纯文本歌词会按行均分到歌曲时长，至少能顺序展示。
|
||||
func embyLyricsTrackEvents(content string, durationSec int) []gin.H {
|
||||
lines := parseLrc(content)
|
||||
if len(lines) == 0 {
|
||||
texts := make([]string, 0, 32)
|
||||
for _, l := range strings.Split(strings.ReplaceAll(content, "\r\n", "\n"), "\n") {
|
||||
if t := strings.TrimSpace(l); t != "" {
|
||||
texts = append(texts, t)
|
||||
}
|
||||
}
|
||||
if len(texts) > 0 {
|
||||
total := int64(durationSec) * 1000
|
||||
if total <= 0 {
|
||||
total = int64(len(texts)) * 5000
|
||||
}
|
||||
step := total / int64(len(texts))
|
||||
for i, t := range texts {
|
||||
lines = append(lines, gin.H{"Text": t, "Start": int64(i) * step})
|
||||
}
|
||||
}
|
||||
}
|
||||
events := make([]gin.H, 0, len(lines))
|
||||
for _, l := range lines {
|
||||
ms, _ := l["Start"].(int64)
|
||||
events = append(events, gin.H{
|
||||
"StartPositionTicks": ms * 10000,
|
||||
"Text": l["Text"],
|
||||
})
|
||||
}
|
||||
return events
|
||||
}
|
||||
|
||||
// embySubtitleStream 处理 /Items/{id}/{mediaSourceId}/Subtitles/{index}/Stream.js
|
||||
// Emby Theater 用它拉取歌词行（JSON 形式的 TrackEvents）
|
||||
func embySubtitleStream(c *gin.Context, user *model.User, id string) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
item, ok := lib.songByID(lib.origID(id))
|
||||
if !ok || item.Lyrics == "" {
|
||||
c.JSON(200, gin.H{"TrackEvents": []any{}})
|
||||
return
|
||||
}
|
||||
events := embyLyricsTrackEvents(item.Lyrics, item.Duration)
|
||||
c.JSON(200, gin.H{"TrackEvents": events, "TotalRecordCount": len(events)})
|
||||
}
|
||||
|
||||
// ==================== 播放会话 / 用户状态 ====================
|
||||
|
||||
type embyPlaybackBody struct {
|
||||
ItemId string `json:"ItemId"`
|
||||
ItemID string `json:"ItemID"`
|
||||
PositionTicks int64 `json:"PositionTicks"`
|
||||
PlayMethod string `json:"PlayMethod"`
|
||||
EventName string `json:"EventName"`
|
||||
MediaSourceId string `json:"MediaSourceId"`
|
||||
}
|
||||
|
||||
func (b embyPlaybackBody) itemID() string {
|
||||
if b.ItemId != "" {
|
||||
return b.ItemId
|
||||
}
|
||||
return b.ItemID
|
||||
}
|
||||
|
||||
func embySessionPlaying(c *gin.Context, user *model.User, action string) {
|
||||
var body embyPlaybackBody
|
||||
_ = c.ShouldBindJSON(&body)
|
||||
if action == "ping" {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
id := body.itemID()
|
||||
if id == "" {
|
||||
id = c.Query("ItemId")
|
||||
}
|
||||
if id == "" {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
stopped := action == "stopped" || action == "stop"
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
position := body.PositionTicks
|
||||
if position == 0 {
|
||||
if v, err := strconv.ParseInt(c.DefaultQuery("PositionTicks", "0"), 10, 64); err == nil {
|
||||
position = v
|
||||
}
|
||||
}
|
||||
played := false
|
||||
if stopped && target.Duration > 0 {
|
||||
ratio := float64(position) / float64(embyTicks(target.Duration))
|
||||
played = ratio >= 0.9
|
||||
}
|
||||
applyProgress(user, target, position, played, stopped)
|
||||
c.Status(204)
|
||||
}
|
||||
|
||||
func embySaveProgress(c *gin.Context, user *model.User, id string, start bool) {
|
||||
var body embyPlaybackBody
|
||||
_ = c.ShouldBindJSON(&body)
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
position := body.PositionTicks
|
||||
if position == 0 {
|
||||
position, _ = strconv.ParseInt(c.DefaultQuery("PositionTicks", "0"), 10, 64)
|
||||
}
|
||||
played := false
|
||||
stopped := c.Request.Method == "DELETE"
|
||||
if stopped && target.Duration > 0 {
|
||||
played = float64(position)/float64(embyTicks(target.Duration)) >= 0.9
|
||||
}
|
||||
applyProgress(user, target, position, played, stopped)
|
||||
c.Status(204)
|
||||
}
|
||||
|
||||
func applyProgress(user *model.User, target *embyPlayTarget, position int64, played bool, stopped bool) {
|
||||
if user == nil || target == nil {
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
d.PositionTicks = position
|
||||
if played {
|
||||
d.Played = true
|
||||
d.PositionTicks = 0
|
||||
d.PlayCount++
|
||||
}
|
||||
now := time.Now()
|
||||
d.LastPlayedAt = &now
|
||||
_ = db.SaveMediaUserData(d)
|
||||
}
|
||||
|
||||
func embySetPlayed(c *gin.Context, user *model.User, id string, played bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
id = lib.origID(id)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
// 剧集/季整体标记
|
||||
if strings.HasPrefix(id, "sr-") || strings.HasPrefix(id, "sn-") {
|
||||
embyMarkContainerPlayed(c, user, lib, id, played)
|
||||
return
|
||||
}
|
||||
embyNotFound(c)
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
d.Played = played
|
||||
if played {
|
||||
d.PositionTicks = 0
|
||||
d.PlayCount++
|
||||
} else {
|
||||
d.PlayCount = 0
|
||||
}
|
||||
now := time.Now()
|
||||
d.LastPlayedAt = &now
|
||||
_ = db.SaveMediaUserData(d)
|
||||
c.JSON(200, gin.H{
|
||||
"PlaybackPositionTicks": d.PositionTicks,
|
||||
"PlayCount": d.PlayCount,
|
||||
"Played": d.Played,
|
||||
"IsFavorite": d.IsFavorite,
|
||||
})
|
||||
}
|
||||
|
||||
func embyMarkContainerPlayed(c *gin.Context, user *model.User, lib *embyLibrary, id string, played bool) {
|
||||
var targets []*embyPlayTarget
|
||||
if strings.HasPrefix(id, "sr-") {
|
||||
key := strings.TrimPrefix(id, "sr-")
|
||||
for _, ep := range lib.episodesOf(key, 0) {
|
||||
if t, ok := lib.playTarget(ep.ID); ok {
|
||||
targets = append(targets, t)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
rest := strings.TrimPrefix(id, "sn-")
|
||||
idx := strings.LastIndex(rest, "-")
|
||||
if idx > 0 {
|
||||
key := rest[:idx]
|
||||
season, _ := strconv.Atoi(rest[idx+1:])
|
||||
for _, ep := range lib.episodesOf(key, season) {
|
||||
if t, ok := lib.playTarget(ep.ID); ok {
|
||||
targets = append(targets, t)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, t := range targets {
|
||||
d, err := db.GetMediaUserData(user.ID, t.ItemID, t.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: t.ItemID, EpisodeIndex: t.EpisodeNo}
|
||||
}
|
||||
d.Played = played
|
||||
d.PositionTicks = 0
|
||||
if !played {
|
||||
d.PlayCount = 0
|
||||
}
|
||||
_ = db.SaveMediaUserData(d)
|
||||
}
|
||||
c.Status(204)
|
||||
}
|
||||
|
||||
func embySetFavorite(c *gin.Context, user *model.User, id string, fav bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
embyNotFound(c)
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
d.IsFavorite = fav
|
||||
_ = db.SaveMediaUserData(d)
|
||||
c.JSON(200, gin.H{
|
||||
"PlaybackPositionTicks": d.PositionTicks,
|
||||
"PlayCount": d.PlayCount,
|
||||
"Played": d.Played,
|
||||
"IsFavorite": d.IsFavorite,
|
||||
})
|
||||
}
|
||||
|
||||
// embyUpdateUserData 部分更新用户状态
|
||||
func embyUpdateUserData(c *gin.Context, user *model.User, id string) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
embyNotFound(c)
|
||||
return
|
||||
}
|
||||
var body map[string]any
|
||||
_ = c.ShouldBindJSON(&body)
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
if v, ok := body["Played"].(bool); ok {
|
||||
d.Played = v
|
||||
if v {
|
||||
d.PositionTicks = 0
|
||||
}
|
||||
}
|
||||
if v, ok := body["IsFavorite"].(bool); ok {
|
||||
d.IsFavorite = v
|
||||
}
|
||||
if v, ok := body["PlaybackPositionTicks"].(float64); ok {
|
||||
d.PositionTicks = int64(v)
|
||||
}
|
||||
if v, ok := body["PlayCount"].(float64); ok {
|
||||
d.PlayCount = int(v)
|
||||
}
|
||||
_ = db.SaveMediaUserData(d)
|
||||
c.JSON(200, gin.H{
|
||||
"PlaybackPositionTicks": d.PositionTicks,
|
||||
"PlayCount": d.PlayCount,
|
||||
"Played": d.Played,
|
||||
"IsFavorite": d.IsFavorite,
|
||||
})
|
||||
}
|
||||
|
||||
// embyRefreshItem 触发条目所属扫描路径的增量扫描
|
||||
func embyRefreshItem(c *gin.Context, id string) {
|
||||
lib := loadEmbyLibrary(nil)
|
||||
var itemID uint
|
||||
switch {
|
||||
case strings.HasPrefix(id, "mv-"), strings.HasPrefix(id, "so-"):
|
||||
n, _ := strconv.ParseUint(id[3:], 10, 64)
|
||||
itemID = uint(n)
|
||||
case strings.HasPrefix(id, "ep-"):
|
||||
parts := strings.Split(id, "-")
|
||||
if len(parts) == 3 {
|
||||
n, _ := strconv.ParseUint(parts[1], 10, 64)
|
||||
itemID = uint(n)
|
||||
}
|
||||
case strings.HasPrefix(id, "sr-"):
|
||||
if s := lib.seriesByKey(strings.TrimPrefix(id, "sr-")); s != nil && len(s.Items) > 0 {
|
||||
itemID = s.Items[0].ID
|
||||
}
|
||||
case strings.HasPrefix(id, "al-"):
|
||||
if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 {
|
||||
itemID = al.Items[0].ID
|
||||
}
|
||||
}
|
||||
if itemID == 0 {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(itemID)
|
||||
if err != nil {
|
||||
c.Status(204)
|
||||
return
|
||||
}
|
||||
if sp, err := db.GetMediaScanPath(item.ScanPathID); err == nil && sp != nil {
|
||||
media.ScanMediaPath(sp)
|
||||
}
|
||||
c.Status(204)
|
||||
}
|
||||
|
||||
// ==================== 内部工具 ====================
|
||||
|
||||
func common_ApiBase(c *gin.Context) string {
|
||||
return strings.TrimSuffix(common.GetApiUrl(c.Request.Context()), "/")
|
||||
}
|
||||
|
||||
// embyBodyJSON 便于调试时输出请求体
|
||||
func embyBodyJSON(c *gin.Context) string {
|
||||
raw, _ := c.GetRawData()
|
||||
if len(raw) == 0 {
|
||||
return ""
|
||||
}
|
||||
var v any
|
||||
if err := json.Unmarshal(raw, &v); err != nil {
|
||||
return string(raw)
|
||||
}
|
||||
b, _ := json.Marshal(v)
|
||||
return string(b)
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,277 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
stdpath "path"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// MediaStatus 媒体库状态（前端据此决定是否渲染媒体库入口/侧栏）
|
||||
type MediaStatus struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Types []string `json:"types"`
|
||||
EmbyEnabled bool `json:"emby_enabled"`
|
||||
SubsonicEnabled bool `json:"subsonic_enabled"`
|
||||
EmbyBaseURL string `json:"emby_base_url"`
|
||||
SubsonicBaseURL string `json:"subsonic_base_url"`
|
||||
StrmPath string `json:"strm_path"`
|
||||
StrmOnScan bool `json:"strm_on_scan"`
|
||||
StrmSameDir bool `json:"strm_same_dir"`
|
||||
ThumbPath string `json:"thumb_path"`
|
||||
Version string `json:"version"`
|
||||
ProxyCover bool `json:"proxy_cover"`
|
||||
}
|
||||
|
||||
// PublicMediaStatus 公开的媒体库状态
|
||||
func PublicMediaStatus(c *gin.Context) {
|
||||
// Types 必须是非 nil 切片：前端会直接调用 .includes()，nil 序列化成 null 会抛异常
|
||||
status := MediaStatus{Enabled: MediaEnabled(), Types: []string{}}
|
||||
apiBase := common.GetApiUrl(c.Request.Context())
|
||||
status.StrmPath = media.DefaultStrmOptions(apiBase).OutputPath
|
||||
status.StrmOnScan = setting.GetBool(conf.MediaStrmOnScan)
|
||||
status.StrmSameDir = setting.GetBool(conf.MediaStrmSameDir)
|
||||
status.ThumbPath = setting.GetStr(conf.MediaThumbnailPath, "/imgs")
|
||||
status.Version = conf.Version
|
||||
status.ProxyCover = setting.GetBool(conf.MediaProxyCover)
|
||||
if !status.Enabled {
|
||||
common.SuccessResp(c, status)
|
||||
return
|
||||
}
|
||||
for _, mt := range MediaTypeEnabledList() {
|
||||
status.Types = append(status.Types, string(mt))
|
||||
}
|
||||
status.EmbyEnabled = MediaEmbyEnabled()
|
||||
status.SubsonicEnabled = MediaSubsonicEnabled()
|
||||
status.EmbyBaseURL = strings.TrimSuffix(apiBase, "/") + "/@emby"
|
||||
status.SubsonicBaseURL = strings.TrimSuffix(apiBase, "/") + "/@rest"
|
||||
common.SuccessResp(c, status)
|
||||
}
|
||||
|
||||
// ==================== 应用密钥 ====================
|
||||
|
||||
// ListMediaAppKeys 列出当前管理员的媒体应用密钥
|
||||
func ListMediaAppKeys(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "未登录", 401)
|
||||
return
|
||||
}
|
||||
keys, err := db.ListMediaAppKeys(user.ID)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, keys)
|
||||
}
|
||||
|
||||
// CreateMediaAppKey 创建应用密钥（明文只在创建时返回一次）
|
||||
func CreateMediaAppKey(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "未登录", 401)
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
_ = c.ShouldBindJSON(&req)
|
||||
name := strings.TrimSpace(req.Name)
|
||||
if name == "" {
|
||||
name = "Media Key"
|
||||
}
|
||||
plain, err := media.NewAppKeyPlain()
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
secret, err := media.EncryptAppKey(plain)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
key := &model.MediaAppKey{
|
||||
UserID: user.ID,
|
||||
Name: name,
|
||||
Prefix: media.AppKeyPrefix(plain),
|
||||
Secret: secret,
|
||||
}
|
||||
if err := db.CreateMediaAppKey(key); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, gin.H{
|
||||
"id": key.ID,
|
||||
"name": key.Name,
|
||||
"prefix": key.Prefix,
|
||||
"key": plain,
|
||||
})
|
||||
}
|
||||
|
||||
// DeleteMediaAppKey 删除应用密钥
|
||||
func DeleteMediaAppKey(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "未登录", 401)
|
||||
return
|
||||
}
|
||||
id, err := strconv.ParseUint(c.Query("id"), 10, 64)
|
||||
if err != nil {
|
||||
common.ErrorStrResp(c, "无效的ID", 400)
|
||||
return
|
||||
}
|
||||
if err := db.DeleteMediaAppKey(user.ID, uint(id)); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, nil)
|
||||
}
|
||||
|
||||
// ==================== STRM ====================
|
||||
|
||||
// GenerateMediaStrm 手动批量生成 STRM
|
||||
func GenerateMediaStrm(c *gin.Context) {
|
||||
var req struct {
|
||||
MediaType model.MediaType `json:"media_type"`
|
||||
ScanPathID uint `json:"scan_path_id"`
|
||||
}
|
||||
_ = c.ShouldBindJSON(&req)
|
||||
apiRoot := common.GetApiUrl(c.Request.Context())
|
||||
media.SetAPIRoot(apiRoot)
|
||||
opts := media.DefaultStrmOptions(apiRoot)
|
||||
if !opts.SameDir && opts.OutputPath == "" {
|
||||
common.ErrorStrResp(c, "未配置 STRM 输出目录", 400)
|
||||
return
|
||||
}
|
||||
n, err := media.GenerateStrmForType(c.Request.Context(), req.MediaType, req.ScanPathID, opts)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
outPath := opts.OutputPath
|
||||
if opts.SameDir {
|
||||
outPath = "与媒体文件同目录"
|
||||
}
|
||||
common.SuccessResp(c, gin.H{"written": n, "path": outPath})
|
||||
}
|
||||
|
||||
// GetMediaStreamURL 返回条目（或其中一集）的签名直链，供本地播放器/外链使用
|
||||
func GetMediaStreamURL(c *gin.Context) {
|
||||
// 直接按路径取签名链接（用于选集等场景）
|
||||
if rawPath := c.Query("path"); rawPath != "" {
|
||||
if !CanUserAccessPath(mediaUser(c), rawPath) {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, gin.H{
|
||||
"url": SignedStreamURL(c, rawPath),
|
||||
"path": rawPath,
|
||||
"name": stdpath.Base(rawPath),
|
||||
"proxied": SignedProxyURL(c, rawPath),
|
||||
})
|
||||
return
|
||||
}
|
||||
id, err := strconv.ParseUint(c.Query("id"), 10, 64)
|
||||
if err != nil {
|
||||
common.ErrorStrResp(c, "无效的ID", 400)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(uint(id))
|
||||
if err != nil || !CanUserAccessItem(mediaUser(c), item) {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
episodeIndex, _ := strconv.Atoi(c.DefaultQuery("episode", "0"))
|
||||
rawPath := MediaItemFilePath(item)
|
||||
if episodeIndex > 0 {
|
||||
var eps []model.EpisodeInfo
|
||||
if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil {
|
||||
for i := range eps {
|
||||
if eps[i].Index == episodeIndex || i+1 == episodeIndex {
|
||||
rawPath = MediaEpisodeFilePath(item, &eps[i])
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 选集是目录里的另一个文件，必须对最终文件再校验一次：
|
||||
// 目录可见不代表其中每个文件都可见（例如 meta 隐藏规则单独禁止某个文件），
|
||||
// 而签名之后的下载链路只验签名，不会重新执行用户的隐藏规则。
|
||||
if !CanUserAccessPath(mediaUser(c), rawPath) {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, gin.H{
|
||||
"url": SignedStreamURL(c, rawPath),
|
||||
"path": rawPath,
|
||||
"name": stdpath.Base(rawPath),
|
||||
"proxied": SignedProxyURL(c, rawPath),
|
||||
})
|
||||
}
|
||||
|
||||
// GetMediaStrm 动态生成单个条目的 .strm 内容（详情页下载）
|
||||
func GetMediaStrm(c *gin.Context) {
|
||||
id, err := strconv.ParseUint(c.Query("id"), 10, 64)
|
||||
if err != nil {
|
||||
common.ErrorStrResp(c, "无效的ID", 400)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(uint(id))
|
||||
if err != nil {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
if !CanUserAccessItem(mediaUser(c), item) {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
episodeIndex, _ := strconv.Atoi(c.DefaultQuery("episode", "0"))
|
||||
|
||||
var rawPath string
|
||||
var filename string
|
||||
if episodeIndex > 0 {
|
||||
var eps []model.EpisodeInfo
|
||||
if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil {
|
||||
for i := range eps {
|
||||
if eps[i].Index == episodeIndex || i+1 == episodeIndex {
|
||||
rawPath = MediaEpisodeFilePath(item, &eps[i])
|
||||
filename = fmt.Sprintf("%s S%02dE%02d.strm", item.ScrapedName, item.Season, episodeIndex)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if rawPath == "" {
|
||||
rawPath = MediaItemFilePath(item)
|
||||
if item.IsFolder {
|
||||
filename = item.ScrapedName + ".strm"
|
||||
} else {
|
||||
filename = strings.TrimSuffix(item.FileName, stdpath.Ext(item.FileName)) + ".strm"
|
||||
}
|
||||
}
|
||||
// 与 GetMediaStreamURL 同理：选集解析出来的是另一个文件，需要按最终路径再校验一次
|
||||
if !CanUserAccessPath(mediaUser(c), rawPath) {
|
||||
common.ErrorStrResp(c, "资源不存在", 404)
|
||||
return
|
||||
}
|
||||
if filename == "" {
|
||||
filename = "media.strm"
|
||||
}
|
||||
|
||||
apiRoot := common.GetApiUrl(c.Request.Context())
|
||||
opts := media.DefaultStrmOptions(apiRoot)
|
||||
content := media.RenderStrmContent(opts.Template, apiRoot, rawPath)
|
||||
c.Header("Content-Type", "text/plain; charset=utf-8")
|
||||
c.Header("Content-Disposition", fmt.Sprintf("attachment; filename*=UTF-8''%s", url.PathEscape(filename)))
|
||||
c.String(200, content)
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"crypto/subtle"
|
||||
"fmt"
|
||||
"net/url"
|
||||
stdpath "path"
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/errs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/op"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/sign"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// MediaEnabled 媒体库总开关是否开启
|
||||
func MediaEnabled() bool {
|
||||
return setting.GetBool(conf.MediaEnabled)
|
||||
}
|
||||
|
||||
// MediaEmbyEnabled Emby 兼容 API 是否可用（依赖总开关）
|
||||
func MediaEmbyEnabled() bool {
|
||||
return MediaEnabled() && setting.GetBool(conf.MediaEmbyEnabled)
|
||||
}
|
||||
|
||||
// MediaSubsonicEnabled OpenSubsonic API 是否可用（依赖总开关）
|
||||
func MediaSubsonicEnabled() bool {
|
||||
return MediaEnabled() && setting.GetBool(conf.MediaSubsonicEnabled)
|
||||
}
|
||||
|
||||
// RequireMediaEnabled 媒体库关闭时统一返回 404，使其表现为"功能不存在"
|
||||
func RequireMediaEnabled(c *gin.Context) {
|
||||
if !MediaEnabled() {
|
||||
c.AbortWithStatus(404)
|
||||
}
|
||||
}
|
||||
|
||||
// RequireMediaProtocol 校验指定协议开关
|
||||
func RequireMediaProtocol(enabled func() bool) gin.HandlerFunc {
|
||||
return func(c *gin.Context) {
|
||||
if !enabled() {
|
||||
c.AbortWithStatus(404)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// mediaUser 返回当前请求的登录用户，未登录时为 nil
|
||||
func mediaUser(c *gin.Context) *model.User {
|
||||
user, _ := c.Request.Context().Value(conf.UserKey).(*model.User)
|
||||
return user
|
||||
}
|
||||
|
||||
// CanUserAccessPath 判断用户是否有权访问指定 VFS 路径
|
||||
// 规则与文件浏览保持一致：base_path 前缀 + 元信息(meta) 读权限 + 隐藏文件规则。
|
||||
//
|
||||
// user 为 nil 一律拒绝（fail-closed）：所有调用点都应经过 Auth / embyAuth 拿到用户，
|
||||
// 缺失用户说明鉴权链路出了问题，不能默认放行；被禁用的访客同样拒绝。
|
||||
func CanUserAccessPath(user *model.User, reqPath string) bool {
|
||||
if user == nil || (user.IsGuest() && user.Disabled) {
|
||||
return false
|
||||
}
|
||||
reqPath = utils.FixAndCleanPath(reqPath)
|
||||
base := utils.FixAndCleanPath(user.BasePath)
|
||||
if base == "" {
|
||||
base = "/"
|
||||
}
|
||||
if base != "/" && !utils.IsSubPath(base, reqPath) && !utils.PathEqual(base, reqPath) {
|
||||
return false
|
||||
}
|
||||
meta, err := op.GetNearestMeta(reqPath)
|
||||
if err != nil && !errors.Is(errors.Cause(err), errs.MetaNotFound) {
|
||||
return false
|
||||
}
|
||||
// media 播放链路无法携带目录密码，因此受密码保护的目录视为不可访问
|
||||
return common.CanAccess(user, meta, reqPath, "")
|
||||
}
|
||||
|
||||
// userPathPrefix 返回用于 SQL 前缀过滤的用户目录，全盘用户返回空串
|
||||
func userPathPrefix(user *model.User) string {
|
||||
if user == nil {
|
||||
return ""
|
||||
}
|
||||
base := utils.FixAndCleanPath(user.BasePath)
|
||||
if base == "" || base == "/" {
|
||||
return ""
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// mediaFolderOf 返回条目所在目录
|
||||
func mediaFolderOf(item *model.MediaItem) string {
|
||||
if item == nil {
|
||||
return "/"
|
||||
}
|
||||
folder := strings.TrimSuffix(item.FolderPath, "/")
|
||||
if folder == "" {
|
||||
folder = "/"
|
||||
}
|
||||
return folder
|
||||
}
|
||||
|
||||
// MediaItemFilePath 拼接媒体条目对应的实际文件路径
|
||||
func MediaItemFilePath(item *model.MediaItem) string {
|
||||
if item == nil {
|
||||
return ""
|
||||
}
|
||||
folder := strings.TrimSuffix(item.FolderPath, "/")
|
||||
if folder == "" {
|
||||
folder = "/"
|
||||
}
|
||||
return stdpath.Join(folder, item.FileName)
|
||||
}
|
||||
|
||||
// MediaEpisodeFilePath 拼接文件夹型条目的单集文件路径
|
||||
func MediaEpisodeFilePath(item *model.MediaItem, ep *model.EpisodeInfo) string {
|
||||
if item == nil || ep == nil {
|
||||
return ""
|
||||
}
|
||||
return stdpath.Join(MediaItemFilePath(item), ep.FileName)
|
||||
}
|
||||
|
||||
// CanUserAccessItem 判断用户是否可访问某个媒体条目（含所属媒体类型开关）
|
||||
func CanUserAccessItem(user *model.User, item *model.MediaItem) bool {
|
||||
if item == nil || item.Hidden {
|
||||
return false
|
||||
}
|
||||
if !MediaTypeEnabled(item.MediaType) {
|
||||
return false
|
||||
}
|
||||
return CanUserAccessPath(user, MediaItemFilePath(item))
|
||||
}
|
||||
|
||||
// MediaTypeEnabled 判断某媒体类型是否启用扫描
|
||||
func MediaTypeEnabled(mediaType model.MediaType) bool {
|
||||
cfg, err := db.GetMediaConfig(mediaType)
|
||||
if err != nil || cfg == nil {
|
||||
return false
|
||||
}
|
||||
return cfg.Enabled
|
||||
}
|
||||
|
||||
// MediaTypeEnabledList 返回已启用的媒体类型
|
||||
func MediaTypeEnabledList() []model.MediaType {
|
||||
all := []model.MediaType{model.MediaTypeVideo, model.MediaTypeMusic, model.MediaTypeImage, model.MediaTypeBook}
|
||||
enabled := make([]model.MediaType, 0, len(all))
|
||||
for _, mt := range all {
|
||||
if MediaTypeEnabled(mt) {
|
||||
enabled = append(enabled, mt)
|
||||
}
|
||||
}
|
||||
return enabled
|
||||
}
|
||||
|
||||
// LookUpAppKeyUser 按明文应用密钥查找对应用户与密钥记录。
|
||||
// Emby 与 Subsonic 的所有密钥认证都走这里，避免两套实现产生行为差异。
|
||||
func LookUpAppKeyUser(key string) (*model.User, *model.MediaAppKey, bool) {
|
||||
key = strings.TrimSpace(key)
|
||||
if !strings.HasPrefix(key, "olmp_") {
|
||||
return nil, nil, false
|
||||
}
|
||||
keys, err := db.ListMediaAppKeysByPrefix(media.AppKeyPrefix(key))
|
||||
if err != nil {
|
||||
return nil, nil, false
|
||||
}
|
||||
for i := range keys {
|
||||
plain, err := media.DecryptAppKey(keys[i].Secret)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if subtle.ConstantTimeCompare([]byte(plain), []byte(key)) != 1 {
|
||||
continue
|
||||
}
|
||||
user, err := op.GetUserById(keys[i].UserID)
|
||||
if err != nil || user.Disabled || user.IsGuest() {
|
||||
return nil, nil, false
|
||||
}
|
||||
db.TouchMediaAppKey(keys[i].ID)
|
||||
return user, &keys[i], true
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// SignedStreamURL 构造带签名的 /d 直链（绝对地址，供 Emby/Subsonic/STRM 使用）
|
||||
func SignedStreamURL(c *gin.Context, rawPath string) string {
|
||||
return SignedStreamURLWithBase(common.GetApiUrl(c.Request.Context()), rawPath)
|
||||
}
|
||||
|
||||
// SignedStreamURLWithBase 使用指定站点前缀构造签名直链
|
||||
func SignedStreamURLWithBase(apiBase, rawPath string) string {
|
||||
apiBase = strings.TrimSuffix(apiBase, "/")
|
||||
p := rawPath
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
p = "/" + p
|
||||
}
|
||||
return fmt.Sprintf("%s/d%s?sign=%s", apiBase, utils.EncodePath(p), url.QueryEscape(sign.Sign(rawPath)))
|
||||
}
|
||||
|
||||
// SignedProxyURL 构造经服务端代理的签名链接（/p）
|
||||
func SignedProxyURL(c *gin.Context, rawPath string) string {
|
||||
apiBase := strings.TrimSuffix(common.GetApiUrl(c.Request.Context()), "/")
|
||||
p := rawPath
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
p = "/" + p
|
||||
}
|
||||
return fmt.Sprintf("%s/p%s?sign=%s", apiBase, utils.EncodePath(p), url.QueryEscape(sign.Sign(rawPath)))
|
||||
}
|
||||
|
||||
// MediaStreamURL 返回媒体条目（或其中一集）的签名直链
|
||||
func MediaStreamURL(c *gin.Context, item *model.MediaItem, ep *model.EpisodeInfo) string {
|
||||
if ep != nil {
|
||||
return SignedStreamURL(c, MediaEpisodeFilePath(item, ep))
|
||||
}
|
||||
return SignedStreamURL(c, MediaItemFilePath(item))
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
stdpath "path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/imgmime"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media/scraper"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// 允许代理的封面来源域名（TMDB / 豆瓣 / Discogs）。
|
||||
// 仅代理图片，避免该接口被当作通用反向代理滥用。
|
||||
var coverAllowedHosts = []string{
|
||||
"image.tmdb.org",
|
||||
"tmdb.org",
|
||||
"doubanio.com",
|
||||
"douban.com",
|
||||
"discogs.com",
|
||||
}
|
||||
|
||||
func hostAllowed(host string) bool {
|
||||
host = strings.ToLower(strings.TrimSuffix(host, "."))
|
||||
if host == "" || host == "localhost" || strings.HasSuffix(host, ".local") {
|
||||
return false
|
||||
}
|
||||
// 显式拒绝 IP 字面量，避免白名单域名被解析/重定向到内网
|
||||
if ip := net.ParseIP(host); ip != nil {
|
||||
return false
|
||||
}
|
||||
for _, h := range coverAllowedHosts {
|
||||
if host == h || strings.HasSuffix(host, "."+h) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// safeCoverURL 校验封面地址的协议与域名（含重定向后的每一跳）
|
||||
func safeCoverURL(u *url.URL) error {
|
||||
if u == nil {
|
||||
return fmt.Errorf("empty url")
|
||||
}
|
||||
if u.Scheme != "http" && u.Scheme != "https" {
|
||||
return fmt.Errorf("unsupported scheme")
|
||||
}
|
||||
if !hostAllowed(u.Hostname()) {
|
||||
return fmt.Errorf("host not allowed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PublicMediaProxy 代理外部封面图（TMDB/豆瓣/Discogs），解决客户端直连被墙的问题。
|
||||
// 用法：/api/public/media/proxy?url=https%3A%2F%2Fimage.tmdb.org%2F...
|
||||
func PublicMediaProxy(c *gin.Context) {
|
||||
if !MediaEnabled() {
|
||||
c.Status(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
raw := c.Query("url")
|
||||
if raw == "" {
|
||||
c.Status(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || safeCoverURL(u) != nil {
|
||||
c.Status(http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
resp, err := fetchCover(c.Request.Context(), u.String())
|
||||
if err != nil && setting.GetBool(conf.MediaCoverGateway) {
|
||||
// 直连失败时回退到公共图片网关（部分网络环境下 TMDB 图片域名不可达）。
|
||||
// 这会把封面 URL 交给第三方 wsrv.nl，因此默认关闭，需管理员显式开启。
|
||||
gateway := "https://wsrv.nl/?url=" + url.QueryEscape(strings.TrimPrefix(strings.TrimPrefix(u.String(), "https://"), "http://"))
|
||||
resp, err = fetchCover(c.Request.Context(), gateway)
|
||||
}
|
||||
if err != nil {
|
||||
utils.Log.Warnf("media cover proxy failed: %s: %v", u.String(), err)
|
||||
c.Status(http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
// 不信任上游 Content-Type：读取有限字节后按魔数判定，只输出四种栅格图片，
|
||||
// 避免 image/svg+xml 等可执行脚本的类型以本站源被浏览器渲染。
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
if err != nil || len(data) == 0 {
|
||||
c.Status(http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
ct := imgmime.Sniff(data)
|
||||
if ct == "" {
|
||||
c.Status(http.StatusUnsupportedMediaType)
|
||||
return
|
||||
}
|
||||
setSafeImageHeaders(c)
|
||||
c.Header("Cache-Control", "public, max-age=604800")
|
||||
c.Data(http.StatusOK, ct, data)
|
||||
}
|
||||
|
||||
// setSafeImageHeaders 为用户可影响内容的图片响应加上防 XSS 头。
|
||||
func setSafeImageHeaders(c *gin.Context) {
|
||||
c.Header("X-Content-Type-Options", "nosniff")
|
||||
c.Header("Content-Disposition", `inline; filename="cover"`)
|
||||
c.Header("Content-Security-Policy", "default-src 'none'; sandbox")
|
||||
}
|
||||
|
||||
// fetchCover 拉取图片，仅接受 200 且 Content-Type 为 image/*
|
||||
// 重定向的每一跳都重新做域名校验，避免白名单域名 302 到内网造成 SSRF。
|
||||
func fetchCover(ctx context.Context, raw string) (*http.Response, error) {
|
||||
client := &http.Client{
|
||||
Timeout: 20 * time.Second,
|
||||
CheckRedirect: func(req *http.Request, via []*http.Request) error {
|
||||
if len(via) >= 3 {
|
||||
return fmt.Errorf("too many redirects")
|
||||
}
|
||||
return safeCoverURL(req.URL)
|
||||
},
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, raw, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; OpenList-Media/1.0)")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
_ = resp.Body.Close()
|
||||
return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// thumbnailRoot 返回运行目录模式下缩略图的本地根目录（<data>/<thumbnail_path>）
|
||||
func thumbnailRoot() string {
|
||||
dir := setting.GetStr(conf.MediaThumbnailPath, "imgs")
|
||||
return scraper.RuntimeThumbRoot(dir)
|
||||
}
|
||||
|
||||
// PublicMediaThumb 读取本地缩略图目录中的文件。
|
||||
// 缩略图由刮削器写入本地磁盘（非 VFS），因此不能用 /d 读取。
|
||||
// 用法：/api/public/media/thumb?path=/imgs/xxx.jpg
|
||||
func PublicMediaThumb(c *gin.Context) {
|
||||
if !MediaEnabled() {
|
||||
c.Status(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
reqPath := strings.TrimPrefix(stdpath.Clean("/"+c.Query("path")), "/")
|
||||
// stdpath.Clean 已消解 ../ 段；这里不能再用 Contains("..") 粗暴过滤，会误杀 a..b.jpg 这类合法文件名。
|
||||
// 越界由下方 filepath.Rel 校验兜底。
|
||||
if reqPath == "" || reqPath == "." {
|
||||
c.Status(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
base := strings.Trim(setting.GetStr(conf.MediaThumbnailPath, "/imgs"), "/")
|
||||
if base != "" {
|
||||
if !strings.HasPrefix(reqPath, base+"/") {
|
||||
c.Status(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
reqPath = strings.TrimPrefix(reqPath, base+"/")
|
||||
}
|
||||
root := thumbnailRoot()
|
||||
full := filepath.Join(root, filepath.FromSlash(reqPath))
|
||||
rel, err := filepath.Rel(root, full)
|
||||
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
c.Status(http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
// 只按扩展名白名单输出栅格图片，拒绝 svg / html 等可执行类型
|
||||
var ct string
|
||||
switch strings.ToLower(filepath.Ext(full)) {
|
||||
case ".jpg", ".jpeg":
|
||||
ct = "image/jpeg"
|
||||
case ".png":
|
||||
ct = "image/png"
|
||||
case ".gif":
|
||||
ct = "image/gif"
|
||||
case ".webp":
|
||||
ct = "image/webp"
|
||||
default:
|
||||
c.Status(http.StatusUnsupportedMediaType)
|
||||
return
|
||||
}
|
||||
setSafeImageHeaders(c)
|
||||
c.Header("Content-Type", ct)
|
||||
c.Header("Cache-Control", "public, max-age=604800")
|
||||
c.File(full)
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/go-cache"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// 媒体协议（Emby / Subsonic）的认证入口不经过 /api/auth/login，
|
||||
// 因此必须自行复用 LoginCache 的失败计数与锁定策略，否则等于附送一个无限次数的爆破接口。
|
||||
// 使用独立的 key 前缀，避免 Subsonic 客户端的失败重试把 Web 登录一并锁死。
|
||||
|
||||
func mediaAuthKey(c *gin.Context) string {
|
||||
return "media:" + c.ClientIP()
|
||||
}
|
||||
|
||||
// mediaAuthLocked 当前 IP 是否已达到失败上限（需要拒绝认证尝试）
|
||||
// 注意：这里绝不能刷新过期时间。客户端（尤其是 Subsonic）会持续轮询重试，
|
||||
// 每次被拒都续期会把 IP 永久锁死，连之后正确的密钥也无法登录。
|
||||
func mediaAuthLocked(c *gin.Context) bool {
|
||||
count, ok := model.LoginCache.Get(mediaAuthKey(c))
|
||||
return ok && count >= model.DefaultMaxAuthRetries
|
||||
}
|
||||
|
||||
// bumpMediaAuthCount 增加一次失败计数。
|
||||
// 计数必须带 TTL：key 里包含 IP / 用户名（后者来自请求参数，可被随意伪造），
|
||||
// 不带过期时间会让缓存随请求无限增长；固定窗口也让锁定到期后计数自动归零。
|
||||
// 达到上限后请求会被 mediaAuthLocked / mediaAuthLockedFor 拦截，不会再走到这里，
|
||||
// 因此锁定窗口不会被持续刷新。
|
||||
func bumpMediaAuthCount(key string) {
|
||||
count, _ := model.LoginCache.Get(key)
|
||||
count++
|
||||
model.LoginCache.Set(key, count, cache.WithEx[int](model.DefaultLockDuration))
|
||||
}
|
||||
|
||||
// mediaAuthFailed 记录一次认证失败（IP 维度）
|
||||
func mediaAuthFailed(c *gin.Context) {
|
||||
bumpMediaAuthCount(mediaAuthKey(c))
|
||||
}
|
||||
|
||||
// mediaAuthSucceeded 认证成功后清除失败计数
|
||||
func mediaAuthSucceeded(c *gin.Context) {
|
||||
model.LoginCache.Del(mediaAuthKey(c))
|
||||
}
|
||||
|
||||
// ==================== 用户名维度 ====================
|
||||
// 只按 IP 计数是可以被绕过的：gin 默认信任所有代理（仓库里没有设置
|
||||
// SetTrustedProxies），客户端自带 X-Forwarded-For 就能改写 c.ClientIP()，
|
||||
// 轮换这个头即可无限次尝试。因此再按用户名记一份失败次数：它与 IP 无关，
|
||||
// 即使 IP 被伪造，对同一账号的爆破依然会被锁住。
|
||||
// （代理信任范围属于项目层面的问题，主干 /api/auth/login 现状相同。）
|
||||
|
||||
func mediaAuthUserKey(username string) string {
|
||||
return "media:user:" + strings.ToLower(strings.TrimSpace(username))
|
||||
}
|
||||
|
||||
// mediaAuthLockedFor 本次请求是否应被限流：IP 或用户名任一维度达到上限即拒绝。
|
||||
func mediaAuthLockedFor(c *gin.Context, username string) bool {
|
||||
if mediaAuthLocked(c) {
|
||||
return true
|
||||
}
|
||||
if strings.TrimSpace(username) == "" {
|
||||
return false
|
||||
}
|
||||
count, ok := model.LoginCache.Get(mediaAuthUserKey(username))
|
||||
return ok && count >= model.DefaultMaxAuthRetries
|
||||
}
|
||||
|
||||
// bumpMediaAuthUser 给用户名维度记一次失败
|
||||
func bumpMediaAuthUser(username string) {
|
||||
name := strings.TrimSpace(username)
|
||||
if name == "" {
|
||||
return
|
||||
}
|
||||
bumpMediaAuthCount(mediaAuthUserKey(name))
|
||||
}
|
||||
|
||||
// mediaAuthFailedFor 记录一次认证失败：同时计入 IP 与用户名两个维度。
|
||||
func mediaAuthFailedFor(c *gin.Context, username string) {
|
||||
mediaAuthFailed(c)
|
||||
bumpMediaAuthUser(username)
|
||||
}
|
||||
|
||||
// mediaAuthSucceededFor 认证成功后清除 IP 与用户名两个维度的计数。
|
||||
func mediaAuthSucceededFor(c *gin.Context, username string) {
|
||||
mediaAuthSucceeded(c)
|
||||
if name := strings.TrimSpace(username); name != "" {
|
||||
model.LoginCache.Del(mediaAuthUserKey(name))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
const (
|
||||
// 单个条目最多保存的批注数，防止恶意请求把表撑大
|
||||
maxMediaAnnotations = 500
|
||||
// 单条批注对象（含绘制点集）的 JSON 上限
|
||||
maxMediaAnnotationAnchor = 512 << 10
|
||||
// 单次请求的批注对象总量上限，避免一次塞进几百个大体积批注
|
||||
maxMediaAnnotationTotal = 8 << 20
|
||||
)
|
||||
|
||||
// mediaAnnotationTarget 校验条目存在且当前用户可以写入批注。
|
||||
// 批注是用户私有数据，不允许访客：访客是公共账号，多设备之间的批注会互相覆盖。
|
||||
func mediaAnnotationTarget(c *gin.Context, itemID uint) (*model.MediaItem, *model.User, bool) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return nil, nil, false
|
||||
}
|
||||
if user.IsGuest() {
|
||||
common.ErrorStrResp(c, "guest cannot save annotations", 403)
|
||||
return nil, nil, false
|
||||
}
|
||||
if itemID == 0 {
|
||||
common.ErrorStrResp(c, "invalid request", 400)
|
||||
return nil, nil, false
|
||||
}
|
||||
item, err := db.GetMediaItemByID(itemID)
|
||||
if err != nil || item.Hidden || item.DeletedAt.Valid || !CanUserAccessItem(user, item) {
|
||||
common.ErrorStrResp(c, "not found", 404)
|
||||
return nil, nil, false
|
||||
}
|
||||
return item, user, true
|
||||
}
|
||||
|
||||
// ListMediaAnnotationsAPI 列出当前用户对某个条目保存的批注
|
||||
// GET /fs/media/annotations?item_id=1
|
||||
func ListMediaAnnotationsAPI(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
itemID := uint(atoiDefault(c.Query("item_id"), 0))
|
||||
// 未登录 / 访客 / 无权访问一律返回空列表：这是私有数据，不对访客暴露
|
||||
if user == nil || user.IsGuest() || itemID == 0 {
|
||||
common.SuccessResp(c, []model.MediaAnnotation{})
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(itemID)
|
||||
if err != nil || item.Hidden || item.DeletedAt.Valid || !CanUserAccessItem(user, item) {
|
||||
common.SuccessResp(c, []model.MediaAnnotation{})
|
||||
return
|
||||
}
|
||||
list, err := db.ListMediaAnnotations(user.ID, item.ID)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, list)
|
||||
}
|
||||
|
||||
type mediaAnnotationPayload struct {
|
||||
RemoteID string `json:"remote_id"`
|
||||
Anchor string `json:"anchor"`
|
||||
}
|
||||
|
||||
type mediaAnnotationSyncReq struct {
|
||||
ItemID uint `json:"item_id"`
|
||||
Upserts []mediaAnnotationPayload `json:"upserts"`
|
||||
Deletes []string `json:"deletes"`
|
||||
}
|
||||
|
||||
// SyncMediaAnnotationsAPI 增量保存批注：upserts 覆盖同一 remote_id，deletes 按 remote_id 删除。
|
||||
// 查看器每次改动只会带上变化的那几条，不需要整份文档重传。
|
||||
// POST /fs/media/annotations
|
||||
func SyncMediaAnnotationsAPI(c *gin.Context) {
|
||||
var req mediaAnnotationSyncReq
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorStrResp(c, "invalid request", 400)
|
||||
return
|
||||
}
|
||||
item, user, ok := mediaAnnotationTarget(c, req.ItemID)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if len(req.Upserts) > maxMediaAnnotations || len(req.Deletes) > maxMediaAnnotations {
|
||||
common.ErrorStrResp(c, "too many annotations", 400)
|
||||
return
|
||||
}
|
||||
|
||||
upserts := make([]model.MediaAnnotation, 0, len(req.Upserts))
|
||||
seen := make(map[string]struct{}, len(req.Upserts))
|
||||
anchorBytes := 0
|
||||
for _, p := range req.Upserts {
|
||||
remoteID := strings.TrimSpace(p.RemoteID)
|
||||
// remote_id 是查看器分配的批注主键，多设备同步时靠它去重，必须存在且唯一
|
||||
if remoteID == "" || p.Anchor == "" || len(remoteID) > 128 {
|
||||
common.ErrorStrResp(c, "invalid annotation", 400)
|
||||
return
|
||||
}
|
||||
if _, dup := seen[remoteID]; dup {
|
||||
continue
|
||||
}
|
||||
seen[remoteID] = struct{}{}
|
||||
if len(p.Anchor) > maxMediaAnnotationAnchor {
|
||||
common.ErrorStrResp(c, "annotation too large", 400)
|
||||
return
|
||||
}
|
||||
anchorBytes += len(p.Anchor)
|
||||
if anchorBytes > maxMediaAnnotationTotal {
|
||||
common.ErrorStrResp(c, "annotations too large", 400)
|
||||
return
|
||||
}
|
||||
upserts = append(upserts, model.MediaAnnotation{
|
||||
RemoteID: remoteID,
|
||||
Anchor: p.Anchor,
|
||||
})
|
||||
}
|
||||
|
||||
deletes := make([]string, 0, len(req.Deletes))
|
||||
for _, id := range req.Deletes {
|
||||
id = strings.TrimSpace(id)
|
||||
if id == "" || len(id) > 128 {
|
||||
continue
|
||||
}
|
||||
deletes = append(deletes, id)
|
||||
}
|
||||
|
||||
if err := db.SyncMediaAnnotations(user.ID, item.ID, upserts, deletes); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, gin.H{"saved": len(upserts), "deleted": len(deletes)})
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
const mediaTicksPerSecond = 10_000_000
|
||||
|
||||
// GetMediaUserDataBatch 批量查询当前用户对条目的播放状态（收藏/进度/已看）
|
||||
// GET /fs/media/userdata?ids=1,2,3
|
||||
func GetMediaUserDataBatch(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.SuccessResp(c, []model.MediaUserData{})
|
||||
return
|
||||
}
|
||||
raw := c.Query("ids")
|
||||
ids := make([]uint, 0)
|
||||
for _, part := range strings.Split(raw, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
if id, err := strconv.ParseUint(part, 10, 64); err == nil {
|
||||
ids = append(ids, uint(id))
|
||||
}
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
common.SuccessResp(c, []model.MediaUserData{})
|
||||
return
|
||||
}
|
||||
// 限制批量查询数量，避免超长 IN 子句造成数据库压力
|
||||
if len(ids) > 500 {
|
||||
ids = ids[:500]
|
||||
}
|
||||
list, err := db.ListMediaUserDataByItems(user.ID, ids)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, list)
|
||||
}
|
||||
|
||||
// GetMediaProgress 查询单个条目（或选集）的播放进度
|
||||
func GetMediaProgress(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
itemID := uint(atoiDefault(c.Query("item_id"), 0))
|
||||
episode := atoiDefault(c.Query("episode_index"), 0)
|
||||
if user == nil || itemID == 0 {
|
||||
common.SuccessResp(c, nil)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(itemID)
|
||||
if err != nil || item.Hidden || !CanUserAccessItem(user, item) {
|
||||
common.SuccessResp(c, nil)
|
||||
return
|
||||
}
|
||||
data, err := db.GetMediaUserData(user.ID, itemID, episode)
|
||||
if err != nil {
|
||||
common.SuccessResp(c, nil)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, data)
|
||||
}
|
||||
|
||||
// SaveMediaProgress 上报播放进度
|
||||
type saveProgressReq struct {
|
||||
ItemID uint `json:"item_id"`
|
||||
EpisodeIndex int `json:"episode_index"`
|
||||
PositionSec float64 `json:"position"`
|
||||
DurationSec float64 `json:"duration"`
|
||||
Played bool `json:"played"`
|
||||
}
|
||||
|
||||
func SaveMediaProgress(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
var req saveProgressReq
|
||||
if err := c.ShouldBindJSON(&req); err != nil || req.ItemID == 0 {
|
||||
common.ErrorStrResp(c, "invalid request", 400)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(req.ItemID)
|
||||
if err != nil || item.Hidden || !CanUserAccessItem(user, item) {
|
||||
common.ErrorStrResp(c, "not found", 404)
|
||||
return
|
||||
}
|
||||
data, err := db.GetMediaUserData(user.ID, req.ItemID, req.EpisodeIndex)
|
||||
if err != nil {
|
||||
data = &model.MediaUserData{
|
||||
UserID: user.ID,
|
||||
ItemID: req.ItemID,
|
||||
EpisodeIndex: req.EpisodeIndex,
|
||||
}
|
||||
}
|
||||
now := time.Now()
|
||||
data.PositionTicks = int64(req.PositionSec * mediaTicksPerSecond)
|
||||
if req.DurationSec > 0 {
|
||||
data.PositionTicks = int64(minFloat(req.PositionSec, req.DurationSec) * mediaTicksPerSecond)
|
||||
}
|
||||
data.LastPlayedAt = &now
|
||||
finished := req.Played
|
||||
if req.DurationSec > 0 && req.PositionSec/req.DurationSec >= 0.9 {
|
||||
finished = true
|
||||
}
|
||||
if finished && !data.Played {
|
||||
data.Played = true
|
||||
data.PlayCount++
|
||||
}
|
||||
// 重新播放时清空已看标记
|
||||
if !finished && req.PositionSec < 5 && data.Played {
|
||||
data.Played = false
|
||||
}
|
||||
if err := db.SaveMediaUserData(data); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, data)
|
||||
}
|
||||
|
||||
// SetMediaFavorite 收藏 / 取消收藏
|
||||
type favoriteReq struct {
|
||||
ItemID uint `json:"item_id"`
|
||||
EpisodeIndex int `json:"episode_index"`
|
||||
Favorite bool `json:"favorite"`
|
||||
}
|
||||
|
||||
func SetMediaFavorite(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
if user == nil {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
var req favoriteReq
|
||||
if err := c.ShouldBindJSON(&req); err != nil || req.ItemID == 0 {
|
||||
common.ErrorStrResp(c, "invalid request", 400)
|
||||
return
|
||||
}
|
||||
item, err := db.GetMediaItemByID(req.ItemID)
|
||||
if err != nil || item.Hidden || !CanUserAccessItem(user, item) {
|
||||
common.ErrorStrResp(c, "not found", 404)
|
||||
return
|
||||
}
|
||||
data, err := db.GetMediaUserData(user.ID, req.ItemID, req.EpisodeIndex)
|
||||
if err != nil {
|
||||
data = &model.MediaUserData{
|
||||
UserID: user.ID,
|
||||
ItemID: req.ItemID,
|
||||
EpisodeIndex: req.EpisodeIndex,
|
||||
}
|
||||
}
|
||||
data.IsFavorite = req.Favorite
|
||||
if err := db.SaveMediaUserData(data); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c, gin.H{"favorite": data.IsFavorite})
|
||||
}
|
||||
|
||||
// ListMediaFavoritesAPI 列出当前用户收藏的媒体条目
|
||||
// GET /fs/media/favorites?media_type=video&page=1&page_size=20
|
||||
func ListMediaFavoritesAPI(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
mediaType := model.MediaType(c.Query("media_type"))
|
||||
page := atoiDefault(c.DefaultQuery("page", "1"), 1)
|
||||
pageSize := atoiDefault(c.DefaultQuery("page_size", "20"), 20)
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize <= 0 || pageSize > 100 {
|
||||
pageSize = 20
|
||||
}
|
||||
items := []model.MediaItem{}
|
||||
if user != nil {
|
||||
data, err := db.ListMediaFavorites(user.ID)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
for i := range data {
|
||||
row := data[i]
|
||||
if row.EpisodeIndex != 0 {
|
||||
continue
|
||||
}
|
||||
item, err := db.GetMediaItemByID(row.ItemID)
|
||||
if err != nil || item.Hidden || item.DeletedAt.Valid {
|
||||
continue
|
||||
}
|
||||
if mediaType != "" && item.MediaType != mediaType {
|
||||
continue
|
||||
}
|
||||
if !CanUserAccessItem(user, item) {
|
||||
continue
|
||||
}
|
||||
items = append(items, *item)
|
||||
}
|
||||
}
|
||||
total := len(items)
|
||||
start := (page - 1) * pageSize
|
||||
if start > total {
|
||||
start = total
|
||||
}
|
||||
end := start + pageSize
|
||||
if end > total {
|
||||
end = total
|
||||
}
|
||||
common.SuccessResp(c, common.PageResp{Content: items[start:end], Total: int64(total)})
|
||||
}
|
||||
|
||||
// ListMediaResumeAPI 继续观看列表
|
||||
func ListMediaResumeAPI(c *gin.Context) {
|
||||
user := mediaUser(c)
|
||||
mediaType := model.MediaType(c.Query("media_type"))
|
||||
limit := atoiDefault(c.DefaultQuery("limit", "20"), 20)
|
||||
items := []model.MediaItem{}
|
||||
if user != nil {
|
||||
data, err := db.ListMediaResume(user.ID, 200)
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
for i := range data {
|
||||
row := data[i]
|
||||
if row.PositionTicks <= 0 {
|
||||
continue
|
||||
}
|
||||
item, err := db.GetMediaItemByID(row.ItemID)
|
||||
if err != nil || item.Hidden || item.DeletedAt.Valid {
|
||||
continue
|
||||
}
|
||||
if mediaType != "" && item.MediaType != mediaType {
|
||||
continue
|
||||
}
|
||||
if !CanUserAccessItem(user, item) {
|
||||
continue
|
||||
}
|
||||
items = append(items, *item)
|
||||
if len(items) >= limit {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
common.SuccessResp(c, items)
|
||||
}
|
||||
|
||||
func atoiDefault(s string, def int) int {
|
||||
if v, err := strconv.Atoi(s); err == nil {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func minFloat(a, b float64) float64 {
|
||||
if a < b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
@@ -0,0 +1,772 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
stdpath "path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/op"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
const subsonicAPIVersion = "1.16.1"
|
||||
|
||||
// ==================== 响应结构 ====================
|
||||
|
||||
type subsonicError struct {
|
||||
Code int `json:"code" xml:"code,attr"`
|
||||
Message string `json:"message" xml:"message,attr"`
|
||||
}
|
||||
|
||||
type subsonicResponse struct {
|
||||
XMLName xml.Name `json:"-" xml:"subsonic-response"`
|
||||
|
||||
Status string `json:"status" xml:"status,attr"`
|
||||
Version string `json:"version" xml:"version,attr"`
|
||||
Type string `json:"type" xml:"type,attr"`
|
||||
ServerVersion string `json:"serverVersion" xml:"serverVersion,attr"`
|
||||
OpenSubsonic bool `json:"openSubsonic" xml:"openSubsonic,attr"`
|
||||
|
||||
Error *subsonicError `json:"error,omitempty" xml:"error,omitempty"`
|
||||
|
||||
License *subsonicLicense `json:"license,omitempty" xml:"license,omitempty"`
|
||||
MusicFolders *subsonicMusicFolders `json:"musicFolders,omitempty" xml:"musicFolders,omitempty"`
|
||||
Indexes *subsonicIndexes `json:"indexes,omitempty" xml:"indexes,omitempty"`
|
||||
Artists *subsonicArtists `json:"artists,omitempty" xml:"artists,omitempty"`
|
||||
Artist *subsonicArtist `json:"artist,omitempty" xml:"artist,omitempty"`
|
||||
Album *subsonicAlbum `json:"album,omitempty" xml:"album,omitempty"`
|
||||
Song *subsonicChild `json:"song,omitempty" xml:"song,omitempty"`
|
||||
AlbumList *subsonicAlbumList `json:"albumList,omitempty" xml:"albumList,omitempty"`
|
||||
AlbumList2 *subsonicAlbumList `json:"albumList2,omitempty" xml:"albumList2,omitempty"`
|
||||
SearchResult2 *subsonicSearchResult `json:"searchResult2,omitempty" xml:"searchResult2,omitempty"`
|
||||
SearchResult3 *subsonicSearchResult `json:"searchResult3,omitempty" xml:"searchResult3,omitempty"`
|
||||
Playlists *subsonicPlaylists `json:"playlists,omitempty" xml:"playlists,omitempty"`
|
||||
Playlist *subsonicPlaylist `json:"playlist,omitempty" xml:"playlist,omitempty"`
|
||||
Lyrics *subsonicLyrics `json:"lyrics,omitempty" xml:"lyrics,omitempty"`
|
||||
LyricsList *subsonicLyricsList `json:"lyricsList,omitempty" xml:"lyricsList,omitempty"`
|
||||
Starred *subsonicStarred `json:"starred,omitempty" xml:"starred,omitempty"`
|
||||
Starred2 *subsonicStarred `json:"starred2,omitempty" xml:"starred2,omitempty"`
|
||||
Genres *subsonicGenres `json:"genres,omitempty" xml:"genres,omitempty"`
|
||||
RandomSongs *subsonicSongs `json:"randomSongs,omitempty" xml:"randomSongs,omitempty"`
|
||||
SongsByGenre *subsonicSongs `json:"songsByGenre,omitempty" xml:"songsByGenre,omitempty"`
|
||||
ScanStatus *subsonicScanStatus `json:"scanStatus,omitempty" xml:"scanStatus,omitempty"`
|
||||
User *subsonicUser `json:"user,omitempty" xml:"user,omitempty"`
|
||||
Users *subsonicUsers `json:"users,omitempty" xml:"users,omitempty"`
|
||||
OpenSubsonicExtensions *subsonicExtensions `json:"openSubsonicExtensions,omitempty" xml:"openSubsonicExtensions,omitempty"`
|
||||
NowPlaying *subsonicNowPlaying `json:"nowPlaying,omitempty" xml:"nowPlaying,omitempty"`
|
||||
PlayQueue *subsonicPlayQueue `json:"playQueue,omitempty" xml:"playQueue,omitempty"`
|
||||
ArtistInfo2 *subsonicArtistInfo `json:"artistInfo2,omitempty" xml:"artistInfo2,omitempty"`
|
||||
AlbumInfo *subsonicAlbumInfo `json:"albumInfo,omitempty" xml:"albumInfo,omitempty"`
|
||||
TopSongs *subsonicSongs `json:"topSongs,omitempty" xml:"topSongs,omitempty"`
|
||||
SimilarSongs2 *subsonicSongs `json:"similarSongs2,omitempty" xml:"similarSongs2,omitempty"`
|
||||
MusicDirectory *subsonicDirectory `json:"directory,omitempty" xml:"directory,omitempty"`
|
||||
Bookmarks *subsonicBookmarks `json:"bookmarks,omitempty" xml:"bookmarks,omitempty"`
|
||||
}
|
||||
|
||||
type subsonicLicense struct {
|
||||
Valid bool `json:"valid" xml:"valid,attr"`
|
||||
}
|
||||
|
||||
type subsonicMusicFolder struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
}
|
||||
|
||||
type subsonicMusicFolders struct {
|
||||
MusicFolder []subsonicMusicFolder `json:"musicFolder" xml:"musicFolder"`
|
||||
}
|
||||
|
||||
type subsonicIndex struct {
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Artist []subsonicChild `json:"artist" xml:"artist"`
|
||||
}
|
||||
|
||||
type subsonicIndexes struct {
|
||||
LastModified int64 `json:"lastModified" xml:"lastModified,attr"`
|
||||
IgnoredArticles string `json:"ignoredArticles" xml:"ignoredArticles,attr"`
|
||||
Index []subsonicIndex `json:"index" xml:"index"`
|
||||
}
|
||||
|
||||
type subsonicArtist struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
AlbumCount int `json:"albumCount,omitempty" xml:"albumCount,attr,omitempty"`
|
||||
CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"`
|
||||
Album []subsonicAlbum `json:"album" xml:"album"`
|
||||
}
|
||||
|
||||
type subsonicArtists struct {
|
||||
IgnoredArticles string `json:"ignoredArticles" xml:"ignoredArticles,attr"`
|
||||
Index []subsonicIndex2 `json:"index" xml:"index"`
|
||||
}
|
||||
|
||||
type subsonicIndex2 struct {
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Artist []subsonicArtist `json:"artist" xml:"artist"`
|
||||
}
|
||||
|
||||
type subsonicAlbum struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"`
|
||||
ArtistId string `json:"artistId,omitempty" xml:"artistId,attr,omitempty"`
|
||||
CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"`
|
||||
SongCount int `json:"songCount" xml:"songCount,attr"`
|
||||
Duration int `json:"duration" xml:"duration,attr"`
|
||||
Created string `json:"created" xml:"created,attr"`
|
||||
Year int `json:"year,omitempty" xml:"year,attr,omitempty"`
|
||||
Song []subsonicChild `json:"song" xml:"song"`
|
||||
}
|
||||
|
||||
type subsonicChild struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Parent string `json:"parent,omitempty" xml:"parent,attr,omitempty"`
|
||||
IsDir bool `json:"isDir" xml:"isDir,attr"`
|
||||
Title string `json:"title" xml:"title,attr"`
|
||||
Album string `json:"album,omitempty" xml:"album,attr,omitempty"`
|
||||
Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"`
|
||||
Track int `json:"track,omitempty" xml:"track,attr,omitempty"`
|
||||
Year int `json:"year,omitempty" xml:"year,attr,omitempty"`
|
||||
Genre string `json:"genre,omitempty" xml:"genre,attr,omitempty"`
|
||||
CoverArt string `json:"coverArt,omitempty" xml:"coverArt,attr,omitempty"`
|
||||
Size int64 `json:"size,omitempty" xml:"size,attr,omitempty"`
|
||||
ContentType string `json:"contentType,omitempty" xml:"contentType,attr,omitempty"`
|
||||
Suffix string `json:"suffix,omitempty" xml:"suffix,attr,omitempty"`
|
||||
Duration int `json:"duration,omitempty" xml:"duration,attr,omitempty"`
|
||||
BitRate int `json:"bitRate,omitempty" xml:"bitRate,attr,omitempty"`
|
||||
Path string `json:"path,omitempty" xml:"path,attr,omitempty"`
|
||||
AlbumId string `json:"albumId,omitempty" xml:"albumId,attr,omitempty"`
|
||||
ArtistId string `json:"artistId,omitempty" xml:"artistId,attr,omitempty"`
|
||||
Type string `json:"type,omitempty" xml:"type,attr,omitempty"`
|
||||
IsVideo bool `json:"isVideo,omitempty" xml:"isVideo,attr,omitempty"`
|
||||
PlayCount int `json:"playCount,omitempty" xml:"playCount,attr,omitempty"`
|
||||
DiscNumber int `json:"discNumber,omitempty" xml:"discNumber,attr,omitempty"`
|
||||
Created string `json:"created,omitempty" xml:"created,attr,omitempty"`
|
||||
Starred string `json:"starred,omitempty" xml:"starred,attr,omitempty"`
|
||||
UserRating int `json:"userRating,omitempty" xml:"userRating,attr,omitempty"`
|
||||
}
|
||||
|
||||
type subsonicAlbumList struct {
|
||||
Album []subsonicAlbum `json:"album" xml:"album"`
|
||||
}
|
||||
|
||||
type subsonicSearchResult struct {
|
||||
Offset int `json:"offset" xml:"offset,attr"`
|
||||
Total int `json:"total" xml:"total,attr"`
|
||||
Artist []subsonicArtist `json:"artist" xml:"artist"`
|
||||
Album []subsonicAlbum `json:"album" xml:"album"`
|
||||
Song []subsonicChild `json:"song" xml:"song"`
|
||||
}
|
||||
|
||||
type subsonicSongs struct {
|
||||
Song []subsonicChild `json:"song" xml:"song"`
|
||||
}
|
||||
|
||||
type subsonicDirectory struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Parent string `json:"parent,omitempty" xml:"parent,attr,omitempty"`
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Starred string `json:"starred,omitempty" xml:"starred,attr,omitempty"`
|
||||
Child []subsonicChild `json:"child" xml:"child"`
|
||||
}
|
||||
|
||||
type subsonicPlaylists struct {
|
||||
Playlist []subsonicPlaylist `json:"playlist" xml:"playlist"`
|
||||
}
|
||||
|
||||
type subsonicPlaylist struct {
|
||||
Id string `json:"id" xml:"id,attr"`
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Comment string `json:"comment,omitempty" xml:"comment,attr,omitempty"`
|
||||
Owner string `json:"owner,omitempty" xml:"owner,attr,omitempty"`
|
||||
Public bool `json:"public" xml:"public,attr"`
|
||||
SongCount int `json:"songCount" xml:"songCount,attr"`
|
||||
Duration int `json:"duration" xml:"duration,attr"`
|
||||
Created string `json:"created" xml:"created,attr"`
|
||||
Entry []subsonicChild `json:"entry" xml:"entry"`
|
||||
}
|
||||
|
||||
type subsonicLyrics struct {
|
||||
Artist string `json:"artist,omitempty" xml:"artist,attr,omitempty"`
|
||||
Title string `json:"title,omitempty" xml:"title,attr,omitempty"`
|
||||
Value string `json:"value" xml:",chardata"`
|
||||
}
|
||||
|
||||
type subsonicLyricsList struct {
|
||||
StructuredLyrics []subsonicStructuredLyrics `json:"structuredLyrics" xml:"structuredLyrics"`
|
||||
}
|
||||
|
||||
type subsonicStructuredLyrics struct {
|
||||
DisplayArtist string `json:"displayArtist,omitempty" xml:"displayArtist,attr,omitempty"`
|
||||
DisplayTitle string `json:"displayTitle,omitempty" xml:"displayTitle,attr,omitempty"`
|
||||
Lang string `json:"lang" xml:"lang,attr"`
|
||||
Synced bool `json:"synced" xml:"synced,attr"`
|
||||
Line []subsonicLyricLine `json:"line" xml:"line"`
|
||||
}
|
||||
|
||||
type subsonicLyricLine struct {
|
||||
Start int64 `json:"start" xml:"start,attr"`
|
||||
Value string `json:"value" xml:"value,attr"`
|
||||
}
|
||||
|
||||
type subsonicStarred struct {
|
||||
Artist []subsonicArtist `json:"artist" xml:"artist"`
|
||||
Album []subsonicAlbum `json:"album" xml:"album"`
|
||||
Song []subsonicChild `json:"song" xml:"song"`
|
||||
}
|
||||
|
||||
type subsonicGenre struct {
|
||||
Value string `json:"value" xml:",chardata"`
|
||||
SongCount int `json:"songCount" xml:"songCount,attr"`
|
||||
AlbumCount int `json:"albumCount" xml:"albumCount,attr"`
|
||||
}
|
||||
|
||||
type subsonicGenres struct {
|
||||
Genre []subsonicGenre `json:"genre" xml:"genre"`
|
||||
}
|
||||
|
||||
type subsonicScanStatus struct {
|
||||
Scanning bool `json:"scanning" xml:"scanning,attr"`
|
||||
Count int64 `json:"count" xml:"count,attr"`
|
||||
}
|
||||
|
||||
type subsonicUser struct {
|
||||
Username string `json:"username" xml:"username,attr"`
|
||||
Email string `json:"email,omitempty" xml:"email,attr,omitempty"`
|
||||
ScrobblingEnabled bool `json:"scrobblingEnabled" xml:"scrobblingEnabled,attr"`
|
||||
AdminRole bool `json:"adminRole" xml:"adminRole,attr"`
|
||||
DownloadRole bool `json:"downloadRole" xml:"downloadRole,attr"`
|
||||
PlaylistRole bool `json:"playlistRole" xml:"playlistRole,attr"`
|
||||
StreamRole bool `json:"streamRole" xml:"streamRole,attr"`
|
||||
CoverArtRole bool `json:"coverArtRole" xml:"coverArtRole,attr"`
|
||||
}
|
||||
|
||||
type subsonicUsers struct {
|
||||
User []subsonicUser `json:"user" xml:"user"`
|
||||
}
|
||||
|
||||
type subsonicExtension struct {
|
||||
Name string `json:"name" xml:"name,attr"`
|
||||
Versions []int `json:"versions" xml:"versions"`
|
||||
}
|
||||
|
||||
type subsonicExtensions struct {
|
||||
Extension []subsonicExtension `json:"openSubsonicExtension" xml:"openSubsonicExtension"`
|
||||
}
|
||||
|
||||
type subsonicNowPlaying struct {
|
||||
Entry []subsonicChild `json:"entry" xml:"entry"`
|
||||
}
|
||||
|
||||
type subsonicPlayQueue struct {
|
||||
Current string `json:"current,omitempty" xml:"current,attr,omitempty"`
|
||||
Position int64 `json:"position,omitempty" xml:"position,attr,omitempty"`
|
||||
Username string `json:"username,omitempty" xml:"username,attr,omitempty"`
|
||||
Changed string `json:"changed,omitempty" xml:"changed,attr,omitempty"`
|
||||
Entry []subsonicChild `json:"entry" xml:"entry"`
|
||||
}
|
||||
|
||||
type subsonicArtistInfo struct {
|
||||
Biography string `json:"biography,omitempty" xml:"biography,omitempty"`
|
||||
MusicBrainzId string `json:"musicBrainzId,omitempty" xml:"musicBrainzId,omitempty"`
|
||||
}
|
||||
|
||||
type subsonicAlbumInfo struct {
|
||||
Notes string `json:"notes,omitempty" xml:"notes,omitempty"`
|
||||
}
|
||||
|
||||
type subsonicBookmarks struct {
|
||||
Bookmark []any `json:"bookmark" xml:"bookmark"`
|
||||
}
|
||||
|
||||
// ==================== 分发 ====================
|
||||
|
||||
// SubsonicAPI OpenSubsonic API 入口（/@rest/*）
|
||||
func SubsonicAPI(c *gin.Context) {
|
||||
if !MediaSubsonicEnabled() {
|
||||
c.AbortWithStatus(404)
|
||||
return
|
||||
}
|
||||
name := strings.Trim(c.Param("path"), "/")
|
||||
// 兼容接入地址已带 /@rest、客户端又自动追加 /rest 的情况
|
||||
if first, rest, found := strings.Cut(name, "/"); found && strings.EqualFold(first, "rest") {
|
||||
name = rest
|
||||
}
|
||||
name = strings.TrimSuffix(name, ".view")
|
||||
name = strings.ToLower(name)
|
||||
if name == "" {
|
||||
subError(c, 0, "missing method")
|
||||
return
|
||||
}
|
||||
|
||||
// 免认证端点
|
||||
if name == "getopensubsonicextensions" {
|
||||
subRespond(c, &subsonicResponse{OpenSubsonicExtensions: &subsonicExtensions{
|
||||
Extension: []subsonicExtension{
|
||||
{Name: "apiKeyAuthentication", Versions: []int{1}},
|
||||
{Name: "formPost", Versions: []int{1}},
|
||||
{Name: "songLyrics", Versions: []int{1}},
|
||||
},
|
||||
}})
|
||||
return
|
||||
}
|
||||
|
||||
user, _, ok := subAuth(c)
|
||||
if !ok {
|
||||
// 锁定判定同样要看用户名维度，否则被锁的账号只会收到「密码错误」
|
||||
if mediaAuthLockedFor(c, subParam(c, "u")) {
|
||||
subError(c, 40, "Too many failed attempts, please try again later")
|
||||
return
|
||||
}
|
||||
subError(c, 40, "Wrong username or password; use an app key as the password or enable password auth")
|
||||
return
|
||||
}
|
||||
|
||||
switch name {
|
||||
case "ping":
|
||||
subRespond(c, &subsonicResponse{})
|
||||
case "getlicense":
|
||||
subRespond(c, &subsonicResponse{License: &subsonicLicense{Valid: true}})
|
||||
case "tokeninfo":
|
||||
subRespond(c, &subsonicResponse{User: subUserOf(user)})
|
||||
case "getuser":
|
||||
subRespond(c, &subsonicResponse{User: subUserOf(user)})
|
||||
case "getusers":
|
||||
if !user.IsAdmin() {
|
||||
subError(c, 50, "not authorized")
|
||||
return
|
||||
}
|
||||
users, _, err := op.GetUsers(1, -1)
|
||||
if err != nil {
|
||||
subError(c, 0, err.Error())
|
||||
return
|
||||
}
|
||||
out := &subsonicUsers{}
|
||||
for i := range users {
|
||||
if users[i].IsGuest() {
|
||||
continue
|
||||
}
|
||||
out.User = append(out.User, *subUserOf(&users[i]))
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Users: out})
|
||||
case "getmusicfolders":
|
||||
subRespond(c, &subsonicResponse{MusicFolders: &subsonicMusicFolders{
|
||||
MusicFolder: []subsonicMusicFolder{{Id: "1", Name: "Music"}},
|
||||
}})
|
||||
case "getindexes":
|
||||
subGetIndexes(c, user)
|
||||
case "getartists":
|
||||
subGetArtists(c, user)
|
||||
case "getartist":
|
||||
subGetArtist(c, user)
|
||||
case "getalbum":
|
||||
subGetAlbum(c, user)
|
||||
case "getsong":
|
||||
subGetSong(c, user)
|
||||
case "getmusicdirectory":
|
||||
subGetMusicDirectory(c, user)
|
||||
case "getalbumlist", "getalbumlist2":
|
||||
subGetAlbumList(c, user, name == "getalbumlist")
|
||||
case "getrandomsongs":
|
||||
subGetRandomSongs(c, user)
|
||||
case "getsongsbygenre":
|
||||
subGetSongsByGenre(c, user)
|
||||
case "search2", "search3":
|
||||
subSearch(c, user, name == "search2")
|
||||
case "getstarred", "getstarred2":
|
||||
subGetStarred(c, user, name == "getstarred")
|
||||
case "stream", "download":
|
||||
subStream(c, user)
|
||||
case "getcoverart":
|
||||
subGetCoverArt(c, user)
|
||||
case "getlyrics":
|
||||
subGetLyrics(c, user)
|
||||
case "getlyricsbysongid":
|
||||
subGetLyricsBySongID(c, user)
|
||||
case "star", "unstar":
|
||||
subStar(c, user, name == "star")
|
||||
case "setrating":
|
||||
subSetRating(c, user)
|
||||
case "scrobble":
|
||||
subScrobble(c, user)
|
||||
case "getnowplaying":
|
||||
subRespond(c, &subsonicResponse{NowPlaying: &subsonicNowPlaying{Entry: subNowPlayingEntries()}})
|
||||
case "getplaylists":
|
||||
subGetPlaylists(c, user)
|
||||
case "getplaylist":
|
||||
subGetPlaylist(c, user)
|
||||
case "createplaylist":
|
||||
subCreatePlaylist(c, user)
|
||||
case "updateplaylist":
|
||||
subUpdatePlaylist(c, user)
|
||||
case "deleteplaylist":
|
||||
subDeletePlaylist(c, user)
|
||||
case "getscanstatus":
|
||||
subRespond(c, &subsonicResponse{ScanStatus: &subsonicScanStatus{Scanning: false, Count: subSongCount(user)}})
|
||||
case "startscan":
|
||||
if !user.IsAdmin() {
|
||||
subError(c, 50, "not authorized")
|
||||
return
|
||||
}
|
||||
subStartScan(c)
|
||||
case "saveplayqueue":
|
||||
subSavePlayQueue(c, user)
|
||||
case "getplayqueue":
|
||||
subRespond(c, &subsonicResponse{PlayQueue: subLoadPlayQueue(user)})
|
||||
case "getartistinfo2":
|
||||
subRespond(c, &subsonicResponse{ArtistInfo2: &subsonicArtistInfo{}})
|
||||
case "getalbuminfo", "getalbuminfo2":
|
||||
subRespond(c, &subsonicResponse{AlbumInfo: &subsonicAlbumInfo{}})
|
||||
case "gettopsongs", "getsimilarsongs2":
|
||||
subRespond(c, &subsonicResponse{TopSongs: &subsonicSongs{}})
|
||||
case "getbookmarks":
|
||||
subRespond(c, &subsonicResponse{Bookmarks: &subsonicBookmarks{}})
|
||||
case "getgenres":
|
||||
subGetGenres(c, user)
|
||||
case "getavatars":
|
||||
c.Status(404)
|
||||
// 明确不支持
|
||||
case "getpodcasts", "getnewestpodcasts", "refreshpodcasts", "createpodcastchannel",
|
||||
"deletepodcastchannel", "deletepodcastepisode", "downloadpodcastepisode",
|
||||
"createuser", "updateuser", "deleteuser", "changepassword",
|
||||
"getshares", "createshare", "updateshare", "deleteshare", "jukeboxcontrol":
|
||||
c.String(501, "This endpoint is not implemented, but may be in future releases")
|
||||
case "search", "getchatmessages", "addchatmessage", "getvideos", "getvideoinfo", "getcaptions", "hls":
|
||||
c.String(410, "This endpoint will not be implemented")
|
||||
default:
|
||||
subError(c, 0, "unknown method: "+name)
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 参数 / 认证 / 响应 ====================
|
||||
|
||||
func subParam(c *gin.Context, key string) string {
|
||||
if v := c.Query(key); v != "" {
|
||||
return v
|
||||
}
|
||||
if c.Request.Method == "POST" {
|
||||
_ = c.Request.ParseForm()
|
||||
if v := c.Request.PostForm.Get(key); v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func subIntParam(c *gin.Context, key string, def int) int {
|
||||
v := subParam(c, key)
|
||||
if v == "" {
|
||||
return def
|
||||
}
|
||||
n, err := strconv.Atoi(v)
|
||||
if err != nil {
|
||||
return def
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// subAuth 支持 apiKey / u+t+s / u+p（默认关闭）认证，并对失败次数做 IP + 用户名双维度限流
|
||||
func subAuth(c *gin.Context) (*model.User, *model.MediaAppKey, bool) {
|
||||
username := strings.TrimSpace(subParam(c, "u"))
|
||||
if mediaAuthLockedFor(c, username) {
|
||||
return nil, nil, false
|
||||
}
|
||||
u, k, ok := subAuthInner(c)
|
||||
if ok {
|
||||
// 用真实用户名清除计数：apiKey 认证时 u 参数是空的
|
||||
mediaAuthSucceededFor(c, u.Username)
|
||||
} else if subHasCredential(c) {
|
||||
// 只有「确实带了凭据但校验失败」才计数；
|
||||
// 完全裸请求（部分客户端启动时会先发一次不带参数的探测）不计入失败次数。
|
||||
mediaAuthFailedFor(c, username)
|
||||
}
|
||||
return u, k, ok
|
||||
}
|
||||
|
||||
// subHasCredential 判断请求是否真的携带了认证凭据
|
||||
func subHasCredential(c *gin.Context) bool {
|
||||
return subParam(c, "apiKey") != "" || subParam(c, "u") != "" ||
|
||||
subParam(c, "t") != "" || subParam(c, "p") != ""
|
||||
}
|
||||
|
||||
func subAuthInner(c *gin.Context) (*model.User, *model.MediaAppKey, bool) {
|
||||
// OpenSubsonic apiKey 扩展：直接把应用密钥放在 apiKey 参数里
|
||||
if key := strings.TrimSpace(subParam(c, "apiKey")); key != "" {
|
||||
if u, k, ok := subAuthByAppKey(key); ok {
|
||||
return u, k, true
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
username := strings.TrimSpace(subParam(c, "u"))
|
||||
password := decodeSubsonicPassword(subParam(c, "p"))
|
||||
// 用户名留空、只填了应用密钥（部分客户端只允许填一个"密码"字段）
|
||||
if username == "" {
|
||||
if password != "" {
|
||||
if u, k, ok := subAuthByAppKey(password); ok {
|
||||
return u, k, true
|
||||
}
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
user, err := op.GetUserByName(username)
|
||||
if err != nil || user.Disabled || user.IsGuest() {
|
||||
return nil, nil, false
|
||||
}
|
||||
// t+s：用应用密钥的明文做 md5(密码+salt) 校验（Subsonic token 认证）
|
||||
if salt := subParam(c, "s"); salt != "" {
|
||||
if token := subParam(c, "t"); token != "" {
|
||||
keys, _ := db.ListMediaAppKeys(user.ID)
|
||||
for i := range keys {
|
||||
plain, err := media.DecryptAppKey(keys[i].Secret)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if media.MatchSubsonicToken(plain, salt, token) {
|
||||
db.TouchMediaAppKey(keys[i].ID)
|
||||
return user, &keys[i], true
|
||||
}
|
||||
}
|
||||
// 不直接返回：个别客户端会同时带上 p，继续按明文校验
|
||||
}
|
||||
}
|
||||
// p：OpenList 密码或应用密钥明文
|
||||
if password != "" {
|
||||
// 使用主账号密码认证会把登录密码暴露给第三方客户端，必须由管理员显式开启
|
||||
if setting.GetBool(conf.MediaSubsonicAllowPwd) && user.OtpSecret == "" &&
|
||||
user.ValidateRawPassword(password) == nil {
|
||||
return user, nil, true
|
||||
}
|
||||
if u, k, ok := subAuthByAppKey(password); ok && u.ID == user.ID {
|
||||
return u, k, true
|
||||
}
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// decodeSubsonicPassword 解析 p 参数，兼容 Subsonic 的 enc:<hex> 编码
|
||||
func decodeSubsonicPassword(p string) string {
|
||||
p = strings.TrimSpace(p)
|
||||
if strings.HasPrefix(p, "enc:") {
|
||||
if raw, err := hex.DecodeString(strings.TrimPrefix(p, "enc:")); err == nil {
|
||||
return string(raw)
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func subAuthByAppKey(key string) (*model.User, *model.MediaAppKey, bool) {
|
||||
return LookUpAppKeyUser(key)
|
||||
}
|
||||
|
||||
// subsonicServerVersion 返回给 Subsonic 客户端的服务器版本（仅用于展示）
|
||||
func subsonicServerVersion() string {
|
||||
if conf.Version == "" {
|
||||
return "OpenList"
|
||||
}
|
||||
return conf.Version
|
||||
}
|
||||
|
||||
func subRespond(c *gin.Context, resp *subsonicResponse) {
|
||||
resp.Status = "ok"
|
||||
resp.Version = subsonicAPIVersion
|
||||
resp.Type = "OpenList"
|
||||
resp.ServerVersion = subsonicServerVersion()
|
||||
resp.OpenSubsonic = true
|
||||
format := strings.ToLower(subParam(c, "f"))
|
||||
switch format {
|
||||
case "json":
|
||||
c.JSON(200, gin.H{"subsonic-response": resp})
|
||||
case "jsonp":
|
||||
callback := subParam(c, "callback")
|
||||
if !validJsonpCallback(callback) {
|
||||
c.JSON(200, gin.H{"subsonic-response": &subsonicResponse{
|
||||
Status: "failed", Version: subsonicAPIVersion, Type: "OpenList",
|
||||
ServerVersion: subsonicServerVersion(), OpenSubsonic: true,
|
||||
Error: &subsonicError{Code: 0, Message: "invalid callback parameter"},
|
||||
}})
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(gin.H{"subsonic-response": resp})
|
||||
c.Header("Content-Type", "application/javascript; charset=utf-8")
|
||||
c.String(200, "%s(%s)", callback, string(body))
|
||||
default:
|
||||
c.XML(200, resp)
|
||||
}
|
||||
}
|
||||
|
||||
func subError(c *gin.Context, code int, message string) {
|
||||
resp := &subsonicResponse{
|
||||
Status: "failed",
|
||||
Version: subsonicAPIVersion,
|
||||
Type: "OpenList",
|
||||
ServerVersion: subsonicServerVersion(),
|
||||
OpenSubsonic: true,
|
||||
Error: &subsonicError{Code: code, Message: message},
|
||||
}
|
||||
format := strings.ToLower(subParam(c, "f"))
|
||||
switch format {
|
||||
case "json":
|
||||
c.JSON(200, gin.H{"subsonic-response": resp})
|
||||
case "jsonp":
|
||||
body, _ := json.Marshal(gin.H{"subsonic-response": resp})
|
||||
c.String(200, "%s(%s)", subParam(c, "callback"), string(body))
|
||||
default:
|
||||
c.XML(200, resp)
|
||||
}
|
||||
}
|
||||
|
||||
func validJsonpCallback(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for i, r := range s {
|
||||
if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r == '_' || r == '$' {
|
||||
continue
|
||||
}
|
||||
if i > 0 && (r >= '0' && r <= '9' || r == '.') {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ==================== 数据映射 ====================
|
||||
|
||||
func subUserOf(u *model.User) *subsonicUser {
|
||||
return &subsonicUser{
|
||||
Username: u.Username,
|
||||
ScrobblingEnabled: true,
|
||||
AdminRole: u.IsAdmin(),
|
||||
DownloadRole: true,
|
||||
PlaylistRole: true,
|
||||
StreamRole: true,
|
||||
CoverArtRole: true,
|
||||
}
|
||||
}
|
||||
|
||||
func (l *embyLibrary) subChild(it *model.MediaItem) subsonicChild {
|
||||
title := it.ScrapedName
|
||||
if title == "" {
|
||||
title = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName))
|
||||
}
|
||||
album := it.AlbumName
|
||||
if album == "" {
|
||||
album = "Unknown Album"
|
||||
}
|
||||
artist := albumArtistOf(it)
|
||||
image := fmt.Sprintf("al-%s", albumKeyOf(it))
|
||||
year := 0
|
||||
if len(it.ReleaseDate) >= 4 {
|
||||
year, _ = strconv.Atoi(it.ReleaseDate[:4])
|
||||
}
|
||||
userData := l.userDataOf(it.ID, 0)
|
||||
c := subsonicChild{
|
||||
Id: fmt.Sprintf("so-%d", it.ID),
|
||||
Parent: image,
|
||||
IsDir: false,
|
||||
Title: title,
|
||||
Album: album,
|
||||
Artist: artist,
|
||||
Track: it.TrackNumber,
|
||||
Year: year,
|
||||
Genre: it.Genre,
|
||||
CoverArt: image,
|
||||
Size: it.FileSize,
|
||||
ContentType: it.MimeType,
|
||||
Suffix: strings.TrimPrefix(strings.ToLower(stdpath.Ext(it.FileName)), "."),
|
||||
Duration: it.Duration,
|
||||
Path: MediaItemFilePath(it),
|
||||
AlbumId: image,
|
||||
ArtistId: "ar-" + artistKeyOf(it),
|
||||
Type: "music",
|
||||
PlayCount: userData.PlayCount,
|
||||
Created: it.CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z"),
|
||||
UserRating: userDataRating(it.ID, l),
|
||||
}
|
||||
if userData.IsFavorite {
|
||||
c.Starred = it.UpdatedAt.UTC().Format("2006-01-02T15:04:05.000Z")
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func userDataRating(itemID uint, l *embyLibrary) int {
|
||||
if l == nil {
|
||||
return 0
|
||||
}
|
||||
if d, ok := l.userData[[2]uint{itemID, 0}]; ok {
|
||||
return d.Rating
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (l *embyLibrary) subAlbum(al *embyAlbum) subsonicAlbum {
|
||||
duration := 0
|
||||
for i := range al.Items {
|
||||
duration += al.Items[i].Duration
|
||||
}
|
||||
created := time.Now().UTC().Format("2006-01-02T15:04:05.000Z")
|
||||
if len(al.Items) > 0 {
|
||||
created = al.Items[0].CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z")
|
||||
}
|
||||
return subsonicAlbum{
|
||||
Id: "al-" + al.Key,
|
||||
Name: al.Name,
|
||||
Artist: al.Artist,
|
||||
ArtistId: "ar-" + artistKeyOf(&al.Items[0]),
|
||||
CoverArt: "al-" + al.Key,
|
||||
SongCount: len(al.Items),
|
||||
Duration: duration,
|
||||
Created: created,
|
||||
Year: al.Year,
|
||||
}
|
||||
}
|
||||
|
||||
func (l *embyLibrary) subAlbumWithSongs(al *embyAlbum) subsonicAlbum {
|
||||
out := l.subAlbum(al)
|
||||
for i := range al.Items {
|
||||
out.Song = append(out.Song, l.subChild(&al.Items[i]))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (l *embyLibrary) subArtist(key, name string) subsonicArtist {
|
||||
out := subsonicArtist{Id: "ar-" + key, Name: name}
|
||||
for _, al := range l.albums() {
|
||||
if artistKeyOf(&al.Items[0]) != key {
|
||||
continue
|
||||
}
|
||||
out.Album = append(out.Album, l.subAlbum(al))
|
||||
}
|
||||
out.AlbumCount = len(out.Album)
|
||||
if len(out.Album) > 0 {
|
||||
out.CoverArt = out.Album[0].CoverArt
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (l *embyLibrary) subArtists() []subsonicArtist {
|
||||
seen := map[string]bool{}
|
||||
var out []subsonicArtist
|
||||
for i := range l.music {
|
||||
key := artistKeyOf(&l.music[i])
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, l.subArtist(key, albumArtistOf(&l.music[i])))
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return strings.ToLower(out[i].Name) < strings.ToLower(out[j].Name) })
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,667 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"net/http"
|
||||
stdpath "path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/db"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/media"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/model"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// ==================== 浏览 ====================
|
||||
|
||||
func subInitial(name string) string {
|
||||
if name == "" {
|
||||
return "#"
|
||||
}
|
||||
r := strings.ToUpper(string([]rune(name)[0]))
|
||||
if r < "A" || r > "Z" {
|
||||
return "#"
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func subGetIndexes(c *gin.Context, user *model.User) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
artists := lib.subArtists()
|
||||
groups := map[string][]subsonicChild{}
|
||||
var keys []string
|
||||
for _, a := range artists {
|
||||
idx := subInitial(a.Name)
|
||||
if _, ok := groups[idx]; !ok {
|
||||
keys = append(keys, idx)
|
||||
}
|
||||
groups[idx] = append(groups[idx], subsonicChild{
|
||||
Id: a.Id,
|
||||
Title: a.Name,
|
||||
IsDir: true,
|
||||
Album: a.Name,
|
||||
})
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := &subsonicIndexes{IgnoredArticles: "The El La Los Las Le Les", LastModified: time.Now().UnixMilli()}
|
||||
for _, k := range keys {
|
||||
out.Index = append(out.Index, subsonicIndex{Name: k, Artist: groups[k]})
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Indexes: out})
|
||||
}
|
||||
|
||||
func subGetArtists(c *gin.Context, user *model.User) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
artists := lib.subArtists()
|
||||
groups := map[string][]subsonicArtist{}
|
||||
var keys []string
|
||||
for _, a := range artists {
|
||||
idx := subInitial(a.Name)
|
||||
if _, ok := groups[idx]; !ok {
|
||||
keys = append(keys, idx)
|
||||
}
|
||||
groups[idx] = append(groups[idx], a)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := &subsonicArtists{IgnoredArticles: "The El La Los Las Le Les"}
|
||||
for _, k := range keys {
|
||||
out.Index = append(out.Index, subsonicIndex2{Name: k, Artist: groups[k]})
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Artists: out})
|
||||
}
|
||||
|
||||
func subGetArtist(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
if !strings.HasPrefix(id, "ar-") {
|
||||
subError(c, 70, "artist not found")
|
||||
return
|
||||
}
|
||||
lib := loadEmbyLibrary(user)
|
||||
key := strings.TrimPrefix(id, "ar-")
|
||||
for _, a := range lib.subArtists() {
|
||||
if a.Id == id {
|
||||
_ = key
|
||||
subRespond(c, &subsonicResponse{Artist: &a})
|
||||
return
|
||||
}
|
||||
}
|
||||
subError(c, 70, "artist not found")
|
||||
}
|
||||
|
||||
func subGetAlbum(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
if !strings.HasPrefix(id, "al-") {
|
||||
subError(c, 70, "album not found")
|
||||
return
|
||||
}
|
||||
lib := loadEmbyLibrary(user)
|
||||
al := lib.albumByKey(strings.TrimPrefix(id, "al-"))
|
||||
if al == nil {
|
||||
subError(c, 70, "album not found")
|
||||
return
|
||||
}
|
||||
out := lib.subAlbumWithSongs(al)
|
||||
subRespond(c, &subsonicResponse{Album: &out})
|
||||
}
|
||||
|
||||
func subGetSong(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
it, ok := lib.songByID(id)
|
||||
if !ok {
|
||||
subError(c, 70, "song not found")
|
||||
return
|
||||
}
|
||||
child := lib.subChild(it)
|
||||
subRespond(c, &subsonicResponse{Song: &child})
|
||||
}
|
||||
|
||||
func subGetMusicDirectory(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
switch {
|
||||
case strings.HasPrefix(id, "ar-"):
|
||||
key := strings.TrimPrefix(id, "ar-")
|
||||
dir := &subsonicDirectory{Id: id, Name: id}
|
||||
for _, al := range lib.albums() {
|
||||
if artistKeyOf(&al.Items[0]) != key {
|
||||
continue
|
||||
}
|
||||
a := lib.subAlbum(al)
|
||||
dir.Child = append(dir.Child, subsonicChild{
|
||||
Id: a.Id, IsDir: true, Title: a.Name, Album: a.Name,
|
||||
Artist: a.Artist, CoverArt: a.CoverArt, Year: a.Year,
|
||||
})
|
||||
if dir.Name == id {
|
||||
dir.Name = a.Artist
|
||||
}
|
||||
}
|
||||
subRespond(c, &subsonicResponse{MusicDirectory: dir})
|
||||
case strings.HasPrefix(id, "al-"):
|
||||
al := lib.albumByKey(strings.TrimPrefix(id, "al-"))
|
||||
if al == nil {
|
||||
subError(c, 70, "directory not found")
|
||||
return
|
||||
}
|
||||
dir := &subsonicDirectory{Id: id, Name: al.Name, Parent: "ar-" + artistKeyOf(&al.Items[0])}
|
||||
for i := range al.Items {
|
||||
dir.Child = append(dir.Child, lib.subChild(&al.Items[i]))
|
||||
}
|
||||
subRespond(c, &subsonicResponse{MusicDirectory: dir})
|
||||
default:
|
||||
subError(c, 70, "directory not found")
|
||||
}
|
||||
}
|
||||
|
||||
func subGetAlbumList(c *gin.Context, user *model.User, legacy bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
listType := strings.ToLower(subParam(c, "type"))
|
||||
albums := lib.albums()
|
||||
switch listType {
|
||||
case "newest":
|
||||
sort.SliceStable(albums, func(i, j int) bool { return albums[i].Items[0].CreatedAt.After(albums[j].Items[0].CreatedAt) })
|
||||
case "recent":
|
||||
sort.SliceStable(albums, func(i, j int) bool { return albums[i].Items[0].UpdatedAt.After(albums[j].Items[0].UpdatedAt) })
|
||||
case "alphabeticalbyartist":
|
||||
sort.SliceStable(albums, func(i, j int) bool {
|
||||
if albums[i].Artist == albums[j].Artist {
|
||||
return strings.ToLower(albums[i].Name) < strings.ToLower(albums[j].Name)
|
||||
}
|
||||
return strings.ToLower(albums[i].Artist) < strings.ToLower(albums[j].Artist)
|
||||
})
|
||||
case "random":
|
||||
rand.Shuffle(len(albums), func(i, j int) { albums[i], albums[j] = albums[j], albums[i] })
|
||||
case "byyear":
|
||||
from := subIntParam(c, "fromYear", 0)
|
||||
to := subIntParam(c, "toYear", 0)
|
||||
var filtered []*embyAlbum
|
||||
for _, al := range albums {
|
||||
if from > 0 && al.Year < from {
|
||||
continue
|
||||
}
|
||||
if to > 0 && al.Year > to {
|
||||
continue
|
||||
}
|
||||
filtered = append(filtered, al)
|
||||
}
|
||||
albums = filtered
|
||||
default:
|
||||
sort.SliceStable(albums, func(i, j int) bool { return strings.ToLower(albums[i].Name) < strings.ToLower(albums[j].Name) })
|
||||
}
|
||||
size := subIntParam(c, "size", 50)
|
||||
offset := subIntParam(c, "offset", 0)
|
||||
if offset > len(albums) {
|
||||
offset = len(albums)
|
||||
}
|
||||
end := offset + size
|
||||
if end > len(albums) {
|
||||
end = len(albums)
|
||||
}
|
||||
out := &subsonicAlbumList{}
|
||||
for _, al := range albums[offset:end] {
|
||||
out.Album = append(out.Album, lib.subAlbum(al))
|
||||
}
|
||||
if legacy {
|
||||
subRespond(c, &subsonicResponse{AlbumList: out})
|
||||
return
|
||||
}
|
||||
subRespond(c, &subsonicResponse{AlbumList2: out})
|
||||
}
|
||||
|
||||
func (l *embyLibrary) allSongs() []subsonicChild {
|
||||
out := make([]subsonicChild, 0, len(l.music))
|
||||
for i := range l.music {
|
||||
out = append(out, l.subChild(&l.music[i]))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func subGetRandomSongs(c *gin.Context, user *model.User) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
songs := lib.allSongs()
|
||||
rand.Shuffle(len(songs), func(i, j int) { songs[i], songs[j] = songs[j], songs[i] })
|
||||
size := subIntParam(c, "size", 10)
|
||||
if size > len(songs) {
|
||||
size = len(songs)
|
||||
}
|
||||
subRespond(c, &subsonicResponse{RandomSongs: &subsonicSongs{Song: songs[:size]}})
|
||||
}
|
||||
|
||||
func subGetSongsByGenre(c *gin.Context, user *model.User) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
genre := strings.ToLower(subParam(c, "genre"))
|
||||
var songs []subsonicChild
|
||||
for i := range lib.music {
|
||||
if genre != "" && !strings.Contains(strings.ToLower(lib.music[i].Genre), genre) {
|
||||
continue
|
||||
}
|
||||
songs = append(songs, lib.subChild(&lib.music[i]))
|
||||
}
|
||||
subRespond(c, &subsonicResponse{SongsByGenre: &subsonicSongs{Song: songs}})
|
||||
}
|
||||
|
||||
func subSearch(c *gin.Context, user *model.User, legacy bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
term := strings.ToLower(strings.TrimSpace(subParam(c, "query")))
|
||||
if term == "" {
|
||||
term = strings.ToLower(strings.TrimSpace(subParam(c, "any")))
|
||||
}
|
||||
artistCount := subIntParam(c, "artistCount", 20)
|
||||
albumCount := subIntParam(c, "albumCount", 20)
|
||||
songCount := subIntParam(c, "songCount", 20)
|
||||
|
||||
out := &subsonicSearchResult{}
|
||||
for _, a := range lib.subArtists() {
|
||||
if len(out.Artist) >= artistCount {
|
||||
break
|
||||
}
|
||||
if term == "" || strings.Contains(strings.ToLower(a.Name), term) {
|
||||
out.Artist = append(out.Artist, a)
|
||||
}
|
||||
}
|
||||
for _, al := range lib.albums() {
|
||||
if len(out.Album) >= albumCount {
|
||||
break
|
||||
}
|
||||
if term == "" || strings.Contains(strings.ToLower(al.Name), term) ||
|
||||
strings.Contains(strings.ToLower(al.Artist), term) {
|
||||
out.Album = append(out.Album, lib.subAlbum(al))
|
||||
}
|
||||
}
|
||||
for i := range lib.music {
|
||||
if len(out.Song) >= songCount {
|
||||
break
|
||||
}
|
||||
name := lib.music[i].ScrapedName
|
||||
if name == "" {
|
||||
name = lib.music[i].FileName
|
||||
}
|
||||
if term == "" || strings.Contains(strings.ToLower(name), term) {
|
||||
out.Song = append(out.Song, lib.subChild(&lib.music[i]))
|
||||
}
|
||||
}
|
||||
out.Total = len(out.Artist) + len(out.Album) + len(out.Song)
|
||||
if legacy {
|
||||
subRespond(c, &subsonicResponse{SearchResult2: out})
|
||||
return
|
||||
}
|
||||
subRespond(c, &subsonicResponse{SearchResult3: out})
|
||||
}
|
||||
|
||||
func subGetStarred(c *gin.Context, user *model.User, legacy bool) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
out := &subsonicStarred{}
|
||||
for i := range lib.music {
|
||||
if d, ok := lib.userData[[2]uint{lib.music[i].ID, 0}]; ok && d.IsFavorite {
|
||||
out.Song = append(out.Song, lib.subChild(&lib.music[i]))
|
||||
}
|
||||
}
|
||||
if legacy {
|
||||
subRespond(c, &subsonicResponse{Starred: out})
|
||||
return
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Starred2: out})
|
||||
}
|
||||
|
||||
func subGetGenres(c *gin.Context, user *model.User) {
|
||||
lib := loadEmbyLibrary(user)
|
||||
counts := map[string]int{}
|
||||
for i := range lib.music {
|
||||
for _, g := range strings.Split(lib.music[i].Genre, ",") {
|
||||
g = strings.TrimSpace(g)
|
||||
if g == "" {
|
||||
continue
|
||||
}
|
||||
counts[g]++
|
||||
}
|
||||
}
|
||||
out := &subsonicGenres{}
|
||||
for g, n := range counts {
|
||||
out.Genre = append(out.Genre, subsonicGenre{Value: g, SongCount: n})
|
||||
}
|
||||
sort.Slice(out.Genre, func(i, j int) bool { return out.Genre[i].Value < out.Genre[j].Value })
|
||||
subRespond(c, &subsonicResponse{Genres: out})
|
||||
}
|
||||
|
||||
// ==================== 播放 ====================
|
||||
|
||||
func subStream(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
subError(c, 70, "song not found")
|
||||
return
|
||||
}
|
||||
if setting.GetBool(conf.MediaSubsonicProxyStream) {
|
||||
c.Redirect(http.StatusFound, SignedProxyURL(c, target.RawPath))
|
||||
return
|
||||
}
|
||||
c.Redirect(http.StatusFound, SignedStreamURL(c, target.RawPath))
|
||||
}
|
||||
|
||||
func subGetCoverArt(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
var item *model.MediaItem
|
||||
switch {
|
||||
case strings.HasPrefix(id, "so-"):
|
||||
item, _ = lib.songByID(id)
|
||||
case strings.HasPrefix(id, "al-"):
|
||||
if al := lib.albumByKey(strings.TrimPrefix(id, "al-")); al != nil && len(al.Items) > 0 {
|
||||
item = &al.Items[0]
|
||||
}
|
||||
case strings.HasPrefix(id, "ar-"):
|
||||
key := strings.TrimPrefix(id, "ar-")
|
||||
for i := range lib.music {
|
||||
if artistKeyOf(&lib.music[i]) == key {
|
||||
item = &lib.music[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
case strings.HasPrefix(id, "mv-"), strings.HasPrefix(id, "ep-"):
|
||||
if it, ok := lib.resolveItem(id); ok {
|
||||
_ = it
|
||||
if p, ok := lib.playTarget(id); ok {
|
||||
item = p.Item
|
||||
}
|
||||
}
|
||||
}
|
||||
if item == nil {
|
||||
c.Status(404)
|
||||
return
|
||||
}
|
||||
serveEmbyCover(c, item)
|
||||
}
|
||||
|
||||
func subGetLyrics(c *gin.Context, user *model.User) {
|
||||
artist := subParam(c, "artist")
|
||||
title := subParam(c, "title")
|
||||
lib := loadEmbyLibrary(user)
|
||||
for i := range lib.music {
|
||||
it := &lib.music[i]
|
||||
name := it.ScrapedName
|
||||
if name == "" {
|
||||
name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName))
|
||||
}
|
||||
if title != "" && !strings.EqualFold(title, name) {
|
||||
continue
|
||||
}
|
||||
if artist != "" && !strings.EqualFold(artist, albumArtistOf(it)) {
|
||||
continue
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Lyrics: &subsonicLyrics{
|
||||
Artist: albumArtistOf(it), Title: name, Value: it.Lyrics,
|
||||
}})
|
||||
return
|
||||
}
|
||||
subError(c, 70, "lyrics not found")
|
||||
}
|
||||
|
||||
func subGetLyricsBySongID(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
it, ok := lib.songByID(id)
|
||||
if !ok || it.Lyrics == "" {
|
||||
subError(c, 70, "lyrics not found")
|
||||
return
|
||||
}
|
||||
name := it.ScrapedName
|
||||
if name == "" {
|
||||
name = strings.TrimSuffix(it.FileName, stdpath.Ext(it.FileName))
|
||||
}
|
||||
lines := parseLrc(it.Lyrics)
|
||||
structured := subsonicStructuredLyrics{
|
||||
DisplayArtist: albumArtistOf(it),
|
||||
DisplayTitle: name,
|
||||
Lang: "und",
|
||||
Synced: len(lines) > 0,
|
||||
}
|
||||
for _, l := range lines {
|
||||
structured.Line = append(structured.Line, subsonicLyricLine{
|
||||
Start: l["Start"].(int64),
|
||||
Value: l["Text"].(string),
|
||||
})
|
||||
}
|
||||
subRespond(c, &subsonicResponse{LyricsList: &subsonicLyricsList{
|
||||
StructuredLyrics: []subsonicStructuredLyrics{structured},
|
||||
}})
|
||||
}
|
||||
|
||||
// ==================== 状态 ====================
|
||||
|
||||
func subStar(c *gin.Context, user *model.User, star bool) {
|
||||
id := subParam(c, "id")
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
if album := strings.HasPrefix(id, "al-"); album {
|
||||
subError(c, 70, "not supported")
|
||||
return
|
||||
}
|
||||
subError(c, 70, "song not found")
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
d.IsFavorite = star
|
||||
if err := db.SaveMediaUserData(d); err != nil {
|
||||
subError(c, 0, err.Error())
|
||||
return
|
||||
}
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
func subSetRating(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
rating := subIntParam(c, "rating", 0)
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
subError(c, 70, "song not found")
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
d.Rating = rating
|
||||
_ = db.SaveMediaUserData(d)
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
func subScrobble(c *gin.Context, user *model.User) {
|
||||
id := subParam(c, "id")
|
||||
submission := strings.EqualFold(subParam(c, "submission"), "true")
|
||||
lib := loadEmbyLibrary(user)
|
||||
target, ok := lib.playTarget(id)
|
||||
if !ok {
|
||||
subRespond(c, &subsonicResponse{})
|
||||
return
|
||||
}
|
||||
d, err := db.GetMediaUserData(user.ID, target.ItemID, target.EpisodeNo)
|
||||
if err != nil || d == nil {
|
||||
d = &model.MediaUserData{UserID: user.ID, ItemID: target.ItemID, EpisodeIndex: target.EpisodeNo}
|
||||
}
|
||||
now := time.Now()
|
||||
d.LastPlayedAt = &now
|
||||
if submission {
|
||||
d.Played = true
|
||||
d.PlayCount++
|
||||
}
|
||||
_ = db.SaveMediaUserData(d)
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
// ==================== 播放列表 ====================
|
||||
|
||||
func subGetPlaylists(c *gin.Context, user *model.User) {
|
||||
list, err := db.ListMediaPlaylists(user.ID)
|
||||
if err != nil {
|
||||
subError(c, 0, err.Error())
|
||||
return
|
||||
}
|
||||
out := &subsonicPlaylists{}
|
||||
for i := range list {
|
||||
out.Playlist = append(out.Playlist, subPlaylistOf(&list[i]))
|
||||
}
|
||||
subRespond(c, &subsonicResponse{Playlists: out})
|
||||
}
|
||||
|
||||
func subPlaylistOf(p *model.MediaPlaylist) subsonicPlaylist {
|
||||
return subsonicPlaylist{
|
||||
Id: strconv.FormatUint(uint64(p.ID), 10),
|
||||
Name: p.Name,
|
||||
Comment: p.Comment,
|
||||
Public: p.Public,
|
||||
SongCount: p.ItemCount,
|
||||
Created: p.CreatedAt.UTC().Format("2006-01-02T15:04:05.000Z"),
|
||||
}
|
||||
}
|
||||
|
||||
func subGetPlaylist(c *gin.Context, user *model.User) {
|
||||
id, err := strconv.ParseUint(subParam(c, "id"), 10, 64)
|
||||
if err != nil {
|
||||
subError(c, 70, "playlist not found")
|
||||
return
|
||||
}
|
||||
p, err := db.GetMediaPlaylist(uint(id))
|
||||
if err != nil || (p.UserID != user.ID && !user.IsAdmin()) {
|
||||
subError(c, 70, "playlist not found")
|
||||
return
|
||||
}
|
||||
out := subPlaylistOf(p)
|
||||
items, _ := db.ListMediaPlaylistItems(p.ID)
|
||||
lib := loadEmbyLibrary(user)
|
||||
for _, it := range items {
|
||||
if song, ok := lib.songByID(fmt.Sprintf("so-%d", it.ItemID)); ok {
|
||||
out.Entry = append(out.Entry, lib.subChild(song))
|
||||
}
|
||||
}
|
||||
out.SongCount = len(out.Entry)
|
||||
subRespond(c, &subsonicResponse{Playlist: &out})
|
||||
}
|
||||
|
||||
func subCreatePlaylist(c *gin.Context, user *model.User) {
|
||||
name := subParam(c, "name")
|
||||
if name == "" {
|
||||
subError(c, 10, "missing name")
|
||||
return
|
||||
}
|
||||
p := &model.MediaPlaylist{UserID: user.ID, Name: name}
|
||||
if err := db.CreateMediaPlaylist(p); err != nil {
|
||||
subError(c, 0, err.Error())
|
||||
return
|
||||
}
|
||||
subError(c, 0, "created:"+strconv.FormatUint(uint64(p.ID), 10))
|
||||
}
|
||||
|
||||
func subUpdatePlaylist(c *gin.Context, user *model.User) {
|
||||
id, err := strconv.ParseUint(subParam(c, "playlistId"), 10, 64)
|
||||
if err != nil {
|
||||
subError(c, 10, "missing playlistId")
|
||||
return
|
||||
}
|
||||
p, err := db.GetMediaPlaylist(uint(id))
|
||||
if err != nil || (p.UserID != user.ID && !user.IsAdmin()) {
|
||||
subError(c, 70, "playlist not found")
|
||||
return
|
||||
}
|
||||
if name := subParam(c, "name"); name != "" {
|
||||
p.Name = name
|
||||
}
|
||||
if comment := subParam(c, "comment"); comment != "" {
|
||||
p.Comment = comment
|
||||
}
|
||||
if pub := subParam(c, "public"); pub != "" {
|
||||
p.Public = strings.EqualFold(pub, "true")
|
||||
}
|
||||
_ = db.UpdateMediaPlaylist(p)
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
func subDeletePlaylist(c *gin.Context, user *model.User) {
|
||||
id, err := strconv.ParseUint(subParam(c, "id"), 10, 64)
|
||||
if err != nil {
|
||||
subError(c, 10, "missing id")
|
||||
return
|
||||
}
|
||||
p, err := db.GetMediaPlaylist(uint(id))
|
||||
if err != nil || (p.UserID != user.ID && !user.IsAdmin()) {
|
||||
subError(c, 70, "playlist not found")
|
||||
return
|
||||
}
|
||||
_ = db.DeleteMediaPlaylist(p.ID)
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
// ==================== 扫描 ====================
|
||||
|
||||
func subSongCount(user *model.User) int64 {
|
||||
lib := loadEmbyLibrary(user)
|
||||
return int64(len(lib.music))
|
||||
}
|
||||
|
||||
func subStartScan(c *gin.Context) {
|
||||
for _, mt := range MediaTypeEnabledList() {
|
||||
cfg, err := db.GetMediaConfig(mt)
|
||||
if err != nil || cfg == nil {
|
||||
continue
|
||||
}
|
||||
media.ScanMedia(cfg)
|
||||
}
|
||||
subRespond(c, &subsonicResponse{ScanStatus: &subsonicScanStatus{Scanning: true}})
|
||||
}
|
||||
|
||||
// ==================== 播放队列（内存） ====================
|
||||
|
||||
var (
|
||||
subQueueMu sync.Mutex
|
||||
subQueues = map[uint]*subsonicPlayQueue{}
|
||||
)
|
||||
|
||||
func subSavePlayQueue(c *gin.Context, user *model.User) {
|
||||
ids := c.Query("id")
|
||||
if ids == "" {
|
||||
_ = c.Request.ParseForm()
|
||||
ids = c.Request.PostForm.Get("id")
|
||||
}
|
||||
q := &subsonicPlayQueue{
|
||||
Current: subParam(c, "current"),
|
||||
Position: int64(subIntParam(c, "position", 0)),
|
||||
Username: user.Username,
|
||||
Changed: time.Now().UTC().Format("2006-01-02T15:04:05.000Z"),
|
||||
}
|
||||
if ids != "" {
|
||||
lib := loadEmbyLibrary(user)
|
||||
for _, id := range strings.Split(ids, ",") {
|
||||
if song, ok := lib.songByID(strings.TrimSpace(id)); ok {
|
||||
q.Entry = append(q.Entry, lib.subChild(song))
|
||||
}
|
||||
}
|
||||
}
|
||||
subQueueMu.Lock()
|
||||
subQueues[user.ID] = q
|
||||
subQueueMu.Unlock()
|
||||
subRespond(c, &subsonicResponse{})
|
||||
}
|
||||
|
||||
func subLoadPlayQueue(user *model.User) *subsonicPlayQueue {
|
||||
subQueueMu.Lock()
|
||||
defer subQueueMu.Unlock()
|
||||
if q, ok := subQueues[user.ID]; ok {
|
||||
return q
|
||||
}
|
||||
return &subsonicPlayQueue{Username: user.Username}
|
||||
}
|
||||
|
||||
func subNowPlayingEntries() []subsonicChild { return []subsonicChild{} }
|
||||
@@ -0,0 +1,656 @@
|
||||
package handles
|
||||
|
||||
import (
|
||||
"crypto/subtle"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/conf"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/driver"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/fs"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/op"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/setting"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/sign"
|
||||
"github.com/OpenListTeam/OpenList/v4/internal/transcode"
|
||||
"github.com/OpenListTeam/OpenList/v4/server/common"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// 探测阶段的独立并发上限。
|
||||
//
|
||||
// 转码配额（每用户 3 个活跃任务）只约束「入队」，而配额判定发生在探测之后：
|
||||
// 并发请求可能同时通过配额预检、各起一个最长 30s 的 ffprobe。这里给探测单独
|
||||
// 设一道闸，保证这段窗口内的进程数有界。每用户上限取任务配额值，正常并发播放
|
||||
// 不会被误伤（用户有 3 个任务时预检已经先拒绝了）。
|
||||
const (
|
||||
maxConcurrentProbes = 8
|
||||
maxProbesPerUser = transcode.MaxActiveJobsPerUser
|
||||
)
|
||||
|
||||
type probeLimiter struct {
|
||||
mu sync.Mutex
|
||||
active int
|
||||
users map[string]int
|
||||
}
|
||||
|
||||
var probeSlots = &probeLimiter{users: make(map[string]int)}
|
||||
|
||||
func (p *probeLimiter) acquire(owner string) bool {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.active >= maxConcurrentProbes || p.users[owner] >= maxProbesPerUser {
|
||||
return false
|
||||
}
|
||||
p.active++
|
||||
p.users[owner]++
|
||||
return true
|
||||
}
|
||||
|
||||
func (p *probeLimiter) release(owner string) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.active > 0 {
|
||||
p.active--
|
||||
}
|
||||
if n := p.users[owner]; n <= 1 {
|
||||
delete(p.users, owner)
|
||||
} else {
|
||||
p.users[owner] = n - 1
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 播放端：/api/transcode/play 发起或复用一个转码任务
|
||||
// ============================================================
|
||||
|
||||
// TranscodePlay 入口：POST /api/fs/transcode body: {path}
|
||||
//
|
||||
// 行为：
|
||||
// 1. 检查 transcode_enabled
|
||||
// 2. fs.Get 获取文件并跑 Decide
|
||||
// 3. 不需要转码 → 返回 {transcode:false, direct: true}
|
||||
// 4. 需要转码 → 创建 Job 入队，返回 {transcode:true, job_id, master_url}
|
||||
func TranscodePlay(c *gin.Context) {
|
||||
if !transcode.IsEnabled() {
|
||||
common.SuccessResp(c, gin.H{"transcode": false, "reason": "disabled"})
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Path string `json:"path" binding:"required"`
|
||||
}
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
// 该路由挂在 Auth(true)（允许被禁用的访客通过）之下，必须在这里自行拒绝，
|
||||
// 否则关闭访客后匿名请求仍能触发 ffprobe/ffmpeg 并占用转码缓存。
|
||||
user := mediaUser(c)
|
||||
if user == nil || (user.IsGuest() && user.Disabled) {
|
||||
common.ErrorStrResp(c, "Guest user is disabled, login please", 401)
|
||||
return
|
||||
}
|
||||
if !CanUserAccessPath(user, req.Path) {
|
||||
common.ErrorStrResp(c, "path is not accessible", 403)
|
||||
return
|
||||
}
|
||||
obj, err := fs.Get(c.Request.Context(), req.Path, &fs.GetArgs{NoLog: true})
|
||||
if err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
if obj.IsDir() {
|
||||
common.ErrorStrResp(c, "path is a directory", 400)
|
||||
return
|
||||
}
|
||||
dec := transcode.Decide(transcode.DecideRequest{
|
||||
FileName: obj.GetName(),
|
||||
FileSize: obj.GetSize(),
|
||||
})
|
||||
if !dec.NeedTranscode {
|
||||
common.SuccessResp(c, gin.H{
|
||||
"transcode": false,
|
||||
"reason": dec.Reason,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
mgr := transcode.Default()
|
||||
mgr.Start()
|
||||
|
||||
// apiURL 仅用于返回给浏览器的 master_url（需要跟随请求 Host 才能被客户端访问）。
|
||||
apiURL := common.GetApiUrlFromRequest(c.Request)
|
||||
|
||||
// 配额键：普通用户按 ID，访客按 IP——访客共用一个账号，
|
||||
// 否则一个匿名访客就能占满全部匿名转码配额。
|
||||
ownerID := "user:" + strconv.FormatUint(uint64(user.ID), 10)
|
||||
if user.IsGuest() {
|
||||
ownerID = "guest:" + c.ClientIP()
|
||||
}
|
||||
|
||||
// 同一用户重复请求同一文件：直接复用进行中的任务，避免重复 ffprobe 与重复转码。
|
||||
if exist, ok := mgr.Scheduler.FindReusable(ownerID, req.Path); ok {
|
||||
respondTranscodeJob(c, apiURL, exist)
|
||||
return
|
||||
}
|
||||
|
||||
// 【内存优化】优先尝试解析为本地文件路径，让 ffmpeg 直接读取本地文件，
|
||||
// 完全绕过 HTTP /d 代理路径——避免 net.Downloader 给每个 range 请求分配
|
||||
// 高达 MaxBufferLimit (默认系统内存 5%) 的大缓冲。这是降低 Go 进程内存
|
||||
// 占用最有效的方法。如果不是本地驱动则退回到 HTTP 签名 URL。
|
||||
var sourceURL string
|
||||
if localFile, ok := resolveLocalFilePath(req.Path); ok {
|
||||
// ffmpeg 接受本地路径作为输入，无需 file:// 前缀
|
||||
sourceURL = localFile
|
||||
fmt.Printf("[transcode] using local file path for ffmpeg: %s\n", localFile)
|
||||
} else {
|
||||
// 远程驱动：构造签名 HTTP URL。
|
||||
// 这个 URL 会被服务端 ffmpeg 直接访问，绝不能使用 Host / X-Forwarded-Host：
|
||||
// 未配置 site_url 时攻击者可借此让服务器去拉取任意地址（SSRF）。
|
||||
base := trustedInternalBaseURL()
|
||||
if base == "" {
|
||||
// 仅 HTTPS 监听且未配置 site_url 时无可用基址；
|
||||
// 若继续拼出 "/d/..." 会被当成绝对本地路径，ffmpeg 只会报文件不存在。
|
||||
common.ErrorStrResp(
|
||||
c,
|
||||
"transcode needs site_url (or an HTTP port) to fetch the source",
|
||||
503,
|
||||
)
|
||||
return
|
||||
}
|
||||
signedPath := sign.Sign(req.Path)
|
||||
sourceURL = fmt.Sprintf("%s/d%s?sign=%s", base, encodePath(req.Path), signedPath)
|
||||
}
|
||||
if !transcode.ValidSourceInput(sourceURL) {
|
||||
common.ErrorStrResp(c, "unsupported transcode source", 400)
|
||||
return
|
||||
}
|
||||
|
||||
// 创建 Job
|
||||
job := transcode.NewJob()
|
||||
job.Path = req.Path
|
||||
job.OwnerID = ownerID
|
||||
job.SourceURL = sourceURL
|
||||
// 配额预检：ProbeSource 会起一个最长 30s 的 ffprobe 子进程，
|
||||
// 不能等探测完才告诉用户「已有 3 个活跃任务」。最终判定仍由 SubmitLimited 原子完成。
|
||||
if err := mgr.Scheduler.Capacity(ownerID); err != nil {
|
||||
common.ErrorResp(c, err, 429)
|
||||
return
|
||||
}
|
||||
if !probeSlots.acquire(ownerID) {
|
||||
common.ErrorStrResp(c, "too many concurrent transcode requests", 429)
|
||||
return
|
||||
}
|
||||
// 【关键】探测视频总时长（异步会让首响应变慢，但只有一次，必须等待）
|
||||
// 总时长用于生成完整的 playlist，避免 HLS.js 时长显示错误和拖动跳跃问题
|
||||
probe, perr := func() (transcode.SourceProbe, error) {
|
||||
// 用 defer 释放：探测内部 panic 时也不能把槽位永久占住
|
||||
defer probeSlots.release(ownerID)
|
||||
return transcode.ProbeSource(sourceURL)
|
||||
}()
|
||||
if perr != nil {
|
||||
// 探测失败不阻塞流程，但记录日志；后续 playlist 会缺少总时长信息
|
||||
fmt.Printf("[transcode] probe source failed: %v\n", perr)
|
||||
probe = transcode.SourceProbe{Size: obj.GetSize()}
|
||||
} else if probe.Size == 0 {
|
||||
probe.Size = obj.GetSize()
|
||||
}
|
||||
job.Probe = probe
|
||||
profile := transcode.BuildProfile()
|
||||
job.Profiles = []transcode.Profile{profile}
|
||||
job.Output = transcode.BuildOutputSpec()
|
||||
timeoutMin := setting.GetInt(conf.TranscodeJobTimeoutMin, 120)
|
||||
job.Deadline = time.Now().Add(time.Duration(timeoutMin) * time.Minute).Unix()
|
||||
submitted, _, err := mgr.Scheduler.SubmitLimited(job)
|
||||
if err != nil {
|
||||
// 队列已满 / 用户配额超限：返回 429，前端可降级为直链播放
|
||||
common.ErrorResp(c, err, 429)
|
||||
return
|
||||
}
|
||||
respondTranscodeJob(c, apiURL, submitted)
|
||||
}
|
||||
|
||||
// respondTranscodeJob 返回转码任务的播放信息
|
||||
func respondTranscodeJob(c *gin.Context, apiURL string, job *transcode.Job) {
|
||||
profileName := ""
|
||||
if len(job.Profiles) > 0 {
|
||||
profileName = job.Profiles[0].Name
|
||||
}
|
||||
masterURL := fmt.Sprintf("%s/tc/%s/%s/master.m3u8", apiURL, job.ID, job.PlayToken)
|
||||
common.SuccessResp(c, gin.H{
|
||||
"transcode": true,
|
||||
"job_id": job.ID,
|
||||
"master_url": masterURL,
|
||||
"profile": profileName,
|
||||
})
|
||||
}
|
||||
|
||||
func encodePath(p string) string {
|
||||
// 与 down.go 保持一致：保留 '/' 不转义
|
||||
parts := strings.Split(p, "/")
|
||||
for i, s := range parts {
|
||||
parts[i] = url.PathEscape(s)
|
||||
}
|
||||
return strings.Join(parts, "/")
|
||||
}
|
||||
|
||||
// trustedInternalBaseURL 返回供服务端 ffmpeg 访问本站 /d 接口的可信基址，不读取任何请求头。
|
||||
// 优先使用管理员配置的 site_url；未配置时回环到本机实际监听的 HTTP 端口。
|
||||
// 两者都不可用（例如仅监听 Unix socket）时返回空串，调用方应拒绝转码。
|
||||
func trustedInternalBaseURL() string {
|
||||
if site := strings.TrimSuffix(conf.Conf.SiteURL, "/"); strings.HasPrefix(site, "http") {
|
||||
return site
|
||||
}
|
||||
if p := conf.Conf.Scheme.HttpPort; p > 0 {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", p)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// resolveLocalFilePath 尝试把 OpenList 路径解析为宿主机文件系统的实际路径。
|
||||
// 仅对本地驱动 (Local) 有效，其他驱动（云盘等）返回 false。
|
||||
//
|
||||
// 这是内存优化的关键路径：本地文件场景下让 ffmpeg 直接读文件，可以完全绕过
|
||||
// HTTP /d 代理路径上的 net.Downloader 大块缓冲（默认每个 range 请求最高
|
||||
// 占用 MaxBufferLimit = 系统内存 5%，多 chunk 并发 + 多 range 累计可达数 GB）。
|
||||
func resolveLocalFilePath(rawPath string) (string, bool) {
|
||||
storage, actualPath, err := op.GetStorageAndActualPath(rawPath)
|
||||
if err != nil || storage == nil {
|
||||
return "", false
|
||||
}
|
||||
if storage.Config().Name != "Local" {
|
||||
return "", false
|
||||
}
|
||||
rooter, ok := storage.(driver.IRootPath)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
root := rooter.GetRootPath()
|
||||
if root == "" {
|
||||
return "", false
|
||||
}
|
||||
full := filepath.Join(root, actualPath)
|
||||
// 只接受真实存在的文件，避免传给 ffmpeg 一个无效路径
|
||||
fi, err := os.Stat(full)
|
||||
if err != nil || fi.IsDir() {
|
||||
return "", false
|
||||
}
|
||||
return full, true
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 播放端：/tc/:job/:token/... Master/Variant playlist & 切片
|
||||
// ============================================================
|
||||
|
||||
// TCMaster GET /tc/:job/:token/master.m3u8
|
||||
func TCMaster(c *gin.Context) {
|
||||
jobID := c.Param("job")
|
||||
token := c.Param("token")
|
||||
job, ok := transcode.Default().Scheduler.Get(jobID)
|
||||
if !ok {
|
||||
c.String(http.StatusNotFound, "job not found")
|
||||
return
|
||||
}
|
||||
if !tokenMatches(job.PlayToken, token) {
|
||||
c.String(http.StatusForbidden, "bad token")
|
||||
return
|
||||
}
|
||||
if len(job.Profiles) == 0 {
|
||||
c.String(http.StatusInternalServerError, "no profiles")
|
||||
return
|
||||
}
|
||||
apiURL := common.GetApiUrlFromRequest(c.Request)
|
||||
baseURL := fmt.Sprintf("%s/tc/%s/%s/", apiURL, jobID, token)
|
||||
body := transcode.Default().Cache.MasterPlaylist(job.Profiles, baseURL)
|
||||
job.Touch()
|
||||
c.Header("Content-Type", "application/vnd.apple.mpegurl")
|
||||
// Master playlist 内容在 job 生命周期内不变，设置较长缓存
|
||||
c.Header("Cache-Control", "public, max-age=3600")
|
||||
c.String(http.StatusOK, body)
|
||||
}
|
||||
|
||||
// TCPlaylist GET /tc/:job/:token/:profile/playlist.m3u8
|
||||
// 等待首切片就绪后返回；阻塞最多 30s
|
||||
func TCPlaylist(c *gin.Context) {
|
||||
jobID := c.Param("job")
|
||||
token := c.Param("token")
|
||||
profile := c.Param("profile")
|
||||
mgr := transcode.Default()
|
||||
job, ok := mgr.Scheduler.Get(jobID)
|
||||
if !ok {
|
||||
c.String(http.StatusNotFound, "job not found")
|
||||
return
|
||||
}
|
||||
if !tokenMatches(job.PlayToken, token) {
|
||||
c.String(http.StatusForbidden, "bad token")
|
||||
return
|
||||
}
|
||||
if !transcode.ValidProfileName(profile) {
|
||||
c.String(http.StatusBadRequest, "bad profile")
|
||||
return
|
||||
}
|
||||
// 【关键】如果 job 之前因 idle 被 cancel，这里重新激活：复用 jobID/token 重新入队
|
||||
// 让 worker 再次启动 ffmpeg 转码，避免播放器看到 404 后无法恢复。
|
||||
// 配额/队列已满时不再起新的 ffmpeg，但 playlist 与已缓存的切片仍应可读：
|
||||
// 这里只记下 quotaBlocked，交给切片请求在「缓存也没有」时才返回 429。
|
||||
quotaBlocked := false
|
||||
if st := job.GetStatus(); st == transcode.JobCancelled || st == transcode.JobFinished || st == transcode.JobFailed {
|
||||
if _, rerr := mgr.Scheduler.Reactivate(jobID); rerr == nil {
|
||||
fmt.Printf("[transcode] reactivate job %s on playlist request (was %s)\n", jobID, st)
|
||||
} else if errors.Is(rerr, transcode.ErrUserQuota) || errors.Is(rerr, transcode.ErrQueueFull) {
|
||||
quotaBlocked = true
|
||||
fmt.Printf("[transcode] reactivate job %s on playlist request rejected (%v), serving from cache only\n", jobID, rerr)
|
||||
}
|
||||
}
|
||||
job.Touch()
|
||||
// 等首切片；配额已满时无法启动新的转码，没必要干等 30s
|
||||
if !quotaBlocked {
|
||||
select {
|
||||
case <-job.WaitReady():
|
||||
case <-time.After(30 * time.Second):
|
||||
c.String(http.StatusGatewayTimeout, "timeout waiting first segment")
|
||||
return
|
||||
}
|
||||
}
|
||||
if job.Status == transcode.JobFailed {
|
||||
c.String(http.StatusInternalServerError, "transcode failed: "+job.Error)
|
||||
return
|
||||
}
|
||||
apiURL := common.GetApiUrlFromRequest(c.Request)
|
||||
baseURL := fmt.Sprintf("%s/tc/%s/%s/%s/", apiURL, jobID, token, profile)
|
||||
isFinished := job.Status == transcode.JobFinished
|
||||
// 【关键】传入总时长，让 BuildPlaylist 能够生成完整切片列表（VOD 模式），
|
||||
// 这样 HLS.js 就知道视频真实总时长，避免时长显示错误和拖动跳跃问题。
|
||||
body := transcode.Default().Cache.BuildPlaylist(jobID, profile, job.Output.SegmentDuration,
|
||||
isFinished, baseURL, job.Probe.Duration)
|
||||
c.Header("Content-Type", "application/vnd.apple.mpegurl")
|
||||
if isFinished || job.Probe.Duration > 0 {
|
||||
// VOD 模式：playlist 完整不变，长缓存
|
||||
c.Header("Cache-Control", "public, max-age=3600")
|
||||
} else {
|
||||
// 没有总时长时退回 EVENT 模式：短缓存
|
||||
c.Header("Cache-Control", "public, max-age=2")
|
||||
}
|
||||
c.String(http.StatusOK, body)
|
||||
}
|
||||
|
||||
// TCSegment GET /tc/:job/:token/:profile/seg-:seq.ts
|
||||
// 切片就绪则直接返回；否则等待最多 60s
|
||||
func TCSegment(c *gin.Context) {
|
||||
jobID := c.Param("job")
|
||||
token := c.Param("token")
|
||||
profile := c.Param("profile")
|
||||
segName := c.Param("seg")
|
||||
mgr := transcode.Default()
|
||||
job, ok := mgr.Scheduler.Get(jobID)
|
||||
if !ok {
|
||||
c.String(http.StatusNotFound, "job not found")
|
||||
return
|
||||
}
|
||||
if !tokenMatches(job.PlayToken, token) {
|
||||
c.String(http.StatusForbidden, "bad token")
|
||||
return
|
||||
}
|
||||
if !transcode.ValidProfileName(profile) {
|
||||
c.String(http.StatusBadRequest, "bad profile")
|
||||
return
|
||||
}
|
||||
// 【关键】job 之前因 idle 被 cancel 时，复用同一 jobID/token 重新入队启动转码。
|
||||
// 这样浏览器持有的旧 master.m3u8 + 切片 URL 仍然有效，暂停再继续也不会 404。
|
||||
// 配额/队列已满时不重新起 ffmpeg，但已缓存的切片依然要能命中（回看已转完的视频）。
|
||||
quotaBlocked := false
|
||||
if st := job.GetStatus(); st == transcode.JobCancelled || st == transcode.JobFinished || st == transcode.JobFailed {
|
||||
if _, rerr := mgr.Scheduler.Reactivate(jobID); rerr == nil {
|
||||
fmt.Printf("[transcode] reactivate job %s on segment request (was %s)\n", jobID, st)
|
||||
} else if errors.Is(rerr, transcode.ErrUserQuota) || errors.Is(rerr, transcode.ErrQueueFull) {
|
||||
quotaBlocked = true
|
||||
fmt.Printf("[transcode] reactivate job %s on segment request rejected (%v), serving from cache only\n", jobID, rerr)
|
||||
}
|
||||
}
|
||||
// seg-N.ts
|
||||
name := strings.TrimSuffix(strings.TrimPrefix(segName, "seg-"), ".ts")
|
||||
seq, err := strconv.Atoi(name)
|
||||
if err != nil {
|
||||
c.String(http.StatusBadRequest, "bad segment name")
|
||||
return
|
||||
}
|
||||
cache := mgr.Cache
|
||||
// 【智能调度】触发对应 chunk 的 ffmpeg 启动（如已运行则只更新 LastAccess）
|
||||
// 这是用户拖动进度条到任意位置时能快速响应的关键。
|
||||
// 配额已满时不能触发新的 ffmpeg，否则「限制并发」就没有意义了。
|
||||
if mgr.Chunks != nil && !quotaBlocked {
|
||||
chunkIdx := mgr.Chunks.EnsureChunkRunningForSeg(jobID, profile, seq)
|
||||
fmt.Printf("[tc-segment] req seq=%d chunk=%d job=%s\n", seq, chunkIdx, jobID)
|
||||
}
|
||||
info, err := cache.GetSegment(jobID, profile, seq)
|
||||
if err != nil {
|
||||
if quotaBlocked {
|
||||
// 配额已满且该切片不在缓存里：只能拒绝，让前端降级为直链播放
|
||||
c.String(http.StatusTooManyRequests, "too many active transcode jobs")
|
||||
return
|
||||
}
|
||||
// 等待
|
||||
waitStart := time.Now()
|
||||
info, err = cache.WaitForSegment(jobID, profile, seq, 60*time.Second)
|
||||
if err != nil {
|
||||
fmt.Printf("[tc-segment] req seq=%d FAILED after %.1fs: %v\n", seq, time.Since(waitStart).Seconds(), err)
|
||||
c.String(http.StatusNotFound, "segment not available: "+err.Error())
|
||||
return
|
||||
}
|
||||
fmt.Printf("[tc-segment] req seq=%d ready after %.1fs\n", seq, time.Since(waitStart).Seconds())
|
||||
}
|
||||
job.Touch()
|
||||
// 切片内容不可变，设置强缓存避免浏览器/HLS.js 反复请求同一切片
|
||||
c.Header("Cache-Control", "public, max-age=86400, immutable")
|
||||
// 【内存优化】流式发送切片文件，避免 gin c.File() 将整个切片读入内存
|
||||
// 注意：不能使用 http.ServeFile，它会自动嗅探 Content-Type 并处理 Range 请求，
|
||||
// 与 gin 已写入的响应头冲突，导致 HLS.js 收到异常数据触发 bufferAppendError。
|
||||
f, err := os.Open(info.Path)
|
||||
if err != nil {
|
||||
c.String(http.StatusInternalServerError, "open segment: "+err.Error())
|
||||
return
|
||||
}
|
||||
defer f.Close()
|
||||
fi, err := f.Stat()
|
||||
if err != nil {
|
||||
c.String(http.StatusInternalServerError, "stat segment: "+err.Error())
|
||||
return
|
||||
}
|
||||
c.DataFromReader(http.StatusOK, fi.Size(), "video/mp2t", f, nil)
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Worker 协议端
|
||||
// ============================================================
|
||||
|
||||
// WorkerRegister POST /api/transcode/worker/register
|
||||
func WorkerRegister(c *gin.Context) {
|
||||
if !transcode.IsEnabled() {
|
||||
common.ErrorStrResp(c, "transcode disabled", 403)
|
||||
return
|
||||
}
|
||||
if !checkWorkerSecret(c) {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
var req transcode.RegisterRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
w := transcode.Default().Registry.Register(&req, false)
|
||||
common.SuccessResp(c, transcode.RegisterResponse{
|
||||
WorkerID: w.ID,
|
||||
HeartbeatInterval: 10,
|
||||
ClaimStrategy: "pull",
|
||||
ProtocolVersion: transcode.ProtocolVersion,
|
||||
})
|
||||
}
|
||||
|
||||
// WorkerHeartbeat POST /api/transcode/worker/heartbeat
|
||||
func WorkerHeartbeat(c *gin.Context) {
|
||||
if !checkWorkerSecret(c) {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
var req transcode.HeartbeatRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
mgr := transcode.Default()
|
||||
if _, ok := mgr.Registry.Heartbeat(&req); !ok {
|
||||
common.SuccessResp(c, transcode.HeartbeatResponse{OK: false, Kick: true})
|
||||
return
|
||||
}
|
||||
cancellations := mgr.Scheduler.PopCancellations(req.WorkerID)
|
||||
common.SuccessResp(c, transcode.HeartbeatResponse{OK: true, Cancel: cancellations})
|
||||
}
|
||||
|
||||
// WorkerClaim POST /api/transcode/worker/claim
|
||||
func WorkerClaim(c *gin.Context) {
|
||||
if !checkWorkerSecret(c) {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
var req transcode.ClaimRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
wait := time.Duration(req.Wait) * time.Second
|
||||
if wait > 30*time.Second {
|
||||
wait = 30 * time.Second
|
||||
}
|
||||
jobs := transcode.Default().Scheduler.Claim(req.WorkerID, req.Slots, wait)
|
||||
common.SuccessResp(c, transcode.ClaimResponse{Jobs: jobs})
|
||||
}
|
||||
|
||||
// WorkerSegmentUpload PUT /api/transcode/worker/segment?job=&profile=&seq=&duration=&final=
|
||||
func WorkerSegmentUpload(c *gin.Context) {
|
||||
jobID := c.Query("job")
|
||||
profile := c.Query("profile")
|
||||
seq, _ := strconv.Atoi(c.Query("seq"))
|
||||
dur, _ := strconv.ParseFloat(c.Query("duration"), 64)
|
||||
isFinal, _ := strconv.ParseBool(c.Query("final"))
|
||||
job, ok := transcode.Default().Scheduler.Get(jobID)
|
||||
if !ok {
|
||||
common.ErrorStrResp(c, "job not found", 404)
|
||||
return
|
||||
}
|
||||
// 校验 callback_token (Bearer)，使用恒定时间比较避免时序侧信道
|
||||
auth := c.GetHeader("Authorization")
|
||||
expect := "Bearer " + job.CallbackToken
|
||||
if subtle.ConstantTimeCompare([]byte(auth), []byte(expect)) != 1 {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
// profile 会被拼进缓存目录路径，必须是白名单字符，防止 ../ 路径穿越
|
||||
if !transcode.ValidProfileName(profile) {
|
||||
common.ErrorStrResp(c, "invalid profile", 400)
|
||||
return
|
||||
}
|
||||
// 只接受任务声明过的档位，避免持有 callback_token 的一方往任意档位目录写数据
|
||||
if !jobHasProfile(job, profile) {
|
||||
common.ErrorStrResp(c, "profile not declared for this job", 400)
|
||||
return
|
||||
}
|
||||
defer c.Request.Body.Close()
|
||||
_, err := transcode.Default().Cache.PutSegment(jobID, profile, seq, dur, c.Request.Body, isFinal)
|
||||
if err != nil {
|
||||
switch err {
|
||||
case transcode.ErrInvalidProfile, transcode.ErrInvalidJobID, transcode.ErrInvalidSegment:
|
||||
common.ErrorResp(c, err, 400)
|
||||
case transcode.ErrSegmentTooLarge:
|
||||
common.ErrorResp(c, err, 413)
|
||||
default:
|
||||
common.ErrorResp(c, err, 500)
|
||||
}
|
||||
return
|
||||
}
|
||||
if seq == 0 {
|
||||
job.MarkReady()
|
||||
}
|
||||
c.Status(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// WorkerJobFinish POST /api/transcode/worker/job/finish
|
||||
func WorkerJobFinish(c *gin.Context) {
|
||||
var req transcode.FinishRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
job, ok := transcode.Default().Scheduler.Get(req.JobID)
|
||||
if !ok {
|
||||
common.ErrorStrResp(c, "job not found", 404)
|
||||
return
|
||||
}
|
||||
auth := c.GetHeader("Authorization")
|
||||
if !tokenMatches("Bearer "+job.CallbackToken, auth) {
|
||||
common.ErrorStrResp(c, "unauthorized", 401)
|
||||
return
|
||||
}
|
||||
if err := transcode.Default().Scheduler.Finish(&req); err != nil {
|
||||
common.ErrorResp(c, err, 500)
|
||||
return
|
||||
}
|
||||
common.SuccessResp(c)
|
||||
}
|
||||
|
||||
// ListTranscodeWorkers GET /api/admin/transcode/workers (admin)
|
||||
func ListTranscodeWorkers(c *gin.Context) {
|
||||
common.SuccessResp(c, transcode.Default().Registry.List())
|
||||
}
|
||||
|
||||
// ListTranscodeJobs GET /api/admin/transcode/jobs (admin)
|
||||
func ListTranscodeJobs(c *gin.Context) {
|
||||
common.SuccessResp(c, transcode.Default().Scheduler.ListJobs())
|
||||
}
|
||||
|
||||
// CancelTranscodeJob POST /api/admin/transcode/cancel
|
||||
func CancelTranscodeJob(c *gin.Context) {
|
||||
var req struct {
|
||||
JobID string `json:"job_id" binding:"required"`
|
||||
}
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
common.ErrorResp(c, err, 400)
|
||||
return
|
||||
}
|
||||
transcode.Default().Scheduler.Cancel(req.JobID)
|
||||
common.SuccessResp(c)
|
||||
}
|
||||
|
||||
// jobHasProfile 判断 profile 是否属于任务声明的输出档位
|
||||
func jobHasProfile(job *transcode.Job, profile string) bool {
|
||||
if job == nil {
|
||||
return false
|
||||
}
|
||||
for _, p := range job.Profiles {
|
||||
if p.Name == profile {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// tokenMatches 恒定时间比较凭据，避免按字节比较造成的时序侧信道
|
||||
func tokenMatches(expected, got string) bool {
|
||||
if expected == "" || got == "" {
|
||||
return false
|
||||
}
|
||||
return subtle.ConstantTimeCompare([]byte(expected), []byte(got)) == 1
|
||||
}
|
||||
|
||||
func checkWorkerSecret(c *gin.Context) bool {
|
||||
auth := c.GetHeader("Authorization")
|
||||
auth = strings.TrimPrefix(auth, "Bearer ")
|
||||
return transcode.VerifyWorkerSecret(auth)
|
||||
}
|
||||
@@ -67,6 +67,11 @@ func Init(e *gin.Engine) {
|
||||
g.HEAD("/sad/:sid", middlewares.EmptyPathParse, middlewares.SharingIdParse, handles.SharingArchiveExtract)
|
||||
g.HEAD("/sad/:sid/*path", middlewares.PathParse, middlewares.SharingIdParse, handles.SharingArchiveExtract)
|
||||
|
||||
// 转码播放：master/variant playlist 与切片下发；token 已包含在路径中作为鉴权
|
||||
g.GET("/tc/:job/:token/master.m3u8", handles.TCMaster)
|
||||
g.GET("/tc/:job/:token/:profile/playlist.m3u8", handles.TCPlaylist)
|
||||
g.GET("/tc/:job/:token/:profile/:seg", handles.TCSegment)
|
||||
|
||||
api := g.Group("/api")
|
||||
auth := api.Group("", middlewares.Auth(false))
|
||||
webauthn := api.Group("/authn", middlewares.Authn)
|
||||
@@ -76,6 +81,10 @@ func Init(e *gin.Engine) {
|
||||
api.POST("/auth/login/ldap", handles.LoginLdap)
|
||||
auth.GET("/me", handles.CurrentUser)
|
||||
auth.POST("/me/update", handles.UpdateCurrent)
|
||||
// 媒体应用密钥（每个登录用户都可以为自己的客户端签发密钥）
|
||||
auth.GET("/me/appkey/list", handles.ListMediaAppKeys)
|
||||
auth.POST("/me/appkey/create", handles.CreateMediaAppKey)
|
||||
auth.POST("/me/appkey/delete", handles.DeleteMediaAppKey)
|
||||
auth.GET("/me/sshkey/list", handles.ListMyPublicKey)
|
||||
auth.POST("/me/sshkey/add", handles.AddMyPublicKey)
|
||||
auth.POST("/me/sshkey/delete", handles.DeleteMyPublicKey)
|
||||
@@ -100,12 +109,51 @@ func Init(e *gin.Engine) {
|
||||
// no need auth
|
||||
public := api.Group("/public")
|
||||
public.Any("/settings", handles.PublicSettings)
|
||||
public.GET("/media_status", handles.PublicMediaStatus)
|
||||
// 封面代理与本地缩略图（媒体库页面使用，免鉴权）
|
||||
public.GET("/media/proxy", handles.PublicMediaProxy)
|
||||
public.GET("/media/thumb", handles.PublicMediaThumb)
|
||||
public.Any("/offline_download_tools", handles.OfflineDownloadTools)
|
||||
public.Any("/archive_extensions", handles.ArchiveExtensions)
|
||||
public.Any("/init_status", handles.InitStatus)
|
||||
public.POST("/init/setup", handles.InitSetup)
|
||||
|
||||
// 转码 Worker 协议端点（远程 Worker 节点对接，由共享密钥/token 鉴权，无需登录）
|
||||
transcodeWorker := api.Group("/transcode/worker")
|
||||
transcodeWorker.POST("/register", handles.WorkerRegister)
|
||||
transcodeWorker.POST("/heartbeat", handles.WorkerHeartbeat)
|
||||
transcodeWorker.POST("/claim", handles.WorkerClaim)
|
||||
transcodeWorker.PUT("/segment", handles.WorkerSegmentUpload)
|
||||
transcodeWorker.POST("/job/finish", handles.WorkerJobFinish)
|
||||
|
||||
// Emby 兼容 API（自带鉴权与开关判断，路径大小写由 handler 归一化）
|
||||
// 使用 @ 前缀，避免与用户自己创建的 /emby、/rest 目录冲突
|
||||
g.Any("/@emby/*path", handles.EmbyAPI)
|
||||
g.Any("/@mediabrowser/*path", handles.EmbyAPI)
|
||||
// OpenSubsonic API（自带鉴权与开关判断）
|
||||
g.Any("/@rest/*path", handles.SubsonicAPI)
|
||||
|
||||
_fs(auth.Group("/fs"))
|
||||
// 媒体库公开 API：与文件浏览一致，允许访客访问（访客仍受自身 base_path/权限限制）
|
||||
mediaPublic := auth.Group("/fs/media", handles.RequireMediaEnabled)
|
||||
mediaPublic.GET("/list", handles.PublicListMedia)
|
||||
mediaPublic.GET("/item/:id", handles.PublicGetMedia)
|
||||
mediaPublic.GET("/albums", handles.PublicListAlbums)
|
||||
mediaPublic.GET("/album", handles.PublicGetAlbum)
|
||||
mediaPublic.GET("/folders", handles.PublicListFolders)
|
||||
mediaPublic.GET("/scan_paths", handles.PublicListScanPaths)
|
||||
mediaPublic.GET("/strm", handles.GetMediaStrm)
|
||||
mediaPublic.GET("/stream_url", handles.GetMediaStreamURL)
|
||||
// 播放进度与收藏（与 Emby / Subsonic 共用同一份用户数据）
|
||||
mediaPublic.GET("/userdata", handles.GetMediaUserDataBatch)
|
||||
mediaPublic.GET("/progress", handles.GetMediaProgress)
|
||||
mediaPublic.POST("/progress", handles.SaveMediaProgress)
|
||||
mediaPublic.POST("/favorite", handles.SetMediaFavorite)
|
||||
mediaPublic.GET("/favorites", handles.ListMediaFavoritesAPI)
|
||||
mediaPublic.GET("/resume", handles.ListMediaResumeAPI)
|
||||
// 阅读器批注（登录用户私有数据；访客只读返回空列表，写入直接 403）
|
||||
mediaPublic.GET("/annotations", handles.ListMediaAnnotationsAPI)
|
||||
mediaPublic.POST("/annotations", handles.SyncMediaAnnotationsAPI)
|
||||
fsAndShare(api.Group("/fs", middlewares.Auth(true)))
|
||||
_task(auth.Group("/task", middlewares.AuthNotGuest))
|
||||
_sharing(auth.Group("/share", middlewares.AuthNotGuest))
|
||||
@@ -190,6 +238,42 @@ func admin(g *gin.RouterGroup) {
|
||||
scan.POST("/start", handles.StartManualScan)
|
||||
scan.POST("/stop", handles.StopManualScan)
|
||||
scan.GET("/progress", handles.GetManualScanProgress)
|
||||
|
||||
// 媒体库管理
|
||||
mediaAdmin := g.Group("/media")
|
||||
mediaAdmin.GET("/config/list", handles.ListMediaConfigs)
|
||||
mediaAdmin.POST("/config/save", handles.SaveMediaConfig)
|
||||
mediaAdmin.GET("/items", handles.ListMediaItemsAdmin)
|
||||
mediaAdmin.POST("/items/update", handles.UpdateMediaItemAdmin)
|
||||
mediaAdmin.POST("/items/delete", handles.DeleteMediaItemAdmin)
|
||||
mediaAdmin.POST("/scan/start", handles.StartMediaScan)
|
||||
mediaAdmin.GET("/scan/progress", handles.GetMediaScanProgress)
|
||||
mediaAdmin.POST("/scrape/start", handles.StartMediaScrape)
|
||||
mediaAdmin.GET("/scrape/stats", handles.GetMediaScrapeStats)
|
||||
mediaAdmin.POST("/clear", handles.ClearMediaDB)
|
||||
mediaAdmin.POST("/clear_scrape", handles.ClearMediaScrape)
|
||||
mediaAdmin.POST("/delete_invalid", handles.DeleteInvalidMedia)
|
||||
// 扫描路径管理
|
||||
mediaAdmin.GET("/scan_paths", handles.ListMediaScanPaths)
|
||||
mediaAdmin.POST("/scan_paths/create", handles.CreateMediaScanPath)
|
||||
mediaAdmin.POST("/scan_paths/update", handles.UpdateMediaScanPath)
|
||||
mediaAdmin.POST("/scan_paths/delete", handles.DeleteMediaScanPath)
|
||||
mediaAdmin.POST("/scan_paths/clear", handles.ClearMediaScanPathDB)
|
||||
// 导入 / 导出
|
||||
mediaAdmin.GET("/export", handles.ExportMediaDB)
|
||||
mediaAdmin.POST("/import", handles.ImportMediaDB)
|
||||
// 媒体应用密钥（Subsonic/Emby/MoviePilot 等无交互客户端）
|
||||
mediaAdmin.GET("/appkey/list", handles.ListMediaAppKeys)
|
||||
mediaAdmin.POST("/appkey/create", handles.CreateMediaAppKey)
|
||||
mediaAdmin.POST("/appkey/delete", handles.DeleteMediaAppKey)
|
||||
// STRM 批量生成
|
||||
mediaAdmin.POST("/strm/generate", handles.GenerateMediaStrm)
|
||||
|
||||
// 转码管理
|
||||
transcodeAdmin := g.Group("/transcode")
|
||||
transcodeAdmin.GET("/workers", handles.ListTranscodeWorkers)
|
||||
transcodeAdmin.GET("/jobs", handles.ListTranscodeJobs)
|
||||
transcodeAdmin.POST("/cancel", handles.CancelTranscodeJob)
|
||||
}
|
||||
|
||||
func fsAndShare(g *gin.RouterGroup) {
|
||||
@@ -198,6 +282,9 @@ func fsAndShare(g *gin.RouterGroup) {
|
||||
a := g.Group("/archive")
|
||||
a.Any("/meta", handles.FsArchiveMetaSplit)
|
||||
a.Any("/list", handles.FsArchiveListSplit)
|
||||
|
||||
// 转码 - 用户态：申请播放
|
||||
g.POST("/transcode/play", handles.RequireMediaEnabled, handles.TranscodePlay)
|
||||
}
|
||||
|
||||
func _fs(g *gin.RouterGroup) {
|
||||
|
||||
Reference in new issue
Block a user