* security: harden artifact serialization and loading * ci: restore dependency and title lint compatibility * fix: address artifact test and lint failures * ci: handle current MLflow and Windows OSQP changes * ci: use lxml release with cross-platform wheels * fix: defer optional CVXPY solver imports * docs: mark notebook model artifacts as trusted * docs: mark position artifacts as trusted * style: document intentional lazy import * ci: skip broken released OSQP on Windows Python 3.8 * fix: mark online model artifacts as trusted * test: make MLflow client timing check robust * ci: retry nondeterministic benchmark smoke test * fix: cap filelock before fork safety regression * fix: use filelock release compatible with multiprocessing * fix: preserve artifact loading compatibility after hardening * fix(security): validate derived artifact paths and extend regressions * fix(security): preserve extended data artifact round trips * fix(security): retain legacy NumPy and pandas artifact classes Allow the exact MaskedArray and SparseDtype module paths emitted by the older dependency versions in CI. Preserve restricted loading and cover both old and new paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(data): avoid unused trading calendar lookup in PIT normalization PIT normalization uses report dates and calendar-day offsets, so do not request an unused Baostock calendar. Cover annual and quarterly normalization without online calendar access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(security): require explicit trust in artifact workflows Expose default-off artifact trust at online, delayed-training and meta-workflow entry points. Keep data loading restricted, preserve legacy default recorder calls and restored workflow defaults, and document migration and trust boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(workflow): cover complete DDG-DA and HIST artifact flows Exercise real training, cache and recorder restoration, saved-task replay, signals and portfolio backtests. Fix the cache trust, grouping, relative-path, LightGBM early-stopping and HIST CPU blockers exposed by full execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(security)!: unify artifact trust and publish upgrade guidance Use trusted consistently across public APIs and CLI options. Migrate pre-release saved consent only at serialization boundaries, reject invalid or conflicting settings, and preserve restricted data reads. Add a canonical upgrade guide with README, installation, changelog and runtime-error entry points. BREAKING CHANGE: executable artifact restoration requires explicit trusted=True. HIST mappings use JSON and high-frequency cache paths must remain within artifact_root. See docs/start/artifact_migration.rst for upgrading existing workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(security): remove trial-only trust compatibility Target upgrades from pre-PR Qlib rather than artifacts generated by intermediate PR revisions. Retain restricted defaults for saved objects without consent, unify current APIs, and remove state/keyword migration branches and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: avoid failed dataset probes and space download retries Download the same existing v2 archives used by successful jobs without probing nonexistent package-version assets. Allow five download attempts 60 seconds apart; preserve the full matrix, datasets, noninteractive setup and failure propagation. Cover the exact archives and retry policy in offline CI tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add practical artifact migration examples Show explicit consent for restored online components while preserving the original manager, schema-aware Arrow conversions, timezone-preserving ZoneInfo migration, and restricted round trips into a separate recorder. Explain shared-component scope, staging artifacts and trusted-source prerequisites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflow): preserve fresh artifact workflows with scoped trust Keep immutable cache references intact at open rolling tails. Add explicit per-load record-template consent and per-artifact collector loading options without widening other reads. Restore the original template class after parent-load failures, and support audited business-frequency and NumPy record representations through exact pickle globals. Document the scoped APIs and cover fresh trained-model reads, cache generation, numerical round trips and refusal paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Requirements
Here is the minimal hardware requirements to run the workflow_by_code example.
- Memory: 16G
- Free Disk: 5G
NOTE
The results will slightly vary on different OSs(the variance of annualized return will be less than 2%).
The evaluation results in the README.md page are from Linux OS.
Recorder artifact trust
Recorder loading is restricted by default. Prediction and label data normally need
no opt-in, but resuming a workflow can require executable model, dataset or task
objects. Only enable trusted=True after verifying both the writer and
the artifact store, including who can replace files in a shared experiment.
Unrestricted pickle loading can execute code. Do not automatically retry a refused
load with trust enabled.
See the artifact loading migration guide
for supported NumPy/pandas data, version limitations, custom recorder/loader changes
and low-level load_object(..., trusted=True) usage. Some notebooks and direct-load
examples already opt in explicitly; their model/task loads assume your own trusted
runs, not arbitrary downloaded or shared artifacts.
Online workflows
The three online_srv CLIs expose --trusted, defaulting to False.
For your own artifacts in an access-controlled store, run from examples/:
python online_srv/update_online_pred.py --trusted=True main
# Subsequent daily updates use the same explicit consent:
python online_srv/update_online_pred.py --trusted=True update_online_pred
The simulation and rolling-management examples also accept the flag. Their
main/first_run methods reset experiment data (and rolling task pools); use
dedicated experiment stores and save anything important before running them.
With Qlib data and, for rolling management, your MongoDB service configured:
python online_srv/online_management_simulate.py --trusted=True --exp_name=my_own_rolling_exp main
python online_srv/rolling_online_management.py --trusted=True --task_url=mongodb://localhost:27017/ first_run
python online_srv/rolling_online_management.py --trusted=True --task_url=mongodb://localhost:27017/ add_strategy
python online_srv/rolling_online_management.py --task_url=mongodb://localhost:27017/ routine
Consent is passed to each strategy, including newly added strategies, and through
its online tool/updater to executable artifact loads. Prediction, label and
numerical-report reads stay restricted. OnlineManager has no trust constructor
or global permission.
RollingOnlineExample creates a DelayTrainerRM with the selected consent only
when no trainer is supplied. In Python, configure a caller-supplied delayed trainer
yourself; the example does not change its policy:
from qlib.model.trainer import DelayTrainerR
from online_srv.rolling_online_management import RollingOnlineExample
example = RollingOnlineExample(
trainer=DelayTrainerR(trusted=True),
trusted=True,
)
Ordinary TrainerR/TrainerRM constructors do not accept trusted.
The simulation example leaves those trainers unchanged and configures its strategy.
Rolling management saves a local .RollingOnlineExample pickle. Only restore a
file you independently trust. Restoring it retains its saved strategy, tool and trainer
settings; legacy components without a saved flag default to restricted loading.
An example constructor/CLI flag does not override a subsequently loaded manager.
After reviewing artifact sources, explicitly reconfigure or recreate each
strategy, its strategy.tool, and any delayed trainer; changing a strategy's flag
alone does not update its existing tool. add_strategy uses the current CLI flag
for new strategies only. Remember that this example's first_run is destructive.
Ordinary trusted artifacts do not require deleting experiments or full retraining
to migrate; select consent on the actual components that reload them.
Other migrations
- DDG-DA:
the workflow's opt-in covers necessary recorder and handler/internal-data cache
reads. Verify local
working_dir, configuration directory andh_pathcontents as well as MLflow storage. - HIST: update the
stock-index mapping path to JSON, including on restored models; legacy
object-pickled
.npymappings are not accepted.
These settings cover scoped artifact loads, not every Qlib deserialization API. Other local model files, handler caches, YAML configurations and task stores retain their own trust requirements.