Commit Graph
26 Commits
Author SHA1 Message Date
XianBWandCopilot 7c95268c0c fix(security)!: harden artifact serialization and loading (#2339)
* security: harden artifact serialization and loading

* ci: restore dependency and title lint compatibility

* fix: address artifact test and lint failures

* ci: handle current MLflow and Windows OSQP changes

* ci: use lxml release with cross-platform wheels

* fix: defer optional CVXPY solver imports

* docs: mark notebook model artifacts as trusted

* docs: mark position artifacts as trusted

* style: document intentional lazy import

* ci: skip broken released OSQP on Windows Python 3.8

* fix: mark online model artifacts as trusted

* test: make MLflow client timing check robust

* ci: retry nondeterministic benchmark smoke test

* fix: cap filelock before fork safety regression

* fix: use filelock release compatible with multiprocessing

* fix: preserve artifact loading compatibility after hardening

* fix(security): validate derived artifact paths and extend regressions

* fix(security): preserve extended data artifact round trips

* fix(security): retain legacy NumPy and pandas artifact classes

Allow the exact MaskedArray and SparseDtype module paths emitted by the older dependency versions in CI. Preserve restricted loading and cover both old and new paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(data): avoid unused trading calendar lookup in PIT normalization

PIT normalization uses report dates and calendar-day offsets, so do not request an unused Baostock calendar. Cover annual and quarterly normalization without online calendar access.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(security): require explicit trust in artifact workflows

Expose default-off artifact trust at online, delayed-training and meta-workflow entry points. Keep data loading restricted, preserve legacy default recorder calls and restored workflow defaults, and document migration and trust boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(workflow): cover complete DDG-DA and HIST artifact flows

Exercise real training, cache and recorder restoration, saved-task replay, signals and portfolio backtests. Fix the cache trust, grouping, relative-path, LightGBM early-stopping and HIST CPU blockers exposed by full execution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(security)!: unify artifact trust and publish upgrade guidance

Use trusted consistently across public APIs and CLI options. Migrate pre-release saved consent only at serialization boundaries, reject invalid or conflicting settings, and preserve restricted data reads. Add a canonical upgrade guide with README, installation, changelog and runtime-error entry points.

BREAKING CHANGE: executable artifact restoration requires explicit trusted=True. HIST mappings use JSON and high-frequency cache paths must remain within artifact_root. See docs/start/artifact_migration.rst for upgrading existing workflows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(security): remove trial-only trust compatibility

Target upgrades from pre-PR Qlib rather than artifacts generated by intermediate PR revisions. Retain restricted defaults for saved objects without consent, unify current APIs, and remove state/keyword migration branches and documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci: avoid failed dataset probes and space download retries

Download the same existing v2 archives used by successful jobs without probing nonexistent package-version assets. Allow five download attempts 60 seconds apart; preserve the full matrix, datasets, noninteractive setup and failure propagation. Cover the exact archives and retry policy in offline CI tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add practical artifact migration examples

Show explicit consent for restored online components while preserving the original manager, schema-aware Arrow conversions, timezone-preserving ZoneInfo migration, and restricted round trips into a separate recorder. Explain shared-component scope, staging artifacts and trusted-source prerequisites.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(workflow): preserve fresh artifact workflows with scoped trust

Keep immutable cache references intact at open rolling tails. Add explicit per-load record-template consent and per-artifact collector loading options without widening other reads. Restore the original template class after parent-load failures, and support audited business-frequency and NumPy record representations through exact pickle globals.

Document the scoped APIs and cover fresh trained-model reads, cache generation, numerical round trips and refusal paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-10-08 17:49:08 +08:00
Lee YuntongandLeeYuntong 3a348aec9f Fix typo (#1811)
Co-authored-by: LeeYuntong <nukuihayu@outlook.com>
2024-06-20 18:12:07 +08:00
Cadenza-Li 76f2fb1a1a Add ipynb format check (#1439)
* Update test_qlib_from_source.yml

* add ipynb format check to workflow

* test ipynb CI

* modify nbqa check path

* add pylint flake8 mypy check to ipynb

* check ipynb with black and pylint

* reformat .ipynb files

* format line length

nbqa black . -l 120

* update nbqa .ipynb format CI

* format old ipynb files

* add nbconvert check to CI

* adjust CI order to avoid repeating download data
2023-02-21 09:23:22 +08:00
you-n-g c34051c1ce Be compatible with Google Colab (#1188)
* Update workflow_by_code.ipynb

* Update workflow_by_code.ipynb

* Update workflow_by_code.ipynb

* Update workflow_by_code.ipynb

* Update workflow_by_code.ipynb
2022-07-08 14:23:25 +08:00
you-n-g 2952c443ca Add Qlib notebook tutorial (#1037)
* Add Qlib notebook tutorial

* Update tutorial
2022-04-08 21:29:41 +08:00
Chia-hung Tai 184ce34a34 [807] Move the REG_CONSTANT/EPS to constant.py. (#811)
* [807] Move the REG_CONSTANT to constant.py.

* import REG_US.

* Move EPS to constant.py.
2022-01-09 21:39:46 +08:00
Pengrong Zhu c10c349b20 remove unneeded code from workflow_by_code.ipynb && fix analysis_model_performance (#740) 2021-12-11 13:23:00 +08:00
Chao Wang 0b3b95f22f Update workflow_by_code.ipynb (#697)
changed model_strategy to signal_stragey. there is no model_strategy under qlib.contrib.strategy.
2021-11-18 09:21:45 +08:00
+9 3760a18a8d Merge nested main (#597)
* MVP for Indian Stocks in qlib using yahooquery

* cleaned with black

* cleaned with black

* add YahooNormalizeIN and YahooNormalizeIN1d

* cleaned the code

* added 1min for IN and also updated readme

* update comments

* fix comments

* recorder support upload both raw file and directory

* fix comments

* Update README.md

* Fix docs of QlibRecorder

* sort index after loader (#538)

make sure the fetch method is based on a index-sorted pd.DataFrame

* refactor online serving rolling api

* refactor TRA

* format by black

* fix horizon

* fix TRA when use single head

* clean up

* improve pretrain

* update README

* fix tra when logdir is None

* fix tra when logdir is None

* Update strategy.py

* Update README.md

* Update README.md

* Conda Suggestion

* code standard docs

* Update ensemble.py (#560)

* Fix CI  Bug (#575)


Co-authored-by: yuxwang <anduinnn@foxmail.com>

* Update gen.py (#576)

* Fix multi-process loop calls (#574)

* check lexsort in the 'lazy_sort_index' function (#566)

* check lexsort

* check lexsort

* lexsort comment

* lexsort comment

* Delete .DS_Store

* Update README.md

* bug fix & use oracle transport pretrain

* mend

* Add `backend_freq_config` parameter, support multi-freq uri

* Add sample_config to QlibDataLoader, support multi-freq

* add multi-freq example

* get_cls_kwargs renamed get_callable_kwargs

* support multi-freq uri

* Add inst_processors to D.features

* Fix typo

* Fix the index type of the multi-freq example

* Fix duplicate mlflow directories in tests

* Add DataPathManager to QlibConfig && modify inst_processors to supports list only

* Modify the default value in the multi_freq example

* Modify client-server mode and dataset-cache to disable inst_processor

* Add wheel package to github CI

* fix comment

* Update FAQ.rst

* Update README.md

Fix wrong link

* Update the docs of TaskManager (#586)

* Update manage.py

* update yaml

* update run_all_model

* Modify the Feature to be case sensitive (#589)

* update README

* remove verbose

* fix spell bug

* fix typos (#592)

* Update Release Note

* fix portfolio bug

* Add calendar support for resample

* add freq kwargs

* test.yml: Remove redundant code (#595)

* Supporting shared processor (#596)

* Supporting shared processor

* fix readonly reverse bug

* remove pytests dependency

* with fit bug

* fix parameter error

* fix comments

* Fix undefined names in Python code (#599)

* Update pytorch_tabnet.py

$ `flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics`
```
./qlib/qlib/contrib/model/pytorch_tabnet.py:567:38: F821 undefined name 'inp'
            self.independ.append(GLU(inp, out_dim, vbs=vbs))
                                     ^
./qlib/examples/model_rolling/task_manager_rolling.py:75:18: F821 undefined name 'task_train'
        run_task(task_train, self.task_pool, experiment_name=self.experiment_name)
                 ^
2     F821 undefined name 'task_train'
2
```

* Fix undefined names in Python code

* from qlib.model.trainer import task_train

* update seed

* fix some docstring

* add comments

* Fix SimpleDatasetCache

* Update setup.py

updated classifiers

* Update setup.py

change to matplotlib==3.3

* Update python-publish.yml

added python 3.9

* updategrade version number

* Update model list

* fix the type of filter_pipe

* fix comment

* fix record_temp

* update cvxpy version

* Update code_standard.rst (#587)

* Update code_standard.rst

* Update docs/developer/code_standard.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* Add file lock for MLflowExpManager (#619)

* fix torch version

* Share version number (#620)

* Update initialization.rst (#622)

* Update initialization.rst

* Update docs/start/initialization.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* Update docs/start/initialization.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* fix bugs for running previous exmaple

* fix deal amount bug

* update change doc (#623)

* Add files via upload

* Update README.md

* Update README.md

* Update README.md

* Delete change doc.gif

* Add files via upload

* Update README.md

* Delete change doc.gif

* Add files via upload

* Delete change doc.gif

* Add files via upload

* Update README.md

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* update doc

* simplify run all model

* fix run all model bug

* Fix Models (#483)

* fix gat dataset

* fix tft model

* Update tft.py

* Fix tft.py

Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com>

* type and skip empty exp

* fix model yaml config

* fix tft import bug

* skip empty result

* fix model and yaml bug

* fix wrong generate parameter

* Modify multi-freq example (#626)

* modify the example of multi-freq

* add Copyright

* add a comment to average_ops.py

* modify the example of multi-freq

* add comment to multi_freq_handler.py

* add the Ref expression description to multi_freq_handler.py

* add expression description to multi_freq_handler.py

* update images

* fix workflow and update framework

Co-authored-by: Gaurav <2796gaurav@gmail.com>
Co-authored-by: 2796gaurav <17353992+2796gaurav@users.noreply.github.com>
Co-authored-by: bxdd <bxd98@126.com>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Dong Zhou <Zhou.Dong@microsoft.com>
Co-authored-by: ZhangTP1996 <ztp18@mails.tsinghua.edu.cn>
Co-authored-by: demon143 <59681577+demon143@users.noreply.github.com>
Co-authored-by: Wangwuyi123 <51237097+Wangwuyi123@users.noreply.github.com>
Co-authored-by: yuxwang <anduinnn@foxmail.com>
Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com>
Co-authored-by: Mark Zhao <50850474+markzhao98@users.noreply.github.com>
Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com>
Co-authored-by: Dong Zhou <evanzd@users.noreply.github.com>
Co-authored-by: SaintMalik <37118134+saintmalik@users.noreply.github.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Anurag Kumar <mailanu98@gmail.com>
Co-authored-by: demon143 <785696300@qq.com>
2021-10-01 02:15:30 +08:00
you-n-g 1fb50d521b Merge branch 'main' into nested_decision_exe 2021-07-07 17:30:31 +08:00
Guodong Xu d66733c358 examples/workflow_by_code.ipynd: fix an error in R.get_recorder() parameters
get_recorder() needs specify 'recorder_id='. However workflow_by_code.ipynd
didn't. This patch fixes it.

Without this fix, here is the error message jupyter-notebook reports:

"---------------------------------------------------------------------------
TypeError Traceback (most recent call last)

<ipython-input-7-e6a7b5f4da00> in <module>
26 # backtest and analysis
27 with R.start(experiment_name="backtest_analysis"):
---> 28 recorder = R.get_recorder(rid, experiment_name="train_model")
29 model = recorder.load_object("trained_model")
30

TypeError: get_recorder() takes 1 positional argument but 2 positional arguments (and 1 keyword-only argument) were given"

Signed-off-by: Guodong Xu <guodong.xu@linaro.org>
2021-06-26 18:25:47 +08:00
bxdd f78e90171b fix comments & add VAStrategy & add trade indicator 2021-06-14 21:32:18 +08:00
bxdd 4d48c96d30 fix CI 2021-06-01 18:50:50 +08:00
bxdd 4085b447aa move backtest to core, fix calendar bugs, add some docstring 2021-05-27 21:14:39 +08:00
Jactus 4861552d28 Update notebook 2021-03-25 17:13:52 +08:00
zhupr 627fc628e0 Fix get_data.py detection 2020-11-28 12:57:00 +08:00
Dong Zhou cde5c59b7e fix colab link 2020-11-28 08:38:42 +08:00
Dong Zhou dc8bf6f077 add colab button 2020-11-28 08:26:18 +08:00
zhupr c5a3b74a96 Fix the target repository of get_data.py in google.colab 2020-11-27 22:33:22 +08:00
zhupr be2173d839 Fix 2020-11-27 22:08:35 +08:00
zhupr f8101214eb Fix report, support google colab 2020-11-27 20:00:05 +08:00
Young e2935f214a fix some typo and bug 2020-11-26 08:42:54 +00:00
Jactus 27b573c7d6 Update run_all_model script 2020-11-23 15:10:14 +08:00
Jactus aed566fa00 Fix get_data bug in workflows 2020-11-23 10:53:26 +08:00
Jactus 5aa48524d6 Add run_all_model script 2020-11-20 16:09:40 +08:00
Jactus c91698287a Add catboost config and notebook 2020-11-19 17:18:18 +08:00