## Summary
Rename the repository dependency-resource directory `ragflow_deps/` to
`deps/`, and update every hardcoded reference to that directory name so
the rename does not break resource loading at build/runtime.
This is not a purely cosmetic rename: the Go runtime hardcodes the
directory name as a resource search path (e.g.
`internal/common/model_assets.go` trimming the `deps/` prefix, and the
tokenizers joining `deps/` into candidate paths), and the Dockerfiles
copy assets into `/ragflow/deps`. Those literals must move together with
the directory, otherwise assets fail to resolve.
## Two intentional exceptions (kept unchanged)
The rename deliberately keeps two names that only *coincidentally*
contained the old string:
- The published image tag `infiniflow/ragflow_deps:latest` (used by
`Dockerfile`, `Dockerfile_base`, docs, and CI `image inspect` checks).
- The CI install path `/opt/ragflow_deps` inside the CI runner image
(`build.sh` `SYSTEM_DEPS_TOKENIZER`).
## Changes
- `git mv ragflow_deps deps`.
- `build.sh`: `CL100K_TABLE_RELPATH`, native asset zip path, and related
references updated to `deps/`.
- `tools/scripts/download_dev_deps.py`: `RAGFLOW_DEPS_DIR` points at
`deps/`.
- Go runtime resource search paths (`internal/common`,
`internal/tokenizer`) updated to the `deps/` layout.
- `Dockerfile` / `Dockerfile_base`: container asset path changed to
`/ragflow/deps`; kept tag `infiniflow/ragflow_deps:latest` unchanged.
- Docs / `AGENTS.md` / CI workflow paths updated to the new directory
name.
## Test plan
- Go unit tests in `internal/common` and `internal/tokenizer` updated to
the new `deps/` layout and pass via `bash build.sh --test`.
- Python download scripts import and build URLs correctly (`py_compile`
clean).
- Added `tools/scripts/test_dep_rename_invariants.py` as a regression
guard asserting the two intentional exceptions (`/opt/ragflow_deps` and
`infiniflow/ragflow_deps`) stay intact, so a future blanket rename
cannot silently regress them.
---
## Follow-up fix (commit 2f6857095d)
The original rename used a blanket `sed s/ragflow_deps/deps/g` and only
selectively reverted `build.sh`. That left two intentionally-kept names
rewritten, which this follow-up restores:
- CI install path `/opt/ragflow_deps` — `sep-tests.yml` had been changed
to read `/opt/deps`, breaking the offline cl100k table fallback (the
cached-image path pointed at a directory that never receives the file).
Restored to `/opt/ragflow_deps`.
- Published image tag `infiniflow/ragflow_deps` — left as bare
`infiniflow/deps` in `sep-tests.yml` comments/echo, `.dockerignore`, and
`Dockerfile` / `Dockerfile_base` error messages. Restored to
`infiniflow/ragflow_deps`.
The regression guard `tools/scripts/test_dep_rename_invariants.py` was
added to prevent recurrence.
## Background
This branch started as a focused fix to agentic RAG regexp retrieval
semantics (`f80556585`) and grew into the full agentic RAG path. The
title no longer describes the contents, so it has been rewritten.
The PR now covers three largely independent lines of work:
### 1. The agentic RAG is reachable from the UI
`internal/agentic_rag` (the eino-ADK ReAct explorer) was already built
and wired, but only reachable by hand-crafting an `agent_mode` kwarg. It
is now the sixth option in the chat mode selector (`reasoning` level 5).
One subtlety worth stating plainly: **levels 1-4 and level 5 are not the
same agent.** Levels 1-4 go through `internal/rag/agentic-rag` (the
harness graph) with a depth chosen by `harnessModeForLevel`; level 5
switches engines outright to `internal/agentic_rag`. That is why level 5
must never reach `harnessModeForLevel` — its `level >= 4` case would
silently answer "ultra" for a level outside its domain.
### 2. Per-dialog failover chain
`agenticModelChain` resolved exactly one model and the caller then used
`chain[0]`, so a "chain" was never more than a single element. A dialog
can now configure an ordered list of fallback models in Chat Settings,
handed to `NewFailoverEinoChatModel` (sticky cursor plus a 30s
full-chain cooldown).
The list lives in the dialog's own `llm_setting.failover_llm_ids`, so no
new table is involved. A member that no longer resolves is skipped with
a warning rather than failing the turn.
Also removed: `tenant_model_group` / `tenant_model_group_mapping`, which
nothing ever read (the DAOs were constructed but never called, and no
frontend or Python code referenced the concept). Their removal takes an
explicit drop migration with it, plus the account-deletion cascade that
queried them.
### 3. A hung MiniMax stream (independent of the agentic work)
With any mode selected, a chat rendered its whole answer and then sat on
"thinking" forever. Root cause is `minimax.go:256`: MiniMax sends `data:
[DONE]` but leaves the HTTP connection open, and the code waited for the
scanner goroutine's EOF *after* `HandleStreamingResponse` had already
returned. That receive can only end when `streamCallTimeout` (20
minutes) expires.
Diagnosed by capturing a real SSE stream (the complete answer arrives,
the terminal `final: true` never does) and a goroutine dump (6 requests
parked in `chan receive`).
## Two review findings fixed on the way through
- **KB-scope authorization**: the agentic branch bypassed quote
resolution, and an empty KB scope made `buildBoolQueryFromCondition`
drop the `kb_id` filter — so a citation could resolve a chunk belonging
to a different KB in the same tenant. The agentic branch now requires a
non-empty scope and otherwise falls through to the regular path.
- **Stale documentation**: `agentic-rag-failover-groups.md` described
the "automatically include every tenant model" strategy that upstream
had already removed. It was rewritten for the per-dialog scope and then
dropped entirely, since the design now lives in the code it describes.
## Verification
- `bash build.sh --test`: `admin`, `dao`, `service`, `service/dataset`
and `entity/models` all pass
- The MiniMax fix was verified end-to-end against a live server: before,
the turn hung indefinitely; after, it completes in **1.9s** with `final:
true` present
- Frontend: 9 tests added; type-check and lint clean on the touched
files
## Not included
- **Attachment support in agentic mode.** Text attachments could be
appended safely, but images have no safe fix: the agent's toolset is
built around corpus retrieval and has no image input channel. Fixing
only the text path would leave the feature half-supported and harder to
diagnose than now. Planned as a follow-up PR, with the design synced
here first.
- Tool-calling is not enforced as a group constraint. `is_tools` is a
provider-declared flag rather than a measured capability (187 of 659
chat models do not declare it), so gating on it would reject working
configurations while admitting broken ones.
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Refactor the Go `KnowledgeCompilerComponent` so its parameter is a
**single string template id** instead of a DSL-level `variant` (or
plural group id list). The `variant` is no longer in the DSL — it is now
**derived at runtime from the resolved compilation template's `kind`
field**.
This aligns the Go ingestion port with the frontend Compiler operator,
which emits a singular `compilation_template_group_id` and does not
write `variant` into the generated `compiler.json`.
## Summary
- Add language-aware Snowball stemmer to `RagTokenizer` supporting 16
languages (Dutch, German, French, Spanish, etc.)
- Thread the KB `language` parameter through the full tokenization
pipeline (14 parser modules + task executor)
- Add Dutch to the frontend language lists and cross-language form
## Problem
RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all**
BM25 tokenization, regardless of the knowledge base language setting.
This produces incorrect stems for non-English text. For example:
| Dutch word | Dutch stemmer | English Porter |
|---|---|---|
| documenten | document | documenten (unchanged!) |
| gebruikers | gebruiker | gebruik (over-stemmed) |
| instellingen | instell | instellingen (unchanged!) |
This degrades BM25 recall for any non-English knowledge base.
## Solution
NLTK already ships Snowball stemmers for 16 languages. This PR:
1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with
`set_language()` and `_normalize_token()` that selects the correct NLTK
Snowball stemmer. Falls back to Porter for unmapped languages (Chinese,
Japanese, Korean, etc. — these use character-based tokenization anyway).
2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** +
**`rag/svr/task_executor.py`**: Threads the `language` parameter through
`tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers.
3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`,
`LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`,
cross-language form field, and `en.ts` locale.
## Backward Compatibility
- Default language is `"English"`, preserving existing behavior for all
current users
- Languages without a Snowball stemmer mapping fall back to Porter (no
change)
- No new dependencies — NLTK Snowball is already bundled
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.
The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.
All 21 Python agent components and ~35 tools are ported with functional
parity.
Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.
234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).
## Type of change
- [x] Refactoring
- [x] New feature
- [x] Bug fix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Fixes **Tongyi-Qianwen** (`QWenEmbed`) text embeddings when the
configured `base_url` points at DashScope **international**
(`dashscope-intl.aliyuncs.com`) or **China** (`dashscope.aliyuncs.com`)
hosts, including values copied from Model Studio that use the
**OpenAI-compatible** path (`.../compatible-mode/v1`).
- The `dashscope` Python SDK (`TextEmbedding.call`) expects the
**native** HTTP root (`https://<host>/api/v1`), not the
OpenAI-compatible base URL. Without mapping, international accounts
could hit the wrong host or path.
## Implementation
- Added `_dashscope_native_http_api_url()` to normalize known DashScope
hosts to `.../api/v1`, and wired `QWenEmbed` to set
`dashscope.base_http_api_url` before each embedding call (document and
query).
## Notes
- In-code comments document the Tongyi-Qianwen / DashScope intl vs CN
behavior for future maintainers.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
Fix: Clicking the button in the bottom-right corner of the
`/chats/widget` page fails to display the dialog box.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
bin directory cannot be copied to docker image introduced by
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
feat: Added LLM factory initialization functionality and knowledge base
related API interfaces
refactor(dao): Refactored the TenantLLMDAO query method
feat(handler): Implemented knowledge base related API endpoints
feat(service): Added LLM API key setting functionality
feat(model): Extended the knowledge base model definition
feat(config): Added default user LLM configuration
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
# RAGFlow Go Implementation Plan 🚀
This repository tracks the progress of porting RAGFlow to Go. We'll
implement core features and provide performance comparisons between
Python and Go versions.
## Implementation Checklist
- [x] User Management APIs
- [x] Dataset Management Operations
- [x] Retrieval Test
- [x] Chat Management Operations
- [x] Infinity Go SDK
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
This PR adds comprehensive **Right-to-Left (RTL) language support**,
primarily targeting Arabic and other RTL scripts (Hebrew, Persian, Urdu,
etc.).
Previously, RTL content had multiple rendering issues:
- Incorrect sentence splitting for Arabic punctuation in citation logic
- Misaligned text in chat messages and markdown components
- Improper positioning of blockquotes and “think” sections
- Incorrect table alignment
- Citation placement ambiguity in RTL prompts
- UI layout inconsistencies when mixing LTR and RTL text
This PR introduces backend and frontend improvements to properly detect,
render, and style RTL content while preserving existing LTR behavior.
#### Backend
- Updated sentence boundary regex in `rag/nlp/search.py` to include
Arabic punctuation:
- `،` (comma)
- `؛` (semicolon)
- `؟` (question mark)
- `۔` (Arabic full stop)
- Ensures citation insertion works correctly in RTL sentences.
- Updated citation prompt instructions to clarify citation placement
rules for RTL languages.
#### Frontend
- Introduced a new utility: `text-direction.ts`
- Detects text direction based on Unicode ranges.
- Supports Arabic, Hebrew, Syriac, Thaana, and related scripts.
- Provides `getDirAttribute()` for automatic `dir` assignment.
- Applied dynamic `dir` attributes across:
- Markdown rendering
- Chat messages
- Search results
- Tables
- Hover cards and reference popovers
- Added proper RTL styling in LESS:
- Text alignment adjustments
- Blockquote border flipping
- Section indentation correction
- Table direction switching
- Use of `<bdi>` for figure labels to prevent bidirectional conflicts
#### DevOps / Environment
- Added Windows backend launch script with retry handling.
- Updated dependency metadata.
- Adjusted development-only React debugging behavior.
---
### Type of change
- [x] Bug Fix (non-breaking change which fixes RTL rendering and
citation issues)
- [x] New Feature (non-breaking change which adds RTL detection and
dynamic direction handling)
---------
Co-authored-by: 6ba3i <isbaaoui09@gmail.com>
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
Co-authored-by: Ahmad Intisar <168020872+ahmadintisar@users.noreply.github.com>
Co-authored-by: Liu An <asiro@qq.com>
### What problem does this PR solve?
Add seekdb as doc_engine wich is the lite version of oceanbase.
close#12691
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
```
$ python admin/client/ragflow_cli.py -t user -u aaa@aaa.com -p 9380
ragflow> list datasets;
ragflow> list default models;
ragflow> show version;
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
After I ran this command,
```bash
uv run ./download_deps.py
```
a file was not ignored.
```bash
❯ git status
On branch feat/ignore-uv
Untracked files:
(use "git add <file>..." to include in what will be committed)
uv-x86_64-unknown-linux-gnu.tar.gz
nothing added to commit but untracked files present (use "git add" to track)
```
Add this file name to `.gitignore`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Manage and display memory datasets.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This PR adds a data backup and migration solution for RAGFlow Docker
Compose deployments. Currently, users lack a standardized way to backup
and restore RAGFlow data volumes (MySQL, MinIO, Redis, Elasticsearch),
which is essential for data safety and environment migration.
**Solution:**
- **Migration Script** (`docker/migration.sh`) - Automates
backup/restore operations for all RAGFlow data volumes
- **Documentation**
(`docs/guides/migration/migrate_from_docker_compose.md`) - Usage guide
and best practices
- **Safety Features** - Container conflict detection and user
confirmations to prevent data loss
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
Co-authored-by: treedy <treedy2022@icloud.com>
### What problem does this PR solve?
There are two main changes:
1. Update xgboost to 1.6.0 to build the project on MacOS with Apple
chips, this change refers to the issue:
https://github.com/infiniflow/ragflow/issues/5114.
2. When `use_china_mirrors` is set in `download_deps.py`, the names of
chrome files downloaded by the script will be different from the file
names used in Dockerfile, so I added the file name in `get_urls`
function to solve this problem.
I think it's better to add testing for Docker image
`infiniflow/ragflow_deps` to the test workflow, but since the workflow
is currently running on a self-hosted runner, I'm not sure how to modify
it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Add code_executor_manager. #4977.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Add Support for german language
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
when develop ragflow local there would be a hash file generate that is
kind of not good for develop
this patch add a regex to `.gitignore` for better developing
### Type of change
- [x] Refactoring
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
### What problem does this PR solve?
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Add create_dataset method, test for it, and update SDK->sdk.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: cecilia-uu <konghui1996@163.com>
### What problem does this PR solve?
Create python SDK to return the version of RAGFlow.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: cecilia-uu <konghui1996@163.com>
… incorrectly appear on the model settings page.
### What problem does this PR solve?
This is an issue with the Upload component on the assistant-setting
page. I use the show variable to explicitly control the button component
within it.
see:
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Other (please describe): Add the local models that DeepDoc depends
on to the gitignore file in dev mode.
Signed-off-by: liuchao <lcjia_you@126.com>
### What problem does this PR solve?
Optimize task broker and executor for reduce memory usage and deployment
complexity.
### Type of change
- [x] Performance Improvement
- [x] Refactoring
### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Ignore temporal files to help Mac developers.
### Type of change
- [x] Other (please describe):
Co-authored-by: PLIX870I <plix870i@V-SPDT-XIAOHUI-MB.local>
Zhichang Yu
Wang Qi
xugangqiang
Jin Hai
Paul Y Hui
balibabu
Rodger Blom
sham-srandCursor
Liu An
chanx
longbingljw
AAAkater
Lynn
TreeDyandtreedy
He Wang
Yongteng Lei
so95
Philipp Rien
yihong
cecilia-uu
lcjia_you
Fakai ZhaoandKevin Hu
Liu XiaohuiandPLIX870I