Since the database fields often store HTML fragments or Markdown fragments, a custom file name suffix field was added in the database datasource. Also, when parsing HTML, it should be compatible with parsing HTML fragments (without a body tag), not just full HTML documents (with a body tag).
Add Sitemap data source — ingests web pages from sitemap.xml with incremental sync. Supports HTML→Markdown extraction, PDF discovery, URL filtering, SSRF protection.
Add MonkeyOCRv2-Parsing as a document layout parser. Adapter sends docs to HTTP API, extracts ZIP response, converts JSON layout (text/tables/bbox) to RAGFlow sections. Configurable via MONKEYOCRV2_SERVER_URL.
Add user attribution to embedding calls — forwards user on OpenAI-compatible providers via current_llm_user(). Excludes LocalAI/Xinference (reject unknown fields). Accepts user_id on ingest/parse/chunk endpoints; redacts from logs. Omitted = unchanged. Go/vision/TTS/ASR out of scope.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Fix MCP session cleanup — now properly awaits server task and closes transport contexts/executors before stopping event loop. Wakes queued callers on shutdown.
Fix MCP timeouts — now uses a single deadline across sync/async paths. Expired queued requests are skipped before dispatch; in-flight coroutines are canceled. Tests added. Remote side effects cannot be rolled back (server idempotency required).
Add custom CA certificate support to WebDAV data source. Users can now connect to private/self-signed WebDAV servers via the verify option.
Fixes#16635
### Summary
Add Xquik as a new data source for X (Twitter) post ingestion. Supports Latest/Top queries, cursor pagination, incremental sync, and maps engagement/media to RAGFlow documents. Stops duplicate requests on repeated cursors. Tests and docs included.
---------
Co-authored-by: kriptoburak <kriptoburak@users.noreply.github.com>
Adds Azure DevOps as a new data source, syncing repository files and pull requests. Supports both hosted and self-hosted. Excludes wikis, work items, and binaries. Includes checkpoint/resume, change detection via commit IDs, and handles Azure's quirky 203 auth response. Tests passed against real deployment.
Fix build_connector_for_source() to honor config["bucket_type"] when constructing blob-storage connectors. Previously, an S3 source with bucket_type: s3_compatible was always built as Amazon S3, causing wrong credential validation. Now uses the configured bucket type and falls back to source type when missing.
Fix WebDAV data-loss bug where transient listing errors (network/timeout) returned [] and triggered deletion of all indexed documents. Removed top-level except so errors propagate to prune collector (which aborts on error). Individual file errors still handled. 3 regression tests added. Ruff clean.
## Summary
- Broaden `update_metadata_to` to keep `bool` / `int` / `float` / `None`
and structured values (e.g. PDF `outline` list[dict]) while retaining
string / list[str] merge+dedupe for LLM metadata.
- Fixes document system fields (`_isCurrent`, `_version`,
`_processStatus`) being dropped when parse persists PDF outline via full
`meta_fields` replace.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- route REST retrieval metadata conditions through the existing
metadata-index push-down service
- keep full metadata loading lazy and preserve the Python in-memory
filter as an exact fallback
- add focused coverage for successful and empty push-down results and
multi-valued fallback semantics
## Summary
Fix the silent-failure health-check retry loops in
`common/doc_store/{es_conn_pool, ob_conn_pool, infinity_conn_pool,
infinity_conn_base}.py`. These pool initializers run at module import
via `@singleton`; a brief blip during pod start currently crashes the
whole worker process. The fix retries with exponential backoff, logs
each attempt, and re-raises on exhaustion.
## Related
- #17065 / #17066 — S3 storage
- #17069 / #17070 — non-S3 storage backends
- #17123 / #17125 — `rag/utils/{es_conn, opensearch_conn}`
- #17134 / #17135 — `memory/utils/es_conn`
- Fixes#17195
---------
Co-authored-by: Harsh Kashyap <Harsh23Kashyap@users.noreply.github.com>
## Summary
Fixes#18025. Both merge paths deduplicated IDs by scanning a plain list
(`item not in list`) inside a loop while appending — O(n²) per merge.
Replaced with a set-backed `seen` check alongside the existing ordered
list: same order, same dedup result, O(n).
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## Summary
- allow ISO 8601 normalization callers to provide an explicit fallback
while preserving the existing default behavior
- use the extraction conversation time when `valid_at` is missing or
invalid
- clear an invalid optional `invalid_at` instead of writing an
unparseable value
- include the rejected timestamp value in the error log
This addresses the timestamp write-through portion of #18415. It is
intentionally separate from #18462, which fixes the semantic output
prompt.
Co-authored-by: Loong <184861530+yzl0ng@users.noreply.github.com>
### Summary
GaussDB DocEngine could return no chunks for conversational queries even
when relevant content was available. `Dealer.search()` supplies
`minimum_should_match` (30%, then 10% on retry), but the GaussDB adapter
discarded it and built a single `plainto_tsquery` from every token. This
effectively required all conversational filler terms to match.
Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
### Summary
`format_document_soup` tracks "am I inside a table" and "am I inside a
link" with sticky flags that are meant to be reset by `elif e.name ==
"/table"` and `elif e.name == "/a"`. BeautifulSoup's `.descendants` only
yields opening tags — a `Tag` named `/table` or `/a` never exists — so
both branches are dead code and neither flag is ever cleared.
Everything after the first `<table>` on a page is therefore formatted as
if it were still table content: paragraphs lose their newline, list
items lose their `- ` marker, headings lose their break, and the text is
glued onto the last table cell. Under
`HTML_BASED_CONNECTOR_TRANSFORM_LINKS_STRATEGY=markdown` the same bug
leaks a link's `href` into everything that follows it, including whole
subsequent paragraphs. The Confluence connector
(`confluence_connector.py:948`) goes through this path.
Real output for a Confluence-shaped page (heading, intro, spec table,
then the body) via the public `parse_html_page_basic`:
**Before**
```
prod us-east-1 Rollback procedure If the canary fails, run the rollback script immediately. Drain the load balancer Revert the deployment Escalate to the on-call rota if the rollback stalls. Do not skip the post-mortem.
```
**After**
```
prod us-east-1
Rollback procedure
If the canary fails, run the rollback script immediately.
- Drain the load balancer
- Revert the deployment
Escalate to [the on-call rota](http://oncall.example.com) if the rollback stalls.
Do not skip the post-mortem.
```
Every heading, paragraph and list marker after the table is lost, and
the whole body is indexed as one run-on line hanging off a table cell.
### Fix
Derive both scopes from each element's **ancestors** instead of from
flags that nothing can clear, and drop the two dead branches plus the
two that become redundant.
The scopes are resolved in one up-front pass into `id`-keyed maps
(`table_scope`, `href_scope`) and looked up in O(1) per element. Probing
per element with `find_parent` instead is O(depth) each, which measured
12–13× slower on table-heavy pages and up to 103× on deeply nested
markup; the map version costs a depth-independent 1.13–1.35× over
`main`. Numbers and method are in the round-2 comment below.
This also changes one adjacent behaviour worth calling out explicitly: a
link **inside** a table cell now renders as markdown, where before it
rendered as plain text. That previous behaviour was not by design — it
only held when no link preceded the table. With a link before the table,
`main` stamps the stale href onto every cell:
```
main: '[pre](http://STALE.com)\n\t[cellA](http://STALE.com)\t[cellB](http://STALE.com)'
branch: '[pre](http://STALE.com)\n\tcellA\tcellB'
```
Those cells are not links. Both symptoms are the same sticky-state bug,
so they are fixed together rather than left half-done.
### Testing
`test/unit_test/data_source/test_html_utils.py` is new —
`format_document_soup` had no test coverage. 11 tests: 8 fail on `main`
and pass on this branch, 3 are controls that pass on both (the table
itself still separates rows and cells, anchor text is still linkified,
the default `strip` strategy still strips).
Representative failures on `main`:
```
assert '\nAfter' in 'Before\n\tA\tB After'
assert '\n- item1' in 'Before\n\tA\tB item1 item2'
assert 'see [link](http://x.com) [ after](http://x.com)' == 'see [link](http://x.com) after'
assert '[next paragraph]' not in '[link](http://x.com)\n[next paragraph](http://x.com)'
```
Reverting each clause of the fix independently keeps the anchors honest:
reverting only the table clause fails exactly the 4 table tests and
leaves the link tests green; reverting only the link clause fails
exactly the 3 link tests and leaves the table tests green.
(`test_link_inside_a_table_cell_is_linkified` needs both clauses broken
to fail, so it appears in neither single-clause revert — it is covered
by the 8-fail run against `main`.)
Full `test/unit_test/data_source/` suite: **3 failed, 199 passed**, and
the failure set is byte-identical to clean `main` (**3 failed, 188
passed**) — the 3 are `TestSSRFValidation::*`, which resolve
`api.example.com` against real DNS and are unrelated to this change.
`ruff check` and `ruff format --check` are clean on both touched files.
---
This PR was drafted with AI assistance (Claude). I reviewed the change,
independently reproduced both symptoms against `main`, and take
responsibility for it.
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
`common/misc_utils.once` set `executed=True` before invoking the wrapped
function, so an exception on the first call permanently disabled future
calls and returned the cached `None`.
This change marks `executed=True` only after a successful call, allowing
retries after transient failures while preserving once-only behavior
after success. It also adds regression tests for retry-after-exception
and thread-safe single execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
The **aimlapi.com** provider added in #17311 does not identify itself on
any of its outgoing requests, so its traffic cannot be attributed to the
integration. This PR adds the two headers AIMLAPI expects —
`X-AIMLAPI-Source` and `X-AIMLAPI-Partner-ID` — to every request the
provider makes.
### What problem does this PR solve?
Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.
`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.
Closes#17056Closes#5782Closes#7075
**What it does**
1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.
**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.
**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix Infinity compatibility issues in knowledge compilation.
This change:
- Stores compilation source ID lists as JSON arrays in Infinity.
- Parses JSON array fields when reading compiled documents.
- Uses `json_contains` for filtering JSON array fields.
- Adds the missing `name` column to the Infinity mapping.
- Updates dataset navigation KNN search to use the unified
`MatchDenseExpr` interface.
- Handles unavailable embeddings without querying an invalid `q_0_vec`
field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Harsh Kashyap
Anupam Mediratta
Ren Wei (任卫)
Gonghan-PrincessandGonghan-Princess
Aroh Maurya
Lynn
Wang Qi
Ryker_Feng
VincentLambert
Ziyang Zhang
ÖnderyandCursor
Wenchao An
Lem0nTea2002
MarMar Labs
SYED ALI ABBAS RAHIL
OSHA-BandOti_B
Tai An
buua436
Harsh Kashyap
taek105
Burak Bayırandkriptoburak
Taranum Wasu
Erhan Aydın
Jiaquan Yi
陈志谦andjindou
天海蒼灆andCursor
Elias Shieh
I Kartik ReddyandClaude Opus 5
LoongandLoong
Sevenzuo
zws202606
S
deadtricksterandClaude Opus 4.8
Kevin HuandYingfeng Zhang
Eugene
Xavierando