Commit Graph
56 Commits
Author SHA1 Message Date
天海蒼灆 a1208cd273 fix: surface MinerU API error bodies and re-raise parser failures (#17308)
Fix MinerU/PaddleOCR error handling — send enum values, include response body in failures, and surface real parse errors.
2026-09-14 11:37:17 +08:00
Sourav Rajvi ec0ff067c1 fix(parser): handle large sparse spreadsheets, JSON BOM, and XLSX QA chunking (#19237) 2026-09-13 15:46:34 +08:00
天海蒼灆 792a17cf20 fix(mineru): align backend names with current MinerU API (#19055) 2026-09-09 13:07:42 +08:00
MarMar Labs 1455cdfadc fix(excel): drop the "None" label for a blank spreadsheet header cell (#19175) 2026-09-08 10:41:51 +08:00
Ziyang Zhang fd9f9e7f61 feat(parser): add MonkeyOCRv2-Parsing support (#18887)
Add MonkeyOCRv2-Parsing as a document layout parser. Adapter sends docs to HTTP API, extracts ZIP response, converts JSON layout (text/tables/bbox) to RAGFlow sections. Configurable via MONKEYOCRV2_SERVER_URL.
2026-09-07 21:12:50 +08:00
Paul Yao 0c28d59ea1 fix(parser): handle MinerU chart blocks instead of silently dropping them (#19096) 2026-09-05 11:03:27 +08:00
Sbaaoui Idriss ebb2174236 fix(parser): decode non-UTF-8 document text (#19253) 2026-09-04 15:40:34 +08:00
helloxjade 6ed4970d6d fix(pdf): make _table_transformer_job honor TABLE_AUTO_ROTATE by default (#16079) 2026-09-03 14:51:22 +08:00
Xiaolong Ma 62028e9c78 Feature: support self-hosted PaddleOCR-VL deployments (#19011)
PaddleOCR could only reach the hosted AI Studio service, whose
asynchronous job API (submit -> poll -> fetch) is unreachable in an
air-gapped install. A self-hosted PaddleOCR-VL exposes a PaddleX
pipeline that answers a single request synchronously instead; the two
protocols share nothing but the result schema.

Pick the protocol from the configured address rather than adding a
second provider, so existing configurations keep working and only the
URL decides: just paddleocr.aistudio-app.com serves the job API, and any
other host gets the synchronous /layout-parsing call. The address is
compared by hostname, so an AI Studio deployment on a sibling subdomain
is routed to the synchronous pipeline while a lookalike domain is not
mistaken for the hosted service. Self-hosted deployments usually run
unauthenticated, so the access token is now required only for the hosted
service.

Fix two defects this exposed on the way:

- A block whose content is nothing but a markdown image became an empty
section carrying a position tag, because emptiness was tested before the
images were stripped. Scanned pages that the layout model labels as one
image block hit this on every page.
- HTTP, JSON and errorCode failures on the synchronous path never
reached the callback, so a wrong path or a rejected token left the
document at zero chunks with no visible cause.

Closes #18757
2026-08-31 12:26:00 +08:00
天海蒼灆 61298397f6 feat: locate Excel citation sources in document preview (#18853) 2026-08-29 23:56:57 +08:00
Aeovy 5b3a66fd8c fix(mineru): Fix MinerU Rendering Stability, Error Visibility, and Code Block Parsing Issues (#18617) 2026-08-26 20:12:41 +08:00
Yurii214 7622d6742a fix(deepdoc): harden opendataloader table-cell parsing against bad json (#17316) 2026-08-25 19:49:15 +08:00
Lynn b8b35aa9ab Fix: kimi k3 temperature (#18564) 2026-08-21 10:11:06 +08:00
Aeovy b27623a0c2 feat(MinerU):Add support for MinerU table and image chunk classification (#16703)
This PR fixes two issues with MinerU PDF parser where table and image chunks were not properly classified or associated:
1. **Table chunks** were incorrectly classified as `text` instead of `table`
2. **Image chunks** were missing image resource association and classified as `text`
2026-08-19 09:52:20 +08:00
Elias Shieh 9784814a9b fix(mineru): honor dataset language in figure prompts (#18188)
### Summary

Closes #17885.

MinerU figure enrichment now passes the resolved dataset language to
`vision_llm_figure_describe_prompt`. Missing and empty language values
use `English`, matching the other figure-description paths.

This change is limited to MinerU. PR #18021 already fixed the Mistral
path.
2026-08-13 15:39:39 +08:00
Ziyang Guo 75363af54b fix(mistral): honor dataset language in figure prompts (#18021)
### Summary

Refs #17885.

Mistral figure enrichment now receives the dataset language through the
production parsing path. `by_mistral_ocr` forwards `lang` to
`MistralParser.parse_pdf`; the parser stores the normalized language and
passes it to the figure-description prompt. Empty or missing values
still fall back to English.
2026-08-11 20:38:24 +08:00
Jack f41f866aa1 Fix(parser): keep real line breaks when merging HTML fragments and PDF boxes (#17856)
Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
2026-08-06 09:57:23 +08:00
Jin Hai cf13082a1a Revert "feat: Go knowledge compiler with scheduler-driven dataset compilation" (#17897)
Reverts infiniflow/ragflow#17881
2026-08-05 21:50:28 +08:00
Zhichang Yu 14b943a04a feat: Go knowledge compiler with scheduler-driven dataset compilation (#17881)
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
2026-08-05 20:00:42 +08:00
taek105 b7966c98f2 fix: honor dataset language across VisionFigureParser paths (#17227) 2026-08-05 18:40:10 +08:00
JackandCodeBuddy 9b05e5c67e Fix: delimiter is chunk boundary, drop token_size atom-split (OVER_CAP default) (#17808)
## Summary

Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.

**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.

## Changes

- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.

## Contract (refs #17799)

- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.

## Notes

- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.

---------

Co-authored-by: CodeBuddy <noreply@tencent.com>
2026-08-05 11:50:07 +08:00
buua436 9ccb23e661 fix: align pipeline delimiter chunking (#17723) 2026-08-03 17:47:08 +08:00
deb3d0c201 fix(chunker): enforce strict chunk_token_num cap on .txt / PDF / email paths (#17203)
Fixes #17202 (and complements #12109).

## Problem

`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.

A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.

Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.

## Fix

Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:

1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
   ```python
   if cks[-1] == "":
       cks[-1] = t; tk_nums[-1] = tnum; return
   if tk_nums[-1] + tnum <= chunk_token_num:
       cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
   cks.append(t); tk_nums.append(tnum)
   ```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.

2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.

3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.

A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.

## Result on the dataset above

| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |

## Tests

- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.

All 22 unit tests pass on the host venv:

```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit           OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks   OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge     OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget         OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size                OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge              OK
test_naive_merge.py::test_images_oversized_section_is_split                 OK
test_naive_merge.py::test_images_custom_delimiter_preserved                 OK
test_naive_merge.py::test_images_plain_string_input                         OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty           OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_…              OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated        OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget             OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_…                   OK
test_naive_merge.py::test_strict_cap_single_overlong_section_…              OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits            OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget                 OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines              OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary                     OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace  OK
test_txt_parser.py::test_empty_text_returns_empty                           OK
```

`ruff check` and `ruff format --check` are clean on all four changed
files.

## Out of scope

- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.

Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.

---------

Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-08-01 22:48:51 +08:00
Lynn 0d1cdf157c Fix: validate Mistral OCR model (#17515) 2026-07-29 14:53:32 +08:00
Yash Raj Pandey b08e5f5647 Fix: RAGFlowJsonParser crashes with IndexError on top-level JSON scalars (#16877) 2026-07-28 16:58:42 +08:00
Xavierando 08332501a8 feat: add Mistral OCR (/v1/ocr) as a document parser; fix "Can't find model" mis-tag (#5782, #7075) (#17057)
### What problem does this PR solve?

Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.

`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.

Closes #17056
Closes #5782
Closes #7075

**What it does**

1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.

**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.

**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2026-07-24 21:07:48 +08:00
Yurii214 cd9db94daa fix(deepdoc): guard docling crop against missing page images (#17147) 2026-07-24 11:20:06 +08:00
Yurii214 64be8585aa fix(deepdoc): keep pipe tables inside markdown code fences (#17201) 2026-07-23 13:31:35 +08:00
Andrew Chen 8af2467112 fix(html_parser): keep text before the first block element (#17073) 2026-07-23 12:43:16 +08:00
euvre f45f03a016 Add Go service/handler tests for API contract parity (#16905) 2026-07-20 20:02:41 +08:00
Lynn e37cd86122 Fix: OpenDataLoader parser (#17037) 2026-07-20 09:49:19 +08:00
zcxGGmuandzq afc12b2bff fix: preserve mineru cross-page table positions (#17017)
## Summary
- Read MinerU `_middle.json` alongside the content-list output.
- Preserve each matched table block as a separate PDF position tag so
cross-page tables expose all highlighted ranges.
- Keep enrichment limited to table outputs and require the first-page
bbox anchor before adding matching continuation blocks.

## Tests
-
`/Users/zq/.config/superpowers/worktrees/ragflow/issue-16992-pdf-table-coordinates/.venv/bin/python
-m pytest --confcutdir=test/unit_test/deepdoc/parser
test/unit_test/deepdoc/parser/test_mineru_parser.py -q`
-
`/Users/zq/.config/superpowers/worktrees/ragflow/issue-16992-pdf-table-coordinates/.venv/bin/ruff
check deepdoc/parser/mineru_parser.py
test/unit_test/deepdoc/parser/test_mineru_parser.py`
-
`/Users/zq/.config/superpowers/worktrees/ragflow/issue-16992-pdf-table-coordinates/.venv/bin/python
-m py_compile deepdoc/parser/mineru_parser.py
test/unit_test/deepdoc/parser/test_mineru_parser.py`
- `git diff --check`

Closes #16883

---------

Co-authored-by: zq <zhouquan1511@163.com>
2026-07-18 20:48:52 +08:00
zcxGGmuandzq a7b193d77b fix: align pdf table structure coordinates (#17016)
### What problem does this PR solve?

Table structure recognition rows, columns, headers, and spans are
produced in cropped table image coordinates, while OCR boxes are matched
later in page-cumulative coordinates. Comparing those boxes without
normalization can skip or misassign table row and column metadata.

Closes #16992.

### What is changed?

- Map TSR components from cropped or rotated table-image coordinates
back into page-cumulative coordinates before matching OCR boxes.
- Reuse one inverse rotation transform for rotated OCR boxes and TSR
components.
- Keep TSR layout ids in the same `table-N` form used by table OCR
boxes.
- Sort columns by mapped page x-coordinate after coordinate
normalization.
- Add focused unit coverage for page offsets, zoom scaling, and
90/180/270 degree rotated tables.

### Type of change

- [x] Bug fix
- [x] Test coverage

### How has this been tested?

- `uv run --group test pytest
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run --no-sync --group test pytest
--confcutdir=test/unit_test/deepdoc/parser
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run ruff check deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `uv run --no-sync python -m py_compile deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `git diff --check`

A later dependency-sync attempt was blocked while resolving the
`en-core-web-sm` wheel from GitHub, and the repository-level unit-test
conftest can try to download missing NLTK `wordnet` data when it is not
already present locally. The focused parser test above does not require
that data fixture.

---------

Co-authored-by: zq <zhouquan1511@163.com>
2026-07-18 18:22:33 +08:00
deadtrickster 982b9c7b25 fix(deepdoc): recover word boundaries for non-Latin scripts; skip OCR fallback the recogniser can't serve (#16958) 2026-07-18 00:36:25 +08:00
Harsh Kashyap 81361c2108 fix(mineru): thread page_from/page_to into MinerU API request (#16957) 2026-07-16 12:23:03 +08:00
Harsh KashyapandHarsh23Kashyap 5c96fa51f0 fix(docling): detect chunked response by shape, not request payload (#16921)
Fixes #16917.

## Problem

`deepdoc/parser/docling_parser.py::_parse_pdf_remote` decides whether
the
response is chunked based on which payload was sent, not on what came
back.
Docling Serve silently drops unknown fields such as `do_chunking`
(Pydantic
`extra="ignore"`) and returns a standard `{"document": ..., "status":
...}`
conversion response. The code then:

1. sets `is_chunked_response = True` from the request shape,
2. logs `Successfully used native chunking on: <endpoint>`,
3. extracts 0 chunks from `response_json.get("results", [])`,
4. logs `Native chunks received: 0`,
5. falls through to the existing `md_content` fallback.

The `md_content` fallback path is fine. The misleading log lines are the
problem: operators see "Successfully used native chunking" immediately
followed by "Native chunks received: 0" and "No chunk built", which
looks
like an internal regression rather than a server contract gap.

## Fix

Decide chunked-vs-standard from the **response shape**, not the request:

```python
response_is_chunk = self._looks_like_chunk_response(response_json)
is_chunked_response = chunk_flag and response_is_chunk
```

`_looks_like_chunk_response` returns True iff the response is a
non-empty
list or a dict with a non-empty `results` or `chunks` list. A standard
conversion response (`{"document": ..., "status": ...}`) does not match,
so
a server that ignored the chunking flag is correctly classified as
standard
even when the request payload asked for chunking.

When chunking was requested but the server returned a standard response,
log a single WARNING ("Server ignored chunking request on <endpoint>;
treating response as standard conversion.") instead of the INFO success
line. The misleading "Prioritizes native chunking endpoints" docstring
is
replaced with what the code actually does.

## Tests

`test/unit_test/deepdoc/parser/test_docling_parser_remote.py` (6 tests,
all passing):

- `test_remote_chunked_200_standard_payload_falls_back` (existing —
still
  passes; the `md_content` path is unchanged)
- `test_chunk_shape_helper_recognises_chunk_payloads`
- `test_chunk_shape_helper_rejects_standard_payloads`
- `test_remote_chunked_request_with_results_list_is_treated_as_chunked`
- `test_remote_top_level_list_response_is_treated_as_chunked`
- `test_remote_chunked_request_with_ignored_flag_does_not_log_success`

```
$ uv run pytest test/unit_test/deepdoc/parser/test_docling_parser_remote.py -v
============================== 6 passed in 0.26s ==============================
```

## Files changed

- `deepdoc/parser/docling_parser.py` (+35 / -5)
- `test/unit_test/deepdoc/parser/test_docling_parser_remote.py` (+89 /
-4)

## Backward compatibility

- All four payload/endpoint combinations continue to be tried in the
same order.
- The bundled-docling happy path (`parse_pdf`, not `_parse_pdf_remote`)
is
  untouched.
- A server that returns a real chunked response to a chunked request
still
goes down the chunked branch. A server that returns a standard response
  to a chunked request now goes down the standard branch with
  `is_chunked_response=False` instead of misleadingly logging success.

## Follow-up (out of scope)

Calling the real Docling-Serve native chunk endpoints
(`/v1/chunk/hybrid/source`, `/v1/chunk/hierarchical/source`) with
`HybridChunkerOptions` is a larger feature change and warrants its own
PR after this lands.

Co-authored-by: Harsh23Kashyap <harsh@example.com>
2026-07-16 09:29:09 +08:00
Wang Qi 6a4b9be426 Refactor: reformat all code for lefthook using ruff and gofmt (#16585) 2026-07-03 12:53:39 +08:00
Harsh Kashyap b42414b64a fix(deepdoc): parse bodyless HTML fragments (#16423) 2026-07-01 14:45:22 +08:00
400476f0b3 Feat: SoMark (#16482)
Follow #15486
Co-authored-by: limuting <limuting233@gmail.com>
Co-authored-by: lutianyi <lutianyi233@163.com>
Co-authored-by: justinychuang <huangyicheng@soulcode.cn>
Co-authored-by: maybehokori <138367708+maybehokori@users.noreply.github.com>
2026-07-01 13:29:28 +08:00
Yash Raj Pandey dd2c88b768 fix(excel_parser): keep zero-valued cells when building Excel text chunks (#16287) 2026-06-26 09:30:09 +08:00
Wang Qi 5defb4e7d6 Revert "fix(deepdoc): keep zero and false Excel cells in __call__" (#16366)
Reverts infiniflow/ragflow#16318
2026-06-25 19:56:47 +08:00
Harsh Kashyap 66d86154ab fix(deepdoc): accept GFM table separators with one or more dashes (#16319) 2026-06-25 19:25:57 +08:00
Harsh Kashyap 0af5d43e8d fix(deepdoc): keep zero and false Excel cells in __call__ (#16318) 2026-06-25 19:12:57 +08:00
Yash Raj Pandey 091417980e fix(html_parser): preserve original text when splitting oversized blocks (#16052)
### Bug

`RAGFlowHtmlParser.chunk_block()` splits an oversized block by slicing
the **tokenized** string and storing the joined tokens:

```python
tks_str = rag_tokenizer.tokenize(block)
...
tokens = tks_str.split(" ")
while start < len(tokens):
    chunks.append(" ".join(tokens[start:start + chunk_token_num]))  # tokenized form, not source
```

On the default (Elasticsearch) backend `rag_tokenizer.tokenize`
transforms text: it lowercases/stems Latin words and inserts spaces
between CJK characters. So any text block longer than `chunk_token_num`
is stored as garbled, lowercased, space-segmented text instead of the
source content. The small-block branch correctly stores the original
`block`, so only oversized blocks are corrupted. Affects HTML and EPUB
ingestion (both go through `chunk_block`), degrading retrieved chunks
and the answers generated from them.

### Real tokenizer behavior (infinity-sdk 0.7.0, ES backend)

```
tokenize("Hello World FOO Bar Baz Qux Jumps")  -> "hello world foo bar baz qux jump"   # lowercased + stemmed
tokenize("你好世界这是一个测试")                 -> "你好世界 这 是 一个 测试"            # spaces inserted
```

### Fix

Split the **original** text: break it into atoms (whitespace-delimited
runs for space-separated scripts, per-character for spaceless scripts
such as Chinese) and pack them into pieces of at most `chunk_token_num`
tokens. This preserves the source characters and still splits scripts
that have no whitespace — a plain whitespace split would leave CJK as
one un-splittable chunk.

### Proof (real tokenizer, before/after)

Running the old vs new split against the real `infinity.rag_tokenizer`:

```
ENGLISH "Hello World FOO Bar Baz Qux Lazy Dogs"  (chunk_token_num=4)
  OLD: ['hello world foo bar', 'baz qux jump over', 'lazi dog']          # lowercased + stemmed
  NEW: ['Hello World FOO Bar ', 'Baz Qux Jumps Over ', 'Lazy Dogs']      # preserved; each <= 4 tokens
  NEW preserves text exactly: True

CHINESE "你好世界这是一个测试用例需要被切分成多个块"  (chunk_token_num=3)
  OLD: ['你好世界 这 是', '一个 测试用例 需要', ...]                      # spurious spaces
  NEW: ['你好世', '界这是', '一个测', ...]                               # preserved; each <= 3 tokens
  NEW preserves text exactly: True
```

### Tests

Added `test/unit_test/deepdoc/parser/test_html_parser.py` (English +
Chinese oversized blocks, plus small-block merge). Before the fix the
two oversized tests fail (English shows lowercasing, Chinese shows
inserted spaces); after the fix all pass. `ruff check` clean.
2026-06-25 16:43:35 +08:00
helloxjade 1b2da645c3 fix: deduplicate markdown table chunks (#16143) 2026-06-24 13:22:57 +08:00
jaso0n0818 a70c7e8cc7 fix(deepdoc): attach lone header lines to the following section when delimiter is set (#16109)
## Summary
Fixes #15487 — lone markdown headers are no longer isolated as empty
chunks when a custom `delimiter` is set.

- Merge consecutive lone headers before attaching to the following prose
body
- Skip code fences, tables, lists, and blockquotes via
`_is_attachable_body()`
- Unit tests include the `# Title / ## Intro / Body` regression from
CodeRabbit review

## Validation
- `pytest test/unit_test/deepdoc/parser/test_markdown_parser.py` (11
passed locally)

Closes #15487
2026-06-18 14:24:09 +08:00
Yingfeng cf5cca5cbb Fix wrong unit test path (#15864) 2026-06-09 22:48:33 +08:00
Yash Raj Pandey 14c460a525 Fix: Excel parser emits a spurious header-only chunk at exact chunk_rows multiples (#15490)
### What problem does this PR solve?

`RAGFlowExcelParser.html()` iterates `(len(rows) - 1) // chunk_rows + 1`
times. `rows[0]` is the header, so `len(rows) - 1` is the data-row
count. When that count is an exact multiple of `chunk_rows`, the `+ 1`
over-counts by one: the final iteration's data slice is empty, but the
header row is still appended — producing a chunk that contains only the
table header and no data.

This is reachable via `rag/app/naive.py` (`html4excel`, `chunk_rows=12`)
and `rag/app/one.py`. A sheet with 12/24/36… data rows (or 256/512… with
the default `chunk_rows=256`) produces an extra
`<table><caption>…</caption><tr><th>…</th></tr></table>` chunk. It is
non-empty, so it passes the `if _` filter and gets indexed as a real
(empty) chunk.

| data rows (chunk_rows=12) | before | after |
|---|---|---|
| 12 | 2 chunks (1 header-only) | 1 |
| 24 | 3 chunks (1 header-only) | 2 |
| 13 | 2 (unchanged) | 2 |

### Fix

Iterate `ceil(n_data / chunk_rows)` times instead of `n_data //
chunk_rows + 1`. Adds
`test/unit_test/deepdoc/parser/test_excel_parser.py`; the
header-only-chunk cases fail before this change and pass after.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Used the Claude CLI while working on this.
2026-06-08 17:16:45 +08:00
bitloi 9f3e289b78 Fix: preserve markdown tables during delimiter extraction (#15632)
### What problem does this PR solve?

Markdown extraction can split tables row by row when delimiter-based
extraction uses a newline delimiter. That loses table structure during
chunking even though delimiters should still split normally outside
tables.

This PR keeps the follow-up to #15482 intentionally narrow:

- preserve Markdown pipe tables during delimiter-based extraction
- preserve borderless pipe tables during delimiter-based extraction
- preserve multiline HTML tables during delimiter-based extraction
- keep delimiter splitting unchanged outside protected table ranges

Refs #15482

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

### Testing

- `ruff check deepdoc/parser/markdown_parser.py
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `python3 run_tests.py -t
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `git diff --check`
2026-06-05 10:35:33 +08:00
Yufeng He 5db1b296fb fix: fall back from empty Docling native chunks (#15601)
## Summary
- keep the native Docling chunking path when it returns usable chunks
- fall back to the standard Docling response parser when a chunked
request gets HTTP 200 but returns no usable chunks
- add a regression test for older Docling servers that accept the
chunking request but return a standard conversion payload

## Why
Older external Docling servers can accept a request containing
`do_chunking: true` and still return the standard conversion response
shape. The current code treats any HTTP 200 from the chunked request as
a native chunk response, finds no chunk entries, and returns zero
sections without trying the standard response parser.

Fixes #15569.

## Validation
- `python -m pytest
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py -q`
- `python -m py_compile deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `python -m ruff check deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `git diff --check`
2026-06-04 13:42:58 +08:00