```
SHOW MODEL STATS;
SHOW MODEL STATS DAYS 7;
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31';
SHOW MODEL STATS DAYS 7 'openai'; -- provider
SHOW MODEL STATS DAYS 7 'openai' 'prod'; -- provider + instance
SHOW MODEL STATS DAYS 7 'openai' 'prod' 'gpt-4o'; -- provider + instance + model
SHOW MODEL STATS DAYS 7 BY DATE;
SHOW MODEL STATS DAYS 7 BY MODEL;
SHOW MODEL STATS DAYS 7 BY INSTANCE;
SHOW MODEL STATS DAYS 7 BY PROVIDER;
SHOW MODEL STATS DAYS 30 'openai' BY DATE;
SHOW MODEL STATS DAYS 30 'openai' 'prod' BY MODEL;
SHOW MODEL STATS DAYS 30 'openai' 'prod' 'gpt-4o' BY DATE;
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31' 'azure' BY INSTANCE;
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31' BY PROVIDER;
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
> [!NOTE]
> **AI-generated, not reviewed by a human.** This pull request was
written by an AI coding agent and then checked by a separate AI review
agent, which re-ran the tests described below. No human has reviewed the
code. It is submitted by @monsterlag, who runs these agents. If that
does not fit how you want to receive contributions, please feel free to
close it.
### Summary
Fixes#21260.
Title-family chunkers treated captionless PDF figures carrying only
coordinates as empty, dropping them before preview cropping and merging
text across the missing figure.
Keep structured records when the existing position decoder/merger
retains rows from either `positions` or `_pdf_positions`. Check decoded
content rather than serialized JSON length so `[]`, `null`, empty rows
and short rows still count as empty.
Regression tests cover GroupTitle, HierarchyTitle and Manual with either
position field or both, empty-position records between text chunks, and
`root_chunk_as_heading`. The crop/upload regression verifies a
position-only figure after the root heading supplies text.
This change is limited to record retention. Crop and tokenizer filters
for media whose finalized text remains empty are unchanged; it does not
enable end-to-end indexing of textless figures.
### Testing
- Reproduced the empty-position regression before this revision: 36
strategy/field/value combinations and the root-heading case failed; all
pass after the fix.
- Passed focused position/crop regressions with `bash build.sh --test
-count=1 -run
'TestTitleFamily.*Position|TestCropTitleChunks.*PositionOnly'
./internal/ingestion/component/chunker`.
- Passed `bash build.sh --test -count=1
./internal/ingestion/component/...`, including the full chunker suite
and the wrapper's native unit tests.
- Passed `gofmt`, `git diff --check`, and lefthook pre-commit checks.
Tests used Go 1.27, project-pinned native libraries, and the isolated
Clang/LLD 21 sysroot prepared during implementation. A local
`GOFLAGS=-exec=...` loader wrapper supplies the newer glibc/libstdc++
needed by those libraries; no build configuration or environment files
are included.
## Summary
Rename the repository dependency-resource directory `ragflow_deps/` to
`deps/`, and update every hardcoded reference to that directory name so
the rename does not break resource loading at build/runtime.
This is not a purely cosmetic rename: the Go runtime hardcodes the
directory name as a resource search path (e.g.
`internal/common/model_assets.go` trimming the `deps/` prefix, and the
tokenizers joining `deps/` into candidate paths), and the Dockerfiles
copy assets into `/ragflow/deps`. Those literals must move together with
the directory, otherwise assets fail to resolve.
## Two intentional exceptions (kept unchanged)
The rename deliberately keeps two names that only *coincidentally*
contained the old string:
- The published image tag `infiniflow/ragflow_deps:latest` (used by
`Dockerfile`, `Dockerfile_base`, docs, and CI `image inspect` checks).
- The CI install path `/opt/ragflow_deps` inside the CI runner image
(`build.sh` `SYSTEM_DEPS_TOKENIZER`).
## Changes
- `git mv ragflow_deps deps`.
- `build.sh`: `CL100K_TABLE_RELPATH`, native asset zip path, and related
references updated to `deps/`.
- `tools/scripts/download_dev_deps.py`: `RAGFLOW_DEPS_DIR` points at
`deps/`.
- Go runtime resource search paths (`internal/common`,
`internal/tokenizer`) updated to the `deps/` layout.
- `Dockerfile` / `Dockerfile_base`: container asset path changed to
`/ragflow/deps`; kept tag `infiniflow/ragflow_deps:latest` unchanged.
- Docs / `AGENTS.md` / CI workflow paths updated to the new directory
name.
## Test plan
- Go unit tests in `internal/common` and `internal/tokenizer` updated to
the new `deps/` layout and pass via `bash build.sh --test`.
- Python download scripts import and build URLs correctly (`py_compile`
clean).
- Added `tools/scripts/test_dep_rename_invariants.py` as a regression
guard asserting the two intentional exceptions (`/opt/ragflow_deps` and
`infiniflow/ragflow_deps`) stay intact, so a future blanket rename
cannot silently regress them.
---
## Follow-up fix (commit 2f6857095d)
The original rename used a blanket `sed s/ragflow_deps/deps/g` and only
selectively reverted `build.sh`. That left two intentionally-kept names
rewritten, which this follow-up restores:
- CI install path `/opt/ragflow_deps` — `sep-tests.yml` had been changed
to read `/opt/deps`, breaking the offline cl100k table fallback (the
cached-image path pointed at a directory that never receives the file).
Restored to `/opt/ragflow_deps`.
- Published image tag `infiniflow/ragflow_deps` — left as bare
`infiniflow/deps` in `sep-tests.yml` comments/echo, `.dockerignore`, and
`Dockerfile` / `Dockerfile_base` error messages. Restored to
`infiniflow/ragflow_deps`.
The regression guard `tools/scripts/test_dep_rename_invariants.py` was
added to prevent recurrence.
## Summary
The DSL no longer supports `output_format` selection.
`normalizeParserJSON` (in `internal/ingestion/component/parser.go`)
hardcodes `setup["output_format"] = "json"` for every parser setup, so
the `ParseResult.Text` field — populated only on the now-unreachable
text output path — is dead.
This change removes `ParseResult.Text` and every read/write site:
- `internal/parser/parser/parse_result.go`: drop the `Text` field and
update payload-family helpers (`isPayloadZero`, `payloadFamily`, etc.).
- `internal/parser/parser/email_parser.go` / `doc_parser.go`: drop the
text-output branches and `output_format:"text"` handling.
- `internal/ingestion/component/parser.go`: the normalization already
forces `json`; remove the text branch in `normalizeParserJSON` /
`buildParserOutputs`.
- `internal/service/file/file_content.go`: `parseResultText` no longer
reads `res.Text`; it falls back to `res.Markdown` for legacy
markdown-only payloads.
- `internal/service/document/document_artifact.go`: DOC preview now
reads preview text from the JSON items.
- `internal/parser/parser/pdf_result.go`: drop the dead `case
result.Text != ""` fallback in `NormalizeResult` (no PDF producer sets
`parser.ParseResult.Text`; confirmed by grep).
## Test plan
- Updated / removed obsolete text-path unit tests; added guard tests
(`text_field_removed_test.go`, `text_field_removed_doc_test.go`)
asserting the field is gone and JSON is the only output family.
- `bash build.sh --test ./internal/parser/parser/...
./internal/ingestion/component/... ./internal/service/file/...
./internal/service/document/...` — all green.
- Full `bash build.sh --test` regression — all green (no FAIL).
- `bash build.sh --go` builds both `ragflow-cli` and `ragflow_server`.
## Notes
`deepdoctype.ParseResult` (a different struct in
`internal/deepdoc/parser/type`) still has its own `Text` / `Sections`
fields; those are unrelated and intentionally untouched.
---------
Co-authored-by: CodeBuddy Code <noreply@codebuddy.ai>
## What
Add **typed read-side accessors** for the ingestion canvas envelope so
consumers
read `[]ChunkDoc` / `[]parser.DocItem` instead of bare type assertions,
and remove
the three silent carrier-mismatch skips that the untyped
`map[string]any` form
allowed:
- `chunker` `imageUploadDecorator` — id-stamp + image-upload pass
- `knowledge_compiler` `mergeChunks` / `buildInputs`
- `task/indexdoc` `NormalizeChunks`
Introduce the read-side `schema.GetChunks` accessor plus the lossless
`schema.ChunkCarrierToMaps` bridge. The speculative `PutChunks` /
`PutParseItems` /
`GetParseItems` helpers added alongside it were dead code (no production
callers)
and have since been removed; `GetChunks` accepts the legacy
`[]map[string]any` and `[]any` carriers as well as `[]ChunkDoc`, so no
producer
needs to change for consumers to migrate.
Reconcile `ChunkDoc.MarshalJSON` with `ToMap` via a shared non-recursive
`wireMap`, so the typed `[]ChunkDoc` form and the legacy
`[]map[string]any` wire
form serialize **identically** (parity-gated by test). This matches the
existing
`DocItem` `MarshalJSON`→`ToMap` pattern and fixes the empty-text
media-chunk
divergence (`text` was dropped by `omitempty`).
## Why
The untyped envelope forced every consumer to repeat bare
`.([]map[string]any)`
assertions and defensive `switch`/`clog.Warn` fallbacks; a carrier-type
mismatch
silently skipped work (e.g. the image-upload decorator). Centralizing
the typed
read behind accessors removes that fragility while keeping the wire
format
unchanged.
## Scope / design decision
**Producers continue to emit the documented `[]map[string]any` wire
form** (built
internally as `[]ChunkDoc` and converted via `schema.ChunkDocsToMaps`).
Consumers
read through `GetChunks` / `ChunkCarrierToMaps`, which bridge both
carrier forms,
so this change is strictly additive — no flag day, no producer
migration.
A follow-up attempt to make producers emit the typed `[]ChunkDoc`
carrier directly
(Phase 4) was **reverted** (final commit). Rationale:
- It violates the documented component-output contract: `ChunkerOutputs`
and the
tokenizer output are specified as `list[object]` / `map[string]any`.
- It breaks every consumer that reads `out["chunks"].([]map[string]any)`
at the
**Go value layer** — including concurrent `main` changes (e.g.
`pdf_postprocess_test`) that panic. JSON-byte parity (`MarshalJSON ==
ToMap`) is
necessary but **not sufficient**: consumers assert on the value type,
not the
serialized bytes.
- Changing the producer emission type is a **shared-wire contract
change**. It
must not be done as a side effect of a feature PR; it requires its own
contract-change epic: revise the documented contract, coordinate all
consumers, and land it as a flag-day or with an explicit transition
window.
The original typed-access goal is already fully met by the read-side
accessors
above, so no producer change is needed.
## Testing (TDD)
Failing tests were written first for: the accessors (round-trip,
legacy-map
bridge, empty/nil, garbage-no-panic, parse-items), the serialization
parity gate,
and each migrated consumer (typed-carrier acceptance). Unit tier is
green:
`bash build.sh --test ./internal/ingestion/...` — all packages ok.
`internal/service/document` (only non-ingestion `schema` importer) also
ok.
## Risk
Low. The `Component.Invoke` signature is unchanged, agent/canvas is
untouched,
the wire format is byte-identical, and `GetChunks` bridges both carrier
forms.
---------
Co-authored-by: CodeBuddy Code <noreply@codebuddy.ai>
### Summary
`tableutil.ParseTableHTML` builds table rows in source order. HTML 4
required `<tfoot>` to come before `<tbody>`, and browsers draw it below
the body wherever it stands. Tables written in that order are still
common: generated reports, documentation tools, older CMS exports. For
those tables the footer row landed between the header and the data:
```html
<table>
<thead><tr><th>Item</th><th>Qty</th></tr></thead>
<tfoot><tr><th>Total</th><th>3</th></tr></tfoot>
<tbody><tr><td>pen</td><td>1</td></tr><tr><td>ink</td><td>2</td></tr></tbody>
</table>
```
```text
main: Item | Qty / Total | 3 / pen | 1 / ink | 2 HeaderRows = 2
this branch: Item | Qty / pen | 1 / ink | 2 / Total | 3 HeaderRows = 1
```
A footer of `<th>` cells was also counted as a second header row,
because the header count is the leading run of `<th>` rows.
### What changed
`ParseTableHTML` collects the rows of a `<tfoot>` apart and appends them
after the body. Their `<th>` flags go with them, so they no longer
extend the header. Tables without a `<tfoot>` are unchanged. Nested
tables are still folded into their cell by `cellText`, as before.
The HTML parser's table items, the chunker and the deepdoc converter all
build their grids through `ParseTableHTML` / `BuildTableData`, so they
all get the rendered order.
### Testing
`TestParseTableHTMLFooterRowsComeLast` uses the table above and expects
the rows in rendered order with one header row. It fails on `main` with
`HeaderRows = 2`.
```
CGO_ENABLED=0 go test ./internal/parser/... # ok (chunk, parser, tableutil)
TIKTOKEN_CACHE_DIR=… go test ./internal/ingestion/component/chunker/ ./internal/service/file/
# chunker: only TestPaddleOCRHighlightsThroughPipeline fails, identically on main (local environment)
go vet ./internal/parser/tableutil/ && gofmt -l internal/parser/ # clean
```
dependabot[bot]
Jin Hai
Sbaaoui Idriss
chanx
xugangqiang
maoyifeng
Wang Qi
Yash Raj Pandey
ZZZZY
Bruce-Yii
monsterlag
buua436
Lynn
balibabu
ciel
jay77721
adws
Harsh Kashyap
Lukas