Collapse the nine identical **"cut through the last `</think>`"**
implementations — mirroring Python's `re.sub(r"^.*</think>", "", s,
re.DOTALL)` — into one shared helper `common.StripThinkTrailing`,
preventing future behavior drift between copies.
### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
Make `ResolveModelContentLength` honor the per-model custom **context window length** (`content_length`) — stored in the Python-legacy `tenant_model.extra["max_tokens"]` field, whose semantic meaning is the context window, NOT the generation cap — **before** any provider-catalog read, and remove the parallel service-layer implementation so every consumer shares one resolution path.
Switch the agent LLM component's `fitMessages` from the local map-based `messageFitInRaw`/`countAllTokens`/`stringContent`/`setContent` helpers to the new shared `internal/component/messagefit` package (#18091).
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
- Return the configured `empty_response` when retrieval has no query or
no chunks.
- Preserve `formalized_content` for downstream Message nodes.
## Testing
- `bash build.sh --go`
- `ok ragflow/internal/agent/tool`
- `ok ragflow/internal/agent/component`
Yingfeng
天海蒼灆
Sbaaoui Idriss
qinling0210
Lynn
euvre
WOLIKIMCHENG
buua436
jay77721andCommandCodeBot
Jin Hai
Haruko386
EthanZhang
Hz_
Zhichang Yu
JackandCodeBuddy
futurehua