dify/docs/BACKEND_MODERATION.md

# Strict moderation for model invocations

`ModerationService.check` is a reusable use case for the existing `keywords`,
`openai_moderation`, and `api` capabilities. It receives an admitted
`AppRequestContext`, a `ModerationRule`, text, and an `input` or `output` phase.
It does not depend on Flask, persistence models, or app queue managers.

An enabled phase returns only after approval. Any flagged verdict raises
`ModerationRejectedError`, including API verdicts requesting replacement rather
than rejection. Backend failures propagate instead of silently approving text.
Disabled phases perform no moderation I/O. Strict policies are selected when
assembling models. App OpenAI rules consume typed platform failures to apply
configured replacement behavior; they never schedule a second OpenAI check.

## Automatic platform moderation

`HOSTED_MODERATION_ENABLED` controls automatic platform moderation in Cloud and
defaults to false. It takes effect only when `DEPLOYMENT_EDITION=CLOUD`, matching
the edition boundary used to assemble AI Credits quotas. Non-Cloud deployments
ignore the switch. When disabled or outside Cloud, no automatic policy is
constructed or injected. Explicit model policies and app-level keyword/API
moderation remain available in every edition. App-level Model Moderation is a
silent no-op when the model is not admitted for automatic AI-credit moderation;
enabling its app configuration does not select a tenant moderation provider.

When enabled,
Cloud LLMs using a system provider with a current `TRIAL` or `PAID`
credit-pool quota automatically receive input and output moderation.
This includes `CREDITS` and legacy `TIMES` quotas, which reserve one credit per call.
This applies to manager-created models, credential-cache hits, and direct `ModelInstance`
construction, including exhausted or unlimited credit quotas. Cloud custom-key,
non-LLM, and token-based quota modes do not receive this automatic policy.
Merely configuring a fallback identifier does not enable moderation.

For example, an operator can enable moderation and pin an optional fallback:

```dotenv
DEPLOYMENT_EDITION=CLOUD
HOSTED_MODERATION_ENABLED=true
HOSTED_OPENAI_API_KEY=<server-owned-api-key>
HOSTED_MODERATION_PLUGIN_UNIQUE_IDENTIFIER=langgenius/openai:<version>@<checksum>
```

The policy uses `langgenius/openai/openai` / `omni-moderation-latest` with the
server-owned `HOSTED_OPENAI_API_KEY` and optional OpenAI base/organization settings.
These credentials do not require enabling an OpenAI LLM trial/paid provider or
initializing Cloud hosting quotas. It first discovers the tenant's
`langgenius/openai/openai` provider and requires a Marketplace installation,
moderation capability, the official package identity, and a verified manifest.
If that route is absent, untrusted, or fails, it uses
`HOSTED_MODERATION_PLUGIN_UNIQUE_IDENTIFIER`, a pinned
`langgenius/openai:<version>@<checksum>` fallback supplied by SRE. A positive
violation verdict is final: it never triggers fallback to obtain another verdict.
The fallback identifier is validated only if needed; a working trusted tenant
plugin does not require it. Missing both routes, platform credentials, or a
working backend fails closed. A user's API key is never substituted.

The verified tenant route uses `/plugin/{tenant_id}/dispatch/moderation/invoke`,
which authorizes the plugin through the tenant's installation record. The SRE
fallback instead uses `/v2/invoke/dispatch/moderation/invoke` with a mandatory
`X-Plugin-Unique-Identifier` and the admitted `tenant_id` in JSON. Both routes
keep daemon server-key authentication, platform credentials, and pinned package
identity. An arbitrary client request cannot select direct dispatch or replace
the server-managed fallback identifier.

The v2 route is the daemon's existing direct-runtime invocation surface, originally
introduced for Slim remote CLI calls. It parses the supplied identifier, preserves
tenant execution context and cluster routing, and does not query tenant plugin
installations. It is not an installation endpoint or a public tenant API. The
fallback runtime must already be deployed and available to the daemon; direct
dispatch does not bootstrap a missing package. The fallback remains an SRE-owned
trust decision, while tenant-selected packages still require Marketplace and
verified-manifest checks. Moderation does not construct a
tenant model instance or enter `CreditPoolService`, so it neither reserves nor
deducts user credits. The target LLM retains its normal accounting, including
charging consumed usage when output is rejected.

The former `HostingModerationFeature` and `HOSTED_MODERATION_PROVIDERS` allowlist
have been removed. `HOSTED_MODERATION_ENABLED` controls the Cloud platform policy.
The old
random input-fragment check is replaced by complete segmented checks.

An explicitly supplied policy supplements enabled platform moderation rather
than disabling it. When the switch is off, only the explicit policy applies.

## App-level ownership and replacement

Each model guard exclusively owns its input
and output checks. App Model Moderation never proactively invokes OpenAI, selects
credentials, or caches approved content. Its configuration only selects a reply
for a real platform rejection. Disabling an app phase does not disable mandatory
platform checks. Different model invocations remain independent even when their
text is identical.

App checks and reply selection are stateless functions of the admitted app
configuration. There is no moderation scope, context variable, failure queue,
observer, or cross-thread model-to-response side channel. Parallel workflow model
calls do not serialize their moderation requests.
Startup injects one stateless policy port into the application container; response
pipelines retain that port, not an invocation-scoped session. The kernel does not
import service implementations or require a moderation wrapper around each entry.

Graphon node results retain exception type names, but terminal graph events only
carry display text. The Dify workflow runner propagates the failing node's type
through an internal field on its terminal queue event. Container failures inherit
their failed child's type; handled failures and retries do not classify a later
terminal failure. Platform input and output rejections have distinct exception
types so phase-specific presets survive this boundary without parsing messages.
Because Graphon has no terminal failure source ID, container error text correlates
its selected failure with typed child events. Text never supplies a type or phase;
conflicting child types retain the generic error instead of inventing a verdict.
The internal field is excluded from serialization and does not change public SSE.
The parameter-extraction node has a small compatibility adapter because Graphon
currently drops even the exception type when catching model invocation failures.
It retains only that type during one node run and clears it on every exit.

Keyword and API app rules still use `ModerationService.evaluate` and retain their
direct-reply and text-override semantics. They are independent policies, not
duplicate OpenAI checks, and cannot bypass platform moderation. Output checks run
sequentially on the response consumer without a worker or evaluation lock.
Streaming checks inspect only new segments; completion checks only an unchecked
tail, never the whole stream again. Direct replies replace the whole answer;
API overrides replace the flagged segment while retaining preceding output and
later deltas. Blocking output uses the non-stream segment size. Segments have no
overlap or cross-boundary context. The existing app-level display-then-replace
behavior remains distinct from the platform guard's check-before-delivery guarantee.

Application entry points call the same stateless app adapters; no request wrapper
is required to enable the correct behavior. Standalone kernel callers without an
application container retain the keyword/API factory fallback. Their App OpenAI
toggle is still a no-op and never selects an independent tenant provider.
Independent use of the OpenAI capability retains tenant credentials, accounting,
and failure behavior. It is an explicit independent
operation, not an execution path selected by the App Model Moderation toggle.

The plugin `/invoke/app` entry uses the same app generator behavior. Invocation
validates tenant ownership before model work. Each workflow runner owns its
event state, so nested or subsequent requests cannot inherit another run's failures.

Remote Agent LLM gateway calls do not transport local workflow node error types.
Local App Model Moderation never issues hosted or tenant
checks and cannot infer a typed remote verdict from text. Other configured app
rules remain independent. A client-provided approval flag must never skip platform checks.

An enabled app OpenAI rule uses its nonblank phase-specific preset when the
platform rejects content. Chatbot, Chatflow, and legacy Agent Chat use the
English default `This message can’t be processed.` when no matching
preset is configured, including when app moderation or its phase is disabled.
The default is resolved at runtime, not written into the app configuration.
Other app moderation backends retain their own verdict and replacement rules;
their presets are not reused for a separate platform rejection.

Chat replies use the existing `message` or `message_replace` event followed by
`message_end`, without exposing the rejected segment. A replacement covers the
entire answer, including any previously delivered prefix, and the stored answer
matches the replacement. Rendering it does not invoke a model or estimate token
usage again. A failed chatflow retains its failed workflow status while rendering
the failing node's platform rejection with its preset or default reply.
The reply and `message_end` precede the failed `workflow_finished` event because
the Redis/SSE subscriber stops reading at that terminal event; sending the reply
afterwards would persist it without ever delivering it to the WebApp client.

Unavailable checks are errors, not content verdicts, and never become replies.
Chatflow restores the typed `moderation_unavailable` error from the failing node's
error type. Error text alone cannot classify an unrelated failure, and no request
state is shared between invocations.
The error event precedes `workflow_finished` so public subscribers receive its
503 payload, while blocking requests raise the typed error for HTTP mapping.
No unavailable check becomes a preset or default chat reply.
Explicit non-platform model policy failures are also preserved. Completion,
Workflow, RAG, and other non-chat paths retain the normal moderation error when
there is no matching preset. Remote New Agent errors retain their existing
transport handling; arbitrary upstream errors are not inferred to be platform
rejections from their text.

Platform rejections emit warning logs containing tenant, app, and phase.
Tenant-route failures also log fallback warnings. Moderated content and API keys
are never included in these log messages.

## App errors

Model-bound guards adapt moderation failures to the existing `AppGenerateError`
contract, so task error handling preserves them and response converters emit
the usual SSE `error` event. Console clients use their existing error toast:

Application errors carry stable moderation identifiers and safe messages, not
HTTP status codes or response payloads. HTTP handlers and the SSE response
converter own status mapping. Keyword/API input and output backend failures are
adapted to the same unavailable error; private diagnostics remain in the cause
and server logs, never in the client-facing response.

| Condition | Code | Event status | Message |
| --- | --- | --- | --- |
| Content rejected | `moderation_rejected` | 400 | The configured rejection message, or the phase-specific fallback |
| Check unavailable or failed | `moderation_unavailable` | 503 | This message can’t be processed right now. Please try again later. |

These statuses describe the SSE error payload, not the HTTP status of an already
opened stream. Non-streaming requests on Console, Service API, and Web API return
the same codes and messages with the corresponding HTTP status. Check failures
retain their original cause in backend logs but
never expose platform credentials or plugin diagnostics to clients. Provider
generation failures retain their existing classification. When an error is
surfaced, approved segments remain visible; neither error delivers the rejected
or unchecked segment.
Chat apps may render a platform rejection using a matching preset or the default
reply described above instead of returning this error payload.

App output evaluation failures use the same `moderation_unavailable` contract.
The response consumer immediately stops consuming queued output, persists the
message's error status, and emits the existing error event. The handler retains
the first failure without locks; completion cannot retry it into approval.
Already delivered app-level output is not retracted by this error; platform
model guards continue to check segments before delivery.

## Model invocation

Obtain the service from the composition root and bind the policy during model
assembly. Individual model invocations never accept a moderation override:

```python
from core.plugin.impl.model_runtime_factory import create_plugin_model_assembly
from extensions.application_services.moderation import build_llm_moderation_guard
from extensions.ext_application_services import application_services
from graphon.model_runtime.entities.model_entities import ModelType
from machinery.context import AppRequestContext
from services.moderation_service import ModerationRule

guard = build_llm_moderation_guard(
    AppRequestContext(tenant_id=tenant_id, app_id=app_id),
    service=application_services().moderation,
    rule=ModerationRule(
        type="keywords",
        config={
            "keywords": "blocked phrase\nanother phrase",
            "inputs_config": {"enabled": True, "preset_response": "Input rejected"},
            "outputs_config": {"enabled": True, "preset_response": "Output rejected"},
        },
    ),
)

assembly = create_plugin_model_assembly(tenant_id=tenant_id, moderation=guard)
model_instance = assembly.model_manager.get_model_instance(
    tenant_id=tenant_id,
    provider=provider,
    model_type=ModelType.LLM,
    model=model_name,
)
result = model_instance.invoke_llm(prompt_messages=prompt_messages, stream=False)
```

Use the same configuration shape as app moderation. OpenAI uses
`type="openai_moderation"`; API extensions use `type="api"` with
`api_based_extension_id` scoped to the admitted tenant. An empty `app_id` is
allowed for non-app invocations; API extensions receive it unchanged. The guard
snapshots the configuration so later caller mutations cannot disable its checks.

`ModelManager.for_tenant(..., moderation=guard)`,
`create_plugin_model_manager(..., moderation=guard)`, and direct
`ModelInstance(..., moderation=guard)` construction support the same binding.
The policy follows every manager-created LLM, including cached credentials and
default-model resolution. It is exposed read-only on the instance, and mismatched
tenant bindings fail before credential resolution. Non-LLM capabilities do not
inherit LLM policies, so moderation-model calls do not recursively moderate themselves.

The assembly owner supplies a trusted scope for explicit policies. Automatic
selection uses trusted deployment and provider quota configuration, never
caller-provided request metadata. Raw plugin runtime/client interfaces remain
infrastructure, not moderated business entry points.

Both segment limits are required when constructing a guard. Production assembly
always provides the configured positive limits; there is no alternate unbounded
whole-stream buffering mode selected by an omitted limit.

## Guarantees and trade-offs

- Input checks precede model invocation and quota reservation. All prompt roles,
  text parts of multimodal messages, and assistant tool arguments are inspected
  as independent non-stream segments, so long prompts are fully covered without
  sending one oversized moderation request. Each platform-checked prompt message
  has its own segment sequence. Each model invocation performs its own checks;
  app response handling does not schedule another moderation request.
- Opaque Agent gateway dictionaries are inspected through validated copies. The
  original payload and provider-specific extension fields are forwarded unchanged.
- Output checks inspect completion text, tool arguments, and native structured
  payloads even when their message text is empty. Binary image, audio,
  video, and document content is not inspected by these text moderation backends.
- Non-stream output is checked in independent segments of
  `MODERATION_NON_STREAM_SEGMENT_SIZE` characters. Every segment must pass before
  the original result is returned.
- Streaming output is buffered until it reaches
  `MODERATION_BUFFER_SIZE` characters, then checked before its original
  chunks are released. The final partial segment is also checked before release.
  Provider chunks are kept intact, so a segment can exceed the configured threshold.
  A later rejection stops the stream and preserves earlier delivered segments;
  current model-level consumers cannot retract those bytes.
- Segmentation is the moderation boundary: each segment is checked independently,
  without carrying text across segment boundaries or rechecking the full document
  at completion. Content requiring cross-segment context may not be detected.
  Defaults are 2000 characters for input and non-stream output, and a 300-character
  buffering threshold for streaming output in both app-level and model-level moderation.
- Consumed model usage is settled before output checking, including rejected
  completions. A provider or moderation error is not an approval or a model retry.
- Structured output automatically uses the policy of its model instance. Workflow
  polling applies that same policy before task submission and to successful terminal
  results, after settling usage. Rejected inputs neither submit a task nor reserve
  model quota; rejected successful outputs do not refund consumed quota.
- The guard protects returned results, not provider-level observer callbacks.
  Callbacks must not independently publish unmoderated model content.
- Models outside enabled automatic platform moderation retain their existing invocation and
  streaming behavior unless an explicit policy is supplied.

The API adapter loads tenant-scoped extension records in a short repository
session, then decrypts credentials and performs HTTP outside that session. It
requires an explicit boolean `flagged` field in every API verdict.

## RAG failure handling

`ModerationInvocationError` identifies model moderation failures without exposing
application-service dependencies to RAG consumers. Both rejection and service
unavailability remain terminal for the affected operation, not best-effort empty
results. Input failures precede model usage; output failures retain consumed
model charges.

QA formatting propagates moderation failures in preview and indexing mode.
Bounded worker futures return failures to the indexing coordinator, which marks
the document as errored instead of completing an empty or partial QA index.
Workers retain ten-chunk batches, so a failed batch does not submit subsequent
batches or incur further model calls for them.
Ordinary QA generation failures retain their existing best-effort handling.

Function-call dataset routing also propagates moderation failures. Ordinary
routing failures may still yield no selected dataset, but moderation failure
must never be interpreted as a successful empty retrieval.

Automatic metadata filtering in both legacy RAG and KnowledgeFS stops on
moderation failures instead of continuing with no automatic conditions.
KnowledgeFS emits a failed node event with the moderation exception type and
does not issue a retrieval request. Ordinary extraction and parsing failures
keep their existing fallback behavior.

Paragraph and parent-child summary previews preserve moderation failures from
their worker futures. Indexing-estimate adapters and application services pass
the shared moderation error contract through unchanged, so existing HTTP
handlers return `moderation_rejected` (400) or `moderation_unavailable` (503)
instead of a generic indexing-estimate error. Ordinary preview failures keep
their existing aggregated error messages.

Background summary generation records `ERROR` on the affected summary and does
not save or vectorize the failed LLM output. Other segments can still finish;
the original document's indexing status is independent of its summary status.