Compare commits

...
Author SHA1 Message Date
alvinfeiandClaude Opus 5.5 a218a92d4e fix(memhub): memory writes retry themselves; a broken model provider pauses the queue instead of losing jobs
The SDK keeps one outcome per request id and replays it: a write that failed
under an id fails under it forever, without calling the model. Re-running a
job under its idempotency key (including the old manual retry) therefore never
reached the model again. Every re-run now gets an SDK id of its own:
`idempotency_key` on attempt 0 (unchanged for existing jobs), then
`idempotency_key#rN`, N stored on the job (`sdk_attempt`, migration 61). Before
a re-run the outcome of the previous id is checked; a write that did succeed is
completed from its receipt and not made twice.

- Provider failure (401/403, 404, 408, 429, 5xx, timeout, connection error,
  breaker refusal) during the attempt: the job goes back to `pending` as it
  was: same place in the queue, no attempt or retry spent, no cap. The breaker
  paces it. Attribution: `ModelBackoff.attempt()` charges every provider
  failure to every write in flight, because a request cannot be traced to its
  job (the SDK batches embeddings on one shared worker).
- Any other failure: retried after 30 s, 60 s, 120 s, then `failed` with its
  code; the queue moves on.
- Crash or interruption mid-write (`unknown`): re-run automatically under a
  new id after the outcome check, counted as a retry. Existing `unknown` jobs
  are handled the same way after upgrade, once. A write still running on its
  thread in this process is left until it ends.
- The manual retry (`POST /api/v1/memory-processing/retry-unknown`, the App's
  重新整理 button and its strings) is removed.
- App banner while the provider is paused, in 11 locales; `error_code` and
  `next_retry_at` on `GET /api/v2/memory-writes/{id}` and
  `GET /api/v2/memories/additions/{id}`; `memhub status` shows the pause.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:58:23 +08:00
alvinfeiandClaude Opus 5.5 6845adf083 fix(memhub): WAL stays bounded, disabled sources suspend their jobs, recall no longer starves behind stuck writes
WAL (#245): the writer sets journal_size_limit (64 MiB), Core runs a
TRUNCATE checkpoint when the main database is idle (every minute, with a
50 ms busy wait, skipped rather than raised when busy) and at shutdown, and
the size alert judges pending frames rather than the file's size.

Disabled sources (#245): jobs of a source that is off or revoked are
suspended. Rows keep their state; status counts their waiting jobs as
`suspended` and queue depths leave them out. The claim gate already skips
them, so turning the source back on resumes them.

Gate agreement: a job the execute gate hands back (release_unstarted) is
held for 5 s through its lease, so a persistent disagreement between the
claim gate (lane user) and the execute gate (Evidence user) cannot spin.

Recall (#231): SDK writes run in their own bounded pool (12), recall's SDK
reads in theirs (8). The recall route has a total budget (timeout + 3 s,
at most 12 s) and answers 504 recall_timeout. The broker maps a Core
timeout to 504 and the client maps 504/ECONNABORTED/ETIMEDOUT to
broker_timeout. retry-unknown requeues off the event loop (#162).

Model backoff: a process-wide breaker per model profile revision records
every provider exchange through the SDK transport. After a provider failure
(auth, not found, rate limit, timeout, 5xx, unreachable) the drain claims no
jobs for 5 s, then one probe; each failed probe doubles the pause up to
3 minutes. The first success, or any model profile create, save, activation
or embedding switch, resets it. memory-processing reports
llm_backoff_until and llm_backoff_error.

Fixes #245, Fixes #231, Refs #162

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:58:23 +08:00
feitianxiang 850c728c9d Merge pull request #280 from Tencent-Hunyuan/fix/multi-user-hardening
fix(memhub): multi-user responses echo user_id, cross-user delete answers 404, Hermes profiles isolated for any root
2026-11-02 16:44:04 +08:00
alvinfeiandClaude Opus 5.5 ce9801b449 fix(memhub): resolve a caller's Hermes root through the grant's own helper
The overlap check resolved the caller-supplied root a second time in new
code. Resolve it with the same helper the grant uses (`resolved_source_root`
in access.py); the Hermes check now compares an already-resolved root and
only resolves other users' stored grant roots itself. A root that cannot be
resolved is left to the grant that follows, which refuses it with its own
code; the `profiles` container is still refused by name before that.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:32:17 +08:00
alvinfeiandClaude Opus 5.5 576d9bf408 fix(memhub): multi-user responses echo user_id, cross-user delete answers 404, Hermes profiles isolated for any root
Hardening of the multi-user feature from #250, after review.

- Read responses carry `user_id`, the user actually read (`local` when none
  was named): memory-library, memory-library/layers, user-profile,
  home-summary, memories/graph, raw, context/search, v1/recall and
  memories/ask. ask/stream puts it on its first (`recall`) event. The
  contracts types gain it as optional, and the Host Bridge context validator
  accepts it as an optional key, so an older Core still validates.
- DELETE /api/v2/memories/{id}: 204 only when the caller's own memory is
  deleted; another user's memory, a missing id, and an SDK result with
  deleted_count 0 (a delete that lost a race) all answer the same 404
  `memory_not_found`. The App treats it as already gone; `memhub memories
  delete` prints that it was not found, exits 0, and its JSON is committed
  with `already_gone: true`.
- Hermes skips a `profiles` directory at any depth (discovery helper:
  `excluded_directory_names`), so a root above ~/.hermes never reads another
  user's profile. Connect (other-user root, local connect and authorize)
  refuses a root that is the `profiles` container (422
  `hermes_root_is_profiles_container`) and a root equal to, containing or
  inside another user's active Hermes grant, symlinks resolved (409
  `hermes_root_overlaps_other_user`); `local` on ~/.hermes with another user
  on ~/.hermes/profiles/<name> stays allowed. The CLI explains both codes.
- `sources connect --user-id` with an invalid id says the user id need not
  equal the Hermes profile name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:26:20 +08:00
feitianxiang 34f406e96d Merge pull request #277 from Tencent-Hunyuan/feat/memhub-hy-memory-2.0.12
feat(memhub): hy-memory 2.0.12; intention dedup switch in memory features
2026-11-02 15:16:36 +08:00
alvinfeiandClaude Opus 5.5 a653bd4065 feat(memhub): intention dedup switch in memory features
Expose hy-memory 2.0.12's per-client intention dedup in the memory-features
setting: the feature intention_dedup (recall.intention_dedup_enabled, default
on) and the parameter intention_dedup_threshold
(recall.intention_dedup_threshold, default 0.95, accepted 0.8-1.0, else 422).
Both are part of the features key, so a change rebuilds the SDK clients like
the other switches. CLI: memhub settings features set intention_dedup=false,
intention_dedup_threshold=0.9. Refs #241.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 15:04:24 +08:00
alvinfeiandClaude Opus 5.5 07b426ba7f chore(memhub): pin hy-memory 2.0.12
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 15:04:24 +08:00
feitianxiang 4b83b79b2b Merge pull request #275 from Tencent-Hunyuan/release/hy-memory-2.0.12
release: hy-memory 2.0.12
2026-11-02 14:58:42 +08:00
alvinfeiandClaude Opus 5.5 675cc30e5a release: hy-memory 2.0.12
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:30:15 +08:00
feitianxiang 3511153f9f Merge pull request #274 from Tencent-Hunyuan/fix/deepseek-harness-profiles
fix(memhub): DeepSeek Harness one switch grants every profile; import versioned sessions and v3 tool text
2026-11-02 14:17:44 +08:00
feitianxiang 23facf75d5 Merge pull request #273 from Tencent-Hunyuan/fix/single-runtime-quit
fix(memhub): one runtime; App quit never stops the background service, stops every runtime when it is disabled
2026-11-02 14:17:32 +08:00
alvinfeiandClaude Opus 5.5 384c7476ad fix(memhub): adopt a newer runtime silently and let quit follow the service state
The owner's decision for a runtime with a higher build than the App: startup
neither fails nor shows anything. The App adopts the runtime and never stops,
downgrades or duplicates it at startup. Quit gives it no special pass, so it
follows the same rules as any adopted runtime: preserved when the service is
enabled or its state is unknown, stopped when the service is disabled. The
now-unused "后台版本比 App 新" message is removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:07:57 +08:00
alvinfeiandClaude Opus 5.5 71b8f9be24 Merge github/main into fix/deepseek-harness-profiles
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:04:40 +08:00
feitianxiang d257188d09 Merge pull request #270 from Tencent-Hunyuan/fix/commit-validation-and-zh-language
fix(memhub): one commit format at the packaged-runtime boundary; zh-Hans UI sets the SDK language
2026-11-02 14:03:21 +08:00
feitianxiang bde0220373 Merge pull request #268 from Tencent-Hunyuan/feat/sdk-intention-dedup-switch
feat(sdk): configurable intention dedup (on by default, unchanged behaviour)
2026-11-02 14:03:04 +08:00
alvinfeiandClaude Opus 5.5 a88d227cc0 fix(memhub): one DeepSeek Harness switch for every detected profile
The app page assumed one source maps to one host. Harness has none without a
profile, so the switch was greyed out with `unsupported_host` and an installed
profile stayed `install-only` with no way to approve it (#230).

The one switch now covers every `deepseek-harness:<profile>` through the
existing install, grant, revoke and uninstall APIs. On installs and grants each
profile that is not approved, off removes and revokes each linked one. A
profile that fails is named and the others keep their result. The page lists
each profile with its state.

New profiles are connected while the switch is on. The desktop re-reads the
profile list on each host refresh, and the remembered apply gets one more
attempt per new profile. The Runtime re-reads the disk for an unknown profile
id: its list was fixed at start and it outlives the App, so one new profile
made the host list fail for every app.

Rewrite the harness doc to match, removing the sources page it described,
which is not mounted. Add the strings to all 11 locales.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 13:11:19 +08:00
alvinfeiandClaude Opus 5.5 059125a09f fix(memhub): one runtime; App quit never stops the background service, stops every runtime when it is disabled
Quit no longer trusts what startup saw. `transientRuntime` and
`preserveManagedRuntimeOnQuit` were derived from the launch path, so an
already-running transient Runtime was treated as a managed service and
outlived the App (#237, #238, #239).

- queryRuntimeServiceState asks launchd/systemd at quit time, every time.
  Enabled: a LaunchAgent declaring this data dir is installed, there is no
  `launchctl disable` override, and the enable marker ("Launch at login")
  is present. A failed or timed-out query is `unknown`.
- Quit: enabled -> never stop. Disabled -> stopAllRuntimeProcesses stops
  every Runtime of the data dir (App child, CLI-started, leftover), each
  pid verified via the receipt and the authenticated identity endpoint; an
  unverifiable lease owner is reported, never signalled. Unknown -> stop
  nothing and log it.
- Startup reconciles a running Runtime by build: the Runtime now publishes
  build_number/git_commit in runtime-service.json. Same build is adopted;
  an older service job is replaced through launchd (installUserService
  kickstart/re-bootstrap), an older transient/CLI one is stopped gracefully
  and restarted; a newer one is never downgraded or stopped; an unknown
  build is adopted and never stopped. A stray beside an enabled service is
  stopped so exactly one Runtime remains.
- The tray and the login preference re-read the real state on each refresh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:57:21 +08:00
alvinfeiandClaude Opus 5.5 0d0cefb25e fix(core): leave no cursor on a DeepSeek Harness session it refuses
An unknown session version was skipped record by record. Each skip carried a
cursor and the file ended with a cursor touch, so the refused session sat
behind a checkpoint: the release that learned the format would read nothing,
turning "not imported yet" into "never imported". The skip was also not
reported, since `report_issue` defaults to false.

Refuse the file from its header instead, with the cursorless
`ScanFailure(UNSUPPORTED_SCHEMA)` other sources already use. The parser check
stays for callers that hand it records directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:55:17 +08:00
alvinfeiandClaude Opus 5.5 be8cf028ff test(core): pin DeepSeek Harness text tool output and session name boundary
Cover the #175 tool-output half directly: a `tool/result` written as plain
`text` blocks imports next to the older `tool-result` block shape, goes
through the same credential scrubbing and clipping, and names outside
`session[.vN].jsonl[.zstd]` stay out. Correct the contract-suite comments that
still said only `session.jsonl` was accepted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:47:17 +08:00
alvinfeiandClaude Opus 5.5 ef5ff06590 fix(sdk): intention dedup switch also gates write-batch dedup
When intention_dedup_enabled is off, no intention dedup should run, but
the writer's batch dedup still dropped near-duplicate intentions from
one extraction. Gate it on the same switch. Its threshold stays on
MEMORY_DEDUP_THRESHOLD.

Also skip the embedding backfill when dedup is off, since only dedup
uses it, and fall back to the env default on an invalid configured
threshold so a bad value cannot fail the read.

Refs #241

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:33:56 +08:00
ryanlandClaude Opus 5.5 7eb38aa6ad fix(core): import versioned DeepSeek Harness session files
DSH names a session file after its format generation
(`session.v3.jsonl.zstd`, `session.v4.jsonl.zstd`), but discovery matched only
the two unversioned names, so import jobs completed with `discovered 0` while
the sessions sat on disk.

Discover `session[.vN].jsonl[.zstd]` and gate parsing on the header `version`
field: verified generations (0/3/4) import, anything newer is skipped as
`unsupported_schema` rather than interpreted with today's field rules. v4 tool
results use plain `text` blocks, so read that shape through the same clipping
and credential-scrubbing path as the `tool-result` blocks used by v0/v3.

(cherry picked from commit 2a65e5ab2d8548cfb8d6f72a4b68b702f6b4a3cc; conflicts in CHANGELOG.md and the Core test inventory resolved against main)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:32:02 +08:00
alvinfeiandClaude Opus 5.5 2624228f6f feat(sdk): configurable intention dedup (on by default, unchanged behaviour)
Add MemoryConfig.recall.intention_dedup_enabled (default true, env
MEMORY_INTENTION_DEDUP_ENABLED) and recall.intention_dedup_threshold
(env MEMORY_INTENTION_DEDUP_THRESHOLD, falling back to the shared
MEMORY_DEDUP_THRESHOLD, then 0.95). The reader passes both from the
client config to recall_intentions. When off, read-time intention dedup
is skipped entirely: nothing is hidden and nothing is deleted.

Refs #241

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:25:37 +08:00
136 changed files with 8063 additions and 1259 deletions

No files matched your search

+71
View File
@@ -4,6 +4,77 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
- MemHub multi-user hardening (follow-up to #250):
- Read responses name the user they were read for: `user_id` (the one
asked for, `local` by default) is now on `GET /api/v2/memory-library`,
`/memory-library/layers`, `/user-profile`, `/home-summary`,
`/memories/graph`, `/raw`, and `POST /api/v2/context/search`,
`/api/v1/recall`, `/api/v2/memories/ask`. `ask/stream` carries it on its
first (`recall`) event. The Host Bridge context validator accepts the new
field and still accepts a Core without it.
- `DELETE /api/v2/memories/{id}` answers 404 `memory_not_found` for another
user's memory and for an id that does not exist (the same response, so
existence is still not revealed), and also when the SDK finds nothing to
delete; 204 only when the caller's own memory was deleted. The App treats
that 404 as already gone (a double-click delete no longer shows an
error); `memhub memories delete` prints "未找到记忆 <id>(可能已删除)",
exits 0, and its JSON is `state: "committed"` with `already_gone: true`.
- Hermes never reads a `profiles/` directory at any depth (scan, detect and
live sync), so a root granted above `~/.hermes` cannot pick up another
user's profile. Connecting refuses a root that is the `profiles`
container itself (422 `hermes_root_is_profiles_container`) and a root
equal to, containing or inside another user's active Hermes grant (409
`hermes_root_overlaps_other_user`, symlinks resolved); `local` on
`~/.hermes` with another user on `~/.hermes/profiles/<name>` stays
allowed. The CLI explains both codes.
- `memhub sources connect --user-id` with an invalid id now says the user
id need not equal the Hermes profile name (e.g. `--user-id wife-work
--root ~/.hermes/profiles/wife:work`).
- MemHub's DeepSeek Harness app page can grant recall again (#230). The
"一键插入应用" switch used to be greyed out with `unsupported_host`, and a
profile installed from the CLI stayed `install-only` with no way to approve
it. The one switch now covers every detected profile
(`deepseek-harness:<profile>`): on installs each profile's MCP patch and
grants recall, off removes the patch and revokes the grant. The page lists
each profile with its state, and names any profile that failed while
keeping the others. While the switch is on, a profile created later is
installed and granted automatically, without restarting MemHub; the Runtime
also no longer fails every host read when it meets a profile created after
it started. The harness docs no longer describe an unmounted sources page.
- MemHub's `deepseek-harness` source imports session files whose names carry the
format generation: `session.v3.jsonl.zstd` and `session.v4.jsonl.zstd` are now
discovered alongside the original `session.jsonl[.zstd]`. Discovery accepts
any `session[.vN].jsonl[.zstd]` name and the header's `version` decides
whether the session is parsed (v0/v3/v4) or refused as `unsupported_schema`.
A refused file is reported and keeps no cursor, so the release that learns
its format reads it from the top.
v4 tool results, which are written as plain `text` blocks rather than
`tool-result` blocks, are read through the same clipping and credential
scrubbing path. Previously every versioned session was invisible: the import
job completed with `discovered 0` while the files sat on disk.
## Hy Memory 2.0.12 — 2026-11-02
- Intention dedup is now configurable per client
(`MemoryConfig.recall.intention_dedup_enabled`, default `true`;
`recall.intention_dedup_threshold`, default `MEMORY_DEDUP_THRESHOLD` or
`0.95`). Env defaults: `MEMORY_INTENTION_DEDUP_ENABLED`,
`MEMORY_INTENTION_DEDUP_THRESHOLD`. When on, behaviour is unchanged
(duplicates found at read time are still deleted); when off, no intention
dedup runs, at read time or within a write batch, and nothing is hidden,
dropped or deleted. The threshold applies to read-time dedup only. Refs #241.
- MemHub pins `hy-memory==2.0.12` and exposes that intention dedup switch in
its memory-features setting (`GET`/`PUT /api/v1/settings/memory-features`,
`memhub settings features`): the feature `intention_dedup` (default on) and
the numeric `intention_dedup_threshold` (default 0.95, accepted 0.8–1.0,
anything else is 422 or a CLI usage error). Unset, both keep the SDK
default, so nothing changes for the App; a change rebuilds the SDK clients
like the other feature switches. The API answer gains `parameters`, and
`pipeline.features` can now carry the threshold as a number. Refs #241.
## MemHub 1.0.6 — 2026-11-02
Build 37, on `hy-memory==2.0.11`. Linux x64/arm64 Runtime and macOS arm64
+20
View File
@@ -182,9 +182,29 @@ SDK 不设置 Rerank Search 并发上限。开启后模型在 `HyMemoryClient`
| `MEMORY_RECALL_STRENGTH_TAU` | `180` | 衰减时间常数,天 |
| `MEMORY_RECALL_ACCESS_TRACKING` | `true` | Strength 开启时是否回写访问信息 |
| `MEMORY_ENTITY_STORE_ENABLED` | `false` | 是否维护独立 entity collection;当前 Reader 不直接读取 |
| `MEMORY_INTENTION_DEDUP_ENABLED` | `true` | 是否对 L7 Intention 判重(读侧和写入侧批内);见下文 |
| `MEMORY_INTENTION_DEDUP_THRESHOLD` | `MEMORY_DEDUP_THRESHOLD`,再缺省为 `0.95` | Intention 读侧判重的 cosine 阈值 |
旧数据可调用当前公开方法 `client.build_entity_store(user_id)` 补建 Entity Store。
Intention 判重:开启时(默认,与以往行为一致),Search 召回的 L7 Intention 中,
同 `user_id` + 同 `agent_id` + `memory_at` 同一天 + `valid_until` 相同、且 cosine ≥ 阈值的
条目视为复本,只返回最早写入的一条,其余在后台从向量库**删除**。判重不区分内容中的项目、
对象等语义差异,内容高度相似但对象不同的意图可能被误删;写入时同一批抽取结果也会做一次
批内判重。关闭后这两处都不判重,不隐藏、不丢弃、也不删除意图。两项对应
`MemoryConfig.recall.intention_dedup_enabled` /
`recall.intention_dedup_threshold`,按 client 生效,环境变量只提供默认值:
```python
client = HyMemoryClient.from_config({
"recall": {"intention_dedup_enabled": False},
})
```
`MEMORY_DEDUP_THRESHOLD` 同时被写入侧批内意图判重和 dedup 内核使用;
`MEMORY_INTENTION_DEDUP_THRESHOLD` 只影响读侧 Intention 判重,写入侧批内判重的阈值
仍取 `MEMORY_DEDUP_THRESHOLD`。
## 日志、Trace 与历史
| 变量 | 默认值 | 说明 |
+19
View File
@@ -432,6 +432,14 @@ class RecallConfig:
retrieval_clauses_require_keyword: bool = None # env: MEMORY_RETRIEVAL_CLAUSES_REQUIRE_KEYWORD
retrieval_clauses_min_keyword_score: float = None # normalized BM25 floor
supplemental_propositions_enabled: bool = None # env: MEMORY_SUPPLEMENTAL_PROPOSITIONS_ENABLED
# ── Intention(L7)判重 ──
# 开启(默认):召回时同 user+agent+同天 memory_at+同 valid_until 且
# cosine≥阈值 的意图视为复本,只返回最早一条,其余异步从 VDB 删除(历史行为)。
# 关闭:读侧与写入侧批内都不判重——不隐藏、不丢弃、不删除任何意图。
intention_dedup_enabled: bool = None # env: MEMORY_INTENTION_DEDUP_ENABLED
# 读侧意图判重 cosine 阈值。env: MEMORY_INTENTION_DEDUP_THRESHOLD,
# 未设置时回退到共享的 MEMORY_DEDUP_THRESHOLD(兼容旧部署),默认 0.95。
intention_dedup_threshold: float = None
# 最小分数阈值
min_score_threshold: float = 0.0
@@ -750,6 +758,15 @@ class RecallConfig:
self.supplemental_propositions_enabled = _get_env_bool(
"MEMORY_SUPPLEMENTAL_PROPOSITIONS_ENABLED", False
)
if self.intention_dedup_enabled is None:
self.intention_dedup_enabled = _get_env_bool(
"MEMORY_INTENTION_DEDUP_ENABLED", True
)
if self.intention_dedup_threshold is None:
self.intention_dedup_threshold = _get_env_float(
"MEMORY_INTENTION_DEDUP_THRESHOLD",
_get_env_float("MEMORY_DEDUP_THRESHOLD", 0.95),
)
@dataclass
@@ -1384,6 +1401,8 @@ class MemoryConfig:
"supplemental_propositions_enabled": (
self.recall.supplemental_propositions_enabled
),
"intention_dedup_enabled": self.recall.intention_dedup_enabled,
"intention_dedup_threshold": self.recall.intention_dedup_threshold,
},
"pipeline": {
"default_version": self.pipeline.default_version,
+31 -10
View File
@@ -15,6 +15,9 @@ HY Memory - Intention (L7) 召回 + 惰性过期转换 + 复本判定
→ 视为同一个意图的复本(extractor 复读机 / 多 session 重复表达)。
保留组里 `gmt_created` 最早的(老的先来先得,稳定 id);
- **异步删库**:其余复本 fire-and-forget 从 VDB 删除,不影响本次调用 RT。
- **开关**:判重受 `MemoryConfig.recall.intention_dedup_enabled`(默认开)与
`recall.intention_dedup_threshold` 控制,由 reader 按 client 配置传入。
关闭时完全跳过判重:不隐藏、不删除任何意图(写入侧批内判重同样跳过)。
Q: 为什么要看 memory_at 和 valid_until?
A: 意图是 point-in-time 信号。同一句"我周五要开会",如果一条 memory_at 在
@@ -38,8 +41,14 @@ from ...models.memory import MemoryLayer, MemoryStatus
logger = logging.getLogger(__name__)
def _get_dedup_threshold() -> float:
"""去重 cosine 阈值:沿用 dedup 内核的 MEMORY_DEDUP_THRESHOLD,默认 0.95。"""
def _get_dedup_threshold(configured: Optional[float] = None) -> float:
"""去重 cosine 阈值:优先用 client 配置;否则(或配置非法)沿用 dedup 内核的
MEMORY_DEDUP_THRESHOLD,默认 0.95。"""
try:
if configured is not None:
return float(configured)
except (TypeError, ValueError):
logger.warning(f"[intention] invalid dedup threshold: {configured!r}")
try:
return float(os.environ.get("MEMORY_DEDUP_THRESHOLD", 0.95))
except (TypeError, ValueError):
@@ -244,6 +253,8 @@ async def recall_intentions(
stats_out: Optional[Dict[str, Any]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
dedup_enabled: bool = True,
dedup_threshold: Optional[float] = None,
) -> List[Dict[str, Any]]:
"""
召回未过期的 L7_INTENTION 节点(VDB),惰性过期转 L2_FACT,判重并异步删复本。
@@ -260,6 +271,11 @@ async def recall_intentions(
now: 当前时间(默认 datetime.now(),测试可注入)
stats_out: 可选 dict,本函数把分段耗时与数量**原地回填**进去,供调用方落
READ_INTENTION pipeline log。传 None 则完全不统计(零开销)。
dedup_enabled: 是否执行读侧判重(及复本异步删库)。False 时跳过判重,
所有未过期意图原样返回,不删除任何节点。默认 True(历史行为)。
dedup_threshold: 判重 cosine 阈值。None 时回退到 env
MEMORY_DEDUP_THRESHOLD(默认 0.95),兼容直接调用本函数的旧代码;
reader 总是传入 `recall.intention_dedup_threshold`。
Returns:
[{"node_id", "score", "node", "layer": "l7_intention", "source": "vdb_intention"}, ...]
@@ -272,7 +288,7 @@ async def recall_intentions(
埋点字段(回填进 stats_out):
耗时(ms): vdb_search_ms / expire_ms / backfill_embed_ms / dedup_ms / total_ms
数量: fetched / expired / survivors / dedup_dropped / returned
配置: fetch_limit / out_limit / dedup_threshold
配置: fetch_limit / out_limit / dedup_enabled / dedup_threshold
说明: expire_ms、backfill_embed_ms 只有真的发生了对应操作才非 0,
借此可判断慢在 VDB 检索、逐条过期转换、补拉向量还是判重计算。
"""
@@ -358,7 +374,7 @@ async def recall_intentions(
_t = time.perf_counter()
_backfilled = 0
_missing_cnt = 0
if survivors:
if survivors and dedup_enabled: # 补拉只为判重;关闭时无需这次 VDB 读
missing_ids = [
h.get("node_id") for h in survivors
if h.get("node_id") and not getattr(h.get("node"), "embedding", None)
@@ -391,18 +407,23 @@ async def recall_intentions(
)
# 2. 判重:同 user + 同天 memory_at + valid_until 同日 + cosine≥threshold
# dedup_enabled=False → 完全跳过:不隐藏、不删除。
_t = time.perf_counter()
threshold = _get_dedup_threshold()
try:
kept, delete_ids = _partition_duplicates(survivors, threshold)
except Exception as e:
# 任何 dedup 异常 → 降级为不去重(返回全部 survivors 截断 limit)
logger.warning(f"[intention] dedup partition failed, degrade to raw: {e}")
threshold = _get_dedup_threshold(dedup_threshold)
if dedup_enabled:
try:
kept, delete_ids = _partition_duplicates(survivors, threshold)
except Exception as e:
# 任何 dedup 异常 → 降级为不去重(返回全部 survivors 截断 limit)
logger.warning(f"[intention] dedup partition failed, degrade to raw: {e}")
kept, delete_ids = survivors, []
else:
kept, delete_ids = survivors, []
# 纯 CPU(cosine 两两比较);survivors 多时可能显著
_stat(
dedup_ms=round((time.perf_counter() - _t) * 1000, 2),
dedup_dropped=len(delete_ids),
dedup_enabled=bool(dedup_enabled),
dedup_threshold=threshold,
)
+6
View File
@@ -1856,6 +1856,12 @@ class BaseReadPipeline(ReadPipeline):
stats_out=_intent_stats,
include_sources=request.include_sources,
exclude_sources=request.exclude_sources,
dedup_enabled=bool(getattr(
self.config.recall, "intention_dedup_enabled", True
)),
dedup_threshold=getattr(
self.config.recall, "intention_dedup_threshold", None
),
)
await trace_log.log_intention(
stats=_intent_stats,
+6 -1
View File
@@ -635,6 +635,7 @@ class MemoryWriter(WritePipeline):
不查 VDB、不做"新写入 vs 库里已有"的对比,避免给 VDB 增加读压力。
库内存量污染由 recall 侧异步删除逐步清理。best-effort:任何异常都
回退到"不判重、全写入",绝不影响 upsert 主流程。
`recall.intention_dedup_enabled=False` 时跳过批内判重,全部写入。
"""
if not intentions:
return []
@@ -660,7 +661,11 @@ class MemoryWriter(WritePipeline):
# ============ 批内 dedup(不查 VDB,只对本批 intentions 互斥判重) ============
keep_mask: List[bool] = [True] * len(intentions)
try:
dropped = self._intention_batch_dedup(intentions, embeddings)
dropped = (
self._intention_batch_dedup(intentions, embeddings)
if getattr(self.config.recall, "intention_dedup_enabled", True)
else []
)
for i in dropped:
if 0 <= i < len(keep_mask):
keep_mask[i] = False
+9
View File
@@ -137,6 +137,15 @@ memhub profile
`user-profile` / `memhub profile` 调用 SDK `get_user_profile()`,返回组装后的
用户长期画像,不通过普通 Search 拼装。
读接口都回显 `user_id`(实际读的用户,未指定时为 `local`):
`GET /api/v2/memory-library`、`/memory-library/layers`、`/user-profile`、
`/user-profile/versions`、`/home-summary`、`/memories/graph`、`/raw`,
`POST /api/v2/context/search`、`/api/v1/recall`、`/api/v2/memories/ask`;
`/memories/ask/stream` 写在第一个 `recall` 事件上。
`DELETE /api/v2/memories/{id}` 只在删掉调用者自己的记忆时回 204;另一个用户的
记忆和不存在的 ID 一律回 404 `memory_not_found`(不暴露是否存在),调用方按
「已不存在」处理。
返回值:
```python
+2 -2
View File
@@ -18,7 +18,7 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.11`。MemHub 源码版本为 `1.0.6`
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.12`。MemHub 源码版本为 `1.0.6`
(build 37),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
@@ -118,7 +118,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.11`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.12`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+1 -1
View File
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.11",
"versionInfo": "2.0.12",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+1 -1
View File
@@ -9,7 +9,7 @@ Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.11
## hy-memory 2.0.12
License: Apache-2.0
+29 -13
View File
@@ -1938,7 +1938,7 @@ async function runCommand(
return reject(
descriptor,
asJson,
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。",
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。用户 ID 不必等于 Hermes profile 名,例如 --user-id wife-work --root ~/.hermes/profiles/wife:work。",
);
}
if (!parsed) return usage();
@@ -2328,29 +2328,41 @@ async function runCommand(
...userIdParameters(userId),
}),
);
// A delete answered 404 `memory_not_found` is already gone (deleted
// before, or never this user's): what was asked for holds, so it exits 0
// as committed and says so, instead of failing a retry or a double run.
const alreadyGone =
parsed.kind === "delete" &&
outcome.state === "rejected" &&
outcome.code === "memory_not_found";
const state = alreadyGone ? "committed" : outcome.state;
const document = {
schema_version: 2,
product: PRODUCT,
command: `memories ${parsed.kind}`,
state: outcome.state,
state,
operation: parsed.kind,
memory_id: parsed.memoryId,
...(outcome.state === "committed"
? parsed.kind === "update"
? { result: outcome.data }
: {}
: { code: outcome.code }),
...(alreadyGone
? { already_gone: true }
: outcome.state === "committed"
? parsed.kind === "update"
? { result: outcome.data }
: {}
: { code: outcome.code }),
};
emit(
asJson
? `${JSON.stringify(document)}\n`
: outcome.state === "committed"
? `已${parsed.kind === "update" ? "更新" : "删除"}记忆 ${parsed.memoryId}\n`
: outcome.code === "memory_id_ambiguous"
? "这个 ID 同时存在于多个场景;请加 --kind(以及 --variant)指明是哪一条。\n"
: `操作失败:${outcome.code}\n`,
: alreadyGone
? `未找到记忆 ${parsed.memoryId}(可能已删除),无需再删。\n`
: outcome.state === "committed"
? `已${parsed.kind === "update" ? "更新" : "删除"}记忆 ${parsed.memoryId}\n`
: outcome.code === "memory_id_ambiguous"
? "这个 ID 同时存在于多个场景;请加 --kind(以及 --variant)指明是哪一条。\n"
: `操作失败:${outcome.code}\n`,
);
return outcome.state === "committed" ? EXIT_OK : EXIT_FAILED;
return state === "committed" ? EXIT_OK : EXIT_FAILED;
}
if (
named === "settings" &&
@@ -4371,6 +4383,10 @@ function renderWriteReceiptText(receipt: MemoryWriteReceipt): string {
...(receipt.memory_ids && receipt.memory_ids.length > 0
? [`memory_ids: ${receipt.memory_ids.join(",")}`]
: []),
...(receipt.error_code ? [`error_code: ${receipt.error_code}`] : []),
...(receipt.next_retry_at
? [`next_retry_at: ${receipt.next_retry_at}`]
: []),
"",
].join("\n");
}
+6 -5
View File
@@ -258,7 +258,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "sources",
summary:
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(会自动排除 ~/.hermes/profiles/**,各用户互不读取)。",
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(任何层级的 profiles/ 目录都会被排除,各用户互不读取)。--root 不能是 profiles 目录本身,也不能与另一个用户的 Hermes 目录相同或互相包含(~/.hermes 与 ~/.hermes/profiles/<name> 除外)。",
usage:
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect hermes --user-id <id> --root <path> [--import] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
options: [
@@ -270,7 +270,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
flag: "--user-id <id>",
summary:
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;其他来源只能是 local。",
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;用户 ID 不必等于 profile 名,例如 --user-id wife-work --root ~/.hermes/profiles/wife:work。其他来源只能是 local。",
},
{
flag: "--root <path>",
@@ -397,7 +397,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "memories",
summary:
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes。",
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes;要删的记忆已不存在(或不属于该用户)时提示未找到,按已删除处理并退出 0。",
usage:
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--user-id <id>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--user-id <id>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add-status <request-id> [--user-id <id>] [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--user-id <id>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--user-id <id>] [--json]",
options: [
@@ -461,11 +461,12 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "settings",
summary:
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、few_shot 提取示例;set 需 --yes,不加时只显示计划;reset 恢复默认值。settings language 查看或设置界面语言(App 切换语言时会自动设置):简体中文(zh-Hans)和 system 不指定 SDK 语言,SDK 照旧自动识别中文或英文;其余语言(en、zh-Hant、ja、es、de、pt、fr、ko、ru、id)会让提取等 SDK 提示使用该语言,立即生效。",
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、intention_dedup 意图判重(默认开;关闭则检索与写入都不判重、不删除意图)、few_shot 提取示例,以及数值 intention_dedup_threshold 意图判重阈值(默认 0.95,可设 0.8–1.0);set 需 --yes,不加时只显示计划;reset 恢复默认值。settings language 查看或设置界面语言(App 切换语言时会自动设置):简体中文(zh-Hans)和 system 不指定 SDK 语言,SDK 照旧自动识别中文或英文;其余语言(en、zh-Hant、ja、es、de、pt、fr、ko、ru、id)会让提取等 SDK 提示使用该语言,立即生效。",
usage:
"memhub settings mode [lite|pro|ultra] [--yes] [--json]\n" +
" memhub settings features [--json]\n" +
" memhub settings features set <name>=<true|false> [...] [--yes] [--json]\n" +
" memhub settings features set intention_dedup_threshold=<0.8-1.0> [--yes] [--json]\n" +
" memhub settings features reset [<name> ...] [--json]\n" +
" memhub settings language [<code>|system] [--json]",
options: [
@@ -476,7 +477,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
flag: "features set <name>=<true|false>",
summary:
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。",
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。intention_dedup_threshold 取 0.8–1.0 的数值。",
},
{
flag: "features reset [<name> ...]",
+5
View File
@@ -268,6 +268,11 @@ export interface MemoryWriteReceipt {
detail?: string;
raw_ids?: string[];
memory_ids?: string[];
/** Stable code of the last failure while the write still waits; null when
* none. Optional so older Runtimes stay readable. */
error_code?: string | null;
/** When Core will try this write again (ISO); null when not scheduled. */
next_retry_at?: string | null;
}
export interface OpenAIMessage {
+13
View File
@@ -348,12 +348,25 @@ export function parseApiKey(bytes: Buffer | null): string | undefined {
return value;
}
/** What to do about a refusal whose code alone does not say. */
const FLOW_CODE_HINTS: Record<string, string> = {
hermes_root_is_profiles_container:
"--root 指向的是 profiles 目录本身;请指向其中一个 profile,例如 ~/.hermes/profiles/wife。",
hermes_root_overlaps_other_user:
"--root 与另一个用户已接入的 Hermes 目录相同或互相包含,会读到对方的记录;请换一个目录,或先撤销对方的接入。~/.hermes 与 ~/.hermes/profiles/<name> 可以分属两个用户。",
};
export function renderFlowText(document: FlowDocument): string {
const lines = [
`操作状态:${document.state}`,
`操作 ID:${document.operation_id ?? "无"}`,
"code" in document ? `代码:${document.code}` : "已返回有界回执。",
];
const hint =
"code" in document && typeof document.code === "string"
? FLOW_CODE_HINTS[document.code]
: undefined;
if (hint) lines.push(hint);
// A partial success or an all-rejected batch must name each source that was
// not accepted; a bare "partial" leaves the user with nothing to act on.
if (
+52
View File
@@ -233,6 +233,31 @@ describe("memories add-status", () => {
});
});
it("passes the retry code and time of an addition still waiting", async () => {
transport.get.mockResolvedValue({
status: 200,
data: {
...addReceipt("pending"),
error_code: "model_auth_failed",
next_retry_at: "2026-11-02T10:00:00+00:00",
},
});
const json = await capturedRun([
"memories",
"add-status",
REQUEST,
"--json",
]);
expect(JSON.parse(json.stdout).receipt).toMatchObject({
state: "pending",
error_code: "model_auth_failed",
next_retry_at: "2026-11-02T10:00:00+00:00",
});
const text = await capturedRun(["memories", "add-status", REQUEST]);
expect(text.stdout).toContain("error_code: model_auth_failed");
expect(text.stdout).toContain("next_retry_at: 2026-11-02T10:00:00+00:00");
});
it("reports an unknown request id as not_found, exit 1", async () => {
transport.get.mockResolvedValue({
status: 404,
@@ -366,6 +391,33 @@ describe("memories update / delete pass the namespace", () => {
);
});
it("reports a delete answered 404 as already gone and exits 0", async () => {
transport.delete.mockResolvedValue({
status: 404,
data: { detail: "memory_not_found" },
});
const json = await capturedRun([
"memories",
"delete",
"gone-1",
"--yes",
"--json",
]);
expect(json.code).toBe(0);
expect(JSON.parse(json.stdout)).toEqual({
schema_version: 2,
product: expect.any(String),
command: "memories delete",
state: "committed",
operation: "delete",
memory_id: "gone-1",
already_gone: true,
});
const text = await capturedRun(["memories", "delete", "gone-1", "--yes"]);
expect(text.code).toBe(0);
expect(text.stdout).toContain("未找到记忆 gone-1");
});
it("explains an ambiguous id instead of a bare code", async () => {
transport.delete.mockResolvedValue({
status: 409,
+173 -6
View File
@@ -36,15 +36,17 @@ const DEFAULTS: Record<string, boolean> = {
session_episodes: false,
episode_summary: false,
intention: true,
intention_dedup: true,
few_shot: false,
};
const THRESHOLD_DEFAULT = 0.95;
function featuresState(
configured: Record<string, boolean> = {},
configured: Record<string, boolean | number> = {},
mode = "pro",
extra: Record<string, unknown> = {},
) {
const nonDefault: Record<string, boolean> = {};
const nonDefault: Record<string, boolean | number> = {};
const features = Object.entries(DEFAULTS).map(([name, fallback]) => {
const enabled = configured[name] ?? fallback;
if (enabled !== fallback) nonDefault[name] = enabled;
@@ -70,7 +72,34 @@ function featuresState(
description: name,
};
});
return { mode, features, non_default: nonDefault, ...extra };
const threshold =
(configured.intention_dedup_threshold as number | undefined) ??
THRESHOLD_DEFAULT;
if (threshold !== THRESHOLD_DEFAULT)
nonDefault.intention_dedup_threshold = threshold;
const dedupOn =
(configured.intention_dedup as boolean | undefined) ??
DEFAULTS.intention_dedup;
const parameters = [
{
name: "intention_dedup_threshold",
value: threshold,
configured:
"intention_dedup_threshold" in configured
? configured.intention_dedup_threshold
: null,
default: THRESHOLD_DEFAULT,
minimum: 0.8,
maximum: 1,
active: dedupOn,
reason: dedupOn ? null : "requires_intention_dedup",
requires_feature: "intention_dedup",
sdk_field: "recall.intention_dedup_threshold",
env: "MEMORY_INTENTION_DEDUP_THRESHOLD",
description: "threshold",
},
];
return { mode, features, parameters, non_default: nonDefault, ...extra };
}
beforeEach(() => {
@@ -110,11 +139,16 @@ describe("memhub settings features", () => {
expect(result.code).toBe(0);
const json = JSON.parse(result.stdout);
expect(json.state).toBe("ok");
expect(json.features.features).toHaveLength(7);
expect(json.features.features).toHaveLength(8);
expect(json.features.parameters[0].name).toBe("intention_dedup_threshold");
expect(transport.put).not.toHaveBeenCalled();
const text = (await capturedRun(["settings", "features"])).stdout;
expect(text).toContain("graph_rerank:关(默认)");
expect(text).toContain("intention:开(默认)");
expect(text).toContain("intention_dedup:开(默认)");
expect(text).toContain(
"intention_dedup_threshold:0.95(默认,可设 0.8–1)",
);
expect(text).toContain("先开启 session_episodes");
});
@@ -160,6 +194,130 @@ describe("memhub settings features", () => {
);
});
it("turns intention dedup off and sets its threshold, round trip", async () => {
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState({ intention_dedup: false }, "pro", {
changed: ["intention_dedup"],
}),
});
const off = await capturedRun([
"settings",
"features",
"set",
"intention_dedup=false",
"--yes",
"--json",
]);
expect(off.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup: false },
undefined,
);
const offJson = JSON.parse(off.stdout);
expect(offJson.state).toBe("committed");
expect(offJson.changed).toEqual(["intention_dedup"]);
expect(offJson.features.parameters[0].reason).toBe(
"requires_intention_dedup",
);
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ intention_dedup: false }),
});
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState(
{ intention_dedup: true, intention_dedup_threshold: 0.9 },
"pro",
{ changed: ["intention_dedup", "intention_dedup_threshold"] },
),
});
const on = await capturedRun([
"settings",
"features",
"set",
"intention_dedup=true",
"intention_dedup_threshold=0.9",
"--yes",
]);
expect(on.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup: true, intention_dedup_threshold: 0.9 },
undefined,
);
expect(on.stdout).toContain(
"已更新:intention_dedup、intention_dedup_threshold",
);
expect(on.stdout).toContain("intention_dedup_threshold:0.9(已设置");
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ intention_dedup_threshold: 0.9 }),
});
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState({}, "pro", {
changed: ["intention_dedup_threshold"],
}),
});
const reset = await capturedRun([
"settings",
"features",
"reset",
"intention_dedup_threshold",
]);
expect(reset.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup_threshold: null },
undefined,
);
});
it("previews a threshold without --yes and refuses one out of range", async () => {
const preview = await capturedRun([
"settings",
"features",
"set",
"intention_dedup_threshold=0.85",
]);
expect(preview.code).toBe(0);
expect(preview.stdout).toContain("- intention_dedup_threshold = 0.85");
for (const value of ["0.5", "0.79", "1.01", "2", "-0.9", "abc", "1e0"]) {
const result = await capturedRun([
"settings",
"features",
"set",
`intention_dedup_threshold=${value}`,
"--yes",
]);
expect(result.code, value).toBe(2);
}
expect(transport.put).not.toHaveBeenCalled();
});
it("does not send a threshold to a Runtime that does not list it", async () => {
const { parameters: _unused, ...older } = featuresState();
transport.get.mockResolvedValue({ status: 200, data: older });
const result = await capturedRun([
"settings",
"features",
"set",
"intention_dedup_threshold=0.9",
"--yes",
"--json",
]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout)).toMatchObject({
state: "rejected",
code: "memory_feature_unsupported",
});
expect(transport.put).not.toHaveBeenCalled();
});
it("does nothing when the values are already set", async () => {
transport.get.mockResolvedValue({
status: 200,
@@ -210,6 +368,8 @@ describe("memhub settings features", () => {
["settings", "features", "set", "graph_rerank=yes"],
["settings", "features", "set", "graph_rerank"],
["settings", "features", "set", "few_shot=true", "few_shot=false"],
["settings", "features", "set", "intention_dedup=0.9"],
["settings", "features", "set", "intention_dedup_threshold=true"],
["settings", "features", "reset", "nope"],
["settings", "features", "toggle"],
]) {
@@ -269,7 +429,11 @@ describe("status pipeline features", () => {
last_evolve_at: null,
graph_build: { state: "idle", users: {} },
evolve: { mode: "pro", schedule: null, running: [], users: {} },
features: { episode_recall: true, intention: false },
features: {
episode_recall: true,
intention: false,
intention_dedup_threshold: 0.9,
},
},
};
const report = await composeStatusReport({
@@ -284,9 +448,12 @@ describe("status pipeline features", () => {
expect(report.pipeline?.features).toEqual({
episode_recall: true,
intention: false,
intention_dedup_threshold: 0.9,
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain("SDK 功能:episode_recall=开、intention=关");
expect(text).toContain(
"SDK 功能:episode_recall=开、intention=关、intention_dedup_threshold=0.9",
);
});
});
+118 -28
View File
@@ -2,7 +2,8 @@
* `memhub settings features [set <name>=<true|false> ... | reset [<name> ...]]`.
*
* The SDK's optional retrieval and extraction features (graph rerank, episode
* recall, the episode write side, intention, few-shot). One global setting in
* recall, the episode write side, intention and its dedup, few-shot), plus the
* intention dedup threshold, the one numeric setting. One global setting in
* the Runtime; an unset feature keeps the SDK default, so an install nobody
* touched behaves as before. `set` prints its plan first and needs `--yes`,
* like `settings mode`, because some features spend model calls; `reset`
@@ -20,10 +21,19 @@ export const MEMORY_FEATURES = [
"session_episodes",
"episode_summary",
"intention",
"intention_dedup",
"few_shot",
] as const;
export type MemoryFeature = (typeof MEMORY_FEATURES)[number];
/** Numeric settings, with the range the Runtime accepts (inclusive). */
export const MEMORY_PARAMETERS = {
intention_dedup_threshold: { minimum: 0.8, maximum: 1 },
} as const;
export type MemoryParameter = keyof typeof MEMORY_PARAMETERS;
export type MemorySetting = MemoryFeature | MemoryParameter;
type SettingValues = Partial<Record<MemorySetting, boolean | number>>;
const FEATURE_LABEL: Record<MemoryFeature, string> = {
graph_rerank: "图谱重排(沿知识图谱的边传播支持度;仅 ultra)",
same_series_timeline:
@@ -33,6 +43,8 @@ const FEATURE_LABEL: Record<MemoryFeature, string> = {
session_episodes: "片段写入(同一会话提取的记忆归入同一片段)",
episode_summary: "片段摘要(片段结束时总结一次;每个片段一次模型调用)",
intention: "意图提取(随事实一起提取用户的计划与意图)",
intention_dedup:
"意图判重(检索时把同一天、相似度达到阈值的意图视为重复,只返回最早一条并删除其余;关闭则不判重、不删除)",
few_shot: "提取示例(在提取提示里附上示例;不加调用,输入更长)",
};
@@ -41,21 +53,42 @@ const REASON_TEXT: Record<string, string> = {
requires_pro: "lite 模式下不生效(需要 pro 或 ultra)",
requires_ultra: "当前模式下不生效(需要 ultra)",
requires_graph_rerank: "需要先开启并生效 graph_rerank",
requires_intention_dedup: "intention_dedup 已关闭,阈值不起作用",
};
const PARAMETER_LABEL: Record<MemoryParameter, string> = {
intention_dedup_threshold: "意图判重阈值(余弦相似度,越高保留越多)",
};
export type FeaturesCommand =
| { kind: "show" }
| {
kind: "set";
values: Partial<Record<MemoryFeature, boolean>>;
values: SettingValues;
confirmed: boolean;
}
| { kind: "reset"; names: MemoryFeature[] };
| { kind: "reset"; names: MemorySetting[] };
function isFeature(value: string): value is MemoryFeature {
return (MEMORY_FEATURES as readonly string[]).includes(value);
}
function isParameter(value: string): value is MemoryParameter {
return Object.hasOwn(MEMORY_PARAMETERS, value);
}
function isSetting(value: string): value is MemorySetting {
return isFeature(value) || isParameter(value);
}
/** A parameter's value: a plain decimal in its range; null = usage. */
function parameterValue(name: MemoryParameter, raw: string): number | null {
if (!/^\d+(\.\d+)?$/.test(raw)) return null;
const value = Number(raw);
const { minimum, maximum } = MEMORY_PARAMETERS[name];
return value >= minimum && value <= maximum ? value : null;
}
/** Arguments after `settings features`; null = usage. */
export function parseFeaturesCommand(
values: readonly string[],
@@ -68,18 +101,28 @@ export function parseFeaturesCommand(
const positional = tail.filter((value) => value !== "--yes");
if (verb === "set") {
if (positional.length === 0) return null;
const parsed: Partial<Record<MemoryFeature, boolean>> = {};
const parsed: SettingValues = {};
for (const item of positional) {
const match = /^([a-z_]+)=(true|false)$/.exec(item);
if (!match || !isFeature(match[1]) || match[1] in parsed) return null;
parsed[match[1]] = match[2] === "true";
const match = /^([a-z_]+)=(.+)$/.exec(item);
if (!match || match[1] in parsed) return null;
const [, name, raw] = match;
if (isFeature(name)) {
if (raw !== "true" && raw !== "false") return null;
parsed[name] = raw === "true";
} else if (isParameter(name)) {
const value = parameterValue(name, raw);
if (value === null) return null;
parsed[name] = value;
} else {
return null;
}
}
return { kind: "set", values: parsed, confirmed: yes === 1 };
}
if (verb === "reset") {
if (new Set(positional).size !== positional.length) return null;
if (!positional.every(isFeature)) return null;
return { kind: "reset", names: positional as MemoryFeature[] };
if (!positional.every(isSetting)) return null;
return { kind: "reset", names: positional as MemorySetting[] };
}
return null;
}
@@ -96,11 +139,26 @@ export interface FeatureState {
description: string;
}
export interface ParameterState {
name: MemoryParameter;
value: number;
configured: number | null;
default: number;
minimum: number;
maximum: number;
active: boolean;
reason: string | null;
requires_feature: string;
description: string;
}
export interface FeaturesState {
mode: string;
features: FeatureState[];
non_default: Partial<Record<MemoryFeature, boolean>>;
changed?: MemoryFeature[];
/** Absent from a Runtime older than the intention dedup threshold. */
parameters?: ParameterState[];
non_default: SettingValues;
changed?: MemorySetting[];
}
function featuresState(value: unknown): FeaturesState | null {
@@ -125,6 +183,20 @@ function featuresState(value: unknown): FeaturesState | null {
return null;
}
}
if (item.parameters !== undefined) {
if (!Array.isArray(item.parameters)) return null;
for (const parameter of item.parameters) {
if (
!parameter ||
typeof parameter !== "object" ||
typeof (parameter as ParameterState).name !== "string" ||
typeof (parameter as ParameterState).value !== "number" ||
typeof (parameter as ParameterState).active !== "boolean"
) {
return null;
}
}
}
return item as unknown as FeaturesState;
}
@@ -137,12 +209,12 @@ export type FeaturesDocument = {
| {
state: "preview";
features: FeaturesState;
target: Partial<Record<MemoryFeature, boolean>>;
target: SettingValues;
applied: false;
next_step: string;
}
| { state: "unchanged"; features: FeaturesState }
| { state: "committed"; features: FeaturesState; changed: MemoryFeature[] }
| { state: "committed"; features: FeaturesState; changed: MemorySetting[] }
| { state: "rejected" | "unknown" | "unavailable"; code: string }
);
@@ -170,13 +242,23 @@ export async function runFeaturesCommand(
if (command.kind === "show") {
return { ...BASE, state: "ok", features: current };
}
// What a person set, per name this Runtime lists; null = the default.
const configured = new Map<string, boolean | number | null>([
...current.features.map(
(feature) => [feature.name, feature.configured] as const,
),
...(current.parameters ?? []).map(
(parameter) => [parameter.name, parameter.configured] as const,
),
]);
let body: Record<string, unknown>;
if (command.kind === "set") {
const configured = new Map(
current.features.map((feature) => [feature.name, feature.configured]),
);
if (Object.keys(command.values).some((name) => !configured.has(name))) {
// An older Runtime does not know the name; it would answer 422.
return { ...BASE, state: "rejected", code: "memory_feature_unsupported" };
}
const differs = Object.entries(command.values).some(
([name, value]) => configured.get(name as MemoryFeature) !== value,
([name, value]) => configured.get(name) !== value,
);
if (!differs) return { ...BASE, state: "unchanged", features: current };
if (!command.confirmed) {
@@ -197,16 +279,10 @@ export async function runFeaturesCommand(
const names =
command.names.length > 0
? command.names
: current.features
.filter((feature) => feature.configured !== null)
.map((feature) => feature.name);
if (
names.every(
(name) =>
current.features.find((feature) => feature.name === name)
?.configured === null,
)
) {
: [...configured]
.filter(([, value]) => value !== null)
.map(([name]) => name as MemorySetting);
if (names.every((name) => (configured.get(name) ?? null) === null)) {
return { ...BASE, state: "unchanged", features: current };
}
body = Object.fromEntries(names.map((name) => [name, null]));
@@ -233,6 +309,16 @@ export function featuresExitCode(document: FeaturesDocument): number {
: 1;
}
function parameterLine(parameter: ParameterState): string {
const label = PARAMETER_LABEL[parameter.name] ?? parameter.name;
const origin = parameter.configured === null ? "默认" : "已设置";
const activity =
!parameter.active && parameter.reason
? `,${REASON_TEXT[parameter.reason] ?? parameter.reason}`
: "";
return `${parameter.name}:${parameter.value}(${origin}${activity},可设 ${parameter.minimum}–${parameter.maximum})— ${label}`;
}
function featureLine(feature: FeatureState): string {
const label = FEATURE_LABEL[feature.name] ?? feature.name;
const value = feature.enabled ? "开" : "关";
@@ -251,6 +337,8 @@ export function renderFeaturesText(document: FeaturesDocument): string {
const state = document.features;
const lines = [`当前模式:${state.mode}`];
for (const feature of state.features) lines.push(featureLine(feature));
for (const parameter of state.parameters ?? [])
lines.push(parameterLine(parameter));
lines.push(
"片段召回(episode_recall)只对带片段的记忆有用:先开启 session_episodes,之后提取的记忆才会带片段。",
);
@@ -258,7 +346,9 @@ export function renderFeaturesText(document: FeaturesDocument): string {
if (document.state === "preview") {
lines.push("将设置:");
for (const [name, value] of Object.entries(document.target)) {
lines.push(`- ${name} = ${value ? "开" : "关"}`);
lines.push(
`- ${name} = ${typeof value === "number" ? value : value ? "开" : "关"}`,
);
}
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
}
+10
View File
@@ -248,6 +248,10 @@ function addReceipt(receipt: MemoryAddReceipt): Record<string, unknown> {
task_id: receipt.task_id,
memory_ids: receipt.memory_ids,
error: receipt.error,
error_code:
typeof receipt.error_code === "string" ? receipt.error_code : null,
next_retry_at:
typeof receipt.next_retry_at === "string" ? receipt.next_retry_at : null,
};
}
@@ -347,6 +351,12 @@ export function renderMemoryAddText(document: MemoryAddDocument): string {
...((document.receipt.memory_ids as string[]).length > 0
? [`memory_ids: ${(document.receipt.memory_ids as string[]).join(",")}`]
: []),
...(document.receipt.error_code
? [`error_code: ${document.receipt.error_code}`]
: []),
...(document.receipt.next_retry_at
? [`next_retry_at: ${document.receipt.next_retry_at}`]
: []),
document.command === "memories add"
? `查询进度:memhub memories add-status ${document.request_id};重试请带上同一个 --request-id,不会重复添加。`
: "",
+70
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import { composeDoctorReport, renderDoctorText } from "./doctor-report.js";
import {
composeStatusReport,
@@ -41,6 +42,75 @@ describe("status default output", () => {
});
});
describe("status on a paused model", () => {
const processing = (backoff: {
llm_backoff_until: string | null;
llm_backoff_error: string | null;
}) => ({
pending: 4,
fresh_pending: 4,
backlog_pending: 0,
outside_window_pending: 0,
running: 0,
waiting_for_model: 0,
completed: 1,
failed: 0,
blocked_stages: [],
suspended: 2,
...backoff,
});
const reader =
(summary: unknown): CoreReader =>
async <T>(path: string) =>
(path === "/api/v1/memory-processing"
? { state: "ok", data: summary }
: { state: "unavailable", code: "x" }) as ReadOutcome<T>;
const compose = (summary: unknown) =>
composeStatusReport({
dataDir: "/tmp/isolated/.one-memory",
runtime: {
state: "running",
code: "runtime_verified",
host: "127.0.0.1",
port: 30001,
},
background: null,
reader: reader(summary),
});
it("names the code and the retry time while the breaker pauses", async () => {
const report = await compose(
processing({
llm_backoff_until: "2026-11-02T10:00:00+00:00",
llm_backoff_error: "model_auth_failed",
}),
);
expect(report.sections.processing).toMatchObject({
state: "ok",
pending: 4,
suspended: 2,
llm_backoff_until: "2026-11-02T10:00:00+00:00",
llm_backoff_error: "model_auth_failed",
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain(
"模型暂停:model_auth_failed,将于 2026-11-02T10:00:00+00:00 自动重试",
);
expect(text).not.toContain("llm_backoff");
});
it("prints no pause line and null fields when nothing is paused", async () => {
const report = await compose(
processing({ llm_backoff_until: null, llm_backoff_error: null }),
);
expect(report.sections.processing).toMatchObject({
llm_backoff_until: null,
llm_backoff_error: null,
});
expect(renderStatusText(report, "MemHub")).not.toContain("模型暂停");
});
});
describe("status and doctor on an unfinished upgrade migration", () => {
it("names the migration log and the issues page", async () => {
const report = await composeStatusReport({
+55 -8
View File
@@ -85,7 +85,7 @@ export interface PipelineSummary {
>;
graph_build: string | null;
/** SDK features (`memory_features`) whose value differs from the default. */
features: Record<string, boolean>;
features: Record<string, boolean | number>;
/** The App's display language; null = never chosen or "follow the system". */
ui_language: string | null;
/** The SDK prompt language: `auto` (zh/en detection) or the code passed. */
@@ -134,12 +134,13 @@ export function describePipeline(value: unknown): PipelineSummary | null {
};
}
}
const features: Record<string, boolean> = {};
const features: Record<string, boolean | number> = {};
if (item.features && typeof item.features === "object") {
for (const [name, value] of Object.entries(
item.features as Record<string, unknown>,
)) {
if (typeof value === "boolean") features[name] = value;
if (typeof value === "boolean" || typeof value === "number")
features[name] = value;
}
}
const graph =
@@ -319,6 +320,8 @@ export function renderStatusText(
report.sections.processing?.embedding_switch,
);
if (embeddingSwitch !== null) lines.push(`向量模型:${embeddingSwitch}`);
const backoff = modelBackoffLine(report.sections.processing);
if (backoff !== null) lines.push(backoff);
if (report.summary.state === "partial") {
const notes = [
report.summary.degraded.length > 0
@@ -334,6 +337,21 @@ export function renderStatusText(
return lines.join("\n");
}
/** "Model paused" while Core's circuit breaker holds model work back; null
* otherwise. Nothing to do here: the queue resumes by itself once the model
* answers again (or after the model is fixed). */
function modelBackoffLine(section: Section | undefined): string | null {
if (section?.state !== "ok") return null;
const error = section.llm_backoff_error;
if (typeof error !== "string" || error.length === 0) return null;
const until = section.llm_backoff_until;
return `模型暂停:${error},${
typeof until === "string" && until.length > 0
? `将于 ${until} 自动重试`
: "稍后自动重试"
}`;
}
export function pipelineLine(pipeline: PipelineSummary): string {
const parts: string[] = [pipeline.mode];
if (pipeline.forced_by_env)
@@ -357,7 +375,10 @@ export function pipelineLine(pipeline: PipelineSummary): string {
if (changed.length > 0) {
parts.push(
`;SDK 功能:${changed
.map(([name, value]) => `${name}=${value ? "开" : "关"}`)
.map(
([name, value]) =>
`${name}=${typeof value === "number" ? value : value ? "开" : "关"}`,
)
.join("、")}`,
);
}
@@ -380,15 +401,22 @@ function runtimeLine(report: StatusDocument): string {
})`;
}
/** Fields that have their own line (or none) instead of a place in the
* section's one-line summary. */
const SECTION_LINE_OMITTED = new Set([
"state",
"code",
"embedding_switch",
"llm_backoff_until",
"llm_backoff_error",
]);
function sectionLine(section: Section | undefined): string {
if (!section) return "未检查";
if (section.state === "not_checked") return "未检查";
if (section.state !== "ok") return "暂时不可用,请稍后重试";
return Object.entries(section)
.filter(
([key]) =>
key !== "state" && key !== "code" && key !== "embedding_switch",
)
.filter(([key]) => !SECTION_LINE_OMITTED.has(key))
.map(([key, value]) => `${fieldLabel(key)} ${formatValue(value)}`)
.join(" ");
}
@@ -424,6 +452,7 @@ function fieldLabel(key: string): string {
backlog_pending: "历史待处理",
outside_window_pending: "稍后处理",
waiting_for_model: "等待模型",
suspended: "来源已暂停",
completed: "已完成",
failed: "失败",
blocked: "卡住",
@@ -504,10 +533,28 @@ function describeProcessing(
reason: block.reason,
count: block.count,
})),
suspended: typeof summary.suspended === "number" ? summary.suspended : 0,
...llmBackoffFields(summary),
...embeddingSwitchField(summary),
};
}
/** The model circuit breaker: set only while Core pauses model work because
* the provider keeps failing. Both null when nothing is paused. */
function llmBackoffFields(summary: MemoryProcessingSummary): {
llm_backoff_until: string | null;
llm_backoff_error: string | null;
} {
const until = summary.llm_backoff_until;
const error = summary.llm_backoff_error;
return {
llm_backoff_until:
typeof until === "string" && until.length <= 64 ? until : null,
llm_backoff_error:
typeof error === "string" && /^[a-z_]{1,64}$/.test(error) ? error : null,
};
}
/** The running switch or merge, projected; absent when none runs. */
function embeddingSwitchField(
summary: MemoryProcessingSummary,
+63
View File
@@ -208,6 +208,27 @@ describe("--user-id on the memory commands", () => {
expect(direct.get.mock.calls[0][1].params).toEqual({ user_id: "wife" });
});
it("prints the retry code and time of a write still waiting", async () => {
direct.get.mockResolvedValue({
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T2",
error_code: "model_timeout",
next_retry_at: "2026-11-02T10:00:00+00:00",
},
});
const text = await capturedRun(["write", "status", REQUEST]);
expect(text.stdout).toContain("error_code: model_timeout");
expect(text.stdout).toContain("next_retry_at: 2026-11-02T10:00:00+00:00");
const json = await capturedRun(["write", "status", REQUEST, "--json"]);
expect(JSON.parse(json.stdout).write).toMatchObject({
error_code: "model_timeout",
next_retry_at: "2026-11-02T10:00:00+00:00",
});
});
it("passes the user in the body of memory writes and searches", async () => {
await capturedRun(["memories", "add", "Prefers tea", "--user-id", "wife"]);
expect(transport.post.mock.calls[0][0]).toBe("/api/v2/memories");
@@ -373,6 +394,48 @@ describe("sources connect for a Hermes profile", () => {
}
expect(transport.post).not.toHaveBeenCalled();
});
it("says the user id need not equal the Hermes profile name", async () => {
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"wife:work",
"--root",
"~/.hermes/profiles/wife:work",
]);
expect(result.code).toBe(2);
expect(result.stderr).toContain("用户 ID 不必等于 Hermes profile 名");
expect(result.stderr).toContain(
"--user-id wife-work --root ~/.hermes/profiles/wife:work",
);
expect(transport.post).not.toHaveBeenCalled();
});
it("explains a refused overlapping or profiles-container root", async () => {
for (const code of [
"hermes_root_is_profiles_container",
"hermes_root_overlaps_other_user",
]) {
transport.post.mockResolvedValueOnce({
status: code.endsWith("container") ? 422 : 409,
data: { detail: code },
});
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"kid",
"--root",
"/profiles",
]);
expect(result.code).toBe(1);
expect(result.stdout).toContain(code);
expect(result.stdout).toContain("--root");
}
});
});
async function capturedRun(arguments_: string[]) {
+1 -1
View File
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.11",
"hy-memory==2.0.12",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
+296 -112
View File
@@ -46,7 +46,12 @@ from pydantic import (
from one_memory import __version__, memory_visibility
from one_memory.auth import local_api_guard, renderer_origins
from one_memory.background_work import shutdown_bulk_pool, warm_bulk_pool
from one_memory.background_work import (
in_sdk_read,
in_the_background,
shutdown_bulk_pool,
warm_bulk_pool,
)
from one_memory.config import Settings, load_settings
from one_memory.context_routes import sdk_context_router
from one_memory.conversation_processing import (
@@ -55,6 +60,7 @@ from one_memory.conversation_processing import (
)
from one_memory.database_access import (
MAIN_DATABASE_REPORT_SECONDS,
checkpoint_idle_main_databases,
close_main_databases,
log_main_database_stats,
main_database,
@@ -130,6 +136,7 @@ from one_memory.hy_sdk_types import (
from one_memory.imports.access import (
AuthorizedSourceRoot,
SourceAccessError,
resolved_source_root,
root_identity_matches,
source_root_identity,
)
@@ -169,6 +176,11 @@ from one_memory.imports.locations import SourceLocation, inspect_source_location
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.runner import live_sync_arrival
from one_memory.imports.sources.common import SourceInstallState
from one_memory.imports.sources.hermes import (
HERMES_ROOT_IS_PROFILES_CONTAINER,
hermes_root_conflict,
hermes_root_is_profiles_container,
)
from one_memory.imports.sources.registry import (
RootCandidateKind,
SourceRegistration,
@@ -198,12 +210,17 @@ from one_memory.memory_evolution import (
)
from one_memory.memory_features import (
FEATURES,
PARAMETERS,
SETTING_NAMES,
FeatureCost,
FeatureOverrides,
FeatureReason,
FeatureSide,
MemoryFeatureName,
MemoryFeatures,
MemoryParameterName,
MemorySettingName,
SettingValue,
effective_features,
feature_activity,
non_default_features,
@@ -222,10 +239,10 @@ from one_memory.memory_server.config_state import (
MemoryMode,
)
from one_memory.memory_tasks import (
NON_RETRYABLE_POST_WRITE_CODES,
MemoryTaskStatus,
MemoryTaskStore,
)
from one_memory.model_backoff import BackedOffModelTransport, ModelBackoff
from one_memory.model_presets import seed_preset_profile
from one_memory.model_probe import ModelProber, ModelProbeResult
from one_memory.model_profiles import (
@@ -279,7 +296,7 @@ from one_memory.sdk_search_options import (
SearchLane,
split_csv,
)
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.sdk_write_processor import SdkWriteCoordinator, sdk_write_in_flight
from one_memory.secrets import (
KeyringSecretStore,
SecretStore,
@@ -526,11 +543,25 @@ class SdkMemoryUpdateRequest(BaseModel):
class SdkUserProfileResponse(BaseModel):
model_config = ConfigDict(extra="forbid")
#: The user whose profile this is: the `user_id` asked for, else `local`.
user_id: str
request_id: str
profile: dict[str, object] | None
elapsed_ms: float = Field(ge=0)
class SdkMemoryLibraryPage(HySdkMemoryPage):
"""`GET /api/v2/memory-library`: one SDK page and the user it was read for."""
model_config = ConfigDict(frozen=True, extra="forbid")
user_id: str
@classmethod
def of(cls, page: HySdkMemoryPage, user_id: str) -> "SdkMemoryLibraryPage":
return cls(**dict(page), user_id=user_id)
class SdkProfileVersionsResponse(HySdkProfileVersions):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -585,6 +616,8 @@ class SdkRawItem(HySdkRawRecord):
class SdkRawPage(BaseModel):
model_config = ConfigDict(extra="forbid")
#: The user whose raw turns these are.
user_id: str
items: tuple[SdkRawItem, ...] = ()
total: int = Field(ge=0)
offset: int = Field(ge=0)
@@ -598,6 +631,8 @@ class SdkRawPage(BaseModel):
class SdkLayerCounts(BaseModel):
model_config = ConfigDict(extra="forbid")
#: The user whose memories were counted.
user_id: str
#: Every SDK layer, zero when empty.
layers: dict[str, int]
total: int = Field(ge=0)
@@ -683,6 +718,17 @@ class MemoryProcessingSummary(BaseModel):
#: The running embedding switch or merge, `{job_id, kind, state, percent,
#: eta_s}`; None when none runs. Model writes wait while it is set.
embedding_switch: dict[str, object] | None = None
#: Jobs held back because their source is off or its grant was revoked.
#: Not in `pending` or the queue numbers: they run again, unchanged, once
#: the source is turned back on. Nothing is cancelled or deleted.
suspended: int = 0
#: Until when model work is paused because the provider kept failing
#: (bad key, wrong URL, model missing, provider down); None when it is
#: not. Ends early on success or when the model configuration changes.
llm_backoff_until: datetime | None = None
#: The last provider failure behind the pause, as a stable code
#: (`model_auth_failed`, `model_not_found`, `model_unreachable`, ...).
llm_backoff_error: str | None = None
class MemoryModeState(BaseModel):
@@ -763,8 +809,9 @@ class PipelineStatus(BaseModel):
last_evolve_at: datetime | None = None
graph_build: GraphBuildState
#: The SDK features (`memory_features`) whose value differs from the SDK
#: default; empty while every feature is at its default.
features: dict[MemoryFeatureName, bool] = Field(default_factory=dict)
#: default; empty while every feature is at its default. A parameter
#: (`intention_dedup_threshold`) appears as its number.
features: dict[MemorySettingName, SettingValue] = Field(default_factory=dict)
#: The display language the App last reported (`ui_language`); null =
#: none set (the App sends the language it shows, even following the system).
ui_language: UiLanguage | None = None
@@ -824,6 +871,30 @@ class MemoryFeatureState(BaseModel):
description: str
class MemoryParameterState(BaseModel):
"""One numeric setting in `GET /api/v1/settings/memory-features`."""
model_config = ConfigDict(extra="forbid")
name: MemoryParameterName
#: What clients are built with: `configured`, else `default`.
value: float
#: What a person set; null = not set, the SDK default applies.
configured: float | None
#: The SDK default (its code default, or its own env var when set).
default: float
#: The range a `PUT` accepts, inclusive.
minimum: float
maximum: float
#: Whether it acts: exactly when `requires_feature` acts.
active: bool
reason: FeatureReason | None
requires_feature: MemoryFeatureName
sdk_field: str
env: str
description: str
class MemoryFeaturesState(BaseModel):
"""`GET /api/v1/settings/memory-features`."""
@@ -831,15 +902,16 @@ class MemoryFeaturesState(BaseModel):
mode: MemoryMode
features: tuple[MemoryFeatureState, ...]
parameters: tuple[MemoryParameterState, ...] = ()
#: Same as `pipeline.features`: the values that differ from the defaults.
non_default: dict[MemoryFeatureName, bool]
non_default: dict[MemorySettingName, SettingValue]
class MemoryFeaturesChange(MemoryFeaturesState):
"""`PUT /api/v1/settings/memory-features`: the new state and what changed."""
#: Features whose value changed; empty = nothing rebuilt.
changed: tuple[MemoryFeatureName, ...] = ()
#: Features and parameters whose value changed; empty = nothing rebuilt.
changed: tuple[MemorySettingName, ...] = ()
def memory_mode_effects(
@@ -873,15 +945,6 @@ def memory_mode_effects(
MemoryProcessingSummary.model_rebuild()
class RetryUnknownMemoryTasksRequest(BaseModel):
confirm: Literal[True]
class RetryUnknownMemoryTasksReceipt(BaseModel):
reconciled: int = 0
requeued: int = 0
class GraphWidthStatus(BaseModel):
"""What the app needs to offer the choice before extraction fails on it."""
@@ -967,6 +1030,8 @@ class AskMemoryRequest(SdkSearchFilters):
class SdkHomeSummary(BaseModel):
schema_version: Literal[2] = 2
#: The user the card was read for.
user_id: str
memory_count: int
latest: HySdkMemoryRecord | None
observed_at: datetime
@@ -993,6 +1058,8 @@ class MemoryGraphEdge(BaseModel):
class MemoryGraphResponse(BaseModel):
#: The user whose memories were mapped.
user_id: str
total_memories: int
projected_memories: int
nodes: tuple[MemoryGraphNode, ...]
@@ -1145,7 +1212,6 @@ def create_app(
)
from one_memory.memory_server.data_safety import MIN_FREE_BYTES
from one_memory.memory_server.durable_queue import (
COMPLETED,
DurableWriteDrain,
DurableWriteQueue,
execute_job,
@@ -1304,6 +1370,9 @@ def create_app(
)
app.state.model_request_budget = model_request_budget
app.state.model_request_audit_store = ModelRequestAuditStore(database_path)
#: One pause for every model request while the provider keeps
#: failing (`model_backoff`); reset when the configuration changes.
app.state.model_backoff = ModelBackoff()
# Before anything asks for an active profile: a first launch that
# carries a preset should show it already filled in, so the user
# meets one empty field instead of a form they cannot answer.
@@ -1322,17 +1391,23 @@ def create_app(
profiles=app.state.model_profile_store,
secrets=app.state.secret_store,
data_dir=runtime_settings.data_dir,
# Outermost, so a request the pause refuses never went out and
# leaves no audit row behind.
transport_factory=lambda profile: (
lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
lambda: BackedOffModelTransport(
app.state.model_backoff,
profile_id=f"{profile.profile_id}:{profile.revision}",
inner=AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
),
)
),
@@ -1351,6 +1426,9 @@ def create_app(
data_dir=runtime_settings.data_dir,
memory_mode=lambda: _memory_mode(app),
writes_open=lambda: _embedding_writes_open(app),
# The claim gate's own source question, so what is counted as
# waiting is what the drain would claim (#245).
source_active=lambda source_key: _source_writes_on(app, source_key),
)
app.state.evolve_state_store = EvolveStateStore(runtime_settings.data_dir)
app.state.evolve_service = EvolveService(
@@ -1375,24 +1453,15 @@ def create_app(
drain_budget = production_drain_budget()
async def reconcile_unknown_writes() -> None:
unknown = await asyncio.to_thread(
memory_write_queue.unknown_jobs,
limit=128,
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
# A write whose worker never learned how it ended (a restart,
# a lapsed lease, the work ceiling) goes back into the queue
# under a new SDK request id; the re-run first asks the SDK
# whether the old one landed (`execute_job`). Nothing stays
# `unknown` waiting for a person.
await in_the_background(
memory_write_queue.recover_unknown,
in_flight=sdk_write_in_flight,
)
for job in unknown:
receipt = await app.state.sdk_write_coordinator.inquire_write_outcome(
job.request_id
)
if receipt is None:
continue
await asyncio.to_thread(
memory_write_queue.settle,
job.job_id,
state=COMPLETED,
sdk_raw_ids=receipt.raw_ids,
sdk_memory_ids=receipt.memory_ids,
)
async def admit_hy_invocation(
source_id: str,
@@ -1422,6 +1491,7 @@ def create_app(
evidence=app.state.evidence_store,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
admit_hy_invocation=admit_hy_invocation,
model_failures=app.state.model_backoff.attempt,
),
execute_owned=lambda job_id, owner: execute_job(
job_id,
@@ -1431,6 +1501,7 @@ def create_app(
owner=owner,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
admit_hy_invocation=admit_hy_invocation,
model_failures=app.state.model_backoff.attempt,
),
drain_handle_budget=drain_budget.handle_budget,
drain_cleanup_reserve=drain_budget.cleanup_reserve,
@@ -1458,6 +1529,7 @@ def create_app(
# therefore skipped on every drain tick — the note stayed
# "saved, waiting" with no task behind it, forever.
source_claim_enabled=lambda source_id: _sedimentation_claim_enabled(app, source_id),
model_claim_allowance=app.state.model_backoff.allowance,
)
app.state.memory_write_drain = memory_write_drain
@@ -1495,6 +1567,7 @@ def create_app(
app.state.memory_task_store.rebind_unstarted_profile, profile
)
await asyncio.to_thread(memory_write_queue.readmit_model_profile_waiters)
_reset_model_backoff(app)
_resume_sedimentation(app)
_start_backfill(app)
@@ -1970,6 +2043,10 @@ def create_app(
# Last: what has just been shut down was using it.
shutdown_bulk_pool()
log_main_database_stats("shutdown")
# Leave an empty WAL behind: everything that wrote is stopped
# by now. Busy (another process reading) is fine; the next
# start's idle pass truncates it.
await asyncio.to_thread(checkpoint_idle_main_databases)
close_main_databases()
diagnostic_event(
level="info",
@@ -2667,6 +2744,7 @@ def create_app(
)
if not root_identity_matches(registration.candidate.path, request.expected_root_id):
raise HTTPException(status_code=409, detail="source_identity_changed")
await _refuse_overlapping_root(source_id, registration.candidate.path, DEFAULT_USER_ID)
root_changed = prior is not None and prior.root_id != request.expected_root_id
if root_changed:
source_jobs = [job for job in _job_store(app).list_jobs() if job.source_id == source_id]
@@ -3067,6 +3145,20 @@ def create_app(
registrations = [_require_registration(app, source_id) for source_id in source_ids]
if request.request_id is None:
raise HTTPException(status_code=409, detail="source_confirmation_required")
for registration in registrations:
if registration.descriptor.source_id in MULTI_USER_SOURCES:
try:
resolved = await asyncio.to_thread(
app.state.source_candidate_store.resolve, registration
)
except SourceCandidateError:
# The connect below reports the missing candidate itself.
continue
await _refuse_overlapping_root(
registration.descriptor.source_id,
resolved.candidate.path,
DEFAULT_USER_ID,
)
history_after = await _onboarding_history_after(
request,
registrations,
@@ -3166,6 +3258,42 @@ def create_app(
raise
return ConnectSourcesResponse(**result.model_dump())
async def _refuse_overlapping_root(source_id: str, root: Path, user_id: str) -> None:
"""Refuse a Hermes root that would read another user's files.
422 `hermes_root_is_profiles_container` for `.../profiles` itself;
409 `hermes_root_overlaps_other_user` for a root equal to, holding or
inside another user's active grant, unless one is the other's
`profiles/<name>` (the scan never enters `profiles/`).
"""
if source_id not in MULTI_USER_SOURCES:
return
if hermes_root_is_profiles_container(root):
raise HTTPException(status_code=422, detail=HERMES_ROOT_IS_PROFILES_CONTAINER)
def conflict_with_others() -> str | None:
try:
mine = resolved_source_root(root)
except SourceAccessError:
# Unreadable or missing: the grant that follows refuses it
# with its own code.
return None
others = [
Path(path)
for receipt, path in _grant_store(app).grants()
if receipt.source_id == source_id
and receipt.user_id != user_id
and receipt.revoked_at is None
]
return hermes_root_conflict(mine, others)
conflict = await asyncio.to_thread(conflict_with_others)
if conflict is not None:
raise HTTPException(
status_code=422 if conflict == HERMES_ROOT_IS_PROFILES_CONTAINER else 409,
detail=conflict,
)
async def _connect_other_root(request: ConnectSourcesRequest) -> ConnectSourcesResponse:
"""Grant one Hermes root to a user: `~/.hermes/profiles/<name>` for its own user.
@@ -3195,6 +3323,7 @@ def create_app(
root = Path(request.root)
if not root.is_absolute():
raise HTTPException(status_code=422, detail="source_root_must_be_absolute")
await _refuse_overlapping_root(source_id, root, request.user_id)
# Named outright by the caller, like an environment override:
# no saved candidate may replace it, and naming it confirms it.
registration = replace(
@@ -3428,7 +3557,7 @@ def create_app(
)
def create_model_profile(request: ModelProfileInput) -> ModelProfile:
try:
return _model_profile_store(app).create(request, _secret_store(app))
created = _model_profile_store(app).create(request, _secret_store(app))
except EmbeddingBaseUrlRequiredError:
raise HTTPException(422, "embedding_base_url_required") from None
except ModelCreationOperationConflictError:
@@ -3440,6 +3569,8 @@ def create_app(
status_code=503,
detail=_secret_store_detail(error),
) from None
_reset_model_backoff(app)
return created
@app.get("/api/v1/model-profile-creations/{operation_id}", dependencies=protected)
def model_creation_operation(operation_id: str) -> ModelCreationOperationReceipt:
@@ -3474,7 +3605,7 @@ def create_app(
) -> ModelProfile:
_refuse_embedding_edit_during_switch(app, profile_id, request)
try:
return _model_profile_store(app).update(
updated = _model_profile_store(app).update(
profile_id,
request,
_secret_store(app),
@@ -3490,6 +3621,10 @@ def create_app(
status_code=503,
detail=_secret_store_detail(error),
) from None
# A saved key, URL or model is what the paused provider was waiting
# for: try it now, not after the rest of the pause.
_reset_model_backoff(app)
return updated
@app.post(
"/api/v1/model-profiles/{profile_id}/activate",
@@ -3552,6 +3687,7 @@ def create_app(
# D75: work queued for the previous model continues with this one.
# Only tasks that never reached the model are re-bound.
await asyncio.to_thread(app.state.memory_task_store.rebind_unstarted_profile, profile)
_reset_model_backoff(app)
# A missing profile is an event-driven wait, not a provider outage.
# Release those durable rows only now that activation has changed the
# condition; the ordinary drain poll will claim them without a restart.
@@ -3730,6 +3866,7 @@ def create_app(
raise _sdk_http_error(error, "hy_memory_list_failed") from None
total = stats.get("total") if isinstance(stats, dict) else None
return SdkHomeSummary(
user_id=user_id,
memory_count=total if isinstance(total, int) and total >= 0 else page.total,
latest=page.memories[0] if page.memories else None,
observed_at=datetime.now(UTC),
@@ -3737,7 +3874,7 @@ def create_app(
@app.get(
"/api/v2/memory-library",
response_model=HySdkMemoryPage,
response_model=SdkMemoryLibraryPage,
dependencies=strict,
)
async def sdk_memory_library(
@@ -3754,7 +3891,7 @@ def create_app(
exclude_sources: list[str] | None = Query(default=None), # noqa: B008
session_ids: list[str] | None = Query(default=None), # noqa: B008
agent_ids: list[str] | None = Query(default=None), # noqa: B008
) -> HySdkMemoryPage:
) -> SdkMemoryLibraryPage:
"""One page of memories across chat, working and coding.
Without `query`: `list_memories(scene="all")`, newest update first;
@@ -3800,7 +3937,7 @@ def create_app(
status_code=422,
detail="sdk_search_supports_active_status_only",
)
return await _library_search_page(
searched = await _library_search_page(
app,
client,
query=search_query,
@@ -3813,6 +3950,7 @@ def create_app(
user_id=user_id,
filters=filters,
)
return SdkMemoryLibraryPage.of(searched, user_id)
raw = await asyncio.to_thread(
client.list_memories,
user_id=user_id,
@@ -3830,7 +3968,7 @@ def create_app(
raw, limit=limit, offset=offset
)
_log_skipped_sdk_memories(skipped)
return page
return SdkMemoryLibraryPage.of(page, user_id)
except HTTPException:
raise
except Exception as error:
@@ -3876,6 +4014,7 @@ def create_app(
except Exception as error:
raise _sdk_http_error(error, "hy_memory_list_failed") from None
return SdkLayerCounts(
user_id=user_id,
layers=counts,
total=sum(counts.values()),
statuses=tuple(chosen) if chosen else None,
@@ -4036,6 +4175,7 @@ def create_app(
)
)
return MemoryGraphResponse(
user_id=user_id,
total_memories=total if isinstance(total, int) else len(memories),
projected_memories=len(memories),
nodes=tuple(nodes.values()),
@@ -4057,7 +4197,7 @@ def create_app(
profile = HySdkUserProfile.from_sdk(value)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_profile_read_failed") from None
return SdkUserProfileResponse.model_validate(profile.model_dump())
return SdkUserProfileResponse(user_id=user_id, **profile.model_dump())
@app.get(
"/api/v2/user-profile/versions",
@@ -4181,15 +4321,16 @@ def create_app(
try:
async with app.state.sdk_write_coordinator.use_client() as client:
# `delete` takes no user. Another user's memory is answered
# exactly like an id that does not exist: 204, nothing
# touched, so the status cannot tell that it exists.
# exactly like an id that does not exist: 404
# `memory_not_found`, nothing touched, so the status cannot
# tell that it exists. Callers read 404 as "already gone".
current = await asyncio.to_thread(
client.get, memory_id, **_namespace_kwargs(kind, variant)
)
if current is not None and not owns(
if current is None or not owns(
HySdkMemoryRecord.from_sdk(current).user_id, user_id
):
return Response(status_code=204)
raise HTTPException(status_code=404, detail="memory_not_found")
result = await asyncio.to_thread(
client.delete, memory_id, **_namespace_kwargs(kind, variant)
)
@@ -4198,7 +4339,11 @@ def create_app(
error_code = result.get("error_code")
if error_code or result.get("partial") is True:
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
if result.get("success") is not True and result.get("deleted_count") != 0:
if result.get("success") is not True:
# Read a moment ago, found by nobody now (a second delete won
# the race): the same answer as an id that never existed.
if result.get("deleted_count") == 0:
raise HTTPException(status_code=404, detail="memory_not_found")
raise HySdkError("hy_memory_delete_rejected")
except HTTPException:
raise
@@ -4271,6 +4416,7 @@ def create_app(
and (before is None or at < before)
)
return SdkRawPage(
user_id=user_id,
items=tuple(SdkRawItem.of(record) for record in records),
total=page.total,
offset=offset,
@@ -4303,6 +4449,7 @@ def create_app(
)
except MemoryAnswerError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
answer = answer.model_copy(update={"user_id": request.user_id})
if request.question != "__om_probe__" and access is not None:
from one_memory.call_event_recorder import ask_call_event, try_record
@@ -4344,6 +4491,9 @@ def create_app(
):
if event.get("type") == "final":
final = event
elif event.get("type") == "recall":
# The first event names whose memories answer.
event = {**event, "user_id": request.user_id}
yield _sse(_serialize_ask_event(event))
except MemoryAnswerError:
yield _sse({"type": "error", "code": "memory_answer_failed"})
@@ -4379,7 +4529,10 @@ def create_app(
http_request: Request,
) -> RecallReceipt:
from one_memory.call_event_recorder import record_legacy_result
from one_memory.recall_fallback import recall_with_fallback
from one_memory.recall_fallback import (
recall_total_timeout_seconds,
recall_with_fallback,
)
started = time.monotonic()
await _require_external_recall_scope(
@@ -4395,21 +4548,36 @@ def create_app(
)
raw_options = {"filters": options} if options is not None else {}
access = external_access(http_request.headers)
receipt = await recall_with_fallback(
request,
model_ready=lambda: _recall_model_ready(app),
recall_memories=target.recall_async,
has_memories=lambda: _library_has_memories(app, request.user_id),
raw_sources=lambda: _recall_raw_sources(app, request, access, started),
search_raw=lambda query, limit, sources: _search_sdk_raw(
app,
query,
limit=limit,
sources=sources,
user_id=request.user_id,
**raw_options,
),
)
try:
# The whole answer, not only the search: the checks before and
# after it used to wait for a thread with no limit at all, so a
# pool held by stuck writes left recall hanging while /health
# still answered (#231). Late is a clear 504, not silence.
receipt = await asyncio.wait_for(
recall_with_fallback(
request,
model_ready=lambda: _recall_model_ready(app),
recall_memories=target.recall_async,
has_memories=lambda: _library_has_memories(app, request.user_id),
raw_sources=lambda: _recall_raw_sources(app, request, access, started),
search_raw=lambda query, limit, sources: _search_sdk_raw(
app,
query,
limit=limit,
sources=sources,
user_id=request.user_id,
**raw_options,
),
),
timeout=recall_total_timeout_seconds(request.timeout_ms),
)
except TimeoutError:
logger.warning(
"recall_timed_out timeout_ms=%d elapsed_ms=%.0f",
request.timeout_ms,
(time.monotonic() - started) * 1000,
)
raise HTTPException(status_code=504, detail="recall_timeout") from None
event_store = getattr(app.state, "call_event_store", None)
if event_store is not None and request.query != "__om_probe__":
try:
@@ -4422,7 +4590,7 @@ def create_app(
)
except Exception:
app.state.call_event_recording_error = True
return receipt
return receipt.model_copy(update={"user_id": request.user_id})
@app.get(
"/api/v1/memory-processing",
@@ -4447,6 +4615,8 @@ def create_app(
# running" is how a stalled queue looks busy, so the strand is given
# its own number.
running_rows = counts.get(MemoryTaskStatus.RUNNING, 0)
backoff = cast(ModelBackoff | None, getattr(app.state, "model_backoff", None))
backoff_state = backoff.state() if backoff is not None else None
return MemoryProcessingSummary(
pending=(
sdk_counts.get("pending", 0)
@@ -4487,6 +4657,13 @@ def create_app(
metrics=ProcessingMetrics.model_validate(store.processing_metrics()),
pipeline=_pipeline_status(app),
embedding_switch=_embedding_switch_summary(app),
suspended=sdk_counts.get("suspended", 0),
llm_backoff_until=backoff_state.until if backoff_state is not None else None,
llm_backoff_error=(
backoff_state.last_error
if backoff_state is not None and backoff_state.until is not None
else None
),
)
@app.get(
@@ -4565,7 +4742,8 @@ def create_app(
"""Turn SDK features on or off; one global setting, not per user.
A key sets that feature, null resets it to the SDK default, an absent
key is left as it is; an unknown key or a non-boolean is 422. A
key is left as it is; an unknown key, a non-boolean feature or an
`intention_dedup_threshold` outside its range is 422. A
change rebuilds the SDK clients like a mode change: a write in flight
finishes with the old features, the next job and search use the new
ones. Nothing stored is migrated and no collection is renamed. See
@@ -4581,7 +4759,7 @@ def create_app(
current = _memory_features(app)
before = effective_features(previous, defaults)
after = effective_features(current, defaults)
changed = tuple(spec.name for spec in FEATURES if before[spec.name] != after[spec.name])
changed = tuple(name for name in SETTING_NAMES if before[name] != after[name])
if current != previous:
# Even a reset to the default value rebuilds: the client then
# carries no override at all, byte for byte the default config.
@@ -4727,29 +4905,6 @@ def create_app(
raise HTTPException(status_code=404, detail="evolve_run_not_found")
return receipt
@app.post(
"/api/v1/memory-processing/retry-unknown",
response_model=RetryUnknownMemoryTasksReceipt,
dependencies=protected,
)
async def retry_unknown_memory_tasks(
request: RetryUnknownMemoryTasksRequest,
) -> RetryUnknownMemoryTasksReceipt:
"""Retry unknown writes only behind the user's explicit confirmation."""
del request
reconciled = await _sdk_memory_runtime(app).reconcile_unknown_outputs()
# The startup queue knows the authority. A fresh legacy-mode queue
# would recreate the job-evidence table the SDK cutover dropped.
queue = app.state.memory_write_queue
unknown = queue.unknown_jobs(limit=2048)
if app.state.memory_authority is MemoryAuthority.LEGACY:
for job in unknown:
queue.resolve_unknown_from_linked_output(job.job_id)
requeued = sum(queue.retry_unknown_confirmed(job.job_id) for job in unknown)
if app.state.memory_authority is MemoryAuthority.LEGACY:
requeued += app.state.memory_task_store.retry_unknown_confirmed()
return RetryUnknownMemoryTasksReceipt(reconciled=reconciled, requeued=requeued)
@app.get(
"/api/v1/graph-width",
response_model=GraphWidthStatus,
@@ -4916,7 +5071,7 @@ async def _require_external_recall_scope(
# 299 rows from a covering index, about 52ms warm — enough, repeated and
# serialised on the loop, to answer nobody for seconds.
scope_started = time.monotonic()
known = await asyncio.to_thread(_evidence_store(app).known_workspaces, user_id="local")
known = await in_sdk_read(_evidence_store(app).known_workspaces, user_id="local")
# Measured because nothing else records it. The ~52ms figure is from a copy
# of one database at one point in time; this says what it costs here, on a
# database that has since grown, often enough to show whether the read is
@@ -5852,7 +6007,7 @@ async def _recall_model_ready(app: FastAPI) -> bool:
if _memory_mode(app) == "lite":
return False
try:
profile = await asyncio.to_thread(_model_profile_store(app).active, _secret_store(app))
profile = await in_sdk_read(_model_profile_store(app).active, _secret_store(app))
except Exception:
return False
return profile is not None and bool(profile.capabilities.model_available)
@@ -5891,7 +6046,7 @@ async def _library_has_memories(app: FastAPI, user_id: str = DEFAULT_USER_ID) ->
"""Any active memory at all. Asked only after an empty memory search."""
try:
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
raw = await in_sdk_read(
client.list_memories,
user_id=user_id,
statuses=["active"],
@@ -5931,7 +6086,7 @@ async def _recall_raw_sources(
timeout_ms=request.timeout_ms,
)
try:
authority = await asyncio.to_thread(
authority = await in_sdk_read(
resolve_context_authority,
query,
access,
@@ -5971,7 +6126,7 @@ async def _search_sdk_raw(
if not include:
return ()
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
raw = await in_sdk_read(
client.search,
query,
**{
@@ -6066,7 +6221,7 @@ def _ui_language_state(app: FastAPI) -> UiLanguageState:
def _memory_features_state(
app: FastAPI, defaults: dict[str, bool] | None = None
app: FastAPI, defaults: dict[str, SettingValue] | None = None
) -> MemoryFeaturesState:
defaults = sdk_feature_defaults() if defaults is None else defaults
overrides = _memory_features(app)
@@ -6079,9 +6234,9 @@ def _memory_features_state(
features.append(
MemoryFeatureState(
name=spec.name,
enabled=effective[spec.name],
configured=configured.get(spec.name),
default=defaults[spec.name],
enabled=bool(effective[spec.name]),
configured=cast(bool | None, configured.get(spec.name)),
default=bool(defaults[spec.name]),
active=reason is None,
reason=reason,
side=spec.side,
@@ -6093,10 +6248,32 @@ def _memory_features_state(
description=spec.description,
)
)
parameters = []
for parameter in PARAMETERS:
reason = feature_activity(parameter.name, effective, mode)
parameters.append(
MemoryParameterState(
name=parameter.name,
value=float(effective[parameter.name]),
configured=cast(float | None, configured.get(parameter.name)),
default=float(defaults[parameter.name]),
minimum=parameter.minimum,
maximum=parameter.maximum,
active=reason is None,
reason=reason,
requires_feature=parameter.requires_feature,
sdk_field=parameter.sdk_field,
env=parameter.env,
description=parameter.description,
)
)
return MemoryFeaturesState(
mode=mode,
features=tuple(features),
non_default=cast(dict[MemoryFeatureName, bool], non_default_features(overrides, defaults)),
parameters=tuple(parameters),
non_default=cast(
dict[MemorySettingName, SettingValue], non_default_features(overrides, defaults)
),
)
@@ -6117,7 +6294,7 @@ def _pipeline_status(app: FastAPI) -> PipelineStatus | None:
last_evolve_at=max(finished, default=None),
graph_build=state.graph_build,
features=cast(
dict[MemoryFeatureName, bool],
dict[MemorySettingName, SettingValue],
non_default_features(_memory_features(app), sdk_feature_defaults()),
),
**_ui_language_state(app).model_dump(exclude={"languages"}),
@@ -6160,7 +6337,7 @@ async def _get_sdk_memory(
variant: str | None = None,
) -> HySdkMemoryRecord | None:
client = await handle.client()
raw = await asyncio.to_thread(client.get, memory_id, **_namespace_kwargs(kind, variant))
raw = await in_sdk_read(client.get, memory_id, **_namespace_kwargs(kind, variant))
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
@@ -6322,7 +6499,7 @@ async def _search_sdk_memories(
) -> tuple[object, ...]:
client = await handle.client()
overrides = options.search_overrides() if options is not None else {}
raw = await asyncio.to_thread(
raw = await in_sdk_read(
client.search,
query,
**{
@@ -6398,6 +6575,13 @@ def _user_evolving(app: FastAPI, user_id: str) -> bool:
return bool(callable(evolving) and evolving(user_id))
def _reset_model_backoff(app: FastAPI) -> None:
"""The user changed the model configuration: end any provider pause now."""
backoff = getattr(app.state, "model_backoff", None)
if isinstance(backoff, ModelBackoff):
backoff.reset()
def _embedding_switch(app: FastAPI) -> EmbeddingSwitchService:
return cast(EmbeddingSwitchService, app.state.embedding_switch)
@@ -15,6 +15,11 @@ BULK_WORKERS = 3
SNAPSHOT_WORKERS = 8
INTERACTIVE_READ_WORKERS = 4
SOURCE_READER_WORKERS = 4
# One per source the drain runs side by side (`max_concurrent_sources`), and
# two over for the outcome inquiries that follow them.
SDK_WRITE_WORKERS = 12
# Recall's SDK reads: one search and the `get`s behind it (`_SDK_GET_CONCURRENCY`).
SDK_READ_WORKERS = 8
_pool: ThreadPoolExecutor | None = None
_snapshot_pool: ThreadPoolExecutor | None = None
@@ -22,6 +27,7 @@ _interactive_pool: ThreadPoolExecutor | None = None
_raw_read_pool: ThreadPoolExecutor | None = None
_memory_read_pool: ThreadPoolExecutor | None = None
_source_reader_pool: ThreadPoolExecutor | None = None
_sdk_write_pool: ThreadPoolExecutor | None = None
_warmed_readers: weakref.WeakSet[ThreadPoolExecutor] = weakref.WeakSet()
_reader_warm_lock = threading.Lock()
_interactive_reads = 0
@@ -71,15 +77,38 @@ def raw_read_pool() -> ThreadPoolExecutor:
def memory_read_pool() -> ThreadPoolExecutor:
"""Recall's own threads: SDK searches, gets, and what decides between them.
Recall used to hand these to the default pool, which SDK writes share. A
write that times out keeps its thread until the SDK returns (#231), so a
few stuck writes left recall waiting for a thread while `/health`, which
needs none, still answered.
"""
global _memory_read_pool
if _memory_read_pool is None:
_memory_read_pool = ThreadPoolExecutor(
max_workers=INTERACTIVE_READ_WORKERS,
max_workers=SDK_READ_WORKERS,
thread_name_prefix="one-memory-memory-read",
)
return _memory_read_pool
def sdk_write_pool() -> ThreadPoolExecutor:
"""SDK writes, bounded and apart from everything else.
Each thread here only waits for the SDK's own loop thread to answer
(`_LoopThread.run`), so any thread outside that loop may make the call; a
stuck write therefore costs a thread of this pool and nobody else's.
"""
global _sdk_write_pool
if _sdk_write_pool is None:
_sdk_write_pool = ThreadPoolExecutor(
max_workers=SDK_WRITE_WORKERS,
thread_name_prefix="one-memory-sdk-write",
)
return _sdk_write_pool
def source_reader_pool() -> ThreadPoolExecutor:
"""Bounded process-wide pool for source iteration and parsing."""
global _source_reader_pool
@@ -175,6 +204,30 @@ async def in_interactive_read[T](
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
async def in_sdk_read[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""A recall-path read, in recall's own pool (`memory_read_pool`)."""
return await in_read_pool(memory_read_pool(), work, *args, **kwargs)
async def in_sdk_write[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""One SDK write call, in the write pool (`sdk_write_pool`)."""
return await asyncio.get_running_loop().run_in_executor(
sdk_write_pool(),
copy_context().run,
partial(work, *args, **kwargs),
)
def _retrieve[T](future: "asyncio.Future[T]") -> None:
"""Mark a shielded read's outcome as seen.
@@ -261,7 +314,10 @@ def _warm_reader_pool(reader: ThreadPoolExecutor) -> None:
def shutdown_bulk_pool() -> None:
global _pool, _snapshot_pool, _interactive_pool, _raw_read_pool, _memory_read_pool
global _source_reader_pool
global _source_reader_pool, _sdk_write_pool
if _sdk_write_pool is not None:
_sdk_write_pool.shutdown(wait=False, cancel_futures=True)
_sdk_write_pool = None
if _pool is not None:
_pool.shutdown(wait=False, cancel_futures=True)
_pool = None
@@ -155,6 +155,8 @@ class SdkContextSearchResponse(BaseModel):
used_characters: int = Field(ge=0, le=20000)
truncated: bool
total_ms: float = Field(ge=0)
#: Whose memories were searched: the request's `user_id`, else `local`.
user_id: str = Field(min_length=1, max_length=64)
class ContextLaneResult(BaseModel):
@@ -261,6 +261,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
used_characters=0,
truncated=True,
total_ms=(time.monotonic() - started) * 1000,
user_id=value.user_id,
)
await record(final, timed_out=True)
return final
@@ -306,6 +307,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
used_characters=used,
truncated=truncated,
total_ms=(time.monotonic() - started) * 1000,
user_id=value.user_id,
)
await record(final, timed_out=False)
return final
@@ -111,10 +111,26 @@ MAIN_DATABASE_CHECKPOINT_WRITE_INTERVAL = 64
MAIN_DATABASE_CHECKPOINT_SECONDS = 30.0
# Advisory only: PASSIVE cannot force a frame past a long reader. Crossing this
# threshold is a signal that the WAL is not converging and that read snapshots
# or write volume need investigation.
# threshold with frames still waiting to be copied back is a signal that the
# WAL is not converging and that read snapshots or write volume need
# investigation. A large file whose frames are all checkpointed is not: the
# idle TRUNCATE below gives that space back.
MAIN_DATABASE_WAL_WARN_BYTES = 512 * 1024 * 1024
# What a WAL file is cut back to once a checkpoint has copied every frame and
# the writer starts it over. Without a limit SQLite reuses the file in place
# and never shrinks it, so one import peak left a 639 MB WAL behind for good
# (#245). 64 MiB is about sixteen times what the default autocheckpoint (1000
# pages) lets build up between checkpoints, so ordinary write bursts reuse the
# file without growing and shrinking it each time, and it is still a tenth of
# what that user's disk was left holding.
MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES = 64 * 1024 * 1024
# How long an idle TRUNCATE may wait for the writer or for a reader to leave.
# It runs only when nothing holds the database; anyone arriving in the
# meantime wins, and the checkpoint is tried again on the next pass.
MAIN_DATABASE_IDLE_CHECKPOINT_WAIT_MS = 50
def checkpoint_write_interval() -> int:
raw = os.environ.get(
@@ -276,6 +292,8 @@ class MainDatabase:
self._writes_since_checkpoint = 0
self._last_checkpoint_at = time.monotonic()
self._wal_warned_at = 0.0
#: Learned from the first checkpoint; turns WAL frames into bytes.
self._page_size = 4096
self._reader_started_at: dict[int, float] = {}
@contextmanager
@@ -542,16 +560,59 @@ class MainDatabase:
def _checkpoint_from_maintenance(self, mode: str) -> tuple[int, int, int]:
return self._checkpoint(mode, maintenance=True)
def _checkpoint(self, mode: str, *, maintenance: bool) -> tuple[int, int, int]:
def _checkpoint(
self,
mode: str,
*,
maintenance: bool,
busy_timeout_ms: int = WRITE_LOCK_WAIT_MS,
) -> tuple[int, int, int]:
normalized = mode.strip().upper()
if normalized not in {"PASSIVE", "FULL", "RESTART", "TRUNCATE"}:
raise ValueError("unsupported WAL checkpoint mode")
with self.write(autocommit=True, maintenance=maintenance) as connection:
with self.write(
autocommit=True, maintenance=maintenance, busy_timeout_ms=busy_timeout_ms
) as connection:
row = connection.execute(f"PRAGMA wal_checkpoint({normalized})").fetchone()
page_size = connection.execute("PRAGMA page_size").fetchone()
if row is None or len(row) != 3:
raise sqlite3.DatabaseError("SQLite returned an invalid checkpoint result")
if page_size is not None:
self._page_size = int(page_size[0])
return int(row[0]), int(row[1]), int(row[2])
def idle(self) -> bool:
"""Nobody holds or waits for a connection to this database right now."""
with self._state:
return (
self._writer_owner is None
and not self._reader_holders
and not len(self._write_line)
and not len(self._read_line)
)
def checkpoint_if_idle(self) -> tuple[int, int, int] | None:
"""Give the WAL file back to the disk while nothing is using the database.
PASSIVE copies frames back but leaves the file its size; only TRUNCATE
empties it, and TRUNCATE has to wait for every reader to finish, so it
is attempted only when this process holds no checkout at all, and with
a wait of a few milliseconds. A busy result, or someone arriving in
between, is not a failure: the next idle pass tries again. Returns the
checkpoint result, or None when it was not attempted or not admitted.
"""
if not self.idle() or self._wal_bytes() == 0:
return None
try:
return self._checkpoint(
"TRUNCATE",
maintenance=True,
busy_timeout_ms=MAIN_DATABASE_IDLE_CHECKPOINT_WAIT_MS,
)
except sqlite3.Error:
logger.debug("main_database_idle_checkpoint_skipped file=%s", self.path.name)
return None
def _maybe_checkpoint_after_write(self) -> None:
"""Checkpoint on a write cadence without opening another connection.
@@ -579,17 +640,23 @@ class MainDatabase:
logger.warning("main_database_checkpoint_failed file=%s", self.path.name, exc_info=True)
return
wal_bytes = self._wal_bytes()
if wal_bytes >= MAIN_DATABASE_WAL_WARN_BYTES:
# Frames the checkpoint could not copy back yet: what a reader is
# holding up. The file size alone also counts space already
# checkpointed and only waiting to be reused or truncated, which is
# not the WAL failing to converge.
pending_bytes = max(0, log_frames - checkpointed) * self._page_size
if pending_bytes >= MAIN_DATABASE_WAL_WARN_BYTES:
with self._state:
should_warn = now - self._wal_warned_at >= MAIN_DATABASE_REPORT_SECONDS
if should_warn:
self._wal_warned_at = now
if should_warn:
logger.warning(
"main_database_wal_high file=%s wal_bytes=%d busy=%d "
"main_database_wal_high file=%s wal_bytes=%d pending_bytes=%d busy=%d "
"log_frames=%d checkpointed=%d readers_in_use=%d",
self.path.name,
wal_bytes,
pending_bytes,
busy,
log_frames,
checkpointed,
@@ -784,6 +851,10 @@ class MainDatabase:
try:
if wal:
enable_write_ahead_log(connection, wait_ms=busy_timeout_ms)
# Only the writer restarts the WAL, so only the writer needs the cap.
connection.execute(
f"PRAGMA journal_size_limit = {int(MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES)}"
)
_seal(connection)
if busy_timeout_ms != WRITE_LOCK_WAIT_MS:
# Opened at the caller's wait; every checkout starts at the default.
@@ -1027,6 +1098,18 @@ def main_database_transaction(
yield connection
def checkpoint_idle_main_databases() -> int:
"""TRUNCATE the WAL of every database nobody is using; how many were emptied."""
with _databases_lock:
databases = list(_alive.values())
emptied = 0
for database in databases:
result = database.checkpoint_if_idle()
if result is not None and result[0] == 0:
emptied += 1
return emptied
def close_main_databases() -> None:
"""Close every connection this process holds; used when the Core stops."""
with _databases_lock:
@@ -1050,6 +1133,9 @@ async def report_main_database(seconds: float) -> None:
while True:
await asyncio.sleep(seconds)
log_main_database_stats("periodic")
# The idle signal is this process's own checkouts: none held, none
# waiting. Off the loop, since a checkpoint is disk work however short.
await asyncio.to_thread(checkpoint_idle_main_databases)
def main_database_stats() -> dict[str, float | int]:
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.11"
HY_MEMORY_VERSION = "2.0.12"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -503,7 +503,7 @@ def _ensure_coding_schema(path: Path) -> None:
CodingMemoryStore,
)
# Relies on the pinned SDK's internals (hy-memory==2.0.11 is pinned exactly).
# Relies on the pinned SDK's internals (hy-memory==2.0.12 is pinned exactly).
store = CodingMemoryStore.__new__(CodingMemoryStore)
store._db_path = str(path)
store._conn = None
@@ -47,8 +47,10 @@ from one_memory.local_embedding import LocalEmbedder
from one_memory.log_file import confine_hy_sdk_logging
from one_memory.memory_features import (
FEATURES,
FEATURES_BY_NAME,
PARAMETERS,
SETTINGS_BY_NAME,
FeatureOverrides,
SettingValue,
feature_overrides,
)
from one_memory.model_profiles import (
@@ -321,21 +323,27 @@ def sdk_relevance_floor() -> float:
return min(1.0, max(0.0, value))
def sdk_feature_defaults() -> dict[str, bool]:
"""Each `memory_features` feature's SDK default, as a fresh config has it.
def sdk_feature_defaults() -> dict[str, SettingValue]:
"""Each `memory_features` setting's SDK default, as a fresh config has it.
That is the SDK's code default unless the process environment sets the
SDK's own variable (`FeatureSpec.env`), exactly what an unset feature
leaves on a client.
"""
config = MemoryConfig()
return {
defaults: dict[str, SettingValue] = {
spec.name: bool(getattr(getattr(config, spec.section), spec.field)) for spec in FEATURES
}
for parameter in PARAMETERS:
defaults[parameter.name] = float(
getattr(getattr(config, parameter.section), parameter.field)
)
return defaults
def apply_sdk_features(
config: MemoryConfig, features: Mapping[str, bool] | Iterable[tuple[str, bool]]
config: MemoryConfig,
features: Mapping[str, SettingValue] | Iterable[tuple[str, SettingValue]],
) -> MemoryConfig:
"""Set each feature a person chose on its `MemoryConfig` field.
@@ -346,7 +354,7 @@ def apply_sdk_features(
"""
values = dict(features.items() if isinstance(features, Mapping) else features)
for name, value in feature_overrides(values):
spec = FEATURES_BY_NAME[name]
spec = SETTINGS_BY_NAME[name]
setattr(getattr(config, spec.section), spec.field, value)
return config
@@ -383,7 +391,7 @@ def build_sdk_config(
*,
api_key: str | None,
data_dir: Path,
features: Mapping[str, bool] | Iterable[tuple[str, bool]] = (),
features: Mapping[str, SettingValue] | Iterable[tuple[str, SettingValue]] = (),
language: str | None = None,
) -> MemoryConfig:
"""Map a model snapshot and its embedding space to an SDK configuration.
@@ -436,6 +436,11 @@ def root_identity_matches(root: Path, expected: str) -> bool:
return _root_identity_matches_details(resolved, details, expected)
def resolved_source_root(root: Path) -> Path:
"""The real directory a root names (symlinks followed), as a grant records it."""
return _root_details(root)[0]
def _check_root_identity(root: Path, expected: str | None) -> None:
if expected is not None and not root_identity_matches(root, expected):
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
@@ -27,7 +27,7 @@ def discovered_off_loop(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
excluded_directory_names: frozenset[str] = frozenset(),
) -> AsyncIterator[SourceFileDiscoveryEvent]:
"""The same walk, for callers that have an event loop to keep free.
@@ -42,7 +42,7 @@ def discovered_off_loop(
grant,
suffixes=suffixes,
max_targets=max_targets,
excluded_top_directories=excluded_top_directories,
excluded_directory_names=excluded_directory_names,
),
scope_key=grant.source_id,
)
@@ -53,15 +53,17 @@ def discover_source_files(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
excluded_directory_names: frozenset[str] = frozenset(),
) -> Iterator[SourceFileDiscoveryEvent]:
"""Metadata only: one lstat per candidate, no open and no per-file grant lookup.
The grant and root identity are checked once before the walk. Each listed
file keeps the grant's check, which runs again when that file is opened;
the open itself refuses symlinks and any inode other than the one listed.
`excluded_top_directories` are directories directly under the root that
are never entered, such as another user's Hermes profiles.
`excluded_directory_names` are directory names never entered, at any
depth under the root, such as the `profiles` container that holds other
users' Hermes profiles. Any depth, because the granted root may sit above
the Hermes home (`~/.hermes` inside it is then `.hermes/profiles`).
"""
grant.check_root()
inspected = 0
@@ -69,12 +71,10 @@ def discover_source_files(
grant.root,
followlinks=False,
):
at_root = Path(directory) == grant.root
child_directories[:] = sorted(
child
for child in child_directories
if not (Path(directory) / child).is_symlink()
and not (at_root and child in excluded_top_directories)
if not (Path(directory) / child).is_symlink() and child not in excluded_directory_names
)
for filename in sorted(filenames):
if Path(filename).suffix.lower() not in suffixes:
@@ -209,6 +209,7 @@ class JsonlSourceAdapter:
file_predicate: Callable[[AuthorizedSourceFile], bool] | None = None,
suffixes: frozenset[str] = frozenset({".jsonl"}),
header_selector: Callable[[dict[str, JsonValue]], bool] | None = None,
header_gate: Callable[[dict[str, JsonValue] | None], bool] | None = None,
) -> None:
if descriptor.source_id != grant.source_id:
raise ValueError("Descriptor and authorized root source IDs must match")
@@ -219,6 +220,10 @@ class JsonlSourceAdapter:
self.file_predicate = file_predicate or (lambda _: True)
self.suffixes = suffixes
self.header_selector = header_selector
# False refuses the whole file before any record is read. It reports
# and leaves no cursor, so a release that learns the format reads the
# file from the top instead of finding it already behind a checkpoint.
self.header_gate = header_gate
self._header_cache: dict[str, tuple[str, dict[str, JsonValue] | None]] = {}
self._history_listing_cache: SourceListing | None = None
@@ -371,6 +376,9 @@ class JsonlSourceAdapter:
source_file, checkpoint, header = await asyncio.to_thread(
self._prepare, listed, options
)
if self.header_gate is not None and not self.header_gate(header):
yield ScanFailure(reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA)
continue
_log.debug(
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
self.descriptor.source_id,
@@ -574,6 +582,9 @@ class JsonlSourceAdapter:
# resumed scan starts past the header, and the records after it need
# it just as much as the first batch did.
header = await _read_header(reader, source_file, self.header_selector)
if self.header_gate is not None and not self.header_gate(header):
yield ScanFailure(reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA)
continue
last: SourceCursor = _start_cursor(source_file, options, checkpoint)
async for raw in iterate_in_worker(
partial(reader, source_file, checkpoint=checkpoint),
@@ -1,3 +1,4 @@
import re
from pathlib import PurePosixPath
from pydantic import JsonValue
@@ -37,6 +38,19 @@ DEEPSEEK_HARNESS_DESCRIPTOR = SourceDescriptor(
),
)
#: One session is one file, and the name carries the format generation it was
#: written in: `session.jsonl[.zstd]` for the first format, then
#: `session.v3.jsonl.zstd` and `session.v4.jsonl.zstd`. The version segment is
#: part of the published on-disk contract, so discovery must accept it — the
#: earlier exact-name match imported none of the versioned files.
SESSION_FILE = re.compile(r"^session(?:\.v[0-9]+)?\.jsonl(?:\.zstd)?$")
#: Session generations this parser has been verified against: the first format
#: (header `version: 0`), v3 and v4. Discovery is intentionally wider than
#: parsing: a future generation is found by pattern and then skipped from its
#: header, so it is reported as unsupported instead of silently vanishing.
SUPPORTED_SESSION_VERSIONS = frozenset({0, 3, 4})
def create_deepseek_harness_adapter(
grant: AuthorizedSourceRoot,
@@ -49,25 +63,43 @@ def create_deepseek_harness_adapter(
grant=grant,
parser=parse_deepseek_harness_record,
source_version=source_version,
file_predicate=lambda source_file: (
source_file.relative_path.name in {"session.jsonl", "session.jsonl.zstd"}
),
file_predicate=lambda source_file: bool(SESSION_FILE.match(source_file.relative_path.name)),
suffixes=frozenset({".jsonl", ".zstd"}),
header_selector=_is_session_header,
header_gate=_supported_session,
)
def _is_session_header(record: dict[str, JsonValue]) -> bool:
"""Harness writes the session's `cwd` once, in the first line only.
Verified against a session this machine produced: the header is
`{"type":"session","version":0,"id":...,"cwd":"/private/tmp/dsh-verify"}`
and no later record repeats it. Read per-record, `cwd` is therefore always
absent and every turn imports with no workspace.
Verified against sessions this machine produced: the header is
`{"type":"session","version":0|4,"id":...,"cwd":...}` and no later record
repeats it. Read per-record, `cwd` is therefore always absent and every turn
imports with no workspace.
"""
return record.get("type") == "session"
def _supported_session(header: dict[str, JsonValue] | None) -> bool:
"""Whether the session's declared format generation is one we can read.
A missing header, or a header without `version`, is the first format — the
earliest files this adapter already read. Any declared version outside
`SUPPORTED_SESSION_VERSIONS` is refused rather than interpreted with today's
field rules: a future format may change field semantics, and importing it
wrongly is harder to undo than not importing it yet.
"""
if header is None:
return True
version = header.get("version")
if version is None:
return True
if not isinstance(version, int) or isinstance(version, bool):
return False
return version in SUPPORTED_SESSION_VERSIONS
def parse_deepseek_harness_record(
record: dict[str, JsonValue],
context: SourceRecordContext,
@@ -80,6 +112,11 @@ def parse_deepseek_harness_record(
)
event_type = record.get("type")
data = as_record(record.get("data"))
if not _supported_session(context.header):
return ParsedSkip(
reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA,
conversation_id=conversation_id,
)
if event_type == "session":
return ParsedSkip(reason_code=ScanIssueCode.UNKNOWN_RECORD)
if data is None:
@@ -148,11 +185,18 @@ def _tool_message(
"""`tool/call` carries name and arguments; `tool/result` a user-role message.
The same call also appears as a `tool-call` block in `assistant/message`;
only the event is read, so it is stored once.
only the event is read, so it is stored once. The first format and v3 wrap
the result in `tool-result` blocks; v4 writes plain `text` blocks, so both
shapes are read through the same clipping and credential-scrubbing path.
"""
message = as_record(data.get("message"))
if record.get("type") == "tool/result" and message is not None:
text = tool_blocks_evidence(message.get("content"), block_types=frozenset({"tool-result"}))
if text is None:
text = tool_blocks_evidence(
[{"type": "tool-result", "content": message.get("content")}],
block_types=frozenset({"tool-result"}),
)
message_id = first_string(message.get("id"), data.get("callId"))
else:
text = tool_evidence_text(data)
@@ -1,9 +1,9 @@
import json
import re
import sqlite3
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Iterable, Iterator
from functools import partial
from pathlib import PurePosixPath
from pathlib import Path, PurePosixPath
from pydantic import JsonValue
@@ -88,7 +88,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=10_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
excluded_directory_names=frozenset({HERMES_PROFILES_DIRECTORY}),
):
if isinstance(item, AuthorizedSourceFile) and _is_hermes_target(
item.relative_path.as_posix()
@@ -102,7 +102,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=options.max_scan_targets or 100_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
excluded_directory_names=frozenset({HERMES_PROFILES_DIRECTORY}),
):
yield ScanProgressEvent(
phase=ScanPhase.DISCOVER,
@@ -253,16 +253,62 @@ def parse_hermes_rollout_record(
#: own `state.db` and sessions. A profile is another person's history: it is
#: read only through its own grant (its own MemHub user), never as part of the
#: root above it. Hermes' own export skips this directory for the same reason.
#:
#: Skipped at any depth, not only directly under the root: a root granted
#: above the Hermes home holds `.hermes/profiles/`. Hermes keeps nothing of
#: its own under a directory of that name (its history is `state.db` and
#: `sessions/`), so nothing real is lost.
HERMES_PROFILES_DIRECTORY = "profiles"
def _is_hermes_target(relative_path: str) -> bool:
path = PurePosixPath(relative_path)
if path.parts and path.parts[0] == HERMES_PROFILES_DIRECTORY:
if HERMES_PROFILES_DIRECTORY in path.parts[:-1]:
return False
return path.name == "state.db" or (path.suffix == ".jsonl" and "sessions" in path.parts)
def hermes_root_is_profiles_container(root: Path) -> bool:
"""`profiles` itself holds everyone's profiles; it is never one person's history."""
return root.name == HERMES_PROFILES_DIRECTORY
def hermes_root_conflict(mine: Path, other_users_roots: Iterable[Path]) -> str | None:
"""Why the resolved root `mine` may not be granted beside other users' Hermes roots.
Paths are compared resolved (symlinks followed; the other users' roots are
resolved here). Two roots that are equal or nest would read the same files
for two users, except when one is exactly the other's `profiles/<name>`:
the scan above never enters `profiles/`, so the two stay apart (`local` on
`~/.hermes`, a user on `~/.hermes/profiles/<name>`). None when it may.
"""
if hermes_root_is_profiles_container(mine):
return HERMES_ROOT_IS_PROFILES_CONTAINER
for other in other_users_roots:
theirs = _real(other)
if mine == theirs:
return HERMES_ROOT_OVERLAPS_OTHER_USER
for parent, child in ((mine, theirs), (theirs, mine)):
if parent in child.parents:
relative = child.relative_to(parent).parts
if len(relative) != 2 or relative[0] != HERMES_PROFILES_DIRECTORY:
return HERMES_ROOT_OVERLAPS_OTHER_USER
return None
#: 422: the root is the `profiles` container, not a profile.
HERMES_ROOT_IS_PROFILES_CONTAINER = "hermes_root_is_profiles_container"
#: 409: the root is, holds or sits inside another user's Hermes root.
HERMES_ROOT_OVERLAPS_OTHER_USER = "hermes_root_overlaps_other_user"
def _real(path: Path) -> Path:
try:
return path.expanduser().resolve(strict=False)
except (OSError, RuntimeError):
return path.expanduser().absolute()
def _read_rollout(
source_file: AuthorizedSourceFile,
options: ScanOptions,
@@ -34,6 +34,7 @@ from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.model_backoff import next_retry_at
from one_memory.secrets import SecretStoreUnavailableError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
@@ -77,6 +78,12 @@ class MemoryAddReceipt(BaseModel):
task_id: str | None = None
memory_ids: list[str] = Field(default_factory=list)
error: str | None = None
#: Why a note still being worked on last failed (`model_auth_failed`, ...),
#: or why a failed one failed. A provider failure keeps it `pending`.
error_code: str | None = None
#: When a pending note is next tried: its own retry wait, or the model
#: provider's pause while the provider is failing.
next_retry_at: str | None = None
class LegacyMemoryAdditionReader:
@@ -258,6 +265,8 @@ class MemoryAdditionStore:
durable_state = durable.state if durable is not None else "failed"
mapped: MemoryAddState = {
"pending": "pending",
# Re-run automatically within seconds (`recover_unknown`).
"unknown": "pending",
"running": "running",
"waiting_for_model": "waiting_for_model",
"waiting_for_permission": "waiting_for_permission",
@@ -272,6 +281,12 @@ class MemoryAdditionStore:
state=mapped,
memory_ids=(list(durable.sdk_memory_ids) if durable is not None else []),
error=durable.detail if durable is not None else "write_missing",
error_code=(
durable.error_code
if durable is not None and durable.state != "completed"
else None
),
next_retry_at=durable.retry_at if durable is not None else None,
)
if self.processor is None:
raise RuntimeError("legacy_memory_add_processor_unavailable")
@@ -402,10 +417,15 @@ def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryAddReceipt:
try:
return store(request).receipt_for_request(request_id, user_id=user_id)
receipt = store(request).receipt_for_request(request_id, user_id=user_id)
except KeyError as error:
raise HTTPException(404, "memory_add_not_found") from error
except ValueError as error:
raise HTTPException(409, str(error)) from error
return receipt.model_copy(
update={
"next_retry_at": next_retry_at(request.app, receipt.state, receipt.next_retry_at)
}
)
return router
@@ -21,6 +21,11 @@ exercises (query rewrite needs `queries=[...]`, session narratives need
cross-encoder download, access write-back); and everything MemHub already
controls elsewhere (pipeline log, trace, memory operations, telemetry, mode).
Beside the on/off features there is one numeric parameter,
`intention_dedup_threshold` (`PARAMETERS`): the cosine similarity at which the
read-time intention dedup treats two intentions as copies. It is set, stored,
reset and keyed exactly like a feature.
No SDK import here: only `hy_sdk_runtime` and `hy_sdk_migration` import it.
`config_state` imports this module, so it imports nothing from there.
"""
@@ -31,7 +36,7 @@ from collections.abc import Mapping
from dataclasses import dataclass
from typing import Literal
from pydantic import BaseModel, ConfigDict, StrictBool
from pydantic import BaseModel, ConfigDict, Field, StrictBool
MemoryFeatureName = Literal[
"graph_rerank",
@@ -40,8 +45,13 @@ MemoryFeatureName = Literal[
"session_episodes",
"episode_summary",
"intention",
"intention_dedup",
"few_shot",
]
#: Numeric settings beside the on/off features.
MemoryParameterName = Literal["intention_dedup_threshold"]
#: Every key of the setting: a feature or a parameter.
MemorySettingName = MemoryFeatureName | MemoryParameterName
#: Read features change search; write features change what extraction stores.
FeatureSide = Literal["read", "write"]
#: none: no extra call. extra_query: one more store query per search.
@@ -49,9 +59,25 @@ FeatureSide = Literal["read", "write"]
#: a longer extraction prompt.
FeatureCost = Literal["none", "extra_query", "llm_call", "prompt_tokens"]
#: Why an enabled feature does nothing right now (`active: false`).
FeatureReason = Literal["disabled", "requires_pro", "requires_ultra", "requires_graph_rerank"]
FeatureReason = Literal[
"disabled",
"requires_pro",
"requires_ultra",
"requires_graph_rerank",
"requires_intention_dedup",
]
#: A feature is on or off; a parameter is a number.
type SettingValue = bool | float
#: A client's features as a hashable key: sorted (name, value) pairs, set ones only.
type FeatureOverrides = tuple[tuple[str, bool], ...]
type FeatureOverrides = tuple[tuple[str, SettingValue], ...]
#: The range a person may set `intention_dedup_threshold` to. With dedup on,
#: the SDK deletes every intention at or above it from the store at read time,
#: which cannot be undone; below 0.8 the embedding models MemHub uses score
#: different plans on one topic as that close, so a lower value would delete
#: intentions that are not copies. 1.0 keeps only exact copies' dedup.
INTENTION_DEDUP_THRESHOLD_MIN = 0.8
INTENTION_DEDUP_THRESHOLD_MAX = 1.0
_MODE_RANK: dict[str, int] = {"lite": 0, "pro": 1, "ultra": 2}
@@ -169,6 +195,23 @@ FEATURES: tuple[FeatureSpec, ...] = (
"No extra call."
),
),
FeatureSpec(
name="intention_dedup",
section="recall",
field="intention_dedup_enabled",
env="MEMORY_INTENTION_DEDUP_ENABLED",
side="read",
requires_mode="pro",
requires_feature=None,
cost="none",
description=(
"Treat same-day intentions (L7) with the same expiry and a cosine "
"similarity at or above intention_dedup_threshold as copies: a "
"search returns the earliest and deletes the others from the store. "
"On by default; off hides, drops and deletes no intention, at read "
"time or in a write batch. No extra call."
),
),
FeatureSpec(
name="few_shot",
section="llm",
@@ -189,12 +232,57 @@ FEATURE_NAMES: tuple[MemoryFeatureName, ...] = tuple(spec.name for spec in FEATU
FEATURES_BY_NAME: dict[str, FeatureSpec] = {spec.name: spec for spec in FEATURES}
@dataclass(frozen=True)
class ParameterSpec:
"""A numeric setting; it acts only while `requires_feature` acts."""
name: MemoryParameterName
section: Literal["recall"]
field: str
env: str
minimum: float
maximum: float
requires_feature: MemoryFeatureName
description: str
@property
def sdk_field(self) -> str:
return f"{self.section}.{self.field}"
PARAMETERS: tuple[ParameterSpec, ...] = (
ParameterSpec(
name="intention_dedup_threshold",
section="recall",
field="intention_dedup_threshold",
env="MEMORY_INTENTION_DEDUP_THRESHOLD",
minimum=INTENTION_DEDUP_THRESHOLD_MIN,
maximum=INTENTION_DEDUP_THRESHOLD_MAX,
requires_feature="intention_dedup",
description=(
"Cosine similarity at or above which intention_dedup treats two "
"intentions as copies. Higher keeps more. The SDK default is 0.95 "
"(MEMORY_DEDUP_THRESHOLD also sets it when its own env var is unset)."
),
),
)
PARAMETER_NAMES: tuple[MemoryParameterName, ...] = tuple(spec.name for spec in PARAMETERS)
PARAMETERS_BY_NAME: dict[str, ParameterSpec] = {spec.name: spec for spec in PARAMETERS}
#: Every key of the setting, features first, each in its catalog order.
SETTING_NAMES: tuple[MemorySettingName, ...] = (*FEATURE_NAMES, *PARAMETER_NAMES)
SETTINGS_BY_NAME: dict[str, FeatureSpec | ParameterSpec] = {
**FEATURES_BY_NAME,
**PARAMETERS_BY_NAME,
}
class MemoryFeatures(BaseModel):
"""`config.json` `memory_features`, and the `PUT` body.
Every key is optional; absent (or null) means the SDK default. In a `PUT`
an absent key is left as it is and null resets it. Unknown keys and
non-boolean values are refused (422).
an absent key is left as it is and null resets it. Unknown keys,
non-boolean feature values and a threshold that is not a number in its
range are refused (422).
"""
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
@@ -205,7 +293,14 @@ class MemoryFeatures(BaseModel):
session_episodes: StrictBool | None = None
episode_summary: StrictBool | None = None
intention: StrictBool | None = None
intention_dedup: StrictBool | None = None
few_shot: StrictBool | None = None
intention_dedup_threshold: float | None = Field(
default=None,
ge=INTENTION_DEDUP_THRESHOLD_MIN,
le=INTENTION_DEDUP_THRESHOLD_MAX,
allow_inf_nan=False,
)
def overrides(self) -> FeatureOverrides:
"""The features a person set, as the client key."""
@@ -221,24 +316,47 @@ class MemoryFeatures(BaseModel):
return MemoryFeatures(**{key: value for key, value in values.items() if value is not None})
def feature_overrides(values: Mapping[str, bool]) -> FeatureOverrides:
unknown = set(values) - set(FEATURES_BY_NAME)
def feature_overrides(values: Mapping[str, SettingValue]) -> FeatureOverrides:
unknown = set(values) - set(SETTINGS_BY_NAME)
if unknown:
raise ValueError("unknown_memory_feature")
return tuple(sorted((name, bool(value)) for name, value in values.items()))
return tuple(sorted((name, _setting_value(name, value)) for name, value in values.items()))
def _setting_value(name: str, value: SettingValue) -> SettingValue:
parameter = PARAMETERS_BY_NAME.get(name)
if parameter is None:
return bool(value)
if isinstance(value, bool) or not parameter.minimum <= float(value) <= parameter.maximum:
raise ValueError("invalid_memory_feature_value")
return float(value)
def effective_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""Every feature's value: what a person set, else the SDK default."""
values: dict[str, bool] = {spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES}
overrides: FeatureOverrides, defaults: Mapping[str, SettingValue]
) -> dict[str, SettingValue]:
"""Every setting's value: what a person set, else the SDK default."""
values: dict[str, SettingValue] = {
spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES
}
for parameter in PARAMETERS:
if parameter.name in defaults:
values[parameter.name] = float(defaults[parameter.name])
values.update(dict(overrides))
return values
def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> FeatureReason | None:
"""None when the feature acts in this mode; else why it does not."""
def feature_activity(
name: str, effective: Mapping[str, SettingValue], mode: str
) -> FeatureReason | None:
"""None when the feature acts in this mode; else why it does not.
A parameter acts exactly when the feature it tunes acts.
"""
parameter = PARAMETERS_BY_NAME.get(name)
if parameter is not None:
reason = feature_activity(parameter.requires_feature, effective, mode)
return "requires_intention_dedup" if reason == "disabled" else reason
spec = FEATURES_BY_NAME[name]
if not effective.get(name, False):
return "disabled"
@@ -252,11 +370,12 @@ def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> Fea
def non_default_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""The features whose value differs from the SDK default, for status."""
overrides: FeatureOverrides, defaults: Mapping[str, SettingValue]
) -> dict[str, SettingValue]:
"""The settings whose value differs from the SDK default, for status."""
baseline = effective_features((), defaults)
return {
name: value
for name, value in effective_features(overrides, defaults).items()
if value != bool(defaults.get(name, False))
if value != baseline.get(name)
}
@@ -21,6 +21,7 @@ import shutil
import sqlite3
import time as _time
from collections.abc import Awaitable, Callable
from contextlib import AbstractContextManager, nullcontext
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -52,6 +53,7 @@ from one_memory.memory_tasks import (
from one_memory.memory_tasks import (
POLICY_STOPPED_CODES as _TASK_POLICY_STOPPED_CODES,
)
from one_memory.model_backoff import ModelAttempt
from one_memory.sdk_migration_log import error_chain_text
from one_memory.service_levels import ServiceLevel, service_level_context
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, other_user_lane_prefix
@@ -107,6 +109,43 @@ PARKED_STATES = (WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
#: re-admitted on a timer. The gap keeps a still-down model from spinning.
DEFAULT_MODEL_READMIT_AFTER = timedelta(minutes=1)
#: How long a job the execute gate handed back (`release_unstarted`) is left
#: alone before it can be claimed again.
RELEASE_HOLD = timedelta(seconds=5)
#: Automatic retries of a job whose attempt failed for a reason other than the
#: model provider: content the model or the SDK refused, a parse error, a
#: crash or an interruption mid-write. After the last one the job is `failed`,
#: shown with its code, and the queue moves on. A provider failure (a broken
#: key, a model that is gone, a provider that is down) never counts: the job
#: waits in `pending` for as long as the provider is broken, paced by
#: `model_backoff`.
JOB_RETRY_LIMIT = 3
JOB_RETRY_FIRST_BACKOFF = timedelta(seconds=30)
JOB_RETRY_BACKOFF_CAP = timedelta(minutes=3)
#: The code a job interrupted mid-write carries when it has no better one.
INTERRUPTED_ERROR_CODE = "durable_write_interrupted"
def job_retry_backoff(retry: int) -> timedelta:
"""The wait before automatic retry number `retry` (1-based)."""
wait: timedelta = JOB_RETRY_FIRST_BACKOFF * (2 ** min(max(retry - 1, 0), 16))
return min(wait, JOB_RETRY_BACKOFF_CAP)
def sdk_request_id(idempotency_key: str, attempt: int) -> str:
"""The SDK request id of a job's `attempt`-th write.
The SDK keeps one outcome per request id and replays it: a write that
failed under an id fails under it forever, without calling the model.
Every re-run therefore gets an id of its own. Attempt 0 is the job's own
key, so every job written before this existed keeps the id its outcome
was recorded under.
"""
return idempotency_key if attempt <= 0 else f"{idempotency_key}#r{attempt}"
#: Production drain-budget defaults. These are the single source consumed by
#: the composition root; callers must not invent a second formula or copy a
#: test probe's numbers into the product path.
@@ -337,6 +376,10 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
("time_quality", "TEXT NOT NULL DEFAULT 'approximate'"),
("sdk_raw_ids_json", "TEXT NOT NULL DEFAULT '[]'"),
("sdk_memory_ids_json", "TEXT NOT NULL DEFAULT '[]'"),
#: Which SDK request id the next write uses (`sdk_request_id`).
("sdk_attempt", "INTEGER NOT NULL DEFAULT 0"),
#: Automatic retries spent (`JOB_RETRY_LIMIT`); provider failures spend none.
("retries", "INTEGER NOT NULL DEFAULT 0"),
)
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
@@ -381,6 +424,10 @@ class DurableWriteRecord:
detail: str = ""
sdk_raw_ids: tuple[str, ...] = ()
sdk_memory_ids: tuple[str, ...] = ()
#: The stable code of the last failure, if the row carries one.
error_code: str | None = None
#: When a waiting row may next be claimed, if a retry wait holds it.
retry_at: str | None = None
@dataclass(frozen=True)
@@ -746,10 +793,10 @@ class DurableWriteQueue:
"""Recover expired leases on the only engine D56 permits.
A lease whose owner died before HY was invoked is safe to retry. Once
`hy_invoked` is recorded, the outcome may already exist in HY and its
requests have no idempotency key, so that row becomes `unknown`
instead of `pending`. Unknown is intentionally sticky: a later reclaim
pass cannot turn it back into retryable work.
`hy_invoked` is recorded the write may already have landed, so that
row becomes `unknown` instead of `pending`. `unknown` is only a stop
on the way: `recover_unknown` re-runs it under a new SDK request id,
after asking the SDK whether the old one succeeded.
"""
now = datetime.now(UTC).isoformat()
changed = 0
@@ -1509,13 +1556,15 @@ class DurableWriteQueue:
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
_require_queue_schema(connection)
row = connection.execute(
"SELECT job_id, state, created_at, updated_at, last_error, "
"SELECT job_id, state, created_at, updated_at, last_error, lease_expires_at, "
"sdk_raw_ids_json,sdk_memory_ids_json "
"FROM memory_write_jobs WHERE idempotency_key=?",
(request_id,),
).fetchone()
if row is None:
return None
last_error = str(row["last_error"] or "")
retry_at = row["lease_expires_at"]
return DurableWriteRecord(
job_id=str(row["job_id"]),
request_id=request_id,
@@ -1526,6 +1575,14 @@ class DurableWriteQueue:
detail=str(row["last_error"] or ""),
sdk_raw_ids=tuple(json.loads(str(row["sdk_raw_ids_json"]))),
sdk_memory_ids=tuple(json.loads(str(row["sdk_memory_ids_json"]))),
error_code=last_error if _ERROR_CODE.fullmatch(last_error) else None,
retry_at=(
str(retry_at)
if str(row["state"]) == PENDING
and retry_at is not None
and str(retry_at) > datetime.now(UTC).isoformat()
else None
),
)
def payload_for(self, job_id: str) -> WriteRequest | None:
@@ -1726,9 +1783,23 @@ class DurableWriteQueue:
)
return updated.rowcount == 1
def release_unstarted(self, job_id: str, *, owner: str | None = None) -> bool:
"""Return a claimed job to pending when its source was paused pre-HY."""
now = datetime.now(UTC).isoformat()
def release_unstarted(
self,
job_id: str,
*,
owner: str | None = None,
hold: timedelta = RELEASE_HOLD,
) -> bool:
"""Return a claimed job to pending when its source was paused pre-HY.
The row comes back with its lease still set `hold` into the future.
Claim only takes a pending row whose lease has expired, so a job the
execute gate just refused is not claimed again on the very next tick:
when the claim gate and the execute gate disagree for good (a
pre-upgrade payload whose Evidence belongs to another user than its
lane), the job is retried every few seconds instead of in a tight loop.
"""
now = datetime.now(UTC)
where = "job_id=? AND state=? AND hy_invoked=0"
params: tuple[Any, ...] = (job_id, RUNNING)
if owner is not None:
@@ -1738,11 +1809,175 @@ class DurableWriteQueue:
updated = connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
"lease_expires_at=NULL,updated_at=? WHERE " + where,
(PENDING, now, *params),
"lease_expires_at=?,updated_at=? WHERE " + where,
(PENDING, (now + hold).isoformat(), now.isoformat(), *params),
)
return updated.rowcount == 1
def sdk_attempt(self, job_id: str) -> int:
"""Which SDK request id this job's next write uses (`sdk_request_id`)."""
with main_database(self.database_path).read() as connection:
row = connection.execute(
"SELECT sdk_attempt FROM memory_write_jobs WHERE job_id=?", (job_id,)
).fetchone()
return int(row[0]) if row is not None else 0
def requeue_for_model(
self,
job_id: str,
*,
error_code: str,
owner: str | None = None,
) -> bool:
"""Put a job the model provider failed back in line, as it was.
The provider is broken, not the job: it keeps its place in the queue
(`event_at`, `queue_seq`), spends no attempt and no retry, and is
claimed again once `model_backoff` lets a probe through. Whatever the
SDK may have kept of the failed write stays under that write's request
id; the next one gets an id of its own (`sdk_attempt`), so the SDK
does not replay the failure instead of calling the model.
"""
now = datetime.now(UTC).isoformat()
where = "job_id=? AND state=?"
params: tuple[Any, ...] = (job_id, RUNNING)
if owner is not None:
where += " AND lease_owner=?"
params += (owner,)
with main_database(self.database_path).write() as connection, connection:
updated = connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,lease_expires_at=NULL,"
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
"sdk_attempt=sdk_attempt+hy_invoked,hy_invoked=0,"
"last_error=?,updated_at=? WHERE " + where,
(PENDING, error_code, now, *params),
)
return updated.rowcount == 1
def retry_or_fail(
self,
job_id: str,
*,
error_code: str,
owner: str | None = None,
now: datetime | None = None,
) -> str:
"""Retry a job that failed for a reason of its own, or give up on it.
Up to `JOB_RETRY_LIMIT` retries, each after a longer wait
(`job_retry_backoff`), held by the lease so the claim skips the job
until then. After the last one the job is `failed`: shown with its
code, out of the queue's way. A write that reached the SDK is re-run
under a new request id. Returns the state written, or "" when the row
is no longer this owner's.
"""
now = now or datetime.now(UTC)
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
row = connection.execute(
"SELECT state,lease_owner,retries FROM memory_write_jobs WHERE job_id=?",
(job_id,),
).fetchone()
if (
row is None
or str(row["state"]) != RUNNING
or (owner is not None and row["lease_owner"] != owner)
):
return ""
retry = int(row["retries"]) + 1
if retry > JOB_RETRY_LIMIT:
connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,lease_expires_at=NULL,"
"last_error=?,updated_at=? WHERE job_id=? AND state=?",
(FAILED, error_code, now.isoformat(), job_id, RUNNING),
)
return FAILED
connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,lease_expires_at=?,"
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
"sdk_attempt=sdk_attempt+hy_invoked,hy_invoked=0,retries=?,"
"last_error=?,updated_at=? WHERE job_id=? AND state=?",
(
PENDING,
(now + job_retry_backoff(retry)).isoformat(),
retry,
error_code,
now.isoformat(),
job_id,
RUNNING,
),
)
return PENDING
def recover_unknown(
self,
*,
in_flight: Callable[[str], bool] = lambda _request_id: False,
limit: int = 128,
now: datetime | None = None,
) -> int:
"""Move every `unknown` job back into the queue, or out of it.
A job is `unknown` when its write may have reached the SDK and its
worker never learned how it ended: the process stopped, the lease
lapsed, or the work ceiling gave up on it. It is re-run, counted as a
retry so a crash loop ends in `failed`. The re-run gets a new SDK
request id and, before it starts, asks the SDK how the previous id
ended (`execute_job`): a write that did succeed is completed from its
receipt instead of being made twice.
Jobs from before this existed (upgrade) are handled the same way,
once: from then on they follow the normal rules. A write whose SDK
call is still running in this process (`in_flight`) is left until it
ends. A legacy row whose write is known to have landed against an
authority Core could not verify (`NON_RETRYABLE_POST_WRITE_CODES`) is
never re-run: it is `failed`, which says so.
"""
now = now or datetime.now(UTC)
changed = 0
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id,idempotency_key,sdk_attempt,retries,last_error "
"FROM memory_write_jobs WHERE state=? ORDER BY queue_seq LIMIT ?",
(UNKNOWN, limit),
).fetchall()
for row in rows:
if in_flight(sdk_request_id(str(row["idempotency_key"]), int(row["sdk_attempt"]))):
continue
last_error = str(row["last_error"] or "")
code = last_error if _ERROR_CODE.fullmatch(last_error) else INTERRUPTED_ERROR_CODE
retry = int(row["retries"]) + 1
if last_error in NON_RETRYABLE_POST_WRITE_CODES or retry > JOB_RETRY_LIMIT:
changed += connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"lease_expires_at=NULL,last_error=?,updated_at=? "
"WHERE job_id=? AND state=?",
(FAILED, code, now.isoformat(), str(row["job_id"]), UNKNOWN),
).rowcount
continue
changed += connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,lease_expires_at=?,"
"attempts=0,sdk_attempt=sdk_attempt+1,hy_invoked=0,retries=?,"
"last_error=?,updated_at=? WHERE job_id=? AND state=?",
(
PENDING,
(now + job_retry_backoff(retry)).isoformat(),
retry,
code,
now.isoformat(),
str(row["job_id"]),
UNKNOWN,
),
).rowcount
if changed:
logger.info("durable_write_unknown_recovered count=%d", changed)
return changed
def settle_worker_error(self, job_id: str, *, error_code: str, owner: str | None = None) -> str:
"""Fallback settlement when the injected executor raises past its guard.
@@ -2008,60 +2243,6 @@ class DurableWriteQueue:
)
return str(target)
def retry_unknown_confirmed(self, job_id: str) -> int:
"""Requeue one unknown write only after an explicit user decision.
Automatic workers may never move ``unknown`` back to pending because
HY may already contain the result. This separate command boundary is
intentionally named for the confirmation it requires. The durable
job and any task rows created from the same Evidence are reset in one
short transaction, so the unified queue remains the only execution
entry and a half-reset attempt cannot become permanently stranded.
"""
now = datetime.now(UTC).isoformat()
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
row = connection.execute(
"SELECT state,last_error FROM memory_write_jobs WHERE job_id=?", (job_id,)
).fetchone()
if (
row is None
or str(row["state"]) != UNKNOWN
or str(row["last_error"]) in NON_RETRYABLE_POST_WRITE_CODES
):
return 0
# After the SDK cutover the legacy task tables are gone and the
# job is the only row to reset.
if not self.sdk_authority and _legacy_task_links_present(connection):
connection.execute(
"""
UPDATE memory_tasks
SET status='pending', runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE status='unknown' AND task_id IN (
SELECT inputs.task_id
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE linked.job_id=?
)
""",
(now, job_id),
)
changed = connection.execute(
"""
UPDATE memory_write_jobs
SET state='pending', attempts=0, lease_owner=NULL,
lease_expires_at=NULL, hy_invoked=0,
last_error='model_content_manual_retry_requested', updated_at=?
WHERE job_id=? AND state='unknown'
""",
(now, job_id),
).rowcount
return changed
def readmit_cancelled_permission_confirmed(self, job_id: str, request: WriteRequest) -> int:
"""Requeue one unsent permission cancellation after explicit consent.
@@ -2114,115 +2295,6 @@ class DurableWriteQueue:
).rowcount
return changed
def retry_timed_out_unknown_after_reconcile(
self,
*,
older_than: datetime,
limit: int = 32,
) -> int:
"""Requeue bounded timeout unknowns only after the caller queried HY.
This is not the general unknown retry path. Only a timed-out write with
attempts remaining is eligible, and attempts are preserved so repeated
timeouts stop at the durable ceiling instead of looping forever.
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = datetime.now(UTC).isoformat()
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id FROM memory_write_jobs "
"WHERE state=? AND last_error='hy_memory_timeout' "
"AND attempts < max_attempts AND updated_at <= ? "
"ORDER BY updated_at, job_id LIMIT ?",
(UNKNOWN, older_than.isoformat(), limit),
).fetchall()
job_ids = tuple(str(row["job_id"]) for row in rows)
if not job_ids:
return 0
placeholders = ",".join("?" for _ in job_ids)
connection.execute(
f"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
f"lease_expires_at=NULL, hy_invoked=0, "
f"last_error='model_content_reconciled_absent_retry', updated_at=? "
f"WHERE state=? AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_reconciled_absent_retry', updated_at=? "
f"WHERE status=? AND task_id IN ("
f"SELECT inputs.task_id FROM memory_task_inputs AS inputs "
f"JOIN memory_write_job_evidence AS linked ON EXISTS ("
f"SELECT 1 FROM json_each(inputs.refs_json) AS ref "
f"WHERE json_extract(ref.value, '$.evidence_id')=linked.evidence_id) "
f"WHERE linked.job_id IN ({placeholders}))",
(MemoryTaskStatus.PENDING, now, MemoryTaskStatus.UNKNOWN, *job_ids),
)
return len(job_ids)
def retry_first_unknown_after_reconcile(
self,
*,
older_than: datetime,
limit: int = 32,
) -> int:
"""Retry one unresolved unknown outcome once, after reconciliation.
The caller first asks HY and checks linked governed output. Rows still
unknown may have lost the response after the SDK call, so one bounded
retry is useful. `attempts=1` is the guard: the retry claim increments
it to two, and a second unknown is left for explicit user recovery.
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = datetime.now(UTC).isoformat()
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id FROM memory_write_jobs "
"WHERE state=? AND attempts=1 AND attempts < max_attempts "
f"AND COALESCE(last_error, '') NOT IN "
f"({','.join('?' for _ in NON_RETRYABLE_POST_WRITE_CODES)}) "
"AND updated_at <= ? ORDER BY updated_at, job_id LIMIT ?",
(
UNKNOWN,
*NON_RETRYABLE_POST_WRITE_CODES,
older_than.isoformat(),
limit,
),
).fetchall()
job_ids = tuple(str(row["job_id"]) for row in rows)
if not job_ids:
return 0
placeholders = ",".join("?" for _ in job_ids)
connection.execute(
f"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
f"lease_expires_at=NULL, hy_invoked=0, "
f"last_error='model_content_unknown_auto_retry', updated_at=? "
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
f"WHERE status=? AND task_id IN ("
f"SELECT inputs.task_id FROM memory_task_inputs AS inputs "
f"JOIN memory_write_job_evidence AS linked ON EXISTS ("
f"SELECT 1 FROM json_each(inputs.refs_json) AS ref "
f"WHERE json_extract(ref.value, '$.evidence_id')=linked.evidence_id) "
f"WHERE linked.job_id IN ({placeholders}))",
(MemoryTaskStatus.PENDING, now, MemoryTaskStatus.UNKNOWN, *job_ids),
)
return len(job_ids)
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
"""Close an unknown durable row when local governance output proves success."""
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
@@ -2361,7 +2433,9 @@ class DurableWriteRegistry:
# distinguishing detail. Parked and cancelled states keep their own
# names — collapsing an authorization stop into `failed` is exactly the
# pollution the task layer already rejects.
if record.state in (PENDING, RUNNING):
if record.state in (PENDING, RUNNING, UNKNOWN):
# `unknown` no longer waits for anyone: the drain re-runs it
# within seconds (`recover_unknown`), so to a caller it is queued.
state = "accepted"
elif record.state == DEAD:
state = "failed"
@@ -2385,6 +2459,8 @@ class DurableWriteRegistry:
detail=detail,
raw_ids=record.sdk_raw_ids,
memory_ids=record.sdk_memory_ids,
error_code=record.error_code if record.state != COMPLETED else None,
next_retry_at=record.retry_at,
)
def submit(
@@ -2466,12 +2542,22 @@ async def execute_job(
source_writes_permitted: Callable[[str], bool] | None = None,
admit_hy_invocation: Callable[[str, Callable[[], Awaitable[bool]]], Awaitable[bool]]
| None = None,
model_failures: Callable[[], AbstractContextManager[ModelAttempt | None]] | None = None,
) -> None:
"""Run one durable job through the SDK-authority processor.
``process_sdk_job`` owns the one public SDK write. ``hy_invoked`` is
recorded immediately before that call, so a crash mid-write is later
reclaimed to ``unknown`` instead of being retried blindly.
reclaimed to ``unknown`` and re-run (`recover_unknown`).
Each write uses the SDK request id of the job's current attempt
(`sdk_request_id`). A re-run first asks the SDK how the previous id
ended, and completes from its receipt if it succeeded.
``model_failures`` (`ModelBackoff.attempt`) says whether the provider
failed while this write was in flight: such a job goes back to `pending`
as it was and waits for the provider. Any other failure is retried
`JOB_RETRY_LIMIT` times and then `failed`.
"""
# Every store call here is synchronous SQLite work on the single writer or
# a reader; made from the loop it would block health checks and the API for
@@ -2547,16 +2633,31 @@ async def execute_job(
raise RuntimeError("durable_write_lease_lost_before_invocation")
hy_invoked = True
attempt = await in_the_background(queue.sdk_attempt, job_id)
window = model_failures() if model_failures is not None else nullcontext(None)
provider: ModelAttempt | None = None
try:
level = ServiceLevel.REALTIME if request.priority == "live" else ServiceLevel.BACKGROUND
with service_level_context(level):
sdk_result = await supervisor.process_sdk_job(
records,
request_id=request.idempotency_key,
before_hy_invocation=mark_hy_invocation,
with service_level_context(level), window as provider:
previous = (
await _previous_write_receipt(
supervisor, sdk_request_id(request.idempotency_key, attempt - 1)
)
if attempt > 0
else None
)
raw_ids = sdk_result.raw_ids
memory_ids = sdk_result.memory_ids
if previous is not None:
# The write before this one did land: its worker only never
# learned so. Complete from its receipt rather than write twice.
raw_ids, memory_ids = previous.raw_ids, previous.memory_ids
else:
sdk_result = await supervisor.process_sdk_job(
records,
request_id=sdk_request_id(request.idempotency_key, attempt),
before_hy_invocation=mark_hy_invocation,
)
raw_ids = sdk_result.raw_ids
memory_ids = sdk_result.memory_ids
except Exception as error: # noqa: BLE001 - classified below
# Never persist the raw message: it may be a provider response or the
# user's own text, and `last_error` is shown over HTTP as `detail`.
@@ -2576,51 +2677,56 @@ async def execute_job(
):
await in_the_background(queue.release_unstarted, job_id, owner=owner)
return
# The write coordinator classifies these before any SDK client is leased, so HY
# was provably not called. They must not read as `failed`:
provider_failure = provider.failure if provider is not None else None
if hy_invoked and provider_failure is not None:
# The provider failed this write (a broken key, a model that is
# gone, a provider that is down): the job is fine and waits for
# the provider, in its place, without spending a retry. The
# breaker paces when it is tried again.
await in_the_background(
queue.requeue_for_model, job_id, error_code=provider_failure, owner=owner
)
logger.info(
"durable_write_job_waiting_for_provider job_id=%s error_code=%s sdk_error=%s",
job_id,
provider_failure,
code,
)
return
# Refusals made before the SDK was called. They prove nothing was
# sent, and are states, not failures:
# - authorization stop -> cancelled (the user turned it off);
# - no permission yet -> waiting_for_permission (parked, re-admitted
# when a snapshot admits it);
# - transient model/provider trouble -> waiting_for_model (parked).
# Everything else may have crossed into HY, and HY has no idempotency
# key, so it is `unknown` rather than a blind retry.
# - no permission yet -> waiting_for_permission (re-admitted when a
# snapshot admits it);
# - no usable model yet -> waiting_for_model (re-admitted when one is).
state: str | None = None
if code in PRE_SEND_REFUSAL_CODES:
# The frozen input window is checked before the payload reaches
# HY. This one refusal therefore has a proven pre-send boundary.
state, invoked_marker = FAILED, False
elif hy_invoked:
# Error names do not prove execution phase: permission and
# evidence rechecks also run after HY returns. Once the durable
# marker is set, every failure is possibly-sent and must stay out
# of the automatic retry path.
state, invoked_marker = UNKNOWN, True
elif code in POLICY_STOPPED_CODES:
state, invoked_marker = CANCELLED, False
elif code in PERMISSION_REQUIRED_CODES or code.startswith("model_content_policy"):
state, invoked_marker = WAITING_FOR_PERMISSION, False
elif code in MODEL_NOT_READY_CODES:
state, invoked_marker = WAITING_FOR_MODEL, False
elif code == "model_content_transport_request_failed":
# The generation or embedding HTTP exchange can fail after HY has
# already persisted a node. Treating that as a pre-send failure
# permits a blind duplicate; the confirmed unknown flow first
# reconciles by request identity and only then requeues absence.
state, invoked_marker = UNKNOWN, True
elif code.startswith("model_content_") and code != "model_content_outcome_unknown":
state, invoked_marker = FAILED, False
# Deterministic: the same input exceeds the same window again.
state = FAILED
elif not hy_invoked and code in POLICY_STOPPED_CODES:
state = CANCELLED
elif not hy_invoked and (
code in PERMISSION_REQUIRED_CODES or code.startswith("model_content_policy")
):
state = WAITING_FOR_PERMISSION
elif not hy_invoked and code in MODEL_NOT_READY_CODES:
state = WAITING_FOR_MODEL
if state is not None:
await in_the_background(
queue.settle,
job_id,
state=state,
error_code=code,
hy_invoked=hy_invoked,
owner=owner,
)
else:
# Force the marker even if this row was not claimed first (a direct
# call): the outcome is only `unknown` because HY may have been
# reached, and reclaim must treat it as possibly-sent.
state, invoked_marker = UNKNOWN, True
await in_the_background(
queue.settle,
job_id,
state=state,
error_code=code,
hy_invoked=invoked_marker,
owner=owner,
)
# Anything else is this job's own trouble: content the model or
# the SDK refused, a reply that did not parse, a check that failed
# after the write. Retried a few times, then `failed`.
state = await in_the_background(
queue.retry_or_fail, job_id, error_code=code, owner=owner
)
# Never `logger.exception` here: it prints the exception's message,
# which may be a provider response or the user's own text. Log the
# already-sanitised code instead, matching `hy-failure-classification`.
@@ -2638,9 +2744,10 @@ async def execute_job(
# chain, never its message: `error_code=AttributeError` alone
# named no location.
logger.error(
"durable_write_job_failed job_id=%s error_code=%s chain=%s",
"durable_write_job_failed job_id=%s error_code=%s state=%s chain=%s",
job_id,
code,
state or "stale",
error_chain_text(error),
)
return
@@ -2665,6 +2772,14 @@ async def execute_job(
)
async def _previous_write_receipt(supervisor: Any, request_id: str) -> Any | None:
"""The receipt of an earlier write of this job, if the SDK says it succeeded."""
inquire = getattr(supervisor, "inquire_write_outcome", None)
if inquire is None:
return None
return await inquire(request_id)
class DurableWriteDrain:
"""Run SQLite durable jobs with lease, periodic renewal and no blind retry.
@@ -2694,6 +2809,7 @@ class DurableWriteDrain:
reconcile_unknown: Callable[[], Awaitable[None]] | None = None,
source_writes_permitted: Callable[[str], bool] | None = None,
source_claim_enabled: Callable[[str], bool] | None = None,
model_claim_allowance: Callable[[], int | None] | None = None,
) -> None:
self.queue = queue
self.execute = execute
@@ -2769,6 +2885,11 @@ class DurableWriteDrain:
self.reconcile_unknown = reconcile_unknown
self.source_writes_permitted = source_writes_permitted
self.source_claim_enabled = source_claim_enabled or source_writes_permitted
#: How many jobs may be claimed now while the model provider is being
#: given a rest (`model_backoff`): None for no limit, 0 to claim none.
#: Without it a backlog failing in 50ms each was hundreds of requests
#: a minute against a provider that refused every one (#245).
self.model_claim_allowance = model_claim_allowance
self._last_unknown_reconcile: datetime | None = None
def claim(
@@ -2812,6 +2933,14 @@ class DurableWriteDrain:
return ()
except sqlite3.Error:
pass
if self.model_claim_allowance is not None:
# Asked only once there is work to claim: while the provider is
# paused nothing is, and after the pause one job probes it.
allowance = self.model_claim_allowance()
if allowance is not None:
if allowance < 1:
return ()
limit = min(limit, allowance)
enabled_sources: set[str] | None = None
if self.source_writes_permitted is not None:
with main_database(self.queue.database_path).read() as connection:
@@ -3163,6 +3163,33 @@ async def _060_memory_note_requests(cursor: Any) -> None:
await cursor.execute(statement)
def write_job_retries_ddl(dialect: str = "mysql") -> str:
"""Per-job retry state: which SDK request id is next, and retries spent.
The SDK replays a request id's first outcome forever, so every re-run of
a job needs an id of its own (`sdk_attempt`). `retries` counts automatic
retries after failures of the job itself; provider failures spend none.
"""
if dialect == "sqlite":
return """
ALTER TABLE memory_write_jobs
ADD COLUMN sdk_attempt INTEGER NOT NULL DEFAULT 0;
ALTER TABLE memory_write_jobs
ADD COLUMN retries INTEGER NOT NULL DEFAULT 0;
"""
return """
ALTER TABLE memory_write_jobs
ADD COLUMN sdk_attempt INT NOT NULL DEFAULT 0;
ALTER TABLE memory_write_jobs
ADD COLUMN retries INT NOT NULL DEFAULT 0;
"""
async def _061_write_job_retries(cursor: Any) -> None:
for statement in _split_statements(write_job_retries_ddl(_dialect(cursor))):
await cursor.execute(statement)
def _split_statements(sql: str) -> list[str]:
"""Split a multi-statement script, dropping blanks, PRAGMA and comments.
@@ -3462,4 +3489,5 @@ MIGRATIONS: tuple[Migration, ...] = (
_060_memory_note_requests,
("memory_note_requests",),
),
Migration(61, "write_job_retries", _061_write_job_retries),
)
@@ -57,6 +57,11 @@ class WriteRecord:
detail: str = ""
raw_ids: tuple[str, ...] = ()
memory_ids: tuple[str, ...] = ()
#: The stable code of the last failure of a job still being worked on
#: (or of a failed one).
error_code: str | None = None
#: When a waiting job is next tried, if a retry wait holds it.
next_retry_at: str | None = None
def _now() -> str:
@@ -44,6 +44,7 @@ from one_memory.memory_server.durable_queue import (
)
from one_memory.memory_server.registry import WriteRegistry, WriteState
from one_memory.memory_server.submit import WriteConflictError
from one_memory.model_backoff import next_retry_at
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
#: Durable keys of another user's writes start with this. A `local` request
@@ -72,6 +73,12 @@ class MemoryWriteReceipt(BaseModel):
detail: str = ""
raw_ids: tuple[str, ...] = ()
memory_ids: tuple[str, ...] = ()
#: Why a job still being worked on last failed (`model_auth_failed`, ...),
#: or why a failed one failed. A provider failure keeps `state` queued.
error_code: str | None = None
#: When it is next tried: the job's own retry wait, or the model
#: provider's pause (`model_backoff`) while the provider is failing.
next_retry_at: str | None = None
class OpenAIMessage(BaseModel):
@@ -235,6 +242,8 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
detail=record.detail,
raw_ids=record.raw_ids,
memory_ids=record.memory_ids,
error_code=record.error_code,
next_retry_at=next_retry_at(request.app, record.state, record.next_retry_at),
)
return instance
@@ -0,0 +1,328 @@
"""One pause for every model request while the provider keeps failing (#245).
A broken model configuration (a revoked key, a wrong base URL, a model that
does not exist, a provider that is down) used to fail each queued job on its
own: the drain claimed the next job the moment the last one failed, so a
backlog of hundreds of jobs turned into requests as fast as the provider could
refuse them. The pause is global, not per job: a failure says something about
the provider, not about the conversation that happened to hit it.
After the first failure no further model work is claimed for 5 seconds; each
failed probe after that doubles the pause, up to 3 minutes. A pause admits one
probe job at a time. The first success, or any change the user makes to the
model configuration, ends it at once. Jobs not yet sent stay `pending`.
"""
from __future__ import annotations
import logging
import threading
import time
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any
import httpx
logger = logging.getLogger(__name__)
BACKOFF_FIRST_SECONDS = 5.0
BACKOFF_CAP_SECONDS = 180.0
#: How long one probe may hold the pause open before another is admitted. A
#: probe job that never reaches the provider (nothing to remember, a refusal
#: before the SDK) reports nothing, and the queue must not wait on it forever.
PROBE_HOLD_SECONDS = 30.0
#: Requests already in flight when the pause began are refused without going
#: out once this many failures in a row confirm it is not a one-off.
REFUSE_IN_FLIGHT_AFTER = 2
#: Statuses that say the configuration or the provider is wrong, not the one
#: request: authentication, a missing model or route, throttling, a timeout
#: and the provider's own failures. A 400/413/422 is about the request's
#: content and is left to that job.
_PROVIDER_STATUSES = {401: "model_auth_failed", 403: "model_auth_failed", 404: "model_not_found"}
#: Every code a provider failure is recorded under: a job that failed with one
#: of these waits for the provider instead of spending a retry.
MODEL_FAILURE_CODES = frozenset(
{
"model_auth_failed",
"model_not_found",
"model_timeout",
"model_rate_limited",
"model_server_error",
"model_unreachable",
"model_transport_failed",
}
)
def backoff_seconds(level: int) -> float:
"""The pause after the `level`-th failed attempt in a row (1-based)."""
if level < 1:
return 0.0
return min(BACKOFF_FIRST_SECONDS * float(2 ** min(level - 1, 16)), BACKOFF_CAP_SECONDS)
def failure_code(status: int) -> str | None:
"""The stable code for a provider HTTP status, or None if it is no provider failure."""
if status in _PROVIDER_STATUSES:
return _PROVIDER_STATUSES[status]
if status in {408, 504}:
return "model_timeout"
if status == 429:
return "model_rate_limited"
if status >= 500:
return "model_server_error"
return None
def transport_failure_code(error: BaseException) -> str:
if isinstance(error, httpx.TimeoutException):
return "model_timeout"
if isinstance(error, httpx.ConnectError | httpx.NetworkError):
return "model_unreachable"
return "model_transport_failed"
def request_kind(request: httpx.Request) -> str:
"""`embedding` for the embeddings route, `chat` for everything else."""
return "embedding" if request.url.path.rstrip("/").endswith("/embeddings") else "chat"
@dataclass(frozen=True)
class ModelBackoffState:
#: When the pause ends; None when no pause is in force.
until: datetime | None
#: Failed attempts in a row.
failures: int
#: The last failure's stable code (`model_auth_failed`, ...).
last_error: str | None
profile_id: str | None
class ModelBackoffRefusal(httpx.TransportError):
"""A request refused locally because the provider is being given a rest."""
@dataclass(eq=False)
class ModelAttempt:
"""One job's write, as the provider saw it.
`failure` is the code of the first provider failure (or local refusal)
that happened while this write was in flight, None if there was none.
"""
failure: str | None = None
class ModelBackoff:
"""Process-wide circuit breaker for the active model profile.
Thread-safe: the transport records from the SDK's own loop thread, the
drain asks from its claim thread, and status reads from the event loop.
"""
def __init__(
self,
*,
clock: Callable[[], float] = time.monotonic,
wall_clock: Callable[[], datetime] = lambda: datetime.now(UTC),
) -> None:
self._clock = clock
self._wall_clock = wall_clock
self._lock = threading.Lock()
#: Writes in flight (`attempt`); never cleared by a reset.
self._attempts: set[ModelAttempt] = set()
self._reset_locked()
def _reset_locked(self) -> None:
#: Which kinds of request are failing (`chat`, `embedding`): a working
#: embedding endpoint says nothing about a chat model that is gone.
self._failing: set[str] = set()
self._level = 0
self._failures = 0
self._until = 0.0
self._probe_until = 0.0
self._last_error: str | None = None
self._profile_id: str | None = None
def reset(self, reason: str = "model_configuration_changed") -> None:
"""Forget every failure: the user changed what is being called."""
with self._lock:
was_paused = self._failures > 0
self._reset_locked()
if was_paused:
logger.info("model_backoff_reset reason=%s", reason)
def record_success(self, profile_id: str | None = None, kind: str = "model") -> None:
with self._lock:
if self._failures == 0:
return
if profile_id is not None and self._profile_id not in {None, profile_id}:
return
self._failing.discard(kind)
if self._failing:
return
self._reset_locked()
logger.info("model_backoff_cleared profile_id=%s", profile_id)
def record_failure(self, code: str, profile_id: str | None = None, kind: str = "model") -> None:
now = self._clock()
with self._lock:
if profile_id is not None and self._profile_id not in {None, profile_id}:
# Another profile's history says nothing about this one.
self._reset_locked()
self._profile_id = profile_id
self._failing.add(kind)
self._mark_attempts_locked(code)
self._failures += 1
self._last_error = code
self._probe_until = 0.0
# Requests that were already out when the pause began fail too;
# they are the same outage, not a reason to wait longer.
if now >= self._until:
self._level += 1
self._until = now + backoff_seconds(self._level)
level, pause = self._level, self._until - now
logger.warning(
"model_backoff_failure error_code=%s level=%d pause_s=%.0f profile_id=%s",
code,
level,
pause,
profile_id,
)
@contextmanager
def attempt(self) -> Iterator[ModelAttempt]:
"""Hold one job's write open, and learn whether the provider failed it.
Every provider failure and every local refusal while the write is in
flight is charged to it. A request cannot be traced to the job that
made it: the SDK sends embeddings from one shared batching worker, so
neither the calling thread nor a context variable identifies the job.
Charging every write then in flight can name a write the provider did
not fail (another job's request failed beside it); that write waits
for the provider and is retried with the rest. It can never miss the
write the provider did fail, which would spend that job's retries on
an outage.
"""
record = ModelAttempt()
with self._lock:
self._attempts.add(record)
try:
yield record
finally:
with self._lock:
self._attempts.discard(record)
def record_refusal(self) -> str:
"""A request refused locally; charged to the writes in flight like a failure."""
with self._lock:
code = self._last_error or "model_transport_failed"
self._mark_attempts_locked(code)
return code
def _mark_attempts_locked(self, code: str) -> None:
for record in self._attempts:
if record.failure is None:
record.failure = code
def paused(self) -> bool:
with self._lock:
return self._failures > 0 and self._clock() < self._until
def refuses_requests(self, kind: str = "model") -> bool:
"""Whether a request of this kind should not go out at all right now.
Only the failing kind is refused: a chat model that is gone must not
take recall's embedding calls down with it.
"""
with self._lock:
return (
kind in self._failing
and self._failures >= REFUSE_IN_FLIGHT_AFTER
and self._clock() < self._until
)
def allowance(self) -> int | None:
"""How many model jobs may be claimed now: None for no limit.
0 while paused; once the pause is over, one probe at a time until it
reports or `PROBE_HOLD_SECONDS` pass.
"""
now = self._clock()
with self._lock:
if self._failures == 0:
return None
if now < self._until or now < self._probe_until:
return 0
self._probe_until = now + PROBE_HOLD_SECONDS
return 1
def state(self) -> ModelBackoffState:
now = self._clock()
with self._lock:
remaining = self._until - now if self._failures else 0.0
return ModelBackoffState(
until=(
self._wall_clock() + timedelta(seconds=remaining) if remaining > 0 else None
),
failures=self._failures,
last_error=self._last_error,
profile_id=self._profile_id,
)
def next_retry_at(app: Any, state: str, own: str | None) -> str | None:
"""When a queued job is tried next: its own retry wait, or the provider's pause.
While the breaker pauses, nothing is claimed at all, so the pause holds
every queued job, whatever it last failed with.
"""
if state not in {"accepted", "pending"}:
return None
backoff = getattr(getattr(app, "state", None), "model_backoff", None)
until = backoff.state().until if isinstance(backoff, ModelBackoff) else None
paused = until.isoformat() if until is not None else None
candidates = [value for value in (own, paused) if value]
return max(candidates) if candidates else None
class BackedOffModelTransport(httpx.AsyncBaseTransport):
"""Report each provider exchange to the breaker; refuse while it confirms a pause."""
def __init__(
self,
backoff: ModelBackoff,
*,
profile_id: str,
inner: httpx.AsyncBaseTransport | None = None,
) -> None:
self.backoff = backoff
self.profile_id = profile_id
self.inner = inner or httpx.AsyncHTTPTransport(retries=0, trust_env=False)
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
kind = request_kind(request)
if self.backoff.refuses_requests(kind):
self.backoff.record_refusal()
raise ModelBackoffRefusal("model_backoff_active", request=request)
try:
response = await self.inner.handle_async_request(request)
except ModelBackoffRefusal:
raise
except httpx.TransportError as error:
self.backoff.record_failure(transport_failure_code(error), self.profile_id, kind)
raise
code = failure_code(response.status_code)
if code is not None:
self.backoff.record_failure(code, self.profile_id, kind)
elif response.status_code < 400:
self.backoff.record_success(self.profile_id, kind)
return response
async def aclose(self) -> None:
await self.inner.aclose()
@@ -135,6 +135,8 @@ class RecallReceipt(BaseModel):
#: exists yet (`fallback_reason`). Raw hits are excerpts, not memories.
mode: Literal["memory", "raw"] = "memory"
fallback_reason: Literal["model_unavailable", "no_memories"] | None = None
#: Whose memories were searched: the request's `user_id`, else `local`.
user_id: str = DEFAULT_USER_ID
class RecallTarget(Protocol):
@@ -32,6 +32,19 @@ FallbackReason = Literal["model_unavailable", "no_memories"]
#: caller). An empty set means the caller may read no raw conversation.
RawSources = frozenset[str] | None
#: What the whole recall request may take beyond the caller's own
#: `timeout_ms`: the search keeps to that budget, but the checks around it
#: (is a model configured, is the library empty, which raw sources) and the
#: raw fallback need a little room of their own. Well under the broker's 15 s
#: wait on Core, so a slow Core answers 504 before the broker gives up.
RECALL_TOTAL_GRACE_SECONDS = 3.0
RECALL_TOTAL_CEILING_SECONDS = 12.0
def recall_total_timeout_seconds(timeout_ms: int) -> float:
"""The most one recall request may take from start to answer."""
return min(timeout_ms / 1_000 + RECALL_TOTAL_GRACE_SECONDS, RECALL_TOTAL_CEILING_SECONDS)
async def recall_with_fallback(
request: RecallRequest,
@@ -33,7 +33,7 @@ from one_memory.model_profiles import (
)
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
from one_memory.user_scope import owns
from one_memory.user_scope import DEFAULT_USER_ID, owns
_NO_GROUNDING = "现有记忆不足以回答这个问题。"
_GROUNDING_LIMIT = 6
@@ -62,6 +62,8 @@ class SdkMemoryAnswer(BaseModel):
answer: str
memories: tuple[SdkAskMemory, ...]
mode: Literal["generated", "extractive", "empty"]
#: Whose memories answered: the request's `user_id`, else `local`.
user_id: str = DEFAULT_USER_ID
class _PromptMemory:
@@ -25,7 +25,7 @@ from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import SdkClientHandle, sdk_relevance_floor
from one_memory.hy_sdk_types import HySdkError, parse_sdk_search
from one_memory.memory_queue import ExtractionWindowPreset, QueueDepths
from one_memory.memory_server.durable_queue import COMPLETED, DurableWriteQueue
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.semantic_deadline import SemanticMemoryHit
@@ -34,6 +34,10 @@ _STABLE_CODE = re.compile(r"[a-z][a-z0-9_]{0,95}")
logger = logging.getLogger(__name__)
#: Jobs not yet run that a suspended source holds back.
_WAITING_STATES = frozenset({"pending", "waiting_for_model", "waiting_for_permission"})
_WAITING_SQL = ",".join(f"'{state}'" for state in sorted(_WAITING_STATES))
class SdkMemoryRuntime:
"""Product services that remain after legacy task/governance writes retire."""
@@ -48,12 +52,19 @@ class SdkMemoryRuntime:
data_dir: Path,
memory_mode: Callable[[], str] = lambda: "pro",
writes_open: Callable[[], bool] = lambda: True,
source_active: Callable[[str], bool] = lambda _source_key: True,
) -> None:
self.coordinator = coordinator
#: The SDK mode, for the graph store's path (`hy_graph_admin`).
self.memory_mode = memory_mode
#: False while an embedding switch pauses model writes.
self.writes_open = writes_open
#: Whether a queue lane's source may still become memories: the same
#: question the drain's claim asks of it. A lane that is off (the
#: user turned the source off, or its grant was revoked) has its
#: waiting jobs suspended: kept, never claimed, and not counted as
#: pending until the source is on again.
self.source_active = source_active
self.read_handle = read_handle
self.write_queue = write_queue
self.extraction_settings = extraction_settings
@@ -241,22 +252,11 @@ class SdkMemoryRuntime:
now = datetime.now(UTC)
fresh_since = (now - timedelta(days=7)).isoformat()
with main_database(self.write_queue.database_path).read() as database:
fresh = int(
database.execute(
"SELECT COUNT(*) FROM memory_write_jobs "
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
"AND event_at>=?",
(fresh_since,),
).fetchone()[0]
)
backlog = int(
database.execute(
"SELECT COUNT(*) FROM memory_write_jobs "
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
"AND event_at<?",
(fresh_since,),
).fetchone()[0]
)
rows = database.execute(
"SELECT source_key,event_at>=? AS fresh,COUNT(*) FROM memory_write_jobs "
f"WHERE state IN ({_WAITING_SQL}) GROUP BY source_key,fresh",
(fresh_since,),
).fetchall()
recent = tuple(
datetime.fromisoformat(str(row[0]))
for row in database.execute(
@@ -264,6 +264,9 @@ class SdkMemoryRuntime:
"ORDER BY updated_at DESC LIMIT 8"
)
)
active = self._active_lanes({str(row[0]) for row in rows})
fresh = sum(int(row[2]) for row in rows if str(row[0]) in active and row[1])
backlog = sum(int(row[2]) for row in rows if str(row[0]) in active and not row[1])
return QueueDepths(fresh=fresh, backlog=backlog, recent_completions=recent)
def active_depths(self) -> tuple[int, int]:
@@ -276,13 +279,27 @@ class SdkMemoryRuntime:
return running, running
def status_counts(self) -> dict[str, int]:
"""Jobs per state, with a suspended source's waiting jobs as `suspended`.
Those jobs keep their state in the table: suspension is the source's
switch, not the row's, so turning the source back on is all it takes
for the drain to claim them again, and nothing is moved or deleted.
"""
with main_database(self.write_queue.database_path).read() as database:
return {
str(row[0]): int(row[1])
for row in database.execute(
"SELECT state,COUNT(*) FROM memory_write_jobs GROUP BY state"
)
}
rows = database.execute(
"SELECT source_key,state,COUNT(*) FROM memory_write_jobs GROUP BY source_key,state"
).fetchall()
active = self._active_lanes({str(row[0]) for row in rows if str(row[1]) in _WAITING_STATES})
counts: dict[str, int] = {}
for source_key, state, count in rows:
name = str(state)
if name in _WAITING_STATES and str(source_key) not in active:
name = "suspended"
counts[name] = counts.get(name, 0) + int(count)
return counts
def _active_lanes(self, lanes: set[str]) -> set[str]:
return {lane for lane in lanes if self.source_active(lane)}
def manual_batch_progress(self) -> tuple[int, int]:
return 0, 0
@@ -291,31 +308,6 @@ class SdkMemoryRuntime:
del task_ids
return []
async def reconcile_unknown_outputs(
self,
*,
limit: int = 128,
exclude_error_codes: tuple[str, ...] = (),
) -> int:
reconciled = 0
for job in await asyncio.to_thread(
self.write_queue.unknown_jobs,
limit=limit,
exclude_error_codes=exclude_error_codes,
):
receipt = await self.coordinator.inquire_write_outcome(job.request_id)
if receipt is None:
continue
await asyncio.to_thread(
self.write_queue.settle,
job.job_id,
state=COMPLETED,
sdk_raw_ids=receipt.raw_ids,
sdk_memory_ids=receipt.memory_ids,
)
reconciled += 1
return reconciled
async def resume_allowed(self) -> None:
await asyncio.to_thread(self.write_queue.readmit_parked)
@@ -29,6 +29,7 @@ from collections.abc import Callable
from pathlib import Path
from typing import Any
from one_memory.background_work import in_sdk_read
from one_memory.embedding_health import EmbeddingGuardedClient, EmbeddingHealth
from one_memory.embedding_spaces import (
EmbeddingSpace,
@@ -112,7 +113,7 @@ class ActiveProfileReadHandle(SdkClientHandle):
async def current_handle(self) -> SdkClientHandle:
"""The handle of the live space, held for the asking task."""
space, identity = await asyncio.to_thread(self._resolve)
space, identity = await in_sdk_read(self._resolve)
mode, features = self.memory_mode(), self.current_features()
language = self.current_language()
key = (
@@ -127,9 +128,7 @@ class ActiveProfileReadHandle(SdkClientHandle):
if handle is None:
# Only on a change of space or profile: resolving a key can
# reach the system keychain, so it is not done on every read.
handle = await asyncio.to_thread(
self._build, space, identity, mode, features, language
)
handle = await in_sdk_read(self._build, space, identity, mode, features, language)
self._handles[key] = handle
self._retired.update(value for name, value in self._handles.items() if name != key)
self._handles = {key: handle}
@@ -3,12 +3,14 @@
from __future__ import annotations
import asyncio
import threading
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence
from contextlib import asynccontextmanager
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from one_memory.background_work import in_sdk_write
from one_memory.embedding_spaces import (
EmbeddingSpace,
EmbeddingSpaceStore,
@@ -28,6 +30,61 @@ from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
PRODUCT_AGENT_ID = "all-apps"
#: SDK request ids whose write call is running on a thread right now. A write
#: the drain gave up on (its work ceiling, a lost lease) keeps running on its
#: thread; it must not be re-run beside itself (`recover_unknown`).
_IN_FLIGHT: set[str] = set()
_IN_FLIGHT_LOCK = threading.Lock()
def sdk_write_in_flight(request_id: str) -> bool:
with _IN_FLIGHT_LOCK:
return request_id in _IN_FLIGHT
class _Tracked:
"""One SDK write call, held in `_IN_FLIGHT` from submission until it ends.
If the wait for it is cancelled before a thread picked it up, it is
abandoned: it will not start, and it leaves `_IN_FLIGHT` at once.
"""
def __init__(self, request_id: str, work: Callable[..., Any]) -> None:
self.request_id = request_id
self.work = work
self.started = False
self.abandoned = False
with _IN_FLIGHT_LOCK:
_IN_FLIGHT.add(request_id)
def __call__(self, *args: Any, **kwargs: Any) -> Any:
with _IN_FLIGHT_LOCK:
if self.abandoned:
raise RuntimeError("hy_memory_write_abandoned")
self.started = True
try:
return self.work(*args, **kwargs)
finally:
with _IN_FLIGHT_LOCK:
_IN_FLIGHT.discard(self.request_id)
def abandon(self) -> None:
with _IN_FLIGHT_LOCK:
if not self.started:
self.abandoned = True
_IN_FLIGHT.discard(self.request_id)
async def _write(tracked_id: str, work: Callable[..., Any], /, *args: Any, **kwargs: Any) -> Any:
call = _Tracked(tracked_id, work)
try:
return await in_sdk_write(call, *args, **kwargs)
except asyncio.CancelledError:
call.abandon()
raise
_MESSAGE_ROLES = frozenset({"user", "assistant", "tool"})
_CONVERSATION_ROLES = frozenset({"user", "assistant"})
@@ -82,7 +139,8 @@ class SdkWriteProcessor:
if extracted and not self._lite:
contents = [sanitize_model_input(record.verbatim) for record in records]
await before_hy_invocation()
result = await asyncio.to_thread(
result = await _write(
request_id,
client.add_extracted,
contents,
user_id=scope.user_id,
@@ -109,7 +167,8 @@ class SdkWriteProcessor:
if not messages:
raise RuntimeError("hy_memory_task_without_inputs")
await before_hy_invocation()
result = await asyncio.to_thread(
result = await _write(
request_id,
client.add,
messages,
user_id=scope.user_id,
@@ -130,7 +189,7 @@ class SdkWriteProcessor:
if not request_id:
raise ValueError("request_id is required")
client = await self._client_handle.client()
outcome = await asyncio.to_thread(client.get_write_outcome, request_id)
outcome = await in_sdk_write(client.get_write_outcome, request_id)
if outcome is None:
return None
if not isinstance(outcome, Mapping):
+3 -1
View File
@@ -25,6 +25,7 @@
| `test_c1_cross_source_session_isolation.py` | `contract` | Same session name in two sources must stay two conversations. |
| `test_c1_damage_per_source.py` | `contract` | Damage and unknown schema, per source: a bad record must not sink the import. |
| `test_c1_deepseek_profiles.py` | `contract` | DeepSeek Harness has two profiles; only the headless session form is importable. |
| `test_c1_deepseek_versioned_sessions.py` | `contract` | DeepSeek Harness versions its session file names; every generation must import. |
| `test_c1_import_event_matrix.py` | `contract` | C1 owned matrix: existing history, new file, append, truncate/replace, restart, revoke. |
| `test_c1_import_failure_injection.py` | `contract` | C1 failure injection: atomic receipt, disk-full, read-only source, rotation. |
| `test_c1_import_instrumentation.py` | `contract` | Instrumented import accounting: bytes read, opens, hashes, jobs, duplicates. |
@@ -189,6 +190,7 @@
| `test_memory_write_registry.py` | `contract` | A3-01: acceptance and outcome recording for the unified write path. |
| `test_memory_write_routes.py` | `integration` | A3-01: the HTTP write entry, tested against a real FastAPI app. |
| `test_migrate_off_sqlite.py` | `contract` | A1-06: the migration is only done when reconciliation agrees. |
| `test_model_backoff.py` | `contract` | A broken model configuration must not turn a backlog into a request storm (#245). |
| `test_model_creation_api.py` | `integration` | Normal protected model creation routes with temporary data and in-memory secrets. |
| `test_model_creation_operations.py` | `contract` | 覆盖 lookup without table is read only and legacy create does not initialize、operation commits with profile and version without autoactivation、any replay conflicts before secret put 等 21 个场景。 |
| `test_model_creation_validation_boundary.py` | `integration` | Real ASGI validation must precede creation reservation and secret writes. |
@@ -318,4 +320,4 @@
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
共 313 个测试文件。
共 315 个测试文件。
@@ -14,8 +14,8 @@ from pathlib import Path
import pytest
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
EXPECTED_TEST_FILE_COUNT = 313
EXPECTED_TEST_FILES_SHA256 = "dcada5d5421767196a79e415dd47c2b6c6d67681972dbe81481b31867d318928"
EXPECTED_TEST_FILE_COUNT = 315
EXPECTED_TEST_FILES_SHA256 = "681258b2a13ca852e273707071b286afe6b0b2fe537f90a8a2c64d83ec662772"
# Pure transformations, validation models, parity checks and state machines.
# This list is intentionally small: uncertain files default to contract.
@@ -324,3 +324,48 @@ def test_saturated_raw_reads_do_not_delay_memory_reads() -> None:
finally:
release_raw.set()
shutdown_bulk_pool()
def test_recall_reads_answer_while_every_sdk_write_thread_is_stuck() -> None:
"""A write that hangs in the SDK keeps its thread, not recall's (#231)."""
from one_memory.background_work import SDK_WRITE_WORKERS, in_sdk_read, in_sdk_write
stuck = threading.Barrier(SDK_WRITE_WORKERS + 1)
release = threading.Event()
def hung_write() -> None:
stuck.wait(timeout=2)
assert release.wait(5)
async def scenario() -> tuple[float, int]:
writes = [asyncio.create_task(in_sdk_write(hung_write)) for _ in range(SDK_WRITE_WORKERS)]
# One more than the pool holds: it waits for a write thread, as the
# drain's next write would.
queued = asyncio.create_task(in_sdk_write(lambda: None))
await asyncio.to_thread(stuck.wait, 2)
started = time.monotonic()
answer = await asyncio.wait_for(in_sdk_read(lambda: 7), 1)
elapsed = time.monotonic() - started
assert not queued.done()
release.set()
await asyncio.gather(*writes, queued)
return elapsed, answer
try:
elapsed, answer = asyncio.run(scenario())
assert answer == 7
assert elapsed < 0.2
finally:
release.set()
shutdown_bulk_pool()
def test_sdk_writes_run_on_their_own_bounded_threads() -> None:
from one_memory.background_work import SDK_WRITE_WORKERS, in_sdk_write, sdk_write_pool
try:
name = asyncio.run(in_sdk_write(lambda: threading.current_thread().name))
assert name.startswith("one-memory-sdk-write")
assert sdk_write_pool()._max_workers == SDK_WRITE_WORKERS
finally:
shutdown_bulk_pool()
@@ -0,0 +1,246 @@
"""DeepSeek Harness versions its session file names; every generation must import.
The first format writes `<session>/session.jsonl[.zstd]`; v3 and v4 write
`session.v3.jsonl.zstd` / `session.v4.jsonl.zstd`. Discovery used to match only
the two unversioned names, so every v3/v4 session was invisible to the import
job — `discovered 0` while the files sat on disk. These tests pin the file-name
contract, the header gate that keeps an unknown future version out of the
database, and the upgrade path where a library imported before the fix picks up
the versioned files on its next scan.
"""
import json
from pathlib import Path
import pytest
import zstandard
from one_memory.imports.adapters import ScanFailure, ScanIssueCode
from one_memory.imports.sources import deepseek_harness
from one_memory.imports.sources.deepseek_harness import (
DEEPSEEK_HARNESS_DESCRIPTOR,
create_deepseek_harness_adapter,
)
from tests.test_secondary_source_adapters import (
_cursors,
_evidence,
_evidence_text,
_grant,
_scan,
_skip_codes,
)
def _write_session(directory: Path, name: str, records: list[dict[str, object]]) -> None:
directory.mkdir(parents=True, exist_ok=True)
payload = "".join(f"{json.dumps(record)}\n" for record in records).encode()
(directory / name).write_bytes(zstandard.ZstdCompressor().compress(payload))
def _conversation(version: int | None, *, directory: str = "run-a") -> list[dict[str, object]]:
header: dict[str, object] = {
"type": "session",
"version": version,
"id": directory,
"cwd": "/private/versioned-project",
}
return [
header,
{
"type": "user/message",
"seq": 1,
"data": {
"id": "human",
"role": "user",
"source": {"kind": "user"},
"content": [{"type": "text", "text": "visible human"}],
},
},
{
"type": "assistant/message",
"seq": 2,
"data": {
"message": {
"id": "assistant",
"role": "assistant",
"content": [{"type": "text", "text": "visible assistant"}],
}
},
},
]
def _session_adapter(tmp_path: Path):
root = tmp_path / "dsh"
root.mkdir(parents=True, exist_ok=True)
return root, create_deepseek_harness_adapter(
_grant(DEEPSEEK_HARNESS_DESCRIPTOR.source_id, root, tmp_path)
)
def test_versioned_v4_session_file_is_discovered_and_imported(tmp_path: Path) -> None:
root, adapter = _session_adapter(tmp_path)
session = root / "run-v4"
records = _conversation(4, directory="run-v4")
records.append(
{
"type": "tool/result",
"seq": 3,
"time": 1793341068763,
"data": {
"callId": "call-1",
"message": {"role": "tool", "content": [{"type": "text", "text": "tool output"}]},
},
}
)
_write_session(session, "session.v4.jsonl.zstd", records)
# A rotated backup next to the live file must not be imported as a second copy.
(session / "session.v4.jsonl.zstd.orig-20261027").write_bytes(
zstandard.ZstdCompressor().compress(b'{"type":"session","version":4}\n')
)
events = _scan(adapter)
assert _evidence_text(events) == ["visible human", "visible assistant", "tool output"]
evidence = _evidence(events)
assert {event.evidence.project for event in evidence} == {"versioned-project"}
assert {event.evidence.conversation_id for event in evidence} == {"run-v4"}
def test_versioned_v3_session_file_is_discovered_and_imported(tmp_path: Path) -> None:
root, adapter = _session_adapter(tmp_path)
_write_session(root / "run-v3", "session.v3.jsonl.zstd", _conversation(3))
events = _scan(adapter)
assert _evidence_text(events) == ["visible human", "visible assistant"]
def _tool_result(content: list[dict[str, object]]) -> dict[str, object]:
return {
"type": "tool/result",
"seq": 3,
"data": {"callId": "call-1", "message": {"role": "tool", "content": content}},
}
def test_tool_output_imports_in_either_block_shape(tmp_path: Path) -> None:
"""`tool/result` output written as `text` blocks used to be dropped.
Only `tool-result` blocks were read, so a session that writes plain `text`
blocks lost every tool output. Both shapes must arrive, in any generation.
"""
root, adapter = _session_adapter(tmp_path)
v3 = _conversation(3, directory="run-v3-text")
v3.append(_tool_result([{"type": "text", "text": "v3 text output"}]))
_write_session(root / "run-v3-text", "session.v3.jsonl.zstd", v3)
v0 = _conversation(0, directory="run-v0-block")
v0.append(_tool_result([{"type": "tool-result", "content": "v0 block output"}]))
_write_session(root / "run-v0-block", "session.jsonl.zstd", v0)
texts = _evidence_text(_scan(adapter))
assert "v3 text output" in texts
assert "v0 block output" in texts
def test_text_tool_output_is_scrubbed_like_block_output(tmp_path: Path) -> None:
root, adapter = _session_adapter(tmp_path)
secret = "sk-" + "a" * 40
records = _conversation(4, directory="run-secret")
records.append(_tool_result([{"type": "text", "text": f"token {secret} " + "x" * 5000}]))
_write_session(root / "run-secret", "session.v4.jsonl.zstd", records)
tool = _evidence_text(_scan(adapter))[-1]
assert secret not in tool
assert len(tool) < 2000
def test_session_file_names_outside_the_contract_are_not_imported(tmp_path: Path) -> None:
root, adapter = _session_adapter(tmp_path)
for name in (
"session.v4.jsonl.zstd.orig-20261027",
"session.vx.jsonl.zstd",
"other.v4.jsonl.zstd",
"session.v4.json",
):
_write_session(root / name.replace(".", "-"), name, _conversation(4))
assert _evidence(_scan(adapter)) == []
def test_unknown_future_session_version_imports_nothing(tmp_path: Path) -> None:
root, adapter = _session_adapter(tmp_path)
records = _conversation(5)
records.append(
{
"type": "user/message",
"seq": 9,
"data": {
"id": "future",
"role": "user",
"source": {"kind": "user"},
"content": [{"type": "text", "text": "future generation payload"}],
},
}
)
_write_session(root / "run-v5", "session.v5.jsonl.zstd", records)
events = _scan(adapter)
assert _evidence(events) == []
assert ScanIssueCode.UNSUPPORTED_SCHEMA in _skip_codes(events)
def test_unknown_session_version_is_reported_and_read_once_supported(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Not importing a future format yet must stay "yet".
A cursor left behind by the refused scan would put the whole session behind
the checkpoint, so the release that learns the format would read nothing.
"""
root, adapter = _session_adapter(tmp_path)
_write_session(root / "run-v5", "session.v5.jsonl.zstd", _conversation(5))
refused = _scan(adapter)
assert _cursors(refused) == {}
assert [event.reason_code for event in refused if isinstance(event, ScanFailure)] == [
ScanIssueCode.UNSUPPORTED_SCHEMA
]
monkeypatch.setattr(deepseek_harness, "SUPPORTED_SESSION_VERSIONS", frozenset({0, 3, 4, 5}))
_, upgraded = _session_adapter(tmp_path)
assert _evidence_text(_scan(upgraded, cursors=_cursors(refused))) == [
"visible human",
"visible assistant",
]
def test_upgrade_scan_rediscovers_versioned_files_already_on_disk(tmp_path: Path) -> None:
"""A library imported before this fix must pick up the files on its next scan.
The first scan sees only the unversioned session that the old predicate
matched; after the upgrade, the versioned file next to it is discovered by
the next scan while the already imported session stays deduplicated by the
cursor it left behind.
"""
root, adapter = _session_adapter(tmp_path)
old = root / "run-old"
_write_session(old, "session.jsonl.zstd", _conversation(0, directory="run-old"))
first_scan = _scan(adapter)
assert _evidence_text(first_scan) == ["visible human", "visible assistant"]
assert _evidence(first_scan)[0].evidence.conversation_id == "run-old"
versioned = root / "run-v4"
_write_session(versioned, "session.v4.jsonl.zstd", _conversation(4, directory="run-v4"))
cursors = _cursors(first_scan)
assert cursors # the old session left a cursor behind
upgrade_scan = _scan(adapter, cursors=cursors)
assert _evidence_text(upgrade_scan) == ["visible human", "visible assistant"]
assert {event.evidence.conversation_id for event in _evidence(upgrade_scan)} == {"run-v4"}
@@ -193,7 +193,7 @@ def _qwenwork() -> Harness:
def _deepseek() -> Harness:
def write(path: Path, turns: list[Turn]) -> None:
# deepseek only accepts a file literally named session.jsonl, and nests
# deepseek only accepts files named session[.vN].jsonl[.zstd], and nests
# the turn under "data".
lines: list[dict] = [{"type": "session", "version": 0, "id": "s1", "cwd": "/w"}]
for index, (role, text, _session) in enumerate(turns):
@@ -225,7 +225,7 @@ def _deepseek() -> Harness:
from one_memory.imports.sources.deepseek_harness import create_deepseek_harness_adapter
# deepseek accepts only a file literally named session.jsonl.
# deepseek accepts only session[.vN].jsonl[.zstd]; the first format is used here.
return Harness(
"deepseek-harness",
write,
@@ -336,6 +336,79 @@ def test_main_database_warns_when_wal_stays_above_the_high_water_mark(
assert any("main_database_wal_high" in record.message for record in caplog.records)
def test_a_wal_whose_frames_are_all_checkpointed_is_not_reported_high(
tmp_path, monkeypatch, caplog
):
"""#245: the alert judges frames still waiting, not the file's size."""
path = _database(tmp_path)
database = main_database(path)
monkeypatch.setattr(_access, "MAIN_DATABASE_CHECKPOINT_WRITE_INTERVAL", 1)
# One page: any frame left uncopied would cross it, a fully copied WAL not.
monkeypatch.setattr(_access, "MAIN_DATABASE_WAL_WARN_BYTES", 1)
caplog.set_level("WARNING", logger="one_memory.database_access")
with database.write() as db, db:
db.execute("UPDATE counter SET value = 1 WHERE id = 1")
assert database._wal_bytes() > 0
assert not any("main_database_wal_high" in record.message for record in caplog.records)
def test_the_writer_caps_the_wal_it_leaves_behind(tmp_path):
path = _database(tmp_path)
with main_database(path).write() as db:
limit = db.execute("PRAGMA journal_size_limit").fetchone()[0]
assert limit == _access.MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES
def test_an_idle_checkpoint_shrinks_the_wal_after_a_large_burst(tmp_path, monkeypatch):
"""#245: a WAL that grew once used to stay that size for good."""
path = _database(tmp_path)
database = main_database(path)
# No cadence checkpoint during the burst: the WAL has to grow.
monkeypatch.setattr(_access, "MAIN_DATABASE_CHECKPOINT_WRITE_INTERVAL", 10_000_000)
monkeypatch.setattr(_access, "MAIN_DATABASE_CHECKPOINT_SECONDS", 10_000_000.0)
database._last_checkpoint_at = time.monotonic()
with database.write() as db, db:
db.execute("PRAGMA wal_autocheckpoint = 0")
db.execute("CREATE TABLE blobs (body BLOB)")
db.executemany(
"INSERT INTO blobs VALUES (randomblob(?))", [(64 * 1024,) for _ in range(1_500)]
)
with database.write() as db:
db.execute("PRAGMA wal_autocheckpoint = 1000")
grown = database._wal_bytes()
assert grown > _access.MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES
with database.read() as reader:
reader.execute("SELECT COUNT(*) FROM blobs").fetchone()
# Someone is reading: no truncate is even attempted.
assert database.checkpoint_if_idle() is None
assert _access.checkpoint_idle_main_databases() == 1
assert database._wal_bytes() <= _access.MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES
with database.read() as reader:
assert reader.execute("SELECT COUNT(*) FROM blobs").fetchone() == (1_500,)
def test_a_busy_idle_checkpoint_is_skipped_not_raised(tmp_path):
path = _database(tmp_path)
database = main_database(path)
with database.write() as db, db:
db.execute("UPDATE counter SET value = 1 WHERE id = 1")
# Another process's reader holds a snapshot: TRUNCATE cannot finish.
outsider = sqlite3.connect(path)
try:
outsider.execute("BEGIN")
outsider.execute("SELECT value FROM counter").fetchone()
result = database.checkpoint_if_idle()
assert result is None or result[0] == 1
finally:
outsider.close()
assert database.checkpoint_if_idle() == (0, 0, 0)
assert database._wal_bytes() == 0
def test_main_database_maintenance_checkpoint_does_not_break_an_active_reader(
tmp_path, monkeypatch
):
@@ -293,6 +293,7 @@ def test_sdk_user_profile_http_returns_native_assembled_profile(tmp_path):
response = await client.get("/api/v2/user-profile")
assert response.status_code == 200, response.text
assert response.json() == {
"user_id": "local",
"request_id": "profile-request",
"profile": {
"memory_id": "l0-head",
@@ -660,6 +661,8 @@ def test_memory_add_routes_do_not_collide_with_single_memory_routes(tmp_path):
"task_id",
"memory_ids",
"error",
"error_code",
"next_retry_at",
}
status = await client.get(f"/api/v2/memories/additions/{request_id}")
assert status.json()["evidence_id"] == created.json()["evidence_id"]
+166 -5
View File
@@ -28,6 +28,8 @@ from one_memory.memory_features import (
FEATURE_NAMES,
FEATURES,
FEATURES_BY_NAME,
PARAMETERS,
SETTING_NAMES,
MemoryFeatures,
effective_features,
feature_activity,
@@ -46,15 +48,20 @@ TODAY = {
"session_episodes": False,
"episode_summary": False,
"intention": True,
"intention_dedup": True,
"few_shot": False,
}
#: The same for the numeric settings.
TODAY_PARAMETERS = {"intention_dedup_threshold": 0.95}
@pytest.fixture(autouse=True)
def _clean_sdk_env(monkeypatch):
"""The SDK's own env vars feed its defaults; tests start without them."""
for spec in FEATURES:
for spec in (*FEATURES, *PARAMETERS):
monkeypatch.delenv(spec.env, raising=False)
# The threshold falls back to the SDK's shared dedup variable.
monkeypatch.delenv("MEMORY_DEDUP_THRESHOLD", raising=False)
monkeypatch.delenv("ONE_MEMORY_GRAPH_STORE", raising=False)
@@ -95,17 +102,25 @@ def _field(config, name: str) -> bool:
def test_the_document_names_exactly_the_catalog() -> None:
assert tuple(MemoryFeatures.model_fields) == FEATURE_NAMES
assert tuple(MemoryFeatures.model_fields) == SETTING_NAMES
assert set(TODAY) == set(FEATURE_NAMES)
# Each feature is a MemoryConfig field the SDK has; no env var is written.
defaults = hy_sdk_runtime.sdk_feature_defaults()
assert set(defaults) == set(FEATURE_NAMES)
assert set(defaults) == set(SETTING_NAMES)
def test_the_defaults_are_todays_behaviour() -> None:
assert hy_sdk_runtime.sdk_feature_defaults() == TODAY
everything = {**TODAY, **TODAY_PARAMETERS}
assert hy_sdk_runtime.sdk_feature_defaults() == everything
assert MemoryFeatures().overrides() == ()
assert effective_features((), TODAY) == TODAY
assert effective_features((), everything) == everything
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
def test_intention_dedup_defaults_to_on_at_095_in_the_built_config(tmp_path, profile) -> None:
config = _config(tmp_path, profile)
assert config.recall.intention_dedup_enabled is True
assert config.recall.intention_dedup_threshold == 0.95
def test_the_sdk_env_var_still_moves_the_default_of_an_unset_feature(monkeypatch) -> None:
@@ -126,6 +141,32 @@ def test_unknown_or_non_boolean_values_are_refused() -> None:
)
@pytest.mark.parametrize("bad", (0.79, 1.01, -1, 2, True, "0.9", float("nan"), float("inf")))
def test_an_intention_dedup_threshold_outside_its_range_is_refused(bad) -> None:
with pytest.raises(ValidationError):
MemoryFeatures.model_validate({"intention_dedup_threshold": bad})
def test_the_intention_dedup_threshold_range_is_inclusive() -> None:
for value in (0.8, 0.9, 1, 1.0):
document = MemoryFeatures.model_validate({"intention_dedup_threshold": value})
assert document.overrides() == (("intention_dedup_threshold", float(value)),)
with pytest.raises(ValueError, match="invalid_memory_feature_value"):
hy_sdk_runtime.build_sdk_config(
PROFILES[0],
api_key=None,
data_dir=Path("/tmp/unused"),
features={"intention_dedup_threshold": 0.5},
)
def test_the_sdk_env_vars_still_move_the_threshold_default(monkeypatch) -> None:
monkeypatch.setenv("MEMORY_DEDUP_THRESHOLD", "0.9")
assert hy_sdk_runtime.sdk_feature_defaults()["intention_dedup_threshold"] == 0.9
monkeypatch.setenv("MEMORY_INTENTION_DEDUP_THRESHOLD", "0.97")
assert hy_sdk_runtime.sdk_feature_defaults()["intention_dedup_threshold"] == 0.97
def test_merge_sets_resets_and_leaves_absent_keys_alone() -> None:
saved = MemoryFeatures(graph_rerank=True, few_shot=True)
update = MemoryFeatures.model_validate({"few_shot": None, "episode_recall": True})
@@ -165,6 +206,20 @@ def test_each_feature_reaches_the_built_memory_config(tmp_path, profile) -> None
assert _field(config, other.name) is TODAY[other.name], (spec.name, other.name)
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
def test_intention_dedup_on_off_and_threshold_reach_the_built_memory_config(
tmp_path, profile
) -> None:
off = _config(tmp_path, profile, {"intention_dedup": False})
assert off.recall.intention_dedup_enabled is False
assert off.recall.intention_dedup_threshold == 0.95
on = _config(tmp_path, profile, {"intention_dedup": True, "intention_dedup_threshold": 0.9})
assert on.recall.intention_dedup_enabled is True
assert on.recall.intention_dedup_threshold == 0.9
# Nothing else moves.
assert _field(on, "intention") is True and _field(on, "graph_rerank") is False
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
def test_nothing_set_is_the_config_of_before_byte_for_byte(tmp_path, profile) -> None:
before = repr(dataclasses.asdict(_config(tmp_path, profile)))
@@ -172,6 +227,8 @@ def test_nothing_set_is_the_config_of_before_byte_for_byte(tmp_path, profile) ->
assert repr(dataclasses.asdict(_config(tmp_path, profile, {}))) == before
# Setting every feature to its default value changes nothing either.
assert repr(dataclasses.asdict(_config(tmp_path, profile, TODAY))) == before
everything = {**TODAY, **TODAY_PARAMETERS}
assert repr(dataclasses.asdict(_config(tmp_path, profile, everything))) == before
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
@@ -190,6 +247,7 @@ def test_features_never_change_the_collection_identity(tmp_path, profile) -> Non
{name: True for name in FEATURE_NAMES},
{name: False for name in FEATURE_NAMES},
{"graph_rerank": True, "episode_recall": True},
{"intention_dedup": False, "intention_dedup_threshold": 0.8},
):
assert identity(_config(tmp_path, profile, values)) == baseline
@@ -243,6 +301,13 @@ def test_a_feature_needing_a_higher_mode_is_reported_inactive() -> None:
series = effective_features((("same_series_timeline", True), ("graph_rerank", True)), TODAY)
assert feature_activity("same_series_timeline", series, "pro") == "requires_ultra"
assert feature_activity("same_series_timeline", series, "ultra") is None
defaults = {**TODAY, **TODAY_PARAMETERS}
assert feature_activity("intention_dedup", defaults, "pro") is None
assert feature_activity("intention_dedup_threshold", defaults, "pro") is None
assert feature_activity("intention_dedup_threshold", defaults, "lite") == "requires_pro"
off = effective_features((("intention_dedup", False),), defaults)
assert feature_activity("intention_dedup", off, "pro") == "disabled"
assert feature_activity("intention_dedup_threshold", off, "pro") == "requires_intention_dedup"
# --- rebuilding handles -------------------------------------------------------
@@ -497,3 +562,99 @@ def test_features_api_get_put_reset_and_status(tmp_path, monkeypatch) -> None:
assert (await client.get("/api/v1/settings/memory-features")).json()["non_default"] == {}
run_app(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
def test_intention_dedup_api_on_off_threshold_and_422(tmp_path, monkeypatch) -> None:
built: list[tuple[bool, float]] = []
original = hy_sdk_runtime.build_sdk_config
def recording(*args, **kwargs):
config = original(*args, **kwargs)
built.append(
(config.recall.intention_dedup_enabled, config.recall.intention_dedup_threshold)
)
return config
monkeypatch.setattr(hy_sdk_runtime, "build_sdk_config", recording)
def built_now(app) -> tuple[bool, float]:
"""What a client built with the read handle's current key carries."""
recording(
hy_sdk_runtime.builtin_retrieval_snapshot(),
api_key=None,
data_dir=tmp_path,
features=app.state.sdk_client_handle.current_features(),
)
return built[-1]
async def scenario(app, client):
path = "/api/v1/settings/memory-features"
current = (await client.get(path)).json()
dedup = _by_name(current)["intention_dedup"]
assert dedup["enabled"] is True and dedup["default"] is True
assert dedup["configured"] is None and dedup["active"] is True
assert dedup["sdk_field"] == "recall.intention_dedup_enabled"
assert dedup["env"] == "MEMORY_INTENTION_DEDUP_ENABLED"
(threshold,) = current["parameters"]
assert threshold == {
"name": "intention_dedup_threshold",
"value": 0.95,
"configured": None,
"default": 0.95,
"minimum": 0.8,
"maximum": 1.0,
"active": True,
"reason": None,
"requires_feature": "intention_dedup",
"sdk_field": "recall.intention_dedup_threshold",
"env": "MEMORY_INTENTION_DEDUP_THRESHOLD",
"description": threshold["description"],
}
assert built_now(app) == (True, 0.95)
off = await client.put(path, json={"intention_dedup": False})
assert off.status_code == 200, off.text
body = off.json()
assert body["changed"] == ["intention_dedup"]
assert body["non_default"] == {"intention_dedup": False}
assert body["parameters"][0]["active"] is False
assert body["parameters"][0]["reason"] == "requires_intention_dedup"
assert app.state.sdk_write_coordinator.memory_features() == (("intention_dedup", False),)
assert built_now(app) == (False, 0.95)
on = (
await client.put(path, json={"intention_dedup": True, "intention_dedup_threshold": 0.9})
).json()
assert on["changed"] == ["intention_dedup", "intention_dedup_threshold"]
assert on["non_default"] == {"intention_dedup_threshold": 0.9}
assert on["parameters"][0]["value"] == 0.9 and on["parameters"][0]["active"] is True
expected = (("intention_dedup", True), ("intention_dedup_threshold", 0.9))
assert app.state.sdk_client_handle.current_features() == expected
assert built_now(app) == (True, 0.9)
processing = (await client.get("/api/v1/memory-processing")).json()
assert processing["pipeline"]["features"] == {"intention_dedup_threshold": 0.9}
saved = json.loads((tmp_path / "data" / "config.json").read_text())
assert saved["memory_features"] == {
"intention_dedup": True,
"intention_dedup_threshold": 0.9,
}
for bad in (0.5, 0.79, 1.5, -0.1, "0.9", True):
response = await client.put(path, json={"intention_dedup_threshold": bad})
assert response.status_code == 422, bad
assert (await client.put(path, json={"intention_dedup": "false"})).status_code == 422
# A refused body changes nothing.
assert app.state.sdk_client_handle.current_features() == expected
reset = (
await client.put(
path, json={"intention_dedup": None, "intention_dedup_threshold": None}
)
).json()
assert reset["changed"] == ["intention_dedup_threshold"]
assert reset["non_default"] == {}
assert app.state.sdk_client_handle.current_features() == ()
assert built_now(app) == (True, 0.95)
run_app(tmp_path, monkeypatch, scenario)
@@ -1433,25 +1433,6 @@ def test_a_claimed_row_with_no_readable_payload_is_settled_not_left_running(tmp_
assert _rows(path)[0]["lease_owner"] is None
def test_an_unknown_outcome_is_recorded_as_unknown_not_retried(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request())
supervisor = _Recorder(error=RuntimeError("model_content_outcome_unknown"))
_claim_through_drain(queue, record.job_id)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=supervisor,
evidence=_EvidenceStore({"e1"}),
)
)
assert _state(path, record.job_id) == "unknown"
def test_unpermitted_evidence_parks_and_is_readmitted_after_confirmation(tmp_path) -> None:
"""A note claimed before its confirmation must not become terminal failed."""
path = tmp_path / "one-memory.sqlite3"
@@ -1478,34 +1459,6 @@ def test_unpermitted_evidence_parks_and_is_readmitted_after_confirmation(tmp_pat
assert replay.state == PENDING
def test_a_permission_error_after_hy_invocation_stays_unknown(tmp_path) -> None:
"""An error code cannot erase the durable possibly-sent boundary."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request())
_claim_through_drain(queue, record.job_id)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=_Recorder(
error=RuntimeError("model_content_evidence_changed"),
invoke_hy=True,
),
evidence=_EvidenceStore({"e1"}),
)
)
assert queue.get(record.request_id).state == UNKNOWN
with sqlite3.connect(path) as connection:
assert connection.execute(
"SELECT hy_invoked,last_error FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone() == (1, "model_content_evidence_changed")
assert queue.reclaim_expired() == 0
def test_replayed_live_evidence_readmits_a_pre_send_identity_failure(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
@@ -1623,90 +1576,6 @@ def test_startup_never_repairs_a_possibly_sent_identity_failure(tmp_path) -> Non
assert queue.get(job.request_id).state == UNKNOWN
def test_reconciled_absent_timeout_is_requeued_once_without_resetting_attempts(
tmp_path,
) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request())
with sqlite3.connect(path) as connection, connection:
connection.execute(
"UPDATE memory_write_jobs SET state='unknown', attempts=1, "
"hy_invoked=1, last_error='hy_memory_timeout', updated_at=? WHERE job_id=?",
("2026-01-01T00:00:00+00:00", record.job_id),
)
assert (
queue.retry_timed_out_unknown_after_reconcile(
older_than=datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
)
== 1
)
with sqlite3.connect(path) as connection:
state, attempts, invoked, error = connection.execute(
"SELECT state, attempts, hy_invoked, last_error FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone()
assert (state, attempts, invoked, error) == (
"pending",
1,
0,
"model_content_reconciled_absent_retry",
)
def test_unresolved_unknown_is_automatically_retried_only_once(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request())
with sqlite3.connect(path) as connection, connection:
connection.execute(
"UPDATE memory_write_jobs SET state='unknown',attempts=1,hy_invoked=1,"
"last_error='model_content_outcome_unknown',updated_at=? WHERE job_id=?",
("2026-01-01T00:00:00+00:00", record.job_id),
)
cutoff = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
assert queue.retry_first_unknown_after_reconcile(older_than=cutoff) == 1
first_retry = queue.get(record.request_id)
assert first_retry.state == PENDING
with sqlite3.connect(path) as connection:
assert (
connection.execute(
"SELECT attempts FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone()[0]
== 1
)
drain = DurableWriteDrain(queue, execute=_never_execute, owner="retry-owner")
assert drain.claim(limit=1) == (record.job_id,)
assert queue.mark_invoked(record.job_id, owner="retry-owner")
queue.settle(
record.job_id,
state=UNKNOWN,
owner="retry-owner",
error_code="model_content_outcome_unknown",
hy_invoked=True,
)
with sqlite3.connect(path) as connection:
assert (
connection.execute(
"SELECT attempts FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone()[0]
== 2
)
assert (
queue.retry_first_unknown_after_reconcile(
older_than=datetime.now(UTC) + timedelta(minutes=1)
)
== 0
)
assert queue.get(record.request_id).state == UNKNOWN
def test_drain_runs_unknown_reconciliation_during_busy_safe_maintenance(tmp_path) -> None:
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
calls = 0
@@ -1759,26 +1628,6 @@ def test_a_reclaimed_uninvoked_job_with_attempts_left_is_retryable(tmp_path) ->
assert _rows(path)[0]["attempts"] == 0
def test_post_invocation_transport_failure_stays_unknown_not_pending(tmp_path) -> None:
"""Once the bridge may have been opened, no error is safe to replay."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request())
_claim_through_drain(queue, record.job_id)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=_Recorder(error=RuntimeError("hy_memory_timeout")),
evidence=_EvidenceStore({"e1"}),
)
)
assert _state(path, record.job_id) == UNKNOWN
assert _rows(path)[0]["hy_invoked"] == 1
@pytest.mark.parametrize(
"code,expected",
[
@@ -1900,7 +1749,9 @@ def test_a_failed_job_logs_where_it_failed_but_never_the_message(tmp_path, caplo
for message in caplog.records
if "durable_write_job_failed" in message.getMessage()
]
assert f"job_id={record.job_id} error_code=RuntimeError chain=RuntimeError@" in line
assert (
f"job_id={record.job_id} error_code=RuntimeError state=pending chain=RuntimeError@" in line
)
assert ";AttributeError@" in line
assert "test_memory_server_durable_queue.py:" in line and ":closed_store" in line
assert "secret diary" not in line
@@ -2620,57 +2471,6 @@ def test_a_wedged_job_hits_the_work_ceiling_and_settles_unknown(tmp_path) -> Non
assert (state, owner, expiry) == ("unknown", None, None)
def test_user_confirmed_unknown_retry_requeues_job_and_linked_tasks(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request(evidence=("e1",)))
with sqlite3.connect(path) as connection, connection:
connection.execute(
"UPDATE memory_write_jobs SET state='unknown', attempts=2, hy_invoked=1, "
"last_error='hy_memory_timeout' WHERE job_id=?",
(record.job_id,),
)
connection.executescript(
"""
CREATE TABLE memory_tasks (
task_id TEXT PRIMARY KEY,
status TEXT NOT NULL,
runner_token TEXT,
attempts INTEGER NOT NULL,
last_error_code TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE memory_task_inputs (
task_id TEXT PRIMARY KEY,
refs_json TEXT NOT NULL
);
"""
)
connection.execute(
"INSERT INTO memory_tasks VALUES (?,?,?,?,?,?)",
("task-1", "unknown", None, 2, "model_content_outcome_unknown", "old"),
)
connection.execute(
"INSERT INTO memory_task_inputs VALUES (?,?)",
("task-1", '[{"evidence_id":"e1"}]'),
)
assert queue.retry_unknown_confirmed(record.job_id) == 1
retried = queue.get(record.request_id)
assert retried is not None
assert retried.state == PENDING
with sqlite3.connect(path) as connection:
assert connection.execute(
"SELECT attempts, hy_invoked FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone() == (0, 0)
assert connection.execute(
"SELECT status, attempts, last_error_code FROM memory_tasks WHERE task_id='task-1'"
).fetchone() == ("pending", 0, "model_content_manual_retry_requested")
assert queue.retry_unknown_confirmed(record.job_id) == 0
def test_restart_redrains_a_job_claimed_by_a_worker_that_died(tmp_path) -> None:
"""End-to-end restart recovery: a claimed-but-not-run job runs once.
@@ -2922,30 +2722,6 @@ def test_forever_drain_admits_a_new_source_while_an_earlier_source_is_running(tm
assert queue.get("workbuddy-1").state == COMPLETED
def test_sqlite_drain_preserves_unknown_after_execute_job(tmp_path) -> None:
"""A drain failure after invocation must remain unknown, not look retryable."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
queue.submit(_request(evidence=("e1",)))
async def execute(job_id: str) -> None:
await execute_job(
job_id,
queue=queue,
supervisor=_Recorder(error=RuntimeError("model_content_outcome_unknown")),
evidence=_EvidenceStore({"e1"}),
)
async def scenario() -> None:
drain = DurableWriteDrain(queue, execute=execute)
assert await drain.run_one() == 1
asyncio.run(scenario())
assert queue.get("idem-1").state == "unknown"
assert queue.reclaim_expired() == 0
assert queue.get("idem-1").state == "unknown"
def test_sqlite_drain_keeps_running_after_an_executor_raises(tmp_path) -> None:
"""One executor exception must not kill the drain or strand other rows.
@@ -4222,27 +3998,6 @@ def test_set_state_rejects_an_unvalidated_state(tmp_path) -> None:
assert registry.get(record.request_id).state == "accepted"
def test_set_state_cannot_resolve_unknown_without_an_inquiry(tmp_path) -> None:
"""The HTTP-facing setter must not bypass the inquiry-only exit.
`DurableWriteQueue.settle` deliberately treats `unknown` as sticky; a
direct `set_state("completed"/"failed")` would otherwise turn a
possibly-delivered write into a terminal claim without asking HY. The
durable registry is the shared caller surface, so it must preserve that
boundary even though today only tests call it.
"""
from one_memory.memory_server.durable_queue import DurableWriteRegistry
path = tmp_path / "one-memory.sqlite3"
registry = DurableWriteRegistry(path)
durable = registry.queue.submit(_request())
registry.queue.settle(durable.job_id, state=UNKNOWN, hy_invoked=True)
assert registry.set_state(durable.request_id, "completed") is None
assert registry.set_state(durable.request_id, "failed") is None
assert registry.get(durable.request_id).state == UNKNOWN
@pytest.mark.parametrize("state", (WAITING_FOR_PERMISSION, WAITING_FOR_MODEL))
def test_set_state_accepts_the_route_parked_states(tmp_path, state: str) -> None:
"""The durable setter must accept every non-terminal route state.
@@ -5435,83 +5190,6 @@ def test_readmit_paths_never_requeue_a_possibly_sent_row(tmp_path) -> None:
assert DurableWriteDrain(queue, execute=_never_execute).claim(limit=5) == ()
def test_a_refused_source_context_is_nonblocking_unknown(
tmp_path,
) -> None:
"""An ambiguous call-boundary failure is retained but never blocks FIFO."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request(key="refused-scope", source="opencode"))
claimer = DurableWriteDrain(queue, execute=_never_execute, owner="refused-owner")
assert claimer.claim(limit=1) == (record.job_id,)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=_Recorder(
error=RuntimeError("hy_memory_write_verification_context_scope"),
invoke_hy=True,
),
evidence=_EvidenceStore({"e1"}),
owner="refused-owner",
)
)
with sqlite3.connect(path.as_uri() + "?mode=ro", uri=True) as db:
state, invoked, last_error = db.execute(
"SELECT state, hy_invoked, last_error FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone()
assert state == UNKNOWN
assert invoked == 1
assert last_error == "hy_memory_write_verification_context_scope"
assert DurableWriteDrain(queue, execute=_never_execute).claim(limit=1) == ()
def test_a_failure_after_the_call_still_parks_as_unknown(tmp_path) -> None:
"""The fix must not widen: a post-call refusal is still possibly-sent."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request(key="after-call", source="opencode"))
claimer = DurableWriteDrain(queue, execute=_never_execute, owner="after-call-owner")
assert claimer.claim(limit=1) == (record.job_id,)
recorder = _Recorder(
error=RuntimeError("hy_memory_write_verification_authority_invalid"),
invoke_hy=True,
)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=recorder,
evidence=_EvidenceStore({"e1"}),
owner="after-call-owner",
)
)
with sqlite3.connect(path.as_uri() + "?mode=ro", uri=True) as db:
state, invoked = db.execute(
"SELECT state, hy_invoked FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone()
assert state == UNKNOWN
assert invoked == 1
assert len(recorder.seen) == 1
assert (
queue.retry_first_unknown_after_reconcile(
older_than=datetime.now(UTC) + timedelta(minutes=1)
)
== 0
)
assert queue.retry_unknown_confirmed(record.job_id) == 0
assert DurableWriteDrain(queue, execute=_never_execute).claim(limit=1) == ()
assert len(recorder.seen) == 1
def test_event_time_not_enqueue_order_drives_the_window(
tmp_path,
) -> None:
@@ -5674,3 +5352,93 @@ def test_hand_added_memories_still_run_one_at_a_time_in_order(tmp_path) -> None:
assert drain.claim(limit=5) == ()
assert queue.get("note-2").state == PENDING
assert second.job_id
def test_a_job_the_execute_gate_releases_is_not_reclaimed_while_it_still_refuses(
tmp_path,
) -> None:
"""The claim gate and the execute gate can disagree for good (#245).
The claim asks about the lane's user, the execute gate about the user of
the job's Evidence. An id-only pre-upgrade payload sits in the `local`
lane even when its Evidence belongs to someone else: while that user
evolves, the claim admits the job and the execute gate hands it back.
Released with no hold, the very next tick claimed it again, in a loop.
"""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request(key="evolving-user", source="cursor"))
drain = DurableWriteDrain(queue, execute=_never_execute, owner="gate-owner")
assert drain.claim(limit=1) == (record.job_id,)
asyncio.run(
execute_job(
record.job_id,
queue=queue,
supervisor=_Recorder(error=RuntimeError("memory_evolve_in_progress")),
evidence=_EvidenceStore({"e1"}),
owner="gate-owner",
)
)
assert queue.get(record.request_id).state == PENDING
now = datetime.now(UTC)
assert drain.claim(limit=1, now=now) == ()
assert drain.claim(limit=1, now=now + timedelta(seconds=4)) == ()
assert drain.claim(limit=1, now=now + timedelta(seconds=6)) == (record.job_id,)
with sqlite3.connect(path) as connection:
assert connection.execute(
"SELECT attempts,hy_invoked FROM memory_write_jobs WHERE job_id=?",
(record.job_id,),
).fetchone() == (1, 0)
def _suspension_runtime(queue: DurableWriteQueue, enabled: set[str]):
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
return SdkMemoryRuntime(
coordinator=SimpleNamespace(),
read_handle=SimpleNamespace(), # type: ignore[arg-type]
write_queue=queue,
extraction_settings=SimpleNamespace(), # type: ignore[arg-type]
data_dir=queue.database_path.parent,
source_active=lambda source_key: source_key in enabled,
)
def test_a_disabled_source_suspends_its_jobs_and_enabling_it_resumes_them(tmp_path) -> None:
"""A source the user turned off keeps its jobs, out of the pending count (#245)."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
recent = datetime.now(UTC)
for index in range(3):
queue.submit(_request(key=f"cursor-{index}", source="cursor", event_at=recent))
codex = queue.submit(_request(key="codex-0", source="codex", event_at=recent))
enabled = {"cursor", "codex"}
runtime = _suspension_runtime(queue, enabled)
drain = DurableWriteDrain(
queue,
execute=_never_execute,
owner="suspend-owner",
source_writes_permitted=lambda source_key: source_key in enabled,
)
assert runtime.status_counts() == {PENDING: 4}
assert runtime.queue_depths().fresh == 4
enabled.discard("cursor")
assert runtime.status_counts() == {PENDING: 1, "suspended": 3}
assert runtime.queue_depths().fresh == 1
assert drain.claim(limit=10) == (codex.job_id,)
queue.settle(codex.job_id, state=COMPLETED, hy_invoked=True, owner="suspend-owner")
assert drain.claim(limit=10) == ()
assert runtime.status_counts() == {COMPLETED: 1, "suspended": 3}
# Nothing was cancelled, moved or deleted: the rows are as they were.
assert [row["state"] for row in _rows(path) if row["source_key"] == "cursor"] == [PENDING] * 3
enabled.add("cursor")
assert runtime.status_counts() == {PENDING: 3, COMPLETED: 1}
assert runtime.queue_depths().fresh == 3
assert len(drain.claim(limit=10)) == 1
+3 -3
View File
@@ -1181,7 +1181,7 @@ def test_authority_invalid_task_is_never_reinvoked(tmp_path) -> None:
assert retained.last_error_code == "hy_memory_write_verification_authority_invalid"
def test_linked_unknown_is_retried_only_by_the_durable_job(tmp_path) -> None:
def test_linked_unknown_is_rerun_only_by_the_durable_job(tmp_path) -> None:
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
from one_memory.memory_server.durable_queue import DurableWriteQueue
@@ -1241,8 +1241,8 @@ def test_linked_unknown_is_retried_only_by_the_durable_job(tmp_path) -> None:
cutoff = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
assert store.retry_first_unknown_after_reconcile(older_than=cutoff) == 0
assert store.require(task.task_id).status is MemoryTaskStatus.UNKNOWN
assert queue.retry_first_unknown_after_reconcile(older_than=cutoff) == 1
assert store.require(task.task_id).status is MemoryTaskStatus.PENDING
# The durable job is the only execution entry: it re-runs itself.
assert queue.recover_unknown() == 1
assert queue.get(job.request_id).state == "pending"
assert (
store.active_window(
@@ -0,0 +1,607 @@
"""A broken model configuration must not turn a backlog into a request storm (#245)."""
from __future__ import annotations
import asyncio
import sqlite3
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
import httpx
import pytest
from httpx import ASGITransport, AsyncClient
from pydantic import SecretStr
from one_memory.api import create_app
from one_memory.config import Settings
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
from one_memory.memory_server.durable_queue import (
FAILED,
PENDING,
DurableWriteDrain,
DurableWriteQueue,
execute_job,
)
from one_memory.model_backoff import (
BACKOFF_CAP_SECONDS,
BackedOffModelTransport,
ModelBackoff,
ModelBackoffRefusal,
backoff_seconds,
)
from one_memory.secrets import InMemorySecretStore
CHAT = "https://models.example.com/v1/chat/completions"
EMBEDDINGS = "https://models.example.com/v1/embeddings"
class _Clock:
def __init__(self) -> None:
self.now = 0.0
def __call__(self) -> float:
return self.now
def _request(index: int) -> WriteRequest:
session = f"sess-{index}"
return WriteRequest(
internal_session_id=session,
batch_identity=BatchIdentity(internal_session_id=session, first_round=1, last_round=5),
evidence_ids=[f"e-{index}"],
idempotency_key=f"idem-{index}",
# Twenty sources, so the drain could claim ten jobs at once.
source_id=f"source-{index % 20}",
)
async def _never_execute(job_id: str) -> None:
raise AssertionError(f"the simulation settles {job_id} itself")
def _send(transport: BackedOffModelTransport, url: str = CHAT) -> int | None:
try:
response = asyncio.run(transport.handle_async_request(httpx.Request("POST", url)))
except ModelBackoffRefusal:
return None
return response.status_code
def test_the_pause_doubles_from_five_seconds_and_stops_at_three_minutes() -> None:
assert [backoff_seconds(level) for level in range(1, 9)] == [
5.0,
10.0,
20.0,
40.0,
80.0,
160.0,
180.0,
180.0,
]
assert backoff_seconds(1000) == BACKOFF_CAP_SECONDS == 180.0
def test_an_always_failing_provider_sees_a_bounded_number_of_calls(tmp_path) -> None:
"""200 pending jobs against a provider that refuses every one.
Before the pause the drain claimed the next job the moment the last one
failed: hundreds of requests a minute. Now the calls follow the schedule.
"""
clock = _Clock()
backoff = ModelBackoff(clock=clock)
calls: list[float] = []
def provider(_request: httpx.Request) -> httpx.Response:
calls.append(clock.now)
return httpx.Response(401)
transport = BackedOffModelTransport(
backoff, profile_id="p:1", inner=httpx.MockTransport(provider)
)
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
for index in range(200):
queue.submit(_request(index))
drain = DurableWriteDrain(
queue,
execute=_never_execute,
owner="sim",
model_claim_allowance=backoff.allowance,
)
claimed = 0
for tick in range(6001):
# Ten minutes in tenths of a second, without float drift.
clock.now = tick / 10
for job_id in drain.claim(limit=10):
claimed += 1
_send(transport)
queue.settle(job_id, state=FAILED, owner="sim", error_code="model_auth_failed")
# The first claim was taken before anything had failed: two of its
# requests reached the provider and the rest were refused locally. After
# that one probe per pause: 5, 10, 20, 40, 80, 160, then 180 seconds.
assert calls[:2] == [0.0, 0.0]
probes = calls[2:]
assert probes == pytest.approx([5.0, 15.0, 35.0, 75.0, 155.0, 315.0, 495.0], abs=0.11)
gaps = [later - earlier for earlier, later in zip(probes, probes[1:], strict=False)]
assert max(gaps) <= BACKOFF_CAP_SECONDS + 0.11
assert len(calls) == 9
assert claimed == 17
with sqlite3.connect(tmp_path / "one-memory.sqlite3") as database:
(pending,) = database.execute(
"SELECT COUNT(*) FROM memory_write_jobs WHERE state=?", (PENDING,)
).fetchone()
assert pending == 200 - claimed
def test_a_success_ends_the_pause_at_once() -> None:
clock = _Clock()
backoff = ModelBackoff(clock=clock)
replies = [httpx.Response(503), httpx.Response(503), httpx.Response(200)]
transport = BackedOffModelTransport(
backoff, profile_id="p:1", inner=httpx.MockTransport(lambda _r: replies.pop(0))
)
assert _send(transport) == 503
clock.now = 5.0
assert backoff.allowance() == 1
assert _send(transport) == 503
assert backoff.allowance() == 0
assert backoff.state().last_error == "model_server_error"
clock.now = 15.0
assert backoff.allowance() == 1
assert _send(transport) == 200
assert backoff.allowance() is None
assert backoff.state().until is None
assert backoff.state().failures == 0
def test_a_working_embedding_endpoint_does_not_clear_a_broken_chat_model() -> None:
clock = _Clock()
backoff = ModelBackoff(clock=clock)
def provider(request: httpx.Request) -> httpx.Response:
return httpx.Response(200 if request.url.path.endswith("/embeddings") else 404)
transport = BackedOffModelTransport(
backoff, profile_id="p:1", inner=httpx.MockTransport(provider)
)
assert _send(transport) == 404
assert _send(transport) == 404
# Recall's embeddings still go out while the chat model is refused.
assert _send(transport, EMBEDDINGS) == 200
assert _send(transport) is None
assert backoff.state().last_error == "model_not_found"
assert backoff.paused()
def test_an_unreachable_provider_is_paused_like_a_refusing_one() -> None:
clock = _Clock()
backoff = ModelBackoff(clock=clock)
def provider(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("name resolution failed", request=request)
transport = BackedOffModelTransport(
backoff, profile_id="p:1", inner=httpx.MockTransport(provider)
)
with pytest.raises(httpx.ConnectError):
_send(transport)
assert backoff.state().last_error == "model_unreachable"
assert backoff.allowance() == 0
def test_a_request_problem_is_left_to_its_job() -> None:
backoff = ModelBackoff(clock=_Clock())
transport = BackedOffModelTransport(
backoff, profile_id="p:1", inner=httpx.MockTransport(lambda _r: httpx.Response(400))
)
assert _send(transport) == 400
assert backoff.allowance() is None
def test_a_configuration_change_ends_the_pause() -> None:
clock = _Clock()
backoff = ModelBackoff(clock=clock)
for _ in range(6):
clock.now += 200.0
backoff.record_failure("model_auth_failed", "p:1", "chat")
assert backoff.state().until is not None
backoff.reset()
assert backoff.allowance() is None
assert backoff.state().until is None
# A new revision of the profile is a new configuration as well.
backoff.record_failure("model_auth_failed", "p:1", "chat")
backoff.record_failure("model_auth_failed", "p:2", "chat")
assert backoff.state().failures == 1
def test_a_probe_that_never_reports_does_not_hold_the_queue_forever() -> None:
clock = _Clock()
backoff = ModelBackoff(clock=clock)
backoff.record_failure("model_timeout", "p:1", "chat")
clock.now = 5.0
assert backoff.allowance() == 1
assert backoff.allowance() == 0
clock.now = 35.0
assert backoff.allowance() == 1
def test_memory_processing_shows_the_pause_and_a_profile_save_ends_it(tmp_path) -> None:
async def scenario() -> None:
app = create_app(
Settings(
data_dir=tmp_path / "data",
source_home=tmp_path,
api_token=SecretStr("local-token"),
),
secret_store=InMemorySecretStore(),
)
headers = {"Authorization": "Bearer local-token", "Origin": "http://127.0.0.1:5173"}
async with (
app.router.lifespan_context(app),
AsyncClient(
transport=ASGITransport(app=app), base_url="http://test", headers=headers
) as client,
):
created = await client.post(
"/api/v1/model-profiles",
json={
"name": "Provider",
"provider_kind": "openai_compatible",
"base_url": "https://models.example.com/v1",
"model": "memory-model",
"embedding_model": "embedding-model",
"embedding_base_url": "https://models.example.com/v1",
"api_key": "api-route-secret",
},
)
assert created.status_code == 200
backoff: ModelBackoff = app.state.model_backoff
backoff.record_failure("model_auth_failed", "p:1", "chat")
paused = (await client.get("/api/v1/memory-processing")).json()
assert paused["llm_backoff_error"] == "model_auth_failed"
until = datetime.fromisoformat(paused["llm_backoff_until"])
assert until > datetime.now(UTC)
assert paused["suspended"] == 0
saved = await client.patch(
f"/api/v1/model-profiles/{created.json()['profile_id']}",
json={"expected_revision": created.json()["revision"], "model": "other-model"},
)
assert saved.status_code == 200
cleared = (await client.get("/api/v1/memory-processing")).json()
assert cleared["llm_backoff_until"] is None
assert cleared["llm_backoff_error"] is None
assert backoff.allowance() is None
asyncio.run(scenario())
# --- Every write retries itself: provider failures wait, others retry 3 times ---
class _FakeSdk:
"""Stands in for the SDK write path, with its real replay rule.
Like `HyMemoryClient`, it keeps one outcome per request id and replays a
failed one without calling the model. Every write goes out through the
breaker's transport, so the provider sees exactly what MemHub sends.
"""
def __init__(self, transport: BackedOffModelTransport | None = None) -> None:
self.transport = transport
self.outcomes: dict[str, str] = {}
self.written: list[str] = []
self.request_ids: list[str] = []
self.fail_with: str | None = None
async def process_sdk_job(self, records, *, request_id, before_hy_invocation):
await before_hy_invocation()
self.request_ids.append(request_id)
if request_id in self.outcomes:
if self.outcomes[request_id] == "failed":
raise RuntimeError("hy_sdk_write_failed")
raise AssertionError("a succeeded id is never sent again")
if self.fail_with is not None:
self.outcomes[request_id] = "failed"
raise RuntimeError(self.fail_with)
if self.transport is not None:
try:
response = await self.transport.handle_async_request(httpx.Request("POST", CHAT))
except httpx.TransportError:
self.outcomes[request_id] = "unknown"
raise
if response.status_code >= 400:
self.outcomes[request_id] = "failed"
raise RuntimeError("hy_sdk_write_failed")
self.outcomes[request_id] = "succeeded"
self.written.extend(record.evidence_id for record in records)
return SimpleNamespace(raw_ids=(f"raw-{request_id}",), memory_ids=(f"m-{request_id}",))
async def inquire_write_outcome(self, request_id):
if self.outcomes.get(request_id) != "succeeded":
return None
return SimpleNamespace(raw_ids=(f"raw-{request_id}",), memory_ids=(f"m-{request_id}",))
class _Evidence:
def get(self, evidence_id: str):
return SimpleNamespace(evidence_id=evidence_id, source_id="cursor", user_id="local")
def _ordered_request(index: int, *, source: str = "cursor") -> WriteRequest:
return WriteRequest(
internal_session_id="sess-1",
batch_identity=BatchIdentity(
internal_session_id="sess-1", first_round=index + 1, last_round=index + 1
),
evidence_ids=[f"e-{index:03d}"],
idempotency_key=f"idem-{index:03d}",
source_id=source,
event_at=datetime(2026, 10, 1, tzinfo=UTC) + timedelta(minutes=index),
)
def _jobs(path) -> list[tuple[str, str, int, int, str | None]]:
with sqlite3.connect(path) as database:
return database.execute(
"SELECT idempotency_key,state,sdk_attempt,retries,last_error "
"FROM memory_write_jobs ORDER BY queue_seq"
).fetchall()
def test_an_outage_with_200_queued_jobs_ends_with_every_job_written_once_in_order(
tmp_path,
) -> None:
"""The provider refuses every request for five minutes, then recovers.
No job is lost, skipped, written twice or left `unknown`; the order is
the order of the conversation; the provider sees a handful of probes
during the outage, not one request per job.
"""
clock = _Clock()
backoff = ModelBackoff(clock=clock)
calls: list[tuple[float, int]] = []
def provider(_request: httpx.Request) -> httpx.Response:
status = 401 if clock.now < 300.0 else 200
calls.append((clock.now, status))
return httpx.Response(status)
sdk = _FakeSdk(
BackedOffModelTransport(backoff, profile_id="p:1", inner=httpx.MockTransport(provider))
)
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
for index in range(200):
queue.submit(_ordered_request(index))
drain = DurableWriteDrain(
queue,
execute=_never_execute,
owner="outage",
model_claim_allowance=backoff.allowance,
)
async def tick() -> None:
await asyncio.gather(
*(
execute_job(
job_id,
queue=queue,
supervisor=sdk,
evidence=_Evidence(),
owner="outage",
model_failures=backoff.attempt,
)
for job_id in drain.claim(limit=10)
)
)
for second in range(800):
clock.now = float(second)
asyncio.run(tick())
jobs = _jobs(path)
assert [state for _, state, *_ in jobs] == ["completed"] * 200
assert sdk.written == [f"e-{index:03d}" for index in range(200)]
outage = [at for at, status in calls if status == 401]
# One request at 0 s, then one probe per pause: 5, 15, 35, 75, 155 s.
assert outage == [0.0, 5.0, 15.0, 35.0, 75.0, 155.0]
# The first job was tried seven times; each try had its own SDK id, so
# the recovered provider was really called, not a replayed failure.
assert jobs[0][2:4] == (6, 0)
assert sdk.request_ids[:3] == ["idem-000", "idem-000#r1", "idem-000#r2"]
assert all(retries == 0 for *_, retries, _ in jobs)
def test_a_failure_of_the_job_itself_is_retried_three_times_then_skipped(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
first = queue.submit(_ordered_request(0))
second = queue.submit(_ordered_request(1))
sdk = _FakeSdk()
drain = DurableWriteDrain(queue, execute=_never_execute, owner="skip")
now = datetime.now(UTC)
def run(at: datetime) -> tuple[str, ...]:
claimed = drain.claim(limit=1, now=at)
for job_id in claimed:
asyncio.run(
execute_job(
job_id,
queue=queue,
supervisor=sdk,
evidence=_Evidence(),
owner="skip",
model_failures=ModelBackoff().attempt,
)
)
return claimed
sdk.fail_with = "model_content_json_parse_failed"
assert run(now) == (first.job_id,)
# Each retry waits longer: 30 s, 60 s, 120 s. Until then the claim skips
# the job, and the job after it in the conversation waits behind it.
assert run(now + timedelta(seconds=20)) == ()
assert run(now + timedelta(seconds=40)) == (first.job_id,)
assert run(now + timedelta(seconds=150)) == (first.job_id,)
assert run(now + timedelta(seconds=400)) == (first.job_id,)
assert _jobs(path)[0][1:] == ("failed", 3, 3, "model_content_json_parse_failed")
assert queue.get("idem-000").error_code == "model_content_json_parse_failed"
sdk.fail_with = None
assert run(now + timedelta(seconds=401)) == (second.job_id,)
assert _jobs(path)[1][1] == "completed"
assert sdk.request_ids == [
"idem-000",
"idem-000#r1",
"idem-000#r2",
"idem-000#r3",
"idem-001",
]
def test_a_write_interrupted_mid_call_is_rerun_under_a_new_request_id(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_ordered_request(0))
drain = DurableWriteDrain(queue, execute=_never_execute, owner="crashed")
assert drain.claim(limit=1) == (record.job_id,)
assert queue.mark_invoked(record.job_id, owner="crashed")
# The process stops; on restart the lease has lapsed.
with sqlite3.connect(path) as database:
database.execute(
"UPDATE memory_write_jobs SET lease_expires_at=? WHERE job_id=?",
("2000-01-01T00:00:00+00:00", record.job_id),
)
assert queue.reclaim_expired() == 1
assert queue.get("idem-000").state == "unknown"
# Still running on its thread in this process: left alone.
assert queue.recover_unknown(in_flight=lambda request_id: request_id == "idem-000") == 0
assert queue.recover_unknown() == 1
assert _jobs(path)[0][1:] == ("pending", 1, 1, "durable_write_interrupted")
assert queue.recover_unknown() == 0
sdk = _FakeSdk()
later = datetime.now(UTC) + timedelta(seconds=31)
rerun = DurableWriteDrain(queue, execute=_never_execute, owner="restarted")
assert rerun.claim(limit=1, now=later) == (record.job_id,)
asyncio.run(
execute_job(
record.job_id, queue=queue, supervisor=sdk, evidence=_Evidence(), owner="restarted"
)
)
assert sdk.request_ids == ["idem-000#r1"]
assert queue.get("idem-000").state == "completed"
def test_a_write_that_did_land_is_completed_from_its_receipt_not_rerun(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_ordered_request(0))
with sqlite3.connect(path) as database:
database.execute(
"UPDATE memory_write_jobs SET state='unknown',hy_invoked=1,"
"last_error='durable_work_ceiling_exceeded' WHERE job_id=?",
(record.job_id,),
)
sdk = _FakeSdk()
sdk.outcomes["idem-000"] = "succeeded"
assert queue.recover_unknown() == 1
drain = DurableWriteDrain(queue, execute=_never_execute, owner="landed")
assert drain.claim(limit=1, now=datetime.now(UTC) + timedelta(seconds=31)) == (record.job_id,)
asyncio.run(
execute_job(
record.job_id, queue=queue, supervisor=sdk, evidence=_Evidence(), owner="landed"
)
)
assert sdk.request_ids == []
completed = queue.get("idem-000")
assert completed.state == "completed"
assert completed.sdk_memory_ids == ("m-idem-000",)
def test_unknown_jobs_from_before_the_upgrade_are_handled_once(tmp_path) -> None:
from one_memory.memory_tasks import NON_RETRYABLE_POST_WRITE_CODES
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
plain = queue.submit(_ordered_request(0, source="a"))
verified = queue.submit(_ordered_request(1, source="b"))
looping = queue.submit(_ordered_request(2, source="c"))
with sqlite3.connect(path) as database:
for job, code, retries in (
(plain, "hy_sdk_write_failed", 0),
(verified, NON_RETRYABLE_POST_WRITE_CODES[0], 0),
(looping, "durable_write_interrupted", 3),
):
database.execute(
"UPDATE memory_write_jobs SET state='unknown',hy_invoked=1,attempts=1,"
"last_error=?,retries=? WHERE job_id=?",
(code, retries, job.job_id),
)
assert queue.recover_unknown() == 3
assert queue.recover_unknown() == 0
# Reopening the queue (a restart) changes nothing either.
assert DurableWriteQueue(path).recover_unknown() == 0
states = {key: (state, attempt) for key, state, attempt, _, _ in _jobs(path)}
assert states == {
"idem-000": ("pending", 1),
# The write landed against an authority Core could not verify.
"idem-001": ("failed", 0),
# A crash loop ends instead of spinning.
"idem-002": ("failed", 0),
}
def test_concurrent_writes_are_all_charged_with_a_provider_failure() -> None:
"""The breaker cannot tell whose request failed, so every write in flight waits.
A write the provider did fail must never spend its retries on the outage;
a write that failed beside it for its own reason waits for the provider
once, and is retried with the rest.
"""
backoff = ModelBackoff(clock=_Clock())
with backoff.attempt() as first, backoff.attempt() as second:
backoff.record_failure("model_server_error", "p:1", "chat")
with backoff.attempt() as later:
pass
assert (first.failure, second.failure, later.failure) == (
"model_server_error",
"model_server_error",
None,
)
def test_the_manual_retry_endpoint_is_gone(tmp_path) -> None:
async def scenario() -> int:
app = create_app(
Settings(
data_dir=tmp_path / "data",
source_home=tmp_path,
api_token=SecretStr("local-token"),
),
secret_store=InMemorySecretStore(),
)
async with (
app.router.lifespan_context(app),
AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
headers={"Authorization": "Bearer local-token"},
) as client,
):
response = await client.post(
"/api/v1/memory-processing/retry-unknown", json={"confirm": True}
)
return response.status_code
assert asyncio.run(scenario()) in {404, 405}
@@ -334,7 +334,9 @@ def test_another_users_memory_id_answers_404_for_read_update_and_delete(
assert (await client.get(path)).status_code == 404
assert (await client.put(path, json={"content": "stolen"})).status_code == 404
assert (await client.delete(path)).status_code == 204
refused = await client.delete(path)
assert refused.status_code == 404
assert refused.json()["detail"] == "memory_not_found"
assert sdk.memories[wife_memory]["content"] == "wife drinks coffee"
own = await client.get(path, params={"user_id": "wife"})
@@ -343,6 +345,36 @@ def test_another_users_memory_id_answers_404_for_read_update_and_delete(
assert updated.status_code == 200
assert (await client.delete(path, params={"user_id": "wife"})).status_code == 204
assert wife_memory not in sdk.memories
# A second delete (a double click) finds nothing: 404, already gone.
again = await client.delete(path, params={"user_id": "wife"})
assert again.status_code == 404
assert again.json()["detail"] == "memory_not_found"
run_with_sdk(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
def test_a_delete_that_loses_the_race_answers_404_like_a_missing_id(tmp_path, monkeypatch) -> None:
"""Read as the caller's own, then gone before the SDK deletes it."""
async def scenario(app, sdk, client):
await _write_both_users(app, sdk, client)
[mine] = [
memory_id
for memory_id, row in sdk.memories.items()
if row["content"] == "local drinks tea"
]
original = sdk.delete
def raced(memory_id, kind=None, variant=None):
sdk.memories.pop(memory_id, None)
return original(memory_id, kind=kind, variant=variant)
sdk.delete = raced # type: ignore[method-assign]
response = await client.delete(f"/api/v2/memories/{mine}")
missing = await client.delete("/api/v2/memories/no-such-memory")
assert _observed(response) == _observed(missing)
assert response.status_code == 404
run_with_sdk(tmp_path, monkeypatch, scenario)
@@ -426,6 +458,52 @@ async def _true() -> bool:
return True
@pytest.mark.fresh_sdk
@pytest.mark.parametrize("user", [None, "wife"])
def test_every_read_echoes_the_user_it_was_read_for(tmp_path, monkeypatch, user) -> None:
"""Each read names the user it answered for: the one asked for, else `local`."""
async def scenario(app, sdk, client):
await _write_both_users(app, sdk, client)
monkeypatch.setattr(
"one_memory.api._recall_model_ready", lambda _app: _true(), raising=True
)
app.state.recall_target = _sdk_recall_target(app, "local")
expected = user or "local"
params = {} if user is None else {"user_id": user}
body = {} if user is None else {"user_id": user}
reads = [
await client.get("/api/v2/memory-library", params=params),
await client.get("/api/v2/memory-library", params={**params, "query": "route"}),
await client.get("/api/v2/memory-library/layers", params=params),
await client.get("/api/v2/user-profile", params=params),
await client.get("/api/v2/home-summary", params=params),
await client.get("/api/v2/memories/graph", params=params),
await client.get("/api/v2/raw", params=params),
await client.post(
"/api/v2/context/search", json={"query": "drinks", "kinds": ["memory"], **body}
),
await client.post("/api/v1/recall", json={"query": "drinks", **body}),
await client.post("/api/v2/memories/ask", json={"question": "what drink", **body}),
]
for response in reads:
assert response.status_code == 200, (response.request.url, response.text)
assert response.json()["user_id"] == expected, response.request.url
async with client.stream(
"POST", "/api/v2/memories/ask/stream", json={"question": "what drink", **body}
) as streamed:
events = [
json.loads(line[5:])
async for line in streamed.aiter_lines()
if line.startswith("data:")
]
assert events[0]["type"] == "recall"
assert events[0]["user_id"] == expected
run_with_sdk(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
@pytest.mark.parametrize(
("method", "path", "kwargs"),
@@ -569,6 +647,26 @@ def test_a_hermes_root_scan_never_reads_its_profiles(tmp_path) -> None:
assert _scan_texts(wife, tmp_path) == ["I am the wife", "hello", "wife's transcript"]
def test_a_profiles_directory_is_skipped_at_any_depth(tmp_path) -> None:
"""A root above the Hermes home still never reads `.hermes/profiles/**`."""
home = tmp_path / "home"
hermes, _wife = _hermes_homes(home)
(hermes / "sessions").mkdir()
(hermes / "sessions" / "own.jsonl").write_text(
json.dumps({"type": "message", "role": "user", "content": "own transcript"}) + "\n"
)
nested = hermes / "sessions" / "profiles" / "kid" / "sessions"
nested.mkdir(parents=True)
(nested / "k.jsonl").write_text(
json.dumps({"type": "message", "role": "user", "content": "kid's transcript"}) + "\n"
)
above = _scan_texts(home, tmp_path)
assert "I am the wife" not in above and "kid's transcript" not in above
assert set(above) == {"I am the husband", "hi", "own transcript"}
assert "kid's transcript" not in _scan_texts(hermes, tmp_path)
def _jobs(database: Path) -> list[sqlite3.Row]:
with sqlite3.connect(database) as db:
db.row_factory = sqlite3.Row
@@ -719,6 +817,87 @@ def test_a_migrated_home_with_the_old_grant_triggers_can_connect_and_revoke(tmp_
asyncio.run(scenario())
async def _connect(client, user_id: str, root: Path | None = None, **extra):
body: dict[str, object] = {"source_ids": ["hermes"], "request_id": str(uuid4()), **extra}
if user_id != "local":
body["user_id"] = user_id
if root is not None:
body["root"] = str(root)
return await client.post("/api/v1/sources/connect", json=body)
@pytest.mark.fresh_sdk
def test_a_hermes_root_that_is_the_profiles_container_is_refused(tmp_path) -> None:
async def scenario():
home = tmp_path / "home"
hermes, _wife = _hermes_homes(home)
app = application(tmp_path, home=home)
async with app.router.lifespan_context(app), client_for(app) as client:
for user in ("wife", "local"):
refused = await _connect(client, user, hermes / "profiles")
assert refused.status_code == 422, refused.text
assert refused.json()["detail"] == "hermes_root_is_profiles_container"
assert app.state.source_grant_store.grants() == ()
asyncio.run(scenario())
@pytest.mark.fresh_sdk
def test_two_users_on_one_hermes_root_or_nested_roots_are_refused(tmp_path) -> None:
async def scenario():
home = tmp_path / "home"
hermes, wife = _hermes_homes(home)
(wife / "sessions").mkdir()
link = home / "hermes-link"
link.symlink_to(hermes, target_is_directory=True)
app = application(tmp_path, home=home)
async with app.router.lifespan_context(app), client_for(app) as client:
location = (await client.post("/api/v1/sources/hermes/location")).json()
local = await _connect(client, "local", source_roots={"hermes": location["root_id"]})
assert local.status_code == 200, local.text
# Same root, the same root through a symlink, a root inside it
# that is not `profiles/<name>`, and a root above it.
for root in (hermes, link, wife / "sessions", home):
refused = await _connect(client, "kid", root)
assert refused.status_code == 409, (root, refused.text)
assert refused.json()["detail"] == "hermes_root_overlaps_other_user"
# The standard layout: `local` on ~/.hermes, wife on its profile.
accepted = await _connect(client, "wife", wife)
assert accepted.status_code == 200, accepted.text
# Another user on wife's own root is refused too.
twin = await _connect(client, "wife2", wife)
assert twin.status_code == 409
# Once wife's grant is revoked her root is free again.
await client.post("/api/v1/sources/hermes/revoke", params={"user_id": "wife"})
assert (await _connect(client, "wife2", wife)).status_code == 200
asyncio.run(scenario())
@pytest.mark.fresh_sdk
def test_local_cannot_take_another_users_hermes_root(tmp_path) -> None:
"""The check runs for `local`'s own connect too, not only for other users."""
async def scenario():
home = tmp_path / "home"
hermes, _wife = _hermes_homes(home)
other = home / "elsewhere" / ".hermes"
_hermes_state_db(other, [(1, "s", "user", "elsewhere")])
app = application(tmp_path, home=home)
async with app.router.lifespan_context(app), client_for(app) as client:
# wife took ~/.hermes itself, which is `local`'s default root.
taken = await _connect(client, "wife", hermes)
assert taken.status_code == 200, taken.text
location = (await client.post("/api/v1/sources/hermes/location")).json()
refused = await _connect(client, "local", source_roots={"hermes": location["root_id"]})
assert refused.status_code == 409, refused.text
assert refused.json()["detail"] == "hermes_root_overlaps_other_user"
# `local` on a root of its own that does not overlap is fine.
assert (await _connect(client, "local", other)).status_code == 200
asyncio.run(scenario())
@pytest.mark.fresh_sdk
def test_only_hermes_takes_another_user_or_a_custom_root(tmp_path) -> None:
async def scenario():
@@ -852,7 +1031,7 @@ def test_another_users_id_answers_exactly_like_an_id_that_does_not_exist(
other = await client.request(method, theirs, **kwargs)
absent = await client.request(method, missing, **kwargs)
assert _observed(other) == _observed(absent), (method, theirs)
assert other.status_code == (204 if method == "DELETE" else 404)
assert other.status_code == 404
assert sdk.memories == before
run_with_sdk(tmp_path, monkeypatch, scenario)
@@ -259,3 +259,52 @@ def test_recall_route_falls_back_and_keeps_the_raw_grant(tmp_path, monkeypatch):
# A host granted recall alone may not read original conversations.
assert recall_only["mode"] == "raw" and recall_only["hits"] == []
assert len(searched) == 1
def test_the_total_recall_budget_stays_under_the_brokers_wait():
from one_memory.recall_fallback import recall_total_timeout_seconds
assert recall_total_timeout_seconds(800) == pytest.approx(3.8)
# The broker gives Core 15 seconds; Core answers before it gives up.
assert recall_total_timeout_seconds(60_000) == 12.0
@pytest.mark.fresh_sdk
def test_a_recall_that_cannot_finish_in_time_is_a_clear_timeout(tmp_path, monkeypatch):
"""A recall stuck behind the SDK answers 504 `recall_timeout`, not nothing (#231)."""
from one_memory import recall_fallback
async def stuck(*_args, **_kwargs):
await asyncio.Event().wait()
monkeypatch.setattr(recall_fallback, "recall_with_fallback", stuck)
monkeypatch.setattr(recall_fallback, "recall_total_timeout_seconds", lambda _ms: 0.2)
async def scenario():
app = create_app(
Settings(
data_dir=tmp_path / "data",
source_home=tmp_path,
api_token=SecretStr("recall-test"),
imports_enabled=False,
automatic_memory_processing=False,
live_sync_enabled=False,
),
secret_store=InMemorySecretStore(),
)
async with (
app.router.lifespan_context(app),
AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
headers={"Authorization": "Bearer recall-test"},
) as client,
):
return await asyncio.wait_for(
client.post("/api/v1/recall", json={"query": "数据库"}), 5
)
response = asyncio.run(scenario())
assert response.status_code == 504
assert response.json()["detail"] == "recall_timeout"
@@ -1124,13 +1124,13 @@ def test_pre_upgrade_unknown_writes_are_settled_at_cutover_not_retried(store):
surviving = DurableWriteQueue(store.database_path, sdk_authority=True)
assert surviving.unknown_jobs() == ()
# Never requeued: they may already have written.
assert surviving.retry_unknown_confirmed(unknown[0]) == 0
assert surviving.recover_unknown() == 0
assert surviving.readmit_parked() == 0
assert PENDING not in {state for state, _, _ in rows.values()}
def test_retrying_unknown_writes_works_after_the_legacy_tables_are_dropped(store):
"""The retry button raised `no such table: memory_tasks` after an upgrade."""
def test_recovering_unknown_writes_works_after_the_legacy_tables_are_dropped(store):
"""The old retry button raised `no such table: memory_tasks` after an upgrade."""
from one_memory.memory_server.durable_queue import PENDING, UNKNOWN
from one_memory.target_schema import drop_legacy_tables
@@ -1162,14 +1162,10 @@ def test_retrying_unknown_writes_works_after_the_legacy_tables_are_dropped(store
surviving = DurableWriteQueue(store.database_path, sdk_authority=sdk_authority)
make_unknown("model_content_outcome_unknown", 1)
assert surviving.resolve_unknown_from_linked_output(identifier) is False
assert surviving.retry_unknown_confirmed(identifier) == 1
assert surviving.recover_unknown() == 1
assert surviving.get("after-upgrade").state == PENDING
make_unknown("hy_memory_timeout", 1)
since = datetime.now(UTC)
assert surviving.retry_timed_out_unknown_after_reconcile(older_than=since) == 1
make_unknown("model_content_outcome_unknown", 1)
assert surviving.retry_first_unknown_after_reconcile(older_than=since) == 1
with main_database(store.database_path).write() as database, database:
database.execute("UPDATE memory_write_jobs SET retries=0 WHERE job_id=?", (identifier,))
if sdk_authority:
# The SDK-authority queue never brings a dropped table back.
assert legacy_tables_present(store.database_path) == ()
@@ -91,7 +91,7 @@ def test_bundle_contains_management_sdk_config_and_verified_manifest(tmp_path):
assert manifest["bundle_id"] == "migration-1"
assert manifest["sdk_namespace"] == "governed-v2"
assert manifest["runtime_build"] == "build-13"
assert manifest["snapshot_sdk_version"] == "2.0.11"
assert manifest["snapshot_sdk_version"] == "2.0.12"
paths = {item["path"]: item for item in manifest["files"]}
assert {
"management/one-memory.sqlite3",
+4 -4
View File
@@ -991,7 +991,7 @@ wheels = [
[[package]]
name = "hy-memory"
version = "2.0.11"
version = "2.0.12"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "chromadb" },
@@ -1008,9 +1008,9 @@ dependencies = [
{ name = "scikit-learn" },
{ name = "tiktoken" },
]
sdist = { url = "https://files.pythonhosted.org/packages/01/93/aec7d243259f6a315f38e8ebc21a9eea153a48fc13648fda149be7672788/hy_memory-2.0.11.tar.gz", hash = "sha256:96d6bd606d61052f127cd274aa6f8de8e7151d6250f1e89e4ef5b90e18ec7cb0", size = 1245738, upload-time = "2026-11-01T14:15:02.474Z" }
sdist = { url = "https://files.pythonhosted.org/packages/44/19/bcd0de44ae2011ee1ad70579996aad72e18f0ca292df22d1545db05db850/hy_memory-2.0.12.tar.gz", hash = "sha256:c1231596b4cf12595775541f81680d4819461cb69e4aa4dc449f876d4c3b6742", size = 1246969, upload-time = "2026-11-02T06:30:37.448Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/1a/85/1f5e6a13507a0cfcdce26f67c590df00d327a0f7d5c5b5483c3bd0fa2f85/hy_memory-2.0.11-py3-none-any.whl", hash = "sha256:c0173444a83c1197a49acaccb123f8286cea205b4838c1e83d106974f5bad259", size = 1315871, upload-time = "2026-11-01T14:14:58.764Z" },
{ url = "https://files.pythonhosted.org/packages/75/9e/b64c7643c1bad1ddca4623a39bb62423bed0c22e7fd0d86be1ff73b2a17a/hy_memory-2.0.12-py3-none-any.whl", hash = "sha256:7a2332288197fafb89d5037eac7fef97df4c80794ead18159d21da84cde431df", size = 1316922, upload-time = "2026-11-02T06:30:35.105Z" },
]
[[package]]
@@ -1837,7 +1837,7 @@ requires-dist = [
{ name = "fastapi", specifier = ">=0.141.1" },
{ name = "filelock", specifier = ">=3.32.4" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "hy-memory", specifier = "==2.0.11" },
{ name = "hy-memory", specifier = "==2.0.12" },
{ name = "keyring", specifier = ">=25.7.0" },
{ name = "numpy", specifier = "==2.5.2" },
{ name = "onnxruntime", specifier = "==1.23.2" },
@@ -9,7 +9,7 @@ import type {
Tray,
} from "electron";
import { describe, expect, it, vi } from "vitest";
import { installBackgroundTray } from "./background-tray";
import { installBackgroundTray, type RuntimeHosting } from "./background-tray";
import type { LoginPreference } from "./login-preference";
async function flush() {
@@ -27,6 +27,7 @@ function fixture(
status: "disabled",
};
let changed: (() => void) | undefined;
let hosting: RuntimeHosting | Error = "background-service";
const languageSnapshot: LanguageSnapshot = {
preference: "zh-Hans",
locale: "zh-Hans",
@@ -50,6 +51,10 @@ function fixture(
};
return state;
});
const readRuntimeHosting = vi.fn(async () => {
if (hosting instanceof Error) throw hosting;
return hosting;
});
const open = vi.fn();
const reportError = vi.fn();
const install = () =>
@@ -77,6 +82,7 @@ function fixture(
reportError,
readLoginPreference,
setLoginPreference,
readRuntimeHosting,
...(trial
? { allowLoginStartup: false, productName: "One Memory Trial" }
: {}),
@@ -91,6 +97,7 @@ function fixture(
menus,
readLoginPreference,
setLoginPreference,
readRuntimeHosting,
open,
reportError,
install,
@@ -103,6 +110,9 @@ function fixture(
setState(value: LoginPreference) {
state = value;
},
setHosting(value: RuntimeHosting | Error) {
hosting = value;
},
};
}
@@ -179,6 +189,50 @@ describe("native macOS background tray", () => {
expect(f.tray.eventNames()).toEqual([]);
});
it("says quitting stops the memory service when the App hosts it", async () => {
const f = fixture();
f.setHosting("app-hosted");
f.install();
await flush();
expect(f.menu()[3]?.label).toBe("退出 MemHub(同时停止记忆服务)");
f.changeLanguage("es");
await flush();
expect(f.menu()[3]?.label).toBe(
"Salir de MemHub (también detiene el servicio de memoria)",
);
});
it("re-reads who hosts the Runtime every time the menu opens", async () => {
const f = fixture();
f.install();
await flush();
expect(f.menu()[3]?.label).toContain("后台服务继续运行");
// The user turned the background service off in Settings.
f.setHosting("app-hosted");
f.tray.emit("mouse-down");
await flush();
expect(f.menu()[3]?.label).toContain("同时停止记忆服务");
f.setHosting("background-service");
f.tray.emit("right-click");
await flush();
expect(f.menu()[3]?.label).toContain("后台服务继续运行");
expect(f.readRuntimeHosting).toHaveBeenCalledTimes(3);
});
it("never claims quit stops the service when the state is unknown", async () => {
const f = fixture();
f.setHosting(new Error("launchctl timed out"));
f.install();
await flush();
expect(f.menu()[3]?.label).toContain("后台服务继续运行");
f.setHosting("unknown");
f.tray.emit("mouse-down");
await flush();
expect(f.menu()[3]?.label).toContain("后台服务继续运行");
});
it("blocks login changes for trial, development and non-macOS builds", async () => {
for (const f of [
fixture(true, "darwin", true),
+16 -1
View File
@@ -10,6 +10,8 @@ import type {
import type { LoginPreference } from "./login-preference";
import type { NativeLanguageSource } from "./native-language";
export type RuntimeHosting = "background-service" | "app-hosted" | "unknown";
export const LOGIN_SERVICE_NAME = "com.tencent.onememory.runtime.plist";
interface TrayOptions {
@@ -24,6 +26,9 @@ interface TrayOptions {
reportError(message: string): void;
readLoginPreference(): Promise<LoginPreference>;
setLoginPreference(enabled: boolean): Promise<LoginPreference>;
/** Who keeps the Runtime alive, re-read on every menu refresh: the
* background service (quit leaves it running) or this App (quit stops it). */
readRuntimeHosting?(): Promise<RuntimeHosting>;
allowLoginStartup?: boolean;
productName?: string;
language?: NativeLanguageSource;
@@ -67,6 +72,13 @@ export function installBackgroundTray(options: TrayOptions): () => void {
} else {
label += t("nativeDevelopmentUnavailable");
}
let hosting: RuntimeHosting = "unknown";
try {
hosting = (await options.readRuntimeHosting?.()) ?? "unknown";
} catch {
hosting = "unknown";
}
if (disposed || current !== generation) return;
tray.setContextMenu(
options.buildMenu([
{ label: t("nativeOpen"), click: options.open },
@@ -92,7 +104,10 @@ export function installBackgroundTray(options: TrayOptions): () => void {
},
{ type: "separator" },
{
label: t("nativeQuit"),
// Unknown keeps the plain label: quit then stops nothing.
label: t(
hosting === "app-hosted" ? "nativeQuitStopsService" : "nativeQuit",
),
click: () => {
if (!disposed) app.quit();
},
@@ -5,6 +5,7 @@ import {
claimRuntimeServiceForApp,
claimRuntimeServiceForCli,
enableRuntimeService,
type RuntimeServiceState,
runtimeServiceEnableMarkerPath,
} from "@one-memory/runtime-control";
import { expect, it } from "vitest";
@@ -98,3 +99,25 @@ it("rejects unsupported and unconfirmed writes", async () => {
),
).rejects.toThrow("login_preference_unsupported");
});
it("reads a launchctl-disabled service as off and not switchable, asked fresh", async () => {
const dataDir = await appOwned();
let state: RuntimeServiceState = {
state: "disabled",
reason: "service-disabled-by-launchctl",
jobPid: null,
};
const serviceState = async () => state;
await expect(
readLoginPreference(dataDir, "darwin", true, serviceState),
).resolves.toEqual({
supported: false,
enabled: false,
status: "disabled-by-system",
});
state = { state: "enabled", reason: "service-enabled", jobPid: 4321 };
await expect(
readLoginPreference(dataDir, "darwin", true, serviceState),
).resolves.toEqual({ supported: true, enabled: true, status: "enabled" });
});
+15 -1
View File
@@ -1,6 +1,7 @@
import {
disableRuntimeService,
enableRuntimeService,
type RuntimeServiceState,
readRuntimeServiceOwner,
runtimeServiceEnabled,
setAppRuntimeStartAtLogin,
@@ -12,10 +13,16 @@ export interface LoginPreference {
status: string;
}
/** "Launch at login" is the background service switch: it writes the enable
* marker the LaunchAgent's `KeepAlive` waits on. On it means the Runtime
* outlives the App and quitting leaves it running; off means quitting stops
* it. `serviceState`, when given, is asked fresh so a `launchctl disable`
* override (which the switch cannot undo) reads as off, not as on. */
export async function readLoginPreference(
dataDir: string,
platform: NodeJS.Platform,
supported: boolean,
serviceState?: () => Promise<RuntimeServiceState>,
): Promise<LoginPreference> {
if (platform !== "darwin" || !supported) {
return { supported: false, enabled: false, status: "unsupported" };
@@ -28,6 +35,12 @@ export async function readLoginPreference(
if (owner?.owner !== "app") {
return { supported: false, enabled: false, status: "not-installed" };
}
if (
serviceState &&
(await serviceState()).reason === "service-disabled-by-launchctl"
) {
return { supported: false, enabled: false, status: "disabled-by-system" };
}
return {
supported: true,
enabled: owner.start_at_login && (await runtimeServiceEnabled(dataDir)),
@@ -46,6 +59,7 @@ export async function setLoginPreference(
platform: NodeJS.Platform,
input: unknown,
supported: boolean,
serviceState?: () => Promise<RuntimeServiceState>,
): Promise<LoginPreference> {
if (
!input ||
@@ -63,5 +77,5 @@ export async function setLoginPreference(
await setAppRuntimeStartAtLogin(dataDir, input.openAtLogin);
if (input.openAtLogin) await enableRuntimeService(dataDir);
else await disableRuntimeService(dataDir);
return readLoginPreference(dataDir, platform, supported);
return readLoginPreference(dataDir, platform, supported, serviceState);
}
+140 -21
View File
@@ -23,7 +23,9 @@ import {
type HostPermissionClient,
installPackagedRuntime,
installUserService,
queryRuntimeServiceState,
RecallBrokerSupervisor,
type RuntimeServiceState,
RuntimeSupervisor,
readInstalledRuntimeSource,
readPublishedCorePort,
@@ -110,6 +112,7 @@ import { showMigrationFailureDialog } from "./migration-failure-dialog";
import { isAllowedNavigation } from "./navigation-policy";
import { installDenyAllPermissionPolicy } from "./permissions";
import { isHostPresent, recallHosts } from "./recall-hosts";
import { reconcileRunningRuntime } from "./runtime-adoption";
import { shutdownRuntime } from "./runtime-shutdown";
import { createSourceChangeHandler } from "./source-change-handler";
import { SourceDirectoryPicker } from "./source-directory-picker";
@@ -140,7 +143,12 @@ let runtimeSupervisor: RuntimeSupervisor | null = null;
// Where the Runtime keeps its receipt. Stopping it on a full quit needs this,
// and `stop` runs outside `initialize`, where the directory is chosen.
let runtimeDataDir: string | null = null;
let preserveManagedRuntimeOnQuit = false;
// Whether quitting stops the Runtime is asked of the service manager at quit
// time (`currentRuntimeServiceState`), never remembered from startup. The one
// startup fact kept is a Runtime whose build could not be read: this App never
// stops one it could not compare with its own.
let preserveRuntimeOnQuitReason: string | null = null;
let runtimeServiceManagedByApp = false;
let disposeTray: (() => void) | null = null;
let disposeApplicationMenu: (() => void) | null = null;
let languagePreferences: LanguagePreferences | null = null;
@@ -247,6 +255,25 @@ if (trialProfile) {
app.setPath("userData", hourlyUserDataDir);
}
/** Ask the service manager, now, whether the background service for this data
* dir is on. Never cached: quit and the tray each re-read it, so a switch made
* in Settings, by `memhub service`, or by `launchctl` counts immediately. */
function currentRuntimeServiceState(): Promise<RuntimeServiceState> {
if (!runtimeDataDir) {
return Promise.resolve({
state: "unknown",
reason: "service-state-query-failed",
jobPid: null,
});
}
return queryRuntimeServiceState({
dataDir: runtimeDataDir,
home: app.getPath("home"),
platform: process.platform,
uid: process.getuid?.(),
});
}
function prepareProtocol(): void {
protocol.registerSchemesAsPrivileged([
{
@@ -498,6 +525,7 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
!preview &&
(hostIntegrationEnabled || desktopTestIsolation !== null) &&
installedApplication;
runtimeServiceManagedByApp = appManagedRuntime;
const runningPackagedHost =
app.isPackaged && !isolatedHourly
? await resolveRunningPackagedHostPaths({
@@ -594,9 +622,11 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
throw new Error("host_permission_store_unavailable");
}
if (isQuitting()) return;
const hosts = (await recallHosts(app.getPath("home"))).filter(
(host) => host.hostId !== "openclaw",
);
const scanHosts = async () =>
(await recallHosts(app.getPath("home"))).filter(
(host) => host.hostId !== "openclaw",
);
const hosts = await scanHosts();
readDiagnosticSummary = async () => {
const core = coreSupervisor;
if (!core) return {};
@@ -851,14 +881,25 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
);
ipcMain.handle("one-memory:get-login-preference", (event) => {
assertAllowedRenderer(event.senderFrame?.url);
return readLoginPreference(dataDir, process.platform, appManagedRuntime);
return readLoginPreference(
dataDir,
process.platform,
appManagedRuntime,
currentRuntimeServiceState,
);
});
ipcMain.handle("one-memory:set-login-preference", (event, input: unknown) => {
assertAllowedRenderer(event.senderFrame?.url);
return observeDesktopAction(
"login_preference",
() =>
setLoginPreference(dataDir, process.platform, input, appManagedRuntime),
setLoginPreference(
dataDir,
process.platform,
input,
appManagedRuntime,
currentRuntimeServiceState,
),
{ action: "set" },
);
});
@@ -876,10 +917,14 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
});
ipcMain.handle("one-memory:recall-hosts", async (event) => {
assertAllowedRenderer(event.senderFrame?.url);
// A Harness profile can appear at any time, and which ones exist is only
// knowable from the disk. In place, so `requireHost` sees it too. One
// directory read per list, next to an IPC round trip per host below.
hosts.splice(0, hosts.length, ...(await scanHosts()));
if (connectionClient?.host) {
const inspector = createConnectionClient({ dataDir });
const rows: unknown[] = [];
for (const host of hosts) {
for (const host of [...hosts]) {
const inspected = await inspector.host?.({
hostId: host.hostId,
operation: "inspect",
@@ -1169,7 +1214,6 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
const startAtLogin = owner?.owner === "app" ? owner.start_at_login : true;
const previousOwner = owner;
let runtimeChanged = false;
let transientRuntime = false;
// Install whoever owns the service. Owner decides who starts and stops
// it, not which build runs: skipping the install for a CLI owner (also
// the App's own guess when its receipt is missing) left an upgraded App
@@ -1239,12 +1283,68 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
const frozen = await readSdkMigrationBlock(dataDir);
if (frozen?.frozen !== undefined) throw new Error(frozen.code);
}
const alreadyRunning = runtimeChanged
? false
: await waitForRuntimeProcess({ dataDir, attempts: 4 }).then(
() => true,
() => false,
);
// One Runtime per data dir. A Runtime already serving it is adopted
// when it is this App's build or newer (silently; a newer one is never
// stopped or downgraded at startup); an older one is replaced (through
// launchd when it is the service's job), and one whose build cannot be
// read is adopted but never stopped.
const reconciled = await reconcileRunningRuntime({
app: {
...(packagedBuildNumber === undefined
? {}
: { buildNumber: packagedBuildNumber }),
...(packagedCommit === undefined ? {} : { commit: packagedCommit }),
},
installedChanged: runtimeChanged,
findRunning: () =>
waitForRuntimeProcess({ dataDir, attempts: 4 }).then(
(attachment) =>
attachment.servicePid === null
? null
: {
pid: attachment.servicePid,
build: attachment.runtimeBuild,
},
() => null,
),
queryServiceState: currentRuntimeServiceState,
stopRuntime: () => stopRuntimeProcess({ dataDir }),
log: ({ decision, service, running }) =>
diagnostics.log({
level: decision.action === "adopt-unverified" ? "warn" : "info",
component: "desktop.stable_runtime",
event: "stable_runtime.adoption",
outcome:
decision.action === "adopt-unverified" ? "degraded" : "succeeded",
details: {
action: decision.action,
reason:
"reason" in decision
? decision.reason
: (service?.reason ?? "no-runtime-running"),
status: service?.state ?? "not-queried",
version:
running === null
? "none"
: running.build === undefined
? "legacy"
: String(running.build.buildNumber ?? "unknown"),
},
}),
});
preserveRuntimeOnQuitReason = reconciled.preserveReason;
if (reconciled.decision.action === "adopt-unverified") {
void dialog
.showMessageBox({
type: "warning",
message: "MemHub",
detail: stableRuntimeFailureDetail(
new Error("runtime_build_unknown"),
),
})
.catch(() => undefined);
}
const alreadyRunning = reconciled.adopted;
if (owner?.owner !== "cli") {
const appBundle = path.resolve(path.dirname(process.execPath), "../..");
owner = await claimRuntimeServiceForApp({
@@ -1258,8 +1358,13 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
// A durable `launchctl disable` override is a state to report, not a
// failure to undo: `installUserService` never re-enables a service the
// user or the system disabled. An explicit App launch may still own a
// transient Runtime for this UI session. Shutdown stops that process.
const serviceMode = await installServiceWithDisabledFallback({
// transient Runtime for this UI session; quit stops it because the
// service is disabled, which it decides by asking launchd again then.
// For an enabled service, `installUserService` restarts the job with
// `kickstart -k` (or re-bootstraps a changed plist), so an older
// Runtime is replaced through launchd rather than killed into a
// respawn of the old binary.
await installServiceWithDisabledFallback({
install: async () => {
await installUserService(layout, {
uid: process.getuid?.(),
@@ -1291,12 +1396,10 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
});
},
});
transientRuntime = serviceMode === "transient";
}
if (owner?.owner === "app" && !startAtLogin) {
await disableRuntimeService(dataDir);
}
preserveManagedRuntimeOnQuit = !transientRuntime;
} else if (standaloneRuntime) {
if (
!desktopBackgroundLaunch &&
@@ -1665,7 +1768,8 @@ if (desktopVersion !== null) {
? readLoginPreference(
runtimeDataDir,
process.platform,
preserveManagedRuntimeOnQuit,
runtimeServiceManagedByApp,
currentRuntimeServiceState,
)
: Promise.resolve({
supported: false,
@@ -1678,9 +1782,21 @@ if (desktopVersion !== null) {
runtimeDataDir,
process.platform,
{ openAtLogin: enabled, userConfirmed: true },
preserveManagedRuntimeOnQuit,
runtimeServiceManagedByApp,
currentRuntimeServiceState,
);
},
readRuntimeHosting: async () => {
if (!runtimeDataDir || !app.isPackaged) return "unknown";
// A Runtime whose build could not be read is never stopped by it.
if (preserveRuntimeOnQuitReason) return "unknown";
const service = await currentRuntimeServiceState();
return service.state === "enabled"
? "background-service"
: service.state === "disabled"
? "app-hosted"
: "unknown";
},
...(trialProfile
? { allowLoginStartup: false, productName: app.getName() }
: {}),
@@ -1712,7 +1828,10 @@ if (desktopVersion !== null) {
// stop and says so by taking the embedded path instead.
dataDir: runtimeDataDir ?? "",
standalone: app.isPackaged && runtimeDataDir !== null,
preserveManagedRuntime: preserveManagedRuntimeOnQuit,
// Asked now, not remembered from startup: the user may have switched
// the background service on or off since.
queryServiceState: currentRuntimeServiceState,
preserveReason: preserveRuntimeOnQuitReason,
stopSupervisor: async () => {
const results = await Promise.allSettled([
runtimeSupervisor?.stop() ?? coreSupervisor?.stop(),
@@ -0,0 +1,233 @@
import type { RuntimeServiceState } from "@one-memory/runtime-control";
import { describe, expect, it, vi } from "vitest";
import {
compareRuntimeBuild,
decideRuntimeAdoption,
type RunningRuntime,
reconcileRunningRuntime,
} from "./runtime-adoption";
const app = { buildNumber: 38, commit: "abcdef1234567" };
const enabled = (jobPid: number | null): RuntimeServiceState => ({
state: "enabled",
reason: "service-enabled",
jobPid,
});
const disabled: RuntimeServiceState = {
state: "disabled",
reason: "service-disabled-by-launchctl",
jobPid: null,
};
const unknown: RuntimeServiceState = {
state: "unknown",
reason: "service-state-timeout",
jobPid: null,
};
const running = (
build: RunningRuntime["build"],
pid = 4321,
): RunningRuntime => ({ pid, build });
function reconcile(options: {
running: RunningRuntime | null;
service: RuntimeServiceState;
installedChanged?: boolean;
}) {
const stopRuntime = vi.fn(async () => "stopped");
const queryServiceState = vi.fn(async () => options.service);
const result = reconcileRunningRuntime({
app,
installedChanged: options.installedChanged ?? false,
findRunning: async () => options.running,
queryServiceState,
stopRuntime,
});
return { result, stopRuntime, queryServiceState };
}
describe("comparing the running Runtime's build with the App's", () => {
it("compares build numbers, then commits both sides know", () => {
expect(compareRuntimeBuild(app, { buildNumber: 38, commit: null })).toBe(
"same",
);
expect(
compareRuntimeBuild(app, { buildNumber: 38, commit: "abcdef1" }),
).toBe("same");
expect(
compareRuntimeBuild(app, { buildNumber: 38, commit: "0123456" }),
).toBe("app-newer");
expect(compareRuntimeBuild(app, { buildNumber: 37, commit: null })).toBe(
"app-newer",
);
expect(compareRuntimeBuild(app, { buildNumber: 39, commit: null })).toBe(
"runtime-newer",
);
});
it("reads a receipt without build fields as a Runtime older than this App", () => {
expect(compareRuntimeBuild(app, undefined)).toBe("app-newer");
});
it("is unknown when either side cannot say", () => {
expect(compareRuntimeBuild(app, { buildNumber: null, commit: null })).toBe(
"unknown",
);
expect(compareRuntimeBuild({}, { buildNumber: 38, commit: null })).toBe(
"unknown",
);
});
});
describe("App start adopts or replaces a running Runtime", () => {
it("starts one when nothing runs, without asking the service manager", async () => {
const { result, stopRuntime, queryServiceState } = reconcile({
running: null,
service: enabled(null),
});
await expect(result).resolves.toMatchObject({
decision: { action: "start" },
adopted: false,
});
expect(queryServiceState).not.toHaveBeenCalled();
expect(stopRuntime).not.toHaveBeenCalled();
});
it("adopts the same build instead of starting a second Runtime", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 38, commit: null }),
service: disabled,
});
await expect(result).resolves.toEqual({
decision: { action: "adopt" },
adopted: true,
preserveReason: null,
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("adopts the service's own job when the service is enabled", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 38, commit: null }, 4321),
service: enabled(4321),
});
await expect(result).resolves.toMatchObject({
decision: { action: "adopt" },
adopted: true,
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("ends with one Runtime when a stray holds the lease beside an enabled service", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 38, commit: null }, 777),
service: enabled(null),
});
await expect(result).resolves.toMatchObject({
decision: { action: "stop-then-start", reason: "stray-runtime" },
adopted: false,
});
expect(stopRuntime).toHaveBeenCalledOnce();
});
it("does not treat a Runtime as a stray when the service state is unknown", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 38, commit: null }, 777),
service: unknown,
});
await expect(result).resolves.toMatchObject({
decision: { action: "adopt" },
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("replaces an older Runtime that is the service's job through the service manager", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 37, commit: null }, 4321),
service: enabled(4321),
});
await expect(result).resolves.toMatchObject({
decision: { action: "replace-via-service" },
adopted: false,
preserveReason: null,
});
// No plain kill: launchd would answer it with a respawn of the old job.
expect(stopRuntime).not.toHaveBeenCalled();
});
it("replaces a legacy service job whose receipt has no build through launchd", async () => {
const { result, stopRuntime } = reconcile({
running: running(undefined, 4321),
service: enabled(4321),
});
await expect(result).resolves.toMatchObject({
decision: { action: "replace-via-service" },
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("stops an older transient or CLI-started Runtime, then starts the right one", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 36, commit: null }, 999),
service: disabled,
});
await expect(result).resolves.toMatchObject({
decision: { action: "stop-then-start", reason: "app-newer" },
adopted: false,
});
expect(stopRuntime).toHaveBeenCalledOnce();
});
it("replaces a same-build Runtime when this launch moved `current`", () => {
expect(
decideRuntimeAdoption({
running: running({ buildNumber: 38, commit: null }, 4321),
app,
service: enabled(4321),
installedChanged: true,
}),
).toEqual({ action: "replace-via-service" });
});
it("adopts a newer Runtime silently and leaves quit to the service state", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 40, commit: null }, 999),
service: disabled,
});
// No special quit pass: with the service disabled, quit stops it like any
// adopted Runtime; with it enabled, quit preserves it.
await expect(result).resolves.toEqual({
decision: { action: "adopt-newer" },
adopted: true,
preserveReason: null,
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("never stops or downgrades a newer Runtime at startup, even beside an enabled service", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: 40, commit: null }, 999),
service: enabled(4321),
installedChanged: true,
});
await expect(result).resolves.toMatchObject({
decision: { action: "adopt-newer" },
adopted: true,
});
expect(stopRuntime).not.toHaveBeenCalled();
});
it("never kills a Runtime whose build cannot be read", async () => {
const { result, stopRuntime } = reconcile({
running: running({ buildNumber: null, commit: null }, 999),
service: disabled,
installedChanged: true,
});
await expect(result).resolves.toEqual({
decision: { action: "adopt-unverified" },
adopted: true,
preserveReason: "runtime-build-unknown",
});
expect(stopRuntime).not.toHaveBeenCalled();
});
});
+158
View File
@@ -0,0 +1,158 @@
import type {
RuntimeReceiptBuild,
RuntimeServiceState,
} from "@one-memory/runtime-control";
/** The build this App carries. */
export interface AppBuild {
buildNumber?: number;
commit?: string;
}
/** The Runtime found serving the data dir when the App starts. */
export interface RunningRuntime {
pid: number;
/** As published in its receipt; `undefined` for a receipt without the
* fields, which only a Runtime older than this App writes. */
build: RuntimeReceiptBuild | undefined;
}
export type RuntimeVersionRelation =
| "same"
| "app-newer"
| "runtime-newer"
| "unknown";
export type RuntimeAdoption =
/** Nothing is running: install or start this App's Runtime. */
| { action: "start" }
/** Same build: use it as it is. */
| { action: "adopt" }
/** Its build cannot be compared: use it, never stop it, tell the user. */
| { action: "adopt-unverified" }
/** Older, and it is the service manager's job: replace it through the
* service manager, never with a plain kill launchd would answer with a
* respawn of whatever it last loaded. */
| { action: "replace-via-service" }
/** Older and transient/CLI-started, or a stray beside an enabled service:
* stop it gracefully, then install or start the right one. */
| { action: "stop-then-start"; reason: "app-newer" | "stray-runtime" }
/** Newer than this App: use it as it is, silently. Never stopped or
* downgraded at startup and never joined by a second Runtime; quit treats
* it like any adopted Runtime. */
| { action: "adopt-newer" };
/** Compare by build number, then by commit when both sides know theirs. */
export function compareRuntimeBuild(
app: AppBuild,
runtime: RuntimeReceiptBuild | undefined,
): RuntimeVersionRelation {
if (app.buildNumber === undefined) return "unknown";
// Every Runtime that publishes its build carries the fields, even as null.
// A receipt without them predates that contract, and so this App.
if (runtime === undefined) return "app-newer";
if (runtime.buildNumber === null) return "unknown";
if (runtime.buildNumber < app.buildNumber) return "app-newer";
if (runtime.buildNumber > app.buildNumber) return "runtime-newer";
const appCommit = app.commit?.trim().toLowerCase();
if (
appCommit &&
runtime.commit &&
!appCommit.startsWith(runtime.commit) &&
!runtime.commit.startsWith(appCommit)
) {
// Same build, different commit: the App's bundle takes over, exactly as
// `installPackagedRuntime({ replaceSameBuild: true })` already does.
return "app-newer";
}
return "same";
}
export function decideRuntimeAdoption(input: {
running: RunningRuntime | null;
app: AppBuild;
service: RuntimeServiceState;
/** This launch just moved `current` to the App's bundled Runtime, so the
* process still running was started from the install it replaced. */
installedChanged?: boolean;
}): RuntimeAdoption {
const { running, service } = input;
if (running === null) return { action: "start" };
let relation = compareRuntimeBuild(input.app, running.build);
if (relation === "same" && input.installedChanged) relation = "app-newer";
if (relation === "runtime-newer") return { action: "adopt-newer" };
if (relation === "unknown") return { action: "adopt-unverified" };
const serviceJob = service.jobPid !== null && service.jobPid === running.pid;
if (relation === "app-newer") {
return serviceJob
? { action: "replace-via-service" }
: { action: "stop-then-start", reason: "app-newer" };
}
// One Runtime per data dir: with the service on, the one that stays is the
// service's own job, not a transient or CLI process holding its lease.
if (service.state === "enabled" && !serviceJob) {
return { action: "stop-then-start", reason: "stray-runtime" };
}
return { action: "adopt" };
}
export interface RuntimeReconcileDependencies {
app: AppBuild;
installedChanged?: boolean;
findRunning(): Promise<RunningRuntime | null>;
queryServiceState(): Promise<RuntimeServiceState>;
/** Graceful stop of the one verified Runtime: in-flight writes finish. */
stopRuntime(): Promise<unknown>;
log?(event: {
decision: RuntimeAdoption;
service: RuntimeServiceState | null;
running: RunningRuntime | null;
}): Promise<void> | void;
}
export interface RuntimeReconcileResult {
decision: RuntimeAdoption;
/** The running Runtime stays and the caller must not start another. */
adopted: boolean;
/** Quit must leave the adopted Runtime alone whatever the service state:
* this App could not prove it is its own build. */
preserveReason: string | null;
}
/** Decide what to do with a Runtime already serving the data dir at App start,
* and do the part that has to happen before the normal start path runs. */
export async function reconcileRunningRuntime(
dependencies: RuntimeReconcileDependencies,
): Promise<RuntimeReconcileResult> {
const running = await dependencies.findRunning();
const service =
running === null ? null : await dependencies.queryServiceState();
const decision = decideRuntimeAdoption({
running,
app: dependencies.app,
service: service ?? {
state: "unknown",
reason: "service-state-query-failed",
jobPid: null,
},
installedChanged: dependencies.installedChanged ?? false,
});
await dependencies.log?.({ decision, service, running });
switch (decision.action) {
case "start":
case "replace-via-service":
return { decision, adopted: false, preserveReason: null };
case "stop-then-start":
await dependencies.stopRuntime();
return { decision, adopted: false, preserveReason: null };
case "adopt":
case "adopt-newer":
return { decision, adopted: true, preserveReason: null };
case "adopt-unverified":
return {
decision,
adopted: true,
preserveReason: "runtime-build-unknown",
};
}
}
+174 -32
View File
@@ -1,96 +1,162 @@
import type {
RuntimeServiceState,
StopAllRuntimesResult,
} from "@one-memory/runtime-control";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { shutdownRuntime } from "./runtime-shutdown";
const { stopRuntimeProcess } = vi.hoisted(() => ({
stopRuntimeProcess: vi.fn<() => Promise<"stopped" | "not-running">>(
async () => "stopped",
),
const { stopAllRuntimeProcesses } = vi.hoisted(() => ({
stopAllRuntimeProcesses: vi.fn<
(options: { dataDir: string }) => Promise<StopAllRuntimesResult>
>(async () => ({ stopped: [4321], unverified: [] })),
}));
vi.mock("@one-memory/runtime-control", () => ({ stopRuntimeProcess }));
vi.mock("@one-memory/runtime-control", () => ({ stopAllRuntimeProcesses }));
const stopMock = vi.mocked(stopRuntimeProcess);
const stopMock = vi.mocked(stopAllRuntimeProcesses);
const enabled: RuntimeServiceState = {
state: "enabled",
reason: "service-enabled",
jobPid: 4321,
};
const disabled = (
reason: RuntimeServiceState["reason"] = "service-enable-marker-absent",
): RuntimeServiceState => ({ state: "disabled", reason, jobPid: null });
const unknown: RuntimeServiceState = {
state: "unknown",
reason: "service-state-timeout",
jobPid: null,
};
beforeEach(() => {
stopMock.mockClear();
stopMock.mockResolvedValue("stopped");
stopMock.mockReset();
stopMock.mockResolvedValue({ stopped: [4321], unverified: [] });
});
describe("BUG-12: a full quit stops the standalone Runtime process", () => {
it("detaches the UI without stopping a managed Runtime service", async () => {
describe("App quit with the background service enabled", () => {
it("never stops the Runtime", async () => {
const stopSupervisor = vi.fn(async () => {});
const events: string[] = [];
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
preserveManagedRuntime: true,
queryServiceState: async () => enabled,
stopSupervisor,
onDiagnostic: (event) => {
events.push(event.detail);
},
});
expect(result).toEqual({ runtime: "unchanged" });
expect(result).toEqual({
runtime: "preserved",
reason: "background-service-enabled",
});
expect(stopSupervisor).toHaveBeenCalledOnce();
expect(stopMock).not.toHaveBeenCalled();
expect(events).toEqual(["managed-runtime-preserved"]);
expect(events).toEqual(["runtime-preserved:background-service-enabled"]);
});
it("stops the Runtime process in a packaged build", async () => {
const stopSupervisor = vi.fn(async () => {});
it("does not stop a transient Runtime this App started before the service was enabled", async () => {
// Startup fell back to a transient child (service launchctl-disabled) and
// the user re-enabled the service later. Nothing startup saw is an input
// here: quit asks the service manager again and preserves the Runtime.
const queryServiceState = vi.fn(async () => enabled);
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
stopSupervisor,
queryServiceState,
stopSupervisor: async () => {},
});
expect(queryServiceState).toHaveBeenCalledOnce();
expect(result.runtime).toBe("preserved");
expect(stopMock).not.toHaveBeenCalled();
});
});
describe("App quit with the background service disabled", () => {
it("stops every Runtime of the data dir: the App's child and an external one", async () => {
stopMock.mockResolvedValue({ stopped: [1111, 2222], unverified: [] });
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => disabled(),
stopSupervisor: async () => {},
});
expect(result).toEqual({ runtime: "stopped" });
expect(stopMock).toHaveBeenCalledWith({ dataDir: "/tmp/runtime-data" });
expect(stopSupervisor).toHaveBeenCalledOnce();
});
it("does not touch a separate Runtime process in an embedded build", async () => {
const stopSupervisor = vi.fn(async () => {});
it.each([
"service-not-installed",
"service-disabled-by-launchctl",
"service-enable-marker-absent",
] as const)("treats %s as disabled", async (reason) => {
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => disabled(reason),
});
expect(result).toEqual({ runtime: "stopped" });
expect(stopMock).toHaveBeenCalledOnce();
});
it("reports a live lease owner it could not verify instead of signalling it", async () => {
stopMock.mockResolvedValue({ stopped: [], unverified: [9999] });
const events: string[] = [];
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: false,
stopSupervisor,
standalone: true,
queryServiceState: async () => disabled(),
onDiagnostic: (event) => {
events.push(`${event.outcome}:${event.detail}`);
},
});
expect(result).toEqual({ runtime: "unchanged" });
expect(stopMock).not.toHaveBeenCalled();
expect(stopSupervisor).toHaveBeenCalledOnce();
expect(result).toEqual({
runtime: "failed",
detail: "unverified-runtime-left-running",
});
expect(events).toEqual(["degraded:unverified-runtime-left-running"]);
});
it("tears the supervisor down before it stops the process", async () => {
const order: string[] = [];
stopMock.mockImplementation(async () => {
order.push("process");
return "stopped";
return { stopped: [4321], unverified: [] };
});
await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => {
order.push("query");
return disabled();
},
stopSupervisor: async () => {
order.push("supervisor");
},
});
expect(order).toEqual(["supervisor", "process"]);
expect(order).toEqual(["supervisor", "query", "process"]);
});
it("reports not-running instead of throwing when nothing is attached", async () => {
stopMock.mockResolvedValue("not-running");
it("reports not-running when nothing serves the data dir", async () => {
stopMock.mockResolvedValue({ stopped: [], unverified: [] });
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
stopSupervisor: async () => {},
queryServiceState: async () => disabled(),
});
expect(result).toEqual({ runtime: "not-running" });
@@ -103,7 +169,7 @@ describe("BUG-12: a full quit stops the standalone Runtime process", () => {
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
stopSupervisor: async () => {},
queryServiceState: async () => disabled(),
onDiagnostic: async (event) => {
events.push(event.outcome);
},
@@ -117,24 +183,100 @@ describe("BUG-12: a full quit stops the standalone Runtime process", () => {
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => disabled(),
stopSupervisor: async () => {
throw new Error("supervisor exploded");
},
});
// The process is what must not be left behind; a supervisor error is not
// an excuse to skip it.
expect(stopMock).toHaveBeenCalledOnce();
expect(result).toEqual({
runtime: "failed",
detail: "supervisor exploded",
});
});
});
describe("App quit when the service state cannot be determined", () => {
it("stops nothing on a timeout and says so", async () => {
const events: string[] = [];
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => unknown,
onDiagnostic: (event) => {
events.push(`${event.outcome}:${event.detail}`);
},
});
expect(result).toEqual({
runtime: "preserved",
reason: "service-state-unknown:service-state-timeout",
});
expect(stopMock).not.toHaveBeenCalled();
expect(events).toEqual([
"degraded:runtime-preserved:service-state-unknown:service-state-timeout",
]);
});
it("stops nothing when the query itself throws", async () => {
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState: async () => {
throw new Error("launchctl exploded");
},
});
expect(result).toEqual({
runtime: "preserved",
reason: "service-state-unknown:service-state-query-failed",
});
expect(stopMock).not.toHaveBeenCalled();
});
});
describe("Runtimes quit never touches", () => {
it("leaves a Runtime whose build could not be read, even with the service disabled", async () => {
const queryServiceState = vi.fn(async () => disabled());
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: true,
queryServiceState,
preserveReason: "runtime-build-unknown",
});
expect(result).toEqual({
runtime: "preserved",
reason: "runtime-build-unknown",
});
expect(stopMock).not.toHaveBeenCalled();
});
it("does not touch a separate Runtime process in an embedded build", async () => {
const stopSupervisor = vi.fn(async () => {});
const queryServiceState = vi.fn(async () => disabled());
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: false,
queryServiceState,
stopSupervisor,
});
expect(result).toEqual({ runtime: "unchanged" });
expect(stopMock).not.toHaveBeenCalled();
expect(queryServiceState).not.toHaveBeenCalled();
expect(stopSupervisor).toHaveBeenCalledOnce();
});
it("reports a supervisor failure in an embedded build too", async () => {
const result = await shutdownRuntime({
dataDir: "/tmp/runtime-data",
standalone: false,
queryServiceState: async () => disabled(),
stopSupervisor: async () => {
throw new Error("supervisor exploded");
},
+68 -24
View File
@@ -1,13 +1,24 @@
import { stopRuntimeProcess } from "@one-memory/runtime-control";
import {
type RuntimeServiceState,
type StopAllRuntimesResult,
stopAllRuntimeProcesses,
} from "@one-memory/runtime-control";
export interface RuntimeShutdownOptions {
dataDir: string;
/** Packaged builds run the Runtime as their own process; a dev build's Core
* is this process's child and is stopped by the supervisor. */
standalone: boolean;
/** A managed OS service outlives Electron; a transient App-spawned Runtime
* is still stopped on quit. */
preserveManagedRuntime?: boolean;
/** Asks the service manager whether the background service is on. Called
* at quit time, every time: what startup saw may no longer be true. */
queryServiceState: () => Promise<RuntimeServiceState>;
/** Set when startup adopted a Runtime whose build it could not read. This
* App never stops such a Runtime. A newer Runtime gets no such pass: it
* follows the service state like any adopted Runtime. */
preserveReason?: string | null;
stopRuntimes?: (options: {
dataDir: string;
}) => Promise<StopAllRuntimesResult>;
/** Tear down this process's supervisor state (Core child, broker). */
stopSupervisor?: () => Promise<void>;
onDiagnostic?: (event: {
@@ -19,16 +30,22 @@ export interface RuntimeShutdownOptions {
export type RuntimeShutdownResult =
| { runtime: "stopped" }
| { runtime: "not-running" }
| { runtime: "preserved"; reason: string }
| { runtime: "unchanged" }
| { runtime: "failed"; detail: string };
/** Stop everything a full quit has to stop.
/** Stop everything a full quit has to stop, and nothing else.
*
* Quitting used to tear down this process's own handles and nothing else. In a
* packaged build the Runtime is a separate process, so it outlived the app: the
* window and the tray were gone and the Runtime was still there, and its
* receipt still named a pid, so the next launch concluded it was already
* running and attached to a process that no longer answered.
* Exactly one Runtime serves a data dir. Whether quitting the App ends it is
* decided here, at quit time, from the service manager's answer:
*
* - background service enabled: the Runtime is the user's background memory
* service. Quit never stops it, whoever started the process, including a
* transient Runtime this App started earlier.
* - background service disabled: the Runtime lives only while the App does.
* Quit stops every Runtime of this data dir: the App's own child, one
* `memhub start` launched, a leftover from an earlier session.
* - state unknown (query failed or timed out): stop nothing, and say so.
*
* A failure here is reported, never thrown. By the time this runs the app is
* already quitting, and throwing would only turn a failure someone could see
@@ -39,15 +56,6 @@ export async function shutdownRuntime(
): Promise<RuntimeShutdownResult> {
const stopSupervisor = options.stopSupervisor ?? (async () => {});
const supervisorFailure = await settle(stopSupervisor);
if (options.preserveManagedRuntime) {
await options.onDiagnostic?.({
outcome: supervisorFailure ? "degraded" : "succeeded",
detail: supervisorFailure ?? "managed-runtime-preserved",
});
return supervisorFailure
? { runtime: "failed", detail: supervisorFailure }
: { runtime: "unchanged" };
}
if (!options.standalone) {
if (supervisorFailure) {
await options.onDiagnostic?.({
@@ -58,16 +66,52 @@ export async function shutdownRuntime(
}
return { runtime: "unchanged" };
}
try {
const result = await stopRuntimeProcess({ dataDir: options.dataDir });
const preserved = async (
reason: string,
outcome: "succeeded" | "degraded",
): Promise<RuntimeShutdownResult> => {
await options.onDiagnostic?.({
outcome: supervisorFailure ? "degraded" : "succeeded",
detail: supervisorFailure ?? result,
outcome: supervisorFailure ? "degraded" : outcome,
detail: supervisorFailure ?? `runtime-preserved:${reason}`,
});
return supervisorFailure
? { runtime: "failed", detail: supervisorFailure }
: { runtime: "preserved", reason };
};
if (options.preserveReason) {
return preserved(options.preserveReason, "succeeded");
}
const service = await options.queryServiceState().catch(
(): RuntimeServiceState => ({
state: "unknown",
reason: "service-state-query-failed",
jobPid: null,
}),
);
if (service.state === "enabled") {
return preserved("background-service-enabled", "succeeded");
}
if (service.state !== "disabled") {
// Fail safe: a Runtime stopped by mistake takes the user's background
// memory with it; one left running by mistake is only a process.
return preserved(`service-state-unknown:${service.reason}`, "degraded");
}
try {
const result = await (options.stopRuntimes ?? stopAllRuntimeProcesses)({
dataDir: options.dataDir,
});
const stopped = result.stopped.length > 0 ? "stopped" : "not-running";
const unverified =
result.unverified.length > 0 ? "unverified-runtime-left-running" : null;
await options.onDiagnostic?.({
outcome: supervisorFailure || unverified ? "degraded" : "succeeded",
detail: supervisorFailure ?? unverified ?? `${stopped}:${service.reason}`,
});
if (supervisorFailure) {
return { runtime: "failed", detail: supervisorFailure };
}
return { runtime: result };
if (unverified) return { runtime: "failed", detail: unverified };
return { runtime: stopped };
} catch (error) {
const detail = error instanceof Error ? error.message : "unknown";
await options.onDiagnostic?.({ outcome: "degraded", detail });
@@ -143,6 +143,7 @@ const SAFE_STABLE_RUNTIME_ERROR_CODES = new Set([
"sdk_migration_status_unavailable",
"sdk_migration_incomplete",
"runtime_service_not_ready",
"runtime_build_unknown",
]);
/** Every migration code is a stable token Core or the orchestrator chose;
@@ -242,6 +243,8 @@ export const STABLE_RUNTIME_USER_MESSAGES: Record<string, string> = {
"记忆数据格式升级没有完成,MemHub 暂时无法打开。请重新启动重试;如果仍然失败,请把这个提示反馈给我们,原有记忆数据不会丢失。",
runtime_service_not_ready:
"记忆服务仍在首次启动,请稍候后重试。你的记忆数据不会受到影响。",
runtime_build_unknown:
"无法确认正在运行的 MemHub 后台版本,App 已连接到它,但不会停止或替换它。如遇问题,请更新 App 或重启后台服务;你的记忆数据不会受到影响。",
desktop_start_failed:
"MemHub 启动失败。请重新启动;如果仍然打不开,请重新安装,你的记忆数据不会受到影响。",
};
@@ -0,0 +1,185 @@
// @vitest-environment jsdom
/**
* #230: DeepSeek Harness has no single host, so the app page used to grey the
* switch out with `unsupported_host` and offered no other way to grant recall.
* The one switch now covers every detected profile.
*/
import type { SourceResponse } from "@one-memory/contracts";
import {
cleanup,
fireEvent,
render,
screen,
waitFor,
within,
} from "@testing-library/react";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { ApplicationDetail } from "./ApplicationDetail";
import { readConnectionPreferences } from "./connection-preferences";
import { resetConnectionReconcile } from "./connection-reconcile";
import { harnessBridge } from "./harness-profiles.test-fixture";
const source = {
source_id: "deepseek-harness",
display_name: "DeepSeek Harness",
authorized: true,
sync_enabled: true,
capabilities: {
historical_import: "implemented",
incremental_import: "implemented",
live_sync: "implemented",
recall: "implemented",
installer: "implemented",
verification: "real_replayed",
},
} as SourceResponse;
function renderDetail() {
return render(
<ApplicationDetail
source={source}
sources={[source]}
stale={false}
setSync={vi.fn()}
refresh={vi.fn().mockResolvedValue({ sources: [source] })}
onBack={vi.fn()}
/>,
);
}
function profileItems() {
return within(
screen.getByRole("list", { name: "DeepSeek Harness 的 Profile" }),
)
.getAllByRole("listitem")
.map((item) => item.textContent);
}
beforeEach(() => {
window.localStorage.clear();
resetConnectionReconcile();
});
afterEach(() => {
cleanup();
window.oneMemory = undefined;
window.localStorage.clear();
resetConnectionReconcile();
});
it("lists every detected profile with its own state", async () => {
harnessBridge({ desktop: "approved", web: "not-installed" });
renderDetail();
await waitFor(() =>
expect(profileItems()).toEqual(["desktop:已接入", "web:未接入"]),
);
const insertion = screen.getByRole("switch", { name: "一键插入应用" });
expect(insertion.hasAttribute("disabled")).toBe(false);
expect(screen.queryByText(/当前安装不支持一键插入/)).toBeNull();
});
it("turns every profile on with the one switch and remembers the choice", async () => {
const { api, state } = harnessBridge({
desktop: "not-installed",
web: "not-installed",
});
renderDetail();
const insertion = await screen.findByRole("switch", { name: "一键插入应用" });
await waitFor(() => expect(profileItems()).toHaveLength(2));
expect(insertion.getAttribute("aria-checked")).toBe("false");
fireEvent.click(insertion);
await waitFor(() =>
expect(profileItems()).toEqual(["desktop:已接入", "web:已接入"]),
);
expect(api.installRecall.mock.calls.map(([hostId]) => hostId)).toEqual([
"deepseek-harness:desktop",
"deepseek-harness:web",
]);
expect([...state.values()]).toEqual(["approved", "approved"]);
expect(insertion.getAttribute("aria-checked")).toBe("true");
// Not "未接通": the self-checks the grants recorded were read back.
await waitFor(() =>
expect(
screen.getByRole("button", { name: /已接通|未接通|检测中/ }).textContent,
).toBe("已接通"),
);
// Remembered, so a profile found on a later launch is connected too.
expect(readConnectionPreferences().hostSourceIds).toEqual([
"deepseek-harness",
]);
});
it("names the profile that failed and keeps the others on", async () => {
const { state } = harnessBridge(
{ desktop: "not-installed", web: "not-installed" },
{ failInstall: ["web"] },
);
renderDetail();
const insertion = await screen.findByRole("switch", { name: "一键插入应用" });
await waitFor(() => expect(profileItems()).toHaveLength(2));
fireEvent.click(insertion);
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("以下 Profile 未能接入:web");
expect(alert.textContent).toContain("应用控制通道暂不可用");
await waitFor(() =>
expect(profileItems()).toEqual(["desktop:已接入", "web:未接入"]),
);
expect(state.get("deepseek-harness:desktop")).toBe("approved");
expect(insertion.getAttribute("aria-checked")).toBe("true");
});
it("turns every linked profile off with the one switch", async () => {
const { api, state } = harnessBridge({
desktop: "approved",
web: "approved",
});
renderDetail();
const insertion = await screen.findByRole("switch", { name: "一键插入应用" });
await waitFor(() =>
expect(insertion.getAttribute("aria-checked")).toBe("true"),
);
fireEvent.click(insertion);
await waitFor(() =>
expect(profileItems()).toEqual(["desktop:未接入", "web:未接入"]),
);
expect(api.uninstallRecall).toHaveBeenCalledTimes(2);
expect([...state.values()]).toEqual(["not-installed", "not-installed"]);
expect(insertion.getAttribute("aria-checked")).toBe("false");
expect(readConnectionPreferences().hostSourceIds).toEqual([]);
});
it("names the profile whose connection could not be removed", async () => {
harnessBridge(
{ desktop: "approved", web: "approved" },
{
failUninstall: ["desktop"],
},
);
renderDetail();
const insertion = await screen.findByRole("switch", { name: "一键插入应用" });
await waitFor(() =>
expect(insertion.getAttribute("aria-checked")).toBe("true"),
);
fireEvent.click(insertion);
// Removal failed but revocation went through: the grant is gone, the file
// is left for the person to check.
expect((await screen.findByRole("alert")).textContent).toContain(
"已撤销以下 Profile 的记忆权限,但连接文件没有全部移除,请核对:desktop",
);
});
it("says how to get a profile when none is detected", async () => {
harnessBridge({});
renderDetail();
expect(
await screen.findByText(
"还没有检测到 DeepSeek Harness 的 Profile。创建 Profile 后再回到此页。",
),
).toBeTruthy();
expect(
screen
.getByRole("switch", { name: "一键插入应用" })
.hasAttribute("disabled"),
).toBe(true);
});
+126 -8
View File
@@ -2,15 +2,21 @@ import type { SourceResponse } from "@one-memory/contracts";
import type { MessageKey } from "@one-memory/i18n";
import { useCallback, useEffect, useRef, useState } from "react";
import { AppMark } from "./app-mark";
import { connectionHostId, preferenceHasSource } from "./connection-hosts";
import {
connectionHostId,
isProfileSource,
preferenceHasSource,
profileName,
} from "./connection-hosts";
import {
readConnectionPreferences,
writeConnectionPreference,
} from "./connection-preferences";
import { noteUserHostChoice } from "./connection-reconcile";
import type { RecallLinkCheck } from "./global";
import type { RecallHostStatus, RecallLinkCheck } from "./global";
import {
disconnectHost,
disconnectProfiles,
enableHostConnection,
hostCapabilities,
hostConnectionApproved,
@@ -19,8 +25,11 @@ import {
hostWriteReason,
matchHost,
probeHost,
profileHosts,
profileWriteFailures,
refreshRecallHosts,
setHostMemoryPermission,
sourceConnectionApproved,
UNSUPPORTED_NOTE,
useRecallHosts,
} from "./host-state";
@@ -63,6 +72,24 @@ function connectionLabel(input: {
return "未开启关联";
}
/** The recorded self-check across the profiles that may use memory: wired
* only when every one of them is, otherwise the first one that is not. */
function profilesProbeState(profiles: RecallHostStatus[]) {
const linked = profiles.filter(hostConnectionApproved);
if (!linked.length) return null;
for (const host of linked) {
const state = host.selfCheck?.state ?? null;
if (state !== "wired") return state;
}
return "wired";
}
function profileList(hosts: readonly ProfileHost[]) {
return hosts.map((host) => profileName(host.hostId)).join(", ");
}
type ProfileHost = Pick<RecallHostStatus, "hostId">;
function SourceIcon({ source }: { source: SourceResponse }) {
return (
<AppMark
@@ -215,7 +242,10 @@ export function ApplicationDetail({
if (!mounted.current || version !== epoch.current) return;
setRead({ sourceId: id, state: "ready" });
setCheckedAt(new Date().toISOString());
if (matched?.installed) void probeLink(matched.hostId);
// A per-profile source has no single host to probe; its status comes
// from each profile's recorded self-check instead.
if (matched?.installed && !isProfileSource(id))
void probeLink(matched.hostId);
else setLink(null);
} catch {
if (mounted.current && version === epoch.current)
@@ -292,10 +322,14 @@ export function ApplicationDetail({
}
const current = read.sourceId === source.source_id ? read : null;
const host = matchHost(hosts, source.source_id);
const profileSource = isProfileSource(source.source_id);
const profiles = profileHosts(hosts, source.source_id);
const loading = !current || current.state === "loading";
const readFailed = current?.state === "error";
const approved = hostConnectionApproved(host);
const unsupported = hostConnectionUnsupported(host);
const approved = profileSource
? sourceConnectionApproved(hosts, source.source_id)
: hostConnectionApproved(host);
const unsupported = !profileSource && hostConnectionUnsupported(host);
const insertionChecked = insertionIntent ?? approved;
const memoryChecked = memoryIntent ?? host?.permission?.state === "approved";
const probing =
@@ -305,9 +339,10 @@ export function ApplicationDetail({
loading ||
readFailed ||
(stale && current?.state !== "ready");
const probeState =
link?.probe.state ??
(linkPending || linkFailed ? null : (host?.selfCheck?.state ?? null));
const probeState = profileSource
? profilesProbeState(profiles)
: (link?.probe.state ??
(linkPending || linkFailed ? null : (host?.selfCheck?.state ?? null)));
const stateText = connectionLabel({
approved,
insertionOn: insertionChecked,
@@ -326,10 +361,43 @@ export function ApplicationDetail({
? { reason: failureNote(installError.failure) }
: {}),
});
async function applyProfiles(enabled: boolean) {
if (enabled) {
try {
await enableHostConnection(source.source_id);
} finally {
// Each grant records its self-check on the desktop, and the status row
// reads those; nothing else re-reads them while this page is open.
await refreshRecallHosts().catch(() => {});
}
return;
}
const results = await disconnectProfiles(source.source_id);
const failed = results.filter((item) => item.outcome.state === "failed");
const retained = results.filter(
(item) =>
item.outcome.state === "revoked" || item.outcome.state === "review",
);
if (retained.length)
setInstallError({
key: "appProfilesRevokedRetained",
params: { profiles: profileList(retained) },
});
const first = failed[0]?.outcome;
if (first?.state === "failed")
throw Object.assign(Error("disconnect_failed"), {
profiles: failed.map((item) => profileName(item.hostId)),
failure: {
reason: first.reason,
rereadFailed: first.rereadFailed,
} satisfies HostFailure,
});
}
async function applyInsertion(enabled: boolean) {
const bridge = window.oneMemory;
if (!bridge) return;
setInstallError("");
if (profileSource) return applyProfiles(enabled);
if (enabled) {
await enableHostConnection(source.source_id);
void probeLink(hostId);
@@ -385,6 +453,30 @@ export function ApplicationDetail({
}
// A failed turn-on keeps the stored choice on; a failed turn-off
// shows the grant that is really still there after the re-read.
// Per-profile sources name the profiles that failed; the others
// keep what they got.
const failedProfiles =
profileWriteFailures(error) ??
(error as { profiles?: string[] }).profiles ??
null;
if (failedProfiles) {
setInstallError({
key: wanted
? "appProfilesConnectFailed"
: "appProfilesDisconnectFailed",
params: { profiles: failedProfiles.join(", ") },
failure: wanted
? {
reason: hostWriteReason(error),
rereadFailed: hostRereadFailed(error),
}
: ((error as { failure?: HostFailure }).failure ?? {
reason: null,
rereadFailed: false,
}),
});
continue;
}
setInstallError(
wanted
? {
@@ -561,6 +653,31 @@ export function ApplicationDetail({
<div>
<h2>{text("一键插入应用")}</h2>
<p>{text("安装记忆连接,并默认允许此应用使用已授权记忆。")}</p>
{profileSource &&
(profiles.length ? (
<>
<ul
className="r6-app-profiles"
aria-label={t("appProfilesLabel", {
name: source.display_name,
})}
>
{profiles.map((profile) => (
<li key={profile.hostId}>
{t(
hostConnectionApproved(profile)
? "appProfileOn"
: "appProfileOff",
{ profile: profileName(profile.hostId) },
)}
</li>
))}
</ul>
<p>{t("appProfilesHint")}</p>
</>
) : (
<p>{t("appProfilesNone", { name: source.display_name })}</p>
))}
{unsupported && (
<p>{`${source.display_name} ${text(UNSUPPORTED_NOTE)}`}</p>
)}
@@ -578,6 +695,7 @@ export function ApplicationDetail({
stale ||
appLocationNote !== null ||
unsupported ||
(profileSource && !profiles.length) ||
source.capabilities?.installer !== "implemented" ||
source.capabilities?.recall !== "implemented" ||
!hostCapabilities().install
+23 -18
View File
@@ -31,6 +31,8 @@ import {
hostBridgeAvailable,
hostConnectionApproved,
hostConnectionUnknown,
sourceConnectionApproved,
sourceConnectionIncomplete,
useRecallHosts,
} from "./host-state";
import { subscribePoll } from "./poll-scheduler";
@@ -737,12 +739,7 @@ export function ApplicationsHome({
failure.kind === "host"
? preferences.hostSourceIds.some((id) =>
sameConnectionHost(id, failure.sourceId),
) &&
!hosts.some(
(host) =>
hostConnectionApproved(host) &&
hostMatchesSource(failure.sourceId, host.hostId),
)
) && sourceConnectionIncomplete(hosts, failure.sourceId)
: preferences.syncSourceIds.includes(failure.sourceId) &&
!imports.sources.some(
(source) =>
@@ -754,14 +751,22 @@ export function ApplicationsHome({
const autoFailureText = autoFailures
.map((failure) =>
failure.kind === "host"
? t("apps.autoHostFailed", {
name: nameOf(failure.sourceId),
reason: failure.reason
? t("apps.failureReason", {
reason: localizeFailureReason(failure.reason),
})
: "",
})
? t(
failure.profiles
? "apps.autoHostProfilesFailed"
: "apps.autoHostFailed",
{
name: nameOf(failure.sourceId),
...(failure.profiles
? { profiles: failure.profiles.join(", ") }
: {}),
reason: failure.reason
? t("apps.failureReason", {
reason: localizeFailureReason(failure.reason),
})
: "",
},
)
: t("apps.autoSyncFailed", {
name: nameOf(failure.sourceId),
reason: localizeFailureReason(failure.reason),
@@ -1181,8 +1186,9 @@ export function ApplicationsHome({
t,
callState === "unavailable",
) || sourceSyncActivity(source.source_id, syncHealth, locale, t);
const host = hosts.find((item) =>
sourceMatchesHost(source.source_id, item.hostId),
const recallAllowed = sourceConnectionApproved(
hosts,
source.source_id,
);
return (
<div className="r6-home-app-row" key={source.source_id}>
@@ -1193,8 +1199,7 @@ export function ApplicationsHome({
<p>
{source.sync_enabled
? t("apps.syncEnabled")
: host?.installed &&
host.permission?.state === "approved"
: recallAllowed
? t("apps.recallAllowed")
: source.authorized
? t("apps.historyAllowed")
+99 -22
View File
@@ -4,7 +4,6 @@ import type { GovernedMemory, MemoryDetail } from "@one-memory/contracts";
import {
act,
cleanup,
fireEvent,
render,
screen,
waitFor,
@@ -75,7 +74,6 @@ const {
askMemoriesStream,
setExtractionWindow,
manualExtract,
retryUnknownMemoryProcessing,
getGraphWidth,
decideGraphWidth,
getModelProfiles,
@@ -89,7 +87,6 @@ const {
askMemoriesStream: vi.fn(),
setExtractionWindow: vi.fn(),
manualExtract: vi.fn(),
retryUnknownMemoryProcessing: vi.fn(),
getGraphWidth: vi.fn(),
decideGraphWidth: vi.fn(),
getModelProfiles: vi.fn(),
@@ -107,7 +104,6 @@ vi.mock("./api", () => ({
askMemoriesStream,
setExtractionWindow,
manualExtract,
retryUnknownMemoryProcessing,
getGraphWidth,
decideGraphWidth,
getModelProfiles,
@@ -159,7 +155,6 @@ describe("MemorySpace", () => {
in_flight_count: 0,
seconds_per_completion: null,
});
retryUnknownMemoryProcessing.mockResolvedValue({ requeued: 0 });
getGraphWidth.mockResolvedValue({ mismatch: null, memory_count: 1 });
decideGraphWidth.mockResolvedValue({
archived_path: null,
@@ -394,11 +389,9 @@ describe("MemorySpace", () => {
).toBeTruthy();
});
it("shows unknown results once and only retries after confirmation", async () => {
const confirm = vi
.spyOn(window, "confirm")
.mockReturnValueOnce(false)
.mockReturnValueOnce(true);
it("leaves unknown outcomes to Core: no manual retry, no alarm", async () => {
// An unknown outcome is a transient "checking" state Core settles by
// itself within seconds; there is nothing for the reader to do.
getMemoryProcessing.mockResolvedValue({
pending: 0,
fresh_pending: 0,
@@ -420,20 +413,104 @@ describe("MemorySpace", () => {
in_flight_count: 0,
seconds_per_completion: 20,
});
retryUnknownMemoryProcessing.mockResolvedValue({ requeued: 17 });
render(<MemorySpace />);
expect(
await screen.findByText("有 17 段对话的整理结果不确定"),
).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "重新整理" }));
expect(retryUnknownMemoryProcessing).not.toHaveBeenCalled();
fireEvent.click(screen.getByRole("button", { name: "重新整理" }));
await waitFor(() =>
expect(retryUnknownMemoryProcessing).toHaveBeenCalledOnce(),
);
expect(confirm).toHaveBeenCalledTimes(2);
confirm.mockRestore();
expect(await screen.findByText("整理状态")).toBeTruthy();
expect(screen.queryByText("整理需要处理")).toBeNull();
expect(screen.queryByText(/整理结果不确定/)).toBeNull();
expect(screen.queryByRole("button", { name: "重新整理" })).toBeNull();
});
describe("模型熔断提示", () => {
function backoff(
error: string | null,
pending = 4,
): Record<string, unknown> {
return {
pending,
running: 0,
fresh_pending: pending,
backlog_pending: 0,
recent_completions: [],
waiting_for_model: 0,
completed: 1,
failed: 0,
blocked_stages: [],
outside_window_pending: 0,
extraction_window_preset: "3m",
manual_extract_batches_planned: 0,
manual_extract_batches_current: 0,
active_queue: 0,
in_flight_count: 0,
seconds_per_completion: null,
suspended: 0,
llm_backoff_until: error ? "2026-11-02T10:00:00+00:00" : null,
llm_backoff_error: error,
};
}
it("names the mapped reason and the queued count, and links to the model", async () => {
const onConfigureModel = vi.fn();
getMemoryProcessing.mockResolvedValue(backoff("model_auth_failed"));
render(
<MemorySpace
onConnect={() => {}}
onConfigureModel={onConfigureModel}
/>,
);
const banner = await screen.findByRole("status", {
name: "模型暂停处理",
});
expect(banner.textContent).toContain(
"模型配置有问题(API Key 无效或没有权限),记忆暂停处理,已有 4 条在排队",
);
await userEvent.click(
within(banner).getByRole("button", { name: "配置整理模型" }),
);
expect(onConfigureModel).toHaveBeenCalledTimes(1);
});
it("reads an unrecognised code as a plain connection failure", async () => {
getMemoryProcessing.mockResolvedValue(backoff("model_something_new", 1));
render(<MemorySpace onConnect={() => {}} onConfigureModel={() => {}} />);
const banner = await screen.findByRole("status", {
name: "模型暂停处理",
});
expect(banner.textContent).toContain("(连接模型服务失败)");
expect(banner.textContent).toContain("已有 1 条在排队");
});
it("shows nothing while the breaker is closed", async () => {
getMemoryProcessing.mockResolvedValue(backoff(null));
render(<MemorySpace onConnect={() => {}} onConfigureModel={() => {}} />);
expect(await screen.findByText("正在整理记忆")).toBeTruthy();
expect(screen.queryByRole("status", { name: "模型暂停处理" })).toBeNull();
expect(screen.queryByText(/模型配置有问题/)).toBeNull();
});
it("goes away by itself once a refresh clears the pause", async () => {
vi.useFakeTimers({ shouldAdvanceTime: true });
getMemoryProcessing.mockResolvedValue(backoff("model_timeout"));
render(<MemorySpace onConnect={() => {}} onConfigureModel={() => {}} />);
const banner = await screen.findByRole("status", {
name: "模型暂停处理",
});
expect(banner.textContent).toContain("模型响应超时");
getMemoryProcessing.mockResolvedValue(backoff(null));
await act(async () => {
await vi.advanceTimersByTimeAsync(3_500);
});
await waitFor(() =>
expect(
screen.queryByRole("status", { name: "模型暂停处理" }),
).toBeNull(),
);
});
});
it("splits the queue into new and catch-up rather than one total", async () => {
+51 -47
View File
@@ -37,7 +37,6 @@ import {
getSdkMemoryGraph,
getSdkMemoryLibrary,
manualExtract,
retryUnknownMemoryProcessing,
setExtractionWindow,
updateSdkMemory,
} from "./api";
@@ -46,7 +45,6 @@ import { useGuiLanguage } from "./gui-language";
import { MemoryDeleteDialog } from "./MemoryDeleteDialog";
import { MemoryMap } from "./MemoryMap";
import { subscribePoll } from "./poll-scheduler";
import { R6Button } from "./R6Controls";
import { SdkMemoryLibrary } from "./SdkMemoryLibrary";
import { SdkRawHistory } from "./SdkRawHistory";
import {
@@ -65,6 +63,7 @@ import type { MemoryArrival } from "./memory-arrival-state";
import {
blockReasonLabel,
extractionWindowPresetLabel,
llmBackoffReasonLabel,
modelBlock,
splitBlockedStages,
taskKindLabel,
@@ -691,44 +690,15 @@ export function MemorySpace({
}
}
async function retryUnknown() {
if (
busy ||
!window.confirm(
t(
"memory:重新整理可能再次调用模型。只会重跑结果不确定的对话,是否继续?",
),
)
)
return;
setBusy(true);
setError(null);
const trace = traceUiAction("memory", "retry");
try {
const receipt = await retryUnknownMemoryProcessing(trace.traceId);
trace.done("succeeded", { count: receipt.requeued });
setProcessing(await getMemoryProcessing());
} catch (reason) {
trace.done(writeTraceStatus(reason), {
reason: isDefiniteRefusal(reason)
? "memory_unknown_retry_rejected"
: "memory_unknown_retry_result_unknown",
});
setError(message(reason, t));
} finally {
setBusy(false);
}
}
// What the reader owes attention, and what the pipeline is already handling
// on its own. The second half is counted, never listed: an internal guard
// that clears itself on the next pass is not something to act on. See
// `worthNamingBlock`.
// `worthNamingBlock`. An unknown outcome is in the second half too: Core
// settles it by itself within seconds.
const blocked = processing
? splitBlockedStages(processing.blocked_stages)
: { named: [], retrying: 0 };
const processingNeedsAttention =
blocked.named.length > 0 || (processing?.unknown ?? 0) > 0;
const processingNeedsAttention = blocked.named.length > 0;
return (
<section
@@ -933,19 +903,6 @@ export function MemorySpace({
</span>
) : null}
</p>
{(processing.unknown ?? 0) > 0 ? (
<div className="memory-processing-unknown" role="status">
<span>
{tn(
"memory:有 {count} 段对话的整理结果不确定",
processing.unknown ?? 0,
)}
</span>
<R6Button disabled={busy} onClick={() => void retryUnknown()}>
{t("memory:重新整理")}
</R6Button>
</div>
) : null}
<ExtractionWindowControls
processing={processing}
busy={busy}
@@ -963,6 +920,10 @@ export function MemorySpace({
</details>
) : null}
<ModelBackoffBanner
processing={processing}
onConfigureModel={onConfigureModel}
/>
<ModelPrompt
processing={processing}
hasMemories={(sdkLibrary.page?.total ?? 0) > 0}
@@ -1232,6 +1193,49 @@ function ModelPrompt({
);
}
/** Core's circuit breaker paused model work because the provider keeps
* failing (bad key, wrong URL, missing model, provider down).
*
* Nothing is lost and nothing needs retrying by hand: the queued work stays
* pending and Core keeps probing on its own, so this only says what is wrong
* and where to fix it. It is driven by the polled summary alone and goes away
* on the first read that no longer carries `llm_backoff_error`. */
function ModelBackoffBanner({
processing,
onConfigureModel,
}: {
processing: MemoryProcessingSummary | null;
onConfigureModel(): void;
}) {
const { t, tn } = useGuiLanguage();
const code = processing?.llm_backoff_error;
if (!processing || !code) return null;
return (
<p
className="model-prompt"
role="status"
aria-label={t("memory:模型暂停处理")}
>
<span>
{tn(
"memory:模型配置有问题({reason}),记忆暂停处理,已有 {count} 条在排队",
processing.pending,
{
reason: t(
`memory:${llmBackoffReasonLabel(code)}` as Parameters<
typeof t
>[0],
),
},
)}
</span>
<button type="button" onClick={onConfigureModel}>
{t("memory:配置整理模型")}
</button>
</p>
);
}
function MemoryInspector({
detail,
busy,
@@ -13,9 +13,18 @@ vi.mock("axios", () => ({
delete: del,
interceptors: { request: { use: vi.fn() } },
}),
isAxiosError: (value: unknown) =>
typeof value === "object" && value !== null && "isAxiosError" in value,
},
}));
function httpError(status: number, detail: string) {
return Object.assign(new Error(`http_${status}`), {
isAxiosError: true,
response: { status, data: { detail } },
});
}
import {
deleteSdkMemory,
getHomeSummary,
@@ -100,6 +109,18 @@ describe("paged memory library client", () => {
expect(del).toHaveBeenLastCalledWith("/api/v2/memories/c1", undefined);
});
it("treats a delete answered 404 memory_not_found as already gone", async () => {
del.mockRejectedValueOnce(httpError(404, "memory_not_found"));
await expect(deleteSdkMemory("gone")).resolves.toBeUndefined();
const other = httpError(404, "source_not_found");
del.mockRejectedValueOnce(other);
await expect(deleteSdkMemory("x")).rejects.toBe(other);
const failed = httpError(503, "hy_memory_delete_failed");
del.mockRejectedValueOnce(failed);
await expect(deleteSdkMemory("x")).rejects.toBe(failed);
});
it("forwards cancellation for inactive or superseded views", async () => {
const controller = new AbortController();
get.mockResolvedValueOnce({
+19 -14
View File
@@ -763,9 +763,25 @@ export async function deleteSdkMemory(
...(traceId ? { headers: { "X-One-Memory-Trace-Id": traceId } } : {}),
...(params ? { params } : {}),
};
await client.delete(
`/api/v2/memories/${encodeURIComponent(memoryId)}`,
Object.keys(config).length ? config : undefined,
try {
await client.delete(
`/api/v2/memories/${encodeURIComponent(memoryId)}`,
Object.keys(config).length ? config : undefined,
);
} catch (reason) {
// 404 `memory_not_found`: already gone (a second click, or another
// window deleted it first). The memory is not there, which is what the
// person asked for, so it is not an error.
if (isMemoryNotFound(reason)) return;
throw reason;
}
}
function isMemoryNotFound(reason: unknown): boolean {
return (
axios.isAxiosError(reason) &&
reason.response?.status === 404 &&
apiErrorCode(reason) === "memory_not_found"
);
}
@@ -985,17 +1001,6 @@ export async function getModelUsage(
return value;
}
export async function retryUnknownMemoryProcessing(
traceId?: string,
): Promise<{ requeued: number }> {
const response = await client.post<{ reconciled: number; requeued: number }>(
"/api/v1/memory-processing/retry-unknown",
{ confirm: true },
traceId ? { headers: { "X-One-Memory-Trace-Id": traceId } } : undefined,
);
return response.data;
}
export async function getExtractionSettings(): Promise<ExtractionSettings> {
const response = await client.get<ExtractionSettings>(
"/api/v1/extraction-settings",
@@ -118,6 +118,14 @@
margin: 0;
overflow-wrap: anywhere;
}
.r6-app-profiles {
font-size: 13px;
color: #637786;
line-height: 1.8;
margin: 4px 0;
padding-left: 18px;
overflow-wrap: anywhere;
}
.r6-app-detail-actions {
display: flex;
gap: 12px;
+20 -2
View File
@@ -1,5 +1,6 @@
/** Mirrors desktop `SOURCE_HOST`. OpenClaw and DeepSeek Harness profiles
* are not accepted by one-click insertion. */
/** Mirrors desktop `SOURCE_HOST`. OpenClaw is not accepted by one-click
* insertion; DeepSeek Harness has no single host and goes through
* `PROFILE_SOURCES` instead. */
export const SOURCE_HOST: Readonly<Record<string, string>> = {
cursor: "cursor",
"claude-code": "claude-code",
@@ -14,6 +15,23 @@ export const SOURCE_HOST: Readonly<Record<string, string>> = {
export const ONE_CLICK_SOURCES = new Set(Object.keys(SOURCE_HOST));
/** Sources configured per profile. Each profile is its own host
* (`deepseek-harness:<profile>`, with its own patch, credential and grant), so
* the app's one switch covers every profile the desktop detected. */
export const PROFILE_SOURCES: ReadonlySet<string> = new Set([
"deepseek-harness",
]);
export function isProfileSource(sourceId: string) {
return PROFILE_SOURCES.has(sourceId);
}
/** `deepseek-harness:web` -> `web`. */
export function profileName(hostId: string) {
const colon = hostId.indexOf(":");
return colon < 0 ? hostId : hostId.slice(colon + 1);
}
export function connectionHostId(sourceId: string): string | null {
return SOURCE_HOST[sourceId] ?? null;
}
+38 -5
View File
@@ -1,6 +1,10 @@
import type { SourceResponse } from "@one-memory/contracts";
import { useQuery } from "@tanstack/react-query";
import { connectionHostId, sameConnectionHost } from "./connection-hosts";
import {
connectionHostId,
isProfileSource,
sameConnectionHost,
} from "./connection-hosts";
import {
readConnectionPreferences,
writeConnectionPreference,
@@ -8,11 +12,13 @@ import {
import {
cachedRecallHosts,
enableHostConnection,
hostConnectionApproved,
hostConnectionUnsupported,
hostWriteInFlight,
hostWriteReason,
matchHost,
profileHosts,
profileWriteFailures,
sourceConnectionIncomplete,
} from "./host-state";
import { onClientReset, queryClient } from "./query-client";
import { latestSource, sourceWriteInFlight } from "./source-state";
@@ -37,6 +43,8 @@ export type AutoConnectFailure = {
kind: "sync" | "host";
sourceId: string;
reason: string | null;
/** Which profiles of a per-profile source it was; the others are kept. */
profiles?: string[];
};
export type AutoConnectState = {
running: string[];
@@ -53,8 +61,10 @@ queryClient.setQueryDefaults(AUTO_KEY, {
});
let flight: Promise<AutoConnectFailure[]> | null = null;
const seenProfiles = new Map<string, Set<string>>();
onClientReset(() => {
flight = null;
seenProfiles.clear();
});
function autoState() {
@@ -148,8 +158,26 @@ export async function reconcileConnectionPreferences(
function hostNeedsApply(sourceId: string) {
const hosts = cachedRecallHosts();
if (hosts === undefined) return false;
const host = matchHost(hosts, sourceId);
return !hostConnectionApproved(host) && !hostConnectionUnsupported(host);
return sourceConnectionIncomplete(hosts, sourceId);
}
/** `attempted` holds one apply per source, which is right until a per-profile
* source grows a profile: that one was never tried, so the remembered switch
* gets one more apply. A profile that fails is not retried on every poll. */
function rearmNewProfiles(sourceIds: string[]) {
const hosts = cachedRecallHosts();
if (hosts === undefined) return;
for (const sourceId of sourceIds) {
if (!isProfileSource(sourceId)) continue;
const current = profileHosts(hosts, sourceId).map((host) => host.hostId);
const seen = seenProfiles.get(sourceId);
seenProfiles.set(sourceId, new Set([...(seen ?? []), ...current]));
if (!seen || current.every((hostId) => seen.has(hostId))) continue;
updateAuto((state) => ({
...state,
attempted: state.attempted.filter((key) => key !== hostKey(sourceId)),
}));
}
}
/** Whether a remembered host is still waiting for (or in) its one apply. */
@@ -191,6 +219,7 @@ export function autoReconcile(
if (hostConnectionUnsupported(matchHost(hosts, id)))
writeConnectionPreference("host", id, false);
const preferences = readConnectionPreferences();
rearmNewProfiles(preferences.hostSourceIds);
const { attempted } = autoState();
const only = options.only;
const wantedHosts = only
@@ -256,6 +285,7 @@ export function retryAutoReconcile(options: ReconcileOptions) {
export function resetConnectionReconcile() {
flight = null;
seenProfiles.clear();
queryClient.removeQueries({ queryKey: AUTO_KEY });
}
@@ -301,12 +331,15 @@ async function runReconcile(
// Per-app: other hosts keep going. Unsupported never succeeds on retry.
if (error instanceof Error && error.message === "host_unsupported")
writeConnectionPreference("host", sourceId, false);
else
else {
const profiles = profileWriteFailures(error);
failures.push({
kind: "host",
sourceId,
reason: hostWriteReason(error),
...(profiles ? { profiles } : {}),
});
}
}
}
return failures;
@@ -0,0 +1,120 @@
/** A fake desktop bridge holding DeepSeek Harness profile hosts, for tests. */
import { vi } from "vitest";
import type { RecallHostStatus, RecallPermissionStatus } from "./global";
type Bridge = NonNullable<typeof window.oneMemory>;
type Grant =
| "not-installed"
| "install-only"
| "approved"
| "revoked"
| "unsupported";
function permission(hostId: string, state: Grant): RecallPermissionStatus {
return {
hostId,
state,
reason: null,
receipt:
state === "not-installed" || state === "unsupported"
? null
: {
hostId,
installationId: `install-${hostId}`,
revision: 1,
status: state,
allowedTools: state === "approved" ? ["om_recall"] : [],
allowedScopes: state === "approved" ? ["all-apps"] : [],
},
};
}
/** A desktop that knows a set of Harness profiles and keeps their state. */
export function harnessBridge(
initial: Record<string, Grant>,
options: { failInstall?: string[]; failUninstall?: string[] } = {},
) {
const state = new Map(
Object.entries(initial).map(([profile, grant]) => [
`deepseek-harness:${profile}`,
grant,
]),
);
const hosts = (): RecallHostStatus[] =>
[...state].map(([hostId, grant]) => ({
hostId,
label: `DeepSeek Harness · ${hostId.split(":")[1]}`,
present: true,
installed: grant === "install-only" || grant === "approved",
permission: permission(hostId, grant),
// Recorded by the desktop when the grant lands, as on a real install.
selfCheck:
grant === "approved"
? { state: "wired", detail: "", probed_at: "2026-11-02" }
: null,
skillDrift: null,
}));
const failing = (hostId: string, list: string[] = []) =>
list.some((profile) => hostId === `deepseek-harness:${profile}`);
const api = {
getRecallHosts: vi.fn(async () => hosts()),
getRecallPermission: vi.fn(async (hostId: string) =>
permission(hostId, state.get(hostId) ?? "not-installed"),
),
previewRecallInstall: vi.fn(async (hostId: string) => ({
hostId,
hostLabel: hostId,
files: [],
fingerprint: `fp-${hostId}`,
})),
installRecall: vi.fn(async (hostId: string, _approved?: string) => {
if (failing(hostId, options.failInstall))
throw Error(
"Error invoking remote method 'one-memory:install-recall': Error: host_control_unavailable",
);
state.set(hostId, "install-only");
return {
hostId,
installed: true,
permission: permission(hostId, "install-only"),
};
}),
setRecallPermission: vi.fn(async (hostId: string, _input: unknown) => {
state.set(hostId, "approved");
return permission(hostId, "approved");
}),
revokeRecallPermission: vi.fn(async (hostId: string, _input: unknown) => {
state.set(hostId, "revoked");
return permission(hostId, "revoked");
}),
previewRecallUninstall: vi.fn(async (hostId: string) => ({
hostId,
hostLabel: hostId,
files: [],
review: [],
fingerprint: `fp-${hostId}`,
})),
uninstallRecall: vi.fn(async (hostId: string, _approved?: string) => {
if (failing(hostId, options.failUninstall))
throw Error(
"Error invoking remote method 'one-memory:uninstall-recall': Error: host_control_unavailable",
);
state.set(hostId, "not-installed");
return {
hostId,
installed: false,
permission: permission(hostId, "not-installed"),
};
}),
repairRecallCommands: vi.fn(async () => []),
};
window.oneMemory = api as unknown as Bridge;
return {
api,
state,
addProfile(profile: string) {
state.set(`deepseek-harness:${profile}`, "not-installed");
},
};
}
@@ -0,0 +1,210 @@
// @vitest-environment jsdom
/**
* DeepSeek Harness is configured per profile: every `deepseek-harness:<profile>`
* is its own host with its own patch, credential and grant. The app page has
* one switch for all of them (#230). These pin what that switch does: on
* installs and grants every detected profile, off removes and revokes every
* linked one, a failing profile is named without undoing the others, and a
* profile that appears later is picked up by the remembered switch.
*/
import { afterEach, describe, expect, it } from "vitest";
import { saveConnectionPreferences } from "./connection-preferences";
import {
autoReconcile,
resetConnectionReconcile,
} from "./connection-reconcile";
import { harnessBridge } from "./harness-profiles.test-fixture";
import {
disconnectProfiles,
enableHostConnection,
hostWriteReason,
ProfileWriteError,
profileHosts,
refreshRecallHosts,
sourceConnectionApproved,
sourceConnectionIncomplete,
} from "./host-state";
afterEach(() => {
window.oneMemory = undefined;
window.localStorage.clear();
resetConnectionReconcile();
});
describe("DeepSeek Harness profile hosts", () => {
it("lists only the profiles of the per-profile source", async () => {
harnessBridge({ desktop: "not-installed", web: "approved" });
const hosts = await refreshRecallHosts();
expect(
profileHosts(hosts, "deepseek-harness").map((host) => host.hostId),
).toEqual(["deepseek-harness:desktop", "deepseek-harness:web"]);
expect(profileHosts(hosts, "cursor")).toEqual([]);
// One approved profile reads as on; the other is still to do.
expect(sourceConnectionApproved(hosts, "deepseek-harness")).toBe(true);
expect(sourceConnectionIncomplete(hosts, "deepseek-harness")).toBe(true);
});
});
describe("turning the DeepSeek Harness switch on", () => {
it("installs and grants every profile that is not approved yet", async () => {
const { api, state } = harnessBridge({
desktop: "not-installed",
headless: "revoked",
web: "approved",
});
await refreshRecallHosts();
await enableHostConnection("deepseek-harness");
expect(api.installRecall.mock.calls.map(([hostId]) => hostId)).toEqual([
"deepseek-harness:desktop",
"deepseek-harness:headless",
]);
expect(api.setRecallPermission).toHaveBeenCalledWith(
"deepseek-harness:desktop",
expect.objectContaining({
allowedTools: ["om_recall", "om_conversations", "om_diagnostic_trace"],
allowedScopes: ["all-apps", "source:deepseek-harness"],
userConfirmed: true,
}),
);
expect([...state.values()]).toEqual(["approved", "approved", "approved"]);
});
it("keeps the profiles that worked and names the one that failed", async () => {
const { api, state } = harnessBridge(
{ desktop: "not-installed", web: "not-installed" },
{ failInstall: ["web"] },
);
await refreshRecallHosts();
const error = await enableHostConnection("deepseek-harness").catch(
(reason: unknown) => reason,
);
expect(error).toBeInstanceOf(ProfileWriteError);
expect((error as ProfileWriteError).profiles).toEqual(["web"]);
expect((error as Error).message).toContain("host_control_unavailable");
expect(state.get("deepseek-harness:desktop")).toBe("approved");
expect(state.get("deepseek-harness:web")).toBe("not-installed");
expect(api.setRecallPermission).toHaveBeenCalledTimes(1);
// The failure re-read lands the real state of every profile.
const hosts = await refreshRecallHosts();
expect(sourceConnectionApproved(hosts, "deepseek-harness")).toBe(true);
});
});
describe("turning the DeepSeek Harness switch off", () => {
it("removes and revokes every linked profile and leaves unlinked ones alone", async () => {
const { api, state } = harnessBridge({
desktop: "approved",
headless: "not-installed",
web: "approved",
});
await refreshRecallHosts();
const results = await disconnectProfiles("deepseek-harness");
expect(results.map((item) => [item.hostId, item.outcome.state])).toEqual([
["deepseek-harness:desktop", "removed"],
["deepseek-harness:web", "removed"],
]);
expect(api.uninstallRecall).toHaveBeenCalledTimes(2);
expect(state.get("deepseek-harness:desktop")).toBe("not-installed");
expect(state.get("deepseek-harness:web")).toBe("not-installed");
});
it("revokes a profile whose files cannot be removed and still reaches the rest", async () => {
const { api, state } = harnessBridge(
{ desktop: "approved", web: "approved" },
{ failUninstall: ["desktop"] },
);
await refreshRecallHosts();
const results = await disconnectProfiles("deepseek-harness");
expect(results.map((item) => item.outcome.state)).toEqual([
"revoked",
"removed",
]);
expect(api.revokeRecallPermission).toHaveBeenCalledWith(
"deepseek-harness:desktop",
expect.objectContaining({ userConfirmed: true }),
);
expect(state.get("deepseek-harness:desktop")).toBe("revoked");
expect(state.get("deepseek-harness:web")).toBe("not-installed");
});
});
describe("a profile created after the switch was turned on", () => {
it("is installed and granted by the remembered switch while the app is running", async () => {
saveConnectionPreferences({
syncSourceIds: [],
hostSourceIds: ["deepseek-harness"],
});
const harness = harnessBridge({ desktop: "not-installed" });
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toEqual([]);
expect(harness.api.installRecall).toHaveBeenCalledTimes(1);
// One attempt was already spent on this source; the new profile gets its own.
harness.addProfile("web");
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toEqual([]);
expect(harness.api.installRecall.mock.calls.at(-1)).toEqual([
"deepseek-harness:web",
"fp-deepseek-harness:web",
]);
expect(harness.state.get("deepseek-harness:web")).toBe("approved");
});
it("does not retry a failing profile on every poll", async () => {
saveConnectionPreferences({
syncSourceIds: [],
hostSourceIds: ["deepseek-harness"],
});
const harness = harnessBridge(
{ desktop: "approved", web: "not-installed" },
{ failInstall: ["web"] },
);
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toEqual([
{
kind: "host",
sourceId: "deepseek-harness",
reason: "应用控制通道暂不可用",
profiles: ["web"],
},
]);
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toBeNull();
expect(harness.api.installRecall).toHaveBeenCalledTimes(1);
});
it("stays out while the switch is off", async () => {
const harness = harnessBridge({ desktop: "not-installed" });
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toBeNull();
harness.addProfile("web");
await refreshRecallHosts();
expect(await autoReconcile({ sources: [] })).toBeNull();
expect(harness.api.installRecall).not.toHaveBeenCalled();
});
});
describe("a profile whose name MemHub cannot grant", () => {
it("is named with the real reason and does not hold up the others", async () => {
const { state } = harnessBridge({
My_Profile: "unsupported",
web: "not-installed",
});
await refreshRecallHosts();
const error = await enableHostConnection("deepseek-harness").catch(
(reason: unknown) => reason,
);
expect((error as ProfileWriteError).profiles).toEqual(["My_Profile"]);
expect(hostWriteReason(error)).toBe("appProfileNameUnsupported");
expect(state.get("deepseek-harness:web")).toBe("approved");
// Nothing is left for the remembered switch to keep trying.
expect(
sourceConnectionIncomplete(
await refreshRecallHosts(),
"deepseek-harness",
),
).toBe(false);
});
});
+187 -19
View File
@@ -1,5 +1,10 @@
import { MutationObserver, useQuery } from "@tanstack/react-query";
import { connectionHostId, hostMatchesSource } from "./connection-hosts";
import {
connectionHostId,
hostMatchesSource,
isProfileSource,
profileName,
} from "./connection-hosts";
import type {
RecallHostStatus,
RecallPermissionStatus,
@@ -226,6 +231,7 @@ export function matchHost(
const REASONS: Record<string, string> = {
host_unsupported: "它没有 MCP 通道,无法发放可撤销的记忆凭证",
profile_name_unsupported: "appProfileNameUnsupported",
unsupported_host: "当前安装不支持一键插入",
connection_client_unavailable: "桌面端的连接组件不可用",
noncanonical_data_dir_host_integration_disabled: "测试实例不接入应用",
@@ -350,18 +356,77 @@ export function hostWriteInFlight() {
return queryClient.isMutating({ mutationKey: HOST_WRITE }) > 0;
}
/** A per-profile source writes to every profile, so it waits on each of them. */
function sourceHost(sourceId: string) {
if (isProfileSource(sourceId))
return [
sourceId,
...profileHosts(cachedRecallHosts(), sourceId).map((host) => host.hostId),
];
return [connectionHostId(sourceId) ?? sourceId];
}
/** Every detected profile host of a per-profile source, e.g. DeepSeek Harness. */
export function profileHosts(
hosts: readonly RecallHostStatus[] | undefined,
sourceId: string,
) {
if (!isProfileSource(sourceId)) return [];
return (hosts ?? []).filter((host) => host.hostId.startsWith(`${sourceId}:`));
}
/** The app switch reads as on: its host, or for a per-profile source any
* profile, may use memory. A failed profile is reported, not hidden. */
export function sourceConnectionApproved(
hosts: readonly RecallHostStatus[] | undefined,
sourceId: string,
) {
if (isProfileSource(sourceId))
return profileHosts(hosts, sourceId).some(hostConnectionApproved);
return hostConnectionApproved(matchHost(hosts, sourceId));
}
/** Something the switch would still do: a host, or any profile, that is
* neither approved nor unsupported. */
export function sourceConnectionIncomplete(
hosts: readonly RecallHostStatus[] | undefined,
sourceId: string,
) {
const pending = (host: RecallHostStatus | null) =>
!hostConnectionApproved(host) && !hostConnectionUnsupported(host);
if (isProfileSource(sourceId))
return profileHosts(hosts, sourceId).some(pending);
return pending(matchHost(hosts, sourceId));
}
/** Some profiles of a per-profile source could not be written; the others
* were kept. The message keeps the first cause so `hostWriteReason` names it. */
export class ProfileWriteError extends Error {
readonly profiles: string[];
constructor(profiles: string[], cause: unknown) {
super(
`profile_write_failed ${cause instanceof Error ? cause.message : ""}`.trim(),
);
this.profiles = profiles;
}
}
/** The profile names a failed write left behind, if it was a profile write. */
export function profileWriteFailures(error: unknown): string[] | null {
return error instanceof ProfileWriteError ? error.profiles : null;
}
type Connected = {
hostId: string;
installed: boolean;
permission?: RecallPermissionStatus;
};
async function connect(sourceId: string): Promise<Connected | null> {
const hostId = connectionHostId(sourceId);
type Bridge = NonNullable<ReturnType<typeof bridge>>;
async function connect(sourceId: string): Promise<Connected[]> {
const profiles = isProfileSource(sourceId);
const hostId = profiles ? sourceId : connectionHostId(sourceId);
const api = bridge();
if (
!hostId ||
@@ -371,12 +436,65 @@ async function connect(sourceId: string): Promise<Connected | null> {
!api.setRecallPermission
)
throw Error("unsupported_host");
const existing = matchHost(await refreshRecallHosts(), sourceId);
if (hostConnectionUnsupported(existing)) throw Error("host_unsupported");
if (hostConnectionApproved(existing)) {
await api.repairRecallCommands?.();
return null;
const hosts = await refreshRecallHosts();
if (profiles) return connectProfiles(api, sourceId, hosts);
const result = await grantHost(
api,
hostId,
sourceId,
matchHost(hosts, sourceId),
);
await api.repairRecallCommands?.();
return result ? [result] : [];
}
/** One switch, every detected profile. A profile that fails does not undo
* the ones that worked; the error names it. Profiles already approved are
* left alone, so turning the switch on again only reaches what is missing. */
async function connectProfiles(
api: Bridge,
sourceId: string,
hosts: RecallHostStatus[],
): Promise<Connected[]> {
const profiles = profileHosts(hosts, sourceId);
if (!profiles.length) throw Error("profiles_not_found");
const connected: Connected[] = [];
const failed: string[] = [];
let cause: unknown;
for (const host of profiles) {
try {
const result = await grantHost(api, host.hostId, sourceId, host);
if (result) connected.push(result);
} catch (error) {
failed.push(profileName(host.hostId));
// Harness speaks MCP, so "unsupported" can only mean the id grammar
// refused this name; "no MCP channel" would send the person astray.
cause ??= hostConnectionUnsupported(host)
? Error("profile_name_unsupported")
: error;
}
}
if (failed.length < profiles.length) await api.repairRecallCommands?.();
if (failed.length) throw new ProfileWriteError(failed, cause);
return connected;
}
/** Installs and grants one host; null when it was already approved. */
async function grantHost(
api: Bridge,
hostId: string,
sourceId: string,
existing: RecallHostStatus | null,
): Promise<Connected | null> {
if (
!api.previewRecallInstall ||
!api.installRecall ||
!api.getRecallPermission ||
!api.setRecallPermission
)
throw Error("unsupported_host");
if (hostConnectionUnsupported(existing)) throw Error("host_unsupported");
if (hostConnectionApproved(existing)) return null;
const preview = await api.previewRecallInstall(hostId);
const installed = await api.installRecall(hostId, preview.fingerprint);
let permission =
@@ -394,19 +512,19 @@ async function connect(sourceId: string): Promise<Connected | null> {
}
if (permission.receipt?.status !== "approved")
throw Error("permission_not_approved");
await api.repairRecallCommands?.();
return { hostId, installed: true, permission };
}
function landConnected(result: Connected | null) {
if (result)
function landConnected(results: Connected[]) {
for (const result of results)
landHost(result.hostId, {
installed: result.installed,
permission: result.permission,
});
}
/** Installs and grants one host; resolves only once the grant is approved. */
/** Installs and grants one host (or every profile of a per-profile source);
* resolves only once the grants are approved. */
export const enableHostConnection = hostWrite(
sourceHost,
connect,
@@ -479,6 +597,22 @@ async function disconnect(hostId: string): Promise<Disconnected> {
}
}
function landDisconnected({
hostId,
outcome,
}: {
hostId: string;
outcome: DisconnectOutcome;
}) {
if (outcome.state === "removed")
landHost(hostId, {
installed: false,
...(outcome.permission ? { permission: outcome.permission } : {}),
});
else if (outcome.state !== "failed" && outcome.permission)
landHost(hostId, { permission: outcome.permission });
}
export const disconnectHost = hostWrite(
(hostId: string) => [hostId],
async (
@@ -489,14 +623,48 @@ export const disconnectHost = hostWrite(
const rereadFailed = !(await rereadOnce());
return { hostId, outcome: { ...result, rereadFailed } };
},
({ hostId, outcome }) => {
if (outcome.state === "removed")
landHost(hostId, {
installed: false,
...(outcome.permission ? { permission: outcome.permission } : {}),
landDisconnected,
);
export type ProfileDisconnect = { hostId: string; outcome: DisconnectOutcome };
/** Linked to MemHub at all: installed, or holding a receipt that is not yet
* revoked. A profile that never was has nothing to remove. */
function profileLinked(host: RecallHostStatus) {
const status = host.permission?.receipt?.status;
return host.installed || (status !== undefined && status !== "revoked");
}
/** Turn a per-profile source off: each linked profile is removed and its grant
* revoked, exactly as one host would be. One failure does not stop the rest;
* every outcome is returned so the page can name the profile it concerns. */
export const disconnectProfiles = hostWrite(
sourceHost,
async (sourceId: string): Promise<ProfileDisconnect[]> => {
const profiles = profileHosts(await refreshRecallHosts(), sourceId).filter(
profileLinked,
);
const results: ProfileDisconnect[] = [];
let anyFailed = false;
for (const host of profiles) {
const outcome = await disconnect(host.hostId);
anyFailed ||= outcome.state === "failed";
results.push({
hostId: host.hostId,
outcome:
outcome.state === "failed"
? { ...outcome, rereadFailed: false }
: outcome,
});
else if (outcome.state !== "failed" && outcome.permission)
landHost(hostId, { permission: outcome.permission });
}
if (anyFailed && !(await rereadOnce()))
for (const result of results)
if (result.outcome.state === "failed")
result.outcome = { ...result.outcome, rereadFailed: true };
return results;
},
(results) => {
for (const result of results) landDisconnected(result);
},
);
@@ -165,6 +165,25 @@ export function modelBlock(
);
}
/** Why Core's circuit breaker paused model work, from `llm_backoff_error`.
*
* Said as what is wrong with the model, never as the code: the reader's next
* step is the model settings, and the code alone does not say which part of
* them to check. Unknown codes read as a plain connection failure. */
export function llmBackoffReasonLabel(code: string): string {
return LLM_BACKOFF_REASON_LABELS[code] ?? "连接模型服务失败";
}
const LLM_BACKOFF_REASON_LABELS: Record<string, string> = {
model_auth_failed: "API Key 无效或没有权限",
model_not_found: "找不到模型或地址",
model_rate_limited: "请求过于频繁,被服务商限流",
model_timeout: "模型响应超时",
model_server_error: "模型服务出错",
model_unreachable: "无法连接模型服务",
model_transport_failed: "连接模型服务失败",
};
/** What went wrong reading a source, said without the code.
*
* These are `ScanIssueCode` plus the one access-level refusal, and they land
+8 -1
View File
@@ -91,7 +91,14 @@ describe("single state layer", () => {
});
it("reads and writes host connections only through host-state", () => {
expect(offenders(hostUses, ["host-state.ts", "global.d.ts"])).toEqual([]);
expect(
offenders(hostUses, [
"host-state.ts",
"global.d.ts",
// Test support: it is the fake bridge, and only tests import it.
"harness-profiles.test-fixture.ts",
]),
).toEqual([]);
});
it("reads and writes sources only through source-state", () => {
@@ -88,7 +88,6 @@ describe("presentation adapters", () => {
"批量处理结果已保留,但列表刷新失败,请重新读取记忆库。",
"批量处理结果已保留,但列表刷新失败,稍后会自动重新读取。",
"召回范围已保存,但列表刷新失败。请稍后重新打开记忆页。",
"重新整理可能再次调用模型。只会重跑结果不确定的对话,是否继续?",
"原始对话暂时无法读取",
"没有匹配的原始对话",
"还没有导入的原始对话",
+1 -1
View File
@@ -282,7 +282,7 @@ Support is declared per immutable GitHub Release manifest:
- Linux x64/arm64 Runtime on native GHA runners.
- macOS arm64/x64 App/Runtime: Developer ID, notarization, native acceptance.
- Windows is not supported.
- `hy-memory==2.0.11` is the current stable SDK dependency; the release
- `hy-memory==2.0.12` is the current stable SDK dependency; the release
lock and [`../MEMORY.md`](../MEMORY.md) are authoritative when the pin changes.
See [`INSTALLATION.md`](INSTALLATION.md).
+13 -3
View File
@@ -69,8 +69,14 @@ node apps/cli/dist/one-memory.cjs sources list
```
- 只有 `hermes` 能带 `--user-id`(非 `local`)或 `--root`;其他来源带了会退 2,什么都不发。
- 用户 ID 不必等于 profile 名:profile 名里有 `:` 等字符时另起一个,例如
`--user-id wife-work --root ~/.hermes/profiles/wife:work`。
- `--root` 里的 `~` 由 CLI 展开为主目录,相对路径按当前目录解析成绝对路径。
- 扫描 `~/.hermes` 时会排除 `profiles/**`,所以 `local` 不会读到 wife 的 `state.db`。
- 扫描时任何层级名为 `profiles` 的目录都会被排除,所以 `local` 不会读到 wife 的 `state.db`,
即使授权的根在 `~/.hermes` 之上。
- `--root` 不能是 `profiles` 目录本身(`hermes_root_is_profiles_container`),也不能与另一个
用户已接入的 Hermes 目录相同、包含它或在它之内(`hermes_root_overlaps_other_user`,解析
symlink 后比较);`~/.hermes` 给 `local`、`~/.hermes/profiles/<name>` 给另一个用户是允许的。
- App 不变:它只读写 `local`,其他来源也都固定是 `local`。
## 3. 看导入进度
@@ -140,7 +146,11 @@ node apps/cli/dist/one-memory.cjs write status <request_id> --user-id wife
- 用户 ID 只能是 1–64 位字母、数字、`.`、`_`、`-`;其他写法退 2,不连 Runtime。
- 不带 `--user-id`(或写 `--user-id local`)时请求里不出现 `user_id`,和以前完全一样。
- 用户之间互相隔离:读、改、删另一个用户的记忆 ID 返回「未找到」;write 的 request id 也只属于提交它的用户。
- 用户之间互相隔离:读、改、删另一个用户的记忆 ID 返回「未找到」(HTTP 404 `memory_not_found`,
与不存在的 ID 完全一样);write 的 request id 也只属于提交它的用户。
- `memories delete <id> --yes` 遇到「未找到」时按已删除处理:提示「未找到记忆 <id>(可能已删除)」,
退出 0;`--json` 为 `state: "committed"` 加 `already_gone: true`。App 里重复点删除也不再报错。
- 读接口(`memories list`、`profile`、`recall` 等背后的 HTTP 响应)都带 `user_id`,即实际读的是哪个用户。
### 画像版本(`profile --versions`)
@@ -177,7 +187,7 @@ node apps/cli/dist/one-memory.cjs settings features reset
```
- 全局设置,不分用户;没设置的功能保持 SDK 默认值,什么都不设时与以前完全一样(App 不变)。
- 可设:`graph_rerank`(图谱重排,仅 ultra)、`same_series_timeline`(需 graph_rerank)、`episode_recall`(片段召回,pro 即可)、`session_episodes`(片段写入)、`episode_summary`(片段摘要,每个片段一次模型调用)、`intention`(默认开)、`few_shot`。
- 可设:`graph_rerank`(图谱重排,仅 ultra)、`same_series_timeline`(需 graph_rerank)、`episode_recall`(片段召回,pro 即可)、`session_episodes`(片段写入)、`episode_summary`(片段摘要,每个片段一次模型调用)、`intention`(默认开)、`intention_dedup`(意图判重,默认开;关闭后检索和写入都不判重、不删除意图)、`few_shot`;以及数值 `intention_dedup_threshold`(意图判重的余弦阈值,默认 0.95,可设 0.8–1.0,例如 `settings features set intention_dedup_threshold=0.9 --yes`)。
- 片段召回只对带片段的记忆有用:同时开 `session_episodes`,之后提取的记忆才按会话归入片段。
- 在当前模式下不生效的功能照样保存,并注明原因(如 `graph_rerank` 在 pro 下「需要 ultra」)。
- 修改会重建 SDK 客户端:正在写的一批按旧设置写完,之后的写入和检索用新设置;不迁移数据,不改集合名。
@@ -5,6 +5,11 @@ Post-cutover hosts use SDK-native Context v2 at
`POST /api/v2/context/search`. A v2 response is never converted into v1 by
inventing revisions, Evidence IDs, paths, or adoption state.
Since MemHub 1.0.6 the v2 response also carries `user_id`, the user whose
memories were searched (`local` unless the request named one). The Host
Bridge validator accepts it as an optional field, so a response from an older
Core without it still validates; any other unknown field still fails closed.
## Transport
- The Broker `/context` route (used by host hooks and `recall-cli --context`;
+73 -17
View File
@@ -32,6 +32,7 @@ prompt and no extra call.
| `session_episodes` | `recall.session_episode_enabled` (`MEMORY_SESSION_EPISODE_ENABLED`) | off | pro or ultra | none |
| `episode_summary` | `recall.episode_summary_enabled` (`MEMORY_EPISODE_SUMMARY_ENABLED`) | off | pro or ultra | LLM: one call per closed episode of 3+ memories |
| `intention` | `llm.enable_intention` (`MEMORY_ENABLE_INTENTION`) | **on** | pro or ultra | none (off shortens the prompt) |
| `intention_dedup` | `recall.intention_dedup_enabled` (`MEMORY_INTENTION_DEDUP_ENABLED`) | **on** | pro or ultra | none |
| `few_shot` | `llm.few_shot_enabled` (`MEMORY_FEW_SHOT_ENABLED`) | off | pro or ultra | tokens (extraction and reconcile) |
- **Graph rerank (usage A).** It propagates support over the graph edges
@@ -52,8 +53,32 @@ prompt and no extra call.
write half of episodes, and both are off by default. Usage B only helps once
episodes are being written. The SDK's session narrative (L3, `finalize_session`)
is a separate feature and is not exposed (section 2).
- **`intention`** is the only feature that is on by default. Turning it off
stops L7 intention memories from being extracted.
- **`intention`** is on by default. Turning it off stops L7 intention
memories from being extracted.
- **`intention_dedup`** (hy-memory 2.0.12) is on by default, which is what
every earlier version did. A search treats intentions of one user and agent
with the same `memory_at` day, the same `valid_until` day and a cosine
similarity at or above `intention_dedup_threshold` as copies: it returns the
earliest and deletes the others from the store. Off, no intention dedup
runs, at read time or within a write batch, and nothing is hidden, dropped
or deleted.
### Parameters
Numeric settings beside the features. Each tunes one feature and acts only
while that feature acts.
| Name | SDK field (env) | Default | Range | Tunes |
|---|---|---|---|---|
| `intention_dedup_threshold` | `recall.intention_dedup_threshold` (`MEMORY_INTENTION_DEDUP_THRESHOLD`, else `MEMORY_DEDUP_THRESHOLD`) | 0.95 | 0.8–1.0 inclusive | `intention_dedup` (read time only) |
Why 0.8–1.0: with dedup on, every intention at or above the threshold is
deleted from the store at read time, and that cannot be undone. Below about
0.8 the embedding models MemHub ships or connects score different plans on
one topic as that close, so a lower value would delete intentions that are
not copies. Cosine similarity cannot exceed 1.0, and 1.0 keeps dedup for
exact copies only. Someone who wants no dedup turns `intention_dedup` off
rather than picking a threshold.
### Left out, and why
@@ -81,11 +106,12 @@ prompt and no extra call.
holds only the features they set:
```json
{"memory_mode": "pro", "memory_features": {"episode_recall": true, "session_episodes": true}}
{"memory_mode": "pro", "memory_features": {"episode_recall": true, "intention_dedup_threshold": 0.9}}
```
The values are strict booleans and unknown keys are refused, so a hand-edited
file with an unknown feature fails to load (`config_invalid`), like any other
Feature values are strict booleans, a parameter is a number in its range, and
unknown keys are refused. So a hand-edited file with an unknown feature or an
out-of-range threshold fails to load (`config_invalid`), like any other
malformed setting.
## 3. How a setting reaches the SDK
@@ -104,7 +130,8 @@ malformed setting.
- Each client's features are fixed when it is built, like its mode. Both the
write handles (`SdkWriteCoordinator`) and the read handle
(`ActiveProfileReadHandle`) are keyed by (profile, revision, mode,
features).
features). The features part of the key is the sorted (name, value) pairs
a person set, parameters included, so a new threshold is a new key.
- A change runs `retire_stale_handles`, the #218 use count. An idle write
handle closes at once, and a write in flight finishes with the features it
started with. The next read builds a client with the new key. The old read
@@ -135,6 +162,22 @@ malformed setting.
"description": "Usage A: propagate support over knowledge-graph edges ..."
}
],
"parameters": [
{
"name": "intention_dedup_threshold",
"value": 0.95,
"configured": null,
"default": 0.95,
"minimum": 0.8,
"maximum": 1.0,
"active": true,
"reason": null,
"requires_feature": "intention_dedup",
"sdk_field": "recall.intention_dedup_threshold",
"env": "MEMORY_INTENTION_DEDUP_THRESHOLD",
"description": "Cosine similarity at or above which ..."
}
],
"non_default": {"graph_rerank": true}
}
```
@@ -143,27 +186,32 @@ malformed setting.
- `configured` is null when unset.
- `default` is the SDK default: its code default, or its own env var when set.
- `active` is false with a `reason` when the feature does nothing right now.
The reason is one of `disabled`, `requires_pro` (lite), `requires_ultra`, or
`requires_graph_rerank` (for `same_series_timeline`).
The reason is one of `disabled`, `requires_pro` (lite), `requires_ultra`,
`requires_graph_rerank` (for `same_series_timeline`), or
`requires_intention_dedup` (for the threshold while `intention_dedup` is off).
- `cost` is one of `none`, `extra_query`, `llm_call`, `prompt_tokens`.
- `features` always lists every feature, in the table's order.
- `parameters` lists every parameter. `value` is what clients are built with;
`minimum` and `maximum` are the range a `PUT` accepts. A parameter is
active exactly when the feature it tunes is.
## 5. `PUT /api/v1/settings/memory-features`
The body has any of the feature names as keys:
- `true` or `false` sets that feature.
- A JSON number sets a parameter, if it is in range (inclusive).
- `null` resets it to the default.
- An absent key is left as it is.
- Anything else answers `422`, including a non-boolean, `"true"`, `1`, or an
unknown name.
- Anything else answers `422`, including a non-boolean feature, `"true"`, `1`,
an unknown name, or a threshold outside 0.8–1.0, a string, or a boolean.
```json
{"graph_rerank": true, "episode_recall": true, "few_shot": null}
{"graph_rerank": true, "intention_dedup_threshold": 0.9, "few_shot": null}
```
The answer is the state of section 4, plus `changed`: the features whose
effective value changed, in the table's order. When the set of configured
The answer is the state of section 4, plus `changed`: the features and then
the parameters whose effective value changed, in the tables' order. When the set of configured
features changes, the SDK clients are rebuilt (section 3). That includes a
reset to a value equal to the default, after which the client carries no
override at all.
@@ -172,10 +220,11 @@ override at all.
`pipeline` in `GET /api/v1/memory-processing` (memory-mode contract §7) gains
`features`, which lists the features whose effective value differs from the
default. It is `{}` while every feature is at its default:
default, and the parameters (as numbers) likewise. It is `{}` while every
setting is at its default:
```json
{"pipeline": {"mode": "pro", "...": "...", "features": {"episode_recall": true, "intention": false}}}
{"pipeline": {"mode": "pro", "...": "...", "features": {"episode_recall": true, "intention_dedup_threshold": 0.9}}}
```
`memhub status --json` and `memhub progress --json` carry it as
@@ -188,14 +237,19 @@ to the memory-mode line.
memhub settings features # list them
memhub settings features set episode_recall=true session_episodes=true # plan only
memhub settings features set episode_recall=true session_episodes=true --yes
memhub settings features set intention_dedup=false --yes # no intention dedup
memhub settings features set intention_dedup_threshold=0.9 --yes # a looser dedup
memhub settings features reset few_shot # back to the default
memhub settings features reset # every set feature
```
`set` prints its plan and needs `--yes`, like `settings mode`, because some
features spend model calls. `reset` only returns features to their defaults,
so it applies at once. Unknown names or values other than `true` / `false` are
a usage error (exit 2) before the Runtime is asked.
so it applies at once. Unknown names, feature values other than `true` /
`false`, and a threshold that is not a plain decimal in 0.8–1.0 are a usage
error (exit 2) before the Runtime is asked. Against an older Runtime that does
not list the threshold, `set` answers `rejected` /
`memory_feature_unsupported` without sending it.
## 8. Rules kept
@@ -216,6 +270,8 @@ a usage error (exit 2) before the Runtime is asked.
| session_episodes | 片段写入:同一会话中提取的记忆归入同一片段。 | Episodes per conversation: memories extracted from one conversation share one episode. | Episodios por conversación: las memorias extraídas de una conversación comparten un episodio. |
| episode_summary | 片段摘要:片段结束时总结一次(每个片段一次模型调用)。 | Episode summaries: each finished episode is summarized once (one model call per episode). | Resúmenes de episodio: cada episodio terminado se resume una vez (una llamada al modelo por episodio). |
| intention | 意图提取:随事实一起记住你提到的计划和意图。 | Intentions: remembers the plans and intentions you mention, along with facts. | Intenciones: recuerda los planes e intenciones que mencionas, junto con los hechos. |
| intention_dedup | 意图去重:同一天里几乎相同的意图只保留最早一条,其余删除。 | Intention dedup: of near-identical intentions from one day, keeps the earliest and deletes the rest. | Deduplicación de intenciones: de las intenciones casi idénticas de un mismo día, conserva la más antigua y borra las demás. |
| intention_dedup_threshold | 去重阈值:越高,越少的意图被视为重复。 | Dedup threshold: the higher it is, the fewer intentions count as copies. | Umbral de deduplicación: cuanto más alto, menos intenciones cuentan como copias. |
| few_shot | 提取示例:在提取提示中加入示例,可帮助较弱的模型(输入更长)。 | Extraction examples: adds examples to the extraction prompt, which can help a weaker model (longer input). | Ejemplos de extracción: añade ejemplos a la instrucción de extracción, lo que puede ayudar a un modelo más débil (entrada más larga). |
| default | 默认 | Default | Predeterminado |
| requires_ultra | 当前模式下不生效,需要 ultra。 | Has no effect in this mode; needs ultra. | No tiene efecto en este modo; requiere ultra. |
@@ -20,6 +20,20 @@ cursor。旧文件后续追加也必须能被再次发现,不能只有一个
用户授权是读取前置条件。环境变量只用于发现候选根,不能代替授权。每个候选
文件必须在打开时再次验证 realpath 归属、symlink、普通文件类型和大小预算。
多用户(仅 Hermes):一个授权根属于一个 `user_id`。Hermes 的扫描、检测和
持续同步共用同一次遍历,**任何层级**名为 `profiles` 的目录都不进入(发现
helper 的 `excluded_directory_names`),所以授权根即使在 `~/.hermes` 之上,
也不会读到 `.hermes/profiles/<name>`。接入时 Core 拒绝:
- 根的最后一段是 `profiles`(profiles 容器本身):422
`hermes_root_is_profiles_container`;
- 根(解析 symlink 后)与另一个用户未撤销的 Hermes 授权根相同、包含它或在
它之内:409 `hermes_root_overlaps_other_user`。唯一例外是一方正好是另一方
的 `profiles/<name>`(`local` 在 `~/.hermes`、另一个用户在
`~/.hermes/profiles/<name>`),因为上面的排除已把两者隔开。
本机 `local` 的接入(`/sources/connect`、`/sources/{id}/authorize`)也走同一检查。
## Imported source record
Adapter 输出的 `ScannedEvidence` 是兼容命名的导入中间表示。原文持久化与 raw
+1 -1
View File
@@ -40,7 +40,7 @@ Preview 不会弹出给用户;开关点击本身被视为确认。关闭时通
| [Qwen Code](qwen-code.md) | 开放 | MCP + Skill | 无自动 Hook | `$QWEN_HOME/projects` 扫描 |
| [Pi](pi.md) | 开放 | CLI Skill,不使用 MCP | JS Capture Extension | Pi session 扫描 + Extension 实时入口 |
| [OpenClaw](openclaw.md) | 正式产品冻结 | 当前没有;仅有未开放的底层 MCP 写入器 | 没有 Hook、Plugin 或 Prefetch | Adapter 也被冻结,开关不可用 |
| [DeepSeek Harness](deepseek-harness.md) | 详情页排除,来源管理页按 Profile 接入 | Profile 级 MCP Patch | 无自动 Hook | `$DSH_HOME/sessions` 扫描 |
| [DeepSeek Harness](deepseek-harness.md) | 开放;一个开关覆盖全部已检测 Profile | Profile 级 MCP Patch | 无自动 Hook | `$DSH_HOME/sessions` 扫描 |
## 同步开关的统一精确语义
+37 -26
View File
@@ -1,28 +1,45 @@
# DeepSeek Harness 一键插入支持
更新日期:2026-10-18
更新日期:2026-11-02
## 结论
DeepSeek Harness 的 Profile 级 MCP 安装器已经实现,Profile Scoped Host ID(`deepseek-harness:<profile>`)在权限层按不透明标签处理,可正常走完 Preview 与安装。
DeepSeek Harness 按 Profile 配置 MCP:每个 Profile 是一个独立 Host(`deepseek-harness:<profile>`),有自己的 `cordis.patch.yml`、Credential 与 Grant。应用详情页只有一个“一键插入应用”开关,它一次覆盖桌面检测到的全部 Profile:打开时逐个安装并授权,关闭时逐个移除并撤销。
## 接入方式和产品状态
| 项目 | DeepSeek Harness 当前实现 |
|---|---|
| 应用详情页一键插入 | 有意排除,因为无法选择 Profile |
| 来源管理页 Profile 安装 | 可用;每个 Profile 独立 Patch、Credential 与 Grant |
| 应用详情页一键插入 | 可用;一个开关覆盖全部已检测 Profile,并列出每个 Profile 的状态 |
| 首次导入批量连接 | 排除;在应用详情页打开开关 |
| Harness 查询 MemHub | 通过 Profile 级 Cordis MCP Client |
| Skill | 没有 |
| Prompt 前自动 Prefetch | 没有 |
| Capture Hook/Plugin | 没有 |
| Harness 新会话进入 MemHub | Core 扫描 `$DSH_HOME/sessions` |
| Harness 新会话进入 MemHub | Core 扫描 `$DSH_HOME/sessions`,按 `session[.vN].jsonl[.zstd]` 识别会话文件 |
会话文件按写入时的格式版本命名:首版为 `session.jsonl` / `session.jsonl.zstd`,v3/v4 起为 `session.v3.jsonl.zstd` / `session.v4.jsonl.zstd`。来源扫描接受带版本号的名称;解析前再读文件头 `version` 字段,只导入已验证的 v0/v3/v4,未来未知版本不会被解析:整个文件以 `unsupported_schema` 报告,且不留下读取进度,将来支持该版本的发布会从头读取(错误导入比暂时漏导更难回滚)。`tool/result` 的输出无论写成 `tool-result` 块还是纯 `text` 块,都按同一条裁剪与凭据清洗路径导入。
## 两个开关分别做什么
### “一键插入应用”
### 打开“一键插入应用”
应用详情页不可用。来源管理页对每个 Profile 写 `cordis.patch.yml`、创建独立 Credential 并授权;Host ID 形如 `deepseek-harness:<profile>`,冒号只作为标签存在,权限层的安装目录按 Host ID 摘要生成,不参与路径拼接。
应用详情页对每个已检测、尚未获批的 Profile:
1. 预览并写入该 Profile 的 `cordis.patch.yml` 受管块,创建独立 Credential;
2. 授予 `om_recall`、`om_conversations`、`om_diagnostic_trace`,范围为 `all-apps` 与 `source:deepseek-harness`(与其他应用的一键插入相同)。
已获批的 Profile 不会重装。某个 Profile 失败时,其他 Profile 的结果保留,页面提示失败的 Profile 名称;开关只要有一个 Profile 获批就显示为开。Host ID 里的冒号只作为标签存在,权限层的安装目录按 Host ID 摘要生成,不参与路径拼接。
### 关闭“一键插入应用”
对每个已接入的 Profile 先撤销 Credential,再删除受管块(与其他应用的关闭语义相同)。受管块被改过、无法删除时仍会撤销授权,页面列出需要核对的 Profile;撤销也失败的 Profile 会单独列出,它仍可使用记忆。从未接入的 Profile 不受影响。
### 之后新建的 Profile
桌面每次刷新应用接入状态时都会重新枚举 Profile(`$DSH_HOME/profiles/<profile>/cordis.yml`),应用首页打开时每 3 秒刷新一次;常驻的 Runtime 收到未知的 Profile Host ID 时也会重新读盘。开关的选择会被记住:开关为开时,新出现的 Profile 由应用首页自动安装并授权,不需要重启 MemHub。
每个新 Profile 触发一次自动尝试;失败时应用首页会指明 Profile,不会每次轮询都重试,下次启动或点击重试时再试一次。开关为关时不会自动接入任何 Profile。
### 打开“同步开关”
@@ -32,12 +49,10 @@ DeepSeek Harness 的 Profile 级 MCP 安装器已经实现,Profile Scoped Host
Core 停止扫描和 pending Job claim,running Job可结束,已有 Memory 保留。它不会修改任何 Profile 的 `cordis.patch.yml` 或 Credential。
### 修复一键安装后的组合语义
### 组合语义
未来修复 Profile Host ID 后:
- 某个 Profile 的一键插入只决定该 Profile 是否能查询 MemHub;
- 全局同步开关决定 MemHub 是否继续读取整个 DeepSeek Harness 的 Session;
- 一键插入决定已检测 Profile 是否能查询 MemHub;
- 同步开关决定 MemHub 是否继续读取整个 DeepSeek Harness 的 Session;
- 没有自动 Prefetch,也没有 Capture Hook。
## Profile 粒度
@@ -58,7 +73,7 @@ deepseek-harness:headless
...
```
理论上每个 Profile 拥有独立 Patch、Journal、Credential 和 Grant;Core 会话同步仍按统一的 `deepseek-harness` 来源扫描 `$DSH_HOME/sessions`。
每个 Profile 拥有独立 Patch、Journal、Credential 和 Grant;Core 会话同步仍按统一的 `deepseek-harness` 来源扫描 `$DSH_HOME/sessions`。
## MCP Patch
@@ -85,13 +100,15 @@ DeepSeek Harness 不安装 Skill,也不安装 Capture Hook;新会话进入 M
## UI 状态
- 应用详情页排除 DeepSeek Harness,因为单一来源页无法选择 Profile。
- 首次导入批量连接也排除它。
- 来源管理页会列出每个 Profile,并显示独立安装按钮。
- 应用详情页的“一键插入应用”下方列出每个已检测 Profile 及其“已接入 / 未接入”状态;没有检测到 Profile 时开关不可用,并提示创建 Profile 后回到此页。
- Host ID 只接受小写字母、数字和连字符(最长 32 个字符)。名称不符合的 Profile 仍会列出,但无法授权,页面会指明它和原因,不影响其他 Profile。
- 连接状态取各已接入 Profile 记录的自检结果:全部接通才显示“已接通”。
- 首次导入批量连接排除 DeepSeek Harness。
- CLI 的 `memhub host install deepseek-harness:<profile>` 只安装,不授权(状态停在 `install-only`);授权入口只有应用详情页的开关,CLI 没有授权命令。
## 已修复:Profile Host ID 校验
此前来源管理页安装会在写入 `cordis.patch.yml` 和创建 Credential 之前失败,原因是:
此前 Profile 安装会在写入 `cordis.patch.yml` 和创建 Credential 之前失败,原因是:
```text
系统生成 hostId = deepseek-harness:<profile>
@@ -101,18 +118,12 @@ HostGrantStore 也不接受冒号
Host ID 是不可解释的标签:安装目录对它取摘要,不会把它拼进路径。现在权限层与 Grant 存储都接受一个可选的 `:<profile>` 段,因此 Profile 安装可以正常走完。这不是 DeepSeek Harness 不支持 MCP 的问题,而是 Profile Host ID 格式与权限校验不一致,现已对齐。
## 修复后的权限语义风险
## 已修复:应用详情页返回 `unsupported_host`
来源管理页当前默认授权选择可能包含:
- `om_recall`、`om_conversations`、`om_diagnostic_trace`(`om_ask` 已随 `MemHub_ask` 移除,旧授权仍可读取但不再授权任何工具);
- `all-apps`;
- 所有已授权 Source 和 Workspace。
这比应用详情页通用授权更宽。Host ID 修复后仍应确认是否希望每个 Profile 默认获得如此宽的权限。
此前应用详情页按“一个来源对应一个 Host”接入,而 DeepSeek Harness 没有不带 Profile 的 Host,开关因此置灰并返回 `unsupported_host`,安装后的 Profile 也只能停在 `install-only`(#230)。现在开关直接作用于全部 `deepseek-harness:<profile>` Host,复用与其他应用相同的安装、授权、撤销与卸载接口。
## 重复安装与卸载
底层 Patch 安装是幂等的:已有受管块时替换,不追加第二份。相同 Row ID 出现在 MemHub 哨兵块外时默认拒绝接管。
正常卸载设计是先撤销该 Profile Credential,再删除受管块并保留其他 Patch;如果删完没有其他 Row,恢复 `[]`。用户修改过受管块时不强删。
卸载先撤销该 Profile Credential,再删除受管块并保留其他 Patch;如果删完没有其他 Row,恢复 `[]`。用户修改过受管块时不强删。
+3
View File
@@ -118,6 +118,9 @@ export interface SdkContextSearchResponse {
used_characters: number;
truncated: boolean;
total_ms: number;
/** Whose memories were searched; `local` unless the request named a user.
* Absent from a Core older than 1.0.6. */
user_id?: string;
}
/** Native Host Bridge wire response. Schemas stay distinct and are never adapted. */
+31 -1
View File
@@ -425,6 +425,8 @@ export interface HomeSummary {
export interface SdkHomeSummary {
schema_version: 2;
/** The user the card was read for (`local` unless one was named). */
user_id?: string;
memory_count: number;
latest: SdkMemoryRecord | null;
observed_at: string;
@@ -605,6 +607,8 @@ export interface SdkMemoryRecord {
}
export interface SdkMemoryPage {
/** `GET /api/v2/memory-library`: the user the page was read for. */
user_id?: string;
memories: SdkMemoryRecord[];
total: number;
limit: number;
@@ -621,6 +625,8 @@ export interface SdkMemoryPage {
}
export interface SdkUserProfileResponse {
/** The user whose profile this is (`local` unless one was named). */
user_id?: string;
request_id: string;
profile: Record<string, unknown> | null;
elapsed_ms: number;
@@ -649,6 +655,8 @@ export interface SdkRawRecord {
}
export interface SdkRawPage {
/** The user whose raw turns these are. */
user_id?: string;
items: SdkRawRecord[];
total: number;
limit: number;
@@ -659,6 +667,8 @@ export interface SdkRawPage {
/** `GET /api/v2/memory-library/layers`: memories per SDK layer. */
export interface SdkLayerCounts {
/** The user whose memories were counted. */
user_id?: string;
layers: Record<string, number>;
total: number;
statuses: string[] | null;
@@ -763,7 +773,8 @@ export type AskMemoryEvent =
| { type: "error"; code: string };
export type SdkAskMemoryEvent =
| { type: "recall"; memories: SdkAskMemory[] }
/** `user_id`: whose memories answer (Core 1.0.6+). */
| { type: "recall"; memories: SdkAskMemory[]; user_id?: string }
| { type: "delta"; text: string }
| {
type: "final";
@@ -833,6 +844,8 @@ export interface RecallReceipt {
fallback_reason?: "model_unavailable" | "no_memories" | null;
/** Sources the caller was not allowed to read, left out of this result. */
excluded_sources?: string[];
/** Whose memories were searched (`local` unless the request named a user). */
user_id?: string;
}
export interface LiveSyncHealth {
@@ -874,6 +887,8 @@ export interface MemoryGraphEdge {
}
export interface MemoryGraphResponse {
/** The user whose memories were mapped. */
user_id?: string;
total_memories: number;
projected_memories: number;
nodes: MemoryGraphNode[];
@@ -939,6 +954,15 @@ export interface MemoryProcessingSummary {
* Workers own a task from the moment they take it until they finish, fail or
* give up, so unlike `running` this cannot include work nobody is doing. */
in_flight_count: number;
/** Jobs held back because their source is off or its grant was revoked.
* Not part of `pending`: they run again, unchanged, once the source is on.
* Optional so older producers stay valid. */
suspended?: number;
/** Until when model work is paused because the provider kept failing
* (bad key, wrong URL, missing model, provider down); null when not. */
llm_backoff_until?: string | null;
/** The provider failure behind the pause, as a stable code. */
llm_backoff_error?: string | null;
metrics?: {
raw_evidence: number;
complete_turns: number;
@@ -1110,6 +1134,12 @@ export interface MemoryAddReceipt {
task_id: string | null;
memory_ids: string[];
error: string | null;
/** Stable code of the last failure while the addition still waits (for
* example `model_auth_failed`); null when none. Optional so older
* Runtimes stay valid. */
error_code?: string | null;
/** When Core will try this addition again (ISO); null when not scheduled. */
next_retry_at?: string | null;
}
export type {
@@ -70,6 +70,7 @@ export type BrokerClientErrorCode =
| "memory_add_conflict"
| "memory_add_invalid"
| "broker_request_failed"
| "broker_timeout"
| "broker_unavailable";
const CLIENT_MESSAGES: Record<BrokerClientErrorCode, string> = {
@@ -100,6 +101,10 @@ const CLIENT_MESSAGES: Record<BrokerClientErrorCode, string> = {
// `broker_unavailable` now means only what it says: nothing answered.
broker_request_failed:
"MemHub is running, but this request failed. Check MemHub for the reason.",
// A request that got no answer in time is not a MemHub that is down: the
// broker or Core accepted it and is still busy (#231). Telling the user to
// start MemHub sent them to restart a healthy Runtime.
broker_timeout: "MemHub is running, but did not respond in time.",
broker_unavailable: "MemHub is not running or its broker is unavailable",
};
@@ -318,12 +323,13 @@ export async function recallThroughBroker(
discoveryPath: string,
request: RecallRequest,
options?: BrokerClientOptions,
timeout = 20_000,
): Promise<RecallReceipt> {
return postThroughBroker(
discoveryPath,
"recall",
{ ...request, timeout_ms: request.timeout_ms ?? HOST_CONTEXT_TIMEOUT_MS },
20_000,
timeout,
options,
);
}
@@ -461,6 +467,8 @@ async function postThroughBroker<T>(
: "host_authorization_required",
);
}
if (error.response?.status === 504 || isTimeout(error))
throw new BrokerClientError("broker_timeout");
// Anything the broker actually answered means it is running, however
// badly the request went. Only a failure to reach it says otherwise.
if (error.response) throw new BrokerClientError("broker_request_failed");
@@ -470,6 +478,11 @@ async function postThroughBroker<T>(
}
}
/** A connection that was accepted and then not answered in time. */
function isTimeout(error: { code?: string }): boolean {
return error.code === "ECONNABORTED" || error.code === "ETIMEDOUT";
}
/** Late credential/discovery completion cannot dispatch after a cancelled wait. */
function abortable<T>(promise: Promise<T>, signal?: AbortSignal): Promise<T> {
if (!signal) return promise;
Loaded 100 of 136 files, more files were not shown because too many files have changed in this diff. Show more