Compare commits

...
16 Commits
Author SHA1 Message Date
alvinfeiandClaude Opus 5.5 302fd30af1 feat(sdk): re-embed vector stores and the graph into a new embedding space
Add hy_memory.reembed for switching the embedding model while writes are
paused. Source and target are named explicitly; a space's identity is the
model name plus the width.

- vector_space_names, plan_reembed, reembed_vector_space (resumable,
  cancellable at batch boundaries, widths validated before any upsert, target
  wins on existing ids, source never written), verify_reembed,
  drop_vector_space.
- scan_vector_spaces and diff_spaces for MemHub's startup reconcile of stray
  spaces.
- reembed_graph builds a new Kuzu DB at the target width, copies every node
  and edge, recomputes Memory/beh/Topic/HierarchyTopic embeddings and rebuilds
  the HNSW index; the source graph is opened read-only. verify_graph_reembed
  and drop_graph.

EmbedService.embed_batch now chunks to embedder.batch_size (default 64),
honours Retry-After on 429, takes a per-call max_retries, and rejects vectors
whose width differs from embedding_dims.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:33:52 +08:00
feitianxiang f5ea0af75d Merge pull request #251 from Tencent-Hunyuan/fix/hermes-injected-content
feat(memhub): land #250 on main — multi-user Runtime, hy-memory 2.0.9
2026-11-01 14:39:43 +08:00
alvinfeiandClaude Opus 5.5 1c2f48579f Merge main into fix/hermes-injected-content to land #250
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 14:33:00 +08:00
feitianxiang d29615abeb Merge pull request #250 from Tencent-Hunyuan/feat/memhub-multi-user
feat(memhub): multi-user Runtime — user_id on memory APIs and CLI, per-user Hermes profiles, hy-memory 2.0.9
2026-11-01 14:25:05 +08:00
feitianxiang 0965519e16 Merge pull request #249 from Tencent-Hunyuan/fix/hermes-injected-content
fix(memhub): drop Hermes gateway notices and unwrap relayed messages on import; no false 'damaged log' warning
2026-11-01 14:24:05 +08:00
alvinfeiandClaude Opus 5.5 7e9f0d98d8 fix(memhub): another user's memory id answers like a missing one
DELETE of another user's id answered 404 while a missing id answered 204,
so comparing the two told that the id existed. It now answers 204 and
deletes nothing; GET and PUT answer the same 404 as a missing id, and
receipt and addition reads already matched. A test compares every by-id
route's response for another user's id with a nonexistent id's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:23:10 +08:00
alvinfeiandClaude Opus 5.5 0ff4538f36 feat(memhub-cli): --user-id for memory commands, connect a Hermes profile
- Global --user-id <id> (default local) for memories list/show/add/
  add-status/update/delete, recall, profile and write submit/status; sent
  only when it is not the default, so default requests are unchanged.
- memhub sources connect hermes --user-id <id> --root <path> [--import]
  grants one Hermes profile to its own user; refused for other sources.
- memhub sources list shows the user of every grant (from
  GET /api/v1/source-grants, skipped on an older Core).
- Help catalog, contract tests and docs/cli/EXPERIENCE.md updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:12:31 +08:00
alvinfeiandClaude Opus 5.5 a9709788d8 feat(memhub): multi-user Core — user_id on memory APIs, per-user Hermes grants
One Runtime now serves many user_ids. A user is the root that is read,
never a session or platform field; the App never sends user_id and keeps
reading and writing `local`, and only Hermes may hold another user's grant.

- R1: every memory read/write route takes an optional user_id (query for
  GET/DELETE, body for POST/PUT), default `local`, pattern
  ^[A-Za-z0-9._-]{1,64}$ (no `:`).
- R2: user_id reaches the SDK: Evidence, durable payload and client.add
  carry it; reads filter by user. get/update/delete/history and
  get_write_outcome take no user in hy-memory 2.0.9, so Core reads the
  memory first and answers 404 for another user's id; memory-write keys
  are namespaced per user and addition receipts are keyed by the user's
  Evidence id.
- R3: grants, cursors, jobs, live sync and the claim gate work per
  (user_id, source_id) through lane keys (`hermes` for local,
  `hermes@wife` otherwise). POST /api/v1/sources/connect takes user_id and
  root (Hermes only, 422 otherwise); revoke takes ?user_id; new
  GET /api/v1/source-grants lists every user's grants. Another user's
  grant syncs while it stands; the App's per-source switch stays local's.
- R4: a Hermes root scan never enters profiles/** (discovery, history and
  live sync share the same walk).
- R5: session keys, assembling rows, FIFO source keys and request ids are
  per user, so two users' turns never share a batch.
- R6: memory routes refuse undeclared query parameters and body fields
  with 422.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:12:31 +08:00
alvinfeiandClaude Opus 5.5 c1e5916b51 build(memhub): pin hy-memory 2.0.9
Core pins and locks the published hy-memory 2.0.9 from PyPI; the
version constant, README, architecture notes, SBOM, notices and their
tests follow. CHANGELOG records 2.0.9 as published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 09:12:21 +08:00
alvinfeiandClaude Opus 5.5 97de5480bd Merge github/main (hy-memory 2.0.9 release) into feat/memhub-multi-user
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 09:11:25 +08:00
feitianxiang 97d9309ca8 Merge pull request #248 from Tencent-Hunyuan/feat/sdk-2.0.9
fix(sdk): placeholder values never overwrite the basic profile (2.0.9)
2026-11-01 08:26:49 +08:00
alvinfeiandClaude Opus 5.5 b1b3e0a291 fix(sdk): drop the user-turn filter for basic_info
Owner decision on #248: the writer no longer filters basic_info against
the user turns. The placeholder guard (with healing on read) and the
prompt hard rules stay; the guard alone keeps an assistant quoting
"name is 用户" out of the profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 08:18:32 +08:00
alvinfeiandClaude Opus 5.5 53507c7abe fix(memhub): only call non-JSON diagnostic lines damaged in memhub logs
After an upgrade the diagnostic files still hold an older build's records.
They are valid JSON that the reader leaves out, but were counted as damaged,
so `memhub logs` warned "跳过 N 行损坏的诊断记录" and a support export
failed with diagnostic_export_invalid_lines. Count them apart as
`unrecognized_lines` (reported only in the JSON empty-state record) and
keep the warning for lines that are not JSON.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 05:10:19 +08:00
alvinfeiandClaude Opus 5.5 5a1d252787 fix(memhub): drop Hermes gateway notices and unwrap relayed messages on import
Hermes stores its gateway's own posts in `user` rows: finished background
process notices (often with the gateway's "not written by a human" note),
voice placeholders that only name an audio file, and "Reply with exactly:
PONG" probes. Relayed Telegram messages arrive wrapped in out-of-band
markers and a gateway-origin JSON block. All of it reached extraction as
the person's words.

The Hermes adapter now cleans `user` rows (state.db and session
transcripts) before evidence is built, so the content hash and dedup see
the cleaned text:
- skipped with new reason codes `injected_machine_notice`,
  `voice_placeholder` and `probe` (media-only rows as `no_text`), counted
  in `import.parse.issue_summary` like other parse skips;
- out-of-band markers, the gateway-origin block and the interrupted
  response label are removed, keeping only the person's message.

Assistant and tool rows are unchanged. Tests use real rows with the
Telegram ids and home paths replaced by fake values.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 05:10:19 +08:00
alvinfeiandClaude Opus 5.5 8895fea775 release: hy-memory 2.0.9
Placeholder values no longer overwrite the user's basic profile (L0). Not
yet published to PyPI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
alvinfeiandClaude Opus 5.5 78882f3300 fix(sdk): placeholder values never overwrite the basic profile
A weak extraction model filled basic_info with "用户", "The user" or "N/A"
when the user stated nothing, and each placeholder superseded the real L0
value. upsert_basic_profile now drops placeholder values and writes no new
version when nothing is left; profile assembly skips stored placeholders.
The writer keeps a basic_info field only when its value appears in a user
message of the request. The zh and en extraction prompts add a hard rule
and a small-talk counter-example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
79 changed files with 6241 additions and 230 deletions

No files matched your search

+1
View File
@@ -40,6 +40,7 @@ MEMORY_EMBEDDING_DIMS=1024
# MEMORY_EMBEDDER_TIMEOUT=15
# MEMORY_EMBEDDER_MAX_RETRIES=2
# MEMORY_EMBEDDER_RETRY_DELAY=0.3
# MEMORY_EMBEDDER_BATCH_SIZE=64
# MEMORY_EMBEDDER_EXTRA_HEADERS={}
# MEMORY_EMBEDDER_EXTRA_BODY={}
+63
View File
@@ -4,6 +4,46 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
- Hy Memory: new `hy_memory.reembed` module for switching the embedding
model. Writers must be paused for the whole run. Nothing is derived from
the provider or endpoint: source and target are named explicitly, and the
identity of a space is the model name plus the width (`space_identity`).
Only Chroma and Kuzu are supported.
- `vector_space_names(config, base_name=None)` returns every physical
collection a client opens for the six vector roles: memory, entities,
coding keys and working facts, each in the working and coding variants.
- `reembed_vector_space(source, target, embedding=...)` pages every raw
source collection, covering all users and all layers including L1_RAW. It
embeds the text the write path embedded (`reembed_text`: the canonical
state's `state_retrieval_text`, a schema's `retrieval_text`, otherwise the
document). It then writes each record into the target with the same id,
document and metadata. Each batch's widths are checked before anything is
written. A cancel stops at a batch boundary (`ReembedCancelled`). A rerun
skips the ids the target already holds, so it resumes, and a merge never
overwrites the target. The source is never written.
- `plan_reembed`, `verify_reembed` and `drop_vector_space` complete the
vector side. `verify_reembed` checks ids, metadata, per-user counts and
the width, and runs a self-recall sample. `drop_vector_space` refuses
collections that a live client in this process holds open.
- `scan_vector_spaces(persist_directory)` is read-only. It lists every
space on disk with its roles, counts, stored width, per-user counts and
the models recorded in metadata. `diff_spaces(source, target)` returns the
ids that the source holds and the target lacks, whatever their status.
- `reembed_graph(source_path, target_path, embedding=..., target_dims=...)`
builds a new Kuzu graph at the new width. It copies every node and edge
and recomputes `Memory.embedding`, `Memory.beh_embedding`, and the Topic
and HierarchyTopic embeddings, then rebuilds the HNSW index. The source
graph is opened read-only and kept for rollback. A run that fails or is
cancelled leaves no target behind, and a rerun starts over.
`verify_graph_reembed` and `drop_graph` complete the graph side.
- Hy Memory: `EmbedService.embed_batch` now splits large batches into
requests of `embedder.batch_size` texts each (`MEMORY_EMBEDDER_BATCH_SIZE`,
default 64); before, it sent them all in one request. On a 429 it waits for
the `Retry-After` time plus jitter, capped at 60 s. It takes a per-call
`max_retries`. A returned vector whose width differs from the configured
`embedding_dims` now raises `EmbeddingDimensionMismatchError` and is not
retried.
## MemHub 1.0.5 — 2026-10-28
Build 36, on `hy-memory==2.0.8`. Linux x64/arm64 Runtime and macOS arm64
@@ -94,6 +134,29 @@ authority on first launch.
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.9 — 2026-11-01
- Published
[`hy-memory==2.0.9`](https://pypi.org/project/hy-memory/2.0.9/).
- Placeholder values no longer overwrite the user's basic profile (L0). A
small extraction model filled `basic_info` with `{"name": "用户"}`,
`The user` or `N/A` when the user had said nothing about themselves, and
each placeholder superseded the real value, so the L0 chain read
费天翔 → Fei → 用户 → The user. Two defences:
- `upsert_basic_profile` drops placeholder values: empty, `用户`,
`the user`, `user`, `n/a`, `na`, `none`, `null`, `unknown`, `未提供`,
`未知`, `不详`, `-`, `?` and the field's own name, compared
case-insensitively with whitespace and punctuation stripped. When nothing
is left it writes no new version and no new L0 head. Placeholders already
stored in the chain no longer override earlier real values when the
profile is assembled, so `get_user_profile` of an affected user shows the
last real value again without any migration. This also stops an
assistant quoting an old memory ("The user's name is 用户.") from being
written back.
- The extraction prompts (zh and en) state the rule: only what the user
said in a `user` turn, never the field name or a placeholder, `{}`
otherwise, with a small-talk counter-example.
## Hy Memory 2.0.8 — 2026-10-28
- Published
+1
View File
@@ -48,6 +48,7 @@
| `MEMORY_EMBEDDER_TIMEOUT` | `15` | 单次超时,秒 |
| `MEMORY_EMBEDDER_MAX_RETRIES` | `2` | 最大重试次数 |
| `MEMORY_EMBEDDER_RETRY_DELAY` | `0.3` | 重试间隔,秒 |
| `MEMORY_EMBEDDER_BATCH_SIZE` | `64` | `embed_batch` 单次请求最多发送的文本条数,超出按此切块 |
| `MEMORY_EMBEDDER_EXTRA_HEADERS` | — | JSON object,附加 HTTP headers |
| `MEMORY_EMBEDDER_EXTRA_BODY` | — | JSON object,附加请求 body |
+53
View File
@@ -55,6 +55,33 @@ except Exception:
from .client import HyMemoryClient
from .inspector import MemoryInspector
from .runtime import SharedRuntime
from .reembed import (
EmbeddingSpaceIdentity,
GraphReembedPlan,
GraphReembedProgress,
GraphReembedReport,
GraphVerification,
ReembedCancelled,
ReembedPlan,
ReembedProgress,
ReembedReport,
ReembedVerification,
ScannedSpace,
SpaceDiff,
VectorSpace,
diff_spaces,
drop_graph,
drop_vector_space,
plan_reembed,
reembed_graph,
reembed_text,
reembed_vector_space,
scan_vector_spaces,
space_identity,
vector_space_names,
verify_graph_reembed,
verify_reembed,
)
from .snapshot import (
DataPlaneRestore,
DataPlaneSnapshot,
@@ -129,6 +156,32 @@ __all__ = [
"DataPlaneSnapshotRejected",
"restore_local_data_plane",
"snapshot_local_data_plane",
# 换 embedding 模型:把向量库和图重新 embed 进新的 embedding 空间
"EmbeddingSpaceIdentity",
"GraphReembedPlan",
"GraphReembedProgress",
"GraphReembedReport",
"GraphVerification",
"ReembedCancelled",
"ReembedPlan",
"ReembedProgress",
"ReembedReport",
"ReembedVerification",
"ScannedSpace",
"SpaceDiff",
"VectorSpace",
"diff_spaces",
"drop_graph",
"drop_vector_space",
"plan_reembed",
"reembed_graph",
"reembed_text",
"reembed_vector_space",
"scan_vector_spaces",
"space_identity",
"vector_space_names",
"verify_graph_reembed",
"verify_reembed",
# 配置
"MemoryConfig",
# 公共扩展协议 / 精确读取类型
+33
View File
@@ -682,6 +682,23 @@ Output:
```
John's occupation and location are HIS attributes, so they go into `memory` with "John" as the subject. They must NEVER land in `basic_info.occupation` / `basic_info.location` — `basic_info` is exclusively the user's own profile. Only `name` (Tim, from the user message's `name` field) belongs there.
## Example 11 — Counter-example: small talk has no basic_info
New Messages:
[{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello! The user's name is The user. How can I help?"}]
Memory Date: 2025-09-03
Output:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
The user stated nothing about themselves. The assistant's line is not a user turn, and "The user" is a placeholder, not a name. Output `{}` — never `{"name": "The user"}` or `{"name": "user"}`.
---
"""
@@ -749,6 +766,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
@@ -817,6 +842,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
+31
View File
@@ -365,6 +365,13 @@ EXTRACT_PROMPT_ZH = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -440,6 +447,13 @@ EXTRACT_PROMPT_ZH_NO_INTENTION = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -669,6 +683,23 @@ Memory Date:2025-09-02
```
职业和所在地是 **John 的**属性,所以进 `memory` 并以"John"作主体,**绝不能**落到 `basic_info` 的 `occupation` / `location` —— `basic_info` 专属于用户本人。只有 `name`(Tim,来自用户消息的 `name` 字段)才该放进去。注意输出语言随输入语言(此例为英文)。
## 示例 11 —— 反例:寒暄没有 basic_info
新消息:
[{"role": "user", "content": "你好"},
{"role": "assistant", "content": "你好!我可以帮你。之前记得 The user's name is 用户."}]
Memory Date:2025-09-03
输出:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
用户没有说任何关于自己的信息。assistant 那句不是 user 回合,"用户" 也是占位符而不是名字。必须输出 `{}`——**绝不能**输出 `{"name": "用户"}`。
---
"""
+77
View File
@@ -26,6 +26,8 @@ from __future__ import annotations
import asyncio
import logging
import re
import unicodedata
import uuid
import weakref
from dataclasses import dataclass
@@ -208,6 +210,76 @@ def _sanitize_kv(
return result
# ================================================================
# 占位符守卫(P0-1)
# ================================================================
# 弱模型(如 qwen3.5-2b)在用户什么都没说时,会把 schema 字段说明里的词当值
# 填进 basic_info:{"name": "用户"} / "The user" / "N/A"。每个占位符都会
# supersede 掉真实值,演化链变成 费天翔 → Fei → 用户 → The user。
# 规则:占位符永远不入库;读侧组装 KV 时占位符也不覆盖已有真实值。
_PLACEHOLDER_VALUES = frozenset({
"用户", "该用户", "本用户", "用户本人",
"the user", "the user's", "user", "a user", "this user",
"n/a", "na", "none", "null", "nil", "unknown", "unspecified",
"not provided", "not specified", "not mentioned",
"未提供", "未知", "不详", "未说明", "未提及", "无",
"-", "?",
})
# 内置字段的中文标签:弱模型也会把字段说明的中文词当值。
_FIELD_LABELS: Dict[str, tuple] = {
"name": ("姓名", "名字", "用户名", "用户姓名", "用户名字"),
"age": ("年龄", "用户年龄"),
"location": ("所在地", "地点", "城市", "用户所在地"),
"occupation": ("职业", "工作", "用户职业"),
"employer": ("雇主", "公司", "用户雇主"),
}
# 去掉标点/符号后再比较,让 "The user." / "用户。" / "N/A" 都命中。
# "-" / "?" 本身就是纯标点,剥完为空串,同样判为占位符。
_PUNCT_RE = re.compile(r"[\s\.,;:!?'\"`~()\[\]{}<>/\\|_\-\u3000-\u303f\uff00-\uff0f\uff1a-\uff20]+")
def _placeholder_key(value: str) -> str:
s = unicodedata.normalize("NFKC", str(value or "")).strip().lower()
s = s.replace("\u2019", "'")
s = _PUNCT_RE.sub(" ", s)
return " ".join(s.split())
_PLACEHOLDER_KEYS = frozenset(_placeholder_key(v) for v in _PLACEHOLDER_VALUES)
def is_placeholder_value(field_name: str, value: Any) -> bool:
"""value 是否为占位符(空 / 用户 / The user / N/A / 字段名本身 等)。
大小写不敏感、首尾空白与标点不敏感。数字(如 age=30)永远不是占位符。
"""
if value is None:
return True
if isinstance(value, bool):
return False
if isinstance(value, (int, float)):
return False
key = _placeholder_key(value)
if not key:
return True
if key in _PLACEHOLDER_KEYS:
return True
field_key = _placeholder_key(field_name)
if field_key and key in (field_key, field_key.replace("_", " ")):
return True
for label in _FIELD_LABELS.get(field_name, ()):
if key == _placeholder_key(label):
return True
return False
def _drop_placeholders(kv: Dict[str, Any]) -> Dict[str, Any]:
return {k: v for k, v in kv.items() if not is_placeholder_value(k, v)}
# ================================================================
# 公共入口:upsert_basic_profile
# ================================================================
@@ -287,6 +359,8 @@ async def _upsert_basic_profile_unlocked(
# L0 Basic Info 从输入、唯一性到落盘都只属于 user_id。
new_kv = _sanitize_kv(kv or {}, allowed_fields=allowed_fields)
# 单调性守卫:占位符永远不覆盖已有值,过滤后为空就不写新版本、不产生新链头。
new_kv = _drop_placeholders(new_kv)
if not new_kv:
return BasicProfileUpsertResult(
success=True, changed=False, reason="no valid fields after sanitize"
@@ -513,5 +587,8 @@ def _assemble_full_kv_from_nodes(
for k, v in kv.items():
if allowed_set is not None and k not in allowed_set:
continue
# 历史上已落盘的占位符版本(如 name=用户)不覆盖更早的真实值。
if is_placeholder_value(k, v):
continue
full_kv[k] = v
return full_kv
+5
View File
@@ -260,6 +260,7 @@ class EmbedderConfig:
max_retries: int = None # 最大重试次数
retry_delay: float = None # 重试间隔(秒)
timeout: int = None # 请求超时(秒)
batch_size: int = None # 单次 HTTP 请求最多带多少条 input(embed_batch 按此切块)
# 内部平台扩展(extra_headers / extra_body 会透传给 OpenAI client)
extra_headers: Optional[Dict[str, str]] = None
@@ -294,6 +295,10 @@ class EmbedderConfig:
# 请求切断并制造重试风暴。SDK 内层重试已关闭,15s 是每个显式 attempt
# 的真实上限;部署可通过 MEMORY_EMBEDDER_TIMEOUT 覆盖。
self.timeout = _get_env_int("MEMORY_EMBEDDER_TIMEOUT", 15)
if self.batch_size is None:
# 大批量(如重新 embed 整个库)不能一次 HTTP 发几千条:多数服务对
# input 数组有上限,超了直接 400。按 batch_size 切块逐块发送。
self.batch_size = _get_env_int("MEMORY_EMBEDDER_BATCH_SIZE", 64)
if self.extra_headers is None:
val = os.getenv("MEMORY_EMBEDDER_EXTRA_HEADERS")
if val:
+105 -19
View File
@@ -38,10 +38,12 @@ import importlib.util
import json
from json import JSONDecodeError
import logging
import random
import threading
import time
from ..config import MemoryConfig
from ..embedding import EmbeddingDimensionMismatchError
from .workload import embedding_slot
from ..utils.log_setup import get_request_id
@@ -187,6 +189,41 @@ def _redact_headers(headers: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any
return out
# Retry-After 的上限:服务端给出超长等待(或恶意值)时不让单次调用挂几十分钟。
_RETRY_AFTER_MAX_S = 60.0
def _retry_after_seconds(exc: Exception) -> Optional[float]:
"""从 429 异常的响应头里取 Retry-After(秒数或 HTTP 日期),取不到返回 None。"""
response = getattr(exc, "response", None)
headers = getattr(response, "headers", None)
if not headers:
return None
try:
raw = headers.get("retry-after") or headers.get("Retry-After")
except Exception:
return None
if raw is None:
return None
raw = str(raw).strip()
try:
seconds = float(raw)
except ValueError:
try:
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
when = parsedate_to_datetime(raw)
if when.tzinfo is None:
when = when.replace(tzinfo=timezone.utc)
seconds = (when - datetime.now(timezone.utc)).total_seconds()
except Exception:
return None
if seconds != seconds: # NaN
return None
return min(max(0.0, seconds), _RETRY_AFTER_MAX_S)
def _embeddings_api_url(base_url: str) -> str:
"""OpenAI SDK posts to ``{base_url}/embeddings`` (base_url usually ends with /v1)."""
base = (base_url or "").rstrip("/")
@@ -205,7 +242,8 @@ class EmbedServiceConfig:
api_key: str = ""
base_url: str = ""
embedding_dims: int = 1536
batch_size: int = 100
# embed_batch 单次 HTTP 最多带多少条 input;超出按块顺序发送
batch_size: int = 64
enable_cache: bool = True
cache_size: int = 10000
max_retries: int = 5
@@ -287,6 +325,7 @@ class EmbedService:
max_retries=embedder.max_retries if embedder.max_retries is not None else 5,
retry_delay=embedder.retry_delay if embedder.retry_delay is not None else 3.0,
timeout=embedder.timeout if embedder.timeout is not None else 60,
batch_size=max(1, int(getattr(embedder, "batch_size", None) or 64)),
queue_batch_size=int(os.getenv("MEMORY_EMBED_BATCH_SIZE", "8")),
queue_batch_window_ms=float(os.getenv("MEMORY_EMBED_BATCH_WINDOW_MS", "100")),
extra_headers=embedder.extra_headers,
@@ -640,6 +679,8 @@ class EmbedService:
self,
texts: List[str],
use_cache: bool = True,
*,
max_retries: Optional[int] = None,
) -> List[List[float]]:
"""
批量向量化
@@ -647,17 +688,23 @@ class EmbedService:
Args:
texts: 文本列表
use_cache: 是否使用缓存
max_retries: 本次调用的重试次数;None 用配置值。长批量任务(重新
embed 整个库)可以给更宽的重试预算,不影响在线路径。
Returns:
向量列表
"""
vectors, _metas = await self.embed_batch_with_meta(texts, use_cache=use_cache)
vectors, _metas = await self.embed_batch_with_meta(
texts, use_cache=use_cache, max_retries=max_retries,
)
return vectors
async def embed_batch_with_meta(
self,
texts: List[str],
use_cache: bool = True,
*,
max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
"""批量向量化,同时返回每条的 provider metadata(非 vector)。"""
if not texts:
@@ -682,7 +729,12 @@ class EmbedService:
indices_to_embed.append(i)
if texts_to_embed:
vectors, batch_metas = await self._embed_texts_with_meta(texts_to_embed)
# max_retries 只在显式给出时透传:保持 `_embed_texts_with_meta(texts)`
# 这个既有调用形状不变(测试与下游替身按它打桩)。
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
vectors, batch_metas = await self._embed_texts_with_meta(
texts_to_embed, **retry_kwargs,
)
for j, vector in enumerate(vectors):
idx = indices_to_embed[j]
@@ -700,9 +752,9 @@ class EmbedService:
return results, metas
async def _embed_texts_with_meta(
self, texts: List[str]
self, texts: List[str], *, max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
"""实际的向量化调用:**一次 HTTP 带走整批**;返回 (vectors, metas)。
"""实际的向量化调用:**一次 HTTP 带走一块**;返回 (vectors, metas)。
早期实现是 `for text in texts: await _embed_openai([text])` 逐条串行,
原意是"避免 batch size 限制",但代价极大:线上一批 8 条 × ~5.6s 串行
@@ -719,21 +771,34 @@ class EmbedService:
截断在这里做(发送前逐条过 `_truncate_for_embedding`):所有 embedding
路径(embed / embed_batch / embed_queued / *_with_meta)最终都落到这个
方法 —— 放在各入口会漏。理由与阈值见模块头部 `_EMBED_MAX_TOKENS`。
切块:一次带走整批只适用于小批。几千条的 input 数组会被多数服务直接
400(input 上限),所以按 `batch_size`(默认 64)切块,块之间顺序发送。
不超过一块时行为与切块前逐字相同(`batch_mode="single_request"`)。
"""
if not texts:
return [], []
safe_texts = [_truncate_for_embedding(t) for t in texts]
vectors, meta = await self._embed_openai(safe_texts)
base_meta = dict(meta or {})
base_meta["batch_mode"] = "single_request"
base_meta["batch_input_count"] = len(safe_texts)
chunk = max(1, int(self._embed_config.batch_size or 64))
vectors: List[List[float]] = []
all_metas: List[Dict[str, Any]] = []
for idx in range(len(safe_texts)):
m = dict(base_meta)
m["batch_index"] = idx
all_metas.append(m)
chunk_count = (len(safe_texts) + chunk - 1) // chunk
for start in range(0, len(safe_texts), chunk):
part = safe_texts[start:start + chunk]
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
part_vectors, meta = await self._embed_openai(part, **retry_kwargs)
base_meta = dict(meta or {})
base_meta["batch_mode"] = "single_request" if chunk_count == 1 else "chunked"
base_meta["batch_input_count"] = len(part)
if chunk_count > 1:
base_meta["batch_chunk_index"] = start // chunk
base_meta["batch_chunk_count"] = chunk_count
for idx in range(len(part)):
m = dict(base_meta)
m["batch_index"] = start + idx
all_metas.append(m)
vectors.extend(part_vectors)
self._last_embed_meta = all_metas[-1] if all_metas else {}
return vectors, all_metas
@@ -783,7 +848,7 @@ class EmbedService:
)
async def _embed_openai(
self, texts: List[str]
self, texts: List[str], *, max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], Dict[str, Any]]:
"""
使用 OpenAI 兼容接口向量化
@@ -873,7 +938,9 @@ class EmbedService:
"endpoint_overridden": bool(_ov),
}
_max_retries = max(0, int(self._embed_config.max_retries or 0))
_max_retries = max(0, int(
(self._embed_config.max_retries or 0) if max_retries is None else max_retries
))
_retry_delay = max(0.0, float(self._embed_config.retry_delay or 0.0))
last_exc: Optional[Exception] = None
@@ -919,6 +986,17 @@ class EmbedService:
f"got={len(vectors)}"
)
dims = [len(v) for v in vectors]
expected_dims = self._embed_config.embedding_dims
if expected_dims:
bad = [d for d in dims if d != int(expected_dims)]
if bad:
# 宽度不对的向量写进集合会把集合的维度钉死在错误值上(Chroma
# 首次 add 定维),或在已有集合上整批失败。确定性错误,不重试。
raise EmbeddingDimensionMismatchError(
"embedding dimension mismatch: "
f"expected={int(expected_dims)} got={bad[0]} "
f"model={self._embed_config.model}"
)
meta = {
"http_status": raw.status_code,
"response_headers": resp_headers,
@@ -954,6 +1032,8 @@ class EmbedService:
},
)
return vectors, meta
except EmbeddingDimensionMismatchError:
raise
except Exception as e:
last_exc = e
_err_name = type(e).__name__
@@ -987,12 +1067,18 @@ class EmbedService:
if _retryable and attempt < _max_retries:
# 逐出可能卡死的 client,下个 attempt 重建干净连接(恢复原防御语义)
self._evict_client(_eff_base_url, _eff_api_key, _eff_timeout)
_delay = _retry_delay
_retry_after = _retry_after_seconds(e) if _status == 429 else None
if _retry_after is not None:
# 限流时服务端明确告诉了要等多久:按它等,再加一点抖动,
# 避免多个 worker 在同一时刻一起醒来再次撞限流。
_delay = max(_delay, _retry_after) + random.uniform(0, 0.25 * _retry_after)
logger.warning(
f"[embed] attempt {attempt + 1}/{_max_retries + 1} failed "
f"({_err_name}): {_err_msg[:200]}; retry in {_retry_delay}s"
f"({_err_name}): {_err_msg[:200]}; retry in {_delay:.2f}s"
)
if _retry_delay > 0:
await asyncio.sleep(_retry_delay)
if _delay > 0:
await asyncio.sleep(_delay)
continue
self._log_embed_call(
status="error",
+18
View File
@@ -22,6 +22,7 @@ import logging
import json
import threading
import time
import weakref
from ..models.memory import MemoryNode, MemoryLayer, MemoryStatus
from ..config import MemoryConfig
@@ -77,6 +78,21 @@ async def _run_in_kuzu_pool(func, *args, **kwargs):
return await run_in_executor_guarded(_kuzu_executor, _measured)
# 本进程里已打开、尚未 close 的 store。drop_graph 据此拒绝删掉一个正被
# 活着的 client 使用的图库。WeakSet:store 被回收后自动消失。
_LIVE_STORES: "weakref.WeakSet[KuzuGraphStore]" = weakref.WeakSet()
def open_graph_paths() -> List[str]:
"""Real paths of the graph databases live stores in this process hold open."""
import os
return [
os.path.realpath(str(store._db_path))
for store in list(_LIVE_STORES)
if store._db is not None
]
def kuzu_executor_snapshot() -> Dict[str, Any]:
with _kuzu_stats_lock:
return {
@@ -350,6 +366,7 @@ class KuzuGraphStore(GraphStoreBase):
)
self._db = db
self._conn = conn
_LIVE_STORES.add(self)
except asyncio.TimeoutError:
logger.warning(
f"GraphStore init timed out after 5s (Kuzu file lock contention). "
@@ -2441,6 +2458,7 @@ class KuzuGraphStore(GraphStoreBase):
async def close(self) -> None:
"""关闭连接"""
conn, db = self._conn, self._db
_LIVE_STORES.discard(self)
self._conn = None
self._db = None
self._available = False
+14 -6
View File
@@ -89,6 +89,17 @@ def passes_created_after(
logger = logging.getLogger(__name__)
def physical_collection_name(base_name: Optional[str], dims: Optional[int]) -> str:
"""The collection name a store opens for ``base_name`` at ``dims``.
The dims suffix keeps two embedding widths from sharing one collection.
`hy_memory.reembed.vector_space_names` uses this too, so callers never
re-derive the rule.
"""
base = base_name or "agent_memories"
return f"{base}_{dims}" if dims else base
class VectorStoreBase(ABC):
"""
向量存储抽象基类
@@ -98,13 +109,10 @@ class VectorStoreBase(ABC):
def __init__(self, config: MemoryConfig):
self.config = config
base_name = config.vector_store.collection_name or "agent_memories"
dims = config.vector_store.embedding_dims
# 自动拼接维度后缀,避免不同 embedding 模型写入同一 collection 导致维度冲突
if dims:
self._collection_name = f"{base_name}_{dims}"
else:
self._collection_name = base_name
self._collection_name = physical_collection_name(
config.vector_store.collection_name, config.vector_store.embedding_dims,
)
_supports_retrieval_clauses: bool = False
_supports_narrow_episode_queries: bool = False
+29 -1
View File
@@ -9,10 +9,12 @@ Agent Memory V2 - VectorStore (Chroma)
- 同步 API → _run_in_vdb_pool() 包装为异步
"""
from typing import Optional, List, Dict, Any
from typing import Optional, List, Dict, Any, Set, Tuple
import concurrent.futures
import functools
import logging
import os
import weakref
from ..models.memory import (
MemoryNode, MemoryLayer, MemoryStatus,
@@ -33,6 +35,30 @@ _vdb_executor = concurrent.futures.ThreadPoolExecutor(
)
# 本进程里已 initialize、尚未 close 的 store。drop_vector_space 据此拒绝删掉
# 一个正被活着的 client 使用的集合。WeakSet:store 被回收后自动消失。
_LIVE_STORES: "weakref.WeakSet[ChromaVectorStore]" = weakref.WeakSet()
def _location_key(vs_config) -> Tuple[str, str]:
if getattr(vs_config, "host", None):
return ("http", f"{vs_config.host}:{vs_config.port or 8000}")
return ("local", os.path.realpath(str(vs_config.persist_directory or "")))
def open_collection_names(location: Tuple[str, str]) -> Set[str]:
"""Collections that live stores in this process hold open at ``location``."""
names: Set[str] = set()
for store in list(_LIVE_STORES):
if store._client is None or _location_key(store.config.vector_store) != location:
continue
names.add(store._collection_name)
if store._entity_collection is not None:
from ..pipelines._retrieval.config import entity_collection_name
names.add(entity_collection_name(store._collection_name))
return names
async def _run_in_vdb_pool(func, *args, **kwargs):
"""Run a synchronous Chroma call in the dedicated VDB executor."""
call = functools.partial(func, *args, **kwargs) if args or kwargs else func
@@ -88,6 +114,7 @@ class ChromaVectorStore(VectorStoreBase):
return client, collection
self._client, self._collection = await _run_in_vdb_pool(_init)
_LIVE_STORES.add(self)
logger.debug(f"VectorStore initialized (Chroma), collection={self._collection_name}")
# ================================================================
@@ -561,6 +588,7 @@ class ChromaVectorStore(VectorStoreBase):
async def close(self) -> None:
"""关闭客户端"""
# Chroma PersistentClient 无需显式关闭
_LIVE_STORES.discard(self)
self._client = None
self._collection = None
self._entity_collection = None
+1777
View File
File diff suppressed because it is too large. Load diff
+2 -2
View File
@@ -18,7 +18,7 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.8`。MemHub 源码版本为 `1.0.5`
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.9`。MemHub 源码版本为 `1.0.5`
(build 36),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
@@ -118,7 +118,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.8`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.9`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+1 -1
View File
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.8",
"versionInfo": "2.0.9",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+1 -1
View File
@@ -9,7 +9,7 @@ Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.8
## hy-memory 2.0.9
License: Apache-2.0
+45 -1
View File
@@ -162,10 +162,54 @@ it("never emits a secret from a malformed manually written event", async () => {
command: "logs",
state: "empty",
events: 0,
invalid_lines: 1,
invalid_lines: 0,
unrecognized_lines: 1,
});
});
// After an upgrade the files still hold an older build's records. They are
// whole JSON lines the reader leaves out, not damage, so the text output must
// not warn about them; only a line that is not JSON is called damaged.
it("warns only about lines that are not JSON, not about older records", async () => {
const dataDir = await fixture();
const older = {
schema_version: 0,
service: "desktop",
component: "desktop.lifecycle",
event: "desktop.started",
};
await writeFile(
path.join(dataDir, "logs", "diagnostic-desktop.jsonl"),
`${JSON.stringify(older)}\n${JSON.stringify({ ...older, schema_version: 1 })}\n`,
{ mode: 0o600 },
);
let stdout = "";
let stderr = "";
vi.spyOn(process.stdout, "write").mockImplementation((value) => {
stdout += String(value);
return true;
});
vi.spyOn(process.stderr, "write").mockImplementation((value) => {
stderr += String(value);
return true;
});
expect(await run(["logs"], { ONE_MEMORY_DATA_DIR: dataDir })).toBe(0);
expect(stdout).toContain("core.start_failed");
expect(stdout).not.toContain("损坏");
expect(stderr).toBe("");
await writeFile(
path.join(dataDir, "logs", "diagnostic-cli.jsonl"),
'{"schema_version": 1, "event": \n',
{ mode: 0o600 },
);
stdout = "";
expect(await run(["logs"], { ONE_MEMORY_DATA_DIR: dataDir })).toBe(0);
expect(stdout).toContain("跳过 1 行损坏的诊断记录。");
expect(stderr).toBe("memhub: 跳过 1 行损坏的诊断记录。\n");
});
it("rejects invalid filters without touching the log directory", async () => {
const dataDir = path.join(
await mkdtemp(path.join(os.tmpdir(), "om-cli-no-log-")),
+143 -11
View File
@@ -190,8 +190,10 @@ import {
} from "./managed-cli-link.js";
import {
addMemory,
DEFAULT_USER_ID,
isMemoryId,
isRequestId,
isUserId,
MEMORY_KINDS,
MEMORY_SCENES,
MEMORY_VARIANTS,
@@ -209,6 +211,7 @@ import {
renderMemoryAddText,
renderMemoryDetailText,
renderMemoryPageText,
userIdParameters,
validMemoryText,
} from "./memory-report.js";
import {
@@ -538,6 +541,23 @@ async function runCommand(
environment = { ...environment, ONE_MEMORY_DATA_DIR: lifted.dataDir };
}
}
// `--user-id <id>` is global to the memory commands: it names whose
// memories are read or written. It is lifted before dispatch so each
// command's own parser keeps seeing exactly the arguments it always did.
// `sources connect` gives the flag its own meaning and parses it itself.
let userId: string | undefined;
if (USER_ID_COMMANDS.has(arguments_[0] ?? "")) {
const lifted = liftUserIdOption(arguments_);
if (lifted === null) {
return reject(
findCommand(arguments_[0] ?? ""),
asJson,
"--user-id 无效:只能是 1–64 位字母、数字、点、下划线或连字符。",
);
}
arguments_ = lifted.arguments_;
userId = lifted.userId;
}
environment = await environmentWithManagedDataDir(environment);
const command = arguments_[0];
if (command === undefined) return reject(undefined, asJson, "缺少命令。");
@@ -776,9 +796,11 @@ async function runCommand(
const attachment = await attachToRuntimeOrNull(environment);
if (attachment === null) return runtimeUnavailable(descriptor, asJson);
if (parsed.kind === "status") {
const receipt = await readMemoryWrite(attachment, parsed.id).catch(
() => undefined,
);
const receipt = await readMemoryWrite(
attachment,
parsed.id,
userId,
).catch(() => undefined);
if (receipt === undefined) {
return emitFailure(asJson, "write status", "memory_write_read_failed");
}
@@ -812,6 +834,7 @@ async function runCommand(
? {}
: { memory_at: parsed.memory_at }),
...(parsed.priority === undefined ? {} : { priority: parsed.priority }),
...(userId === undefined ? {} : { user_id: userId }),
});
if (outcome.state !== "accepted") {
emit(
@@ -957,9 +980,12 @@ async function runCommand(
state: "empty",
events: 0,
invalid_lines: report.invalid_lines,
unrecognized_lines: report.unrecognized_lines,
})}\n`
: renderLogText(report.events, report.invalid_lines),
);
// Only lines that are not JSON are damage. Records an older build wrote
// are left out without a warning: after an upgrade they are expected.
if (report.invalid_lines) {
process.stderr.write(
`${PRODUCT}: 跳过 ${report.invalid_lines} 行损坏的诊断记录。\n`,
@@ -1582,6 +1608,7 @@ async function runCommand(
limit: parsed.limit,
character_budget: parsed.characterBudget,
timeout_ms: parsed.timeoutMs,
...userIdParameters(userId),
})
: ({
schema_version: 2,
@@ -1780,6 +1807,16 @@ async function runCommand(
}
if (named === "sources" && arguments_[1] !== "list") {
const parsed = parseSourcesCommand(arguments_.slice(1));
if (
!parsed &&
arguments_.some((item) => item === "--user-id" || item === "--root")
) {
return reject(
descriptor,
asJson,
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。",
);
}
if (!parsed) return usage();
if (parsed.kind === "receipt") {
const connected = await connectRuntime(environment);
@@ -2004,7 +2041,11 @@ async function runCommand(
);
return connected.exitCode;
}
const outcome = await connected.reader<unknown>("/api/v2/user-profile");
const profileUser = userIdParameters(userId);
const outcome =
profileUser.user_id === undefined
? await connected.reader<unknown>("/api/v2/user-profile")
: await connected.reader<unknown>("/api/v2/user-profile", profileUser);
const value =
outcome.state === "ok"
? (outcome.data as {
@@ -2109,7 +2150,7 @@ async function runCommand(
}
const { reader, operations } = connected;
if (parsed.kind === "list") {
const document = await readMemoryPage(reader, parsed.query);
const document = await readMemoryPage(reader, parsed.query, userId);
emit(
asJson
? `${JSON.stringify(document)}\n`
@@ -2122,6 +2163,7 @@ async function runCommand(
reader,
parsed.memoryId,
parsed.namespace,
userId,
);
emit(
asJson
@@ -2133,8 +2175,8 @@ async function runCommand(
if (parsed.kind === "add" || parsed.kind === "add-status") {
const document =
parsed.kind === "add"
? await addMemory(operations, text ?? "", parsed.requestId)
: await readMemoryAddStatus(reader, parsed.requestId);
? await addMemory(operations, text ?? "", parsed.requestId, userId)
: await readMemoryAddStatus(reader, parsed.requestId, userId);
emit(
asJson
? `${JSON.stringify(document)}\n`
@@ -2149,9 +2191,13 @@ async function runCommand(
content: text,
...namespaceParameters(parsed.namespace),
...(parsed.task === undefined ? {} : { task: parsed.task }),
...userIdParameters(userId),
})
: await operations.delete<unknown>(
withQuery(memoryPath, namespaceParameters(parsed.namespace)),
withQuery(memoryPath, {
...namespaceParameters(parsed.namespace),
...userIdParameters(userId),
}),
);
const document = {
schema_version: 2,
@@ -3675,13 +3721,21 @@ async function connectSources(
asJson,
);
}
if (parsed.importHistory) {
// Another user's Hermes profile is granted and imported in one connect
// request: only that request can carry a user and a root.
if (
parsed.importHistory ||
parsed.userId !== undefined ||
parsed.root !== undefined
) {
const document = await connectSourcesWithImport(reader, operations, {
sourceIds: parsed.sourceIds,
confirmOverrides: parsed.confirmOverrides,
...(parsed.requestId === undefined
? {}
: { requestId: parsed.requestId }),
...(parsed.userId === undefined ? {} : { userId: parsed.userId }),
...(parsed.root === undefined ? {} : { root: parsed.root }),
});
emitFlowDocument(document, asJson);
return flowExitCode(document);
@@ -5131,14 +5185,23 @@ type SourcesCommand =
importHistory: boolean;
confirmOverrides: boolean;
requestId?: string;
/** `--user-id`: whose grant (Hermes only). Absent for `local`. */
userId?: string;
/** `--root`: an absolute root other than the candidate (Hermes only). */
root?: string;
}
| { kind: "all"; importHistory: boolean; confirmOverrides: boolean }
| { kind: "receipt"; requestId: string };
const SOURCE_ID_SHAPE = /^[a-z0-9][a-z0-9-]{0,127}$/;
/** Only Hermes keeps one history per profile, so only Hermes may be
* connected for another user or from another root. */
const MULTI_USER_SOURCES = new Set(["hermes"]);
/** `sources connect <id> [<id> ...] | --all-detected [--import]
* [--confirm-overrides] [--request-id <uuid>]` and `sources receipt <id>`.
* [--confirm-overrides] [--request-id <uuid>]
* [--user-id <id>] [--root <path>]` and `sources receipt <id>`.
*
* Sources may be named positionally or with repeated `--source` (the old
* `init --source` spelling is rewritten to this command). */
@@ -5154,6 +5217,8 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
const sourceIds: string[] = [];
const switches = new Set<string>();
let requestId: string | undefined;
let userId: string | undefined;
let root: string | undefined;
for (let index = 1; index < tokens.length; index += 1) {
const token = tokens[index];
if (["--import", "--confirm-overrides", "--all-detected"].includes(token)) {
@@ -5161,6 +5226,21 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
switches.add(token);
continue;
}
if (token === "--user-id" || token === "--root") {
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (token === "--user-id") {
if (userId !== undefined || !isUserId(value)) return null;
userId = value;
} else {
if (root !== undefined) return null;
const expanded = expandRootPath(value);
if (expanded === null) return null;
root = expanded;
}
index += 1;
continue;
}
if (token === "--source" || token === "--request-id") {
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
@@ -5184,10 +5264,23 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
const confirmOverrides = switches.has("--confirm-overrides");
if (
requestId !== undefined &&
(!OPERATION_ID.test(requestId) || !importHistory)
(!OPERATION_ID.test(requestId) ||
!(importHistory || userId !== undefined || root !== undefined))
) {
return null;
}
// `--user-id local` is the default and changes nothing.
if (userId === DEFAULT_USER_ID) userId = undefined;
if (userId !== undefined || root !== undefined) {
// One Hermes profile per connect: the user and root name one grant.
if (
switches.has("--all-detected") ||
sourceIds.length !== 1 ||
!MULTI_USER_SOURCES.has(sourceIds[0])
) {
return null;
}
}
if (switches.has("--all-detected")) {
// One names exact sources, the other means "whatever is detected".
if (sourceIds.length > 0 || requestId !== undefined) return null;
@@ -5200,9 +5293,24 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
importHistory,
confirmOverrides,
...(requestId === undefined ? {} : { requestId }),
...(userId === undefined ? {} : { userId }),
...(root === undefined ? {} : { root }),
};
}
/** `--root` as an absolute path: a leading `~` is the home directory, a
* relative path is taken from the working directory. Core expands nothing. */
function expandRootPath(value: string): string | null {
if (value.length === 0 || /[\0\r\n]/.test(value)) return null;
const expanded =
value === "~"
? os.homedir()
: value.startsWith("~/")
? path.join(os.homedir(), value.slice(2))
: value;
return path.resolve(expanded);
}
function parseGraphWidthCommand(
rest: readonly string[],
): { decision: GraphWidthDecision; confirmed: boolean } | null {
@@ -6596,6 +6704,30 @@ const OWN_DATA_DIR_COMMANDS = new Set([
/** Remove one `--data-dir <absolute path>` from argv; null when it is
* repeated, missing its value, or not an absolute path. */
/** Memory commands that honour the global `--user-id`. */
const USER_ID_COMMANDS = new Set(["memories", "recall", "profile", "write"]);
/**
* Take `--user-id <id>` out of the arguments. The default user (`local`) is
* returned as undefined, so a default request names no user at all and stays
* byte-identical to one sent before multi-user support.
*/
function liftUserIdOption(
arguments_: readonly string[],
): { arguments_: string[]; userId?: string } | null {
const index = arguments_.indexOf("--user-id");
if (index === -1) return { arguments_: [...arguments_] };
if (arguments_.indexOf("--user-id", index + 1) !== -1) return null;
const value = arguments_[index + 1];
if (value === undefined || value.startsWith("--") || !isUserId(value)) {
return null;
}
return {
arguments_: [...arguments_.slice(0, index), ...arguments_.slice(index + 2)],
...(value === DEFAULT_USER_ID ? {} : { userId: value }),
};
}
function liftDataDirOption(
arguments_: readonly string[],
): { arguments_: string[]; dataDir?: string } | null {
+26 -7
View File
@@ -68,6 +68,12 @@ const JSON_OPTION: CommandOption = {
flag: "--json",
summary: "以 JSON 输出到 stdout,供脚本与 Agent 消费。",
};
/** Global to the memory commands: whose memories are read or written. */
const USER_ID_OPTION: CommandOption = {
flag: "--user-id <id>",
summary:
"读写哪个用户的记忆,默认 local(App 用的就是 local)。1–64 位字母、数字、点、下划线或连字符;其他用户的记忆来自用 sources connect hermes --user-id 接入的 Hermes profile。",
};
const WATCH_OPTION: CommandOption = {
flag: "--watch",
summary: "串行持续刷新;Ctrl-C 只停止观察,不停止 Runtime。",
@@ -252,15 +258,25 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "sources",
summary:
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。",
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(会自动排除 ~/.hermes/profiles/**,各用户互不读取)。",
usage:
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect hermes --user-id <id> --root <path> [--import] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
options: [
{
flag: "--import",
summary:
"接入后立即导入历史会话并开启持续同步;不加时只授权,处理保持冻结。",
},
{
flag: "--user-id <id>",
summary:
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;其他来源只能是 local。",
},
{
flag: "--root <path>",
summary:
"只用于 hermes:读取的根目录(~ 会展开为主目录)。默认是 Hermes 自己的目录 ~/.hermes。",
},
{
flag: "--all-detected",
summary:
@@ -373,7 +389,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes。",
usage:
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--json]\n memhub memories add-status <request-id> [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--json]",
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--user-id <id>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--user-id <id>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add-status <request-id> [--user-id <id>] [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--user-id <id>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--user-id <id>] [--json]",
options: [
{
flag: "--scene <all|chat|working|coding>",
@@ -411,6 +427,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
"add 的幂等键;用同一个 id 重试只会读回第一次的结果,不会重复添加。不给时自动生成并输出。",
},
DESTRUCTIVE_CONFIRM_OPTION,
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
@@ -418,8 +435,8 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "profile",
summary: "读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充)。",
usage: "memhub profile [--json]",
options: [JSON_OPTION],
usage: "memhub profile [--user-id <id>] [--json]",
options: [USER_ID_OPTION, JSON_OPTION],
exit_codes: RUNTIME_EXITS,
},
{
@@ -475,7 +492,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"从原始对话与已沉淀记忆统一检索上下文;本地只读,不调用模型、不触发补索引。文本输出标明每条是「原始对话摘录」还是「记忆」(JSON 里是 kind: raw|memory)。",
usage:
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--user-id <id>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
options: [
{
flag: "--query-file <path>",
@@ -502,6 +519,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
flag: "--timeout-ms <n>",
summary: "Core 查询时限 50..30000 毫秒,默认 2000。",
},
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
@@ -650,7 +668,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"提交一段 OpenAI 格式的对话(messages)到写入队列,异步加工成记忆;或按 request id 查询状态。request id 是幂等键:不给时自动生成并输出,重试请带上同一个。新增一条结论式记忆请用 memories add。",
usage:
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--json]\n memhub write status <request-id> [--json]",
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--user-id <id>] [--json]\n memhub write status <request-id> [--user-id <id>] [--json]",
options: [
{
flag: "--messages-file <path>",
@@ -666,6 +684,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
{ flag: "--memory-at <RFC3339>", summary: "对话发生时间,必须带时区。" },
{ flag: "--live", summary: "优先处理。" },
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
+2
View File
@@ -10,6 +10,8 @@ export interface ContextSearchInput {
limit: number;
character_budget: number;
timeout_ms: number;
/** Whose memories to search. Omitted for the default user (`local`). */
user_id?: string;
}
/** The CLI's JSON document mirrors Core's SDK-native context schema v2. */
+11
View File
@@ -12,6 +12,7 @@ const READ_ONLY_PATHS = [
"/api/v1/live-sync",
"/api/v1/memory-processing",
"/api/v1/model-profiles",
"/api/v1/source-grants",
"/api/v1/sources",
"/api/v2/home-summary",
"/api/v2/memory-library",
@@ -310,6 +311,8 @@ export async function submitMemoryWrite(
messages: readonly OpenAIMessage[];
memory_at?: string;
priority?: "live" | "normal";
/** Whose memory this is. Omitted for the default user (`local`). */
user_id?: string;
},
): Promise<MemoryWriteOutcome> {
let response: { status: number; data: unknown };
@@ -327,6 +330,9 @@ export async function submitMemoryWrite(
...(request.priority === undefined
? {}
: { priority: request.priority }),
...(request.user_id === undefined || request.user_id === "local"
? {}
: { user_id: request.user_id }),
},
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
@@ -469,11 +475,16 @@ export async function deleteModelProfile(
export async function readMemoryWrite(
attachment: RuntimeAttachmentLike,
requestId: string,
/** A receipt belongs to the user that submitted it; omit for `local`. */
userId?: string,
): Promise<MemoryWriteReceipt | null> {
try {
const response = await axios.get<MemoryWriteReceipt>(
`${baseUrlOf(attachment)}/api/v2/memory-writes/${encodeURIComponent(requestId)}`,
{
...(userId === undefined || userId === "local"
? {}
: { params: { user_id: userId } }),
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 10_000,
proxy: false,
+3 -2
View File
@@ -6,9 +6,10 @@ const WRITE_PATHS = [
] as const;
/** `POST /api/v2/memories` adds one memory (the app's 添加记忆). */
const MEMORY_ADD_PATH = "/api/v2/memories";
/** One SDK memory; delete names its namespace in the query string. */
/** One SDK memory; delete names its namespace, and a non-default user, in
* the query string. */
const SDK_MEMORY_PATH =
/^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}(?:\?kind=(?:chat|experience|fact)(?:&variant=(?:working|coding))?)?$/;
/^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}(?:\?(?:kind=(?:chat|experience|fact)(?:&variant=(?:working|coding))?(?:&user_id=[A-Za-z0-9._-]{1,64})?|user_id=[A-Za-z0-9._-]{1,64}))?$/;
const MODEL_PROBE_PATH =
/^\/api\/v1\/model-profiles\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\/probe$/;
const MODEL_PROFILE_PATH =
+29 -7
View File
@@ -110,9 +110,18 @@ export async function connectSourcesWithImport(
sourceIds: readonly string[];
confirmOverrides: boolean;
requestId?: string;
/** Whose grant this is. Omitted for the default user (`local`). */
userId?: string;
/** An absolute root other than the source's candidate (a Hermes
* profile). Core reads its identity itself, so the source's candidate
* root id and revision are not sent for it. */
root?: string;
},
): Promise<FlowDocument> {
const sourceIds = input.sourceIds;
// A second user's grant is not a first run, and a custom root has no
// candidate the app showed: both are a plain connect.
const profile = input.root !== undefined || input.userId !== undefined;
if (
sourceIds.length < 1 ||
sourceIds.length > 32 ||
@@ -134,9 +143,14 @@ export async function connectSourcesWithImport(
return invalid("source_unavailable");
}
const items = selected.map(([candidate]) => candidate);
const needsOverride = items
.filter((item) => item.requires_override_confirmation)
.map((item) => item.source_id);
// A custom root is the caller's own choice, not the environment override
// the confirmation exists for.
const needsOverride =
input.root !== undefined
? []
: items
.filter((item) => item.requires_override_confirmation)
.map((item) => item.source_id);
if (needsOverride.length > 0 && !input.confirmOverrides) {
return {
schema_version: 1,
@@ -154,18 +168,22 @@ export async function connectSourcesWithImport(
request = buildSourceImportRequest({
sourceIds: [...sourceIds],
sourceRoots: Object.fromEntries(
items.map((item) => [item.source_id, item.candidate_root_id]),
items.flatMap((item) =>
input.root === undefined
? [[item.source_id, item.candidate_root_id]]
: [],
),
),
sourceCandidateRevisions: Object.fromEntries(
items.flatMap((item) =>
item.candidate_revision === undefined
item.candidate_revision === undefined || input.root !== undefined
? []
: [[item.source_id, item.candidate_revision]],
),
),
confirmEnvironmentOverrides: needsOverride.length > 0,
fullRescan: false,
onboarding: true,
onboarding: !profile,
enableSync: true,
requestId,
});
@@ -175,7 +193,11 @@ export async function connectSourcesWithImport(
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
const result = await operations.post<unknown>(request.path, {
...request.body,
...(input.userId === undefined ? {} : { user_id: input.userId }),
...(input.root === undefined ? {} : { root: input.root }),
});
if (result.state === "rejected") {
return rejected(requestId, result.code);
}
+6
View File
@@ -107,7 +107,11 @@ export async function readLogReport(
query: LogQuery,
): Promise<{
events: DiagnosticEventV1[];
/** Lines that are not JSON: the only ones `memhub logs` calls damaged. */
invalid_lines: number;
/** Whole JSON lines the reader leaves out, such as an older build's
* records. Nothing is wrong with them, so the text output says nothing. */
unrecognized_lines: number;
failure?: "log_directory_unreadable";
}> {
// `readDiagnosticEvents` treats an unreadable logs path as "no events". That
@@ -119,6 +123,7 @@ export async function readLogReport(
return {
events: [],
invalid_lines: 0,
unrecognized_lines: 0,
failure: "log_directory_unreadable",
};
}
@@ -129,6 +134,7 @@ export async function readLogReport(
return {
events: [],
invalid_lines: 0,
unrecognized_lines: 0,
failure: "log_directory_unreadable",
};
}
+37 -3
View File
@@ -14,6 +14,25 @@ import { type CoreReader, RESERVED_MEMORY_SEGMENTS } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
/** A MemHub user id, the same shape Core accepts. `local` is the default. */
const USER_ID = /^[A-Za-z0-9._-]{1,64}$/;
export const DEFAULT_USER_ID = "local";
export function isUserId(value: string): boolean {
return USER_ID.test(value);
}
/**
* The `user_id` to send, or nothing. The default user is never named on the
* wire, so a request without `--user-id` stays exactly what it was before
* multi-user support and still works against an older Core.
*/
export function userIdParameters(userId?: string): { user_id?: string } {
return userId === undefined || userId === DEFAULT_USER_ID
? {}
: { user_id: userId };
}
/** Output budget: a long memory is shortened visibly, never dropped silently. */
const CONTENT_BUDGET = 240;
@@ -101,10 +120,12 @@ export interface MemoryDetailDocument {
export async function readMemoryPage(
reader: CoreReader,
query: MemoryQuery,
userId?: string,
): Promise<MemoryPageDocument> {
const parameters: Record<string, string | number> = {
offset: query.offset,
limit: query.limit,
...userIdParameters(userId),
};
if (query.status !== undefined) parameters.statuses = query.status;
if (query.source !== undefined) parameters.source = query.source;
@@ -156,8 +177,12 @@ export async function readMemoryDetail(
reader: CoreReader,
memoryId: string,
namespace: MemoryNamespace = {},
userId?: string,
): Promise<MemoryDetailDocument> {
const parameters = namespaceParameters(namespace);
const parameters = {
...namespaceParameters(namespace),
...userIdParameters(userId),
};
const outcome = await reader<unknown>(
`/api/v2/memories/${memoryId}`,
Object.keys(parameters).length > 0 ? parameters : undefined,
@@ -236,6 +261,7 @@ export async function addMemory(
operations: CoreOperations,
text: string,
requestId: string = randomUUID(),
userId?: string,
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
@@ -259,7 +285,10 @@ export async function addMemory(
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
const result = await operations.post<unknown>(request.path, {
...request.body,
...userIdParameters(userId),
});
if (result.state !== "committed") {
return { ...base, state: result.state, code: result.code };
}
@@ -273,6 +302,7 @@ export async function addMemory(
export async function readMemoryAddStatus(
reader: CoreReader,
requestId: string,
userId?: string,
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
@@ -283,7 +313,11 @@ export async function readMemoryAddStatus(
if (!UUID.test(requestId)) {
return { ...base, state: "invalid", code: "invalid_request_id" };
}
const outcome = await reader<unknown>(memoryAddStatusPath(requestId));
const user = userIdParameters(userId);
const outcome =
user.user_id === undefined
? await reader<unknown>(memoryAddStatusPath(requestId))
: await reader<unknown>(memoryAddStatusPath(requestId), user);
if (outcome.state === "not_found") {
return { ...base, state: "not_found", code: "memory_add_not_found" };
}
+84 -1
View File
@@ -38,6 +38,21 @@ export interface SourcesDocument {
state: ReadState["state"];
code?: string;
sources: Array<Record<string, unknown>>;
/** Every grant of every user (`GET /api/v1/source-grants`). Absent when
* Core is too old to list them; `sources` is the default user's view. */
grants?: SourceGrant[];
}
/** One source grant. Only Hermes has grants for users other than `local`,
* one per Hermes profile root. */
export interface SourceGrant {
user_id: string;
source_id: string;
root_label: string | null;
root_path: string | null;
authorized: boolean;
revoked_at: string | null;
sync_enabled: boolean;
}
export interface SyncDocument {
@@ -86,14 +101,66 @@ export async function readSources(
sources: [],
};
}
const grants = await readSourceGrants(reader);
return {
schema_version: SCHEMA_VERSION,
product: PRODUCT,
state: "ok",
sources: result.data.map(projectSource),
// `/api/v1/sources` is the default user's view of every source.
sources: result.data.map((source) => ({
...projectSource(source),
user_id: "local",
})),
...(grants === null ? {} : { grants }),
};
}
/** The grants of every user, or null when Core cannot list them (an older
* Core answers 404): the source list is then shown as before. */
async function readSourceGrants(
reader: CoreReader,
): Promise<SourceGrant[] | null> {
let outcome: ReadOutcome<unknown>;
try {
outcome = await reader<unknown>("/api/v1/source-grants");
} catch {
return null;
}
if (outcome.state !== "ok") return null;
const value = outcome.data;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const items = (value as { grants?: unknown }).grants;
if (!Array.isArray(items)) return null;
const grants: SourceGrant[] = [];
const text = (candidate: unknown): string | null =>
typeof candidate === "string" && candidate.length <= 4096
? candidate
: null;
for (const item of items) {
if (!item || typeof item !== "object" || Array.isArray(item)) return null;
const grant = item as Record<string, unknown>;
if (
typeof grant.user_id !== "string" ||
!/^[A-Za-z0-9._-]{1,64}$/.test(grant.user_id) ||
typeof grant.source_id !== "string" ||
!/^[a-z0-9][a-z0-9-]{0,127}$/.test(grant.source_id) ||
typeof grant.authorized !== "boolean"
) {
return null;
}
grants.push({
user_id: grant.user_id,
source_id: grant.source_id,
root_label: text(grant.root_label),
root_path: text(grant.root_path),
authorized: grant.authorized,
revoked_at: text(grant.revoked_at),
sync_enabled: grant.sync_enabled === true,
});
}
return grants;
}
export async function readSync(reader: CoreReader): Promise<SyncDocument> {
const result = validated(
await reader<unknown>("/api/v1/live-sync"),
@@ -231,6 +298,22 @@ export function renderSourcesText(document: SourcesDocument): string {
);
}
if (document.sources.length === 0) lines.push("(没有来源)");
// The list above is the default user's (`local`). Another user's grant,
// a Hermes profile, is listed with its user so it is not mistaken for it.
const others = (document.grants ?? []).filter(
(grant) => grant.user_id !== "local",
);
if (others.length > 0) {
lines.push("其他用户的连接:");
for (const grant of others) {
const name =
document.sources.find((source) => source.source_id === grant.source_id)
?.display_name ?? grant.source_id;
lines.push(
`- ${name}(用户 ${grant.user_id}):${grant.authorized ? "已连接" : grant.revoked_at ? "已撤销" : "未连接"};处理${grant.sync_enabled ? "已开启" : "已冻结"}${grant.root_path ? `;目录 ${grant.root_path}` : ""}`,
);
}
}
lines.push("");
return lines.join("\n");
}
+62 -1
View File
@@ -179,12 +179,72 @@ describe("CLI-07/10 source and synchronization reads", () => {
sync_enabled: true,
sync_completeness: "additions_only",
});
expect(requestedPaths()).toEqual(["/api/v1/sources"]);
// The grants list answers 404 here, as an older Core does: the source
// list is shown as before, every row the default user's.
expect(report.sources[0].user_id).toBe("local");
expect(report.grants).toBeUndefined();
expect(requestedPaths()).toEqual([
"/api/v1/sources",
"/api/v1/source-grants",
]);
expect(requestedPaths().some((path) => path.includes("detect"))).toBe(
false,
);
});
it("shows every user's grant, a Hermes profile with its user", async () => {
const grants = {
schema_version: 1,
grants: [
{
user_id: "local",
source_id: "codex",
root_label: ".codex",
root_path: "/home/synthetic/.codex",
authorized: true,
revoked_at: null,
sync_enabled: true,
},
{
user_id: "wife",
source_id: "hermes",
root_label: "wife",
root_path: "/home/synthetic/.hermes/profiles/wife",
authorized: true,
revoked_at: null,
sync_enabled: true,
},
],
};
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/source-grants"
? { status: 200, data: grants }
: (RESPONSES.get(url) ?? { status: 404, data: {} }),
);
let streams = captureStreams();
try {
expect(await run(["sources", "list", "--json"], {})).toBe(0);
} finally {
streams.restore();
}
const report = JSON.parse(streams.stdout());
expect(report.grants).toEqual(grants.grants);
expect(report.sources[0]).toMatchObject({
source_id: "codex",
user_id: "local",
});
streams = captureStreams();
try {
expect(await run(["sources", "list"], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain("其他用户的连接:");
expect(streams.stdout()).toContain(
"hermes(用户 wife):已连接;处理已开启;目录 /home/synthetic/.hermes/profiles/wife",
);
});
it("keeps degraded synchronization distinct and prints its error code", async () => {
const streams = captureStreams();
try {
@@ -552,6 +612,7 @@ describe("CLI-19 progress and issue reads", () => {
requestedPaths().every((path) =>
[
"/api/v1/sources",
"/api/v1/source-grants",
"/api/v1/live-sync",
"/api/v1/import-jobs",
"/api/v1/memory-processing",
+405
View File
@@ -0,0 +1,405 @@
import { mkdtemp, writeFile } from "node:fs/promises";
import os, { tmpdir } from "node:os";
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
const direct = vi.hoisted(() => ({ get: vi.fn(), post: vi.fn() }));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: direct.get,
post: direct.post,
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const REQUEST = "33333333-3333-4333-8333-333333333333";
const HERMES = {
source_id: "hermes",
display_name: "Hermes",
storage_kinds: ["sqlite"],
available: true,
authorized: true,
detected: null,
root_label: ".hermes",
candidate_path: "~/.hermes",
candidate_kind: "default",
candidate_revision: 0,
candidate_root_id: "hermes-root-identity",
imported_messages: 0,
imported_conversations: 0,
capabilities: {
historical_import: "implemented",
incremental_import: "implemented",
live_sync: "implemented",
recall: "implemented",
installer: "not_implemented",
verification: "fixture_only",
},
sync_completeness: "additions_only",
location_state: "found",
};
const CODEX = {
...HERMES,
source_id: "codex",
display_name: "Codex",
candidate_root_id: "codex-root-identity",
};
function addReceipt() {
return {
evidence_id: "44444444-4444-4444-8444-444444444444",
text: "Prefers tea",
scope: "all-apps",
created_at: "2026-10-27T00:00:00Z",
state: "pending",
task_id: null,
memory_ids: [],
error: null,
};
}
beforeEach(() => {
vi.clearAllMocks();
for (const method of [
...Object.values(transport),
...Object.values(direct),
]) {
method.mockReset();
}
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
transport.get.mockImplementation(async (url: string) => {
if (url === "/api/v1/sources") {
return { status: 200, data: [HERMES, CODEX] };
}
if (url === "/api/v2/user-profile") {
return {
status: 200,
data: { request_id: "r-1", profile: null, elapsed_ms: 1 },
};
}
return { status: 404, data: {} };
});
transport.post.mockImplementation(
async (url: string, body: Record<string, unknown>) =>
url === "/api/v1/sources/connect"
? {
status: 200,
data: {
request_id: body.request_id,
accepted_source_ids: ["hermes"],
failed_sources: {},
jobs: [],
},
}
: url === "/api/v2/memories"
? { status: 201, data: addReceipt() }
: { status: 404, data: {} },
);
transport.put.mockResolvedValue({ status: 200, data: {} });
transport.delete.mockResolvedValue({ status: 204, data: "" });
direct.post.mockResolvedValue({
status: 202,
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T1",
},
});
direct.get.mockResolvedValue({
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T1",
},
});
});
describe("--user-id on the memory commands", () => {
it("passes the user as a query parameter to memory reads and deletes", async () => {
await capturedRun(["memories", "list", "--user-id", "wife", "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memory-library", {
params: { offset: 0, limit: 100, user_id: "wife" },
});
// Global: the flag may come before the subcommand too.
await capturedRun(["memories", "--user-id", "wife", "show", "m-1"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memories/m-1", {
params: { user_id: "wife" },
});
await capturedRun(["memories", "add-status", REQUEST, "--user-id", "wife"]);
expect(transport.get).toHaveBeenCalledWith(
`/api/v2/memories/additions/${REQUEST}`,
{ params: { user_id: "wife" } },
);
await capturedRun(["profile", "--user-id", "wife", "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/user-profile", {
params: { user_id: "wife" },
});
await capturedRun([
"memories",
"delete",
"fact-1",
"--kind",
"fact",
"--yes",
"--user-id",
"wife",
]);
expect(transport.delete).toHaveBeenCalledWith(
"/api/v2/memories/fact-1?kind=fact&user_id=wife",
undefined,
);
await capturedRun([
"memories",
"delete",
"m-1",
"--yes",
"--user-id",
"wife",
]);
expect(transport.delete).toHaveBeenCalledWith(
"/api/v2/memories/m-1?user_id=wife",
undefined,
);
await capturedRun(["write", "status", REQUEST, "--user-id", "wife"]);
expect(direct.get.mock.calls[0][0]).toBe(
`http://127.0.0.1:8765/api/v2/memory-writes/${REQUEST}`,
);
expect(direct.get.mock.calls[0][1].params).toEqual({ user_id: "wife" });
});
it("passes the user in the body of memory writes and searches", async () => {
await capturedRun(["memories", "add", "Prefers tea", "--user-id", "wife"]);
expect(transport.post.mock.calls[0][0]).toBe("/api/v2/memories");
expect(transport.post.mock.calls[0][1]).toMatchObject({
text: "Prefers tea",
scope: "all-apps",
user_id: "wife",
});
const directory = await mkdtemp(path.join(tmpdir(), "om-user-id-"));
const content = path.join(directory, "content.txt");
await writeFile(content, "Prefers green tea\n");
await capturedRun([
"memories",
"update",
"m-1",
"--content-file",
content,
"--user-id",
"wife",
]);
expect(transport.put.mock.calls[0][0]).toBe("/api/v2/memories/m-1");
expect(transport.put.mock.calls[0][1]).toMatchObject({ user_id: "wife" });
await capturedRun(["recall", "tea", "--user-id", "wife", "--json"]);
const search = transport.post.mock.calls.find(
([url]) => url === "/api/v2/context/search",
);
expect(search?.[1]).toMatchObject({ query: "tea", user_id: "wife" });
const messages = path.join(directory, "messages.json");
await writeFile(
messages,
JSON.stringify([{ role: "user", content: "I like tea" }]),
);
await capturedRun([
"write",
"submit",
"--messages-file",
messages,
"--session",
"s-1",
"--user-id",
"wife",
]);
expect(direct.post.mock.calls[0][0]).toBe(
"http://127.0.0.1:8765/api/v2/memory-writes",
);
expect(direct.post.mock.calls[0][1]).toMatchObject({
session_id: "s-1",
user_id: "wife",
});
});
it("names no user by default or for --user-id local", async () => {
for (const extra of [[], ["--user-id", "local"]]) {
transport.get.mockClear();
transport.post.mockClear();
await capturedRun(["memories", "list", ...extra, "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memory-library", {
params: { offset: 0, limit: 100 },
});
await capturedRun(["profile", ...extra]);
expect(transport.get).toHaveBeenCalledWith(
"/api/v2/user-profile",
undefined,
);
await capturedRun(["memories", "add", "Prefers tea", ...extra]);
expect(transport.post.mock.calls[0][1]).not.toHaveProperty("user_id");
await capturedRun(["recall", "tea", ...extra, "--json"]);
const search = transport.post.mock.calls.find(
([url]) => url === "/api/v2/context/search",
);
expect(search?.[1]).not.toHaveProperty("user_id");
}
});
it("refuses a malformed user id before contacting Core", async () => {
for (const value of ["bad:id", "", "x".repeat(65), "a b", "--json"]) {
const result = await capturedRun([
"memories",
"list",
"--user-id",
value,
]);
expect(result.code).toBe(2);
}
expect(
(await capturedRun(["profile", "--user-id", "a", "--user-id", "b"])).code,
).toBe(2);
expect((await capturedRun(["recall", "tea", "--user-id"])).code).toBe(2);
expect(transport.get).not.toHaveBeenCalled();
expect(transport.post).not.toHaveBeenCalled();
});
});
describe("sources connect for a Hermes profile", () => {
it("connects hermes for another user from its own root", async () => {
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"wife",
"--root",
"~/.hermes/profiles/wife",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledTimes(1);
const [url, body] = transport.post.mock.calls[0];
expect(url).toBe("/api/v1/sources/connect");
expect(body).toMatchObject({
source_ids: ["hermes"],
// Core reads the custom root's identity itself.
source_roots: {},
source_candidate_revisions: {},
onboarding: false,
user_id: "wife",
root: path.join(os.homedir(), ".hermes/profiles/wife"),
});
});
it("keeps a plain connect unchanged and accepts --import with a profile", async () => {
await capturedRun(["sources", "connect", "hermes", "--import", "--json"]);
const plain = transport.post.mock.calls[0][1];
expect(plain).not.toHaveProperty("user_id");
expect(plain).not.toHaveProperty("root");
expect(plain).toMatchObject({
source_roots: { hermes: "hermes-root-identity" },
onboarding: true,
});
transport.post.mockClear();
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--import",
"--user-id",
"wife",
"--root",
"/profiles/wife",
"--request-id",
REQUEST,
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post.mock.calls[0][1]).toMatchObject({
request_id: REQUEST,
user_id: "wife",
root: "/profiles/wife",
});
});
it("refuses --user-id and --root for any source but hermes", async () => {
for (const argv of [
["sources", "connect", "codex", "--user-id", "wife"],
["sources", "connect", "codex", "--root", "/somewhere", "--import"],
["sources", "connect", "hermes", "codex", "--user-id", "wife"],
["sources", "connect", "--all-detected", "--user-id", "wife"],
["sources", "connect", "hermes", "--user-id", "bad:id"],
]) {
const result = await capturedRun(argv);
expect(result.code).toBe(2);
expect(result.stderr).toContain("hermes");
}
expect(transport.post).not.toHaveBeenCalled();
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+1 -1
View File
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.8",
"hy-memory==2.0.9",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
+401 -50
View File
@@ -14,6 +14,7 @@ from collections.abc import (
MutableMapping,
)
from contextlib import asynccontextmanager, nullcontext, suppress
from dataclasses import replace
from datetime import UTC, datetime, timedelta
from functools import partial
from pathlib import Path
@@ -117,6 +118,7 @@ from one_memory.imports.access import (
AuthorizedSourceRoot,
SourceAccessError,
root_identity_matches,
source_root_identity,
)
from one_memory.imports.adapters import (
ScanFailure,
@@ -157,6 +159,7 @@ from one_memory.imports.sources.common import SourceInstallState
from one_memory.imports.sources.registry import (
RootCandidateKind,
SourceRegistration,
SourceRootCandidate,
build_source_catalog,
workbuddy_default_homes,
)
@@ -244,7 +247,17 @@ from one_memory.service_levels import (
from one_memory.source_import_receipts import SourceImportReceipt, build_source_import_receipt
from one_memory.source_read_access import SourceReadAccess
from one_memory.source_totals import SourceTotals
from one_memory.strict_query import forbid_unknown_query_params
from one_memory.target_schema import bootstrap_fresh_management_schema
from one_memory.user_scope import (
DEFAULT_USER_ID,
MULTI_USER_SOURCES,
USER_ID_PATTERN,
UserId,
lane_key,
owns,
split_lane,
)
from one_memory.workspace_resolution import resolve_workspace_scopes
logger = logging.getLogger(__name__)
@@ -387,6 +400,12 @@ class CreateImportJobRequest(BaseModel):
class ConnectSourcesRequest(BaseModel):
source_ids: tuple[str, ...] = Field(min_length=1, max_length=32)
#: Whose grant this is. Only Hermes may connect a root for a user other
#: than `local`: a Hermes profile is another person's history.
user_id: UserId = DEFAULT_USER_ID
#: A root chosen by the caller instead of the source's candidate, such as
#: `~/.hermes/profiles/<name>`. Hermes only; Core computes its identity.
root: str | None = Field(default=None, min_length=1, max_length=4096)
confirm_environment_overrides: bool = False
full_rescan: bool = False
onboarding: bool = False
@@ -422,6 +441,8 @@ class SdkMemoryUpdateRequest(BaseModel):
variant: SdkMemoryVariant | None = None
# Experiences only: also replace the task.
task: str | None = Field(default=None, min_length=1, max_length=200_000)
#: Whose memory this is; another user's id answers 404.
user_id: UserId = DEFAULT_USER_ID
class SdkUserProfileResponse(BaseModel):
@@ -584,11 +605,15 @@ class ManualExtractReceipt(BaseModel):
class AskMemoryRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
question: str = Field(min_length=1, max_length=2_000)
scope: str = Field(default="all-apps", min_length=1, max_length=512)
# Host-injected project directory, same boundary as recall. Optional so
# the web UI and older callers keep working unchanged.
workspace_path: str | None = Field(default=None, max_length=4_096)
#: Whose memories answer. Omitted, it is `local`, the App's library.
user_id: UserId = DEFAULT_USER_ID
class SdkHomeSummary(BaseModel):
@@ -626,6 +651,27 @@ class MemoryGraphResponse(BaseModel):
hy_graph_available: bool = True
class SourceGrantView(BaseModel):
"""One `(user_id, source_id)` grant, as `GET /api/v1/source-grants` lists it."""
model_config = ConfigDict(frozen=True, extra="forbid")
user_id: str
source_id: str
root_label: str
root_path: str
authorized: bool
revoked_at: datetime | None
sync_enabled: bool
class SourceGrantList(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
schema_version: Literal[1] = 1
grants: tuple[SourceGrantView, ...]
class SdkMigrationRequired(RuntimeError):
"""A serving Core was started on a home that is not on SDK authority."""
@@ -928,6 +974,7 @@ def create_app(
write_queue=memory_write_queue,
extraction_settings=app.state.extraction_settings_store,
data_dir=runtime_settings.data_dir,
user_ids=lambda: _known_user_ids(app),
)
# C2 owns the single production budget (handle 5s + reserve 250ms,
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
@@ -1100,6 +1147,13 @@ def create_app(
source_id
for source_id, registration in app.state.source_catalog.items()
if registration.capabilities.live_sync is CapabilityState.IMPLEMENTED
) + tuple(
lane
for lane in _other_user_lanes(
app.state.source_catalog, app.state.source_grant_store
)
if app.state.source_catalog[split_lane(lane)[1]].capabilities.live_sync
is CapabilityState.IMPLEMENTED
)
# Resolved before the supervisor is built so it can state truthfully
# how it is being driven: watching only applies to roots that exist
@@ -1496,9 +1550,12 @@ def create_app(
Depends(local_api_guard(runtime_settings, valid_ui_session=ui_session_valid)),
Depends(require_external_endpoint),
]
app.include_router(memory_add_router(protected))
# Memory reads and writes refuse a query parameter they do not declare: an
# ignored `user_id` would read as isolation that never happened.
strict = [*protected, Depends(forbid_unknown_query_params)]
app.include_router(memory_add_router(strict))
app.include_router(file_import_router(protected))
app.include_router(sdk_context_router(protected))
app.include_router(sdk_context_router(strict))
# A4-01: scan-root configuration, so the CLI's `path` command has something
# to read instead of refusing.
@@ -1512,7 +1569,7 @@ def create_app(
if getattr(app.state, "memory_write_registry", None) is None:
app.state.memory_write_registry = WriteRegistry()
app.include_router(sdk_memory_write_router(protected))
app.include_router(sdk_memory_write_router(strict))
from one_memory.call_event_routes import call_event_router
app.include_router(call_event_router(protected))
@@ -2287,7 +2344,11 @@ def create_app(
return SourceSyncState(source_id=source_id, sync_enabled=request.enabled)
async def _prepare_source_resume(source_id: str) -> None:
"""Prepare catch-up work before committing the new desired state."""
"""Prepare catch-up work before committing the new desired state.
`source_id` is a lane key, so another user's lane prepares that
user's own cursors and history job.
"""
await _live_sync(app).initialize_source(source_id, baseline_missing_files=True)
active = {
ImportStatus.PENDING,
@@ -2297,14 +2358,17 @@ def create_app(
ImportStatus.FAILED,
}
if any(
job.source_id == source_id and job.operation == "history" and job.status in active
lane_key(job.user_id, job.source_id) == source_id
and job.operation == "history"
and job.status in active
for job in _job_store(app).list_jobs()
):
return
user_id, bare_source_id = split_lane(source_id)
await asyncio.to_thread(
_job_store(app).create,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
operation="history",
)
@@ -2312,7 +2376,7 @@ def create_app(
"""Run work prepared for a source after config.json has committed."""
try:
for job in _job_store(app).list_jobs():
if job.source_id != source_id:
if lane_key(job.user_id, job.source_id) != source_id:
continue
if job.status is ImportStatus.PENDING:
_supervisor(app).enqueue(job.job_id)
@@ -2341,8 +2405,13 @@ def create_app(
response_model=SourceGrantReceipt,
dependencies=protected,
)
async def revoke_source(source_id: str) -> SourceGrantReceipt:
async def revoke_source(
source_id: str,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> SourceGrantReceipt:
_require_registration(app, source_id)
if user_id != DEFAULT_USER_ID:
return await _revoke_other_user_grant(source_id, user_id)
receipt = _grant_store(app).receipt(
user_id="local",
source_id=source_id,
@@ -2378,6 +2447,65 @@ def create_app(
await _refresh_live_watch(app)
return receipt
async def _revoke_other_user_grant(source_id: str, user_id: str) -> SourceGrantReceipt:
"""Stop reading another user's root. The App's switch is left alone."""
if source_id not in MULTI_USER_SOURCES:
raise HTTPException(status_code=422, detail="user_id_requires_multi_user_source")
lane = lane_key(user_id, source_id)
receipt = await asyncio.to_thread(
partial(_grant_store(app).receipt, user_id=user_id, source_id=source_id)
)
if receipt is None:
raise HTTPException(status_code=404, detail="grant_not_found")
active_statuses = {
ImportStatus.PENDING,
ImportStatus.RUNNING,
ImportStatus.PAUSED,
ImportStatus.INTERRUPTED,
ImportStatus.FAILED,
}
for job in _job_store(app).list_jobs():
if lane_key(job.user_id, job.source_id) == lane and job.status in active_statuses:
await _supervisor(app).cancel(job.job_id)
await _live_sync(app).stop_source(lane)
_supervisor(app).unregister_adapter(lane)
receipt = await asyncio.to_thread(
partial(_grant_store(app).revoke, user_id=user_id, source_id=source_id)
)
await _refresh_live_watch(app)
return receipt
@app.get(
"/api/v1/source-grants",
response_model=SourceGrantList,
dependencies=protected,
)
async def list_source_grants() -> SourceGrantList:
"""Every grant of every user. `/api/v1/sources` stays `local`'s, as the App reads it."""
def describe() -> SourceGrantList:
config_store = _config_store(app)
views = []
for receipt, root_path in _grant_store(app).grants():
lane = lane_key(receipt.user_id, receipt.source_id)
authorized = receipt.revoked_at is None and receipt.revision > 0
views.append(
SourceGrantView(
user_id=receipt.user_id,
source_id=receipt.source_id,
root_label=receipt.root_label,
root_path=_display_candidate_path(
Path(root_path), runtime_settings.source_home
),
authorized=authorized,
revoked_at=receipt.revoked_at,
sync_enabled=authorized and _lane_sync_enabled(config_store, lane),
)
)
return SourceGrantList(grants=tuple(views))
return await asyncio.to_thread(describe)
@app.post(
"/api/v1/import-jobs",
response_model=ImportJob,
@@ -2406,6 +2534,8 @@ def create_app(
async def connect_sources(
request: ConnectSourcesRequest,
) -> ConnectSourcesResponse:
if request.user_id != DEFAULT_USER_ID or request.root is not None:
return await _connect_other_root(request)
if request.onboarding:
app.state.history_import_gate_released = False
source_ids = tuple(dict.fromkeys(request.source_ids))
@@ -2511,6 +2641,103 @@ def create_app(
raise
return ConnectSourcesResponse(**result.model_dump())
async def _connect_other_root(request: ConnectSourcesRequest) -> ConnectSourcesResponse:
"""Grant one Hermes root to a user: `~/.hermes/profiles/<name>` for its own user.
Only Hermes holds more than one person's history (one profile per
person), so a non-`local` user or a caller-chosen root is refused for
every other source. The grant is `(user_id, source_id)`; its cursors,
jobs and live sync follow that key.
"""
source_ids = tuple(dict.fromkeys(request.source_ids))
if any(source_id not in MULTI_USER_SOURCES for source_id in source_ids):
raise HTTPException(status_code=422, detail="user_id_requires_multi_user_source")
if request.root is not None and len(source_ids) != 1:
raise HTTPException(status_code=422, detail="source_root_requires_one_source")
if request.user_id != DEFAULT_USER_ID and request.root is None:
# The user is the root that is read: another user must name theirs,
# or they would be handed `local`'s history.
raise HTTPException(status_code=422, detail="source_root_required_for_user")
if request.onboarding or request.import_upload_id is not None:
raise HTTPException(status_code=422, detail="user_id_unsupported_for_request")
if request.request_id is None:
raise HTTPException(status_code=409, detail="source_confirmation_required")
registrations = []
source_roots = dict(request.source_roots)
for source_id in source_ids:
registration = _require_registration(app, source_id)
if request.root is not None:
root = Path(request.root)
if not root.is_absolute():
raise HTTPException(status_code=422, detail="source_root_must_be_absolute")
# Named outright by the caller, like an environment override:
# no saved candidate may replace it, and naming it confirms it.
registration = replace(
registration,
candidate=SourceRootCandidate(
path=root, kind=RootCandidateKind.ENVIRONMENT_OVERRIDE
),
)
try:
source_roots[source_id] = await asyncio.to_thread(source_root_identity, root)
except SourceAccessError as error:
raise HTTPException(status_code=409, detail=error.code) from None
registrations.append(registration)
existing_intent = await asyncio.to_thread(
partial(
app.state.source_connection_store.get,
request.request_id,
user_id=request.user_id,
)
)
try:
result = await asyncio.to_thread(
partial(
app.state.source_connection_store.connect,
request_id=request.request_id,
registrations=tuple(registrations),
source_roots=source_roots,
confirm_overrides=(
request.confirm_environment_overrides or request.root is not None
),
full_rescan=request.full_rescan,
user_id=request.user_id,
)
)
except ConnectionIntentError as error:
raise HTTPException(status_code=409, detail=str(error)) from None
if existing_intent is not None:
return ConnectSourcesResponse(**result.model_dump())
for job in result.jobs:
registration = _require_registration(app, job.source_id)
grant = await asyncio.to_thread(
partial(
_grant_store(app).authorized_root,
user_id=job.user_id,
source_id=job.source_id,
)
)
if grant is None:
continue
lane = lane_key(job.user_id, job.source_id)
_supervisor(app).register_adapter(
registration.create_adapter(grant), user_id=job.user_id
)
_detection(app).pop(lane, None)
if job.user_id != DEFAULT_USER_ID:
if registration.capabilities.live_sync is CapabilityState.IMPLEMENTED:
_live_sync(app).ensure_source(lane)
# Another user's grant syncs while it stands (see
# `_lane_sync_enabled`); the App's switch is not touched.
await _prepare_source_resume(lane)
await _activate_source_resume(lane)
elif request.enable_sync:
await _prepare_source_resume(lane)
await asyncio.to_thread(_config_store(app).set_sync, job.source_id, True)
await _activate_source_resume(lane)
await _refresh_live_watch(app)
return ConnectSourcesResponse(**result.model_dump())
@app.get(
"/api/v1/source-connections/{request_id}",
response_model=ConnectionIntentReceipt,
@@ -2527,8 +2754,12 @@ def create_app(
response_model=list[ImportJob],
dependencies=protected,
)
def list_import_jobs() -> list[ImportJob]:
return _job_store(app).list_jobs()
def list_import_jobs(
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> list[ImportJob]:
# One user's jobs: the App reads `local`'s and must not show another
# user's Hermes import as its own.
return [job for job in _job_store(app).list_jobs() if job.user_id == user_id]
@app.get(
"/api/v1/live-sync",
@@ -2857,9 +3088,11 @@ def create_app(
@app.get(
"/api/v2/home-summary",
response_model=SdkHomeSummary,
dependencies=protected,
dependencies=strict,
)
async def sdk_home_summary() -> SdkHomeSummary:
async def sdk_home_summary(
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> SdkHomeSummary:
"""The home card: the memory page's count and its newest memory.
The count is `memory_stats` total (chat + working + coding
@@ -2871,12 +3104,12 @@ def create_app(
client = await _sdk_client_handle(app).client()
stats = await asyncio.to_thread(
client.memory_stats,
user_id="local",
user_id=user_id,
statuses=["active"],
)
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
scene="all",
statuses=["active"],
limit=1,
@@ -2897,7 +3130,7 @@ def create_app(
@app.get(
"/api/v2/memory-library",
response_model=HySdkMemoryPage,
dependencies=protected,
dependencies=strict,
)
async def sdk_memory_library(
statuses: list[str] = Query(default=["active"]), # noqa: B008
@@ -2907,6 +3140,7 @@ def create_app(
limit: int = Query(default=LIBRARY_PAGE_SIZE, ge=1, le=500),
window_id: str | None = Query(default=None, min_length=1, max_length=64),
scene: Literal["all", "chat", "working", "coding"] = Query(default="all"),
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> HySdkMemoryPage:
"""One page of memories across chat, working and coding.
@@ -2941,10 +3175,11 @@ def create_app(
limit=limit,
window_id=window_id,
scene=scene,
user_id=user_id,
)
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
scene=scene,
statuses=list(statuses),
include_sources=[source] if source else None,
@@ -2965,12 +3200,13 @@ def create_app(
@app.get(
"/api/v2/memories/graph",
response_model=MemoryGraphResponse,
dependencies=protected,
dependencies=strict,
)
async def sdk_memory_graph(
limit: int = Query(default=160, ge=1, le=300),
offset: int = Query(default=0, ge=0),
source: str | None = Query(default=None, min_length=1),
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryGraphResponse:
"""The memory map on SDK authority: SDK memories plus the Kuzu overlay.
@@ -2984,7 +3220,7 @@ def create_app(
# left list's 20-row pagination window.
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
scene="all",
statuses=["active"],
include_sources=[source] if source else None,
@@ -3054,7 +3290,9 @@ def create_app(
# source withholds the overlay, as on the legacy map.
if not await asyncio.to_thread(_read_access(app).has_revoked_sources):
try:
hy_graph = await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
hy_graph = await _sdk_memory_runtime(app).graph_snapshot(
limit=limit, user_id=user_id
)
except HySdkError:
hy_graph = None
if hy_graph is not None:
@@ -3104,12 +3342,14 @@ def create_app(
@app.get(
"/api/v2/user-profile",
response_model=SdkUserProfileResponse,
dependencies=protected,
dependencies=strict,
)
async def sdk_user_profile() -> SdkUserProfileResponse:
async def sdk_user_profile(
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> SdkUserProfileResponse:
try:
client = await _sdk_client_handle(app).client()
value = await asyncio.to_thread(client.get_user_profile, user_id="local")
value = await asyncio.to_thread(client.get_user_profile, user_id=user_id)
profile = HySdkUserProfile.from_sdk(value)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_profile_read_failed") from None
@@ -3118,18 +3358,23 @@ def create_app(
@app.get(
"/api/v2/memories/{memory_id}",
response_model=HySdkMemoryRecord,
dependencies=protected,
dependencies=strict,
)
async def get_sdk_memory(
memory_id: str,
kind: SdkMemoryKind | None = None,
variant: SdkMemoryVariant | None = None,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> HySdkMemoryRecord:
_check_memory_namespace(kind, variant)
try:
memory = await _get_sdk_memory(
_sdk_client_handle(app), memory_id, kind=kind, variant=variant
)
# The SDK's `get` takes no user: another user's memory reads as
# missing, never as existing.
if memory is not None and not owns(memory.user_id, user_id):
memory = None
except AmbiguousMemoryIdError:
raise HTTPException(status_code=409, detail="memory_id_ambiguous") from None
except Exception as error:
@@ -3141,7 +3386,7 @@ def create_app(
@app.put(
"/api/v2/memories/{memory_id}",
response_model=HySdkMemoryRecord,
dependencies=protected,
dependencies=strict,
)
async def update_sdk_memory(
memory_id: str,
@@ -3153,6 +3398,13 @@ def create_app(
namespace = _namespace_kwargs(request.kind, request.variant)
try:
async with app.state.sdk_write_coordinator.use_client() as client:
# `update` takes no user: read the memory first and refuse
# another user's id exactly as a missing one.
current = await asyncio.to_thread(client.get, memory_id, **namespace)
if current is None or not owns(
HySdkMemoryRecord.from_sdk(current).user_id, request.user_id
):
raise HTTPException(status_code=404, detail="memory_not_found")
update_kwargs: dict[str, object] = dict(namespace)
if request.task is not None:
update_kwargs["task"] = request.task
@@ -3190,16 +3442,27 @@ def create_app(
@app.delete(
"/api/v2/memories/{memory_id}",
status_code=204,
dependencies=protected,
dependencies=strict,
)
async def delete_sdk_memory(
memory_id: str,
kind: SdkMemoryKind | None = None,
variant: SdkMemoryVariant | None = None,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> Response:
_check_memory_namespace(kind, variant)
try:
async with app.state.sdk_write_coordinator.use_client() as client:
# `delete` takes no user. Another user's memory is answered
# exactly like an id that does not exist: 204, nothing
# touched, so the status cannot tell that it exists.
current = await asyncio.to_thread(
client.get, memory_id, **_namespace_kwargs(kind, variant)
)
if current is not None and not owns(
HySdkMemoryRecord.from_sdk(current).user_id, user_id
):
return Response(status_code=204)
result = await asyncio.to_thread(
client.delete, memory_id, **_namespace_kwargs(kind, variant)
)
@@ -3210,6 +3473,8 @@ def create_app(
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
if result.get("success") is not True and result.get("deleted_count") != 0:
raise HySdkError("hy_memory_delete_rejected")
except HTTPException:
raise
except AmbiguousMemoryIdError:
raise HTTPException(status_code=409, detail="memory_id_ambiguous") from None
except Exception as error:
@@ -3219,18 +3484,19 @@ def create_app(
@app.get(
"/api/v2/raw",
response_model=SdkRawPage,
dependencies=protected,
dependencies=strict,
)
async def sdk_raw_library(
source: str | None = Query(default=None, min_length=1),
offset: int = Query(default=0, ge=0),
limit: int = Query(default=100, ge=1, le=500),
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> SdkRawPage:
try:
client = await _sdk_client_handle(app).client()
value = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
layers=["l1_raw"],
include_sources=[source] if source else None,
offset=offset,
@@ -3244,7 +3510,7 @@ def create_app(
@app.post(
"/api/v2/memories/ask",
response_model=SdkMemoryAnswer,
dependencies=protected,
dependencies=strict,
)
async def ask_sdk_memories(
request: AskMemoryRequest,
@@ -3261,7 +3527,9 @@ def create_app(
"om_ask",
)
try:
answer = await _sdk_request_answerer(app).answer(request.question)
answer = await _sdk_request_answerer(app).answer(
request.question, user_id=request.user_id
)
except MemoryAnswerError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
if request.question != "__om_probe__" and access is not None:
@@ -3278,7 +3546,7 @@ def create_app(
@app.post(
"/api/v2/memories/ask/stream",
dependencies=protected,
dependencies=strict,
)
async def ask_sdk_memories_stream(
request: AskMemoryRequest,
@@ -3300,7 +3568,9 @@ def create_app(
final: dict[str, object] | None = None
try:
async for event in answerer.answer_stream(request.question):
async for event in answerer.answer_stream(
request.question, user_id=request.user_id
):
if event.get("type") == "final":
final = event
yield _sse(_serialize_ask_event(event))
@@ -3331,7 +3601,7 @@ def create_app(
@app.post(
"/api/v1/recall",
response_model=RecallReceipt,
dependencies=protected,
dependencies=strict,
)
async def recall_memories(
request: RecallRequest,
@@ -3344,16 +3614,20 @@ def create_app(
await _require_external_recall_scope(
app, http_request, request.scope, request.workspace_path
)
target = _recall_target(app)
target = (
_recall_target(app)
if request.user_id == DEFAULT_USER_ID
else _sdk_recall_target(app, request.user_id)
)
access = external_access(http_request.headers)
receipt = await recall_with_fallback(
request,
model_ready=lambda: _recall_model_ready(app),
recall_memories=target.recall_async,
has_memories=lambda: _library_has_memories(app),
has_memories=lambda: _library_has_memories(app, request.user_id),
raw_sources=lambda: _recall_raw_sources(app, request, access, started),
search_raw=lambda query, limit, sources: _search_sdk_raw(
app, query, limit=limit, sources=sources
app, query, limit=limit, sources=sources, user_id=request.user_id
),
)
event_store = getattr(app.state, "call_event_store", None)
@@ -3730,21 +4004,60 @@ def _sync_refused(code: str) -> HTTPException:
return HTTPException(status_code=409, detail=code)
def _lane_sync_enabled(config_store: ConfigStore, lane: str) -> bool:
"""Whether a lane may sync, before its grant is asked.
`local` keeps the App's per-source switch in the configuration document,
which stays global. Another user's Hermes grant syncs for as long as it
stands: connecting it is the consent, revoking it stops it, and it never
turns the App's switch on or off.
"""
user_id, source_id = split_lane(lane)
if user_id == DEFAULT_USER_ID:
return config_store.sync_enabled(source_id)
return source_id in MULTI_USER_SOURCES
def _other_user_lanes(
catalog: dict[str, SourceRegistration],
grant_store: SourceGrantStore,
) -> tuple[str, ...]:
"""Lanes of every unrevoked grant held by a user other than `local`."""
return tuple(
lane_key(receipt.user_id, receipt.source_id)
for receipt, _root in grant_store.grants()
if receipt.user_id != DEFAULT_USER_ID
and receipt.revoked_at is None
and receipt.source_id in MULTI_USER_SOURCES
and receipt.source_id in catalog
)
def _known_user_ids(app: FastAPI) -> tuple[str, ...]:
"""`local` and every user holding a grant: the users whose data Core keeps."""
users = [DEFAULT_USER_ID]
for receipt, _root in _grant_store(app).grants():
if receipt.user_id not in users:
users.append(receipt.user_id)
return tuple(users)
def _granted_roots(
source_ids: Iterable[str],
grant_store: SourceGrantStore,
config_store: ConfigStore,
) -> dict[str, Path]:
roots: dict[str, Path] = {}
for source_id in source_ids:
if not config_store.sync_enabled(source_id):
for lane in source_ids:
if not _lane_sync_enabled(config_store, lane):
continue
user_id, source_id = split_lane(lane)
try:
grant = grant_store.authorized_root(user_id="local", source_id=source_id)
grant = grant_store.authorized_root(user_id=user_id, source_id=source_id)
except SourceAccessError:
continue
if grant is not None:
roots[source_id] = grant.root
roots[lane] = grant.root
return roots
@@ -3753,6 +4066,14 @@ def _active_adapters(
grant_store: SourceGrantStore,
) -> dict[str, SourceAdapter]:
adapters: dict[str, SourceAdapter] = {}
for lane in _other_user_lanes(catalog, grant_store):
user_id, source_id = split_lane(lane)
try:
grant = grant_store.authorized_root(user_id=user_id, source_id=source_id)
if grant is not None:
adapters[lane] = catalog[source_id].create_adapter(grant)
except SourceAccessError:
continue
for source_id, registration in catalog.items():
try:
grant = grant_store.authorized_root(
@@ -3915,10 +4236,14 @@ def _config_store(app: FastAPI) -> ConfigStore:
def _source_processing_enabled(app: FastAPI, source_id: str) -> bool:
if not _config_store(app).sync_enabled(source_id):
"""`source_id` is a lane key: another user's lane asks that user's grant."""
if not _lane_sync_enabled(_config_store(app), source_id):
return False
user_id, bare_source_id = split_lane(source_id)
try:
return _grant_store(app).authorized_root(user_id="local", source_id=source_id) is not None
return (
_grant_store(app).authorized_root(user_id=user_id, source_id=bare_source_id) is not None
)
except SourceAccessError:
return False
@@ -4437,13 +4762,32 @@ async def _recall_model_ready(app: FastAPI) -> bool:
return profile is not None and bool(profile.capabilities.model_available)
async def _library_has_memories(app: FastAPI) -> bool:
def _sdk_recall_target(app: FastAPI, user_id: str) -> SdkRecallTarget:
"""Recall over one user's SDK memories; `get` results are kept only if theirs."""
handle = _sdk_client_handle(app)
async def owned(memory_id: str) -> HySdkMemoryRecord | None:
memory = await _get_sdk_memory(handle, memory_id)
return memory if memory is not None and owns(memory.user_id, user_id) else None
return SdkRecallTarget(
semantic_search=lambda query: _search_sdk_memories(
handle,
query,
user_id=user_id,
agent_id="*",
),
get_memory=owned,
)
async def _library_has_memories(app: FastAPI, user_id: str = DEFAULT_USER_ID) -> bool:
"""Any active memory at all. Asked only after an empty memory search."""
try:
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
statuses=["active"],
limit=1,
offset=0,
@@ -4501,13 +4845,14 @@ async def _search_sdk_raw(
*,
limit: int,
sources: frozenset[str] | None,
user_id: str = DEFAULT_USER_ID,
) -> tuple[HySdkRawHit, ...]:
"""The imported conversations nearest the query: the SDK raw layer."""
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.search,
query,
user_ids=["local"],
user_ids=[user_id],
agent_ids=[],
include_sources=sorted(sources) if sources else None,
limit=limit,
@@ -4610,6 +4955,7 @@ async def _library_search_page(
limit: int,
window_id: str | None,
scene: str = "all",
user_id: str = DEFAULT_USER_ID,
) -> HySdkMemoryPage:
"""One page of a keyword search, cut from a cached result window.
@@ -4621,7 +4967,9 @@ async def _library_search_page(
SDK so edits and deletes show. The whole library is never listed in Core.
"""
windows = _library_search_windows(app)
key = window_key(query, source, statuses, scene)
# The user is part of the window's identity: one user's `window_id` must
# never cut a page from another user's matches.
key = window_key(query, source, statuses, scene, user_id)
window = windows.get(window_id, key) if window_id else None
window_reset = window_id is not None and window is None
if window is None:
@@ -4631,7 +4979,7 @@ async def _library_search_page(
query,
scene="all",
rerank=False,
user_ids=["local"],
user_ids=[user_id],
agent_ids=[],
include_sources=[source] if source else None,
limit=fetch,
@@ -4667,7 +5015,7 @@ async def _library_search_page(
except HySdkReadError:
skipped += 1
continue
if record.status == "active":
if record.status == "active" and owns(record.user_id, user_id):
memories.append(record)
_log_skipped_sdk_memories(skipped)
return HySdkMemoryPage(
@@ -4724,8 +5072,10 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
forever with attempts=0 while later jobs ran. Intake, claim and execute
all ask this one question.
"""
if source_id == "one-memory-manual":
receipt = _grant_store(app).receipt(user_id="local", source_id=source_id)
if split_lane(source_id)[1] == "one-memory-manual":
# One switch for hand-added memories, whoever adds them: it is a
# management setting, and management stays global.
receipt = _grant_store(app).receipt(user_id="local", source_id="one-memory-manual")
return receipt is None or receipt.revoked_at is None
return _source_processing_enabled(app, source_id)
@@ -4759,6 +5109,7 @@ def _history_import_gate_released(app: FastAPI) -> bool:
job
for job in jobs.list_jobs()
if job.operation == "history"
and job.user_id == DEFAULT_USER_ID
and job.source_id in selected
and (state.started_at is None or job.created_at >= state.started_at)
]
@@ -5,6 +5,8 @@ from typing import Annotated, Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
from one_memory.user_scope import DEFAULT_USER_ID, UserId
ContextKind = Literal["raw", "memory"]
ContextSearchChannel = Literal["lexical", "semantic"]
ContextSemanticStatus = Literal["not_searched", "completed", "failed", "timed_out"]
@@ -58,6 +60,8 @@ class SdkContextSearchRequest(ContextSearchRequest):
limit: int = Field(default=10, ge=1, le=20, strict=True)
schema_limit: int = Field(default=3, ge=0, le=20, strict=True)
intention_limit: int = Field(default=1, ge=0, le=20, strict=True)
#: Whose memories to search. The App never sends it and reads `local`.
user_id: UserId = DEFAULT_USER_ID
class ContextMessage(BaseModel):
@@ -28,6 +28,7 @@ from one_memory.hy_sdk_types import (
parse_sdk_search,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.user_scope import owns
CALL_EVENT_RESPONSE_BUDGET_MS = 250
@@ -73,7 +74,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
raw = await asyncio.to_thread(
client.search,
value.query,
user_ids=["local"],
user_ids=[value.user_id],
agent_ids=[],
include_sources=list(authority.raw_sources or ()),
limit=value.limit,
@@ -108,7 +109,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
raw = await asyncio.to_thread(
client.search,
value.query,
user_ids=["local"],
user_ids=[value.user_id],
agent_ids=[],
limit=value.limit,
min_score=sdk_relevance_floor(),
@@ -123,7 +124,12 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
for hit in hits:
record = await asyncio.to_thread(client.get, hit.memory_id)
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
if memory is None or memory.status != "active":
# `get` takes no user: a hit is kept only if it is the caller's.
if (
memory is None
or memory.status != "active"
or not owns(memory.user_id, value.user_id)
):
continue
items.append(
SdkMemoryContextItem(
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.8"
HY_MEMORY_VERSION = "2.0.9"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -497,7 +497,7 @@ def _ensure_coding_schema(path: Path) -> None:
CodingMemoryStore,
)
# Relies on the pinned SDK's internals (hy-memory==2.0.8 is pinned exactly).
# Relies on the pinned SDK's internals (hy-memory==2.0.9 is pinned exactly).
store = CodingMemoryStore.__new__(CodingMemoryStore)
store._db_path = str(path)
store._conn = None
@@ -31,6 +31,16 @@ class ScanIssueCode(StrEnum):
#: import it. Named for the reason, so a reader can tell a setting from
#: a record the parser could not use.
SUBAGENT_TRACE_DISABLED = "subagent_trace_disabled"
#: A host's own notice wearing the user's role — Hermes posts a finished
#: background process, and its gateway marks the post as not written by a
#: human. Nothing in it is the person's, so it never reaches extraction.
INJECTED_MACHINE_NOTICE = "injected_machine_notice"
#: The host could not transcribe a voice message and left a placeholder
#: naming the audio file. The person did speak, but the placeholder holds
#: only a path, which an extractor would otherwise read as a fact.
VOICE_PLACEHOLDER = "voice_placeholder"
#: A health check such as "Reply with exactly: PONG": no one's words.
PROBE = "probe"
SOURCE_BUSY = "source_busy"
UNSUPPORTED_SCHEMA = "unsupported_schema"
BUDGET_EXCEEDED = "budget_exceeded"
@@ -27,6 +27,7 @@ def discovered_off_loop(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
) -> AsyncIterator[SourceFileDiscoveryEvent]:
"""The same walk, for callers that have an event loop to keep free.
@@ -41,6 +42,7 @@ def discovered_off_loop(
grant,
suffixes=suffixes,
max_targets=max_targets,
excluded_top_directories=excluded_top_directories,
),
scope_key=grant.source_id,
)
@@ -51,12 +53,15 @@ def discover_source_files(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
) -> Iterator[SourceFileDiscoveryEvent]:
"""Metadata only: one lstat per candidate, no open and no per-file grant lookup.
The grant and root identity are checked once before the walk. Each listed
file keeps the grant's check, which runs again when that file is opened;
the open itself refuses symlinks and any inode other than the one listed.
`excluded_top_directories` are directories directly under the root that
are never entered, such as another user's Hermes profiles.
"""
grant.check_root()
inspected = 0
@@ -64,8 +69,12 @@ def discover_source_files(
grant.root,
followlinks=False,
):
at_root = Path(directory) == grant.root
child_directories[:] = sorted(
child for child in child_directories if not (Path(directory) / child).is_symlink()
child
for child in child_directories
if not (Path(directory) / child).is_symlink()
and not (at_root and child in excluded_top_directories)
)
for filename in sorted(filenames):
if Path(filename).suffix.lower() not in suffixes:
@@ -252,6 +252,18 @@ class SourceGrantStore:
).fetchone()
return self._to_receipt(row) if row else None
def grants(self) -> tuple[tuple[SourceGrantReceipt, str], ...]:
"""Every grant of every user, with its granted root path.
One source may hold several grants, one per `(user_id, source_id)`:
a Hermes root for `local` and a Hermes profile for another user.
"""
with self._connect() as connection:
rows = connection.execute(
"SELECT * FROM source_grants ORDER BY source_id, user_id"
).fetchall()
return tuple((self._to_receipt(row), str(row["requested_path"])) for row in rows)
def require_receipt(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
receipt = self.receipt(user_id=user_id, source_id=source_id)
if receipt is None:
@@ -28,6 +28,7 @@ from one_memory.imports.adapters import (
)
from one_memory.imports.jobs import ImportJob, ImportJobStore, ImportStatus
from one_memory.imports.supervisor import ImportSupervisor
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, split_lane
#: How long an automatic retry will wait for a wedged worker to notice cancel.
ABANDON_TIMEOUT_SECONDS = 5.0
@@ -64,7 +65,12 @@ class LiveSyncHealth(BaseModel):
class PollingLiveSyncSupervisor:
"""Near-live sync that reuses each adapter's persisted incremental cursor."""
"""Near-live sync that reuses each adapter's persisted incremental cursor.
A "source" here is a lane key (`user_scope.lane_key`): the bare source id
for `local`, `source@user` for another user's grant. Cursors, jobs and
locks are therefore per `(user_id, source_id)`.
"""
def __init__(
self,
@@ -92,8 +98,22 @@ class PollingLiveSyncSupervisor:
}
self._locks = {source_id: asyncio.Lock() for source_id in self.source_ids}
self._task: asyncio.Task[None] | None = None
self._group: asyncio.TaskGroup | None = None
self._observers: set[asyncio.Task[None]] = set()
def ensure_source(self, source_id: str) -> None:
"""Follow a lane granted after start, such as a new Hermes profile."""
if source_id in self._locks:
return
self.source_ids = (*self.source_ids, source_id)
self._health.setdefault(
source_id,
LiveSyncHealth(source_id=source_id, mode=self.mode, state=LiveSyncState.IDLE),
)
self._locks[source_id] = asyncio.Lock()
if self._group is not None:
self._group.create_task(self._poll_source(source_id))
def _set_health(self, source_id: str, value: LiveSyncHealth) -> None:
previous = self._health.get(source_id)
self._health[source_id] = value
@@ -152,7 +172,12 @@ class PollingLiveSyncSupervisor:
self._watched_sources = frozenset(source_ids)
def all_health(self) -> tuple[LiveSyncHealth, ...]:
return tuple(self.health(source_id) for source_id in self.source_ids)
"""`local`'s lanes only: what the App has always been shown."""
return tuple(
self.health(source_id)
for source_id in self.source_ids
if split_lane(source_id)[0] == DEFAULT_USER_ID
)
async def initialize_source(
self, source_id: str, *, baseline_missing_files: bool = True
@@ -164,11 +189,12 @@ class PollingLiveSyncSupervisor:
baselines = (
await adapter.live_baseline() if isinstance(adapter, _LiveBaselineAdapter) else {}
)
user_id, bare_source_id = split_lane(source_id)
await in_the_background(
partial(
self.job_store.ensure_live_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
baselines=baselines,
baseline_missing_files=baseline_missing_files,
)
@@ -228,7 +254,7 @@ class PollingLiveSyncSupervisor:
(
job
for job in reversed(source_jobs)
if job.source_id == source_id
if lane_key(job.user_id, job.source_id) == source_id
and job.operation == "sync"
and job.status
in {
@@ -277,13 +303,13 @@ class PollingLiveSyncSupervisor:
competing = [
job
for job in source_jobs
if job.source_id == source_id
if lane_key(job.user_id, job.source_id) == source_id
and job.operation != "sync"
and job.status
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
]
bounded_history_incomplete = any(
job.source_id == source_id
lane_key(job.user_id, job.source_id) == source_id
and job.operation == "history"
and job.history_after is not None
and job.status is not ImportStatus.COMPLETED
@@ -379,11 +405,12 @@ class PollingLiveSyncSupervisor:
await close()
async def _live_cursors(self, source_id: str, adapter: object) -> dict[str, SourceCursor]:
user_id, bare_source_id = split_lane(source_id)
cursors = await in_the_background(
partial(
self.job_store.get_source_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
operation="sync",
)
)
@@ -395,8 +422,8 @@ class PollingLiveSyncSupervisor:
return await in_the_background(
partial(
self.job_store.ensure_live_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
baselines=baselines,
)
)
@@ -520,9 +547,13 @@ class PollingLiveSyncSupervisor:
async def _run(self) -> None:
# A source owns its cadence; one large catch-up cannot delay every
# other source. The group keeps shutdown responsible for every poller.
async with asyncio.TaskGroup() as group:
for source_id in self.source_ids:
group.create_task(self._poll_source(source_id))
try:
async with asyncio.TaskGroup() as group:
self._group = group
for source_id in self.source_ids:
group.create_task(self._poll_source(source_id))
finally:
self._group = None
async def _poll_source(self, source_id: str) -> None:
while True:
@@ -3,7 +3,7 @@ import logging
import math
import sqlite3
from collections import Counter
from collections.abc import Callable
from collections.abc import AsyncIterator, Callable
from contextlib import suppress
from contextvars import ContextVar
from dataclasses import dataclass, field
@@ -227,14 +227,17 @@ class ImportRunner:
heartbeat = asyncio.create_task(self._keep_alive(job_id=job_id, runner_token=runner_token))
pending = _PendingSkips()
writes = _PendingWrites()
stream = adapter.scan(
ScanOptions(
cursors=cursors,
reuse_listing=(
job.operation == "history"
or (job.operation == "sync" and job.source_id == "workbuddy")
),
)
stream = _owned_by(
adapter.scan(
ScanOptions(
cursors=cursors,
reuse_listing=(
job.operation == "history"
or (job.operation == "sync" and job.source_id == "workbuddy")
),
)
),
job.user_id,
)
slice_cursors = dict(cursors)
next_event: asyncio.Future[ScanEvent] | None = None
@@ -924,3 +927,30 @@ async def _settled_work[T](
with suppress(asyncio.CancelledError, Exception):
task.result()
raise
def _owned_by(stream: AsyncIterator[ScanEvent], user_id: str) -> AsyncIterator[ScanEvent]:
"""Stamp every scanned record with the user whose root is being read.
Adapters know nothing of users and emit `local`. Which user a record
belongs to is decided by the grant the job reads (a Hermes profile's
`state.db` is that profile's user), so the job's user is the record's.
`local` passes the stream through untouched.
"""
if user_id == "local":
return stream
return _restamped(stream, user_id)
async def _restamped(stream: AsyncIterator[ScanEvent], user_id: str) -> AsyncIterator[ScanEvent]:
try:
async for event in stream:
if isinstance(event, ScannedEvidence) and event.evidence.user_id != user_id:
event = event.model_copy(
update={"evidence": event.evidence.model_copy(update={"user_id": user_id})}
)
yield event
finally:
close = getattr(stream, "aclose", None)
if close is not None:
await close()
@@ -1,4 +1,5 @@
import json
import re
import sqlite3
from collections.abc import AsyncIterator, Iterator
from functools import partial
@@ -87,6 +88,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=10_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
):
if isinstance(item, AuthorizedSourceFile) and _is_hermes_target(
item.relative_path.as_posix()
@@ -100,6 +102,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=options.max_scan_targets or 100_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
):
yield ScanProgressEvent(
phase=ScanPhase.DISCOVER,
@@ -214,7 +217,11 @@ def parse_hermes_rollout_record(
message.get("content"),
allowed_block_types=frozenset({"text", "input_text", "output_text"}),
)
text = strip_injected_memory(text)
text = strip_injected_memory(text or "")
if role == "user":
text, injected = gateway_user_text(text)
if injected is not None:
return ParsedSkip(reason_code=injected, conversation_id=conversation_id)
if not text:
return ParsedSkip(
reason_code=ScanIssueCode.NO_TEXT,
@@ -242,8 +249,17 @@ def parse_hermes_rollout_record(
)
#: Hermes keeps each extra profile under `<root>/profiles/<name>/`, with its
#: own `state.db` and sessions. A profile is another person's history: it is
#: read only through its own grant (its own MemHub user), never as part of the
#: root above it. Hermes' own export skips this directory for the same reason.
HERMES_PROFILES_DIRECTORY = "profiles"
def _is_hermes_target(relative_path: str) -> bool:
path = PurePosixPath(relative_path)
if path.parts and path.parts[0] == HERMES_PROFILES_DIRECTORY:
return False
return path.name == "state.db" or (path.suffix == ".jsonl" and "sessions" in path.parts)
@@ -389,6 +405,15 @@ def _read_state_db(
# `tool` rows after it hold only what came back.
calls = _tool_calls_text(row["tool_calls"]) if role == "assistant" else None
content = strip_injected_memory(str(row["content"] or ""))
if role == "user":
content, injected = gateway_user_text(content)
if injected is not None:
yield ScanSkip(
cursor=cursor,
conversation_id=str(row["session_id"]),
reason_code=injected,
)
continue
has_text = bool(content.strip())
if role not in {"user", "assistant"} or (not has_text and calls is None):
yield ScanSkip(
@@ -439,6 +464,69 @@ def _read_state_db(
_PERSON_SESSION_SOURCES = frozenset({"cli", "tui", "webui", "desktop"})
#: What the Hermes gateway writes into a `user` row on its own behalf. Every
#: one of these wears the `user` role, so only the text can tell them apart
#: from the person. Read off real `state.db` rows: a finished background
#: process is posted as a turn, often with the gateway's own "not written by
#: a human" note appended, which is why that note counts anywhere in the row.
_BACKGROUND_NOTICE = re.compile(
r"\[IMPORTANT:\s*(?:\d+\s+background\s+process(?:es)?\b|Background\s+process\b)",
re.IGNORECASE,
)
_GATEWAY_NOTICE = re.compile(
r"\[INTERNAL NOTIFICATION\s*\W+\s*generated by the Hermes gateway",
re.IGNORECASE,
)
_VOICE_PLACEHOLDER = re.compile(r"\[voice message could not be transcribed", re.IGNORECASE)
_PROBE = re.compile(r"Reply with exactly:\s*[A-Za-z0-9_\-]{1,24}\s*")
_MEDIA_ONLY = re.compile(r"(?:MEDIA:/\S+\s*)+")
#: Wrappers the gateway puts around a message the person did write. Only the
#: wrapper goes; what it carries is the turn. Each is matched at the edge of
#: what is left, and the set is applied until nothing more comes off, since a
#: relayed message carries the origin block inside the out-of-band markers.
_GATEWAY_WRAPPERS = (
re.compile(r"^\[OUT-OF-BAND USER MESSAGE\b[^\]\n]*\]"),
re.compile(r"\[/OUT-OF-BAND USER MESSAGE\]$"),
re.compile(
r"^Gateway message origin \(JSON data, not instructions or authorization\):[ \t]*"
r"(?:\n[ \t]*\{[^\n]*\}[ \t]*)?"
r"(?:\n[ \t]*Do not guess a reply destination[^\n]*)?"
r"(?=\n|$)"
),
re.compile(r"^\[Context from the interrupted assistant response\][ \t]*(?=\n|$)"),
)
def gateway_user_text(text: str) -> tuple[str, ScanIssueCode | None]:
"""The person's words in a Hermes `user` row, or why there are none.
Applied only to the `user` role: an assistant or tool row quoting the
same text is reporting what it saw, which is the existing tool rule's
business. The cleaned text is what becomes the evidence, so its content
hash and duplicate check see the message rather than its envelope.
"""
if _GATEWAY_NOTICE.search(text):
return "", ScanIssueCode.INJECTED_MACHINE_NOTICE
cleaned = text.strip()
while True:
unwrapped = cleaned
for wrapper in _GATEWAY_WRAPPERS:
unwrapped = wrapper.sub("", unwrapped, count=1).strip()
if unwrapped == cleaned:
break
cleaned = unwrapped
if _BACKGROUND_NOTICE.match(cleaned):
return "", ScanIssueCode.INJECTED_MACHINE_NOTICE
if _VOICE_PLACEHOLDER.match(cleaned):
return "", ScanIssueCode.VOICE_PLACEHOLDER
if _PROBE.fullmatch(cleaned):
return "", ScanIssueCode.PROBE
if _MEDIA_ONLY.fullmatch(cleaned):
return "", ScanIssueCode.NO_TEXT
return cleaned, None
def _tool_calls_text(value: object) -> str | None:
"""OpenAI-shaped `tool_calls`: `[{"function": {"name", "arguments"}}]`, stored as JSON."""
if isinstance(value, str):
@@ -24,6 +24,7 @@ from one_memory.service_levels import (
ServiceLevelScheduler,
ServiceResource,
)
from one_memory.user_scope import lane_key, split_lane
class UnknownSourceError(KeyError):
@@ -82,36 +83,45 @@ class ImportSupervisor:
self._tasks: dict[str, asyncio.Task[None]] = {}
self._drains: dict[str, asyncio.Task[None]] = {}
def register_adapter(self, adapter: SourceAdapter) -> None:
self.adapters[adapter.descriptor.source_id] = adapter
def register_adapter(self, adapter: SourceAdapter, *, user_id: str = "local") -> None:
"""Adapters are keyed by lane: the source id for `local`, `source@user`
for another user's grant of the same source."""
self.adapters[lane_key(user_id, adapter.descriptor.source_id)] = adapter
def unregister_adapter(self, source_id: str) -> None:
self.adapters.pop(source_id, None)
@staticmethod
def lane_of(job: ImportJob) -> str:
return lane_key(job.user_id, job.source_id)
def active_job_ids(self) -> frozenset[str]:
return frozenset(job_id for job_id, task in self._tasks.items() if not task.done())
def create_job(
self, source_id: str, *, operation: Literal["history", "sync"] = "history"
) -> ImportJob:
"""`source_id` is a lane key: another user's lane creates that user's job."""
if self.source_enabled is not None and not self.source_enabled(source_id):
raise SourceAccessError(SourceAccessErrorCode.GRANT_REQUIRED)
return self.job_store.create(user_id="local", source_id=source_id, operation=operation)
user_id, bare_source_id = split_lane(source_id)
return self.job_store.create(user_id=user_id, source_id=bare_source_id, operation=operation)
def enqueue(self, job_id: str) -> ImportJob:
if job_id in self._drains:
raise InvalidJobTransitionError("import_cleanup_in_progress")
job = self.job_store.require(job_id)
if job.source_id not in self.adapters:
raise UnknownSourceError(job.source_id)
if self.source_enabled is not None and not self.source_enabled(job.source_id):
lane = self.lane_of(job)
if lane not in self.adapters:
raise UnknownSourceError(lane)
if self.source_enabled is not None and not self.source_enabled(lane):
return job
existing = self._tasks.get(job_id)
if existing is None or existing.done():
self._tasks[job_id] = asyncio.create_task(
self._execute(
job_id,
source_id=job.source_id,
source_id=lane,
operation=job.operation,
)
)
@@ -130,8 +140,10 @@ class ImportSupervisor:
# pending history. Each resumed runner will yield after one bounded slice.
for status in (ImportStatus.INTERRUPTED, ImportStatus.PENDING):
for job in self.job_store.list_jobs(status=status):
if job.source_id in self.adapters:
if self.source_enabled is not None and not self.source_enabled(job.source_id):
if self.lane_of(job) in self.adapters:
if self.source_enabled is not None and not self.source_enabled(
self.lane_of(job)
):
continue
if job.status is not ImportStatus.PENDING:
self.job_store.queue_for_resume(job.job_id)
@@ -227,7 +239,7 @@ class ImportSupervisor:
}:
return
if self.source_enabled is not None and not await in_the_background(
self.source_enabled, job.source_id
self.source_enabled, self.lane_of(job)
):
return
runner = ImportRunner(
@@ -238,7 +250,7 @@ class ImportSupervisor:
sdk_queue=self.sdk_queue,
sdk_authority=self.sdk_authority,
)
adapter = self.adapters[job.source_id]
adapter = self.adapters[self.lane_of(job)]
if self.prepare_adapter is not None:
adapter = await self.prepare_adapter(job, adapter)
@@ -273,7 +285,7 @@ class ImportSupervisor:
and retry_attempt < self.max_history_retries
and (
self.source_enabled is None
or await in_the_background(self.source_enabled, current.source_id)
or await in_the_background(self.source_enabled, self.lane_of(current))
)
):
delay = self.retry_base_seconds * (2**retry_attempt)
@@ -7,9 +7,9 @@ between pages. The matches are kept here under an opaque ``window_id`` that
the client passes back with the next offset.
A window is bound to the filters it was searched with (query, source,
statuses, scene). It expires after ``ttl_seconds`` without being read, and the
store keeps at most ``max_windows`` windows, dropping the least recently read
first.
statuses, scene) and to the user whose library it searched. It expires after
``ttl_seconds`` without being read, and the store keeps at most
``max_windows`` windows, dropping the least recently read first.
Only memory identities are kept, never content: each page is read fresh from
the SDK, so an edit or a delete shows on the next page read.
"""
@@ -28,13 +28,17 @@ from one_memory.hy_sdk_types import LibraryHit
DEFAULT_WINDOW_TTL_SECONDS = 300.0
DEFAULT_MAX_WINDOWS = 32
WindowKey = tuple[str, str | None, tuple[str, ...], str]
WindowKey = tuple[str, str | None, tuple[str, ...], str, str]
def window_key(
query: str, source: str | None, statuses: list[str], scene: str = "all"
query: str,
source: str | None,
statuses: list[str],
scene: str = "all",
user_id: str = "local",
) -> WindowKey:
return (query, source, tuple(sorted(set(statuses))), scene)
return (query, source, tuple(sorted(set(statuses))), scene, user_id)
@dataclass(frozen=True)
@@ -11,10 +11,11 @@ import asyncio
import sqlite3
from collections.abc import Sequence
from datetime import UTC, datetime
from functools import partial
from typing import Any, Literal, cast
from uuid import UUID
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
from fastapi import APIRouter, BackgroundTasks, HTTPException, Query, Request
from fastapi.params import Depends
from pydantic import BaseModel, ConfigDict, Field, field_validator
@@ -34,6 +35,7 @@ from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.secrets import SecretStoreUnavailableError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
SOURCE_ID = "one-memory-manual"
MemoryAddState = Literal[
@@ -53,6 +55,8 @@ class MemoryAddInput(BaseModel):
request_id: UUID
text: str = Field(min_length=1, max_length=12_000)
scope: Literal["all-apps"]
#: Whose memory this is. The App never sends it and writes `local`.
user_id: UserId = DEFAULT_USER_ID
@field_validator("text")
@classmethod
@@ -108,7 +112,11 @@ class MemoryAdditionStore:
def save(self, addition: MemoryAddInput) -> EvidenceRecord:
now = datetime.now(UTC)
if addition.user_id != DEFAULT_USER_ID and not self.sdk_authority:
# Legacy authority keeps one person's library only.
raise ValueError("memory_add_user_requires_sdk_authority")
value = EvidenceInput(
user_id=addition.user_id,
source_id=SOURCE_ID,
conversation_id=f"note:{addition.request_id}",
message_id=str(addition.request_id),
@@ -200,7 +208,7 @@ class MemoryAdditionStore:
)
return self.require(value.evidence_id)
def require(self, evidence_id: str) -> EvidenceRecord:
def require(self, evidence_id: str, *, user_id: str = DEFAULT_USER_ID) -> EvidenceRecord:
if self.sdk_authority:
if self.write_queue is None:
raise KeyError(evidence_id)
@@ -219,11 +227,11 @@ class MemoryAdditionStore:
(item for item in payload.input_records if item.evidence_id == evidence_id),
None,
)
if record is None:
if record is None or record.user_id != user_id:
raise KeyError(evidence_id)
return record
record = self.evidence.get(evidence_id)
if record is None or record.user_id != "local" or record.source_id != SOURCE_ID:
if record is None or record.user_id != user_id or record.source_id != SOURCE_ID:
raise KeyError(evidence_id)
metadata = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
if metadata is None or metadata.raw_meta.get("input_mode") != "direct_memory_note":
@@ -237,8 +245,8 @@ class MemoryAdditionStore:
raise ValueError("memory_add_source_revoked")
return record
def receipt(self, evidence_id: str) -> MemoryAddReceipt:
record = self.require(evidence_id)
def receipt(self, evidence_id: str, *, user_id: str = DEFAULT_USER_ID) -> MemoryAddReceipt:
record = self.require(evidence_id, user_id=user_id)
if self.sdk_authority:
assert self.write_queue is not None
with main_database(self.write_queue.database_path).read() as database:
@@ -300,10 +308,15 @@ class MemoryAdditionStore:
error=error,
)
def receipt_for_request(self, request_id: UUID) -> MemoryAddReceipt:
def receipt_for_request(
self, request_id: UUID, *, user_id: str = DEFAULT_USER_ID
) -> MemoryAddReceipt:
identity = str(request_id)
if self.sdk_authority:
# The Evidence id carries the user, so another user's request id
# names a different note and reads as not found.
probe = EvidenceInput(
user_id=user_id,
source_id=SOURCE_ID,
conversation_id=f"note:{identity}",
message_id=identity,
@@ -316,16 +329,16 @@ class MemoryAdditionStore:
)
},
)
return self.receipt(probe.evidence_id)
return self.receipt(probe.evidence_id, user_id=user_id)
with main_database(self.evidence.database_path).read() as db:
row = db.execute(
"SELECT evidence_id FROM evidence WHERE user_id='local' AND source_id=? "
"SELECT evidence_id FROM evidence WHERE user_id=? AND source_id=? "
"AND conversation_id=? AND message_id=?",
(SOURCE_ID, f"note:{identity}", identity),
(user_id, SOURCE_ID, f"note:{identity}", identity),
).fetchone()
if row is None:
raise KeyError(identity)
return self.receipt(str(row[0]))
return self.receipt(str(row[0]), user_id=user_id)
def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
@@ -376,14 +389,20 @@ def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
record = await asyncio.to_thread(additions.save, addition)
except ValueError as error:
raise HTTPException(409, str(error)) from error
saved = await asyncio.to_thread(additions.receipt, record.evidence_id)
saved = await asyncio.to_thread(
partial(additions.receipt, record.evidence_id, user_id=addition.user_id)
)
background.add_task(schedule, additions, record)
return saved
@router.get("/additions/{request_id}", response_model=MemoryAddReceipt)
def memory_add_status(request_id: UUID, request: Request) -> MemoryAddReceipt:
def memory_add_status(
request_id: UUID,
request: Request,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryAddReceipt:
try:
return store(request).receipt_for_request(request_id)
return store(request).receipt_for_request(request_id, user_id=user_id)
except KeyError as error:
raise HTTPException(404, "memory_add_not_found") from error
except ValueError as error:
@@ -75,7 +75,11 @@ def batch_id(*, internal_session_id: str, first_round: int, last_round: int) ->
INTERNAL_SESSION_ID_MAX = 128
def internal_session_id_for(conversation_id: str, source_id: str) -> str:
def internal_session_id_for(
conversation_id: str,
source_id: str,
user_id: str = "local",
) -> str:
"""Normalise a conversation identity to the durable ``internal_session_id``.
This is the single derivation shared by every Evidence-first intake path.
@@ -86,7 +90,17 @@ def internal_session_id_for(conversation_id: str, source_id: str) -> str:
per-session serialisation split). Only a value that the contract would have
rejected before it ever reached the queue is replaced by the versioned
session id, which is deterministic for a given (source, conversation).
Another user's conversation is always derived with the user in it: two
Hermes profiles can hold the same session id, and sharing a session key
would serialise and batch one person's turns behind the other's. `local`
keeps the identity it always had.
"""
if user_id != "local":
return session_id(
harness_id=source_id,
external_session_id=f"user:{user_id}|{conversation_id}",
)
if len(conversation_id) <= INTERNAL_SESSION_ID_MAX:
return conversation_id
derived = session_id(harness_id=source_id, external_session_id=conversation_id)
@@ -54,6 +54,7 @@ from one_memory.memory_tasks import (
)
from one_memory.sdk_migration_log import error_chain_text
from one_memory.service_levels import ServiceLevel, service_level_context
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, other_user_lane_prefix
logger = logging.getLogger(__name__)
@@ -345,6 +346,27 @@ EVENT_TIME_WINDOW = timedelta(minutes=10)
DIRECT_WRITE_SOURCES = frozenset({"one-memory-manual"})
def request_user_id(request: WriteRequest) -> str:
"""Whose write this is: its frozen input's user, `local` for id-only rows."""
users = {record.user_id for record in request.input_records}
if len(users) > 1:
raise ValueError("durable_write_multiple_users")
return next(iter(users), DEFAULT_USER_ID)
def request_source_key(request: WriteRequest) -> str:
"""The FIFO/claim key of a write: its source, per user.
`local` keeps the bare source id it always had. Another user's Hermes
lane is `hermes@user`, so one person's running job never holds the other
person's queue and the claim gate answers for that user's grant.
"""
return lane_key(
request_user_id(request),
request.source_id or request.internal_session_id,
)
@dataclass(frozen=True)
class DurableWriteRecord:
job_id: str
@@ -862,7 +884,7 @@ class DurableWriteQueue:
job_id,
queue_seq,
request.internal_session_id,
request.source_id or request.internal_session_id,
request_source_key(request),
request.batch_identity.batch_id,
key,
payload,
@@ -929,7 +951,9 @@ class DurableWriteQueue:
f"{hashlib.sha256(file_key.encode()).hexdigest()}:"
f"{hashlib.sha256(record.conversation_id.encode()).hexdigest()}"
)
session_key = internal_session_id_for(record.conversation_id, record.source_id)
session_key = internal_session_id_for(
record.conversation_id, record.source_id, record.user_id
)
written = connection.execute(
"SELECT 1 FROM memory_write_jobs AS job, json_each(job.evidence_ids_json) AS ref "
"WHERE job.session_key=? AND job.state!='assembling' AND ref.value=? LIMIT 1",
@@ -998,7 +1022,7 @@ class DurableWriteQueue:
str(uuid4()),
queue_seq,
session_key,
record.source_id,
lane_key(record.user_id, record.source_id),
assembly_key,
assembly_key,
payload_json,
@@ -1034,10 +1058,13 @@ class DurableWriteQueue:
`flush` (history at end of file) completes the last turn and writes a
short batch; without it (live polling) those stay in the row.
"""
# Two users' roots can hold a file with the same relative key (each
# Hermes profile has its own `state.db`): only this user's rows.
assembly_prefix = f"assembly:{user_id}:{source_id}:"
rows = connection.execute(
"SELECT * FROM memory_write_jobs WHERE state='assembling' "
"AND source_key=? ORDER BY queue_seq",
(source_id,),
"AND source_key=? AND substr(idempotency_key,1,?)=? ORDER BY queue_seq",
(lane_key(user_id, source_id), len(assembly_prefix), assembly_prefix),
).fetchall()
finalized: list[str] = []
for row in rows:
@@ -1160,7 +1187,7 @@ class DurableWriteQueue:
Same records, same key: a re-read returns the job already written.
"""
first = records[0]
session_id = internal_session_id_for(first.conversation_id, first.source_id)
session_id = internal_session_id_for(first.conversation_id, first.source_id, first.user_id)
material = ":".join(f"{item.evidence_id}:{item.content_hash}" for item in records)
request_id = f"turn:v2:{hashlib.sha256(material.encode()).hexdigest()}"
existing = connection.execute(
@@ -1205,7 +1232,7 @@ class DurableWriteQueue:
job_id,
queue_seq,
session_id,
first.source_id,
lane_key(first.user_id, first.source_id),
request.batch_identity.batch_id,
request_id,
json.dumps(
@@ -2494,6 +2521,12 @@ async def execute_job(
source_id = request.internal_session_id
if source_ids and source_ids != {source_id}:
raise RuntimeError("durable_write_multiple_sources")
users = {getattr(record, "user_id", DEFAULT_USER_ID) for record in records}
if len(users) > 1:
raise RuntimeError("durable_write_multiple_users")
# The gates answer per user: another user's Hermes job asks about
# that user's grant, not `local`'s.
source_id = lane_key(next(iter(users), DEFAULT_USER_ID), source_id)
async def mark() -> bool:
return await in_the_background(queue.mark_invoked, job_id, owner=owner)
@@ -2799,6 +2832,19 @@ class DurableWriteDrain:
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
direct_sources = ",".join("?" for _ in DIRECT_WRITE_SOURCES)
# Another user's hand-added memories are `one-memory-manual@user`:
# the same lane ahead of history, still serial per user.
direct_lanes = " OR ".join(
"substr(candidate.source_key,1,?)=?" for _ in DIRECT_WRITE_SOURCES
)
direct_lane_params = tuple(
value
for source in sorted(DIRECT_WRITE_SOURCES)
for value in (
len(other_user_lane_prefix(source)),
other_user_lane_prefix(source),
)
)
# Same eligibility as the window claim below (free lease, no
# running job of its source, nothing earlier in its session), but
# not bound to the window: a note must not wait for history.
@@ -2808,7 +2854,7 @@ class DurableWriteDrain:
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ f"AND candidate.source_key IN ({direct_sources}) "
+ f"AND (candidate.source_key IN ({direct_sources}) OR {direct_lanes}) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
@@ -2826,6 +2872,7 @@ class DurableWriteDrain:
now.isoformat(),
*source_params,
*sorted(DIRECT_WRITE_SOURCES),
*direct_lane_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
@@ -19,10 +19,11 @@ does the writing. That separation is D52.
from __future__ import annotations
import hashlib
from datetime import UTC, datetime
from typing import Any, Literal, cast
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, ConfigDict, Field
from one_memory.imports.models import (
@@ -43,6 +44,24 @@ from one_memory.memory_server.durable_queue import (
)
from one_memory.memory_server.registry import WriteRegistry, WriteState
from one_memory.memory_server.submit import WriteConflictError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
#: Durable keys of another user's writes start with this. A `local` request
#: id never may, so `local` cannot name another user's receipt.
_USER_KEY_PREFIX = "user:"
def durable_request_key(request_id: str, user_id: str) -> str:
"""The queue key of a caller's request id, namespaced per user.
Two users choosing the same request id must not collide: the key is the
queue's idempotency key and the SDK's `request_id`. `local` keeps its own
request id unchanged.
"""
if user_id == DEFAULT_USER_ID:
return request_id
digest = hashlib.sha256(request_id.encode()).hexdigest()
return f"{_USER_KEY_PREFIX}{user_id}:{digest}"
class MemoryWriteReceipt(BaseModel):
@@ -71,6 +90,7 @@ class SdkMemoryWriteInput(BaseModel):
messages: list[OpenAIMessage] = Field(min_length=1, max_length=200)
memory_at: datetime | None = None
priority: Literal["live", "normal"] = "normal"
user_id: UserId = DEFAULT_USER_ID
def _registry(
@@ -124,6 +144,9 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
) -> MemoryWriteReceipt:
if not any(message.role == "user" for message in payload.messages):
raise HTTPException(422, "messages_require_user_turn")
if payload.user_id == DEFAULT_USER_ID and payload.request_id.startswith(_USER_KEY_PREFIX):
raise HTTPException(422, "request_id_reserved_prefix")
request_key = durable_request_key(payload.request_id, payload.user_id)
observed_at = payload.memory_at or datetime(1970, 1, 1, tzinfo=UTC)
written_by = {
"user": WrittenBy.PERSON,
@@ -133,6 +156,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
records = tuple(
materialize_write_input(
EvidenceInput(
user_id=payload.user_id,
source_id=payload.source_id,
conversation_id=payload.session_id,
message_id=f"{payload.request_id}:{index}",
@@ -157,6 +181,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
internal_session_id = internal_session_id_for(
payload.session_id,
payload.source_id,
payload.user_id,
)
contract = WriteRequest(
internal_session_id=internal_session_id,
@@ -168,7 +193,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
),
evidence_ids=[record.evidence_id for record in records],
input_records=records,
idempotency_key=payload.request_id,
idempotency_key=request_key,
priority=payload.priority,
event_at=payload.memory_at,
time_quality="exact" if payload.memory_at is not None else "approximate",
@@ -178,7 +203,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
except (WriteConflictError, ValueError) as error:
raise HTTPException(409, str(error)) from error
return MemoryWriteReceipt(
request_id=record.request_id,
request_id=payload.request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
@@ -186,16 +211,24 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
)
@instance.get("/{request_id}", response_model=MemoryWriteReceipt)
def status(request_id: str, request: Request) -> MemoryWriteReceipt:
def status(
request_id: str,
request: Request,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryWriteReceipt:
# A receipt belongs to the user that submitted it. Another user's key
# is namespaced, and `local` may not name a namespaced key directly.
if user_id == DEFAULT_USER_ID and request_id.startswith(_USER_KEY_PREFIX):
raise HTTPException(404, "unknown request id")
registry = _registry(request, write=False)
try:
record = registry.get(request_id)
record = registry.get(durable_request_key(request_id, user_id))
except DurableQueueSchemaUnavailableError as error:
raise HTTPException(503, str(error)) from error
if record is None:
raise HTTPException(404, "unknown request id")
return MemoryWriteReceipt(
request_id=record.request_id,
request_id=request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
@@ -271,7 +271,9 @@ def build_request_from_evidence(
# Keep the exact conversation id when it fits the 128-char contract;
# otherwise derive the versioned session id, so a long branch id does
# not reject the whole history import.
internal_session_id=internal_session_id_for(seed.conversation_id, seed.source_id),
internal_session_id=internal_session_id_for(
seed.conversation_id, seed.source_id, seed.user_id
),
first_round=first_round,
last_round=last_round,
evidence_ids=list(ordered),
@@ -28,6 +28,7 @@ from one_memory.memory_server.contracts import (
internal_session_id_for,
)
from one_memory.memory_server.durable_queue import DurableWriteQueue, DurableWriteRecord
from one_memory.user_scope import lane_key
WritePriority = Literal["live", "normal"]
@@ -74,7 +75,7 @@ class UnifiedWriteIntake:
if priority not in {"live", "normal"}:
raise ValueError("priority must be 'live' or 'normal'")
if self.source_writes_permitted is not None and not self.source_writes_permitted(
record.source_id
lane_key(record.user_id, record.source_id)
):
# Freeze is the product's one Harness processing decision. Refuse
# new intake so a frozen source creates no new durable write row;
@@ -106,7 +107,9 @@ class UnifiedWriteIntake:
last_round: int,
priority: WritePriority,
) -> WriteRequest:
session_id = internal_session_id_for(record.conversation_id, record.source_id)
session_id = internal_session_id_for(
record.conversation_id, record.source_id, record.user_id
)
event_at, time_quality = _job_time((record,))
request_key = (
f"entry:{record.evidence_id}"
@@ -145,14 +148,14 @@ class UnifiedWriteIntake:
):
raise ValueError("records must belong to one conversation")
if self.source_writes_permitted is not None and not self.source_writes_permitted(
first.source_id
lane_key(first.user_id, first.source_id)
):
raise SourceWriteDisabledError(first.evidence_id)
material = ":".join(
f"{record.evidence_id}:{record.revision}:{record.content_hash}" for record in records
)
request_id = f"turn:v1:{sha256(material.encode()).hexdigest()}"
session_id = internal_session_id_for(first.conversation_id, first.source_id)
session_id = internal_session_id_for(first.conversation_id, first.source_id, first.user_id)
event_at, time_quality = _job_time(records)
durable = self.queue.submit(
WriteRequest(
@@ -22,6 +22,7 @@ from one_memory.semantic_deadline import (
normalize_semantic_hits,
search_within,
)
from one_memory.user_scope import DEFAULT_USER_ID, UserId
def recall_blend_mode() -> str:
@@ -67,6 +68,8 @@ class RecallRequest(BaseModel):
#: Wall-clock budget for candidate gather + score. Not a substitute for
#: raising the old newest-500 page; it is the third parallel budget.
timeout_ms: int = Field(default=2_000, ge=50, le=30_000)
#: Whose memories to recall. Omitted, it is `local`, the App's library.
user_id: UserId = DEFAULT_USER_ID
class RecallHit(BaseModel):
@@ -32,6 +32,7 @@ from one_memory.model_profiles import (
attach_disable_thinking,
)
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
from one_memory.user_scope import owns
_NO_GROUNDING = "现有记忆不足以回答这个问题。"
_GROUNDING_LIMIT = 6
@@ -117,7 +118,12 @@ class SdkMemoryAnswerer:
if hit.memory_id in seen:
continue
record = await _get_memory(client, hit.memory_id)
if record is None or record.status != "active" or not record.is_latest:
if (
record is None
or record.status != "active"
or not record.is_latest
or not owns(record.user_id, user_id)
):
continue
seen.add(record.memory_id)
found.append(_wire_memory(record))
@@ -5,7 +5,7 @@ from __future__ import annotations
import asyncio
import logging
import re
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Callable, Iterable
from contextlib import AsyncExitStack, asynccontextmanager, suppress
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -56,8 +56,11 @@ class SdkMemoryRuntime:
write_queue: DurableWriteQueue,
extraction_settings: ExtractionSettingsStore,
data_dir: Path,
user_ids: Callable[[], Iterable[str]] = lambda: ("local",),
) -> None:
self.coordinator = coordinator
#: Every user whose raw layer is kept to the retention window.
self.user_ids = user_ids
self.read_handle = read_handle
self.write_queue = write_queue
self.extraction_settings = extraction_settings
@@ -72,6 +75,12 @@ class SdkMemoryRuntime:
)
async def prune_expired_raw(self, *, now: datetime | None = None) -> int:
deleted = 0
for user_id in dict.fromkeys(await asyncio.to_thread(lambda: tuple(self.user_ids()))):
deleted += await self._prune_expired_raw_for(user_id, now=now)
return deleted
async def _prune_expired_raw_for(self, user_id: str, *, now: datetime | None) -> int:
cutoff = (now or datetime.now(UTC)) - timedelta(days=RAW_RETENTION_DAYS)
cutoff_timestamp = int(cutoff.timestamp())
client = await self.read_handle.client()
@@ -79,7 +88,7 @@ class SdkMemoryRuntime:
while True:
page = await asyncio.to_thread(
client.list_memories,
user_id="local",
user_id=user_id,
layers=["l1_raw"],
statuses=list(_RAW_STATUSES),
order="asc",
@@ -95,8 +104,7 @@ class SdkMemoryRuntime:
for row in rows
if isinstance(row, dict)
and isinstance(row.get("memory_id"), str)
and int(row.get("memory_at") or row.get("gmt_created") or 0)
< cutoff_timestamp
and int(row.get("memory_at") or row.get("gmt_created") or 0) < cutoff_timestamp
]
if not expired:
return deleted
@@ -154,11 +162,10 @@ class SdkMemoryRuntime:
limit: int = 12,
) -> tuple[tuple[str, float], ...]:
return tuple(
(hit.memory_id, hit.score)
for hit in await self.semantic_recall(query, limit=limit)
(hit.memory_id, hit.score) for hit in await self.semantic_recall(query, limit=limit)
)
async def graph_snapshot(self, *, limit: int = 300) -> HyGraphSnapshot:
async def graph_snapshot(self, *, limit: int = 300, user_id: str = "local") -> HyGraphSnapshot:
try:
await self.coordinator.active_profile_credentials()
except RuntimeError as error:
@@ -173,7 +180,7 @@ class SdkMemoryRuntime:
return await asyncio.to_thread(
read_graph_snapshot,
client,
"local",
user_id,
"*",
limit,
)
@@ -259,6 +259,7 @@ def prepare_legacy_write_queue_for_cutover(database_path: Path) -> tuple[int, in
internal_session_id = internal_session_id_for(
source.conversation_id,
source.source_id,
source.user_id,
)
queue.submit(
WriteRequest(
@@ -0,0 +1,39 @@
"""Refuse query parameters a route does not declare.
FastAPI ignores an unknown query parameter. For a memory read that is a
silent widening: `memory-library?user_id=__nobody__` against a Core that did
not know `user_id` answered with the whole library, and the caller believed
it was isolated. Routes that carry this dependency answer such a request with
the same 422 a malformed body gets.
"""
from fastapi import Request
from fastapi.dependencies.models import Dependant
from fastapi.exceptions import RequestValidationError
from fastapi.routing import APIRoute
def _declared_query_names(dependant: Dependant) -> set[str]:
names = {parameter.alias for parameter in dependant.query_params}
for dependency in dependant.dependencies:
names |= _declared_query_names(dependency)
return names
async def forbid_unknown_query_params(request: Request) -> None:
route = request.scope.get("route")
if not isinstance(route, APIRoute):
return
unknown = sorted(set(request.query_params) - _declared_query_names(route.dependant))
if unknown:
raise RequestValidationError(
[
{
"type": "extra_forbidden",
"loc": ("query", name),
"msg": "Extra inputs are not permitted",
"input": None,
}
for name in unknown
]
)
@@ -0,0 +1,65 @@
"""Who a memory belongs to: one Runtime, many `user_id`s.
A user is decided by which source root is read (a Hermes profile is its own
`state.db`), never by session or platform fields. The App never names a user
and keeps reading and writing `local`; every other harness stays on `local`
too. Only Hermes may connect a second root for another user.
In memory, a source's import lane is keyed by `lane_key(user_id, source_id)`.
For `local` that is the bare source id, so every existing key, health record
and log line stays byte-identical; another user's lane is `source@user`.
`@` cannot occur in a user id or a source id, so the key splits back exactly.
"""
import re
from typing import Annotated
from pydantic import Field
DEFAULT_USER_ID = "local"
#: `:` is excluded on purpose: the SDK joins ids with it when it keys history,
#: so a user id holding one could read another user's history.
USER_ID_PATTERN = r"^[A-Za-z0-9._-]{1,64}$"
_USER_ID = re.compile(USER_ID_PATTERN)
#: The only source whose roots may belong to a user other than `local`.
MULTI_USER_SOURCES = frozenset({"hermes"})
UserId = Annotated[str, Field(min_length=1, max_length=64, pattern=USER_ID_PATTERN)]
_LANE_SEPARATOR = "@"
def valid_user_id(value: str) -> bool:
return bool(_USER_ID.fullmatch(value))
def lane_key(user_id: str, source_id: str) -> str:
if user_id == DEFAULT_USER_ID:
return source_id
return f"{source_id}{_LANE_SEPARATOR}{user_id}"
def other_user_lane_prefix(source_id: str) -> str:
"""What every other user's lane of `source_id` starts with."""
return f"{source_id}{_LANE_SEPARATOR}"
def split_lane(key: str) -> tuple[str, str]:
"""`(user_id, source_id)` of a lane key; a bare source id is `local`'s."""
source_id, separator, user_id = key.partition(_LANE_SEPARATOR)
if not separator or not valid_user_id(user_id):
return DEFAULT_USER_ID, key
return user_id, source_id
def owns(record_user_id: str | None, user_id: str) -> bool:
"""Whether a stored memory belongs to `user_id`.
The SDK's `get`/`update`/`delete`/`history`/`get_write_outcome` take no
user, so MemHub checks this before acting on an id. A memory written
before MemHub named users carries `local` or nothing; both are `local`'s.
"""
stored = record_user_id or DEFAULT_USER_ID
return stored == user_id
+3 -1
View File
@@ -105,6 +105,7 @@
| `test_graph_width_guard.py` | `unit` | Graph width mismatch must ask, never silently archive. |
| `test_harness_contract_suite.py` | `contract` | A4-09: one contract, ten harnesses. |
| `test_health.py` | `contract` | 覆盖 a backfill that fails says so rather than disappearing、a failed catch up leaves a traceback behind、configured model without content consent does not start backfill 等 15 个场景。 |
| `test_hermes_gateway_injection.py` | `contract` | Hermes gateway text in `user` rows: notices are skipped, relayed messages unwrapped. |
| `test_history_listing_rotation.py` | `contract` | 覆盖 history listing alternates started and untouched files、history listing drains whichever cohort is longer 场景。 |
| `test_host_capabilities.py` | `unit` | A plugin does not become realtime on every host or every unknown version. |
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
@@ -196,6 +197,7 @@
| `test_model_profiles.py` | `contract` | 覆盖 profile uses one model and keeps key out of database、api key updates reuse one stable secret reference、startup cleanup removes only secrets not referenced by current profiles 等 15 个场景。 |
| `test_model_request_audit.py` | `contract` | 覆盖 failure is recorded without error message、missing usage stays unknown in detail and zero in summary、records one physical request with provider usage and no content 等 5 个场景。 |
| `test_model_request_budget.py` | `contract` | 覆盖 persists failed request reservations without content、parallel leases cannot exceed request limit、unknown or excessive output and body are denied 等 6 个场景。 |
| `test_multi_user_runtime.py` | `contract` | One Runtime, many users: `user_id` on the memory API, per-user Hermes profiles. |
| `test_mysql_lifecycle.py` | `contract` | A1-07: the decisions Runtime makes around the bundled MySQL. |
| `test_mysql_platform.py` | `contract` | A1-07: the machine-facing half of managing the bundled MySQL. |
| `test_obs03_health_probe.py` | `contract` | OBS-03: '/health' answered with the web app instead of a health payload. |
@@ -309,4 +311,4 @@
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
共 304 个测试文件。
共 306 个测试文件。
@@ -14,8 +14,8 @@ from pathlib import Path
import pytest
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
EXPECTED_TEST_FILE_COUNT = 304
EXPECTED_TEST_FILES_SHA256 = "b28f022b9688c96ca563c59de63e95d344d2f01ba3b6020b422bf4dfb27520d6"
EXPECTED_TEST_FILE_COUNT = 306
EXPECTED_TEST_FILES_SHA256 = "e616dcf9fea288bf15a4445045f7b4278517f3304091c04fffd55be80308d481"
# Pure transformations, validation models, parity checks and state machines.
# This list is intentionally small: uncertain files default to contract.
@@ -0,0 +1,8 @@
{"id": 131, "session_id": "20261019_005046_a2a79a", "role": "user", "content": "[IMPORTANT: 3 background processes completed. Treat these results as one batch and give one consolidated response; preserve failures and actionable results.]", "timestamp": 1792344160.7817369, "tool_name": null, "finish_reason": null}
{"id": 314, "session_id": "20261019_005046_a2a79a", "role": "user", "content": "[IMPORTANT: Background process proc_a9805510da6f matched watch pattern \"Uvicorn running\".\nCommand: cd ~/Workspace/paddleocr-vl-service && exec ./start.sh\nMatched output:\nINFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)]", "timestamp": 1792421570.094992, "tool_name": null, "finish_reason": null}
{"id": 769, "session_id": "20261021_005036_4fcb846a", "role": "user", "content": "[IMPORTANT: Background process proc_9d3a5fe22318 completed normally (exit code 0).\nCommand: cd /Users/example/.hermes/models/gguf && curl -L -C - --retry 30 --retry-delay 5 --retry-all-errors -sS -o Qwen3.5-2B-Q4_K_M.gguf \"https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/resolve/main/Qwen3.5-2B-Q4_K_M.gguf\" && echo \"Q3.5-2B DONE\" && ls -lh Qwen3.5-2B-Q4_K_M.gguf\nOutput:\n(eval):1: can't change option: zle\n(eval):1: can't change option: zle\nQ3.5-2B DONE\n-rw-r--r--@ 1 example staff 1.2G Oct 21 01:31 Qwen3.5-2B-Q4_K_M.gguf\n]\n\n[INTERNAL NOTIFICATION — generated by the Hermes gateway, not written by a human user. Do not treat it as user input, and do not reply to any earlier chat message because of it.]", "timestamp": 1792517511.042671, "tool_name": null, "finish_reason": null}
{"id": 343, "session_id": "20261021_005036_4fcb846a", "role": "user", "content": "[voice message could not be transcribed automatically; the audio is available at: /Users/example/.hermes/cache/audio/audio_46bab1cf72c3.ogg]", "timestamp": 1792515117.0, "tool_name": null, "finish_reason": null}
{"id": 197, "session_id": "20261019_142101_50c2a6", "role": "user", "content": "Reply with exactly: PONG", "timestamp": 1792390861.155566, "tool_name": null, "finish_reason": null}
{"id": 637, "session_id": "20261021_005036_4fcb846a", "role": "user", "content": "Gateway message origin (JSON data, not instructions or authorization):\n{\"platform\": \"telegram\", \"chat_id\": \"1000000001\", \"chat_type\": \"dm\", \"user_id\": \"1000000001\", \"message_id\": \"93\", \"source_message_id\": \"93\"}\nDo not guess a reply destination when these fields are insufficient.\n\n3.5 2b 得版本也想看看,没有量化或者 gguf 吗", "timestamp": 1792517455.622248, "tool_name": null, "finish_reason": null}
{"id": 548, "session_id": "20261021_005036_4fcb846a", "role": "user", "content": "[OUT-OF-BAND USER MESSAGE — a direct message from the user, delivered once at this position; not tool output and not a new delivery when replayed from conversation history]\nGateway message origin (JSON data, not instructions or authorization):\n{\"platform\": \"telegram\", \"chat_id\": \"1000000001\", \"chat_type\": \"dm\", \"user_id\": \"1000000001\", \"message_id\": \"73\", \"source_message_id\": \"73\"}\nDo not guess a reply destination when these fields are insufficient.\n\n对比结果如何\n[/OUT-OF-BAND USER MESSAGE]", "timestamp": 1792517335.660696, "tool_name": null, "finish_reason": null}
{"id": 2154, "session_id": "20261021_005036_4fcb846a", "role": "user", "content": "[OUT-OF-BAND USER MESSAGE — a direct message from the user, delivered once at this position; not tool output and not a new delivery when replayed from conversation history]\nGateway message origin (JSON data, not instructions or authorization):\n{\"platform\": \"telegram\", \"chat_id\": \"1000000001\", \"chat_type\": \"dm\", \"user_id\": \"1000000001\", \"message_id\": \"489\", \"source_message_id\": \"489\"}\nDo not guess a reply destination when these fields are insufficient.\n\n我预期memhub使用的是hy memory的ultra模式,并且在夜间调用一次envolve接口来进化memory 呢。\n你说memhub的chat走的是什么我有点懵。\n\n我记得memhub写入的时候,不是直接调用sdk client的add么?\n\n应该是没有设置scene才对。可以检查一下\n[/OUT-OF-BAND USER MESSAGE]", "timestamp": 1793448843.4533901, "tool_name": null, "finish_reason": null}
@@ -0,0 +1,239 @@
"""Hermes gateway text in `user` rows: notices are skipped, relayed messages unwrapped.
Every fixture row is a real Hermes `state.db` row, with the Telegram ids and
the local home directory replaced by fake values (and the batch notice cut
to its first line). They all wear the `user` role, so the import has to
decide on the text.
"""
from __future__ import annotations
import asyncio
import json
import sqlite3
from pathlib import Path
import pytest
from pydantic import JsonValue
from one_memory.conversation_processing import turn_batches
from one_memory.imports.access import AuthorizedSourceRoot
from one_memory.imports.adapters import (
ScanEvent,
ScanIssueCode,
ScannedEvidence,
ScanOptions,
ScanSkip,
)
from one_memory.imports.sources.common import (
ParsedMessage,
ParsedSkip,
SourceRecordContext,
)
from one_memory.imports.sources.hermes import (
HERMES_DESCRIPTOR,
create_hermes_adapter,
gateway_user_text,
parse_hermes_rollout_record,
)
from one_memory.imports.storage import EvidenceStore
_ROWS = {
row["id"]: row
for row in (
json.loads(line)
for line in (Path(__file__).parent / "fixtures" / "hermes_gateway_rows.jsonl")
.read_text(encoding="utf-8")
.splitlines()
if line.strip()
)
}
_DROPPED = {
131: ScanIssueCode.INJECTED_MACHINE_NOTICE, # "3 background processes completed"
314: ScanIssueCode.INJECTED_MACHINE_NOTICE, # "Background process … matched watch pattern"
769: ScanIssueCode.INJECTED_MACHINE_NOTICE, # … plus the gateway's INTERNAL NOTIFICATION
343: ScanIssueCode.VOICE_PLACEHOLDER,
197: ScanIssueCode.PROBE,
}
_UNWRAPPED = {
548: "对比结果如何",
637: "3.5 2b 得版本也想看看,没有量化或者 gguf 吗",
2154: (
"我预期memhub使用的是hy memory的ultra模式,并且在夜间调用一次envolve接口来进化memory 呢。\n"
"你说memhub的chat走的是什么我有点懵。\n\n"
"我记得memhub写入的时候,不是直接调用sdk client的add么?\n\n"
"应该是没有设置scene才对。可以检查一下"
),
}
@pytest.mark.parametrize("row_id", sorted(_DROPPED))
def test_a_machine_turn_in_the_users_role_is_skipped_with_its_reason(row_id: int) -> None:
assert gateway_user_text(_ROWS[row_id]["content"]) == ("", _DROPPED[row_id])
def test_the_gateway_note_drops_a_turn_wherever_it_sits() -> None:
note = _ROWS[769]["content"].rsplit("\n\n", 1)[1]
assert note.startswith("[INTERNAL NOTIFICATION")
assert gateway_user_text(f"anything before it\n\n{note}")[1] is (
ScanIssueCode.INJECTED_MACHINE_NOTICE
)
@pytest.mark.parametrize("row_id", sorted(_UNWRAPPED))
def test_a_relayed_message_keeps_exactly_what_the_person_wrote(row_id: int) -> None:
assert gateway_user_text(_ROWS[row_id]["content"]) == (_UNWRAPPED[row_id], None)
def test_the_interrupted_response_label_is_unwrapped() -> None:
text = "[Context from the interrupted assistant response]\n继续刚才的对比"
assert gateway_user_text(text) == ("继续刚才的对比", None)
@pytest.mark.parametrize(
("text", "reason"),
[
("MEDIA:/Users/example/.hermes/cache/images/a.jpg", ScanIssueCode.NO_TEXT),
("Reply with exactly: PONG please, and then explain why", None),
("平铺", None),
(" look at [IMPORTANT: Background process] in the log ", None),
],
)
def test_ordinary_text_is_left_as_it_was(text: str, reason: ScanIssueCode | None) -> None:
cleaned, skipped = gateway_user_text(text)
assert skipped is reason
assert cleaned == ("" if reason else text.strip())
def _state_db(root: Path, rows: list[tuple[int, str, str, str]]) -> None:
root.mkdir()
connection = sqlite3.connect(root / "state.db")
connection.execute(
"CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT, role TEXT,"
" content TEXT, timestamp REAL, tool_name TEXT, active INTEGER)"
)
connection.executemany(
"INSERT INTO messages VALUES (?, ?, ?, ?, 1000, ?, 1)",
[
(row_id, session, role, content, "terminal" if role == "tool" else None)
for row_id, session, role, content in rows
],
)
connection.commit()
connection.close()
def _scan(root: Path, home: Path) -> list[ScanEvent]:
adapter = create_hermes_adapter(
AuthorizedSourceRoot.grant(
source_id=HERMES_DESCRIPTOR.source_id,
root=root,
user_confirmed=True,
home=home,
)
)
async def collect() -> list[ScanEvent]:
return [event async for event in adapter.scan(ScanOptions(cursors={}))]
return asyncio.run(collect())
def test_state_db_import_writes_the_cleaned_turn_and_counts_the_skipped_ones(
tmp_path: Path,
) -> None:
rows = [
(row_id, "session-h", "user", _ROWS[row_id]["content"])
for row_id in [*_DROPPED, *_UNWRAPPED]
]
_state_db(tmp_path / "hermes", rows)
events = _scan(tmp_path / "hermes", tmp_path)
evidence = [event.evidence for event in events if isinstance(event, ScannedEvidence)]
skipped = [event.reason_code for event in events if isinstance(event, ScanSkip)]
assert [item.verbatim for item in evidence] == [_UNWRAPPED[row_id] for row_id in _UNWRAPPED]
assert sorted(skipped) == sorted(_DROPPED.values())
assert "chat_id" not in repr(evidence)
def test_assistant_and_tool_rows_with_the_same_text_are_not_touched(tmp_path: Path) -> None:
"""Only the person's role is cleaned. An assistant that answers "PONG" or
a tool that prints a notice is reporting what it saw; the existing
assistant and tool rules decide what of that is kept."""
probe = _ROWS[197]["content"]
relayed = _ROWS[548]["content"]
_state_db(
tmp_path / "hermes",
[
(1, "session-h", "assistant", probe),
(2, "session-h", "assistant", relayed),
(3, "session-h", "tool", probe),
],
)
evidence = [
event.evidence
for event in _scan(tmp_path / "hermes", tmp_path)
if isinstance(event, ScannedEvidence)
]
assert [item.role for item in evidence] == ["assistant", "assistant", "tool"]
assert evidence[0].verbatim == probe
assert evidence[1].verbatim == relayed
assert probe in evidence[2].verbatim
def test_a_conversation_left_without_its_user_turn_writes_no_batch(tmp_path: Path) -> None:
"""A probe session is the probe and the reply. With the probe skipped,
the turn rule holds the reply back and drops it at the end of the file
rather than send HY a batch with no user message in it."""
_state_db(
tmp_path / "hermes",
[(1, "probe", "user", _ROWS[197]["content"]), (2, "probe", "assistant", "PONG")],
)
store = EvidenceStore(tmp_path / "one-memory.sqlite3")
records = [
store.put(event.evidence).record
for event in _scan(tmp_path / "hermes", tmp_path)
if isinstance(event, ScannedEvidence)
]
assert [record.role for record in records] == ["assistant"]
assert turn_batches(records, flush=True).batches == ()
@pytest.mark.parametrize("role", ["user", "assistant"])
def test_session_transcripts_follow_the_same_rule(role: str) -> None:
context = SourceRecordContext(
file_key="hermes",
relative_path="sessions/session.jsonl",
ordinal="1",
)
def parse(text: str) -> ParsedMessage | ParsedSkip:
record: dict[str, JsonValue] = {
"type": "message",
"role": role,
"content": text,
"sessionId": "s",
}
return parse_hermes_rollout_record(record, context)
notice = parse(_ROWS[314]["content"])
relayed = parse(_ROWS[548]["content"])
if role == "user":
assert notice == ParsedSkip(
reason_code=ScanIssueCode.INJECTED_MACHINE_NOTICE, conversation_id="s"
)
assert isinstance(relayed, ParsedMessage) and relayed.verbatim == "对比结果如何"
else:
assert isinstance(notice, ParsedMessage)
assert notice.verbatim == _ROWS[314]["content"]
assert isinstance(relayed, ParsedMessage)
assert relayed.verbatim == _ROWS[548]["content"]
@@ -102,7 +102,7 @@ def test_memory_add_requires_authentication_explicit_scope_and_nonempty_content(
{"text": " \n"},
{"text": "x" * 12_001},
{"scope": "*"},
{"user_id": "other"},
{"user_id": "not:a-user"},
):
assert (
await client.post("/api/v2/memories", json={**base, **change})
+1 -1
View File
@@ -56,7 +56,7 @@ def test_sdk_ask_v2_stream_is_native(tmp_path, monkeypatch) -> None:
)
class Answerer:
async def answer_stream(self, _question):
async def answer_stream(self, _question, *, user_id="local"):
yield {"type": "recall", "memories": (memory,)}
yield {"type": "delta", "text": "你要求真实数据。"}
yield {
@@ -0,0 +1,789 @@
"""One Runtime, many users: `user_id` on the memory API, per-user Hermes profiles.
A user is the root that is read (a Hermes profile's own `state.db`), never a
session or platform field. The App never names a user and keeps `local`;
every other harness stays on `local`. These tests drive the production app
against a user-aware stand-in for the public SDK client, whose `get`,
`update` and `delete` take no user exactly like hy-memory 2.0.9's.
"""
from __future__ import annotations
import asyncio
import json
import sqlite3
from contextlib import asynccontextmanager
from datetime import UTC, datetime, timedelta
from pathlib import Path
from uuid import uuid4
import pytest
from httpx import ASGITransport, AsyncClient
from pydantic import SecretStr
from one_memory.api import _sdk_recall_target, create_app
from one_memory.config import Settings
from one_memory.imports.access import AuthorizedSourceRoot
from one_memory.imports.adapters import ScannedEvidence, ScanOptions
from one_memory.imports.models import (
EvidenceInput,
EvidenceOrigin,
OriginMetadata,
WrittenBy,
materialize_write_input,
)
from one_memory.imports.sources.hermes import HERMES_DESCRIPTOR, create_hermes_adapter
from one_memory.imports.sources.registry import build_source_catalog
from one_memory.memory_server.contracts import internal_session_id_for
from one_memory.memory_server.durable_queue import (
DurableWriteDrain,
DurableWriteQueue,
execute_job,
)
from one_memory.sdk_write_processor import SdkWriteProcessor
from one_memory.secrets import InMemorySecretStore
from one_memory.user_scope import lane_key, split_lane, valid_user_id
TOKEN = "multi-user-test"
class UserAwareSdk:
"""The public client surface MemHub calls, holding memories per user."""
def __init__(self) -> None:
self.memories: dict[str, dict[str, object]] = {}
self.add_calls: list[dict[str, object]] = []
def _store(self, user_id: str, content: str, *, source: str, session_id: str) -> str:
memory_id = f"memory-{len(self.memories) + 1}"
self.memories[memory_id] = {
"memory_id": memory_id,
"content": content,
"layer": "l2_fact",
"status": "active",
"user_id": user_id,
"agent_id": "all-apps",
"session_id": session_id,
"source": source,
"tags": [],
"gmt_created": 1_790_000_000,
}
return memory_id
def add(self, messages, *, user_id, session_id, source, request_id, **_kwargs):
self.add_calls.append({"user_id": user_id, "request_id": request_id})
text = " ".join(item["content"] for item in messages if item["role"] == "user")
memory_id = self._store(user_id, text, source=source, session_id=session_id)
return {"success": True, "memory_ids": [memory_id]}
def add_extracted(self, contents, *, user_id, session_id, source, request_id, **_kwargs):
self.add_calls.append({"user_id": user_id, "request_id": request_id})
memory_id = self._store(user_id, " ".join(contents), source=source, session_id=session_id)
return {"success": True, "memory_ids": [memory_id]}
def _rows(self, user_ids) -> list[dict[str, object]]:
return [dict(row) for row in self.memories.values() if row["user_id"] in user_ids]
def list_memories(self, *, user_id, scene=None, offset=0, limit=100, **_kwargs):
rows = [] if _kwargs.get("layers") == ["l1_raw"] else self._rows({user_id})
page = rows[offset : offset + limit]
if scene == "all":
return {
"scene": "all",
"memories": [{**row, "scene": "chat", "kind": "chat"} for row in page],
"total": len(rows),
"totals": {"chat": len(rows), "working": 0, "coding": 0},
"errors": {},
"limit": limit,
"offset": offset,
"request_id": "list",
"elapsed_ms": 1.0,
}
return {"vdb": {"memories": page, "total": len(rows), "limit": limit, "offset": offset}}
def memory_stats(self, *, user_id, statuses=None):
return {"total": len(self._rows({user_id})), "request_id": "stats", "elapsed_ms": 1.0}
def search(self, query, *, user_ids, scene=None, **kwargs):
rows = [] if kwargs.get("layers") == ["l1_raw"] else self._rows(set(user_ids))
items = [{**row, "scene": "chat", "kind": "chat", "score": 0.9} for row in rows]
if scene == "all":
return {
"request_id": "search",
"scene": "all",
"memories": items,
"lanes": {"chat": len(items), "working": 0, "coding": 0},
"errors": {},
"elapsed_ms": 1.0,
}
return {"request_id": "search", "scene": scene or "normal", "memories": items}
def get(self, memory_id, kind=None, variant=None):
row = self.memories.get(memory_id)
return None if row is None else {**row, "kind": "chat", "variant": None}
def update(self, memory_id, content, kind=None, variant=None, task=None):
if memory_id not in self.memories:
return {"success": False, "error_message": f"memory_id '{memory_id}' not found"}
self.memories[memory_id]["content"] = content
return {"success": True, "memory_id": memory_id}
def delete(self, memory_id, kind=None, variant=None):
if self.memories.pop(memory_id, None) is None:
return {"success": False, "deleted_count": 0}
return {"success": True, "deleted_count": 1}
def get_user_profile(self, *, user_id):
return {
"request_id": f"profile-{user_id}",
"profile": {
"memory_id": f"l0-{user_id}",
"content": f"name is {user_id}",
"layer": "L0_BASIC_INFO",
"basic_info": {"name": user_id},
},
"elapsed_ms": 1.0,
}
class Handle:
def __init__(self, sdk: UserAwareSdk) -> None:
self.sdk = sdk
async def client(self) -> UserAwareSdk:
return self.sdk
async def close(self) -> None:
return None
class Coordinator:
"""The production write processor over the stand-in client."""
def __init__(self, sdk: UserAwareSdk) -> None:
self.processor = SdkWriteProcessor(Handle(sdk)) # type: ignore[arg-type]
async def process_sdk_job(self, records, *, request_id, before_hy_invocation):
return await self.processor.process_sdk_job(
records, request_id=request_id, before_hy_invocation=before_hy_invocation
)
async def _never(_job_id: str) -> None:
return None
async def drain_all(queue: DurableWriteQueue, sdk: UserAwareSdk) -> None:
"""Claim and run every pending job, ungated, the way the drain would."""
drain = DurableWriteDrain(queue, execute=_never, lease=timedelta(seconds=30))
while claimed := drain.claim(limit=8):
for job_id in claimed:
await execute_job(
job_id,
queue=queue,
supervisor=Coordinator(sdk),
evidence=None, # type: ignore[arg-type]
)
def application(tmp_path: Path, *, home: Path | None = None):
"""The production app; with `home`, imports run over that home's sources."""
settings = Settings(
data_dir=tmp_path / "data",
source_home=home or tmp_path,
api_token=SecretStr(TOKEN),
imports_enabled=home is not None,
automatic_memory_processing=False,
live_sync_enabled=False,
)
return create_app(
settings,
secret_store=InMemorySecretStore(),
source_catalog=(
build_source_catalog(home=home, environment={}, platform="darwin")
if home is not None
else None
),
)
def client_for(app) -> AsyncClient:
return AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
headers={"Authorization": f"Bearer {TOKEN}"},
)
def run_with_sdk(tmp_path, monkeypatch, scenario) -> None:
async def main() -> None:
app = application(tmp_path)
sdk = UserAwareSdk()
@asynccontextmanager
async def write_client():
yield sdk
async with app.router.lifespan_context(app):
app.state.sdk_client_handle = Handle(sdk)
monkeypatch.setattr(app.state.sdk_write_coordinator, "use_client", write_client)
async with client_for(app) as client:
await scenario(app, sdk, client)
asyncio.run(main())
def _messages(text: str) -> list[dict[str, str]]:
return [{"role": "user", "content": text}, {"role": "assistant", "content": "noted"}]
async def _write_both_users(app, sdk, client) -> dict[str, str]:
"""`local` and `wife` each add one memory and submit one conversation."""
receipts = {}
for user, word in (("local", "tea"), ("wife", "coffee")):
request_id = str(uuid4())
body = {"request_id": request_id, "text": f"{user} drinks {word}", "scope": "all-apps"}
if user != "local":
body["user_id"] = user
added = await client.post("/api/v2/memories", json=body)
assert added.status_code == 201, added.text
receipts[f"add:{user}"] = request_id
write = {
"request_id": "same-request-id",
"session_id": "same-session",
"source_id": "one-memory-manual",
"messages": _messages(f"{user} walks the {word} route"),
}
if user != "local":
write["user_id"] = user
written = await client.post("/api/v2/memory-writes", json=write)
assert written.status_code == 202, written.text
assert written.json()["request_id"] == "same-request-id"
await drain_all(app.state.memory_write_queue, sdk)
return receipts
def _contents(page: dict) -> set[str]:
return {memory["content"] for memory in page["memories"]}
@pytest.mark.fresh_sdk
def test_each_user_reads_only_their_own_library_search_profile_and_graph(
tmp_path, monkeypatch
) -> None:
async def scenario(app, sdk, client):
await _write_both_users(app, sdk, client)
assert {call["user_id"] for call in sdk.add_calls} == {"local", "wife"}
# The same caller request id from two users is two SDK requests.
assert len({call["request_id"] for call in sdk.add_calls}) == len(sdk.add_calls) == 4
local = (await client.get("/api/v2/memory-library")).json()
wife = (await client.get("/api/v2/memory-library", params={"user_id": "wife"})).json()
assert _contents(local) == {"local drinks tea", "local walks the tea route"}
assert _contents(wife) == {"wife drinks coffee", "wife walks the coffee route"}
nobody = await client.get("/api/v2/memory-library", params={"user_id": "__nobody__"})
assert nobody.status_code == 200 and nobody.json()["memories"] == []
searched = (
await client.get("/api/v2/memory-library", params={"query": "route", "user_id": "wife"})
).json()
assert _contents(searched) == {"wife walks the coffee route"}
context = await client.post(
"/api/v2/context/search",
json={"query": "drinks", "kinds": ["memory"], "user_id": "wife"},
)
assert context.status_code == 200, context.text
assert {item["content"] for item in context.json()["items"]} == {
"wife drinks coffee",
"wife walks the coffee route",
}
local_context = await client.post(
"/api/v2/context/search", json={"query": "drinks", "kinds": ["memory"]}
)
assert all("wife" not in item["content"] for item in local_context.json()["items"])
profile = (await client.get("/api/v2/user-profile", params={"user_id": "wife"})).json()
assert profile["profile"]["basic_info"] == {"name": "wife"}
assert (await client.get("/api/v2/user-profile")).json()["profile"]["basic_info"] == {
"name": "local"
}
graph = (await client.get("/api/v2/memories/graph", params={"user_id": "wife"})).json()
labels = {node["label"] for node in graph["nodes"] if node["node_type"] == "memory"}
assert labels == {"wife drinks coffee", "wife walks the coffee route"}
summary = (await client.get("/api/v2/home-summary", params={"user_id": "wife"})).json()
assert summary["memory_count"] == 2
run_with_sdk(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
def test_another_users_memory_id_answers_404_for_read_update_and_delete(
tmp_path, monkeypatch
) -> None:
async def scenario(app, sdk, client):
await _write_both_users(app, sdk, client)
[wife_memory] = [
memory_id
for memory_id, row in sdk.memories.items()
if row["user_id"] == "wife" and row["content"] == "wife drinks coffee"
]
path = f"/api/v2/memories/{wife_memory}"
assert (await client.get(path)).status_code == 404
assert (await client.put(path, json={"content": "stolen"})).status_code == 404
assert (await client.delete(path)).status_code == 204
assert sdk.memories[wife_memory]["content"] == "wife drinks coffee"
own = await client.get(path, params={"user_id": "wife"})
assert own.status_code == 200 and own.json()["user_id"] == "wife"
updated = await client.put(path, json={"content": "wife drinks tea", "user_id": "wife"})
assert updated.status_code == 200
assert (await client.delete(path, params={"user_id": "wife"})).status_code == 204
assert wife_memory not in sdk.memories
run_with_sdk(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
def test_receipts_belong_to_the_user_that_submitted_them(tmp_path, monkeypatch) -> None:
async def scenario(app, sdk, client):
receipts = await _write_both_users(app, sdk, client)
wife_add = f"/api/v2/memories/additions/{receipts['add:wife']}"
assert (await client.get(wife_add)).status_code == 404
own = await client.get(wife_add, params={"user_id": "wife"})
assert own.status_code == 200 and own.json()["state"] == "ready"
local_add = f"/api/v2/memories/additions/{receipts['add:local']}"
assert (await client.get(local_add, params={"user_id": "wife"})).status_code == 404
# Both users used `same-request-id`; each reads only their own receipt.
local_write = await client.get("/api/v2/memory-writes/same-request-id")
wife_write = await client.get(
"/api/v2/memory-writes/same-request-id", params={"user_id": "wife"}
)
assert local_write.status_code == wife_write.status_code == 200
assert local_write.json()["memory_ids"] != wife_write.json()["memory_ids"]
assert (
await client.get("/api/v2/memory-writes/same-request-id", params={"user_id": "kid"})
).status_code == 404
# `local` cannot name another user's namespaced queue key directly.
with sqlite3.connect(app.state.settings.data_dir / "one-memory.sqlite3") as db:
keys = [row[0] for row in db.execute("SELECT idempotency_key FROM memory_write_jobs")]
wife_key = next(key for key in keys if key.startswith("user:wife:"))
assert (await client.get(f"/api/v2/memory-writes/{wife_key}")).status_code == 404
run_with_sdk(tmp_path, monkeypatch, scenario)
@pytest.mark.fresh_sdk
def test_omitting_user_id_keeps_local_and_recall_and_ask_pass_it_through(
tmp_path, monkeypatch
) -> None:
async def scenario(app, sdk, client):
await _write_both_users(app, sdk, client)
seen: list[str] = []
original = sdk.list_memories
def recording(**kwargs):
seen.append(kwargs["user_id"])
return original(**kwargs)
sdk.list_memories = recording # type: ignore[method-assign]
# Built at startup over the real handle; rebuilt here over the stand-in.
app.state.recall_target = _sdk_recall_target(app, "local")
await client.get("/api/v2/memory-library")
await client.get("/api/v2/raw")
await client.get("/api/v2/home-summary")
assert set(seen) == {"local"}
monkeypatch.setattr(
"one_memory.api._recall_model_ready", lambda _app: _true(), raising=True
)
recalled = await client.post("/api/v1/recall", json={"query": "drinks", "user_id": "wife"})
assert recalled.status_code == 200, recalled.text
assert {hit["content"] for hit in recalled.json()["hits"]} == {
"wife drinks coffee",
"wife walks the coffee route",
}
local = await client.post("/api/v1/recall", json={"query": "drinks"})
assert all("wife" not in hit["content"] for hit in local.json()["hits"])
answered = await client.post(
"/api/v2/memories/ask", json={"question": "what does she drink", "user_id": "wife"}
)
assert answered.status_code == 200, answered.text
assert {memory["content"] for memory in answered.json()["memories"]} <= {
"wife drinks coffee",
"wife walks the coffee route",
}
run_with_sdk(tmp_path, monkeypatch, scenario)
async def _true() -> bool:
return True
@pytest.mark.fresh_sdk
@pytest.mark.parametrize(
("method", "path", "kwargs"),
[
("get", "/api/v2/memory-library", {"params": {"nobody": "1"}}),
("get", "/api/v2/home-summary", {"params": {"userid": "wife"}}),
("get", "/api/v2/user-profile", {"params": {"user": "wife"}}),
("get", "/api/v2/raw", {"params": {"scope": "x"}}),
("get", "/api/v2/memories/graph", {"params": {"scene": "all"}}),
("get", "/api/v2/memories/some-id", {"params": {"owner": "wife"}}),
("delete", "/api/v2/memories/some-id", {"params": {"owner": "wife"}}),
("get", "/api/v2/memory-writes/some-id", {"params": {"owner": "wife"}}),
("get", f"/api/v2/memories/additions/{uuid4()}", {"params": {"owner": "wife"}}),
("post", "/api/v2/memories/ask", {"json": {"question": "q", "user": "wife"}}),
("post", "/api/v1/recall", {"json": {"query": "q", "user": "wife"}}),
("post", "/api/v1/recall", {"json": {"query": "q"}, "params": {"user_id": "wife"}}),
("post", "/api/v2/context/search", {"json": {"query": "q", "owner": "wife"}}),
(
"post",
"/api/v2/memories",
{"json": {"request_id": str(uuid4()), "text": "t", "scope": "all-apps", "u": 1}},
),
("put", "/api/v2/memories/some-id", {"json": {"content": "c", "owner": "wife"}}),
],
)
def test_unknown_query_and_body_parameters_answer_422(tmp_path, method, path, kwargs) -> None:
async def scenario():
app = application(tmp_path)
async with app.router.lifespan_context(app), client_for(app) as client:
response = await client.request(method.upper(), path, **kwargs)
assert response.status_code == 422, response.text
assert response.json()["detail"] == "request_validation_failed"
asyncio.run(scenario())
@pytest.mark.fresh_sdk
@pytest.mark.parametrize("bad", ["a:b", "", "x" * 65, "white space", "ü", "a/b"])
def test_a_malformed_user_id_answers_422(tmp_path, bad) -> None:
async def scenario():
app = application(tmp_path)
async with app.router.lifespan_context(app), client_for(app) as client:
responses = [
await client.get("/api/v2/memory-library", params={"user_id": bad}),
await client.get("/api/v2/user-profile", params={"user_id": bad}),
await client.post(
"/api/v2/memories",
json={
"request_id": str(uuid4()),
"text": "t",
"scope": "all-apps",
"user_id": bad,
},
),
await client.post(
"/api/v2/memory-writes",
json={
"request_id": "r",
"session_id": "s",
"messages": _messages("hello"),
"user_id": bad,
},
),
await client.post("/api/v1/recall", json={"query": "q", "user_id": bad}),
]
assert [response.status_code for response in responses] == [422] * len(responses)
asyncio.run(scenario())
def test_user_ids_and_lanes() -> None:
assert valid_user_id("wife") and valid_user_id("A.b_c-9")
assert not valid_user_id("a:b") and not valid_user_id("wife\n")
assert lane_key("local", "hermes") == "hermes"
assert lane_key("wife", "hermes") == "hermes@wife"
assert split_lane("hermes@wife") == ("wife", "hermes")
assert split_lane("hermes") == ("local", "hermes")
# `local` keeps the session identity it always had; another user's never equals it.
assert internal_session_id_for("s-1", "hermes") == "s-1"
assert internal_session_id_for("s-1", "hermes", "wife") != "s-1"
assert internal_session_id_for("s-1", "hermes", "wife") != internal_session_id_for(
"s-1", "hermes", "kid"
)
def _hermes_state_db(root: Path, rows: list[tuple[int, str, str, str]]) -> None:
root.mkdir(parents=True, exist_ok=True)
connection = sqlite3.connect(root / "state.db")
connection.execute(
"CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT, role TEXT,"
" content TEXT, timestamp REAL, tool_name TEXT, active INTEGER)"
)
connection.executemany(
"INSERT INTO messages VALUES (?, ?, ?, ?, 1790000000, NULL, 1)",
rows,
)
connection.commit()
connection.close()
def _hermes_homes(home: Path) -> tuple[Path, Path]:
hermes = home / ".hermes"
wife = hermes / "profiles" / "wife"
# The same Hermes session id in both profiles: identities must not merge.
_hermes_state_db(
hermes,
[(1, "session-1", "user", "I am the husband"), (2, "session-1", "assistant", "hi")],
)
_hermes_state_db(
wife,
[(1, "session-1", "user", "I am the wife"), (2, "session-1", "assistant", "hello")],
)
return hermes, wife
def _scan_texts(root: Path, home: Path) -> list[str]:
adapter = create_hermes_adapter(
AuthorizedSourceRoot.grant(
source_id=HERMES_DESCRIPTOR.source_id, root=root, user_confirmed=True, home=home
)
)
async def collect() -> list[str]:
return [
event.evidence.verbatim
async for event in adapter.scan(ScanOptions(cursors={}))
if isinstance(event, ScannedEvidence)
]
return asyncio.run(collect())
def test_a_hermes_root_scan_never_reads_its_profiles(tmp_path) -> None:
hermes, wife = _hermes_homes(tmp_path)
(hermes / "profiles" / "wife" / "sessions").mkdir()
(hermes / "profiles" / "wife" / "sessions" / "s.jsonl").write_text(
json.dumps({"type": "message", "role": "user", "content": "wife's transcript"}) + "\n"
)
assert _scan_texts(hermes, tmp_path) == ["I am the husband", "hi"]
assert _scan_texts(wife, tmp_path) == ["I am the wife", "hello", "wife's transcript"]
def _jobs(database: Path) -> list[sqlite3.Row]:
with sqlite3.connect(database) as db:
db.row_factory = sqlite3.Row
return db.execute(
"SELECT session_key,source_key,idempotency_key,state,payload_json "
"FROM memory_write_jobs ORDER BY queue_seq"
).fetchall()
@pytest.mark.fresh_sdk
def test_two_hermes_grants_each_read_their_own_root_into_separate_batches(tmp_path) -> None:
async def scenario():
home = tmp_path / "home"
_hermes, wife = _hermes_homes(home)
app = application(tmp_path, home=home)
async with app.router.lifespan_context(app), client_for(app) as client:
location = (await client.post("/api/v1/sources/hermes/location")).json()
local = await client.post(
"/api/v1/sources/connect",
json={
"source_ids": ["hermes"],
"request_id": str(uuid4()),
"source_roots": {"hermes": location["root_id"]},
"enable_sync": True,
},
)
assert local.status_code == 200, local.text
profile = await client.post(
"/api/v1/sources/connect",
json={
"source_ids": ["hermes"],
"request_id": str(uuid4()),
"user_id": "wife",
"root": str(wife),
},
)
assert profile.status_code == 200, profile.text
[wife_job] = profile.json()["jobs"]
assert wife_job["user_id"] == "wife"
jobs = [*local.json()["jobs"], wife_job]
for job in jobs:
finished = await app.state.import_supervisor.wait(job["job_id"])
assert finished.status == "completed", finished
rows = [row for row in _jobs(app.state.settings.data_dir / "one-memory.sqlite3")]
assert rows, "history import wrote no batches"
by_user: dict[str, list[str]] = {}
for row in rows:
records = json.loads(row["payload_json"])["input_records"]
users = {record["user_id"] for record in records}
assert len(users) == 1, "a batch mixed users"
[user] = users
by_user.setdefault(user, []).extend(record["verbatim"] for record in records)
assert row["source_key"] == lane_key(user, "hermes")
assert row["session_key"] == internal_session_id_for("session-1", "hermes", user)
assert sorted(by_user["local"]) == ["I am the husband", "hi"]
assert sorted(by_user["wife"]) == ["I am the wife", "hello"]
grants = (await client.get("/api/v1/source-grants")).json()["grants"]
assert {(grant["user_id"], grant["source_id"]) for grant in grants} >= {
("local", "hermes"),
("wife", "hermes"),
}
# The App's view stays `local`'s.
listed = (await client.get("/api/v1/import-jobs")).json()
assert {job["user_id"] for job in listed} == {"local"}
wife_jobs = (await client.get("/api/v1/import-jobs", params={"user_id": "wife"})).json()
assert {job["user_id"] for job in wife_jobs} == {"wife"}
assert all(
item["source_id"] != "hermes@wife"
for item in (await client.get("/api/v1/live-sync")).json()
)
revoked = await client.post("/api/v1/sources/hermes/revoke", params={"user_id": "wife"})
assert revoked.status_code == 200 and revoked.json()["revoked_at"] is not None
local_grant = app.state.source_grant_store.receipt(user_id="local", source_id="hermes")
assert local_grant.revoked_at is None
asyncio.run(scenario())
@pytest.mark.fresh_sdk
def test_only_hermes_takes_another_user_or_a_custom_root(tmp_path) -> None:
async def scenario():
home = tmp_path / "home"
(home / ".claude" / "projects").mkdir(parents=True)
app = application(tmp_path, home=home)
async with app.router.lifespan_context(app), client_for(app) as client:
for extra in ({"user_id": "wife"}, {"root": str(home / ".claude" / "projects")}):
response = await client.post(
"/api/v1/sources/connect",
json={
"source_ids": ["claude-code"],
"request_id": str(uuid4()),
"source_roots": {},
**extra,
},
)
assert response.status_code == 422, response.text
assert response.json()["detail"] == "user_id_requires_multi_user_source"
revoke = await client.post(
"/api/v1/sources/claude-code/revoke", params={"user_id": "wife"}
)
assert revoke.status_code == 422
# Hermes too: another user must name their own root.
rootless = await client.post(
"/api/v1/sources/connect",
json={
"source_ids": ["hermes"],
"request_id": str(uuid4()),
"source_roots": {},
"user_id": "wife",
},
)
assert rootless.status_code == 422
assert rootless.json()["detail"] == "source_root_required_for_user"
asyncio.run(scenario())
def _record(user_id: str, message_id: str, role: str, text: str):
return materialize_write_input(
EvidenceInput(
user_id=user_id,
source_id="hermes",
conversation_id="session-1",
message_id=message_id,
role=role,
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
created_at=datetime(2026, 10, 1, tzinfo=UTC),
verbatim=text,
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
)
def test_assembling_rows_and_batches_are_keyed_by_user(tmp_path) -> None:
"""Two users' `state.db` share a relative file key and a session id.
Finishing one user's file must neither take nor drop the other user's
staged turns, and the batches it writes hold one user each.
"""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path, sdk_authority=True)
file_key = "f" * 64
from one_memory.database_access import main_database
with main_database(path).write(row_factory=sqlite3.Row) as connection, connection:
for user in ("local", "wife"):
for index, role in enumerate(("user", "assistant")):
queue.stage_sdk_input(
_record(user, f"m{index}", role, f"{user} {role}"),
file_key=file_key,
priority="normal",
connection=connection,
)
queue.finalize_sdk_file(
user_id="local", source_id="hermes", file_key=file_key, connection=connection
)
rows = _jobs(path)
pending = [row for row in rows if row["state"] == "pending"]
assembling = [row for row in rows if row["state"] == "assembling"]
assert [
{record["user_id"] for record in json.loads(row["payload_json"])["input_records"]}
for row in pending
] == [{"local"}]
assert [split_lane(row["source_key"]) for row in assembling] == [("wife", "hermes")]
assert pending[0]["session_key"] != assembling[0]["session_key"]
def _observed(response) -> tuple[int, bytes]:
return response.status_code, response.content
@pytest.mark.fresh_sdk
def test_another_users_id_answers_exactly_like_an_id_that_does_not_exist(
tmp_path, monkeypatch
) -> None:
"""Comparing responses must not tell whether another user's id exists."""
async def scenario(app, sdk, client):
receipts = await _write_both_users(app, sdk, client)
[wife_memory] = [
memory_id
for memory_id, row in sdk.memories.items()
if row["user_id"] == "wife" and row["content"] == "wife drinks coffee"
]
before = {key: dict(row) for key, row in sdk.memories.items()}
with sqlite3.connect(app.state.settings.data_dir / "one-memory.sqlite3") as db:
keys = [row[0] for row in db.execute("SELECT idempotency_key FROM memory_write_jobs")]
wife_key = next(key for key in keys if key.startswith("user:wife:"))
pairs = [
("GET", f"/api/v2/memories/{wife_memory}", "/api/v2/memories/no-such-memory", {}),
(
"PUT",
f"/api/v2/memories/{wife_memory}",
"/api/v2/memories/no-such-memory",
{"json": {"content": "stolen"}},
),
("DELETE", f"/api/v2/memories/{wife_memory}", "/api/v2/memories/no-such-memory", {}),
(
"GET",
f"/api/v2/memories/additions/{receipts['add:wife']}",
f"/api/v2/memories/additions/{uuid4()}",
{},
),
("GET", f"/api/v2/memory-writes/{wife_key}", "/api/v2/memory-writes/no-such-id", {}),
]
for method, theirs, missing, kwargs in pairs:
other = await client.request(method, theirs, **kwargs)
absent = await client.request(method, missing, **kwargs)
assert _observed(other) == _observed(absent), (method, theirs)
assert other.status_code == (204 if method == "DELETE" else 404)
assert sdk.memories == before
run_with_sdk(tmp_path, monkeypatch, scenario)
@@ -173,12 +173,14 @@ def test_first_import_gate_waits_for_every_selected_history_job() -> None:
jobs = [
SimpleNamespace(
operation="history",
user_id="local",
source_id="cursor",
created_at=datetime(2026, 10, 18, tzinfo=UTC),
status="completed",
),
SimpleNamespace(
operation="history",
user_id="local",
source_id="codex",
created_at=datetime(2026, 10, 18, tzinfo=UTC),
status="running",
@@ -210,7 +210,7 @@ def test_recall_route_falls_back_and_keeps_the_raw_grant(tmp_path, monkeypatch):
async def no_model(_app):
return False
async def raw(_app, query, *, limit, sources):
async def raw(_app, query, *, limit, sources, user_id="local"):
searched.append(sources)
return (_raw("r1", "上周决定用 Postgres", source="cursor"),)
@@ -1353,7 +1353,7 @@ def test_the_sdk_memory_map_draws_sdk_memories_without_legacy_tables(tmp_path) -
original = app.state.sdk_client_handle
app.state.sdk_client_handle = Handle()
async def graph_snapshot(*, limit):
async def graph_snapshot(*, limit, user_id="local"):
del limit
return HyGraphSnapshot(
nodes=(HyGraphNode(node_id="t1", content="topic", layer="TOPIC"),),
@@ -91,7 +91,7 @@ def test_bundle_contains_management_sdk_config_and_verified_manifest(tmp_path):
assert manifest["bundle_id"] == "migration-1"
assert manifest["sdk_namespace"] == "governed-v2"
assert manifest["runtime_build"] == "build-13"
assert manifest["snapshot_sdk_version"] == "2.0.8"
assert manifest["snapshot_sdk_version"] == "2.0.9"
paths = {item["path"]: item for item in manifest["files"]}
assert {
"management/one-memory.sqlite3",
@@ -57,10 +57,10 @@ def client_for(app, headers=None):
class Answerer:
async def answer(self, question):
async def answer(self, question, *, user_id="local"):
return SdkMemoryAnswer(answer="真实数据。", memories=(MEMORY,), mode="extractive")
async def answer_stream(self, question):
async def answer_stream(self, question, *, user_id="local"):
yield {"type": "recall", "memories": (MEMORY,)}
yield {
"type": "final",
+4 -4
View File
@@ -991,7 +991,7 @@ wheels = [
[[package]]
name = "hy-memory"
version = "2.0.8"
version = "2.0.9"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "chromadb" },
@@ -1008,9 +1008,9 @@ dependencies = [
{ name = "scikit-learn" },
{ name = "tiktoken" },
]
sdist = { url = "https://files.pythonhosted.org/packages/21/15/83ca83b649ead238f186fdb33efdc73615fbba279efbbdca3857c1a86beb/hy_memory-2.0.8.tar.gz", hash = "sha256:0fa0ed1a1160b389a3d7c13a9fb6528b140d7ec23cf49e65f1be5974820a576b", size = 856025, upload-time = "2026-10-28T07:35:47.783Z" }
sdist = { url = "https://files.pythonhosted.org/packages/48/28/e01fd9a92cd090c8236df0dd06dae049c5de66345e5bd9653497ed56125e/hy_memory-2.0.9.tar.gz", hash = "sha256:5f55ea92044ced19a00b4e56c565215adedd3439a1bdb9cfe9ffe42387c8f1a1", size = 858323, upload-time = "2026-11-01T00:29:49.914Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/11/6a/c699cac0ff06bd089fceeff90446130763d8187e649dbdb0313014491a10/hy_memory-2.0.8-py3-none-any.whl", hash = "sha256:68bd789df24e51e64bb231e97a8d24ee89b4aa5f9ba14c3e0824589e9213bf7c", size = 921119, upload-time = "2026-10-28T07:35:45.571Z" },
{ url = "https://files.pythonhosted.org/packages/d5/44/4368555a408a58daf902c81b62184a3c99f12abd45d4f1144bf79f171a6f/hy_memory-2.0.9-py3-none-any.whl", hash = "sha256:95dbea232d2041f9b0ea3d08dd751603c2a5c0e4422b97e74f664e6ca913549e", size = 923250, upload-time = "2026-11-01T00:29:47.814Z" },
]
[[package]]
@@ -1837,7 +1837,7 @@ requires-dist = [
{ name = "fastapi", specifier = ">=0.141.1" },
{ name = "filelock", specifier = ">=3.32.4" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "hy-memory", specifier = "==2.0.8" },
{ name = "hy-memory", specifier = "==2.0.9" },
{ name = "keyring", specifier = ">=25.7.0" },
{ name = "numpy", specifier = "==2.5.2" },
{ name = "onnxruntime", specifier = "==1.23.2" },
@@ -184,6 +184,9 @@ const SCAN_ISSUE_LABELS: Record<string, string> = {
tool_content_excluded: "工具输出不计入原文",
internal_reasoning_excluded: "模型的内部思考不计入原文",
not_written_by_person: "不是本人写的,未计入",
injected_machine_notice: "来源程序自己发的通知,未计入",
voice_placeholder: "语音没有转成文字,未计入",
probe: "连通性检查消息,未计入",
source_busy: "来源正被占用,稍后会自动补读",
unsupported_schema: "这个来源的文件格式暂不支持",
budget_exceeded: "超出本次处理上限,其余未读",
+1 -1
View File
@@ -282,7 +282,7 @@ Support is declared per immutable GitHub Release manifest:
- Linux x64/arm64 Runtime on native GHA runners.
- macOS arm64/x64 App/Runtime: Developer ID, notarization, native acceptance.
- Windows is not supported.
- `hy-memory==2.0.8` is the current stable SDK dependency; the release
- `hy-memory==2.0.9` is the current stable SDK dependency; the release
lock and [`../MEMORY.md`](../MEMORY.md) are authoritative when the pin changes.
See [`INSTALLATION.md`](INSTALLATION.md).
+37
View File
@@ -53,6 +53,26 @@ node apps/cli/dist/one-memory.cjs sources connect --all-detected --import
(如 `DSH_HOME`)的来源必须加 `--confirm-overrides`,否则被拒绝(或在
`--all-detected` 时跳过并列出)。用同一个 `--request-id <uuid>` 重试不会重复导入。
### 接入 Hermes 的另一个 profile(多用户)
一个 Runtime 可以给多个用户记忆。**用户由读的是哪个目录决定**,不看会话或平台账号:
`~/.hermes/state.db` 属于默认用户 `local`,`~/.hermes/profiles/<name>/state.db` 是另一个用户。
```bash
# 把 wife 这个 Hermes profile 接成用户 wife:授权 + 导入历史,一步完成
node apps/cli/dist/one-memory.cjs sources connect hermes --user-id wife --root ~/.hermes/profiles/wife
# 也可以显式加 --import,效果相同;可带 --request-id <uuid> 幂等重试
node apps/cli/dist/one-memory.cjs sources connect hermes --user-id wife --root ~/.hermes/profiles/wife --import --json
# 列出来源;其他用户的连接单独列在「其他用户的连接」下,--json 里在 grants 字段
node apps/cli/dist/one-memory.cjs sources list
```
- 只有 `hermes` 能带 `--user-id`(非 `local`)或 `--root`;其他来源带了会退 2,什么都不发。
- `--root` 里的 `~` 由 CLI 展开为主目录,相对路径按当前目录解析成绝对路径。
- 扫描 `~/.hermes` 时会排除 `profiles/**`,所以 `local` 不会读到 wife 的 `state.db`。
- App 不变:它只读写 `local`,其他来源也都固定是 `local`。
## 3. 看导入进度
```bash
@@ -105,6 +125,23 @@ node apps/cli/dist/one-memory.cjs write status <request_id> --json
两条都返回 request id;它是幂等键,结果未知时用**同一个** `--request-id` 重试不会重复写入。
### 按用户读写(`--user-id`)
记忆命令都接受全局 `--user-id <id>`(默认 `local`,即 App 看到的那份):
`memories list/show/add/add-status/update/delete`、`recall`、`profile`、`write submit/status`。
```bash
node apps/cli/dist/one-memory.cjs memories list --user-id wife --json
node apps/cli/dist/one-memory.cjs memories add "我更喜欢喝绿茶" --user-id wife
node apps/cli/dist/one-memory.cjs recall "喝什么茶" --user-id wife
node apps/cli/dist/one-memory.cjs profile --user-id wife
node apps/cli/dist/one-memory.cjs write status <request_id> --user-id wife
```
- 用户 ID 只能是 1–64 位字母、数字、`.`、`_`、`-`;其他写法退 2,不连 Runtime。
- 不带 `--user-id`(或写 `--user-id local`)时请求里不出现 `user_id`,和以前完全一样。
- 用户之间互相隔离:读、改、删另一个用户的记忆 ID 返回「未找到」;write 的 request id 也只属于提交它的用户。
## 8. 召回(recall)
`recall` 两条路径(raw / memory)此前已真机验证。文本输出会标明每条是「原始对话摘录」
@@ -90,6 +90,7 @@ describe("diagnostic log", () => {
expect(await readDiagnosticEvents(dataDir)).toEqual({
events: [],
invalid_lines: 0,
unrecognized_lines: 0,
});
});
@@ -496,6 +497,52 @@ describe("diagnostic log", () => {
expect(await readdir(outputDir)).toEqual([]);
});
it("counts only non-JSON lines as damaged and leaves older records out", async () => {
const dataDir = await root();
const logger = new DiagnosticLogger({ dataDir, service: "core" });
await logger.log({
level: "info",
component: "core.service",
event: "core.ready",
outcome: "succeeded",
});
const older = {
schema_version: 0,
service: "desktop",
component: "desktop.lifecycle",
event: "desktop.started",
};
await writeFile(
path.join(dataDir, "logs", "diagnostic-desktop.jsonl"),
[
JSON.stringify(older),
JSON.stringify({ ...older, schema_version: 1 }),
JSON.stringify(["not", "an", "event"]),
"",
].join("\n"),
{ mode: 0o600 },
);
const result = await readDiagnosticEvents(dataDir);
expect(result.events.map((event) => event.event)).toEqual(["core.ready"]);
expect(result.invalid_lines).toBe(0);
expect(result.unrecognized_lines).toBe(3);
// An older build's records do not stop a support export either.
const target = path.join(await root(), "bundle.json");
await expect(
exportDiagnosticBundle({ dataDir, target, productVersion: "1.2.3" }),
).resolves.toEqual({ event_count: 1, invalid_lines: 0 });
await writeFile(
path.join(dataDir, "logs", "diagnostic-cli.jsonl"),
'{"schema_version": 1, "event": \n',
{ mode: 0o600 },
);
const damaged = await readDiagnosticEvents(dataDir);
expect(damaged.invalid_lines).toBe(1);
expect(damaged.unrecognized_lines).toBe(3);
});
it("fails closed when the diagnostic directory cannot be read", async () => {
if (process.platform === "win32") return;
const dataDir = await root();
@@ -429,10 +429,18 @@ export async function readDiagnosticEvents(
errorsOnly?: boolean;
tail?: number;
} = {},
): Promise<{ events: DiagnosticEventV1[]; invalid_lines: number }> {
): Promise<{
events: DiagnosticEventV1[];
invalid_lines: number;
unrecognized_lines: number;
}> {
const files = await diagnosticLogFiles(dataDir);
let events: DiagnosticEventV1[] = [];
// Only a line that is not JSON at all is damage. A whole JSON line this
// reader does not accept (an older build's record, another schema) is left
// out and counted apart, so an upgrade does not read as a broken log.
let invalidLines = 0;
let unrecognizedLines = 0;
for (const file of files) {
const content = await readRegularFileNoFollow(
file,
@@ -440,11 +448,17 @@ export async function readDiagnosticEvents(
);
for (const line of content.split("\n")) {
if (!line) continue;
let value: unknown;
try {
const item = parseDiagnosticEvent(JSON.parse(line));
events.push(item);
value = JSON.parse(line);
} catch {
invalidLines += 1;
continue;
}
try {
events.push(parseDiagnosticEvent(value));
} catch {
unrecognizedLines += 1;
}
}
}
@@ -480,7 +494,11 @@ export async function readDiagnosticEvents(
return !(filter.errorsOnly && item.level !== "error");
});
const tail = Math.min(Math.max(filter.tail ?? 200, 1), 5_000);
return { events: events.slice(-tail), invalid_lines: invalidLines };
return {
events: events.slice(-tail),
invalid_lines: invalidLines,
unrecognized_lines: unrecognizedLines,
};
}
export async function exportDiagnosticBundle(options: {
+1 -1
View File
@@ -36,7 +36,7 @@ MEMMY_LICENSE = (
MEMMY_PROVENANCE = MEMMY_LICENSE.with_name("PROVENANCE.md")
MEMMY_PIN = "25acd5f29fc7df73b943713abe61f1b2f9839887"
HY_MEMORY_NAME = "hy-memory"
HY_MEMORY_VERSION = "2.0.8"
HY_MEMORY_VERSION = "2.0.9"
class SbomError(RuntimeError):
+1 -1
View File
@@ -12,7 +12,7 @@ test("installation docs disclose publication and lifecycle boundaries", async ()
"utf8",
);
assert.match(readme, /`hy-memory==2\.0\.8`/);
assert.match(readme, /`hy-memory==2\.0\.9`/);
assert.match(readme, /`v1\.0\.5` 的发行目标是 Linux x64\/arm64 Runtime/);
assert.match(readme, /Linux 制品由 GitHub Actions/);
assert.match(readme, /macOS.*Developer ID 签名、公证/s);
+3 -3
View File
@@ -13,7 +13,7 @@ spec.loader.exec_module(module)
def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) -> None:
dist_info = tmp_path / "hy_memory-2.0.8.dist-info"
dist_info = tmp_path / "hy_memory-2.0.9.dist-info"
licenses = dist_info / "licenses"
licenses.mkdir(parents=True)
(dist_info / "METADATA").write_text(
@@ -21,7 +21,7 @@ def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) ->
(
"Metadata-Version: 2.4",
"Name: hy-memory",
"Version: 2.0.8",
"Version: 2.0.9",
"License-Expression: Apache-2.0",
"",
)
@@ -38,7 +38,7 @@ def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) ->
notice = module.hy_memory_notice(dist_info)
assert notice["name"] == "hy-memory"
assert notice["version"] == "2.0.8"
assert notice["version"] == "2.0.9"
assert notice["license"] == "Apache-2.0"
assert notice["copyright"] == "Copyright (C) 2026 Tencent. All rights reserved."
assert "Licensed under the Apache License" in notice["text"]
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hy-memory"
version = "2.0.8"
version = "2.0.9"
description = "HY Memory - Industrial-grade dual-system cognitive memory framework"
readme = "README.md"
requires-python = ">=3.9"
+175
View File
@@ -0,0 +1,175 @@
"""P0-1:占位符不得污染 L0 基础画像。
弱模型(qwen3.5-2b)在用户什么都没说时把 basic_info 填成 {"name": "用户"} /
"The user" / "N/A",每个占位符都 supersede 掉真实值,演化链变成
费天翔 → Fei → 用户 → The user。防御:(a) upsert 占位符守卫 + 读侧愈合;
(c) prompt 硬约束。验收 2(assistant 引用 "name is 用户")由占位符守卫单独兜住。
本文件覆盖改进建议第 10 节验收 1–4。
"""
import asyncio
from datetime import datetime, timedelta
import pytest
from hy_memory.agent.extractor import EXTRACT_FEW_SHOT_EN
from hy_memory.agent.prompts_zh import EXTRACT_FEW_SHOT_ZH
from hy_memory.agent.tools.basic_profile import (
_assemble_full_kv_from_nodes,
is_placeholder_value,
upsert_basic_profile,
)
from hy_memory.models.memory import MemoryLayer, MemoryNode, MemoryStatus
FIELDS = ["name", "age", "location", "occupation", "employer"]
class FakeStore:
def __init__(self, nodes=None):
self.nodes = list(nodes or [])
async def list_by_user(self, **kwargs):
return [n for n in self.nodes if n.user_id == kwargs.get("user_id")]
async def get_by_id(self, node_id):
return next((n for n in self.nodes if n.node_id == node_id), None)
async def upsert(self, node):
self.nodes.append(node)
async def update_payload(self, node_id, updates):
node = await self.get_by_id(node_id)
for key, value in updates.items():
if key == "status" and isinstance(value, str):
value = MemoryStatus(value)
setattr(node, key, value)
class FakeEmbed:
async def embed_queued(self, text):
return [0.1] * 8
def l0_versions(store):
return [n for n in store.nodes if n.layer == MemoryLayer.L0_BASIC_INFO]
def write_basic_info(store, llm_basic_info):
"""模拟 writer 的 basic_info 段:LLM 给出的非空 basic_info 直接 upsert。"""
if not llm_basic_info:
return None
return asyncio.run(upsert_basic_profile(
user_id="u1", kv=llm_basic_info, vector_store=store,
embed_service=FakeEmbed(), allowed_fields=FIELDS,
))
# ── 验收 1:寒暄不产生新的 L0 版本 ──
def test_small_talk_creates_no_l0_version():
# [user]: 你好 / [assistant]: 我可以帮你。弱模型输出 {"name": "用户"}
store = FakeStore()
write_basic_info(store, {"name": "用户"})
write_basic_info(store, {})
assert l0_versions(store) == []
def test_placeholder_only_upsert_writes_nothing():
store = FakeStore()
for kv in ({"name": "用户"}, {"name": "The user"}, {"name": "N/A"},
{"location": "unknown", "occupation": "未提供"}, {"name": "name"}):
res = write_basic_info(store, kv)
assert res.success and not res.changed
assert l0_versions(store) == []
# ── 验收 2:assistant 引用 "name is 用户" 不写入(仅靠占位符守卫)──
def test_assistant_quoting_placeholder_does_not_write():
# [assistant]: 那条 The user's name is 用户. 更说明问题
store = FakeStore()
for kv in ({"name": "用户"}, {"name": "The user"}, {"name": "The user."}):
res = write_basic_info(store, kv)
assert res.success and not res.changed
assert l0_versions(store) == []
# ── 验收 3:用户自报姓名 ──
def test_user_states_name():
# [user]: 我叫费天翔,常住杭州,做 AI 基础设施。
store = FakeStore()
res = write_basic_info(store, {
"name": "费天翔", "location": "杭州", "occupation": "AI 基础设施从业者",
})
assert res.success and res.created
assert l0_versions(store)[0].custom["basic_info_kv"] == {
"name": "费天翔", "location": "杭州", "occupation": "AI 基础设施从业者",
}
# ── 验收 4:已有真实值不被后续占位符覆盖 ──
def test_existing_name_not_overwritten_by_placeholder():
store = FakeStore()
write_basic_info(store, {"name": "费天翔"})
assert len(l0_versions(store)) == 1
for kv in ({"name": "用户"}, {"name": "The user."}, {"name": "N/A"}):
res = write_basic_info(store, kv)
assert not res.changed
assert len(l0_versions(store)) == 1
assert _assemble_full_kv_from_nodes(list(store.nodes)) == {"name": "费天翔"}
def test_mixed_kv_keeps_real_fields_and_drops_placeholders():
store = FakeStore()
res = write_basic_info(store, {"name": "用户", "location": "杭州"})
assert res.diff_kv == {"location": "杭州"}
def test_already_poisoned_chain_heals_on_read():
now = datetime.utcnow()
def node(nid, kv, days_ago):
return MemoryNode(
node_id=nid, user_id="u1", agent_id="",
layer=MemoryLayer.L0_BASIC_INFO, content="profile",
custom={"basic_info_kv": kv},
gmt_created=now - timedelta(days=days_ago),
)
chain = [
node("a", {"name": "费天翔"}, 4), node("b", {"name": "Fei"}, 3),
node("c", {"name": "用户"}, 2), node("d", {"name": "The user"}, 1),
]
assert _assemble_full_kv_from_nodes(chain) == {"name": "Fei"}
@pytest.mark.parametrize("value", [
"", " ", "用户", "用户。", "The user", "the user.", "THE USER", "user",
"The user's", "n/a", "N/A", "na", "None", "null", "unknown", "未提供",
"未知", "不详", "-", "?", "name", " Name ", "姓名",
])
def test_placeholder_values(value):
assert is_placeholder_value("name", value)
@pytest.mark.parametrize("field,value", [
("name", "费天翔"), ("name", "Fei"), ("name", "Username42"),
("age", 30), ("age", 0), ("location", "杭州"), ("employer", "Tencent"),
])
def test_real_values_are_not_placeholders(field, value):
assert not is_placeholder_value(field, value)
# ── (c) prompt 硬约束 ──
def test_prompts_carry_hard_rule_and_small_talk_counter_example():
from hy_memory.agent import extractor, prompts_zh
en_rules = [v for v in vars(extractor).values()
if isinstance(v, str) and "HARD RULE: fill a field ONLY" in v]
zh_rules = [v for v in vars(prompts_zh).values()
if isinstance(v, str) and "**硬性规则**" in v]
assert len(en_rules) >= 2 and len(zh_rules) >= 2
assert "small talk has no basic_info" in EXTRACT_FEW_SHOT_EN
assert "寒暄没有 basic_info" in EXTRACT_FEW_SHOT_ZH
+642
View File
@@ -0,0 +1,642 @@
"""Re-embed into a new embedding space: vector stores, scan, diff and merge.
The switch model: writes are paused, every vector is computed again with the
new model into a new space, the source is never touched. The graph half lives
in ``test_reembed_graph.py``.
"""
import hashlib
import json
import math
import random
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from hy_memory import (
EmbeddingDimensionMismatchError,
MemoryConfig,
ReembedCancelled,
VectorSpace,
diff_spaces,
drop_vector_space,
plan_reembed,
reembed_text,
reembed_vector_space,
scan_vector_spaces,
space_identity,
vector_space_names,
verify_reembed,
)
from hy_memory.config import EmbedderConfig
from hy_memory.core.embed_service import EmbedService
from hy_memory.models.memory import MemoryLayer, MemoryNode, MemoryStatus
from hy_memory.reembed import VECTOR_ROLES, parse_collection_name
class FakeEmbedder:
"""Deterministic vectors: same (model, text) -> same unit vector."""
def __init__(self, model: str, dims: int, *, wrong_width_after: int = -1):
self.model = model
self.dims = dims
self.calls = []
self.texts = 0
self.wrong_width_after = wrong_width_after
def vector(self, text: str):
seed = int(hashlib.sha256(f"{self.model}\0{text}".encode()).hexdigest()[:16], 16)
rng = random.Random(seed)
values = [rng.gauss(0.0, 1.0) for _ in range(self.dims)]
norm = math.sqrt(sum(v * v for v in values)) or 1.0
return [v / norm for v in values]
async def embed_batch(self, texts, use_cache=True):
self.calls.append(list(texts))
out = []
for text in texts:
self.texts += 1
vector = self.vector(text)
if 0 <= self.wrong_width_after < self.texts:
vector = vector + [0.0]
out.append(vector)
return out
async def embed(self, text, use_cache=True):
return self.vector(text)
async def embed_queued(self, text, use_cache=True):
return self.vector(text)
def get_embedding_dims(self):
return self.dims
def _client(path):
import chromadb
return chromadb.PersistentClient(path=str(path))
def _space(path, base, dims, model=None):
return VectorSpace(base_name=base, dims=dims, persist_directory=str(path), model=model)
def _custom_state():
return {
"memory_kind": "canonical_state",
"state_key": "home_city",
"state_value": "Berlin",
"state_retrieval_text": "user lives in Berlin (home city)",
}
async def _seed(path, base, dims, *, model="old-model", users=("alice", "bob"), per_user=7):
"""Write a realistic source space through the SDK's own stores."""
from hy_memory.data.vector_store_chroma import ChromaVectorStore
old = FakeEmbedder(model, dims)
cfg = MemoryConfig()
cfg.vector_store.provider = "chroma"
cfg.vector_store.persist_directory = str(path)
cfg.vector_store.collection_name = base
cfg.vector_store.embedding_dims = dims
store = ChromaVectorStore(cfg)
await store.initialize()
nodes = []
for user in users:
for i in range(per_user):
layer = [MemoryLayer.L1_RAW, MemoryLayer.L2_FACT, MemoryLayer.L3_SUMMARY][i % 3]
node = MemoryNode(
user_id=user, agent_id="a", session_id=f"s{i % 2}",
layer=layer, content=f"{user} fact {i}",
status=MemoryStatus.SUPERSEDED if i == 4 else MemoryStatus.ACTIVE,
tags=["t1", f"t{i}"],
custom={"k": i, "nested": {"x": [1, 2]}},
)
if i == 4:
node.is_latest = False
node.superseded_by = ["later"]
if i == 5:
node.custom = _custom_state()
node.embedding = old.vector(node.content)
nodes.append(node)
await store.upsert_batch(nodes)
entities = await store._get_entity_collection()
entities.add(
ids=["e1", "e2"],
embeddings=[old.vector("Berlin"), old.vector("Dylan")],
documents=["Berlin", "Dylan"],
metadatas=[{"user_id": "alice", "label": "GPE"}, {"user_id": "bob", "label": "PERSON"}],
)
await store.close()
client = _client(path)
names = vector_space_names(_space(path, base, dims))
for role, doc in (("coding_keys", "fix the flaky test"), ("working_facts_cv", "repo: uses uv")):
coll = client.get_or_create_collection(names[role], metadata={"hnsw:space": "cosine"})
coll.add(
ids=[f"{role}:0"],
embeddings=[old.vector(doc)],
documents=[doc],
metadatas=[{"user_id": "alice", "layer": "l1_raw",
"custom": json.dumps({"coding_key_text": doc})}],
)
return names
def _snapshot(path, names):
client = _client(path)
out = {}
for role, name in names.items():
try:
coll = client.get_collection(name)
except Exception:
continue
got = coll.get(include=["documents", "metadatas", "embeddings"])
out[role] = sorted(
(rid, doc, json.dumps(meta, sort_keys=True), [round(float(x), 6) for x in emb])
for rid, doc, meta, emb in zip(got["ids"], got["documents"], got["metadatas"], got["embeddings"])
)
return hashlib.sha256(json.dumps(out, sort_keys=True).encode()).hexdigest()
def _records(path, name):
got = _client(path).get_collection(name).get(include=["documents", "metadatas", "embeddings"])
return {
rid: (doc, meta, list(emb))
for rid, doc, meta, emb in zip(got["ids"], got["documents"], got["metadatas"], got["embeddings"])
}
# ----------------------------------------------------------------------
# naming
# ----------------------------------------------------------------------
class TestVectorSpaceNames:
@pytest.mark.parametrize("dims", [512, None])
async def test_names_equal_what_the_stores_open(self, tmp_path, dims):
from hy_memory.coding.store import CodingMemoryStore
from hy_memory.data.vector_store_chroma import ChromaVectorStore
cfg = MemoryConfig()
cfg.vector_store.provider = "chroma"
cfg.vector_store.persist_directory = str(tmp_path / "v")
cfg.vector_store.collection_name = "memhub_abc123"
cfg.vector_store.embedding_dims = dims
cfg.history.db_path = str(tmp_path / "history.sqlite3")
store = ChromaVectorStore(cfg)
await store.initialize()
await store._get_entity_collection()
coding = []
for variant in ("working", "coding"):
cs = CodingMemoryStore(cfg, FakeEmbedder("m", dims or 8), variant=variant)
await cs.initialize()
coding.append(cs)
opened = {c.name for c in _client(tmp_path / "v").list_collections()}
names = vector_space_names(cfg)
assert set(names) == set(VECTOR_ROLES)
assert set(names.values()) == opened
suffix = f"_{dims}" if dims else ""
assert names["memory"] == f"memhub_abc123{suffix}"
assert names["coding_keys_cv"] == f"memhub_abc123_coding_keys_cv{suffix}"
await store.close()
for cs in coding:
await cs.close()
def test_explicit_base_name_and_space(self, tmp_path):
cfg = MemoryConfig()
cfg.vector_store.provider = "chroma"
cfg.vector_store.embedding_dims = 1024
names = vector_space_names(cfg, base_name="staging_x")
assert names["memory"] == "staging_x_1024"
assert names["entities"] == "staging_x_1024_entities"
assert names["working_facts"] == "staging_x_working_facts_1024"
assert vector_space_names(_space(tmp_path, "b", 512))["memory"] == "b_512"
def test_parse_collection_name_round_trips(self):
for dims in (512, None):
for role, name in vector_space_names(VectorSpace("memhub_9f", dims)).items():
assert parse_collection_name(name) == ("memhub_9f", dims, role)
assert parse_collection_name("memhub_9f_512_tag_index") is None
def test_space_identity_is_model_and_width_only(self):
a = space_identity("BAAI/bge-small-zh-v1.5", 512)
b = space_identity(" BAAI/bge-small-zh-v1.5 ", 512)
assert a == b and a.key == "BAAI/bge-small-zh-v1.5#512"
assert a.digest != space_identity("BAAI/bge-small-zh-v1.5", 1024).digest
with pytest.raises(ValueError):
space_identity("", 512)
def test_non_chroma_provider_is_not_supported(self):
cfg = MemoryConfig()
cfg.vector_store.provider = "qdrant"
with pytest.raises(NotImplementedError):
vector_space_names(cfg)
class TestReembedText:
def test_canonical_state_uses_state_retrieval_text(self):
meta = {"content": "住在柏林", "custom": json.dumps(_custom_state())}
assert reembed_text("memory", "住在柏林", meta) == "user lives in Berlin (home city)"
def test_retrieval_text_when_present(self):
meta = {"custom": json.dumps({"retrieval_text": " pattern: plans ahead "})}
assert reembed_text("memory", "画像", meta) == "pattern: plans ahead"
def test_document_otherwise_with_content_fallback(self):
assert reembed_text("memory", "plain fact", {"custom": "{}"}) == "plain fact"
assert reembed_text("memory", None, {"content": "from metadata"}) == "from metadata"
assert reembed_text("memory", "doc", {"custom": "not json"}) == "doc"
def test_coding_key_and_working_fact_use_the_document(self):
coding = {"custom": json.dumps({"coding_key_text": "fix flaky test",
"retrieval_text": "ignored outside memory"})}
assert reembed_text("coding_keys", "fix flaky test", coding) == "fix flaky test"
assert reembed_text("working_facts", "repo: uses uv", {}) == "repo: uses uv"
# ----------------------------------------------------------------------
# re-embed
# ----------------------------------------------------------------------
@pytest.mark.parametrize(
"src_dims,tgt_dims,src_model,tgt_model",
[(512, 512, "old-model", "new-model"), (512, 1024, "old", "new"), (1024, 512, "old", "new")],
)
async def test_reembed_copies_everything_and_never_touches_source(
tmp_path, src_dims, tgt_dims, src_model, tgt_model,
):
path = tmp_path / "v"
names = await _seed(path, "memhub_src", src_dims, model=src_model)
before = _snapshot(path, names)
source = _space(path, "memhub_src", src_dims)
target = _space(path, "memhub_dst", tgt_dims, model=tgt_model)
new = FakeEmbedder(tgt_model, tgt_dims)
plan = await plan_reembed(source, target)
assert plan.per_role["memory"] == 14 and plan.per_role["entities"] == 2
assert plan.total_vectors == 18 == plan.estimated_vectors
assert plan.source_dims == src_dims and plan.target_dims == tgt_dims
seen = []
report = await reembed_vector_space(
source, target, embedding=new, batch_size=4, on_progress=seen.append,
)
assert report.embedded == 18 and report.reused == 0
assert seen and seen[-1].done == seen[-1].total
assert _snapshot(path, names) == before # source unchanged, byte for byte
target_names = vector_space_names(target)
for role in ("memory", "entities", "coding_keys", "working_facts_cv"):
src = _records(path, names[role])
dst = _records(path, target_names[role])
assert set(src) == set(dst)
for rid, (doc, meta, _) in src.items():
assert dst[rid][0] == doc
assert dst[rid][1] == meta # status, is_latest, superseded_by, timestamps, custom
assert len(dst[rid][2]) == tgt_dims
assert dst[rid][2] == pytest.approx(new.vector(reembed_text(role, doc, meta)), abs=1e-5)
memory = _records(path, target_names["memory"])
layers = {meta["layer"] for _, meta, _ in memory.values()}
users = {meta["user_id"] for _, meta, _ in memory.values()}
assert "l1_raw" in layers and users == {"alice", "bob"}
statuses = {meta["status"] for _, meta, _ in memory.values()}
assert statuses == {"active", "superseded"}
state = [r for r in memory.values() if "canonical_state" in r[1]["custom"]]
assert state and state[0][2] == pytest.approx(
new.vector("user lives in Berlin (home city)"), abs=1e-5,
)
meta = _client(path).get_collection(target_names["memory"]).metadata
assert meta["hnsw:space"] == "cosine" and meta["hy_embedding_model"] == tgt_model
verified = await verify_reembed(source, target, embedding=new)
assert verified.ok, verified.problems
assert verified.per_user == {"alice": 10, "bob": 8}
assert verified.self_recall_samples == 18 and verified.self_recall_hits == 18
async def test_resume_after_kill_only_embeds_the_rest(tmp_path):
path = tmp_path / "v"
await _seed(path, "src", 512, per_user=20)
source, target = _space(path, "src", 512), _space(path, "dst", 1024)
class Killed(Exception):
pass
first = FakeEmbedder("new", 1024)
batches = {"n": 0}
def _kill():
batches["n"] += 1
if batches["n"] > 3:
raise Killed()
return False
with pytest.raises(Killed):
await reembed_vector_space(
source, target, embedding=first, batch_size=5, max_inflight=1,
roles=["memory"], cancel=_kill,
)
assert first.texts == 15
second = FakeEmbedder("new", 1024)
report = await reembed_vector_space(
source, target, embedding=second, batch_size=5, max_inflight=1, roles=["memory"],
)
assert second.texts == 40 - 15
assert report.per_role["memory"].reused == 15
# The end state equals an uninterrupted run: every source record, verbatim,
# with the vector of its own text.
resumed = _records(path, "dst_1024")
src = _records(path, "src_512")
assert set(resumed) == set(src)
for rid, (doc, meta, vector) in resumed.items():
assert (doc, meta) == src[rid][:2]
assert vector == pytest.approx(second.vector(reembed_text("memory", doc, meta)), abs=1e-5)
assert (await verify_reembed(source, target, roles=["memory"], samples=0)).ok
async def test_cancel_at_batch_boundary_leaves_no_partial_batch(tmp_path):
path = tmp_path / "v"
await _seed(path, "src", 512)
calls = {"n": 0}
def _cancel():
calls["n"] += 1
return calls["n"] > 2
with pytest.raises(ReembedCancelled) as info:
await reembed_vector_space(
_space(path, "src", 512), _space(path, "dst", 512),
embedding=FakeEmbedder("new", 512), batch_size=3, max_inflight=1,
roles=["memory"], cancel=_cancel,
)
assert info.value.report.per_role["memory"].embedded == 6
assert _client(path).get_collection("dst_512").count() == 6
async def test_wrong_width_raises_before_any_upsert(tmp_path):
path = tmp_path / "v"
await _seed(path, "src", 512)
bad = FakeEmbedder("new", 1024, wrong_width_after=2)
with pytest.raises(EmbeddingDimensionMismatchError):
await reembed_vector_space(
_space(path, "src", 512), _space(path, "dst", 1024),
embedding=bad, batch_size=4, roles=["memory"],
)
assert _client(path).get_collection("dst_1024").count() == 0
declared_wrong = FakeEmbedder("new", 768)
with pytest.raises(EmbeddingDimensionMismatchError):
await reembed_vector_space(
_space(path, "src", 512), _space(path, "dst", 1024), embedding=declared_wrong,
)
async def test_refuses_to_write_into_the_source_collection(tmp_path):
path = tmp_path / "v"
await _seed(path, "src", 512)
with pytest.raises(ValueError):
await reembed_vector_space(
_space(path, "src", 512), _space(path, "src", 512),
embedding=FakeEmbedder("new", 512),
)
async def test_verify_detects_missing_extra_drift_and_width(tmp_path):
path = tmp_path / "v"
await _seed(path, "src", 512)
source, target = _space(path, "src", 512), _space(path, "dst", 512)
new = FakeEmbedder("new", 512)
await reembed_vector_space(source, target, embedding=new, roles=["memory"])
coll = _client(path).get_collection("dst_512")
got = coll.get(include=["documents", "metadatas"])
victim, drifted = got["ids"][0], got["ids"][1]
coll.delete(ids=[victim])
meta = dict(got["metadatas"][1])
meta["status"] = "archived"
coll.update(ids=[drifted], metadatas=[meta])
coll.add(ids=["stranger"], embeddings=[new.vector("x")], documents=["x"], metadatas=[{"user_id": "eve"}])
result = await verify_reembed(source, target, embedding=new, roles=["memory"], samples=5)
assert not result.ok
counts = result.per_role["memory"]
assert (counts["missing"], counts["extra"], counts["mismatched"]) == (1, 1, 1)
assert any("eve" in p for p in result.problems)
wide = _space(path, "dst", 1024)
wide_coll = _client(path).get_or_create_collection("dst_1024", metadata={"hnsw:space": "cosine"})
src = _records(path, "src_512")
wide_coll.add(
ids=list(src),
embeddings=[[0.0] * 512 for _ in src],
documents=[v[0] for v in src.values()],
metadatas=[v[1] for v in src.values()],
)
result = await verify_reembed(source, wide, roles=["memory"], samples=0)
assert result.per_role["memory"]["bad_dims"] == len(src) and not result.ok
async def test_drop_vector_space_refuses_live_collections(tmp_path):
from hy_memory.data.vector_store_chroma import ChromaVectorStore
path = tmp_path / "v"
await _seed(path, "src", 512)
cfg = MemoryConfig()
cfg.vector_store.provider = "chroma"
cfg.vector_store.persist_directory = str(path)
cfg.vector_store.collection_name = "src"
cfg.vector_store.embedding_dims = 512
live = ChromaVectorStore(cfg)
await live.initialize()
with pytest.raises(RuntimeError):
await drop_vector_space(_space(path, "src", 512))
await live.close()
dropped = await drop_vector_space(_space(path, "src", 512))
assert set(dropped) == {"src_512", "src_512_entities", "src_coding_keys_512", "src_working_facts_cv_512"}
assert not {c.name for c in _client(path).list_collections()}
# ----------------------------------------------------------------------
# scan, diff, merge
# ----------------------------------------------------------------------
async def _two_spaces(path):
"""A live 1024 space and a stray 512 space sharing some ids."""
from hy_memory.data.vector_store_chroma import ChromaVectorStore
async def _store(base, dims, model, contents):
cfg = MemoryConfig()
cfg.vector_store.provider = "chroma"
cfg.vector_store.persist_directory = str(path)
cfg.vector_store.collection_name = base
cfg.vector_store.embedding_dims = dims
store = ChromaVectorStore(cfg)
await store.initialize()
emb = FakeEmbedder(model, dims)
nodes = []
for node_id, user, content, status in contents:
node = MemoryNode(
node_id=node_id, user_id=user, agent_id="a", session_id="s",
layer=MemoryLayer.L2_FACT, content=content, status=status,
)
node.embedding = emb.vector(content)
nodes.append(node)
await store.upsert_batch(nodes)
await store.close()
ids = [f"00000000-0000-0000-0000-00000000000{i}" for i in range(6)]
await _store("memhub_old", 512, "bge", [
(ids[0], "alice", "shared one", MemoryStatus.ACTIVE),
(ids[1], "alice", "shared two (stray wording)", MemoryStatus.ACTIVE),
(ids[2], "bob", "only in stray", MemoryStatus.ACTIVE),
(ids[3], "bob", "only in stray, deleted", MemoryStatus.SHADOW),
])
await _store("memhub_new", 1024, "e5", [
(ids[0], "alice", "shared one", MemoryStatus.ACTIVE),
(ids[1], "alice", "shared two (live wording)", MemoryStatus.ARCHIVED),
(ids[4], "carol", "only in live", MemoryStatus.ACTIVE),
])
return ids
async def test_scan_diff_and_merge(tmp_path):
path = tmp_path / "v"
ids = await _two_spaces(path)
spaces = await scan_vector_spaces(path)
by_base = {(s.base_name, s.dims): s for s in spaces}
assert set(by_base) == {("memhub_old", 512), ("memhub_new", 1024)}
old, live = by_base[("memhub_old", 512)], by_base[("memhub_new", 1024)]
assert old.roles["memory"].count == 4 and old.width == 512
assert live.roles["memory"].count == 3 and live.width == 1024
assert old.per_user == {"alice": 2, "bob": 2}
assert live.per_user == {"alice": 2, "carol": 1}
stray, target = old.space(), _space(path, "memhub_new", 1024, model="e5")
diff = await diff_spaces(stray, target)
assert set(diff.missing["memory"]) == {ids[2], ids[3]} # status ignored
assert diff.total_missing == 2
live_before = _records(path, "memhub_new_1024")
stray_before = _snapshot(path, stray.names())
merger = FakeEmbedder("e5", 1024)
report = await reembed_vector_space(stray, target, embedding=merger)
assert merger.texts == 2 and report.embedded == 2
assert report.reused == 1 and report.kept == 1
after = _records(path, "memhub_new_1024")
assert set(after) == set(live_before) | {ids[2], ids[3]}
for rid, record in live_before.items(): # the target wins
assert after[rid][:2] == record[:2]
assert after[rid][2] == pytest.approx(record[2], abs=1e-6)
assert _snapshot(path, stray.names()) == stray_before
assert (await diff_spaces(stray, target)).empty
again = FakeEmbedder("e5", 1024)
second = await reembed_vector_space(stray, target, embedding=again)
assert again.texts == 0 and second.embedded == 0
assert _records(path, "memhub_new_1024").keys() == after.keys()
rescanned = {(s.base_name, s.dims): s for s in await scan_vector_spaces(path)}
assert rescanned[("memhub_new", 1024)].roles["memory"].count == 5
async def test_scan_of_missing_directory_is_empty(tmp_path):
assert await scan_vector_spaces(tmp_path / "nope") == []
# ----------------------------------------------------------------------
# EmbedService fixes
# ----------------------------------------------------------------------
def _fake_raw(n, dims):
items = []
for i in range(n):
item = MagicMock()
item.embedding = [0.1] * dims
item.index = i
items.append(item)
resp = MagicMock()
resp.model = "m"
resp.usage = None
resp.data = items
raw = MagicMock()
raw.status_code = 200
raw.headers = {}
raw.parse.return_value = resp
return raw
def _service(dims=8, **embedder):
cfg = MemoryConfig(embedder=EmbedderConfig(
provider="openai", model="m", api_key="sk", base_url="https://e.invalid/v1",
embedding_dims=dims, max_retries=2, retry_delay=0.0, **embedder,
))
return EmbedService(cfg)
async def test_embed_batch_chunks_to_batch_size():
svc = _service()
sizes = []
async def _create(**kwargs):
sizes.append(len(kwargs["input"]))
return _fake_raw(len(kwargs["input"]), 8)
client = MagicMock()
client.embeddings.with_raw_response.create = AsyncMock(side_effect=_create)
with patch("openai.AsyncOpenAI", return_value=client):
vectors = await svc.embed_batch([f"t{i}" for i in range(1000)], use_cache=False)
assert len(vectors) == 1000
assert len(sizes) == math.ceil(1000 / 64) and max(sizes) == 64
small = _service(batch_size=10)
sizes.clear()
with patch("openai.AsyncOpenAI", return_value=client):
_, metas = await small.embed_batch_with_meta([f"u{i}" for i in range(25)], use_cache=False)
assert sizes == [10, 10, 5]
assert [m["batch_index"] for m in metas] == list(range(25))
async def test_429_honours_retry_after():
svc = _service()
class RateLimited(Exception):
status_code = 429
def __init__(self):
super().__init__("rate limited")
self.response = MagicMock()
self.response.headers = {"retry-after": "2"}
client = MagicMock()
client.embeddings.with_raw_response.create = AsyncMock(
side_effect=[RateLimited(), _fake_raw(1, 8)],
)
sleeps = []
async def _sleep(seconds):
sleeps.append(seconds)
with patch("openai.AsyncOpenAI", return_value=client), \
patch("hy_memory.core.embed_service.asyncio.sleep", _sleep):
vectors = await svc.embed_batch(["x"], use_cache=False)
assert len(vectors) == 1
assert len(sleeps) == 1 and 2.0 <= sleeps[0] <= 2.5
async def test_per_call_max_retries():
svc = _service()
class Down(Exception):
status_code = 503
client = MagicMock()
client.embeddings.with_raw_response.create = AsyncMock(side_effect=Down("down"))
with patch("openai.AsyncOpenAI", return_value=client), pytest.raises(Down):
await svc.embed_batch(["x"], use_cache=False, max_retries=5)
assert client.embeddings.with_raw_response.create.await_count == 6
async def test_returned_width_is_validated_and_not_retried():
svc = _service(dims=8)
client = MagicMock()
client.embeddings.with_raw_response.create = AsyncMock(return_value=_fake_raw(2, 7))
with patch("openai.AsyncOpenAI", return_value=client), \
pytest.raises(EmbeddingDimensionMismatchError):
await svc.embed_batch(["a", "b"], use_cache=False)
assert client.embeddings.with_raw_response.create.await_count == 1
+252
View File
@@ -0,0 +1,252 @@
"""Re-embed the Kuzu graph into a new embedding space.
The source graph is opened read-only and never modified. A new graph is built
at the target path with the target width: every node and edge is copied, the
embedding columns are recomputed from their text, the HNSW index is rebuilt.
Runs in CI: kuzu is a core dependency.
"""
import hashlib
import json
import os
import pytest
from hy_memory import (
EmbeddingDimensionMismatchError,
MemoryConfig,
ReembedCancelled,
drop_graph,
reembed_graph,
verify_graph_reembed,
)
from hy_memory.models.memory import MemoryLayer, MemoryNode
from hy_memory.reembed import hierarchy_topic_label, plan_graph_reembed
from .test_reembed import FakeEmbedder
def _cfg(path, dims):
cfg = MemoryConfig()
cfg.graph_store.provider = "kuzu"
cfg.graph_store.db_path = str(path)
cfg.vector_store.embedding_dims = dims
return cfg
async def _open(path, dims):
from hy_memory.data.graph_store_kuzu import KuzuGraphStore
store = KuzuGraphStore(_cfg(path, dims))
await store.initialize()
assert store._available
return store
def _node(content, *, layer=MemoryLayer.L2_FACT, custom=None, user="alice"):
return MemoryNode(
user_id=user, agent_id="a", session_id="s1",
layer=layer, content=content, custom=custom or {},
)
async def _seed_graph(path, dims, model="old"):
"""A graph with every node and edge type the SDK writes."""
old = FakeEmbedder(model, dims)
gs = await _open(path, dims)
schema = _node(
"画像:喜欢提前规划", layer=MemoryLayer.L6_SCHEMA,
custom={"schema_type": "basic", "retrieval_text": "plans trips weeks ahead",
"_has_beh": True, "beh_abstraction": "prepares early under uncertainty"},
)
schema._graph_embedding = old.vector("plans trips weeks ahead")
core = _node("core: anticipates", layer=MemoryLayer.L6_SCHEMA, custom={"schema_type": "abstract"})
core._graph_embedding = old.vector("core: anticipates")
stale_beh = _node("beh without text", layer=MemoryLayer.L6_SCHEMA, custom={})
stale_beh._graph_embedding = old.vector("beh without text")
facts = [_node(f"fact {i}", user="bob" if i % 2 else "alice") for i in range(5)]
for node in [schema, core, stale_beh, *facts]:
await gs.upsert_memory_node(node)
await gs.update_embedding(schema.node_id, beh_embedding=old.vector("prepares early under uncertainty"))
await gs.update_embedding(stale_beh.node_id, beh_embedding=old.vector("lost text"))
await gs.add_cross_abstracts_to(schema.node_id, core.node_id)
ik = facts[0].get_isolation_key()
await gs.add_edge(facts[0].node_id, facts[1].node_id, "RELATED_TO", {"relation_type": "co_occurs", "weight": 0.7})
await gs.add_temporal_edge(facts[0].node_id, facts[2].node_id, topic_id="t")
for fact in facts[:3]:
await gs.add_topic_tag(fact.node_id, "travel", ik, embed_service=old)
await gs.add_topic_tag(facts[3].node_id, "work", ik) # no embed service: NULL embedding
await gs.add_topic_path(
facts[0].node_id, ["Travel", "Japan trip"], ik, "alice",
node_types=["topic", "series"], embed_service=old,
)
await gs.add_topic_path(facts[1].node_id, ["Work", "Standups"], ik, "alice", embed_service=old)
await gs.add_entity_mention(facts[0].node_id, "Kyoto", "GPE", ik)
await gs.ensure_vdbref("raw-1", "l1_raw")
await gs.add_derived_from(schema.node_id, "raw-1")
await gs.close()
return old, {"schema": schema, "core": core, "stale_beh": stale_beh, "facts": facts}
def _digest(path):
"""Hash of the Kuzu DB file and any sidecar files next to it."""
h = hashlib.sha256()
for name in sorted(os.listdir(path.parent)):
if name == path.name or name.startswith(path.name + "."):
h.update(name.encode())
with open(path.parent / name, "rb") as handle:
h.update(handle.read())
return h.hexdigest()
def _read(path, query):
import kuzu
db = kuzu.Database(str(path), read_only=True)
conn = kuzu.Connection(db)
try:
result = conn.execute(query)
rows = []
while result.has_next():
rows.append(result.get_next())
return rows
finally:
conn.close()
db.close()
@pytest.mark.parametrize(
"src_dims,tgt_dims,src_model,tgt_model",
[(512, 512, "old", "new"), (512, 1024, "old", "new"), (1024, 512, "old", "new")],
)
async def test_reembed_graph_copies_everything_at_the_new_width(
tmp_path, src_dims, tgt_dims, src_model, tgt_model,
):
source, target = tmp_path / "graph", tmp_path / "graph-new"
_, nodes = await _seed_graph(source, src_dims, model=src_model)
before = _digest(source)
plan = await plan_graph_reembed(source, target_dims=tgt_dims)
assert plan.source_width == src_dims and plan.target_width == tgt_dims
assert plan.memory_nodes == 8 and plan.topics == 2 and plan.hierarchy_topics == 4
assert plan.memory_embeddings == 3 and plan.beh_embeddings == 2
assert plan.topic_embeddings == 1 and plan.hierarchy_topic_embeddings == 4
new = FakeEmbedder(tgt_model, tgt_dims)
seen = []
report = await reembed_graph(
source, target, embedding=new, target_dims=tgt_dims, batch_size=3, on_progress=seen.append,
)
assert _digest(source) == before # the source graph is untouched
assert not os.path.exists(str(target) + ".reembed-partial")
assert report.source_width == src_dims and report.target_width == tgt_dims
assert report.node_counts == plan.node_counts
assert report.rel_counts == plan.rel_counts
assert report.embedded == {
"Memory.embedding": 3, "Memory.beh_embedding": 1,
"Topic.embedding": 1, "HierarchyTopic.embedding": 4,
}
assert report.cleared == {"Memory.beh_embedding": 1} # beh vector with no beh_abstraction
assert report.hierarchy_labels_exact == 4 and report.hierarchy_labels_inexact == ()
assert report.vector_indexes == ("memory_content_idx",)
assert seen
check = await verify_graph_reembed(source, target, target_dims=tgt_dims)
assert check.ok, check.problems
assert check.vector_index and check.target_width == tgt_dims
assert all(src == dst for src, dst in check.node_counts.values())
assert all(src == dst for src, dst in check.rel_counts.values())
schema = nodes["schema"]
rows = _read(target, f"MATCH (m:Memory {{node_id: '{schema.node_id}'}}) RETURN m.embedding, m.beh_embedding, m.custom_json, m.content;")
emb, beh, custom, content = rows[0]
assert len(emb) == tgt_dims
assert emb == pytest.approx(new.vector("plans trips weeks ahead"), abs=1e-5)
assert beh == pytest.approx(new.vector("prepares early under uncertainty"), abs=1e-5)
assert json.loads(custom)["retrieval_text"] == "plans trips weeks ahead" and content == schema.content
rows = _read(target, f"MATCH (m:Memory {{node_id: '{nodes['core'].node_id}'}}) RETURN m.embedding;")
assert rows[0][0] == pytest.approx(new.vector("core: anticipates"), abs=1e-5)
rows = _read(target, f"MATCH (m:Memory {{node_id: '{nodes['facts'][0].node_id}'}}) RETURN m.embedding IS NULL;")
assert rows[0][0] is True # a NULL source vector stays NULL
topics = dict(_read(target, "MATCH (t:Topic) RETURN t.name, t.embedding;"))
assert topics["travel"] == pytest.approx(new.vector("travel"), abs=1e-5)
assert topics["work"] is None
for subject, path_json, embedding in _read(target, "MATCH (t:HierarchyTopic) RETURN t.subject, t.path_json, t.embedding;"):
label, exact = hierarchy_topic_label(subject, path_json)
assert exact and embedding == pytest.approx(new.vector(label), abs=1e-5)
# The rebuilt graph serves the SDK: vector search on the new index works.
gs = await _open(target, tgt_dims)
try:
hits = await gs.vector_search(
query_embedding=new.vector("plans trips weeks ahead"),
isolation_key=schema.get_isolation_key(), limit=1,
)
assert hits and hits[0]["node_id"] == schema.node_id
width = await gs._run("CALL table_info('Memory') RETURN *;")
assert dict((r[1], r[2]) for r in width)["embedding"] == f"FLOAT[{tgt_dims}]"
finally:
await gs.close()
async def test_source_labels_rebuild_exactly(tmp_path):
"""The stored HierarchyTopic vectors were computed from the rebuilt label."""
source = tmp_path / "graph"
old, _ = await _seed_graph(source, 64)
rows = _read(source, "MATCH (t:HierarchyTopic) RETURN t.subject, t.path_json, t.embedding;")
assert len(rows) == 4
for subject, path_json, embedding in rows:
label, exact = hierarchy_topic_label(subject, path_json)
assert exact
assert embedding == pytest.approx(old.vector(label), abs=1e-6)
def test_legacy_path_json_is_flagged_inexact():
label, exact = hierarchy_topic_label("alice", json.dumps(["Travel", "Japan"]), "Japan")
assert label == "alice: topic:Travel > facet:Japan" and not exact
label, exact = hierarchy_topic_label("alice", "", "Japan")
assert label == "alice: Japan" and not exact
async def test_cancel_and_failure_leave_no_target(tmp_path):
source, target = tmp_path / "graph", tmp_path / "graph-new"
await _seed_graph(source, 64)
with pytest.raises(ReembedCancelled):
await reembed_graph(
source, target, embedding=FakeEmbedder("new", 32), target_dims=32,
cancel=lambda: True,
)
with pytest.raises(EmbeddingDimensionMismatchError):
await reembed_graph(
source, target, embedding=FakeEmbedder("new", 32, wrong_width_after=1), target_dims=32,
)
assert not os.path.exists(target)
assert not os.path.exists(str(target) + ".reembed-partial")
# A restart starts over and succeeds.
report = await reembed_graph(source, target, embedding=FakeEmbedder("new", 32), target_dims=32)
assert report.node_counts["Memory"] == 8
with pytest.raises(FileExistsError):
await reembed_graph(source, target, embedding=FakeEmbedder("new", 32), target_dims=32)
again = await reembed_graph(
source, target, embedding=FakeEmbedder("new", 32), target_dims=32, overwrite=True,
)
assert again.rel_counts == report.rel_counts
async def test_refuses_open_source_and_drops_only_closed_graphs(tmp_path):
source, target = tmp_path / "graph", tmp_path / "graph-new"
await _seed_graph(source, 64)
live = await _open(source, 64)
try:
with pytest.raises(RuntimeError):
await reembed_graph(source, target, embedding=FakeEmbedder("new", 64), target_dims=64)
with pytest.raises(RuntimeError):
await drop_graph(source)
finally:
await live.close()
with pytest.raises(ValueError):
await reembed_graph(source, source, embedding=FakeEmbedder("new", 64), target_dims=64)
await reembed_graph(source, target, embedding=FakeEmbedder("new", 64), target_dims=64)
assert await drop_graph(target) is True
assert not os.path.exists(target)
assert await drop_graph(target) is False
assert os.path.exists(source)