Compare commits

...
Author SHA1 Message Date
alvinfeiandCursor c4da617d59 fix(migration): stage SDK data before cutover
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:39:09 +08:00
alvinfeiandCursor 729e699eab refactor(sdk): keep legacy migration out of client APIs
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:31:22 +08:00
d926844173 refactor(memhub): remove private Hy Memory bridge (#145)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: kristinhe <kristinhe@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:02:27 +08:00
de4292ed4e feat(hermes): add managed native prefetch provider (#143)
* feat: add native Hermes memory provider

* feat(hermes): support scoped private CA trust

* feat(hermes): bound write timeout separately from reads

* fix(hermes): support truststore scoped CA snapshots

* feat(hermes): add managed prefetch provider

Co-authored-by: kindong <727777477@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: kindong <727777477@qq.com>
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 16:51:15 +08:00
134 changed files with 8585 additions and 23387 deletions

No files matched your search

+49
View File
@@ -0,0 +1,49 @@
name: Hermes provider
on:
pull_request:
paths:
- "plugins/hermes/**"
- ".github/workflows/hermes-provider.yml"
push:
branches: [main]
paths:
- "plugins/hermes/**"
- ".github/workflows/hermes-provider.yml"
permissions:
contents: read
jobs:
provider:
runs-on: ubuntu-24.04
timeout-minutes: 3
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
with:
persist-credentials: false
- uses: astral-sh/setup-uv@d0d8abe699bfb85fec6de9f7adb5ae17292296ff
with:
python-version: "3.12"
enable-cache: false
- name: Download and verify native Hermes contract
run: |
uv run --no-project --python 3.12 python - <<'PY'
import hashlib
import os
from pathlib import Path
import urllib.request
url = "https://raw.githubusercontent.com/NousResearch/hermes-agent/818c13be1dc4fd28987e1e881a9408224afd4535/agent/memory_provider.py"
with urllib.request.urlopen(url, timeout=30) as response:
data = response.read()
expected = "a4e44a293013fff831ac136ab77c556541cd18817f53bd662e97090cd80bd5a6"
if hashlib.sha256(data).hexdigest() != expected:
raise SystemExit("Hermes contract SHA-256 mismatch")
snapshot = Path(os.environ["RUNNER_TEMP"]) / "hermes-memory-provider.py"
snapshot.write_bytes(data)
with open(os.environ["GITHUB_ENV"], "a") as env:
env.write(f"HERMES_MEMORY_PROVIDER_SNAPSHOT={snapshot}\n")
PY
- name: Test provider with native contract
run: uv run --no-project --python 3.12 --with pytest python -m pytest plugins/hermes/tests -o addopts= -o log_cli=false -q
+20 -9
View File
@@ -40,6 +40,17 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
- Removed MemHub's private `hy_memory_bridge` implementation. MemHub now calls
public `HyMemoryClient` methods at runtime; prompt/reconcile/response
monkey-patches and private VDB/Coding runtime access were deleted. Kuzu Graph
projection/admin and stopped-writer App migration are the documented product
exceptions.
- Extended public SDK `get()` to resolve Chat, Coding Experience, and Working
Fact IDs, including raw records, owner/session fields, and explicit ambiguity
handling; added constructor-time public embedding-service injection.
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
persistent per-session memory-ID deduplication, authorized Broker reads, and
recall-context stripping to prevent memory recursion.
- Added an extensible MemHub interface-localization layer with Simplified
Chinese, English, and Spanish catalogs shared by Web and Electron-native
surfaces. Interface language remains independent from memory content and
@@ -47,13 +58,12 @@ Hy Memory and MemHub are versioned and released independently.
- `search(..., layers=["l1_raw"])` now explicitly searches SDK raw records in
pro/ultra mode and keeps existing user/agent/session/source filters; default
search continues to hide raw.
- Added `import_memory_exact` for idempotent migration of an existing semantic
memory with its canonical ID, lifecycle and supersession links, without
extraction or reconciliation.
- Added SDK-authoritative lifecycle listing with explicit status filters,
idempotent durable write outcomes, exact raw import/get/list APIs, and
quiesced local data-plane snapshot/restore contracts for the MemHub
authority migration.
idempotent durable write outcomes, and quiesced local data-plane
snapshot/restore contracts for the MemHub authority migration. Exact legacy
backfill is now an App-owned privileged offline operation rather than a
public `HyMemoryClient` API. Backfill runs against a complete staging copy;
cutover atomically swaps SDK roots and restores the previous root on failure.
- Physical delete now reports incomplete VDB/Graph cleanup as a partial failure
instead of claiming success after only one store was changed.
- Coding search no longer injects the version-evolution diff block
@@ -76,9 +86,10 @@ Hy Memory and MemHub are versioned and released independently.
retry scheduling for their memory tasks.
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
while persistent caller business context belongs in `meta`.
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
memories, including memories written in lite mode. The default remains
hidden for backward compatibility.
- Unified exact reads and lists across memory layers: `get(memory_id)` returns
the matched layer, while `list_memories(layers=["l1_raw"])` explicitly lists
raw records. Separate raw get/list methods and the `include_raw` switch were
removed.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
+1
View File
@@ -253,6 +253,7 @@ That version is usually not in the working tree or at HEAD either, so it cannot
- [Graph memory design](docs/pro_graph_design.md) — graph projection, rerank, relation paths, and deployment limits
- [Environment template](.env.example) — a copy-ready configuration template
- [MemHub README](memhub/README.md) — local-first desktop & CLI product overview, runtime commands, and privacy boundaries
- [Hermes native memory provider](plugins/hermes/README.md) — explicit owner/agent scope; separate HTTP service; automatic capture and recall opt-in.
- [OpenClaw plugin](plugins/openclaw/README.md) — agent integrations live under `plugins/` (overview: [plugins/README.md](plugins/README.md))
### License
+1
View File
@@ -253,6 +253,7 @@ Coding 与 Working **复用同一套双层 pipeline 和数据 schema**,区别
- [Graph 记忆设计](docs/pro_graph_design.md) — 图投影、重排、关系路径与部署限制
- [环境变量模板](.env.example) — 可直接复制的配置模板
- [MemHub 说明文档](memhub/README.md) — 本地优先桌面端与 CLI 产品总览、Runtime 命令与隐私边界
- [Hermes 原生记忆 Provider](plugins/hermes/README.md) — 显式用户与 Agent 隔离,独立 HTTP 服务,自动记录与召回默认关闭。
- [OpenClaw 插件](plugins/openclaw/README.md) — Agent 集成见 `plugins/`(总览:[plugins/README.md](plugins/README.md))
### License
+7 -6
View File
@@ -273,8 +273,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
```python
memory = client.get("memory_id")
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
raw_memory = client.get("memory_id", include_raw=True)
# raw ID 返回 L1_RAW,fact ID 返回对应事实层。
raw_memory = client.get("raw_id")
result = client.update("memory_id", "更新后的内容")
result = client.delete("memory_id")
result = client.delete_all(user_id="user_001")
@@ -284,7 +284,7 @@ page = client.list_memories(
limit=100,
offset=0,
order="desc",
layers=None,
layers=["l1_raw"], # 显式列出 raw;不传仍沿用默认语义层列表
with_evolution=False,
)
@@ -294,9 +294,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
`get()` / `async_get()` 是统一的 ID 精确读取:返回结果的 `layer`、`kind` 和
`variant` 表明实际类型。授权由调用 SDK 前的产品边界完成,不由 SDK 根据 layer
猜测。`list_memories()` 默认仍不包含 `L1_RAW`;显式传
`layers=["l1_raw"]` 时按同一列表合同返回 raw。
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
+13
View File
@@ -65,6 +65,12 @@ from .snapshot import (
# ====== 配置 ======
from .config import MemoryConfig
from .embedding import (
EmbeddingDimensionMismatchError,
EmbeddingService,
EmbeddingServiceProtocol,
)
from .retrieval import AmbiguousMemoryIdError, MemoryKind, MemoryVariant
# ====== Embedding 端点覆盖(per-call,不改 model/dims,只换接入端点)======
from .core.embed_service import embed_endpoint_override
@@ -125,6 +131,13 @@ __all__ = [
"snapshot_local_data_plane",
# 配置
"MemoryConfig",
# 公共扩展协议 / 精确读取类型
"EmbeddingService",
"EmbeddingServiceProtocol",
"EmbeddingDimensionMismatchError",
"MemoryKind",
"MemoryVariant",
"AmbiguousMemoryIdError",
# Embedding 端点覆盖
"embed_endpoint_override",
# 高级 API
+280 -456
View File
@@ -38,6 +38,8 @@ from typing import List, Dict, Any, Optional, Tuple, Union, TYPE_CHECKING
from .config import MemoryConfig
from .core.embed_service import EmbedService, embed_endpoint_override
from .embedding import EmbeddingDimensionMismatchError, EmbeddingService
from .retrieval import AmbiguousMemoryIdError, MemoryKind, MemoryVariant
from .data.vector_store import create_vector_store
from .data.history_store import HistoryStore
from .data.cache import create_cache
@@ -434,6 +436,8 @@ class HyMemoryClient:
Args:
config: 自定义 MemoryConfig(默认从环境变量加载)
mode: 处理模式 ("lite" / "pro" / "ultra"),默认从环境变量 MEMORY_MODE 读取
embedding_service: 可选的调用方自管 embedding service;须满足公开的
EmbeddingService Protocol,client 不接管其生命周期。
"""
# 合法的 mode 值
@@ -445,12 +449,15 @@ class HyMemoryClient:
mode: Optional[str] = None,
coding_writer: Optional[str] = None,
runtime: Optional["SharedRuntime"] = None,
embedding_service: Optional[EmbeddingService] = None,
):
# 确保日志系统已配置(幂等,多次调用安全)
setup_logging()
# 配置
self._config = config or MemoryConfig.from_env()
self._borrowed_embedding_service = embedding_service
self._validate_embedding_service(embedding_service)
# coding_writer 显式传入则覆盖 config / env(默认 None → 沿用 config.coding.writer)
# 取值: "legacy" | "agent"
@@ -537,8 +544,12 @@ class HyMemoryClient:
)
logger.info(f"[init] Default ThreadPoolExecutor max_workers={_max_workers}")
# 1. EmbedService(纯配置,首次调用时真正连接 API)
self._embed_service = EmbedService(self._config)
# 1. EmbedService(注入发生在 ComponentFactory / pipeline 预热之前)
self._embed_service = (
self._borrowed_embedding_service
if self._borrowed_embedding_service is not None
else EmbedService(self._config)
)
logger.debug(
f"[init] EmbedService created: model={self._config.embedder.model}"
)
@@ -568,8 +579,10 @@ class HyMemoryClient:
)
# 4. ComponentFactory(注入已初始化的共享组件)
self._registry = ComponentFactory(config=self._config)
self._registry._shared_embed_service = self._embed_service
self._registry = ComponentFactory(
config=self._config,
embedding_service=self._embed_service,
)
self._registry._shared_vector_store = self._vector_store
self._registry._shared_vector_store_initialized = True
if self._graph_store is not None:
@@ -703,6 +716,7 @@ class HyMemoryClient:
cls,
config_dict: Dict[str, Any],
mode: Optional[str] = None,
embedding_service: Optional[EmbeddingService] = None,
) -> "HyMemoryClient":
"""
从字典创建客户端(类似 mem0 风格)。
@@ -734,7 +748,49 @@ class HyMemoryClient:
"""
config = MemoryConfig.from_dict(config_dict)
resolved_mode = mode or config_dict.get("mode")
return cls(config=config, mode=resolved_mode)
return cls(
config=config,
mode=resolved_mode,
embedding_service=embedding_service,
)
def _validate_embedding_service(
self, embedding_service: Optional[EmbeddingService]
) -> None:
"""Validate a borrowed service before any store or pipeline is created."""
if embedding_service is None:
return
required = (
"embed",
"embed_with_meta",
"embed_batch",
"embed_queued",
"embed_queued_with_meta",
"get_embedding_dims",
)
missing = [
name for name in required
if not callable(getattr(embedding_service, name, None))
]
if missing:
raise TypeError(
"embedding_service does not implement EmbeddingService; "
f"missing: {', '.join(missing)}"
)
try:
actual_dims = int(embedding_service.get_embedding_dims())
except Exception as exc:
raise TypeError(
"embedding_service.get_embedding_dims() must return an integer"
) from exc
expected_dims = getattr(self._config.vector_store, "embedding_dims", None)
if expected_dims is None:
expected_dims = getattr(self._config.embedder, "embedding_dims", None)
if expected_dims is not None and actual_dims != int(expected_dims):
raise EmbeddingDimensionMismatchError(
"embedding service dimension mismatch: "
f"service={actual_dims}, vector_store={int(expected_dims)}"
)
# ================================================================
# 属性
@@ -1060,6 +1116,7 @@ class HyMemoryClient:
llm_provider=self._coding_llm_provider,
embed_service=self._embed_service,
cache=self._cache,
variant=variant,
)
self._coding_stores[variant] = store
self._coding_writers[variant] = writer
@@ -1829,21 +1886,26 @@ class HyMemoryClient:
self,
memory_id: str,
*,
include_raw: bool = False,
kind: Optional[MemoryKind] = None,
variant: Optional[MemoryVariant] = None,
) -> Optional[Dict[str, Any]]:
"""
获取单条记忆。
Args:
memory_id: 记忆 ID
include_raw: 是否允许返回 L1_RAW 原始记忆。默认 False,保持原有隐藏行为;
lite 模式的记忆均为 L1_RAW,需显式设为 True 才能按 ID 读取。
kind: 可选 namespace("chat" / "experience" / "fact")。
variant: 可选生产力变体("working" / "coding")。
Returns:
{"memory_id", "content", "layer", "request_id", "elapsed_ms"} 或 None(不存在时)
带 kind/variant 的统一 memory dict,或 None(不存在时)。
"""
return self._loop_thread.run(
self.async_get(memory_id, include_raw=include_raw)
self.async_get(
memory_id,
kind=kind,
variant=variant,
)
)
def update(
@@ -1912,6 +1974,9 @@ class HyMemoryClient:
branch: Optional[str] = None,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
with_evolution: bool = False,
@@ -1934,6 +1999,9 @@ class HyMemoryClient:
user_ids: 可选,多用户批量列举。非空时优先于 user_id,返回并集
(分页在合并排序后统一施加)。
agent_ids: 可选,多 agent 批量列举。非空时优先于 agent_id。
session_ids: 可选,只返回指定 session。
include_sources: 可选,只返回指定来源。
exclude_sources: 可选,排除指定来源;与 include_sources 冲突时排除优先。
layers: 可选,只返回指定层级(字符串或 MemoryLayer)。不传时返回
L1_RAW 以外的全部层级,与改动前一致。
statuses: 可选,只返回指定生命周期(字符串或 MemoryStatus)。不传时
@@ -1949,6 +2017,9 @@ class HyMemoryClient:
self.async_list_memories(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
workspace_id=workspace_id, branch=branch,
layers=layers, statuses=statuses, with_evolution=with_evolution,
@@ -2737,6 +2808,8 @@ class HyMemoryClient:
request_id=request_id,
source=source,
)
# Public receipt always identifies the productivity namespace.
resp["variant"] = scene_variant
# 版本演进链层(scene="coding" 独有)。放在 writer.write 之后、返回
# 之前,因为它只依赖对话原文,与向量层写入结果无关。
# best-effort:这层挂不上不影响本次 add 的向量层产出。
@@ -2907,106 +2980,6 @@ class HyMemoryClient:
include_runtime_diag=True,
)
def import_raw(
self,
content: str,
*,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
raw_id: Optional[str] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Persist one exact raw record without extraction or reconciliation."""
return self._loop_thread.run(
self.async_import_raw(
content,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
memory_at=memory_at,
metadata=metadata,
raw_id=raw_id,
request_id=request_id,
),
include_runtime_diag=True,
)
def import_memory_exact(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
layer: Any = "l2_fact",
status: Any = "active",
owner: Optional[str] = None,
tags: Optional[List[str]] = None,
supersedes: Optional[List[str]] = None,
superseded_by: Optional[List[str]] = None,
is_latest: bool = True,
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Import one existing semantic memory without extraction/reconcile."""
return self._loop_thread.run(
self.async_import_memory_exact(
content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
status=status,
owner=owner,
tags=tags,
supersedes=supersedes,
superseded_by=superseded_by,
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
include_runtime_diag=True,
)
def get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
"""Read one SDK raw record through the explicit raw API."""
return self._loop_thread.run(self.async_get_raw(raw_id))
def list_raw(
self,
*,
user_id: str,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
source: Optional[str] = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Dict[str, Any]:
"""List raw records for an authorized SDK owner/scope."""
return self._loop_thread.run(
self.async_list_raw(
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
limit=limit,
offset=offset,
order=order,
)
)
@_ensure_internal_loop
async def async_add_extracted(
self,
@@ -3159,316 +3132,6 @@ class HyMemoryClient:
}
@_ensure_internal_loop
async def async_import_raw(
self,
content: str,
*,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
raw_id: Optional[str] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Persist exact source content as L1_RAW without running a model."""
from .utils.source_filter import normalize_source
if not isinstance(content, str) or not content:
raise ValueError("content is required")
if not user_id:
raise ValueError("user_id is required")
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
operation="import_raw",
)
if replay is not None:
return replay
try:
result = await self._async_import_raw_impl(
content,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalize_source(source),
memory_at=memory_at,
metadata=metadata,
raw_id=raw_id,
request_id=request_id,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
raise
await self._finish_write_outcome(request_id, "succeeded", result)
return result
async def _async_import_raw_impl(
self,
content: str,
*,
user_id: str,
agent_id: str,
session_id: str,
source: str,
memory_at: Optional[datetime],
metadata: Optional[Dict[str, Any]],
raw_id: Optional[str],
request_id: str,
) -> Dict[str, Any]:
from .models.memory import MemoryLayer, MemoryNode, MemoryStatus, SourceType
node_id = raw_id or str(uuid.uuid4())
existing = await self._vector_store.get_by_id(node_id)
if existing is not None:
identity = (
existing.layer == MemoryLayer.L1_RAW,
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == source,
)
if not all(identity):
raise ValueError("raw_id is already bound to different source content")
return {
"success": True,
"raw_id": node_id,
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata or {})
custom["request_id"] = request_id
custom["input_kind"] = "raw_exact_import"
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=node_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=MemoryLayer.L1_RAW,
content=content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != node_id:
raise RuntimeError("raw import changed the requested identity")
return {"success": True, "raw_id": node_id, "request_id": request_id}
@_ensure_internal_loop
async def async_import_memory_exact(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
layer: Any = "l2_fact",
status: Any = "active",
owner: Optional[str] = None,
tags: Optional[List[str]] = None,
supersedes: Optional[List[str]] = None,
superseded_by: Optional[List[str]] = None,
is_latest: bool = True,
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Import a canonical memory node exactly, with no model invocation."""
from .models.memory import MemoryLayer, MemoryStatus
from .utils.source_filter import normalize_source
if not content or not memory_id or not user_id:
raise ValueError("content, memory_id and user_id are required")
normalized_layer = (
layer if isinstance(layer, MemoryLayer) else MemoryLayer.from_string(str(layer))
)
if normalized_layer == MemoryLayer.L1_RAW:
raise ValueError("use import_raw for L1_RAW")
normalized_status = (
status if isinstance(status, MemoryStatus) else MemoryStatus(str(status).lower())
)
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
operation="import_memory_exact",
)
if replay is not None:
return replay
try:
result = await self._async_import_memory_exact_impl(
content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalize_source(source),
layer=normalized_layer,
status=normalized_status,
owner=owner,
tags=tags or [],
supersedes=supersedes or [],
superseded_by=superseded_by or [],
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
raise
await self._finish_write_outcome(request_id, "succeeded", result)
return result
async def _async_import_memory_exact_impl(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
layer,
status,
owner: Optional[str],
tags: List[str],
supersedes: List[str],
superseded_by: List[str],
is_latest: bool,
memory_at: Optional[datetime],
metadata: Optional[Dict[str, Any]],
request_id: str,
) -> Dict[str, Any]:
from .models.memory import MemoryNode, SourceType
existing = await self._vector_store.get_by_id(memory_id)
if existing is not None:
identity = (
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == source,
existing.layer == layer,
existing.status == status,
existing.owner == owner,
list(existing.tags or []) == tags,
list(existing.supersedes or []) == supersedes,
list(existing.superseded_by or []) == superseded_by,
existing.is_latest == is_latest,
)
if not all(identity):
raise ValueError("memory_id is already bound to different memory content")
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata or {})
custom["request_id"] = request_id
custom["input_kind"] = "memory_exact_import"
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
content=content,
source_type=SourceType.EXPLICIT,
status=status,
owner=owner,
tags=list(tags),
supersedes=list(supersedes),
superseded_by=list(superseded_by),
is_latest=is_latest,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != memory_id:
raise RuntimeError("exact memory import changed the requested identity")
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
}
@_ensure_internal_loop
async def async_get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
from .models.memory import MemoryLayer
node = await self._vector_store.get_by_id(raw_id)
if node is None or node.layer != MemoryLayer.L1_RAW:
return None
return self._raw_node_to_public(node)
@_ensure_internal_loop
async def async_list_raw(
self,
*,
user_id: str,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
source: Optional[str] = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Dict[str, Any]:
from .models.memory import MemoryLayer, MemoryStatus
if not user_id:
raise ValueError("user_id is required")
if limit < 1 or offset < 0:
raise ValueError("limit must be positive and offset must be non-negative")
nodes = await self._vector_store.list_by_user(
user_id=user_id,
agent_id=agent_id,
status_filter=[MemoryStatus.ACTIVE],
layers=[MemoryLayer.L1_RAW],
include_sources=[source] if source else None,
limit=100_000,
only_latest=False,
with_vectors=False,
allow_raw_layer=True,
)
if session_id:
nodes = [node for node in nodes if node.session_id == session_id]
nodes = self._sort_memory_nodes(nodes, order=order)
total = len(nodes)
return {
"raw": [
self._raw_node_to_public(node)
for node in nodes[offset: offset + limit]
],
"total": total,
"limit": limit,
"offset": offset,
}
async def _search_explicit_raw_layer(
self,
query: str,
@@ -4374,35 +4037,100 @@ class HyMemoryClient:
# CRUD API(异步)— get / update / delete / delete_all
# ================================================================
async def _coding_store_for_exact_get(self, variant: str):
"""Return an existing namespace store without creating an empty DB."""
stores = getattr(self, "_coding_stores", None) or {}
store = stores.get(variant)
if store is not None:
return store
from .coding.store import CodingMemoryStore
config = getattr(self, "_config", None)
coding_config = getattr(config, "coding", None)
if config is None or coding_config is None:
return None
if not CodingMemoryStore.database_exists(
config,
db_path=coding_config.db_path,
variant=variant,
):
return None
await self._get_coding_writer(variant=variant)
return (getattr(self, "_coding_stores", None) or {}).get(variant)
@staticmethod
def _format_productivity_get(
item: Any,
*,
kind: str,
variant: str,
request_id: str,
elapsed_ms: float,
) -> Dict[str, Any]:
payload = item.to_dict()
item_id = (
payload.get("experience_id")
if kind == "experience"
else payload.get("fact_id")
)
return {
"memory_id": item_id,
"kind": kind,
"variant": variant,
"scene": "productivity",
"content": (
payload.get("solution", "")
if kind == "experience"
else payload.get("statement", "")
),
"layer": kind,
**payload,
"request_id": request_id,
"elapsed_ms": elapsed_ms,
}
@_ensure_internal_loop
async def async_get(
self,
memory_id: str,
*,
include_raw: bool = False,
kind: Optional[MemoryKind] = None,
variant: Optional[MemoryVariant] = None,
) -> Optional[Dict[str, Any]]:
"""
获取单条记忆(异步)。
VDB + Graph 双路并行查询,哪个有结果返回哪个。
默认查询 chat、working/coding experience 和 working/coding fact。
若同一 ID 命中多个 namespace,则 fail closed;调用方必须用 kind/variant
显式消歧。
Args:
memory_id: 记忆 ID
include_raw: 是否允许返回 L1_RAW 原始记忆,默认 False。
kind: 可选 namespace("chat" / "experience" / "fact")。
variant: 可选生产力变体("working" / "coding")。
Returns:
{"memory_id", "content", "layer", "source", "request_id", "elapsed_ms"} 或 None
统一 memory dict,或 None。
"""
import asyncio as _asyncio_get
from .models.memory import MemoryLayer
if kind not in (None, "chat", "experience", "fact"):
raise ValueError(
"kind must be one of: chat, experience, fact"
)
if variant not in (None, "working", "coding"):
raise ValueError("variant must be one of: working, coding")
if kind == "chat" and variant is not None:
raise ValueError("variant is only valid for experience/fact lookups")
request_id = str(uuid.uuid4())
set_request_id(request_id)
t0 = time.perf_counter()
logger.info(f"[get] memory_id={memory_id}")
logger.info(
"[get] memory_id=%s kind=%s variant=%s",
memory_id, kind, variant,
)
# 双路并行
async def _get_vdb():
return await self._vector_store.get_by_id(memory_id)
@@ -4415,39 +4143,63 @@ class HyMemoryClient:
logger.debug(f"[get] graph_store query failed: {e}")
return None
vdb_node, graph_node = await _asyncio_get.gather(_get_vdb(), _get_graph())
query_chat = kind in (None, "chat") and variant is None
coding_kinds = (
("experience", "fact") if kind is None else (kind,)
)
coding_variants = (variant,) if variant is not None else ("working", "coding")
# L1_RAW 默认不对外可见:按 ID 直取走主键查询、绕过向量库的 layer
# filter,故在此统一挡掉。SDK 调用方可通过 include_raw=True 显式读取;
# HTTP 接口不传该参数,继续保持隐藏。内部写链路不经过此方法。
#
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
if (
not include_raw
and vdb_node is not None
and vdb_node.layer == MemoryLayer.L1_RAW
):
vdb_node = None
if (
not include_raw
and graph_node is not None
and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW
):
graph_node = None
stores = {}
if kind != "chat":
for candidate_variant in coding_variants:
store = await self._coding_store_for_exact_get(candidate_variant)
if store is not None:
stores[candidate_variant] = store
tasks = []
task_names = []
if query_chat:
tasks.extend((_get_vdb(), _get_graph()))
task_names.extend(("chat:vdb", "chat:graph"))
for candidate_variant, store in stores.items():
if "experience" in coding_kinds:
tasks.append(store.get_by_id(memory_id))
task_names.append(f"{candidate_variant}:experience")
if "fact" in coding_kinds:
tasks.append(store.get_fact(memory_id))
task_names.append(f"{candidate_variant}:fact")
values = await asyncio.gather(*tasks) if tasks else []
by_name = dict(zip(task_names, values))
vdb_node = by_name.get("chat:vdb")
graph_node = by_name.get("chat:graph")
matches = []
chat_node = vdb_node if vdb_node is not None else graph_node
if chat_node is not None:
matches.append(("chat", chat_node))
for name, value in by_name.items():
if not name.startswith("chat:") and value is not None:
matches.append((name, value))
if len(matches) > 1:
raise AmbiguousMemoryIdError(
memory_id, [namespace for namespace, _ in matches]
)
elapsed = round((time.perf_counter() - t0) * 1000, 2)
# pipeline trace(best-effort,受 MEMORY_PIPELINE_TRACE_ENABLED 控制)
_hit_node = vdb_node if vdb_node is not None else graph_node
_source = "vdb" if vdb_node is not None else ("graph" if graph_node is not None else "")
_hit_namespace, _hit_node = matches[0] if matches else ("", None)
_source = (
"vdb" if vdb_node is not None
else ("graph" if graph_node is not None else _hit_namespace)
)
_ident = self._backend_identity()
_parsed = {
"memory_id": memory_id,
**_ident,
"hit": _hit_node is not None,
"source": _source,
"namespace": _hit_namespace,
}
if _hit_node is not None:
try:
@@ -4464,10 +4216,25 @@ class HyMemoryClient:
elapsed_ms=elapsed,
)
# VDB 优先(如果两边都有,以 VDB 为准)
if not matches:
logger.info(f"[get] not found: memory_id={memory_id}")
return None
if _hit_namespace != "chat":
hit_variant, hit_kind = _hit_namespace.split(":", 1)
return self._format_productivity_get(
_hit_node,
kind=hit_kind,
variant=hit_variant,
request_id=request_id,
elapsed_ms=elapsed,
)
if vdb_node is not None:
return {
"memory_id": vdb_node.node_id,
"kind": "chat",
"variant": None,
"content": vdb_node.content,
"layer": vdb_node.layer.value,
"status": vdb_node.status.value if vdb_node.status else "active",
@@ -4481,10 +4248,13 @@ class HyMemoryClient:
# 不透出 source_raw_memory_id:见 async_search 同处说明。
"user_id": vdb_node.user_id,
"agent_id": vdb_node.agent_id,
"session_id": vdb_node.session_id,
"owner": vdb_node.owner,
"confidence": vdb_node.confidence,
"custom": vdb_node.custom or {},
"meta": list((vdb_node.custom or {}).get("meta") or []),
"source": "vdb",
"source": vdb_node.source,
"retrieval_source": "vdb",
"request_id": request_id,
"elapsed_ms": elapsed,
}
@@ -4492,6 +4262,8 @@ class HyMemoryClient:
if graph_node is not None:
return {
"memory_id": graph_node.node_id,
"kind": "chat",
"variant": None,
"content": graph_node.content,
"layer": graph_node.layer.value if graph_node.layer else "",
"status": graph_node.status.value if graph_node.status else "active",
@@ -4505,14 +4277,16 @@ class HyMemoryClient:
# 不透出 source_raw_memory_id:见 async_search 同处说明。
"user_id": graph_node.user_id,
"agent_id": graph_node.agent_id,
"session_id": getattr(graph_node, "session_id", "") or "",
"owner": getattr(graph_node, "owner", None),
"confidence": graph_node.confidence,
"custom": getattr(graph_node, "custom", {}) or {},
"source": "graph",
"source": getattr(graph_node, "source", "") or "",
"retrieval_source": "graph",
"request_id": request_id,
"elapsed_ms": elapsed,
}
logger.info(f"[get] not found: memory_id={memory_id}")
return None
@_ensure_internal_loop
@@ -5081,6 +4855,7 @@ class HyMemoryClient:
"user_id": node.user_id,
"agent_id": node.agent_id,
"session_id": getattr(node, "session_id", None),
"source": getattr(node, "source", None),
"speculate": getattr(node, "speculate", None),
# 不透出 source_raw_memory_id:见 async_search 同处说明。
"tags": node.tags or [],
@@ -5152,6 +4927,9 @@ class HyMemoryClient:
order: str,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List] = None,
statuses: Optional[List] = None,
with_evolution: bool = False,
@@ -5177,6 +4955,9 @@ class HyMemoryClient:
status_filter=effective_statuses,
layers=layers,
with_vectors=False,
allow_raw_layer=bool(
layers is not None and MemoryLayer.L1_RAW in layers
),
)
# L0 是 user-level,不应被 agent 过滤。默认列表或显式包含 L0 时,额外按
@@ -5198,6 +4979,22 @@ class HyMemoryClient:
])
all_nodes.extend(hit["node"] for hit in l0_hits if hit.get("node") is not None)
if session_ids:
wanted_sessions = set(session_ids)
all_nodes = [
node for node in all_nodes if node.session_id in wanted_sessions
]
if include_sources:
wanted_sources = set(include_sources)
all_nodes = [
node for node in all_nodes if node.source in wanted_sources
]
if exclude_sources:
blocked_sources = set(exclude_sources)
all_nodes = [
node for node in all_nodes if node.source not in blocked_sources
]
all_nodes = self._sort_memory_nodes(all_nodes, order=order)
total = len(all_nodes)
page_nodes = all_nodes[offset: offset + limit]
@@ -5261,6 +5058,9 @@ class HyMemoryClient:
order: str,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List] = None,
statuses: Optional[List] = None,
) -> Optional[Dict[str, Any]]:
@@ -5308,6 +5108,21 @@ class HyMemoryClient:
node for node in graph_nodes
if getattr(node, "status", None) in effective_statuses
]
if session_ids:
wanted_sessions = set(session_ids)
graph_nodes = [
node for node in graph_nodes if node.session_id in wanted_sessions
]
if include_sources:
wanted_sources = set(include_sources)
graph_nodes = [
node for node in graph_nodes if node.source in wanted_sources
]
if exclude_sources:
blocked_sources = set(exclude_sources)
graph_nodes = [
node for node in graph_nodes if node.source not in blocked_sources
]
graph_nodes = self._sort_memory_nodes(graph_nodes, order=order)
total = len(graph_nodes)
@@ -5419,6 +5234,9 @@ class HyMemoryClient:
branch: Optional[str] = None,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
with_evolution: bool = False,
@@ -5495,6 +5313,9 @@ class HyMemoryClient:
vdb = await self._list_vdb_bucket(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
layers=effective_layers, statuses=effective_statuses,
with_evolution=with_evolution,
@@ -5502,6 +5323,9 @@ class HyMemoryClient:
graph = await self._list_graph_bucket(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
layers=effective_layers, statuses=effective_statuses,
)
+30
View File
@@ -416,6 +416,36 @@ class CodingMemoryStore:
self._initialized = False
@staticmethod
def database_path(
config: MemoryConfig,
db_path: Optional[str] = None,
variant: str = "working",
) -> str:
"""Return the authoritative SQLite path without creating anything."""
if db_path is None:
data_dir = (
os.path.dirname(config.history.db_path)
if config.history.db_path else "./data"
)
db_path = os.path.join(data_dir, "coding_memory.db")
normalized = (
"coding"
if str(variant or "").strip().lower() == "coding"
else "working"
)
return _variant_db_path(db_path, "_cv" if normalized == "coding" else "")
@classmethod
def database_exists(
cls,
config: MemoryConfig,
db_path: Optional[str] = None,
variant: str = "working",
) -> bool:
"""Whether a coding namespace has an existing SQLite database."""
return os.path.isfile(cls.database_path(config, db_path, variant))
async def initialize(self) -> None:
if self._initialized:
return
+9 -2
View File
@@ -50,6 +50,7 @@ class CodingWriter:
embed_service: EmbedService,
fact_reconciler: Optional[WorkingFactReconciler] = None,
cache: Any = None,
variant: str = "working",
):
self.store = store
self.extractor = extractor
@@ -59,6 +60,11 @@ class CodingWriter:
self.fact_reconciler = fact_reconciler or WorkingFactReconciler(
store, embed_service, llm_provider
)
self.variant = (
"coding"
if str(variant or "").strip().lower() == "coding"
else "working"
)
# cache 仅用于把 request_id 串给各阶段(各阶段自己打点);writer 不集中打点。
self.cache = cache
@@ -217,6 +223,7 @@ class CodingWriter:
return {
"success": True,
"scene": "productivity",
"variant": self.variant,
"request_id": request_id,
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
"ops": [op.to_dict() for op in ops],
@@ -278,11 +285,11 @@ class CodingWriter:
f"[coding-write] crosslink fact {f.fact_id} failed: {e}"
)
@staticmethod
def _empty_response(request_id: str, t0: float) -> Dict[str, Any]:
def _empty_response(self, request_id: str, t0: float) -> Dict[str, Any]:
return {
"success": True,
"scene": "productivity",
"variant": self.variant,
"request_id": request_id,
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
"ops": [],
+53
View File
@@ -0,0 +1,53 @@
"""Public structural contract for injected embedding services."""
from typing import Any, Dict, List, Protocol, Tuple, runtime_checkable
@runtime_checkable
class EmbeddingService(Protocol):
"""Embedding service accepted by :class:`HyMemoryClient`.
Implementations are borrowed by the client: their lifecycle remains owned
by the caller and ``HyMemoryClient.close()`` will not close them.
"""
async def embed(self, text: str, use_cache: bool = True) -> List[float]:
...
async def embed_with_meta(
self, text: str, use_cache: bool = True
) -> Tuple[List[float], Dict[str, Any]]:
...
async def embed_batch(
self, texts: List[str], use_cache: bool = True
) -> List[List[float]]:
...
async def embed_queued(
self, text: str, use_cache: bool = True
) -> List[float]:
...
async def embed_queued_with_meta(
self, text: str, use_cache: bool = True
) -> Tuple[List[float], Dict[str, Any]]:
...
def get_embedding_dims(self) -> int:
...
# Explicit alias for callers that prefer the conventional Protocol suffix.
EmbeddingServiceProtocol = EmbeddingService
class EmbeddingDimensionMismatchError(ValueError):
"""Raised before initialization when service and store dimensions differ."""
__all__ = [
"EmbeddingService",
"EmbeddingServiceProtocol",
"EmbeddingDimensionMismatchError",
]
+9 -3
View File
@@ -19,6 +19,7 @@ import logging
from .base import WritePipeline, ReadPipeline
from ..config import MemoryConfig
from ..core.embed_service import EmbedService
from ..embedding import EmbeddingService
from ..data.vector_store import create_vector_store
from ..data.vector_store_base import VectorStoreBase
@@ -50,7 +51,12 @@ class ComponentFactory:
s2 = await factory.get_system2_writer() # System2Writer (ultra mode)
"""
def __init__(self, config: Optional[MemoryConfig] = None):
def __init__(
self,
config: Optional[MemoryConfig] = None,
*,
embedding_service: Optional[EmbeddingService] = None,
):
self.config = config or MemoryConfig.from_env()
# 实例缓存(单例)
@@ -62,7 +68,7 @@ class ComponentFactory:
self._readers: Dict[str, ReadPipeline] = {}
# 共享资源
self._shared_embed_service: Optional[EmbedService] = None
self._shared_embed_service: Optional[EmbeddingService] = embedding_service
self._shared_vector_store: Optional[VectorStoreBase] = None
self._shared_vector_store_initialized: bool = False
self._shared_graph_store = None
@@ -72,7 +78,7 @@ class ComponentFactory:
# 共享资源
# ================================================================
def _get_shared_embed_service(self) -> EmbedService:
def _get_shared_embed_service(self) -> EmbeddingService:
if self._shared_embed_service is None:
self._shared_embed_service = EmbedService(self.config)
return self._shared_embed_service
+23
View File
@@ -0,0 +1,23 @@
"""Public types and errors for exact memory-ID lookup."""
from typing import Literal, Sequence
MemoryKind = Literal["chat", "experience", "fact"]
MemoryVariant = Literal["working", "coding"]
class AmbiguousMemoryIdError(LookupError):
"""An ID exists in more than one public memory namespace."""
def __init__(self, memory_id: str, namespaces: Sequence[str]):
self.memory_id = memory_id
self.namespaces = tuple(namespaces)
joined = ", ".join(self.namespaces)
super().__init__(
f"memory ID {memory_id!r} is ambiguous across namespaces: {joined}; "
"pass kind and/or variant to disambiguate"
)
__all__ = ["MemoryKind", "MemoryVariant", "AmbiguousMemoryIdError"]
+68 -6
View File
@@ -8,6 +8,7 @@ Uses a two-tier strategy:
This avoids langdetect's known weakness with short Chinese text.
"""
import json
import logging
import re
from typing import List, Optional
@@ -135,6 +136,30 @@ def detect_language(text: str) -> str:
return "zh" if ratio > 0.1 else "en"
def _spoken_text(value: str) -> str:
"""Ignore text-block metadata when a message content is structured JSON."""
text = value.strip()
if text.startswith("["):
try:
blocks = json.loads(text)
except (ValueError, TypeError):
pass
else:
if (
isinstance(blocks, list)
and blocks
and all(
isinstance(block, dict)
and isinstance(block.get("type"), str)
and block["type"] in {"text", "input_text", "output_text"}
and isinstance(block.get("text"), str)
for block in blocks
)
):
return "\n".join(block["text"] for block in blocks)
return text
def extract_content_for_detection(
content: str = "",
messages: Optional[List] = None,
@@ -153,20 +178,57 @@ def extract_content_for_detection(
Concatenated content text suitable for language detection.
"""
if content and content.strip():
return content.strip()
value = content.strip()
# Writer serializes message batches to JSON before calling Extractor.
# Detect the spoken text, not the leading ASCII keys (role/content),
# without changing the raw evidence sent to the model.
if value.startswith("["):
try:
decoded = json.loads(value)
except (ValueError, TypeError):
pass
else:
if (
isinstance(decoded, list)
and decoded
and all(
isinstance(item, dict)
and isinstance(item.get("role"), str)
and isinstance(item.get("content"), str)
for item in decoded
)
):
# In a mixed batch, the latest user turn is the clearest
# instruction for the language; assistant/tool metadata
# must not decide it first.
user_text = [
_spoken_text(item["content"])
for item in decoded
if item["role"] == "user" and item["content"].strip()
]
if user_text:
return user_text[-1]
spoken = "\n".join(_spoken_text(item["content"]) for item in decoded)
if spoken.strip():
return spoken
return _spoken_text(value)
if messages:
parts = []
user_parts = []
for msg in messages:
if isinstance(msg, dict):
c = msg.get("content", "")
c, role = msg.get("content", ""), msg.get("role")
elif hasattr(msg, "content"):
c = msg.content
c, role = msg.content, getattr(msg, "role", None)
else:
continue
if c and c.strip():
parts.append(c.strip())
return "\n".join(parts)
if isinstance(c, str) and c.strip():
text = _spoken_text(c)
parts.append(text)
if role == "user":
user_parts.append(text)
return user_parts[-1] if user_parts else "\n".join(parts)
return ""
+13 -41
View File
@@ -9,10 +9,13 @@
## Architecture Principles
1. Memory 读写与存储 backend **只经 SDK**(config / env)。
2. MemHub 对 memory 的原子操作 **只经 SDK**;缺接口先补 SDK。
2. MemHub 运行时对 memory 的原子操作 **只经 SDK 公共 Client**;缺正常业务
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
但不得把迁移专用方法加入公共 Client。
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
生产入口仍建议经 `HyMemoryBridge`(共享 Runtime、模型配置、密钥、harness/model 门禁),但 bridge **不得**再把 SDK 结果镜像成第二套 memory 正文权威。
生产入口通过共享 Runtime 租约取得公开 `HyMemoryClient`;MemHub 只在调用前执行
模型配置、密钥和 harness/model 门禁,不维护第二套 memory 正文权威。
来源授权、模型授权和宿主召回授权仍是 MemHub 管理面门禁;memory
数据平面的权威在 SDK。
@@ -30,46 +33,15 @@ reader: hybrid
## MemHub 运行时边界
MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtime_config` 注入):
MemHub 通过公开 `HyMemoryClient` 配置使用可复现的本地策略:
- vector / graph / cache / history / coding 等 **一律 SDK config/env**;
- 模型与 embedding 来自版本化 Model Profile;密钥进安全存储;
- MemHub 管理库可存 harness/model/队列状态与 **`memory_id` 引用**。
发布包默认 SDK 行为环境变量仍集中在
`apps/core/src/one_memory/hy_runtime_config.py`,并由下表锁定:
| 变量 | 值 |
|---|---|
| `MEMORY_MODE` | `ultra` |
| `MEMORY_ENABLE_GRAPH` | `true` |
| `MEMORY_GRAPH_RERANK_ENABLED` | `false` |
| `MEMORY_GRAPH_TEMPORAL_SPINE` | `true` |
| `MEMORY_TOPIC_SUMMARY_ENABLED` | `false` |
| `MEMORY_EPISODE_SUMMARY_ENABLED` | `false` |
| `MEMORY_EPISODE_RECALL_ENABLED` | `true` |
| `MEMORY_DYNAMIC_EPISODE_ENABLED` | `true` |
| `MEMORY_SESSION_EPISODE_ENABLED` | `false` |
| `MEMORY_SESSION_EPISODE_CONTEXT_ENABLED` | `false` |
| `MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED` | `true` |
| `MEMORY_EPISODE_CAPSULE_QUERY_MIN` | `0.65` |
| `MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN` | `0.60` |
| `MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX` | `3` |
| `MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT` | `3` |
| `MEMORY_ENABLE_INTENTION` | `false` |
| `MEMORY_ENABLE_QUERY_REWRITE` | `false` |
| `MEMORY_FEW_SHOT_ENABLED` | `true` |
| `MEMORY_HYBRID_V2_QUERY_EXPAND` | `1` |
| `MEMORY_HYBRID_V2_GEO_INTENT_PRIOR` | `0.6` |
| `MEMORY_HYBRID_V2_DIVERSITY` | `1` |
| `MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR` | `0.55` |
| `MEMORY_HYBRID_V2_TEMPORAL_BOOST` | `0.18` |
| `MEMORY_CROSS_DOMAIN_CON_THRESHOLD` | `0.5` |
| `MEMORY_READER_ENABLE_SUMMARY` | `false` |
| `MEMORY_SUMMARY_ENABLED_IN_SYS2` | `false` |
| `MEMORY_LOCAL_RERANK_ENABLED` | `false` |
| `RECONCILE_ENABLED` | `true` |
| `HY_MEMORY_READER` | `hybrid` |
MemHub 不再通过 import-time 环境变量覆盖 SDK 的提取、reconcile、episode、
intention、few-shot、summary 或去重策略。产品只配置模型/密钥/endpoint、本地
backend 路径、公开 embedding service,以及每次公开 Client 调用的参数。
独立脚本直连 SDK 时须自配存储目录与模型端点。
@@ -77,7 +49,7 @@ MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtim
下面是独立调试示例。
产品代码经 `HyMemoryBridge` 获取已配置的 client(管理面门禁),memory I/O
产品代码经 `hy_sdk_runtime.SdkClientHandle` 获取已配置的公开 client;memory I/O
由 SDK 完成。
```python
@@ -228,7 +200,8 @@ result = client.search(
- 区分空结果、权限拒绝、不可用、超时、部分结果;
- 未发布的 Matter 占位功能不保留表或 API;未来组织功能另立合同。
生产入口:`hy_memory_bridge.py`、`recall.py`、`packages/host-bridge`。
生产入口:`hy_sdk_runtime.py`、`sdk_write_processor.py`、`recall.py` 和
`packages/host-bridge`。
## 关闭客户端
@@ -245,8 +218,7 @@ Runtime 长期复用 client;仅在停止、配置切换或受控恢复时重
```text
harness / user action (+ authz, model, queue)
-> HyMemoryBridge
-> Hy Memory SDK
-> public HyMemoryClient
-> SDK stores (config/env)
-> optional MemHub persist memory_id / org metadata
-> Recall Broker / UI (SDK read + MemHub authz)
@@ -42,12 +42,10 @@ TARGETED = (
"tests/test_database_contention.py",
"tests/test_main_database_fairness.py",
"tests/test_soak_probe_effectiveness.py",
"tests/test_evidence_reconcile_primitive.py",
"tests/test_sqlite_semantics_gate.py",
"tests/test_store_binding.py",
"tests/test_store_binding_ratchet.py",
"tests/test_main_database_entry.py",
"tests/test_memory_processing.py",
"tests/test_extraction_window.py",
"tests/test_extraction_defer.py",
"tests/test_active_memory_window.py",
@@ -2,11 +2,5 @@
import os
from one_memory.hy_runtime_config import configure_hy_behavior_environment
# HY has a few reader switches evaluated at module import time. Fix the
# product-owned, non-secret behavior before any one_memory module imports HY.
configure_hy_behavior_environment()
_version = os.environ.get("ONE_MEMORY_VERSION", "").strip()
__version__ = _version if _version else "development"
+246 -197
View File
@@ -50,6 +50,10 @@ from one_memory.auth import local_api_guard, renderer_origins
from one_memory.background_work import shutdown_bulk_pool, warm_bulk_pool
from one_memory.config import Settings, load_settings
from one_memory.context_routes import context_router, sdk_context_router
from one_memory.conversation_processing import (
_conversation_chunks,
memory_task_timeout_seconds,
)
from one_memory.database_access import (
MAIN_DATABASE_REPORT_SECONDS,
close_main_databases,
@@ -78,23 +82,30 @@ from one_memory.external_memory_access import external_access, require_external_
from one_memory.extraction_settings import ExtractionSettings, ExtractionSettingsStore
from one_memory.file_imports import FileImportStore, file_import_router
from one_memory.home_summary import HomeSummary, read_home_summary
from one_memory.hy_memory_bridge import (
GraphWidthDecision,
GraphWidthMismatch,
HyMemoryBridge,
HyMemoryBridgeError,
HySdkMemoryPage,
HySdkMemoryRecord,
HySdkRawRecord,
builtin_retrieval_snapshot,
)
from one_memory.hy_graph_admin import GraphWidthDecision, GraphWidthMismatch
from one_memory.hy_runtime_config import (
HY_MEMORY_READER,
DiagnosticLogLevel,
detailed_logging_enabled,
diagnostic_log_level,
set_detailed_logging,
set_diagnostic_log_level,
)
from one_memory.hy_sdk_runtime import (
SdkClientHandle,
builtin_retrieval_snapshot,
close_all_shared_runtimes,
sdk_relevance_floor,
)
from one_memory.hy_sdk_types import (
HySdkError,
HySdkMemoryPage,
HySdkMemoryRecord,
HySdkRawPage,
HySdkRawRecord,
HySdkUserProfile,
parse_sdk_search,
)
from one_memory.imports.access import (
AuthorizedSourceRoot,
SourceAccessError,
@@ -167,11 +178,10 @@ from one_memory.memory_library import (
read_conversation_memories,
read_memory_library,
)
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
from one_memory.memory_processing import (
MemoryProcessingSupervisor,
_conversation_chunks,
memory_task_timeout_seconds,
from one_memory.memory_notes import (
LegacyMemoryNoteReader,
MemoryNoteStore,
memory_note_router,
)
from one_memory.memory_qa import MemoryAnswerer, MemoryAnswerError
from one_memory.memory_queue import (
@@ -201,7 +211,10 @@ from one_memory.model_profiles import (
ModelProfileStore,
ModelProfileUnavailableError,
)
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
from one_memory.model_request_audit import (
AuditedModelTransport,
ModelRequestAuditStore,
)
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
from one_memory.onboarding import OnboardingState, OnboardingStore
from one_memory.recall import (
@@ -225,6 +238,8 @@ from one_memory.sdk_memory_qa import (
SdkMemoryAnswer,
SdkMemoryAnswerer,
)
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.secrets import (
KeyringSecretStore,
SecretStore,
@@ -845,7 +860,7 @@ def create_app(
runtime = CoreRuntime(runtime_settings) if manage_runtime else nullcontext()
with runtime:
database_path = runtime_settings.data_dir / "one-memory.sqlite3"
app.state.sdk_library_bridge = None
app.state.sdk_client_handle = None
bootstrap_fresh_management_schema(database_path)
# Before any store: each one below changes the library as it opens.
try:
@@ -891,6 +906,7 @@ def create_app(
)
from one_memory.memory_server.data_safety import MIN_FREE_BYTES
from one_memory.memory_server.durable_queue import (
COMPLETED,
DurableWriteDrain,
DurableWriteQueue,
execute_job,
@@ -968,29 +984,21 @@ def create_app(
await asyncio.to_thread(app.state.evidence_store.release_completed_sdk_staging)
evidence_store = app.state.evidence_store
if app.state.memory_authority is MemoryAuthority.SDK:
sdk_bridge = _LazySdkBridge(app)
async def read_sdk_memory(
memory_id: str,
) -> HySdkMemoryRecord | None:
return cast(
HySdkMemoryRecord | None,
await sdk_bridge.get_sdk_memory(memory_id),
)
sdk_handle = _sdk_client_handle(app)
app.state.recall_target = SdkRecallTarget(
semantic_search=lambda query: sdk_bridge.search_memories(
semantic_search=lambda query: _search_sdk_memories(
sdk_handle,
query,
user_id="local",
agent_id="*",
),
get_memory=read_sdk_memory,
get_memory=lambda memory_id: _get_sdk_memory(sdk_handle, memory_id),
)
else:
app.state.recall_target = GovernedRecallTarget(
app.state.memory_governance_store,
filter_memories=app.state.source_read_access.memories,
semantic_search=lambda query: _memory_processing_supervisor(
semantic_search=lambda query: _sdk_memory_runtime(
app
).semantic_recall(query),
# Read per request rather than cached: an import that lands
@@ -1094,7 +1102,7 @@ def create_app(
authorize_generation=lambda profile, question, grounding: (
_authorize_answer_generation(app, profile, question, grounding)
),
semantic_search=lambda query: _memory_processing_supervisor(app).semantic_recall(
semantic_search=lambda query: _sdk_memory_runtime(app).semantic_recall(
query
),
resolve_workspace=lambda workspace_path: resolve_workspace_scopes(
@@ -1116,43 +1124,32 @@ def create_app(
store=app.state.model_profile_store,
secrets=app.state.secret_store,
)
app.state.memory_processing_supervisor = MemoryProcessingSupervisor(
tasks=app.state.memory_task_store,
app.state.sdk_write_coordinator = SdkWriteCoordinator(
profiles=app.state.model_profile_store,
secrets=app.state.secret_store,
evidence=app.state.evidence_store,
governance=app.state.memory_governance_store,
data_dir=runtime_settings.data_dir,
extraction_settings=app.state.extraction_settings_store,
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
profile=profile,
api_key=api_key,
data_dir=data_dir,
http_transport_factory=lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
transport_factory=lambda profile: lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
),
# `None` means "whatever the active profile allows". Pinned to
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
# package, and left pinned after it: a hardcoded 1 is rule 9's
# exact anti-pattern (a small number baked into code that
# cannot be tuned per machine). Measured before removing — at
# profile concurrency 4, recall latency is unchanged while
# precipitation goes 4x.
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
service_levels=app.state.service_levels,
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
sdk_authority=lambda: app.state.memory_authority is MemoryAuthority.SDK,
sedimentation_enabled=lambda: _config_store(
app
).sedimentation_enabled(),
)
app.state.sdk_memory_runtime = SdkMemoryRuntime(
coordinator=app.state.sdk_write_coordinator,
read_handle=_sdk_client_handle(app),
write_queue=memory_write_queue,
extraction_settings=app.state.extraction_settings_store,
data_dir=runtime_settings.data_dir,
)
# C2 owns the single production budget (handle 5s + reserve 250ms,
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
@@ -1162,34 +1159,26 @@ def create_app(
drain_budget = production_drain_budget()
async def reconcile_unknown_writes() -> None:
if app.state.memory_authority is MemoryAuthority.SDK:
# Unknown SDK writes must be reconciled by SDK request ID.
# Until that direct path is available, retain them as
# unknown rather than re-running a possibly successful add.
return
await app.state.memory_processing_supervisor.reconcile_unknown_outputs(
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
)
unknown = await asyncio.to_thread(
memory_write_queue.unknown_jobs,
limit=128,
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
)
for job in unknown:
receipt = (
await app.state.sdk_write_coordinator.inquire_write_outcome(
job.request_id
)
)
if receipt is None:
continue
await asyncio.to_thread(
memory_write_queue.resolve_unknown_from_linked_output,
memory_write_queue.settle,
job.job_id,
state=COMPLETED,
sdk_raw_ids=receipt.raw_ids,
sdk_memory_ids=receipt.memory_ids,
)
await asyncio.to_thread(
memory_write_queue.retry_first_unknown_after_reconcile,
older_than=datetime.now(UTC) - timedelta(seconds=10),
limit=32,
)
await asyncio.to_thread(
app.state.memory_task_store.retry_first_unknown_after_reconcile,
older_than=datetime.now(UTC) - timedelta(seconds=10),
limit=128,
)
async def admit_hy_invocation(
source_id: str,
@@ -1211,7 +1200,7 @@ def create_app(
execute=lambda job_id: execute_job(
job_id,
queue=memory_write_queue,
supervisor=app.state.memory_processing_supervisor,
supervisor=app.state.sdk_write_coordinator,
evidence=app.state.evidence_store,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
admit_hy_invocation=admit_hy_invocation,
@@ -1219,7 +1208,7 @@ def create_app(
execute_owned=lambda job_id, owner: execute_job(
job_id,
queue=memory_write_queue,
supervisor=app.state.memory_processing_supervisor,
supervisor=app.state.sdk_write_coordinator,
evidence=app.state.evidence_store,
owner=owner,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
@@ -1293,7 +1282,14 @@ def create_app(
app.state.memory_note_store = MemoryNoteStore(
app.state.evidence_store,
app.state.memory_processing_supervisor,
(
LegacyMemoryNoteReader(
app.state.memory_task_store,
app.state.memory_governance_store,
)
if app.state.memory_authority is MemoryAuthority.LEGACY
else None
),
app.state.memory_write_intake,
sdk_authority=app.state.memory_authority is MemoryAuthority.SDK,
write_queue=memory_write_queue,
@@ -1435,18 +1431,9 @@ def create_app(
# starts its threads wherever the first work arrives from, and
# from a busy loop that wait was 2.5 seconds.
warm_bulk_pool()
if app.state.memory_authority is MemoryAuthority.LEGACY:
_start_backfill(app)
_start_visibility_recompute(app)
app.state.deletion_sync = watched_task(
asyncio.get_running_loop(),
lambda: _memory_processing_supervisor(app).keep_deletions_in_sync(),
name="one-memory-deletion-sync",
)
else:
app.state.memory_backfill = None
app.state.memory_visibility_task = None
app.state.deletion_sync = None
app.state.memory_backfill = None
app.state.memory_visibility_task = None
app.state.deletion_sync = None
# Both times the loop was blocked here it was found from outside,
# by hand, days later. This says so at the time, and says what was
# holding it.
@@ -1550,16 +1537,12 @@ def create_app(
app.state.memory_write_drain.stop()
with suppress(asyncio.CancelledError):
await app.state.memory_write_drain_task
await app.state.memory_processing_supervisor.shutdown()
if app.state.sdk_library_bridge is not None:
await app.state.sdk_library_bridge.close()
app.state.sdk_library_bridge = None
# HY clients share two process-owned loop threads. Close them
# after every bridge is down, while HY's SQLite executor is
# still available; interpreter atexit is too late for that.
from one_memory.hy_memory_bridge import close_shared_hy_runtimes
await asyncio.to_thread(close_shared_hy_runtimes)
await app.state.sdk_memory_runtime.shutdown()
await app.state.sdk_write_coordinator.close()
if app.state.sdk_client_handle is not None:
await app.state.sdk_client_handle.close()
app.state.sdk_client_handle = None
close_all_shared_runtimes()
# Last: what has just been shut down was using it.
shutdown_bulk_pool()
log_main_database_stats("shutdown")
@@ -1622,12 +1605,12 @@ def create_app(
"/api/v1/context",
"/api/v1/boundary",
"/api/v1/home-summary",
"/api/v1/memory-tasks",
"/api/v1/memory-processing",
)
if getattr(
app.state, "memory_authority", MemoryAuthority.LEGACY
) is MemoryAuthority.SDK and request.url.path.startswith(legacy_paths):
) is MemoryAuthority.SDK and request.url.path.startswith(
legacy_paths
):
return JSONResponse(
status_code=409,
content={"detail": "sdk_memory_authority_required"},
@@ -1820,7 +1803,7 @@ def create_app(
) -> dict[str, object]:
"""Semantic search IDs for the Reader process. Reader-only."""
try:
hits = await _memory_processing_supervisor(app).semantic_search(query, limit=limit)
hits = await _sdk_memory_runtime(app).semantic_search(query, limit=limit)
except Exception:
return {"ids": []}
return {"ids": [memory_id for memory_id, _score in hits]}
@@ -1850,7 +1833,7 @@ def create_app(
) -> dict[str, object]:
"""HY graph overlay for the Reader process. Reader-only."""
try:
snapshot = await _memory_processing_supervisor(app).graph_snapshot(limit=limit)
snapshot = await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
except Exception:
return {"consulted": False, "nodes": [], "edges": []}
return snapshot.model_dump(mode="json")
@@ -1861,7 +1844,7 @@ def create_app(
error = cast(str | None, getattr(app.state, "memory_backfill_error", None))
status = _backfill_state(backfill, error)
governance = getattr(app.state, "memory_governance_store", None)
supervisor = getattr(app.state, "memory_processing_supervisor", None)
supervisor = getattr(app.state, "sdk_memory_runtime", None)
if supervisor is not None:
accepted = supervisor.accepted_backfill_status()
if accepted is not None:
@@ -3092,15 +3075,18 @@ def create_app(
if app.state.memory_authority is not MemoryAuthority.SDK:
raise HTTPException(409, "legacy_memory_authority_required")
try:
page = await _sdk_library_bridge(app).list_sdk_memories(
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
statuses=("active",),
statuses=["active"],
limit=1,
offset=0,
order="desc",
)
except HyMemoryBridgeError as error:
raise HTTPException(503, str(error)) from None
page = HySdkMemoryPage.from_sdk(raw, limit=1, offset=0)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_list_failed") from None
return SdkHomeSummary(
memory_count=page.total,
latest=page.memories[0] if page.memories else None,
@@ -3159,14 +3145,17 @@ def create_app(
limit: int = Query(default=100, ge=1, le=500),
) -> HySdkMemoryPage:
try:
return await _sdk_library_bridge(app).list_sdk_memories(
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
statuses=tuple(statuses),
statuses=list(statuses),
offset=offset,
limit=limit,
)
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
return HySdkMemoryPage.from_sdk(raw, limit=limit, offset=offset)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_list_failed") from None
@app.get(
"/api/v2/user-profile",
@@ -3175,10 +3164,12 @@ def create_app(
)
async def sdk_user_profile() -> SdkUserProfileResponse:
try:
value = await _sdk_library_bridge(app).get_user_profile(user_id="local")
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
return SdkUserProfileResponse.model_validate(value)
client = await _sdk_client_handle(app).client()
value = await asyncio.to_thread(client.get_user_profile, user_id="local")
profile = HySdkUserProfile.from_sdk(value)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_profile_read_failed") from None
return SdkUserProfileResponse.model_validate(profile.model_dump())
@app.get(
"/api/v2/memories/{memory_id}",
@@ -3187,9 +3178,9 @@ def create_app(
)
async def get_sdk_memory(memory_id: str) -> HySdkMemoryRecord:
try:
memory = await _sdk_library_bridge(app).get_sdk_memory(memory_id)
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
memory = await _get_sdk_memory(_sdk_client_handle(app), memory_id)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_get_failed") from None
if memory is None:
raise HTTPException(status_code=404, detail="memory_not_found")
return memory
@@ -3203,12 +3194,15 @@ def create_app(
memory_id: str,
request: SdkMemoryUpdateRequest,
) -> HySdkMemoryRecord:
bridge = _sdk_library_bridge(app)
try:
await bridge.update(memory_id, request.content)
memory = await bridge.get_sdk_memory(memory_id)
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
client = await app.state.sdk_write_coordinator.client()
result = await asyncio.to_thread(client.update, memory_id, request.content)
if not isinstance(result, dict) or result.get("success") is not True:
raise HySdkError("hy_memory_update_rejected")
raw = await asyncio.to_thread(client.get, memory_id)
memory = None if raw is None else HySdkMemoryRecord.from_sdk(raw)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_update_failed") from None
if memory is None:
raise HTTPException(status_code=404, detail="memory_not_found")
return memory
@@ -3220,9 +3214,17 @@ def create_app(
)
async def delete_sdk_memory(memory_id: str) -> Response:
try:
await _sdk_library_bridge(app).delete(memory_id)
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
client = await app.state.sdk_write_coordinator.client()
result = await asyncio.to_thread(client.delete, memory_id)
if not isinstance(result, dict):
raise HySdkError("hy_memory_delete_rejected")
error_code = result.get("error_code")
if error_code or result.get("partial") is True:
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
if result.get("success") is not True and result.get("deleted_count") != 0:
raise HySdkError("hy_memory_delete_rejected")
except Exception as error:
raise _sdk_http_error(error, "hy_memory_delete_failed") from None
return Response(status_code=204)
@app.get(
@@ -3236,15 +3238,19 @@ def create_app(
limit: int = Query(default=100, ge=1, le=500),
) -> SdkRawPage:
try:
items, total = await _sdk_library_bridge(app).list_sdk_raw(
client = await _sdk_client_handle(app).client()
value = await asyncio.to_thread(
client.list_memories,
user_id="local",
source=source,
layers=["l1_raw"],
include_sources=[source] if source else None,
offset=offset,
limit=limit,
)
except HyMemoryBridgeError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
return SdkRawPage(items=items, total=total, offset=offset, limit=limit)
page = HySdkRawPage.from_memory_list(value, limit=limit, offset=offset)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_raw_list_failed") from None
return SdkRawPage(items=page.raw, total=page.total, offset=offset, limit=limit)
@app.get(
"/api/v1/memories",
@@ -3777,9 +3783,9 @@ def create_app(
hy_graph = (
None
if revoked
else await _memory_processing_supervisor(app).graph_snapshot(limit=limit)
else await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
)
except HyMemoryBridgeError:
except HySdkError:
hy_graph = None
revoked_now = await asyncio.to_thread(_read_access(app).has_revoked_sources)
@@ -3856,8 +3862,8 @@ def create_app(
if await asyncio.to_thread(_read_access(app).has_revoked_sources):
raise HTTPException(status_code=403, detail="memory_source_revoked")
try:
return await _memory_processing_supervisor(app).digest()
except HyMemoryBridgeError as error:
return await _sdk_memory_runtime(app).digest()
except HySdkError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
@app.get(
@@ -3881,17 +3887,26 @@ def create_app(
# Two numbers, not one: 675 as a single total reads as "stuck", when
# what is true is that the three from this morning are nearly done
# and the rest is catch-up (R-02).
supervisor = app.state.memory_processing_supervisor
supervisor = app.state.sdk_memory_runtime
depths = supervisor.queue_depths()
active_queue, in_flight_count = supervisor.active_depths()
planned, current = supervisor.manual_batch_progress()
sdk_counts = (
supervisor.status_counts()
if app.state.memory_authority is MemoryAuthority.SDK
else {}
)
# `running` counts rows the store still marks running, which includes
# work a worker has already given up on. Reporting that as "what is
# running" is how a stalled queue looks busy, so the strand is given
# its own number.
running_rows = counts.get(MemoryTaskStatus.RUNNING, 0)
return MemoryProcessingSummary(
pending=counts.get(MemoryTaskStatus.PENDING, 0),
pending=(
sdk_counts.get("pending", 0)
if sdk_counts
else counts.get(MemoryTaskStatus.PENDING, 0)
),
fresh_pending=depths.fresh,
backlog_pending=depths.backlog,
outside_window_pending=counts.get(
@@ -3899,20 +3914,28 @@ def create_app(
0,
),
recent_completions=depths.recent_completions,
running=running_rows,
stranded_running=max(0, running_rows - in_flight_count),
running=sdk_counts.get("running", running_rows),
stranded_running=(
0 if sdk_counts else max(0, running_rows - in_flight_count)
),
unexecuted_policy_failures=store.unexecuted_policy_failures(),
waiting_for_model=counts.get(MemoryTaskStatus.WAITING_FOR_MODEL, 0),
completed=counts.get(MemoryTaskStatus.COMPLETED, 0),
failed=counts.get(MemoryTaskStatus.FAILED, 0),
unknown=counts.get(MemoryTaskStatus.UNKNOWN, 0),
waiting_for_model=sdk_counts.get(
"waiting_for_model",
counts.get(MemoryTaskStatus.WAITING_FOR_MODEL, 0),
),
completed=sdk_counts.get(
"completed",
counts.get(MemoryTaskStatus.COMPLETED, 0),
),
failed=sdk_counts.get("failed", counts.get(MemoryTaskStatus.FAILED, 0)),
unknown=sdk_counts.get("unknown", counts.get(MemoryTaskStatus.UNKNOWN, 0)),
unknown_reasons=store.unknown_error_counts(),
blocked_stages=tuple(
MemoryProcessingBlock(kind=kind, reason=reason, count=count)
for kind, reason, count in store.blocked_stages()[:6]
),
seconds_per_completion=store.seconds_per_completion(),
extraction_window_preset=supervisor.extraction_window_settings().window_preset,
extraction_window_preset=app.state.extraction_settings_store.get().window_preset,
manual_extract_batches_planned=planned,
manual_extract_batches_current=current,
active_queue=active_queue,
@@ -3930,15 +3953,17 @@ def create_app(
) -> RetryUnknownMemoryTasksReceipt:
"""Retry unknown writes only behind the user's explicit confirmation."""
del request
reconciled = await _memory_processing_supervisor(app).reconcile_unknown_outputs()
reconciled = await _sdk_memory_runtime(app).reconcile_unknown_outputs()
from one_memory.memory_server.durable_queue import DurableWriteQueue
queue = DurableWriteQueue(app.state.memory_write_queue_path)
unknown = queue.unknown_jobs(limit=2048)
for job in unknown:
queue.resolve_unknown_from_linked_output(job.job_id)
if app.state.memory_authority is MemoryAuthority.LEGACY:
for job in unknown:
queue.resolve_unknown_from_linked_output(job.job_id)
requeued = sum(queue.retry_unknown_confirmed(job.job_id) for job in unknown)
requeued += app.state.memory_task_store.retry_unknown_confirmed()
if app.state.memory_authority is MemoryAuthority.LEGACY:
requeued += app.state.memory_task_store.retry_unknown_confirmed()
return RetryUnknownMemoryTasksReceipt(reconciled=reconciled, requeued=requeued)
@app.get(
@@ -3949,8 +3974,8 @@ def create_app(
async def get_graph_width() -> GraphWidthStatus:
"""Report the mismatch. Never writes: this is polled by the app."""
try:
mismatch = await _memory_processing_supervisor(app).graph_width_mismatch()
except HyMemoryBridgeError as error:
mismatch = await _sdk_memory_runtime(app).graph_width_mismatch()
except HySdkError as error:
raise HTTPException(status_code=503, detail=str(error)) from None
if mismatch is None:
return GraphWidthStatus()
@@ -3973,10 +3998,10 @@ def create_app(
decision is rejected by validation rather than assumed, because the
assumption that cost a live graph once was made in code, not by a user.
"""
supervisor = _memory_processing_supervisor(app)
supervisor = _sdk_memory_runtime(app)
try:
outcome = await supervisor.decide_graph_width(request.decision)
except HyMemoryBridgeError as error:
except HySdkError as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except OSError as error:
# A failed export left nothing behind, but the user must be told
@@ -3994,7 +4019,7 @@ def create_app(
dependencies=protected,
)
def get_extraction_settings() -> ExtractionSettings:
return _memory_processing_supervisor(app).extraction_window_settings()
return app.state.extraction_settings_store.get()
@app.put(
"/api/v1/extraction-settings",
@@ -4002,7 +4027,7 @@ def create_app(
dependencies=protected,
)
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
return app.state.extraction_settings_store.set_window_preset(request.window_preset)
@app.get(
"/api/v1/sedimentation",
@@ -4052,7 +4077,7 @@ def create_app(
dependencies=protected,
)
def manual_extract(request: ManualExtractRequest) -> ManualExtractReceipt:
supervisor = _memory_processing_supervisor(app)
supervisor = _sdk_memory_runtime(app)
task_ids: tuple[str, ...] | None = None
if request.evidence_ids:
resolved: list[str] = []
@@ -4170,11 +4195,11 @@ def create_app(
"""
store = _memory_store(app)
try:
await _memory_processing_supervisor(app).update_hy_memory(
await _sdk_memory_runtime(app).update_hy_memory(
memory.memory_id,
memory.content,
)
except HyMemoryBridgeError as error:
except HySdkError as error:
return store.fail_index_sync(
memory.memory_id,
user_id="local",
@@ -4213,7 +4238,7 @@ def create_app(
if action == "delete":
deletion = _memory_store(app).deletion(memory_id, user_id="local")
if deletion is None or deletion.status is not DeletionStatus.COMPLETED:
await _memory_processing_supervisor(app).finish_deletion(memory_id)
await _sdk_memory_runtime(app).finish_deletion(memory_id)
return memory
except KeyError:
raise HTTPException(status_code=404, detail="memory_not_found") from None
@@ -4378,7 +4403,7 @@ async def _request_answerer(
def _sdk_request_answerer(app: FastAPI) -> SdkMemoryAnswerer:
return SdkMemoryAnswerer(
bridge=_sdk_library_bridge(app),
client_handle=_sdk_client_handle(app),
profiles=_model_profile_store(app),
secrets=_secret_store(app),
authorize_generation=lambda profile, question, grounding: _authorize_sdk_answer_generation(
@@ -4403,9 +4428,9 @@ async def _authorize_sdk_answer_generation(
identity = await asyncio.to_thread(_model_profile_store(app).active_identity)
if identity != (profile.profile_id, profile.revision):
raise MemoryAnswerError("memory_answer_generation_not_authorized")
bridge = _sdk_library_bridge(app)
handle = _sdk_client_handle(app)
for memory in grounding:
current = await bridge.get_sdk_memory(memory.memory_id)
current = await _get_sdk_memory(handle, memory.memory_id)
if current is None:
raise MemoryAnswerError("memory_answer_generation_not_authorized")
projected = SdkAskMemory(
@@ -4488,20 +4513,6 @@ class _LazyASGIApp:
app = _LazyASGIApp()
class _LazySdkBridge:
"""Resolve the active model only when an SDK operation is actually used."""
def __init__(self, app: FastAPI) -> None:
self.app = app
def __getattr__(self, name: str) -> Callable[..., Awaitable[Any]]:
async def call(*args: Any, **kwargs: Any) -> Any:
method = getattr(_sdk_library_bridge(self.app), name)
return await method(*args, **kwargs)
return call
def _sync_refused(code: str) -> HTTPException:
"""A sync refusal that says which stage refused.
@@ -5200,7 +5211,7 @@ def _resume_sedimentation(app: FastAPI) -> None:
return
app.state.sedimentation_resume = watched_task(
asyncio.get_running_loop(),
lambda: _memory_processing_supervisor(app).resume_allowed(),
lambda: _sdk_memory_runtime(app).resume_allowed(),
component="memory.sedimentation",
name="one-memory-sedimentation-resume",
)
@@ -5368,9 +5379,8 @@ async def _run_backfill(app: FastAPI) -> None:
def _evidence_excerpt(app: FastAPI, record: EvidenceRecord) -> EvidenceExcerpt:
supervisor = _memory_processing_supervisor(app)
task = _memory_task_store(app).task_for_evidence(record.evidence_id)
window = extraction_window_for(supervisor.extraction_window_settings().window_preset)
window = extraction_window_for(app.state.extraction_settings_store.get().window_preset)
return EvidenceExcerpt(
evidence_id=record.evidence_id,
available=True,
@@ -5390,23 +5400,62 @@ def _evidence_excerpt(app: FastAPI, record: EvidenceRecord) -> EvidenceExcerpt:
)
def _memory_processing_supervisor(app: FastAPI) -> MemoryProcessingSupervisor:
def _sdk_memory_runtime(app: FastAPI) -> SdkMemoryRuntime:
return cast(
MemoryProcessingSupervisor,
app.state.memory_processing_supervisor,
SdkMemoryRuntime,
app.state.sdk_memory_runtime,
)
def _sdk_library_bridge(app: FastAPI) -> HyMemoryBridge:
bridge = cast(HyMemoryBridge | None, app.state.sdk_library_bridge)
if bridge is None:
bridge = HyMemoryBridge(
def _sdk_client_handle(app: FastAPI) -> SdkClientHandle:
handle = cast(SdkClientHandle | None, app.state.sdk_client_handle)
if handle is None:
handle = SdkClientHandle(
profile=builtin_retrieval_snapshot(),
api_key=None,
data_dir=cast(Settings, app.state.settings).data_dir,
)
app.state.sdk_library_bridge = bridge
return bridge
app.state.sdk_client_handle = handle
return handle
async def _get_sdk_memory(
handle: SdkClientHandle,
memory_id: str,
) -> HySdkMemoryRecord | None:
client = await handle.client()
raw = await asyncio.to_thread(client.get, memory_id)
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
async def _search_sdk_memories(
handle: SdkClientHandle,
query: str,
*,
user_id: str,
agent_id: str,
limit: int = 12,
) -> tuple[object, ...]:
client = await handle.client()
raw = await asyncio.to_thread(
client.search,
query,
user_ids=[user_id],
agent_ids=[] if agent_id == "*" else [agent_id],
limit=limit,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=limit,
intention_limit=limit,
scene="normal",
reader=HY_MEMORY_READER,
)
return parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
def _sdk_http_error(error: Exception, fallback: str) -> HTTPException:
detail = error.code if isinstance(error, HySdkError) else fallback
return HTTPException(status_code=503, detail=detail)
def _evidence_store(app: FastAPI) -> EvidenceStore:
+81 -16
View File
@@ -421,8 +421,9 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
return 4
from one_memory.database_access import close_main_databases
from one_memory.hy_memory_bridge import HyMemoryBridge
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE, HY_MEMORY_VERSION
from one_memory.hy_sdk_migration import HySdkMigration
from one_memory.hy_sdk_runtime import acquire_client
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
from one_memory.memory_server.migrations import MIGRATIONS
from one_memory.model_profiles import ModelProfileStore
@@ -430,7 +431,12 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
MigrationBundleRejected,
inspect_sdk_migration_bundle,
)
from one_memory.sdk_migration_runner import BackfillResult, run_sdk_data_plane_backfill
from one_memory.sdk_migration_runner import (
BackfillResult,
prepare_sdk_migration_staging,
run_sdk_data_plane_backfill,
sdk_migration_staging,
)
from one_memory.secrets import KeyringSecretStore, SecretStoreUnavailableError
database = settings.data_dir / "one-memory.sqlite3"
@@ -481,17 +487,32 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
separators=(",", ":"),
)
backend_fingerprint = hashlib.sha256(backend_material.encode()).hexdigest()
staging = sdk_migration_staging(
data_dir=settings.data_dir,
migration_id=migration_id,
namespace=HY_MEMORY_NAMESPACE,
)
try:
staging_data_dir = prepare_sdk_migration_staging(
paths=staging,
database_path=database,
)
except Exception:
code = "sdk_migration_staging_failed"
staging_data_dir = None
async def run_backfill() -> BackfillResult:
bridge = HyMemoryBridge(
profile=profile,
assert staging_data_dir is not None
lease = await asyncio.to_thread(
acquire_client,
profile,
api_key=secret,
data_dir=settings.data_dir,
data_dir=staging_data_dir,
)
try:
return await run_sdk_data_plane_backfill(
database_path=database,
bridge=bridge,
client=HySdkMigration(lease.client),
migration_id=migration_id,
bundle_id=bundle_id,
source_runtime_build=str(manifest.get("runtime_build") or ""),
@@ -501,13 +522,21 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
backend_fingerprint=backend_fingerprint,
)
finally:
await bridge.close()
await asyncio.to_thread(lease.close)
try:
result = asyncio.run(run_backfill())
result = (
asyncio.run(run_backfill())
if staging_data_dir is not None
else None
)
except Exception:
code = "sdk_migration_backfill_failed"
else:
if result is None:
document = {"schema_version": 1, "state": "failed", "code": code}
_write_restore(stream, as_json, document, f"SDK backfill 失败:{code}")
return 1
document = {
"schema_version": 1,
"state": result.state,
@@ -583,20 +612,39 @@ def sdk_cutover(options: Sequence[str], stream: TextIO) -> int:
_write_restore(stream, as_json, document, "预览通过;加 --yes 后提交 SDK authority。")
return 4
from one_memory.hy_runtime_config import HY_MEMORY_VERSION
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE, HY_MEMORY_VERSION
from one_memory.sdk_data_plane_migration import DataPlaneMigrationError
from one_memory.sdk_migration_bundle import MigrationBundleRejected
from one_memory.sdk_migration_runner import commit_reconciled_cutover
from one_memory.sdk_migration_runner import (
activate_sdk_migration_staging,
commit_reconciled_cutover,
resume_legacy_cleanup,
sdk_migration_staging,
)
try:
marker = commit_reconciled_cutover(
database_path = settings.data_dir / "one-memory.sqlite3"
staging = sdk_migration_staging(
data_dir=settings.data_dir,
database_path=settings.data_dir / "one-memory.sqlite3",
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=HY_MEMORY_VERSION,
namespace=HY_MEMORY_NAMESPACE,
)
except (DataPlaneMigrationError, MigrationBundleRejected) as error:
marker = activate_sdk_migration_staging(
paths=staging,
database_path=database_path,
commit=lambda: commit_reconciled_cutover(
data_dir=settings.data_dir,
database_path=database_path,
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=HY_MEMORY_VERSION,
),
)
resume_legacy_cleanup(
data_dir=settings.data_dir,
database_path=database_path,
)
except (DataPlaneMigrationError, MigrationBundleRejected, OSError) as error:
code = getattr(error, "code", str(error))
document = {
"schema_version": 1,
@@ -638,10 +686,12 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
import sqlite3
from one_memory.database_access import main_database
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.sdk_data_plane_migration import (
MemoryAuthority,
SdkDataPlaneMigrationStore,
)
from one_memory.sdk_migration_runner import sdk_migration_staging
from one_memory.target_schema import legacy_tables_present
migration_store = SdkDataPlaneMigrationStore(database_path)
@@ -651,10 +701,25 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
generation = hashlib.sha256(f"{stat.st_size}:{stat.st_mtime_ns}".encode()).hexdigest()[:16]
if authority is MemoryAuthority.SDK:
legacy_tables = legacy_tables_present(database_path)
staging_cleanup = False
if marker is not None:
staging = sdk_migration_staging(
data_dir=settings.data_dir,
migration_id=marker.migration_id,
namespace=HY_MEMORY_NAMESPACE,
)
staging_cleanup = (
staging.protective_root.exists()
or staging.staging_home.exists()
)
document = {
"schema_version": 1,
"authority": "sdk",
"state": "cleanup_required" if legacy_tables else "complete",
"state": (
"cleanup_required"
if legacy_tables or staging_cleanup
else "complete"
),
"migration_id": marker.migration_id if marker else None,
"generation": generation,
"legacy_tables": list(legacy_tables),
@@ -42,6 +42,13 @@ from one_memory.context_raw import revalidate_raw_context
from one_memory.context_search import search_context
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception
from one_memory.external_memory_access import external_access
from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import sdk_relevance_floor
from one_memory.hy_sdk_types import (
HySdkMemoryRecord,
parse_sdk_raw_search,
parse_sdk_search,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.retrieval.raw_lane import read_raw_indexed_context
from one_memory.semantic_deadline import hy_wait_budget, search_within
@@ -455,19 +462,33 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
except Exception:
raise HTTPException(503, "context_authority_unavailable") from None
bridge = getattr(request.app.state, "sdk_library_bridge", None)
if bridge is None:
client_handle = getattr(request.app.state, "sdk_client_handle", None)
if client_handle is None:
raise HTTPException(503, "sdk_context_unavailable")
try:
client = await client_handle.client()
except Exception:
raise HTTPException(503, "sdk_context_unavailable") from None
async def raw_lane() -> tuple[SdkRawContextItem, ...]:
if "raw" not in value.kinds or not authority.raw_allowed:
return ()
hits = await bridge.search_sdk_raw(
raw = await asyncio.to_thread(
client.search,
value.query,
user_id="local",
source_ids=tuple(authority.raw_sources or ()),
user_ids=["local"],
agent_ids=[],
include_sources=list(authority.raw_sources or ()),
limit=value.limit,
min_score=0.0,
schema_min_score=0.0,
schema_limit=0,
intention_limit=0,
scene="chat",
reader=HY_MEMORY_READER,
layers=["l1_raw"],
)
hits = parse_sdk_raw_search(raw)
return tuple(
SdkRawContextItem(
raw_id=hit.raw_id,
@@ -487,15 +508,24 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
if "memory" not in value.kinds or not authority.memory_allowed:
return ()
hits = await bridge.search_memories(
raw = await asyncio.to_thread(
client.search,
value.query,
user_id="local",
agent_id="*",
user_ids=["local"],
agent_ids=[],
limit=value.limit,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=value.limit,
intention_limit=value.limit,
scene="normal",
reader=HY_MEMORY_READER,
)
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
items: list[SdkMemoryContextItem] = []
for hit in hits:
memory = await bridge.get_sdk_memory(hit.memory_id)
record = await asyncio.to_thread(client.get, hit.memory_id)
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
if memory is None or memory.status != "active":
continue
items.append(
@@ -0,0 +1,277 @@
"""Pure conversation-window helpers shared by capture, retrieval, and writes."""
from __future__ import annotations
import os
import re
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
from one_memory.memory_tasks import segment_evidence_record
MEMORY_TASK_TIMEOUT_MARGIN_SECONDS = 60.0
MEMORY_TASK_TIMEOUT_MIN_SECONDS = 180.0
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
_HARNESS_PREFIXES = (
"<codex_internal_context",
"# agents.md instructions",
"<environment_context>",
"<permissions instructions>",
"<system-reminder>",
"<system_notification>",
"<instructions>",
)
_HOST_ENVELOPE = re.compile(
r"(?:"
r"^<timestamp>.*?<system_(?:notification|reminder)>|"
r"^<system_(?:notification|reminder)>|"
r'^\{\s*"request_id"\s*:.*?"steps"\s*:'
r")",
re.IGNORECASE | re.DOTALL,
)
_WHOLE_TURN_CHARACTERS = 60_000
_WINDOW_BYTE_CEILING = 2_000_000
_HISTORY_MERGE_TURNS = 8
_HISTORY_MERGE_CHARACTERS = 8_000
def memory_task_timeout_seconds(model_timeout_seconds: float) -> float:
"""Bound a whole write without treating one model request as its budget."""
return max(
MEMORY_TASK_TIMEOUT_MIN_SECONDS,
float(model_timeout_seconds) + MEMORY_TASK_TIMEOUT_MARGIN_SECONDS,
)
def _coding_tool_result_max_bytes() -> int:
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
if raw is None:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
try:
value = int(raw)
except ValueError:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
_HY_TOOL_CHARACTERS = _coding_tool_result_max_bytes()
def _conversation_chunks(
records: list[EvidenceRecord],
*,
max_characters: int | None = None,
max_turns: int | None = None,
completed_only: bool = False,
) -> list[tuple[EvidenceRecord, ...]]:
"""Build bounded, complete user/assistant/tool windows."""
turns: list[list[EvidenceRecord]] = []
for record in records:
if not _is_hy_turn_record(record):
continue
pending_proposal = bool(turns) and not _opens_turn(turns[-1][0])
if not turns or (_opens_turn(record) and not pending_proposal):
turns.append([record])
else:
turns[-1].append(record)
if completed_only and turns and _trailing_user_turn_is_open(turns[-1]):
turns.pop()
whole = _whole_turns() and max_characters is None
if whole:
size = _hy_read_length
max_characters = _WHOLE_TURN_CHARACTERS
max_turns = max_turns or 1_000_000
else:
size = lambda record: len(record.verbatim) # noqa: E731
max_characters = max_characters or 12_000
max_turns = max_turns or 24
chunks: list[tuple[EvidenceRecord, ...]] = []
for turn in turns:
original_question = turn[0] if _opens_turn(turn[0]) else None
expanded: list[EvidenceRecord] = []
for record in turn:
reserve = (
size(original_question)
if original_question is not None
and record is not original_question
and size(original_question) < max_characters
else 0
)
segment_limit = max(1, max_characters - reserve)
if size(record) <= segment_limit:
expanded.append(record)
continue
spans = _semantic_segment_spans(record.verbatim, segment_limit)
expanded.extend(
segment_evidence_record(record, index=index, start=start, end=end)
if len(spans) > 1
else record
for index, (start, end) in enumerate(spans)
)
question = expanded[0] if expanded and _opens_turn(expanded[0]) else None
current: list[EvidenceRecord] = []
for record in expanded:
if current and (
len(current) >= max_turns
or sum(size(item) for item in current) + size(record) > max_characters
or sum(_window_bytes(item) for item in current) + _window_bytes(record)
> _WINDOW_BYTE_CEILING
):
chunks.append(tuple(current))
current = (
[question]
if question is not None
and max_turns > 1
and size(question) + size(record) <= max_characters
else []
)
current.append(record)
chunks.append(tuple(current))
return _merge_tool_only_chunks(chunks)
def _semantic_segment_spans(text: str, limit: int) -> list[tuple[int, int]]:
if len(text) <= limit:
return [(0, len(text))]
spans: list[tuple[int, int]] = []
start = 0
separators = ("\n\n", "\n", "。", "!", "?", ". ", "! ", "? ")
while start < len(text):
hard_end = min(len(text), start + limit)
if hard_end == len(text):
end = hard_end
else:
floor = start + max(1, limit // 2)
candidates = [text.rfind(separator, floor, hard_end) for separator in separators]
boundary = max(candidates)
if boundary < floor:
end = hard_end
else:
matched = next(
separator for separator in separators if text.startswith(separator, boundary)
)
end = boundary + len(matched)
spans.append((start, end))
start = end
return spans
def _merge_tool_only_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
) -> list[tuple[EvidenceRecord, ...]]:
if not any(chunk and all(record.role == "tool" for record in chunk) for chunk in chunks):
return chunks
merged: list[tuple[EvidenceRecord, ...]] = []
leading_tools: list[EvidenceRecord] = []
for chunk in chunks:
if not chunk:
continue
if all(record.role == "tool" for record in chunk):
if merged:
merged[-1] = (*merged[-1], *chunk)
else:
leading_tools.extend(chunk)
continue
if leading_tools:
chunk = (*leading_tools, *chunk)
leading_tools = []
merged.append(chunk)
return [] if leading_tools else merged
def _merge_historical_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
*,
max_turns: int = _HISTORY_MERGE_TURNS,
max_characters: int = _HISTORY_MERGE_CHARACTERS,
) -> list[tuple[EvidenceRecord, ...]]:
if not chunks:
return []
merged: list[tuple[EvidenceRecord, ...]] = []
current: list[EvidenceRecord] = []
current_turns = 0
current_characters = 0
for chunk in chunks:
chunk_characters = sum(len(record.verbatim) for record in chunk)
if current and (
current_turns >= max_turns or current_characters + chunk_characters > max_characters
):
merged.append(tuple(current))
current = []
current_turns = 0
current_characters = 0
current.extend(chunk)
current_turns += 1
current_characters += chunk_characters
if current:
merged.append(tuple(current))
return merged
def _chunk_input_ref(chunk: tuple[EvidenceRecord, ...]) -> str:
user_records = tuple(item for item in chunk if item.role == "user")
return user_records[-1].evidence_id if user_records else chunk[-1].evidence_id
def _opens_turn(record: EvidenceRecord) -> bool:
return record.role == "user"
def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
if len(turn) != 1:
return False
opener = turn[0]
return (
_opens_turn(opener)
and opener.content_kind is EvidenceContentKind.RAW_CONVERSATION
and not _is_memory_note(opener)
)
def _is_hy_turn_record(record: EvidenceRecord) -> bool:
if record.role == "tool":
return record.content_kind is EvidenceContentKind.RAW_CONVERSATION and bool(
record.verbatim.strip()
)
return _is_hy_input_turn(record)
def _is_hy_input_turn(record: EvidenceRecord) -> bool:
source = record.verbatim.strip()
if not source:
return False
if record.content_kind is EvidenceContentKind.MEMORY_CONTENT:
return True
if record.role == "assistant":
return True
if record.role != "user":
return False
if source.casefold().startswith(_HARNESS_PREFIXES):
return False
return _HOST_ENVELOPE.search(source[:2_000]) is None
def _is_memory_note(record: EvidenceRecord) -> bool:
meta = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
return (
record.source_id == "one-memory-manual"
and meta is not None
and meta.raw_meta.get("input_mode") == "direct_memory_note"
)
def _window_bytes(record: EvidenceRecord) -> int:
return len(record.verbatim.encode("utf-8"))
def _whole_turns() -> bool:
return os.environ.get("ONE_MEMORY_SPLIT_TURNS", "").strip() != "1"
def _hy_read_length(record: EvidenceRecord) -> int:
if record.role == "tool":
return min(len(record.verbatim), _HY_TOOL_CHARACTERS)
return len(record.verbatim)
@@ -0,0 +1,271 @@
"""Kuzu graph-width inspection and explicit user-directed administration.
This module is the second graph architecture exception. It may open Kuzu
directly to inspect an existing graph's schema, but owns no prompts, memory
reconciliation, coding data, vector database, or legacy governance behavior.
"""
from __future__ import annotations
import logging
import os
import re
import shutil
import threading
from dataclasses import dataclass
from datetime import UTC, datetime
from enum import StrEnum
from importlib import import_module
from pathlib import Path
from typing import Any
from pydantic import BaseModel, ConfigDict
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
class GraphWidthError(RuntimeError):
"""Base error for graph-width inspection and administration."""
class GraphWidthDecision(StrEnum):
"""What to do when an existing graph disagrees with the embedding width."""
KEEP_EXISTING = "keep_existing"
REBUILD_WITH_NEW = "rebuild_with_new"
EXPORT_FIRST = "export_first"
class GraphWidthMismatch(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
stored_width: int
expected_width: int
graph_path: str
class GraphWidthOutcome(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
archived_path: str | None = None
exported_path: str | None = None
class GraphWidthMismatchError(GraphWidthError):
"""Raised instead of silently archiving a live graph."""
def __init__(self, mismatch: GraphWidthMismatch) -> None:
super().__init__(
f"graph_width_mismatch:stored={mismatch.stored_width}"
f":expected={mismatch.expected_width}"
)
self.mismatch = mismatch
def graph_store_enabled() -> bool:
"""Whether HY should open the Kuzu graph store."""
raw = os.environ.get("ONE_MEMORY_GRAPH_STORE", "").strip().lower()
if not raw:
return False
return raw != "none"
def inspect_graph_width_mismatch(config: Any) -> GraphWidthMismatch | None:
"""Return a mismatch without renaming or deleting anything."""
db_path = Path(config.graph_store.db_path)
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
return _inspect_graph_width(db_path, expected)
def inspect_profile_graph_width(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
) -> GraphWidthMismatch | None:
"""Inspect the graph path and width selected by the active profile."""
del api_key
return _inspect_graph_width(
_profile_graph_path(data_dir),
_profile_embedding_width(profile),
)
def decide_profile_graph_width(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
decision: GraphWidthDecision,
) -> GraphWidthOutcome:
"""Apply the user's decision to the active profile's graph."""
del api_key
config = _GraphAdminConfig(
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir))),
embedder=_EmbeddingWidthConfig(_profile_embedding_width(profile)),
)
return apply_graph_width_decision(
config,
decision,
export_root=data_dir.parent,
)
def apply_graph_width_decision(
config: Any,
decision: GraphWidthDecision,
*,
export_root: Path | None = None,
) -> GraphWidthOutcome:
"""Act only on a current mismatch and only as explicitly requested."""
mismatch = inspect_graph_width_mismatch(config)
if mismatch is None:
return GraphWidthOutcome()
if decision is GraphWidthDecision.REBUILD_WITH_NEW:
archive = retire_mismatched_graph_store(config)
return GraphWidthOutcome(archived_path=str(archive) if archive is not None else None)
if decision is GraphWidthDecision.EXPORT_FIRST:
if export_root is None:
raise GraphWidthError("export_root_required")
return GraphWidthOutcome(
exported_path=str(export_graph_store(config, export_root=export_root))
)
return GraphWidthOutcome()
def export_graph_store(config: Any, *, export_root: Path) -> Path:
"""Copy the graph outside its data directory, atomically."""
db_path = Path(config.graph_store.db_path)
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")
destination = export_root / f"one-memory-graph-{stamp}"
staging = destination.with_name(f"{destination.name}.partial")
sources = [db_path] + [
sibling
for suffix in (".wal", ".shadow", ".tmp")
if (sibling := db_path.with_name(f"{db_path.name}{suffix}")).exists()
]
try:
staging.mkdir(parents=True)
for source in sources:
shutil.copy2(source, staging / source.name)
staging.rename(destination)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
raise
logging.getLogger(__name__).info(
"exported hy graph store to %s (%d files)",
destination,
len(sources),
)
return destination
def retire_mismatched_graph_store(config: Any) -> Path | None:
"""Archive a mismatched Kuzu graph only after the user chooses rebuild."""
db_path = Path(config.graph_store.db_path)
if not db_path.exists():
return None
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
if not expected:
return None
stored = _graph_embedding_width(db_path)
if stored is None or stored == int(expected):
return None
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
archive = db_path.with_name(f"{db_path.name}.dim{stored}.{stamp}.bak")
db_path.rename(archive)
for suffix in (".wal", ".shadow", ".tmp"):
sibling = db_path.with_name(f"{db_path.name}{suffix}")
if sibling.exists():
sibling.rename(archive.with_name(f"{archive.name}{suffix}"))
logging.getLogger(__name__).warning(
"retired hy graph store with embedding width %s (model expects %s): %s",
stored,
expected,
archive.name,
)
return archive
_GRAPH_WIDTH_CACHE: dict[str, int | None] = {}
_GRAPH_WIDTH_LOCK = threading.Lock()
def _graph_embedding_width(db_path: Path) -> int | None:
"""Read the Memory embedding width directly from Kuzu's schema."""
try:
kuzu = import_module("kuzu")
except ModuleNotFoundError:
return None
key = str(db_path.resolve())
with _GRAPH_WIDTH_LOCK:
if key in _GRAPH_WIDTH_CACHE:
return _GRAPH_WIDTH_CACHE[key]
try:
database = kuzu.Database(str(db_path), read_only=True)
except Exception:
return None
width: int | None = None
try:
connection = kuzu.Connection(database)
result = connection.execute("CALL TABLE_INFO('Memory') RETURN *")
while result.has_next():
row = result.get_next()
if len(row) < 3 or row[1] != "embedding":
continue
match = re.search(r"\[(\d+)\]", str(row[2]))
width = int(match.group(1)) if match else None
break
except Exception:
width = None
with _GRAPH_WIDTH_LOCK:
_GRAPH_WIDTH_CACHE[key] = width
return width
def _inspect_graph_width(
db_path: Path,
expected_width: object,
) -> GraphWidthMismatch | None:
if not db_path.exists() or not expected_width:
return None
stored = _graph_embedding_width(db_path)
if stored is None or stored == int(expected_width):
return None
return GraphWidthMismatch(
stored_width=int(stored),
expected_width=int(expected_width),
graph_path=str(db_path),
)
def _profile_graph_path(data_dir: Path) -> Path:
name = "graph" if graph_store_enabled() else "graph-disabled"
return data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / name
def _profile_embedding_width(profile: ModelProfileSnapshot) -> int | None:
if profile.embedding_dimensions is not None:
return profile.embedding_dimensions
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
from one_memory.embedding_assets import BUILTIN_EMBEDDING_DIMENSIONS
return BUILTIN_EMBEDDING_DIMENSIONS
return None
@dataclass(frozen=True)
class _GraphStoreConfig:
db_path: str
@dataclass(frozen=True)
class _EmbeddingWidthConfig:
embedding_dims: int | None
@dataclass(frozen=True)
class _GraphAdminConfig:
graph_store: _GraphStoreConfig
embedder: _EmbeddingWidthConfig
@@ -0,0 +1,185 @@
"""Read-only projection of HY's Kuzu graph for MemHub's existing graph UI.
This module is one of the two explicit graph architecture exceptions. The
public SDK does not expose graph projection yet, so it may use the client's
private graph handle and loop thread. It must remain read-only and must not
grow memory, vector, coding, prompt, or governance responsibilities.
"""
from __future__ import annotations
from typing import Any, cast
from pydantic import BaseModel, ConfigDict
class HyGraphNode(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
node_id: str
content: str
layer: str | None = None
confidence: float | None = None
class HyGraphEdge(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
source_node_id: str
target_node_id: str
relation: str
weight: float | None = None
class HyGraphSnapshot(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
nodes: tuple[HyGraphNode, ...] = ()
edges: tuple[HyGraphEdge, ...] = ()
# False when HY was never consulted, which is not the same as HY having
# induced no relations.
consulted: bool = True
def read_graph_snapshot(
client: object,
user_id: str,
agent_id: str,
limit: int,
) -> HyGraphSnapshot:
raw_nodes, raw_edges = _read_graph_snapshot(client, user_id, agent_id, limit)
return HyGraphSnapshot(
nodes=tuple(
HyGraphNode(
node_id=str(row[0]),
content=str(row[1]),
layer=str(row[2]) if row[2] is not None else None,
confidence=_safe_float(row[3]),
)
for row in raw_nodes
if len(row) >= 4 and row[0] and row[1]
),
edges=tuple(
HyGraphEdge(
source_node_id=str(row[0]),
target_node_id=str(row[1]),
relation=str(row[2]),
weight=_safe_float(row[3]),
)
for row in raw_edges
if len(row) >= 4 and row[0] and row[1] and row[2]
),
)
def _read_graph_snapshot(
client: object,
user_id: str,
agent_id: str,
limit: int,
) -> tuple[list[list[object]], list[list[object]]]:
"""Read the graph through the SDK handles until a public API exists."""
concrete = cast(Any, client)
# Keep the existing architecture gate unchanged; this module is the named
# graph-only exception while the rest of MemHub remains on public handles.
graph_store = getattr(concrete, "_graph_store") # noqa: B009
if graph_store is None:
return ([], [])
loop_thread = getattr(concrete, "_loop_thread") # noqa: B009
# Kuzu rejects a prepared statement that is handed a parameter its text
# never references, so a wildcard read must not bind agent_id at all.
params: dict[str, object] = {"user_id": user_id}
if agent_id != "*":
params["agent_id"] = agent_id
safe_limit = max(1, min(int(limit), 1_000))
memory_agent = "" if agent_id == "*" else " AND m.agent_id = $agent_id"
related_agent = (
"" if agent_id == "*" else " AND a.agent_id = $agent_id AND b.agent_id = $agent_id"
)
source_agent = "" if agent_id == "*" else " AND a.agent_id = $agent_id"
nodes = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN m.node_id, m.content, m.layer, m.confidence
LIMIT {safe_limit}
""",
params,
)
)
or []
)
related_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (a:Memory)-[r:RELATED_TO]->(b:Memory)
WHERE a.user_id = $user_id{related_agent}
AND b.user_id = $user_id
AND a.status = 'active' AND b.status = 'active'
RETURN a.node_id, b.node_id, r.relation_type, r.weight
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
derived_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (a:Memory)-[:DERIVED_FROM]->(v:VdbRef)
WHERE a.user_id = $user_id{source_agent}
AND a.status = 'active'
RETURN a.node_id, v.node_id, 'derived_from', 1.0
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
topic_nodes = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN DISTINCT t.topic_id, t.name, 'TOPIC', 1.0
LIMIT {safe_limit}
""",
params,
)
)
or []
)
tagged_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN m.node_id, t.topic_id, 'tagged_with', 1.0
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
return (
[*nodes, *topic_nodes],
[*related_edges, *derived_edges, *tagged_edges],
)
def _safe_float(value: object) -> float | None:
if isinstance(value, (int, float)):
return float(value)
return None
@@ -1,128 +0,0 @@
"""Which memories HY no longer counts as current, read from HY itself.
HY owns a memory's lifecycle: it supersedes, deduplicates, shadows and
restores on its own schedule — during a search, while deleting the head of a
chain — and most of that never passes through a write the ledger sees. So
the ledger does not keep a copy of any of it; readers ask HY, once per read,
for the nodes it has set aside.
Only the vector store is read. HY writes a node's status there on every
transition, and the graph holds no status the vector store lacks.
"""
import json
from collections.abc import Collection, Mapping
from dataclasses import dataclass, field
from importlib import import_module
from types import MappingProxyType
from typing import Any
#: HY's one status for a node that is current. Everything else it writes —
#: superseded, shadow, archived, negated, conflicted — takes it out of recall.
_ACTIVE = "active"
#: HY's verbatim chunks. Stored as shadow from the start and never a memory,
#: so never something a reader could have shown.
_RAW_LAYER = "l1_raw"
class HyLifecycleUnreadable(RuntimeError):
"""HY's store could not be read; never to be taken for "nothing is hidden"."""
@dataclass(frozen=True, eq=False)
class HyLifecycle:
"""HY's set-aside nodes for one user, each with what HY says replaced it."""
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
def __post_init__(self) -> None:
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
def __eq__(self, other: object) -> bool:
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(other.hidden)
def __hash__(self) -> int:
return hash(tuple(sorted(self.hidden.items())))
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
return self.hidden.get(memory_id, ())
def replacing(self, memory_id: str) -> tuple[str, ...]:
"""The set-aside memories HY names this one as the replacement of."""
return tuple(sorted(old for old, by in self.hidden.items() if memory_id in by))
def read_hy_lifecycle(
client: object, user_id: str, ids: Collection[str] | None = None
) -> HyLifecycle:
"""Every node of `user_id` HY holds as anything but active, in one read.
With `ids`, only those nodes are looked up — what a single-memory read
needs, and a lookup by key rather than a scan; `replacing` is then only
known among them.
Read one level below HY's own listing, which drops a failed read and
answers empty: a hidden memory would then read as current.
"""
native: Any = client
try:
chroma = import_module("hy_memory.data.vector_store_chroma")
store = native._vector_store
collection = store._collection
if not isinstance(store, chroma.ChromaVectorStore) or collection is None:
raise HyLifecycleUnreadable("hy_lifecycle_store_unsupported")
found = collection.get(
ids=None if ids is None else list(ids),
where={
"$and": [
{"user_id": {"$eq": user_id}},
{"status": {"$ne": _ACTIVE}},
{"layer": {"$ne": _RAW_LAYER}},
]
},
include=["metadatas"],
)
metadatas = found["metadatas"]
point_ids = found["ids"]
if (
not isinstance(point_ids, list)
or not isinstance(metadatas, list)
or len(point_ids) != len(metadatas)
):
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
except HyLifecycleUnreadable:
raise
except Exception as error:
raise HyLifecycleUnreadable("hy_lifecycle_unreadable") from error
hidden: dict[str, tuple[str, ...]] = {}
for point_id, metadata in zip(point_ids, metadatas, strict=True):
if not isinstance(point_id, str) or not point_id or not isinstance(metadata, Mapping):
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
payload = chroma.ChromaVectorStore._chroma_metadata_to_payload(dict(metadata))
node_id = payload.get("node_id") or point_id
if not isinstance(node_id, str) or not node_id or node_id != point_id:
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
hidden[node_id] = _replacements(payload)
return HyLifecycle(hidden=hidden)
def _replacements(payload: Mapping[str, object]) -> tuple[str, ...]:
"""What HY names as taking this node's place: its chain successor or the kept duplicate."""
names: list[str] = []
superseded_by = payload.get("superseded_by")
if isinstance(superseded_by, list):
names.extend(value for value in superseded_by if isinstance(value, str) and value)
kept = payload.get("dedup_kept_id")
custom = payload.get("custom")
if isinstance(custom, str):
try:
custom = json.loads(custom)
except ValueError:
custom = None
if not isinstance(kept, str) and isinstance(custom, Mapping):
audit = custom.get("hy_dedup")
if isinstance(audit, Mapping):
kept = audit.get("kept_id")
if isinstance(kept, str) and kept:
names.append(kept)
return tuple(dict.fromkeys(names))
File diff suppressed because it is too large. Load diff
@@ -1,331 +0,0 @@
"""Transport validation for the pinned HY SDK; HY's prompts and decisions pass through unchanged."""
import logging
from contextlib import nullcontext
from time import monotonic
from typing import Any
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
from one_memory.hy_source_context import HyWriteBindings
from one_memory.hy_step_timing import HyStepTimer
def _http_status(value: Any) -> int | None:
return value if type(value) is int and 100 <= value <= 599 else None
def _exception_status(error: Exception) -> int | None:
try:
status = _http_status(getattr(error, "status_code", None))
if status is not None:
return status
status = _http_status(getattr(getattr(error, "response", None), "status_code", None))
if status is not None:
return status
# Native LLMRetryError retains sanitized attempt dictionaries. Only the
# final attempt describes the terminal failure; never parse error text.
attempts = getattr(error, "attempts", None)
if isinstance(attempts, (list, tuple)) and attempts and isinstance(attempts[-1], dict):
return _http_status(attempts[-1].get("status_code"))
except Exception:
# Diagnostic attribute access must not replace the provider exception.
return None
return None
def _model_failure_code(status: int | None) -> str:
specific = {
401: "hy_memory_model_authentication_failed",
402: "hy_memory_model_payment_required",
403: "hy_memory_model_access_denied",
408: "hy_memory_model_timeout",
429: "hy_memory_model_rate_limited",
}
if status in specific:
return specific[status]
if status is not None and 500 <= status <= 599:
return "hy_memory_model_service_unavailable"
if status is not None and 400 <= status <= 422:
return "hy_memory_model_request_incompatible"
return "hy_memory_model_request_failed"
def _terminal_client_timeout(error: Exception) -> bool:
try:
attempts = getattr(error, "attempts", None)
if not isinstance(attempts, (list, tuple)) or not attempts:
return False
final = attempts[-1]
if type(final) is not dict:
return False
kind = final.get("failure_kind")
return type(kind) is str and kind == "timeout"
except Exception:
return False
class HyResponseGuard:
def __init__(self, provider: Any, *, phase: str = "extract") -> None:
self.provider = provider
#: Diagnostic label only; the request passes through unchanged.
self.phase = phase
self.failures: list[str] = []
def __getattr__(self, name: str) -> Any:
return getattr(self.provider, name)
async def complete(self, *args: Any, **kwargs: Any) -> Any:
return await self._complete("complete", args, kwargs)
async def complete_messages(self, *args: Any, **kwargs: Any) -> Any:
return await self._complete("complete_messages", args, kwargs)
async def _complete(self, method: str, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
for attempt in range(2):
started = monotonic()
try:
response = await getattr(self.provider, method)(*args, **kwargs)
except Exception as error:
status = _exception_status(error)
code = (
"hy_memory_model_timeout"
if status is None and _terminal_client_timeout(error)
else _model_failure_code(status)
)
self.failures.append(code)
logging.getLogger(__name__).warning(
"%s: status_code=%s", code, status if status is not None else "unknown"
)
diagnostic_event(
level="error",
component="hy.model",
event="hy.model_call.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code=code,
**diagnostic_exception(error),
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
)
raise
truncated = getattr(response, "finish_reason", "") == "length"
empty = not (response.content or "").strip() and not getattr(
response, "tool_calls", None
)
if not truncated and not empty:
diagnostic_event(
level="info",
component="hy.model",
event="hy.model_call.completed",
outcome="succeeded",
duration_ms=elapsed_ms(started),
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
)
return response
code = (
"hy_memory_model_output_truncated"
if truncated
else "hy_memory_model_empty_response"
)
if attempt:
self.failures.append(code)
raise ValueError(code)
logging.getLogger(__name__).warning("%s: retrying the same HY model request", code)
if truncated and "max_tokens" in kwargs:
kwargs = {**kwargs, "max_tokens": min(16000, kwargs["max_tokens"] * 2)}
raise AssertionError("unreachable")
class HyReconcileGuard:
#: Set only for the span of one leased write.
step_timer: HyStepTimer | None = None
def __init__(self, reconciler: Any) -> None:
self.reconciler = reconciler
self.failures: list[str] = []
self.bindings: HyWriteBindings | None = None
self.imported_contents: frozenset[str] | None = None
self.mutation_target_ids: set[str] = set()
def begin_write(self) -> None:
self.mutation_target_ids.clear()
def __getattr__(self, name: str) -> Any:
return getattr(self.reconciler, name)
async def reconcile(self, *args: Any, **kwargs: Any) -> Any:
timer = self.step_timer
with timer.step("reconcile") if timer is not None else nullcontext():
return await self._reconcile(args, kwargs)
async def _reconcile(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
started = monotonic()
if self.imported_contents is not None:
metadata = kwargs.get("new_memories_with_meta") or []
texts = kwargs.get("new_memories") or []
if (
not metadata
or not texts
or any(
not isinstance(item, dict) or item.get("content") not in self.imported_contents
for item in metadata
)
or any(text not in self.imported_contents for text in texts)
or {item["content"] for item in metadata} != set(texts)
):
code = "hy_memory_write_verification_import_input_mismatch"
self.failures.append(code)
raise ValueError(code)
try:
result = await self.reconciler.reconcile(*args, **kwargs)
except Exception as error:
diagnostic_event(
level="error",
component="hy.reconcile",
event="hy.reconcile.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code="hy_memory_reconcile_failed",
**diagnostic_exception(error),
)
raise
if not result.success:
self.failures.append("hy_memory_reconcile_failed")
diagnostic_event(
level="error",
component="hy.reconcile",
event="hy.reconcile.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code="hy_memory_reconcile_failed",
details={"result_count": len(result.ops)},
)
return result
try:
await self._check_targets(result, kwargs)
except ValueError as error:
code = str(error)
self.failures.append(code)
raise
except Exception as error:
code = "hy_memory_write_verification_failed"
self.failures.append(code)
raise ValueError(code) from error
owners: dict[str, set[str]] = {}
for item in kwargs.get("new_memories_with_meta") or []:
if item.get("owner") in {"user", "agent"}:
owners.setdefault(item["content"], set()).add(item["owner"])
# HY's no-candidate ADD path copies content verbatim but omits owner.
# Restore only an unambiguous exact match, never infer from wording.
for operation in result.ops:
source_owners = owners.get(operation.content, set())
if operation.op == "ADD" and operation.owner is None and len(source_owners) == 1:
operation.owner = next(iter(source_owners))
operation_counts = {"ADD": 0, "UPDATE": 0, "SUPERSEDE": 0, "DELETE": 0}
for operation in result.ops:
raw_name = getattr(operation, "op", "")
name = str(getattr(raw_name, "value", raw_name)).upper()
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
target = getattr(operation, "memory_id", None)
if isinstance(target, str) and target:
self.mutation_target_ids.add(target)
self.mutation_target_ids.update(
key
for key in getattr(operation, "memory_ids", ())
if isinstance(key, str) and key
)
if name == "ADD":
operation_counts["ADD"] += 1
elif name in {"UPDATE", "MERGE", "NOOP"}:
operation_counts["UPDATE"] += 1
elif name == "SUPERSEDE":
operation_counts["SUPERSEDE"] += 1
elif name in {"DELETE", "SHADOW"}:
operation_counts["DELETE"] += 1
diagnostic_event(
level="info",
component="hy.reconcile",
event="hy.reconcile.completed",
outcome="succeeded",
duration_ms=elapsed_ms(started),
details={
"created_count": operation_counts["ADD"],
"updated_count": operation_counts["UPDATE"],
"superseded_count": operation_counts["SUPERSEDE"],
"deleted_count": operation_counts["DELETE"],
"result_count": len(result.ops),
},
)
return result
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
"""HY may only touch nodes of this user and scope that this write was allowed to see."""
targets = {op.memory_id for op in result.ops if op.memory_id}
targets.update(key for op in result.ops for key in getattr(op, "memory_ids", []))
for key in targets:
node = await kwargs["vector_store"].get_by_id(key)
if (
node is None
or node.user_id != kwargs["user_id"]
or node.agent_id != kwargs["agent_id"]
):
raise ValueError("hy_memory_write_verification_target_invalid")
if self.bindings is not None:
known = self.bindings.known.get(key)
if known is None or (known.content, known.owner) != (node.content, node.owner):
raise ValueError("hy_memory_write_verification_target_unbound")
if self.bindings is None:
return
for op in result.ops:
if not op.content:
continue
linked = {key for key in (op.memory_id, *getattr(op, "memory_ids", [])) if key}
self.bindings.record_operation_support(op.content, op.owner, linked)
if op.memory_id:
self.bindings.verified_operations[op.memory_id] = (op.content, op.owner)
def install_hy_write_guards(client: Any) -> None:
"""The pinned HY SDK treats an empty reconcile response as a successful empty op list.
Bind this client's real write paths. Pro/lite use the ordinary writer;
ultra uses the System2 writer's inner writer. Keep HY prompts, extraction,
retrieval, reconciliation and storage intact; never patch the shared SDK
installation.
"""
async def install() -> None:
from one_memory.hy_write_verification import HyExtractionGuard
ordinary = await client._registry.get_writer()
system2 = await client._registry.get_system2_writer()
guards: list[Any] = []
seen_writers: set[int] = set()
seen_guards: set[int] = set()
for writer in (ordinary, system2._writer):
if id(writer) in seen_writers:
continue
seen_writers.add(id(writer))
agent = writer.mem_agent
if not isinstance(agent.llm_provider, HyResponseGuard):
agent.llm_provider = HyResponseGuard(agent.llm_provider)
agent.extractor.llm = agent.llm_provider
if not isinstance(agent.extractor, HyExtractionGuard):
agent.extractor = HyExtractionGuard(agent.extractor)
reconciler = writer.reconciler
provider = reconciler._get_llm()
if not isinstance(provider, HyResponseGuard):
reconciler._llm = HyResponseGuard(provider, phase="reconcile")
if not isinstance(reconciler, HyReconcileGuard):
writer._reconciler = HyReconcileGuard(reconciler)
for guard in (
agent.llm_provider,
reconciler._llm,
writer._reconciler,
agent.extractor,
):
if id(guard) not in seen_guards:
seen_guards.add(id(guard))
guards.append(guard)
client._one_memory_write_guards = tuple(guards)
client._loop_thread.run(install())
@@ -33,49 +33,8 @@ _COLLECTION_LOGGERS = (
)
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
# One Memory deliberately stays on ultra because its existing, user-visible
# profile/graph flow calls digest(), which the pinned public SDK exposes only
# for ultra clients. Reader and non-reranker behavior remain pinned for
# compatibility with the SDK authority-migration development preview.
HY_MEMORY_MODE = "ultra"
# The SDK only recognises `legacy` and `hybrid` (see
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
# READER_HYBRID)`). The old spelling `hybrid_v2` was silently rejected and
# fell back to legacy, so the hybrid-v2 recall strategy never ran at all.
HY_MEMORY_READER = "hybrid"
HY_MEMORY_BEHAVIOR_ENV: dict[str, str] = {
"MEMORY_MODE": HY_MEMORY_MODE,
"MEMORY_ENABLE_GRAPH": "true",
"MEMORY_GRAPH_RERANK_ENABLED": "false",
"MEMORY_GRAPH_TEMPORAL_SPINE": "true",
"MEMORY_TOPIC_SUMMARY_ENABLED": "false",
"MEMORY_EPISODE_SUMMARY_ENABLED": "false",
"MEMORY_EPISODE_RECALL_ENABLED": "true",
"MEMORY_DYNAMIC_EPISODE_ENABLED": "true",
"MEMORY_SESSION_EPISODE_ENABLED": "false",
"MEMORY_SESSION_EPISODE_CONTEXT_ENABLED": "false",
"MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED": "true",
"MEMORY_EPISODE_CAPSULE_QUERY_MIN": "0.65",
"MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN": "0.60",
"MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX": "3",
"MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT": "3",
"MEMORY_ENABLE_INTENTION": "false",
"MEMORY_ENABLE_QUERY_REWRITE": "false",
"MEMORY_FEW_SHOT_ENABLED": "true",
"MEMORY_HYBRID_V2_QUERY_EXPAND": "1",
"MEMORY_HYBRID_V2_GEO_INTENT_PRIOR": "0.6",
"MEMORY_HYBRID_V2_DIVERSITY": "1",
"MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR": "0.55",
"MEMORY_HYBRID_V2_TEMPORAL_BOOST": "0.18",
"MEMORY_CROSS_DOMAIN_CON_THRESHOLD": "0.5",
"MEMORY_READER_ENABLE_SUMMARY": "false",
"MEMORY_SUMMARY_ENABLED_IN_SYS2": "false",
"MEMORY_LOCAL_RERANK_ENABLED": "false",
"RECONCILE_ENABLED": "true",
"HY_MEMORY_READER": HY_MEMORY_READER,
}
def detailed_logging_enabled(data_dir: Path) -> bool:
return diagnostic_log_level(data_dir) == "debug"
@@ -168,7 +127,6 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
log_dir = memory_root / "logs"
memory_root.mkdir(parents=True, exist_ok=True)
log_dir.mkdir(parents=True, exist_ok=True)
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
os.environ["MEMORY_DATA_DIR"] = str(memory_root)
os.environ["MEMORY_LOG_DIR"] = str(log_dir)
level = diagnostic_log_level(data_dir)
@@ -178,9 +136,3 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
os.environ["MEMORY_MEMORY_OPERATIONS_ENABLED"] = "false"
os.environ["ANONYMIZED_TELEMETRY"] = "False"
os.environ.pop("CHROMA_OTEL_COLLECTION_ENDPOINT", None)
def configure_hy_behavior_environment() -> None:
"""Set import-time HY behavior flags without touching paths or secrets."""
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
@@ -0,0 +1,599 @@
"""Privileged offline access to the HY SDK data plane during App migration.
Runtime product code must use public ``HyMemoryClient`` methods. This module
is the deliberately isolated exception: App startup has stopped all writers,
so the one-time legacy migration may access SDK stores directly without
turning migration mechanics into public Client APIs.
"""
from __future__ import annotations
import asyncio
from collections.abc import Callable, Mapping
from datetime import datetime
from pathlib import Path
from typing import Any, Protocol
from hy_memory import ( # type: ignore[import-untyped]
DataPlaneRestore,
DataPlaneSnapshot,
restore_local_data_plane,
snapshot_local_data_plane,
)
from hy_memory.models.memory import ( # type: ignore[import-untyped]
MemoryLayer,
MemoryNode,
MemoryStatus,
SourceType,
)
from hy_memory.utils.source_filter import normalize_source # type: ignore[import-untyped]
from one_memory.hy_sdk_types import (
HySdkMemoryPage,
HySdkMemoryRecord,
HySdkMigrationError,
HySdkRawPage,
HySdkRawRecord,
)
class MigrationClient(Protocol):
_loop_thread: Any
_vector_store: Any
_embed_service: Any
_graph_store: Any
def list_memories(
self,
*,
user_id: str,
agent_id: str | None = None,
statuses: list[str] | None = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Mapping[str, object]: ...
def delete(self, memory_id: str) -> Mapping[str, object]: ...
def get_write_outcome(self, request_id: str) -> Mapping[str, object] | None: ...
class HySdkMigration:
"""Typed offline adapter over one initialized SDK data plane."""
def __init__(self, client: MigrationClient) -> None:
self.client = client
self._loop_thread = client._loop_thread
self._vector_store = client._vector_store
self._embed_service = client._embed_service
self._graph_store = client._graph_store
async def list_memories(
self,
*,
user_id: str,
agent_id: str | None = None,
statuses: tuple[str, ...] = ("active",),
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> HySdkMemoryPage:
raw = await _sdk_call(
"hy_sdk_memory_list_failed",
self.client.list_memories,
user_id=user_id,
agent_id=agent_id,
statuses=list(statuses),
limit=limit,
offset=offset,
order=order,
)
return _memory_page(raw, limit=limit, offset=offset)
async def write_legacy_memory(
self,
*,
content: str,
memory_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
layer: str,
status: str,
owner: str | None,
tags: tuple[str, ...],
supersedes: tuple[str, ...],
superseded_by: tuple[str, ...],
is_latest: bool,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_memory_import_failed",
self._loop_thread.run,
self._write_legacy_memory(
content=content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
status=status,
owner=owner,
tags=tags,
supersedes=supersedes,
superseded_by=superseded_by,
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
)
return _successful_result(raw, "hy_sdk_memory_import_failed")
async def _write_legacy_memory(
self,
*,
content: str,
memory_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
layer: str,
status: str,
owner: str | None,
tags: tuple[str, ...],
supersedes: tuple[str, ...],
superseded_by: tuple[str, ...],
is_latest: bool,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
if not content or not memory_id or not user_id:
raise ValueError("content, memory_id and user_id are required")
normalized_layer = MemoryLayer.from_string(layer)
if normalized_layer is MemoryLayer.L1_RAW:
raise ValueError("raw content must use the raw migration lane")
normalized_status = MemoryStatus(status.lower())
normalized_source = normalize_source(source)
existing = await self._vector_store.get_by_id(memory_id)
if existing is not None:
identity = (
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == normalized_source,
existing.layer == normalized_layer,
existing.status == normalized_status,
existing.owner == owner,
list(existing.tags or []) == list(tags),
list(existing.supersedes or []) == list(supersedes),
list(existing.superseded_by or []) == list(superseded_by),
existing.is_latest == is_latest,
)
if not all(identity):
raise ValueError("memory_id is already bound to different memory content")
await self._project_graph(existing)
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata)
custom["migration_request_id"] = request_id
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalized_source,
layer=normalized_layer,
content=content,
source_type=SourceType.EXPLICIT,
status=normalized_status,
owner=owner,
tags=list(tags),
supersedes=list(supersedes),
superseded_by=list(superseded_by),
is_latest=is_latest,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != memory_id:
raise RuntimeError("exact migration changed the requested memory identity")
await self._project_graph(node)
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
}
async def _project_graph(self, node: MemoryNode) -> None:
if self._graph_store is not None:
await self._graph_store.upsert_memory_node(node)
async def read_raw_inventory(
self,
*,
user_id: str,
agent_id: str | None = None,
session_id: str | None = None,
source: str | None = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> HySdkRawPage:
raw = await _sdk_call(
"hy_sdk_raw_list_failed",
self._loop_thread.run,
self._read_raw_inventory(
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
limit=limit,
offset=offset,
order=order,
),
)
return _raw_page(raw, limit=limit, offset=offset)
async def _read_raw_inventory(
self,
*,
user_id: str,
agent_id: str | None,
session_id: str | None,
source: str | None,
limit: int,
offset: int,
order: str,
) -> dict[str, object]:
if not user_id:
raise ValueError("user_id is required")
nodes = await self._vector_store.list_by_user(
user_id=user_id,
agent_id=agent_id,
status_filter=[MemoryStatus.ACTIVE],
layers=[MemoryLayer.L1_RAW],
include_sources=[source] if source else None,
limit=100_000,
only_latest=False,
with_vectors=False,
allow_raw_layer=True,
)
if session_id:
nodes = [node for node in nodes if node.session_id == session_id]
nodes.sort(
key=_node_timestamp,
reverse=order.lower() != "asc",
)
return {
"raw": [_raw_node(node) for node in nodes[offset : offset + limit]],
"total": len(nodes),
"limit": limit,
"offset": offset,
}
async def write_legacy_raw(
self,
*,
content: str,
raw_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_raw_import_failed",
self._loop_thread.run,
self._write_legacy_raw(
content=content,
raw_id=raw_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
)
return _successful_result(raw, "hy_sdk_raw_import_failed")
async def _write_legacy_raw(
self,
*,
content: str,
raw_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
if not content or not raw_id or not user_id:
raise ValueError("content, raw_id and user_id are required")
normalized_source = normalize_source(source)
existing = await self._vector_store.get_by_id(raw_id)
if existing is not None:
identity = (
existing.layer is MemoryLayer.L1_RAW,
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == normalized_source,
)
if not all(identity):
raise ValueError("raw_id is already bound to different source content")
return {
"success": True,
"raw_id": raw_id,
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata)
custom["migration_request_id"] = request_id
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=raw_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalized_source,
layer=MemoryLayer.L1_RAW,
content=content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != raw_id:
raise RuntimeError("exact migration changed the requested raw identity")
return {"success": True, "raw_id": raw_id, "request_id": request_id}
async def delete(self, memory_id: str) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_memory_delete_failed",
self.client.delete,
memory_id,
)
return _successful_result(raw, "hy_sdk_memory_delete_failed")
async def get_write_outcome(self, request_id: str) -> dict[str, object] | None:
raw = await _sdk_call(
"hy_sdk_write_outcome_failed",
self.client.get_write_outcome,
request_id,
)
if raw is None:
return None
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_write_outcome_shape_invalid")
return dict(raw)
def snapshot_data_plane(
*,
source_root: Path,
target_root: Path,
writers_stopped: bool,
) -> DataPlaneSnapshot:
"""Snapshot an explicitly quiesced SDK-owned local data plane."""
return snapshot_local_data_plane(
source_root=source_root,
target_root=target_root,
writers_stopped=writers_stopped,
)
def restore_data_plane(
*,
snapshot_root: Path,
target_root: Path,
writers_stopped: bool,
protective_root: Path | None = None,
) -> DataPlaneRestore:
"""Restore an explicitly quiesced SDK-owned local data plane."""
return restore_local_data_plane(
snapshot_root=snapshot_root,
target_root=target_root,
writers_stopped=writers_stopped,
protective_root=protective_root,
)
async def _sdk_call[T](
code: str,
call: Callable[..., T],
*args: object,
**kwargs: object,
) -> T:
pending = asyncio.create_task(asyncio.to_thread(call, *args, **kwargs))
try:
return await asyncio.shield(pending)
except asyncio.CancelledError:
await _finish_cancelled_call(pending)
raise
except Exception as error:
raise HySdkMigrationError(code) from error
async def _finish_cancelled_call[T](pending: asyncio.Task[T]) -> None:
while not pending.done():
try:
await asyncio.shield(pending)
except asyncio.CancelledError:
continue
except Exception:
return
if not pending.cancelled():
pending.exception()
def _memory_page(raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
rows: list[object] = []
total = 0
for bucket_name, row_name in (("vdb", "memories"), ("graph", "nodes")):
bucket = raw.get(bucket_name)
if bucket is None and bucket_name == "graph":
continue
if not isinstance(bucket, Mapping):
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
bucket_rows = bucket.get(row_name)
bucket_total = bucket.get("total")
if not isinstance(bucket_rows, list) or type(bucket_total) is not int or bucket_total < 0:
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
rows.extend(bucket_rows)
total += bucket_total
records: dict[str, HySdkMemoryRecord] = {}
for row in rows:
record = _memory_record(row)
records.setdefault(record.memory_id, record)
return HySdkMemoryPage(
memories=tuple(records.values()),
total=total,
limit=limit,
offset=offset,
)
def _memory_record(raw: object) -> HySdkMemoryRecord:
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid")
required = ("memory_id", "content", "layer", "status", "user_id", "agent_id")
if any(not isinstance(raw.get(name), str) or not raw[name] for name in required):
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid")
custom = raw.get("custom")
source_raw_ids: list[object] = []
if isinstance(custom, Mapping) and isinstance(custom.get("source_raw_memory_ids"), list):
source_raw_ids.extend(custom["source_raw_memory_ids"])
if isinstance(raw.get("source_raw_memory_id"), str):
source_raw_ids.append(raw["source_raw_memory_id"])
try:
return HySdkMemoryRecord(
memory_id=raw["memory_id"],
content=raw["content"],
layer=raw["layer"],
status=raw["status"],
is_latest=bool(raw.get("is_latest", True)),
supersedes=_string_tuple(raw.get("supersedes")),
superseded_by=_string_tuple(raw.get("superseded_by")),
tags=_string_tuple(raw.get("tags")),
owner=raw.get("owner") if raw.get("owner") in {"user", "agent"} else None,
source_raw_ids=_string_tuple(source_raw_ids),
user_id=raw["user_id"],
agent_id=raw["agent_id"],
session_id=_optional_string(raw.get("session_id")),
source=_optional_string(raw.get("source")),
memory_at=_optional_int(raw.get("memory_at")),
gmt_created=_optional_int(raw.get("gmt_created")),
)
except (TypeError, ValueError) as error:
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid") from error
def _raw_page(raw: object, *, limit: int, offset: int) -> HySdkRawPage:
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_raw_page_shape_invalid")
rows = raw.get("raw")
total = raw.get("total")
if not isinstance(rows, list) or type(total) is not int or total < 0:
raise HySdkMigrationError("hy_sdk_raw_page_shape_invalid")
return HySdkRawPage(
raw=tuple(_raw_record(row) for row in rows),
total=total,
limit=limit,
offset=offset,
)
def _node_timestamp(node: Any) -> float:
value = getattr(node, "memory_at", None) or getattr(node, "gmt_created", None)
return value.timestamp() if isinstance(value, datetime) else 0.0
def _raw_node(node: Any) -> dict[str, object]:
def timestamp(value: object) -> int | None:
return int(value.timestamp()) if isinstance(value, datetime) else None
return {
"raw_id": node.node_id,
"content": node.content,
"user_id": node.user_id,
"agent_id": node.agent_id,
"session_id": node.session_id,
"source": node.source,
"memory_at": timestamp(node.memory_at),
"gmt_created": timestamp(node.gmt_created),
"custom": node.custom or {},
}
def _raw_record(raw: object) -> HySdkRawRecord:
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid")
required = ("raw_id", "content", "user_id", "agent_id", "session_id", "source")
if any(not isinstance(raw.get(name), str) or not raw[name] for name in required):
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid")
custom = raw.get("custom")
try:
return HySdkRawRecord(
raw_id=raw["raw_id"],
content=raw["content"],
user_id=raw["user_id"],
agent_id=raw["agent_id"],
session_id=raw["session_id"],
source=raw["source"],
memory_at=_optional_int(raw.get("memory_at")),
gmt_created=_optional_int(raw.get("gmt_created")),
custom=dict(custom) if isinstance(custom, Mapping) else {},
)
except (TypeError, ValueError) as error:
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid") from error
def _successful_result(raw: object, code: str) -> dict[str, object]:
if not isinstance(raw, Mapping) or raw.get("success") is not True:
raise HySdkMigrationError(code)
return dict(raw)
def _string_tuple(value: object) -> tuple[str, ...]:
if not isinstance(value, list | tuple):
return ()
return tuple(item for item in value if isinstance(item, str) and item)
def _optional_string(value: object) -> str | None:
return value if isinstance(value, str) else None
def _optional_int(value: object) -> int | None:
return value if type(value) is int else None
@@ -0,0 +1,406 @@
"""HY SDK client construction and shared-runtime leasing.
This module deliberately depends only on the SDK's package-root exports. A
lease owns one client while the process shares one ``SharedRuntime`` per
physical MemHub data directory.
"""
from __future__ import annotations
import asyncio
import hashlib
import os
import threading
from collections.abc import Callable, Coroutine
from dataclasses import dataclass
from pathlib import Path
from typing import Any, ClassVar
from hy_memory import ( # type: ignore[import-untyped]
HyMemoryClient,
MemoryConfig,
SharedRuntime,
)
from one_memory.embedding_assets import (
BUILTIN_EMBEDDING_DIMENSIONS,
BUILTIN_EMBEDDING_MODEL,
)
from one_memory.hy_graph_admin import (
GraphWidthMismatchError,
graph_store_enabled,
inspect_graph_width_mismatch,
)
from one_memory.hy_sdk_types import HySdkRuntimeError
from one_memory.local_embedding import LocalEmbedder
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
_RUNTIME_LOCK = threading.RLock()
BUILTIN_RETRIEVAL_PROFILE_ID = "builtin-retrieval"
_LOCAL_ONNX_PLACEHOLDER_URL = "http://127.0.0.1/local-onnx"
DEFAULT_RELEVANCE_FLOOR = 0.35
@dataclass
class _RuntimeEntry:
runtime: SharedRuntime
references: int
_RUNTIMES: dict[Path, _RuntimeEntry] = {}
class LocalOnnxEmbeddingService:
"""Public ``EmbeddingService`` implementation backed by MemHub's ONNX model."""
_embedder: ClassVar[LocalEmbedder | None] = None
_embedder_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self, embedder: LocalEmbedder | None = None) -> None:
if embedder is None:
with self._embedder_lock:
if self._embedder is None:
self.__class__._embedder = LocalEmbedder()
embedder = self._embedder
if embedder is None: # pragma: no cover - guarded by the assignment above
raise HySdkRuntimeError("hy_sdk_local_embedding_unavailable")
self._local_embedder = embedder
async def embed(self, text: str, use_cache: bool = True) -> list[float]:
del use_cache
vectors = await asyncio.to_thread(self._local_embedder.encode, [text])
return vectors[0]
async def embed_with_meta(
self,
text: str,
use_cache: bool = True,
) -> tuple[list[float], dict[str, Any]]:
return await self.embed(text, use_cache), {"provider": "local-onnx"}
async def embed_batch(
self,
texts: list[str],
use_cache: bool = True,
) -> list[list[float]]:
del use_cache
if not texts:
return []
return await asyncio.to_thread(self._local_embedder.encode, list(texts))
async def embed_queued(self, text: str, use_cache: bool = True) -> list[float]:
return await self.embed(text, use_cache)
async def embed_queued_with_meta(
self,
text: str,
use_cache: bool = True,
) -> tuple[list[float], dict[str, Any]]:
return await self.embed_with_meta(text, use_cache)
def get_embedding_dims(self) -> int:
return BUILTIN_EMBEDDING_DIMENSIONS
class ClientLease:
"""Own a client and release its shared runtime exactly once."""
def __init__(self, client: HyMemoryClient, data_dir: Path) -> None:
self.client = client
self.data_dir = data_dir.resolve()
self._closed = False
self._close_lock = threading.Lock()
@property
def closed(self) -> bool:
return self._closed
def close(self) -> None:
with self._close_lock:
if self._closed:
return
self._closed = True
try:
self.client.close()
finally:
_release_runtime(self.data_dir)
def __enter__(self) -> ClientLease:
return self
def __exit__(self, *_exc: object) -> None:
self.close()
class SdkClientHandle:
"""Lazy async owner for one profile-bound public SDK client lease."""
def __init__(
self,
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> None:
self._profile = profile
self._api_key = api_key
self._data_dir = data_dir
self._http_transport_factory = http_transport_factory
self._lease: ClientLease | None = None
self._lock = asyncio.Lock()
async def client(self) -> HyMemoryClient:
if self._lease is not None:
return self._lease.client
async with self._lock:
if self._lease is None:
self._lease = await asyncio.to_thread(
acquire_client,
self._profile,
api_key=self._api_key,
data_dir=self._data_dir,
http_transport_factory=self._http_transport_factory,
)
return self._lease.client
async def close(self) -> None:
async with self._lock:
lease, self._lease = self._lease, None
if lease is not None:
await asyncio.to_thread(lease.close)
def builtin_retrieval_snapshot() -> ModelProfileSnapshot:
"""Return the model-free profile used by SDK-native read operations."""
return ModelProfileSnapshot(
profile_id=BUILTIN_RETRIEVAL_PROFILE_ID,
revision=0,
provider_kind=ProviderKind.LOCAL_ONNX,
base_url=_LOCAL_ONNX_PLACEHOLDER_URL,
model=BUILTIN_EMBEDDING_MODEL,
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_model=BUILTIN_EMBEDDING_MODEL,
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
timeout_seconds=60,
max_concurrency=1,
context_window=None,
has_api_key=False,
)
def sdk_relevance_floor() -> float:
raw = os.environ.get("ONE_MEMORY_HY_RELEVANCE_FLOOR", "").strip()
if not raw:
return DEFAULT_RELEVANCE_FLOOR
try:
value = float(raw)
except ValueError:
return DEFAULT_RELEVANCE_FLOOR
return min(1.0, max(0.0, value))
def build_sdk_config(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
) -> MemoryConfig:
"""Map a model snapshot to SDK models, endpoints, backends, and local paths."""
generation_endpoint, generation_key = _generation_access(profile, api_key)
embedding_provider = profile.embedding_provider_kind or profile.provider_kind
embedding_service_is_local = embedding_provider is ProviderKind.LOCAL_ONNX
embedding_model = profile.embedding_model or (
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else None
)
if not embedding_model:
raise HySdkRuntimeError("hy_sdk_embedding_model_required")
dimensions: int | None
if embedding_service_is_local:
embedding_endpoint = None
embedding_key = None
dimensions = profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
else:
embedding_endpoint = _openai_endpoint(
embedding_provider,
profile.embedding_base_url or profile.base_url,
)
embedding_key = "ollama" if embedding_provider is ProviderKind.OLLAMA else api_key
if embedding_provider is ProviderKind.OPENAI_COMPATIBLE and not embedding_key:
raise HySdkRuntimeError("hy_sdk_embedding_api_key_missing")
if embedding_provider not in {
ProviderKind.OPENAI_COMPATIBLE,
ProviderKind.OLLAMA,
}:
raise HySdkRuntimeError("hy_sdk_embedding_provider_unsupported")
dimensions = profile.embedding_dimensions
root = data_dir.resolve() / "hy-memory" / "governed-v2"
root.mkdir(parents=True, exist_ok=True)
collection_identity = "\0".join(
(
embedding_provider.value,
embedding_endpoint or "local",
embedding_model,
str(dimensions or "auto"),
)
)
collection = "memhub_" + hashlib.sha256(collection_identity.encode()).hexdigest()[:16]
return MemoryConfig.from_dict(
{
"llm": {
"provider": "openai",
"model": profile.model,
"api_key": generation_key,
"base_url": generation_endpoint,
},
"embedder": {
"provider": "openai",
"model": (
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else embedding_model
),
"api_key": embedding_key,
"base_url": embedding_endpoint,
"embedding_dims": dimensions,
},
"vector_store": {
"provider": "chroma",
"collection_name": collection,
"persist_directory": str(root / "vector"),
"embedding_dims": dimensions,
"host": None,
"port": None,
"api_key": None,
},
"graph_store": {
"provider": "kuzu",
"db_path": str(root / "graph"),
"url": None,
"username": None,
"password": None,
"database": None,
},
"cache": {
"backend": "sqlite",
"db_path": str(root / "cache.sqlite3"),
},
"history": {"db_path": str(root / "history.sqlite3")},
"coding": {"db_path": str(root / "coding.sqlite3")},
}
)
def acquire_client(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> ClientLease:
"""Construct a public SDK client and return its explicit runtime lease."""
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
config.http_transport_factory = http_transport_factory
mismatch = inspect_graph_width_mismatch(config)
if mismatch is not None:
raise GraphWidthMismatchError(mismatch)
key = data_dir.resolve()
runtime = _acquire_runtime(config, key)
embedding_service: LocalOnnxEmbeddingService | None = None
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
embedding_service = LocalOnnxEmbeddingService()
try:
client = HyMemoryClient(
config=config,
mode="ultra" if graph_store_enabled() else "pro",
runtime=runtime,
embedding_service=embedding_service,
)
except BaseException:
_release_runtime(key)
raise
return ClientLease(client, key)
def close_all_shared_runtimes() -> None:
"""Close process runtimes only after every client lease has been released."""
with _RUNTIME_LOCK:
if _RUNTIMES:
raise HySdkRuntimeError("hy_sdk_runtime_leases_active")
def _generation_access(
profile: ModelProfileSnapshot,
api_key: str | None,
) -> tuple[str, str]:
if profile.provider_kind is ProviderKind.LOCAL_ONNX:
return _LOCAL_ONNX_PLACEHOLDER_URL, "local-read-only"
if profile.provider_kind not in {
ProviderKind.OPENAI_COMPATIBLE,
ProviderKind.OLLAMA,
}:
raise HySdkRuntimeError("hy_sdk_generation_provider_unsupported")
endpoint = _openai_endpoint(profile.provider_kind, profile.base_url)
if profile.provider_kind is ProviderKind.OLLAMA:
return endpoint, "ollama"
if not api_key:
raise HySdkRuntimeError("hy_sdk_model_api_key_missing")
return endpoint, api_key
def _openai_endpoint(provider: ProviderKind, endpoint: str | None) -> str:
value = (endpoint or "").rstrip("/")
if not value:
raise HySdkRuntimeError("hy_sdk_model_endpoint_missing")
if provider is ProviderKind.OLLAMA and not value.endswith("/v1"):
return f"{value}/v1"
return value
def _acquire_runtime(config: MemoryConfig, key: Path) -> SharedRuntime:
with _RUNTIME_LOCK:
entry = _RUNTIMES.get(key)
if entry is None:
runtime = _run_sync(SharedRuntime.create(config))
_RUNTIMES[key] = _RuntimeEntry(runtime=runtime, references=1)
return runtime
entry.references += 1
return entry.runtime
def _release_runtime(key: Path) -> None:
runtime: SharedRuntime | None = None
with _RUNTIME_LOCK:
entry = _RUNTIMES.get(key)
if entry is None:
return
entry.references -= 1
if entry.references == 0:
runtime = entry.runtime
del _RUNTIMES[key]
if runtime is not None:
runtime.close()
def _run_sync[T](coroutine: Coroutine[Any, Any, T]) -> T:
"""Run SDK async construction from either sync or async caller threads."""
try:
asyncio.get_running_loop()
except RuntimeError:
return asyncio.run(coroutine)
result: list[T] = []
failure: list[BaseException] = []
def run() -> None:
try:
result.append(asyncio.run(coroutine))
except BaseException as error:
failure.append(error)
thread = threading.Thread(target=run, name="hy-sdk-runtime-create")
thread.start()
thread.join()
if failure:
raise failure[0]
return result[0]
@@ -0,0 +1,402 @@
"""Validated records and response parsers for public HY SDK operations."""
from __future__ import annotations
from collections.abc import Mapping
from pydantic import BaseModel, ConfigDict, Field
from one_memory.semantic_deadline import SemanticMemoryHit
class HySdkError(RuntimeError):
"""Base error carrying a stable, product-facing code."""
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
class HySdkRuntimeError(HySdkError):
"""The public SDK runtime could not be configured or leased."""
class HySdkMigrationError(HySdkError):
"""The privileged offline SDK data-plane migration failed."""
class HySdkReadError(HySdkError):
"""A public SDK read operation failed or returned an invalid shape."""
class HySdkMemoryRecord(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
memory_id: str
content: str
layer: str
status: str
is_latest: bool
supersedes: tuple[str, ...] = ()
superseded_by: tuple[str, ...] = ()
tags: tuple[str, ...] = ()
owner: str | None = None
source_raw_ids: tuple[str, ...] = ()
user_id: str
agent_id: str
session_id: str | None = None
source: str | None = None
memory_at: int | None = None
gmt_created: int | None = None
@classmethod
def from_sdk(cls, raw: object) -> HySdkMemoryRecord:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_record_shape_unknown")
if any(
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
for key in ("memory_id", "content")
):
raise HySdkReadError("hy_memory_record_shape_unknown")
try:
return cls(
memory_id=str(raw["memory_id"]),
content=str(raw["content"]),
layer=str(raw.get("layer") or ""),
status=str(raw.get("status") or ""),
is_latest=bool(raw.get("is_latest", True)),
supersedes=_string_tuple(raw.get("supersedes")),
superseded_by=_string_tuple(raw.get("superseded_by")),
tags=_string_tuple(raw.get("tags")),
owner=(str(raw["owner"]) if raw.get("owner") in {"user", "agent"} else None),
source_raw_ids=_source_raw_ids(raw),
user_id=str(raw.get("user_id") or ""),
agent_id=str(raw.get("agent_id") or ""),
session_id=(
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
gmt_created=(
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
),
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
class HySdkMemoryPage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
memories: tuple[HySdkMemoryRecord, ...]
total: int = Field(ge=0)
limit: int = Field(ge=1)
offset: int = Field(ge=0)
@classmethod
def from_sdk(cls, raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_list_shape_unknown")
rows: list[object] = []
total = 0
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
bucket = raw.get(bucket_name)
if bucket is None and bucket_name == "graph":
continue
if not isinstance(bucket, Mapping) or not isinstance(bucket.get(rows_name), list):
raise HySdkReadError("hy_memory_list_shape_unknown")
bucket_total = bucket.get("total")
if (
not isinstance(bucket_total, int)
or isinstance(bucket_total, bool)
or bucket_total < 0
):
raise HySdkReadError("hy_memory_list_shape_unknown")
total += bucket_total
rows.extend(bucket[rows_name])
deduplicated: dict[str, HySdkMemoryRecord] = {}
for row in rows:
record = HySdkMemoryRecord.from_sdk(row)
deduplicated.setdefault(record.memory_id, record)
return cls(
memories=tuple(deduplicated.values()),
total=total,
limit=limit,
offset=offset,
)
class HySdkRawRecord(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
raw_id: str
content: str
user_id: str
agent_id: str
session_id: str
source: str
memory_at: int | None = None
gmt_created: int | None = None
custom: dict[str, object] = Field(default_factory=dict)
@classmethod
def from_sdk(cls, raw: object) -> HySdkRawRecord:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_shape_unknown")
if any(
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
for key in ("raw_id", "content", "user_id", "agent_id", "session_id", "source")
):
raise HySdkReadError("hy_memory_raw_shape_unknown")
custom = raw.get("custom")
if not isinstance(custom, Mapping):
custom = {}
try:
return cls(
raw_id=str(raw["raw_id"]),
content=str(raw["content"]),
user_id=str(raw["user_id"]),
agent_id=str(raw["agent_id"]),
session_id=str(raw["session_id"]),
source=str(raw["source"]),
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
gmt_created=(
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
),
custom=dict(custom),
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_raw_shape_unknown") from error
class HySdkRawHit(HySdkRawRecord):
score: float = Field(ge=0, le=1)
@classmethod
def from_sdk_search(cls, raw: object) -> HySdkRawHit:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
raw_id = raw.get("memory_id")
score = raw.get("score")
if (
not isinstance(raw_id, str)
or not isinstance(score, int | float)
or isinstance(score, bool)
):
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
base = HySdkRawRecord.from_sdk({**raw, "raw_id": raw_id})
return cls(
**base.model_dump(),
score=max(0.0, min(1.0, float(score))),
)
class HySdkRawPage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
raw: tuple[HySdkRawRecord, ...]
total: int = Field(ge=0)
limit: int = Field(ge=1)
offset: int = Field(ge=0)
@classmethod
def from_memory_list(
cls,
raw: object,
*,
limit: int,
offset: int,
) -> HySdkRawPage:
"""Parse ``list_memories(layers=['l1_raw'])`` into the App raw DTO."""
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
bucket = raw.get("vdb")
if not isinstance(bucket, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
rows = bucket.get("memories")
total = bucket.get("total")
if (
not isinstance(rows, list)
or not isinstance(total, int)
or isinstance(total, bool)
or total < 0
):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted = []
for row in rows:
if not isinstance(row, Mapping) or row.get("layer") != "l1_raw":
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted.append({**row, "raw_id": row.get("memory_id")})
return cls(
raw=tuple(HySdkRawRecord.from_sdk(row) for row in converted),
total=total,
limit=limit,
offset=offset,
)
class HySdkUserProfile(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
request_id: str
profile: dict[str, object] | None
elapsed_ms: float = Field(ge=0)
@classmethod
def from_sdk(cls, raw: object) -> HySdkUserProfile:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_profile_shape_unknown")
request_id = raw.get("request_id")
profile = raw.get("profile")
elapsed_ms = raw.get("elapsed_ms")
if (
not isinstance(request_id, str)
or (profile is not None and not isinstance(profile, Mapping))
or not isinstance(elapsed_ms, int | float)
or isinstance(elapsed_ms, bool)
):
raise HySdkReadError("hy_memory_profile_shape_unknown")
return cls(
request_id=request_id,
profile=None if profile is None else dict(profile),
elapsed_ms=float(elapsed_ms),
)
def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
rows = _reply_rows(raw, error_code)
found: list[SemanticMemoryHit] = []
seen: set[str] = set()
for item in sorted(rows, key=_score, reverse=True):
memory_id = item.get("memory_id") or item.get("id")
if not isinstance(memory_id, str) or not memory_id or memory_id in seen:
continue
seen.add(memory_id)
content = item.get("content")
member_ids, formatting_complete = _formatted_member_ids(item)
found.append(
SemanticMemoryHit(
memory_id=memory_id,
score=_score(item),
formatted_content=(
content.strip() if isinstance(content, str) and content.strip() else None
),
member_ids=member_ids,
sdk_formatted=True,
formatting_complete=formatting_complete,
)
)
return tuple(found)
def parse_sdk_raw_search(raw: object) -> tuple[HySdkRawHit, ...]:
return tuple(
HySdkRawHit.from_sdk_search(row)
for row in _reply_rows(raw, "hy_memory_raw_search_shape_unknown")
)
def _string_tuple(value: object) -> tuple[str, ...]:
if not isinstance(value, list):
return ()
return tuple(
dict.fromkeys(item.strip() for item in value if isinstance(item, str) and item.strip())
)
def _source_raw_ids(raw: Mapping[str, object]) -> tuple[str, ...]:
custom = raw.get("custom")
values: list[object] = []
if isinstance(custom, Mapping):
listed = custom.get("source_raw_memory_ids")
if isinstance(listed, list):
values.extend(listed)
singular = raw.get("source_raw_memory_id")
if isinstance(singular, str):
values.append(singular)
return tuple(
dict.fromkeys(value.strip() for value in values if isinstance(value, str) and value.strip())
)
def _reply_rows(raw: object, error_code: str) -> tuple[Mapping[str, object], ...]:
if not isinstance(raw, Mapping):
raise HySdkReadError(error_code)
keys = tuple(key for key in ("memories", "coding_memories") if key in raw)
if "scene" in raw:
scene = raw["scene"]
if not isinstance(scene, str):
raise HySdkReadError(error_code)
if scene in ("normal", "chat"):
selected = "memories"
elif scene in ("productivity", "working"):
selected = "coding_memories"
else:
raise HySdkReadError(error_code)
if selected not in keys:
raise HySdkReadError(error_code)
elif len(keys) == 1:
selected = keys[0]
else:
raise HySdkReadError(error_code)
result: list[Mapping[str, object]] = []
for key in keys:
value = raw[key]
if key == "memories" and isinstance(value, Mapping):
if set(value) != {"schema", "proactive", "normal"}:
raise HySdkReadError(error_code)
channels = tuple(value.values())
else:
channels = (value,)
for channel in channels:
if not isinstance(channel, list) or any(
not isinstance(row, Mapping) for row in channel
):
raise HySdkReadError(error_code)
if key == selected:
result.extend(channel)
return tuple(result)
def _score(memory: Mapping[str, object]) -> float:
score = memory.get("score")
return float(score) if isinstance(score, int | float) and not isinstance(score, bool) else 0.0
def _formatted_member_ids(memory: Mapping[str, object]) -> tuple[tuple[str, ...], bool]:
represented: dict[str, None] = {}
composite = False
raw_members = memory.get("member_ids")
if raw_members is not None:
composite = True
if not isinstance(raw_members, list | tuple) or any(
not isinstance(member_id, str) or not member_id for member_id in raw_members
):
return (), False
for member_id in raw_members:
represented.setdefault(member_id, None)
chain = memory.get("evolution_chain")
if chain is not None:
composite = True
if not isinstance(chain, list | tuple):
return (), False
for entry in chain:
if not isinstance(entry, Mapping):
return (), False
member_id = entry.get("node_id") or entry.get("memory_id")
if not isinstance(member_id, str) or not member_id:
return (), False
represented.setdefault(member_id, None)
if memory.get("capsule_memories") is not None or memory.get("timeline") is not None:
composite = True
if raw_members is None:
return (), False
content = memory.get("content")
if isinstance(content, str) and content.lstrip().startswith(
("Related memories:", "Matched memories:", "Memory timeline:")
):
composite = True
if not represented:
return (), False
return tuple(represented), not composite or bool(represented)
@@ -1,153 +0,0 @@
"""Per-write original Evidence and where each HY memory came from; never a memory source."""
import json
from collections.abc import Set as AbstractSet
from importlib import import_module
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.input_sanitization import sanitize_model_input
class HySupportedMemory(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
hy_memory_id: str
governed_memory_id: str
revision: int
content: str
owner: Literal["user", "agent"]
evidence_ids: tuple[str, ...]
class HySourceContext(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
current: tuple[EvidenceRecord, ...]
history: tuple[EvidenceRecord, ...] = ()
supporting: tuple[EvidenceRecord, ...] = ()
known_memories: tuple[HySupportedMemory, ...] = ()
class HyWriteBindings:
def __init__(self, context: HySourceContext) -> None:
if not context.current:
raise ValueError("hy_memory_write_verification_context_scope")
self.context = context
seed = context.current[0]
records = (*context.current, *context.history, *context.supporting)
workspaces = {record.workspace for record in records}
# A missing workspace is a legacy identity, not a wildcard that may
# absorb a named project. Keep NULL and named workspaces disjoint.
if None in workspaces and len(workspaces) != 1:
raise ValueError("hy_memory_write_verification_context_scope")
ids = [record.evidence_id for record in records]
if len(ids) != len(set(ids)):
raise ValueError("hy_memory_write_verification_context_scope")
for record in records:
if (
(
record.user_id,
record.source_id,
record.conversation_id,
record.workspace,
)
!= (
seed.user_id,
seed.source_id,
seed.conversation_id,
seed.workspace,
)
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
or record.role not in {"user", "assistant", "tool"}
):
raise ValueError("hy_memory_write_verification_context_scope")
self.all_sources: list[dict[str, Any]] = [
{
"id": record.evidence_id,
"role": record.role,
"content": sanitize_model_input(record.verbatim),
"new": index < len(context.current),
"revision": record.revision,
"created_at": record.created_at.isoformat() if record.created_at else None,
}
for index, record in enumerate(records)
]
self.sources = self.all_sources[: len(context.current) + len(context.history)]
self._supports: dict[tuple[str, str], tuple[str, ...]] = {}
self.profile_order: list[str] | None = None
self.verified_operations: dict[str, tuple[str, str]] = {}
self.known = {memory.hy_memory_id: memory for memory in context.known_memories}
for memory in context.known_memories:
if not memory.evidence_ids or not set(memory.evidence_ids) <= set(ids):
raise ValueError("hy_memory_write_verification_context_scope")
self._supports[(memory.content, memory.owner)] = memory.evidence_ids
def check_content(self, content: str) -> None:
try:
parsed = json.loads(content)
except ValueError as error:
raise ValueError("hy_memory_write_verification_input_mismatch") from error
expected = [
{"role": record.role, "content": sanitize_model_input(record.verbatim)}
for record in self.context.current
]
# HY strips tool messages itself before chat extraction when it judges a turn chat.
without_tools = [item for item in expected if item["role"] != "tool"]
if parsed != expected and (without_tools == expected or parsed != without_tools):
raise ValueError("hy_memory_write_verification_input_mismatch")
def history_json(self) -> str:
return json.dumps(
[
{
"evidence_id": source["id"],
**{k: source[k] for k in ("role", "content", "revision", "created_at")},
}
for source in self.sources
if not source["new"]
],
ensure_ascii=False,
)
def turn_support(
self, owner: str | None, targets: AbstractSet[str] = frozenset()
) -> tuple[str, ...]:
"""This turn's words by the memory's author, plus what a merged memory already cited."""
roles = {"user": {"user"}, "agent": {"assistant"}}.get(owner or "")
current = [source for source in self.all_sources if source["new"]]
ids = [source["id"] for source in current if roles is None or source["role"] in roles]
if not ids:
ids = [source["id"] for source in current]
for target in sorted(targets):
if target in self.known:
ids.extend(self.known[target].evidence_ids)
return tuple(dict.fromkeys(ids))
def record_turn_supports(self, items: list[dict[str, Any]]) -> None:
for item in items:
key = (item["content"], item["owner"] or "")
self._supports.setdefault(key, self.turn_support(item["owner"]))
def record_operation_support(self, content: str, owner: str | None, targets: set[str]) -> None:
key = (content, owner or "")
merged = (*self._supports.get(key, ()), *self.turn_support(owner, targets))
self._supports[key] = tuple(dict.fromkeys(merged))
def support_for(self, content: str, owner: str | None) -> tuple[str, ...]:
return self._supports.get((content, owner or ""), ())
def profile_support(self, content: str, values: dict[str, Any]) -> tuple[str, ...]:
render = import_module("hy_memory.agent.tools.basic_profile")._render_content
if not values or render(values, field_order=self.profile_order) != content:
return ()
ids: dict[str, None] = {}
for key, value in values.items():
support = self.support_for(f"{key}: {value}", "user")
if not support:
return ()
ids.update(dict.fromkeys(support))
return tuple(ids)
@@ -1,53 +0,0 @@
"""Input checks around HY's native extractor; HY's output is published as it returns it."""
import asyncio
from typing import Any
from one_memory.hy_source_context import HyWriteBindings
def _extracted_items(info: dict[str, Any]) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
for section in ("memory", "intentions"):
for item in info.get(section) or []:
if isinstance(item, dict) and isinstance(item.get("content"), str):
items.append({"content": item["content"], "owner": item.get("owner")})
for name, value in (info.get("basic_info") or {}).items():
items.append({"content": f"{name}: {value}", "owner": "user"})
return items
class HyExtractionGuard:
def __init__(self, extractor: Any) -> None:
self.extractor = extractor
self.failures: list[str] = []
self._lock = asyncio.Lock()
self.bindings: HyWriteBindings | None = None
def __getattr__(self, name: str) -> Any:
return getattr(self.extractor, name)
async def extract(self, *args: Any, **kwargs: Any) -> Any:
async with self._lock:
try:
return await self._extract(args, kwargs)
except ValueError as error:
if str(error).startswith("hy_memory_write_verification"):
self.failures.append(str(error))
raise
async def _extract(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
content = kwargs.get("content", args[0] if args else "")
bindings = self.bindings
if bindings is not None:
bindings.check_content(content)
fields = kwargs.get("basic_profile_fields")
bindings.profile_order = list(fields) if fields else None
kwargs = {**kwargs, "history_context": bindings.history_json()}
else:
# No raw-history identity was supplied; do not trust HY's text prefixes.
kwargs = {**kwargs, "history_context": ""}
result = await self.extractor.extract(*args, **kwargs)
if bindings is not None and result.success:
bindings.record_turn_supports(_extracted_items(result.info))
return result
@@ -6,6 +6,7 @@ from pathlib import PurePosixPath
from pydantic import JsonValue
from one_memory.derived_echo import strip_injected_memory
from one_memory.imports.access import (
AuthorizedSourceFile,
AuthorizedSourceRoot,
@@ -213,6 +214,7 @@ def parse_hermes_rollout_record(
message.get("content"),
allowed_block_types=frozenset({"text", "input_text", "output_text"}),
)
text = strip_injected_memory(text)
if not text:
return ParsedSkip(
reason_code=ScanIssueCode.NO_TEXT,
@@ -386,7 +388,8 @@ def _read_state_db(
# The assistant's call sits on its own row as `tool_calls`; the
# `tool` rows after it hold only what came back.
calls = _tool_calls_text(row["tool_calls"]) if role == "assistant" else None
has_text = bool(str(row["content"] or "").strip())
content = strip_injected_memory(str(row["content"] or ""))
has_text = bool(content.strip())
if role not in {"user", "assistant"} or (not has_text and calls is None):
yield ScanSkip(
cursor=cursor,
@@ -401,7 +404,7 @@ def _read_state_db(
role=str(role) if has_text else "tool",
written_by=_written_by(role, row["session_source"]) if has_text else WrittenBy.TOOL,
created_at=parse_timestamp(row["timestamp"]),
verbatim=str(row["content"]) if has_text else str(calls),
verbatim=content if has_text else str(calls),
workspace=workspace,
project=safe_project_label(workspace),
record_type="state_db_message" if has_text else "state_db_message.tool",
@@ -0,0 +1,35 @@
"""Passive lifecycle projection used only by retired Governance readers."""
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass, field
from types import MappingProxyType
@dataclass(frozen=True, eq=False)
class HyLifecycle:
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
def __post_init__(self) -> None:
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
def __eq__(self, other: object) -> bool:
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(
other.hidden
)
def __hash__(self) -> int:
return hash(tuple(sorted(self.hidden.items())))
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
return self.hidden.get(memory_id, ())
def replacing(self, memory_id: str) -> tuple[str, ...]:
return tuple(
sorted(
old
for old, replacements in self.hidden.items()
if memory_id in replacements
)
)
@@ -224,7 +224,7 @@ def record_capture_rejection(
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
if not changed or not _sync_open(store.database_path, record.source_id):
return
from one_memory.memory_processing import _conversation_chunks
from one_memory.conversation_processing import _conversation_chunks
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
@@ -121,7 +121,7 @@ class LocalEmbedder:
raise MissingEmbeddingAssetsError(f"{_FETCH_HINT} Looked in {directory}.")
# Imported by name for the same reason kuzu and hy_memory are in
# hy_memory_bridge: onnxruntime ships no py.typed, and a dynamic
# onnxruntime ships no py.typed, and a dynamic
# import keeps the strict checker on everything around it.
ort = import_module("onnxruntime")
from tokenizers import Tokenizer
@@ -22,7 +22,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.database import enable_write_ahead_log
from one_memory.database_access import main_database
from one_memory.hy_lifecycle import HyLifecycle
from one_memory.legacy_lifecycle_types import HyLifecycle
from one_memory.library_upgrade import LibraryUpgradeError, PreparedLibrary, holds_legacy_rows
from one_memory.memory_server.migrations import governance_ddl
@@ -10,7 +10,7 @@ from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
from one_memory.database_access import main_database
from one_memory.hy_lifecycle import HyLifecycle
from one_memory.legacy_lifecycle_types import HyLifecycle
from one_memory.memory_evidence_freshness import register_memory_freshness
from one_memory.memory_governance import (
GovernedMemory,
@@ -3,7 +3,7 @@ import json
import sqlite3
from collections.abc import Sequence
from datetime import UTC, datetime
from typing import Literal, cast
from typing import Any, Literal, cast
from uuid import UUID
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
@@ -21,11 +21,10 @@ from one_memory.imports.models import (
)
from one_memory.imports.storage import EvidenceStore
from one_memory.input_sanitization import sanitize_model_input
from one_memory.memory_governance import MemoryStatus
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.secrets import SecretStoreUnavailableError
SOURCE_ID = "one-memory-manual"
@@ -82,11 +81,23 @@ class SavedMemoryNoteEvidence(BaseModel):
verbatim: str
class LegacyMemoryNoteReader:
"""Read pre-cutover note receipts without retaining any write runtime."""
def __init__(
self,
tasks: MemoryTaskStore,
governance: MemoryGovernanceStore,
) -> None:
self.tasks = tasks
self.governance = governance
class MemoryNoteStore:
def __init__(
self,
evidence: EvidenceStore,
processor: MemoryProcessingSupervisor,
processor: Any | None,
intake: UnifiedWriteIntake | None = None,
*,
sdk_authority: bool = False,
@@ -252,6 +263,8 @@ class MemoryNoteStore:
memory_ids=(list(durable.sdk_memory_ids) if durable is not None else []),
error=durable.detail if durable is not None else "write_missing",
)
if self.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
task = self.processor.tasks.task_for_evidence(evidence_id)
memory_ids = []
if task:
@@ -356,6 +369,8 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
try:
if notes.sdk_authority:
return await asyncio.to_thread(notes.receipt, record.evidence_id)
if notes.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
if notes.intake is None:
# Isolated tests construct the store without a queue.
await notes.processor.enqueue_conversation_off_thread(record)
File diff suppressed because it is too large. Load diff
@@ -69,8 +69,8 @@ _TYPE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$")
def safe_error_code(error: BaseException) -> str:
"""Reduce an exception to a storable code, never its message text.
A recognised code (the message *is* the code, as `HyMemoryBridgeError`
and the permission guard raise them) is kept so callers can classify it.
A recognised code (the message *is* a stable domain code) is kept so
callers can classify it.
Everything else collapses to its exception type, which is a fixed
identifier rather than content.
"""
@@ -239,8 +239,8 @@ PERMISSION_REQUIRED_CODES = frozenset(
}
)
#: Model-side codes `process_records` raises *before* it can open the bridge
#: (no active profile, missing credential, secure storage down). They prove
#: Model-side codes the SDK write coordinator raises before it can lease a
#: client (no active profile, missing credential, secure storage down). They prove
#: nothing was sent, so the write parks instead of being lost. Transport codes
#: that can arrive after a request was in flight stay `unknown`.
MODEL_NOT_READY_CODES = frozenset(
@@ -251,9 +251,9 @@ MODEL_NOT_READY_CODES = frozenset(
}
)
#: Deterministic, pre-bridge refusals: the frozen input no longer matches
#: Deterministic, pre-SDK refusals: the frozen input no longer matches
#: current Evidence, or a turn exceeds the model's input window. Both are
#: raised while validating the records, before any bridge or HY call, so
#: raised while validating the records, before any SDK call, so
#: nothing was sent and retrying the same intent changes nothing — a terminal
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
#: write) and not a retry (which would fail identically).
@@ -358,12 +358,10 @@ class UnknownWriteJob:
`request_id` is the durable queue's idempotency key. It identifies the row
for queue inspection and is the value `DurableWriteQueue.get()` accepts.
It is **not** the HY request id: `MemoryProcessingSupervisor` derives the
HY handle from the task it creates, so an inquiry implementation must
obtain that task-layer handle separately rather than passing this value to
HY. This is discovery only: it does not decide whether HY has the write,
and it deliberately exposes no transition method that could turn
`unknown` back into retryable work.
It is the public SDK request id used for exact outcome inquiry. This is
discovery only: it does not decide whether HY has the write, and it
deliberately exposes no transition method that could turn `unknown` back
into retryable work.
"""
job_id: str
@@ -1759,8 +1757,8 @@ class DurableWriteQueue:
"""Retire an `unknown` job after an inquiry, never by guessing.
`unknown` means HY may already hold the write; the only safe way out is
to ask by request id (`hy_capabilities` measured that the SDK supports
this) and then record what the inquiry found. `found=True` closes it as
to call the public SDK `get_write_outcome(request_id)` and then record
what the inquiry found. `found=True` closes it as
`completed`; `found=False` gives up and marks it `dead`. The state
machine's legal transitions are the authority, so this refuses any
change that `job_state.next_state` would reject — in particular it can
@@ -2258,12 +2256,11 @@ async def execute_job(
admit_hy_invocation: Callable[[str, Callable[[], Awaitable[bool]]], Awaitable[bool]]
| None = None,
) -> None:
"""Run one durable job through the existing authorized processing path.
"""Run one durable job through the SDK-authority processor.
The executor does not call HY directly: `process_records` owns the
permission guard, the runner and governance. `hy_invoked` is recorded
before that call, so a crash mid-write is later reclaimed to `unknown`
instead of being retried blindly.
``process_sdk_job`` owns the one public SDK write. ``hy_invoked`` is
recorded immediately before that call, so a crash mid-write is later
reclaimed to ``unknown`` instead of being retried blindly.
"""
# Every store call here is synchronous SQLite work on the single writer or
# a reader; made from the loop it would block health checks and the API for
@@ -2336,38 +2333,13 @@ async def execute_job(
try:
level = ServiceLevel.REALTIME if request.priority == "live" else ServiceLevel.BACKGROUND
with service_level_context(level):
authority = getattr(supervisor, "sdk_authority", None)
if callable(authority) and authority():
sdk_result = await supervisor.process_sdk_job(
records,
request_id=request.idempotency_key,
before_hy_invocation=mark_hy_invocation,
)
raw_ids = sdk_result.raw_ids
memory_ids = tuple(memory.memory_id for memory in sdk_result.memories)
else:
completed_tasks = (
await supervisor.process_records(
records,
before_hy_invocation=mark_hy_invocation,
source_admitted=True,
)
or ()
)
receipts = []
for task in completed_tasks:
receipt = await in_the_background(supervisor.tasks.sdk_receipt, task.task_id)
if receipt is None:
raise RuntimeError("sdk_write_receipt_missing")
receipts.append(receipt)
raw_ids = tuple(
dict.fromkeys(raw_id for receipt in receipts for raw_id in receipt.raw_ids)
)
memory_ids = tuple(
dict.fromkeys(
memory_id for receipt in receipts for memory_id in receipt.memory_ids
)
)
sdk_result = await supervisor.process_sdk_job(
records,
request_id=request.idempotency_key,
before_hy_invocation=mark_hy_invocation,
)
raw_ids = sdk_result.raw_ids
memory_ids = sdk_result.memory_ids
except Exception as error: # noqa: BLE001 - classified below
# Never persist the raw message: it may be a provider response or the
# user's own text, and `last_error` is shown over HTTP as `detail`.
@@ -2382,7 +2354,7 @@ async def execute_job(
):
await in_the_background(queue.release_unstarted, job_id, owner=owner)
return
# The task layer classifies these before any bridge is opened, so HY
# The write coordinator classifies these before any SDK client is leased, so HY
# was provably not called. They must not read as `failed`:
# - authorization stop -> cancelled (the user turned it off);
# - no permission yet -> waiting_for_permission (parked, re-admitted
@@ -2445,13 +2417,10 @@ async def execute_job(
sdk_raw_ids=raw_ids,
sdk_memory_ids=memory_ids,
)
authority = getattr(supervisor, "sdk_authority", None)
release = getattr(evidence, "release_sdk_staging", None)
durable_receipt = await in_the_background(queue.sdk_receipt, job_id)
if (
callable(authority)
and authority()
and callable(release)
callable(release)
and durable_receipt is not None
and (durable_receipt.raw_ids or durable_receipt.memory_ids)
):
@@ -1,182 +0,0 @@
"""A0-03: measured answer to "can a HY write be made idempotent?".
The result decides whether the `unknown` job state can ever be retired, so it
is probed against the installed SDK rather than written down from memory.
What is verified against `hy-memory==2.0.5.dev3`:
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
atomically claims it in the SDK cache before invoking a write. Replays return
the durable terminal result; running/unknown claims are not invoked again.
- `HyMemoryClient.get_write_outcome(request_id)` reads that durable outcome
directly; it does not depend on optional pipeline tracing.
A first pass at this probe got both answers wrong, and the failure mode is
worth keeping in mind: searching SDK source for the string "idempotent" hits a
docstring about *episode* nodes ("calls are idempotent") and a "dedup" comment
about search over-sampling. Neither is write idempotency. So the probe checks
the `add` path specifically and names the inspector class explicitly, instead
of grepping for a word that appears in unrelated prose.
Net: MemHub can query and settle an uncertain request without blind replay.
`unknown` remains a legitimate operational state while the SDK claim itself is
non-terminal, but the same request id cannot write twice.
"""
from __future__ import annotations
import importlib
import inspect
from enum import StrEnum
from typing import Any
class HyIdempotencySupport(StrEnum):
"""Whether the SDK can dedupe a replayed write."""
#: A caller key is honoured and a replay is rejected or folded together.
KEY = "key"
#: Same content is folded by the engine, without an explicit key.
CONTENT_HASH = "content_hash"
#: A replay writes a second time. Retrying after a crash duplicates.
NONE = "none"
#: Never probed. Must not survive A0-03.
UNKNOWN = "unknown"
class HyResultLookupSupport(StrEnum):
"""Whether the outcome of a past write can be queried afterwards."""
BY_REQUEST_ID = "by_request_id"
BY_MEMORY_ID_ONLY = "by_memory_id_only"
NONE = "none"
UNKNOWN = "unknown"
def _module(name: str) -> Any | None:
try:
return importlib.import_module(name)
except ImportError:
return None
def _has_callable(obj: Any, name: str) -> bool:
return obj is not None and callable(getattr(obj, name, None))
def hy_write_accepts_request_id() -> bool:
"""True when the write path carries a caller-supplied request id."""
client_mod = _module("hy_memory.client")
if client_mod is None:
return False
client = getattr(client_mod, "HyMemoryClient", None)
if client is None:
return False
add = getattr(client, "add", None)
if add is None:
return False
return "request_id" in inspect.signature(add).parameters
def _add_path_declares_idempotency() -> bool:
"""Whether the *write* path de-duplicates, judged on its own source.
Deliberately scoped to `add`/`async_add`. A whole-module search matches
unrelated prose: "calls are idempotent" describes episode-node upserts, and
a "dedup" comment describes search over-sampling. Both are about other
features and would make this probe answer `key` for the wrong reason.
"""
client_mod = _module("hy_memory.client")
if client_mod is None:
return False
client = getattr(client_mod, "HyMemoryClient", None)
if client is None:
return False
sources: list[str] = []
for name in ("add", "async_add"):
func = getattr(client, name, None)
if func is None:
continue
try:
sources.append(inspect.getsource(func).lower())
except (OSError, TypeError):
continue
if not sources:
return False
if _has_callable(client, "get_write_outcome"):
claim = getattr(client, "_claim_write_outcome", None)
if claim is not None:
return True
# Older SDKs may expose an explicit key under one of these names.
markers = ("idempotency_key", "idempotent_key", "dedupe_key", "dedup_key")
return any(marker in source for source in sources for marker in markers)
def _inspector_can_lookup_by_request_id() -> bool:
"""True when pipeline logs can be queried by the write's request id."""
inspector = _module("hy_memory.inspector")
if inspector is None:
return False
for cls_name in ("MemoryInspector", "Inspector", "PipelineInspector"):
cls = getattr(inspector, cls_name, None)
if cls is None:
continue
for method in ("get_pipeline_logs", "async_get_pipeline_logs"):
func = getattr(cls, method, None)
if func is None:
continue
# The parameter is `request_id` on `MemoryInspector`; older
# docstrings call the same thing `n`. Accept either so a rename
# does not silently downgrade this to "no lookup available".
if {"request_id", "n"} & set(inspect.signature(func).parameters):
return True
return False
def _probe_idempotency() -> HyIdempotencySupport:
if not hy_write_accepts_request_id():
# No handle at all: not even a correlation id to investigate later.
return HyIdempotencySupport.NONE
if _add_path_declares_idempotency():
return HyIdempotencySupport.KEY
# A request_id is present but nothing de-duplicates on it.
return HyIdempotencySupport.NONE
def _probe_result_lookup() -> HyResultLookupSupport:
client_mod = _module("hy_memory.client")
client = getattr(client_mod, "HyMemoryClient", None) if client_mod is not None else None
if _has_callable(client, "get_write_outcome"):
return HyResultLookupSupport.BY_REQUEST_ID
if _inspector_can_lookup_by_request_id():
return HyResultLookupSupport.BY_REQUEST_ID
if _has_callable(client, "get"):
return HyResultLookupSupport.BY_MEMORY_ID_ONLY
return HyResultLookupSupport.NONE
IDEMPOTENCY_SUPPORT: HyIdempotencySupport = _probe_idempotency()
RESULT_LOOKUP_SUPPORT: HyResultLookupSupport = _probe_result_lookup()
def probe_hy_write_idempotency() -> dict[str, Any]:
"""Full probe report, for STATUS evidence and for debugging a bumped SDK.
Re-probed on every call so a dependency upgrade that changes the answer is
visible, rather than being frozen at import time.
"""
client_mod = _module("hy_memory.client")
client = getattr(client_mod, "HyMemoryClient", None)
add = getattr(client, "add", None)
return {
"sdk_importable": client_mod is not None,
"add_signature": str(inspect.signature(add)) if add is not None else "",
"accepts_request_id": hy_write_accepts_request_id(),
"add_declares_idempotency": _add_path_declares_idempotency(),
"durable_write_outcome": (
client is not None and _has_callable(client, "get_write_outcome")
),
"inspector_lookup_by_request_id": _inspector_can_lookup_by_request_id(),
"idempotency": _probe_idempotency().value,
"result_lookup": _probe_result_lookup().value,
}
@@ -9,9 +9,8 @@ Evidence→queue crash window stayed open.
This adapter is deliberately small. It turns one committed Evidence record
into the one-record complete-turn request, submits it once under an evidence
derived idempotency key, and returns the durable record. The existing
``DurableWriteDrain`` remains the only consumer; it calls
``process_records``, which preserves the complete-turn/debounce/settle rules
in ``MemoryProcessingSupervisor``.
``DurableWriteDrain`` remains the only consumer; it calls the SDK write
coordinator once and persists only the public receipt ids.
"""
from __future__ import annotations
@@ -477,7 +477,7 @@ class MemoryTaskStore:
#: Statuses whose input snapshot nothing reads back afterwards.
#:
#: Deliberately narrow. A `completed` task's inputs are read after the fact:
#: `memory_processing` compares them against the evidence actually sent to
#: SDK write processing compares them against the evidence actually sent to
#: HY to prove coverage, and `matters/discovery` reads them to report how
#: many records HY really saw. Clearing those would silently turn a
#: verification into a pass and a report into a lie, which is far worse
@@ -10,7 +10,6 @@ from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from datetime import UTC, datetime
from importlib import import_module
from pathlib import Path
from typing import TypedDict
from uuid import uuid4
@@ -46,14 +45,7 @@ def audit_precipitation_request(request_id: str) -> Iterator[None]:
def _current_prompt_stage() -> str:
try:
resolver = getattr( # noqa: B009 - optional SDK version boundary
import_module("hy_memory.utils.model_call_context"),
"current_model_call_stage",
)
except (ImportError, AttributeError):
return "other"
return str(resolver())
return "other"
class ModelRequestAuditStore:
+1 -1
View File
@@ -12,7 +12,7 @@ from typing import Any, Literal, Protocol
from pydantic import BaseModel, ConfigDict, Field
from one_memory.hy_memory_bridge import HySdkMemoryRecord
from one_memory.hy_sdk_types import HySdkMemoryRecord
from one_memory.memory_governance import GovernedMemory, MemoryStatus
from one_memory.semantic_deadline import (
SemanticMemoryHit,
@@ -2,10 +2,10 @@
from collections.abc import Sequence
from one_memory.conversation_processing import _is_hy_input_turn
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.imports.sources.parsing import is_host_generated_user_turn, strip_host_wrappers
from one_memory.input_sanitization import sanitize_model_input
from one_memory.memory_processing import _is_hy_input_turn
from one_memory.retrieval.chunking import TextChunk
from one_memory.retrieval.mapped_content import PreparedMappedContent, prepare_mapped_content
@@ -5,6 +5,7 @@ import json
from dataclasses import asdict, dataclass, field
from typing import Literal
from one_memory.conversation_processing import _is_hy_input_turn
from one_memory.embedding_assets import BUILTIN_EMBEDDING_MAX_TOKENS
from one_memory.imports.models import (
MAX_VERBATIM_CHARACTERS,
@@ -17,7 +18,6 @@ from one_memory.imports.sources.parsing import (
)
from one_memory.input_sanitization import sanitize_model_input_with_spans
from one_memory.local_embedding import TokenizedText
from one_memory.memory_processing import _is_hy_input_turn
from one_memory.retrieval.chunking import ChunkingError, OffsetTokenizer, TextChunk, chunk_evidence
MAP_VERSION = "raw-content-map-v1"
@@ -6,13 +6,15 @@ import asyncio
import json
from collections.abc import AsyncIterator, Awaitable, Callable
from contextlib import aclosing
from typing import Literal
from typing import Any, Literal, Protocol
import httpx
from pydantic import BaseModel, ConfigDict
from one_memory.http_client import create_async_client
from one_memory.hy_memory_bridge import HyMemoryBridge, HySdkMemoryRecord
from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import sdk_relevance_floor
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_search
from one_memory.memory_qa import (
MemoryAnswerError,
_DeferredSecretStore,
@@ -36,6 +38,10 @@ _GROUNDING_LIMIT = 6
type _Recheck = Callable[[], Awaitable[None]]
class SdkClientHandleLike(Protocol):
async def client(self) -> Any: ...
class SdkAskMemory(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -69,7 +75,7 @@ class SdkMemoryAnswerer:
def __init__(
self,
*,
bridge: HyMemoryBridge,
client_handle: SdkClientHandleLike,
profiles: ModelProfileStore,
secrets: SecretStore,
authorize_generation: Callable[
@@ -78,7 +84,7 @@ class SdkMemoryAnswerer:
| None = None,
transport: httpx.AsyncBaseTransport | None = None,
) -> None:
self.bridge = bridge
self.client_handle = client_handle
self.profiles = profiles
self.secrets = secrets
self.authorize_generation = authorize_generation
@@ -90,18 +96,27 @@ class SdkMemoryAnswerer:
*,
user_id: str = "local",
) -> tuple[SdkAskMemory, ...]:
hits = await self.bridge.search_memories(
client = await self.client_handle.client()
raw = await asyncio.to_thread(
client.search,
question,
user_id=user_id,
agent_id="*",
user_ids=[user_id],
agent_ids=[],
limit=12,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=12,
intention_limit=12,
scene="normal",
reader=HY_MEMORY_READER,
)
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
found: list[SdkAskMemory] = []
seen: set[str] = set()
for hit in hits:
if hit.memory_id in seen:
continue
record = await self.bridge.get_sdk_memory(hit.memory_id)
record = await _get_memory(client, hit.memory_id)
if record is None or record.status != "active" or not record.is_latest:
continue
seen.add(record.memory_id)
@@ -211,8 +226,9 @@ class SdkMemoryAnswerer:
self,
memories: tuple[SdkAskMemory, ...],
) -> None:
client = await self.client_handle.client()
for memory in memories:
current = await self.bridge.get_sdk_memory(memory.memory_id)
current = await _get_memory(client, memory.memory_id)
if current is None or _wire_memory(current) != memory:
raise MemoryAnswerError("memory_answer_context_changed")
@@ -340,6 +356,11 @@ def _wire_memory(record: HySdkMemoryRecord) -> SdkAskMemory:
)
async def _get_memory(client: Any, memory_id: str) -> HySdkMemoryRecord | None:
raw = await asyncio.to_thread(client.get, memory_id)
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
def _summary(memories: tuple[SdkAskMemory, ...]) -> str:
statements = [
memory.content.replace("用户", "你", 1).replace("The user", "You", 1) for memory in memories
@@ -0,0 +1,300 @@
"""Narrow SDK-native runtime for graph, retrieval, and queue status operations."""
from __future__ import annotations
import asyncio
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
from one_memory.database_access import main_database
from one_memory.extraction_settings import ExtractionSettings, ExtractionSettingsStore
from one_memory.hy_graph_admin import (
GraphWidthDecision,
GraphWidthMismatch,
GraphWidthOutcome,
decide_profile_graph_width,
inspect_profile_graph_width,
)
from one_memory.hy_graph_projection import HyGraphSnapshot, read_graph_snapshot
from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import SdkClientHandle, sdk_relevance_floor
from one_memory.hy_sdk_types import HySdkError, parse_sdk_search
from one_memory.memory_queue import ExtractionWindowPreset, QueueDepths
from one_memory.memory_server.durable_queue import COMPLETED, DurableWriteQueue
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.semantic_deadline import SemanticMemoryHit
class SdkMemoryRuntime:
"""Product services that remain after legacy task/governance writes retire."""
def __init__(
self,
*,
coordinator: SdkWriteCoordinator,
read_handle: SdkClientHandle,
write_queue: DurableWriteQueue,
extraction_settings: ExtractionSettingsStore,
data_dir: Path,
) -> None:
self.coordinator = coordinator
self.read_handle = read_handle
self.write_queue = write_queue
self.extraction_settings = extraction_settings
self.data_dir = data_dir
async def semantic_recall(
self,
query: str,
*,
limit: int = 12,
) -> tuple[SemanticMemoryHit, ...]:
client = await self.read_handle.client()
raw = await asyncio.to_thread(
client.search,
query,
user_ids=["local"],
agent_ids=[],
limit=limit,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=limit,
intention_limit=limit,
scene="normal",
reader=HY_MEMORY_READER,
)
return parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
async def semantic_search(
self,
query: str,
*,
limit: int = 12,
) -> tuple[tuple[str, float], ...]:
return tuple(
(hit.memory_id, hit.score)
for hit in await self.semantic_recall(query, limit=limit)
)
async def graph_snapshot(self, *, limit: int = 300) -> HyGraphSnapshot:
try:
await self.coordinator.active_profile_credentials()
except RuntimeError as error:
if str(error) == "model_profile_not_active":
return HyGraphSnapshot(consulted=False)
raise
try:
client = await self.coordinator.client()
except Exception as error:
raise HySdkError("hy_memory_graph_read_failed") from error
return await asyncio.to_thread(
read_graph_snapshot,
client,
"local",
"*",
limit,
)
async def digest(self) -> dict[str, object]:
client = await self.coordinator.client()
try:
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
limit=1_000,
offset=0,
)
agent_ids = _agent_ids(raw)
digests = [
{
"agent_id": agent_id,
"schemas_recorded": 0,
**dict(
await asyncio.to_thread(
client.evolve,
user_id="local",
agent_id=agent_id,
)
),
}
for agent_id in agent_ids
]
except Exception as error:
raise HySdkError("hy_memory_digest_failed") from error
return {
"success": all(result.get("success") is not False for result in digests),
"scope_count": len(digests),
"skipped_conversation_scopes": 0,
"schemas_recorded": 0,
"digests": digests,
}
async def graph_width_mismatch(self) -> GraphWidthMismatch | None:
try:
profile, secret = await self.coordinator.active_profile_credentials()
except RuntimeError as error:
if str(error) == "model_profile_not_active":
return None
raise
return await asyncio.to_thread(
inspect_profile_graph_width,
profile,
api_key=secret,
data_dir=self.data_dir,
)
async def decide_graph_width(
self,
decision: GraphWidthDecision,
) -> GraphWidthOutcome:
try:
profile, secret = await self.coordinator.active_profile_credentials()
except RuntimeError as error:
raise HySdkError(str(error)) from error
return await asyncio.to_thread(
decide_profile_graph_width,
profile,
api_key=secret,
data_dir=self.data_dir,
decision=decision,
)
async def update_hy_memory(self, memory_id: str, content: str) -> dict[str, object]:
client = await self.coordinator.client()
try:
result = await asyncio.to_thread(client.update, memory_id, content)
except Exception as error:
raise HySdkError("hy_memory_update_failed") from error
if not isinstance(result, dict) or result.get("success") is not True:
raise HySdkError("hy_memory_update_rejected")
return result
async def finish_deletion(self, memory_id: str) -> None:
client = await self.coordinator.client()
try:
result = await asyncio.to_thread(client.delete, memory_id)
except Exception as error:
raise HySdkError("hy_memory_delete_failed") from error
if not isinstance(result, dict):
raise HySdkError("hy_memory_delete_rejected")
error_code = result.get("error_code")
if error_code or result.get("partial") is True:
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
if result.get("success") is not True and result.get("deleted_count") != 0:
raise HySdkError("hy_memory_delete_rejected")
def extraction_window_settings(self) -> ExtractionSettings:
return self.extraction_settings.get()
def set_extraction_window(
self,
preset: ExtractionWindowPreset,
) -> ExtractionSettings:
return self.extraction_settings.set_window_preset(preset)
def queue_depths(self) -> QueueDepths:
now = datetime.now(UTC)
fresh_since = (now - timedelta(days=7)).isoformat()
with main_database(self.write_queue.database_path).read() as database:
fresh = int(
database.execute(
"SELECT COUNT(*) FROM memory_write_jobs "
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
"AND event_at>=?",
(fresh_since,),
).fetchone()[0]
)
backlog = int(
database.execute(
"SELECT COUNT(*) FROM memory_write_jobs "
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
"AND event_at<?",
(fresh_since,),
).fetchone()[0]
)
recent = tuple(
datetime.fromisoformat(str(row[0]))
for row in database.execute(
"SELECT updated_at FROM memory_write_jobs WHERE state='completed' "
"ORDER BY updated_at DESC LIMIT 8"
)
)
return QueueDepths(fresh=fresh, backlog=backlog, recent_completions=recent)
def active_depths(self) -> tuple[int, int]:
with main_database(self.write_queue.database_path).read() as database:
running = int(
database.execute(
"SELECT COUNT(*) FROM memory_write_jobs WHERE state='running'"
).fetchone()[0]
)
return running, running
def status_counts(self) -> dict[str, int]:
with main_database(self.write_queue.database_path).read() as database:
return {
str(row[0]): int(row[1])
for row in database.execute(
"SELECT state,COUNT(*) FROM memory_write_jobs GROUP BY state"
)
}
def manual_batch_progress(self) -> tuple[int, int]:
return 0, 0
def promote_next_manual_batch(self, *, task_ids: tuple[str, ...] | None = None) -> list[Any]:
del task_ids
return []
async def reconcile_unknown_outputs(
self,
*,
limit: int = 128,
exclude_error_codes: tuple[str, ...] = (),
) -> int:
reconciled = 0
for job in await asyncio.to_thread(
self.write_queue.unknown_jobs,
limit=limit,
exclude_error_codes=exclude_error_codes,
):
receipt = await self.coordinator.inquire_write_outcome(job.request_id)
if receipt is None:
continue
await asyncio.to_thread(
self.write_queue.settle,
job.job_id,
state=COMPLETED,
sdk_raw_ids=receipt.raw_ids,
sdk_memory_ids=receipt.memory_ids,
)
reconciled += 1
return reconciled
async def resume_allowed(self) -> None:
await asyncio.to_thread(self.write_queue.readmit_parked)
def accepted_backfill_status(self) -> None:
return None
async def shutdown(self) -> None:
return None
def _agent_ids(raw: object) -> tuple[str, ...]:
if not isinstance(raw, dict):
raise HySdkError("hy_memory_list_shape_unknown")
found: set[str] = set()
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
bucket = raw.get(bucket_name)
if bucket is None:
continue
if not isinstance(bucket, dict) or not isinstance(bucket.get(rows_name), list):
raise HySdkError("hy_memory_list_shape_unknown")
for row in bucket[rows_name]:
if isinstance(row, dict):
agent_id = row.get("agent_id")
if isinstance(agent_id, str) and agent_id:
found.add(agent_id)
return tuple(sorted(found))
@@ -22,6 +22,12 @@ from one_memory.hy_runtime_config import (
HY_MEMORY_NAMESPACE,
HY_MEMORY_VERSION,
)
from one_memory.hy_sdk_migration import (
restore_data_plane as restore_sdk_data_plane,
)
from one_memory.hy_sdk_migration import (
snapshot_data_plane,
)
class MigrationBundleRejected(RuntimeError):
@@ -113,11 +119,7 @@ def create_sdk_migration_bundle(
if not sdk_root.is_dir():
raise MigrationBundleRejected("sdk_migration_bundle_sdk_root_invalid")
if snapshotter is None:
try:
from one_memory.hy_memory_bridge import snapshot_hy_data_plane
except ImportError as error:
raise MigrationBundleRejected("sdk_migration_snapshot_unavailable") from error
snapshotter = snapshot_hy_data_plane
snapshotter = snapshot_data_plane
try:
snapshotter(
source_root=sdk_root,
@@ -275,11 +277,7 @@ def restore_sdk_migration_bundle(
raise MigrationBundleRejected("sdk_migration_bundle_manifest_invalid")
if restore_data_plane is None:
try:
from one_memory.hy_memory_bridge import restore_hy_data_plane
except ImportError as error:
raise MigrationBundleRejected("sdk_migration_restore_unavailable") from error
restore_data_plane = restore_hy_data_plane
restore_data_plane = restore_sdk_data_plane
stamp = (now or datetime.now(UTC)).strftime("%Y%m%dT%H%M%S%fZ")
database_live = home / "one-memory.sqlite3"
@@ -12,7 +12,7 @@ from pathlib import Path
from typing import Protocol
from one_memory.database_access import main_database
from one_memory.hy_memory_bridge import HySdkMemoryPage, HySdkRawRecord
from one_memory.hy_sdk_types import HySdkMemoryPage, HySdkRawPage
from one_memory.sdk_data_plane_migration import SdkDataPlaneMigrationStore
@@ -103,8 +103,8 @@ SDK_LIFECYCLE_STATUSES = (
)
class SdkMigrationBridge(Protocol):
def list_sdk_memories(
class SdkMigrationClient(Protocol):
def list_memories(
self,
*,
user_id: str,
@@ -115,7 +115,7 @@ class SdkMigrationBridge(Protocol):
order: str = "desc",
) -> Awaitable[HySdkMemoryPage]: ...
def import_sdk_memory_exact(
def write_legacy_memory(
self,
*,
content: str,
@@ -133,7 +133,7 @@ class SdkMigrationBridge(Protocol):
request_id: str,
) -> Awaitable[dict[str, object]]: ...
def list_sdk_raw(
def read_raw_inventory(
self,
*,
user_id: str,
@@ -143,9 +143,9 @@ class SdkMigrationBridge(Protocol):
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Awaitable[tuple[tuple[HySdkRawRecord, ...], int]]: ...
) -> Awaitable[HySdkRawPage]: ...
def import_sdk_raw_exact(
def write_legacy_raw(
self,
*,
content: str,
@@ -360,7 +360,7 @@ def classify_memory_inventory(
async def read_sdk_memory_inventory(
bridge: SdkMigrationBridge,
client: SdkMigrationClient,
*,
user_id: str = "local",
page_size: int = 500,
@@ -368,7 +368,7 @@ async def read_sdk_memory_inventory(
records: dict[str, SdkMemoryInventory] = {}
offset = 0
while True:
page = await bridge.list_sdk_memories(
page = await client.list_memories(
user_id=user_id,
statuses=SDK_LIFECYCLE_STATUSES,
limit=page_size,
@@ -425,7 +425,7 @@ def record_memory_migration_plan(
async def execute_memory_migration_plan(
bridge: SdkMigrationBridge,
client: SdkMigrationClient,
store: SdkDataPlaneMigrationStore,
*,
migration_id: str,
@@ -457,7 +457,7 @@ async def execute_memory_migration_plan(
continue
if decision.action in {"delete", "verified_deleted"}:
if decision.action == "delete":
await bridge.delete(decision.canonical_memory_id)
await client.delete(decision.canonical_memory_id)
store.settle_item(
migration_id=migration_id,
item_key=decision.item_key,
@@ -472,12 +472,13 @@ async def execute_memory_migration_plan(
request_id = hashlib.sha256(
f"{migration_id}:{decision.item_key}:import_exact".encode()
).hexdigest()
await bridge.import_sdk_memory_exact(
await client.write_legacy_memory(
content=item.content,
memory_id=item.canonical_memory_id,
user_id=item.user_id,
agent_id=item.agent_id,
session_id=item.session_id,
source="legacy-governance",
layer=item.layer,
status=item.status,
owner=item.owner,
@@ -485,6 +486,8 @@ async def execute_memory_migration_plan(
supersedes=item.supersedes,
superseded_by=item.superseded_by,
is_latest=item.status == "active",
memory_at=None,
metadata={},
request_id=request_id,
)
if decision.action == "record_alias":
@@ -503,7 +506,7 @@ async def execute_memory_migration_plan(
async def read_sdk_raw_inventory(
bridge: SdkMigrationBridge,
client: SdkMigrationClient,
*,
user_id: str = "local",
page_size: int = 500,
@@ -511,13 +514,13 @@ async def read_sdk_raw_inventory(
records: dict[str, SdkRawInventory] = {}
offset = 0
while True:
rows, total = await bridge.list_sdk_raw(
page = await client.read_raw_inventory(
user_id=user_id,
limit=page_size,
offset=offset,
order="asc",
)
for raw in rows:
for raw in page.raw:
raw_id = str(raw.raw_id)
records[raw_id] = SdkRawInventory(
raw_id=raw_id,
@@ -527,9 +530,9 @@ async def read_sdk_raw_inventory(
source=str(raw.source),
content_hash=content_hash(str(raw.content)),
)
consumed = len(rows)
consumed = len(page.raw)
offset += consumed
if consumed == 0 or offset >= total:
if consumed == 0 or offset >= page.total:
break
return tuple(records[key] for key in sorted(records))
@@ -632,7 +635,7 @@ def record_raw_migration_plan(
async def execute_raw_migration_plan(
bridge: SdkMigrationBridge,
client: SdkMigrationClient,
store: SdkDataPlaneMigrationStore,
*,
migration_id: str,
@@ -654,9 +657,9 @@ async def execute_raw_migration_plan(
raise RuntimeError(f"missing legacy raw inventory for {decision.item_key}")
if decision.action == "import_exact":
request_id = hashlib.sha256(
f"{migration_id}:{decision.item_key}:import_raw".encode()
f"{migration_id}:{decision.item_key}:write_legacy_raw".encode()
).hexdigest()
await bridge.import_sdk_raw_exact(
await client.write_legacy_raw(
content=item.content,
raw_id=decision.raw_id,
user_id=item.user_id,
@@ -4,7 +4,10 @@ from __future__ import annotations
import hashlib
import json
import shutil
import sqlite3
from collections.abc import Callable
from contextlib import suppress
from dataclasses import dataclass
from pathlib import Path
@@ -24,7 +27,7 @@ from one_memory.sdk_data_plane_migration import (
)
from one_memory.sdk_migration_bundle import inspect_sdk_migration_bundle
from one_memory.sdk_migration_inventory import (
SdkMigrationBridge,
SdkMigrationClient,
execute_memory_migration_plan,
execute_raw_migration_plan,
read_legacy_memory_inventory,
@@ -47,6 +50,139 @@ class BackfillResult:
raw_items: int
@dataclass(frozen=True)
class SdkMigrationStaging:
migration_id: str
staging_home: Path
live_root: Path
staging_root: Path
protective_root: Path
state_file: Path
def sdk_migration_staging(
*,
data_dir: Path,
migration_id: str,
namespace: str,
) -> SdkMigrationStaging:
"""Return same-filesystem paths without trusting the migration ID as a path."""
token = hashlib.sha256(migration_id.encode()).hexdigest()[:16]
home = data_dir.resolve()
staging_home = home / ".sdk-migration" / token
live_root = home / "hy-memory" / namespace
return SdkMigrationStaging(
migration_id=migration_id,
staging_home=staging_home,
live_root=live_root,
staging_root=staging_home / "hy-memory" / namespace,
protective_root=live_root.parent / f".{namespace}.pre-migration-{token}",
state_file=staging_home / "swap-state.json",
)
def _write_staging_state(paths: SdkMigrationStaging, phase: str) -> None:
paths.staging_home.mkdir(parents=True, exist_ok=True)
temporary = paths.state_file.with_suffix(".tmp")
temporary.write_text(
json.dumps(
{"schema_version": 1, "migration_id": paths.migration_id, "phase": phase},
sort_keys=True,
),
encoding="utf-8",
)
temporary.replace(paths.state_file)
def recover_sdk_migration_swap(
*,
paths: SdkMigrationStaging,
database_path: Path,
) -> None:
"""Recover an interrupted directory swap before opening any SDK store."""
if not paths.protective_root.exists():
return
authority = SdkDataPlaneMigrationStore(database_path).authority()
if authority.value == "sdk":
if not paths.live_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_committed_store_missing")
shutil.rmtree(paths.protective_root)
shutil.rmtree(paths.staging_home, ignore_errors=True)
return
if paths.live_root.exists():
if paths.staging_root.exists():
raise DataPlaneMigrationError("sdk_migration_swap_state_ambiguous")
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
paths.live_root.replace(paths.staging_root)
paths.protective_root.replace(paths.live_root)
_write_staging_state(paths, "recovered")
def prepare_sdk_migration_staging(
*,
paths: SdkMigrationStaging,
database_path: Path,
) -> Path:
"""Build a private copy; incomplete retries restart from the unchanged live root."""
recover_sdk_migration_swap(paths=paths, database_path=database_path)
migration_store = SdkDataPlaneMigrationStore(database_path)
try:
progress = migration_store.progress(paths.migration_id)
reuse_ready = progress.state == "ready" and progress.phase == "reconciled"
except Exception:
reuse_ready = False
if paths.staging_root.exists():
if paths.staging_root.is_symlink() or not paths.staging_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_staging_invalid")
if reuse_ready:
return paths.staging_home
shutil.rmtree(paths.staging_root)
paths.live_root.mkdir(parents=True, exist_ok=True)
from one_memory.hy_sdk_migration import snapshot_data_plane
snapshot_data_plane(
source_root=paths.live_root,
target_root=paths.staging_root,
writers_stopped=True,
)
_write_staging_state(paths, "prepared")
return paths.staging_home
def activate_sdk_migration_staging(
*,
paths: SdkMigrationStaging,
database_path: Path,
commit: Callable[[], CutoverMarker],
) -> CutoverMarker:
"""Atomically expose staged SDK files and roll them back if cutover fails."""
recover_sdk_migration_swap(paths=paths, database_path=database_path)
if not paths.staging_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_staging_missing")
paths.live_root.mkdir(parents=True, exist_ok=True)
_write_staging_state(paths, "activating")
paths.live_root.replace(paths.protective_root)
try:
paths.staging_root.replace(paths.live_root)
_write_staging_state(paths, "staging_live")
marker = commit()
except BaseException:
if paths.live_root.exists():
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
paths.live_root.replace(paths.staging_root)
if paths.protective_root.exists():
paths.protective_root.replace(paths.live_root)
_write_staging_state(paths, "rolled_back")
raise
with suppress(OSError):
_write_staging_state(paths, "committed")
# Authority is already committed; leftover paths are detected by
# sdk-migration-status and removed by sdk-cleanup on the next startup.
shutil.rmtree(paths.protective_root, ignore_errors=True)
shutil.rmtree(paths.staging_home, ignore_errors=True)
return marker
def prepare_legacy_write_queue_for_cutover(database_path: Path) -> tuple[int, int]:
"""Freeze old jobs and move unstarted task rows into the surviving queue."""
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
@@ -217,7 +353,6 @@ def commit_reconciled_cutover(
reconciliation_digest=digest,
)
inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=marker.backup_bundle_id)
drop_legacy_tables(database_path)
return marker
@@ -232,13 +367,23 @@ def resume_legacy_cleanup(*, data_dir: Path, database_path: Path) -> tuple[str,
data_dir=data_dir,
bundle_id=marker.backup_bundle_id,
)
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
recover_sdk_migration_swap(
paths=sdk_migration_staging(
data_dir=data_dir,
migration_id=marker.migration_id,
namespace=HY_MEMORY_NAMESPACE,
),
database_path=database_path,
)
return drop_legacy_tables(database_path)
async def run_sdk_data_plane_backfill(
*,
database_path: Path,
bridge: SdkMigrationBridge,
client: SdkMigrationClient,
migration_id: str,
bundle_id: str,
source_runtime_build: str,
@@ -261,7 +406,7 @@ async def run_sdk_data_plane_backfill(
store.set_phase(migration_id, phase="inventory", state="running")
legacy_memories = read_legacy_memory_inventory(database_path)
sdk_memories = await read_sdk_memory_inventory(bridge)
sdk_memories = await read_sdk_memory_inventory(client)
memory_decisions = record_memory_migration_plan(
store,
migration_id=migration_id,
@@ -269,7 +414,7 @@ async def run_sdk_data_plane_backfill(
sdk=sdk_memories,
)
legacy_raw = read_legacy_raw_inventory(database_path)
sdk_raw = await read_sdk_raw_inventory(bridge)
sdk_raw = await read_sdk_raw_inventory(client)
raw_decisions = record_raw_migration_plan(
store,
migration_id=migration_id,
@@ -279,14 +424,14 @@ async def run_sdk_data_plane_backfill(
store.set_phase(migration_id, phase="backfill", state="running")
await execute_memory_migration_plan(
bridge,
client,
store,
migration_id=migration_id,
legacy=legacy_memories,
decisions=memory_decisions,
)
await execute_raw_migration_plan(
bridge,
client,
store,
migration_id=migration_id,
legacy=legacy_raw,
@@ -0,0 +1,330 @@
"""Single-write adapter from durable Evidence payloads to the public HY SDK."""
from __future__ import annotations
import asyncio
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from one_memory.hy_sdk_runtime import SdkClientHandle
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
from one_memory.input_sanitization import sanitize_model_input
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
PRODUCT_AGENT_ID = "all-apps"
_MESSAGE_ROLES = frozenset({"user", "assistant", "tool"})
@dataclass(frozen=True)
class SdkWriteReceipt:
"""The only SDK write identity retained by MemHub."""
raw_ids: tuple[str, ...] = ()
memory_ids: tuple[str, ...] = ()
class SdkWriteProcessor:
"""Turn one durable job into exactly one public SDK write call."""
def __init__(self, client_handle: SdkClientHandle) -> None:
self._client_handle = client_handle
async def process_sdk_job(
self,
records: tuple[EvidenceRecord, ...],
*,
request_id: str,
before_hy_invocation: Callable[[], Awaitable[None]],
) -> SdkWriteReceipt:
if not records:
raise RuntimeError("hy_memory_task_without_inputs")
scope = _write_scope(records)
metadata = _write_metadata(records, scope)
client = await self._client_handle.client()
if all(_is_extracted_content(record) for record in records):
contents = [sanitize_model_input(record.verbatim) for record in records]
await before_hy_invocation()
result = await asyncio.to_thread(
client.add_extracted,
contents,
user_id=scope.user_id,
agent_id=PRODUCT_AGENT_ID,
session_id=scope.session_id,
source=scope.source,
metadata=metadata,
memory_at=scope.memory_at,
request_id=request_id,
)
else:
messages = [
{
"role": record.role,
"content": sanitize_model_input(record.verbatim),
}
for record in records
if record.content_kind is EvidenceContentKind.RAW_CONVERSATION
and record.role in _MESSAGE_ROLES
]
if not messages:
raise RuntimeError("hy_memory_task_without_inputs")
await before_hy_invocation()
result = await asyncio.to_thread(
client.add,
messages,
user_id=scope.user_id,
agent_id=PRODUCT_AGENT_ID,
session_id=scope.session_id,
source=scope.source,
metadata=metadata,
memory_at=scope.memory_at,
scene=None,
workspace_id=scope.workspace,
request_id=request_id,
)
return _receipt_from_result(result)
async def inquire_write_outcome(self, request_id: str) -> SdkWriteReceipt | None:
"""Return a confirmed successful receipt without scanning SDK stores."""
if not request_id:
raise ValueError("request_id is required")
client = await self._client_handle.client()
outcome = await asyncio.to_thread(client.get_write_outcome, request_id)
if outcome is None:
return None
if not isinstance(outcome, Mapping):
raise RuntimeError("hy_sdk_write_outcome_shape_invalid")
status = outcome.get("status")
if status != "succeeded":
return None
result = outcome.get("result")
if not isinstance(result, Mapping):
raise RuntimeError("hy_sdk_write_outcome_shape_invalid")
return _receipt_from_result(result)
async def close(self) -> None:
await self._client_handle.close()
class SdkWriteCoordinator:
"""Resolve the active model profile, then execute one public SDK write."""
def __init__(
self,
*,
profiles: ModelProfileStore,
secrets: SecretStore,
data_dir: Any,
transport_factory: Callable[[ModelProfileSnapshot], Callable[[], Any] | None],
sedimentation_enabled: Callable[[], bool],
) -> None:
self.profiles = profiles
self.secrets = secrets
self.data_dir = data_dir
self.transport_factory = transport_factory
self.sedimentation_enabled = sedimentation_enabled
self._handles: dict[tuple[str, int], SdkClientHandle] = {}
self._lock = asyncio.Lock()
async def process_sdk_job(
self,
records: tuple[EvidenceRecord, ...],
*,
request_id: str,
before_hy_invocation: Callable[[], Awaitable[None]],
) -> SdkWriteReceipt:
if not self.sedimentation_enabled():
raise RuntimeError("model_sedimentation_disabled")
handle = await self._active_handle()
return await SdkWriteProcessor(handle).process_sdk_job(
records,
request_id=request_id,
before_hy_invocation=before_hy_invocation,
)
async def inquire_write_outcome(
self, request_id: str
) -> SdkWriteReceipt | None:
return await SdkWriteProcessor(
await self._active_handle()
).inquire_write_outcome(request_id)
async def close(self) -> None:
async with self._lock:
handles, self._handles = tuple(self._handles.values()), {}
await asyncio.gather(*(handle.close() for handle in handles))
async def client(self) -> Any:
"""Return the public SDK client selected by the active write profile."""
return await (await self._active_handle()).client()
async def active_profile_credentials(
self,
) -> tuple[ModelProfileSnapshot, str | None]:
"""Resolve one stable profile snapshot for graph administration."""
_identity, profile, secret = await asyncio.to_thread(self._read_active_credentials)
return profile, secret
async def _active_handle(self) -> SdkClientHandle:
identity, profile, secret = await asyncio.to_thread(
self._read_active_credentials
)
async with self._lock:
handle = self._handles.get(identity)
if handle is None:
handle = SdkClientHandle(
profile,
api_key=secret,
data_dir=self.data_dir,
http_transport_factory=self.transport_factory(profile),
)
self._handles[identity] = handle
stale = [
value
for key, value in self._handles.items()
if key != identity
]
self._handles = {identity: handle}
await asyncio.gather(*(value.close() for value in stale))
return handle
def _read_active_credentials(
self,
) -> tuple[tuple[str, int], ModelProfileSnapshot, str | None]:
identity = self.profiles.active_identity()
if identity is None:
raise RuntimeError("model_profile_not_active")
profile = self.profiles.snapshot(*identity)
try:
secret = self.profiles.resolve_snapshot_secret(
*identity,
self.secrets,
)
except SecretStoreUnavailableError:
raise RuntimeError("secure_storage_unavailable") from None
if self.profiles.active_identity() != identity:
raise RuntimeError("model_profile_changed")
if profile.has_api_key and secret is None:
raise RuntimeError("model_secret_unavailable")
return identity, profile, secret
@dataclass(frozen=True)
class _WriteScope:
user_id: str
session_id: str
source: str
workspace: str | None
workspaces: tuple[str, ...]
memory_at: datetime | None
def _write_scope(records: tuple[EvidenceRecord, ...]) -> _WriteScope:
first = records[0]
expected = (first.user_id, first.source_id, first.conversation_id)
if any(
(record.user_id, record.source_id, record.conversation_id) != expected
for record in records[1:]
):
raise RuntimeError("durable_write_multiple_sources")
workspaces = tuple(
sorted({record.workspace for record in records if record.workspace is not None})
)
return _WriteScope(
user_id=first.user_id,
session_id=first.conversation_id,
source=first.source_id,
workspace=workspaces[0] if len(workspaces) == 1 else None,
workspaces=workspaces,
memory_at=max(
(record.created_at for record in records if record.created_at is not None),
default=None,
),
)
def _write_metadata(
records: tuple[EvidenceRecord, ...],
scope: _WriteScope,
) -> dict[str, Any]:
return {
"source_id": scope.source,
"session_id": scope.session_id,
"workspace_ids": list(scope.workspaces),
"memory_at": scope.memory_at.isoformat() if scope.memory_at is not None else None,
"evidence_ids": [record.evidence_id for record in records],
"input_kind": (
"memory_content_without_raw"
if all(_is_extracted_content(record) for record in records)
else "raw_conversation"
),
}
def _is_extracted_content(record: EvidenceRecord) -> bool:
if record.content_kind is EvidenceContentKind.MEMORY_CONTENT:
return True
manual = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
return (
record.source_id == "one-memory-manual"
and manual is not None
and manual.raw_meta.get("input_mode") == "direct_memory_note"
)
def _receipt_from_result(value: object) -> SdkWriteReceipt:
if not isinstance(value, Mapping):
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
if value.get("success") is not True:
raise RuntimeError("hy_sdk_write_failed")
# Read only documented public receipt fields. ``variant`` identifies the
# productivity namespace; IDs from both variants are canonical memory IDs.
variant = value.get("variant")
if variant is not None and not isinstance(variant, str):
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
raw_ids = _public_ids(value, singular="raw_id", plural="raw_ids")
memory_ids = _ordered_unique(
(
*_public_ids(value, singular="memory_id", plural="memory_ids"),
*_public_ids(value, plural="experience_ids"),
*_public_ids(value, plural="fact_ids"),
)
)
return SdkWriteReceipt(raw_ids=raw_ids, memory_ids=memory_ids)
def _public_ids(
value: Mapping[object, object],
*,
singular: str | None = None,
plural: str,
) -> tuple[str, ...]:
found: list[str] = []
if singular is not None:
item = value.get(singular)
if item is not None:
if not isinstance(item, str):
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
if item:
found.append(item)
items = value.get(plural)
if items is not None:
if isinstance(items, (str, bytes)) or not isinstance(items, Sequence):
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
for item in items:
if not isinstance(item, str):
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
if item:
found.append(item)
return _ordered_unique(found)
def _ordered_unique(values: Sequence[str]) -> tuple[str, ...]:
return tuple(dict.fromkeys(values))
+1 -19
View File
@@ -4,7 +4,6 @@
| 测试文件 | 等级 | 一句话说明 |
| --- | --- | --- |
| `test_active_credentials.py` | `contract` | Active credentials are read once per call, never reused after a refusal. |
| `test_anchor_invalid_rewind.py` | `contract` | Bug 2 real branch: an in-place same-length rewrite invalidates the anchor. |
| `test_architecture_layer_gate.py` | `contract` | A0-01: L0-L4 dependency-direction gate. |
| `test_archive_detection.py` | `contract` | S4.1: what an uploaded export is, decided before anything parses it. |
@@ -91,7 +90,6 @@
| `test_disable_thinking.py` | `contract` | 运行期生成请求须按档案可选关闭思考;默认不向厂商下发未知字段。 |
| `test_dual_engine_helper.py` | `contract` | Tests for the dual-engine harness itself. |
| `test_event_loop_blocking.py` | `contract` | Proof, as a test, that blocking work on the loop freezes it — and the fix works. |
| `test_evidence_reconcile_primitive.py` | `integration` | A bounded, read-only discovery primitive for the H23 Evidence reconcile. |
| `test_evidence_store.py` | `contract` | 覆盖 same source message merges origins without duplication、identical text from different sources is not deduplicated、rescanning an unchanged file does not write the message again 等 26 个场景。 |
| `test_export_probe.py` | `contract` | What a probe may say about a stranger's chat history, and what it may not. |
| `test_external_installation_permissions.py` | `integration` | 覆盖 external marker is not an installation grant、external diagnostic trace uses its own read only tool、raw source grants filter before limit and private body loading 场景。 |
@@ -103,8 +101,6 @@
| `test_file_import_operations_review.py` | `contract` | Upload refusal/legacy compatibility review with bounded synthetic files only. |
| `test_frozen_core_layout.py` | `contract` | 覆盖 single file layout remains available without private preset、folder layout keeps same core and all native dependencies、unknown layout does not silently build another package 等 5 个场景。 |
| `test_golden_corpus.py` | `contract` | A7-01: the golden corpus comparison. |
| `test_governance_permission_commit.py` | `contract` | 覆盖 permission guard is rechecked inside write transaction and rolls back、guard context does not leak into subsequent manual actions、guard check and commit hold database write authority 场景。 |
| `test_governance_read_path_does_not_take_the_writer.py` | `contract` | The recall read path must not acquire the single global writer. |
| `test_graph_width_endpoint.py` | `integration` | The width mismatch has to reach the interface, not just the log. |
| `test_graph_width_export.py` | `unit` | EXPORT_FIRST has to actually export. |
| `test_graph_width_guard.py` | `unit` | Graph width mismatch must ask, never silently archive. |
@@ -115,16 +111,9 @@
| `test_host_capabilities.py` | `unit` | A plugin does not become realtime on every host or every unknown version. |
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
| `test_hourly_c1_recall_stage_probe_contract.py` | `contract` | Guard the packaged recall stage probe's command and error semantics. |
| `test_hy_coding_reply.py` | `contract` | R-07: HY's coding-shaped replies read as "no memories at all". |
| `test_hy_log_privacy.py` | `contract` | HY SDK records are not rewritten by One Memory's logging setup. |
| `test_hy_memory_bridge.py` | `contract` | 覆盖 hy client initialization is serial across bridges、native hy clients share one runtime per data directory、ollama profile ignores ambient remote storage and models 等 41 个场景。 |
| `test_hy_metrics_lifecycle.py` | `contract` | What the pinned public SDK's MetricsCollector does, and what that costs us. |
| `test_hy_network_egress.py` | `contract` | Where a real HY client is allowed to send bytes, and where it is not. |
| `test_hy_response_guard.py` | `contract` | 覆盖 invalid response retries same hy provider without changing prompt、repeated empty response fails instead of becoming no op、failure receipt survives a later successful model call 等 23 个场景。 |
| `test_hy_response_shapes.py` | `contract` | HY response shape selection only; no provider, write routing or model calls. |
| `test_hy_runtime_config.py` | `contract` | MEMORY.md is the audited record of what we send HY; keep it answerable. |
| `test_hy_sdk_idempotency_probe.py` | `contract` | A0-03: probe whether the HY SDK can make a write idempotent. |
| `test_hy_source_context.py` | `contract` | 覆盖 bound context rejects a same named conversation in another workspace、bound context rejects a duplicated evidence id、bound context rejects a same conversation from another source 等 9 个场景。 |
| `test_hy_step_trace_switch.py` | `contract` | Rule 15: the per-step HY trace is off unless the debug tier asks for it. |
| `test_import_api.py` | `integration` | 覆盖 resume during cleanup returns conflict without internal error、authorize detect import and revoke api flow、protected api rejects missing token and foreign origin 等 5 个场景。 |
| `test_import_control_races_review.py` | `contract` | Read-only review repro: real runner/store, synthetic delayed adapter cleanup. |
@@ -168,16 +157,11 @@
| `test_memory_batch_extraction.py` | `contract` | 覆盖 batch extractor keeps only unique known evidence ids、batch extractor maps provider failure to machine error 场景。 |
| `test_memory_citation_query_plan.py` | `contract` | Correlated citation checks must look links up by memory, not scan by 'active'. |
| `test_memory_content_import.py` | `integration` | 覆盖 existing memory preview keeps unknown author and exact original、invalid existing memory records never leave a partial upload、memory content requires explicit mode and rejects vendor guessing 等 5 个场景。 |
| `test_memory_correction_sync.py` | `integration` | R-27: a correction that only lands locally, while HY keeps the old text. |
| `test_memory_evidence_freshness.py` | `contract` | 新鲜度拆除后:读路径恒为 current;证据修订 / 畸形 proof / segment ref 不再藏记忆或 503。 |
| `test_memory_freshness_integration_review.py` | `integration` | Independent synthetic read-path review; no lifespan, model or user database. |
| `test_memory_governance.py` | `contract` | 覆盖 correction keeps verbatim and appends history、mutations are revision checked and request idempotent、delete result survives restart and request replay 等 42 个场景。 |
| `test_memory_governance_api.py` | `integration` | 覆盖 memory governance api preserves evidence and reports cleanup、memory validation does not echo content 场景。 |
| `test_memory_graph_api.py` | `integration` | 覆盖 graph labels where every edge came from、a created memory is on the map with its source links、drawing the map does not stop the app answering 等 12 个场景。 |
| `test_memory_language.py` | `contract` | R-03: memory language follows the message, so Kristin's gets written in English. |
| `test_memory_notes.py` | `integration` | 覆盖 empty library can save a note without a matter or configured model、note requires authentication explicit scope and nonempty content、saved note evidence returns raw and consent reference from same record 等 6 个场景。 |
| `test_memory_processing.py` | `contract` | 覆盖 stale known reason diagnostics never fail dispatch、memory task logs six step durations without content、assistant turns reach hy so references can be resolved 等 95 个场景。 |
| `test_memory_processing_diagnostics.py` | `contract` | 覆盖 conversation prepare failure includes safe message and stack、deletion sync failure includes safe message and stack 场景。 |
| `test_memory_qa.py` | `contract` | 覆盖 search terms keep meaningful single chinese character、streamed answer emits recall then incremental grounded text、governance change during generation stops the old answer 等 14 个场景。 |
| `test_memory_qa_api.py` | `integration` | 覆盖 ask stream endpoint emits recall deltas and final、ask stream endpoint requires the local token 场景。 |
| `test_memory_qa_permissions.py` | `contract` | 覆盖 initial refusal precedes all secret access and network、invalid authorization result fails before secret access、authorization receives exact question and only outbound grounding 等 16 个场景。 |
@@ -289,7 +273,6 @@
| `test_secret_store_error_code_api.py` | `integration` | 3925a kernel item 3: the error code reaches the HTTP boundary. |
| `test_secret_store_error_codes.py` | `integration` | 3925a kernel item 3: keychain failure classification and backoff that heals. |
| `test_secure_storage.py` | `integration` | 覆盖 isolated keychain service namespace、custom keychain service is refused outside isolated tests、packaged keychain budget covers loaded process startup 等 20 个场景。 |
| `test_sedimentation_concurrency.py` | `contract` | Sedimentation runs several harnesses at once, but one harness at a time. |
| `test_sedimentation_switch.py` | `contract` | Stopping model-driven sedimentation without losing the configuration. |
| `test_service_level_pressure_acceptance.py` | `contract` | 覆盖 workbuddy recall remains available during 100k import 场景。 |
| `test_service_levels.py` | `contract` | 覆盖 background never consumes the realtime reservation、realtime is next for the same source after background yields、import history gets one slice after nine realtime admissions 等 8 个场景。 |
@@ -320,7 +303,6 @@
| `test_source_sync_idempotent.py` | `integration` | Final-state Harness sync is idempotent and resumes missed source work. |
| `test_source_sync_permissions.py` | `integration` | S1.3 source permission tests use temporary files only, never real host history. |
| `test_source_totals.py` | `contract` | 覆盖 the first ask is answered from the rows、same named sessions in two workspaces count as two conversations、insert during count is not marked as already counted 等 7 个场景。 |
| `test_source_write_switch.py` | `contract` | A frozen Harness produces no extraction tasks. |
| `test_sql_compat.py` | `contract` | The shared compatibility layer, especially 'rowcount'. |
| `test_sqlite_semantics_gate.py` | `contract` | D56: prevent SQLite implementation details spreading outside the data layer. |
| `test_sqlite_write_contention.py` | `contract` | Measure whether SQLite's single writer actually blocks us. |
@@ -345,4 +327,4 @@
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
共 340 个测试文件。
共 322 个测试文件。
@@ -14,8 +14,8 @@ from pathlib import Path
import pytest
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
EXPECTED_TEST_FILE_COUNT = 340
EXPECTED_TEST_FILES_SHA256 = "f69d3f33df5277febb09557d5bce7adb7d8719ed47e48a71f852e3e4ecf8fd74"
EXPECTED_TEST_FILE_COUNT = 322
EXPECTED_TEST_FILES_SHA256 = "d242783adb4015ebdbb5c099f3b5aa88a99c828ca6fddc8ca8d6c74c56e48945"
# Pure transformations, validation models, parity checks and state machines.
# This list is intentionally small: uncertain files default to contract.
@@ -62,7 +62,6 @@ INTEGRATION_FILES = frozenset(
"test_d93_same_process_load.py",
"test_diagnostic_trace.py",
"test_diagnostic_trace_api.py",
"test_evidence_reconcile_primitive.py",
"test_external_installation_permissions.py",
"test_fault_matrix.py",
"test_file_import_operations.py",
@@ -77,7 +76,6 @@ INTEGRATION_FILES = frozenset(
"test_manual_transcript_import.py",
"test_measure_c1_loop_blocking_contract.py",
"test_memory_content_import.py",
"test_memory_correction_sync.py",
"test_memory_freshness_integration_review.py",
"test_memory_governance_api.py",
"test_memory_graph_api.py",
+46
View File
@@ -0,0 +1,46 @@
from __future__ import annotations
import gc
from pathlib import Path
import pytest
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
def model_snapshot(
provider: ProviderKind,
*,
base_url: str | None,
has_api_key: bool,
embedding_provider: ProviderKind | None = None,
embedding_base_url: str | None = None,
) -> ModelProfileSnapshot:
return ModelProfileSnapshot(
profile_id="profile",
revision=3,
provider_kind=provider,
base_url=base_url,
model="memory-model",
embedding_model="embedding-model",
embedding_provider_kind=embedding_provider,
embedding_base_url=embedding_base_url,
embedding_dimensions=768,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
has_api_key=has_api_key,
)
def kuzu_graph_with_embedding_width(db_path: Path, width: int) -> None:
kuzu = pytest.importorskip("kuzu")
database = kuzu.Database(str(db_path))
connection = kuzu.Connection(database)
connection.execute(
f"CREATE NODE TABLE Memory("
f"node_id STRING, embedding FLOAT[{width}], PRIMARY KEY(node_id))"
)
del connection
del database
gc.collect()
@@ -1,225 +0,0 @@
"""Active credentials are read once per call, never reused after a refusal."""
import asyncio
import threading
import pytest
from pydantic import SecretStr
from one_memory.background_work import shutdown_bulk_pool
from one_memory.hy_memory_bridge import HyMemoryBridgeError
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import MemoryGovernanceStore
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_tasks import MemoryTaskStore
from one_memory.model_profiles import (
ModelCapabilities,
ModelProfileInput,
ModelProfilePatch,
ModelProfileStore,
)
from one_memory.secrets import InMemorySecretStore, SecretStoreUnavailableError
class CountingSecrets(InMemorySecretStore):
def __init__(self):
super().__init__()
self.reads = []
self.denied = False
self.after_read = None
def has(self, secret_ref):
# The OS store's has() resolves the secret; the in-memory default does not.
return self.resolve(secret_ref) is not None
def resolve(self, secret_ref):
self.reads.append((secret_ref, threading.get_ident()))
if self.denied:
raise SecretStoreUnavailableError("synthetic reader denial, must not leak")
value = super().resolve(secret_ref)
callback, self.after_read = self.after_read, None
if callback is not None:
callback()
return value
def create_profile(profiles, secrets, *, keyed=True):
profile = profiles.create(
ModelProfileInput(
name="Synthetic credentials",
provider_kind="openai_compatible" if keyed else "ollama",
base_url="http://127.0.0.1:1/v1",
model="synthetic-model",
api_key=SecretStr("synthetic-credential") if keyed else None,
),
secrets,
)
profiles.record_capabilities(
profile.profile_id,
expected_revision=profile.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True,
model_available=True,
structured_output=True,
embedding_available=True,
),
secrets=secrets,
)
return profiles.activate(profile.profile_id, secrets)
@pytest.fixture
def setup(tmp_path):
database = tmp_path / "one-memory.sqlite3"
secrets = CountingSecrets()
profiles = ModelProfileStore(database)
profile = create_profile(profiles, secrets)
supervisor = MemoryProcessingSupervisor(
tasks=MemoryTaskStore(database),
profiles=profiles,
secrets=secrets,
evidence=EvidenceStore(database),
governance=MemoryGovernanceStore(database),
data_dir=tmp_path,
)
secrets.reads.clear()
try:
yield supervisor, profiles, secrets, profile
finally:
shutdown_bulk_pool()
def test_resolves_active_immutable_snapshot_once_off_loop(setup):
supervisor, profiles, secrets, profile = setup
expected = profiles.snapshot(profile.profile_id, profile.revision)
caller = threading.get_ident()
result = asyncio.run(supervisor._active_credentials())
assert result == (expected, "synthetic-credential")
assert secrets.reads == [(profiles.secret_ref(profile.profile_id), secrets.reads[0][1])]
assert secrets.reads[0][1] != caller
assert supervisor.tasks.list() == []
assert supervisor._bridges == {}
@pytest.mark.parametrize("concurrent", [False, True])
def test_each_call_resolves_again_without_cross_request_cache(setup, concurrent):
supervisor, _profiles, secrets, _profile = setup
async def scenario():
if concurrent:
return await asyncio.gather(*(supervisor._active_credentials() for _ in range(4)))
return [await supervisor._active_credentials() for _ in range(4)]
results = asyncio.run(scenario())
assert all(result is not None and result[1] == "synthetic-credential" for result in results)
assert len(secrets.reads) == 4
@pytest.mark.parametrize("failure", ["missing", "denied"])
def test_prior_success_cannot_mask_current_reader_refusal(setup, failure):
supervisor, profiles, secrets, profile = setup
async def scenario():
assert (await supervisor._active_credentials())[1] == "synthetic-credential"
secrets.reads.clear()
if failure == "missing":
secrets.delete(profiles.secret_ref(profile.profile_id))
else:
secrets.denied = True
code = "model_secret_unavailable" if failure == "missing" else "secure_storage_unavailable"
with pytest.raises(HyMemoryBridgeError, match=f"^{code}$"):
await supervisor._active_credentials()
assert len(secrets.reads) == 1
asyncio.run(scenario())
@pytest.mark.parametrize("change", ["deactivate", "switch", "revision"])
def test_profile_change_during_secret_read_rejects_old_credentials(setup, change):
supervisor, profiles, secrets, profile = setup
other = create_profile(profiles, secrets)
profiles.activate(profile.profile_id, secrets)
secrets.reads.clear()
def change_identity():
# Concurrent committed identity change while the original reader is in flight.
with profiles._connect(writing=True) as db:
if change == "revision":
db.execute(
"UPDATE model_profiles SET revision=revision+1 WHERE profile_id=?",
(profile.profile_id,),
)
else:
db.execute("UPDATE model_profiles SET is_active=0")
if change == "switch":
db.execute(
"UPDATE model_profiles SET is_active=1 WHERE profile_id=?",
(other.profile_id,),
)
secrets.after_read = change_identity
with pytest.raises(HyMemoryBridgeError, match="^model_profile_changed$"):
asyncio.run(supervisor._active_credentials())
assert len(secrets.reads) == 1
assert supervisor._bridges == {}
def test_rotation_between_calls_returns_new_snapshot_and_secret(setup):
supervisor, profiles, secrets, profile = setup
before = asyncio.run(supervisor._active_credentials())
updated = profiles.update(
profile.profile_id,
ModelProfilePatch(expected_revision=profile.revision, api_key=SecretStr("synthetic-new")),
secrets,
)
assert updated.is_active is False
assert updated.capabilities == ModelCapabilities()
assert profiles.active_identity() is None
secrets.reads.clear()
assert asyncio.run(supervisor._active_credentials()) is None
assert secrets.reads == []
assert supervisor.tasks.list() == []
assert supervisor._bridges == {}
# Synthetic probe evidence and explicit activation belong to the new revision.
profiles.record_capabilities(
updated.profile_id,
expected_revision=updated.revision,
capabilities=profile.capabilities,
secrets=secrets,
)
updated = profiles.activate(updated.profile_id, secrets)
assert updated.is_active is True
secrets.reads.clear()
after = asyncio.run(supervisor._active_credentials())
assert before[0].revision == 1 and before[1] == "synthetic-credential"
assert after == (profiles.snapshot(updated.profile_id, updated.revision), "synthetic-new")
assert after[0].revision == 2
assert len(secrets.reads) == 1
def test_no_active_profile_does_not_read_credentials(setup):
supervisor, profiles, secrets, _profile = setup
with profiles._connect(writing=True) as db:
db.execute("UPDATE model_profiles SET is_active=0")
assert asyncio.run(supervisor._active_credentials()) is None
assert secrets.reads == []
def test_keyless_profile_does_not_require_or_resolve_a_secret(setup):
supervisor, profiles, secrets, _profile = setup
keyless = create_profile(profiles, secrets, keyed=False)
secrets.reads.clear()
result = asyncio.run(supervisor._active_credentials())
assert result == (profiles.snapshot(keyless.profile_id, keyless.revision), None)
assert result[0].has_api_key is False
assert secrets.reads == []
@@ -4,12 +4,11 @@ The public target architecture is defined by ``docs/ARCHITECTURE.md``:
* Hy Memory SDK owns memory/raw/vector/graph/coding data;
* MemHub owns the management plane and may retain ``memory_id`` references;
* product code reaches the SDK through ``hy_memory_bridge`` only.
* product code reaches the SDK only through narrow runtime/migration modules;
* MemHub calls public ``HyMemoryClient`` methods only.
The legacy implementation still contains Evidence/Governance writers and a
small number of private SDK-store accesses. Those are migration debt, not an
allowance for new code. The explicit debt lists below may shrink during the
refactor; new entries must not be added.
The sole exception is the read-only Kuzu projection/admin surface used by the
existing graph UI and Graph Width workflow.
The gate is deliberately not a blanket ban on ``sqlite3``. Adapters may read a
host product's SQLite through the guarded read-only module; they may not own
@@ -58,13 +57,13 @@ HOST_READONLY_MODULE = "one_memory.imports.sqlite_access"
SRC_ROOT = PACKAGE_ROOT.parents[0]
PACKAGE_LABEL = PACKAGE_ROOT.name
HY_BRIDGE = f"{PACKAGE_LABEL}/hy_memory_bridge.py"
#: The one module allowed to *invoke* the SDK.
SDK_CALLER = HY_BRIDGE
#: Type-only importers of HY datatypes; they hold no client and write nothing.
HY_TYPE_ONLY: frozenset[str] = frozenset()
SDK_MODULES = frozenset(
{
f"{PACKAGE_LABEL}/hy_sdk_runtime.py",
f"{PACKAGE_LABEL}/hy_sdk_migration.py",
}
)
SDK_CLIENT_CONSTRUCTOR = f"{PACKAGE_LABEL}/hy_sdk_runtime.py"
# Frozen migration debt: modules that still write the legacy memory/raw
# authority. The target count is zero. A new module cannot join this set.
@@ -84,19 +83,37 @@ LEGACY_AUTHORITY_WRITE = re.compile(
re.IGNORECASE,
)
# Existing accesses to SDK implementation details. These are exact file /
# attribute debts so deleting one is always green, while adding one fails.
SDK_PRIVATE_ACCESS_DEBT: dict[str, frozenset[str]] = {
f"{PACKAGE_LABEL}/hy_lifecycle.py": frozenset({"_collection", "_vector_store"}),
f"{PACKAGE_LABEL}/hy_memory_bridge.py": frozenset(
{"_coding_store", "_graph_store", "_loop_thread", "_vector_store"}
),
f"{PACKAGE_LABEL}/hy_response_guard.py": frozenset({"_loop_thread", "_registry"}),
}
SDK_PRIVATE_ATTRIBUTES = frozenset(
attribute for attributes in SDK_PRIVATE_ACCESS_DEBT.values() for attribute in attributes
{
"_coding_store",
"_coding_stores",
"_collection",
"_embed_service",
"_graph_store",
"_loop_thread",
"_readers",
"_reader",
"_reconciler",
"_registry",
"_run",
"_shared_embed_service",
"_spawn_search_dedup",
"_system2_writer",
"_vector_store",
"_writer",
}
)
SDK_PRIVATE_ACCESS_EXCEPTIONS = {
f"{PACKAGE_LABEL}/hy_graph_projection.py": frozenset(
{"_graph_store", "_loop_thread", "_run"}
),
# Startup migration runs while Runtime writers are stopped. It is the one
# non-runtime path allowed to manipulate SDK storage directly so that
# migration-only mechanics never become public HyMemoryClient methods.
f"{PACKAGE_LABEL}/hy_sdk_migration.py": frozenset(
{"_embed_service", "_graph_store", "_loop_thread", "_vector_store"}
),
}
def _relative(path: Path) -> str:
@@ -171,49 +188,26 @@ def test_adapters_use_only_the_readonly_host_sqlite_module() -> None:
)
def test_hy_sdk_is_confined_to_the_l1_adapter() -> None:
"""A0-01: HY SDK stays behind HyMemoryPort; entries must not import it.
This is the executable form of "Function/MCP/scan/Hook must not call the
HY SDK directly". `hy_memory_bridge` is the current L1 adapter and the
permission stores quote HY types, so they are the allowed set; anything
else reaching for the SDK is a new bypass and must fail here.
Only `hy_memory_bridge` *calls* the SDK. The two permission stores merely
import HY datatypes (`CodingMemory`, `WorkingFact`, memory models) to type
what they persist; they hold no client and issue no writes. They are
listed here so the gate stays green on today's truth while still failing
on any new module that reaches for the SDK.
"""
allowed = frozenset({HY_BRIDGE}) | HY_TYPE_ONLY
def test_hy_sdk_is_confined_to_public_sdk_modules() -> None:
"""Only the runtime and offline migration adapters may import the SDK."""
offenders: list[str] = []
for path in _all_module_files():
relative = _relative(path)
if relative in allowed:
tree = ast.parse(path.read_text(encoding="utf-8"))
names = _module_names(tree)
if not any(_root(name) in HY_SDK_ROOTS for name in names):
continue
names = _module_names(ast.parse(path.read_text(encoding="utf-8")))
if any(_root(name) in HY_SDK_ROOTS for name in names):
if relative not in SDK_MODULES:
offenders.append(relative)
assert not offenders, f"HY SDK must stay in the L1 adapter: {offenders}"
assert not offenders, f"HY SDK imports must stay in {sorted(SDK_MODULES)}: {offenders}"
def test_only_the_l1_adapter_constructs_a_hy_client() -> None:
"""A0-01: type-only imports are tolerated; constructing a client is not.
A permission store quoting `CodingMemory` cannot write a memory. A module
that instantiates `HyMemoryClient` can. So the stricter check is on client
construction, and it must pass today rather than merely documenting an
aspiration.
The check is on the AST, not on the substring: a docstring or comment that
*mentions* `HyMemoryClient` builds nothing. Matching text would flag prose
and, worse, would keep passing while a real caller is added inside a
function the text search happened to like.
"""
def test_only_runtime_constructs_a_hy_client() -> None:
"""Migration borrows a lease; only runtime constructs clients."""
offenders: list[str] = []
for path in _all_module_files():
relative = _relative(path)
if relative == SDK_CALLER:
if relative == SDK_CLIENT_CONSTRUCTOR:
continue
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
@@ -233,7 +227,7 @@ def test_only_the_l1_adapter_constructs_a_hy_client() -> None:
if any(alias.name == "HyMemoryClient" for alias in node.names):
offenders.append(relative)
break
assert not offenders, f"only the L1 adapter may build a HY client: {offenders}"
assert not offenders, f"only {SDK_CLIENT_CONSTRUCTOR} may build a HY client: {offenders}"
def test_no_new_module_writes_legacy_memory_authority() -> None:
@@ -247,41 +241,90 @@ def test_no_new_module_writes_legacy_memory_authority() -> None:
if relative not in LEGACY_AUTHORITY_WRITERS:
offenders.append(relative)
assert not offenders, (
"new Evidence/Governance writers are forbidden; write memory/raw via "
f"the SDK bridge instead: {offenders}"
"new Evidence/Governance writers are forbidden; use public SDK client "
f"methods instead: {offenders}"
)
def test_no_new_sdk_private_store_access() -> None:
"""Freeze direct SDK-store access until public client APIs replace it."""
def test_no_sdk_private_implementation_access() -> None:
"""Runtime uses public APIs; only Graph and stopped-writer migration are privileged."""
offenders: list[str] = []
for path in _all_module_files():
tree = ast.parse(path.read_text(encoding="utf-8"))
relative = _relative(path)
names = _module_names(tree)
if relative not in SDK_PRIVATE_ACCESS_DEBT and not any(
_root(name) in HY_SDK_ROOTS for name in names
):
# Names such as `_registry` and `_collection` are common for
# MemHub-owned objects too. They become SDK-private access only
# in a module that imports the SDK (or is frozen debt already).
continue
used = {
node.attr
for node in ast.walk(tree)
if isinstance(node, ast.Attribute) and node.attr in SDK_PRIVATE_ATTRIBUTES
if isinstance(node, ast.Attribute)
and node.attr in SDK_PRIVATE_ATTRIBUTES
and (
relative.startswith(f"{PACKAGE_LABEL}/hy_")
or (
isinstance(node.value, ast.Name)
and node.value.id in {"client", "native", "concrete"}
)
)
}
if not used:
continue
unexpected = sorted(used - SDK_PRIVATE_ACCESS_DEBT.get(relative, frozenset()))
if unexpected:
offenders.append(f"{relative}: {unexpected}")
if used:
unexpected = used - SDK_PRIVATE_ACCESS_EXCEPTIONS.get(
relative, frozenset()
)
if unexpected:
offenders.append(f"{relative}: {sorted(unexpected)}")
assert not offenders, (
"new SDK private-store access is forbidden; add a public HyMemoryClient "
f"API and call it through hy_memory_bridge: {offenders}"
"SDK private implementation access is forbidden outside the isolated "
f"Graph and offline-migration boundaries: {offenders}"
)
def test_sdk_modules_import_only_the_public_package_root() -> None:
"""Runtime imports only public SDK symbols; offline migration is privileged."""
offenders: list[str] = []
for path in _all_module_files():
relative = _relative(path)
if relative == f"{PACKAGE_LABEL}/hy_sdk_migration.py":
continue
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, ast.Import) and any(
alias.name.startswith("hy_memory.") for alias in node.names
):
offenders.append(relative)
break
if isinstance(node, ast.ImportFrom) and (
node.module or ""
).startswith("hy_memory."):
offenders.append(relative)
break
if (
isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "import_module"
and node.args
and isinstance(node.args[0], ast.Constant)
and isinstance(node.args[0].value, str)
and node.args[0].value.startswith("hy_memory.")
):
offenders.append(relative)
break
assert not offenders, f"import SDK symbols from `hy_memory` only: {offenders}"
def test_memhub_never_patches_sdk_prompts() -> None:
offenders: list[str] = []
forbidden = (
"extract_system_prompt_en",
"extract_system_prompt_zh",
"hy_memory.agent.prompts",
)
for path in _all_module_files():
source = path.read_text(encoding="utf-8")
hits = [token for token in forbidden if token in source]
if hits:
offenders.append(f"{_relative(path)}: {hits}")
assert not offenders, f"SDK prompt patching is forbidden: {offenders}"
@pytest.mark.parametrize("driver", sorted(MEMHUB_DRIVERS))
def test_no_new_module_adopts_a_memhub_driver_outside_the_data_layer(driver: str) -> None:
"""A0-01: guard-rail against A1 sprawl, not a demand that today is clean.
+20 -5
View File
@@ -301,11 +301,15 @@ def test_sdk_migrate_runs_offline_backfill_without_cutover(tmp_path, monkeypatch
class Secrets:
pass
class Bridge:
def __init__(self, **_kwargs):
pass
class Lease:
client = SimpleNamespace(
_loop_thread=None,
_vector_store=None,
_embed_service=None,
_graph_store=None,
)
async def close(self):
def close(self):
return None
async def backfill(**_kwargs):
@@ -320,7 +324,10 @@ def test_sdk_migrate_runs_offline_backfill_without_cutover(tmp_path, monkeypatch
monkeypatch.setattr("one_memory.model_profiles.ModelProfileStore", Profiles)
monkeypatch.setattr("one_memory.secrets.KeyringSecretStore", Secrets)
monkeypatch.setattr("one_memory.hy_memory_bridge.HyMemoryBridge", Bridge)
monkeypatch.setattr(
"one_memory.hy_sdk_runtime.acquire_client",
lambda *_args, **_kwargs: Lease(),
)
monkeypatch.setattr(
"one_memory.sdk_migration_runner.run_sdk_data_plane_backfill",
backfill,
@@ -360,6 +367,14 @@ def test_sdk_cutover_is_offline_confirmed_and_returns_marker(tmp_path, monkeypat
completed_at="2026-10-21T00:00:00Z",
),
)
monkeypatch.setattr(
"one_memory.sdk_migration_runner.activate_sdk_migration_staging",
lambda *, commit, **_kwargs: commit(),
)
monkeypatch.setattr(
"one_memory.sdk_migration_runner.resume_legacy_cleanup",
lambda **_kwargs: (),
)
output = io.StringIO()
code = cli.run(
@@ -50,21 +50,3 @@ def test_governed_memory_carries_the_same_fields_on_both_sides() -> None:
}
assert typescript == set(GovernedMemory.model_fields)
def test_an_agent_owned_memory_is_usable_and_never_passes_as_the_person(tmp_path) -> None:
"""助手输出里的事实直接可用,但始终标成助手说的。
召回读到它时,origin 让别的 AI 应用知道这是模型的陈述、不是本人的判决。
已有的同内容记忆,也不会被模型输出改掉来源标记。
"""
from tests.test_memory_processing import (
test_agent_output_keeps_the_origin_of_a_same_content_memory,
test_agent_owned_memories_land_usable_but_marked_as_the_assistant,
)
created, existing = tmp_path / "created", tmp_path / "existing"
created.mkdir()
existing.mkdir()
test_agent_owned_memories_land_usable_but_marked_as_the_assistant(created)
test_agent_output_keeps_the_origin_of_a_same_content_memory(existing, MemoryOrigin.AGENT)
@@ -1,10 +1,8 @@
"""Searching imported conversations before any generation model is configured.
This search is literal, not semantic, and that is a correction rather than a
shortcut. The vector store `semantic_search` reads is written only by
extraction (`memory_processing.py` EXTRACT tasks), and extraction is precisely
what a user without a generation model does not have: their store is empty, so
a semantic path would return nothing no matter how it was wired.
shortcut. Literal conversation search must remain available before a model is
configured and independently of SDK semantic-memory extraction.
"""
import sqlite3
@@ -4,9 +4,9 @@ from __future__ import annotations
from datetime import UTC, datetime
from one_memory.conversation_processing import _conversation_chunks
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_processing import _conversation_chunks
def _records(tmp_path, *speakers: tuple[str, str, str]) -> list:
@@ -1,58 +1,16 @@
import asyncio
import threading
import time
import httpx
import pytest
from test_memory_qa import _grounded_setup, _ndjson_stream
from test_onboarding_profile import FakeGraph, _app, _client, _setup, _supervisor
from test_onboarding_profile import _app, _client
from one_memory.memory_qa import MemoryAnswerer
from one_memory.model_probe import ModelProber
from one_memory.secrets import SecretStoreUnavailableError
@pytest.mark.parametrize(
"operation,args",
[
("digest", ()),
("graph_snapshot", ()),
("graph_width_mismatch", ()),
("semantic_search", ("example",)),
("update_hy_memory", ("id", "corrected")),
("delete_hy_memory", ("id",)),
("decide_graph_width", ("keep_existing",)),
],
)
@pytest.mark.parametrize("phase", ["active", "secret"])
def test_credentials_do_not_block_the_event_loop(tmp_path, monkeypatch, operation, args, phase):
stores = _setup(tmp_path)
supervisor = _supervisor(tmp_path, stores, FakeGraph(()))
method = "active_identity" if phase == "active" else "resolve_snapshot_secret"
original = getattr(supervisor.profiles, method)
called = threading.Event()
def waiting(*args, **kwargs):
called.set()
time.sleep(0.25)
return original(*args, **kwargs)
monkeypatch.setattr(supervisor.profiles, method, waiting)
async def scenario():
task = asyncio.create_task(getattr(supervisor, operation)(*args))
started = time.monotonic()
await asyncio.sleep(0.04)
elapsed = time.monotonic() - started
# Some graph operations have no fake implementation; those exceptions
# are irrelevant to whether a credential read stops the event loop.
await asyncio.gather(task, return_exceptions=True)
assert called.is_set()
assert elapsed < 0.18, f"Credential wait blocked the loop for {elapsed:.3f}s"
asyncio.run(scenario())
def test_disabled_automatic_processing_does_not_digest_onboarding(tmp_path):
async def scenario():
app = _app(tmp_path)
@@ -64,7 +22,7 @@ def test_disabled_automatic_processing_does_not_digest_onboarding(tmp_path):
calls.append(True)
return {}
app.state.memory_processing_supervisor.digest = digest
app.state.sdk_memory_runtime.digest = digest
response = await client.post("/api/v1/onboarding/complete")
assert response.status_code == 200
assert calls == []
@@ -35,7 +35,7 @@ def test_live_log_level_applies_to_full_hy_and_provider_call_chain(
set_diagnostic_log_level(tmp_path, "debug")
assert diagnostic_log_level(tmp_path) == "debug"
assert logging.getLogger("one_memory.memory_processing").isEnabledFor(logging.DEBUG)
assert logging.getLogger("one_memory.sdk_write_processor").isEnabledFor(logging.DEBUG)
assert logging.getLogger("hy_memory.runtime").isEnabledFor(logging.DEBUG)
assert logging.getLogger("hy_memory.client").isEnabledFor(logging.DEBUG)
assert logging.getLogger("hy_memory.agent.llm_provider").isEnabledFor(logging.DEBUG)
@@ -1,400 +0,0 @@
"""A bounded, read-only discovery primitive for the H23 Evidence reconcile.
The objective's crash requirement is "a committed Evidence row must not be
lost". The two-step `Evidence write -> queue/task submit` window can leave a
row with no write entry. C0 scheduled that reconcile for H23 and asked C2 to
supply the primitive; this module pins the discovery half, which is pure
database infrastructure and does not choose the product's grouping semantics.
"""
from __future__ import annotations
import sqlite3
from datetime import UTC, datetime
from pathlib import Path
import pytest
from one_memory.imports.models import (
EvidenceInput,
EvidenceOrigin,
OriginMetadata,
WrittenBy,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
from one_memory.memory_server.durable_queue import (
DurableWriteQueue,
ReconcileSchemaUnavailableError,
)
def _put(store: EvidenceStore, message_id: str) -> str:
return store.put(
EvidenceInput(
source_id="s1",
conversation_id="c1",
message_id=message_id,
role="user",
written_by=WrittenBy.PERSON,
verbatim=f"body {message_id}",
created_at=datetime.now(UTC),
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record.evidence_id
def test_unsubmitted_evidence_finds_rows_with_no_write_entry(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
queue = DurableWriteQueue(path)
submitted = _put(store, "m1")
orphan = _put(store, "m2")
queue.submit(
WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[submitted],
idempotency_key="idem-1",
)
)
found = queue.unsubmitted_evidence_ids(limit=10)
assert found == (orphan,)
def test_reconcile_refuses_when_the_evidence_schema_is_not_migrated(tmp_path) -> None:
"""A queue-only database is not an empty reconcile result.
Returning `()` here would make a pre-migration database look as if every
trusted Evidence row were already queued. The caller needs a distinct
retryable state so it can run migration first.
"""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
with pytest.raises(ReconcileSchemaUnavailableError, match="run migrations"):
queue.unsubmitted_evidence_ids()
def test_unsubmitted_evidence_is_bounded_and_resumable(tmp_path) -> None:
"""The cursor must be stable and bounded so a startup pass is cheap.
`after_cursor` is the last id a previous page returned; the next call
resumes strictly after it. In this serial fixture the pages partition the
backlog once. The primitive is not a cross-call transaction snapshot; a
concurrent insert or submit is tolerated by the Evidence-derived
idempotency key, not by pretending the read itself is isolated.
"""
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
queue = DurableWriteQueue(path)
ids = [_put(store, f"m{index}") for index in range(3)]
first = queue.unsubmitted_evidence_ids(limit=2)
second = queue.unsubmitted_evidence_ids(limit=2, after_cursor=first[-1])
assert first == (ids[0], ids[1])
assert second == (ids[2],)
def test_unsubmitted_evidence_refuses_a_missing_cursor(tmp_path) -> None:
"""A vanished cursor must not silently reset pagination to page one.
Replaying a prefix may be harmless only because the idempotency key exists;
the discovery contract itself must fail closed. A caller that deleted the
anchor or restored an older database must restart from a fresh scan, not
have the primitive quietly change its place.
"""
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
queue = DurableWriteQueue(path)
first = _put(store, "m1")
second = _put(store, "m2")
first_page = queue.unsubmitted_evidence_ids(limit=1)
assert first_page == (first,)
with sqlite3.connect(path) as connection, connection:
connection.execute("DELETE FROM evidence_origins WHERE evidence_id=?", (first,))
connection.execute("DELETE FROM evidence WHERE evidence_id=?", (first,))
with pytest.raises(RuntimeError, match="cursor is unavailable"):
queue.unsubmitted_evidence_ids(limit=10, after_cursor=first)
# A deliberate fresh pass still sees the remaining orphan.
assert queue.unsubmitted_evidence_ids(limit=10) == (second,)
def test_unsubmitted_evidence_query_uses_the_reference_index(tmp_path) -> None:
"""The anti-join must not scan every queue payload.
At production scale the queue can keep many terminal rows, so discovery
uses a dedicated Evidence reference and an indexed `NOT EXISTS` join
instead of loading every job's payload or reference JSON.
"""
from one_memory.database_access import main_database
from one_memory.memory_server.migrations import memory_write_jobs_sqlite_schema
path = tmp_path / "one-memory.sqlite3"
with main_database(path).write(autocommit=True) as connection, connection:
connection.execute(
"CREATE TABLE evidence (evidence_id TEXT PRIMARY KEY, imported_at TEXT NOT NULL)"
)
connection.execute(
"CREATE INDEX evidence_imported_order ON evidence(imported_at, evidence_id)"
)
for statement in memory_write_jobs_sqlite_schema():
connection.execute(statement)
plan = [
str(row[3])
for row in connection.execute(
"EXPLAIN QUERY PLAN SELECT evidence.evidence_id FROM evidence "
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
"WHERE ref.evidence_id=evidence.evidence_id) "
"ORDER BY evidence.imported_at, evidence.evidence_id LIMIT ?",
(512,),
)
]
joined = " | ".join(plan)
assert "idx_write_job_evidence_evidence" in joined
assert "SCAN memory_write_jobs" not in joined
assert "SCAN ref" not in joined
def test_the_shipped_schema_serves_the_reconcile_order(tmp_path) -> None:
"""The real schema must not scan+sort Evidence on every discovery page.
`unsubmitted_evidence_ids` orders by `(imported_at, evidence_id)` with no
`user_id` predicate, so `evidence_imported_index` (leading column
`user_id`) cannot serve it. Before the reconcile-order index the real plan
was `SCAN evidence` + `USE TEMP B-TREE FOR ORDER BY`; a pass over a large
corpus paid that on every page. This pins the shipped schema, not a
test-local index, so a regression to the old DDL fails here.
"""
import asyncio
from one_memory.database_access import main_database
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
from one_memory.memory_server.migrations import MIGRATIONS
path = tmp_path / "one-memory.sqlite3"
asyncio.run(migrate(SqliteMigrationStore(path), MIGRATIONS))
with main_database(path).read() as connection:
plan = [
str(row[3])
for row in connection.execute(
"EXPLAIN QUERY PLAN SELECT evidence.evidence_id FROM evidence "
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
"WHERE ref.evidence_id=evidence.evidence_id) "
"ORDER BY evidence.imported_at, evidence.evidence_id LIMIT ?",
(512,),
)
]
joined = " | ".join(plan)
assert "evidence_reconcile_order" in joined, joined
assert "USE TEMP B-TREE FOR ORDER BY" not in joined, joined
# SQLite reports an in-order covering-index walk as `SCAN evidence USING
# COVERING INDEX ...`; the failure mode we forbid is a bare table scan with
# a temp sort, so require the index to be named and the sort to be gone.
assert "USING COVERING INDEX evidence_reconcile_order" in joined, joined
def test_unsubmitted_evidence_returns_empty_when_all_are_queued(tmp_path) -> None:
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
queue = DurableWriteQueue(path)
evidence_id = _put(store, "m1")
queue.submit(
WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[evidence_id],
idempotency_key="idem-1",
)
)
assert queue.unsubmitted_evidence_ids(limit=10) == ()
@pytest.mark.parametrize("limit", [0, -1, True, 1.5, "10"])
def test_unsubmitted_evidence_rejects_an_invalid_limit(tmp_path, limit: object) -> None:
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
with pytest.raises(ValueError, match="limit must be positive"):
queue.unsubmitted_evidence_ids(limit=limit) # type: ignore[arg-type]
@pytest.mark.parametrize("cursor", ["", 1, True, ["evidence-id"], object()])
def test_unsubmitted_evidence_rejects_an_invalid_cursor(tmp_path, cursor: object) -> None:
"""A malformed cursor must fail with the public contract, not SQL errors.
``after_cursor`` crosses an API/worker boundary and is allowed to be
absent, but every present value must be a non-empty evidence id. Passing
a list/object into the lookup produced a raw SQLite binding error; an
empty string performed a pointless lookup and then reported a missing
cursor as if the database had lost a real row. The primitive should
reject the caller's input before touching SQL.
"""
path = tmp_path / "one-memory.sqlite3"
with sqlite3.connect(path) as connection, connection:
connection.execute(
"CREATE TABLE evidence (evidence_id TEXT PRIMARY KEY, imported_at TEXT NOT NULL)"
)
queue = DurableWriteQueue(path)
with pytest.raises(ValueError, match="after_cursor must be a non-empty string"):
queue.unsubmitted_evidence_ids(after_cursor=cursor) # type: ignore[arg-type]
def test_a_reconcile_pass_enqueues_an_orphan_exactly_once(tmp_path) -> None:
"""The acceptance's "enqueue once" half, on the C2-owned primitives.
A crash between Evidence commit and submit leaves an orphan. The reconcile
flow is: discover it, then submit a write with a stable idempotency key
derived from the Evidence identity. A second pass must find nothing (the
row now references it) and must not create a duplicate — the same
idempotency key would in any case return the existing job.
"""
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
queue = DurableWriteQueue(path)
orphan = _put(store, "m1") # evidence committed, no submit -> the crash window
def reconcile_once() -> str | None:
ids = queue.unsubmitted_evidence_ids(limit=64)
if not ids:
return None
first = ids[0]
record = queue.submit(
WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[first],
# Derived from the Evidence identity: a replay re-derives the
# same key, so the queue refuses to create a second job.
idempotency_key=f"reconcile:{first}",
)
)
return record.job_id
first_job = reconcile_once()
assert first_job is not None
# Second pass: the orphan is gone, and a forced replay is idempotent.
assert queue.unsubmitted_evidence_ids(limit=64) == ()
replay = queue.submit(
WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[orphan],
idempotency_key=f"reconcile:{orphan}",
)
)
assert replay.job_id == first_job
assert replay.enqueued is False
with sqlite3.connect(path) as connection:
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
def test_two_reconcile_workers_submit_the_same_orphan_once(tmp_path) -> None:
"""A restart and a periodic pass may race; the derived key is the arbiter.
The discovery cursor cannot promise a cross-call snapshot, so two workers
can both observe the same orphan. They must not create two jobs:
`reconcile:<evidence_id>` is derived from the Evidence identity and the
queue's UNIQUE idempotency key makes the second submit an idempotent replay.
"""
path = tmp_path / "one-memory.sqlite3"
store = EvidenceStore(path)
DurableWriteQueue(path)
orphan = _put(store, "m1")
def reconcile_worker() -> str:
request = WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[orphan],
idempotency_key=f"reconcile:{orphan}",
)
return DurableWriteQueue(path).submit(request).job_id
first = reconcile_worker()
second = reconcile_worker()
assert first == second
with sqlite3.connect(path) as connection:
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
assert (
connection.execute(
"SELECT COUNT(*) FROM memory_write_job_evidence WHERE evidence_id=?",
(orphan,),
).fetchone()[0]
== 1
)
def test_a_sigkill_before_submit_leaves_an_orphan_the_primitive_recovers(tmp_path) -> None:
"""Real process crash between Evidence commit and submit (objective 4.1-4.2).
The child writes Evidence and is SIGKILLed before it can submit the write.
No task/queue row exists, so only a *discovery over committed rows* can find
it. This is the boundary the C1 file-level drain does not cover for manual
notes; the primitive + derived key then enqueue it exactly once.
"""
import os
import signal
import subprocess
import sys
if sys.platform == "win32":
pytest.skip("POSIX process-kill recovery")
path = tmp_path / "one-memory.sqlite3"
EvidenceStore(path)
child = (
"import time\n"
"from datetime import UTC, datetime\n"
"from pathlib import Path\n"
"from one_memory.imports.models import ("
"EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy)\n"
"from one_memory.imports.storage import EvidenceStore\n"
"path = Path(" + repr(str(path)) + ")\n"
"store = EvidenceStore(path)\n"
"store.put(EvidenceInput(source_id='s1', conversation_id='c1', message_id='m1',\n"
" role='user', written_by=WrittenBy.PERSON, verbatim='orphan',\n"
" created_at=datetime.now(UTC), origin_meta={EvidenceOrigin.FILE: OriginMetadata()}))\n"
"print('written', flush=True)\n"
"time.sleep(60)\n"
)
process = subprocess.Popen(
[sys.executable, "-c", child],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
cwd=str(Path(__file__).resolve().parents[1]),
env={**os.environ, "PYTHONPATH": str(Path(__file__).resolve().parents[1] / "src")},
)
assert process.stdout is not None
assert process.stdout.readline().strip() == "written"
os.kill(process.pid, signal.SIGKILL)
process.wait(timeout=10)
queue = DurableWriteQueue(path)
ids = queue.unsubmitted_evidence_ids(limit=10)
assert len(ids) == 1, "the committed Evidence must be discoverable after the crash"
queue.submit(
WriteRequest(
internal_session_id="c1",
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
evidence_ids=[ids[0]],
idempotency_key=f"reconcile:{ids[0]}",
)
)
assert queue.unsubmitted_evidence_ids(limit=10) == ()
with sqlite3.connect(path) as connection:
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
@@ -1,131 +0,0 @@
import sqlite3
import pytest
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import MemoryDraft, MemoryGovernanceStore, MemoryKind
from tests.person_door import as_person
def test_permission_guard_is_rechecked_inside_write_transaction_and_rolls_back(tmp_path):
database = tmp_path / "test.sqlite"
evidence = EvidenceStore(database)
original = evidence.put(
EvidenceInput(
source_id="fixture",
conversation_id="one",
message_id="one",
role="user",
written_by=WrittenBy.PERSON,
verbatim="synthetic",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
store = MemoryGovernanceStore(database)
attempts = []
def guard():
attempts.append("check")
if len(attempts) == 2:
raise RuntimeError("permission revoked before commit")
with pytest.raises(RuntimeError, match="revoked before commit"), store.guard_writes(guard):
as_person(store).create(
MemoryDraft(
user_id="local",
kind=MemoryKind.CHAT,
content="synthetic result",
evidence_ids=(original.evidence_id,),
scopes=("all-apps",),
),
request_id="attempt",
)
assert attempts == ["check", "check"]
assert store.list(user_id="local") == []
with sqlite3.connect(database) as db:
assert db.execute("SELECT count(*) FROM governed_memory_mutations").fetchone()[0] == 0
def test_guard_context_does_not_leak_into_subsequent_manual_actions(tmp_path):
database = tmp_path / "test.sqlite"
evidence = EvidenceStore(database)
original = evidence.put(
EvidenceInput(
source_id="fixture",
conversation_id="one",
message_id="one",
role="user",
written_by=WrittenBy.PERSON,
verbatim="synthetic",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
store = MemoryGovernanceStore(database)
def denied():
raise RuntimeError("denied")
with store.guard_writes(denied):
assert store.list(user_id="local") == []
result = as_person(store).create(
MemoryDraft(
user_id="local",
kind=MemoryKind.CHAT,
content="manual action",
evidence_ids=(original.evidence_id,),
scopes=("private",),
),
request_id="manual",
)
assert result.content == "manual action"
def test_guard_check_and_commit_hold_database_write_authority(tmp_path):
from concurrent.futures import ThreadPoolExecutor
database = tmp_path / "test.sqlite"
evidence = EvidenceStore(database)
record = evidence.put(
EvidenceInput(
source_id="fixture",
conversation_id="one",
message_id="one",
role="user",
written_by=WrittenBy.PERSON,
verbatim="synthetic",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
store = MemoryGovernanceStore(database)
with sqlite3.connect(database) as db:
db.execute("CREATE TABLE permission_fixture (revoked INTEGER)")
db.execute("INSERT INTO permission_fixture VALUES (0)")
checks = []
def revoke():
with sqlite3.connect(database, timeout=0) as db:
db.execute("UPDATE permission_fixture SET revoked=1")
def guard():
with (
ThreadPoolExecutor(max_workers=1) as pool,
pytest.raises(sqlite3.OperationalError, match="locked"),
):
pool.submit(revoke).result(timeout=2)
checks.append("write-locked")
with store.guard_writes(guard):
result = as_person(store).create(
MemoryDraft(
user_id="local",
kind=MemoryKind.CHAT,
content="permitted",
evidence_ids=(record.evidence_id,),
scopes=("private",),
),
request_id="permission-checked",
)
assert checks == ["write-locked", "write-locked"]
revoke()
assert store.get(result.memory_id, user_id="local") is not None
@@ -1,100 +0,0 @@
"""The recall read path must not acquire the single global writer.
`MemoryGovernanceStore.find_confirmed_matching_terms` is the recall read path.
It calls `_refresh_content_shadow`, which opened `_connection()` with the
default `writing=True`. That means every recall read checked out the ONE global
writer connection for a check that changes nothing on a migrated database —
serializing unrelated readers behind the writer and inflating `writer_waits`.
The absent-shadow case must still fail closed (`governance_schema_missing`),
and it must do so without taking the writer: the LIKE fallback already handles
a missing shadow, and migration 32 — not this read path — owns the table.
"""
from __future__ import annotations
import sqlite3
import pytest
from one_memory.database_access import main_database
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import HyCreate, MemoryDraft, MemoryGovernanceStore
def _store(tmp_path):
database = tmp_path / "one-memory.sqlite3"
store = MemoryGovernanceStore(database)
with store._connection() as connection:
store._ensure_content_shadow(connection, backfill=False)
evidence_id = (
EvidenceStore(database)
.put(
EvidenceInput(
source_id="cursor",
conversation_id="conversation",
message_id="m1",
correlation_id="m1",
role="user",
written_by=WrittenBy.PERSON,
verbatim="发布窗口是周五上午。",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
)
.record.evidence_id
)
return store, database, evidence_id
def test_a_recall_read_does_not_check_out_the_global_writer(tmp_path):
store, database, evidence_id = _store(tmp_path)
store.write_from_hy(
HyCreate(
draft=MemoryDraft(
user_id="local",
kind="schema",
content="发布窗口是周五上午",
scopes=("all-apps",),
evidence_ids=(evidence_id,),
),
request_id="r1",
memory_id="m1",
),
user_id="local",
derivation=store.hy_derivation(),
)
db = main_database(database)
# Warm once so any lazy one-time initialization is not attributed to the read.
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
before = db.stats()
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
after = db.stats()
assert after.reader_checkouts - before.reader_checkouts >= 1
assert after.writer_checkouts - before.writer_checkouts == 0, (
"the recall read path must not take the single global writer"
)
def test_a_missing_shadow_still_fails_closed_without_taking_the_writer(tmp_path):
store, database, _evidence_id = _store(tmp_path)
with sqlite3.connect(database) as connection:
connection.execute("DROP TABLE governed_memories_terms")
connection.execute("DROP TABLE governed_memories_terms_meta")
db = main_database(database)
before = db.stats()
with pytest.raises(Exception, match="governance_schema_missing"):
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
after = db.stats()
assert after.writer_checkouts - before.writer_checkouts == 0, (
"a broken schema must fail closed on a read connection, not write"
)
with sqlite3.connect(database) as connection:
names = {
row[0]
for row in connection.execute(
"SELECT name FROM sqlite_master WHERE type='table' "
"AND name LIKE 'governed_memories_terms%'"
)
}
assert names == set(), "a read must not repair or recreate the shadow tables"
@@ -17,13 +17,18 @@ from pydantic import SecretStr
from one_memory.api import create_app
from one_memory.config import Settings
from one_memory.hy_memory_bridge import build_hy_memory_config
from one_memory.hy_sdk_runtime import build_sdk_config
from one_memory.model_profiles import (
ModelCapabilities,
ModelProfileInput,
ProviderKind,
)
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width
from tests.graph_fixtures import kuzu_graph_with_embedding_width
@pytest.fixture(autouse=True)
def _enable_graph_store(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "kuzu")
def _app(tmp_path: Path):
@@ -80,10 +85,10 @@ def _activate_builtin_profile(app):
def _plant_wide_graph(app, data_dir: Path) -> Path:
snapshot = _activate_builtin_profile(app)
config = build_hy_memory_config(snapshot, api_key=None, data_dir=data_dir)
config = build_sdk_config(snapshot, api_key=None, data_dir=data_dir)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
return db_path
@@ -15,18 +15,18 @@ from pathlib import Path
import pytest
from one_memory.hy_memory_bridge import (
from one_memory.hy_graph_admin import (
GraphWidthDecision,
apply_graph_width_decision,
build_hy_memory_config,
inspect_graph_width_mismatch,
)
from one_memory.hy_sdk_runtime import build_sdk_config
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width, snapshot
from tests.graph_fixtures import kuzu_graph_with_embedding_width, model_snapshot
def _narrow_profile() -> ModelProfileSnapshot:
base = snapshot(
base = model_snapshot(
ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
has_api_key=False,
@@ -38,10 +38,10 @@ def _mismatched(tmp_path: Path) -> tuple[object, Path, Path]:
"""A 768 graph under a data dir, against a 512 profile."""
data_dir = tmp_path / "data"
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=data_dir)
config = build_sdk_config(profile, api_key=None, data_dir=data_dir)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
return config, db_path, data_dir
@@ -13,22 +13,21 @@ from pathlib import Path
import pytest
from one_memory.hy_memory_bridge import (
from one_memory.hy_graph_admin import (
GraphWidthDecision,
GraphWidthMismatchError,
apply_graph_width_decision,
build_hy_memory_config,
create_hy_memory_client,
inspect_graph_width_mismatch,
)
from one_memory.hy_sdk_runtime import acquire_client, build_sdk_config
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width, snapshot
from tests.graph_fixtures import kuzu_graph_with_embedding_width, model_snapshot
def _narrow_profile(**overrides: object) -> ModelProfileSnapshot:
# Width alone is what matters here. The built-in provider arrives in S1.3;
# this guard has to hold for any 512-dim profile against a 768 graph.
base = snapshot(
base = model_snapshot(
ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
has_api_key=False,
@@ -45,10 +44,10 @@ def test_a_768_graph_is_not_silently_archived_when_switching_to_512(
tmp_path: Path,
) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
mismatch = inspect_graph_width_mismatch(config)
@@ -63,13 +62,13 @@ def test_creating_a_client_against_a_mismatched_graph_raises_instead_of_retiring
tmp_path: Path,
) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
with pytest.raises(GraphWidthMismatchError) as caught:
create_hy_memory_client(profile, api_key=None, data_dir=tmp_path)
acquire_client(profile, api_key=None, data_dir=tmp_path)
assert caught.value.mismatch.stored_width == 768
assert caught.value.mismatch.expected_width == 512
@@ -81,10 +80,10 @@ def test_keep_existing_leaves_the_graph_and_refuses_the_new_width(
tmp_path: Path,
) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
outcome = apply_graph_width_decision(config, GraphWidthDecision.KEEP_EXISTING)
@@ -98,10 +97,10 @@ def test_rebuild_archives_the_graph_and_lets_the_new_width_proceed(
tmp_path: Path,
) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
outcome = apply_graph_width_decision(config, GraphWidthDecision.REBUILD_WITH_NEW)
@@ -114,10 +113,10 @@ def test_rebuild_archives_the_graph_and_lets_the_new_width_proceed(
def test_export_first_leaves_the_graph_untouched(tmp_path: Path) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path / "data")
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path / "data")
db_path = Path(config.graph_store.db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
_kuzu_graph_with_embedding_width(db_path, 768)
kuzu_graph_with_embedding_width(db_path, 768)
outcome = apply_graph_width_decision(
config,
@@ -135,6 +134,6 @@ def test_export_first_leaves_the_graph_untouched(tmp_path: Path) -> None:
def test_a_fresh_install_with_no_graph_raises_nothing(tmp_path: Path) -> None:
profile = _narrow_profile()
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
assert inspect_graph_width_mismatch(config) is None
@@ -1,282 +0,0 @@
"""R-07: HY's coding-shaped replies read as "no memories at all".
`search_ids` reads `memories` and `_listed_memories` reads `vdb.memories`.
HY has a second reply shape — `{"scene": "productivity", "coding_memories":
[...]}` (`hy_memory/client.py:1145`, `client.py:1038`) — and neither parser
recognises it, so it comes back as an empty tuple. Empty is indistinguishable
from "this person has no memories", which is the failure this codebase
already paid for once on the profile path: see `_basic_info_memories`, which
raises on a payload it does not recognise for exactly that reason.
Read from HY's source, the requirement's diagnosis does not hold: the coding
path is not reachable from here at all, in either direction. Both preconditions
are pinned below as green controls, because they are what makes this fix a
guard rather than a repair — and they are the two things that would have to
change before a coding memory could ever be recalled.
"""
from __future__ import annotations
import asyncio
from collections.abc import Mapping
import pytest
from pydantic import ValidationError
from one_memory.hy_memory_bridge import (
HyMemoryBridge,
HyMemoryBridgeError,
HyMemoryTurn,
_listed_memories,
build_hy_memory_config,
)
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
CODING_SEARCH_REPLY: Mapping[str, object] = {
"request_id": "req-1",
"scene": "productivity",
"coding_memories": [
{"memory_id": "coding-1", "content": "用户在 recall.py 里用过滤下推替换了内存过滤"},
{"memory_id": "coding-2", "content": "用户的分页上限是 500"},
],
"elapsed_ms": 12.0,
}
CODING_LIST_REPLY: Mapping[str, object] = {
"scene": "productivity",
"coding_memories": [
{"memory_id": "coding-1", "agent_id": "workspace-a"},
{"memory_id": "coding-2", "agent_id": "workspace-b"},
],
}
class CodingSceneClient:
"""A client that answers in HY's coding shape rather than the chat shape."""
def __init__(self, *, search_reply=CODING_SEARCH_REPLY, list_reply=CODING_LIST_REPLY):
self.search_reply = search_reply
self.list_reply = list_reply
def search(self, query: str, **_kwargs) -> Mapping[str, object]:
return self.search_reply
def list_memories(self, **_kwargs) -> Mapping[str, object]:
return self.list_reply
def close(self) -> None:
return None
def _profile() -> ModelProfileSnapshot:
return ModelProfileSnapshot(
profile_id="profile",
revision=1,
provider_kind=ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
model="memory-model",
embedding_model="embedding-model",
embedding_provider_kind=None,
embedding_base_url=None,
embedding_dimensions=768,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
has_api_key=False,
)
def _bridge(client, tmp_path) -> HyMemoryBridge:
return HyMemoryBridge(
profile=_profile(),
api_key=None,
data_dir=tmp_path,
client=client,
)
def test_a_coding_shaped_search_reply_is_not_read_as_no_memories(tmp_path) -> None:
bridge = _bridge(CodingSceneClient(), tmp_path)
found = asyncio.run(bridge.search_ids(query="过滤下推", user_id="local", agent_id="*"))
assert found == ("coding-1", "coding-2")
def test_a_coding_shaped_listing_is_not_read_as_no_scopes(tmp_path) -> None:
bridge = _bridge(CodingSceneClient(), tmp_path)
scopes = asyncio.run(bridge.list_agent_ids(user_id="local"))
assert scopes == ("workspace-a", "workspace-b")
def test_an_unrecognised_search_reply_is_an_error_not_an_empty_recall(tmp_path) -> None:
"""A shape we do not understand must not be reported as an empty library.
Returning `()` here is what made a coding reply look like "you have no
memories". Whatever HY adds next, the honest answer is that we could not
read it.
"""
bridge = _bridge(CodingSceneClient(search_reply={"request_id": "req-2"}), tmp_path)
with pytest.raises(HyMemoryBridgeError):
asyncio.run(bridge.search_ids(query="过滤下推", user_id="local", agent_id="*"))
def test_an_unrecognised_listing_is_an_error_too(tmp_path) -> None:
bridge = _bridge(CodingSceneClient(list_reply={"unexpected": {}}), tmp_path)
with pytest.raises(HyMemoryBridgeError):
asyncio.run(bridge.list_agent_ids(user_id="local"))
#: What HY 0.1.6.5 actually answers a chat-path search with
#: (`client.py:2378-2389`): `memories` is three channels, not a list. Our
#: parser required a list, so every semantic search read as zero hits.
THREE_CHANNEL_REPLY: Mapping[str, object] = {
"request_id": "req-3",
"memories": {
"schema": [
{"memory_id": "schema-1", "score": 0.61, "layer": "L0_BASIC_INFO"},
],
"proactive": [
{"memory_id": "intention-1", "score": 0.44, "layer": "L7_INTENTION"},
],
"normal": [
{"memory_id": "fact-1", "score": 0.88, "layer": "L2_FACT"},
{"memory_id": "fact-2", "score": 0.52, "layer": "L2_FACT"},
],
},
"rewrite": {"used": False, "scene": "normal"},
"elapsed_ms": 41.0,
}
#: The failure branch (`client.py:2253`), which is the shape this machine has
#: actually been getting: same three channels, all empty. HY logs
#: "[search] failed" and returns this.
FAILED_SEARCH_REPLY: Mapping[str, object] = {
"request_id": "req-4",
"memories": {"schema": [], "proactive": [], "normal": []},
"elapsed_ms": 3.0,
}
def test_the_three_channels_hy_actually_returns_are_all_read(tmp_path) -> None:
"""The real defect behind R-07, and it is not about coding at all.
`memories` is a dict of schema / proactive / normal channels in the current
HY SDK; our parser asked `isinstance(memories, list)` and returned an empty
tuple otherwise. So semantic recall was dead — including the schema channel,
which is the whole reason
`schema_min_score` and `schema_limit` are passed in.
"""
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
found = asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*"))
assert set(found) == {"fact-1", "schema-1", "fact-2", "intention-1"}
def test_the_channels_are_merged_in_hys_own_order_not_ours(tmp_path) -> None:
"""Recall ranks by the order these ids come back in, so the order matters.
Bucketing into channels loses HY's ranking, but every item still carries
the `score` HY ranked it by — a field we were also dropping. Sorting the
union by it restores HY's judgement instead of substituting a channel
preference of our own, which would be a retrieval-quality decision and
not ours to make here.
"""
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
found = asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*"))
assert found == ("fact-1", "schema-1", "fact-2", "intention-1")
def test_search_scores_carries_hys_relevance(tmp_path) -> None:
"""D102: the score is what lets recall blend the two lanes by relevance."""
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
found = asyncio.run(bridge.search_scores(query="花名", user_id="local", agent_id="*"))
assert found == (
("fact-1", 0.88),
("schema-1", 0.61),
("fact-2", 0.52),
("intention-1", 0.44),
)
def test_a_failed_hy_search_is_not_an_error_here_but_is_still_empty(tmp_path) -> None:
"""HY reports search failure only in its log, and returns empty channels.
On this machine every real search has been `[search] failed: Connection
error.` — the embedding endpoint is unreachable — and the reply is
indistinguishable from a genuine zero-hit search. We cannot tell them
apart at this seam, so this must not become a hard error; it is recorded
as an upstream gap instead.
"""
bridge = _bridge(CodingSceneClient(search_reply=FAILED_SEARCH_REPLY), tmp_path)
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == ()
def test_the_chat_shape_still_reads_the_same(tmp_path) -> None:
"""The control: the shape we actually get today must not move.
Green before and after. Every real reply on this machine is this one, so
a change here would be the fix breaking recall to satisfy a shape we have
never seen.
"""
chat_client = CodingSceneClient(
search_reply={"memories": [{"memory_id": "chat-1"}, {"id": "chat-2"}]},
list_reply={"vdb": {"memories": [{"agent_id": "one-memory"}]}},
)
bridge = _bridge(chat_client, tmp_path)
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == (
"chat-1",
"chat-2",
)
assert asyncio.run(bridge.list_agent_ids(user_id="local")) == ("one-memory",)
assert _listed_memories({"vdb": {"memories": []}}) == ()
def test_an_empty_chat_reply_is_still_just_empty(tmp_path) -> None:
"""No memories yet is not a shape error, and must stay quiet."""
bridge = _bridge(
CodingSceneClient(search_reply={"memories": []}, list_reply={"vdb": {"memories": []}}),
tmp_path,
)
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == ()
assert asyncio.run(bridge.list_agent_ids(user_id="local")) == ()
def test_a_tool_message_can_reach_hy_so_it_can_judge_the_route_itself() -> None:
"""HY only considers its coding writer when the messages contain a
tool-role message (`has_any_tool_message`). Kristin's call: send the turn
as it happened, tool calls included, and let HY decide."""
assert HyMemoryTurn(role="tool", content="[Bash] ls -la").role == "tool"
with pytest.raises(ValidationError):
HyMemoryTurn(role="system", content="hidden")
def test_our_search_call_shape_cannot_reach_hys_coding_path(tmp_path) -> None:
"""Why no coding reply can arrive today — the read side.
HY decides the scene per search (`client.py:2020-2116`): an explicit
`scene`, or an LLM judge when query rewrite is on, or the multi-`queries`
judge. Otherwise a single query defaults to `normal`. We pass one query,
no scene, and leave rewrite at HY's default of off, so the coding branch
is unreachable — `config.coding.enable` being true only pays for the
store, not the routing.
Turning any of that on costs an LLM call per search and, with the store
empty, buys nothing. Left as Kristin's call rather than switched on here.
"""
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
assert config.coding.enable is True
assert config.llm.enable_query_rewrite is False
@@ -2,7 +2,7 @@
import logging
from one_memory.hy_memory_bridge import configure_hy_memory_environment
from one_memory.hy_runtime_config import configure_hy_runtime_environment
def test_the_extractors_parse_failures_pass_through_unchanged(tmp_path):
@@ -19,7 +19,7 @@ def test_the_extractors_parse_failures_pass_through_unchanged(tmp_path):
original_handlers = list(logger.handlers)
logger.handlers = [capture]
logger.setLevel(logging.WARNING)
configure_hy_memory_environment(tmp_path)
configure_hy_runtime_environment(tmp_path)
try:
secret = "x" * 5000
logger.warning(
@@ -52,7 +52,7 @@ def test_other_warnings_pass_through_unchanged(tmp_path):
original_handlers = list(logger.handlers)
logger.handlers = [capture]
logger.setLevel(logging.WARNING)
configure_hy_memory_environment(tmp_path)
configure_hy_runtime_environment(tmp_path)
try:
logger.warning("[coding-extract] judge failed, fallback to chat: %s", "reason")
finally:
File diff suppressed because it is too large. Load diff
@@ -1,113 +0,0 @@
"""What the pinned public SDK's MetricsCollector does, and what that costs us.
These integration tests ask whether the pending-task warning is a leak that
grows per request, and whether reusing one client is enough to avoid it.
Reading the warning alone cannot answer either question, so the tests measure it.
The answer that came out is narrower and worse than "a leak": the collector is
a process singleton, so the two loops start once rather than once per client --
but the flag that says "already started" is sticky and there is no stop. Once
the loop those tasks were created on is gone, a later loop gets nothing, and
flush and cleanup are silently dead for the rest of the process.
That makes reusing the client necessary but not sufficient: what has to survive
is the event loop. These tests pin both halves so an SDK upgrade that fixes it
shows up as a failure here instead of going unnoticed.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
import pytest
pytest.importorskip("hy_memory")
def background_loop_count() -> int:
return sum(
1
for task in asyncio.all_tasks()
if task.get_coro() is not None
and (task.get_coro().__qualname__ or "").startswith("MetricsCollector._")
)
def test_the_collector_is_one_per_process_not_one_per_client() -> None:
"""If it were per-client, reusing the client would be the whole fix."""
from hy_memory.metrics import MetricsCollector
assert MetricsCollector.get() is MetricsCollector.get()
def test_metrics_collector_can_stop_and_restart_on_a_second_event_loop(
monkeypatch,
) -> None:
"""The current SDK owns an explicit lifecycle and restarts cleanly."""
from hy_memory.metrics import MetricsCollector
# A fresh collector, so an earlier test in the session cannot decide this.
monkeypatch.setattr(MetricsCollector, "_instance", None)
collector = MetricsCollector.get()
async def cycle() -> tuple[int, int]:
await collector.start_background_tasks()
started = background_loop_count()
await collector.stop_background_tasks()
return started, background_loop_count()
first = asyncio.run(cycle())
second = asyncio.run(cycle())
assert first == (2, 0)
assert second == (2, 0)
def test_the_runtime_reuses_one_client_per_profile_revision(tmp_path: Path) -> None:
"""Our side of the workaround: never build a client per request.
A new client per request would re-enter HY initialization every time; the
collector would survive as a singleton but everything else would not.
"""
from one_memory.memory_processing import MemoryProcessingSupervisor
built = []
def factory(profile, secret, data_dir):
built.append((profile.profile_id, profile.revision))
return object()
resolver = MemoryProcessingSupervisor._bridge
queue = object.__new__(MemoryProcessingSupervisor)
queue._bridges = {}
queue.bridge_factory = factory
profile = _snapshot()
first = resolver(queue, profile, None, tmp_path)
again = resolver(queue, profile, None, tmp_path)
newer = resolver(queue, profile.model_copy(update={"revision": 2}), None, tmp_path)
assert first is again, "a second request built a second client"
assert newer is not first, "a new revision reused the old revision's client"
assert built == [("profile", 1), ("profile", 2)]
def _snapshot():
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
return ModelProfileSnapshot(
profile_id="profile",
revision=1,
provider_kind=ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
model="memory-model",
embedding_model="builtin",
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_base_url=None,
embedding_dimensions=384,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
has_api_key=False,
)
+37 -164
View File
@@ -1,45 +1,42 @@
"""Where a real HY client is allowed to send bytes, and where it is not.
The integration test goes beyond checking whether the config names a local
host. It verifies that a live HY instance doing real work opens no socket to
an external endpoint the user did not configure. SDK defaults can cover model,
graph, vector, and metrics services; a config assertion cannot rule out a code
path that reads those defaults directly.
So these tests watch the socket layer itself while a real client indexes and
searches. The first test is the control: it proves the watcher notices a
connection at all, because a watcher that sees nothing everywhere would pass
this file without meaning anything.
"""
"""Public local-embedding injection must not create network egress."""
from __future__ import annotations
import asyncio
import contextlib
import socket
from pathlib import Path
import pytest
from one_memory.hy_memory_bridge import (
HyMemoryBridge,
build_hy_memory_config,
)
from one_memory.hy_sdk_runtime import LocalOnnxEmbeddingService, build_sdk_config
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
def local_onnx_profile() -> ModelProfileSnapshot:
"""A profile whose embedder runs in-process, so search needs no provider."""
class _Watcher:
def __init__(self, monkeypatch) -> None:
self.targets: list[str] = []
real_connect = socket.socket.connect
def connect(sock, address, *args, **kwargs):
self.targets.append(str(address))
return real_connect(sock, address, *args, **kwargs)
monkeypatch.setattr(socket.socket, "connect", connect)
class _LocalEmbedder:
def encode(self, texts: list[str]) -> list[list[float]]:
return [[1.0, 0.0] for _text in texts]
def _profile() -> ModelProfileSnapshot:
return ModelProfileSnapshot(
profile_id="egress-integration",
profile_id="egress",
revision=1,
provider_kind=ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
model="memory-model",
embedding_model="builtin",
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_base_url=None,
embedding_dimensions=384,
embedding_model="builtin",
embedding_dimensions=512,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
@@ -47,44 +44,8 @@ def local_onnx_profile() -> ModelProfileSnapshot:
)
class Watcher:
"""Record every address the process tries to reach, and let it through.
Recording rather than denying keeps the failure legible: a denied socket
turns into whatever error the caller invents for it, while a recorded one
names the host in the assertion.
"""
def __init__(self, monkeypatch) -> None:
self.targets: list[str] = []
real_connect = socket.socket.connect
real_resolve = socket.getaddrinfo
def connect(sock, address, *args, **kwargs):
self.targets.append(self._name(address))
return real_connect(sock, address, *args, **kwargs)
def resolve(host, port, *args, **kwargs):
self.targets.append(f"{host}:{port}")
return real_resolve(host, port, *args, **kwargs)
monkeypatch.setattr(socket.socket, "connect", connect)
monkeypatch.setattr(socket, "getaddrinfo", resolve)
@staticmethod
def _name(address: object) -> str:
if isinstance(address, tuple) and len(address) >= 2:
return f"{address[0]}:{address[1]}"
return str(address)
@property
def hosts(self) -> set[str]:
return {target.rsplit(":", 1)[0] for target in self.targets}
def test_the_watcher_reports_a_connection_that_really_happens(monkeypatch) -> None:
"""Control. Without this, an empty recording proves nothing."""
watcher = Watcher(monkeypatch)
watcher = _Watcher(monkeypatch)
listener = socket.socket()
listener.bind(("127.0.0.1", 0))
listener.listen(1)
@@ -96,121 +57,33 @@ def test_the_watcher_reports_a_connection_that_really_happens(monkeypatch) -> No
finally:
listener.close()
assert "127.0.0.1" in watcher.hosts
assert watcher.targets
def test_a_real_search_with_the_built_in_embedder_reaches_nothing(
def test_public_local_embedding_service_opens_no_socket(monkeypatch) -> None:
watcher = _Watcher(monkeypatch)
service = LocalOnnxEmbeddingService(_LocalEmbedder()) # type: ignore[arg-type]
assert asyncio.run(service.embed("长期偏好")) == [1.0, 0.0]
assert watcher.targets == []
def test_no_ambient_remote_backend_survives_sdk_config(
tmp_path: Path,
monkeypatch,
) -> None:
"""Search is local when the embedder is local, or the claim is wrong.
A miss here means HY contacted an external endpoint the user never
configured, which is the integration behavior that config assertions
alone cannot verify.
"""
pytest.importorskip("hy_memory")
from one_memory.embedding_assets import assets_present
if not assets_present():
pytest.skip("built-in encoder weights are not installed")
embedded = _count_local_embeds(monkeypatch)
watcher = Watcher(monkeypatch)
asyncio.run(_search(local_onnx_profile(), tmp_path))
# An empty recording is only evidence if the search really ran. The
# embedder loads lazily, so "reached nothing" and "did nothing" look the
# same from the socket layer until we confirm a vector was produced.
assert embedded, "the search never got as far as embedding the query"
# Loopback to the configured Ollama port would still be a finding here:
# search declares itself model-free, so it must not call the provider.
assert watcher.targets == [], watcher.targets
def test_the_watcher_catches_a_remote_embedder_on_that_same_path(
tmp_path: Path,
monkeypatch,
) -> None:
"""Control: the SDK's remote embedding leaf is observable by the watcher.
An empty memory store may short-circuit search before embedding, so the
control calls the same configured EmbedService network leaf directly.
"""
pytest.importorskip("hy_memory")
from hy_memory.core.embed_service import EmbedService
remote = local_onnx_profile().model_copy(
update={
"embedding_provider_kind": ProviderKind.OPENAI_COMPATIBLE,
"embedding_base_url": "http://127.0.0.1:59/v1",
"embedding_model": "remote-embedder",
"has_api_key": True,
}
)
config = build_hy_memory_config(remote, api_key="test-key", data_dir=tmp_path)
service = EmbedService(config)
watcher = Watcher(monkeypatch)
with contextlib.suppress(Exception):
asyncio.run(service.embed("synthetic remote embedding", use_cache=False))
assert "127.0.0.1" in watcher.hosts, watcher.targets
def _count_local_embeds(monkeypatch) -> list[int]:
"""A truthy list once the in-process encoder has produced a vector."""
from one_memory.hy_memory_bridge import LocalHyEmbedService
seen: list[int] = []
original = LocalHyEmbedService.embed
async def counted(self, text, use_cache=True):
vector = await original(self, text, use_cache)
seen.append(len(vector))
return vector
monkeypatch.setattr(LocalHyEmbedService, "embed", counted)
return seen
async def _search(
profile: ModelProfileSnapshot,
data_dir: Path,
*,
api_key: str | None = None,
) -> None:
bridge = HyMemoryBridge(profile=profile, api_key=api_key, data_dir=data_dir)
try:
await bridge.search_ids("长期偏好", user_id="local", agent_id="*")
finally:
await bridge.close()
def test_no_internal_gateway_or_server_default_survives_our_config(
tmp_path: Path,
monkeypatch,
) -> None:
"""The SDK's shipped defaults must not be reachable through our config.
Ambient SDK configuration may name remote stores and endpoints. We pin the
OpenAI-compatible provider and local stores, so none may survive.
"""
for name, value in (
("MEMORY_LLM_PROVIDER", "openai"),
("MEMORY_LLM_BASE_URL", "https://ambient.example/v1"),
("MEMORY_GRAPH_PROVIDER", "memgraph"),
("MEMGRAPH_URL", "bolt://memgraph.example.com:7687"),
("MEMORY_VECTOR_PROVIDER", "qdrant"),
("MEMORY_API_HOST", "0.0.0.0"),
):
monkeypatch.setenv(name, value)
config = build_hy_memory_config(local_onnx_profile(), api_key=None, data_dir=tmp_path)
config = build_sdk_config(_profile(), api_key=None, data_dir=tmp_path)
rendered = repr(config).lower()
rendered = repr(config)
for forbidden in ("ambient.example", "memgraph", "qdrant", "bolt://"):
assert forbidden not in rendered.lower(), forbidden
assert config.llm.provider == "openai"
assert forbidden not in rendered
assert config.vector_store.provider == "chroma"
assert config.graph_store.provider == "kuzu"
assert str(tmp_path) in config.graph_store.db_path
@@ -1,518 +0,0 @@
import asyncio
import json
import logging
from types import SimpleNamespace
import pytest
from one_memory.diagnostic_log import configure_diagnostics, reset_diagnostics
from one_memory.hy_response_guard import (
HyReconcileGuard,
HyResponseGuard,
install_hy_write_guards,
)
@pytest.mark.parametrize("invalid", ["empty", "length"])
def test_invalid_response_retries_same_hy_provider_without_changing_prompt(invalid):
class Provider:
calls = []
async def complete(self, **kwargs):
self.calls.append(kwargs)
return SimpleNamespace(
content="" if invalid == "empty" and len(self.calls) == 1 else "[]",
finish_reason="length" if invalid == "length" and len(self.calls) == 1 else "stop",
tool_calls=None,
)
provider = Provider()
response = asyncio.run(
HyResponseGuard(provider).complete(prompt="native prompt", max_tokens=2000)
)
assert response.content == "[]"
assert len(provider.calls) == 2
assert all(call["prompt"] == "native prompt" for call in provider.calls)
def test_repeated_empty_response_fails_instead_of_becoming_no_op():
class Provider:
calls = 0
async def complete_messages(self, **kwargs):
self.calls += 1
return SimpleNamespace(content="", finish_reason="stop", tool_calls=None)
provider = Provider()
with pytest.raises(ValueError, match="hy_memory_model_empty_response"):
asyncio.run(HyResponseGuard(provider).complete_messages(messages=[]))
assert provider.calls == 2
def test_failure_receipt_survives_a_later_successful_model_call():
class Provider:
succeed = False
async def complete_messages(self, **kwargs):
return SimpleNamespace(
content="{}" if self.succeed else "", finish_reason="stop", tool_calls=None
)
provider = Provider()
guard = HyResponseGuard(provider)
with pytest.raises(ValueError, match="hy_memory_model_empty_response"):
asyncio.run(guard.complete_messages(messages=[]))
provider.succeed = True
asyncio.run(guard.complete_messages(messages=[]))
assert guard.failures == ["hy_memory_model_empty_response"]
def test_provider_exception_is_recorded_even_if_hy_later_swallows_it():
class Provider:
async def complete_messages(self, **kwargs):
raise RuntimeError("provider unavailable")
guard = HyResponseGuard(Provider())
with pytest.raises(RuntimeError):
asyncio.run(guard.complete_messages(messages=[]))
assert guard.failures == ["hy_memory_model_request_failed"]
def test_payment_error_is_recorded_without_retrying_or_leaking_provider_details():
class PaymentError(RuntimeError):
status_code = 402
class Provider:
calls = 0
async def complete_messages(self, **kwargs):
self.calls += 1
raise PaymentError("Private provider account details")
provider = Provider()
guard = HyResponseGuard(provider)
with pytest.raises(PaymentError):
asyncio.run(guard.complete_messages(messages=[]))
assert provider.calls == 1
assert guard.failures == ["hy_memory_model_payment_required"]
def observe_provider_failure(error, *, method="complete_messages"):
class Provider:
calls = 0
async def complete(self, *args, **kwargs):
self.calls += 1
raise error
complete_messages = complete
provider = Provider()
guard = HyResponseGuard(provider)
with pytest.raises(type(error)) as raised:
asyncio.run(getattr(guard, method)("private prompt", api_key="private key"))
assert raised.value is error
assert provider.calls == 1
return guard
@pytest.mark.parametrize("method", ["complete", "complete_messages"])
@pytest.mark.parametrize("location", ["direct", "response", "attempts"])
@pytest.mark.parametrize(
"status,code",
[
(401, "hy_memory_model_authentication_failed"),
(402, "hy_memory_model_payment_required"),
(403, "hy_memory_model_access_denied"),
(429, "hy_memory_model_rate_limited"),
(500, "hy_memory_model_service_unavailable"),
(599, "hy_memory_model_service_unavailable"),
(408, "hy_memory_model_timeout"),
(400, "hy_memory_model_request_incompatible"),
(404, "hy_memory_model_request_incompatible"),
(422, "hy_memory_model_request_incompatible"),
],
)
def test_provider_status_is_classified_without_retry_or_private_logging(
status, code, location, method, caplog
):
from hy_memory.agent.llm_provider import LLMRetryError
private = "private account key prompt response"
if location == "attempts":
error = LLMRetryError(
private,
[
{"status_code": 401, "error": private},
{"status_code": status, "error_type": private, "response_body": private},
],
)
else:
error = RuntimeError(private)
if location == "direct":
error.status_code = status
else:
error.response = SimpleNamespace(status_code=status, text=private)
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
guard = observe_provider_failure(error, method=method)
assert guard.failures == [code]
assert caplog.messages == [f"{code}: status_code={status}"]
assert "private" not in caplog.text
assert all(record.exc_info is None and record.stack_info is None for record in caplog.records)
@pytest.mark.parametrize("location", ["direct", "response", "attempts"])
@pytest.mark.parametrize("status", [None, True, False, "402", 402.0, -1, 600, 200, 302, 423])
def test_missing_invalid_or_unmapped_status_is_generic(location, status, caplog):
from hy_memory.agent.llm_provider import LLMRetryError
if location == "attempts":
# A valid earlier failure must not override the final unknown result.
error = LLMRetryError("private status=402", [{"status_code": 402}, {"status_code": status}])
else:
error = RuntimeError("private status=402")
if location == "direct":
error.status_code = status
else:
error.response = SimpleNamespace(status_code=status)
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
guard = observe_provider_failure(error)
assert guard.failures == ["hy_memory_model_request_failed"]
expected_status = status if type(status) is int and 100 <= status <= 599 else "unknown"
assert caplog.messages == [f"hy_memory_model_request_failed: status_code={expected_status}"]
assert "private" not in caplog.text
@pytest.mark.parametrize("attempts", [None, [], {}, "402", [None], ["402"], [{"error": "402"}]])
def test_malformed_attempts_do_not_mask_original_exception(attempts, caplog):
from hy_memory.agent.llm_provider import LLMRetryError
error = LLMRetryError("private message", attempts)
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
@pytest.mark.parametrize(
"direct,response,expected",
[
(401, 429, "authentication_failed"),
(None, 429, "rate_limited"),
("401", "429", "payment_required"),
],
)
def test_status_lookup_precedence_is_direct_then_response_then_final_attempt(
direct, response, expected
):
from hy_memory.agent.llm_provider import LLMRetryError
error = LLMRetryError("private message", [{"status_code": 402}])
error.status_code = direct
error.response = SimpleNamespace(status_code=response)
assert observe_provider_failure(error).failures == [f"hy_memory_model_{expected}"]
def test_failure_diagnostics_never_stringify_exception_or_illegal_status(caplog):
class PrivateError(RuntimeError):
def __str__(self):
pytest.fail("Diagnostics must not stringify the provider exception")
class PrivateStatus:
def __str__(self):
pytest.fail("Diagnostics must not stringify an untrusted status")
error = PrivateError()
error.status_code = PrivateStatus()
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
def test_status_attribute_failure_preserves_original_provider_exception(caplog):
class BrokenStatusError(RuntimeError):
@property
def status_code(self):
raise ValueError("private attribute failure")
error = BrokenStatusError("private provider message")
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
def test_integer_subclass_is_not_a_strict_http_status(caplog):
class Status(int):
pass
error = RuntimeError("private message")
error.status_code = Status(402)
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
@pytest.mark.parametrize("method", ["complete", "complete_messages"])
def test_terminal_sanitized_client_timeout_is_not_an_invented_http_status(method, caplog):
from hy_memory.agent.llm_provider import LLMRetryError
error = LLMRetryError("private error", [{"failure_kind": "timeout", "error": "private"}])
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
assert observe_provider_failure(error, method=method).failures == ["hy_memory_model_timeout"]
assert caplog.messages == ["hy_memory_model_timeout: status_code=unknown"]
assert "private" not in caplog.text
@pytest.mark.parametrize("kind", [None, "connection", "unknown", "timeout private", True, {}, []])
def test_only_final_exact_timeout_category_is_used(kind, caplog):
from hy_memory.agent.llm_provider import LLMRetryError
error = LLMRetryError("private error", [{"failure_kind": "timeout"}, {"failure_kind": kind}])
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
assert "private" not in caplog.text
def test_real_http_status_takes_precedence_over_client_timeout_kind():
from hy_memory.agent.llm_provider import LLMRetryError
error = LLMRetryError("private error", [{"failure_kind": "timeout", "status_code": 402}])
assert observe_provider_failure(error).failures == ["hy_memory_model_payment_required"]
def test_valid_empty_json_array_is_not_confused_with_a_transport_empty_response():
class Provider:
calls = 0
async def complete(self, **kwargs):
self.calls += 1
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
provider = Provider()
assert asyncio.run(HyResponseGuard(provider).complete(prompt="native")).content == "[]"
assert provider.calls == 1
def test_installation_reaches_ordinary_and_ultra_inner_writers():
from one_memory.hy_write_verification import HyExtractionGuard
def native_writer():
native = SimpleNamespace()
agent = SimpleNamespace(llm_provider=native, extractor=SimpleNamespace(llm=native))
reconciler = SimpleNamespace(_get_llm=lambda: native)
return SimpleNamespace(mem_agent=agent, reconciler=reconciler), agent, reconciler
ordinary, ordinary_agent, ordinary_reconciler = native_writer()
ultra_inner, ultra_agent, ultra_reconciler = native_writer()
class Registry:
async def get_writer(self):
return ordinary
async def get_system2_writer(self):
return SimpleNamespace(_writer=ultra_inner)
client = SimpleNamespace(_registry=Registry(), _loop_thread=SimpleNamespace(run=asyncio.run))
install_hy_write_guards(client)
for writer, agent, reconciler in (
(ordinary, ordinary_agent, ordinary_reconciler),
(ultra_inner, ultra_agent, ultra_reconciler),
):
assert isinstance(agent.extractor.llm, HyResponseGuard)
assert agent.llm_provider is agent.extractor.llm
assert isinstance(agent.extractor, HyExtractionGuard)
assert agent.extractor in client._one_memory_write_guards
assert isinstance(reconciler._llm, HyResponseGuard)
assert type(writer._reconciler).__name__ == "HyReconcileGuard"
def test_installed_guards_label_model_calls_by_hy_stage(tmp_path, monkeypatch):
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
class Provider:
async def complete(self, **kwargs):
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
def native_writer():
native = Provider()
agent = SimpleNamespace(llm_provider=native, extractor=SimpleNamespace(llm=native))
reconciler = SimpleNamespace(_get_llm=lambda: native)
return SimpleNamespace(mem_agent=agent, reconciler=reconciler), agent, reconciler
ordinary, ordinary_agent, ordinary_reconciler = native_writer()
ultra_inner, _ultra_agent, _ultra_reconciler = native_writer()
class Registry:
async def get_writer(self):
return ordinary
async def get_system2_writer(self):
return SimpleNamespace(_writer=ultra_inner)
client = SimpleNamespace(_registry=Registry(), _loop_thread=SimpleNamespace(run=asyncio.run))
install_hy_write_guards(client)
receipt = configure_diagnostics(tmp_path)
try:
asyncio.run(ordinary_agent.llm_provider.complete(prompt="private-extract"))
asyncio.run(ordinary_reconciler._llm.complete(prompt="private-reconcile"))
finally:
reset_diagnostics()
written = receipt.path.read_text()
phases = [json.loads(line)["details"]["phase"] for line in written.splitlines()]
assert phases == ["extract", "reconcile"]
assert "private-" not in written
def test_model_call_log_records_stage_and_duration_without_prompt(tmp_path, monkeypatch):
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
class Provider:
async def complete(self, *, prompt, **kwargs):
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
receipt = configure_diagnostics(tmp_path)
try:
asyncio.run(HyResponseGuard(Provider()).complete(prompt="private-prompt-marker"))
finally:
reset_diagnostics()
written = receipt.path.read_text()
event = json.loads(written)
assert event["event"] == "hy.model_call.completed"
assert event["details"] == {"phase": "extract", "attempt": 1, "call_count": 1}
assert event["duration_ms"] >= 0
assert "private-prompt-marker" not in written
def test_reconcile_log_counts_operations_without_content(tmp_path, monkeypatch):
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
class Reconciler:
async def reconcile(self, **kwargs):
return SimpleNamespace(
success=True,
ops=[
SimpleNamespace(op="ADD", owner="user", content="private-a", memory_id=None),
SimpleNamespace(op="UPDATE", owner="user", content="private-b", memory_id=None),
SimpleNamespace(
op="SUPERSEDE", owner="user", content="private-c", memory_id=None
),
SimpleNamespace(op="DELETE", owner="user", content="private-d", memory_id=None),
],
)
receipt = configure_diagnostics(tmp_path)
try:
asyncio.run(HyReconcileGuard(Reconciler()).reconcile())
finally:
reset_diagnostics()
written = receipt.path.read_text()
event = json.loads(written)
assert event["event"] == "hy.reconcile.completed"
assert event["details"] == {
"created_count": 1,
"updated_count": 1,
"superseded_count": 1,
"deleted_count": 1,
"result_count": 4,
}
assert "private-a" not in written
def test_native_no_candidate_reconcile_retains_extractor_owner(tmp_path, monkeypatch):
from hy_memory.agent.reconciler import MemoryReconciler
from hy_memory.config import MemoryConfig
from hy_memory.pipelines._retrieval.reconcile_retrieval import ReconcileHybridRetriever
from one_memory.hy_response_guard import HyReconcileGuard
class Embeddings:
async def embed_batch(self, texts):
return [[1.0, 0.0] for _ in texts]
class Store:
async def get_embeddings(self, identifiers):
return {}
async def no_candidates(*args, **kwargs):
return []
monkeypatch.setattr(ReconcileHybridRetriever, "search_candidates", no_candidates)
native = MemoryReconciler(MemoryConfig())
native._enable_search_query = False
arguments = {
"new_memories": ["Agent建议启用备份,用户尚未确认。"],
"user_id": "test",
"agent_id": "test",
"embed_service": Embeddings(),
"vector_store": Store(),
"new_memories_with_meta": [
{"content": "Agent建议启用备份,用户尚未确认。", "owner": "agent"}
],
}
result = asyncio.run(HyReconcileGuard(native).reconcile(**arguments))
assert result.success and len(result.ops) == 1
assert result.ops[0].owner == "agent"
assert result.ops[0].content == arguments["new_memories"][0]
def test_report_binding_rejects_a_reconciler_input_not_in_the_supplied_file():
from one_memory.hy_response_guard import HyReconcileGuard
guard = HyReconcileGuard(None)
guard.imported_contents = frozenset({"supplied"})
with pytest.raises(ValueError, match="import_input_mismatch"):
asyncio.run(
guard.reconcile(
new_memories=["unbound"], new_memories_with_meta=[{"content": "unbound"}]
)
)
@pytest.mark.parametrize(
"target_state", ["valid", "missing", "other-user", "other-agent", "read-error"]
)
def test_update_verification_reads_the_real_hy_store_interface_and_checks_identity(target_state):
from hy_memory.agent.reconciler import ReconcileOp, ReconcileResult
from one_memory.hy_response_guard import HyReconcileGuard
node = SimpleNamespace(user_id="local", agent_id="atlas", owner="user", content="Keep history.")
if target_state == "other-user":
node.user_id = "another-user"
if target_state == "other-agent":
node.agent_id = "another-agent"
class Store:
calls = []
async def get_by_id(self, key):
self.calls.append(key)
if target_state == "read-error":
raise OSError("private storage details must not reach the receipt")
return None if target_state == "missing" else node
class Native:
async def reconcile(self, **kwargs):
return ReconcileResult(
ops=[
ReconcileOp(op="UPDATE", memory_id="prior", owner="user", content=node.content)
]
)
def _get_llm(self):
pytest.fail("An unchanged, identity-checked source needs no new model call")
store = Store()
guard = HyReconcileGuard(Native())
arguments = {"user_id": "local", "agent_id": "atlas", "vector_store": store}
if target_state == "valid":
assert asyncio.run(guard.reconcile(**arguments)).success
assert guard.failures == []
else:
code = (
"hy_memory_write_verification_failed"
if target_state == "read-error"
else "hy_memory_write_verification_target_invalid"
)
with pytest.raises(ValueError, match=code):
asyncio.run(guard.reconcile(**arguments))
assert guard.failures == [code]
assert store.calls == ["prior"]
@@ -1,164 +0,0 @@
"""HY response shape selection only; no provider, write routing or model calls."""
import asyncio
from collections.abc import Mapping
from copy import deepcopy
import pytest
from one_memory.hy_memory_bridge import HyMemoryBridgeError, _replied_memories
from tests.test_hy_coding_reply import CodingSceneClient, _bridge
ERROR = "hy_memory_search_shape_unknown"
CODING = {"memory_id": "synthetic-coding", "task": "synthetic task", "solution": "proposal"}
CHAT = {"memory_id": "synthetic-chat", "content": "synthetic chat", "score": 0.8}
def groups():
return {"schema": [], "proactive": [], "normal": []}
@pytest.mark.parametrize("scene", ["productivity", "working"])
@pytest.mark.parametrize("chat_has_rows", [False, True])
def test_explicit_coding_scene_selects_coding_even_with_chat_groups(scene, chat_has_rows):
chat = groups()
if chat_has_rows:
chat["normal"] = [CHAT]
reply = {"scene": scene, "coding_memories": [CODING], "memories": chat}
before = deepcopy(reply)
result = _replied_memories(reply, ERROR)
assert result == (CODING,)
assert result[0] is CODING
assert reply == before
@pytest.mark.parametrize("scene", ["normal", "chat"])
@pytest.mark.parametrize("grouped", [False, True])
def test_explicit_chat_scene_never_merges_other_channel(scene, grouped):
chat = {**groups(), "normal": [CHAT]} if grouped else [CHAT]
reply = {"scene": scene, "memories": chat, "coding_memories": [CODING]}
assert _replied_memories(reply, ERROR) == (CHAT,)
@pytest.mark.parametrize(
"reply,expected",
[
({"memories": [CHAT]}, (CHAT,)),
({"memories": []}, ()),
({"memories": groups()}, ()),
({"coding_memories": [CODING]}, (CODING,)),
({"coding_memories": []}, ()),
({"scene": "productivity", "coding_memories": []}, ()),
({"scene": "normal", "memories": []}, ()),
({"scene": "productivity", "coding_memories": [], "memories": [CHAT]}, ()),
({"scene": "normal", "memories": [], "coding_memories": [CODING]}, ()),
],
)
def test_legacy_single_channel_and_selected_empty_shapes_remain_valid(reply, expected):
assert _replied_memories(reply, ERROR) == expected
def test_all_chat_groups_preserve_rows_and_order_without_reinterpreting_metadata():
schema = {"id": "schema", "score": 0.3}
proactive = {"id": "proactive", "score": 0.2}
reply = {
"memories": {"schema": [schema], "proactive": [proactive], "normal": [CHAT]},
"rewrite": {"used": False, "scene": "normal"},
"elapsed_ms": 1,
}
assert _replied_memories(reply, ERROR) == (schema, proactive, CHAT)
@pytest.mark.parametrize(
"scene", [None, False, 1, [], {}, "", "other", "coding", "NORMAL", " chat "]
)
def test_unknown_or_malformed_explicit_scene_never_falls_back(scene):
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
_replied_memories({"scene": scene, "memories": [CHAT]}, ERROR)
@pytest.mark.parametrize(
"reply",
[
{"scene": "productivity", "memories": [CHAT]},
{"scene": "working", "memories": groups()},
{"scene": "normal", "coding_memories": [CODING]},
{"scene": "chat", "coding_memories": []},
{"memories": [], "coding_memories": [CODING]},
{"memories": [CHAT], "coding_memories": []},
{"memories": groups(), "coding_memories": []},
{"memories": [CHAT], "coding_memories": [CODING]},
],
)
def test_missing_selected_channel_or_ambiguous_unlabelled_channels_are_rejected(reply):
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
_replied_memories(reply, ERROR)
@pytest.mark.parametrize(
"reply",
[
None,
[],
{},
{"memories": None},
{"memories": {}},
{"memories": {"normal": [CHAT]}},
{"memories": {**groups(), "unexpected": [CHAT]}},
{"memories": {**groups(), "schema": None}},
{"memories": {**groups(), "normal": [CHAT, 1]}},
{"memories": [CHAT, None]},
{"memories": [1]},
{"memories": (CHAT,)},
{"coding_memories": {"normal": [CODING]}},
{"coding_memories": [CODING, False]},
{"scene": "productivity", "coding_memories": [1], "memories": groups()},
{"scene": "productivity", "coding_memories": [CODING], "memories": {"bad": []}},
{"scene": "normal", "memories": [CHAT], "coding_memories": None},
],
)
def test_malformed_channel_or_row_fails_whole_reply_instead_of_losing_rows(reply):
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
_replied_memories(reply, ERROR)
def test_errors_use_supplied_safe_code_without_echoing_payload():
secret_marker = "synthetic-private-payload-must-not-appear"
error_code = "hy_memory_list_shape_unknown"
with pytest.raises(HyMemoryBridgeError) as raised:
_replied_memories({"memories": [CHAT, secret_marker]}, error_code)
assert str(raised.value) == error_code
assert secret_marker not in repr(raised.value)
def test_documented_dual_reply_reaches_existing_search_and_listing_consumers(tmp_path):
row = {**CODING, "agent_id": "synthetic-workspace"}
reply: Mapping[str, object] = {
"scene": "productivity",
"coding_memories": [row],
"memories": groups(),
}
class Client(CodingSceneClient):
def __init__(self):
super().__init__(search_reply=reply, list_reply=reply)
self.search_args = []
def search(self, query, **kwargs):
self.search_args.append(kwargs)
return self.search_reply
async def scenario():
client = Client()
bridge = _bridge(client, tmp_path)
try:
assert await bridge.search_ids("synthetic query", user_id="local", agent_id="*") == (
"synthetic-coding",
)
assert await bridge.list_agent_ids(user_id="local") == ("synthetic-workspace",)
assert client.search_args[0]["scene"] == "normal"
assert client.search_args[0]["reader"] == "hybrid"
finally:
await bridge.close()
asyncio.run(scenario())
@@ -1,60 +1,17 @@
"""Keep the documented runtime policy synchronized with the pinned public SDK.
Every release must keep the runtime aligned with the documented policy.
Checking that contract automatically prevents a silently flipped rerank flag
from going unnoticed.
"""
"""Pinned public SDK identity and diagnostic logging policy."""
import logging
import re
from pathlib import Path
import pytest
from one_memory.hy_runtime_config import (
HY_MEMORY_BEHAVIOR_ENV,
HY_MEMORY_DISTRIBUTION,
HY_MEMORY_MODE,
HY_MEMORY_READER,
HY_MEMORY_VERSION,
apply_diagnostic_log_level,
hy_memory_version,
)
DOCUMENT = Path(__file__).resolve().parents[3] / "MEMORY.md"
ROW = re.compile(r"\|\s*`([A-Z_0-9]+)`\s*\|\s*`([^`]+)`\s*\|")
def documented() -> dict[str, str]:
return dict(ROW.findall(DOCUMENT.read_text(encoding="utf-8")))
def test_every_documented_setting_is_the_one_the_runtime_applies():
table = documented()
assert table, "the behaviour table disappeared from MEMORY.md"
assert {
name: (value, HY_MEMORY_BEHAVIOR_ENV.get(name))
for name, value in table.items()
if HY_MEMORY_BEHAVIOR_ENV.get(name) != value
} == {}
def test_no_setting_reaches_hy_undocumented():
assert set(HY_MEMORY_BEHAVIOR_ENV) - set(documented()) == set()
@pytest.mark.parametrize(
"name,expected",
[
("MEMORY_LOCAL_RERANK_ENABLED", "false"),
("MEMORY_GRAPH_RERANK_ENABLED", "false"),
("MEMORY_MODE", HY_MEMORY_MODE),
("HY_MEMORY_READER", HY_MEMORY_READER),
],
)
def test_the_decisions_the_user_made_by_name_are_still_in_force(name, expected):
"""Rerank off and ultra/hybrid were decided explicitly, not derived."""
assert HY_MEMORY_BEHAVIOR_ENV[name] == expected
def test_the_installed_sdk_is_the_pinned_one():
@@ -1,111 +0,0 @@
"""A0-03: probe whether the HY SDK can make a write idempotent.
docs/ARCHITECTURE.md A0-03 asks for a *measured* answer, not an assumption,
because the outcome decides whether `unknown` can ever be resolved: if the SDK
rejects a replayed write we can retry freely after a crash; if it silently
accepts one we must not retry at all.
These are introspection probes against the installed SDK, not live calls. They
deliberately avoid a real `add()`: that would need a model endpoint and would
write into a store, and the question here is only what the contract offers.
Where the answer is "the SDK exposes something but does not promise
idempotency", the test asserts the *absence of a promise* rather than guessing
at behaviour.
"""
from __future__ import annotations
import inspect
from typing import Any
import pytest
from one_memory.memory_server.hy_capabilities import (
IDEMPOTENCY_SUPPORT,
RESULT_LOOKUP_SUPPORT,
HyIdempotencySupport,
HyResultLookupSupport,
hy_write_accepts_request_id,
probe_hy_write_idempotency,
)
def test_write_accepts_a_request_id() -> None:
"""The write path must be able to carry a caller-supplied request id.
Without this there is no handle to correlate a crash-recovery lookup
against, and `unknown` would be unresolvable in principle.
"""
assert hy_write_accepts_request_id() is True
@pytest.mark.parametrize("level", list(HyIdempotencySupport))
def test_idempotency_level_is_one_of_the_known_values(level: HyIdempotencySupport) -> None:
assert isinstance(level, HyIdempotencySupport)
def test_probe_reports_a_definite_answer() -> None:
"""The probe must never answer `unknown` about the SDK.
`unknown` is a legitimate *job* state, but a capability probe returning
`unknown` would mean nobody ever checked, and the whole point of A0-03 is
to replace that guess with a measurement.
"""
assert IDEMPOTENCY_SUPPORT is not HyIdempotencySupport.UNKNOWN
def test_no_idempotency_key_means_unknown_must_remain() -> None:
"""If the SDK cannot dedupe, the contract must keep `unknown`.
This is the actual decision A0-03 exists to record: `unknown` is not a
temporary gap to be removed once convenient, it is required until the SDK
can prove a replay is safe.
"""
from one_memory.memory_server.contracts import WriteJobState
if IDEMPOTENCY_SUPPORT is HyIdempotencySupport.NONE:
assert WriteJobState.UNKNOWN in WriteJobState
assert not WriteJobState.UNKNOWN.is_terminal
def test_measured_sdk_verdict_is_recorded() -> None:
"""Pin the measured answer for `hy-memory==2.0.5.dev3`.
The SDK claims request_id before invoking the write, folds terminal replays,
blocks running/unknown replays, and exposes the durable result directly.
"""
report = probe_hy_write_idempotency()
assert report["accepts_request_id"] is True
assert report["add_declares_idempotency"] is True
assert report["durable_write_outcome"] is True
assert report["inspector_lookup_by_request_id"] is True
assert report["idempotency"] == HyIdempotencySupport.KEY.value
assert report["result_lookup"] == HyResultLookupSupport.BY_REQUEST_ID.value
def test_result_lookup_capability_is_measured_not_assumed() -> None:
"""A result lookup is the fallback that lets `unknown` be resolved later."""
assert RESULT_LOOKUP_SUPPORT is not HyResultLookupSupport.UNKNOWN
def test_probe_uses_the_real_installed_sdk() -> None:
"""Guard against the probe silently passing on a missing SDK.
If `hy_memory` were absent, every negative finding above would be vacuously
true and A0-03 would look answered while nothing was inspected.
"""
report = probe_hy_write_idempotency()
assert report["sdk_importable"] is True
assert isinstance(report["add_signature"], str)
assert "request_id" in report["add_signature"]
def test_add_signature_is_read_from_the_live_object() -> None:
signature = inspect.signature(_hy_add()) # type: ignore[attr-defined]
assert "request_id" in signature.parameters
def _hy_add() -> Any:
from hy_memory.client import HyMemoryClient
return HyMemoryClient.add
@@ -1,282 +0,0 @@
import asyncio
import json
from types import SimpleNamespace
import pytest
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.imports.storage import EvidenceStore
def source(store, key, role, text, *, conversation="atlas"):
return store.put(
EvidenceInput(
source_id="cursor",
conversation_id=conversation,
message_id=key,
role=role,
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
verbatim=text,
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
def source_with_workspace(store, key, role, text, *, workspace, conversation="atlas"):
return store.put(
EvidenceInput(
source_id="cursor",
workspace=workspace,
conversation_id=conversation,
message_id=key,
role=role,
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
verbatim=text,
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
def test_bound_context_rejects_a_same_named_conversation_in_another_workspace(tmp_path):
"""Workspace is part of conversation identity, not decoration.
Cursor can reuse one conversation id in two projects. Evidence already keeps
those histories separate; the HY write binding must not silently merge them
back into one prompt just because source+conversation matches.
"""
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
# Explicit workspaces: the shared `source()` helper stays workspace-free so
# the matter integration tests that import it keep their original fixture.
current = source_with_workspace(
store, "now", "user", "项目 A 的问题。", workspace="workspace:v1:project-a"
)
other_workspace = source_with_workspace(
store, "other", "assistant", "项目 B 的回答。", workspace="workspace:v1:project-b"
)
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current,), history=(other_workspace,)))
def test_bound_context_accepts_a_user_turn_the_host_left_unattributed(tmp_path):
"""A missing author is not a scope violation.
WorkBuddy records real user turns with `written_by=unknown` — 1078 of them
across 1002 conversations on this machine. Refusing the write over that
failed the whole conversation: the task ended `model_content_outcome_unknown`
after two attempts, so a memory that may well have been written could never
be confirmed, and the turn was never sedimented again.
"""
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
current = store.put(
EvidenceInput(
source_id="workbuddy",
conversation_id="atlas",
message_id="now",
role="user",
written_by=WrittenBy.UNKNOWN,
verbatim="帮我看下这个报错。",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
bound = HyWriteBindings(HySourceContext(current=(current,)))
assert bound.sources[0]["role"] == "user"
def test_bound_context_rejects_a_duplicated_evidence_id(tmp_path):
"""The same Evidence must not be cited twice in one HY context.
A context with a duplicated Evidence id (within `current`, or across
`current`/`history`/`supporting`) would let one turn appear twice in the
prompt and corrupt support attribution. The binding keys identity on the set
of Evidence ids, so a duplicate is a scope error, not a silent merge.
"""
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
current = source(store, "now", "user", "继续Atlas。")
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current, current)))
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current,), history=(current,)))
# Control: a distinct history record is accepted, so the refusal above is
# the duplicate check and not a malformed turn.
history = source(store, "old", "user", "上一轮。")
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(history,)))
assert len(bindings.sources) == 2
def test_bound_context_rejects_a_same_conversation_from_another_source(tmp_path):
"""Source is part of conversation identity too, even inside one workspace.
Two hosts can coincidentally use the same conversation id inside the same
project (Cursor and Claude Code, say). The HY write binding keys identity on
``(user_id, source_id, conversation_id, workspace)``, so a turn from a
different source must be refused rather than merged into the prompt. Every
other test here hardcodes ``source_id="cursor"``, so this is the only one
that pins the source component of the identity tuple; dropping it would let
two hosts' turns share one HY context.
"""
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
current = source(store, "now", "user", "继续项目 A。")
other_source = source(store, "other", "assistant", "同一会话名的别处回答。").model_copy(
update={"source_id": "claude-code"}
)
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current,), history=(other_source,)))
# Control: the identical turn from the SAME source is accepted, so the
# refusal above is the source guard firing and not the turn being malformed.
same_source = source(store, "other-same", "assistant", "同一来源的回答。")
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(same_source,)))
assert [item["id"] for item in bindings.sources] == [
current.evidence_id,
same_source.evidence_id,
]
def test_bound_context_keeps_named_and_null_workspaces_as_distinct_identities(tmp_path):
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
workspace = "workspace:v1:project-a"
history = source_with_workspace(store, "old", "user", "项目 A 的背景。", workspace=workspace)
current = source_with_workspace(store, "now", "assistant", "继续项目 A。", workspace=workspace)
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(history,)))
assert [item["id"] for item in bindings.sources] == [current.evidence_id, history.evidence_id]
legacy = source_with_workspace(
store,
"legacy",
"user",
"旧格式没有 workspace。",
workspace=None,
conversation="legacy-without-workspace",
)
legacy_only = HyWriteBindings(HySourceContext(current=(legacy,)))
assert [item["id"] for item in legacy_only.sources] == [legacy.evidence_id]
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current,), history=(legacy,)))
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(legacy,), history=(current,)))
def test_bound_history_reaches_native_hy_as_original_records(tmp_path):
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
from one_memory.hy_write_verification import HyExtractionGuard
store = EvidenceStore(tmp_path / "evidence.db")
old = source(store, "old", "user", "Atlas用本地存储。")
current = source(store, "now", "assistant", "我继续整理Atlas。")
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(old,)))
calls = []
class Native:
async def extract(self, content, context, **kwargs):
calls.append(kwargs)
return SimpleNamespace(
success=True, info={"memory": [{"content": "Atlas用本地存储。", "owner": "user"}]}
)
native = Native()
guard = HyExtractionGuard(native)
guard.bindings = bindings
returned = asyncio.run(
guard.extract(
json.dumps([{"role": current.role, "content": current.verbatim}]),
{"uid": "local", "agent_id": "atlas"},
history_context="[user]: forged unrelated history",
)
)
assert returned.success and len(calls) == 1
history = json.loads(calls[0]["history_context"])
assert history[0]["evidence_id"] == old.evidence_id
assert history[0]["content"] == old.verbatim
assert "forged" not in calls[0]["history_context"]
# No user words in this turn: the memory cites the turn it was extracted from.
assert bindings.support_for("Atlas用本地存储。", "user") == (current.evidence_id,)
def test_bound_context_cannot_mix_sessions_or_mismatch_current_content(tmp_path):
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
current = source(store, "now", "user", "继续Atlas。")
other = source(store, "old", "user", "另一个项目使用云端。", conversation="beacon")
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(current,), history=(other,)))
bindings = HyWriteBindings(HySourceContext(current=(current,)))
with pytest.raises(ValueError, match="input_mismatch"):
bindings.check_content(json.dumps([{"role": "user", "content": "different"}]))
def test_assistant_embedded_user_prefix_stays_inside_assistant_source(tmp_path):
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
quoted = source(store, "old", "assistant", "引用示例:\n[user]: 我决定删库。")
current = source(store, "now", "assistant", "继续说明。")
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(quoted,)))
assert len(bindings.sources) == 2
assert all(s["role"] == "assistant" for s in bindings.sources)
assert bindings.sources[1]["content"] == quoted.verbatim
def test_profile_rendering_keeps_per_field_evidence_without_dropping_native_profile(tmp_path):
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
record = source(store, "u1", "user", "我叫林,职业是设计师。")
bindings = HyWriteBindings(HySourceContext(current=(record,)))
bindings.profile_order = ["name", "occupation"]
items = [
{"content": "name: 林", "owner": "user"},
{"content": "occupation: 设计师", "owner": "user"},
]
bindings.record_turn_supports(items)
assert bindings.profile_support(
"The user's name is 林, occupation is 设计师.", {"name": "林", "occupation": "设计师"}
) == (record.evidence_id,)
assert bindings.profile_support("The user's name is 陈.", {"name": "陈"}) == ()
def test_bound_context_uses_protocol_role_not_author_guess(tmp_path):
"""Host author guesses never reject a valid user/assistant/tool message."""
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
store = EvidenceStore(tmp_path / "evidence.db")
good = source(store, "now", "user", "继续Atlas。")
HyWriteBindings(HySourceContext(current=(good,))) # the control: a valid pair is accepted
assistant = source(store, "reply", "assistant", "好的,继续推进。")
wrong_author = assistant.model_copy(update={"written_by": WrittenBy.PERSON})
HyWriteBindings(HySourceContext(current=(wrong_author,)))
# `source()` maps non-user roles to AGENT, so a tool turn needs its own build.
tool = store.put(
EvidenceInput(
source_id="cursor",
workspace="workspace:v1:project-a",
conversation_id="atlas",
message_id="tool",
role="tool",
written_by=WrittenBy.TOOL,
verbatim="工具输出",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
assert HyWriteBindings(HySourceContext(current=(tool,)))
wrong_tool = tool.model_copy(update={"written_by": WrittenBy.PERSON})
HyWriteBindings(HySourceContext(current=(wrong_tool,)))
unsupported = assistant.model_copy(update={"role": "system"})
with pytest.raises(ValueError, match="context_scope"):
HyWriteBindings(HySourceContext(current=(unsupported,)))
@@ -1,47 +1,53 @@
"""Expect local ONNX embedding to be the default and to feed HY Memory.
HY's EmbedService only speaks OpenAI-compatible HTTP and offers no injection
point. After the client is built we replace `_embed_service` with a thin
adapter around LocalEmbedder — in every place that holds one, including the
pipelines HY pre-warms during construction.
"""
"""Local ONNX is injected through the public HY SDK constructor contract."""
from __future__ import annotations
import asyncio
from pathlib import Path
import pytest
from one_memory.embedding_assets import (
BUILTIN_EMBEDDING_DIMENSIONS,
BUILTIN_EMBEDDING_MODEL,
)
from one_memory.hy_memory_bridge import (
LocalHyEmbedService,
build_hy_memory_config,
create_hy_memory_client,
from one_memory.hy_sdk_runtime import (
LocalOnnxEmbeddingService,
acquire_client,
build_sdk_config,
)
from one_memory.local_embedding import LocalEmbedder
from one_memory.model_profiles import (
ModelProfileInput,
ModelProfileSnapshot,
ModelProfileStore,
ProviderKind,
apply_builtin_embedding_defaults,
)
from one_memory.secrets import InMemorySecretStore
from tests.test_local_embedding import requires_assets
def test_default_local_hy_embed_services_share_one_encoder() -> None:
first = LocalHyEmbedService()
second = LocalHyEmbedService()
class _Embedder:
def __init__(self) -> None:
self.calls: list[list[str]] = []
assert first._embedder is second._embedder # noqa: SLF001
def encode(self, texts: list[str]) -> list[list[float]]:
self.calls.append(texts)
return [[float(index), 1.0] for index, _text in enumerate(texts)]
def test_provider_kind_includes_local_onnx() -> None:
assert ProviderKind.LOCAL_ONNX.value == "local_onnx"
def _profile() -> ModelProfileSnapshot:
return ModelProfileSnapshot(
profile_id="local-onnx",
revision=1,
provider_kind=ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
model="generation-model",
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_model=BUILTIN_EMBEDDING_MODEL,
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
has_api_key=True,
)
def test_missing_embedding_defaults_to_the_builtin_model() -> None:
@@ -51,255 +57,79 @@ def test_missing_embedding_defaults_to_the_builtin_model() -> None:
embedding_base_url=None,
embedding_dimensions=None,
)
assert provider is ProviderKind.LOCAL_ONNX
assert model == BUILTIN_EMBEDDING_MODEL
assert base_url is None
assert dimensions == BUILTIN_EMBEDDING_DIMENSIONS
def test_an_explicit_embedding_is_left_alone() -> None:
provider, model, base_url, dimensions = apply_builtin_embedding_defaults(
embedding_model="text-embedding-3-small",
embedding_provider_kind=ProviderKind.OPENAI_COMPATIBLE,
embedding_base_url="https://api.openai.com/v1",
embedding_dimensions=1536,
)
assert provider is ProviderKind.OPENAI_COMPATIBLE
assert model == "text-embedding-3-small"
assert base_url == "https://api.openai.com/v1"
assert dimensions == 1536
def test_creating_a_profile_without_embedding_gets_the_builtin(
tmp_path: Path,
) -> None:
def test_creating_a_profile_without_embedding_gets_the_builtin(tmp_path: Path) -> None:
store = ModelProfileStore(tmp_path / "profiles.db")
secrets = InMemorySecretStore()
profile = store.create(
ModelProfileInput(
name="gen-only",
provider_kind=ProviderKind.OLLAMA,
model="qwen2.5",
),
secrets,
InMemorySecretStore(),
)
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
assert profile.embedding_model == BUILTIN_EMBEDDING_MODEL
assert profile.embedding_dimensions == BUILTIN_EMBEDDING_DIMENSIONS
def test_local_embedding_is_exempt_from_the_shared_origin_rule(
tmp_path: Path,
) -> None:
# Remote embedding must share the generation origin and key. Local
# embedding never leaves the machine, so the same rule would block the
# exact combination this feature exists for: remote generation + local
# embedding.
store = ModelProfileStore(tmp_path / "profiles.db")
secrets = InMemorySecretStore()
profile = store.create(
ModelProfileInput(
name="mixed",
provider_kind=ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
model="gpt-4o-mini",
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_model=BUILTIN_EMBEDDING_MODEL,
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
api_key="sk-test",
),
secrets,
)
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
def test_local_embedding_config_needs_no_embedding_key(tmp_path: Path) -> None:
config = build_sdk_config(_profile(), api_key="generation-key", data_dir=tmp_path)
def test_build_config_for_local_embedding_needs_no_embedding_key(
tmp_path: Path,
) -> None:
from tests.test_hy_memory_bridge import snapshot
profile = snapshot(
ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
has_api_key=True,
embedding_provider=ProviderKind.LOCAL_ONNX,
).model_copy(
update={
"embedding_model": BUILTIN_EMBEDDING_MODEL,
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
}
)
config = build_hy_memory_config(profile, api_key="sk-gen", data_dir=tmp_path)
assert config.vector_store.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
assert config.embedder.api_key is None
assert config.embedder.base_url is None
assert config.embedder.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
assert config.vector_store.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
@requires_assets
def test_local_hy_embed_service_returns_unit_512_vectors() -> None:
import asyncio
def test_public_local_embedding_service_covers_single_batch_and_metadata() -> None:
embedder = _Embedder()
service = LocalOnnxEmbeddingService(embedder) # type: ignore[arg-type]
service = LocalHyEmbedService(LocalEmbedder())
vector = asyncio.run(service.embed("本地嵌入不需要出网"))
assert len(vector) == 512
length = sum(value * value for value in vector) ** 0.5
assert length == pytest.approx(1.0, abs=1e-4)
single = asyncio.run(service.embed("单条"))
batch = asyncio.run(service.embed_batch(["第一条", "第二条"]))
vector, metadata = asyncio.run(service.embed_with_meta("带元数据"))
assert single == [0.0, 1.0]
assert batch == [[0.0, 1.0], [1.0, 1.0]]
assert vector == [0.0, 1.0]
assert metadata == {"provider": "local-onnx"}
assert service.get_embedding_dims() == BUILTIN_EMBEDDING_DIMENSIONS
@requires_assets
def test_create_client_swaps_in_the_local_embed_service(tmp_path: Path) -> None:
from tests.test_hy_memory_bridge import snapshot
profile = snapshot(
ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
has_api_key=True,
embedding_provider=ProviderKind.LOCAL_ONNX,
).model_copy(
update={
"embedding_model": BUILTIN_EMBEDDING_MODEL,
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
}
)
# HyMemoryClient opens local chroma/kuzu on a fresh data_dir and leaves
# the embedder alone until first call, so this does not reach the network.
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
try:
assert isinstance(client._embed_service, LocalHyEmbedService)
finally:
close = getattr(client, "close", None)
if callable(close):
close()
def test_local_embed_service_covers_every_method_hy_calls() -> None:
"""我们的替代品必须覆盖 HY 真会调的全部方法 —— 名单从 HY 源码里数,不写死。
上一版这个类只实现了 `embed`,注释还写着「HY only ever calls .embed on it」。
装着的这个 HY 版本里,写入流水线调的是 `embed_queued`(17 处)、`embed_batch`
(12 处)、`embed`(11 处)、`embed_with_meta`(1 处)。缺的那些不会报缺方法,
而是让 HY 退回它自己那个指向占位地址的服务,于是抽取以连接失败告终、
一条记忆都产不出来。
名单是数出来的而不是抄下来的:抄下来的名单会随 HY 升级一起烂掉,
而这条测试的全部意义就是替我们盯住那次升级。
"""
import re
from pathlib import Path as _Path
import hy_memory
root = _Path(hy_memory.__file__).parent
called: set[str] = set()
for path in root.rglob("*.py"):
if path.name == "embed_service.py":
continue
called.update(re.findall(r"embed_service\.([a-z_]+)", path.read_text(encoding="utf-8")))
assert called, "一个调用点都没数到,说明这条测试自己坏了"
missing = {name for name in called if not callable(getattr(LocalHyEmbedService, name, None))}
assert missing == set(), f"HY 会调但我们没实现:{sorted(missing)}"
@requires_assets
def test_the_pipelines_get_the_local_embed_service_not_only_the_client(
def test_acquire_client_injects_public_embedding_service(
tmp_path: Path,
monkeypatch,
) -> None:
"""真正干活的模块拿到的必须也是本地那个。
captured: dict[str, object] = {}
这是漏掉的那一步:client 在 __init__ 里把它自己那个 EmbedService 交给了
组件工厂(client.py:300),而所有流水线都从工厂取(registry.py:103/137/177)。
只替换 client 上的引用,工厂手里攥着的还是原来那个 —— 指向
http://127.0.0.1/local-onnx,连 TCP 都建不起来。
class Client:
def __init__(self, **kwargs: object) -> None:
captured.update(kwargs)
旧测试只断言 client 上那个引用,所以它一直是绿的,而抽取一直是零产出。
"""
from tests.test_hy_memory_bridge import snapshot
def close(self) -> None:
return None
profile = snapshot(
ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
has_api_key=True,
embedding_provider=ProviderKind.LOCAL_ONNX,
).model_copy(
update={
"embedding_model": BUILTIN_EMBEDDING_MODEL,
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
}
monkeypatch.setattr("one_memory.hy_sdk_runtime.HyMemoryClient", Client)
monkeypatch.setattr(
"one_memory.hy_sdk_runtime._acquire_runtime",
lambda _config, _key: object(),
)
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
try:
shared = client._registry._shared_embed_service
assert isinstance(shared, LocalHyEmbedService)
# 同一个对象,不是两份各自缓存的替代品:HY 的服务自带批处理队列,
# 两份会各攒一队,而队列的意义就是同一批一起走。
assert shared is client._embed_service
finally:
close = getattr(client, "close", None)
if callable(close):
close()
@requires_assets
def test_the_write_pipeline_itself_gets_the_local_embed_service(tmp_path: Path) -> None:
"""真正写记忆的那条流水线手里必须是本地那个。
这是前两条守卫都没守住的一层,而它才是决定成败的一层。实测:改完
client 与工厂两处引用之后,真实导入仍然 67 次 ConnectError、零记忆产出。
原因是 client.__init__ 在 client.py:332 就把 System2Writer 预热建好了 ——
那时替换还没发生,于是它抓走了 HY 原本那个服务并传给内层 writer,而
registry._system2_writer 一旦有值就永不重建。
调用链是实测出来的,不是读出来的:给真实 EmbedService 的 embed_queued 挂
栈回溯,它自己报出 client.async_add -> system2_writer.write ->
writer.write -> embed_queued。
"""
from tests.test_hy_memory_bridge import snapshot
profile = snapshot(
ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
has_api_key=True,
embedding_provider=ProviderKind.LOCAL_ONNX,
).model_copy(
update={
"embedding_model": BUILTIN_EMBEDDING_MODEL,
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
}
lease = acquire_client(
_profile(),
api_key="generation-key",
data_dir=tmp_path,
)
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
try:
writer = asyncio.run(client._registry.get_system2_writer())
assert isinstance(writer._embed_service, LocalHyEmbedService)
# 内层 writer 也要 —— 栈回溯显示 embed_queued 是它调的,不是 System2 调的。
inner = getattr(writer, "_writer", None) or getattr(writer, "writer", None)
assert inner is not None, "找不到内层 writer,这条测试自己坏了"
assert isinstance(inner._embed_service, LocalHyEmbedService)
assert isinstance(captured["embedding_service"], LocalOnnxEmbeddingService)
assert "runtime" in captured
finally:
close = getattr(client, "close", None)
if callable(close):
close()
@requires_assets
def test_named_read_pipeline_uses_local_embedding_after_cache_replacement(tmp_path: Path) -> None:
from tests.test_hy_memory_bridge import snapshot
profile = snapshot(
ProviderKind.OPENAI_COMPATIBLE,
base_url="https://api.example.com/v1",
has_api_key=True,
embedding_provider=ProviderKind.LOCAL_ONNX,
).model_copy(
update={
"embedding_model": BUILTIN_EMBEDDING_MODEL,
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
}
)
client = create_hy_memory_client(profile, api_key="test-generation-key", data_dir=tmp_path)
try:
reader = client._loop_thread.run(client._registry.get_reader(reader_name="legacy"))
assert isinstance(reader.embed_service, LocalHyEmbedService)
assert reader.embed_service is client._embed_service
assert client._registry._readers["legacy"] is reader
finally:
client.close()
lease.close()
@@ -91,9 +91,6 @@ ALLOWED = {
"Chroma's own chroma.sqlite3, read-only, checked before Chroma opens it",
opened_with='"?mode=ro"',
),
("hy_memory_bridge.py", "reattach_legacy_vector_collection"): Allowed(
"HY's vector store chroma.sqlite3 under its persist directory, renaming a collection"
),
("imports/sqlite_access.py", "open_sqlite_readonly"): Allowed(
"a host app's own SQLite (Cursor state.vscdb / store.db): one direct "
"read-only open when there is no WAL, and one read-only open of the "
@@ -1,738 +0,0 @@
"""R-27: a correction that only lands locally, while HY keeps the old text.
Deleting a memory cascades to HY (`api.py:1129` → `bridge.delete`). Correcting
one does not: `POST /api/v1/memories/{id}/correct` writes the governance
record and stops. The semantic half of recall comes from HY, so after a
correction HY still matches and ranks on the old wording while the governance
store returns the new — the user fixed it, saw it fixed, and search kept
using the sentence they replaced.
`client.update(memory_id, content)` re-embeds and upserts in place, keeping
the id (`hy_memory/client.py:2500-2561`), so the capability was already here
and simply never called.
"""
from __future__ import annotations
import asyncio
from fastapi import BackgroundTasks, Request
from httpx import ASGITransport, AsyncClient
from pydantic import SecretStr
from one_memory.api import create_app
from one_memory.config import Settings
from one_memory.hy_memory_bridge import HyMemoryBridgeError
from one_memory.imports.models import (
EvidenceInput,
EvidenceOrigin,
OriginMetadata,
WrittenBy,
)
from one_memory.model_profiles import (
ModelCapabilities,
ModelProfileInput,
ProviderKind,
)
from one_memory.recall import RecallRequest, RecallState
class FakeHy:
"""HY's side of the seam: an index keyed by memory id, plus a kill switch.
Holds content rather than just recording calls, so a test can ask the
question the user asks — does searching the new wording find it, and does
the old wording stop finding it.
"""
def __init__(self) -> None:
self.index: dict[str, str] = {}
self.available = True
#: HY holds no such memory, so no retry can ever land it.
self.rejects = False
self.updates: list[tuple[str, str]] = []
self.deletes: list[str] = []
self.adds: list[object] = []
async def update(self, memory_id: str, content: str) -> dict[str, object]:
if not self.available:
raise HyMemoryBridgeError("hy_memory_update_failed")
if self.rejects:
raise HyMemoryBridgeError("hy_memory_update_rejected")
self.updates.append((memory_id, content))
self.index[memory_id] = content
return {"success": True, "memory_id": memory_id}
async def delete(self, memory_id: str) -> dict[str, object]:
self.deletes.append(memory_id)
self.index.pop(memory_id, None)
return {"success": True}
async def ingest(self, request: object) -> object:
self.adds.append(request)
raise AssertionError("a correction must not re-add the memory")
async def search_ids(self, query: str, **_kwargs) -> tuple[str, ...]:
return tuple(memory_id for memory_id, content in self.index.items() if query in content)
async def close(self) -> None:
return None
def _settings(tmp_path) -> Settings:
return Settings(
data_dir=tmp_path / "data",
source_home=tmp_path,
api_token=SecretStr("local-token"),
)
def _endpoint(app, path: str):
"""The route's own function, so a test can hold the response and the work
scheduled after it apart — over HTTP the two are indistinguishable."""
for route in app.router.routes:
if getattr(route, "path", None) == path:
return route.endpoint
raise AssertionError(f"no route at {path}")
def _client(app) -> AsyncClient:
return AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
headers={
"Authorization": "Bearer local-token",
"Origin": "http://127.0.0.1:5173",
},
)
async def _confirmed_memory(app, client: AsyncClient, content: str, tag: str = "1") -> str:
# `tag` keeps the request ids apart. They are the mutation ledger's key, so
# a second memory created under the first one's id would be served the
# first one's result as a replay and never exist.
evidence = app.state.evidence_store.put(
EvidenceInput(
source_id="cursor",
conversation_id=f"conversation-{tag}",
message_id=f"message-{tag}",
correlation_id=f"message-{tag}",
role="user",
written_by=WrittenBy.PERSON,
verbatim=content,
origin_meta={EvidenceOrigin.FILE: OriginMetadata(raw_meta={})},
)
).record
created = await client.post(
"/api/v1/memories",
json={
"request_id": f"create-{tag}",
"draft": {
"user_id": "local",
"kind": "chat",
"content": content,
"scopes": ["all-apps"],
"evidence_ids": [evidence.evidence_id],
},
},
)
assert created.status_code == 200
memory_id = created.json()["memory_id"]
return memory_id
def _attach(app, hy: FakeHy) -> None:
"""Put the fake where the supervisor resolves its bridge.
The correction path has to go through the supervisor like deletion does,
because that is where the active profile and its secret are resolved —
which is also why the app needs an active profile before either cascade
can reach HY at all.
`supervisor._bridge` built `HyMemoryBridge` directly until R-27 and never
consulted `bridge_factory`, so neither cascade had a seam and the delete
one had no test. Setting the factory is now enough to stay off the
network, which is why the delete control below can exist at all.
"""
profiles = app.state.model_profile_store
secrets = app.state.secret_store
profile = profiles.create(
ModelProfileInput(
name="Local",
provider_kind=ProviderKind.OLLAMA,
model="hy-bot-phi4",
),
secrets,
)
profiles.record_capabilities(
profile.profile_id,
expected_revision=profile.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True,
model_available=True,
structured_output=True,
embedding_available=True,
),
secrets=secrets,
)
profiles.activate(profile.profile_id, secrets)
supervisor = app.state.memory_processing_supervisor
supervisor.bridge_factory = lambda _profile, _secret, _data_dir: hy
def test_remote_correction_updates_sdk_without_retired_content_consent(tmp_path):
async def scenario():
app = create_app(_settings(tmp_path))
async with app.router.lifespan_context(app), _client(app) as client:
hy = FakeHy()
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "original")
profiles, secrets = app.state.model_profile_store, app.state.secret_store
profile = profiles.create(
ModelProfileInput(
name="Remote embedding",
provider_kind=ProviderKind.OLLAMA,
model="fixture",
embedding_model="remote-embedding",
embedding_provider_kind=ProviderKind.OLLAMA,
embedding_base_url="http://127.0.0.1:1/v1",
),
secrets,
)
profiles.record_capabilities(
profile.profile_id,
expected_revision=profile.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True, model_available=True, embedding_available=True
),
secrets=secrets,
)
profiles.activate(profile.profile_id, secrets)
changed = await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "local-edit",
"expected_revision": 1,
"content": "updated",
"scopes": ["all-apps"],
},
)
assert changed.status_code == 200
assert changed.json()["content"] == "updated"
assert hy.updates == [(memory_id, "updated")]
asyncio.run(scenario())
def test_correcting_a_memory_updates_hy_in_place(tmp_path) -> None:
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
corrected = await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
assert corrected.status_code == 200
assert hy.updates == [(memory_id, "用户的花名是白泽")]
assert hy.index[memory_id] == "用户的花名是白泽"
# The boundary: never delete-and-re-add, which would change the id
# and break every reference already handed out.
assert hy.deletes == []
assert hy.adds == []
assert corrected.json()["memory_id"] == memory_id
asyncio.run(scenario())
def test_search_follows_the_correction(tmp_path) -> None:
"""The point of the sync: the new wording is findable, the old is not."""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
assert await hy.search_ids("白泽") == (memory_id,)
assert await hy.search_ids("麒麟") == ()
asyncio.run(scenario())
def test_a_correction_is_not_lost_when_hy_is_down(tmp_path) -> None:
"""The user's intent outranks the downstream.
The correction lands locally whatever HY does, and the fact that HY is
behind is recorded rather than swallowed.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
hy.available = False
corrected = await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
assert corrected.status_code == 200
assert corrected.json()["content"] == "用户的花名是白泽"
detail = await client.get(f"/api/v1/memories/{memory_id}")
sync = detail.json()["index_sync"]
assert sync["status"] == "pending"
assert sync["attempts"] == 1
assert sync["last_error_code"] == "hy_memory_update_failed"
asyncio.run(scenario())
def test_the_app_can_tell_the_user_the_index_is_current(tmp_path) -> None:
"""A synced correction must be distinguishable from a pending one.
Otherwise the interface has to choose between claiming success it cannot
see and worrying the user every time.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
detail = await client.get(f"/api/v1/memories/{memory_id}")
assert detail.json()["index_sync"]["status"] == "synced"
asyncio.run(scenario())
def test_a_retry_after_hy_returns_reaches_consistency(tmp_path) -> None:
""" "Eventually consistent" needs something that actually retries.
The retry sends the memory's current content rather than a copy taken at
correction time, so a memory corrected twice while HY was down converges
on the latest wording rather than replaying an intermediate one.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
hy.available = False
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-2",
"expected_revision": 2,
"content": "用户的花名是白泽,读作 bái zé",
"scopes": ["all-apps"],
},
)
hy.available = True
retried = await client.post(f"/api/v1/memories/{memory_id}/index-sync")
assert retried.status_code == 200
assert retried.json()["status"] == "synced"
assert hy.index[memory_id] == "用户的花名是白泽,读作 bái zé"
assert len(hy.updates) == 1
# Both failed corrections and the retry that worked: the count is
# how anyone tells a downstream that is slow from one that is
# broken, so a second correction must not restart it at zero.
assert retried.json()["attempts"] == 3
asyncio.run(scenario())
def test_a_pending_sync_outlives_the_process(tmp_path) -> None:
"""Restarting must not lose the knowledge that HY is stale.
A correction the user made and a downstream that never caught up is
exactly the state a crash would otherwise erase, leaving a memory that
searches wrong with nothing recording why.
"""
async def scenario() -> None:
hy = FakeHy()
settings = _settings(tmp_path)
app = create_app(settings)
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.available = False
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
# A second app over the same data_dir: new stores, nothing carried
# over in memory, so anything it still knows it read back from disk.
restarted = create_app(settings)
async with restarted.router.lifespan_context(restarted), _client(restarted) as client:
detail = await client.get(f"/api/v1/memories/{memory_id}")
sync = detail.json()["index_sync"]
assert sync["status"] == "pending"
assert sync["last_error_code"] == "hy_memory_update_failed"
asyncio.run(scenario())
def test_undoing_a_correction_syncs_hy_back_too(tmp_path) -> None:
"""Undo moves the text as much as correcting it did.
Taking a correction back and leaving HY matching on the withdrawn wording
is the same defect from the other direction, and the one the user reaches
for right after a correction they regret.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
undone = await client.post(
f"/api/v1/memories/{memory_id}/actions/undo",
json={"request_id": "undo-1", "expected_revision": 2},
)
assert undone.status_code == 200
assert undone.json()["content"] == "用户的花名是麒麟"
assert hy.index[memory_id] == "用户的花名是麒麟"
assert await hy.search_ids("白泽") == ()
detail = await client.get(f"/api/v1/memories/{memory_id}")
assert detail.json()["index_sync"]["status"] == "synced"
asyncio.run(scenario())
def test_an_untouched_memory_has_nothing_to_sync(tmp_path) -> None:
"""The control: a memory nobody corrected must not look out of date.
Green before and after. If every memory reported a sync state, the
interface would have to explain a status the user never caused — and
confirming this one moved its status without moving a word of its text,
which is the distinction the reindex has to be keyed on.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
detail = await client.get(f"/api/v1/memories/{memory_id}")
assert detail.json()["index_sync"] is None
assert hy.updates == []
asyncio.run(scenario())
async def _stale_memory(app, client: AsyncClient, hy: FakeHy) -> str:
"""A memory the user corrected while HY was unreachable.
The state this whole section is about: the app reads 白泽, HY still
matches on 麒麟, and nothing has caught up.
"""
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.index[memory_id] = "用户的花名是麒麟"
hy.available = False
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
hy.available = True
return memory_id
def test_a_recall_that_returns_a_stale_memory_catches_hy_up(tmp_path) -> None:
"""Where the retry belongs: the moment the memory is actually read.
Not a sweep at startup — the night the office lost its outbound network,
calling HY on launch would have turned one outage into an app that will
not open. A memory's index being behind can only be observed through
recall, so recall is where catching up is worth anything.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _stale_memory(app, client, hy)
recalled = await client.post(
"/api/v1/recall",
json={"query": "白泽", "scope": "all-apps"},
)
assert recalled.status_code == 200
assert recalled.json()["state"] == "grounded"
assert hy.index[memory_id] == "用户的花名是白泽"
detail = await client.get(f"/api/v1/memories/{memory_id}")
assert detail.json()["index_sync"]["status"] == "synced"
asyncio.run(scenario())
def test_the_recall_itself_does_not_wait_on_hy_being_back(tmp_path) -> None:
"""The recall answers from the governance store, which already holds the
corrected text. Letting a still-unreachable HY turn that into an error, or
into a wait, would make the fix worse than the defect."""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _stale_memory(app, client, hy)
hy.available = False
recalled = await client.post(
"/api/v1/recall",
json={"query": "白泽", "scope": "all-apps"},
)
assert recalled.status_code == 200
assert recalled.json()["state"] == "grounded"
assert recalled.json()["hits"][0]["content"] == "用户的花名是白泽"
detail = await client.get(f"/api/v1/memories/{memory_id}")
sync = detail.json()["index_sync"]
assert sync["status"] == "pending"
assert sync["attempts"] == 2
asyncio.run(scenario())
def test_a_memory_no_recall_returned_is_left_alone(tmp_path) -> None:
"""Scoping the work to what was read is the whole reason this is cheap.
A memory nobody recalls is a memory nobody reads, so its index being
behind is a difference no user can observe — and paying for it would mean
a call to HY per stale memory on every recall.
So the recall here does ground on something: a catch-up that skipped this
memory only because nothing at all was returned would prove nothing.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
stale_id = await _stale_memory(app, client, hy)
await _confirmed_memory(app, client, "用户在用 Cursor 写代码", tag="2")
recalled = await client.post(
"/api/v1/recall",
json={"query": "Cursor", "scope": "all-apps"},
)
assert recalled.status_code == 200
assert recalled.json()["state"] == "grounded"
assert stale_id not in {hit["memory_id"] for hit in recalled.json()["hits"]}
assert hy.updates == []
assert hy.index[stale_id] == "用户的花名是麒麟"
asyncio.run(scenario())
def test_the_recall_answers_before_the_catch_up_runs(tmp_path) -> None:
"""The ordering is the reason recall is a safe place to do this at all.
Called through the route so the two halves are separable: the receipt is
built without HY in the picture, and the catch-up is left as work for
afterwards. Done inline instead, every recall would wait on a network
call — the exact cost that ruled out doing this at startup.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _stale_memory(app, client, hy)
recall = _endpoint(app, "/api/v1/recall")
background = BackgroundTasks()
receipt = await recall(
RecallRequest(query="白泽", scope="all-apps"),
background,
Request({"type": "http", "headers": []}),
)
assert receipt.state is RecallState.GROUNDED
assert hy.updates == []
assert background.tasks
await background()
assert hy.index[memory_id] == "用户的花名是白泽"
asyncio.run(scenario())
def test_a_retry_hy_can_never_accept_is_not_made_on_every_recall(tmp_path) -> None:
"""`hy_memory_update_rejected` means HY has no such memory to re-embed.
Retrying it is a call that cannot succeed, and recall is the one path that
runs constantly — so this is the difference between a harmless catch-up
and a request to HY every time anyone recalls anything.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
hy.rejects = True
await client.post(
f"/api/v1/memories/{memory_id}/correct",
json={
"request_id": "correct-1",
"expected_revision": 1,
"content": "用户的花名是白泽",
"scopes": ["all-apps"],
},
)
hy.updates.clear()
await client.post(
"/api/v1/recall",
json={"query": "白泽", "scope": "all-apps"},
)
assert hy.updates == []
detail = await client.get(f"/api/v1/memories/{memory_id}")
sync = detail.json()["index_sync"]
assert sync["status"] == "pending"
assert sync["attempts"] == 1
asyncio.run(scenario())
def test_recall_touches_hy_for_nothing_when_no_memory_is_behind(tmp_path) -> None:
"""The control: recall is the hot path, and it stays quiet.
Green before and after. Every recall in the product runs through here, so
a catch-up that fired on memories nobody corrected would be a permanent
tax on the one call that has to stay fast.
"""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
await _confirmed_memory(app, client, "用户的花名是麒麟")
recalled = await client.post(
"/api/v1/recall",
json={"query": "麒麟", "scope": "all-apps"},
)
assert recalled.json()["state"] == "grounded"
assert hy.updates == []
asyncio.run(scenario())
def test_deletion_still_cascades_the_way_it_did(tmp_path) -> None:
"""The control for the path that already worked."""
async def scenario() -> None:
app = create_app(_settings(tmp_path))
hy = FakeHy()
async with app.router.lifespan_context(app), _client(app) as client:
_attach(app, hy)
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
deleted = await client.post(
f"/api/v1/memories/{memory_id}/actions/delete",
json={"request_id": "delete-1", "expected_revision": 1},
)
assert deleted.status_code == 200
assert hy.deletes == [memory_id]
asyncio.run(scenario())
File diff suppressed because it is too large. Load diff
@@ -1,256 +0,0 @@
"""R-03: memory language follows the message, so Kristin's gets written in English.
HY picks its extraction prompt from the content: `is_chinese(content)` in
`hy_memory/agent/extractor.py:868`, which is a CJK character ratio with a 0.3
threshold (`hy_memory/utils/lang_detect.py`). Kristin's everyday shape —
paste an English traceback, ask about it in Chinese — measures 0.09, so the
extractor is told to write in English and the memory comes out in English.
Read from HY's source rather than assumed, and it contradicts the plan of
record: `extract_system_prompt_zh` is only consulted *inside* the
`is_chinese(...)` true branch (`extractor.py:891-901`), so setting it changes
nothing for the case that is broken. The branch that actually runs reads
`extract_system_prompt_en` (`extractor.py:922-930`).
These tests drive HY's real `Extractor` with a stub LLM and look at the
system prompt it actually sends, so they pin HY's behaviour rather than a
copy of its logic.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass, field
from typing import Any
import pytest
from hy_memory.agent.extractor import Extractor
from hy_memory.utils.lang_detect import _chinese_char_ratio, is_chinese
from one_memory import hy_memory_bridge
from one_memory.hy_memory_bridge import build_hy_memory_config
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
#: A common shape: an English traceback pasted in, with
#: the question asked in Chinese. Measured CJK ratio 0.09.
CHINESE_QUESTION_WITH_ENGLISH_ERROR = """这个报错是什么意思,我该怎么修?
Traceback (most recent call last):
File "/Users/example/one-memory/apps/core/src/one_memory/recall.py", line 177, in _recall
for memory in self.governance.list(user_id=user_id, status=MemoryStatus.CONFIRMED, limit=500):
File "/Users/example/one-memory/apps/core/src/one_memory/memory_governance.py", line 388
raise ValueError("invalid pagination")
ValueError: invalid pagination
"""
@dataclass
class StubResponse:
content: str = '{"memories": [], "intentions": [], "basic_info": {}}'
tokens_used: int = 0
prompt_tokens: int = 0
completion_tokens: int = 0
tool_calls: Any = None
@dataclass
class RecordingLLM:
"""Stands in for the model, and keeps what it was told to do."""
messages: list[list[dict[str, Any]]] = field(default_factory=list)
async def complete_messages(self, *, messages, **_kwargs) -> StubResponse:
self.messages.append(messages)
return StubResponse()
async def complete(self, **_kwargs) -> StubResponse:
return StubResponse()
def system_prompt(self) -> str:
return next(item["content"] for item in self.messages[0] if item["role"] == "system")
def user_prompt(self) -> str:
return next(item["content"] for item in self.messages[0] if item["role"] == "user")
def _profile() -> ModelProfileSnapshot:
return ModelProfileSnapshot(
profile_id="profile",
revision=1,
provider_kind=ProviderKind.OLLAMA,
base_url="http://127.0.0.1:11434",
model="memory-model",
embedding_model="embedding-model",
embedding_provider_kind=None,
embedding_base_url=None,
embedding_dimensions=768,
timeout_seconds=30,
max_concurrency=1,
context_window=8192,
has_api_key=False,
)
def _extract(content: str, tmp_path) -> RecordingLLM:
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
llm = RecordingLLM()
extractor = Extractor(llm_provider=llm, llm_config=config.llm)
asyncio.run(extractor.extract(content=content))
return llm
def _reads_as_chinese(prompt: str) -> bool:
return _chinese_char_ratio(prompt) > 0.2
def _follows_the_input_language(prompt: str) -> bool:
return (
"输出语言必须与输入语言一致" in prompt
or "Output language must match input language" in prompt
)
def test_hy_keeps_a_chinese_question_with_an_english_traceback_in_chinese(
tmp_path,
) -> None:
"""The premise, pinned to HY's own detector rather than described.
Green before the fix and after: it is the control. R-03 must not be
"made to pass" by pushing the detector over its threshold — the input is
genuinely 9% CJK, and it is the memory language that has to stop
depending on that.
"""
assert _chinese_char_ratio(CHINESE_QUESTION_WITH_ENGLISH_ERROR) < 0.3
assert is_chinese(CHINESE_QUESTION_WITH_ENGLISH_ERROR) is True
def test_the_extractor_is_told_to_write_chinese_even_when_the_input_reads_as_english(
tmp_path,
) -> None:
"""The memory language is a property of the user, not of one message.
Before the fix this sample takes HY's English branch and gets the English
system prompt, so the memory is written in English — for a user whose
every other memory is Chinese, and who never asked for either.
"""
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
assert _reads_as_chinese(llm.system_prompt()), (
f"extractor was given an English system prompt: {llm.system_prompt()[:120]!r}"
)
assert not _follows_the_input_language(llm.system_prompt()), (
"the prompt still hands the language decision back to the input"
)
assert "始终" in llm.system_prompt() and "中文" in llm.system_prompt()
@pytest.mark.parametrize("ratio", [0.05, 0.3, 0.7])
def test_the_memory_language_does_not_depend_on_the_cjk_ratio(ratio: float, tmp_path) -> None:
"""Three samples either side of the 0.3 threshold, all Chinese memories.
0.3 is the extraction threshold and 0.7 is System 2's, so a sample at
each says whether the outcome still moves with the ratio.
"""
chinese_length = max(1, round(40 * ratio))
content = "我" * chinese_length + "x" * (40 - chinese_length)
llm = _extract(content, tmp_path)
assert _reads_as_chinese(llm.system_prompt()), (
f"CJK ratio {ratio} produced an English system prompt"
)
assert not _follows_the_input_language(llm.system_prompt()), (
f"CJK ratio {ratio} left the language up to the input"
)
def test_the_input_reaches_hy_untouched(tmp_path) -> None:
"""No Chinese prefix glued onto the content to fool the detector.
That shortcut would work and is forbidden: the content HY extracts from
is the person's own words, and padding it pollutes the verbatim the whole
evidence chain rests on. Green before and after — a guard, not a repair.
"""
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
assert CHINESE_QUESTION_WITH_ENGLISH_ERROR in llm.user_prompt()
assert "ValueError: invalid pagination" in llm.user_prompt()
def test_hys_own_chinese_prompt_hands_the_language_back_to_the_input(tmp_path) -> None:
"""Why shipping HY's Chinese prompt as-is is not the fix.
Both of HY's extraction system prompts carry one rule — "输出语言必须与输入
语言一致" / "Output language must match input language" — so a Chinese
prompt over an English-reading input still asks for English. Serving HY's
text verbatim would have made every assertion about the *prompt* pass
while the memory came out in English anyway. Green before and after: it
pins the SDK text this fix has to work around, and goes red the day HY
drops that rule, which is the day the fix can shrink.
"""
prompts_zh = pytest.importorskip("hy_memory.agent.prompts_zh")
extractor_module = pytest.importorskip("hy_memory.agent.extractor")
assert _follows_the_input_language(prompts_zh.EXTRACT_SYSTEM_PROMPT_ZH)
assert _follows_the_input_language(extractor_module.EXTRACT_SYSTEM_PROMPT)
def test_the_english_traceback_is_still_not_to_be_translated(tmp_path) -> None:
"""Fixing the language must not start rewriting what was pasted in.
HY's other two language rules keep terms, brands, formulas and code
exactly as written. Pinning the memory language to Chinese must leave
those standing, or a traceback becomes a translated traceback and stops
matching anything.
"""
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
assert "保留原始术语、品牌名、公式、代码" in llm.system_prompt()
assert "不要转写或翻译专有名词" in llm.system_prompt()
def test_a_reworded_sdk_prompt_still_gets_a_chinese_instruction(
tmp_path, monkeypatch, caplog
) -> None:
"""The fix edits one line of HY's prompt, so it has to survive a reword.
If a future SDK changes that line's wording the replacement silently
stops matching, and silently going back to "follow the input" is the
failure mode worth spending a branch on: the override is appended
instead, and it says so in the log.
"""
from hy_memory.agent.prompts_zh import EXTRACT_SYSTEM_PROMPT_ZH as native_prompt
class RewordedPrompts:
EXTRACT_SYSTEM_PROMPT_ZH = native_prompt.replace(
"* 输出语言必须与输入语言一致", "* HY 换了语言规则的措辞"
)
real_import = hy_memory_bridge.import_module
monkeypatch.setattr(
hy_memory_bridge,
"import_module",
lambda name: RewordedPrompts if name == "hy_memory.agent.prompts_zh" else real_import(name),
)
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
assert "始终是简体中文" in config.llm.extract_system_prompt_en
assert "hy_language_rule_not_found" in caplog.text
def test_setting_the_chinese_override_controls_the_mixed_traceback_prompt(tmp_path) -> None:
"""Pins why R-03 is not the one-line config change it was planned as.
`extract_system_prompt_zh` sits inside the `is_chinese(...)` true branch,
so on the failing input it is never read. If a later HY version moves the
override outside the branch this test goes red, which is the moment to
simplify the fix rather than keep carrying it.
"""
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
llm = RecordingLLM()
config.llm.extract_system_prompt_en = None
config.llm.extract_system_prompt_zh = "只用中文写记忆。"
extractor = Extractor(llm_provider=llm, llm_config=config.llm)
asyncio.run(extractor.extract(content=CHINESE_QUESTION_WITH_ENGLISH_ERROR))
assert llm.system_prompt() == "只用中文写记忆。"
+6 -2
View File
@@ -240,7 +240,7 @@ def test_sdk_memory_write_accepts_openai_messages_without_evidence_tables(tmp_pa
@pytest.mark.fresh_sdk
def test_sdk_user_profile_http_returns_native_assembled_profile(tmp_path):
class Bridge:
async def get_user_profile(self, *, user_id: str):
def get_user_profile(self, *, user_id: str):
assert user_id == "local"
return {
"request_id": "profile-request",
@@ -253,13 +253,17 @@ def test_sdk_user_profile_http_returns_native_assembled_profile(tmp_path):
"elapsed_ms": 1.25,
}
class Handle:
async def client(self):
return Bridge()
async def close(self):
return None
async def scenario():
app = application(tmp_path)
async with app.router.lifespan_context(app), client_for(app) as client:
app.state.sdk_library_bridge = Bridge()
app.state.sdk_client_handle = Handle()
response = await client.get("/api/v2/user-profile")
assert response.status_code == 200, response.text
assert response.json() == {
File diff suppressed because it is too large. Load diff
@@ -1,37 +0,0 @@
from __future__ import annotations
from one_memory import memory_processing
def _failure() -> RuntimeError:
try:
memory_processing.memory_task_timeout_seconds("database is locked") # type: ignore[arg-type]
except ValueError as error:
return RuntimeError("database is locked").with_traceback(error.__traceback__)
raise AssertionError("expected invalid timeout")
def test_conversation_prepare_failure_includes_safe_message_and_stack(monkeypatch) -> None:
events: list[dict[str, object]] = []
monkeypatch.setattr(memory_processing, "diagnostic_event", lambda **event: events.append(event))
memory_processing._report_conversation_prepare_failure("workbuddy", _failure())
assert events[0]["error_type"] == "RuntimeError"
assert events[0]["details"] == {
"source_id": "workbuddy",
"reason": "database is locked",
}
assert events[0]["safe_stack"]
assert "/Users/" not in str(events[0]["safe_stack"])
def test_deletion_sync_failure_includes_safe_message_and_stack(monkeypatch) -> None:
events: list[dict[str, object]] = []
monkeypatch.setattr(memory_processing, "diagnostic_event", lambda **event: events.append(event))
memory_processing._report_deletion_sync_failure(_failure())
assert events[0]["error_type"] == "RuntimeError"
assert events[0]["details"] == {"reason": "database is locked"}
assert events[0]["safe_stack"]
@@ -16,7 +16,6 @@ from hashlib import sha256
import pytest
from one_memory.database_access import main_database
from one_memory.hy_memory_bridge import HyMemoryBridgeError
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
from one_memory.memory_server.durable_queue import (
CANCELLED,
@@ -821,20 +820,20 @@ def test_same_turn_replayed_by_live_sync_upgrades_priority_without_conflict(tmp_
class _Recorder:
"""Stands in for `MemoryProcessingSupervisor.process_records`."""
"""Stands in for the SDK-authority processor."""
def __init__(self, error: Exception | None = None, *, invoke_hy: bool = False) -> None:
self.seen: list[tuple[str, ...]] = []
self.error = error
self.invoke_hy = invoke_hy
async def process_records(self, records, **_kwargs):
async def process_sdk_job(self, records, **_kwargs):
self.seen.append(tuple(record.evidence_id for record in records))
if self.invoke_hy:
await _kwargs["before_hy_invocation"]()
if self.error is not None:
raise self.error
return ()
return type("Receipt", (), {"raw_ids": (), "memory_ids": ()})()
class _TwoInvocationRecorder:
@@ -842,24 +841,28 @@ class _TwoInvocationRecorder:
self.pause_after_first = pause_after_first
self.invocations = 0
async def process_records(self, _records, **kwargs):
async def process_sdk_job(self, _records, **kwargs):
await kwargs["before_hy_invocation"]()
self.invocations += 1
self.pause_after_first()
await kwargs["before_hy_invocation"]()
self.invocations += 1
return ()
return type("Receipt", (), {"raw_ids": (), "memory_ids": ()})()
class _EvidenceStore:
def __init__(self, known: set[str]) -> None:
self.known = known
self.released: list[tuple[str, ...]] = []
def get(self, evidence_id: str):
if evidence_id not in self.known:
return None
return type("Record", (), {"evidence_id": evidence_id})()
def release_sdk_staging(self, evidence_ids: tuple[str, ...]) -> None:
self.released.append(evidence_ids)
def _state(path, job_id: str) -> str:
rows = _rows(path)
@@ -879,7 +882,7 @@ def _claim_through_drain(queue: DurableWriteQueue, job_id: str) -> None:
def test_a_durable_job_is_processed_through_the_authorized_path(tmp_path) -> None:
"""The executor must not bypass `process_records` (permission + runner)."""
"""The executor must use the SDK-authority processor."""
path = tmp_path / "one-memory.sqlite3"
queue = DurableWriteQueue(path)
record = queue.submit(_request(evidence=("e1", "e2")))
@@ -906,7 +909,6 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
_claim_through_drain(queue, record.job_id)
class Supervisor:
sdk_authority = staticmethod(lambda: True)
calls = 0
@classmethod
@@ -918,21 +920,11 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
(),
{
"raw_ids": ("raw-1",),
"memories": (
type("Memory", (), {"memory_id": "memory-1"})(),
type("Memory", (), {"memory_id": "memory-2"})(),
),
"memory_ids": ("memory-1", "memory-2"),
},
)()
class Evidence(_EvidenceStore):
released: tuple[str, ...] = ()
def release_sdk_staging(self, evidence_ids):
self.released = evidence_ids
return len(evidence_ids)
evidence = Evidence({"e1"})
evidence = _EvidenceStore({"e1"})
asyncio.run(
execute_job(
record.job_id,
@@ -947,7 +939,210 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
assert Supervisor.calls == 1
assert receipt.raw_ids == ("raw-1",)
assert receipt.memory_ids == ("memory-1", "memory-2")
assert evidence.released == ("e1",)
assert evidence.released == [("e1",)]
class _SdkPublicClient:
def __init__(self) -> None:
self.calls: list[tuple[str, object, dict[str, object]]] = []
self.events: list[str] = []
self.outcome: object = None
def add(self, data, **kwargs):
self.events.append("add")
self.calls.append(("add", data, kwargs))
return {
"success": True,
"raw_ids": ["raw-1"],
"memory_id": "memory-1",
"experience_ids": ["experience-1"],
"fact_ids": ["fact-1"],
"variant": "coding",
}
def add_extracted(self, contents, **kwargs):
self.events.append("add_extracted")
self.calls.append(("add_extracted", contents, kwargs))
return {"success": True, "memory_ids": ["memory-note"]}
def get_write_outcome(self, request_id):
self.calls.append(("get_write_outcome", request_id, {}))
return self.outcome
class _SdkPublicHandle:
def __init__(self, client: _SdkPublicClient) -> None:
self.value = client
async def client(self):
return self.value
async def close(self):
pass
def _sdk_evidence(
evidence_id: str,
*,
role: str = "user",
text: str = "hello",
memory_content: bool = False,
source: str = "cursor",
manual_note: bool = False,
created_at: datetime | None = None,
):
from one_memory.imports.models import (
EvidenceContentKind,
EvidenceInput,
EvidenceOrigin,
OriginMetadata,
WrittenBy,
materialize_write_input,
)
origin = (
OriginMetadata(raw_meta={"input_mode": "direct_memory_note"})
if manual_note
else OriginMetadata()
)
return materialize_write_input(
EvidenceInput(
source_id=source,
workspace="repo-a",
conversation_id="session-1",
message_id=evidence_id,
content_kind=(
EvidenceContentKind.MEMORY_CONTENT
if memory_content
else EvidenceContentKind.RAW_CONVERSATION
),
role=role,
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
created_at=created_at,
verbatim=text,
origin_meta={EvidenceOrigin.MANUAL_IMPORT: origin},
)
)
def test_sdk_write_processor_calls_add_once_with_public_scope_and_receipt() -> None:
from one_memory.sdk_write_processor import PRODUCT_AGENT_ID, SdkWriteProcessor, SdkWriteReceipt
client = _SdkPublicClient()
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
memory_at = datetime(2026, 10, 22, 8, tzinfo=UTC)
records = (
_sdk_evidence("one", text="question", created_at=memory_at),
_sdk_evidence(
"two",
role="assistant",
text="answer",
created_at=memory_at + timedelta(seconds=1),
),
_sdk_evidence("three", role="tool", text="result", created_at=memory_at),
)
async def run() -> SdkWriteReceipt:
async def before() -> None:
client.events.append("before")
return await processor.process_sdk_job(
records,
request_id="request-1",
before_hy_invocation=before,
)
receipt = asyncio.run(run())
assert client.events == ["before", "add"]
assert len(client.calls) == 1
method, messages, kwargs = client.calls[0]
assert method == "add"
assert messages == [
{"role": "user", "content": "question"},
{"role": "assistant", "content": "answer"},
{"role": "tool", "content": "result"},
]
assert kwargs["user_id"] == "local"
assert kwargs["agent_id"] == PRODUCT_AGENT_ID
assert kwargs["session_id"] == "session-1"
assert kwargs["source"] == "cursor"
assert kwargs["workspace_id"] == "repo-a"
assert kwargs["memory_at"] == memory_at + timedelta(seconds=1)
assert kwargs["request_id"] == "request-1"
assert kwargs["metadata"] == {
"source_id": "cursor",
"session_id": "session-1",
"workspace_ids": ["repo-a"],
"memory_at": (memory_at + timedelta(seconds=1)).isoformat(),
"evidence_ids": [record.evidence_id for record in records],
"input_kind": "raw_conversation",
}
assert receipt == SdkWriteReceipt(
raw_ids=("raw-1",),
memory_ids=("memory-1", "experience-1", "fact-1"),
)
@pytest.mark.parametrize(
"record",
(
_sdk_evidence("memory", text="remember this", memory_content=True),
_sdk_evidence(
"note",
text="save this note",
source="one-memory-manual",
manual_note=True,
),
),
)
def test_sdk_write_processor_uses_add_extracted_for_memory_content_and_notes(record) -> None:
from one_memory.sdk_write_processor import SdkWriteProcessor, SdkWriteReceipt
client = _SdkPublicClient()
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
async def run() -> SdkWriteReceipt:
async def before() -> None:
client.events.append("before")
return await processor.process_sdk_job(
(record,),
request_id=record.evidence_id,
before_hy_invocation=before,
)
assert asyncio.run(run()) == SdkWriteReceipt(memory_ids=("memory-note",))
assert client.events == ["before", "add_extracted"]
assert client.calls[0][1] == [record.verbatim]
assert len(client.calls) == 1
def test_sdk_unknown_outcome_inquiry_uses_only_public_receipt() -> None:
from one_memory.sdk_write_processor import SdkWriteProcessor, SdkWriteReceipt
client = _SdkPublicClient()
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
client.outcome = {"status": "unknown", "result": {"success": False}}
assert asyncio.run(processor.inquire_write_outcome("request-1")) is None
client.outcome = {
"status": "succeeded",
"result": {
"success": True,
"raw_ids": ["raw-1"],
"fact_ids": ["fact-1"],
"variant": "working",
},
}
assert asyncio.run(processor.inquire_write_outcome("request-1")) == SdkWriteReceipt(
raw_ids=("raw-1",),
memory_ids=("fact-1",),
)
assert [call[0] for call in client.calls] == [
"get_write_outcome",
"get_write_outcome",
]
def test_missing_evidence_is_refused_without_calling_the_model(tmp_path) -> None:
@@ -1978,7 +2173,7 @@ def test_execute_job_crash_after_marking_hy_invoked_becomes_sticky_unknown(tmp_p
drain.claim(limit=1)
class _CancelledBeforeReturn:
async def process_records(self, _records, **_kwargs):
async def process_sdk_job(self, _records, **_kwargs):
raise asyncio.CancelledError
with pytest.raises(asyncio.CancelledError):
@@ -4978,7 +5173,7 @@ def test_a_refused_source_context_is_nonblocking_unknown(
record.job_id,
queue=queue,
supervisor=_Recorder(
error=HyMemoryBridgeError("hy_memory_write_verification_context_scope"),
error=RuntimeError("hy_memory_write_verification_context_scope"),
invoke_hy=True,
),
evidence=_EvidenceStore({"e1"}),
@@ -5006,7 +5201,7 @@ def test_a_failure_after_the_call_still_parks_as_unknown(tmp_path) -> None:
claimer = DurableWriteDrain(queue, execute=_never_execute, owner="after-call-owner")
assert claimer.claim(limit=1) == (record.job_id,)
recorder = _Recorder(
error=HyMemoryBridgeError("hy_memory_write_verification_authority_invalid"),
error=RuntimeError("hy_memory_write_verification_authority_invalid"),
invoke_hy=True,
)
@@ -509,10 +509,10 @@ def test_the_api_shows_the_chain_refuses_edits_and_still_deletes_through_hy(tmp_
cascaded: list[str] = []
async def delete_hy_memory(memory_id: str) -> None:
async def finish_deletion(memory_id: str) -> None:
cascaded.append(memory_id)
app.state.memory_processing_supervisor.delete_hy_memory = delete_hy_memory
app.state.sdk_memory_runtime.finish_deletion = finish_deletion
deleted = await client.post(
"/api/v1/memories/old/actions/delete",
json={"request_id": "delete-x", "expected_revision": replaced.revision},
+1 -1
View File
@@ -861,7 +861,7 @@ def test_clearing_drops_payloads_of_a_failed_task_at_the_retry_ceiling(tmp_path)
def test_clearing_never_touches_completed_tasks(tmp_path) -> None:
"""Completed inputs are evidence of what HY was actually given.
`memory_processing` compares them against the records sent to HY to prove
SDK write processing compares them against the records sent to HY to prove
coverage, and `matters/discovery` reads them to report how many records HY
saw. Clearing them would make a failed coverage check look like a pass.
"""
+28 -310
View File
@@ -1,263 +1,28 @@
"""R-09:首次安装引导把画像录下来,不问用户一句话。
引导导完来源就结束了,从不归纳一次画像。`digest()` 全仓唯一的调用方是
记忆空间里那个按钮(`MemorySpace.tsx:265`),也就是说首次安装走完,
System 2 归纳出的画像一条都不在治理层里,新装的库对「我是谁」一无所知,
而证据其实已经导进来了。
归纳本身还漏了一半:`_record_schemas` 只挑 `L6_SCHEMA` 节点,
`L0_BASIC_INFO` 直接跳过。`_read_graph_snapshot` 的查询不带 layer 过滤,
L0 节点本来就在快照里,取回来了又丢掉。HY 自己把这两层并称画像
(`_PROFILE_LAYERS = {"l0_basic_info", "l6_schema"}`),
basic info 那半边留在 Kuzu 里,本地看不见。
"""
"""Onboarding endpoints manage UI state without touching SDK memory."""
import asyncio
import sqlite3
from pathlib import Path
from httpx import ASGITransport, AsyncClient
from pydantic import SecretStr
from one_memory.api import create_app
from one_memory.config import Settings
from one_memory.hy_memory_bridge import HyGraphNode, HyGraphSnapshot, HyMemoryBridgeError
from one_memory.imports.models import (
EvidenceContentKind,
EvidenceInput,
EvidenceOrigin,
OriginMetadata,
WrittenBy,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import (
MemoryDraft,
MemoryGovernanceStore,
MemoryKind,
MemoryStatus,
)
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_tasks import MemoryTaskStore
from one_memory.model_profiles import (
ModelCapabilities,
ModelProfileInput,
ModelProfileStore,
ProviderKind,
)
from one_memory.secrets import InMemorySecretStore
from tests.person_door import as_person
class FakeGraph:
"""归纳只用得上四个方法,其余不实现,少一个假实现少一处失真。"""
def __init__(self, graph_nodes: tuple[HyGraphNode, ...]) -> None:
self.graph_nodes = graph_nodes
self.graph_reads: list[tuple[str, str]] = []
self.digest_calls: list[tuple[str, str]] = []
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
self.digest_calls.append((user_id, agent_id))
return {"success": True, "tasks_processed": 1}
async def graph_snapshot(
self,
*,
user_id: str,
agent_id: str,
limit: int = 300,
) -> HyGraphSnapshot:
self.graph_reads.append((user_id, agent_id))
return HyGraphSnapshot(nodes=self.graph_nodes, edges=())
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
return ()
async def close(self) -> None:
return None
def _setup(tmp_path):
database = tmp_path / "one-memory.sqlite3"
secrets = InMemorySecretStore()
evidence = EvidenceStore(database)
profiles = ModelProfileStore(database)
tasks = MemoryTaskStore(database)
governance = MemoryGovernanceStore(database)
profile = profiles.create(
ModelProfileInput(
name="Local",
provider_kind=ProviderKind.OLLAMA,
model="hy-bot-phi4",
embedding_model="nomic-embed-text",
def _app(tmp_path: Path):
return create_app(
Settings(
_env_file=None,
data_dir=tmp_path / "data",
source_home=tmp_path / "home",
api_token=SecretStr("test-token"),
imports_enabled=False,
automatic_memory_processing=False,
live_sync_enabled=False,
),
secrets,
source_catalog={},
)
profiles.record_capabilities(
profile.profile_id,
expected_revision=profile.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True,
model_available=True,
structured_output=True,
embedding_available=True,
),
secrets=secrets,
)
profile = profiles.activate(profile.profile_id, secrets)
return evidence, profiles, tasks, governance, secrets, profile
def _imported(evidence: EvidenceStore, *, workspace: str | None = "workspace:v1:project"):
return evidence.put(
EvidenceInput(
source_id="cursor",
workspace=workspace,
conversation_id="conversation",
message_id="m1",
correlation_id="m1",
content_kind=EvidenceContentKind.RAW_CONVERSATION,
role="user",
written_by=WrittenBy.PERSON,
verbatim="导入进来的一段对话。",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
).record
def _confirmed_from(governance: MemoryGovernanceStore, evidence_id: str) -> None:
"""归纳只对「还有本地证据的 scope」动手,所以得先有一条确认记忆。"""
as_person(governance).create(
MemoryDraft(
user_id="local",
kind=MemoryKind.CHAT,
content="项目里的一条既有记忆。",
scopes=("workspace:v1:project",),
evidence_ids=(evidence_id,),
),
request_id="candidate-1",
)
def _supervisor(tmp_path, stores, bridge) -> MemoryProcessingSupervisor:
evidence, profiles, tasks, governance, secrets, profile = stores
supervisor = MemoryProcessingSupervisor(
tasks=tasks,
profiles=profiles,
secrets=secrets,
evidence=evidence,
governance=governance,
data_dir=tmp_path,
)
supervisor._bridges[(profile.profile_id, profile.revision, None)] = bridge
return supervisor
def test_the_basic_info_half_of_the_profile_is_written_down_too(tmp_path) -> None:
"""L0_BASIC_INFO 取回来了又被丢掉,画像只进了一半。
HY 自己把 L0_basic_info 与 L6_schema 并称画像,快照里两层都在,
而 `_record_schemas` 只留 L6。零提问录画像要的正是 L0 这半边:
「用户是一名产品经理」这种事实,只会出现在 basic info 层。
"""
stores = _setup(tmp_path)
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
record = _imported(evidence)
_confirmed_from(governance, record.evidence_id)
bridge = FakeGraph(
graph_nodes=(
HyGraphNode(
node_id="basic-1",
content="用户是一名非技术背景的产品经理。",
layer="l0_basic_info",
),
HyGraphNode(
node_id="schema-1",
content="用户偏好可验证的结论。",
layer="l6_schema",
),
),
)
supervisor = _supervisor(tmp_path, stores, bridge)
result = asyncio.run(supervisor.digest())
profile_contents = {
memory.content: memory
for memory in governance.list(user_id="local")
if memory.scopes == ("all-apps",)
}
assert "用户是一名非技术背景的产品经理。" in profile_contents
assert "用户偏好可验证的结论。" in profile_contents
assert result["schemas_recorded"] == 2
basic = profile_contents["用户是一名非技术背景的产品经理。"]
assert basic.status is MemoryStatus.CONFIRMED
# 说得出为什么相信它,和 L6 那半边同一个标准。
assert basic.evidence_ids == (record.evidence_id,)
# kind 用仓库既有的 layer→kind 映射,不为画像新造一个种类
# (种类是 API 契约的一部分,新增要人放行)。
assert basic.kind is MemoryKind.CHAT
def test_a_layer_hy_does_not_call_profile_still_never_reaches_governance(tmp_path) -> None:
"""绿对照,也是这一项的防假修复守卫。
把「只挑 L6」改成「快照里的都带回来」能让上面那条变绿,代价是
整段原始对话(L1_RAW)被当成画像广播给每个 app。这条盯着它。
"""
stores = _setup(tmp_path)
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
record = _imported(evidence)
_confirmed_from(governance, record.evidence_id)
bridge = FakeGraph(
graph_nodes=tuple(
HyGraphNode(node_id=f"node-{layer}", content=f"{layer} 的内容。", layer=layer)
for layer in ("l1_raw", "l2_fact", "l3_summary", "l4_identity", "l7_intention")
),
)
supervisor = _supervisor(tmp_path, stores, bridge)
result = asyncio.run(supervisor.digest())
assert result["schemas_recorded"] == 0
assert [memory.content for memory in governance.list(user_id="local")] == [
"项目里的一条既有记忆。"
]
def test_the_same_basic_info_is_not_written_twice(tmp_path) -> None:
"""绿对照:归纳会被反复触发(引导一次、按钮随时),不能越攒越多。"""
stores = _setup(tmp_path)
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
record = _imported(evidence)
_confirmed_from(governance, record.evidence_id)
nodes = (
HyGraphNode(
node_id="basic-1",
content="用户是一名非技术背景的产品经理。",
layer="l0_basic_info",
),
)
supervisor = _supervisor(tmp_path, stores, FakeGraph(graph_nodes=nodes))
first = asyncio.run(supervisor.digest())
second = asyncio.run(supervisor.digest())
assert first["schemas_recorded"] == 1
assert second["schemas_recorded"] == 0
assert [
memory.content
for memory in governance.list(user_id="local")
if memory.scopes == ("all-apps",)
] == ["用户是一名非技术背景的产品经理。"]
def _app(tmp_path):
settings = Settings(
data_dir=tmp_path / "data",
source_home=tmp_path / "home",
api_token=SecretStr("test-token"),
)
return create_app(settings, source_catalog={})
def _client(app) -> AsyncClient:
@@ -271,83 +36,47 @@ def _client(app) -> AsyncClient:
)
def test_finishing_onboarding_is_not_model_content_consent(tmp_path) -> None:
"""S1.3: completion and skip cannot authorize profile induction."""
def test_finishing_onboarding_is_not_model_content_consent(tmp_path: Path) -> None:
async def scenario() -> None:
app = _app(tmp_path)
async with app.router.lifespan_context(app), _client(app) as client:
calls: list[str] = []
async def spy() -> dict[str, object]:
async def digest() -> dict[str, object]:
calls.append("digest")
return {"success": True, "schemas_recorded": 0}
app.state.memory_processing_supervisor.digest = spy # type: ignore[method-assign]
# 只是看一眼引导状态,不该触发归纳。
assert (await client.get("/api/v1/onboarding")).status_code == 200
assert calls == []
return {"success": True}
app.state.sdk_memory_runtime.digest = digest
finished = await client.post("/api/v1/onboarding/complete")
assert finished.status_code == 200
assert finished.json()["status"] == "completed"
assert calls == []
asyncio.run(scenario())
def test_the_way_back_to_the_first_run_deletes_nothing(tmp_path) -> None:
"""重置只动完成标记,记忆一条不少。
完成态和记忆放在同一个库里是有意的:重装一遍 App 不该把一个已经有记忆的人
当成新用户。代价是想再看一次冷启动,重装不管用。这条盯着那个出口:
状态回到首次运行,别的都不动。
"""
def test_reset_returns_to_first_run_without_touching_sdk_data(tmp_path: Path) -> None:
async def scenario() -> None:
app = _app(tmp_path)
async with app.router.lifespan_context(app), _client(app) as client:
database = tmp_path / "data" / "one-memory.sqlite3"
with sqlite3.connect(database) as connection:
connection.execute(
"INSERT INTO governed_memories "
"(memory_id,user_id,kind,status,content,scopes_json,"
"revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?)",
(
"keep-me",
"local",
"chat",
"confirmed",
"记忆必须保留",
'["all-apps"]',
1,
"2026-10-17T00:00:00+00:00",
"2026-10-17T00:00:00+00:00",
),
)
finished = await client.post("/api/v1/onboarding/complete")
assert finished.json()["status"] == "completed"
sdk_marker = tmp_path / "data" / "hy-memory" / "governed-v2" / "memory.marker"
sdk_marker.parent.mkdir(parents=True, exist_ok=True)
sdk_marker.write_text("keep-sdk-memory", encoding="utf-8")
assert (await client.post("/api/v1/onboarding/complete")).status_code == 200
reset = await client.post("/api/v1/onboarding/reset")
assert reset.status_code == 200
assert reset.json()["status"] == "not_started"
assert reset.json()["completed_at"] is None
# 落盘了,不是只在响应里改一下:下一次启动读到的也是首次运行。
assert (await client.get("/api/v1/onboarding")).json()["status"] == "not_started"
with sqlite3.connect(database) as connection:
assert connection.execute(
"SELECT content,revision FROM governed_memories WHERE memory_id='keep-me'"
).fetchone() == ("记忆必须保留", 1)
assert sdk_marker.read_text(encoding="utf-8") == "keep-sdk-memory"
asyncio.run(scenario())
def test_resetting_before_anything_was_completed_is_not_an_error(tmp_path) -> None:
"""绿对照:新库里还没有那一行,重置不能炸。"""
def test_resetting_before_completion_is_not_an_error(tmp_path: Path) -> None:
async def scenario() -> None:
app = _app(tmp_path)
async with app.router.lifespan_context(app), _client(app) as client:
@@ -359,28 +88,17 @@ def test_resetting_before_anything_was_completed_is_not_an_error(tmp_path) -> No
asyncio.run(scenario())
def test_hy_being_down_does_not_leave_the_first_install_unfinished(tmp_path) -> None:
"""守卫:这一步可跳过,归纳失败不能把引导卡住。
首次安装时模型档案可能还没配好,通用出站网络也可能不可用。
归纳失败只该少一条画像,不该让人装不完。
"""
def test_sdk_unavailability_cannot_block_onboarding_completion(tmp_path: Path) -> None:
async def scenario() -> None:
app = _app(tmp_path)
async with app.router.lifespan_context(app), _client(app) as client:
async def unavailable() -> dict[str, object]:
raise HyMemoryBridgeError("hy_memory_unavailable")
app.state.memory_processing_supervisor.digest = unavailable # type: ignore[method-assign]
raise RuntimeError("hy_sdk_unavailable")
app.state.sdk_memory_runtime.digest = unavailable
finished = await client.post("/api/v1/onboarding/complete")
assert finished.status_code == 200
assert finished.json()["status"] == "completed"
assert finished.json()["completed_at"] is not None
# 状态是真的落了盘,不是只在这次响应里。
assert (await client.get("/api/v1/onboarding")).json()["status"] == "completed"
asyncio.run(scenario())
+1 -1
View File
@@ -3,7 +3,7 @@ import hashlib
import pytest
from one_memory.hy_memory_bridge import HySdkMemoryRecord
from one_memory.hy_sdk_types import HySdkMemoryRecord
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import MemoryDraft, MemoryGovernanceStore, MemoryKind
@@ -127,9 +127,17 @@ def test_both_lanes_answer_through_one_real_hy_instance(tmp_path, monkeypatch):
linked.raise_for_status()
job = await app.state.import_supervisor.wait(linked.json()["jobs"][0]["job_id"])
assert job.stored == 1 and job.errors == 0, job
processor = app.state.memory_processing_supervisor
await asyncio.gather(*tuple(processor._debounces.values()), return_exceptions=True)
await asyncio.wait_for(asyncio.gather(*tuple(processor._workers.values())), timeout=300)
deadline = asyncio.get_running_loop().time() + 300
while True:
counts = app.state.sdk_memory_runtime.status_counts()
if not any(
counts.get(state, 0)
for state in ("pending", "running", "waiting_for_model")
):
break
if asyncio.get_running_loop().time() >= deadline:
raise AssertionError(f"SDK write queue did not drain: {counts}")
await asyncio.sleep(0.05)
# Imported content is not shared with every app by default. Sharing
# it here is what the memory lane is later asked to find.
@@ -216,13 +216,13 @@ def test_retrieval_stop_failure_still_drains_other_owned_services(tmp_path, monk
)
with pytest.raises(failure):
async with app.router.lifespan_context(app):
original = app.state.memory_processing_supervisor.shutdown
original = app.state.sdk_memory_runtime.shutdown
async def shutdown():
await original()
events.append("processing_drained")
monkeypatch.setattr(app.state.memory_processing_supervisor, "shutdown", shutdown)
monkeypatch.setattr(app.state.sdk_memory_runtime, "shutdown", shutdown)
assert events == ["start", "retrieval_drained", "processing_drained"]
assert not (tmp_path / "runtime.json").exists()
@@ -4,9 +4,16 @@ import asyncio
import json
import pytest
from hy_memory.models.memory import MemoryLayer, MemoryStatus
from one_memory.database_access import close_main_databases, main_database
from one_memory.hy_memory_bridge import HySdkMemoryPage, HySdkMemoryRecord, HySdkRawRecord
from one_memory.hy_sdk_migration import HySdkMigration
from one_memory.hy_sdk_types import (
HySdkMemoryPage,
HySdkMemoryRecord,
HySdkRawPage,
HySdkRawRecord,
)
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
@@ -35,10 +42,15 @@ from one_memory.sdk_migration_inventory import (
record_raw_migration_plan,
)
from one_memory.sdk_migration_runner import (
activate_sdk_migration_staging,
commit_reconciled_cutover,
prepare_legacy_write_queue_for_cutover,
prepare_sdk_migration_staging,
recover_sdk_migration_swap,
resume_legacy_cleanup,
run_sdk_data_plane_backfill,
sdk_cutover_blockers,
sdk_migration_staging,
)
from one_memory.target_schema import legacy_tables_present
@@ -210,6 +222,8 @@ def test_startup_cutover_freezes_pending_job_input_and_preserves_the_queue(store
sdk_version="2.0.5.dev3",
)
assert marker.migration_id == "m1"
assert legacy_tables_present(store.database_path)
resume_legacy_cleanup(data_dir=data_dir, database_path=store.database_path)
assert legacy_tables_present(store.database_path) == ()
assert store.authority() is MemoryAuthority.SDK
preserved = DurableWriteQueue(store.database_path, sdk_authority=True)
@@ -410,7 +424,7 @@ def test_inventory_classification_covers_every_legacy_and_sdk_state():
assert len(decisions) == 8
class _Bridge:
class _MigrationClient:
def __init__(self, memories=()):
self.memories = tuple(memories)
self.imported = []
@@ -418,7 +432,7 @@ class _Bridge:
self.deleted = []
self.raw = ()
async def list_sdk_memories(self, *, limit, offset, **_kwargs):
async def list_memories(self, *, limit, offset, **_kwargs):
page = self.memories[offset : offset + limit]
return HySdkMemoryPage(
memories=page,
@@ -427,7 +441,7 @@ class _Bridge:
offset=offset,
)
async def import_sdk_memory_exact(self, **kwargs):
async def write_legacy_memory(self, **kwargs):
self.imported.append(kwargs)
return {"success": True, "memory_id": kwargs["memory_id"]}
@@ -435,11 +449,16 @@ class _Bridge:
self.deleted.append(memory_id)
return {"success": True, "deleted_count": 1}
async def list_sdk_raw(self, *, limit, offset, **_kwargs):
async def read_raw_inventory(self, *, limit, offset, **_kwargs):
rows = self.raw[offset : offset + limit]
return rows, len(self.raw)
return HySdkRawPage(
raw=rows,
total=len(self.raw),
limit=limit,
offset=offset,
)
async def import_sdk_raw_exact(self, **kwargs):
async def write_legacy_raw(self, **kwargs):
self.imported_raw.append(kwargs)
return {"success": True, "raw_id": kwargs["raw_id"]}
@@ -466,14 +485,14 @@ def _record(memory_id, *, content="body", status="active"):
def test_sdk_inventory_is_paged_without_dropping_lifecycle_records():
bridge = _Bridge(
client = _MigrationClient(
(
_record("active"),
_record("old", status="superseded"),
_record("shadow", status="shadow"),
)
)
inventory = asyncio.run(read_sdk_memory_inventory(bridge, page_size=2))
inventory = asyncio.run(read_sdk_memory_inventory(client, page_size=2))
assert [(item.memory_id, item.status) for item in inventory] == [
("active", "active"),
("old", "superseded"),
@@ -502,10 +521,10 @@ def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
legacy=legacy,
sdk=sdk,
)
bridge = _Bridge()
client = _MigrationClient()
asyncio.run(
execute_memory_migration_plan(
bridge,
client,
store,
migration_id="m1",
legacy=legacy,
@@ -513,8 +532,8 @@ def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
)
)
assert [item["memory_id"] for item in bridge.imported] == ["missing"]
assert bridge.deleted == ["delete-existing"]
assert [item["memory_id"] for item in client.imported] == ["missing"]
assert client.deleted == ["delete-existing"]
progress = store.progress("m1")
assert progress.counts == {"adopted": 1, "deleted": 2, "verified": 3}
@@ -590,23 +609,23 @@ def test_raw_inventory_matches_exact_scope_imports_missing_and_adopts_sdk_only(s
legacy=legacy,
sdk=sdk,
)
bridge = _Bridge()
client = _MigrationClient()
asyncio.run(
execute_raw_migration_plan(
bridge,
client,
store,
migration_id="m1",
legacy=legacy,
decisions=recorded,
)
)
assert [item["raw_id"] for item in bridge.imported_raw] == ["evidence:e2"]
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e2"]
assert store.progress("m1").counts == {"adopted": 1, "verified": 2}
def test_sdk_raw_inventory_pages_all_records():
bridge = _Bridge()
bridge.raw = (
client = _MigrationClient()
client.raw = (
HySdkRawRecord(
raw_id="r1",
content="one",
@@ -624,7 +643,7 @@ def test_sdk_raw_inventory_pages_all_records():
source="claude",
),
)
inventory = asyncio.run(read_sdk_raw_inventory(bridge, page_size=1))
inventory = asyncio.run(read_sdk_raw_inventory(client, page_size=1))
assert [item.raw_id for item in inventory] == ["r1", "r2"]
@@ -656,11 +675,11 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
""",
(content_hash("verbatim"),),
)
bridge = _Bridge()
client = _MigrationClient()
result = asyncio.run(
run_sdk_data_plane_backfill(
database_path=database,
bridge=bridge,
client=client,
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
@@ -674,15 +693,15 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
assert result.state == "ready"
assert result.phase == "reconciled"
assert result.counts == {"verified": 2}
assert [item["memory_id"] for item in bridge.imported] == ["memory-1"]
assert [item["raw_id"] for item in bridge.imported_raw] == ["evidence:e1"]
assert [item["memory_id"] for item in client.imported] == ["memory-1"]
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e1"]
assert store.authority() is MemoryAuthority.LEGACY
# Simulate a process dying after both SDK side effects but before startup
# could finish. The next inventory observes them as matched rather than
# import_exact; that reclassification must resume the settled ledger
# instead of conflicting with its original operation key.
bridge.memories = (_record("memory-1", content="preference"),)
bridge.raw = (
client.memories = (_record("memory-1", content="preference"),)
client.raw = (
HySdkRawRecord(
raw_id="evidence:e1",
content="verbatim",
@@ -696,7 +715,7 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
resumed = asyncio.run(
run_sdk_data_plane_backfill(
database_path=database,
bridge=bridge,
client=client,
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
@@ -708,8 +727,8 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
)
assert resumed.state == "ready"
assert resumed.counts == {"verified": 2}
assert len(bridge.imported) == 1
assert len(bridge.imported_raw) == 1
assert len(client.imported) == 1
assert len(client.imported_raw) == 1
def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
@@ -729,7 +748,7 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
result = asyncio.run(
run_sdk_data_plane_backfill(
database_path=database,
bridge=_Bridge(),
client=_MigrationClient(),
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
@@ -744,3 +763,217 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
assert result.phase == "review_required"
assert result.counts == {"quarantine": 1}
assert store.authority() is MemoryAuthority.LEGACY
class _MigrationLoop:
def run(self, coroutine):
return asyncio.run(coroutine)
class _MigrationVectorStore:
def __init__(self):
self.nodes = {}
async def get_by_id(self, memory_id):
return self.nodes.get(memory_id)
async def upsert(self, node):
self.nodes[node.node_id] = node
return node.node_id
async def list_by_user(self, *, user_id, layers, **_kwargs):
return [
node
for node in self.nodes.values()
if node.user_id == user_id and node.layer in layers
]
class _MigrationEmbedder:
async def embed(self, _content):
return [0.1, 0.2]
class _MigrationGraphStore:
def __init__(self):
self.nodes = {}
async def upsert_memory_node(self, node):
self.nodes[node.node_id] = node
return node.node_id
class _PrivilegedMigrationClient:
def __init__(self):
self._loop_thread = _MigrationLoop()
self._vector_store = _MigrationVectorStore()
self._embed_service = _MigrationEmbedder()
self._graph_store = _MigrationGraphStore()
def list_memories(self, **_kwargs):
return {"vdb": {"memories": [], "total": 0}, "request_id": "list"}
def delete(self, _memory_id):
return {"success": True, "deleted_count": 1}
def get_write_outcome(self, _request_id):
return None
def test_offline_migration_writes_sdk_stores_without_public_client_import_methods():
client = _PrivilegedMigrationClient()
migration = HySdkMigration(client)
raw_result = asyncio.run(
migration.write_legacy_raw(
content="verbatim",
raw_id="raw-1",
user_id="local",
agent_id="one-memory",
session_id="session",
source="cursor",
memory_at=None,
metadata={"role": "user"},
request_id="migrate-raw-1",
)
)
memory_result = asyncio.run(
migration.write_legacy_memory(
content="prefers tea",
memory_id="memory-1",
user_id="local",
agent_id="one-memory",
session_id="session",
source="legacy-governance",
layer="l2_fact",
status="active",
owner="user",
tags=("preference",),
supersedes=(),
superseded_by=(),
is_latest=True,
memory_at=None,
metadata={},
request_id="migrate-memory-1",
)
)
raw_page = asyncio.run(migration.read_raw_inventory(user_id="local"))
assert raw_result["raw_id"] == "raw-1"
assert memory_result["memory_id"] == "memory-1"
assert [item.raw_id for item in raw_page.raw] == ["raw-1"]
assert client._vector_store.nodes["raw-1"].layer is MemoryLayer.L1_RAW
assert client._vector_store.nodes["memory-1"].status is MemoryStatus.ACTIVE
assert client._graph_store.nodes["memory-1"].content == "prefers tea"
assert not hasattr(client, "import_raw")
assert not hasattr(client, "import_memory_exact")
def test_raw_http_dto_uses_the_unified_memory_list_shape():
page = HySdkRawPage.from_memory_list(
{
"vdb": {
"memories": [
{
"memory_id": "raw-1",
"content": "verbatim",
"layer": "l1_raw",
"status": "active",
"user_id": "local",
"agent_id": "one-memory",
"session_id": "session",
"source": "cursor",
"custom": {"role": "user"},
}
],
"total": 1,
"limit": 100,
"offset": 0,
}
},
limit=100,
offset=0,
)
assert page.raw[0].raw_id == "raw-1"
assert page.raw[0].custom == {"role": "user"}
def _staging_fixture(tmp_path, store):
data_dir = tmp_path / "home"
live = data_dir / "hy-memory" / "governed-v2"
live.mkdir(parents=True)
(live / "sdk-only.txt").write_text("old", encoding="utf-8")
paths = sdk_migration_staging(
data_dir=data_dir,
migration_id="migration-1",
namespace="governed-v2",
)
prepare_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
)
return paths
def test_staging_preserves_sdk_only_data_and_swaps_after_commit(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
staged = paths.staging_root / "sdk-only.txt"
assert staged.read_text(encoding="utf-8") == "old"
staged.write_text("migrated", encoding="utf-8")
marker = object()
result = activate_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
commit=lambda: marker,
)
assert result is marker
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
assert not paths.protective_root.exists()
assert not paths.staging_home.exists()
def test_incomplete_staging_retry_restarts_from_unchanged_live_root(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("partial", encoding="utf-8")
prepare_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
)
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
def test_failed_cutover_restores_live_store_and_keeps_staging(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
def fail():
raise DataPlaneMigrationError("cutover_failed")
with pytest.raises(DataPlaneMigrationError, match="cutover_failed"):
activate_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
commit=fail,
)
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
def test_restart_recovers_swap_when_authority_was_not_committed(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
paths.live_root.replace(paths.protective_root)
paths.staging_root.replace(paths.live_root)
recover_sdk_migration_swap(
paths=paths,
database_path=store.database_path,
)
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
+25 -15
View File
@@ -1,11 +1,10 @@
from __future__ import annotations
import asyncio
from types import SimpleNamespace
import pytest
from one_memory.hy_memory_bridge import HySdkMemoryRecord
from one_memory.hy_sdk_types import HySdkMemoryRecord
from one_memory.memory_qa import MemoryAnswerError
from one_memory.sdk_memory_qa import SdkMemoryAnswerer
@@ -24,18 +23,29 @@ def record(content: str = "用户偏好本地优先。") -> HySdkMemoryRecord:
)
class Bridge:
class Client:
def __init__(self) -> None:
self.current = record()
self.searches = 0
async def search_memories(self, *_args, **_kwargs):
def search(self, *_args, **_kwargs):
self.searches += 1
return (SimpleNamespace(memory_id="memory-1"),)
return {
"scene": "normal",
"memories": [{"memory_id": "memory-1", "score": 0.9}],
}
async def get_sdk_memory(self, memory_id):
def get(self, memory_id):
assert memory_id == "memory-1"
return self.current
return self.current.model_dump()
class Handle:
def __init__(self, client: Client) -> None:
self.value = client
async def client(self):
return self.value
class Profiles:
@@ -45,10 +55,10 @@ class Profiles:
def test_sdk_ask_uses_native_sdk_memory_without_governance_fields() -> None:
bridge = Bridge()
client = Client()
answer = asyncio.run(
SdkMemoryAnswerer(
bridge=bridge, # type: ignore[arg-type]
client_handle=Handle(client),
profiles=Profiles(), # type: ignore[arg-type]
secrets=object(), # type: ignore[arg-type]
).answer("偏好是什么?")
@@ -65,18 +75,18 @@ def test_sdk_ask_uses_native_sdk_memory_without_governance_fields() -> None:
def test_sdk_ask_revalidates_memory_before_returning_extract() -> None:
bridge = Bridge()
client = Client()
class ChangingProfiles:
@staticmethod
def active(_secrets):
bridge.current = record("内容已变化")
client.current = record("内容已变化")
return None
with pytest.raises(MemoryAnswerError, match="memory_answer_context_changed"):
asyncio.run(
SdkMemoryAnswerer(
bridge=bridge, # type: ignore[arg-type]
client_handle=Handle(client),
profiles=ChangingProfiles(), # type: ignore[arg-type]
secrets=object(), # type: ignore[arg-type]
).answer("偏好是什么?")
@@ -84,14 +94,14 @@ def test_sdk_ask_revalidates_memory_before_returning_extract() -> None:
def test_sdk_ask_propagates_sdk_search_failure_without_legacy_fallback() -> None:
class Failed(Bridge):
async def search_memories(self, *_args, **_kwargs):
class Failed(Client):
def search(self, *_args, **_kwargs):
raise RuntimeError("sdk unavailable")
with pytest.raises(RuntimeError, match="sdk unavailable"):
asyncio.run(
SdkMemoryAnswerer(
bridge=Failed(), # type: ignore[arg-type]
client_handle=Handle(Failed()),
profiles=Profiles(), # type: ignore[arg-type]
secrets=object(), # type: ignore[arg-type]
).answer("偏好是什么?")
@@ -1,190 +1,97 @@
"""What `semantic_search` should do once the profile that filled the store is gone.
Deactivating or deleting a model profile does not delete the memories that
profile produced, but `semantic_search` gave up as soon as it could not find
an active one. The retrieval it abandoned needs no generation model at all:
the built-in ONNX encoder is what wrote those vectors in the first place.
"""
"""Model-free SDK search remains available without an active write profile."""
from pathlib import Path
from types import SimpleNamespace
import pytest
from one_memory.hy_memory_bridge import (
HY_MEMORY_NAMESPACE,
GraphWidthMismatch,
GraphWidthMismatchError,
from one_memory.hy_sdk_runtime import (
SdkClientHandle,
builtin_retrieval_snapshot,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_governance import MemoryGovernanceStore
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_tasks import MemoryTaskStore
from one_memory.model_profiles import (
ModelCapabilities,
ModelProfileInput,
ModelProfileStore,
ProviderKind,
)
from one_memory.secrets import InMemorySecretStore
from one_memory.model_profiles import ProviderKind
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
class RecordingBridge:
def __init__(self, snapshot, hits: tuple[str, ...] = ()) -> None:
self.snapshot = snapshot
self.hits = hits
class _SearchClient:
def __init__(self, reply: dict[str, object]) -> None:
self.reply = reply
self.calls: list[tuple[str, dict[str, object]]] = []
async def search_scores(self, query: str, **_: object) -> tuple[object, ...]:
del query
return tuple((memory_id, 0.9) for memory_id in self.hits)
def search(self, query: str, **options: object) -> dict[str, object]:
self.calls.append((query, options))
return self.reply
def _supervisor(tmp_path, *, hits: tuple[str, ...] = (), built: list | None = None):
database = tmp_path / "one-memory.sqlite3"
secrets = InMemorySecretStore()
profiles = ModelProfileStore(database)
class _Handle:
def __init__(self, client: _SearchClient) -> None:
self._client = client
def factory(snapshot, _secret, data_dir):
# Building the real client is what creates the store on disk, so the
# stand-in has to create it too: without that, "no store was built"
# is a sentence no assertion could ever falsify.
(data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / "graph").mkdir(
parents=True,
exist_ok=True,
)
bridge = RecordingBridge(snapshot, hits)
if built is not None:
built.append(bridge)
return bridge
async def client(self) -> _SearchClient:
return self._client
supervisor = MemoryProcessingSupervisor(
tasks=MemoryTaskStore(database),
profiles=profiles,
secrets=secrets,
evidence=EvidenceStore(database),
governance=MemoryGovernanceStore(database),
data_dir=tmp_path,
bridge_factory=factory,
def _runtime(client: _SearchClient) -> SdkMemoryRuntime:
return SdkMemoryRuntime(
coordinator=SimpleNamespace(),
read_handle=_Handle(client), # type: ignore[arg-type]
write_queue=SimpleNamespace(), # type: ignore[arg-type]
extraction_settings=SimpleNamespace(), # type: ignore[arg-type]
data_dir=Path("/unused"),
)
return supervisor, profiles, secrets
def _plant_store(tmp_path: Path) -> None:
"""Give the data directory the graph an earlier extraction would have left."""
graph = tmp_path / "hy-memory" / HY_MEMORY_NAMESPACE / "graph"
graph.mkdir(parents=True)
(graph / "catalog.kz").write_bytes(b"")
def test_builtin_retrieval_handle_needs_no_profile_or_key() -> None:
profile = builtin_retrieval_snapshot()
def _activate(profiles, secrets):
created = profiles.create(
ModelProfileInput(
name="Local",
provider_kind=ProviderKind.OLLAMA,
model="hy-bot-phi4",
embedding_model="nomic-embed-text",
),
secrets,
)
profiles.record_capabilities(
created.profile_id,
expected_revision=created.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True,
model_available=True,
structured_output=True,
embedding_available=True,
),
secrets=secrets,
)
return profiles.activate(created.profile_id, secrets)
assert profile.profile_id == "builtin-retrieval"
assert profile.provider_kind is ProviderKind.LOCAL_ONNX
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
assert profile.has_api_key is False
@pytest.mark.anyio
async def test_a_library_left_by_a_retired_profile_is_still_searchable(tmp_path) -> None:
supervisor, _profiles, _secrets = _supervisor(tmp_path, hits=("memory-1",))
_plant_store(tmp_path)
found = await supervisor.semantic_search("抽取")
assert found == (("memory-1", 0.9),)
@pytest.mark.anyio
async def test_no_store_on_disk_is_not_a_reason_to_build_one(tmp_path) -> None:
"""Searching must not leave a 512-wide graph behind for an undecided user.
Building the client is what creates the store, and the width it creates is
the built-in encoder's. A user who has not chosen a model yet would then
meet a width mismatch on their first 768-wide profile: an accident this
search invented, in the same shape as the one that retired a live graph.
"""
supervisor, _profiles, _secrets = _supervisor(tmp_path, hits=("memory-1",))
found = await supervisor.semantic_search("抽取")
assert found == ()
assert not (tmp_path / "hy-memory").exists()
@pytest.mark.anyio
async def test_a_store_written_wider_is_left_to_the_width_prompt(tmp_path) -> None:
supervisor, _profiles, _secrets = _supervisor(tmp_path)
_plant_store(tmp_path)
class WiderStore(RecordingBridge):
async def search_scores(self, query: str, **_: object) -> tuple[object, ...]:
# Where the real one raises: the client is built lazily, on the
# first search, and it is building it that reads the stored width.
del query
raise GraphWidthMismatchError(
GraphWidthMismatch(
stored_width=768,
expected_width=512,
graph_path=str(tmp_path / "graph"),
)
)
supervisor.bridge_factory = lambda snapshot, _secret, _data_dir: WiderStore(snapshot)
assert await supervisor.semantic_search("抽取") == ()
@pytest.mark.anyio
async def test_retrieval_reaches_hy_as_a_local_encoder_needing_no_key(tmp_path) -> None:
built: list[RecordingBridge] = []
supervisor, _profiles, _secrets = _supervisor(tmp_path, built=built)
_plant_store(tmp_path)
await supervisor.semantic_search("抽取")
snapshot = built[0].snapshot
assert snapshot.embedding_provider_kind is ProviderKind.LOCAL_ONNX
assert snapshot.has_api_key is False
assert snapshot == builtin_retrieval_snapshot()
@pytest.mark.anyio
async def test_the_retrieval_bridge_does_not_occupy_a_real_profiles_cache_slot(
tmp_path,
async def test_sdk_client_handle_lazily_reuses_the_model_free_client(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Otherwise pasting a key would change nothing the user can see.
client = _SearchClient({"scene": "normal", "memories": []})
lease = SimpleNamespace(client=client, close=lambda: None)
acquired: list[object] = []
The cache is keyed by profile id and revision. Were the retrieval snapshot
to collide with a real profile's key, the bridge built with no generation
model would be handed back after the user configured one, and extraction
would go on doing nothing with no error anywhere to explain it.
"""
built: list[RecordingBridge] = []
supervisor, profiles, secrets = _supervisor(tmp_path, built=built)
_plant_store(tmp_path)
def acquire(*args: object, **kwargs: object) -> object:
acquired.append((args, kwargs))
return lease
await supervisor.semantic_search("抽取")
active = _activate(profiles, secrets)
await supervisor._prewarm_write_bridges(("cursor",)) # noqa: SLF001
await supervisor.semantic_search("抽取")
monkeypatch.setattr("one_memory.hy_sdk_runtime.acquire_client", acquire)
handle = SdkClientHandle(
builtin_retrieval_snapshot(),
api_key=None,
data_dir=tmp_path,
)
assert len(built) == 3
assert len({id(bridge) for bridge in built}) == 3
assert built[0].snapshot == builtin_retrieval_snapshot()
assert {bridge.snapshot.profile_id for bridge in built[1:]} == {active.profile_id}
assert await handle.client() is client
assert await handle.client() is client
assert len(acquired) == 1
await handle.close()
@pytest.mark.anyio
async def test_search_without_profile_uses_public_sdk_search() -> None:
client = _SearchClient(
{
"scene": "normal",
"memories": [
{"memory_id": "memory-1", "content": "偏好简短回答", "score": 0.91},
],
}
)
found = await _runtime(client).semantic_search("回答偏好")
assert found == (("memory-1", 0.91),)
query, options = client.calls[0]
assert query == "回答偏好"
assert options["user_ids"] == ["local"]
assert options["agent_ids"] == []
assert options["scene"] == "normal"
@@ -35,6 +35,7 @@ from one_memory.imports.sources.deepseek_harness import (
from one_memory.imports.sources.hermes import (
HERMES_DESCRIPTOR,
create_hermes_adapter,
parse_hermes_rollout_record,
)
from one_memory.imports.sources.openclaw import (
OPENCLAW_DESCRIPTOR,
@@ -233,6 +234,55 @@ def test_qwen_code_reads_only_the_transcript_beside_its_bookkeeping(
assert SECRET not in repr(events)
@pytest.mark.parametrize("role", ["user", "assistant"])
def test_hermes_strips_native_provider_recall_context(role: str) -> None:
context = SourceRecordContext(
file_key="hermes",
relative_path="sessions/session.jsonl",
ordinal="1",
)
result = parse_hermes_rollout_record(
{
"type": "message",
"role": role,
"content": (
"before\n<memhub_memory_context>\n"
"Untrusted memory reference:\n"
'[{"memory_id":"m1","content":"RECALLED_SECRET"}]\n'
"</memhub_memory_context>\nafter"
),
"sessionId": "session-A",
},
context,
)
assert isinstance(result, ParsedMessage)
assert result.verbatim == "before\n\nafter"
assert "RECALLED_SECRET" not in result.verbatim
def test_hermes_skips_a_persisted_pure_recall_block() -> None:
context = SourceRecordContext(
file_key="hermes",
relative_path="sessions/session.jsonl",
ordinal="1",
)
result = parse_hermes_rollout_record(
{
"type": "message",
"role": "user",
"content": (
"<memhub_memory_context>\n"
'[{"memory_id":"m1","content":"RECALLED_SECRET"}]\n'
"</memhub_memory_context>"
),
"sessionId": "session-A",
},
context,
)
assert isinstance(result, ParsedSkip)
assert result.reason_code == ScanIssueCode.NO_TEXT
@pytest.mark.parametrize(
("role", "record_type", "provenance", "written_by"),
[
@@ -1,494 +0,0 @@
"""Sedimentation runs several harnesses at once, but one harness at a time.
Extraction was serial: `max_concurrent_tasks` defaulted to 1 and nothing ever
passed anything else, so the queue on the user's machine stood at 1,440 with
one task in flight and the scheduler logging `queue_saturated`. Meanwhile the
model profile has carried a `max_concurrency` of 4 all along, copied into
every task's snapshot and never once consulted by the scheduler.
Turning the number up on its own would be wrong. HY decides whether a new
memory supersedes an older one by what it has already been told, so two
extractions from the same harness running at once can each conclude without
knowing about the other. Harnesses run side by side up to the profile's limit.
"""
import asyncio
import sqlite3
import threading
import time
import pytest
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_tasks import MemoryTaskStatus
from one_memory.model_profiles import ModelCapabilities, ModelProfilePatch
from tests.test_memory_processing import FakeBridge, _put, _runner, _setup
#: Long enough that an overlap is unmistakable, short enough to stay a unit test.
MODEL_SECONDS = 0.25
class WatchedBridge(FakeBridge):
"""A model that takes a moment, and says who was inside it when.
Overlap is recorded rather than inferred from timings: every call notes
which conversations were already in the model when it arrived.
"""
def __init__(self, seconds: float = MODEL_SECONDS) -> None:
super().__init__(outputs=())
self.seconds = seconds
self.inside: set[str] = set()
self.peak = 0
self.overlaps: list[tuple[str, ...]] = []
self.same_conversation_overlaps: list[str] = []
self._guard = threading.Lock()
async def _spend_a_moment(self, request) -> None:
conversation = getattr(request, "session_id", None) or "?"
with self._guard:
if self.inside:
self.overlaps.append(tuple(sorted(self.inside)))
if conversation in self.inside:
self.same_conversation_overlaps.append(conversation)
self.inside.add(conversation)
self.peak = max(self.peak, len(self.inside))
try:
await asyncio.sleep(self.seconds)
finally:
with self._guard:
self.inside.discard(conversation)
async def ingest(self, request):
await self._spend_a_moment(request)
return await super().ingest(request)
async def ingest_extracted(self, request, contents):
await self._spend_a_moment(request)
return await super().ingest_extracted(request, contents)
def _conversations(
evidence,
*,
count: int,
turns: int = 1,
sources: tuple[str, ...] = ("cursor",),
) -> None:
for conversation in range(count):
source = sources[conversation % len(sources)]
for turn in range(turns):
evidence.put(
EvidenceInput(
source_id=source,
message_id=f"c{conversation}-t{turn}",
role="user",
written_by=WrittenBy.PERSON,
verbatim=f"第 {conversation} 段对话的第 {turn} 句。",
conversation_id=f"conversation-{conversation}",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
)
evidence.put(
EvidenceInput(
source_id=source,
message_id=f"c{conversation}-t{turn}-answer",
role="assistant",
written_by=WrittenBy.AGENT,
verbatim=f"第 {conversation} 段对话第 {turn} 句的回复。",
conversation_id=f"conversation-{conversation}",
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
)
)
async def _drain(supervisor, tasks, *, expected: int, timeout: float = 60.0) -> float:
started = time.monotonic()
supervisor.backfill()
deadline = started + timeout
while time.monotonic() < deadline:
recorded = tasks.list()
if len(recorded) >= expected and all(
task.status is MemoryTaskStatus.COMPLETED for task in recorded
):
break
await asyncio.sleep(0.01)
else: # pragma: no cover - only on a real stall
unfinished = [
(t.task_id, str(t.status))
for t in tasks.list()
if t.status is not MemoryTaskStatus.COMPLETED
]
await supervisor.shutdown()
raise AssertionError(
f"{expected} expected, {len(tasks.list())} recorded, still unfinished after "
f"{timeout}s: {unfinished}. A drain that gives up quietly would let every "
"measurement below be taken off a run that never finished."
)
elapsed = time.monotonic() - started
await supervisor.shutdown()
return elapsed
def _supervise(tmp_path, stores, bridge, **kwargs):
evidence, profiles, tasks, governance, secrets, _profile = stores
return MemoryProcessingSupervisor(
tasks=tasks,
profiles=profiles,
secrets=secrets,
evidence=evidence,
governance=governance,
data_dir=tmp_path,
runner_factory=lambda: _runner(tmp_path, stores, bridge),
**kwargs,
)
def test_two_turns_of_one_conversation_are_never_in_the_model_together(tmp_path) -> None:
"""HY cannot tell what supersedes what if it is told both at once."""
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
for turn in range(4):
_put(
evidence,
message_id=f"same-{turn}",
role="user",
written_by=WrittenBy.PERSON,
content=f"同一段对话的第 {turn} 句,说的是同一件事的不同版本。",
conversation_id="the-one-conversation",
)
bridge = WatchedBridge()
async def exercise() -> None:
# Four workers on purpose: with room to overlap, only the scheduler's
# own rule can keep these apart.
supervisor = _supervise(tmp_path, stores, bridge, max_concurrent_tasks=4)
await _drain(supervisor, tasks, expected=1)
asyncio.run(exercise())
assert bridge.requests, "nothing reached the model, so this run proves nothing"
assert bridge.same_conversation_overlaps == [], (
f"{len(bridge.same_conversation_overlaps)} extractions from the same "
"conversation were inside the model at the same time; HY decides "
"supersede from what it has already been told, so neither could see "
"the other's conclusion"
)
def test_separate_harnesses_run_side_by_side(tmp_path) -> None:
"""Harness ordering does not serialize unrelated sources."""
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
_conversations(evidence, count=8, sources=("cursor", "codex"))
bridge = WatchedBridge()
async def exercise() -> None:
# No max_concurrent_tasks: whatever the profile says is what should
# happen, and nobody in the product passes one.
supervisor = _supervise(tmp_path, stores, bridge)
await _drain(supervisor, tasks, expected=8)
asyncio.run(exercise())
assert bridge.peak > 1, (
f"two separate harnesses were extracted {bridge.peak} at a time; "
"the scheduler is still working through harnesses one after another"
)
def test_durable_drain_and_scheduler_share_one_source_lock(tmp_path) -> None:
"""A direct durable write cannot overlap the scheduler for one source."""
stores = _setup(tmp_path)
evidence, profiles, tasks, governance, secrets, _profile = stores
_put(
evidence,
message_id="scheduled",
role="user",
written_by=WrittenBy.PERSON,
content="后台调度中的 WorkBuddy 对话。",
conversation_id="scheduled-conversation",
source_id="workbuddy",
)
_put(
evidence,
message_id="scheduled-answer",
role="assistant",
written_by=WrittenBy.AGENT,
content="后台调度中的 WorkBuddy 回复。",
conversation_id="scheduled-conversation",
source_id="workbuddy",
)
with sqlite3.connect(tmp_path / "one-memory.sqlite3") as database:
database.executescript(
"""
CREATE TABLE source_grants (
user_id TEXT NOT NULL,
source_id TEXT NOT NULL,
root_path TEXT NOT NULL,
candidate_kind TEXT NOT NULL,
granted_at TEXT NOT NULL,
revoked_at TEXT,
PRIMARY KEY (user_id, source_id)
);
INSERT INTO source_grants VALUES (
'local','workbuddy','/authorized','default',
'2026-10-04T00:00:00+00:00',NULL
);
"""
)
bridge = WatchedBridge(seconds=0.2)
async def exercise() -> None:
scheduled_supervisor = _supervise(
tmp_path,
stores,
bridge,
max_concurrent_tasks=4,
)
direct_supervisor = _supervise(
tmp_path,
stores,
bridge,
max_concurrent_tasks=4,
)
scheduled_supervisor.backfill()
for _ in range(100):
if bridge.inside:
break
await asyncio.sleep(0.01)
assert bridge.inside, "scheduler never entered the model"
direct = _put(
evidence,
message_id="direct",
role="user",
written_by=WrittenBy.PERSON,
content="durable drain 同时送来的 WorkBuddy 对话。",
conversation_id="direct-conversation",
source_id="workbuddy",
)
direct_answer = _put(
evidence,
message_id="direct-answer",
role="assistant",
written_by=WrittenBy.AGENT,
content="durable drain 同时送来的 WorkBuddy 回复。",
conversation_id="direct-conversation",
source_id="workbuddy",
)
await direct_supervisor.process_records((direct, direct_answer))
for _ in range(100):
if len(bridge.requests) >= 2:
break
await asyncio.sleep(0.01)
await asyncio.gather(
scheduled_supervisor.shutdown(),
direct_supervisor.shutdown(),
)
asyncio.run(exercise())
assert len(bridge.requests) >= 2, "both execution paths must reach the model"
assert bridge.overlaps == [], (
f"the durable drain and scheduler entered HY together for one source: {bridge.overlaps}"
)
def test_side_by_side_stops_at_the_profile_limit(tmp_path) -> None:
"""The limit is the model's own, not a number invented by the scheduler."""
stores = _setup(tmp_path)
evidence, profiles, tasks, _governance, secrets, profile = stores
allowed = 3
profile = profiles.update(
profile.profile_id,
ModelProfilePatch(expected_revision=profile.revision, max_concurrency=allowed),
secrets,
)
# A changed profile has to be probed again before it can be the active one.
profiles.record_capabilities(
profile.profile_id,
expected_revision=profile.revision,
capabilities=ModelCapabilities(
endpoint_reachable=True,
model_available=True,
structured_output=True,
embedding_available=True,
),
secrets=secrets,
)
profiles.activate(profile.profile_id, secrets)
_conversations(
evidence,
count=8,
sources=("cursor", "codex", "claude-code", "workbuddy"),
)
bridge = WatchedBridge()
async def exercise() -> None:
supervisor = _supervise(tmp_path, stores, bridge)
await _drain(supervisor, tasks, expected=8)
asyncio.run(exercise())
assert bridge.peak > 1, "nothing ran side by side, so the ceiling is untested"
assert bridge.peak <= allowed, (
f"the profile allows {allowed} at a time and {bridge.peak} were in the model together"
)
def test_a_queue_that_is_entirely_blocked_stops_starting_workers(tmp_path) -> None:
"""Serialising by conversation must not turn into a spin.
A worker that finds nothing to take exits, and a worker exiting is what
starts the next one. So a queue that is full but entirely blocked on the
one conversation already running will start workers forever unless the
question asked before starting one is "is there anything runnable",
not "is the queue non-empty".
The bound is deliberately loose: the difference between correct and
wrong here is a handful against thousands, not a tuned number.
"""
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
turns = 6
_conversations(evidence, count=1, turns=turns)
bridge = WatchedBridge(seconds=0.05)
starts = 0
async def exercise() -> None:
nonlocal starts
supervisor = _supervise(tmp_path, stores, bridge)
start_one = supervisor._start_worker # noqa: SLF001 - counting the thing under test
def counted() -> None:
nonlocal starts
starts += 1
start_one()
supervisor._start_worker = counted # noqa: SLF001
await _drain(supervisor, tasks, expected=turns)
asyncio.run(exercise())
assert starts > 0, "no worker was ever started, so the count proves nothing"
assert starts <= 10 * turns, (
f"{starts} workers were started for {turns} pieces of work, all of one "
"conversation; each one found nothing to take and its exit started the next"
)
def test_nothing_queued_is_lost_when_the_core_stops(tmp_path) -> None:
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
_conversations(evidence, count=8)
bridge = WatchedBridge(seconds=0.5)
async def exercise() -> None:
supervisor = _supervise(tmp_path, stores, bridge)
supervisor.backfill()
deadline = asyncio.get_running_loop().time() + 10.0
try:
while not tasks.list():
if asyncio.get_running_loop().time() >= deadline:
raise AssertionError(
"backfill recorded no task within 10s; an unbounded wait "
"would hang the complete Core release gate"
)
await asyncio.sleep(0.01)
# Stop in the middle, which is what closing the window does.
finally:
await supervisor.shutdown()
asyncio.run(exercise())
recorded = tasks.list()
assert recorded, "no task was ever recorded, so nothing was at risk"
stranded = [
task for task in recorded if task.status not in {MemoryTaskStatus.COMPLETED, *_CLAIMABLE}
]
assert stranded == [], (
f"{len(stranded)} tasks were left in a state no later run will pick up: "
f"{[(t.task_id, str(t.status)) for t in stranded]}"
)
_CLAIMABLE = {
MemoryTaskStatus.PENDING,
MemoryTaskStatus.WAITING_FOR_MODEL,
MemoryTaskStatus.FAILED,
MemoryTaskStatus.RUNNING,
}
def test_running_several_at_once_never_reports_the_database_locked(tmp_path) -> None:
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
_conversations(evidence, count=12)
bridge = WatchedBridge(seconds=0.05)
async def exercise() -> None:
supervisor = _supervise(tmp_path, stores, bridge)
await _drain(supervisor, tasks, expected=12)
asyncio.run(exercise())
refused = [
(task.task_id, task.last_error_code)
for task in tasks.list()
if task.last_error_code and "lock" in task.last_error_code.lower()
]
assert refused == [], (
f"extraction was refused the database while running side by side: {refused}"
)
assert all(task.status is MemoryTaskStatus.COMPLETED for task in tasks.list()), (
"not every task finished, so a locked database could simply not have been reached yet"
)
@pytest.mark.parametrize("workers", [1, 4])
def test_no_database_connection_is_held_while_the_model_thinks(tmp_path, workers) -> None:
"""Waiting on a model must not cost anyone a connection.
Asked of the waiting thread itself, not of the database as a whole: with
several workers, another one being mid-write while this one waits is
exactly what is supposed to happen, and a process-wide reading calls that
a fault. What must not happen is the thread that is about to sit on an
`await` still holding a checkout of its own — the loop thread does every
model call, so one held there is one nobody else can have for as long as
the model takes.
Checked at both settings because holding one only costs something when
there is someone else to take it from.
"""
from one_memory.database_access import main_database
stores = _setup(tmp_path)
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
_conversations(evidence, count=4)
access = main_database(tmp_path / "one-memory.sqlite3")
held: list[tuple[bool, int]] = []
class Sampling(WatchedBridge):
async def _spend_a_moment(self, request) -> None:
me = threading.get_ident()
held.append(
(
access._writer_owner == me, # noqa: SLF001 - the point of the test
access._reader_holders.get(me, 0), # noqa: SLF001
)
)
await super()._spend_a_moment(request)
bridge = Sampling(seconds=0.1)
async def exercise() -> None:
supervisor = _supervise(tmp_path, stores, bridge, max_concurrent_tasks=workers)
await _drain(supervisor, tasks, expected=4)
asyncio.run(exercise())
assert held, "the model was never reached, so nothing was sampled"
assert [sample for sample in held if sample != (False, 0)] == [], (
f"the thread about to wait on the model was still holding a checkout of its own: {held}"
)
Loaded 100 of 134 files, more files were not shown because too many files have changed in this diff. Show more