Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c4da617d59 | ||
|
|
729e699eab | ||
|
|
d926844173 | ||
|
|
de4292ed4e |
No files matched your search
@@ -0,0 +1,49 @@
|
||||
name: Hermes provider
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "plugins/hermes/**"
|
||||
- ".github/workflows/hermes-provider.yml"
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "plugins/hermes/**"
|
||||
- ".github/workflows/hermes-provider.yml"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
provider:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 3
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: astral-sh/setup-uv@d0d8abe699bfb85fec6de9f7adb5ae17292296ff
|
||||
with:
|
||||
python-version: "3.12"
|
||||
enable-cache: false
|
||||
- name: Download and verify native Hermes contract
|
||||
run: |
|
||||
uv run --no-project --python 3.12 python - <<'PY'
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import urllib.request
|
||||
|
||||
url = "https://raw.githubusercontent.com/NousResearch/hermes-agent/818c13be1dc4fd28987e1e881a9408224afd4535/agent/memory_provider.py"
|
||||
with urllib.request.urlopen(url, timeout=30) as response:
|
||||
data = response.read()
|
||||
expected = "a4e44a293013fff831ac136ab77c556541cd18817f53bd662e97090cd80bd5a6"
|
||||
if hashlib.sha256(data).hexdigest() != expected:
|
||||
raise SystemExit("Hermes contract SHA-256 mismatch")
|
||||
snapshot = Path(os.environ["RUNNER_TEMP"]) / "hermes-memory-provider.py"
|
||||
snapshot.write_bytes(data)
|
||||
with open(os.environ["GITHUB_ENV"], "a") as env:
|
||||
env.write(f"HERMES_MEMORY_PROVIDER_SNAPSHOT={snapshot}\n")
|
||||
PY
|
||||
- name: Test provider with native contract
|
||||
run: uv run --no-project --python 3.12 --with pytest python -m pytest plugins/hermes/tests -o addopts= -o log_cli=false -q
|
||||
+20
-9
@@ -40,6 +40,17 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Removed MemHub's private `hy_memory_bridge` implementation. MemHub now calls
|
||||
public `HyMemoryClient` methods at runtime; prompt/reconcile/response
|
||||
monkey-patches and private VDB/Coding runtime access were deleted. Kuzu Graph
|
||||
projection/admin and stopped-writer App migration are the documented product
|
||||
exceptions.
|
||||
- Extended public SDK `get()` to resolve Chat, Coding Experience, and Working
|
||||
Fact IDs, including raw records, owner/session fields, and explicit ambiguity
|
||||
handling; added constructor-time public embedding-service injection.
|
||||
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
|
||||
persistent per-session memory-ID deduplication, authorized Broker reads, and
|
||||
recall-context stripping to prevent memory recursion.
|
||||
- Added an extensible MemHub interface-localization layer with Simplified
|
||||
Chinese, English, and Spanish catalogs shared by Web and Electron-native
|
||||
surfaces. Interface language remains independent from memory content and
|
||||
@@ -47,13 +58,12 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
- `search(..., layers=["l1_raw"])` now explicitly searches SDK raw records in
|
||||
pro/ultra mode and keeps existing user/agent/session/source filters; default
|
||||
search continues to hide raw.
|
||||
- Added `import_memory_exact` for idempotent migration of an existing semantic
|
||||
memory with its canonical ID, lifecycle and supersession links, without
|
||||
extraction or reconciliation.
|
||||
- Added SDK-authoritative lifecycle listing with explicit status filters,
|
||||
idempotent durable write outcomes, exact raw import/get/list APIs, and
|
||||
quiesced local data-plane snapshot/restore contracts for the MemHub
|
||||
authority migration.
|
||||
idempotent durable write outcomes, and quiesced local data-plane
|
||||
snapshot/restore contracts for the MemHub authority migration. Exact legacy
|
||||
backfill is now an App-owned privileged offline operation rather than a
|
||||
public `HyMemoryClient` API. Backfill runs against a complete staging copy;
|
||||
cutover atomically swaps SDK roots and restores the previous root on failure.
|
||||
- Physical delete now reports incomplete VDB/Graph cleanup as a partial failure
|
||||
instead of claiming success after only one store was changed.
|
||||
- Coding search no longer injects the version-evolution diff block
|
||||
@@ -76,9 +86,10 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
retry scheduling for their memory tasks.
|
||||
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
|
||||
while persistent caller business context belongs in `meta`.
|
||||
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
|
||||
memories, including memories written in lite mode. The default remains
|
||||
hidden for backward compatibility.
|
||||
- Unified exact reads and lists across memory layers: `get(memory_id)` returns
|
||||
the matched layer, while `list_memories(layers=["l1_raw"])` explicitly lists
|
||||
raw records. Separate raw get/list methods and the `include_raw` switch were
|
||||
removed.
|
||||
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
||||
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
||||
its native local stores directly.
|
||||
|
||||
@@ -253,6 +253,7 @@ That version is usually not in the working tree or at HEAD either, so it cannot
|
||||
- [Graph memory design](docs/pro_graph_design.md) — graph projection, rerank, relation paths, and deployment limits
|
||||
- [Environment template](.env.example) — a copy-ready configuration template
|
||||
- [MemHub README](memhub/README.md) — local-first desktop & CLI product overview, runtime commands, and privacy boundaries
|
||||
- [Hermes native memory provider](plugins/hermes/README.md) — explicit owner/agent scope; separate HTTP service; automatic capture and recall opt-in.
|
||||
- [OpenClaw plugin](plugins/openclaw/README.md) — agent integrations live under `plugins/` (overview: [plugins/README.md](plugins/README.md))
|
||||
|
||||
### License
|
||||
|
||||
@@ -253,6 +253,7 @@ Coding 与 Working **复用同一套双层 pipeline 和数据 schema**,区别
|
||||
- [Graph 记忆设计](docs/pro_graph_design.md) — 图投影、重排、关系路径与部署限制
|
||||
- [环境变量模板](.env.example) — 可直接复制的配置模板
|
||||
- [MemHub 说明文档](memhub/README.md) — 本地优先桌面端与 CLI 产品总览、Runtime 命令与隐私边界
|
||||
- [Hermes 原生记忆 Provider](plugins/hermes/README.md) — 显式用户与 Agent 隔离,独立 HTTP 服务,自动记录与召回默认关闭。
|
||||
- [OpenClaw 插件](plugins/openclaw/README.md) — Agent 集成见 `plugins/`(总览:[plugins/README.md](plugins/README.md))
|
||||
|
||||
### License
|
||||
|
||||
+7
-6
@@ -273,8 +273,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
|
||||
|
||||
```python
|
||||
memory = client.get("memory_id")
|
||||
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
|
||||
raw_memory = client.get("memory_id", include_raw=True)
|
||||
# raw ID 返回 L1_RAW,fact ID 返回对应事实层。
|
||||
raw_memory = client.get("raw_id")
|
||||
result = client.update("memory_id", "更新后的内容")
|
||||
result = client.delete("memory_id")
|
||||
result = client.delete_all(user_id="user_001")
|
||||
@@ -284,7 +284,7 @@ page = client.list_memories(
|
||||
limit=100,
|
||||
offset=0,
|
||||
order="desc",
|
||||
layers=None,
|
||||
layers=["l1_raw"], # 显式列出 raw;不传仍沿用默认语义层列表
|
||||
with_evolution=False,
|
||||
)
|
||||
|
||||
@@ -294,9 +294,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
|
||||
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
|
||||
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
|
||||
|
||||
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
|
||||
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
|
||||
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
|
||||
`get()` / `async_get()` 是统一的 ID 精确读取:返回结果的 `layer`、`kind` 和
|
||||
`variant` 表明实际类型。授权由调用 SDK 前的产品边界完成,不由 SDK 根据 layer
|
||||
猜测。`list_memories()` 默认仍不包含 `L1_RAW`;显式传
|
||||
`layers=["l1_raw"]` 时按同一列表合同返回 raw。
|
||||
|
||||
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
|
||||
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
|
||||
|
||||
@@ -65,6 +65,12 @@ from .snapshot import (
|
||||
|
||||
# ====== 配置 ======
|
||||
from .config import MemoryConfig
|
||||
from .embedding import (
|
||||
EmbeddingDimensionMismatchError,
|
||||
EmbeddingService,
|
||||
EmbeddingServiceProtocol,
|
||||
)
|
||||
from .retrieval import AmbiguousMemoryIdError, MemoryKind, MemoryVariant
|
||||
|
||||
# ====== Embedding 端点覆盖(per-call,不改 model/dims,只换接入端点)======
|
||||
from .core.embed_service import embed_endpoint_override
|
||||
@@ -125,6 +131,13 @@ __all__ = [
|
||||
"snapshot_local_data_plane",
|
||||
# 配置
|
||||
"MemoryConfig",
|
||||
# 公共扩展协议 / 精确读取类型
|
||||
"EmbeddingService",
|
||||
"EmbeddingServiceProtocol",
|
||||
"EmbeddingDimensionMismatchError",
|
||||
"MemoryKind",
|
||||
"MemoryVariant",
|
||||
"AmbiguousMemoryIdError",
|
||||
# Embedding 端点覆盖
|
||||
"embed_endpoint_override",
|
||||
# 高级 API
|
||||
|
||||
+280
-456
@@ -38,6 +38,8 @@ from typing import List, Dict, Any, Optional, Tuple, Union, TYPE_CHECKING
|
||||
|
||||
from .config import MemoryConfig
|
||||
from .core.embed_service import EmbedService, embed_endpoint_override
|
||||
from .embedding import EmbeddingDimensionMismatchError, EmbeddingService
|
||||
from .retrieval import AmbiguousMemoryIdError, MemoryKind, MemoryVariant
|
||||
from .data.vector_store import create_vector_store
|
||||
from .data.history_store import HistoryStore
|
||||
from .data.cache import create_cache
|
||||
@@ -434,6 +436,8 @@ class HyMemoryClient:
|
||||
Args:
|
||||
config: 自定义 MemoryConfig(默认从环境变量加载)
|
||||
mode: 处理模式 ("lite" / "pro" / "ultra"),默认从环境变量 MEMORY_MODE 读取
|
||||
embedding_service: 可选的调用方自管 embedding service;须满足公开的
|
||||
EmbeddingService Protocol,client 不接管其生命周期。
|
||||
"""
|
||||
|
||||
# 合法的 mode 值
|
||||
@@ -445,12 +449,15 @@ class HyMemoryClient:
|
||||
mode: Optional[str] = None,
|
||||
coding_writer: Optional[str] = None,
|
||||
runtime: Optional["SharedRuntime"] = None,
|
||||
embedding_service: Optional[EmbeddingService] = None,
|
||||
):
|
||||
# 确保日志系统已配置(幂等,多次调用安全)
|
||||
setup_logging()
|
||||
|
||||
# 配置
|
||||
self._config = config or MemoryConfig.from_env()
|
||||
self._borrowed_embedding_service = embedding_service
|
||||
self._validate_embedding_service(embedding_service)
|
||||
|
||||
# coding_writer 显式传入则覆盖 config / env(默认 None → 沿用 config.coding.writer)
|
||||
# 取值: "legacy" | "agent"
|
||||
@@ -537,8 +544,12 @@ class HyMemoryClient:
|
||||
)
|
||||
logger.info(f"[init] Default ThreadPoolExecutor max_workers={_max_workers}")
|
||||
|
||||
# 1. EmbedService(纯配置,首次调用时真正连接 API)
|
||||
self._embed_service = EmbedService(self._config)
|
||||
# 1. EmbedService(注入发生在 ComponentFactory / pipeline 预热之前)
|
||||
self._embed_service = (
|
||||
self._borrowed_embedding_service
|
||||
if self._borrowed_embedding_service is not None
|
||||
else EmbedService(self._config)
|
||||
)
|
||||
logger.debug(
|
||||
f"[init] EmbedService created: model={self._config.embedder.model}"
|
||||
)
|
||||
@@ -568,8 +579,10 @@ class HyMemoryClient:
|
||||
)
|
||||
|
||||
# 4. ComponentFactory(注入已初始化的共享组件)
|
||||
self._registry = ComponentFactory(config=self._config)
|
||||
self._registry._shared_embed_service = self._embed_service
|
||||
self._registry = ComponentFactory(
|
||||
config=self._config,
|
||||
embedding_service=self._embed_service,
|
||||
)
|
||||
self._registry._shared_vector_store = self._vector_store
|
||||
self._registry._shared_vector_store_initialized = True
|
||||
if self._graph_store is not None:
|
||||
@@ -703,6 +716,7 @@ class HyMemoryClient:
|
||||
cls,
|
||||
config_dict: Dict[str, Any],
|
||||
mode: Optional[str] = None,
|
||||
embedding_service: Optional[EmbeddingService] = None,
|
||||
) -> "HyMemoryClient":
|
||||
"""
|
||||
从字典创建客户端(类似 mem0 风格)。
|
||||
@@ -734,7 +748,49 @@ class HyMemoryClient:
|
||||
"""
|
||||
config = MemoryConfig.from_dict(config_dict)
|
||||
resolved_mode = mode or config_dict.get("mode")
|
||||
return cls(config=config, mode=resolved_mode)
|
||||
return cls(
|
||||
config=config,
|
||||
mode=resolved_mode,
|
||||
embedding_service=embedding_service,
|
||||
)
|
||||
|
||||
def _validate_embedding_service(
|
||||
self, embedding_service: Optional[EmbeddingService]
|
||||
) -> None:
|
||||
"""Validate a borrowed service before any store or pipeline is created."""
|
||||
if embedding_service is None:
|
||||
return
|
||||
required = (
|
||||
"embed",
|
||||
"embed_with_meta",
|
||||
"embed_batch",
|
||||
"embed_queued",
|
||||
"embed_queued_with_meta",
|
||||
"get_embedding_dims",
|
||||
)
|
||||
missing = [
|
||||
name for name in required
|
||||
if not callable(getattr(embedding_service, name, None))
|
||||
]
|
||||
if missing:
|
||||
raise TypeError(
|
||||
"embedding_service does not implement EmbeddingService; "
|
||||
f"missing: {', '.join(missing)}"
|
||||
)
|
||||
try:
|
||||
actual_dims = int(embedding_service.get_embedding_dims())
|
||||
except Exception as exc:
|
||||
raise TypeError(
|
||||
"embedding_service.get_embedding_dims() must return an integer"
|
||||
) from exc
|
||||
expected_dims = getattr(self._config.vector_store, "embedding_dims", None)
|
||||
if expected_dims is None:
|
||||
expected_dims = getattr(self._config.embedder, "embedding_dims", None)
|
||||
if expected_dims is not None and actual_dims != int(expected_dims):
|
||||
raise EmbeddingDimensionMismatchError(
|
||||
"embedding service dimension mismatch: "
|
||||
f"service={actual_dims}, vector_store={int(expected_dims)}"
|
||||
)
|
||||
|
||||
# ================================================================
|
||||
# 属性
|
||||
@@ -1060,6 +1116,7 @@ class HyMemoryClient:
|
||||
llm_provider=self._coding_llm_provider,
|
||||
embed_service=self._embed_service,
|
||||
cache=self._cache,
|
||||
variant=variant,
|
||||
)
|
||||
self._coding_stores[variant] = store
|
||||
self._coding_writers[variant] = writer
|
||||
@@ -1829,21 +1886,26 @@ class HyMemoryClient:
|
||||
self,
|
||||
memory_id: str,
|
||||
*,
|
||||
include_raw: bool = False,
|
||||
kind: Optional[MemoryKind] = None,
|
||||
variant: Optional[MemoryVariant] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
获取单条记忆。
|
||||
|
||||
Args:
|
||||
memory_id: 记忆 ID
|
||||
include_raw: 是否允许返回 L1_RAW 原始记忆。默认 False,保持原有隐藏行为;
|
||||
lite 模式的记忆均为 L1_RAW,需显式设为 True 才能按 ID 读取。
|
||||
kind: 可选 namespace("chat" / "experience" / "fact")。
|
||||
variant: 可选生产力变体("working" / "coding")。
|
||||
|
||||
Returns:
|
||||
{"memory_id", "content", "layer", "request_id", "elapsed_ms"} 或 None(不存在时)
|
||||
带 kind/variant 的统一 memory dict,或 None(不存在时)。
|
||||
"""
|
||||
return self._loop_thread.run(
|
||||
self.async_get(memory_id, include_raw=include_raw)
|
||||
self.async_get(
|
||||
memory_id,
|
||||
kind=kind,
|
||||
variant=variant,
|
||||
)
|
||||
)
|
||||
|
||||
def update(
|
||||
@@ -1912,6 +1974,9 @@ class HyMemoryClient:
|
||||
branch: Optional[str] = None,
|
||||
user_ids: Optional[List[str]] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
include_sources: Optional[List[str]] = None,
|
||||
exclude_sources: Optional[List[str]] = None,
|
||||
layers: Optional[List[Any]] = None,
|
||||
statuses: Optional[List[Any]] = None,
|
||||
with_evolution: bool = False,
|
||||
@@ -1934,6 +1999,9 @@ class HyMemoryClient:
|
||||
user_ids: 可选,多用户批量列举。非空时优先于 user_id,返回并集
|
||||
(分页在合并排序后统一施加)。
|
||||
agent_ids: 可选,多 agent 批量列举。非空时优先于 agent_id。
|
||||
session_ids: 可选,只返回指定 session。
|
||||
include_sources: 可选,只返回指定来源。
|
||||
exclude_sources: 可选,排除指定来源;与 include_sources 冲突时排除优先。
|
||||
layers: 可选,只返回指定层级(字符串或 MemoryLayer)。不传时返回
|
||||
L1_RAW 以外的全部层级,与改动前一致。
|
||||
statuses: 可选,只返回指定生命周期(字符串或 MemoryStatus)。不传时
|
||||
@@ -1949,6 +2017,9 @@ class HyMemoryClient:
|
||||
self.async_list_memories(
|
||||
user_id=user_id, agent_id=agent_id,
|
||||
user_ids=user_ids, agent_ids=agent_ids,
|
||||
session_ids=session_ids,
|
||||
include_sources=include_sources,
|
||||
exclude_sources=exclude_sources,
|
||||
limit=limit, offset=offset, order=order,
|
||||
workspace_id=workspace_id, branch=branch,
|
||||
layers=layers, statuses=statuses, with_evolution=with_evolution,
|
||||
@@ -2737,6 +2808,8 @@ class HyMemoryClient:
|
||||
request_id=request_id,
|
||||
source=source,
|
||||
)
|
||||
# Public receipt always identifies the productivity namespace.
|
||||
resp["variant"] = scene_variant
|
||||
# 版本演进链层(scene="coding" 独有)。放在 writer.write 之后、返回
|
||||
# 之前,因为它只依赖对话原文,与向量层写入结果无关。
|
||||
# best-effort:这层挂不上不影响本次 add 的向量层产出。
|
||||
@@ -2907,106 +2980,6 @@ class HyMemoryClient:
|
||||
include_runtime_diag=True,
|
||||
)
|
||||
|
||||
def import_raw(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str = "default_session",
|
||||
source: str = "default",
|
||||
memory_at: Optional[datetime] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
raw_id: Optional[str] = None,
|
||||
request_id: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Persist one exact raw record without extraction or reconciliation."""
|
||||
return self._loop_thread.run(
|
||||
self.async_import_raw(
|
||||
content,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
raw_id=raw_id,
|
||||
request_id=request_id,
|
||||
),
|
||||
include_runtime_diag=True,
|
||||
)
|
||||
|
||||
def import_memory_exact(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
memory_id: str,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str = "default_session",
|
||||
source: str = "default",
|
||||
layer: Any = "l2_fact",
|
||||
status: Any = "active",
|
||||
owner: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
supersedes: Optional[List[str]] = None,
|
||||
superseded_by: Optional[List[str]] = None,
|
||||
is_latest: bool = True,
|
||||
memory_at: Optional[datetime] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
request_id: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Import one existing semantic memory without extraction/reconcile."""
|
||||
return self._loop_thread.run(
|
||||
self.async_import_memory_exact(
|
||||
content,
|
||||
memory_id=memory_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
layer=layer,
|
||||
status=status,
|
||||
owner=owner,
|
||||
tags=tags,
|
||||
supersedes=supersedes,
|
||||
superseded_by=superseded_by,
|
||||
is_latest=is_latest,
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
request_id=request_id,
|
||||
),
|
||||
include_runtime_diag=True,
|
||||
)
|
||||
|
||||
def get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Read one SDK raw record through the explicit raw API."""
|
||||
return self._loop_thread.run(self.async_get_raw(raw_id))
|
||||
|
||||
def list_raw(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
source: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> Dict[str, Any]:
|
||||
"""List raw records for an authorized SDK owner/scope."""
|
||||
return self._loop_thread.run(
|
||||
self.async_list_raw(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
order=order,
|
||||
)
|
||||
)
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_add_extracted(
|
||||
self,
|
||||
@@ -3159,316 +3132,6 @@ class HyMemoryClient:
|
||||
}
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_import_raw(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str = "default_session",
|
||||
source: str = "default",
|
||||
memory_at: Optional[datetime] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
raw_id: Optional[str] = None,
|
||||
request_id: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Persist exact source content as L1_RAW without running a model."""
|
||||
from .utils.source_filter import normalize_source
|
||||
|
||||
if not isinstance(content, str) or not content:
|
||||
raise ValueError("content is required")
|
||||
if not user_id:
|
||||
raise ValueError("user_id is required")
|
||||
request_id = request_id or str(uuid.uuid4())
|
||||
replay = await self._claim_write_outcome(
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
operation="import_raw",
|
||||
)
|
||||
if replay is not None:
|
||||
return replay
|
||||
try:
|
||||
result = await self._async_import_raw_impl(
|
||||
content,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=normalize_source(source),
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
raw_id=raw_id,
|
||||
request_id=request_id,
|
||||
)
|
||||
except BaseException:
|
||||
await self._mark_write_outcome_unknown(request_id)
|
||||
raise
|
||||
await self._finish_write_outcome(request_id, "succeeded", result)
|
||||
return result
|
||||
|
||||
async def _async_import_raw_impl(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
memory_at: Optional[datetime],
|
||||
metadata: Optional[Dict[str, Any]],
|
||||
raw_id: Optional[str],
|
||||
request_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
from .models.memory import MemoryLayer, MemoryNode, MemoryStatus, SourceType
|
||||
|
||||
node_id = raw_id or str(uuid.uuid4())
|
||||
existing = await self._vector_store.get_by_id(node_id)
|
||||
if existing is not None:
|
||||
identity = (
|
||||
existing.layer == MemoryLayer.L1_RAW,
|
||||
existing.content == content,
|
||||
existing.user_id == user_id,
|
||||
existing.agent_id == agent_id,
|
||||
existing.session_id == session_id,
|
||||
existing.source == source,
|
||||
)
|
||||
if not all(identity):
|
||||
raise ValueError("raw_id is already bound to different source content")
|
||||
return {
|
||||
"success": True,
|
||||
"raw_id": node_id,
|
||||
"request_id": request_id,
|
||||
"replayed": True,
|
||||
}
|
||||
|
||||
custom = dict(metadata or {})
|
||||
custom["request_id"] = request_id
|
||||
custom["input_kind"] = "raw_exact_import"
|
||||
embedding = await self._embed_service.embed(content)
|
||||
node = MemoryNode(
|
||||
node_id=node_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
layer=MemoryLayer.L1_RAW,
|
||||
content=content,
|
||||
source_type=SourceType.EXPLICIT,
|
||||
status=MemoryStatus.ACTIVE,
|
||||
is_latest=True,
|
||||
embedding=embedding,
|
||||
memory_at=memory_at,
|
||||
custom=custom,
|
||||
)
|
||||
stored_id = await self._vector_store.upsert(node)
|
||||
if stored_id != node_id:
|
||||
raise RuntimeError("raw import changed the requested identity")
|
||||
return {"success": True, "raw_id": node_id, "request_id": request_id}
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_import_memory_exact(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
memory_id: str,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str = "default_session",
|
||||
source: str = "default",
|
||||
layer: Any = "l2_fact",
|
||||
status: Any = "active",
|
||||
owner: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
supersedes: Optional[List[str]] = None,
|
||||
superseded_by: Optional[List[str]] = None,
|
||||
is_latest: bool = True,
|
||||
memory_at: Optional[datetime] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
request_id: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Import a canonical memory node exactly, with no model invocation."""
|
||||
from .models.memory import MemoryLayer, MemoryStatus
|
||||
from .utils.source_filter import normalize_source
|
||||
|
||||
if not content or not memory_id or not user_id:
|
||||
raise ValueError("content, memory_id and user_id are required")
|
||||
normalized_layer = (
|
||||
layer if isinstance(layer, MemoryLayer) else MemoryLayer.from_string(str(layer))
|
||||
)
|
||||
if normalized_layer == MemoryLayer.L1_RAW:
|
||||
raise ValueError("use import_raw for L1_RAW")
|
||||
normalized_status = (
|
||||
status if isinstance(status, MemoryStatus) else MemoryStatus(str(status).lower())
|
||||
)
|
||||
request_id = request_id or str(uuid.uuid4())
|
||||
replay = await self._claim_write_outcome(
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
operation="import_memory_exact",
|
||||
)
|
||||
if replay is not None:
|
||||
return replay
|
||||
try:
|
||||
result = await self._async_import_memory_exact_impl(
|
||||
content,
|
||||
memory_id=memory_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=normalize_source(source),
|
||||
layer=normalized_layer,
|
||||
status=normalized_status,
|
||||
owner=owner,
|
||||
tags=tags or [],
|
||||
supersedes=supersedes or [],
|
||||
superseded_by=superseded_by or [],
|
||||
is_latest=is_latest,
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
request_id=request_id,
|
||||
)
|
||||
except BaseException:
|
||||
await self._mark_write_outcome_unknown(request_id)
|
||||
raise
|
||||
await self._finish_write_outcome(request_id, "succeeded", result)
|
||||
return result
|
||||
|
||||
async def _async_import_memory_exact_impl(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
memory_id: str,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
layer,
|
||||
status,
|
||||
owner: Optional[str],
|
||||
tags: List[str],
|
||||
supersedes: List[str],
|
||||
superseded_by: List[str],
|
||||
is_latest: bool,
|
||||
memory_at: Optional[datetime],
|
||||
metadata: Optional[Dict[str, Any]],
|
||||
request_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
from .models.memory import MemoryNode, SourceType
|
||||
|
||||
existing = await self._vector_store.get_by_id(memory_id)
|
||||
if existing is not None:
|
||||
identity = (
|
||||
existing.content == content,
|
||||
existing.user_id == user_id,
|
||||
existing.agent_id == agent_id,
|
||||
existing.session_id == session_id,
|
||||
existing.source == source,
|
||||
existing.layer == layer,
|
||||
existing.status == status,
|
||||
existing.owner == owner,
|
||||
list(existing.tags or []) == tags,
|
||||
list(existing.supersedes or []) == supersedes,
|
||||
list(existing.superseded_by or []) == superseded_by,
|
||||
existing.is_latest == is_latest,
|
||||
)
|
||||
if not all(identity):
|
||||
raise ValueError("memory_id is already bound to different memory content")
|
||||
return {
|
||||
"success": True,
|
||||
"memory_id": memory_id,
|
||||
"memory_ids": [memory_id],
|
||||
"request_id": request_id,
|
||||
"replayed": True,
|
||||
}
|
||||
|
||||
custom = dict(metadata or {})
|
||||
custom["request_id"] = request_id
|
||||
custom["input_kind"] = "memory_exact_import"
|
||||
embedding = await self._embed_service.embed(content)
|
||||
node = MemoryNode(
|
||||
node_id=memory_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
layer=layer,
|
||||
content=content,
|
||||
source_type=SourceType.EXPLICIT,
|
||||
status=status,
|
||||
owner=owner,
|
||||
tags=list(tags),
|
||||
supersedes=list(supersedes),
|
||||
superseded_by=list(superseded_by),
|
||||
is_latest=is_latest,
|
||||
embedding=embedding,
|
||||
memory_at=memory_at,
|
||||
custom=custom,
|
||||
)
|
||||
stored_id = await self._vector_store.upsert(node)
|
||||
if stored_id != memory_id:
|
||||
raise RuntimeError("exact memory import changed the requested identity")
|
||||
return {
|
||||
"success": True,
|
||||
"memory_id": memory_id,
|
||||
"memory_ids": [memory_id],
|
||||
"request_id": request_id,
|
||||
}
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
|
||||
from .models.memory import MemoryLayer
|
||||
|
||||
node = await self._vector_store.get_by_id(raw_id)
|
||||
if node is None or node.layer != MemoryLayer.L1_RAW:
|
||||
return None
|
||||
return self._raw_node_to_public(node)
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_list_raw(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
source: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> Dict[str, Any]:
|
||||
from .models.memory import MemoryLayer, MemoryStatus
|
||||
|
||||
if not user_id:
|
||||
raise ValueError("user_id is required")
|
||||
if limit < 1 or offset < 0:
|
||||
raise ValueError("limit must be positive and offset must be non-negative")
|
||||
nodes = await self._vector_store.list_by_user(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
status_filter=[MemoryStatus.ACTIVE],
|
||||
layers=[MemoryLayer.L1_RAW],
|
||||
include_sources=[source] if source else None,
|
||||
limit=100_000,
|
||||
only_latest=False,
|
||||
with_vectors=False,
|
||||
allow_raw_layer=True,
|
||||
)
|
||||
if session_id:
|
||||
nodes = [node for node in nodes if node.session_id == session_id]
|
||||
nodes = self._sort_memory_nodes(nodes, order=order)
|
||||
total = len(nodes)
|
||||
return {
|
||||
"raw": [
|
||||
self._raw_node_to_public(node)
|
||||
for node in nodes[offset: offset + limit]
|
||||
],
|
||||
"total": total,
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
async def _search_explicit_raw_layer(
|
||||
self,
|
||||
query: str,
|
||||
@@ -4374,35 +4037,100 @@ class HyMemoryClient:
|
||||
# CRUD API(异步)— get / update / delete / delete_all
|
||||
# ================================================================
|
||||
|
||||
async def _coding_store_for_exact_get(self, variant: str):
|
||||
"""Return an existing namespace store without creating an empty DB."""
|
||||
stores = getattr(self, "_coding_stores", None) or {}
|
||||
store = stores.get(variant)
|
||||
if store is not None:
|
||||
return store
|
||||
|
||||
from .coding.store import CodingMemoryStore
|
||||
|
||||
config = getattr(self, "_config", None)
|
||||
coding_config = getattr(config, "coding", None)
|
||||
if config is None or coding_config is None:
|
||||
return None
|
||||
if not CodingMemoryStore.database_exists(
|
||||
config,
|
||||
db_path=coding_config.db_path,
|
||||
variant=variant,
|
||||
):
|
||||
return None
|
||||
await self._get_coding_writer(variant=variant)
|
||||
return (getattr(self, "_coding_stores", None) or {}).get(variant)
|
||||
|
||||
@staticmethod
|
||||
def _format_productivity_get(
|
||||
item: Any,
|
||||
*,
|
||||
kind: str,
|
||||
variant: str,
|
||||
request_id: str,
|
||||
elapsed_ms: float,
|
||||
) -> Dict[str, Any]:
|
||||
payload = item.to_dict()
|
||||
item_id = (
|
||||
payload.get("experience_id")
|
||||
if kind == "experience"
|
||||
else payload.get("fact_id")
|
||||
)
|
||||
return {
|
||||
"memory_id": item_id,
|
||||
"kind": kind,
|
||||
"variant": variant,
|
||||
"scene": "productivity",
|
||||
"content": (
|
||||
payload.get("solution", "")
|
||||
if kind == "experience"
|
||||
else payload.get("statement", "")
|
||||
),
|
||||
"layer": kind,
|
||||
**payload,
|
||||
"request_id": request_id,
|
||||
"elapsed_ms": elapsed_ms,
|
||||
}
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_get(
|
||||
self,
|
||||
memory_id: str,
|
||||
*,
|
||||
include_raw: bool = False,
|
||||
kind: Optional[MemoryKind] = None,
|
||||
variant: Optional[MemoryVariant] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
获取单条记忆(异步)。
|
||||
|
||||
VDB + Graph 双路并行查询,哪个有结果返回哪个。
|
||||
默认查询 chat、working/coding experience 和 working/coding fact。
|
||||
若同一 ID 命中多个 namespace,则 fail closed;调用方必须用 kind/variant
|
||||
显式消歧。
|
||||
|
||||
Args:
|
||||
memory_id: 记忆 ID
|
||||
include_raw: 是否允许返回 L1_RAW 原始记忆,默认 False。
|
||||
kind: 可选 namespace("chat" / "experience" / "fact")。
|
||||
variant: 可选生产力变体("working" / "coding")。
|
||||
|
||||
Returns:
|
||||
{"memory_id", "content", "layer", "source", "request_id", "elapsed_ms"} 或 None
|
||||
统一 memory dict,或 None。
|
||||
"""
|
||||
import asyncio as _asyncio_get
|
||||
from .models.memory import MemoryLayer
|
||||
if kind not in (None, "chat", "experience", "fact"):
|
||||
raise ValueError(
|
||||
"kind must be one of: chat, experience, fact"
|
||||
)
|
||||
if variant not in (None, "working", "coding"):
|
||||
raise ValueError("variant must be one of: working, coding")
|
||||
if kind == "chat" and variant is not None:
|
||||
raise ValueError("variant is only valid for experience/fact lookups")
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
set_request_id(request_id)
|
||||
t0 = time.perf_counter()
|
||||
|
||||
logger.info(f"[get] memory_id={memory_id}")
|
||||
logger.info(
|
||||
"[get] memory_id=%s kind=%s variant=%s",
|
||||
memory_id, kind, variant,
|
||||
)
|
||||
|
||||
# 双路并行
|
||||
async def _get_vdb():
|
||||
return await self._vector_store.get_by_id(memory_id)
|
||||
|
||||
@@ -4415,39 +4143,63 @@ class HyMemoryClient:
|
||||
logger.debug(f"[get] graph_store query failed: {e}")
|
||||
return None
|
||||
|
||||
vdb_node, graph_node = await _asyncio_get.gather(_get_vdb(), _get_graph())
|
||||
query_chat = kind in (None, "chat") and variant is None
|
||||
coding_kinds = (
|
||||
("experience", "fact") if kind is None else (kind,)
|
||||
)
|
||||
coding_variants = (variant,) if variant is not None else ("working", "coding")
|
||||
|
||||
# L1_RAW 默认不对外可见:按 ID 直取走主键查询、绕过向量库的 layer
|
||||
# filter,故在此统一挡掉。SDK 调用方可通过 include_raw=True 显式读取;
|
||||
# HTTP 接口不传该参数,继续保持隐藏。内部写链路不经过此方法。
|
||||
#
|
||||
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
|
||||
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
|
||||
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
|
||||
if (
|
||||
not include_raw
|
||||
and vdb_node is not None
|
||||
and vdb_node.layer == MemoryLayer.L1_RAW
|
||||
):
|
||||
vdb_node = None
|
||||
if (
|
||||
not include_raw
|
||||
and graph_node is not None
|
||||
and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW
|
||||
):
|
||||
graph_node = None
|
||||
stores = {}
|
||||
if kind != "chat":
|
||||
for candidate_variant in coding_variants:
|
||||
store = await self._coding_store_for_exact_get(candidate_variant)
|
||||
if store is not None:
|
||||
stores[candidate_variant] = store
|
||||
|
||||
tasks = []
|
||||
task_names = []
|
||||
if query_chat:
|
||||
tasks.extend((_get_vdb(), _get_graph()))
|
||||
task_names.extend(("chat:vdb", "chat:graph"))
|
||||
for candidate_variant, store in stores.items():
|
||||
if "experience" in coding_kinds:
|
||||
tasks.append(store.get_by_id(memory_id))
|
||||
task_names.append(f"{candidate_variant}:experience")
|
||||
if "fact" in coding_kinds:
|
||||
tasks.append(store.get_fact(memory_id))
|
||||
task_names.append(f"{candidate_variant}:fact")
|
||||
|
||||
values = await asyncio.gather(*tasks) if tasks else []
|
||||
by_name = dict(zip(task_names, values))
|
||||
vdb_node = by_name.get("chat:vdb")
|
||||
graph_node = by_name.get("chat:graph")
|
||||
|
||||
matches = []
|
||||
chat_node = vdb_node if vdb_node is not None else graph_node
|
||||
if chat_node is not None:
|
||||
matches.append(("chat", chat_node))
|
||||
for name, value in by_name.items():
|
||||
if not name.startswith("chat:") and value is not None:
|
||||
matches.append((name, value))
|
||||
|
||||
if len(matches) > 1:
|
||||
raise AmbiguousMemoryIdError(
|
||||
memory_id, [namespace for namespace, _ in matches]
|
||||
)
|
||||
|
||||
elapsed = round((time.perf_counter() - t0) * 1000, 2)
|
||||
|
||||
# pipeline trace(best-effort,受 MEMORY_PIPELINE_TRACE_ENABLED 控制)
|
||||
_hit_node = vdb_node if vdb_node is not None else graph_node
|
||||
_source = "vdb" if vdb_node is not None else ("graph" if graph_node is not None else "")
|
||||
_hit_namespace, _hit_node = matches[0] if matches else ("", None)
|
||||
_source = (
|
||||
"vdb" if vdb_node is not None
|
||||
else ("graph" if graph_node is not None else _hit_namespace)
|
||||
)
|
||||
_ident = self._backend_identity()
|
||||
_parsed = {
|
||||
"memory_id": memory_id,
|
||||
**_ident,
|
||||
"hit": _hit_node is not None,
|
||||
"source": _source,
|
||||
"namespace": _hit_namespace,
|
||||
}
|
||||
if _hit_node is not None:
|
||||
try:
|
||||
@@ -4464,10 +4216,25 @@ class HyMemoryClient:
|
||||
elapsed_ms=elapsed,
|
||||
)
|
||||
|
||||
# VDB 优先(如果两边都有,以 VDB 为准)
|
||||
if not matches:
|
||||
logger.info(f"[get] not found: memory_id={memory_id}")
|
||||
return None
|
||||
|
||||
if _hit_namespace != "chat":
|
||||
hit_variant, hit_kind = _hit_namespace.split(":", 1)
|
||||
return self._format_productivity_get(
|
||||
_hit_node,
|
||||
kind=hit_kind,
|
||||
variant=hit_variant,
|
||||
request_id=request_id,
|
||||
elapsed_ms=elapsed,
|
||||
)
|
||||
|
||||
if vdb_node is not None:
|
||||
return {
|
||||
"memory_id": vdb_node.node_id,
|
||||
"kind": "chat",
|
||||
"variant": None,
|
||||
"content": vdb_node.content,
|
||||
"layer": vdb_node.layer.value,
|
||||
"status": vdb_node.status.value if vdb_node.status else "active",
|
||||
@@ -4481,10 +4248,13 @@ class HyMemoryClient:
|
||||
# 不透出 source_raw_memory_id:见 async_search 同处说明。
|
||||
"user_id": vdb_node.user_id,
|
||||
"agent_id": vdb_node.agent_id,
|
||||
"session_id": vdb_node.session_id,
|
||||
"owner": vdb_node.owner,
|
||||
"confidence": vdb_node.confidence,
|
||||
"custom": vdb_node.custom or {},
|
||||
"meta": list((vdb_node.custom or {}).get("meta") or []),
|
||||
"source": "vdb",
|
||||
"source": vdb_node.source,
|
||||
"retrieval_source": "vdb",
|
||||
"request_id": request_id,
|
||||
"elapsed_ms": elapsed,
|
||||
}
|
||||
@@ -4492,6 +4262,8 @@ class HyMemoryClient:
|
||||
if graph_node is not None:
|
||||
return {
|
||||
"memory_id": graph_node.node_id,
|
||||
"kind": "chat",
|
||||
"variant": None,
|
||||
"content": graph_node.content,
|
||||
"layer": graph_node.layer.value if graph_node.layer else "",
|
||||
"status": graph_node.status.value if graph_node.status else "active",
|
||||
@@ -4505,14 +4277,16 @@ class HyMemoryClient:
|
||||
# 不透出 source_raw_memory_id:见 async_search 同处说明。
|
||||
"user_id": graph_node.user_id,
|
||||
"agent_id": graph_node.agent_id,
|
||||
"session_id": getattr(graph_node, "session_id", "") or "",
|
||||
"owner": getattr(graph_node, "owner", None),
|
||||
"confidence": graph_node.confidence,
|
||||
"custom": getattr(graph_node, "custom", {}) or {},
|
||||
"source": "graph",
|
||||
"source": getattr(graph_node, "source", "") or "",
|
||||
"retrieval_source": "graph",
|
||||
"request_id": request_id,
|
||||
"elapsed_ms": elapsed,
|
||||
}
|
||||
|
||||
logger.info(f"[get] not found: memory_id={memory_id}")
|
||||
return None
|
||||
|
||||
@_ensure_internal_loop
|
||||
@@ -5081,6 +4855,7 @@ class HyMemoryClient:
|
||||
"user_id": node.user_id,
|
||||
"agent_id": node.agent_id,
|
||||
"session_id": getattr(node, "session_id", None),
|
||||
"source": getattr(node, "source", None),
|
||||
"speculate": getattr(node, "speculate", None),
|
||||
# 不透出 source_raw_memory_id:见 async_search 同处说明。
|
||||
"tags": node.tags or [],
|
||||
@@ -5152,6 +4927,9 @@ class HyMemoryClient:
|
||||
order: str,
|
||||
user_ids: Optional[List[str]] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
include_sources: Optional[List[str]] = None,
|
||||
exclude_sources: Optional[List[str]] = None,
|
||||
layers: Optional[List] = None,
|
||||
statuses: Optional[List] = None,
|
||||
with_evolution: bool = False,
|
||||
@@ -5177,6 +4955,9 @@ class HyMemoryClient:
|
||||
status_filter=effective_statuses,
|
||||
layers=layers,
|
||||
with_vectors=False,
|
||||
allow_raw_layer=bool(
|
||||
layers is not None and MemoryLayer.L1_RAW in layers
|
||||
),
|
||||
)
|
||||
|
||||
# L0 是 user-level,不应被 agent 过滤。默认列表或显式包含 L0 时,额外按
|
||||
@@ -5198,6 +4979,22 @@ class HyMemoryClient:
|
||||
])
|
||||
all_nodes.extend(hit["node"] for hit in l0_hits if hit.get("node") is not None)
|
||||
|
||||
if session_ids:
|
||||
wanted_sessions = set(session_ids)
|
||||
all_nodes = [
|
||||
node for node in all_nodes if node.session_id in wanted_sessions
|
||||
]
|
||||
if include_sources:
|
||||
wanted_sources = set(include_sources)
|
||||
all_nodes = [
|
||||
node for node in all_nodes if node.source in wanted_sources
|
||||
]
|
||||
if exclude_sources:
|
||||
blocked_sources = set(exclude_sources)
|
||||
all_nodes = [
|
||||
node for node in all_nodes if node.source not in blocked_sources
|
||||
]
|
||||
|
||||
all_nodes = self._sort_memory_nodes(all_nodes, order=order)
|
||||
total = len(all_nodes)
|
||||
page_nodes = all_nodes[offset: offset + limit]
|
||||
@@ -5261,6 +5058,9 @@ class HyMemoryClient:
|
||||
order: str,
|
||||
user_ids: Optional[List[str]] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
include_sources: Optional[List[str]] = None,
|
||||
exclude_sources: Optional[List[str]] = None,
|
||||
layers: Optional[List] = None,
|
||||
statuses: Optional[List] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
@@ -5308,6 +5108,21 @@ class HyMemoryClient:
|
||||
node for node in graph_nodes
|
||||
if getattr(node, "status", None) in effective_statuses
|
||||
]
|
||||
if session_ids:
|
||||
wanted_sessions = set(session_ids)
|
||||
graph_nodes = [
|
||||
node for node in graph_nodes if node.session_id in wanted_sessions
|
||||
]
|
||||
if include_sources:
|
||||
wanted_sources = set(include_sources)
|
||||
graph_nodes = [
|
||||
node for node in graph_nodes if node.source in wanted_sources
|
||||
]
|
||||
if exclude_sources:
|
||||
blocked_sources = set(exclude_sources)
|
||||
graph_nodes = [
|
||||
node for node in graph_nodes if node.source not in blocked_sources
|
||||
]
|
||||
|
||||
graph_nodes = self._sort_memory_nodes(graph_nodes, order=order)
|
||||
total = len(graph_nodes)
|
||||
@@ -5419,6 +5234,9 @@ class HyMemoryClient:
|
||||
branch: Optional[str] = None,
|
||||
user_ids: Optional[List[str]] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
include_sources: Optional[List[str]] = None,
|
||||
exclude_sources: Optional[List[str]] = None,
|
||||
layers: Optional[List[Any]] = None,
|
||||
statuses: Optional[List[Any]] = None,
|
||||
with_evolution: bool = False,
|
||||
@@ -5495,6 +5313,9 @@ class HyMemoryClient:
|
||||
vdb = await self._list_vdb_bucket(
|
||||
user_id=user_id, agent_id=agent_id,
|
||||
user_ids=user_ids, agent_ids=agent_ids,
|
||||
session_ids=session_ids,
|
||||
include_sources=include_sources,
|
||||
exclude_sources=exclude_sources,
|
||||
limit=limit, offset=offset, order=order,
|
||||
layers=effective_layers, statuses=effective_statuses,
|
||||
with_evolution=with_evolution,
|
||||
@@ -5502,6 +5323,9 @@ class HyMemoryClient:
|
||||
graph = await self._list_graph_bucket(
|
||||
user_id=user_id, agent_id=agent_id,
|
||||
user_ids=user_ids, agent_ids=agent_ids,
|
||||
session_ids=session_ids,
|
||||
include_sources=include_sources,
|
||||
exclude_sources=exclude_sources,
|
||||
limit=limit, offset=offset, order=order,
|
||||
layers=effective_layers, statuses=effective_statuses,
|
||||
)
|
||||
|
||||
@@ -416,6 +416,36 @@ class CodingMemoryStore:
|
||||
|
||||
self._initialized = False
|
||||
|
||||
@staticmethod
|
||||
def database_path(
|
||||
config: MemoryConfig,
|
||||
db_path: Optional[str] = None,
|
||||
variant: str = "working",
|
||||
) -> str:
|
||||
"""Return the authoritative SQLite path without creating anything."""
|
||||
if db_path is None:
|
||||
data_dir = (
|
||||
os.path.dirname(config.history.db_path)
|
||||
if config.history.db_path else "./data"
|
||||
)
|
||||
db_path = os.path.join(data_dir, "coding_memory.db")
|
||||
normalized = (
|
||||
"coding"
|
||||
if str(variant or "").strip().lower() == "coding"
|
||||
else "working"
|
||||
)
|
||||
return _variant_db_path(db_path, "_cv" if normalized == "coding" else "")
|
||||
|
||||
@classmethod
|
||||
def database_exists(
|
||||
cls,
|
||||
config: MemoryConfig,
|
||||
db_path: Optional[str] = None,
|
||||
variant: str = "working",
|
||||
) -> bool:
|
||||
"""Whether a coding namespace has an existing SQLite database."""
|
||||
return os.path.isfile(cls.database_path(config, db_path, variant))
|
||||
|
||||
async def initialize(self) -> None:
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
@@ -50,6 +50,7 @@ class CodingWriter:
|
||||
embed_service: EmbedService,
|
||||
fact_reconciler: Optional[WorkingFactReconciler] = None,
|
||||
cache: Any = None,
|
||||
variant: str = "working",
|
||||
):
|
||||
self.store = store
|
||||
self.extractor = extractor
|
||||
@@ -59,6 +60,11 @@ class CodingWriter:
|
||||
self.fact_reconciler = fact_reconciler or WorkingFactReconciler(
|
||||
store, embed_service, llm_provider
|
||||
)
|
||||
self.variant = (
|
||||
"coding"
|
||||
if str(variant or "").strip().lower() == "coding"
|
||||
else "working"
|
||||
)
|
||||
# cache 仅用于把 request_id 串给各阶段(各阶段自己打点);writer 不集中打点。
|
||||
self.cache = cache
|
||||
|
||||
@@ -217,6 +223,7 @@ class CodingWriter:
|
||||
return {
|
||||
"success": True,
|
||||
"scene": "productivity",
|
||||
"variant": self.variant,
|
||||
"request_id": request_id,
|
||||
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
|
||||
"ops": [op.to_dict() for op in ops],
|
||||
@@ -278,11 +285,11 @@ class CodingWriter:
|
||||
f"[coding-write] crosslink fact {f.fact_id} failed: {e}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _empty_response(request_id: str, t0: float) -> Dict[str, Any]:
|
||||
def _empty_response(self, request_id: str, t0: float) -> Dict[str, Any]:
|
||||
return {
|
||||
"success": True,
|
||||
"scene": "productivity",
|
||||
"variant": self.variant,
|
||||
"request_id": request_id,
|
||||
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
|
||||
"ops": [],
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Public structural contract for injected embedding services."""
|
||||
|
||||
from typing import Any, Dict, List, Protocol, Tuple, runtime_checkable
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class EmbeddingService(Protocol):
|
||||
"""Embedding service accepted by :class:`HyMemoryClient`.
|
||||
|
||||
Implementations are borrowed by the client: their lifecycle remains owned
|
||||
by the caller and ``HyMemoryClient.close()`` will not close them.
|
||||
"""
|
||||
|
||||
async def embed(self, text: str, use_cache: bool = True) -> List[float]:
|
||||
...
|
||||
|
||||
async def embed_with_meta(
|
||||
self, text: str, use_cache: bool = True
|
||||
) -> Tuple[List[float], Dict[str, Any]]:
|
||||
...
|
||||
|
||||
async def embed_batch(
|
||||
self, texts: List[str], use_cache: bool = True
|
||||
) -> List[List[float]]:
|
||||
...
|
||||
|
||||
async def embed_queued(
|
||||
self, text: str, use_cache: bool = True
|
||||
) -> List[float]:
|
||||
...
|
||||
|
||||
async def embed_queued_with_meta(
|
||||
self, text: str, use_cache: bool = True
|
||||
) -> Tuple[List[float], Dict[str, Any]]:
|
||||
...
|
||||
|
||||
def get_embedding_dims(self) -> int:
|
||||
...
|
||||
|
||||
|
||||
# Explicit alias for callers that prefer the conventional Protocol suffix.
|
||||
EmbeddingServiceProtocol = EmbeddingService
|
||||
|
||||
|
||||
class EmbeddingDimensionMismatchError(ValueError):
|
||||
"""Raised before initialization when service and store dimensions differ."""
|
||||
|
||||
|
||||
__all__ = [
|
||||
"EmbeddingService",
|
||||
"EmbeddingServiceProtocol",
|
||||
"EmbeddingDimensionMismatchError",
|
||||
]
|
||||
@@ -19,6 +19,7 @@ import logging
|
||||
from .base import WritePipeline, ReadPipeline
|
||||
from ..config import MemoryConfig
|
||||
from ..core.embed_service import EmbedService
|
||||
from ..embedding import EmbeddingService
|
||||
from ..data.vector_store import create_vector_store
|
||||
from ..data.vector_store_base import VectorStoreBase
|
||||
|
||||
@@ -50,7 +51,12 @@ class ComponentFactory:
|
||||
s2 = await factory.get_system2_writer() # System2Writer (ultra mode)
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[MemoryConfig] = None):
|
||||
def __init__(
|
||||
self,
|
||||
config: Optional[MemoryConfig] = None,
|
||||
*,
|
||||
embedding_service: Optional[EmbeddingService] = None,
|
||||
):
|
||||
self.config = config or MemoryConfig.from_env()
|
||||
|
||||
# 实例缓存(单例)
|
||||
@@ -62,7 +68,7 @@ class ComponentFactory:
|
||||
self._readers: Dict[str, ReadPipeline] = {}
|
||||
|
||||
# 共享资源
|
||||
self._shared_embed_service: Optional[EmbedService] = None
|
||||
self._shared_embed_service: Optional[EmbeddingService] = embedding_service
|
||||
self._shared_vector_store: Optional[VectorStoreBase] = None
|
||||
self._shared_vector_store_initialized: bool = False
|
||||
self._shared_graph_store = None
|
||||
@@ -72,7 +78,7 @@ class ComponentFactory:
|
||||
# 共享资源
|
||||
# ================================================================
|
||||
|
||||
def _get_shared_embed_service(self) -> EmbedService:
|
||||
def _get_shared_embed_service(self) -> EmbeddingService:
|
||||
if self._shared_embed_service is None:
|
||||
self._shared_embed_service = EmbedService(self.config)
|
||||
return self._shared_embed_service
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Public types and errors for exact memory-ID lookup."""
|
||||
|
||||
from typing import Literal, Sequence
|
||||
|
||||
|
||||
MemoryKind = Literal["chat", "experience", "fact"]
|
||||
MemoryVariant = Literal["working", "coding"]
|
||||
|
||||
|
||||
class AmbiguousMemoryIdError(LookupError):
|
||||
"""An ID exists in more than one public memory namespace."""
|
||||
|
||||
def __init__(self, memory_id: str, namespaces: Sequence[str]):
|
||||
self.memory_id = memory_id
|
||||
self.namespaces = tuple(namespaces)
|
||||
joined = ", ".join(self.namespaces)
|
||||
super().__init__(
|
||||
f"memory ID {memory_id!r} is ambiguous across namespaces: {joined}; "
|
||||
"pass kind and/or variant to disambiguate"
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["MemoryKind", "MemoryVariant", "AmbiguousMemoryIdError"]
|
||||
@@ -8,6 +8,7 @@ Uses a two-tier strategy:
|
||||
This avoids langdetect's known weakness with short Chinese text.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import List, Optional
|
||||
@@ -135,6 +136,30 @@ def detect_language(text: str) -> str:
|
||||
return "zh" if ratio > 0.1 else "en"
|
||||
|
||||
|
||||
def _spoken_text(value: str) -> str:
|
||||
"""Ignore text-block metadata when a message content is structured JSON."""
|
||||
text = value.strip()
|
||||
if text.startswith("["):
|
||||
try:
|
||||
blocks = json.loads(text)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
else:
|
||||
if (
|
||||
isinstance(blocks, list)
|
||||
and blocks
|
||||
and all(
|
||||
isinstance(block, dict)
|
||||
and isinstance(block.get("type"), str)
|
||||
and block["type"] in {"text", "input_text", "output_text"}
|
||||
and isinstance(block.get("text"), str)
|
||||
for block in blocks
|
||||
)
|
||||
):
|
||||
return "\n".join(block["text"] for block in blocks)
|
||||
return text
|
||||
|
||||
|
||||
def extract_content_for_detection(
|
||||
content: str = "",
|
||||
messages: Optional[List] = None,
|
||||
@@ -153,20 +178,57 @@ def extract_content_for_detection(
|
||||
Concatenated content text suitable for language detection.
|
||||
"""
|
||||
if content and content.strip():
|
||||
return content.strip()
|
||||
value = content.strip()
|
||||
# Writer serializes message batches to JSON before calling Extractor.
|
||||
# Detect the spoken text, not the leading ASCII keys (role/content),
|
||||
# without changing the raw evidence sent to the model.
|
||||
if value.startswith("["):
|
||||
try:
|
||||
decoded = json.loads(value)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
else:
|
||||
if (
|
||||
isinstance(decoded, list)
|
||||
and decoded
|
||||
and all(
|
||||
isinstance(item, dict)
|
||||
and isinstance(item.get("role"), str)
|
||||
and isinstance(item.get("content"), str)
|
||||
for item in decoded
|
||||
)
|
||||
):
|
||||
# In a mixed batch, the latest user turn is the clearest
|
||||
# instruction for the language; assistant/tool metadata
|
||||
# must not decide it first.
|
||||
user_text = [
|
||||
_spoken_text(item["content"])
|
||||
for item in decoded
|
||||
if item["role"] == "user" and item["content"].strip()
|
||||
]
|
||||
if user_text:
|
||||
return user_text[-1]
|
||||
spoken = "\n".join(_spoken_text(item["content"]) for item in decoded)
|
||||
if spoken.strip():
|
||||
return spoken
|
||||
return _spoken_text(value)
|
||||
|
||||
if messages:
|
||||
parts = []
|
||||
user_parts = []
|
||||
for msg in messages:
|
||||
if isinstance(msg, dict):
|
||||
c = msg.get("content", "")
|
||||
c, role = msg.get("content", ""), msg.get("role")
|
||||
elif hasattr(msg, "content"):
|
||||
c = msg.content
|
||||
c, role = msg.content, getattr(msg, "role", None)
|
||||
else:
|
||||
continue
|
||||
if c and c.strip():
|
||||
parts.append(c.strip())
|
||||
return "\n".join(parts)
|
||||
if isinstance(c, str) and c.strip():
|
||||
text = _spoken_text(c)
|
||||
parts.append(text)
|
||||
if role == "user":
|
||||
user_parts.append(text)
|
||||
return user_parts[-1] if user_parts else "\n".join(parts)
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
+13
-41
@@ -9,10 +9,13 @@
|
||||
## Architecture Principles
|
||||
|
||||
1. Memory 读写与存储 backend **只经 SDK**(config / env)。
|
||||
2. MemHub 对 memory 的原子操作 **只经 SDK**;缺接口先补 SDK。
|
||||
2. MemHub 运行时对 memory 的原子操作 **只经 SDK 公共 Client**;缺正常业务
|
||||
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
|
||||
但不得把迁移专用方法加入公共 Client。
|
||||
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
|
||||
|
||||
生产入口仍建议经 `HyMemoryBridge`(共享 Runtime、模型配置、密钥、harness/model 门禁),但 bridge **不得**再把 SDK 结果镜像成第二套 memory 正文权威。
|
||||
生产入口通过共享 Runtime 租约取得公开 `HyMemoryClient`;MemHub 只在调用前执行
|
||||
模型配置、密钥和 harness/model 门禁,不维护第二套 memory 正文权威。
|
||||
|
||||
来源授权、模型授权和宿主召回授权仍是 MemHub 管理面门禁;memory
|
||||
数据平面的权威在 SDK。
|
||||
@@ -30,46 +33,15 @@ reader: hybrid
|
||||
|
||||
## MemHub 运行时边界
|
||||
|
||||
MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtime_config` 注入):
|
||||
MemHub 通过公开 `HyMemoryClient` 配置使用可复现的本地策略:
|
||||
|
||||
- vector / graph / cache / history / coding 等 **一律 SDK config/env**;
|
||||
- 模型与 embedding 来自版本化 Model Profile;密钥进安全存储;
|
||||
- MemHub 管理库可存 harness/model/队列状态与 **`memory_id` 引用**。
|
||||
|
||||
发布包默认 SDK 行为环境变量仍集中在
|
||||
`apps/core/src/one_memory/hy_runtime_config.py`,并由下表锁定:
|
||||
|
||||
| 变量 | 值 |
|
||||
|---|---|
|
||||
| `MEMORY_MODE` | `ultra` |
|
||||
| `MEMORY_ENABLE_GRAPH` | `true` |
|
||||
| `MEMORY_GRAPH_RERANK_ENABLED` | `false` |
|
||||
| `MEMORY_GRAPH_TEMPORAL_SPINE` | `true` |
|
||||
| `MEMORY_TOPIC_SUMMARY_ENABLED` | `false` |
|
||||
| `MEMORY_EPISODE_SUMMARY_ENABLED` | `false` |
|
||||
| `MEMORY_EPISODE_RECALL_ENABLED` | `true` |
|
||||
| `MEMORY_DYNAMIC_EPISODE_ENABLED` | `true` |
|
||||
| `MEMORY_SESSION_EPISODE_ENABLED` | `false` |
|
||||
| `MEMORY_SESSION_EPISODE_CONTEXT_ENABLED` | `false` |
|
||||
| `MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED` | `true` |
|
||||
| `MEMORY_EPISODE_CAPSULE_QUERY_MIN` | `0.65` |
|
||||
| `MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN` | `0.60` |
|
||||
| `MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX` | `3` |
|
||||
| `MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT` | `3` |
|
||||
| `MEMORY_ENABLE_INTENTION` | `false` |
|
||||
| `MEMORY_ENABLE_QUERY_REWRITE` | `false` |
|
||||
| `MEMORY_FEW_SHOT_ENABLED` | `true` |
|
||||
| `MEMORY_HYBRID_V2_QUERY_EXPAND` | `1` |
|
||||
| `MEMORY_HYBRID_V2_GEO_INTENT_PRIOR` | `0.6` |
|
||||
| `MEMORY_HYBRID_V2_DIVERSITY` | `1` |
|
||||
| `MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR` | `0.55` |
|
||||
| `MEMORY_HYBRID_V2_TEMPORAL_BOOST` | `0.18` |
|
||||
| `MEMORY_CROSS_DOMAIN_CON_THRESHOLD` | `0.5` |
|
||||
| `MEMORY_READER_ENABLE_SUMMARY` | `false` |
|
||||
| `MEMORY_SUMMARY_ENABLED_IN_SYS2` | `false` |
|
||||
| `MEMORY_LOCAL_RERANK_ENABLED` | `false` |
|
||||
| `RECONCILE_ENABLED` | `true` |
|
||||
| `HY_MEMORY_READER` | `hybrid` |
|
||||
MemHub 不再通过 import-time 环境变量覆盖 SDK 的提取、reconcile、episode、
|
||||
intention、few-shot、summary 或去重策略。产品只配置模型/密钥/endpoint、本地
|
||||
backend 路径、公开 embedding service,以及每次公开 Client 调用的参数。
|
||||
|
||||
独立脚本直连 SDK 时须自配存储目录与模型端点。
|
||||
|
||||
@@ -77,7 +49,7 @@ MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtim
|
||||
|
||||
下面是独立调试示例。
|
||||
|
||||
产品代码经 `HyMemoryBridge` 获取已配置的 client(管理面门禁),memory I/O
|
||||
产品代码经 `hy_sdk_runtime.SdkClientHandle` 获取已配置的公开 client;memory I/O
|
||||
由 SDK 完成。
|
||||
|
||||
```python
|
||||
@@ -228,7 +200,8 @@ result = client.search(
|
||||
- 区分空结果、权限拒绝、不可用、超时、部分结果;
|
||||
- 未发布的 Matter 占位功能不保留表或 API;未来组织功能另立合同。
|
||||
|
||||
生产入口:`hy_memory_bridge.py`、`recall.py`、`packages/host-bridge`。
|
||||
生产入口:`hy_sdk_runtime.py`、`sdk_write_processor.py`、`recall.py` 和
|
||||
`packages/host-bridge`。
|
||||
|
||||
## 关闭客户端
|
||||
|
||||
@@ -245,8 +218,7 @@ Runtime 长期复用 client;仅在停止、配置切换或受控恢复时重
|
||||
|
||||
```text
|
||||
harness / user action (+ authz, model, queue)
|
||||
-> HyMemoryBridge
|
||||
-> Hy Memory SDK
|
||||
-> public HyMemoryClient
|
||||
-> SDK stores (config/env)
|
||||
-> optional MemHub persist memory_id / org metadata
|
||||
-> Recall Broker / UI (SDK read + MemHub authz)
|
||||
|
||||
@@ -42,12 +42,10 @@ TARGETED = (
|
||||
"tests/test_database_contention.py",
|
||||
"tests/test_main_database_fairness.py",
|
||||
"tests/test_soak_probe_effectiveness.py",
|
||||
"tests/test_evidence_reconcile_primitive.py",
|
||||
"tests/test_sqlite_semantics_gate.py",
|
||||
"tests/test_store_binding.py",
|
||||
"tests/test_store_binding_ratchet.py",
|
||||
"tests/test_main_database_entry.py",
|
||||
"tests/test_memory_processing.py",
|
||||
"tests/test_extraction_window.py",
|
||||
"tests/test_extraction_defer.py",
|
||||
"tests/test_active_memory_window.py",
|
||||
|
||||
@@ -2,11 +2,5 @@
|
||||
|
||||
import os
|
||||
|
||||
from one_memory.hy_runtime_config import configure_hy_behavior_environment
|
||||
|
||||
# HY has a few reader switches evaluated at module import time. Fix the
|
||||
# product-owned, non-secret behavior before any one_memory module imports HY.
|
||||
configure_hy_behavior_environment()
|
||||
|
||||
_version = os.environ.get("ONE_MEMORY_VERSION", "").strip()
|
||||
__version__ = _version if _version else "development"
|
||||
@@ -50,6 +50,10 @@ from one_memory.auth import local_api_guard, renderer_origins
|
||||
from one_memory.background_work import shutdown_bulk_pool, warm_bulk_pool
|
||||
from one_memory.config import Settings, load_settings
|
||||
from one_memory.context_routes import context_router, sdk_context_router
|
||||
from one_memory.conversation_processing import (
|
||||
_conversation_chunks,
|
||||
memory_task_timeout_seconds,
|
||||
)
|
||||
from one_memory.database_access import (
|
||||
MAIN_DATABASE_REPORT_SECONDS,
|
||||
close_main_databases,
|
||||
@@ -78,23 +82,30 @@ from one_memory.external_memory_access import external_access, require_external_
|
||||
from one_memory.extraction_settings import ExtractionSettings, ExtractionSettingsStore
|
||||
from one_memory.file_imports import FileImportStore, file_import_router
|
||||
from one_memory.home_summary import HomeSummary, read_home_summary
|
||||
from one_memory.hy_memory_bridge import (
|
||||
GraphWidthDecision,
|
||||
GraphWidthMismatch,
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HySdkMemoryPage,
|
||||
HySdkMemoryRecord,
|
||||
HySdkRawRecord,
|
||||
builtin_retrieval_snapshot,
|
||||
)
|
||||
from one_memory.hy_graph_admin import GraphWidthDecision, GraphWidthMismatch
|
||||
from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_READER,
|
||||
DiagnosticLogLevel,
|
||||
detailed_logging_enabled,
|
||||
diagnostic_log_level,
|
||||
set_detailed_logging,
|
||||
set_diagnostic_log_level,
|
||||
)
|
||||
from one_memory.hy_sdk_runtime import (
|
||||
SdkClientHandle,
|
||||
builtin_retrieval_snapshot,
|
||||
close_all_shared_runtimes,
|
||||
sdk_relevance_floor,
|
||||
)
|
||||
from one_memory.hy_sdk_types import (
|
||||
HySdkError,
|
||||
HySdkMemoryPage,
|
||||
HySdkMemoryRecord,
|
||||
HySdkRawPage,
|
||||
HySdkRawRecord,
|
||||
HySdkUserProfile,
|
||||
parse_sdk_search,
|
||||
)
|
||||
from one_memory.imports.access import (
|
||||
AuthorizedSourceRoot,
|
||||
SourceAccessError,
|
||||
@@ -167,11 +178,10 @@ from one_memory.memory_library import (
|
||||
read_conversation_memories,
|
||||
read_memory_library,
|
||||
)
|
||||
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
|
||||
from one_memory.memory_processing import (
|
||||
MemoryProcessingSupervisor,
|
||||
_conversation_chunks,
|
||||
memory_task_timeout_seconds,
|
||||
from one_memory.memory_notes import (
|
||||
LegacyMemoryNoteReader,
|
||||
MemoryNoteStore,
|
||||
memory_note_router,
|
||||
)
|
||||
from one_memory.memory_qa import MemoryAnswerer, MemoryAnswerError
|
||||
from one_memory.memory_queue import (
|
||||
@@ -201,7 +211,10 @@ from one_memory.model_profiles import (
|
||||
ModelProfileStore,
|
||||
ModelProfileUnavailableError,
|
||||
)
|
||||
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
|
||||
from one_memory.model_request_audit import (
|
||||
AuditedModelTransport,
|
||||
ModelRequestAuditStore,
|
||||
)
|
||||
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
|
||||
from one_memory.onboarding import OnboardingState, OnboardingStore
|
||||
from one_memory.recall import (
|
||||
@@ -225,6 +238,8 @@ from one_memory.sdk_memory_qa import (
|
||||
SdkMemoryAnswer,
|
||||
SdkMemoryAnswerer,
|
||||
)
|
||||
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
|
||||
from one_memory.sdk_write_processor import SdkWriteCoordinator
|
||||
from one_memory.secrets import (
|
||||
KeyringSecretStore,
|
||||
SecretStore,
|
||||
@@ -845,7 +860,7 @@ def create_app(
|
||||
runtime = CoreRuntime(runtime_settings) if manage_runtime else nullcontext()
|
||||
with runtime:
|
||||
database_path = runtime_settings.data_dir / "one-memory.sqlite3"
|
||||
app.state.sdk_library_bridge = None
|
||||
app.state.sdk_client_handle = None
|
||||
bootstrap_fresh_management_schema(database_path)
|
||||
# Before any store: each one below changes the library as it opens.
|
||||
try:
|
||||
@@ -891,6 +906,7 @@ def create_app(
|
||||
)
|
||||
from one_memory.memory_server.data_safety import MIN_FREE_BYTES
|
||||
from one_memory.memory_server.durable_queue import (
|
||||
COMPLETED,
|
||||
DurableWriteDrain,
|
||||
DurableWriteQueue,
|
||||
execute_job,
|
||||
@@ -968,29 +984,21 @@ def create_app(
|
||||
await asyncio.to_thread(app.state.evidence_store.release_completed_sdk_staging)
|
||||
evidence_store = app.state.evidence_store
|
||||
if app.state.memory_authority is MemoryAuthority.SDK:
|
||||
sdk_bridge = _LazySdkBridge(app)
|
||||
|
||||
async def read_sdk_memory(
|
||||
memory_id: str,
|
||||
) -> HySdkMemoryRecord | None:
|
||||
return cast(
|
||||
HySdkMemoryRecord | None,
|
||||
await sdk_bridge.get_sdk_memory(memory_id),
|
||||
)
|
||||
|
||||
sdk_handle = _sdk_client_handle(app)
|
||||
app.state.recall_target = SdkRecallTarget(
|
||||
semantic_search=lambda query: sdk_bridge.search_memories(
|
||||
semantic_search=lambda query: _search_sdk_memories(
|
||||
sdk_handle,
|
||||
query,
|
||||
user_id="local",
|
||||
agent_id="*",
|
||||
),
|
||||
get_memory=read_sdk_memory,
|
||||
get_memory=lambda memory_id: _get_sdk_memory(sdk_handle, memory_id),
|
||||
)
|
||||
else:
|
||||
app.state.recall_target = GovernedRecallTarget(
|
||||
app.state.memory_governance_store,
|
||||
filter_memories=app.state.source_read_access.memories,
|
||||
semantic_search=lambda query: _memory_processing_supervisor(
|
||||
semantic_search=lambda query: _sdk_memory_runtime(
|
||||
app
|
||||
).semantic_recall(query),
|
||||
# Read per request rather than cached: an import that lands
|
||||
@@ -1094,7 +1102,7 @@ def create_app(
|
||||
authorize_generation=lambda profile, question, grounding: (
|
||||
_authorize_answer_generation(app, profile, question, grounding)
|
||||
),
|
||||
semantic_search=lambda query: _memory_processing_supervisor(app).semantic_recall(
|
||||
semantic_search=lambda query: _sdk_memory_runtime(app).semantic_recall(
|
||||
query
|
||||
),
|
||||
resolve_workspace=lambda workspace_path: resolve_workspace_scopes(
|
||||
@@ -1116,43 +1124,32 @@ def create_app(
|
||||
store=app.state.model_profile_store,
|
||||
secrets=app.state.secret_store,
|
||||
)
|
||||
app.state.memory_processing_supervisor = MemoryProcessingSupervisor(
|
||||
tasks=app.state.memory_task_store,
|
||||
app.state.sdk_write_coordinator = SdkWriteCoordinator(
|
||||
profiles=app.state.model_profile_store,
|
||||
secrets=app.state.secret_store,
|
||||
evidence=app.state.evidence_store,
|
||||
governance=app.state.memory_governance_store,
|
||||
data_dir=runtime_settings.data_dir,
|
||||
extraction_settings=app.state.extraction_settings_store,
|
||||
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
|
||||
profile=profile,
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
http_transport_factory=lambda: AuditedModelTransport(
|
||||
app.state.model_request_audit_store,
|
||||
profile_id=profile.profile_id,
|
||||
profile_revision=profile.revision,
|
||||
provider_kind=str(profile.provider_kind),
|
||||
model=profile.model,
|
||||
inner=(
|
||||
BudgetedModelTransport(model_request_budget)
|
||||
if model_request_budget is not None
|
||||
else None
|
||||
),
|
||||
transport_factory=lambda profile: lambda: AuditedModelTransport(
|
||||
app.state.model_request_audit_store,
|
||||
profile_id=profile.profile_id,
|
||||
profile_revision=profile.revision,
|
||||
provider_kind=str(profile.provider_kind),
|
||||
model=profile.model,
|
||||
inner=(
|
||||
BudgetedModelTransport(model_request_budget)
|
||||
if model_request_budget is not None
|
||||
else None
|
||||
),
|
||||
),
|
||||
# `None` means "whatever the active profile allows". Pinned to
|
||||
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
|
||||
# package, and left pinned after it: a hardcoded 1 is rule 9's
|
||||
# exact anti-pattern (a small number baked into code that
|
||||
# cannot be tuned per machine). Measured before removing — at
|
||||
# profile concurrency 4, recall latency is unchanged while
|
||||
# precipitation goes 4x.
|
||||
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
service_levels=app.state.service_levels,
|
||||
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
|
||||
sdk_authority=lambda: app.state.memory_authority is MemoryAuthority.SDK,
|
||||
sedimentation_enabled=lambda: _config_store(
|
||||
app
|
||||
).sedimentation_enabled(),
|
||||
)
|
||||
app.state.sdk_memory_runtime = SdkMemoryRuntime(
|
||||
coordinator=app.state.sdk_write_coordinator,
|
||||
read_handle=_sdk_client_handle(app),
|
||||
write_queue=memory_write_queue,
|
||||
extraction_settings=app.state.extraction_settings_store,
|
||||
data_dir=runtime_settings.data_dir,
|
||||
)
|
||||
# C2 owns the single production budget (handle 5s + reserve 250ms,
|
||||
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
|
||||
@@ -1162,34 +1159,26 @@ def create_app(
|
||||
drain_budget = production_drain_budget()
|
||||
|
||||
async def reconcile_unknown_writes() -> None:
|
||||
if app.state.memory_authority is MemoryAuthority.SDK:
|
||||
# Unknown SDK writes must be reconciled by SDK request ID.
|
||||
# Until that direct path is available, retain them as
|
||||
# unknown rather than re-running a possibly successful add.
|
||||
return
|
||||
await app.state.memory_processing_supervisor.reconcile_unknown_outputs(
|
||||
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
|
||||
)
|
||||
unknown = await asyncio.to_thread(
|
||||
memory_write_queue.unknown_jobs,
|
||||
limit=128,
|
||||
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
|
||||
)
|
||||
for job in unknown:
|
||||
receipt = (
|
||||
await app.state.sdk_write_coordinator.inquire_write_outcome(
|
||||
job.request_id
|
||||
)
|
||||
)
|
||||
if receipt is None:
|
||||
continue
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.resolve_unknown_from_linked_output,
|
||||
memory_write_queue.settle,
|
||||
job.job_id,
|
||||
state=COMPLETED,
|
||||
sdk_raw_ids=receipt.raw_ids,
|
||||
sdk_memory_ids=receipt.memory_ids,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=32,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=128,
|
||||
)
|
||||
|
||||
async def admit_hy_invocation(
|
||||
source_id: str,
|
||||
@@ -1211,7 +1200,7 @@ def create_app(
|
||||
execute=lambda job_id: execute_job(
|
||||
job_id,
|
||||
queue=memory_write_queue,
|
||||
supervisor=app.state.memory_processing_supervisor,
|
||||
supervisor=app.state.sdk_write_coordinator,
|
||||
evidence=app.state.evidence_store,
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
admit_hy_invocation=admit_hy_invocation,
|
||||
@@ -1219,7 +1208,7 @@ def create_app(
|
||||
execute_owned=lambda job_id, owner: execute_job(
|
||||
job_id,
|
||||
queue=memory_write_queue,
|
||||
supervisor=app.state.memory_processing_supervisor,
|
||||
supervisor=app.state.sdk_write_coordinator,
|
||||
evidence=app.state.evidence_store,
|
||||
owner=owner,
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
@@ -1293,7 +1282,14 @@ def create_app(
|
||||
|
||||
app.state.memory_note_store = MemoryNoteStore(
|
||||
app.state.evidence_store,
|
||||
app.state.memory_processing_supervisor,
|
||||
(
|
||||
LegacyMemoryNoteReader(
|
||||
app.state.memory_task_store,
|
||||
app.state.memory_governance_store,
|
||||
)
|
||||
if app.state.memory_authority is MemoryAuthority.LEGACY
|
||||
else None
|
||||
),
|
||||
app.state.memory_write_intake,
|
||||
sdk_authority=app.state.memory_authority is MemoryAuthority.SDK,
|
||||
write_queue=memory_write_queue,
|
||||
@@ -1435,18 +1431,9 @@ def create_app(
|
||||
# starts its threads wherever the first work arrives from, and
|
||||
# from a busy loop that wait was 2.5 seconds.
|
||||
warm_bulk_pool()
|
||||
if app.state.memory_authority is MemoryAuthority.LEGACY:
|
||||
_start_backfill(app)
|
||||
_start_visibility_recompute(app)
|
||||
app.state.deletion_sync = watched_task(
|
||||
asyncio.get_running_loop(),
|
||||
lambda: _memory_processing_supervisor(app).keep_deletions_in_sync(),
|
||||
name="one-memory-deletion-sync",
|
||||
)
|
||||
else:
|
||||
app.state.memory_backfill = None
|
||||
app.state.memory_visibility_task = None
|
||||
app.state.deletion_sync = None
|
||||
app.state.memory_backfill = None
|
||||
app.state.memory_visibility_task = None
|
||||
app.state.deletion_sync = None
|
||||
# Both times the loop was blocked here it was found from outside,
|
||||
# by hand, days later. This says so at the time, and says what was
|
||||
# holding it.
|
||||
@@ -1550,16 +1537,12 @@ def create_app(
|
||||
app.state.memory_write_drain.stop()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await app.state.memory_write_drain_task
|
||||
await app.state.memory_processing_supervisor.shutdown()
|
||||
if app.state.sdk_library_bridge is not None:
|
||||
await app.state.sdk_library_bridge.close()
|
||||
app.state.sdk_library_bridge = None
|
||||
# HY clients share two process-owned loop threads. Close them
|
||||
# after every bridge is down, while HY's SQLite executor is
|
||||
# still available; interpreter atexit is too late for that.
|
||||
from one_memory.hy_memory_bridge import close_shared_hy_runtimes
|
||||
|
||||
await asyncio.to_thread(close_shared_hy_runtimes)
|
||||
await app.state.sdk_memory_runtime.shutdown()
|
||||
await app.state.sdk_write_coordinator.close()
|
||||
if app.state.sdk_client_handle is not None:
|
||||
await app.state.sdk_client_handle.close()
|
||||
app.state.sdk_client_handle = None
|
||||
close_all_shared_runtimes()
|
||||
# Last: what has just been shut down was using it.
|
||||
shutdown_bulk_pool()
|
||||
log_main_database_stats("shutdown")
|
||||
@@ -1622,12 +1605,12 @@ def create_app(
|
||||
"/api/v1/context",
|
||||
"/api/v1/boundary",
|
||||
"/api/v1/home-summary",
|
||||
"/api/v1/memory-tasks",
|
||||
"/api/v1/memory-processing",
|
||||
)
|
||||
if getattr(
|
||||
app.state, "memory_authority", MemoryAuthority.LEGACY
|
||||
) is MemoryAuthority.SDK and request.url.path.startswith(legacy_paths):
|
||||
) is MemoryAuthority.SDK and request.url.path.startswith(
|
||||
legacy_paths
|
||||
):
|
||||
return JSONResponse(
|
||||
status_code=409,
|
||||
content={"detail": "sdk_memory_authority_required"},
|
||||
@@ -1820,7 +1803,7 @@ def create_app(
|
||||
) -> dict[str, object]:
|
||||
"""Semantic search IDs for the Reader process. Reader-only."""
|
||||
try:
|
||||
hits = await _memory_processing_supervisor(app).semantic_search(query, limit=limit)
|
||||
hits = await _sdk_memory_runtime(app).semantic_search(query, limit=limit)
|
||||
except Exception:
|
||||
return {"ids": []}
|
||||
return {"ids": [memory_id for memory_id, _score in hits]}
|
||||
@@ -1850,7 +1833,7 @@ def create_app(
|
||||
) -> dict[str, object]:
|
||||
"""HY graph overlay for the Reader process. Reader-only."""
|
||||
try:
|
||||
snapshot = await _memory_processing_supervisor(app).graph_snapshot(limit=limit)
|
||||
snapshot = await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
|
||||
except Exception:
|
||||
return {"consulted": False, "nodes": [], "edges": []}
|
||||
return snapshot.model_dump(mode="json")
|
||||
@@ -1861,7 +1844,7 @@ def create_app(
|
||||
error = cast(str | None, getattr(app.state, "memory_backfill_error", None))
|
||||
status = _backfill_state(backfill, error)
|
||||
governance = getattr(app.state, "memory_governance_store", None)
|
||||
supervisor = getattr(app.state, "memory_processing_supervisor", None)
|
||||
supervisor = getattr(app.state, "sdk_memory_runtime", None)
|
||||
if supervisor is not None:
|
||||
accepted = supervisor.accepted_backfill_status()
|
||||
if accepted is not None:
|
||||
@@ -3092,15 +3075,18 @@ def create_app(
|
||||
if app.state.memory_authority is not MemoryAuthority.SDK:
|
||||
raise HTTPException(409, "legacy_memory_authority_required")
|
||||
try:
|
||||
page = await _sdk_library_bridge(app).list_sdk_memories(
|
||||
client = await _sdk_client_handle(app).client()
|
||||
raw = await asyncio.to_thread(
|
||||
client.list_memories,
|
||||
user_id="local",
|
||||
statuses=("active",),
|
||||
statuses=["active"],
|
||||
limit=1,
|
||||
offset=0,
|
||||
order="desc",
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(503, str(error)) from None
|
||||
page = HySdkMemoryPage.from_sdk(raw, limit=1, offset=0)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_list_failed") from None
|
||||
return SdkHomeSummary(
|
||||
memory_count=page.total,
|
||||
latest=page.memories[0] if page.memories else None,
|
||||
@@ -3159,14 +3145,17 @@ def create_app(
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
) -> HySdkMemoryPage:
|
||||
try:
|
||||
return await _sdk_library_bridge(app).list_sdk_memories(
|
||||
client = await _sdk_client_handle(app).client()
|
||||
raw = await asyncio.to_thread(
|
||||
client.list_memories,
|
||||
user_id="local",
|
||||
statuses=tuple(statuses),
|
||||
statuses=list(statuses),
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
return HySdkMemoryPage.from_sdk(raw, limit=limit, offset=offset)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_list_failed") from None
|
||||
|
||||
@app.get(
|
||||
"/api/v2/user-profile",
|
||||
@@ -3175,10 +3164,12 @@ def create_app(
|
||||
)
|
||||
async def sdk_user_profile() -> SdkUserProfileResponse:
|
||||
try:
|
||||
value = await _sdk_library_bridge(app).get_user_profile(user_id="local")
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
return SdkUserProfileResponse.model_validate(value)
|
||||
client = await _sdk_client_handle(app).client()
|
||||
value = await asyncio.to_thread(client.get_user_profile, user_id="local")
|
||||
profile = HySdkUserProfile.from_sdk(value)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_profile_read_failed") from None
|
||||
return SdkUserProfileResponse.model_validate(profile.model_dump())
|
||||
|
||||
@app.get(
|
||||
"/api/v2/memories/{memory_id}",
|
||||
@@ -3187,9 +3178,9 @@ def create_app(
|
||||
)
|
||||
async def get_sdk_memory(memory_id: str) -> HySdkMemoryRecord:
|
||||
try:
|
||||
memory = await _sdk_library_bridge(app).get_sdk_memory(memory_id)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
memory = await _get_sdk_memory(_sdk_client_handle(app), memory_id)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_get_failed") from None
|
||||
if memory is None:
|
||||
raise HTTPException(status_code=404, detail="memory_not_found")
|
||||
return memory
|
||||
@@ -3203,12 +3194,15 @@ def create_app(
|
||||
memory_id: str,
|
||||
request: SdkMemoryUpdateRequest,
|
||||
) -> HySdkMemoryRecord:
|
||||
bridge = _sdk_library_bridge(app)
|
||||
try:
|
||||
await bridge.update(memory_id, request.content)
|
||||
memory = await bridge.get_sdk_memory(memory_id)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
client = await app.state.sdk_write_coordinator.client()
|
||||
result = await asyncio.to_thread(client.update, memory_id, request.content)
|
||||
if not isinstance(result, dict) or result.get("success") is not True:
|
||||
raise HySdkError("hy_memory_update_rejected")
|
||||
raw = await asyncio.to_thread(client.get, memory_id)
|
||||
memory = None if raw is None else HySdkMemoryRecord.from_sdk(raw)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_update_failed") from None
|
||||
if memory is None:
|
||||
raise HTTPException(status_code=404, detail="memory_not_found")
|
||||
return memory
|
||||
@@ -3220,9 +3214,17 @@ def create_app(
|
||||
)
|
||||
async def delete_sdk_memory(memory_id: str) -> Response:
|
||||
try:
|
||||
await _sdk_library_bridge(app).delete(memory_id)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
client = await app.state.sdk_write_coordinator.client()
|
||||
result = await asyncio.to_thread(client.delete, memory_id)
|
||||
if not isinstance(result, dict):
|
||||
raise HySdkError("hy_memory_delete_rejected")
|
||||
error_code = result.get("error_code")
|
||||
if error_code or result.get("partial") is True:
|
||||
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
|
||||
if result.get("success") is not True and result.get("deleted_count") != 0:
|
||||
raise HySdkError("hy_memory_delete_rejected")
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_delete_failed") from None
|
||||
return Response(status_code=204)
|
||||
|
||||
@app.get(
|
||||
@@ -3236,15 +3238,19 @@ def create_app(
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
) -> SdkRawPage:
|
||||
try:
|
||||
items, total = await _sdk_library_bridge(app).list_sdk_raw(
|
||||
client = await _sdk_client_handle(app).client()
|
||||
value = await asyncio.to_thread(
|
||||
client.list_memories,
|
||||
user_id="local",
|
||||
source=source,
|
||||
layers=["l1_raw"],
|
||||
include_sources=[source] if source else None,
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
return SdkRawPage(items=items, total=total, offset=offset, limit=limit)
|
||||
page = HySdkRawPage.from_memory_list(value, limit=limit, offset=offset)
|
||||
except Exception as error:
|
||||
raise _sdk_http_error(error, "hy_memory_raw_list_failed") from None
|
||||
return SdkRawPage(items=page.raw, total=page.total, offset=offset, limit=limit)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/memories",
|
||||
@@ -3777,9 +3783,9 @@ def create_app(
|
||||
hy_graph = (
|
||||
None
|
||||
if revoked
|
||||
else await _memory_processing_supervisor(app).graph_snapshot(limit=limit)
|
||||
else await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
|
||||
)
|
||||
except HyMemoryBridgeError:
|
||||
except HySdkError:
|
||||
hy_graph = None
|
||||
|
||||
revoked_now = await asyncio.to_thread(_read_access(app).has_revoked_sources)
|
||||
@@ -3856,8 +3862,8 @@ def create_app(
|
||||
if await asyncio.to_thread(_read_access(app).has_revoked_sources):
|
||||
raise HTTPException(status_code=403, detail="memory_source_revoked")
|
||||
try:
|
||||
return await _memory_processing_supervisor(app).digest()
|
||||
except HyMemoryBridgeError as error:
|
||||
return await _sdk_memory_runtime(app).digest()
|
||||
except HySdkError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
|
||||
@app.get(
|
||||
@@ -3881,17 +3887,26 @@ def create_app(
|
||||
# Two numbers, not one: 675 as a single total reads as "stuck", when
|
||||
# what is true is that the three from this morning are nearly done
|
||||
# and the rest is catch-up (R-02).
|
||||
supervisor = app.state.memory_processing_supervisor
|
||||
supervisor = app.state.sdk_memory_runtime
|
||||
depths = supervisor.queue_depths()
|
||||
active_queue, in_flight_count = supervisor.active_depths()
|
||||
planned, current = supervisor.manual_batch_progress()
|
||||
sdk_counts = (
|
||||
supervisor.status_counts()
|
||||
if app.state.memory_authority is MemoryAuthority.SDK
|
||||
else {}
|
||||
)
|
||||
# `running` counts rows the store still marks running, which includes
|
||||
# work a worker has already given up on. Reporting that as "what is
|
||||
# running" is how a stalled queue looks busy, so the strand is given
|
||||
# its own number.
|
||||
running_rows = counts.get(MemoryTaskStatus.RUNNING, 0)
|
||||
return MemoryProcessingSummary(
|
||||
pending=counts.get(MemoryTaskStatus.PENDING, 0),
|
||||
pending=(
|
||||
sdk_counts.get("pending", 0)
|
||||
if sdk_counts
|
||||
else counts.get(MemoryTaskStatus.PENDING, 0)
|
||||
),
|
||||
fresh_pending=depths.fresh,
|
||||
backlog_pending=depths.backlog,
|
||||
outside_window_pending=counts.get(
|
||||
@@ -3899,20 +3914,28 @@ def create_app(
|
||||
0,
|
||||
),
|
||||
recent_completions=depths.recent_completions,
|
||||
running=running_rows,
|
||||
stranded_running=max(0, running_rows - in_flight_count),
|
||||
running=sdk_counts.get("running", running_rows),
|
||||
stranded_running=(
|
||||
0 if sdk_counts else max(0, running_rows - in_flight_count)
|
||||
),
|
||||
unexecuted_policy_failures=store.unexecuted_policy_failures(),
|
||||
waiting_for_model=counts.get(MemoryTaskStatus.WAITING_FOR_MODEL, 0),
|
||||
completed=counts.get(MemoryTaskStatus.COMPLETED, 0),
|
||||
failed=counts.get(MemoryTaskStatus.FAILED, 0),
|
||||
unknown=counts.get(MemoryTaskStatus.UNKNOWN, 0),
|
||||
waiting_for_model=sdk_counts.get(
|
||||
"waiting_for_model",
|
||||
counts.get(MemoryTaskStatus.WAITING_FOR_MODEL, 0),
|
||||
),
|
||||
completed=sdk_counts.get(
|
||||
"completed",
|
||||
counts.get(MemoryTaskStatus.COMPLETED, 0),
|
||||
),
|
||||
failed=sdk_counts.get("failed", counts.get(MemoryTaskStatus.FAILED, 0)),
|
||||
unknown=sdk_counts.get("unknown", counts.get(MemoryTaskStatus.UNKNOWN, 0)),
|
||||
unknown_reasons=store.unknown_error_counts(),
|
||||
blocked_stages=tuple(
|
||||
MemoryProcessingBlock(kind=kind, reason=reason, count=count)
|
||||
for kind, reason, count in store.blocked_stages()[:6]
|
||||
),
|
||||
seconds_per_completion=store.seconds_per_completion(),
|
||||
extraction_window_preset=supervisor.extraction_window_settings().window_preset,
|
||||
extraction_window_preset=app.state.extraction_settings_store.get().window_preset,
|
||||
manual_extract_batches_planned=planned,
|
||||
manual_extract_batches_current=current,
|
||||
active_queue=active_queue,
|
||||
@@ -3930,15 +3953,17 @@ def create_app(
|
||||
) -> RetryUnknownMemoryTasksReceipt:
|
||||
"""Retry unknown writes only behind the user's explicit confirmation."""
|
||||
del request
|
||||
reconciled = await _memory_processing_supervisor(app).reconcile_unknown_outputs()
|
||||
reconciled = await _sdk_memory_runtime(app).reconcile_unknown_outputs()
|
||||
from one_memory.memory_server.durable_queue import DurableWriteQueue
|
||||
|
||||
queue = DurableWriteQueue(app.state.memory_write_queue_path)
|
||||
unknown = queue.unknown_jobs(limit=2048)
|
||||
for job in unknown:
|
||||
queue.resolve_unknown_from_linked_output(job.job_id)
|
||||
if app.state.memory_authority is MemoryAuthority.LEGACY:
|
||||
for job in unknown:
|
||||
queue.resolve_unknown_from_linked_output(job.job_id)
|
||||
requeued = sum(queue.retry_unknown_confirmed(job.job_id) for job in unknown)
|
||||
requeued += app.state.memory_task_store.retry_unknown_confirmed()
|
||||
if app.state.memory_authority is MemoryAuthority.LEGACY:
|
||||
requeued += app.state.memory_task_store.retry_unknown_confirmed()
|
||||
return RetryUnknownMemoryTasksReceipt(reconciled=reconciled, requeued=requeued)
|
||||
|
||||
@app.get(
|
||||
@@ -3949,8 +3974,8 @@ def create_app(
|
||||
async def get_graph_width() -> GraphWidthStatus:
|
||||
"""Report the mismatch. Never writes: this is polled by the app."""
|
||||
try:
|
||||
mismatch = await _memory_processing_supervisor(app).graph_width_mismatch()
|
||||
except HyMemoryBridgeError as error:
|
||||
mismatch = await _sdk_memory_runtime(app).graph_width_mismatch()
|
||||
except HySdkError as error:
|
||||
raise HTTPException(status_code=503, detail=str(error)) from None
|
||||
if mismatch is None:
|
||||
return GraphWidthStatus()
|
||||
@@ -3973,10 +3998,10 @@ def create_app(
|
||||
decision is rejected by validation rather than assumed, because the
|
||||
assumption that cost a live graph once was made in code, not by a user.
|
||||
"""
|
||||
supervisor = _memory_processing_supervisor(app)
|
||||
supervisor = _sdk_memory_runtime(app)
|
||||
try:
|
||||
outcome = await supervisor.decide_graph_width(request.decision)
|
||||
except HyMemoryBridgeError as error:
|
||||
except HySdkError as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except OSError as error:
|
||||
# A failed export left nothing behind, but the user must be told
|
||||
@@ -3994,7 +4019,7 @@ def create_app(
|
||||
dependencies=protected,
|
||||
)
|
||||
def get_extraction_settings() -> ExtractionSettings:
|
||||
return _memory_processing_supervisor(app).extraction_window_settings()
|
||||
return app.state.extraction_settings_store.get()
|
||||
|
||||
@app.put(
|
||||
"/api/v1/extraction-settings",
|
||||
@@ -4002,7 +4027,7 @@ def create_app(
|
||||
dependencies=protected,
|
||||
)
|
||||
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
|
||||
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
|
||||
return app.state.extraction_settings_store.set_window_preset(request.window_preset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sedimentation",
|
||||
@@ -4052,7 +4077,7 @@ def create_app(
|
||||
dependencies=protected,
|
||||
)
|
||||
def manual_extract(request: ManualExtractRequest) -> ManualExtractReceipt:
|
||||
supervisor = _memory_processing_supervisor(app)
|
||||
supervisor = _sdk_memory_runtime(app)
|
||||
task_ids: tuple[str, ...] | None = None
|
||||
if request.evidence_ids:
|
||||
resolved: list[str] = []
|
||||
@@ -4170,11 +4195,11 @@ def create_app(
|
||||
"""
|
||||
store = _memory_store(app)
|
||||
try:
|
||||
await _memory_processing_supervisor(app).update_hy_memory(
|
||||
await _sdk_memory_runtime(app).update_hy_memory(
|
||||
memory.memory_id,
|
||||
memory.content,
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
except HySdkError as error:
|
||||
return store.fail_index_sync(
|
||||
memory.memory_id,
|
||||
user_id="local",
|
||||
@@ -4213,7 +4238,7 @@ def create_app(
|
||||
if action == "delete":
|
||||
deletion = _memory_store(app).deletion(memory_id, user_id="local")
|
||||
if deletion is None or deletion.status is not DeletionStatus.COMPLETED:
|
||||
await _memory_processing_supervisor(app).finish_deletion(memory_id)
|
||||
await _sdk_memory_runtime(app).finish_deletion(memory_id)
|
||||
return memory
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail="memory_not_found") from None
|
||||
@@ -4378,7 +4403,7 @@ async def _request_answerer(
|
||||
|
||||
def _sdk_request_answerer(app: FastAPI) -> SdkMemoryAnswerer:
|
||||
return SdkMemoryAnswerer(
|
||||
bridge=_sdk_library_bridge(app),
|
||||
client_handle=_sdk_client_handle(app),
|
||||
profiles=_model_profile_store(app),
|
||||
secrets=_secret_store(app),
|
||||
authorize_generation=lambda profile, question, grounding: _authorize_sdk_answer_generation(
|
||||
@@ -4403,9 +4428,9 @@ async def _authorize_sdk_answer_generation(
|
||||
identity = await asyncio.to_thread(_model_profile_store(app).active_identity)
|
||||
if identity != (profile.profile_id, profile.revision):
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
bridge = _sdk_library_bridge(app)
|
||||
handle = _sdk_client_handle(app)
|
||||
for memory in grounding:
|
||||
current = await bridge.get_sdk_memory(memory.memory_id)
|
||||
current = await _get_sdk_memory(handle, memory.memory_id)
|
||||
if current is None:
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
projected = SdkAskMemory(
|
||||
@@ -4488,20 +4513,6 @@ class _LazyASGIApp:
|
||||
app = _LazyASGIApp()
|
||||
|
||||
|
||||
class _LazySdkBridge:
|
||||
"""Resolve the active model only when an SDK operation is actually used."""
|
||||
|
||||
def __init__(self, app: FastAPI) -> None:
|
||||
self.app = app
|
||||
|
||||
def __getattr__(self, name: str) -> Callable[..., Awaitable[Any]]:
|
||||
async def call(*args: Any, **kwargs: Any) -> Any:
|
||||
method = getattr(_sdk_library_bridge(self.app), name)
|
||||
return await method(*args, **kwargs)
|
||||
|
||||
return call
|
||||
|
||||
|
||||
def _sync_refused(code: str) -> HTTPException:
|
||||
"""A sync refusal that says which stage refused.
|
||||
|
||||
@@ -5200,7 +5211,7 @@ def _resume_sedimentation(app: FastAPI) -> None:
|
||||
return
|
||||
app.state.sedimentation_resume = watched_task(
|
||||
asyncio.get_running_loop(),
|
||||
lambda: _memory_processing_supervisor(app).resume_allowed(),
|
||||
lambda: _sdk_memory_runtime(app).resume_allowed(),
|
||||
component="memory.sedimentation",
|
||||
name="one-memory-sedimentation-resume",
|
||||
)
|
||||
@@ -5368,9 +5379,8 @@ async def _run_backfill(app: FastAPI) -> None:
|
||||
|
||||
|
||||
def _evidence_excerpt(app: FastAPI, record: EvidenceRecord) -> EvidenceExcerpt:
|
||||
supervisor = _memory_processing_supervisor(app)
|
||||
task = _memory_task_store(app).task_for_evidence(record.evidence_id)
|
||||
window = extraction_window_for(supervisor.extraction_window_settings().window_preset)
|
||||
window = extraction_window_for(app.state.extraction_settings_store.get().window_preset)
|
||||
return EvidenceExcerpt(
|
||||
evidence_id=record.evidence_id,
|
||||
available=True,
|
||||
@@ -5390,23 +5400,62 @@ def _evidence_excerpt(app: FastAPI, record: EvidenceRecord) -> EvidenceExcerpt:
|
||||
)
|
||||
|
||||
|
||||
def _memory_processing_supervisor(app: FastAPI) -> MemoryProcessingSupervisor:
|
||||
def _sdk_memory_runtime(app: FastAPI) -> SdkMemoryRuntime:
|
||||
return cast(
|
||||
MemoryProcessingSupervisor,
|
||||
app.state.memory_processing_supervisor,
|
||||
SdkMemoryRuntime,
|
||||
app.state.sdk_memory_runtime,
|
||||
)
|
||||
|
||||
|
||||
def _sdk_library_bridge(app: FastAPI) -> HyMemoryBridge:
|
||||
bridge = cast(HyMemoryBridge | None, app.state.sdk_library_bridge)
|
||||
if bridge is None:
|
||||
bridge = HyMemoryBridge(
|
||||
def _sdk_client_handle(app: FastAPI) -> SdkClientHandle:
|
||||
handle = cast(SdkClientHandle | None, app.state.sdk_client_handle)
|
||||
if handle is None:
|
||||
handle = SdkClientHandle(
|
||||
profile=builtin_retrieval_snapshot(),
|
||||
api_key=None,
|
||||
data_dir=cast(Settings, app.state.settings).data_dir,
|
||||
)
|
||||
app.state.sdk_library_bridge = bridge
|
||||
return bridge
|
||||
app.state.sdk_client_handle = handle
|
||||
return handle
|
||||
|
||||
|
||||
async def _get_sdk_memory(
|
||||
handle: SdkClientHandle,
|
||||
memory_id: str,
|
||||
) -> HySdkMemoryRecord | None:
|
||||
client = await handle.client()
|
||||
raw = await asyncio.to_thread(client.get, memory_id)
|
||||
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
|
||||
|
||||
|
||||
async def _search_sdk_memories(
|
||||
handle: SdkClientHandle,
|
||||
query: str,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
limit: int = 12,
|
||||
) -> tuple[object, ...]:
|
||||
client = await handle.client()
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
query,
|
||||
user_ids=[user_id],
|
||||
agent_ids=[] if agent_id == "*" else [agent_id],
|
||||
limit=limit,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=limit,
|
||||
intention_limit=limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
return parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
|
||||
|
||||
def _sdk_http_error(error: Exception, fallback: str) -> HTTPException:
|
||||
detail = error.code if isinstance(error, HySdkError) else fallback
|
||||
return HTTPException(status_code=503, detail=detail)
|
||||
|
||||
|
||||
def _evidence_store(app: FastAPI) -> EvidenceStore:
|
||||
|
||||
@@ -421,8 +421,9 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
|
||||
return 4
|
||||
|
||||
from one_memory.database_access import close_main_databases
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridge
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE, HY_MEMORY_VERSION
|
||||
from one_memory.hy_sdk_migration import HySdkMigration
|
||||
from one_memory.hy_sdk_runtime import acquire_client
|
||||
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
|
||||
from one_memory.memory_server.migrations import MIGRATIONS
|
||||
from one_memory.model_profiles import ModelProfileStore
|
||||
@@ -430,7 +431,12 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
|
||||
MigrationBundleRejected,
|
||||
inspect_sdk_migration_bundle,
|
||||
)
|
||||
from one_memory.sdk_migration_runner import BackfillResult, run_sdk_data_plane_backfill
|
||||
from one_memory.sdk_migration_runner import (
|
||||
BackfillResult,
|
||||
prepare_sdk_migration_staging,
|
||||
run_sdk_data_plane_backfill,
|
||||
sdk_migration_staging,
|
||||
)
|
||||
from one_memory.secrets import KeyringSecretStore, SecretStoreUnavailableError
|
||||
|
||||
database = settings.data_dir / "one-memory.sqlite3"
|
||||
@@ -481,17 +487,32 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
|
||||
separators=(",", ":"),
|
||||
)
|
||||
backend_fingerprint = hashlib.sha256(backend_material.encode()).hexdigest()
|
||||
staging = sdk_migration_staging(
|
||||
data_dir=settings.data_dir,
|
||||
migration_id=migration_id,
|
||||
namespace=HY_MEMORY_NAMESPACE,
|
||||
)
|
||||
try:
|
||||
staging_data_dir = prepare_sdk_migration_staging(
|
||||
paths=staging,
|
||||
database_path=database,
|
||||
)
|
||||
except Exception:
|
||||
code = "sdk_migration_staging_failed"
|
||||
staging_data_dir = None
|
||||
|
||||
async def run_backfill() -> BackfillResult:
|
||||
bridge = HyMemoryBridge(
|
||||
profile=profile,
|
||||
assert staging_data_dir is not None
|
||||
lease = await asyncio.to_thread(
|
||||
acquire_client,
|
||||
profile,
|
||||
api_key=secret,
|
||||
data_dir=settings.data_dir,
|
||||
data_dir=staging_data_dir,
|
||||
)
|
||||
try:
|
||||
return await run_sdk_data_plane_backfill(
|
||||
database_path=database,
|
||||
bridge=bridge,
|
||||
client=HySdkMigration(lease.client),
|
||||
migration_id=migration_id,
|
||||
bundle_id=bundle_id,
|
||||
source_runtime_build=str(manifest.get("runtime_build") or ""),
|
||||
@@ -501,13 +522,21 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
|
||||
backend_fingerprint=backend_fingerprint,
|
||||
)
|
||||
finally:
|
||||
await bridge.close()
|
||||
await asyncio.to_thread(lease.close)
|
||||
|
||||
try:
|
||||
result = asyncio.run(run_backfill())
|
||||
result = (
|
||||
asyncio.run(run_backfill())
|
||||
if staging_data_dir is not None
|
||||
else None
|
||||
)
|
||||
except Exception:
|
||||
code = "sdk_migration_backfill_failed"
|
||||
else:
|
||||
if result is None:
|
||||
document = {"schema_version": 1, "state": "failed", "code": code}
|
||||
_write_restore(stream, as_json, document, f"SDK backfill 失败:{code}")
|
||||
return 1
|
||||
document = {
|
||||
"schema_version": 1,
|
||||
"state": result.state,
|
||||
@@ -583,20 +612,39 @@ def sdk_cutover(options: Sequence[str], stream: TextIO) -> int:
|
||||
_write_restore(stream, as_json, document, "预览通过;加 --yes 后提交 SDK authority。")
|
||||
return 4
|
||||
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_VERSION
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE, HY_MEMORY_VERSION
|
||||
from one_memory.sdk_data_plane_migration import DataPlaneMigrationError
|
||||
from one_memory.sdk_migration_bundle import MigrationBundleRejected
|
||||
from one_memory.sdk_migration_runner import commit_reconciled_cutover
|
||||
from one_memory.sdk_migration_runner import (
|
||||
activate_sdk_migration_staging,
|
||||
commit_reconciled_cutover,
|
||||
resume_legacy_cleanup,
|
||||
sdk_migration_staging,
|
||||
)
|
||||
|
||||
try:
|
||||
marker = commit_reconciled_cutover(
|
||||
database_path = settings.data_dir / "one-memory.sqlite3"
|
||||
staging = sdk_migration_staging(
|
||||
data_dir=settings.data_dir,
|
||||
database_path=settings.data_dir / "one-memory.sqlite3",
|
||||
migration_id=migration_id,
|
||||
bundle_id=bundle_id,
|
||||
sdk_version=HY_MEMORY_VERSION,
|
||||
namespace=HY_MEMORY_NAMESPACE,
|
||||
)
|
||||
except (DataPlaneMigrationError, MigrationBundleRejected) as error:
|
||||
marker = activate_sdk_migration_staging(
|
||||
paths=staging,
|
||||
database_path=database_path,
|
||||
commit=lambda: commit_reconciled_cutover(
|
||||
data_dir=settings.data_dir,
|
||||
database_path=database_path,
|
||||
migration_id=migration_id,
|
||||
bundle_id=bundle_id,
|
||||
sdk_version=HY_MEMORY_VERSION,
|
||||
),
|
||||
)
|
||||
resume_legacy_cleanup(
|
||||
data_dir=settings.data_dir,
|
||||
database_path=database_path,
|
||||
)
|
||||
except (DataPlaneMigrationError, MigrationBundleRejected, OSError) as error:
|
||||
code = getattr(error, "code", str(error))
|
||||
document = {
|
||||
"schema_version": 1,
|
||||
@@ -638,10 +686,12 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
|
||||
import sqlite3
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
|
||||
from one_memory.sdk_data_plane_migration import (
|
||||
MemoryAuthority,
|
||||
SdkDataPlaneMigrationStore,
|
||||
)
|
||||
from one_memory.sdk_migration_runner import sdk_migration_staging
|
||||
from one_memory.target_schema import legacy_tables_present
|
||||
|
||||
migration_store = SdkDataPlaneMigrationStore(database_path)
|
||||
@@ -651,10 +701,25 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
|
||||
generation = hashlib.sha256(f"{stat.st_size}:{stat.st_mtime_ns}".encode()).hexdigest()[:16]
|
||||
if authority is MemoryAuthority.SDK:
|
||||
legacy_tables = legacy_tables_present(database_path)
|
||||
staging_cleanup = False
|
||||
if marker is not None:
|
||||
staging = sdk_migration_staging(
|
||||
data_dir=settings.data_dir,
|
||||
migration_id=marker.migration_id,
|
||||
namespace=HY_MEMORY_NAMESPACE,
|
||||
)
|
||||
staging_cleanup = (
|
||||
staging.protective_root.exists()
|
||||
or staging.staging_home.exists()
|
||||
)
|
||||
document = {
|
||||
"schema_version": 1,
|
||||
"authority": "sdk",
|
||||
"state": "cleanup_required" if legacy_tables else "complete",
|
||||
"state": (
|
||||
"cleanup_required"
|
||||
if legacy_tables or staging_cleanup
|
||||
else "complete"
|
||||
),
|
||||
"migration_id": marker.migration_id if marker else None,
|
||||
"generation": generation,
|
||||
"legacy_tables": list(legacy_tables),
|
||||
|
||||
@@ -42,6 +42,13 @@ from one_memory.context_raw import revalidate_raw_context
|
||||
from one_memory.context_search import search_context
|
||||
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception
|
||||
from one_memory.external_memory_access import external_access
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_READER
|
||||
from one_memory.hy_sdk_runtime import sdk_relevance_floor
|
||||
from one_memory.hy_sdk_types import (
|
||||
HySdkMemoryRecord,
|
||||
parse_sdk_raw_search,
|
||||
parse_sdk_search,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.retrieval.raw_lane import read_raw_indexed_context
|
||||
from one_memory.semantic_deadline import hy_wait_budget, search_within
|
||||
@@ -455,19 +462,33 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
except Exception:
|
||||
raise HTTPException(503, "context_authority_unavailable") from None
|
||||
|
||||
bridge = getattr(request.app.state, "sdk_library_bridge", None)
|
||||
if bridge is None:
|
||||
client_handle = getattr(request.app.state, "sdk_client_handle", None)
|
||||
if client_handle is None:
|
||||
raise HTTPException(503, "sdk_context_unavailable")
|
||||
try:
|
||||
client = await client_handle.client()
|
||||
except Exception:
|
||||
raise HTTPException(503, "sdk_context_unavailable") from None
|
||||
|
||||
async def raw_lane() -> tuple[SdkRawContextItem, ...]:
|
||||
if "raw" not in value.kinds or not authority.raw_allowed:
|
||||
return ()
|
||||
hits = await bridge.search_sdk_raw(
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
value.query,
|
||||
user_id="local",
|
||||
source_ids=tuple(authority.raw_sources or ()),
|
||||
user_ids=["local"],
|
||||
agent_ids=[],
|
||||
include_sources=list(authority.raw_sources or ()),
|
||||
limit=value.limit,
|
||||
min_score=0.0,
|
||||
schema_min_score=0.0,
|
||||
schema_limit=0,
|
||||
intention_limit=0,
|
||||
scene="chat",
|
||||
reader=HY_MEMORY_READER,
|
||||
layers=["l1_raw"],
|
||||
)
|
||||
hits = parse_sdk_raw_search(raw)
|
||||
return tuple(
|
||||
SdkRawContextItem(
|
||||
raw_id=hit.raw_id,
|
||||
@@ -487,15 +508,24 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
|
||||
if "memory" not in value.kinds or not authority.memory_allowed:
|
||||
return ()
|
||||
hits = await bridge.search_memories(
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
value.query,
|
||||
user_id="local",
|
||||
agent_id="*",
|
||||
user_ids=["local"],
|
||||
agent_ids=[],
|
||||
limit=value.limit,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=value.limit,
|
||||
intention_limit=value.limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
items: list[SdkMemoryContextItem] = []
|
||||
for hit in hits:
|
||||
memory = await bridge.get_sdk_memory(hit.memory_id)
|
||||
record = await asyncio.to_thread(client.get, hit.memory_id)
|
||||
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
|
||||
if memory is None or memory.status != "active":
|
||||
continue
|
||||
items.append(
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
"""Pure conversation-window helpers shared by capture, retrieval, and writes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
|
||||
from one_memory.memory_tasks import segment_evidence_record
|
||||
|
||||
MEMORY_TASK_TIMEOUT_MARGIN_SECONDS = 60.0
|
||||
MEMORY_TASK_TIMEOUT_MIN_SECONDS = 180.0
|
||||
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
|
||||
|
||||
_HARNESS_PREFIXES = (
|
||||
"<codex_internal_context",
|
||||
"# agents.md instructions",
|
||||
"<environment_context>",
|
||||
"<permissions instructions>",
|
||||
"<system-reminder>",
|
||||
"<system_notification>",
|
||||
"<instructions>",
|
||||
)
|
||||
_HOST_ENVELOPE = re.compile(
|
||||
r"(?:"
|
||||
r"^<timestamp>.*?<system_(?:notification|reminder)>|"
|
||||
r"^<system_(?:notification|reminder)>|"
|
||||
r'^\{\s*"request_id"\s*:.*?"steps"\s*:'
|
||||
r")",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_WHOLE_TURN_CHARACTERS = 60_000
|
||||
_WINDOW_BYTE_CEILING = 2_000_000
|
||||
_HISTORY_MERGE_TURNS = 8
|
||||
_HISTORY_MERGE_CHARACTERS = 8_000
|
||||
|
||||
|
||||
def memory_task_timeout_seconds(model_timeout_seconds: float) -> float:
|
||||
"""Bound a whole write without treating one model request as its budget."""
|
||||
return max(
|
||||
MEMORY_TASK_TIMEOUT_MIN_SECONDS,
|
||||
float(model_timeout_seconds) + MEMORY_TASK_TIMEOUT_MARGIN_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
def _coding_tool_result_max_bytes() -> int:
|
||||
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
|
||||
if raw is None:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
|
||||
|
||||
_HY_TOOL_CHARACTERS = _coding_tool_result_max_bytes()
|
||||
|
||||
|
||||
def _conversation_chunks(
|
||||
records: list[EvidenceRecord],
|
||||
*,
|
||||
max_characters: int | None = None,
|
||||
max_turns: int | None = None,
|
||||
completed_only: bool = False,
|
||||
) -> list[tuple[EvidenceRecord, ...]]:
|
||||
"""Build bounded, complete user/assistant/tool windows."""
|
||||
turns: list[list[EvidenceRecord]] = []
|
||||
for record in records:
|
||||
if not _is_hy_turn_record(record):
|
||||
continue
|
||||
pending_proposal = bool(turns) and not _opens_turn(turns[-1][0])
|
||||
if not turns or (_opens_turn(record) and not pending_proposal):
|
||||
turns.append([record])
|
||||
else:
|
||||
turns[-1].append(record)
|
||||
|
||||
if completed_only and turns and _trailing_user_turn_is_open(turns[-1]):
|
||||
turns.pop()
|
||||
|
||||
whole = _whole_turns() and max_characters is None
|
||||
if whole:
|
||||
size = _hy_read_length
|
||||
max_characters = _WHOLE_TURN_CHARACTERS
|
||||
max_turns = max_turns or 1_000_000
|
||||
else:
|
||||
size = lambda record: len(record.verbatim) # noqa: E731
|
||||
max_characters = max_characters or 12_000
|
||||
max_turns = max_turns or 24
|
||||
|
||||
chunks: list[tuple[EvidenceRecord, ...]] = []
|
||||
for turn in turns:
|
||||
original_question = turn[0] if _opens_turn(turn[0]) else None
|
||||
expanded: list[EvidenceRecord] = []
|
||||
for record in turn:
|
||||
reserve = (
|
||||
size(original_question)
|
||||
if original_question is not None
|
||||
and record is not original_question
|
||||
and size(original_question) < max_characters
|
||||
else 0
|
||||
)
|
||||
segment_limit = max(1, max_characters - reserve)
|
||||
if size(record) <= segment_limit:
|
||||
expanded.append(record)
|
||||
continue
|
||||
spans = _semantic_segment_spans(record.verbatim, segment_limit)
|
||||
expanded.extend(
|
||||
segment_evidence_record(record, index=index, start=start, end=end)
|
||||
if len(spans) > 1
|
||||
else record
|
||||
for index, (start, end) in enumerate(spans)
|
||||
)
|
||||
question = expanded[0] if expanded and _opens_turn(expanded[0]) else None
|
||||
current: list[EvidenceRecord] = []
|
||||
for record in expanded:
|
||||
if current and (
|
||||
len(current) >= max_turns
|
||||
or sum(size(item) for item in current) + size(record) > max_characters
|
||||
or sum(_window_bytes(item) for item in current) + _window_bytes(record)
|
||||
> _WINDOW_BYTE_CEILING
|
||||
):
|
||||
chunks.append(tuple(current))
|
||||
current = (
|
||||
[question]
|
||||
if question is not None
|
||||
and max_turns > 1
|
||||
and size(question) + size(record) <= max_characters
|
||||
else []
|
||||
)
|
||||
current.append(record)
|
||||
chunks.append(tuple(current))
|
||||
return _merge_tool_only_chunks(chunks)
|
||||
|
||||
|
||||
def _semantic_segment_spans(text: str, limit: int) -> list[tuple[int, int]]:
|
||||
if len(text) <= limit:
|
||||
return [(0, len(text))]
|
||||
spans: list[tuple[int, int]] = []
|
||||
start = 0
|
||||
separators = ("\n\n", "\n", "。", "!", "?", ". ", "! ", "? ")
|
||||
while start < len(text):
|
||||
hard_end = min(len(text), start + limit)
|
||||
if hard_end == len(text):
|
||||
end = hard_end
|
||||
else:
|
||||
floor = start + max(1, limit // 2)
|
||||
candidates = [text.rfind(separator, floor, hard_end) for separator in separators]
|
||||
boundary = max(candidates)
|
||||
if boundary < floor:
|
||||
end = hard_end
|
||||
else:
|
||||
matched = next(
|
||||
separator for separator in separators if text.startswith(separator, boundary)
|
||||
)
|
||||
end = boundary + len(matched)
|
||||
spans.append((start, end))
|
||||
start = end
|
||||
return spans
|
||||
|
||||
|
||||
def _merge_tool_only_chunks(
|
||||
chunks: list[tuple[EvidenceRecord, ...]],
|
||||
) -> list[tuple[EvidenceRecord, ...]]:
|
||||
if not any(chunk and all(record.role == "tool" for record in chunk) for chunk in chunks):
|
||||
return chunks
|
||||
merged: list[tuple[EvidenceRecord, ...]] = []
|
||||
leading_tools: list[EvidenceRecord] = []
|
||||
for chunk in chunks:
|
||||
if not chunk:
|
||||
continue
|
||||
if all(record.role == "tool" for record in chunk):
|
||||
if merged:
|
||||
merged[-1] = (*merged[-1], *chunk)
|
||||
else:
|
||||
leading_tools.extend(chunk)
|
||||
continue
|
||||
if leading_tools:
|
||||
chunk = (*leading_tools, *chunk)
|
||||
leading_tools = []
|
||||
merged.append(chunk)
|
||||
return [] if leading_tools else merged
|
||||
|
||||
|
||||
def _merge_historical_chunks(
|
||||
chunks: list[tuple[EvidenceRecord, ...]],
|
||||
*,
|
||||
max_turns: int = _HISTORY_MERGE_TURNS,
|
||||
max_characters: int = _HISTORY_MERGE_CHARACTERS,
|
||||
) -> list[tuple[EvidenceRecord, ...]]:
|
||||
if not chunks:
|
||||
return []
|
||||
merged: list[tuple[EvidenceRecord, ...]] = []
|
||||
current: list[EvidenceRecord] = []
|
||||
current_turns = 0
|
||||
current_characters = 0
|
||||
for chunk in chunks:
|
||||
chunk_characters = sum(len(record.verbatim) for record in chunk)
|
||||
if current and (
|
||||
current_turns >= max_turns or current_characters + chunk_characters > max_characters
|
||||
):
|
||||
merged.append(tuple(current))
|
||||
current = []
|
||||
current_turns = 0
|
||||
current_characters = 0
|
||||
current.extend(chunk)
|
||||
current_turns += 1
|
||||
current_characters += chunk_characters
|
||||
if current:
|
||||
merged.append(tuple(current))
|
||||
return merged
|
||||
|
||||
|
||||
def _chunk_input_ref(chunk: tuple[EvidenceRecord, ...]) -> str:
|
||||
user_records = tuple(item for item in chunk if item.role == "user")
|
||||
return user_records[-1].evidence_id if user_records else chunk[-1].evidence_id
|
||||
|
||||
|
||||
def _opens_turn(record: EvidenceRecord) -> bool:
|
||||
return record.role == "user"
|
||||
|
||||
|
||||
def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
|
||||
if len(turn) != 1:
|
||||
return False
|
||||
opener = turn[0]
|
||||
return (
|
||||
_opens_turn(opener)
|
||||
and opener.content_kind is EvidenceContentKind.RAW_CONVERSATION
|
||||
and not _is_memory_note(opener)
|
||||
)
|
||||
|
||||
|
||||
def _is_hy_turn_record(record: EvidenceRecord) -> bool:
|
||||
if record.role == "tool":
|
||||
return record.content_kind is EvidenceContentKind.RAW_CONVERSATION and bool(
|
||||
record.verbatim.strip()
|
||||
)
|
||||
return _is_hy_input_turn(record)
|
||||
|
||||
|
||||
def _is_hy_input_turn(record: EvidenceRecord) -> bool:
|
||||
source = record.verbatim.strip()
|
||||
if not source:
|
||||
return False
|
||||
if record.content_kind is EvidenceContentKind.MEMORY_CONTENT:
|
||||
return True
|
||||
if record.role == "assistant":
|
||||
return True
|
||||
if record.role != "user":
|
||||
return False
|
||||
if source.casefold().startswith(_HARNESS_PREFIXES):
|
||||
return False
|
||||
return _HOST_ENVELOPE.search(source[:2_000]) is None
|
||||
|
||||
|
||||
def _is_memory_note(record: EvidenceRecord) -> bool:
|
||||
meta = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
|
||||
return (
|
||||
record.source_id == "one-memory-manual"
|
||||
and meta is not None
|
||||
and meta.raw_meta.get("input_mode") == "direct_memory_note"
|
||||
)
|
||||
|
||||
|
||||
def _window_bytes(record: EvidenceRecord) -> int:
|
||||
return len(record.verbatim.encode("utf-8"))
|
||||
|
||||
|
||||
def _whole_turns() -> bool:
|
||||
return os.environ.get("ONE_MEMORY_SPLIT_TURNS", "").strip() != "1"
|
||||
|
||||
|
||||
def _hy_read_length(record: EvidenceRecord) -> int:
|
||||
if record.role == "tool":
|
||||
return min(len(record.verbatim), _HY_TOOL_CHARACTERS)
|
||||
return len(record.verbatim)
|
||||
@@ -0,0 +1,271 @@
|
||||
"""Kuzu graph-width inspection and explicit user-directed administration.
|
||||
|
||||
This module is the second graph architecture exception. It may open Kuzu
|
||||
directly to inspect an existing graph's schema, but owns no prompts, memory
|
||||
reconciliation, coding data, vector database, or legacy governance behavior.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
|
||||
class GraphWidthError(RuntimeError):
|
||||
"""Base error for graph-width inspection and administration."""
|
||||
|
||||
|
||||
class GraphWidthDecision(StrEnum):
|
||||
"""What to do when an existing graph disagrees with the embedding width."""
|
||||
|
||||
KEEP_EXISTING = "keep_existing"
|
||||
REBUILD_WITH_NEW = "rebuild_with_new"
|
||||
EXPORT_FIRST = "export_first"
|
||||
|
||||
|
||||
class GraphWidthMismatch(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
stored_width: int
|
||||
expected_width: int
|
||||
graph_path: str
|
||||
|
||||
|
||||
class GraphWidthOutcome(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
archived_path: str | None = None
|
||||
exported_path: str | None = None
|
||||
|
||||
|
||||
class GraphWidthMismatchError(GraphWidthError):
|
||||
"""Raised instead of silently archiving a live graph."""
|
||||
|
||||
def __init__(self, mismatch: GraphWidthMismatch) -> None:
|
||||
super().__init__(
|
||||
f"graph_width_mismatch:stored={mismatch.stored_width}"
|
||||
f":expected={mismatch.expected_width}"
|
||||
)
|
||||
self.mismatch = mismatch
|
||||
|
||||
|
||||
def graph_store_enabled() -> bool:
|
||||
"""Whether HY should open the Kuzu graph store."""
|
||||
raw = os.environ.get("ONE_MEMORY_GRAPH_STORE", "").strip().lower()
|
||||
if not raw:
|
||||
return False
|
||||
return raw != "none"
|
||||
|
||||
|
||||
def inspect_graph_width_mismatch(config: Any) -> GraphWidthMismatch | None:
|
||||
"""Return a mismatch without renaming or deleting anything."""
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
|
||||
return _inspect_graph_width(db_path, expected)
|
||||
|
||||
|
||||
def inspect_profile_graph_width(
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
) -> GraphWidthMismatch | None:
|
||||
"""Inspect the graph path and width selected by the active profile."""
|
||||
del api_key
|
||||
return _inspect_graph_width(
|
||||
_profile_graph_path(data_dir),
|
||||
_profile_embedding_width(profile),
|
||||
)
|
||||
|
||||
|
||||
def decide_profile_graph_width(
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
decision: GraphWidthDecision,
|
||||
) -> GraphWidthOutcome:
|
||||
"""Apply the user's decision to the active profile's graph."""
|
||||
del api_key
|
||||
config = _GraphAdminConfig(
|
||||
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir))),
|
||||
embedder=_EmbeddingWidthConfig(_profile_embedding_width(profile)),
|
||||
)
|
||||
return apply_graph_width_decision(
|
||||
config,
|
||||
decision,
|
||||
export_root=data_dir.parent,
|
||||
)
|
||||
|
||||
|
||||
def apply_graph_width_decision(
|
||||
config: Any,
|
||||
decision: GraphWidthDecision,
|
||||
*,
|
||||
export_root: Path | None = None,
|
||||
) -> GraphWidthOutcome:
|
||||
"""Act only on a current mismatch and only as explicitly requested."""
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is None:
|
||||
return GraphWidthOutcome()
|
||||
if decision is GraphWidthDecision.REBUILD_WITH_NEW:
|
||||
archive = retire_mismatched_graph_store(config)
|
||||
return GraphWidthOutcome(archived_path=str(archive) if archive is not None else None)
|
||||
if decision is GraphWidthDecision.EXPORT_FIRST:
|
||||
if export_root is None:
|
||||
raise GraphWidthError("export_root_required")
|
||||
return GraphWidthOutcome(
|
||||
exported_path=str(export_graph_store(config, export_root=export_root))
|
||||
)
|
||||
return GraphWidthOutcome()
|
||||
|
||||
|
||||
def export_graph_store(config: Any, *, export_root: Path) -> Path:
|
||||
"""Copy the graph outside its data directory, atomically."""
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")
|
||||
destination = export_root / f"one-memory-graph-{stamp}"
|
||||
staging = destination.with_name(f"{destination.name}.partial")
|
||||
sources = [db_path] + [
|
||||
sibling
|
||||
for suffix in (".wal", ".shadow", ".tmp")
|
||||
if (sibling := db_path.with_name(f"{db_path.name}{suffix}")).exists()
|
||||
]
|
||||
try:
|
||||
staging.mkdir(parents=True)
|
||||
for source in sources:
|
||||
shutil.copy2(source, staging / source.name)
|
||||
staging.rename(destination)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
logging.getLogger(__name__).info(
|
||||
"exported hy graph store to %s (%d files)",
|
||||
destination,
|
||||
len(sources),
|
||||
)
|
||||
return destination
|
||||
|
||||
|
||||
def retire_mismatched_graph_store(config: Any) -> Path | None:
|
||||
"""Archive a mismatched Kuzu graph only after the user chooses rebuild."""
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
if not db_path.exists():
|
||||
return None
|
||||
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
|
||||
if not expected:
|
||||
return None
|
||||
stored = _graph_embedding_width(db_path)
|
||||
if stored is None or stored == int(expected):
|
||||
return None
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
archive = db_path.with_name(f"{db_path.name}.dim{stored}.{stamp}.bak")
|
||||
db_path.rename(archive)
|
||||
for suffix in (".wal", ".shadow", ".tmp"):
|
||||
sibling = db_path.with_name(f"{db_path.name}{suffix}")
|
||||
if sibling.exists():
|
||||
sibling.rename(archive.with_name(f"{archive.name}{suffix}"))
|
||||
logging.getLogger(__name__).warning(
|
||||
"retired hy graph store with embedding width %s (model expects %s): %s",
|
||||
stored,
|
||||
expected,
|
||||
archive.name,
|
||||
)
|
||||
return archive
|
||||
|
||||
|
||||
_GRAPH_WIDTH_CACHE: dict[str, int | None] = {}
|
||||
_GRAPH_WIDTH_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _graph_embedding_width(db_path: Path) -> int | None:
|
||||
"""Read the Memory embedding width directly from Kuzu's schema."""
|
||||
try:
|
||||
kuzu = import_module("kuzu")
|
||||
except ModuleNotFoundError:
|
||||
return None
|
||||
key = str(db_path.resolve())
|
||||
with _GRAPH_WIDTH_LOCK:
|
||||
if key in _GRAPH_WIDTH_CACHE:
|
||||
return _GRAPH_WIDTH_CACHE[key]
|
||||
try:
|
||||
database = kuzu.Database(str(db_path), read_only=True)
|
||||
except Exception:
|
||||
return None
|
||||
width: int | None = None
|
||||
try:
|
||||
connection = kuzu.Connection(database)
|
||||
result = connection.execute("CALL TABLE_INFO('Memory') RETURN *")
|
||||
while result.has_next():
|
||||
row = result.get_next()
|
||||
if len(row) < 3 or row[1] != "embedding":
|
||||
continue
|
||||
match = re.search(r"\[(\d+)\]", str(row[2]))
|
||||
width = int(match.group(1)) if match else None
|
||||
break
|
||||
except Exception:
|
||||
width = None
|
||||
with _GRAPH_WIDTH_LOCK:
|
||||
_GRAPH_WIDTH_CACHE[key] = width
|
||||
return width
|
||||
|
||||
|
||||
def _inspect_graph_width(
|
||||
db_path: Path,
|
||||
expected_width: object,
|
||||
) -> GraphWidthMismatch | None:
|
||||
if not db_path.exists() or not expected_width:
|
||||
return None
|
||||
stored = _graph_embedding_width(db_path)
|
||||
if stored is None or stored == int(expected_width):
|
||||
return None
|
||||
return GraphWidthMismatch(
|
||||
stored_width=int(stored),
|
||||
expected_width=int(expected_width),
|
||||
graph_path=str(db_path),
|
||||
)
|
||||
|
||||
|
||||
def _profile_graph_path(data_dir: Path) -> Path:
|
||||
name = "graph" if graph_store_enabled() else "graph-disabled"
|
||||
return data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / name
|
||||
|
||||
|
||||
def _profile_embedding_width(profile: ModelProfileSnapshot) -> int | None:
|
||||
if profile.embedding_dimensions is not None:
|
||||
return profile.embedding_dimensions
|
||||
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
|
||||
from one_memory.embedding_assets import BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
return BUILTIN_EMBEDDING_DIMENSIONS
|
||||
return None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _GraphStoreConfig:
|
||||
db_path: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _EmbeddingWidthConfig:
|
||||
embedding_dims: int | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _GraphAdminConfig:
|
||||
graph_store: _GraphStoreConfig
|
||||
embedder: _EmbeddingWidthConfig
|
||||
@@ -0,0 +1,185 @@
|
||||
"""Read-only projection of HY's Kuzu graph for MemHub's existing graph UI.
|
||||
|
||||
This module is one of the two explicit graph architecture exceptions. The
|
||||
public SDK does not expose graph projection yet, so it may use the client's
|
||||
private graph handle and loop thread. It must remain read-only and must not
|
||||
grow memory, vector, coding, prompt, or governance responsibilities.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, cast
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
|
||||
class HyGraphNode(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
node_id: str
|
||||
content: str
|
||||
layer: str | None = None
|
||||
confidence: float | None = None
|
||||
|
||||
|
||||
class HyGraphEdge(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
source_node_id: str
|
||||
target_node_id: str
|
||||
relation: str
|
||||
weight: float | None = None
|
||||
|
||||
|
||||
class HyGraphSnapshot(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
nodes: tuple[HyGraphNode, ...] = ()
|
||||
edges: tuple[HyGraphEdge, ...] = ()
|
||||
# False when HY was never consulted, which is not the same as HY having
|
||||
# induced no relations.
|
||||
consulted: bool = True
|
||||
|
||||
|
||||
def read_graph_snapshot(
|
||||
client: object,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
limit: int,
|
||||
) -> HyGraphSnapshot:
|
||||
raw_nodes, raw_edges = _read_graph_snapshot(client, user_id, agent_id, limit)
|
||||
return HyGraphSnapshot(
|
||||
nodes=tuple(
|
||||
HyGraphNode(
|
||||
node_id=str(row[0]),
|
||||
content=str(row[1]),
|
||||
layer=str(row[2]) if row[2] is not None else None,
|
||||
confidence=_safe_float(row[3]),
|
||||
)
|
||||
for row in raw_nodes
|
||||
if len(row) >= 4 and row[0] and row[1]
|
||||
),
|
||||
edges=tuple(
|
||||
HyGraphEdge(
|
||||
source_node_id=str(row[0]),
|
||||
target_node_id=str(row[1]),
|
||||
relation=str(row[2]),
|
||||
weight=_safe_float(row[3]),
|
||||
)
|
||||
for row in raw_edges
|
||||
if len(row) >= 4 and row[0] and row[1] and row[2]
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _read_graph_snapshot(
|
||||
client: object,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
limit: int,
|
||||
) -> tuple[list[list[object]], list[list[object]]]:
|
||||
"""Read the graph through the SDK handles until a public API exists."""
|
||||
concrete = cast(Any, client)
|
||||
# Keep the existing architecture gate unchanged; this module is the named
|
||||
# graph-only exception while the rest of MemHub remains on public handles.
|
||||
graph_store = getattr(concrete, "_graph_store") # noqa: B009
|
||||
if graph_store is None:
|
||||
return ([], [])
|
||||
loop_thread = getattr(concrete, "_loop_thread") # noqa: B009
|
||||
# Kuzu rejects a prepared statement that is handed a parameter its text
|
||||
# never references, so a wildcard read must not bind agent_id at all.
|
||||
params: dict[str, object] = {"user_id": user_id}
|
||||
if agent_id != "*":
|
||||
params["agent_id"] = agent_id
|
||||
safe_limit = max(1, min(int(limit), 1_000))
|
||||
memory_agent = "" if agent_id == "*" else " AND m.agent_id = $agent_id"
|
||||
related_agent = (
|
||||
"" if agent_id == "*" else " AND a.agent_id = $agent_id AND b.agent_id = $agent_id"
|
||||
)
|
||||
source_agent = "" if agent_id == "*" else " AND a.agent_id = $agent_id"
|
||||
nodes = (
|
||||
loop_thread.run(
|
||||
graph_store._run(
|
||||
f"""
|
||||
MATCH (m:Memory)
|
||||
WHERE m.user_id = $user_id{memory_agent}
|
||||
AND m.status = 'active'
|
||||
RETURN m.node_id, m.content, m.layer, m.confidence
|
||||
LIMIT {safe_limit}
|
||||
""",
|
||||
params,
|
||||
)
|
||||
)
|
||||
or []
|
||||
)
|
||||
related_edges = (
|
||||
loop_thread.run(
|
||||
graph_store._run(
|
||||
f"""
|
||||
MATCH (a:Memory)-[r:RELATED_TO]->(b:Memory)
|
||||
WHERE a.user_id = $user_id{related_agent}
|
||||
AND b.user_id = $user_id
|
||||
AND a.status = 'active' AND b.status = 'active'
|
||||
RETURN a.node_id, b.node_id, r.relation_type, r.weight
|
||||
LIMIT {safe_limit * 4}
|
||||
""",
|
||||
params,
|
||||
)
|
||||
)
|
||||
or []
|
||||
)
|
||||
derived_edges = (
|
||||
loop_thread.run(
|
||||
graph_store._run(
|
||||
f"""
|
||||
MATCH (a:Memory)-[:DERIVED_FROM]->(v:VdbRef)
|
||||
WHERE a.user_id = $user_id{source_agent}
|
||||
AND a.status = 'active'
|
||||
RETURN a.node_id, v.node_id, 'derived_from', 1.0
|
||||
LIMIT {safe_limit * 4}
|
||||
""",
|
||||
params,
|
||||
)
|
||||
)
|
||||
or []
|
||||
)
|
||||
topic_nodes = (
|
||||
loop_thread.run(
|
||||
graph_store._run(
|
||||
f"""
|
||||
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
|
||||
WHERE m.user_id = $user_id{memory_agent}
|
||||
AND m.status = 'active'
|
||||
RETURN DISTINCT t.topic_id, t.name, 'TOPIC', 1.0
|
||||
LIMIT {safe_limit}
|
||||
""",
|
||||
params,
|
||||
)
|
||||
)
|
||||
or []
|
||||
)
|
||||
tagged_edges = (
|
||||
loop_thread.run(
|
||||
graph_store._run(
|
||||
f"""
|
||||
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
|
||||
WHERE m.user_id = $user_id{memory_agent}
|
||||
AND m.status = 'active'
|
||||
RETURN m.node_id, t.topic_id, 'tagged_with', 1.0
|
||||
LIMIT {safe_limit * 4}
|
||||
""",
|
||||
params,
|
||||
)
|
||||
)
|
||||
or []
|
||||
)
|
||||
return (
|
||||
[*nodes, *topic_nodes],
|
||||
[*related_edges, *derived_edges, *tagged_edges],
|
||||
)
|
||||
|
||||
|
||||
def _safe_float(value: object) -> float | None:
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
return None
|
||||
@@ -1,128 +0,0 @@
|
||||
"""Which memories HY no longer counts as current, read from HY itself.
|
||||
|
||||
HY owns a memory's lifecycle: it supersedes, deduplicates, shadows and
|
||||
restores on its own schedule — during a search, while deleting the head of a
|
||||
chain — and most of that never passes through a write the ledger sees. So
|
||||
the ledger does not keep a copy of any of it; readers ask HY, once per read,
|
||||
for the nodes it has set aside.
|
||||
|
||||
Only the vector store is read. HY writes a node's status there on every
|
||||
transition, and the graph holds no status the vector store lacks.
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import Collection, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from importlib import import_module
|
||||
from types import MappingProxyType
|
||||
from typing import Any
|
||||
|
||||
#: HY's one status for a node that is current. Everything else it writes —
|
||||
#: superseded, shadow, archived, negated, conflicted — takes it out of recall.
|
||||
_ACTIVE = "active"
|
||||
#: HY's verbatim chunks. Stored as shadow from the start and never a memory,
|
||||
#: so never something a reader could have shown.
|
||||
_RAW_LAYER = "l1_raw"
|
||||
|
||||
|
||||
class HyLifecycleUnreadable(RuntimeError):
|
||||
"""HY's store could not be read; never to be taken for "nothing is hidden"."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, eq=False)
|
||||
class HyLifecycle:
|
||||
"""HY's set-aside nodes for one user, each with what HY says replaced it."""
|
||||
|
||||
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
|
||||
|
||||
def __eq__(self, other: object) -> bool:
|
||||
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(other.hidden)
|
||||
|
||||
def __hash__(self) -> int:
|
||||
return hash(tuple(sorted(self.hidden.items())))
|
||||
|
||||
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
|
||||
return self.hidden.get(memory_id, ())
|
||||
|
||||
def replacing(self, memory_id: str) -> tuple[str, ...]:
|
||||
"""The set-aside memories HY names this one as the replacement of."""
|
||||
return tuple(sorted(old for old, by in self.hidden.items() if memory_id in by))
|
||||
|
||||
|
||||
def read_hy_lifecycle(
|
||||
client: object, user_id: str, ids: Collection[str] | None = None
|
||||
) -> HyLifecycle:
|
||||
"""Every node of `user_id` HY holds as anything but active, in one read.
|
||||
|
||||
With `ids`, only those nodes are looked up — what a single-memory read
|
||||
needs, and a lookup by key rather than a scan; `replacing` is then only
|
||||
known among them.
|
||||
|
||||
Read one level below HY's own listing, which drops a failed read and
|
||||
answers empty: a hidden memory would then read as current.
|
||||
"""
|
||||
native: Any = client
|
||||
try:
|
||||
chroma = import_module("hy_memory.data.vector_store_chroma")
|
||||
store = native._vector_store
|
||||
collection = store._collection
|
||||
if not isinstance(store, chroma.ChromaVectorStore) or collection is None:
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_store_unsupported")
|
||||
found = collection.get(
|
||||
ids=None if ids is None else list(ids),
|
||||
where={
|
||||
"$and": [
|
||||
{"user_id": {"$eq": user_id}},
|
||||
{"status": {"$ne": _ACTIVE}},
|
||||
{"layer": {"$ne": _RAW_LAYER}},
|
||||
]
|
||||
},
|
||||
include=["metadatas"],
|
||||
)
|
||||
metadatas = found["metadatas"]
|
||||
point_ids = found["ids"]
|
||||
if (
|
||||
not isinstance(point_ids, list)
|
||||
or not isinstance(metadatas, list)
|
||||
or len(point_ids) != len(metadatas)
|
||||
):
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
|
||||
except HyLifecycleUnreadable:
|
||||
raise
|
||||
except Exception as error:
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_unreadable") from error
|
||||
hidden: dict[str, tuple[str, ...]] = {}
|
||||
for point_id, metadata in zip(point_ids, metadatas, strict=True):
|
||||
if not isinstance(point_id, str) or not point_id or not isinstance(metadata, Mapping):
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
|
||||
payload = chroma.ChromaVectorStore._chroma_metadata_to_payload(dict(metadata))
|
||||
node_id = payload.get("node_id") or point_id
|
||||
if not isinstance(node_id, str) or not node_id or node_id != point_id:
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
|
||||
hidden[node_id] = _replacements(payload)
|
||||
return HyLifecycle(hidden=hidden)
|
||||
|
||||
|
||||
def _replacements(payload: Mapping[str, object]) -> tuple[str, ...]:
|
||||
"""What HY names as taking this node's place: its chain successor or the kept duplicate."""
|
||||
names: list[str] = []
|
||||
superseded_by = payload.get("superseded_by")
|
||||
if isinstance(superseded_by, list):
|
||||
names.extend(value for value in superseded_by if isinstance(value, str) and value)
|
||||
kept = payload.get("dedup_kept_id")
|
||||
custom = payload.get("custom")
|
||||
if isinstance(custom, str):
|
||||
try:
|
||||
custom = json.loads(custom)
|
||||
except ValueError:
|
||||
custom = None
|
||||
if not isinstance(kept, str) and isinstance(custom, Mapping):
|
||||
audit = custom.get("hy_dedup")
|
||||
if isinstance(audit, Mapping):
|
||||
kept = audit.get("kept_id")
|
||||
if isinstance(kept, str) and kept:
|
||||
names.append(kept)
|
||||
return tuple(dict.fromkeys(names))
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,331 +0,0 @@
|
||||
"""Transport validation for the pinned HY SDK; HY's prompts and decisions pass through unchanged."""
|
||||
|
||||
import logging
|
||||
from contextlib import nullcontext
|
||||
from time import monotonic
|
||||
from typing import Any
|
||||
|
||||
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
|
||||
from one_memory.hy_source_context import HyWriteBindings
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
|
||||
|
||||
def _http_status(value: Any) -> int | None:
|
||||
return value if type(value) is int and 100 <= value <= 599 else None
|
||||
|
||||
|
||||
def _exception_status(error: Exception) -> int | None:
|
||||
try:
|
||||
status = _http_status(getattr(error, "status_code", None))
|
||||
if status is not None:
|
||||
return status
|
||||
status = _http_status(getattr(getattr(error, "response", None), "status_code", None))
|
||||
if status is not None:
|
||||
return status
|
||||
# Native LLMRetryError retains sanitized attempt dictionaries. Only the
|
||||
# final attempt describes the terminal failure; never parse error text.
|
||||
attempts = getattr(error, "attempts", None)
|
||||
if isinstance(attempts, (list, tuple)) and attempts and isinstance(attempts[-1], dict):
|
||||
return _http_status(attempts[-1].get("status_code"))
|
||||
except Exception:
|
||||
# Diagnostic attribute access must not replace the provider exception.
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _model_failure_code(status: int | None) -> str:
|
||||
specific = {
|
||||
401: "hy_memory_model_authentication_failed",
|
||||
402: "hy_memory_model_payment_required",
|
||||
403: "hy_memory_model_access_denied",
|
||||
408: "hy_memory_model_timeout",
|
||||
429: "hy_memory_model_rate_limited",
|
||||
}
|
||||
if status in specific:
|
||||
return specific[status]
|
||||
if status is not None and 500 <= status <= 599:
|
||||
return "hy_memory_model_service_unavailable"
|
||||
if status is not None and 400 <= status <= 422:
|
||||
return "hy_memory_model_request_incompatible"
|
||||
return "hy_memory_model_request_failed"
|
||||
|
||||
|
||||
def _terminal_client_timeout(error: Exception) -> bool:
|
||||
try:
|
||||
attempts = getattr(error, "attempts", None)
|
||||
if not isinstance(attempts, (list, tuple)) or not attempts:
|
||||
return False
|
||||
final = attempts[-1]
|
||||
if type(final) is not dict:
|
||||
return False
|
||||
kind = final.get("failure_kind")
|
||||
return type(kind) is str and kind == "timeout"
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class HyResponseGuard:
|
||||
def __init__(self, provider: Any, *, phase: str = "extract") -> None:
|
||||
self.provider = provider
|
||||
#: Diagnostic label only; the request passes through unchanged.
|
||||
self.phase = phase
|
||||
self.failures: list[str] = []
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.provider, name)
|
||||
|
||||
async def complete(self, *args: Any, **kwargs: Any) -> Any:
|
||||
return await self._complete("complete", args, kwargs)
|
||||
|
||||
async def complete_messages(self, *args: Any, **kwargs: Any) -> Any:
|
||||
return await self._complete("complete_messages", args, kwargs)
|
||||
|
||||
async def _complete(self, method: str, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
|
||||
for attempt in range(2):
|
||||
started = monotonic()
|
||||
try:
|
||||
response = await getattr(self.provider, method)(*args, **kwargs)
|
||||
except Exception as error:
|
||||
status = _exception_status(error)
|
||||
code = (
|
||||
"hy_memory_model_timeout"
|
||||
if status is None and _terminal_client_timeout(error)
|
||||
else _model_failure_code(status)
|
||||
)
|
||||
self.failures.append(code)
|
||||
logging.getLogger(__name__).warning(
|
||||
"%s: status_code=%s", code, status if status is not None else "unknown"
|
||||
)
|
||||
diagnostic_event(
|
||||
level="error",
|
||||
component="hy.model",
|
||||
event="hy.model_call.failed",
|
||||
outcome="failed",
|
||||
duration_ms=elapsed_ms(started),
|
||||
error_code=code,
|
||||
**diagnostic_exception(error),
|
||||
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
|
||||
)
|
||||
raise
|
||||
truncated = getattr(response, "finish_reason", "") == "length"
|
||||
empty = not (response.content or "").strip() and not getattr(
|
||||
response, "tool_calls", None
|
||||
)
|
||||
if not truncated and not empty:
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="hy.model",
|
||||
event="hy.model_call.completed",
|
||||
outcome="succeeded",
|
||||
duration_ms=elapsed_ms(started),
|
||||
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
|
||||
)
|
||||
return response
|
||||
code = (
|
||||
"hy_memory_model_output_truncated"
|
||||
if truncated
|
||||
else "hy_memory_model_empty_response"
|
||||
)
|
||||
if attempt:
|
||||
self.failures.append(code)
|
||||
raise ValueError(code)
|
||||
logging.getLogger(__name__).warning("%s: retrying the same HY model request", code)
|
||||
if truncated and "max_tokens" in kwargs:
|
||||
kwargs = {**kwargs, "max_tokens": min(16000, kwargs["max_tokens"] * 2)}
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
class HyReconcileGuard:
|
||||
#: Set only for the span of one leased write.
|
||||
step_timer: HyStepTimer | None = None
|
||||
|
||||
def __init__(self, reconciler: Any) -> None:
|
||||
self.reconciler = reconciler
|
||||
self.failures: list[str] = []
|
||||
self.bindings: HyWriteBindings | None = None
|
||||
self.imported_contents: frozenset[str] | None = None
|
||||
self.mutation_target_ids: set[str] = set()
|
||||
|
||||
def begin_write(self) -> None:
|
||||
self.mutation_target_ids.clear()
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.reconciler, name)
|
||||
|
||||
async def reconcile(self, *args: Any, **kwargs: Any) -> Any:
|
||||
timer = self.step_timer
|
||||
with timer.step("reconcile") if timer is not None else nullcontext():
|
||||
return await self._reconcile(args, kwargs)
|
||||
|
||||
async def _reconcile(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
|
||||
started = monotonic()
|
||||
if self.imported_contents is not None:
|
||||
metadata = kwargs.get("new_memories_with_meta") or []
|
||||
texts = kwargs.get("new_memories") or []
|
||||
if (
|
||||
not metadata
|
||||
or not texts
|
||||
or any(
|
||||
not isinstance(item, dict) or item.get("content") not in self.imported_contents
|
||||
for item in metadata
|
||||
)
|
||||
or any(text not in self.imported_contents for text in texts)
|
||||
or {item["content"] for item in metadata} != set(texts)
|
||||
):
|
||||
code = "hy_memory_write_verification_import_input_mismatch"
|
||||
self.failures.append(code)
|
||||
raise ValueError(code)
|
||||
try:
|
||||
result = await self.reconciler.reconcile(*args, **kwargs)
|
||||
except Exception as error:
|
||||
diagnostic_event(
|
||||
level="error",
|
||||
component="hy.reconcile",
|
||||
event="hy.reconcile.failed",
|
||||
outcome="failed",
|
||||
duration_ms=elapsed_ms(started),
|
||||
error_code="hy_memory_reconcile_failed",
|
||||
**diagnostic_exception(error),
|
||||
)
|
||||
raise
|
||||
if not result.success:
|
||||
self.failures.append("hy_memory_reconcile_failed")
|
||||
diagnostic_event(
|
||||
level="error",
|
||||
component="hy.reconcile",
|
||||
event="hy.reconcile.failed",
|
||||
outcome="failed",
|
||||
duration_ms=elapsed_ms(started),
|
||||
error_code="hy_memory_reconcile_failed",
|
||||
details={"result_count": len(result.ops)},
|
||||
)
|
||||
return result
|
||||
try:
|
||||
await self._check_targets(result, kwargs)
|
||||
except ValueError as error:
|
||||
code = str(error)
|
||||
self.failures.append(code)
|
||||
raise
|
||||
except Exception as error:
|
||||
code = "hy_memory_write_verification_failed"
|
||||
self.failures.append(code)
|
||||
raise ValueError(code) from error
|
||||
owners: dict[str, set[str]] = {}
|
||||
for item in kwargs.get("new_memories_with_meta") or []:
|
||||
if item.get("owner") in {"user", "agent"}:
|
||||
owners.setdefault(item["content"], set()).add(item["owner"])
|
||||
# HY's no-candidate ADD path copies content verbatim but omits owner.
|
||||
# Restore only an unambiguous exact match, never infer from wording.
|
||||
for operation in result.ops:
|
||||
source_owners = owners.get(operation.content, set())
|
||||
if operation.op == "ADD" and operation.owner is None and len(source_owners) == 1:
|
||||
operation.owner = next(iter(source_owners))
|
||||
operation_counts = {"ADD": 0, "UPDATE": 0, "SUPERSEDE": 0, "DELETE": 0}
|
||||
for operation in result.ops:
|
||||
raw_name = getattr(operation, "op", "")
|
||||
name = str(getattr(raw_name, "value", raw_name)).upper()
|
||||
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
|
||||
target = getattr(operation, "memory_id", None)
|
||||
if isinstance(target, str) and target:
|
||||
self.mutation_target_ids.add(target)
|
||||
self.mutation_target_ids.update(
|
||||
key
|
||||
for key in getattr(operation, "memory_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
)
|
||||
if name == "ADD":
|
||||
operation_counts["ADD"] += 1
|
||||
elif name in {"UPDATE", "MERGE", "NOOP"}:
|
||||
operation_counts["UPDATE"] += 1
|
||||
elif name == "SUPERSEDE":
|
||||
operation_counts["SUPERSEDE"] += 1
|
||||
elif name in {"DELETE", "SHADOW"}:
|
||||
operation_counts["DELETE"] += 1
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="hy.reconcile",
|
||||
event="hy.reconcile.completed",
|
||||
outcome="succeeded",
|
||||
duration_ms=elapsed_ms(started),
|
||||
details={
|
||||
"created_count": operation_counts["ADD"],
|
||||
"updated_count": operation_counts["UPDATE"],
|
||||
"superseded_count": operation_counts["SUPERSEDE"],
|
||||
"deleted_count": operation_counts["DELETE"],
|
||||
"result_count": len(result.ops),
|
||||
},
|
||||
)
|
||||
return result
|
||||
|
||||
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
|
||||
"""HY may only touch nodes of this user and scope that this write was allowed to see."""
|
||||
targets = {op.memory_id for op in result.ops if op.memory_id}
|
||||
targets.update(key for op in result.ops for key in getattr(op, "memory_ids", []))
|
||||
for key in targets:
|
||||
node = await kwargs["vector_store"].get_by_id(key)
|
||||
if (
|
||||
node is None
|
||||
or node.user_id != kwargs["user_id"]
|
||||
or node.agent_id != kwargs["agent_id"]
|
||||
):
|
||||
raise ValueError("hy_memory_write_verification_target_invalid")
|
||||
if self.bindings is not None:
|
||||
known = self.bindings.known.get(key)
|
||||
if known is None or (known.content, known.owner) != (node.content, node.owner):
|
||||
raise ValueError("hy_memory_write_verification_target_unbound")
|
||||
if self.bindings is None:
|
||||
return
|
||||
for op in result.ops:
|
||||
if not op.content:
|
||||
continue
|
||||
linked = {key for key in (op.memory_id, *getattr(op, "memory_ids", [])) if key}
|
||||
self.bindings.record_operation_support(op.content, op.owner, linked)
|
||||
if op.memory_id:
|
||||
self.bindings.verified_operations[op.memory_id] = (op.content, op.owner)
|
||||
|
||||
|
||||
def install_hy_write_guards(client: Any) -> None:
|
||||
"""The pinned HY SDK treats an empty reconcile response as a successful empty op list.
|
||||
|
||||
Bind this client's real write paths. Pro/lite use the ordinary writer;
|
||||
ultra uses the System2 writer's inner writer. Keep HY prompts, extraction,
|
||||
retrieval, reconciliation and storage intact; never patch the shared SDK
|
||||
installation.
|
||||
"""
|
||||
|
||||
async def install() -> None:
|
||||
from one_memory.hy_write_verification import HyExtractionGuard
|
||||
|
||||
ordinary = await client._registry.get_writer()
|
||||
system2 = await client._registry.get_system2_writer()
|
||||
guards: list[Any] = []
|
||||
seen_writers: set[int] = set()
|
||||
seen_guards: set[int] = set()
|
||||
for writer in (ordinary, system2._writer):
|
||||
if id(writer) in seen_writers:
|
||||
continue
|
||||
seen_writers.add(id(writer))
|
||||
agent = writer.mem_agent
|
||||
if not isinstance(agent.llm_provider, HyResponseGuard):
|
||||
agent.llm_provider = HyResponseGuard(agent.llm_provider)
|
||||
agent.extractor.llm = agent.llm_provider
|
||||
if not isinstance(agent.extractor, HyExtractionGuard):
|
||||
agent.extractor = HyExtractionGuard(agent.extractor)
|
||||
reconciler = writer.reconciler
|
||||
provider = reconciler._get_llm()
|
||||
if not isinstance(provider, HyResponseGuard):
|
||||
reconciler._llm = HyResponseGuard(provider, phase="reconcile")
|
||||
if not isinstance(reconciler, HyReconcileGuard):
|
||||
writer._reconciler = HyReconcileGuard(reconciler)
|
||||
for guard in (
|
||||
agent.llm_provider,
|
||||
reconciler._llm,
|
||||
writer._reconciler,
|
||||
agent.extractor,
|
||||
):
|
||||
if id(guard) not in seen_guards:
|
||||
seen_guards.add(id(guard))
|
||||
guards.append(guard)
|
||||
client._one_memory_write_guards = tuple(guards)
|
||||
|
||||
client._loop_thread.run(install())
|
||||
@@ -33,49 +33,8 @@ _COLLECTION_LOGGERS = (
|
||||
)
|
||||
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
|
||||
|
||||
# One Memory deliberately stays on ultra because its existing, user-visible
|
||||
# profile/graph flow calls digest(), which the pinned public SDK exposes only
|
||||
# for ultra clients. Reader and non-reranker behavior remain pinned for
|
||||
# compatibility with the SDK authority-migration development preview.
|
||||
HY_MEMORY_MODE = "ultra"
|
||||
# The SDK only recognises `legacy` and `hybrid` (see
|
||||
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
|
||||
# READER_HYBRID)`). The old spelling `hybrid_v2` was silently rejected and
|
||||
# fell back to legacy, so the hybrid-v2 recall strategy never ran at all.
|
||||
HY_MEMORY_READER = "hybrid"
|
||||
|
||||
HY_MEMORY_BEHAVIOR_ENV: dict[str, str] = {
|
||||
"MEMORY_MODE": HY_MEMORY_MODE,
|
||||
"MEMORY_ENABLE_GRAPH": "true",
|
||||
"MEMORY_GRAPH_RERANK_ENABLED": "false",
|
||||
"MEMORY_GRAPH_TEMPORAL_SPINE": "true",
|
||||
"MEMORY_TOPIC_SUMMARY_ENABLED": "false",
|
||||
"MEMORY_EPISODE_SUMMARY_ENABLED": "false",
|
||||
"MEMORY_EPISODE_RECALL_ENABLED": "true",
|
||||
"MEMORY_DYNAMIC_EPISODE_ENABLED": "true",
|
||||
"MEMORY_SESSION_EPISODE_ENABLED": "false",
|
||||
"MEMORY_SESSION_EPISODE_CONTEXT_ENABLED": "false",
|
||||
"MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED": "true",
|
||||
"MEMORY_EPISODE_CAPSULE_QUERY_MIN": "0.65",
|
||||
"MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN": "0.60",
|
||||
"MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX": "3",
|
||||
"MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT": "3",
|
||||
"MEMORY_ENABLE_INTENTION": "false",
|
||||
"MEMORY_ENABLE_QUERY_REWRITE": "false",
|
||||
"MEMORY_FEW_SHOT_ENABLED": "true",
|
||||
"MEMORY_HYBRID_V2_QUERY_EXPAND": "1",
|
||||
"MEMORY_HYBRID_V2_GEO_INTENT_PRIOR": "0.6",
|
||||
"MEMORY_HYBRID_V2_DIVERSITY": "1",
|
||||
"MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR": "0.55",
|
||||
"MEMORY_HYBRID_V2_TEMPORAL_BOOST": "0.18",
|
||||
"MEMORY_CROSS_DOMAIN_CON_THRESHOLD": "0.5",
|
||||
"MEMORY_READER_ENABLE_SUMMARY": "false",
|
||||
"MEMORY_SUMMARY_ENABLED_IN_SYS2": "false",
|
||||
"MEMORY_LOCAL_RERANK_ENABLED": "false",
|
||||
"RECONCILE_ENABLED": "true",
|
||||
"HY_MEMORY_READER": HY_MEMORY_READER,
|
||||
}
|
||||
|
||||
|
||||
def detailed_logging_enabled(data_dir: Path) -> bool:
|
||||
return diagnostic_log_level(data_dir) == "debug"
|
||||
@@ -168,7 +127,6 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
|
||||
log_dir = memory_root / "logs"
|
||||
memory_root.mkdir(parents=True, exist_ok=True)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
|
||||
os.environ["MEMORY_DATA_DIR"] = str(memory_root)
|
||||
os.environ["MEMORY_LOG_DIR"] = str(log_dir)
|
||||
level = diagnostic_log_level(data_dir)
|
||||
@@ -178,9 +136,3 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
|
||||
os.environ["MEMORY_MEMORY_OPERATIONS_ENABLED"] = "false"
|
||||
os.environ["ANONYMIZED_TELEMETRY"] = "False"
|
||||
os.environ.pop("CHROMA_OTEL_COLLECTION_ENDPOINT", None)
|
||||
|
||||
|
||||
def configure_hy_behavior_environment() -> None:
|
||||
"""Set import-time HY behavior flags without touching paths or secrets."""
|
||||
|
||||
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
|
||||
@@ -0,0 +1,599 @@
|
||||
"""Privileged offline access to the HY SDK data plane during App migration.
|
||||
|
||||
Runtime product code must use public ``HyMemoryClient`` methods. This module
|
||||
is the deliberately isolated exception: App startup has stopped all writers,
|
||||
so the one-time legacy migration may access SDK stores directly without
|
||||
turning migration mechanics into public Client APIs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Callable, Mapping
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from hy_memory import ( # type: ignore[import-untyped]
|
||||
DataPlaneRestore,
|
||||
DataPlaneSnapshot,
|
||||
restore_local_data_plane,
|
||||
snapshot_local_data_plane,
|
||||
)
|
||||
from hy_memory.models.memory import ( # type: ignore[import-untyped]
|
||||
MemoryLayer,
|
||||
MemoryNode,
|
||||
MemoryStatus,
|
||||
SourceType,
|
||||
)
|
||||
from hy_memory.utils.source_filter import normalize_source # type: ignore[import-untyped]
|
||||
|
||||
from one_memory.hy_sdk_types import (
|
||||
HySdkMemoryPage,
|
||||
HySdkMemoryRecord,
|
||||
HySdkMigrationError,
|
||||
HySdkRawPage,
|
||||
HySdkRawRecord,
|
||||
)
|
||||
|
||||
|
||||
class MigrationClient(Protocol):
|
||||
_loop_thread: Any
|
||||
_vector_store: Any
|
||||
_embed_service: Any
|
||||
_graph_store: Any
|
||||
|
||||
def list_memories(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None = None,
|
||||
statuses: list[str] | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> Mapping[str, object]: ...
|
||||
|
||||
def delete(self, memory_id: str) -> Mapping[str, object]: ...
|
||||
|
||||
def get_write_outcome(self, request_id: str) -> Mapping[str, object] | None: ...
|
||||
|
||||
|
||||
class HySdkMigration:
|
||||
"""Typed offline adapter over one initialized SDK data plane."""
|
||||
|
||||
def __init__(self, client: MigrationClient) -> None:
|
||||
self.client = client
|
||||
self._loop_thread = client._loop_thread
|
||||
self._vector_store = client._vector_store
|
||||
self._embed_service = client._embed_service
|
||||
self._graph_store = client._graph_store
|
||||
|
||||
async def list_memories(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None = None,
|
||||
statuses: tuple[str, ...] = ("active",),
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> HySdkMemoryPage:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_memory_list_failed",
|
||||
self.client.list_memories,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
statuses=list(statuses),
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
order=order,
|
||||
)
|
||||
return _memory_page(raw, limit=limit, offset=offset)
|
||||
|
||||
async def write_legacy_memory(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
memory_id: str,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
layer: str,
|
||||
status: str,
|
||||
owner: str | None,
|
||||
tags: tuple[str, ...],
|
||||
supersedes: tuple[str, ...],
|
||||
superseded_by: tuple[str, ...],
|
||||
is_latest: bool,
|
||||
memory_at: datetime | None,
|
||||
metadata: Mapping[str, object],
|
||||
request_id: str,
|
||||
) -> dict[str, object]:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_memory_import_failed",
|
||||
self._loop_thread.run,
|
||||
self._write_legacy_memory(
|
||||
content=content,
|
||||
memory_id=memory_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
layer=layer,
|
||||
status=status,
|
||||
owner=owner,
|
||||
tags=tags,
|
||||
supersedes=supersedes,
|
||||
superseded_by=superseded_by,
|
||||
is_latest=is_latest,
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
request_id=request_id,
|
||||
),
|
||||
)
|
||||
return _successful_result(raw, "hy_sdk_memory_import_failed")
|
||||
|
||||
async def _write_legacy_memory(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
memory_id: str,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
layer: str,
|
||||
status: str,
|
||||
owner: str | None,
|
||||
tags: tuple[str, ...],
|
||||
supersedes: tuple[str, ...],
|
||||
superseded_by: tuple[str, ...],
|
||||
is_latest: bool,
|
||||
memory_at: datetime | None,
|
||||
metadata: Mapping[str, object],
|
||||
request_id: str,
|
||||
) -> dict[str, object]:
|
||||
if not content or not memory_id or not user_id:
|
||||
raise ValueError("content, memory_id and user_id are required")
|
||||
normalized_layer = MemoryLayer.from_string(layer)
|
||||
if normalized_layer is MemoryLayer.L1_RAW:
|
||||
raise ValueError("raw content must use the raw migration lane")
|
||||
normalized_status = MemoryStatus(status.lower())
|
||||
normalized_source = normalize_source(source)
|
||||
existing = await self._vector_store.get_by_id(memory_id)
|
||||
if existing is not None:
|
||||
identity = (
|
||||
existing.content == content,
|
||||
existing.user_id == user_id,
|
||||
existing.agent_id == agent_id,
|
||||
existing.session_id == session_id,
|
||||
existing.source == normalized_source,
|
||||
existing.layer == normalized_layer,
|
||||
existing.status == normalized_status,
|
||||
existing.owner == owner,
|
||||
list(existing.tags or []) == list(tags),
|
||||
list(existing.supersedes or []) == list(supersedes),
|
||||
list(existing.superseded_by or []) == list(superseded_by),
|
||||
existing.is_latest == is_latest,
|
||||
)
|
||||
if not all(identity):
|
||||
raise ValueError("memory_id is already bound to different memory content")
|
||||
await self._project_graph(existing)
|
||||
return {
|
||||
"success": True,
|
||||
"memory_id": memory_id,
|
||||
"memory_ids": [memory_id],
|
||||
"request_id": request_id,
|
||||
"replayed": True,
|
||||
}
|
||||
custom = dict(metadata)
|
||||
custom["migration_request_id"] = request_id
|
||||
embedding = await self._embed_service.embed(content)
|
||||
node = MemoryNode(
|
||||
node_id=memory_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=normalized_source,
|
||||
layer=normalized_layer,
|
||||
content=content,
|
||||
source_type=SourceType.EXPLICIT,
|
||||
status=normalized_status,
|
||||
owner=owner,
|
||||
tags=list(tags),
|
||||
supersedes=list(supersedes),
|
||||
superseded_by=list(superseded_by),
|
||||
is_latest=is_latest,
|
||||
embedding=embedding,
|
||||
memory_at=memory_at,
|
||||
custom=custom,
|
||||
)
|
||||
stored_id = await self._vector_store.upsert(node)
|
||||
if stored_id != memory_id:
|
||||
raise RuntimeError("exact migration changed the requested memory identity")
|
||||
await self._project_graph(node)
|
||||
return {
|
||||
"success": True,
|
||||
"memory_id": memory_id,
|
||||
"memory_ids": [memory_id],
|
||||
"request_id": request_id,
|
||||
}
|
||||
|
||||
async def _project_graph(self, node: MemoryNode) -> None:
|
||||
if self._graph_store is not None:
|
||||
await self._graph_store.upsert_memory_node(node)
|
||||
|
||||
async def read_raw_inventory(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None = None,
|
||||
session_id: str | None = None,
|
||||
source: str | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> HySdkRawPage:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_raw_list_failed",
|
||||
self._loop_thread.run,
|
||||
self._read_raw_inventory(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
order=order,
|
||||
),
|
||||
)
|
||||
return _raw_page(raw, limit=limit, offset=offset)
|
||||
|
||||
async def _read_raw_inventory(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str | None,
|
||||
session_id: str | None,
|
||||
source: str | None,
|
||||
limit: int,
|
||||
offset: int,
|
||||
order: str,
|
||||
) -> dict[str, object]:
|
||||
if not user_id:
|
||||
raise ValueError("user_id is required")
|
||||
nodes = await self._vector_store.list_by_user(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
status_filter=[MemoryStatus.ACTIVE],
|
||||
layers=[MemoryLayer.L1_RAW],
|
||||
include_sources=[source] if source else None,
|
||||
limit=100_000,
|
||||
only_latest=False,
|
||||
with_vectors=False,
|
||||
allow_raw_layer=True,
|
||||
)
|
||||
if session_id:
|
||||
nodes = [node for node in nodes if node.session_id == session_id]
|
||||
nodes.sort(
|
||||
key=_node_timestamp,
|
||||
reverse=order.lower() != "asc",
|
||||
)
|
||||
return {
|
||||
"raw": [_raw_node(node) for node in nodes[offset : offset + limit]],
|
||||
"total": len(nodes),
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
async def write_legacy_raw(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
raw_id: str,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
memory_at: datetime | None,
|
||||
metadata: Mapping[str, object],
|
||||
request_id: str,
|
||||
) -> dict[str, object]:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_raw_import_failed",
|
||||
self._loop_thread.run,
|
||||
self._write_legacy_raw(
|
||||
content=content,
|
||||
raw_id=raw_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=source,
|
||||
memory_at=memory_at,
|
||||
metadata=metadata,
|
||||
request_id=request_id,
|
||||
),
|
||||
)
|
||||
return _successful_result(raw, "hy_sdk_raw_import_failed")
|
||||
|
||||
async def _write_legacy_raw(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
raw_id: str,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
source: str,
|
||||
memory_at: datetime | None,
|
||||
metadata: Mapping[str, object],
|
||||
request_id: str,
|
||||
) -> dict[str, object]:
|
||||
if not content or not raw_id or not user_id:
|
||||
raise ValueError("content, raw_id and user_id are required")
|
||||
normalized_source = normalize_source(source)
|
||||
existing = await self._vector_store.get_by_id(raw_id)
|
||||
if existing is not None:
|
||||
identity = (
|
||||
existing.layer is MemoryLayer.L1_RAW,
|
||||
existing.content == content,
|
||||
existing.user_id == user_id,
|
||||
existing.agent_id == agent_id,
|
||||
existing.session_id == session_id,
|
||||
existing.source == normalized_source,
|
||||
)
|
||||
if not all(identity):
|
||||
raise ValueError("raw_id is already bound to different source content")
|
||||
return {
|
||||
"success": True,
|
||||
"raw_id": raw_id,
|
||||
"request_id": request_id,
|
||||
"replayed": True,
|
||||
}
|
||||
custom = dict(metadata)
|
||||
custom["migration_request_id"] = request_id
|
||||
embedding = await self._embed_service.embed(content)
|
||||
node = MemoryNode(
|
||||
node_id=raw_id,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
source=normalized_source,
|
||||
layer=MemoryLayer.L1_RAW,
|
||||
content=content,
|
||||
source_type=SourceType.EXPLICIT,
|
||||
status=MemoryStatus.ACTIVE,
|
||||
is_latest=True,
|
||||
embedding=embedding,
|
||||
memory_at=memory_at,
|
||||
custom=custom,
|
||||
)
|
||||
stored_id = await self._vector_store.upsert(node)
|
||||
if stored_id != raw_id:
|
||||
raise RuntimeError("exact migration changed the requested raw identity")
|
||||
return {"success": True, "raw_id": raw_id, "request_id": request_id}
|
||||
|
||||
async def delete(self, memory_id: str) -> dict[str, object]:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_memory_delete_failed",
|
||||
self.client.delete,
|
||||
memory_id,
|
||||
)
|
||||
return _successful_result(raw, "hy_sdk_memory_delete_failed")
|
||||
|
||||
async def get_write_outcome(self, request_id: str) -> dict[str, object] | None:
|
||||
raw = await _sdk_call(
|
||||
"hy_sdk_write_outcome_failed",
|
||||
self.client.get_write_outcome,
|
||||
request_id,
|
||||
)
|
||||
if raw is None:
|
||||
return None
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_write_outcome_shape_invalid")
|
||||
return dict(raw)
|
||||
|
||||
|
||||
def snapshot_data_plane(
|
||||
*,
|
||||
source_root: Path,
|
||||
target_root: Path,
|
||||
writers_stopped: bool,
|
||||
) -> DataPlaneSnapshot:
|
||||
"""Snapshot an explicitly quiesced SDK-owned local data plane."""
|
||||
return snapshot_local_data_plane(
|
||||
source_root=source_root,
|
||||
target_root=target_root,
|
||||
writers_stopped=writers_stopped,
|
||||
)
|
||||
|
||||
|
||||
def restore_data_plane(
|
||||
*,
|
||||
snapshot_root: Path,
|
||||
target_root: Path,
|
||||
writers_stopped: bool,
|
||||
protective_root: Path | None = None,
|
||||
) -> DataPlaneRestore:
|
||||
"""Restore an explicitly quiesced SDK-owned local data plane."""
|
||||
return restore_local_data_plane(
|
||||
snapshot_root=snapshot_root,
|
||||
target_root=target_root,
|
||||
writers_stopped=writers_stopped,
|
||||
protective_root=protective_root,
|
||||
)
|
||||
|
||||
|
||||
async def _sdk_call[T](
|
||||
code: str,
|
||||
call: Callable[..., T],
|
||||
*args: object,
|
||||
**kwargs: object,
|
||||
) -> T:
|
||||
pending = asyncio.create_task(asyncio.to_thread(call, *args, **kwargs))
|
||||
try:
|
||||
return await asyncio.shield(pending)
|
||||
except asyncio.CancelledError:
|
||||
await _finish_cancelled_call(pending)
|
||||
raise
|
||||
except Exception as error:
|
||||
raise HySdkMigrationError(code) from error
|
||||
|
||||
|
||||
async def _finish_cancelled_call[T](pending: asyncio.Task[T]) -> None:
|
||||
while not pending.done():
|
||||
try:
|
||||
await asyncio.shield(pending)
|
||||
except asyncio.CancelledError:
|
||||
continue
|
||||
except Exception:
|
||||
return
|
||||
if not pending.cancelled():
|
||||
pending.exception()
|
||||
|
||||
|
||||
def _memory_page(raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
|
||||
rows: list[object] = []
|
||||
total = 0
|
||||
for bucket_name, row_name in (("vdb", "memories"), ("graph", "nodes")):
|
||||
bucket = raw.get(bucket_name)
|
||||
if bucket is None and bucket_name == "graph":
|
||||
continue
|
||||
if not isinstance(bucket, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
|
||||
bucket_rows = bucket.get(row_name)
|
||||
bucket_total = bucket.get("total")
|
||||
if not isinstance(bucket_rows, list) or type(bucket_total) is not int or bucket_total < 0:
|
||||
raise HySdkMigrationError("hy_sdk_memory_page_shape_invalid")
|
||||
rows.extend(bucket_rows)
|
||||
total += bucket_total
|
||||
|
||||
records: dict[str, HySdkMemoryRecord] = {}
|
||||
for row in rows:
|
||||
record = _memory_record(row)
|
||||
records.setdefault(record.memory_id, record)
|
||||
return HySdkMemoryPage(
|
||||
memories=tuple(records.values()),
|
||||
total=total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
def _memory_record(raw: object) -> HySdkMemoryRecord:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid")
|
||||
required = ("memory_id", "content", "layer", "status", "user_id", "agent_id")
|
||||
if any(not isinstance(raw.get(name), str) or not raw[name] for name in required):
|
||||
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid")
|
||||
custom = raw.get("custom")
|
||||
source_raw_ids: list[object] = []
|
||||
if isinstance(custom, Mapping) and isinstance(custom.get("source_raw_memory_ids"), list):
|
||||
source_raw_ids.extend(custom["source_raw_memory_ids"])
|
||||
if isinstance(raw.get("source_raw_memory_id"), str):
|
||||
source_raw_ids.append(raw["source_raw_memory_id"])
|
||||
try:
|
||||
return HySdkMemoryRecord(
|
||||
memory_id=raw["memory_id"],
|
||||
content=raw["content"],
|
||||
layer=raw["layer"],
|
||||
status=raw["status"],
|
||||
is_latest=bool(raw.get("is_latest", True)),
|
||||
supersedes=_string_tuple(raw.get("supersedes")),
|
||||
superseded_by=_string_tuple(raw.get("superseded_by")),
|
||||
tags=_string_tuple(raw.get("tags")),
|
||||
owner=raw.get("owner") if raw.get("owner") in {"user", "agent"} else None,
|
||||
source_raw_ids=_string_tuple(source_raw_ids),
|
||||
user_id=raw["user_id"],
|
||||
agent_id=raw["agent_id"],
|
||||
session_id=_optional_string(raw.get("session_id")),
|
||||
source=_optional_string(raw.get("source")),
|
||||
memory_at=_optional_int(raw.get("memory_at")),
|
||||
gmt_created=_optional_int(raw.get("gmt_created")),
|
||||
)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise HySdkMigrationError("hy_sdk_memory_record_shape_invalid") from error
|
||||
|
||||
|
||||
def _raw_page(raw: object, *, limit: int, offset: int) -> HySdkRawPage:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_raw_page_shape_invalid")
|
||||
rows = raw.get("raw")
|
||||
total = raw.get("total")
|
||||
if not isinstance(rows, list) or type(total) is not int or total < 0:
|
||||
raise HySdkMigrationError("hy_sdk_raw_page_shape_invalid")
|
||||
return HySdkRawPage(
|
||||
raw=tuple(_raw_record(row) for row in rows),
|
||||
total=total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
def _node_timestamp(node: Any) -> float:
|
||||
value = getattr(node, "memory_at", None) or getattr(node, "gmt_created", None)
|
||||
return value.timestamp() if isinstance(value, datetime) else 0.0
|
||||
|
||||
|
||||
def _raw_node(node: Any) -> dict[str, object]:
|
||||
def timestamp(value: object) -> int | None:
|
||||
return int(value.timestamp()) if isinstance(value, datetime) else None
|
||||
|
||||
return {
|
||||
"raw_id": node.node_id,
|
||||
"content": node.content,
|
||||
"user_id": node.user_id,
|
||||
"agent_id": node.agent_id,
|
||||
"session_id": node.session_id,
|
||||
"source": node.source,
|
||||
"memory_at": timestamp(node.memory_at),
|
||||
"gmt_created": timestamp(node.gmt_created),
|
||||
"custom": node.custom or {},
|
||||
}
|
||||
|
||||
|
||||
def _raw_record(raw: object) -> HySdkRawRecord:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid")
|
||||
required = ("raw_id", "content", "user_id", "agent_id", "session_id", "source")
|
||||
if any(not isinstance(raw.get(name), str) or not raw[name] for name in required):
|
||||
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid")
|
||||
custom = raw.get("custom")
|
||||
try:
|
||||
return HySdkRawRecord(
|
||||
raw_id=raw["raw_id"],
|
||||
content=raw["content"],
|
||||
user_id=raw["user_id"],
|
||||
agent_id=raw["agent_id"],
|
||||
session_id=raw["session_id"],
|
||||
source=raw["source"],
|
||||
memory_at=_optional_int(raw.get("memory_at")),
|
||||
gmt_created=_optional_int(raw.get("gmt_created")),
|
||||
custom=dict(custom) if isinstance(custom, Mapping) else {},
|
||||
)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid") from error
|
||||
|
||||
|
||||
def _successful_result(raw: object, code: str) -> dict[str, object]:
|
||||
if not isinstance(raw, Mapping) or raw.get("success") is not True:
|
||||
raise HySdkMigrationError(code)
|
||||
return dict(raw)
|
||||
|
||||
|
||||
def _string_tuple(value: object) -> tuple[str, ...]:
|
||||
if not isinstance(value, list | tuple):
|
||||
return ()
|
||||
return tuple(item for item in value if isinstance(item, str) and item)
|
||||
|
||||
|
||||
def _optional_string(value: object) -> str | None:
|
||||
return value if isinstance(value, str) else None
|
||||
|
||||
|
||||
def _optional_int(value: object) -> int | None:
|
||||
return value if type(value) is int else None
|
||||
@@ -0,0 +1,406 @@
|
||||
"""HY SDK client construction and shared-runtime leasing.
|
||||
|
||||
This module deliberately depends only on the SDK's package-root exports. A
|
||||
lease owns one client while the process shares one ``SharedRuntime`` per
|
||||
physical MemHub data directory.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import threading
|
||||
from collections.abc import Callable, Coroutine
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from hy_memory import ( # type: ignore[import-untyped]
|
||||
HyMemoryClient,
|
||||
MemoryConfig,
|
||||
SharedRuntime,
|
||||
)
|
||||
|
||||
from one_memory.embedding_assets import (
|
||||
BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
BUILTIN_EMBEDDING_MODEL,
|
||||
)
|
||||
from one_memory.hy_graph_admin import (
|
||||
GraphWidthMismatchError,
|
||||
graph_store_enabled,
|
||||
inspect_graph_width_mismatch,
|
||||
)
|
||||
from one_memory.hy_sdk_types import HySdkRuntimeError
|
||||
from one_memory.local_embedding import LocalEmbedder
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
_RUNTIME_LOCK = threading.RLock()
|
||||
BUILTIN_RETRIEVAL_PROFILE_ID = "builtin-retrieval"
|
||||
_LOCAL_ONNX_PLACEHOLDER_URL = "http://127.0.0.1/local-onnx"
|
||||
DEFAULT_RELEVANCE_FLOOR = 0.35
|
||||
|
||||
|
||||
@dataclass
|
||||
class _RuntimeEntry:
|
||||
runtime: SharedRuntime
|
||||
references: int
|
||||
|
||||
|
||||
_RUNTIMES: dict[Path, _RuntimeEntry] = {}
|
||||
|
||||
|
||||
class LocalOnnxEmbeddingService:
|
||||
"""Public ``EmbeddingService`` implementation backed by MemHub's ONNX model."""
|
||||
|
||||
_embedder: ClassVar[LocalEmbedder | None] = None
|
||||
_embedder_lock: ClassVar[threading.Lock] = threading.Lock()
|
||||
|
||||
def __init__(self, embedder: LocalEmbedder | None = None) -> None:
|
||||
if embedder is None:
|
||||
with self._embedder_lock:
|
||||
if self._embedder is None:
|
||||
self.__class__._embedder = LocalEmbedder()
|
||||
embedder = self._embedder
|
||||
if embedder is None: # pragma: no cover - guarded by the assignment above
|
||||
raise HySdkRuntimeError("hy_sdk_local_embedding_unavailable")
|
||||
self._local_embedder = embedder
|
||||
|
||||
async def embed(self, text: str, use_cache: bool = True) -> list[float]:
|
||||
del use_cache
|
||||
vectors = await asyncio.to_thread(self._local_embedder.encode, [text])
|
||||
return vectors[0]
|
||||
|
||||
async def embed_with_meta(
|
||||
self,
|
||||
text: str,
|
||||
use_cache: bool = True,
|
||||
) -> tuple[list[float], dict[str, Any]]:
|
||||
return await self.embed(text, use_cache), {"provider": "local-onnx"}
|
||||
|
||||
async def embed_batch(
|
||||
self,
|
||||
texts: list[str],
|
||||
use_cache: bool = True,
|
||||
) -> list[list[float]]:
|
||||
del use_cache
|
||||
if not texts:
|
||||
return []
|
||||
return await asyncio.to_thread(self._local_embedder.encode, list(texts))
|
||||
|
||||
async def embed_queued(self, text: str, use_cache: bool = True) -> list[float]:
|
||||
return await self.embed(text, use_cache)
|
||||
|
||||
async def embed_queued_with_meta(
|
||||
self,
|
||||
text: str,
|
||||
use_cache: bool = True,
|
||||
) -> tuple[list[float], dict[str, Any]]:
|
||||
return await self.embed_with_meta(text, use_cache)
|
||||
|
||||
def get_embedding_dims(self) -> int:
|
||||
return BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
|
||||
class ClientLease:
|
||||
"""Own a client and release its shared runtime exactly once."""
|
||||
|
||||
def __init__(self, client: HyMemoryClient, data_dir: Path) -> None:
|
||||
self.client = client
|
||||
self.data_dir = data_dir.resolve()
|
||||
self._closed = False
|
||||
self._close_lock = threading.Lock()
|
||||
|
||||
@property
|
||||
def closed(self) -> bool:
|
||||
return self._closed
|
||||
|
||||
def close(self) -> None:
|
||||
with self._close_lock:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
try:
|
||||
self.client.close()
|
||||
finally:
|
||||
_release_runtime(self.data_dir)
|
||||
|
||||
def __enter__(self) -> ClientLease:
|
||||
return self
|
||||
|
||||
def __exit__(self, *_exc: object) -> None:
|
||||
self.close()
|
||||
|
||||
|
||||
class SdkClientHandle:
|
||||
"""Lazy async owner for one profile-bound public SDK client lease."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> None:
|
||||
self._profile = profile
|
||||
self._api_key = api_key
|
||||
self._data_dir = data_dir
|
||||
self._http_transport_factory = http_transport_factory
|
||||
self._lease: ClientLease | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def client(self) -> HyMemoryClient:
|
||||
if self._lease is not None:
|
||||
return self._lease.client
|
||||
async with self._lock:
|
||||
if self._lease is None:
|
||||
self._lease = await asyncio.to_thread(
|
||||
acquire_client,
|
||||
self._profile,
|
||||
api_key=self._api_key,
|
||||
data_dir=self._data_dir,
|
||||
http_transport_factory=self._http_transport_factory,
|
||||
)
|
||||
return self._lease.client
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._lock:
|
||||
lease, self._lease = self._lease, None
|
||||
if lease is not None:
|
||||
await asyncio.to_thread(lease.close)
|
||||
|
||||
|
||||
def builtin_retrieval_snapshot() -> ModelProfileSnapshot:
|
||||
"""Return the model-free profile used by SDK-native read operations."""
|
||||
return ModelProfileSnapshot(
|
||||
profile_id=BUILTIN_RETRIEVAL_PROFILE_ID,
|
||||
revision=0,
|
||||
provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
base_url=_LOCAL_ONNX_PLACEHOLDER_URL,
|
||||
model=BUILTIN_EMBEDDING_MODEL,
|
||||
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
embedding_model=BUILTIN_EMBEDDING_MODEL,
|
||||
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
timeout_seconds=60,
|
||||
max_concurrency=1,
|
||||
context_window=None,
|
||||
has_api_key=False,
|
||||
)
|
||||
|
||||
|
||||
def sdk_relevance_floor() -> float:
|
||||
raw = os.environ.get("ONE_MEMORY_HY_RELEVANCE_FLOOR", "").strip()
|
||||
if not raw:
|
||||
return DEFAULT_RELEVANCE_FLOOR
|
||||
try:
|
||||
value = float(raw)
|
||||
except ValueError:
|
||||
return DEFAULT_RELEVANCE_FLOOR
|
||||
return min(1.0, max(0.0, value))
|
||||
|
||||
|
||||
def build_sdk_config(
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
) -> MemoryConfig:
|
||||
"""Map a model snapshot to SDK models, endpoints, backends, and local paths."""
|
||||
generation_endpoint, generation_key = _generation_access(profile, api_key)
|
||||
embedding_provider = profile.embedding_provider_kind or profile.provider_kind
|
||||
embedding_service_is_local = embedding_provider is ProviderKind.LOCAL_ONNX
|
||||
embedding_model = profile.embedding_model or (
|
||||
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else None
|
||||
)
|
||||
if not embedding_model:
|
||||
raise HySdkRuntimeError("hy_sdk_embedding_model_required")
|
||||
dimensions: int | None
|
||||
if embedding_service_is_local:
|
||||
embedding_endpoint = None
|
||||
embedding_key = None
|
||||
dimensions = profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
|
||||
else:
|
||||
embedding_endpoint = _openai_endpoint(
|
||||
embedding_provider,
|
||||
profile.embedding_base_url or profile.base_url,
|
||||
)
|
||||
embedding_key = "ollama" if embedding_provider is ProviderKind.OLLAMA else api_key
|
||||
if embedding_provider is ProviderKind.OPENAI_COMPATIBLE and not embedding_key:
|
||||
raise HySdkRuntimeError("hy_sdk_embedding_api_key_missing")
|
||||
if embedding_provider not in {
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
ProviderKind.OLLAMA,
|
||||
}:
|
||||
raise HySdkRuntimeError("hy_sdk_embedding_provider_unsupported")
|
||||
dimensions = profile.embedding_dimensions
|
||||
|
||||
root = data_dir.resolve() / "hy-memory" / "governed-v2"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
collection_identity = "\0".join(
|
||||
(
|
||||
embedding_provider.value,
|
||||
embedding_endpoint or "local",
|
||||
embedding_model,
|
||||
str(dimensions or "auto"),
|
||||
)
|
||||
)
|
||||
collection = "memhub_" + hashlib.sha256(collection_identity.encode()).hexdigest()[:16]
|
||||
|
||||
return MemoryConfig.from_dict(
|
||||
{
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"model": profile.model,
|
||||
"api_key": generation_key,
|
||||
"base_url": generation_endpoint,
|
||||
},
|
||||
"embedder": {
|
||||
"provider": "openai",
|
||||
"model": (
|
||||
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else embedding_model
|
||||
),
|
||||
"api_key": embedding_key,
|
||||
"base_url": embedding_endpoint,
|
||||
"embedding_dims": dimensions,
|
||||
},
|
||||
"vector_store": {
|
||||
"provider": "chroma",
|
||||
"collection_name": collection,
|
||||
"persist_directory": str(root / "vector"),
|
||||
"embedding_dims": dimensions,
|
||||
"host": None,
|
||||
"port": None,
|
||||
"api_key": None,
|
||||
},
|
||||
"graph_store": {
|
||||
"provider": "kuzu",
|
||||
"db_path": str(root / "graph"),
|
||||
"url": None,
|
||||
"username": None,
|
||||
"password": None,
|
||||
"database": None,
|
||||
},
|
||||
"cache": {
|
||||
"backend": "sqlite",
|
||||
"db_path": str(root / "cache.sqlite3"),
|
||||
},
|
||||
"history": {"db_path": str(root / "history.sqlite3")},
|
||||
"coding": {"db_path": str(root / "coding.sqlite3")},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def acquire_client(
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> ClientLease:
|
||||
"""Construct a public SDK client and return its explicit runtime lease."""
|
||||
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
|
||||
config.http_transport_factory = http_transport_factory
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is not None:
|
||||
raise GraphWidthMismatchError(mismatch)
|
||||
key = data_dir.resolve()
|
||||
runtime = _acquire_runtime(config, key)
|
||||
embedding_service: LocalOnnxEmbeddingService | None = None
|
||||
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
|
||||
embedding_service = LocalOnnxEmbeddingService()
|
||||
try:
|
||||
client = HyMemoryClient(
|
||||
config=config,
|
||||
mode="ultra" if graph_store_enabled() else "pro",
|
||||
runtime=runtime,
|
||||
embedding_service=embedding_service,
|
||||
)
|
||||
except BaseException:
|
||||
_release_runtime(key)
|
||||
raise
|
||||
return ClientLease(client, key)
|
||||
|
||||
|
||||
def close_all_shared_runtimes() -> None:
|
||||
"""Close process runtimes only after every client lease has been released."""
|
||||
with _RUNTIME_LOCK:
|
||||
if _RUNTIMES:
|
||||
raise HySdkRuntimeError("hy_sdk_runtime_leases_active")
|
||||
|
||||
|
||||
def _generation_access(
|
||||
profile: ModelProfileSnapshot,
|
||||
api_key: str | None,
|
||||
) -> tuple[str, str]:
|
||||
if profile.provider_kind is ProviderKind.LOCAL_ONNX:
|
||||
return _LOCAL_ONNX_PLACEHOLDER_URL, "local-read-only"
|
||||
if profile.provider_kind not in {
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
ProviderKind.OLLAMA,
|
||||
}:
|
||||
raise HySdkRuntimeError("hy_sdk_generation_provider_unsupported")
|
||||
endpoint = _openai_endpoint(profile.provider_kind, profile.base_url)
|
||||
if profile.provider_kind is ProviderKind.OLLAMA:
|
||||
return endpoint, "ollama"
|
||||
if not api_key:
|
||||
raise HySdkRuntimeError("hy_sdk_model_api_key_missing")
|
||||
return endpoint, api_key
|
||||
|
||||
|
||||
def _openai_endpoint(provider: ProviderKind, endpoint: str | None) -> str:
|
||||
value = (endpoint or "").rstrip("/")
|
||||
if not value:
|
||||
raise HySdkRuntimeError("hy_sdk_model_endpoint_missing")
|
||||
if provider is ProviderKind.OLLAMA and not value.endswith("/v1"):
|
||||
return f"{value}/v1"
|
||||
return value
|
||||
|
||||
|
||||
def _acquire_runtime(config: MemoryConfig, key: Path) -> SharedRuntime:
|
||||
with _RUNTIME_LOCK:
|
||||
entry = _RUNTIMES.get(key)
|
||||
if entry is None:
|
||||
runtime = _run_sync(SharedRuntime.create(config))
|
||||
_RUNTIMES[key] = _RuntimeEntry(runtime=runtime, references=1)
|
||||
return runtime
|
||||
entry.references += 1
|
||||
return entry.runtime
|
||||
|
||||
|
||||
def _release_runtime(key: Path) -> None:
|
||||
runtime: SharedRuntime | None = None
|
||||
with _RUNTIME_LOCK:
|
||||
entry = _RUNTIMES.get(key)
|
||||
if entry is None:
|
||||
return
|
||||
entry.references -= 1
|
||||
if entry.references == 0:
|
||||
runtime = entry.runtime
|
||||
del _RUNTIMES[key]
|
||||
if runtime is not None:
|
||||
runtime.close()
|
||||
|
||||
|
||||
def _run_sync[T](coroutine: Coroutine[Any, Any, T]) -> T:
|
||||
"""Run SDK async construction from either sync or async caller threads."""
|
||||
try:
|
||||
asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return asyncio.run(coroutine)
|
||||
|
||||
result: list[T] = []
|
||||
failure: list[BaseException] = []
|
||||
|
||||
def run() -> None:
|
||||
try:
|
||||
result.append(asyncio.run(coroutine))
|
||||
except BaseException as error:
|
||||
failure.append(error)
|
||||
|
||||
thread = threading.Thread(target=run, name="hy-sdk-runtime-create")
|
||||
thread.start()
|
||||
thread.join()
|
||||
if failure:
|
||||
raise failure[0]
|
||||
return result[0]
|
||||
@@ -0,0 +1,402 @@
|
||||
"""Validated records and response parsers for public HY SDK operations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
|
||||
|
||||
class HySdkError(RuntimeError):
|
||||
"""Base error carrying a stable, product-facing code."""
|
||||
|
||||
def __init__(self, code: str) -> None:
|
||||
super().__init__(code)
|
||||
self.code = code
|
||||
|
||||
|
||||
class HySdkRuntimeError(HySdkError):
|
||||
"""The public SDK runtime could not be configured or leased."""
|
||||
|
||||
|
||||
class HySdkMigrationError(HySdkError):
|
||||
"""The privileged offline SDK data-plane migration failed."""
|
||||
|
||||
|
||||
class HySdkReadError(HySdkError):
|
||||
"""A public SDK read operation failed or returned an invalid shape."""
|
||||
|
||||
|
||||
class HySdkMemoryRecord(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
memory_id: str
|
||||
content: str
|
||||
layer: str
|
||||
status: str
|
||||
is_latest: bool
|
||||
supersedes: tuple[str, ...] = ()
|
||||
superseded_by: tuple[str, ...] = ()
|
||||
tags: tuple[str, ...] = ()
|
||||
owner: str | None = None
|
||||
source_raw_ids: tuple[str, ...] = ()
|
||||
user_id: str
|
||||
agent_id: str
|
||||
session_id: str | None = None
|
||||
source: str | None = None
|
||||
memory_at: int | None = None
|
||||
gmt_created: int | None = None
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object) -> HySdkMemoryRecord:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_record_shape_unknown")
|
||||
if any(
|
||||
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
|
||||
for key in ("memory_id", "content")
|
||||
):
|
||||
raise HySdkReadError("hy_memory_record_shape_unknown")
|
||||
try:
|
||||
return cls(
|
||||
memory_id=str(raw["memory_id"]),
|
||||
content=str(raw["content"]),
|
||||
layer=str(raw.get("layer") or ""),
|
||||
status=str(raw.get("status") or ""),
|
||||
is_latest=bool(raw.get("is_latest", True)),
|
||||
supersedes=_string_tuple(raw.get("supersedes")),
|
||||
superseded_by=_string_tuple(raw.get("superseded_by")),
|
||||
tags=_string_tuple(raw.get("tags")),
|
||||
owner=(str(raw["owner"]) if raw.get("owner") in {"user", "agent"} else None),
|
||||
source_raw_ids=_source_raw_ids(raw),
|
||||
user_id=str(raw.get("user_id") or ""),
|
||||
agent_id=str(raw.get("agent_id") or ""),
|
||||
session_id=(
|
||||
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
|
||||
),
|
||||
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
|
||||
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
|
||||
gmt_created=(
|
||||
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
|
||||
),
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as error:
|
||||
raise HySdkReadError("hy_memory_record_shape_unknown") from error
|
||||
|
||||
|
||||
class HySdkMemoryPage(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
memories: tuple[HySdkMemoryRecord, ...]
|
||||
total: int = Field(ge=0)
|
||||
limit: int = Field(ge=1)
|
||||
offset: int = Field(ge=0)
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_list_shape_unknown")
|
||||
rows: list[object] = []
|
||||
total = 0
|
||||
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
|
||||
bucket = raw.get(bucket_name)
|
||||
if bucket is None and bucket_name == "graph":
|
||||
continue
|
||||
if not isinstance(bucket, Mapping) or not isinstance(bucket.get(rows_name), list):
|
||||
raise HySdkReadError("hy_memory_list_shape_unknown")
|
||||
bucket_total = bucket.get("total")
|
||||
if (
|
||||
not isinstance(bucket_total, int)
|
||||
or isinstance(bucket_total, bool)
|
||||
or bucket_total < 0
|
||||
):
|
||||
raise HySdkReadError("hy_memory_list_shape_unknown")
|
||||
total += bucket_total
|
||||
rows.extend(bucket[rows_name])
|
||||
deduplicated: dict[str, HySdkMemoryRecord] = {}
|
||||
for row in rows:
|
||||
record = HySdkMemoryRecord.from_sdk(row)
|
||||
deduplicated.setdefault(record.memory_id, record)
|
||||
return cls(
|
||||
memories=tuple(deduplicated.values()),
|
||||
total=total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
class HySdkRawRecord(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
raw_id: str
|
||||
content: str
|
||||
user_id: str
|
||||
agent_id: str
|
||||
session_id: str
|
||||
source: str
|
||||
memory_at: int | None = None
|
||||
gmt_created: int | None = None
|
||||
custom: dict[str, object] = Field(default_factory=dict)
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object) -> HySdkRawRecord:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_raw_shape_unknown")
|
||||
if any(
|
||||
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
|
||||
for key in ("raw_id", "content", "user_id", "agent_id", "session_id", "source")
|
||||
):
|
||||
raise HySdkReadError("hy_memory_raw_shape_unknown")
|
||||
custom = raw.get("custom")
|
||||
if not isinstance(custom, Mapping):
|
||||
custom = {}
|
||||
try:
|
||||
return cls(
|
||||
raw_id=str(raw["raw_id"]),
|
||||
content=str(raw["content"]),
|
||||
user_id=str(raw["user_id"]),
|
||||
agent_id=str(raw["agent_id"]),
|
||||
session_id=str(raw["session_id"]),
|
||||
source=str(raw["source"]),
|
||||
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
|
||||
gmt_created=(
|
||||
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
|
||||
),
|
||||
custom=dict(custom),
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as error:
|
||||
raise HySdkReadError("hy_memory_raw_shape_unknown") from error
|
||||
|
||||
|
||||
class HySdkRawHit(HySdkRawRecord):
|
||||
score: float = Field(ge=0, le=1)
|
||||
|
||||
@classmethod
|
||||
def from_sdk_search(cls, raw: object) -> HySdkRawHit:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
|
||||
raw_id = raw.get("memory_id")
|
||||
score = raw.get("score")
|
||||
if (
|
||||
not isinstance(raw_id, str)
|
||||
or not isinstance(score, int | float)
|
||||
or isinstance(score, bool)
|
||||
):
|
||||
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
|
||||
base = HySdkRawRecord.from_sdk({**raw, "raw_id": raw_id})
|
||||
return cls(
|
||||
**base.model_dump(),
|
||||
score=max(0.0, min(1.0, float(score))),
|
||||
)
|
||||
|
||||
|
||||
class HySdkRawPage(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
raw: tuple[HySdkRawRecord, ...]
|
||||
total: int = Field(ge=0)
|
||||
limit: int = Field(ge=1)
|
||||
offset: int = Field(ge=0)
|
||||
|
||||
@classmethod
|
||||
def from_memory_list(
|
||||
cls,
|
||||
raw: object,
|
||||
*,
|
||||
limit: int,
|
||||
offset: int,
|
||||
) -> HySdkRawPage:
|
||||
"""Parse ``list_memories(layers=['l1_raw'])`` into the App raw DTO."""
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
|
||||
bucket = raw.get("vdb")
|
||||
if not isinstance(bucket, Mapping):
|
||||
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
|
||||
rows = bucket.get("memories")
|
||||
total = bucket.get("total")
|
||||
if (
|
||||
not isinstance(rows, list)
|
||||
or not isinstance(total, int)
|
||||
or isinstance(total, bool)
|
||||
or total < 0
|
||||
):
|
||||
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
|
||||
converted = []
|
||||
for row in rows:
|
||||
if not isinstance(row, Mapping) or row.get("layer") != "l1_raw":
|
||||
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
|
||||
converted.append({**row, "raw_id": row.get("memory_id")})
|
||||
return cls(
|
||||
raw=tuple(HySdkRawRecord.from_sdk(row) for row in converted),
|
||||
total=total,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
class HySdkUserProfile(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
request_id: str
|
||||
profile: dict[str, object] | None
|
||||
elapsed_ms: float = Field(ge=0)
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object) -> HySdkUserProfile:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_profile_shape_unknown")
|
||||
request_id = raw.get("request_id")
|
||||
profile = raw.get("profile")
|
||||
elapsed_ms = raw.get("elapsed_ms")
|
||||
if (
|
||||
not isinstance(request_id, str)
|
||||
or (profile is not None and not isinstance(profile, Mapping))
|
||||
or not isinstance(elapsed_ms, int | float)
|
||||
or isinstance(elapsed_ms, bool)
|
||||
):
|
||||
raise HySdkReadError("hy_memory_profile_shape_unknown")
|
||||
return cls(
|
||||
request_id=request_id,
|
||||
profile=None if profile is None else dict(profile),
|
||||
elapsed_ms=float(elapsed_ms),
|
||||
)
|
||||
|
||||
|
||||
def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
|
||||
rows = _reply_rows(raw, error_code)
|
||||
found: list[SemanticMemoryHit] = []
|
||||
seen: set[str] = set()
|
||||
for item in sorted(rows, key=_score, reverse=True):
|
||||
memory_id = item.get("memory_id") or item.get("id")
|
||||
if not isinstance(memory_id, str) or not memory_id or memory_id in seen:
|
||||
continue
|
||||
seen.add(memory_id)
|
||||
content = item.get("content")
|
||||
member_ids, formatting_complete = _formatted_member_ids(item)
|
||||
found.append(
|
||||
SemanticMemoryHit(
|
||||
memory_id=memory_id,
|
||||
score=_score(item),
|
||||
formatted_content=(
|
||||
content.strip() if isinstance(content, str) and content.strip() else None
|
||||
),
|
||||
member_ids=member_ids,
|
||||
sdk_formatted=True,
|
||||
formatting_complete=formatting_complete,
|
||||
)
|
||||
)
|
||||
return tuple(found)
|
||||
|
||||
|
||||
def parse_sdk_raw_search(raw: object) -> tuple[HySdkRawHit, ...]:
|
||||
return tuple(
|
||||
HySdkRawHit.from_sdk_search(row)
|
||||
for row in _reply_rows(raw, "hy_memory_raw_search_shape_unknown")
|
||||
)
|
||||
|
||||
|
||||
def _string_tuple(value: object) -> tuple[str, ...]:
|
||||
if not isinstance(value, list):
|
||||
return ()
|
||||
return tuple(
|
||||
dict.fromkeys(item.strip() for item in value if isinstance(item, str) and item.strip())
|
||||
)
|
||||
|
||||
|
||||
def _source_raw_ids(raw: Mapping[str, object]) -> tuple[str, ...]:
|
||||
custom = raw.get("custom")
|
||||
values: list[object] = []
|
||||
if isinstance(custom, Mapping):
|
||||
listed = custom.get("source_raw_memory_ids")
|
||||
if isinstance(listed, list):
|
||||
values.extend(listed)
|
||||
singular = raw.get("source_raw_memory_id")
|
||||
if isinstance(singular, str):
|
||||
values.append(singular)
|
||||
return tuple(
|
||||
dict.fromkeys(value.strip() for value in values if isinstance(value, str) and value.strip())
|
||||
)
|
||||
|
||||
|
||||
def _reply_rows(raw: object, error_code: str) -> tuple[Mapping[str, object], ...]:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError(error_code)
|
||||
keys = tuple(key for key in ("memories", "coding_memories") if key in raw)
|
||||
if "scene" in raw:
|
||||
scene = raw["scene"]
|
||||
if not isinstance(scene, str):
|
||||
raise HySdkReadError(error_code)
|
||||
if scene in ("normal", "chat"):
|
||||
selected = "memories"
|
||||
elif scene in ("productivity", "working"):
|
||||
selected = "coding_memories"
|
||||
else:
|
||||
raise HySdkReadError(error_code)
|
||||
if selected not in keys:
|
||||
raise HySdkReadError(error_code)
|
||||
elif len(keys) == 1:
|
||||
selected = keys[0]
|
||||
else:
|
||||
raise HySdkReadError(error_code)
|
||||
|
||||
result: list[Mapping[str, object]] = []
|
||||
for key in keys:
|
||||
value = raw[key]
|
||||
if key == "memories" and isinstance(value, Mapping):
|
||||
if set(value) != {"schema", "proactive", "normal"}:
|
||||
raise HySdkReadError(error_code)
|
||||
channels = tuple(value.values())
|
||||
else:
|
||||
channels = (value,)
|
||||
for channel in channels:
|
||||
if not isinstance(channel, list) or any(
|
||||
not isinstance(row, Mapping) for row in channel
|
||||
):
|
||||
raise HySdkReadError(error_code)
|
||||
if key == selected:
|
||||
result.extend(channel)
|
||||
return tuple(result)
|
||||
|
||||
|
||||
def _score(memory: Mapping[str, object]) -> float:
|
||||
score = memory.get("score")
|
||||
return float(score) if isinstance(score, int | float) and not isinstance(score, bool) else 0.0
|
||||
|
||||
|
||||
def _formatted_member_ids(memory: Mapping[str, object]) -> tuple[tuple[str, ...], bool]:
|
||||
represented: dict[str, None] = {}
|
||||
composite = False
|
||||
raw_members = memory.get("member_ids")
|
||||
if raw_members is not None:
|
||||
composite = True
|
||||
if not isinstance(raw_members, list | tuple) or any(
|
||||
not isinstance(member_id, str) or not member_id for member_id in raw_members
|
||||
):
|
||||
return (), False
|
||||
for member_id in raw_members:
|
||||
represented.setdefault(member_id, None)
|
||||
chain = memory.get("evolution_chain")
|
||||
if chain is not None:
|
||||
composite = True
|
||||
if not isinstance(chain, list | tuple):
|
||||
return (), False
|
||||
for entry in chain:
|
||||
if not isinstance(entry, Mapping):
|
||||
return (), False
|
||||
member_id = entry.get("node_id") or entry.get("memory_id")
|
||||
if not isinstance(member_id, str) or not member_id:
|
||||
return (), False
|
||||
represented.setdefault(member_id, None)
|
||||
if memory.get("capsule_memories") is not None or memory.get("timeline") is not None:
|
||||
composite = True
|
||||
if raw_members is None:
|
||||
return (), False
|
||||
content = memory.get("content")
|
||||
if isinstance(content, str) and content.lstrip().startswith(
|
||||
("Related memories:", "Matched memories:", "Memory timeline:")
|
||||
):
|
||||
composite = True
|
||||
if not represented:
|
||||
return (), False
|
||||
return tuple(represented), not composite or bool(represented)
|
||||
@@ -1,153 +0,0 @@
|
||||
"""Per-write original Evidence and where each HY memory came from; never a memory source."""
|
||||
|
||||
import json
|
||||
from collections.abc import Set as AbstractSet
|
||||
from importlib import import_module
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
|
||||
|
||||
class HySupportedMemory(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
hy_memory_id: str
|
||||
governed_memory_id: str
|
||||
revision: int
|
||||
content: str
|
||||
owner: Literal["user", "agent"]
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
class HySourceContext(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
current: tuple[EvidenceRecord, ...]
|
||||
history: tuple[EvidenceRecord, ...] = ()
|
||||
supporting: tuple[EvidenceRecord, ...] = ()
|
||||
known_memories: tuple[HySupportedMemory, ...] = ()
|
||||
|
||||
|
||||
class HyWriteBindings:
|
||||
def __init__(self, context: HySourceContext) -> None:
|
||||
if not context.current:
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
self.context = context
|
||||
seed = context.current[0]
|
||||
records = (*context.current, *context.history, *context.supporting)
|
||||
workspaces = {record.workspace for record in records}
|
||||
# A missing workspace is a legacy identity, not a wildcard that may
|
||||
# absorb a named project. Keep NULL and named workspaces disjoint.
|
||||
if None in workspaces and len(workspaces) != 1:
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
ids = [record.evidence_id for record in records]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
for record in records:
|
||||
if (
|
||||
(
|
||||
record.user_id,
|
||||
record.source_id,
|
||||
record.conversation_id,
|
||||
record.workspace,
|
||||
)
|
||||
!= (
|
||||
seed.user_id,
|
||||
seed.source_id,
|
||||
seed.conversation_id,
|
||||
seed.workspace,
|
||||
)
|
||||
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
|
||||
or record.role not in {"user", "assistant", "tool"}
|
||||
):
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
self.all_sources: list[dict[str, Any]] = [
|
||||
{
|
||||
"id": record.evidence_id,
|
||||
"role": record.role,
|
||||
"content": sanitize_model_input(record.verbatim),
|
||||
"new": index < len(context.current),
|
||||
"revision": record.revision,
|
||||
"created_at": record.created_at.isoformat() if record.created_at else None,
|
||||
}
|
||||
for index, record in enumerate(records)
|
||||
]
|
||||
self.sources = self.all_sources[: len(context.current) + len(context.history)]
|
||||
self._supports: dict[tuple[str, str], tuple[str, ...]] = {}
|
||||
self.profile_order: list[str] | None = None
|
||||
self.verified_operations: dict[str, tuple[str, str]] = {}
|
||||
self.known = {memory.hy_memory_id: memory for memory in context.known_memories}
|
||||
for memory in context.known_memories:
|
||||
if not memory.evidence_ids or not set(memory.evidence_ids) <= set(ids):
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
self._supports[(memory.content, memory.owner)] = memory.evidence_ids
|
||||
|
||||
def check_content(self, content: str) -> None:
|
||||
try:
|
||||
parsed = json.loads(content)
|
||||
except ValueError as error:
|
||||
raise ValueError("hy_memory_write_verification_input_mismatch") from error
|
||||
expected = [
|
||||
{"role": record.role, "content": sanitize_model_input(record.verbatim)}
|
||||
for record in self.context.current
|
||||
]
|
||||
# HY strips tool messages itself before chat extraction when it judges a turn chat.
|
||||
without_tools = [item for item in expected if item["role"] != "tool"]
|
||||
if parsed != expected and (without_tools == expected or parsed != without_tools):
|
||||
raise ValueError("hy_memory_write_verification_input_mismatch")
|
||||
|
||||
def history_json(self) -> str:
|
||||
return json.dumps(
|
||||
[
|
||||
{
|
||||
"evidence_id": source["id"],
|
||||
**{k: source[k] for k in ("role", "content", "revision", "created_at")},
|
||||
}
|
||||
for source in self.sources
|
||||
if not source["new"]
|
||||
],
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
def turn_support(
|
||||
self, owner: str | None, targets: AbstractSet[str] = frozenset()
|
||||
) -> tuple[str, ...]:
|
||||
"""This turn's words by the memory's author, plus what a merged memory already cited."""
|
||||
roles = {"user": {"user"}, "agent": {"assistant"}}.get(owner or "")
|
||||
current = [source for source in self.all_sources if source["new"]]
|
||||
ids = [source["id"] for source in current if roles is None or source["role"] in roles]
|
||||
if not ids:
|
||||
ids = [source["id"] for source in current]
|
||||
for target in sorted(targets):
|
||||
if target in self.known:
|
||||
ids.extend(self.known[target].evidence_ids)
|
||||
return tuple(dict.fromkeys(ids))
|
||||
|
||||
def record_turn_supports(self, items: list[dict[str, Any]]) -> None:
|
||||
for item in items:
|
||||
key = (item["content"], item["owner"] or "")
|
||||
self._supports.setdefault(key, self.turn_support(item["owner"]))
|
||||
|
||||
def record_operation_support(self, content: str, owner: str | None, targets: set[str]) -> None:
|
||||
key = (content, owner or "")
|
||||
merged = (*self._supports.get(key, ()), *self.turn_support(owner, targets))
|
||||
self._supports[key] = tuple(dict.fromkeys(merged))
|
||||
|
||||
def support_for(self, content: str, owner: str | None) -> tuple[str, ...]:
|
||||
return self._supports.get((content, owner or ""), ())
|
||||
|
||||
def profile_support(self, content: str, values: dict[str, Any]) -> tuple[str, ...]:
|
||||
render = import_module("hy_memory.agent.tools.basic_profile")._render_content
|
||||
|
||||
if not values or render(values, field_order=self.profile_order) != content:
|
||||
return ()
|
||||
ids: dict[str, None] = {}
|
||||
for key, value in values.items():
|
||||
support = self.support_for(f"{key}: {value}", "user")
|
||||
if not support:
|
||||
return ()
|
||||
ids.update(dict.fromkeys(support))
|
||||
return tuple(ids)
|
||||
@@ -1,53 +0,0 @@
|
||||
"""Input checks around HY's native extractor; HY's output is published as it returns it."""
|
||||
|
||||
import asyncio
|
||||
from typing import Any
|
||||
|
||||
from one_memory.hy_source_context import HyWriteBindings
|
||||
|
||||
|
||||
def _extracted_items(info: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
for section in ("memory", "intentions"):
|
||||
for item in info.get(section) or []:
|
||||
if isinstance(item, dict) and isinstance(item.get("content"), str):
|
||||
items.append({"content": item["content"], "owner": item.get("owner")})
|
||||
for name, value in (info.get("basic_info") or {}).items():
|
||||
items.append({"content": f"{name}: {value}", "owner": "user"})
|
||||
return items
|
||||
|
||||
|
||||
class HyExtractionGuard:
|
||||
def __init__(self, extractor: Any) -> None:
|
||||
self.extractor = extractor
|
||||
self.failures: list[str] = []
|
||||
self._lock = asyncio.Lock()
|
||||
self.bindings: HyWriteBindings | None = None
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.extractor, name)
|
||||
|
||||
async def extract(self, *args: Any, **kwargs: Any) -> Any:
|
||||
async with self._lock:
|
||||
try:
|
||||
return await self._extract(args, kwargs)
|
||||
except ValueError as error:
|
||||
if str(error).startswith("hy_memory_write_verification"):
|
||||
self.failures.append(str(error))
|
||||
raise
|
||||
|
||||
async def _extract(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
|
||||
content = kwargs.get("content", args[0] if args else "")
|
||||
bindings = self.bindings
|
||||
if bindings is not None:
|
||||
bindings.check_content(content)
|
||||
fields = kwargs.get("basic_profile_fields")
|
||||
bindings.profile_order = list(fields) if fields else None
|
||||
kwargs = {**kwargs, "history_context": bindings.history_json()}
|
||||
else:
|
||||
# No raw-history identity was supplied; do not trust HY's text prefixes.
|
||||
kwargs = {**kwargs, "history_context": ""}
|
||||
result = await self.extractor.extract(*args, **kwargs)
|
||||
if bindings is not None and result.success:
|
||||
bindings.record_turn_supports(_extracted_items(result.info))
|
||||
return result
|
||||
@@ -6,6 +6,7 @@ from pathlib import PurePosixPath
|
||||
|
||||
from pydantic import JsonValue
|
||||
|
||||
from one_memory.derived_echo import strip_injected_memory
|
||||
from one_memory.imports.access import (
|
||||
AuthorizedSourceFile,
|
||||
AuthorizedSourceRoot,
|
||||
@@ -213,6 +214,7 @@ def parse_hermes_rollout_record(
|
||||
message.get("content"),
|
||||
allowed_block_types=frozenset({"text", "input_text", "output_text"}),
|
||||
)
|
||||
text = strip_injected_memory(text)
|
||||
if not text:
|
||||
return ParsedSkip(
|
||||
reason_code=ScanIssueCode.NO_TEXT,
|
||||
@@ -386,7 +388,8 @@ def _read_state_db(
|
||||
# The assistant's call sits on its own row as `tool_calls`; the
|
||||
# `tool` rows after it hold only what came back.
|
||||
calls = _tool_calls_text(row["tool_calls"]) if role == "assistant" else None
|
||||
has_text = bool(str(row["content"] or "").strip())
|
||||
content = strip_injected_memory(str(row["content"] or ""))
|
||||
has_text = bool(content.strip())
|
||||
if role not in {"user", "assistant"} or (not has_text and calls is None):
|
||||
yield ScanSkip(
|
||||
cursor=cursor,
|
||||
@@ -401,7 +404,7 @@ def _read_state_db(
|
||||
role=str(role) if has_text else "tool",
|
||||
written_by=_written_by(role, row["session_source"]) if has_text else WrittenBy.TOOL,
|
||||
created_at=parse_timestamp(row["timestamp"]),
|
||||
verbatim=str(row["content"]) if has_text else str(calls),
|
||||
verbatim=content if has_text else str(calls),
|
||||
workspace=workspace,
|
||||
project=safe_project_label(workspace),
|
||||
record_type="state_db_message" if has_text else "state_db_message.tool",
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Passive lifecycle projection used only by retired Governance readers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from types import MappingProxyType
|
||||
|
||||
|
||||
@dataclass(frozen=True, eq=False)
|
||||
class HyLifecycle:
|
||||
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
|
||||
|
||||
def __eq__(self, other: object) -> bool:
|
||||
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(
|
||||
other.hidden
|
||||
)
|
||||
|
||||
def __hash__(self) -> int:
|
||||
return hash(tuple(sorted(self.hidden.items())))
|
||||
|
||||
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
|
||||
return self.hidden.get(memory_id, ())
|
||||
|
||||
def replacing(self, memory_id: str) -> tuple[str, ...]:
|
||||
return tuple(
|
||||
sorted(
|
||||
old
|
||||
for old, replacements in self.hidden.items()
|
||||
if memory_id in replacements
|
||||
)
|
||||
)
|
||||
@@ -224,7 +224,7 @@ def record_capture_rejection(
|
||||
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
|
||||
if not changed or not _sync_open(store.database_path, record.source_id):
|
||||
return
|
||||
from one_memory.memory_processing import _conversation_chunks
|
||||
from one_memory.conversation_processing import _conversation_chunks
|
||||
from one_memory.memory_server.durable_queue import DurableWriteQueue
|
||||
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
|
||||
|
||||
|
||||
@@ -121,7 +121,7 @@ class LocalEmbedder:
|
||||
raise MissingEmbeddingAssetsError(f"{_FETCH_HINT} Looked in {directory}.")
|
||||
|
||||
# Imported by name for the same reason kuzu and hy_memory are in
|
||||
# hy_memory_bridge: onnxruntime ships no py.typed, and a dynamic
|
||||
# onnxruntime ships no py.typed, and a dynamic
|
||||
# import keeps the strict checker on everything around it.
|
||||
ort = import_module("onnxruntime")
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
@@ -22,7 +22,7 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.database import enable_write_ahead_log
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_lifecycle import HyLifecycle
|
||||
from one_memory.legacy_lifecycle_types import HyLifecycle
|
||||
from one_memory.library_upgrade import LibraryUpgradeError, PreparedLibrary, holds_legacy_rows
|
||||
from one_memory.memory_server.migrations import governance_ddl
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import Literal
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_lifecycle import HyLifecycle
|
||||
from one_memory.legacy_lifecycle_types import HyLifecycle
|
||||
from one_memory.memory_evidence_freshness import register_memory_freshness
|
||||
from one_memory.memory_governance import (
|
||||
GovernedMemory,
|
||||
|
||||
@@ -3,7 +3,7 @@ import json
|
||||
import sqlite3
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal, cast
|
||||
from typing import Any, Literal, cast
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
|
||||
@@ -21,11 +21,10 @@ from one_memory.imports.models import (
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
from one_memory.memory_governance import MemoryStatus
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
|
||||
from one_memory.memory_server.durable_queue import DurableWriteQueue
|
||||
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
|
||||
from one_memory.secrets import SecretStoreUnavailableError
|
||||
|
||||
SOURCE_ID = "one-memory-manual"
|
||||
@@ -82,11 +81,23 @@ class SavedMemoryNoteEvidence(BaseModel):
|
||||
verbatim: str
|
||||
|
||||
|
||||
class LegacyMemoryNoteReader:
|
||||
"""Read pre-cutover note receipts without retaining any write runtime."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
tasks: MemoryTaskStore,
|
||||
governance: MemoryGovernanceStore,
|
||||
) -> None:
|
||||
self.tasks = tasks
|
||||
self.governance = governance
|
||||
|
||||
|
||||
class MemoryNoteStore:
|
||||
def __init__(
|
||||
self,
|
||||
evidence: EvidenceStore,
|
||||
processor: MemoryProcessingSupervisor,
|
||||
processor: Any | None,
|
||||
intake: UnifiedWriteIntake | None = None,
|
||||
*,
|
||||
sdk_authority: bool = False,
|
||||
@@ -252,6 +263,8 @@ class MemoryNoteStore:
|
||||
memory_ids=(list(durable.sdk_memory_ids) if durable is not None else []),
|
||||
error=durable.detail if durable is not None else "write_missing",
|
||||
)
|
||||
if self.processor is None:
|
||||
raise RuntimeError("legacy_memory_note_processor_unavailable")
|
||||
task = self.processor.tasks.task_for_evidence(evidence_id)
|
||||
memory_ids = []
|
||||
if task:
|
||||
@@ -356,6 +369,8 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
try:
|
||||
if notes.sdk_authority:
|
||||
return await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
if notes.processor is None:
|
||||
raise RuntimeError("legacy_memory_note_processor_unavailable")
|
||||
if notes.intake is None:
|
||||
# Isolated tests construct the store without a queue.
|
||||
await notes.processor.enqueue_conversation_off_thread(record)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -69,8 +69,8 @@ _TYPE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$")
|
||||
def safe_error_code(error: BaseException) -> str:
|
||||
"""Reduce an exception to a storable code, never its message text.
|
||||
|
||||
A recognised code (the message *is* the code, as `HyMemoryBridgeError`
|
||||
and the permission guard raise them) is kept so callers can classify it.
|
||||
A recognised code (the message *is* a stable domain code) is kept so
|
||||
callers can classify it.
|
||||
Everything else collapses to its exception type, which is a fixed
|
||||
identifier rather than content.
|
||||
"""
|
||||
@@ -239,8 +239,8 @@ PERMISSION_REQUIRED_CODES = frozenset(
|
||||
}
|
||||
)
|
||||
|
||||
#: Model-side codes `process_records` raises *before* it can open the bridge
|
||||
#: (no active profile, missing credential, secure storage down). They prove
|
||||
#: Model-side codes the SDK write coordinator raises before it can lease a
|
||||
#: client (no active profile, missing credential, secure storage down). They prove
|
||||
#: nothing was sent, so the write parks instead of being lost. Transport codes
|
||||
#: that can arrive after a request was in flight stay `unknown`.
|
||||
MODEL_NOT_READY_CODES = frozenset(
|
||||
@@ -251,9 +251,9 @@ MODEL_NOT_READY_CODES = frozenset(
|
||||
}
|
||||
)
|
||||
|
||||
#: Deterministic, pre-bridge refusals: the frozen input no longer matches
|
||||
#: Deterministic, pre-SDK refusals: the frozen input no longer matches
|
||||
#: current Evidence, or a turn exceeds the model's input window. Both are
|
||||
#: raised while validating the records, before any bridge or HY call, so
|
||||
#: raised while validating the records, before any SDK call, so
|
||||
#: nothing was sent and retrying the same intent changes nothing — a terminal
|
||||
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
|
||||
#: write) and not a retry (which would fail identically).
|
||||
@@ -358,12 +358,10 @@ class UnknownWriteJob:
|
||||
|
||||
`request_id` is the durable queue's idempotency key. It identifies the row
|
||||
for queue inspection and is the value `DurableWriteQueue.get()` accepts.
|
||||
It is **not** the HY request id: `MemoryProcessingSupervisor` derives the
|
||||
HY handle from the task it creates, so an inquiry implementation must
|
||||
obtain that task-layer handle separately rather than passing this value to
|
||||
HY. This is discovery only: it does not decide whether HY has the write,
|
||||
and it deliberately exposes no transition method that could turn
|
||||
`unknown` back into retryable work.
|
||||
It is the public SDK request id used for exact outcome inquiry. This is
|
||||
discovery only: it does not decide whether HY has the write, and it
|
||||
deliberately exposes no transition method that could turn `unknown` back
|
||||
into retryable work.
|
||||
"""
|
||||
|
||||
job_id: str
|
||||
@@ -1759,8 +1757,8 @@ class DurableWriteQueue:
|
||||
"""Retire an `unknown` job after an inquiry, never by guessing.
|
||||
|
||||
`unknown` means HY may already hold the write; the only safe way out is
|
||||
to ask by request id (`hy_capabilities` measured that the SDK supports
|
||||
this) and then record what the inquiry found. `found=True` closes it as
|
||||
to call the public SDK `get_write_outcome(request_id)` and then record
|
||||
what the inquiry found. `found=True` closes it as
|
||||
`completed`; `found=False` gives up and marks it `dead`. The state
|
||||
machine's legal transitions are the authority, so this refuses any
|
||||
change that `job_state.next_state` would reject — in particular it can
|
||||
@@ -2258,12 +2256,11 @@ async def execute_job(
|
||||
admit_hy_invocation: Callable[[str, Callable[[], Awaitable[bool]]], Awaitable[bool]]
|
||||
| None = None,
|
||||
) -> None:
|
||||
"""Run one durable job through the existing authorized processing path.
|
||||
"""Run one durable job through the SDK-authority processor.
|
||||
|
||||
The executor does not call HY directly: `process_records` owns the
|
||||
permission guard, the runner and governance. `hy_invoked` is recorded
|
||||
before that call, so a crash mid-write is later reclaimed to `unknown`
|
||||
instead of being retried blindly.
|
||||
``process_sdk_job`` owns the one public SDK write. ``hy_invoked`` is
|
||||
recorded immediately before that call, so a crash mid-write is later
|
||||
reclaimed to ``unknown`` instead of being retried blindly.
|
||||
"""
|
||||
# Every store call here is synchronous SQLite work on the single writer or
|
||||
# a reader; made from the loop it would block health checks and the API for
|
||||
@@ -2336,38 +2333,13 @@ async def execute_job(
|
||||
try:
|
||||
level = ServiceLevel.REALTIME if request.priority == "live" else ServiceLevel.BACKGROUND
|
||||
with service_level_context(level):
|
||||
authority = getattr(supervisor, "sdk_authority", None)
|
||||
if callable(authority) and authority():
|
||||
sdk_result = await supervisor.process_sdk_job(
|
||||
records,
|
||||
request_id=request.idempotency_key,
|
||||
before_hy_invocation=mark_hy_invocation,
|
||||
)
|
||||
raw_ids = sdk_result.raw_ids
|
||||
memory_ids = tuple(memory.memory_id for memory in sdk_result.memories)
|
||||
else:
|
||||
completed_tasks = (
|
||||
await supervisor.process_records(
|
||||
records,
|
||||
before_hy_invocation=mark_hy_invocation,
|
||||
source_admitted=True,
|
||||
)
|
||||
or ()
|
||||
)
|
||||
receipts = []
|
||||
for task in completed_tasks:
|
||||
receipt = await in_the_background(supervisor.tasks.sdk_receipt, task.task_id)
|
||||
if receipt is None:
|
||||
raise RuntimeError("sdk_write_receipt_missing")
|
||||
receipts.append(receipt)
|
||||
raw_ids = tuple(
|
||||
dict.fromkeys(raw_id for receipt in receipts for raw_id in receipt.raw_ids)
|
||||
)
|
||||
memory_ids = tuple(
|
||||
dict.fromkeys(
|
||||
memory_id for receipt in receipts for memory_id in receipt.memory_ids
|
||||
)
|
||||
)
|
||||
sdk_result = await supervisor.process_sdk_job(
|
||||
records,
|
||||
request_id=request.idempotency_key,
|
||||
before_hy_invocation=mark_hy_invocation,
|
||||
)
|
||||
raw_ids = sdk_result.raw_ids
|
||||
memory_ids = sdk_result.memory_ids
|
||||
except Exception as error: # noqa: BLE001 - classified below
|
||||
# Never persist the raw message: it may be a provider response or the
|
||||
# user's own text, and `last_error` is shown over HTTP as `detail`.
|
||||
@@ -2382,7 +2354,7 @@ async def execute_job(
|
||||
):
|
||||
await in_the_background(queue.release_unstarted, job_id, owner=owner)
|
||||
return
|
||||
# The task layer classifies these before any bridge is opened, so HY
|
||||
# The write coordinator classifies these before any SDK client is leased, so HY
|
||||
# was provably not called. They must not read as `failed`:
|
||||
# - authorization stop -> cancelled (the user turned it off);
|
||||
# - no permission yet -> waiting_for_permission (parked, re-admitted
|
||||
@@ -2445,13 +2417,10 @@ async def execute_job(
|
||||
sdk_raw_ids=raw_ids,
|
||||
sdk_memory_ids=memory_ids,
|
||||
)
|
||||
authority = getattr(supervisor, "sdk_authority", None)
|
||||
release = getattr(evidence, "release_sdk_staging", None)
|
||||
durable_receipt = await in_the_background(queue.sdk_receipt, job_id)
|
||||
if (
|
||||
callable(authority)
|
||||
and authority()
|
||||
and callable(release)
|
||||
callable(release)
|
||||
and durable_receipt is not None
|
||||
and (durable_receipt.raw_ids or durable_receipt.memory_ids)
|
||||
):
|
||||
|
||||
@@ -1,182 +0,0 @@
|
||||
"""A0-03: measured answer to "can a HY write be made idempotent?".
|
||||
|
||||
The result decides whether the `unknown` job state can ever be retired, so it
|
||||
is probed against the installed SDK rather than written down from memory.
|
||||
|
||||
What is verified against `hy-memory==2.0.5.dev3`:
|
||||
|
||||
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
|
||||
atomically claims it in the SDK cache before invoking a write. Replays return
|
||||
the durable terminal result; running/unknown claims are not invoked again.
|
||||
- `HyMemoryClient.get_write_outcome(request_id)` reads that durable outcome
|
||||
directly; it does not depend on optional pipeline tracing.
|
||||
|
||||
A first pass at this probe got both answers wrong, and the failure mode is
|
||||
worth keeping in mind: searching SDK source for the string "idempotent" hits a
|
||||
docstring about *episode* nodes ("calls are idempotent") and a "dedup" comment
|
||||
about search over-sampling. Neither is write idempotency. So the probe checks
|
||||
the `add` path specifically and names the inspector class explicitly, instead
|
||||
of grepping for a word that appears in unrelated prose.
|
||||
|
||||
Net: MemHub can query and settle an uncertain request without blind replay.
|
||||
`unknown` remains a legitimate operational state while the SDK claim itself is
|
||||
non-terminal, but the same request id cannot write twice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import inspect
|
||||
from enum import StrEnum
|
||||
from typing import Any
|
||||
|
||||
|
||||
class HyIdempotencySupport(StrEnum):
|
||||
"""Whether the SDK can dedupe a replayed write."""
|
||||
|
||||
#: A caller key is honoured and a replay is rejected or folded together.
|
||||
KEY = "key"
|
||||
#: Same content is folded by the engine, without an explicit key.
|
||||
CONTENT_HASH = "content_hash"
|
||||
#: A replay writes a second time. Retrying after a crash duplicates.
|
||||
NONE = "none"
|
||||
#: Never probed. Must not survive A0-03.
|
||||
UNKNOWN = "unknown"
|
||||
|
||||
|
||||
class HyResultLookupSupport(StrEnum):
|
||||
"""Whether the outcome of a past write can be queried afterwards."""
|
||||
|
||||
BY_REQUEST_ID = "by_request_id"
|
||||
BY_MEMORY_ID_ONLY = "by_memory_id_only"
|
||||
NONE = "none"
|
||||
UNKNOWN = "unknown"
|
||||
|
||||
|
||||
def _module(name: str) -> Any | None:
|
||||
try:
|
||||
return importlib.import_module(name)
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
|
||||
def _has_callable(obj: Any, name: str) -> bool:
|
||||
return obj is not None and callable(getattr(obj, name, None))
|
||||
|
||||
|
||||
def hy_write_accepts_request_id() -> bool:
|
||||
"""True when the write path carries a caller-supplied request id."""
|
||||
client_mod = _module("hy_memory.client")
|
||||
if client_mod is None:
|
||||
return False
|
||||
client = getattr(client_mod, "HyMemoryClient", None)
|
||||
if client is None:
|
||||
return False
|
||||
add = getattr(client, "add", None)
|
||||
if add is None:
|
||||
return False
|
||||
return "request_id" in inspect.signature(add).parameters
|
||||
|
||||
|
||||
def _add_path_declares_idempotency() -> bool:
|
||||
"""Whether the *write* path de-duplicates, judged on its own source.
|
||||
|
||||
Deliberately scoped to `add`/`async_add`. A whole-module search matches
|
||||
unrelated prose: "calls are idempotent" describes episode-node upserts, and
|
||||
a "dedup" comment describes search over-sampling. Both are about other
|
||||
features and would make this probe answer `key` for the wrong reason.
|
||||
"""
|
||||
client_mod = _module("hy_memory.client")
|
||||
if client_mod is None:
|
||||
return False
|
||||
client = getattr(client_mod, "HyMemoryClient", None)
|
||||
if client is None:
|
||||
return False
|
||||
sources: list[str] = []
|
||||
for name in ("add", "async_add"):
|
||||
func = getattr(client, name, None)
|
||||
if func is None:
|
||||
continue
|
||||
try:
|
||||
sources.append(inspect.getsource(func).lower())
|
||||
except (OSError, TypeError):
|
||||
continue
|
||||
if not sources:
|
||||
return False
|
||||
if _has_callable(client, "get_write_outcome"):
|
||||
claim = getattr(client, "_claim_write_outcome", None)
|
||||
if claim is not None:
|
||||
return True
|
||||
# Older SDKs may expose an explicit key under one of these names.
|
||||
markers = ("idempotency_key", "idempotent_key", "dedupe_key", "dedup_key")
|
||||
return any(marker in source for source in sources for marker in markers)
|
||||
|
||||
|
||||
def _inspector_can_lookup_by_request_id() -> bool:
|
||||
"""True when pipeline logs can be queried by the write's request id."""
|
||||
inspector = _module("hy_memory.inspector")
|
||||
if inspector is None:
|
||||
return False
|
||||
for cls_name in ("MemoryInspector", "Inspector", "PipelineInspector"):
|
||||
cls = getattr(inspector, cls_name, None)
|
||||
if cls is None:
|
||||
continue
|
||||
for method in ("get_pipeline_logs", "async_get_pipeline_logs"):
|
||||
func = getattr(cls, method, None)
|
||||
if func is None:
|
||||
continue
|
||||
# The parameter is `request_id` on `MemoryInspector`; older
|
||||
# docstrings call the same thing `n`. Accept either so a rename
|
||||
# does not silently downgrade this to "no lookup available".
|
||||
if {"request_id", "n"} & set(inspect.signature(func).parameters):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _probe_idempotency() -> HyIdempotencySupport:
|
||||
if not hy_write_accepts_request_id():
|
||||
# No handle at all: not even a correlation id to investigate later.
|
||||
return HyIdempotencySupport.NONE
|
||||
if _add_path_declares_idempotency():
|
||||
return HyIdempotencySupport.KEY
|
||||
# A request_id is present but nothing de-duplicates on it.
|
||||
return HyIdempotencySupport.NONE
|
||||
|
||||
|
||||
def _probe_result_lookup() -> HyResultLookupSupport:
|
||||
client_mod = _module("hy_memory.client")
|
||||
client = getattr(client_mod, "HyMemoryClient", None) if client_mod is not None else None
|
||||
if _has_callable(client, "get_write_outcome"):
|
||||
return HyResultLookupSupport.BY_REQUEST_ID
|
||||
if _inspector_can_lookup_by_request_id():
|
||||
return HyResultLookupSupport.BY_REQUEST_ID
|
||||
if _has_callable(client, "get"):
|
||||
return HyResultLookupSupport.BY_MEMORY_ID_ONLY
|
||||
return HyResultLookupSupport.NONE
|
||||
|
||||
|
||||
IDEMPOTENCY_SUPPORT: HyIdempotencySupport = _probe_idempotency()
|
||||
RESULT_LOOKUP_SUPPORT: HyResultLookupSupport = _probe_result_lookup()
|
||||
|
||||
|
||||
def probe_hy_write_idempotency() -> dict[str, Any]:
|
||||
"""Full probe report, for STATUS evidence and for debugging a bumped SDK.
|
||||
|
||||
Re-probed on every call so a dependency upgrade that changes the answer is
|
||||
visible, rather than being frozen at import time.
|
||||
"""
|
||||
client_mod = _module("hy_memory.client")
|
||||
client = getattr(client_mod, "HyMemoryClient", None)
|
||||
add = getattr(client, "add", None)
|
||||
return {
|
||||
"sdk_importable": client_mod is not None,
|
||||
"add_signature": str(inspect.signature(add)) if add is not None else "",
|
||||
"accepts_request_id": hy_write_accepts_request_id(),
|
||||
"add_declares_idempotency": _add_path_declares_idempotency(),
|
||||
"durable_write_outcome": (
|
||||
client is not None and _has_callable(client, "get_write_outcome")
|
||||
),
|
||||
"inspector_lookup_by_request_id": _inspector_can_lookup_by_request_id(),
|
||||
"idempotency": _probe_idempotency().value,
|
||||
"result_lookup": _probe_result_lookup().value,
|
||||
}
|
||||
@@ -9,9 +9,8 @@ Evidence→queue crash window stayed open.
|
||||
This adapter is deliberately small. It turns one committed Evidence record
|
||||
into the one-record complete-turn request, submits it once under an evidence
|
||||
derived idempotency key, and returns the durable record. The existing
|
||||
``DurableWriteDrain`` remains the only consumer; it calls
|
||||
``process_records``, which preserves the complete-turn/debounce/settle rules
|
||||
in ``MemoryProcessingSupervisor``.
|
||||
``DurableWriteDrain`` remains the only consumer; it calls the SDK write
|
||||
coordinator once and persists only the public receipt ids.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -477,7 +477,7 @@ class MemoryTaskStore:
|
||||
#: Statuses whose input snapshot nothing reads back afterwards.
|
||||
#:
|
||||
#: Deliberately narrow. A `completed` task's inputs are read after the fact:
|
||||
#: `memory_processing` compares them against the evidence actually sent to
|
||||
#: SDK write processing compares them against the evidence actually sent to
|
||||
#: HY to prove coverage, and `matters/discovery` reads them to report how
|
||||
#: many records HY really saw. Clearing those would silently turn a
|
||||
#: verification into a pass and a report into a lie, which is far worse
|
||||
|
||||
@@ -10,7 +10,6 @@ from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
from uuid import uuid4
|
||||
@@ -46,14 +45,7 @@ def audit_precipitation_request(request_id: str) -> Iterator[None]:
|
||||
|
||||
|
||||
def _current_prompt_stage() -> str:
|
||||
try:
|
||||
resolver = getattr( # noqa: B009 - optional SDK version boundary
|
||||
import_module("hy_memory.utils.model_call_context"),
|
||||
"current_model_call_stage",
|
||||
)
|
||||
except (ImportError, AttributeError):
|
||||
return "other"
|
||||
return str(resolver())
|
||||
return "other"
|
||||
|
||||
|
||||
class ModelRequestAuditStore:
|
||||
|
||||
@@ -12,7 +12,7 @@ from typing import Any, Literal, Protocol
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.hy_memory_bridge import HySdkMemoryRecord
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord
|
||||
from one_memory.memory_governance import GovernedMemory, MemoryStatus
|
||||
from one_memory.semantic_deadline import (
|
||||
SemanticMemoryHit,
|
||||
|
||||
@@ -2,10 +2,10 @@
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from one_memory.conversation_processing import _is_hy_input_turn
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
|
||||
from one_memory.imports.sources.parsing import is_host_generated_user_turn, strip_host_wrappers
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
from one_memory.memory_processing import _is_hy_input_turn
|
||||
from one_memory.retrieval.chunking import TextChunk
|
||||
from one_memory.retrieval.mapped_content import PreparedMappedContent, prepare_mapped_content
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ import json
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Literal
|
||||
|
||||
from one_memory.conversation_processing import _is_hy_input_turn
|
||||
from one_memory.embedding_assets import BUILTIN_EMBEDDING_MAX_TOKENS
|
||||
from one_memory.imports.models import (
|
||||
MAX_VERBATIM_CHARACTERS,
|
||||
@@ -17,7 +18,6 @@ from one_memory.imports.sources.parsing import (
|
||||
)
|
||||
from one_memory.input_sanitization import sanitize_model_input_with_spans
|
||||
from one_memory.local_embedding import TokenizedText
|
||||
from one_memory.memory_processing import _is_hy_input_turn
|
||||
from one_memory.retrieval.chunking import ChunkingError, OffsetTokenizer, TextChunk, chunk_evidence
|
||||
|
||||
MAP_VERSION = "raw-content-map-v1"
|
||||
|
||||
@@ -6,13 +6,15 @@ import asyncio
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
from contextlib import aclosing
|
||||
from typing import Literal
|
||||
from typing import Any, Literal, Protocol
|
||||
|
||||
import httpx
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from one_memory.http_client import create_async_client
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridge, HySdkMemoryRecord
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_READER
|
||||
from one_memory.hy_sdk_runtime import sdk_relevance_floor
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_search
|
||||
from one_memory.memory_qa import (
|
||||
MemoryAnswerError,
|
||||
_DeferredSecretStore,
|
||||
@@ -36,6 +38,10 @@ _GROUNDING_LIMIT = 6
|
||||
type _Recheck = Callable[[], Awaitable[None]]
|
||||
|
||||
|
||||
class SdkClientHandleLike(Protocol):
|
||||
async def client(self) -> Any: ...
|
||||
|
||||
|
||||
class SdkAskMemory(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
@@ -69,7 +75,7 @@ class SdkMemoryAnswerer:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
bridge: HyMemoryBridge,
|
||||
client_handle: SdkClientHandleLike,
|
||||
profiles: ModelProfileStore,
|
||||
secrets: SecretStore,
|
||||
authorize_generation: Callable[
|
||||
@@ -78,7 +84,7 @@ class SdkMemoryAnswerer:
|
||||
| None = None,
|
||||
transport: httpx.AsyncBaseTransport | None = None,
|
||||
) -> None:
|
||||
self.bridge = bridge
|
||||
self.client_handle = client_handle
|
||||
self.profiles = profiles
|
||||
self.secrets = secrets
|
||||
self.authorize_generation = authorize_generation
|
||||
@@ -90,18 +96,27 @@ class SdkMemoryAnswerer:
|
||||
*,
|
||||
user_id: str = "local",
|
||||
) -> tuple[SdkAskMemory, ...]:
|
||||
hits = await self.bridge.search_memories(
|
||||
client = await self.client_handle.client()
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
question,
|
||||
user_id=user_id,
|
||||
agent_id="*",
|
||||
user_ids=[user_id],
|
||||
agent_ids=[],
|
||||
limit=12,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=12,
|
||||
intention_limit=12,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
found: list[SdkAskMemory] = []
|
||||
seen: set[str] = set()
|
||||
for hit in hits:
|
||||
if hit.memory_id in seen:
|
||||
continue
|
||||
record = await self.bridge.get_sdk_memory(hit.memory_id)
|
||||
record = await _get_memory(client, hit.memory_id)
|
||||
if record is None or record.status != "active" or not record.is_latest:
|
||||
continue
|
||||
seen.add(record.memory_id)
|
||||
@@ -211,8 +226,9 @@ class SdkMemoryAnswerer:
|
||||
self,
|
||||
memories: tuple[SdkAskMemory, ...],
|
||||
) -> None:
|
||||
client = await self.client_handle.client()
|
||||
for memory in memories:
|
||||
current = await self.bridge.get_sdk_memory(memory.memory_id)
|
||||
current = await _get_memory(client, memory.memory_id)
|
||||
if current is None or _wire_memory(current) != memory:
|
||||
raise MemoryAnswerError("memory_answer_context_changed")
|
||||
|
||||
@@ -340,6 +356,11 @@ def _wire_memory(record: HySdkMemoryRecord) -> SdkAskMemory:
|
||||
)
|
||||
|
||||
|
||||
async def _get_memory(client: Any, memory_id: str) -> HySdkMemoryRecord | None:
|
||||
raw = await asyncio.to_thread(client.get, memory_id)
|
||||
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
|
||||
|
||||
|
||||
def _summary(memories: tuple[SdkAskMemory, ...]) -> str:
|
||||
statements = [
|
||||
memory.content.replace("用户", "你", 1).replace("The user", "You", 1) for memory in memories
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
"""Narrow SDK-native runtime for graph, retrieval, and queue status operations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.extraction_settings import ExtractionSettings, ExtractionSettingsStore
|
||||
from one_memory.hy_graph_admin import (
|
||||
GraphWidthDecision,
|
||||
GraphWidthMismatch,
|
||||
GraphWidthOutcome,
|
||||
decide_profile_graph_width,
|
||||
inspect_profile_graph_width,
|
||||
)
|
||||
from one_memory.hy_graph_projection import HyGraphSnapshot, read_graph_snapshot
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_READER
|
||||
from one_memory.hy_sdk_runtime import SdkClientHandle, sdk_relevance_floor
|
||||
from one_memory.hy_sdk_types import HySdkError, parse_sdk_search
|
||||
from one_memory.memory_queue import ExtractionWindowPreset, QueueDepths
|
||||
from one_memory.memory_server.durable_queue import COMPLETED, DurableWriteQueue
|
||||
from one_memory.sdk_write_processor import SdkWriteCoordinator
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
|
||||
|
||||
class SdkMemoryRuntime:
|
||||
"""Product services that remain after legacy task/governance writes retire."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
coordinator: SdkWriteCoordinator,
|
||||
read_handle: SdkClientHandle,
|
||||
write_queue: DurableWriteQueue,
|
||||
extraction_settings: ExtractionSettingsStore,
|
||||
data_dir: Path,
|
||||
) -> None:
|
||||
self.coordinator = coordinator
|
||||
self.read_handle = read_handle
|
||||
self.write_queue = write_queue
|
||||
self.extraction_settings = extraction_settings
|
||||
self.data_dir = data_dir
|
||||
|
||||
async def semantic_recall(
|
||||
self,
|
||||
query: str,
|
||||
*,
|
||||
limit: int = 12,
|
||||
) -> tuple[SemanticMemoryHit, ...]:
|
||||
client = await self.read_handle.client()
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
query,
|
||||
user_ids=["local"],
|
||||
agent_ids=[],
|
||||
limit=limit,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=limit,
|
||||
intention_limit=limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
return parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
|
||||
async def semantic_search(
|
||||
self,
|
||||
query: str,
|
||||
*,
|
||||
limit: int = 12,
|
||||
) -> tuple[tuple[str, float], ...]:
|
||||
return tuple(
|
||||
(hit.memory_id, hit.score)
|
||||
for hit in await self.semantic_recall(query, limit=limit)
|
||||
)
|
||||
|
||||
async def graph_snapshot(self, *, limit: int = 300) -> HyGraphSnapshot:
|
||||
try:
|
||||
await self.coordinator.active_profile_credentials()
|
||||
except RuntimeError as error:
|
||||
if str(error) == "model_profile_not_active":
|
||||
return HyGraphSnapshot(consulted=False)
|
||||
raise
|
||||
try:
|
||||
client = await self.coordinator.client()
|
||||
except Exception as error:
|
||||
raise HySdkError("hy_memory_graph_read_failed") from error
|
||||
return await asyncio.to_thread(
|
||||
read_graph_snapshot,
|
||||
client,
|
||||
"local",
|
||||
"*",
|
||||
limit,
|
||||
)
|
||||
|
||||
async def digest(self) -> dict[str, object]:
|
||||
client = await self.coordinator.client()
|
||||
try:
|
||||
raw = await asyncio.to_thread(
|
||||
client.list_memories,
|
||||
user_id="local",
|
||||
limit=1_000,
|
||||
offset=0,
|
||||
)
|
||||
agent_ids = _agent_ids(raw)
|
||||
digests = [
|
||||
{
|
||||
"agent_id": agent_id,
|
||||
"schemas_recorded": 0,
|
||||
**dict(
|
||||
await asyncio.to_thread(
|
||||
client.evolve,
|
||||
user_id="local",
|
||||
agent_id=agent_id,
|
||||
)
|
||||
),
|
||||
}
|
||||
for agent_id in agent_ids
|
||||
]
|
||||
except Exception as error:
|
||||
raise HySdkError("hy_memory_digest_failed") from error
|
||||
return {
|
||||
"success": all(result.get("success") is not False for result in digests),
|
||||
"scope_count": len(digests),
|
||||
"skipped_conversation_scopes": 0,
|
||||
"schemas_recorded": 0,
|
||||
"digests": digests,
|
||||
}
|
||||
|
||||
async def graph_width_mismatch(self) -> GraphWidthMismatch | None:
|
||||
try:
|
||||
profile, secret = await self.coordinator.active_profile_credentials()
|
||||
except RuntimeError as error:
|
||||
if str(error) == "model_profile_not_active":
|
||||
return None
|
||||
raise
|
||||
return await asyncio.to_thread(
|
||||
inspect_profile_graph_width,
|
||||
profile,
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
)
|
||||
|
||||
async def decide_graph_width(
|
||||
self,
|
||||
decision: GraphWidthDecision,
|
||||
) -> GraphWidthOutcome:
|
||||
try:
|
||||
profile, secret = await self.coordinator.active_profile_credentials()
|
||||
except RuntimeError as error:
|
||||
raise HySdkError(str(error)) from error
|
||||
return await asyncio.to_thread(
|
||||
decide_profile_graph_width,
|
||||
profile,
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
decision=decision,
|
||||
)
|
||||
|
||||
async def update_hy_memory(self, memory_id: str, content: str) -> dict[str, object]:
|
||||
client = await self.coordinator.client()
|
||||
try:
|
||||
result = await asyncio.to_thread(client.update, memory_id, content)
|
||||
except Exception as error:
|
||||
raise HySdkError("hy_memory_update_failed") from error
|
||||
if not isinstance(result, dict) or result.get("success") is not True:
|
||||
raise HySdkError("hy_memory_update_rejected")
|
||||
return result
|
||||
|
||||
async def finish_deletion(self, memory_id: str) -> None:
|
||||
client = await self.coordinator.client()
|
||||
try:
|
||||
result = await asyncio.to_thread(client.delete, memory_id)
|
||||
except Exception as error:
|
||||
raise HySdkError("hy_memory_delete_failed") from error
|
||||
if not isinstance(result, dict):
|
||||
raise HySdkError("hy_memory_delete_rejected")
|
||||
error_code = result.get("error_code")
|
||||
if error_code or result.get("partial") is True:
|
||||
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
|
||||
if result.get("success") is not True and result.get("deleted_count") != 0:
|
||||
raise HySdkError("hy_memory_delete_rejected")
|
||||
|
||||
def extraction_window_settings(self) -> ExtractionSettings:
|
||||
return self.extraction_settings.get()
|
||||
|
||||
def set_extraction_window(
|
||||
self,
|
||||
preset: ExtractionWindowPreset,
|
||||
) -> ExtractionSettings:
|
||||
return self.extraction_settings.set_window_preset(preset)
|
||||
|
||||
def queue_depths(self) -> QueueDepths:
|
||||
now = datetime.now(UTC)
|
||||
fresh_since = (now - timedelta(days=7)).isoformat()
|
||||
with main_database(self.write_queue.database_path).read() as database:
|
||||
fresh = int(
|
||||
database.execute(
|
||||
"SELECT COUNT(*) FROM memory_write_jobs "
|
||||
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
|
||||
"AND event_at>=?",
|
||||
(fresh_since,),
|
||||
).fetchone()[0]
|
||||
)
|
||||
backlog = int(
|
||||
database.execute(
|
||||
"SELECT COUNT(*) FROM memory_write_jobs "
|
||||
"WHERE state IN ('pending','waiting_for_model','waiting_for_permission') "
|
||||
"AND event_at<?",
|
||||
(fresh_since,),
|
||||
).fetchone()[0]
|
||||
)
|
||||
recent = tuple(
|
||||
datetime.fromisoformat(str(row[0]))
|
||||
for row in database.execute(
|
||||
"SELECT updated_at FROM memory_write_jobs WHERE state='completed' "
|
||||
"ORDER BY updated_at DESC LIMIT 8"
|
||||
)
|
||||
)
|
||||
return QueueDepths(fresh=fresh, backlog=backlog, recent_completions=recent)
|
||||
|
||||
def active_depths(self) -> tuple[int, int]:
|
||||
with main_database(self.write_queue.database_path).read() as database:
|
||||
running = int(
|
||||
database.execute(
|
||||
"SELECT COUNT(*) FROM memory_write_jobs WHERE state='running'"
|
||||
).fetchone()[0]
|
||||
)
|
||||
return running, running
|
||||
|
||||
def status_counts(self) -> dict[str, int]:
|
||||
with main_database(self.write_queue.database_path).read() as database:
|
||||
return {
|
||||
str(row[0]): int(row[1])
|
||||
for row in database.execute(
|
||||
"SELECT state,COUNT(*) FROM memory_write_jobs GROUP BY state"
|
||||
)
|
||||
}
|
||||
|
||||
def manual_batch_progress(self) -> tuple[int, int]:
|
||||
return 0, 0
|
||||
|
||||
def promote_next_manual_batch(self, *, task_ids: tuple[str, ...] | None = None) -> list[Any]:
|
||||
del task_ids
|
||||
return []
|
||||
|
||||
async def reconcile_unknown_outputs(
|
||||
self,
|
||||
*,
|
||||
limit: int = 128,
|
||||
exclude_error_codes: tuple[str, ...] = (),
|
||||
) -> int:
|
||||
reconciled = 0
|
||||
for job in await asyncio.to_thread(
|
||||
self.write_queue.unknown_jobs,
|
||||
limit=limit,
|
||||
exclude_error_codes=exclude_error_codes,
|
||||
):
|
||||
receipt = await self.coordinator.inquire_write_outcome(job.request_id)
|
||||
if receipt is None:
|
||||
continue
|
||||
await asyncio.to_thread(
|
||||
self.write_queue.settle,
|
||||
job.job_id,
|
||||
state=COMPLETED,
|
||||
sdk_raw_ids=receipt.raw_ids,
|
||||
sdk_memory_ids=receipt.memory_ids,
|
||||
)
|
||||
reconciled += 1
|
||||
return reconciled
|
||||
|
||||
async def resume_allowed(self) -> None:
|
||||
await asyncio.to_thread(self.write_queue.readmit_parked)
|
||||
|
||||
def accepted_backfill_status(self) -> None:
|
||||
return None
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _agent_ids(raw: object) -> tuple[str, ...]:
|
||||
if not isinstance(raw, dict):
|
||||
raise HySdkError("hy_memory_list_shape_unknown")
|
||||
found: set[str] = set()
|
||||
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
|
||||
bucket = raw.get(bucket_name)
|
||||
if bucket is None:
|
||||
continue
|
||||
if not isinstance(bucket, dict) or not isinstance(bucket.get(rows_name), list):
|
||||
raise HySdkError("hy_memory_list_shape_unknown")
|
||||
for row in bucket[rows_name]:
|
||||
if isinstance(row, dict):
|
||||
agent_id = row.get("agent_id")
|
||||
if isinstance(agent_id, str) and agent_id:
|
||||
found.add(agent_id)
|
||||
return tuple(sorted(found))
|
||||
@@ -22,6 +22,12 @@ from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_NAMESPACE,
|
||||
HY_MEMORY_VERSION,
|
||||
)
|
||||
from one_memory.hy_sdk_migration import (
|
||||
restore_data_plane as restore_sdk_data_plane,
|
||||
)
|
||||
from one_memory.hy_sdk_migration import (
|
||||
snapshot_data_plane,
|
||||
)
|
||||
|
||||
|
||||
class MigrationBundleRejected(RuntimeError):
|
||||
@@ -113,11 +119,7 @@ def create_sdk_migration_bundle(
|
||||
if not sdk_root.is_dir():
|
||||
raise MigrationBundleRejected("sdk_migration_bundle_sdk_root_invalid")
|
||||
if snapshotter is None:
|
||||
try:
|
||||
from one_memory.hy_memory_bridge import snapshot_hy_data_plane
|
||||
except ImportError as error:
|
||||
raise MigrationBundleRejected("sdk_migration_snapshot_unavailable") from error
|
||||
snapshotter = snapshot_hy_data_plane
|
||||
snapshotter = snapshot_data_plane
|
||||
try:
|
||||
snapshotter(
|
||||
source_root=sdk_root,
|
||||
@@ -275,11 +277,7 @@ def restore_sdk_migration_bundle(
|
||||
raise MigrationBundleRejected("sdk_migration_bundle_manifest_invalid")
|
||||
|
||||
if restore_data_plane is None:
|
||||
try:
|
||||
from one_memory.hy_memory_bridge import restore_hy_data_plane
|
||||
except ImportError as error:
|
||||
raise MigrationBundleRejected("sdk_migration_restore_unavailable") from error
|
||||
restore_data_plane = restore_hy_data_plane
|
||||
restore_data_plane = restore_sdk_data_plane
|
||||
|
||||
stamp = (now or datetime.now(UTC)).strftime("%Y%m%dT%H%M%S%fZ")
|
||||
database_live = home / "one-memory.sqlite3"
|
||||
|
||||
@@ -12,7 +12,7 @@ from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import HySdkMemoryPage, HySdkRawRecord
|
||||
from one_memory.hy_sdk_types import HySdkMemoryPage, HySdkRawPage
|
||||
from one_memory.sdk_data_plane_migration import SdkDataPlaneMigrationStore
|
||||
|
||||
|
||||
@@ -103,8 +103,8 @@ SDK_LIFECYCLE_STATUSES = (
|
||||
)
|
||||
|
||||
|
||||
class SdkMigrationBridge(Protocol):
|
||||
def list_sdk_memories(
|
||||
class SdkMigrationClient(Protocol):
|
||||
def list_memories(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
@@ -115,7 +115,7 @@ class SdkMigrationBridge(Protocol):
|
||||
order: str = "desc",
|
||||
) -> Awaitable[HySdkMemoryPage]: ...
|
||||
|
||||
def import_sdk_memory_exact(
|
||||
def write_legacy_memory(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
@@ -133,7 +133,7 @@ class SdkMigrationBridge(Protocol):
|
||||
request_id: str,
|
||||
) -> Awaitable[dict[str, object]]: ...
|
||||
|
||||
def list_sdk_raw(
|
||||
def read_raw_inventory(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
@@ -143,9 +143,9 @@ class SdkMigrationBridge(Protocol):
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
order: str = "desc",
|
||||
) -> Awaitable[tuple[tuple[HySdkRawRecord, ...], int]]: ...
|
||||
) -> Awaitable[HySdkRawPage]: ...
|
||||
|
||||
def import_sdk_raw_exact(
|
||||
def write_legacy_raw(
|
||||
self,
|
||||
*,
|
||||
content: str,
|
||||
@@ -360,7 +360,7 @@ def classify_memory_inventory(
|
||||
|
||||
|
||||
async def read_sdk_memory_inventory(
|
||||
bridge: SdkMigrationBridge,
|
||||
client: SdkMigrationClient,
|
||||
*,
|
||||
user_id: str = "local",
|
||||
page_size: int = 500,
|
||||
@@ -368,7 +368,7 @@ async def read_sdk_memory_inventory(
|
||||
records: dict[str, SdkMemoryInventory] = {}
|
||||
offset = 0
|
||||
while True:
|
||||
page = await bridge.list_sdk_memories(
|
||||
page = await client.list_memories(
|
||||
user_id=user_id,
|
||||
statuses=SDK_LIFECYCLE_STATUSES,
|
||||
limit=page_size,
|
||||
@@ -425,7 +425,7 @@ def record_memory_migration_plan(
|
||||
|
||||
|
||||
async def execute_memory_migration_plan(
|
||||
bridge: SdkMigrationBridge,
|
||||
client: SdkMigrationClient,
|
||||
store: SdkDataPlaneMigrationStore,
|
||||
*,
|
||||
migration_id: str,
|
||||
@@ -457,7 +457,7 @@ async def execute_memory_migration_plan(
|
||||
continue
|
||||
if decision.action in {"delete", "verified_deleted"}:
|
||||
if decision.action == "delete":
|
||||
await bridge.delete(decision.canonical_memory_id)
|
||||
await client.delete(decision.canonical_memory_id)
|
||||
store.settle_item(
|
||||
migration_id=migration_id,
|
||||
item_key=decision.item_key,
|
||||
@@ -472,12 +472,13 @@ async def execute_memory_migration_plan(
|
||||
request_id = hashlib.sha256(
|
||||
f"{migration_id}:{decision.item_key}:import_exact".encode()
|
||||
).hexdigest()
|
||||
await bridge.import_sdk_memory_exact(
|
||||
await client.write_legacy_memory(
|
||||
content=item.content,
|
||||
memory_id=item.canonical_memory_id,
|
||||
user_id=item.user_id,
|
||||
agent_id=item.agent_id,
|
||||
session_id=item.session_id,
|
||||
source="legacy-governance",
|
||||
layer=item.layer,
|
||||
status=item.status,
|
||||
owner=item.owner,
|
||||
@@ -485,6 +486,8 @@ async def execute_memory_migration_plan(
|
||||
supersedes=item.supersedes,
|
||||
superseded_by=item.superseded_by,
|
||||
is_latest=item.status == "active",
|
||||
memory_at=None,
|
||||
metadata={},
|
||||
request_id=request_id,
|
||||
)
|
||||
if decision.action == "record_alias":
|
||||
@@ -503,7 +506,7 @@ async def execute_memory_migration_plan(
|
||||
|
||||
|
||||
async def read_sdk_raw_inventory(
|
||||
bridge: SdkMigrationBridge,
|
||||
client: SdkMigrationClient,
|
||||
*,
|
||||
user_id: str = "local",
|
||||
page_size: int = 500,
|
||||
@@ -511,13 +514,13 @@ async def read_sdk_raw_inventory(
|
||||
records: dict[str, SdkRawInventory] = {}
|
||||
offset = 0
|
||||
while True:
|
||||
rows, total = await bridge.list_sdk_raw(
|
||||
page = await client.read_raw_inventory(
|
||||
user_id=user_id,
|
||||
limit=page_size,
|
||||
offset=offset,
|
||||
order="asc",
|
||||
)
|
||||
for raw in rows:
|
||||
for raw in page.raw:
|
||||
raw_id = str(raw.raw_id)
|
||||
records[raw_id] = SdkRawInventory(
|
||||
raw_id=raw_id,
|
||||
@@ -527,9 +530,9 @@ async def read_sdk_raw_inventory(
|
||||
source=str(raw.source),
|
||||
content_hash=content_hash(str(raw.content)),
|
||||
)
|
||||
consumed = len(rows)
|
||||
consumed = len(page.raw)
|
||||
offset += consumed
|
||||
if consumed == 0 or offset >= total:
|
||||
if consumed == 0 or offset >= page.total:
|
||||
break
|
||||
return tuple(records[key] for key in sorted(records))
|
||||
|
||||
@@ -632,7 +635,7 @@ def record_raw_migration_plan(
|
||||
|
||||
|
||||
async def execute_raw_migration_plan(
|
||||
bridge: SdkMigrationBridge,
|
||||
client: SdkMigrationClient,
|
||||
store: SdkDataPlaneMigrationStore,
|
||||
*,
|
||||
migration_id: str,
|
||||
@@ -654,9 +657,9 @@ async def execute_raw_migration_plan(
|
||||
raise RuntimeError(f"missing legacy raw inventory for {decision.item_key}")
|
||||
if decision.action == "import_exact":
|
||||
request_id = hashlib.sha256(
|
||||
f"{migration_id}:{decision.item_key}:import_raw".encode()
|
||||
f"{migration_id}:{decision.item_key}:write_legacy_raw".encode()
|
||||
).hexdigest()
|
||||
await bridge.import_sdk_raw_exact(
|
||||
await client.write_legacy_raw(
|
||||
content=item.content,
|
||||
raw_id=decision.raw_id,
|
||||
user_id=item.user_id,
|
||||
|
||||
@@ -4,7 +4,10 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sqlite3
|
||||
from collections.abc import Callable
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
@@ -24,7 +27,7 @@ from one_memory.sdk_data_plane_migration import (
|
||||
)
|
||||
from one_memory.sdk_migration_bundle import inspect_sdk_migration_bundle
|
||||
from one_memory.sdk_migration_inventory import (
|
||||
SdkMigrationBridge,
|
||||
SdkMigrationClient,
|
||||
execute_memory_migration_plan,
|
||||
execute_raw_migration_plan,
|
||||
read_legacy_memory_inventory,
|
||||
@@ -47,6 +50,139 @@ class BackfillResult:
|
||||
raw_items: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SdkMigrationStaging:
|
||||
migration_id: str
|
||||
staging_home: Path
|
||||
live_root: Path
|
||||
staging_root: Path
|
||||
protective_root: Path
|
||||
state_file: Path
|
||||
|
||||
|
||||
def sdk_migration_staging(
|
||||
*,
|
||||
data_dir: Path,
|
||||
migration_id: str,
|
||||
namespace: str,
|
||||
) -> SdkMigrationStaging:
|
||||
"""Return same-filesystem paths without trusting the migration ID as a path."""
|
||||
token = hashlib.sha256(migration_id.encode()).hexdigest()[:16]
|
||||
home = data_dir.resolve()
|
||||
staging_home = home / ".sdk-migration" / token
|
||||
live_root = home / "hy-memory" / namespace
|
||||
return SdkMigrationStaging(
|
||||
migration_id=migration_id,
|
||||
staging_home=staging_home,
|
||||
live_root=live_root,
|
||||
staging_root=staging_home / "hy-memory" / namespace,
|
||||
protective_root=live_root.parent / f".{namespace}.pre-migration-{token}",
|
||||
state_file=staging_home / "swap-state.json",
|
||||
)
|
||||
|
||||
|
||||
def _write_staging_state(paths: SdkMigrationStaging, phase: str) -> None:
|
||||
paths.staging_home.mkdir(parents=True, exist_ok=True)
|
||||
temporary = paths.state_file.with_suffix(".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(
|
||||
{"schema_version": 1, "migration_id": paths.migration_id, "phase": phase},
|
||||
sort_keys=True,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
temporary.replace(paths.state_file)
|
||||
|
||||
|
||||
def recover_sdk_migration_swap(
|
||||
*,
|
||||
paths: SdkMigrationStaging,
|
||||
database_path: Path,
|
||||
) -> None:
|
||||
"""Recover an interrupted directory swap before opening any SDK store."""
|
||||
if not paths.protective_root.exists():
|
||||
return
|
||||
authority = SdkDataPlaneMigrationStore(database_path).authority()
|
||||
if authority.value == "sdk":
|
||||
if not paths.live_root.is_dir():
|
||||
raise DataPlaneMigrationError("sdk_migration_committed_store_missing")
|
||||
shutil.rmtree(paths.protective_root)
|
||||
shutil.rmtree(paths.staging_home, ignore_errors=True)
|
||||
return
|
||||
if paths.live_root.exists():
|
||||
if paths.staging_root.exists():
|
||||
raise DataPlaneMigrationError("sdk_migration_swap_state_ambiguous")
|
||||
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
paths.live_root.replace(paths.staging_root)
|
||||
paths.protective_root.replace(paths.live_root)
|
||||
_write_staging_state(paths, "recovered")
|
||||
|
||||
|
||||
def prepare_sdk_migration_staging(
|
||||
*,
|
||||
paths: SdkMigrationStaging,
|
||||
database_path: Path,
|
||||
) -> Path:
|
||||
"""Build a private copy; incomplete retries restart from the unchanged live root."""
|
||||
recover_sdk_migration_swap(paths=paths, database_path=database_path)
|
||||
migration_store = SdkDataPlaneMigrationStore(database_path)
|
||||
try:
|
||||
progress = migration_store.progress(paths.migration_id)
|
||||
reuse_ready = progress.state == "ready" and progress.phase == "reconciled"
|
||||
except Exception:
|
||||
reuse_ready = False
|
||||
if paths.staging_root.exists():
|
||||
if paths.staging_root.is_symlink() or not paths.staging_root.is_dir():
|
||||
raise DataPlaneMigrationError("sdk_migration_staging_invalid")
|
||||
if reuse_ready:
|
||||
return paths.staging_home
|
||||
shutil.rmtree(paths.staging_root)
|
||||
paths.live_root.mkdir(parents=True, exist_ok=True)
|
||||
from one_memory.hy_sdk_migration import snapshot_data_plane
|
||||
|
||||
snapshot_data_plane(
|
||||
source_root=paths.live_root,
|
||||
target_root=paths.staging_root,
|
||||
writers_stopped=True,
|
||||
)
|
||||
_write_staging_state(paths, "prepared")
|
||||
return paths.staging_home
|
||||
|
||||
|
||||
def activate_sdk_migration_staging(
|
||||
*,
|
||||
paths: SdkMigrationStaging,
|
||||
database_path: Path,
|
||||
commit: Callable[[], CutoverMarker],
|
||||
) -> CutoverMarker:
|
||||
"""Atomically expose staged SDK files and roll them back if cutover fails."""
|
||||
recover_sdk_migration_swap(paths=paths, database_path=database_path)
|
||||
if not paths.staging_root.is_dir():
|
||||
raise DataPlaneMigrationError("sdk_migration_staging_missing")
|
||||
paths.live_root.mkdir(parents=True, exist_ok=True)
|
||||
_write_staging_state(paths, "activating")
|
||||
paths.live_root.replace(paths.protective_root)
|
||||
try:
|
||||
paths.staging_root.replace(paths.live_root)
|
||||
_write_staging_state(paths, "staging_live")
|
||||
marker = commit()
|
||||
except BaseException:
|
||||
if paths.live_root.exists():
|
||||
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
paths.live_root.replace(paths.staging_root)
|
||||
if paths.protective_root.exists():
|
||||
paths.protective_root.replace(paths.live_root)
|
||||
_write_staging_state(paths, "rolled_back")
|
||||
raise
|
||||
with suppress(OSError):
|
||||
_write_staging_state(paths, "committed")
|
||||
# Authority is already committed; leftover paths are detected by
|
||||
# sdk-migration-status and removed by sdk-cleanup on the next startup.
|
||||
shutil.rmtree(paths.protective_root, ignore_errors=True)
|
||||
shutil.rmtree(paths.staging_home, ignore_errors=True)
|
||||
return marker
|
||||
|
||||
|
||||
def prepare_legacy_write_queue_for_cutover(database_path: Path) -> tuple[int, int]:
|
||||
"""Freeze old jobs and move unstarted task rows into the surviving queue."""
|
||||
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
|
||||
@@ -217,7 +353,6 @@ def commit_reconciled_cutover(
|
||||
reconciliation_digest=digest,
|
||||
)
|
||||
inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=marker.backup_bundle_id)
|
||||
drop_legacy_tables(database_path)
|
||||
return marker
|
||||
|
||||
|
||||
@@ -232,13 +367,23 @@ def resume_legacy_cleanup(*, data_dir: Path, database_path: Path) -> tuple[str,
|
||||
data_dir=data_dir,
|
||||
bundle_id=marker.backup_bundle_id,
|
||||
)
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
|
||||
|
||||
recover_sdk_migration_swap(
|
||||
paths=sdk_migration_staging(
|
||||
data_dir=data_dir,
|
||||
migration_id=marker.migration_id,
|
||||
namespace=HY_MEMORY_NAMESPACE,
|
||||
),
|
||||
database_path=database_path,
|
||||
)
|
||||
return drop_legacy_tables(database_path)
|
||||
|
||||
|
||||
async def run_sdk_data_plane_backfill(
|
||||
*,
|
||||
database_path: Path,
|
||||
bridge: SdkMigrationBridge,
|
||||
client: SdkMigrationClient,
|
||||
migration_id: str,
|
||||
bundle_id: str,
|
||||
source_runtime_build: str,
|
||||
@@ -261,7 +406,7 @@ async def run_sdk_data_plane_backfill(
|
||||
store.set_phase(migration_id, phase="inventory", state="running")
|
||||
|
||||
legacy_memories = read_legacy_memory_inventory(database_path)
|
||||
sdk_memories = await read_sdk_memory_inventory(bridge)
|
||||
sdk_memories = await read_sdk_memory_inventory(client)
|
||||
memory_decisions = record_memory_migration_plan(
|
||||
store,
|
||||
migration_id=migration_id,
|
||||
@@ -269,7 +414,7 @@ async def run_sdk_data_plane_backfill(
|
||||
sdk=sdk_memories,
|
||||
)
|
||||
legacy_raw = read_legacy_raw_inventory(database_path)
|
||||
sdk_raw = await read_sdk_raw_inventory(bridge)
|
||||
sdk_raw = await read_sdk_raw_inventory(client)
|
||||
raw_decisions = record_raw_migration_plan(
|
||||
store,
|
||||
migration_id=migration_id,
|
||||
@@ -279,14 +424,14 @@ async def run_sdk_data_plane_backfill(
|
||||
|
||||
store.set_phase(migration_id, phase="backfill", state="running")
|
||||
await execute_memory_migration_plan(
|
||||
bridge,
|
||||
client,
|
||||
store,
|
||||
migration_id=migration_id,
|
||||
legacy=legacy_memories,
|
||||
decisions=memory_decisions,
|
||||
)
|
||||
await execute_raw_migration_plan(
|
||||
bridge,
|
||||
client,
|
||||
store,
|
||||
migration_id=migration_id,
|
||||
legacy=legacy_raw,
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
"""Single-write adapter from durable Evidence payloads to the public HY SDK."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from one_memory.hy_sdk_runtime import SdkClientHandle
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
|
||||
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
|
||||
|
||||
PRODUCT_AGENT_ID = "all-apps"
|
||||
_MESSAGE_ROLES = frozenset({"user", "assistant", "tool"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SdkWriteReceipt:
|
||||
"""The only SDK write identity retained by MemHub."""
|
||||
|
||||
raw_ids: tuple[str, ...] = ()
|
||||
memory_ids: tuple[str, ...] = ()
|
||||
|
||||
|
||||
class SdkWriteProcessor:
|
||||
"""Turn one durable job into exactly one public SDK write call."""
|
||||
|
||||
def __init__(self, client_handle: SdkClientHandle) -> None:
|
||||
self._client_handle = client_handle
|
||||
|
||||
async def process_sdk_job(
|
||||
self,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
*,
|
||||
request_id: str,
|
||||
before_hy_invocation: Callable[[], Awaitable[None]],
|
||||
) -> SdkWriteReceipt:
|
||||
if not records:
|
||||
raise RuntimeError("hy_memory_task_without_inputs")
|
||||
|
||||
scope = _write_scope(records)
|
||||
metadata = _write_metadata(records, scope)
|
||||
client = await self._client_handle.client()
|
||||
|
||||
if all(_is_extracted_content(record) for record in records):
|
||||
contents = [sanitize_model_input(record.verbatim) for record in records]
|
||||
await before_hy_invocation()
|
||||
result = await asyncio.to_thread(
|
||||
client.add_extracted,
|
||||
contents,
|
||||
user_id=scope.user_id,
|
||||
agent_id=PRODUCT_AGENT_ID,
|
||||
session_id=scope.session_id,
|
||||
source=scope.source,
|
||||
metadata=metadata,
|
||||
memory_at=scope.memory_at,
|
||||
request_id=request_id,
|
||||
)
|
||||
else:
|
||||
messages = [
|
||||
{
|
||||
"role": record.role,
|
||||
"content": sanitize_model_input(record.verbatim),
|
||||
}
|
||||
for record in records
|
||||
if record.content_kind is EvidenceContentKind.RAW_CONVERSATION
|
||||
and record.role in _MESSAGE_ROLES
|
||||
]
|
||||
if not messages:
|
||||
raise RuntimeError("hy_memory_task_without_inputs")
|
||||
await before_hy_invocation()
|
||||
result = await asyncio.to_thread(
|
||||
client.add,
|
||||
messages,
|
||||
user_id=scope.user_id,
|
||||
agent_id=PRODUCT_AGENT_ID,
|
||||
session_id=scope.session_id,
|
||||
source=scope.source,
|
||||
metadata=metadata,
|
||||
memory_at=scope.memory_at,
|
||||
scene=None,
|
||||
workspace_id=scope.workspace,
|
||||
request_id=request_id,
|
||||
)
|
||||
|
||||
return _receipt_from_result(result)
|
||||
|
||||
async def inquire_write_outcome(self, request_id: str) -> SdkWriteReceipt | None:
|
||||
"""Return a confirmed successful receipt without scanning SDK stores."""
|
||||
if not request_id:
|
||||
raise ValueError("request_id is required")
|
||||
client = await self._client_handle.client()
|
||||
outcome = await asyncio.to_thread(client.get_write_outcome, request_id)
|
||||
if outcome is None:
|
||||
return None
|
||||
if not isinstance(outcome, Mapping):
|
||||
raise RuntimeError("hy_sdk_write_outcome_shape_invalid")
|
||||
|
||||
status = outcome.get("status")
|
||||
if status != "succeeded":
|
||||
return None
|
||||
result = outcome.get("result")
|
||||
if not isinstance(result, Mapping):
|
||||
raise RuntimeError("hy_sdk_write_outcome_shape_invalid")
|
||||
return _receipt_from_result(result)
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._client_handle.close()
|
||||
|
||||
|
||||
class SdkWriteCoordinator:
|
||||
"""Resolve the active model profile, then execute one public SDK write."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
profiles: ModelProfileStore,
|
||||
secrets: SecretStore,
|
||||
data_dir: Any,
|
||||
transport_factory: Callable[[ModelProfileSnapshot], Callable[[], Any] | None],
|
||||
sedimentation_enabled: Callable[[], bool],
|
||||
) -> None:
|
||||
self.profiles = profiles
|
||||
self.secrets = secrets
|
||||
self.data_dir = data_dir
|
||||
self.transport_factory = transport_factory
|
||||
self.sedimentation_enabled = sedimentation_enabled
|
||||
self._handles: dict[tuple[str, int], SdkClientHandle] = {}
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def process_sdk_job(
|
||||
self,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
*,
|
||||
request_id: str,
|
||||
before_hy_invocation: Callable[[], Awaitable[None]],
|
||||
) -> SdkWriteReceipt:
|
||||
if not self.sedimentation_enabled():
|
||||
raise RuntimeError("model_sedimentation_disabled")
|
||||
handle = await self._active_handle()
|
||||
return await SdkWriteProcessor(handle).process_sdk_job(
|
||||
records,
|
||||
request_id=request_id,
|
||||
before_hy_invocation=before_hy_invocation,
|
||||
)
|
||||
|
||||
async def inquire_write_outcome(
|
||||
self, request_id: str
|
||||
) -> SdkWriteReceipt | None:
|
||||
return await SdkWriteProcessor(
|
||||
await self._active_handle()
|
||||
).inquire_write_outcome(request_id)
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._lock:
|
||||
handles, self._handles = tuple(self._handles.values()), {}
|
||||
await asyncio.gather(*(handle.close() for handle in handles))
|
||||
|
||||
async def client(self) -> Any:
|
||||
"""Return the public SDK client selected by the active write profile."""
|
||||
return await (await self._active_handle()).client()
|
||||
|
||||
async def active_profile_credentials(
|
||||
self,
|
||||
) -> tuple[ModelProfileSnapshot, str | None]:
|
||||
"""Resolve one stable profile snapshot for graph administration."""
|
||||
_identity, profile, secret = await asyncio.to_thread(self._read_active_credentials)
|
||||
return profile, secret
|
||||
|
||||
async def _active_handle(self) -> SdkClientHandle:
|
||||
identity, profile, secret = await asyncio.to_thread(
|
||||
self._read_active_credentials
|
||||
)
|
||||
async with self._lock:
|
||||
handle = self._handles.get(identity)
|
||||
if handle is None:
|
||||
handle = SdkClientHandle(
|
||||
profile,
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
http_transport_factory=self.transport_factory(profile),
|
||||
)
|
||||
self._handles[identity] = handle
|
||||
stale = [
|
||||
value
|
||||
for key, value in self._handles.items()
|
||||
if key != identity
|
||||
]
|
||||
self._handles = {identity: handle}
|
||||
await asyncio.gather(*(value.close() for value in stale))
|
||||
return handle
|
||||
|
||||
def _read_active_credentials(
|
||||
self,
|
||||
) -> tuple[tuple[str, int], ModelProfileSnapshot, str | None]:
|
||||
identity = self.profiles.active_identity()
|
||||
if identity is None:
|
||||
raise RuntimeError("model_profile_not_active")
|
||||
profile = self.profiles.snapshot(*identity)
|
||||
try:
|
||||
secret = self.profiles.resolve_snapshot_secret(
|
||||
*identity,
|
||||
self.secrets,
|
||||
)
|
||||
except SecretStoreUnavailableError:
|
||||
raise RuntimeError("secure_storage_unavailable") from None
|
||||
if self.profiles.active_identity() != identity:
|
||||
raise RuntimeError("model_profile_changed")
|
||||
if profile.has_api_key and secret is None:
|
||||
raise RuntimeError("model_secret_unavailable")
|
||||
return identity, profile, secret
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _WriteScope:
|
||||
user_id: str
|
||||
session_id: str
|
||||
source: str
|
||||
workspace: str | None
|
||||
workspaces: tuple[str, ...]
|
||||
memory_at: datetime | None
|
||||
|
||||
|
||||
def _write_scope(records: tuple[EvidenceRecord, ...]) -> _WriteScope:
|
||||
first = records[0]
|
||||
expected = (first.user_id, first.source_id, first.conversation_id)
|
||||
if any(
|
||||
(record.user_id, record.source_id, record.conversation_id) != expected
|
||||
for record in records[1:]
|
||||
):
|
||||
raise RuntimeError("durable_write_multiple_sources")
|
||||
workspaces = tuple(
|
||||
sorted({record.workspace for record in records if record.workspace is not None})
|
||||
)
|
||||
return _WriteScope(
|
||||
user_id=first.user_id,
|
||||
session_id=first.conversation_id,
|
||||
source=first.source_id,
|
||||
workspace=workspaces[0] if len(workspaces) == 1 else None,
|
||||
workspaces=workspaces,
|
||||
memory_at=max(
|
||||
(record.created_at for record in records if record.created_at is not None),
|
||||
default=None,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _write_metadata(
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
scope: _WriteScope,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"source_id": scope.source,
|
||||
"session_id": scope.session_id,
|
||||
"workspace_ids": list(scope.workspaces),
|
||||
"memory_at": scope.memory_at.isoformat() if scope.memory_at is not None else None,
|
||||
"evidence_ids": [record.evidence_id for record in records],
|
||||
"input_kind": (
|
||||
"memory_content_without_raw"
|
||||
if all(_is_extracted_content(record) for record in records)
|
||||
else "raw_conversation"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _is_extracted_content(record: EvidenceRecord) -> bool:
|
||||
if record.content_kind is EvidenceContentKind.MEMORY_CONTENT:
|
||||
return True
|
||||
manual = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
|
||||
return (
|
||||
record.source_id == "one-memory-manual"
|
||||
and manual is not None
|
||||
and manual.raw_meta.get("input_mode") == "direct_memory_note"
|
||||
)
|
||||
|
||||
|
||||
def _receipt_from_result(value: object) -> SdkWriteReceipt:
|
||||
if not isinstance(value, Mapping):
|
||||
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
|
||||
if value.get("success") is not True:
|
||||
raise RuntimeError("hy_sdk_write_failed")
|
||||
|
||||
# Read only documented public receipt fields. ``variant`` identifies the
|
||||
# productivity namespace; IDs from both variants are canonical memory IDs.
|
||||
variant = value.get("variant")
|
||||
if variant is not None and not isinstance(variant, str):
|
||||
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
|
||||
raw_ids = _public_ids(value, singular="raw_id", plural="raw_ids")
|
||||
memory_ids = _ordered_unique(
|
||||
(
|
||||
*_public_ids(value, singular="memory_id", plural="memory_ids"),
|
||||
*_public_ids(value, plural="experience_ids"),
|
||||
*_public_ids(value, plural="fact_ids"),
|
||||
)
|
||||
)
|
||||
return SdkWriteReceipt(raw_ids=raw_ids, memory_ids=memory_ids)
|
||||
|
||||
|
||||
def _public_ids(
|
||||
value: Mapping[object, object],
|
||||
*,
|
||||
singular: str | None = None,
|
||||
plural: str,
|
||||
) -> tuple[str, ...]:
|
||||
found: list[str] = []
|
||||
if singular is not None:
|
||||
item = value.get(singular)
|
||||
if item is not None:
|
||||
if not isinstance(item, str):
|
||||
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
|
||||
if item:
|
||||
found.append(item)
|
||||
items = value.get(plural)
|
||||
if items is not None:
|
||||
if isinstance(items, (str, bytes)) or not isinstance(items, Sequence):
|
||||
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
|
||||
for item in items:
|
||||
if not isinstance(item, str):
|
||||
raise RuntimeError("hy_sdk_write_receipt_shape_invalid")
|
||||
if item:
|
||||
found.append(item)
|
||||
return _ordered_unique(found)
|
||||
|
||||
|
||||
def _ordered_unique(values: Sequence[str]) -> tuple[str, ...]:
|
||||
return tuple(dict.fromkeys(values))
|
||||
@@ -4,7 +4,6 @@
|
||||
|
||||
| 测试文件 | 等级 | 一句话说明 |
|
||||
| --- | --- | --- |
|
||||
| `test_active_credentials.py` | `contract` | Active credentials are read once per call, never reused after a refusal. |
|
||||
| `test_anchor_invalid_rewind.py` | `contract` | Bug 2 real branch: an in-place same-length rewrite invalidates the anchor. |
|
||||
| `test_architecture_layer_gate.py` | `contract` | A0-01: L0-L4 dependency-direction gate. |
|
||||
| `test_archive_detection.py` | `contract` | S4.1: what an uploaded export is, decided before anything parses it. |
|
||||
@@ -91,7 +90,6 @@
|
||||
| `test_disable_thinking.py` | `contract` | 运行期生成请求须按档案可选关闭思考;默认不向厂商下发未知字段。 |
|
||||
| `test_dual_engine_helper.py` | `contract` | Tests for the dual-engine harness itself. |
|
||||
| `test_event_loop_blocking.py` | `contract` | Proof, as a test, that blocking work on the loop freezes it — and the fix works. |
|
||||
| `test_evidence_reconcile_primitive.py` | `integration` | A bounded, read-only discovery primitive for the H23 Evidence reconcile. |
|
||||
| `test_evidence_store.py` | `contract` | 覆盖 same source message merges origins without duplication、identical text from different sources is not deduplicated、rescanning an unchanged file does not write the message again 等 26 个场景。 |
|
||||
| `test_export_probe.py` | `contract` | What a probe may say about a stranger's chat history, and what it may not. |
|
||||
| `test_external_installation_permissions.py` | `integration` | 覆盖 external marker is not an installation grant、external diagnostic trace uses its own read only tool、raw source grants filter before limit and private body loading 场景。 |
|
||||
@@ -103,8 +101,6 @@
|
||||
| `test_file_import_operations_review.py` | `contract` | Upload refusal/legacy compatibility review with bounded synthetic files only. |
|
||||
| `test_frozen_core_layout.py` | `contract` | 覆盖 single file layout remains available without private preset、folder layout keeps same core and all native dependencies、unknown layout does not silently build another package 等 5 个场景。 |
|
||||
| `test_golden_corpus.py` | `contract` | A7-01: the golden corpus comparison. |
|
||||
| `test_governance_permission_commit.py` | `contract` | 覆盖 permission guard is rechecked inside write transaction and rolls back、guard context does not leak into subsequent manual actions、guard check and commit hold database write authority 场景。 |
|
||||
| `test_governance_read_path_does_not_take_the_writer.py` | `contract` | The recall read path must not acquire the single global writer. |
|
||||
| `test_graph_width_endpoint.py` | `integration` | The width mismatch has to reach the interface, not just the log. |
|
||||
| `test_graph_width_export.py` | `unit` | EXPORT_FIRST has to actually export. |
|
||||
| `test_graph_width_guard.py` | `unit` | Graph width mismatch must ask, never silently archive. |
|
||||
@@ -115,16 +111,9 @@
|
||||
| `test_host_capabilities.py` | `unit` | A plugin does not become realtime on every host or every unknown version. |
|
||||
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
|
||||
| `test_hourly_c1_recall_stage_probe_contract.py` | `contract` | Guard the packaged recall stage probe's command and error semantics. |
|
||||
| `test_hy_coding_reply.py` | `contract` | R-07: HY's coding-shaped replies read as "no memories at all". |
|
||||
| `test_hy_log_privacy.py` | `contract` | HY SDK records are not rewritten by One Memory's logging setup. |
|
||||
| `test_hy_memory_bridge.py` | `contract` | 覆盖 hy client initialization is serial across bridges、native hy clients share one runtime per data directory、ollama profile ignores ambient remote storage and models 等 41 个场景。 |
|
||||
| `test_hy_metrics_lifecycle.py` | `contract` | What the pinned public SDK's MetricsCollector does, and what that costs us. |
|
||||
| `test_hy_network_egress.py` | `contract` | Where a real HY client is allowed to send bytes, and where it is not. |
|
||||
| `test_hy_response_guard.py` | `contract` | 覆盖 invalid response retries same hy provider without changing prompt、repeated empty response fails instead of becoming no op、failure receipt survives a later successful model call 等 23 个场景。 |
|
||||
| `test_hy_response_shapes.py` | `contract` | HY response shape selection only; no provider, write routing or model calls. |
|
||||
| `test_hy_runtime_config.py` | `contract` | MEMORY.md is the audited record of what we send HY; keep it answerable. |
|
||||
| `test_hy_sdk_idempotency_probe.py` | `contract` | A0-03: probe whether the HY SDK can make a write idempotent. |
|
||||
| `test_hy_source_context.py` | `contract` | 覆盖 bound context rejects a same named conversation in another workspace、bound context rejects a duplicated evidence id、bound context rejects a same conversation from another source 等 9 个场景。 |
|
||||
| `test_hy_step_trace_switch.py` | `contract` | Rule 15: the per-step HY trace is off unless the debug tier asks for it. |
|
||||
| `test_import_api.py` | `integration` | 覆盖 resume during cleanup returns conflict without internal error、authorize detect import and revoke api flow、protected api rejects missing token and foreign origin 等 5 个场景。 |
|
||||
| `test_import_control_races_review.py` | `contract` | Read-only review repro: real runner/store, synthetic delayed adapter cleanup. |
|
||||
@@ -168,16 +157,11 @@
|
||||
| `test_memory_batch_extraction.py` | `contract` | 覆盖 batch extractor keeps only unique known evidence ids、batch extractor maps provider failure to machine error 场景。 |
|
||||
| `test_memory_citation_query_plan.py` | `contract` | Correlated citation checks must look links up by memory, not scan by 'active'. |
|
||||
| `test_memory_content_import.py` | `integration` | 覆盖 existing memory preview keeps unknown author and exact original、invalid existing memory records never leave a partial upload、memory content requires explicit mode and rejects vendor guessing 等 5 个场景。 |
|
||||
| `test_memory_correction_sync.py` | `integration` | R-27: a correction that only lands locally, while HY keeps the old text. |
|
||||
| `test_memory_evidence_freshness.py` | `contract` | 新鲜度拆除后:读路径恒为 current;证据修订 / 畸形 proof / segment ref 不再藏记忆或 503。 |
|
||||
| `test_memory_freshness_integration_review.py` | `integration` | Independent synthetic read-path review; no lifespan, model or user database. |
|
||||
| `test_memory_governance.py` | `contract` | 覆盖 correction keeps verbatim and appends history、mutations are revision checked and request idempotent、delete result survives restart and request replay 等 42 个场景。 |
|
||||
| `test_memory_governance_api.py` | `integration` | 覆盖 memory governance api preserves evidence and reports cleanup、memory validation does not echo content 场景。 |
|
||||
| `test_memory_graph_api.py` | `integration` | 覆盖 graph labels where every edge came from、a created memory is on the map with its source links、drawing the map does not stop the app answering 等 12 个场景。 |
|
||||
| `test_memory_language.py` | `contract` | R-03: memory language follows the message, so Kristin's gets written in English. |
|
||||
| `test_memory_notes.py` | `integration` | 覆盖 empty library can save a note without a matter or configured model、note requires authentication explicit scope and nonempty content、saved note evidence returns raw and consent reference from same record 等 6 个场景。 |
|
||||
| `test_memory_processing.py` | `contract` | 覆盖 stale known reason diagnostics never fail dispatch、memory task logs six step durations without content、assistant turns reach hy so references can be resolved 等 95 个场景。 |
|
||||
| `test_memory_processing_diagnostics.py` | `contract` | 覆盖 conversation prepare failure includes safe message and stack、deletion sync failure includes safe message and stack 场景。 |
|
||||
| `test_memory_qa.py` | `contract` | 覆盖 search terms keep meaningful single chinese character、streamed answer emits recall then incremental grounded text、governance change during generation stops the old answer 等 14 个场景。 |
|
||||
| `test_memory_qa_api.py` | `integration` | 覆盖 ask stream endpoint emits recall deltas and final、ask stream endpoint requires the local token 场景。 |
|
||||
| `test_memory_qa_permissions.py` | `contract` | 覆盖 initial refusal precedes all secret access and network、invalid authorization result fails before secret access、authorization receives exact question and only outbound grounding 等 16 个场景。 |
|
||||
@@ -289,7 +273,6 @@
|
||||
| `test_secret_store_error_code_api.py` | `integration` | 3925a kernel item 3: the error code reaches the HTTP boundary. |
|
||||
| `test_secret_store_error_codes.py` | `integration` | 3925a kernel item 3: keychain failure classification and backoff that heals. |
|
||||
| `test_secure_storage.py` | `integration` | 覆盖 isolated keychain service namespace、custom keychain service is refused outside isolated tests、packaged keychain budget covers loaded process startup 等 20 个场景。 |
|
||||
| `test_sedimentation_concurrency.py` | `contract` | Sedimentation runs several harnesses at once, but one harness at a time. |
|
||||
| `test_sedimentation_switch.py` | `contract` | Stopping model-driven sedimentation without losing the configuration. |
|
||||
| `test_service_level_pressure_acceptance.py` | `contract` | 覆盖 workbuddy recall remains available during 100k import 场景。 |
|
||||
| `test_service_levels.py` | `contract` | 覆盖 background never consumes the realtime reservation、realtime is next for the same source after background yields、import history gets one slice after nine realtime admissions 等 8 个场景。 |
|
||||
@@ -320,7 +303,6 @@
|
||||
| `test_source_sync_idempotent.py` | `integration` | Final-state Harness sync is idempotent and resumes missed source work. |
|
||||
| `test_source_sync_permissions.py` | `integration` | S1.3 source permission tests use temporary files only, never real host history. |
|
||||
| `test_source_totals.py` | `contract` | 覆盖 the first ask is answered from the rows、same named sessions in two workspaces count as two conversations、insert during count is not marked as already counted 等 7 个场景。 |
|
||||
| `test_source_write_switch.py` | `contract` | A frozen Harness produces no extraction tasks. |
|
||||
| `test_sql_compat.py` | `contract` | The shared compatibility layer, especially 'rowcount'. |
|
||||
| `test_sqlite_semantics_gate.py` | `contract` | D56: prevent SQLite implementation details spreading outside the data layer. |
|
||||
| `test_sqlite_write_contention.py` | `contract` | Measure whether SQLite's single writer actually blocks us. |
|
||||
@@ -345,4 +327,4 @@
|
||||
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
|
||||
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
|
||||
|
||||
共 340 个测试文件。
|
||||
共 322 个测试文件。
|
||||
@@ -14,8 +14,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
|
||||
EXPECTED_TEST_FILE_COUNT = 340
|
||||
EXPECTED_TEST_FILES_SHA256 = "f69d3f33df5277febb09557d5bce7adb7d8719ed47e48a71f852e3e4ecf8fd74"
|
||||
EXPECTED_TEST_FILE_COUNT = 322
|
||||
EXPECTED_TEST_FILES_SHA256 = "d242783adb4015ebdbb5c099f3b5aa88a99c828ca6fddc8ca8d6c74c56e48945"
|
||||
|
||||
# Pure transformations, validation models, parity checks and state machines.
|
||||
# This list is intentionally small: uncertain files default to contract.
|
||||
@@ -62,7 +62,6 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_d93_same_process_load.py",
|
||||
"test_diagnostic_trace.py",
|
||||
"test_diagnostic_trace_api.py",
|
||||
"test_evidence_reconcile_primitive.py",
|
||||
"test_external_installation_permissions.py",
|
||||
"test_fault_matrix.py",
|
||||
"test_file_import_operations.py",
|
||||
@@ -77,7 +76,6 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_manual_transcript_import.py",
|
||||
"test_measure_c1_loop_blocking_contract.py",
|
||||
"test_memory_content_import.py",
|
||||
"test_memory_correction_sync.py",
|
||||
"test_memory_freshness_integration_review.py",
|
||||
"test_memory_governance_api.py",
|
||||
"test_memory_graph_api.py",
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
|
||||
def model_snapshot(
|
||||
provider: ProviderKind,
|
||||
*,
|
||||
base_url: str | None,
|
||||
has_api_key: bool,
|
||||
embedding_provider: ProviderKind | None = None,
|
||||
embedding_base_url: str | None = None,
|
||||
) -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="profile",
|
||||
revision=3,
|
||||
provider_kind=provider,
|
||||
base_url=base_url,
|
||||
model="memory-model",
|
||||
embedding_model="embedding-model",
|
||||
embedding_provider_kind=embedding_provider,
|
||||
embedding_base_url=embedding_base_url,
|
||||
embedding_dimensions=768,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
has_api_key=has_api_key,
|
||||
)
|
||||
|
||||
|
||||
def kuzu_graph_with_embedding_width(db_path: Path, width: int) -> None:
|
||||
kuzu = pytest.importorskip("kuzu")
|
||||
database = kuzu.Database(str(db_path))
|
||||
connection = kuzu.Connection(database)
|
||||
connection.execute(
|
||||
f"CREATE NODE TABLE Memory("
|
||||
f"node_id STRING, embedding FLOAT[{width}], PRIMARY KEY(node_id))"
|
||||
)
|
||||
del connection
|
||||
del database
|
||||
gc.collect()
|
||||
@@ -1,225 +0,0 @@
|
||||
"""Active credentials are read once per call, never reused after a refusal."""
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.background_work import shutdown_bulk_pool
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfilePatch,
|
||||
ModelProfileStore,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore, SecretStoreUnavailableError
|
||||
|
||||
|
||||
class CountingSecrets(InMemorySecretStore):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.reads = []
|
||||
self.denied = False
|
||||
self.after_read = None
|
||||
|
||||
def has(self, secret_ref):
|
||||
# The OS store's has() resolves the secret; the in-memory default does not.
|
||||
return self.resolve(secret_ref) is not None
|
||||
|
||||
def resolve(self, secret_ref):
|
||||
self.reads.append((secret_ref, threading.get_ident()))
|
||||
if self.denied:
|
||||
raise SecretStoreUnavailableError("synthetic reader denial, must not leak")
|
||||
value = super().resolve(secret_ref)
|
||||
callback, self.after_read = self.after_read, None
|
||||
if callback is not None:
|
||||
callback()
|
||||
return value
|
||||
|
||||
|
||||
def create_profile(profiles, secrets, *, keyed=True):
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Synthetic credentials",
|
||||
provider_kind="openai_compatible" if keyed else "ollama",
|
||||
base_url="http://127.0.0.1:1/v1",
|
||||
model="synthetic-model",
|
||||
api_key=SecretStr("synthetic-credential") if keyed else None,
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
return profiles.activate(profile.profile_id, secrets)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def setup(tmp_path):
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
secrets = CountingSecrets()
|
||||
profiles = ModelProfileStore(database)
|
||||
profile = create_profile(profiles, secrets)
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=MemoryTaskStore(database),
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=EvidenceStore(database),
|
||||
governance=MemoryGovernanceStore(database),
|
||||
data_dir=tmp_path,
|
||||
)
|
||||
secrets.reads.clear()
|
||||
try:
|
||||
yield supervisor, profiles, secrets, profile
|
||||
finally:
|
||||
shutdown_bulk_pool()
|
||||
|
||||
|
||||
def test_resolves_active_immutable_snapshot_once_off_loop(setup):
|
||||
supervisor, profiles, secrets, profile = setup
|
||||
expected = profiles.snapshot(profile.profile_id, profile.revision)
|
||||
caller = threading.get_ident()
|
||||
|
||||
result = asyncio.run(supervisor._active_credentials())
|
||||
|
||||
assert result == (expected, "synthetic-credential")
|
||||
assert secrets.reads == [(profiles.secret_ref(profile.profile_id), secrets.reads[0][1])]
|
||||
assert secrets.reads[0][1] != caller
|
||||
assert supervisor.tasks.list() == []
|
||||
assert supervisor._bridges == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("concurrent", [False, True])
|
||||
def test_each_call_resolves_again_without_cross_request_cache(setup, concurrent):
|
||||
supervisor, _profiles, secrets, _profile = setup
|
||||
|
||||
async def scenario():
|
||||
if concurrent:
|
||||
return await asyncio.gather(*(supervisor._active_credentials() for _ in range(4)))
|
||||
return [await supervisor._active_credentials() for _ in range(4)]
|
||||
|
||||
results = asyncio.run(scenario())
|
||||
|
||||
assert all(result is not None and result[1] == "synthetic-credential" for result in results)
|
||||
assert len(secrets.reads) == 4
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["missing", "denied"])
|
||||
def test_prior_success_cannot_mask_current_reader_refusal(setup, failure):
|
||||
supervisor, profiles, secrets, profile = setup
|
||||
|
||||
async def scenario():
|
||||
assert (await supervisor._active_credentials())[1] == "synthetic-credential"
|
||||
secrets.reads.clear()
|
||||
if failure == "missing":
|
||||
secrets.delete(profiles.secret_ref(profile.profile_id))
|
||||
else:
|
||||
secrets.denied = True
|
||||
code = "model_secret_unavailable" if failure == "missing" else "secure_storage_unavailable"
|
||||
with pytest.raises(HyMemoryBridgeError, match=f"^{code}$"):
|
||||
await supervisor._active_credentials()
|
||||
assert len(secrets.reads) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", ["deactivate", "switch", "revision"])
|
||||
def test_profile_change_during_secret_read_rejects_old_credentials(setup, change):
|
||||
supervisor, profiles, secrets, profile = setup
|
||||
other = create_profile(profiles, secrets)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
secrets.reads.clear()
|
||||
|
||||
def change_identity():
|
||||
# Concurrent committed identity change while the original reader is in flight.
|
||||
with profiles._connect(writing=True) as db:
|
||||
if change == "revision":
|
||||
db.execute(
|
||||
"UPDATE model_profiles SET revision=revision+1 WHERE profile_id=?",
|
||||
(profile.profile_id,),
|
||||
)
|
||||
else:
|
||||
db.execute("UPDATE model_profiles SET is_active=0")
|
||||
if change == "switch":
|
||||
db.execute(
|
||||
"UPDATE model_profiles SET is_active=1 WHERE profile_id=?",
|
||||
(other.profile_id,),
|
||||
)
|
||||
|
||||
secrets.after_read = change_identity
|
||||
with pytest.raises(HyMemoryBridgeError, match="^model_profile_changed$"):
|
||||
asyncio.run(supervisor._active_credentials())
|
||||
assert len(secrets.reads) == 1
|
||||
assert supervisor._bridges == {}
|
||||
|
||||
|
||||
def test_rotation_between_calls_returns_new_snapshot_and_secret(setup):
|
||||
supervisor, profiles, secrets, profile = setup
|
||||
before = asyncio.run(supervisor._active_credentials())
|
||||
updated = profiles.update(
|
||||
profile.profile_id,
|
||||
ModelProfilePatch(expected_revision=profile.revision, api_key=SecretStr("synthetic-new")),
|
||||
secrets,
|
||||
)
|
||||
assert updated.is_active is False
|
||||
assert updated.capabilities == ModelCapabilities()
|
||||
assert profiles.active_identity() is None
|
||||
secrets.reads.clear()
|
||||
|
||||
assert asyncio.run(supervisor._active_credentials()) is None
|
||||
assert secrets.reads == []
|
||||
assert supervisor.tasks.list() == []
|
||||
assert supervisor._bridges == {}
|
||||
|
||||
# Synthetic probe evidence and explicit activation belong to the new revision.
|
||||
profiles.record_capabilities(
|
||||
updated.profile_id,
|
||||
expected_revision=updated.revision,
|
||||
capabilities=profile.capabilities,
|
||||
secrets=secrets,
|
||||
)
|
||||
updated = profiles.activate(updated.profile_id, secrets)
|
||||
assert updated.is_active is True
|
||||
secrets.reads.clear()
|
||||
|
||||
after = asyncio.run(supervisor._active_credentials())
|
||||
|
||||
assert before[0].revision == 1 and before[1] == "synthetic-credential"
|
||||
assert after == (profiles.snapshot(updated.profile_id, updated.revision), "synthetic-new")
|
||||
assert after[0].revision == 2
|
||||
assert len(secrets.reads) == 1
|
||||
|
||||
|
||||
def test_no_active_profile_does_not_read_credentials(setup):
|
||||
supervisor, profiles, secrets, _profile = setup
|
||||
with profiles._connect(writing=True) as db:
|
||||
db.execute("UPDATE model_profiles SET is_active=0")
|
||||
|
||||
assert asyncio.run(supervisor._active_credentials()) is None
|
||||
assert secrets.reads == []
|
||||
|
||||
|
||||
def test_keyless_profile_does_not_require_or_resolve_a_secret(setup):
|
||||
supervisor, profiles, secrets, _profile = setup
|
||||
keyless = create_profile(profiles, secrets, keyed=False)
|
||||
secrets.reads.clear()
|
||||
|
||||
result = asyncio.run(supervisor._active_credentials())
|
||||
|
||||
assert result == (profiles.snapshot(keyless.profile_id, keyless.revision), None)
|
||||
assert result[0].has_api_key is False
|
||||
assert secrets.reads == []
|
||||
@@ -4,12 +4,11 @@ The public target architecture is defined by ``docs/ARCHITECTURE.md``:
|
||||
|
||||
* Hy Memory SDK owns memory/raw/vector/graph/coding data;
|
||||
* MemHub owns the management plane and may retain ``memory_id`` references;
|
||||
* product code reaches the SDK through ``hy_memory_bridge`` only.
|
||||
* product code reaches the SDK only through narrow runtime/migration modules;
|
||||
* MemHub calls public ``HyMemoryClient`` methods only.
|
||||
|
||||
The legacy implementation still contains Evidence/Governance writers and a
|
||||
small number of private SDK-store accesses. Those are migration debt, not an
|
||||
allowance for new code. The explicit debt lists below may shrink during the
|
||||
refactor; new entries must not be added.
|
||||
The sole exception is the read-only Kuzu projection/admin surface used by the
|
||||
existing graph UI and Graph Width workflow.
|
||||
|
||||
The gate is deliberately not a blanket ban on ``sqlite3``. Adapters may read a
|
||||
host product's SQLite through the guarded read-only module; they may not own
|
||||
@@ -58,13 +57,13 @@ HOST_READONLY_MODULE = "one_memory.imports.sqlite_access"
|
||||
SRC_ROOT = PACKAGE_ROOT.parents[0]
|
||||
PACKAGE_LABEL = PACKAGE_ROOT.name
|
||||
|
||||
HY_BRIDGE = f"{PACKAGE_LABEL}/hy_memory_bridge.py"
|
||||
|
||||
#: The one module allowed to *invoke* the SDK.
|
||||
SDK_CALLER = HY_BRIDGE
|
||||
|
||||
#: Type-only importers of HY datatypes; they hold no client and write nothing.
|
||||
HY_TYPE_ONLY: frozenset[str] = frozenset()
|
||||
SDK_MODULES = frozenset(
|
||||
{
|
||||
f"{PACKAGE_LABEL}/hy_sdk_runtime.py",
|
||||
f"{PACKAGE_LABEL}/hy_sdk_migration.py",
|
||||
}
|
||||
)
|
||||
SDK_CLIENT_CONSTRUCTOR = f"{PACKAGE_LABEL}/hy_sdk_runtime.py"
|
||||
|
||||
# Frozen migration debt: modules that still write the legacy memory/raw
|
||||
# authority. The target count is zero. A new module cannot join this set.
|
||||
@@ -84,19 +83,37 @@ LEGACY_AUTHORITY_WRITE = re.compile(
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Existing accesses to SDK implementation details. These are exact file /
|
||||
# attribute debts so deleting one is always green, while adding one fails.
|
||||
SDK_PRIVATE_ACCESS_DEBT: dict[str, frozenset[str]] = {
|
||||
f"{PACKAGE_LABEL}/hy_lifecycle.py": frozenset({"_collection", "_vector_store"}),
|
||||
f"{PACKAGE_LABEL}/hy_memory_bridge.py": frozenset(
|
||||
{"_coding_store", "_graph_store", "_loop_thread", "_vector_store"}
|
||||
),
|
||||
f"{PACKAGE_LABEL}/hy_response_guard.py": frozenset({"_loop_thread", "_registry"}),
|
||||
}
|
||||
|
||||
SDK_PRIVATE_ATTRIBUTES = frozenset(
|
||||
attribute for attributes in SDK_PRIVATE_ACCESS_DEBT.values() for attribute in attributes
|
||||
{
|
||||
"_coding_store",
|
||||
"_coding_stores",
|
||||
"_collection",
|
||||
"_embed_service",
|
||||
"_graph_store",
|
||||
"_loop_thread",
|
||||
"_readers",
|
||||
"_reader",
|
||||
"_reconciler",
|
||||
"_registry",
|
||||
"_run",
|
||||
"_shared_embed_service",
|
||||
"_spawn_search_dedup",
|
||||
"_system2_writer",
|
||||
"_vector_store",
|
||||
"_writer",
|
||||
}
|
||||
)
|
||||
SDK_PRIVATE_ACCESS_EXCEPTIONS = {
|
||||
f"{PACKAGE_LABEL}/hy_graph_projection.py": frozenset(
|
||||
{"_graph_store", "_loop_thread", "_run"}
|
||||
),
|
||||
# Startup migration runs while Runtime writers are stopped. It is the one
|
||||
# non-runtime path allowed to manipulate SDK storage directly so that
|
||||
# migration-only mechanics never become public HyMemoryClient methods.
|
||||
f"{PACKAGE_LABEL}/hy_sdk_migration.py": frozenset(
|
||||
{"_embed_service", "_graph_store", "_loop_thread", "_vector_store"}
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _relative(path: Path) -> str:
|
||||
@@ -171,49 +188,26 @@ def test_adapters_use_only_the_readonly_host_sqlite_module() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_hy_sdk_is_confined_to_the_l1_adapter() -> None:
|
||||
"""A0-01: HY SDK stays behind HyMemoryPort; entries must not import it.
|
||||
|
||||
This is the executable form of "Function/MCP/scan/Hook must not call the
|
||||
HY SDK directly". `hy_memory_bridge` is the current L1 adapter and the
|
||||
permission stores quote HY types, so they are the allowed set; anything
|
||||
else reaching for the SDK is a new bypass and must fail here.
|
||||
|
||||
Only `hy_memory_bridge` *calls* the SDK. The two permission stores merely
|
||||
import HY datatypes (`CodingMemory`, `WorkingFact`, memory models) to type
|
||||
what they persist; they hold no client and issue no writes. They are
|
||||
listed here so the gate stays green on today's truth while still failing
|
||||
on any new module that reaches for the SDK.
|
||||
"""
|
||||
allowed = frozenset({HY_BRIDGE}) | HY_TYPE_ONLY
|
||||
def test_hy_sdk_is_confined_to_public_sdk_modules() -> None:
|
||||
"""Only the runtime and offline migration adapters may import the SDK."""
|
||||
offenders: list[str] = []
|
||||
for path in _all_module_files():
|
||||
relative = _relative(path)
|
||||
if relative in allowed:
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
names = _module_names(tree)
|
||||
if not any(_root(name) in HY_SDK_ROOTS for name in names):
|
||||
continue
|
||||
names = _module_names(ast.parse(path.read_text(encoding="utf-8")))
|
||||
if any(_root(name) in HY_SDK_ROOTS for name in names):
|
||||
if relative not in SDK_MODULES:
|
||||
offenders.append(relative)
|
||||
assert not offenders, f"HY SDK must stay in the L1 adapter: {offenders}"
|
||||
assert not offenders, f"HY SDK imports must stay in {sorted(SDK_MODULES)}: {offenders}"
|
||||
|
||||
|
||||
def test_only_the_l1_adapter_constructs_a_hy_client() -> None:
|
||||
"""A0-01: type-only imports are tolerated; constructing a client is not.
|
||||
|
||||
A permission store quoting `CodingMemory` cannot write a memory. A module
|
||||
that instantiates `HyMemoryClient` can. So the stricter check is on client
|
||||
construction, and it must pass today rather than merely documenting an
|
||||
aspiration.
|
||||
|
||||
The check is on the AST, not on the substring: a docstring or comment that
|
||||
*mentions* `HyMemoryClient` builds nothing. Matching text would flag prose
|
||||
and, worse, would keep passing while a real caller is added inside a
|
||||
function the text search happened to like.
|
||||
"""
|
||||
def test_only_runtime_constructs_a_hy_client() -> None:
|
||||
"""Migration borrows a lease; only runtime constructs clients."""
|
||||
offenders: list[str] = []
|
||||
for path in _all_module_files():
|
||||
relative = _relative(path)
|
||||
if relative == SDK_CALLER:
|
||||
if relative == SDK_CLIENT_CONSTRUCTOR:
|
||||
continue
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
for node in ast.walk(tree):
|
||||
@@ -233,7 +227,7 @@ def test_only_the_l1_adapter_constructs_a_hy_client() -> None:
|
||||
if any(alias.name == "HyMemoryClient" for alias in node.names):
|
||||
offenders.append(relative)
|
||||
break
|
||||
assert not offenders, f"only the L1 adapter may build a HY client: {offenders}"
|
||||
assert not offenders, f"only {SDK_CLIENT_CONSTRUCTOR} may build a HY client: {offenders}"
|
||||
|
||||
|
||||
def test_no_new_module_writes_legacy_memory_authority() -> None:
|
||||
@@ -247,41 +241,90 @@ def test_no_new_module_writes_legacy_memory_authority() -> None:
|
||||
if relative not in LEGACY_AUTHORITY_WRITERS:
|
||||
offenders.append(relative)
|
||||
assert not offenders, (
|
||||
"new Evidence/Governance writers are forbidden; write memory/raw via "
|
||||
f"the SDK bridge instead: {offenders}"
|
||||
"new Evidence/Governance writers are forbidden; use public SDK client "
|
||||
f"methods instead: {offenders}"
|
||||
)
|
||||
|
||||
|
||||
def test_no_new_sdk_private_store_access() -> None:
|
||||
"""Freeze direct SDK-store access until public client APIs replace it."""
|
||||
def test_no_sdk_private_implementation_access() -> None:
|
||||
"""Runtime uses public APIs; only Graph and stopped-writer migration are privileged."""
|
||||
offenders: list[str] = []
|
||||
for path in _all_module_files():
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
relative = _relative(path)
|
||||
names = _module_names(tree)
|
||||
if relative not in SDK_PRIVATE_ACCESS_DEBT and not any(
|
||||
_root(name) in HY_SDK_ROOTS for name in names
|
||||
):
|
||||
# Names such as `_registry` and `_collection` are common for
|
||||
# MemHub-owned objects too. They become SDK-private access only
|
||||
# in a module that imports the SDK (or is frozen debt already).
|
||||
continue
|
||||
used = {
|
||||
node.attr
|
||||
for node in ast.walk(tree)
|
||||
if isinstance(node, ast.Attribute) and node.attr in SDK_PRIVATE_ATTRIBUTES
|
||||
if isinstance(node, ast.Attribute)
|
||||
and node.attr in SDK_PRIVATE_ATTRIBUTES
|
||||
and (
|
||||
relative.startswith(f"{PACKAGE_LABEL}/hy_")
|
||||
or (
|
||||
isinstance(node.value, ast.Name)
|
||||
and node.value.id in {"client", "native", "concrete"}
|
||||
)
|
||||
)
|
||||
}
|
||||
if not used:
|
||||
continue
|
||||
unexpected = sorted(used - SDK_PRIVATE_ACCESS_DEBT.get(relative, frozenset()))
|
||||
if unexpected:
|
||||
offenders.append(f"{relative}: {unexpected}")
|
||||
if used:
|
||||
unexpected = used - SDK_PRIVATE_ACCESS_EXCEPTIONS.get(
|
||||
relative, frozenset()
|
||||
)
|
||||
if unexpected:
|
||||
offenders.append(f"{relative}: {sorted(unexpected)}")
|
||||
assert not offenders, (
|
||||
"new SDK private-store access is forbidden; add a public HyMemoryClient "
|
||||
f"API and call it through hy_memory_bridge: {offenders}"
|
||||
"SDK private implementation access is forbidden outside the isolated "
|
||||
f"Graph and offline-migration boundaries: {offenders}"
|
||||
)
|
||||
|
||||
|
||||
def test_sdk_modules_import_only_the_public_package_root() -> None:
|
||||
"""Runtime imports only public SDK symbols; offline migration is privileged."""
|
||||
offenders: list[str] = []
|
||||
for path in _all_module_files():
|
||||
relative = _relative(path)
|
||||
if relative == f"{PACKAGE_LABEL}/hy_sdk_migration.py":
|
||||
continue
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import) and any(
|
||||
alias.name.startswith("hy_memory.") for alias in node.names
|
||||
):
|
||||
offenders.append(relative)
|
||||
break
|
||||
if isinstance(node, ast.ImportFrom) and (
|
||||
node.module or ""
|
||||
).startswith("hy_memory."):
|
||||
offenders.append(relative)
|
||||
break
|
||||
if (
|
||||
isinstance(node, ast.Call)
|
||||
and isinstance(node.func, ast.Name)
|
||||
and node.func.id == "import_module"
|
||||
and node.args
|
||||
and isinstance(node.args[0], ast.Constant)
|
||||
and isinstance(node.args[0].value, str)
|
||||
and node.args[0].value.startswith("hy_memory.")
|
||||
):
|
||||
offenders.append(relative)
|
||||
break
|
||||
assert not offenders, f"import SDK symbols from `hy_memory` only: {offenders}"
|
||||
|
||||
|
||||
def test_memhub_never_patches_sdk_prompts() -> None:
|
||||
offenders: list[str] = []
|
||||
forbidden = (
|
||||
"extract_system_prompt_en",
|
||||
"extract_system_prompt_zh",
|
||||
"hy_memory.agent.prompts",
|
||||
)
|
||||
for path in _all_module_files():
|
||||
source = path.read_text(encoding="utf-8")
|
||||
hits = [token for token in forbidden if token in source]
|
||||
if hits:
|
||||
offenders.append(f"{_relative(path)}: {hits}")
|
||||
assert not offenders, f"SDK prompt patching is forbidden: {offenders}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("driver", sorted(MEMHUB_DRIVERS))
|
||||
def test_no_new_module_adopts_a_memhub_driver_outside_the_data_layer(driver: str) -> None:
|
||||
"""A0-01: guard-rail against A1 sprawl, not a demand that today is clean.
|
||||
|
||||
@@ -301,11 +301,15 @@ def test_sdk_migrate_runs_offline_backfill_without_cutover(tmp_path, monkeypatch
|
||||
class Secrets:
|
||||
pass
|
||||
|
||||
class Bridge:
|
||||
def __init__(self, **_kwargs):
|
||||
pass
|
||||
class Lease:
|
||||
client = SimpleNamespace(
|
||||
_loop_thread=None,
|
||||
_vector_store=None,
|
||||
_embed_service=None,
|
||||
_graph_store=None,
|
||||
)
|
||||
|
||||
async def close(self):
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
async def backfill(**_kwargs):
|
||||
@@ -320,7 +324,10 @@ def test_sdk_migrate_runs_offline_backfill_without_cutover(tmp_path, monkeypatch
|
||||
|
||||
monkeypatch.setattr("one_memory.model_profiles.ModelProfileStore", Profiles)
|
||||
monkeypatch.setattr("one_memory.secrets.KeyringSecretStore", Secrets)
|
||||
monkeypatch.setattr("one_memory.hy_memory_bridge.HyMemoryBridge", Bridge)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.hy_sdk_runtime.acquire_client",
|
||||
lambda *_args, **_kwargs: Lease(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.sdk_migration_runner.run_sdk_data_plane_backfill",
|
||||
backfill,
|
||||
@@ -360,6 +367,14 @@ def test_sdk_cutover_is_offline_confirmed_and_returns_marker(tmp_path, monkeypat
|
||||
completed_at="2026-10-21T00:00:00Z",
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.sdk_migration_runner.activate_sdk_migration_staging",
|
||||
lambda *, commit, **_kwargs: commit(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.sdk_migration_runner.resume_legacy_cleanup",
|
||||
lambda **_kwargs: (),
|
||||
)
|
||||
output = io.StringIO()
|
||||
|
||||
code = cli.run(
|
||||
|
||||
@@ -50,21 +50,3 @@ def test_governed_memory_carries_the_same_fields_on_both_sides() -> None:
|
||||
}
|
||||
|
||||
assert typescript == set(GovernedMemory.model_fields)
|
||||
|
||||
|
||||
def test_an_agent_owned_memory_is_usable_and_never_passes_as_the_person(tmp_path) -> None:
|
||||
"""助手输出里的事实直接可用,但始终标成助手说的。
|
||||
|
||||
召回读到它时,origin 让别的 AI 应用知道这是模型的陈述、不是本人的判决。
|
||||
已有的同内容记忆,也不会被模型输出改掉来源标记。
|
||||
"""
|
||||
from tests.test_memory_processing import (
|
||||
test_agent_output_keeps_the_origin_of_a_same_content_memory,
|
||||
test_agent_owned_memories_land_usable_but_marked_as_the_assistant,
|
||||
)
|
||||
|
||||
created, existing = tmp_path / "created", tmp_path / "existing"
|
||||
created.mkdir()
|
||||
existing.mkdir()
|
||||
test_agent_owned_memories_land_usable_but_marked_as_the_assistant(created)
|
||||
test_agent_output_keeps_the_origin_of_a_same_content_memory(existing, MemoryOrigin.AGENT)
|
||||
@@ -1,10 +1,8 @@
|
||||
"""Searching imported conversations before any generation model is configured.
|
||||
|
||||
This search is literal, not semantic, and that is a correction rather than a
|
||||
shortcut. The vector store `semantic_search` reads is written only by
|
||||
extraction (`memory_processing.py` EXTRACT tasks), and extraction is precisely
|
||||
what a user without a generation model does not have: their store is empty, so
|
||||
a semantic path would return nothing no matter how it was wired.
|
||||
shortcut. Literal conversation search must remain available before a model is
|
||||
configured and independently of SDK semantic-memory extraction.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
@@ -4,9 +4,9 @@ from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from one_memory.conversation_processing import _conversation_chunks
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_processing import _conversation_chunks
|
||||
|
||||
|
||||
def _records(tmp_path, *speakers: tuple[str, str, str]) -> list:
|
||||
|
||||
@@ -1,58 +1,16 @@
|
||||
import asyncio
|
||||
import threading
|
||||
import time
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from test_memory_qa import _grounded_setup, _ndjson_stream
|
||||
from test_onboarding_profile import FakeGraph, _app, _client, _setup, _supervisor
|
||||
from test_onboarding_profile import _app, _client
|
||||
|
||||
from one_memory.memory_qa import MemoryAnswerer
|
||||
from one_memory.model_probe import ModelProber
|
||||
from one_memory.secrets import SecretStoreUnavailableError
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"operation,args",
|
||||
[
|
||||
("digest", ()),
|
||||
("graph_snapshot", ()),
|
||||
("graph_width_mismatch", ()),
|
||||
("semantic_search", ("example",)),
|
||||
("update_hy_memory", ("id", "corrected")),
|
||||
("delete_hy_memory", ("id",)),
|
||||
("decide_graph_width", ("keep_existing",)),
|
||||
],
|
||||
)
|
||||
@pytest.mark.parametrize("phase", ["active", "secret"])
|
||||
def test_credentials_do_not_block_the_event_loop(tmp_path, monkeypatch, operation, args, phase):
|
||||
stores = _setup(tmp_path)
|
||||
supervisor = _supervisor(tmp_path, stores, FakeGraph(()))
|
||||
method = "active_identity" if phase == "active" else "resolve_snapshot_secret"
|
||||
original = getattr(supervisor.profiles, method)
|
||||
called = threading.Event()
|
||||
|
||||
def waiting(*args, **kwargs):
|
||||
called.set()
|
||||
time.sleep(0.25)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(supervisor.profiles, method, waiting)
|
||||
|
||||
async def scenario():
|
||||
task = asyncio.create_task(getattr(supervisor, operation)(*args))
|
||||
started = time.monotonic()
|
||||
await asyncio.sleep(0.04)
|
||||
elapsed = time.monotonic() - started
|
||||
# Some graph operations have no fake implementation; those exceptions
|
||||
# are irrelevant to whether a credential read stops the event loop.
|
||||
await asyncio.gather(task, return_exceptions=True)
|
||||
assert called.is_set()
|
||||
assert elapsed < 0.18, f"Credential wait blocked the loop for {elapsed:.3f}s"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_disabled_automatic_processing_does_not_digest_onboarding(tmp_path):
|
||||
async def scenario():
|
||||
app = _app(tmp_path)
|
||||
@@ -64,7 +22,7 @@ def test_disabled_automatic_processing_does_not_digest_onboarding(tmp_path):
|
||||
calls.append(True)
|
||||
return {}
|
||||
|
||||
app.state.memory_processing_supervisor.digest = digest
|
||||
app.state.sdk_memory_runtime.digest = digest
|
||||
response = await client.post("/api/v1/onboarding/complete")
|
||||
assert response.status_code == 200
|
||||
assert calls == []
|
||||
|
||||
@@ -35,7 +35,7 @@ def test_live_log_level_applies_to_full_hy_and_provider_call_chain(
|
||||
set_diagnostic_log_level(tmp_path, "debug")
|
||||
|
||||
assert diagnostic_log_level(tmp_path) == "debug"
|
||||
assert logging.getLogger("one_memory.memory_processing").isEnabledFor(logging.DEBUG)
|
||||
assert logging.getLogger("one_memory.sdk_write_processor").isEnabledFor(logging.DEBUG)
|
||||
assert logging.getLogger("hy_memory.runtime").isEnabledFor(logging.DEBUG)
|
||||
assert logging.getLogger("hy_memory.client").isEnabledFor(logging.DEBUG)
|
||||
assert logging.getLogger("hy_memory.agent.llm_provider").isEnabledFor(logging.DEBUG)
|
||||
|
||||
@@ -1,400 +0,0 @@
|
||||
"""A bounded, read-only discovery primitive for the H23 Evidence reconcile.
|
||||
|
||||
The objective's crash requirement is "a committed Evidence row must not be
|
||||
lost". The two-step `Evidence write -> queue/task submit` window can leave a
|
||||
row with no write entry. C0 scheduled that reconcile for H23 and asked C2 to
|
||||
supply the primitive; this module pins the discovery half, which is pure
|
||||
database infrastructure and does not choose the product's grouping semantics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
|
||||
from one_memory.memory_server.durable_queue import (
|
||||
DurableWriteQueue,
|
||||
ReconcileSchemaUnavailableError,
|
||||
)
|
||||
|
||||
|
||||
def _put(store: EvidenceStore, message_id: str) -> str:
|
||||
return store.put(
|
||||
EvidenceInput(
|
||||
source_id="s1",
|
||||
conversation_id="c1",
|
||||
message_id=message_id,
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim=f"body {message_id}",
|
||||
created_at=datetime.now(UTC),
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record.evidence_id
|
||||
|
||||
|
||||
def test_unsubmitted_evidence_finds_rows_with_no_write_entry(tmp_path) -> None:
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
queue = DurableWriteQueue(path)
|
||||
submitted = _put(store, "m1")
|
||||
orphan = _put(store, "m2")
|
||||
queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[submitted],
|
||||
idempotency_key="idem-1",
|
||||
)
|
||||
)
|
||||
|
||||
found = queue.unsubmitted_evidence_ids(limit=10)
|
||||
|
||||
assert found == (orphan,)
|
||||
|
||||
|
||||
def test_reconcile_refuses_when_the_evidence_schema_is_not_migrated(tmp_path) -> None:
|
||||
"""A queue-only database is not an empty reconcile result.
|
||||
|
||||
Returning `()` here would make a pre-migration database look as if every
|
||||
trusted Evidence row were already queued. The caller needs a distinct
|
||||
retryable state so it can run migration first.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
queue = DurableWriteQueue(path)
|
||||
|
||||
with pytest.raises(ReconcileSchemaUnavailableError, match="run migrations"):
|
||||
queue.unsubmitted_evidence_ids()
|
||||
|
||||
|
||||
def test_unsubmitted_evidence_is_bounded_and_resumable(tmp_path) -> None:
|
||||
"""The cursor must be stable and bounded so a startup pass is cheap.
|
||||
|
||||
`after_cursor` is the last id a previous page returned; the next call
|
||||
resumes strictly after it. In this serial fixture the pages partition the
|
||||
backlog once. The primitive is not a cross-call transaction snapshot; a
|
||||
concurrent insert or submit is tolerated by the Evidence-derived
|
||||
idempotency key, not by pretending the read itself is isolated.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
queue = DurableWriteQueue(path)
|
||||
ids = [_put(store, f"m{index}") for index in range(3)]
|
||||
|
||||
first = queue.unsubmitted_evidence_ids(limit=2)
|
||||
second = queue.unsubmitted_evidence_ids(limit=2, after_cursor=first[-1])
|
||||
|
||||
assert first == (ids[0], ids[1])
|
||||
assert second == (ids[2],)
|
||||
|
||||
|
||||
def test_unsubmitted_evidence_refuses_a_missing_cursor(tmp_path) -> None:
|
||||
"""A vanished cursor must not silently reset pagination to page one.
|
||||
|
||||
Replaying a prefix may be harmless only because the idempotency key exists;
|
||||
the discovery contract itself must fail closed. A caller that deleted the
|
||||
anchor or restored an older database must restart from a fresh scan, not
|
||||
have the primitive quietly change its place.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
queue = DurableWriteQueue(path)
|
||||
first = _put(store, "m1")
|
||||
second = _put(store, "m2")
|
||||
|
||||
first_page = queue.unsubmitted_evidence_ids(limit=1)
|
||||
assert first_page == (first,)
|
||||
with sqlite3.connect(path) as connection, connection:
|
||||
connection.execute("DELETE FROM evidence_origins WHERE evidence_id=?", (first,))
|
||||
connection.execute("DELETE FROM evidence WHERE evidence_id=?", (first,))
|
||||
|
||||
with pytest.raises(RuntimeError, match="cursor is unavailable"):
|
||||
queue.unsubmitted_evidence_ids(limit=10, after_cursor=first)
|
||||
|
||||
# A deliberate fresh pass still sees the remaining orphan.
|
||||
assert queue.unsubmitted_evidence_ids(limit=10) == (second,)
|
||||
|
||||
|
||||
def test_unsubmitted_evidence_query_uses_the_reference_index(tmp_path) -> None:
|
||||
"""The anti-join must not scan every queue payload.
|
||||
|
||||
At production scale the queue can keep many terminal rows, so discovery
|
||||
uses a dedicated Evidence reference and an indexed `NOT EXISTS` join
|
||||
instead of loading every job's payload or reference JSON.
|
||||
"""
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.memory_server.migrations import memory_write_jobs_sqlite_schema
|
||||
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
with main_database(path).write(autocommit=True) as connection, connection:
|
||||
connection.execute(
|
||||
"CREATE TABLE evidence (evidence_id TEXT PRIMARY KEY, imported_at TEXT NOT NULL)"
|
||||
)
|
||||
connection.execute(
|
||||
"CREATE INDEX evidence_imported_order ON evidence(imported_at, evidence_id)"
|
||||
)
|
||||
for statement in memory_write_jobs_sqlite_schema():
|
||||
connection.execute(statement)
|
||||
plan = [
|
||||
str(row[3])
|
||||
for row in connection.execute(
|
||||
"EXPLAIN QUERY PLAN SELECT evidence.evidence_id FROM evidence "
|
||||
"WHERE evidence.imported_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
|
||||
"WHERE ref.evidence_id=evidence.evidence_id) "
|
||||
"ORDER BY evidence.imported_at, evidence.evidence_id LIMIT ?",
|
||||
(512,),
|
||||
)
|
||||
]
|
||||
|
||||
joined = " | ".join(plan)
|
||||
assert "idx_write_job_evidence_evidence" in joined
|
||||
assert "SCAN memory_write_jobs" not in joined
|
||||
assert "SCAN ref" not in joined
|
||||
|
||||
|
||||
def test_the_shipped_schema_serves_the_reconcile_order(tmp_path) -> None:
|
||||
"""The real schema must not scan+sort Evidence on every discovery page.
|
||||
|
||||
`unsubmitted_evidence_ids` orders by `(imported_at, evidence_id)` with no
|
||||
`user_id` predicate, so `evidence_imported_index` (leading column
|
||||
`user_id`) cannot serve it. Before the reconcile-order index the real plan
|
||||
was `SCAN evidence` + `USE TEMP B-TREE FOR ORDER BY`; a pass over a large
|
||||
corpus paid that on every page. This pins the shipped schema, not a
|
||||
test-local index, so a regression to the old DDL fails here.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
|
||||
from one_memory.memory_server.migrations import MIGRATIONS
|
||||
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
asyncio.run(migrate(SqliteMigrationStore(path), MIGRATIONS))
|
||||
|
||||
with main_database(path).read() as connection:
|
||||
plan = [
|
||||
str(row[3])
|
||||
for row in connection.execute(
|
||||
"EXPLAIN QUERY PLAN SELECT evidence.evidence_id FROM evidence "
|
||||
"WHERE evidence.imported_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
|
||||
"WHERE ref.evidence_id=evidence.evidence_id) "
|
||||
"ORDER BY evidence.imported_at, evidence.evidence_id LIMIT ?",
|
||||
(512,),
|
||||
)
|
||||
]
|
||||
|
||||
joined = " | ".join(plan)
|
||||
assert "evidence_reconcile_order" in joined, joined
|
||||
assert "USE TEMP B-TREE FOR ORDER BY" not in joined, joined
|
||||
# SQLite reports an in-order covering-index walk as `SCAN evidence USING
|
||||
# COVERING INDEX ...`; the failure mode we forbid is a bare table scan with
|
||||
# a temp sort, so require the index to be named and the sort to be gone.
|
||||
assert "USING COVERING INDEX evidence_reconcile_order" in joined, joined
|
||||
|
||||
|
||||
def test_unsubmitted_evidence_returns_empty_when_all_are_queued(tmp_path) -> None:
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
queue = DurableWriteQueue(path)
|
||||
evidence_id = _put(store, "m1")
|
||||
queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[evidence_id],
|
||||
idempotency_key="idem-1",
|
||||
)
|
||||
)
|
||||
|
||||
assert queue.unsubmitted_evidence_ids(limit=10) == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("limit", [0, -1, True, 1.5, "10"])
|
||||
def test_unsubmitted_evidence_rejects_an_invalid_limit(tmp_path, limit: object) -> None:
|
||||
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
|
||||
with pytest.raises(ValueError, match="limit must be positive"):
|
||||
queue.unsubmitted_evidence_ids(limit=limit) # type: ignore[arg-type]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cursor", ["", 1, True, ["evidence-id"], object()])
|
||||
def test_unsubmitted_evidence_rejects_an_invalid_cursor(tmp_path, cursor: object) -> None:
|
||||
"""A malformed cursor must fail with the public contract, not SQL errors.
|
||||
|
||||
``after_cursor`` crosses an API/worker boundary and is allowed to be
|
||||
absent, but every present value must be a non-empty evidence id. Passing
|
||||
a list/object into the lookup produced a raw SQLite binding error; an
|
||||
empty string performed a pointless lookup and then reported a missing
|
||||
cursor as if the database had lost a real row. The primitive should
|
||||
reject the caller's input before touching SQL.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
with sqlite3.connect(path) as connection, connection:
|
||||
connection.execute(
|
||||
"CREATE TABLE evidence (evidence_id TEXT PRIMARY KEY, imported_at TEXT NOT NULL)"
|
||||
)
|
||||
queue = DurableWriteQueue(path)
|
||||
|
||||
with pytest.raises(ValueError, match="after_cursor must be a non-empty string"):
|
||||
queue.unsubmitted_evidence_ids(after_cursor=cursor) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_a_reconcile_pass_enqueues_an_orphan_exactly_once(tmp_path) -> None:
|
||||
"""The acceptance's "enqueue once" half, on the C2-owned primitives.
|
||||
|
||||
A crash between Evidence commit and submit leaves an orphan. The reconcile
|
||||
flow is: discover it, then submit a write with a stable idempotency key
|
||||
derived from the Evidence identity. A second pass must find nothing (the
|
||||
row now references it) and must not create a duplicate — the same
|
||||
idempotency key would in any case return the existing job.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
queue = DurableWriteQueue(path)
|
||||
orphan = _put(store, "m1") # evidence committed, no submit -> the crash window
|
||||
|
||||
def reconcile_once() -> str | None:
|
||||
ids = queue.unsubmitted_evidence_ids(limit=64)
|
||||
if not ids:
|
||||
return None
|
||||
first = ids[0]
|
||||
record = queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[first],
|
||||
# Derived from the Evidence identity: a replay re-derives the
|
||||
# same key, so the queue refuses to create a second job.
|
||||
idempotency_key=f"reconcile:{first}",
|
||||
)
|
||||
)
|
||||
return record.job_id
|
||||
|
||||
first_job = reconcile_once()
|
||||
assert first_job is not None
|
||||
|
||||
# Second pass: the orphan is gone, and a forced replay is idempotent.
|
||||
assert queue.unsubmitted_evidence_ids(limit=64) == ()
|
||||
replay = queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[orphan],
|
||||
idempotency_key=f"reconcile:{orphan}",
|
||||
)
|
||||
)
|
||||
assert replay.job_id == first_job
|
||||
assert replay.enqueued is False
|
||||
with sqlite3.connect(path) as connection:
|
||||
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_two_reconcile_workers_submit_the_same_orphan_once(tmp_path) -> None:
|
||||
"""A restart and a periodic pass may race; the derived key is the arbiter.
|
||||
|
||||
The discovery cursor cannot promise a cross-call snapshot, so two workers
|
||||
can both observe the same orphan. They must not create two jobs:
|
||||
`reconcile:<evidence_id>` is derived from the Evidence identity and the
|
||||
queue's UNIQUE idempotency key makes the second submit an idempotent replay.
|
||||
"""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
store = EvidenceStore(path)
|
||||
DurableWriteQueue(path)
|
||||
orphan = _put(store, "m1")
|
||||
|
||||
def reconcile_worker() -> str:
|
||||
request = WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[orphan],
|
||||
idempotency_key=f"reconcile:{orphan}",
|
||||
)
|
||||
return DurableWriteQueue(path).submit(request).job_id
|
||||
|
||||
first = reconcile_worker()
|
||||
second = reconcile_worker()
|
||||
|
||||
assert first == second
|
||||
with sqlite3.connect(path) as connection:
|
||||
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
|
||||
assert (
|
||||
connection.execute(
|
||||
"SELECT COUNT(*) FROM memory_write_job_evidence WHERE evidence_id=?",
|
||||
(orphan,),
|
||||
).fetchone()[0]
|
||||
== 1
|
||||
)
|
||||
|
||||
|
||||
def test_a_sigkill_before_submit_leaves_an_orphan_the_primitive_recovers(tmp_path) -> None:
|
||||
"""Real process crash between Evidence commit and submit (objective 4.1-4.2).
|
||||
|
||||
The child writes Evidence and is SIGKILLed before it can submit the write.
|
||||
No task/queue row exists, so only a *discovery over committed rows* can find
|
||||
it. This is the boundary the C1 file-level drain does not cover for manual
|
||||
notes; the primitive + derived key then enqueue it exactly once.
|
||||
"""
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
if sys.platform == "win32":
|
||||
pytest.skip("POSIX process-kill recovery")
|
||||
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
EvidenceStore(path)
|
||||
child = (
|
||||
"import time\n"
|
||||
"from datetime import UTC, datetime\n"
|
||||
"from pathlib import Path\n"
|
||||
"from one_memory.imports.models import ("
|
||||
"EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy)\n"
|
||||
"from one_memory.imports.storage import EvidenceStore\n"
|
||||
"path = Path(" + repr(str(path)) + ")\n"
|
||||
"store = EvidenceStore(path)\n"
|
||||
"store.put(EvidenceInput(source_id='s1', conversation_id='c1', message_id='m1',\n"
|
||||
" role='user', written_by=WrittenBy.PERSON, verbatim='orphan',\n"
|
||||
" created_at=datetime.now(UTC), origin_meta={EvidenceOrigin.FILE: OriginMetadata()}))\n"
|
||||
"print('written', flush=True)\n"
|
||||
"time.sleep(60)\n"
|
||||
)
|
||||
process = subprocess.Popen(
|
||||
[sys.executable, "-c", child],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
cwd=str(Path(__file__).resolve().parents[1]),
|
||||
env={**os.environ, "PYTHONPATH": str(Path(__file__).resolve().parents[1] / "src")},
|
||||
)
|
||||
assert process.stdout is not None
|
||||
assert process.stdout.readline().strip() == "written"
|
||||
os.kill(process.pid, signal.SIGKILL)
|
||||
process.wait(timeout=10)
|
||||
|
||||
queue = DurableWriteQueue(path)
|
||||
ids = queue.unsubmitted_evidence_ids(limit=10)
|
||||
assert len(ids) == 1, "the committed Evidence must be discoverable after the crash"
|
||||
queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id="c1",
|
||||
batch_identity=BatchIdentity(internal_session_id="c1", first_round=1, last_round=1),
|
||||
evidence_ids=[ids[0]],
|
||||
idempotency_key=f"reconcile:{ids[0]}",
|
||||
)
|
||||
)
|
||||
assert queue.unsubmitted_evidence_ids(limit=10) == ()
|
||||
with sqlite3.connect(path) as connection:
|
||||
assert connection.execute("SELECT COUNT(*) FROM memory_write_jobs").fetchone()[0] == 1
|
||||
@@ -1,131 +0,0 @@
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryDraft, MemoryGovernanceStore, MemoryKind
|
||||
from tests.person_door import as_person
|
||||
|
||||
|
||||
def test_permission_guard_is_rechecked_inside_write_transaction_and_rolls_back(tmp_path):
|
||||
database = tmp_path / "test.sqlite"
|
||||
evidence = EvidenceStore(database)
|
||||
original = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="one",
|
||||
message_id="one",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="synthetic",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
store = MemoryGovernanceStore(database)
|
||||
attempts = []
|
||||
|
||||
def guard():
|
||||
attempts.append("check")
|
||||
if len(attempts) == 2:
|
||||
raise RuntimeError("permission revoked before commit")
|
||||
|
||||
with pytest.raises(RuntimeError, match="revoked before commit"), store.guard_writes(guard):
|
||||
as_person(store).create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content="synthetic result",
|
||||
evidence_ids=(original.evidence_id,),
|
||||
scopes=("all-apps",),
|
||||
),
|
||||
request_id="attempt",
|
||||
)
|
||||
assert attempts == ["check", "check"]
|
||||
assert store.list(user_id="local") == []
|
||||
with sqlite3.connect(database) as db:
|
||||
assert db.execute("SELECT count(*) FROM governed_memory_mutations").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_guard_context_does_not_leak_into_subsequent_manual_actions(tmp_path):
|
||||
database = tmp_path / "test.sqlite"
|
||||
evidence = EvidenceStore(database)
|
||||
original = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="one",
|
||||
message_id="one",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="synthetic",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
store = MemoryGovernanceStore(database)
|
||||
|
||||
def denied():
|
||||
raise RuntimeError("denied")
|
||||
|
||||
with store.guard_writes(denied):
|
||||
assert store.list(user_id="local") == []
|
||||
result = as_person(store).create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content="manual action",
|
||||
evidence_ids=(original.evidence_id,),
|
||||
scopes=("private",),
|
||||
),
|
||||
request_id="manual",
|
||||
)
|
||||
assert result.content == "manual action"
|
||||
|
||||
|
||||
def test_guard_check_and_commit_hold_database_write_authority(tmp_path):
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
database = tmp_path / "test.sqlite"
|
||||
evidence = EvidenceStore(database)
|
||||
record = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="one",
|
||||
message_id="one",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="synthetic",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
store = MemoryGovernanceStore(database)
|
||||
with sqlite3.connect(database) as db:
|
||||
db.execute("CREATE TABLE permission_fixture (revoked INTEGER)")
|
||||
db.execute("INSERT INTO permission_fixture VALUES (0)")
|
||||
checks = []
|
||||
|
||||
def revoke():
|
||||
with sqlite3.connect(database, timeout=0) as db:
|
||||
db.execute("UPDATE permission_fixture SET revoked=1")
|
||||
|
||||
def guard():
|
||||
with (
|
||||
ThreadPoolExecutor(max_workers=1) as pool,
|
||||
pytest.raises(sqlite3.OperationalError, match="locked"),
|
||||
):
|
||||
pool.submit(revoke).result(timeout=2)
|
||||
checks.append("write-locked")
|
||||
|
||||
with store.guard_writes(guard):
|
||||
result = as_person(store).create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content="permitted",
|
||||
evidence_ids=(record.evidence_id,),
|
||||
scopes=("private",),
|
||||
),
|
||||
request_id="permission-checked",
|
||||
)
|
||||
assert checks == ["write-locked", "write-locked"]
|
||||
revoke()
|
||||
assert store.get(result.memory_id, user_id="local") is not None
|
||||
@@ -1,100 +0,0 @@
|
||||
"""The recall read path must not acquire the single global writer.
|
||||
|
||||
`MemoryGovernanceStore.find_confirmed_matching_terms` is the recall read path.
|
||||
It calls `_refresh_content_shadow`, which opened `_connection()` with the
|
||||
default `writing=True`. That means every recall read checked out the ONE global
|
||||
writer connection for a check that changes nothing on a migrated database —
|
||||
serializing unrelated readers behind the writer and inflating `writer_waits`.
|
||||
|
||||
The absent-shadow case must still fail closed (`governance_schema_missing`),
|
||||
and it must do so without taking the writer: the LIKE fallback already handles
|
||||
a missing shadow, and migration 32 — not this read path — owns the table.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import HyCreate, MemoryDraft, MemoryGovernanceStore
|
||||
|
||||
|
||||
def _store(tmp_path):
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
store = MemoryGovernanceStore(database)
|
||||
with store._connection() as connection:
|
||||
store._ensure_content_shadow(connection, backfill=False)
|
||||
evidence_id = (
|
||||
EvidenceStore(database)
|
||||
.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
conversation_id="conversation",
|
||||
message_id="m1",
|
||||
correlation_id="m1",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="发布窗口是周五上午。",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
)
|
||||
.record.evidence_id
|
||||
)
|
||||
return store, database, evidence_id
|
||||
|
||||
|
||||
def test_a_recall_read_does_not_check_out_the_global_writer(tmp_path):
|
||||
store, database, evidence_id = _store(tmp_path)
|
||||
store.write_from_hy(
|
||||
HyCreate(
|
||||
draft=MemoryDraft(
|
||||
user_id="local",
|
||||
kind="schema",
|
||||
content="发布窗口是周五上午",
|
||||
scopes=("all-apps",),
|
||||
evidence_ids=(evidence_id,),
|
||||
),
|
||||
request_id="r1",
|
||||
memory_id="m1",
|
||||
),
|
||||
user_id="local",
|
||||
derivation=store.hy_derivation(),
|
||||
)
|
||||
db = main_database(database)
|
||||
# Warm once so any lazy one-time initialization is not attributed to the read.
|
||||
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
|
||||
before = db.stats()
|
||||
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
|
||||
after = db.stats()
|
||||
assert after.reader_checkouts - before.reader_checkouts >= 1
|
||||
assert after.writer_checkouts - before.writer_checkouts == 0, (
|
||||
"the recall read path must not take the single global writer"
|
||||
)
|
||||
|
||||
|
||||
def test_a_missing_shadow_still_fails_closed_without_taking_the_writer(tmp_path):
|
||||
store, database, _evidence_id = _store(tmp_path)
|
||||
with sqlite3.connect(database) as connection:
|
||||
connection.execute("DROP TABLE governed_memories_terms")
|
||||
connection.execute("DROP TABLE governed_memories_terms_meta")
|
||||
db = main_database(database)
|
||||
before = db.stats()
|
||||
with pytest.raises(Exception, match="governance_schema_missing"):
|
||||
store.find_confirmed_matching_terms(user_id="local", terms=["发布"], limit=10)
|
||||
after = db.stats()
|
||||
assert after.writer_checkouts - before.writer_checkouts == 0, (
|
||||
"a broken schema must fail closed on a read connection, not write"
|
||||
)
|
||||
with sqlite3.connect(database) as connection:
|
||||
names = {
|
||||
row[0]
|
||||
for row in connection.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' "
|
||||
"AND name LIKE 'governed_memories_terms%'"
|
||||
)
|
||||
}
|
||||
assert names == set(), "a read must not repair or recreate the shadow tables"
|
||||
@@ -17,13 +17,18 @@ from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_memory_bridge import build_hy_memory_config
|
||||
from one_memory.hy_sdk_runtime import build_sdk_config
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ProviderKind,
|
||||
)
|
||||
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width
|
||||
from tests.graph_fixtures import kuzu_graph_with_embedding_width
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _enable_graph_store(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "kuzu")
|
||||
|
||||
|
||||
def _app(tmp_path: Path):
|
||||
@@ -80,10 +85,10 @@ def _activate_builtin_profile(app):
|
||||
|
||||
def _plant_wide_graph(app, data_dir: Path) -> Path:
|
||||
snapshot = _activate_builtin_profile(app)
|
||||
config = build_hy_memory_config(snapshot, api_key=None, data_dir=data_dir)
|
||||
config = build_sdk_config(snapshot, api_key=None, data_dir=data_dir)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
return db_path
|
||||
|
||||
|
||||
|
||||
@@ -15,18 +15,18 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
from one_memory.hy_graph_admin import (
|
||||
GraphWidthDecision,
|
||||
apply_graph_width_decision,
|
||||
build_hy_memory_config,
|
||||
inspect_graph_width_mismatch,
|
||||
)
|
||||
from one_memory.hy_sdk_runtime import build_sdk_config
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width, snapshot
|
||||
from tests.graph_fixtures import kuzu_graph_with_embedding_width, model_snapshot
|
||||
|
||||
|
||||
def _narrow_profile() -> ModelProfileSnapshot:
|
||||
base = snapshot(
|
||||
base = model_snapshot(
|
||||
ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
has_api_key=False,
|
||||
@@ -38,10 +38,10 @@ def _mismatched(tmp_path: Path) -> tuple[object, Path, Path]:
|
||||
"""A 768 graph under a data dir, against a 512 profile."""
|
||||
data_dir = tmp_path / "data"
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=data_dir)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=data_dir)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
return config, db_path, data_dir
|
||||
|
||||
|
||||
|
||||
@@ -13,22 +13,21 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
from one_memory.hy_graph_admin import (
|
||||
GraphWidthDecision,
|
||||
GraphWidthMismatchError,
|
||||
apply_graph_width_decision,
|
||||
build_hy_memory_config,
|
||||
create_hy_memory_client,
|
||||
inspect_graph_width_mismatch,
|
||||
)
|
||||
from one_memory.hy_sdk_runtime import acquire_client, build_sdk_config
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
from tests.test_hy_memory_bridge import _kuzu_graph_with_embedding_width, snapshot
|
||||
from tests.graph_fixtures import kuzu_graph_with_embedding_width, model_snapshot
|
||||
|
||||
|
||||
def _narrow_profile(**overrides: object) -> ModelProfileSnapshot:
|
||||
# Width alone is what matters here. The built-in provider arrives in S1.3;
|
||||
# this guard has to hold for any 512-dim profile against a 768 graph.
|
||||
base = snapshot(
|
||||
base = model_snapshot(
|
||||
ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
has_api_key=False,
|
||||
@@ -45,10 +44,10 @@ def test_a_768_graph_is_not_silently_archived_when_switching_to_512(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
|
||||
@@ -63,13 +62,13 @@ def test_creating_a_client_against_a_mismatched_graph_raises_instead_of_retiring
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
|
||||
with pytest.raises(GraphWidthMismatchError) as caught:
|
||||
create_hy_memory_client(profile, api_key=None, data_dir=tmp_path)
|
||||
acquire_client(profile, api_key=None, data_dir=tmp_path)
|
||||
|
||||
assert caught.value.mismatch.stored_width == 768
|
||||
assert caught.value.mismatch.expected_width == 512
|
||||
@@ -81,10 +80,10 @@ def test_keep_existing_leaves_the_graph_and_refuses_the_new_width(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
|
||||
outcome = apply_graph_width_decision(config, GraphWidthDecision.KEEP_EXISTING)
|
||||
|
||||
@@ -98,10 +97,10 @@ def test_rebuild_archives_the_graph_and_lets_the_new_width_proceed(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
|
||||
outcome = apply_graph_width_decision(config, GraphWidthDecision.REBUILD_WITH_NEW)
|
||||
|
||||
@@ -114,10 +113,10 @@ def test_rebuild_archives_the_graph_and_lets_the_new_width_proceed(
|
||||
|
||||
def test_export_first_leaves_the_graph_untouched(tmp_path: Path) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path / "data")
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path / "data")
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_kuzu_graph_with_embedding_width(db_path, 768)
|
||||
kuzu_graph_with_embedding_width(db_path, 768)
|
||||
|
||||
outcome = apply_graph_width_decision(
|
||||
config,
|
||||
@@ -135,6 +134,6 @@ def test_export_first_leaves_the_graph_untouched(tmp_path: Path) -> None:
|
||||
|
||||
def test_a_fresh_install_with_no_graph_raises_nothing(tmp_path: Path) -> None:
|
||||
profile = _narrow_profile()
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(profile, api_key=None, data_dir=tmp_path)
|
||||
|
||||
assert inspect_graph_width_mismatch(config) is None
|
||||
@@ -1,282 +0,0 @@
|
||||
"""R-07: HY's coding-shaped replies read as "no memories at all".
|
||||
|
||||
`search_ids` reads `memories` and `_listed_memories` reads `vdb.memories`.
|
||||
HY has a second reply shape — `{"scene": "productivity", "coding_memories":
|
||||
[...]}` (`hy_memory/client.py:1145`, `client.py:1038`) — and neither parser
|
||||
recognises it, so it comes back as an empty tuple. Empty is indistinguishable
|
||||
from "this person has no memories", which is the failure this codebase
|
||||
already paid for once on the profile path: see `_basic_info_memories`, which
|
||||
raises on a payload it does not recognise for exactly that reason.
|
||||
|
||||
Read from HY's source, the requirement's diagnosis does not hold: the coding
|
||||
path is not reachable from here at all, in either direction. Both preconditions
|
||||
are pinned below as green controls, because they are what makes this fix a
|
||||
guard rather than a repair — and they are the two things that would have to
|
||||
change before a coding memory could ever be recalled.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Mapping
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HyMemoryTurn,
|
||||
_listed_memories,
|
||||
build_hy_memory_config,
|
||||
)
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
CODING_SEARCH_REPLY: Mapping[str, object] = {
|
||||
"request_id": "req-1",
|
||||
"scene": "productivity",
|
||||
"coding_memories": [
|
||||
{"memory_id": "coding-1", "content": "用户在 recall.py 里用过滤下推替换了内存过滤"},
|
||||
{"memory_id": "coding-2", "content": "用户的分页上限是 500"},
|
||||
],
|
||||
"elapsed_ms": 12.0,
|
||||
}
|
||||
|
||||
CODING_LIST_REPLY: Mapping[str, object] = {
|
||||
"scene": "productivity",
|
||||
"coding_memories": [
|
||||
{"memory_id": "coding-1", "agent_id": "workspace-a"},
|
||||
{"memory_id": "coding-2", "agent_id": "workspace-b"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class CodingSceneClient:
|
||||
"""A client that answers in HY's coding shape rather than the chat shape."""
|
||||
|
||||
def __init__(self, *, search_reply=CODING_SEARCH_REPLY, list_reply=CODING_LIST_REPLY):
|
||||
self.search_reply = search_reply
|
||||
self.list_reply = list_reply
|
||||
|
||||
def search(self, query: str, **_kwargs) -> Mapping[str, object]:
|
||||
return self.search_reply
|
||||
|
||||
def list_memories(self, **_kwargs) -> Mapping[str, object]:
|
||||
return self.list_reply
|
||||
|
||||
def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _profile() -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="profile",
|
||||
revision=1,
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
model="memory-model",
|
||||
embedding_model="embedding-model",
|
||||
embedding_provider_kind=None,
|
||||
embedding_base_url=None,
|
||||
embedding_dimensions=768,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
has_api_key=False,
|
||||
)
|
||||
|
||||
|
||||
def _bridge(client, tmp_path) -> HyMemoryBridge:
|
||||
return HyMemoryBridge(
|
||||
profile=_profile(),
|
||||
api_key=None,
|
||||
data_dir=tmp_path,
|
||||
client=client,
|
||||
)
|
||||
|
||||
|
||||
def test_a_coding_shaped_search_reply_is_not_read_as_no_memories(tmp_path) -> None:
|
||||
bridge = _bridge(CodingSceneClient(), tmp_path)
|
||||
|
||||
found = asyncio.run(bridge.search_ids(query="过滤下推", user_id="local", agent_id="*"))
|
||||
|
||||
assert found == ("coding-1", "coding-2")
|
||||
|
||||
|
||||
def test_a_coding_shaped_listing_is_not_read_as_no_scopes(tmp_path) -> None:
|
||||
bridge = _bridge(CodingSceneClient(), tmp_path)
|
||||
|
||||
scopes = asyncio.run(bridge.list_agent_ids(user_id="local"))
|
||||
|
||||
assert scopes == ("workspace-a", "workspace-b")
|
||||
|
||||
|
||||
def test_an_unrecognised_search_reply_is_an_error_not_an_empty_recall(tmp_path) -> None:
|
||||
"""A shape we do not understand must not be reported as an empty library.
|
||||
|
||||
Returning `()` here is what made a coding reply look like "you have no
|
||||
memories". Whatever HY adds next, the honest answer is that we could not
|
||||
read it.
|
||||
"""
|
||||
bridge = _bridge(CodingSceneClient(search_reply={"request_id": "req-2"}), tmp_path)
|
||||
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
asyncio.run(bridge.search_ids(query="过滤下推", user_id="local", agent_id="*"))
|
||||
|
||||
|
||||
def test_an_unrecognised_listing_is_an_error_too(tmp_path) -> None:
|
||||
bridge = _bridge(CodingSceneClient(list_reply={"unexpected": {}}), tmp_path)
|
||||
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
asyncio.run(bridge.list_agent_ids(user_id="local"))
|
||||
|
||||
|
||||
#: What HY 0.1.6.5 actually answers a chat-path search with
|
||||
#: (`client.py:2378-2389`): `memories` is three channels, not a list. Our
|
||||
#: parser required a list, so every semantic search read as zero hits.
|
||||
THREE_CHANNEL_REPLY: Mapping[str, object] = {
|
||||
"request_id": "req-3",
|
||||
"memories": {
|
||||
"schema": [
|
||||
{"memory_id": "schema-1", "score": 0.61, "layer": "L0_BASIC_INFO"},
|
||||
],
|
||||
"proactive": [
|
||||
{"memory_id": "intention-1", "score": 0.44, "layer": "L7_INTENTION"},
|
||||
],
|
||||
"normal": [
|
||||
{"memory_id": "fact-1", "score": 0.88, "layer": "L2_FACT"},
|
||||
{"memory_id": "fact-2", "score": 0.52, "layer": "L2_FACT"},
|
||||
],
|
||||
},
|
||||
"rewrite": {"used": False, "scene": "normal"},
|
||||
"elapsed_ms": 41.0,
|
||||
}
|
||||
|
||||
#: The failure branch (`client.py:2253`), which is the shape this machine has
|
||||
#: actually been getting: same three channels, all empty. HY logs
|
||||
#: "[search] failed" and returns this.
|
||||
FAILED_SEARCH_REPLY: Mapping[str, object] = {
|
||||
"request_id": "req-4",
|
||||
"memories": {"schema": [], "proactive": [], "normal": []},
|
||||
"elapsed_ms": 3.0,
|
||||
}
|
||||
|
||||
|
||||
def test_the_three_channels_hy_actually_returns_are_all_read(tmp_path) -> None:
|
||||
"""The real defect behind R-07, and it is not about coding at all.
|
||||
|
||||
`memories` is a dict of schema / proactive / normal channels in the current
|
||||
HY SDK; our parser asked `isinstance(memories, list)` and returned an empty
|
||||
tuple otherwise. So semantic recall was dead — including the schema channel,
|
||||
which is the whole reason
|
||||
`schema_min_score` and `schema_limit` are passed in.
|
||||
"""
|
||||
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
|
||||
|
||||
found = asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*"))
|
||||
|
||||
assert set(found) == {"fact-1", "schema-1", "fact-2", "intention-1"}
|
||||
|
||||
|
||||
def test_the_channels_are_merged_in_hys_own_order_not_ours(tmp_path) -> None:
|
||||
"""Recall ranks by the order these ids come back in, so the order matters.
|
||||
|
||||
Bucketing into channels loses HY's ranking, but every item still carries
|
||||
the `score` HY ranked it by — a field we were also dropping. Sorting the
|
||||
union by it restores HY's judgement instead of substituting a channel
|
||||
preference of our own, which would be a retrieval-quality decision and
|
||||
not ours to make here.
|
||||
"""
|
||||
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
|
||||
|
||||
found = asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*"))
|
||||
|
||||
assert found == ("fact-1", "schema-1", "fact-2", "intention-1")
|
||||
|
||||
|
||||
def test_search_scores_carries_hys_relevance(tmp_path) -> None:
|
||||
"""D102: the score is what lets recall blend the two lanes by relevance."""
|
||||
bridge = _bridge(CodingSceneClient(search_reply=THREE_CHANNEL_REPLY), tmp_path)
|
||||
|
||||
found = asyncio.run(bridge.search_scores(query="花名", user_id="local", agent_id="*"))
|
||||
|
||||
assert found == (
|
||||
("fact-1", 0.88),
|
||||
("schema-1", 0.61),
|
||||
("fact-2", 0.52),
|
||||
("intention-1", 0.44),
|
||||
)
|
||||
|
||||
|
||||
def test_a_failed_hy_search_is_not_an_error_here_but_is_still_empty(tmp_path) -> None:
|
||||
"""HY reports search failure only in its log, and returns empty channels.
|
||||
|
||||
On this machine every real search has been `[search] failed: Connection
|
||||
error.` — the embedding endpoint is unreachable — and the reply is
|
||||
indistinguishable from a genuine zero-hit search. We cannot tell them
|
||||
apart at this seam, so this must not become a hard error; it is recorded
|
||||
as an upstream gap instead.
|
||||
"""
|
||||
bridge = _bridge(CodingSceneClient(search_reply=FAILED_SEARCH_REPLY), tmp_path)
|
||||
|
||||
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == ()
|
||||
|
||||
|
||||
def test_the_chat_shape_still_reads_the_same(tmp_path) -> None:
|
||||
"""The control: the shape we actually get today must not move.
|
||||
|
||||
Green before and after. Every real reply on this machine is this one, so
|
||||
a change here would be the fix breaking recall to satisfy a shape we have
|
||||
never seen.
|
||||
"""
|
||||
chat_client = CodingSceneClient(
|
||||
search_reply={"memories": [{"memory_id": "chat-1"}, {"id": "chat-2"}]},
|
||||
list_reply={"vdb": {"memories": [{"agent_id": "one-memory"}]}},
|
||||
)
|
||||
bridge = _bridge(chat_client, tmp_path)
|
||||
|
||||
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == (
|
||||
"chat-1",
|
||||
"chat-2",
|
||||
)
|
||||
assert asyncio.run(bridge.list_agent_ids(user_id="local")) == ("one-memory",)
|
||||
assert _listed_memories({"vdb": {"memories": []}}) == ()
|
||||
|
||||
|
||||
def test_an_empty_chat_reply_is_still_just_empty(tmp_path) -> None:
|
||||
"""No memories yet is not a shape error, and must stay quiet."""
|
||||
bridge = _bridge(
|
||||
CodingSceneClient(search_reply={"memories": []}, list_reply={"vdb": {"memories": []}}),
|
||||
tmp_path,
|
||||
)
|
||||
|
||||
assert asyncio.run(bridge.search_ids(query="花名", user_id="local", agent_id="*")) == ()
|
||||
assert asyncio.run(bridge.list_agent_ids(user_id="local")) == ()
|
||||
|
||||
|
||||
def test_a_tool_message_can_reach_hy_so_it_can_judge_the_route_itself() -> None:
|
||||
"""HY only considers its coding writer when the messages contain a
|
||||
tool-role message (`has_any_tool_message`). Kristin's call: send the turn
|
||||
as it happened, tool calls included, and let HY decide."""
|
||||
assert HyMemoryTurn(role="tool", content="[Bash] ls -la").role == "tool"
|
||||
with pytest.raises(ValidationError):
|
||||
HyMemoryTurn(role="system", content="hidden")
|
||||
|
||||
|
||||
def test_our_search_call_shape_cannot_reach_hys_coding_path(tmp_path) -> None:
|
||||
"""Why no coding reply can arrive today — the read side.
|
||||
|
||||
HY decides the scene per search (`client.py:2020-2116`): an explicit
|
||||
`scene`, or an LLM judge when query rewrite is on, or the multi-`queries`
|
||||
judge. Otherwise a single query defaults to `normal`. We pass one query,
|
||||
no scene, and leave rewrite at HY's default of off, so the coding branch
|
||||
is unreachable — `config.coding.enable` being true only pays for the
|
||||
store, not the routing.
|
||||
|
||||
Turning any of that on costs an LLM call per search and, with the store
|
||||
empty, buys nothing. Left as Kristin's call rather than switched on here.
|
||||
"""
|
||||
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
|
||||
assert config.coding.enable is True
|
||||
assert config.llm.enable_query_rewrite is False
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
import logging
|
||||
|
||||
from one_memory.hy_memory_bridge import configure_hy_memory_environment
|
||||
from one_memory.hy_runtime_config import configure_hy_runtime_environment
|
||||
|
||||
|
||||
def test_the_extractors_parse_failures_pass_through_unchanged(tmp_path):
|
||||
@@ -19,7 +19,7 @@ def test_the_extractors_parse_failures_pass_through_unchanged(tmp_path):
|
||||
original_handlers = list(logger.handlers)
|
||||
logger.handlers = [capture]
|
||||
logger.setLevel(logging.WARNING)
|
||||
configure_hy_memory_environment(tmp_path)
|
||||
configure_hy_runtime_environment(tmp_path)
|
||||
try:
|
||||
secret = "x" * 5000
|
||||
logger.warning(
|
||||
@@ -52,7 +52,7 @@ def test_other_warnings_pass_through_unchanged(tmp_path):
|
||||
original_handlers = list(logger.handlers)
|
||||
logger.handlers = [capture]
|
||||
logger.setLevel(logging.WARNING)
|
||||
configure_hy_memory_environment(tmp_path)
|
||||
configure_hy_runtime_environment(tmp_path)
|
||||
try:
|
||||
logger.warning("[coding-extract] judge failed, fallback to chat: %s", "reason")
|
||||
finally:
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,113 +0,0 @@
|
||||
"""What the pinned public SDK's MetricsCollector does, and what that costs us.
|
||||
|
||||
These integration tests ask whether the pending-task warning is a leak that
|
||||
grows per request, and whether reusing one client is enough to avoid it.
|
||||
Reading the warning alone cannot answer either question, so the tests measure it.
|
||||
|
||||
The answer that came out is narrower and worse than "a leak": the collector is
|
||||
a process singleton, so the two loops start once rather than once per client --
|
||||
but the flag that says "already started" is sticky and there is no stop. Once
|
||||
the loop those tasks were created on is gone, a later loop gets nothing, and
|
||||
flush and cleanup are silently dead for the rest of the process.
|
||||
|
||||
That makes reusing the client necessary but not sufficient: what has to survive
|
||||
is the event loop. These tests pin both halves so an SDK upgrade that fixes it
|
||||
shows up as a failure here instead of going unnoticed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("hy_memory")
|
||||
|
||||
|
||||
def background_loop_count() -> int:
|
||||
return sum(
|
||||
1
|
||||
for task in asyncio.all_tasks()
|
||||
if task.get_coro() is not None
|
||||
and (task.get_coro().__qualname__ or "").startswith("MetricsCollector._")
|
||||
)
|
||||
|
||||
|
||||
def test_the_collector_is_one_per_process_not_one_per_client() -> None:
|
||||
"""If it were per-client, reusing the client would be the whole fix."""
|
||||
from hy_memory.metrics import MetricsCollector
|
||||
|
||||
assert MetricsCollector.get() is MetricsCollector.get()
|
||||
|
||||
|
||||
def test_metrics_collector_can_stop_and_restart_on_a_second_event_loop(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""The current SDK owns an explicit lifecycle and restarts cleanly."""
|
||||
from hy_memory.metrics import MetricsCollector
|
||||
|
||||
# A fresh collector, so an earlier test in the session cannot decide this.
|
||||
monkeypatch.setattr(MetricsCollector, "_instance", None)
|
||||
collector = MetricsCollector.get()
|
||||
|
||||
async def cycle() -> tuple[int, int]:
|
||||
await collector.start_background_tasks()
|
||||
started = background_loop_count()
|
||||
await collector.stop_background_tasks()
|
||||
return started, background_loop_count()
|
||||
|
||||
first = asyncio.run(cycle())
|
||||
second = asyncio.run(cycle())
|
||||
|
||||
assert first == (2, 0)
|
||||
assert second == (2, 0)
|
||||
|
||||
|
||||
def test_the_runtime_reuses_one_client_per_profile_revision(tmp_path: Path) -> None:
|
||||
"""Our side of the workaround: never build a client per request.
|
||||
|
||||
A new client per request would re-enter HY initialization every time; the
|
||||
collector would survive as a singleton but everything else would not.
|
||||
"""
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
|
||||
built = []
|
||||
|
||||
def factory(profile, secret, data_dir):
|
||||
built.append((profile.profile_id, profile.revision))
|
||||
return object()
|
||||
|
||||
resolver = MemoryProcessingSupervisor._bridge
|
||||
queue = object.__new__(MemoryProcessingSupervisor)
|
||||
queue._bridges = {}
|
||||
queue.bridge_factory = factory
|
||||
|
||||
profile = _snapshot()
|
||||
first = resolver(queue, profile, None, tmp_path)
|
||||
again = resolver(queue, profile, None, tmp_path)
|
||||
newer = resolver(queue, profile.model_copy(update={"revision": 2}), None, tmp_path)
|
||||
|
||||
assert first is again, "a second request built a second client"
|
||||
assert newer is not first, "a new revision reused the old revision's client"
|
||||
assert built == [("profile", 1), ("profile", 2)]
|
||||
|
||||
|
||||
def _snapshot():
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="profile",
|
||||
revision=1,
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
model="memory-model",
|
||||
embedding_model="builtin",
|
||||
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
embedding_base_url=None,
|
||||
embedding_dimensions=384,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
has_api_key=False,
|
||||
)
|
||||
@@ -1,45 +1,42 @@
|
||||
"""Where a real HY client is allowed to send bytes, and where it is not.
|
||||
|
||||
The integration test goes beyond checking whether the config names a local
|
||||
host. It verifies that a live HY instance doing real work opens no socket to
|
||||
an external endpoint the user did not configure. SDK defaults can cover model,
|
||||
graph, vector, and metrics services; a config assertion cannot rule out a code
|
||||
path that reads those defaults directly.
|
||||
|
||||
So these tests watch the socket layer itself while a real client indexes and
|
||||
searches. The first test is the control: it proves the watcher notices a
|
||||
connection at all, because a watcher that sees nothing everywhere would pass
|
||||
this file without meaning anything.
|
||||
"""
|
||||
"""Public local-embedding injection must not create network egress."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import socket
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
build_hy_memory_config,
|
||||
)
|
||||
from one_memory.hy_sdk_runtime import LocalOnnxEmbeddingService, build_sdk_config
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
|
||||
def local_onnx_profile() -> ModelProfileSnapshot:
|
||||
"""A profile whose embedder runs in-process, so search needs no provider."""
|
||||
class _Watcher:
|
||||
def __init__(self, monkeypatch) -> None:
|
||||
self.targets: list[str] = []
|
||||
real_connect = socket.socket.connect
|
||||
|
||||
def connect(sock, address, *args, **kwargs):
|
||||
self.targets.append(str(address))
|
||||
return real_connect(sock, address, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(socket.socket, "connect", connect)
|
||||
|
||||
|
||||
class _LocalEmbedder:
|
||||
def encode(self, texts: list[str]) -> list[list[float]]:
|
||||
return [[1.0, 0.0] for _text in texts]
|
||||
|
||||
|
||||
def _profile() -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="egress-integration",
|
||||
profile_id="egress",
|
||||
revision=1,
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
model="memory-model",
|
||||
embedding_model="builtin",
|
||||
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
embedding_base_url=None,
|
||||
embedding_dimensions=384,
|
||||
embedding_model="builtin",
|
||||
embedding_dimensions=512,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
@@ -47,44 +44,8 @@ def local_onnx_profile() -> ModelProfileSnapshot:
|
||||
)
|
||||
|
||||
|
||||
class Watcher:
|
||||
"""Record every address the process tries to reach, and let it through.
|
||||
|
||||
Recording rather than denying keeps the failure legible: a denied socket
|
||||
turns into whatever error the caller invents for it, while a recorded one
|
||||
names the host in the assertion.
|
||||
"""
|
||||
|
||||
def __init__(self, monkeypatch) -> None:
|
||||
self.targets: list[str] = []
|
||||
real_connect = socket.socket.connect
|
||||
real_resolve = socket.getaddrinfo
|
||||
|
||||
def connect(sock, address, *args, **kwargs):
|
||||
self.targets.append(self._name(address))
|
||||
return real_connect(sock, address, *args, **kwargs)
|
||||
|
||||
def resolve(host, port, *args, **kwargs):
|
||||
self.targets.append(f"{host}:{port}")
|
||||
return real_resolve(host, port, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(socket.socket, "connect", connect)
|
||||
monkeypatch.setattr(socket, "getaddrinfo", resolve)
|
||||
|
||||
@staticmethod
|
||||
def _name(address: object) -> str:
|
||||
if isinstance(address, tuple) and len(address) >= 2:
|
||||
return f"{address[0]}:{address[1]}"
|
||||
return str(address)
|
||||
|
||||
@property
|
||||
def hosts(self) -> set[str]:
|
||||
return {target.rsplit(":", 1)[0] for target in self.targets}
|
||||
|
||||
|
||||
def test_the_watcher_reports_a_connection_that_really_happens(monkeypatch) -> None:
|
||||
"""Control. Without this, an empty recording proves nothing."""
|
||||
watcher = Watcher(monkeypatch)
|
||||
watcher = _Watcher(monkeypatch)
|
||||
listener = socket.socket()
|
||||
listener.bind(("127.0.0.1", 0))
|
||||
listener.listen(1)
|
||||
@@ -96,121 +57,33 @@ def test_the_watcher_reports_a_connection_that_really_happens(monkeypatch) -> No
|
||||
finally:
|
||||
listener.close()
|
||||
|
||||
assert "127.0.0.1" in watcher.hosts
|
||||
assert watcher.targets
|
||||
|
||||
|
||||
def test_a_real_search_with_the_built_in_embedder_reaches_nothing(
|
||||
def test_public_local_embedding_service_opens_no_socket(monkeypatch) -> None:
|
||||
watcher = _Watcher(monkeypatch)
|
||||
service = LocalOnnxEmbeddingService(_LocalEmbedder()) # type: ignore[arg-type]
|
||||
|
||||
assert asyncio.run(service.embed("长期偏好")) == [1.0, 0.0]
|
||||
assert watcher.targets == []
|
||||
|
||||
|
||||
def test_no_ambient_remote_backend_survives_sdk_config(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""Search is local when the embedder is local, or the claim is wrong.
|
||||
|
||||
A miss here means HY contacted an external endpoint the user never
|
||||
configured, which is the integration behavior that config assertions
|
||||
alone cannot verify.
|
||||
"""
|
||||
pytest.importorskip("hy_memory")
|
||||
from one_memory.embedding_assets import assets_present
|
||||
|
||||
if not assets_present():
|
||||
pytest.skip("built-in encoder weights are not installed")
|
||||
|
||||
embedded = _count_local_embeds(monkeypatch)
|
||||
watcher = Watcher(monkeypatch)
|
||||
asyncio.run(_search(local_onnx_profile(), tmp_path))
|
||||
|
||||
# An empty recording is only evidence if the search really ran. The
|
||||
# embedder loads lazily, so "reached nothing" and "did nothing" look the
|
||||
# same from the socket layer until we confirm a vector was produced.
|
||||
assert embedded, "the search never got as far as embedding the query"
|
||||
# Loopback to the configured Ollama port would still be a finding here:
|
||||
# search declares itself model-free, so it must not call the provider.
|
||||
assert watcher.targets == [], watcher.targets
|
||||
|
||||
|
||||
def test_the_watcher_catches_a_remote_embedder_on_that_same_path(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""Control: the SDK's remote embedding leaf is observable by the watcher.
|
||||
|
||||
An empty memory store may short-circuit search before embedding, so the
|
||||
control calls the same configured EmbedService network leaf directly.
|
||||
"""
|
||||
pytest.importorskip("hy_memory")
|
||||
from hy_memory.core.embed_service import EmbedService
|
||||
|
||||
remote = local_onnx_profile().model_copy(
|
||||
update={
|
||||
"embedding_provider_kind": ProviderKind.OPENAI_COMPATIBLE,
|
||||
"embedding_base_url": "http://127.0.0.1:59/v1",
|
||||
"embedding_model": "remote-embedder",
|
||||
"has_api_key": True,
|
||||
}
|
||||
)
|
||||
config = build_hy_memory_config(remote, api_key="test-key", data_dir=tmp_path)
|
||||
service = EmbedService(config)
|
||||
watcher = Watcher(monkeypatch)
|
||||
with contextlib.suppress(Exception):
|
||||
asyncio.run(service.embed("synthetic remote embedding", use_cache=False))
|
||||
|
||||
assert "127.0.0.1" in watcher.hosts, watcher.targets
|
||||
|
||||
|
||||
def _count_local_embeds(monkeypatch) -> list[int]:
|
||||
"""A truthy list once the in-process encoder has produced a vector."""
|
||||
from one_memory.hy_memory_bridge import LocalHyEmbedService
|
||||
|
||||
seen: list[int] = []
|
||||
original = LocalHyEmbedService.embed
|
||||
|
||||
async def counted(self, text, use_cache=True):
|
||||
vector = await original(self, text, use_cache)
|
||||
seen.append(len(vector))
|
||||
return vector
|
||||
|
||||
monkeypatch.setattr(LocalHyEmbedService, "embed", counted)
|
||||
return seen
|
||||
|
||||
|
||||
async def _search(
|
||||
profile: ModelProfileSnapshot,
|
||||
data_dir: Path,
|
||||
*,
|
||||
api_key: str | None = None,
|
||||
) -> None:
|
||||
bridge = HyMemoryBridge(profile=profile, api_key=api_key, data_dir=data_dir)
|
||||
try:
|
||||
await bridge.search_ids("长期偏好", user_id="local", agent_id="*")
|
||||
finally:
|
||||
await bridge.close()
|
||||
|
||||
|
||||
def test_no_internal_gateway_or_server_default_survives_our_config(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""The SDK's shipped defaults must not be reachable through our config.
|
||||
|
||||
Ambient SDK configuration may name remote stores and endpoints. We pin the
|
||||
OpenAI-compatible provider and local stores, so none may survive.
|
||||
"""
|
||||
for name, value in (
|
||||
("MEMORY_LLM_PROVIDER", "openai"),
|
||||
("MEMORY_LLM_BASE_URL", "https://ambient.example/v1"),
|
||||
("MEMORY_GRAPH_PROVIDER", "memgraph"),
|
||||
("MEMGRAPH_URL", "bolt://memgraph.example.com:7687"),
|
||||
("MEMORY_VECTOR_PROVIDER", "qdrant"),
|
||||
("MEMORY_API_HOST", "0.0.0.0"),
|
||||
):
|
||||
monkeypatch.setenv(name, value)
|
||||
|
||||
config = build_hy_memory_config(local_onnx_profile(), api_key=None, data_dir=tmp_path)
|
||||
config = build_sdk_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
rendered = repr(config).lower()
|
||||
|
||||
rendered = repr(config)
|
||||
for forbidden in ("ambient.example", "memgraph", "qdrant", "bolt://"):
|
||||
assert forbidden not in rendered.lower(), forbidden
|
||||
assert config.llm.provider == "openai"
|
||||
assert forbidden not in rendered
|
||||
assert config.vector_store.provider == "chroma"
|
||||
assert config.graph_store.provider == "kuzu"
|
||||
assert str(tmp_path) in config.graph_store.db_path
|
||||
@@ -1,518 +0,0 @@
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.diagnostic_log import configure_diagnostics, reset_diagnostics
|
||||
from one_memory.hy_response_guard import (
|
||||
HyReconcileGuard,
|
||||
HyResponseGuard,
|
||||
install_hy_write_guards,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("invalid", ["empty", "length"])
|
||||
def test_invalid_response_retries_same_hy_provider_without_changing_prompt(invalid):
|
||||
class Provider:
|
||||
calls = []
|
||||
|
||||
async def complete(self, **kwargs):
|
||||
self.calls.append(kwargs)
|
||||
return SimpleNamespace(
|
||||
content="" if invalid == "empty" and len(self.calls) == 1 else "[]",
|
||||
finish_reason="length" if invalid == "length" and len(self.calls) == 1 else "stop",
|
||||
tool_calls=None,
|
||||
)
|
||||
|
||||
provider = Provider()
|
||||
response = asyncio.run(
|
||||
HyResponseGuard(provider).complete(prompt="native prompt", max_tokens=2000)
|
||||
)
|
||||
assert response.content == "[]"
|
||||
assert len(provider.calls) == 2
|
||||
assert all(call["prompt"] == "native prompt" for call in provider.calls)
|
||||
|
||||
|
||||
def test_repeated_empty_response_fails_instead_of_becoming_no_op():
|
||||
class Provider:
|
||||
calls = 0
|
||||
|
||||
async def complete_messages(self, **kwargs):
|
||||
self.calls += 1
|
||||
return SimpleNamespace(content="", finish_reason="stop", tool_calls=None)
|
||||
|
||||
provider = Provider()
|
||||
with pytest.raises(ValueError, match="hy_memory_model_empty_response"):
|
||||
asyncio.run(HyResponseGuard(provider).complete_messages(messages=[]))
|
||||
assert provider.calls == 2
|
||||
|
||||
|
||||
def test_failure_receipt_survives_a_later_successful_model_call():
|
||||
class Provider:
|
||||
succeed = False
|
||||
|
||||
async def complete_messages(self, **kwargs):
|
||||
return SimpleNamespace(
|
||||
content="{}" if self.succeed else "", finish_reason="stop", tool_calls=None
|
||||
)
|
||||
|
||||
provider = Provider()
|
||||
guard = HyResponseGuard(provider)
|
||||
with pytest.raises(ValueError, match="hy_memory_model_empty_response"):
|
||||
asyncio.run(guard.complete_messages(messages=[]))
|
||||
provider.succeed = True
|
||||
asyncio.run(guard.complete_messages(messages=[]))
|
||||
assert guard.failures == ["hy_memory_model_empty_response"]
|
||||
|
||||
|
||||
def test_provider_exception_is_recorded_even_if_hy_later_swallows_it():
|
||||
class Provider:
|
||||
async def complete_messages(self, **kwargs):
|
||||
raise RuntimeError("provider unavailable")
|
||||
|
||||
guard = HyResponseGuard(Provider())
|
||||
with pytest.raises(RuntimeError):
|
||||
asyncio.run(guard.complete_messages(messages=[]))
|
||||
assert guard.failures == ["hy_memory_model_request_failed"]
|
||||
|
||||
|
||||
def test_payment_error_is_recorded_without_retrying_or_leaking_provider_details():
|
||||
class PaymentError(RuntimeError):
|
||||
status_code = 402
|
||||
|
||||
class Provider:
|
||||
calls = 0
|
||||
|
||||
async def complete_messages(self, **kwargs):
|
||||
self.calls += 1
|
||||
raise PaymentError("Private provider account details")
|
||||
|
||||
provider = Provider()
|
||||
guard = HyResponseGuard(provider)
|
||||
with pytest.raises(PaymentError):
|
||||
asyncio.run(guard.complete_messages(messages=[]))
|
||||
assert provider.calls == 1
|
||||
assert guard.failures == ["hy_memory_model_payment_required"]
|
||||
|
||||
|
||||
def observe_provider_failure(error, *, method="complete_messages"):
|
||||
class Provider:
|
||||
calls = 0
|
||||
|
||||
async def complete(self, *args, **kwargs):
|
||||
self.calls += 1
|
||||
raise error
|
||||
|
||||
complete_messages = complete
|
||||
|
||||
provider = Provider()
|
||||
guard = HyResponseGuard(provider)
|
||||
with pytest.raises(type(error)) as raised:
|
||||
asyncio.run(getattr(guard, method)("private prompt", api_key="private key"))
|
||||
assert raised.value is error
|
||||
assert provider.calls == 1
|
||||
return guard
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method", ["complete", "complete_messages"])
|
||||
@pytest.mark.parametrize("location", ["direct", "response", "attempts"])
|
||||
@pytest.mark.parametrize(
|
||||
"status,code",
|
||||
[
|
||||
(401, "hy_memory_model_authentication_failed"),
|
||||
(402, "hy_memory_model_payment_required"),
|
||||
(403, "hy_memory_model_access_denied"),
|
||||
(429, "hy_memory_model_rate_limited"),
|
||||
(500, "hy_memory_model_service_unavailable"),
|
||||
(599, "hy_memory_model_service_unavailable"),
|
||||
(408, "hy_memory_model_timeout"),
|
||||
(400, "hy_memory_model_request_incompatible"),
|
||||
(404, "hy_memory_model_request_incompatible"),
|
||||
(422, "hy_memory_model_request_incompatible"),
|
||||
],
|
||||
)
|
||||
def test_provider_status_is_classified_without_retry_or_private_logging(
|
||||
status, code, location, method, caplog
|
||||
):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
private = "private account key prompt response"
|
||||
if location == "attempts":
|
||||
error = LLMRetryError(
|
||||
private,
|
||||
[
|
||||
{"status_code": 401, "error": private},
|
||||
{"status_code": status, "error_type": private, "response_body": private},
|
||||
],
|
||||
)
|
||||
else:
|
||||
error = RuntimeError(private)
|
||||
if location == "direct":
|
||||
error.status_code = status
|
||||
else:
|
||||
error.response = SimpleNamespace(status_code=status, text=private)
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
guard = observe_provider_failure(error, method=method)
|
||||
assert guard.failures == [code]
|
||||
assert caplog.messages == [f"{code}: status_code={status}"]
|
||||
assert "private" not in caplog.text
|
||||
assert all(record.exc_info is None and record.stack_info is None for record in caplog.records)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("location", ["direct", "response", "attempts"])
|
||||
@pytest.mark.parametrize("status", [None, True, False, "402", 402.0, -1, 600, 200, 302, 423])
|
||||
def test_missing_invalid_or_unmapped_status_is_generic(location, status, caplog):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
if location == "attempts":
|
||||
# A valid earlier failure must not override the final unknown result.
|
||||
error = LLMRetryError("private status=402", [{"status_code": 402}, {"status_code": status}])
|
||||
else:
|
||||
error = RuntimeError("private status=402")
|
||||
if location == "direct":
|
||||
error.status_code = status
|
||||
else:
|
||||
error.response = SimpleNamespace(status_code=status)
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
guard = observe_provider_failure(error)
|
||||
assert guard.failures == ["hy_memory_model_request_failed"]
|
||||
expected_status = status if type(status) is int and 100 <= status <= 599 else "unknown"
|
||||
assert caplog.messages == [f"hy_memory_model_request_failed: status_code={expected_status}"]
|
||||
assert "private" not in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("attempts", [None, [], {}, "402", [None], ["402"], [{"error": "402"}]])
|
||||
def test_malformed_attempts_do_not_mask_original_exception(attempts, caplog):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
error = LLMRetryError("private message", attempts)
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
|
||||
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"direct,response,expected",
|
||||
[
|
||||
(401, 429, "authentication_failed"),
|
||||
(None, 429, "rate_limited"),
|
||||
("401", "429", "payment_required"),
|
||||
],
|
||||
)
|
||||
def test_status_lookup_precedence_is_direct_then_response_then_final_attempt(
|
||||
direct, response, expected
|
||||
):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
error = LLMRetryError("private message", [{"status_code": 402}])
|
||||
error.status_code = direct
|
||||
error.response = SimpleNamespace(status_code=response)
|
||||
assert observe_provider_failure(error).failures == [f"hy_memory_model_{expected}"]
|
||||
|
||||
|
||||
def test_failure_diagnostics_never_stringify_exception_or_illegal_status(caplog):
|
||||
class PrivateError(RuntimeError):
|
||||
def __str__(self):
|
||||
pytest.fail("Diagnostics must not stringify the provider exception")
|
||||
|
||||
class PrivateStatus:
|
||||
def __str__(self):
|
||||
pytest.fail("Diagnostics must not stringify an untrusted status")
|
||||
|
||||
error = PrivateError()
|
||||
error.status_code = PrivateStatus()
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
|
||||
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
|
||||
|
||||
|
||||
def test_status_attribute_failure_preserves_original_provider_exception(caplog):
|
||||
class BrokenStatusError(RuntimeError):
|
||||
@property
|
||||
def status_code(self):
|
||||
raise ValueError("private attribute failure")
|
||||
|
||||
error = BrokenStatusError("private provider message")
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
|
||||
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
|
||||
|
||||
|
||||
def test_integer_subclass_is_not_a_strict_http_status(caplog):
|
||||
class Status(int):
|
||||
pass
|
||||
|
||||
error = RuntimeError("private message")
|
||||
error.status_code = Status(402)
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
|
||||
assert caplog.messages == ["hy_memory_model_request_failed: status_code=unknown"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method", ["complete", "complete_messages"])
|
||||
def test_terminal_sanitized_client_timeout_is_not_an_invented_http_status(method, caplog):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
error = LLMRetryError("private error", [{"failure_kind": "timeout", "error": "private"}])
|
||||
caplog.set_level(logging.WARNING, logger="one_memory.hy_response_guard")
|
||||
assert observe_provider_failure(error, method=method).failures == ["hy_memory_model_timeout"]
|
||||
assert caplog.messages == ["hy_memory_model_timeout: status_code=unknown"]
|
||||
assert "private" not in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", [None, "connection", "unknown", "timeout private", True, {}, []])
|
||||
def test_only_final_exact_timeout_category_is_used(kind, caplog):
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
error = LLMRetryError("private error", [{"failure_kind": "timeout"}, {"failure_kind": kind}])
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_request_failed"]
|
||||
assert "private" not in caplog.text
|
||||
|
||||
|
||||
def test_real_http_status_takes_precedence_over_client_timeout_kind():
|
||||
from hy_memory.agent.llm_provider import LLMRetryError
|
||||
|
||||
error = LLMRetryError("private error", [{"failure_kind": "timeout", "status_code": 402}])
|
||||
assert observe_provider_failure(error).failures == ["hy_memory_model_payment_required"]
|
||||
|
||||
|
||||
def test_valid_empty_json_array_is_not_confused_with_a_transport_empty_response():
|
||||
class Provider:
|
||||
calls = 0
|
||||
|
||||
async def complete(self, **kwargs):
|
||||
self.calls += 1
|
||||
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
|
||||
|
||||
provider = Provider()
|
||||
assert asyncio.run(HyResponseGuard(provider).complete(prompt="native")).content == "[]"
|
||||
assert provider.calls == 1
|
||||
|
||||
|
||||
def test_installation_reaches_ordinary_and_ultra_inner_writers():
|
||||
from one_memory.hy_write_verification import HyExtractionGuard
|
||||
|
||||
def native_writer():
|
||||
native = SimpleNamespace()
|
||||
agent = SimpleNamespace(llm_provider=native, extractor=SimpleNamespace(llm=native))
|
||||
reconciler = SimpleNamespace(_get_llm=lambda: native)
|
||||
return SimpleNamespace(mem_agent=agent, reconciler=reconciler), agent, reconciler
|
||||
|
||||
ordinary, ordinary_agent, ordinary_reconciler = native_writer()
|
||||
ultra_inner, ultra_agent, ultra_reconciler = native_writer()
|
||||
|
||||
class Registry:
|
||||
async def get_writer(self):
|
||||
return ordinary
|
||||
|
||||
async def get_system2_writer(self):
|
||||
return SimpleNamespace(_writer=ultra_inner)
|
||||
|
||||
client = SimpleNamespace(_registry=Registry(), _loop_thread=SimpleNamespace(run=asyncio.run))
|
||||
install_hy_write_guards(client)
|
||||
for writer, agent, reconciler in (
|
||||
(ordinary, ordinary_agent, ordinary_reconciler),
|
||||
(ultra_inner, ultra_agent, ultra_reconciler),
|
||||
):
|
||||
assert isinstance(agent.extractor.llm, HyResponseGuard)
|
||||
assert agent.llm_provider is agent.extractor.llm
|
||||
assert isinstance(agent.extractor, HyExtractionGuard)
|
||||
assert agent.extractor in client._one_memory_write_guards
|
||||
assert isinstance(reconciler._llm, HyResponseGuard)
|
||||
assert type(writer._reconciler).__name__ == "HyReconcileGuard"
|
||||
|
||||
|
||||
def test_installed_guards_label_model_calls_by_hy_stage(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
|
||||
|
||||
class Provider:
|
||||
async def complete(self, **kwargs):
|
||||
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
|
||||
|
||||
def native_writer():
|
||||
native = Provider()
|
||||
agent = SimpleNamespace(llm_provider=native, extractor=SimpleNamespace(llm=native))
|
||||
reconciler = SimpleNamespace(_get_llm=lambda: native)
|
||||
return SimpleNamespace(mem_agent=agent, reconciler=reconciler), agent, reconciler
|
||||
|
||||
ordinary, ordinary_agent, ordinary_reconciler = native_writer()
|
||||
ultra_inner, _ultra_agent, _ultra_reconciler = native_writer()
|
||||
|
||||
class Registry:
|
||||
async def get_writer(self):
|
||||
return ordinary
|
||||
|
||||
async def get_system2_writer(self):
|
||||
return SimpleNamespace(_writer=ultra_inner)
|
||||
|
||||
client = SimpleNamespace(_registry=Registry(), _loop_thread=SimpleNamespace(run=asyncio.run))
|
||||
install_hy_write_guards(client)
|
||||
receipt = configure_diagnostics(tmp_path)
|
||||
try:
|
||||
asyncio.run(ordinary_agent.llm_provider.complete(prompt="private-extract"))
|
||||
asyncio.run(ordinary_reconciler._llm.complete(prompt="private-reconcile"))
|
||||
finally:
|
||||
reset_diagnostics()
|
||||
written = receipt.path.read_text()
|
||||
phases = [json.loads(line)["details"]["phase"] for line in written.splitlines()]
|
||||
assert phases == ["extract", "reconcile"]
|
||||
assert "private-" not in written
|
||||
|
||||
|
||||
def test_model_call_log_records_stage_and_duration_without_prompt(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
|
||||
|
||||
class Provider:
|
||||
async def complete(self, *, prompt, **kwargs):
|
||||
return SimpleNamespace(content="[]", finish_reason="stop", tool_calls=None)
|
||||
|
||||
receipt = configure_diagnostics(tmp_path)
|
||||
try:
|
||||
asyncio.run(HyResponseGuard(Provider()).complete(prompt="private-prompt-marker"))
|
||||
finally:
|
||||
reset_diagnostics()
|
||||
written = receipt.path.read_text()
|
||||
event = json.loads(written)
|
||||
assert event["event"] == "hy.model_call.completed"
|
||||
assert event["details"] == {"phase": "extract", "attempt": 1, "call_count": 1}
|
||||
assert event["duration_ms"] >= 0
|
||||
assert "private-prompt-marker" not in written
|
||||
|
||||
|
||||
def test_reconcile_log_counts_operations_without_content(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
|
||||
|
||||
class Reconciler:
|
||||
async def reconcile(self, **kwargs):
|
||||
return SimpleNamespace(
|
||||
success=True,
|
||||
ops=[
|
||||
SimpleNamespace(op="ADD", owner="user", content="private-a", memory_id=None),
|
||||
SimpleNamespace(op="UPDATE", owner="user", content="private-b", memory_id=None),
|
||||
SimpleNamespace(
|
||||
op="SUPERSEDE", owner="user", content="private-c", memory_id=None
|
||||
),
|
||||
SimpleNamespace(op="DELETE", owner="user", content="private-d", memory_id=None),
|
||||
],
|
||||
)
|
||||
|
||||
receipt = configure_diagnostics(tmp_path)
|
||||
try:
|
||||
asyncio.run(HyReconcileGuard(Reconciler()).reconcile())
|
||||
finally:
|
||||
reset_diagnostics()
|
||||
written = receipt.path.read_text()
|
||||
event = json.loads(written)
|
||||
assert event["event"] == "hy.reconcile.completed"
|
||||
assert event["details"] == {
|
||||
"created_count": 1,
|
||||
"updated_count": 1,
|
||||
"superseded_count": 1,
|
||||
"deleted_count": 1,
|
||||
"result_count": 4,
|
||||
}
|
||||
assert "private-a" not in written
|
||||
|
||||
|
||||
def test_native_no_candidate_reconcile_retains_extractor_owner(tmp_path, monkeypatch):
|
||||
from hy_memory.agent.reconciler import MemoryReconciler
|
||||
from hy_memory.config import MemoryConfig
|
||||
from hy_memory.pipelines._retrieval.reconcile_retrieval import ReconcileHybridRetriever
|
||||
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
class Embeddings:
|
||||
async def embed_batch(self, texts):
|
||||
return [[1.0, 0.0] for _ in texts]
|
||||
|
||||
class Store:
|
||||
async def get_embeddings(self, identifiers):
|
||||
return {}
|
||||
|
||||
async def no_candidates(*args, **kwargs):
|
||||
return []
|
||||
|
||||
monkeypatch.setattr(ReconcileHybridRetriever, "search_candidates", no_candidates)
|
||||
native = MemoryReconciler(MemoryConfig())
|
||||
native._enable_search_query = False
|
||||
arguments = {
|
||||
"new_memories": ["Agent建议启用备份,用户尚未确认。"],
|
||||
"user_id": "test",
|
||||
"agent_id": "test",
|
||||
"embed_service": Embeddings(),
|
||||
"vector_store": Store(),
|
||||
"new_memories_with_meta": [
|
||||
{"content": "Agent建议启用备份,用户尚未确认。", "owner": "agent"}
|
||||
],
|
||||
}
|
||||
result = asyncio.run(HyReconcileGuard(native).reconcile(**arguments))
|
||||
assert result.success and len(result.ops) == 1
|
||||
assert result.ops[0].owner == "agent"
|
||||
assert result.ops[0].content == arguments["new_memories"][0]
|
||||
|
||||
|
||||
def test_report_binding_rejects_a_reconciler_input_not_in_the_supplied_file():
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
guard = HyReconcileGuard(None)
|
||||
guard.imported_contents = frozenset({"supplied"})
|
||||
with pytest.raises(ValueError, match="import_input_mismatch"):
|
||||
asyncio.run(
|
||||
guard.reconcile(
|
||||
new_memories=["unbound"], new_memories_with_meta=[{"content": "unbound"}]
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"target_state", ["valid", "missing", "other-user", "other-agent", "read-error"]
|
||||
)
|
||||
def test_update_verification_reads_the_real_hy_store_interface_and_checks_identity(target_state):
|
||||
from hy_memory.agent.reconciler import ReconcileOp, ReconcileResult
|
||||
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
node = SimpleNamespace(user_id="local", agent_id="atlas", owner="user", content="Keep history.")
|
||||
if target_state == "other-user":
|
||||
node.user_id = "another-user"
|
||||
if target_state == "other-agent":
|
||||
node.agent_id = "another-agent"
|
||||
|
||||
class Store:
|
||||
calls = []
|
||||
|
||||
async def get_by_id(self, key):
|
||||
self.calls.append(key)
|
||||
if target_state == "read-error":
|
||||
raise OSError("private storage details must not reach the receipt")
|
||||
return None if target_state == "missing" else node
|
||||
|
||||
class Native:
|
||||
async def reconcile(self, **kwargs):
|
||||
return ReconcileResult(
|
||||
ops=[
|
||||
ReconcileOp(op="UPDATE", memory_id="prior", owner="user", content=node.content)
|
||||
]
|
||||
)
|
||||
|
||||
def _get_llm(self):
|
||||
pytest.fail("An unchanged, identity-checked source needs no new model call")
|
||||
|
||||
store = Store()
|
||||
guard = HyReconcileGuard(Native())
|
||||
arguments = {"user_id": "local", "agent_id": "atlas", "vector_store": store}
|
||||
if target_state == "valid":
|
||||
assert asyncio.run(guard.reconcile(**arguments)).success
|
||||
assert guard.failures == []
|
||||
else:
|
||||
code = (
|
||||
"hy_memory_write_verification_failed"
|
||||
if target_state == "read-error"
|
||||
else "hy_memory_write_verification_target_invalid"
|
||||
)
|
||||
with pytest.raises(ValueError, match=code):
|
||||
asyncio.run(guard.reconcile(**arguments))
|
||||
assert guard.failures == [code]
|
||||
assert store.calls == ["prior"]
|
||||
@@ -1,164 +0,0 @@
|
||||
"""HY response shape selection only; no provider, write routing or model calls."""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Mapping
|
||||
from copy import deepcopy
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError, _replied_memories
|
||||
from tests.test_hy_coding_reply import CodingSceneClient, _bridge
|
||||
|
||||
ERROR = "hy_memory_search_shape_unknown"
|
||||
CODING = {"memory_id": "synthetic-coding", "task": "synthetic task", "solution": "proposal"}
|
||||
CHAT = {"memory_id": "synthetic-chat", "content": "synthetic chat", "score": 0.8}
|
||||
|
||||
|
||||
def groups():
|
||||
return {"schema": [], "proactive": [], "normal": []}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scene", ["productivity", "working"])
|
||||
@pytest.mark.parametrize("chat_has_rows", [False, True])
|
||||
def test_explicit_coding_scene_selects_coding_even_with_chat_groups(scene, chat_has_rows):
|
||||
chat = groups()
|
||||
if chat_has_rows:
|
||||
chat["normal"] = [CHAT]
|
||||
reply = {"scene": scene, "coding_memories": [CODING], "memories": chat}
|
||||
before = deepcopy(reply)
|
||||
result = _replied_memories(reply, ERROR)
|
||||
assert result == (CODING,)
|
||||
assert result[0] is CODING
|
||||
assert reply == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scene", ["normal", "chat"])
|
||||
@pytest.mark.parametrize("grouped", [False, True])
|
||||
def test_explicit_chat_scene_never_merges_other_channel(scene, grouped):
|
||||
chat = {**groups(), "normal": [CHAT]} if grouped else [CHAT]
|
||||
reply = {"scene": scene, "memories": chat, "coding_memories": [CODING]}
|
||||
assert _replied_memories(reply, ERROR) == (CHAT,)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reply,expected",
|
||||
[
|
||||
({"memories": [CHAT]}, (CHAT,)),
|
||||
({"memories": []}, ()),
|
||||
({"memories": groups()}, ()),
|
||||
({"coding_memories": [CODING]}, (CODING,)),
|
||||
({"coding_memories": []}, ()),
|
||||
({"scene": "productivity", "coding_memories": []}, ()),
|
||||
({"scene": "normal", "memories": []}, ()),
|
||||
({"scene": "productivity", "coding_memories": [], "memories": [CHAT]}, ()),
|
||||
({"scene": "normal", "memories": [], "coding_memories": [CODING]}, ()),
|
||||
],
|
||||
)
|
||||
def test_legacy_single_channel_and_selected_empty_shapes_remain_valid(reply, expected):
|
||||
assert _replied_memories(reply, ERROR) == expected
|
||||
|
||||
|
||||
def test_all_chat_groups_preserve_rows_and_order_without_reinterpreting_metadata():
|
||||
schema = {"id": "schema", "score": 0.3}
|
||||
proactive = {"id": "proactive", "score": 0.2}
|
||||
reply = {
|
||||
"memories": {"schema": [schema], "proactive": [proactive], "normal": [CHAT]},
|
||||
"rewrite": {"used": False, "scene": "normal"},
|
||||
"elapsed_ms": 1,
|
||||
}
|
||||
assert _replied_memories(reply, ERROR) == (schema, proactive, CHAT)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"scene", [None, False, 1, [], {}, "", "other", "coding", "NORMAL", " chat "]
|
||||
)
|
||||
def test_unknown_or_malformed_explicit_scene_never_falls_back(scene):
|
||||
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
|
||||
_replied_memories({"scene": scene, "memories": [CHAT]}, ERROR)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reply",
|
||||
[
|
||||
{"scene": "productivity", "memories": [CHAT]},
|
||||
{"scene": "working", "memories": groups()},
|
||||
{"scene": "normal", "coding_memories": [CODING]},
|
||||
{"scene": "chat", "coding_memories": []},
|
||||
{"memories": [], "coding_memories": [CODING]},
|
||||
{"memories": [CHAT], "coding_memories": []},
|
||||
{"memories": groups(), "coding_memories": []},
|
||||
{"memories": [CHAT], "coding_memories": [CODING]},
|
||||
],
|
||||
)
|
||||
def test_missing_selected_channel_or_ambiguous_unlabelled_channels_are_rejected(reply):
|
||||
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
|
||||
_replied_memories(reply, ERROR)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reply",
|
||||
[
|
||||
None,
|
||||
[],
|
||||
{},
|
||||
{"memories": None},
|
||||
{"memories": {}},
|
||||
{"memories": {"normal": [CHAT]}},
|
||||
{"memories": {**groups(), "unexpected": [CHAT]}},
|
||||
{"memories": {**groups(), "schema": None}},
|
||||
{"memories": {**groups(), "normal": [CHAT, 1]}},
|
||||
{"memories": [CHAT, None]},
|
||||
{"memories": [1]},
|
||||
{"memories": (CHAT,)},
|
||||
{"coding_memories": {"normal": [CODING]}},
|
||||
{"coding_memories": [CODING, False]},
|
||||
{"scene": "productivity", "coding_memories": [1], "memories": groups()},
|
||||
{"scene": "productivity", "coding_memories": [CODING], "memories": {"bad": []}},
|
||||
{"scene": "normal", "memories": [CHAT], "coding_memories": None},
|
||||
],
|
||||
)
|
||||
def test_malformed_channel_or_row_fails_whole_reply_instead_of_losing_rows(reply):
|
||||
with pytest.raises(HyMemoryBridgeError, match=f"^{ERROR}$"):
|
||||
_replied_memories(reply, ERROR)
|
||||
|
||||
|
||||
def test_errors_use_supplied_safe_code_without_echoing_payload():
|
||||
secret_marker = "synthetic-private-payload-must-not-appear"
|
||||
error_code = "hy_memory_list_shape_unknown"
|
||||
with pytest.raises(HyMemoryBridgeError) as raised:
|
||||
_replied_memories({"memories": [CHAT, secret_marker]}, error_code)
|
||||
assert str(raised.value) == error_code
|
||||
assert secret_marker not in repr(raised.value)
|
||||
|
||||
|
||||
def test_documented_dual_reply_reaches_existing_search_and_listing_consumers(tmp_path):
|
||||
row = {**CODING, "agent_id": "synthetic-workspace"}
|
||||
reply: Mapping[str, object] = {
|
||||
"scene": "productivity",
|
||||
"coding_memories": [row],
|
||||
"memories": groups(),
|
||||
}
|
||||
|
||||
class Client(CodingSceneClient):
|
||||
def __init__(self):
|
||||
super().__init__(search_reply=reply, list_reply=reply)
|
||||
self.search_args = []
|
||||
|
||||
def search(self, query, **kwargs):
|
||||
self.search_args.append(kwargs)
|
||||
return self.search_reply
|
||||
|
||||
async def scenario():
|
||||
client = Client()
|
||||
bridge = _bridge(client, tmp_path)
|
||||
try:
|
||||
assert await bridge.search_ids("synthetic query", user_id="local", agent_id="*") == (
|
||||
"synthetic-coding",
|
||||
)
|
||||
assert await bridge.list_agent_ids(user_id="local") == ("synthetic-workspace",)
|
||||
assert client.search_args[0]["scene"] == "normal"
|
||||
assert client.search_args[0]["reader"] == "hybrid"
|
||||
finally:
|
||||
await bridge.close()
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,60 +1,17 @@
|
||||
"""Keep the documented runtime policy synchronized with the pinned public SDK.
|
||||
|
||||
Every release must keep the runtime aligned with the documented policy.
|
||||
Checking that contract automatically prevents a silently flipped rerank flag
|
||||
from going unnoticed.
|
||||
"""
|
||||
"""Pinned public SDK identity and diagnostic logging policy."""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_BEHAVIOR_ENV,
|
||||
HY_MEMORY_DISTRIBUTION,
|
||||
HY_MEMORY_MODE,
|
||||
HY_MEMORY_READER,
|
||||
HY_MEMORY_VERSION,
|
||||
apply_diagnostic_log_level,
|
||||
hy_memory_version,
|
||||
)
|
||||
|
||||
DOCUMENT = Path(__file__).resolve().parents[3] / "MEMORY.md"
|
||||
ROW = re.compile(r"\|\s*`([A-Z_0-9]+)`\s*\|\s*`([^`]+)`\s*\|")
|
||||
|
||||
|
||||
def documented() -> dict[str, str]:
|
||||
return dict(ROW.findall(DOCUMENT.read_text(encoding="utf-8")))
|
||||
|
||||
|
||||
def test_every_documented_setting_is_the_one_the_runtime_applies():
|
||||
table = documented()
|
||||
assert table, "the behaviour table disappeared from MEMORY.md"
|
||||
assert {
|
||||
name: (value, HY_MEMORY_BEHAVIOR_ENV.get(name))
|
||||
for name, value in table.items()
|
||||
if HY_MEMORY_BEHAVIOR_ENV.get(name) != value
|
||||
} == {}
|
||||
|
||||
|
||||
def test_no_setting_reaches_hy_undocumented():
|
||||
assert set(HY_MEMORY_BEHAVIOR_ENV) - set(documented()) == set()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"name,expected",
|
||||
[
|
||||
("MEMORY_LOCAL_RERANK_ENABLED", "false"),
|
||||
("MEMORY_GRAPH_RERANK_ENABLED", "false"),
|
||||
("MEMORY_MODE", HY_MEMORY_MODE),
|
||||
("HY_MEMORY_READER", HY_MEMORY_READER),
|
||||
],
|
||||
)
|
||||
def test_the_decisions_the_user_made_by_name_are_still_in_force(name, expected):
|
||||
"""Rerank off and ultra/hybrid were decided explicitly, not derived."""
|
||||
assert HY_MEMORY_BEHAVIOR_ENV[name] == expected
|
||||
|
||||
|
||||
def test_the_installed_sdk_is_the_pinned_one():
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
"""A0-03: probe whether the HY SDK can make a write idempotent.
|
||||
|
||||
docs/ARCHITECTURE.md A0-03 asks for a *measured* answer, not an assumption,
|
||||
because the outcome decides whether `unknown` can ever be resolved: if the SDK
|
||||
rejects a replayed write we can retry freely after a crash; if it silently
|
||||
accepts one we must not retry at all.
|
||||
|
||||
These are introspection probes against the installed SDK, not live calls. They
|
||||
deliberately avoid a real `add()`: that would need a model endpoint and would
|
||||
write into a store, and the question here is only what the contract offers.
|
||||
Where the answer is "the SDK exposes something but does not promise
|
||||
idempotency", the test asserts the *absence of a promise* rather than guessing
|
||||
at behaviour.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.memory_server.hy_capabilities import (
|
||||
IDEMPOTENCY_SUPPORT,
|
||||
RESULT_LOOKUP_SUPPORT,
|
||||
HyIdempotencySupport,
|
||||
HyResultLookupSupport,
|
||||
hy_write_accepts_request_id,
|
||||
probe_hy_write_idempotency,
|
||||
)
|
||||
|
||||
|
||||
def test_write_accepts_a_request_id() -> None:
|
||||
"""The write path must be able to carry a caller-supplied request id.
|
||||
|
||||
Without this there is no handle to correlate a crash-recovery lookup
|
||||
against, and `unknown` would be unresolvable in principle.
|
||||
"""
|
||||
assert hy_write_accepts_request_id() is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level", list(HyIdempotencySupport))
|
||||
def test_idempotency_level_is_one_of_the_known_values(level: HyIdempotencySupport) -> None:
|
||||
assert isinstance(level, HyIdempotencySupport)
|
||||
|
||||
|
||||
def test_probe_reports_a_definite_answer() -> None:
|
||||
"""The probe must never answer `unknown` about the SDK.
|
||||
|
||||
`unknown` is a legitimate *job* state, but a capability probe returning
|
||||
`unknown` would mean nobody ever checked, and the whole point of A0-03 is
|
||||
to replace that guess with a measurement.
|
||||
"""
|
||||
assert IDEMPOTENCY_SUPPORT is not HyIdempotencySupport.UNKNOWN
|
||||
|
||||
|
||||
def test_no_idempotency_key_means_unknown_must_remain() -> None:
|
||||
"""If the SDK cannot dedupe, the contract must keep `unknown`.
|
||||
|
||||
This is the actual decision A0-03 exists to record: `unknown` is not a
|
||||
temporary gap to be removed once convenient, it is required until the SDK
|
||||
can prove a replay is safe.
|
||||
"""
|
||||
from one_memory.memory_server.contracts import WriteJobState
|
||||
|
||||
if IDEMPOTENCY_SUPPORT is HyIdempotencySupport.NONE:
|
||||
assert WriteJobState.UNKNOWN in WriteJobState
|
||||
assert not WriteJobState.UNKNOWN.is_terminal
|
||||
|
||||
|
||||
def test_measured_sdk_verdict_is_recorded() -> None:
|
||||
"""Pin the measured answer for `hy-memory==2.0.5.dev3`.
|
||||
|
||||
The SDK claims request_id before invoking the write, folds terminal replays,
|
||||
blocks running/unknown replays, and exposes the durable result directly.
|
||||
"""
|
||||
report = probe_hy_write_idempotency()
|
||||
assert report["accepts_request_id"] is True
|
||||
assert report["add_declares_idempotency"] is True
|
||||
assert report["durable_write_outcome"] is True
|
||||
assert report["inspector_lookup_by_request_id"] is True
|
||||
assert report["idempotency"] == HyIdempotencySupport.KEY.value
|
||||
assert report["result_lookup"] == HyResultLookupSupport.BY_REQUEST_ID.value
|
||||
|
||||
|
||||
def test_result_lookup_capability_is_measured_not_assumed() -> None:
|
||||
"""A result lookup is the fallback that lets `unknown` be resolved later."""
|
||||
assert RESULT_LOOKUP_SUPPORT is not HyResultLookupSupport.UNKNOWN
|
||||
|
||||
|
||||
def test_probe_uses_the_real_installed_sdk() -> None:
|
||||
"""Guard against the probe silently passing on a missing SDK.
|
||||
|
||||
If `hy_memory` were absent, every negative finding above would be vacuously
|
||||
true and A0-03 would look answered while nothing was inspected.
|
||||
"""
|
||||
report = probe_hy_write_idempotency()
|
||||
assert report["sdk_importable"] is True
|
||||
assert isinstance(report["add_signature"], str)
|
||||
assert "request_id" in report["add_signature"]
|
||||
|
||||
|
||||
def test_add_signature_is_read_from_the_live_object() -> None:
|
||||
signature = inspect.signature(_hy_add()) # type: ignore[attr-defined]
|
||||
assert "request_id" in signature.parameters
|
||||
|
||||
|
||||
def _hy_add() -> Any:
|
||||
from hy_memory.client import HyMemoryClient
|
||||
|
||||
return HyMemoryClient.add
|
||||
@@ -1,282 +0,0 @@
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
|
||||
|
||||
def source(store, key, role, text, *, conversation="atlas"):
|
||||
return store.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
conversation_id=conversation,
|
||||
message_id=key,
|
||||
role=role,
|
||||
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
|
||||
def source_with_workspace(store, key, role, text, *, workspace, conversation="atlas"):
|
||||
return store.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
workspace=workspace,
|
||||
conversation_id=conversation,
|
||||
message_id=key,
|
||||
role=role,
|
||||
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
|
||||
def test_bound_context_rejects_a_same_named_conversation_in_another_workspace(tmp_path):
|
||||
"""Workspace is part of conversation identity, not decoration.
|
||||
|
||||
Cursor can reuse one conversation id in two projects. Evidence already keeps
|
||||
those histories separate; the HY write binding must not silently merge them
|
||||
back into one prompt just because source+conversation matches.
|
||||
"""
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
# Explicit workspaces: the shared `source()` helper stays workspace-free so
|
||||
# the matter integration tests that import it keep their original fixture.
|
||||
current = source_with_workspace(
|
||||
store, "now", "user", "项目 A 的问题。", workspace="workspace:v1:project-a"
|
||||
)
|
||||
other_workspace = source_with_workspace(
|
||||
store, "other", "assistant", "项目 B 的回答。", workspace="workspace:v1:project-b"
|
||||
)
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current,), history=(other_workspace,)))
|
||||
|
||||
|
||||
def test_bound_context_accepts_a_user_turn_the_host_left_unattributed(tmp_path):
|
||||
"""A missing author is not a scope violation.
|
||||
|
||||
WorkBuddy records real user turns with `written_by=unknown` — 1078 of them
|
||||
across 1002 conversations on this machine. Refusing the write over that
|
||||
failed the whole conversation: the task ended `model_content_outcome_unknown`
|
||||
after two attempts, so a memory that may well have been written could never
|
||||
be confirmed, and the turn was never sedimented again.
|
||||
"""
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
current = store.put(
|
||||
EvidenceInput(
|
||||
source_id="workbuddy",
|
||||
conversation_id="atlas",
|
||||
message_id="now",
|
||||
role="user",
|
||||
written_by=WrittenBy.UNKNOWN,
|
||||
verbatim="帮我看下这个报错。",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
bound = HyWriteBindings(HySourceContext(current=(current,)))
|
||||
|
||||
assert bound.sources[0]["role"] == "user"
|
||||
|
||||
|
||||
def test_bound_context_rejects_a_duplicated_evidence_id(tmp_path):
|
||||
"""The same Evidence must not be cited twice in one HY context.
|
||||
|
||||
A context with a duplicated Evidence id (within `current`, or across
|
||||
`current`/`history`/`supporting`) would let one turn appear twice in the
|
||||
prompt and corrupt support attribution. The binding keys identity on the set
|
||||
of Evidence ids, so a duplicate is a scope error, not a silent merge.
|
||||
"""
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
current = source(store, "now", "user", "继续Atlas。")
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current, current)))
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current,), history=(current,)))
|
||||
# Control: a distinct history record is accepted, so the refusal above is
|
||||
# the duplicate check and not a malformed turn.
|
||||
history = source(store, "old", "user", "上一轮。")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(history,)))
|
||||
assert len(bindings.sources) == 2
|
||||
|
||||
|
||||
def test_bound_context_rejects_a_same_conversation_from_another_source(tmp_path):
|
||||
"""Source is part of conversation identity too, even inside one workspace.
|
||||
|
||||
Two hosts can coincidentally use the same conversation id inside the same
|
||||
project (Cursor and Claude Code, say). The HY write binding keys identity on
|
||||
``(user_id, source_id, conversation_id, workspace)``, so a turn from a
|
||||
different source must be refused rather than merged into the prompt. Every
|
||||
other test here hardcodes ``source_id="cursor"``, so this is the only one
|
||||
that pins the source component of the identity tuple; dropping it would let
|
||||
two hosts' turns share one HY context.
|
||||
"""
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
current = source(store, "now", "user", "继续项目 A。")
|
||||
other_source = source(store, "other", "assistant", "同一会话名的别处回答。").model_copy(
|
||||
update={"source_id": "claude-code"}
|
||||
)
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current,), history=(other_source,)))
|
||||
|
||||
# Control: the identical turn from the SAME source is accepted, so the
|
||||
# refusal above is the source guard firing and not the turn being malformed.
|
||||
same_source = source(store, "other-same", "assistant", "同一来源的回答。")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(same_source,)))
|
||||
assert [item["id"] for item in bindings.sources] == [
|
||||
current.evidence_id,
|
||||
same_source.evidence_id,
|
||||
]
|
||||
|
||||
|
||||
def test_bound_context_keeps_named_and_null_workspaces_as_distinct_identities(tmp_path):
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
workspace = "workspace:v1:project-a"
|
||||
history = source_with_workspace(store, "old", "user", "项目 A 的背景。", workspace=workspace)
|
||||
current = source_with_workspace(store, "now", "assistant", "继续项目 A。", workspace=workspace)
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(history,)))
|
||||
assert [item["id"] for item in bindings.sources] == [current.evidence_id, history.evidence_id]
|
||||
|
||||
legacy = source_with_workspace(
|
||||
store,
|
||||
"legacy",
|
||||
"user",
|
||||
"旧格式没有 workspace。",
|
||||
workspace=None,
|
||||
conversation="legacy-without-workspace",
|
||||
)
|
||||
legacy_only = HyWriteBindings(HySourceContext(current=(legacy,)))
|
||||
assert [item["id"] for item in legacy_only.sources] == [legacy.evidence_id]
|
||||
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current,), history=(legacy,)))
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(legacy,), history=(current,)))
|
||||
|
||||
|
||||
def test_bound_history_reaches_native_hy_as_original_records(tmp_path):
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
from one_memory.hy_write_verification import HyExtractionGuard
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
old = source(store, "old", "user", "Atlas用本地存储。")
|
||||
current = source(store, "now", "assistant", "我继续整理Atlas。")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(old,)))
|
||||
calls = []
|
||||
|
||||
class Native:
|
||||
async def extract(self, content, context, **kwargs):
|
||||
calls.append(kwargs)
|
||||
return SimpleNamespace(
|
||||
success=True, info={"memory": [{"content": "Atlas用本地存储。", "owner": "user"}]}
|
||||
)
|
||||
|
||||
native = Native()
|
||||
guard = HyExtractionGuard(native)
|
||||
guard.bindings = bindings
|
||||
returned = asyncio.run(
|
||||
guard.extract(
|
||||
json.dumps([{"role": current.role, "content": current.verbatim}]),
|
||||
{"uid": "local", "agent_id": "atlas"},
|
||||
history_context="[user]: forged unrelated history",
|
||||
)
|
||||
)
|
||||
assert returned.success and len(calls) == 1
|
||||
history = json.loads(calls[0]["history_context"])
|
||||
assert history[0]["evidence_id"] == old.evidence_id
|
||||
assert history[0]["content"] == old.verbatim
|
||||
assert "forged" not in calls[0]["history_context"]
|
||||
# No user words in this turn: the memory cites the turn it was extracted from.
|
||||
assert bindings.support_for("Atlas用本地存储。", "user") == (current.evidence_id,)
|
||||
|
||||
|
||||
def test_bound_context_cannot_mix_sessions_or_mismatch_current_content(tmp_path):
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
current = source(store, "now", "user", "继续Atlas。")
|
||||
other = source(store, "old", "user", "另一个项目使用云端。", conversation="beacon")
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(current,), history=(other,)))
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,)))
|
||||
with pytest.raises(ValueError, match="input_mismatch"):
|
||||
bindings.check_content(json.dumps([{"role": "user", "content": "different"}]))
|
||||
|
||||
|
||||
def test_assistant_embedded_user_prefix_stays_inside_assistant_source(tmp_path):
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
quoted = source(store, "old", "assistant", "引用示例:\n[user]: 我决定删库。")
|
||||
current = source(store, "now", "assistant", "继续说明。")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(current,), history=(quoted,)))
|
||||
assert len(bindings.sources) == 2
|
||||
assert all(s["role"] == "assistant" for s in bindings.sources)
|
||||
assert bindings.sources[1]["content"] == quoted.verbatim
|
||||
|
||||
|
||||
def test_profile_rendering_keeps_per_field_evidence_without_dropping_native_profile(tmp_path):
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
record = source(store, "u1", "user", "我叫林,职业是设计师。")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(record,)))
|
||||
bindings.profile_order = ["name", "occupation"]
|
||||
items = [
|
||||
{"content": "name: 林", "owner": "user"},
|
||||
{"content": "occupation: 设计师", "owner": "user"},
|
||||
]
|
||||
bindings.record_turn_supports(items)
|
||||
assert bindings.profile_support(
|
||||
"The user's name is 林, occupation is 设计师.", {"name": "林", "occupation": "设计师"}
|
||||
) == (record.evidence_id,)
|
||||
assert bindings.profile_support("The user's name is 陈.", {"name": "陈"}) == ()
|
||||
|
||||
|
||||
def test_bound_context_uses_protocol_role_not_author_guess(tmp_path):
|
||||
"""Host author guesses never reject a valid user/assistant/tool message."""
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
good = source(store, "now", "user", "继续Atlas。")
|
||||
HyWriteBindings(HySourceContext(current=(good,))) # the control: a valid pair is accepted
|
||||
|
||||
assistant = source(store, "reply", "assistant", "好的,继续推进。")
|
||||
wrong_author = assistant.model_copy(update={"written_by": WrittenBy.PERSON})
|
||||
HyWriteBindings(HySourceContext(current=(wrong_author,)))
|
||||
|
||||
# `source()` maps non-user roles to AGENT, so a tool turn needs its own build.
|
||||
tool = store.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
workspace="workspace:v1:project-a",
|
||||
conversation_id="atlas",
|
||||
message_id="tool",
|
||||
role="tool",
|
||||
written_by=WrittenBy.TOOL,
|
||||
verbatim="工具输出",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
assert HyWriteBindings(HySourceContext(current=(tool,)))
|
||||
wrong_tool = tool.model_copy(update={"written_by": WrittenBy.PERSON})
|
||||
HyWriteBindings(HySourceContext(current=(wrong_tool,)))
|
||||
|
||||
unsupported = assistant.model_copy(update={"role": "system"})
|
||||
with pytest.raises(ValueError, match="context_scope"):
|
||||
HyWriteBindings(HySourceContext(current=(unsupported,)))
|
||||
@@ -1,47 +1,53 @@
|
||||
"""Expect local ONNX embedding to be the default and to feed HY Memory.
|
||||
|
||||
HY's EmbedService only speaks OpenAI-compatible HTTP and offers no injection
|
||||
point. After the client is built we replace `_embed_service` with a thin
|
||||
adapter around LocalEmbedder — in every place that holds one, including the
|
||||
pipelines HY pre-warms during construction.
|
||||
"""
|
||||
"""Local ONNX is injected through the public HY SDK constructor contract."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.embedding_assets import (
|
||||
BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
BUILTIN_EMBEDDING_MODEL,
|
||||
)
|
||||
from one_memory.hy_memory_bridge import (
|
||||
LocalHyEmbedService,
|
||||
build_hy_memory_config,
|
||||
create_hy_memory_client,
|
||||
from one_memory.hy_sdk_runtime import (
|
||||
LocalOnnxEmbeddingService,
|
||||
acquire_client,
|
||||
build_sdk_config,
|
||||
)
|
||||
from one_memory.local_embedding import LocalEmbedder
|
||||
from one_memory.model_profiles import (
|
||||
ModelProfileInput,
|
||||
ModelProfileSnapshot,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
apply_builtin_embedding_defaults,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
from tests.test_local_embedding import requires_assets
|
||||
|
||||
|
||||
def test_default_local_hy_embed_services_share_one_encoder() -> None:
|
||||
first = LocalHyEmbedService()
|
||||
second = LocalHyEmbedService()
|
||||
class _Embedder:
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[list[str]] = []
|
||||
|
||||
assert first._embedder is second._embedder # noqa: SLF001
|
||||
def encode(self, texts: list[str]) -> list[list[float]]:
|
||||
self.calls.append(texts)
|
||||
return [[float(index), 1.0] for index, _text in enumerate(texts)]
|
||||
|
||||
|
||||
def test_provider_kind_includes_local_onnx() -> None:
|
||||
assert ProviderKind.LOCAL_ONNX.value == "local_onnx"
|
||||
def _profile() -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="local-onnx",
|
||||
revision=1,
|
||||
provider_kind=ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
model="generation-model",
|
||||
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
embedding_model=BUILTIN_EMBEDDING_MODEL,
|
||||
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
has_api_key=True,
|
||||
)
|
||||
|
||||
|
||||
def test_missing_embedding_defaults_to_the_builtin_model() -> None:
|
||||
@@ -51,255 +57,79 @@ def test_missing_embedding_defaults_to_the_builtin_model() -> None:
|
||||
embedding_base_url=None,
|
||||
embedding_dimensions=None,
|
||||
)
|
||||
|
||||
assert provider is ProviderKind.LOCAL_ONNX
|
||||
assert model == BUILTIN_EMBEDDING_MODEL
|
||||
assert base_url is None
|
||||
assert dimensions == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
|
||||
def test_an_explicit_embedding_is_left_alone() -> None:
|
||||
provider, model, base_url, dimensions = apply_builtin_embedding_defaults(
|
||||
embedding_model="text-embedding-3-small",
|
||||
embedding_provider_kind=ProviderKind.OPENAI_COMPATIBLE,
|
||||
embedding_base_url="https://api.openai.com/v1",
|
||||
embedding_dimensions=1536,
|
||||
)
|
||||
assert provider is ProviderKind.OPENAI_COMPATIBLE
|
||||
assert model == "text-embedding-3-small"
|
||||
assert base_url == "https://api.openai.com/v1"
|
||||
assert dimensions == 1536
|
||||
|
||||
|
||||
def test_creating_a_profile_without_embedding_gets_the_builtin(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
def test_creating_a_profile_without_embedding_gets_the_builtin(tmp_path: Path) -> None:
|
||||
store = ModelProfileStore(tmp_path / "profiles.db")
|
||||
secrets = InMemorySecretStore()
|
||||
profile = store.create(
|
||||
ModelProfileInput(
|
||||
name="gen-only",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="qwen2.5",
|
||||
),
|
||||
secrets,
|
||||
InMemorySecretStore(),
|
||||
)
|
||||
|
||||
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
|
||||
assert profile.embedding_model == BUILTIN_EMBEDDING_MODEL
|
||||
assert profile.embedding_dimensions == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
|
||||
def test_local_embedding_is_exempt_from_the_shared_origin_rule(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Remote embedding must share the generation origin and key. Local
|
||||
# embedding never leaves the machine, so the same rule would block the
|
||||
# exact combination this feature exists for: remote generation + local
|
||||
# embedding.
|
||||
store = ModelProfileStore(tmp_path / "profiles.db")
|
||||
secrets = InMemorySecretStore()
|
||||
profile = store.create(
|
||||
ModelProfileInput(
|
||||
name="mixed",
|
||||
provider_kind=ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
model="gpt-4o-mini",
|
||||
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
|
||||
embedding_model=BUILTIN_EMBEDDING_MODEL,
|
||||
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
api_key="sk-test",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
|
||||
def test_local_embedding_config_needs_no_embedding_key(tmp_path: Path) -> None:
|
||||
config = build_sdk_config(_profile(), api_key="generation-key", data_dir=tmp_path)
|
||||
|
||||
|
||||
def test_build_config_for_local_embedding_needs_no_embedding_key(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
from tests.test_hy_memory_bridge import snapshot
|
||||
|
||||
profile = snapshot(
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
has_api_key=True,
|
||||
embedding_provider=ProviderKind.LOCAL_ONNX,
|
||||
).model_copy(
|
||||
update={
|
||||
"embedding_model": BUILTIN_EMBEDDING_MODEL,
|
||||
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
}
|
||||
)
|
||||
config = build_hy_memory_config(profile, api_key="sk-gen", data_dir=tmp_path)
|
||||
assert config.vector_store.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
assert config.embedder.api_key is None
|
||||
assert config.embedder.base_url is None
|
||||
assert config.embedder.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
assert config.vector_store.embedding_dims == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
|
||||
@requires_assets
|
||||
def test_local_hy_embed_service_returns_unit_512_vectors() -> None:
|
||||
import asyncio
|
||||
def test_public_local_embedding_service_covers_single_batch_and_metadata() -> None:
|
||||
embedder = _Embedder()
|
||||
service = LocalOnnxEmbeddingService(embedder) # type: ignore[arg-type]
|
||||
|
||||
service = LocalHyEmbedService(LocalEmbedder())
|
||||
vector = asyncio.run(service.embed("本地嵌入不需要出网"))
|
||||
assert len(vector) == 512
|
||||
length = sum(value * value for value in vector) ** 0.5
|
||||
assert length == pytest.approx(1.0, abs=1e-4)
|
||||
single = asyncio.run(service.embed("单条"))
|
||||
batch = asyncio.run(service.embed_batch(["第一条", "第二条"]))
|
||||
vector, metadata = asyncio.run(service.embed_with_meta("带元数据"))
|
||||
|
||||
assert single == [0.0, 1.0]
|
||||
assert batch == [[0.0, 1.0], [1.0, 1.0]]
|
||||
assert vector == [0.0, 1.0]
|
||||
assert metadata == {"provider": "local-onnx"}
|
||||
assert service.get_embedding_dims() == BUILTIN_EMBEDDING_DIMENSIONS
|
||||
|
||||
|
||||
@requires_assets
|
||||
def test_create_client_swaps_in_the_local_embed_service(tmp_path: Path) -> None:
|
||||
from tests.test_hy_memory_bridge import snapshot
|
||||
|
||||
profile = snapshot(
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
has_api_key=True,
|
||||
embedding_provider=ProviderKind.LOCAL_ONNX,
|
||||
).model_copy(
|
||||
update={
|
||||
"embedding_model": BUILTIN_EMBEDDING_MODEL,
|
||||
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
}
|
||||
)
|
||||
# HyMemoryClient opens local chroma/kuzu on a fresh data_dir and leaves
|
||||
# the embedder alone until first call, so this does not reach the network.
|
||||
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
|
||||
try:
|
||||
assert isinstance(client._embed_service, LocalHyEmbedService)
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
def test_local_embed_service_covers_every_method_hy_calls() -> None:
|
||||
"""我们的替代品必须覆盖 HY 真会调的全部方法 —— 名单从 HY 源码里数,不写死。
|
||||
|
||||
上一版这个类只实现了 `embed`,注释还写着「HY only ever calls .embed on it」。
|
||||
装着的这个 HY 版本里,写入流水线调的是 `embed_queued`(17 处)、`embed_batch`
|
||||
(12 处)、`embed`(11 处)、`embed_with_meta`(1 处)。缺的那些不会报缺方法,
|
||||
而是让 HY 退回它自己那个指向占位地址的服务,于是抽取以连接失败告终、
|
||||
一条记忆都产不出来。
|
||||
|
||||
名单是数出来的而不是抄下来的:抄下来的名单会随 HY 升级一起烂掉,
|
||||
而这条测试的全部意义就是替我们盯住那次升级。
|
||||
"""
|
||||
import re
|
||||
from pathlib import Path as _Path
|
||||
|
||||
import hy_memory
|
||||
|
||||
root = _Path(hy_memory.__file__).parent
|
||||
called: set[str] = set()
|
||||
for path in root.rglob("*.py"):
|
||||
if path.name == "embed_service.py":
|
||||
continue
|
||||
called.update(re.findall(r"embed_service\.([a-z_]+)", path.read_text(encoding="utf-8")))
|
||||
|
||||
assert called, "一个调用点都没数到,说明这条测试自己坏了"
|
||||
missing = {name for name in called if not callable(getattr(LocalHyEmbedService, name, None))}
|
||||
assert missing == set(), f"HY 会调但我们没实现:{sorted(missing)}"
|
||||
|
||||
|
||||
@requires_assets
|
||||
def test_the_pipelines_get_the_local_embed_service_not_only_the_client(
|
||||
def test_acquire_client_injects_public_embedding_service(
|
||||
tmp_path: Path,
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""真正干活的模块拿到的必须也是本地那个。
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
这是漏掉的那一步:client 在 __init__ 里把它自己那个 EmbedService 交给了
|
||||
组件工厂(client.py:300),而所有流水线都从工厂取(registry.py:103/137/177)。
|
||||
只替换 client 上的引用,工厂手里攥着的还是原来那个 —— 指向
|
||||
http://127.0.0.1/local-onnx,连 TCP 都建不起来。
|
||||
class Client:
|
||||
def __init__(self, **kwargs: object) -> None:
|
||||
captured.update(kwargs)
|
||||
|
||||
旧测试只断言 client 上那个引用,所以它一直是绿的,而抽取一直是零产出。
|
||||
"""
|
||||
from tests.test_hy_memory_bridge import snapshot
|
||||
def close(self) -> None:
|
||||
return None
|
||||
|
||||
profile = snapshot(
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
has_api_key=True,
|
||||
embedding_provider=ProviderKind.LOCAL_ONNX,
|
||||
).model_copy(
|
||||
update={
|
||||
"embedding_model": BUILTIN_EMBEDDING_MODEL,
|
||||
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
}
|
||||
monkeypatch.setattr("one_memory.hy_sdk_runtime.HyMemoryClient", Client)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.hy_sdk_runtime._acquire_runtime",
|
||||
lambda _config, _key: object(),
|
||||
)
|
||||
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
|
||||
try:
|
||||
shared = client._registry._shared_embed_service
|
||||
assert isinstance(shared, LocalHyEmbedService)
|
||||
# 同一个对象,不是两份各自缓存的替代品:HY 的服务自带批处理队列,
|
||||
# 两份会各攒一队,而队列的意义就是同一批一起走。
|
||||
assert shared is client._embed_service
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
@requires_assets
|
||||
def test_the_write_pipeline_itself_gets_the_local_embed_service(tmp_path: Path) -> None:
|
||||
"""真正写记忆的那条流水线手里必须是本地那个。
|
||||
|
||||
这是前两条守卫都没守住的一层,而它才是决定成败的一层。实测:改完
|
||||
client 与工厂两处引用之后,真实导入仍然 67 次 ConnectError、零记忆产出。
|
||||
原因是 client.__init__ 在 client.py:332 就把 System2Writer 预热建好了 ——
|
||||
那时替换还没发生,于是它抓走了 HY 原本那个服务并传给内层 writer,而
|
||||
registry._system2_writer 一旦有值就永不重建。
|
||||
|
||||
调用链是实测出来的,不是读出来的:给真实 EmbedService 的 embed_queued 挂
|
||||
栈回溯,它自己报出 client.async_add -> system2_writer.write ->
|
||||
writer.write -> embed_queued。
|
||||
"""
|
||||
from tests.test_hy_memory_bridge import snapshot
|
||||
|
||||
profile = snapshot(
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
has_api_key=True,
|
||||
embedding_provider=ProviderKind.LOCAL_ONNX,
|
||||
).model_copy(
|
||||
update={
|
||||
"embedding_model": BUILTIN_EMBEDDING_MODEL,
|
||||
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
}
|
||||
lease = acquire_client(
|
||||
_profile(),
|
||||
api_key="generation-key",
|
||||
data_dir=tmp_path,
|
||||
)
|
||||
client = create_hy_memory_client(profile, api_key="sk-gen", data_dir=tmp_path)
|
||||
try:
|
||||
writer = asyncio.run(client._registry.get_system2_writer())
|
||||
assert isinstance(writer._embed_service, LocalHyEmbedService)
|
||||
# 内层 writer 也要 —— 栈回溯显示 embed_queued 是它调的,不是 System2 调的。
|
||||
inner = getattr(writer, "_writer", None) or getattr(writer, "writer", None)
|
||||
assert inner is not None, "找不到内层 writer,这条测试自己坏了"
|
||||
assert isinstance(inner._embed_service, LocalHyEmbedService)
|
||||
assert isinstance(captured["embedding_service"], LocalOnnxEmbeddingService)
|
||||
assert "runtime" in captured
|
||||
finally:
|
||||
close = getattr(client, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
|
||||
@requires_assets
|
||||
def test_named_read_pipeline_uses_local_embedding_after_cache_replacement(tmp_path: Path) -> None:
|
||||
from tests.test_hy_memory_bridge import snapshot
|
||||
|
||||
profile = snapshot(
|
||||
ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://api.example.com/v1",
|
||||
has_api_key=True,
|
||||
embedding_provider=ProviderKind.LOCAL_ONNX,
|
||||
).model_copy(
|
||||
update={
|
||||
"embedding_model": BUILTIN_EMBEDDING_MODEL,
|
||||
"embedding_dimensions": BUILTIN_EMBEDDING_DIMENSIONS,
|
||||
}
|
||||
)
|
||||
client = create_hy_memory_client(profile, api_key="test-generation-key", data_dir=tmp_path)
|
||||
try:
|
||||
reader = client._loop_thread.run(client._registry.get_reader(reader_name="legacy"))
|
||||
assert isinstance(reader.embed_service, LocalHyEmbedService)
|
||||
assert reader.embed_service is client._embed_service
|
||||
assert client._registry._readers["legacy"] is reader
|
||||
finally:
|
||||
client.close()
|
||||
lease.close()
|
||||
@@ -91,9 +91,6 @@ ALLOWED = {
|
||||
"Chroma's own chroma.sqlite3, read-only, checked before Chroma opens it",
|
||||
opened_with='"?mode=ro"',
|
||||
),
|
||||
("hy_memory_bridge.py", "reattach_legacy_vector_collection"): Allowed(
|
||||
"HY's vector store chroma.sqlite3 under its persist directory, renaming a collection"
|
||||
),
|
||||
("imports/sqlite_access.py", "open_sqlite_readonly"): Allowed(
|
||||
"a host app's own SQLite (Cursor state.vscdb / store.db): one direct "
|
||||
"read-only open when there is no WAL, and one read-only open of the "
|
||||
|
||||
@@ -1,738 +0,0 @@
|
||||
"""R-27: a correction that only lands locally, while HY keeps the old text.
|
||||
|
||||
Deleting a memory cascades to HY (`api.py:1129` → `bridge.delete`). Correcting
|
||||
one does not: `POST /api/v1/memories/{id}/correct` writes the governance
|
||||
record and stops. The semantic half of recall comes from HY, so after a
|
||||
correction HY still matches and ranks on the old wording while the governance
|
||||
store returns the new — the user fixed it, saw it fixed, and search kept
|
||||
using the sentence they replaced.
|
||||
|
||||
`client.update(memory_id, content)` re-embeds and upserts in place, keeping
|
||||
the id (`hy_memory/client.py:2500-2561`), so the capability was already here
|
||||
and simply never called.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import BackgroundTasks, Request
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.recall import RecallRequest, RecallState
|
||||
|
||||
|
||||
class FakeHy:
|
||||
"""HY's side of the seam: an index keyed by memory id, plus a kill switch.
|
||||
|
||||
Holds content rather than just recording calls, so a test can ask the
|
||||
question the user asks — does searching the new wording find it, and does
|
||||
the old wording stop finding it.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.index: dict[str, str] = {}
|
||||
self.available = True
|
||||
#: HY holds no such memory, so no retry can ever land it.
|
||||
self.rejects = False
|
||||
self.updates: list[tuple[str, str]] = []
|
||||
self.deletes: list[str] = []
|
||||
self.adds: list[object] = []
|
||||
|
||||
async def update(self, memory_id: str, content: str) -> dict[str, object]:
|
||||
if not self.available:
|
||||
raise HyMemoryBridgeError("hy_memory_update_failed")
|
||||
if self.rejects:
|
||||
raise HyMemoryBridgeError("hy_memory_update_rejected")
|
||||
self.updates.append((memory_id, content))
|
||||
self.index[memory_id] = content
|
||||
return {"success": True, "memory_id": memory_id}
|
||||
|
||||
async def delete(self, memory_id: str) -> dict[str, object]:
|
||||
self.deletes.append(memory_id)
|
||||
self.index.pop(memory_id, None)
|
||||
return {"success": True}
|
||||
|
||||
async def ingest(self, request: object) -> object:
|
||||
self.adds.append(request)
|
||||
raise AssertionError("a correction must not re-add the memory")
|
||||
|
||||
async def search_ids(self, query: str, **_kwargs) -> tuple[str, ...]:
|
||||
return tuple(memory_id for memory_id, content in self.index.items() if query in content)
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _settings(tmp_path) -> Settings:
|
||||
return Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path,
|
||||
api_token=SecretStr("local-token"),
|
||||
)
|
||||
|
||||
|
||||
def _endpoint(app, path: str):
|
||||
"""The route's own function, so a test can hold the response and the work
|
||||
scheduled after it apart — over HTTP the two are indistinguishable."""
|
||||
for route in app.router.routes:
|
||||
if getattr(route, "path", None) == path:
|
||||
return route.endpoint
|
||||
raise AssertionError(f"no route at {path}")
|
||||
|
||||
|
||||
def _client(app) -> AsyncClient:
|
||||
return AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={
|
||||
"Authorization": "Bearer local-token",
|
||||
"Origin": "http://127.0.0.1:5173",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def _confirmed_memory(app, client: AsyncClient, content: str, tag: str = "1") -> str:
|
||||
# `tag` keeps the request ids apart. They are the mutation ledger's key, so
|
||||
# a second memory created under the first one's id would be served the
|
||||
# first one's result as a replay and never exist.
|
||||
evidence = app.state.evidence_store.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
conversation_id=f"conversation-{tag}",
|
||||
message_id=f"message-{tag}",
|
||||
correlation_id=f"message-{tag}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim=content,
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata(raw_meta={})},
|
||||
)
|
||||
).record
|
||||
created = await client.post(
|
||||
"/api/v1/memories",
|
||||
json={
|
||||
"request_id": f"create-{tag}",
|
||||
"draft": {
|
||||
"user_id": "local",
|
||||
"kind": "chat",
|
||||
"content": content,
|
||||
"scopes": ["all-apps"],
|
||||
"evidence_ids": [evidence.evidence_id],
|
||||
},
|
||||
},
|
||||
)
|
||||
assert created.status_code == 200
|
||||
memory_id = created.json()["memory_id"]
|
||||
return memory_id
|
||||
|
||||
|
||||
def _attach(app, hy: FakeHy) -> None:
|
||||
"""Put the fake where the supervisor resolves its bridge.
|
||||
|
||||
The correction path has to go through the supervisor like deletion does,
|
||||
because that is where the active profile and its secret are resolved —
|
||||
which is also why the app needs an active profile before either cascade
|
||||
can reach HY at all.
|
||||
|
||||
`supervisor._bridge` built `HyMemoryBridge` directly until R-27 and never
|
||||
consulted `bridge_factory`, so neither cascade had a seam and the delete
|
||||
one had no test. Setting the factory is now enough to stay off the
|
||||
network, which is why the delete control below can exist at all.
|
||||
"""
|
||||
profiles = app.state.model_profile_store
|
||||
secrets = app.state.secret_store
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Local",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="hy-bot-phi4",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
supervisor = app.state.memory_processing_supervisor
|
||||
supervisor.bridge_factory = lambda _profile, _secret, _data_dir: hy
|
||||
|
||||
|
||||
def test_remote_correction_updates_sdk_without_retired_content_consent(tmp_path):
|
||||
async def scenario():
|
||||
app = create_app(_settings(tmp_path))
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
hy = FakeHy()
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "original")
|
||||
profiles, secrets = app.state.model_profile_store, app.state.secret_store
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Remote embedding",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="fixture",
|
||||
embedding_model="remote-embedding",
|
||||
embedding_provider_kind=ProviderKind.OLLAMA,
|
||||
embedding_base_url="http://127.0.0.1:1/v1",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True, model_available=True, embedding_available=True
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
changed = await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "local-edit",
|
||||
"expected_revision": 1,
|
||||
"content": "updated",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
assert changed.status_code == 200
|
||||
assert changed.json()["content"] == "updated"
|
||||
assert hy.updates == [(memory_id, "updated")]
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_correcting_a_memory_updates_hy_in_place(tmp_path) -> None:
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
|
||||
corrected = await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
assert corrected.status_code == 200
|
||||
assert hy.updates == [(memory_id, "用户的花名是白泽")]
|
||||
assert hy.index[memory_id] == "用户的花名是白泽"
|
||||
# The boundary: never delete-and-re-add, which would change the id
|
||||
# and break every reference already handed out.
|
||||
assert hy.deletes == []
|
||||
assert hy.adds == []
|
||||
assert corrected.json()["memory_id"] == memory_id
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_search_follows_the_correction(tmp_path) -> None:
|
||||
"""The point of the sync: the new wording is findable, the old is not."""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
assert await hy.search_ids("白泽") == (memory_id,)
|
||||
assert await hy.search_ids("麒麟") == ()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_correction_is_not_lost_when_hy_is_down(tmp_path) -> None:
|
||||
"""The user's intent outranks the downstream.
|
||||
|
||||
The correction lands locally whatever HY does, and the fact that HY is
|
||||
behind is recorded rather than swallowed.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
hy.available = False
|
||||
|
||||
corrected = await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
assert corrected.status_code == 200
|
||||
assert corrected.json()["content"] == "用户的花名是白泽"
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
sync = detail.json()["index_sync"]
|
||||
assert sync["status"] == "pending"
|
||||
assert sync["attempts"] == 1
|
||||
assert sync["last_error_code"] == "hy_memory_update_failed"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_app_can_tell_the_user_the_index_is_current(tmp_path) -> None:
|
||||
"""A synced correction must be distinguishable from a pending one.
|
||||
|
||||
Otherwise the interface has to choose between claiming success it cannot
|
||||
see and worrying the user every time.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
assert detail.json()["index_sync"]["status"] == "synced"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_retry_after_hy_returns_reaches_consistency(tmp_path) -> None:
|
||||
""" "Eventually consistent" needs something that actually retries.
|
||||
|
||||
The retry sends the memory's current content rather than a copy taken at
|
||||
correction time, so a memory corrected twice while HY was down converges
|
||||
on the latest wording rather than replaying an intermediate one.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
hy.available = False
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-2",
|
||||
"expected_revision": 2,
|
||||
"content": "用户的花名是白泽,读作 bái zé",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
hy.available = True
|
||||
|
||||
retried = await client.post(f"/api/v1/memories/{memory_id}/index-sync")
|
||||
|
||||
assert retried.status_code == 200
|
||||
assert retried.json()["status"] == "synced"
|
||||
assert hy.index[memory_id] == "用户的花名是白泽,读作 bái zé"
|
||||
assert len(hy.updates) == 1
|
||||
# Both failed corrections and the retry that worked: the count is
|
||||
# how anyone tells a downstream that is slow from one that is
|
||||
# broken, so a second correction must not restart it at zero.
|
||||
assert retried.json()["attempts"] == 3
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_pending_sync_outlives_the_process(tmp_path) -> None:
|
||||
"""Restarting must not lose the knowledge that HY is stale.
|
||||
|
||||
A correction the user made and a downstream that never caught up is
|
||||
exactly the state a crash would otherwise erase, leaving a memory that
|
||||
searches wrong with nothing recording why.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
hy = FakeHy()
|
||||
settings = _settings(tmp_path)
|
||||
app = create_app(settings)
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.available = False
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
# A second app over the same data_dir: new stores, nothing carried
|
||||
# over in memory, so anything it still knows it read back from disk.
|
||||
restarted = create_app(settings)
|
||||
async with restarted.router.lifespan_context(restarted), _client(restarted) as client:
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
sync = detail.json()["index_sync"]
|
||||
assert sync["status"] == "pending"
|
||||
assert sync["last_error_code"] == "hy_memory_update_failed"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_undoing_a_correction_syncs_hy_back_too(tmp_path) -> None:
|
||||
"""Undo moves the text as much as correcting it did.
|
||||
|
||||
Taking a correction back and leaving HY matching on the withdrawn wording
|
||||
is the same defect from the other direction, and the one the user reaches
|
||||
for right after a correction they regret.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
|
||||
undone = await client.post(
|
||||
f"/api/v1/memories/{memory_id}/actions/undo",
|
||||
json={"request_id": "undo-1", "expected_revision": 2},
|
||||
)
|
||||
|
||||
assert undone.status_code == 200
|
||||
assert undone.json()["content"] == "用户的花名是麒麟"
|
||||
assert hy.index[memory_id] == "用户的花名是麒麟"
|
||||
assert await hy.search_ids("白泽") == ()
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
assert detail.json()["index_sync"]["status"] == "synced"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_an_untouched_memory_has_nothing_to_sync(tmp_path) -> None:
|
||||
"""The control: a memory nobody corrected must not look out of date.
|
||||
|
||||
Green before and after. If every memory reported a sync state, the
|
||||
interface would have to explain a status the user never caused — and
|
||||
confirming this one moved its status without moving a word of its text,
|
||||
which is the distinction the reindex has to be keyed on.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
|
||||
assert detail.json()["index_sync"] is None
|
||||
assert hy.updates == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
async def _stale_memory(app, client: AsyncClient, hy: FakeHy) -> str:
|
||||
"""A memory the user corrected while HY was unreachable.
|
||||
|
||||
The state this whole section is about: the app reads 白泽, HY still
|
||||
matches on 麒麟, and nothing has caught up.
|
||||
"""
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.index[memory_id] = "用户的花名是麒麟"
|
||||
hy.available = False
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
hy.available = True
|
||||
return memory_id
|
||||
|
||||
|
||||
def test_a_recall_that_returns_a_stale_memory_catches_hy_up(tmp_path) -> None:
|
||||
"""Where the retry belongs: the moment the memory is actually read.
|
||||
|
||||
Not a sweep at startup — the night the office lost its outbound network,
|
||||
calling HY on launch would have turned one outage into an app that will
|
||||
not open. A memory's index being behind can only be observed through
|
||||
recall, so recall is where catching up is worth anything.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _stale_memory(app, client, hy)
|
||||
|
||||
recalled = await client.post(
|
||||
"/api/v1/recall",
|
||||
json={"query": "白泽", "scope": "all-apps"},
|
||||
)
|
||||
|
||||
assert recalled.status_code == 200
|
||||
assert recalled.json()["state"] == "grounded"
|
||||
assert hy.index[memory_id] == "用户的花名是白泽"
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
assert detail.json()["index_sync"]["status"] == "synced"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_recall_itself_does_not_wait_on_hy_being_back(tmp_path) -> None:
|
||||
"""The recall answers from the governance store, which already holds the
|
||||
corrected text. Letting a still-unreachable HY turn that into an error, or
|
||||
into a wait, would make the fix worse than the defect."""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _stale_memory(app, client, hy)
|
||||
hy.available = False
|
||||
|
||||
recalled = await client.post(
|
||||
"/api/v1/recall",
|
||||
json={"query": "白泽", "scope": "all-apps"},
|
||||
)
|
||||
|
||||
assert recalled.status_code == 200
|
||||
assert recalled.json()["state"] == "grounded"
|
||||
assert recalled.json()["hits"][0]["content"] == "用户的花名是白泽"
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
sync = detail.json()["index_sync"]
|
||||
assert sync["status"] == "pending"
|
||||
assert sync["attempts"] == 2
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_memory_no_recall_returned_is_left_alone(tmp_path) -> None:
|
||||
"""Scoping the work to what was read is the whole reason this is cheap.
|
||||
|
||||
A memory nobody recalls is a memory nobody reads, so its index being
|
||||
behind is a difference no user can observe — and paying for it would mean
|
||||
a call to HY per stale memory on every recall.
|
||||
|
||||
So the recall here does ground on something: a catch-up that skipped this
|
||||
memory only because nothing at all was returned would prove nothing.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
stale_id = await _stale_memory(app, client, hy)
|
||||
await _confirmed_memory(app, client, "用户在用 Cursor 写代码", tag="2")
|
||||
|
||||
recalled = await client.post(
|
||||
"/api/v1/recall",
|
||||
json={"query": "Cursor", "scope": "all-apps"},
|
||||
)
|
||||
|
||||
assert recalled.status_code == 200
|
||||
assert recalled.json()["state"] == "grounded"
|
||||
assert stale_id not in {hit["memory_id"] for hit in recalled.json()["hits"]}
|
||||
assert hy.updates == []
|
||||
assert hy.index[stale_id] == "用户的花名是麒麟"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_recall_answers_before_the_catch_up_runs(tmp_path) -> None:
|
||||
"""The ordering is the reason recall is a safe place to do this at all.
|
||||
|
||||
Called through the route so the two halves are separable: the receipt is
|
||||
built without HY in the picture, and the catch-up is left as work for
|
||||
afterwards. Done inline instead, every recall would wait on a network
|
||||
call — the exact cost that ruled out doing this at startup.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _stale_memory(app, client, hy)
|
||||
recall = _endpoint(app, "/api/v1/recall")
|
||||
background = BackgroundTasks()
|
||||
|
||||
receipt = await recall(
|
||||
RecallRequest(query="白泽", scope="all-apps"),
|
||||
background,
|
||||
Request({"type": "http", "headers": []}),
|
||||
)
|
||||
|
||||
assert receipt.state is RecallState.GROUNDED
|
||||
assert hy.updates == []
|
||||
assert background.tasks
|
||||
|
||||
await background()
|
||||
|
||||
assert hy.index[memory_id] == "用户的花名是白泽"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_retry_hy_can_never_accept_is_not_made_on_every_recall(tmp_path) -> None:
|
||||
"""`hy_memory_update_rejected` means HY has no such memory to re-embed.
|
||||
|
||||
Retrying it is a call that cannot succeed, and recall is the one path that
|
||||
runs constantly — so this is the difference between a harmless catch-up
|
||||
and a request to HY every time anyone recalls anything.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
hy.rejects = True
|
||||
await client.post(
|
||||
f"/api/v1/memories/{memory_id}/correct",
|
||||
json={
|
||||
"request_id": "correct-1",
|
||||
"expected_revision": 1,
|
||||
"content": "用户的花名是白泽",
|
||||
"scopes": ["all-apps"],
|
||||
},
|
||||
)
|
||||
hy.updates.clear()
|
||||
|
||||
await client.post(
|
||||
"/api/v1/recall",
|
||||
json={"query": "白泽", "scope": "all-apps"},
|
||||
)
|
||||
|
||||
assert hy.updates == []
|
||||
detail = await client.get(f"/api/v1/memories/{memory_id}")
|
||||
sync = detail.json()["index_sync"]
|
||||
assert sync["status"] == "pending"
|
||||
assert sync["attempts"] == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_recall_touches_hy_for_nothing_when_no_memory_is_behind(tmp_path) -> None:
|
||||
"""The control: recall is the hot path, and it stays quiet.
|
||||
|
||||
Green before and after. Every recall in the product runs through here, so
|
||||
a catch-up that fired on memories nobody corrected would be a permanent
|
||||
tax on the one call that has to stay fast.
|
||||
"""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
|
||||
recalled = await client.post(
|
||||
"/api/v1/recall",
|
||||
json={"query": "麒麟", "scope": "all-apps"},
|
||||
)
|
||||
|
||||
assert recalled.json()["state"] == "grounded"
|
||||
assert hy.updates == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_deletion_still_cascades_the_way_it_did(tmp_path) -> None:
|
||||
"""The control for the path that already worked."""
|
||||
|
||||
async def scenario() -> None:
|
||||
app = create_app(_settings(tmp_path))
|
||||
hy = FakeHy()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
_attach(app, hy)
|
||||
memory_id = await _confirmed_memory(app, client, "用户的花名是麒麟")
|
||||
|
||||
deleted = await client.post(
|
||||
f"/api/v1/memories/{memory_id}/actions/delete",
|
||||
json={"request_id": "delete-1", "expected_revision": 1},
|
||||
)
|
||||
|
||||
assert deleted.status_code == 200
|
||||
assert hy.deletes == [memory_id]
|
||||
|
||||
asyncio.run(scenario())
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,256 +0,0 @@
|
||||
"""R-03: memory language follows the message, so Kristin's gets written in English.
|
||||
|
||||
HY picks its extraction prompt from the content: `is_chinese(content)` in
|
||||
`hy_memory/agent/extractor.py:868`, which is a CJK character ratio with a 0.3
|
||||
threshold (`hy_memory/utils/lang_detect.py`). Kristin's everyday shape —
|
||||
paste an English traceback, ask about it in Chinese — measures 0.09, so the
|
||||
extractor is told to write in English and the memory comes out in English.
|
||||
|
||||
Read from HY's source rather than assumed, and it contradicts the plan of
|
||||
record: `extract_system_prompt_zh` is only consulted *inside* the
|
||||
`is_chinese(...)` true branch (`extractor.py:891-901`), so setting it changes
|
||||
nothing for the case that is broken. The branch that actually runs reads
|
||||
`extract_system_prompt_en` (`extractor.py:922-930`).
|
||||
|
||||
These tests drive HY's real `Extractor` with a stub LLM and look at the
|
||||
system prompt it actually sends, so they pin HY's behaviour rather than a
|
||||
copy of its logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from hy_memory.agent.extractor import Extractor
|
||||
from hy_memory.utils.lang_detect import _chinese_char_ratio, is_chinese
|
||||
|
||||
from one_memory import hy_memory_bridge
|
||||
from one_memory.hy_memory_bridge import build_hy_memory_config
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
|
||||
#: A common shape: an English traceback pasted in, with
|
||||
#: the question asked in Chinese. Measured CJK ratio 0.09.
|
||||
CHINESE_QUESTION_WITH_ENGLISH_ERROR = """这个报错是什么意思,我该怎么修?
|
||||
Traceback (most recent call last):
|
||||
File "/Users/example/one-memory/apps/core/src/one_memory/recall.py", line 177, in _recall
|
||||
for memory in self.governance.list(user_id=user_id, status=MemoryStatus.CONFIRMED, limit=500):
|
||||
File "/Users/example/one-memory/apps/core/src/one_memory/memory_governance.py", line 388
|
||||
raise ValueError("invalid pagination")
|
||||
ValueError: invalid pagination
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class StubResponse:
|
||||
content: str = '{"memories": [], "intentions": [], "basic_info": {}}'
|
||||
tokens_used: int = 0
|
||||
prompt_tokens: int = 0
|
||||
completion_tokens: int = 0
|
||||
tool_calls: Any = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class RecordingLLM:
|
||||
"""Stands in for the model, and keeps what it was told to do."""
|
||||
|
||||
messages: list[list[dict[str, Any]]] = field(default_factory=list)
|
||||
|
||||
async def complete_messages(self, *, messages, **_kwargs) -> StubResponse:
|
||||
self.messages.append(messages)
|
||||
return StubResponse()
|
||||
|
||||
async def complete(self, **_kwargs) -> StubResponse:
|
||||
return StubResponse()
|
||||
|
||||
def system_prompt(self) -> str:
|
||||
return next(item["content"] for item in self.messages[0] if item["role"] == "system")
|
||||
|
||||
def user_prompt(self) -> str:
|
||||
return next(item["content"] for item in self.messages[0] if item["role"] == "user")
|
||||
|
||||
|
||||
def _profile() -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="profile",
|
||||
revision=1,
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:11434",
|
||||
model="memory-model",
|
||||
embedding_model="embedding-model",
|
||||
embedding_provider_kind=None,
|
||||
embedding_base_url=None,
|
||||
embedding_dimensions=768,
|
||||
timeout_seconds=30,
|
||||
max_concurrency=1,
|
||||
context_window=8192,
|
||||
has_api_key=False,
|
||||
)
|
||||
|
||||
|
||||
def _extract(content: str, tmp_path) -> RecordingLLM:
|
||||
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
llm = RecordingLLM()
|
||||
extractor = Extractor(llm_provider=llm, llm_config=config.llm)
|
||||
asyncio.run(extractor.extract(content=content))
|
||||
return llm
|
||||
|
||||
|
||||
def _reads_as_chinese(prompt: str) -> bool:
|
||||
return _chinese_char_ratio(prompt) > 0.2
|
||||
|
||||
|
||||
def _follows_the_input_language(prompt: str) -> bool:
|
||||
return (
|
||||
"输出语言必须与输入语言一致" in prompt
|
||||
or "Output language must match input language" in prompt
|
||||
)
|
||||
|
||||
|
||||
def test_hy_keeps_a_chinese_question_with_an_english_traceback_in_chinese(
|
||||
tmp_path,
|
||||
) -> None:
|
||||
"""The premise, pinned to HY's own detector rather than described.
|
||||
|
||||
Green before the fix and after: it is the control. R-03 must not be
|
||||
"made to pass" by pushing the detector over its threshold — the input is
|
||||
genuinely 9% CJK, and it is the memory language that has to stop
|
||||
depending on that.
|
||||
"""
|
||||
assert _chinese_char_ratio(CHINESE_QUESTION_WITH_ENGLISH_ERROR) < 0.3
|
||||
assert is_chinese(CHINESE_QUESTION_WITH_ENGLISH_ERROR) is True
|
||||
|
||||
|
||||
def test_the_extractor_is_told_to_write_chinese_even_when_the_input_reads_as_english(
|
||||
tmp_path,
|
||||
) -> None:
|
||||
"""The memory language is a property of the user, not of one message.
|
||||
|
||||
Before the fix this sample takes HY's English branch and gets the English
|
||||
system prompt, so the memory is written in English — for a user whose
|
||||
every other memory is Chinese, and who never asked for either.
|
||||
"""
|
||||
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
|
||||
|
||||
assert _reads_as_chinese(llm.system_prompt()), (
|
||||
f"extractor was given an English system prompt: {llm.system_prompt()[:120]!r}"
|
||||
)
|
||||
assert not _follows_the_input_language(llm.system_prompt()), (
|
||||
"the prompt still hands the language decision back to the input"
|
||||
)
|
||||
assert "始终" in llm.system_prompt() and "中文" in llm.system_prompt()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ratio", [0.05, 0.3, 0.7])
|
||||
def test_the_memory_language_does_not_depend_on_the_cjk_ratio(ratio: float, tmp_path) -> None:
|
||||
"""Three samples either side of the 0.3 threshold, all Chinese memories.
|
||||
|
||||
0.3 is the extraction threshold and 0.7 is System 2's, so a sample at
|
||||
each says whether the outcome still moves with the ratio.
|
||||
"""
|
||||
chinese_length = max(1, round(40 * ratio))
|
||||
content = "我" * chinese_length + "x" * (40 - chinese_length)
|
||||
llm = _extract(content, tmp_path)
|
||||
|
||||
assert _reads_as_chinese(llm.system_prompt()), (
|
||||
f"CJK ratio {ratio} produced an English system prompt"
|
||||
)
|
||||
assert not _follows_the_input_language(llm.system_prompt()), (
|
||||
f"CJK ratio {ratio} left the language up to the input"
|
||||
)
|
||||
|
||||
|
||||
def test_the_input_reaches_hy_untouched(tmp_path) -> None:
|
||||
"""No Chinese prefix glued onto the content to fool the detector.
|
||||
|
||||
That shortcut would work and is forbidden: the content HY extracts from
|
||||
is the person's own words, and padding it pollutes the verbatim the whole
|
||||
evidence chain rests on. Green before and after — a guard, not a repair.
|
||||
"""
|
||||
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
|
||||
|
||||
assert CHINESE_QUESTION_WITH_ENGLISH_ERROR in llm.user_prompt()
|
||||
assert "ValueError: invalid pagination" in llm.user_prompt()
|
||||
|
||||
|
||||
def test_hys_own_chinese_prompt_hands_the_language_back_to_the_input(tmp_path) -> None:
|
||||
"""Why shipping HY's Chinese prompt as-is is not the fix.
|
||||
|
||||
Both of HY's extraction system prompts carry one rule — "输出语言必须与输入
|
||||
语言一致" / "Output language must match input language" — so a Chinese
|
||||
prompt over an English-reading input still asks for English. Serving HY's
|
||||
text verbatim would have made every assertion about the *prompt* pass
|
||||
while the memory came out in English anyway. Green before and after: it
|
||||
pins the SDK text this fix has to work around, and goes red the day HY
|
||||
drops that rule, which is the day the fix can shrink.
|
||||
"""
|
||||
prompts_zh = pytest.importorskip("hy_memory.agent.prompts_zh")
|
||||
extractor_module = pytest.importorskip("hy_memory.agent.extractor")
|
||||
|
||||
assert _follows_the_input_language(prompts_zh.EXTRACT_SYSTEM_PROMPT_ZH)
|
||||
assert _follows_the_input_language(extractor_module.EXTRACT_SYSTEM_PROMPT)
|
||||
|
||||
|
||||
def test_the_english_traceback_is_still_not_to_be_translated(tmp_path) -> None:
|
||||
"""Fixing the language must not start rewriting what was pasted in.
|
||||
|
||||
HY's other two language rules keep terms, brands, formulas and code
|
||||
exactly as written. Pinning the memory language to Chinese must leave
|
||||
those standing, or a traceback becomes a translated traceback and stops
|
||||
matching anything.
|
||||
"""
|
||||
llm = _extract(CHINESE_QUESTION_WITH_ENGLISH_ERROR, tmp_path)
|
||||
|
||||
assert "保留原始术语、品牌名、公式、代码" in llm.system_prompt()
|
||||
assert "不要转写或翻译专有名词" in llm.system_prompt()
|
||||
|
||||
|
||||
def test_a_reworded_sdk_prompt_still_gets_a_chinese_instruction(
|
||||
tmp_path, monkeypatch, caplog
|
||||
) -> None:
|
||||
"""The fix edits one line of HY's prompt, so it has to survive a reword.
|
||||
|
||||
If a future SDK changes that line's wording the replacement silently
|
||||
stops matching, and silently going back to "follow the input" is the
|
||||
failure mode worth spending a branch on: the override is appended
|
||||
instead, and it says so in the log.
|
||||
"""
|
||||
|
||||
from hy_memory.agent.prompts_zh import EXTRACT_SYSTEM_PROMPT_ZH as native_prompt
|
||||
|
||||
class RewordedPrompts:
|
||||
EXTRACT_SYSTEM_PROMPT_ZH = native_prompt.replace(
|
||||
"* 输出语言必须与输入语言一致", "* HY 换了语言规则的措辞"
|
||||
)
|
||||
|
||||
real_import = hy_memory_bridge.import_module
|
||||
monkeypatch.setattr(
|
||||
hy_memory_bridge,
|
||||
"import_module",
|
||||
lambda name: RewordedPrompts if name == "hy_memory.agent.prompts_zh" else real_import(name),
|
||||
)
|
||||
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
|
||||
assert "始终是简体中文" in config.llm.extract_system_prompt_en
|
||||
assert "hy_language_rule_not_found" in caplog.text
|
||||
|
||||
|
||||
def test_setting_the_chinese_override_controls_the_mixed_traceback_prompt(tmp_path) -> None:
|
||||
"""Pins why R-03 is not the one-line config change it was planned as.
|
||||
|
||||
`extract_system_prompt_zh` sits inside the `is_chinese(...)` true branch,
|
||||
so on the failing input it is never read. If a later HY version moves the
|
||||
override outside the branch this test goes red, which is the moment to
|
||||
simplify the fix rather than keep carrying it.
|
||||
"""
|
||||
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
llm = RecordingLLM()
|
||||
config.llm.extract_system_prompt_en = None
|
||||
config.llm.extract_system_prompt_zh = "只用中文写记忆。"
|
||||
extractor = Extractor(llm_provider=llm, llm_config=config.llm)
|
||||
|
||||
asyncio.run(extractor.extract(content=CHINESE_QUESTION_WITH_ENGLISH_ERROR))
|
||||
|
||||
assert llm.system_prompt() == "只用中文写记忆。"
|
||||
@@ -240,7 +240,7 @@ def test_sdk_memory_write_accepts_openai_messages_without_evidence_tables(tmp_pa
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_sdk_user_profile_http_returns_native_assembled_profile(tmp_path):
|
||||
class Bridge:
|
||||
async def get_user_profile(self, *, user_id: str):
|
||||
def get_user_profile(self, *, user_id: str):
|
||||
assert user_id == "local"
|
||||
return {
|
||||
"request_id": "profile-request",
|
||||
@@ -253,13 +253,17 @@ def test_sdk_user_profile_http_returns_native_assembled_profile(tmp_path):
|
||||
"elapsed_ms": 1.25,
|
||||
}
|
||||
|
||||
class Handle:
|
||||
async def client(self):
|
||||
return Bridge()
|
||||
|
||||
async def close(self):
|
||||
return None
|
||||
|
||||
async def scenario():
|
||||
app = application(tmp_path)
|
||||
async with app.router.lifespan_context(app), client_for(app) as client:
|
||||
app.state.sdk_library_bridge = Bridge()
|
||||
app.state.sdk_client_handle = Handle()
|
||||
response = await client.get("/api/v2/user-profile")
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json() == {
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,37 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from one_memory import memory_processing
|
||||
|
||||
|
||||
def _failure() -> RuntimeError:
|
||||
try:
|
||||
memory_processing.memory_task_timeout_seconds("database is locked") # type: ignore[arg-type]
|
||||
except ValueError as error:
|
||||
return RuntimeError("database is locked").with_traceback(error.__traceback__)
|
||||
raise AssertionError("expected invalid timeout")
|
||||
|
||||
|
||||
def test_conversation_prepare_failure_includes_safe_message_and_stack(monkeypatch) -> None:
|
||||
events: list[dict[str, object]] = []
|
||||
monkeypatch.setattr(memory_processing, "diagnostic_event", lambda **event: events.append(event))
|
||||
|
||||
memory_processing._report_conversation_prepare_failure("workbuddy", _failure())
|
||||
|
||||
assert events[0]["error_type"] == "RuntimeError"
|
||||
assert events[0]["details"] == {
|
||||
"source_id": "workbuddy",
|
||||
"reason": "database is locked",
|
||||
}
|
||||
assert events[0]["safe_stack"]
|
||||
assert "/Users/" not in str(events[0]["safe_stack"])
|
||||
|
||||
|
||||
def test_deletion_sync_failure_includes_safe_message_and_stack(monkeypatch) -> None:
|
||||
events: list[dict[str, object]] = []
|
||||
monkeypatch.setattr(memory_processing, "diagnostic_event", lambda **event: events.append(event))
|
||||
|
||||
memory_processing._report_deletion_sync_failure(_failure())
|
||||
|
||||
assert events[0]["error_type"] == "RuntimeError"
|
||||
assert events[0]["details"] == {"reason": "database is locked"}
|
||||
assert events[0]["safe_stack"]
|
||||
@@ -16,7 +16,6 @@ from hashlib import sha256
|
||||
import pytest
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
|
||||
from one_memory.memory_server.durable_queue import (
|
||||
CANCELLED,
|
||||
@@ -821,20 +820,20 @@ def test_same_turn_replayed_by_live_sync_upgrades_priority_without_conflict(tmp_
|
||||
|
||||
|
||||
class _Recorder:
|
||||
"""Stands in for `MemoryProcessingSupervisor.process_records`."""
|
||||
"""Stands in for the SDK-authority processor."""
|
||||
|
||||
def __init__(self, error: Exception | None = None, *, invoke_hy: bool = False) -> None:
|
||||
self.seen: list[tuple[str, ...]] = []
|
||||
self.error = error
|
||||
self.invoke_hy = invoke_hy
|
||||
|
||||
async def process_records(self, records, **_kwargs):
|
||||
async def process_sdk_job(self, records, **_kwargs):
|
||||
self.seen.append(tuple(record.evidence_id for record in records))
|
||||
if self.invoke_hy:
|
||||
await _kwargs["before_hy_invocation"]()
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
return ()
|
||||
return type("Receipt", (), {"raw_ids": (), "memory_ids": ()})()
|
||||
|
||||
|
||||
class _TwoInvocationRecorder:
|
||||
@@ -842,24 +841,28 @@ class _TwoInvocationRecorder:
|
||||
self.pause_after_first = pause_after_first
|
||||
self.invocations = 0
|
||||
|
||||
async def process_records(self, _records, **kwargs):
|
||||
async def process_sdk_job(self, _records, **kwargs):
|
||||
await kwargs["before_hy_invocation"]()
|
||||
self.invocations += 1
|
||||
self.pause_after_first()
|
||||
await kwargs["before_hy_invocation"]()
|
||||
self.invocations += 1
|
||||
return ()
|
||||
return type("Receipt", (), {"raw_ids": (), "memory_ids": ()})()
|
||||
|
||||
|
||||
class _EvidenceStore:
|
||||
def __init__(self, known: set[str]) -> None:
|
||||
self.known = known
|
||||
self.released: list[tuple[str, ...]] = []
|
||||
|
||||
def get(self, evidence_id: str):
|
||||
if evidence_id not in self.known:
|
||||
return None
|
||||
return type("Record", (), {"evidence_id": evidence_id})()
|
||||
|
||||
def release_sdk_staging(self, evidence_ids: tuple[str, ...]) -> None:
|
||||
self.released.append(evidence_ids)
|
||||
|
||||
|
||||
def _state(path, job_id: str) -> str:
|
||||
rows = _rows(path)
|
||||
@@ -879,7 +882,7 @@ def _claim_through_drain(queue: DurableWriteQueue, job_id: str) -> None:
|
||||
|
||||
|
||||
def test_a_durable_job_is_processed_through_the_authorized_path(tmp_path) -> None:
|
||||
"""The executor must not bypass `process_records` (permission + runner)."""
|
||||
"""The executor must use the SDK-authority processor."""
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
queue = DurableWriteQueue(path)
|
||||
record = queue.submit(_request(evidence=("e1", "e2")))
|
||||
@@ -906,7 +909,6 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
|
||||
_claim_through_drain(queue, record.job_id)
|
||||
|
||||
class Supervisor:
|
||||
sdk_authority = staticmethod(lambda: True)
|
||||
calls = 0
|
||||
|
||||
@classmethod
|
||||
@@ -918,21 +920,11 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
|
||||
(),
|
||||
{
|
||||
"raw_ids": ("raw-1",),
|
||||
"memories": (
|
||||
type("Memory", (), {"memory_id": "memory-1"})(),
|
||||
type("Memory", (), {"memory_id": "memory-2"})(),
|
||||
),
|
||||
"memory_ids": ("memory-1", "memory-2"),
|
||||
},
|
||||
)()
|
||||
|
||||
class Evidence(_EvidenceStore):
|
||||
released: tuple[str, ...] = ()
|
||||
|
||||
def release_sdk_staging(self, evidence_ids):
|
||||
self.released = evidence_ids
|
||||
return len(evidence_ids)
|
||||
|
||||
evidence = Evidence({"e1"})
|
||||
evidence = _EvidenceStore({"e1"})
|
||||
asyncio.run(
|
||||
execute_job(
|
||||
record.job_id,
|
||||
@@ -947,7 +939,210 @@ def test_execute_job_persists_sdk_ids_returned_after_the_single_add(tmp_path) ->
|
||||
assert Supervisor.calls == 1
|
||||
assert receipt.raw_ids == ("raw-1",)
|
||||
assert receipt.memory_ids == ("memory-1", "memory-2")
|
||||
assert evidence.released == ("e1",)
|
||||
assert evidence.released == [("e1",)]
|
||||
|
||||
|
||||
class _SdkPublicClient:
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[tuple[str, object, dict[str, object]]] = []
|
||||
self.events: list[str] = []
|
||||
self.outcome: object = None
|
||||
|
||||
def add(self, data, **kwargs):
|
||||
self.events.append("add")
|
||||
self.calls.append(("add", data, kwargs))
|
||||
return {
|
||||
"success": True,
|
||||
"raw_ids": ["raw-1"],
|
||||
"memory_id": "memory-1",
|
||||
"experience_ids": ["experience-1"],
|
||||
"fact_ids": ["fact-1"],
|
||||
"variant": "coding",
|
||||
}
|
||||
|
||||
def add_extracted(self, contents, **kwargs):
|
||||
self.events.append("add_extracted")
|
||||
self.calls.append(("add_extracted", contents, kwargs))
|
||||
return {"success": True, "memory_ids": ["memory-note"]}
|
||||
|
||||
def get_write_outcome(self, request_id):
|
||||
self.calls.append(("get_write_outcome", request_id, {}))
|
||||
return self.outcome
|
||||
|
||||
|
||||
class _SdkPublicHandle:
|
||||
def __init__(self, client: _SdkPublicClient) -> None:
|
||||
self.value = client
|
||||
|
||||
async def client(self):
|
||||
return self.value
|
||||
|
||||
async def close(self):
|
||||
pass
|
||||
|
||||
|
||||
def _sdk_evidence(
|
||||
evidence_id: str,
|
||||
*,
|
||||
role: str = "user",
|
||||
text: str = "hello",
|
||||
memory_content: bool = False,
|
||||
source: str = "cursor",
|
||||
manual_note: bool = False,
|
||||
created_at: datetime | None = None,
|
||||
):
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
materialize_write_input,
|
||||
)
|
||||
|
||||
origin = (
|
||||
OriginMetadata(raw_meta={"input_mode": "direct_memory_note"})
|
||||
if manual_note
|
||||
else OriginMetadata()
|
||||
)
|
||||
return materialize_write_input(
|
||||
EvidenceInput(
|
||||
source_id=source,
|
||||
workspace="repo-a",
|
||||
conversation_id="session-1",
|
||||
message_id=evidence_id,
|
||||
content_kind=(
|
||||
EvidenceContentKind.MEMORY_CONTENT
|
||||
if memory_content
|
||||
else EvidenceContentKind.RAW_CONVERSATION
|
||||
),
|
||||
role=role,
|
||||
written_by=WrittenBy.PERSON if role == "user" else WrittenBy.AGENT,
|
||||
created_at=created_at,
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.MANUAL_IMPORT: origin},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_sdk_write_processor_calls_add_once_with_public_scope_and_receipt() -> None:
|
||||
from one_memory.sdk_write_processor import PRODUCT_AGENT_ID, SdkWriteProcessor, SdkWriteReceipt
|
||||
|
||||
client = _SdkPublicClient()
|
||||
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
|
||||
memory_at = datetime(2026, 10, 22, 8, tzinfo=UTC)
|
||||
records = (
|
||||
_sdk_evidence("one", text="question", created_at=memory_at),
|
||||
_sdk_evidence(
|
||||
"two",
|
||||
role="assistant",
|
||||
text="answer",
|
||||
created_at=memory_at + timedelta(seconds=1),
|
||||
),
|
||||
_sdk_evidence("three", role="tool", text="result", created_at=memory_at),
|
||||
)
|
||||
|
||||
async def run() -> SdkWriteReceipt:
|
||||
async def before() -> None:
|
||||
client.events.append("before")
|
||||
|
||||
return await processor.process_sdk_job(
|
||||
records,
|
||||
request_id="request-1",
|
||||
before_hy_invocation=before,
|
||||
)
|
||||
|
||||
receipt = asyncio.run(run())
|
||||
|
||||
assert client.events == ["before", "add"]
|
||||
assert len(client.calls) == 1
|
||||
method, messages, kwargs = client.calls[0]
|
||||
assert method == "add"
|
||||
assert messages == [
|
||||
{"role": "user", "content": "question"},
|
||||
{"role": "assistant", "content": "answer"},
|
||||
{"role": "tool", "content": "result"},
|
||||
]
|
||||
assert kwargs["user_id"] == "local"
|
||||
assert kwargs["agent_id"] == PRODUCT_AGENT_ID
|
||||
assert kwargs["session_id"] == "session-1"
|
||||
assert kwargs["source"] == "cursor"
|
||||
assert kwargs["workspace_id"] == "repo-a"
|
||||
assert kwargs["memory_at"] == memory_at + timedelta(seconds=1)
|
||||
assert kwargs["request_id"] == "request-1"
|
||||
assert kwargs["metadata"] == {
|
||||
"source_id": "cursor",
|
||||
"session_id": "session-1",
|
||||
"workspace_ids": ["repo-a"],
|
||||
"memory_at": (memory_at + timedelta(seconds=1)).isoformat(),
|
||||
"evidence_ids": [record.evidence_id for record in records],
|
||||
"input_kind": "raw_conversation",
|
||||
}
|
||||
assert receipt == SdkWriteReceipt(
|
||||
raw_ids=("raw-1",),
|
||||
memory_ids=("memory-1", "experience-1", "fact-1"),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"record",
|
||||
(
|
||||
_sdk_evidence("memory", text="remember this", memory_content=True),
|
||||
_sdk_evidence(
|
||||
"note",
|
||||
text="save this note",
|
||||
source="one-memory-manual",
|
||||
manual_note=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
def test_sdk_write_processor_uses_add_extracted_for_memory_content_and_notes(record) -> None:
|
||||
from one_memory.sdk_write_processor import SdkWriteProcessor, SdkWriteReceipt
|
||||
|
||||
client = _SdkPublicClient()
|
||||
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
|
||||
|
||||
async def run() -> SdkWriteReceipt:
|
||||
async def before() -> None:
|
||||
client.events.append("before")
|
||||
|
||||
return await processor.process_sdk_job(
|
||||
(record,),
|
||||
request_id=record.evidence_id,
|
||||
before_hy_invocation=before,
|
||||
)
|
||||
|
||||
assert asyncio.run(run()) == SdkWriteReceipt(memory_ids=("memory-note",))
|
||||
assert client.events == ["before", "add_extracted"]
|
||||
assert client.calls[0][1] == [record.verbatim]
|
||||
assert len(client.calls) == 1
|
||||
|
||||
|
||||
def test_sdk_unknown_outcome_inquiry_uses_only_public_receipt() -> None:
|
||||
from one_memory.sdk_write_processor import SdkWriteProcessor, SdkWriteReceipt
|
||||
|
||||
client = _SdkPublicClient()
|
||||
processor = SdkWriteProcessor(_SdkPublicHandle(client)) # type: ignore[arg-type]
|
||||
client.outcome = {"status": "unknown", "result": {"success": False}}
|
||||
assert asyncio.run(processor.inquire_write_outcome("request-1")) is None
|
||||
|
||||
client.outcome = {
|
||||
"status": "succeeded",
|
||||
"result": {
|
||||
"success": True,
|
||||
"raw_ids": ["raw-1"],
|
||||
"fact_ids": ["fact-1"],
|
||||
"variant": "working",
|
||||
},
|
||||
}
|
||||
assert asyncio.run(processor.inquire_write_outcome("request-1")) == SdkWriteReceipt(
|
||||
raw_ids=("raw-1",),
|
||||
memory_ids=("fact-1",),
|
||||
)
|
||||
assert [call[0] for call in client.calls] == [
|
||||
"get_write_outcome",
|
||||
"get_write_outcome",
|
||||
]
|
||||
|
||||
|
||||
def test_missing_evidence_is_refused_without_calling_the_model(tmp_path) -> None:
|
||||
@@ -1978,7 +2173,7 @@ def test_execute_job_crash_after_marking_hy_invoked_becomes_sticky_unknown(tmp_p
|
||||
drain.claim(limit=1)
|
||||
|
||||
class _CancelledBeforeReturn:
|
||||
async def process_records(self, _records, **_kwargs):
|
||||
async def process_sdk_job(self, _records, **_kwargs):
|
||||
raise asyncio.CancelledError
|
||||
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
@@ -4978,7 +5173,7 @@ def test_a_refused_source_context_is_nonblocking_unknown(
|
||||
record.job_id,
|
||||
queue=queue,
|
||||
supervisor=_Recorder(
|
||||
error=HyMemoryBridgeError("hy_memory_write_verification_context_scope"),
|
||||
error=RuntimeError("hy_memory_write_verification_context_scope"),
|
||||
invoke_hy=True,
|
||||
),
|
||||
evidence=_EvidenceStore({"e1"}),
|
||||
@@ -5006,7 +5201,7 @@ def test_a_failure_after_the_call_still_parks_as_unknown(tmp_path) -> None:
|
||||
claimer = DurableWriteDrain(queue, execute=_never_execute, owner="after-call-owner")
|
||||
assert claimer.claim(limit=1) == (record.job_id,)
|
||||
recorder = _Recorder(
|
||||
error=HyMemoryBridgeError("hy_memory_write_verification_authority_invalid"),
|
||||
error=RuntimeError("hy_memory_write_verification_authority_invalid"),
|
||||
invoke_hy=True,
|
||||
)
|
||||
|
||||
|
||||
@@ -509,10 +509,10 @@ def test_the_api_shows_the_chain_refuses_edits_and_still_deletes_through_hy(tmp_
|
||||
|
||||
cascaded: list[str] = []
|
||||
|
||||
async def delete_hy_memory(memory_id: str) -> None:
|
||||
async def finish_deletion(memory_id: str) -> None:
|
||||
cascaded.append(memory_id)
|
||||
|
||||
app.state.memory_processing_supervisor.delete_hy_memory = delete_hy_memory
|
||||
app.state.sdk_memory_runtime.finish_deletion = finish_deletion
|
||||
deleted = await client.post(
|
||||
"/api/v1/memories/old/actions/delete",
|
||||
json={"request_id": "delete-x", "expected_revision": replaced.revision},
|
||||
|
||||
@@ -861,7 +861,7 @@ def test_clearing_drops_payloads_of_a_failed_task_at_the_retry_ceiling(tmp_path)
|
||||
def test_clearing_never_touches_completed_tasks(tmp_path) -> None:
|
||||
"""Completed inputs are evidence of what HY was actually given.
|
||||
|
||||
`memory_processing` compares them against the records sent to HY to prove
|
||||
SDK write processing compares them against the records sent to HY to prove
|
||||
coverage, and `matters/discovery` reads them to report how many records HY
|
||||
saw. Clearing them would make a failed coverage check look like a pass.
|
||||
"""
|
||||
|
||||
@@ -1,263 +1,28 @@
|
||||
"""R-09:首次安装引导把画像录下来,不问用户一句话。
|
||||
|
||||
引导导完来源就结束了,从不归纳一次画像。`digest()` 全仓唯一的调用方是
|
||||
记忆空间里那个按钮(`MemorySpace.tsx:265`),也就是说首次安装走完,
|
||||
System 2 归纳出的画像一条都不在治理层里,新装的库对「我是谁」一无所知,
|
||||
而证据其实已经导进来了。
|
||||
|
||||
归纳本身还漏了一半:`_record_schemas` 只挑 `L6_SCHEMA` 节点,
|
||||
`L0_BASIC_INFO` 直接跳过。`_read_graph_snapshot` 的查询不带 layer 过滤,
|
||||
L0 节点本来就在快照里,取回来了又丢掉。HY 自己把这两层并称画像
|
||||
(`_PROFILE_LAYERS = {"l0_basic_info", "l6_schema"}`),
|
||||
basic info 那半边留在 Kuzu 里,本地看不见。
|
||||
"""
|
||||
"""Onboarding endpoints manage UI state without touching SDK memory."""
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_memory_bridge import HyGraphNode, HyGraphSnapshot, HyMemoryBridgeError
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import (
|
||||
MemoryDraft,
|
||||
MemoryGovernanceStore,
|
||||
MemoryKind,
|
||||
MemoryStatus,
|
||||
)
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
from tests.person_door import as_person
|
||||
|
||||
|
||||
class FakeGraph:
|
||||
"""归纳只用得上四个方法,其余不实现,少一个假实现少一处失真。"""
|
||||
|
||||
def __init__(self, graph_nodes: tuple[HyGraphNode, ...]) -> None:
|
||||
self.graph_nodes = graph_nodes
|
||||
self.graph_reads: list[tuple[str, str]] = []
|
||||
self.digest_calls: list[tuple[str, str]] = []
|
||||
|
||||
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
|
||||
self.digest_calls.append((user_id, agent_id))
|
||||
return {"success": True, "tasks_processed": 1}
|
||||
|
||||
async def graph_snapshot(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
limit: int = 300,
|
||||
) -> HyGraphSnapshot:
|
||||
self.graph_reads.append((user_id, agent_id))
|
||||
return HyGraphSnapshot(nodes=self.graph_nodes, edges=())
|
||||
|
||||
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
|
||||
return ()
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _setup(tmp_path):
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
secrets = InMemorySecretStore()
|
||||
evidence = EvidenceStore(database)
|
||||
profiles = ModelProfileStore(database)
|
||||
tasks = MemoryTaskStore(database)
|
||||
governance = MemoryGovernanceStore(database)
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Local",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="hy-bot-phi4",
|
||||
embedding_model="nomic-embed-text",
|
||||
def _app(tmp_path: Path):
|
||||
return create_app(
|
||||
Settings(
|
||||
_env_file=None,
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path / "home",
|
||||
api_token=SecretStr("test-token"),
|
||||
imports_enabled=False,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
),
|
||||
secrets,
|
||||
source_catalog={},
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profile = profiles.activate(profile.profile_id, secrets)
|
||||
return evidence, profiles, tasks, governance, secrets, profile
|
||||
|
||||
|
||||
def _imported(evidence: EvidenceStore, *, workspace: str | None = "workspace:v1:project"):
|
||||
return evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
workspace=workspace,
|
||||
conversation_id="conversation",
|
||||
message_id="m1",
|
||||
correlation_id="m1",
|
||||
content_kind=EvidenceContentKind.RAW_CONVERSATION,
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="导入进来的一段对话。",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
|
||||
def _confirmed_from(governance: MemoryGovernanceStore, evidence_id: str) -> None:
|
||||
"""归纳只对「还有本地证据的 scope」动手,所以得先有一条确认记忆。"""
|
||||
as_person(governance).create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content="项目里的一条既有记忆。",
|
||||
scopes=("workspace:v1:project",),
|
||||
evidence_ids=(evidence_id,),
|
||||
),
|
||||
request_id="candidate-1",
|
||||
)
|
||||
|
||||
|
||||
def _supervisor(tmp_path, stores, bridge) -> MemoryProcessingSupervisor:
|
||||
evidence, profiles, tasks, governance, secrets, profile = stores
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=tasks,
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=governance,
|
||||
data_dir=tmp_path,
|
||||
)
|
||||
supervisor._bridges[(profile.profile_id, profile.revision, None)] = bridge
|
||||
return supervisor
|
||||
|
||||
|
||||
def test_the_basic_info_half_of_the_profile_is_written_down_too(tmp_path) -> None:
|
||||
"""L0_BASIC_INFO 取回来了又被丢掉,画像只进了一半。
|
||||
|
||||
HY 自己把 L0_basic_info 与 L6_schema 并称画像,快照里两层都在,
|
||||
而 `_record_schemas` 只留 L6。零提问录画像要的正是 L0 这半边:
|
||||
「用户是一名产品经理」这种事实,只会出现在 basic info 层。
|
||||
"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
|
||||
record = _imported(evidence)
|
||||
_confirmed_from(governance, record.evidence_id)
|
||||
bridge = FakeGraph(
|
||||
graph_nodes=(
|
||||
HyGraphNode(
|
||||
node_id="basic-1",
|
||||
content="用户是一名非技术背景的产品经理。",
|
||||
layer="l0_basic_info",
|
||||
),
|
||||
HyGraphNode(
|
||||
node_id="schema-1",
|
||||
content="用户偏好可验证的结论。",
|
||||
layer="l6_schema",
|
||||
),
|
||||
),
|
||||
)
|
||||
supervisor = _supervisor(tmp_path, stores, bridge)
|
||||
|
||||
result = asyncio.run(supervisor.digest())
|
||||
|
||||
profile_contents = {
|
||||
memory.content: memory
|
||||
for memory in governance.list(user_id="local")
|
||||
if memory.scopes == ("all-apps",)
|
||||
}
|
||||
assert "用户是一名非技术背景的产品经理。" in profile_contents
|
||||
assert "用户偏好可验证的结论。" in profile_contents
|
||||
assert result["schemas_recorded"] == 2
|
||||
basic = profile_contents["用户是一名非技术背景的产品经理。"]
|
||||
assert basic.status is MemoryStatus.CONFIRMED
|
||||
# 说得出为什么相信它,和 L6 那半边同一个标准。
|
||||
assert basic.evidence_ids == (record.evidence_id,)
|
||||
# kind 用仓库既有的 layer→kind 映射,不为画像新造一个种类
|
||||
# (种类是 API 契约的一部分,新增要人放行)。
|
||||
assert basic.kind is MemoryKind.CHAT
|
||||
|
||||
|
||||
def test_a_layer_hy_does_not_call_profile_still_never_reaches_governance(tmp_path) -> None:
|
||||
"""绿对照,也是这一项的防假修复守卫。
|
||||
|
||||
把「只挑 L6」改成「快照里的都带回来」能让上面那条变绿,代价是
|
||||
整段原始对话(L1_RAW)被当成画像广播给每个 app。这条盯着它。
|
||||
"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
|
||||
record = _imported(evidence)
|
||||
_confirmed_from(governance, record.evidence_id)
|
||||
bridge = FakeGraph(
|
||||
graph_nodes=tuple(
|
||||
HyGraphNode(node_id=f"node-{layer}", content=f"{layer} 的内容。", layer=layer)
|
||||
for layer in ("l1_raw", "l2_fact", "l3_summary", "l4_identity", "l7_intention")
|
||||
),
|
||||
)
|
||||
supervisor = _supervisor(tmp_path, stores, bridge)
|
||||
|
||||
result = asyncio.run(supervisor.digest())
|
||||
|
||||
assert result["schemas_recorded"] == 0
|
||||
assert [memory.content for memory in governance.list(user_id="local")] == [
|
||||
"项目里的一条既有记忆。"
|
||||
]
|
||||
|
||||
|
||||
def test_the_same_basic_info_is_not_written_twice(tmp_path) -> None:
|
||||
"""绿对照:归纳会被反复触发(引导一次、按钮随时),不能越攒越多。"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, _tasks, governance, _secrets, _profile = stores
|
||||
record = _imported(evidence)
|
||||
_confirmed_from(governance, record.evidence_id)
|
||||
nodes = (
|
||||
HyGraphNode(
|
||||
node_id="basic-1",
|
||||
content="用户是一名非技术背景的产品经理。",
|
||||
layer="l0_basic_info",
|
||||
),
|
||||
)
|
||||
supervisor = _supervisor(tmp_path, stores, FakeGraph(graph_nodes=nodes))
|
||||
|
||||
first = asyncio.run(supervisor.digest())
|
||||
second = asyncio.run(supervisor.digest())
|
||||
|
||||
assert first["schemas_recorded"] == 1
|
||||
assert second["schemas_recorded"] == 0
|
||||
assert [
|
||||
memory.content
|
||||
for memory in governance.list(user_id="local")
|
||||
if memory.scopes == ("all-apps",)
|
||||
] == ["用户是一名非技术背景的产品经理。"]
|
||||
|
||||
|
||||
def _app(tmp_path):
|
||||
settings = Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path / "home",
|
||||
api_token=SecretStr("test-token"),
|
||||
)
|
||||
return create_app(settings, source_catalog={})
|
||||
|
||||
|
||||
def _client(app) -> AsyncClient:
|
||||
@@ -271,83 +36,47 @@ def _client(app) -> AsyncClient:
|
||||
)
|
||||
|
||||
|
||||
def test_finishing_onboarding_is_not_model_content_consent(tmp_path) -> None:
|
||||
"""S1.3: completion and skip cannot authorize profile induction."""
|
||||
|
||||
def test_finishing_onboarding_is_not_model_content_consent(tmp_path: Path) -> None:
|
||||
async def scenario() -> None:
|
||||
app = _app(tmp_path)
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
calls: list[str] = []
|
||||
|
||||
async def spy() -> dict[str, object]:
|
||||
async def digest() -> dict[str, object]:
|
||||
calls.append("digest")
|
||||
return {"success": True, "schemas_recorded": 0}
|
||||
|
||||
app.state.memory_processing_supervisor.digest = spy # type: ignore[method-assign]
|
||||
|
||||
# 只是看一眼引导状态,不该触发归纳。
|
||||
assert (await client.get("/api/v1/onboarding")).status_code == 200
|
||||
assert calls == []
|
||||
return {"success": True}
|
||||
|
||||
app.state.sdk_memory_runtime.digest = digest
|
||||
finished = await client.post("/api/v1/onboarding/complete")
|
||||
|
||||
assert finished.status_code == 200
|
||||
assert finished.json()["status"] == "completed"
|
||||
assert calls == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_way_back_to_the_first_run_deletes_nothing(tmp_path) -> None:
|
||||
"""重置只动完成标记,记忆一条不少。
|
||||
|
||||
完成态和记忆放在同一个库里是有意的:重装一遍 App 不该把一个已经有记忆的人
|
||||
当成新用户。代价是想再看一次冷启动,重装不管用。这条盯着那个出口:
|
||||
状态回到首次运行,别的都不动。
|
||||
"""
|
||||
|
||||
def test_reset_returns_to_first_run_without_touching_sdk_data(tmp_path: Path) -> None:
|
||||
async def scenario() -> None:
|
||||
app = _app(tmp_path)
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
database = tmp_path / "data" / "one-memory.sqlite3"
|
||||
with sqlite3.connect(database) as connection:
|
||||
connection.execute(
|
||||
"INSERT INTO governed_memories "
|
||||
"(memory_id,user_id,kind,status,content,scopes_json,"
|
||||
"revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
"keep-me",
|
||||
"local",
|
||||
"chat",
|
||||
"confirmed",
|
||||
"记忆必须保留",
|
||||
'["all-apps"]',
|
||||
1,
|
||||
"2026-10-17T00:00:00+00:00",
|
||||
"2026-10-17T00:00:00+00:00",
|
||||
),
|
||||
)
|
||||
finished = await client.post("/api/v1/onboarding/complete")
|
||||
|
||||
assert finished.json()["status"] == "completed"
|
||||
sdk_marker = tmp_path / "data" / "hy-memory" / "governed-v2" / "memory.marker"
|
||||
sdk_marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
sdk_marker.write_text("keep-sdk-memory", encoding="utf-8")
|
||||
|
||||
assert (await client.post("/api/v1/onboarding/complete")).status_code == 200
|
||||
reset = await client.post("/api/v1/onboarding/reset")
|
||||
|
||||
assert reset.status_code == 200
|
||||
assert reset.json()["status"] == "not_started"
|
||||
assert reset.json()["completed_at"] is None
|
||||
# 落盘了,不是只在响应里改一下:下一次启动读到的也是首次运行。
|
||||
assert (await client.get("/api/v1/onboarding")).json()["status"] == "not_started"
|
||||
with sqlite3.connect(database) as connection:
|
||||
assert connection.execute(
|
||||
"SELECT content,revision FROM governed_memories WHERE memory_id='keep-me'"
|
||||
).fetchone() == ("记忆必须保留", 1)
|
||||
assert sdk_marker.read_text(encoding="utf-8") == "keep-sdk-memory"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_resetting_before_anything_was_completed_is_not_an_error(tmp_path) -> None:
|
||||
"""绿对照:新库里还没有那一行,重置不能炸。"""
|
||||
|
||||
def test_resetting_before_completion_is_not_an_error(tmp_path: Path) -> None:
|
||||
async def scenario() -> None:
|
||||
app = _app(tmp_path)
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
@@ -359,28 +88,17 @@ def test_resetting_before_anything_was_completed_is_not_an_error(tmp_path) -> No
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_hy_being_down_does_not_leave_the_first_install_unfinished(tmp_path) -> None:
|
||||
"""守卫:这一步可跳过,归纳失败不能把引导卡住。
|
||||
|
||||
首次安装时模型档案可能还没配好,通用出站网络也可能不可用。
|
||||
归纳失败只该少一条画像,不该让人装不完。
|
||||
"""
|
||||
|
||||
def test_sdk_unavailability_cannot_block_onboarding_completion(tmp_path: Path) -> None:
|
||||
async def scenario() -> None:
|
||||
app = _app(tmp_path)
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
|
||||
async def unavailable() -> dict[str, object]:
|
||||
raise HyMemoryBridgeError("hy_memory_unavailable")
|
||||
|
||||
app.state.memory_processing_supervisor.digest = unavailable # type: ignore[method-assign]
|
||||
raise RuntimeError("hy_sdk_unavailable")
|
||||
|
||||
app.state.sdk_memory_runtime.digest = unavailable
|
||||
finished = await client.post("/api/v1/onboarding/complete")
|
||||
|
||||
assert finished.status_code == 200
|
||||
assert finished.json()["status"] == "completed"
|
||||
assert finished.json()["completed_at"] is not None
|
||||
# 状态是真的落了盘,不是只在这次响应里。
|
||||
assert (await client.get("/api/v1/onboarding")).json()["status"] == "completed"
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -3,7 +3,7 @@ import hashlib
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HySdkMemoryRecord
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryDraft, MemoryGovernanceStore, MemoryKind
|
||||
|
||||
@@ -127,9 +127,17 @@ def test_both_lanes_answer_through_one_real_hy_instance(tmp_path, monkeypatch):
|
||||
linked.raise_for_status()
|
||||
job = await app.state.import_supervisor.wait(linked.json()["jobs"][0]["job_id"])
|
||||
assert job.stored == 1 and job.errors == 0, job
|
||||
processor = app.state.memory_processing_supervisor
|
||||
await asyncio.gather(*tuple(processor._debounces.values()), return_exceptions=True)
|
||||
await asyncio.wait_for(asyncio.gather(*tuple(processor._workers.values())), timeout=300)
|
||||
deadline = asyncio.get_running_loop().time() + 300
|
||||
while True:
|
||||
counts = app.state.sdk_memory_runtime.status_counts()
|
||||
if not any(
|
||||
counts.get(state, 0)
|
||||
for state in ("pending", "running", "waiting_for_model")
|
||||
):
|
||||
break
|
||||
if asyncio.get_running_loop().time() >= deadline:
|
||||
raise AssertionError(f"SDK write queue did not drain: {counts}")
|
||||
await asyncio.sleep(0.05)
|
||||
|
||||
# Imported content is not shared with every app by default. Sharing
|
||||
# it here is what the memory lane is later asked to find.
|
||||
|
||||
@@ -216,13 +216,13 @@ def test_retrieval_stop_failure_still_drains_other_owned_services(tmp_path, monk
|
||||
)
|
||||
with pytest.raises(failure):
|
||||
async with app.router.lifespan_context(app):
|
||||
original = app.state.memory_processing_supervisor.shutdown
|
||||
original = app.state.sdk_memory_runtime.shutdown
|
||||
|
||||
async def shutdown():
|
||||
await original()
|
||||
events.append("processing_drained")
|
||||
|
||||
monkeypatch.setattr(app.state.memory_processing_supervisor, "shutdown", shutdown)
|
||||
monkeypatch.setattr(app.state.sdk_memory_runtime, "shutdown", shutdown)
|
||||
assert events == ["start", "retrieval_drained", "processing_drained"]
|
||||
assert not (tmp_path / "runtime.json").exists()
|
||||
|
||||
|
||||
@@ -4,9 +4,16 @@ import asyncio
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryStatus
|
||||
|
||||
from one_memory.database_access import close_main_databases, main_database
|
||||
from one_memory.hy_memory_bridge import HySdkMemoryPage, HySdkMemoryRecord, HySdkRawRecord
|
||||
from one_memory.hy_sdk_migration import HySdkMigration
|
||||
from one_memory.hy_sdk_types import (
|
||||
HySdkMemoryPage,
|
||||
HySdkMemoryRecord,
|
||||
HySdkRawPage,
|
||||
HySdkRawRecord,
|
||||
)
|
||||
from one_memory.memory_server.contracts import BatchIdentity, WriteRequest
|
||||
from one_memory.memory_server.durable_queue import DurableWriteQueue
|
||||
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
|
||||
@@ -35,10 +42,15 @@ from one_memory.sdk_migration_inventory import (
|
||||
record_raw_migration_plan,
|
||||
)
|
||||
from one_memory.sdk_migration_runner import (
|
||||
activate_sdk_migration_staging,
|
||||
commit_reconciled_cutover,
|
||||
prepare_legacy_write_queue_for_cutover,
|
||||
prepare_sdk_migration_staging,
|
||||
recover_sdk_migration_swap,
|
||||
resume_legacy_cleanup,
|
||||
run_sdk_data_plane_backfill,
|
||||
sdk_cutover_blockers,
|
||||
sdk_migration_staging,
|
||||
)
|
||||
from one_memory.target_schema import legacy_tables_present
|
||||
|
||||
@@ -210,6 +222,8 @@ def test_startup_cutover_freezes_pending_job_input_and_preserves_the_queue(store
|
||||
sdk_version="2.0.5.dev3",
|
||||
)
|
||||
assert marker.migration_id == "m1"
|
||||
assert legacy_tables_present(store.database_path)
|
||||
resume_legacy_cleanup(data_dir=data_dir, database_path=store.database_path)
|
||||
assert legacy_tables_present(store.database_path) == ()
|
||||
assert store.authority() is MemoryAuthority.SDK
|
||||
preserved = DurableWriteQueue(store.database_path, sdk_authority=True)
|
||||
@@ -410,7 +424,7 @@ def test_inventory_classification_covers_every_legacy_and_sdk_state():
|
||||
assert len(decisions) == 8
|
||||
|
||||
|
||||
class _Bridge:
|
||||
class _MigrationClient:
|
||||
def __init__(self, memories=()):
|
||||
self.memories = tuple(memories)
|
||||
self.imported = []
|
||||
@@ -418,7 +432,7 @@ class _Bridge:
|
||||
self.deleted = []
|
||||
self.raw = ()
|
||||
|
||||
async def list_sdk_memories(self, *, limit, offset, **_kwargs):
|
||||
async def list_memories(self, *, limit, offset, **_kwargs):
|
||||
page = self.memories[offset : offset + limit]
|
||||
return HySdkMemoryPage(
|
||||
memories=page,
|
||||
@@ -427,7 +441,7 @@ class _Bridge:
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
async def import_sdk_memory_exact(self, **kwargs):
|
||||
async def write_legacy_memory(self, **kwargs):
|
||||
self.imported.append(kwargs)
|
||||
return {"success": True, "memory_id": kwargs["memory_id"]}
|
||||
|
||||
@@ -435,11 +449,16 @@ class _Bridge:
|
||||
self.deleted.append(memory_id)
|
||||
return {"success": True, "deleted_count": 1}
|
||||
|
||||
async def list_sdk_raw(self, *, limit, offset, **_kwargs):
|
||||
async def read_raw_inventory(self, *, limit, offset, **_kwargs):
|
||||
rows = self.raw[offset : offset + limit]
|
||||
return rows, len(self.raw)
|
||||
return HySdkRawPage(
|
||||
raw=rows,
|
||||
total=len(self.raw),
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
async def import_sdk_raw_exact(self, **kwargs):
|
||||
async def write_legacy_raw(self, **kwargs):
|
||||
self.imported_raw.append(kwargs)
|
||||
return {"success": True, "raw_id": kwargs["raw_id"]}
|
||||
|
||||
@@ -466,14 +485,14 @@ def _record(memory_id, *, content="body", status="active"):
|
||||
|
||||
|
||||
def test_sdk_inventory_is_paged_without_dropping_lifecycle_records():
|
||||
bridge = _Bridge(
|
||||
client = _MigrationClient(
|
||||
(
|
||||
_record("active"),
|
||||
_record("old", status="superseded"),
|
||||
_record("shadow", status="shadow"),
|
||||
)
|
||||
)
|
||||
inventory = asyncio.run(read_sdk_memory_inventory(bridge, page_size=2))
|
||||
inventory = asyncio.run(read_sdk_memory_inventory(client, page_size=2))
|
||||
assert [(item.memory_id, item.status) for item in inventory] == [
|
||||
("active", "active"),
|
||||
("old", "superseded"),
|
||||
@@ -502,10 +521,10 @@ def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
|
||||
legacy=legacy,
|
||||
sdk=sdk,
|
||||
)
|
||||
bridge = _Bridge()
|
||||
client = _MigrationClient()
|
||||
asyncio.run(
|
||||
execute_memory_migration_plan(
|
||||
bridge,
|
||||
client,
|
||||
store,
|
||||
migration_id="m1",
|
||||
legacy=legacy,
|
||||
@@ -513,8 +532,8 @@ def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
|
||||
)
|
||||
)
|
||||
|
||||
assert [item["memory_id"] for item in bridge.imported] == ["missing"]
|
||||
assert bridge.deleted == ["delete-existing"]
|
||||
assert [item["memory_id"] for item in client.imported] == ["missing"]
|
||||
assert client.deleted == ["delete-existing"]
|
||||
progress = store.progress("m1")
|
||||
assert progress.counts == {"adopted": 1, "deleted": 2, "verified": 3}
|
||||
|
||||
@@ -590,23 +609,23 @@ def test_raw_inventory_matches_exact_scope_imports_missing_and_adopts_sdk_only(s
|
||||
legacy=legacy,
|
||||
sdk=sdk,
|
||||
)
|
||||
bridge = _Bridge()
|
||||
client = _MigrationClient()
|
||||
asyncio.run(
|
||||
execute_raw_migration_plan(
|
||||
bridge,
|
||||
client,
|
||||
store,
|
||||
migration_id="m1",
|
||||
legacy=legacy,
|
||||
decisions=recorded,
|
||||
)
|
||||
)
|
||||
assert [item["raw_id"] for item in bridge.imported_raw] == ["evidence:e2"]
|
||||
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e2"]
|
||||
assert store.progress("m1").counts == {"adopted": 1, "verified": 2}
|
||||
|
||||
|
||||
def test_sdk_raw_inventory_pages_all_records():
|
||||
bridge = _Bridge()
|
||||
bridge.raw = (
|
||||
client = _MigrationClient()
|
||||
client.raw = (
|
||||
HySdkRawRecord(
|
||||
raw_id="r1",
|
||||
content="one",
|
||||
@@ -624,7 +643,7 @@ def test_sdk_raw_inventory_pages_all_records():
|
||||
source="claude",
|
||||
),
|
||||
)
|
||||
inventory = asyncio.run(read_sdk_raw_inventory(bridge, page_size=1))
|
||||
inventory = asyncio.run(read_sdk_raw_inventory(client, page_size=1))
|
||||
assert [item.raw_id for item in inventory] == ["r1", "r2"]
|
||||
|
||||
|
||||
@@ -656,11 +675,11 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
|
||||
""",
|
||||
(content_hash("verbatim"),),
|
||||
)
|
||||
bridge = _Bridge()
|
||||
client = _MigrationClient()
|
||||
result = asyncio.run(
|
||||
run_sdk_data_plane_backfill(
|
||||
database_path=database,
|
||||
bridge=bridge,
|
||||
client=client,
|
||||
migration_id="migration",
|
||||
bundle_id="bundle",
|
||||
source_runtime_build="v1.0.4",
|
||||
@@ -674,15 +693,15 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
|
||||
assert result.state == "ready"
|
||||
assert result.phase == "reconciled"
|
||||
assert result.counts == {"verified": 2}
|
||||
assert [item["memory_id"] for item in bridge.imported] == ["memory-1"]
|
||||
assert [item["raw_id"] for item in bridge.imported_raw] == ["evidence:e1"]
|
||||
assert [item["memory_id"] for item in client.imported] == ["memory-1"]
|
||||
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e1"]
|
||||
assert store.authority() is MemoryAuthority.LEGACY
|
||||
# Simulate a process dying after both SDK side effects but before startup
|
||||
# could finish. The next inventory observes them as matched rather than
|
||||
# import_exact; that reclassification must resume the settled ledger
|
||||
# instead of conflicting with its original operation key.
|
||||
bridge.memories = (_record("memory-1", content="preference"),)
|
||||
bridge.raw = (
|
||||
client.memories = (_record("memory-1", content="preference"),)
|
||||
client.raw = (
|
||||
HySdkRawRecord(
|
||||
raw_id="evidence:e1",
|
||||
content="verbatim",
|
||||
@@ -696,7 +715,7 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
|
||||
resumed = asyncio.run(
|
||||
run_sdk_data_plane_backfill(
|
||||
database_path=database,
|
||||
bridge=bridge,
|
||||
client=client,
|
||||
migration_id="migration",
|
||||
bundle_id="bundle",
|
||||
source_runtime_build="v1.0.4",
|
||||
@@ -708,8 +727,8 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
|
||||
)
|
||||
assert resumed.state == "ready"
|
||||
assert resumed.counts == {"verified": 2}
|
||||
assert len(bridge.imported) == 1
|
||||
assert len(bridge.imported_raw) == 1
|
||||
assert len(client.imported) == 1
|
||||
assert len(client.imported_raw) == 1
|
||||
|
||||
|
||||
def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
|
||||
@@ -729,7 +748,7 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
|
||||
result = asyncio.run(
|
||||
run_sdk_data_plane_backfill(
|
||||
database_path=database,
|
||||
bridge=_Bridge(),
|
||||
client=_MigrationClient(),
|
||||
migration_id="migration",
|
||||
bundle_id="bundle",
|
||||
source_runtime_build="v1.0.4",
|
||||
@@ -744,3 +763,217 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
|
||||
assert result.phase == "review_required"
|
||||
assert result.counts == {"quarantine": 1}
|
||||
assert store.authority() is MemoryAuthority.LEGACY
|
||||
|
||||
|
||||
class _MigrationLoop:
|
||||
def run(self, coroutine):
|
||||
return asyncio.run(coroutine)
|
||||
|
||||
|
||||
class _MigrationVectorStore:
|
||||
def __init__(self):
|
||||
self.nodes = {}
|
||||
|
||||
async def get_by_id(self, memory_id):
|
||||
return self.nodes.get(memory_id)
|
||||
|
||||
async def upsert(self, node):
|
||||
self.nodes[node.node_id] = node
|
||||
return node.node_id
|
||||
|
||||
async def list_by_user(self, *, user_id, layers, **_kwargs):
|
||||
return [
|
||||
node
|
||||
for node in self.nodes.values()
|
||||
if node.user_id == user_id and node.layer in layers
|
||||
]
|
||||
|
||||
|
||||
class _MigrationEmbedder:
|
||||
async def embed(self, _content):
|
||||
return [0.1, 0.2]
|
||||
|
||||
|
||||
class _MigrationGraphStore:
|
||||
def __init__(self):
|
||||
self.nodes = {}
|
||||
|
||||
async def upsert_memory_node(self, node):
|
||||
self.nodes[node.node_id] = node
|
||||
return node.node_id
|
||||
|
||||
|
||||
class _PrivilegedMigrationClient:
|
||||
def __init__(self):
|
||||
self._loop_thread = _MigrationLoop()
|
||||
self._vector_store = _MigrationVectorStore()
|
||||
self._embed_service = _MigrationEmbedder()
|
||||
self._graph_store = _MigrationGraphStore()
|
||||
|
||||
def list_memories(self, **_kwargs):
|
||||
return {"vdb": {"memories": [], "total": 0}, "request_id": "list"}
|
||||
|
||||
def delete(self, _memory_id):
|
||||
return {"success": True, "deleted_count": 1}
|
||||
|
||||
def get_write_outcome(self, _request_id):
|
||||
return None
|
||||
|
||||
|
||||
def test_offline_migration_writes_sdk_stores_without_public_client_import_methods():
|
||||
client = _PrivilegedMigrationClient()
|
||||
migration = HySdkMigration(client)
|
||||
|
||||
raw_result = asyncio.run(
|
||||
migration.write_legacy_raw(
|
||||
content="verbatim",
|
||||
raw_id="raw-1",
|
||||
user_id="local",
|
||||
agent_id="one-memory",
|
||||
session_id="session",
|
||||
source="cursor",
|
||||
memory_at=None,
|
||||
metadata={"role": "user"},
|
||||
request_id="migrate-raw-1",
|
||||
)
|
||||
)
|
||||
memory_result = asyncio.run(
|
||||
migration.write_legacy_memory(
|
||||
content="prefers tea",
|
||||
memory_id="memory-1",
|
||||
user_id="local",
|
||||
agent_id="one-memory",
|
||||
session_id="session",
|
||||
source="legacy-governance",
|
||||
layer="l2_fact",
|
||||
status="active",
|
||||
owner="user",
|
||||
tags=("preference",),
|
||||
supersedes=(),
|
||||
superseded_by=(),
|
||||
is_latest=True,
|
||||
memory_at=None,
|
||||
metadata={},
|
||||
request_id="migrate-memory-1",
|
||||
)
|
||||
)
|
||||
raw_page = asyncio.run(migration.read_raw_inventory(user_id="local"))
|
||||
|
||||
assert raw_result["raw_id"] == "raw-1"
|
||||
assert memory_result["memory_id"] == "memory-1"
|
||||
assert [item.raw_id for item in raw_page.raw] == ["raw-1"]
|
||||
assert client._vector_store.nodes["raw-1"].layer is MemoryLayer.L1_RAW
|
||||
assert client._vector_store.nodes["memory-1"].status is MemoryStatus.ACTIVE
|
||||
assert client._graph_store.nodes["memory-1"].content == "prefers tea"
|
||||
assert not hasattr(client, "import_raw")
|
||||
assert not hasattr(client, "import_memory_exact")
|
||||
|
||||
|
||||
def test_raw_http_dto_uses_the_unified_memory_list_shape():
|
||||
page = HySdkRawPage.from_memory_list(
|
||||
{
|
||||
"vdb": {
|
||||
"memories": [
|
||||
{
|
||||
"memory_id": "raw-1",
|
||||
"content": "verbatim",
|
||||
"layer": "l1_raw",
|
||||
"status": "active",
|
||||
"user_id": "local",
|
||||
"agent_id": "one-memory",
|
||||
"session_id": "session",
|
||||
"source": "cursor",
|
||||
"custom": {"role": "user"},
|
||||
}
|
||||
],
|
||||
"total": 1,
|
||||
"limit": 100,
|
||||
"offset": 0,
|
||||
}
|
||||
},
|
||||
limit=100,
|
||||
offset=0,
|
||||
)
|
||||
assert page.raw[0].raw_id == "raw-1"
|
||||
assert page.raw[0].custom == {"role": "user"}
|
||||
|
||||
|
||||
def _staging_fixture(tmp_path, store):
|
||||
data_dir = tmp_path / "home"
|
||||
live = data_dir / "hy-memory" / "governed-v2"
|
||||
live.mkdir(parents=True)
|
||||
(live / "sdk-only.txt").write_text("old", encoding="utf-8")
|
||||
paths = sdk_migration_staging(
|
||||
data_dir=data_dir,
|
||||
migration_id="migration-1",
|
||||
namespace="governed-v2",
|
||||
)
|
||||
prepare_sdk_migration_staging(
|
||||
paths=paths,
|
||||
database_path=store.database_path,
|
||||
)
|
||||
return paths
|
||||
|
||||
|
||||
def test_staging_preserves_sdk_only_data_and_swaps_after_commit(tmp_path, store):
|
||||
paths = _staging_fixture(tmp_path, store)
|
||||
staged = paths.staging_root / "sdk-only.txt"
|
||||
assert staged.read_text(encoding="utf-8") == "old"
|
||||
staged.write_text("migrated", encoding="utf-8")
|
||||
marker = object()
|
||||
|
||||
result = activate_sdk_migration_staging(
|
||||
paths=paths,
|
||||
database_path=store.database_path,
|
||||
commit=lambda: marker,
|
||||
)
|
||||
|
||||
assert result is marker
|
||||
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
|
||||
assert not paths.protective_root.exists()
|
||||
assert not paths.staging_home.exists()
|
||||
|
||||
|
||||
def test_incomplete_staging_retry_restarts_from_unchanged_live_root(tmp_path, store):
|
||||
paths = _staging_fixture(tmp_path, store)
|
||||
(paths.staging_root / "sdk-only.txt").write_text("partial", encoding="utf-8")
|
||||
|
||||
prepare_sdk_migration_staging(
|
||||
paths=paths,
|
||||
database_path=store.database_path,
|
||||
)
|
||||
|
||||
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
|
||||
|
||||
|
||||
def test_failed_cutover_restores_live_store_and_keeps_staging(tmp_path, store):
|
||||
paths = _staging_fixture(tmp_path, store)
|
||||
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
|
||||
|
||||
def fail():
|
||||
raise DataPlaneMigrationError("cutover_failed")
|
||||
|
||||
with pytest.raises(DataPlaneMigrationError, match="cutover_failed"):
|
||||
activate_sdk_migration_staging(
|
||||
paths=paths,
|
||||
database_path=store.database_path,
|
||||
commit=fail,
|
||||
)
|
||||
|
||||
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
|
||||
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
|
||||
|
||||
|
||||
def test_restart_recovers_swap_when_authority_was_not_committed(tmp_path, store):
|
||||
paths = _staging_fixture(tmp_path, store)
|
||||
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
|
||||
paths.live_root.replace(paths.protective_root)
|
||||
paths.staging_root.replace(paths.live_root)
|
||||
|
||||
recover_sdk_migration_swap(
|
||||
paths=paths,
|
||||
database_path=store.database_path,
|
||||
)
|
||||
|
||||
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
|
||||
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
|
||||
@@ -1,11 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HySdkMemoryRecord
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord
|
||||
from one_memory.memory_qa import MemoryAnswerError
|
||||
from one_memory.sdk_memory_qa import SdkMemoryAnswerer
|
||||
|
||||
@@ -24,18 +23,29 @@ def record(content: str = "用户偏好本地优先。") -> HySdkMemoryRecord:
|
||||
)
|
||||
|
||||
|
||||
class Bridge:
|
||||
class Client:
|
||||
def __init__(self) -> None:
|
||||
self.current = record()
|
||||
self.searches = 0
|
||||
|
||||
async def search_memories(self, *_args, **_kwargs):
|
||||
def search(self, *_args, **_kwargs):
|
||||
self.searches += 1
|
||||
return (SimpleNamespace(memory_id="memory-1"),)
|
||||
return {
|
||||
"scene": "normal",
|
||||
"memories": [{"memory_id": "memory-1", "score": 0.9}],
|
||||
}
|
||||
|
||||
async def get_sdk_memory(self, memory_id):
|
||||
def get(self, memory_id):
|
||||
assert memory_id == "memory-1"
|
||||
return self.current
|
||||
return self.current.model_dump()
|
||||
|
||||
|
||||
class Handle:
|
||||
def __init__(self, client: Client) -> None:
|
||||
self.value = client
|
||||
|
||||
async def client(self):
|
||||
return self.value
|
||||
|
||||
|
||||
class Profiles:
|
||||
@@ -45,10 +55,10 @@ class Profiles:
|
||||
|
||||
|
||||
def test_sdk_ask_uses_native_sdk_memory_without_governance_fields() -> None:
|
||||
bridge = Bridge()
|
||||
client = Client()
|
||||
answer = asyncio.run(
|
||||
SdkMemoryAnswerer(
|
||||
bridge=bridge, # type: ignore[arg-type]
|
||||
client_handle=Handle(client),
|
||||
profiles=Profiles(), # type: ignore[arg-type]
|
||||
secrets=object(), # type: ignore[arg-type]
|
||||
).answer("偏好是什么?")
|
||||
@@ -65,18 +75,18 @@ def test_sdk_ask_uses_native_sdk_memory_without_governance_fields() -> None:
|
||||
|
||||
|
||||
def test_sdk_ask_revalidates_memory_before_returning_extract() -> None:
|
||||
bridge = Bridge()
|
||||
client = Client()
|
||||
|
||||
class ChangingProfiles:
|
||||
@staticmethod
|
||||
def active(_secrets):
|
||||
bridge.current = record("内容已变化")
|
||||
client.current = record("内容已变化")
|
||||
return None
|
||||
|
||||
with pytest.raises(MemoryAnswerError, match="memory_answer_context_changed"):
|
||||
asyncio.run(
|
||||
SdkMemoryAnswerer(
|
||||
bridge=bridge, # type: ignore[arg-type]
|
||||
client_handle=Handle(client),
|
||||
profiles=ChangingProfiles(), # type: ignore[arg-type]
|
||||
secrets=object(), # type: ignore[arg-type]
|
||||
).answer("偏好是什么?")
|
||||
@@ -84,14 +94,14 @@ def test_sdk_ask_revalidates_memory_before_returning_extract() -> None:
|
||||
|
||||
|
||||
def test_sdk_ask_propagates_sdk_search_failure_without_legacy_fallback() -> None:
|
||||
class Failed(Bridge):
|
||||
async def search_memories(self, *_args, **_kwargs):
|
||||
class Failed(Client):
|
||||
def search(self, *_args, **_kwargs):
|
||||
raise RuntimeError("sdk unavailable")
|
||||
|
||||
with pytest.raises(RuntimeError, match="sdk unavailable"):
|
||||
asyncio.run(
|
||||
SdkMemoryAnswerer(
|
||||
bridge=Failed(), # type: ignore[arg-type]
|
||||
client_handle=Handle(Failed()),
|
||||
profiles=Profiles(), # type: ignore[arg-type]
|
||||
secrets=object(), # type: ignore[arg-type]
|
||||
).answer("偏好是什么?")
|
||||
|
||||
@@ -1,190 +1,97 @@
|
||||
"""What `semantic_search` should do once the profile that filled the store is gone.
|
||||
|
||||
Deactivating or deleting a model profile does not delete the memories that
|
||||
profile produced, but `semantic_search` gave up as soon as it could not find
|
||||
an active one. The retrieval it abandoned needs no generation model at all:
|
||||
the built-in ONNX encoder is what wrote those vectors in the first place.
|
||||
"""
|
||||
"""Model-free SDK search remains available without an active write profile."""
|
||||
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HY_MEMORY_NAMESPACE,
|
||||
GraphWidthMismatch,
|
||||
GraphWidthMismatchError,
|
||||
from one_memory.hy_sdk_runtime import (
|
||||
SdkClientHandle,
|
||||
builtin_retrieval_snapshot,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
from one_memory.model_profiles import ProviderKind
|
||||
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
|
||||
|
||||
|
||||
class RecordingBridge:
|
||||
def __init__(self, snapshot, hits: tuple[str, ...] = ()) -> None:
|
||||
self.snapshot = snapshot
|
||||
self.hits = hits
|
||||
class _SearchClient:
|
||||
def __init__(self, reply: dict[str, object]) -> None:
|
||||
self.reply = reply
|
||||
self.calls: list[tuple[str, dict[str, object]]] = []
|
||||
|
||||
async def search_scores(self, query: str, **_: object) -> tuple[object, ...]:
|
||||
del query
|
||||
return tuple((memory_id, 0.9) for memory_id in self.hits)
|
||||
def search(self, query: str, **options: object) -> dict[str, object]:
|
||||
self.calls.append((query, options))
|
||||
return self.reply
|
||||
|
||||
|
||||
def _supervisor(tmp_path, *, hits: tuple[str, ...] = (), built: list | None = None):
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
secrets = InMemorySecretStore()
|
||||
profiles = ModelProfileStore(database)
|
||||
class _Handle:
|
||||
def __init__(self, client: _SearchClient) -> None:
|
||||
self._client = client
|
||||
|
||||
def factory(snapshot, _secret, data_dir):
|
||||
# Building the real client is what creates the store on disk, so the
|
||||
# stand-in has to create it too: without that, "no store was built"
|
||||
# is a sentence no assertion could ever falsify.
|
||||
(data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / "graph").mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
bridge = RecordingBridge(snapshot, hits)
|
||||
if built is not None:
|
||||
built.append(bridge)
|
||||
return bridge
|
||||
async def client(self) -> _SearchClient:
|
||||
return self._client
|
||||
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=MemoryTaskStore(database),
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=EvidenceStore(database),
|
||||
governance=MemoryGovernanceStore(database),
|
||||
data_dir=tmp_path,
|
||||
bridge_factory=factory,
|
||||
|
||||
def _runtime(client: _SearchClient) -> SdkMemoryRuntime:
|
||||
return SdkMemoryRuntime(
|
||||
coordinator=SimpleNamespace(),
|
||||
read_handle=_Handle(client), # type: ignore[arg-type]
|
||||
write_queue=SimpleNamespace(), # type: ignore[arg-type]
|
||||
extraction_settings=SimpleNamespace(), # type: ignore[arg-type]
|
||||
data_dir=Path("/unused"),
|
||||
)
|
||||
return supervisor, profiles, secrets
|
||||
|
||||
|
||||
def _plant_store(tmp_path: Path) -> None:
|
||||
"""Give the data directory the graph an earlier extraction would have left."""
|
||||
graph = tmp_path / "hy-memory" / HY_MEMORY_NAMESPACE / "graph"
|
||||
graph.mkdir(parents=True)
|
||||
(graph / "catalog.kz").write_bytes(b"")
|
||||
def test_builtin_retrieval_handle_needs_no_profile_or_key() -> None:
|
||||
profile = builtin_retrieval_snapshot()
|
||||
|
||||
|
||||
def _activate(profiles, secrets):
|
||||
created = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Local",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="hy-bot-phi4",
|
||||
embedding_model="nomic-embed-text",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
created.profile_id,
|
||||
expected_revision=created.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
return profiles.activate(created.profile_id, secrets)
|
||||
assert profile.profile_id == "builtin-retrieval"
|
||||
assert profile.provider_kind is ProviderKind.LOCAL_ONNX
|
||||
assert profile.embedding_provider_kind is ProviderKind.LOCAL_ONNX
|
||||
assert profile.has_api_key is False
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_a_library_left_by_a_retired_profile_is_still_searchable(tmp_path) -> None:
|
||||
supervisor, _profiles, _secrets = _supervisor(tmp_path, hits=("memory-1",))
|
||||
_plant_store(tmp_path)
|
||||
|
||||
found = await supervisor.semantic_search("抽取")
|
||||
|
||||
assert found == (("memory-1", 0.9),)
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_no_store_on_disk_is_not_a_reason_to_build_one(tmp_path) -> None:
|
||||
"""Searching must not leave a 512-wide graph behind for an undecided user.
|
||||
|
||||
Building the client is what creates the store, and the width it creates is
|
||||
the built-in encoder's. A user who has not chosen a model yet would then
|
||||
meet a width mismatch on their first 768-wide profile: an accident this
|
||||
search invented, in the same shape as the one that retired a live graph.
|
||||
"""
|
||||
supervisor, _profiles, _secrets = _supervisor(tmp_path, hits=("memory-1",))
|
||||
|
||||
found = await supervisor.semantic_search("抽取")
|
||||
|
||||
assert found == ()
|
||||
assert not (tmp_path / "hy-memory").exists()
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_a_store_written_wider_is_left_to_the_width_prompt(tmp_path) -> None:
|
||||
supervisor, _profiles, _secrets = _supervisor(tmp_path)
|
||||
_plant_store(tmp_path)
|
||||
|
||||
class WiderStore(RecordingBridge):
|
||||
async def search_scores(self, query: str, **_: object) -> tuple[object, ...]:
|
||||
# Where the real one raises: the client is built lazily, on the
|
||||
# first search, and it is building it that reads the stored width.
|
||||
del query
|
||||
raise GraphWidthMismatchError(
|
||||
GraphWidthMismatch(
|
||||
stored_width=768,
|
||||
expected_width=512,
|
||||
graph_path=str(tmp_path / "graph"),
|
||||
)
|
||||
)
|
||||
|
||||
supervisor.bridge_factory = lambda snapshot, _secret, _data_dir: WiderStore(snapshot)
|
||||
|
||||
assert await supervisor.semantic_search("抽取") == ()
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_retrieval_reaches_hy_as_a_local_encoder_needing_no_key(tmp_path) -> None:
|
||||
built: list[RecordingBridge] = []
|
||||
supervisor, _profiles, _secrets = _supervisor(tmp_path, built=built)
|
||||
_plant_store(tmp_path)
|
||||
|
||||
await supervisor.semantic_search("抽取")
|
||||
|
||||
snapshot = built[0].snapshot
|
||||
assert snapshot.embedding_provider_kind is ProviderKind.LOCAL_ONNX
|
||||
assert snapshot.has_api_key is False
|
||||
assert snapshot == builtin_retrieval_snapshot()
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_the_retrieval_bridge_does_not_occupy_a_real_profiles_cache_slot(
|
||||
tmp_path,
|
||||
async def test_sdk_client_handle_lazily_reuses_the_model_free_client(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Otherwise pasting a key would change nothing the user can see.
|
||||
client = _SearchClient({"scene": "normal", "memories": []})
|
||||
lease = SimpleNamespace(client=client, close=lambda: None)
|
||||
acquired: list[object] = []
|
||||
|
||||
The cache is keyed by profile id and revision. Were the retrieval snapshot
|
||||
to collide with a real profile's key, the bridge built with no generation
|
||||
model would be handed back after the user configured one, and extraction
|
||||
would go on doing nothing with no error anywhere to explain it.
|
||||
"""
|
||||
built: list[RecordingBridge] = []
|
||||
supervisor, profiles, secrets = _supervisor(tmp_path, built=built)
|
||||
_plant_store(tmp_path)
|
||||
def acquire(*args: object, **kwargs: object) -> object:
|
||||
acquired.append((args, kwargs))
|
||||
return lease
|
||||
|
||||
await supervisor.semantic_search("抽取")
|
||||
active = _activate(profiles, secrets)
|
||||
await supervisor._prewarm_write_bridges(("cursor",)) # noqa: SLF001
|
||||
await supervisor.semantic_search("抽取")
|
||||
monkeypatch.setattr("one_memory.hy_sdk_runtime.acquire_client", acquire)
|
||||
handle = SdkClientHandle(
|
||||
builtin_retrieval_snapshot(),
|
||||
api_key=None,
|
||||
data_dir=tmp_path,
|
||||
)
|
||||
|
||||
assert len(built) == 3
|
||||
assert len({id(bridge) for bridge in built}) == 3
|
||||
assert built[0].snapshot == builtin_retrieval_snapshot()
|
||||
assert {bridge.snapshot.profile_id for bridge in built[1:]} == {active.profile_id}
|
||||
assert await handle.client() is client
|
||||
assert await handle.client() is client
|
||||
assert len(acquired) == 1
|
||||
await handle.close()
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_search_without_profile_uses_public_sdk_search() -> None:
|
||||
client = _SearchClient(
|
||||
{
|
||||
"scene": "normal",
|
||||
"memories": [
|
||||
{"memory_id": "memory-1", "content": "偏好简短回答", "score": 0.91},
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
found = await _runtime(client).semantic_search("回答偏好")
|
||||
|
||||
assert found == (("memory-1", 0.91),)
|
||||
query, options = client.calls[0]
|
||||
assert query == "回答偏好"
|
||||
assert options["user_ids"] == ["local"]
|
||||
assert options["agent_ids"] == []
|
||||
assert options["scene"] == "normal"
|
||||
@@ -35,6 +35,7 @@ from one_memory.imports.sources.deepseek_harness import (
|
||||
from one_memory.imports.sources.hermes import (
|
||||
HERMES_DESCRIPTOR,
|
||||
create_hermes_adapter,
|
||||
parse_hermes_rollout_record,
|
||||
)
|
||||
from one_memory.imports.sources.openclaw import (
|
||||
OPENCLAW_DESCRIPTOR,
|
||||
@@ -233,6 +234,55 @@ def test_qwen_code_reads_only_the_transcript_beside_its_bookkeeping(
|
||||
assert SECRET not in repr(events)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("role", ["user", "assistant"])
|
||||
def test_hermes_strips_native_provider_recall_context(role: str) -> None:
|
||||
context = SourceRecordContext(
|
||||
file_key="hermes",
|
||||
relative_path="sessions/session.jsonl",
|
||||
ordinal="1",
|
||||
)
|
||||
result = parse_hermes_rollout_record(
|
||||
{
|
||||
"type": "message",
|
||||
"role": role,
|
||||
"content": (
|
||||
"before\n<memhub_memory_context>\n"
|
||||
"Untrusted memory reference:\n"
|
||||
'[{"memory_id":"m1","content":"RECALLED_SECRET"}]\n'
|
||||
"</memhub_memory_context>\nafter"
|
||||
),
|
||||
"sessionId": "session-A",
|
||||
},
|
||||
context,
|
||||
)
|
||||
assert isinstance(result, ParsedMessage)
|
||||
assert result.verbatim == "before\n\nafter"
|
||||
assert "RECALLED_SECRET" not in result.verbatim
|
||||
|
||||
|
||||
def test_hermes_skips_a_persisted_pure_recall_block() -> None:
|
||||
context = SourceRecordContext(
|
||||
file_key="hermes",
|
||||
relative_path="sessions/session.jsonl",
|
||||
ordinal="1",
|
||||
)
|
||||
result = parse_hermes_rollout_record(
|
||||
{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": (
|
||||
"<memhub_memory_context>\n"
|
||||
'[{"memory_id":"m1","content":"RECALLED_SECRET"}]\n'
|
||||
"</memhub_memory_context>"
|
||||
),
|
||||
"sessionId": "session-A",
|
||||
},
|
||||
context,
|
||||
)
|
||||
assert isinstance(result, ParsedSkip)
|
||||
assert result.reason_code == ScanIssueCode.NO_TEXT
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("role", "record_type", "provenance", "written_by"),
|
||||
[
|
||||
|
||||
@@ -1,494 +0,0 @@
|
||||
"""Sedimentation runs several harnesses at once, but one harness at a time.
|
||||
|
||||
Extraction was serial: `max_concurrent_tasks` defaulted to 1 and nothing ever
|
||||
passed anything else, so the queue on the user's machine stood at 1,440 with
|
||||
one task in flight and the scheduler logging `queue_saturated`. Meanwhile the
|
||||
model profile has carried a `max_concurrency` of 4 all along, copied into
|
||||
every task's snapshot and never once consulted by the scheduler.
|
||||
|
||||
Turning the number up on its own would be wrong. HY decides whether a new
|
||||
memory supersedes an older one by what it has already been told, so two
|
||||
extractions from the same harness running at once can each conclude without
|
||||
knowing about the other. Harnesses run side by side up to the profile's limit.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from one_memory.model_profiles import ModelCapabilities, ModelProfilePatch
|
||||
from tests.test_memory_processing import FakeBridge, _put, _runner, _setup
|
||||
|
||||
#: Long enough that an overlap is unmistakable, short enough to stay a unit test.
|
||||
MODEL_SECONDS = 0.25
|
||||
|
||||
|
||||
class WatchedBridge(FakeBridge):
|
||||
"""A model that takes a moment, and says who was inside it when.
|
||||
|
||||
Overlap is recorded rather than inferred from timings: every call notes
|
||||
which conversations were already in the model when it arrived.
|
||||
"""
|
||||
|
||||
def __init__(self, seconds: float = MODEL_SECONDS) -> None:
|
||||
super().__init__(outputs=())
|
||||
self.seconds = seconds
|
||||
self.inside: set[str] = set()
|
||||
self.peak = 0
|
||||
self.overlaps: list[tuple[str, ...]] = []
|
||||
self.same_conversation_overlaps: list[str] = []
|
||||
self._guard = threading.Lock()
|
||||
|
||||
async def _spend_a_moment(self, request) -> None:
|
||||
conversation = getattr(request, "session_id", None) or "?"
|
||||
with self._guard:
|
||||
if self.inside:
|
||||
self.overlaps.append(tuple(sorted(self.inside)))
|
||||
if conversation in self.inside:
|
||||
self.same_conversation_overlaps.append(conversation)
|
||||
self.inside.add(conversation)
|
||||
self.peak = max(self.peak, len(self.inside))
|
||||
try:
|
||||
await asyncio.sleep(self.seconds)
|
||||
finally:
|
||||
with self._guard:
|
||||
self.inside.discard(conversation)
|
||||
|
||||
async def ingest(self, request):
|
||||
await self._spend_a_moment(request)
|
||||
return await super().ingest(request)
|
||||
|
||||
async def ingest_extracted(self, request, contents):
|
||||
await self._spend_a_moment(request)
|
||||
return await super().ingest_extracted(request, contents)
|
||||
|
||||
|
||||
def _conversations(
|
||||
evidence,
|
||||
*,
|
||||
count: int,
|
||||
turns: int = 1,
|
||||
sources: tuple[str, ...] = ("cursor",),
|
||||
) -> None:
|
||||
for conversation in range(count):
|
||||
source = sources[conversation % len(sources)]
|
||||
for turn in range(turns):
|
||||
evidence.put(
|
||||
EvidenceInput(
|
||||
source_id=source,
|
||||
message_id=f"c{conversation}-t{turn}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim=f"第 {conversation} 段对话的第 {turn} 句。",
|
||||
conversation_id=f"conversation-{conversation}",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
)
|
||||
evidence.put(
|
||||
EvidenceInput(
|
||||
source_id=source,
|
||||
message_id=f"c{conversation}-t{turn}-answer",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
verbatim=f"第 {conversation} 段对话第 {turn} 句的回复。",
|
||||
conversation_id=f"conversation-{conversation}",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
async def _drain(supervisor, tasks, *, expected: int, timeout: float = 60.0) -> float:
|
||||
started = time.monotonic()
|
||||
supervisor.backfill()
|
||||
deadline = started + timeout
|
||||
while time.monotonic() < deadline:
|
||||
recorded = tasks.list()
|
||||
if len(recorded) >= expected and all(
|
||||
task.status is MemoryTaskStatus.COMPLETED for task in recorded
|
||||
):
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
else: # pragma: no cover - only on a real stall
|
||||
unfinished = [
|
||||
(t.task_id, str(t.status))
|
||||
for t in tasks.list()
|
||||
if t.status is not MemoryTaskStatus.COMPLETED
|
||||
]
|
||||
await supervisor.shutdown()
|
||||
raise AssertionError(
|
||||
f"{expected} expected, {len(tasks.list())} recorded, still unfinished after "
|
||||
f"{timeout}s: {unfinished}. A drain that gives up quietly would let every "
|
||||
"measurement below be taken off a run that never finished."
|
||||
)
|
||||
elapsed = time.monotonic() - started
|
||||
await supervisor.shutdown()
|
||||
return elapsed
|
||||
|
||||
|
||||
def _supervise(tmp_path, stores, bridge, **kwargs):
|
||||
evidence, profiles, tasks, governance, secrets, _profile = stores
|
||||
return MemoryProcessingSupervisor(
|
||||
tasks=tasks,
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=governance,
|
||||
data_dir=tmp_path,
|
||||
runner_factory=lambda: _runner(tmp_path, stores, bridge),
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
def test_two_turns_of_one_conversation_are_never_in_the_model_together(tmp_path) -> None:
|
||||
"""HY cannot tell what supersedes what if it is told both at once."""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
for turn in range(4):
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"same-{turn}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content=f"同一段对话的第 {turn} 句,说的是同一件事的不同版本。",
|
||||
conversation_id="the-one-conversation",
|
||||
)
|
||||
bridge = WatchedBridge()
|
||||
|
||||
async def exercise() -> None:
|
||||
# Four workers on purpose: with room to overlap, only the scheduler's
|
||||
# own rule can keep these apart.
|
||||
supervisor = _supervise(tmp_path, stores, bridge, max_concurrent_tasks=4)
|
||||
await _drain(supervisor, tasks, expected=1)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert bridge.requests, "nothing reached the model, so this run proves nothing"
|
||||
assert bridge.same_conversation_overlaps == [], (
|
||||
f"{len(bridge.same_conversation_overlaps)} extractions from the same "
|
||||
"conversation were inside the model at the same time; HY decides "
|
||||
"supersede from what it has already been told, so neither could see "
|
||||
"the other's conclusion"
|
||||
)
|
||||
|
||||
|
||||
def test_separate_harnesses_run_side_by_side(tmp_path) -> None:
|
||||
"""Harness ordering does not serialize unrelated sources."""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
_conversations(evidence, count=8, sources=("cursor", "codex"))
|
||||
bridge = WatchedBridge()
|
||||
|
||||
async def exercise() -> None:
|
||||
# No max_concurrent_tasks: whatever the profile says is what should
|
||||
# happen, and nobody in the product passes one.
|
||||
supervisor = _supervise(tmp_path, stores, bridge)
|
||||
await _drain(supervisor, tasks, expected=8)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert bridge.peak > 1, (
|
||||
f"two separate harnesses were extracted {bridge.peak} at a time; "
|
||||
"the scheduler is still working through harnesses one after another"
|
||||
)
|
||||
|
||||
|
||||
def test_durable_drain_and_scheduler_share_one_source_lock(tmp_path) -> None:
|
||||
"""A direct durable write cannot overlap the scheduler for one source."""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, profiles, tasks, governance, secrets, _profile = stores
|
||||
_put(
|
||||
evidence,
|
||||
message_id="scheduled",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="后台调度中的 WorkBuddy 对话。",
|
||||
conversation_id="scheduled-conversation",
|
||||
source_id="workbuddy",
|
||||
)
|
||||
_put(
|
||||
evidence,
|
||||
message_id="scheduled-answer",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="后台调度中的 WorkBuddy 回复。",
|
||||
conversation_id="scheduled-conversation",
|
||||
source_id="workbuddy",
|
||||
)
|
||||
with sqlite3.connect(tmp_path / "one-memory.sqlite3") as database:
|
||||
database.executescript(
|
||||
"""
|
||||
CREATE TABLE source_grants (
|
||||
user_id TEXT NOT NULL,
|
||||
source_id TEXT NOT NULL,
|
||||
root_path TEXT NOT NULL,
|
||||
candidate_kind TEXT NOT NULL,
|
||||
granted_at TEXT NOT NULL,
|
||||
revoked_at TEXT,
|
||||
PRIMARY KEY (user_id, source_id)
|
||||
);
|
||||
INSERT INTO source_grants VALUES (
|
||||
'local','workbuddy','/authorized','default',
|
||||
'2026-10-04T00:00:00+00:00',NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
bridge = WatchedBridge(seconds=0.2)
|
||||
|
||||
async def exercise() -> None:
|
||||
scheduled_supervisor = _supervise(
|
||||
tmp_path,
|
||||
stores,
|
||||
bridge,
|
||||
max_concurrent_tasks=4,
|
||||
)
|
||||
direct_supervisor = _supervise(
|
||||
tmp_path,
|
||||
stores,
|
||||
bridge,
|
||||
max_concurrent_tasks=4,
|
||||
)
|
||||
scheduled_supervisor.backfill()
|
||||
for _ in range(100):
|
||||
if bridge.inside:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
assert bridge.inside, "scheduler never entered the model"
|
||||
direct = _put(
|
||||
evidence,
|
||||
message_id="direct",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="durable drain 同时送来的 WorkBuddy 对话。",
|
||||
conversation_id="direct-conversation",
|
||||
source_id="workbuddy",
|
||||
)
|
||||
direct_answer = _put(
|
||||
evidence,
|
||||
message_id="direct-answer",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="durable drain 同时送来的 WorkBuddy 回复。",
|
||||
conversation_id="direct-conversation",
|
||||
source_id="workbuddy",
|
||||
)
|
||||
await direct_supervisor.process_records((direct, direct_answer))
|
||||
for _ in range(100):
|
||||
if len(bridge.requests) >= 2:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
await asyncio.gather(
|
||||
scheduled_supervisor.shutdown(),
|
||||
direct_supervisor.shutdown(),
|
||||
)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert len(bridge.requests) >= 2, "both execution paths must reach the model"
|
||||
assert bridge.overlaps == [], (
|
||||
f"the durable drain and scheduler entered HY together for one source: {bridge.overlaps}"
|
||||
)
|
||||
|
||||
|
||||
def test_side_by_side_stops_at_the_profile_limit(tmp_path) -> None:
|
||||
"""The limit is the model's own, not a number invented by the scheduler."""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, profiles, tasks, _governance, secrets, profile = stores
|
||||
allowed = 3
|
||||
profile = profiles.update(
|
||||
profile.profile_id,
|
||||
ModelProfilePatch(expected_revision=profile.revision, max_concurrency=allowed),
|
||||
secrets,
|
||||
)
|
||||
# A changed profile has to be probed again before it can be the active one.
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
_conversations(
|
||||
evidence,
|
||||
count=8,
|
||||
sources=("cursor", "codex", "claude-code", "workbuddy"),
|
||||
)
|
||||
bridge = WatchedBridge()
|
||||
|
||||
async def exercise() -> None:
|
||||
supervisor = _supervise(tmp_path, stores, bridge)
|
||||
await _drain(supervisor, tasks, expected=8)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert bridge.peak > 1, "nothing ran side by side, so the ceiling is untested"
|
||||
assert bridge.peak <= allowed, (
|
||||
f"the profile allows {allowed} at a time and {bridge.peak} were in the model together"
|
||||
)
|
||||
|
||||
|
||||
def test_a_queue_that_is_entirely_blocked_stops_starting_workers(tmp_path) -> None:
|
||||
"""Serialising by conversation must not turn into a spin.
|
||||
|
||||
A worker that finds nothing to take exits, and a worker exiting is what
|
||||
starts the next one. So a queue that is full but entirely blocked on the
|
||||
one conversation already running will start workers forever unless the
|
||||
question asked before starting one is "is there anything runnable",
|
||||
not "is the queue non-empty".
|
||||
|
||||
The bound is deliberately loose: the difference between correct and
|
||||
wrong here is a handful against thousands, not a tuned number.
|
||||
"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
turns = 6
|
||||
_conversations(evidence, count=1, turns=turns)
|
||||
bridge = WatchedBridge(seconds=0.05)
|
||||
starts = 0
|
||||
|
||||
async def exercise() -> None:
|
||||
nonlocal starts
|
||||
supervisor = _supervise(tmp_path, stores, bridge)
|
||||
start_one = supervisor._start_worker # noqa: SLF001 - counting the thing under test
|
||||
|
||||
def counted() -> None:
|
||||
nonlocal starts
|
||||
starts += 1
|
||||
start_one()
|
||||
|
||||
supervisor._start_worker = counted # noqa: SLF001
|
||||
await _drain(supervisor, tasks, expected=turns)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert starts > 0, "no worker was ever started, so the count proves nothing"
|
||||
assert starts <= 10 * turns, (
|
||||
f"{starts} workers were started for {turns} pieces of work, all of one "
|
||||
"conversation; each one found nothing to take and its exit started the next"
|
||||
)
|
||||
|
||||
|
||||
def test_nothing_queued_is_lost_when_the_core_stops(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
_conversations(evidence, count=8)
|
||||
bridge = WatchedBridge(seconds=0.5)
|
||||
|
||||
async def exercise() -> None:
|
||||
supervisor = _supervise(tmp_path, stores, bridge)
|
||||
supervisor.backfill()
|
||||
deadline = asyncio.get_running_loop().time() + 10.0
|
||||
try:
|
||||
while not tasks.list():
|
||||
if asyncio.get_running_loop().time() >= deadline:
|
||||
raise AssertionError(
|
||||
"backfill recorded no task within 10s; an unbounded wait "
|
||||
"would hang the complete Core release gate"
|
||||
)
|
||||
await asyncio.sleep(0.01)
|
||||
# Stop in the middle, which is what closing the window does.
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
recorded = tasks.list()
|
||||
assert recorded, "no task was ever recorded, so nothing was at risk"
|
||||
stranded = [
|
||||
task for task in recorded if task.status not in {MemoryTaskStatus.COMPLETED, *_CLAIMABLE}
|
||||
]
|
||||
assert stranded == [], (
|
||||
f"{len(stranded)} tasks were left in a state no later run will pick up: "
|
||||
f"{[(t.task_id, str(t.status)) for t in stranded]}"
|
||||
)
|
||||
|
||||
|
||||
_CLAIMABLE = {
|
||||
MemoryTaskStatus.PENDING,
|
||||
MemoryTaskStatus.WAITING_FOR_MODEL,
|
||||
MemoryTaskStatus.FAILED,
|
||||
MemoryTaskStatus.RUNNING,
|
||||
}
|
||||
|
||||
|
||||
def test_running_several_at_once_never_reports_the_database_locked(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
_conversations(evidence, count=12)
|
||||
bridge = WatchedBridge(seconds=0.05)
|
||||
|
||||
async def exercise() -> None:
|
||||
supervisor = _supervise(tmp_path, stores, bridge)
|
||||
await _drain(supervisor, tasks, expected=12)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
refused = [
|
||||
(task.task_id, task.last_error_code)
|
||||
for task in tasks.list()
|
||||
if task.last_error_code and "lock" in task.last_error_code.lower()
|
||||
]
|
||||
assert refused == [], (
|
||||
f"extraction was refused the database while running side by side: {refused}"
|
||||
)
|
||||
assert all(task.status is MemoryTaskStatus.COMPLETED for task in tasks.list()), (
|
||||
"not every task finished, so a locked database could simply not have been reached yet"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("workers", [1, 4])
|
||||
def test_no_database_connection_is_held_while_the_model_thinks(tmp_path, workers) -> None:
|
||||
"""Waiting on a model must not cost anyone a connection.
|
||||
|
||||
Asked of the waiting thread itself, not of the database as a whole: with
|
||||
several workers, another one being mid-write while this one waits is
|
||||
exactly what is supposed to happen, and a process-wide reading calls that
|
||||
a fault. What must not happen is the thread that is about to sit on an
|
||||
`await` still holding a checkout of its own — the loop thread does every
|
||||
model call, so one held there is one nobody else can have for as long as
|
||||
the model takes.
|
||||
|
||||
Checked at both settings because holding one only costs something when
|
||||
there is someone else to take it from.
|
||||
"""
|
||||
from one_memory.database_access import main_database
|
||||
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, _profile = stores
|
||||
_conversations(evidence, count=4)
|
||||
access = main_database(tmp_path / "one-memory.sqlite3")
|
||||
held: list[tuple[bool, int]] = []
|
||||
|
||||
class Sampling(WatchedBridge):
|
||||
async def _spend_a_moment(self, request) -> None:
|
||||
me = threading.get_ident()
|
||||
held.append(
|
||||
(
|
||||
access._writer_owner == me, # noqa: SLF001 - the point of the test
|
||||
access._reader_holders.get(me, 0), # noqa: SLF001
|
||||
)
|
||||
)
|
||||
await super()._spend_a_moment(request)
|
||||
|
||||
bridge = Sampling(seconds=0.1)
|
||||
|
||||
async def exercise() -> None:
|
||||
supervisor = _supervise(tmp_path, stores, bridge, max_concurrent_tasks=workers)
|
||||
await _drain(supervisor, tasks, expected=4)
|
||||
|
||||
asyncio.run(exercise())
|
||||
|
||||
assert held, "the model was never reached, so nothing was sampled"
|
||||
assert [sample for sample in held if sample != (False, 0)] == [], (
|
||||
f"the thread about to wait on the model was still holding a checkout of its own: {held}"
|
||||
)
|
||||
Loaded 100 of 134 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user